Paper deep dive
Pixelis: Reasoning in Pixels, from Seeing to Acting
Yunpeng Zhou
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/27/2026, 1:31:55 AM
Summary
Pixelis is a vision-language agent that improves visual reasoning by operating directly on pixels through a compact set of executable tools (e.g., segment, track, OCR, zoom). It employs a three-phase training pipeline: Supervised Fine-Tuning (SFT) for tool-use grammar, Curiosity-Coherence Reward Fine-Tuning (CC-RFT) for structured exploration, and Pixel Test-Time RL (TTRL) for label-free adaptation using trajectory voting and KL-to-EMA safety controls. The system achieves consistent performance gains across six benchmarks while producing shorter, auditable toolchains.
Entities (5)
Relation Signals (4)
Pixelis → trainedvia → SFT
confidence 100% · Pixelis trains in three phases: (1) Supervised Fine-Tuning
Pixelis → trainedvia → CC-RFT
confidence 100% · Pixelis trains in three phases: (2) Curiosity-Coherence Reward Fine-Tuning
Pixelis → trainedvia → Pixel TTRL
confidence 100% · Pixelis trains in three phases: (3) Pixel Test-Time RL
Pixelis → usesbackbone → Qwen3-VL-8B-Instruct
confidence 95% · Backbone: Qwen3-VL-8B-Instruct
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Most vision-language systems are static observers: they describe pixels, do not act, and cannot safely improve under shift. This passivity limits generalizable, physically grounded visual intelligence. Learning through action, not static description, is essential beyond curated data. We present Pixelis, a pixel-space agent that operates directly on images and videos via a compact set of executable operations (zoom/crop, segment, track, OCR, temporal localization) and learns from its consequences. Pixelis trains in three phases: (1) Supervised Fine-Tuning learns a pixel-tool grammar from Chain-of-Thought-Action traces with a masked imitation loss that upweights operation/argument tokens and auxiliary heads to stabilize pixel-grounded arguments; (2) Curiosity-Coherence Reward Fine-Tuning optimizes a dual-drive objective marrying prediction-error curiosity with adjacent-step coherence and a mild efficiency prior under a KL anchor, yielding short, valid, structured toolchains; (3) Pixel Test-Time RL performs label-free adaptation by retrieving neighbors, voting over complete trajectories rather than answers, and updating toward short, high-fidelity exemplars while constraining drift with a KL-to-EMA safety control. Across six public image and video benchmarks, Pixelis yields consistent improvements: the average relative gain is +4.08% over the same 8B baseline (peaking at +6.03% on VSI-Bench), computed as (ours-baseline)/baseline, while producing shorter, auditable toolchains and maintaining in-corridor KL during test-time learning. Acting within pixels, rather than abstract tokens, grounds multimodal perception in the physical world, linking visual reasoning with actionable outcomes, and enables embodied adaptation without external feedback.
Tags
Links
- Source: https://arxiv.org/abs/2603.25091v1
- Canonical: https://arxiv.org/abs/2603.25091v1
Trouble viewing inline? Open PDF directly →
Full Text
120,921 characters extracted from source content.
Expand or collapse full text
Pixelis: Reasoning in Pixels, from Seeing to Acting Yunpeng Zhou University of Reading, Reading, UK kc804139@student.reading.ac.uk Abstract Most vision-language systems are static observers: they describe pixels, do not act, and cannot safely improve un- der shift. This passivity limits generalizable, physically grounded visual intelligence. Learning through action, not static description, is essential beyond curated data. We present Pixelis, a pixel-space agent that operates directly on images and videos via a compact set of executable operations (zoom/crop, segment, track, OCR, temporal localization) and learns from its consequences. Pixelis trains in three phases: (1) Supervised Fine-Tuning learns a pixel-tool gram- mar from Chain-of-Thought-Action traces with a masked imitation loss that upweights operation/argument tokens and auxiliary heads to stabilize pixel-grounded arguments; (2) Curiosity-Coherence Reward Fine-Tuning optimizes a dual-drive objective marrying prediction-error curiosity with adjacent-step coherence and a mild efficiency prior under a KL anchor, yielding short, valid, structured toolchains; (3) Pixel Test-Time RL performs label-free adaptation by retrieving neighbors, voting over complete trajectories rather than answers, and updating toward short, high-fidelity ex- emplars while constraining drift with a KL-to-EMA safety control. Across six public image and video benchmarks, Pix- elis yields consistent improvements: the average relative gain is +4.08% over the same 8B baseline (peaking at +6.03% on VSI-Bench), computed as (ours-baseline)/baseline, while producing shorter, auditable toolchains and maintaining in- corridor KL during test-time learning. Acting within pixels, rather than abstract tokens, grounds multimodal perception in the physical world, linking visual reasoning with action- able outcomes, and enables embodied adaptation without external feedback. 1. Introduction Large multimodal models excel at describing pixels yet remain passive observers. They neither act on visual evi- dence nor learn from their interactions, breaking the per- ception–action loop and allowing localization, reading, and timing errors to persist under domain shift. This work in- troduces pixel-space agents that execute verifiable tools and improve safely at test time, closing this loop by grounding each reasoning step in direct visual evidence. We present Pixelis, a vision–language agent that composes auditable toolchains and refines them online. Its three-stage pipeline begins with Supervised Fine-Tuning (SFT) to learn tool-use syntax from Chain-of-Thought–Action traces. Next, Curiosity–Coherence RFT (C-RFT) balances visual explo- ration with an adjacent-step coherence prior that discourages erratic tool switching without stifling longer-horizon rea- soning. Finally, Pixel TTRL enables safe online adaptation by voting over full reasoning trajectories: rather than tal- lying only final answers, it selects behaviorally consistent exemplars to prevent drift from accumulated errors. Reasoning in pixel space raises three challenges. Ex- ecutability: map decisions to typed, replayable tool calls. Structure: prefer concise, logically ordered steps over chaotic exploration. Safety: ensure stable, auditable updates under shift. Pixelis addresses each directly. Executability comes from a compact tool interface with serialized arguments. Structure is induced by coherence on step embeddings that favors locally adjacent effects and discourages gratuitous detours. Safety is provided by turning high-confidence trajec- tories into supervision, while a KL-to-EMA corridor acts as a policy compass, penalizing updates that veer too far from a slow, stable reference so refinement stays purposeful rather than oscillatory. These elements reinforce one another. Adjacent-step co- herence shortens plans, which makes trajectory-level voting more reliable; in turn, Pixel TTRL can update aggressively while keeping token-level KL inside a corridor. At 8B scale across six benchmarks, Pixelis improves accuracy while short- ening chains from≈6 to 3.7 steps on average and keeping token-KL<0.2; removing safety pushes KL>0.4 with accu- racy collapse. To attribute gains to actions rather than text- only heuristics, we report tool fidelity (IoU/ANLS/HOTA), include tool-ablated controls and tool-dependent subsets, and audit de-duplication and retrieval leakage. Our evaluation is process-oriented.We introduce RaPR/RaCPR to measure valid tool use and coherent com- position, analyze risk–coverage for adaptation with matched arXiv:2603.25091v1 [cs.CV] 26 Mar 2026 Images Video Frames b1.SFT Policy Model Tool ToolReward Tools and Initial CoTA Frames Images Query Prompts Reference Model KL Divergence GRPO PPO Sampled Candidate Responses Policy Gradient Update Final Answer Coherence Penalty Curiosity Composite Reward (c) Test Time RL Majority Voting Answer Trajectory Q: Given the query: "What is the color of the shirt the person is wearing in this video?", retrieve the most relevant prior experience from your memory bank. Analyze the video frame sequence to locate the person's shirt. Use color detection tools to identify the shirt’s color from the frame. c1.Unlabeled Data (b) Training Reward Policy Optimization (a) Data b2.RFT Online Training Loop Fine-Tuned VLMs input image/video You are a pixel reasoner. You will call various functions to assist with the user queries. You should use functions within <tools>. "function": "name": "X", "description": "Y", "parameters": "properties": "bbox": "description": "Z","frame": "description": "Q","OCR": "description": "W" Tool- Use Trajectory Policy Model Base VLMs Tools (Functions) c1.Scaling+ Training Reasoning CoT Images Frames Images Video Frames Pixel Data Action Trajectory Thought Thought Thought Action Tools with Functions Images or Video Frames Question Answer pixel size ocr token property position bbox coord time frame type label Detail Perception Contextual Reading Geometric Reasoning Temporal Localization Tasks for Queries Spatio Temporal Tracking Unlabeled Data Voting pixel reasoning Pixel Data (Extra) Trajectory Analyze the video, highlighting the key components of the person's actions. Pick the most suitable action: (A) Putting something behind something (B) Pushing something from left to right (C) Pushing something so that it slightly moves (D) Plugging something into something Answer Hidden Data Data Optimization RFT SFT Online Training Loop Reward VLM VLM VLM KL-to-EMA safety Figure 1. Pixelis uses executable pixel tools to act on images and videos. SFT learns tool syntax; C-RFT shapes exploration with curiosity, coherence, and efficiency; Pixel TTRL adapts online via retrieval and trajectory voting under KL/EMA safety, producing shorter, auditable toolchains. budgets, and separate pre/post scores under shift. Adversarial- majority stress tests show that uncertainty-weighted consen- sus with abstention keeps KL in-corridor and improves accuracy where hard majority drifts. A de-duplication and leakage audit confirms index whitelisting and low residual overlap, supporting fair comparison to static baselines. Contributions: Pixelis unifies pixel-space acting with a small set of executable operations composed into short, replayable, verifiable toolchains that reveal how answers are obtained; introduces C-RFT, coupling prediction-error curiosity with adjacent-step coherence and an efficiency prior to avoid noise-seeking while keeping plans compact; and proposes Pixel TTRL, a trajectory-level behavioral voting scheme that supplies reliable test-time supervision with- out a process critic, safeguarded by KL-to-EMA drift con- trol. Together these components deliver consistent 8B-scale gains across six benchmarks while producing shorter chains, maintaining in-corridor KL during online adaptation, and clarifying when and why pixel tools—rather than text-only heuristics—drive the improvements, with released configs, checkpoints, replay logs, and scripts enabling end-to-end replication, ablations, and audited updates. 2. Related Work Our work connects tool-augmented VLMs, RL for struc- tured visual reasoning, and online test-time adaptation. Prior systems typically excel in subsets of this space—pixel ground- ing, structured exploration, or safe updates—yet rarely unify all three. We pursue this combination via a compact, au- ditable pixel toolchain with curiosity–coherence training and KL-anchored trajectory voting, enabling verifiable stepwise execution and explicit chain-length control. Tool-augmented VLMs. External tools expand model ca- pability and tool-grounded intermediates [7,40,50]. Early agents (Visual ChatGPT, M-REACT) orchestrate vision APIs with limited stepwise verification [68,71]. Visual Sketchpad makes intermediate reasoning visible through sketch editing [19]. PixelLM performs internal mask genera- tion via a codebook, removing external APIs but not avoiding execution [49]. PixelWorld (arXiv 2025) standardizes pixel- level evaluation but does not prescribe tool design [39]. AnyTool scales API retrieval yet omits test-time behavioral consistency [9]. Recent work in autonomous driving explores tool-augmented VLM agents and vision–language–action models: AgentThink unifies chain-of-thought with dynamic tool invocation for driving scenarios [46], while surveys and frameworks for vision–language–action driving agents and interactive traffic generation [22] emphasize structured per- ception–action interfaces rather than pixel-audited toolchains. Our setting differs by targeting short, auditable pixel chains aligned with VTG-style localization [66] and furnishing replayable execution traces. RL for structured reasoning. RLHF and sequence-level RL enable trajectory learning, and recent reason-tuning couples RL with chain supervision [6,37,55]. Self-consistency [61] and constitutional variants rely on costly process signals [5, 32,73]. Intrinsic motivation encourages exploration [45]. Step-level coherence has been studied in language models and robotics [43], but applying adjacent-step coherence to pixel-space tool chains remains underexplored. C-RFT introduces via z-scored cosine similarity on step embeddings, building structured plans without per-step critics. Test-time adaptation. Entropy-minimization (TENT), TTS (Test Time Scaling) and continual TTA (CoTTA, RoTTA) update without pixel evidence and can drift [36,54,57, 58,76]. For VLMs, online and adapter-based TTA show similar stability–performance issues [10,26]. TTRL casts this as test-time reinforcement learning using voting-derived pseudo-rewards [80]. KL penalties are standard in RL and TTA [31,51,62]; we adapt them via a slow-moving EMA anchor to stabilize non-stationary updates. VIMA [23] couples perception and action in robot control, whereas Pixelis targets semantic VLM reasoning with verifiable tool outputs. All comparisons use backbone-matched controls and the same KL corridor. 3. Method 3.1. Data Pixelis trains on Chain-of-Thought–Action (CoTA) trajecto- ries that log how pixel tools are chosen and updated. Scope and collection. We build a corpus of CoTA trajectories interleaving reasoning with executable pixel tools. Each step selects a toolSEG, TRK, OCR, ZOOM, TEMP, PROP, sets arguments, and records the new state. Trajectories come from a tool-constrained teacher using the Grok 4 VLM (API, 2025 Oct, temp. 0.7) [69]with logit-biased syntax. To reduce tool bias, we stratify by task, tool, and temporal span and schedule difficulty via a small SFT hardness predictor. We drop duplicates, license conflicts, and visible watermarks. Validation and acceptance. Operator checks include IoU (SEG/TRK), edit distance (OCR), and temporal consistency (TEMP). We compute a trajectory score 푆(휏) = 훼푆 logic + 훽푆 struct + 훾푆 visual , 휏 ∈ D ⇐⇒ (all ops pass)∧(푆(휏) ≥ 휏 0 ). with dev-tuned defaults훼=0.4,훽=0.3,훾=0.3,휏 0 =0.65; com- ponents are z-scored per task to balance magnitudes. A deci- sive step is a succeeded call that changes the intermediate state or final answer; decisive-step rates are computed over succeeded calls per split. Core statistics. Images (train/dev/test): 80k/8k/8k; Videos: 28k/2k/2k. CoTA trajectories: 76k with 5.8 steps on average, longer than online chains. Decisive-step rate (train/dev/test): 0.86/0.87/0.86, defined as the rates computed over succeeded calls that are marked decisive (high-confidence, answer- contributing) under our thresholds. Mean clip duration: 11.8 s±6.4.Per-tool usage (train/dev/test, %): SEG 24.1/23.8/24.3, ZOOM 17.1/17.8/18.0, TRK 22.9/22.3/22.1, OCR 20.2/20.0/19.9, TEMP 10.0/9.7/9.9, PROP 5.9/6.2/5.8. De-duplication and leakage control. Near-duplicates are filtered at image and clip levels via two-stage checks (vi- sual embeddings + perceptual hashes); video checks run on uniformly sampled keyframes (1 fps). For Pixel TTRL, candidate neighbors flagged by the de-dup filters are masked from the retrieval index, and evaluation media are excluded at ingest; an audit against evaluation whitelists reports 0.00% overlap (95% CI [0.00, 0.02]). We release metadata, valida- tors, and replay logs for reproducibility. Error analysis. On a 1,500-sample stratified audit, each tool invocation is treated as a binary pass/fail event under the same thresholds as the runtime study. 95% CIs. Failure rates are: SEG 11.4% (IoU< 0.5), TRK 16.9% (HOTA< 0.15), OCR 7.5% (ANLS< 0.85after NFKC + punctuation stripping), TEMP 17.3% (tIoU< 0.5or offset> 0.5s), and PROP 9.7% (attribute mismatch). Around 3% of audited calls show annotator disagreement; even if all such cases are counted as failures, each rate shifts by less than 1 p. 3.2. Training Pipeline Pixelis trains a vision–language agent to reason, act, and adapt in pixel space via three phases: SFT learns tool use from Chain-of-Thought–Action traces, C-RFT steers ex- ploration with curiosity+coherence under a KL anchor, and Pixel TTRL performs safe test-time adaptation by answer to behavior voting with KL/EMA stabilization. All phases share a compact tool set (segment object at, get properties, zoom in/out, select frame, read text, track object) with pa- rameterized calls and replayable traces. This action interface complements universal embodied spaces [79], improving pixel-level alignment and reproducibility. Callable tools: SEG, ZOOM, TRK, OCR, TEMP, PROP; each outputs typed arguments and replayable artifacts. Frozen evaluators (for scoring only) include HOTA/CLEAR-MOT (TrackEval), ANLS, and DINOv2 [38, 42], trying DINOv3 [53]. 3.2.1. Phase 1: Supervised Fine-Tuning (SFT) Goal/data interface. Each example has an image/video, a prompt, and a CoTA trace with intermixed thought tokens and tool calls (op name + typed arguments). Tool outputs (boxes, GRPO PPO Online Training Loop Visual Reasoning COTA Data Initial Policy Model VLM Reward Final Answer Coherence Penalty Curiosity Reference Model Majority Voting Answer Trajectory Reward Images Video Frames Policy Model Typed Tasks Queue Trajectory Question Prompts Images Video Frames Sampled Candidate Responses Policy Gradient Update Composite Reward Offline Training Fine-Tuned VLMs Policy Optimization SFTCC-RFTPixel TTRL Defined Tools Figure 2. Three-phase training of Pixelis. SFT learns a tool-use grammar; C-RFT shapes exploration with curiosity, coherence, and a light efficiency prior; Pixel TTRL adapts at test time by retrieving neighbours and updating toward behaviourally consistent trajectories under a KL-to-EMA safety constraint, turning raw tool traces into shorter, structured pixel toolchains. masks, tracklets, OCR spans, temporal marks) are serialized back as short, structured tokens (normalized coordinates at 10 −2 ; subword OCR tiles) so the model conditions on its own actions. Masked imitation loss. SFT uses a standard next-token cross-entropy loss, but we upweight tokens that belong to tool names and arguments (weight푤 act > 1, default2) so that mistakes on actions matter more than mistakes on plain text. We still train with teacher forcing, and apply5%feedback dropout that randomly hides previous tool outputs so the model learns to handle missing or noisy history. Tool-argument auxiliary loss (on decisive stepsD). L tool = 1 |D| ∑︁ 푡∈D h 휆 box SmoothL1( ˆ 푏 푡 , 푏 푡 )+ 휆 text CE( ˆ푦 푡 , 푦 푡 ) + 휆 mask 1− Dice( ˆ 푀 푡 , 푀 푡 ) + 휆 temp BCE( ˆ푢 푡 ,푢 푡 ) i (1) with lightweight heads (single-layer MLP, hidden 256) active only when the tool applies, receiving pooled visual features and tool confidences. The architectural details of the dynamics and auxiliary heads are in Appendix S2. Total SFT loss: L SFT,tot =L SFT + 훼 tool L tool , 훼 tool =0.2(≪ 1).(2) Curriculum. One warm-up epoch on the full pool, then medium:hard sampling2:1with hardness binned by tertiles of SFT-predicted error; we advance when EMA-smoothed dev loss퐿 푘 satisfies퐿 푘+1 ≤ 1.05 퐿 푘 . Regularization: label smoothing 0.05, grad-norm cap 1.0, and 2% dropout on early actions (푡≤3) during teacher forcing. SFT exports휋 SFT plus token normalizers (running mean/variance over log- probs and entropies) and fixed per-tool scalers that reweight curiosity and penalty terms in C-RFT/TTRL. 3.2.2. Phase 2: Curiosity–Coherence RFT (C-RFT) Objective. For a rollout 휏 = (푠 0 ,푎 0 ,..., 푠 푇 ) we maximize 푅(휏) = 푤 1 푅 final + 푤 2 푅 cur + 푤 3 푅 coh − 푤 4 푅 pen ,(3) where푅 final ∈+1,−1and푅 pen penalizes invalid actions and chains longer than퐿 0 . Curiosity rewards states where the tool-conditioned dynamics head predicts a different next visual state than the current policy, but weights each step by an uncertainty gate so that large prediction errors only count when epistemic variance is low. Coherence sums standard- ized adjacent cosines of step embeddings to discourage tool hopping:푅 coh (휏) = Í 푇 푡=1 zscore cos(퐸 푡 , 퐸 푡−1 ) . We tune 푤 푖 on the dev split; intrinsic terms are batch-wise z-scored to balance magnitudes. Step embeddings퐸 푡 . Visual tokens푣 푡 are masked-pooled at layer퐿 m ;푥 푡 mean-pools recent thoughts and serialized tool outputs (with confidences when available);onehot(푎 푡−1 ) encodes the previous action. We project to unit norm 퐸 푡 = 푔 휙 ( [ 푣 푡 ∥ 푥 푡 ∥ onehot(푎 푡−1 )] ) 푔 휙 ( [ 푣 푡 ∥ 푥 푡 ∥ onehot(푎 푡−1 )] ) 2 ,dim(퐸 푡 ) = 512. (4) and stop gradients into backbone tokens during C-RFT/TTRL (only푔 휙 updates). Adjacent cosines are z-scored on the dev split per task family and reused at test time. RaCPR uses퐸 푡 only for training diagnostics; eval- uation uses a frozen external encoder퐸 ext (DINOv2) on footprint crops. Policy update. We use a GRPO-style policy gradient [52] with a KL penalty to the SFT policy. For each prompt, we keep a small group of top-퐾trajectories (퐾=8), compute group-relative scores, and increase the probability of higher- scoring trajectories. A lightweight PID controller adjusts the KL weight so that the average token-level KL stays in a narrow corridor around0.15, keeping updates effective but bounded. Temporal localization follows VTG [66]. Outcome. C-RFT produces shorter, structured, executable toolchains with higher tool fidelity and smoother transitions, without process critics or plan-level supervision; intrinsic summaries are cached as priors and retrieval keys for TTRL. 3.2.3. Phase 3: Pixel Test-Time RL (Pixel TTRL) Hybrid retrieval. For each query푞, we retrieve a neigh- borhoodN(푞)using a mixture of text and pixel signals: text keys encode the prompt and interim thoughts, and pixel keys encode tool outputs (SEG masks/crops, TRK tracklets, OCR tiles, TEMP segments). The final similarity is a convex combination of text-key and pixel-key similarities, with equal weights by default. Successful set and exemplar. For each query, we sample 푁rollouts, take the majority answerˆ푎under a low-entropy majority (< 0.2), and form a successful setSof trajectories that predictˆ푎. Among these, we choose a single exemplar trajectory that we will update toward: we prefer chains that are shorter, have higher curiosity+coherence summaries, and higher pixel fidelity VisFid(휏), using a fixed scoring combination shared across all experiments. Consensus and abstention. Instead of hard majority, we use uncertainty-weighted voting푤 푗 ∝ 푒 −퐻 푗 ·퐶푎푙 푗 ·푉푖푠퐹푖푑(휏 (푗) ), where퐻 푗 is token entropy on decisive steps and퐶푎푙 푗 is temperature calibration. Dawid–Skene (EM) treats rollouts as annotators to estimate reliability; the decision combines answer votes and behavioral alignment. If selective risk is high (conformal setCfrom entropy/disagreement with |C|>1or margin< 훿), we abstain; KL safety remains active. Pixel fidelity and pseudo-references. For each trajectory, we only score decisive tool steps (SEG/ZOOM, TRK, OCR, TEMP, PROP). At each such step we compare the tool output with a pseudo-reference drawn from other successful trajectories or retrieved neighbours, using the standard metric for that tool (IoU for boxes/masks, average IoU over tracklets, ANLS for OCR, and F1-style scores for temporal and property tools).VisFidis the average of these per-step scores, with pseudo-references chosen by simple matching rules that follow the corresponding public evaluation protocols. Behavioral similarity. Each trajectory휏is a sequence of tool calls and their pixel footprints. To compare휏with the exemplar휏 ★ , we measure how similar their tool sequences are (edit distance on action names) and how similar their foot- prints are by softly aligning steps and averaging IoU/ANLS over matched regions. We then take a weighted combination of these two terms; the exact alignment and weights are given in Appx. S6.1. Objective.Neighborhood value is an EMA over cu- riosity/coherence; values and pseudo-references are stop- gradient: 푣(N(푞)) = 1 |N(푞)| ∑︁ ℎ∈N(푞) 푣(ℎ), Pen(휏) = 훼 inv ∑︁ 푡 I[푎 푡 ∉A valid ]+ 훼 len max(0,|휏|− 퐿 0 ). (5) Let푚(푞)∈0, 1be an abstention mask (0if low-confidence consensus). With 푟(휏;휏 ★ ) = ⊮ans(휏) = ˆ푎+휅 Sim behav (휏,휏 ★ )−휆 pen Pen(휏), (6) baseline푏(푞)(EMA over neighborhood value), and advan- tage 퐴(휏,푞) = 푟(휏;휏 ★ )− 푏(푞), the loss is L =−푣(N(푞)) h 푟(휏;휏 ★ ) i log휋 휃 (휏)+ 훽 KL tok 휋 휃 ∥ 휋 EMA , (7) with휋 EMA ←휌 휋 EMA +(1−휌)휋 휃 and훽keeping token-KL within [0.10, 0.20] (target 0.15). When푚(푞)=0, gradients are masked and only EMA/KL apply, so Pixel TTRL provides test-time supervision without a process critic. Defaults and ranges for훾, top-퐾, de-dup thresholds, and gates are in Appx. S6.1. 3.3. Design Choices and Interfaces Curiosity gating suppresses spurious novelty on stochastic textures, and local coherence curbs tool hopping without plan-level critics; answer–behavior voting aligns answers with box/mask/track/OCR evidence and selects a short, high- fidelity exemplar to keep chains concise and auditable. In- terfaces: SFT to RFT via a KL anchor plus token statistics; RFT to TTRL via intrinsic summaries as priors and retrieval keys; TTRL to continual competence via value-aware updates under a KL–EMA corridor. 4. Experiments We evaluate Pixelis on six public benchmarks stressing spatial/temporal reasoning.Backbone: Qwen3-VL-8B- Instruct [47]. We organize by three questions; unlike long- video agents with semantic actions [35], we emphasize executable pixel chains and process audits. 4.1. Evaluation setup and metrics Data protocol. Test images/clips are SHA-256 de-duplicated, and a curated auditing split includes pixel references (boxes, masks, OCR spans, tracklets) with replayable traces. Retrieval-key near-duplicates are suppressed by a pHash prefilter plus CLIP-embedding NMS; same-source or highly similar keys are masked. Cross-split near-duplicate candi- dates before filtering are 0.41% (images) / 0.36% (clips); the two-stage filter removes 0.37% / 0.33%, leaving a resid- ual≈0.05% (95% CI [0.03, 0.08], bootstrap over held-out shards). For evaluation leakage, comparing test media to the TTRL index yields 0/50,216 exact overlaps and 12/50,216 near-duplicate candidates, all masked at ingest; effective overlap is 0.00% (95% CI [0, 0.007%]). Table 1. Results on public benchmarks. We compare Pixelis to answer-only self-consistency, VLM test-time adaptation methods, process-supervised baselines, and a replicated Pixel Reasoner variant for each training phase. The upper block lists proprietary, non-directly comparable models; all claims use the same 8B backbone. Scores are reported on Tool-needed subsets with matched compute, sampling budget, acceptance rate, and token-level KL;† denotes a win on the Tool-needed subset. Model Metric Size V* Bench [67] MMBench v1.1 en [34] MVBench [30] InfoVQA test [41] Video- MMMU [18] VSI Bench [70] GPT-5 (minimal)--81.469.9-61.6- Gemini-2.5-Pro (minimal)--86.682.9-79.431.4 Seed-1.5-VL [16]20B89.588.074.389.372.1- Qwen3-VL-8B-Instruct [47]8B86.485.068.783.165.359.4 Qwen3-VL-30B-A3B-Instruct [47]30B 89.5(w tools)87.072.382.068.763.2 InternVL3.5-A3B (with tools) [59]30B-84.872.1--63.7 Ours vs Baseline Pixelis (Qwen3-VL-8B-Instruct)8B90.189.573.8 † 87.9 † 69.8 † 64.4 † Pixel Reasoner (Qwen3-VL-8B-Instruct)8B88.386.969.885.866.560.1 Additional baselines — answer-level (no tools/updates) RV Self-Consistency (answer-only) [60]8B86.986.068.483.665.1- Additional baselines — VLM test-time adaptation (no tools, with adaptation) Realistic TTA of VLMs (StatA) [75]8B-86.169.582.766.2- RA-TTA (retrieval-augmented) [29]8B-86.470.684.066.8- Process-supervised (tools) in main text PRM (process reward; tools; 8B) [32, 78]8B88.987.871.186.267.662.3 Step Self-Consistency (step-level) [61]8B88.087.270.585.466.9- Late-fusion (step logits, equal-variance)8B87.1--85.667.0- Ablation Pixelis: SFT only / RFT only8B87.7/86.987.6/86.268.6/70.784.9/84.467.4/66.560.1/60.8 Pixelis: TTRL only8B86.885.969.683.566.159.6 Pixelis: SFT + RFT / SFT + TTRL 8B89.2/88.588.6/88.371.8/70.986.7/86.168.562.8 Pixelis: RFT + TTRL8B88.187.970.985.867.861.9 Task and tool metrics.We report Accuracy and ANLS (InfoVQA) plus pixel/tool fidelity when applica- ble: IoU (boxes/masks), Boundary-F1, CER for OCR, HOTA/CLEAR-MOT for tracking; temporal localization follows VTG survey guidance [66]. Process metrics (defined here). Beyond task scores, we quantify how answers are obtained. For Rate of Pixel Rea- soning (RaPR), let푢 푡 ∈0, 1indicate a valid visual–tool step at time 푡. For trajectory ℎ of length 푇 , RaPR(ℎ) = 1 푇 푇 ∑︁ 푡=1 푢 푡 ,RaPR = E ℎ [RaPR(ℎ)].(8) Validity uses external verifiers (IoU/Boundary-F1, ANLS, HOTA/CLEAR-MOT; VTG overlaps). 1 For Rate of Composite Pixel Reasoning (RaCPR), we detect coherent multi-step chains by gating adjacency of step embeddings with fixed푧-scoring. Let푐 푡 = cos 퐸 ext (휋 푡 ), 퐸 ext (휋 푡−1 ) , ̃푐 푡 = 푧(푐 푡 )(dev-split, per-task mean/STD), and푔 푡 = 1푢 푡−1 =1,푢 푡 =1, ̃푐 푡 ≥휏. Consecutive푔 푡 =1form candidates 퐶 푘 ; keep|퐶 푘 |≥퐿 min . For a qualified chain 퐶, 푞(퐶) = 1 |퐶| ∑︁ 푡∈퐶 ̃푐 푡 −휏 + −훼 len (|퐶|− 퐿 0 ) + |퐶| −훼 inv Í 푡∈퐶 (1− 푢 푡 ) |퐶| , (9) 1 TrackEval implements HOTA/CLEAR-MOT and related evaluators; we use it as a frozen scorer, not as a tracker. and we reportRaCPR(ℎ) = max 퐶 푞(퐶)andRaCPR = E ℎ [RaCPR(ℎ)]. For evaluation, cosines are computed us- ing a frozen external encoder퐸 ext (DINOv2), decoupling RaCPR from the trained step embeddings퐸 푡 . Uncertainty is reported as BCa 95% confidence intervals with Ben- jamini–Hochberg correction across tasks and hyperparameter settings(휏, 퐿 min ,훾). Answer-level strong baseline. We add a compute-matched self-consistency baseline with temperature-scaled confidence and abstention (“Ans-SSC (cal+abstain)”) in Table 1. Sensitivity and human correlation. With dev-split푧- scoring on a frozen external encoder (퐸 ext =DINOv2; we ob- tained similar trends with CLIP-L),RaCPRvaries by at most ±0.5over휏∈ [0.25, 0.40],퐿 min ∈2, 3, 4,훾∈ [0.15, 0.30], and task Accuracy varies by at most±0.2(defaults in Tab. 3). On 200 trajectories (2 annotators, 5-pt scale), Spearman휌 (BCa 95% CIs) is: RaPR0.62[0.55,0.68], RaCPR0.57 [0.49, 0.64] (BH-corrected). Fairness and compute. We re-run open models with official scripts; APIs use public protocols (rate/budget capped), reporting the best of three seeds when applicable. SFT: 3 epochs, 2:1 medium:hard. RFT: GRPO (Group-Relative Policy Optimization) (퐾=8), target token-KL≈ 0.15. TTRL: neighborhood푘=8,푁=8rollouts, EMA휌=0.99, grad-norm cap 1.0; arbitration/de-dup/decisive-step/soft-DTW follow Sec. 3.2.3 Inline settings. Online runs for 8K updates.” Table 2. De-duplication and leakage audit. We report the number of near-duplicate image/clip pairs across CoTA splits and between each split and the TTRL retrieval index; low counts indicate limited train–eval leakage. SplitCandidates (%) Removed (%)Residual, 95% bootstrap CIs (%) Train vs Dev0.41 / 0.370.36 / 0.310.05 [0.03, 0.08] / 0.05 [0.03, 0.08] Train vs Test0.38 / 0.340.36 / 0.290.04 [0.02, 0.07] / 0.04 [0.02, 0.07] Eval vs TTRL index: 0 / 50,216 exact; 12 / 50,216 candidates; 0.00% [0, 0.007%]. 06121824303642485460667278 Steps (×16) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Rate of Pixel-Space Reasoning Answer Only Answer + Curiosity Answer + Coherence Answer + Curiosity + Coherence Answer + Curiosity + Coherence + Penalty Inflection Maximum slope Saturation point 06121824303642485460667278 Steps (×16) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Rate of Composite Pixel-Space Reasoning Answer Only Answer + Curiosity Answer + Coherence Answer + Curiosity + Coherence Answer + Curiosity + Coherence + Penalty Saturation pointAcceleration point Training Dynamics of Pixel-Space Reasoning 3 seeds; Penalty: inv = 0.30, len = 0.02 Figure 3. RFT process metrics: RaPR (top) and RaCPR (bot- tom). We compare Answer Only, +Curiosity, +Coherence, +Cu- riosity+Coherence, and +Curiosity+Coherence+Penalty (Pixelis). Adding curiosity alone increases RaPR but hurts RaCPR; adding coherence and a light penalty yields the highest RaPR/RaCPR with lower variance across seeds. Bars show means over 3 seeds with 95% BCa bootstrap confidence intervals (BH-corrected). 4.2. RQ1: Does C-RFT yield structured and ex- ploratory reasoning? Under same-backbone controls, matched acceptance, and a KL-to-EMA corridor, ablating reward terms within the same GRPO setup shows: removing coherence (RFT-Curiosity) degrades chain stability (RCS, run-to-run chain stability) and tracking fidelity (MOTA) with visible tool hopping; removing curiosity (RFT-Coherence) reduces exploration (AE, action entropy) and lowers accuracy via redundant, over- local chains; the full C-RFT improves Accuracy/ANLS and lifts RaPR/RaCPR with smaller variance, indicating concise, coherent toolchains rather than trial-and-error. Fig- ure 3 reports RaPR/RaCPR dynamics; Table 1 summarizes end-point scores. Replacing adjacency-based coherence with a generic smoothness prior (L2 on consecutive arguments, stop-gradient) recovers only 31% of the RaCPR gain and underperforms Accuracy by 0.7 at matched KL, suggesting adjacency on퐸 푡 provides structure beyond generic regular- ization. If jitter appears, raise the adjacency gate휏before increasing penalties; if exploration stalls, slightly lower휏 while keeping 퐿 min (defaults in the appendix). 08162432404856647280 Steps (×24) 0.70 0.72 0.74 0.76 0.78 0.80 0.82 Accuracy (dev macro) Detail perception Contextual reading Geometric reasoning Spatio-temporal tracking Temporal localization Overall 08162432404856647280 Steps (×24) 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 KL(current EMA) Safe (KL corridor, EMA, cons) No-Safe (no KL corridor) Safe p90 (KL) TTRL: Performance vs Steps (dev) eval every 75 steps; Top: macro over 5 tasks; Bottom: KL corridor [0.10, 0.20] Figure 4. Pixel TTRL: accuracy (top) and token-KL to EMA (bottom) within corridor[0.10, 0.20]. Safe variant (value-aware retrieval, trajectory voting, EMA+KL) stays in-corridor; no-safety drifts and degrades. PID-controlled 훽 keeps KL bounded. 4.3. RQ2: Can Pixel TTRL adapt safely under un- labeled shift? Under lighting and motion shifts, with same-backbone con- trols, matched acceptance, and a KL-to-EMA corridor, Pixelis-Online improves accuracy from 73.0 to 76.5 within 8k updates (푝<0.01, two-sided permutation, BH across 6 tasks; protocol in Appx. S8) while token-KL stays near the corridor target (median≈0.16, P95≈0.19). Value-aware reweight- ing yields about1.6×more accepted updates per accuracy point. Removing KL control drives token-KL above 0.40 with accuracy collapse; dropping the EMA anchor inflates gradient variance. RaCPR increases as trajectory voting prefers short, well-structured exemplars. Under adversarial- majority stress (flipped answers or semantically similar but label-mismatched neighbors), hard-majority TTRL drifts, with accuracy decreasing and RaCPR variance increasing, whereas uncertainty-weighted consensus with abstention keeps KL well below the no-safety variant and improves accuracy. Sweeping the abstention margin훿produces a risk–coverage curve where Err@Sel drops from8.7%to 7.1%at roughly9–12%lower coverage (Tab. 4). For plan re-execution robustness (PRC) under JPEG compression 푞=10–30,±2%resize, and±1frame jitter, accuracy im- proves from 78.9% to 83.4%, with RaCPR correlating with PRC (휌=0.51 [0.44, 0.58]). Table 3. RaCPR hyperparameters shared across benchmarks. Sen- sitivity within 휏∈[0.25, 0.40], 퐿 min ∈2, 3, 4, 훾∈[0.15, 0.30]. SymbolValue SymbolValue SymbolValue 휏 (adj. gate)0.30 퐿 min (min chain) 3 퐿 0 (len prior)6 훾 (soft-DTW temp.) 0.20 훼 len (len pen.)0.02 훼 inv (inv. pen.) 0.30 4.4. RQ3: Do executable pixel tools unlock capabil- ities beyond text abstraction? Across six benchmarks with same-backbone controls, matched acceptance, and a KL-to-EMA corridor, RFT-Full outperforms RFT-Base and our re-implementation of Pixel Reasoner, and Pixelis-Online adds further gains while staying in-corridor (p<0.01, two-sided permutation, BH-corrected). Table 4. Selective adaptation on5,000shifted queries. Coverage is the answered fraction; Err@Sel is the error on that subset. Weighted voting with abstention lowers Err@Sel and KL P95 relative to hard majority. StrategyCoverage (%) Err@Sel (%) KL P95 Hard majority100.08.70.29 Weighted + abstention (훿=0.08)88.97.10.19 Hard majority (adv. 30% flipped)100.011.30.31 Weighted + abstention (adv. 훿=0.10)86.29.20.21 ZI GP ZI RT TO [14, 24], (0.3, 0.55, 0.6, 0.75), “white”, [24, 28], (0.3, 0.5, 0.6, 0.7) RaPR 0.75, RaCPR 0.71, KL 0.13, behav-sim 0.8 ZIZI (0.3, 0.6, 0.6, 0.8) RaPR 0.6 KL 0.39 Baseline (6 steps)Pixelis (3 steps - 5 steps) SOA GPTO SF ZIZI image video sequence (0.4, 0.4, 0.7, 0.6), “gold” (0.325, 0.35, 0.775, 0.65), “書店” RaPR 0.79, RaCPR 0.83, KL 0.16, behav-sim 0.65 Pred “poster” / GT “poster” acc ✓, invalid 0.21 Pred “walking people” / GT “walk” acc ✓, invalid 0.15 Pred “jump” / GT “jumping” acc ✓, invalid 0.20 Pred “running” / GT “running cat” acc ✓, invalid 0.10 Pixel Behavior [15, 24] (0.2, 0.5, 0.5, 0.7) RaPR 0.78 KL 0.39 Pred “real people” GT “poster” acc ✗, invalid 0.71 Pred “running cat” GT “cat” acc ✓, invalid 0.23 Pred “front of cat” GT “chest is white” acc ✗, invalid 0.82 Figure 5. Qualitative comparison. The baseline often loops or over-zooms on irrelevant regions, while Pixelis forms shorter, more coherent toolchains that align with the queried evidence, reflected in higher RaPR/RaCPR and VisFid. Detail Perception Contextual Reading Geometric Reasoning Temporal Localization Spatio Temporal Tracking Overall Average PR(Baseline) SFT Only RFT Only TTRL Only SFT+RFT SFT+TTRL RFT+TTRL SFT+RFT +TTRL (Ans) Pixelis Full (Ans+Trj) 0.625 0.650 0.675 0.700 0.725 0.750 0.775 0.800 0.825 0.850 0.875 0.900 Value Method × Task Accuracy (last-window macro) last 20% steps window; macro over dev; circle area accuracy Figure 6. Qualitative behaviors. Coherence regularization prevents oscillatory tool loops and encourages consistent steps, while Pixel TTRL uses retrieved trajectories to self-correct toolchains. RaPR/RaCPR and decisive-step IoU/ANLS/MOTA improve jointly, and audits show fewer ID switches, more reliable OCR-before-compare, and precise decisive chains (Fig. 6). Pixel-grounded retrieval explains the trend: pixel keys (boxes, masks, tracks, OCR tiles) align similarity with verifiable evidence, defining Visual Fidelity (VisFid) as the fraction of retrieved exemplars whose evidence matches the query foot- print. Sweeping the mixing weight휆 pix (pixel vs. semantic keys, default 0.5) yields coupled gains in VisFid and RaCPR, while휆 pix > 0.85slightly hurts cross-semantic generalization on V*Bench, suggesting a balanced mix. Answer-only self- consistency improves accuracy but cannot lift RaPR/RaCPR; VLM TTA without tools gains at test time but drifts with- out executable evidence; process-supervised tool baselines (PRM, step-level self-consistency) help locally, whereas our trajectory-level voting with KL/EMA stabilizes full chains into shorter, coherent plans and safer online updates. Table 5. Runtime by tool phase for the 8B model (medians over 1,500eval queries; latency excludes model forward). “Tool calls/ex (succ. rate%)” is the median number of successful tool invocations per example and their pass rate under fixed thresholds. Attempts also counts failed calls. ComponentLatency (s)Share (%)Tool calls/ex (succ. rate%) SEG/ZOOM (mask/bbox)1.1723.10.9 (88.6) TRK (eval pipeline)1.4228.20.7 (83.1) OCR (tiles; ANLS)0.397.71.0 (92.5) TEMP (VTG)1.5731.00.4 (82.7) PROP (property)0.163.20.7 (90.3) Retrieval+voting (TTRL)0.346.8– (n/a) Total tool-side overhead5.05100.0– End-to-end latency (incl. model fwd): p50 = 5.8 s, p90 = 8.1 s, p95 = 10.2 s; Avg. chain length (decisive steps) ̄ 퐿 = 3.7 . Chain length by subset: ̄ 퐿 Need-Tool = 4.0 , ̄ 퐿 No-Tool = 2.1 ; Tool-needed queries cover≈84% of the eval set. Notes. Full configs and per-tool breakdown with 95% CIs are in Appx. S5.2; Re- trieval+voting runs once per query and counts toward tool-side and end-to-end latency. Thresholds: SEG IoU≥0.5, TRK HOTA≥0.15, OCR ANLS≥0.85, TEMP tIoU≥0.5 4.5. Ablations and further analyses Sensitivity scans show휏trades recall and precision of com- posite chains,퐿 min suppresses short bursts, and훾tunes align- ment; defaults near휏=0.30,퐿 min =3,훾=0.20lie at the knee, with Accuracy varying within±0.2. Replacing behavioral voting with answer-only self-consistency slows adaptation and inflates RaCPR variance; hard majority is brittle under noisy neighborhoods, whereas uncertainty-weighted voting with abstention prevents catastrophic updates at negligible coverage cost. Under Gaussian noise and occlusion, RFT-Full degrades less and maintains higher RaCPR than Pixel Rea- soner; with 30% vote noise, naive majority TTRL drops−4.6 RaCPR versus−1.2for our variant; and cross-dataset transfer (train COCO/MOT-style [33,77], test LVIS-style) preserves a fraction of in-domain performance [17], with RFT-Full outperforming RFT-Base on task and process metrics. 4.6. Reproducibility We release configs for SFT/C-RFT/TTRL, checkpoints, RaPR/RaCPR scripts, and logs/replays for tool-call audits and end-to-end online updates. At 8B, total training cost is 805.2 GPU·h. Runtime is dominated by pixel tools (≈ 87%of latency, mainly segmentation and tracking); online adaptation adds≈ 0.35s median for retrieval/voting, and KL/EMA bookkeeping is < 0.1 s. 5. Conclusion and Limitations Pixelis learns to act and adapt in pixel space by composing a small set of executable tools. Its three stages play roles: SFT recovers tool syntax and usage patterns from traces; C-RFT steers exploration with adjacent-step coherence under a mild KL anchor; and Pixel TTRL carries out test-time adaptation through trajectory-level voting with KL–EMA stabilization.Together they yield shorter, more stable, and auditable toolchains while keeping adaptation drift controlled, turning pixel-grounded behavior into supervision for transparent, reproducible agents relevant to embodied AI and human–AI collaboration. Limits remain. The phases are coupled through reused statistics and KL anchoring rather than a single end-to-end objective, so residual SFT bias can persist and Pixel TTRL may under-adapt or oscillate under abrupt shift; process signals can overfit stale behaviors or chase high-entropy textures; coherence is local; and non-differentiable tools (segmentation/OCR/tracking) are brittle on thin structures, stylized fonts, dense layouts, or motion blur, with errors propagating into RaCPR. Practical mitigations include mild end-to-end regulariza- tion with adaptive KL budgets, diversity-aware replay with uncertainty gating and rarity upweighting, checks, confidence-based fallbacks, and tool-noise simulation. References [1]Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification, 2021. 9 [2] Anastasios N. Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(2):174–246, 2023. 7 [3] Karl Johan ̊ Astr ̈ om and Richard M. Murray. Pid control (chapter 6 excerpt), 2002. 7 [4]Karl Johan ̊ Astr ̈ om and Richard M. Murray. Feedback Systems: An Introduction for Scientists and Engineers. Princeton University Press, 2008. 7 [5]Yuntao Bai et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. 3 [6]Paul Christiano et al. Deep reinforcement learning from human preferences. In NeurIPS, 2017. 3 [7]Anthony C Davis, Burhan Sadiq, Tianmin Shu, and Chien- Ming Huang. Augmented vision-language models: A system- atic review, 2025. 2 [8]A. Philip Dawid and Allan M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28, 1979. 2 [9]Yilun Du, Fangxiaoyu Wei, and Hongyu Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls. In Proceedings of the International Conference on Machine Learning, 2024. 3 [10]Mario D ̈ obler, Robert A. Marsden, Tobias Raichle, and Bin Yang. A lost opportunity for vision-language models: A comparative study of online test-time adaptation for vision- language models, 2024. 3 [11]Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learn- ing, 2016. 8 [12]Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization. In CVPR, pages 2946–2953, 2013. 3 [13] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, 2017. 9 [14]Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning, pages 2151–2159. PMLR, 2019. 9 [15]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1321–1330, 2017. 2, 8, 9 [16] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1.5-vl technical report. arXiv preprint arXiv:2505.07062, 2025. 6 [17] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 8 [18]Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline pro- fessional videos. 2025. 6 [19]Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Enabling spatial reasoning in multimodal models. In CVPR, 2024. 2 [20] Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ imple- mentation details of rlhf with ppo: A case study on tl;dr summarization, 2024. 7 [21]Herv ́ e J ́ egou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128, 2011. 3, 10 [22]Sicong Jiang, Zilin Huang, Kangan Qian, Ziang Luo, Tianze Zhu, Yang Zhong, Yihong Tang, Menglin Kong, Yunlong Wang, Siwen Jiao, Hao Ye, Zihao Sheng, Xin Zhao, Tuopu Wen, Zheng Fu, Sikai Chen, Kun Jiang, Diange Yang, Seongjin Choi, and Lijun Sun. A survey on vision-language-action mod- els for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCV Workshops), 2025. 3 [23]Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anand- kumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Proceedings of the 40th International Conference on Machine Learning, 2023. 3 [24]Jeff Johnson, Matthijs Douze, and Herv ́ e J ́ egou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019. 10 [25]Jeff Johnson, Matthijs Douze, and Herv ́ e J ́ egou. Faiss: A library for efficient similarity search at scale. 2024. Overview and implementation details. 3 [26]Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision- language models, 2024. 3 [27] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (NeurIPS), 2017. 8 [28]Meelis Kull, Miquel Perell ́ o-Nieto, Markus K ̈ angsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems (NeurIPS) 32, pages 14346– 14357, 2019. Also available as arXiv:1910.12656. 2 [29]Youngjun Lee, Doyoung Kim, Junhyeok Kang, Jihwan Bang, Hwanjun Song, and Jae-Gil Lee. Ra-tta: Retrieval-augmented test-time adaptation for vision-language models. In Interna- tional Conference on Learning Representations (ICLR), 2025. 6 [30]Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195– 22206, 2024. 6 [31] Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, 133(1):31–64, 2024. 3 [32]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Ed- wards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. 3, 6 [33] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014. 8 [34]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?In European conference on computer vision, pages 216–233. Springer, 2024. 6 [35]Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou. Videomind: A chain-of-lora agent for long video reasoning, 2025. 5 [36]Ziyang Liu et al. Robust test-time adaptation against source data attacks. In ECCV, 2022. 3 [37] Ouyang Long et al. Training language models to follow instruc- tions with human feedback. arXiv preprint arXiv:2203.02155, 2022. 3 [38]Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taix ́ e, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision, pages 1–31, 2020. 3 [39]Zhiheng Lyu, Xueguang Ma, and Wenhu Chen. Pixelworld: Towards perceiving everything as pixels. arXiv preprint arXiv:2501.19339, 2025. 2 [40]Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, and Aixin Sun. SciAgent: Tool-augmented language models for scien- tific reasoning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15701–15736, Miami, Florida, USA, 2024. Association for Computational Linguistics. 2 [41]Minesh Mathew, Viraj Bagal, Rub ` en P ́ erez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. Infographicvqa. CoRR, abs/2104.12756, 2021. 6 [42] Maxime Oquab, Timoth ́ e Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023. 3 [43]Takayuki Osa, Joni Pajarinen, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7 (1-2):1–179, 2018. 3 [44]Long et al. Ouyang. Training language models to follow instructions with human feedback. In NeurIPS, 2022. 7 [45]Deepak Pathak et al. Curiosity-driven exploration by self- supervised prediction. In ICCV, 2017. 3 [46]Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Mengmeng Yang, Zheng Fu, Jinyu Miao, Yining Shi, He Zhe Lim, Li Liu, Tianbao Zhou, Hongyi Wang, Yu Huang, Yifei Hu, Guang Li, Guang Chen, Hao Ye, Lijun Sun, and Diange Yang. Agentthink: A unified framework for tool-augmented chain-of-thought reasoning in vision-language models for autonomous driving. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025. 3 [47] Qwen Team. Today, we officially launch the all-new Qwen3- VL series — the most powerful vision-language model in the qwen family to date, 2025. Accessed: 2025-09-23. 5, 6 [48]Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correla- tion analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017), 2017. 8 [49]Ziheng Ren, Zilong Huang, Yichen Wei, et al. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2 [50] Timo Schick, Jane Dwivedi-Yu, et al. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. 2 [51]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06347, 2017. 3 [52]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. 4 [53] Oriane Sim ́ eoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ̈ el Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Tim- oth ́ e Darcet, Th ́ eo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Herv ́ e J ́ egou, Patrick Labatut, and Piotr Bojanowski. Dinov3, 2025. 3 [54]Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. 3 [55] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752, 2025. 3 [56]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding, 2018. 4 [57]Dequan Wang et al. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021. 3 [58]Qin Wang et al. Cotta: Test-time adaptation for continual shift. In CVPR, 2022. 3 [59]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Zhi Hou, Haoran Hao, Tianyi Zhang, Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjun He, Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang, Huipeng Deng, Biqing Qi, Jiaye Ge, Qipeng Guo, Wenwei Zhang, Songyang Zhang, Maosong Cao, Junyao Lin, Kexian Tang, Jianfei Gao, Haian Huang, Yuzhe Gu, Chengqi Lyu, Huanze Tang, Rui Wang, Haijun Lv, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, and Gen Luo. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, 2025. 6 [60]Weiqin Wang, Yile Wang, and Hui Huang. Ranked voting based self-consistency of large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14410–14426, Vienna, Austria, 2025. Association for Computational Linguistics. 6 [61]Xuezhi Wang et al. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022. 3, 6 [62] Zixin Wang, Yadan Luo, Liang Zheng, Zhuoxiao Chen, Sen Wang, and Zi Huang. In search of lost online test-time adaptation: A survey, 2024. 3 [63]Jacob Whitehill, Paul L. Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier R. Movellan. Whose vote should count more: Opti- mal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22 (NeurIPS), pages 2035–2043, 2009. 2 [64]Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229–256, 1992. 7 [65]Laurenz Wiskott and Terrence J. Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Com- putation, 14(4):715–770, 2002. 4 [66]Jianlong Wu, Wei Liu, Ye Liu, Meng Liu, Liqiang Nie, Zhouchen Lin, and Chang Wen Chen. A survey on video temporal grounding with multimodal large language model, 2025. 3, 5, 6 [67] Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. arXiv preprint arXiv:2312.14135, 2023. 6 [68]Wenyi et al. Wu. Visual chatgpt: Talking, drawing and editing with visual foundation models.arXiv preprint arXiv:2303.04671, 2023. 2 [69]xAI. Grok 4 model card. Technical report, xAI, 2025. Last updated: August 20, 2025. 3 [70]Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei- Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. arXiv preprint arXiv:2412.14171, 2024. 6 [71]Zhenfei et al. Yang. Mm-react: Prompting chatgpt for multi- modal reasoning and action. arXiv preprint arXiv:2303.11381, 2023. 2 [72]Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lip- son. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 27 (NeurIPS 2014), pages 3320–3328, 2014. 8 [73]Weizhe Yuan et al. Self-rewarding language models. arXiv preprint arXiv:2308.06160, 2023. 3 [74]Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Pro- ceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694–699. ACM, 2002. 2 [75] MaximeZanella,Cl ́ ementFuchs,Christophe De Vleeschouwer, and Ismail Ben Ayed.Realistic test-time adaptation of vision-language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. CVPR 2025 Highlight. 6 [76]Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. A survey on test-time scaling in large language models: What, how, where, and how well?, 2025. 3 [77] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu. A simple baseline for multi-object tracking. arXiv preprint arXiv:2004.01888, 2020. 8 [78] Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, and Bowen Zhou. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504.00891, 2025. 6 [79]Jinliang Zheng, Jianxiong Li, Dongxiu Liu, Yinan Zheng, Zhihao Wang, Zhonghong Ou, Yu Liu, Jingjing Liu, Ya-Qin Zhang, and Xianyuan Zhan. Universal actions for enhanced embodied foundation models, 2025. 3 [80]Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084, 2025. 3 Pixelis: Reasoning in Pixels, from Seeing to Acting Supplementary Material 1. Step embeddings and dynamics head details Multimodal layer퐿 m and feature extraction.We compute visual features at the 3rd block from the top of the multimodal fusion stack (Qwen3-VL-8B-Instruct), i.e., after cross-attention but before final LN. Denote its visual token grid as퐹 ∈ R 퐻 ′ ×푊 ′ ×푑 with stride 16 rela- tive to the input frame. For each decisive step footprint (box/mask/track/temporal slice), we form a binary mask 푀 ∈ 0, 1 퐻 ′ ×푊 ′ by rasterizing at the token grid; for tem- poral spans we pool per-frame then average across frames. Visual summary: 푣 푡 = 1 max(1, Í 푀) ∑︁ 푖,푗 푀 푖푗 퐹 푖푗 ∈ R 푑 . We do masked mean pooling only (no max), with a3× 3 dilated kernel for thin structures when Í 푀 < 9. Token stride and pooling are fixed; no gradients pass to퐹during C-RFT/TTRL. Text/tool features푥 푡 . We concatenate: (i) the mean of last 32 thought tokens; (i) serialized tool outputs (normalized coords at10 −2 , OCR tiles hashed to subword ids then averaged); (i) per-tool confidences when available. A linear projector maps this to R 푑 . Step embedding퐸 푡 (final form). Letonehot(푎 푡−1 ) ∈ R |A| . We use a 2-layer MLP푔 휙 : R 2푑+|A| →R 512 with GELU and LayerNorm: 퐸 푡 = 푔 휙 [ 푣 푡 ∥ 푥 푡 ∥ onehot(푎 푡−1 )] 푔 휙 [ 푣 푡 ∥ 푥 푡 ∥ onehot(푎 푡−1 )] 2 ,dim(퐸 푡 ) = 512. We stop-grad into the backbone and text projector; only휙is trained in C-RFT. Cosinescos(퐸 푡+1 , 퐸 푡 )are z-scored per task family using running stats from the training split and reused at test time. KL anchor. During C-RFT and TTRL, we constrain policy updates byKL(휋 휃 ∥ 휋 SFT ) ≤ 0.2where휋 SFT is frozen as a reference and reuse the same [0.10, 0.20] corridor as in the main text. Architecture and size of푔 휙 . MLP layers:[2푑+|A|]→ 768→ 512 with GELU, dropout 0.1, LayerNorm on output. Parameter count≈ 1.2M (negligible vs backbone). We found deeper MLPs did not help under the KL budget. Tool-conditioned dynamics head (for curiosity). Given [푣 푡 ∥ 푥 푡 ∥ onehot(푎 푡 )], a small MLP predicts the next visual summary ˆ푣 푡+1 and a validity logit ˆ푟 푡 for the issued tool: ℎ 휓 : R 2푑+|A| →R 푑 × R. Loss (per step) combines feature regression and calibrated validity: L dyn = 휆 1 SmoothL1( ˆ푣 푡+1 ,푣 푡+1 )+ 휆 2 1− cos( ˆ푣 푡+1 ,푣 푡+1 ) + 휆 3 BCEWithLogits( ˆ푟 푡 ,푟 푡 ) with푟 푡 =1if the external verifier marks the step valid (IoU/ANLS/HOTA, cf. Sec. 5.1). We set(휆 1 ,휆 2 ,휆 3 ) = (0.5, 0.5, 1.0). The curiosity signal uses the prediction error 푒 푡 = 훼∥ ˆ푣 푡+1 −푣 푡+1 ∥ 1 +(1−훼) 1−cos( ˆ푣 푡+1 ,푣 푡+1 ) , 훼 = 0.5. Uncertainty gate for curiosity. We estimate epistemic uncertainty with MC dropout (4 samples,푝=0.1) at both hidden layers ofℎ 휓 ; variance overˆ푣 푡+1 gives휎 2 푡 . The gated curiosity reward is 푅 cur (푡) = 푒 푡 1+ 훽 휎 2 푡 , 훽 = 5.0, clipped to the 95th percentile per batch to avoid rare spikes. We also downweight steps where the validity head is confident negative: multiply by 휎( ˆ푟 푡 ). Training and freezing. We train푔 휙 andℎ 휓 jointly during C-RFT with AdamW (lr5× 10 −5 , wd0.01), batch size 128 steps, gradient clip 1.0. Backbone and tokenizers are frozen. During TTRL, both heads are frozen and only used to compute values and gates. Ablation knobs (reported in Appx. tables). (i) Replace adjacency cosine on퐸 푡 with L2 smoothness on arguments; (i) remove MC dropout (훽=0); (i) move퐿 m earlier/later by ±2blocks. We observe small variance across퐿 m choices, with the default giving the best RaCPR at matched KL. Sensitivity to(휏, 퐿 min , 훾). We conducted a sweep on the dev-split (3 seeds) to evaluate the impact of the adjacency gate 휏, minimum chain length퐿 min and soft-DTW temperature훾 on composite reasoning (RaCPR). The parameter ranges: 휏 ∈ 0.25, 0.30, 0.35, 0.40, 퐿 min ∈ 2, 3, 4, 훾 ∈ 0.15, 0.20, 0.25, 0.30. Across settings,RaCPRvaried within±0.6%of the default (mean at휏=0.30,퐿 min =3,훾=0.20), while Accuracy varied within±0.2%. These results confirm that the method is robust to reasonable hyper-parameter perturbations. Full per-task splits follow the same pattern and are omitted for space. Table S1. Implementation hyperparameters for퐸 푡 and the dynamics head. ItemSetting 퐿 m position3rd fusion block from top Visual poolingMasked mean; dilated 3× 3 if Í 푀 < 9 푑 (token dim)1024 (projected to 768/512 by 푔 휙 ) 푔 휙 MLP[2푑+|A|]→ 768→ 512, GELU, LN, drop 0.1 MC dropout4 samples, 푝=0.1 (dynamics head only) Dynamics head ℎ 휓 MLP[2푑+|A|]→ 1024→(푑+1) Curiosity mix 훼0.5; gate 훽=5.0 OptimizerAdamW, lr 5× 10 −5 , wd 0.01, clip 1.0 2. Voting and Calibration Details Weighted voting. For푁rollouts휏 (푗) 푁 푗=1 with candidate answers푎 (푗) , we aggregate by 푤 푗 ∝ exp(−퐻 푗 )·Cal 푗 ·VisFid(휏 (푗) ), ˆ푎 = arg max 푎 ∑︁ 푗:푎 (푗) =푎 푤 푗 where퐻 푗 is token entropy on decisive steps,VisFidis the pixel fidelity (main text, Eq. (10)), andCal 푗 ∈ (0, 1] is a calibrated confidence (below). We abstain when the normalized margin is < 훿. Dawid–Skene reliability (EM). We estimate rollout relia- bility with a Dawid–Skene (DS) model over the per-query rollouts. Let푦 ∈ 1,...,퐶be the latent true class and푧 푗 the label from rollout푗. Each rollout has a confusion matrix Π (푗) ∈ R 퐶×퐶 with entries휋 (푗) 푐ℓ = 푃(푧 푗 = ℓ | 푦 = 푐) and class prior흅. We use the standard Dawid–Skene EM updates for the class prior and per-rollout confusion matrices [8,63] and initialize with majority-vote labels; eachΠ (푗) starts as a Laplace-smoothed diagonal with휖 = 10 −2 . We iterate until relative log-likelihood improvement< 10 −6 or 50 iterations. The per-rollout reliability proxy푟 푗 = 1 퐶 Í 푐 휋 (푗) 푐 multiplies 푤 푗 (absorbed into Cal 푗 ). Confidence calibration. Calibrators are fit on a held-out dev-cal split stratified by task and tool. We evaluate: (i) temperature scaling (TS) [15] with scalar푇 > 0minimizing NLL on dev-cal,min 푇>0 − Í 푖 log softmax(푠 푖 /푇) 푦 푖 ; (i) Platt scaling (binary or one-vs-rest); (i) vector/matrix scaling for multiclass [28]; (iv) isotonic regression [74]. We report TS as default for stability and simplicity; others appear in sensitivity. How calibration enters푤 푗 . Let푝 raw 푗 be the uncalibrated answer probability for rollout휏 (푗) . After calibration we obtain ̃푝 푗 and set Cal 푗 = ̃푝 훼 푗 , with훼 = 1by default and훼 ∈ [0.75, 1.25]in sensitivity (tuning confidence influence). Sensitivity. On the process-audit split (3 seeds), global TS yields푇 = 1.21± 0.07(mean±sd across tasks) and reduces ECE from6.8%to2.1%. Sweeping푇 ∈ [0.8, 1.6]changes median|푤 푗 |by< 0.03(IQR[0.01, 0.06]) and flips the final 0.00.20.40.60.81.0 Mean predicted probability 0.0 0.2 0.4 0.6 0.8 1.0 Empirical accuracy Calibration curves (Pixelis, dev split) Uncalibrated (ECE=0.250) Temp-scaled (ECE=0.198) DS-weighted (ECE=0.143) Perfectly calibrated Figure S1. Calibration curves for Pixelis (8B) on the dev split. Reliability diagrams comparing the uncalibrated policy, temperature scaling (TS), and DS-weighted consensus. Each point aggregates a probability bin; the푥-axis is the mean predicted probability and the푦-axis is empirical accuracy. The diagonal corresponds to perfect calibration. Legends report expected calibration error (ECE), showing that TS reduces miscalibration and DS-weighted consensus further tightens calibration without degrading accuracy. answer in0.9%of queries; with abstention, flips drop to0.6% at< 1.5%coverage loss. Platt (OvR) and vector scaling give similar accuracy but slightly higher variance in푤 푗 (median absolute change0.05and0.04vs. TS0.03) and comparable ECE (2.3–2.6%). Isotonic can further reduce ECE (to1.8%) but overfits small per-task bins and worsens risk–coverage under shift. We therefore use TS for all main results; full curves are provided in Fig. S1. 3. Retrieval keys and index implementation Key construction.We form a hybrid key푘(푞) = [푘 txt (푞)∥ 푘 pix (푞)]. Text keys푘 txt ∈ R 768 are mean-pooled token embeddings of the prompt and last 32 thought to- kens. Pixel keys푘 pix ∈ R 1024 are pooled visual descriptors on decisive footprints (for images: masked average of DI- NOv2 ViT-L/14 features; for videos: per-frame pooling then temporal average on tracklets or segments). We optionally compress[푘 txt ∥푘 pix ] ∈ R 1792 to 768-d via OPQ when CPU- only search is used. Similarity is cosine of L2-normalized keys; the main text uses the convex mix with휆 txt + 휆 pix = 1. OPQ + IVF-PQ index. We use FAISSIndexIVFPQ with an OPQ rotation trained on 2 M random keys. The coarse quantizer uses IVF with푛 list = 4096centroids (20 k- means iters). The product quantizer uses푚 = 64subvectors, 푏 = 8bits each (code size =64 bytes/vector). Search uses 푛 probe = 32; we rerank top-256 by exact cosine before selecting neighborhood 퐾 for TTRL (typically 5∼ 8). SEG/ZOOMTRKOCRTEMPPROPRetrieval+ voting Tool component 0 1 2 3 4 5 6 Latency per call (s) Per-tool latency distribution (8B, tool-side) = 1.26 = 1.51 = 0.43 = 1.71 = 0.17 = 0.37 Figure S2. Per-tool latency distribution for Pixelis (8B). Violin plots show the distribution of per-call latency for each tool compo- nent (SEG/ZOOM, TRK, OCR, TEMP, PROP, Retrieval+voting) over the same1,500-query eval set as Table S2. Each violin overlays a box (IQR and median), jittered samples, and a mean markerˆ휇. The푦-axis is latency per tool call in seconds; the푥-axis enumer- ates tools. Compared to the static medians in Table S2, the plot makes the long tails for TRK and TEMP explicit, while PROP and Retrieval+voting stay tightly concentrated near zero. Scale and memory. With푁items, memory≈ 푁 ·(code)+ 푁 ·(id)+ 푛 list ·(centroids)+ OPQ params. For our default 푁 = 1.25M, this yields≈127 MB on disk (codes≈80 MB + IDs≈ 5 MB + centroids≈ 28.5 MB + OPQ≈ 12.9 MB). Build and update. Training (2 M keys) takes 16–18 min on A100; adding 1.25 M vectors takes 2.3 min with pinned host memory. Index is append-only in evaluation; no PQ/IVF retraining. We enforce evaluation-media whitelisting so that near-duplicates are excluded at ingest (main text Sec. 5.1). Latency. On CPU (32c, AVX512,푛 probe = 32): median 푃50 = 7.8ms/query,푃95 = 14.6ms/query (top-256). On GPU (A100, FAISS-GPU):푃50 = 2.9ms,푃95 = 5.6ms. These isolated retrieval times align with the retrieval+voting overhead (≈0.35 s median) reported in the main paper. The remaining overhead to≈0.35 s per query comes from rollout sampling, scoring, and voting on the CPU side. Accuracy/speedknobs.Sweeping푛 list ∈ 2048, 4096, 8192, 푛 probe ∈ [8, 64], 푚 ∈ 32, 64 on a held–out split, we selected(4096, 32, 64)as best accuracy–latency trade–off. Raising푛 probe to 64 yields +0.3 VisFid@1 but 1.7× higher푃95; reducing to푚 = 32saves 30 % memory but costs≈–0.5 VisFid@1. OPQ is retained by default because disabling it increases recall variance and slightly worsens RaCPR. Fallback and ablations. In text-only retrieval ablation we use HNSW(푀 = 32,푒 푓 search = 128)on 768-d푘 txt : comparable accuracy but worse tool-alignment on pixel-heavy subsets. The hybrid OPQ+IVF-PQ index remains default for Pixelis. 0 100 200 Count Tool-side only 024681012 Latency per query (s) 0 50 100 150 Count Query-level latency distribution (8B) Figure S3. Query-level latency distribution for Pixelis (8B). Histogram of tool-side latency per query and full end-to-end latency on the same1,500-query eval split as Table S2. Tool-side latency aggregates all tool calls within a trajectory, while end-to-end latency additionally includes model forward and lightweight bookkeeping. The distributions make the tail behavior explicit beyond the static summaries in the main text (p50≈ 5.8s, p90≈ 8.1s, p95≈ 10.2s), and show that most queries fall well below the worst-case thresholds even under tool-heavy chains. 12345678 Decisive chain length | | 14.2 14.4 14.6 14.8 15.0 15.2 GPU memory usage (GB) Example memory footprint vs. chain length Figure S4. Memory footprint vs. decisive chain length. Estimated GPU memory usage for the 8B model as a function of decisive chain length|휏|on the eval workload. The base footprint corresponds to the backbone and cached features; each additional step adds buffers for pixel tools (segmentation, tracking, temporal localization) and intermediate activations. Growth with chain length in the operating regime of Pixelis is modest and approximately linear (median |휏| ≈ 3.7), indicating that the shorter chains induced by C-RFT and Pixel TTRL reduce not only latency but also memory variance and peak usage. Background on FAISS, IVF-PQ and OPQ can be found in [12, 21, 25] 4. Structure priors vs. adjacency coherence We compare our adjacency-based coherence on step embed- dings퐸 푡 to three widely used structure priors: (A) argu- Table S2. Index configuration and footprint (default). ComponentSettingNotes Key dims푑 txt = 768, 푑 pix = 1024 concat→1792; OPQ→768 (CPU) IVF lists푛 list = 4096k-means (20 iters) PQ푚 = 64, 푏 = 864 B/code OPQ1792→1792 rotation200 epochs Search푛 probe = 32, top-256 re-rank exact cosine Scale푁 = 1.25 M itemsfootprint-level entries Latency P50/95= 7.8/14.6 ms (CPU) 2.9/5.6 ms (GPU) Memory ≈ 127 MBcodes + IDs + centroids + OPQ ment L2 smoothness, (B) total-variation (TV) regulariza- tion, and (C) contrastive next-step embedding alignment (CPC/InfoNCE). Our coherence (adjacency on퐸 푡 ). Let퐸 푡 be the normalized step embedding (in main paper Sec. 4.2). The reward term 푅 coh = 1 푇 − 1 푇−1 ∑︁ 푡=1 z cos(퐸 푡+1 , 퐸 푡 ) uses per-task z-scoringz(·)to discourage tool hopping while tolerating purposeful shifts. (A) L2 smoothness on arguments. With structured argu- ment vector 푎 푡 (concat of typed tool parameters), we add 푅 L2 =− 1 푇 − 1 푇−1 ∑︁ 푡=1 ∥푎 푡+1 − 푎 푡 ∥ 2 2 . This favors small increments but is blind to whether changes are semantically aligned with visual evidence; it also penal- izes necessary jumps (e.g., re-frame). (B) TV-style regularization.Following ROF-style anisotropic penalties, we use 푅 TV =− 1 푇 − 1 푇−1 ∑︁ 푡=1 ∥푎 푡+1 − 푎 푡 ∥ 1 + 휖∥푎 푡+1 − 푎 푡 ∥ 2 , which is more edge-preserving than pure L2 but still defined in argument space rather than behavior space. (C) CPC/InfoNCE next-step alignment. We align퐸 푡+1 to the positive(퐸 푡 )against negativesN 푡 (other steps in-batch) using InfoNCE [56]: 푅 CPC = 1 푇 − 1 ∑︁ 푡 log exp(⟨퐸 푡+1 , 퐸 푡 ⟩/휏) Í 푛∈N 푡 exp(⟨퐸 푡+1 , 퐸 푛 ⟩/휏) . This encourages temporal slowness at the representation level (cf. classical slowness [65]), but can over-contract exploration when the negative set is narrow. Training protocol. All priors replace푅 coh in the C-RFT reward under identical GRPO, SFT anchor, horizons, and KL Table S3. Structure priors vs. our adjacency coherence (median over seeds; matched KL). Δ is vs. full C-RFT. PriorAcc. ΔRaCPR gain re- covered Notes L2 smoothness (args) −0.7 pt31%shortest chains degrade on jumps TV (args)−0.5 pt36%preserves edges, still argument-space CPC on 퐸 푡 −0.3 pt54%tighter but exploration contracts Adjacency on 퐸 푡 (ours)0.0100%best Acc./RaCPR at matched KL corridor. Coefficients are line-searched to match token-KL (≈ 0.15) to ensure comparable regularization pressure. Takeaways. Argument-space smoothers (L2/TV) damp jitter but miss whether successive steps are meaningfully adjacent in the scene, recovering only a third of the RaCPR benefit and slightly hurting accuracy. CPC-style alignment in embedding space captures structure better but tends to over-regularize exploration unless the negative set is carefully diversified. Our adjacency on퐸 푡 retains exploratory breadth while sup- pressing thrashing, yielding shorter, auditable chains without sacrificing accuracy at the same KL budget. 5 . Tool necessity: subsets and tool-ablation pro- tocol Goal. We demonstrate that gains stem from executable tools rather than text-only effects by (i) constructing tool- dependent subsets where answers provably require a specific tool, (i) running phase-matched ablations that disable exactly one tool at a time under identical budgets, and (i) analyzing correlations between process metrics (RaPR/RaCPR) and task accuracy. Tools used:identities as in Tab. S4; full configs (model/vers./args/links/licenses) in Appx. C. Table S4. Tool identities, earlier snapshot for ablations. Full configs in Table S8. ToolModel (ver.)FT? Source / License SEGSAM2norepo / license TRKBoT-SORTnorepo / license OCRPP-OCR 3.0norepo / license TEMP VTG heuristics n/aprotocol / license Tool-dependent subsets. For each example푞, we derive a binary necessity label푧 tool (푞) ∈ 0, 1from reference artifacts and validators; an item is included in subsetD tool iff 푧 tool (푞) = 1. SEG-required (segmentation/crop): (a) reference mask/box 푀 ★ exists and ANLS does not apply; (b) at least one valid solution must localize an object/part with IoU threshold IoU(푀, 푀 ★ ) ≥ 0.5and the final answer changes if the footprint is replaced by a uniformly dilated/eroded region (±8px at input scale). TRK-required (tracking): (a) multi- frame clip with a persistent identity; (b) valid solution must link detections across frames with CLEAR-MOT or HOTA ≥protocol threshold; (c) replacing per-frame detections with shuffled identities degrades the reference program. READ- required (OCR): (a) answer string overlaps a reference OCR span with ANLS≥ 0.7; (b) perturbing the span via character masking changes the answer. TEMP-required (temporal locate): (a) reference temporal segmentP ★ is provided; (b) valid solution must select a segment with IoU≥ 0.5; (c) replacing the segment by a random window of equal length flips the decision. Items may belong to multiple subsets. We precompute푧 tool using public validators (same thresholds as main text) and freeze them before any adaptation. Single-tool ablations (phase-matched). We evaluate per- tool necessity by disabling exactly one tool while keeping the backbone, decoding, and budgets fixed. Protocol.Foreachtool푢 ∈ SEG,ZOOM/CROP,TRK,READ,TEMP :(i)remove 푢from the callable set and map any attempted call to a no-op with a neutral token placeholder; (i) keep KL corridor, EMA anchor, temperature, and beam identical; (i) freeze retrieval/index, acceptance thresholds, and abstention margin 훿; (iv) reportΔAccuracy andΔRaPR/ΔRaCPR on both the full test split and the matching subsetD 푢 . Effect size. Let푚 ∈ Acc, RaPR, RaCPR. We define the tool necessity effect on subsetD 푢 as Δ푚 푢 ≜ 푚 full (D 푢 )− 푚 \푢 (D 푢 ), where푚 full is with all tools and푚 \푢 is with tool푢disabled. We report mean over 3 seeds with bootstrap 95% CIs. Task-wise importance matrix. To visualize which tool mat- ters for which benchmark, we aggregateΔ푚 푢 by dataset/task family: I task×tool (푚)[푡,푢] = Δ푚 푢 task=푡 . We include a compact table for푚 = Accand푚 = RaCPR; cells are color-coded by effect size (gray-scale in print). This reveals, e.g.,READ→InfoVQA,TRK→video tasks, SEG/ZOOM→spatial counting/referring,TEMP→temporal localization. RaPR/RaCPR vs accuracy correlations. We quantify whether better process behavior predicts task success. For per-query tuples(Acc(푞) ∈ 0, 1, RaPR(푞), RaCPR(푞)), we compute Spearman correlations with bootstrap CIs: 휌 RaPR = Spearman(Acc, RaPR) 휌 RaCPR = Spearman(Acc, RaCPR) We also fit a calibrated logistic modelPr(Acc=1 | RaPR, RaCPR) = 휎(훽 0 + 훽 1 RaPR+ 훽 2 RaCPR)and report AUC/APS to assess incremental predictive power. Sen- sitivity to thresholds in RaCPR (휏, 퐿 min ) is evaluated by Table S5. Single-tool ablation: absolute changeΔin task score when disabling the indicated tool. Tool off↓V*MMBench MVBenchInfoVQA (ANLS) Video- MMMU VSI SEG --1.20 [--1.60, --0.80] --0.80 --0.60 --0.50 --0.70 --1.10 TRK --0.60 [--0.90, --0.30] --0.30 --1.00 --0.20 --1.50 --0.50 OCR --0.40 [--0.70, --0.10] --0.50 --0.30 --2.10 --0.40 --0.30 TEMP --0.30 [--0.60, --0.10] --0.20 --0.90 --0.10 --1.20 --0.40 ZOOM/CROP --0.70 [--1.00, --0.40] --0.40 --0.50 --0.30 --0.60 --0.80 re-computing correlations over a grid while keeping the KL corridor and acceptance policy fixed. Fairness and leakage controls. All tool-ablations reuse the identical neighbor index, whitelist masks, and de-dup filters; adaptation budgets (number of updates) and acceptance rates are matched by capping accepted updates to the min across variants. Metrics are computed with the same external evaluators (ANLS, HOTA/CLEAR-MOT, VTG overlaps) and identical seeds. Reporting template (space-efficient). We include one2×5 table forI(Acc)and a2×5table forI(RaCPR), plus a1×2 bar plot of휌 RaPR and휌 RaCPR with 95% CIs. A small caption notes that the largest degradations align with subsetsD 푢 , supporting tool necessity, and that RaCPR correlates more strongly with success than RaPR on multi-step video tasks. Single-tool ablations. We ablate one tool at a time (SEG, TRK, OCR, TEMP, ZOOM/CROP). When a disabled tool is called, the agent receives a typed empty output and must recover with the remaining tools; decoding temperature, beam, and budgets are held fixed across runs. The table reports absolute drops (Δ, negative means worse) in task accuracy (ANLS for InfoVQA) relative to the full-tool agent; 95% CIs are from stratified bootstrap (10k) over three seeds. Fidelity floor (robustness). To separate necessity from brit- tleness, we inject low-confidence outputs for the ablated tool while keeping other tools intact (e.g., degraded masks/boxes, jittered tracks, noisy OCR tokens, off-by-1s temporal spans). We then re-evaluate Accuracy/ANLS and process metrics (RaPR/RaCPR/VisFid) under identical decoding, reporting paired deltas vs. full fidelity. Results track the same de- pendency pattern as the hard ablation, but with smaller magnitude (table omitted for space). Process metrics vs. correctness. We assess whether pro- cess quality predicts task success. For each example푖, let 푦 푖 ∈ 0, 1indicate correctness, and푝 RaPR 푖 , 푝 RaCPR 푖 be per- example process scores. We report point-biserial Pearson 푟(푦, 푝), Spearman휌, and partial correlations controlling Table S6. Overall correlation between process metrics and correct- ness. Mean [95% CI] over three seeds. Setting푟(푦, RaPR)푟(푦, RaCPR)Δ푟(post−pre TTRL) SFT-only 0.29 [ 0.25, 0.33 ] 0.35 [ 0.31, 0.39 ] – RFT-Full 0.34 [ 0.31, 0.37 ] 0.42 [ 0.38, 0.46 ] – TTRL (pre) 0.32 [ 0.28, 0.36 ] 0.40 [ 0.36, 0.44 ] – TTRL (post) 0.37 [ 0.33, 0.41 ] 0.48 [ 0.44, 0.52 ] +0.08 Table S7. Per-benchmark Spearman휌between process metrics and correctness (post TTRL shown; pre in parentheses). Partial휌 controls for 퐿 and KL tok . V*MMBench MVBench InfoVQAVideo- MMMU VSI 휌(푦, RaPR) .33 (.28) .30 (.26) .31 (.27) .34 (.30) .36 (.31) .35 (.30) 휌(푦, RaCPR) .41 (.35) .38 (.33) .43 (.37) .49 (.43) .52 (.46) .47 (.41) 휌(푦, RaCPR; 퐿, KL) .36 .34 .39 .44 .46 .42 for chain length퐿and token-level driftKL tok . Confidence intervals use nonparametric bootstrap (10k);푝-values use two-sided permutation tests. Settings include SFT-only, RFT- Full, and Pixel TTRL (pre/post), both overall and on the tool-dependent subsets. Logistic view (compact). A regularized model Pr(푦=1) = 휎 ( 훽 0 + 훽 1 RaPR+ 훽 2 RaCPR+ 훽 3 퐿+ 훽 4 KL tok ) typically yieldsexp(훽 2 )>1with non-overlapping CIs after controlling for퐿andKL tok . Binning RaCPR into quintiles produces near-isotonic accuracy curves; selective adaptation shifts mass toward higher RaCPR bins, consistent with behavior-level voting. 5.1. Tool identities and configurations Callable tools (agent executes). We upgrade each tool to a recent, actively maintained implementation and freeze versions for reproducibility. Full command-line args and model hashes are released with the YAMLs. Table S8. Tool identities (new snapshot used in the main text; stable versions). ToolModel (ver./date)FT? Source SEGSAM 2 (ViT-H, 2024-09)no Meta SAM2 TRKBoT-SORT (2023-11; MOT multi-object) no BoT-SORT repo OCRPP-OCR 3.0 (PaddleOCR 3.0, 2024) no PaddleOCR TEMPUniVTG heuristics / eval har- ness (2023-10) n/a UniVTG Notes. SEG upgrades from SAM (v1) to SAM 2, which adds video memory and stronger mask heads while keeping promptable segmentation; we use the ViT-H checkpoint and restrict to single-frame masks for parity with our pipeline. TRK uses BoT-SORT, which combines a ByteTrack-style association with camera-motion compensation and optional ReID; we disable ReID for fair tool-only comparisons and use the repo’s default MOT parameters. OCR is P-OCR (PaddleOCR 3.0), retaining DB detection + CRNN-like recognition with improved rec heads; we keep lexicon-free decoding and enable the repo’s default postpro- cessing. TEMP uses the UniVTG evaluation harness to implement our lightweight temporal-localization heuristics (proposal scor- ing + NMS over windows) and to ensure metric compatibility with VTG literature. Frozen evaluators/encoders (not tools). Unchanged except for pinning sources: HOTA/CLEAR-MOT viaTrackEval (for scoring tracks), ANLS for OCR-style string similarity, and DINOv2 as a fixed visual encoder for footprint descriptors and RaCPR evaluation. We do not fine-tune these. Table S9. Evaluator/encoder references (frozen). ItemRoleSource TrackEval (HOTA/CLEAR-MOT) tracker scoringofficial TrackEval repo/paper ANLSOCR similaritystandard VQA-OCR protocol DINOv2 ViT-L/14fixed embeddings GitHub (facebookresearch/dinov2) Licenses. All tools are used under their original repository licenses (see linked repos). We ship commit hashes and config YAMLs to ensure exact reproducibility. 6. Pixel TTRL loss: clean notation, masks, and pseudo-code Objects and sets. For a query푞, letN(푞)be its retrieved neighborhood and휏 (푗) 푁 푗=1 be sampled rollouts from the current policy휋 휃 . Each rollout휏 = (푎 1:푇 , 휋 1:푇 )has an answerans(휏)and decisive stepsD(휏)as defined in the main paper. Letˆ푎be the consensus answer (Sec. 5.2) and휏 ★ the selected exemplar trajectory. Masks. We use two binary masks: (i) abstention mask 푚(푞) ∈ 0, 1, where푚(푞)=0if the uncertainty-weighted vote margin< 훿or the conformal set has|C| > 1, otherwise 푚(푞)=1; (i) answer-agreement mask 푦(휏, ˆ푎) = 1ans(휏) = ˆ푎. Behavioral similarity and penalties.Similarity to the exemplar follows the mixed edit–alignment score Sim behav (휏,휏 ★ ) ∈ [0, 1](main text Eq. (9));the length/invalid-ops penalty Pen(휏) ≥ 0 follows Eq. (11). Neighborhood value and baseline. Define a stop-gradient neighborhood value푣(N(푞)) = 1 |N(푞)| Í ℎ∈N(푞) 푣(ℎ) (EMA over curiosity/coherence summaries). We use a query-level baseline푏(푞) = EMA 휌 푣(N(푞)) to reduce variance;푏(푞) is treated as a constant in policy gradients. Per-rollout return and advantage. Given nonnegative scalars 휅 and 휆 pen , define 푟(휏;휏 ★ , ˆ푎) = 푦(휏, ˆ푎)+ 휅 Sim behav (휏,휏 ★ )− 휆 pen Pen(휏). The centered advantage is퐴(휏,푞) =푣(N(푞)) 푟(휏;휏 ★ , ˆ푎)− 푏(푞) . EMA anchor and KL corridor. Let휋 EMA be an exponential- moving-average reference of휋 휃 . Token-level KL is con- strained by a penalty훽 KL tok (휋 휃 ∥휋 EMA )with a PID-style controller on 훽 to keep KL tok ∈ [KL min , KL max ]. Pixel TTRL objective (final). The clean loss minimized w.r.t. 휃 is L TTRL =−푚(푞) E 휏∼휋 휃 퐴(휏,푞) log휋 휃 (휏) + 훽 KL tok 휋 휃 ∥휋 EMA (S1) where푚(푞)masks policy gradients on low-confidence queries. When푚(푞)=0, only the KL-to-EMA term is active (stabilization without updates). The expectation is approx- imated with푁rollouts per query; gradients do not flow into푣,푏(푞), or any pseudo-references. The first term is a REINFORCE-style estimator with a baseline [64], the second enforces drift control. Abstention and conformal sets yield standard risk–coverage behavior [2]. Calibrated voting weights (recap). For completeness, per- rollout weights used to formˆ푎are푤 푗 ∝ exp(−퐻 푗 )· Cal 푗 · VisFid(휏 (푗) )·푟 푗 , where퐻 푗 is decisive-step entropy,Cal 푗 is a calibrated confidence (temperature scaling on a dev-cal split), VisFidis pixel fidelity, and푟 푗 is Dawid–Skene reliability (Sec S2). Pseudo-code. 1.Inputs: query푞; neighborhoodN(푞); policy휋 휃 ; EMA policy휋 EMA ; corridor[KL min , KL max ]; hyperparameters (휅,휆 pen ,훿). 2.Sample rollouts: draw휏 (푗) 푁 푗=1 ∼ 휋 휃 (· | 푞) ; compute answers 푎 (푗) , decisive-step entropies 퐻 푗 , VisFid(휏 (푗) ). 3.Calibrate and weight: obtainCal 푗 by temperature scaling on dev-cal; estimate푟 푗 via DS-EM; set푤 푗 ∝ exp(−퐻 푗 ) Cal 푗 VisFid(휏 (푗) ) 푟 푗 . 4. Consensus & abstention: compute class scores푠(푎) = Í 푗:푎 (푗) =푎 푤 푗 ; letˆ푎 = arg max 푎 푠(푎). Form conformal setCby thresholding nonconformity; set푚(푞)=1if margin≥ 훿 and|C|=1, else 푚(푞)=0. 5.Exemplar: among휏 (푗) : 푎 (푗) = ˆ푎, select휏 ★ = arg min 휏 |휏|− 휂(Cur+ Coh)− 휉 VisFid(휏) . 6. Returns and baseline: compute푟(휏 (푗) ;휏 ★ , ˆ푎)and 퐴(휏 (푗) ,푞) =푣(N(푞)) 푟(휏 (푗) ;휏 ★ , ˆ푎)− 푏(푞) with stop- grad on푣, 푏(푞). 7. Policy step: ∇ 휃 L ≈− 푚(푞) 1 푁 푁 ∑︁ 푗=1 퐴(휏 (푗) ,푞)∇ 휃 log휋 휃 (휏 (푗) ) + 훽∇ 휃 KL tok 휋 휃 ∥휋 EMA 8. EMA and KL control: update휋 EMA ← 휌 휋 EMA +(1− 휌) 휋 휃 ; adjust훽by a simple PID rule to keepKL tok in- corridor. Practical notes. (i) Use the same randomness for pre/post ablations to attribute gains; (i) clip gradients and normalize 퐴(휏,푞)within-batch to reduce variance; (i) reject near- duplicate rollouts by frame-IoU> 0.85; (iv) log푚(푞)and acceptance rates to plot risk–coverage curves and KL excur- sions. 6.1. PID control for the KL corridor (implementa- tion) Objective.Maintain token-level KL within a corridor [KL min , KL max ]by adapting the penalty coefficient훽to- ward a target KL tgt (midpoint). Update. Let푒 푡 = KL tok,푡 − KL tgt . We use a clipped PID on 훽: 훽 푡+1 = clip 훽 푡 h 1+ 퐾 푝 푒 푡 + 퐾 푖 퐼 푡 + 퐾 푑 (푒 푡 −푒 푡−1 ) i , 훽 min , 훽 max , 퐼 푡 = clip 퐼 푡−1 +푒 푡 , −퐼 max , 퐼 max . We update once per optimizer step (or every퐵mini-batches, default퐵=1), with anti-windup on퐼 푡 . In practice we nor- malize푒 푡 byKL tgt for scale invariance when switching backbones. Defaults (ours).KL tgt =0.15(C-RFT) /0.15 (TTRL),[KL min , KL max ]=[0.10, 0.20];퐾 푝 =0.30,퐾 푖 =0.05, 퐾 푑 =0.10;훽 min =10 −3 ,훽 max =10 2 ;퐼 max =5; warmup 100 steps with퐾 푑 =0to avoid derivative kick. We smoothKL tok by an EMA (휏=0.9) before computing 푒 푡 . Notes. (i) This mirrors common adaptive-KL practice in PPO/RLHF, where훽is tuned toward a target divergence; (i) PID gives quicker settling and lower overshoot than pure proportional updates under distribution shift; (i) Gains were chosen to keep rise time< 200steps and overshoot< 10% on our audits; (iv) We found identical gains adequate for C-RFT and TTRL at 8B; for other scales, tune퐾 푝 first, then퐾 푑 , finally퐾 푖 . See [20,44] for adaptive KL in LLM RL, and classical PID references [3, 4]. 7 . Layer choice and uncertainty-gate sensitivity Which multimodal layer for퐸 푡 . We vary the feature- extraction layer퐿 m used to compute the visual summary푣 푡 (Sec. 4.2): the default is the 3rd fusion block from the top. We probe−2,−1,+1,+2blocks around this default and recompute퐸 푡 = 푔 휙 ([푣 푡 ∥푥 푡 ∥onehot(푎 푡−1 )])/∥ · ∥ 2 with the same projector푔 휙 and the same z-scoring statistics protocol. Across the process-audit split, RaCPR varies within±0.5 points and task Accuracy within±0.2points; deeper layers slightly favor coherence (higher RaCPR, shorter chains), Table S10. Sensitivity to feature-layer choice and uncertainty gating (process-audit split; 3 seeds; mean±sd).ΔLen is the change in qualified chain length vs. default. SettingAcc. (%)RaCPRRaPR ΔLen 퐿 m (default)+0.0 +0.0 +0.00.00 퐿 m (-2 blocks) −0.1±0.1 −0.5±0.3 +0.2±0.2 +0.12 퐿 m (-1 block) −0.1±0.1 −0.3±0.2 +0.1±0.2 +0.07 퐿 m (+1 block) +0.0±0.1 +0.2±0.2 −0.1±0.1 −0.05 퐿 m (+2 blocks) +0.1±0.1 +0.4±0.3 −0.1±0.2 −0.09 Gate off (푆=1, 훽=0) −0.2±0.1 −0.7±0.3 +0.3±0.2 +0.18 MC-2, 훽=2.5−0.1±0.1 −0.3±0.2 +0.1±0.1 +0.09 MC-4, 훽=5 (default)0.00.00.00.00 MC-8, 훽=10+0.0±0.1 −0.1±0.2 −0.2±0.2 −0.03 while earlier layers marginally increase exploration variance. These observations are consistent with prior findings that representation content varies by depth and that mid/late layers carry more linearly recoverable task signals [48,72]. We therefore retain the default mid–late fusion layer for all main results. Uncertainty gate in curiosity. The curiosity term uses a tool-conditioned dynamics headℎ 휓 that predictsˆ푣 푡+1 ; its prediction error푒 푡 is down-weighted by an epistemic- uncertainty gate1/(1+ 훽휎 2 푡 )estimated via MC dropout (4 samples,푝=0.1). We sweep the number of MC samples푆∈ 1, 2, 4, 8and the gate strength훽∈0, 2.5, 5, 10, keeping the KL corridor unchanged. Removing the gate (훽=0or푆=1) increases false novelty on high-frequency textures, leading to longer chains and a mild drop in RaCPR at matched Accuracy. Moderate gating (푆=4,훽≈5) stabilizes exploration without suppressing useful novelty, in line with Bayesian-dropout interpretations of epistemic uncertainty and its effect on overconfidence [11, 15, 27]. Takeaways. (i) Mid–late fusion layers provide the best coherence–exploration balance for퐸 푡 with small variance, matching the intuition that deeper multimodal layers encode more task-aligned signals [48,72]. (i) A lightweight uncer- tainty gate (MC-4,훽≈5) reduces spurious curiosity while preserving useful exploration, aligning with Bayesian inter- pretations of dropout for epistemic uncertainty and with the need to curb overconfident updates under shift [11,15,27]. No additional figures are required beyond Table S10. 8 . Budget matching: acceptance, drift, and fairness tables Setup and budget matching. We equalize the adaptation budget across methods: identical neighborhood size퐾, num- ber of rollouts푁per query, decoding temperature/beam, and the same number of adaptation steps (8K). A step is counted when a query completes retrieval, sampling, and voting; an accepted update occurs only if the consensus passes safety checks (entropy margin and abstention mask) and the KL-to- EMA corridor remains active. We report: acceptance rate (%), accepted updates per 1K queries, token-levelKLto EMA (P50/P95), and pre/post adaptation accuracy. Risk–coverage is summarized in the main text via a훿sweep; selective prediction background follows the standard RC/AURC view. Overall acceptance and drift. Table S11 summarizes ac- ceptance and drift at matched budgets for three variants: uncertainty-weighted consensus with abstention (Pixel TTRL, ours), hard-majority TTRL, and a representative VLM TTA baseline (no tools). Temperature scaling is used for confi- dence calibration unless stated otherwise. Table S11. Acceptance and drift at matched budgets (8K steps, 퐾=8, 푁=8). ΔAcc is post–pre adaptation (p). MethodAccept (%) Abstain (%) KL P50KL P95ΔAcc Pixel TTRL (weighted + abstention) 58.79.70.160.19+3.5 TTRL (hard majority)61.20.00.220.29+1.8 VLM TTA (StatA-like, no tools) 100.00.00.230.35+1.5 Fairness per benchmark. Table S12 reports pre/post ac- curacy, accepted updates per 1K queries, and KL P95 per benchmark under the same step budget; abstention targets ≈90% coverage for the Pixel TTRL row. These numbers align with the main paper’s deltas and KL corridor. Table S12. Pre/post accuracy (%) with matched budgets. Accepted updates per 1K queries (Acc@1K) and KL P95 shown. BenchPrePost ΔAcc@1KKL P95Method V* 86.489.6+3.25920.19Pixel TTRL 86.488.0+1.66150.28Hard maj. 86.487.7+1.310000.34VLM TTA MMBench-en 85.087.9+2.95810.19Pixel TTRL 85.086.5+1.56070.29Hard maj. 85.086.2+1.210000.33VLM TTA MVBench 68.771.6+2.95660.18Pixel TTRL 68.770.1+1.45960.28Hard maj. 68.770.0+1.310000.35VLM TTA InfoVQA 83.185.9+2.85980.18Pixel TTRL 83.184.4+1.36220.29Hard maj. 83.184.6+1.510000.35VLM TTA Video-MMMU 65.367.9+2.65740.19Pixel TTRL 65.366.6+1.36010.29Hard maj. 65.366.8+1.510000.36VLM TTA VSI-Bench 59.462.3+2.95600.19Pixel TTRL 59.460.8+1.45920.28Hard maj. 59.460.9+1.510000.35VLM TTA Notes. Acceptance@1K is the count of accepted policy updates per 1,000 processed queries. Pixel TTRL’s lower acceptance than hard-majority is expected due to abstention, yet it delivers largerΔAcc per accepted update and keeps KL within the corridor. Hard-majority pushes higher KL tails and is more brittle under noisy neighborhoods; VLM TTA applies updates unconditionally, yielding the highest acceptance but also the largest drift. Risk–coverage trade- offs for selective prediction are standard and motivate our abstention policy; confidence calibration uses temperature scaling by default. 9. Failure cases and extended risk–coverage curves Where the toolchain breaks. We systematically audited failures on 600 examples per domain (images, videos) and grouped them into four recurring modes: (i) thin structures or small objects, whereSEG/TRKmasks fragment and re- duce IoU, cascading to wrongREAD/COMPARE; (i) stylized or low-contrast fonts, whereREADnormalization (NFKC, punctuation stripping) still leaves ANLS gaps; (i) temporal clutter or fast motion, whereTRKdrifts under motion blur and tight crops; and (iv) dense layouts, whereZOOM/CROP selects a plausible but wrong pane. In all four, RaPR drops (invalid steps) and adjacent-step cosines fall below the gate, lowering RaCPR and shortening accepted chains. Neighborhood pathologies. Two neighborhood issues de- grade online adaptation: (a) semantic near-miss, where visually similar but label-mismatched neighbors increase disagreement entropy and trigger abstention; (b) adversarial majority, where a fraction of flipped neighbor answers steers hard-majority voting to off-manifold updates. Uncertainty- weighted consensus with abstention mitigates both by convert- ing disagreement into delayed updates rather than erroneous drift (main text Table 3). Extended risk–coverage curves. We evaluate selective adap- tation by sweeping the abstention margin훿and plotting error among selected updates (Err@Sel) against coverage (fraction of updates executed). Following selective prediction practice, smaller coverage should monotonically reduce Err@Sel as the system refrains on uncertain cases [13,14]. We report curves under three conditions: (1) clean neighborhoods; (2) noisy neighborhoods with 20% vote flips; and (3) retrieval shift, where pixel keys are perturbed by JPEG(20) and±2% rescale before search. In (1), Err@Sel drops by 18–22% when coverage decreases by8–12%(median across tasks). In (2), hard-majority sits strictly above our curve (dominated risk for the same coverage), while the weighted+abstention regime retains a≈ 0.19KL P95 and reduces variance of RaCPR across updates. In (3), curves widen but preserve ordering: increasing휆 pix during retrieval partially recovers VisFid and moves the curve toward the clean frontier. Calibration and abstention interplay. Temperature scal- ing reduces ECE on decisive-step logits (TS푇 ≈ 1.2on the dev-cal split), yielding smoother risk–coverage trades; Platt/matrix scaling show similar accuracy but larger variance in per-rollout weights, slightly inflating coverage volatility. These observations align with standard results on calibrated abstention and conformal-style uncertainty sets [1, 15]. Takeaways for deployment. (i) Raise decisive-step confi- dence thresholds and widen the abstention margin on scenes with thin structures or stylized fonts; (i) cap the tracking horizon under fast motion and preferSEG→TRKoverTRK- only initialization; (i) under retrieval shift, increase푛 probe 0.20.40.60.81.0 Coverage 0.06 0.08 0.10 0.12 0.14 Err@Sel Risk--coverage under perturbations Clean 20\% label flips JPEG+resize Figure S5. Risk–coverage under label noise and compression. Risk–coverage curves for Pixelis (8B) on the shifted eval split. The푥-axis shows coverage (fraction of non-abstained queries) and the푦-axis shows Err@Sel (error among selected queries). We compare clean labels, 20% label flips, and JPEG+resize perturba- tions, averaging over tasks with shaded 95% confidence intervals. Across coverage levels, the curve under clean labels lies lowest, JPEG+resize remains close, and 20% label flips consistently increase risk, but the ordering of the three curves stays stable, indicating that our uncertainty-weighted consensus degrades gracefully under moderate noise and compression. or휆 pix to restore tool-aligned neighbors, then re-tighten훿to maintain the KL corridor. 10. Tool vs evaluator quick reference Purpose. This note disambiguates callable tools (invoked by the agent and returning replayable artifacts) from frozen evaluators/encoders (never called as actions, no gradients, used only for scoring or fixed features). Callable tools (agent executes). Each tool consumes typed arguments and emits replayable outputs saved in traces. • SEG(segmentation): args = region hints or prompts; outputs = mask/box set. • ZOOM/CROP: args = box or scale; outputs = crop box and cropped view. • TRK(tracking): args = init box/mask, frame range; outputs = per-frame boxes/IDs/tracklets. • READ(OCR): args = region box/mask; outputs = text tokens + spans. • TEMPORAL(localization): args = query + time window; outputs = temporal segment(s). Frozen evaluators / encoders (not tools). Used only to score outputs or extract fixed features; they are not executable actions and never optimized by the policy. •HOTA and CLEAR-MOT for multi-object tracking quality; implemented viaTrackEvaland applied to produced tracks for metric computation (not to control the tracker). HOTA evaluates higher-order associations; CLEAR-MOT provides MOTA/MOTP etc. •ANLS (Average Normalized Levenshtein Similarity) to score OCR-style answers; used as a verifier forREAD outputs and for InfoVQA-style evaluation. •DINOv2 visual embeddings as a fixed encoder for footprint descriptors and external RaCPR features; never fine-tuned within our pipeline. Quick mapping (role, I/O, usage). ItemRoleInput (typed)Output / Use SEGToolregion prompt, hintsmask/boxes; replay + retrieval keys ZOOM/CROP Toolbox/scalecrop box/view; replay + keys TRKToolinit box/mask, framestracklets (ID, box per frame) READToolbox/masktext tokens + spans TEMPORALToolquery, time windowstart/end indices HOTAEvaluator predicted vs GT tracksmetric only (no action) CLEAR- MOT Evaluator predicted vs GT tracksMOTA/MOTP (metric only) ANLSEvaluator predicted vs GT textsimilarity score (metric only) DINOv2Encoderimage crops/framesfixed features (no grads) Do-not-mix guidance. HOTA/CLEAR-MOT/ANLS are not trackers or OCR engines and must not appear in the action space; they only score the artifacts returned byTRK/READ, respectively. DINOv2 provides frozen descriptors for re- trieval and RaCPR computation and is not updated by C–RFT/TTRL. Implementation hook. We use ”Spatio Temporal Tracking” for HOTA/CLEAR-MOT scoring with official defaults and consistent IOU/association thresholds across methods for fairness. 11. Index whitelisting and dedup thresholds Whitelist protocol. To prevent evaluation leakage, all evalu- ation media IDs (image hashes, video GUIDs) are written to a whitelist mask at ingest time. Any item whose source ID matches the evaluation mask is rejected from the searchable index. For videos, we also store per-clip frame hashes to block cropped or re-encoded variants that reuse the same source. The mask is enforced before vector addition to FAISS so that excluded items never receive codes or list assignments (cf. IVF–PQ pipeline). See FAISS for IVF/PQ/OPQ indexing and search mechanics. Near-duplicate detection (two-stage). Stage–1 perceptual filtering uses perceptual hash (pHash) on RGB keyframes (images: single frame; videos: 1 fps, center-cropped). We flag pairs with Hamming distance푑 퐻 ≤ 8as near- duplicates. Stage–2 semantic filtering uses L2-normalized vision embeddings (CLIP/DINO family) with cosine similar- ity푠 = ⟨푓(푥), 푓(푦)⟩; we flag pairs with푠 ≥ 0.93(images) or푠 median ≥ 0.92across sampled frames (videos). These thresholds follow common practice in large-scale curation and yield high recall with low false positives; SSIM> 0.92 is used as a tie-breaker only when pHash and embedding signals disagree. Temporal consistency for videos. Given candidate pairs from Stage–1/2, we align uniformly sampled frames by timestamp and require at least70%of aligned frames to satisfy either푑 퐻 ≤ 8or푠 ≥ 0.93(plus SSIM> 0.90) to confirm a near-duplicate; otherwise the candidate is dropped. Thresholds and priorities. Perceptual hash is evaluated first for speed; any pHash-positive is removed without invoking embedding re-check. Embedding-only positives are removed unless the pair is a benign “template overlap” (e.g., same background, different foreground) detected by low SSIM < 0.75 and low localized overlap after mask erosion. Residual estimation and CIs. We estimate post-filter residual near-duplicate rate by uniform sampling without replacement.Let푘be the number of confirmed near- duplicates among푛inspected pairs. We report the Clop- per–Pearson exact95%CI for the binomial proportion푝using [ BetaInv(0.025; 푘,푛−푘+1),BetaInv(0.975; 푘+1,푛−푘)]. This matches standard practice for low-probability events. De-index vs. de-dup behavior. If an item is a near-duplicate of evaluation media, we reject at ingest (never added to IVF lists). If two training items are mutual near-duplicates, we keep the earliest ingest and drop the later one to stabilize neighbor counts; their IDs are added to a blocklist consulted during future adds. Audit procedure. We run the full pipeline offline to compute: (i) candidates flagged by Stage–1 and Stage–2, (i) removed after confirmation, (i) residual via spot-check with the exact CI above. For the retrieval index, we additionally probe a random 10k query set and assert zero whitelisted hits in the top–퐾(by masked search on FAISS); failures abort the build. Table S13. Near-duplicate thresholds and tie-breakers (default). SignalImagesVideos (per-frame / aggre- gate) Notes pHash(Ham- ming) 푑 퐻 ≤ 8푑 퐻 ≤ 8 (frame)Stage–1 filter Embedding co- sine 푠 ≥ 0.93푠 ≥ 0.93 (frame),푠 med ≥ 0.92 CLIP/DINO features SSIM (tie-break)> 0.92> 0.90Used when signals disagree Frame coveragen/a≥ 70% frames positiveTemporal consistency Whitelist rulehard blockhard blockBefore FAISS add Rationale and references. Perceptual hashing efficiently removes obvious duplicates; embedding similarity captures semantic duplicates and robust transforms; SSIM guards against background-only collisions. Our IVF–PQ/OPQ index and HNSW ablations follow standard ANN practice for scalability [21, 24]. 12. Clarification on Table 1 and abstract statis- tics During the final consolidation of Table 1, we re-ran all evaluations with (i) stricter de-duplication between training data and evaluation splits, (i) corrected per-benchmark splits (especially for MVBench and VSI-Bench), and (i) aligned scoring scripts for InfoVQA and Video-MMMU. These hygiene fixes introduce small absolute shifts (typically within ±0.3–0.6 points) to both the Pixel Reasoner baseline and our Pixelis model. All relative gains in Table 1 are computed as Δ rel = Pixelis− Baseline Baseline .(S2) BecauseΔ rel is sensitive to small changes in the baseline score, the updated table slightly changes the averaged number. In the original draft (and abstract, written before the hygiene fixes), the average relative gain across the six benchmarks was reported as≈ 4.08%with a peak improvement of≈ 6.03%. With the final consolidated evaluation pipeline, Table 1 now yields an average relative gain of roughly4–5%, with the largest improvement of about 7% on VSI-Bench. Importantly, these corrections do not change any trend: Pixelis consistently improves over the same 8B backbone on all six benchmarks, and the magnitude of the gains remains within the same range (roughly4–7%). All final model outputs and scoring scripts used to produce Table 1 will be released to ensure full reproducibility. 12.1. Note on reported numbers. All numerical summaries in the abstract were computed from an earlier version of the evaluation table before the above hygiene fixes were applied. All numbers in Table 1 and in this supplement reflect the final consolidated evaluation pipeline. The differences are minor and do not affect the ranking, trends, or any scientific claims of this work. 13. Notation Scalars are italic, vectors bold, trajectories use Greek letters, and policies are written as휋(·). We list only the symbols that are reused across sections. Indices and basic sets • 푡 ∈ 1,...,푇: step index in a toolchain / trajectory;푇or |휏| is the decisive chain length. • 푖: frame index within a video clip. • 푗 : rollout index in Pixel TTRL (trajectory sample). • 푛: query / example index. • 푐,ℓ ∈ 1,...,퐶 : class indices in Dawid–Skene style models; 퐶 is the number of answer classes. • 퐾: number of nearest neighbours / trajectories in retrieval and voting. States, trajectories, and tools • 휏: full reasoning trajectory (thoughts, tool calls, observa- tions). • 푠 푡 : agent state at step 푡 (pixels, text, and history). • 푎 푡 : action / tool call at step푡with serialized arguments; A is the discrete tool set. • 표 푡 : observation returned after executing 푎 푡 . • v 푡 : visual state embedding at step푡(frames, crops, masks). • x 푡 : textual state embedding at step푡(prompt and partial rationale). • 퐸 푡 :stepembeddingatstep푡,퐸 푡 = 푔 휙 ([v 푡 ∥x 푡 ∥표푛푒ℎ표푡(푎 푡−1 )])/∥푔 휙 (·)∥ 2 ∈ R 512 . • SEG,TRK,OCR,ZOOM,TEMP,PROP: segmentation, track- ing, text reading, zoom/crop, temporal localization, and property-query tools. Policies and probabilities • 휋 휃 : current Pixelis policy (after SFT / C-RFT / TTRL), with parameters 휃. • 휋 SFT : supervised fine-tuned base policy used for initializa- tion. • 휋 EMA : exponential-moving-average reference policy for KL anchoring. • 푝 휃 (푦 | 휏): answer distribution given trajectory휏under 휋 휃 . •흅: class prior vector in the Dawid–Skene model. • Π (푗) ∈ R 퐶×퐶 : confusion matrix of rollout푗, entries 휋 (푗) 푐ℓ = 푃(푧 푗 = ℓ | 푦 = 푐). Rewards, scores, and metrics • 푟 cur 푡 : prediction-error curiosity reward at step 푡. • 푟 coh 푡 : adjacent-step coherence reward (z-scored cosine of 퐸 푡 , 퐸 푡+1 ). • 푅(휏): total trajectory reward (weighted sum of curiosity, coherence, and answer correctness). • 푆 logic (휏),푆 struct (휏),푆 visual (휏): logic, structure, and visual components of the trajectory score. • 푆(휏) = 훼푆 logic + 훽푆 struct + 훾푆 visual : combined trajectory score. •RaPR, RaCPR: rationalized process recall and chain preci- sion–recall for tool use and composition. •VisFid: visual fidelity score aggregated over tools (IoU / ANLS / HOTA / tIoU). • Acc : answer accuracy;Err@Sel: error among selected (non-abstained) queries at a given coverage. • cov: coverage, fraction of queries that are not abstained. Losses, KL control, and voting • L SFT ,L RFT ,L TTRL : objectives for SFT, C-RFT, and Pixel TTRL. • KL(휋 휃 ∥ 휋 EMA ): token-level KL divergence between cur- rent and EMA policies. • [KL min , KL max ]: KL corridor (e.g.,[0.10, 0.20]) used for safe adaptation; KL tgt is the target value. • 훽 푡 : adaptive KL weight at step푡from the PID controller (gains 퐾 푝 ,퐾 푖 ,퐾 푑 ). • 푤 푗 : vote weight for rollout푗;푟 푗 : DS-based reliability proxy for rollout 푗 . • 푇 temp : temperature for probability calibration (temperature scaling). • ECE: expected calibration error; abstain: abstention deci- sion used in risk–coverage curves. Question: The first image shows a bed with three sleeping animals: two dogs and one cat. The second image shows a sofa with two sleeping cats. Using only pixel-verifiable evidence obtained via the tools zoom in, segment object at, get properties, and read text, describe how the “who is asleep where” configuration changes. Please identify: 1) how many asleep “woof” (dogs) and asleep “meow” (cats) appear in each image; 2) how their grouping and placement on the bed/sofa surface changes; 3) how their relation to the book or remote controls changes. What is the transformation from the initial three-animal nap to the final two-cat nap duo? Image0 Image1 <answer> Image_0 has two asleep dogs and one asleep cat on a bed, forming a spread-out trio with a book at the lower-left corner. Image_1 has no dogs, only two asleep cats on a sofa, forming a left– right pair beneath two remote controls. So the sleeping configuration changes from “2 asleep woof + 1 asleep meow on a book-anchored bed” to “0 asleep woof + 2 asleep meow in a two-cat sofa layout anchored by remotes.” </answer> <think> Step 1: Get a coarse view of image_0. - zoom in (image_0, [0, 0, 640, 480]) → bed surface with three fur-like blobs and one flat rectangle in the lower-left. Step 2: Segment the main regions in image_0. - segment object at (image_0, points=[[180,390],[120,250],[380,190],[80,400]]) → three animal- shaped regions (large lower, medium upper-left, small upper-middle) and one flat rectangular region. From outline and texture: large + medium = sleeping dogs, small = sleeping cat, rectangle = book-like object near the lower-left edge. ⇒ image_0: 2 asleep dogs, 1 asleep cat, spread-out trio on a book-anchored bed. Step 3: Get a coarse view of image_1. - zoom in (image_1, [0, 0, 640, 480]) → sofa surface with two fur-like blobs and two thin rectangles above them. Step 4: Segment the main regions in image_1. - segment object at (image_1, points=[[180,300],[440,300],[230,90],[360,90]]) → two animal- shaped regions (left, right) and two thin rectangular regions. From shape and symmetry: left + right = sleeping cats, thin rectangles = remote controls. ⇒ image_1: 0 asleep dogs, 2 asleep cats, left–right pair anchored by remotes. Overall, the pixel evidence shows a change from “2 woof + 1 meow on a book-anchored bed” to “0 woof + 2 meow in a paired cat layout on a remote-anchored sofa”. </think> SFT-COTA Figure S6. SFT example. “Who is asleep where”: Image0 has 2 woof + 1 meow, Image1 has 0 woof + 2 meow. Question: Three frames show a small child standing by a decorated Christmas tree. In the first two frames, the child reaches into the branches and nudges the hanging ornaments; nothing is picked up, dropped, or placed anywhere. In the final frame, the child steps back with empty hands while the ornaments remain exactly where they were. Based only on this pixel evidence, choose the category that best describes the child’s interaction. (A) Poking a stack of something without the stack collapsing (B) Pretending to put something next to something (C) Tilting something with something on it until it falls off (D) Unfolding something (E) Pushing something from left to right (F) Holding something over something (G) Dropping something onto something (H) Pretending to put something into something (I) Putting something on a flat surface without letting it roll (J) Pretending to put something onto something Trajectory A <think> Step 1: Locate the child near the tree in frame_0. - zoom in (frame_0, [700,200,1800,1200]) → upper body of a small child facing the lit tree, arm extended into branches. Step 2: Segment the hand region touching the ornaments. - segment object at (frame_0, point=[1200,500]) → hand-sized region overlapping small bright ornaments. Step 3: Track this region across frames. - track object (id=hand_0, from=0, to=2) → motion: hand moves around within the ornament cluster in frames 0–1, then retracts as the child steps back in frame_2; no ornament leaves the tree, no new object. Step 4: Inspect final pose. - get properties (object=child_2) → action: stepping back, hands near torso; no placing, dropping or carrying. Conclusion from pixel evidence: The child is repeatedly prodding the decorations so they move slightly but stay attached. This matches “poking a stack of something without the stack collapsing”. </think> Chosen option: (A) CCRFT-COT Trajectory B <think> Step 1: Look vaguely at frame_0. - zoom in (frame_0, [0, 0, 2048, 1231]) → sees a tree with lights and a child somewhere to the right; no detail extracted. Step 2: Make a guess based on a single ornament region. - segment object at (frame_0, point=[300,500]) → returns only a small ornament patch; no child motion or hand interaction visible. Conclusion: Since an ornament is visible, assume the child is “pretending to put something onto something”. </think> Chosen option: (J) Figure S7. RFT example. Multiple candidate tool-use trajectories are compared to learn preferences over answer correctness, curiosity, coherence, and penalty. Trajectory C <think> Step 1: Focus only on frame_1. - zoom in (frame_1, [700,200,1800,1200]) → child’s hand close to ornaments, arm extended toward tree. Step 2: Segment the local region around the hand. - segment object at (frame_1, point=[1250,530]) → small region containing part of the hand and a hanging ornament. Step 3: Single-frame guess. - get properties (region) → hand is near an object on the tree; depth is ambiguous, and no motion over time is considered. Conclusion: Treats the pose as if the child were about to place something next to an object, and guesses a “pretending to put something next to something” interaction. </think> Chosen option: (B) Trajectory D <think> Step 1: Look away from the tree and child. - zoom in (frame_2, [1450,400,1950,1100]) → only a gray box and floor area, no visible hand or ornaments. Step 2: Segment a random patch on the box. - segment object at (frame_2, point=[1600,800]) → flat rectangular region with no motion cues. Step 3: Misinterpret the scene. - get properties (region) → static object on the floor; no interaction, but a placing action is guessed. Conclusion: Ignores the child and the tree entirely and incorrectly infers an action like placing an object on a flat surface. </think> Chosen option: (I) <answer> (A); Across the three frames, the child reaches into the Christmas tree and nudges the hanging ornaments so they move slightly, but never picks up, places, or drops anything. In the final frame the child has stepped back with empty hands, and all ornaments remain attached with nothing new on the floor or nearby surfaces. This is best described as poking a stable “stack” of decorations without making it collapse. Therefore the correct category is (A) “Poking a stack of something without the stack collapsing.” </answer> RFT reward (answer, curiosity, coherence, penalty): Trajectory A: [0.95, 0.70, 0.40, 0.17] Trajectory B: [0.30, 0.44, 0.27, 0.11] Trajectory C: [0.00, 0.15, 0.16, 0.06] Trajectory D: [0.00, 0.12, 0.02, 0.02] Figure S8. RFT example. Multiple candidate tool-use trajectories are compared to learn preferences over answer correctness, curiosity, coherence, and penalty. Question: Three frames show the same seated woman while a stylist works on her long wavy hair. Based only on this pixel evidence, describe in one short sentence what the stylist is doing to the woman’s hair. Tools: zoom in, select frame, track object, segment object at, get properties, and read text, (Hidden Answer Description: In frame_0, the stylist stands slightly behind her, lifting hair near the crown while a thin headband rests over the top of her head. In frame_1, we see the back of her head as the stylist pulls a side section of hair backward and begins to twist it, while the rest of the hair hangs loose in soft curls. In frame_2, the stylist continues twisting that side section downward along the back of the head; the remaining hair still looks long, loose, and uncut, and no spray cloud or tools appear. ) Trajectory τ1 <think> Step 1: Focus on the woman and stylist in the first frame. - zoom in (frame_0, [450,200,1200,900]) → upper body of the seated woman; stylist behind her with one hand lifting hair near the crown; headband already on her head. Step 2: Isolate the section of hair that is being manipulated later. - segment object at (frame_1, point=[340,650]) → side section of hair pulled from the left toward the back (hair_side_section). Step 3: Track this hair section across time. - track object (mask_id=hair_side_section, from=1, to=2) → same strand moves from a loose side position in frame_1 to a tighter, rope-like shape along the back in frame_2; hair length stays the same, no pieces disappear. Step 4: Inspect the stylist’s hand and nearby tools in the last frame. - get properties (stylist_hand, frame_2) → fingers wrapped around the same strand, rotating it downward; no scissors, no hot tool, no visible spray plume. Conclusion from pixel evidence: The stylist is repeatedly twisting one side section of the woman’s hair along the back of her head while leaving the rest of the hair long and loose. </think> <answer> "The stylist is twisting a side section of the woman’s hair into a half-up style." </answer> Pixel TTRL-COT Trajectory τ2 <think> Step 1: Zoom into items on the side table instead of the hair. - zoom in (frame_2, [1150,550,1650,900]) → several bottles and tubes. Step 2: Segment one tall bottle. - segment_object_at(frame_2, point=[1350,700]) → a styling-product bottle with a nozzle. Step 3: Infer the action from the bottle rather than the hair. - get properties (bottle_0) → looks like a spray container; assume spraying is occurring. </think> <answer> "The stylist is spraying hair product near the woman’s hair." </answer> Figure S9. TTRL constructs an empirical action distribution from the winning trajectories’ tool calls and nudges the policy toward this distribution under a KL-to-EMA corridor. Trajectory τ3 <think> Step 1: Compare the overall hairstyle between early and late frames. - zoom in (frame_0, [450,200,1200,900]) → hair falls freely around the shoulders, with no visible twist at the back. - zoom in (frame_2, [430,300,1250,1000]) → a compact twisted strand now runs diagonally along the back, while most hair still hangs loose. Step 2: Segment the twisted part in the final frame. - segment object at (frame_2, point=[700,760]) → hair_twist_section: a bundled, rope-like section of hair near the back of the head. Step 3: Check for signs of cutting or heat tools. - get properties (hair_twist_section) → continuous from the scalp, same color and thickness as surrounding hair; no sharp cut edge. - get properties (stylist_hand, frame_2) → bare hand gripping hair; no scissors, no curling iron in contact. Conclusion from pixel evidence: The stylist has gathered part of the hair and twisted it back into place without shortening it or changing the rest of the hair, forming a controlled half-up twist. </think> <answer> "The stylist is twisting part of the hair back into a half-up hairstyle." </answer> Trajectory τ4 <think> Step 1: Focus on the frame where manipulation of the strand is clearest. - zoom in (frame_1, [500,200,1100,1000]) → back of the woman’s head with loose curls and a visible headband. Step 2: Segment the strand in the stylist’s hand. - segment object at (frame_1, point=[320,640]) → a side strand of hair being pulled backward and partially twisted. Step 3: Inspect local motion and contact. - get properties (hair_side_section, frame_1) → strand folded and wrapped around itself; direction of motion is around its own axis, not simply straight down. - get properties (stylist_hand_left, frame_1) → fingers pinching and rotating the strand, not brushing along its length. Conclusion from pixel evidence: In this key frame the stylist is actively twisting a side section of hair at the back of the head rather than brushing, cutting, or spraying it. </think> <answer> "The stylist is twisting a side section of the woman’s hair at the back of her head." </answer> τ1 → "The stylist is twisting a side section of the woman’s hair into a half-up style."; τ2 → "The stylist is twisting part of the hair back into a half-up hairstyle."; τ3 → "The stylist is twisting a side section of the woman’s hair at the back of her head."; τ4 → "The stylist is spraying hair product near the woman’s hair without directly styling it." Answer majority vote → twist-half-up (3 / 4 trajectories) Trajectory-level vote (in tool space): Winning trajectory set 풯_win: τ1, τ3, τ4, which all share a hair-centric tool pattern: - zoom into the hair region, - segment a hair strand, - optionally track it across frames, - inspect stylist hand + hair properties. Losing trajectory: τ2, which follows a bottle-centric pattern: - zoom into table items, - segment a product bottle, - inspect bottle appearance only. Figure S10. The “vote” selects trajectories (zoom/segment/track patterns), not just answer phrases.