Paper deep dive
LPA-CWM: A Learned Physical Adjudicator for Motion Reasoning with Counterfactual World Models
Kunwei Wu, Xiang Liu, Guocai Yao, Junming Chen, Zhikang Chen, Min Zhang, Pengwei Wang, Sen Cui
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/20/2026, 4:17:26 AM
Summary
The paper introduces LPA-CWM, a method for motion reasoning that uses a lightweight Learned Physical Adjudicator (LPA) to weight counterfactual world model (CWM) responses based on reliability, rather than uniform aggregation. It also proposes the Completeness-aware Motion Correspondence (CMC) evaluation protocol. LPA-CWM significantly improves motion tracking accuracy on DAVIS and Kinetics datasets compared to baseline methods.
Entities (10)
Relation Signals (8)
LPA-CWM → usescomponent → Learned Physical Adjudicator
confidence 95% · We formulate response aggregation as candidate reliability learning and propose LPA-CWM with a lightweight Learned Physical Adjudicator (LPA).
Completeness-aware Motion Correspondence → proposedby → LPA-CWM
confidence 94% · We also introduce Completeness-aware Motion Correspondence (CMC), a ground-truth-anchored evaluation protocol
LPA-CWM → outperforms → Uniform CWM
confidence 92% · LPA-CWM improves DCAavg over Uniform CWM by 60.0% and 29.0%, respectively
LPA-CWM → evaluatedon → Kinetics
confidence 90% · On the evaluated DAVIS and Kinetics subsets, LPA-CWM improves DCAavg over Uniform CWM by 60.0% and 29.0%, respectively
LPA-CWM → evaluatedon → Davis
confidence 90% · On the evaluated DAVIS and Kinetics subsets, LPA-CWM improves DCAavg over Uniform CWM by 60.0%
LPA-CWM → trainedon → MOVi-F
confidence 88% · Trained on dense MOVi-F trajectories, the 3.0M-parameter LPA compares visual context and response structure
LPA-CWM → outperforms → APA-CWM
confidence 85% · Compared with external methods, Uniform CWM, and APA-CWM, LPA-CWM performs strongly across all four dimensions.
APA-CWM → usescomponent →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Counterfactual world models (CWM) extract motion from pretrained video predictors by comparing factual and intervened predictions. However, responses generated under different target-frame masks vary in reliability, while uniform aggregation weights them equally. We formulate response aggregation as candidate reliability learning and propose LPA-CWM with a lightweight Learned Physical Adjudicator (LPA). Trained on dense MOVi-F trajectories, the 3.0M-parameter LPA compares visual context and response structure across an unordered candidate set to predict relative weights, while the CWM predictor and intervention generator remain frozen. The weighted responses undergo windowed localization and one paired re-evaluation to recover motion. We also introduce Completeness-aware Motion Correspondence (CMC), a ground-truth-anchored evaluation protocol that jointly measures localization, trajectory completeness, visibility, and continuity, counting missing predictions as failures on visible dynamic points. On the evaluated DAVIS and Kinetics subsets, LPA-CWM improves $\mathrm{DCA}_{\mathrm{avg}}$ over Uniform CWM by 60.0\% and 29.0\%, respectively, and also improves tracking accuracy under TAP-Vid First. A quick overview is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.14073v1
- Canonical: https://arxiv.org/abs/2609.14073v1
Trouble viewing inline? Open PDF directly →
Full Text
74,860 characters extracted from source content.
Expand or collapse full text
LPA-CWM: A Learned Physical Adjudicator for Motion Reasoning with Counterfactual World Models Kunwei Wu1,* Xiang Liu2,* Guocai Yao3 Junming Chen4 Zhikang Chen5 Min Zhang6 Pengwei Wang3 Sen Cui2,† 1National University of Singapore 2Tsinghua University 3Beijing Academy of Artificial Intelligence 4Facebook 5University of Oxford 6East China Normal University *Equal contribution † Leader & Corresponding Author Contact: wukunwei@u.nus.edu | xiang-liu25@mail.tsinghua.edu.cn | cuis@mail.tsinghua.edu.cn Abstract Counterfactual world models (CWM) extract motion from pretrained video predictors by comparing factual and intervened predictions. However, responses generated under different target-frame masks vary in reliability, while uniform aggregation weights them equally. We formulate response aggregation as candidate reliability learning and propose LPA-CWM with a lightweight Learned Physical Adjudicator (LPA). Trained on dense MOVi-F trajectories, the 3.0M-parameter LPA compares visual context and response structure across an unordered candidate set to predict relative weights, while the CWM predictor and intervention generator remain frozen. The weighted responses undergo windowed localization and one paired re-evaluation to recover motion. We also introduce Completeness-aware Motion Correspondence (CMC), a ground-truth-anchored evaluation protocol that jointly measures localization, trajectory completeness, visibility, and continuity, counting missing predictions as failures on visible dynamic points. On the evaluated DAVIS and Kinetics subsets, LPA-CWM improves DCAavgDCA_avg over Uniform CWM by 60.0% and 29.0%, respectively, and also improves tracking accuracy under TAP-Vid First. A quick overview is available at https://LPA-CWM.github.io. Figure 1: An introduction of our work. Uniform CWM can suffer from three characteristic failures in motion reasoning: drifting from moving objects, being distracted by competing motion, and losing track under appearance changes. Our Learned Physical Adjudicator (LPA) identifies reliable counterfactual motion evidence, enabling LPA-CWM to recover more coherent trajectories. We further introduce the CMC Benchmark to evaluate motion correspondence across localization, completeness, visibility, and continuity. Compared with external methods, Uniform CWM, and APA-CWM, LPA-CWM performs strongly across all four dimensions. 1 Introduction World models learn predictive representations of visual dynamics (Ha and Schmidhuber, 2018; Hafner et al., 2019; Hafner et al., 2020). Counterfactual World Modeling (CWM) extracts motion from these representations by comparing predictions with and without localized interventions (Bear et al., 2023; Venkatesh et al., 2024; Stojanov et al., 2025). Different target-frame masks can produce inconsistent responses for the same query: some localize the motion well, while others are affected by background content, competing motion, or visual ambiguity. Uniform aggregation treats all responses as equally reliable and can dilute useful evidence. This raises two questions: how should counterfactual motion evidence be combined, and how should the recovered trajectories be evaluated? To improve CWM inference, we formulate multi-mask aggregation as candidate reliability estimation. We first derive APA-CWM, an Analytic Physical Adjudicator (APA) from the Normalized Spatiotemporal Gradient (NSG) (Zhang et al., 2025). APA reads the NSG-derived spatiotemporal consistency around each candidate response peak and converts these local scores into closed-form reliability weights. It provides a training-free physical reference for testing how far an analytic consistency prior can resolve candidate ambiguity. We then propose LPA-CWM, whose Learned Physical Adjudicator (LPA) combines local visual evidence, response morphology, shared frame-pair context, and candidate statistics. A Transformer without candidate-index positional embeddings jointly compares the unordered candidate set and predicts relative weights. We train only the 3.0M-parameter LPA on dense Kubric MOVi-F trajectories (Greff et al., 2022). For each query, dense forward motion provides the oracle target endpoint; candidate endpoint errors define relative reliability targets and supervise distribution matching, weighted-coordinate localization, and best-candidate ranking. This gives direct supervision for which CWM responses are trustworthy while leaving the pretrained CWM unchanged. At inference, LPA weights the initial responses, followed by windowed localization and one paired CWM re-evaluation to obtain the final motion. Existing benchmarks assess video quality, temporal consistency, physical and commonsense faithfulness, controllability, embodied utility, and counterfactual consistency (Huang et al., 2024; Zheng et al., 2025; Duan et al., 2025; Shang et al., 2026; Begiristain et al., 2026). They broaden evaluation beyond visual plausibility, but do not directly measure point correspondence recovered through CWM interventions. In motion recovery, low error on returned predictions can hide missing trajectory segments, while smooth trajectories may still follow incorrect motion. We therefore introduce Completeness-aware Motion Correspondence (CMC), a ground-truth-anchored protocol built on TAP-Vid annotations (Doersch et al., 2022). CMC evaluates localization, completeness, visibility, and temporal continuity on dynamic point tracks. Dynamic Correspondence Accuracy requires visible predictions within a distance threshold and counts missing predictions as failures on ground-truth-visible points. Track-level success further requires sufficient coverage and bounded localization error, while visibility and continuity diagnostics expose interrupted trajectories. CMC complements TAP-Vid First by making completeness on moving tracks an explicit target. We evaluate on DAVIS and Kinetics (Pont-Tuset et al., 2017; Kay et al., 2017) against SMURF, GMRW, DODUO, CoTracker3, RAFT, and SEA-RAFT, covering self-supervised flow, global correspondence, dense optical flow, and dedicated point tracking (Stone et al., 2021; Shrivastava and Owens, 2024; Jiang et al., 2024; Karaev et al., 2025; Teed and Deng, 2020; Wang et al., 2024). LPA-CWM improves DCAavgDCA_avg over Uniform CWM by 60.0% on DAVIS and 29.0% on Kinetics. On Kinetics it achieves the best value across all reported CMC metrics; on DAVIS it gives the highest coverage, Complete@0.8, and continuity, while CoTracker3 remains stronger on several conventional tracking measures. Within the CWM family, LPA-CWM also improves all TAP-Vid First measures over Uniform CWM and APA-CWM. In summary, our main contributions are: • We formulate multi-mask CWM inference as candidate reliability estimation and introduce analytic and learned reliability weighting for counterfactual motion responses. • We propose LPA-CWM, a lightweight physically supervised adjudicator trained from dense MOVi-F trajectories, with windowed localization and one paired CWM re-evaluation. • We introduce CMC for joint evaluation of localization, completeness, visibility, and continuity, and show strong gains over Uniform CWM, APA-CWM, and external baselines. 2 Related Work World models and counterfactual motion. Video world models learn predictive representations through feature prediction, as in V-JEPA (Bardes et al., 2024), or action-controllable generation, as in Genie (Bruce et al., 2024). CWM extracts correspondence by comparing factual and locally intervened predictions (Bear et al., 2023), and later work applies this interface to physical dynamics (Venkatesh et al., 2024). Opt-CWM learns query-conditioned perturbations for flow and occlusion (Stojanov et al., 2025); KL-tracing measures the response of generative video models through KL divergence (Kim et al., 2025); and Point Prompting propagates colored markers through video diffusion models (Shrivastava et al., 2026). LPA-CWM focuses on learning the relative reliability of the multiple responses already produced by a frozen CWM. Diffusion models as structural priors. Diffusion features also support visual correspondence. SD4Match adapts Stable Diffusion features with image-pair-conditioned prompts (Li et al., 2024), while Track4Gen jointly trains video generation and point correspondence to reduce appearance drift (Jeong et al., 2025). NSG relates a diffusion score field to observed inter-frame changes as a spatiotemporal consistency signal (Zhang et al., 2025). APA-CWM uses this signal for analytic candidate weighting, while LPA-CWM learns the weights from trajectory supervision. Flow, correspondence, and point tracking. Our external baselines cover complementary motion interfaces. RAFT uses all-pairs correlations with iterative flow refinement (Teed and Deng, 2020), and SEA-RAFT improves the framework with probabilistic training and a more efficient design (Wang et al., 2024). SMURF learns optical flow through full-image warping and multi-frame self-supervision (Stone et al., 2021). GMRW learns global correspondence with contrastive random walks (Shrivastava and Owens, 2024), whereas DODUO combines semantic priors with self-supervised flow (Jiang et al., 2024). CoTracker3 is a dedicated point tracker trained with pseudo-labels from real videos (Karaev et al., 2025); AllTracker extends dense long-range tracking with multi-frame flow and temporal attention (Harley et al., 2025). Evaluation of motion consistency. General video and world-model benchmarks measure complementary properties: VBench targets visual quality and temporal consistency (Huang et al., 2024), VBench-2.0 adds physical and commonsense faithfulness (Zheng et al., 2025), WorldScore evaluates controllability, quality, and dynamics (Duan et al., 2025), WorldArena includes embodied task utility (Shang et al., 2026), and CRONOS tests counterfactual physical consistency (Begiristain et al., 2026). Motion-specific evaluation includes FVMD for point-track motion distributions (Liu et al., 2024), Physics-IQ for prediction against real physical experiments (Motamed et al., 2026), and TAPVid-3D for long-range 3D correspondence and visibility (Koppula et al., 2024). CMC instead evaluates recovered 2D dynamic tracks, counting missing visible predictions as failures and reporting completeness and continuity alongside localization. 3 Method Given two consecutive frames and a query point, our method first uses a frozen counterfactual world model (CWM) to produce multiple motion candidates. We then compare two ways to weight these candidates: an analytic rule, APA-CWM, and a learned adjudicator, LPA-CWM. Both variants combine the candidate response maps before applying the same localization and re-evaluation procedure. Figure 2 summarizes the method. Figure 2: Overview of APA-CWM and LPA-CWM. Given consecutive frames (t,t+1)(I_t,I_t+1) and a query point tp_t, a frozen CWM generates motion candidates m\ _m\. APA-CWM derives candidate weights from the frozen ADM score map P around candidate peaks ^t+1(m) p_t+1^(m). LPA-CWM instead encodes local crops, full responses, the frame pair, and response statistics; the resulting tokens mz_m are processed by a trainable Set Transformer to predict weights. The weighted response is locally decoded and re-evaluated once by the CWM to obtain refined motion. During LPA training, endpoint errors from MOVi-F trajectories are converted into soft targets q. 3.1 Counterfactual Motion Candidates from a Frozen World Model Let t,t+1∈[0,1]256×256×3I_t,I_t+1∈[0,1]^256× 256× 3 be two consecutive RGB frames, and let t∈Ωp_t∈ be a query location in the first frame, where Ω is the 256×256256× 256 image grid. We build on counterfactual world models (Bear et al., 2023; Venkatesh et al., 2024; Stojanov et al., 2025), consisting of an RGB predictor ΨRGB ^RGB and a query-conditioned intervention generator θ δ_θ. Both ΨRGB ^RGB and θ δ_θ are pretrained and remain fixed. The generator produces the intervention t=θ(t,t)∈ℝ256×256×3 δ_t= δ_θ(I_t,p_t) ^256× 256× 3. For each target-frame mask ℳmM_m, CWM measures how this intervention changes the target prediction: m=ΨRGB(t+t,ℳm(t+1))−ΨRGB(t,ℳm(t+1)). _m= ^RGB\! (I_t+ δ_t,M_m(I_t+1) )- ^RGB\! (I_t,M_m(I_t+1) ). (1) Thus, m() _m(u) measures how much the prediction at target pixel u changes when the source query is intervened. A large response indicates a strong dependency between that target location and the queried source region. Repeating this process for M masks produces t=m,^t+1(m)m=1M,^t+1(m)=argmax∈Ω‖m()‖1.C_t= \ _m, p_t+1^(m) \_m=1^M, p_t+1^(m)= _u∈ \| _m(u) \|_1. (2) Here, u indexes target-frame pixels and the ℓ1 _1 norm reduces the three RGB channels. The peak ^t+1(m) p_t+1^(m) is the target location with the strongest response under mask m, and therefore serves as that mask’s proposed endpoint for the query tp_t. Because different masks expose different target contexts, their response maps and candidate endpoints can disagree. Uniform CWM averages all responses equally, whereas APA-CWM and LPA-CWM estimate candidate-specific weights. 3.2 Analytic Candidate Weighting APA-CWM provides a closed-form reference based on the Normalized Spatiotemporal Gradient (NSG) (Zhang et al., 2025). Let t,t+1∈[−1,1]224×224×3x_t,x_t+1∈[-1,1]^224× 224× 3 denote the frame pair (t,t+1)(I_t,I_t+1) after preprocessing for a frozen Ablated Diffusion Model (ADM) (Dhariwal and Nichol, 2021). The notation distinguishes these ADM inputs from the 256×256256× 256 RGB frames used by CWM. Let ϕs_φ be the frozen ADM score network, tst_s its diffusion time, and u a location on the ADM grid. Following NSG, we compute a spatial score P that compares the local ADM score magnitude with its alignment to the observed frame change: ()=‖ϕ(t,ts)()‖2|⟨ϕ(t,ts)(),t+1()−t()⟩|+10−6.P(u)= \|s_φ(x_t,t_s)(u) \|_2 | _φ(x_t,t_s)(u),x_t+1(u)-x_t(u) |+10^-6. (3) The numerator measures the local score magnitude, while the denominator measures its projection onto the observed inter-frame change. The constant 10−610^-6 prevents numerical instability when this inner product is close to zero. APA treats the resulting P map as a local physical consistency cue for judging the CWM candidates. We bilinearly resize P to the CWM response grid and keep the same notation for the aligned map. For each candidate, APA averages this score in the radius-six neighborhood 6N_6 around its response peak and then normalizes the candidate scores: rmAPA=1|6(^t+1(m))|∑∈6(^t+1(m))(),wmAPA=clip(rmAPA,10−3,103)∑jclip(rjAPA,10−3,103).r_m^APA= 1 |N_6( p_t+1^(m)) | _u _6( p_t+1^(m))P(u), w_m^APA= clip(r_m^APA,10^-3,10^3) _jclip(r_j^APA,10^-3,10^3). (4) Here, rmAPAr_m^APA is the local NSG score for candidate m. Clipping limits extreme values, and normalization converts these scores into nonnegative candidate weights wmAPAw_m^APA that sum to one. APA thus provides a training-free way to favor candidates supported by the NSG cue. Since this cue is not directly supervised by endpoint accuracy, we next learn candidate reliability from dense motion trajectories. 3.3 Learned Physical Adjudicator Candidate representation and set reasoning. LPA represents each candidate using four sources of information. (1) Local visual evidence: the local branch receives a 10-channel 32×3232× 32 tensor containing source and target RGB crops, their absolute RGB difference, and a crop of the candidate response. These inputs correspond to 64×6464× 64 regions centered at tp_t and ^t+1(m) p_t+1^(m). Before cropping, the RGB response is reduced and normalized as d~m()=‖m()‖1/(max‖m()‖1+10−8) d_m(u)=\| _m(u)\|_1/( _v\| _m(v)\|_1+10^-8). (2) Full response: a second branch encodes the complete d~m d_m map to retain the global shape and ambiguity of the response. (3) Frame context: a third branch encodes the full frame pair to provide scene-level visual information shared by all candidates. (4) Candidate statistics: the final branch encodes 16 scalar descriptors covering displacement, response concentration, ambiguity, photometric change, and edge strength. The exact descriptors and encoder architectures are given in Appendix A. The four encoded features are combined and projected to a 256-dimensional candidate token mz_m. We denote the Transformer encoder without candidate-order embeddings by SetEncSetEnc. It compares all candidate tokens, after which a shared scoring head hscoreh_score produces one score per candidate: [~1,…,~M]=SetEnc([1,…,M]),wmLPA=exp(hscore(~m))∑jexp(hscore(~j)). [ z_1,…, z_M ]=SetEnc ([z_1,…,z_M] ), w_m^LPA= (h_score( z_m)) _j (h_score( z_j)). (5) The softmax converts these scores into normalized LPA weights wmLPAw_m^LPA. Without candidate-order embeddings, permuting the input candidates only permutes their output weights, while the weighted response remains unchanged. We also randomize candidate order during training. Dense-trajectory supervision. We train LPA on M=10M=10 candidate sets generated from adjacent MOVi-F frames (Greff et al., 2022); training and validation videos are disjoint. MOVi-F provides a dense ground-truth forward motion field t→t+1F_t→ t+1, where t→t+1(t)F_t→ t+1(p_t) is the displacement of tp_t from frame t to t+1t+1. The corresponding ground-truth endpoint is t+1∗=t+t→t+1(t)p_t+1^*=p_t+F_t→ t+1(p_t). We use this endpoint to measure how reliable each candidate is. Specifically, eme_m is the endpoint error of candidate m, and these errors are converted into a soft reliability target =(q1,…,qM)q=(q_1,…,q_M): em=‖^t+1(m)−t+1∗‖2,qm=exp(−em/τq)∑jexp(−ej/τq),τq=4.e_m= \| p_t+1^(m)-p_t+1^* \|_2, q_m= (-e_m/ _q) _j (-e_j/ _q), _q=4. (6) Candidates closer to the ground-truth endpoint receive larger qmq_m. Unlike a hard label that selects only the single best candidate, q retains the relative quality of all candidates. The temperature τq _q controls how strongly the target distribution favors candidates with lower endpoint error. Training objective. We train LPA with the following objective: ℒ=DKL(∥LPA)+λ1SmoothL1(∑mwmLPA^t+1(m),t+1∗)+λ2CE(,argminmem),L=D_KL (q\|w^LPA )+ _1SmoothL1\! ( _mw_m^LPA p_t+1^(m),p_t+1^* )+ _2CE\! (a, _me_m ), (7) where =hscore(~m)m=1Ma= \h_score( z_m) \_m=1^M denotes the pre-softmax scores. The KL term matches the predicted weights to the soft reliability targets q. The SmoothL1 term encourages the weighted candidate endpoint toward the ground-truth endpoint, while the cross-entropy term encourages the lowest-error candidate to receive the highest score. In our experiments, we set λ1=2 _1=2 and λ2=0.2 _2=0.2. The weighted candidate endpoint in the SmoothL1 term is used only as a differentiable training signal. During inference, LPA does not directly average candidate peak coordinates; instead, its predicted weights are applied to the full CWM response maps, which are then decoded as described next. MOVi-F correspondence filtering and further training details are given in Appendix A. 3.4 Weighted Aggregation and Localized Decoding Weighted response aggregation. Let wmw_m denote the uniform, analytic, or learned candidate weight used by Uniform CWM, APA-CWM, or LPA-CWM, respectively. We first combine the full candidate responses and reduce the three RGB channels to a scalar localization map: ¯w=∑m=1Mwmm,d¯w()=13‖¯w()‖1,wm≥0,∑mwm=1. _w= _m=1^Mw_m _m, d_w(u)= 13 \| _w(u) \|_1, w_m≥ 0, _mw_m=1. (8) Here, ¯w _w is the weighted CWM response and d¯w() d_w(u) measures its response strength at pixel u. Larger values indicate locations more strongly supported by the aggregated candidates. Windowed localization and re-evaluation. To locate the endpoint, we turn d¯w d_w into a normalized spatial distribution π(0)()∝exp(βπd¯w())π^(0)(u) ( _π d_w(u)), with βπ=200 _π=200, normalized over all ∈Ωu∈ . Thus, π(0)()π^(0)(u) gives the relative support for pixel u as the endpoint, while π(k)π^(k) denotes this distribution after k windowing updates. Standard localization directly selects the peak of π(0)π^(0); our windowed localization instead applies two updates to suppress distant secondary peaks. Figure 3: Windowed localization progressively suppresses distant modes. Top: flattened maps from π(0)π^(0) to π(2)π^(2). Bottom: the corresponding 3D views. At each update, we compute the spatial mean (k) μ^(k) and spread σ(k)σ^(k) of the current distribution, then apply a Gaussian window centered on the dominant region: (k)=∑π(k)(),σ(k)=[∑π(k)()‖−(k)‖22]1/2,π(k+1)()∝π(k)()exp[−‖−(k)‖222[3max(σ(k),0.5)]2]. cases μ^(k)= _uπ^(k)(u)u,\\ σ^(k)= [ _uπ^(k)(u) \|u- μ^(k) \|_2^2 ]^1/2,\\ π^(k+1)(u) π^(k)(u) \! [- \|u- μ^(k) \|_2^22[3 (σ^(k),0.5)]^2 ]. cases (9) Each update preserves the main response region while reducing the influence of distant modes. The window scale follows the current response spread, while max(σ(k),0.5) (σ^(k),0.5) prevents it from collapsing when the distribution becomes highly concentrated. We apply this update twice as our inference setting. The pixel with the largest value in π(2)π^(2) is taken as the initial endpoint. Finally, this initial endpoint determines the target-mask center for one fresh, full-field CWM evaluation, while the source query remains tp_t. The new M responses are averaged uniformly and decoded to obtain the refined endpoint ^t+1 p_t+1; APA and LPA are not run again, and their initial weights are not reused. The pairwise motion is ϕ^t=^t+1−t φ_t= p_t+1-p_t, and the refined endpoint becomes the query for the next frame pair. Visibility is inherited from the original CWM response-strength signal; LPA introduces no separate visibility head. Further decoding details are provided in Appendix A. 4 Experiments Our experiments mainly address three questions: (1) Does learned candidate weighting improve motion recovered from a frozen CWM? (2) How does LPA-CWM compare with analytic weighting and external motion baselines under CMC and standard point-tracking evaluation? (3) Which components and weighting behaviors contribute to the improvement? 4.1 Experimental Setup Models and data. Uniform CWM, APA-CWM, and LPA-CWM share the same frozen predictor, intervention generator, and cached initial responses. They differ only in candidate weighting and use the same localization and paired re-evaluation procedure. We fix M=10M=10, use one paired re-evaluation (R=1R=1), set βπ=200 _π=200, and apply two windowed localization updates. Only the 3.0M-parameter LPA is trained on MOVi-F trajectories, with videos split before query sampling and checkpoint selection performed on held-out MOVi-F videos. We evaluate on 30 videos each from DAVIS (Pont-Tuset et al., 2017) and Kinetics (Kay et al., 2017), using TAP-Vid point tracks and visibility annotations (Doersch et al., 2022). The same evaluation videos and queries are used for all methods; further protocol details are provided in Appendix B. RoboCOIN (Wu et al., 2025) and RoboTAP (Vecerik et al., 2024) are additionally used for qualitative manipulation examples. Evaluation and comparisons. We evaluate motion recovery using our CMC protocol and the standard TAP-Vid First protocol. External baselines include SMURF, GMRW, DODUO, CoTracker3, RAFT, and SEA-RAFT, covering optical flow, correspondence, and dedicated point tracking. All methods receive identical evaluation frames and queries, and no baseline uses post-hoc smoothing or repair. Appendix D describes the flow and correspondence adapters. 4.2 Completeness-aware Motion Correspondence Figure 4: CMC summary on DAVIS. One representative metric from each CMC dimension; higher is better. CMC evaluates post-query motion on ground-truth dynamic tracks. We retain tracks with at least 4 pixels of cumulative post-query displacement at 256×256256× 256 resolution and count missing predictions on ground-truth-visible points as failures. CMC organizes its metrics into four diagnostic dimensions. Localization is summarized by DCAavg, which averages visible-and-correct predictions over 1,2,4,8,16\1,2,4,8,16\-pixel thresholds. Completeness is measured by Coverage and Complete@0.8, visibility by VisBA and occlusion F1, and continuity by LCC, the longest uninterrupted predicted-visible run. Figure 4 shows one representative metric from each dimension, while Table 1 reports the full set. CTS@(4,0.8) is an additional track-level success measure requiring at least 80% coverage and mean EPE at most 4 pixels on jointly visible points. Full definitions, threshold sweeps, and fragmentation diagnostics appear in Appendix C. Table 1: CMC results on DAVIS and Kinetics. DCAavg is the primary metric; CTS@(4,0.8) measures track-level success. All entries are percentages. Bold marks the best result among the three CWM variants. Best-performing external methods (shaded) are considered separately. Additional threshold-wise, visibility, continuity, and fragmentation results are reported in Appendix C. Method DAVIS Kinetics Coverage↑ Complete@0.8↑ DCAavg↑_ avg CTS@(4,0.8)↑ VisBA↑ Occ. F1↑ LCC↑ Coverage↑ Complete@0.8↑ DCAavg↑_ avg CTS@(4,0.8)↑ VisBA↑ Occ. F1↑ LCC↑ SMURF 91.98 85.54 11.25 2.67 56.64 14.92 83.29 89.67 81.89 19.16 14.17 62.69 35.11 82.40 GMRW 37.02 20.62 23.92 17.00 63.24 45.22 35.46 55.28 36.25 32.38 30.42 65.99 46.34 51.43 DODUO 31.44 15.04 22.25 14.42 58.94 45.83 28.44 67.04 49.87 40.21 39.87 78.37 58.39 59.83 CoTracker3 92.31 85.33 71.81 70.50 76.43 58.97 81.01 87.06 79.67 50.15 46.61 78.00 58.56 80.41 RAFT 68.74 49.58 49.00 31.77 70.16 41.66 62.02 80.83 69.24 44.89 43.94 74.93 60.72 71.71 SEA-RAFT 57.76 31.25 47.39 42.50 65.78 39.56 54.07 78.59 65.73 43.36 40.71 72.67 51.96 70.65 Uniform CWM 88.10 80.96 36.87 21.46 59.91 33.91 85.20 87.14 77.11 40.26 34.54 67.66 44.97 79.95 APA-CWM (Ours) 94.74 89.98 55.79 47.91 62.07 38.48 90.13 89.08 83.78 46.22 37.10 77.66 59.22 84.60 LPA-CWM (Ours) 98.21 95.63 58.98 57.54 75.89 48.14 93.18 94.65 89.97 51.92 58.25 82.97 62.67 86.84 We evaluate all methods under CMC on DAVIS and Kinetics (Table 1). Within the CWM family, LPA-CWM improves every reported metric over Uniform CWM and APA-CWM on both datasets. On DAVIS, CoTracker3 remains stronger in localization and several visibility-related measures, while LPA-CWM gives the strongest completeness and continuity results. On Kinetics, LPA-CWM achieves the best value among all evaluated methods on every main-paper CMC metric. Relative to Uniform CWM, learned adjudication improves DCAavg by 60.0% on DAVIS and 29.0% on Kinetics. 4.3 TAP-Vid First and Qualitative Results TAP-Vid First is a standard evaluation protocol from TAP-Vid that queries each point at its first visible frame and evaluates the subsequent trajectory (Doersch et al., 2022). Within the CWM family, LPA-CWM achieves the best result on all five reported measures on both DAVIS and Kinetics (Table 2), consistently improving over Uniform CWM and APA-CWM. External trackers provide an additional reference under this conventional tracking protocol; CoTracker3 remains stronger on several measures, particularly on DAVIS. Together with the CMC results, these comparisons show that learned candidate weighting improves both conventional tracking accuracy and the completeness of motion recovered from the frozen CWM. Table 2: TAP-Vid First results on DAVIS and Kinetics. AJ, points within threshold, and occlusion accuracy are percentages; distances are pixels. Bold marks the best result among the three CWM variants. Best-performing external methods (shaded) are considered separately. Method DAVIS Kinetics AJ↑ Pts. within thresh.↑ Avg. dist.↓ Med. dist.↓ Occ. acc.↑ AJ↑ Pts. within thresh.↑ Avg. dist.↓ Med. dist.↓ Occ. acc.↑ SMURF 7.39 14.34 32.34 30.06 78.14 17.28 29.43 27.67 20.56 70.48 GMRW 21.85 41.55 20.59 13.38 50.77 29.83 48.48 17.87 12.13 53.52 DODUO 26.15 52.49 8.79 2.90 49.46 36.89 46.32 13.80 4.15 67.69 CoTracker3 62.90 76.81 4.20 2.01 86.11 50.86 63.21 9.85 3.60 84.80 RAFT 37.08 58.79 15.83 5.60 69.86 37.65 47.67 18.20 4.00 80.52 SEA-RAFT 40.72 51.84 19.17 7.78 65.22 37.88 47.65 18.61 3.71 79.45 Uniform CWM 24.31 40.21 13.10 3.98 76.56 35.44 46.62 15.26 7.44 85.99 APA-CWM (Ours) 41.73 57.35 10.27 2.94 83.47 35.66 47.40 12.15 4.01 87.19 LPA-CWM (Ours) 45.29 63.92 6.73 2.07 84.18 41.91 53.12 10.87 3.45 88.96 Figure 5: Qualitative motion correspondence on four cases. Five methods receive identical source/target frames and visible queries across manipulation, human-action, and animal-motion scenes. Lines connect source queries to predicted endpoints, and badges report mean EPE in pixels. LPA-CWM produces accurate and coherent correspondences across the displayed cases. Figure 5 provides four qualitative comparisons, including a manipulation case together with human-action and animal-motion scenes. LPA-CWM gives the lowest displayed mean EPE in each example and remains stable under large motion and appearance changes. Additional qualitative results are provided in Appendix Figure F.1. 4.4 Ablation Studies Mask count and re-evaluation. We select M and R using 40 fixed query groups from 10 MOVi-F videos, with nested masks and shared queries. The sweep over M∈5,10,20M∈\5,10,20\ and R∈0,1,2R∈\0,1,2\ selects M=10,R=1M=10,R=1 as the best accuracy–cost setting. Increasing to 20 masks increases latency without improving endpoint accuracy, while a second re-evaluation provides no further reduction in EPE. This indicates that moderate candidate diversity together with a single refinement step is sufficient in our setting. Appendix Figure E.1 reports the full sweep and timing results. Architecture, objectives, and decoding. Table 3 isolates the main design choices under matched inference settings. Removing candidate-set interaction or visual cues consistently reduces performance, showing that both candidate comparison and visual context contribute to localization. For supervision, adding either coordinate or best-candidate guidance improves over KL matching alone, while the full objective performs best across all five measures. Windowed localization also improves each weighting rule, with the largest gain obtained by LPA. These results suggest that the gains come from several complementary components rather than a single design choice. Detailed controls are provided in Appendix E. Candidate weighting. Figure 6 examines the learned weighting before localization and re-evaluation. LPA substantially reduces candidate-space EPE and approaches the best-of-10 oracle, closing most of the gap from uniform weighting. It also assigns progressively larger weights to better candidates and more than doubles APA’s top-1 best-candidate hit rate. The learned distribution remains soft instead of collapsing to a single candidate. These results directly support the intended role of LPA: learning which CWM responses should contribute most strongly to the final motion. Table 3: Architecture, objective, and inference-stage ablations on DAVIS under TAP-Vid First. All configurations use M=10M=10 and R=1R=1. Architecture and objective variants use windowed localization; “Std.” and “Win.” denote localization without and with the two Gaussian-window updates, respectively. The best value within each block is bold. Configuration AJ↑ Pts. within thresh.↑ Avg. dist.↓ Med. dist.↓ Occ. acc.↑ Architecture ablation (windowed localization) Uniform CWM (reference) 24.31 40.21 13.10 3.98 76.56 w/o candidate-set interaction 41.52 56.82 8.99 2.66 82.75 Response + scalar cues only 41.89 57.20 8.46 3.26 83.61 Full LPA-CWM 45.29 63.92 6.73 2.07 84.18 Objective ablation (full architecture, windowed localization) KL only 41.56 57.48 9.09 2.80 82.84 KL + coordinate supervision 42.51 58.21 8.11 2.12 83.34 KL + best-candidate supervision 42.92 57.96 7.95 2.16 83.73 Full objective 45.29 63.92 6.73 2.07 84.18 Candidate aggregation and localization Uniform + Std. 23.25 39.46 13.61 4.26 76.18 APA + Std. 40.54 56.19 10.54 3.12 83.29 LPA + Std. 42.80 58.47 7.17 2.85 83.85 Uniform + Win. 24.31 40.21 13.10 3.98 76.56 APA + Win. 41.73 57.35 10.27 2.94 83.47 LPA + Win. 45.29 63.92 6.73 2.07 84.18 Figure 6: Candidate weighting on MOVi-F validation videos. (a) Candidate-space weighted-coordinate EPE; points denote videos and diamonds denote video-macro means. (b) Assigned weight versus ground-truth candidate quality rank from best to worst; the inset reports top-1 best-candidate hit rates. Error bars and shading show 95% video-cluster bootstrap intervals. 5 Conclusion and Limitations We presented LPA-CWM for improving motion reasoning with a frozen counterfactual world model. LPA learns relative reliability across competing CWM candidates from dense trajectory supervision, while APA-CWM provides a training-free analytic reference based on NSG. We also introduced CMC to evaluate localization, trajectory completeness, visibility, and continuity on dynamic tracks. Across DAVIS and Kinetics, LPA-CWM consistently improves over Uniform CWM and APA-CWM under both CMC and TAP-Vid First, while the ablations confirm the contributions of candidate interaction, trajectory supervision, and windowed localization. Three limitations remain. First, LPA can only reweight motion evidence already present in the frozen candidate set, so missing or consistently incorrect candidates cannot be recovered; improving candidate generation or jointly adapting the underlying CWM may address this limitation. Second, LPA is trained on synthetic MOVi-F trajectories, while the current quantitative evaluation covers DAVIS and Kinetics within one CWM family. Extending trajectory supervision to real and embodied data and evaluating across additional datasets and world-model architectures will better establish generalization. Third, multi-mask inference and paired re-evaluation increase per-query computation. More efficient candidate selection, shared computation across masks, and reducing the number of required CWM evaluations are promising directions for lowering this cost. References Bardes et al. (2024) A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471. External Links: Link Cited by: §2. Bear et al. (2023) D. M. Bear, K. T. Feigelis, H. Chen, W. Lee, R. M. Venkatesh, K. Kotar, A. Durango, and D. L. K. Yamins Unifying (machine) vision via counterfactual world modeling. External Links: 2306.01828 Cited by: §1, §2, §3.1. Begiristain et al. (2026) L. Begiristain, O. Dünkel, and A. Kortylewski CRONOS: benchmarking counterfactual physical consistency in video models. arXiv preprint arXiv:2605.23699. Cited by: §1, §2. Bruce et al. (2024) J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. M. E. Bechtle, F. Behbahani, S. C. Y. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel Genie: generative interactive environments. In Proceedings of the 41st International Conference on Machine Learning, p. 4603–4623. External Links: Link Cited by: §2. Dhariwal and Nichol (2021) P. Dhariwal and A. Nichol Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, Cited by: §3.2. Doersch et al. (2022) C. Doersch, A. Gupta, L. Markeeva, A. Recasens, L. Smaira, Y. Aytar, J. Carreira, A. Zisserman, and Y. Yang TAP-Vid: a benchmark for tracking any point in a video. In Advances in Neural Information Processing Systems, Cited by: Appendix B, Appendix B, Appendix C, §1, §4.1, §4.3. Duan et al. (2025) H. Duan, H. Yu, S. Chen, L. Fei-Fei, and J. Wu WorldScore: a unified evaluation benchmark for world generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Cited by: §1, §2. Greff et al. (2022) K. Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, D. J. Fleet, D. Gnanapragasam, F. Golemo, C. Herrmann, T. Kipf, A. Kundu, D. Lagun, I. Laradji, H. D. Liu, H. Meyer, Y. Miao, D. Nowrouzezahrai, C. Oztireli, E. Pot, N. Radwan, D. Rebain, S. Sabour, M. S. M. Sajjadi, M. Sela, V. Sitzmann, A. Stone, D. Sun, S. Vora, Z. Wang, T. Wu, K. M. Yi, F. Zhong, and A. Tagliasacchi Kubric: a scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3749–3761. Cited by: §1, §3.3. Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber World models. arXiv preprint arXiv:1803.10122. Cited by: §1. Hafner et al. (2020) D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi Dream to control: learning behaviors by latent imagination. In International Conference on Learning Representations, Cited by: §1. Hafner et al. (2019) D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, p. 2555–2565. Cited by: §1. Harley et al. (2025) A. W. Harley, Y. You, X. Sun, Y. Zheng, N. Raghuraman, Y. Gu, S. Liang, W. Chu, A. Dave, P. Tokmakov, S. You, R. Ambrus, K. Fragkiadaki, and L. J. Guibas AllTracker: efficient dense point tracking at high resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, External Links: Link Cited by: §2. Huang et al. (2024) Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y. Wang, X. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu VBench: comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 21807–21818. Cited by: §1, §2. Jeong et al. (2025) H. Jeong, C. P. Huang, J. C. Ye, N. J. Mitra, and D. Ceylan Track4Gen: teaching video diffusion models to track points improves video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: §2. Jiang et al. (2024) Z. Jiang, H. Jiang, and Y. Zhu Doduo: dense visual correspondence from unsupervised semantic-aware flow. In IEEE International Conference on Robotics and Automation, External Links: Document, Link Cited by: §1, §2. Karaev et al. (2025) N. Karaev, Y. Makarov, J. Wang, N. Neverova, A. Vedaldi, and C. Rupprecht CoTracker3: simpler and better point tracking by pseudo-labelling real videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Cited by: §1, §2. Kay et al. (2017) W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman The Kinetics human action video dataset. arXiv preprint arXiv:1705.06950. Cited by: Appendix B, §1, §4.1. Kim et al. (2025) S. Kim, K. L. Aw, K. Kotar, C. Eyzaguirre, W. Lee, Y. Liu, J. Watrous, S. Stojanov, J. C. Niebles, J. Wu, and D. L. K. Yamins Taming generative video models for zero-shot optical flow extraction. arXiv preprint arXiv:2507.09082. External Links: Link Cited by: §2. Koppula et al. (2024) S. Koppula, I. Rocco, Y. Yang, J. Heyward, J. Carreira, A. Zisserman, G. Brostow, and C. Doersch TAPVid-3D: a benchmark for tracking any point in 3D. In Advances in Neural Information Processing Systems, Vol. 37, p. 82149–82165. Cited by: §2. Li et al. (2024) X. Li, J. Lu, K. Han, and V. A. Prisacariu SD4Match: learning to prompt stable diffusion model for semantic matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 27558–27568. External Links: Link Cited by: §2. Liu et al. (2024) J. Liu, Y. Qu, Q. Yan, X. Zeng, L. Wang, and R. Liao Fréchet video motion distance: a metric for evaluating motion consistency in videos. arXiv preprint arXiv:2407.16124. External Links: Link Cited by: §2. Motamed et al. (2026) S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos Do generative video models understand physical principles?. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Cited by: §2. Pont-Tuset et al. (2017) J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbelaez, A. Sorkine-Hornung, and L. Van Gool The 2017 DAVIS challenge on video object segmentation. arXiv preprint arXiv:1704.00675. Cited by: Appendix B, §1, §4.1. Shang et al. (2026) Y. Shang, Z. Li, Y. Ma, W. Su, X. Jin, Z. Wang, L. Jin, X. Zhang, Y. Tang, H. Su, C. Gao, W. Wu, X. Liu, D. Shah, Z. Zhang, Z. Chen, J. Zhu, Y. Tian, T. Chua, W. Zhu, and Y. Li WorldArena: a unified benchmark for evaluating perception and functional utility of embodied world models. arXiv preprint arXiv:2602.08971. Cited by: §1, §2. Shrivastava et al. (2026) A. Shrivastava, S. Mehta, D. Geng, and A. Owens Point prompting: counterfactual tracking with video diffusion models. In International Conference on Learning Representations, Cited by: §2. Shrivastava and Owens (2024) A. Shrivastava and A. Owens Self-supervised any-point tracking by contrastive random walks. In European Conference on Computer Vision, Cited by: §1, §2. Stojanov et al. (2025) S. Stojanov, D. Wendt, S. Kim, R. Venkatesh, K. Feigelis, K. Kotar, K. L. Aw, J. Wu, and D. L. K. Yamins Self-supervised learning of motion concepts by optimizing counterfactuals. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §1, §2, §3.1. Stone et al. (2021) A. Stone, D. Maurer, A. Ayvaci, A. Angelova, and R. Jonschkowski SMURF: self-teaching multi-frame unsupervised RAFT with full-image warping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3887–3896. Cited by: §1, §2. Teed and Deng (2020) Z. Teed and J. Deng RAFT: recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision, p. 402–419. Cited by: §1, §2. Vecerik et al. (2024) M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and J. Scholz RoboTAP: tracking arbitrary points for few-shot visual imitation. In IEEE International Conference on Robotics and Automation, External Links: Link Cited by: §4.1. Venkatesh et al. (2024) R. Venkatesh, H. Chen, K. Feigelis, D. M. Bear, K. Jedoui, K. Kotar, F. J. Binder, W. Lee, S. Liu, K. A. Smith, J. E. Fan, and D. L. K. Yamins Understanding physical dynamics with counterfactual world modeling. In European Conference on Computer Vision, External Links: Document Cited by: §1, §2, §3.1. Wang et al. (2024) Y. Wang, L. Lipson, and J. Deng SEA-RAFT: simple, efficient, accurate RAFT for optical flow. In European Conference on Computer Vision, Cited by: §1, §2. Wu et al. (2025) S. Wu, X. Liu, S. Xie, P. Wang, X. Li, et al. RoboCOIN: an open-sourced bimanual robotic data COllection for INtegrated manipulation. arXiv preprint arXiv:2511.17441. External Links: Link Cited by: §4.1. Zhang et al. (2025) S. Zhang, Z. Lian, J. Yang, D. Li, G. Pang, F. Liu, B. Han, S. Li, and M. Tan Physics-driven spatiotemporal modeling for AI-generated video detection. In Advances in Neural Information Processing Systems, Cited by: §1, §2, §3.2. Zheng et al. (2025) D. Zheng, Z. Huang, H. Liu, K. Zou, Y. He, F. Zhang, L. Gu, Y. Zhang, J. He, W. Zheng, Y. Qiao, and Z. Liu VBench-2.0: advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755. Cited by: §1, §2. Appendix A Implementation Details Frozen candidate generator. The counterfactual predictor and intervention generator remain fixed during LPA training and inference. CWM uses a 32×3232× 32 patch-grid mask at the 256×256256× 256 input resolution, where each mask element corresponds to one 8×88× 8-pixel image patch; approximately 89.9% of the target-frame patch grid is hidden. Each mask-conditioned counterfactual response is evaluated on a 256×256256× 256 response grid, and its candidate endpoint is obtained from the peak of the channel-reduced absolute RGB response. Averaging rather than summing over RGB channels changes only a constant factor and therefore does not change the peak location. LPA architecture. The complete LPA module contains 3,000,257 trainable parameters. The main text uses height–width–channel notation, whereas the PyTorch implementation stores image-like tensors in channel-first format. The local candidate encoder therefore receives a 10×32×3210× 32× 32 tensor and applies four 3×33× 3 convolutional blocks with stride 2 and padding 1, using channel dimensions 10→32→64→128→19210→ 32→ 64→ 128→ 192. Each block uses GroupNorm and GELU, followed by global average pooling to produce a 192-dimensional local feature. The global response encoder receives the complete normalized single-channel response in 1×32×321× 32× 32 format. It uses four stride-2 convolutional blocks with channel dimensions 1→16→32→64→961→ 16→ 32→ 64→ 96, with 3×33× 3 kernels, padding 1, GroupNorm, and GELU, followed by global average pooling to produce a 96-dimensional feature. The shared frame-pair encoder receives the concatenated source and target RGB frames in 6×64×646× 64× 64 format and uses four analogous blocks with channel dimensions 6→32→64→128→1926→ 32→ 64→ 128→ 192, producing a 192-dimensional shared context feature. The scalar encoder maps the 16-dimensional candidate descriptor through an MLP with dimensions 16→64→6416→ 64→ 64, followed by GELU and LayerNorm, to obtain a 64-dimensional feature. The four branches are concatenated into a 544-dimensional representation and projected by Linear(544,256)(544,256), LayerNorm, and GELU to form a 256-dimensional candidate token. The set encoder contains four pre-normalized Transformer layers with eight attention heads, a 512-dimensional feed-forward block, dropout 0.1, and GELU activation. No candidate-order positional embedding is used. The shared scoring head applies LayerNorm(256)(256), Linear(256,128)(256,128), GELU, and Linear(128,1)(128,1), followed by a softmax over the candidate dimension. Candidate statistics. The scalar branch receives the 16-dimensional descriptor listed in Table A.1. These statistics complement the learned visual and response encoders with compact information about displacement, response concentration, ambiguity, photometric change, and local edge structure. Table A.1: Candidate scalar descriptor used by LPA. The support column indicates the spatial region used to compute each statistic. Dim. Statistic Definition Support 1 Normalized vertical displacement Candidate y displacement normalized by image height Global 2 Normalized horizontal displacement Candidate x displacement normalized by image width Global 3 Endpoint response Response magnitude at the candidate endpoint Single pixel 4 Global response mean Mean response magnitude Full response 5 Global response standard deviation Population standard deviation of response magnitude Full response 6 Global response median Median response magnitude Full response 7 Local peak ratio (7×77× 7) Endpoint response divided by local mean response 7×77× 7 8 Local peak ratio (15×1515× 15) Endpoint response divided by local mean response 15×1515× 15 9 Peak-to-median ratio Endpoint response divided by global response median Full response 10 Peak-to-second-peak ratio Endpoint response divided by the strongest response outside an 11×1111× 11 exclusion region Global + local exclusion 11 Local response mass ratio Local response mass divided by total response mass 7×77× 7 12 Local response mass ratio Local response mass divided by total response mass 15×1515× 15 13 Local response mass ratio Local response mass divided by total response mass 31×3131× 31 14 Normalized spatial entropy Entropy of the normalized spatial response distribution Full response 15 Local photometric change Mean frame difference around the candidate endpoint 7×77× 7 16 Local source-edge strength Mean source-frame edge magnitude around the candidate endpoint 7×77× 7 Dimensions 3–10 first undergo signed-log compression. All 16 dimensions are then standardized using population means and standard deviations computed over all candidate rows in the MOVi-F training split; the same normalization statistics are stored with the checkpoint and reused at inference. Candidate-centered scalar statistics use the rounded and image-clipped candidate endpoint. Local windows are clipped to valid image support without padding, and local means are normalized by the number of valid pixels. Local patch sampling. Each local visual patch is centered at the source query or candidate endpoint, covers a continuous 64×6464× 64 spatial extent, and is sampled directly onto a 32×3232× 32 bilinear grid. If the sampling region extends beyond the image domain, out-of-bound values use border replication; candidate centers are neither discarded nor recentered. MOVi-F training correspondence filtering. Training samples are drawn from adjacent frames in the official MOVi-F training split using the native 128×128128× 128 dense ground-truth forward and backward flow fields. For a native-resolution source point p, let =+t→t+1().q=p+F_t→ t+1(p). The backward field is read at the rounded target coordinate, without bilinear interpolation, and the forward–backward consistency error is efb()=‖t→t+1()+t+1→t(round())‖2.e_fb(p)= \|F_t→ t+1(p)+B_t+1→ t (round(q) ) \|_2. A correspondence is retained only when efb≤2.0e_fb≤ 2.0 pixels at the native 128×128128× 128 resolution and both the source and warped target are at least four pixels from the image boundary. Out-of-frame targets, points too close to the boundary, and non-finite consistency errors are discarded rather than clamped back into the valid domain. The sampling procedure additionally requires a minimum moving magnitude of 0.25 pixels at the native resolution. Accepted frames and coordinates are subsequently mapped to the common 256×256256× 256 training scale. LPA training. MOVi-F is partitioned at the source-video level before frame-pair and query sampling, and each training sample retains the complete M=10M=10 candidate set. Candidate order is randomly permuted during training. AdamW uses a learning rate of 3×10−43× 10^-4, batch size 16, and a cosine schedule. Checkpoint selection uses candidate-space weighted-coordinate EPE on held-out MOVi-F validation videos. This criterion corresponds to the weighted-coordinate supervision in Section 3.3 and is distinct from the final deployed endpoint EPE reported in the M–R configuration study. Analytic precursor. APA-CWM uses a frozen Ablated Diffusion Model (ADM) score network with approximately 552M parameters, evaluated in FP16 at 224×224224× 224 resolution and diffusion time ts=0.006t_s=0.006. The score map P is computed once per frame pair using the same numerical stabilizer 10−610^-6 as in the main text and bilinearly resampled to the 256×256256× 256 CWM response grid. APA averages the aligned statistic within a radius-six neighborhood around each candidate response peak, clips the candidate values to [10−3,103][10^-3,10^3], and applies L1 normalization across candidates. The score map is used only by APA-CWM; LPA-CWM does not consume the diffusion score map. Figure A.1: Evidence chain for the NSG-derived analytic precursor on a real frame pair. The reference frame and observed change define the temporal evidence; score magnitude and score–change alignment produce the analytic reliability map P. APA-CWM samples this map at CWM response peaks. LPA-CWM does not use the diffusion score map; it learns candidate reliability from the features described in Section 3.3. Heatmaps are independently normalized for visualization. Counterfactual decoding. For each frame pair and query, M denotes the number of mask-conditioned candidate responses in the initial CWM set, whereas R denotes the number of paired re-evaluation passes after the initial localization. The two hyperparameters therefore control different stages of inference. When R=0R=0, only the initial candidate set is weighted and localized, with no additional CWM pass. For the default R=1R=1, the decoder-specific initial endpoint conditions one coarse-conditioned paired re-evaluation. The source location remains the original query, while the initial endpoint determines the center of the heuristic masking applied to the target frame. At 256×256256× 256 resolution, both source and target retain complete spatial support. A fresh set of M masks is sampled, and the frozen CWM recomputes M mask-conditioned responses. APA and LPA are not rerun, the initial candidate weights are not reused, and the fresh responses are uniformly aggregated. The discrete peak of the resulting response distribution gives the refined endpoint. Thus, the default R=1R=1 stage is a full-field coarse-conditioned paired re-evaluation rather than a local crop refinement. For R>1R>1, later passes may become progressively local. The crop side follows sr=s1ηr−1s_r=s_1η^r-1 with fixed η=0.75η=0.75; η changes only the crop size and is neither learned nor used in candidate weighting. For non-square inputs, the first square crop uses the shorter image dimension and is shifted to remain inside the valid image region rather than padded. Standard and windowed localization. Both decoders start from the same aggregated response distribution using the softmax inverse-temperature parameter βπ=200 _π=200. Standard localization applies no Gaussian-window update and passes the discrete argmax of the unwindowed response distribution to the subsequent R stage. Windowed localization performs two Gaussian-window reweighting steps. At each step, the continuous distribution mean and spatial standard deviation determine the window center and scale, with width 3max(σ,0.5)3 (σ,0.5); the distribution is renormalized after each update. The coordinate passed to the R stage is the discrete argmax after the second update, not the continuous expectation. The two window updates operate within a fixed response field and are distinct from the paired CWM re-evaluation passes. Trajectory construction. For each adjacent frame pair, the refined endpoint predicted for the current query is used as the source query in the next pair. The complete post-query trajectory is constructed before any CMC metric is computed. LPA-CWM inherits the CWM response-strength visibility signal and does not train a separate visibility or occlusion head. All compared methods use the same sampled frame sequence, image coordinates, and query identities. Appendix B Datasets and Evaluation Protocols Evaluation datasets. We evaluate on TAP-Vid DAVIS and TAP-Vid Kinetics (Doersch et al., 2022), which provide human-annotated point tracks and visibility labels for object-centric and in-the-wild videos, respectively. DAVIS contains challenging deformation, occlusion, distractors, and fast motion (Pont-Tuset et al., 2017), whereas Kinetics-400 introduces broader variation in actions, camera motion, viewpoint, and human–object interaction (Kay et al., 2017). We use fixed evaluation subsets and identical sampled frames and query tracks for all methods. No model parameter is adapted to either evaluation dataset. Evaluation protocol. Following the TAP-Vid First protocol (Doersch et al., 2022), each query is introduced at its first visible frame and evaluated on subsequent frames. Ground-truth coordinates and visibility are taken directly from the TAP-Vid annotations. The same query sets are used for Uniform CWM, APA-CWM, LPA-CWM, and all external baselines, and missing predictions on annotated visible points are retained as failures rather than removed from evaluation. The CWM variants also share the same cached initial response sets; only their candidate weighting differs before the common localization and paired re-evaluation procedure. TAP-Vid metrics. We report Average Jaccard, points within the benchmark distance thresholds, mean and median endpoint distance, and occlusion accuracy. Average Jaccard jointly reflects localization and visibility, while the distance metrics separately summarize typical and tail localization errors. CMC uses the same ground-truth tracks but evaluates motion correspondence from the complementary perspectives of localization, completeness, visibility, and temporal continuity, as detailed in Appendix C. Appendix C Completeness-aware Motion Correspondence We introduce Completeness-aware Motion Correspondence (CMC) to evaluate motion correspondence on ground-truth dynamic tracks while explicitly penalizing incomplete predictions. Dynamic tracks are selected from ground truth, and missing visible-point predictions remain in the accuracy denominator. For both datasets, the ground-truth coordinates and visibility labels are read directly from TAP-Vid point-track annotations. We use TAP-Vid DAVIS point tracks for the DAVIS videos and TAP-Vid Kinetics point tracks for the Kinetics videos; DAVIS segmentation annotations are not converted into trajectories (Doersch et al., 2022). Let qiq_i be the query frame of trajectory i, xi,tgtx gt_i,t and x^i,t x_i,t the ground-truth and predicted coordinates, and vi,tgt,v^i,t∈0,1v gt_i,t, v_i,t∈\0,1\ their visibility labels. Only post-query, ground-truth-visible frames are evaluated: Ei=t:t>qi,vi,tgt=1.E_i=\t:t>q_i,\ v gt_i,t=1\. The post-query ground-truth path length is Li,>qgt=∑t≥qi[vi,tgt=vi,t+1gt=1]∥xi,t+1gt−xi,tgt∥2.L gt_i,>q= _t≥ q_i1[v gt_i,t=v gt_i,t+1=1] x gt_i,t+1-x gt_i,t _2. The dynamic set =i:Li,>qgt≥4D=\i:L gt_i,>q≥ 4\ is defined on a common 256-pixel coordinate scale and is independent of all predictions. Completeness. Per-track visibility coverage and its thresholded counterpart are Ci=∑t∈Ei[v^i,t=1]|Ei|,Coverage=1||∑i∈Ci,C_i= _t∈ E_i1[ v_i,t=1]|E_i|, = 1|D| _i C_i, Complete@0.8=1||∑i∈[Ci≥0.8].Complete@0.8= 1|D| _i 1[C_i≥ 0.8]. These are completeness diagnostics, not localization-accuracy measures. Dynamic Correspondence Accuracy. With ei,t=∥x^i,t−xi,tgt∥2e_i,t= x_i,t-x gt_i,t _2, we define DCAτ=∑i∈∑t∈Ei[v^i,t=1][ei,t<τ]∑i∈|Ei|.DCA_τ= _i _t∈ E_i1[ v_i,t=1]1[e_i,t<τ] _i |E_i|. Thus a prediction marked invisible contributes zero rather than disappearing from the denominator. The primary scalar summary is DCAavg=15∑τ∈1,2,4,8,16DCAτ.DCA_ avg= 15 _τ∈\1,2,4,8,16\DCA_τ. Complete Track Success. Let e¯ijoint e_i joint be the mean EPE over frames in EiE_i that are also predicted visible. The track-level criterion is CTS(τ,c)=1||∑i∈[Ci≥c][e¯ijoint≤τ].CTS(τ,c)= 1|D| _i 1[C_i≥ c]1[ e_i joint≤τ]. We report CTS@(4,0.8) and CTS@(8,0.8). CTS is secondary to DCA because it still allows up to 20% of a track to be absent. Visibility and continuity. Visibility balanced accuracy is VisBA=12(TPRvisible+TNRoccluded),VisBA= 12(TPR_ visible+TNR_ occluded), and occlusion F1 is computed from post-query occlusion labels. For each ground-truth-visible segment, longest continuous coverage (LCC) is the longest predicted-visible run divided by |Ei||E_i|. Fragmentation counts the number of →11\!→\!0\!→\!1 visibility recoveries within such segments; Table C.2 reports the mean count. Table C.1: Full threshold-wise CMC accuracy results. All entries use the same 0–100 scale as Table 1. Bold marks the best result among the three CWM variants. Best-performing external methods (shaded) are considered separately. Method Coverage↑ Complete@0.8↑ DCA@1↑ DCA@2↑ DCA@4↑ DCA@8↑ DCA@16↑ DCAavg↑_ avg CTS@(4,0.8)↑ CTS@(8,0.8)↑ Dataset: DAVIS SMURF 91.98 85.54 1.72 4.11 8.72 14.07 27.62 11.25 2.67 3.33 GMRW 37.02 20.62 11.47 21.20 26.10 28.57 32.25 23.92 17.00 17.00 DODUO 31.44 15.04 6.76 18.23 25.89 29.59 30.79 22.25 14.42 15.04 CoTracker3 92.31 85.33 40.55 64.40 80.49 85.20 88.41 71.81 70.50 74.82 RAFT 68.74 49.58 29.93 46.92 53.02 56.58 58.53 49.00 31.77 34.59 SEA-RAFT 57.76 31.25 26.38 43.73 53.43 56.32 57.06 47.39 42.50 43.13 Uniform CWM 88.10 80.96 6.75 19.30 39.85 52.99 65.46 36.87 21.46 37.96 APA-CWM (Ours) 94.74 89.98 19.31 41.71 61.91 73.94 82.05 55.79 47.91 58.14 LPA-CWM (Ours) 98.21 95.63 17.87 44.73 67.49 80.13 84.67 58.98 57.54 66.79 Dataset: Kinetics SMURF 89.67 81.89 4.61 9.36 16.87 24.29 40.65 19.16 14.17 14.79 GMRW 55.28 36.25 11.23 24.66 36.71 41.66 47.66 32.38 30.42 34.38 DODUO 67.04 49.87 6.95 22.30 46.45 59.92 65.42 40.21 39.87 48.62 CoTracker3 87.06 79.67 13.04 32.29 54.89 71.99 78.56 50.15 46.61 60.89 RAFT 80.83 69.24 10.02 26.02 49.49 67.38 71.55 44.89 43.94 60.75 SEA-RAFT 78.59 65.73 9.49 25.19 47.29 65.01 69.80 43.36 40.71 56.55 Uniform CWM 87.14 77.11 4.47 20.93 44.69 60.91 70.31 40.26 34.54 50.58 APA-CWM (Ours) 89.08 83.78 8.44 22.86 47.03 72.16 80.61 46.22 37.10 64.66 LPA-CWM (Ours) 94.65 89.97 9.58 35.22 61.37 72.47 80.97 51.92 58.25 67.04 Table C.2: Visibility and continuity diagnostics for CMC. VisBA, occlusion F1, and LCC are percentages; Frag. is the mean fragmentation count ×100× 100. Bold marks the best result among the three CWM variants. Best-performing external methods (shaded) are considered separately. Method DAVIS Kinetics VisBA↑ Occlusion F1↑ LCC↑ Frag.↓ VisBA↑ Occlusion F1↑ LCC↑ Frag.↓ SMURF 56.64 14.92 83.29 21.54 62.69 35.11 82.40 9.66 GMRW 63.24 45.22 35.46 8.46 65.99 46.34 51.43 7.16 DODUO 58.94 45.83 28.44 12.88 78.37 58.39 59.83 16.82 CoTracker3 76.43 58.97 81.01 1.25 78.00 58.56 80.41 8.41 RAFT 70.16 41.66 62.02 7.62 74.93 60.72 71.71 9.38 SEA-RAFT 65.78 39.56 54.07 9.38 72.67 51.96 70.65 10.91 Uniform CWM 59.91 33.91 85.20 3.57 67.66 44.97 79.95 10.55 APA-CWM (Ours) 62.07 38.48 90.13 2.86 77.66 59.22 84.60 4.03 LPA-CWM (Ours) 75.89 48.14 93.18 0.63 82.97 62.67 86.84 2.78 Threshold sensitivity. Figure C.1 evaluates whether the CMC conclusions depend on the default dynamic-motion and coverage thresholds. With the coverage threshold fixed at c=0.8c=0.8, we vary the ground-truth motion threshold Lmotion∈2,4,8L_ motion∈\2,4,8\ pixels and report DCAavg. LPA-CWM changes only from 59.42 to 58.30 across this range and remains above APA-CWM and Uniform CWM at every setting. We then fix Lmotion=4L_ motion=4 pixels and vary the coverage threshold c∈0.6,0.7,0.8,0.9c∈\0.6,0.7,0.8,0.9\ for CTS@(4,c). LPA-CWM remains best throughout, decreasing moderately from 58.79 to 56.29 as the coverage requirement becomes stricter. The method ordering is therefore stable around the default Lmotion=4L_ motion=4 pixels and c=0.8c=0.8. Figure C.1: CMC threshold sensitivity on DAVIS. (a) DCAavg as the dynamic-track motion threshold varies over 2,4,8\2,4,8\ pixels. (b) CTS@(4,c) as the coverage threshold varies over 0.6,0.7,0.8,0.9\0.6,0.7,0.8,0.9\ with Lmotion=4L_ motion=4 pixels. Dashed vertical lines indicate the default settings. Appendix D Adapting External Estimators Coordinates. CoTracker3 is evaluated with the official offline point-tracking interface: the full video and first-visible query are provided jointly, and the predictor returns the complete trajectory directly. It therefore requires neither a flow-to-coordinate adapter nor recursive flow propagation. RAFT, SEA-RAFT, DODUO, and SMURF instead predict each post-query coordinate directly from the original query frame. For a query introduced at frame q, each query–target pair (Iq,It)(I_q,I_t) for t>qt>q is evaluated independently, and the forward displacement at the original query location gives the target coordinate. Reverse predictions, when computed, are used only for visibility estimation. GMRW uses its official tracking utility to propagate correspondences through adjacent frames. Visibility. Table D.1 summarizes the visibility rules used by the external estimators. CoTracker3 uses the Boolean visibility returned by its official offline predictor, which thresholds the native visibility score at 0.9 and forces the annotated query point to be visible. No additional forward–backward consistency test is applied. Table D.1: Visibility adaptation for external estimators. Thresholds are defined on the corresponding evaluator grid. Method Visibility source Threshold Sampling RAFT Direct query-to-target forward–backward consistency 6 px Forward at integer query; backward bilinear SEA-RAFT Direct query-to-target forward–backward consistency 6 px Forward at integer query; backward bilinear DODUO Direct query-to-target forward–backward consistency 6 px Forward at integer query; backward bilinear CoTracker3 Native visibility from official offline predictor 0.9 Native sparse-query trajectory; no additional FB test SMURF Native occlusion map 0.5 Query-location lookup GMRW Official tracking utility with adjacent-frame forward–backward consistency 3 px Bilinear at propagated positions For the flow and correspondence estimators, out-of-frame or non-finite predictions are treated as invisible by the corresponding adapters. CoTracker3 retains the native visibility returned by the official predictor without an additional out-of-frame, non-finite, or forward–backward filter. These evaluation-time visibility rules are distinct from the ground-truth forward–backward filtering used to construct MOVi-F training samples in Appendix A. Aggregation. All metrics are first computed from the same per-video query set and then aggregated with equal video weight. No point or video is selected using a method’s predicted error, coverage, or visibility. Missing predictions enter DCA as failures. Appendix E Ablation Protocols Joint M–R sensitivity. The audit uses 40 query groups from 10 held-out MOVi-F videos, shared across configurations. We evaluate M∈5,10,20×R∈0,1,2M∈\5,10,20\× R∈\0,1,2\. Masks are nested prefixes of the same ordered 20-mask set, avoiding independent draws for smaller M. All runs use full LPA-CWM on identical videos, frame pairs, and queries. We report final endpoint EPE after localization and R paired re-evaluations. The sweep selects M=10,R=1M=10,R=1 for the main comparison. (a) Final deployed endpoint EPE (b) End-to-end inference time Figure E.1: Joint sensitivity to mask count and paired re-evaluation depth. Full LPA-CWM results on 40 fixed MOVi-F query groups. (a) Final deployed endpoint EPE. (b) End-to-end inference time per query. R=0R=0 omits paired re-evaluation; M=10,R=1M=10,R=1 is the deployed setting. Latency measurement. Figure E.1 uses a single NVIDIA GeForce RTX 5090 with batch size 1, where each timed sample contains one adjacent frame pair and one query group. CUDA is synchronized immediately before and after the complete model call. Timing includes the factual frozen-CWM forward pass, M mask-conditioned counterfactual evaluations, LPA feature extraction and set processing, candidate aggregation, localization, the specified R paired CWM re-evaluation passes, and final endpoint extraction. Disk I/O, CPU preprocessing, host-to-device transfer, and model or checkpoint initialization are excluded. We therefore report end-to-end per-query latency under this protocol rather than throughput-optimized peak speed. Candidate-weight analysis. Candidates are ranked within each complete ten-candidate group by ground-truth endpoint EPE. The top-1 oracle hit rate measures how often the highest-weight candidate is the minimum-EPE candidate. Effective support is exp(−∑mwmlogwm) (- _mw_m w_m): it equals 1 for one-hot weights and 10 for uniform weights. Figure 6 reports video-macro summaries. This analysis uses the MOVi-F validation split used for checkpoint selection and serves as mechanism evidence; real-video tracking is evaluated separately. EPE definitions across analyses. Figure E.1 evaluates 40 fixed query groups using the final deployed endpoint after localization and the requested R re-evaluation passes; the default M=10,R=1M=10,R=1 setting obtains 2.1363 pixels. Figure 6 instead evaluates 640 complete M=10M=10 candidate groups using the candidate-space weighted-coordinate surrogate before response-level localization and paired re-evaluation, where LPA obtains 2.41 pixels. These values are not directly comparable because they use different validation subsets and endpoint definitions. Inference-stage ablation. All six configurations use the same frozen CWM, M=10M=10, R=1R=1, evaluation seed, DAVIS batches, frame pairs, queries, and initial masks. The rows are Uniform + Std., APA + Std., Uniform + Win., APA + Win., LPA + Std., and LPA + Win.. “Std.” uses the discrete peak of the unwindowed aggregated response distribution as the initial endpoint. “Win.” applies two Gaussian-window reweighting updates and uses the discrete peak of the final distribution. Uniform, APA, and LPA respectively denote equal, analytic, and learned weights for the initial candidate set. All six rows then use the same one-pass coarse-conditioned paired re-evaluation after their decoder-specific initial endpoint. The second pass does not rerun APA or LPA, does not reuse the initial weights, and uniformly aggregates the freshly generated responses. This 3×23× 2 design therefore separates initial candidate weighting from localization. Learned architecture and objective ablations. Architecture variants use the complete three-term objective, the same windowed localization, and the same R=1R=1 paired re-evaluation. “w/o candidate-set interaction” removes the permutation-equivariant set Transformer, while “response + scalar cues only” removes the local RGB and shared frame-pair raw encoders but retains response maps, 16 scalar statistics, and candidate-set interaction. Objective variants retain the full LPA architecture and compare ℒKLL_ KL, ℒKL+ℒcoordL_ KL+L_ coord, ℒKL+ℒbestL_ KL+L_ best, and the full three-term objective. Table 3 reports every row under the TAP-Vid First protocol on the same DAVIS evaluation set and cached initial CWM candidate responses. Its five columns are TAP-Vid localization and occlusion measures. The inference procedure is fixed except for the named architecture or objective component. Appendix F Additional Qualitative Results Figure F.1: Additional qualitative results on fourteen cases distinct from Figure 5. Columns show SEA-RAFT, CoTracker3, Uniform CWM, APA-CWM, and LPA-CWM in this order. Every method receives the same frame pair and the same five query identities. A method uses one color throughout; continuous lines connect source queries to target predictions, and badges report mean EPE over the displayed queries.