Paper deep dive
Grounding What Shapes the Plan: Rethinking Groundedness for Physical Intelligence in Autonomous Driving
Minkyoung Cho, Zewei Zhou, Wenhao Ding, Shuhan Tan, Boyi Li, Yuxiao Chen, Yan Wang, Zheng Lian, Min-Hung Chen, Chaowei Xiao, Zhuoqing Mao, Boris Ivanovic, Marco Pavone, Yulong Cao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/13/2026, 3:32:07 AM
Summary
The paper introduces GroundAct, a framework for autonomous driving that redefines 'groundedness' by using physical entities as the unit of grounding. Unlike previous methods that ground reasoning in text or tokens (Token-as-Content), GroundAct uses a 'Token-as-Reference' mechanism to link symbolic reasoning to continuous entity states. This allows for 'grounded planning,' where the plan is explicitly corrected based on the predicted interactions between the ego vehicle and the referenced entities, improving safety and performance in open-loop and closed-loop simulations.
Entities (8)
Relation Signals (7)
GroundAct → evaluatedon → AlpaSim
confidence 95% · ...and the AlpaSim closed-loop driving benchmark...
GroundAct → evaluatedon → PAI-AV
confidence 95% · We evaluate GroundAct on three PAI-AV-based open-loop benchmarks...
GroundAct → uses → Token-as-Reference
confidence 95% · GroundAct instantiates our three design choices: entity as the grounding unit, Token-as-Reference as the grounding mechanism...
Token-as-Content → contrastedwith → Token-as-Reference
confidence 90% · GroundAct draws on... Token-as-Reference... Whereas these methods focus on grounding intermediate reasoning, GroundAct preserves the reference link...
GroundAct → usesbackbone → Qwen3-VL-2B
confidence 90% · We instantiate GroundAct with a Qwen3-VL-2B vision–language backbone...
GroundAct → outperforms → Alpamayo 2 Super
confidence 85% · GroundAct... outperforming it on several metrics.
GroundAct → usescomponent → Flow Matching
confidence 85% · ...a flow-matching action expert...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Driving models increasingly ground reasoning in causal relations, spatial structure, perceptual evidence, and predicted futures. These advances make reasoning more faithful to the driving scene, but leave a fundamental question unresolved: what should groundedness mean when the model ultimately outputs an action? Correctly grounded reasoning does not, by itself, ensure desirable driving outcomes. We introduce GroundAct, which starts from a simple premise: driving unfolds through physical entities and their interactions. Entities therefore become the unit of grounding; a lightweight reference token keeps each selected entity's continuous state addressable through symbolic reasoning; and only the referenced entities' interactions with the evolving proposal correct the plan. The result is an explicit path from what reasoning grounds to what the plan does, which we call grounded planning. To assess its practical value, we evaluate GroundAct in both open- and closed-loop settings. GroundAct shows strong open-loop planning across normal, out-of-distribution, and safety-critical scenarios, with closed-loop results extending this evidence to driving in simulation.
Tags
Links
- Source: https://arxiv.org/abs/2610.07521v1
- Canonical: https://arxiv.org/abs/2610.07521v1
Trouble viewing inline? Open PDF directly →
Full Text
70,752 characters extracted from source content.
Expand or collapse full text
Grounding What Shapes the Plan: Rethinking Groundedness for Physical Intelligence in Autonomous Driving Minkyoung Cho† Affiliation: UMich2 Affiliation: UCLA3 Zewei Zhou† Affiliation: NVIDIA1 Affiliation: UMich2 Wenhao Ding Affiliation: UMich2 Shuhan Tan Affiliation: UMich2 Boyi Li Affiliation: UMich2 Yuxiao Chen Affiliation: UMich2 Yan Wang Affiliation: UMich2 Zheng Lian Affiliation: UMich2 Min-Hung Chen Affiliation: UMich2 Chaowei Xiao Affiliation: UMich2 Affiliation: JHU4 Z. Morley Mao Affiliation: USC5 Boris Ivanovic Affiliation: UMich2 Marco Pavone Affiliation: UMich2 Affiliation: Stanford6 Yulong Cao Affiliation: UMich2 Abstract Driving models increasingly ground reasoning in causal relations, spatial structure, perceptual evidence, and predicted futures. These advances make reasoning more faithful to the driving scene, but leave a fundamental question unresolved: what should groundedness mean when the model ultimately outputs an action? Correctly grounded reasoning does not, by itself, ensure desirable driving outcomes. We introduce GroundAct, which starts from a simple premise: driving unfolds through physical entities and their interactions. Entities therefore become the unit of grounding; a lightweight reference token keeps each selected entity’s continuous state addressable through symbolic reasoning; and only the referenced entities’ interactions with the evolving proposal correct the plan. The result is an explicit path from what reasoning grounds to what the plan does, which we call grounded planning. To assess its practical value, we evaluate GroundAct in both open- and closed-loop settings. GroundAct shows strong open-loop planning across normal, out-of-distribution, and safety-critical scenarios, with closed-loop results extending this evidence to driving in simulation. † † done during an internship at NVIDIA. Correspondence: minkycho@umich.edu. 1 Introduction Autonomous-driving models increasingly incorporate reasoning for its potential to support generalization across diverse and long-tail scenarios, grounding it in causal relationships, spatial structure, perceptual evidence, and predicted future states (Zeng et al., 2025; Shang et al., 2026; Peng et al., 2026; Tan et al., 2026). Structured reasoning traces further connect grounded observations and predictions to high-level intentions for how the vehicle should behave (NVIDIA et al., 2025c). Collectively, these advances improve what a driving model can explain about the scene, why a decision may be appropriate, and what may happen next. Yet reasoning is not the endpoint of driving: the output is a plan. Correctly grounded reasoning does not by itself ensure desirable driving outcomes, such as safety and comfort. Our motivating audit measures this transfer gap: the collision rate in the audited interactions, conditioned on correctly grounded reasoning (Figure 1(a)). This gap appears across causal, semantic, and spatiotemporal grounding. Figure 1(b) further illustrates how the same reasoning trace can arise in different driving contexts, even when the appropriate speed profiles differ. Swapping the speed profiles between the two contexts can lead to safety-critical outcomes, such as collisions, despite correctly grounded reasoning in both cases. This leads to our central question: What should groundedness mean when the output is an action? Our answer begins from a physical-AI view of driving: the ego vehicle acts in a world of physical entities and their interactions. Realizing a maneuver requires accounting for where relevant entities are, how they move, and how the ego plan interacts with them. We therefore choose the entity as our grounding unit: it provides a common physical referent linking what reasoning identifies as relevant to what planning must account for. Our principle is to carry grounded reasoning into action generation by preserving this physical connection, so that the selected entities’ interactions shape the evolving plan and improve the resulting driving behavior. We call this grounded planning. In this work, we introduce GroundAct, whose Token-as-Reference interface interleaves generated reasoning content with references to continuous entity states (Figure 2). Resolving these references gives subsequent reasoning and planning access to the same selected states. The action expert retains the full scene context, while an explicit refinement pathway evaluates the referenced states against the evolving ego proposal and uses their predicted interactions to guide selective correction. (a) Grounding-to-Plan Transfer Gap. Measured on PAI-AV∗ safety-critical samples (Appendix A). (b) Same Reasoning Trace, Different Actions Required. Blue: ego; orange/green: surrounding vehicles. Figure 1: Grounded reasoning does not by itself ensure desirable driving outcomes. We evaluate GroundAct on three PAI-AV-based open-loop benchmarks (Normal, OOD, and Safety-Critical) and the AlpaSim closed-loop driving benchmark (NVIDIA, 2025; NVIDIA et al., 2025b). With 3.9B parameters, GroundAct demonstrates strong planning and safety performance across these benchmarks, remaining competitive with the 34B state-of-the-art driving foundation model (NVIDIA, 2026b) and outperforming it on several metrics. Controlled ablations test whether reasoning-selected physical states remain consequential for planning by removing or replacing their references and by comparing reasoning-based selection against heuristic alternatives. Matched correction analyses further examine whether GroundAct corrects unsafe plans while preserving those that are already safe. Our contributions are: (i) Grounded planning, which establishes an explicit and traceable path from the entity grounded in reasoning to its consequence for action; (i) Token-as-Reference, which keeps continuous entity state in the model while reasoning selects planning-critical entities through lightweight symbolic references; and (i) Interaction-driven selective correction, which uses the referenced entities’ predicted interactions with the evolving ego plan to guide selective refinement. Figure 2: Token-as-Content vs. Token-as-Reference. [Left] Grounded information is encoded in generated content. [Right] GroundAct retains this content path and adds entity-reference tokens <|entity_i|>, each indexing a continuous entity state. These references keep the planning-critical entities selected during reasoning addressable throughout reasoning and planning. Only the referenced states enter the explicit refinement pathway. 2 Background and Related Work Grounded Reasoning as Token-as-Content. Driving VLAs ground their reasoning in three main forms, as illustrated on the left of Figure 2. Causally grounded reasoning states the factors behind a decision in language, as in Alpamayo-R1’s causal chains (NVIDIA et al., 2025c) and in adaptive or self-reflective reasoning before action (Zhou et al., 2025; Peng et al., 2026; Zhou et al., 2026). Spatially grounded reasoning carries positions: object coordinates written into the reasoning text (Tian et al., 2025) or object boxes drawn into generated frames (Zeng et al., 2025). Future-state grounded reasoning incorporates predicted futures, represented as image tokens (Zeng et al., 2025), latent tokens (Tan et al., 2026; Luo et al., 2026; Jia et al., 2026), or compact dynamics tokens (Shang et al., 2026). Despite these differences, the three forms share one interface, which we call Token-as-Content: whatever is grounded is expressed in the generated tokens and is consumed as such by subsequent reasoning or action generation. Grounding is thereby a property of the reasoning process; how that grounding should remain consequential for action generation is left unspecified. Token-as-Reference draws inspiration from recent visual-grounding work: Chung et al. (2025) points to image patches and copies their embeddings into the reasoning stream, while Zhang et al. (2026) serializes explicit points or boxes. Whereas these methods focus on grounding intermediate reasoning, GroundAct preserves the reference link from reasoning through downstream planning. Related Directions. GroundAct draws on two further lines of work, discussed in Appendix B. Prediction- and interaction-aware planners show that structured prediction and physical interaction can shape planning directly (Hu et al., 2023; Huang et al., 2023; Sadat et al., 2020; Li et al., 2026; Jia et al., 2026); GroundAct asks which entities should receive that computation and how the choice stays connected to reasoning. Guided and residual generative planners modify a pretrained generator without replacing it (Lipman et al., 2023; Song et al., 2021; Zhang et al., 2023a; Jing et al., 2026); GroundAct uses individual referenced entities, rather than a global intent, as the mediator. Both lines make predictions or guidance signals consequential for planning; GroundAct makes the reasoning’s grounding consequential, by keeping the selected entities addressable until they act on the plan. 3 GroundAct 3.1 Overview GroundAct instantiates our three design choices: entity as the grounding unit, Token-as-Reference as the grounding mechanism, and interaction-driven correction for planning. These choices span continuous multimodal encoding, discrete autoregressive reasoning, and iterative continuous action generation (Figure 3). Figure 3 summarizes the computation. A physical entity is first represented by a continuous, addressable state (§3.2). During reasoning, an entity mention is associated with its corresponding state in the current entity pool and expressed through a symbolic reference, without serializing the underlying geometry into text (§3.3). For planning, the referenced states are retrieved and explicitly evaluated against the evolving ego proposal; their predicted physical interactions then produce selective corrections to the generated action (§3.4). The representation changes across these computations, from continuous state to discrete reference to interaction feature, while the physical referent remains fixed. Section 3.5 describes how this correspondence is learned. Figure 3: GroundAct Architecture. The three columns correspond to §3.2 through §3.4. The row below follows the computation from input tokens through the Physical AI reasoner to the action expert’s refined plan, while the bottom bar summarizes the design invariant preserved throughout. 3.2 Prefill-Time Grounding For reasoning and planning to refer to the same physical entity, the model first needs a continuous state that is aligned with that entity and can be addressed later. Given temporal observations and driving context O, we append K entity tokens q1,…,qKq_1,…,q_K (K=64K=64) to the multimodal prefill. Each acts as a learnable query that attends to and aggregates temporal visual features together with the surrounding context; its final hidden state Mi=Hθ(O,q1,…,qK)[qi],M_i=H_θ(O,q_1,…,q_K)[q_i], the Transformer output HθH_θ read at the position of qiq_i, is the entity state at position i. The entity states ℳ=Mii=1KM=\M_i\_i=1^K stay inside the model, each addressable by its index i. Each index i is scene-local: the same position can represent different entities in different scenes. Subsequent reasoning and planning must therefore bind to the state an index currently addresses. 3.3 Reference-based Reasoning Reasoning interprets the driving situation and the intended maneuver, and in doing so singles out the planning-critical entities. To carry this choice into subsequent reasoning and planning, Token-as-Reference binds a discrete reference ei=⟨|entityi|⟩e_i= |entity\ i| to the continuous prefill state MiM_i. The token names the entity; its physical information remains in the addressable state. Reference Generation. At decoding step t, let hth_t be the reasoning hidden state and yt+1y_t+1 the next token. A gate assigns probability mass πt _t to references, and an entity pointer distributes it according to the correspondence between hth_t and the current states: p(yt+1=ei∣ht,ℳ)=πtsoftmaxi[τcos(Wqht,WkMi)],πt=σ(g(ht)).p(y_t+1=e_i h_t,M)= _t\,softmax_i\! [τ (W_qh_t,W_kM_i) ], _t=σ(g(h_t)). (1) Here Wq,WkW_q,W_k are learned projections, τ>0τ>0 controls score sharpness, and g is a scalar gate. The pointer softmax excludes detector-null slots whenever non-null candidates are available. The vocabulary head distributes the remaining mass 1−πt1- _t over non-reference tokens. Here, the division of probability mass between vocabulary and reference outputs is motivated by the gated pointer-generator models (See et al., 2017). This separates when to reference from which entity to name within one next-token distribution. Reference Resolution and Consumption. When yt+1=eiy_t+1=e_i, resolving the address directly reads the original prefill state MiM_i from the pool. The next decoding step consumes this state through the input embedding ut+1u_t+1: Resolve(ei,ℳ) (e_i,M) =Mi,ut+1 =M_i, u_t+1 =(1−α)bref+α(bi+P(Mi)). =(1-α)b_ref+α (b_i+P(M_i) ). (2) Here bib_i combines the entity-query base embedding and slot identity, P projects and scales the retrieved state into the decoder input space, and α∈[0,1]α∈[0,1] interpolates between this input and a shared reference embedding brefb_ref. The decoder processes ut+1u_t+1 with its existing context to produce a new reasoning hidden state; the retrieved MiM_i remains the prefill representation. Planning reads the same MiM_i (§3.4), preserving the selected physical referent across both computations. 3.4 Reference-driven Planning A referenced entity does not prescribe an action: what it implies depends on where the ego proposal (i.e., the planner’s current candidate trajectory) is heading. Let ℛ(r)⊆1,…,KR(r) \1,…,K\ denote the entities referenced in reasoning sequence r. Only their states ℳℛ=Mi:i∈ℛ(r)M_R=\M_i:i (r)\ enter explicit physical prediction and interaction modeling for residual correction, while the action expert—a flow-matching trajectory generator conditioned on the reasoning context—keeps its full view of the scene. Reasoning thus decides where additional physical computation is spent. Interaction with the Evolving Proposal. In the continuous action mode, the action expert produces a trajectory by integrating a learned vector field over a fixed number of solver steps. At step k, it evaluates the base velocity vbase(k)v_base^(k) at the current flow state x(k)x^(k); from the two we extrapolate a sequence of normalized acceleration and curvature and decode it through vehicle dynamics into the ego proposal P^(k) P^(k). In the forecast-based formulation, each referenced entity has a predicted motion S^i(k) S_i^(k) over the planning horizon (in the joint variant, conditioned on the current proposal), and the proposal-relative interaction Ci(k)=Interact(P^(k),S^i(k)),i∈ℛ(r),C_i^(k)=Interact\! ( P^(k), S_i^(k) ), i (r), encodes quantities such as clearance, relative motion, and validity over that horizon. Because the proposal changes at every step, so do these interactions; k indexes refinement of the whole proposal, not time within the horizon. Selective Correction. In this general formulation, a permutation-invariant set encoder aggregates the referenced states, their forecasts, and their interactions, and a residual adapter RϕR_φ maps the result, together with the current state and base velocity, to a correction of the base field: zℛ(k)=SetEnc(Mi,S^i(k),Ci(k)i∈ℛ(r)),vGroundAct(k)=vbase(k)+γ(k)Rϕ(x(k),vbase(k),zℛ(k)),z_R^(k)=SetEnc\! ( \M_i, S_i^(k),C_i^(k) \_i (r) ), v_GroundAct^(k)=v_base^(k)+γ^(k)\,R_φ\! (x^(k),v_base^(k),z_R^(k) ), where γ(k)γ^(k) combines a risk latch, activated by predicted overlap for the remaining solver steps, with a flow-time schedule. The solver integrates the corrected field, and the new proposal is evaluated again. The correction is defined jointly over the referenced set, and the pathway is bypassed (i.e., recovering the base field) when no valid entity is referenced or the gate is inactive. Our implementation predicts clearance from four observed frames, the ego proposal, and the states of valid referenced entities; thus, clearance conditions the residual. 3.5 Training Objectives And Optimization Training preserves physical correspondence from entity states to references and selective action correction. The principal objectives below apply to different parameter sets and optimization intervals. Entity States. Entity-level 3D supervision establishes physical correspondence for the entity states. Prediction heads decode an object category, including a null class, and an oriented BEV box from each MiM_i. Hungarian matching assigns predictions one-to-one to annotated objects; unmatched queries receive the null target. Class/null prediction and box regression are combined with language modeling: ℒentity=λclsℒcls+λboxℒbox+λLMℒLM.L_entity= _clsL_cls+ _boxL_box+ _LML_LM. (3) Queries and prediction heads first warm up with frozen backbones. Joint adaptation then unfreezes the backbones and includes language supervision. Physical References. Offline annotations link accepted referring expressions in reasoning to physical objects. At each training pass, those objects are rematched to class-consistent states in the current entity pool by BEV-center distance, so reference targets follow physical correspondences rather than fixed indices. After pointer alignment, decoder adaptation combines masked token prediction ℒtextL_text, matched-entity association ℒassocL_assoc, and reference timing ℒgateL_gate: ℒreason=λtextℒtext+λassocℒassoc+λgateℒgate.L_reason= _textL_text+ _assocL_assoc+ _gateL_gate. (4) Selective Correction. The base action expert is trained with conditional flow matching and then frozen for physical prediction and residual learning. For forecast-based prediction, annotated entity motion supervises the forecasts through ℒfutureL_future (Appendix C, Equation 12). As an alternative, physical supervision targets proposal-relative clearance computed from recorded entity motion, with higher weight near contact. In addition, offline planner screens for target-clearance gain, retained progress, and no newly overlapping entities define residual targets relative to the frozen planner’s field at the same stored flow state. Distillation fits the applied correction with ℒfieldL_field and the ungated residual with ℒrawL_raw, penalizes raw corrections in risk-inactive states with ℒoffL_off, and regularizes their consistency and magnitude with ℒregL_reg: ℒres=λfieldℒfield+λrawℒraw+λoffℒoff+ℒreg.L_res= _fieldL_field+ _rawL_raw+ _offL_off+L_reg. (5) The residual fitting uses ground-truth-derived interaction features; inference uses predicted physical information. Appendix C defines the component losses, teacher targets, and optimization scope. 4 Evaluation 4.1 Experimental Setup Table 1: Open-Loop Driving Performance by Grounding Paradigm. Generation mode: D discrete; C continuous; best, second, and third distinct values per column. PAI-AV Normal (n=4Kn=4K) PAI-AV OOD (n=290n=290) Model Token represents Size Mode ADE Collision Corner Dist. ADE Collision Corner Dist. 2.5 s 5.0 s 6.4 s 2.5 s 5.0 s 6.4 s 2.5 s 5.0 s 6.4 s 2.5 s 5.0 s 6.4 s Token-as-Content FSDrive† spatio-temporal states 2.1B D 0.417 1.809 3.112 2.943 10.721 3.231 0.546 2.145 3.404 3.549 20.469 3.480 Alpamayo-R1 causes 10B D 0.551 2.612 4.417 4.269 13.443 4.599 0.596 2.691 4.402 5.805 24.844 4.568 Alpamayo-R1 causes 10B C 0.535 2.263 3.802 3.714 12.472 3.945 0.541 2.295 3.751 4.182 21.036 3.872 AutoVLA† semantic knowledge 2.1B D 0.555 1.780 2.848 2.557 9.014 2.910 0.563 1.973 3.096 2.809 14.170 3.148 LCDrive action-conditioned futures 0.6B D 0.332 1.503 2.615 2.292 9.100 2.676 0.475 1.994 3.268 2.809 17.353 3.330 Alpamayo 1.5 causes 10B D 0.425 1.881 3.163 3.311 10.951 3.279 0.522 2.148 3.412 3.995 21.223 3.513 Alpamayo 1.5 causes 10B C 0.575 2.247 3.647 3.952 12.806 3.770 0.585 2.240 3.528 4.370 21.848 3.627 SpanVLA† semantic knowledge 3.8B C 0.496 1.818 2.844 2.531 8.339 2.931 0.482 2.059 3.167 2.310 14.232 3.532 Alpamayo 2 Super causes 34B D 0.345 1.338 2.245 2.591 6.028 2.304 0.470 1.696 2.683 2.871 9.613 2.756 Alpamayo 2 Super causes 34B C 0.299 1.304 2.214 2.097 5.473 2.260 0.391 1.606 2.601 2.060 8.864 2.645 Token-as-Reference (Ours) GroundAct entity-state addresses 3.9B D 0.266 1.204 2.091 2.149 5.967 2.138 0.371 1.545 2.540 2.185 11.923 2.592 GroundAct entity-state addresses 3.9B C 0.325 1.332 2.210 2.474 6.873 2.278 0.459 1.692 2.680 2.310 10.861 2.747 PAI-AV∗ Safety-Critical (n=512n=512) Model Token represents Size Mode ADE Collision Off-road Corner Dist. 2.5 s 5.0 s 6.4 s 2.5 s 5.0 s 2.5 s 5.0 s 6.4 s Token-as-Content FSDrive† spatio-temporal states 2.1B D 0.533 2.249 3.823 3.797 20.161 5.712 28.394 3.992 Alpamayo-R1 causes 10B D 0.977 4.084 6.584 6.770 29.490 12.695 37.565 6.939 Alpamayo-R1 causes 10B C 1.014 4.054 6.521 7.230 28.840 14.388 38.411 6.872 AutoVLA† semantic knowledge 2.1B D 0.570 1.907 3.027 1.465 14.290 4.297 17.383 3.069 LCDrive action-conditioned futures 0.6B D 0.384 1.585 2.633 1.074 12.370 1.530 9.668 2.709 Alpamayo 1.5 causes 10B D 0.782 3.278 5.281 6.413 29.330 10.286 36.198 5.548 Alpamayo 1.5 causes 10B C 0.865 3.384 5.396 6.152 29.460 10.221 37.402 5.665 SpanVLA† semantic knowledge 3.8B C 0.561 2.213 3.182 1.302 15.625 2.539 17.643 5.195 Alpamayo 2 Super causes 34B D 0.528 1.768 2.797 1.335 7.585 3.092 7.878 2.887 Alpamayo 2 Super causes 34B C 0.384 1.515 2.491 0.390 7.420 1.497 7.585 2.558 Token-as-Reference (Ours) GroundAct entity-state addresses 3.9B D 0.382 1.528 2.506 0.553 7.096 1.107 6.868 2.580 GroundAct entity-state addresses 3.9B C 0.431 1.565 2.501 0.488 9.668 1.042 8.496 2.571 Models and Comparisons. We instantiate GroundAct with a Qwen3-VL-2B vision–language backbone (Bai et al., 2025) and a flow-matching action expert (2.1B and 1.8B parameters; Table 1). Actions are generated either as discrete action tokens from the VLM backbone or as a continuous trajectory from the action expert; the selective correction of §3.4 modifies the expert’s vector field, so it is active only in the continuous mode. We nonetheless report discrete generation in the open-loop evaluation, which isolates the effect of entity grounding on downstream reasoning from that of the correction. We compare against Alpamayo-R1 (NVIDIA et al., 2025c), Alpamayo 1.5 (NVIDIA, 2026a), Alpamayo 2 Super (NVIDIA, 2026b), FSDrive† (Zeng et al., 2025), LCDrive (Tan et al., 2026), AutoVLA† (Zhou et al., 2025), and SpanVLA† (Zhou et al., 2026). The Alpamayo models and LCDrive are evaluated from their released checkpoints without fine-tuning. FSDrive†, AutoVLA† and SpanVLA† are the matched controls: each method’s defining component is re-implemented on our backbone and fine-tuned, without subsequent reinforcement learning, on the same 140.5K PAI-AV training samples as GroundAct. See Appendix D for further details. Driving Evaluation Protocols. Open-loop evaluation uses the PAI-AV Normal (4K samples) and OOD (290 samples) splits, along with 512 high-risk samples from PAI-AV∗ collected under the same data configuration (Appendix E). All models receive four temporally ordered front-wide camera frames (t−0.3t-0.3 s, t−0.2t-0.2 s, t−0.1t-0.1 s, t) and generate six candidate trajectories over a 6.4 s horizon. Following LCDrive (Tan et al., 2026), we report ADE, corner distance, collision rate, and off-road rate averaged over the six candidates. Closed-loop evaluation uses 913 scenes from the AlpaSim ‘public_2601’ test suite (NVIDIA et al., 2025b), with the same four-frame input. We report progress, three safety metrics, and two diagnostic distances (Table 2; Appendix H). 4.2 Driving Performance Open-Loop Planning. Table 1 evaluates GroundAct under both discrete and continuous generation. Across all three datasets, GroundAct achieves the best or comparable performance on nearly all metrics, despite being substantially smaller than Alpamayo 2 Super. The discrete variant operates solely with the 2.1B VLM, rather than the full 3.4B model. Here, entity references retrieve the corresponding physical states into the reasoning process, yielding strong trajectory accuracy across all three datasets and long-horizon safety on the safety-critical set. The continuous variant additionally incorporates state-based plan correction through the action expert. Its performance remains competitive but slightly trails Alpamayo 2 Super on several metrics. Table 5 attributes this gap to the flow-matching head (i.e., action expert trained from scratch on substantially less data and without Alpamayo’s RL post-training) rather than to the correction itself. Together, these results show the benefit of entity grounding in both reasoning and continuous plan refinement. Table 2: Closed-Loop Driving Performance on AlpaSim. Each model uses its representative generation mode: discrete generation for FSDrive, AutoVLA, and LCDrive, and continuous generation otherwise; best, second, and third distinct values for Progress and Safety Metrics. Metric definitions are provided in Appendix H. Safety Metrics ↓ Additional Analysis (m) Model # Params Progress ↑ At-fault collision Off-road Wrong lane Distance to GT Traj. Min. obstacle distance FSDrive† (Zeng et al., 2025) 2.1B 0.604 0.193 0.271 0.210 2.577 4.447 Alpamayo-R1 (NVIDIA et al., 2025c) 10B 0.682 0.156 0.189 0.295 2.781 4.649 AutoVLA† (Zhou et al., 2025) 2.1B 0.706 0.118 0.185 0.211 2.544 3.989 LCDrive (Tan et al., 2026) 0.6B 0.639 0.259 0.263 0.189 2.825 3.871 Alpamayo 1.5 (NVIDIA, 2026a) 10B 0.682 0.166 0.155 0.297 2.843 4.407 SpanVLA† (Zhou et al., 2026) 3.8B 0.696 0.156 0.184 0.207 2.896 3.794 Alpamayo 2 Super (NVIDIA, 2026b) 34B 0.713 0.084 0.147 0.220 2.799 3.721 GroundAct 3.9B 0.752 0.105 0.153 0.207 2.511 3.641 Closed-Loop Driving. In Table 2, GroundAct, with the correction of §3.4 active, attains the highest progress, ahead of the much larger, RL post-trained Alpamayo 2 Super, and is the only model in the top two on progress and all three safety metrics. It is better than or tied with every matched control, which shares its backbone and training data, on each of these metrics, including SpanVLA† at comparable size. It is also comparable to Alpamayo 2 Super, with only marginal gaps in at-fault collision and off-road performance. The diagnostic columns indicate that this safety does not come from conservative margins: GroundAct deviates least from the GT path despite driving furthest and has the smallest clearance to other actors, consistent with selective correction (Table 4.3(b)), whereas the two earlier Alpamayo models keep wider clearance yet show the highest wrong-lane rates. Figure 4: Reference-Based Reasoning with Varying Entity References. Table 3: Make Grounding Consequential. Continuous generation; Best; second-best. VLM backbone Spatial grounding Grounded reasoning GroundAct (Ours) Ground – ✓ ✓ ✓ Reference – – ✓ ✓ Act – – – ✓ Trajectory (m)↓ ADE 2.5 s 0.551 0.479 0.463 0.431 ADE 5.0 s 2.308 1.893 1.651 1.565 ADE 6.4 s 3.498 2.985 2.581 2.501 Safety (%)↓ Coll. 2.5 s 1.465 1.237 0.716 0.488 Coll. 5.0 s 17.741 13.639 11.979 9.668 Off-road 2.5 s 2.637 1.855 1.302 1.042 Off-road 5.0 s 19.857 17.415 12.988 8.496 Figure 5: Learn to Reference. 4.3 Ablation and Analysis We examine three aspects of GroundAct’s grounding pathway: whether the grounding connection should be preserved through action refinement, whether the model learns current entity–reference associations, and whether reasoning-selected references remain consequential for planning and support selective correction. Grounding Connection. Table 5 examines the value of carrying grounded information through reasoning into action refinement. With a shared backbone and continuous-generation protocol, grounded reasoning improves on spatial grounding alone, and interaction-driven correction further reduces trajectory error, collision rates, and off-road rates. These results support the combined design, without isolating each component’s standalone contribution. Learning Current Entity Associations. Under the reported evaluation, successful language generation can coexist with failed entity referencing (Figure 5). The comparison supports explicitly supervising both when to refer and which current entity state to select, accommodating changing entity–token associations. Reference Dependence and Selection. The intervention audit shows that references are consequential for planning, with evidence of safety utility on the Safety cohort (Table 4.3(a)). It does not establish that reasoning-based selection is safer than the tested heuristics, or equivalent to them. Table 4: Reference Dependence and Selective Correction. (a) Collision-rate increase relative to Ours (p). Matched distractor: another same-class entity at a similar distance, with at least 1 m clearance from the base plan. Highest predicted risk: the entity with the highest predicted risk to the ego plan. (b) Preserved (X m): percentage of initially safe plans whose maximum time-aligned distance from the base plan is at most X m over 6.4 s. (a) Reference-to-Action Dependence Reference Selection PAI-AV* Safety PAI-AV OOD No reference +4.54% +7.03% Matched distractor +3.15% +7.67% Nearest entity +1.43% +2.61% Highest predicted risk +1.26% +1.51% (b) Selective Correction Outcome Rate Safe → Unsafe (Collision or Off-road) 2.2% Preserved (0m; exactly the same) 53.9% Preserved (3m) 70.4% Figure 6: Selective Correction. BEV details show six matched trajectories. Selective Correction and Preservation. The final policy can repair unsafe plans while preserving some already-safe outputs, but may also introduce new safety issues (Table 4.3(b)) or leave existing ones insufficiently corrected (Figure 6). Preservation reflects both correction and final arbitration and should therefore be attributed to the complete policy. We discuss these failures further in §4.4. Grounding Reasoning through Entity References. Figure 4 illustrates how GroundAct references specific perceived entities when reasoning about following, lane keeping, lane changing, and stopping. These references enable retrieval of the corresponding continuous physical states, grounding subsequent reasoning and planning in the entities selected during reasoning. Their physical states and interactions then guide downstream decisions and shape the evolving plan. 4.4 Additional Analysis Reasoning Alignment with Predicted and GT Behaviors. We further investigate how grounding affects reasoning–action alignment and alignment with ground-truth behavior. Specifically, we measure the consistency between meta-actions (e.g., stop, decelerate) inferred from the reasoning and those derived from either the predicted or ground-truth trajectories (Table 4.4). Our method yields substantial improvements in both settings. We attribute these gains to two key factors: (1) improved spatial awareness of surrounding objects and (2) an explicit grounding chain that links reasoning to planning. Table 5: Reasoning-Action Alignment. D discrete; C continuous. Appendix I. Model F1 Recall Precision vs. meta-action from predicted trajectory Alpamayo-1.5 D 55.98 47.24 68.70 Alpamayo-1.5 C 51.61 44.74 60.96 GroundAct (Ours) C 68.44 69.38 67.52 vs. meta-action from GT trajectory Alpamayo-1.5 D 58.48 48.77 73.03 Alpamayo-1.5 C 58.52 49.06 72.51 GroundAct (Ours) C 73.79 71.81 75.89 Object Detection and Referencing. We follow Rex-Omni (Jiang et al., 2026), which evaluates object detection and pointing in 2D images, and adapt its evaluation protocol and metrics to BEV space. See Appendix I for evaluation settings and baseline details. As shown in Table 4.4, GroundAct performs strongly on both detection and geometry-aware referencing of planning-critical entities. These results indicate that our entity representation captures both semantic identity and spatial geometry, providing the physical information needed to ground prediction and planning in entity states and compute their physical risks. Table 6: Detection and Referencing. Appendix I for baseline details. Detection (%) Referencing (%) Method mAP NDS (BEV) F1@ Point F1@ IoU Critical mIoU WildDet3Dz 11.71 17.07 28.00 19.43 52.99 FCOS3D 27.07 44.40 49.30 36.25 69.22 MonoDETR 21.98 39.43 43.26 30.72 74.52 PETR 33.52 46.75 41.11 26.50 69.02 PETRv2 31.27 47.95 55.02 40.09 70.29 GroundAct 35.96 48.65 54.52 42.44 75.20 ∗ WildDet3Dz: zero-shot; others: fine-tuned. Figure 7: Failure Case. Pedestrians occluded at planning time emerge during execution. Failure Case: Stale References. The correction acts only on entities referenced during reasoning, using their states at planning time. After the plan is generated, references can become outdated when occluded entities emerge (Figure 7) or surrounding entities move rapidly. In either case, the correction may lack up-to-date referents to act on. Despite this limitation, GroundAct maintains strong planning performance in the safety-critical and OOD scenarios in Table 1. A natural next step is anticipatory reasoning: predicting future states to identify which entities will become planning-critical, and when, during execution. 5 Conclusion We revisit what groundedness should mean when the output is an action. GroundAct preserves access to the continuous states of reasoning-selected entities through Token-as-Reference, allowing their predicted interactions to selectively shape the ego vehicle’s evolving plan. Open- and closed-loop evaluations show strong planning performance, while a controlled audit shows that disabling the referencing pathway from reasoning to planning can compromise driving safety. These results support grounded planning: preserving the physical connection between what reasoning identifies as relevant and how the vehicle acts. References Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu Qwen3-vl technical report. External Links: 2511.21631, Link Cited by: §4.1. Caesar et al. (2020) H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom NuScenes: a multimodal dataset for autonomous driving. External Links: 1903.11027, Link Cited by: Appendix I. Cho et al. (2025) M. Cho, R. Ohana, C. Jacobsen, A. Jothi, M. Chen, Z. M. Mao, and E. Can TC-lora: temporally modulated conditional lora for adaptive diffusion control. External Links: 2510.09561, Link Cited by: Appendix B. Chung et al. (2025) J. Chung, J. Kim, S. Kim, J. Lee, M. S. Kim, and Y. Yu V1: learning to point visual tokens for multimodal grounded reasoning. External Links: 2505.18842, Link Cited by: §2. Hu et al. (2023) Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu, J. Dai, Y. Qiao, and H. Li Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17853–17862. Cited by: Appendix B, §2. Huang et al. (2026) W. Huang, J. Zhang, S. Li, T. Jia, J. Duan, Y. Cheng, J. Cho, M. Wallingford, R. Soraki, C. D. Kim, S. Liu, D. Clay, T. Anderson, W. Han, A. Farhadi, B. Hariharan, Z. Ren, and R. Krishna WildDet3D: scaling promptable 3d detection in the wild. External Links: 2604.08626, Link Cited by: Appendix I. Huang et al. (2023) Z. Huang, H. Liu, and C. Lv GameFormer: game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3903–3913. Cited by: Appendix B, §2. Jia et al. (2026) F. Jia, L. Liu, Z. Song, C. Jia, H. Ye, X. Hao, and L. Chen DriveWorld-VLA: unified latent-space world modeling with vision–language–action for autonomous driving. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: Appendix B, §2, §2. Jiang et al. (2026) Q. Jiang, J. Huo, X. Chen, Y. Xiong, Z. Zeng, Y. Chen, T. Ren, J. Yu, and L. Zhang Detect anything via next point prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 25472–25483. Cited by: Appendix I, §4.4. Jing et al. (2026) P. Jing, V. S. Huang, H. Lu, J. Dai, Y. Xie, and B. Zhu Action emergence from streaming intent. External Links: 2605.12622, Link Cited by: Appendix B, §2. Li et al. (2026) Y. Li, S. Shang, W. Liu, B. Zhan, H. Wang, Y. Wang, Y. Chen, X. Wang, Y. An, C. Tang, L. Hou, L. Fan, and Z. Zhang DriveVLA-w0: world models amplify data scaling law in autonomous driving. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix B, §2. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Appendix B, §2. Liu et al. (2022) Y. Liu, T. Wang, X. Zhang, and J. Sun PETR: position embedding transformation for multi-view 3d object detection. In European Conference on Computer Vision (ECCV), p. 531–548. External Links: Document Cited by: Appendix I. Liu et al. (2023) Y. Liu, J. Yan, F. Jia, S. Li, A. Gao, T. Wang, and X. Zhang PETRv2: a unified framework for 3d perception from multi-camera images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3262–3272. Cited by: Appendix I. Luo et al. (2026) Y. Luo, F. Li, S. Xu, Y. Ji, Z. Zhang, B. Wang, Y. Shen, J. Cui, L. Chen, G. Chen, H. Ye, Z. Yang, and F. Wen LaST-vla: thinking in latent spatio-temporal space for vision-language-action in autonomous driving. External Links: 2603.01928, Link Cited by: §2. NVIDIA et al. (2025a) NVIDIA, H. A. Alhaija, J. Alvarez, M. Bala, T. Cai, T. Cao, L. Cha, J. Chen, M. Chen, F. Ferroni, S. Fidler, D. Fox, Y. Ge, J. Gu, A. Hassani, M. Isaev, P. Jannaty, S. Lan, T. Lasser, H. Ling, M. Liu, X. Liu, Y. Lu, A. Luo, Q. Ma, H. Mao, F. Ramos, X. Ren, T. Shen, X. Sun, S. Tang, T. Wang, J. Wu, J. Xu, S. Xu, K. Xie, Y. Ye, X. Yang, X. Zeng, and Y. Zeng Cosmos-Transfer1: conditional world generation with adaptive multimodal control. External Links: 2503.14492, Link Cited by: Appendix B. NVIDIA et al. (2025b) AlpaSim: a modular, lightweight, and data-driven research simulator for autonomous driving External Links: Link Cited by: Appendix H, §1, §4.1. NVIDIA et al. (2025c) NVIDIA, Y. Wang, W. Luo, J. Bai, Y. Cao, T. Che, K. Chen, Y. Chen, J. Diamond, Y. Ding, W. Ding, L. Feng, G. Heinrich, J. Huang, P. Karkus, B. Li, P. Li, T. Lin, D. Liu, M. Liu, L. Liu, Z. Liu, J. Lu, Y. Mao, P. Molchanov, L. Pavao, Z. Peng, M. Ranzinger, E. Schmerling, S. Shen, Y. Shi, S. Tariq, R. Tian, T. Wekel, X. Weng, T. Xiao, E. Yang, X. Yang, Y. You, X. Zeng, W. Zhang, B. Ivanovic, and M. Pavone Alpamayo-R1: bridging reasoning and action prediction for generalizable autonomous driving in the long tail. arXiv preprint arXiv:2511.00088. Cited by: §1, §2, §4.1, Table 2. NVIDIA (2025) NVIDIA Physical AI Autonomous Vehicles. Note: Hugging Face dataset cardAccessed September 2026 External Links: Link Cited by: §1. NVIDIA (2026a) NVIDIA Alpamayo 1.5 10B. Note: Hugging Face model cardAccessed September 2026 External Links: Link Cited by: §4.1, Table 2. NVIDIA (2026b) NVIDIA Alpamayo 2 Super. Note: Hugging Face model cardAccessed September 2026 External Links: Link Cited by: §1, §4.1, Table 2. Peng et al. (2026) Z. Peng, W. Ding, Y. You, Y. Chen, W. Luo, T. Tian, Y. Cao, A. Sharma, D. Xu, B. Ivanovic, B. Li, Y. Wang, and M. Pavone Counterfactual VLA: self-reflective vision-language-action model with adaptive reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 4022–4031. Cited by: §1, §2. Sadat et al. (2020) A. Sadat, S. Casas, M. Ren, X. Wu, P. Dhawan, and R. Urtasun Perceive, predict, and plan: safe motion planning through interpretable semantic representations. External Links: 2008.05930, Link Cited by: Appendix B, §2. See et al. (2017) A. See, P. J. Liu, and C. D. Manning Get to the point: summarization with pointer-generator networks. External Links: 1704.04368, Link Cited by: §3.3. Shang et al. (2026) S. Shang, B. Zhan, Y. Yan, Y. Wang, Y. Li, Y. An, X. Wang, J. Liu, L. Hou, L. Fan, Z. Zhang, and T. Tan DynVLA: learning world dynamics for action reasoning in autonomous driving. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1, §2. Song et al. (2021) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, External Links: Link Cited by: Appendix B, §2. Tan et al. (2026) S. Tan, K. Chitta, Y. Chen, R. Tian, Y. You, Y. Wang, W. Luo, Y. Cao, P. Krähenbühl, M. Pavone, and B. Ivanovic Latent chain-of-thought world modeling for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 39724–39733. Cited by: §1, §2, §4.1, §4.1, Table 2. Tian et al. (2025) T. Tian, B. Li, X. Weng, Y. Chen, E. Schmerling, Y. Wang, B. Ivanovic, and M. Pavone Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. In Proceedings of The 8th Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research, Vol. 270, p. 3656–3673. External Links: Link Cited by: §2. Wang et al. (2021) T. Wang, X. Zhu, J. Pang, and D. Lin FCOS3D: fully convolutional one-stage monocular 3d object detection. External Links: 2104.10956, Link Cited by: Appendix I. Wang et al. (2022) Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon DETR3D: 3d object detection from multi-view images via 3d-to-2d queries. In Proceedings of the 5th Conference on Robot Learning, A. Faust, D. Hsu, and G. Neumann (Eds.), Proceedings of Machine Learning Research, Vol. 164, p. 180–191. Cited by: Appendix C. Zeng et al. (2025) S. Zeng, X. Chang, M. Xie, X. Liu, Y. Bai, Z. Pan, M. Xu, and X. Wei FutureSightDrive: thinking visually with spatio-temporal cot for autonomous driving. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §2, §4.1, Table 2. Zhang et al. (2026) J. Zhang, Y. Deng, K. Chang, and W. Wang Thinking with visual grounding. External Links: 2606.16122, Link Cited by: §2. Zhang et al. (2023a) L. Zhang, A. Rao, and M. Agrawala Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3836–3847. Cited by: Appendix B, §2. Zhang et al. (2023b) R. Zhang, H. Qiu, T. Wang, Z. Guo, Z. Cui, Y. Qiao, H. Li, and P. Gao MonoDETR: depth-guided transformer for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9155–9166. Cited by: Appendix I. Zhou et al. (2025) Z. Zhou, T. Cai, S. Z. Zhao, Y. Zhang, Z. Huang, B. Zhou, and J. Ma AutoVLA: a vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2, §4.1, Table 2. Zhou et al. (2026) Z. Zhou, R. Yang, Xuewei, Qi, Y. Guo, S. X. Chen, T. Feng, K. Pistunova, Y. Shen, L. Su, and J. Ma SpanVLA: efficient action bridging and learning from negative-recovery samples for vision-language-action model. External Links: 2604.19710, Link Cited by: §2, §4.1, Table 2. Appendix A Grounding Criteria And Transfer Gap An actor is an annotated non-ego traffic participant with a GT track ID (e.g., a vehicle, pedestrian, or cyclist. We manually designate each scene’s target actor and constraint-relevant physical fact based on driving situation. Using each model’s native representation (Table 7), we assign G=1G=1 for correct grounding and G=0G=0 for an established failure; missing or ambiguous results. Table 7: Model-Specific Grounding Criteria. Model Criterion For Correct Grounding Alpamayo-1.5 & SpanVLA Reasoning correctly identifies the target actor and relevant physical fact at the specified time, without contradiction. FSDrive The decoded future image at approximately +0.5+0.5 s preserves actor class, coarse location relative to ego and road, and local road geometry. GroundAct Entity-reference validation requires a valid generated reference to the target actor in the same-run entity representation, a correct relevant fact, and BEV IoU ≥0.5≥ 0.5 with GT after temporal alignment. The above text criterion is used to validate textual reasoning. For candidate i, Ci=1C_i=1 denotes ego–actor footprint overlap or contact within 0.10.1–5.05.0 s, evaluated at 10 Hz without at-fault attribution. Let ℰE denote candidates with both grounding and collision labels. TransferGap is then computed as: TransferGap=∑i∈ℰGiCi∑i∈ℰGi.TransferGap= _i G_iC_i _i G_i. Appendix B Extended Related Work Prediction and Interaction-Aware Planning. Future-aware driving methods use visual futures, latent dynamics, or action-conditioned world models to inform planning (Li et al., 2026; Jia et al., 2026). UniAD and GameFormer couple perception, multi-agent prediction, interaction, and planning through query-based or game-theoretic structures (Hu et al., 2023; Huang et al., 2023), while P3 exposes semantic occupancy as a differentiable planning cost (Sadat et al., 2020). These works establish that structured prediction and physical interaction can directly shape planning. GroundAct asks a complementary question: which physical entities should receive explicit interaction-mediated computation, and how should that choice remain connected to reasoning? We use reasoning-referenced entities as the interface to downstream future and interaction modeling, making the selected grounding explicit in the planning computation. Reasoning-Guided Generative Planning. Flow matching and diffusion-based policies generate trajectories through iterative refinement (Lipman et al., 2023; Song et al., 2021), while guidance and residual pathways can modify a pretrained generator or controller without replacing its base behavior (Zhang et al., 2023a; NVIDIA et al., 2025a; Cho et al., 2025). Streaming Intent further connects reasoning to generative planning by deriving a discrete driving intent from chain-of-thought reasoning and using classifier-free guidance to steer a rectified-flow trajectory generator (Jing et al., 2026). GroundAct shares the goal of making reasoning consequential for generative action, but differs in the mediator. Rather than steering generation with a global semantic intent, GroundAct preserves references to individual physical entities, evaluates their interactions with the evolving ego proposal, and uses the resulting physical signal for selective residual correction. Appendix C Training Implementation Details This section specifies the targets, loss reductions, and optimization scope underlying Section 3.5. The objectives preserve the physical correspondence as the representation, reasoning interface, and planner are adapted. Physical Labels and Losses. For the training objective, we follow the prior query-based object detection framework (Wang et al., 2022). Let A denote the Hungarian assignment between entity slots and annotated objects. Matched queries receive their objects’ class and oriented BEV box, and unmatched queries receive the null class. For class logit zicz_ic and one-hot target yicy_ic, including the null category, define pic=σ(zic)p_ic=σ(z_ic), pict=yicpic+(1−yic)(1−pic)p_ic^\,t=y_icp_ic+(1-y_ic)(1-p_ic), and αict=yicαf+(1−yic)(1−αf) _ic^\,t=y_ic _f+(1-y_ic)(1- _f). The per-scene classification and box losses are ℒcls _cls =−1K∑i=1K∑c=1C+1αict(1−pict)γflogpict,ℒbox =- 1K _i=1^K _c=1^C+1 _ic^\,t(1-p_ic^\,t) _f p_ic^\,t, _box =16max(1,||)∑(i,j)∈‖b^i−bj⋆‖1. = 16 (1,|A|) _(i,j) \| b_i-b_j \|_1. (6) Here αf=0.25 _f=0.25 and γf=2 _f=2 are the focal-loss parameters, and ηbox=5 _box=5 is the box regression weight. The box code contains normalized (cx,cy,l,w,sinψ,cosψ)(c_x,c_y,l,w, ψ, ψ). Losses are averaged over scenes; matching combines class compatibility and box distance. Reference Targets and Reduction. Each annotated referent is matched to the class-consistent entity within a BEV-center distance threshold (2 m). All K=64K=64 entity states participate in Equation 7. A reference without a valid physical match is excluded from association and gate losses. Association and Gating. Let V contain reference positions with a valid match, and let it⋆i_t be the matched slot for target token yt+1y_t+1. The association objective is ℒassoc=−1max(1,||)∑t∈logexp[τcos(Wqht,WkMit⋆)]∑j=1Kexp[τcos(Wqht,WkMj)].L_assoc=- 1 (1,|V|) _t [τ (W_qh_t,W_kM_i_t )] _j=1^K [τ (W_qh_t,W_kM_j)]. (7) Gate supervision balances reference positions 1G_1 and ordinary-text positions 0G_0. With =b∈0,1:|b|>0C=\b∈\0,1\:|G_b|>0\, the objectives are ℒgate _gate =−1max(1,||)∑b∈1|b|∑t∈b[blogπt+(1−b)log(1−πt)], =- 1 (1,|C|) _b 1|G_b| _t _b [b _t+(1-b) (1- _t) ], (8) If no gate labels are valid, its loss is zero. The index sets V and bG_b collect valid entries across the current training microbatch, with scene indices suppressed. These independently normalized losses form the reference objective in Equation 4. Token Prediction. Ordinary language targets exclude reference-token positions. Let textT_text and actionT_action denote the supervised non-reference text and discrete-action positions. Their masked next-token losses are reduced separately: ℒtext=∑q∈text,actionηq[−1max(1,|q|)∑t∈qlogpLM,t(yt+1⋆)].L_text= _q∈\text,action\ _q [- 1 (1,|T_q|) _t _q p_LM,t(y_t+1 ) ]. (9) Here pLM,tp_LM,t is the vocabulary-head softmax over the full token vocabulary, and qT_q collects supervised positions across the current training microbatch. Reference supervision is excluded through target-position masking. The prefill objective ℒLML_LM uses the same weighted, separately reduced cross-entropy form on the original supervised token groups, before reference adaptation. The balanced gate loss averages the positive and negative group means when both are present and uses the remaining mean when only one group is present. Initializing Address Feedback. Association alignment first learns the pointer projections against a fixed entity representation. The feedback projection is additionally initialized to make a consumed reference resolve to its own entity. Let ai=bi+P(sg[Mi])a_i=b_i+P(sg[M_i]) be the address representation before feedback interpolation. With the pointer fixed and gradients stopped through the entity bank, address retrieval uses ℒaddr=−1max(1,|addr|)∑i∈addrlogpptrfixed(i∣ai,sg[ℳ]).L_addr=- 1 (1,|V_addr|) _i _addr p_ptr^fixed\! (i a_i,sg[M] ). (10) This objective trains the input projection to preserve the intended address. Subsequent teacher-forced reasoning consumes the corresponding live entity state at the next token position, matching the reference-consumption operation used during generation. Base Action Expert Training. The expert learns recorded actions a⋆a through conditional flow matching, conditioned on the complete frozen VLM key/value context c, including observations and reasoning: xs=(1−s)ϵ+sa⋆,ℒAE=a⋆,c,ϵ,s‖vω(xs,s,c)−(a⋆−ϵ)‖av2.x_s=(1-s)ε+sa , _AE=E_a ,c,ε,s \|v_ω(x_s,s;c)-(a -ε) \|_av^2. (11) Here ϵ∼(0,I)ε (0,I), s is noise-to-action flow time, and the squared norm averages over action coordinates and horizon positions. The objective is evaluated in the action expert’s encoded, normalized action space; geometric checks operate on decoded ego trajectories. The expectation samples Gaussian noise and flow time s∼pss p_s. Flow time is distinct from physical future time and pointer sharpness τ. The stored flow-matching sampler uses a transformed beta distribution, s=smax(1−B)s=s_ (1-B) with B∼Beta(as,bs)B (a_s,b_s), rather than a uniform flow-time distribution. Its source configuration sets smax=0.999s_ =0.999, as=1.5a_s=1.5, and bs=1b_s=1. The trained expert remains frozen during physical prediction and residual learning. Physical-Prediction Supervision. For the forecast-based formulation described in §3.4, let Sih⋆S_ih be the annotated state of the object matched to slot i at physical horizon index h, and let mihm_ih mark valid supervision. The forecast objective can be written as ℒfuture=[∑i,hmihℓstate(S^ih(k),Sih⋆)max(1,∑i,hmih)].L_future=E [ _i,hm_ih\, _state( S_ih^(k),S_ih ) (1, _i,hm_ih) ]. (12) The state penalty supervises the physical quantities represented by the forecast. The expectation includes training examples and sampled proposal states indexed by k; h indexes physical time, whereas k indexes refinement of the proposal. Teacher Construction and Training Inputs. Teachers are selected from frozen-planner candidates, and selection uses a 44 s evaluation horizon. The recorded teacher selector requires finite actions and trajectories, action bounds, a target-clearance gain of at least 0.50.5 m, and a progress ratio of at least 0.950.95 relative to the base proposal. If the base overlaps the target entity, the alternative must remove that overlap. It must also introduce no newly overlapping entity identities across the scene. Each stored teacher record retains the flow state, base field, flow time, integration step, and physical features used to construct its target. At a stored state (xs,s)(x_s,s), let us⋆u_s denote the teacher field and define its raw and applied residual targets as rs⋆=sg[us⋆−vbase(xs,s,c)],ds⋆=γs⋆rs⋆.r_s =sg\! [u_s -v_base(x_s,s;c) ], d_s = _s r_s . (13) The base and teacher fields are evaluated at the same state in the same action coordinates. The teacher gate γs⋆ _s determines how much of the raw correction is applied. For a retained alternative a+a^+, the recorded endpoint-based field and gate are us⋆=a+−xsmax(1−s,δ),γs⋆=bsriskmin(1,κs2),ds⋆=γs⋆sg[us⋆−vbase].u_s = a^+-x_s (1-s,δ), _s =b_s^risk (1,κ s^2), d_s = _s sg[u_s -v_base]. (14) The binary risk latch bsriskb_s^risk remains active once a teacher trace has triggered correction. The recorded ten-step configuration uses δ=0.1δ=0.1 and κ=2κ=2. The denominator floor limits endpoint conversion near the end of the flow. Residual Distillation and Risk-Off Preservation. Let zℛtrainz_R^train denote the ground-truth-derived interaction features described above. With adapter prediction r^s=Rϕ(xs,vbase,zℛtrain) r_s=R_φ(x_s,v_base,z_R^train) and d^s=γs⋆r^s d_s= _s r_s, distillation uses ℒfield=+‖d^s−ds⋆‖av2,ℒraw=on‖r^s−rs⋆‖av2.L_field=E_D_+\| d_s-d_s \|_av^2, _raw=E_D_on\| r_s-r_s \|_av^2. (15) Here +D_+ contains retained teacher traces and onD_on their risk-active states. Applying the teacher gate to both sides preserves the target’s intended correction strength; the raw term additionally supervises the adapter before gating. For risk-inactive states offD_off in the teacher traces, preservation discourages a nonzero raw correction: ℒoff=off‖r^s‖av2.L_off=E_D_off\| r_s\|_av^2. (16) Each masked term is normalized by the number of selected scalar elements in the minibatch and is zero when none are selected. These penalties supervise the correction field. Residual Consistency and Preservation. Let Δs s be the integration increment recorded with a teacher state. The predicted and teacher next states are x^′=xs+Δs(vbase+d^s) x =x_s+ s(v_base+ d_s) and x⋆′=xs+Δs(vbase+ds⋆)x =x_s+ s(v_base+d_s ). With Dh2D_h^2 the second difference along the physical action horizon, the remaining terms in Equation 5 are ℒstep _step =‖x^′−x⋆′‖av2, =E\| x -x \|_av^2, (17) ℒsmooth _smooth =∥Dh2d^s−Dh2ds⋆∥av2,ℒnorm=∥r^s∥av2, =E\|D_h^2 d_s-D_h^2d_s \|_av^2, _norm=E\| r_s\|_av^2, ℒreg _reg =λstepℒstep+λsmoothℒsmooth+λnormℒnorm. = _stepL_step+ _smoothL_smooth+ _normL_norm. The recorded distillation configuration uses (λfield,λraw,λstep,λoff,λsmooth,λnorm)=(1, 0.25, 2, 0.1, 0.01, 0.001).( _field, _raw, _step, _off, _smooth, _norm)=(1,\;0.25,\;2,\;0.1,\;0.01,\;0.001). (18) The risk-off penalty acts on the raw adapter output, so it remains informative when the applied correction is zeroed by the gate. It encourages preservation without asserting an explicit loss on the final integrated action. Table 8: Optimization Scope. Each interface is adapted while preserving the upstream information it uses. Interval Trainable parameters Fixed components Entity warm-up Entity queries and class/box heads Vision and language backbones Joint adaptation Backbones, queries, physical heads, and vocabulary head — Association alignment Pointer projections and temperature Entity encoder and language model Language adaptation Language decoder, vocabulary head, and reference gate Vision encoder, queries, physical heads, and pointer parameters Address initialization Feedback projection Pointer and source entity states Action learning Base action expert Multimodal encoder and reasoner Physical prediction Prediction module Base planner Residual distillation Residual adapter Base planner and teacher records Appendix D Experimental Setup Details Baseline Configurations. Alpamayo-R1, Alpamayo 1.5, Alpamayo 2 Super, and LCDrive are evaluated from their authors’ released checkpoints without fine-tuning; all were trained on PAI-AV, with training sets that differ in size from ours. The evaluated LCDrive uses Qwen2.5-0.5B and DINOv2-Small, with approximately 0.6B parameters. FSDrive†, AutoVLA† and SpanVLA† share GroundAct’s Qwen3-VL-2B backbone: we re-implement each method’s defining component on this backbone (e.g., action tokenization) and fine-tune the resulting model on the same 140.5K PAI-AV training samples as GroundAct, without subsequent reinforcement learning. Open-Loop Sample Selection. Normal evaluation uses the first 4,000 samples of the PAI-AV default split, and OOD evaluation uses its 290-sample OOD split. The PAI-AV∗ set contains 512 high-risk samples (Appendix E). Computational Setup. Training uses 32 NVIDIA A100-SXM4-80GB GPUs throughout all stages. For each evaluated sample, open-loop evaluation runs on a single NVIDIA A100-SXM4-80GB GPU and closed-loop evaluation on a single NVIDIA H100 80GB node. Appendix E Safety-Critical Evaluation Dataset Figure 8: Distribution of safety-critical factors in PAI-AV∗ (n=512n=512). Factors are identified from the open-loop plans generated by the released Alpamayo-R1 10B model. Bars represent mutually exclusive factor combinations, with labels indicating counts and percentages and colors indicating the number of factors. “None” denotes samples with none of the listed factors. The PAI-AV∗ open-loop evaluation set contains 512 PAI-AV samples selected as high-risk scenes; none overlaps the 140.5K training samples used by GroundAct, FSDrive†, AutoVLA†, and SpanVLA†. Figure 8 characterizes the set through the open-loop plans of the released Alpamayo-R1 10B checkpoint: at least one of collision, off-road, or wrong-lane occurs in 390 samples (76.17%), including 264 (51.56%) with two or three factors, while 122 (23.83%) show none. These factors describe a baseline’s behavior on the set rather than a property of the scenes. Appendix F Reference-Objective Evaluation Metrics Table 9: Reference-Objective Evaluation Metrics. All five reported scores are percentages. TPTP, FPFP, and FNFN denote matched predictions, unmatched predictions, and missed targets in physical-reference matching. Metric Definition Valid Emission Validity of the references emitted in generated reasoning. Sentence Complete Completion of generated reasoning, assessed separately from the correctness of its physical references. Grounding Precision Count-based precision: 100TP/(TP+FP)100\,TP/(TP+FP), where TPTP and FPFP denote matched and unmatched predicted physical referents. Grounding Recall Count-based recall: 100TP/(TP+FN)100\,TP/(TP+FN), where FNFN denotes unmatched target referents. Grounding F1 Count-based harmonic mean: 100 2TP/(2TP+FP+FN)100\,2TP/(2TP+FP+FN). Appendix G Metrics for Open-Loop Planning Evaluation Each sample is a clip and a prediction timestamp. Every model generates six candidate trajectories per sample, each with 64 poses at 10 Hz (6.4 s). Table 10 defines the four metrics. Each metric is averaged over the six candidates of a sample and then over samples. Collision and off-road are per-candidate binary events, counted once per candidate, evaluated at the discrete poses without at-fault filtering, and reported as percentages. ADE and corner distance use all samples; collision uses the annotation-available subsets. Table 10: Open-loop evaluation metrics. Lower is better for all four. Metric Horizons Definition Samples ADE (m) ↓ 2.5, 5.0, 6.4 s Mean ground-plane distance between predicted and ground-truth ego positions over the horizon. All Corner distance (m) ↓ 6.4 s The same distance averaged over the four corners of the oriented ego box. All Collision (%) ↓ 2.5, 5.0 s Fraction of candidates whose predicted ego footprint intersects any annotated actor footprint within the horizon. 3,846 / 267 / 512a Off-road (%) ↓ 2.5, 5.0 s Fraction of candidates whose predicted ego footprint contains a road-boundary point within the horizon. PAI-AV∗ onlyb aAnnotation-available subsets of Normal / OOD / PAI-AV∗. bNormal and OOD lack map data. Appendix H Metrics for Closed-Loop Driving Evaluation For Table 2, we evaluate each model on the same N=913N=913 public_2601 scenes in AlpaSim (NVIDIA et al., 2025b), with one rollout per scene. The valid evaluation interval extends from the start of policy control through the first collision of any type, off-road event, or deviation of at least 4m4\,m from the recorded ground-truth path. All metrics are computed using AlpaSim’s default protocol. Table 11: Closed-loop evaluation metrics. The two distance metrics are diagnostic and have no preferred direction. Metric Definition Progress ↑ Distance along the ground-truth path reached by projecting the final ego position onto the path, normalized by its total length. At-fault collision ↓ Fraction of scenes with at least one front or lateral ego collision. Off-road ↓ Fraction of scenes where the ego footprint contacts a road edge while not fully contained within the lane area. Wrong lane ↓ Fraction of scenes where the ego heading differs by more than 120∘120 from the direction of a lane containing the ego center. Maximum Distance to GT Trajectory (m) Maximum distance from the ego center to the nearest point on the GT path. Min. obstacle distance (m) Minimum distance between the ego footprint and any other actor’s footprint. Appendix I Setting Details of Additional Analysis Perception Baselines. We fine-tune FCOS3D (Wang et al., 2021), MonoDETR (Zhang et al., 2023b), PETR (Liu et al., 2022), and PETRv2 (Liu et al., 2023) from released checkpoints on 140,554 training frames, using one current front-wide image and camera calibration. PETRv2 uses no temporal inputs. WildDet3D (Huang et al., 2026) is evaluated zero-shot. Table 12 summarizes training settings; model names link to official implementations. Table 12: Perception Baseline Training Settings. Batch is global; LR and WD denote base learning rate and default weight decay. WildDet3D is evaluated without fine-tuning. Model / Backbone Optimizer Batch LR WD FCOS3D / R101-DCN SGD 16 ×10−32\!×\!10^-3 10−410^-4 MonoDETR / R50 AdamW 16 ×10−42\!×\!10^-4 10−410^-4 PETR / R50-DCN AdamW 8 ×10−42\!×\!10^-4 10−210^-2 PETRv2 / V-99-eSE AdamW 8 ×10−42\!×\!10^-4 10−210^-2 WildDet3D – – – – Detection And Referencing Metrics. We adapt nuScenes detection metrics (Caesar et al., 2020) and Rex-Omni pointing and overlap criteria (Jiang et al., 2026) to BEV space (Table 13). Tables 4.4 and 4.4 report percentages (higher is better), with the best value per metric and evaluation target in bold. Table 13: Detection And Referencing Metrics. All scores are reported as percentages. For the two F1 metrics, F(H)=2H/(NP+NG)F(H)=2H/(N_P+N_G), where NPN_P and NGN_G count retained predictions and GT objects, and H counts GT objects hit at least once. Metric Definition mAP Average precision using BEV center-distance matching, averaged over evaluated categories and thresholds 0.5,1,2,4\0.5,1,2,4\ m. NDS (BEV) Combines mAP with BEV translation, scale, and orientation errors; velocity and attribute terms are omitted: [5mAP+∑e∈mATE,mASE,mAOEmax(0,1−e)]/8[5\,mAP+ _e∈\mATE,mASE,mAOE\ (0,1-e)]/8, using unscaled mAP. F1@Point F(H)F(H) with a GT hit when any predicted center falls inside its BEV footprint, irrespective of category. F1@IoU F(H)F(H) with a GT hit when any prediction has rotated BEV IoU ≥0.5≥ 0.5 with it, irrespective of category. Critical mIoU Mean BEV IoU for matched GT-CoC-referenced objects, excluding unmatched objects. Pairs are selected after class-agnostic, one-to-one center matching to all GT objects within 2 m. Appendix J Inference Efficiency. Table 14: E2E Latency. Config. Ground Ref. Act Time (s) Driving VLM backbone – – – 0.88±0.140.88± 0.14 Spatial grounding ✓ – – 0.94±0.120.94± 0.12 Grounded reasoning ✓ ✓ – 0.99±0.140.99± 0.14 GroundAct ✓ ✓ ✓ 1.10±0.191.10± 0.19 As shown in Table 14, GroundAct adds 0.220.22 s to the backbone VLM’s mean inference latency (0.880.88 s), bringing the total to 1.101.10 s on a single NVIDIA A100-SXM4-80GB GPU without model- or inference-level optimization. Potential avenues for reducing latency include compiled inference and TensorRT deployment. Beyond latency reduction, another deployment direction is to decouple high-level planning from low-level control. GroundAct’s strong open-loop planning performance over a 6.46.4 s prediction horizon motivates exploring lower-frequency grounded planning (e.g., approximately 11 Hz) paired with a faster low-level controller.