Paper deep dive
Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies
Ali Ahmadi, Hamed Rahimi, Adrien Jacquet Cretides, Marie Samson, Mahdi Khoramshahi, Mohamed Chetouani
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/18/2026, 1:47:27 PM
Summary
The paper introduces HUMA, a hybrid social robot navigation framework that dynamically balances the computational efficiency of Reinforcement Learning (RL) policies with the semantic reasoning capabilities of Vision-Language Models (VLMs). HUMA uses a reactive RL policy for routine navigation and activates a post-trained VLM only when humans enter sensitive proximity zones, thereby reducing personal space violations and collisions while maintaining real-time performance. The approach is evaluated on Social-MP3D and Social-HM3D benchmarks, showing significant improvements in task success and social compliance compared to state-of-the-art baselines.
Entities (10)
Relation Signals (8)
HUMA → uses → Vision-Language Models
confidence 95% · conditioning it on a post-trained high-level VLM when a human enters sensitive situations
HUMA → uses → Reinforcement Learning
confidence 95% · HUMA uses a reactive RL policy to handle low-density, routine navigation tasks
HUMA → evaluatedon → Social-MP3D
confidence 92% · We evaluate HUMA on the Social-MP3D and Social-HM3D benchmarks
HUMA → evaluatedon → Social-HM3D
confidence 92% · We evaluate HUMA on the Social-MP3D and Social-HM3D benchmarks
LoRA → appliedto → Qwen3-VL-2B
confidence 90% · we employ LoRA within the PEFT framework... To efficiently learn the parameters θVLM
Falcon → basepolicyfor → HUMA
confidence 88% · We adopt the RL policy architecture introduced by Falcon [8] as our base navigation policy.
HUMA → deployedon → Mirokaï
confidence 85% · real-world deployment on the Mirokaï mobile robot further demonstrates the practical viability of our approach.
Qwen3-VL-2B → usedas → Vision-Language Models
confidence 85% · we initialize our VLM using Qwen3-VL 2B
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As mobile robots become more integrated into everyday human environments, social robot navigation is becoming essential for ensuring human comfort, safety, and trust. While reinforcement learning (RL) navigation policies provide the fast inference and reactive behavior necessary for real-time deployment, they still lack flexible semantic reasoning capabilities and often fail to generalize to complex social scenarios. Recent approaches have increasingly turned to vision-language models (VLMs) in place of RL policies to improve semantic and social reasoning in robot navigation. Nevertheless, their high computational cost and slow inference remain major barriers to real-time deployment. To overcome these limitations, we introduce HUMA (Hybrid Understanding for Multi-modal social Navigation), a hybrid architecture that dynamically balances the computational efficiency of RL policies with the deep semantic understanding of VLMs. Our approach uses a reactive RL policy to handle low-density, routine navigation tasks, while conditioning it on a post-trained high-level VLM when a human enters sensitive situations, such as the robot's proximity zone. We evaluate HUMA on the Social-MP3D and Social-HM3D benchmarks, where it achieves task success improvements of 20% and 3%, respectively, while significantly reducing personal space violations and human collisions against state-of-the-art baselines. Extensive ablation studies validate each architectural component, and real-world deployment on the Mirokaï mobile robot further demonstrates the practical viability of our approach.
Tags
Links
- Source: https://arxiv.org/abs/2607.10991v1
- Canonical: https://arxiv.org/abs/2607.10991v1
Trouble viewing inline? Open PDF directly →
Full Text
49,370 characters extracted from source content.
Expand or collapse full text
Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies Ali Ahmadi, Hamed Rahimi, Adrien Jacquet Crétides, Marie Samson, Mahdi Khoramshahi, Mohamed Chetouani Institut des Systèmes Intelligents et de Robotique (ISIR) Sorbonne Université France lastname@isir.upmc.fr Abstract As mobile robots become more integrated into everyday human environments, social robot navigation is becoming essential for ensuring human comfort, safety, and trust. While reinforcement learning (RL) navigation policies provide the fast inference and reactive behavior necessary for real-time deployment, they still lack flexible semantic reasoning capabilities and often fail to generalize to complex social scenarios. Recent approaches have increasingly turned to vision-language models (VLMs) in place of RL policies to improve semantic and social reasoning in robot navigation. Nevertheless, their high computational cost and slow inference remain major barriers to real-time deployment. To overcome these limitations, we introduce HUMA (Hybrid Understanding for Multi-modal social Navigation), a hybrid architecture that dynamically balances the computational efficiency of RL policies with the deep semantic understanding of VLMs. Our approach uses a reactive RL policy to handle low-density, routine navigation tasks, while conditioning it on a post-trained high-level VLM when a human enters sensitive situations, such as the robot’s proximity zone. We evaluate HUMA on the Social-MP3D and Social-HM3D benchmarks, where it achieves task success improvements of 20% and 3%, respectively, while significantly reducing personal space violations and human collisions against state-of-the-art baselines. Extensive ablation studies validate each architectural component, and real-world deployment on the Mirokaï mobile robot further demonstrates the practical viability of our approach. 1 Introduction As mobile robots increasingly transition from isolated settings into dynamic, human-centric environments such as airports or hospitals, social robot navigation appears as one of the main challenges of Human-Robot Interactions (HRI) in modern societies [31, 37]. Social Navigation (also referred to as Human-aware or Socially-aware Navigation) lies at the intersection of Human-Robot Interaction (HRI) and Robot Motion Planning [15, 31]. It concerns the ability of robots to navigate safely and efficiently in human-populated environments while respecting social norm and human safety such as judging a path’s clearance relative to human movements and obstacles [1]. This requires an agent to dynamically read social cues and anticipate human intent, transforming classical Motion Planning into a socially-aware optimization task. By adapting its trajectory to maintain a non-disruptive presence, a socially aware robot allows for safe, intuitive and comfortable interactions in shared spaces [21]. Figure 1: Overview. Existing approaches for social navigation face a fundamental trade-off: RL-based policies (left) offer fast, reactive inference but lack semantic reasoning for complex social scenarios, while VLM-based methods (middle) provide rich contextual understanding at the cost of high computational overhead and slow inference, preventing real-time deployment. Our method (right), combines a reactive RL policy, that handles routine navigation, with a post-trained VLM selectively invoked when necessary, enabling socially compliant, real-time navigation. While early frameworks relied on fixed geometric rules, current approaches to social navigation have leveraged learning-based paradigms to tackle complex environments [27, 29]. Specifically, policies trained with reinforcement learning (RL) have become a standard approach, largely due to their fast inference and reactive performance in real-time deployment. However, although RL methods are often computationally intensive and data-hungry during training, modeling the nuances of human social behavior typically requires large-scale datasets, and purely RL-based approaches frequently struggle to generalize to rare or edge-case human interactions, resulting in behavior that can be rigid or unnatural. With recent advances in foundation models, Vision-Language Models (VLMs) have been increasingly incorporated into social robot navigation frameworks due to their strong semantic understanding of visually rich and socially complex environments. VLMs leverages spatial and behavioral reasoning capabilities to decode social contexts, such as pedestrian orientation, group dynamics, or intent, that cannot be captured effectively by RL observations. In spite of their abilities, their massive computational overhead and slow inference time makes their deployment limited for real time navigation scenarios [22, 11]. Consequently, the literature lacks a balanced approach that can leverage the deep context-awareness of VLMs without sacrificing the rapid execution speed required for fluid mobile navigation. Our core insight is that high-level semantic reasoning is an unnecessary overhead in classical, uncrowded environments, but becomes critical when maneuvering in proximity to humans. We believe this insight is addressable by employing a adaptive arbitration approach, using a RL policy to handle standard, low-density navigation tasks, while a VLM is activated as the planner the moment a human enters the agent’s proximity zone. This allows the robot to interpret complex social contexts and behaviors to guide the underlying navigation controller. As shown in Figure 2, this paper introduces HUMA, a hybrid human-aware navigation framework that dynamically balances computational efficiency with deep semantic understanding. The HUMA architecture consists of an efficient RL policy for standard navigation, conditioned on a Personal Space Compliance (PSC) switch that enables a VLM for social navigation (trained by LoRA adapters) to intervene under socially sensitive conditions, such as close-proximity human interactions. Through extensive evaluation on the Social-HM3D and Social-MP3D benchmarks [8], we demonstrate that HUMA significantly outperforms existing baselines, achieving higher social acceptance and improved navigation success rates. Moreover, we perform an ablation study to investigate different configurations of HUMA, and deploy it on the Mirokai robot from Enchanted Tools to showcase its efficiency in real-world scenarios. 2 Related Work Motion Planning in Social Navigation Motion Planning is a fundamental challenge in Social Navigation [15], requiring robots to navigate safely, efficiently, and naturally in human-populated environments while adhering to social norms and interaction dynamics. Beyond basic collision avoidance, socially aware navigation demands behaviours that account for human comfort and predictability, such as respecting personal space, adapting motion and speed to surrounding pedestrians, and communicating navigation intent in a socially compliant manner [31]. Approaches for this high-level decision-making task mainly consist of planning-based approaches (search and sampling) [14, 2, 32, 13, 36, 41] and learning-based approaches [18, 25, 12, 3]. Classical planning methods for robot navigation include search-based approaches such as A* and D* and sampling-based methods such as PRM and RRT [19, 5, 14]. Although recent work improves adaptability through replanning and global-local mapping strategies [24, 33], these methods still largely depend on fixed geometric heuristics and explicit maps, motivating the development of learning-based approaches. Learning-based Motion Planning Learning-based methods leverage data-driven models such as Deep Reinforcement Learning (DRL), Convolutional Neural Networks (CNNs), and Inverse RL to predict decisions and adapt to complex environments [40, 25, 3, 38, 29]. Early RL-based methods learn navigation policies end-to-end from sensor observations to control commands, enabling fast inference and efficient obstacle avoidance in crowded environments [16]. For instance, Gong et al. [8] proposed Falcon, a future-aware RL framework for social navigation that explicitly predicts human trajectories. While these approaches improve high-level decision-making and navigation planning, they are often domain-specific, require extensive task-dependent training and data, and generalize poorly to unseen social scenarios due to limited semantic understanding and low interpretability. VLM-based Motion Planning Recent advances in socially aware robot navigation have increasingly shifted toward generative approaches [35], particularly VLM [6, 20]. Recent VLM-based frameworks introduce higher-level contextual reasoning and improved social awareness by leveraging multimodal semantic priors learned from large-scale data. Systems such as CoNVOI [30], Walk with Me [42], VLM-Social-Nav [34], and VLM-Informed Path Selection [7] exploit vision-language reasoning for socially compliant trajectory evaluation, scene understanding, and instruction following, achieving improved human-aware navigation and stronger generalization across diverse environments. While these methods significantly improve success rates and human-aware navigation metrics, they remain computationally expensive, require massive training data and long training times, and often suffer from high inference latency, particularly in long-horizon interactive navigation tasks [11].When comparing VLM-based motion planning with DRL-based approaches, each exhibits complementary strengths and weaknesses: DRL methods typically offer lower planning latency, whereas VLM-based methods provide superior semantic understanding, accuracy, and generalizability. This motivates the exploration of a balanced framework that leverages the advantages of both paradigms to improve overall navigation performance and safety. Figure 2: HUMA is a hybrid human-aware navigation framework that dynamically balances computational efficiency with deep semantic understanding. Input signals are routed via a PSC switch to either a lightweight RL policy for standard navigation, or a post-trained VLM pipeline with visual input pre-processing for socially sensitive conditions such as close-proximity human interactions. 3 Method 3.1 Problem Formulation We formulate Robot Navigation as a Goal-Conditioned Partially Observable Markov Decision Process (GC-POMDP), ℳ=(,,,,,)M=(S,O,Z,G,A,T), where S is the state space, A is the action space, a discrete set of high-level navigational commands, and (⋅|s,a)T(·|s,a) is the state transition function. The agent’s goal g∈g represents the relative coordinate vector (xg,yg)∈ℝ2(x_g,y_g) ^2 to the target location. At each step t, the agent processes an observation ot∈o_t from the state st∈s_t according to the observation function (ot|st,at−1)Z(o_t|s_t,a_t-1). oto_t is composed of a depth map tD_t, the relative goal vector gtg_t to the robot, and the distance to the closest human dℋd_H. The agent then queries a policy π to select an action π(ot)=atπ(o_t)=a_t, with at∈a_t , and transitions to the next state st+1∼(⋅|st,at)s_t+1 (·|s_t,a_t). In the case of Social Robot Navigation, the state space S and observation space O are fundamentally partitioned into distinct regimes depending on the presence and proximity of humans. The objective is to find a global policy π∗(at|ot)π^*(a_t|o_t) that successfully reaches a terminal goal state st≈gs_t≈ g while minimizing collisions and maintaining socially compliant behavior. 3.2 HUMA We propose HUMA, a hybrid social robot navigation framework that dynamically switches between a computationally efficient RL policy for standard navigation and a VLM post-trained for socially-aware navigation in the presence of humans. The core architecture comprises three main components: the RL navigation policy, the VLM reasoning module, and a human-dependent switching mechanism. RL Navigation Policy We adopt the RL policy architecture introduced by Falcon [8] as our base navigation policy. At each timestep t, the policy processes egocentric depth images DtD_t via a visual encoder alongside point-goal coordinate data gtg_t transformed via a linear encoder. A two-layer LSTM recurrent state encoder extracts latent spatio-temporal features to output discrete navigation actions at∈a_t . The policy is trained via Decentralized Distributed Proximal Policy Optimization (D-PPO). To enforce adherence to human social norms without global maps or prior path knowledge, the primary policy updates are driven by a composite reward function RsocialnavtR_socialnav^t, which explicitly subtracts a multi-component Social Cognition Penalty RscptR_scp^t from goal-directed PointNav rewards such that: Rsocialnavt=Rpointnavt−RscptR_socialnav^t=R_pointnav^t-R_scp^t The goal-directed PointNav component is defined as Rpointnavt=−βdΔd−rslack+βsucc⋅IsuccR_pointnav^t=- _d _d-r_slack+ _succ· I_succ, which accounts for the change in distance to the target Δd _d, a step penalty rslackr_slack, and a success indicator IsuccI_succ. This is counterbalanced by the Social Cognition Penalty, formulated as Rscpt=rcoll+rprox+rtrajR_scp^t=r_coll+r_prox+r_traj. Within this penalty, the collision component rcollr_coll imposes discrete weight penalties for physical contact with static obstacles or human agents. The proximity penalty rproxr_prox scales exponentially if the robot breaches a 2.0 m2.0 m safety envelope around any human agent, computed as rprox=∑i=1Nβprox⋅exp(−dit)r_prox= _i=1^N _prox· (-d_i^t) if dit<2.0 md_i^t<2.0 m and 0 if dit≥2.0 md_i^t≥ 2.0 m. Here N is the total number of dynamic humans in the environment, βprox _prox represents the penalty weight term for human proximity, and ditd_i^t denotes the distance between the robot and the i-th human at the current timestep t. Finally, the trajectory obstruction penalty (rtrajr_traj) penalizes the robot if its immediate path intersects within 0.05 m0.05 m of a human’s projected H-step future path, applying heavier weight to imminent path conflicts: rtraj=∑k=t+1t+H∑i=1Nβtraj⋅(1k−t+1)if dtraj,ik<0.05 m0if dtraj,ik≥0.05 mr_traj= _k=t+1^t+H _i=1^N cases _traj· ( 1k-t+1 )&if d_traj,i^k<0.05 m\\ 0&if d_traj,i^k≥ 0.05 m cases where H is the look-ahead horizon for trajectory forecasting, βtraj _traj is the penalty weight scaling factor for trajectory obstructions, k represents the specific future timestep being evaluated, and dtraj,ikd_traj,i^k represents the distance between the robot’s and the i-th human’s predicted future positions at the k-th forward timestep. Additionally, the training phase utilizes a Spatial-Temporal Precognition Module. Consequently, the total loss function ℒtotalL_total balances primary navigation PPO updates ℒmainL_main with the auxiliary objectives loss ℒauxL_aux such that ℒtotal=βmainℒmain+βauxℒauxL_total= _mainL_main+ _auxL_aux. ℒauxL_aux is defined as ℒaux=ℒcount+ℒpos+ℒtrajL_aux=L_count+L_pos+L_traj, where ℒcountL_count is optimized using cross-entropy loss to estimate the total human count , while mean squared error (MSE) is applied to both the current human position tracking ℒposL_pos and the multi-step future trajectory forecasting ℒtrajL_traj objectives. Vision Language Model Reasoning Module In contrast to RL Policy, the input domain for the VLM, denoted as otVLMo_t^VLM, integrates pre-processed the multimodal data streams into a single composite image tfusedI_t^fused and a text prompt tP_t: otVLM=(tfused,t)o_t^VLM= (I_t^fused,P_t ). To optimize downstream context awareness for the model, we perform a visual token pre-processing step. Raw depth map inputs tD_t are converted into information-dense visual tokens before being fed to the VLM. The depth map is normalized to an 8-bit unsigned integer space, mapped to a 3-channel JET colormap to partition distance zones (close, medium, far), and processed via a Canny edge detector to map structural boundaries. Finally, these binary edges are superimposed as a white contour mask over the colormap alongside the rendered GPS goal vector gtg_t to create the unified image tfusedI_t^fused. In parallel, the GPS goal state is encoded linguistically, with the text prompt tP_t being updated at each timestep t to append state-dependent telemetry to the static social navigation rules: t=[static,ℋtext(gt)]P_t= [P_static,H_text(g_t) ] where ℋtextH_text formats the relative goal coordinates into explicit linguistic descriptions, such as the distance in meters or the target angle in radians. The final inference step passes these inputs to the VLM framework, parameterized by θVLM _VLM, to generate the next n actions at:t+n∈a_t:t+n to perform: at:t+n=πVLM(otVLM;θVLM)a_t:t+n= _VLM (o_t^VLM; _VLM ) (1) To bridge the linguistic output of the VLM with the discrete action space A, the model generates specific textual action tokens (e.g., [FORWARD], [LEFT], [STOP]). To efficiently learn the parameters θVLM _VLM, our model is adapted using Parameter-Efficient Fine-Tuning (PEFT) via Low-Rank Adaptation (LoRA), allowing the model to translate the multimodal image-text input (otVLMo_t^VLM) into a sequence of discrete navigation commands at:t+na_t:t+n. Switching Mechanism To balance computational efficiency with high-level social reasoning, HUMA employs a dynamic policy switching mechanism. Let dt=minh‖probott−pht‖2d_t= _h\|p_robot^t-p_h^t\|_2 represent the minimum Euclidean distance between the robot and any human at timestep t. Following Falcon’s definition of the PSC Score [8], a binary compliance signal is defined based on a 1.0 m1.0 m threshold (accounting for a 0.3 m0.3 m human collision radius and 0.25 m0.25 m robot radius). To prevent recent spatial violations from being smoothed out by long-term metrics, our mechanism monitors a rolling history of the last W binary signals. The localized compliance metric, PSCWPSC_W, is formulated as: PSCW=1W∑i=t−W+1t[di>1.0 m]PSC_W= 1W _i=t-W+1^t1[d_i>1.0 m] At each timestep, the active policy is selected based on the weighted PSC score PSCWPSC_W. The robot follows the efficient RL policy when social compliance remains above a threshold (PSCW≥θPSC_W≥θ), and switches to the VLM policy when personal space violations occur (PSCW<θPSC_W<θ) to better handle socially sensitive situations. 4 Experimental Setup We investigate two main questions in our experiments: (1) whether the high-level social reasoning of the VLM with proximity-based triggering improves performance over recent RL policy baselines, and (2) under which conditions our proposed framework achieves its best performance. To address these questions, we design a controlled experimental setting that systematically evaluates our method against strong RL baselines across varying task configurations and environmental conditions. Benchmark To answer the first research question and evaluate the proposed method in realistic multi-agent settings, we conduct experiments on two photo-realistic simulation benchmarks: Social-HM3D and Social-MP3D [8]. Social-HM3D comprises 844 unique indoor scenes derived from the Habitat-Matterport 3D (HM3D) dataset [28].On the other hand, Social-MP3D is built upon the Matterport3D (MP3D) dataset [4] and contains 72 diverse indoor scenes spanning similar semantic categories such as residences, offices, and gyms. To further investigate the training and evaluation of the VLM component, we additionally evaluate on two social navigation visual question answering datasets: SNEI [23] and MUSON [17]. SNEI consists of socially grounded navigation reasoning samples in the form of image-question-answer pairs, constructed on top of the SCAN benchmark [12]. Metrics To comprehensively evaluate navigation performance in dynamic human environments, we report a set of complementary metrics that capture success, efficiency, and social compliance. We measure Success Rate (SR) as the proportion of episodes in which the agent successfully reaches the goal within a predefined threshold and time limit. To quantify social awareness, we compute PSC, defined as the fraction of timesteps during which the robot avoids violating human interpersonal comfort zones, with higher values indicating more socially compliant behavior. Navigation efficiency is assessed using Success weighted by Path Length (SPL) [43], which jointly evaluates success and trajectory optimality relative to the shortest path, where Si∈0,1S_i∈\0,1\ denotes success, lil_i is the shortest path length, and pip_i is the executed path length. Finally, safety is measured via Human Collision Rate (H-Coll), defined as the percentage of episodes in which the agent collides with at least one pedestrian, where lower values indicate safer navigation behavior. Additionally, to evaluate the VLM component, we employ SBERT and ROUGE-1 metrics to measure semantic and syntactic alignment between generated and reference responses across multiple social navigation reasoning tasks, including perception and scene understanding, prediction of future human and environmental states, and reasoning and explanation of perceived context, predicted outcomes, and recommended navigation actions in natural language form. Overall, higher values are desirable for SR, PSC, SPL, SBERT, ROUGE-1, while lower values are preferred for H-Coll. Baselines We compare the proposed approach against both classical and learning-based social robot navigation baselines. For classical approaches, we consider A* [9] and ORCA [39]. For learning-based methods, we evaluate against the official Habitat navigation baseline [26] and Falcon [8], which employ RL for embodied navigation tasks in photo-realistic simulation environments. Finally, we compare against NavThinker [10], a hybrid planning and RL framework that is closely related to our approach in its use of RL as the primary optimization mechanism, while differing substantially in its methodology for imagination and scene understanding. Configuration For the policy network, we adapt our method on top of the pretrained policy introduced in Falcon, where we consider the human collision radius is 0.3m, the robot radius is 0.25m, the PSC distance threshold is set to 1.0m. On the other hand, we initialize our VLM using Qwen3-VL 2B, a multimodal transformer with approximately 2.1B parameters. The model is fine-tuned using supervised fine-tuning (SFT) with category-specific prompts spanning five task dimensions: perception, prediction, reasoning, action, and explanation. To enable efficient adaptation, we employ LoRA within the PEFT framework, restricting updates to a small subset of parameters while preserving the majority of pretrained weights. Specifically, LoRA is applied with rank r=16r=16, scaling factor α=32α=32, and a dropout rate of 0.05, without bias terms. The adaptation targets the projection and feed-forward modules. This configuration results in approximately 17 million trainable parameters, corresponding to 0.81% of the full model. We use a per-device batch size of 1 with gradient accumulation over 8 steps, yielding an effective batch size of 8. A warmup ratio of 0.1 is applied to stabilize early-stage optimization. All experiments are conducted on NVIDIA Thor GPUs for both training and evaluation. Table 1: Social navigation results on Social-HM3D and Social-MP3D. Our method achieves the best SR and H-Coll on both benchmarks, with the largest gains on Social-MP3D, while remaining competitive in SPL and PSC. †Results taken from the original paper. Method Dataset and Metrics Social-HM3D Social-MP3D SR↑ SPL↑ PSC↑ H-Coll↓ SR↑ SPL↑ PSC↑ H-Coll↓ A*† 44.81 43.99 90.38 54.80 45.67 44.69 91.97 54.00 ORCA† 37.44 32.91 92.23 39.77 38.81 34.65 94.03 39.86 Habitat-official† 38.99 33.53 90.37 55.48 37.00 31.76 92.03 52.33 Falcon† 56.26 52.05 89.76 41.22 51.67 45.54 92.53 40.67 NavThinker† 59.46 55.00 89.91 39.09 47.33 41.71 93.68 37.67 Ours 62.07 54.54 92.32 34.48 70.35 59.41 92.96 20.82 5 Results Comparative Evaluation In the first level of evaluation, we compare the proposed method against baseline approaches on the Social-HM3D and Social-MP3D datasets. As reported in Table 1, our model achieves an overall accuracy of 62%62\%, demonstrating consistent improvements across both benchmarks. In particular, we observe an increase of approximately 20%20\% in success rate on Social-MP3D and 3%3\% on Social-HM3D relative to the baselines, while simultaneously reducing collision rates by 20%20\% and 5%5\%, respectively. These results highlight the effectiveness of the proposed approach in improving both task completion and safety. Notably, despite optimizing for higher success rates and lower collisions, the model maintains competitive performance in PSC and SPL metrics, indicating that gains in navigation efficiency and safety do not come at the expense of path quality or adherence to personal space constraints. Ablation Analysis on VLM In this ablation study, we investigate the trade-off between inference latency and task performance for the VLM, with the goal of identifying a compact model that maintains strong performance. As shown in Figure 3(a), we evaluate multiple model variants and observe that our trained model achieves performance comparable to the highest-performing baseline, with only a 1%1\% difference, while reducing the number of parameters by approximately 33%33\%. Furthermore, Figure 3(b) illustrates the performance-to-parameter ratio, where our approach consistently achieves a significantly higher efficiency compared to competing methods. (a) Accuracy comparison across VLM variants (b) Accuracy vs. model size trade-off Figure 3: VLM selection ablation. Our fine-tuned (ft) and zero-shot (zs) 2B models achieve accuracy within 1% of the best baseline while using 33% fewer parameters. Table 2: Effect of training dataset on VLM performance. Training on SNEI alone yields the best action accuracy, while MUSON improves semantic and syntactic output quality. As action prediction is the primary requirement for our navigation task, SNEI is selected as the training dataset. Training Dataset Act. Acc ↑ SBERT ↑ ROUGE-1 ↑ None (Zero-shot) 0.825 0.507 0.317 SNEI 0.850 0.580 0.370 MUSON 0.762 0.693 0.488 MUSON+SNEI 0.762 0.676 0.472 On the other hand, as shown in Table 2, we investigate the impact of different training dataset combinations, specifically SNEI and MUSON, on the performance of our model. Our results indicate that when the Qwen3-VL-2B model is trained on SNEI, it achieves the highest action prediction accuracy on the MUSON evaluation set. In contrast, training on MUSON leads to better semantic and syntactic alignment in the generated reasoning outputs. However, since explicit reasoning generation is not required for our target social navigation task, we prioritize action prediction performance and therefore consider SNEI to be a more suitable training dataset for this setting. Ablation Analysis on Augmentation of Visual Input The visual observations provided by both simulation environments and real-world robotic platforms typically consist of RGB images and depth information. As shown in Table 3, we investigate the impact of different input modalities and augmentation strategies on the performance of the proposed framework. In particular, we evaluate standard RGB inputs, colorized depth representations, and their combinations. For the colorized depth setting, depth maps are transformed into colored visual representations accompanied by prompt descriptions, enabling the VLM to better interpret spatial structure and distance information. Furthermore, we analyze the effect of additional visual guidance cues, such as arrows and pointer annotations, by comparing performance with and without these augmentations. Our findings suggest that colorized mapping, together with colored arrow indications for navigation, improves the model’s understanding of the environment, while directional annotations further enhance the reasoning capability of the VLM by providing clearer spatial and navigational context. Table 3: Ablation of input modality and visual augmentation. Depth JET colormap yields the best SR, H-Coll, and SPL with the lowest latency, outperforming RGB-based and mixed inputs. Adding directional arrows provides no further gain for depth inputs but degrades performance in RGB settings, confirming colorized depth as the optimal input representation. Type Example Setting SR↑ H-Coll↓ SPL↑ ms/call Depth Depth JET (no arrow) 62.07 34.48 54.54 2,425 Depth JET+ green arrow 62.07 34.48 54.54 2,425 RGB RGB only (CLAHE) 59.77 35.63 52.93 4,580 RGB + white arrow 54.02 40.80 48.09 4,260 Mix RGB + depth side-by-side 60.34 35.63 53.41 5,427 RGB + depth + arrow 58.62 38.51 52.93 5,318 Ablation Analysis on Proximity Threshold and Evolution of Environment We further study the impact of the proximity decision threshold and environment update strategy on navigation performance, as shown in Table 4 and Table 5. For the proximity threshold, we observe that r=0.80r=0.80 m yields the best overall performance, achieving the highest success rate and SPL while maintaining the lowest collision rate among all settings. This suggests that an intermediate decision distance provides an effective balance between early intervention and navigation efficiency. Table 4: Ablation of PSC switch proximity threshold r. A threshold of 0.80 m achieves the best scores, confirming that intermediate intervention distance optimally trades off safety and navigation efficiency. rr (m) SR↑ SPL↑ PSC↑ H-Coll↓ #Calls 0.70 59.20 53.85 91.62 36.78 390 0.75 55.75 49.93 91.75 40.23 500 0.80 62.07 54.54 92.32 34.48 1090 0.85 58.05 51.90 92.20 37.93 2080 0.90 55.75 47.66 92.72 42.53 3250 In addition, we analyze different environment evolution strategies during VLM reasoning. As shown in Table 5, a fully frozen environment yields the best performance (62.07% success rate), although this setting is not realistic for real-world deployment. When reducing the frequency of environment updates to more realistic settings, performance slightly degrades from 62% to 56% success rate, indicating a trade-off between simulation fidelity and decision stability during VLM inference. Table 5: Ablation of environment evolution strategy during VLM inference. Freezing the environment yields the best performance but is unrealistic for deployment; reducing update frequency to more realistic settings incurs a modest 5–6% SR drop, reflecting a trade-off between decision stability and simulation fidelity. Strategy SR↑ SPL↑ PSC↑ H-Coll↓ #Calls Freeze 62.07 54.54 92.32 34.48 109 Pause 55.17 49.46 91.61 42.53 56 Realistic 56.90 52.31 91.62 40.23 46 Figure 4: Real-World Deployment of HUMA on the Mirokai robot. The chronological sequence showcases the robot relying on the reactive RL policy to navigate around the chair and reasoning about the human as it gets close to him, before executing a socially compliant trajectory. 6 Deployment To demonstrate HUMA’s applicability in a real-world environment, as shown in Figure 4, we integrated it into the Mirokai humanoid robotics platform developed by Enchanted Tools. The deployment architecture bridges real-time data with the robot’s base via a ROS2 Humble ecosystem. Perception Setup Human positions and environment dynamics are captured using a combination of onboard depth camera and an external OptiTrack motion capture system. The motion capture system provides high-precision tracking of agents and goals in the shared space, transformed into the robot’s base frame, while the camera data feeds perceptual data to the policy. Software Architecture The pipeline, shown in Figure 5, is entirely implemented in ROS2, with dedicated nodes subscribing to the motion capture and camera topics before forwarding the data as input to HUMA. The positions provided by the motion capture system are converted into relative distances between the agents, which are then used as navigation inputs for the policy. HUMA is encapsulated within a dedicated ROS2 node that processes both the computed distances and the video stream from the onboard camera. The model outputs high-level navigation actions such as left, right, forward, backward, stop, or pause. These discrete actions are then converted within the ROS2 interface node into standard linear and angular velocity commands, which are sent to the robot controller for execution. By continuously repeating this process in a closed-loop architecture, the robot is able to navigate toward the target while avoiding surrounding obstacles and humans using HUMA as a ROS2-based social navigation module. Figure 5: Real-world deployment architecture of HUMA. Agent positions from the OptiTrack motion capture system and perceptual data from the Mirokaï onboard camera are fed as relative distances and raw observations into a ROS2-based social navigation node running HUMA. The framework outputs standard linear and angular velocity commands sent directly to the Mirokaï robot. Limitations Figure 6: VLM reasoning in a proximity-triggered social navigation scenario. (a) Raw RGB, (b) JET colormap depth, and (c) fused VLM input with Canny edge overlay. With PSC = 0.800 and a human at 1.58 m, the PSC switch activates the VLM, which generates a forward action sequence while describing the social context in natural language. Detailed example provided in Figure 7. Despite promising results, as an example shown in Figure 6, HUMA has several limitations that motivate future research. As explained in Section 5, the current freeze timing assumption, where the simulator pauses during VLM inference, does not hold in real deployment, costing 5.2%5.2\% SR in our realistic timing ablation, and addressing this through better hardware or more efficient models remains an open challenge. Additionally, HUMA reacts to observed PSC violations rather than predicting future ones; integrating Falcon’s trajectory prediction module could enable truly proactive intervention. Moreover, as explained in Section 5, the fixed trigger threshold r=0.8r=0.8 treats all scenes equally, motivating future adaptive mechanisms that adjust based on scene density or pedestrian dynamics. A recurring failure mode is that the VLM correctly handles the immediate social situation but fills remaining plan steps with ”forward,” suggesting that shorter plans with higher-frequency VLM calls would better support continuous scene reassessment. Moreover, as explained in Section 5, the domain gap between the VLM’s real-world RGB training data and Habitat’s synthetic images also limits performance, particularly in degraded or near-black frames where the model may hallucinate. To further illustrate the behavior of HUMA during inference, Figure 7 presents two consecutive VLM invocations triggered by PSC threshold violations. In each case, the raw RGB observation, the JET colormap depth representation, and the GPS-encoded navigation context are passed to the VLM reasoning module. The VLM interprets the social scene and produces an 8-action plan that is subsequently executed by the RL controller. This example highlights two key properties of HUMA: first, the PSC switch correctly identifies socially sensitive situations and activates the VLM only when necessary; second, the fused visual input provides sufficient spatial and semantic context for the VLM to generate socially compliant action sequences. Notably, consecutive invocations demonstrate that the switching mechanism remains responsive across successive timesteps, ensuring continuous social compliance without falling back to the RL policy prematurely. Figure 7: An overview Example. Two consecutive VLM invocations triggered by PSC threshold violations. At each trigger, the robot passes raw RGB, depth JET colormap, and GPS-encoded navigation context to the VLM. The VLM reasons about the social scene and outputs an 8-action plan executed by the robot. RGB and depth images are rendered from the Habitat simulator during evaluation. Conclusion In this paper, we introduced HUMA, a hybrid human-aware navigation framework that dynamically balances the computational efficiency of an RL policy with the deep semantic understanding of a post-trained VLM, activated conditionally via a PSC switch. Evaluated on Social-HM3D and Social-MP3D, HUMA significantly outperforms existing baselines in success rate and human collision rate while maintaining competitive SPL and PSC scores. Extensive ablation studies validate each component of the architecture, and real-world deployment on the Mirokaï robot from Enchanted Tools further demonstrates its practical viability for socially compliant navigation in human-populated environments. References [1] M. Aghzal, X. Yue, E. Plaku, and Z. Yao (2025) Evaluating vision-language models as evaluators in path planning. External Links: 2411.18711, Link Cited by: §1. [2] S. B. Banisetty, S. Forer, L. Yliniemi, M. Nicolescu, and D. Feil-Seifer (2021) Socially aware navigation: a non-linear multi-objective optimization approach. ACM Transactions on Interactive Intelligent Systems (TiiS) 11 (2), p. 1–26. Cited by: §2. [3] B. Brito, M. Everett, J. P. How, and J. Alonso-Mora (2021) Where to go next: learning a subgoal recommendation policy for navigation in dynamic environments. IEEE Robotics and Automation Letters 6 (3), p. 4616–4623. Cited by: §2, §2. [4] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang (2017) Matterport3d: learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158. Cited by: §4. [5] K. Charalampous, I. Kostavelis, and A. Gasteratos (2016) Robot navigation in large-scale social maps: an action recognition approach. Expert Systems with Applications 66, p. 261–273. Cited by: §2. [6] J. Chen, Y. Li, P. Jiang, J. Du, Z. Chen, C. Tie, J. Deng, and L. Shao (2025) LISN: language-instructed social navigation with vlm-based controller modulating. arXiv preprint arXiv:2512.09920. Cited by: §2. [7] Z. Fang, A. Xiao, D. Hsu, and G. H. Lee (2026) From obstacles to etiquette: robot social navigation with vlm-informed path selection. IEEE Robotics and Automation Letters. Cited by: §2. [8] Z. Gong, T. Hu, R. Qiu, and J. Liang (2025) From cognition to precognition: a future-aware framework for social navigation. External Links: 2409.13244, Link Cited by: §1, §2, §3.2, §3.2, §4, §4. [9] P. E. Hart, N. J. Nilsson, and B. Raphael (1968) A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics 4 (2), p. 100–107. External Links: Document Cited by: §4. [10] T. Hu, Z. Gong, L. Kong, X. Mei, Y. Ding, Q. Zeng, A. Liang, R. Li, Y. Zhong, and J. Liang (2026) NavThinker: action-conditioned world models for coupled prediction and planning in social navigation. External Links: 2603.15359, Link Cited by: §4. [11] Y. Huang, J. Sansom, Z. Ma, F. Gervits, and J. Chai (2024) Drivlme: enhancing llm-based autonomous driving agents with embodied and social experiences. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 3153–3160. Cited by: §1, §2. [12] H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone (2022) Socially compliant navigation dataset (scand): a large-scale dataset of demonstrations for social navigation. IEEE Robotics and Automation Letters 7 (4), p. 11807–11814. Cited by: §2, §4. [13] M. Kollmitz, K. Hsiao, J. Gaa, and W. Burgard (2015-11-10) Time dependent planning on a layered social cost map for human-aware robot navigation. In 2015 European Conference on Mobile Robots, ECMR 2015 - Proceedings, United States (English). Note: European Conference on Mobile Robots, ECMR 2015 ; Conference date: 02-09-2015 Through 04-09-2015 External Links: Document Cited by: §2. [14] M. Korkmaz (2021) Human-aware dynamic path planning. In 2021 International Conference on INnovations in Intelligent SysTems and Applications (INISTA), p. 1–5. Cited by: §2. [15] T. Kruse, A. K. Pandey, R. Alami, and A. Kirsch (2013) Human-aware robot navigation: a survey. Robotics and Autonomous Systems 61 (12), p. 1726–1743. External Links: ISSN 0921-8890, Document, Link Cited by: §1, §2. [16] L. Liu, D. Dugas, G. Cesari, R. Siegwart, and R. Dubé (2020) Robot navigation in crowded environments using deep reinforcement learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 5671–5677. Cited by: §2. [17] Z. Liu, X. Zhang, Z. Wang, T. Kawabata, X. Xiao, and L. Xiao (2025) MUSON: a reasoning-oriented multimodal dataset for socially compliant navigation in urban environments. arXiv preprint arXiv:2512.22867. Cited by: §4. [18] M. Luber, L. Spinello, J. Silva, and K. O. Arras (2012) Socially-aware robot navigation: a learning approach. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 902–907. Cited by: §2. [19] Y. Luo, P. Cai, A. Bera, D. Hsu, W. S. Lee, and D. Manocha (2018) Porca: modeling and planning for autonomous driving among many pedestrians. IEEE Robotics and Automation Letters 3 (4), p. 3418–3425. Cited by: §2. [20] S. Narasimhan, A. H. Tan, D. Choi, and G. Nejat (2025) Olivia-nav: an online lifelong vision language approach for mobile robot social navigation. In 2025 IEEE international conference on robotics and automation (ICRA), p. 9130–9137. Cited by: §2. [21] L. C. Nogueira, P. Thomaz Aquino, and F. Tonidandel (2025) Legibility on social robot navigation: a survey. In 2025 Brazilian Conference on Robotics (CROS), Vol. 1, p. 1–6. External Links: Document Cited by: §1. [22] H. Pan, S. Huang, J. Yang, J. Mi, K. Li, X. You, P. Liang, J. Yang, Y. Liu, J. Zhang, M. Wang, J. Yang, X. Zhang, L. Zhao, M. Chen, J. Zhou, and X. Wei (2026-04) Robot navigation via foundation language models: a review. ACM Comput. Surv. 58 (11). External Links: ISSN 0360-0300, Link, Document Cited by: §1. [23] A. Payandeh, D. Song, M. Nazeri, J. Liang, P. Mukherjee, A. H. Raj, Y. Kong, D. Manocha, and X. Xiao (2024) Social-llava: enhancing robot navigation through human-language reasoning in social spaces. arXiv preprint arXiv:2501.09024. Cited by: §4. [24] R. Peddi, C. Di Franco, S. Gao, and N. Bezzo (2020) A data-driven framework for proactive intention-aware motion planning of a robot in a human environment. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 5738–5744. Cited by: §2. [25] N. Pérez-Higueras, F. Caballero, and L. Merino (2018) Learning human-aware path planning with fully convolutional networks. In 2018 IEEE international conference on robotics and automation (ICRA), p. 5897–5902. Cited by: §2, §2. [26] X. Puig, E. Undersander, A. Szot, M. Dallaire Cote, T. Yang, R. Partsey, R. Desai, A. Clegg, M. Hlavac, S. Y. Min, et al. (2024) Habitat 3.0: a co-habitat for humans, avatars, and robots. In International Conference on Learning Representations, Vol. 2024, p. 15306–15336. Cited by: §4. [27] A. R, C. C, R. R, and P. D (2025) Social robot navigation: a review and benchmarking of learning-based methods. Front. Robot. AI 12:1658643. External Links: Document Cited by: §1. [28] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, et al. (2021) Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08238. Cited by: §4. [29] O. A. I. Ramírez, H. Khambhaita, R. Chatila, M. Chetouani, and R. Alami (2016) Robots learning how and where to approach people. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Vol. , p. 347–353. External Links: Document Cited by: §1, §2. [30] A. J. Sathyamoorthy, K. Weerakoon, M. Elnoor, A. Zore, B. Ichter, F. Xia, J. Tan, W. Yu, and D. Manocha (2024) Convoi: context-aware navigation using vision language models in outdoor and indoor environments. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 13837–13844. Cited by: §2. [31] P. T. Singamaneni, P. Bachiller-Burgos, L. J. Manso, A. Garrell, A. Sanfeliu, A. Spalanzani, and R. Alami (2024) A survey on socially aware robot navigation: taxonomy and future challenges. The International Journal of Robotics Research 43 (10), p. 1533–1572. Cited by: §1, §2. [32] P. T. Singamaneni, A. Favier, and R. Alami (2021) Human-aware navigation planner for diverse human-robot interaction contexts. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 5817–5824. Cited by: §2. [33] P. T. Singamaneni, A. Favier, and R. Alami (2022) Watch out! there may be a human. addressing invisible humans in social navigation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 11344–11351. Cited by: §2. [34] D. Song, J. Liang, A. Payandeh, A. H. Raj, X. Xiao, and D. Manocha (2024) Vlm-social-nav: socially aware robot navigation through scoring using vision-language models. IEEE Robotics and Automation Letters 10 (1), p. 508–515. Cited by: §2. [35] X. Sun, Y. Zhang, X. Tang, A. S. Bedi, and A. Bera (2024) Trustnavgpt: modeling uncertainty to improve trustworthiness of audio-guided llm-based robot navigation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 8794–8801. Cited by: §2. [36] Z. Talebpour, D. Viswanathan, R. Ventura, G. Englebienne, and A. Martinoli (2016) Incorporating perception uncertainty in human-aware navigation: a comparative study. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), p. 570–577. Cited by: §2. [37] R. Triebel, K. Arras, R. Alami, L. Beyer, S. Breuers, R. Chatila, M. Chetouani, D. Cremers, V. Evers, M. Fiore, H. Hung, O. A. I. Ramírez, M. Joosse, H. Khambhaita, T. Kucner, B. Leibe, A. J. Lilienthal, T. Linder, M. Lohse, M. Magnusson, B. Okal, L. Palmieri, U. Rafi, M. van Rooij, and L. Zhang (2016) SPENCER: a socially aware service robot for passenger guidance and help in busy airports. In Field and Service Robotics: Results of the 10th International Conference, D. S. Wettergreen and T. D. Barfoot (Eds.), p. 607–622. External Links: ISBN 978-3-319-27702-8, Document, Link Cited by: §1. [38] R. Valiente, B. Toghi, R. Pedarsani, and Y. P. Fallah (2022) Robustness and adaptability of reinforcement learning-based cooperative autonomous driving in mixed-autonomy traffic. IEEE Open Journal of Intelligent Transportation Systems 3, p. 397–410. Cited by: §2. [39] J. van den Berg, S. J. Guy, M. Lin, and D. Manocha (2011) Reciprocal n-body collision avoidance. In Robotics Research, C. Pradalier, R. Siegwart, and G. Hirzinger (Eds.), Berlin, Heidelberg, p. 3–19. External Links: ISBN 978-3-642-19457-3 Cited by: §4. [40] D. Vasquez, B. Okal, and K. O. Arras (2014) Inverse reinforcement learning algorithms and features for robot navigation in crowds: an experimental comparison. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 1341–1346. Cited by: §2. [41] A. Vega-Magro, L. Manso, P. Bustos, P. Núñez, and D. G. Macharet (2017) Socially acceptable robot navigation over groups of people. In 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), p. 1182–1187. Cited by: §2. [42] L. Zhang, X. Hao, X. Bu, Y. Tang, H. Li, J. Lu, X. Wei, J. Ma, Y. Liu, J. Zhang, et al. (2026) Walk with me: long-horizon social navigation for human-centric outdoor assistance. arXiv preprint arXiv:2604.26839. Cited by: §2. [43] L. Zhang, E. Xiao, X. Hao, H. Fu, Z. Gong, L. Chen, X. Liang, R. Xu, H. Ye, and W. Ding (2025) SocialNav-map: dynamic mapping with human trajectory prediction for zero-shot social navigation. arXiv preprint arXiv:2511.12232. Cited by: §4.