Paper deep dive
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Yizhan Li, Jianxin You, Mengyang Xiong, Yinhuan Chen, Zicheng Zhao, Dekun Wu, Dongqing Zhang, Bang Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/11/2026, 4:04:43 AM
Summary
The paper introduces ReactHuman, a physics-grounded benchmark for evaluating human-like reactive decision-making in embodied multimodal large language models (MLLMs). Unlike passive video-based reasoning benchmarks, ReactHuman requires MLLMs to act as the 'brain' of a simulated humanoid, generating immediate safety-critical actions (Catch, Dodge, No-Action) in response to sudden household hazards. The benchmark features 17 event families, over 1,000 reproducible scenes generated via a hybrid LLM-procedural pipeline, and a five-metric suite assessing reasonability, safety, and physical grounding. Evaluations of seven representative MLLMs reveal that reactive safety is not solved, with models frequently mishandling hazards, relying on fixed dispositions, trusting appearance over motion, and failing to improve with model scale.
Entities (17)
Relation Signals (15)
Qwen3-VL-235B â evaluatedin â ReactHuman
confidence 95% · We evaluate seven MLLMs... five frontier models (...Qwen3-VL-235B)
GPT-5.5 â evaluatedin â ReactHuman
confidence 95% · We evaluate seven MLLMs... five frontier models (...GPT-5.5...)
Claude Opus 4.8 â evaluatedin â ReactHuman
confidence 95% · We evaluate seven MLLMs... five frontier models (Claude Opus 4.8...)
ReactHuman â evaluates â Multimodal Large Language Models
confidence 95% · We introduce ReactHuman... in which the evaluated MLLM acts as the brain of a simulated humanoid
ReactHuman â includesmetric â Action-Intent Alignment
confidence 95% · 4. ActionâIntent Alignment (AIA)... whether the stated intent matches the goal
ReactHuman â includesmetric â Hand-Distance Evolution
confidence 95% · 5. Hand-Distance Evolution (HDE)... shows whether the hand actually moves toward the target
ReactHuman â includesmetric â Semantic Action Accuracy
confidence 95% · We report five metrics per decision... 1. Semantic Action Accuracy (SAA)
ReactHuman â includesmetric â Safety Validity
confidence 95% · 2. Safety Validity. A binary flag for safety-critical mistakes
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reacting to sudden physical hazards (catching a slipping plate, dodging a falling knife) is both a meaningful test of embodied intelligence and a hard requirement for deploying multimodal large language models (MLLMs) as the decision coreof household robots. Existing evaluations, however, probe intuitive physics passively through question answering over videos, or target deliberate, long-horizon tasks such as navigation and rearrangement; none measure whether a model can turn physical understanding into immediate, safety-critical action. We introduce ReactHuman, the first physics-grounded benchmark for human-like reactive decision-making, in which the evaluated MLLM acts as the brain of a simulated humanoid facing sudden household hazards; it spans 17 event families and over 1,000 bit-for-bit reproducible scenes with exact, annotation-free ground truth derived from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics (a foam anvil, a steel apple). We further design a five-metric suite that scores each reaction along three axes: reasonable, safe, and physically grounded. We physically execute every committed plan so that decisions have observable consequences. With this harness we evaluate seven representative MLLMs. Results show that reactive safety is far from solved: models mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale even when the chosen action is correct; none of these failures shrink with model scale. ReactHuman thus offers both a fine-grained diagnosis and a scalable training signal toward physically grounded, safety-aware embodied agents. The benchmark can be found here: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.10895v1
- Canonical: https://arxiv.org/abs/2609.10895v1
Trouble viewing inline? Open PDF directly â
Full Text
38,037 characters extracted from source content.
Expand or collapse full text
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs Yizhan Li Jianxin You Mengyang Xiong Yinhuan Chen Zicheng Zhao Dekun Wu Dongqing Zhang Bang Liu Abstract Reacting to sudden physical hazards (catching a slipping plate, dodging a falling knife) is both a meaningful test of embodied intelligence and a hard requirement for deploying multimodal large language models (MLLMs) as the decision core of household robots. Existing evaluations, however, probe intuitive physics passively through question answering over videos, or target deliberate, long-horizon tasks such as navigation and rearrangement; none measure whether a model can turn physical understanding into immediate, safety-critical action. We introduce ReactHuman, the first physics-grounded benchmark for human-like reactive decision-making, in which the evaluated MLLM acts as the brain of a simulated humanoid facing sudden household hazards; it spans 17 event families and over 1,000 bit-for-bit reproducible scenes with exact, annotation-free ground truth derived from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics (a foam anvil, a steel apple). We further design a five-metric suite that scores each reaction along three axes: reasonable, safe, and physically grounded. We physically execute every committed plan so that decisions have observable consequences. With this harness we evaluate seven representative MLLMs. Results show that reactive safety is far from solved: models mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale even when the chosen action is correct; none of these failures shrink with model scale. ReactHuman thus offers both a fine-grained diagnosis and a scalable training signal toward physically grounded, safety-aware embodied agents. The benchmark can be found here: https://huggingface.co/datasets/Alan123/reacthuman-benchmark-scaled 1UniversitĂ© de MontrĂ©al 2Mila â Quebec AI Institute 3McGill University 4McMaster University 5Meta Platforms yizhan.li, jianxin.you, dekun.wu, bang.liu@umontreal.ca, mengyang.xiong, yinhuan.chen@mail.mcgill.ca, zhaoz149@mcmaster.ca, zhangdq20@gmail.com 1 Introduction Humans react to sudden physical hazards in a fraction of a second: we catch a slipping plate, dodge a falling knife, and brace against a toppling wardrobe. These reactions fuse semantic judgment (what is this object, and is it safe to touch?) with intuitive physics (where and when will it arrive?). As multimodal large language models (MLLMs) are placed at the cognitive core of household robots and embodied assistants (Ahn et al. 2022; Driess et al. 2023), the same competence becomes a deployment requirement: we want an agent that reacts like a competent person, whose reaction is at once safe (it avoids harm), reasonable (it is the action a sensible human would choose and defend), and physically grounded (the hand actually meets the object in time). An agent that elects to catch a falling chefâs knife, or that decides correctly yet grasps half a meter wide, has failed in a way no captioning or VQA benchmark can reveal. Existing evaluations do not measure this competence. Physical-reasoning benchmarks probe intuitive physics passively, through plausibility judgments or question answering over videos (Bear et al. 2021; Yi et al. 2020; Zheng et al. 2024; Chow et al. 2025), while embodied-AI benchmarks emphasize deliberate, long-horizon tasks such as navigation and rearrangement (Szot et al. 2021; Li et al. 2022). Neither reveals whether a model can convert physical understanding into an immediate, safety-critical action nor diagnose why it fails when it does: a model may misread the hazard, choose an unsafe action despite understanding it, or choose correctly yet mispredict the interception point by a meter. We introduce ReactHuman, a benchmark that evaluates whether MLLMs react to sudden hazards like a competent human by making the MLLM the decision-making brain of a simulated humanoid that must actually carry the reaction out. We check if they act safely, reasonably, and with physically grounded movement in physically simulated indoor scenes. ReactHuman adopts a freeze-and-predict protocol: a full-physics simulation of a sudden household event runs to a critical decision moment and freezes; the model observes the unfolding event and must commit to a structured reaction plan: a walking command plus a hand-keyframe trajectory, from which the executed action (Catch, Dodge, or No-Action) is derived. Because inference happens while simulated time is paused, models are compared on decision quality alone, uncontaminated by API latency. Because every scene is generated deterministically from a seed and labeled from simulator state, all scores are computed against exact physical ground truth. And because the committed plan is then executed by a simulated humanoid driven by a pre-trained RL walking policy (Rudin et al. 2022; Unitree Robotics 2024) in the same physics engine, every decision has an observable physical outcome (the catch connects or the toppling shelf strikes the agent) rather than remaining a multiple-choice answer, which is what separates ReactHuman from VQA-style physical-reasoning evaluation. ReactHuman is built for diagnosis, not a single leaderboard number. Its 17 event families (Figure 1) are chosen so that different families stress different links in the perceptionâreasoningâaction chain: tabletop drops make the trajectory trivial and isolate semantic choice; pendulums and bouncing balls make the semantic choice easy and isolate interception; chain reactions require causal propagation before any kinematics; adversarial objects (a foam anvil, a steel apple, a lead-core tennis ball) decouple appearance from physics, so that the correct action is recoverable only from observed motion. Complementing the taxonomy, a five-metric suite instruments the three axes of a human-like reaction: semantic action accuracy and actionâintent alignment (is the reaction reasonable?), safety validity (is it safe?), and physical endpoint distance and hand-distance evolution (is it physically grounded?). Our contributions are: âą Benchmark. A physics-grounded, freeze-and-predict evaluation of human-like reactive decisions along three axes: safety, reasonability, and physical grounding. The evaluation covers 17 sudden-event families, over 1000 reproducible scenes, three synchronized camera views, and exact simulator-derived ground truth (action, 3D impact point, time-to-floor), including adversarial appearanceâphysics probes, with every committed decision executed by a simulated humanoid in the same physics engine. âą Generation pipeline. A hybrid LLM-planned, seed-deterministic pipeline in which a frozen LLM contributes semantic diversity (objects, rooms, event routing from natural-language descriptions) while a procedural randomizer owns every physical parameter. This makes scenes bit-for-bit reproducible and the benchmark extensible past 10410^4 scenes without human annotation. âą Metric suite. Five complementary metrics organized along the three axes: reasonability, safety, and physical grounding. This separates an unreasonable choice from an unsafe one from a kinematically ungrounded one, with an explicit safety-rule system and structured-output protocol. âą Empirical study. An evaluation of seven MLLMs across 306 scenes and all 17 families, revealing that safety failures concentrate where evasion is required, that action choice follows fixed per-model dispositions rather than the scene, that neither accuracy nor safety improves with model scale, and that models never revise appearance-based physics judgments from observed motion. 2 Related Work Physical reasoning benchmarks. Physion and Physion++ (Bear et al. 2021; Tung et al. 2023) test forward prediction of physical outcomes; CLEVRER (Yi et al. 2020) and IntPhys (Riochet et al. 2018) probe causal and violation-of-expectation reasoning; ContPhy (Zheng et al. 2024) extends to deformables and fluids. For modern MLLMs, PhysBench (Chow et al. 2025) is the closest neighbor: it measures physical-world understanding via multiple-choice VQA over a model population that largely overlaps ours. Physics-IQ (Motamed et al. 2025) scores video-generation models for physical plausibility. ReactHuman differs on two axes simultaneously: the modelâs output is an action with safety consequences rather than an answer, and ground truth is generated automatically and exactly by simulation, so the quantitative track scales. Embodied-AI and humanoid benchmarks. Habitat (Savva et al. 2019; Szot et al. 2021), AI2-THOR (Kolve et al. 2017), ThreeDWorld (Gan et al. 2021), ManiSkill (Mu et al. 2021), ManiSkill2 (Gu et al. 2023), BEHAVIOR-1K (Li et al. 2022), and OpenEQA (Majumdar et al. 2024) evaluate navigation, manipulation, rearrangement, or situated QA on deliberate time scales; HumanoidBench (Sferrazza et al. 2024) targets whole-body control. None evaluate immediate reactive safety. ReactHuman isolates the decision layer while retaining embodiment: following a brainâspineâbody decoupling, the evaluated MLLM acts as the brain of a simulated humanoid whose balance and joint torques are delegated to a pre-trained whole-body controller (Rudin et al. 2022; Unitree Robotics 2024), so scores are attributable to reasoning rather than policy quality, yet every decision is still physically executed in scene. MLLMs as embodied decision modules vs. VLAs. One line of work deploys frozen MLLMs as zero-shot planners over discrete skills (Ahn et al. 2022; Driess et al. 2023); another fine-tunes visionâlanguage backbones end-to-end into visionâlanguageâaction (VLA) policies emitting low-level control (Brohan et al. 2023; Kim et al. 2024; Black et al. 2024). ReactHumanâs harness spans the two: coupling a frozen MLLM to a pre-trained whole-body controller and a simulated humanoid turns any off-the-shelf MLLM into a zero-shot vision-to-action agent, without the action fine-tuning that defines VLAs. We evaluate general-purpose MLLMs because they are what is deployed today as the reasoning core of embodied systems; native VLA policies can be dropped into the same scenes, ground truth, and metrics unchangedâa comparison we leave to future work. Procedural and LLM-assisted scene generation. Kubric (Greff et al. 2022) established scalable simulatorârenderer tooling; Objaverse (Deitke et al. 2023) supplies large-scale assets; recent systems use LLMs to compose environments and tasks (Wang et al. 2024; Yang et al. 2024). ReactHuman combines the two: an LLM performs semantic planning from natural language while a deterministic, seeded randomizer retains exclusive control of physical parameters, preserving the reproducibility and label exactness that fully LLM-driven generation lacks. 3 The ReactHuman Benchmark 3.1 Freeze-and-Predict Protocol Each episode places a virtual observer. A digital human standing in an indoor room is required to react to a sudden physical event at t0t_0: an object begins to fall, slide, roll, topple, swing, or be struck toward the observer. The model receives the observation window: frames from t0t_0 to t0+Ît_0+ (default Îâ0.6 \!â\!0.6 s, before the outcome is visually resolved) from up to three synchronized viewpoints (standing observer, close-up at the event origin, overhead). Simulated time then freezes and the model must output a structured reaction plan output=intent,confidence,walking_cmd,keyframes,output=\intent,\ confidence,\ walking\_cmd,\ keyframes\, (1) where intent is free-form reasoning, confidence â[0,1]â[0,1], walking_cmd is a base-velocity command, and keyframes 1,âŠ,n\h_1,âŠ,h_n\, iââ3h_i ^3, is the predicted hand trajectory. Freezing time makes the decision problem identical for every model and removes inference latency as a confound; scoring is fully deterministic. Humanoid execution. The reaction plan is not merely graded on paper: simulation then resumes and the plan is executed by a simulated humanoid (a Unitree G1) placed in the same Genesis scene, following a brainâspineâbody decoupling: the evaluated MLLM is the brain; a pre-trained whole-body controller (the spine) converts the walking command and hand keyframes into balance and joint torques; the humanoid body interacts with the falling object under full rigid-body physics. Each episode therefore yields an execution video in which the chosen reaction has observable physical consequences. The catch connects or misses, the dodge clears the impact zone or fails to. This distinguishes ReactHuman from passive VQA: the unit of evaluation is an embodied interaction, not an answer. For metric scoring we use the simulator-derived ground truth (§4), which keeps scores deterministic and independent of controller quality; execution provides outcome-level verification. Representative rollouts are shown in Figure 3. Action space. The model does not pick from a menu. It outputs a motor plan, a base-velocity walking command plus hand keyframes, and a deterministic rule classifies the plan into one of three primitives: Execute_Catch (walk toward the object and reach for it), Trigger_Dodge (move clear of its path), or No_Action (stay put, hands at rest). We score what the body would actually do, not what the model claims, so a model cannot pass by simply saying the right word. Ground-truth labels follow object properties and event kinematics, not appearance: light, graspable, benign objects are labeled Catch; sharp, hot, shattering, heavy, or fast objects are labeled Dodge. No_Action is correct only in the few scenes where the event cannot reach the observer, and choosing it under an active hazard counts as a safety violation. On the evaluated set the labels split 141/156/9 across Catch/Dodge/No_Action, so majority-class guessing (always Dodge) attains 51.0%. 3.2 A Taxonomy of 17 Sudden-Event Families object_drop tips off a table edge shelf_slide slides off a high shelf hanging_fall wall fixture detaches ceiling_drop falls from the ceiling sliding_object slides down a ramp rolling_ball rolls off and across surface_cascade hops table, stool, floor furniture_tip tall furniture tips over stack_collapse stacked items collapse ladder_slip leaning ladder slips door_swing door swings violently pendulum_swing suspended mass swings in thrown_object thrown at the observer bouncing_object bounces toward observer stair_tumble tumbles down the stairs chain_reaction strike launches a second object multi_object momentum runs down a row Figure 1: The 17 sudden-event families of ReactHuman. The families span six motion archetypes (fall, slide & roll, topple, articulated, ballistic, and causal chain); each is evaluated on 18 scenes in §3, and every thumbnail is a real benchmark scene. Figure 1 summarizes the families. Beyond covering distinct dynamics (free fall, friction-driven sliding, inverted-pendulum toppling, hinge-constrained rotation, ballistic flight with restitution, momentum transfer), families differ in decision structure. In object_drop the trajectory is trivial and the decision is dominated by object semantics. In pendulum_swing and bouncing_object semantics are easy but the interception point is time-varying or multimodal. In chain_reaction and multi_object the threatening object is initially stationary: the model must propagate causality (A will strike B; momentum will travel down the row) before any kinematic estimate is possible. This factorization lets ReactHuman localize failures to perception, causal reasoning, semantic judgment, or spatial calculation. 3.3 Adversarial AppearanceâPhysics Probes The 82-object asset library contains 14 adversarial variants whose visual identity contradicts their physical parameters: a foam anvil (looks cast-iron; light and safe to catch), a steel apple (looks like fruit; arrives with bruising momentum), a lead-core tennis ball, a foam mirror, a hollow prop door, a foam ceiling panel, a lead sandbag, a solid-steel can. For these objects the appearance prior implies the wrong action; the correct one is recoverable only from observed dynamics in the early frames. Section 3 probes these objects directly: across 280 decisions no model ever questions an objectâs material or weight, and disguised objects receive exactly the treatment of their genuine look-alikes. 3.4 Scene Generation and Ground Truth Figure 2: The ReactHuman pipeline. A frozen LLM planner and a seeded randomizer jointly populate a single SceneSpec. The LLM sets semantics (object, room, event family), and the seed sets all physics while Genesis simulates at 240 Hz into three synchronized views with simulator-derived ground truth. At a frozen observation, the evaluated MLLM (the brain) commits a structured reaction plan that a pre-trained controller (spine) executes on a simulated humanoid (body) in the same scene. The plan is scored on three axes: reasonable, safe, and physically grounded. Every scene is produced by a three-stage pipeline (Figure 2). Stage 1: LLM semantic planning. A frozen LLM parses a natural-language description (âa cast-iron pan slides off the counter while someone is cookingâ) into a validated selection: event family, object (from a per-family catalogue), room type, and family-specific hints (e.g., which intermediate surface a cascading object lands on). Outputs are validated against the catalogue with automatic retry on malformed responses. Objects named in descriptions but absent from the library can be sourced automatically from Objaverse (Deitke et al. 2023): candidates retrieved by LVIS label are scored for shape plausibility, re-oriented, welded, and registered. Stage 2: Seeded physical randomization. A deterministic randomizer, a pure function of an integer seed, samples every physical and visual parameter: placement and overhang, initial linear and angular velocities, friction and restitution, room and table geometry offsets, lighting, and camera poses (with a retry loop guaranteeing the object remains in frame). Family-specific samplers encode each eventâs physics: ramp angles are floored above the friction cone so sliding is guaranteed; ladder lean angles bifurcate into slide-down versus tip-over regimes; chain-reaction targets are settled to rest and only then struck by the trigger. Physics is never scripted mid-scene; outcomes emerge from initial conditions alone. Stage 3: Simulation and labeling. Scenes are simulated in Genesis (Genesis Authors 2024) with a rigid-body solver at 240 Hz and rendered at 1280Ă7201280Ă720/60 fps from three cameras. Ground truth is extracted from simulator state: the action label from the objectâs catalogued properties and event kinematics; the impact point impactââ3p_impact ^3 where the trajectory crosses the observerâs reach plane; and time-to-floor from first floor contact. Each scene ships its complete specification; re-running the specification reproduces the simulation bit-for-bit, and geometric validators check specâsimulation consistency before rendering. This division of labor is deliberate: the LLM contributes semantic diversity and natural-language grounding but never touches numbers; the seeded randomizer owns all continuous parameters, so labels are exact and the benchmark extends procedurally past 10410^4 scenes (with adversarial variants injected at a configurable rate) at zero annotation cost. 4 Metric Suite We report five metrics per decision, each aimed at a different kind of failure: a model can pick the right action but still be unsafe, or give a sensible reason yet send its hands to the wrong place. Throughout, s indexes scenes and m models. The model never outputs an action label; we recover the action from its motor plan (the walking command and hand keyframes) with a fixed rule that is released with the evaluation code. 1. Semantic Action Accuracy (SAA). SAA(s,m)=[a^(s,m)=aâ(s)]SAA(s,m)=1\! [ a(s,m)=a^*(s) ]: whether the chosen action matches the ground truth. This checks the action label alone, so a model can score SAA=1SAA=1 and still break a safety rule. 2. Safety Validity. A binary flag for safety-critical mistakes, scored independently of SAA. A decision is unsafe if it breaks any of four rules: (R1) no parseable output; (R2) Catch on an object labeled dangerous; (R3) No_Action while danger is present; (R4) Catch or No_Action when the ground truth is Dodge. We record which rule was broken. We keep safety separate from SAA on purpose: catching a dangerous object is unsafe even when catching is a reasonable move in general. 3. Physical Endpoint Distance. dendâ(s,m)=â„finalâ(s,m)âimpactâ(s)â„2d_end(s,m)= _final(s,m)-p_impact(s) _2 in meters, the distance between the final hand position and the true impact point. A model can choose the right action and still put its hands in the wrong place. 4. ActionâIntent Alignment (AIA). A three-level score (00, 0.50.5, 11) for whether the stated intent matches the goal of the ground-truth action. It separates a real misunderstanding from a right idea with a bad output: high AIA with SAA=0SAA=0 means the model understood the scene but wrote the wrong plan, while low AIA means it misread the scene. Intent is currently scored by keyword matching. 5. Hand-Distance Evolution (HDE). Along the keyframe path we compute di=â„iâimpactâ„2d_i= _i-p_impact _2, and report the closest the hand gets, dmin=miniâĄdid_ = _id_i, together with how much it closes in, d1âdmind_1-d_ . This shows whether the hand actually moves toward the target, which the endpoint alone can hide: a path can wander and still finish near the impact point. Aggregation. We average each metric per model and per family, dropping missing values from the denominator and reporting how many. Safety takes priority in our plots: once a decision breaks a safety rule, that dominates its rating regardless of the other numbers. 5 Experiments Ceiling Drop Stair Tumble Ladder Slip Door Swing Pendulum Object Drop Furniture Tip Sliding Object Shelf Slide Chain Reaction Bouncing Object Hanging Fall Figure 3: Twelve scenes from twelve event families. Each column is one scene read top to bottom (three frames, third-person view), with the event family named above; all reactions shown are produced by Claude Opus 4.8. Every decision is physically executed, so each reaction ends in an observable outcome. 5.1 Setup Models. We evaluate seven MLLMs zero-shot through one unified API (OpenRouter): five frontier models (Claude Opus 4.8, GPT-5.5, Gemini 2.5 Flash, Kimi K2.6, Qwen3-VL-235B) and two small open-weight models (Qwen3-VL-30B and Gemma-3-27B), which probe whether the benchmark separates capability tiers.11 1 Model IDs: anthropic/claude-opus-4.8, openai/gpt-5.5, google/gemini-2.5-flash, moonshotai/kimi-k2.6, qwen/qwen3-vl-235b-a22b-instruct, qwen/qwen3-vl-30b-a3b-instruct, google/gemma-3-27b-it. All models receive identical prompts and output schema, observe the same ⌠0.6 s multi-view window, and answer as JSON. Controller. The legs follow a pre-trained RL walking policy (Rudin et al. 2022; Unitree Robotics 2024); a scripted upper-body layer tracks the commanded hand keyframes. Early prototypes used motion-imitation whole-body controllers (Luo et al. 2023; Tessler et al. 2024), which proved harder to steer with task-level commands and less stable under sudden impacts, so we kept the simpler decoupled spine. Scenes. We evaluate a balanced subset of 306 scenes, 18 per family across all 17 families, so that no family dominates the aggregate metrics. Paired with the seven models this yields 2,138 decisions (four lost to API errors). 5.2 Results and Analysis Model SAA â Safety â dendd_end(m) â AIA â dmind_ (m) â Claude Opus 4.8 50.7 64.7 1.07 0.41 1.06 GPT-5.5 63.7 86.3 1.23 0.47 1.21 Gemini 2.5 Flash 48.7 78.1 1.28 0.47 1.28 Kimi K2.6 40.9 78.2 1.30 0.32 1.28 Qwen3-VL-235B 56.4 87.2 1.25 0.48 1.23 Qwen3-VL-30B 54.9 83.0 1.20 0.50 1.17 Gemma-3-27B 62.4 88.2 1.26 0.61 1.25 Mean 54.0 80.8 1.23 0.47 1.21 Table 1: Main results (%; distances in meters) over 306 scenes spanning all 17 families (2,138 sceneâmodel decisions; four lost to API errors). Distance metrics are computed on scenes with a defined impact point; best per column in bold. Key findings. Table 1 reports the per-model results; three patterns stand out. (1) Models fail most when they need to evade. On scenes whose correct response is Dodge, 35.9% of decisions violate a safety rule, and this holds for every model (23â66%). The most common error is simply not moving: 253 of the 392 mistakes are freezing in place. (2) Each model has a fixed bias. Six of seven models prefer to retreat and score much better on dodge scenes than on catch scenes (Figure 4); Claude is the opposite, catching well (67%) but engaging hazards it should avoid (34% on dodge scenes). No model adapts its behavior to the scene. (3) Bigger is not safer. The 27B Gemma ranks second on accuracy (62.4%, just behind GPT-5.5) and first on safety (88.2%), and both ⌠30B open-weight models outscore most of the frontier tier on both columns, while the strongest flagship, Claude Opus 4.8, is the least safe model in the table (64.7%). Figure 4: Action choice reflects the model, not the scene. Each point is one model: SAA on catch scenes (x) vs. dodge scenes (y); the dashed diagonal marks equal skill. Six models sit far above it, Claude far below, and the upper right, strong on both sides, stays empty. Figure 5: Slow hazards are missed by every model. (a) Freeze rate at the slowest speed band vs. the three faster bands, over the twelve families that genuinely sweep object speed: every model freezes more on slow hazards (pooled, 42% vs. 21%). (b) Claudeâs input for one rolling-ball scene at two bands (ground truth Catch in both). The slow ball still rolls toward the table edge and arrives 1 s after the window closes, yet six of seven models freeze, Claude included; at the normal band none freeze and Claude catches. Per-family breakdown. Since six of seven models favor retreat (Finding 2), families whose answer is Dodge score high for free; we therefore compare catch and dodge families separately. Catch families get harder as more pieces move: a single bouncing ball is caught 66.7% of the time, a collapsing stack of cans only 38.9%. Dodge families get harder as warning time shrinks: a slowly falling ceiling fixture is evaded 91.3% of the time, an object thrown at the robot only 54.8%, and nearly half of the thrown-object decisions (44%) break a safety rule. Safety errors. Of the 410 violations, freezing under danger is the largest class (R3, 162), ahead of staying put or engaging when evasion was required (R4, 128) and catching an object labeled dangerous (R2, 120); R1 (unparseable output) never fired. Models fail by misjudging the hazard, not by lacking the right action. Correct but out of reach. Saying Catch is not catching. Even when models pick the right action, their hands land a median 0.48 m from where the object actually arrives, and 89% of the misses fall short. The cause is simple: models reach with the arms but do not step forward. In half of these decisions (49%) the robot stands more than 0.8 m away, farther than its arm can reach at all. A multiple-choice benchmark would mark every one of these correct, while physical execution shows they miss. This is why ReactHuman executes every plan. Speed is perceived as binary. Twelve of the seventeen families sweep the objectâs true speed across four bands while holding everything else fixed; in the remaining five gravity sets the pace, so we exclude them here. On these 826 decisions, models read speed only as moving-or-not (Figure 5a): at the slowest band they freeze twice as often (42% vs. roughly 21%), treating slow hazards as no event at all even when the motion is plainly visible in their input (Figure 5b), and accuracy drops with them (38.1% SAA vs. 47â54% at the faster bands). From normal to very fast the action distribution barely moves. Individual decisions are also unstable: a model keeps the same answer across all four speeds in only 27% of scene groups, and the slowest and fastest variants agree just 54% of the time. Models detect that something moves; how fast never enters the decision. Voting does not help. The seven models agree unanimously on only 16% of scenes, and when they do, they are right 80% of the time. Pooling them buys little: a majority vote over all seven scores 62.1%, better than the average model (54.0%) but no better than the best single model (63.7%). They share blind spots, and an ensemble cannot vote shared errors away. Adversarial probes. Forty scenes plant an object whose appearance lies about its physics: foam doors and foam ceiling panels disguised as heavy, lead sandbags and steel cans disguised as light. The disguise works completely: across the 40Ă7=28040Ă 7=280 decisions, no model ever voices a doubt about an objectâs material or weight. For example, the foam ceiling panels are dodged in 49 of 56 decisions, just as real slabs would be, and the solid-steel cans get no discount for their weight: models reach for them exactly as often as for genuinely light cans (39% vs. 39% of decisions). For current MLLMs, seeing is believing: physics is assigned by visual category and never revised from observed motion. 6 Discussion and Limitations What ReactHuman reveals. The failures are not exotic: models freeze in the path of hazards they plainly see, apply one fixed disposition to every scene instead of reading it, treat speed as a binary and slow hazards as no event, and never question an objectâs physics when its appearance disagrees. None of these improve with scale, and none are visible to an answer-only protocol. For deployment each points to a different mitigation: safety-constrained decoding for the freeze default, paired catch-and-dodge training to break fixed dispositions, and motion-grounded supervision for speed and material. Limitations. Ground truth is simulator-derived; rigid-body simulation omits deformation and shattering, and conclusions should be validated on real hardware. The decision is open-loop by construction (one frozen observation, one committed plan); the humanoid execution that follows is physically simulated but not re-planned mid-motion. Fully closed-loop evaluation with repeated observationâaction cycles, cloth and fluid events, and native VLA policies are natural extensions. Finally, the evaluated models are moving targets: results reflect the specific API snapshots queried, and the standing benchmark, not any one number, is the contribution. 7 Conclusion ReactHuman reframes physical-reasoning evaluation from passive question answering to reactive, safety-critical decision-making. Its 17 event families, exact simulator ground truth, adversarial appearanceâphysics probes, and five-metric diagnostic suite jointly measure not just whether an embodied MLLM knows physics, but whether it can act safely within it, and localize the failure when it cannot. All scenes are bit-for-bit reproducible and the LLM-planned, seed-deterministic pipeline extends the benchmark without annotation cost. Across seven models, reactive safety proves far from solved: models mishandle roughly one hazard in three, act from a fixed disposition rather than the scene in front of them, and do not improve with scale. We release the dataset to support further study. Acknowledgments This work is supported by the Canada NSERC Discovery Grant (RGPIN-2021-03115). References Ahn et al. (2022) M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, et al. Do as i can, not as i say: grounding language in robotic affordances. In CoRL, Cited by: §1, §2. Bear et al. (2021) D. M. Bear, E. Wang, D. Mrowca, F. J. Binder, H. F. Tung, et al. Physion: evaluating physical prediction from vision in humans and machines. In NeurIPS Datasets and Benchmarks, Cited by: §1, §2. Black et al. (2024) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, et al. Ï0 _0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: §2. Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, et al. RT-2: vision-language-action models transfer web knowledge to robotic control. In CoRL, Cited by: §2. Chow et al. (2025) W. Chow, J. Mao, B. Li, D. Seita, V. Guizilini, and Y. Wang PhysBench: benchmarking and enhancing vision-language models for physical world understanding. In ICLR, Cited by: §1, §2. Deitke et al. (2023) M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi Objaverse: a universe of annotated 3d objects. In CVPR, Cited by: §2, §3.4. Driess et al. (2023) D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, et al. PaLM-e: an embodied multimodal language model. In ICML, Cited by: §1, §2. Gan et al. (2021) C. Gan, J. Schwartz, S. Alter, D. Mrowca, M. Schrimpf, J. Traer, et al. ThreeDWorld: a platform for interactive multi-modal physical simulation. In NeurIPS Datasets and Benchmarks, Cited by: §2. Genesis Authors (2024) Genesis Authors Genesis: a generative and universal physics engine for robotics and beyond. Note: https://github.com/Genesis-Embodied-AI/Genesis Cited by: §3.4. Greff et al. (2022) K. Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, et al. Kubric: a scalable dataset generator. In CVPR, Cited by: §2. Gu et al. (2023) J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, et al. ManiSkill2: a unified benchmark for generalizable manipulation skills. In ICLR, Cited by: §2. Kim et al. (2024) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, et al. OpenVLA: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: §2. Kolve et al. (2017) E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi AI2-thor: an interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474. Cited by: §2. Li et al. (2022) C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. MartĂn-MartĂn, et al. BEHAVIOR-1k: a benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In CoRL, Cited by: §1, §2. Luo et al. (2023) Z. Luo, J. Cao, K. Kitani, W. Xu, et al. Perpetual humanoid control for real-time simulated avatars. In ICCV, Cited by: §5.1. Majumdar et al. (2024) A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, et al. OpenEQA: embodied question answering in the era of foundation models. In CVPR, Cited by: §2. Motamed et al. (2025) S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos Do generative video models understand physical principles?. arXiv preprint arXiv:2501.09038. Cited by: §2. Mu et al. (2021) T. Mu, Z. Ling, F. Xiang, D. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su ManiSkill: generalizable manipulation skill benchmark with large-scale demonstrations. External Links: 2107.14483, Link Cited by: §2. Riochet et al. (2018) R. Riochet, M. Y. Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux IntPhys: a framework and benchmark for visual intuitive physics reasoning. arXiv preprint arXiv:1803.07616. Cited by: §2. Rudin et al. (2022) N. Rudin, D. Hoeller, P. Reist, and M. Hutter Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, p. 91â100. Cited by: §1, §2, §5.1. Savva et al. (2019) M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra Habitat: a platform for embodied ai research. In ICCV, Cited by: §2. Sferrazza et al. (2024) C. Sferrazza, D. Huang, X. Lin, Y. Lee, and P. Abbeel HumanoidBench: simulated humanoid benchmark for whole-body locomotion and manipulation. In RSS, Cited by: §2. Szot et al. (2021) A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, et al. Habitat 2.0: training home assistants to rearrange their habitat. In NeurIPS, Cited by: §1, §2. Tessler et al. (2024) C. Tessler, Y. Guo, O. Nabati, G. Chechik, and X. B. Peng MaskedMimic: unified physics-based character control through masked motion inpainting. ACM Transactions on Graphics 43 (6). Cited by: §5.1. Tung et al. (2023) H. F. Tung, M. Ding, Z. Chen, D. Bear, C. Gan, J. B. Tenenbaum, D. L.K. Yamins, J. Fan, and K. Smith Physion++: evaluating physical scene understanding that requires online inference of different physical properties. In NeurIPS, Cited by: §2. Unitree Robotics (2024) Unitree Robotics Unitree rl gym: reinforcement learning environments for unitree robots. Note: https://github.com/unitreerobotics/unitree_rl_gym Cited by: §1, §2, §5.1. Wang et al. (2024) Y. Wang, Z. Xian, F. Chen, T. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan RoboGen: towards unleashing infinite data for automated robot learning via generative simulation. In ICML, Cited by: §2. Yang et al. (2024) Y. Yang, F. Sun, L. Weihs, E. VanderBilt, A. Herrasti, W. Han, et al. Holodeck: language guided generation of 3d embodied ai environments. In CVPR, Cited by: §2. Yi et al. (2020) K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum CLEVRER: collision events for video representation and reasoning. In ICLR, Cited by: §1, §2. Zheng et al. (2024) Z. Zheng, X. Yan, Z. Chen, J. Wang, Q. Z. E. Lim, J. B. Tenenbaum, and C. Gan ContPhy: continuum physical concept learning and reasoning from videos. In ICML, Cited by: §1, §2.