Paper deep dive
KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis
Mehdi Hosseinzadeh, King Hang Wong, Feras Dayoub
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 3:47:39 AM
Summary
KITE (Keyframe-Indexed Tokenized Evidence) is a training-free, layout-grounded front-end for vision-language models (VLMs) that converts long robot-execution videos into compact, interpretable evidence. By distilling trajectories into motion-salient keyframes paired with schematic bird's-eye-view (BEV) representations, robot-profile tokens, and scene-context, KITE enables off-the-shelf VLMs to perform failure detection, identification, localization, explanation, and correction with high accuracy on the RoboFAC benchmark.
Entities (5)
Relation Signals (3)
KITE → evaluatedon → RoboFAC
confidence 100% · On the RoboFAC benchmark, KITE with Qwen2.5-VL substantially improves over vanilla Qwen2.5-VL
KITE → improvesperformanceof → Qwen2.5-VL
confidence 95% · KITE with Qwen2.5-VL substantially improves over vanilla Qwen2.5-VL in the training-free setting
DART → demonstratesapplicabilityof → KITE
confidence 90% · We also report qualitative results on real dual-arm robots, demonstrating the practical applicability of KITE
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present KITE, a training-free, keyframe-anchored, layout-grounded front-end that converts long robot-execution videos into compact, interpretable tokenized evidence for vision-language models (VLMs). KITE distills each trajectory into a small set of motion-salient keyframes with open-vocabulary detections and pairs each keyframe with a schematic bird's-eye-view (BEV) representation that encodes relative object layout, axes, timestamps, and detection confidence. These visual cues are serialized with robot-profile and scene-context tokens into a unified prompt, allowing the same front-end to support failure detection, identification, localization, explanation, and correction with an off-the-shelf VLM. On the RoboFAC benchmark, KITE with Qwen2.5-VL substantially improves over vanilla Qwen2.5-VL in the training-free setting, with especially large gains on simulation failure detection, identification, and localization, while remaining competitive with a RoboFAC-tuned baseline. A small QLoRA fine-tune further improves explanation and correction quality. We also report qualitative results on real dual-arm robots, demonstrating the practical applicability of KITE as a structured and interpretable front-end for robot failure analysis. Code and models are released on our project page: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2604.07034v1
- Canonical: https://arxiv.org/abs/2604.07034v1
Trouble viewing inline? Open PDF directly →
Full Text
43,827 characters extracted from source content.
Expand or collapse full text
KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis Mehdi Hosseinzadeh, King Hang Wong, and Feras Dayoub https://m80hz.github.io/kite/ Q: Failure Explanation ... A: The robot failed to complete the task because it did not successfully grasp the cup with its left hand. The robot's arm was positioned above the cup but did not make contact with it, as indicated by the lack of movement or change in position of the cup in the BEV diagram. This suggests that the robot's sensors may have failed to detect the cup, or there may have been an issue with the gripper's ability to grasp the cup. VLM === ROBOT PROFILE === Robot name: DART. Morphology: mobile-base dual-arm. Arms: 2. Grippers: 2. End-effector: dexterous hands. ... === PREDICTED PLAN STEPS === 1. Grasp the cup with the left hand. 2. Pick up the cup from the table. 3. Move the cup towards the mouth of the robot. === Global Scene Graph and Tracks === "timeline": "keyframes_sec": ..., "global_scene": "tracks": ["track_id": 0, "cls": "robot arm", ... "relations": ["subj": "robot arm", "rel": "left_of", "obj": "fork", ...] ... KITE Structured Context , s Language Instructions Images Fig. 1: Failure explanation in real-world settings with KITE. Example sequence from the Dual-Arm Robot (DART) in the lab. Left (top to bottom): optical flow estimates used for keyframe selection; RGB keyframe with object detection overlays; single-view depth estimates; and a pseudo-BEV schematic (circles with radius∝confidence; X/Z axes; timestamp). Notably, in its failure explanation, the VLM references the BEV diagram to infer that the cup’s position remains unchanged. Abstract— We present KITE, a training-free, keyframe- anchored, layout-grounded front-end that converts long robot- execution videos into compact, interpretable tokenized evidence for vision-language models (VLMs). KITE distills each trajectory into a small set of motion-salient keyframes with open-vocabulary detections and pairs each keyframe with a schematic bird’s-eye- view (BEV) representation that encodes relative object layout, axes, timestamps, and detection confidence. These visual cues are serialized with robot-profile and scene-context tokens into a unified prompt, allowing the same front-end to support failure detection, identification, localization, explanation, and correction with an off-the-shelf VLM. On the RoboFAC benchmark, KITE with Qwen2.5-VL substantially improves over vanilla Qwen2.5- VL in the training-free setting, with especially large gains on simulation failure detection, identification, and localization, while remaining competitive with a RoboFAC-tuned baseline. A small QLoRA fine-tune further improves explanation and correction quality. We also report qualitative results on real dual-arm robots, demonstrating the practical applicability of KITE as a structured and interpretable front-end for robot failure analysis. Code and models are released on our project page: https://m80hz.github.io/kite/ I. INTRODUCTION Robots executing long-horizon manipulation in the wild still fail in mundane but consequential ways: a gripper Authors are with the Australian Institute for Machine Learning (AIML), Adelaide University, Australia. approaches a mug off-axis and slides off, a handle is contacted too late relative to the arm motion, or a bimanual handover misaligns in space and time. Explaining such failures requires combining where (layout, contact, relative pose), when (the moment the execution deviates), and what (task intent and subgoals). In practice, these cues are often subtle, distributed across time, and difficult to recover from raw video alone. This challenge helps explain why the robotics community has increasingly turned to large language models (LLMs) and vision-language models (VLMs) as general-purpose reasoning interfaces. Foundation models offer open-vocabulary percep- tion, natural-language conditioning, reusable commonsense priors, and a single interface that can support planning, monitoring, explanation, and recovery across many tasks and embodiments [1]–[8]. Yet their strengths are blunted when the input is a long raw execution video: subtle failure cues are easily buried in dense visual detail, temporal context is diluted, and the evidence needed for diagnosis is rarely presented in a form that is immediately legible to the model. Prior work has begun to address this problem by summariz- ing robot experiences for an LLM [9], training failure-specific VLMs [10], or introducing failure-analysis QA benchmarks [11]. These directions are important, but they either depend on additional task-specific training, or they still leave open arXiv:2604.07034v1 [cs.RO] 8 Apr 2026 the representation problem: how should a long execution be converted into a compact form that preserves the key spatiotemporal evidence needed for post-hoc diagnosis? To the best of our knowledge, there is still no simple training- free front-end that makes those facts immediately accessible to an off-the-shelf generalist VLM. To address this gap, we propose Keyframe-Indexed Tok- enized Evidence (KITE), a compact and interpretable front- end that converts an execution into a small set of motion- salient keyframes, each paired with a schematic pseudo-BEV of relative layout and a minimal scene/interaction summary. As previewed in Fig. 1, KITE externalizes the evidence a VLM needs: object detections, coarse depth ordering, robot- profile information, scene relations, timestamps, and contact- transition cues are turned into a temporally indexed storyboard and serialized prompt context. The resulting representation supports failure detection, identification, localization, expla- nation, and correction without task-specific prompt redesign. In summary, the contributions of this paper are as follows: •A training-free, keyframe-indexed, layout-grounded front-end that converts long robot-execution videos into compact evidence consumable by general-purpose VLMs; •A multimodal evidence representation that combines pseudo-BEV schematics, robot-profile information, scene relations, and contact-transition cues in a single inter- pretable prompt format; •A keyframe-indexed failure localization method, in contrast to approaches that localize failures only at coarse plan steps. On RoboFAC [11], KITE with a general-purpose VLM substantially improves over vanilla Qwen2.5-VL and re- mains competitive with a RoboFAC-tuned baseline, while a lightweight QLoRA fine-tune further improves explanation and correction quality. We also demonstrate qualitative effectiveness on rollout episodes from two real dual-arm robots, the RealMan Dual Arm Compound Robot [12] and ALOHA-2 [13]. I. RELATED WORK Foundation Models for Robotics. Recent advances in Large-Language-Models (LLMs) [14]–[17] and Vision- Language-Models (VLMs) [18]–[22] have catalyzed their in- tegration into robotics across planning, control, and interaction domains. Numerous works leverage pretrained LLMs as high- level planners for robots, using natural language understanding and commonsense to decompose tasks and guide actions. For example, [1] pioneered grounding an LLM in robotic affordances for instruction-following, and subsequent systems [2]–[5], [23]–[27] have combined language and perception in embodied models that reason over visual inputs to plan robot behavior [6]–[8]. LLM-driven policies have been applied to navigation and mobile manipulation tasks, including domestic assistive robots that follow instructional/guiding prompts to tidy environments or perform user requests. Such LLM-based planners and embodied agents have demonstrated flexible task generalization and improved semantic understanding in novel scenarios. This broader shift motivates our setting: if foundation models are increasingly used to plan and guide robot behavior, they should also be able to analyze and explain robot failures. The key difficulty is that post-hoc failure analysis requires evidence to be presented in a temporally compact and spatially legible form. Robotic Failure Analysis, Retrospection, and Recovery. Failure explanation has long been studied in explainable robotics and human-robot interaction (HRI), including verbal- ization, user-facing explanations, and recovery support [28]– [33]. More recently, LLM-based methods have been used to summarize experiences, diagnose failures, and suggest corrective actions [9], [34]–[36]. In parallel, VLM-based approaches have been explored as success or failure detectors, and instruction-tuned models have been trained specifically for robotic failure reasoning [10], [37]–[40]. These methods show the promise of foundation-model reasoning, but many rely on task-specific fine-tuning or large memory structures. Structured Representations for Multimodal Reasoning. Recent work suggests that structured scene abstractions can inject useful inductive bias into language-guided reasoning without requiring full retraining. Examples include 3D scene graphs for task grounding [41], bird’s-eye-view (BEV) interfaces for multimodal reasoning [42], and diagrammatic abstractions for improved visual understanding [43]. KITE is most closely aligned with this direction. Compared with RE- FLECT [9], which reasons over summarized robot memories and multisensory logs, KITE focuses on the representation in- terface itself: it converts long execution videos into a compact, keyframe-indexed, layout-grounded evidence bundle that can be consumed directly by a pretrained VLM. Compared with tuning-heavy failure-analysis approaches [10], KITE remains training-free at the front end and is designed to preserve interpretability through explicit keyframes, schematic layouts, and serialized evidence tokens. I. METHOD: KITE FRONT-END KITE is a training-free, model-agnostic front-end that con- verts a long robot-execution video into a compact bundle of motion-salient keyframes, schematic pseudo-BEVs, and serialized evidence tokens. Each component is designed to externalize a factor that is otherwise difficult for a VLM to infer reliably from raw video alone: when via keyframes, where via pseudo-BEV and scene relations, and what/how via robot-profile and task-context tokens. The resulting representation is layout-grounded, temporally indexed, and directly consumable by a general-purpose VLM. An overview appears in Fig. 2. A. Preliminaries and Notation Let a video beV =(I t , t) T t=1 , whereI t ∈R H×W×3 is an RGB frame andtis its timestamp. We select up toM keyframes K =(I t k , t k , i k ) M k=1 , wherei k is the frame index. For each keyframek, we compute: (i) an open-vocabulary detection set O k =(b j , c j , s j ) with boxb j , classc j , and confidences j ; and (i) a relative depth Vision Language Model s Keyframe Selection motion (optical flow) s Open-Vocabulary Object Detection GroundingDINO s Single-view Depth Estimation Depth-Anything-V2 s Scene Graph 3D s Pseudo-BEV Schematics Qwen2.5-VL Language Output Response (Failure Detection/Explanation) Narrative Summary Input Video s Robot Description 3D Point-cloud Structured Text Images , KITE Structured Context ([ROBOT], [PLAN],[KF], [CONTACT],[GLOBAL SCENE]) s Language Instruction Fig. 2: Overview of KITE. The proposed pipeline takes a raw video and distills it into a small set of salient keyframes, identified using motion-based peaks. For each keyframe, we run open-vocabulary detection to localize the robot and surrounding objects, and render a pseudo-BEV schematic that depicts the scene layout with simple, interpretable symbols. These visual elements are paired with a structured context and form a compact, interpretable front-end for prompting a vision-language model. The model can then answer diverse failure analysis QA tasks, as well as generate grounded explanations and final narratives. The sequence illustrated here comes from the real-world subset of the RoboFAC benchmark [11]. mapD k (single-view, up to scale). For each consecutive keyframe pair(k, k + 1), we additionally compute a contact- transition token γ k→k+1 ∈GAIN, LOSS, STABLE, which summarizes coarse interaction changes between the robot gripper and its nearest object. B. Keyframe Selection We operate under a small keyframe budgetMand prioritize motion-salient frames. To detect salient events, we compute dense optical flow between consecutive frames and score each frame by its average flow magnitude. Keyframes are selected as local peaks in this score using temporal non-maximum suppression. If fewer thanMsalient frames are identified, we supplement them with uniformly spaced frames to preserve contextual coverage. We use dense rather than sparse flow because our goal is not correspondence tracking, but a stable scene-wide saliency signal: manipulation failures often involve distributed motion of the arm, gripper, and object, while sparse keypoints can be unreliable in texture-poor or partially occluded robot scenes. The selector is modular and can be replaced by entropy- based or learned policies; our ablations compare motion-peak selection against uniform sampling. C. Per-Keyframe Perception Open-Vocabulary Detection (OVD): We run an OVD module (e.g., GroundingDINO [44]) to detect objects of interest and robot arms/grippers. Detections are temporally linked across keyframes into short tracks (instance IDs), and timestampst k are rendered as overlays on the RGB keyframes. Single-View Depth Estimation: We estimate relative depth per keyframe (e.g., Depth-Anything-V2 [45]) and associate depth statistics with each detection. We use depth only as a coarse ordering cue, not as metric geometry. Contact-Transition Proxy: To provide a simple but informative interaction signal, we compute a coarse contact- transition token whenever a robot hand or gripper and a candidate object are both detected with high confidence in consecutive keyframes. Letd k denote the nearest-center distance between the gripper and its closest object at keyframe k, and letIoU k denote the corresponding bounding-box IoU. With thresholds τ IoU , τ d > 0, we define ∆IoU k = IoU k+1 − IoU k ,∆d k = d k+1 − d k . γ k→k+1 = GAIN,∆IoU k ≥ τ IoU , ∆d k ≤−τ d , LOSS, −∆IoU k ≥ τ IoU , ∆d k ≥ τ d , STABLE, otherwise. This token is intentionally coarse: it captures interaction trends that are often useful for failure analysis, without requiring force sensing or precise contact estimation. D. 3D Scene Graph For each keyframe we build a local scene graphG k whose nodes are detections inO k with 3D centroids approximated from relative depth and camera geometry. We encode pairwise relations from the set LEFT_OF, ABOVE, IN_FRONT_OF using the sign and magnitude of centroid offsets with small tolerance thresholds. Local graphsG k M k=1 are aggregated into a global graph by maintaining instance tracks across keyframes. We use a coarse 3D scene graph rather than a purely 2D graph because front/back ordering is often ambiguous in image coordinates alone, while relative depth provides enough signal to recover non-metric spatial ordering useful for failure diagnosis. Our goal is not full geometric reconstruction, but a compact relational scaffold that can disambiguate layout cues for the VLM. E. Robot Description We include a concise robot profile describing morphology (#arms, #grippers, end-effector types), sensors, workspace, and salient embodiment constraints. This allows the VLM to condition explanations and corrections on the robot platform and environment. F. Pseudo-BEV Schematic (Layout Prior) Photorealistic reconstructions are costly and not necessarily aligned with what current VLMs parse most reliably. We there- fore render a schematic, non-metric top-down pseudo-bird’s- eye-view (pseudo-BEV) for each keyframe that externalizes relative layout while preserving identity consistency across modalities: • fixed axes (X right, Z forward) with arrows; •one circle per tracked object, with radius proportional to confidence s j ; •the object class label and the same instance ID used in the RGB overlay; • overlaid timestamp t k and keyframe index. Pseudo-BEVs are not metrically accurate maps; they are schematic layout cues intended to make spatial relationships easier for the VLM to read. G. KITE: Keyframe-Indexed Tokenized Evidence We serialize a compact context prefix that acts as a single front-end across all QA tasks. LetTdenote the KITE context string: T = [ROBOT] short description | z morphology, gripper, workspace ∥ [PLAN] high-level plan |z optional task context ∥ [KF i k @ t k ] |z timestamped keyframe tags ∥ [CONTACT k→ k+1] γ k→k+1 |z GAIN/LOSS/STABLE ∥ [GLOBAL_SCENE] tracks & relations | z IDs consistent with RGB/pseudo-BEV . If plan steps are unavailable, the [PLAN] field is omitted. H. Prompting and Failure Localization For each question, we provide a compact image bundle consisting of RGB keyframe overlays and their corresponding pseudo-BEVs, and prependTto the text prompt. We include a brief instruction stating that the pseudo-BEV is a schematic, not to scale, and should be used only for relative layout reasoning. For frame-level failure localization, we request strict JSON: "candidates":["frame_num": INT, "confidence": FLOAT, ...] with up to three candidates and confidence values in[0, 1]. A simple parser extracts the top candidate, and subsequent analysis can then be aligned to that evidence frame. I. Narrative Summary GivenTand a storyboard montage containing all selected keyframes and pseudo-BEVs, we prompt the VLM for a concise causal narrative that explicitly references keyframe IDs and timestamps, and proposes one high-level and one low-level correction. Since all perception is performed only on the selected keyframes, the overall cost scales linearly withMand is independent of the original video length once M is fixed. IV. EXPERIMENTS We evaluate KITE on RoboFAC [11], a large-scale bench- mark for robotic failure analysis, using both quantitative and qualitative analyses. Our main question is whether the proposed front-end improves a strong off-the-shelf VLM with- out task-specific training. Accordingly, our core comparison is vanilla Qwen2.5-VL versus KITE + Qwen2.5-VL, with the RoboFAC-tuned model and larger closed-source VLMs reported as reference baselines. We also provide ablations for pseudo-BEV and keyframe selection, and include qualitative rollouts from our lab robots—a RealMan dual-arm compound robot [12] (DART) and ALOHA-2 Stationary [13]—to illustrate transfer beyond the benchmark. RoboFAC contains only single-arm tasks, whereas our in-lab examples include dual-arm failures. A. Datasets and Tasks RoboFAC: RoboFAC [11] is a QA-style benchmark for robotic failure analysis containing both simulation and real-world sequences. It provides more than 60K training QA pairs from simulation, together with 10K simulated and 8K real-world QA pairs for testing. The benchmark defines eight question types: Task identification (TI), Task planning (TP), Failure detection (FD), Failure identification (FI), Failure locating (FL), Failure explanation (FE), High- level correction (HL), and Low-level correction (L). In this paper, we report the seven tasks directly relevant to failure analysis and correction (TI, FD, FI, FL, FE, HL, L); TP is used only as optional contextual information in KITE rather than as a primary evaluation target. We follow the official data splits and evaluation protocols where applicable. DART and ALOHA-2 (in-lab): We additionally test KITE qualitatively on in-lab sequences from DART and ALOHA-2. These examples are zero-shot with respect to our method and are intended to illustrate transfer to real dual-arm platforms and failure modes outside the single-arm benchmark setting. B. Backbones and Baselines We adopt Qwen2.5-VL [46] as the main backbone due to its strong vision-language capabilities and open-source availabil- ity. We additionally report results for Gemini-2.0 [22], GPT- 4o [18], vanilla Qwen2.5-VL-3B and 7B models (without KITE; RGB keyframes only), the RoboFAC-7B model fine- tuned on RoboFAC [11], our training-free KITE + Qwen2.5- VL, and KITE + Qwen2.5-VL further adapted with QLoRA. Object DetectionsOptical FlowPseudo-BEVDepth Estimation === ROBOT PROFILE === Robot name: So-100. Morphology: single-arm. Arms: 1. Grippers: 1. End-effector: parallel_jaw. Sensors: RGB camera. Workspace: tabletop. === PREDICTED PLAN STEPS === 1. The robot arm moves towards the red object. 2. The robot arm picks up the red object. 3. The robot arm moves towards the yellow object. 4. The robot arm places the red object on the yellow object. === Global Scene Graph and Tracks === "timeline": "keyframes_sec": [0.47, 1.13, 2.0, 2.4, 4.33], "global_scene": "tracks": ["track_id": 0, "cls": "cube", "mean_pos": [0.64, 0.13, 2.45], "time_span": [0.47, 2.4], "track_id": 1, "cls": "robot arm", "mean_pos": [-0.42, -0.1, 2.18], "time_span": [0.47, 1.13], ...,"relations": ["subj": "robot arm", "rel": "left_of", "obj": "cube", ...] KITE Structured Context Q: You are given key frames from a task execution plus structured context. Identify up to 3 candidate failure frames (by its frame number 0, 1, 2, ...) where failure likely occurred. Confidence in [0,1]. Return STRICT JSON only: "candidates":["frame_num": int, "confidence": float]. Candidates: 0:0.47s, 1:1.13s, 2:2.00s, 3:2.40s, 4:4.33s A: [ "frame_num": 3, "confidence": 0.9 ] Failure Locating The robot, SO-100, is attempting to pick up a red object from the table and place it on a yellow cube. The intended plan involves first picking the red object and then placing it on the yellow cube. In the video, the robot successfully picks up the red object and moves it towards the yellow cube. However, instead of placing the red object on the yellow cube, the robot continues to move the red object around the table, indicating a failure in the placement step. This failure occurs between keyframes 2.4 and 4.33 seconds, as the robot's end-effector is still holding the red object and moving it around the table rather than placing it on the yellow cube. Narrative Summary Fig. 3: Qualitative results in simulation (RoboFAC dataset). Each panel shows: RGB keyframe with object-detection overlays; optical-flow estimates; pseudo-BEV schematic (consistent object IDs; circle radius∝confidence; timestamp); and single-view depth estimates, all for the corresponding keyframes. We also illustrate a short structured-context excerpt, KITE’s response to a failure-localization query, and a final narrative summary. TABLE I: Performance of multi-modal baseline models on the RoboFAC Benchmark [11]. Success rate for MCQ questions is reported (higher is better) for both simulation and real-world tasks.†denotes the models that are finetuned on RoboFAC benchmark. Model SimulationReal-world FDFIFLFDFIFL Gemini-2.00.48 0.270.75 0.60 0.11 0.18 GPT-4o0.640.21 0.710.96 0.430.52 Qwen2.5-VL-3B0.38 0.04 0.51 0.04 0.03 0.07 Qwen2.5-VL-7B0.52 0.26 0.22 0.83 0.38 0.72 KITE + Qwen2.5-VL-7B0.88 0.44 0.55 0.840.43 0.74 RoboFAC-7B † 0.91 0.63 0.94 0.80 0.56 0.71 KITE+Qwen2.5-7B+QLoRA † 0.93 0.69 0.92 0.89 0.58 0.77 C. Metrics For multiple-choice questions (FD, FI, FL), we report success rate. For free-language tasks (TI, FE, HL, L), following [10], we report ROUGE-L F1 and Sentence-BERT cosine similarity between the generated answer and the reference answer. D. Simulation and Real-world Results Table I reports MCQ accuracy, and Table I reports free- language results measured by ROUGE-L and Sentence-BERT similarity, for both simulation and real-world settings. Training-free KITE substantially improves over vanilla Qwen2.5-VL-7B in simulation, with gains of+36points on FD,+18on FI, and+33on FL. On real-world MCQ tasks, the gains over vanilla Qwen2.5-VL-7B are smaller but consistently positive (+1on FD,+5on FI, and+2 on FL). For free-language tasks, KITE improves ROUGE- L in all reported TI/FE/HL/L settings and improves or closely matches Sentence-BERT similarity in nearly all cases. Applying QLoRA further improves performance across most reported dimensions and brings KITE close to, and in some cases beyond, the RoboFAC-tuned baseline. E. Ablations We isolate the contribution of pseudo-BEV and the keyframe selector on real-world tasks, as shown in Table I. In the↓pseudo-BEV setting, we remove only the pseudo- BEV images while keeping the RGB keyframes and all serialized text tokens unchanged. In the uniform keyframe setting, we replace motion-based selection with uniformly spaced keyframes while keeping the rest of KITE fixed. Removing pseudo-BEV reduces performance most clearly on failure explanation, where FE drops by 0.05 ROUGE-L, and also hurts FD/FI/FL. Replacing motion-based keyframes with uniform sampling causes a larger degradation overall, especially for questions that depend on identifying when and where the failure first becomes visible. F. Qualitative Analyses For a sequence from thePegInsertionSidesimula- tion task in RoboFAC, Fig. 3 shows the selected keyframes, object detections, optical-flow estimates, pseudo-BEV render- ings, depth estimates, a short excerpt of the KITE context, the failure-localization output, and the final narrative summary. The example illustrates how KITE makes the evidence chain legible to the VLM. We also present the same intermediate representations and outputs for real-world sequences recorded in our lab: TABLE I: Performance of multi-modal baseline models on the RoboFAC Benchmark [11]. ROUGE-L and SBERT cosine similarity metrics (higher is better) are reported for free-language reasoning tasks for both simulation and real-world tasks.† denotes the models that are finetuned on RoboFAC benchmark. Model Sim (ROUGE-L)Sim (SBERT Cosine)Real (ROUGE-L)Real (SBERT Cosine) TIFEHLLLTIFEHLLLTIFEHLLLTIFEHLLL Qwen2.5-VL-7B0.206 0.194 0.230 0.157 0.546 0.448 0.683 0.6570.264 0.233 0.219 0.197 0.689 0.786 0.792 0.785 KITE + Qwen2.5-VL-7B0.295 0.248 0.241 0.190 0.680 0.829 0.798 0.7790.300 0.252 0.223 0.232 0.696 0.832 0.791 0.804 RoboFAC-7B † 0.323 0.299 0.301 0.245 0.701 0.842 0.808 0.7940.337 0.361 0.228 0.305 0.722 0.856 0.798 0.813 KITE+Qwen2.5-7B+QLoRA † 0.326 0.314 0.302 0.296 0.698 0.845 0.806 0.8030.338 0.365 0.229 0.313 0.724 0.860 0.798 0.815 TABLE I: Ablation study of our method. The↓indicates the feature is removed. Success rate for MCQ and ROUGE-L metric for other question dimensions are reported for real- world tasks. Config success rateROUGE-L FDFIFLTIFEHLLL Full (KITE)0.84 0.43 0.74 0.300 0.252 0.223 0.232 ↓ pseudo-BEV0.81 0.37 0.70 0.302 0.202 0.221 0.228 uniform keyframe 0.69 0.33 0.56 0.298 0.189 0.217 0.190 DART in Fig. 1, and ALOHA-2 in Fig. 4. In Fig. 4, the object is dropped during a dual-arm handover. The generated explanation explicitly ties the failure to the robot embodiment and the observed sequence, illustrating the value of including robot-profile information in the structured context. G. Implementation Details a) Keyframes.: We use up toM = 8keyframes per video. This budget was chosen to balance temporal coverage against the multimodal context length that can be passed to the VLM. Keyframes are selected primarily from motion-salient peaks detected via optical flow; if fewer thanMsalient frames are found, we add uniformly spaced frames to preserve coverage. All images are resized to 512× 512 for VLM input. b) Optical Flow, OVD, Depth, and Contact.: We com- pute per-frame mean optical-flow magnitude using dense flow [47] to obtain a scene-wide motion saliency score for keyframe proposal. We use GroundingDINO [44] (Swin- T backbone) for open-vocabulary detection, capped at five detections per keyframe. For monocular depth, we use Depth- Anything-V2-Large [45]; to reduce extreme outliers, we suppress depth values beyond the 0.8 quantile bound. Contact- transition tokens use IoU and nearest-center trends across adjacent keyframes, as described in Section I. c) Contact-Transition Thresholds.: For the contact- transition proxy, we setτ IoU = 0.1for bounding-box overlap and τ d = 0.15 for nearest-center distance. d) Pseudo-BEV.: Pseudo-BEV schematics are rendered on a256× 256white canvas withX/Zaxes, projected semantic dots for tracked objects, confidence-scaled circle radii clipped to[r min = 3, r max = 10]pixels, class labels, and OCR-friendly timestamps. e) VLM Calls.: For each QA, we provide2× Mimages: the RGB keyframes and their corresponding pseudo-BEVs. The text prompt consists of the KITE prefix, a short instruction explaining that the pseudo-BEV is a schematic top-down layout used for relative spatial reasoning, and the target question. f) QLoRA.: To study how well the proposed evidence representation transfers under lightweight adaptation, we also fine-tune the VLM with QLoRA [48]. We use rank 8, 4-bit quantization, one epoch, and a learning rate of1× 10 −5 , with the LLM backbone and merger parameters unfrozen. All training and evaluation are run on a single NVIDIA A6000 GPU. V. LIMITATIONS AND FUTURE WORK KITE deliberately favors compact, interpretable evidence over full geometric fidelity. It relies on open-vocabulary detection and monocular relative depth, which can struggle with small, occluded, reflective, or visually ambiguous objects. Its contact-transition proxy captures coarse interaction trends rather than precise force events. Likewise, the current scene graph is intentionally lightweight: it uses a small set of coarse relations and omits potentially useful predicates such as ON_TOP_OF and INSIDE, so the system is better suited to diagnosing high-level spatial inconsistencies than precise geometric deviations. The pseudo-BEV is non-metric and flattens vertical structure, which further limits low-level geometric analysis. The keyframe selector is also a simplification. Motion saliency can miss low-motion or very brief failures, and identity tracking across sparse keyframes can switch in cluttered scenes. More broadly, our quantitative evaluation is centered on RoboFAC, while the DART and ALOHA-2 studies are qualitative; broader cross-benchmark evaluation and user-facing assessment of explanation quality remain important future directions. Finally, results still depend on the reasoning quality of the chosen VLM backend. Future work can therefore explore stronger perception modules, richer relation vocabularies, multi-view layout cues, adaptive keyframe policies, and broader human-centered evaluation. VI. CONCLUSION We introduced KITE, a training-free, keyframe-indexed, pseudo-BEV-grounded front-end that converts long robot- execution videos into compact, interpretable tokenized evi- dence for VLMs. By combining object-overlaid keyframe RGBs, schematic layout cues, robot-profile information, Object DetectionsOptical FlowPseudo-BEVDepth Estimation === ROBOT PROFILE === Robot name: Aloha Stationary. Morphology: dual-arm. Arms: 2. Grippers: 2. End-effector: parallel_jaws. Sensors: RGB-D camera. Workspace: tabletop. === PREDICTED PLAN STEPS === 1. The left robotic arm picks up the spoon from the black box. 2. The left robotic arm moves the spoon away from the black box. 3. The left robotic arm places the spoon on the table. === Global Scene Graphand Tracks === "timeline": "keyframes_sec": [0.13, 2.27, 3.8, 4.27, 12.27], "global_scene": "tracks": ["track_id": 0, "cls": "robot arm", "mean_pos": [-0.0, -0.0, 0.0], "time_span": [0.13, 4.27], "track_id": 1, "cls": "fork", "mean_pos": [0.6, 0.89, 5.24], "time_span": [0.13, 0.13], ...], "relations": ["subj": "robot arm", "rel": "left_of", "obj": "fork", ...] KITE Structured Context Q: You are given key frames from a task execution plus structured context. Identify up to 3 candidate failure frames (by its frame number 0, 1, 2, ...) where failure likely occurred. Confidence in [0,1]. Return STRICT JSON only: "candidates":["frame_num": int, "confidence": float]. Candidates: 0:0.13s, 1:2.27s, 2:3.80s, 3:4.27s, 4:12.27s A: [ "frame_num": 4, "confidence": 0.9 ] Failure Locating The robot, named Aloha Stationary, is a dual-arm robot equipped with parallel jaw grippers and an RGB-D camera. Its intended task involves manipulating a spoon placed on a black box. The robot's plan includes picking up the spoon from the black box, moving it away, and placing it on the table. In the video, the left robotic arm successfully picks up the spoon from the black box, lifts it slightly, and then places it back onto the black box. The right robotic arm remains stationary throughout the sequence. The robot did not move the spoon away from the black box as per the intended plan. This could be due to a lack of coordination between the arms or an error in the programming. The robot's performance can be improved by ensuring better synchronization between the arms and refining the control algorithms for more accurate movements. Narrative Summary Fig. 4: Qualitative results in real-world (ALOHA-2). Each panel shows: RGB keyframe with object-detection overlays; optical-flow estimates; pseudo-BEV schematic (consistent object IDs; circle radius∝confidence; timestamp); and single-view depth estimates, all for the corresponding keyframes. We also illustrate a short structured-context excerpt, KITE’s response to a failure-localization query, and a final narrative summary. contact-transition tokens, and serialized scene relations, KITE provides a structured interface for failure detection, identifica- tion, localization, explanation, and correction. On RoboFAC, KITE substantially improves a strong vanilla VLM baseline in the training-free setting and remains competitive with task- tuned alternatives, while a lightweight QLoRA adaptation yields further gains. Qualitative results on DART and ALOHA- 2 suggest that the representation transfers beyond a single benchmark and extends naturally to dual-arm real-world failures. REFERENCES [1]A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al., “Do as i can, not as i say: Grounding language in robotic affordances,” in Conference on robot learning. PMLR, 2023, p. 287–318. [2]D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al., “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378, 2023. [3]S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al., “A generalist agent,” arXiv preprint arXiv:2205.06175, 2022. [4]Y. Hu, Q. Xie, V. Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, Z. Zhao, et al., “Toward general-purpose robots via foundation models: A survey and meta-analysis,” arXiv preprint arXiv:2312.08782, 2023. [5]R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu, S. Song, A. Kapoor, K. Hausman, et al., “Foundation models in robotics: Applications, challenges, and the future,” arXiv preprint arXiv:2312.07843, 2023. [6]F. Liu, K. Fang, P. Abbeel, and S. Levine, “Moka: Open-vocabulary robotic manipulation through mark-based visual prompting,” arXiv preprint arXiv:2403.03174, 2024. [7]H. Huang, F. Lin, Y. Hu, S. Wang, and Y. Gao, “Copa: General robotic manipulation through spatial constraints of parts with foundation models,” arXiv preprint arXiv:2403.08248, 2024. [8]W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei, “Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,” arXiv preprint arXiv:2409.01652, 2024. [9]Z. Liu, A. Bahety, and S. Song, “Reflect: Summarizing robot experi- ences for failure explanation and correction,” in CoRL, 2023. [10]J. Duan, W. Pumacay, N. Kumar, Y. R. Wang, S. Tian, W. Yuan, R. Krishna, D. Fox, A. Mandlekar, and Y. Guo, “Aha: A vision- language-model for detecting and reasoning over failures in robotic manipulation,” in ICLR, 2025. [11]W. Lu, M. Ye, Z. Ye, R. Tao, S. Yang, and B. Zhao, “Robofac: A comprehensive framework for robotic failure analysis and correction,” 2025. [Online]. Available: https://arxiv.org/abs/2505.12224 [12]RealMan Robotics, “Compound robot - realman robotics,” https://w. realman-robotics.com/compound-robot, 2024, accessed: 2024-09-07. [13]A. . Team, J. Aldaco, T. Armstrong, R. Baruch, J. Bingham, S. Chan, K. Draper, D. Dwibedi, C. Finn, P. Florence, S. Goodrich, W. Gramlich, T. Hage, A. Herzog, J. Hoech, T. Nguyen, I. Storz, B. Taban- pour, L. Takayama, J. Tompson, A. Wahid, T. Wahrburg, S. Xu, S. Yaroshenko, K. Zakka, and T. Z. Zhao, “Aloha 2: An enhanced low-cost hardware for bimanual teleoperation,” 2024. [14]F. Zeng, W. Gan, Y. Wang, N. Liu, and P. S. Yu, “Large language models for robotics: A survey,” arXiv preprint arXiv:2311.07226, 2023. [15]C. Zhang, J. Chen, J. Li, Y. Peng, and Z. Mao, “Large language models for human–robot interaction: A review,” Biomimetic Intelligence and Robotics, vol. 3, no. 4, p. 100131, 2023. [16] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [17]H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. [18] O. team, “Gpt-4o system card,” 2024. [Online]. Available: https://arxiv.org/abs/2410.21276 [19]H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” 2023. [20]H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-01-30-llava-next [21]M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024. [22]G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023. [23]J. Duan, S. Yu, H. L. Tan, H. Zhu, and C. Tan, “A survey of embodied ai: From simulators to research tasks,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 6, no. 2, p. 230–244, 2022. [24] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, p. 27 730–27 744, 2022. [25] M. Crosby, M. Rovatsos, and R. Petrick, “Automated agent decom- position for classical planning,” in Proceedings of the International Conference on Automated Planning and Scheduling, vol. 23, 2013, p. 46–54. [26]B. Xu, Z. Peng, B. Lei, S. Mukherjee, Y. Liu, and D. Xu, “Rewoo: Decoupling reasoning from observations for efficient augmented language models,” arXiv preprint arXiv:2305.18323, 2023. [27]T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large lan- guage models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, p. 22 199–22 213, 2022. [28]D. Das, S. Banerjee, and S. Chernova, “Explainable ai for robot failures: Generating explanations that improve user assistance in fault recovery,” in Proceedings of the 2021 ACM/IEEE international conference on human-robot interaction, 2021, p. 351–360. [29]S. Rosenthal, S. P. Selvaraj, and M. M. Veloso, “Verbalization: Narration of autonomous robot experience.” in IJCAI, vol. 16, 2016, p. 862–868. [30]S. Ye, G. Neville, M. Schrum, M. Gombolay, S. Chernova, and A. Howard, “Human trust after robot mistakes: Study of the effects of different forms of robot communication,” in 2019 28th IEEE Interna- tional Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 2019, p. 1–7. [31]P. Khanna, E. Yadollahi, M. Björkman, I. Leite, and C. Smith, “User study exploring the role of explanation of failures by robots in human robot collaboration tasks,” arXiv preprint arXiv:2303.16010, 2023. [32] J. Arkin, D. Park, S. Roy, M. R. Walter, N. Roy, T. M. Howard, and R. Paul, “Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions,” The International Journal of Robotics Research, vol. 39, no. 10-11, p. 1279–1304, 2020. [33]A. Bucker, L. Figueredo, S. Haddadin, A. Kapoor, S. Ma, S. Vemprala, and R. Bonatti, “Latte: Language trajectory transformer,” arXiv preprint arXiv:2208.02918, 2022. [34] S. S. Raman, V. Cohen, I. Idrees, E. Rosen, R. Mooney, S. Tellex, and D. Paulius, “Cape: Corrective actions from precondition errors using large language models,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, p. 14 070–14 077. [35]Z. Wang, B. Liang, V. Dhat, Z. Brumbaugh, N. Walker, R. Krishna, and M. Cakmak, “I can tell what i am doing: Toward real-world natural language grounding of robot experiences,” arXiv preprint arXiv:2411.12960, 2024. [36]C. DeChant, I. Akinola, and D. Bauer, “Learning to summarize and answer questions about a virtual robot’s past actions,” Autonomous robots, vol. 47, no. 8, p. 1103–1118, 2023. [37]Y. Du, K. Konyushkova, M. Denil, A. Raju, J. Landon, F. Hill, N. de Freitas, and S. Cabi, “Vision-language models as success detectors,” arXiv preprint arXiv:2303.07280, 2023. [38]Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang, “Vip: Towards universal visual reward and representation via value-implicit pre-training,” arXiv preprint arXiv:2210.00030, 2022. [39]H. Ha, P. Florence, and S. Song, “Scaling up and distilling down: Language-guided robot skill acquisition,” in Conference on Robot Learning. PMLR, 2023, p. 3766–3777. [40]L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang, “Gensim: Generating robotic simulation tasks via large language models,” arXiv preprint arXiv:2310.01361, 2023. [41]K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suender- hauf, “Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,” arXiv preprint arXiv:2307.06135, 2023. [42] T. Choudhary, V. Dewangan, S. Chandhok, S. Priyadarshan, A. Jain, A. K. Singh, S. Srivastava, K. M. Jatavallabhula, and K. M. Krishna, “Talk2bev: Language-enhanced bird’s-eye view maps for autonomous driving,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, p. 16 345–16 352. [43]C. Deng, S. Chen, D. Chen, Y. He, and Q. Wu, “Sketch, ground, and refine: Top-down dense video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, p. 234–243. [44]S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al., “Grounding dino: Marrying dino with grounded pre- training for open-set object detection,” arXiv preprint arXiv:2303.05499, 2023. [45]L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” arXiv:2406.09414, 2024. [46]S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025. [47]G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Scandinavian conference on Image analysis.Springer, 2003, p. 363–370. [48]T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Ef- ficient finetuning of quantized llms,” arXiv preprint arXiv:2305.14314, 2023.