Paper deep dive
STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration
Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu, Songpo Li, Soshi Iba
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/28/2026, 4:54:23 AM
Summary
The paper introduces STEP (State-aware Task Estimator and Planner), a framework for human-robot collaboration that uses Multi-modal Large Language Models (MM-LLMs) to estimate system states and predict action sequences. Unlike previous methods that rely on natural language plans which can be ambiguous or hallucinated, STEP explicitly estimates structured state representations (JSON) and propagates them to forecast future states. This allows for quantitative measurement of progress toward a goal and the generation of precise assistance parameters. Evaluated on a simulated block assembly task, STEP outperforms state-of-the-art baselines in action executability and final-state error.
Entities (6)
Relation Signals (5)
STEP → evaluatedon → Block Assembly
confidence 95% · We evaluate STEP in a simulated environment using a robot assembly task.
STEP → uses → MM-LLM
confidence 95% · STEP prompts a MM-LLM to explicitly estimate the state of the system
STEP → employs → State Representation
confidence 92% · STEP employs an MM-LLM for explicitly estimating and predicting a structured state representation
STEP → outperforms → AntGPT
confidence 90% · Our approach outperforms the state-of-the-art by 32.8% in action executability
MM-LLM → lacks → State tracking
confidence 85% · MM-LLMs inherently lack an understanding of system states and do not track state transitions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8% in action executability and 14.8% in final-state error.
Tags
Links
- Source: https://arxiv.org/abs/2608.27225v1
- Canonical: https://arxiv.org/abs/2608.27225v1
Trouble viewing inline? Open PDF directly →
Full Text
45,719 characters extracted from source content.
Expand or collapse full text
STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration Maitrey Gramopadhye Affiliation: University of North Carolina at Chapel Hill, United States Prakash Baskaran Affiliation: Honda Research Institute, San Jose, CA, USA Xiao Liu Affiliation: Honda Research Institute, San Jose, CA, USA Songpo Li Affiliation: Honda Research Institute, San Jose, CA, USA Soshi Iba †thanks: This work was done during an internship at Honda Research Institute†thanks: Correspondence email: maitrey@cs.unc.edu Affiliation: Honda Research Institute, San Jose, CA, USA Abstract Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (M-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, M-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a M-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8%32.8\% in action executability and 14.8%14.8\% in final-state error. Fig. 1: STEP with robot system setup. Predicting structured state representations along with future actions allows STEP to quantitatively measure the distance of a state from the goal and deduce assistance parameters required for executing the predicted actions. I Introduction As robots have become more affordable and versatile, their applications have expanded, particularly in industries requiring greater human interaction and generalization. Human-robot collaboration improves efficiency by drawing on their respective strengths. However, effective teamwork requires shared control and mutual understanding. For seamless collaboration, robots must be able to infer human intentions and take over some of the workload. Relieving control over to a robot can enable a person to supervise multiple robots or take on a creative role by developing a vision and delegating execution to the robot [1]. However, several tasks in which robot assistance would help are intrinsically human and designed for human-centric environments, and training robots to infer human goals and generate actionable plans requires extensive data collection, which remains a significant challenge [2, 3, 4]. An alternative approach that has recently gained popularity is the use of M-LLMs for long-term action anticipation (LTA) from observations [5, 6, 7, 8]. Prior work has explored the in-context performance of M-LLMs to infer a human’s goal and predict the future actions needed to achieve it, based on the sequence of actions performed by the human [9]. However, directly querying an M-LLM for planning robot actions has several limitations. Since M-LLMs lack real-world interaction during pre-training, they struggle to track the effects of their planned actions [10, 11, 12]. As a result, an agent that executes the generated actions often deviates from the intended goal state. To address this, recent work has proposed explicitly querying M-LLMs to track the state of the system, allowing them to measure progress toward the goal [13, 14, 15, 16, 17]. However, these approaches either rely on assumptions such as the ability to comprehensively describe system states in natural language or affordably query the environment for state information; or they constrain their focus by treating state tracking separately from task planning. In this paper, we extend prior research on LTA by relaxing some of the assumptions and constraints to better match a real-world industrial setting. Another key challenge in applying M-LLMs to robotics is handling the high dimensionality of the action space. Since M-LLMs fail to consider the nuances of real-world execution, several previous works limit their output to a cursory action plan of “verb, noun” pairs (e.g., “pick, cup”, “pour from, cup”, “place on, table”), while relying on predefined functions to abstract away the other parameters [11, 14, 18, 19, 20]. However, this often leads to ambiguity during action execution. For instance, executing “place on, table” with a cup requires determining a placement location on the table, while accounting for other objects, and the orientation of the cup after placement (upright, upside down, etc.). Some works mitigate this by incorporating feedback, either from humans in the loop or by querying the environment, and re-planning for any failures. In this paper, we propose STEP to explicitly estimate and propagate a structured representation of the environment state. Specifically, we consider an ubiquitous industrial scenario in which an operator is assembling a structure on a workstation and wishes to hand over the planning for building the structure to a robot. Our mock task serves as a proxy for real-world industrial tasks, such as machine assembly or modular fixture construction, where human-robot collaboration would be beneficial. As shown in Fig. 1, we use the actions already performed by the operator and an image of the workstation, along with a small curated set of in-context examples, to prompt an M-LLM to estimate - the overall structure the operator is trying to build (task estimation) (See §I-A) and a structured state representation of the workstation in a JSON format (state representation generation) (See §I-B), followed by the remaining actions required to construct the structure (action prediction) (See §I-C) and the future states of the environment after executing the proposed actions (state propagation) (See §I-D), without requiring any interaction with the environment. Propagating a structured representation of the environment allows us to predict assistance parameters required to complete the predicted actions that could not be expressed as a “verb, noun” pair. It also allows us to track the distance from the inferred goal quantitatively. We use this distance metric to truncate and regenerate the future action plan and states iteratively, to ensure progress towards the inferred goal (rollouts) (See §I-E). STEP employs an M-LLM for explicitly estimating and predicting a structured state representation to support robotic action planning in an industrial setting. Such industrial settings (1) offer a valuable opportunity for human–robot collaboration, (2) are inherently structured, making our choice of a structured state representation well-suited, and (3) remain underrepresented in existing M-LLM pre-training datasets, posing unique challenges for model generalization. We evaluated our approach on a dataset from a simulated block assembly scenario, in which human operators teleoperate two Franka robot arms situated on either side of a table (See §IV-A) [21]. The goal of the assembly scenario is to construct a structure using five identical wooden blocks. We use several metrics (See §IV-B), including executability, longest common subsequence (LCS), and final-state error to test our generated action plans. Overall, our method increased action executability by 32.8%32.8\% and decreased final-state error by 14.8%14.8\%, compared to a state-of-the-art baseline (See §V). Fig. 2: After a hand-over by the operator, the overall task and current system state are estimated. They are then used to iteratively rollout next actions and states multiple times. Each rollout selects the closest state to the goal to guide actions towards the estimated task. I Related Work STEP is inspired by papers on shared autonomy and LTA using LLMs. We address some of their limitations by extending prior work on tracking a state representation of the environment for LTA. I-A Shared autonomy in teleoperation The idea of shared teleoperation has gained popularity in industry as a way to increase production while empowering humans to assume a supervisory role [1]. Previous work has approached shared teleoperation by having a semi-autonomous robot provide assistance to an operator [22, 23, 24]. Usually, the robot starts as non-autonomous and attempts to infer the user’s goal [25, 26, 27, 28], or form a reasonable estimate of it [29], by observing their actions. The robot then provides assistance to complete the task by taking control [30] or providing visual, haptic, or multi-modal cues [31, 32, 33, 24]. In this paper, we follow a similar pipeline, by taking as input the actions performed by the user as a sequence of natural language actions (e.g. “Pick block 1”, “Stand block 1”, “Withdraw”) and predicting the user’s goal, followed by the actions required to complete the goal. I-B Large language models and long-term action anticipation Large Language Models (LLMs), and their images + text counterparts, M-LLMs, have shown great performance on diverse tasks [34, 35, 36, 37, 38, 39, 40]. Particularly useful is their in-context learning ability, where LLMs used as frozen models can generalize across domains when few domain-specific examples are provided during inference [41, 42]. Several studies have proposed using in-context learning to ground LLM output in robot-actionable steps [11, 18, 19, 20]. In-context learning is especially useful in robot task planning for industry, as LLMs are often not trained on data that is relevant for these scenarios. Like shared autonomy planning, LTA involves forecasting an agent’s future actions from an initial sequence of observed behavior [5, 6, 7, 8]. Previous research has explored converting a video into a sequence of natural language actions and using that as input to LLMs to predict future actions. AntGPT [9] proposes a method in which an LLM predicts the overall goal of an agent from a sequence of natural language actions and then generates the remaining actions required to achieve that goal. VidAssist [43] proposes to take as input the initial sequence of actions and the overall goal to iteratively build a tree of possible future action sequences and find the sequence that will best satisfy the given goal. Inspired by these approaches, our prediction pipeline further extends this framework by also predicting and propagating the state of the environment. I-C State estimation using language models LLMs often struggle with planning and reasoning tasks. Being trained for auto-regressive sequence-to-sequence translation, LLMs lack any motivation to keep track of the state of a system [12]. As a result, LLMs often hallucinate actions that diverge from the intended goal. To overcome this limitation, previous research has explored using chain-of-thought reasoning to explicitly track the state of the system with LLMs [44, 45]. RAFA [13] proposes iteratively building a tree of action and state sequences, followed by utilizing these sequences to query an LLM to select the best action to perform. However, RAFA requires access to the initial ground-truth state of the system to begin planning. Similar to RAFA, LLM-State [14] and Kaige et al., [15] also output the future state representations in natural language to guide an agent’s actions. However, a natural language state representation is often not expressive enough to represent the state of a robot workspace and only allows prior works to get a vague guidance towards the goal state. In our method, we query an M-LLM to estimate the environment state and propagate it as a structured JSON object, allowing us to calculate a quantitative distance from the goal state. Statler [16] proposes querying an LLM to infer and update the state of a system as a structured representation. However, Statler does not consider the problem of action planning and only propagates the state for one action. Wall-E [17] proposes to predict future actions and propagate a state representation. However, it needs the initial ground-truth state of the system as input and an exploratory phase to learn the rules about the world, which might not be feasible in the real world. I Approach STEP takes as input the actions performed by an operator and an image of the workstation, along with a set of example interactions to prompt an M-LLM. As illustrated in Fig. 2, our pipeline predicts the overall task the operator is trying to accomplish. Simultaneously, it also estimates a structured state representation of the current workspace from its image. It then uses the estimated task and action history to predict future actions required to complete the task. The current state is then propagated to estimate the future states of the workstation. Using the structured future state representations, it calculates their distance from the goal state of the estimated task and truncates the future actions at the state with the least distance. STEP also repeats the action prediction and state propagation steps for multiple rollouts to iteratively get closer to completing the task. I-A Task Estimation Following prior work [9], we employ a top-down approach by first estimating the overall task (T) that the operator wants to complete. STEP takes as input the actions performed by the operator so far (A1,A2,…,AkA_1,A_2,...,A_k) and the image of the current workstation (I) to prompt an M-LLM for the task. In the prompt, we also include examples (of the format (actions, <image>) -> Task) from a small pre-collected dataset, and a list of all possible tasks (see §IV-A) to guide and constrain the M-LLM output. I-B State Representation Generation We prompt an M-LLM with the image of the workstation (I) for a structured representation of the state (S). We follow a chain-of-thought approach to query the model multiple times, each time asking it to describe various parts of the workspace layout. First, we query it to get the locations (top right, top left, center right, etc.) of each of the 5 blocks in the image. These locations of the blocks in the image help us ground our subsequent prompts in the image, where we query the model to provide the following information for each block: (i) the block’s placement location (including the number of the block it is placed on, if any) (i) the block’s orientation (i.e., standing, lying face down, or lying on side) (i) the block’s top face orientation (facing parallel or perpendicular to the camera) (iv) If the block is on another block, describe its location relative to the base block. We also include a brief description of the workstation and example images of the workstation in the prompts. In the end, we combine the collected information and ask an M-LLM to format it as a JSON object (See Fig. 3 for more details). Fig. 3: For each of the blocks our structured state representation records the block location (table/another block with its number), block orientation (stand/lie on face/lie on side/NA), top face orientation (long-side parallel/perpendicular to camera/NA), specific location (left/right/center if the block is on another block, otherwise NA). The modular state design also allows extension to other scenarios by adding descriptors - ‘nearby objects’, ‘facing’, ‘containing objects’ etc. I-C Action Prediction We use the estimated task (T) and the action history (A1,A2,…,AkA_1,A_2,...,A_k) to prompt the M-LLM for the future actions (Ak+1,Ak+2,…,AnA_k+1,A_k+2,...,A_n) that need to be performed to complete the task. We also include examples (of the format - performed actions -> future actions) in the prompt, taken from the pre-collected dataset for completing the estimated task. During our experiments, we discovered that including multi-modal data in the prompt for this stage degraded the performance, so we did not use the state or image information for action prediction. I-D State Propagation We propagate the estimated current state of the workstation (S) to track the changes resulting from executing the predicted actions. We prompt the M-LLM using A1,A2,…,AkA_1,A_2,...,A_k, S and Ak+1A_k+1, to get as output the next state (Sk+1S_k+1), also as a JSON object. We then iteratively set the propagated state as the current state and predict the states resulting from executing all predicted actions (Sk+1,Sk+2,…,SnS_k+1,S_k+2,...,S_n). For example, getting Sk+2S_k+2 would involve - A1,A2,…,Ak,Ak+1A_1,A_2,...,A_k,A_k+1, Sk+1S_k+1, Ak+2A_k+2 -> Sk+2S_k+2. Examples of the same format relevant to the estimated task are also included in the prompt. During our experiments, we found that using a chain-of-thought approach to break down the state propagation into two parts improved performance. For propagating the state at each action, we first ask the M-LLM to describe the changes in the state in natural language, and then use the description to get a JSON representation of the next state. I-E Rollouts Using a structured state representation of the workstation allows us to manually calculate its distance (dTd_T) from the estimated task completion state. We calculate the distance as the total number of differences between the state and the task completion state. Since our state representation has 5 blocks and each block has 4 different characteristics, dTd_T can range from 00 to 2020 (See Fig. 3). For every rollout, we query the M-LLM for the future actions and states. We then calculate dTd_T for all states (current and predicted), and truncate the predicted actions and states after the state with the first instance of the lowest dTd_T (e.g. SlS_l could be the state with the lowest dTd_T). For the next rollout, we set SlS_l as the current state and use A1,…,Ak,…,AlA_1,...,A_k,...,A_l and SlS_l for action prediction and state propagation. Since SlS_l has a dTd_T equal to or lower than S, with each rollout STEP identifies a state closer to the inferred goal than the state at the start of the rollout, and helps guide the action plan towards the goal. We repeat the rollouts for max_rolloutmax\_rollout times, which is a hyperparameter, or until we reach a state with dT=0d_T=0. After the final rollout, we return the predicted action sequence (Ak+1,…,AlA_k+1,...,A_l) and state sequence (S,Sk+1,…,SlS,S_k+1,...,S_l), where dTd_T of SlS_l is the lowest among all states. IV Evaluation We compared our method with a state-of-the-art LTA baseline inspired by the top-down approach proposed by AntGPT [9]. The baseline consisted of using the same inputs as our method, for task estimation (§I-A) followed by a single rollout of action prediction (§I-C), with no state information. We selected this comparison to directly examine the impact of explicitly estimating and tracking the system state on action prediction accuracy. To isolate and study this effect, we deliberately excluded potential confounding factors such as interactive planning or human-in-the-loop feedback. Also, several related works we examined impose constraints such as requiring access to the initial ground-truth state of the system for planning, an assumption that is often unrealistic in real-world industrial human–robot collaboration scenarios. Moreover, many approaches address only specific subcomponents of the overall action planning problem rather than the complete pipeline. Considering these factors, we identified AntGPT [9] as a suitable inspiration for our baseline. IV-A Experimental Setup and Dataset We evaluated our method on a dataset of teleoperated manipulation sequences collected by Baskaran et al. [21]. The collected manipulation sequences involved users teleoperating two robots in real time within a simulated environment using HTC Vive Pro controllers [46]. The simulated workstation composed of a table mounted with two Franka Emika robot arms [47] on either side, and five identical wooden block assembly pieces. The collected dataset had 495 instances from 19 users, where each instance involved operators accomplishing the goal of assembling the blocks into one of the 88 different structures shown in Fig. 4 - Tuning fork, Low base tuning fork, Bridge, Arch, Snake, Horse, Frame and Stacking. Please see Baskaran et al. [21] for more details on the dataset collection procedure. We manually annotated the dataset instances to add labels and post-processed them to get the following features per instance: Task (T), Recorded actions (A1,A2,…,AnA_1,A_2,...,A_n), Images of the workspace (I0,I1,…,InI_0,I_1,...,I_n), States of the workspace (S0,S1,…,SnS_0,S_1,...,S_n) and Possible tasks (Tp0,Tp1,…,TpnTp_0,Tp_1,...,Tp_n). T denotes the goal structure. A recorded action AiA_i is of the format <Left robot action (AilA_il), Right robot action (AirA_ir)>. AilA_il and AirA_ir can be one of the nine actions - Pick block BxB_x, Stand block BxB_x, Lie block BxB_x, Side-lie block BxB_x, Stand-on-block block BxB_x, Lie-on-block block BxB_x, Side-lie-on-block block BxB_x, Idle, and Withdraw. An image IiI_i is taken from the video where AiA_i ends and Ai+1A_i+1 begins. We also added block numbers to the images in post-processing to distinguish the blocks (See Fig. 2). SiS_i is the structured state representation of IiI_i. TpiTp_i refers to all possible structures that share SiS_i as an intermediate state, as some tasks may overlap with others. For example, the Arch task overlaps with the Bridge and Horse tasks (see Fig. 4). For our evaluation we used the initial part of the recorded actions (A1,A2,…,AkA_1,A_2,...,A_k) and the image of the workstation after completing those actions (IkI_k), to predict the remaining actions required to complete the task (T). Fig. 4: The eight block assembly tasks. IV-B Metrics To calculate quantitative metrics for the predicted actions, we first manually propagate the ground-truth current state of the workstation using hand-crafted rules. Our rules take as input the predicted actions, and the predicted states which help to determine assistance parameters to remove ambiguity from action execution. The baseline (See §IV) does not predict future states and thus suffers from ambiguity in execution. So we randomly select a state from the set of possible states that can result from executing the actions predicted by the baseline. Note that here we select a state only from those reachable by the predicted actions, not from all possible states. Following existing research [18, 19], we show results across three metrics: executability, final-state error, and longest common subsequence. We also provide the task correctness and current state error to measure the correctness in predicting the overall task and the error in inferring the current state of the workspace. Executability ensures that the predicted actions follow a logical order and satisfy execution constraints (e.g., proper object poses, action preconditions, etc.). We perform checks for executability and truncate the predicted action sequence at the first action that fails the checks. We report executability as the percentage of predicted actions that passed all the checks. Final-state error measures the error of the ground truth task completion state from the final state that could be successfully achieved by executing the predicted actions. For the final achievable state, we select the resulting state from the last predicted action that can be executed successfully. This metric measures the difference between the result and the expected goal. Longest Common Sub-sequence (LCS) is defined as the length of the longest common sub-sequence of actions in the predicted actions and the remaining recorded actions from our dataset, divided by the length of the longer sequence between predicted actions and remaining recorded actions. Following prior work [18, 19, 48], we allowed gaps to exist between the actions in the common sub-sequence, provided their order was the same. We report LCS as a percentage. A high LCS can mean that the predicted actions are relevant to the task and have short-term order. However, it does not provide a complete picture of correctness on its own, as there can be multiple correct action sequences with varying LCS values. A high LCS also does not guarantee that the predicted actions will execute successfully. Task correctness measures the accuracy of task estimation (See §I-A). It is assigned 11 if the predicted task matches any of the possible tasks at inference time; otherwise, it is set to 00. Current state error compares the distance between the current state output of state representation generation (See §I-B) and the ground-truth state at the time of inference. V Results & Discussion TABLE I: Comparison of executability, final error, and LCS of STEP with the baseline, along with task correctness and current state error for multiple task completion percentages. We used the GPT-4o [49] model offered by the OpenAI API [50] for our main results. Every time we prompt the M-LLM, we sample 55 outputs and pick the output with the maximum log-probability returned by the model. V-A Comparison with Baseline To model variations in user delegations to STEP, we conduct experiments at multiple task completion percentages of 30%, 50%, 70%, and 90%. A task completion of x% signifies that the initial x% of the actions from the recorded action sequence were used as input for evaluation. As can be seen in Table I, STEP predicted action plans that were more executable than the baseline, for all percentages of task completion. Also, by predicting future states along with actions, our method resulted in a state that was closer to the intended goal of the operator, when compared to the baseline. However, we noted that the baseline method had a higher LCS than our method. This behavior can be attributed to STEP truncating the predicted actions after the state with the least distance from the goal. As a result, our predicted action sequence was on average approximately 33 actions shorter in length than the baseline’s predictions. We suspect that the extra actions produced by the baseline caused it to have more number of actions in common with the recorded actions, thus increasing the baseline LCS. However, the higher LCS of the baseline resulted in unnecessary actions that were less executable and less correct in the resultant state. Action plans generated by STEP were more efficient, as they had a lower final error compared to the baseline, while being shorter on average. We hypothesize that noise in the recorded actions, caused by operator errors when collecting the data also contributed to LCS being a questionable metric in our evaluation. We noticed that with increasing task completion percentage, both the methods achieved a better final error, since the robot started from a state closer to the goal state, as more of the assembly was completed by the operator. With increasing task completion, the task correctness first declines due to increasing noise in longer input action sequences and more occlusion of the block structures in the images, and then increases as the structure being built becomes more discernible. We suspect that the current state error increased as the task progressed because the structure became more complicated and blocks got closer together in the image. V-B Ablations TABLE I: Ablation of rollouts for 50% task completion. Final error improves with rollouts showing that additional rollouts decrease the distance from goal. Rollouts Exec. % ↑ Final Err. ↓ LCS % ↑ 1 88.98 4.40 31.06 2 85.60 4.11 37.4 3 84.55 4.05 38.82 We ablated the number of rollouts conducted before calculating the metrics. Table I shows the effect of the number of rollouts on performance for 50% task completion. We noticed that each rollout predicted a longer action sequence than the preceding rollout, with action plan length increasing by 0.70.7 actions on average with each rollout. LCS increased with more rollouts, which is explained by the increased number of actions in the action plan. The increasing actions did cause the executability to decrease. However, as seen from the decreasing final error with more rollouts, the added actions that could execute were necessary as they were able to result in a final state closer to the goal with increasing rollouts. TABLE I: Ablating M-LLM to compare executability and final error of STEP with the baseline for 50% task completion. Results for 4545 random dataset instances. Model Exec. % ↑ Final Error ↓ Task Corr. Base STEP Base STEP % ↑ GPT-4o 71.68 78.12 4.53 4.38 62.22 GPT-4o mini 82.60 82.16 4.49 4.93 26.67 GPT-4.1 82.28 92.02 4.80 4.11 71.11 GPT-5.1 69.24 82.70 4.80 3.71 73.33 GPT-5.2 76.64 84.44 4.73 4.49 71.11 We compared STEP with the baseline for various M-LLMs - GPT-4o, GPT-4o mini, GPT-4.1, GPT-5.1, and GPT-5.2, offered by the OpenAI API [50]. These models were selected to cover a range of model sizes and generation paradigms (regular LLMs and reasoning LLMs). To reduce computational cost and runtime, this ablation was conducted on a representative subset of 4545 dataset instances. These instances were randomly sampled from our dataset while ensuring coverage across all eight task categories (see Fig. 4). As seen in Table I, STEP outperformed the baseline across most models in terms of both executability and final error, demonstrating its robustness and generalizability across different M-LLMs. For GPT‑4o mini, however, the baseline achieved slightly better results than STEP. We observed that GPT‑4o mini exhibited a notably low task correctness score, which measures the task estimation stage, the first step in our pipeline, and consequently propagated errors throughout subsequent stages for both methods. As a result, the outputs corresponded to an incorrect task, rendering these results less reliable compared to those from other models. We attribute this behavior to GPT‑4o mini’s relatively smaller model size, which likely limits its task estimation capabilities. TABLE IV: Ablation of task correctness for 50% task completion. All metrics improve with correct task prediction. Improvement in earlier stages improves downstream performance. Task corr. Exec. % ↑ Final Err. ↓ LCS % ↑ 1 84.63 3.64 41.16 0 84.37 4.9 33.45 TABLE V: Ablation of current state error for 50% task completion. Executability and final error improve with no error in the current state estimation. Curr. st. err. Exec. % ↑ Final Err. ↓ LCS % ↑ 0 86.38 3.93 36.08 >>0 84.41 4.06 39.07 We also studied the impact of task and state estimation on action prediction and state propagation. Table IV compares the results for the data instances that had a correct task prediction with the instances with an incorrect task prediction. Similarly, Table V shows the results for instances that did not have any errors in their current state prediction compared to the other instances. On average, we saw an improvement in performance of the predicted actions if the task and current state were predicted correctly. This ablation also shows the modularity of our approach as stages can be worked on independently and an improved performance earlier in the pipeline can reliably be transferred to better results for the final predicted actions. VI Conclusion We introduce STEP, a method to enhance long-term action anticipation with M-LLMs by explicitly modeling state transitions, leading to more accurate and executable robot action plans in collaborative industrial tasks. We construct a multi-staged pipeline composed of first estimating the overall task a robot operator is trying to perform, along with a structured representation of the current state of the robot’s workstation. We then pass this information on to later stages to predict the actions required to complete the task and propagate the state representation to obtain the assistance parameters required to successfully execute the predicted actions for the desired task. We also track the distance between the propagated states and the goal state and guide predicted actions towards the goal. We extensively evaluated our method on a dataset of an assembly scenario with teleoperated robots. Our results highlight our contribution towards bridging some of the gaps in the existing literature by demonstrating improved performance in executability and final error over the existing state-of-the-art. However, we see some areas for improvement. One limitation of STEP is that its current implementation is specific to the block assembly scenario. Industry may extend STEP to other scenarios by using information relevant to the new scenarios, by modifying the prompts, state representation structure, and function implementations. This investment in modifying STEP for a new scenario would only be required once at the start. However, future work can explore ways to generalize STEP to various scenarios with minimal modifications. A potential avenue of research could be using an M-LLM to analyze a scenario and automatically craft the required prompts, state representation structure, and function implementations. Additional work is also required to make STEP real-time before it can be adopted for practical use. In order to propagate states and guide the generation of action plans towards the goal, STEP queries an M-LLM several more times than the baseline. However, this causes STEP to trade off speed for performance accuracy. In our experiments, we observed that the baseline had a shorter run time (M = 18.24, SD = 4.23 sec) than STEP (M = 148.07, SD = 55.03 sec). Future research could investigate hosting an M-LLM locally or finetuning a smaller model to reduce the latency of each query. We hope that our work encourages further inquiries into the potential of M-LLMs in long-horizon reasoning for human-robot collaboration. References [1] A. Pichler, S. C. Akkaladevi, M. Ikeda, M. Hofmann, M. Plasch, C. Wögerer, and G. Fritz (2017) Towards shared autonomy for robotic tasks in manufacturing. Procedia Manufacturing 11, p. 72–82. Cited by: §I, §I-A. [2] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. (2023) Rt-2: vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. Cited by: §I. [3] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. Cited by: §I. [4] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: §I. [5] F. Sener, D. Singhania, and A. Yao (2020) Temporal aggregate representations for long-range video understanding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, p. 154–171. Cited by: §I, §I-B. [6] Y. Abu Farha, A. Richard, and J. Gall (2018) When will you do what?-anticipating temporal occurrences of activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 5343–5352. Cited by: §I, §I-B. [7] Y. Abu Farha, Q. Ke, B. Schiele, and J. Gall (2021) Long-term anticipation of activities with cycle consistency. In Pattern Recognition: 42nd DAGM German Conference, DAGM GCPR 2020, Tübingen, Germany, September 28–October 1, 2020, Proceedings 42, p. 159–173. Cited by: §I, §I-B. [8] H. Gammulle, S. Denman, S. Sridharan, and C. Fookes (2019) Forecasting future action sequences with neural memory networks. arXiv preprint arXiv:1909.09278. Cited by: §I, §I-B. [9] Q. Zhao, S. Wang, C. Zhang, C. Fu, M. Q. Do, N. Agarwal, K. Lee, and C. Sun (2024) AntGPT: can large language models help long-term action anticipation from videos?. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §I, §I-B, §I-A, §IV. [10] E. M. Bender and A. Koller (2020) Climbing towards nlu: on meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 5185–5198. Cited by: §I. [11] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. (2022) Do as i can, not as i say: grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. Cited by: §I, §I, §I-B. [12] S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu (2023) Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992. Cited by: §I, §I-C. [13] Z. Liu, H. Hu, S. Zhang, H. Guo, S. Ke, B. Liu, and Z. Wang (2023) Reason for future, act for now: a principled framework for autonomous llm agents with provable sample efficiency. arXiv preprint arXiv:2309.17382. Cited by: §I, §I-C. [14] S. Chen, A. Xiao, and D. Hsu (2024) Llm-state: open world state representation for long-horizon task planning with large language model. arXiv preprint arXiv:2311.17406. Cited by: §I, §I, §I-C. [15] K. Xie, I. Yang, J. Gunerli, and M. Riedl (2024) Making large language models into world models with precondition and effect knowledge. arXiv preprint arXiv:2409.12278. Cited by: §I, §I-C. [16] T. Yoneda, J. Fang, P. Li, H. Zhang, T. Jiang, S. Lin, B. Picker, D. Yunis, H. Mei, and M. R. Walter (2024) Statler: state-maintaining language models for embodied reasoning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 15083–15091. Cited by: §I, §I-C. [17] S. Zhou, T. Zhou, Y. Yang, G. Long, D. Ye, J. Jiang, and C. Zhang (2024) Wall-e: world alignment by rule learning improves world model-based llm agents. arXiv preprint arXiv:2410.07484. Cited by: §I, §I-C. [18] M. Gramopadhye and D. Szafir (2023) Generating executable action plans with environmentally-aware language models. External Links: 2210.04964 Cited by: §I, §I-B, §IV-B, §IV-B. [19] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch (2022) Language models as zero-shot planners: extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, p. 9118–9147. Cited by: §I, §I-B, §IV-B, §IV-B. [20] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg (2023) Progprompt: generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 11523–11530. Cited by: §I, §I-B. [21] P. Baskaran, X. Liu, S. Li, and S. Iba (2025) EXplainable intention estimation in teleoperated manipulation using deep dynamic graph neural networks. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Cited by: §I, §IV-A. [22] G. Li, Q. Li, C. Yang, Y. Su, Z. Yuan, and X. Wu (2023) The classification and new trends of shared control strategies in telerobotic systems: a survey. IEEE Transactions on Haptics 16 (2), p. 118–133. Cited by: §I-A. [23] M. Selvaggio, M. Cognetti, S. Nikolaidis, S. Ivaldi, and B. Siciliano (2021) Autonomy in physical human-robot interaction: a brief survey. IEEE Robotics and Automation Letters 6 (4), p. 7989–7996. Cited by: §I-A. [24] M. Cai, K. Patel, S. Iba, and S. Li (2024) Hierarchical deep learning for intention estimation of teleoperation manipulation in assembly tasks. arXiv preprint arXiv:2403.19770. Cited by: §I-A. [25] W. Yu, R. Alqasemi, R. Dubey, and N. Pernalete (2005) Telemanipulation assistance based on motion intention recognition. In Proceedings of the 2005 IEEE international conference on robotics and automation, p. 1121–1126. Cited by: §I-A. [26] M. Gao, J. Oberländer, T. Schamm, and J. M. Zöllner (2014) Contextual task-aware shared autonomy for assistive mobile robot teleoperation. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 3311–3318. Cited by: §I-A. [27] M. Li and A. M. Okamura (2003) Recognition of operator motions for real-time assistance using virtual fixtures. In 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings., p. 125–131. Cited by: §I-A. [28] D. Aarno and D. Kragic (2008) Motion intention recognition in robot assisted applications. Robotics and Autonomous Systems 56 (8), p. 692–705. Cited by: §I-A. [29] S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, and J. A. Bagnell (2018) Shared autonomy via hindsight optimization for teleoperation and teaming. The International Journal of Robotics Research 37 (7), p. 717–742. Cited by: §I-A. [30] T. Lin, A. U. Krishnan, and Z. Li (2020) Shared autonomous interface for reducing physical effort in robot teleoperation via human motion mapping. In 2020 IEEE International Conference on Robotics and Automation (ICRA), p. 9157–9163. Cited by: §I-A. [31] S. Manschitz and D. Ruiken (2022) Shared autonomy for intuitive teleoperation. In ICRA Workshop: Shared Autonomy in Physical Human-Robot Interaction: Adaptability and Trust, Cited by: §I-A. [32] D. Zhang, R. Tron, and R. P. Khurshid (2021) Haptic feedback improves human-robot agreement and user satisfaction in shared-autonomy teleoperation. In 2021 ieee international conference on robotics and automation (icra), p. 3306–3312. Cited by: §I-A. [33] C. Wang, S. Hasler, D. Tanneberg, F. Ocker, F. Joublin, A. Ceravola, J. Deigmoeller, and M. Gienger (2024) LaMI: large language models for multi-modal human-robot interaction. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, p. 1–10. Cited by: §I-A. [34] J. Davison, J. Feldman, and A. Rush (2019) Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, p. 1173–1178. External Links: Link, Document Cited by: §I-B. [35] Z. Jiang, F. F. Xu, J. Araki, and G. Neubig (2019) How can we know what language models know?. CoRR abs/1911.12543. External Links: Link, 1911.12543 Cited by: §I-B. [36] F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel (2019) Language Models as Knowledge Bases?. ArXiv abs/1909.01066. Cited by: §I-B. [37] G. Ilharco, R. Zellers, A. Farhadi, and H. Hajishirzi (2020) Probing Text Models for Common Ground with Visual Representations. ArXiv abs/2005.00619. Cited by: §I-B. [38] K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training Verifiers to Solve Math Word Problems. ArXiv abs/2110.14168. Cited by: §I-B. [39] J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu (2021) Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034. Cited by: §I-B. [40] K. Lu, A. Grover, P. Abbeel, and I. Mordatch (2021) Pretrained Transformers as Universal Computation Engines. arXiv preprint arXiv:2103.05247. Cited by: §I-B. [41] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §I-B. [42] M. Tsimpoukelli, J. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill (2021) Multimodal few-shot learning with frozen language models. Proc. Neural Information Processing Systems. Cited by: §I-B. [43] M. M. Islam, T. Nagarajan, H. Wang, F. Chu, K. Kitani, G. Bertasius, and X. Yang (2024) Propose, assess, search: harnessing llms for goal-oriented planning in instructional videos. In European Conference on Computer Vision, p. 436–452. Cited by: §I-B. [44] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: §I-C. [45] Y. Mu, Q. Zhang, M. Hu, W. Wang, M. Ding, J. Jin, B. Wang, J. Dai, Y. Qiao, and P. Luo (2023) Embodiedgpt: vision-language pre-training via embodied chain of thought. arXiv preprint arXiv:2305.15021. Cited by: §I-C. [46] H. Corporation (2025) HTC vive pro controllers. Note: https://w.vive.com/ Cited by: §IV-A. [47] F. E. GmbH. (2021) Franka emika robot: instruction handbook. Cited by: §IV-A. [48] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba (2018) Virtualhome: simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 8494–8502. Cited by: §IV-B. [49] OpenAI (2024) Hello GPT-4o. External Links: Link Cited by: §V. [50] G. Brockman, P. Welinder, M. Murati, and OpenAI (2020) OpenAI: openai api. External Links: Link Cited by: §V-B, §V.