Paper deep dive

LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music

Snehasis Banerjee, Ranjan Dasgupta

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 10

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%

Last extracted: 7/5/2026, 5:48:07 AM

Summary

This paper introduces a novel framework for human-robot interaction (HRI) that uses Large Language Models (LLMs) to synthesize complex robotic actions from multimodal inputs: natural speech, hand gestures, and music/sound beats. The system architecture consists of three stages: multimodal input processing (using Whisper for speech, a gesture recognition module, and a beat detection pipeline), LLM-based action reasoning (using Qwen3:0.6b via Ollama to map inputs to a predefined action space), and robotic execution (on a Unitree Go2 quadruped robot via ROS/CycloneDDS). The framework emphasizes safety through a constrained action space and a deterministic validator to prevent LLM hallucinations.

Entities (9)

CycloneDDS · middleware · 100%Hand Gestures · input-modality · 100%Large Language Model · technology · 100%LLM-Powered Interactive Robotic Action Synthesis Framework · framework · 100%Music/Sound Beats · input-modality · 100%Qwen3:0.6b · model · 100%ROS · middleware · 100%Unitree Go2 · robot · 100%Whisper · model · 100%

Relation Signals (5)

Unitree Go2 → executesactionsvia → ROS

confidence 100% · This sequence is dispatched to an action queue for execution on a quadruped robot over ROS.

Hand Gestures → isprocessedby → Gesture Recognition Module

confidence 100% · A real-time gesture recognition module ([4]) processes the video stream to identify and classify static or dynamic gestures.

Qwen3:0.6b → isthecorellmof → LLM-Powered Interactive Robotic Action Synthesis Framework

confidence 100% · Due to near real-time demands, Qwen3:0.6b [7] was used as Ollama-based LLM.

Whisper → isusedfor → Natural Speech Processing

confidence 100% · User’s vocal commands are captured by a microphone and fed into an automatic speech recognition (ASR) model (Whisper [1]).

LLM-Powered Interactive Robotic Action Synthesis Framework → uses → Large Language Model

confidence 100% · This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs)...

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats. Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection. These processed inputs are contextualized using prompt templates and fed into a LLM. The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions. This sequence is dispatched to an action queue for execution on a quadruped robot over ROS. The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music. This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

9,640 characters extracted from source content.

Expand or collapse full text

LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music Snehasis Banerjee, Ranjan Dasgupta Abstract The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot’s expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats. Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection. These processed inputs are contextualized using prompt templates and fed into a LLM. The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions. This sequence is dispatched to an action queue for execution on a quadruped robot over ROS. The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music. This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner. The ability of robots to seamlessly collaborate with and act upon instructions from humans is a cornerstone of next-generation automation. While significant strides have been made in isolated command modalities like speech [2] or gesture control [6], these interactions often lack the richness and contextual nuance of human communication. Humans naturally combine speech, gestures, and even respond to environmental cues like music to convey intent. Enabling robots to understand and act upon this multimodal stream of information is crucial for achieving truly symbiotic HRI. Recent advancements in Large Language Models (LLMs) have demonstrated unprecedented capabilities in natural language understanding, reasoning, and planning [5]. This has inspired a new wave of research in robotics, where LLMs are used as a central ‘brain’ to translate high-level human goals into low-level robot actions. However, most existing work focuses primarily on text-based inputs, neglecting the valuable information contained in non-verbal modalities. This paper presents a framework, illustrated in Figure 1, for multimodal robotic action synthesis. Our key contribution is the integration of three distinct input channels—natural speech, hand gestures, and music beats — into a unified system orchestrated by an LLM with tailored prompts. We hypothesize that this fusion allows for more expressive and sophisticated robot control. For example, a user can say, ‘Do this trick’, while performing a specific gesture, and add, ‘on the beat’, prompting the robot to synthesize an action that is not only semantically correct but also rhythmically synchronized. We validate our approach on a quadruped robot, demonstrating its ability to perform complex maneuvers based on these multimodal commands. Additionally, for near real time system response, the model specifics have been carefully chosen post ablation, that are capable of running in low compute. Figure 1: System architecture for LLM-powered multimodal action synthesis. Natural speech, hand gestures, and music are processed into structured inputs (transcripts, recognized gestures, and beats) for the LLM, to generate an action sequence for the robot. Example is shown for command ‘stand on your hand’. I System Architecture The proposed framework consists of three main stages: multimodal input processing, LLM-based action reasoning, and robotic execution, as depicted in Figure 1. I-A Multimodal Input Processing • Natural Speech: User’s vocal commands are captured by a microphone and fed into an automatic speech recognition (ASR) model (Whisper [1]). The model outputs a text transcript, which serves as the primary semantic instruction for the LLM. • Hand Gestures: A camera captures the user’s hand movements. A real-time gesture recognition module ( [4]) processes the video stream to identify and classify static or dynamic gestures. The recognized gesture (e.g., ’open palm’, ’thumbs up’) provides complementary, often deictic information. • Music/Sound: An audio input stream is processed to extract rhythmic information. A signal processing pipeline applies beat detection algorithms [3] to identify the tempo and timing of beats in ambient music or sound. This provides a temporal structure for the robot’s actions. I-B LLM-based Action Reasoning The processed inputs from all three modalities are aggregated and formatted into a structured prompt using a set of predefined templates, before sending to the LLM. The LLM’s task is to act as a translator and planner. It takes the combined command (e.g., Transcript: ‘Do a flip’, Gesture: ‘pointing up’, Beats: [20 BPM]) and maps it to a plausible sequence of executable actions defined in the robot’s ‘Action Space’. The Action Space is a database or a structured description of all primitive skills the robot can perform (e.g., stand, sit, jump, forward, rotate). The LLM reasons about the goal (user instruction) and constraints (obstacles detected by sensors) to compose these primitives into a high-level plan. I-C Robotic Execution The action sequence generated by the LLM is sent to an ‘Action Queue’. Each action in the queue is translated into a specific API command 111Unitree Go2 SDK - https://github.com/unitreerobotics/unitree_sdk2 and sent to the robot’s controller, via ROS or OMG-DDS222Implementation as CycloneDDS: https://github.com/eclipse-cyclonedds. The robot then transitions from its initial state to execute the commanded action, such as the ‘Hand Stand Action’ shown in the figure. I Implementation and Results Our experimental setup utilizes a Unitree Go2 quadruped robot. For speech processing, we employed OpenAI’s Whisper ‘small’ model for transcription. Due to near real-time demands, Qwen3:0.6b [7] was used as Ollama-based LLM. In a preliminary experiment, a user instructed the robot, ‘Do something cool like standing on your limbs when my fist opens’ while making a ‘fist’ gesture. Upon the user opening their hand (an implicit ‘go’ gesture) and the start of a beat, the system successfully generated and executed the sequence for a handstand, that lasted till the duration of beat. This demonstrates the LLM’s ability to fuse the semantic meaning (‘stand on limbs’), the triggering event (gesture change), and the rhythmic constraint (the beat) into a single, coherent robotic action. To mitigate risks from LLM hallucinations, our framework employs a multi-layered safety system. The most critical ‘Guardrail’ is a constrained Action Space; the LLM can only select from a pre-vetted library of safe, high-level skills and cannot generate raw motor commands. Additionally, structured prompts guide the LLM to refuse ambiguous requests. Finally, a deterministic ‘Action Sequence Validator’ verifies any generated plan for correctness before it is sent for execution. This multi-layered approach ensures the robot operates within a predictable and safe envelope, grounding the LLM’s generative capabilities in physical reality. I Conclusion and Future Work We presented a novel LLM-powered framework for interactive robotic action synthesis that integrates speech, gestures, and music beats. Our initial results show that this multimodal approach enables more expressive and context-aware human-robot interaction. By allowing an LLM to reason over this rich input, robots can perform complex actions that are aligned with a user’s multifaceted intent. Future work will focus on expanding the action space, improving the real-time responsiveness of the system by optimizing the LLM inference pipeline, and incorporating a feedback loop where the robot’s state can inform the LLM’s subsequent planning steps. References [1] A. Andreyev (2025) Quantization for openai’s whisper models: a comparative analysis. arXiv preprint arXiv:2503.09905. Cited by: 1st item. [2] C. Deuerlein, M. Langer, J. Seßner, P. Heß, and J. Franke (2021) Human-robot-interaction using cloud-based speech recognition systems. Procedia Cirp 97, p. 130–135. Cited by: LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music. [3] F. Foscarin, J. Schlüter, and G. Widmer (2024) Beat this! accurate beat tracking without dbn postprocessing. arXiv preprint arXiv:2407.21658. Cited by: 3rd item. [4] A. Kapitanov, K. Kvanchiani, A. Nagaev, R. Kraynov, and A. Makhliarchuk (2024) HaGRID–hand gesture recognition image dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 4572–4581. Cited by: 2nd item. [5] Y. Kim, D. Kim, J. Choi, J. Park, N. Oh, and D. Park (2024) A survey on integration of large language models with intelligent robots. Intelligent Service Robotics 17 (5), p. 1091–1107. Cited by: LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music. [6] X. Wang, H. Shen, H. Yu, J. Guo, and X. Wei (2022) Hand and arm gesture-based human-robot interaction: a review. In Proceedings of the 6th International Conference on Algorithms, Computing and Systems, p. 1–7. Cited by: LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music. [7] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §I.

Instant research discovery