Paper deep dive

Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMs

Chris Lee, Flora Salim, Benjamin Tag, Francisco Cruz

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 48

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 6/21/2026, 3:01:02 AM

Summary

This paper presents an iterative Reinforcement Learning with Human Feedback (RLHF) system designed to improve the expressiveness and naturalness of co-speech gestures in the humanoid robot Pepper. Using GPT-4, the system generates executable Python code for low-level joint movements based on natural language input and a library of motion primitives. To overcome the stiffness of baseline LLM-generated motions, the authors implement a dual-phase learning approach: offline fine-tuning using Direct Preference Optimization (DPO) based on pairwise comparisons, and online fine-tuning where human feedback is summarized and injected into the conversation history via few-shot prompting. An iterative user study with 250 participants across five rounds demonstrated that this RLHF process successfully improves the perceived expressiveness, relevance, and fluidity of the robot's gestures.

Entities (6)

DPO · optimization-algorithm · 100%GPT-4 · large-language-model · 100%Pepper · robot · 100%Prolific · platform · 100%RLHF · machine-learning-method · 100%Aldebaran · organization · 90%

Relation Signals (4)

GPT-4 → generates → gesture code

confidence 100% · In this paper, we investigate GPT-4 for generating expressive co-speech gestures on the Pepper robot

Human Feedback → guides → LLM

confidence 100% · The iterative process improves the LLM's co-speech generative capabilities, producing more expressive, relevant and fluid movements.

DPO → usedfor → offline finetuning

confidence 100% · For offline finetuning, the LLM learns to predict l f ... by using direct preference optimisation (DPO)

RLHF → improves → Pepper

confidence 90% · Our results show that RLHF improved the LLM's co-speech generative capabilities, producing more expressive, relevant and fluid movements.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Expressive gestures are essential for natural and effective communication, complementing speech when verbal cues alone are insufficient (e.g., pointing). For social robots such as the humanoid Pepper, producing natural and expressive movements is critical for improving human-robot interaction (HRI) and long-term acceptance. However, generating gestures remains challenging due to reliance on expert-authored animations, resulting in rigid behaviors that are impractical for dynamic and diverse environments. Alternatively, machine learning approaches often struggle to capture perceived naturalness, becoming increasingly challenging with more degrees of freedom. Consequently, producing expressive robot gestures requires a system that can adapt to the environment while adhering to social norms and physical constraints. Recent advances in large language models (LLMs) enable dynamic code generation, offering new opportunities for runtime gesture synthesis from natural language. In this paper, we integrate ChatGPT into the humanoid robot Pepper to generate co-speech gestures aligned with conversational output. While this baseline enables flexible gesture generation, the resulting motions are often perceived as stiff and unnatural. To address this limitation, we introduce an iterative reinforcement learning with human feedback (RLHF) system that finetunes gesture generation based on user evaluations, leveraging an iterative user study to compare Pepper's generated gestures. Our results show that RLHF improved the LLM's co-speech generative capabilities, producing more expressive, relevant and fluid movements.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

47,988 characters extracted from source content.

Expand or collapse full text

Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMs Chris Lee 1 , Flora Salim 1 , Benjamin Tag 1 , Francisco Cruz 1,2 Abstract— Expressive gestures are essential for natural and effective communication, complementing speech when verbal cues alone are insufficient (e.g., pointing). For social robots such as the humanoid Pepper, producing natural and expressive movements is critical for improving human-robot interaction (HRI) and long-term acceptance. However, generating ges- tures remains challenging due to reliance on expert-authored animations, resulting in rigid behaviors that are impractical for dynamic and diverse environments. Alternatively, machine learning approaches often struggle to capture perceived natu- ralness, becoming increasingly challenging with more degrees of freedom. Consequently, producing expressive robot gestures requires a system that can adapt to the environment while adhering to social norms and physical constraints. Recent advances in large language models (LLMs) enable dynamic code generation, offering new opportunities for runtime gesture synthesis from natural language. In this paper, we integrate ChatGPT into the humanoid robot Pepper to generate co- speech gestures aligned with conversational output. While this baseline enables flexible gesture generation, the resulting motions are often perceived as stiff and unnatural. To address this limitation, we introduce an iterative reinforcement learning with human feedback (RLHF) system that finetunes gesture generation based on user evaluations, leveraging an iterative user study to compare Pepper’s generated gestures. Our results show that RLHF improved the LLM’s co-speech generative capabilities, producing more expressive, relevant and fluid movements. I. INTRODUCTION The rise of generative artificial intelligence (AI) has opened new opportunities for robotic systems to adapt and operate in dynamic, real-world environments. As robots are increasingly deployed in socially interactive settings, they require learning mechanisms that go beyond static pretraining and hand-crafted design. While current robotics research primarily targets functional tasks such as manipu- lation, navigation, and speech-based communication [1]–[4], domains that are well-defined and objectively measurable, socially expressive behaviors like gesture generation remain challenging due to their subjective and perception-driven nature [5], [6]. Ultimately, to enable social robots that are not only functional but also preferred by users, robotic systems must therefore be capable of continuously learning from human feedback. The performance of social robots is often evaluated using subjective human ratings, such as the GODSPEED question- naires [7], which introduce personal bias and abstraction. However, this subjectivity underscores the necessity of hu- man feedback when optimizing social behaviors, as improve- 1 Chris Lee, Flora Salim, Benjamin Tag and Francisco Cruz are with the School of Computer Science and Engineering, Univer- sity of New South Wales, Sydney, Australia. Corresponding Author: christopher.lee1956@student.unsw.edu.au 2 Francisco Cruz is with Universidad Central de Chile, Santiago, Chile ments in functional metrics do not necessarily correspond to users’ perceptions of intelligence or interaction quality [8]. Perceived qualities such as naturalness or smoothness cannot be fully captured by predefined or task-driven metrics. There- fore, developing anthropomorphic and likable social robots requires integrating learning mechanisms that adapt directly to communicated human preferences, such as interactive reinforcement learning. Supporting users with varying robotics expertise is also crucial, as they interpret robot behavior differently [8]. Intu- itive human-robot interaction (HRI) often relies on natural language [9], however, non-verbal cues such as gestures, perceived emotion or proxemics can serve as implicit reward signals [10], [11]. This motivates the use of large language models (LLMs), whose ability to incorporate human feed- back or examples (e.g., few-shot learning) [9], [12], can help align robot behavior with human preferences. The rapid growth of LLM research has enabled robotics to integrate multiple modalities [1], [3] and generate motion code in real time [2], [13]–[15]. In embodied agents, their modular structure allows coordinated processing of multi- modal data and environmentally grounded actions [3], [4], while richer feedback modalities can capture more nuanced user preferences [16]. Despite these advantages, challenges remain in contextual grounding, embodiment, and real-time adaptation, and reliance on offline pretraining can introduce cost and dataset rigidity bottlenecks [1]–[3]. Consequently, robots operating in real-world environments require the abil- ity to learn and adapt dynamically, rather than depending solely on predefined or curated knowledge. In this paper, we investigate GPT-4 for generating ex- pressive co-speech gestures on the Pepper robot [17]. Ini- tial LLM-generated motions were often stiff and unnatural, potentially limiting user acceptance if used in an HRI ex- periment. To address this, we conducted an iterative online user study in which participants rated videos of Pepper per- forming expressive co-speech gestures aligned with spoken phrases. Participants’ ratings were then used as feedback to finetune the LLM’s gesture code generation, allowing us to evaluate whether this iterative process improves perceived expressiveness. Hence, we introduce an iterative reinforce- ment learning with human feedback (RLHF) system (Fig. 1 that finetunes gesture generation based on user evaluations. This leads to our research questions; RQ1: Can human feedback be used to finetune an LLM’s code generation to improve the perceived expressiveness of generated robot gestures? RQ2: To what extent can LLMs generate code for expressive co-speech gestures on a humanoid robot? The online user study had five iterations, finetuning GPT- 4 a total of four times, each with new human feedback. arXiv:2606.18747v1 [cs.RO] 17 Jun 2026 How are you today? LLM OUTPUT: System Prompt Human Input Conversation History CONVERSATION HISTORY: [H: How are you today?, <L: I’m great, thank you for asking!, def execute_movement(motion): >, H: You’re not looking too energetic] You’re not looking too energetic SYSTEM PROMPT: You are Pepper, an assistive social robot... FUNCTION LIBRARY def head_rotate_up()... CURRENT JOINT ANGLES “RShoulderPitch”: 0.1... I’m great, thank you for asking! ｌ in 휏 (i) ｌ out 휋 (i) def execute_movement(motion): head_rotate_right (motion, 0.3, 1.0, 1.0) left_elbow_extend (motion, 0.5, 1.5, -0.9) right_elbow_contract (motion, 0.4, 1.5, 1.0) left_hand_open (motion, 0.4, 1.0, 1.0) [ｌ f,1 (i) ,...,ｌ f,p (i) ] C h ｌ f ｌ in 휏 + Iterative Finetuning + Few-Shot Prompting Human Feedback Human InputLLM Output Speech Output Gesture Output Robot Execution Human Observation Fig. 1.Overview of expressive co-speech gesture generation system with iterative reinforcement learning with human feedback. Our work demonstrates that LLMs can be finetuned in both offline and online settings to learn human preferences from subjective feedback. By coupling the model with a library of low-level motion primitives, we enable it to generate gesture code that can adapt flexibly to new contexts and embodiment constraints. Integrating human feedback directly into the reinforcement learning loop further allows the model to be steered toward increasingly expressive and human-aligned robot gesture code over time. I. RELATED WORKS A. Expressive Motion and Social Behavior in Humanoid Robots In real-world settings, verbal communication is often am- biguous, noisy, or incomplete, making non-verbal behaviors such as body language and gestures critical for conveying intent and meaning [18]. In HRI, gestures like pointing, can improve robot legibility and user understanding, leading to shorter interactions, faster error detection, and fewer overall errors [19]. However, gestures must be used carefully, as they can become confusing without sufficient contextual ground- ing [20], especially when they are symbolic or metaphorical. Overall, gestures are essential for HRI performance, as they shape task efficiency, error rates, and users’ interpretation of robot intent. Beyond functional roles, expressive movements convey affective and social information, shaping perception, engage- ment, and trust during interaction. Such movements can sig- nal responsiveness to nearby people [21] and communicate emotional cues that language alone cannot express [22]. Because these cues directly influence how humans interpret and relate to robots, any system that autonomously learns and generates expressive motions must ultimately be aligned with human perception and preferences. B. Reinforcement Learning with Human Feedback As social robots take on increasingly complex and long- term interactions, they must be able to learn incremen- tally from human feedback, often provided by non-expert users [10]. However, gathering feedback can be costly at large scales and when factoring fatigue. Designing RLHF remains challenging, particularly in shaping reward signals that reflect human intent while remaining practical for learn- ing [11], [23], [24]. Early approaches like TAMER relied on scalar feedback or structured mappings from verbal advice to reward variables [25], [26], while pairwise preference comparisons later enabled RLHF to scale to deep reinforce- ment learning [27]. More recent work has explored implicit feedback signals [28] and learned reward models [29], as well as leveraging LLMs to support richer and more scalable human feedback [12], [30]. Integrating LLMs with RL combines their strengths as LLMs provide broad pretrained knowledge, while RL en- ables contextual grounding through environmental interac- tion, with popular models already applying RLHF [31]. This synergy benefits robotics by supporting multimodal understanding, multitask generalization, efficient learning, and flexible reward shaping [11], [31], [32]. LLMs can also substitute human feedback with AI-generated feedback in self-rewarding systems [11], [32], [33], that ultimately improves scalability but weakens human-alignment. Human interpretability and alignment are crucial in critical settings, such as robots deployed in medical institutions, making traditional RL potentially unsafe or inefficient and the environment difficult to reproduce [24]. Interactive RL is particularly effective in early learning stages, where correc- tive feedback often provides more guidance than evaluative rewards [23]. Human feedback can be incorporated via offline and online learning (e.g., i-sim2real and SKiP), the of- fline phase stabilizes behavior while improving long-horizon performance [34], [35]. While offline finetuning maximizes performance on fixed data, online learning is essential to maintain alignment with evolving human preferences [36]. It- erating between these phases enables more efficient learning by mitigating early-stage exploration failures and adapting to dynamic user expectations. In robot motion generation, additional constraints further limit safe exploration. For instance, SEED [37] learns high- level primitives like reaching and grasping rather than low- level joint commands, highlighting the need to align human feedback with the action space. Combining pretrained LLM knowledge with RLHF’s sample efficiency offers a promising approach for learning expressive, human-aligned robot low- level motions iteratively. C. LLM based code generation and robotic code GenAI for robot code spans outputs from low-level move- ment functions to high-level policies, with higher-level ab- stractions generally being more human-aligned due to their explicit intent. A key challenge is enabling GenAI to com- pose low-level functions that reflect high-level intent. While larger datasets can improve generation quality, collecting human-aligned data at scale is costly, motivating approaches that learn preferences efficiently. Prompt-based methods are commonly used to inject human feedback into LLM-driven robot code generation [9], [13], [38], [39], with systems such as ECLAIR, GenEM++, and B ̈ armann et al.’s framework focusing on high-level actions (e.g., pick-and-place) rather than low-level motion. This exposes a gap in using GenAI and human feedback to generate expressive, unconstrained low-level robot motions for co-speech gesture generation. Creating robot gestures is inherently non-deterministic, as multiple gestures may be appropriate for the same in- put, particularly in co-speech contexts, making objective evaluation challenging [40], [41]. In prolonged interactions, it is natural and desirable for robots to avoid repetitive behaviors [6], and LLMs should also be capable of gener- ating diverse co-speech gesture code [15]. Quantitative and objective evaluations struggle to capture human preference across non-deterministic generative outputs. Likewise, al- though low-level primitive movement functions offer adapt- ability and generalization, they do not inherently encode the expressive, human-aligned patterns seen in higher-level movement policies. These limitations make iterative human feedback especially important for steering the model toward producing appropriate and expressive co-speech gestures. I. METHODOLOGY Overview: Our RLHF system inherits the procedural structure of recent gesture generation processes [13], [39], [42], using them to generate co-speech gestures that cannot be evaluated functionally, leveraging human evaluation to optimize their expressiveness. As seen in Fig. 1, it takes natural language input l in ∈ L and generates a co-speech tuple τ = ⟨l out ,π⟩ of conversational output l out ∈ L and a continuous sequence of parameterized primitive robotic functions, with length s to compose the expressive gesture π = [a 1 ,...,a s ]. We denote the sequence of parameterized functions as a temporally extended policy π, with each function as an action a. Each policy will then be subject to human evaluation, elaborated in the user study section, to act as the reward for finetuning. Only the motion sequence will be finetuned and not the conversational output, while the co-speech tuple is generated by OpenAI’s GPT-4.1 (gpt-4.1-2025-04-14). GPT has strengths in general purpose and linguistic under- standing [3], suited for co-speech and our prompt structure enforces safe code generation. The system will also keep a conversation history C h = [l in,1 ,τ 1 ,...,l in,c ,τ c ] where c is the number of interaction cycles between person and robot. C h is injected into the prompt to give the LLM contextual understanding of previous interactions with the person. Motion Sequence Generation: To ensure that the LLM- generated motion code is executable on the Pepper robot under the NAOqi v2.5 framework, we adhere to the official Aldebaran documentation 1 specifying joint names and angle limits. We implement a library of parameterized low-level joint motion functions with docstrings, which is injected into the LLM prompt as a structured knowledge base, enabling the model to generate properly embodied motions in real time from natural language input (Fig. 2). Each primitive function corresponds to a single joint (e.g., RShoulderPitch) and specifies the target angle in radians, execution speed, and a post-motion delay. Temporal coordination across the motion sequence is achieved by encouraging the LLM to insert explicit timing intervals between primitives, enabling synchronized co-speech gesture generation. Altogether, the sequence of reusable primitive functions and timing intervals will compose π. LLM Prompting: By default, when prompting GPT to generate expressive co-speech gestures alongside dialogue, the base model will lack embodiment and environmental awareness. We address this through a system prompt (Fig. 2) that constrains the LLM to produce safe, executable robot code while adopting a personality aligned with Pepper’s role as a social robot. As co-speech gestures are non-functional, the injected state information is limited to Pepper’s joint limits, represented via a library of primitive motions, and its current joint angles. We used role-playing prompts to define Pepper’s conversational persona (e.g., “You are Pepper, an assistive social robot”) and directional prompting to guide natural, polite, and concise speech (e.g., ”Use simple words when possible, stay positive and patient”), enabling consis- tent and human-appropriate conversational output. As co-speech gestures can have multiple motions rele- vant for the same utterance, we set the temperature at 1.5 despite other implementations being close to 0 [39], [42], to encourage expressive variability. Despite this increased stochasticity, strict output schemas were maintained through constraint-based code generation (e.g., ”You must NOT in- clude imports, comments...”, ”All motion function calls must follow this exact parameter order”), instruction prompting 1 Pepper Aldebaran Documentation: http://doc.aldebaran.com/ 2-4/home_pepper.html FUNCTION LIBRARY: Import time def head_rotate_up(motion, speed, delay, radians): """ Docstring for head_rotate_up This function rotates the robot's head up by the specified radians. Its angle limit is -0.7068 radians. """ current = motion.getAngles("HeadPitch", True)[0] if (current - radians) < -0.7068: radians = -0.7 - current motion.changeAngles("HeadPitch", -1*abs(radians), speed) time.sleep(delay) . . . def head_rotate_down(motion, speed, delay, radians): . . . def left_shoulder_rotate_up(motion, speed, delay, radians):: SYSTEM PROMPT: f"""You are Pepper, an assistive social robot created by SoftBank Robotics. You support people with friendly, helpful, and easy-to-speak verbal responses. . . . After generating the spoken response, you must also generate a coordinated expressive gesture to match the meaning, mood, and timing of the spoken response. You may use only the following robot movement functions for specific sequential joint movements: functions_library_doc . . . 2. Pepper's joint angles must stay within these recommended, visible ranges, when writing the code: - HeadPitch (head up/down): -0.7 to 0.6 rad (use ±0.3-0.5 for natural nods) - HeadYaw (head left/right): -2.0 to 2.0 rad (use ±0.5-1.0 for visible turns) . . . The current joint angles of Pepper are: current_angles When generating the gesture code, ensure that the movements calculate the current angle and do not exceed the joint limits. . . . Here is the required output structure again. You must follow it exactly: SPOKEN_RESPONSE: <short spoken sentence> GESTURE_CODE: def execute_movement(motion): <valid movement code using only the allowed functions> “” HUMAN INPUT: How are you today? LLM OUTPUT: SPOKEN_RESPONSE: I'm great, thank you for asking! GESTURE_CODE: def execute_movement(motion): head_rotate_right(motion, 0.3, 1.0, 1.0) left_elbow_extend(motion, 0.5, 1.5, -0.9) right_elbow_contract(motion, 0.4, 1.5, 1.0) left_hand_open(motion, 0.4, 1.0, 1.0) SPOKEN RESPONSE EXECUTION: tts.post.say(speech_output) GESTURE CODE EXECUTION: def execute_generated_movement(gesture_code, motion): setup_code = """ import time from function_library import * """ full_code = setup_code + " " + gesture_code namespace = 'motion': motion, exec full_code in namespace if 'execute_movement' in namespace: func = namespace['execute_movement'] . . . func(*args_to_pass) return True Fig. 2.Pipeline from function library of primitive robotic motions to final execution. (e.g., ”STRICT OUTPUT RULES - YOU MUST FOLLOW ALL OF THEM”), and program synthesis that ensures co- herent primitive motion sequences. Prompt robustness was further improved using meta-prompting, in which the system prompt was iteratively refined using LLM-assisted analy- sis. Additionally, for contextual grounding and real-time adaptation, we applied few-shot prompting by maintaining a conversation history of previous user interactions. IterativeHumanFeedback:Similarto EMOTION++ [39], we incorporate human feedback through an iterative combination of offline and online learning, where each iteration i ∈ [1, 2,...,i max ] will receive human feedback l f to reproduce reinforcement learning (Fig. 1). Each i will use the same set of l in with length n to produce corresponding co-speech tuples [l (i) in,1 ,...,l (i) in,n ] 7→ [τ i 1 ,...,τ i n ]. However, while EMOTION++ uses their research team for feedback, we crowdsource our human evaluations from an online user study to have a diversified understanding of human preferences and also scalability [24], [42]. We collect several evaluations on the same motion se- quence generated by the LLM to allow us to mimic episodic learning in RL, which is difficult to reproduce in HRI [24]. Every gesture π (i) ∈ [π (i) 1 ,...,π (i) n ] will receive l f ∈ [l f,1 ,...,l f,p ], where p is the number of participants, creating a total of n ∗ p samples of l f . For offline finetuning, the LLM learns to predict l f ←[⟨π (i) 1 ,π (i) 2 ,l out ⟩ by using direct preference optimisation (DPO) and pairwise comparison, identifying which of the two candidate gestures π (i) 1 or π (i) 2 produces higher l f and is preferred by participants. l out remains the same for each iteration of τ i . Meanwhile, online finetuning will have the LLM summarize l f,sum ←[ [l f,1 ,...,l f,p ], to append to the conversation history C h injected into the prompt as few-shot when given l in . Each subsequent i will retain C h from previous iterations, where each i will have c = i−1. To analyze the effectiveness of our system, the human evaluations of every i ∈ [1, 2,...,i max ] will be compared. IV. USER STUDY A. Overview To investigate how human feedback enhances LLM-based motion sequence generation for expressive co-speech ges- tures, we conducted an iterative online user study 2 organized into five evaluation rounds, where i max = 5. In each iteration, participants were shown video recordings of the Pepper robot speaking and gesturing, and asked to evaluate the generated co-speech gestures. Across all iterations, the verbal output remained fixed, while feedback collected in each round was used to fine-tune the LLM and refine gesture generation for subsequent iterations. This iterative design isolates the effect of human feedback on low-level 2 The full survey instrument preview can be viewed online at: https://unsw.au1.qualtrics.com/jfe4/preview/ previewId/df7d60a3-d99b-4049-bee7-6d6237f599d5/ SV_4Zrg5aTNYYUI2qy?Q_CHL=preview&Q_SurveyVersionID= current gesture generation and enables a systematic analysis of how expressive co-speech behaviors evolve over time, despite the inherent non-determinism and subjectivity of gesture inter- pretation. This user study addresses the research questions introduced earlier by comparing human evaluations across iterations, and we hypothesize for RQ1 that: H1: Co-speech gestures produced in the final iteration will be perceived as more satisfactory than those generated in the first iteration. B. Participants Participants were recruited via the online Prolific 3 plat- form, widely used for academic and industry research stud- ies. We prioritized native English speakers as inclusion criterion and recruited participants from AU, the UK, and the US, as the LLM verbal output was in English in the recorded videos. To mitigate language barriers when evaluating co- speech gestures, non-native speakers were also allowed, with English proficiency self-determined. A total of 250 adults (18+) participated across five study iterations of 50 participants each, targeting the general public to capture di- verse perspectives on perceived expressiveness. Nonetheless, cultural and language biases remain due to the use of English stimuli. Ethics approval 4 was obtained from the authors’ institution prior to data collection. C. Survey Procedure The online survey was hosted on the Qualtrics platform and deployed five times, each with a new cohort of par- ticipants. The participants are linked the documentation of the Pepper robot’s joints, to familiarize themselves with its movement capabilities and limits. This ensures that no feedback is given where the movements are physically un- achievable. In each deployment, participants viewed a ran- domized subset of 20 short videos showing the Pepper robot performing generated co-speech gestures. After each video, participants rated the perceived expressiveness, relevance, and fluidity of the gesture using a five-point Likert scale (1 = “Not at all”, 5 = “Very much”). Participants could also provide optional free-text comments (e.g., “arms barely moved”) to capture more natural feedback. Prior to each study iteration, co-speech gestures were generated using the LLM from a fixed set of phrases and recorded through Pepper. The first iteration used the base GPT-4.1 model, while each subsequent iteration fine-tuned the model from the previous round using the collected human feedback. A total of 50 distinct phrases were used in the study, grouped into five categories with 10 phrases each: greet- ing, encouragement, apology, explanation, and collaboration. Phrases were generated using GPT-4.1 with the same social embodiment prompting guidelines described in the method- ology, ensuring reproducibility in live deployment settings. The phrases were designed to reflect casual, everyday speech (e.g., “Good job, keep it up!”, “Hi, how are you doing today?”, and “Sorry if that caused any confusion”). Se- mantic overlap was intentionally introduced across phrases to encourage non-deterministic co-speech gesture generation 3 Prolific Website Link: https://w.prolific.com/ 4 UNSW HRECS Office - Reference No. iRECS8045 for similar meanings. In each iteration, 1,000 feedback sam- ples were collected, with each phrase-gesture pair receiving 20 evaluations. To ensure balanced exposure across phrase categories, the full set was divided into five subsets, each containing two phrases from every category. Participants evaluated two randomly selected subsets, resulting in an equal distribution of phrase types across the study. After each survey deployment, except the final iteration, collected human feedback was used to finetune the LLM. Offline finetuning via the OpenAI API trained the model to learn preferred gesture code sequences for semantically similar phrases, reinforcing patterns linked to greater per- ceived expressiveness, relevance, and fluidity For pairwise comparison we permutated within the five categories, then had a separate comparison for each combined participant average rating leading to a total of 1350 gesture code compar- isons. Another 150 comparisons between iterations were in- cluded. In parallel, online finetuning emulated conversational feedback, again using combined participant averages, and free-text comments were summarized into a single feedback description using the LLM. This consolidated feedback, to- gether with the phrase and generated code, was appended to the conversation history C h , as few-shot prompting examples. Both offline and online finetuning accumulated feedback across iterations and by the final iteration, the model had been trained on 4000 feedback samples while the conversa- tion history contained c = 4 prior feedback cycles. V. RESULTS & DISCUSSION 1) Quantitative Results: Using Welch’s unpaired t-test, we evaluate H1, testing whether perceived expressiveness, relevance, and fluidity improve across iterations of our RLHF system. For each iteration, user ratings were first averaged per phrase and then aggregated across phrase categories. As shown in Fig. 3, Iteration 5 exhibits a statistically significant improvement over Iteration 1 across all metrics and cate- gories (p<0.001), ultimately supporting H1. Most intermedi- ate iterations also show statistically significant gains relative to the baseline. Notably, Iteration 2 demonstrates the largest immediate improvement, highlighting the strong effect of incorporating human feedback through few-shot prompting. Subsequent iterations exhibit partial regression before stabi- lizing, suggesting a shift from early rapid adaptation toward more constrained and incremental refinement. Relevance and fluidity improved consistently across it- erations, exhibiting larger gains than expressiveness (e.g., Encouragement–Relevance increased from 2.00 to 2.78). In contrast, the largest overall improvement in expressiveness was more modest (∆ = 0.32 for Apology–Expressiveness), likely reflecting the more abstract and subjective nature of ex- pressiveness compared to the more functional dimensions of relevance and fluidity. Apology showed the strongest overall improvement across all metrics, with Greeting and Encour- agement also demonstrating substantial gains. This suggests that gestures associated with clear semantic intent and salient expressive cues were more readily reinforced through hu- man feedback, whereas more ambiguous categories such as Collaboration and Explanation required additional learning to achieve comparable improvements. Overall, our results Greeting Collaboration Encouragement Apology Explanation 0.0 0.5 1.0 1.5 2.0 2.5 3.0 *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ** Expressiveness Greeting Collaboration Encouragement Apology Explanation *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ** *** *** Relevance Greeting Collaboration Encouragement Apology Explanation *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** Fluidity Iterations Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Significance * p < 0.05 ** p < 0.01 *** p < 0.001 Fig. 3.Perceived average expressiveness, relevance, and fluidity user ratings across categories per iteration. Although Iteration 1 shows a significant improvement, subsequent iterations fluctuate downward before stabilising and then improving consistently through Iteration 5. 050100150200250300350 Step 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Error Rate Iterations Iteration 1 Iteration 2 Iteration 3 Iteration 4 Fig. 4.Smoothed Error Rate per iteration during DPO finetuning. address RQ2 by showing positive attitudes towards gesture quality with roughly half of the fifth iteration evaluations exceeding 3 out of 5. We conducted additional ablation experiments (Fig. 5) to assess the relative contributions of few-shot prompting and DPO. We compared the base model from Iteration 1 with the Iteration 5 combined system, alongside two ablations, removing few-shot prompting from the Iteration 5 model and injecting the few-shot conversation history into the base model. Error rates decreased across iterations (see Fig. 4), indicating that the model learned user preferences, however, as observed in the qualitative analysis, this learning alone often produced robust yet unnatural gesture patterns. Few- shot prompting, both in the base and combined models, yielded the largest overall gains, alternately improving ex- pressiveness. In contrast, the contribution of DPO was most evident in relevance and fluidity, where the combined model consistently outperformed the base model, suggesting that iterative offline fine-tuning enabled the LLM to internalize and prioritize relevant primitive motion-level traits associated with smoother and more coherent gesture execution. 2) Qualitative Gesture Generation Changes: Across iter- ations, we observed systematic changes in co-speech gesture generation in response to human feedback. In the first iteration, gestures showed limited diversity, dominated by repetitive arm-raising and open-hand movements, reflecting early exploration for reward signals. The second iteration demonstrated the strong influence of few-shot feedback in constraining behavior, where comments discouraging arm use led the model to largely suppress arm motion. Similarly, the third iteration further reduced gesture complexity, with the model favoring head nods over full-body gestures for high-context utterances, suggesting a conservative strategy under semantic ambiguity. In the fourth iteration, gesture categories began to converge, with pruning of redundant functions (see Fig. 6) and improved fluidity, alongside the emergence of novel behaviors (e.g., forward torso bends during apologies). Arm movements became slower and more deliberate, though head motion remained abrupt. In the fifth iteration, gesture diversity declined in the com- bined model, where few-shot prompting encouraged reuse of motion sequences from earlier iterations. This reuse of prior gestures suggests that the LLM captures which gestures are preferred, but struggles to reliably embody these preferences in executable robot motion. When conversational history was removed, the model exhibited severe mode collapse, reusing the same four to five gestures across all categories. This suggests that DPO alone was insufficient to translate semantic preference into diverse motion code. In contrast, using the full conversation history with the base GPT model produced similar gestures to the combined model and occa- sionally novel motions, albeit with reduced fluidity. Overall, these observations indicate that human feedback effectively prunes inappropriate gestures and enables limited novelty, however, sustaining diversity, and fluidity in low-level co- speech motion likely requires additional mechanisms, such as pose interpolation or higher-level motion representations, particularly for longer continuous sequences. Across all iterations, participant feedback revealed sev- eral recurring themes. Users consistently preferred motion quality over quantity, with expressiveness associated more with smooth, fluid movements than with large or numerous joint activations. Participants were generally more tolerant of under-expressiveness, particularly when unnecessary arm movements were omitted in favor of natural head motions, which may explain why semantically clear categories such as apologies expressed through head bowing were rated more favorably. Synchronization between speech and gesture was also repeatedly emphasized, however, use of Pepper’s native text-to-speech system limited temporal alignment, further encouraging preference for simpler gestures. Finally, Greeting Collaboration Encouragement Apology Explanation 0.0 0.5 1.0 1.5 2.0 2.5 3.0 *** *** *** *** *** *** *** *** *** *** *** *** ** *** *** Expressiveness Greeting Collaboration Encouragement Apology Explanation *** ** *** *** *** *** *** *** *** *** *** *** *** *** *** Relevance Greeting Collaboration Encouragement Apology Explanation *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** Fluidity Iterations Iteration 1 Iteration 5 Combined Iteration 5 DPO Iteration 1 Few-shot Significance * p < 0.05 ** p < 0.01 *** p < 0.001 Fig. 5.Perceived average expressiveness, relevance, and fluidity user ratings across categories under ablation testing. Greeting Collaboration Encouragement Apology Explanation Category 0 2 4 6 8 10 Average Function Call Count Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Fig. 6. Average primitive movement functions called per co-speech gesture sequence across categories per iteration. participants criticized gestures that appeared semantically unrelated, non-human, or excessively jerky, fast and over- pronounced. Collectively, these observations highlight key priorities for improving GenAI-driven co-speech gesture generation using low-level motion primitives. VI. CONCLUSIONS AND FUTURE WORK This work demonstrates that large language models, when paired with iterative reinforcement learning from human feedback (RLHF), meaningfully improve the generation of expressive co-speech gestures for humanoid robots. By inte- grating GPT-4.1 with a function library of low-level motion primitives and iteratively refining its outputs through five rounds of human evaluation, we show that subjective human preferences can be translated into more expressive, relevant, and fluid robot motion code. Across iterations, user ratings revealed statistically significant improvements in all mea- sured dimensions, with relevance and fluidity benefiting most from the combined effects of few-shot prompting and direct preference optimization-based finetuning. Qualitative analy- sis further highlighted how human feedback shaped gesture selection, reduced inappropriate or overly complex motions, and encouraged more semantically aligned behaviors. At the same time, our findings expose the limitations of relying solely on low-level function composition for ex- pressive motion generation. While RLHF effectively pruned undesirable behaviors, such as unnecessary and unnatural arm movements, sustaining gesture diversity and achieving smooth, human-like motion remained challenging, particu- larly as the model gravitated toward conservative or repetitive patterns (e.g., head nodding). These observations suggest that future systems may benefit from hybrid approaches that combine LLM-driven code generation with higher-level motion abstractions, pose interpolation, movement gradients or learned dynamical models to better capture the richness and nuances of human expressiveness. Overall, this study provides evidence that human-in-the- loop learning can enhance the dynamic capabilities of LLM- powered robots, offering a scalable path toward more adap- tive and human-aligned gesture generation. There is future opportunities of integrating other modalities such as VLM self-critique or visual embodiment, to further ground the LLM. As generative AI continues to advance, integrating human feedback into embodied systems will be essential for developing robots that communicate not only effectively, but expressively and intuitively in real-world interactions. VII. ACKNOWLEDGMENT This research was supported by the Commonwealth through an Australian Government RTP Scholarship [DOI: https://doi.org/10.82133/C42F-K220] and by CSIRO, Aus- tralia’s national science organisation, through the Next Gen- eration Graduate Program (NGGP) scholarship. We sincerely thank Takero Izuhara and Tao Lu, former associates from our industry partner ST Solutions PTY LTD, for their ongoing support and ideas, providing research resources, including access to the Pepper robot used in our work. REFERENCES [1] C. Zhang, J. Chen, J. Li, Y. Peng, and Z. Mao, “Large language models for human–robot interaction: A review,” Biomimetic Intelligence and Robotics, vol. 3, no. 4, p. 100131, 2023. [2] K. Zhang, P. Yun, J. Cen, J. Cai, D. Zhu, H. Yuan, C. Zhao, and et al, “Generative Artificial Intelligence in Robotic Manipulation: A Survey,” 2025. [3] F. Zeng, W. Gan, Y. Wang, N. Liu, and P. S. Yu, “Large Language Models for Robotics: A Survey,” 2023. [4] J. Wang, E. Shi, H. Hu, C. Ma, Y. Liu, X. Wang, Y. Yao, and et al, “Large language models for robotics: Opportunities, challenges, and perspectives,” Journal of Automation and Intelligence (2025), p. S2949855424000613, 2024. [5] Y. Yoon, B. Cha, J.-H. Lee, M. Jang, J. Lee, J. Kim, and G. Lee, “Speech gesture generation from the trimodal context of text, audio, and speaker identity,” ACM Trans. Graph., vol. 39, no. 6, p. 222:1– 222:16, 2020. [6] U. Zabala, I. Rodriguez, J. M. Mart ́ ınez-Otzeta, and E. Lazkano, “Modeling and evaluating beat gestures for social robots,” Multimed Tools Appl, vol. 81, no. 3, p. 3421–3438, 2022. [7] C. Bartneck, D. Kuli ́ c, E. Croft, and S. Zoghbi, “Measurement In- struments for the Anthropomorphism, Animacy, Likeability, Perceived Intelligence, and Perceived Safety of Robots,” Int J of Soc Robotics, vol. 1, no. 1, p. 71–81, 2009. [8] G. Hoffman and X. Zhao, “A primer for conducting experiments in human–robot interaction,” ACM Transactions of Human-Robot Interaction, vol. 10, no. 1, 2021. [9] L. B ̈ armann, R. Kartmann, F. Peller-Konrad, J. Niehues, A. Waibel, and T. Asfour, “Incremental learning of humanoid robot behavior from natural interaction and large language models,” Front. Robot. AI, vol. 11, 2024. [10] N. Akalin and A. Loutfi, “Reinforcement Learning Approaches in Social Robotics,” Sensors, vol. 21, no. 4, p. 1292, 2021. [11] T. Kaufmann, P. Weng, V. Bengs, and E. H ̈ ullermeier, “A Survey of Reinforcement Learning from Human Feedback,” 2025. [12] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, et al., “Training language models to follow instructions with human feedback,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, NeurIPS ’22, (Red Hook, NY, USA), p. 27730–27744, Curran Associates Inc., 2022. [13] K. Mahadevan, J. Chien, N. Brown, Z. Xu, C. Parada, F. Xia, A. Zeng, and et al, “Generative Expressive Robot Behaviors using Large Language Models,” in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, p. 482–491, 2024. [14] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for Robotics: Design Principles and Model Abilities,” 2023. [15] A. Sobo, A. Mubarak, A. Baimagambetov, and N. Polatidis, “Eval- uating LLMs for Code Generation in HRI: A Comparative Study of ChatGPT, Gemini, and Claude,” Applied Artificial Intelligence, vol. 39, no. 1, p. 2439610, 2025. [16] J. De Heuvel, D. Marta, S. Holk, I. Leite, and M. Bennewitz, “The Impact of VR and 2D Interfaces on Human Feedback in Preference- Based Robot Learning,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 19024–19030, 2025. [17] A. K. Pandey and R. Gelin, “A Mass-Produced Sociable Humanoid Robot: Pepper: The First Machine of Its Kind,” IEEE Robot. Automat. Mag., vol. 25, no. 3, p. 40–48, 2018. [18] A. Mehrabian, Nonverbal Communication.New York: Routledge, 2017. [19] C. Breazeal, C. Kidd, A. Thomaz, G. Hoffman, and M. Berlin, “Effects of nonverbal communication on efficiency and robustness in human- robot teamwork,” in 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 708–713, 2005. [20] T. Ende, S. Haddadin, S. Parusel, T. W ̈ usthoff, M. Hassenzahl, and A. Albu-Sch ̈ affer, “A human-centered approach to robot gesture based communication within collaborative working processes,” in 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 3367–3374, 2011. [21] M. Lombardi, C. Calabrese, D. Ghiglino, C. Foglino, D. De Tommaso, G. Da Lisca, L. Natale, and et al, “Would you let a humanoid play storytelling with your child? A usability study on LLM-powered nar- rative Humanoid-Robot Interaction,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 20066– 20073, 2025. [22] V. Ng-Thow-Hing, P. Luo, and S. Okita, “Synchronized gesture and speech production for humanoid robots,” in 2010 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, p. 4617–4624, 2010. [23] I. Moreira, J. Rivas, F. Cruz, R. Dazeley, A. Ayala, and B. Fernandes, “Deep Reinforcement Learning with Interactive Feedback in a Hu- man–Robot Environment,” Applied Sciences, vol. 10, no. 16, p. 5574, 2020. [24] M. Balsells, M. T. Villasevil, Z. Wang, S. Desai, P. Agrawal, and A. Gupta, “Autonomous Robotic Reinforcement Learning with Asyn- chronous Human Feedback,” in Proceedings of The 7th Conference on Robot Learning, p. 774–799, PMLR, 2023. [25] W. B. Knox and P. Stone, “Interactively shaping agents via human reinforcement: the TAMER framework,” in Proceedings of the fifth international conference on Knowledge capture, K-CAP ’09, (New York, NY, USA), p. 9–16, Association for Computing Machinery, 2009. [26] G. Kuhlmann, P. Stone, R. Mooney, and J. Shavlik, “Guiding a reinforcement learner with natural language advice: Initial results in RoboCup soccer,” AAAI Workshop - Technical Report, 2004. [27] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Proceedings of the 31st International Conference on Neural In- formation Processing Systems, NeurIPS’17, (Red Hook, NY, USA), p. 4302–4310, Curran Associates Inc., 2017. [28] H. Wang, J. Lin, Z. Ma, Y. Vasylkiv, H. Brock, K. Nakamura, R. Gomez, and et al, “Affective Behavior Learning for Social Robot Haru with Implicit Evaluative Feedback,” in 2022 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), p. 3881– 3888, 2022. [29] R. Juan, J. Huang, R. Gomez, K. Nakamura, Q. Sha, B. He, and G. Li, “Shaping Progressive Net of Reinforcement Learning for Policy Transfer with Human Evaluative Feedback,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 1281–1288, 2021. [30] D. Jin, S. Mehri, D. Hazarika, A. Padmakumar, S. Lee, Y. Liu, and M. Namazifar, “Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language,” 2023. [31] Y. Cao, H. Zhao, Y. Cheng, T. Shu, Y. Chen, G. Liu, G. Liang, and et al, “Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 6, p. 9737–9757, 2025. [32] S. Wang, S. Zhang, J. Zhang, R. Hu, X. Li, T. Zhang, J. Li, and et al, “Reinforcement Learning Enhanced LLMs: A Survey,” arXiv preprint arXiv:2412.10400, 2024. [33] W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston, “Self-rewarding language models,” in Proceedings of the 41st International Conference on Machine Learning, vol. 235 of ICML’24, (Vienna, Austria), p. 57905–57923, JMLR.org, 2024. [34] S. W. Abeyruwan, L. Graesser, D. B. D’Ambrosio, A. Singh, A. Shankar, A. Bewley, D. Jain, and et al, “i-Sim2Real: Reinforce- ment Learning of Robotic Policies in Tight Human-Robot Interaction Loops,” in Proceedings of The 6th Conference on Robot Learning, p. 212–224, PMLR, 2023. [35] X. Wang, K. Lee, K. Hakhamaneshi, P. Abbeel, and M. Laskin, “Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback,” in Proceedings of the 5th Conference on Robot Learning, p. 1259–1268, PMLR, 2022. [36] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, and et al, “Learning to summarize from human feed- back,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, NeurIPS ’20, (Red Hook, NY, USA), p. 3008–3021, Curran Associates Inc., 2020. [37] A. Hiranaka, M. Hwang, S. Lee, C. Wang, L. Fei-Fei, J. Wu, and R. Zhang, “Primitive Skill-Based Robot Learning from Human Evaluative Feedback,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 7817–7824, 2023. [38] I. Tarakli, S. Vinanzi, and A. D. Nuovo, “Interactive Reinforcement Learning from Natural Language Feedback,” in 2024 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), p. 11478–11484, 2024. [39] P. Huang, Y. Hu, N. Nechyporenko, D. Kim, W. Talbott, and J. Zhang, “EMOTION: Expressive Motion Sequence Generation for Humanoid Robots With In-Context Learning,” IEEE Robotics and Automation Letters, vol. 10, no. 8, p. 7699–7706, 2025. [40] S. Lu, Y. Yoon, and A. Feng, “Co-Speech Gesture Synthesis using Discrete Gesture Token Learning,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 9808–9815, 2023. [41] P. J. Yazdian, M. Chen, and A. Lim, “Gesture2Vec: Clustering Ges- tures using Representation Learning Methods for Co-speech Gesture Generation,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 3100–3107, 2022. [42] L. Roy, E. A. Croft, A. Ramirez, and D. Kuli ́ c, “GPT-Driven Gestures: Leveraging Large Language Models to Generate Expressive Robot Motion for Enhanced Human-Robot Interaction,” IEEE Robotics and Automation Letters, vol. 10, no. 5, p. 4172–4179, 2025.

Instant research discovery