Paper deep dive
Auto-HSI: Personalized human control of a robot swarm on demand by using LLMs for online automatic code generation
Alessandro Nazzari, Nathan Cerisara, Dorian Tonnis, Raina Zakir, Lorenzo Labarile, Weixu Zhu, Marco Dorigo, Mary Katherine Heinrich
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper presents Auto-HSI, a method for generating personalized human-swarm interaction (HSI) interfaces on demand. The objective is to enable untrained operators to use natural language descriptions and gesture demonstrations to explain how they want the robots to collectively behave in response to their gestures. Based on these inputs, the code should automatically be generated for personalized state machines that will control the robots as desired, in response to the desired gesture inputs. In the developed Auto-HSI prototype, the generated code produces a personalized interface for centralized control using one- and two-handed gestures, enabling a user to teleoperate the robots' motion, formation shape, and shape deformation. We test the gesture tracking and code generation components of Auto-HSI against performance benchmarks. We then test the full Auto-HSI prototype in ``live'' operation experiments, in which real human operators centrally control 50 simulated robots in a physics-based simulator, under nominal and noisy conditions. In these experiments, robots are teleoperated to: score a goal, traverse a maze that requires shape deformation, and score two simultaneous goals by splitting into two groups. We also demonstrate a real human operator making live updates to their personalized Auto-HSI interface during operation (in simulation). Finally, we demonstrate live operation of real robots.
Tags
Links
- Source: https://arxiv.org/abs/2609.16346v1
- Canonical: https://arxiv.org/abs/2609.16346v1
Trouble viewing inline? Open PDF directly →
Full Text
43,887 characters extracted from source content.
Expand or collapse full text
Auto-HSI: Personalized human control of a robot swarm on demand by using LLMs for online automatic code generation* A. Nazzari Affiliation: IRIDIA, Université Libre de Bruxelles, Brussels, Belgium Affiliation: Department of Aerospace Science and Technology, Politecnico di Milano, Milan, Italy N. Cerisara Affiliation: IRIDIA, Université Libre de Bruxelles, Brussels, Belgium Affiliation: Télécom Physique Strasbourg, Université de Strasbourg, France D. Tonnis Affiliation: IRIDIA, Université Libre de Bruxelles, Brussels, Belgium Affiliation: Polytech Lyon, Lyon 1 Université Claude Bernard, Lyon, France R. Zakir Affiliation: IRIDIA, Université Libre de Bruxelles, Brussels, Belgium L. Labarile Affiliation: IRIDIA, Université Libre de Bruxelles, Brussels, Belgium Affiliation: University of Pisa, Pisa, Italy; Scuola Superiore Sant’Anna, Pisa, Italy W. Zhu Affiliation: IRIDIA, Université Libre de Bruxelles, Brussels, Belgium M. Dorigo Affiliation: IRIDIA, Université Libre de Bruxelles, Brussels, Belgium M. K. Heinrich †thanks: *R.˜Zakir, M.˜Dorigo, and M.K.˜Heinrich acknowledge support from the Belgian F.R.S.-FNRS, of which they are a FRIA Doctoral student, a Research Director, and a Research Associate, respectively.†thanks: $†$ A. Nazzari, N. Cerisara, and D. Tonnis contributed equally to this work. Affiliation: IRIDIA, Université Libre de Bruxelles, Brussels, Belgium Abstract This paper presents Auto-HSI, a method for generating personalized human-swarm interaction (HSI) interfaces on demand. The objective is to enable untrained operators to use natural language descriptions and gesture demonstrations to explain how they want the robots to collectively behave in response to their gestures. Based on these inputs, the code should automatically be generated for personalized state machines that will control the robots as desired, in response to the desired gesture inputs. In the developed Auto-HSI prototype, the generated code produces a personalized interface for centralized control using one- and two-handed gestures, enabling a user to teleoperate the robots’ motion, formation shape, and shape deformation. We test the gesture tracking and code generation components of Auto-HSI against performance benchmarks. We then test the full Auto-HSI prototype in “live” operation experiments, in which real human operators centrally control 50 simulated robots in a physics-based simulator, under nominal and noisy conditions. In these experiments, robots are teleoperated to: score a goal, traverse a maze that requires shape deformation, and score two simultaneous goals by splitting into two groups. We also demonstrate a real human operator making live updates to their personalized Auto-HSI interface during operation (in simulation). Finally, we demonstrate live operation of real robots. I Introduction A key challenge in human-swarm interaction (HSI) [1] is to understand how a human operator can control a large group of robots using only one or a few inputs: that is, far fewer inputs than the total number of robots. A common approach is teleoperation via a central control station that generates and sends commands to all or some of the robots, using inputs from gesture or pose tracking [2, 3, 4, 5], joysticks [6], or haptic feedback devices [7]. We build on this existing work and follow the same standard control station setup, focusing on teleportation using gestures. In existing HSI studies, the set of possible swarm behaviors that an operator is able to trigger is usually predefined, and often quite limited. The interface (that is, the possible operator inputs and which robot control outputs they produce) is also usually predefined, and assumes a trained operator: the operator should already have been informed which inputs should be used to steer the swarm, and how. An important exception is [4], in which a linear model was trained to map user motion to robot motion. In this study, we aim to expand to the personalized on-demand generation of the full state machine registry and constituent code blocks that run locally on each robot, based on open-ended natural language inputs. In our Auto-HSI approach, a personalized HSI interface is automatically generated to allow users to operate the robots in the manner they desire. In the prototype reported here, a new user needs to be instructed to use hand-based gestures, and to mark the beginning and end of a gesture with a closed fist, but the interface personalization options are otherwise open-ended. The user can request that any demonstrated gesture become associated to any of the desired multi-robot shape and motion behaviors described in natural language. To approach this objective, Auto-HSI makes use of existing external Large Language Models (LLMs); we demonstrate that different LLMs can be interchanged and used with our presented Auto-HSI prototype. Automatic code generation using LLMs has been demonstrated for human control of individual robots [8], but not yet for self-organized human-swarm interaction. Here, we demonstrate our Auto-HSI prototype for robots that can compile and run their own local copies of the generated code, but also can communicate with a central control station. Future work is to extend the Auto-HSI prototype to the control of robot swarms that coordinate exclusively in a self-organized manner. I Methods: User interaction with Auto-HSI From the user’s perspective, interaction with Auto-HSI is organized in two phases: personalization, in which the user demonstrates the desired gestures and explains in natural language the robot swarm behaviors desired for those gestures, and live operation, in which the swarm is operated through gestures alone. The LLM is involved in the personalization phase only; it never issues commands to the robots, but rather generates and edits code that the robots execute at 1010\,Hz. Note that the personalization phase is not restricted to system setup: the personalization phase can be called again during operation of the swarm, if the user wishes to update their personalized Auto-HSI. I-A Personalization phase The user demonstrates a gesture and provides a natural language description of the desired multi-robot behavior the gesture should trigger, via a generated state machine (SM) registry. The system automatically assembles a query that includes the user’s description, some operating guidelines, and the current configuration of the SM registry. To implement the described behavior, the LLM inspects the SM registry to determine whether an existing SM already covers similar behaviors. If so, the LLM modifies the existing SM, for example by adding a node with a transition associated to the newly demonstrated gesture. Otherwise, the LLM creates a new SM. At each code generation step, the update is validated before being committed, so that syntactically invalid code never reaches the robots. I-B Live operation phase In this phase, there are no calls to the external LLM. When a gesture is detected, it is compared via dynamic time warping (DTW) to the gesture database to determine the most similar entry: if the most similar database entry has a DTW cost below the acceptance threshold (parameter for tuning), it is considered a match. The gesture identifier of the match is then broadcast to all robots. On the following control step each robot evaluates the outgoing transitions of its active SM node and, if a transition is associated to the broadcast identifier, it is executed. Transitions are evaluated independently in every SM, so a gesture can trigger transitions in more than one SM. I-C Updates during operation: re-triggering personalization Once the interface is complete, the personalization phase can be re-triggered during operation if needed. The user can explain in natural language which behavior is to be modified and how; the LLM produces the corresponding change, which is applied in the same manner as in the initial personalization phase. I Methods: Auto-HSI software components Fig. 1: Diagram illustrating an example of an automatically generated Auto-HSI instance: gesture database (left) and state machine (SM) registry (right) resulting from an example personalization phase. The gestures (labeled in blue text) demonstrated by the user are automatically associated to transitions in generated SMs for the robots’ movement, formation rotation, and formation shape; these transitions are triggered by the recognition of the respective gesture. Additional transitions in the generated SMs proceed automatically (labeled in red text) after one control step and return the machine to its base node. Note that, in addition to the SM structures and transitions being automatically generated, the executable code inside the nodes of the SMs is also generated. The executable code inside the SMs can call functions to reference a database of shape definitions that is also generated, simultaneously to the generation of SMs. The Auto-HSI software includes two main internal components, gesture tracking and recognition (see Sec. I-A) and code generation (see Sec. I-B); and two components for communicating with external elements, an external LLM (see Sec. I-C) and the robots (see Sec. I-D). The full Auto-HSI prototype, along with all software used in all experimental setups, demonstrations, and analyses of results is provided in the online code repository11 1 Online code repository: https://doi.org/10.5281/zenodo.22724336. I-A Gesture tracking and recognition From each camera frame, our Auto-HSI prototype uses 2121 points =(x,y)k=(x_k,y_k) per visible hand: 44 points (fingertip, finger base, and two intermediate joints) per finger if_i, and 11 for the wrist.22 2 The 21 points-per-hand that the Auto-HSI prototype uses are extracted from camera frames using the Mediapipe hand-landmark model [9]. From the 21 points ∈k per hand, we first classify fist or non-fist, where a fist is a hand configuration K with no finger extended. A fist is distinguished by comparing fingertip–wrist and finger-base–wrist distances for each finger ∋index,middle,ring,pinkyF \f_ index,f_ middle,f_ ring,f_ pinky\, as follows: ∈fistif ∀i∈∥tipi−w∥≤∥basei−w∥,non-fistotherwise ∈ cases aligned fist~~&if~~ _f_i ~~~ _ tip_i-k_w ≤ _ base_i-k_w ,\\ non-fist~~&otherwise~~ aligned cases (1) where wk_w is the wrist and tipik_ tip_i and baseik_ base_i are the fingertip and finger-base, respectively, of finger if_i. For each K classified as non-fist (Eq. 1), we extract the center of the palm =(xc,yc)c=(x_c,y_c), defined as the geometric centroid of: wk_w, baseindexk_ base_ index, and basepinkyk_ base_ pinky. We combine the left and right palm centers, leftc_ left and rightc_ right, into the two-handed configuration =(xcleft,ycleft,xcright,ycright)C=(x_c_ left,y_c_ left,x_c_ right,y_c_ right). For an undetected hand, ←(0,0).c←(0,0). Gesture detection is segmented using fists: a gesture segment is an uninterrupted sequence of frames with at least one visible ∈non-fistK∈ non-fist, preceded and followed by frames that contain exclusively ∈fistK∈ fist. After segmentation, a gesture G is defined as the sequence of vector differences from the segment’s two-handed configurations [2,n][C_2,C_n] to its initial 1C_1, calculated as: =(2−1,3−1,…,n−1), =(C_2-C_1,\;C_3-C_1,\;…,\;C_n-C_1)~, (2) such that the resulting G is a sequence of 4-dimensional vectors. Gestures that are demonstrated by the user during the personalization phase are written to a database, against which new gestures need to be compared during the live operation phase. To account for the fact that the same gesture may be performed at varying speeds and thus two sequences G need to be aligned along the temporal axis, we score gesture similarity using DTW, calculating cumulative cost DTW(i,j) DTW(i,j) from starting point (1,1)(1,1) to (i,j)(i,j) in the standard manner [10] as: DTW(i,j)=d(xi,yj)+minDTW(i−1,j)DTW(i,j−1)DTW(i−1,j−1), DTW(i,j)=d(x_i,y_j)+ cases DTW(i-1,j)\\ DTW(i,j-1)\\ DTW(i-1,j-1) cases, (3) where d(xi,yj)d(x_i,y_j) denotes Euclidean distance between 4-dimensional vectors xix_i and yjy_j, and sequences X=(x1,x2,…,xn)X=(x_1,x_2,...,x_n) and Y=(y1,y2,…,yn)Y=(y_1,y_2,...,y_n) are two different gestures 1G_1 and 2G_2 (Eq. 2). I-B Code generation During the personalization phase, a registry of SMs is populated incrementally as the user defines new behaviors. An SM consists of a neutral base node and a set of behavior nodes. Each node carries code blocks executed 1) on entry, 2) on every control step, and 3) on exit. Each transition is triggered either by a gesture event or automatically after a fixed number of control steps. Transitions are routed through the base node. For each SM, a label is generated that should indicate a general behavior category to which the SM belongs, so that a later request in the same category can be applied to the same SM in an ad hoc manner. Fig. 1 shows an example registry produced by the personalization phase: seven recorded gestures and three SMs, handling movement, rotation, and shape. For example, in the SM for handling movement, the swipe up gesture moves the machine into the move forward node, whose entry block sets the initial velocity to move the robots forward, control-step block maintains that velocity, and exit block reverts the velocity to zero. The LLM receives a request in natural language and runs a sequence of tool calls that create or modify an SM registry. Each natural language request includes the user’s natural language input, the static Auto-HSI baseline prompt, and the current state of the SM registry. The full history of interactions is also sent to the LLM, to support the ability to perform incremental changes. The static Auto-HSI baseline prompt is created at initialization and includes the following content. First, it lists the functions available to interact with the SM registry: • retrieve SMs from the registry; • retrieve the nodes of an SM; • create or remove an SM (including its label and initial node); • create, remove, or modify a node (including its transitions, and thereby the gestures that trigger transitions, and its calls to a simultaneously generated database of shapes); • modify the three code blocks of a node (on entry; on exit and on step); as well as the functions to add a gesture with a generated label to a gesture database, add a generated shape definition to a shape database, or modify an existing shape definition (e.g., to deform its polygon points). Note that generated shape definitions can be based on the user drawing the desired shape with a gesture or the user’s natural language description of the desired shape. It also specifies some requests for the generated SMs, e.g., the duration of a control step and the robot coordinate frame conventions. Second, it specifies the instructions for the generated code blocks of an SM node. A block is free-form generated Lua code that makes use of some pre-existing functions, with generated arguments. The provided list of Lua functions to be used in generated code, whether for the SMs or the shape database, includes: create a formation shape, deform a formation shape, scale a formation shape, set the formation velocity, rotate a formation, and obtain state information (formation shape, formation rotation, formation center, number of robots). Third, it provides two examples. The first example is a minimal SM registry: a single SM labeled “movement,” containing the base node and a node labeled “forward,” which has an on enter code block and on exit code bock, respectively starting and then stopping the robots’ motion. The second example is a request from a user during a personalization phase, accompanied by the ordered list of functions that should be called as a result: inspect the SM registry and verify if a new SM should be created, inspect the gesture database and verify if the requested gesture exists, register a new gesture if needed, inspect the shape database and verify if a new shape should be created, create or modify a shape as needed, create or modify an SM as needed, update the code blocks of new SM nodes as needed, and finally verify the SM has been correctly added to the registry and can be retrieved. Finally, it also lists some general instructions, including: guidelines on how to interact with the user by avoiding technical terms and instead using a conversational tone; suggestions on avoiding overly complex code structures when creating SM nodes; and prohibition to ask the user for feedback on the generated code. I-C Interface to an external LLM The natural language input from the user can be typed or dictated through a local speech-to-text engine. The LLM returns either a text answer, which is displayed to the operator and closes the query, or a set of tool calls, which are executed against the registry before the LLM is queried again, with the results appended to the history. Requests are issued through OpenRouter’s API, which enables different external LLMs to be used interchangeably by the Auto-HSI prototype. The registry manager verifies that modifications applied to the code of any node result in valid Lua code, compiles the code fragments before accepting them, and snapshots the SM registry. A rejected fragment is returned to the LLM and is corrected without involving the operator, while a runtime error raised on a robot is reported back to the server, which issues an automatic repair request. Requests to the LLM run in a dedicated thread, so that neither the interface nor the communication loop is blocked while a query is resolved. I-D Communication with the robots In the simulation setup, each robot holds its own TCP connection to the server and runs its own local copy of the generated code for robot control (i.e., the SM registry with its required shapes). When a modification of one SM is committed, the affected SM is serialized to JSON and pushed over the open sockets. A full node update rebuilds the SM in memory while preserving the identifier of the active node. A partial node update recompiles only the relevant code blocks. IV Results First, the two main internal components of Auto-HSI (gesture tracking and recognition; code generation), are evaluated against performance benchmarks. Then, we present experiments and demonstrations of real human operators33 3 Note that, in this paper, black boxes have been added to images containing faces, for de-identification. using the full Auto-HSI prototype. All experimental results data, raw videos, and LLM logs are provided in the online data repository44 4 Online data repository: https://doi.org/10.5281/zenodo.22727703. [Note for double-blind review: Temporarily, repository has been partially redacted because of faces of authors in raw experiment videos.]. IV-A Benchmark: Gesture recognition To assess the gesture recognition performance of the tracking and recognition method (Sec. I-A), we use a benchmark matrix m×n∗G_m× n^* that consists of n=41n=41 ground-truth gestures (for descriptions of the 41 ground-truth gestures, see Table S1 in the online data repository)) performed m=4m=4 times, by 4 different individual operators, for a total of 164 ground-truth recordings ij∗∈∗G_ij^* ^*. The test matrix ^m×n G_m× n consists of the same n=41n=41 gestures, but performed m=16m=16 times (4 times each by 4 operators), for a total of 656 test recordings ^ij∈ G_ij∈ G. For each test sample ^ij∈ G_ij∈ G, we match it to the ground-truth sample ij∗∈∗G_ij^* ^* that has the lowest DTW cost according to Eq. 3. We then assess performance according to: true positives (TP), when the test column j j (i.e., ^:j∈ G_:j∈ G) was judged to match the selected ground-truth column j∗j^* (i.e., :j∗∈∗G_:j^* ^*) and the judgment is correct; false positives (FP), when j j was judged to match j∗j^* but does not; and false negatives (FN), when j j was judged to not match j∗j^* but does. We assess performance according to recall Rj^R_ j , the true positive rate for test column j j, calculated as: Rj^=TPj^/(TPj^+FNj^), R_ j= TP_ j~/~( TP_ j+ FN_ j)~, (4) and precision Pj∗P_j^* for ground-truth column j∗j^*, calculated as: Pj∗=TPj∗/(TPj∗+FPj∗). P_j^*= TP_j^*~/~( TP_j^*+ FP_j^*)~. (5) Recall R measures the probability that the personalized Auto-HSI interface correctly identifies a gesture performed during the live operation phase against those demonstrated during the personalization phase. Low recall would result in robots failing to perform a behavior requested by the operator. Precision P measures the probability that a gesture selected from among those demonstrated during the personalization phase is indeed the gesture performed during the live operation phase. Low precision would result in robots performing a behavior that was not requested. Table I reports the results, with gestures (i.e., ground-truth column j∗j^* and its accompanying test column j j) categorized into four gesture types: one-handed swipes, two-handed swipes, one-handed shapes, and two-handed shapes. TABLE I: Gesture recognition performance by gesture type, with the number of ground-truth columns j∗j^* (“Qty”) and test samples i,j i,j (“Trials”) listed for each gesture type. We report recall R (Eq. 4), and precision P (Eq. 5), with their 95% confidence intervals. Recognition is tested against the full confusion matrix, i.e. each of the 656 test samples is matched against all 164 ground-truth samples, no matter the gesture type. Gesture type Qty Trials R, Eq. 4, 95% CI (%) P, Eq. 5, 95% CI (%) One-hand swipe 16 256 99.2 [97.2, 99.8] 97.7 [95.1, 98.9] Two-hand swipe 10 160 95.0 [90.4, 97.4] 93.3 [88.3, 96.2] One-hand shape 8 128 90.6 [84.3, 94.6] 90.6 [84.3, 94.6] Two-hand shape 7 112 86.6 [79.1, 91.7] 92.4 [85.7, 96.1] Overall 41 656 94.4 [92.3, 95.9] 94.4 [92.3, 95.9] The results in Table I show that swipe gestures are recognized reliably, both one- and two-handed, with one-handed swipes reaching a recall of 99.2%99.2\% (254 of 256 test samples). As expected, the more complex shape gestures, both one- and two-handed, are recognized less reliably, although performance remains high: recall drops to 90.6%90.6\% and 86.6%86.6\% respectively, the latter being the lowest value observed across gesture types. Overall, 619 of 656 test samples are classified correctly, corresponding to an accuracy of 94.4%94.4\% (95%95\% CI [92.3,95.9][92.3,95.9]). IV-B Benchmark: Code generation To assess the code generation method (Sec. I-B), we test the reliability of the robot behaviors that result from the generated SMs (including the generated SM structure and its generated code blocks). The benchmark is structured in four baseline categories: basic motion, shape deformation, shape creation, and SM modification. Each category includes 2–5 baseline SMs that were manually designed to fulfill a pre-defined behavior description, for a total of 15 baseline SMs (for descriptions of the 15 baseline SMs, see Table S3 in the online data repository). For each baseline SM, a set of test SMs is generated from a natural language description of the desired behavior. We run 18 test trials per baseline SM (a total of 270 trials), with the following variations. For each baseline SM, we test three different amounts of detail in the natural language prompt (high, mid, or low), with two different prompt versions (single-message and multi-message), for a total of 6 test prompts per baseline SM. Each test prompt is input to three different example external LLMs: DeepSeek v4 Pro, DeepSeek v4 Flash, and Gemma 4 31B. TABLE I: Percentage of the code generation trials, per experimental variation, in which the robot behaviors produced by the test SMs are sufficiently visually similar to those produced by the baseline SMs, according to the test metrics (for threshold details, see Table S2 in the online data repository). The three example external LLMs are DeepSeek v4 Pro (LLM-1), DeepSeek v4 Flash (LLM-2) and Gemma 4 31B (LLM-3). A checkmark (✓ ) indicates 100% of trials in that variation. There are 270 trials in total. Category Metric Prompt detail LLM-1 LLM-2 LLM-3 Motion (72 trials) dtrackd_ track High ✓ ✓ ✓ Mid ✓ ✓ ✓ Low ✓ ✓ ✓ Deform shape (90 trials) dshaped_ shape High ✓ ✓ ✓ Mid ✓ ✓ 80% Low 90%90\% 90%90\% 77.8%77.8\% Create shape (72 trials) dshapeλd_ shape^λ High ✓ 75% ✓ Mid 62.5%62.5\% 50%50\% 87.5%87.5\% Low 75%75\% 62.5%62.5\% 75%75\% Modify SM (36 trials) dshaped_ shape High ✓ ✓ ✓ Mid ✓ ✓ ✓ Low ✓ ✓ ✓ We compare the output of the generated and the baseline SM in numerical simulation, by comparing the target robot positions produced by the SMs, in terms of both shape and movement over time. We make the comparisons using two metrics. First, motion tracking error dtrackd_ track, which we define as the Euclidean distance between the geometric centroids of the robots’ target positions, at each time step. Second, the formation error dshaped_ shape, which compares the inter-robot distances of the robots’ target positions. To calculate dshaped_ shape, we measure the Euclidean distance between each robot position i and every other robot position j (when the distances are normalized, we denote the metric as dshapeλd_ shape^λ). For each formation, we then sort the list of distances in ascending order. The formation error dshaped_ shape then is the sum of absolute differences between the elements of the two lists. The multi-robot behaviors produced by the test SMs are sufficiently visually similar to those produced by the baseline SMs , and thus are considered successful generations of the desired multi-robot behavior, in 91.4% of the 270 trials. As seen in Table I, shape creation is the most demanding category: with medium and low levels of prompt detail, the desired robot behaviors are produced in only 68.75% of trials. This is likely attributable to shape creation involving far more degrees of freedom, compared to defining motion directions or modifying an existing shape or SM. The fact that the desired behaviors are not always produced in all trials, even when using the highest level of prompt detail, necessitates our Auto-HSI design choice to conduct the personalization phase before the live operation phase, and to include a multi-robot simulator in-the-loop in the personalization phase, to provide feedback to the operator and enable multi-shot queries until the produced robot behaviors match what is desired. IV-C Real human teleoperation of 50 simulated robots We test teleoperation of simulated robots by real human operators, using the full Auto-HSI prototype, in three different tasks: scoring a goal (Task-1), traversing a maze with narrow gaps (Task-2), and scoring two goals at the same time (Task-3). Together, the three tasks demonstrate personalized teleoperation of the robot swarm’s formation shape, motion trajectory, shape deformation, and splitting into two groups. We use 50 differential-drive ground robots, modeled after the e-puck [11] robot, in the physics-based simulator ARGoS [12]. Each task is tested: a) in the nominal condition, i.e., no noise injection; and b) with noisy motion, added using the noise parameter e=1.5e=1.5. At each time step, noise is added independently to each wheel speed w of each robot (i.e., 100 wheels in total), sampled uniformly randomly, as follows: wi←wi+|wi|⋅Uniform[−e,e]. w_i← w_i+|w_i|· Uniform[-e,e]~. (6) Each task and noise condition has been executed by three different operators (i.e., a total of 18 trials). The operators each approach the personalization phase differently, demonstrating different gestures and explaining different desired meanings in natural language, thus resulting in three distinct personalized Auto-HSI interfaces. The operators use their different personalized interfaces to complete the same tasks, but in different manners (see an example of this personalization in Fig. 5). IV-C1 Scoring a goal (Task-1) (a) Nominal condition; no noise (e=0.0e=0.0), 30 s frame interval. (b) Condition with wheel-speed noise (e=1.5e=1.5), 30 s frame interval. Fig. 2: Scoring a goal (Task-1), video timelapse: example trials of the same operator in both the (a) nominal condition and (b) noisy condition. The task is to use the robots to push a ball (dark-green circle) through an environment with obstacles, then into the goal position at the end (i.e., the small gap between two parallel walls at the right-hand boundary of the environment). (b) In the condition with wheel-speed noise, it is more difficult for the operator to retain control of the ball. At one point (see light-green dashed box), the operator loses the ball due to the wheel-speed noise and is forced to rotate the formation to recapture it. Additional time steps of this maneuver are shown in Fig. 3. (a) Move forward (155 s) (b) Rotate (167 s) (c) Move forward (176 s) (d) Rotate (189 s) (e) Swipe up (130 s) (f) Swipe right (157 s) (g) Swipe up (168 s) (h) Swipe right (178 s) Fig. 3: Highlight from Task-1 (Fig. 2(b), time 130–178 s): operator performs ball recovery to compensate for robots’ wheel-speed noise. (a–d) Zoom-in views of the area in the light-green dashed box in Fig. 2(b), with (e–h) operator gestures that triggered their motion. (a) Ball slips through a gap between robots while they move forward in response to (e) swipe up. Operator needs to turn back to retrieve the ball. Robots (b) rotate in response to (f) swipe right, then (c) move forward in response to (g) swipe up. When the operator judges the ball to be enclosed again, the task needs to be resumed: robots (d) rotate in response to (h) swipe right. Note that gestures are described according to camera view, not operator view. (a) Task-2, 30 s frame interval. (b) Task-3, 30 s frame interval. Fig. 4: Traversing a maze with narrow gaps (Task-2) and scoring two goals at the same time (Task-3), video timelapse: example trials in the nominal condition (no noise; e=0.0e=0.0). (a) In Task-2, the task is to start with a large shape and then deform it to fit through narrow gaps in the environment, until reaching the goal position at the end (at the right-hand boundary). (b) In Task-3, the task is to split the robots into two groups, then use them to push two different balls (dark-green circles) along two identical paths simultaneously, then into two different goal positions (at the right-hand boundary). In Task-1, the user needs to navigate through an arena to move a ball to a target location (example trials shown in Fig. 2). IV-C2 Traversing a maze with narrow gaps (Task-2) In Task-2, the user needs to deform the robots’ formation shape to traverse a maze with two narrow gaps and reach a target location (example trial shown in Fig. 4(a)). The operator initially forms the desired rectangular shape, then compresses it vertically when passing through a narrow gap, then compresses it horizontally to navigate between the walls to get to the next narrow gap. To navigate through the different areas of the environment, the operator must iteratively deform the shape in a trial-and-error process to get it to fit and not lose robots from getting stuck along the way (see full videos in the online data repository). IV-C3 Scoring two goals at the same time (Task-3) In Task-3, the user needs to split the robots into two groups and navigate both groups through an arena to move two balls into two target locations (example trial shown in Fig. 4(b)). IV-C4 Results of all trials (Tasks-1, -2, and -3) In all 18 trials (i.e., both with and without wheel-speed noise), the operators are able to complete the task. To summarize the comparison between the conditions with and without wheel-speed noise, Table I reports the completion time and number of utilized gestures for each of the three operators, for each noise condition. The completion time is measured from the execution of the first gesture. In the nominal condition (e=0e=0) the three operators perform comparably: completion times range from 123 s to 241 s and the number of gestures from 22 to 54, with no operator notably outperforming the other two. In the condition with wheel-speed noise (e=1.5e=1.5), performance compared to the nominal condition is similar (or sometimes a little better) in Task-2, while notably worse in Task-1 and Task-3. This can be attributed to the fact that corrective actions in Task-2 manage the robots only, while corrective actions in Tasks-2 and -3 also need to manage the ball(s) that might need to be retrieved (see Fig. 3) The performance degradation varies notably between operators in Task-1: User-1 requires 1.5×1.5× the time and 2.1×2.1× the gestures used in the nominal condition, compared to 2.0×2.0× and 3.7×3.7× for User-3 and 2.3×2.3× and 4.6×4.6× for User-2. In Task-3, User-1 and User-2 require 1.8×1.8× to 1.9×1.9× the time as in the nominal condition. User-3 shows the worst degradation, requiring 720 s and 205 gestures, compared to 125 s and 26 gestures in the nominal condition. TABLE I: Comparison of completion time and number of gestures used in the three tasks, for three different human operators (User-1, User-2, User-3). Completion time (s) Number of gestures Task Noise e User-1 User-2 User-3 User-1 User-2 User-3 Task-1 0.0 185 241 157 22 36 23 1.5 270 549 308 46 164 84 Task-2 0.0 237 205 123 49 44 23 1.5 197 186 153 44 46 27 Task-3 0.0 150 210 125 26 54 26 1.5 270 400 720 71 93 205 (a) Trial 1 of shape deformation (from purple to orange) (b) Trial 2 of shape deformation (from purple to orange) (c) Swipe right (d) Swipe outwards Fig. 5: Highlight from Task-2: two different operators’ personalized Auto-HSI interfaces. In both example trials, (a–b) the robots’ shape is being deformed in a similar manner, from approximately a square (purple) to a more elongated rectangle (orange), although two different gestures are used: (a) is produced by gesture (c) swipe right, and (b) is produced by gesture (d) swipe outwards. Note that gestures are described according to camera view, not operator view. IV-D Demo: Re-triggering personalization during operation We perform a demonstration of re-triggering the personalization phase after starting live operation. The raw videos of the demonstration are available in the online data repository. In the demo, after robot operation begins, the operator decides to stop the robots’ motion and request the addition of two new behaviors: shape deformation along the vertical axis (similar to the deformation shown in Fig. 5(a)) and along the horizontal axis. The operator then observes that both of the new gestures are producing the same robot behavior, and requests a correction by using natural language to describe that the behaviors should be different. The next code generation step fixes the newly added SM, after which the operator’s desired behaviors are performed successfully and the operator proceeds to complete the task. In the results (logs available in the online data repository), the first update step produces 1 new SM and adds 3 new nodes, with the full interaction and update taking 62 s in total, and the second update modifies one line of code, taking 25 s. IV-E Demo: Teleoperation of real robots We perform two demos with real e-puck [11] robots: one demo in which the operator forms 4 robots into an approximately square shape and navigates them through an arena (Fig. 6) and one in which the operator forms 6 robots into an approximately circle shape and deforms the shape by changing its overall size (Fig. 7). The operator interacts with the Auto-HSI prototype using a base station computer. On the base station computer, the nodes of the generated SMs send motor commands to the real robots wirelessly. (a) Starting navigation (b) Navigating around obstacle (c) Reaching goal position Fig. 6: Demo with real robots: navigating an arena. (a) Starting shape: larger circle (b) Ending shape: smaller circle Fig. 7: Demo with real robots: deforming a shape. V DISCUSSION AND FUTURE WORK A major current limitation is that robots are coordinated using external motion tracking and a central base station that issues motion commands. The most crucial future work is therefore to integrate with a swarm architecture that allows global targets to be disseminated from one robot to the rest of the swarm in a self-organized manner, while allowing the robots to maintain individual behaviors such as local collision avoidance. One architecture that delivers these capabilities is the Self-organizing Nervous System (SoNS) architecture [13]; integration should be feasible, as the SoNS already provides an operator with a single access-point for programming behaviors, and communication from a SoNS-brain to an off-board LLM (without a human in the loop) has already been demonstrated [14]. Future work could also investigate allowing a user to describe both individual robot and swarm-wide desired behaviors in natural language [15]. Additionally, the current Auto-HSI runs on a base station computer, which then sends inputs to the robots. Therefore, another important aspect of future work is to transition to more capable robot platforms that have powerful onboard cameras, microphones, and processing, so Auto-HSI can run directly on the robots without any intermediary base station computer (but still assuming communication with an off-board LLM). Using such onboard equipment with a SoNS-enabled swarm, or a swarm with similar architecture capabilities, an operator would only need to stay in view and audio range of any robot in the swarm, rather than being restricted to a specific pre-allocated robot that is responsible for human interaction. Longer-term, if smaller-size LLMs and/or onboard processing capabilities improve sufficiently to support the required code generation based on natural language audio inputs using only onboard models, then the LLM can also run onboard the robots. Finally, transferring to real-world untrained operators and application scenarios with more open-ended gesture inputs and swarm behaviors would of course require further development: including of the gesture tracking, audio processing, emergency intervention mode, and expansion of the state machine generation mechanism to cover actuation beyond formation control. Furthermore, during real-world scenarios, operators are highly unlikely to have a bird-eye view of the robot swarm, as they did in this study. Therefore, the development of mechanisms providing feedback to the user about the swarm’s state, performance, and environment would also be required. ACKNOWLEDGMENT The authors would like to thank Giuseppe Patarino and Alexandre Pacheco for help with the real-robot demos. References [1] A. Kolling, P. Walker, N. Chakraborty, K. Sycara, and M. Lewis, “Human interaction with robot swarms: A survey,” IEEE Transactions on Human-Machine Systems, vol. 46, no. 1, 2015. [2] N. Ayanian, A. Spielberg, M. Arbesfeld, J. Strauss, and D. Rus, “Controlling a team of robots with a single input,” in 2014 IEEE Int. Conf. on Robotics and Automation (ICRA), IEEE, 2014. [3] J. Alonso-Mora, S. H. Lohaus, P. Leemann, R. Siegwart, and P. Beardsley, “Gesture based human-multi-robot swarm interaction and its application to an interactive display,” in 2015 IEEE Int. Conf. on Robotics and Automation (ICRA), IEEE, 2015. [4] M. Macchini, L. De Matteïs, F. Schiano, and D. Floreano, “Personalized human-swarm interaction through hand motion,” IEEE Robotics and Automation Letters, vol. 6, no. 4, 2021. [5] G. Podevijn, R. O’Grady, Y. S. Nashed, and M. Dorigo, “Gesturing at subswarms: Towards direct human control of robot swarms,” in Conf. Towards Autonomous Robotic Systems, Springer, 2013. [6] D. Zhou and M. Schwager, “Assistive collision avoidance for quadrotor swarm teleoperation,” in 2016 IEEE Int. Conf. on Robotics and Automation (ICRA), IEEE, 2016. [7] D. Lee, A. Franchi, H. I. Son, C. Ha, H. H. Bülthoff, and P. R. Giordano, “Semiautonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles,” IEEE/ASME Transactions on mechatronics, vol. 18, no. 4, 2013. [8] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in 2023 IEEE Int. Conf. on Robotics and Automation (ICRA), IEEE, 2023. [9] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. Yong, J. Lee, et al., “Mediapipe: A framework for perceiving and processing reality,” in Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR), vol. 2019, Long Beach, CA, 2019. [10] T. Giorgino, “Computing and visualizing dynamic time warping alignments in R,” Journal of Statistical Software, vol. 31, 2009. [11] F. Mondada, M. Bonani, X. Raemy, J. Pugh, C. Cianci, A. Klaptocz, S. Magnenat, J.-C. Zufferey, D. Floreano, and A. Martinoli, “The e-puck, a robot designed for education in engineering,” in Proc. of the 9th Conf. on Autonomous Robot Systems and Competitions, vol. 1, IPCB: Instituto Politécnico de Castelo Branco, 2009. [12] C. Pinciroli, V. Trianni, R. O’Grady, G. Pini, A. Brutschy, M. Brambilla, N. Mathews, E. Ferrante, G. Di Caro, F. Ducatelle, et al., “ARGoS: a modular, parallel, multi-engine simulator for multi-robot systems,” Swarm intelligence, vol. 6, no. 4, 2012. [13] W. Zhu, S. Oğuz, M. K. Heinrich, M. Allwright, M. Wahby, A. L. Christensen, E. Garone, and M. Dorigo, “Self-organizing nervous systems for robot swarms,” Science Robotics, vol. 9, no. 96, 2024. [14] W. Zhu, M. Dorigo, and M. K. Heinrich, “Online automatic code generation for robot swarms: LLMs and self-organizing hierarchy,” arXiv:2510.04774, 2025. “Multi-Agent Cooperative Systems and Swarm Robotics in the Era of Generative AI” workshop, at IROS 2025. [15] A. Nazzari, R. Rubinacci, and M. Lovera, “TACOS: task agnostic coordinator of a multi-drone system,” Drones, vol. 10, no. 4, 2026.