Paper deep dive
A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration
Alexandra Coroiu, Andrea Vogt, Viktor Werbilo, Andreas Poppele, Johann Christensen, Sven Hallerbach
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/11/2026, 3:27:45 AM
Summary
This paper presents a scoping review and experimental study on Reinforcement Learning from Human Feedback (RLHF) for Human-Robot Collaboration (HRC). The review analyzes 20 publications (2020-2025) focusing on bidirectional, closed-loop RLHF designs, identifying challenges in safety, feedback quality, and adaptation. An empirical VR experiment compared system-initiated versus user-initiated feedback on robot proxemic behavior, finding that user-initiated feedback better captures perceived safety. The study highlights the importance of feedback timing and realistic evaluation metrics for developing safe, adaptive collaborative robots.
Entities (17)
Relation Signals (10)
Alexandra Coroiu → affiliatedwith → DLR Institute for AI Safety and Security
confidence 95% · Alexandra Coroiu alexandra.coroiu@dlr.de ... organization=DLR Institute for AI Safety and Security
Viktor Werbilo → affiliatedwith → DLR Institute for AI Safety and Security
confidence 95% · Viktor Werbilo ... organization=DLR Institute for AI Safety and Security
Johann Christensen → affiliatedwith → DLR Institute for AI Safety and Security
confidence 95% · Johann Christensen ... organization=DLR Institute for AI Safety and Security
Sven Hallerbach → affiliatedwith → DLR Institute for AI Safety and Security
confidence 95% · Sven Hallerbach ... organization=DLR Institute for AI Safety and Security
Reinforcement Learning from Human Feedback → appliedin → Human-Robot Collaboration
confidence 95% · RLHF representing a promising approach for developing safe AI-based robots in HRC systems.
User-Initiated Feedback → outperforms → System-Initiated Feedback
confidence 92% · Results show that user-initiated feedback captures perceived safety better than system-initiated feedback
Scoping Review → follows → PRISMA
confidence 90% · Following PRISMA guidelines, we screened 199 records
Scoping Review → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Human-Robot Collaboration (HRC) can facilitate mass customisation in Industry 4.0, with Reinforcement Learning from Human Feedback (RLHF) representing a promising approach for developing safe AI-based robots. Practical challenges remain regarding safety during AI development, human feedback quality, and bidirectional human-robot adaptation. We conducted a scoping review of RLHF in HRC systems, mapping methods that address these challenges. Following PRISMA guidelines, we screened 199 records and included 20 peer-reviewed publications (2020-2025) spanning multiple HRC domains. To our knowledge, this is the first review focused on the bidirectional, closed-loop design of RLHF. Our review found multiple feedback modalities enabling data collection in various feedback formats. Collected data can be integrated at different stages of AI training, resulting in a multi-step development process. Pilot experiments are commonly used to evaluate HRC systems based on both human and robot metrics. To empirically test a key gap identified in the review, we conducted a between-subjects VR experiment comparing system- and user-initiated feedback on robot proxemic behaviour for safe navigation. Using Bayesian models, we analysed the relation between the collected feedback and safety metrics: psychological safety (post-experiment questionnaire) and physical safety (inverse time-to-collision). Results show that user-initiated feedback captures perceived safety better than system-initiated feedback, indicating that feedback timing directly affects feedback quality. Our review and experiment findings show that RLHF relies on appropriate feedback methods to ensure AI safety in HRC, and future RLHF research should prioritise realistic HRC experiments evaluating the effects of feedback collection methods on relevant human and robot metrics.
Tags
Links
- Source: https://arxiv.org/abs/2610.09891v1
- Canonical: https://arxiv.org/abs/2610.09891v1
Trouble viewing inline? Open PDF directly →
Full Text
61,618 characters extracted from source content.
Expand or collapse full text
0009-0005-6643-7306Alexandra Coroiu 0009-0008-9203-2312Viktor Werbilo 0000-0001-9871-122XJohann Christensen 0009-0007-9163-0567Sven Hallerbach A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration Alexandra Coroiu alexandra.coroiu@dlr.de Andrea Vogt Viktor Werbilo Andreas Poppele Johann Christensen Sven Hallerbach organization=DLR Institute for AI Safety and Security, addressline=Wilhelm-Runge-Straße 10, postcode=89081,city = Ulm, country=Germany organization=DLR Institute for AI Safety and Security, addressline=Rathausallee 12, postcode=53757, city = Sankt Augustin, country=Germany Abstract Human-Robot Collaboration (HRC) can facilitate mass customisation in Industry 4.0, with Reinforcement Learning from Human Feedback (RLHF) representing a promising approach for developing safe AI-based robots. Practical challenges remain regarding safety during AI development, human feedback quality, and bidirectional human-robot adaptation. We conducted a scoping review of RLHF in HRC systems, mapping methods that address these challenges. Following PRISMA guidelines, we screened 199 records and included 20 peer-reviewed publications (2020-2025) spanning multiple HRC domains. To our knowledge, this is the first review focused on the bidirectional, closed-loop design of RLHF. Our review found multiple feedback modalities enabling data collection in various feedback formats. Collected data can be integrated at different stages of AI training, resulting in a multi-step development process. Pilot experiments are commonly used to evaluate HRC systems based on both human and robot metrics. To empirically test a key gap identified in the review, we conducted a between-subjects VR experiment comparing system- and user-initiated feedback on robot proxemic behaviour for safe navigation. Using Bayesian models, we analysed the relation between the collected feedback and safety metrics: psychological safety (post-experiment questionnaire) and physical safety (inverse time-to-collision). Results show that user-initiated feedback captures perceived safety better than system-initiated feedback, indicating that feedback timing directly affects feedback quality. Our review and experiment findings show that RLHF relies on appropriate feedback methods to ensure AI safety in HRC, and future RLHF research should prioritise realistic HRC experiments evaluating the effects of feedback collection methods on relevant human and robot metrics. keywords Human-Robot Collaboration ,Human-in-the-Loop ,Reinforcement Learning from Human Feedback ,AI Safety ,Human-Robot Proxemics †corresponding: Corresponding author 1 Introduction 1.1 Background Industry 4.0 has seen a surge of increasingly autonomous robots working alongside humans across various sectors, particularly in manufacturing and logistics applications. Industrial environments are transitioning from traditional isolated workcells towards Human-Robot Collaboration (HRC), where humans and robots work in shared spaces. These HRC systems combine the efficiency of automation with human flexibility, enabling robots to operate in dynamic environments [26, 48]. Such collaborative robots can be deployed for a wide range of industrial tasks: mobile robots enable navigation and transport on the work floor, and robotic arms enable variable pick-and-place operations for manipulation and assembly [28, 4]. By rapidly automating a wide range of repetitive tasks, HRC systems facilitate working with smaller, customised product batches, and allow humans to focus on higher-level decision-making. This combination of efficiency and adaptability can help address the growing need for mass customization [12, 4], offering a competitive advantage over traditional automation. 1.2 Motivation However, without barriers to separate humans from robots, safety risks pose a significant challenge for the implementation of HRC systems in industrial settings [12]. Specifically, collisions and non-functional contacts between humans and robots represent sources of injury in the shared workspace [14, 43]. This risk is particularly critical for industrial robots capable of generating high forces. Safety strategies focus on preventing hazardous contact (pre-collision strategies) and limiting the impact of the contact (post-collision strategies) [18, 48]. A series of ISO standards apply to HRC and provide guidelines for the development of safe collaborative robots. ISO 10218-2 [22] prescribes safety requirements based on the interaction characteristics between the human and the robot. Specifically, collaborative robots require inherent safety features, such as power and force limiting. The main practical challenge in developing safety methods for HRC concerns ensuring ISO compliance while preserving high system performance [43]. Therefore, balancing safety and performance becomes an optimisation problem. Besides immediate hazards like collisions, human factors represent an important latent safety concern in HRC. Human-centred design of collaborative robots should account for physical and psychological human capabilities, not only to reduce hazard risks and strain for workers [7], but also to ensure optimal system performance [13, 34, 43]. The most notable psychological factors for HRC are trust, cognitive workload, stress, and perceived safety [27, 19]. Previous research [12] identifies cognitive factors as the “most promising research challenge” for developing safe collaborative robots. The mental processes involved in HRC (i.e., perception, memory, reasoning, motor response) [7, 3] need to be carefully considered in order to neither overload nor underload workers when collaborating with robots on shared tasks [19]. These human factors are particularly relevant for ensuring safety and performance over prolonged periods of interaction with collaborative robots. Currently, there is an ongoing effort to integrate human factors in the development process of safe collaborative robots. The main goals are to design appropriate interaction modalities for HRC [26, 28, 18], and to develop adaptable robots that can accommodate the needs and limitations of workers [13, 26, 28, 7, 14]. Existing industrial collaborative robots typically rely only on predefined safety rules, lacking the ability to process real-time human data and dynamically adapt. Processing high volumes of human data remains an open challenge, due to the difficulty in interpreting complex multimodal data and extracting relevant information about human states. Solving this challenge requires new technical solutions capable of combining a variety of sensor data and computing safe robot actions accordingly. Therefore, the development of an adaptive collaborative robot solution relies on measurement methods developed for real-time monitoring of human factors in the real workspace [27, 19, 28], and advanced artificial intelligence (AI) methods that can improve robot understanding of its environment, including human agents [48, 3]. Additionally, these monitoring and AI methods must comply with existing legal and ethical frameworks, such as the the European AI Act [10] and General Data Protection Regulation [9]. 1.3 Problem Statement The field of HRC requires new AI engineering methods to develop safe collaborative robots that can adapt to complex human environments. Human-in-the-loop (HITL) approaches constitute one potential solution that helps ensure safety-by-design by directly integrating human input in the AI development process [48]. A promising AI method is reinforcement learning, which allows robots to learn from continuous interaction with their environment, including human coworkers [13, 12]. The use of HITL methods for reinforcement learning relies on diverse interaction modalities (e.g., vision, motion, language) to collect human data that is used as feedback for RL algorithms [37, 8]. These methods can support two main robot learning techniques: imitation learning and reinforcement learning from human feedback (RLHF) [8]. Imitation learning allows a robot to quickly learn a new task from human demonstrations (e.g., using recorded video samples or hand-guiding data), facilitating human-to-robot skill transfer. RLHF helps to iteratively refine a learnt task based on human feedback (e.g., through pairwise preferences or ratings), without the need to design a rule-based reward model. RLHF can be especially useful in complex environments where robots need to adapt to workers’ subjective evaluations of safety. Therefore, RLHF can contribute to increased HRC safety by integrating human factors into AI training. While RLHF represents a promising approach for developing adaptive collaborative robots, there are still practical challenges that need to be overcome before it can be confidently applied in real-world HRC systems. Firstly, an important aspect of developing HRC with RLHF methods is ensuring human safety during the development of the AI model. For this purpose, simulations have been identified as a safe and flexible tool to collect human data for training and testing the AI model before the AI-based robot is deployed in physical environments [13]. However, while simulations are useful in the early stages of development, further testing is necessary for validating the AI-based robot within real-world constraints. Secondly, all RLHF approaches rely on high-quality human data, which is limited by inherent uncertainty (e.g., unpredictable behaviours and human errors), as well as the difficulty of collecting enough data. To address this, human digital twins can be used in simulation to model human behaviour for training the AI-based robot [13, 12]. Nevertheless, collecting real human data remains essential for customising and validating the safety and performance of these methods in HRC applications. Lastly, RLHF approaches are characterised by the bidirectional adaptation between the human and the robot. As the robot adapts to human feedback, the human responds to the robot. This creates a non-stationary learning environment, where iterative feedback loops lead to reciprocal learning [35]. Therefore, in order to ensure optimal HRC system safety and performance, a closed-loop design is needed to account for both directions of the HITL approach: human-to-robot and robot-to-human [19]. Our work contributes towards closed-loop RLHF design with a scoping literature review on state-of-the-art methods, restricted to bidirectional studies, and a VR experiment comparing the effects of two feedback collection conditions on human factors and feedback quality. 2 Literature Review 2.1 Research Questions As industry shifts towards higher levels of autonomy and mass customisation, robots face increasing performance demands both in terms of efficiency and flexibility. HRC systems can help meet these performance needs, but also introduce new safety concerns in the workspace. While classic automation uses rigid constraints to ensure safety, collaborative robots should dynamically adapt to human workers to balance the safety-performance trade-off (see Figure 1). Figure 1: Requirements for adaptive collaborative robots compared to classic automation, and RLHF as a potential solution HITL approaches represent a promising solution for developing such adaptive collaborative robots. In particular, RLHF methods provide a promising paradigm to directly integrate human data into robot learning. However, the current development of RLHF methods for collaborative robots is still limited by methodological challenges regarding safety during AI development, the complexity of human feedback data, and the effects of human-robot co-adaptation. Addressing these challenges is crucial for developing RLHF applications that can be safely deployed in real-world HRC systems. Therefore, this scoping literature review aims to investigate the state-of-the-art methods used for RLHF development in HRC systems, focusing on solutions that account for the bidirectional adaptation between the human and AI-based robot. Accordingly, we structured this review around the following research questions (RQs): • RQ1: What HRC systems employ RLHF models for robot task learning? • RQ2: What feedback methods are used to collect human data for RLHF models in HRC systems? • RQ3: What AI development methods are used for training RLHF models in HRC systems? • RQ4: What metrics are used for evaluating the effects between the human and the RLHF-trained robot in HRC systems? 2.2 Methods To conduct this scoping literature review we searched for relevant publications using the following search string: (robot* OR cobot*) AND ("human in the loop" OR "human feedback" OR "user feedback" OR "human reward" OR "user reward" OR "human preference*" OR "user preference*") AND ("reinforcement learning" OR rl OR "preference learning" OR rlhf). We queried the Scopus database for journal articles and conference papers published between 2020 and 2025, reflecting the emergence of RLHF methods in robotics. The search resulted in 199 records. We screened the title and abstract of these records to select reports that match our research scope (see Figure 2). Figure 2: PRISMA flow diagram [36] of the Scopus search (2020-2025), 199 records screened, 20 included Each report was screened by two reviewers independently. We used the following relevance criteria: • RLHF is used for learning a robot specific task (e.g., navigation, manipulation); excluding general AI tasks (e.g., language processing, visual processing). • Humans directly participate in robot learning, beyond initial AI algorithm development (e.g., not limited to designing the reward model). • The AI-based robot integrates human data as feedback for training the RLHF algorithm. • Experiments involve real human participants; excluding the use of purely synthetic/simulated human data. • Metrics are evaluated for both directions of the HITL: human-to-robot effects and the robot-to-human effects. Consistent with our research questions, we recorded the following data items from each included report: application domain, specific robot task, human data measurement, feedback format, experiment environment, AI development methods, human metrics (cognitive factors and feedback behaviour), AI-based robot metrics. 2.3 Results 2.3.1 RLHF Applications The use of HITL approaches spans across a wide range of robotic applications. RLHF is used for training robots to fulfil roles in industry [2, 42, 46], assistance and care [17, 25, 29, 32, 39, 44, 45] or emergency and disaster situations [20]. Depending on the type of robot, different tasks can be trained with RLHF. Mobile robots can be fine-tuned for navigating in busy environments where robot and human paths can easily cross [23, 31, 30, 32, 38, 42, 44]. Robotic arms can be trained to perform complex tasks that require fine movements in proximity to humans. Human feedback is then used to fine-tune the RLHF policy for robotic arm navigation in a 3D space [2, 16, 24, 46, 47, 49] or for the control of the robotic grippers [17, 25, 45]. These RLHF use cases can highly benefit from close collaboration between humans and robots to enhance capabilities for flexible tasks in uncertain environments. 2.3.2 Human Feedback Methods There is a wide variety of proposed solutions for collecting human feedback to be used in RLHF algorithms (see Figure 3). These solutions encompass both direct and indirect feedback measurements, each requiring different modalities and different levels of data pre-processing. One of the most common feedback methods involves using a graphical user interface to collect subjective evaluations of the interaction with the robot [16, 17, 31, 47]. Language-based feedback represents another, more complex form of subjective measurement [45]. This method requires an intermediate natural language processing step, e.g., to evaluate the message or emotional valence in verbal or spoken text. Indirect, objective feedback measurements also require more advanced data processing to evaluate complex input from user behaviour [46] (captured through cameras and sensors) or physiological signals like brain-computer interfaces [24]. Interpreting these signals relies on statistical or AI-based analysis to extract meaningful feedback information. Lastly, multi-modal setups can be used to collect robust feedback from combinations of various measurements [29, 32]. Figure 3: Feedback methods in RLHF for robotic applications: measurement modalities and formats The types of feedback collected through the different measurement methods can vary significantly (see Figure 3). First, simple feedback formats consist of subjective preferences [6, 23, 30, 39, 44, 49, 16] and binary or scalar ratings [17, 29, 31, 32, 47, 49, 16] which can be more easily collected through GUIs. However, a key consideration in presenting preferences to users is the need to pre-select relevant pairs or groups of items for comparison [49, 6]. Second, corrections can be used to provide negative feedback for undesirable robot behaviours either as simple binary input or as a scalar value as a measure of intensity [25, 42, 45, 20]. Lastly, feedback from demonstrations [2] represents a more complex format that requires more advanced user behaviour input (e.g., from cameras, force-feedback sensors) and data processing to compare the desired actions against the robot motion and compute feedback values from this difference. As with the different feedback measurements, feedback formats can also be combined to acquire more robust feedback data [46, 38, 24]. Regardless of feedback measurement or format, collected data ultimately needs to be converted into a reward score that can be used in the RL algorithm. The key challenge with human feedback is the inherent uncertainty within human data. While some noise is expected, the design and implementation of feedback methods can influence the noise levels in multiple ways [31, 46, 49]. For example, [47] compares the noise levels between binary and scalar ratings. Similarly, when it comes to preferences, the specific options presented can also highly impact human feedback [6]. Furthermore, complex methods that require extensive data pre-processing make it harder to isolate the specific factors that shape the final reward score distribution. Another challenge with human feedback is collecting sufficiently large and heterogeneous datasets to create representative distributions that are not biased towards a certain type of feedback response [2, 30]. 2.3.3 AI Development Methods Developing RLHF algorithms for HRC generally requires a multi-step process to achieve the desired AI-based robot learning performance (see Figure 4). Using pre-training for the RL task before collecting human feedback [24, 39, 44, 45, 46] ensures a functioning initial policy that can then be fine-tuned and customised according to user input through subsequent learning sessions. RL algorithms can be pre-trained using trial-and-error approaches in simulation, allowing the robot to learn independently, or by incorporating synthetic human feedback to introduce human evaluation early in the learning process [20, 31, 30, 45]. These methods enable training on a large quantity of trials and data that would otherwise be difficult, if not impossible, to collect through real interaction with the environment and users. For modelling both the synthetic and the real human feedback, Bayesian methods can be employed to create distributions of the human data and account for feedback uncertainty [31, 46, 49]. Figure 4: RLHF development methodology for robotic applications with iterative training stages Experiments for collecting human feedback can be set up in various environments that provide different levels of realism. On the one hand, 2D or 3D computer simulations offer an easy-to-use framework for displaying AI-based robot behaviour and collecting human feedback [20, 31, 30, 38, 47, 23]. On the other hand, realistic (mock-up) environments require more space and functional physical assets [23, 32, 44, 45, 47], but offer greater realism and the opportunity to collect more representative feedback data. Additionally, an important consideration for meaningful experiments in realistic environments is the participant count and background. Pilot studies are carried out with a relatively small number of participants (<25<25) [20, 24, 25, 29, 32, 42, 44, 45, 46, 49], while fewer large-scale studies exist [31, 30, 38, 39, 47]. Furthermore, although these large-scale studies include more participants, they are not representative of the intended target groups. 2.3.4 Evaluation Metrics The human-to-robot direction within HTL concerns the effects on the AI-based robot and collaboration outcomes (see Figure 5). Within the context of RLHF for HRC applications, AI-based robot performance can be evaluated on two levels. Firstly, AI learning properties determine the suitability of the RL model for the intended task and target users. AI learning can be quantified using quality metrics such as learning efficiency [17, 20, 23, 30, 31, 44, 45], often assessed alongside adaptability [39, 32, 2, 24, 25], or explainability (especially in terms of predictability) [16]. These metrics represent AI characteristics necessary for enabling the robot to perform highly flexible tasks. Secondly, the robot’s resulting behaviour directly impacts overall HRC system outcomes. While most studies focus only on robot task performance metrics (e.g., efficiency in task execution) [16, 25, 31, 42, 46, 47, 49], others specifically focus on robot actions that impact system safety (e.g., collisions in navigation) [2, 44, 17, 20]. Figure 5: HRC safety and performance for the closed-loop RLHF design: AI-based robot, human and feedback metrics The robot-to-human direction within HTL concerns the effects on human factors and collected feedback (see Figure 5). Within the context of RLHF for HRC applications, two main aspects must be considered for evaluating human factors. Firstly, psychological factors play a central role in evaluating the human-robot dynamic. The most frequently studied factor is satisfaction [6, 25, 32, 44, 46] (typically evaluated using the System Usability Scale questionnaire [5]) often assessed alongside cognitive load [2, 16, 38, 42, 49] (typically evaluated using the standard NASA TLX questionnaire [15]). Trust [39] and perceived safety [45] are only measured in one study each. Additionally, a small number of studies account for user experience and learning effects [17, 47, 39, 42]. Secondly, behavioural aspects encompass both the primary task performance [29, 47] and the provision of human feedback, which can be considered a secondary task in the context of HRC. The most studied human feedback metric is the total number of collected data points [2, 17, 20, 23, 24, 31, 42, 46], which is highly relevant for developing a robust reward function in the RL process. Additionally, it can also be relevant to evaluate feedback timing [49] or feedback correctness [24, 38]. 2.4 Discussion 2.4.1 Main Findings For RQ1, the results reveal that a wide range of HRC applications employ RLHF for robot training: from navigation AI for disaster robots to manipulation AI tasks for assistive robot arms. The common ground among all these applications is the need to operate despite high uncertainty, either because the area is unknown (e.g., in disaster sites), users are diverse (e.g., in assistive care), or tasks change frequently (e.g., mass customisation in industry). These applications often also require very close collaboration with humans: assistive robots work in close physical contact with humans, while emergency or industrial robots need to complement human efforts. Therefore, any HRC system that requires flexibility can benefit from RLHF approaches to optimise the AI-based robot using human input. For RQ2, the results reveal a wide range of state-of-the-art technical solutions for integrating human feedback in RL algorithms. The research appears to be in an exploratory phase where various interaction tools (classic GUIs, NLP, cameras and sensors, or even BCIs) are used to collect feedback in various formats (e.g., ratings, preferences, corrections, or demonstrations). There are no universally preferred methods, as the feedback requirements can vary by application use case or AI development stage. A common issue arising across feedback methods is data uncertainty, stemming from noise and bias in human responses. Feedback strategies are often focused on maximising feedback input counts, maximising information gain from input data points, or minimising uncertainty. For RQ3, the results show that human feedback alone cannot be directly used to produce desirable RL model performance. The AI development methodology often requires pre-training in simulation and synthetic human data to train the robot initially. Additionally, when training with real human feedback data, AI methods can benefit from Bayesian modelling to represent noisy feedback distributions. Furthermore, the collection of real human feedback currently depends on low-fidelity prototypes of the intended HRC system (computer simulations or mock-up environments), which are often used to run pilot studies. These findings suggest that the development of RLHF methods requires iterative steps to train and test the AI, from low-level simulations in the first steps to realistic environments with real users in the later steps. For RQ4, several human and AI-based robot metrics have been identified for HRC systems. For the AI-based robot, metrics which reflect flexibility are most relevant: how well and how fast the AI model can learn to adapt to new users, tasks or environments. These AI metrics shape the overall robot behaviour, subsequently affecting HRC system outcomes. When it comes to human behaviour, the research is predominantly focused on the quantity of provided feedback, which is highly relevant within the context of RLHF. In terms of psychological factors, existing research is mainly focused on satisfaction, while the only safety-relevant metric commonly evaluated is cognitive load. This highlights the early exploratory nature of RLHF research in HRC, which currently focuses more on AI development rather than overall system safety and performance. 2.4.2 Limitations Two main limitations can be noted. First, this scoping literature review sits at the intersection of two related, but distinct research fields: Human-Robot Interaction and AI development with HITL methods. Constructing a search string that covers relevant RLHF methods applicable to HRC systems proved to be difficult. This is an emerging field, so standard terminology is not well established yet. HRC literature focuses on human factors and robotics using vocabulary specific to the psychology and engineering domains, whereas HITL-RLHF literature focuses on AI development using technical vocabulary specific to AI model optimisation and evaluation. Our search string captures keywords specific to our research scope, but may have omitted relevant reports that use other terminology. Furthermore, we restricted our search to the Scopus database due to its multidisciplinary coverage of both HRC and AI literature, which is well suited for this emerging field, but publications indexed only in other databases may have been missed. Second, we restricted our selection criteria to studies that included both directions of the HITL approach, assuming that a closed-loop design requires specific methods to capture the bidirectional adaptation between human and AI-based robots. Nevertheless, unidirectional studies might also contain applicable feedback and AI development methods, but these methods should be validated for applicability within bidirectional studies. 2.4.3 Future Research RLHF research for HRC can benefit from focusing solutions on target applications. When it comes to feedback formats and collection methods, tailored solutions should be developed for specific use cases [27, 28]. For example, an industrial environment with specific HRC tasks might impose various constraints on the RLHF methodology, such as environment sound and light conditions, monitoring capabilities, available interaction modalities, time and cognitive limitations, procedural standards, resource availability. These constraints limit the viability of different feedback methods and should be considered early in the design and development of RLHF strategies. Therefore, future research should attempt to integrate and test in realistic set-ups with target user groups. Another aspect of bringing RLHF research closer to real-world applications of HRC systems is focusing on system safety metrics. Currently, there is limited attention given to human factors that impact HRC safety: trust, perceived safety, and stress [27, 19]. These human factors should be evaluated in realistic environments, under different experimental conditions, and with varied feedback strategies. Particularly, cognitive load can be further explored, as the secondary feedback task can affect both safety and the overall outcome of the primary collaborative task. All these psychological factors are relevant not only for avoiding hazards in a shared human-robot workspace, but also for ensuring optimal HRC system performance. Furthermore, safety metrics should be evaluated in realistic experiments that best reflect the target HRC application. Finally, the closed-loop design [19] of RLHF methods would benefit from evaluating the impact of human-robot collaboration on the collected feedback. Human feedback behaviour can be influenced by both psychological factors and real-world system properties (e.g., environmental conditions, task requirements, available feedback methods), resulting in varying levels of noise or bias. Similarly, the AI-based robot behavioural policy, defined by prior training phases (e.g., pre-training on synthetic human data), ultimately shapes the interaction with the human and the characteristics of the provided feedback. These factors should be accounted for when modelling feedback as rewards for RLHF in order to reduce uncertainty in the collected human data. A well-structured feedback strategy should capture multiple dimensions of the collaboration, including psychological and physical safety. 3 Experiment 3.1 Research Questions RLHF applications in HRC systems rely on human data for training high-performing and safe collaborative robots. Collected feedback is characterised by both volume and quality of data. First, a high volume of feedback input is beneficial for model training, but increases effort for the human collaborator. Therefore, RLHF applications aim to have the robot learn as fast as possible with as little feedback as possible [2, 17, 49, 23, 31, 20, 24]. Feedback volume can be measured not only in terms of counts, but also as time spent per feedback query [23]. Additionally, human effort can be directly quantified as cognitive load [2, 20, 38, 49]. Second, feedback quality can be broadly defined in terms of information value for the RLHF reward model. Two key considerations that impact feedback quality are correctness [24, 38] and timing [49]. Information value per feedback point can be maximised by designing relevant feedback queries [20, 31]. The ideal feedback strategy for RLHF requires optimising the trade-off between human effort and information value obtained from feedback interactions to ensure practicality for real-world HRC systems. Binary ratings represent a fast, low-effort feedback strategy, but the lower resolution of binary signal requires more feedback queries to train the RL policy [21]. The demand for higher feedback volume could be addressed through system-initiated feedback queries at a predetermined sampling rate. In this study, we want to explore the difference between system-initiated and user-initiated binary feedback. To address both directions of a closed-loop design for HRC [19], we want to evaluate both the effects of feedback condition on the human and the effects of feedback on RLHF robot learning. Therefore, we conducted a VR experiment to answer the following research questions: • RQ5: How do system-initiated versus user-initiated binary feedback strategies compare in terms of information value? • RQ6: How do system-initiated versus user-initiated binary feedback strategies compare in terms of human effort? 3.2 Methods 3.2.1 Application Domain The experiment focuses on comparing feedback conditions for RLHF methods applied to robot navigation tasks in HRC systems. Previous research [31, 32] successfully employed RLHF to integrate human proxemic zones into robot navigation and improve human factors outcomes such as perceived safety. This is particularly relevant for industrial applications where repeated proxemic violations can decrease psychological safety for workers, which in turn degrades overall HRC performance and safety [34]. Therefore, safe robot navigation in HRC systems should consider both physical and psychological safety. Furthermore, proxemic zones are not static, but dynamically change based on robot navigation speed and interaction context [33]. RLHF solutions can more accurately capture these dynamic proxemic zones based on human input, instead of relying on AI navigation models with static proxemic rules, resulting in better safety-performance trade-offs. 3.2.2 Environment and Task The experiment was conducted in a VR environment developed in Unreal Engine (UE) version 5.4, and displayed on an HTC Vive headset to participants. The virtual environment simulated an industrial workshop area featuring a sorting station and a conveyor belt (see Figure 6). The environment featured three humanoid robots: one co-working robot and two additional robots operating in the background. The co-working robot was responsible for delivering unsorted crates to the workstation, where the human sorts the crates by object colour using the HTC controller to interact with the virtual objects. The co-working robot is then responsible for transporting the sorted crates to the conveyor belt and storing empty crates away. As the co-working robot navigates the workshop area, its path crosses with the other two robots. Robot navigation was controlled with the UE navigation AI, and collisions were avoided using safety stops when trespassing the 1m radius around other actors. Figure 6: First-person perspective of the VR environment, including the human workstation, co-working robot and background robots The human has to provide feedback on the proxemic behaviour of the co-working robot. A pop-up feedback interface displays the following question: “Did the robot keep sufficient distance to other actors?” (see Figure 7). The human provides binary feedback by choosing between the thumbs-up and thumbs-down buttons. Providing feedback represents a secondary task to the primary task of sorting crates. The two feedback conditions are designed to facilitate data collection while maintaining a smooth workflow. In the user-initiated condition, the participant is free to open the feedback pop-up interface whenever they want with the click of a button on the controller. In contrast, in the system-initiated condition, the participant is prompted to provide feedback for every third crate transported to the conveyor belt. Figure 7: Feedback pop-up prompt displaying the binary feedback buttons 3.2.3 Participants and Procedures Given the exploratory nature of this experiment, we used a convenience sampling strategy. Participants were recruited internally at the two institute locations in Germany. Written informed consent was obtained from all participants, including explanations on data protection and potential VR side effects. We collected demographic data and VR familiarity (6-item questionnaire) in terms of three dimensions: VR expertise, VR frequency, and VR confidence. A total of 48 participants completed the experiment, but four were excluded for not providing any feedback during the VR session. As a result, 44 participants (Male=30Male=30, Female=13Female=13, NonBinary=1NonBinary=1), aged between 25 and 63 years (M=33.3M=33.3, SD=8.8SD=8.8) were included in the study. Most participants reported moderate VR expertise (M=3.07M=3.07, SD=1.29SD=1.29) and confidence (M=2.92M=2.92, SD=0.97SD=0.97), despite infrequent use (M=1.55M=1.55, SD=1.09SD=1.09). The study employed a between-subjects design with two feedback conditions: user-initiated and system-initiated. Participants were randomly split between the two experimental conditions. The final distribution of participants between the user-initiated (N=23N=23) and the system-initiated (N=21N=21) feedback conditions was balanced. To compare the two experimental conditions, we collected data on three safety-relevant psychological factors [27, 19]: perceived safety (8 item questionnaire [1]), cognitive load (NASA TLX [15]), and sense of control (3 item questionnaire [41]). The experiment followed a standardised procedure: 1. Pre-survey: Participants completed the demographic data and VR familiarity questionnaires. 2. VR introduction: Participants received an introduction to the VR equipment, environment and tasks. 3. VR session: Participants performed the task in VR. The session is capped at 15 minutes. 4. Post-survey: Participants completed perceived safety, cognitive load and sense of control questionnaires. 3.2.4 Metrics and Analysis Human effort and feedback information value were quantified based on the survey results and data logs from UE collected during the VR sessions. First, human effort is defined by two metrics: the average duration (seconds) spent on a feedback prompt in the feedback interface, and normalised cognitive load computed from the post-survey data (in line with [2, 49, 20]). Second, feedback information value was defined by two dimensions: psychological and physical safety. Psychological safety was quantified by the two normalised human factors metrics computed from survey data: perceived safety and sense of control (in line with [31]). Physical safety was defined by a proxemic threat score quantified as inverse time-to-collision (iTTC = TTC−1TTC^-1) computed from data logs [40]. We computed threat scores between the co-working robot and the human, and between the co-working robot and the other two robots in the environment. The maximum threat values were extracted from the five-second window preceding each time the feedback interface was opened and feedback was provided. Bayesian models [11] were fitted to compare the two feedback conditions (see Appendix A). Given the nested data structure, we distinguish participant-level data from feedback-level data in our analysis to account for inter-individual variability and avoid pseudo-replication. First, to estimate posterior differences in human effort between the feedback conditions, Bayesian models were constructed for each metric: a Gaussian model for cognitive load (participant-level), and a log-gamma model for average feedback duration (feedback-level). Second, to estimate posterior differences in information value between the feedback conditions, Bayesian models were constructed for each safety dimension: one model for psychological safety and one for physical safety. Maximum threat to the human and to the other agents were used as predictors in a Bernoulli model with feedback-level data. The psychological safety metrics were not strongly correlated (|r|=0.41|r|=0.41) and could be used as individual predictors in a binomial model with participant-level data. 3.3 Results The system-initiated feedback led to higher feedback volume (M=9.22M=9.22, SD=0.76SD=0.76) than user-initiated feedback (M=5.24M=5.24, SD=2.4SD=2.4). Despite more frequent interaction with the feedback interface in the system-initiated condition, there were no credible differences in cognitive load (M=−0.15M=-0.15, 95% CrI [−0.78,0.48][-0.78,0.48]) or feedback duration (M=−0.10M=-0.10, 95% CrI [−0.17,0.39][-0.17,0.39]) between conditions. Therefore, the system-initiated condition provides more feedback data without additional human effort. Feedback valence was generally positive across both conditions, with system-initiated data showing particularly high values (M=0.93M=0.93, SD=0.13SD=0.13), and user-initiated values being moderately high as well (M=0.78M=0.78, SD=0.28SD=0.28). However, there are credible differences in how the two feedback conditions capture psychological safety. User-initiated feedback reflects perceived safety far better than system-initiated feedback (M=1.0M=1.0, 95% CrI [0.13,1.9][0.13,1.9]) (see Figure 8). This credible interval excludes zero, providing strong evidence of the effect, with model convergence confirmed (R^≈1.00 R≈ 1.00). Additionally, there is also a weak difference in how feedback encodes the maximum proxemic threat posed to the human (M−0.16M-0.16, 95% CrI [−0.01,0.35][-0.01,0.35]). No interaction effects were identified for sense of control or the maximum proxemic threat posed to the other robots. Full parameter estimates for the psychological and physical safety models are provided in Appendix A (Tables A.1 and A.2). Figure 8: Feedback information value for perceived safety for user- and system-initiated conditions, Bayesian posterior sigmoids (mean and 95% CrI) 3.4 Discussion The results indicate that user-initiated feedback provides greater information value than system-initiated feedback. This finding builds on previous research [32] demonstrating that real-time user feedback improves robot social behaviour. Additionally, it extends previous research [44] showing better outcomes with feedback-efficient preference learning. Despite a lower volume of feedback data points, the user-initiated condition captures more meaningful information about the psychological safety for the human co-worker. Capturing latent safety constructs is highly valuable for RLHF, because it allows the encoding of not only physical state variables, but also subjective human perception [31, 23]. Additionally, the physical safety data suggests that user-initiated feedback is more sensitive to safety violations. Therefore, these findings show that user-initiated feedback compensates for the lower data volume by providing more timely and efficient feedback. The collected feedback for the system-initiated condition was predominantly positive. This suggests that the system-initiated condition fails to capture both physical safety violations and psychological safety concerns. It appears that although human co-workers form a negative impression of perceived safety, this cannot be easily captured without timely feedback action. When prompted with the feedback interface, participants might respond positively to recent events, even though their overall safety perception is affected by proxemic violations happening outside of the immediate time window. However, two main limitations to the design of this feedback condition must be noted. First, the timing of the system-initiated prompt might influence the participants towards a positive response because it overlaps with task completion. Second, the phrasing of the feedback question might not prompt a strong negative response to proxemic violations. Because the safety stop prevents collisions, the distance between actors is always technically “sufficient”, even when it violates proxemic expectations. The findings from this study can directly inform the design of more realistic future experiments. Moving towards higher fidelity VR simulation or physical set-up requires new feedback interfaces. A pop-up prompt similar to what was used in UE can be transferred to a mounted display on a physical workstation. However, the interaction workflow changes as the human needs to switch attention from the primary physical task to a secondary digital feedback task. The design of the workflow can impact both human effort and the quality of collected feedback. Additionally, for system-initiated feedback, a different modality might be necessary for gaining the attention of the human in a busy industrial environment. More advanced feedback modalities can be employed by having the human give feedback directly to the robot. This would considerably increase the complexity of the human-robot collaboration loop, but could provide a better effort vs information value trade-off. 4 Conclusion RLHF enables the development of flexible AI-based robot applications across multiple domains where close HRC is required, such as manufacturing and logistics. Our work provides two contributions to RLHF in HRC: a scoping literature review focused on the bidirectional, closed-loop design of RLHF, and a VR experiment assessing the interplay between human factors, robot behaviour, and feedback quality. The review shows that RLHF is an emerging research field, currently exploring a wide range of methods for integrating human feedback into robot training during various stages of the AI development process. Our experimental results indicate that feedback timing is a key aspect in ensuring AI safety, as human data quality depends on immediate feedback action following a perceived safety threat. Future RLHF research would benefit from conducting studies in high-fidelity realistic settings to achieve the desired safety and performance goals for HRC systems. Particular attention should be paid to ensuring that feedback methods are viable in real environments, and able to produce high-quality data for RL algorithms without compromising human factors. Ultimately, RLHF methods have the potential to provide AI-based robots with the needed adaptability to bring HRC systems into highly variable real-world applications like Industry 4.0. References [1] N. Akalin, A. Kristoffersson, and A. Loutfi (2019) Evaluating the sense of safety and security in human–robot interaction with older people. In Social Robots: Technological, Societal and Ethical Aspects of Human-Robot Interaction, O. Korn (Ed.), Human–Computer Interaction Series, p. 237–264. External Links: ISBN 978-3-030-17106-3, Document Cited by: §3.2.3. [2] A. Avaei, L. van der Spaa, L. Peternel, and J. Kober (2023) An incremental inverse reinforcement learning approach for motion planning with separated path and velocity preferences. Robotics 12 (2), p. 61. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.4, §2.3.4, §3.1, §3.2.4. [3] L. Benos, A. Bechar, and D. Bochtis (2020) Safety and ergonomics in human-robot interactive agricultural operations. Biosystems Engineering 200, p. 55–72. External Links: Document Cited by: §1.2, §1.2. [4] Z. M. Bi, C. Luo, Z. Miao, B. Zhang, W. J. Zhang, and L. Wang (2021) Safety assurance mechanisms of collaborative robotic systems in manufacturing. Robotics and Computer-Integrated Manufacturing 67. External Links: Document Cited by: §1.1. [5] J. Brooke (1996) SUS: a ”quick and dirty” usability scale. In Usability Evaluation in Industry, P. W. Jordan, B. Thomas, B. A. Weerdmeester, and I. L. McClelland (Eds.), p. 189–194. External Links: ISBN 978-0-7484-0467-5 Cited by: §2.3.4. [6] E. Bıyık, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh (2022) Learning reward functions from diverse sources of human feedback: optimally integrating demonstrations and preferences. International Journal of Robotics Research 41 (1), p. 45–67. External Links: Document Cited by: §2.3.2, §2.3.2, §2.3.4. [7] A. Cardoso, A. Colim, E. Bicho, A. C. Braga, M. Menozzi, and P. Arezes (2021) Ergonomics and human factors as a requirement to implement safer collaborative robotic workstations: a literature review. Safety 7 (4). External Links: Document Cited by: §1.2, §1.2. [8] H. Chen, S. Li, J. Fan, A. Duan, C. Yang, D. Navarro-Alarcon, and P. Zheng (2025) Human-in-the-loop robot learning for smart manufacturing: a human-centric perspective. IEEE Transactions on Automation Science and Engineering 22. External Links: Document Cited by: §1.3. [9] European Parliament and Council of the European Union (2016) Regulation (eu) no 2016/679 … general data protection regulation. Regulation Technical Report 32016R0679, European Union. External Links: Link Cited by: §1.2. [10] European Parliament and Council of the European Union (2024) Regulation (eu) 2024/… of the european parliament and of the council laying down harmonised rules on artificial intelligence. Regulation Technical Report 32024R798, European UnionOfficial Journal of the European Union, Brussels, Belgium. External Links: Link Cited by: §1.2. [11] A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin (2013) Bayesian data analysis. 3rd edition, CRC Press. Cited by: §3.2.4. [12] A. Giallanza, G. La Scalia, R. Micale, and C. M. La Fata (2024) Occupational health and safety issues in human-robot collaboration: state of the art and open challenges. Safety Science 169. External Links: Document Cited by: §1.1, §1.2, §1.2, §1.3, §1.3. [13] A. Gonzalez-Santocildes, J. Vazquez, and A. Eguiluz (2024) Enhancing robot behavior with eeg, reinforcement learning and beyond: a review of techniques in collaborative robotics. Applied Sciences 14 (14). External Links: Document Cited by: §1.2, §1.2, §1.3, §1.3. [14] L. Gualtieri, E. Rauch, and R. Vidoni (2021) Emerging research fields in safety and ergonomics in industrial collaborative robotics: a systematic literature review. Robotics and Computer-Integrated Manufacturing 67. External Links: Document Cited by: §1.2, §1.2. [15] S. G. Hart and L. E. Staveland (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. Technical Report Technical Report NAS-98-003, NASA Ames Research Center, Moffett Field, CA. Cited by: §2.3.4, §3.2.3. [16] L. Hindemith, O. Bruns, A. M. Noller, N. Hemion, S. Schneider, and A. Vollmer (2023) Interactive robot task learning: human teaching proficiency with different feedback approaches. IEEE Transactions on Cognitive and Developmental Systems 15 (4), p. 1938–1947. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.4, §2.3.4. [17] A. Hiranaka, M. Hwang, S. Lee, C. Wang, L. Fei-Fei, J. Wu, and R. Zhang (2023) Primitive skill-based robot learning from human evaluative feedback. In IEEE International Conference on Intelligent Robots and Systems, p. 7817–7824. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.4, §2.3.4, §3.1. [18] S. Hjorth and D. Chrysostomou (2022) Human–robot collaboration in industrial environments: a literature review on non-destructive disassembly. Robotics and Computer-Integrated Manufacturing 73. External Links: Document Cited by: §1.2, §1.2. [19] S. Hopko, J. Wang, and R. Mehta (2022) Human factors considerations and metrics in shared space human-robot collaboration: a systematic review. Frontiers in Robotics and AI 9. External Links: Document Cited by: §1.2, §1.2, §1.3, §2.4.3, §2.4.3, §3.1, §3.2.3. [20] C. Huang, W. Luo, and R. Liu (2021) Meta preference learning for fast user adaptation in human-supervisory multi-robot deployments. In IEEE International Conference on Intelligent Robots and Systems, p. 5851–5856. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.3, §2.3.3, §2.3.4, §2.3.4, §3.1, §3.2.4. [21] M. Hwang, L. Weihs, C. Park, K. Lee, A. Kembhavi, and K. Ehsani (2024) Promptable behaviors: personalizing multi-objective rewards from human preferences. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p. 16216–16226. External Links: Document Cited by: §3.1. [22] International Organization for Standardization (2025) Robotics – Safety requirements – Part 2: Industrial robot applications and robot cells. Technical report Technical Report ISO 10218-2, International Organization for Standardization, Geneva, Switzerland. External Links: Link Cited by: §1.2. [23] I. Kazantzidis, T. J. Norman, Y. Du, and C. T. Freeman (2022) How to train your agent: active learning from human preferences and justifications in safety-critical environments. In International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 1654–1656. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.3, §2.3.4, §2.3.4, §3.1, §3.4. [24] S. K. Kim, E. A. Kirchner, L. Schloßmüller, and F. Kirchner (2020) Errors in human-robot interactions and their effects on robot learning. Frontiers in Robotics and AI 7, p. 558531. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.3, §2.3.3, §2.3.4, §2.3.4, §3.1. [25] T. Kumar Shastha, M. Kyrarini, and A. Gräser (2020) Application of reinforcement learning to a robotic drinking assistant. Robotics 9 (1), p. 1. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.3, §2.3.4, §2.3.4. [26] Y. Liu, G. Caldwell, M. Rittenbruch, M. Belek Fialho Teixeira, A. Burden, and M. Guertler (2024) What affects human decision making in human–robot collaboration?: a scoping review. Robotics 13 (2). External Links: Document Cited by: §1.1, §1.2. [27] M. Lorenzini, M. Lagomarsino, L. Fortini, S. Gholami, and A. Ajoudani (2023) Ergonomic human-robot collaboration in industry: a review. Frontiers in Robotics and AI 9. External Links: Document Cited by: §1.2, §1.2, §2.4.3, §2.4.3, §3.2.3. [28] L. Lu, Z. Xie, H. Wang, L. Li, and X. Xu (2022) Mental stress and safety awareness during human-robot collaboration - review. Applied Ergonomics 105. External Links: Document Cited by: §1.1, §1.2, §2.4.3. [29] M. Maroto-Gómez, M. Malfaz, J. C. Castillo, Á. Castro-González, and M. Á. Salichs (2024) Personalizing activity selection in assistive social robots from explicit and implicit user feedback. International Journal of Social Robotics. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.3, §2.3.4. [30] D. Marta, S. Holk, C. Pek, J. Tumova, and I. Leite (2023) VARIQuery: vae segment-based active learning for query selection in preference-based reinforcement learning. In IEEE International Conference on Intelligent Robots and Systems, p. 7878–7885. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.3, §2.3.3, §2.3.4. [31] D. Marta, C. Pek, G. I. Melsion, J. Tumova, and I. Leite (2022) Human-feedback shield synthesis for perceived safety in deep reinforcement learning. IEEE Robotics and Automation Letters 7 (1), p. 406–413. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.2, §2.3.3, §2.3.3, §2.3.4, §2.3.4, §3.1, §3.2.1, §3.2.4, §3.4. [32] E. McQuillin, N. Churamani, and H. Gunes (2022) Learning socially appropriate robo-waiter behaviours through real-time user feedback. In ACM/IEEE International Conference on Human-Robot Interaction (HRI), p. 541–550. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.3, §2.3.4, §2.3.4, §3.2.1, §3.4. [33] M. M.E. Neggers, R. H. Cuijpers, P. A.M. Ruijten, and W. A. IJsselsteijn (2022) The effect of robot speed on comfortable passing distances. Frontiers in Robotics and AI 9, p. 915972. External Links: Document, ISSN 2296-9144 Cited by: §3.2.1. [34] W. P. Neumann, S. Winkelhaus, E. H. Grosse, and C. H. Glock (2021) Industry 4.0 and the human factor – a systems framework and analysis methodology for successful development. International Journal of Production Economics 233. External Links: ISSN 09255273, Document Cited by: §1.2, §3.2.1. [35] S. Nixdorf, M. Zhang, E. H. Grosse, and F. Ansari (2026) Reciprocal learning in human–machine collaboration: a systematic literature review and implications for production and logistics. International Journal of Production Research 64 (4), p. 1561–1585. External Links: ISSN 0020-7543, Document Cited by: §1.3. [36] M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, L. Shamseer, J. M. Tetzlaff, E. A. Akl, S. E. Brennan, R. Chou, J. Glanville, J. M. Grimshaw, A. Hróbjartsson, M. M. Lalu, T. Li, E. W. Loder, E. Mayo-Wilson, S. McDonald, L. McGeehan, and D. … Moher (2021) The prisma 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, p. n71. External Links: Document Cited by: Figure 2. [37] M. Puerta-Beldarrain, O. Gómez-Carmona, R. Sánchez-Corcuera, D. Casado-Mansilla, D. López-de-Ipiña, and L. Chen (2025) A multifaceted vision of the human-ai collaboration: a comprehensive review. IEEE Access 13, p. 29375–29405. External Links: Document Cited by: §1.3. [38] S. A. Raza and M. Williams (2019) Human feedback as action assignment in interactive reinforcement learning. ACM Transactions on Autonomous and Adaptive Systems 14 (4), p. 1–24. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.3, §2.3.4, §3.1. [39] S. Schneider and F. Kummert (2021) Comparing robot and human guided personalization: adaptive exercise robots are perceived as more competent and trustworthy. International Journal of Social Robotics 13 (2), p. 169–185. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.3, §2.3.3, §2.3.4, §2.3.4. [40] P. T. Singamaneni, A. Favier, and R. Alami (2023) Towards benchmarking human-aware social robot navigation: a new perspective and metrics. In 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Vol. , p. 914–921. External Links: Document Cited by: §3.2.4. [41] M. J. Strube and C. Werner (1984) Personal space claims as a function of interpersonal threat: the mediating role of need for control. Journal of Nonverbal Behavior 8 (3), p. 195–209. External Links: Document Cited by: §3.2.3. [42] K. Tahara and M. Niitsuma (2025) Feedback-driven adaptive task estimation and human error handling in human-robot collaboration. In IEEE/SICE International Symposium on System Integration (SII), p. 891–896. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.3, §2.3.4, §2.3.4. [43] F. Vicentini (2020) Collaborative robotics: a survey. Journal of Mechanical Design 143 (4). External Links: Document Cited by: §1.2, §1.2. [44] R. Wang, W. Wang, and B. Min (2022) Feedback-efficient active preference learning for socially aware robot navigation. In IEEE International Conference on Intelligent Robots and Systems (IROS), p. 11336–11343. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.3, §2.3.3, §2.3.4, §2.3.4, §3.4. [45] R. Wang, D. Zhao, Z. Yuan, I. Obi, and B. Min (2025) PrefCLM: enhancing preference-based reinforcement learning with crowdsourced large language models. IEEE Robotics and Automation Letters 10 (3), p. 2486–2493. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.3, §2.3.3, §2.3.4, §2.3.4. [46] E. Yousefi, M. Chen, and I. Sharf (2025) Shared autonomy policy fine-tuning and alignment for robotic tasks. International Journal of Robotics Research 0 (0). External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.2, §2.3.3, §2.3.3, §2.3.4, §2.3.4. [47] H. Yu, R. M. Aronson, K. H. Allen, and E. S. Short (2023) From “thumbs up” to “10 out of 10”: reconsidering scalar feedback in interactive reinforcement learning. In IEEE International Conference on Intelligent Robots and Systems, p. 4121–4128. External Links: Document Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.2, §2.3.3, §2.3.4, §2.3.4. [48] A. Zacharaki, I. Kostavelis, A. Gasteratos, and I. Dokas (2020) Safety bounds in human robot interaction: a survey. Safety Science 127. External Links: Document Cited by: §1.1, §1.2, §1.2, §1.3. [49] R. Zhang, D. Bansal, Y. Hao, A. Hiranaka, J. Gao, C. Wang, R. Martín-Martín, L. Fei-Fei, and J. Wu (2023) A dual representation framework for robot learning with human guidance. In Conference on Robot Learning (PMLR), Vol. 205, p. 738–750. Cited by: §2.3.1, §2.3.2, §2.3.2, §2.3.3, §2.3.3, §2.3.4, §2.3.4, §3.1, §3.2.4. Appendix A Bayesian Statistics Bambi specification for Gaussian model on cognitive load:cognitive_load = bmb.Model( "NASATLX ~ condition", part_df, family = "gaussian" ) Bambi specification for gamma model on feedback duration:feedback_duration = bmb.Model( "fb_dur ~ condition + (1|pid)", df, family = "gamma", link = "log" ) Bambi specification for binomial model on feedback value for psychological safety:feedback_info_psychological = bmb.Model( "p(fb_sum, fb_count) ~ " "(PERCEIVED_SAFETY + SENSE_OF_CONTROL) * condition", df, family="binomial" ) Bambi specification for Bernoulli model on feedback value for physical safety:feedback_info_physical = bmb.Model( "fb_val ~ (log_max_threat_pr + max_threat_ar) * condition + (1|pid)", df, family="bernoulli", ) Table A.1: Feedback on physical safety, parameter estimates of the Bernoulli model Parameter Mean SD 95% EtiLB 95% EtiUB R-hat condition[UIF] -0.21 0.81 -1.8 1.3 1.00 max_threat_human -0.114 0.083 -0.29 0.039 1.00 max_threat_robots -1.03 1.41 -3.7 1.7 1.00 max_threat_human:condition[UIF] 0.16 0.091 -0.01 0.35 1.00 max_threat_robots:condition[UIF] -1.12 1.47 -4 1.8 1.00 Note: UIF: user-initiated feedback (reference = system-initiated feedback); Eti: equal-tailed interval; LB = lower bound, UB = upper bound. Table A.2: Feedback on psychological safety, parameter estimates of the binomial model Parameter Mean SD 95% EtiLB 95% EtiUB R-hat condition[UIF] -1.191 0.372 -1.9 -0.47 1.00 PERCEIVED_SAFETY 0.066 0.252 -0.41 0.57 1.00 SENSE_OF_CONTROL 0.357 0.303 -0.24 0.94 1.00 PERCEIVED_SAFETY:condition[UIF] 1 0.439 0.13 1.9 1.00 SENSE_OF_CONTROL:condition[UIF] -0.34 0.404 -1.1 0.46 1.00 Note: UIF: user-initiated feedback (reference = system-initiated feedback); Eti: equal-tailed interval; LB = lower bound, UB = upper bound.