Paper deep dive
Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning
Benjamin Poole, Minwoo Lee
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/10/2026, 2:38:01 AM
Summary
The paper introduces Feedback Manipulation Regularization (FMR), an algorithm-agnostic method for offline agent alignment in imitation learning. FMR leverages evaluative feedback to apply temperature scaling, redistributing policy probability mass toward human-aligned behaviors via reverse KL divergence minimization. Evaluated on Safety Gymnasium environments, FMR significantly reduces misalignment (up to 98%) across various imitation learning baselines while maintaining task aptitude, even in data-scarce and noisy demonstration regimes.
Entities (11)
Relation Signals (10)
Feedback Manipulation Regularization (FMR) → improves → Imitation Learning
confidence 95% · We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies.
Feedback Manipulation Regularization (FMR) → reduces → Misalignment
confidence 94% · demonstrating improved aptitude and up to a 98% reduction in misalignment across a range of imitation learning algorithms.
Feedback Manipulation Regularization (FMR) → utilizes → Evaluative Feedback
confidence 93% · FMR uses demonstrations to learn the state space and leverages evaluative feedback to directly manipulate the policy toward human aligned behavior via temperature scaling.
Feedback Manipulation Regularization (FMR) → evaluatedon → Safety Gymnasium
confidence 92% · Adapting various Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98% reduction in misalignment
Feedback Manipulation Regularization (FMR) → applies → Temperature Scaling
confidence 90% · FMR uses demonstrations to learn the state space and leverages evaluative feedback to directly manipulate the policy toward human aligned behavior via temperature scaling.
Feedback Manipulation Regularization (FMR) → minimizes → Reverse KL Divergence
confidence 88% · We align π_θ to a feedback-tempered surrogate of π_h by minimizing the reverse KL divergence J(θ;τ) := E_{(s,a)~D_U} [D_KL(π_θ || π_τ)]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.
Tags
Links
- Source: https://arxiv.org/abs/2607.07859v1
- Canonical: https://arxiv.org/abs/2607.07859v1
Trouble viewing inline? Open PDF directly →
Full Text
97,367 characters extracted from source content.
Expand or collapse full text
Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning Benjamin D. Poole Department of Computer Science University of North Carolina at Charlotte bpoole16@charlotte.edu Minwoo Lee Department of Computer Science University of North Carolina at Charlotte minwoo.lee@charlotte.edu Abstract Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing ap- proaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environ- ments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations. 1 Introduction Reinforcement learning (RL) has traditionally faced two major challenges: aptitude and alignment. While early RL focused on aptitude, improving an agent’s ability to learn and master a problem, recent concerns have shifted toward alignment, ensuring agents learn in ways that adhere to human values, intentions, and preferences [1]. These alignment challenges are expected to intensify as agents become more capable [2,3,4]. One prominent solution is Interactive RL (Int-RL), which integrates human input directly into the RL framework [5,6,7]. Demonstrations and feedback are of particular interest as intuitive mediums for communication. Demonstrations allow humans to explicitly depict desired behavior, while feedback (e.g., preferences) excels at refining behavior by indicating what is good, bad, or simply preferred [8, 7, 9, 10]. Recently, these two modalities have seen significant use in fine-tuning Large Language Models via RL to better align model responses with human expectations [11]. These approaches commonly employ Learning from Preferences (LfP) or Reinforcement Learning from Human Feedback (RLHF), yet rely on sequential, multi-stage pipelines that can require environment access or an additional supervised trained policy [11,12,13]. Only recent works have begun to explore combining demonstrations with feedback in a single-stage offline scenario [14, 15, 16, 17]. Despite this progress, these methods remain primarily designed within the contextual bandit framing of language generation, which does not readily translate to traditional, fully sequential decision- making environments. Consequently, only limited work has attempted to apply preference-based learning to such settings [16,17]. Yet, in terms of alignment, preference-based feedback carries several notable limitations in these fully sequential settings. Firstly, since preference is a relative judgment rather than an absolute declaration of alignment, a preferred behavior is not necessarily an Preprint. arXiv:2607.07859v1 [cs.AI] 8 Jul 2026 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR Figure 1: Imitation learning algorithms suffer from high misalignment. Our proposed method, FMR (solid lines) is able to greatly reduce misalignment for each corresponding baseline (dotted lines) without sacrificing aptitude performance. Results depict misalignment learning curves for the PathM navigation policy using a 10-50 data ratio. The shaded region represents the standard error over 5 seeds. aligned one. Secondly, this ambiguity is further compounded when comparison pairs are generated or sampled randomly, making it difficult to target specific behaviors without explicitly curating aligned and misaligned pairs. Therefore, preferences are fundamentally insufficient for aligning sequential decision-making behavior in RL settings where behavioral precision is essential. To address the limitations of preference based approaches, we turn to evaluative feedback. Inspired by TAMER and COACH [18,10], evaluative feedback operates at the state-action pair level, asking "is this behavior good or bad?" rather than "which of the two is better?". This removes the relational ambiguity of preference feedback and enables direct targeting of specific behaviors. This feedback structure motivates a reframing of how demonstrations are incorporated, which we formalize through imitation learning. Pairing evaluative feedback with demonstrations allows aligned sub-trajectories to be encouraged and misaligned behaviors to be suppressed. This formulation can further be extended to Learning from Noisy Demonstrations (LfND), where demonstrations of high alignment quality are treated as expert data and demonstrations containing noisy mixtures of aligned and misaligned behaviors are treated as imperfect data. This formulation enables the combination of demonstrations and evaluative feedback in a purely offline, single-stage pipeline for fully sequential decision-making environments. To incorporate evaluative feedback into the imitation learning framework, we introduce Feedback Manipulation Regularization (FMR), a framework that is agnostic to the choice of imitation learning algorithm. FMR uses demonstrations to learn the state space and leverages evaluative feedback to directly manipulate the policy toward human aligned behavior via temperature scaling. Adapting various Safety Gymnasium environments, we evaluate alignment by harnessing cost signals to indicate deviation from aligned behavior, providing a principled measure of misalignment [19]. We show that FMR improves alignment across a wide variety of algorithms in a limited data regime (Figure 1). Importantly, as the expert-to-imperfect data ratio shrinks, FMR helps maintain performance in both high and low data overlap scenarios where the imperfect demonstrations contain little to no helpful behaviors. Finally, FMR outperforms alternative approaches adapted to utilize evaluative feedback for reward or as a scoring signal to determine preferences between pairwise comparisons. 2 Related Work Learning for Noisy Demonstrations Common imitation learning methods include Behavioral Cloning (BC), which maps observations to actions via supervised learning, and IQ-Learn, which frames imitation as inverse soft-RL without explicit reward models [20]. Meanwhile, Learning from Noisy Demonstrations (LfND) addresses the realistic mixed-quality demonstration setting by learning from distinct expert and imperfect datasets [8]. Notable LfND methods include DemoDICE, which matches occupancy to expert data while regularizing against suboptimal data [21], and ReCOIL, which relaxes occupancy-matching to enable imitation from arbitrary off-policy data [22]. Despite outperforming standard imitation learning baselines, LfND methods frequently yield misaligned 2 policies even when expert demonstrations consist entirely of aligned behaviors, a limitation that worsens as the expert-to-imperfect demonstration ratio decreases. Interactive Reinforcement Learning Int-RL integrates human input directly into RL, typically online, in the form of advice (e.g., feedback) or demonstrations [7,8]. Classical methods like TAMER [18] and COACH [10] apply evaluative feedback at the state-action level to label behaviors as “good” or “bad”, with credit assignment enabling targeted sub-trajectory labeling. In contrast, Learning from Preferences (LfP) methods use comparative feedback on trajectory pairs to infer a reward function or policy directly [9,11,13]. While works such as [17] and [16] learn policies directly from offline preferences, this feedback is fundamentally relative rather than absolute. This means a preferred behavior is not necessarily an aligned one. When both behaviors in a comparison pair are misaligned, the least misaligned may still be preferred. Furthermore, randomly sampled comparison pairs limit the ability to target specific aligned or misaligned behaviors, meaning preference feedback provides only a coarse, relative signal ill-suited for the behavioral precision required for alignment in sequential decision-making settings. Alignment EvaluationHistorically, RL performance has focused on total return, leaving the open question of how to properly quantify agent alignment. Return-based alignment metrics [12] fail when misaligned behaviors achieve similar returns, while reward model evaluations do not generalize to methods that forgo modeling reward functions [23]. Meanwhile, SafeRL has long employed Con- strained Markov Decision Processes, where environment-elicited costs quantify constraint violations for evaluating the “safety” of behaviors [24,19]. Generalizing this idea, cost serves as a natural evaluative metric for alignment, not as a training objective, directly detecting when a policy exhibits misaligned behavior. Temperature Scaling Temperature scaling divides logits by a temperatureτ > 0before softmax, producing class probabilitiesp i = e z i /τ P j e z j /τ , and has seen wide adoption in knowledge distillation [25], calibration [26,27], and contrastive learning [28]. Two extensions are particularly relevant: adaptive temperature scaling predicts a per-input temperature for flexible adjustment [27], while vector temperature scaling assigns a distinct temperature to each class [26]. FMR takes inspiration from both, using feedback to selectively redistribute probability mass toward more aligned behaviors, but does so in probability space rather than in logit space. 3 Preliminaries We model the environment as a modified Markov Decision Process (MDP) to account for human feedbackM = (S,A,p,r,h,γ)with state spaceS, action spaceA, transition functionp :S×A→ ∆(S)over a distribution of states, reward functionr : S ×A → R, human feedback function h : S ×A → −1, 0, 1, and discount factorγ ∈ [0, 1]. The goal of imitation learning and RL is to learn a policyπ :S → ∆(A)mapping states to distributions over actions. In RL, this is done by maximizing the return,G = P ∞ t=0 γ t r(s t ,a t ) discounting the future byγ. In imitation learning,ris unknown, which means it is often estimated, explicitly or implicitly. We defineD U = D E ∪ D I as the union of the expert and imperfect datasets containing tuples without rewardM\r. Classical approaches to imitation learning such as BC aim to learn a policyπthat maps statesto action a via supervised learning by minimizing the negative log-likelihood: min π L BC (π) = min π − 1 |D U | X (s,a)∈D U logπ(a|s).(1) Policy learning methods in LfND build upon this core idea to improve robustness against noise inD I [21, 22]. Additionally, many imitation works aim to estimate the value of a state, whereV π :S → Rdenotes the state value function ofπ. The state value function estimates the expected returnV π (s) = E π G|s 0 = s when starting atsand thereafter followingπ. Likewise,Q π : S ×A → Ris the state-action value function ofπwhereQ π (s,a) = E π G|s 0 = s,a 0 = a . Although traditional value function estimation relies on rewardsr, imitation learning algorithms have developed alternative techniques for estimating value functions without explicit reward signals [21, 22, 20]. 3 4 Feedback Manipulation Regularization In this section, we introduce our proposed method of feedback manipulation regularization (FMR) for harnessing both demonstrations and feedback as an interconnected source of information. For this, the goal of FMR is to enable humans to refine an agent’s policy by providing evaluative feedbackh on demonstrations. LetD U denote an imbalanced dataset of demonstrations composed of aligned (expert) and imperfect trajectories where|D E |≪|D I |. We make the following assumption about the learned policyπ θ and latent human-aligned policy π h . Assumption 4.1 (Coverage). For any s∈S , if π h (a| s) > 0 then π θ (a| s) > 0. Sinceπ h is unknown, we approximate it usingπ θ , where feedback acts as a corrective signal that redistributes probability mass towards the underlying aligned policy. To achieve this, we take inspiration from temperature scaling for rescaling distributions. Definition 4.2 (Feedback and Temperature). Human feedbackh(s,a j )induces a per-action tempera- tureτ (s,a j )(for simplicity, denoted byτ j hereafter) via a strictly decreasing map. Hence, positiveh implies τ j < 1 (encourage), and negative h implies τ j > 1 (discourage). Remark 4.3 (Feedback-Adaptive Temperature). We propose a simple exponential equation given as: τ j = β −h if h < 0 and j = selected action, β h if h > 0 and j ̸= selected action, 1if h = 0, (2) whereβ > 1is a hyperparameter. In a discrete action space, if negative feedback is received, the action selected receivesτ > 1. If positive feedback is received, all other actions 1 , other than the action selected, receiveτ > 1. When no feedback is receivedτ j = 1, which has no impact on learning. Feedback magnitude can be achieved by using credit assignment [18] to map feedback to prior states. Repeatedly elicited feedback given in quick succession will then lead to larger magnitude and a greater redistribution of the probability mass. Definition 4.4 (Temperature Scaling for Human Aligned Policy Approximation). Givenπ θ (· | s) and τ (·| s), define the approximated human-aligned policy as π τ (a j | s) := π θ (a j | s)τ −1 j Z θ (s) , where Z θ (s) = P a k ∈A π θ (a k | s)τ −1 k . By operating directly in probability space, Definition 4.4 requires no assumptions on howπ θ is computed. Nevertheless, the temperature scaling intuition is preserved where misaligned behaviors are discouraged through redistribution of probability mass, making aligned demonstrations more probable even without explicit positive reinforcement. Withπ τ defined as a feedback-tempered surrogate forπ h , we formulate policy alignment as a distribution matching problem via reverse KL divergence, a common approach in imitation learning [21, 22]. Definition 4.5 (Reverse-KL Formulation). We alignπ θ to a feedback-tempered surrogate ofπ h by minimizing the reverse KL divergence J (θ;τ ) := E (s,a)∼D U h D KL π θ ∥π τ i . Proposition 4.6 (Reverse-KL Decomposition). For any fixed s, D KL π θ ∥π τ = E a j ∼π θ (·|s) logτ (s,a j ) + logZ θ (s). Proof.From Definition 4.4,logπ τ (a j | s) = logπ θ (a j | s)− logτ j − logZ θ (s). Substituting into the KL divergence and noting that the logπ θ (a j | s) terms cancel, X j π θ (a j | s) log π θ (a j | s) π τ (a j | s) = X j π θ (a j | s) logτ j + logZ θ (s). 1 This is akin to assigning τ < 1 to the selected action, but prevents negative loss due to the logarithm. 4 Corollary 4.7 (Generalized KL). Let the unnormalized target measure be ̃π τ (a | s) ∝ π θ (a | s)τ (s,a) −1 . Then D KL π θ (·| s)∥ ̃π τ (·| s) = E a∼π θ (·|s) logτ (s,a) + const. Remark 4.8 (Connection to Entropy Regularization). Settingτ (s,a) = π θ (a| s)recovers entropy regularization [29,30]. More generally, sinceτis derived fromh, feedback modulates the entropy of π θ where negative feedback increases entropy while positive feedback decreases it. Definition 4.9 (Feedback Manipulation Regularization). Givenτinduced by feedback via Defini- tion 4.2, the surrogate regularizer is R FMR (θ) := E s∼D U E a∼π θ (·|s) logτ (s,a) . Any imitation learning objective can incorporate FMR as L(θ) =L IL (θ) + αR FMR (θ),(3) whereL IL is the imitation learning loss and α > 0 controls alignment strength. Together, Definitions 4.2 and 4.4 specify how feedback systematically reshapesπ θ , while Defi- nition 4.9 provides a model-agnostic regularization term that enforces this alignment under any imitation learning objective. 5 Experiments This section aims to address the following questions: (1) Can FMR enhance the task alignment performance of imitation learning algorithms? (2) How does FMR compare to alternative approaches adapted to utilize evaluative feedback, and can it outperform methods that substitute feedback for reward or use it for pairwise preference scoring? We evaluate all methods across aligned-to-imperfect demonstration data ratios of 10-50 (1:5), 25-50 (1:2), and 50-50 (1:1) to test algorithmic robustness as aligned dataD E becomes increasingly scarce. Additional details regarding the task, aligned policy, and data collection, along with visualizations, are provided in Appendix B. Results for the velocity tasks, additional plots for navigation results, and extensive additional experiments are provided in Appendix C. 5.1 Experimental Setup Environments We adapt environments from Safety Gymnasium [19], built on Gymnasium [31] and MuJoCo [32], using cost as a principled measure of misalignment exclusively for evaluation. We evaluate on a 3D navigation task and three velocity-restricted locomotion tasks (Hopper, Swimmer, Walker2D), each with a discretized action space for human demonstration collection. We adapt the original navigation task so that an agent (red sphere) must reach a fixed goal (green cube) using lidar observations, spawning in a randomized upper-right area. To evaluate alignment of the navigation task, two unobservable hazard variants are introduced: passable floor hazards and blocking hazard walls, each incurring a cost of 1 per violation. For velocity tasks, a cost of 1 is incurred per step exceeding a task-specific x-velocity threshold, inducing velocity-restricted gaits as aligned policies. Aligned PoliciesWe define two aligned policies for the navigation task. PathM requires the agent to move diagonally through the center via a narrow hazard-free corridor, while PathBB requires the agent to move along the right boundary then the bottom boundary, entering the goal from below. Hazards are placed to highlight each aligned path and measure deviation, with hazard walls positioned around the goal to enforce directional entry for PathBB. For each aligned policy,D E comprises aligned human demonstrations that follow the aligned policy’s designated path. All demonstrations withinD I do not reach the goal and circle the goal to obfuscate the objective. Critically, PathM exhibits high overlap betweenD E andD I , as imperfect demonstrations frequently traverse the center corridor, while PathBB exhibits low overlap, as imperfect demonstrations rarely follow the boundary bottom path. High overlap helps to evaluate whether aligned sub-trajectories inD I can be identified from misaligned demonstrations, while low overlap looks at generalization to underrepresented aligned behaviors withinD I . Data overlap helps to evaluate how aligned sub-trajectories within otherwise misaligned demonstrations contribute to policy learning, with lower overlap making the problem more challenging due to containing fewer relevant sub-trajectories. 5 For the velocity tasks, we define one aligned policy per environment by imposing an x-velocity threshold, producing a distinctive gait that contrasts with the optimal high-speed policy. SlowHop (v ≤ 0.74) performs slow forward hops with brief pauses, SlowSwim (v ≤ 0.75) shortens the peaks of the optimal sinusoidal gait into a slower serpentine motion, and SlowWalk (v ≤ 1.50) produces a measured bipedal walk in contrast to the optimal running gait. For each velocity task,D E comprises aligned demonstrations that follow the aligned policy’s velocity-restricted gait. Swimmer data has high overlap asD E contains similar sinusoidal gait to optimal trajectories present inD I but with reduced amplitude. Hopper, by contrast, has low overlap asD I predominantly contains demonstrations exceeding the velocity threshold. Walker2D falls between the two, having medium overlap, where the real difficulty comes with the large action space (|A| = 3 6 ). Data and Feedback CollectionFor navigation data,D E comprises 50 aligned human demonstra- tions collected per aligned policy, andD I consists of 50 imperfect human demonstrations shared across policies. For velocity data, human proxy demonstrations are collected, whereD E comprises 50 aligned demonstrations from a cost-augmented SAC-Discrete agent [33,34], andD I comprises 50 demonstrations sampled across standard SAC-Discrete training, yielding variation in quality and dataset overlap. Navigation feedback is collected from a real human evaluator onD I via a visual replay interface where a 600ms credit assignment window is used to map feedback to prior states [18]. For velocity data, proxy feedback is generated by converting cost violations into negative feedback only. Swimmer and Walker2D use dense feedback per violation, while Hopper uses sparser feedback given at violation onset and for 50 subsequent steps. Walker2D additionally receives negative feedback for the 15 steps preceding failure. Baselines To assess the effectiveness of FMR in enhancing the alignment of baseline imitation learning methods trained onD U , we evaluate both traditional imitation learning and robust LfND algorithms such as BC, IQL [20], DemoDICE [21], and ReCOIL [22], with and without FMR augmentation. These represent diverse algorithmic paradigms: actor-critic (DemoDICE, ReCOIL), soft Q-learning (IQL), and supervised learning (BC). Notably, BC and IQL assume minimal imperfect demonstrations, making LfND challenging for them, while DemoDICE and ReCOIL are explicitly designed for this setting. To examine alternative approaches to harnessing evaluative feedback, we use Dual Value Learning (DVL) [22], an offline RL variant of ReCOIL, where we substitute evaluative feedback for reward. Likewise, we employ Contrastive Preference Learning (CPL) [16], a single-stage offline learning from preferences (LfP) algorithm, in which evaluative feedback serves as the comparison score for determining preference. This is achieved by calculating the total feedback accumulated per trajectory segment. Metrics To assess the aptitude performance for the navigation policies, we report success rate, defined as the ratio of episodes that reach the goal to total evaluation episodes. Alternatively, for the velocity policies, we report total normalized return where return is normalized based on the performance ofD E (Appendix B.2). For all tasks, alignment performance is measured using misalignment, defined as the ratio between the number of steps that incur a costc t and the total number of steps in the episode T as 1 T P T t=1 c t . Implementation Details All algorithms trained for 1 million batches with evaluations every 10k iterations (50 episodes each). To evaluate the final average performance and standard deviation for each metric, we compute the mean over the last 10 evaluations across all 5 random seeds. When decreasing the ratio betweenD E andD I , we find oversamplingD E helps to mitigate lose in performance due to dataset imbalance across all algorithms. For the navigation data, we remove duplicate states occurring at episode initialization. For FMR, we useβ = 10andα = 1for all algorithms unless stated otherwise. Additionally, for DemoDICE+FMR, ReCOIL+FMR and DVL, we found that they benefit from cosine learning rate decay when oversampling, while the other algorithms showed no significant benefit. For additional implementation details see Appendix A. 5.2 Can FMR enhance alignment performance? Figure 2 depicts the aptitude (i.e., success rate) and alignment (i.e., misalignment) performance results for the navigation policies PathM and PathBB across all data ratios. PathM exhibits higher baseline performance than PathBB, likely due to higher overlap betweenD E andD I . For both policies, as data ratio decrease, we observe substantial drops in success rate and corresponding increases 6 BCDemoDICEIQLReCOIL 0.0 0.2 0.4 0.6 0.8 1.0 Success Rate BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 PathMPathBB BCDemoDICEIQLReCOIL 0.0 0.2 0.4 0.6 0.8 1.0 Success Rate BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 BCDemoDICEIQLReCOIL 0.0 0.2 0.4 0.6 0.8 1.0 Success Rate BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 BCDemoDICEIQLReCOIL 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 BCDemoDICEIQLReCOIL 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 Figure 2: Navigation results across all baseline algorithms and data ratios. Results show mean success rate and misalignment scores with 95% confidence intervals, averaged over the last 10 evaluations across 5 seeds. in misalignment across all baselines. The notable exception is ReCOIL, which demonstrates the strongest baseline performance, particularly on PathM, though still at the cost of higher misalignment. FMR yields dramatic improvements across all baselines, consistently improving success rates and reducing misalignment, even as aligned demonstrations are reduced. ReCOIL+FMR achieves the strongest overall performance, combining ReCOIL’s higher success rates with drastically reduced misalignment, achieving up to 92% reduction on PathM and 67% on PathBB. Among the velocity policies, SlowSwim (high overlap data) achieves the strongest baseline perfor- mance while SlowHop and SlowWalker prove considerably more challenging (see Appendix C.2). For SlowSwim, ReCOIL achieves a return close toD E with low misalignment, while other baselines reach comparable returns but with significantly higher misalignment that grows as the data ratio decreases. In contrast, all FMR variants demonstrate remarkable stability across data ratios with returns consistently near SlowSwim and up to 98% reduction in misalignment, with ReCOIL+FMR approaching near-zero misalignment. SlowHop baselines exhibit more varied performance, with BC and IQL significantly exceedingD E performance and producing much larger misalignment scores, while misalignment worsens as the data ratio decreases across all baselines. FMR variants substantially reduce misalignment by 75% across all ratios and algorithms, with ReCOIL+FMR maintaining the best alignment. SlowWalk follows a similar but more pronounced degradation trend, with baseline misalignment remaining persistently high and returns declining at lower data ratios. For this task, adding positive feedback to all state-action pairs inD E and increasingα = 2for FMR improved performance, an effect attributed to the challenges posed by the large action space. FMR variants consistently reduce misalignment across all algorithms and ratios with ReCOIL+FMR achieving the best trade-off between return and alignment, reducing misalignment by up to 76% overall. DemoDICE+FMR at the 10-50 ratio is a notable outlier showing little reduction, likely due to the large action space and potentially addressable through better hyperparameter tuning of the base algorithm. 5.3 How Does FMR Compare to Alternative Uses of Evaluative Feedback? To evaluate FMR’s effectiveness at harnessing evaluative feedback, we adapt two popular alternative methods for comparison. Specifically, we compare FMR against DVL [22], the offline RL version of ReCOIL adapted by replacing its reward signal with evaluative feedback, and CPL [16], an offline single-stage LfP algorithm adapted by summing evaluative feedback over trajectory segments to derive preference scores between pairs. For both DVL and CPL, in addition to the evaluative feedback forD I , we add positive feedback for all state-action pairs inD E . Without this additional positive feedback, both algorithms can exhibit significant performance degradation. Results for PathM and PathBB are presented in Figure 3. On PathM, ReCOIL+FMR outperforms all alternatives across all data ratios, achieving the highest success rates with substantially lower 7 50-5025-5010-50 0.00 0.25 0.50 0.75 1.00 Success Rate CPLDVLReCOILReCOIL+FMR PathMPathBB 50-5025-5010-50 0.00 0.25 0.50 0.75 1.00 Success Rate CPLDVLReCOILReCOIL+FMR 50-5025-5010-50 0.00 0.25 0.50 0.75 1.00 Success Rate CPLDVLReCOILReCOIL+FMR 50-5025-5010-50 0.0 0.1 0.2 0.3 Misalignment CPLDVLReCOILReCOIL+FMR 50-5025-5010-50 0.0 0.1 0.2 0.3 Misalignment CPLDVLReCOILReCOIL+FMR Figure 3: Navigation policies data ratio comparison between FMR, DVL, and CPL with ReCOIL baseline. Results show mean success rate and misalignment scores with 95% confidence intervals, averaged over the last 10 evaluations across 5 seeds. misalignment. While CPL and DVL achieve moderate success rates at higher data ratios, both degrade considerably as the ratio decreases, with misalignment rising sharply at the 10-50 ratio. PathBB follows a similar trend, with all methods showing degraded performance at lower data ratios. ReCOIL+FMR again achieves the strongest overall performance, maintaining the highest success rates and lowest misalignment across all ratios. CPL performs comparably to ReCOIL at higher ratios but degrades more sharply as the data ratio decreases, while DVL consistently underperforms both, particularly at the 10-50 ratio. Across the velocity policies, DVL consistently exhibits poor misalignment despite achieving returns comparable toD E , worsening as the data ratio decreases. Notably, on SlowWalker DVL achieves better performance than both ReCOIL and CPL, though it remains substantially higher than Re- COIL+FMR. CPL performs comparably to ReCOIL+FMR on SlowSwim and SlowHop, achieving competitive returns with low misalignment, but degrades substantially on SlowWalk likely due to the higher-dimensional action space. ReCOIL+FMR demonstrates the most robust performance across all tasks and data ratios, consistently achieving the best trade-off between return and misalignment. Overall, these results demonstrate that FMR more effectively harnesses evaluative feedback than DVL or CPL. These results suggest DVL cannot properly utilize feedback, consistent with prior work indicating that naively interpreting feedback as reward is ineffective [10,35]. Moreover, CPL falls short even when using evaluative feedback to alleviate the limitations of typical preference feedback. 6 Discussion Only Negative Feedback Results for the velocity policies demonstrate that FMR learns effec- tively from negative feedback alone. Removing positive feedback from aligned navigation policies (Appendix C.4) yields negligible changes in PathM and PathBB performance, confirming negative feedback as the primary driver of FMR’s advantage. We hypothesize this is particularly critical at deviation points, where suboptimal trajectories diverge from the aligned policy, supported by SlowHop’s performance as it receives feedback exclusively at such points. Positive feedback forD E does, however, boost Walker2D results, suggesting it may play a role in high-dimensional action spaces. Impact of Imperfect Demonstrations Both navigation and velocity results reveal a consistent performance gap between high overlap policies (PathM, SlowSwim) and low overlap policies (PathBB, SlowHop) withD I . We attribute this to noisy demonstrations inD I , which provide limited learning signal that feedback alone cannot fully correct. For instance, in PathBB, the majority ofD I trajectories are misaligned, with only a small subset reflecting the aligned path. This underscores that imperfect demonstrations must retain some relevance to aligned behavior for FMR to be effective. While FMR mitigates noise through behavior reweighting, its effectiveness is ultimately constrained by data quality. WhenD I consists predominantly of irrelevant sub-trajectories, feedback on those segments 8 becomes uninformative, explaining the observed performance decline asD E shrinks in low overlap tasks, where the agent receives fewer aligned demonstrations with little meaningful feedback from D I to compensate. We also examine whether FMR can improve performance using imperfect demonstrations whenD E andD I have high overlap. Comparing ReCOIL+FMR trained onD U with a 10-50 demonstration split against BC and IQL trained exclusively onD E (Appendix C.5), ReCOIL+FMR consistently outperforms or matches both baselines across high overlap tasks. This demonstrates that feedback enables learners to benefit from imperfect data when it is structurally aligned with the target behav- ior. Results on low overlap tasks further corroborate that FMR is most effective when imperfect demonstrations meaningfully intersect with the aligned behavior. Feedback Scalability To assess feedback scalability, we reduce the proportion of demonstrations inD I receiving feedback, under a 50-50 data ratio, by randomly selecting a subset of trajectories and dropping all its feedback (see Appendix C.6). For navigation policies, ReCOIL+FMR exhibits a gradual performance decay as feedback coverage decreases, bounded below by baseline ReCOIL. At 20% feedback coverage performance marginally exceeds ReCOIL, while 50% coverage yields a slight degradation relative to full feedback. This decay is likely amplified by random feedback selection, as dropping all feedback for underrepresented or misaligned trajectories risks reinforcing poor behaviors, further motivating targeted feedback at points of deviation. Among velocity policies, only SlowHop follows this trend, while SlowSwim and SlowWalk appear unaffected. Although SlowHop already has strong baseline performance, almost all trajectories inD I follow the same initial path, meaning that feedback must be provided in sufficient quantity to override the dominant behavioral prior in the data. A promising direction for future work is the identification of similar state-action pairs to generalize feedback across regions of the state-action space, reducing the feedback burden [36]. Generalized Feedback-Adaptive Temperature All evaluated environments employ discretized action spaces to facilitate the collection of human demonstrations. We therefore consider a natural extension of FMR to continuous action spaces, generalizing feedback-adaptive temperature (Eq. (2)) toτ gen (s,a) = β −h , hereafter referred to as generalized feedback-adaptive temperature. In this formulation, when positive feedback is provided, the temperature adjustment is applied exclusively to the selected action. Positive feedback then yieldsτ gen (s,a) < 1, resulting in a negative loss that reduces interpretability. Preliminary experiments in discrete action spaces reveal negligible performance differences across all tasks (see Appendix C.7). As FMR operates in probability space and is therefore model-agnostic, these findings suggest thatτ gen facilitates straightforward extension to continuous action spaces. 7 Conclusion We propose Feedback Manipulation Regularization (FMR), a method that leverages the rich, inter- connected information contained in demonstration feedback to improve alignment performance and, in many cases, aptitude performance in an offline setting. FMR is compatible with any imitation learning algorithm, functioning analogously to human controlled entropy regularization. Specifically, feedback modulates the entropy of state–action pairs, whereby negative feedback increases entropy to discourage undesirable actions, while positive feedback decreases entropy to reinforce preferred behaviors. Through extensive experiments, we demonstrate that FMR consistently enhances a variety of imitation learning baselines in a limited data regime, yielding policies that better align with human intentions. Limitations and Future Work FMR’s primary limitation is its scalability, though it already demonstrates improved scalability over alternative methods via its superior performance. This scalability nonetheless remains dependent on task complexity and the dynamics betweenD E andD I . Natural future directions include generalizing feedback across similar state-action pairs to reduce annotation burden [36], and providing targeted feedback forD I at points which deviate from the aligned policy to improve efficacy. Additionally, while this work is limited to discrete action spaces, the τ gen formulation suggests a principled extension to continuous action spaces. 9 References [1]Jiaming Ji, Tianyi Qiu, Boyuan Chen, Jiayi Zhou, Borong Zhang, Donghai Hong, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Lukas Vierling, Zhaowei Zhang, Fanzhi Zeng, Juntao Dai, Xuehai Pan, Hua Xu, Aidan O’Gara, Kwan Ng, Brian Tse, Jie Fu, Stephen Mcaleer, Yanfeng Wang, Mingchuan Yang, Yunhuai Liu, Yizhou Wang, Song-Chun Zhu, Yike Guo, Yaodong Yang, and Wen Gao. AI Alignment: A Contemporary Survey. ACM Computing Survey, 58(5):132:1–132:38, November 2025. [2] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction, 2018. [3]Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety, July 2016. [4]Gabriel Dulac-Arnold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, bench- marks and analysis. Machine Learning, 110(9):2419–2468, September 2021. [5]Guangliang Li, Randy Gomez, Keisuke Nakamura, and Bo He. Human-Centered Reinforcement Learning: A Survey. IEEE Transactions on Human-Machine Systems, 49(4):337–349, August 2019. [6]Christian Arzate Cruz and Takeo Igarashi. A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, pages 1195–1209. Association for Computing Machinery, July 2020. [7]Anis Najar and Mohamed Chetouani. Reinforcement Learning With Human Advice: A Survey. Frontiers in Robotics and AI, 8:74, 2021. [8] Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges. IEEE Transactions on Cybernetics, 54(12):7173–7186, December 2024. [9]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. [10]James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning, volume 70, pages 2285–2294. JMLR.org, 2017. [11]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730– 27744, 2022. [12]Xu Liu, Haobo Fu, Stefano V. Albrecht, QIANG FU, and Shuai Li. Online-to-offline rl for agent alignment. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Representation Learning, volume 2025, pages 56116–56136, 2025. [13] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc., 2023. [14] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems, 2024. 10 [15]Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization without reference model. In Conference on Empirical Methods in Natural Language Processing, pages 11170–11189. Association for Computational Linguistics, November 2024. [16]Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In International Conference on Learning Representations, 2024. [17]Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. In Advances in Neural Information Processing Systems, volume 36, pages 18806–18827. Curran Associates, Inc., 2023. [18] W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: the TAMER framework. In Proceedings of the fifth international conference on Knowledge capture, pages 9–16. Association for Computing Machinery, September 2009. [19] Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Advances in Neural Information Processing Systems, volume 36, pages 18964–18993. Curran Associates, Inc., 2023. [20]Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. IQ- Learn: Inverse soft-Q Learning for Imitation. In Advances in Neural Information Processing Systems, volume 34, pages 4028–4039. Curran Associates, Inc., 2021. [21]Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2022. [22]Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual RL: Unification and new methods for reinforcement and imitation learning. In The Twelfth International Conference on Learning Representations, 2024. [23]Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation, 2025. [24]Javier García, Fern, and O Fernández. A Comprehensive Survey on Safe Reinforcement Learning. Journal of Machine Learning Research, 16(42):1437–1480, 2015. [25]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. [26]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1321–1330, 2017. ISSN: 2640-3498. [27] Johnathan Xie, Annie S Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. Calibrating language models with adaptive temperature scaling. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18128–18138. Association for Computational Linguistics, November 2024. [28]Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2495–2504, 2021. [29]Ronald J. Williams and Jing Peng. Function Optimization using Connectionist Reinforcement Learning Algorithms. Connection Science, 3(3):241–268, January 1991. [30]Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928–1937, New York, New York, USA, 20–22 Jun 2016. PMLR. 11 [31]Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A standard interface for reinforcement learning environments, 2025. [32] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, October 2012. ISSN: 2153-0866. [33]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861–1870, 10–15 Jul 2018. [34] Petros Christodoulou. Soft Actor-Critic for Discrete Action Settings, 2019. [35] Andrea L. Thomaz and Cynthia Breazeal. Reinforcement learning with human teachers: evidence of feedback and guidance with implications for learning performance. In Proceedings of the 21st national conference on Artificial intelligence, pages 1000–1005. AAAI Press, July 2006. [36] Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets. In Robotics: Science and Systems XIX. Robotics: Science and Systems Foundation, July 2023. 12 A Implementation Details A.1 Data Preprocessing For all tasks, we oversampleD E as the number of demonstrations decreases. Oversampling is performed so that the number of samples inD E is close to, or slightly lower than, that inD I . Oversampling is done by repeating entire trajectories, and we typically oversampleD E to the size of D I (i.e., 50 demonstrations). The exceptions are SlowHop and SlowWalk, where the oversample target is 40 demonstrations to preventD E from exceeding the number of samples inD I . Additionally, for the navigation task, we removed duplicate starting states. These duplicates arise from a delay in the human beginning their demonstration, which creates repeated states within the first 120 steps. A.2 Algorithms All algorithm networks are implemented in PyTorch using two-layer neural networks with 256 hidden units and ReLU activation. During training, algorithms are trained for 1 million batches (i.e., training iterations) using a learning rate of3× 10 −4 and a batch size of 128. Evaluations are conducted every 10k iterations and consist of 50 episodes each. To evaluate the final average performance and standard deviation for each metric, we compute the mean over the last 10 evaluations from training, across all 5 random seeds. Algorithm hyperparameters are selected based on author recommendations or limited searches. Algorithm-specific hyperparameters and details are provided below. FMR Appendix C.8 depicts the search over hyperparameters using PathBB ratio 10–50:β ∈ 1.5, 10, 100, 1000 ,α ∈ 0.1, 1, 10, 100, and credit assignment∈ 1.5, 10, 100, 1000. For all tasks, unless stated otherwise, we set the hyperparameters as follows:β = 10for the feedback- adaptive temperature,α = 1for the regularization strength (for both actor and critic), and the credit assignment window is set to 600 ms. When oversampling we found using cosine learning rate decay to be helpful for DemoDICE+FMR and ReCOIL+FMR. For Walker2D, we found that using positive feedback for all expert state-action pairs inD E and a slightly higherα = 2led to better performance due to the large action space. Although feedback is not typically provided forD E , we observed that positive feedback for Walker2D helped elevate aligned actions. While adding such feedback is trivial, it was generally unnecessary in spaces with smaller action dimensions. For critic-based algorithms (e.g., IQL, DemoDICE, ReCOIL), FMR can be applied to the critic under the assumption of a discrete action space. This requires transforming the value function into probability space using a function such as softmax. However, the softmax function is highly sensitive to input scale. When value function outputs have large magnitudes, the softmax produces overly peaked distributions that concentrate probability on a single action, likely limiting the ability to modulate entropy. For actor-critic algorithms, we apply FMR to both the actor and the critic. IQL Following [20], our implementation of IQL uses soft Q-learning to test FMR’s ability to manipulate a critic-only algorithm. We implement IQL as a critic-only algorithm to verify FMR’s ability to work with critic-only algorithms in discrete spaces. A hyperparameter search was performed over the reward loss0.5, 10using PathM and all ratios. We set the following hyperparameters for all tasks: χ 2 regularization for the reward loss is 10, and the soft value temperature is 1. DemoDICEOur implementation of DemoDICE follows [21]. We set the following hyperparame- ters for all tasks: the discount factor is0.99, and the gradient penalty coefficients for the discriminator (i.e., cost) and critic are 0.1 and 1× 10 −4 , respectively. ReCOILOur implementation of ReCOIL follows [22] using actor-critic with target networks. We set the following hyperparameters for all tasks: the discount factor is0.99, actor temperature is0.1, value temperature is1, max clip is7, reward gap is2, target Q gap is200, and the target network is updated using a value of 5× 10 −3 . DVLOur implementation of DVL follows [22] using actor-critic with target networks. As DVL is the offline RL variant of ReCOIL that additionally accounts for reward, we adopt largely the same hyperparameters used for ReCOIL across all tasks: aχ 2 -based value loss, discount factor of0.99, actor temperature of0.1, value temperature of1, max clip of5, and a target network update rate of 13 5× 10 −3 . To improve performance across all tasks, we set positive feedback for all expert state-action pairs in D E . When oversampling we found using cosine learning rate decay to be helpful. CPL Our implementation of CPL follows [16]. To select hyperparameters, we performed a grid search over the following values: segment length16, 32, 64, segment count20k, 40k, contrastive bias0.25, 0.5, 0.75, 0.85, 0.95, and temperature0.1, 0.3, 0.5, 1. Based on performance across the PathM, PathBB, and Walker2D tasks across various ratios, we selected a segment length of64, a segment count of 20k, a contrastive bias of 0.85, and a temperature of 0.1. We use a discount factor of1and dense preference comparisons as recommended in [16]. The preference score is computed using the sum of evaluative feedback per trajectory segment. To improve performance across all tasks, we set positive feedback for all expert state-action pairs in D E . A.3 Computing Resources All experiments are run in a SLURM cluster environment across five different types of GPU nodes: NVIDIA A40 with an Intel Xeon Gold 6326 CPU, NVIDIA A100 with an Intel Xeon Gold 6326 CPU, NVIDIA A100 with an AMD EPYC 7502 CPU, NVIDIA L40s with an Intel Xeon Gold 6338 CPU or Intel Xeon Platinum 8362 CPU, and NVIDIA H200 NVL with a dual 64-core AMD EPYC 9555 CPU. All nodes used 40 GB of RAM. 14 B Task and Data Collection Details This section provides various visualizations of the different tasks, along with their corresponding trajectories and feedback. B.1 Navigation Policies Figure 4 depicts the navigation task environment for the aligned PathM and PathBB policies. Blue areas represent hazards, the red sphere with a blue box (indicating the front) represents the agent, and the green cube represents the goal. The yellow walls mark the environment boundaries. These are observable to the agent and do not allow the agent to pass through them. When the agent intersects with a hazard, a red sphere will encompass the agent, indicating it is violating a constraint. This is purely a visual effect for observers. Data CollectionFor the navigation policies, human demonstrations are collected for bothD E and D I . For each aligned policy, 50 expert demonstrations are collected forD E with zero cost violations and 100% success rate, while 50 imperfect demonstrations are collected forD I by having humans explore the state space without reaching the goal. Figure 5 visualizes a 2D slice of the environment and trajectories for both aligned navigation policies. The green lines indicate the expert trajectories for each respective policy. The gray lines (shared across policies) represent the imperfect trajectories. The majority ofD I demonstrations move diagonally through the center (similar to PathM), while a minority follow the top or right boundary before moving toward the goal (similar to PathBB), with all demonstrations circling the goal to obfuscate the objective. Human feedback is collected by replaying demonstrations through a visual interface, where negative or positive feedback can be provided via keyboard input. When a feedback key is pressed, the feedback is mapped to the nearest state. Feedback is collected only forD I , as it contains demonstrations requiring correction. Policy-specific feedback is collected for PathM and PathBB. The feedback methodology is defined such that the human evaluator provides positive feedback when the agent follows or moves toward the aligned policy, and negative feedback when it diverges or appears to be diverging. Figure 6 displays a heat map of the human feedback forD I , provided with respect to each aligned navigation policy. Notice that feedback for Figure 6a encourages the agent to move diagonally towards the goal or back towards the middle of the environment (indicated by green positive feedback). Likewise, Figure 6b encourages the agent to move towards the bottom path, although there are few trajectories that do so. 15 (a) PathM(b) PathBB Figure 4: Example of navigation task environment setups. 3210123 x-coordiante 3 2 1 0 1 2 3 y-coordiante (a) PathM 3210123 x-coordiante 3 2 1 0 1 2 3 y-coordiante (b) PathBB Figure 5: Trajectories for the navigation aligned policies depicted as a 2D slice of the environment. The green lines represent expert trajectories, the gray lines represent imperfect trajectories, and the blue areas represent hazards. 3210123 x-coordiante 3 2 1 0 1 2 3 y-coordiante 9 6 3 00 1 2 3 Density (a) PathM 3210123 x-coordiante 3 2 1 0 1 2 3 y-coordiante 13.00 8.67 4.33 0.000.00 2.00 4.00 6.00 Density (b) PathBB Figure 6: Feedback heatmaps ofD I demonstrations for the aligned navigation policies with credit assignment. Color is determined by the feedback density in a particular area in the state space. 16 B.2 Velocity Policies Figure 7 depicts the velocity task environment setups for Swimmer, Hopper, and Walker2D. The sphere above each agent represents whether the agent is exceeding the velocity threshold (this is for visualization purposes only). If the sphere is green, the agent is under the threshold. If the sphere is red, the agent is exceeding the threshold. Data CollectionFor the velocity policies, proxy demonstrations are collected for bothD E andD I using SAC-Discrete. Our implementation of SAC-Discrete follows [34]. For all velocity tasks, we used a two-layer neural networks with 256 hidden units and ReLU activation for the actor and critic networks. Additionally, we used a learning rate of3× 10 −4 for the actor and critic and1× 10 −3 for entropy learning. For Swimmer, we used a discount factor of0.995to reach near optimal performance. The SAC-Discrete agent for Swimmer and Hopper is trained for 5 millions steps while Walk2D is trained for 10 million due to the large action space slowly down learning. For Hopper and Swimmer, samples forD I are collected every 1 million steps while for Walker2D samples are collected at steps 2, 3, 4, 5 and 8 million. The sampling for Walker2D is done to ensure aD I contains samples that have lower overlap with D E but are of higher quality. Figure 8 depicts the expert and the imperfect demonstrations for SlowSwim where the blue dotted line indicates the velocity threshold. Figure 9 depicts the expert and the imperfect demonstrations for SlowHop where the imperfect demonstrations only cross the threshold. Figure 10 depicts the expert and the imperfect demonstrations for SlowWalk which demonstrations act as a middle ground between SlopSwim and SlowHop. For each velocity aligned policy, demonstrations inD E do contain some cost violations. The performance of the expert demonstrationsD E for SlowSwim has a return of283with a misalignment score of8× 10 −5 . SlowHop has a return of1451with a misalignment score of0.01. SlowWalk has a return of2147with a misalignment score of8× 10 −3 . Each policy’sD E return is normalized based on these returns. Feedback is generated using an oracle instead of a human. Oracle feedback is generated by converting the cost into negative feedback, meaning no positive feedback is provided by the oracle. Oracle feedback is used for two reasons: first, to evaluate the viability of synthetic feedback that might be required when human feedback is difficult to provide; second, to evaluate the effectiveness of using only negative feedback. 17 (a) Swimmer(b) Hopper(c) Walker2D Figure 7: Example of velocity task environment setups. 024681012 x-coordiante 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 x-velocity (a) Expert 024681012 x-coordiante 2 1 0 1 2 x-velocity (b) Imperfect Figure 8: Trajectories for Swimmer plotted using coordinate and x-velocity. The blue dotted line represents the 0.74 velocity threshold. 0.00.51.01.52.02.53.03.5 x-coordiante 0.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0 x-velocity (a) Expert 05101520 x-coordiante 0 1 2 3 4 5 x-velocity (b) Imperfect Figure 9: Trajectories for Hopper plotted using coordinate and x-velocity. The blue dotted line represents the 0.75 velocity threshold. 18 02468 x-coordiante 1.5 1.0 0.5 0.0 0.5 1.0 1.5 x-velocity (a) Expert 0510152025 x-coordiante 4 2 0 2 4 6 8 x-velocity (b) Imperfect Figure 10: Trajectories for Walker2D plotted using coordinate and x-velocity. The blue dotted line represents the 1.5 velocity threshold. 19 C Extended Results This section provides a collection of additional experimental results mentioned throughout the paper. All learning curves results are averaged over 5 seeds and smoothed using a moving average of 10. The shaded region represents the standard error. C.1 Navigation Task The following are the results for the navigation tasks. Table 1: PathM performance comparison across algorithms and data ratios. Results show mean ± std for the last 10 evaluations, over 5 seeds. BCIQLDemoDICEReCOIL Variant RatioReturnMis.ReturnMis.ReturnMis.ReturnMis. Base 50-50 0.45 ± 0.50 0.37 ± 0.31 0.46 ± 0.50 0.36 ± 0.31 0.54 ± 0.50 0.33 ± 0.33 0.95 ± 0.22 0.11 ± 0.21 25-50 0.37 ± 0.48 0.42 ± 0.31 0.39 ± 0.49 0.42 ± 0.31 0.43 ± 0.50 0.39 ± 0.33 0.92 ± 0.28 0.12 ± 0.22 10-50 0.31 ± 0.46 0.46 ± 0.29 0.30 ± 0.46 0.47 ± 0.29 0.33 ± 0.47 0.45 ± 0.30 0.86 ± 0.35 0.15 ± 0.24 FMR 50-50 0.87 ± 0.34 0.10 ± 0.23 0.87 ± 0.33 0.09 ± 0.22 0.94 ± 0.24 0.04 ± 0.15 0.99 ± 0.11 0.01 ± 0.08 25-50 0.88 ± 0.32 0.10 ± 0.22 0.81 ± 0.39 0.14 ± 0.26 0.87 ± 0.33 0.07 ± 0.18 0.97 ± 0.16 0.01 ± 0.08 10-50 0.81 ± 0.39 0.15 ± 0.25 0.70 ± 0.46 0.21 ± 0.30 0.87 ± 0.34 0.09 ± 0.21 0.95 ± 0.22 0.02 ± 0.09 Table 2: PathBB performance comparison across algorithms and data ratios. Results show mean ± std for the last 10 evaluations, over 5 seeds. BCIQLDemoDICEReCOIL Variant RatioReturnMis.ReturnMis.ReturnMis.ReturnMis. Base 50-50 0.46 ± 0.50 0.31 ± 0.32 0.48 ± 0.50 0.30 ± 0.33 0.58 ± 0.49 0.27 ± 0.35 0.78 ± 0.41 0.15 ± 0.31 25-50 0.34 ± 0.47 0.39 ± 0.34 0.38 ± 0.48 0.36 ± 0.33 0.43 ± 0.50 0.38 ± 0.39 0.72 ± 0.45 0.15 ± 0.30 10-50 0.25 ± 0.43 0.44 ± 0.30 0.25 ± 0.43 0.43 ± 0.33 0.31 ± 0.46 0.46 ± 0.36 0.59 ± 0.49 0.24 ± 0.33 FMR 50-50 0.71 ± 0.45 0.16 ± 0.27 0.65 ± 0.48 0.20 ± 0.29 0.84 ± 0.37 0.13 ± 0.25 0.92 ± 0.27 0.05 ± 0.17 25-50 0.65 ± 0.48 0.20 ± 0.31 0.57 ± 0.49 0.26 ± 0.33 0.76 ± 0.43 0.22 ± 0.33 0.88 ± 0.32 0.07 ± 0.19 10-50 0.47 ± 0.50 0.33 ± 0.34 0.40 ± 0.49 0.38 ± 0.36 0.67 ± 0.47 0.26 ± 0.35 0.71 ± 0.45 0.10 ± 0.21 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR Figure 11: PathM learning curves for baselines and FMR. The shaded region represents the standard error. 20 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR Figure 12: PathBB learning curves for baselines and FMR. The shaded region represents the standard error. 21 C.2 Velocity Tasks The following are the results for the velocity tasks where return is normalized based onD E perfor- mance. BCDemoDICEIQLReCOIL 0.00 0.25 0.50 0.75 1.00 Return BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 SwimmerHopperWalker2D BCDemoDICEIQLReCOIL 0.00 0.25 0.50 0.75 1.00 Return BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 BCDemoDICEIQLReCOIL 0.0 0.5 1.0 1.5 Return BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 BCDemoDICEIQLReCOIL 0.00 0.25 0.50 0.75 1.00 Return BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 BCDemoDICEIQLReCOIL 0.0 0.1 0.2 Misalignment BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 BCDemoDICEIQLReCOIL 0.0 0.2 0.4 0.6 Misalignment BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 BCDemoDICEIQLReCOIL 0.0 0.2 0.4 0.6 Misalignment BaseFMRRatio=50-50Ratio=25-50Ratio=10-50 Figure 13: Velocity results across all baseline algorithms and data ratios. Results show mean success rate and misalignment scores with 95% confidence intervals, averaged over the last 10 evaluations across 5 seeds Table 3: SlowSwim performance comparison across algorithms and data ratios. Results show mean ± std for the last 10 evaluations, over 5 seeds. The ∗ symbol indicates a very small non-zero number. BCIQLDemoDICEReCOIL Variant RatioReturnMis.ReturnMis.ReturnMis.ReturnMis. Base 50-50 0.96 ± 0.18 0.15 ± 0.15 0.97 ± 0.15 0.14 ± 0.15 1.00 ± 0.050.04 ± 0.100.99 ± 0.07 0.00 ∗ ± 0.02 25-50 0.95 ± 0.20 0.18 ± 0.15 0.95 ± 0.19 0.17 ± 0.15 1.00 ± 0.080.09 ± 0.140.99 ± 0.08 0.00 ∗ ± 0.02 10-50 0.93 ± 0.23 0.23 ± 0.14 0.96 ± 0.17 0.21 ± 0.14 0.98 ± 0.140.16 ± 0.150.97 ± 0.150.02 ± 0.05 FMR 50-50 0.97 ± 0.13 0.02 ± 0.04 0.97 ± 0.09 0.03 ± 0.07 0.99 ± 0.08 0.00 ∗ ± 0.02 1.00 ± 0.01 0.00 ∗ ± 0.00 ∗ 25-50 0.95 ± 0.14 0.03 ± 0.06 0.99 ± 0.07 0.03 ± 0.05 1.00 ± 0.030.01 ± 0.041.00 ± 0.01 0.00 ∗ ± 0.00 ∗ 10-50 0.99 ± 0.07 0.03 ± 0.05 0.95 ± 0.13 0.05 ± 0.08 1.00 ± 0.060.01 ± 0.031.00 ± 0.01 0.00 ∗ ± 0.00 ∗ Table 4: SlowHop performance comparison across algorithms and data ratios. Results show mean ± std for the last 10 evaluations, over 5 seeds. BCIQLDemoDICEReCOIL Variant RatioReturnMis.ReturnMis.ReturnMis.ReturnMis. Base 50-50 1.18 ± 0.51 0.43 ± 0.45 1.18 ± 0.49 0.45 ± 0.45 0.98 ± 0.28 0.17 ± 0.32 1.01 ± 0.18 0.05 ± 0.16 25-50 1.26 ± 0.55 0.51 ± 0.45 1.27 ± 0.55 0.58 ± 0.43 0.94 ± 0.28 0.24 ± 0.37 1.07 ± 0.31 0.14 ± 0.30 10-50 1.37 ± 0.61 0.65 ± 0.40 1.31 ± 0.59 0.65 ± 0.40 1.06 ± 0.43 0.42 ± 0.41 1.07 ± 0.34 0.16 ± 0.30 FMR 50-50 1.02 ± 0.31 0.13 ± 0.28 1.05 ± 0.38 0.19 ± 0.34 0.95 ± 0.19 0.06 ± 0.16 0.99 ± 0.13 0.03 ± 0.10 25-50 1.05 ± 0.40 0.20 ± 0.34 1.09 ± 0.41 0.24 ± 0.37 0.95 ± 0.21 0.06 ± 0.17 1.00 ± 0.18 0.05 ± 0.13 10-50 1.09 ± 0.48 0.28 ± 0.38 1.09 ± 0.48 0.32 ± 0.39 0.91 ± 0.29 0.11 ± 0.22 0.99 ± 0.23 0.07 ± 0.17 22 Table 5: SlowWalk performance comparison across algorithms and data ratios. Results show mean ± std for the last 10 evaluations, over 5 seeds. BCIQLDemoDICEReCOIL Variant RatioReturnMis.ReturnMis.ReturnMis.ReturnMis. Base 50-50 1.08 ± 0.47 0.40 ± 0.39 1.08 ± 0.45 0.40 ± 0.39 0.85 ± 0.32 0.08 ± 0.17 0.95 ± 0.32 0.17 ± 0.27 25-50 0.99 ± 0.53 0.42 ± 0.38 1.06 ± 0.49 0.43 ± 0.39 0.62 ± 0.38 0.18 ± 0.20 0.91 ± 0.38 0.24 ± 0.30 10-50 0.84 ± 0.57 0.46 ± 0.33 0.98 ± 0.57 0.58 ± 0.32 0.39 ± 0.29 0.25 ± 0.18 0.75 ± 0.46 0.34 ± 0.27 FMR 50-50 0.90 ± 0.27 0.13 ± 0.26 0.93 ± 0.23 0.11 ± 0.25 0.87 ± 0.29 0.06 ± 0.13 0.95 ± 0.17 0.04 ± 0.12 25-50 0.79 ± 0.36 0.21 ± 0.29 0.86 ± 0.30 0.18 ± 0.28 0.81 ± 0.32 0.09 ± 0.15 0.93 ± 0.21 0.06 ± 0.12 10-50 0.56 ± 0.35 0.33 ± 0.27 0.58 ± 0.36 0.37 ± 0.27 0.33 ± 0.25 0.25 ± 0.17 0.69 ± 0.32 0.17 ± 0.16 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 Misalignment 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 Misalignment 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 Misalignment 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR Figure 14: SlowSwim learning curves for baselines and FMR. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Return 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Return 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Return 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Return 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Misalignment 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR Figure 15: SlowHop learning curves for baselines and FMR. The shaded region represents the standard error. 23 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Return 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Return 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 Misalignment 1e6 50-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 25-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 10-50 BC BC+FMR DemoDICE DemoDICE+FMR IQL IQL+FMR ReCOIL ReCOIL+FMR Figure 16: SlowWalk learning curves for baselines and FMR. The shaded region represents the standard error. 24 C.3 Feedback as Preferences (CPL) and Reward (DVL) The following results compare FMR with CPL when using the sum feedback to score sub-trajectory preference and DVL when using feedback to replace reward. Results for velocity tasks and additional learning curves are provided. 50-5025-5010-50 0.0 0.2 0.4 0.6 0.8 1.0 Return CPLDVLReCOILReCOIL+FMR SwimmerHopperWalker2D 50-5025-5010-50 0.0 0.2 0.4 0.6 0.8 1.0 Return CPLDVLReCOILReCOIL+FMR 50-5025-5010-50 0.0 0.2 0.4 0.6 0.8 1.0 Return CPLDVLReCOILReCOIL+FMR 50-5025-5010-50 0.0 0.2 0.4 0.6 0.8 1.0 Return CPLDVLReCOILReCOIL+FMR 50-5025-5010-50 0.00 0.02 0.04 0.06 Misalignment CPLDVLReCOILReCOIL+FMR 50-5025-5010-50 0.0 0.1 0.2 Misalignment CPLDVLReCOILReCOIL+FMR 50-5025-5010-50 0.0 0.1 0.2 0.3 0.4 Misalignment CPLDVLReCOILReCOIL+FMR Figure 17: Velocity policies data ratio comparison between FMR, DVL, and CPL with ReCOIL baseline. Results show mean success rate and misalignment scores with 95% confidence intervals, averaged over the last 10 evaluations across 5 seeds. Table 6: PathM comparison of FMR against alternative methods for utilizing evaluative feedback. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmRatioSuc.Mis. ReCOIL50-500.951 ± 0.2150.113 ± 0.208 25-500.916 ± 0.2780.123 ± 0.222 10-500.855 ± 0.3520.150 ± 0.239 CPL50-500.872 ± 0.3340.098 ± 0.237 25-500.796 ± 0.4030.160 ± 0.282 10-500.565 ± 0.4960.302 ± 0.328 DVL50-500.773 ± 0.4190.148 ± 0.260 25-500.655 ± 0.4760.230 ± 0.291 10-500.520 ± 0.5000.308 ± 0.302 ReCOIL+FMR50-500.988 ± 0.1070.015 ± 0.084 25-500.974 ± 0.1600.014 ± 0.076 10-500.947 ± 0.2240.020 ± 0.092 Table 7: PathBB comparison of FMR against alternative methods for utilizing evaluative feedback. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmRatioSuc.Mis. ReCOIL10-500.592 ± 0.4920.237 ± 0.334 25-500.721 ± 0.4490.154 ± 0.296 50-500.780 ± 0.4150.153 ± 0.305 CPL10-500.573 ± 0.4950.273 ± 0.359 25-500.685 ± 0.4650.203 ± 0.341 50-500.781 ± 0.4140.124 ± 0.267 DVL10-500.400 ± 0.4900.287 ± 0.286 25-500.652 ± 0.4760.175 ± 0.274 50-500.771 ± 0.4200.108 ± 0.222 ReCOIL+FMR10-500.708 ± 0.4550.097 ± 0.211 25-500.880 ± 0.3250.073 ± 0.192 50-500.919 ± 0.2730.048 ± 0.167 25 Table 8: SlowSwim comparison of FMR against alternative methods for utilizing evaluative feedback. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmRatioReturnMis. ReCOIL50-500.994 ± 0.0680.003 ± 0.022 25-500.992 ± 0.0760.004 ± 0.024 10-500.967 ± 0.1450.018 ± 0.055 CPL50-501.001 ± 0.0260.001 ± 0.007 25-501.002 ± 0.0100.002 ± 0.005 10-501.000 ± 0.0230.004 ± 0.026 DVL50-500.997 ± 0.0480.012 ± 0.055 25-500.998 ± 0.0550.022 ± 0.077 10-500.993 ± 0.0830.056 ± 0.116 ReCOIL+FMR50-501.001 ± 0.0080.001 ± 0.003 25-501.002 ± 0.0080.000 ± 0.002 10-501.002 ± 0.0080.001 ± 0.003 Table 9: SlowHop comparison of FMR against alternative methods for utilizing evaluative feedback. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmRatioReturnMis. CPL50-500.990 ± 0.0890.022 ± 0.059 25-500.990 ± 0.1110.024 ± 0.059 10-500.995 ± 0.1730.071 ± 0.152 DVL50-501.046 ± 0.2730.103 ± 0.262 25-501.076 ± 0.3020.121 ± 0.275 10-501.135 ± 0.4180.248 ± 0.357 ReCOIL50-501.012 ± 0.1810.050 ± 0.164 25-501.066 ± 0.3100.140 ± 0.296 10-501.067 ± 0.3400.157 ± 0.304 ReCOIL+FMR50-500.990 ± 0.1270.029 ± 0.099 25-501.001 ± 0.1790.047 ± 0.135 10-500.986 ± 0.2250.066 ± 0.168 Table 10: SlowWalk comparison of FMR against alternative methods for utilizing evaluative feedback. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmRatioReturnMis. ReCOIL50-500.951 ± 0.3170.166 ± 0.273 25-500.912 ± 0.3790.243 ± 0.297 10-500.746 ± 0.4600.344 ± 0.275 CPL50-500.735 ± 0.3520.200 ± 0.268 25-500.717 ± 0.4110.326 ± 0.313 10-500.562 ± 0.4490.414 ± 0.294 DVL50-501.009 ± 0.2210.111 ± 0.225 25-501.005 ± 0.2670.154 ± 0.248 10-500.943 ± 0.4360.305 ± 0.282 ReCOIL+FMR50-500.954 ± 0.1690.040 ± 0.119 25-500.929 ± 0.2060.055 ± 0.122 10-500.690 ± 0.3230.165 ± 0.162 26 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 10-50 CPLDVLReCOILReCOIL+FMR Figure 18: PathM learning curves comparing FMR to alternative methods for utilizing evaluative feedback. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 10-50 CPLDVLReCOILReCOIL+FMR Figure 19: PathBB learning curves comparing FMR to alternative methods for utilizing evaluative feedback. The shaded region represents the standard error. 27 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 10-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 Misalignment 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 Misalignment 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 Misalignment 1e6 10-50 CPLDVLReCOILReCOIL+FMR Figure 20: SlowSwim learning curves comparing FMR to alternative methods for utilizing evaluative feedback. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Return 1e6 10-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Misalignment 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Misalignment 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Misalignment 1e6 10-50 CPLDVLReCOILReCOIL+FMR Figure 21: SlowHop learning curves comparing FMR to alternative methods for utilizing evaluative feedback. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 10-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Misalignment 1e6 50-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Misalignment 1e6 25-50 CPLDVLReCOILReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 Misalignment 1e6 10-50 CPLDVLReCOILReCOIL+FMR Figure 22: SlowWalk learning curves comparing FMR to alternative methods for utilizing evaluative feedback. The shaded region represents the standard error. 28 C.4 Only Negative Feedback Below are the results for the navigation aligned policies, comparing with and without positive feedback for D I using ReCOIL+FMR. Table 11: PathM performance comparison for ReCOIL+FMR with and without positive feedback. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmRatioSuc.Mis. w/ pos50-500.988 ± 0.110.015 ± 0.08 25-500.974 ± 0.160.014 ± 0.08 10-500.947 ± 0.220.020 ± 0.09 w/o pos50-500.993 ± 0.080.008 ± 0.06 25-500.982 ± 0.130.009 ± 0.06 10-500.933 ± 0.250.007 ± 0.05 Table 12: PathBB performance comparison for ReCOIL+FMR with and without positive feedback. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmRatioSuc.Mis. w/ pos50-500.919 ± 0.270.048 ± 0.17 25-500.880 ± 0.320.073 ± 0.19 10-500.708 ± 0.450.097 ± 0.21 w/o pos50-500.938 ± 0.240.036 ± 0.17 25-500.892 ± 0.310.055 ± 0.20 10-500.746 ± 0.440.134 ± 0.29 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 25-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Misalignment 1e6 50-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.00 0.02 0.04 0.06 0.08 0.10 Misalignment 1e6 25-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.00 0.02 0.04 0.06 0.08 0.10 Misalignment 1e6 10-50 w/ posw/o pos Figure 23: PathM learning curves for ReCOIL+FMR with and without positive feedback. The shaded region represents the standard error. 29 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 25-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Misalignment 1e6 50-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 25-50 w/ posw/o pos 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Misalignment 1e6 10-50 w/ posw/o pos Figure 24: PathBB learning curves for ReCOIL+FMR with and without positive feedback. The shaded region represents the standard error. 30 C.5 Usefulness of Imperfect Demonstrations Subsequent results are presented for testing the usefulness of imperfect demonstrations across all tasks. Results compare ReCOIL+FMR trained using a 10–50 data ratio against BC and IQL when trained on 10 randomly selected expert demonstrations with oversampling. Table 13: PathM performance comparison between ReCOIL+FMR and BC/IQL trained using only expert demonstrations. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmSuc.Mis. BC Expert0.710 ± 0.450.167 ± 0.31 IQL Expert0.801 ± 0.400.088 ± 0.24 ReCOIL+FMR 0.947 ± 0.220.020 ± 0.09 Table 14: PathBB performance comparison between ReCOIL+FMR and BC/IQL trained using only expert demonstrations. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmSuc.Mis. BC Expert0.854 ± 0.350.068 ± 0.20 IQL Expert0.883 ± 0.320.076 ± 0.22 ReCOIL+FMR0.708 ± 0.450.097 ± 0.21 Table 15: SlowSwim performance comparison between ReCOIL+FMR and BC/IQL trained using only expert demonstrations. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmReturnMis. BC Expert0.9991 ± 0.0340.0004 ± 0.001 IQL Expert1.0017 ± 0.0080.0003 ± 0.001 ReCOIL+FMR1.0018 ± 0.0080.0009 ± 0.003 31 Table 16: SlowHop performance comparison between ReCOIL+FMR and BC/IQL trained using only expert demonstrations. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmReturnMis. BC Expert0.974 ± 0.1250.014 ± 0.006 IQL Expert0.973 ± 0.1300.016 ± 0.007 ReCOIL+FMR0.986 ± 0.2250.066 ± 0.168 Table 17: SlowWalk performance comparison between ReCOIL+FMR and BC/IQL trained using only expert demonstrations. Results show mean ± std for the last 10 evaluations, over 5 seeds. AlgorithmReturnMis. BC Expert0.944 ± 0.1820.071 ± 0.086 IQL Expert0.769 ± 0.3150.136 ± 0.110 ReCOIL+FMR0.690 ± 0.3230.165 ± 0.162 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 Misalignment 1e6 BC ExpertIQL ExpertReCOIL+FMR Figure 25: PathM learning curves for ReCOIL+FMR and BC/IQL trained using only expert demon- strations. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Misalignment 1e6 BC ExpertIQL ExpertReCOIL+FMR Figure 26: PathBB learning curves for ReCOIL+FMR and BC/IQL trained using only expert demonstrations. The shaded region represents the standard error. 32 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.000 0.002 0.004 0.006 0.008 0.010 0.012 0.014 Misalignment 1e6 BC ExpertIQL ExpertReCOIL+FMR Figure 27: SlowSwim learning curves for ReCOIL+FMR and BC/IQL trained using only expert demonstrations. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 Misalignment 1e6 BC ExpertIQL ExpertReCOIL+FMR Figure 28: SlowHop learning curves for ReCOIL+FMR and BC/IQL trained using only expert demonstrations. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 BC ExpertIQL ExpertReCOIL+FMR 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Misalignment 1e6 BC ExpertIQL ExpertReCOIL+FMR Figure 29: SlowWalk learning curves for ReCOIL+FMR and BC/IQL trained using only expert demonstrations. The shaded region represents the standard error. 33 C.6 Feedback Scale The following results test the feedback scale using a 50-50 data ratio. Results are reported for ReCOIL+FMR where 0% corresponds to ReCOIL, 20% corresponds to 10 demonstrations with feedback, 50% corresponds to 25 demonstrations with feedback, and 100% corresponds to all demonstrations having feedback. Demonstrations are randomly selected to receive feedback. When a demonstration is selected, feedback for the entire demonstration is used. All remaining demonstrations do not have feedback. 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 50-50 0%20%50%100% Figure 30: PathM learning curves for ReCOIL+FMR with reduced feedback. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Misalignment 1e6 50-50 0%20%50%100% Figure 31: PathBB learning curves for ReCOIL+FMR with reduced feedback.The shaded region represents the standard error. 34 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 Misalignment 1e6 50-50 0%20%50%100% Figure 32: SlowSwim learning curves for ReCOIL+FMR with reduced feedback. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Misalignment 1e6 50-50 0%20%50%100% Figure 33: SlowHop learning curves for ReCOIL+FMR with reduced feedback. The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 0%20%50%100% 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Misalignment 1e6 50-50 0%20%50%100% Figure 34: SlowWalk learning curves for ReCOIL+FMR with reduced feedback. The shaded region represents the standard error. 35 C.7 Generalized Feedback-Adaptive Temperature The following are the results when comparing feedback-adaptive temperatureτ j with general- ized feedback-adaptive temperatureτ gen . The comparison is performed across all tasks using Re- COIL+FMR in environments with a discrete action space, demonstrating negligible performance differences. This indicates that either formulation is valid, whereτ gen generalizesτ j beyond discrete action spaces to support continuous action spaces. Table 18: PathM performance comparison for ReCOIL+FMR when usingτ j andτ gen . Results show mean ± std for the last 10 evaluations, over 5 seeds. τ TypeRatioSuc.Mis. τ gen 50-500.986 ± 0.120.014 ± 0.08 25-500.974 ± 0.160.019 ± 0.09 10-500.941 ± 0.240.020 ± 0.08 τ j 50-500.988 ± 0.110.015 ± 0.08 25-500.974 ± 0.160.014 ± 0.08 10-500.947 ± 0.220.020 ± 0.09 Table 19: PathBB performance comparison for ReCOIL+FMR when usingτ j andτ gen . Results show mean ± std for the last 10 evaluations, over 5 seeds. τ TypeRatioSuc.Mis. τ gen 50-500.919 ± 0.270.058 ± 0.19 25-500.880 ± 0.330.053 ± 0.15 10-500.706 ± 0.460.109 ± 0.24 τ j 50-500.919 ± 0.270.048 ± 0.17 25-500.880 ± 0.320.073 ± 0.19 10-500.708 ± 0.450.097 ± 0.21 Table 20: SlowSwim performance comparison for ReCOIL+FMR when usingτ j andτ gen . Results show mean ± std for the last 10 evaluations, over 5 seeds. τ TypeRatioReturnMis. τ gen 50-501.0014 ± 0.0140.0007 ± 0.005 25-501.0019 ± 0.0080.0003 ± 0.001 10-501.0021 ± 0.0150.0007 ± 0.003 τ j 50-501.0011 ± 0.0080.0005 ± 0.003 25-501.0017 ± 0.0080.0004 ± 0.002 10-501.0018 ± 0.0080.0009 ± 0.003 36 Table 21: SlowHop performance comparison for ReCOIL+FMR when usingτ j andτ gen . Results show mean ± std for the last 10 evaluations, over 5 seeds. τ TypeRatioReturnMis. τ gen 50-500.988 ± 0.1240.033 ± 0.114 25-500.993 ± 0.1660.042 ± 0.133 10-500.984 ± 0.2170.061 ± 0.157 τ j 50-500.990 ± 0.1270.029 ± 0.099 25-501.001 ± 0.1790.047 ± 0.135 10-500.986 ± 0.2250.066 ± 0.168 Table 22: SlowWalk performance comparison for ReCOIL+FMR when usingτ j andτ gen . Results show mean ± std for the last 10 evaluations, over 5 seeds. τ TypeRatioReturnMis. τ gen 50-500.952 ± 0.1790.035 ± 0.105 25-500.936 ± 0.1970.046 ± 0.107 10-500.679 ± 0.3280.152 ± 0.150 τ j 50-500.954 ± 0.1690.040 ± 0.119 25-500.929 ± 0.2060.055 ± 0.122 10-500.690 ± 0.3230.165 ± 0.162 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 j gen 0.00.20.40.60.81.0 Iterations 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Misalignment 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.00 0.02 0.04 0.06 0.08 0.10 Misalignment 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.00 0.02 0.04 0.06 0.08 0.10 Misalignment 1e6 10-50 j gen Figure 35: PathM learning curves for ReCOIL+FMR when usingτ j andτ gen . The shaded region represents the standard error. 37 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 Misalignment 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 Misalignment 1e6 10-50 j gen Figure 36: PathBB learning curves for ReCOIL+FMR when usingτ j andτ gen . The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 10-50 j gen 0.00.20.40.60.81.0 Iterations 0.000 0.002 0.004 0.006 0.008 0.010 0.012 0.014 Misalignment 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.000 0.002 0.004 0.006 0.008 0.010 0.012 0.014 0.016 Misalignment 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.000 0.002 0.004 0.006 0.008 0.010 0.012 0.014 Misalignment 1e6 10-50 j gen Figure 37: SlowSwim learning curves for ReCOIL+FMR when usingτ j andτ gen . The shaded region represents the standard error. 38 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 10-50 j gen 0.00.20.40.60.81.0 Iterations 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 Misalignment 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 Misalignment 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 Misalignment 1e6 10-50 j gen Figure 38: SlowHop learning curves for ReCOIL+FMR when usingτ j andτ gen . The shaded region represents the standard error. 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Return 1e6 10-50 j gen 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Misalignment 1e6 50-50 j gen 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Misalignment 1e6 25-50 j gen 0.00.20.40.60.81.0 Iterations 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Misalignment 1e6 10-50 j gen Figure 39: SlowWalk learning curves for ReCOIL+FMR when usingτ j andτ gen . The shaded region represents the standard error. 39 C.8 Hyperparameters The following subsections conduct various hyperparameter experiments for FMR. C.9 β Values Below are the results for variousβvalues when computing the temperatureτ j based on feedback for ReCOIL+FM. Table 23: PathBB comparison of differentβvalues for ReCOIL+FMR with a 10-50 data ratio. Results show mean ± std for the last 10 evaluations, over 5 seeds. βSuc.Mis. 1.50.592 ± 0.490.134 ± 0.24 100.708 ± 0.450.097 ± 0.21 1000.702 ± 0.460.191 ± 0.32 10000.709 ± 0.450.098 ± 0.24 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 1.5101001000 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 1.5101001000 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 10-50 1.5101001000 Figure 40: PathBB learning curves ofβvalues for ReCOIL+FMR with a 10-50 data ratio. The shaded region represents the standard error. 40 C.10 α Values Below are the results for variousαvalues for controlling the regularization or alignment strength of ReCOIL+FM. Table 24: PathBB performance comparison of differentαvalues for ReCOIL+FMR with a 10-50 data ratio. Results show mean ± std for the last 10 evaluations, over 5 seeds. αSuc.Mis. 0.10.506 ± 0.500.209 ± 0.31 10.708 ± 0.450.097 ± 0.21 100.672 ± 0.470.092 ± 0.21 1000.510 ± 0.500.074 ± 0.17 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 0.1110100 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 0.1110100 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 10-50 0.1110100 Figure 41: PathBB learning curves ofαvalues for ReCOIL+FMR using PathBB with a 10-50 data ratio. The shaded region represents the standard error. 41 C.11 Credit Assignment Values Below are the results for time windows when applying credit assignment to feedback for Re- COIL+FMR. Table 25: PathBB performance comparison of different credit assignment times for ReCOIL+FMR with a 10-50 data ratio. Results show mean ± std for the last 10 evaluations, over 5 seeds. TimeSuc.Mis. 300ms0.688 ± 0.460.124 ± 0.25 600ms0.708 ± 0.450.097 ± 0.21 1000ms0.758 ± 0.430.119 ± 0.30 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 300ms600ms1000ms 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Success Rate 1e6 10-50 300ms600ms1000ms 0.00.20.40.60.81.0 Iterations 0.0 0.1 0.2 0.3 0.4 0.5 Misalignment 1e6 10-50 300ms600ms1000ms Figure 42: PathBB Learning curves of various credit assignment times for ReCOIL+FMR with a 10-50 data ratio. The shaded region represents the standard error. 42 C.12 FMR Induces Feedback-Modulated Entropy This subsection visualizes state-space entropy for the aligned navigation policies using ReCOIL models. Figure 43 and Figure 44 depict the heatmaps for normalized entropy. To generate these figures, normalized entropy is computed using the final actor model from training, with the expert and imperfect trajectories as input. The average over 5 seeds is then computed. FMR increases entropy for positive feedback while decreasing it for negative feedback. Expert demonstrations receive no feedback and therefore converge to high entropy by default. These entropy heatmaps correspond closely with the original feedback heatmaps given in Figure 6a and Figure 6b. 3210123 x-coordiante 3 2 1 0 1 2 3 y-coordiante 0.0000 0.0833 0.1667 0.25000.2500 0.3333 0.4167 0.5000 Entropy (a) ReCOIL 3210123 x-coordiante 3 2 1 0 1 2 3 y-coordiante 0.0000 0.0833 0.1667 0.25000.2500 0.3333 0.4167 0.5000 Entropy (b) ReCOIL+FMR Figure 43: PathM heatmap of normalized entropy for ReCOIL and ReCOIL+FMR. 3210123 x-coordiante 3 2 1 0 1 2 3 y-coordiante 0.0000 0.0833 0.1667 0.25000.2500 0.3333 0.4167 0.5000 Entropy (a) ReCOIL 3210123 x-coordiante 3 2 1 0 1 2 3 y-coordiante 0.0000 0.0833 0.1667 0.25000.2500 0.3333 0.4167 0.5000 Entropy (b) ReCOIL+FMR Figure 44: PathBB heatmap of normalized entropy for ReCOIL and ReCOIL+FMR. 43