Paper deep dive
PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning
Hang Yu, James Staley, Cheng Xi Tsou, Xiujin Liu, Wenchang Gao, Jindan Huang, Shijie Fang, Zhegong Shangguan, Angelo Cangelosi, Reuben Aronson, Elaine Short
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/29/2026, 12:00:29 PM
Summary
The paper introduces PHIRL (Progress-Heuristicized Inverse Reinforcement Learning), a data-efficient framework that learns robust reward functions by jointly leveraging human demonstrations and progress feedback. PHIRL aligns learned rewards with cumulative task completion annotations across four dimensions: temporal delta, magnitudes, potential, and completion. It significantly outperforms baselines like AIRL and AIRLPref in simulated and real robot tasks, requiring only 20% annotated demonstrations and showing resilience against reward hacking.
Entities (10)
Relation Signals (6)
PHIRL → alignswith → Progress
confidence 95% · aligns the rewards with progress annotations over four dimensions
PHIRL → uses → Inverse Reinforcement Learning
confidence 95% · PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning
PHIRL → evaluatedon → Robosuite
confidence 90% · We evaluate PHIRL on real and simulated robot tasks... from Robosuite
PHIRL → outperforms → AIRL
confidence 90% · Results demonstrate that PHIRL significantly outperforms the baselines
PHIRL → usesasbase → AIRL
confidence 90% · We use the Adversarial Inverse Reinforcement Learning (AIRL) [35] algorithm to learn the initial reward distribution.
PHIRL → extendswith → Vision-Language Model
confidence 85% · we extend PHIRL by integrating a fine-tuned Vision-Language Model (VLM) to provide online progress feedback.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.
Tags
Links
- Source: https://arxiv.org/abs/2609.31855v1
- Canonical: https://arxiv.org/abs/2609.31855v1
Trouble viewing inline? Open PDF directly →
Full Text
46,434 characters extracted from source content.
Expand or collapse full text
PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning Thanks: *Corresponding author: hyu08@tufts.edu Hang Yu* James Staley Cheng Xi Tsou Xiujin Liu Wenchang Gao Jindan Huang Affiliation: Shijie Fang, Zhegong Shangguan, Angelo Cangelosi, Reuben M Aronson, Elaine Short Abstract Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL’s learned reward functions are reliable against exploitation. I Introduction An appropriate reward function is a crucial component for Reinforcement Learning (RL) agents to perform effectively [1, 2]. While sparse rewards are intuitive to specify, they offer limited guidance during early exploration. Conversely, hand-crafted dense rewards accelerate learning, but designing them requires substantial engineering effort and tedious manual tuning [1]. Inverse Reinforcement Learning (IRL) offers an alternative by inferring a dense reward function directly from human demonstrations or human feedback, mitigating the need for manual reward designing. Compared to imitation learning methods that directly map states to actions [3], IRL may require higher-quality demonstrations [4], but the learning outcomes can be easier for post-training [5] or revising [6]. In this work, we aim to learn robust reward functions by combining human demonstrations and feedback, while reducing human engagement. Recent work has demonstrated that using human demonstrations and feedback jointly can improve data efficiency [7] and lower quality requirements for human demonstrations [8]. Many reward learning methods often assume the demonstrations are optimal [9] or near-optimal [4], but providing high-quality demonstrations for manipulation tasks is known to be difficult even for experts. High-quality human feedback, on the other hand, is more accessible and requires fewer skills. Prior reward learning work often relies on preference feedback [8, 10, 11, 12]. Forcing users to pick a preferred trajectory does not always benefit learning. For instance, consider a pick-and-place task where the robot never manages to grasp the object. In this case, all sampled trajectory clips are undesirable. The preference feedback is therefore uninformative. Moreover, prior work has achieved huge success in simulating human feedback [13, 14, 15], but for scenarios where feedback from real human users is required [16], users often need to stay in the training loop to provide feedback on newly generated trajectories. Technologies and dataset-sharing communities, such as UMI [17], ego-centric data [18], and Hugging Face [19], have made robot demonstrations significantly easier to access. In this setting, human-in-the-loop teaching effort has become a more expensive resource than stock demonstrations. In this work, we mitigate these gaps by proposing a data-efficient reward learning framework, PHIRL, that learns from demonstrations and human progress [20]. Progress describes the cumulative task completion based on the task observation. Our key intuition is that the rewards inferred from demonstrations should be consistent with the annotations of the demonstrations. This process mimics human learning: much like a student cross-referencing answers to correct their own solutions, the method generates a potential reward function and compares it against human feedback to refine its learned rewards. In PHIRL, we iteratively use an IRL algorithm to learn a reward function, calculate rewards over the annotated demonstrations, and then shape the learned reward function in four dimensions: temporal delta, magnitudes, potential, and completion. In addition, we extend PHIRL by integrating a fine-tuned Vision-Language Model (VLM) to provide online progress feedback. We validate PHIRL with two robot tasks from Robosuite [21] and one real robot task with four baselines. Our results show that PHIRL significantly outperforms IRL baselines with only 20% of demonstrations being annotated offline, and the online extension of PHIRL is effective with VLM-generated annotations. We conducted a qualitative analysis against AIRL, which confirms that the reward functions shaped by PHIRL are resilient to two reward-hacking behaviors. Code is available at github.com/PHIRL2026. Fig. 1: PHIRL framework. We randomly sample a subset of demonstrations and ask users to annotate them with progress. We then iteratively use an IRL algorithm to learn a reward model from the demonstrations and align the learned reward model with progress over four dimensions. Optionally, the generated trajectories will be sampled and annotated by a VLM for data augmentation. The generated trajectories will be used for reward shaping only. I Background Interactive machine learning grants robots the ability to adapt to human needs or learn new skills by leveraging human knowledge [22], such as human preference [23], facial expressions [24], numerical evaluation [25], and human demonstrations [26]. Reward Learning from Human Feedback Human feedback is a popular way for agents to learn from humans via interactive reinforcement learning or inverse reinforcement learning [26, 24, 27]. Human feedback contains relatively sparse but precise information [8]. To provide feedback, humans first observe agents performing a task, and then evaluate the performance of the agents implicitly or explicitly [22]. Humans can use feedback to assess robot behaviors [28, 29] and indicate their preferred policy, which can then be used to extract rewards [30]. Human feedback can also be applied to post-train trained models, such as foundation models [5]. Reward Learning from Human Demonstrations Learning from Demonstration (LfD) allows learning agents to learn to perform new tasks by imitating humans [26, 4, 31]. LfD approaches have many advantages, such as reducing the need for expert programming [32] and data efficiency [3]. LfD generally solves the learning problem mainly in two ways: (1) inferring a policy [3, 33], and (2) inferring a reward function [34, 35]. Learning a reward function from demonstrations, while more difficult, has multiple advantages: it enables the learning outcome to be transferred to new tasks [36], supports rapid re-optimization [37], and can be combined with optimal control and RL machinery [38]. However, learning an accurate reward function purely from demonstrations is challenging [8], and the majority of demonstrations often need to be high-quality [4, 39]. Reward Learning from Both Human Demonstration and Human Feedback Recent work indicates that combining human demonstrations and human feedback could improve learning efficiency and overcome the limitations of using human feedback or demonstrations alone [10, 40, 8, 26]. Specifically, Ibarz et al. [10] proposed to use demonstrations to pre-train a learning agent, and then used the pre-trained agent to generate preference queries, which significantly improves the sample efficiency for preference learning. Building on Ibarz et al. [10]’s work, Palan et al. [8] not only used the demonstrations to improve sample efficiency but also used the demonstrations to learn an initial reward model. Two related works [9, 41] showed that learned reward functions can be improved by forcing the models to assign more rewards to expert demonstrations. However, this improvement is based on the assumption that expert demonstrations always make at least some task progress. Kim et al. [37], Li et al. [7] showed that the learned reward function can be improved by aligning with human preference comparisons. Our work differs from prior work by improving the data efficiency and reducing human-in-the-loop effort. We do not assume that behaviors in demonstrations always contribute to task completion, and we align the learned rewards with progress across multiple dimensions rather than temporal delta alone. We align the learned rewards with progress annotations collected prior to learning, thereby making them more robust to non-progressive behaviors while requiring less human engagement. I Methodology Our goal is to effectively learn robust reward functions. As previously mentioned, human demonstrations can be combined with human feedback to improve learning efficiency. Our work is built on the key insight that rewards inferred from a set of demonstrations should be consistent with the feedback on the same set of demonstrations. I-A Progress We use a numerical human feedback form, progress [20] for reward function alignment. Progress is a signal that describes the cumulative degree of task completion, ranging from fully incomplete to fully complete. Previous work has demonstrated that progress is an informative teaching signal and has many advantages for reward learning [42, 14]. For any given task t, current state s, any action aia_i and aja_j, and any previous state sis_i and sjs_j, the progress: progt(s)=progt(s∣si,ai)=progt(s∣sj,aj),∀i,j,prog_t(s)\;=\;prog_t\! (s s_i,a_i )\;=\;prog_t\! (s s_j,a_j ), ∀ i,j, (1) In this work, we use progress in a range of 0 to 100 and ask participants to provide progress labels only based on the current observation, where 100 indicates the task is fully complete, and 0 means that no progress has been made yet. Progress is accurate for low-quality demonstrations and consistent across non-experts without requiring additional cognitive load compared to providing preference evaluations [20]. I-B Reward Learning from Demonstrations Given a set of human demonstrations =d0,…,dnD=\d_0,...,d_n\, where d=(s0,a0,…,sn,an)d=(s_0,a_0,...,s_n,a_n), the objective of IRL is to find a reward function r(s,a,s′)r(s,a,s ). If we assume that the demonstrations are from an optimal policy π∗π^*, we can then interpret the IRL problem as a maximum likelihood problem: maxθd∼[logℙθ(d)], _θE_d [ _θ(d)], (2) where ℙθ(d)∝ _θ(d) ℙ(s0)(∏t=0T−1ℙ(st+1∣st,at)) (s_0) ( _t=0^T-1P(s_t+1 s_t,a_t) ) ×exp(∑t=0T−1γtrθ(st,at)) × ( _t=0^T-1γ^tr_θ(s_t,a_t) ) We use the Adversarial Inverse Reinforcement Learning (AIRL) [35] algorithm to learn the initial reward distribution. AIRL casts the optimization of equation 2 as a GAN [43] optimization problem. The discriminator uses a particular fθf_θ: Dθ(s,a,s′)=expfθ(s,a,s′)expfθ(s,a,s′)+π(a|s)D_θ(s,a,s )= \f_θ(s,a,s )\ \f_θ(s,a,s )\+π(a|s) (3) and π is trained to maximize:r=logDθ−log(1−Dθ)r= D_θ- (1-D_θ), where fθ(s,a,s0)f_θ(s,a,s_0) can be interpreted as the advantage under deterministic dynamics: f∗(s,a,s′)=r∗(s)+γV∗(s′)⏟Q(s,a)−V∗(s)⏟V(s)=A∗(s,a)f^*(s,a,s )= r^*(s)+γ V^*(s )_Q(s,a)- V^*(s)_V(s)=A^*(s,a) (4) We select AIRL as our reward learning method over other methods because AIRL infers reward functions from state-only functions r(s)r(s) and V(s)V(s), and progress is also path-independent. I-C Reward Function Shaping We improve the learned reward function by shaping it to align with progress annotations during reward learning. Specifically, we exploit four key properties of the information provided by progress. Temporal Delta: positive changes in progress correspond to positive rewards. Given a trajectory segment st→atst+1s_t a_ts_t+1, if the robot’s action sequence is appropriate, the progress pt+1p_t+1 at state st+1s_t+1 should be higher than the progress ptp_t at sts_t. Similarly, the learned incremental reward rθ(st,st+1)=rθ(st+1)−rθ(st)r_θ(s_t,s_t+1)=r_θ(s_t+1)-r_θ(s_t) should be positive, encouraging transitions that move the robot closer to task completion. We enforce this relationship via the following loss: ℒΔp→rθ(θ)=(st,st+1)∼[ℓ(Δpt,rθ(st,st+1))],L_ p→ r_θ(θ)=\;E_(s_t,s_t+1) \! [\, ( p_t,\,r_θ(s_t,s_t+1) ) ], (5) where Δpt=pt+1−pt p_t=p_t+1-p_t , and ℓ(⋅,⋅) (·,·) is a strictly increasing monotonic hinge alignment function, where ℓ(a,b)=max(0,m−sign(a)b),m≥0. (a,b)= (0,\;m-sign(a)\,b ),m≥ 0. Magnitudes: more increase in progress indicates more rewards. Reward learning from human demonstrations relies more on distinguishing between policies covered by the demonstrations or policies outside the demonstrations, resulting in the learned reward values tending to be binary based on the similarity to expert behaviors. However, even in optimal demonstrations, not all behaviors in a demonstration are equally critical. Some steps, like picking up the target object, are inherently more progressive than simply moving closer, and thus should intuitively receive a larger reward. So transitions with larger progress improvements should receive larger rewards. We enforce this by: ℒΔp→Δrθ(θ) _ p→ r_θ(θ) =(si,si+1,sj,sj+1)∼[ℓ(Δpi−Δpj, =E_(s_i,s_i+1,s_j,s_j+1) [ \! ( p_i- p_j,\, (6) rθ(si,si+1)−rθ(sj,sj+1))]. r_θ(s_i,s_i+1)-r_θ(s_j,s_j+1) ) ]. Note that in this condition specifically, si,si+1,sjs_i,s_i+1,s_j, and sj+1s_j+1 are preferably from the same demonstration D since different users might scale progress differently. Potential: high progress means high potential. The potential function Φ was introduced in [44], and describes the distance between the current state and the goal state: Φ(s)=−dist(s,sgoal) (s)=-dist(s,s_goal). The use of potential function will not alter the original optimal policy, and can reduce random exploration by getting heuristics from: F(st,at,st+1)=γΦ(st+1)−Φ(st)F(s_t,a_t,s_t+1)\;=\;γ\, (s_t+1)\;-\; (s_t). We set Φ(s)=V(s) (s)=V(s) in this work. If a state has higher progress, the learned potential should also be higher. We capture this by: ℒΔp→Φ(Φ) _ p→ ( ) =(st,st+1)∼[ℓ(Δpt,F(st,at,st+1))]. =E_(s_t,s_t+1) [ (\, p_t,F(s_t,a_t,s_t+1) ) ]. (7) Completion: success represents more total rewards than failure. The ultimate goal of reward learning is to learn a reward function that guides the agent to complete the task, and thus, task completion imposes a total reward constraint. We therefore hypothesize that any demonstration ds∈succd_s _succ that completed the task should receive a higher total reward Rθ(d)=∑trθ(stst+1)R_θ(d)\;=\; _tr_θ\! (s_t\,s_t+1 ) than any demonstration df∈faild_f _fail that did not complete the task (in this work, progress<progress< 90 at the end) , no matter the horizon of the demonstrations. We capture this condition by: ℒcomp(θ) _comp(θ) =∑df∈failmax0,Rθ(df) = _d_f _fail \0,\,R_θ(d_f) (8) −1|succ|∑ds∈succRθ(ds), - 1|D_succ| _d_s _succR_θ(d_s) \, where succD_succ are successful demonstrations and failD_fail are unsuccessful demonstrations. I-D PHIRL In PHIRL, we learn reward functions by alternately updating a reward function by using AIRL and aligning the learned reward function with progress. Given demonstration dataset D and their annotated subset pD_p, a learned reward rθr_θ is aligned by minimizing a sum of all shaping components: ℒalign(θ,Φ) _align(θ, ) =λ1ℒΔp→rθ(θ)+λ2ℒΔp→Δrθ(θ) = _1L_ p→ r_θ(θ)+ _2L_ p→ r_θ(θ) (9) +λ3ℒΔp→Φ(Φ)+λ4ℒcomp(θ), + _3L_ p→ ( )+ _4L_comp(θ), where we configured λ1=λ2=λ3=1 _1= _2= _3=1 and λ4=1|| _4= 1|D|. In offline PHIRL, all demonstrations and annotations are collected before the learning phase. We show PHIRL in Algorithm 1. Algorithm 1 PHIRL 1: Collect demonstration dataset D, annotate a subset pD_p, initialize discriminator DθD_θ and policy π 2: while not converged do 3: Sample trajectories τ∼τ and τπ∼π _π π, update DθD_θ to distinguish τ from τπ _π 4: Sample labeled trajectories τp∼p _p _p and corresponding labels pτp_τ 5: Compute reward rθ(τp)r_θ( _p), potential Φ(τp) ( _p) 6: Update rθr_θ and Φ with respect to updated p by minimizing Lalign,θ,ΦL_align,θ, 7: Optional annotate τπ _π with VLM-generated progress, update p←p∪τπD_p _p∪\ _π\ 8: end while Online Extension While human-in-the-loop teaching is not mandatory for our method, PHIRL can benefit from adding generated demonstrations into the annotation dataset. The only change to the method for online learning is that pD_p is updated throughout learning with new annotated demonstrations chosen from generated trajectories. PHIRL Online can improve the robustness of the learning outcome, and better align the reward function with actual human intentions. Scalability We choose to apply PHIRL to AIRL over other IRL algorithms because AIRL’s training objectives are well-aligned with progress in multiple aspects, but the PHIRL framework itself is not inherently tied to AIRL. For instance, for methods without a potential component, PHIRL can still be effective with the remaining three alignments (see subsection IV-C). The key idea of the framework is shaping the learned rewards to align with the evaluation information specified by progress. IV Experiments and Results In this section, we run PHIRL with four baselines using one task with a real robot and two tasks from Robosuite [21] to validate PHIRL’s performance and our claimed contributions. In addition, we demonstrated that the reward functions learned by PHIRL are robust to reward hacking in two representative scenarios. IV-A Setups Domains We selected two domains from Robosuite, Lift and PickPlaceCan, and one real robot task, animal-brushing. For the Lift domain, the goal is to reach the block and lift it to a certain height. For the PickPlaceCan, the goal is to pick up the can, navigate to the goal location, and drop the can into the right bin. For the real robot task, we used a Kinova Gen 3 Lite arm and an animal-brushing brush task. The goal was to navigate to the stuffed animal and then brush the pom-poms off the stuffed animal. Datasets For the Robomimic domains, we trained baselines and PHIRL on a high-quality dataset and a mixed-quality dataset. The high-quality datasets are the Proficient Human (PH) datasets, which were provided by a single proficient operator. Each dataset consists of 200 demonstrations. The mixed-quality datasets are the Multi Human (MH) datasets, which were collected by six operators using the RoboTurk platform. Each operator provided 50 demonstrations. Two operators were “worse” operators, two were “okay” operators, and two were “better” (i.e., proficient) operators. For the real robot domain, we used a similar setup, recruited six participants (two “worse”, two “okay”, and two “better”, 10 demonstrations each), and constructed an MH dataset. Fig. 2: Performance comparison and ablation studies Results in the Lift and PickPlaceCan domains. Error bars represent standard deviation. Note that the standard PHIRL model uses 20% annotation (equivalent to PHIRL-20%). (Top) Average environmental return rewards across all baselines. PHIRL significantly outperforms baselines in both domains, particularly when the task horizon is long or demonstration quality is low (MH). (Middle) Ablation study on the effect of removing individual reward shaping components. Completion shaping is most critical when there are failed demonstrations, while reward shaping is most effective overall. (Bottom) Sensitivity analysis of annotation percentages. PHIRL improves learning significantly with 20% of the demonstrations. The benefits of the offline annotation saturate at 50% if the demonstrations are relatively high quality. Baselines We include four baseline methods, AIRL (IRL with no human feedback) [35], Pref (preference-learning based IRL with no demonstrations) [45], AIRLPref (IRL with both feedback and demonstrations) [8], and Rank2Reward (reward learning method with a progress-ish signal) [41]. We use PPO [46] as the adversarial learners and trajectory generators. We set the standard PHIRL model to use a 20% annotated demonstration dataset. We trained all baselines and PHIRL for 500 episodes, with 10 parallel environments and an environmental horizon of 256. Hyperparameters, including learning rate, discount factor, clip range, and discriminator batch size, are selected by using a hyperparameter search [47]. Data Collection We separately recruited six participants to annotate the Robomimic datasets, six participants to construct the real robot dataset, and two participants to annotate the real robot dataset. When collecting progress and preferences, each demonstration was equally divided into 10 sub-trajectories. Each sub-trajectory was annotated with one progress label, and for preferences, we asked participants to pick a preferable sub-trajectory from two sub-trajectories. To reduce bias in progress annotation, we avoided imposing specific patterns on the progress values by only asking them to provide values from 0 to 100. The VLM base model used in this work is Gemma 3 12B [48], we fine-tuned the base model using LoRA [49]. More details about the implementation, prompts used, and the code can be found at github.com/PHIRL2026/PHIRL_appendix. Fig. 3: Real Robot Study Results. The goal is to brush the pom-pom off the stuffed animal. We calculated the number of pom-poms that were brushed off the stuffed animal. PHIRL-online achieves the highest. Fig. 4: Reward Hacking Scenarios and Reward Curves. The lighter colors and darker colors indicate increases and decreases in rewards. The Lift-Drop Loop was generated by a PPO agent that learns from AIRL rewards. The robot repeatedly lifted the block and then dropped it. We found that AIRL encouraged the robot to collect more rewards by lifting and dropping the block repeatedly, while PHIRL appropriately punished dropping behaviors and rewarded lifting behaviors. The Touch-And-Stay was generated by a PPO agent that learns from Robosuite environmental rewards. The robot navigated to the can and kept poking it instead of picking and placing it. PHIRL gave a negative but near-zero reward, indicating the robot is not making progress. IV-B PHIRL Outperforms Baselines Simulation. We trained baselines and PHIRL on a high-quality dataset (PH) and a mixed-quality (MH) dataset from Robomimic [50]. The results of simulation experiments are presented in Figure 2. The top row presents the average environmental return rewards. Since Pref does not learn from demonstrations, we used the same data for both PH and MH conditions. Error bars and tests are computed over 100 evaluation rollouts (environment seeds 1 to 100). We used Welch’s t-test to determine statistical significance. In the Lift domain, PHIRL achieved a mean environmental reward of M=188.12M=188.12 in PH and M=167.16M=167.16 in MH, which is higher than all other baselines (5.6% and 17.1% higher than the best baseline). In the more challenging PickPlaceCan domain, where the average demonstration horizon is approximately twice that of Lift, PHIRL achieved greater performance advantages (91.3% higher than the best baseline) and also became more significant in the MH dataset(2.3 times more return rewards), where demonstration quality is lower. These results demonstrate that PHIRL learns effective rewards for RL agents, and is more reliable when the task horizon is longer or the demonstration quality is lower. Real Robot For the real robot domain, we recruited six participants and constructed an MH dataset in a similar way (two “worse”, two “okay”, and two “better”, 10 demonstrations each). We trained PHIRL and baselines over the constructed dataset and calculated the number of pom-poms being brushed off from the stuffed animal over 10 episodes. For each episode, 10 pom-poms were placed randomly. Results are shown in Figure 3. Our results confirmed our findings in simulations. PHIRL performed well with mixture quality demonstrations and outperformed baselines AIRL (39), AIRL-Pref (62), Pref (58), and Rank2Reward (54). PHIRL vs. PHIRL-online We found that using VLM-generated progress labels (400 for PH and 600 for MH in addition, the same number as human annotations) and for online training resulted in a consistent improvement for PHIRL. In the Lift-PH condition, we found that adding online progress labels resulted in a statistically significant improvement at the converged performance between PHIRL and PHIRL Online (M=188.12M=188.12 vs M=205.08M=205.08, p<0.05p<0.05). PHIRL-online also demonstrated a decent improvement on the real robot study (84 vs. 73). The impact of online labels became even more pronounced when the demonstrations were noisier (MH settings) or the task was harder (PickPlaceCan Domain). For instance, in the challenging Can-MH scenario, returns increased from M=116.99M=116.99 to M=147.28M=147.28 ( p<0.001p<0.001). IV-C Ablation Study and Sensitivity Analysis Ablation. We investigated the contribution of each individual reward-shaping term to the overall reward function, as shown in the middle row of Figure 2. We treat the standard PHIRL model as the anchor. The potential shaping component is generally influential across all conditions. Other ablations also showed statistically significant decreases, indicating the full suite of reward shaping components is necessary for maximal performance. The only exception is the completion shaping term, as in most conditions demonstrations are marked as fully completed (progress >> 90 for the last frame). Sensitivity. The bottom row of Figure 2 shows the sensitivity of PHIRL to the annotation coverage. Our standard PHIRL utilizes 20% of the available annotations. The results indicate that increasing the coverage in general will increase the model performance. The improvement from annotation is more significant when the dataset has a lower quality. In most conditions, the improvements saturated at 50% of coverage, suggesting the strong data efficiency of PHIRL. More details and results, including more statistical analysis, learning curves, success rate, annotation examples, and additional domains for simulation studies, are in github.com/PHIRL2026/PHIRL_appendix. IV-D PHIRL Learns More Robust Reward Functions When training PHIRL, AIRL, and PPO, we found two reward hacking scenarios: Lift-Drop-Loop and Touch-And-Stay. We show the keyframes of the reward-hacking trajectories and cumulative rewards for PHIRL, AIRL, and PPO in the Robosuite environment. The rewards are normalized to 0 to 100 for visualizing the curves. We use a lighter color to indicate increases in rewards and a darker color to represent decreases in rewards. Lift-Drop-Loop We show keyframes of the trajectory and a normalized reward curve for each method in Figure 4. This scenario was generated by a PPO agent that learns from the AIRL reward function. The robot repeatedly picks up the block, lifts it, and then drops it. This is because AIRL rewards the robot for picking up the block, but does not continue to reward or even punish the robot for having the block once it has been picked up. Thus the robot can achieve a higher cumulative reward by repeatedly lifting and dropping. PHIRL produced a more robust reward function. It positively rewarded the repeated reaching and lifting behaviors, and punished the dropping behaviors since dropping the block back to the table brings the state back to a low progress status. Touch-And-Stay The scenario was produced by a PPO agent that learns from the environmental rewards, shown in Figure 4. Instead of picking up the can and placing it in the target bin, the robot navigated to the can, closed the gripper, and poked the can with the gripper. This reward hacking is likely caused by an imbalanced reward design and a bottleneck in the task. Grasping the can is challenging and has risks of losing future rewards, so the PPO agent learned a “safer” strategy, collecting small rewards by poking the can. Without carefully tuning the hyperparameters, the agent can easily get stuck in this local optimum. In contrast, AIRL and PHIRL both negatively rewarded the poking behaviors, while PHIRL gave a very small negative reward since poking behaviors are not harmful to the task completion. V Discussion and Limitation Sampling Annotated Demonstrations In this work, when we trained PHIRL with partially annotated datasets, we randomly sampled demonstrations from the dataset instead of intentionally selecting imperfect demonstrations. We believe that the process of classifying human demonstrations into high-quality and low-quality introduces additional information to the learning process, which can bias our comparison with baselines. However, we agree that preferentially selecting lower-quality demonstrations will improve sample efficiency and thus improve model performance, especially when there are limited human feedback labels. Exploring Generating Progress Labels without Human Annotations with A VLM We conducted a formative study to explore generating progress labels without any human annotations. We used a ReWiND-style method [30] and a Gemma 3 12B model. We took the key idea of ReWiND: “use language instructions to represent key subgoals in a task, and then combine the language instructions with a VLM to indicate task progress.” We modified the prompt we used for the fine-tuned model, specify the subgoals with instructions, and use an unfine-tuned Gemma 3 12B to annotate the human demonstrations. The results are in Figure 5. Fig. 5: Comparisons Between Human Annotations and Auto-Generated Annotations. We compare the performance of the model at different annotation coverages. The auto-generated annotations from an unfine-tuned VLM can improve learning from demonstrations with PHIRL. Our results show that generated annotations from an unfine-tuned VLM can improve learning with PHIRL in most cases (ph: 169.62 vs. 167.33, 175.80, 180.13, and 182.54, and mh: 118.30 vs. 123.35, 141.92, 138.52, and 145.91), despite the performance is lower than using real human feedback. While the performance gain is less significant and saturates faster than with real human feedback, it demonstrates that PHIRL and other progress-based learning algorithms can benefit from a VLM trained with general knowledge. Limitations and Future Work We acknowledge several limitations in our work: 1. Our method assumes that the task has a finite horizon or a complete status, making it unsuitable for tasks with an infinite horizon or cyclic observations, such as ant_walk from DeepMind Control Suite [51]. 2. Methods are evaluated in one training run, as AIRL-based methods are generative adversarial inverse reinforcement learning methods that learn online with adversarial training; repeating every condition with multiple training seeds was beyond our budget. 3. As we discussed earlier, we randomly sampled demonstrations from the dataset for annotations rather than actively selecting imperfect demonstrations. We expect that preferentially selecting lower-quality demonstrations will improve the sample efficiency, and thus improve model performance, especially when the number of human feedback labels is limited. PHIRL currently relies on progress feedback. Recent advances show that semantic instructions from humans [28, 52] or VLMs [53, 54, 55] can serve as compelling modalities for reward generation and alignment. One future work is to improve the learned function by blending trajectory-level progress with language instructions and structured rubrics. VI Conclusions In conclusion, we proposed PHIRL, an inverse reinforcement learning method that learns a reward function by alternately inferring a reward function and aligning the inferred reward function with progress annotations over annotated demonstrations. PHIRL is flexible to varying human participation levels since the annotations can be collected before or during learning. We showed that PHIRL significantly outperforms baselines with 20% of demonstrations being annotated before learning. We also showed that VLM-generated progress could enhance PHIRL’s converged performance. PHIRL exhibited a robust performance in two reward hacking scenarios, Lift-Drop-Loop and Touch-And-Stay. We believe that PHIRL is an effective framework for learning robust reward functions, and the intuition of aligning the learned policy with human annotation is an efficient way of leveraging human demonstrations and human feedback. References [1] S. Ibrahim, M. Mostafa, A. Jnadi, H. Salloum, and P. Osinenko, “Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications,” 2024. [Online]. Available: https://arxiv.org/abs/2408.10215 [2] S. Fang, W. Gao, S. Goel, C. Thierauf, M. Scheutz, and J. Sinapov, “FLEX: A framework for learning robot-agnostic force-based skills involving sustained contact object manipulation,” in 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, p. 4782–4788. [3] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” The International Journal of Robotics Research, p. 02783649241273668, 2023. [4] F. Sasaki and R. Yamashina, “Behavioral cloning from noisy demonstrations,” in Int. Conf. on Learning Representations, 2020. [5] H. Sun and M. van der Schaar, “Inverse reinforcement learning meets large language model post-training: Basics, advances, and opportunities,” 2025. [Online]. Available: https://arxiv.org/abs/2507.13158 [6] G.-Y. Hao, L. van der Laan, A. Bibaut, and N. Kallus, “Reward transfer from inverse reinforcement learning: A coupled minimax approach,” 2026. [Online]. Available: https://arxiv.org/abs/2605.27834 [7] C. Li, S. Zeng, Z. Liao, J. Li, D. Kang, A. Garcia, and M. Hong, “Learning reward and policy jointly from demonstration and preference improves alignment,” 2024. [Online]. Available: https://arxiv.org/abs/2406.06874 [8] M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh, “Learning reward functions by integrating human demonstrations and preferences,” arXiv preprint arXiv:1906.08928, 2019. [9] T. W. Ayalew, X. Zhang, K. Y. Wu, T. Jiang, M. Maire, and M. R. Walter, “Progressor: A perceptually guided reward estimator with self-supervised online refinement,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, p. 10 297–10 306. [10] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei, “Reward learning from human preferences and demonstrations in atari,” Advances in neural information processing systems, vol. 31, 2018. [11] D. S. Brown, W. Goo, and S. Niekum, “Ranking-based reward extrapolation without rankings,” arXiv preprint arXiv:1907.03976, 2019. [12] W. Gao, P. Sheng, L. Qiu, Y. Ma, J. Zhao, B. Chen, K. Wang, Y. Tian, S. Mao, and T. He, “Step-level preference learning for generative agents in social simulations,” 2026. [Online]. Available: https://arxiv.org/abs/2607.14485 [13] S. Chen, C. Harrison, Y.-C. Lee, A. J. Yang, Z. Ren, L. J. Ratliff, J. Duan, D. Fox, and R. Krishna, “Topreward: Token probabilities as hidden zero-shot rewards for robotics,” arXiv preprint arXiv:2602.19313, 2026. [14] C. Yu, C. Sima, G. Jiang, H. Zhang, H. Mai, H. Li, H. Wang, J. Chen, K. Wu, L. Chen, L. Zhao, M. Shi, P. Luo, Q. Bu, S. Peng, T. Li, and Y. Yuan, “χ0 _0: Resource-aware robust manipulation via taming distributional inconsistencies,” 2026. [Online]. Available: https://arxiv.org/abs/2602.09021 [15] Q. Fang, H. Yu, S. Fang, J. Huang, Q. Chen, R. M. Aronson, and E. S. Short, “Charm: Considering human attributes for reinforcement modeling,” in 2025 34th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2025, p. 7–14. [16] X. Zhou, W. Sun, Q. Ma, Y. Xie, J. Liu, W. Du, S. Welleck, Y. Yang, G. Neubig, S. T. Wu, and M. Sap, “Mind the sim2real gap in user simulation for agentic tasks,” 2026. [Online]. Available: https://arxiv.org/abs/2603.11245 [17] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song, “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” 2024. [Online]. Available: https://arxiv.org/abs/2402.10329 [18] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu, “Egomimic: Scaling imitation learning via egocentric video,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, p. 13 226–13 233. [19] S. M. Jain, “Hugging face,” in Introduction to transformers for NLP: With the hugging face library and models to solve problems. Springer, 2022, p. 51–67. [20] H. Yu, Q. Fang, S. Fang, R. M. Aronson, and E. S. Short, “How much progress did i make? an unexplored human feedback signal for teaching robots,” 2025. [Online]. Available: https://arxiv.org/abs/2407.06459 [21] Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, K. Lin, A. Maddukuri, S. Nasiriany, and Y. Zhu, “robosuite: A modular simulation framework and benchmark for robot learning,” 2025. [Online]. Available: https://arxiv.org/abs/2009.12293 [22] C. Arzate Cruz and T. Igarashi, “A survey on interactive reinforcement learning: Design principles and open challenges,” in Proc. of the 2020 ACM designing interactive systems Conf., 2020, p. 1195–1209. [23] R. Akrour, M. Schoenauer, and M. Sebag, “Preference-based policy learning,” in Machine Learning and Knowledge Discovery in Databases: European Conf., ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proc., Part I 11. Springer, 2011, p. 12–27. [24] Y. Cui, Q. Zhang, B. Knox, A. Allievi, P. Stone, and S. Niekum, “The empathic framework for task learning from implicit human feedback,” in Conf. on Robot Learning. PMLR, 2021, p. 604–626. [25] W. B. Knox and P. Stone, “Interactively shaping agents via human reinforcement: The tamer framework,” in Proc. of the fifth Int. Conf. on Knowledge capture, 2009, p. 9–16. [26] S. A. Mehta and D. P. Losey, “Unified learning from demonstrations, corrections, and preferences during physical human–robot interaction,” ACM Transactions on Human-Robot Interaction, vol. 13, no. 3, p. 1–25, 2024. [27] H. Yu, R. M. Aronson, K. H. Allen, and E. S. Short, “From “thumbs up” to “10 out of 10”: Reconsidering scalar feedback in interactive reinforcement learning,” in 2023 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS). IEEE, 2023, p. 4121–4128. [28] A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. Hendryx, “Rubrics as rewards: Reinforcement learning beyond verifiable domains,” arXiv preprint arXiv:2507.17746, 2025. [29] H. Yu and E. S. Short, “Active feedback learning with rich feedback,” in Companion of the 2021 ACM/IEEE international conference on human-robot interaction, 2021, p. 430–433. [30] J. Zhang, Y. Luo, A. Anwar, S. A. Sontakke, J. J. Lim, J. Thomason, E. Biyik, and J. Zhang, “Rewind: Language-guided rewards teach robot policies without new demonstrations,” arXiv preprint arXiv:2505.10911, 2025. [31] S. Fang, H. Yu, Q. Fang, R. M. Aronson, and E. S. Short, “Demonstration sidetracks: Categorizing systematic non-optimality in human demonstrations,” in 2025 34th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2025, p. 857–863. [32] Z. Zhu and H. Hu, “Robot learning from demonstration in robotic assembly: A survey,” Robotics, vol. 7, no. 2, p. 17, 2018. [33] S. Hu and J. Clune, “Thought cloning: Learning to think while acting by imitating human thinking,” Advances in Neural Information Processing Systems, vol. 36, 2024. [34] J. Ren, G. Swamy, Z. S. Wu, J. A. Bagnell, and S. Choudhury, “Hybrid inverse reinforcement learning,” arXiv preprint arXiv:2402.08848, 2024. [35] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” 2018. [36] S.-W. Yoo and S.-W. Seo, “Learning multi-task transferable rewards via variational inverse reinforcement learning,” in 2022 International Conference on Robotics and Automation (ICRA). IEEE, May 2022, p. 434–440. [Online]. Available: http://dx.doi.org/10.1109/ICRA46639.2022.9811697 [37] K. Kim, J. Zhang, A. Ozdaglar, and P. A. Parrilo, “A unified linear programming framework for offline reward learning from human demonstrations and feedback,” 2024. [Online]. Available: https://arxiv.org/abs/2405.12421 [38] S. Zhang, Z. Chen, S. Chen, Y. Shen, Z. Sun, and C. Gan, “Improving reinforcement learning from human feedback with efficient reward model ensemble,” 2024. [Online]. Available: https://arxiv.org/abs/2401.16635 [39] Y. Duan, Y. Wang, W. Qiu, and H. Zhu, “Learning from demonstrations via capability-aware goal sampling,” arXiv preprint arXiv:2601.08731, 2026. [40] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum, “Safe imitation learning via fast bayesian reward inference from preferences,” in Int. Conf. on Machine Learning. PMLR, 2020, p. 1165–1177. [41] D. Yang, D. Tjia, J. Berg, D. Damen, P. Agrawal, and A. Gupta, “Rank2reward: Learning shaped reward functions from passive video,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, p. 2806–2813. [42] J. Zhang, C. Qian, H. Sun, H. Lu, D. Wang, L. Xue, and H. Liu, “Progresslm: Towards progress reasoning in vision-language models,” 2026. [Online]. Available: https://arxiv.org/abs/2601.15224 [43] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM, vol. 63, no. 11, p. 139–144, 2020. [44] A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in Icml, vol. 99. Citeseer, 1999, p. 278–287. [45] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” Advances in neural information processing systems, vol. 30, 2017. [46] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347 [47] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019. [48] Gemma Team, “Gemma 3 technical report,” arXiv:2503.19786 [cs.CL], mar 2025, google DeepMind. [Online]. Available: https://arxiv.org/abs/2503.19786 [49] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2106.09685 [50] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín, “What matters in learning from offline human demonstrations for robot manipulation,” 2021. [Online]. Available: https://arxiv.org/abs/2108.03298 [51] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq et al., “Deepmind control suite,” arXiv preprint arXiv:1801.00690, 2018. [52] T. Liu, R. Xu, T. Yu, I. Hong, C. Yang, T. Zhao, and H. Wang, “Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment,” arXiv preprint arXiv:2510.07743, 2025. [53] Y. J. Ma, J. Hejna, A. Wahid, C. Fu, D. Shah, J. Liang, Z. Xu, S. Kirmani, P. Xu, D. Driess, T. Xiao, J. Tompson, O. Bastani, D. Jayaraman, W. Yu, T. Zhang, D. Sadigh, and F. Xia, “Vision language models are in-context value learners,” 2024. [Online]. Available: https://arxiv.org/abs/2411.04549 [54] Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li, “Univla: Learning to act anywhere with task-centric latent actions,” arXiv preprint arXiv:2505.06111, 2025. [55] Y. Shukla, W. Gao, V. Sarathy, A. Velasquez, R. Wright, and J. Sinapov, “LgTS: Dynamic task sampling using LLM-generated sub-goals for reinforcement learning agents,” 2023. [Online]. Available: https://arxiv.org/abs/2310.09454