Paper deep dive
SynIL: Leveraging Synergy for Offline Imitation Learning from Imperfect Demonstration Datasets
Yuto Tanaka, Kyo Kutsuzawa, Martina Doku, Dai Owaki, Mitsuhiro Hayashibe
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/1/2026, 7:18:48 AM
Summary
The paper introduces SynIL, a framework for offline imitation learning that automatically assesses demonstration quality using motor synergy concepts from neuroscience. SynIL computes synergy scores from state-action trajectories via PCA and uses self-supervised reward regression to generate dense, transition-level reward signals. These rewards are then used in offline reinforcement learning (specifically IQL) to train policies. The method is evaluated on D4RL locomotion benchmarks and Robomimic manipulation datasets, showing it outperforms Behavior Cloning and matches or exceeds offline RL trained on true environment rewards, particularly in scenarios with imperfect or noisy demonstrations.
Entities (8)
Relation Signals (6)
SynIL → evaluatedon → D4RL
confidence 96% · Comprehensive evaluations on D4RL locomotion benchmarks
SynIL → evaluatedon → Robomimic
confidence 96% · multi-human Robomimic manipulation datasets demonstrate that synergy-derived rewards correlate strongly with ground-truth rewards.
SynIL → uses → Motor Synergy
confidence 95% · Grounded in neuroscientific evidence that motor synergy... correlates directly with motor proficiency, SynIL algorithmically quantifies synergy manifestation
SynIL → outperforms → Behavior Cloning
confidence 94% · SynIL substantially outperforms Behavior Cloning (BC)
SynIL → employs → Self-Supervised Reward Regression
confidence 92% · SynIL algorithmically quantifies synergy manifestation to generate dense, transition-level reward signals via self-supervised reward regression.
SynIL → usesasbase → IQL
confidence 90% · Train a policy by offline reinforcement learning(e.g. IQL)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Imitation learning enables robots to acquire complex skills directly from massive demonstration datasets, but its performance degrades severely when datasets are contaminated with suboptimal or noisy demonstrations. While prior quality-assessment methods attempt to filter or reweight data, they typically rely on manual pre-selection of expert reference data or task-specific heuristics, limiting scalability. To address this challenge, we introduce SynIL (Synergy-based Imitation Learning), a novel framework for automated, label-free demonstration quality assessment in offline reinforcement learning. Grounded in neuroscientific evidence that motor synergy, a low-dimensional coordinated structure in movement, correlates directly with motor proficiency, SynIL algorithmically quantifies synergy manifestation to generate dense, transition-level reward signals via self-supervised reward regression. Comprehensive evaluations on D4RL locomotion benchmarks and multi-human Robomimic manipulation datasets demonstrate that synergy-derived rewards correlate strongly with ground-truth rewards. Furthermore, SynIL substantially outperforms Behavior Cloning (BC) and achieves performance comparable to, and in sparse-reward human teleoperation scenarios, superior to, offline reinforcement learning trained on true environment rewards.
Tags
Links
- Source: https://arxiv.org/abs/2609.38225v1
- Canonical: https://arxiv.org/abs/2609.38225v1
Trouble viewing inline? Open PDF directly →
Full Text
46,825 characters extracted from source content.
Expand or collapse full text
[orcid=0000-0002-5326-7847] [orcid=0009-0008-8022-9206] [orcid=0000-0003-1217-3892] [orcid=0000-0001-6179-5706] †highlights: Synergy, a coordinated modular structure proposed in neuroscience, enables automated, label-free demonstration quality assessment in mixed-proficiency datasets. The proposed synergy scores provide dense transition-level rewards for offline reinforcement learning through self-supervised reward regression. The proposed SynIL framework significantly outperforms Behavior Cloning on D4RL benchmarks and substantially improves task success rates on multi-human Robomimic environments. SynIL: Leveraging Synergy for Offline Imitation Learning from Imperfect Demonstration Datasets Yuto Tanaka yuto.tanaka.r8@alumni.tohoku.ac.jp organization=Department of Robotics, Graduate School of Engineering, Tohoku University, addressline=6-6-01 Aoba, Aramaki, Aoba-ku, city=Sendai, postcode=980-8579, state=Miyagi, country=Japan Kyo Kutsuzawa kutsuzawa@tohoku.ac.jp Martina Doku martina.doku.t7@dc.tohoku.ac.jp Dai Owaki owaki@tohoku.ac.jp Mitsuhiro Hayashibe hayashibe@tohoku.ac.jp https://neuro.mech.tohoku.ac.jp/ Abstract Imitation learning enables robots to acquire complex skills directly from massive demonstration datasets, but its performance degrades severely when datasets are contaminated with suboptimal or noisy demonstrations. While prior quality-assessment methods attempt to filter or reweight data, they typically rely on manual pre-selection of expert reference data or task-specific heuristics, limiting scalability. To address this challenge, we introduce SynIL (Synergy-based Imitation Learning), a novel framework for automated, label-free demonstration quality assessment in offline reinforcement learning. Grounded in neuroscientific evidence that motor synergy—a low-dimensional coordinated structure in movement—correlates directly with motor proficiency, SynIL algorithmically quantifies synergy manifestation to generate dense, transition-level reward signals via self-supervised reward regression. Comprehensive evaluations on D4RL locomotion benchmarks and multi-human Robomimic manipulation datasets demonstrate that synergy-derived rewards correlate strongly with ground-truth rewards. Furthermore, SynIL substantially outperforms Behavior Cloning (BC) and achieves performance comparable to—and in sparse-reward human teleoperation scenarios, superior to—offline reinforcement learning trained on true environment rewards. keywords synergy ,imitation learning ,offline reinforcement learning ,imperfect demonstrations ,demonstration quality assessment †titlenote: This work was supported by JSPS KAKENHI Grant Number JP24K00841.†credit: Conceptualization of this study, Methodology, Software, Analysis and interpretation of data, Drafting the article†credit: Conceptualization of this study, Methodology, Software, Analysis and interpretation of data, Drafting the article†credit: Conceptualization of this study, Methodology, Drafting the article†credit: Conceptualization of this study, Methodology, Drafting the article†credit: Conceptualization of this study, Methodology, Drafting the article†corresponding: Corresponding author 1 Introduction τ1 _1 =s1,a1,s2,a2,⋯,sT,aT =\s_1,a_1,s_2,a_2,·s,s_T,a_T\ τ2 _2 =s1,a1,s2,a2,⋯,sT,aT =\s_1,a_1,s_2,a_2,·s,s_T,a_T\ τN _N =s1,a1,s2,a2,⋯,sT,aT =\s_1,a_1,s_2,a_2,·s,s_T,a_T\ Collect a datasetX,X^X, XState, ActionCCActivatity coefficients⋅· WWSynergiesDecomposeComposeCompute synergy score ξ by PCAD LSSRR(θ)=τ∼[(∑t′=t+H−1rθ(st′,at′)−ξt)2] L_SSRR(θ)=E_τ [ ( _t =t^t+H-1r_θ(s_t ,a_t )- _t )^2 ] Train a reward model rθr_θ by SSRRξ π(a|s) π(a|s) Train a policy by offline reinforcement learning(e.g. IQL)rθr_θ Figure 1: Overview of SynIL. Synergy scores ξ are calculated from trajectories of a dataset D (top-right), and a reward model is trained based on the synergy scores (bottom-right). Offline reinforcement learning is finally performed with rewards computed by the trained reward model (bottom-left). Imitation learning, a method for mimicking expert demonstrations, has gained attention as a new learning paradigm that can potentially outperform manual programming requiring considerable time and a high level of expertise in coding and control. Unlike traditional manual programming methods, imitation learning is a data-driven approach, allowing learning without the need for complex design, programming skills, and specialized knowledge. Imitation learning is highly regarded as a more realistic approach in fields such as robotics, healthcare, and autonomous driving (Zare et al., 2024; Kim et al., 2024; Pan et al., 2017). In robotics, imitation learning has successfully automated numerous actions across various robots, as demonstrated by robotic foundation models (Kawaharazuka et al., 2025). In spite of its potential, the performance of imitation learning is adversely affected by low quality teaching data (Mandlekar et al., 2021; Belkhale et al., 2023). A large amount of optimal demonstration data is required to ensure the stable performance of imitation learning. However, due to factors such as the difficulty of the task, the skill level of the expert, and issues like fatigue and concentration when collecting demonstration data, the data obtained may include suboptimal data. As the dataset grows larger, the likelihood of including incomplete data and noise increases. Presence of suboptimal or poor-quality demonstrations gets imitation learning to mimic these poor-quality demonstrations, resulting in decreased performance. Evaluating the quality of demonstrations will overcome the above issue. For example, DemoDICE (Kim et al., 2022), ISWBC (Li et al., 2024), and DWBC (Xu et al., 2022) pre-divide the training data into optimal data DED_E from experts and DUD_U with unknown proficiency, enhancing imitation learning performance by focusing more on expert data or adjusting the distributional distance with non-optimal data. Additionally, CLUE (Liu et al., 2023) and ORIL (Zolna et al., 2020) address this issue by converting distributional distances with optimal data into rewards and conducting offline reinforcement learning, which considers these rewards. This approach has the potential to enable to learn behaviors beyond the (possibly poor) quality of the demonstrations. However, in all these studies, it is necessary for humans to pre-determine the expert data. Furthermore, when it comes to qualitative evaluation, it may be challenging to discern which instructional actions are optimal depending on the task. Therefore, a method to automatically and quantitatively evaluate optimality is needed. In this study, we focus on the concept of synergy, which is proposed in neuroscience, for automated demonstration-quality assessment. Synergy refers to the low-dimensional structure of coordinated muscles and joints, which are often observed in human proficient movements (Latash et al., 2014; Turvey, 1990). Many studies have reported that the degree of synergy manifestation corresponds to the proficiency of movements (Zaal et al., 1999; Gentner et al., 2010). This phenomenon has also been observed from reinforcement learning in locomotion (Chai and Hayashibe, 2020) and reaching movements (Han et al., 2021). Synergies and their degree of manifestation can be obtained algorithmically. Thus, we expect that synergy can be used for automated demonstration-quality assessment. In this study, we propose a method of imitation learning with automatic assessment of demonstration quality. We propose a method to compute rewards from unlabelled demonstrations based on synergy and use them in offline reinforcement learning. Our contributions are as follows: • We propose a method to algorithmically compute the quality of demonstrations based on synergy and use it as reward for learning. • This study is the first application of the concept of synergy to assess the quality of demonstration movements in imitation learning. • We successfully improved imitation learning performance across various datasets, including real human demonstrations. 2 Related Work 2.1 Imitation Learning Imitation learning is a learning method aimed at mimicking the demonstrated actions of experts (such as humans) (Zare et al., 2024). This approach enables robots to learn behaviors directly from expert behaviors, making it beneficial for tasks where manually programming movements or rules is difficult. Standard imitation learning includes offline methods, which do not involve real environments, such as Behavior Cloning (BC). In general, the policy πθ _θ with parameters θ learns to approximate the action a taken in state s to the state-action pairs (s,a)(s,a) in the expert’s dataset D. The loss function for BC can be expressed as follows: Lπ(θ)=(s,a)∼[−logπθ(a|s)].L_π(θ)=E_(s,a) [- _θ(a|s)]. (1) For continuous actions, a deterministic policy is often trained with a mean-squared error objective. BC is fully offline and requires neither an explicit reward nor further interaction with the environment. Recent robotic foundation models (Kawaharazuka et al., 2025) are trained with almost the same procedure as BC. Although BC is simple and powerful technique, it has several issues. First, BC tends to accumulate approximation errors when the trained policy interacts with the environment in a closed loop, unlike the open-loop training procedure in Equation 1. It motivates extended techniques such as DAgger (Ross and Bagnell, 2010), inverse reinforcement learning (Abbeel and Ng, 2004), and adversarial learning (Ho and Ermon, 2016). Second, as Equation 1 assigns the same importance to all samples, its performance is sensitive to the quality and consistency of the demonstrations. Offline reinforcement learning techniques will address this issue by associating rewards with states and actions. 2.2 Offline Reinforcement Learning Standard reinforcement learning (online reinforcement learning) learns by interacting with the environment. However, online reinforcement learning can be inefficient because it requires collecting data again to change the policy, and there is a risk of interacting with robots or vehicles online. Therefore, while reinforcement learning research is conducted in simulation environments that consider safety and efficiency, there are still few applications in the real world. Offline reinforcement learning is a method that learns only from a pre-collected static dataset without interacting with the environment. Unlike online reinforcement learning, offline reinforcement learning does not interact with the real world, making it effective in settings where data collection is high-cost or dangerous, and online interaction is impractical (such as in robotics (Singh et al., 2022) and autonomous driving (Fang et al., 2022)). In principle, any off-policy algorithm (an online reinforcement learning algorithm that predicts policies using past experiences) can be applied in offline reinforcement learning. However, in many offline settings, off-policy algorithms perform poorly. This is because the value function overestimates actions not supported by the dataset (OOD, Out-of-Distribution), which is due to distribution shifts where the distribution of the dataset differs from the distribution of the learned policy (Prudencio et al., 2023). Many offline reinforcement learning algorithms attempt to address distribution shifts, with representative methods including BCQ (Fujimoto et al., 2019) and IQL (Kostrikov et al., 2021). IQL trains a policy with the following loss function: Lπ(θ)=(s,a)∼[−exp(β(Qϕ(s,a)−Vψ(s)))logπθ(a|s)],L_π(θ)=E_(s,a) [- (β(Q_φ(s,a)-V_ψ(s))) _θ(a|s)], (2) where Qϕ(s,a)Q_φ(s,a) and Vψ(s)V_ψ(s) are the action-value and value functions, trained from rewards. This can be viewed as an extension of BC, weighting each state-action pair by its advantage. SynIL uses IQL because its weighted policy extraction provides a natural way to reflect continuous estimates of demonstration quality in the learned policy. 2.3 Algorithms for Demonstration Quality Research has been conducted to improve the performance of imitation learning using datasets of varying quality. For example, DemoDICE by (Kim et al., 2022), ISWBC by (Li et al., 2024), and DWBC (Xu et al., 2022) pre-divide the training data into optimal data DED_E from experts and DUD_U with unknown proficiency, enhancing imitation learning performance by focusing more on expert data or adjusting the distributional distance with non-optimal data. Additionally, CLUE by Liu et al. (2023) and ORIL by Zolna et al. (2020) address this issue by converting distributional distances with optimal data into rewards and conducting offline reinforcement learning, which considers these rewards. However, in these studies, it is necessary for humans to pre-determine the expert data. Furthermore, when it comes to qualitative evaluation, it may be challenging to discern which instructional actions are optimal depending on the task. Therefore, a method to automatically and quantitatively evaluate optimality is needed. Several studies have aimed to evaluate demonstration quality automatically. Zhang et al. (2026) proposed a method for eliminating suboptimal samples from demonstrations automatically by using a self-supervised task progress predictor and a deduplication module. Kulkarni et al. (2026) proposed using trajectory smoothness as a measure of demonstration quality to reduce the effect of noise and demonstrated that extracting smooth trajectories resulted in better performance than simply using the entire demonstrations. However, these methods simply remove bad trajectories from training. Even bad trajectories contain information about state transitions in the environment, so that information is also lost. Also, Kulkarni et al. (2026) evaluated quality based on trajectories rather than state-action pairs. This will eliminate the good parts of bad trajectories as well, reducing sample efficiency. Therefore, scoring demonstration quality for each state-action pair or transition without eliminating samples would be effective. 3 Methodology 3.1 Overview This section describes the proposed method, SynIL, which derives its name from synergy-based imitation learning. This method allows a policy to imitate demonstrations by performing offline reinforcement learning based on rewards computed from the degree of synergy manifestation. In SynIL, demonstrations do not have to include rewards; instead, rewards are computed from demonstrations using synergy, a concept proposed in neuroscience that quantifies the degree of movement coordination. Unlike most conventional methods, SynIL does not require manual evaluation of demonstrations, except for setting several hyperparameters. An overview of SynIL is provided in Figure 1. First, the degree of synergy manifestation is quantified as a synergy score ξ. Next, the synergy scores ξ are converted into rewards by using self-supervised reward regression (SSRR) (Chen et al., 2021). Finally, offline reinforcement learning is performed using the synergy-based rewards associated with state-action pairs. We assume a dataset D=τii=1M,τi=(t,t,t+1)t=0Ti−1,D= \ _i \_i=1^M, _i= \( s_t, a_t, s_t+1) \_t=0^T_i-1, (3) where t s_t and t a_t indicate the state and action at time t, respectively. No reward, success, or proficiency label is available during SynIL training. 3.2 Acquisition of Synergy Scores based on Synergy Manifestation As the first step of SynIL, the quality of demonstration is quantified based on synergy. Among various types of synergy, we employ the spatial synergy, which represents low-dimensional coordination patterns in multi-dimensional spatial data. The spatial synergy is defined by the following equation: t=tntn+¯t+residuals, X_t= W_t^n C_t^n+ X_t+residuals, (4) where t=[t+1⋯t+(H−1)t+1⋯t+(H−1)]. X_t= bmatrix s_t& s_t+1&·s& s_t+(H-1)\\ a_t& a_t+1&·s& a_t+(H-1) bmatrix. (5) Here, data t X_t is the H-step trajectory of the state-actions cut from a demonstraion, n is the number of spatial synergies, tn W_t^n corresponds to the spatial synergies, and tn C_t^n represents the activation level of the spatial synergies at each time. Also, ¯t X_t incidates the row-wise mean values of t X_t. Equation 4 is computed by the principle component analysys (PCA), which computes tn W_t^n and tn C_t^n to minimize residualsresiduals. It can be said that the smaller the residualsresiduals are, the more significant the spatial synergies are manifest. Note that each state and action dimension is normalized using statistics from the entire training dataset before applying PCA. This prevents PCA from letting variables with large numerical scales dominate the decomposition independently of their importance. Finally, PCA can score how compactly the local movement segment can be represented, regardless of the global semantics of the entire trajectory and dataset. Based on Equation 4, the degree of synergy manifestation is quantified by a metric denoted by (R2)tn(R^2)_t^n, as follows: (R2)tn=1−‖t−¯t−tntn‖F2‖t−¯t‖F2.(R^2)_t^n=1- \| X_t- X_t- W_t^n C_t^n\|_F^2\| X_t- X_t\|_F^2. (6) Here, ∥∙∥F\| \|_F represents the Frobenius norm. (R2)tn(R^2)_t^n ranges from 00 to 11, and the larger the value of (R2)(R^2) is, the higher the degree of synergy manifestation is. However, the above (R2)tn(R^2)_t^n metric varies depending on the number of synergies, n. Instead, this study uses the synergy level (Chai and Hayashibe, 2020), ζt _t, as an indicator of synergy manifestation independent of n, defined as follows: ζt=1N∑n=1N(R2)tn, _t= 1N _n=1^N(R^2)_t^n, (7) where N indicates the degree of freedom of demonstrations. Note that (R2)t0≡0(R^2)_t^0≡ 0, and (R2)tn≡1(R^2)_t^n≡ 1 for n≥Nn≥ N, so the synergy score sums up with a range of n=1,…,Nn=1,…,N. In addition to the synergy level, ζ, we also consider the standard deviation of the demonstrations. This is because only use of the synergy level can lead to the problem of high evaluation of demonstrations with overly simplistic states and actions, such as a demonstration with no movement. It is undesirable in many motor tasks to highly valuing the movement of doing nothing. To address this issue, we use the following synergy score, ξt _t, that considers the standard deviation of state-action pairs as well as the synergy level: ξt=logζt+ηlogσ¯t. _t= _t+η σ_t. (8) Here, σ¯t σ_t is the mean value of the standard deviation of columns of t X_t. η is a hyperparameter to weight the standard deviation of the demonstration action. 3.3 Reward Estimation Based on Synergy As shown in Equations 5–8, the synergy score ξt _t are computed from an H-step state-action trajectory. However, to use offline reinforcement learning, we need rewards associated with each state-action pair. To bridge this gap, we consider that the synergy score ξt _t, computed from an H-step state-action trajectory, approximates the total reward of the H-step state-action trajectory, as expressed in the following equation: ξt≈∑t′=t+H−1rt′, _t≈ _t =t^t+H-1r_t , (9) where rt′r_t indicates a reward at time t′t . To acquire such rewards, we construct a reward model that maps a state-action pair to a reward value. Let a reward model be rθ(t,t)r_θ( s_t, a_t), where t s_t and t a_t denote a state and action at time t, respectively, and θ indicates the parameters of the reward model. The reward model is trained to minimize the following loss function: LSSRR(θ)=τ∼[(∑t′=t+H−1rθ(st′,at′)−ξt)2].L_SSRR(θ)=E_τ [ ( _t =t^t+H-1r_θ(s_t ,a_t )- _t )^2 ]. (10) Minimizing this loss function gets the reward model to generate rewards which satisfy Equation 9. This method is similar to the self-supervised reward regression (SSRR) (Chen et al., 2021), with the noise level replaced with the synergy score. 4 Evaluation 4.1 Dataset To evaluate the proposed method, we used D4RL (Fu et al., 2020) and Robomimic (Mandlekar et al., 2021) datasets as demonstration datasets. D4RL is a dataset for imitation learning and offline reinforcement learning, offering datasets for various environments and problem settings. Among various tasks, we specifically used the locomotion dataset as a typical motor task. The locomotion dataset includes three types of robots: Hopper, HalfCheetah, and Walker2d. D4RL contains several types of datasets with varying quality in terms of the performance and rewards of the expert policies. The “medium” dataset contains demonstrations with moderate quality, the “medium-replay” dataset is a mixed dataset of low to moderate quality demonstrations, and the “medium-expert” dataset is also a mix dataset of medium to high quality demonstrations. Each dataset was generated by policies at different training stages of the reinforcement learning algorithm, soft actor-critic (Haarnoja et al., 2018). These datasets are suitable for our problem setting, as they involve varying-quality demonstrations. It is known that typical imitation learning, such as behavior cloning (BC), decreases the performance with these types of datasets. Robomimic, on the other hand, is an imitation-learning dataset with situations similar to real-world applications. The dataset is composed of demonstraions in various tasks with a seven degrees-of-freedom manipulator with a gripper. The demonstrations have been acquired by human operators with tele-operation in a simulator. Among datasets in Robomimic, we used the multi-human (MH) datasets of Can and Square tasks. The MH dataset contains a total of 300 demonstrations, with 50 collected from each of six individuals with varying levels of expertise. 4.2 Problem Setting In the evaluation, we assume a dataset can be expressed as follows: =(t,t,t+1)|t=0,…,T−1,D= \( s_t, a_t, s_t+1) |t=0,…,T-1 \, (11) where t s_t and t a_t indicates a state and action at time t. We assume that the dataset does not have rewards or any sort of quality values for each tuple of (t,t,t+1)( s_t, a_t, s_t+1). Although the D4RL datasets contains rewards, we excluded them to emulate the sitations where only demonstrations were provided. Thus, reinforcement learning methods cannot be applied unless we compute reward values. We compared the proposed method with BC, a common imitation learning method. Additionally, for reference, we also compared them with the implicit Q-learning (IQL) (Kostrikov et al., 2021) with datasets containing the true rewards. This comparison was conducted to discuss how our method, which uses estimated rewards, is comparable with offline reinforcement learning with true rewards. The algorithm for SynIL is presented in Algorithm 1. Algorithm 1 SynIL 0: dataset D of state-action trajectories, as defined in Equation 11 0: a policy π and a reward model rθr_θ for each trajectory τ∼τ do Compute ζt _t and σ¯t σ_t from H-step trajectories cut from τ by Equation=7 Compute ξt _t by Equation 8 end for Initialize reward function parameters θ for each epoch do for each trajectory τ in D do Compute loss ℒSSRR(θ)L_SSRR(θ) by Equation 10 Update θ to minimize ℒSSRR(θ)L_SSRR(θ) through gradient descent end for end for Train a policy π by offline reinforcement learning with the trained reward model rθr_θ 4.3 Hyperparameters The neural network and Synergy Score hyperparameters used in the SSRR are listed in Table 1. The input dimension of the reward estimation model (RewardNet) was equal to the sum of the state and action dimensions, and the output dimension was one-dimensional since it represents the reward. H and S indicate the horizon and stride; H-step trajectories are cut from a demonstration while shifting the start position by S-steps. Additionally, η was set through hand-tuning to achieve the highest correlation with the true reward. Hyperparameters for SynIL, IQL, and BC are shown in Table 2. IQL becomes equivalent to the BC algorithm by setting the hyperparameter β to 0. Therefore, in D4RL, BC experiments were conducted by setting IQL’s β to 0. The hyperparameters were hand-tuned to achieve the highest performance for each algorithm. Table 1: Hyperparameters for SSRR Hyperparameter Value in D4RL Value in Robomimic RewardNet Hidden layers 2 2 Hidden Size 512 1024 Activation ReLU ReLU Last Activation Linear Linear Epochs 500 10000 Batch Size 64 64 Learning Rate 1×10−41× 10^-4 1×10−31× 10^-3 Weight Decay 1×10−21× 10^-2 00 Criterion MSELoss MSELoss Optimizer Adam Adam Synergy Score Horizon H 200 20 Stride S 100 20 η 0.05 0.05 Table 2: Hyperparameters for imitation learning Hyperparameter SynIL and IQL BC Hidden size 256 256 Hidden layers 2 2 Activation ReLU ReLU Optimizer Adam Adam Epochs 10610^6 10610^6 Evaluate period 5000 5000 Batch size 256 256 Learning rate 3×10−43× 10^-4 3×10−43× 10^-4 Discount 0.99 - α 0.005 - τ 0.7 - β 3.0 - 4.4 Results Figure 2: Comparison of synergy-based/true rewards for different datasets in D4RL. The vertical axes correspond to the cumulative rewards over 200 steps estimated by SSRR, and the horizontal axes correspond to the true cumulative rewards over 200 steps. Table 3: Correlation coefficients between the synergy-based and true rewards in D4RL Dataset Correlation coefficient halfcheetah-medium-expert-v2 0.99590.9959 walker2d-medium-expert-v2 0.86550.8655 hopper-medium-expert-v2 0.87180.8718 halfcheetah-medium-v2 0.94820.9482 walker2d-medium-v2 0.58090.5809 hopper-medium-v2 0.95260.9526 halfcheetah-medium-replay-v2 0.80390.8039 walker2d-medium-replay-v2 0.82630.8263 hopper-medium-replay-v2 0.53140.5314 Table 4: Performance of each algorithm in D4RL (Average sum of rewards over 10 episodes) Dataset BC SynIL (Ours) IQL (ground-truth reward) halfcheetah-medium-v2 42.7±0.342.7± 0.3 45.4±0.2 45.4± 0.2 47.4±0.247.4± 0.2 hopper-medium-v2 54.4±1.7 54.4± 1.7 46.6±3.846.6± 3.8 54.4±3.554.4± 3.5 walker2d-medium-v2 71.7±6.371.7± 6.3 72.0±5.4 72.0± 5.4 80.0±4.980.0± 4.9 halfcheetah-medium-replay-v2 37.1±2.437.1± 2.4 40.6±1.4 40.6± 1.4 44.3±0.544.3± 0.5 hopper-medium-replay-v2 14.4±2.414.4± 2.4 99.9±0.5 99.9± 0.5 66.5±8.166.5± 8.1 walker2d-medium-replay-v2 15.5±6.415.5± 6.4 74.5±3.0 74.5± 3.0 69.5±7.969.5± 7.9 halfcheetah-medium-expert-v2 60.9±5.060.9± 5.0 90.9±1.3 90.9± 1.3 87.0±3.187.0± 3.1 hopper-medium-expert-v2 51.4±1.851.4± 1.8 80.9±8.0 80.9± 8.0 2.2±0.02.2± 0.0 walker2d-medium-expert-v2 80.0±5.880.0± 5.8 108.9±0.3 108.9± 0.3 110.7±0.5110.7± 0.5 Table 5: Performance comparison in Robomimic (average success rate, 3 seeds × 50 rollouts) Dataset BC SynIL (Ours) IQL (sparse reward) Can (MH) 56.7±10.056.7± 10.0 94.0±3.3 94.0± 3.3 67.3±5.067.3± 5.0 Square (MH) 24.0±4.324.0± 4.3 49.3±8.2 49.3± 8.2 16.7±5.216.7± 5.2 Table 6: Execution time in Robomimic (seconds) Dataset BC SynIL (Ours) IQL (sparse reward) Can (MH) 2.1±0.22.1± 0.2 1.3±0.0 1.3± 0.0 2.2±0.22.2± 0.2 Square (MH) 2.9±0.02.9± 0.0 2.0±0.1 2.0± 0.1 3.4±0.23.4± 0.2 Figure 3: Learning curves of all environments. Figure 4: Learning curves of Robomimic. 4.4.1 Correlation between Estimated Reward and Ground-Truth Reward First, we investigated the relationship between the synergy-based rewards and ground-truth rewards in D4RL datasets. Figure 2 shows scatter plots of the rewards estimated by the SSRR (vertical axis, cumulative reward over 200 steps) and true rewards (horizontal axis, cumulative reward over 200 steps). Additionally, Table 3 presents the correlation coefficients between the rewards estimated by the SSRR and true rewards. As a result, the correlation coefficients were more than 0.80.8 for all datasets except for the walker2d-medium-v2 and hopper-medium-replay-v2. Particularly, the correlation coefficients were more than 0.860.86 in all medium-expert datasets. 4.4.2 Performance Comparison We evaluated the performance of imitation learning of SynIL (the proposed method), BC, and IQL in D4RL datasets. Table 4 shows the performance, which was computed as the average sum of rewards over 10 episodes. The values were normalized according to Fu et al. (2020). As a result, according to Table 4, SynIL outperformed BC on most datasets and was comparable to IQL with true rewards. This is particularly evident in datasets of varying quality such as medium-replay and medium-expert. On the other hand, BC, which performs imitation by supervised learning, shows decreased performance on datasets with diverse quality like medium-expert. We then evaluated the performance of imitation learning in Robomimic environments. Table 5 presents the performance, which was evaluated as the mean success rates for 50 trials across three different initial values (i.e. a total of 150 trials). Table 6 shows the execution time in seconds. Note that, since the Robomimic datasets did not contain reward data, we used sparse rewards, where a reward of 11 was applied at the final steps of successful episodes, and 00 was applied otherwise. As a result, according to Table 5, SynIL significantly outperformed both BC and even IQL. Additionally, from Table 6, it is evident that SynIL had a shorter execution time than both BC and IQL. Learning curves in D4RL and Robomimic are shown in Figures 3 and 4. 5 Discussion 5.1 Effectiveness of Synergy Rewards in Imitation Learning The results suggest that the synergy-based rewards, calculated from the degree of synergy manifestation, can be used as surrogate rewards quantifying the quality of demonstrations. The synergy-based rewards have high correlation with the ground-truth rewards, as shown in Tables 3. Also, the use of the synergy-based rewards resulted in high performance in imitation learning, as shown in Tables 4, and 5. Those results demonstrate the effectiveness of the proposed method, that is, the use of synergy for evaluating the quality of demonstrations. The improvement over BC is explained as selective use from mixed-quality data. BC treats all collected actions as targets, suffering from suboptimal samples. In contrast, SynIL first constructs a continuous quality signal, then allows IQL to propagate that signal through a long-horizon value function. High-quality segments can contribute even if they occur within a mediocre trajectory, while coordinated failures can be downweighted if their inferred long-term value is low. This argument is supported by the results of Hopper and Walker2d medium-replay and all medium-expert datasets, as described in Table 4; these are precisely the settings where the dataset contains behavior from substantially different stages of learning. Hopper medium is the exception, showing that the proxy can be unhelpful when quality differences are weak, when relevant task aspects are not expressed by linear coordination, or when reward-model errors alter IQL training. The use of synergy-based rewards even outperformed IQL with ground-truth rewards in some cases of D4RL and in Robomimic. This is because the ground-truth rewards we used in Robomimic were sparse, which was hard to train a well-performing policy. The synergy-based rewards, in contrast, could densely provide more information and function as an important guide for successful completion in the task. This also highlights the advantage of the synergy-based rewards that densely evaluate movement coordination. Note that this comparison should not be interpreted as proof that synergy is a more correct objective than the environment reward. Offline RL performance is sensitive to reward scale, terminal handling, implementation details, and hyperparameters. The feasible conclusion is that the synergy-based rewards contain enough structure to support strong offline policy learning in these datasets. 5.2 Relationship Between Motor Learning and Synergy The results shown in Tables 4 and 5 suggest that the synergy-based reward was effective in improving the performance of imitation learning. Here, although it has not fully been proven, we discuss why the synergy worked well in D4RL and Robomimic. The success of the synergy-based method in D4RL would be caused by the nature of reinforcement learning. The demonstraions in D4RL datasets have been generated by neural-network policies trained by deep reinforcement learning. In reinforcement learning, an agent initially takes random state-action pairs, but as learning progresses, it takes more regular states and actions. During this learning process, the dimensionality of the state-action distribution decreases. This links to the increase of synergy manifestation. Therefore, learning state-action relationship where synergy is highly manifested will result in the selective learning of more proficient behaviors. This would be a reason why the proposed method resulted in a high performance in the D4RL datasets. A similar phenomenon has been reported in Chai and Hayashibe (2020); the degree of synergy manifestation increases as deep reinforcement learning progresses. In future work, these findings may lead to a new reinforcement learning framework that prefers to learn coordinated movements to accelerate motor learning. Moreover, previous works have reported that synergies obtained in some tasks can generalize to new tasks (Al Borno et al., 2020; Kutsuzawa and Hayashibe, 2022). These findings also suggest that the combination of synergy and reinforcement learning can improve the ability of motor learning. On the other hand, the reason of the success of the synergy-based method in Robomimic would relate to the nature of the human motor learning. It is known that many human movements become more regular, stable, and low-dimensional as learning progresses (Davids et al., 2012). Even in unpredictable tasks, diverse actions are initially taken, but ultimately they converge to typical patterns of adaptive responses due to the principle of optimality (Braun et al., 2009). In such cases, it is considered that synergy in state actions also manifests. In addition to the above, the effectiveness of synergy can also be considered from the perspective of computational neuroscience. A recent theory in computational neuroscience argues that human’s central nervous systems perform recognition and action generation so as to minimize the variational free energy (Friston and Stephan, 2007). Minimizing the variational free energy over time requires to decrease the amount of uncertainty, resulting in reduction of the entropy of the sensory signals (Friston, 2012). Reduction in the entropy will bring about more coordinated distributions of the sensory signals and actions, resulting in the increase of the degree of synergy manifestation. Therefore, it is possible that as a human becomes more proficient in a task, the variational free energy decreases more efficiently, thereby the synergy is highly manifested. 5.3 Limitations The main limitation is that synergy is not universally equivalent to optimality. A repetitive failure can be strongly coordinated, while a successful recovery can be high-dimensional. SynIL is most plausible when inefficient demonstrations contain unstable corrections or inconsistent coupling and when proficient demonstrations use repeatable motor patterns. Some studies have reported the relationship between synergy manifestation and optimality (Todorov and Jordan, 2002; Chai and Hayashibe, 2020), but we could not provide solid evidence its generality. Another limitation of this study is hyperparameter tuning. Synergy score, defined in Equation 8, has several hyperparameters: η, H, and S. We need to tune them appropriately to bring out the potential of the proposed method. In particular, in the evaluations, we manually tuned η to obtain high correlation with true rewards. This means we implicitly used true reward information when tuning η, so the setup is not yet a fully reward-free procedure for hyperparameter selection. However, even with this setup, it is possible to argue that synergy contains sufficient information to evaluate demonstration quality and construct a surrogate reward, given an appropriate parametrization. We also used the same values across diverse tasks and agent morphologies in D4RL, suggesting that the hyperparameters are not very task-sensitive. 6 Conclusion To overcome the vulnerability of imitation learning to imperfect demonstration datasets, this study introduced SynIL, an offline reinforcement learning framework utilizing synergy-derived quality evaluation. We demonstrated that biological motor synergy, extracted algorithmically via PCA, serves as an effective quality score for unannotated trajectory segments and successfully generates dense rewards through self-supervised regression. SynIL outperformed Behavior Cloning and demonstrated strong performance across benchmark locomotion and human teleoperation datasets, confirming the utility of synergy as a surrogate reward signal. Beyond robotic control, our findings underscore the potential of integrating neuroscientific principles into machine learning pipelines. Future work will aim to address current limitations, particularly by eliminating the reliance on ground-truth rewards for hyperparameter selection, clarifying the conditions under which synergy aligns with task optimality, and deploying the proposed framework to real-world physical AI platforms. References Abbeel and Ng (2004) Abbeel, P., Ng, A.Y., 2004. Apprenticeship learning via inverse reinforcement learning, in: Twenty-first international conference on Machine learning - ICML ’04, p. 1–8. doi:10.1145/1015330.1015430. Al Borno et al. (2020) Al Borno, M., Hicks, J.L., Delp, S.L., 2020. The effects of motor modularity on performance, learning and generalizability in upper-extremity reaching: a computational analysis. Journal of The Royal Society Interface 17, 20200011. doi:10.1098/rsif.2020.0011. Belkhale et al. (2023) Belkhale, S., Cui, Y., Sadigh, D., 2023. Data quality in imitation learning, in: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. p. 80375–80395. doi:10.52202/075280-3524. Braun et al. (2009) Braun, D.A., Aertsen, A., Wolpert, D.M., Mehring, C., 2009. Learning optimal adaptation strategies in unpredictable motor tasks. Journal of Neuroscience 29, 6472–6478. Chai and Hayashibe (2020) Chai, J., Hayashibe, M., 2020. Motor synergy development in high-performing deep reinforcement learning algorithms. IEEE Robotics and Automation Letters 5, 1271–1278. Chen et al. (2021) Chen, L., Paleja, R., Gombolay, M., 2021. Learning from suboptimal demonstration via self-supervised reward regression, in: Conference on robot learning, PMLR. p. 1262–1277. Davids et al. (2012) Davids, K., Araújo, D., Hristovski, R., Passos, P., Chow, J.Y., 2012. Ecological dynamics and motor learning design in sport. Skill acquisition in sport: Research, theory and practice 2, 112–130. Fang et al. (2022) Fang, X., Zhang, Q., Gao, Y., Zhao, D., 2022. Offline reinforcement learning for autonomous driving with real world driving data, in: 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), IEEE. p. 3417–3422. Friston (2012) Friston, K., 2012. A free energy principle for biological systems. Entropy 14, 2100–2121. Friston and Stephan (2007) Friston, K.J., Stephan, K.E., 2007. Free-energy and the brain. Synthese 159, 417–458. Fu et al. (2020) Fu, J., Kumar, A., Nachum, O., Tucker, G., Levine, S., 2020. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 . Fujimoto et al. (2019) Fujimoto, S., Meger, D., Precup, D., 2019. Off-policy deep reinforcement learning without exploration, in: International conference on machine learning, PMLR. p. 2052–2062. Gentner et al. (2010) Gentner, R., Gorges, S., Weise, D., aufm Kampe, K., Buttmann, M., Classen, J., 2010. Encoding of motor skill in the corticomuscular system of musicians. Current Biology 20, 1869–1874. Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Abbeel, P., Levine, S., 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: International conference on machine learning, PMLR. p. 1861–1870. Han et al. (2021) Han, J., Chai, J., Hayashibe, M., 2021. Synergy emergence in deep reinforcement learning for full-dimensional arm manipulation. IEEE Transactions on Medical Robotics and Bionics 3, 498–509. Ho and Ermon (2016) Ho, J., Ermon, S., 2016. Generative adversarial imitation learning, in: Computers & Electrical Engineering, p. 4565–4573. doi:10.1016/j.compeleceng.2013.11.024. Kawaharazuka et al. (2025) Kawaharazuka, K., Oh, J., Yamada, J., Posner, I., Zhu, Y., 2025. Vision-language-action models for robotics: A review towards real-world applications. IEEE Access , 1–1doi:10.1109/ACCESS.2025.3609980. Kim et al. (2022) Kim, G.H., Seo, S., Lee, J., Jeon, W., Hwang, H., Yang, H., Kim, K.E., 2022. Demodice: Offline imitation learning with supplementary imperfect demonstrations, in: International Conference on Learning Representations, p. 15908–15933. Kim et al. (2024) Kim, J., Zhao, T., Schmidgall, S., Deguet, A., Kobilarov, M., Finn, C., Krieger, A., 2024. Surgical robot transformer (srt): Imitation learning for surgical tasks. arxiv 2024. arXiv preprint arXiv:2407.12998 . Kostrikov et al. (2021) Kostrikov, I., Nair, A., Levine, S., 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 . Kulkarni et al. (2026) Kulkarni, S., Dhar, R., Cui, Y., 2026. Learning from the best: Smoothness-driven metrics for data quality in imitation learning. arXiv preprint arXiv:2604.23000 . Kutsuzawa and Hayashibe (2022) Kutsuzawa, K., Hayashibe, M., 2022. Motor synergy generalization framework for new targets in multi-planar and multi-directional reaching task. Royal Society Open Science 9, 211721. Latash et al. (2014) Latash, M.L., Bernstein, N.A., Turvey, M.T., 2014. Dexterity and its development. Psychology Press. Li et al. (2024) Li, Z., Xu, T., Qin, Z., Yu, Y., Luo, Z.Q., 2024. Imitation learning from imperfection: Theoretical justifications and algorithms. Advances in Neural Information Processing Systems 36. Liu et al. (2023) Liu, J., Zu, L., He, L., Wang, D., 2023. Clue: Calibrated latent guidance for offline reinforcement learning, in: Conference on Robot Learning, PMLR. p. 906–927. Mandlekar et al. (2021) Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., Martín-Martín, R., 2021. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298 . Pan et al. (2017) Pan, Y., Cheng, C.A., Saigol, K., Lee, K., Yan, X., Theodorou, E., Boots, B., 2017. Agile autonomous driving using end-to-end deep imitation learning. arXiv preprint arXiv:1709.07174 . Prudencio et al. (2023) Prudencio, R.F., Maximo, M.R., Colombini, E.L., 2023. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems . Ross and Bagnell (2010) Ross, S., Bagnell, D., 2010. Efficient reductions for imitation learning, in: Teh, Y.W., Titterington, M. (Eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR, Chia Laguna Resort, Sardinia, Italy. p. 661–668. URL: https://proceedings.mlr.press/v9/ross10a.html. Singh et al. (2022) Singh, B., Kumar, R., Singh, V.P., 2022. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review 55, 945–990. Todorov and Jordan (2002) Todorov, E., Jordan, M.I., 2002. Optimal feedback control as a theory of motor coordination. Nature Neuroscience 5, 1226–1235. doi:10.1038/n963. Turvey (1990) Turvey, M.T., 1990. Coordination. American psychologist 45, 938. Xu et al. (2022) Xu, H., Zhan, X., Yin, H., Qin, H., 2022. Discriminator-weighted offline imitation learning from suboptimal demonstrations, in: International Conference on Machine Learning, PMLR. p. 24725–24742. Zaal et al. (1999) Zaal, F.T., Daigle, K., Gottlieb, G.L., Thelen, E., 1999. An unlearned principle for controlling natural movements. Journal of Neurophysiology 82, 255–259. Zare et al. (2024) Zare, M., Kebria, P.M., Khosravi, A., Nahavandi, S., 2024. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics 54, 7173–7186. doi:10.1109/TCYB.2024.3395626. Zhang et al. (2026) Zhang, Y., Xie, Y., Liu, H., Shah, R., Wan, M., Fan, L., Zhu, Y., 2026. Scizor: A self-supervised approach to data curation for large-scale imitation learning, in: IEEE International Conference on Robotics and Automation (ICRA). Zolna et al. (2020) Zolna, K., Novikov, A., Konyushkova, K., Gulcehre, C., Wang, Z., Aytar, Y., Denil, M., de Freitas, N., Reed, S., 2020. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011.13885 .