Paper deep dive
Integrating AIRL-Based Reward Estimation into a Control Method Using Fuzzy Inference and Policy Gradient
Seiji Ishihara
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A method integrating policy gradient with fuzzy inference has been proposed. This method, hereafter referred to as the base model, has high flexibility with respect to the configuration of fuzzy inference rules and fuzzy representation. It can also automatically adjust the rule weights and the membership functions in the antecedent and consequent parts using the same framework for learning. In methods that integrate reinforcement learning with fuzzy inference, however, the reward function must be designed in advance. It is therefore useful to add learning-based reward estimation to the base model. This paper proposes a fuzzy control method that integrates a reward estimation mechanism based on the adversarial inverse reinforcement learning (AIRL) framework into the base model. The generative adversarial nets (GAN) framework adopted in AIRL, however, tends to suffer from unstable learning. To address this, the proposed method treats the policy of the base model as the generator and also uses the set of membership functions in the policy as the cost function of the discriminator. This configuration allows the discriminator update for reward estimation and the generator update for policy estimation to be performed simultaneously on the parameters shared by both, without estimating a value function. In simulations of an illustrative vehicle speed control task, the proposed method acquired policies with high generalization performance through stable and efficient learning.
Tags
Links
- Source: https://arxiv.org/abs/2610.14723v1
- Canonical: https://arxiv.org/abs/2610.14723v1
Trouble viewing inline? Open PDF directly →
Full Text
77,999 characters extracted from source content.
Expand or collapse full text
Integrating AIRL-Based Reward Estimation into a Control Method Using Fuzzy Inference and Policy Gradient SeijiIshihara Email: ishihara_s@mail.dendai.ac.jp Affiliation: School of Science and Engineering, Tokyo Denki University, Saitama, Japan Abstract A method integrating policy gradient with fuzzy inference has been proposed. This method, hereafter referred to as the base model, has high flexibility with respect to the configuration of fuzzy inference rules and fuzzy representation. It can also automatically adjust the rule weights and the membership functions in the antecedent and consequent parts using the same framework for learning. In methods that integrate reinforcement learning with fuzzy inference, however, the reward function must be designed in advance. It is therefore useful to add learning-based reward estimation to the base model. This paper proposes a fuzzy control method that integrates a reward estimation mechanism based on the adversarial inverse reinforcement learning (AIRL) framework into the base model. The generative adversarial nets (GAN) framework adopted in AIRL, however, tends to suffer from unstable learning. To address this, the proposed method treats the policy of the base model as the generator and also uses the set of membership functions in the policy as the cost function of the discriminator. This configuration allows the discriminator update for reward estimation and the generator update for policy estimation to be performed simultaneously on the parameters shared by both, without estimating a value function. In simulations of an illustrative vehicle speed control task, the proposed method acquired policies with high generalization performance through stable and efficient learning. keywordsFuzzy control, Policy gradient, Adversarial inverse reinforcement learning, Reward estimation, Membership function Article Highlights • Fuzzy rule parameters that both decide an action and evaluate it are acquired automatically through learning. • Sharing the parameters lets both kinds of learning proceed at once, without another function that evaluates an action. • Desirable speed control was learned stably with far less training than by a conventional alternating learning method. 1 Introduction Control based on fuzzy inference has high explainability in its process because the knowledge representation is readily understandable by humans. Methods for integrating reinforcement learning into such fuzzy control systems have been studied Glorennec (1994); Horiuchi et al. (1996); Jouffe (1998); Er and Deng (2004); Wang et al. (2004); Tavakol Aghaei and Onat (2017); Imanina and Igarashi (2014); Ishihara et al. (2024). These methods automatically adjust the parameters constituting if-then rules over continuous state and action spaces, using rewards designed according to the control task. Among these is a method Imanina and Igarashi (2014); Ishihara et al. (2024), hereafter referred to as the base model, using the policy gradient method Williams (1992); Baxter and Bartlett (2001); Ishihara and Igarashi (2006) to learn the weights representing the reliability of each fuzzy inference rule (rule weights) and the parameters that specify the shape of each membership function in the antecedent and consequent parts. This method has high flexibility with respect to rule configuration and fuzzy representation. It can also use the same learning framework for the automatic adjustment of the rule weights and that of the membership functions. In methods that integrate reinforcement learning with fuzzy inference, however, the reward function must be designed in advance. Meanwhile, studies on imitation learning and inverse reinforcement learning (IRL) have been reported in recent years Arora and Doshi (2021); Zare et al. (2024). These methods acquire a policy equivalent to that of an expert, or an effective reward function, from the state-action trajectories generated by the expert’s strategy for a given task. The approach of formulating the task as an optimization problem based on the maximum entropy principle and solving it by a gradient method Ziebart et al. (2008) can be readily combined with neural networks. Among methods based on this approach, adversarial IRL (AIRL) Fu et al. (2018) is efficient in that it updates the policy and the reward alternately and partially, and in that it takes as input the state-action pair at each time step. It also has the advantage of extracting a reward function that does not include environment dynamics. The generative adversarial nets (GAN) framework Goodfellow et al. (2014) adopted in AIRL, however, tends to suffer from unstable learning. In such an integration of reinforcement learning and fuzzy inference, it is useful to add a mechanism that estimates the reward function through learning. This paper therefore proposes a fuzzy control method that integrates a reward estimation mechanism based on the AIRL framework into the base model. The proposed method does not adopt the AIRL discriminator as it stands. Instead, to make the learning of both the reward function and the policy stable and efficient, it treats the policy of the base model as the generator and also uses the set of membership functions in the policy as the internal function of the discriminator, namely its cost function11 1 Since the internal function is not explicitly named in AIRL, this paper borrows the term from GCL Finn et al. (2016b) and defines the cost function so that a larger value is more desirable.. This cost function approximates a quantity corresponding to the advantage of an action with respect to the optimal policy, using the set of membership functions rather than a value function. In the learning stage, this allows the discriminator update for reward estimation and the generator update for policy estimation to be performed simultaneously on the parameters shared by both. Since the discriminator and the generator share the parameters of the membership functions and no value function has to be estimated, learning becomes stable and efficient. The proposed method thus forms an integrated structure, in which the membership functions of the policy-gradient-based generator are also applied to the configuration of the discriminator. Simulations of an illustrative vehicle speed control task show that the proposed method achieves stable and efficient learning and acquires policies with high generalization performance. The main contributions of this paper are as follows. 1. A fuzzy control method is proposed that integrates a reward estimation mechanism based on the AIRL framework into the base model. The reward function no longer has to be designed in advance. 2. A cost function of the discriminator is defined using the set of membership functions in the policy of the generator. The parameters of the membership functions are thus shared between the generator and the discriminator. Under the optimal policy, the cost function approximates a quantity corresponding to the advantage of an action without a value function. 3. The discriminator update for reward estimation and the generator update for policy estimation are performed simultaneously on the shared parameters, and no value function has to be estimated. Learning thus becomes stable and efficient. The paper is organized as follows. Section 2 reviews related work on methods integrating reinforcement learning with fuzzy inference and on inverse reinforcement learning. Section 3 describes the base model. Section 4 outlines AIRL. Section 5 presents the configuration of the generator and the discriminator in the proposed method, together with their learning rules. Section 6 explains the control task and its application. Section 7 details the computer experiments and their results, and discusses the findings. Section 8 concludes the paper. 2 Related Work 2.1 Methods Integrating Reinforcement Learning with Fuzzy Inference These integration methods can be broadly divided into two types according to the reinforcement learning they adopt. Under the assumption of a Markov decision process (MDP) for the environment and the agent’s policy, the first type uses value-based reinforcement learning. Glorennec’s fuzzy Q-learning Glorennec (1994), which combines Q-learning with fuzzy inference, and its extensions Horiuchi et al. (1996); Jouffe (1998) belong to this type. Methods of this type generally apply fuzzy representation to the state input variables of the antecedent part, while the parameter values or functions corresponding to the Q-value of each action are given as the output of the consequent part without fuzzy representation. They also have no mechanism for adaptively strengthening or suppressing the rules themselves. One method adds rules as needed according to the matching degree of the antecedent part or the temporal difference (TD) error during reinforcement learning Er and Deng (2004), but it does not weight the rules directly. The second type uses policy-based reinforcement learning, specifically the policy gradient method Williams (1992); Baxter and Bartlett (2001); Ishihara and Igarashi (2006), which learns the policy directly by a gradient method without using a value function. As in the first type, in one method Wang et al. (2004) that uses GPOMDP Baxter and Bartlett (2001), each rule consists of an antecedent part with fuzzy representation and a consequent part that outputs the parameters corresponding to the policy without fuzzy representation. Another method Tavakol Aghaei and Onat (2017) has a consequent part that outputs the matching degree of a fuzzy representation defined by the parameters corresponding to the policy. This method integrates policy gradient methods such as GPOMDP with fuzzy inference. In these methods, however, the membership functions are fixed and the rule weights are not taken into account. The base model Imanina and Igarashi (2014); Ishihara et al. (2024), by contrast, has high flexibility with respect to the configuration of fuzzy inference rules: it supports fuzzy representation not only in the antecedent part but also in the consequent part, and it allows rule weights corresponding to the reliability of each rule to be set. It can also be applied as a general-purpose method that uses the same framework for learning to automatically acquire the reliability of each rule and the concept of fuzzy representation based on the shape of each membership function Ishihara et al. (2024). 2.2 Inverse Reinforcement Learning Inverse reinforcement learning is a type of imitation learning that automatically acquires an expert’s policy from the expert’s trajectories, and it differs from other methods of imitation learning in that it can estimate the reward function explicitly. Multiple types of IRL methods have been studied Arora and Doshi (2021). Among them, methods combining a gradient method with maximum entropy IRL (ME-IRL) Ziebart et al. (2008), a later approach, have been developed. Finn et al. proposed guided cost learning (GCL) Finn et al. (2016b), which alternately updates the policy and the cost function and estimates the partition function through sampling. GCL extends the applicability of ME-IRL and improves efficiency. They subsequently showed that GCL can be reformulated within the GAN framework (GAN-GCL) Finn et al. (2016a). Fu et al. proposed AIRL Fu et al. (2018), which takes as input the state-action pair at each time step rather than an entire trajectory and thereby learns more efficiently. By incorporating a value function into the discriminator, AIRL also estimates a reward function that does not include environment dynamics. In AIRL, the cost function of the discriminator has been shown to be equivalent to the advantage function under the optimal policy. The GAN framework adopted in these methods, however, tends to suffer from unstable learning. Approaches to stabilizing the training of GANs have therefore been explored Goodfellow (2015); Salimans et al. (2016); Karras et al. (2018). For adversarial learning in imitation learning and IRL, a method has also been proposed that constrains the information flow in the discriminator by an information bottleneck to maintain informative gradients Peng et al. (2019). In the context of fuzzy inference, no method has been reported that uses membership functions as the cost function to make learning in the AIRL framework stable and efficient. 3 The Base Model 3.1 Fuzzy Inference Rules The following fuzzy inference rules are used to control the agent’s behavior. Rule i: i: if (x1 is A1i) and … and (xM is AMi) (x_1 is A^i_1) and … and (x_M is A^i_M) then (y1 is B1i) and … and (yN is BNi) (y_1 is B^i_1) and … and (y_N is B^i_N) with ηi _i (1) Here, i (=1,2,⋯,nR=1,2,·s,n_ R) is the rule number, xjx_j (j=1,2,⋯,Mj=1,2,·s,M) is an input variable representing the state, and yky_k (k=1,2,⋯,Nk=1,2,·s,N) is an output variable representing the action. AjiA^i_j and BkiB^i_k are linguistic fuzzy expressions for the variables xjx_j and yky_k in the j-th antecedent part and the k-th consequent part of rule i, respectively. Aji(xj,αji)A^i_j(x_j;α^i_j) and Bki(yk,βki)B^i_k(y_k;β^i_k) are their membership functions, specified by the parameters αjiα^i_j and βkiβ^i_k. ηi _i (≥0≥ 0) is a parameter representing the rule weight. 3.2 Objective Function and Policy The agent decides its action at each time step so as to minimize the objective function E(y;x)=−∑i=1nRηiAi(x;αi)Bi(y;βi)E(y;x)=- _i=1^n_ R _iA^i(x;α^i)B^i(y;β^i) (2) Here, the matching degrees of the input value x in the antecedent part and of the output value y in the consequent part of rule i are defined as Ai(x,αi)≡∏j=1MAji(xj,αji)A^i(x;α^i)≡ _j=1^MA^i_j(x_j;α^i_j) and Bi(y,βi)≡∏k=1NBki(yk,βki)B^i(y;β^i)≡ _k=1^NB^i_k(y_k;β^i_k), respectively. The objective function in Eq. (2) represents the total degree of support that the rules give to the output value y for the input value x. Deciding the output value y(t)y(t) for the input value x(t)x(t) at time t is the policy of the agent, or learning system. The policy is defined as a probability given by the Boltzmann distribution with temperature T, in which E(y(t),x(t))E(y(t);x(t)) is used as the energy function. π(y(t)∣x(t))≡exp(−E(y(t);x(t))/T)∑yexp(−E(y;x(t))/T)π(y(t)\! \!x(t))≡ (-E(y(t);x(t))/T) _y (-E(y;x(t))/T) (3) The policy π is thus stochastic, and T represents the degree of randomness in deciding the action. 3.3 Learning Rule for the Parameters by the Policy Gradient Method The same reinforcement learning framework applies to all the parameters μ∈α,β,ημ∈\α,β,η\. The policy gradient method gives the following learning rule, which maximizes the expected reward τ∼π[r(τ)]E_τ π[r(τ)] for a trajectory τ=(x(0),y(0),⋯,x(L−1),y(L−1))τ=(x(0),y(0),·s,x(L-1),y(L-1)) in an episode. Δμ=ετ∼π[r(τ)∑t=0L−1eμ(t)] μ= _τ π [r(τ) _t=0^L-1e_μ(t) ] (4) Here, L is the episode length and ε (>0>0) is the learning coefficient. The characteristic eligibility Williams (1992) eμ(t)e_μ(t) is given by the following expressions Ishihara and Igarashi (2006). eμ(t)≡∂μlnπ(y(t)∣x(t)) e_μ(t)≡ ∂μ π (y(t)\! \!x(t) ) (5) =−1T∂E(y(t),x(t))∂μ−π(y∣x(t))[∂E(y,x(t))∂μ] =- 1T\! \\! ∂ E(y(t);x(t))∂μ-E_π(y x(t))\!\! [\! ∂ E(y;x(t))∂μ\! ]\! \ (6) Here, π(y∣x)[f(y)]≡∑yf(y)π(y∣x)E_π(y x)[f(y)]≡ _yf(y)\ π(y\! \!x). 4 Adversarial Inverse Reinforcement Learning AIRL is an IRL method that adopts the GAN framework, which consists of a discriminator and a generator. To obtain the optimal policy, AIRL alternately updates the discriminator and the policy in the generator. Specifically, with the expert trajectories prepared in advance, the process repeats a cycle consisting of generating sample trajectories with the policy of the generator, updating the discriminator, estimating the reward from the updated discriminator, and updating the policy in the generator. 4.1 Discriminator Let x′x denote the state one step ahead, reached from state x by action y. The output of the discriminator defined using the cost function fθ,ϕ(x,y,x′)f_θ,φ(x,y,x ) is given by D(x,y,x′)=exp(fθ,ϕ(x,y,x′))exp(fθ,ϕ(x,y,x′))+π(y∣x)D(x,y,x )= (f_θ,φ(x,y,x )) (f_θ,φ(x,y,x ))+π(y\! \!x) (7) The cost function is specified by the parameters θ and ϕφ and is decomposed as follows: fθ,ϕ(x,y,x′)=g(x,y,θ)+κh(x′,ϕ)−h(x,ϕ)f_θ,φ(x,y,x )=g(x,y;θ)+κ h(x ;φ)-h(x;φ) (8) Here, g is an approximation of the reward function, h is an approximation of the state value function, and κ is the discount rate. Under the optimal policy, fθ,ϕ(x,y,x′)f_θ,φ(x,y,x ) has been shown to be equivalent to the advantage function. The value of D(x,y,x′)D(x,y,x ) corresponds to the probability that the input (x,y,x′)(x,y,x ) is contained in the expert trajectory τD _ D, the demonstration. It is also the probability that the input is not contained in the trajectory τS _ S , the sample, generated by the actions that the generator outputs following the policy π(y∣x)π(y\! \!x). Let tD_t denote the set of element triples (x(t),y(t),x(t+1))(x(t),y(t),x(t+1)) based at time t in the demonstration τD _ D. For correct discrimination between the demonstration and the sample, the parameters of the discriminator are learned so as to minimize the following cross-entropy loss function. ℒ(θ,ϕ)=∑t=0L−1 (θ,φ)= _t=0^L-1 −t[ln(D(x,y,x′))] \-E_D_t\! [ (D(x,y,x )) ] . −πt[ln(1−D(x,y,x′))] \ .-E_ _t\! [ (1-D(x,y,x )) ] \ (9) Here, πt _t denotes the distribution of (x(t),y(t),x(t+1))(x(t),y(t),x(t+1)) generated by rolling out the policy π. The gradient-based learning rule for the discriminator is given by Δλ=δ∑t=0L−1 λ=δ\! _t=0^L-1 t[(1−D(x,y,x′))∂λfθ,ϕ(x,y,x′)] \! \\!E_D_t\!\! [\!(1-D(x,y,x )) ∂λf_θ,φ(x,y,x )\! ] . −πt[D(x,y,x′)∂λfθ,ϕ(x,y,x′)] \ .-E_ _t\!\! [D(x,y,x ) ∂λf_θ,φ(x,y,x ) ]\! \ (10) Here, λ∈θ,ϕλ∈\θ,φ\ and δ is the learning coefficient. 4.2 Generator The output y of the generator is the action decided following the policy π(y∣x)π(y\! \!x). The reward r^(x,y,x′) r(x,y,x ) is estimated from the output of the discriminator as follows: r^(x,y,x′)=ln(D(x,y,x′))−ln(1−D(x,y,x′)) r(x,y,x )= (D(x,y,x ))- (1-D(x,y,x )) (11) The policy π(y∣x)π(y\! \!x) of the generator is learned with any reinforcement learning method using this estimated reward. Figure 1: Configuration of the proposed method 5 Proposed Method The proposed method integrates the reward estimation mechanism of AIRL presented in Section 4 into the base model presented in Section 3 by sharing the membership functions. Fig. 1 shows the configuration of the proposed method. The proposed method treats the policy of the base model as the generator and also uses the set of membership functions (MF) in the policy as the cost function of the discriminator. Specifically, a cost function is defined using each membership function of the antecedent part, and this gives a discriminator for each fuzzy representation of each rule. From the viewpoint of the j-th fuzzy representation in the antecedent part of rule i, the j-th (=1,2,⋯,M=1,2,·s,M) discriminator of rule i thus gives a score indicating how likely a trajectory of actions and states is to be contained in the demonstration. The outputs of the discriminators are used for two updates, one minimizing the loss function of the discriminator (Update D) and the other maximizing the expected reward of the generator (Update G), which are performed simultaneously on the parameters of the membership functions of the antecedent part shared by the discriminator and the generator. The following subsections present the proposed cost function and discriminator, together with the learning rules for the discriminator and the generator of the proposed method. 5.1 Cost Function and Discriminator Sharing the Membership Functions This paper proposes the following cost function defined using each membership function in the antecedent part. fαji(x(t),y(t),x(t+1)) f_α^i_j(x(t),y(t),x(t+1)) =ln1−|vαji(x(t+1),x(t),y(t))| = \1- |v_α^i_j(x(t+1);x(t),y(t)) | \ (12) where vαji(x(t+1),x(t),y(t)) v_α^i_j(x(t+1);x(t),y(t)) ≡t[Aji(xj′,αji)]−Aji(xj(t+1),αji) _D_t [A^i_j(x _j;α^i_j) ]-A^i_j(x_j(t+1);α^i_j) (13) Under the assumption that 0<Aji(xj)<10<A^i_j(x_j)<1, it follows that 0≤|vαji|<10≤|v_α^i_j|<1 and fαji≤0f_α^i_j≤ 0. Eqs. (12) and (13) show that the value of fαjif_α^i_j approaches zero as the matching degree Aji(xj(t+1))A^i_j(x_j(t+1)) for the input approaches the expected matching degree t[Aji(xj′)]E_D_t[A^i_j(x _j)] for the demonstration. The cost function fαjif_α^i_j therefore evaluates the value of (x(t),y(t))(x(t),y(t)) by the matching degree for xj(t+1)x_j(t+1). Under the optimal policy, πt[Aji(xj′)]E_ _t[A^i_j(x _j)] equals t[Aji(xj′)]E_D_t[A^i_j(x _j)]. In this case, the difference fαji(x(t),y(t),x(t+1))−πt[fαji(x,y,x′)]f_α^i_j(x(t),y(t),x(t+1))-E_ _t[f_α^i_j(x,y,x )] between the value of a given (x(t),y(t))(x(t),y(t)) and the expected value over all samples generated by π represents the relative value of the action y(t)y(t) with respect to the optimal policy π. This difference can therefore be regarded as a form of approximation of the advantage without using a value function. Using the cost function in Eq. (12), the output of the j-th discriminator of rule i is defined as Dji(x(t),y(t),x(t+1)) D^i_j(x(t),y(t),x(t+1)) =exp(fαji(x(t),y(t),x(t+1)))exp(fαji(x(t),y(t),x(t+1)))+π(y(t)∣x(t)) =\! (f_\!α^i_j(x(t),y(t),x(t\!+\!1))) (f_\!α^i_j\!(x(t),y(t),x(t\!+\!1)\!)\!)\!+\!π(y(t)\! \!x(t)\!) (14) 5.2 Learning Rule for the Discriminator The loss function of the discriminator in the proposed method is ℒ(αji)=∑t=0L−1 (α^i_j)= _t=0^L-1 −t[ln(Dji(x,y,x′))] \-E_D_t\! [ (D^i_j(x,y,x )) ] . −πt[ln(1−Dji(x,y,x′))] \ .-E_ _t\! [ (1-D^i_j(x,y,x )) ] \ (15) In updating the discriminator, the parameters of the policy π of the generator at time t are fixed, and π(y(t)∣x(t))π(y(t)\! \!x(t)) in Eq. (14) is treated as a constant. The learning rule that minimizes this loss function is then given by Δαij=δ∑t=0L−1 α^i_j\!=δ\! _t=0^L-1 t[(−Dji(x,y,x′))∂αjifαji(x,y,x′)] \!\! \\!E_D_t\!\! [\!(1\!-\!D^i_j(x,y,x )\!) ∂α^i_jf_\!α^i_j\!(x,y,x )\! ] . −πt[Dji(x,y,x′)∂αjifαji(x,y,x′)] \ .-E_ _t\!\! [\!D^i_j(x,y,x ) ∂α^i_jf_α^i_j(x,y,x )\! ]\! \ (16) The gradient used for this minimization is thus only that of the cost function fαjif_α^i_j. Even though the policy π of the generator shares the parameters with the discriminator, it is treated as a constant in the learning rule for the discriminator. The derivation of Eq. (16) is given in Appendix 9. 5.3 Learning Rule for the Generator The evaluation function of the generator in the proposed method, which the policy gradient method maximizes, is defined as J(μ)=τ∼π[∑t=0L−1γtr^(x(t),y(t),x(t+1))]J(μ)=E_τ π\! [ _t=0^L-1γ^t r(x(t),y(t),x(t+1)) ] (17) where r^(x,y,x′)=r^ji(x,y,x′)ifμ=αji∑j=1Mr^ji(x,y,x′)ifμ∈βki,ηi r(x,y,x )= \ array[]l\! r^i_j(x,y,x )& if\ \ μ=α^i_j\\[5.69054pt] \! _j=1^M r^i_j(x,y,x )& if\ \ μ∈\β^i_k, _i\ array . r^ji(x,y,x′)=ln(Dji(x,y,x′))−ln(−Dji(x,y,x′)) r^i_j(x,y,x )= (D^i_j(x,y,x ))- (1\!-\!D^i_j(x,y,x )) (20) Here, γ is the discount rate. In updating the generator, the parameters of the discriminators DjiD^i_j at time t are fixed, and r^(x(t),y(t),x(t+1)) r(x(t),y(t),x(t+1)) in Eq. (17) is treated as a constant. The learning rule that maximizes this evaluation function is then given by Δμ μ =ετ∼π[∑t=0L−1∑u=qL−1γu−qr^(x(u),y(u),x(u+))eμ(t)] =\! _τ π\!\! [ _t=0^L-1\!\! \\! _u=q^L-1\!γ^u-q r(x(\!u\!),y(\!u\!),x(\!u+1\!)\!)\!\! \\!e_μ(t)\! ] (21) Here, q=0q=0 corresponds to Eq. (4) and q=tq=t corresponds to the REINFORCE learning rule. The gradient used for this maximization is thus only the characteristic eligibility eμe_μ. Even though the reward r r is given by the outputs of the discriminators DjiD^i_j, which share the parameters with the generator, it is treated as a constant in the learning rule for the generator. With the learning rule in Eq. (21), the parameters of the membership functions in the consequent part and of the rule weights can also be learned, in addition to those in the antecedent part. In the following discussion, however, only the parameters αjiα^i_j of the membership functions in the antecedent part shared with the discriminator are considered as the learning target in the generator. The characteristic eligibility eαji(t)e_α^i_j(t) for the update Δαji α^i_j is given by eαji(t) e_α^i_j(t) =−1Tηiπ(y∣x(t))[Bi(y)]−Bi(y(t))∂αjiAi(x(t)) =\!- 1T _i\! \E_π(y x(t)\!)\!\! [B^i(y) ]\!-\!B^i(y(t))\! \\! ∂α^i_jA^i(x(t)) (22) The derivation of Eq. (22) is given in Appendix 10. 5.4 Simultaneous Learning In the proposed method, the discriminator and the generator share the parameters, and αjiα^i_j is updated by Eqs. (16) and (21) simultaneously. A single update with one set of samples therefore covers both the discriminator and the reward estimation for the generator, just as one alternating cycle does in AIRL. In this case, the amount of a single update of αjiα^i_j is given by Δαji=δ∑t=0L−1t[(−Dji(x,y,x′))∂αjifαji(x,y,x′)] α^i_j\!=\!δ\! _t=0^L-1\!\! \\!E_D_t\!\! [\!(1\!-\!D^i_j(x,y,x )\!) ∂α^i_jf_\!α^i_j(x,y,x )\! ] . −πt[Dji(x,y,x′)∂αjifαji(x,y,x′)] .-E_ _t\!\! [\!D^i_j(x,y,x ) ∂α^i_jf_α^i_j(x,y,x )\! ]\! \ +ετ∼π[∑t=0L−1∑u=qL−1γu−qr^(x(u),y(u),x(u+))eαji(t)] +\! _τ π\!\! [ _t=0^L-1\!\! \\! _u=q^L-1\!γ^u-q r(x(\!u\!),y(\!u\!),x(\!u+1\!)\!)\!\! \\!e_α^i_j\!( -0.50003ptt -0.50003pt)\! ] (23) In Eq. (23), the first term on the right-hand side corresponds to an update that increases the difference in the output of the membership function AjiA^i_j between the demonstration and the sample, while the second term corresponds to an update that decreases it. If the update of the generator can bring the shapes of the membership functions close to a constant function, the output of the cost function easily becomes large regardless of the state of the sample, and a policy that generates trajectories similar to the demonstration does not necessarily have to be learned in the generator. Because the first term keeps the membership function away from such a constant-like shape, the second term can update the membership function so that the policy generates samples close to the states of the demonstration. 6 Control Task and Its Application This section describes a simulation problem Ishihara et al. (2024) that assumes an illustrative vehicle speed control task, and the application of the proposed method to this problem. 6.1 Vehicle Speed Control Problem Two vehicles are assumed to travel in the same direction along a one-dimensional straight line representing a road, with the speed of the leading vehicle vfrontv_ front (km/h) held constant. The task of this problem is to control the speed of the following vehicle xvx_ v (km/h) by accelerating and braking, so that the inter-vehicle distance xdx_ d (m) enters the target range [lnear,lfar][l_ near,l_ far] (m) within 80 seconds from the initial state at t=0t=0 (sec) and then remains within that range until t=110t=110 (sec). With the time discretized at Δt=1 t=1 (sec), the speed control of the following vehicle starts at t=0t=0 and ends when t=110t=110 is reached. The length of one episode is therefore L=110L=110. However, if the inter-vehicle distance becomes less than 0 m, which means a rear-end collision, or reaches 400 m or more, the control is terminated at that point and the episode is regarded as ended. 6.2 Fuzzy Inference Rules The fuzzy inference rules for controlling the speed of the following vehicle are given by Eq. (1) with M=2M=2 and N=1N=1. The input variables in the antecedent part are 1=(xd,lnear,lfar) x_1=(x_ d,l_ near,l_ far) and 2=(xv,vfront) x_2=(x_ v,v_ front). The output variable y1y_1 in the consequent part corresponds to the accelerator input (0≤y1≤50≤ y_1≤ 5) and the brake input (−5≤y1≤0-5≤ y_1≤ 0), and the resulting acceleration of the following vehicle is 2y12y_1 (km/h/sec). The fuzzy sets A1longA_1 long and A1shortA_1 short correspond to fuzzy representations of the inter-vehicle distance, long and short, respectively. The fuzzy sets A2fastA_2 fast and A2slowA_2 slow correspond to fuzzy representations of the speed of the following vehicle, fast and slow, respectively. The fuzzy sets B1high-ac.B_1 high-ac., B1low-ac.B_1 low-ac., B1high-br.B_1 high-br., B1low-br.B_1 low-br., and B1no-op.B_1 no-op. correspond to fuzzy representations of high accelerator input (ac.), low accelerator input, high brake input (br.), low brake input, and no operation (no-op.), respectively. With these combinations (2×2×5=202× 2× 5=20), there are 20 fuzzy inference rules and rule weights ηi _i for i=1i=1 to i=20i=20. 6.3 Membership Functions and Parameters The membership functions for the nine fuzzy sets are defined as A1i(1,α1i)≡A1long(1)ifi=1,…,10A1short(1)ifi=11,…,20 A^i_1( x_1;α^i_1)\!≡\! \ array[]l\!\!A long_1( x_1)& if\ \ i=1,...,10\\[5.69054pt] \!\!A short_1( x_1)& if\ \ i=11,...,20 array . A2i(2,α2i)≡A2fast(2)ifi=1,…,5,11,…,15A2slow(2)ifi=6,…,10,16,…,20 A^i_2( x_2;α^i_2)\!≡\! \ array[]l\!\!A fast_2( x_2)& if\ \ i=1,...,5,11,...,15\\[5.69054pt] \!\!A slow_2( x_2)& if\ \ i=6,...,10,16,...,20\\ array . B1i(y1,β1i)≡B1high-ac.(y1)ifi=1,6,11,16B1low-ac.(y1)ifi=2,7,12,17B1high-br.(y1)ifi=3,8,13,18B1low-br.(y1)ifi=4,9,14,19B1no-op.(y1)ifi=5,10,15,20 B_1^i(y_1;β^i_1)\!≡\! \ array[]l\!\!B_1 high-ac.(y_1)& if\ \ i=1,6,11,16\\[5.69054pt] \!\!B_1 low-ac.(y_1)& if\ \ i=2,7,12,17\\[5.69054pt] \!\!B_1 high-br.(y_1)& if\ \ i=3,8,13,18\\[5.69054pt] \!\!B_1 low-br.(y_1)& if\ \ i=4,9,14,19\\[5.69054pt] \!\!B_1 no-op.(y_1)& if\ \ i=5,10,15,20\\ array . A common membership function is thus used for the same fuzzy set even in different rules. For example, the membership functions for the fuzzy sets A11A_1^1 through A110A_1^10 are all A1long(1)A long_1( x_1), which corresponds to the fuzzy representation “long.” 6.4 Deciding the Action In the training phase, the output value for the action at time t is selected probabilistically from the discrete values of y1y_1 according to the policy π(y1∣(t))π(y_1\! \! x(t)). In the evaluation phase, the output value is decided deterministically as the expected value ∑y1y1π(y1∣(t)) _y_1y_1π(y_1\! \! x(t)) over the discrete values of y1y_1. Figure 2: Configuration of AIRL-BMP 7 Computer Experiments In the computer experiments, the parameters were learned by applying the proposed method to the control task described in Section 6, and the results were evaluated. For comparison, experiments were also conducted with AIRL-BMP (AIRL with Base-Model Policy), an IRL method that uses the discriminator of AIRL described in Section 4 as it stands and takes the policy of the base model as the generator. Fig. 2 shows the configuration of AIRL-BMP. In AIRL-BMP, the discriminator and the generator share no parameters, and they are updated alternately rather than simultaneously. The learning rule for the discriminator in AIRL-BMP is Eq. (10), and that for the generator is given by substituting Eq. (11) into r r in Eq. (21), the learning rule of the proposed method. In these experiments, each membership function in the antecedent part, namely A1long(1)A long_1( x_1), A1short(1)A short_1( x_1), A2fast(2)A fast_2( x_2), and A2slow(2)A slow_2( x_2), was represented by a neural network. Among the parameters α,β,η\α,β,η\ of the policy, the weight parameters αji∈α1long,α1short,α2fast,α2slowα^i_j∈\α long_1,α short_1,α fast_2,α slow_2\ of these neural networks were the learning target. The membership functions B1i(y1)B_1^i(y_1) in the consequent part were fixed to those adopted in the previous experiments on the base model Ishihara et al. (2024). The definitions of B1i(y1)B_1^i(y_1) and the discretization of y1y_1 are given in Appendix 12. The rule weights ηi _i were fixed as ηi=1ifi=5,6,13,200otherwise _i= \ array[]l\!\!1& if\ \ i=5,6,13,20\\ \!\!0& otherwise array . In Eq. (7), for each of the four combinations of antecedent fuzzy representations, a weight of 1 was assigned to the fuzzy inference rule whose consequent fuzzy representation was considered the most appropriate, and 0 to the others. For both methods, the temperature T in Eq. (3) for the policy was set to 0.020.02. In Eq. (21) for the learning rule of the generator, q and γ were set to 00 and 11, respectively. For the proposed method, the learning coefficients δ in Eq. (16) and ε in Eq. (21) were set to 0.000020.00002 and 0.0000010.000001, respectively. For AIRL-BMP, the learning coefficients δ in Eq. (10) and ε in Eq. (21) were set to 0.00020.0002 and 0.000030.00003, respectively. The discount rate κ in Eq. (8) for the cost function of AIRL-BMP was set to 0.90.9. These values were chosen empirically based on the results of preliminary experiments. 7.1 Configuration of the Neural Networks 7.1.1 Membership Functions in the Antecedent Part Each of the neural networks representing the four membership functions in the antecedent part consisted of an input layer with three or two units, a hidden layer with 120 units, and an output layer with one unit that outputs the matching degree. The activation functions were PReLU in the hidden layer and the sigmoid function in the output layer. The initial negative slope of PReLU was 0.250.25. The weight parameters corresponding to the biases were initialized with a uniform distribution over [0,0.01][0,0.01], and the other weight parameters were initialized with a normal distribution of standard deviation 1/n1/ n, where n is the number of input units of each layer. The input values to the neural networks were scaled to one hundredth of the original values, namely (xv,vfront)/100(x_ v,v_ front)/100 or (xd,lnear,lfar)/100(x_ d,l_ near,l_ far)/100. 7.1.2 Discriminator of AIRL-BMP In the discriminator of AIRL-BMP, the input of the function g(1,2,y1,θ)g( x_1, x_2,y_1;θ) was six-dimensional and that of the function h(1,2,ϕ)h( x_1, x_2;φ) was five-dimensional. Each of these functions was represented by a neural network consisting of an input layer with six or five units, a hidden layer with 120 units, and an output layer with one unit. The activation functions were PReLU in the hidden layer and the identity function in the output layer. The initial negative slope of PReLU, the initial values of the weight parameters, and the scaling of 1 x_1 and 2 x_2 were the same as in the neural networks representing the membership functions in the antecedent part. In addition, y1y_1 was scaled to one fifth of the original value when it was input to the network. threeparttable Table 1: Control results for the 580 training problems (task success rate, number of successful tasks, and breakdown of failed tasks) over the 20 sets of learned parameters Avg. task Successful tasks Avg. no. of failed tasks success rate min. max. avg. SD collision out of range delay Proposed 0.981 565 572 569.25 2.34 10.50 0.00 0.25 AIRL-BMP 0.721 285 539 418.45 74.59 19.45 44.15 97.95 Base model 0.983 519 580 570.05 12.16 6.90 3.05 tablenotes Values quoted from Ishihara et al. (2024) Combined count for “out of range” and “delay” failures, which are not distinguished in the base model Table 2: Number of training epochs over the 20 training runs Training epochs min. max. avg. SD Proposed 35 78 48.80 10.51 AIRL-BMP 327 1350 746.65 257.83 7.2 Experimental Procedure in the Training Phase 7.2.1 Training Problems A total of 5×5×5×5=6255× 5× 5× 5=625 speed control problems were prepared by exhaustively combining the speed of the leading vehicle (km/h) from 10,30,50,70,90\10,30,50,70,90\, the initial speed of the following vehicle (km/h) from 0,20,40,60,80\0,20,40,60,80\, the initial inter-vehicle distance (m) from 10,30,50,70,90\10,30,50,70,90\, and the target range of the inter-vehicle distance (m) from [10,15],[30,35],[50,55],[70,75],[90,95]\[10,15],[30,35], [50,55],[70,75],[90,95]\. Problems in which a rear-end collision was theoretically unavoidable or hard to prevent were then excluded: those in which the initial speed of the following vehicle exceeded the speed of the leading vehicle by 50 km/h or more and the initial inter-vehicle distance was 30 m or less, and those in which the initial speed of the following vehicle exceeded the speed of the leading vehicle by 70 km/h or more. The remaining 580 problems were used for training. For each of these 580 problems, one successful episode of the demonstration τD _ D was created by manual control and used as the expert trajectory. 7.2.2 Training Procedure After the parameters to be learned were initialized, the following procedure was used. Step 1. Generate a trajectory sample τS _ S for a training problem from the output of the generator, using the action decision in the training phase. Step 2. For each training problem, repeat the sample generation of Step 1 and the update below in sequence. Proceed to Step 3 when the parameters have been updated once for each training problem. Proposed method: Update the parameters αjiα^i_j shared by the discriminator and the generator using the learning rule in Eq. (23). AIRL-BMP: Update the parameters θ,ϕ\θ,φ\ of the discriminator using the learning rule in Eq. (10). Then update the parameters αjiα^i_j of the generator using the learning rule in Eq. (21), into which the reward in Eq. (11) from the updated discriminator is substituted. Step 3. Generate a trajectory sample τS _ S for a training problem from the output of the generator, using the action decision in the evaluation phase, then check and record whether the task succeeds. Proceed to Step 4 when this has been done for all training problems. Step 4. If the task succeeds for all training problems, adopt the current parameters as the result and end the training. If any problem fails, return to Step 1. However, the number of epochs, each consisting of Steps 1 to 4, is limited to 500 for the proposed method and 2500 for AIRL-BMP. When this maximum is reached, adopt as the result the parameters that gave the largest number of successful problems, and end the training. If more than one set of parameters qualifies, adopt the one obtained at the smallest number of epochs. For both methods, stochastic gradient descent (SGD) was used as the optimizer. The update equations derived from each learning rule are given in Appendix 11. The procedure described above, from the initialization of the parameters to the end of the training, is counted as one training run. Each training run therefore gives one set of learned parameters. In this study, 20 training runs were conducted, giving 20 sets of learned parameters. 7.3 Experimental Results and Discussion in the Training Phase The control results for the training problems with the weight parameters from each training run are shown in Table 1, and the number of epochs for each run is shown in Table 2. For comparison, Table 1 also includes the control results reported in Ishihara et al. (2024) for the same training problems with the base model, which uses a reward function set manually in advance. Table 1 shows that the proposed method achieved an average task success rate of 98.1%98.1\%, close to the 98.3%98.3\% of the base model, with small variation across the 20 sets of results. The average task success rate of AIRL-BMP, in contrast, was 72.1%72.1\%, about 2626 points lower than that of the proposed method. Moreover, AIRL-BMP gave a smaller minimum and a smaller maximum number of successfully controlled problems than the proposed method, and its variation across the sets was also greater, with a standard deviation about 32 times as large. Table 2 shows that the number of epochs required for training by the proposed method was about one fifteenth of that for AIRL-BMP, and that its variation across training runs also tended to be small, with a standard deviation about one twenty-fifth as large. A Mann-Whitney U test between the proposed method and AIRL-BMP showed that the proposed method achieved a significantly larger number of successful tasks (median 570.00570.00, IQR [567.75,571.25][567.75,571.25] versus median 407.50407.50, IQR [370.50,478.75][370.50,478.75]; U=400U=400, p<0.001p<0.001, r=0.855r=0.855). It also required significantly fewer epochs (median 47.0047.00, IQR [40.75,55.00][40.75,55.00] versus median 662.50662.50, IQR [584.25,887.25][584.25,887.25]; U=0U=0, p<0.001p<0.001, r=0.855r=0.855), with n=20n=20 for each method22 2 The statistical test is applied only to the results obtained in the experiments of this paper.. Task failures can be classified into three types: collision, being out of the target range at t=110t=110 (out of range), and remaining within the target range until t=110t=110 with the start of that period later than t=80t=80 (delay). The breakdown of the average number of failed tasks in Table 1 shows that the proposed method had no cases of “out of range” and only 0.25 cases of “delay,” and the base model also had only 3.05 cases of “out of range” and “delay” combined. AIRL-BMP, in contrast, had a large total average number of failed tasks, with “delay” the most frequent at 97.95 cases, followed by “out of range” at 44.15 cases. These results suggest that during training AIRL-BMP could not improve many trajectories of “delay” and “out of range,” which are relatively close to those of successful tasks, whereas the proposed method was able to learn to succeed in these tasks. Most of the 10.50 cases of “collision” in the proposed method occurred in relatively demanding problem settings in which the initial speed of the following vehicle was 50 km/h higher than the speed of the leading vehicle, as was also the case for “collision” in the base model. Figure 3: Average reward per time step for all fuzzy representations at each training epoch. a Proposed method, b AIRL-BMP threeparttable Table 3: Trajectory characteristics for the successful training problems Time to settle Settling delay Exits from range SD after settling avg. SD avg. SD avg. SD pooled value Proposed 22.83 1.84 5.71 1.28 0.48 0.13 0.28 AIRL-BMP 47.91 6.65 29.51 10.22 1.35 0.41 0.96 Demonstration 14.60 — 2.68 — 0.21 — 0.35 Figure 4: Trajectories of the inter-vehicle distance with the proposed method, AIRL-BMP, and the demonstration. a Problem T1, b Problem T2 The average reward per time step for all fuzzy representations at each training epoch is shown in Fig. 3. The value at each epoch is averaged over all the training problems in all the training runs, and the error bars show the minimum and maximum of the averages over the 580 problems in each training run. Fig. 3 shows that the average reward for the proposed method decreased almost monotonically toward zero, the theoretical value of the reward under the optimal policy, within a small number of epochs. This suggests that the proposed method efficiently learned parameters achieving a high average task success rate. The average reward for AIRL-BMP, in contrast, started from a negative value, dropped sharply, and then rose sharply once. It never reached zero in any of the training runs, however, and thereafter decreased almost monotonically. When the trajectories of the samples from the generator differ from those of the demonstration, with large deviations in the inter-vehicle distance or the speed, the output of the discriminator generally becomes small and the reward tends to take a large negative value. Since the average reward for AIRL-BMP never became zero or higher throughout training with five times as many epochs as for the proposed method, AIRL-BMP was less effective than the proposed method in learning to bring the trajectories of the samples for each problem closer to those of the demonstration on average. As successful examples among the control results for the training problems, the trajectories of the inter-vehicle distance after training with the proposed method and with AIRL-BMP are shown in Fig. 4, together with the trajectory of the demonstration, for problem T1, which required acceleration from the initial state, and problem T2, which required deceleration. In problems T1 and T2, (lnear,lfar,vfront)(l_ near,l_ far,v_ front) were (50,55,90)(50,55,90) and (50,55,10)(50,55,10), and the initial values of (xd,xv)(x_ d,x_ v) were (70,60)(70,60) and (50,40)(50,40), respectively. For both methods, the trajectories in Fig. 4 were generated by the policies using the parameters from the training run that gave the largest number of successful problems. Fig. 4 shows that the proposed method, as in the demonstration, tended to settle the inter-vehicle distance within the target range earlier than AIRL-BMP and to give trajectories with smaller oscillations. The trajectories from AIRL-BMP, in contrast, tended to settle within the target range later and to require more time for the oscillations to converge. Table 3 quantifies these characteristics for the successful training problems. The time to settle is the start of the period in which the inter-vehicle distance remained within the target range until t=110t=110, the settling delay is the interval between the first time the distance was within the range and the time to settle, the exits from range is the number of times the distance moved from inside to outside the target range, and the SD after settling is the standard deviation of the distance from the time to settle to t=110t=110. For the first three measures, the values were averaged over the successful problems for each set of learned parameters, and Table 3 shows the mean and the standard deviation of these averages over the 20 sets. The last measure is pooled over all the successful problems in all the sets. Table 3 also includes the values for the demonstration for reference. Since only one demonstration was created for each problem, the values were averaged over the 580 problems and no standard deviation over sets is available. A Mann-Whitney U test between the proposed method and AIRL-BMP showed that the differences in all four measures were significant (p<0.001p<0.001, n=20n=20 for each method), where the pooled standard deviation for each set was used for the last measure. Continuous oscillations of the trajectory do not occur in the demonstration and represent unnatural control. They not only increase the possibility of task failure but can also reduce usefulness even when the task succeeds, and they are generally undesirable. For the proposed method, in contrast, such oscillations requiring time to converge hardly occurred. These results indicate that the proposed method was less prone to continuous oscillations of the trajectory than AIRL-BMP. threeparttable Table 4: Control results for the 722 evaluation problems (task success rate, number of successful tasks, and breakdown of failed tasks) over the 20 sets of learned parameters Avg. task Successful tasks Avg. no. of failed tasks success rate min. max. avg. SD collision out of range delay Proposed 1.000 722 722 722.00 0.00 0.00 0.00 0.00 AIRL-BMP 0.924 586 722 667.15 47.05 0.00 12.65 42.20 Base model 0.994 637 722 717.65 18.99 0.00 4.35 tablenotes Values quoted from Ishihara et al. (2024) Combined count for “out of range” and “delay” failures, which are not distinguished in the base model 7.4 Experimental Procedure in the Evaluation Phase As speed control problems different from those used for training, 5×5×5×5=6255× 5× 5× 5=625 problems were prepared by exhaustively combining the speed of the leading vehicle (km/h) from 45,55,65,75,85\45,55,65,75,85\, the initial speed of the following vehicle (km/h) from 0,10,30,50,70\0,10,30,50,70\, the initial inter-vehicle distance (m) from 30,45,65,70,80\30,45,65,70,80\, and the target range of the inter-vehicle distance (m) from [10,30],[20,40],[40,60],[50,60],[60,70]\[10,30],[20,40],[40,60],[50,60],[60,70]\. In addition, 3×3×3×3=813× 3× 3× 3=81 problems were prepared by exhaustively combining the speed of the leading vehicle (km/h) from 40,50,60\40,50,60\, the initial speed of the following vehicle (km/h) from 20,40,60\20,40,60\, the initial inter-vehicle distance (m) from 20,40,60\20,40,60\, and the target range of the inter-vehicle distance (m) from [10,20],[40,50],[45,60]\[10,20],[40,50],[45,60]\. A further 4×1×2×2=164× 1× 2× 2=16 problems were prepared by exhaustively combining the speed of the leading vehicle (km/h) from 20,30,50,60\20,30,50,60\, the initial speed of the following vehicle (km/h) from 30\30\, the initial inter-vehicle distance (m) from 10,50\10,50\, and the target range of the inter-vehicle distance (m) from [10,15],[30,45]\[10,15],[30,45]\. For these 722 evaluation problems, one set of control results was obtained with each of the 20 sets of learned weight parameters and under the action decision in the evaluation phase. threeparttable Table 5: Trajectory characteristics for the successful evaluation problems Time to settle Settling delay Exits from range SD after settling avg. SD avg. SD avg. SD pooled value Proposed 17.51 2.06 4.56 0.47 0.28 0.02 0.81 AIRL-BMP 31.26 3.22 11.61 3.66 0.45 0.19 2.20 7.5 Experimental Results and Discussion in the Evaluation Phase The control results for the evaluation problems with the weight parameters from each training run are shown in Table 4. For comparison, as in Table 1, Table 4 also includes the results reported in Ishihara et al. (2024) for the same evaluation problems with the base model. Table 4 shows that the proposed method achieved an average task success rate of 100%100\%, higher than the 99.4%99.4\% of the base model, with no variation across the sets. AIRL-BMP, in contrast, had an average task success rate of 92.4%92.4\%, lower than that of the base model, and its variation across the sets was relatively large, as was also the case for the training problems. A Mann-Whitney U test between the proposed method and AIRL-BMP showed that the former achieved a significantly larger number of successful tasks (median 722.00722.00, IQR [722.00,722.00][722.00,722.00] versus median 671.50671.50, IQR [638.50,715.25][638.50,715.25]; U=360U=360, p<0.001p<0.001, r=0.684r=0.684, n=20n=20 for each method). In the breakdown of the average number of failed tasks for AIRL-BMP, “delay” was the most frequent at 42.20 cases, followed by “out of range” at 12.65 cases. As was also the case for the training problems, a tendency toward insufficient learning performance was suggested. For all three methods, no case of “collision” occurred in any of the results. This is thought to be partly because there was no evaluation problem in which the initial speed of the following vehicle exceeded the speed of the leading vehicle by 50 km/h or more. Figure 5: Trajectories of the inter-vehicle distance with the proposed method and AIRL-BMP. a Problem E1, b Problem E2 As successful examples among the control results for the evaluation problems, the trajectories of the inter-vehicle distance after training with the proposed method and with AIRL-BMP are shown in Fig. 5, for problem E1, which required acceleration from the initial state, and problem E2, which required deceleration. In problems E1 and E2, (lnear,lfar,vfront)(l_ near,l_ far,v_ front) were (10,20,60)(10,20,60) and (60,70,45)(60,70,45), and the initial values of (xd,xv)(x_ d,x_ v) were (40,20)(40,20) and (45,70)(45,70), respectively. For both methods, the trajectories in Fig. 5 were generated by the policies using the parameters from the training run that gave the largest number of successful problems. Fig. 5 shows that the proposed method tended to settle the inter-vehicle distance within the target range earlier than AIRL-BMP and to give trajectories with smaller oscillations, as was the case for the training problems. Table 5 quantifies these characteristics for the successful evaluation problems. For the first three measures, the values were averaged over the successful problems for each set of learned parameters, and Table 5 shows the mean and the standard deviation of these averages over the 20 sets. The last measure is pooled over all the successful problems in all the sets. A Mann-Whitney U test between the proposed method and AIRL-BMP showed that the differences in all four measures were significant (p<0.001p<0.001, n=20n=20 for each method), where the pooled standard deviation for each set was used for the last measure. These results suggest that the proposed method acquired higher generalization performance than AIRL-BMP, capable of generating trajectories for successful tasks that settled earlier and with smaller oscillations. 8 Conclusion This paper proposed a fuzzy control method that integrates a reward estimation mechanism based on the AIRL framework into the base model, in which reinforcement learning is integrated with fuzzy inference. In the proposed method, the policy of the base model is treated as the generator, and the set of membership functions in the policy is also used as the cost function of the discriminator. This cost function approximates a quantity corresponding to the advantage of an action with respect to the optimal policy, using the set of membership functions rather than a value function. With this configuration, the proposed method makes stable and efficient learning possible by performing the discriminator update for reward estimation and the generator update for policy estimation simultaneously on the parameters shared by both, without estimating a value function. Simulations of an illustrative vehicle speed control task were conducted to compare the performance of the proposed method with that of AIRL-BMP, which uses the discriminator of AIRL as it stands and takes the policy of the base model as the generator. The results were also compared with those reported on the same problems in previous work for the base model, which uses a reward function set manually in advance. For the training problems, the proposed method achieved an average task success rate almost equal to that of the base model and much higher than that of AIRL-BMP, with far fewer epochs, and the variation in the success rate across training runs was smaller for the proposed method than for the other methods. For the evaluation problems, the proposed method achieved the highest average task success rate among the three methods, 100%100\%, with no variation across the 20 sets of results. These results show that the proposed method achieves stable and efficient learning and can acquire policies with high generalization performance. In this paper, the proposed method was verified on an illustrative vehicle speed control task. Applying the method to more realistic problems, in which the task and the shapes of the membership functions differ from those in this example, is one of the next steps. The simultaneous learning of the parameters of the membership functions in the consequent part and of the rule weights was not addressed in this paper, and will be examined in future work. Funding No funding was received to assist with the preparation of this manuscript. Data availability The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request. Declarations Conflict of interest The author has no conflicts of interest to declare that are relevant to the content of this article. 9 Derivation of Eq. (16) Substituting Eq. (14) into the right-hand side of Eq. (15) gives ℒ(αji)=∑t=0L−1−t[fαji(x,y,x′) (α^i_j)\!=\!\!\! _t=0^L-1\!\! \\!-E_D_t\!\! [\!f_\!α^i_j\!(x,y,x ) . . −ln(exp(fαji(x,y,x′))+π(y∣x))] \ \ .- \!\! (\! (f_\!α^i_j\!(x,y,x )\!)\!+\!π(y\!\! \!\!x)\! )\! ] −πt[ln(π(y∣x)) -\!E_ _t\! [ (π(y\! \!x)\!) −ln(exp(fαji(x,y,x′))+π(y∣x))] \ \ .\!- \!\! (\! (f_\!α^i_j\!(x,y,x )\!)\!+\!π(y\!\! \!\!x)\! )\!\! ]\! \ Treating π as a constant and differentiating Eq. () partially with respect to αjiα^i_j gives ∂αjiℒ(αji) ∂α^i_jL(α^i_j) =∑t=0L−1−t[∂αjifαji(x,y,x′) =\!\!\! _t=0^L-1\!\! \\!\!-E_D_t\!\! [\! ∂α^i_jf_\!α^i_j\!(x,y,x ) . . −exp(fαji(x,y,x′))∂αjifαji(x,y,x′)exp(fαji(x,y,x′))+π(y∣x)] .- (f_\!α^i_j\!(x,y,x )) ∂α^i_jf_\!α^i_j\!(x,y,x ) (f_\!α^i_j\!(x,y,x ))+π(y\! \!x) ] +πt[exp(fαji(x,y,x′))∂αjifαji(x,y,x′)exp(fαji(x,y,x′))+π(y∣x)] \ \, .+E_ _t\!\! [ (f_\!α^i_j\!(x,y,x )) ∂α^i_jf_\!α^i_j\!(x,y,x ) (f_\!α^i_j\!(x,y,x ))+π(y\! \!x) ]\! \ (2) =∑t=0L−1−t[(1−Dji(x,y,x′))∂αjifαji(x,y,x′)] =\!\!\! _t=0^L-1\!\! \\!\!-E_D_t\!\! [\!(1-D^i_j(x,y,x )) ∂α^i_jf_\!α^i_j\!(x,y,x )\! ] . +πt[Dji(x,y,x′)∂αjifαji(x,y,x′)] \ \, .+E_ _t\!\! [D^i_j(x,y,x ) ∂α^i_jf_α^i_j(x,y,x )\! ]\! \ (3) Multiplying the right-hand side of Eq. (3) by −δ-δ gives Eq. (16). 10 Derivation of Eq. (22) From Eq. (6), the characteristic eligibility eαji(t)e_α^i_j(t) is expressed as eαji(t) e_α^i_j(t) =−1T∂E(y(t),x(t))∂αji−π(y∣x(t))[∂E(y,x(t))∂αji] =\!- 1T\! \\!\! ∂ E(y(t);\!x(t)\!)∂α^i_j-E_π(y x(t)\!)\!\! [\! ∂ E(y;\!x(t)\!)∂α^i_j\! ]\! \ (1) From Eq. (2), the first term in the braces on the right-hand side of Eq. (1) is ∂E(y(t),x(t))∂αji=−ηiBi(y(t))∂αjiAi(x(t)) ∂ E(y(t);x(t))∂α^i_j=- _iB^i(y(t)) ∂α^i_jA^i(x(t)) (2) By replacing y(t)y(t) with y in Eq. (2) and taking the expectation of both sides, the second term in the braces on the right-hand side of Eq. (1) is expressed as π(y∣x(t))[∂E(y,x(t))∂αji] _π(y x(t))\!\! [\! ∂ E(y;\!x(t))∂α^i_j\! ] =−ηiπ(y∣x(t))[Bi(y)]∂αjiAi(x(t)) =- _iE_π(y x(t)) [B^i(y) ] ∂α^i_jA^i(x(t)) (3) Substituting Eqs. (2) and (3) into the right-hand side of Eq. (1) gives Eq. (22). 11 Update Equations Used in the Computer Experiments 11.1 Discriminator in the Proposed Method From Eqs. (12) and (13), the cost function for (x(t),y(t),x(t+1))(x(t),y(t),x(t+1)) contained in the trajectory of the demonstration is fαji(x(t),y(t),x(t+1)) f_α^i_j(x(t),y(t),x(t+1)) =ln1−|t[Aji(xj′,αji)]−Aji(xj(t+1),αji)| = \! \1- |E_D_t\!\! [A^i_j(x _j;α^i_j) ]-A^i_j(x_j(t+1);α^i_j) | \ When the trajectory of the demonstration for each problem consists of one episode, t[Aji(xj′,αji)]=Aji(xj(t+1),αji)E_D_t[A^i_j(x _j;α^i_j)]=A^i_j(x_j(t+1);α^i_j) holds, and the right-hand side of Eq. () is therefore zero. The gradient of Eq. () with respect to the parameter αjiα^i_j is then also zero, and the first term in the braces on the right-hand side of Eq. (16) is zero. Consequently, in the computer experiments with the learning rule of the proposed method, the update equation for the parameters αji∈α1long,α1short,α2fast,α2slowα^i_j∈\α long_1,α short_1,α fast_2,α slow_2\ in the cost function fαjif_α^i_j of the discriminator DjiD^i_j is Δαji α^i_j =δ∑t=0L−1−πt[Dji(x,y,x′)∂αjifαji(x,y,x′)] =δ\! _t=0^L-1\! \\!-E_ _t\!\! [\!D^i_j(x,y,x ) ∂α^i_jf_\!α^i_j(x,y,x )\! ]\! \ =δ∑t=0L−1πt[∂αji|vαji(x′,x,y)|1−|vαji(x′,x,y)|+π(y∣x)] =δ\! _t=0^L-1\! \\!E_ _t\!\! [ ∂α^i_j|v_α^i_j(x ;x,y)|1-|v_α^i_j(x ;x,y)|+π(y\! \!x)\! ]\! \ In these experiments, A1long(1)A long_1( x_1) and A2fast(2)A fast_2( x_2) were assumed to be monotonically increasing, and A1short(1)A short_1( x_1) and A2slow(2)A slow_2( x_2) were assumed to be monotonically decreasing. Under these assumptions, the sign of vαjiv_α^i_j is determined by the magnitude relation between xdx_ d or xvx_ v of the demonstration and that of the sample. The gradient of |vαji||v_α^i_j| was therefore computed as shown in Table 1. Table 1: Gradient of |vαji||v_α^i_j| in each case demonstration ≥ sample demonstration << sample xdx_ d xvx_ v xdx_ d xvx_ v ∂vα1long∂α1long ∂ v_α long_1∂α long_1 ∂vα2fast∂α2fast ∂ v_α fast_2∂α fast_2 −∂vα1long∂α1long- ∂ v_α long_1∂α long_1 −∂vα2fast∂α2fast- ∂ v_α fast_2∂α fast_2 −∂vα1short∂α1short- ∂ v_α short_1∂α short_1 −∂vα2slow∂α2slow- ∂ v_α slow_2∂α slow_2 ∂vα1short∂α1short ∂ v_α short_1∂α short_1 ∂vα2slow∂α2slow ∂ v_α slow_2∂α slow_2 threeparttable Table D1: Node values defining the consequent membership functions for the accelerator and brake inputs y1y_1 −5-5 −4-4 −3-3 −2-2 −1-1 00 11 22 33 44 55 B1high-ac.B_1 high-ac. 0 0 0 0 0 0 0.2 0.3 0.6 0.8 0.9 B1low-ac.B_1 low-ac. 0 0 0 0 0 1 0.8 0.6 0.5 0.4 0.2 B1high-br.B_1 high-br. 0.9 0.7 0.5 0.3 0.2 0 0 0 0 0 0 B1low-br.B_1 low-br. 0.1 0.2 0.4 0.6 0.7 1 0 0 0 0 0 11.2 Generator in the Proposed Method From Eq. (22), in the computer experiments with the learning rule of the proposed method, the characteristic eligibility for the policy of the generator is expressed as eαji(t) e_α^i_j(t) =−1Tηiπ(y1∣x(t))[B1i(y1)]−B1i(y1(t)) =- 1T _i \E_π(y_1 x(t)) [B^i_1(y_1) ]-B^i_1(y_1(t)) \ ⋅∂αjiA1i(x1(t))A2i(x2(t)) · ∂α^i_j \A^i_1(x_1(t))A^i_2(x_2(t)) \ (3) Eqs. (5.3) and (20) for the case μ=αjiμ=α^i_j and Eq. (3) are substituted into Eq. (21). For q=0q=0 and γ=1γ=1, the update equation for the parameters αji∈α1long,α1short,α2fast,α2slowα^i_j∈\α long_1,α short_1,α fast_2,α slow_2\ in the membership functions AjiA^i_j that constitute the policy π of the generator is then expressed as Δαji α^i_j =ετ∼π[∑t=0L−1∑u=0L−1(ln(Dji(x(u),y(u),x(u+1))) =\! _τ π\! [ _t=0^L-1\! \\! _u=0^L-1\!\! ( (D^i_j(x(u),y(u),x(u\!+\!1))) . −ln(−Dji(x(u),y(u),x(u+))))eαji(t)] .- (1\!-\!D^i_j(x(\!u\!)\!,y(\!u\!)\!,x(\!u\!+\!1\!)\!)\!) )\!\! \e_α^i_j\!(t)\! ] =ετ∼π[∑t=0L−1∑u=0L−1(fαji(x(u),y(u),x(u+1)) =\! _τ π\! [ _t=0^L-1\! \\! _u=0^L-1\! (f_α^i_j(x(u),y(u),x(u\!+\!1)) −ln(π(y(u)∣x(u))))eαji(t)] -\! (π(y(u)\! \!x(u))) )\! \e_α^i_j\!(t)\! ] (4) For example, in the case of rule i=5i=5, from Eqs. (6.3), (6.3), and (6.3), the characteristic eligibility eα15(t)=eα1long(t)e_α^5_1(t)=e_α long_1(t) is eα1long(t) e_α long_1(t) =−1Tη5π(y1∣x(t))[B1no-op.(y1)]−B1no-op.(y1(t)) =- 1T _5\! \E_π(y_1 x(t))\! [B no-op._1(y_1) ]\!-\!B no-op._1(y_1(t)) \ ⋅A2fast(x2(t))∂A1long(x1(t))∂α1long · A fast_2(x_2(t)) ∂ A long_1(x_1(t))∂α long_1 (5) Substituting Eq. (5) into Eq. (4) gives the update equation for the parameter α15=α1longα^5_1=α long_1 in the generator. 11.3 Discriminator in AIRL-BMP From Eqs. (8) and (10), the update equations for the parameters θ and ϕφ in the cost function fθ,ϕf_θ,φ of the discriminator D in AIRL-BMP are Δθ=δ∑t=0L−1 θ=δ\! _t=0^L-1\! t[(1−D(x,y,x′))∂θg(x,y)] \E_D_t\!\! [(1-D(x,y,x )) ∂θg(x,y) ] . −πt[D(x,y,x′)∂θg(x,y)] \ .-E_ _t\!\! [D(x,y,x ) ∂θg(x,y) ]\! \ (6) Δϕ=δ∑t=0L−1 φ\!=\!δ\!\! _t=0^L-1\!\! t[(−D(x,y,x′))∂ϕ(κh(x′)−h(x))] \\!E_D_t\!\! [\!(1\!-\!D(x,y,x )\!) ∂φ(κ h(x )\!-\!h(x)\!)\! ] . −πt[D(x,y,x′)∂ϕ(κh(x′)−h(x))] \ .-E_ _t\!\! [D(x,y,x ) ∂φ(κ h(x )-h(x)) ]\! \ (7) 11.4 Generator in AIRL-BMP Eqs. (11) and (3) are substituted into Eq. (21). For q=0q=0 and γ=1γ=1, the update equation for the parameters αji∈α1long,α1short,α2fast,α2slowα^i_j∈\α long_1,α short_1,α fast_2,α slow_2\ in the membership functions AjiA^i_j that constitute the policy π of the generator in AIRL-BMP is then expressed as Δαji α^i_j =ετ∼π[∑t=0L−1∑u=0L−1(ln(D(x(u),y(u),x(u+1))) = _τ π\! [ _t=0^L-1\! \\! _u=0^L-1\! ( (D(x(u),y(u),x(u\!+\!1))) −ln(−D(x(u),y(u),x(u+1))))eαji(t)] -\! (1\!-\!D(x(u),y(u),x(u\!+\!1))) )\!\! \e_\!α^i_j\!(t)\! ] =ετ∼π[∑t=0L−1∑u=0L−1(fθ,ϕ(x(u),y(u),x(u+1)) = _τ π\! [ _t=0^L-1\! \\! _u=0^L-1 (f_θ,φ(x(u),y(u),x(u\!+\!1)) −ln(π(y(u)∣x(u))))eαji(t)] -\! (π(y(u)\! \!x(u))) )\!\! \e_\!α^i_j\!(t)\! ] (8) 12 Definition of the Consequent Membership Functions The output variable y1y_1 was discretized with a step of 0.1 over [−5,5][-5,5]. The membership functions for the accelerator and brake inputs were defined as the piecewise linear interpolation of the node values listed in Table D1, with the functions for the accelerator input zero for y1<0y_1<0 and those for the brake input zero for y1>0y_1>0. B1no-op.B_1 no-op. takes the value 1 only at y1=0y_1=0. References Arora and Doshi (2021) Arora S, Doshi P (2021) A survey of inverse reinforcement learning: challenges, methods and progress. Artif Intell 297:103500. 10.1016/j.artint.2021.103500 Baxter and Bartlett (2001) Baxter J, Bartlett PL (2001) Infinite-horizon policy-gradient estimation. J Artif Intell Res 15:319–350. 10.1613/jair.806 Er and Deng (2004) Er MJ, Deng C (2004) Online tuning of fuzzy inference systems using dynamic fuzzy Q-learning. IEEE Trans Syst Man Cybern B 34(3):1478–1489. 10.1109/TSMCB.2004.825938 Finn et al. (2016a) Finn C, Christiano P, Abbeel P, et al (2016a) A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. In: NIPS 2016 Workshop on Adversarial Training, arXiv:1611.03852 Finn et al. (2016b) Finn C, Levine S, Abbeel P (2016b) Guided cost learning: deep inverse optimal control via policy optimization. In: Proc. of the 33rd International Conference on Machine Learning, New York, USA, p 49–58 Fu et al. (2018) Fu J, Luo K, Levine S (2018) Learning robust rewards with adversarial inverse reinforcement learning. In: Proc. of the 6th International Conference on Learning Representations, Vancouver, Canada Glorennec (1994) Glorennec PY (1994) Fuzzy Q-learning and dynamical fuzzy Q-learning. In: Proc. of the 3rd IEEE International Conference on Fuzzy Systems, Orlando, USA, p 474–479, 10.1109/FUZZY.1994.343739 Goodfellow (2015) Goodfellow IJ (2015) On distinguishability criteria for estimating generative models. In: ICLR 2015 Workshop, arXiv:1412.6515 Goodfellow et al. (2014) Goodfellow IJ, Pouget-Abadie J, Mirza M, et al (2014) Generative adversarial nets. In: Advances in Neural Information Processing Systems 27, p 2672–2680 Horiuchi et al. (1996) Horiuchi T, Fujino A, Katai O, et al (1996) Fuzzy interpolation-based Q-learning with continuous states and actions. In: Proc. of the 5th IEEE International Conference on Fuzzy Systems, New Orleans, USA, p 594–600, 10.1109/FUZZY.1996.551807 Imanina and Igarashi (2014) Imanina N, Igarashi H (2014) Policy gradient method using fuzzy controller in policies and its application. In: Proc. of the International Conference on Artificial Intelligence and Pattern Recognition, Kuala Lumpur, Malaysia, p 167–174 Ishihara and Igarashi (2006) Ishihara S, Igarashi H (2006) Applying the policy gradient method to behavior learning in multiagent systems: the pursuit problem. Syst Comput Jpn 37(10):101–109. 10.1002/scj.20248 Ishihara et al. (2024) Ishihara S, Ichige S, Igarashi H (2024) Learning of membership functions in a control method integrating fuzzy inference and the policy gradient method. J Jpn Soc Fuzzy Theory Intell Inform 36(3):666–675. 10.3156/jsoft.36.3_666, (in Japanese) Jouffe (1998) Jouffe L (1998) Fuzzy inference system learning by reinforcement methods. IEEE Trans Syst Man Cybern C 28(3):338–355. 10.1109/5326.704563 Karras et al. (2018) Karras T, Aila T, Laine S, et al (2018) Progressive growing of GANs for improved quality, stability, and variation. In: Proc. of the 6th International Conference on Learning Representations, Vancouver, Canada Peng et al. (2019) Peng XB, Kanazawa A, Toyer S, et al (2019) Variational discriminator bottleneck: improving imitation learning, inverse RL, and GANs by constraining information flow. In: Proc. of the 7th International Conference on Learning Representations, New Orleans, USA Salimans et al. (2016) Salimans T, Goodfellow I, Zaremba W, et al (2016) Improved techniques for training GANs. In: Advances in Neural Information Processing Systems 29, p 2234–2242 Tavakol Aghaei and Onat (2017) Tavakol Aghaei V, Onat A (2017) Tuning scaling factors of fuzzy logic controllers via reinforcement learning policy gradient algorithms. In: Proc. of the 3rd International Conference on Mechatronics and Robotics Engineering, Paris, France, p 146–151, 10.1145/3068796.3068827 Wang et al. (2004) Wang X, Xu X, He H (2004) Policy gradient fuzzy reinforcement learning. In: Proc. of the 3rd International Conference on Machine Learning and Cybernetics, Shanghai, China, p 992–995, 10.1109/ICMLC.2004.1382332 Williams (1992) Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8:229–256. 10.1007/BF00992696 Zare et al. (2024) Zare M, Kebria PM, Khosravi A, et al (2024) A survey of imitation learning: algorithms, recent developments, and challenges. IEEE Trans Cybern 54(12):7173–7186. 10.1109/TCYB.2024.3395626 Ziebart et al. (2008) Ziebart BD, Maas A, Bagnell JA, et al (2008) Maximum entropy inverse reinforcement learning. In: Proc. of the 23rd AAAI Conference on Artificial Intelligence, Chicago, USA, p 1433–1438