Paper deep dive
Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms
Zhenjie Ren, Xiaoli Wei, Xiang Yu, Xun Yu Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/8/2026, 7:21:33 AM
Summary
This paper develops continuous-time q-learning algorithms for mean-field control (MFC) problems with controlled common noise. It establishes martingale conditions for the value function and integrated q-function (Iq-function), quantifies approximation errors when transitioning from theoretical relaxed control formulations to practical exploratory formulations, and proposes both an optimal q-learning algorithm for linear quadratic (LQ) settings and a general Actor-Critic q-learning algorithm. Theoretical convergence of inner iterations is proven for the LQ framework, with numerical examples demonstrating satisfactory performance.
Entities (10)
Relation Signals (7)
This paper → proposes → q-learning algorithms
confidence 97% · aiming to further devise q-learning algorithms for mean-field control
q-learning algorithms → includes → Actor-Critic q-learning algorithm
confidence 96% · allows us to propose several algorithms including the Actor-Critic q-learning algorithm
Mean-field control (MFC) → incorporates → Common Noise
confidence 95% · mean-field control (MFC) with controlled common noise
Actor-Critic q-learning algorithm → utilizes → Iq-function
confidence 94% · policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function
Iq-function → satisfies → Martingale condition
confidence 93% · establish the martingale condition of the value function and the Iq-function
This paper → proves → convergence in LQ framework
confidence 92% · establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework
This paper → analyzes → error between relaxed and exploratory formulations
confidence 90% · quantify the error incurred when they are replaced by the observable ones in the exploratory formulation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establish the martingale condition of the value function and the Iq-function by evaluating along the conditional state distributions generated by all test policies. As the data in the relaxed control formulation are not observable in practice, we quantify the error incurred when they are replaced by the observable ones in the exploratory formulation under discretely sampled actions. This, together with a two-layer fixed point characterization of an optimal policy in Ren et al. (2026), allows us to propose several algorithms including the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function, and the value function and Iq-function are updated in the Critic-step based on the martingale orthogonality condition using the data from the exploratory formulation. We also establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework. In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance.
Tags
Links
- Source: https://arxiv.org/abs/2604.27378v1
- Canonical: https://arxiv.org/abs/2604.27378v1
Trouble viewing inline? Open PDF directly →
Full Text
125,527 characters extracted from source content.
Expand or collapse full text
Continuous-time q-learning for mean-field control with common noise, part-I: q-learning algorithms Zhenjie Ren Email: zhenjie.ren@univ-evry.fr, LaMME, Université Évry Paris-Saclay, Évry, France. Xiaoli Wei Email: tyswxl@gmail.com. Xiang Yu Email: xiang.yu@polyu.edu.hk, Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong. Xun Yu Zhou Email: xz2574@columbia.edu, Department of Industrial Engineering and Operations Research, Columbia University, New York, USA. (This version: April 30, 2026) Abstract This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establish the martingale condition of the value function and the Iq-function by evaluating along the conditional state distributions generated by all test policies. As the data in the relaxed control formulation are not observable in practice, we quantify the error incurred when they are replaced by the observable ones in the exploratory formulation under discretely sampled actions. This, together with a two-layer fixed point characterization of an optimal policy in Ren et al. (2026), allows us to propose several algorithms including the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function, and the value function and Iq-function are updated in the Critic-step based on the martingale orthogonality condition using the data from the exploratory formulation. We also establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework. In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance. Keywords: Mean-field control, common noise, martingale characterization, optimal q-learning algorithm, Actor-Critic q-learning algorithm 1 Introduction Thanks to its tractability and scalability, MFC has become a popular paradigm to approximate the large system of cooperative agents who are coordinated by the social planner to achieve the social optimum. In most conventional methods for solving MFC problems, one needs to assume the full knowledge or precise estimations of the underlying model. However, in real-life applications, the agent or the social planner may only have limited or no information about the environment, which may cause huge errors or inefficiency in implementing the theoretical solutions. This motivated fast-growing developments in reinforcement learning (RL) approaches for MFC problems. Based on the principle of trial-and-error in the unknown environment, the social planner can gradually learn to select best actions from the exploration and exploitation procedure. See some early studies on discrete-time mean-field Markov decision processes and RL algorithms among Gu et al. (2021, 2024); Angiuli et al. (2022, 2023b); Carmona et al (2023); Cui et al. (2021), as well as Laurière et al. (2022) for an overview of discrete-time RL approaches in mean-field models. However, many real-world stochastic systems, particularly in finance and engineering, evolve continuously through time, which call for suitable RL algorithms in a continuous-time framework. Recently, for classical single agent’s stochastic control problems, Wang et al. (2020), Jia and Zhou (2022a, b, 2023) have laid the theoretical foundation for RL with entropy regularization in continuous time models with continuous state space and action space. In particular, Wang et al. (2020) discussed the optimal policy in an entropy-regularized exploratory formulation for continuous-time RL, where the entropy term plays the crucial role to encourage the exploration. Jia and Zhou (2022a) examined the policy evaluation problem by establishing a martingale condition of the value function. Jia and Zhou (2022b) investigated the policy gradient algorithm by connecting it to the martingale condition and policy evaluation in Jia and Zhou (2022a). Jia and Zhou (2023) pioneered the continuous-time q-learning theory by introducing the q-function as the first order time derivative of the advantage function and establishing a joint martingale characterization of the q-function and the value function. The continuous-time entropy-regularized RL approach has been rapidly generalized in different single agent’s settings and applications, see, for instance, Wang et al. (2023); Han et al. (2023); Dong (2024); Dai et al. (2023a); Bo et al. (2025); Dai et al. (2023b); Giegrich et al. (2024); Huang et al. (2025); Jia (2026). Recently, the continuous-time exploratory formulation and some RL algorithms have also been developed in mean-field models without common noise, see among Guo et al. (2022); Frikha et al. (2025); Liang et al. (2024); Wei and Yu (2025); Wei et al. (2024). In Ren et al. (2026), we proposed a proper definition of Iq-function and contributed some new theoretical foundations for the exploratory MFC problem with controlled common noise. The social planner therein is responsible for the learning task on behalf of the population, who assigns randomized policies to agents and lets them interact with the environment based on their own current states and population’s state distribution. Based on agents’ interactions, the social planner collects the population’s distribution and agent’s individual rewards to generate the aggregated reward and improve the policy through iterations. Some applications that fit into this framework could be the centralized traffic management system that coordinates the vehicles on the road and the central bank that sets economic policy such as tax rates to maximize a social welfare function. In previous studies on single agent’s control problems and MFC without common noise, the optimal policy resulting from the exploratory HJB equation is typically characterized as a one-layer fixed point of the Gibbs measure. In sharp contrast, the controlled common noise in Ren et al. (2026) creates an additional nonlinear functional of policy, which significantly complicates the definition of Iq-function and its connection to the optimal policy. As a main finding of Ren et al. (2026), it is revealed that the optimal policy corresponds to a two-layer fixed point of the argmax operator of the Iq-function, which poses some interesting open problems in devising the model-free RL algorithms: How to learn the Iq-function and how to learn the optimal policy via the implicit policy iteration and the learnt Iq-function? As a continuation of our work Ren et al. (2026), the present paper is devoted to addressing these new challenges in q-learning algorithms based on whether the two-layer fixed point admits an explicit expression or not. We summarize the main contributions of this paper as three-fold: (i) First, based on the relaxed control formulation, we establish the martingale conditions of the Iq-function and the value function for a given policy and the optimal policy, respectively. Note that these martingale conditions rely on the data generated in the relaxed control formulation, including the conditional state distribution and the reward trajectories, which are not directly observable in practice. In the implementation step, we need to utilize observable data generated from the exploratory formulation under discretely sampled actions. Therefore, it becomes important to discuss whether the theoretical martingale condition can still be employed in the policy evaluation. In fact, unlike the result in Jia and Zhou (2023) in the single agent’s setting that the martingale conditions remain valid with data from the sampled SDE, the counterpart in our mean-field model may generally fail. In response, we establish a new error analysis of the martingale condition in our MFC problem when we substitute the data by the one generated by discretely sampled actions (see Theorem 3.4 and Proposition 4.1). (i) When the optimal policy as a two-layer fixed point can be obtained explicitly such as in the LQ setting in Ren et al. (2026), we devise an optimal q-learning algorithm (see Figure 4 and Algorithm 1) by taking the advantage of the fact that the optimal Iq-function and the optimal policy share the same parameters in the algorithm. On the other hand, for the general case when the two-layer fixed point may not admit an explicit expression, we devise an Actor-Critic q-learning algorithm to allow different parameterizations of the Iq-function and the policy. In the Actor step, we update the policy on strength of the implicit policy improvement rule via the improved Iq-function, leading to the operator for parameter iterations involving the partial linear functional derivative of the unregularized Iq-function (see (4.10)). In the Critic step, we update the parameters in Iq-function and the value function simultaneously based on the averaged martingale orthogonal condition (see Figure 4 and Algorithm 2). We also discuss the subtle issue in the Actor step to implement the implicit policy improvement iterations, i.e. the inner iterations of parameters to find the optimal one-step iterated policy. The convergence of the inner fixed point iterations is rigorously established in the LQ framework over an infinite horizon. (i) To illustrate the efficiency of our proposed algorithms, we study two examples within and beyond the LQ framework. In both examples, we are able to derive the explicit form of the optimal policy and the resulting optimal value function and the optimal Iq-function, allowing us to obtain the exact parameterizations of these target functions. We then implement and compare three algorithms, namely the optimal q-learning algorithm, the Actor-Critic q-learning algorithm without inner fixed point iterations, and the Actor-Critic q-learning algorithm with inner fixed point iterations. In numerical illustrations, all three algorithms show satisfactory performance, and it is interesting to observe in two examples that the inner fixed point iterations to learn the optimal one-step iterated policy may not be necessary as the alternative iterations in Actor and Critic steps will achieve the same goal to converge to the two-layer fixed point. The remainder of this paper is organized as follows. Section 2 reviews the formulations and theoretical findings of Iq-function in continuous-time MFC problems with common noise. Section 3 establishes the martingale conditions for the Iq-function and the value function. Section 4 presents two q-learning algorithms, namely the optimal q-learning algorithm when the two-layer fixed point characterization of the optimal policy admits an explicit form and the Actor-Critic q-learning algorithm otherwise by resorting to the policy improvement result. Section 5 verifies the convergence of the inner iterations for the LQ MFC problems over an infinite horizon. Section 6 presents two numerical examples using our proposed q-learning algorithms. Notations Given two Polish spaces (S,)(S, S) and (T,)(T, T), for any p>0p>0, we denote by p(S) P_p(S) the space of all probability measures with finite p-th moments on S and equip p(S) P_p(S) with the p-Wasserstein metric p W_p defined by p(μ,ν)=inf(∫S×S|x−y|pγ(dx,dy))1/p:γ∈p(S×S)has marginalsμandν. W_p(μ,ν)= \ ( _S× S|x-y|^pγ(dx,dy) )^1/p:γ∈ P_p(S× S)\; has marginals\;μ\; and\;ν \. For μ∈p(ℝd)μ∈ P_p(R^d), denote ‖μ‖p:=(∫ℝd|x|pμ(dx))1/p\|μ\|_p:= ( _R^d|x|^pμ(dx) )^1/p. We denote by ac(S) P_ac(S) the space of probability measures on S that are absolutely continuous with respect to the Lebesgue measure. We denote by ac(T|S) P_ac(T|S) the space of all probability kernels from S to ac(T) P_ac(T). We denote L2(Ω,ℱ,ℙ;ℝd)L^2( , F,P;R^d) the space of all ℱ F-adapted ℝdR^d-valued square-integrable random variables on the probability space (Ω,ℱ,ℙ)( , F,P). For any measurable function g:S→ℝkg:S ^k, we denote ∫Sg(x)μ(dx) _Sg(x)μ(dx) as ⟨g,μ⟩ g,μ . For a functional F:2(S)→ℝF: P_2(S) , we denote by ∂μF(μ)(x) _μF(μ)(x), ∂x∂μF(μ)(x) _x _μF(μ)(x) and ∂μ2F(μ)(x,x′) _μ^2F(μ)(x,x ) the L derivative in μ, the mixed second-order derivative with respect to μ and x, and the second-order derivative in measure μ, respectively. We adopt (μ,Σ) N(μ, ) and ([p,q]) U([p,q]) for a Gaussian distribution with mean μ and covariance Σ and a uniform distribution on [p,q][p,q], respectively. 2 Setup and Preparations 2.1 Exploratory MFC with common noise in RL We work with an unknown mean-field model, where the social planner is responsible for the RL task who needs to learn the optimal policy based on the trial-and-error procedure. To begin with, let us first introduce two RL formulations of MFC with common noise as discussed in Ren et al. (2026): a relaxed control formulation that is needed for theoretical analysis and an exploratory formulation under discretely sampled actions that is suitable for the implementation of learning. Let (Ωe,ℱe,ℙe)( ^e, F^e,P^e) be a complete probability space with a product structure (Ω0×Ω1×Ω2,ℱ0⊗ℱ1⊗ℱ2,ℙ0⊗ℙ1⊗ℙ2)( ^0× ^1× ^2, F^0 F^1 F^2,P^0 ^1 ^2), where (Ω0,ℱ0,ℙ0)( ^0, F^0,P^0) supports a n-dimensional Brownian motion B=(Bs)s∈[0,T]B=(B_s)_s∈[0,T] that serves as common noise, (Ω1,ℱ1,ℙ1)( ^1, F^1,P^1) supports an m-dimensional Brownian motion W=(Ws)s∈[0,T]W=(W_s)_s∈[0,T], and (Ω2,ℱ2,ℙ2)( ^2, F^2,P^2) supports a sequence of i.i.d. uniform random variables (Ui)i∈ℕ(U_i)_i for the action randomization. We denote by B,W=(ℱsB,W)0≤s≤TF^B,W=( F_s^B,W)_0≤ s≤ T the filtration generated by B,WB,W and by =(s)0≤s≤TG=( G_s)_0≤ s≤ T the filtration generated by B, respectively. It is also assumed that there exists a sub-algebra ℋ H of ℱ1 F^1 for the initial state such that ℋ H is independent of ℱ1 F^1 and it is “rich enough” in the sense that for any μ∈2(ℝd)μ∈ P_2(R^d), there exists an ℋ H-measurable random variable ξ on (Ω1,ℙ1)( ^1,P^1) such that ℙ1∘ξ−1=μP^1 ξ^-1=μ. We denote by ℱse=ℱsB,W∨ℋ∨σ(Ui,si≤s) F_s^e= F_s^B,W H σ(U_i,s_i≤ s). Throughout the paper, we shall make the following assumption. Assumption 2.1. The following conditions for the dynamics and reward functions hold true. (i) b, σ, σo _o, r:r: [0,T]×ℝd×2(ℝd)×[0,T]×R^d× P_2(R^d)× A → ℝd,ℝd×mR^d,R^d× m, ℝd×nR^d× n, ℝR are jointly continuous, and g:ℝd×2(ℝd)→ℝg:R^d× P_2(R^d) is jointly continuous; (i) b, σ, σo _o are uniformly Lipschitz continuous in x and μ, that is, for f∈b,σ,σof∈\b,σ, _o\, there exists a constant C>0C>0 such that |f(t,x,μ)−f(t,x′,μ′)|≤C(|x−x′|+2(μ,μ′))|f(t,x,μ)-f(t,x ,μ )|≤ C (|x-x |+ W_2(μ,μ ) ); (i) There exists some constant C>0C>0 such that for any (t,x,μ,a)∈[0,T]×ℝd×2(ℝd)×(t,x,μ,a)∈[0,T]×R^d× P_2(R^d)× A, |b(t,x,μ,a)| |b(t,x,μ,a)| ≤C(1+|x|+‖μ‖2+|a|), ≤ C (1+|x|+\|μ\|_2+|a| ), |(σσ⊺+σoσo⊺)(t,x,μ,a)| |(σ + _o _o )(t,x,μ,a) | ≤C(1+|x|2+‖μ‖22+|a|2), ≤ C (1+|x|^2+\|μ\|_2^2+|a|^2 ), |r(t,x,μ,a)| |r(t,x,μ,a)| ≤C(1+|x|2+‖μ‖22+|a|2). ≤ C (1+|x|^2+\|μ\|_2^2+|a|^2 ). For theoretical analysis, we first consider the relaxed control formulation. Let Π stand for the set of admissible policies (see Definition 2.2 in Ren et al. (2026)). For ∈Π π∈ , we denote by f(t,x,μ) f_ π(t,x,μ) :=∫f(t,x,μ,a)(a|t,x,μ)a,forf∈b,σ,σo, := _ Af(t,x,μ,a) π(a|t,x,μ)da,\;\;\; for\;f\;∈\b,σ, _o\, std(f) std_ π(f) :=(∫ff⊺(t,x,μ,a)(a|t,x,μ)a−ff⊺(t,x,μ))1/2,forf∈b,σ,σo, := ( _ Af (t,x,μ,a) π(a|t,x,μ)da-f_ πf_ π (t,x,μ) )^1/2,\;\; for\;f\;∈\b,σ, _o\, E(t,x,μ): E_ π(t,x,μ): =−∫log(a|t,x,μ)(a|t,x,μ)a,ℰ(t,μ,):=∫ℝdE(t,x,μ)μ(dx), =- _ A π(a|t,x,μ) π(a|t,x,μ)da, (t,μ, π):= _R^dE_ π(t,x,μ)μ(dx), r^(t,μ): r_ π(t,μ): =∫ℝdr(t,x,μ)μ(dx),g^(t,μ):=∫ℝdg(x,μ)μ(dx). = _R^dr_ π(t,x,μ)μ(dx), g(t,μ):= _R^dg(x,μ)μ(dx). The state dynamics in the relaxed control formulation is given by dXs dX_s π =b(s,Xs,μs)ds+σ(s,Xs,μs)dWs+σo,(s,Xs,μs)dBs =b_ π(s,X_s π, _s π)ds+ _ π(s,X_s π, _s π)dW_s+ _o, π(s,X_s π, _s π)dB_s +std(σ)(s,Xs,μs)dW¯s+std(σo)(s,Xs,μs)dB¯s,s∈[t,T], \;\;\;+ std_ π(σ)(s,X_s π, _s π)d W_s+ std_ π( _o)(s,X_s π, _s π)d B_s,\;s∈[t,T], (2.1) where Xt=ξ∼μX_t π=ξ μ, μs _s π is the conditional law of XsX_s π given s G_s, and W¯ W and B¯ B are extra one-dimensional Brownian motions independent of W and B. The optimal value function of the MFC problem is J~∗(t,μ)=sup∈ΠJ~(t,μ;) J^*(t,μ)= _ π∈ J(t,μ; π), where J~(t,μ;) J(t,μ; π) =e[∫tTe−β(s−t)(r^(s,μs)+γℰ(s,μs,))s+e−β(T−t)g^(μT)]. =E^e [ _t^Te^-β(s-t) ( r_ π(s, _s π)+ (s, _s π, π) )ds+e^-β(T-t) g( _T π) ]. (2.2) The relaxed control formulation, while suitable for theoretical analysis, cannot be directly observed in the learning procedure. Instead, we consider the exploratory formulation for the implementation. Moreover, to avoid measure-theoretical issues caused by continuous sampling, we adopt the exploratory formulation under discretely sampled actions. That is, for a given time gird =t=s0<…<sn=T D=\t=s_0<…<s_n=T\ with ||=max0≤i≤n−1|si+1−si|| D|= _0≤ i≤ n-1|s_i+1-s_i|, the representative agent takes action randomization only at time grids. Her state process is then given by dXs, dX_s D, π =b(s,Xs,,μs,,aδ(s),)ds+σ(s,Xs,,μs,,aδ(s),)dWs =b(s,X_s D, π, _s D, π,a_δ(s) D, π)ds+σ(s,X_s D, π, _s D, π,a_δ(s) D, π)dW_s (2.3) +σo(s,Xs,,μs,,aδ(s),)dBs,s∈[t,T],Xt,=ξ∼μ, \;\;\;\;\;+ _o(s,X_s D, π, _s D, π,a_δ(s) D, π)dB_s,\;s∈[t,T],\;X_t D, π=ξ μ, where μs,=ℒ(Xs,|s) _s D, π= L(X_s D, π| G_s), and for s∈[si,si+1)s∈[s_i,s_i+1), δ(s)=siδ(s)=s_i and asi=ϕ(si,Xsi,,μsi,,Usi)∼(⋅|si,Xsi,,μsi,)a_s_i= _ π(s_i,X_s_i D, π, _s_i D, π,U_s_i) π(·|s_i,X_s_i D, π, _s_i D, π). The social planner’s goal is to maximize the objective function as ||→0| D|→ 0, i.e., J∗(t,ξ)=sup∈Πlim||→0J(t,ξ;)J^*(t,ξ)= _ π∈ _| D|→ 0J D(t,ξ; π), where J(t,ξ;)J D(t,ξ; π) is given by J(t,ξ;) J D(t,ξ; π) =e[∫tTe−β(s−t)(r(s,Xs,,μs,,aδ(s),)+γE(δ(s),Xδ(s),μδ(s),))ds =E^e [ _t^Te^-β(s-t) (r(s,X_s D, π, _s D, π,a_δ(s) D, π)+γ E_ π(δ(s),X_δ(s) π, _δ(s) D, π) )ds +e−β(T−t)g(XT,,μT,)|Xt,=ξ]. \;\;\;\;\;\;+e^-β(T-t)g(X_T D, π, _T D, π) |X_t D, π=ξ ]. It is shown in Ren et al. (2026) that, under suitable model assumptions, two value functions in the relaxed control and the exploratory formulations coincide, i.e., J~(t,μ;)=J(t,ξ;) J(t,μ; π)=J(t,ξ; π). We therefore shall use J by slight abuse of notation in the rest of the paper. 2.2 Continuous-time integrated q-function As the foundation of continuous-time q-learning for MFC with common noise, Ren et al. (2026) proposes the definition of the integrated q-function (Iq-function). Specifically, given a policy ∈Π π∈ , for any (t,μ,)∈[0,T]×2(ℝd)×ac(|ℝd)(t,μ, h)∈[0,T]× P_2(R^d)× P_ac( A|R^d), the Iq-function is defined by qγ(t,μ,;)=∂J∂t(t,μ;)−βJ(t,μ;)+ℋ(t,μ,;)+γℰ(t,μ,), q^γ(t,μ, h; π)= ∂ J∂ t(t,μ; π)-β J(t,μ; π)+ H(t,μ, h; π)+ (t,μ, h), (2.4) where ℋ:[0,T]×2(ℝd)×ac(|ℝd)→ℝ H:[0,T]× P_2(R^d)× P_ac( A|R^d) denotes the integrated Hamiltonian ℋ(t,μ,;) H(t,μ, h; π) :=∫ℝd×H(t,x,μ,a,∂μJ(t,μ;)(x),∂x∂μJ(t,μ;)(x))(a|x)aμ(dx) := _R^d× AH (t,x,μ,a, _μJ(t,μ; π)(x), _x _μJ(t,μ; π)(x) ) h(a|x)daμ(dx) +12∫ℝd×ℝdTr(σo,(t,x,μ)σo,(t,x′,μ)⊺∂μ2J(t,μ;)(x,x′))μ(dx)⊗μ(dx′). \;\;\;+ 12 _R^d×R^d Tr ( _o, h(t,x,μ) _o, h(t,x ,μ) _μ^2J(t,μ; π)(x,x ) )μ(dx) μ(dx ). We call q0(t,μ,;)=qγ(t,μ,;)−γℰ(t,μ,)q^0(t,μ, h; π)=q^γ(t,μ, h; π)- (t,μ, h) the unregularized Iq-function. Building upon Ren et al. (2026), we summarize some key results of Iq-function (Lemma 3.4, Corollary 4.2 and Theorem 3.10 in Ren et al. (2026)) in the following proposition. Proposition 2.2. (i) The dynamic programming equation of the value function J in (2.2) can be written in terms of Iq-function q0(t,μ,;)+γℰ(t,μ,)=0. q^0(t,μ, π; π)+ (t,μ, π)=0. (2.5) (i) ∗ h^* is the unique maximizer of max∈ac(|ℝd)qγ(t,μ,;) _ h∈ P_ac( A|R^d)q^γ(t,μ, h; π) if and only if ∗ h^* is the fixed-point of the map Φ:Π→Π _ π: → ∗(a|t,x,μ)=Φ(∗)(t,x,μ)=exp1γδq0δ(t,μ,∗;)(x,a)∫exp1γδq0δ(t,μ,∗;)(x,a)a,t∈[0,T], h^*(a|t,x,μ)= _ π( h^*)(t,x,μ)= \ 1γ δ q^0δ h(t,μ, h^*; π)(x,a) \ _ A \ 1γ δ q^0δ h(t,μ, h^*; π)(x,a) \da,\;t∈[0,T], (2.6) where δq0δ(s,μ,;)(x,a) δ q^0δ h(s,μ, h; π)(x,a) is the partial linear functional derivative of q0q^0 with respect to h, see Definition 3.7 in Ren et al. (2026). (i) For a given ∈Π π∈ , if we select a new policy ′ π such that either qγ(s,μ,′;)≥qγ(s,μ,;)q^γ(s,μ, π ; π)≥ q^γ(s,μ, π; π) for any s∈[t,T]s∈[t,T] or e[∫tTe−β(s−t)qγ(s,μs′,′;)s]≥e[∫tTe−β(s−t)qγ(s,μs′,;)s]E^e[ _t^Te^-β(s-t)q^γ(s, _s π , π ; π)ds] ^e[ _t^Te^-β(s-t)q^γ(s, _s π , π; π)ds] holds, we then have J(t,μ;′)≥J(t,μ;)J(t,μ; π )≥ J(t,μ; π). In particular, if ∗ h^* is the fixed point of the map Φ _ π, then qγ(t,μ,∗;)≥qγ(t,μ,;)q^γ(t,μ, h^*; π)≥ q^γ(t,μ, π; π) for any (t,μ)∈[0,T]×2(ℝd)(t,μ)∈[0,T]× P_2(R^d). (iv) An optimal policy ∗ π^* satisfies the two-layer fixed point condition ∗(a|t,x,μ)=exp1γδq0,∗δ(t,μ,∗)(x,a)∫exp1γδq0,∗δ(t,μ,∗)(x,a)a. π^*(a|t,x,μ)= \ 1γ δ q^0,*δ h(t,μ, π^*)(x,a) \ _ A \ 1γ δ q^0,*δ h(t,μ, π^*)(x,a) \da. (2.7) 3 Martingale Characterization In this section, we provide the martingale characterization of the value function and the Iq-function in a joint manner in the mean-field setting with common noise. The proofs are similar to those for MFC without common noise (see Theorems 4.3 and 4.4 in Wei and Yu (2025)) and hence are omitted. Theorem 3.1 (Characterization of value function and Iq-function). Let a continuous function J^:[0,T]×2(ℝd)→ℝ J:[0,T]× P_2(R^d) and a continuous function q^0:[0,T]×2(ℝd)×ac(|ℝd)→ℝ q^0:[0,T]× P_2(R^d)× P_ac( A|R^d) be given. Let ∈Π π∈ be a root of the following equation of h q^0(t,μ,)+γℰ(t,μ,)=0 q^0(t,μ, h)+γ E(t,μ, h)=0 (3.1) for each fixed (t,μ)∈[0,T]×2(ℝd)(t,μ)∈[0,T]× P_2(R^d). Then J J and q^0 q^0 are the value function and the unregularized Iq-function associated with π being the root of (3.1) if and only if J J and q^0 q^0 satisfy J^(T,μ)=g^(μ) J(T,μ)= g(μ), and for any (t,μ,)∈[0,T]×2(ℝd)×Π(t,μ, h)∈[0,T]× P_2(R^d)× , the process Ms:=e−βsJ^(s,μs)+∫tse−βu(r^(u,μu)−q^0(u,μu,))u M_s h:=e^-β s J(s, _s h)+ _t^se^-β u ( r_ h(u, _u h)- q^0(u, _u h, h) )du (3.2) is a (ℙe,)(P^e,G)-martingale, where μs=ℒ(Xs|s) _s h= L(X_s h| G_s) is the conditional state distribution in (2.1). We also provide the martingale characterization of the optimal value function and the optimal Iq-function, which will be used to design the optimal q-learning algorithm. Theorem 3.2 (Characterization of the optimal value function and optimal Iq-function). Let a policy ^:[0,T]×ℝd×2(ℝd)→() π:[0,T]×R^d× P_2(R^d)→ P( A), a continuous function J^:[0,T]×2(ℝd)→ℝ J:[0,T]× P_2(R^d) and a continuous function q^0:[0,T]×2(ℝd)×ac(|ℝd)→ℝ q^0:[0,T]× P_2(R^d)× P_ac( A|R^d) be given. Then π, J J and q^0 q^0 are respectively an optimal policy, the optimal value function and the unregularized optimal Iq-function if and only if the following three conditions hold (i) (Consistency condition) J J satisfies J^(T,μ)=g^(μ) J(T,μ)= g(μ), (i) (Two-layer fixed point condition) π satisfies q^0(t,μ,^)+γℰ(t,μ,^)=0,^(a|t,x,μ)=exp1γδq^0δ(t,μ,^)(x,a)∫exp1γδq^0δ(t,μ,^)(x,a)a. \ array[]l q^0(t,μ, π)+γ E(t,μ, π)=0,\\ π(a|t,x,μ)= \ 1γ δ q^0δ h(t,μ, π)(x,a) \ _ A \ 1γ δ q^0δ h(t,μ, π)(x,a) \da. array . (3.5) (i) (Martingale condition) for any (t,μ,)∈[0,T]×2(ℝd)×Π(t,μ, h)∈[0,T]× P_2(R^d)× , the process Ms\M_s h\ in (3.2) is a (ℙe,)(P^e,G)-martingale. Remark 3.3. In Theorem 3.1, the martingale characterization (3.2) utilizes the unobservable measure flow μss\ _s h\_s, generated by all test policies h in the relaxed control formulation (2.1). If the unobservable measure flow μss\ _s h\_s is replaced by the observable μs,s\ _s D, π\_s in the exploratory formulation (2.3) under discretely sampled actions, the arguments to prove Theorem 3.1 become invalid and the process (3.2) may no longer be a martingale in the mean-field model. It is therefore crucial to quantify the error of the martingale condition using μs,s\ _s D, π\_s in place of μss\ _s h\_s in (3.2) if we plan to employ this martingale condition in policy evaluation. Let Ms,M_s D, h denote the counterpart process of MsM_s h in (3.2) by replacing μss\ _s h\_s with μs,s\ _s D, π\_s, we have the following result on error analysis. Theorem 3.4. For any G-adapted process ηss∈[t,T]\ _s h\_s∈[t,T] satisfying sup∈Πe[∫0T|ηs|2s]<+∞ _ h∈ E^e[ _0^T| _s h|^2ds]<+∞, we have lim||→0|e[∫tTηsd(Ms−Ms,)]|=0 _| D|→ 0 |E^e [ _t^T _s hd(M_s h-M_s D, h) ] |=0. Consequently, when Mss\M_s h\_s is a (ℙe,)(P^e,G)-martingale, lim||→0e[∫tTηsMs,]=0 _| D|→ 0E^e [ _t^T _s hdM_s D, h ]=0. Proof. Let qˇ0(t,μ,) q^0(t,μ, h) denote the function obtained from q0q^0 in (2.4) by replacing J(t,μ;)J(t,μ; π) with J^(t,μ) J(t,μ). Also denote Δq0(t,μ,):= q^0(t,μ, h):= qˇ0(t,μ,)−q^0(t,μ,), q^0(t,μ, h)- q^0(t,μ, h), ℒJ^(t,x,x′,μ,a,a′):= L J(t,x,x ,μ,a,a ):= ∂J^∂t(t,μ)−βJ^(t,μ)+b(t,x,μ,a)⊺∂μJ^(t,μ)(x) ∂ J∂ t(t,μ)-β J(t,μ)+b(t,x,μ,a) _μ J(t,μ)(x) +12Tr((σσ⊺+σoσo⊺)(t,x,μ,a)∂x∂μJ^(t,μ)(x)) + 12 Tr ( (σ + _o _o )(t,x,μ,a) _x _μ J(t,μ)(x) ) +12Tr(σo(t,x,μ,a)σo(t,x′,μ,a′)∂μ2J^(t,μ)(x,x′)). + 12 Tr ( _o(t,x,μ,a) _o(t,x ,μ,a ) _μ^2 J(t,μ)(x,x ) ). From the expressions of MsM_s h and Ms,M_s D, h, we obtain that d(Ms−Ms,)= d(M_s h-M_s D, h)= e−βsd(J^(s,μs)−J^(s,μs,))−βe−βs(J^(s,μs)−J^(s,μs,))ds e^-β sd ( J(s, _s h)- J(s, _s D, h) )-β e^-β s( J(s, _s h)- J(s, _s D, h))ds +e−βs(r^(s,μs)−r^(s,μs,)−q^0(s,μs,)+q^0(s,μs,,)) +e^-β s ( r_ h(s, _s h)- r_ h(s, _s D, h)- q^0(s, _s h, h)+ q^0(s, _s D, h, h) ) By applying Itô’s formula to d(J^(s,μs)−J^(s,μs,))d( J(s, _s h)- J(s, _s D, h)), we can rewrite d(Ms−Ms,)d(M_s h-M_s D, h) by d(Ms−Ms,)= d(M_s h-M_s D, h)= e−βs(Δq0(s,μs,)−e¯e[ℒJ^(s,Xs,,X¯s,,μs,,aδ(s),a¯δ(s))+r^(s,μs,) e^-β s ( q^0(s, _s h, h)-E^e E^e [ L J(s,X_s D, h, X_s D, h, _s D, h,a_δ(s), a_δ(s))+ r_ h(s, _s D, h) −q^0(s,μs,,)|s])ds+e−βs(e[∂μJ^(s,μs)(Xs)σo,(s,Xs,μs)|s] - q^0(s, _s D, h, h) | G_s ] )ds+e^-β s (E^e [ _μ J(s, _s h)(X_s h) _o, h(s,X_s h, _s h) | G_s ] −e[∂μJ^(s,μs,)(Xs,)σo(s,Xs,,μs,,aδ(s))|s])dBs. -E^e [ _μ J(s, _s D, h)(X_s D, h) _o(s,X_s D, h, _s D, h,a_δ(s)) | G_s ] )dB_s. Noting that aδ(s)∼(⋅|δ(s),Xδ(s),,μδ(s),)a_δ(s) h(·|δ(s),X_δ(s) D, h, _δ(s) D, h) and using the tower property, we have d(Ms−Ms,)= d(M_s h-M_s D, h)= e−βs(Δq0(s,μs,)−Δq0(s,μs,,)−(qˇ0(δ(s),μδ(s),,)−qˇ0(s,μs,,)) e^-β s ( q^0(s, _s h, h)- q^0(s, _s D, h, h)-( q^0(δ(s), _δ(s) D, h, h)- q^0(s, _s D, h, h)) −(r^(s,μs,)−r^(δ(s),μδ(s),))−e¯e[ℒJ^(s,Xs,,X¯s,,μs,,aδ(s),a¯δ(s)) -( r_ h(s, _s D, h)- r_ h(δ(s), _δ(s) D, h))-E^e E^e [ L J(s,X_s D, h, X_s D, h, _s D, h,a_δ(s), a_δ(s)) −ℒJ^(δ(s),Xδ(s),,X¯δ(s),,μδ(s),,aδ(s),a¯δ(s))|s])ds - L J(δ(s),X_δ(s) D, h, X_δ(s) D, h, _δ(s) D, h,a_δ(s), a_δ(s)) | G_s ] )ds +e−βs(e[∂μJ^(s,μs)(Xs)σo,(s,Xs,μs)|s] +e^-β s (E^e [ _μ J(s, _s h)(X_s h) _o, h(s,X_s h, _s h) | G_s ] −e[∂μJ^(s,μs,)(Xs,)σo(s,Xs,,μs,,aδ(s))|s])dBs -E^e [ _μ J(s, _s D, h)(X_s D, h) _o(s,X_s D, h, _s D, h,a_δ(s))| G_s ] )dB_s =: =: e−βs(Isds+IIsdBs). e^-β s (I_s hds+I_s hdB_s ). It suffices to estimate IsI_s h and IIsII_s h. By Assumptions 2.1 in Ren et al. (2026) and standard estimates on Xs,X_s D, h, it holds that e[|Xs,−Xsi,|2]+e[22(μsi,,μs,)]≤C|s−si|≤C||,s∈[si,si+1], ^e [|X_s D, h-X_s_i D, h|^2 ]+E^e [ W_2^2( _s_i D, h, _s D, h) ]≤ C|s-s_i|≤ C| D|,s∈[s_i,s_i+1], e[sups∈[t,T]|Xs,|2]≤C. ^e[ _s∈[t,T]|X_s D, h|^2]≤ C. By Lemma 3.3 in Ren et al. (2026) with f(⋅)=Δq0(⋅,)f(·)= q^0(·, h), we derive that for s∈[si,si+1]s∈[s_i,s_i+1] e[|Is|2]≤C||E^e[|I_s h|^2]≤ C| D|. In addition, by Assumption 2.1, it holds that e[∫tT|IIs|2s]≤Ce[∫tT(1+|Xs,|2+‖μs,‖22+|Xs|2+‖μs‖22)s]<+∞, ^e [ _t^T|I_s h|^2ds ]≤ CE^e [ _t^T (1+|X_s D, h|^2+\| _s D, h\|_2^2+|X_s h|^2+\| _s h\|_2^2 )ds ]<+∞, and hence e[∫0TηsIIsBs]=0E^e[ _0^T _s hI_s hdB_s]=0. It then follows that |e[∫tTηsd(Ms−Ms,)]|≤ |E^e [ _t^T _s hd(M_s h-M_s D, h) ] |≤ (e[∫tT|ηs|2s])1/2(∑i=0n−1∫sisi+1e[|Is|2]s)1/2 (E^e [ _t^T| _s h|^2ds ] )^1/2 ( _i=0^n-1 _s_i^s_i+1E^e [|I_s h|^2 ]ds )^1/2 ≤ ≤ sup∈Π(e[∫tT|ηs|2s])1/2C||1/2T1/2. _ h∈ (E^e [ _t^T| _s h|^2ds ] )^1/2C| D|^1/2T^1/2. We thus conclude that lim→0|e[∫tTηsd(Ms−Ms,)]|=0 _ D→ 0 |E^e [ _t^T _s hd(M_s h-M_s D, h) ] |=0. ∎ 4 Continuous-Time q-Learning Algorithms This section contributes to the design of q-learning algorithms based on the theoretical results in previous sections. We first devise and present a learning algorithm to learn the optimal value function and the optimal Iq-function when the optimal policy as a two-layer fixed point (3.5) can be derived in an explicit form. As a result, the optimal policy shares the same parameters as that of the optimal Iq-function. In general, however, it could be difficult to explicitly solve the two-layer fixed point (3.5). Therefore, we need to parameterize the optimal value function, the optimal Iq-function and the optimal policy by distinct parameters respectively, which motivates us to devise the Actor-Critic q-learning algorithm that alternately updates the Critic (the value function and the Iq-function) and the Actor (the policy). The main differences between the optimal q-learning and the Actor-Critic q-learning are illustrated in Figures 4 and 4. Value function: JθJ^θ Iq-function: q0,ψq^0,ψ Policy: πψπ^ψ Environment (2) Samples (μtkm,r^tkm,μtk+1m)k(μ^m_t_k, r^m_t_k,μ^m_t_k+1)_k(1) Test policy hϕ~m h φ^m(3) Averaged martingale orthogonal condition (a) Procedure of optimal q-learning algorithm Policy πϕ π^φ Value function: JθJ^θ Iq-function: q0,ψq^0,ψ Environment (1) Test policy hϕ~m h φ^m(2) Samples (μtkm,r^tkm,μtk+1m)k( _t_k^m, r_t_k^m, _t_k+1^m)_k(4) Policy improvement(3) Averaged martingale orthogonal conditionActorCritic (b) Procedure of Actor-Critic q-learning algorithm 4.1 Optimal q-learning algorithm In this section, we apply Theorem 3.2 to learn directly the optimal value function J∗J^* and the optimal Iq-function q0,∗q^0,*. Let JθJ^θ, q0,ψq^0,ψ and ψ π^ψ, (θ,ψ)∈Θ×Ψ(θ,ψ)∈ × , be respectively the parameterized approximators of J∗J^*, q0,∗q^0,* and ∗ π^* such that the consistency condition Jθ(T,μ)=g^(μ)J^θ(T,μ)= g(μ) and the two-layer fixed point condition (3.5) hold at the same time. This happens when we take advantage of the model structure to obtain the explicit form of parameterized function family. The rest is to ensure the martingale condition in (3.2). In contrast to Jia and Zhou (2022b), the feature of mean-field interactions requires us to explore all test policies h to meet the martingale condition. Therefore, the martingale orthogonal condition should hold in the average sense, see section 5 in Wei et al. (2024). It is impossible to use all test policies in the practical implementation. In response, we take a family of parameterized test policies ΠΦ~=ϕ~:ϕ~∈Φ~ = \ h φ: φ∈ \ for some bounded set Φ~⊂ℝLϕ~ ^L_ φ. Denote Mtϕ~,θ,ψ:= M_t h φ,θ,ψ:= e−βtJθ(t,μtϕ)+∫0te−βu[r^ϕ~(u,μuϕ~)−q0,ψ(u,μuϕ~,ϕ~)]u, e^-β tJ^θ(t, _t h^φ)+ _0^te^-β u [ r_ h φ(u, _u h φ)-q^0,ψ(u, _u h φ, h φ) ]du, Mt,ϕ,θ,ψ:= M_t D, h^φ,θ,ψ:= e−βtJθ(t,μt,ϕ~)+∫0te−βu[r^ϕ(u,μu,ϕ~)−q0,ψ(u,μu,ϕ~,ϕ~)]u. e^-β tJ^θ(t, _t D, h φ)+ _0^te^-β u [ r_ h^φ(u, _u D, h φ)-q^0,ψ(u, _u D, h φ, h φ) ]du. According to Theorem 3.2, if there exists a pair (θ∗,ψ∗)∈Θ×Ψ(θ^*,ψ^*)∈ × such that Mtϕ~,θ,ψt\M_t h φ,θ,ψ\_t is a (ℙe,)(P^e,G)-martingale for any ϕ~∈ΠΦ~ h φ∈ , then Jθ∗J^θ^* and q0,ψ∗q^0,ψ^* are the optimal value function and the optimal unregularized Iq-function restricted to ΠΦ~ . A necessary and sufficient condition for the parametrized process Mtϕ~,θ∗,ψ∗t\M_t h φ,θ^*,ψ^*\_t to be a (ℙe,)(P^e,G)-martingale is the following averaged martingale orthogonality condition 0=∫Φ~(e[∫0Tηtϕ~Mtϕ~,θ∗,ψ∗])(dϕ~), 0= _ (E^e [ _0^T _t φdM_t h φ,θ^*,ψ^* ] ) U(d φ), (4.1) where ηϕ~η φ is any G-progressively measurable test process, and U stands for the uniform distribution on the space Φ~ . Recall that Mtϕ~,θ,ψt\M_t h φ,θ,ψ\_t is not accessible to the social planner and therefore cannot be used for the learning the parameters (θ,ψ)(θ,ψ). We thus obtain the following convergence result when the learning of (θ,ψ)(θ,ψ) is based on the observable process Mt,ϕ~,θ,ψt\M_t D, h φ,θ,ψ\_t. Proposition 4.1. Let (θ,∗,ψ,∗)(θ D,*,ψ D,*) be the solution to ∫Φ~(e[∫0Tηtϕ~Mt,ϕ~,θ,ψ])(dϕ~)=0 _ (E^e [ _0^T _t φdM_t D, h φ,θ,ψ ] ) U(d φ)=0. As ||→0| D|→ 0, any convergent subsequence of (θ,∗,ψ,∗)(θ D,*,ψ D,*) converges to the solution (θ∗,ψ∗)(θ^*,ψ^*) of (4.1). Proof. By Theorem 3.4, lim||→0∫Φ~|e[∫0Tηtϕ~d(Mt,ϕ~,θ,ψ−Mtϕ~,θ,ψ)]|(dϕ~)=0 _| D|→ 0 _ |E^e [ _0^T _t φd (M_t D, h φ,θ,ψ-M_t h φ,θ,ψ ) ] | U(d φ)=0. Combining this with Lemma 1 in Jia and Zhou (2022b) yields the desired result. ∎ Using the temporal difference algorithm, we have the following updating rules of θ and ψ: θ←θ+αθ∫Φ~e[∫0Tηtϕ~Mt,ϕ~,θ,ψ](dϕ~), θ←θ+ _θ _ E^e [ _0^T _t φdM_t D, h φ,θ,ψ ] U(d φ), (4.2) ψ←ψ+αψ∫Φ~e[∫0Tζtϕ~Mt,ϕ~,θ,ψ](dϕ~), ψ←ψ+ _ψ _ E^e [ _0^T _t φdM_t D, h φ,θ,ψ ] U(d φ), (4.3) where αθ _θ and αψ _ψ are learning rates of θ and ψ, respectively. In particular, we choose the conventional test functions ηtϕ~=∂Jθ∂θ(t,μt,ϕ~),andζtϕ~=∂q0,ψ∂ψ(t,μt,ϕ~,ϕ~). _t φ= ∂ J^θ∂θ (t, _t D, h φ ),\;and _t φ= ∂ q^0,ψ∂ψ (t, _t D, h φ, h φ ). (4.4) Similar to Wei et al. (2024), in numerical implementation we approximate the integral over the test policy parameter ϕ~ φ in (4.4) by a Monte Carlo method and choose a sequence of test policies ϕ~1 h φ_1, …, ϕ~M h φ_M with the same form yet different parameters from ψ π^ψ. Here, ϕ~1,…,ϕ~M φ_1,…, φ_M are i.i.d. drawn from ψ⋅(p(n),q(n))ψ· U(p(n),q(n)). Recall that the social planner observes the conditional state distribution μs,ϕ~ms\ _s D, h φ^m\_s and the aggregated reward. When the sample data (μtkm,r^tkm,μtk+1m)k( _t_k^m, r_t_k^m, _t_k+1^m)_k generated by ϕ~m h φ_m with tk=kΔt_k=k t, 0≤k≤K−10≤ k≤ K-1, is available, we obtain a discretized version of martingale orthogonal condition (4.2)-(4.3) and the update rules of θ and ψ with test functions (4.4) are written as θ←θ+αθ1M∑m=1MΔmθ,ψ←ψ+αψ1M∑m=1MΔmψ, θ←θ+ _θ 1M _m=1^M ^mθ,\;ψ←ψ+ _ψ 1M _m=1^M ^mψ, (4.5) where Gtk:TmG_t_k:T^m, Δmθ ^mθ and Δmψ ^mψ are defined by Gtk:Tm G_t_k:T^m :=Jθ(tk+1,μtk+1m)−Jθ(tk,μtkm)+(r^tkm−βJθ(tk,μtkm)−q0,ψ(tk,μtkm,ϕ~m))Δt, :=J^θ(t_k+1, _t_k+1^m)-J^θ(t_k, _t_k^m)+ ( r_t_k^m-β J^θ(t_k, _t_k^m)-q^0,ψ(t_k, _t_k^m, h φ_m) ) t, Δmθ ^mθ :=∑k=0K−1e−βtk∂Jθ∂θ(tk,μtkm)Gtk:Tm,Δmψ:=∑k=0K−1e−βtk∂q0,ψ∂ψ(tk,μtkm,ϕ~m)Gtk:Tm. := _k=0^K-1e^-β t_k ∂ J^θ∂θ(t_k, _t_k^m)G_t_k:T^m,\; ^mψ:= _k=0^K-1e^-β t_k ∂ q^0,ψ∂ψ(t_k, _t_k^m, h φ_m)G_t_k:T^m. The offline optimal q-learning algorithm is reported in Algorithm 1. Algorithm 1 Offline Optimal q-Learning Algorithm Inputs: initial conditional state distribution μ0 _0, horizon T, time step Δt t, number of time steps K, number of test policies M, initial learning rates αθ _θ and αψ _ψ, functional forms of parameterized value function Jθ(⋅,⋅)J^θ(·,·) satisfying Jθ(T,⋅)=g^(⋅)J^θ(T,·)= g(·), parameterized (unregularized) Iq-function q0,ψ(⋅,⋅,⋅)q^0,ψ(·,·,·). Required program: Environment simulator (μ′,r^)=EnvironmentΔt(t,μ,)(μ , r)=Environment_ t(t,μ, h) that takes current time–state distribution pair (t,μ)(t,μ) and the policy h as inputs and generates conditional state distribution μ′μ at time t+Δt+ t and the aggregated reward r r at time t as outputs. Learning procedure: Initialize θ and ψ. for episode n=1n=1 to N do Observe the initial state distribution μ0 _0 and store μtkm←μ0 _t_k^m← _0. for m=1m=1 to M do Draw ϕ~m φ^m from ψ⋅([p(n),q(n)])ψ·U([p(n),q(n)]) and set the test policy ϕ~m h φ^m. Initialize k=0k=0. while k<Kk<K do Apply the test policy ϕ~m h φ^m to the simulator (μ,r^)=EnvironmentΔt(tk,μtkm,ϕ~m)(μ, r)=Environment_ t(t_k, _t_k^m, h φ^m), and observe the new state distribution μ and the aggregated reward r r as output. Store μtk+1m←μ _t_k+1^m←μ and r^tkm←r r_t_k^m← r. Update k←k+1k← k+1. end while end forUpdate θ and ψ by θ←θ+αθ1M∑m=1MΔmθ,ψ←ψ+αψ1M∑m=1MΔmψθ←θ+ _θ 1M _m=1^M ^mθ,ψ←ψ+ _ψ 1M _m=1^M ^mψ by (4.5). end for return θ and ψ. 4.2 Actor-Critic q-learning algorithm To cope with the general case where the optimal policy as the two-layer fixed point to (2.7) may not admit an explicit form, we devise in this subsection an Actor-Critic algorithm. Specifically, we rely on Proposition 2.2 (i) to design the Actor step for policy improvement (PI) to update the policy and utilize Theorem 3.1 to design the Critic step for policy evaluation (PE) to update the value function and the Iq-function. Actor Step (or PI step): Let us consider a parameterized policy class ΠΦ=ϕ:ϕ∈Φϕ∈Φ =\ π^φ:φ∈ \_φ∈ . Let ϕnφ^n be the value of ϕφ at the episode n∈ℕn . By Proposition 2.2 (i), if we update it to a new policy ϕ π^φ such that e[∫0Te−βtqγ(t,μtϕ,ϕn;ϕn)t]≤e[∫0Te−βtqγ(t,μtϕ,ϕ;ϕn)t], ^e [ _0^Te^-β tq^γ(t, _t π^φ, π^φ^n; π^φ^n)dt ] ^e [ _0^Te^-β tq^γ(t, _t π^φ, π^φ; π^φ^n)dt ], (4.6) it then holds that J(0,μ;ϕ)≥J(0,μ;ϕn)J(0,μ; π^φ)≥ J(0,μ; π^φ^n). However, μtϕt\ _t π^φ\_t is not known at the episode n in advance. We approximate μtϕ _t π^φ by μ^tn=(1−ρn)μtϕn+ρnμtϕ~0n μ_t^n=(1-ρ^n)μ π^φ^n_t+ρ^n _t h φ_0^n, where μtϕn _t π^φ^n and μtϕ~0n _t h φ_0^n are conditional state distribution generated by ϕn π^φ^n and ϕ~0n h φ_0^n, respectively, ϕ~0n φ^n_0 is drawn from ϕn⋅(p(n),q(n))φ^n· U(p(n),q(n)), and ρn∈(0,1)ρ^n∈(0,1) balances the weighting between the current estimate and the noise using the test policy ϕ~0n h φ_0^n. It is sufficient to solve the following optimization problem at the episode n maxϕ∈Φe[∫0Te−βtqγ(t,μ^tn,ϕ;ϕn)t], _φ∈ E^e [ _0^Te^-β tq^γ(t, μ_t^n, π^φ; π^φ^n)dt ], (4.7) and to update ϕφ using stochastic gradient ascent (SGA) rule. The gradient of the objective function in (4.7) is given below. Lemma 4.2. We have ∇ϕe[∫0Te−βtqγ(t,μ^tn,ϕ;ϕn)t] _φE^e [ _0^Te^-β tq^γ(t, μ_t^n, π^φ; π^φ^n)dt ] (4.8) = = e[∫0Te−βt∫ℝd×[δq0δ(t,μ^tn,ϕ;ϕn)(x,a)−γlogϕ(a|t,x,μ^tn)∇ϕlogϕ(a|t,x,μ^tn)]ϕ(a|t,x,μ^tn)aμ^tn(dx)]. ^e [ _0^Te^-β t _R^d× A [ \ δ q^0δ h(t, μ_t^n, π^φ; π^φ^n)(x,a)-γ π^φ(a|t,x, μ_t^n) \ _φ π^φ(a|t,x, μ_t^n) ] π^φ(a|t,x, μ_t^n)da μ_t^n(dx) ]. Suppose that the sample data μ^tkn0≤k≤K−1\ μ_t_k^n\_0≤ k≤ K-1 generated by ϕn π^φ^n and ϕ~0n h φ^n_0 are available. Based on Lemma 4.2, ϕφ is updated incrementally by SGA with L−1L-1 inner iterations, L≥1L≥ 1, that ϕn,l+1 φ^n,l+1 =ϕn,l+αϕn,lG^ϕn,ln. =φ^n,l+ _ _n,l G_φ^n,l^n. (4.9) where ϕn,0:=ϕnφ^n,0:=φ^n, αϕ _φ is the learning rate, and G^ϕn G_φ^n is a stochastic estimation of (4.8) that G^ϕn G_φ^n =−∑k=0K−1e−βtk∫ℝd×δq0,ψnδ(tk,μ^tkn,ϕ)(x,a)−γlogϕ(a|tk,x,μ^tkn) =- _k=0^K-1e^-β t_k _R^d× A \ δ q^0,ψ^nδ h(t_k, μ_t_k^n, π^φ)(x,a)-γ π^φ(a|t_k,x, μ_t_k^n) \ ∇ϕlogϕ(a|tk,x,μ^tkn)ϕ(a|tk,x,μ^tkn)daμ^tkn(dx). \;\;\; _φ π^φ(a|t_k,x, μ_t_k^n) π^φ(a|t_k,x, μ_t_k^n)da μ_t_k^n(dx). (4.10) Remark 4.3. It is expected that when L is sufficiently large, ϕn,L−2 π^φ^n,L-2 will get very close to the maximizer of qγ(⋅;ϕn)q^γ(·; π^φ^n). Due to the fact that the approximator of qγq^γ with respect to ϕφ is sophisticated and non-concave in general, the convergence of SGA (4.9) for the optimization problem (4.7) becomes complicated to check. We therefore analyze the convergence of gradient ascent (4.9) for the LQ-MFC problem over an infinite horizon in section 5, which may serve as a preliminary step towards a deeper understanding of (4.7) in the general framework in our future study. In two examples in Section 6, our simulation experiments based on the inner iteration (4.9) also show the satisfactory performance. On the other hand, it is expected that limn→∞qγ(t,μ,ϕn+1;ϕn)=0 _n→∞q^γ(t,μ, π^φ^n+1; π^φ^n)=0. Hence we shall modify the objective function in (4.7) by adding another loss function, called the consistency loss function, to enforce the consistency condition qγ,∗(t,μ;∗)=0q^γ,*(t,μ; π^*)=0 when n is large that maxϕ∈Φ∫0Te−βt(woqγ(t,μ^tn,ϕ;ϕn)−wc2|qγ(t,μ^tn,ϕ;ϕn)|2)t, _φ∈ _0^Te^-β t (w_oq^γ(t, μ_t^n, π^φ; π^φ^n)- w_c2|q^γ(t, μ_t^n, π^φ; π^φ^n)|^2 )dt, (4.11) where the parameters wo≥0w_o≥ 0 and wc≥0w_c≥ 0 stand for weights between the optimality loss function and the consistency loss function. At the beginning of the training (n is small), we shall focus on improving the value function by setting wo≫wcw_o w_c. After sufficiently many iterations (n is large), we then set a large wc≫wow_c w_o to enforce the consistency condition limn→∞qγ(t,μ,ϕn+1;ϕn)=0 _n→∞q^γ(t,μ, π^φ^n+1; π^φ^n)=0: during this period, the distance between ϕn+1 π^φ^n+1 and ϕn π^φ^n becomes small and thus qγ(t,μ^tn,ϕn+1;ϕn)q^γ(t, μ_t^n, π^φ^n+1; π^φ^n) is approximately equal to qγ(t,μ^tn,ϕn+1;ϕn+1)q^γ(t, μ_t^n, π^φ^n+1; π^φ^n+1). Therefore, at each episode, after running (L−1)(L-1) inner iterations of (4.9), we also run one-step SGA for (4.11) that ϕn+1=ϕn,L−2+αϕn,L−2G^ϕn,L−2n, φ^n+1=φ^n,L-2+ _ _n,L-2 G_φ^n,L-2^n, (4.12) where G^ϕ G_φ is a stochastic estimation of the gradient of the objective function in (4.11) that G^ϕn G_φ^n =−∑k=0K−1e−βtk(wo−wcq0,ψ(tk,μ^tkn,ϕ))∫ℝd×δq0,ψδ(tk,μ^tkn,ϕ)(x,a) =- _k=0^K-1e^-β t_k (w_o-w_cq^0,ψ(t_k, μ_t_k^n, π^φ) ) _R^d× A \ δ q^0,ψδ h(t_k, μ_t_k^n, π^φ)(x,a) −γlogϕ(a|tk,x,μ^tkn)∇ϕlogϕ(a|tk,x,μ^tkn)ϕ(a|tk,x,μ^tkn)daμ^tkn(dx). \;\;\;-γ π^φ(a|t_k,x, μ_t_k^n) \ _φ π^φ(a|t_k,x, μ_t_k^n) π^φ(a|t_k,x, μ_t_k^n)da μ_t_k^n(dx). (4.13) Remark 4.4. The gradients in (4.10) and (4.13) involve the computation of the partial linear functional derivative δq0,ψδ δ q^0,ψδ h. In general, one can use cylindrical neural network functions (see Pham and Warin (2024, 2023)) to approximate the Iq-function such that q0,ψ(t,μ,)=Φ(t,⟨Ψ,μ⋅⟩), q^0,ψ(t,μ, h)= (t, ,μ· h ), (4.14) where Φ:[0,T]×ℝk→ℝ :[0,T]×R^k and Ψ:ℝd×→ℝk :R^d× A ^k are two neural networks. It follows from the definition of partial linear functional derivative (c.f.Ren et al. (2026)) that δq0,ψδ(t,μ,)(x,a)=∂yΦ(t,⟨Ψ,μ⋅⟩)Ψ(x,a) δ q^0,ψδ h(t,μ, h)(x,a)= _y (t, ,μ· h ) (x,a). Furthermore, the involved integrals with respect to h and μ in (4.10) and (4.14) could be computed approximately with samples drawn from h and μ as in Pham and Warin (2023). Critic Step (or PE step): Given the updated policy ϕ π^φ from the Actor step, the Critic step is essentially the same as the one in Algorithm 1 based on the averaged martingale orthogonality condition to update the value function JθJ^θ and the Iq-function q0,ψq^0,ψ using iterations in (4.5) except that the parameters of the test policies ϕ~1,…,ϕ~M φ^1,…, φ^M are now i.i.d. drawn from ϕ⋅(p(n),q(n))φ· U(p(n),q(n)). In summary, for each episode n, the Actor-Critic q-learning algorithm consists of two steps, whose pseudo code is given in Algorithm 2: (i) The PE step that estimates the value function J(;ϕ)J(; π^φ) and the Iq-function q0(;ϕ)q^0(; π^φ) via the temporal difference algorithm (4.5), where J(;ϕ)J(; π^φ) and q0(;ϕ)q^0(; π^φ) are estimated by parameterized function classes Jθ:θ∈Θ\J^θ:θ∈ \ and q0,ψ:ψ∈Ψ\q^0,ψ:ψ∈ \, respectively; (i) The PI step that has L inner iterations and updates the parameter ϕφ of the policy ϕ π^φ using a stochastic version G^ϕ G_φ of GϕG_φ in (4.13), where q0(;ϕ)q^0(; π^φ) is replaced by the corresponding estimator q0,ψq^0,ψ. Algorithm 2 Offline Actor-Critic q-Learning Algorithm Inputs: initial state distribution μ0 _0, horizon T, time step Δt t, number of mesh grids K, number of test policies M, the number of inner iterations for the Actor L, learning rates αθ _θ, αψ _ψ and αϕ _φ, the parameterized value function Jθ(⋅,⋅)J^θ(·,·) satisfying Jθ(T,⋅)=g^(⋅)J^θ(T,·)= g(·), the parameterized (unregularized) Iq-function q0,ψ(⋅,⋅,⋅)q^0,ψ(·,·,·), and parametrized policy ϕ π^φ. Required program: Environment simulator (μ′,r^)=EnvironmentΔt(t,μ,)(μ , r)=Environment_ t(t,μ, h) that takes current time–state distribution pair (t,μ)(t,μ) and the policy h as inputs and generates state distribution μ′μ at time t+Δt+ t and the aggregated reward r r at time t as outputs. Learning procedure: Initialize θ, ψ and ϕφ. for episode n=1n=1 to N do Observe the initial state distribution μ0 _0 and store μtkm←μ0 _t_k^m← _0. for m=1m=1 to M do Draw ϕ~m φ^m from ϕ⋅([p(n),q(n)])φ·U([p(n),q(n)]) and set the test policy ϕ~m h φ^m. for k=0k=0 to K−1K-1 do Apply the test policy ϕ~m h φ^m, 0≤m≤M0≤ m≤ M, to the simulator (μ,r^)=EnvironmentΔt(tk,μtkm,ϕ~m)(μ, r)=Environment_ t(t_k, _t_k^m, h φ^m), and observe the new state distribution μ and the aggregated reward r r as output. Store μtk+1m←μ _t_k+1^m←μ and r^tkm←r r_t_k^m← r. Apply the policy ϕ π^φ to the environment simulator and store μ^tk+1←μ μ_t_k+1←μ. end for end for Update θ and ψ in PE by θ←θ+αθ1M∑m=1MΔmθ,ψ←ψ+αψ1M∑m=1MΔmψθ←θ+ _θ 1M _m=1^M ^mθ,ψ←ψ+ _ψ 1M _m=1^M ^mψ by (4.5). for iteration l=0l=0 to L−1L-1 do Update ϕφ in PI by ϕ←ϕ+αϕG^ϕφ←φ+ _φ G_φ according to (4.9). end for Update ϕφ by ϕ←ϕ+αϕG^ϕφ←φ+ _φ G_φ according to (4.12). end for return θ, ψ and ϕφ. 5 Inner Iterations in LQ-MFC Over an Infinite Horizon In this section, we discuss the convergence of the gradient ascent iterations for the optimization problem (4.7) in the framework of LQ-MFC problems over an infinite horizon. A theoretical convergence or regret analysis of the Actor-Critic algorithm for LQ-MFC problem, as well as for general MFC problem over a finite horizon remains challenging and is left for future work. 5.1 Infinite-horizon LQ-MFC problems Let us consider a LQ-MFC problem over an infinite horizon with =ℝp A=R^p and assume n=m=1n=m=1 for simplicity. The coefficients of dynamics and the reward function are given by b(x,μ,a) b(x,μ,a) =B(x−μ¯)+(B+B¯)μ¯+Ca, =B(x- μ)+(B+ B) μ+Ca, σ(x,μ,a) σ(x,μ,a) =D(x−μ¯)+(D+D¯)μ¯+Fa, =D(x- μ)+(D+ D) μ+Fa, σo(x,μ,a) _o(x,μ,a) =Do(x−μ¯)+(Do+D¯o)μ¯+Foa, =D_o(x- μ)+(D_o+ D_o) μ+F_oa, r(x,μ,a) r(x,μ,a) =x⊺Mx+μ¯⊺M¯μ¯+a⊺Ra. =x Mx+ μ M μ+a Ra. Here B, B¯ B, D, D¯ D, DoD_o, D¯o D_o, M and M¯ M are deterministic matrices valued in ℝd×dR^d× d, C,F,FoC,F,F_o are deterministic matrices valued in ℝd×pR^d× p, R are deterministic matrices valued in ℝp×pR^p× p. We may assume without loss of generality that M, M¯ M and R are symmetric matrices. Let μ¯=∫ℝdxμ(dx) μ= _R^dxμ(dx) and Var(μ)(Λ)=∫ℝd(x−μ¯)⊺Λ(x−μ¯)μ(dx) Var(μ)( )= _R^d(x- μ) (x- μ)μ(dx) denote the mean and variance of μ, respectively, for a symmetric matrix Λ∈ℝd×d ^d× d. We have the explicit expressions for the optimal value function and the optimal policy. The proof of Theorem 5.1 is similar as that of Theorem 5.1 in Ren et al. (2026) and is thus omitted. Theorem 5.1. Under the condition M⪯0,M+M¯⪯0,R⪯−δIqM 0,\;M+ M 0,\;R -δ I_q, the optimal value function J∗J^* admits the form J∗(μ)=Var(μ)(Λ∗)+μ¯⊺Γ∗μ¯+χ∗, J^*(μ)= Var(μ)( ^*)+ μ ^* μ+χ^*, (5.1) and the optimal policy ∗ π^* is a Gaussian policy that ∗(⋅|x,μ)=(−(U∗)−1S∗(x−μ¯)−(V∗)−1Z∗μ¯,−γ2(U∗)−1), π^*(·|x,μ)= N (-(U^*)^-1S^*(x- μ)-(V^*)^-1Z^* μ,- γ2(U^*)^-1 ), with Λ∗,Γ∗ ^*, ^* and χ∗χ^* satisfying −βΛ∗+M+D⊺Λ∗D+Do⊺Λ∗Do+B⊺Λ∗+Λ∗B−(S∗)⊺(U∗)−1S∗=0, -β ^*+M+D ^*D+D_o ^*D_o+B ^*+ ^*B-(S^*) (U^*)^-1S^*=0, (5.2) −βΓ∗+M+M¯+(D+D¯)⊺Λ∗(D+D¯)+(D¯o+Do)⊺Γ∗(D¯o+Do) -β ^*+M+ M+ (D+ D) ^* (D+ D)+( D_o+D_o) ^*( D_o+D_o) (5.3) +(B+B¯)⊺Γ∗+Γ∗(B+B¯)−(Z∗)⊺(V∗)−1Z∗=0, 85.35826pt+(B+ B) ^*+ ^*(B+ B)-(Z^*) (V^*)^-1Z^*=0, −βχ∗+γp2log(γπ)−γ2log(det(−U∗))=0, -βχ^*+ γ p2 (γπ )- γ2 ( det(-U^*) )=0, (5.4) and U:=UΛU:=U_ , V:=VΓ,ΛV:=V_ , , S:=SΛS:=S_ , Z:=ZΓ,ΛZ:=Z_ , , with U=F⊺ΛF+Fo⊺ΛFo+R,V=F⊺ΛF+Fo⊺ΓFo+R,S=C⊺Λ+F⊺ΛD+Fo⊺ΛDo,Z=C⊺Γ+F⊺Λ(D+D¯)+Fo⊺Γ(Do+D¯o). \ array[]lU&=F F+F_o F_o+R,\\ V&=F F+F_o F_o+R,\\ S&=C +F D+F_o D_o,\\ Z&=C +F (D+ D )+F_o (D_o+ D_o ). array . (5.9) By Theorem 5.1, the optimal policy is Guassian. From this, we restrict our attention to the policy set parameterized with the parameter ϕ=(K,K¯,Σ)φ=(K, K, ) Π=ϕ∈ℝp×d×ℝp×d×ℝd×d:ϕ(⋅|x,μ¯)=(K(x−μ¯)+K¯μ¯,Σ),Σ≻0,Σ⊺=Σ. = \φ ^p× d×R^p× d×R^d× d: π^φ(·|x, μ)= N(K(x- μ)+ K μ, ),\; 0,\; = \. Given any parameter ϕ=(K,K¯,Σ)∈Πφ=(K, K, )∈ , we can easily obtain the expression of J(μ;ϕ)J(μ; π^φ) by applying the dynamic programming equation in (2.5) as well as the explicit expression of Iq-function in the next result. Lemma 5.2. The value function J(⋅;ϕ)J(·; π^φ) takes the form J(μ;ϕ)=Var(μ)(ΛK)+μ¯⊺ΓK¯μ¯+χΣJ(μ; π^φ)= Var(μ)( _K)+ μ _ K μ+ _ , where ΛK _K, ΓK¯ _ K and χΣ _ satisfy −βΛK+D⊺ΛKD+Do⊺ΛKDo+B⊺ΛK+ΛKB+M+2SK⊺K+K⊺UKK=0, -β _K+D _KD+D_o _KD_o+B _K+ _KB+M+2S_K K+K U_KK=0, (5.10) −βΓK¯+(D+D¯)⊺ΛK(D+D¯)+(Do+D¯o)⊺ΓK¯(Do+D¯o)+(B+B¯)⊺ΓK¯ -β _ K+ (D+ D) _K (D+ D)+(D_o+ D_o) _ K(D_o+ D_o)+(B+ B) _ K +ΓK¯(B+B¯)+M+M¯+2ZK,K¯⊺K¯+K¯⊺VK,K¯K¯=0, + _ K(B+ B)+M+ M+2Z_K, K K+ K V_K, K K=0, (5.11) −βχΣ+γp2(log(2π)+1)+γ2log(det(Σ))+Tr(UΣ)=0. -β _ + γ p2 ( (2π)+1 )+ γ2 ( det( ) )+ Tr (U )=0. (5.12) In addition, for any ϕ~∈Π h φ∈ with ϕ~=(K~,K¯~,Σ~) φ=( K, K, ), Iq-function qγ(μ,;ϕ)q^γ(μ, h; π^φ) takes the form qγ(μ,ϕ~;ϕ)=q1γ,ϕ(μ,K~)+q2γ,ϕ(μ,K¯~)+q3γ,ϕ(Σ~), q^γ(μ, h φ; π^φ)=q^γ,φ_1(μ, K)+q^γ,φ_2(μ, K)+q^γ,φ_3( ), (5.13) where ΛK _K, ΓK¯ _ K and χΣ _ satisfy (5.10)-(5.12), q1ϕ(K~)q^φ_1( K), q2ϕ(K¯~)q^φ_2( K) and q3ϕ(Σ~)q^φ_3( ) are respectively given by q1γ,ϕ(μ,K~)= q^γ,φ_1(μ, K)= Var(μ)(−βΛK+D⊺ΛKD+Do⊺ΛKDo+B⊺ΛK+ΛKB+M+2SK⊺K~+K~⊺UKK~), Var(μ) (-β _K+D _KD+D_o _KD_o+B _K+ _KB+M+2S_K K+ K U_K K ), q2γ,ϕ(μ,K¯~)= q^γ,φ_2(μ, K)= μ¯⊺(−βΓK¯+(D+D¯)⊺ΛK(D+D¯)+(Do+D¯o)⊺ΓK¯(Do+D¯o) μ (-β _ K+ (D+ D) _K (D+ D)+(D_o+ D_o) _ K(D_o+ D_o) +(B+B¯)⊺ΓK¯+ΓK¯(B+B¯)+M+M¯+2ZK,K¯⊺K¯~+K¯~⊺VK,K¯K¯)μ¯, +(B+ B) _ K+ _ K(B+ B)+M+ M+2Z_K, K K+ K V_K, K K ) μ, q3γ,ϕ(Σ~)= q^γ,φ_3( )= −βχΣ+γp2(log(2π)+1)+γ2log(det(Σ~))+Tr(UnΣ~). -β _ + γ p2 ( (2π)+1 )+ γ2 ( det( ) )+ Tr (U^n ). 5.2 Approximated Iq-function Note that qγ(μ,ϕ~;ϕ)q^γ(μ, h φ; π^φ) in (5.13) depends on ΛK _K, ΓK¯ _ K and χΣ _ , and hence depends on model parameters and it is unknown. To analyze the convergence of the inner actor iteration in the model-free setting, we need to approximate qγ(μ,ϕ;ϕn)q^γ(μ, π^φ; π^φ^n) by qγ,ψn(μ,ϕ)q^γ,ψ^n(μ, π^φ) for each fixed outer iteration n∈ℕn . By the structure of Iq-function in (5.13), we assume qγ,ψn(μ,ϕ)q^γ,ψ^n(μ, π^φ) takes the form, with ψn=(Λn,Γn,χn,Sn,Un,Zn,Vn)ψ^n=( ^n, ^n, χ^n,S^n,U^n,Z^n,V^n) qγ,ψn(μ,ϕ)=q1γ,ψn(μ,K)+q2γ,ψn(μ,K¯)+q3γ,ψn(Σ), q^γ,ψ^n(μ, π^φ)=q^γ,ψ^n_1(μ,K)+q^γ,ψ^n_2(μ, K)+q^γ,ψ^n_3( ), (5.14) where q1γ,ψn(μ,K) q^γ,ψ^n_1(μ,K) =Var(μ)(Λn+2SnK+K⊺UnK), = Var(μ) ( ^n+2S^nK+K U^nK ), q2γ,ψn(μ,K¯) q^γ,ψ^n_2(μ, K) =μ¯⊺(Γn+2(Zn)⊺K¯+K¯⊺VnK¯)μ¯, = μ ( ^n+2(Z^n) K+ K V^n K ) μ, q3γ,ψn(Σ) q^γ,ψ^n_3( ) =χn+γ2log(det(Σ))+Tr(UnΣ). = χ^n+ γ2 ( det( ) )+ Tr (U^n ). Recall that the inner iteration (4.7) for each fixed n∈ℕn is to solve the optimization problem: maxϕ∈Πqβ,γ,ψn(μ,ϕ): _φ∈ q^β,γ,ψ^n(μ, π^φ): =maxϕ∈Πe[∫0+∞e−βt(q1γ,ψn(μ^tn,K)+q2γ,ψn(μ^tn,K¯)+q3γ,ψn(Σ))t] = _φ∈ E^e [ _0^+∞e^-β t (q_1^γ,ψ^n( μ_t^n,K)+q_2^γ,ψ^n( μ_t^n, K)+q_3^γ,ψ^n( ) )dt ] (5.15) =:q1β,γ,ψn(μ,K)+q2β,γ,ψn(μ,K¯)+q3β,γ,ψn(Σ), =:q_1^β,γ,ψ^n(μ,K)+q_2^β,γ,ψ^n(μ, K)+q_3^β,γ,ψ^n( ), where μ^tn=(1−ρn)μtϕn+ρnμtϕ~0n μ_t^n=(1-ρ^n)μ π^φ^n_t+ρ^n _t h φ_0^n with the initial value μ^0n=μ μ_0^n=μ. The next result directly follows whose proof is omitted. Lemma 5.3. Assume that UnU^n and VnV^n are negative definite. For given ψnψ^n, the approximated Iq-function qβ,γ,ψn(μ,ϕ)q^β,γ,ψ^n(μ, π^φ) is concave in ϕφ. Moreover, the unique maximizer ϕn+1φ^n+1 in (5.15) is Kn+1=−(Un)−1Sn,K¯n+1=−(Vn)−1Zn,Σn+1=−γ2(Un)−1. K^n+1=-(U^n)^-1S^n,\; K^n+1=-(V^n)^-1Z^n,\; ^n+1=- γ2(U^n)^-1. (5.16) For any square matrix Z∈ℝm×mZ ^m× m, we denote σmin(Z) _ (Z), ‖Z‖\|Z\| and ‖Z‖F\|Z\|_F the minimum eigenvalue value, the spectral norm and Frobenius norm of Z , respectively. A function f:ℝm×n→ℝf:R^m× n is said to be Υ -smooth if it is differentiable and ‖∇f(X)−∇f(X′)‖F≤Υ‖X−X′‖F\|∇ f(X)-∇ f(X )\|_F≤ \|X-X \|_F. To implement the gradient ascent update, we first derive the gradient of the objective function. Some direct computations lead to the next result. Lemma 5.4. We have ∇Kqβ,γ,ψn(μ,ϕ) _Kq^β,γ,ψ^n(μ, π^φ) =2(Sn+UnK)Cβ,μ, =2 (S^n+U^nK )C_β,μ, ∇K¯qβ,γ,ψn(μ,ϕ) _ Kq^β,γ,ψ^n(μ, π^φ) =2(Zn+VnK¯)Cβ,μ¯, =2 (Z^n+V^n K )C_β, μ, ∇Σqβ,γ,ψn(ϕ) _ q^β,γ,ψ^n( π^φ) =11−β(γ2Σ−1+Un), = 11-β ( γ2 ^-1+U^n ), where Cβ,μC_β,μ and Cβ,μ¯C_β, μ are defined by Cβ,μ:=e[∫0+∞∫ℝde−βt(x−μ^¯tn)(x−μ^¯tn)⊺μ^tn(dx)t],Cβ,μ¯:=e[∫0+∞e−βtμ^¯tμ^¯t⊺t]. C_β,μ:=E^e [ _0^+∞ _R^de^-β t(x- μ_t^n)(x- μ_t^n) μ^n_t(dx)dt ],\;C_β, μ:=E^e [ _0^+∞e^-β t μ_t μ_t dt ]. Moreover, if Σ⪰aIp aI_p for some a>0a>0,qγ,ψn(μ,ϕ)q^γ,ψ^n(μ, π^φ) is Υ -smooth, where Υ=max2‖Un‖⋅‖Cβ,μ‖,2‖Vn‖⋅‖Cβ,μ¯‖,γ2a2(1−β) = \2\|U^n\|·\|C_β,μ\|,2\|V^n\|·\|C_β, μ\|, γ2a^2(1-β)\. 5.3 Convergence of inner fixed point iteration Consider the gradient ascent for the parameters K, K¯ K and Σ that Kn,(l+1) K^n,(l+1) =Kn,(l)+αK∇Kq1β,γ,ψn(μ,K), =K^n,(l)+ _K _Kq^β,γ,ψ^n_1(μ,K), K¯n,(l+1) K^n,(l+1) =K¯n,(l)+αK¯∇K¯q2β,γ,ψn(μ,K¯), = K^n,(l)+ _ K _ Kq^β,γ,ψ^n_2(μ, K), (5.17) Σn,(l+1) ^n,(l+1) =Σn,(l)+αΣ∇Σq3β,γ,ψn(Σ). = ^n,(l)+ _ _ q^β,γ,ψ^n_3( ). From the explicit expressions in Lemma 5.4, the equation (5.17) is equivalent to Kn,(l+1) K^n,(l+1) =Kn,(l)+2αK(Sn+UnKn,(l))Cβ,μ, =K^n,(l)+2 _K (S^n+U^nK^n,(l) )C_β,μ, K¯n,(l+1) K^n,(l+1) =K¯n,(l)+2αK¯(Zn+VnK¯n,(l))Cβ,μ¯, = K^n,(l)+2 _ K (Z^n+V^n K^n,(l) )C_β, μ, (5.18) Σn,(l+1) ^n,(l+1) =Σn,(l)+αΣ1−β(γ2(Σn,(l))−1+Un). = ^n,(l)+ _ 1-β ( γ2( ^n,(l))^-1+U^n ). With a suitable choice of αΣ _ and initial value Σn,(0) ^n,(0), the next result shows that the iterative update (5.18) ( actor:PG-explicit) for the covariance Σ is invariant in the following sense. Lemma 5.5. Fix n∈ℕn . Given a≥γ2‖Un‖a≥ γ2\|U^n\| and b≥2a2γσmin(−Un)b≥ 2a^2γ _ (-U^n), take aIp⪯Σn,(0)⪯bIpaI_p ^n,(0) bI_p. Let Σn,(l) ^n,(l) be l-th update (5.18) with αΣ=2a2(1−β)γ _ = 2a^2(1-β)γ, then aIp⪯Σn,(l)⪯bIpaI_p ^n,(l) bI_p for every l=0,….l=0,…. Proof. First, we show that aIp⪯Σn,(1)⪯bIpaI_p ^n,(1) bI_p. Define the function g(x)=x+γαΣ2x=x+a2xg(x)=x+ γ _ 2x=x+ a^2x. Then, g is increasing on [a,∞)[a,∞). As Σn,(0)⪰aIp ^n,(0) aI_p, then Σn,(1) ^n,(1) =Σn,(0)+a2(Σn,(0))−1+γ2Un⪰Ip+aIp+γ2Un⪰aIp. = ^n,(0)+a^2( ^n,(0))^-1+ γ2U^n I_p+aI_p+ γ2U^n aI_p. By using Σn,(0)⪯bIp ^n,(0) bI_p, we get that Σn,(1)=Σn,(0)+a2(Σn,(0))−1+γ2Un⪯bIp+a2bIp+γ2Un⪯bIp+γ2σmin(−Un)Ip+γ2Un⪯bIp. ^n,(1)= ^n,(0)+a^2( ^n,(0))^-1+ γ2U^n bI_p+ a^2bI_p+ γ2U^n bI_p+ γ2 _ (-U_n)I_p+ γ2U_n bI_p. By induction, we get aIp⪯Σn,(l)⪯bIpaI_p ^n,(l) bI_p. ∎ We also make the following assumption on Cβ,μC_β,μ and Cβ,μ¯C_β, μ defined in Lemma 5.4. Assumption 5.6. The symmetric matrices Cβ,μC_β,μ and Cβ,μ¯C_β, μ are positive definite, and aIp⪯Σn,(0)⪯bIpaI_p ^n,(0) bI_p with a and b satisfying a≥γ2‖Un‖a≥ γ2\|U^n\| and b≥2a2γσmin(−Un)b≥ 2a^2γ _ (-U^n). Recall that a function f is υ -strongly concave with υ>0 >0 if it is differentiable and −f-f is strongly convex, that is, −f(X′)≥−f(X)+Tr(∇(−f)(X)⊺(X′−X))+υ∥X′−X∥F2. -f(X )≥-f(X)+ Tr (∇(-f)(X) (X -X) )+ \|X -X\|_F^2. By Assumption 5.6, the minimum eigenvalue values of Cβ,μC_β,μ and Cβ,μ¯C_β, μ are positive and hence qnq^n are υ -strongly concave and we can easily derive the next result. Lemma 5.7. Under Assumption 5.6, let υ=minσmin(−Un)σmin(Cβ,μ),σmin(−Vn)σmin(Cβ,μ¯),γ2b2 = \ _ (-U^n) _ (C_β,μ), _ (-V^n) _ (C_β, μ), γ2b^2 \, we have that Tr((∇Kq1β,γ,ψn(μ,K′)−∇Kq1β,γ,ψn(μ,K))⊺(K′−K)]) Tr ( ( _Kq_1^β,γ,ψ^n(μ,K )- _Kq_1^β,γ,ψ^n(μ,K) ) (K -K) ] ) ≤−υ‖K′−K‖F2, ≤- \|K -K\|_F^2, Tr((∇K¯q2β,γ,ψn(μ,K¯′)−∇K¯q2β,γ,ψn(μ,K¯))⊺(K¯′−K¯)]) Tr ( ( _ Kq_2^β,γ,ψ^n(μ, K )- _ Kq_2^β,γ,ψ^n(μ, K) ) ( K - K) ] ) ≤−υ‖K¯′−K¯‖F2, ≤- \| K - K\|^2_F, Tr((∇Σq3β,γ,ψn(Σ′)−∇Σq3β,γ,ψn(Σ))⊺(Σ′−Σ)) Tr ( ( _ q_3^β,γ,ψ^n( )- _ q_3^β,γ,ψ^n( ) ) ( - ) ) ≤−υ‖Σ′−Σ‖F2. ≤- \| - \|_F^2. Furthermore, qβ,γ,ψn(μ)q^β,γ,ψ^n(μ) is υ -strongly concave. When qβ,γ,ψnq^β,γ,ψ^n is Υ -strongly smooth and υ -strongly concave, we have the following convergence of inner fixed point iteration. Proposition 5.8. For fixed n∈ℕn , Let (Kn,(0),K¯n,0,Σn,(0))∈(K^n,(0), K^n,0, ^n,(0))∈ A with aIp⪯Σn,(0)⪯bIpaI_p ^n,(0) bI_p. Under Assumption 5.6, there exists some η∈(0,1)η∈(0,1) such that, after sufficiently large L iterations, the iterative algorithm (5.18) with αK=αK¯=αΣ=υΥ _K= _ K= _ = achieves the following performance bound 0 0 ≤maxϕ∈Πqβ,γ,ψn(μ,ϕ)−qβ,γ,ψn(μ,ϕn,(L))≤(1−η)L(qβ,γ,ψn(μ,ϕn+1)−qβ,γ,ψn(μ,ϕn)), ≤ _φ∈ q^β,γ,ψ^n(μ, π^φ)-q^β,γ,ψ^n(μ, π^φ^n,(L))≤ (1-η )^L (q^β,γ,ψ^n(μ, π^φ^n+1)-q^β,γ,ψ^n(μ, π^φ^n) ), and ‖Kn+1−Kn,(L)‖F2+‖K¯n+1−K¯n,(L)‖F2+‖Σn+1−Σn,(L)‖F2 \|K^n+1-K^n,(L)\|_F^2+\| K^n+1- K^n,(L)\|^2_F+\| ^n+1- ^n,(L)\|_F^2 ≤ ≤ (1−η)L(‖Kn+1−Kn,(0)‖F2+‖K¯n+1−K¯n,(0)‖F2+‖Σn+1−Σn,(0)‖F2), (1-η)^L (\|K^n+1-K^n,(0)\|_F^2+\| K^n+1- K^n,(0)\|^2_F+\| ^n+1- ^n,(0)\|_F^2 ), where (Kn+1,K¯n+1,Σn+1)(K^n+1, K^n+1, ^n+1) is the unique maximizer to the optimization problem (5.15), see Lemma 5.3. Proof. The proof is standard and we only provide the proof of q3β,γ,ψnq^β,γ,ψ^n_3 for the completeness. By Lemma 5.7, we have q3β,γ,ψn(Σ′) q_3^β,γ,ψ^n( ) =q3β,γ,ψn(Σ)+Tr(∇Σq3β,γ,ψn(Σ)⊺(Σ′−Σ)) =q_3^β,γ,ψ^n( )+ Tr ( _ q_3^β,γ,ψ^n( ) ( - ) ) +∫01Tr((∇Σq3β,γ,ψn((1−t)Σ+tΣ′)−∇Σq3β,γ,ψn(Σ))⊺(Σ′−Σ))t + _0^1 Tr ( ( _ q_3^β,γ,ψ^n((1-t) +t )- _ q_3^β,γ,ψ^n( ) ) ( - ) )dt ≤q3β,γ,ψn(Σ)+Tr(∇Σq3β,γ,ψn(Σ)⊺(Σ′−Σ))−υ2‖Σ′−Σ‖F2 ≤ q_3^β,γ,ψ^n( )+ Tr ( _ q_3^β,γ,ψ^n( ) ( - ) )- 2\| - \|_F^2 (5.19) =q3β,γ,ψn(Σ)−12‖(υ(Σ′−Σ)−1υ∇Σq3β,γ,ψn(Σ))‖F2+12υ‖∇Σq3β,γ,ψn(Σ)‖F2 =q_3^β,γ,ψ^n( )- 12\| ( ( - )- 1 _ q_3^β,γ,ψ^n( ) )\|_F^2+ 12 \| _ q_3^β,γ,ψ^n( )\|_F^2 ≤q3β,γ,ψn(Σ)+12υ‖∇Σq3β,γ,ψn(Σ)‖F2. ≤ q_3^β,γ,ψ^n( )+ 12 \| _ q_3^β,γ,ψ^n( )\|_F^2. (5.20) Substituting Σ′=Σn+1 = ^n+1 and Σ=Σn,(l) = ^n,(l) into (5.20), we have 12υ(q3β,γ,ψn(Σn+1)−q3β,γ,ψn(Σn,(l)))≤Tr(∇Σq3β,γ,ψn(Σn,(l))⊺∇Σq3β,γ,ψn(Σn,(l))). 12 (q_3^β,γ,ψ^n( ^n+1)-q_3^β,γ,ψ^n( ^n,(l)) )≤ Tr ( _ q_3^β,γ,ψ^n( ^n,(l)) _ q_3^β,γ,ψ^n( ^n,(l)) ). (5.21) By fundamental theorem of calculus, we have that, for l=0,1,…,L−1l=0,1,…,L-1, q3β,γ,ψn(Σn,(l+1)) q_3^β,γ,ψ^n( ^n,(l+1)) = = q3β,γ,ψn(Σn,(l))+Tr(∇Σq3β,γ,ψn(Σn,l)⊺(Σn,(l+1)−Σn,(l))) q_3^β,γ,ψ^n( ^n,(l))+ Tr ( _ q_3^β,γ,ψ^n( ^n,l) ( ^n,(l+1)- ^n,(l)) ) +∫01Tr((∇Σq3β,γ,ψn(Σn,(l)+t(Σn,(l+1)−Σn,(l)))−∇Σq3β,γ,ψn(Σn,l))⊺(Σn,(l+1)−Σn,(l)))t + _0^1 Tr ( ( _ q_3^β,γ,ψ^n( ^n,(l)+t( ^n,(l+1)- ^n,(l)))- _ q_3^β,γ,ψ^n( ^n,l) ) ( ^n,(l+1)- ^n,(l)) )dt ≥ ≥ q3β,γ,ψn(Σn,(l))+αΣTr(∇Σq3β,γ,ψn(Σn,l)⊺∇Σq3β,γ,ψn(Σn,l)) q_3^β,γ,ψ^n( ^n,(l))+ _ Tr ( _ q_3^β,γ,ψ^n( ^n,l) _ q_3^β,γ,ψ^n( ^n,l) ) −12⋅Υ⋅αΣ2Tr(∇Σq3β,γ,ψn(Σn,l)⊺∇Σq3β,γ,ψn(Σn,l)) - 12· · _ ^2 Tr ( _ q_3^β,γ,ψ^n( ^n,l) _ q_3^β,γ,ψ^n( ^n,l) ) = = q3β,γ,ψn(Σn,(l))+12αΣTr(∇Σq3β,γ,ψn(Σn,(l))⊺∇Σq3β,γ,ψn(Σn,(l))). q_3^β,γ,ψ^n( ^n,(l))+ 12 _ Tr ( _ q_3^β,γ,ψ^n( ^n,(l)) _ q_3^β,γ,ψ^n( ^n,(l)) ). (5.22) Plugging (5.21) into (5.22), we obtain that q3β,γ,ψn(Σn,(l+1)) q_3^β,γ,ψ^n( ^n,(l+1)) ≥q3β,γ,ψn(Σn,(l))+12αΣTr(∇Σq3γ,ψn(Σn,l)⊺∇Σq3β,γ,ψn(Σn,l)) ≥ q_3^β,γ,ψ^n( ^n,(l))+ 12 _ Tr ( _ q_3^γ,ψ^n( ^n,l) _ q_3^β,γ,ψ^n( ^n,l) ) ≥q3β,γ,ψn(Σn,(l))+υΥ(q3β,γ,ψn(Σn+1)−q3β,γ,ψn(Σn,(l))). ≥ q_3^β,γ,ψ^n( ^n,(l))+ (q_3^β,γ,ψ^n( ^n+1)-q_3^β,γ,ψ^n( ^n,(l)) ). Subtracting q3β,γ,ψn(Σn+1)q_3^β,γ,ψ^n( ^n+1) on both sides and rearranging the terms lead to q3β,γ,ψn(Σn+1)−q3β,γ,ψn(Σn,(l+1))≤(1−υΥ)(q3β,γ,ψn(Σn+1)−q3β,γ,ψn(Σn,(l))). q_3^β,γ,ψ^n( ^n+1)-q_3^β,γ,ψ^n( ^n,(l+1))≤ (1- ) (q_3^β,γ,ψ^n( ^n+1)-q_3^β,γ,ψ^n( ^n,(l)) ). Moreover, we have that ‖Σn+1−Σn,(l+1)‖F2 \| ^n+1- ^n,(l+1)\|_F^2 = = ‖Σn+1−Σn,(l)‖F2−2αΣTr(∇Σq3β,γ,ψn(Σn,(l))⊺(Σn+1−Σn,(l)))+αΣ2‖∇Σq3β,γ,ψn(Σn,(l))‖F2 \| ^n+1- ^n,(l)\|_F^2-2 _ Tr ( _ q_3^β,γ,ψ^n( ^n,(l)) ( ^n+1- ^n,(l) ) )+ _ ^2\| _ q_3^β,γ,ψ^n( ^n,(l))\|_F^2 ≤ ≤ (1−υΥ)‖Σn+1−Σn,(l)‖F2−2αΣ(q3β,γ,ψn(Σn+1)−q3β,γ,ψn(Σn,(l)))+αΣ2‖∇Σq3β,γ,ψn(Σn,(l))‖F2 (1- )\| ^n+1- ^n,(l)\|_F^2-2 _ (q_3^β,γ,ψ^n( ^n+1)-q_3^β,γ,ψ^n( ^n,(l)) )+ _ ^2\| _ q_3^β,γ,ψ^n( ^n,(l))\|_F^2 ≤ ≤ (1−υΥ)‖Σn+1−Σn,(l)‖F2+(2αΣ−2αΣ)(q3β,γ,ψn(Σn+1)−q3β,γ,ψn(Σn,(l))), (1- )\| ^n+1- ^n,(l)\|_F^2+ (2 _ -2 _ ) (q_3^β,γ,ψ^n( ^n+1)-q_3^β,γ,ψ^n( ^n,(l)) ), where in the last inequality, we have used (5.19) and (5.22). Repeating the same arguments for q1β,γ,ψnq^β,γ,ψ^n_1 and q2β,γ,ψnq^β,γ,ψ^n_2, we obtain the desired result. ∎ We then readily conclude the next result. Corollary 5.9. For fixed n∈ℕn , Let (Kn,(0),K¯n,0,Σn,(0))∈(K^n,(0), K^n,0, ^n,(0))∈ A with aIp⪯Σn,(0)⪯bIpaI_p ^n,(0) bI_p. Under Assumption 5.6, there exists some η∈(0,1)η∈(0,1) such that, after sufficiently large L iterations, the iterative algorithm (5.18) with αK=αK¯=αΣ=υΥ _K= _ K= _ = achieves qβ,γ(μ,ϕn,(L);ϕn)≥maxϕ∈Πqβ,γ(μ,ϕ;ϕn)−2Δ−(1−η)L(qβ,γ,ψn(μ,ϕn+1)−qβ,γ,ψn(μ,ϕn)), q^β,γ(μ, π^φ^n,(L); π^φ^n)≥ _φ∈ q^β,γ(μ, π^φ; π^φ^n)-2 - (1-η )^L (q^β,γ,ψ^n(μ, π^φ^n+1)-q^β,γ,ψ^n(μ, π^φ^n) ), where Δ:=supϕ|qβ,γ,ψn(μ,ϕ)−qβ,γ(μ,ϕ;ϕn)| := _φ|q^β,γ,ψ^n(μ, π^φ)-q^β,γ(μ, π^φ; π^φ^n)|. By Corollary 5.9, if Δ is small, that is, qβ,γ(μ,ϕ;ϕn)q^β,γ(μ, π^φ; π^φ^n) is well approximated by qβ,γ,ψn(μ,ϕ)q^β,γ,ψ^n(μ, π^φ), then after sufficiently large L iterations, ϕn,(L) π^φ^n,(L) is a near-optimal policy of supϕqβ,γ(μ,ϕ;ϕn) _φq^β,γ(μ, π^φ; π^φ^n). 6 Numerical Examples 6.1 LQ-MFC problem Let us consider a linear model of the wealth process controlled by the social planner on behalf of the population in a mean-field setting, whose dynamics under discretely sampled portfolio actions in the exploratory formulation is given by dXs,=aδ(s)(bds+σdWs+σodBs),s≥t,Xt,=ξ, dX D, π_s=a π_δ(s) (bds+σ dW_s+ _odB_s ), s≥ t, X D, π_t=ξ, (6.1) with constant model parameters b>0,σ≥0,σo≥0b>0,σ≥ 0, _o≥ 0. The BM B stands for common noise. We consider a LQ-type performance measure under entropy regularization for the social planner, whose goal is to maximize the expected mean of the population’s wealth and meanwhile minimize the systemic risk described by the quadratic distance between the wealth and its conditional mean under common noise. That is, the objective function is given by J(t,ξ;)=e[XT,−λ(XT,−e[XT,|T])2]+γ∫tTE(δ(s),Xδ(s),,μδ(s),)ds]. J D(t,ξ; π)=E^e [X D, π_T-λ(X D, π_T-E^e[X D, π_T| G_T])^2 ]+γ _t^TE_ π(δ(s),X_δ(s) D, π, _δ(s) D, π)ds ]. (6.2) This example fits into the framework of LQ-MFC in Ren et al. (2026). By Theorem 5.1 in Ren et al. (2026), the optimal policy, the optimal value function and the optimal unregularized Iq-function admit the explicit expressions that ∗(⋅|t,x,μ) π^*(·|t,x,μ) =(−bσ2+σo2(x−μ¯)+b2λσ2e−b2σ2+σo2(t−T),γ2λ(σ2+σo2)e−b2σ2+σo2(t−T)), = N (- bσ^2+ _o^2(x- μ)+ b2λσ^2e^- b^2σ^2+ _o^2(t-T), γ2λ(σ^2+ _o^2)e^- b^2σ^2+ _o^2(t-T) ), J∗(t,μ) J^*(t,μ) =μ¯−λeb2σ2+σo2(t−T)Var(μ)+γb24(σ2+σo2)(t−T)2−γ2logπγ(σ2+σo2)λ(t−T) = μ-λ e b^2σ^2+ _o^2(t-T) Var(μ)+ γ b^24(σ^2+ _o^2)(t-T)^2- γ2 πγ(σ^2+ _o^2)λ(t-T) +σ2+σo24λσ2(e−b2σ2+σo2(t−T)−1), \;\;+ σ^2+ _o^24λσ^2 (e^- b^2σ^2+ _o^2(t-T)-1 ), q0,∗(t,μ,) q^0,*(t,μ, h) =−(σ2+σo2)λeb2σ2+σo2(t−T)∫ℝ×ℝ(a+bσ2+σo2(x−μ¯)−b2λσσ2+σo2e−b2σ2+σo2(t−T) =-(σ^2+ _o^2)λ e b^2σ^2+ _o^2(t-T) _R×R (a+ bσ^2+ _o^2(x- μ)- b2λσ σ^2+ _o^2e^- b^2σ^2+ _o^2(t-T) +(−1+σ2+σo2)¯)2(a|x)daμ(dx)+γb22(σ2+σo2)(t−T)−γ2logπγ(σ2+σo2)λ, \;\;+ (-1+ σ σ^2+ _o^2 ) h )^2 h(a|x)daμ(dx)+ γ b^22(σ^2+ _o^2)(t-T)- γ2 πγ(σ^2+ _o^2)λ, where ¯:=∫ℝ×ℝa(a|x)aμ(dx) h:= _R×Ra h(a|x)daμ(dx). When model parameters are unknown, we can accordingly consider the precise parametrization of the optimal value function J∗J^* and the optimal unregularized Iq-function q0,∗q^0,* by Jθ(t,μ) J^θ(t,μ) =μ¯−λeθ1(t−T)Var(μ)+γθ14(t−T)2+θ2(t−T)+θ3(e−θ1(t−T)−1), = μ-λ e _1(t-T) Var(μ)+ γ _14(t-T)^2+ _2(t-T)+ _3 (e^- _1(t-T)-1 ), (6.3) q0,ψ(t,μ,) q^0,ψ(t,μ, h) =−12eψ1+ψ2(t−T)∫ℝ×ℝ(a+ψ3(x−μ¯)+ψ4e−ψ2(t−T) =- 12e _1+ _2(t-T) _R×R (a+ _3(x- μ)+ _4e^- _2(t-T) +(ψ5−1)¯)2(a|x)daμ(dx)−γ2log(2πγ)−γ2ψ1+γψ22(t−T), +( _5-1) h )^2 h(a|x)daμ(dx)- γ2 (2πγ)- γ2 _1+ γ _22(t-T), (6.4) δq0,ψδ(t,μ,)(x,a) δ q^0,ψδ h(t,μ, h)(x,a) =−12eψ1+ψ2(t−T)(a2+2ψ3(x−μ¯)a+2ψ4ψ5e−ψ2(t−T)a−2(1−ψ52)¯a). =- 12e _1+ _2(t-T) (a^2+2 _3(x- μ)a+2 _4 _5e^- _2(t-T)a-2(1- _5^2) ha ). (6.5) where θ=(θ1,θ2,θ3)⊺∈ℝ3θ=( _1, _2, _3) ^3, ψ=(ψ1,…,ψ5)⊺∈ℝ5ψ=( _1,…, _5) ^5. The true values of parameters are θ1∗=b2σ2+σo2,θ2∗=−γ2logπγ(σ2+σo2)λ,θ3∗=σ2+σo24λσ2 _1^*= b^2σ^2+ _o^2, _2^*=- γ2 πγ(σ^2+ _o^2)λ, _3^*= σ^2+ _o^24λσ^2, ψ1∗=log(2λ(σ2+σo2)),ψ2∗=b2σ2+σo2,ψ3∗=bσ2+σo2,ψ4∗=−b2λσσ2+σo2,ψ5∗=σ2+σo2 _1^*= (2λ(σ^2+ _o^2) ), _2^*= b^2σ^2+ _o^2, _3^*= bσ^2+ _o^2, _4^*=- b2λσ σ^2+ _o^2, _5^*= σ σ^2+ _o^2, respectively. Accordingly, the optimal policy can be parameterized by ψ=(−ψ3(x−μ¯)−ψ4ψ5e−ψ2(t−T),γe−ψ1−ψ2(t−T)). π^ψ= N (- _3(x- μ)- _4 _5e^- _2(t-T),γ e^- _1- _2(t-T) ). (6.6) To generate data for the simulation experiment, we consider a simulator with access to the conditional mean, conditional variance and aggregated rewards of the social planner with test policies as input. We also choose to parameterize the test policy h in the same form of π but with the different parameter ϕ~ φ. The Environment SimulatorΔt: We first calculate the conditional mean of X,ϕ~X D, h φ and obtain that the conditional mean of the dynamics (6.1), denoted by μ¯t μ_t, satisfies μ¯tk+1≃μ¯tk−ϕ~4e−ϕ~2(t−T)(bΔt+σoΔtΔW), μ_t_k+1 μ_t_k- φ_4e^- φ_2(t-T) (b t+ _o t W ), where Δt t denotes the time step and ΔW∼(0,1) W N(0,1). Then we calculate the conditional variance of the dynamics (6.1) under the policy ϕ~ h φ denoted as Var(μt):=e[(Xt−μ¯t)2|t] Var( _t):=E^e [(X_t D- μ_t)^2 | G_t ]. The Euler approximation of Var(μt) Var( _t) is given by Var(μtk+1) Var( _t_k+1) ≃Var(μtk)+((−2bϕ~3+σ2ϕ~32+σo2ϕ~32)Var(μtk)+σ2ϕ~42e−2ϕ~2(tk−T) Var( _t_k)+ ( (-2b φ_3+σ^2 φ_3^2+ _o^2 φ_3^2 ) Var( _t_k)+σ^2 φ_4^2e^-2 φ_2(t_k-T) +(σ2+σo2)γe−ϕ~1−ϕ~2(tk−T))Δt−2σoϕ~3Var(μtk)ΔtΔW. +(σ^2+ _o^2)γ e^- φ_1- φ_2(t_k-T) ) t-2 _o φ_3 Var( _t_k) t W. In what follows, we consider three algorithms for this LQ-MFC example, namely the optimal q-learning algorithm (Algorithm 1), the Actor-Critic q-learning algorithm without inner iterations that L=1L=1 (Algorithm 2-A), and the Actor-Critic q-learning algorithm with inner iterations that L>1L>1 (Algorithm 2-B). The parametrizations of J, q0q^0 and the partial linear functional derivative of q0q^0 with respect to h are given in (6.3)-(6.5). In Algorithm 1, π takes the form in (6.6). In Algorithm 2-A and Algorithm 2-B, we take advantage of the derived form of π as in (6.6) but would like to implement the Actor-step to update the policy according to (4.9). To this end, we choose to parameterize π with the parameter ϕφ, which differs from the one in the Iq-function. In addition, by employing the form of the optimal Iq-function in (6.4), we can eliminate the consistency loss function in the Actor-step of Algorithm 2-A and Algorithm 2-B by setting wc=0w_c=0. For the simulator, we choose inputs T=1T=1, b=0.25b=0.25, σ=0.5σ=0.5, σ0=0.5 _0=0.5. We also set the model parameters as β=0β=0, γ=0.5γ=0.5 and λ=1.5λ=1.5, the time step Δt=0.1 t=0.1, the number of test policies M=20M=20, and set the lower and upper bounds of the uniform distribution for the test policies as pi(n)=0p_i(n)=0, qi(n)=1n0.225q_i(n)= 1n^0.225, 1≤i≤51≤ i≤ 5. We set the initialization of μ¯0∼([0,1]) μ_0 N([0,1]) and Var(μ0)∼([0,1]) Var( _0) U([0,1]) and choose the initialization of θ=(−0.5,0.5,0.5)⊺θ=(-0.5,0.5,0.5) , ψ=(1,−0.5,1,−0.5,0.1)⊺ψ=(1,-0.5,1,-0.5,0.1) in all three algorithms and set ϕ=(1.5,−1,1.5,−1)⊺φ=(1.5,-1,1.5,-1) in Algorithms 2-A and 2-B, respectively. In Algorithm 1, the number of episodes N=2000N=2000 and the learning rates are set as αθ _θ =(0.02n0.2,0.03n0.15,0.06n0.31),αψ=(0.03n0.15,0.09n0.15,0.02n0.3,0.015n0.2,0.09n0.03). = ( 0.02n^0.2, 0.03n^0.15, 0.06n^0.31 ),\;\; _ψ= ( 0.03n^0.15, 0.09n^0.15, 0.02n^0.3, 0.015n^0.2, 0.09n^0.03 ). Numerical results for the learning of θ and ψ are presented in Figure 1. (a) Convergence of parameters θ (b) Convergence of parameters ψ (c) L1L^1 error of the value function Figure 1: Convergence of the learnt parameters under Algorithm 1 In Algorithm 2-A, the number of episodes N=10000N=10000, there is no inner iteration that L=1L=1. The learning rates are given by αθ=(0.02n0.2,0.03n0.15,0.06n0.31),αψ=(0.04n0.15,0.12n0.1,0.02n0.3,0.015n0.2,0.25n0.2),αϕ=(0.04n0.15,0.1n0.1,0.02n0.3,0.015n0.2). _θ= ( 0.02n^0.2, 0.03n^0.15, 0.06n^0.31 ),\; _ψ= ( 0.04n^0.15, 0.12n^0.1, 0.02n^0.3, 0.015n^0.2, 0.25n^0.2 ),\; _φ= ( 0.04n^0.15, 0.1n^0.1, 0.02n^0.3, 0.015n^0.2 ). Numerical results for the learning of θ and ψ are presented in Figure 2. (a) Convergence of parameters θ (b) Convergence of parameters ψ (c) Convergence of parameters ϕφ (d) L1L^1 error of the value function Figure 2: Convergence of the learnt parameters under Algorithm 2-A In Algorithm 2-B, we set the number of episodes N=2000N=2000, the number of inner iterations L=25L=25, and the learning rates that αθ=(0.02n0.2,0.03n0.15,0.06n0.31),αψ=(0.03n0.1,0.09n0.1,0.02n0.2,0.015n0.2,0.15n0.2),αϕ=(0.03n0.15l0.15,0.08n0.1l0.1,0.02n0.1l0.1,0.015n0.2l0.2), _θ= ( 0.02n^0.2, 0.03n^0.15, 0.06n^0.31 ),\; _ψ= ( 0.03n^0.1, 0.09n^0.1, 0.02n^0.2, 0.015n^0.2, 0.15n^0.2 ),\; _φ= ( 0.03n^0.15l^0.15, 0.08n^0.1l^0.1, 0.02n^0.1l^0.1, 0.015n^0.2l^0.2 ), where n and l are indexes of outer iterations and inner iterations, respectively. In addition, we evaluate the performance of the learnt policy over iterations by illustrating the value function error. At each iteration, we execute the learnt policy 3000 times to output 3000 trajectories by the simulator and estimate the value function under the learnt policy averaged over these trajectories. The optimal value function is used as a benchmark. We adopt the L1L^1 error between the learnt value function and the true value function along the iterations in three algorithms as shown in Figures 1(c), 2(d), 3(d). (a) Convergence of parameters θ (b) Convergence of parameters ψ (c) Convergence of parameters ϕφ (d) L1L^1 error of the value function Figure 3: Convergence of the learnt parameters under Algorithm 2-B 6.2 Non-LQ-MFC problem We next consider an example of non-LQ MFC problem where the state process is governed by the conditional McKean-Vlasov SDE under discretely sampled actions that dXs,=aδ(s),μ¯s,(bds+σdWs+σodBs),s≥t,Xt,=ξ∼μ. dX_s D, π=a_δ(s) D, π μ_s D, π (bds+σ dW_s+ _odB_s ), s≥ t, X_t D, π=ξ μ. (6.7) Here, we assume μ¯=∫ℝxμ(dx)>0 μ= _Rxμ(dx)>0, b>0,σ>0b>0,σ>0, and σo>0 _o>0. Let us consider the entropy-regularized logarithmic utility maximization problem on the population mean with the objective function defined by J(t,ξ;)=e[γ∫tTe−β(s−t)E(δ(s),Xδ(s),,μδ(s),)s+e−β(T−t)logμ¯T,]. J D(t,ξ; π)=E^e [γ _t^Te^-β(s-t)E_ π(δ(s),X_δ(s) D, π, _δ(s) D, π)ds+e^-β(T-t) μ_T D, π ]. Lemma 6.1. The optimal value function J∗J^* and the optimal policy ∗ π^* admit the form that J∗(t,μ) J^*(t,μ) =eβ(t−T)logμ¯+C(t), =e^β(t-T) μ+C(t), ∗(a|t,x,μ) π^*(a|t,x,μ) =Exp(σo2γeβ(t−T)((b2σo2)2+γσo2e−β(t−T)−b2σo2)), =Exp ( _o^2γe^β(t-T) ( ( b2 _o^2)^2+ γ _o^2e^-β(t-T)- b2 _o^2 ) ), where C(t) C(t) =A1eβ(t−T)+A2(T−t)eβ(t−T)+A3eβ(t−T)log(2e2β(t−T)+4γσo2b2eβ(t−T)+2eβ(t−T)+4γσo2b2) =A_1e^β(t-T)+A_2(T-t)e^β(t-T)+A_3e^β(t-T) (2 e^2β(t-T)+ 4γ _o^2b^2e^β(t-T)+2e^β(t-T)+ 4γ _o^2b^2 ) +A4eβ(t−T)1+4γσo2b2e−β(t−T)+A5log(1+4γσo2b2e−β(t−T)+1)+A6, +A_4e^β(t-T) 1+ 4γ _o^2b^2e^-β(t-T)+A_5 ( 1+ 4γ _o^2b^2e^-β(t-T)+1 )+A_6, (6.8) and the constants A1A_1 to A6A_6 are given respectively by A1 A_1 =b24βσo2log(21+4γσo2b2+2+4γσo2b2)+γβlog(b2σo2)−b24βσo21+4γσo2b2, = b^24β _o^2 (2 1+ 4γ _o^2b^2+2+ 4γ _o^2b^2 )+ γβ ( b2 _o^2)- b^24β _o^2 1+ 4γ _o^2b^2, −γβ+γβlog(1+4γσo2b2+1),A2=b24σo2,A3=−b24βσo2, - γβ+ γβ ( 1+ 4γ _o^2b^2+1 ), A_2= b^24 _o^2, A_3=- b^24β _o^2, A4 A_4 =b24βσo2,A5=−γβ,A6=γβ−γβlog(b2σo2). = b^24β _o^2, A_5=- γβ, A_6= γβ- γβ ( b2 _o^2 ). Proof. We first conjecture that J∗J^* takes the form of J∗(t,μ)=A(t)logμ¯+C(t)J^*(t,μ)=A(t) μ+C(t). It follows that ∂μJ∗(t,μ)(x)=A(t)μ¯,∂x∂μJ∗(t,μ)(x)=0,∂μ2J∗(t,μ)(x,x′)=−A(t)μ¯2, _μJ^*(t,μ)(x)= A(t) μ,\; _x _μJ^*(t,μ)(x)=0,\; _μ^2J^*(t,μ)(x,x )=- A(t) μ^2, and ℋ(t,μ,;∗)=A(t)(b¯−12σo2¯2) H(t,μ, h; π^*)=A(t) (b h- 12 _o^2 h^2 ). By the two-layer fixed point condition in (2.7), we get A(t)ba−aσo2A(t)∫ℝ×ℝa′∗(a′|t,x′,μ)a′μ(dx′)−γlog∗(a|t,x,μ)=κ(t,x,μ). A(t)ba-a _o^2A(t) _R×Ra h^*(a |t,x ,μ)da μ(dx )-γ h^*(a|t,x,μ)=κ(t,x,μ). (6.9) To derive the explicit optimal policy as the two-layer fixed point, we consider ∗ h^* as an exponential distribution with the parameter λ(t,x,μ)λ(t,x,μ) that −γlog∗(⋅|t,x,μ)=γλ(t,x,μ)a−γlogλ(t,x,μ). -γ h^*(·|t,x,μ)=γλ(t,x,μ)a-γ λ(t,x,μ). (6.10) Plugging (6.10) into (6.9) and noting that λ(t,x,μ)>0λ(t,x,μ)>0 is independent of x and μ, we get that 1λ(t)=(b2σo2)2+γA(t)σo2+b2σo2>0. 1λ(t)= ( b2 _o^2 )^2+ γA(t) _o^2+ b2 _o^2>0. (6.11) Substituting (6.11) into (2.5), we have that (A′(t)−βA(t))logμ¯+C′(t)−βC(t)+12A(t)b((b2σo2)2+γA(t)σo2+b2σo2)+γ2 (A (t)-β A(t) ) μ+C (t)-β C(t)+ 12A(t)b ( ( b2 _o^2 )^2+ γA(t) _o^2+ b2 _o^2 )+ γ2 −γlog((b2σo2)2+γA(t)σo2+b2σo2)=0. -γ ( ( b2 _o^2 )^2+ γA(t) _o^2+ b2 _o^2 )=0. Together with A(T)=1A(T)=1, we first get that A(t)=eβ(t−T)>0A(t)=e^β(t-T)>0. We then obtain the ODE of C(t)C(t) with the terminal condition C(T)=0C(T)=0 that C′(t)−βC(t)+b24σo2eβ(t−T)+γ2+12b(b2σo2)2e2β(t−T)+γσo2eβ(t−T) C (t)-β C(t)+ b^24 _o^2e^β(t-T)+ γ2+ 12b ( b2 _o^2 )^2e^2β(t-T)+ γ _o^2e^β(t-T) −γlog((b2σo2)2+γσo2eβ(t−T)+b2σo2)=0, -γ ( ( b2 _o^2 )^2+ γ _o^2e^β(t-T)+ b2 _o^2 )=0, for which we can obtain the explicit solution as in (6.8). For the well-posedness of the problem, let us also verify that μ¯s>0 μ_s>0 for any s∈[t,T]s∈[t,T] under the optimal policy ∗=Exp(λ(t)) π^*=Exp(λ(t)) whenever μ¯>0 μ>0. Recall that aδ(r)∼Exp(λ(δ(r)))a_δ(r) (λ(δ(r))), and thus e[aδ(r)]=1λ(δ(r))E^e[a_δ(r)]= 1λ(δ(r)). We take the conditional expectation e[⋅|s]E^e[·| G_s] on both sides of (6.7) and deduce that, for ℙeP^e-a.s. ω0 _0, μ¯s= μ_s= μ¯+e[∫ts1λ(δ(r))(bμ¯r(ω0)dr+σμ¯r(ω0)dWr+σoμ¯r(ω0)dBr(ω0))|s] μ+E^e [ _t^s 1λ(δ(r)) (b μ_r(ω^0)dr+σ μ_r(ω^0)dW_r+ _o μ_r(ω^0)dB_r(ω^0) ) | G_s ] = = μ¯+∫ts1λ(δ(r))(bμ¯r(ω0)dr+σoμ¯r(ω0)dBr(ω0)), μ+ _t^s 1λ(δ(r)) (b μ_r(ω^0)dr+ _o μ_r(ω^0)dB_r(ω^0) ), (6.12) where in the last equality we have used e[∫tsσμ¯rWr|s]=0E^e [ _t^sσ μ_rdW_r | G_s ]=0 by some standard localization arguments and martingale property. Therefore, μ¯s μ_s is a geometric Brownian motion, which implies that μ¯s>0 μ_s>0 whenever μ¯>0 μ>0. ∎ When model parameters b, σ, σo _o are unknown, we can consider the parameterization of J∗J^* and q0,∗q^0,* based on the above explicit expressions that Jθ(t,μ) J^θ(t,μ) =eβ(t−T)logμ¯+θ1eβ(t−T)−θ1+(T−t)eθ2+β(t−T) =e^β(t-T) μ+ _1e^β(t-T)- _1+(T-t)e _2+β(t-T) (6.13) −1βeθ2+β(t−T)log(2e2β(t−T)+γe−θ2+β(t−T)+2eβ(t−T)+γe−θ2) - 1βe _2+β(t-T) (2 e^2β(t-T)+γ e^- _2+β(t-T)+2e^β(t-T)+γ e^- _2 ) +1βeθ2log(21+γe−θ2+2+γe−θ2) + 1βe _2 (2 1+γ e^- _2+2+γ e^- _2 ) −γβlog(1+γe−θ2−β(t−T)+1)+γβlog(1+γe−θ2+1) - γβ ( 1+γ e^- _2-β(t-T)+1 )+ γβ ( 1+γ e^- _2+1 ) +1βeθ2+β(t−T)1+γe−θ2−β(t−T)−1βeθ21+γe−θ2, + 1βe _2+β(t-T) 1+γ e^- _2-β(t-T)- 1βe _2 1+γ e^- _2, q0,ψ(t,μ,) q^0,ψ(t,μ, h) =12eψ1+β(t−T)(4eψ2¯−¯2)−ψ3eβ(t−T)−ψ3eβ(t−T)1+γe−ψ3−β(t−T), = 12e _1+β(t-T) (4e _2 h- h^2 )- _3e^β(t-T)- _3e^β(t-T) 1+γ e^- _3-β(t-T), +γlog(1+γe−ψ3−β(t−T)+1)+γψ2−γ2, \;+γ ( 1+γ e^- _3-β(t-T)+1 )+γ _2- γ2, (6.14) δq0,ψδ(t,μ,)(x,a) δ q^0,ψδ h(t,μ, h)(x,a) =12eψ1+β(t−T)(4eψ2−2¯)a, = 12e _1+β(t-T)(4e _2-2 h)a, (6.15) where θ=(θ1,θ2)∈ℝ2θ=( _1, _2) ^2 and ψ=(ψ1,ψ2,ψ3)∈ℝ3ψ=( _1, _2, _3) ^3. Then, the true values of parameters are θ1∗=b24βσo2log(21+4γσo2b2+2+4γσo2b2)+γβlog(b2σo2)−b24βσo21+4γσo2b2 _1^*= b^24β _o^2 (2 1+ 4γ _o^2b^2+2+ 4γ _o^2b^2 )+ γβ ( b2 _o^2)- b^24β _o^2 1+ 4γ _o^2b^2, θ2∗=log(b24σo2) _2^*= ( b^24 _o^2 ), and ψ1∗=log(σo2),ψ2∗=log(b2σo2),ψ3∗=log(b24σo2) _1^*= ( _o^2), _2^*= ( b2 _o^2), _3^*= ( b^24 _o^2), respectively. Accordingly, the optimal policy can be parameterized by ψ=Exp(1γeψ1+β(t−T)(−eψ2+e2ψ2+γe−ψ1−β(t−T))). π^ψ=Exp ( 1γe _1+β(t-T) (-e _2+ e^2 _2+γ e^- _1-β(t-T) ) ). (6.16) The Environment Simulator Δt: For the simulator, we need to compute the conditional log mean of X D under the policy ϕ~=Exp(1γeϕ~1+β(t−T)(−eϕ~2+e2ϕ~2+γe−ϕ~1−β(t−T))) h φ=Exp ( 1γe φ_1+β(t-T) (-e φ_2+ e^2 φ_2+γ e^- φ_1-β(t-T) ) ). From (6.12), we get the evolution of the conditional log mean denoted by logμ¯t μ_t that logμ¯tk+1 μ_t_k+1 ≃(b(eϕ~2+e2ϕ~2+γe−ϕ~1−β(tk−T)−12σo2(eϕ~2+e2ϕ~2+γe−ϕ~1−β(tk−T))2)Δt (b (e φ_2+ e^2 φ_2+γ e^- φ_1-β(t_k-T)- 12 _o^2 (e φ_2+ e^2 φ_2+γ e^- φ_1-β(t_k-T) )^2 ) t +σo(eϕ~2+e2ϕ~2+γe−ϕ~1−β(tk−T))ΔtΔW, \;\;\;+ _o (e φ_2+ e^2 φ_2+γ e^- φ_1-β(t_k-T) ) t W, where Δt t denotes the time step and ΔW∼(0,1) W N(0,1). Thanks to the exact parameterization of targeted functions in this non-LQ-MFC example, the implementation of the optimal q-learning algorithm (Algorithm 1) is again straightforward and hence omitted. On the other hand, it is still an open problem whether the inner iterations in the Actor-step converge or not in a non-LQ setting. Therefore, we shall implement and present in this example both the Actor-Critic q-learning algorithm without inner iterations that L=1L=1 (Algorithm 2-A) and the Actor-Critic q-learning algorithm with inner iterations that L>1L>1 (Algorithm 2-B). In particular, we would like to numerically confirm the convergence of inner iterations in this example. In Algorithm 2-A and Algorithm 2-B, π takes the similar form of (6.16) but again with the parameter ϕφ differing from the parameter ψ in q0,ψq^0,ψ. For the simulator, we choose inputs T=1T=1, b=1.5b=1.5, σ=0.5σ=0.5, σo=1 _o=1. Moreover, we set the model parameters as β=1β=1, γ=0.2γ=0.2, the time step size Δt=0.05 t=0.05, the number of test policies M=10M=10 and set the lower and upper bounds of the uniform distribution for the test policies as pi(n)=0p_i(n)=0, qi(n)=0.4n0.2q_i(n)= 0.4n^0.2, 1≤i≤31≤ i≤ 3. We set the initialization of logμ¯0∼([0,1]) μ_0 N([0,1]), and choose the initialization of θ=(−0.5,0.5)θ=(-0.5,0.5), ψ=(0.5,0.5,0.5)ψ=(0.5,0.5,0.5) and ϕ=(1.5,−1,1.5,−1)φ=(1.5,-1,1.5,-1) in Algorithms 2-A and 2-B, respectively. In Algorithm 2-A, there is no inner iteration that L=1L=1, and we set the number of episodes N=10000N=10000 and the learning rates as αθ=(0.01n0.49,0.01n0.49),if 1≤n≤2000,(0.014×n0.49,0.013×n0.49),otherwise,αψ=(0.002n0.51,0.002n0.57,0.01n0.32),if 1≤n≤2000,(0.002n0.51,0.002n0.69,0.01n0.64),otherwise. _θ= \ aligned & ( 0.01n^0.49, 0.01n^0.49 ),\;& if\;1≤ n≤ 2000,\\ & ( 0.014× n^0.49, 0.013× n^0.49 ),\;& otherwise, aligned . _ψ= \ aligned & ( 0.002n^0.51, 0.002n^0.57, 0.01n^0.32 ),\;& if\;1≤ n≤ 2000,\\ & ( 0.002n^0.51, 0.002n^0.69, 0.01n^0.64 ),\;& otherwise. aligned . αϕ=(0.06n0.04,0.02n0.2). _φ= ( 0.06n^0.04, 0.02n^0.2 ). Numerical results for the learning of θ and ψ are presented in Figure 4. (a) Convergence of parameters θ (b) Convergence of parameters ψ (c) Convergence of parameters ϕφ (d) L1L^1 error of the value function Figure 4: Convergence of the learnt parameters under Algorithm 2-A In Algorithm 2-B, we choose the number of inner iterations L=30L=30, and we set the number of episodes N=5000N=5000 the learning rates as αθ=(0.01n0.48,0.01n0.47),if 1≤n≤2000,(0.014×n0.49,0.013×n0.49),otherwise,αψ=(0.002n0.51,0.002n0.57,0.01n0.31),if 1≤n≤2000,(0.002n0.51,0.002n0.69,0.01n0.8),otherwise. _θ= \ aligned & ( 0.01n^0.48, 0.01n^0.47 ),\;& if\;1≤ n≤ 2000,\\ & ( 0.014× n^0.49, 0.013× n^0.49 ),\;& otherwise, aligned . _ψ= \ aligned & ( 0.002n^0.51, 0.002n^0.57, 0.01n^0.31 ),\;& if\;1≤ n≤ 2000,\\ & ( 0.002n^0.51, 0.002n^0.69, 0.01n^0.8 ),\;& otherwise. aligned . αϕ=(0.006n0.1,0.002n0.15). _φ= ( 0.006n^0.1, 0.002n^0.15 ). The results for the learning of parameters are reported in Figure 5. (a) Convergence of parameters θ (b) Convergence of parameters ψ (c) Convergence of parameters ϕφ (d) L1L^1 error of the value function Figure 5: Convergence of the learnt parameters under Algorithm 2-B We conclude with some interesting observations from two examples: (i) All algorithms lead to the convergence of parameters with the high accuracy within 10000 iterations. (i) Although the outer iterations of Algorithm 2-B is less than that of Algorithm 2-A, the total iterations of Algorithm 2-B is actually more than that of Algorithm 2-A. Therefore, Algorithm 2-A is more efficient and robust in cases when the characterization of the two-layer fixed point is not available and is also applicable to other cases such as when the normalizing constant in the Gibbs measure is not available (see section 5 in Jia and Zhou (2023)). Acknowledgement: X. Wei is supported by National Natural Science Foundation of China grant under no.12201343 and no.12571509. X. Yu is supported by the Hong Kong RGC General Research Fund (GRF) under grant no. 15211524. References Angiuli et al. (2022) A. Angiuli, J. P. Fouque and M. Laurière (2022). Unified reinforcement Q-learning for mean field game and control problems. Mathematics of Control, Signals, and Systems. 34(2), 217-271. Angiuli et al. (2023b) A. Angiuli, J. P. Fouque, R. Hu and A. Raydan (2023a): Deep reinforcement learning for infinite horizon mean field problems in continuous spaces. Preprint, available at arXiv:2309.10953. To appear in Journal of Machine Learning. Bo et al. (2025) L. Bo, Y. Huang and X. Yu (2025): On optimal tracking portfolio in incomplete markets: The reinforcement learning approach. SIAM Journal on Control and Optimization. 63(1), 321-348. Carmona et al (2023) R. Carmona, M. Laurière and Z. Tan. (2023): Model-free mean-field reinforcement learning: mean-field MDP and mean-field Q-learning. Annals of Applied Probability. 33(6B), 5334-5381. Cui et al. (2021) K. Cui, A. Tahir, M. Sinzger and H. Koeppl (2021): Discrete-time mean field control with environment states. In 2021 60th IEEE Conference on Decision and Control (CDC). Dai et al. (2023a) M. Dai, Y. Dong and Y. Jia (2023): Learning equilibrium mean-variance strategy. Mathematical Finance. 33(4), 1166-1212. Dai et al. (2023b) M. Dai, Y. Dong, Y. Jia and X. Y. Zhou (2023): Data-driven Merton’s strategies via policy randomization. Preprint, available at arXiv:2312.11797. Dong (2024) Y. Dong (2024): Randomized optimal stopping problem in continuous time and reinforcement learning algorithm. SIAM Journal on Control and Optimization. 62(3), 1590-1614. Frikha et al. (2025) N. Frikha, M. Germain, M. Laurière, H. Pham. and X. Song (2023). Actor-Critic learning for mean-field control in continuous time. Journal of Machine Learning Research. 26(127):1-42. Giegrich et al. (2024) M. Giegrich, C. Reisinger and Y. Zhang (2024): Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems. SIAM Journal on Control and Optimization. 62(2):1060-92. Gu et al. (2021) H. Gu, X. Guo, X. Wei and R. Xu (2021): Mean-field controls with Q-learning for cooperative MARL: Convergence and complexity analysis. SIAM Journal on Mathematics of Data Science. 3(4), 1168-1196. Gu et al. (2024) H. Gu, X. Guo, X. Wei and R. Xu (2022): Mean-field multi-agent reinforcement learning: A decentralized network approach. Mathematics of Operations Research. 50(1), 506-536. Guo et al. (2022) X. Guo, R. Xu and T. Zariphopoulou (2022): Entropy regularization for mean field games with learning. Mathematics of Operations Research. 47(4), 3239-3260. Han et al. (2023) X. Han, R. Wang and X. Y. Zhou (2023): Choquet regularization for continuous-time reinforcement learning. SIAM Journal on Control and Optimization. 61(5), 2777-2801. Huang et al. (2025) Y. Huang, M. Li, X. Yu and Z. Zhou (2025): Continuous-time reinforcement learning for optimal switching over multiple regimes. Preprint, available at arXiv:2512.04697 Jia and Zhou (2022a) Y. Jia and X. Y. Zhou (2022a): Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research. 23, 1-50. Jia and Zhou (2022b) Y. Jia and X. Y. Zhou (2022b): Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach. Journal of Machine Learning Research. 23, 1-55. Jia and Zhou (2023) Y. Jia and X. Y. Zhou (2023): q-learning in continuous time. Journal of Machine Learning Research. 24, 1-61. Jia (2026) Y. Jia (2026): Continuous-time risk-sensitive reinforcement learning via quadratic variation penalty. Applied Mathematics &\& Optimization, forthcoming. Laurière et al. (2022) M. Laurière, S. Perrin, J. Pérolat, S. Girgin, P. Muller, R. Élie, M. Geist and O. Pietquin (2022): Learning in mean field games: A survey. Preprint, available at arXiv:2205.12944. Liang et al. (2024) H. Liang, Z. Chen and K. Jing (2024): Actor-critic reinforcement learning algorithms for mean field games in continuous time, state and action spaces. Applied Mathematics and Optimization. 89(3): 72. Pham and Warin (2024) H. Pham and X. Warin (2024): Mean-field neural networks-based algorithms for McKean-Vlasov control problems. Journal of Machine Learning, 3:176-214. Pham and Warin (2023) H. Pham and X. Warin (2023): Mean-field neural networks: learning mappings on Wasserstein space. Neural Networks, 168:380-93. Ren et al. (2026) Z. Ren, X. Wei, X. Yu and X. Y. Zhou (2026): Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations. Working paper. Wang et al. (2020) H. Wang, T. Zariphopoulou and X. Y. Zhou (2020): Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research. 21(1):8145-8178. Wang et al. (2023) Wang, B., X. Gao and L. Li (2023): Reinforcement learning for continuous-time optimal execution: Actor-Critic algorithm and error analysis. Finance and Stochastics, 30, 597-655. Wei and Yu (2025) X. Wei and X. Yu (2025): Continuous-time q-learning for mean-field control problems. Applied Mathematics and Optimization. 91: 10. Wei et al. (2024) X. Wei, X. Yu and F. Yuan (2024): Unified continuous-time q-learning for mean-field game and mean-field control problems. Preprint, available at arXiv:2407.04521. (29)