Paper deep dive
Reinforcement Learning-Guided Evolutionary Policy Optimization for Preference-Adjustable Heterogeneous Agile Earth Observation Satellite Scheduling
He Wang, Junyu Wu, Hui Li, Yanjie Song, Witold Pedrycz, Liang Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Heterogeneous agile Earth observation satellite (AEOS) scheduling requires task selection, satellite assignment, and observation sequencing under satellite-dependent visibility windows, attitude maneuvering requirements, energy consumption, and onboard storage constraints. Since satellites differ in orbital access, maneuvering capability, and payload resources, the same task may have different feasible windows, transition costs, and resource-consumption patterns on different platforms, which increases the difficulty of unified modeling and efficient optimization. To address this problem, this paper proposes an evolutionary policy optimization framework for heterogeneous AEOS scheduling with preference-adjustable weighted objectives. In the modeling layer, assignment-based indirect encoding is combined with decoder-based equivalent-cost evaluation to retain satellite-dependent constraints while integrating task gain, energy saving, and load balance into an interpretable scalar utility. In the optimization layer, schedule decoding, population-based search, and online actor-critic operator control are decoupled, so that reinforcement learning selects high-level search operators rather than constructing schedules directly. Based on this framework, a reinforcement-learning-assisted operator-selection memetic evolutionary algorithm (RLOSMEA) is developed to coordinate global exploration, feasibility recovery, and local refinement under a limited function-evaluation budget. Experiments on different heterogeneous AEOS scenarios show that RLOSMEA achieves higher overall weighted utility and more stable convergence than representative metaheuristic baselines. Sensitivity and learning-behavior analyses further confirm the robustness of the proposed method and the effectiveness of reinforcement-learning-guided operator selection.
Tags
Links
- Source: https://arxiv.org/abs/2608.24470v1
- Canonical: https://arxiv.org/abs/2608.24470v1
Trouble viewing inline? Open PDF directly →
Full Text
71,986 characters extracted from source content.
Expand or collapse full text
Reinforcement Learning-Guided Evolutionary Policy Optimization for Preference-Adjustable Heterogeneous Agile Earth Observation Satellite SchedulingThanks: This research was jointly funded by the National Natural Science Foundation of China (Nos. 62373117, 62403158, 62573150, 72501042), the Fundamental Research Funds for Central Universities (No. 3072025GH0401).(Corresponding author: Liang Li) He Wang, Junyu Wu, Hui Li and Liang Li are with the College of Intelligent Science and Engineering, Harbin Engineering University, Harbin 150001, China (e-mail: wang_he@hrbeu.edu.cn, wujunyu@hrbeu.edu.cn, lihuiheu@hrbeu.edu.cn, liliang@hrbeu.edu.cn). Yanjie Song is with the School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China(e-mail: songyj_2017@163.com). Witold Pedrycz is with the Silesian University of Technology (SUT), Department of Measurement and Control Systems, Gliwice, Akademicka 2, 44-100 Poland, Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 2R3, Canada, Constructor University, Bremen, Germany, also with the Research Center of Performance and Productivity Analysis, Istinye University, Istanbul, Türkiye (e-mail: pedrycz@ualberta.ca).Thanks: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. He Wang Junyu Wu Hui Li Yanjie Song Affiliation: Witold Pedrycz, and Liang Li Abstract Heterogeneous agile Earth observation satellite (AEOS) scheduling requires task selection, satellite assignment, and observation sequencing under satellite-dependent visibility windows, attitude maneuvering requirements, energy consumption, and onboard storage constraints. Since satellites differ in orbital access, maneuvering capability, and payload resources, the same task may have different feasible windows, transition costs, and resource-consumption patterns on different platforms, which increases the difficulty of unified modeling and efficient optimization. To address this problem, this paper proposes an evolutionary policy optimization framework for heterogeneous AEOS scheduling with preference-adjustable weighted objectives. In the modeling layer, assignment-based indirect encoding is combined with decoder-based equivalent-cost evaluation to retain satellite-dependent constraints while integrating task gain, energy saving, and load balance into an interpretable scalar utility. In the optimization layer, schedule decoding, population-based search, and online actor–critic operator control are decoupled, so that reinforcement learning selects high-level search operators rather than constructing schedules directly. Based on this framework, a reinforcement-learning-assisted operator-selection memetic evolutionary algorithm (RLOSMEA) is developed to coordinate global exploration, feasibility recovery, and local refinement under a limited function-evaluation budget. Experiments on different heterogeneous AEOS scenarios show that RLOSMEA achieves higher overall weighted utility and more stable convergence than representative metaheuristic baselines. Sensitivity and learning-behavior analyses further confirm the robustness of the proposed method and the effectiveness of reinforcement-learning-guided operator selection. Index Terms: Satellite scheduling, Earth observation, reinforcement learning, evolutionary policy optimization, Multi-objective optimization. I Introduction Agile Earth observation satellites (AEOSs) provide flexible and time-sensitive observation capabilities for a broad range of applications, such as disaster response, environmental monitoring, infrastructure inspection, and wide-area surveillance [1, 2, 3, 4]. With rapid multi-axis attitude maneuverability, AEOSs can dynamically adjust their observation geometry and access geographically distributed ground targets within flexible visibility windows. Compared with conventional Earth observation satellites with relatively fixed observation geometries, this agility significantly enlarges the set of feasible observation opportunities; however, it also increases the complexity of task scheduling [5]. This complexity becomes more pronounced in heterogeneous multi-satellite systems, where observation opportunities, maneuvering requirements, and onboard resources are no longer uniform across different platforms. In such a heterogeneous AEOS setting, scheduling starts from a set of geographically distributed observation tasks and satellite-specific resources. Each satellite has its own orbit, sensor coverage, maneuvering capability, energy budget, and storage capacity. Consequently, the same task may be observable only by a subset of satellites, and even when it is accessible to multiple satellites, it may correspond to different visibility windows, attitude-transition costs, and resource-consumption patterns. Based on these satellite-task relationships, the scheduler must determine task selection, satellite assignment, observation-window choice, and execution sequence while satisfying visibility, attitude-transition, temporal-conflict, energy, storage, and load-distribution constraints. Owing to the strong coupling among these decisions and constraints, heterogeneous AEOS scheduling is generally regarded as a highly constrained combinatorial optimization problem [6]. Early studies on Earth observation satellite scheduling mainly formulated the problem as task selection and sequencing under visibility-window and resource constraints. Lemaitre et al. [7] investigated the selection and scheduling of observations for agile satellites and showed that satellite agility increases both observation opportunities and scheduling complexity. Wolfe and Sorensen [8] compared dispatching, look-ahead, and genetic scheduling methods in the Earth observing systems domain. Bianchessi and Righini [9] studied planning and scheduling algorithms for the COSMO-SkyMed system, where acquisition and download operations must be jointly arranged. Habet et al. [10] further analyzed the difficulty of agile satellite observation scheduling and developed bounding techniques for this constrained problem. These studies established the basic modeling and algorithmic foundations of satellite scheduling. However, most early formulations mainly focused on task selection, sequencing, and visibility-window feasibility, while large-scale tasks, attitude transitions, coupled resources, and satellite heterogeneity were not fully emphasized. To improve solution quality and provide optimality guarantees, exact optimization methods have been developed for several AEOS scheduling formulations. Branch-and-bound, dynamic programming, mixed-integer programming, branch-and-price, and branch-and-cut-and-price methods can exploit mathematical structures and obtain high-quality or optimal solutions for medium-scale or specially structured cases [11, 12, 13, 14]. Peng et al. [13] interpreted AEOS scheduling as an orienteering problem with time-dependent profits and travel times. Recent branch-and-cut-and-price studies [14] further showed that time-dependent transition times and variable observation windows make AEOS scheduling substantially different from classical team orienteering problems. Nevertheless, exact methods usually face rapidly increasing computational costs as the numbers of satellites, tasks, candidate windows, and temporal coupling relationships grow. This limits their direct use in large-scale or repeatedly updated AEOS scheduling scenarios that require high-quality solutions within a limited computational budget. Fig. 1: Overall scheduling process for heterogeneous AEOS systems, including visibility analysis, satellite–task assignment, scheduling optimization, and final schedule generation. In addition to computational complexity, heterogeneous modeling remains insufficiently addressed in practical multi-satellite AEOS scheduling. Although recent studies have improved the modeling of visibility windows, transition times, and resource constraints, many formulations still focus on single-satellite or homogeneous multi-satellite settings [13, 17, 18, 19]. As illustrated in Fig. 1, heterogeneous AEOS scheduling involves visibility analysis, satellite–task assignment, scheduling optimization, and sequence generation under coupled temporal, attitude-transition, energy, storage, and load-distribution constraints. Since satellites may differ in orbital access, maneuvering capability, energy budget, and storage capacity, the same task can correspond to different feasible windows, maneuvering burdens, and resource-consumption patterns on different platforms. Optimization studies that explicitly preserve these satellite-dependent visibility, transition, and resource characteristics remain relatively limited. Because of these computational and modeling challenges, heuristic and metaheuristic algorithms have become widely used for AEOS scheduling and related mission-planning problems. Classical heuristics, such as greedy insertion, local search, and tabu search, together with metaheuristics, such as adaptive large neighborhood search, genetic algorithms, particle swarm optimization, ant colony optimization, differential evolution, and memetic algorithms, can produce feasible and competitive schedules more efficiently than exact methods in large-scale cases [15, 16, 17, 18, 19, 20]. Liu et al. [18] modeled agile satellite scheduling with time-dependent transition times and developed an adaptive large neighborhood search method. Wu et al. [19] proposed a data-driven improved genetic algorithm for AEOS scheduling with time-dependent transition times. These methods can incorporate problem-specific knowledge, including task priority, observation-window availability, transition feasibility, and resource constraints. However, many existing metaheuristics still rely on fixed operator probabilities, predefined neighborhood structures, or manually designed adaptive rules [21]. Under low population diversity, severe constraint violations, or insufficient capture of rare high-value tasks, such fixed or manually tuned mechanisms may lead to premature convergence, weak feasibility recovery, or unstable performance across different scenarios. Another related research direction is multi-objective or preference-oriented satellite scheduling. Multi-objective formulations have been used to optimize observation profit, energy consumption, image quality, failure risk, and load balance simultaneously [22]. Recent multi-objective AEOS scheduling studies have considered total observation profit and satellite energy consumption as separate objectives and solved the problem using strategy-fusion evolutionary optimization methods [23, 24, 25]. Song et al. [26] proposed a learning-guided NSGA-I for multi-objective satellite range scheduling. These studies demonstrate the value of multi-objective modeling and strategy fusion. However, Pareto-based methods usually return a set of non-dominated schedules rather than a single deployable schedule under a given operational preference. In practical mission planning, decision makers often require one schedule corresponding to a specific preference, such as gain-prioritized, energy-saving, or load-balanced scheduling. Therefore, a preference-adjustable scalar utility model is more convenient for operational decision making, provided that the individual utility components remain interpretable. Reinforcement learning (RL) has also been introduced into satellite scheduling to capture sequential decision-making patterns. Existing studies have combined RL with Monte Carlo tree search for on-board AEOS planning, modeled agile satellite scheduling as an MDP with value-based learning, and developed RL-based models for fair or quality-aware satellite scheduling [19, 27, 28]. More recent studies have employed deep RL architectures, attention-based networks, graph-based policies and RL-controlled neighborhood search to improve scheduling performance. These studies show the potential of RL for adaptive decision making in complex scheduling environments. However, most RL-based AEOS scheduling methods use the learned policy to directly construct schedules or control local neighborhood transformations. Such formulations often involve large discrete action spaces, strict feasibility constraints, sparse or unstable rewards, and substantial training costs. Moreover, a policy trained or tuned for one scenario may not generalize reliably when task density, satellite resources, visibility-window structures, or operational preferences change. Compared with direct RL schedulers, evolutionary and swarm intelligence algorithms provide mature population-based search mechanisms for constrained combinatorial optimization. Their global exploration capability and derivative-free nature make them suitable for AEOS scheduling without requiring a pre-trained policy. Nevertheless, conventional evolutionary algorithms still need effective control over operator selection, parameter adaptation, feasibility recovery, and local refinement. To this end, RL-assisted evolutionary algorithms have been investigated in broader optimization fields, where RL is commonly used for operator selection, parameter control, subpopulation management, or heuristic coordination [29]. Successful applications have been reported in photovoltaic model parameter identification, UAV path planning, distributed flexible job-shop scheduling, and vehicle scheduling [30, 31, 32]. These studies suggest that RL can improve the adaptability of evolutionary search without replacing the optimizer itself. However, in the AEOS scheduling field, the integration of RL and evolutionary optimization is still mainly limited to direct RL schedulers, learning-guided construction methods, or RL-controlled local neighborhood search. A systematic framework that embeds an online actor-critic policy into memetic evolutionary search for high-level AEOS-specific operator-mode selection remains insufficiently explored. To address these limitations, this paper develops an Evolutionary Policy Optimization (EPO) framework for heterogeneous AEOS scheduling. The proposed framework combines a preference-adjustable weighted scheduling model, decoder-based equivalent-cost evaluation, population-based evolutionary search, and online actor-critic operator-mode selection. The policy does not construct schedules directly; instead, it selects high-level search-operator modes that coordinate exploration, feasibility recovery, and schedule refinement. This design keeps the RL action space compact, preserves the strengths of population-based optimization, and provides a modular interface for incorporating additional constraints, utility components, and AEOS-specific operators. The main contributions of this paper are summarized as follows. 1. We formulate a preference-adjustable heterogeneous AEOS scheduling model that combines assignment-based indirect encoding, decoder-based schedule construction, and equivalent-cost evaluation. The model retains satellite-dependent visibility, attitude-transition, energy, storage, and load characteristics while integrating task gain, energy saving, and load balance into an interpretable weighted utility. 2. We propose a modular evolutionary policy optimization framework that separates schedule decoding, population-based search, and online actor-critic operator control. By selecting high-level operator modes rather than constructing schedules directly, the policy layer keeps the reinforcement-learning action space compact and supports extensible constraint, utility, and operator designs. 3. A RL-assisted operator-selection memetic evolutionary algorithm (RLOSMEA) is developed for heterogeneous AEOS scheduling. The algorithm adaptively coordinates global exploration, feasibility recovery, schedule refinement, and diversity maintenance, and its effectiveness is verified on different heterogeneous scheduling scenarios under different preference settings. The remainder of this paper is organized as follows. Section I presents the preference-adjustable weighted AEOS scheduling model, including the utility components, objective function, and constraints. Section I describes the proposed evolutionary policy optimization framework and the RLOSMEA algorithm. Section IV reports the experimental settings, comparative results, and sensitivity analyses. Section V concludes this paper and discusses future work. I Scheduling Model for Heterogeneous AEOS I-A Problem Description Heterogeneous AEOS scheduling aims to determine task selection, satellite assignment, observation-window selection, and execution sequencing under coupled visibility, temporal, attitude-transition, energy, and storage constraints. In the considered setting, satellites differ in orbital access, sensor availability, maneuvering capability, energy budget, and onboard storage capacity. Consequently, the same task may correspond to different feasible windows, transition requirements, and resource-consumption patterns on different satellites. To represent these heterogeneous characteristics while maintaining a compact search space, this paper adopts a preference-adjustable weighted-utility model with three normalized components, namely task gain, energy saving, and load balance. Candidate schedules are encoded indirectly as task-to-satellite assignment vectors, and a schedule decoder is used to determine feasible observation intervals and execution sequences according to satellite-specific visibility windows, task time windows, temporal conflicts, attitude-transition requirements, and onboard resource limits. For compatibility with the minimization interface of the optimizer, the weighted utility is further transformed into an equivalent cost by incorporating penalties for constraint violation and unscheduled feasible tasks. This decoder-centered formulation separates physical feasibility evaluation from search-space representation, thereby facilitating the incorporation of additional satellite-dependent resources, mission rules, and utility components. I-B Notation The main notation used in the proposed scheduling model is summarized in Table I. TABLE I: Main notation Symbol Description S Set of agile satellites T Set of observation tasks NsN_s, NtN_t Numbers of satellites and tasks xjx_j Assignment variable of task j yijy_ij Binary indicator that task j is assigned to satellite i ijW_ij Visibility window set between satellite i and task j EimaxE_i , DimaxD_i Energy budget and storage capacity of satellite i ρiobs _i^obs, ρitr _i^tr Observation and maneuver energy coefficients of satellite i bjb_j, fjf_j Start and finish times of task j djd_j Required observation duration of task j pjp_j Priority coefficient of task j UgU_g, UeU_e, UbU_b Gain, energy-saving, and load-balance utilities U_w Weighted scheduling utility CVnormCV_norm Normalized constraint violation NtfeasN_t^feas Number of tasks observable by at least one satellite I-C Assignment-Based Scheduling Representation An assignment-based indirect encoding is adopted to represent candidate schedules. Compared with a full mixed-integer representation, this encoding reduces the search dimension by leaving the exact observation timing and sequencing decisions to the schedule decoder. Each individual is represented by a task-to-satellite assignment vector: =[x1,x2,…,xNt],x=[x_1,x_2,…,x_N_t], (1) where xj∈0,1,…,Ns,∀j∈.x_j∈\0,1,…,N_s\, ∀ j . (2) Here, xj=0x_j=0 indicates that task j is not assigned to any satellite, whereas xj=ix_j=i indicates that task j is assigned to satellite i. For a compact mathematical description, the corresponding binary assignment indicator is defined as yij=1,xj=i,0,otherwise,i∈,j∈.y_ij= cases1,&x_j=i,\\ 0,&otherwise, cases i ,\;j . (3) After decoding, the scheduling state of task j is given by uj=1,if task j is scheduled,0,otherwise.u_j= cases1,&if task j is scheduled,\\ 0,&otherwise. cases (4) The assignment vector only specifies the candidate satellite for each task and does not directly determine the observation start time or execution order. Given x, the schedule decoder constructs the observation intervals and satellite-specific task sequences by enforcing visibility windows, task time windows, temporal-conflict constraints, attitude-transition feasibility, and onboard resource limits. I-D Utility Components The quality of a decoded schedule is evaluated using three normalized utility components: task gain, energy saving, and load balance. These components are introduced to represent different operational preferences in heterogeneous AEOS scheduling. The task-gain utility measures the value of the scheduled observations, the energy-saving utility reflects the resource efficiency of the schedule, and the load-balance utility characterizes the distribution of scheduling workload among satellites. I-D1 Task-Gain Utility The raw observation gain is defined as P()=∑j∈pjdjuj,P(x)= _j p_jd_ju_j, (5) where pjp_j and djd_j denote the priority coefficient and required observation duration of task j, respectively. The ideal maximum gain is given by Pmax=∑j∈pjdj.P = _j p_jd_j. (6) Accordingly, the normalized task-gain utility is defined as Ug()=P()Pmax+ε,U_g(x)= P(x)P + , (7) where ε is a small positive constant used to avoid division by zero. A larger Ug()U_g(x) indicates that more high-priority observation tasks are successfully scheduled. I-D2 Energy-Saving Utility Let Eiobs()E_i obs(x) and Eitr()E_i tr(x) denote the observation and attitude-transition energy consumptions of satellite i, respectively. In a heterogeneous satellite system, these quantities are satellite dependent because different platforms may have different observation-energy rates, maneuvering efficiencies, and available energy budgets. The total energy consumption of the schedule is expressed as Etot()=∑i∈(Eiobs()+Eitr()).E_ tot(x)= _i (E_i obs(x)+E_i tr(x) ). (8) The normalized energy-saving utility is then defined as Ue()=max0,1−Etot()Eref+ε,U_e(x)= \0,1- E_ tot(x)E_ ref+ \, (9) where ErefE_ ref is the energy-normalization scale. A larger Ue()U_e(x) corresponds to lower energy consumption and therefore better resource efficiency. I-D3 Load-Balance Utility Let Li()L_i(x) denote the normalized scheduling load of satellite i, which is calculated from its scheduled observation workload and resource usage relative to its service capability. The average load over all satellites is defined as L¯()=1Ns∑i∈Li(). L(x)= 1N_s _i L_i(x). (10) The load dispersion is given by DL()=∑i∈(Li()−L¯())2.D_L(x)= _i (L_i(x)- L(x) )^2. (11) The normalized load-balance utility is defined as Ub()=max0,1−DL()Ns−1Ns(∑i∈Li())2+ε.U_b(x)= \0,1- D_L(x) N_s-1N_s ( _i L_i(x) )^2+ \. (12) The denominator normalizes the maximum load dispersion under a fixed total load. Thus, a larger Ub()U_b(x) indicates a more balanced utilization of satellite resources. I-E Preference-Adjustable Weighted Objective To support different operational preferences, the three utility components are integrated through a preference weight vector: =(wg,we,wb),wg+we+wb=1,wg,we,wb≥0,w=(w_g,w_e,w_b),w_g+w_e+w_b=1, w_g,w_e,w_b≥ 0, (13) where wgw_g, wew_e, and wbw_b denote the weights assigned to task gain, energy saving, and load balance, respectively. The weighted scheduling utility is then defined as Uw()=wgUg()+weUe()+wbUb().U_w(x)=w_gU_g(x)+w_eU_e(x)+w_bU_b(x). (14) The theoretical scheduling objective is to maximize the weighted utility: maxUw(). _x\;U_w(x). (15) In the experiments, the best weighted utility obtained in one independent run is reported as Ubest=max∈ΩUw(),U_ best= _x∈ U_w(x), (16) where Ω denotes the set of candidate schedules generated during the search. By adjusting w, the same model can represent different scheduling preferences, such as gain-prioritized, energy-saving, or load-balanced operation. I-F Constraints and Constraint Handling A feasible schedule must satisfy task-assignment uniqueness, visibility-window feasibility, observation-duration consistency, temporal-conflict avoidance, attitude-transition feasibility, and onboard resource constraints. Since the proposed method adopts an indirect assignment encoding, these constraints are enforced and evaluated during schedule decoding rather than being explicitly embedded in the chromosome representation. I-F1 Task Assignment Constraint Each task can be assigned to at most one satellite: ∑i∈yij≤1,∀j∈. _i y_ij≤ 1, ∀ j . (17) This constraint is naturally satisfied by the integer assignment encoding in 2, where each task has only one assignment variable. I-F2 Visibility and Observation-Window Constraint Let WijW_ij denote the set of visibility windows between satellite i and task j. If task j is assigned to satellite i, at least one feasible visibility window must exist: yij≤vij,∀i∈,j∈,y_ij≤ v_ij, ∀ i ,\;j , (18) where vij=1v_ij=1 if Wij≠∅W_ij≠ , and vij=0v_ij=0 otherwise. Let [aj,cj][a_j,c_j] be the allowable time window of task j, and let djd_j be its required observation duration. For a scheduled task, there must exist a visibility window [ℓ,u]∈Wij[ ,u]∈ W_ij such that max(aj,ℓ)≤bj,fj=bj+dj≤min(cj,u). (a_j, )≤ b_j, f_j=b_j+d_j≤ (c_j,u). (19) This condition ensures that the decoded observation interval is simultaneously compatible with the task time window and the satellite-task visibility window. Fig. 2: Evolutionary policy optimization framework for heterogeneous AEOS scheduling I-F3 Temporal and Attitude-Transition Constraint For satellite i, let πi=(j1,j2,…,jmi) _i=(j_1,j_2,…,j_m_i) denote the decoded execution sequence. For any two consecutive tasks jkj_k and jk+1j_k+1 in πi _i, the following condition must hold: bjk+1≥fjk+τi,jk,jk+1,k=1,…,mi−1,b_j_k+1≥ f_j_k+ _i,j_k,j_k+1, k=1,…,m_i-1, (20) where τi,jk,jk+1 _i,j_k,j_k+1 denotes the required attitude-transition time from task jkj_k to task jk+1j_k+1 on satellite i. This constraint prevents temporal overlap and accounts for the maneuvering time required between consecutive observations. I-F4 Energy and Storage Constraints Let EimaxE_i and DimaxD_i denote the available energy budget and onboard storage capacity of satellite i over the planning horizon, respectively. Let Ei(t)E_i(t) and Di(t)D_i(t) denote the remaining energy and onboard data amount of satellite i at time t. The resource constraints are expressed as Ei(t)≥0,Di(t)≤Dimax,∀i∈,t∈[0,Tsim].E_i(t)≥ 0, D_i(t)≤ D_i , ∀ i ,\;t∈[0,T_ sim]. (21) Because satellites are heterogeneous, the energy budgets, storage capacities, and resource-consumption processes are evaluated separately for each satellite during decoding. I-F5 Constraint-Violation Measure and Equivalent Cost Instead of directly discarding infeasible schedules, a normalized constraint-violation measure is introduced to guide the search toward feasible regions. The total constraint violation is defined as CV()=CVwin+CVdur+CVtr+CVene+CVdata,CV(x)=CV_ win+CV_ dur+CV_ tr+CV_ ene+CV_ data, (22) where the five terms denote violations of the time-window, observation-duration, attitude-transition, energy, and data-storage constraints, respectively. A strictly feasible schedule satisfies CV()=0CV(x)=0, and the corresponding normalized violation is denoted by CVnorm()CV_ norm(x). For compatibility with the minimization interface of the optimizer, the weighted utility is converted into an equivalent cost: Cw()=1−Uw()+λcvCVnorm()+λunsRuns(),C_w(x)=1-U_w(x)+ _ cvCV_ norm(x)+ _ unsR_ uns(x), (23) where λcv _ cv and λuns _ uns are penalty coefficients. The unscheduled-task ratio is defined as Runs()=1−1Ntfeas+ε∑j∈uj,R_ uns(x)=1- 1N_t feas+ _j u_j, (24) where NtfeasN_t feas denotes the number of tasks observable by at least one satellite. The optimization problem solved by the proposed algorithm is therefore minCw(). _x\;C_w(x). (25) The schedule with the largest weighted utility is finally reported as the scheduling result. I EPO Framework and RLOSMEA I-A Framework Motivation and Overview The proposed EPO framework follows the assignment-based scheduling model in Section I. Each individual encodes a task-to-satellite assignment vector, and a shared schedule decoder converts it into observation intervals, satellite-specific execution sequences, resource consumption, and constraint-violation information. This decoder-centered design allows the optimizer to search in a compact assignment space while preserving the heterogeneous feasibility and resource characteristics of the original scheduling problem. As shown in Fig. 2, the framework combines population-based evolutionary search with online actor–critic policy control. The evolutionary module generates and updates candidate assignments, whereas the policy module observes the population state and selects a high-level operator mode for offspring generation and schedule modification. Instead of constructing schedules directly, the policy acts as an adaptive operator controller, which keeps the reinforcement-learning action space compact and preserves the global search capability of evolutionary optimization. RLOSMEA is developed as an AEOS-oriented implementation of the proposed framework. It integrates a memetic evolutionary backbone with online operator-mode selection and schedule-specific modification operators, including warm start, feasibility repair, conflict resolution, local improvement, elite intensification, immigrant injection, and diversity rescue. All schedule-oriented modifications are applied before decoding, so each offspring is evaluated only once by the shared decoder. This single-evaluation workflow is suitable for a fixed function-evaluation budget and supports the extension of additional problem-specific operators. I-B The MDP Formulation The adaptive operator-mode selection process is formulated as an online Markov decision process (MDP). To distinguish it from the AEOS scheduling model in Section I, the MDP for operator selection is defined as ℳRL=(,,,r,γ),M_ RL=(X,U,K,r,γ), (26) where X is the population-state space, U is the operator-mode action space, (ξt+1|ξt,αt)K( _t+1| _t, _t) denotes the transition kernel induced by action αt _t, r(ξt,αt,ξt+1)r( _t, _t, _t+1) is the reward function, and γ is the discount factor. At generation t, the actor–critic controller observes the population state ξt∈ _t and selects an operator-mode action αt∈ _t . After offspring generation, schedule modification, decoding evaluation, and environmental selection, the controller receives the immediate reward ϱt=r(ξt,αt,ξt+1). _t=r( _t, _t, _t+1). (27) Learning is performed online in each independent run without using an offline training dataset. The controller is updated generation by generation using the transition tuple (ξt,αt,ϱt,ξt+1).( _t, _t, _t, _t+1). (28) Thus, the operator-selection policy can adapt to the current search trajectory and scheduling scenario. I-C Population-State Representation The population state provides compact contextual information for online operator-mode selection. Instead of encoding a complete schedule, it summarizes the current search condition from four aspects: search progress, scheduling quality, feasibility, and population structure. The state vector observed by the actor–critic controller at generation t is written as ξt=[ξtp,ξtu,ξtf,ξts]T, _t= [ _t^p, _t^u, _t^f, _t^s ]^T, (29) where ξtp _t^p, ξtu _t^u, ξtf _t^f, and ξts _t^s denote the progress, utility, feasibility, and population-structure features, respectively. These features are used only to guide operator-mode selection and do not modify the scheduling objective defined in Section I. I-D Search-Phase Descriptor A search-phase descriptor is introduced to provide stage-dependent information for policy evaluation. The phase set is defined as ℋ=bootstrap,shaping,repair,intensify.H=\bootstrap,shaping,repair,intensify\. (30) The current phase is determined from the population state and recent search diagnostics, including progress, feasibility, constraint violation, diversity, priority capture, load balance, stagnation, and offspring success rate. The phase descriptor is encoded as a one-hot vector ψt _t and concatenated with the population state before action selection. The bootstrap phase promotes the rapid construction of promising assignment structures. The shaping phase emphasizes rare-task enhancement and structural adjustment. The repair phase is activated when feasibility is poor or constraint violation is high. The intensification phase is used when the population enters a promising region and local refinement becomes more beneficial. A rule-based recovery mode is retained as a safeguard against severe stagnation or diversity collapse. When recovery is triggered, the selected action αt _t can be replaced by an exploration-oriented or restructuring action in U. Such recovery-forced actions are not used to update the actor policy, thereby preventing externally forced decisions from biasing policy learning. I-E Operator-Mode Action Space The actor policy selects a high-level operator-mode action rather than a direct scheduling decision. The action space is defined as =α1,α2,α3,α4,α5,α6.U=\ _1, _2, _3, _4, _5, _6\. (31) The six operator modes are summarized in Table I. Each action represents a coordinated search mode for modifying offspring assignment vectors. After action αt _t is selected, it is decoded into the operator-control vector t=(pc,t,pm,t,t), _t=(p_c,t,p_m,t, η_t), (32) where pc,tp_c,t and pm,tp_m,t denote the crossover and mutation parameters, respectively, and t η_t denotes the vector of schedule-specific operator rates. These rates control the activation intensities of warm start, feasibility repair, conflict resolution, local improvement, elite intensification, immigrant injection, rare-task shaping, and diversity rescue. TABLE I: Operator-mode actions Action Mode Main Function α1 _1 gain-bootstrap Gain construction α2 _2 rare-shaping Rare-task enhancement α3 _3 repair-feasible Feasibility recovery α4 _4 cluster-reorder Conflict resolution α5 _5 elite-intensify Elite exploitation α6 _6 diverse-explore Diversity recovery The action decoder follows a sparse-control principle. For each selected action, one dominant problem-oriented operator is activated, whereas the remaining operators are retained as weak background safeguards. This design improves the interpretability of the selected action and provides a clearer reward signal for policy learning. I-F Actor–Critic Policy Optimization Let πθ(α|ξt,ψt) _θ(α| _t, _t) denote the actor policy parameterized by θ. The input feature vector for the actor–critic controller is constructed as ϕt=[ξt;ψt;1], φ_t=[ _t; _t;1], (33) where the last entry is a bias term. The actor computes the action logits as t=ϕt+ψt,o_t=W φ_t+b_ _t, (34) where W is the actor parameter matrix, and ψtb_ _t is the phase-dependent bias vector. A temperature-controlled softmax function is first used to obtain the preliminary policy: πθ0(α|ξt,ψt)=exp(oα,t/σ)∑α′∈exp(oα′,t/σ), _θ^0(α| _t, _t)= (o_α,t/σ) _α (o_α ,t/σ), (35) where σ is the temperature parameter. The phase prior qψt(α)q_ _t(α) is then incorporated as πθ(α|ξt,ψt)=πθ0(α|ξt,ψt)qψt(α)∑α′∈πθ0(α′|ξt,ψt)qψt(α′). _θ(α| _t, _t)= _θ^0(α| _t, _t)q_ _t(α) _α _θ^0(α | _t, _t)q_ _t(α ). (36) To maintain exploration, an ϵε-mixed policy is adopted: π~θ(α|ξt,ψt)=(1−ϵ)πθ(α|ξt,ψt)+ϵ1||. π_θ(α| _t, _t)=(1-ε) _θ(α| _t, _t)+ε 1|U|. (37) The critic estimates the state value using a linear approximation: Vω(ξt,ψt)=Tϕt,V_ω( _t, _t)=v^T φ_t, (38) where v is the critic parameter vector. After action αt _t is applied and the next state is obtained, the temporal-difference error is computed as δt=ϱt+γVω(ξt+1,ψt+1)−Vω(ξt,ψt). _t= _t+γ V_ω( _t+1, _t+1)-V_ω( _t, _t). (39) For numerical stability, δt _t is clipped within a finite interval in the implementation. For a policy-sampled action, the actor and critic are updated as ←+αaδt(αt−~t)ϕtT,W + _a _t(e_ _t- π_t) φ_t^T, (40) ←+αcδtϕt,v + _c _t φ_t, (41) where αa _a and αc _c are the actor and critic learning rates, respectively, αte_ _t is the one-hot vector of the selected action, and ~t π_t is the mixed action-probability vector. If action αt _t is forced by the recovery mode, the actor update is skipped to avoid biasing the learned policy with safeguard decisions. I-G Reward Function The reward function is designed to align online operator-mode selection with the weighted scheduling objective in Section I. It consists of four terms: weighted-utility improvement, best-cost improvement, mean-cost improvement, and offspring success rate. Feasibility, diversity, and stagnation are included in the state and phase descriptors rather than being directly added as separate reward terms. Let UtU_t and Ut+1U_t+1 denote the population-level weighted utilities before and after applying the selected operator mode. The weighted-utility improvement is defined as ΔUt=Ut+1−Ut. U_t=U_t+1-U_t. (42) Since the optimizer minimizes the equivalent cost Cw()C_w(x), the best- and mean-cost improvements are defined as ΔCtbest=Ctbest−Ct+1bestmax(|Ctbest|,εc), C_t best= C_t best-C_t+1 best (|C_t best|, _c), (43) ΔCtmean=Ctmean−Ct+1meanmax(|Ctmean|,εc), C_t mean= C_t mean-C_t+1 mean (|C_t mean|, _c), (44) where CtbestC_t best and CtmeanC_t mean denote the best and mean equivalent costs of the population at generation t, respectively. The offspring success rate is defined as Qtsucc=1|t|∑i∈t(Cw(i)<median(Cw()|∈t)),Q_t succ= 1|Q_t| _x_i _tI (C_w(x_i)< median (\C_w(x)|x _t\ ) ), (45) where tP_t and tQ_t denote the parent and offspring populations, respectively, and (⋅)I(·) is the indicator function. The immediate reward is then written as ϱt=clip(∑j=14λjΓj(t),−1,1), _t= clip ( _j=1^4 _j _j(t),-1,1 ), (46) where Γ1(t)=tanh(ΔUts1),Γ2(t)=tanh(ΔCtbests2), _1(t)= ( U_ts_1 ), _2(t)= ( C_t bests_2 ), (47) Γ3(t)=tanh(ΔCtmeans3),Γ4(t)=tanh(Qtsucc−q0s4). _3(t)= ( C_t means_3 ), _4(t)= ( Q_t succ-q_0s_4 ). (48) This compact reward preserves consistency with both the weighted-utility objective and the equivalent-cost minimization interface. I-H Policy-Controlled Evolutionary Search Algorithm 1 RLOSMEA Under the EPO Framework 0: Scheduling instance ℐI, population size N, maximum function evaluations NFEmaxN_FE , and weight vector w. 0: Best scheduling solution ∗x^*. 1: Initialization: 2: Prepare problem-guidance information G, including task priorities, task durations, feasible satellites, and feasible-window counts. 3: Initialize and evaluate the population 0P_0; extract the initial population state ξ0 _0. 4: Initialize the incumbent best solution ∗x^* and set NFE=|0|N_FE=|P_0|. 5: Initialize the actor–critic controller with actor parameters W, phase-dependent bias, critic parameter v, exploration rate ϵε, and temperature σ. 6: Initialize search diagnostics and set t=0t=0. 7: Evolutionary policy search: 8: while NFE<NFEmaxN_FE<N_FE do 9: Determine the search-phase descriptor ψt _t from the current population state and search diagnostics. 10: Construct the feature vector ϕt=[ξt;ψt;1] φ_t=[ _t; _t;1] by (33). 11: Compute the phase-aware mixed policy by (35)–(37), and sample an operator-mode action αt∈ _t . 12: if severe stagnation or diversity collapse is detected then 13: Activate the recovery mode and replace αt _t with an exploration-oriented or restructuring action. 14: end if 15: Decode αt _t into the operator-control vector t _t by (32). 16: Generate offspring tQ_t through evolutionary variation and apply schedule-specific modifications according to t _t. 17: Preserve elite individuals and evaluate each modified offspring once using the shared schedule decoder. 18: Update NFE←NFE+|t|N_FE← N_FE+|Q_t|. 19: Merge tP_t and tQ_t, and perform feasibility-aware environmental selection to obtain t+1P_t+1. 20: Update the incumbent best solution ∗x^* according to the decoded scheduling utility. 21: Extract the next population state ξt+1 _t+1 and compute the immediate reward ϱt _t by (46)–(48). 22: Compute the temporal-difference error δt _t by (39), and update the critic by (41). 23: if αt _t is not forced by the recovery mode then 24: Update the actor by (40). 25: end if 26: Record search diagnostics and set t←t+1t← t+1. 27: end while 28: Output: Return the best scheduling solution ∗x^*. After action αt _t is selected, the action decoder maps it to the operator-control vector t _t, which specifies the offspring-generation and refinement strategy. Preliminary offspring are first generated through mating selection, crossover, and mutation, and are then refined by schedule-specific operators, including warm start, feasibility repair, conflict resolution, local improvement, elite intensification, immigrant injection, and diversity rescue. All schedule-specific modifications are performed before objective evaluation. Thus, each offspring is decoded and evaluated only once by the shared schedule decoder, avoiding repeated evaluations after individual sub-operators. This workflow is suitable for fixed function-evaluation budgets and maintains a clear interface between operator control and schedule evaluation. After evaluation, the parent and offspring populations are merged and processed by feasibility-aware environmental selection. Feasible individuals are ranked according to the equivalent cost, whereas infeasible individuals are ranked according to normalized constraint violation. The selected individuals form the next population t+1P_t+1 and define the next state ξt+1 _t+1. Algorithm 1 summarizes the overall procedure. I-I Methodological Discussion The proposed EPO framework separates evolutionary search from policy learning. The evolutionary component searches over assignment vectors and relies on the decoder for feasibility checking and equivalent-cost evaluation. The policy component adjusts the search behavior by selecting high-level operator modes, rather than constructing schedules directly. In RLOSMEA, the population-state representation, phase descriptor, and compact reward jointly enable the controller to switch among construction, repair, exploitation, and exploration under a fixed function-evaluation budget. The optimizer operates on the equivalent cost, whereas scheduling performance is reported using the weighted utility. This separation keeps the optimization interface compatible with minimization-based evolutionary search while retaining an interpretable utility-based evaluation of scheduling quality. IV Simulation Experiments and Analysis IV-A Experimental Settings To evaluate the effectiveness and scalability of the proposed method, six heterogeneous AEOS scheduling scenarios, denoted as scenario_01–scenario_06, are constructed. As listed in Table I, the number of candidate tasks increases from 100 to 350, and the number of agile satellites increases from 5 to 12. The increasing task and satellite scales enlarge the assignment space and strengthen the coupling among task selection, satellite assignment, observation-window selection, and execution sequencing. TABLE I: Heterogeneous AEOS scenarios Scenario NtaskN_task NsatN_sat scenario_01 100 5 scenario_02 150 5 scenario_03 200 7 scenario_04 250 9 scenario_05 300 10 scenario_06 350 12 (a) Scenario_01 (b) Scenario_02 (c) Scenario_03 (d) Scenario_04 (e) Scenario_05 (f) Scenario_06 Fig. 3: Weighted-utility convergence curves in six representative heterogeneous scenarios. The solid curves show the mean results over 30 independent runs, and the shaded bands indicate the corresponding standard deviations. (a) scenario_02 (b) scenario_04 (c) scenario_06 Fig. 4: Component-wise utility evolution in representative heterogeneous scenarios. The solid curves show the mean results over 30 independent runs, and the shaded bands indicate the corresponding standard deviations. (a) scenario_01 (b) scenario_03 (c) scenario_05 Fig. 5: Mean weighted utility under different weight settings in three representative heterogeneous scenarios. The solid curves show the mean results over 30 independent runs, and the shaded bands indicate the corresponding standard deviations. (a) scenario_01 (b) scenario_03 (c) scenario_05 Fig. 6: Component-wise response of RLOSMEA under different weight settings in three representative heterogeneous scenarios. Fig. 7: Parameter sensitivity of RLOSMEA on representative heterogeneous scenario_04. Each marker denotes the mean objective value over repeated runs, and the shaded band indicates the corresponding standard deviation. (a) Optimization progress (b) Policy action mixture Fig. 8: RL diagnostics of RLOSMEA in representative scenario_04 Each task is associated with a ground target, a priority value, an observation duration, and a task time window. Satellite–task visibility windows are generated before optimization by considering orbital motion, target motion, sensor-access constraints, and Earth rotation. Each satellite follows a near-circular low-Earth orbit, with altitude and inclination defined as hi=450+50mod(i−1,5)h_i=450+50 (i-1,5) km and Ii=30+10mod(i−1,6)I_i=30+10 (i-1,6) deg, respectively. Initial longitudes and orbital phases are randomly distributed within each scenario to diversify ground-track coverage. The simulation horizon is 86400 s, the preprocessing time step is 1 s, and the maximum observation deviation angle is set to 30 deg. Only satellite–task pairs with valid visibility windows are retained. Heterogeneity is introduced through both orbital access and onboard resources. Different satellite altitudes, inclinations, initial longitudes, and orbital phases lead to different visibility-window sets, while satellite-specific energy budgets, storage capacities, and resource-consumption parameters result in different execution costs. Thus, the same task may have different feasibility conditions, attitude-transition requirements, and resource-consumption patterns on different satellites. During decoding, each assignment is evaluated under the corresponding satellite parameters and must satisfy visibility, observation-duration, temporal-conflict, attitude-transition, energy, and storage constraints. The target set contains both static and moving targets, where 15% of the targets are static and the remaining targets move at speeds up to 45 m/s. Task priorities are uniformly sampled from the integer range [1,5][1,5]. All compared algorithms use the same scenario files, visibility windows, decoder, objective function, constraint-evaluation rules, and stopping criterion. The compared methods include the Memetic Evolutionary Algorithm (MemeticEA)[33], Adaptive Large Neighborhood Search (ALNS)[19], Adaptive Exploration State-Space Particle Swarm Optimization (AESSPSO)[34], Grey Wolf Optimizer (GWO)[35], and the proposed RLOSMEA. The baseline parameter settings follow the corresponding original references and commonly used default configurations, without scenario-specific tuning. For RLOSMEA, all actor–critic and operator-control parameters are fixed across the different scenarios. Specifically, the actor–critic parameters are (actorLR,criticLR)=(0.035,0.08)(actorLR,criticLR)=(0.035,0.08) and (epsilon,temperature)=(0.10,0.82)(epsilon,temperature)=(0.10,0.82). The operator-control base rates are (immigrantBase,repairBase)=(0.10,0.10)(immigrantBase,repairBase)=(0.10,0.10) and (clusterBase,localBase)=(0.10,0.07)(clusterBase,localBase)=(0.10,0.07). Each algorithm is independently run 30 times for each scenario with a budget of 10000 objective evaluations. The convergence curves report the average best-so-far weighted utility, and the shaded bands denote the corresponding standard deviations. All experiments are conducted on a 64-bit Windows platform equipped with an Intel(R) Core(TM) Ultra 9 285H CPU at 2.90 GHz and 64 GB RAM. IV-B Comparative Results in Different Heterogeneous Scenarios This subsection compares RLOSMEA with AESSPSO, ALNS, GWO, and MemeticEA on six heterogeneous AEOS scheduling scenarios. Since all algorithms obtain feasible schedules in all scenarios, the comparison focuses on weighted utility, convergence behavior, and component-wise scheduling performance. IV-B1 Convergence Behavior Fig. 3 shows the best-so-far weighted-utility convergence curves in the six scenarios. The solid curves denote the mean values over 30 independent runs, and the shaded bands indicate the corresponding standard deviations. RLOSMEA achieves the highest weighted utility in all scenarios and maintains this advantage during most stages of the search. AESSPSO and ALNS usually show rapid early improvement but tend to stagnate earlier, whereas GWO remains less competitive in most scenarios. MemeticEA is the strongest baseline, indicating the benefit of local refinement; however, RLOSMEA still obtains consistently better final utilities. This behavior is closely related to the heterogeneous scheduling structure. Because visibility opportunities, attitude-transition costs, and resource budgets vary across satellites, assigning high-value tasks only to a few favorable platforms can quickly introduce conflicts and resource pressure. The policy-guided operator selection in RLOSMEA helps exploit favorable assignments while maintaining the ability to redistribute tasks when feasibility or diversity becomes limiting. The relatively narrow shaded bands further indicate that the proposed method has stable run-to-run performance. IV-B2 Component-Wise Utility Analysis Fig. 4 presents the evolution of the three utility components in representative small-, medium-, and large-scale scenarios. The component-wise results provide further insight into the source of the weighted-utility improvement. The main advantage of RLOSMEA comes from task-gain utility. Across the representative scenarios, RLOSMEA consistently schedules more valuable task-assignment structures than the compared methods. Meanwhile, it maintains a high load-balance utility, indicating that the improvement is not obtained by excessively concentrating tasks on a small number of satellites. This property is important for heterogeneous constellations, where satellites with better access opportunities or stronger resource capacities may otherwise be overused. The energy-saving utility is not always the best among all methods. This result is consistent with the gain-oriented weight setting used in the main comparison, where a moderate increase in energy consumption can be acceptable if it leads to a larger overall weighted utility. Therefore, the advantage of RLOSMEA should be interpreted as a better global tradeoff among task gain, energy consumption, and load distribution, rather than simultaneous dominance in every individual component. IV-B3 Discussion Overall, RLOSMEA achieves the best weighted utility across all six heterogeneous scenarios and preserves its advantage as the problem scale increases. The improvement mainly results from stronger task-gain capture while maintaining acceptable energy-saving and load-balance performance. These results indicate that online operator-mode selection improves both solution quality and convergence stability for preference-adjustable heterogeneous AEOS scheduling. IV-C Weight Sensitivity Weight sensitivity is examined on scenario 01, scenario 03, and scenario 05, which represent small-, medium-, and large-scale heterogeneous scheduling cases, respectively. The tested weight vectors gradually shift the scheduling preference from gain-dominant operation to a more balanced tradeoff among task gain, energy saving, and load balance. For all plots, the solid curves denote the mean results over 30 independent runs, and the shaded bands indicate the corresponding standard deviations. IV-C1 Overall Performance Across Weight Settings Fig. 5 shows the weighted-utility results under different preference weight settings. RLOSMEA achieves the best performance across all tested weights in the three representative scenarios. This indicates that its advantage is not tied to a particular manually selected weight vector, but remains stable when the objective emphasis changes. Although the absolute weighted-utility values vary with the scalarization weights, the ranking pattern is generally consistent. RLOSMEA remains ahead of the compared methods when the preference shifts from task-gain-oriented scheduling toward a more balanced consideration of gain, energy consumption, and load distribution. The relatively narrow shaded bands further suggest that this advantage is repeatable across independent runs. IV-C2 Component-Wise Response Under Different Preferences Fig. 6 shows the response of the three utility components obtained by RLOSMEA under different weight settings. As the gain weight decreases and the energy-saving and load-balance weights increase, the task-gain utility decreases gradually, whereas the energy-saving utility improves. The load-balance utility remains at a high level over the tested range. These trends are consistent with the intended role of the preference-adjustable weighted model. RLOSMEA does not simply optimize one dominant component under all settings; instead, it adjusts the tradeoff among task gain, energy consumption, and load distribution according to the prescribed weights. The same pattern is observed in scenario 01, scenario 03, and scenario 05, indicating that the preference response is not limited to a specific problem scale. IV-C3 Discussion The weight-sensitivity results lead to two observations. First, RLOSMEA maintains superior weighted utility under all tested preference settings, showing robust performance with respect to scalarization changes. Second, the component-wise responses vary smoothly and consistently with the weight vector, suggesting that the proposed method can adjust its search emphasis without destabilizing the decoded schedules. These results support the use of RLOSMEA for preference-adjustable multi-objective scheduling in heterogeneous AEOS systems. IV-D Parameter Sensitivity and Learning Behavior After the cross-scenario and weight-sensitivity comparisons, this subsection further examines the internal behavior of RLOSMEA on scenario 04. The analysis focuses on parameter sensitivity and the learning dynamics of the RL-assisted operator-selection mechanism. IV-D1 Parameter Sensitivity Fig. 7 reports the parameter-sensitivity results of RLOSMEA on scenario 04. Each marker denotes the mean objective value over repeated runs, and the shaded band indicates the corresponding standard deviation. The results show that RLOSMEA maintains stable performance over the tested parameter ranges. The actor learning rate, critic learning rate, and exploration rate only lead to mild variations in the objective value, indicating that the proposed method does not rely on delicate parameter tuning. The softmax temperature has a relatively stronger influence because it directly controls the sharpness of action selection. The conflict-cluster and local-improvement rates also affect the results to some extent, reflecting the importance of structural refinement in heterogeneous scheduling. Nevertheless, no abrupt performance degradation is observed within the tested ranges. This suggests that the default parameter setting lies in a relatively stable region rather than near an isolated optimum. IV-D2 Learning Behavior of Operator Selection Fig. 8 illustrates the learning behavior of RLOSMEA on scenario 04, including optimization progress and the policy action mixture. Fig. 8(a) shows that the weighted utility increases steadily during the search. The gain utility is improved while the load-balance utility remains at a high level, indicating that the algorithm improves the scalar objective mainly by capturing more valuable task assignments without seriously damaging the workload distribution among satellites. Fig. 8(b) shows that the selected operator modes remain diversified throughout the optimization process. No single action dominates the entire search, and several operator modes remain active at different stages. This indicates that the actor–critic controller does not degenerate into a fixed operator pattern. Instead, it adjusts the operator-selection tendency according to the current search condition, thereby maintaining a balance among exploration, feasibility recovery, and exploitation. IV-D3 Discussion The parameter and learning analyses provide two observations. First, RLOSMEA is robust to moderate variations in key control parameters, which supports its practical applicability under different scheduling settings. Second, the RL-assisted controller maintains a diversified and adaptive operator-selection pattern rather than repeatedly selecting one dominant operator. These results support the use of online actor–critic control for coordinating search operators in preference-adjustable heterogeneous AEOS scheduling. IV-E Summary The experimental results demonstrate the effectiveness of RLOSMEA for heterogeneous AEOS scheduling. Across different scenarios, the proposed method achieves higher weighted utility and more stable convergence than the representative metaheuristic baselines. The component-wise analysis shows that the improvement mainly comes from stronger task-gain capture while maintaining acceptable energy-saving and load-balance performance. The weight-sensitivity study further shows that RLOSMEA remains competitive under different preference settings, and the utility components respond consistently to changes in the weight vector. In addition, the parameter-sensitivity and learning-behavior analyses indicate that the method is robust to moderate parameter variations and that the online actor–critic controller provides adaptive operator coordination during the search. Overall, these results verify the effectiveness of the proposed evolutionary policy optimization framework in preference-adjustable heterogeneous AEOS scheduling. V Conclusion This paper investigated heterogeneous AEOS scheduling under coupled visibility, attitude-transition, energy, storage, and load-distribution constraints. A preference-adjustable weighted scheduling model was established by combining assignment-based indirect encoding, decoder-based schedule construction, and equivalent-cost evaluation, thereby preserving satellite-dependent feasibility and resource characteristics while integrating task gain, energy saving, and load balance into an interpretable scalar utility. Based on this model, an evolutionary policy optimization framework was proposed, where reinforcement learning selects high-level operator modes rather than directly constructing schedules. The resulting RLOSMEA coordinates exploration, feasibility recovery, local refinement, and diversity maintenance through online actor–critic operator-mode selection. Experiments on different heterogeneous AEOS scheduling scenarios showed that RLOSMEA achieved higher weighted utility and more stable convergence than representative metaheuristic baselines under the same function-evaluation budget. The component-wise, sensitivity, and learning-behavior analyses further verified the robustness of the proposed method and the effectiveness of reinforcement-learning-guided operator selection. Future work will consider dynamic and uncertain scheduling conditions, including emergency task insertion, cloud-induced visibility changes, and rolling-horizon replanning. References [1] S. Kai, C. YingWu, and W. Pei, “Agile earth observing satellites mission scheduling for disaster and environment monitoring,” Research Journal of Chemistry and Environment, vol. 16, no. 2, p. 139–146, NOV 2012. [2] S. Wang, D. Zhou, M. Sheng, W. Yue, W. Zhang, and B. Pulatov, “Satellite Remote Sensing Mission Scheduling for Ecological Monitoring: A Learning-Based Multi-region Collaborative Approach,” IEEE Transactions on Aerospace and Electronic Systems, vol. 62, p. 4570–4585, 2026. [3] X. Shen, Z. Lu, L. Li, Y. Chen, X. Li, J. Wang, and Y. Wei, “Multistrip Stitching Imaging Mission Planning Method for SAR Satellite Regional Mapping Considering Onboard Energy Consumption,” IEEE Transactions on Geoscience and Remote Sensing, vol. 63, p. 1–13, 2025. [4] Y. Gu, C. Han, Y. Chen, S. Liu, and X. Wang, “Large region targets observation scheduling by multiple satellites using resampling particle swarm optimization,” IEEE Transactions on Aerospace and Electronic Systems, vol. 59, no. 2, p. 1800–1815, 2023. [5] R. Kandepi, H. Saini, R. K. George, S. Konduri, and R. Karidhal, “Agile earth observation satellite constellations scheduling for large area target imaging using heuristic search,” Acta Astronautica, vol. 219, p. 670–677, 2024. [6] L. Wang, Y. Xiang, H. Huang, D. Li, C. Gao, and S. Liu. “Towards Realistic Earth-Observation Constellation Scheduling: Benchmark and Methodology,” in Advances in Neural Information Processing Systems, vol. 38, p. 85923–85944, 2025. [7] M. Lemaitre, G. Verfaillie, F. Jouhaud, J. M. Lachiver, and N. Bataille, “Selecting and scheduling observations of agile satellites,” Aerospace Science and Technology, vol. 6, no. 5, p. 367–381, 2002. [8] W. J. Wolfe and S. E. Sorensen, “Three Scheduling Algorithms Applied to the Earth Observing Systems Domain,” Management Science, vol. 46, no. 1, p. 148–166, 2000. [9] N. Bianchessi and G. Righini, “Planning and scheduling algorithms for the COSMO-SkyMed constellation,” Aerospace Science and Technology, vol. 12, no. 7, p. 535–544, 2008. [10] D. Habet, M. Vasquez, and Y. Vimont, “Bounding the optimum for the problem of scheduling the photographs of an Agile Earth Observing Satellite,” Computational Optimization and Applications, vol. 47, no. 2, p. 307–333, 2010. [11] L. O. Seman, C. A. Rigo, E. Camponogara, P. Munari, and E. A. Bezerra, “Improving energy aware nanosatellite task scheduling by a branch-cut-and-price algorithm,” Computers & Operations Research, vol. 158, p. 106292, 2023. [12] C. A. Rigo, L. O. Seman, E. Camponogara, E. Morsch Filho, E. A. Bezerra, and P. Munari, “A branch-and-price algorithm for nanosatellite task scheduling to improve mission quality-of-service,” European Journal of Operational Research, vol. 303, no. 1, p. 168–183, 2022. [13] G. Peng, R. Dewil, C. Verbeeck, A. Gunawan, L. Xing, and P. Vansteenwegen, “Agile earth observation satellite scheduling: An orienteering problem with time-dependent profits and travel times,” Computers & Operations Research, vol. 111, p. 84–98, 2019. [14] G. Peng, J. Wang, G. Song, A. Gunawan, L. Xing, and P. Vansteenwegen, “Branch-and-Cut-and-Price for Agile Earth Observation Satellite Scheduling,” European Journal of Operational Research, vol. 326, no. 3, p. 427–438, 2025. [15] C. Han, Y. Gu, G. Wu, and X. Wang, “Simulated annealing-based heuristic for multiple agile satellites scheduling under cloud coverage uncertainty,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 53, no. 5, p. 2863–2874, 2023. [16] R. Kandepi, H. Saini, R. K. George, S. Konduri, and R. Karidhal, “Agile earth observation satellite constellations scheduling for large area target imaging using heuristic search,” Acta Astronautica, vol. 219, p. 670–677, 2024. [17] X. Wang, G. Wu, L. Xing, and W. Pedrycz, “Agile earth observation satellite scheduling over 20 years: Formulations, methods, and future directions,” IEEE Systems Journal, vol. 15, no. 3, p. 3881–3892, 2021. [18] X. Liu, G. Laporte, Y. Chen, and R. He, “An adaptive large neighborhood search metaheuristic for agile satellite scheduling with time-dependent transition time,” Computers & Operations Research, vol. 86, p. 41–53, 2017. [19] Y. Du, T. Wang, B. Xin, L. Wang, Y. Chen, and L. Xing, “A data-driven parallel scheduling approach for multiple agile earth observation satellites,” IEEE Transactions on Evolutionary Computation, vol. 24, no. 4, p. 679–693, 2020. [20] H. Wang, W. Huang, S. Magnússon, T. Lindgren, R. Wang, and Y. Song, “A Strategy Fusion-Based Multiobjective Optimization Approach for Agile Earth Observation Satellite Scheduling Problem,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, p. 1–14, Art. no. 5930214, 2024. [21] Y. He, L. Xing, Y. Chen, W. Pedrycz, L. Wang, and G. Wu, “A generic markov decision process model and reinforcement learning method for scheduling agile earth observation satellites,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 3, p. 1463–1474, 2022. [22] M. Qin, X. Zhao, Z. Xu, et al., “A multi-objective scheduling method for agile satellites based on nonlinear utility and deep reinforcement learning,” International Journal of Digital Earth, vol. 19, no. 1, Art. no. 2643501, 2026. [23] L. Wei, M. Chen, L. Xing, Q. Wan, Y. Song, Y. Chen, and Y. Chen, “Knowledge-transfer based genetic programming algorithm for multi-objective dynamic agile earth observation satellite scheduling problem,” Swarm and Evolutionary Computation, vol. 85, p. 101460, 2024. [24] H. Chen, Y. Tian, W. Pedrycz, G. Wu, R. Wang, and L. Wang, “Hyperplane assisted evolutionary algorithm for many-objective optimization problems,” IEEE Transactions on Cybernetics, vol. 50, no. 7, p. 3367–3380, 2020. [25] Y. Du, L. Xing, J. Zhang, Y. Chen, and Y. He, “Moea based memetic algorithms for multi-objective satellite range scheduling problem,” Swarm and Evolutionary Computation, vol. 50, p. 100576, 2019. [26] Y. Song, X. Ma, X. Li, L. Xing, and P. Wang, “Learning-guided nondominated sorting genetic algorithm i for multi-objective satellite range scheduling problem,” Swarm and Evolutionary Computation, vol. 49, p. 194–205, 2019. [27] Y. Zuo, M. Chen, X. Liu, Y. Du, A. Qamar, and Y. Shang, “A Deep Reinforcement Learning-Based Self-Repair Method for Solving the Agile Satellite Scheduling Problem,” Tsinghua Science and Technology, vol. 31, no. 1, p. 180–198, 2026. [28] A. Herrmann and H. Schaub, “Reinforcement Learning for the Agile Earth-Observing Satellite Scheduling Problem,” IEEE Transactions on Aerospace and Electronic Systems, vol. P, p. 1–13, 2023. [29] Y. Song, Y. Wu, Y. Guo, R. Yan, P. N. Suganthan, Y. Zhang, W. Pedrycz, S. Das, R. Mallipeddi, O. S. Ajani, and Q. Feng, “Reinforcement learning-assisted evolutionary algorithm: A survey and research opportunities,” Swarm and Evolutionary Computation, vol. 86, Art. no. 101517, p. 1–?, 2024. [30] X. Zhang, S. Xia, X. Li, and T. Zhang, “Multi-objective particle swarm optimization with multi-mode collaboration based on reinforcement learning for path planning of unmanned air vehicles,” Knowledge-Based Systems, vol. 250, Art. no. 109075, 2022. [31] Z.-Q. Zhang, F.-C. Wu, B. Qian, R. Hu, L. Wang, and H.-P. Jin, “A Q-learning-based hyper-heuristic evolutionary algorithm for the distributed flexible job-shop scheduling problem with crane transportation,” Expert Systems with Applications, vol. 234, Art. no. 121050, 2023. [32] B. Zhou and Z. Zhao, “An adaptive artificial bee colony algorithm enhanced by Deep Q-Learning for milk-run vehicle scheduling problem based on supply hub,” Knowledge-Based Systems, vol. 264, Art. no. 110367, 2023. [33] F. Neri and C. Cotta, “Memetic algorithms and memetic computing optimization: A literature review,” Swarm and Evolutionary Computation, vol. 2, p. 1–14, 2012. [34] M. Alimohammadi and M.-R. Akbarzadeh-T., “State-space adaptive exploration for explainable particle swarm optimization,” Swarm and Evolutionary Computation, vol. 94, Art. no. 101868, 2025. [35] S. Mirjalili, S. M. Mirjalili, and A. Lewis. “Grey Wolf Optimizer,” Advances in Engineering Software, vol. 69, p. 46–61, 2014.