Paper deep dive
SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining
Dengdi Sun, Xiaoya Zhou, Xiao Wang, Wanli Lyu, Jin Tang, Bin Luo
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-scale pre-training has improved the generalization of neural operators across diverse PDEs. However, existing PDE foundation models still struggle with heterogeneous dynamics, where shared representations may cause knowledge interference, while mixture-of-experts (MoE) architectures suffer from increasing expert redundancy. We propose SPEAR, a spectral-disentangled MoE neural operator with knowledge-guided expert aggregation for large-scale PDE pre-training. SPEAR decouples latent features into low- and high-frequency components, enabling shared modeling of transferable dynamics and specialized learning of PDE-specific patterns. To address expert redundancy, we design a knowledge-guided expert aggregation strategy that measures expert similarity from dataset-specific learned knowledge and routing preferences, enabling the identification and consolidation of similar experts. Experiments on twelve PDE datasets and multiple downstream benchmarks demonstrate superior performance in pre-training, fine-tuning, and transfer learning. Furthermore, our aggregation strategy reduces the number of experts by 50\% while maintaining or improving prediction accuracy, achieving a balance between model efficiency and generalization for PDE foundation models.
Tags
Links
- Source: https://arxiv.org/abs/2610.03265v1
- Canonical: https://arxiv.org/abs/2610.03265v1
Trouble viewing inline? Open PDF directly →
Full Text
47,533 characters extracted from source content.
Expand or collapse full text
SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining Dengdi Sun Xiaoya Zhou Xiao Wang* Wanli Lyu Jin Tang Bin Luo Abstract Large-scale pre-training has improved the generalization of neural operators across diverse PDEs. However, existing PDE foundation models still struggle with heterogeneous dynamics, where shared representations may cause knowledge interference, while mixture-of-experts (MoE) architectures suffer from increasing expert redundancy. We propose SPEAR, a spectral-disentangled MoE neural operator with knowledge-guided expert aggregation for large-scale PDE pre-training. SPEAR decouples latent features into low- and high-frequency components, enabling shared modeling of transferable dynamics and specialized learning of PDE-specific patterns. To address expert redundancy, we design a knowledge-guided expert aggregation strategy that measures expert similarity from dataset-specific learned knowledge and routing preferences, enabling the identification and consolidation of similar experts. Experiments on twelve PDE datasets and multiple downstream benchmarks demonstrate superior performance in pre-training, fine-tuning, and transfer learning. Furthermore, our aggregation strategy reduces the number of experts by 50% while maintaining or improving prediction accuracy, achieving a balance between model efficiency and generalization for PDE foundation models. 1. School of Artificial Intelligence, Anhui University, Hefei 230601, China 2. School of Computer Science and Technology, Anhui University, Hefei 230601, China Introduction Neural operators have emerged as a compelling alternative for solving PDEs by learning mappings between infinite-dimensional function spaces, achieving orders-of-magnitude speedups at inference time. However, existing neural operators are typically trained under single-task settings—specific PDE families with fixed parameter ranges—resulting in limited generalization to unseen equation types, parameter regimes, or boundary conditions. Inspired by large language models (Vaswani et al. 2017) and vision foundation models (Radford et al. 2021), recent efforts have introduced large-scale pre-training into the PDE domain to learn transferable physical representations. While pre-training enhances generalization, different PDEs exhibit substantially different underlying dynamics, and modeling multiple physical processes within a single shared parameter space may lead to knowledge conflicts and negative transfer. To scale up modeling capacity while maintaining inference efficiency, recent work has introduced Mixture-of-Experts (MoE) architectures (Shazeer et al. 2017) into neural operator design (Wang et al. 2026; Sun et al. 2026). MoEs employ dynamic routing mechanisms that activate only a subset of expert subnetworks for each input, enabling massive parameter scaling without proportionally increasing per-sample computation. This paradigm shows promise in handling PDE heterogeneity: different experts can specialize in different equation types, physical regimes, or numerical patterns. However, MoE-based neural operators still face the following two critical challenges. As shown in Figure 1, we summarize the motivation and overview of SPEAR. Challenge 1: Spectral Coupling of PDEs. Different PDE families differ drastically in their physical behaviors. For example, elliptic equations describe steady-state equilibrium, with high-frequency features manifesting as smooth local extrema; hyperbolic equations describe wave propagation, with high-frequency features manifesting as sharp discontinuities; parabolic equations describe dissipative processes, with high-frequency features manifesting as decaying spikes. These differences are most pronounced in high-frequency local patterns, while low-frequency dynamics exhibit broad similarity across equation types. However, Fourier-based neural operators such as AFNO (Guibas et al. 2021) couple all frequency components within shared parameters, forcing morphologically distinct high-frequency patterns to coexist and causing severe gradient interference. Existing remedies, including multi-scale operators (Liu et al. 2024) and frequency enhancement modules (Chen et al. 2025), have yet to break this spectral coupling constraint. Therefore, a mechanism that can separate shared dynamics from specific patterns according to frequency characteristics is urgently needed. Challenge 2: Expert Redundancy of MoE. While MoE architectures enhance model capacity through specialization, this comes at a cost: as model scale grows, parameter redundancy proliferates, with many experts learning functionally similar representations while inference overhead increases significantly. Existing expert merging methods (Zhang et al. 2026; Li et al. 2026; Li et al. 2024; Chen et al. 2024a) primarily consolidate similar experts based on routing statistics or expert outputs, rather than the task-specific knowledge semantics that experts truly acquire. This "behavior-based" rather than "knowledge-based" merging strategy lacks physical grounding. In PDE learning, this is particularly problematic: two experts that handle similar physical patterns may never be activated together due to different data distributions, and would thus be judged as "dissimilar" and left unmerged. Figure 1: Motivation of SPEAR. The upper shows the advantage of frequency decoupling for separating shared and PDE-specific dynamics. The lower highlights knowledge-guided expert merging over routing-based merging. It is worth noting that these two challenges are not independent. A key fact connects them: low-frequency dynamics are broadly transferable across PDE types and can be effectively captured by a shared backbone, while high-frequency patterns are PDE-specific and demand expert specialization. This observation naturally suggests a two-stage strategy: first, decouple frequencies to separate sharable low-frequency knowledge from PDE-specific high-frequency patterns, assigning the latter to a dedicated MoE module for expert-level specialization; then, consolidate the high-frequency experts to eliminate redundancy, ensuring that each preserved expert captures truly distinct physical knowledge. In other words: decouple to specialize, then aggregate to refine. Building on this insight, in this paper, we present a SPectral-disentangled MoE neural operator with knowledge-guided Expert AggRegation (SPEAR). Our framework introduces two tightly coupled innovations. For challenge 1, we propose a spectral-disentangled architecture that explicitly partitions the frequency spectrum. The low-frequency branch employs AFNO to capture global evolution patterns, while the high-frequency branch extracts fine-grained local structures via high-pass filtering and models them through an MoE module composed of shared convolutional backbones and LoRA adapters. Dynamic routing based on high-frequency statistics activates a part of experts to efficiently learn local dynamics across PDEs. For challenge 2, we design a knowledge-guided expert aggregation strategy that compresses and reorganizes functionally similar experts by incorporating physical correlations among PDE tasks and expert preference information. Rather than routing statistics, expert similarity is measured by the learned parameters for each PDE dataset. Then, a dataset-level knowledge matrix is constructed and fused via bilinear projection, yielding semantic similarities for expert merging. Our main contributions are summarized as follows: 1) We design a spectral-disentangled architecture that explicitly partitions the frequency spectrum: a shared backbone captures low-frequency dynamics across all PDE types, while a dedicated MoE module handles PDE-specific high-frequency patterns. 2) We propose a knowledge-guided expert aggregation strategy that computes similarities among experts based on their acquired physical knowledge during training and routing preference information, and merges them via bilinear projection and hierarchical clustering. 3) Extensive experiments on large-scale PDE pretraining datasets show that our method learns transferable representations, achieving strong performance across cross-equation pretraining, fine-tuning, and generalization to unseen PDEs with superior parameter efficiency. Related Work Neural Operators for PDE Solving Neural operators learn mappings between infinite-dimensional function spaces (Lu et al. 2021; Li et al. 2020). Various neural operators have been developed for PDE modeling. For example, DeepONet (Lu et al. 2021) adopts a branch–trunk architecture, while the Fourier Neural Operator (FNO) (Li et al. 2020) models global dependencies in the Fourier domain. More recently, methods such as MPP (McCabe et al. 2023), DPOT (Hao et al. 2024), Poseidon (Herde et al. 2024), OmniArch (Chen et al. 2024b), and UniSolver (Zhou et al. 2025) have explored pre-training on diverse PDE datasets, advancing PDE foundation models. However, most existing methods adopt a unified architecture. As PDE datasets become larger and more diverse, differences in physical laws and dynamics may cause knowledge interference across heterogeneous tasks. Therefore, modeling diverse physical knowledge remains an important challenge in large-scale PDE pre-training. Mixture-of-Experts Models Mixture-of-Experts (MoE) expands model capacity through sparse expert activation and has been widely adopted in language and vision models. Representative works include Sparsely-Gated MoE (Shazeer et al. 2017), Switch Transformer (Fedus et al. 2022), and Mixtral (Jiang et al. 2024). By activating only a small subset of experts for each input, MoE increases model capacity with limited additional computation. Recently, MoE has also been introduced into neural operator models. Methods such as MoE-POT (Wang et al. 2026) and NESTOR (Sun et al. 2026) use specialized experts to capture diverse physical knowledge and alleviate interference across heterogeneous PDE tasks. However, some experts may learn similar physical representations, leading to knowledge redundancy and inefficient parameter utilization. Therefore, reducing redundant experts while preserving model performance remains an important challenge in MoE-based PDE modeling. Expert Merging Expert merging is an emerging compression technique that identifies experts with similar functions or representations and integrates them through parameter fusion. Representative methods include SMoE (Li et al. 2024) and HC-SMoE (Chen et al. 2024a), which construct expert relationships from routing patterns or output features and then merge redundant experts. However, most existing methods lack explicit modeling of task semantics and domain knowledge, making it difficult to accurately characterize the knowledge learned by different experts. In PDE modeling, MoE encourages functional specialization, while knowledge overlap may still exist among some experts. Incorporating physical knowledge into expert relationship modeling can therefore improve the identification of truly redundant experts and enable effective compression while preserving expert diversity, which is important for building efficient and scalable PDE foundation models. Figure 2: Overview architecture. The left panel shows the overall pipeline, the upper-right panel presents the FreqMix block, and the lower-right panel illustrates the high-frequency MoE module with low-rank experts. The Proposed Approach We propose SPEAR, a spectral-disentangled MoE neural operator with knowledge-guided expert aggregation, as shown in Fig. 2. SPEAR decouples shared low-frequency dynamics from PDE-specific high-frequency patterns, where the MoE module enables expert specialization for local dynamics. Parameter-efficient design and knowledge-guided expert aggregation further reduce expert redundancy. Preliminaries We consider parameterized PDEs over a spatial domain Ω⊂ℝn ^n and a time interval [0,T][0,T]: ∂t−ℋ(,∇,∇2,…,θ)=0, ∂ t-H(u, ,∇^2u,…;θ)=0, (1) subject to (,0)=u0(),∈Ω,ℬ[](,t)=g(,t),(,t)∈∂Ω×[0,T], casesu(x,0)=u_0(x),&x∈ ,\\ B[u](x,t)=g(x,t),&(x,t)∈∂ ×[0,T], cases (2) where u is the system state, ℋH is the governing operator, θ denotes the physical parameters, and ℬB is the boundary operator. In the data-driven setting, a neural operator wT_w approximates the temporal evolution: ut+1=w(ut−K+1,…,ut),u_t+1=T_w (u_t-K+1,…,u_t ), (3) where w denotes the learnable parameters. Following DPOT (Hao et al. 2024), small random perturbations are added to the input sequence during training to reduce error accumulation and improve long-term prediction robustness. Input Encoding and Time Aggregation Given an input PDE sequence ∈ℝB×H×W×T×Cx ^B× H× W× T× C, we concatenate it with the coordinate grid and reshape it into ~∈ℝ(BT)×(C+3)×H×W x ^(BT)×(C+3)× H× W. Following the patch embedding of ViT (Dosovitskiy et al. 2020), the input is encoded as =PatchEmbed(~)+pos∈ℝ(BT)×D×H′×W′,X=PatchEmbed( x)+E_pos ^(BT)× D× H × W , (4) where H′=H/PHH =H/P_H, W′=W/PWW =W/P_W, and EposE_pos denotes the learnable positional embedding. Then we reshape X into B×H′×W′×T×DB× H × W × T× D and employ a time aggregation layer (Hao et al. 2024) to integrate temporal information: b,h,w=∑t=1Tb,h,w,tt,Y_b,h,w= _t=1^TX_b,h,w,tW_t, (5) where t∈ℝD×DW_t ^D× D is the learnable projection matrix for the t-th time step. The aggregated output is ∈ℝB×H′×W′×DY ^B× H × W × D. Spectral-Disentangled Neural Operator Spatiotemporal physical fields contain smooth global structures and sharp local variations. Therefore, we introduce a spectral-disentangled dual-branch architecture that transforms features into the frequency domain and separates low- and high-frequency components. The low-frequency branch captures shared dynamics, while the high-frequency branch models PDE-specific patterns with MoE. Low-Frequency Global Branch After Fourier transformation of Y to extract low-frequency feature l∈ℝB×C×H×Wx_l ^B× C× H× W, We employ AFNO (Guibas et al. 2021) as the shared expert for low-frequency modeling. The input is first transformed into the frequency domain as ^l=ℱ(l) x_l=F(x_l). The channel dimension is divided into N blocks, and only the first K low-frequency modes are processed using a two-layer complex-valued MLP: :K,:K=2σ(1^1:K,:K+1)+2,z_:K,:K=W_2σ (W_1 x_1_:K,:K+b_1 )+b_2, (6) where 1W_1 and 2W_2 are block-wise complex-valued weights, and σ denotes the activation function. The remaining frequency modes are set to zero, and the transformed features are mapped back to the spatial domain with a residual connection (He et al. 2016), yielding lf=ℱ−1()+lx_lf=F^-1(z)+x_l. High-Frequency Local Branch To capture sharp local variations, such as small-scale vortices and shock boundaries, we apply a hard high-pass filter in the frequency domain. Given high-frequency feature x_h, ^h=ℱ() x_h=F(x_h), a binary mask ∈0,1Hf×WfM∈\0,1\^H_f× W_f is defined as (i,j)=0,i<Kandj<K,1,otherwise.M(i,j)= cases0,&i<K\ and\ j<K,\\ 1,&otherwise. cases (7) The high-frequency component is then obtained as xhf=ℱ−1(^⊙)x_hf=F^-1( x ) and fed into the MoE module to model fine-grained local structures and complex dynamics. Low-Rank Adapter-based Mixture of Experts High-frequency components, such as small-scale vortices and shock boundaries, are difficult to model with a single architecture. We therefore introduce a low-rank adapter-based MoE module that uses dynamic routing to enhance local high-frequency modeling. It contains 12 non-shared experts and one base expert, with the gate activating the Top-2 non-shared experts for each input. (1) Routing Strategy. Since the MoE module processes high-frequency features, the router captures local variations by removing the spatial mean: hf=−MeanH,W()x_hf=x-Mean_H,W(x). A texture-aware routing representation is then constructed by combining channel activation and spatial variation: b=[MeanH,W(b);StdH,W(hf,b)]∈ℝ2C.g_b= [Mean_H,W(x_b);Std_H,W(x_hf,b) ] ^2C. (8) This representation captures both channel responses and local fluctuations. The routing probabilities are computed as b=Softmax(b+)p_b=Softmax(Wg_b+b), where ∈ℝN×2CW ^N× 2C and ∈ℝNb ^N are learnable parameters. Following Top-2 routing (Shazeer et al. 2017), the two experts with the highest probabilities are selected. Let ℐbI_b denote their index set. Their normalized routing weights are p^b(i)=αpb(i)∑j∈ℐbpb(j),i∈ℐb, p_b^(i)=α p_b^(i) _j _bp_b^(j), i _b, (9) where α is the routing scale factor. (2) Depthwise Separable Convolution. To capture local details in high-frequency PDE features, we adopt depthwise separable convolution (DSC) (Howard et al. 2017) as the shared backbone of all experts. Given ∈ℝB×C×H×Wx ^B× C× H× W, DSC applies depthwise convolution for spatial modeling and pointwise convolution for channel fusion, formulated as c=cdw∗cz_c=W^dw_c*x_c and =pw∗u=W^pw*z. Each convolution is followed by Group Normalization (Wu and He 2018) and GELU (Hendrycks and Gimpel 2016). With the residual connection, the DSC backbone is defined as ℱDSC()=+PW(DW()).F_DSC(x)=x+PW(DW(x)). (10) (3) Base Expert. Although PDEs exhibit distinct dynamics, their high-frequency components may share structures. We introduce a shared expert to learn transferable high-frequency representations across PDEs via parameter sharing (Caruana 1997). The base expert adopts a residual depthwise separable convolutional block, formulated as shared=+ℱshared()y_shared=x+F_shared(x). It provides stable shared representations while reducing redundant learning among experts. (4) Specific Experts. The specific experts capture PDE-specific high-frequency patterns using a shared depthwise separable convolutional backbone and expert-specific low-rank adapters (Hu et al. 2022). For the i-th expert, the adapter consists of a down-projection i:C→rB_i:C→ r and an up-projection i:r→CA_i:r→ C, formulated as i=ℱbase()+i(i()).y_i=F_base(x)+A_i\! (B_i(x) ). (11) The low-rank residual introduces expert-specific transformations with limited parameter overhead. The up-projection iA_i is initialized to zero, making each non-shared expert initially equivalent to the shared backbone and improving training stability. (5) Routing and Feature Fusion. The shared expert provides general high-frequency representations across PDE systems, while the selected non-shared experts capture PDE-specific local patterns. For each input sample, the gating network selects the Top-2 non-shared experts. Let ℐbI_b denote the selected expert set for the b-th sample. The selected expert outputs are combined as moe,b=∑i∈ℐbp^b(i)i,b,y_moe,b= _i _b p_b^(i)y_i,b, (12) where p^b(i) p_b^(i) is the normalized routing weight. The specialized and shared representations are fused using a learnable coefficient: fusion=λmoe+(1−λ)shared,λ=Sigmoid(β).y_fusion= _moe+(1-λ)y_shared,\;λ=Sigmoid(β). (13) Finally, a global residual connection preserves the input feature information as =+fusiony=x+y_fusion. This design combines shared and expert-specific representations for effective high-frequency PDE modeling. Knowledge-Guided Expert Aggregation Then, we propose a knowledge-guided expert aggregation strategy that groups experts learning similar high-frequency representations, reducing parameter and memory overhead while preserving expert specialization, as shown in Fig. 3. Expert Similarity To measure expert similarity, we combine dataset-specific knowledge with expert activation preferences. Suppose the model is trained on D datasets with N non-shared experts. The average routing probabilities form a dataset–expert preference matrix ∈ℝD×NR ^D× N. Let 0v_0 denote the pretrained parameters and dv_d the parameters fine-tuned on dataset d. The dataset-specific physical knowledge is represented by Δd=d−0 _d=v_d-v_0. Their pairwise cosine similarities define D(d1,d2)=Δd1⊤Δd2‖Δd1‖2‖Δd2‖2.S_D(d_1,d_2)= _d_1 _d_2\| _d_1\|_2\| _d_2\|_2. (14) We then project the dataset-level similarities into the expert space: Eraw=⊤D∈ℝN×N.S_E^raw=R S_DR ^N× N. (15) To reduce the effect of activation scale, we normalize it as E(i,j)=Eraw(i,j)Eraw(i,i)Eraw(j,j)+ε,S_E(i,j)= S_E^raw(i,j) S_E^raw(i,i)S_E^raw(j,j)+ , (16) where ε ensures stability. Larger E(i,j)S_E(i,j) indicates stronger knowledge similarity between experts i and j. Expert Aggregation Strategy Based on SES_E, we apply threshold-constrained average-linkage hierarchical clustering (Murtagh and Contreras 2012). Each expert is initially treated as an individual cluster, and the similarity between clusters aC_a and bC_b is Sim(a,b)=1|a||b|∑i∈a∑j∈bE(i,j).Sim(C_a,C_b)= 1|C_a||C_b| _i _a _j _bS_E(i,j). (17) The most similar clusters are iteratively merged until their maximum similarity falls below a threshold τ. Experts within each cluster are aggregated according to their global routing frequencies. For expert i∈ki _k, the normalized weight is ωi=fi/∑j∈kfj _i=f_i/ _j _kf_j, where fif_i denotes its global activation frequency. The aggregated adapter parameters are obtained as ¯k=∑i∈kωii. θ_k= _i _k _i θ_i. (18) The resulting aggregated experts replace the original experts, and the routing mapping is updated accordingly for inference. Figure 3: Knowledge-guided expert merging based on bilinear projection and hierarchical clustering. Loss Functions Load Balancing Loss To promote balanced expert utilization, we adopt the load balancing loss (Shazeer et al. 2017): ℒaux=E∑i=1Epifi,L_aux=E _i=1^Ep_if_i, (19) where pip_i and fif_i denote the routing probability and activation frequency of expert i, respectively. Main Task Loss We use the relative L2L_2 reconstruction error (L2RE) (Li et al. 2020) as the main objective: ℒL2RE=‖^i(c)−i(c)‖2‖i(c)‖2.L_L2RE= \| u_i^(c)-u_i^(c) \|_2 \|u_i^(c) \|_2. (20) Orthogonality Loss To encourage expert diversity, we minimize the cosine similarity between the up-projection parameters of different non-shared experts: ℒorth=1|Ω|∑(i,j)∈Ω(i⊤j‖i‖2‖j‖2)2,L_orth= 1| | _(i,j)∈ ( A_i A_j\|A_i\|_2\|A_j\|_2 )^2, (21) where Ω=(i,j)∣i<j =\(i,j) i<j\. Overall Loss The overall objective is ℒtotal=ℒL2RE+λ1ℒaux+λ2ℒorth.L_total=L_L2RE+ _1L_aux+ _2L_orth. (22) We set λ1=0.001 _1=0.001. The coefficient λ2 _2 decays from 0.10.1 to 0.010.01 over the first 400 epochs and reaches 00 at epoch 450. Experiments Datasets and Baselines We evaluate our method on pre-training, fine-tuning, alternative data splits, unseen PDE transfer, model scaling, ablation studies, and expert merging. Datasets. We construct a mixed pre-training set of 12 PDE datasets with diverse equation types and parameter settings from FNO (Li et al. 2020), PDEBench (Takamoto et al. 2022), PDEArena (Gupta and Brandstetter 2022), and CFDBench (Luo et al. 2023). Baselines. We compare against representative neural operators and PDE foundation models, including FNO (Li et al. 2020), UNet (Ronneberger et al. 2015), GK-T (Cao 2021), GNOT (Hao et al. 2023), OFormer (Li et al. 2022), MPP (McCabe et al. 2023), DPOT (Hao et al. 2024), UniSolver (Zhou et al. 2025), MoE-POT (Wang et al. 2026), NESTOR (Sun et al. 2026), and Poseidon (Herde et al. 2024). UNet, GK-T, GNOT, and OFormer are trained separately on each dataset, while the others follow their original pre-training and transfer settings. Table 1: The experiments are divided into four parts: standard small models, pre-training performance of foundation models, fine-tuning results, and model merging results. The evaluation metric is L2RE. The best result within each part is highlighted in bold, while the overall best result is emphasized in blue bold. L2RE Activated FNO-ν PDEBench CNS-(η,ζη,ζ), DR, SWE PDEArena CFDBench Model Params 1e-5 1e-4 1e-3 1,0.1 1,0.01 Avg.(1) 0.1,0.1 0.1,0.01 Avg.(0.1) DR SWE NS NS-cond - Small model FNO 0.5M 0.116 0.0922 0.0156 0.151 0.108 0.130 0.230 0.076 0.153 0.0321 0.0091 0.210 0.384 0.0274 UNet 25M 0.198 0.119 0.0245 0.334 0.291 0.313 0.569 0.357 0.463 0.0971 0.0521 0.102 0.337 0.209 FFNO 1.3M 0.121 0.0503 0.0099 0.0212 0.0520 0.0366 0.162 0.0452 0.104 0.0571 0.0116 0.0839 0.602 0.0071 GK-T 1.6M 0.134 0.0792 0.0098 0.0341 0.0377 0.0359 0.0274 0.0366 0.0320 0.0359 0.0069 0.0952 0.423 0.0105 GNOT 1.8M 0.157 0.0443 0.0125 0.0325 0.0420 0.0373 0.0228 0.0341 0.0285 0.0311 0.0068 0.172 0.325 0.0088 OFormer 1.9M 0.171 0.0645 0.0104 0.0417 0.0625 0.0521 0.0254 0.0205 0.0229 0.0192 0.0072 0.135 0.332 0.0102 Pre-trained MPP 7M - - - - - 0.0442 - - 0.0312 0.0168 0.0066 - - - DPOT 7M 0.0976 0.0606 0.0095 0.0173 0.0397 0.0285 0.0132 0.0220 0.0176 0.0321 0.0056 0.125 0.384 0.0095 NESTOR 13M 0.1195 0.0951 0.0093 0.0167 0.0373 0.0270 0.0120 0.0202 0.0161 0.0308 0.0052 0.132 0.409 0.0112 Unisolver 12M 0.0600 0.0626 0.0148 0.0464 0.1172 0.0818 0.2510 0.1170 0.1840 0.7276 2.6851 0.1306 0.5649 0.0627 Ours 11M 0.0960 0.0770 0.0087 0.0120 0.0372 0.0246 0.0116 0.0184 0.0150 0.0403 0.0048 0.127 0.361 0.0115 Fine-tune DPOT-FT500 7M 0.0520 0.0367 0.0058 0.0112 0.0195 0.0153 0.0174 0.0138 0.0156 0.0148 0.0024 0.0910 0.280 0.0039 NESTOR-FT500 13M 0.0505 0.0217 0.0043 0.0094 0.0134 0.01140 0.0123 0.0083 0.0103 0.0117 0.0026 0.0683 0.285 0.0038 Ours-FT500 11M 0.0543 0.0197 0.0040 0.0090 0.0137 0.01135 0.0132 0.0082 0.0107 0.0213 0.0045 0.0637 0.291 0.0038 Merge Ours-merge 11M 0.0541 0.0192 0.0040 0.0090 0.0137 0.01135 0.0127 0.0079 0.0102 0.0197 0.0040 0.0637 0.279 0.0037 Ours-OPCM 11M 1.5940 1.2863 1.4523 5.0913 1.4027 3.2470 35.4204 4.1313 19.7657 0.8393 0.8393 1.4812 1.3098 0.7957 Ours-SMOE 11M 0.0550 0.0200 0.0044 0.0094 0.0141 0.0118 0.0123 0.0084 0.0104 0.0120 0.0045 0.0640 0.2930 0.0037 Main Results Table 1 compares our method across pre-training, fine-tuning, and expert merging. Overall, our method achieves the best performance on 8 of the 12 tasks. Pre-training Stage. Our method achieves the best performance on 7 tasks, including 5 of the 6 PDEBench tasks, while maintaining low L2RE on most others. This demonstrates that the proposed MoE architecture effectively learns shared physical knowledge across heterogeneous PDEs with high parameter efficiency. Fine-tuning Stage. We fine-tune the pre-trained model for 500 epochs on each downstream dataset. The resulting model achieves the best performance on 6 tasks. Compared with training from scratch, pre-training enables faster adaptation and higher prediction accuracy, indicating strong transferability across PDE tasks. Expert Merging. We further compress the fine-tuned model by aggregating 12 non-shared experts into 6 according to their knowledge-guided similarities. Despite a 50% reduction in expert count, Ours-Merge maintains or improves performance on most tasks. For example, L2RE on PDEArena NS-cond decreases from 0.291 to 0.281, demonstrating nearly lossless compression. In contrast, OPCM (Tang et al. 2026) suffers substantial degradation as the number of merged experts increases, with L2RE increasing by tens or even hundreds of times on some tasks. This suggests that generic merging methods struggle to preserve functional differences among experts. Our method instead identifies redundant experts using cross-task knowledge and routing preferences, reducing redundancy while preserving expert diversity and critical PDE knowledge. Overall, the pre-training and fine-tuning paradigm improves performance and adaptation across PDE tasks, while knowledge-guided expert aggregation enables efficient compression with minimal performance degradation, demonstrating cross-task generalization and parameter efficiency. Table 2: Performance comparison of different methods under the new dataset split. The evaluation metric is L2RE. The best result within each column is highlighted in bold. Model Activated FNO-ν PDEBench CFDBench L2RE Params 1e-5 1e-3 0.1,0.01 SWE DR - FNO 10M 0.1910 0.0245 0.0859 11.0000 0.5300 0.0060 DPOT 7M 0.1070 0.0155 0.0168 0.0063 0.0577 0.0067 MoE-POT 17M 0.0682 0.0077 0.0105 0.0064 0.0411 0.0053 NESTOR 13M 0.0674 0.0076 0.0159 0.0045 0.0184 0.0091 Ours 10M 0.0582 0.0051 0.0165 0.0024 0.0145 0.0056 Table 3: Comparison of generalization and transfer ability. Model Activated Params Wave-Layer Wave-Gauss Turbulence FNO 0.5M - - 0.193 MPP 7M - - 0.152 DPOT 7M 0.0095 0.0075 0.135 Poseidon 21M 0.29 0.29 - MoE-POT 17M 0.05 0.06 0.0621 Ours 10M 0.0085 0.0066 0.0662 Figure 4: Visualization of the predicted physical field at the final time step for Wave-Gauss. Figure 5: Visualization of the predicted physical field at the final time step for Wave-Layer. Experiments on New PDE Dataset Splits To evaluate our model under different pre-training data configurations, we select six subsets from the original 12 PDE datasets and compare our method with state-of-the-art pre-trained models. The results are shown in Table 2. Our method achieves the best performance on four of the six datasets and remains competitive on the others, consistently outperforming pre-trained models on FNO-ν, SWE, and DR. Even with the pre-training data size reduced by half, our method learns transferable physical representations from limited and heterogeneous PDE data, demonstrating strong downstream generalization. Downstream Task Experiments To evaluate generalization and transferability, we evaluate our model on a high-resolution 2D turbulence dataset and two unseen PDEs, Wave-Gauss and Wave-Layer. The results are reported in Table 3 and Figures 4–6. Our method achieves the best performance on both unseen PDEs. It obtains L2RE values of 0.0085 on Wave-Layer and 0.0066 on Wave-Gauss, achieving relative improvements of approximately 10.5% and 12.0% over the best baselines, respectively. It also achieves the second-best result on high-resolution turbulence and outperforms most baselines. These results demonstrate that large-scale pre-training enables transferable representations and improves adaptation to unseen PDEs and complex downstream dynamics. Figure 6: Visualization of the predicted physical variables at the final time step for 2D high-resolution turbulence. Effectiveness Analysis of Expert Aggregation To evaluate the proposed aggregation strategy, we visualize the key matrices involved and compare downstream performance before and after aggregation. Figure 7(1) shows the dataset similarity matrix SDS_D , computed from Δvi=vi−v0 v_i=v_i-v_0. Clear block structure is observed: datasets with similar physical mechanisms exhibit higher similarity, while different equation types show weaker correlations, indicating that the learned knowledge captures dataset-specific characteristics. Figure 7(2) presents the dataset–expert preference matrix R. The sparse and locally concentrated pattern indicates that different datasets activate distinct experts, confirming that routing promotes expert specialization. The knowledge-guided expert similarity matrix SES_E, obtained by combining SDS_D and R, is shown in Figure 7(3). High-similarity regions reflect knowledge overlap among experts, guiding the aggregation of redundant experts while preserving essential knowledge. As shown in Table 1, reducing the number of experts from 12 to 6 achieves 50% compression with negligible performance loss, and even improves performance on certain tasks (e.g., PDEArena NS-cond L2RE decreases from 0.291 to 0.281). Overall, the strategy reduces parameter redundancy and improves representation consistency by merging experts with similar learned knowledge, balancing efficiency and generalization. Figure 7: Visualization of the key matrices used in Knowledge-Guided expert merging: (1) dataset similarity matrix SDS_D, (2) dataset–expert preference matrix R, and (3) expert similarity matrix SES_E. Table 4: The impact of the number of non-shared experts. Num. of experts 1e-5 1e-3 0.1, 0.01 SWE DR CFDBench Avg. ℒ2L_2 4 0.0666 0.0054 0.0162 0.0029 0.0193 0.0079 0.0197 8 0.0679 0.0049 0.0170 0.0029 0.0187 0.0083 0.0200 12 0.0690 0.0050 0.0146 0.0026 0.0190 0.0076 0.0196 16 0.0689 0.0048 0.0165 0.0040 0.0192 0.0077 0.0202 20 0.0700 0.0049 0.0170 0.0028 0.0197 0.0092 0.0206 24 0.0693 0.0049 0.0162 0.0031 0.0187 0.0082 0.0201 Scaling Experiments We pre-train on six subsets of the 12 PDE datasets and vary the number of non-shared experts while fixing the number of activated experts. As shown in Table 4, 12 non-shared experts achieve the best performance. Increasing the expert scale does not always improve performance, as excessive experts complicate routing and may weaken expert collaboration. Therefore, an appropriate expert scale is essential for MoE modeling. Table 5: Ablation experiments of our proposed model. “w/o” denotes the removal of the corresponding component. Method 1e-5 1e-3 0.1, 0.01 SWE DR CFDBench Avg. ℒ2L_2 Promotion Ours 0.0690 0.0050 0.0148 0.0027 0.0196 0.0077 0.0198 - w/o MoE 0.0709 0.0054 0.0165 0.0031 0.0196 0.0079 0.0206 0.0008 w/o High-Freq 0.1070 0.0155 0.0168 0.0063 0.0577 0.0067 0.0350 0.0152 w/o Aux Loss 0.0694 0.0055 0.0154 0.0027 0.0185 0.0098 0.0202 0.0004 w/o Orth Loss 0.0692 0.0051 0.0168 0.0026 0.0189 0.0083 0.0202 0.0004 Ablation Studies We conduct ablation studies on six PDEBench subtasks to evaluate the contribution of each key component in the proposed framework. The results are reported in Table 5. High-Frequency Branch. Removing the high-frequency branch increases the average L2RE by 0.0152, causing the largest performance degradation among all variants. This confirms its critical importance in capturing fine-grained structures and sharp local variations. Mixture-of-Experts Architecture. Removing the MoE module increases the average L2RE by 0.0008, demonstrating that dynamic expert routing effectively improves the modeling of heterogeneous PDE patterns. Load-Balancing Loss. Removing the auxiliary load-balancing loss increases the average L2RE by 0.0004. Although the impact is relatively modest, it promotes balanced expert utilization and stabilizes training. Orthogonality Loss. Removing the orthogonality loss increases the average L2RE by 0.0004, indicating that it encourages diverse expert representations, reduces redundant feature learning, and promotes specialization among experts. Conclusion We presented SPEAR, a spectral-disentangled MoE neural operator with knowledge-guided expert aggregation for large-scale PDE pre-training. SPEAR separates shared low-frequency dynamics from PDE-specific high-frequency patterns and reduces expert redundancy through knowledge-guided aggregation while preserving performance. Extensive experiments demonstrate that SPEAR achieves strong pre-training, transferability, and generalization to unseen PDE systems, providing an efficient and scalable solution for PDE foundation models. References Cao (2021) S. Cao Choose a transformer: fourier or galerkin. Advances in Neural Information Processing Systems 34, p. 24924–24940. Cited by: Datasets and Baselines. Caruana (1997) R. Caruana Multitask learning. Machine learning 28 (1), p. 41–75. Cited by: Low-Rank Adapter-based Mixture of Experts. Chen et al. (2024a) I. Chen, H. Liu, W. Sun, C. Chao, Y. Hsu, C. Lee, et al. Retraining-free merging of sparse moe via hierarchical clustering. arXiv preprint arXiv:2410.08589. Cited by: Introduction, Expert Merging. Chen et al. (2024b) T. Chen, H. Zhou, Y. Li, H. Wang, C. Gao, R. Shi, S. Zhang, and J. Li Omniarch: building foundation model for scientific computing. arXiv preprint arXiv:2402.16014. Cited by: Neural Operators for PDE Solving. Chen et al. (2025) T. Chen, H. Zhou, Y. Li, H. Wang, Z. Zhang, T. Zhu, S. Zhang, and J. Li FreqMoE: dynamic frequency enhancement for neural pde solvers. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, Cited by: Introduction. Dosovitskiy et al. (2020) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: Input Encoding and Time Aggregation. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: Mixture-of-Experts Models. Guibas et al. (2021) J. Guibas, M. Mardani, Z. Li, A. Tao, A. Anandkumar, and B. Catanzaro Adaptive fourier neural operators: efficient token mixers for transformers. arXiv preprint arXiv:2111.13587. Cited by: Introduction, Low-Frequency Global Branch. Gupta and Brandstetter (2022) J. K. Gupta and J. Brandstetter Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616. Cited by: Datasets and Baselines. Hao et al. (2024) Z. Hao, C. Su, S. Liu, J. Berner, C. Ying, H. Su, A. Anandkumar, J. Song, and J. Zhu Dpot: auto-regressive denoising operator transformer for large-scale pde pre-training. arXiv preprint arXiv:2403.03542. Cited by: Neural Operators for PDE Solving, Preliminaries, Input Encoding and Time Aggregation, Datasets and Baselines. Hao et al. (2023) Z. Hao, Z. Wang, H. Su, C. Ying, Y. Dong, S. Liu, Z. Cheng, J. Song, and J. Zhu Gnot: a general neural operator transformer for operator learning. In International conference on machine learning, p. 12556–12569. Cited by: Datasets and Baselines. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778. Cited by: Low-Frequency Global Branch. Hendrycks and Gimpel (2016) D. Hendrycks and K. Gimpel Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. Cited by: Low-Rank Adapter-based Mixture of Experts. Herde et al. (2024) M. Herde, B. Raonić, T. Rohner, R. Käppeli, R. Molinaro, E. De Bezenac, and S. Mishra Poseidon: efficient foundation models for pdes. Advances in Neural Information Processing Systems 37, p. 72525–72624. Cited by: Neural Operators for PDE Solving, Datasets and Baselines. Howard et al. (2017) A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861. Cited by: Low-Rank Adapter-based Mixture of Experts. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: low-rank adaptation of large language models.. Iclr 1 (2), p. 3. Cited by: Low-Rank Adapter-based Mixture of Experts. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: Mixture-of-Experts Models. Li et al. (2026) L. Li, Z. Qiyuan, J. Wang, W. Li, H. Gu, S. Han, and Y. Guo Sub-moe: efficient mixture-of-expert llms compression via subspace expert merging. In AAAI 2026, Cited by: Introduction. Li et al. (2024) P. Li, Z. Zhang, P. Yadav, Y. Sung, Y. Cheng, M. Bansal, and T. Chen Merge, then compress: demystify efficient smoe with hints from its routing policy. In International Conference on Learning Representations, Vol. 2024, p. 14234–14256. Cited by: Introduction, Expert Merging. Li et al. (2022) Z. Li, K. Meidani, and A. B. Farimani Transformer for partial differential equations’ operator learning. arXiv preprint arXiv:2205.13671. Cited by: Datasets and Baselines. Li et al. (2020) Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895. Cited by: Neural Operators for PDE Solving, Main Task Loss, Datasets and Baselines, Datasets and Baselines. Liu et al. (2024) X. Liu, B. Xu, S. Cao, and L. Zhang Mitigating spectral bias for the multiscale operator learning. Journal of Computational Physics 506, p. 112944. External Links: ISSN 0021-9991 Cited by: Introduction. Lu et al. (2021) L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence 3 (3), p. 218–229. Cited by: Neural Operators for PDE Solving. Luo et al. (2023) Y. Luo, Y. Chen, and Z. Zhang CFDBench: a large-scale benchmark for machine learning methods in fluid dynamics. arXiv preprint arXiv:2310.05963. Cited by: Datasets and Baselines. McCabe et al. (2023) M. McCabe, B. R. Blancard, L. H. Parker, R. Ohana, M. Cranmer, A. Bietti, M. Eickenberg, S. Golkar, G. Krawezik, F. Lanusse, et al. Multiple physics pretraining for physical surrogate models. arXiv preprint arXiv:2310.02994. Cited by: Neural Operators for PDE Solving, Datasets and Baselines. Murtagh and Contreras (2012) F. Murtagh and P. Contreras Algorithms for hierarchical clustering: an overview. Wiley interdisciplinary reviews: data mining and knowledge discovery 2 (1), p. 86–97. Cited by: Expert Aggregation Strategy. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: Introduction. Ronneberger et al. (2015) O. Ronneberger, P. Fischer, and T. Brox U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, p. 234–241. Cited by: Datasets and Baselines. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. Cited by: Introduction, Mixture-of-Experts Models, Low-Rank Adapter-based Mixture of Experts, Load Balancing Loss. Sun et al. (2026) D. Sun, X. Zhou, X. Wang, H. Si, W. Lyu, J. Tang, and B. Luo NESTOR: a nested moe-based neural operator for large-scale pde pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6147–6156. Cited by: Introduction, Mixture-of-Experts Models, Datasets and Baselines. Takamoto et al. (2022) M. Takamoto, T. Praditia, R. Leiteritz, D. MacKinlay, F. Alesiani, D. Pflüger, and M. Niepert Pdebench: an extensive benchmark for scientific machine learning. Advances in neural information processing systems 35, p. 1596–1611. Cited by: Datasets and Baselines. Tang et al. (2026) A. Tang, E. Yang, L. Shen, Y. Luo, H. Hu, L. Zhang, B. Du, and D. Tao Merging on the fly without retraining: a sequential approach to scalable continual model merging. Advances in Neural Information Processing Systems 38, p. 82081–82118. Cited by: Main Results. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. Advances in neural information processing systems 30. Cited by: Introduction. Wang et al. (2026) H. Wang, H. Xin, J. Wang, X. Yang, F. Zha, Y. Jiang, et al. Mixture-of-experts operator transformer for large-scale pde pre-training. Advances in Neural Information Processing Systems 38, p. 31498–31527. Cited by: Introduction, Mixture-of-Experts Models, Datasets and Baselines. Wu and He (2018) Y. Wu and K. He Group normalization. In Proceedings of the European conference on computer vision (ECCV), p. 3–19. Cited by: Low-Rank Adapter-based Mixture of Experts. Zhang et al. (2026) D. Zhang, X. Ma, Z. Ni, Z. Wu, H. Shu, X. Jiang, and X. Chen Expert merging: model merging with unsupervised expert alignment and importance-guided layer chunking. In The Fourteenth International Conference on Learning Representations, Cited by: Introduction. Zhou et al. (2025) H. Zhou, Y. Ma, H. Wu, H. Wang, and M. Long Unisolver: pde-conditional transformers towards universal neural pde solvers. In Proceedings of the International Conference on Machine Learning, Cited by: Neural Operators for PDE Solving, Datasets and Baselines.