Paper deep dive
SuperAI: Merging Capabilities Across Heterogeneous Foundation Models
Jingtong Dou, Chuancheng Shi, Wenhua Wu, Wei Bao, Fei Shen, Zhiyong Wang, Tat-Seng Chua
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Enabling the reuse and composition of knowledge and capabilities distributed across foundation models is an important step toward building scalable artificial intelligence (AI) systems. However, architectural differences and incompatible representations make it difficult to transfer and integrate these capabilities through a unified cross-model interface. To address this challenge, we propose SuperAI, a cross-architecture multi-model fusion framework that integrates complementary capabilities from multiple source models into a host model through orthogonal alignment of low-dimensional task-relevant representations. Specifically, we first extract task-relevant representations from the source models and the host model using shared anchor samples, and apply principal component analysis (PCA) to construct low-dimensional functional subspaces with the same dimensionality. Next, we use orthogonal Procrustes mappings to align the source and host coordinate systems, establishing cross-model correspondences while preserving the internal geometry of each source subspace. Finally, we train a lightweight residual multilayer perceptron (MLP) with joint supervision on task outputs and source-model increments, learning input-dependent residual increments that are injected into the host model for multi-model capability fusion. Experiments on three host models across 15 capability domains show that SuperAI achieves higher average domain scores than the evaluated cross-architecture fusion methods and better preserves general capabilities as the number of source models increases. Ablation studies further support the effectiveness of orthogonal alignment, nonlinear residual mapping, and joint supervision.
Tags
Links
- Source: https://arxiv.org/abs/2610.15322v1
- Canonical: https://arxiv.org/abs/2610.15322v1
Trouble viewing inline? Open PDF directly β
Full Text
86,345 characters extracted from source content.
Expand or collapse full text
SUPERAI: MERGING CAPABILITIES ACROSS HETEROGENEOUS FOUNDATION MODELS Jingtong Dou β University of Sydney Sydney, Australia Chuancheng Shi β University of Sydney Sydney, Australia Wenhua Wu University of Sydney Sydney, Australia Wei Bao Edith Cowan University Perth, Australia Fei Shen β National University of Singapore Singapore Zhiyong Wang University of Sydney Sydney, Australia Tat-Seng Chua National University of Singapore Singapore ABSTRACT Enabling the reuse and composition of knowledge and capabilities distributed across foundation models is an important step toward building scalable artificial intelligence (AI) systems. How- ever, architectural differences and incompatible representations make it difficult to transfer and in- tegrate these capabilities through a unified cross-model interface. To address this challenge, we propose SuperAI, a cross-architecture multi-model fusion framework that integrates complemen- tary capabilities from multiple source models into a host model through orthogonal alignment of low-dimensional task-relevant representations. Specifically, we first extract task-relevant represen- tations from the source models and the host model using shared anchor samples, and apply principal component analysis (PCA) to construct low-dimensional functional subspaces with the same di- mensionality. Next, we use orthogonal Procrustes mappings to align the source and host coordinate systems, establishing cross-model correspondences while preserving the internal geometry of each source subspace. Finally, we train a lightweight residual multilayer perceptron (MLP) with joint supervision on task outputs and source-model increments, learning input-dependent residual incre- ments that are injected into the host model for multi-model capability fusion. Experiments on three host models across 15 capability domains show that SuperAI achieves higher average domain scores than the evaluated cross-architecture fusion methods and better preserves general capabilities as the number of source models increases. Ablation studies further support the effectiveness of orthogonal alignment, nonlinear residual mapping, and joint supervision. 1 Introduction Foundation models [Qwen et al., 2025, OLMo et al., 2024, Team et al., 2025, Shi et al., 2026a] develop complementary capabilities shaped by diverse training data, objectives, and architectures. These capabilities remain distributed across models [Ramtoula et al., 2026, Dou et al., 2026a, Wei et al., 2026, Shi et al., 2026b, Dou et al., 2026b, Li et al., 2026, Shi et al., 2026c], while acquiring new ones through further training is costly. Reusing and composing existing capabilities [Akiba et al., 2025] therefore offers an important path toward scalable artificial intelligence (AI) systems. Existing approaches [Wortsman et al., 2022, Ilharco et al., 2022, Yoshida et al., 2026] reuse capabilities through pa- rameter merging [Akiba et al., 2025], knowledge distillation [Ko et al., 2024], and multi-model collaboration [Wang et al., 2025]. Conventional parameter merging combines weights Yadav et al. [2023] or task vectors Ortiz-Jimenez et al. [2023] but generally requires compatible architectures and parameter structures. Knowledge distillation trans- fers information from source-model outputs or intermediate representations through additional host-model training, while routing and collaboration invoke different models according to the input and typically retain multiple models at deployment. β Equal contribution. Email for Jingtong Dou: jdou0056@uni.sydney.edu.au. β Corresponding author. Email: shenfei29@nus.edu.sg. arXiv:2610.15322v1 [cs.AI] 11 Oct 2026 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models (a) CurrentModelMerging Heterogenous Models Different Architecture Host Model Model AModel N Model B Cannot Merge Homogeneous Models Host Model Merge Model AModel NModel B Same Architecture (b) Our Method: SuperAI Multiple Homogeneous Experts Orthogonal Subspace Rotation Host Model Cross-Architecture Multi-Source Merging Model A Model B Model N Merged Model ... ... ... Different Architecture Figure 1: Conventional Model Merging Versus SuperAI. Con- ventional parameter merging requires compatible parameter spaces, whereas SuperAI integrates heterogeneous source capabilities into a merged model through orthogonal subspace alignment and lightweight residual fusion. To accommodate architectural differences, cross- model fusion methods also use neuron match- ing [Xu et al., 2024], optimal transport [Cui et al., 2026, Imfeld et al., 2024], and representation alignment [Dery et al., 2026] to establish corre- spondences. However, as shown in Figure 1, integrating com- plementary information from multiple heteroge- neous source models into a common host repre- sentation space remains challenging, particularly when improving target-task performance must be balanced with retaining the host modelβs existing capabilities. We hypothesize that heterogeneous models can learn structurally corresponding task representa- tions. Using shared anchor samples, we observe that these representations can be approximately aligned through orthogonal mappings in low-dimensional functional subspaces while preserving each source subspaceβs geometry. However, alignment alone does not guarantee task improvements, as transferred information must also be compatible with downstream host computation. Building on this insight, we propose SuperAI, a cross-architecture multi-model capability fusion framework that in- tegrates complementary capabilities from multiple source models into a host model through orthogonal subspace alignment. Specifically, we first extract task-relevant representations from the source models and the host model us- ing shared anchor samples, and apply principal component analysis (PCA) to construct low-dimensional functional subspaces with the same dimensionality. We then use orthogonal Procrustes mappings to align their coordinate sys- tems. Finally, we train a lightweight residual multilayer perceptron (MLP) with joint supervision on task outputs and source-model increments to predict input-dependent residual increments, which are injected into the host model. This joint objective uses source-model information to guide residual learning while adapting the increments to the host modelβs task objectives, connecting representation alignment to task-level capability fusion. Our main contributions are as follows: β’ We propose SuperAI, a cross-architecture multi-model capability fusion framework combining orthogonal subspace alignment with residual injection, providing a unified interface for reusing and composing comple- mentary capabilities from heterogeneous source models. β’ We identify approximately orthogonal cross-model correspondences between task-relevant representations. Ablations support the effectiveness of orthogonal alignment, nonlinear residual mapping, and joint supervi- sion on task outputs and source-model increments. β’ Across three host models and 15 capability domains, the proposed SuperAI outperforms the state-of-the-art (SOTA) fusion methods on average and better preserves general capabilities as the number of source models increases. 2 Related Work Model Merging and Capability Fusion. Parameter-space merging integrates models [Wortsman et al., 2022, Ilharco et al., 2022] with compatible architectures through weight averaging or task-vector addition, as in model soups and task arithmetic. Subsequent methods reduce task interference through sign resolution, filtering, conditioning, or loss bal- ancing [Yadav et al., 2023, Qiu et al., 2026, Yoshida et al., 2026, Im et al., 2026], and improve efficiency through low- rank, orthogonal, or shared parameter structures [Crisostomi et al., 2026, Lee et al., 2026, Yang et al., 2026, Dhasade et al., 2026]. Beyond direct parameter merging, knowledge fusion transfers information through output distributions or token alignment [Wan et al., 2024, 2025], while neuron and component matching reduces representational mismatch before fusion [Ainsworth et al., 2022, Singh and Jaggi, 2020, Imfeld et al., 2024]. For cross-architecture fusion, Trans- port and Merge aligns activations and layers to guide weight fusion [Cui et al., 2026], whereas THESEUS [Rinaldi et al., 2026] combines functional matching with orthogonal Procrustes alignment to transfer task vectors across model widths. Cross-Model Representation Alignment. Representation comparison methods assess statistical similarity using SVCCA and CKA [Raghu et al., 2017, Kornblith et al., 2019], or functional compatibility through model stitch- ing [Lenc and Vedaldi, 2015, Bansal et al., 2021, Athanasiadis et al., 2025]. Relative representations and latent-space 2 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Which planet is known as the Red Planet? Dataset Β§3.1Orthogonal Subspace Alignment Layer1 Layer2 Layerk Layern ... ... Source Model Last Token ... Hidden States Layer1 Layer2 Layerk Layern ... ... Host Model Last Token ... Hidden States Layer Matching All Candidate Layer Pairs Host Layers Source Model Layers Score(ν ν , ν ν ) Best source layer for each host layer. Repeat step 1 for every (ν ν , ν β ) MAX ΰ΄€ νΊβ ν =ΰ· ν=ν ν² ν ν νΊ ν β ν β β ν ,β ν β ν β =νννννν β ν ΰ΄€ νΊβ ν Source Alignment Scores ν ν ν ν ν ν ν ν ν ν ν ν νΊ ν νΊ ν νΊ ν ... ... SourceModel HostModel Science Math ... Social Layer1 Layerk Layern νΊ ν νΊ ν ν ν β Layer Selecting Β§3.2Feature Mapping Host Subspace Β§3.3Multi-Source Joint Training Layer1 Layer2 Layerk Layern ... ... ... Source Model Hidden States Layer1 Layer2 Layerk Layern ... ... Host Model ... Hidden States Host Subspace ν ν ν ν ... Predicted ΰ·‘ ν ν,ν νβ² ν,ν =ν ν,ν +νΆ ΰ·‘ ν ν,ν . Layer 1 Layer 2 Layer k ... ... MLP ... Layer n - 1 Layer n νβ² ν,ν Hidden States Answer Inference Tom has 12 apples and gives away 5. How many are left? Trainable Frozen MLP ... ... ν ν,ν β ΰ·‘ ν ν,ν Layer 1 ... ... Layer 2 Layer k Layer k+1 Layer n-1 Layer n ν ν ν ν = ΰ·© ν νβν ν ν βν ν ν ν ν ν ν ν R s β ΖΆ ν ν ββ =ν ν ν ν β Orthogonal Rotation Eq.5 Eq.6 Eq.7 Eq.3 Figure 2: Overview of Proposed SuperAI Framework. Shared prompts are used to align source and host representations through PCA and orthogonal rotation. All layer pairs are evaluated to select one shared host layer and the corresponding source layers. The aligned source features define capability increments β e , which supervise a shared residual MLP together with task and domain objectives. At inference, only the frozen host model and the trained MLP are required. mappings enable explicit representation reuse [Moschella et al., 2022, Maiorca et al., 2023, Moayeri et al., 2023, Fumero et al., 2024], while the Platonic Representation Hypothesis suggests shared structure across independently trained models [Huh et al., 2024]. Recent work transfers interventions, probes, and adapters between language mod- els [Oozeer et al., 2025, Chen et al., 2026], and extends alignment to multiple models sharing a reference space [Achara et al., 2026]. However, representational correspondence alone does not guarantee effective capability fusion, as aligned information from multiple source models must also be adapted to the host modelβs downstream computation and task objectives to improve its performance. 3 Methodology 3.1 Orthogonal Subspace Alignment Let f h be the host model andf s K s=1 be K source models. Their hidden dimensions are d h and d s , respectively. For source model s, we use shared promptsA s =a i n s i=1 . As illustrated in Figure 2, SuperAI first extracts paired source and host representations from shared prompts and aligns them in low-dimensional subspaces. Let h (β m ) m (a i ) be the hidden state of the last non-padding token at layer β m of model mβs,h. We stack the representations of the same prompts into H (β s ) s βR n s Γd s and H (β h ) h βR n s Γd h . The representations are paired by prompt, so the models may use different tokenizers. For a candidate layer pair, let ΞΌ m be the mean hidden state of model m. We center and normalize each representation. We then project the representations into separate r-dimensional PCA subspaces: u m (a i ) = h (β m ) m (a i )β ΞΌ m max β₯h (β m ) m (a i )β ΞΌ m β₯ 2 ,Ξ΅ , Z m = (U m β 1c β€ m )P m , mβs,h.(1) Here, Ξ΅ > 0 prevents division by zero. The rows of U m are u m (a i ). The PCA mean is c m , and P m βR d m Γr contains the principal components, with P β€ m P m = I r . The vector 1 contains ones. We omit the candidate layer indices below for clarity. Layer Matching. The coordinates Z s and Z h have the same dimension, but their axes may differ. We align them with orthogonal Procrustes: R β s = arg min R β€ R=I r β₯Z s Rβ Z h β₯ 2 F , Z β€ s Z h = LΞ£V β€ , R β s = LV β€ .(2) The aligned source coordinates are b Z sβh = Z s R β s . The orthogonal map preserves pairwise distances within the source PCA subspace. Layer Selecting. Let S s (β s ,β h ) be the alignment score of source layer β s and host layer β h on held-out prompts. For each candidate host layer, we first select the best source layer. We then combine the scores across source models and 3 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models select one shared host layer: β β s (β h ) = arg max β s S s (β s ,β h ), β β h = arg max β h K X s=1 w s S s (β β s (β h ),β h ), K X s=1 w s = 1.(3) Here, w s is the weight of source model s. With equal weights, w s = 1/K. Source models may use different source layers and orthogonal maps, but all are mapped to the same host layer β β h . In the next section, we use this layer to construct target feature increments. We fit the normalization statistics, PCA subspaces, and orthogonal mapping separately for each source model using its domain-specific training prompts. The mapping establishes cross-model correspondences in low-dimensional func- tional subspaces while preserving the geometry of the source model representations. We next examine these properties qualitatively and quantitatively. (a) Before Alignment (b) After Alignment Figure 3: Cross-Model Representations Before and After Orthog- onal Alignment. Source and host representations differ substantially before alignment and become closely aligned afterward. Geometric Correspondence Across Models. To visualize the effect of orthogonal alignment, we compare paired source model and host represen- tations before and after applying the learned map- ping. As shown in Figure 3(a), the representa- tions initially differ substantially in position, ori- entation, and density. After alignment, their point clouds and distribution contours closely overlap, as shown in Figure 3(b). This result suggests that het- erogeneous models encode similar task-relevant information using different coordinate bases while retaining alignable underlying structures, support- ing the effectiveness of orthogonal subspace align- ment in establishing cross-model correspondences. MatchCKA Retrieval@1 ScienceQA-OFT Commonsense-OFT Social-OFT Math-OFT Code-OFT Instruction-3B Math-3B Code-3B Safety-3B Finance-Llama ChemDFM HuatuoGPT-o1 ClimateGPT Foundation-Sec FeynTune Mean 0.941.000.90 0.951.001.00 0.780.911.00 0.971.001.00 0.910.991.00 0.971.001.00 0.991.001.00 0.991.001.00 0.991.001.00 0.880.971.00 0.820.960.77 0.820.980.81 0.76 0.911.00 0.810.941.00 0.890.991.00 0.900.980.97 Llama-3.2-3B MatchCKA Retrieval@1 0.790.930.56 0.921.000.77 0.690.840.94 0.951.000.98 0.870.980.98 0.860.97 1.00 0.951.00 1.00 0.890.980.96 0.901.00 1.00 0.770.900.77 0.660.730.46 0.880.991.00 0.77 0.91 1.00 0.740.870.93 0.720.840.75 0.820.930.87 Qwen2.5-3B 0.50.60.70.80.91.0 Alignment score Figure 4: Cross-model Alignment Evaluation. Match, CKA, and Re- trieval@1 scores across 15 source models with Llama and Qwen as host models. Cross-Model Alignment Evaluation. To evalu- ate whether this geometric correspondence gener- alizes across heterogeneous sourceβhost pairs, we examine the selected layer pairs between 15 source models and two host models, Llama-3.2-3B and Qwen2.5-3B. Match measures overall alignment quality, CKA measures global geometric similar- ity, and Retrieval@1 measures instance-level cor- respondence between representations of the same input. As shown in Figure 4, the average Match, CKA, and Retrieval@1 scores are 0.90, 0.98, and 0.97 for Llama-3.2-3B, and 0.82, 0.93, and 0.87 for Qwen2.5-3B. Despite variation across sourceβ host pairs, the scores indicate that orthogonal alignment preserves global geometry and input- level correspondences across heterogeneous mod- els, supporting its use for subsequent capability in- crement construction. 3.2 Feature Mapping The previous section selects a source layer β β s and an orthogonal map R β s for each source model s. It also selects one shared host layer β β h . For a prompt a i , we rotate the source PCA coordinates into the host subspace and reconstruct them in the host hidden space: bz sβh (a i ) = z s (a i )R β s , b sβh (a i ) = bz sβh (a i )P β€ h + c h .(4) Here, P h and c h are the host PCA component matrix and mean fitted on the prompts paired with source model s. As in the previous section, we omit their source index for clarity. The alignment uses normalized representations. We therefore use b sβh (a i ) to set the direction and the host hidden state to set the magnitude: e h sβh (a i ) = ΞΌ h +β₯h h (a i )β ΞΌ h β₯ 2 b sβh (a i ) max(β₯b sβh (a i )β₯ 2 ,Ξ΅) .(5) 4 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Here, h h (a i ) is the original last non-padding token hidden state at host layer β β h , and ΞΌ h is the mean hidden state at that layer for the prompts paired with source model s. We define the target feature increment as the difference between the mapped representation and the original host representation: β s (a i ) = e h sβh (a i )β h h (a i ), β s (a i )βR d h .(6) The increment β s (a i ) lies in the host hidden space. It provides a representation-level target for the shared residual MLP in the next section. At inference time, the MLP predicts increments directly from host hidden states. Source models and explicit feature maps are not needed. 3.3 Multi-Source Joint Training Section 3.2 constructs a target feature increment β s (a i ) in the host hidden space for each training prompt from source model s. We mix training samples from all source models and train one shared residual MLP at the host layer selected in Section 3.1. The host model remains frozen. For hidden state h i,t at token position t, the MLP predicts an increment and adds it to the hidden state: q i,t = SiLU(W down LN(h i,t )), b β i,t = W up q i,t , h β² i,t = h i,t + Ξ± b β i,t .(7) Here, W down βR bΓd h and W up βR d h Γb , where b is the bottleneck width. The residual scaling factor Ξ± controls the strength of the predicted capability increment injected into the host hidden state. We initialize W up to zero, so the residual update is zero at the start of training. The MLP updates hidden states at all token positions. Increment supervision is applied only at p i , the last non-padding position of prompt a i . This is the position used to extract features in the preceding sections. Let s i denote the source model associated with prompt a i , and letA tr s be the training prompts for source model s. To account for differences in target magnitude across source models, we define the normalization factor and increment loss: Ο s = v u u t 1 |A tr s |d h X a i βA tr s β₯β s (a i )β₯ 2 2 , L inc = 1 Nd h N X i=1 b β i,p i β β s i (a i ) max(Ο s i ,Ξ΅) 2 2 ,(8) where N is the total number of training prompts and Ξ΅ > 0 prevents division by zero. We also compute cross-entropy against the ground-truth answers. LetT i be the answer token positions for prompt a i . The task loss and the joint objective are L task =β 1 N N X i=1 1 |T i | X tβT i logp ΞΈ (y i,t | a i ,y i,<t ), L = Ξ» task L task + Ξ» inc L inc .(9) Here,L task is the cross-entropy loss on the ground-truth answer tokens, andL inc is the source-increment supervision loss. The coefficients Ξ» task and Ξ» inc control the contributions of the two losses. Here, p ΞΈ is the output distribution of the host model with the residual MLP. Only the MLP is updated during training. The source identity s i is used to select the target increment and its normalization factor. At inference time, only the host model and the shared MLP are needed. Source models, source identities, and explicit feature maps are not required. 4 Experiments 4.1 Experimental Setup Source Models and Datasets. We select three model families as host models, covering multiple parameter scales within each family: the Llama includes Llama-3.2-1B, Llama-3.2-3B, and Llama-3.1-8B Grattafiori et al. [2024]; the Pythia includes Pythia-1B, Pythia-2.8B, Pythia-6.9B, and Pythia-12B Biderman et al. [2023]; and the Qwen2.5 includes Qwen2.5-1.5B, Qwen2.5-3B, Qwen2.5-7B, and Qwen2.5-14B Qwen et al. [2025]. We construct a pool of 15 source models spanning science, commonsense and social reasoning, math, code generation, multilingual reasoning, safety, law, finance, astronomy, medicine, climate science, cyber security, chemistry, and physics. The pool includes four OFT sources Yang et al. [2026], Qiu et al. [2023], Qwen2.5-Coder-7B-Instruct Hui et al. [2024], two MergeBench source models He et al. [2026], and eight domain-specialized source models. These source models vary in model family, parameter scale, and training procedure, forming a heterogeneous testbed for cross-architecture capability transfer and multi-source fusion. The complete source benchmark mapping is provided in the appendix. 5 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Table 1: Quantitative Comparison of Multi-source Capability Fusion Across Different Host Models. We report the original host performance and multi-source fusion results across 15 capability domains. The best and second-best results among the fusion methods are shown in bold and underlined , respectively. Green and red arrows in the Avg. column denote gains and decreases relative to the corresponding original host. MethodMathSafetyLawPhys.Sci.C.SenseSocialCodeMulti.Fin./Bus.Astro.Med.ClimateCyberChem.Avg. Domain67.772.046.580.976.581.675.281.569.415.074.358.471.436.249.063.7 Host model: Llama-3.2-3B Original16.756.838.068.367.966.258.739.756.13.760.550.942.331.444.046.7 THESEUS24.157.340.362.963.560.053.842.144.012.754.240.638.434.732.144.0(β 2.7) T&M17.659.5 41.261.464.861.754.241.744.310.949.740.335.831.243.443.8 (β 2.9) SuperAI40.770.940.889.169.868.263.328.355.014.757.948.272.548.944.054.2 (β 7.5) Host model: Qwen2.5-3B Original26.361.339.485.173.077.269.055.671.97.071.253.668.139.245.056.2 THESEUS22.755.640.571.856.254.059.141.557.813.446.737.231.932.629.043.3 (β 12.9) T&M31.261.643.865.963.762.865.953.864.912.755.043.440.834.135.649.0(β 7.2) SuperAI45.776.653.594.982.579.569.544.267.69.372.453.873.853.843.061.3 (β 5.1) Host model: Pythia-2.8B Original3.742.523.924.224.020.735.711.123.60.728.326.227.923.327.022.9 THESEUS1.813.44.354.920.131.925.53.86.26.918.27.938.932.720.519.1 (β 3.8) T&M3.120.16.047.527.140.632.26.09.410.321.811.646.029.223.122.3(β 0.6) SuperAI8.069.931.089.227.618.737.51.329.31.025.027.772.533.021.032.8 (β 9.9) Evaluation Metrics. We use one predefined primary metric for each domain. Most domains are evaluated with accu- racy or task-specific variants of accuracy. The main exceptions are code generation and cybersecurity: MBPP Austin et al. [2021] is evaluated with Pass@1 Chen et al. [2021], requiring the generated program to pass the test suite, while SEVENLLM Ji et al. [2024b] uses token-level F1. All metrics are higher-is-better. For multi-source fusion, we additionally report the unweighted macro average across 15 domains and MMLU Hendrycks et al. [2021] to assess general-capability retention as more source models are merged. More details are in the appendix. Implementation Details. We use Llama-3.2-3B Grattafiori et al. [2024], Qwen2.5-3B Qwen et al. [2025], and Pythia- 2.8B Biderman et al. [2023] as host models. Both the PCA subspace dimension and the bottleneck dimension of the residual MLP are set to 128. We train the models for two epochs using AdamW with a batch size of 4, a learning rate of 2Γ 10 β4 , and a weight decay of 0.01. The feature increment loss weight is set to 0.5. The residual scaling factor Ξ± is set to 1.0, 0.75, and 0.25 for the three host models, respectively. 4.2 Heterogeneous Multi-Source Merging Quantitative Comparison Across Host Models. To evaluate SuperAI across host architectures, we jointly fuse all 15 domain source models into each of three hosts, Llama-3.2-3B, Qwen2.5-3B, and Pythia-2.8B, and compare the resulting models with THESEUS Rinaldi et al. [2026] and Transport and Merge Cui et al. [2026]. The results in Table 1 show that SuperAI achieves the highest average score for all three host models. Specifically, SuperAI improves the average scores of Llama-3.2-3B, Qwen2.5-3B, and Pythia-2.8B from 46.7, 56.2, and 22.9 to 54.2, 61.3, and 32.8, corresponding to gains of 7.5, 5.1, and 9.9 points, respectively. Compared with the strongest fusion baseline for each host, SuperAI improves the average score by 10.2, 12.3, and 10.5 points, respectively. The largest domain-level gains vary across hosts. Llama improves most in climate and math, by 30.2 and 24.0 points, respectively. Qwen obtains substantial gains in math, safety, and cyber, improving by 19.4, 15.3, and 14.6 points. Although Pythia achieves the largest average gain, the post-fusion ranking remains Qwen, Llama, and Pythia, consistent with their original performance. Therefore, orthogonal subspace alignment provides an effective cross-model interface for heterogeneous capability transfer, while capability absorption and final performance remain dependent on the host model. 5565758595100 MMLU Ret. 1 3 5 8 11 15 Experts THESEUST&MSuperAI Figure 5: General capability retention. MMLU re- tention of THESEUS, T&M, and SuperAI with 1β15 source models. General Capability Retention. To evaluate how well SuperAI pre- serves the host modelβs existing general capabilities during multi- source capability fusion, we conduct an MMLU retention analysis as the number of integrated source models increases from 1 to 15. Figure 5 show that MMLU retention decreases for all three methods as more source models are integrated, but SuperAI exhibits a sub- stantially slower decline and an increasing advantage over the base- lines. Specifically, the retention rates of THESEUS and Transport and Merge decrease from 96.8% and 97.9% to 61.7% and 70.9%, respectively, whereas SuperAI retains 88.6% of the original MMLU performance after integrating 15 source models. The widening per- formance gap indicates that interference accumulates more slowly under SuperAI as additional source models are introduced. There- fore, these results suggest that orthogonal subspace alignment and residual injection mitigate interference from accumulating source capabilities, enabling the host model to absorb domain-specific capabilities while better preserving its existing general capabilities. 6 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models 4.3 Ablation Study Table 2: Effect of Training Objectives. We report average scores across 15 domains and cosine-similarity gains for three host models. For each host and metric, the best and second-best results are bolded and underlined, respectively. ObjectiveLlama-3.2-3B Qwen2.5-3B Pythia-2.8B 15-domain average score Original46.756.222.9 Ξ» inc = 050.6(β 3.9)57.5(β 1.3) 27.1(β 4.2) Ξ» task = 044.8 (β 1.9)56.3 (β 0.1) 24.0 (β 1.1) Joint54.2 (β 7.5)61.3 (β 5.1)32.8 (β 9.9) Cosine-similarity gain Ξ» inc = 00.0003β0.0004β0.0004 Ξ» task = 00.02120.10030.2560 Joint0.01280.08400.0913 Effect of Training Objectives. To evaluate the roles of task-output and source-model increment supervision, we compare Ξ» inc = 0, Ξ» task = 0, and the joint objective across three host models, using cosine-similarity gain to measure increment matching. The results in Table 2 show that joint supervision achieves the best downstream perfor- mance, whereas either signal alone is insufficient. Specif- ically, Ξ» inc = 0 improves Llama, Qwen, and Pythia by 3.9, 1.3, and 4.2 points but yields negligible increment- matching gains. Conversely, Ξ» task = 0 achieves the high- est matching gains but provides inconsistent downstream improvements. In contrast, the joint objective improves the three hosts by 7.5, 5.1, and 9.9 points. Therefore, joint su- pervision effectively balances source-model guidance with adaptation to host task objectives, thereby improving the effectiveness of cross-architecture capability transfer. Table 3: Effect of Orthogonal Subspace Alignment. We report 15-domain average scores for three host models, with changes relative to each original host in parentheses. Mapping Method Llama-3.2-3B Qwen2.5-3B Pythia-2.8B Original46.756.222.9 No Alignment47.6 (β 0.9)56.1 (β 0.1) 21.0 (β 1.9) PCA Only47.8 (β 1.1)56.9 (β 0.7) 28.6 (β 5.7) PCA + Linear53.2 (β 6.5)57.5 (β 1.3) 27.2 (β 4.3) PCA + Orthogonal54.2 (β 7.5)61.3 (β 5.1)32.8 (β 9.9) Effect of Orthogonal Subspace Alignment. To evalu- ate the role of orthogonal subspace alignment, we com- pare No Alignment, PCA Only, PCA with Linear Map- ping, and PCA with Orthogonal Mapping under identical training settings. The results in Table 3 show that PCA with Orthogonal Mapping consistently achieves the best performance across all three host models. Specifically, it improves Llama, Qwen, and Pythia by 7.5, 5.1, and 9.9 points, respectively, whereas the other configurations pro- duce smaller or less consistent gains. It outperforms linear mapping by 1.0, 3.8, and 5.6 points on the three hosts, re- spectively. Therefore, orthogonal alignment establishes a reliable cross-model interface by reconciling coordinate differences while preserving the geometry of PCA-projected source representations. Table 4: Effect of Residual Mapper Architecture. We report average scores across 15 domains for three host models. Paren- theses indicate score changes relative to the original host. Mapper Llama-3.2-3B Qwen2.5-3B Pythia-2.8B Original46.756.222.9 Linear50.6 (β 3.9)56.0 (β 0.2) 30.1 (β 7.2) MLP54.2 (β 7.5)61.3 (β 5.1)32.8 (β 9.9) Effect of Residual Mapper Architecture. To evaluate whether nonlinear residual mapping is necessary for adapt- ing aligned source representations to host computation, we compare a linear mapper with the residual MLP used in Su- perAI across three host models. The results in Table 4 show that the linear mapper produces inconsistent performance changes, whereas the residual MLP consistently achieves larger gains across three models. The linear mapper im- proves Llama and Pythia by 3.9 and 7.2 points, respec- tively, but reduces Qwen performance by 0.2 points. In contrast, the residual MLP improves Llama, Qwen, and Pythia by 7.5, 5.1, and 9.9 points, outperforming the linear mapper by 3.6, 5.3, and 2.7 points, respectively. Therefore, the effectiveness of linear mapping for capability transfer varies across host architectures, whereas nonlinear residual mapping learns host-compatible increments and enables consistent capability transfer across architectures. 0.00.51.01.52.0 Residual Scaling Ξ± 58.5 60.0 61.5 Average Score Qwen2.5-3B 0.00.51.01.52.0 Residual Scaling Ξ± 51.0 52.5 54.0 LLaMA-3.2-3B 0.00.51.01.52.0 Residual Scaling Ξ± 28 30 32 Pythia-2.8BGain Figure 6: Sensitivity to Residual Scaling. Average scores across 15 domains for three host models as the residual scal- ing factor Ξ± varies from 0 to 2.0. Sensitivity to Residual Scaling. To assess how residual scaling affects capability transfer, we vary the residual scal- ing factor Ξ± from 0 to 2.0 across three host models. Fig- ure 6 shows that performance varies non-monotonically with Ξ±, with the optimal setting differing across hosts. Specifically, Llama-3.2-3B, Qwen2.5-3B, and Pythia-2.8B achieve their best performance at Ξ± = 1.0, 0.75, and 0.25, respectively. These different optima suggest that the pre- ferred injection strength depends on the host model. In- creasing Ξ± to 1.5 or 2.0 does not yield consistent additional gains in average domain performance. Pythia exhibits a more pronounced decline beyond its optimum than Llama, indicating greater sensitivity to residual scaling. Over- 7 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models all, these results support selecting an appropriate scaling factor for each host, as stronger residual injection does not necessarily improve capability transfer. Table 5: Effect of Layer-pair Alignment Quality. We report 15- domain averages and post-alignment Match scores; parentheses show changes from the baseline. Best and second-best results are bolded and underlined. Layer PairLlama-3.2-3B Qwen2.5-3B Pythia-2.8B 15-domain average score Original46.756.222.9 Weakly Matched51.1 (β 4.4)59.0 (β 2.8) 28.4 (β 5.5) Medium Matched53.5 (β 6.8)60.4(β 4.2) 31.6(β 8.7) Best Matched54.2 (β 7.5)61.3 (β 5.1)32.8 (β 9.9) Post-alignment Match score Weakly Matched0.9070.9120.964 Medium Matched0.9260.9220.967 Best Matched0.9310.9250.976 Effect of Layer-Pair Alignment Quality. To evalu- ate whether post-alignment Match scores can guide the selection of cross-model transfer layers, we compare source and host layer pairs with weak, medium, and optimal match scores while keeping the training data, residual MLP, and optimization settings fixed. The re- sults in Table 5 show that transfer performance improves with layer-pair alignment quality. Specifically, weakly matched layer pairs improve Llama, Qwen, and Pythia by 4.4, 2.8, and 5.5 points, respectively. The gains increase to 6.8, 4.2, and 8.7 points for the medium- matched pairs, and further to 7.5, 5.1, and 9.9 points for the best-matched pairs. Meanwhile, the post-alignment Match scores of all three host models consistently in- crease from weakly matched to best-matched pairs. Higher Match scores indicate more consistent sample re- lationships and functional geometry between source and host representations, making the aligned source-model increments more compatible with downstream host computa- tion. These results show a consistent positive relationship between alignment quality and capability transfer perfor- mance, supporting the use of orthogonal subspace alignment to identify effective representation injection locations for cross-architecture capability transfer. 4.4 Deeper Analysis ,, , , # " )"+'%*$%'&"(! Figure 7: Effect of Host Model Scale on Multi-source Fu- sion Capacity. Bars show mean 15-domain gains at the max- imum stable fusion capacity k β , with error bars indicating standard deviations across three seeds. Lines show k β ; 3/3 denotes positive gains in all three seeds. Missing entries indi- cate unavailable model scales. Maximum Stable Fusion Capacity. To evaluate the im- pact of host model scale on multi-source fusion capability, we conduct scaling experiments across the Qwen, Llama, and Pythia model families, measuring the maximum sta- ble fusion capacity k β and its corresponding average per- formance gain. The results in Figure 7 show that fusion ca- pacity generally exhibits a non-decreasing trend as model scale increases. Specifically, k β increases from 8 to 12 for Qwen, from 7 to 12 for Llama, and from 9 to 12 for Pythia at the 7β8B scale. Pythia reaches a capacity of 14 at the 12β 14B scale. At approximately 3B scale, the average gains for Qwen, Llama, and Pythia are 5.1Β± 0.57, 7.5Β± 0.84, and 9.9Β± 1.31 points, respectively. At the 7β8B scale, all three families can stably accommodate 12 source models, with the corresponding gains decreasing to 4.8Β± 0.61, 5.2Β± 0.69, and 8.7Β± 0.78 points. All reported configurations achieve positive gains across all three random seeds, demonstrating solid stability in the fusion outcomes. Therefore, within the evaluated model families, scaling up the host model increases the number of stably mergeable source models, yet performance gains do not grow monotonically with size, being further shaped by family-specific characteristics and the host modelβs remaining headroom for improvement. Table 6: Domain-wise Capability Absorption Across Qwen2.5 Scales. Each cell reports the post-fusion score and its change from the corresponding base model. Avg. denotes the four-domain average; best and second-best results are bolded and underlined. Host ModelMathSafetyLawPhysicsAvg. Qwen2.5-1.5B 14.3 (β 12.3) 72.5 (β 8.0)49.3 (β 8.5) 94.0 (β 14.1) 57.5 (β 4.6) Qwen2.5-3B45.7 (β 19.4) 76.6 (β 15.3) 53.5 (β 14.1) 94.9 (β 9.8) 67.7 (β 14.6) Qwen2.5-7B57.3 (β 40.7) 79.3(β 3.3)56.3(β 8.5)95.7(β 4.3) 72.2(β 14.2) Qwen2.5-14B 57.7 (β 44.0) 80.2 (β 8.9)59.2 (β 5.6)97.9 (β 1.1) 73.8 (β 12.1) Domain-Wise Capability Absorption.To evaluate how host model scale affects domain-specific capabil- ity absorption, we conduct a scaling analysis within the Qwen2.5 family across four representative domains: Math, Safety, Law, and Physics. The results in Table 6 show that SuperAI achieves positive gains across most model scales and domains. The average gain across the four domains increases from 4.6 points for Qwen2.5- 1.5B to 14.6 points for Qwen2.5-3B, while remaining substantial at 14.2 and 12.1 points for the 7B and 14B models, respectively. Math shows a clear scaling effect, 8 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models with gains of 19.4, 40.7, and 44.0 points for the 3B, 7B, and 14B models. Safety, law, and physics also achieve substantial improvements at different model scales, suggesting that the optimal scale for capability absorption varies across domains. These contrasting trends suggest that model size alone does not determine the magnitude of domain- specific transfer gains. Therefore, increasing host model scale can enhance the absorption of heterogeneous source capabilities, but the benefits exhibit diminishing returns, potentially because stronger baseline capabilities leave less room for complementary capability gains. !! ""# $" ! " !! $" ! " Figure 8: Generalization Across Host Architectures and Scales. Evaluation of SUPERAI in the Climate, Science, Common Sense, and Social domains: (a) generalization to unseen datasets across host architectures; (b) generalization performance across host model scales. Generalization Across Datasets and Host Configu- rations. To evaluate the generalization of SuperAI, we conduct experiments along three dimensions: cross- architecture, cross-dataset, and cross-scale generaliza- tion. The results in Figure 8(a) show that Llama- 3.2-3B, Qwen2.5-3B, and Pythia-2.8B achieve aver- age gains of 5.7, 2.0, and 1.5 points, respectively, on unseen datasets across Climate, Science, Common Sense, and Social. Specifically, Figure 8(b) shows that the Qwen2.5-1.5B, 3B, 7B, and 14B models achieve average gains of 4.4, 2.1, 5.0, and 5.8 points across the four domains, respectively, with the 7B and 14B mod- els exhibiting stronger overall capability absorption. In the Science domain, the gain increases from 1.1 to 9.2 points as the model scale grows, indicating that larger host models can more effectively absorb source capabilities in certain domains. Therefore, SuperAI enables consistent positive transfer across model ar- chitectures, unseen data distributions, and parameter scales, supporting orthogonal subspace alignment as a generaliz- able interface for cross-model capability fusion. However, domain-specific returns remain non-linear and depend on the host modelβs existing capabilities and remaining headroom for improvement. 5 Conclusion In this work, we introduced SuperAI, a framework for integrating multiple heterogeneous source models into a sin- gle host model through orthogonal subspace alignment and lightweight residual fusion. Our analysis reveals that task-relevant representations across different model architectures exhibit approximately orthogonal correspondences in low-dimensional functional subspaces, providing a shared geometric interface for cross-model capability transfer. Experiments across 15 capability domains and three host model families demonstrate that SuperAI consistently out- performs existing cross-architecture merging methods while better preserving the host modelβs general capabilities. Further results across unseen datasets, model architectures, and parameter scales confirm the generalizability of this approach, while the scaling analysis shows that capability absorption is jointly constrained by model capacity and representational characteristics. Overall, our findings suggest that aligning and composing functional representations offers a promising alternative to parameter-level merging for modular and efficient multi-source integration. References Raia Abu Ahmad, Aida Usmanova, and Georg Rehm. The ClimateCheck dataset: Mapping social media claims about climate change to corresponding scholarly articles. In Tirthankar Ghosal, Philipp Mayr, Amanpreet Singh, Aakanksha Naik, Georg Rehm, Dayne Freitag, Dan Li, Sonja Schimmler, and Anita De Waard, editors, Proceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025), pages 42β56, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.sdp-1.5. URL https://aclanthology.org /2025.sdp-1.5/. Akshit Achara, Tatiana Gaintseva, Mateo Mahaut, Pritish Chakraborty, Viktor Stenby Johansson, Melih Barsbey, Emanuele Rodol ` a, and Donato Crisostomi. Multi-way representation alignment. arXiv preprint arXiv:2602.06205, 2026. Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836, 2022. Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195β204, 2025. 9 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Ioannis Athanasiadis, Anmar Karmush, and Michael Felsberg. Grounding functional similarity by invariance-aware model stitching. arXiv preprint arXiv:2505.20142, 2025. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225β236, 2021. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle OβBrien, Eric Hallahan, Moham- mad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https://arxiv.org/abs/2304.01373. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical com- monsense in natural language, 2019. URL https://arxiv.org/abs/1911.11641. Alan Chen, Jack Merullo, Alessandro Stolfo, and Ellie Pavlick. Transferring linear features across language models with model stitching. Advances in Neural Information Processing Systems, 38:48531β48563, 2026. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms, 2024. URL https://arxiv.org/abs/2412.189 25. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Ed- wards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. Finqa: A dataset of numerical reasoning over financial data, 2022. URL https://arxiv.org/abs/2109.00122. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre FT Martins, Fabrizio Esposito, Vera L Μ ucia Raposo, Sofia Morgado, et al. Saullm-7b: A pioneering large language model for law. arXiv preprint arXiv:2403.03883, 2024. Donato Crisostomi, Alessandro Zirilli, Antonio Andrea Gargiulo, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Iacopo Masi, and Emanuele Rodol ` a. Mass: Moerging through adaptive subspace selection. In Interna- tional Conference on Learning Representations, volume 2026, pages 19982β20008, 2026. Chenhang Cui, Binyun Yang, Fei Shen, Yuxin Chen, Jingnan Zheng, Xiang Wang, An Zhang, and Tat-Seng Chua. Transport and merge: Cross-architecture merging for large language models. arXiv preprint arXiv:2602.05495, 2026. Tijmen de Haan, Yuan-Sen Ting, Tirthankar Ghosal, Tuan Dung Nguyen, Alberto Accomazzi, Azton Wells, Nesar Ramachandra, Rui Pan, and Zechang Sun. Achieving gpt-4o level performance in astronomy with a specialized 8b-parameter large language model. Scientific Reports, 15(1), April 2025. ISSN 2045-2322. doi: 10.1038/s41598 -025-97131-y. URL http://dx.doi.org/10.1038/s41598-025-97131-y. Lucio M Dery, Zohar Yahav, Henry Prior, Qixuan Feng, Jiajun Shen, and Arthur Szlam. Latent space communication via kv cache alignment. arXiv preprint arXiv:2601.06123, 2026. Akash Dhasade, Divyansh Jhunjhunwala, Milos Vujasinovic, Gauri Joshi, and Anne-Marie Kermarrec. Navigating the accuracy-size trade-off with flexible model merging. In International Conference on Learning Representations, volume 2026, pages 81010β81041, 2026. Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. Climate- fever: A dataset for verification of real-world climate claims, 2021. URL https://arxiv.org/abs/2012.00614. Jingtong Dou, Chuancheng Shi, Jian Wang, Fei Shen, Zhiyong Wang, and Tat-Seng Chua. Beyond surface artifacts: Capturing shared latent forgery knowledge across modalities. arXiv preprint arXiv:2604.07763, 2026a. Jingtong Dou, Chuancheng Shi, Yemin Wang, Shiming Guo, Anqi Yi, Wenhua Wu, Li Zhang, Fei Shen, and Tat-Seng Chua. Dna: Uncovering universal latent forgery knowledge. arXiv preprint arXiv:2601.22515, 2026b. 10 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Marco Fumero, Marco Pegoraro, Valentino Maiorca, Francesco Locatello, and Emanuele Rodol ` a. Latent functional maps: a spectral framework for representation alignment. Advances in Neural Information Processing Systems, 37: 66178β66203, 2024. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. Yifei He, Siqi Zeng, Yuzheng Hu, Rui Yang, Tong Zhang, and Han Zhao. Mergebench: A benchmark for merging domain-specialized llms. Advances in Neural Information Processing Systems, 38, 2026. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning, 2019. URL https://arxiv.org/abs/1909.00277. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. Kyungjin Im, Miru Kim, Chanin Eom, and Minhae Kwon. Post-hoc merging is not enough: Many-shot model merging with loss-gap balancing. arXiv preprint arXiv:2606.16501, 2026. Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Trans- former fusion with optimal transport. In International Conference on Learning Representations, volume 2024, pages 29313β29336, 2024. Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, Changyu Ren, and Zhoujun Li. Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence, 2024a. URL https://arxiv.org/abs/2405.034 46. Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, et al. Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence. arXiv preprint arXiv:2405.03446, 2024b. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, et al. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31983β32016, 2025. Paul Kassianik, Baturay Saglam, Alexander Chen, Blaine Nelson, Anu Vellore, Massimo Aufiero, Fraser Burch, Dhruv Kedia, Avi Zohary, Sajana Weerawardhena, Aman Priyanshu, Adam Swanda, Amy Chang, Hyrum Anderson, Kojin Oshiba, Omar Santos, Yaron Singer, and Amin Karbasi. Llama-3.1-foundationai-securityllm-base-8b technical report, 2025. URL https://arxiv.org/abs/2504.21039. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. DistiLLM: Towards streamlined distillation for large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 24872β24895. PMLR, 21β27 Jul 2024. URL https://proceedings.mlr.press/v235/ko24c.html. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representa- tions revisited. In International conference on machine learning, pages 3519β3529. PMlR, 2019. Chanhyuk Lee, Jiho Choi, Chanryeol Lee, Donggyun Kim, and Seunghoon Hong. Adarank: Adaptive rank pruning for enhanced model merging. In International Conference on Learning Representations, volume 2026, pages 6484β 6510, 2026. Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equiva- lence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991β999, 2015. 11 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Shaotian Li, Shangze Li, Chuancheng Shi, Wenhua Wu, Yanqiu Wu, Xiaohan Yu, Fei Shen, and Tat-Seng Chua. Latent anomaly knowledge excavation: Unveiling sparse sensitive neurons in vision-language models. arXiv preprint arXiv:2604.07802, 2026. Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodol ` a. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36:55394β 55414, 2023. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018. URL https://arxiv.org/abs/1809.02789. Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi. Text-to-concept (and back) via cross-model align- ment. In International Conference on Machine Learning, pages 25037β25060. PMLR, 2023. Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodol ` a. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430, 2022. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024. Narmeen Oozeer, Dhruv Nathawani, Nirmalendu Prakash, Michael Lan, Abir Harrasse, and Amirali Abdullah. Ac- tivation space interventions can be transferred between large language models. arXiv preprint arXiv:2503.04429, 2025. Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36:66727β66754, 2023. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa : A large-scale multi-subject multi- choice dataset for medical domain question answering, 2022. URL https://arxiv.org/abs/2203.14371. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word prob- lems? In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 2080β2094, 2021. Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Sch Μ olkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in neural information processing systems, 36:79320β79362, 2023. Zihuan Qiu, Lei Wang, Yang Cao, Runtong Zhang, Bing Su, Yi Xu, Fanman Meng, Linfeng Xu, Qingbo Wu, and Hongliang Li. Null-space filtering for data-free continual model merging: Preserving stability, promoting plasticity. In International Conference on Learning Representations, volume 2026, pages 128492β128518, 2026. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017. Benjamin Ramtoula, Pierre-Yves Lajoie, Paul Newman, and Daniele De Martini. Fantastic features and where to find them: A probing method to combine features from multiple foundation models. Advances in Neural Information Processing Systems, 38:89960β89989, 2026. Paul Richmond, Constantinos Papageorgakis, Vasilis Niarchos, Borun Chowdhury, and Prarit Agarwal. Feyntune: large language models for high-energy theory. Machine Learning: Science and Technology, 7(2):025012, 2026. Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Angelo Porrello, and Simone Calderara. Transporting task vectors across different architectures without training. arXiv preprint arXiv:2602.12952, 2026. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reason- ing about social interactions. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 4463β4473, 2019. Chuancheng Shi, Shangze Li, Shiming Guo, Simiao Xie, Wenhua Wu, Jingtong Dou, Chao Wu, Canran Xiao, Cong Wang, Zifeng Cheng, et al. Where culture fades: revealing the cultural gap in text-to-image generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14702β14712, 2026a. 12 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Chuancheng Shi, Shangze Li, Wenjun Lu, Wenhua Wu, Cong Wang, Zifeng Cheng, Fei Shen, and Tat-Seng Chua. Tracerouter: robust safety for large foundation models via path-level intervention. arXiv preprint arXiv:2601.21900, 2026b. Chuancheng Shi, Wenhua Wu, Fei Shen, Xiaogang Zhu, Kun Hu, and Zhiyong Wang. Orthoeraser: coupled-neuron orthogonal projection for concept erasure. arXiv preprint arXiv:2603.11493, 2026c. Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045β22055, 2020. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering chal- lenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149β4158, 2019. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram Μ e, Morgane Rivi ` ere, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. David Thulke, Yingbo Gao, Petrus Pelser, Rein Brune, Rricha Jalota, Floris Fok, Michael Ramos, Ian van Wyk, Ab- dallah Nasir, Hayden Goldstein, Taylor Tragemann, Katie Nguyen, Ariana Fowler, Andrew Stanco, Jon Gabriel, Jordan Taylor, Dean Moro, Evgenii Tsymbalov, Juliette de Waal, Evgeny Matusov, Mudar Yaghi, Mohammad Shi- hadah, Hermann Ney, Christian Dugast, Jonathan Dotan, and Daniel Erasmus. Climategpt: Towards ai synthesizing interdisciplinary research on climate change, 2024. URL https://arxiv.org/abs/2401.09646. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In International Conference on Learning Representations, volume 2024, pages 18303β18322, 2024. Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. Fusechat: Knowledge fusion of chat models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21629β21653, 2025. Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. MixLLM: Dynamic routing in mixed large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10912β10922, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/20 25.naacl-long.545. URL https://aclanthology.org/2025.naacl-long.545/. Yongxian Wei, Runxi Cheng, Weike Jin, Enneng Yang, Li Shen, Lu Hou, Sinan Du, Chun Yuan, Xiaochun Cao, and Dacheng Tao. Optmerge: Unifying multimodal llm capabilities and modalities via model merging. In International Conference on Learning Representations, volume 2026, pages 61571β61595, 2026. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965β23998. Pmlr, 2022. Zhengqi Xu, Han Zheng, Jie Song, Li Sun, and Mingli Song. Training-free heterogeneous model merging. arXiv preprint arXiv:2501.00061, 2024. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in neural information processing systems, 36:7093β7115, 2023. Sihan Yang, Kexuan Shi, and Weiyang Liu. Orthogonal model merging. arXiv preprint arXiv:2602.05943, 2026. Kotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryotaro Shimizu, Ioannis Mitliagkas, and Hiroki Naganuma. Dis- tac: Conditioning task vectors via distillation for robust model merging. In International Conference on Learning Representations, volume 2026, pages 118509β118534, 2026. Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, et al. Chemdfm: A large language foundation model for chemistry. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024. 13 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Appendix This appendix provides additional details, analyses, and discussions that complement the main paper. Appendix A describes the domain-specific source models, training datasets, evaluation benchmarks, and metrics used across the 15 capability domains. Appendix B summarizes the notation used throughout SuperAI. Appendix C presents addi- tional experiments on PCA subspace dimensionality, training-data scale, and multi-source scalability. Appendix D provides further details on cross-layer matching and the selection of the shared host layer. Appendix E discusses the current scope and limitations of our evaluation, together with directions for future work. Finally, Appendix F provides additional discussion on several methodological and experimental questions concerning SuperAI. A Domain-Specific Source Models, Datasets, and Metrics We describe the source model, merging-training data, and evaluation metric for each of the 15 capability domains. Training datasets refer to the data used in our merging procedure, rather than necessarily the original fine-tuning corpora of the source models. All reported metrics are higher-is-better. Science. We use ScienceQA-OFT-8B as the source model and ARC-Challenge Clark et al. [2018] for merging training and evaluation. The task consists of multiple-choice science questions. We report accuracy, measured by whether the predicted option matches the reference answer. Common Sense. We use CommonsenseQA-OFT-8B as the source model and CommonsenseQA Talmor et al. [2019] as the corresponding dataset. The task requires selecting the correct answer to questions involving everyday common- sense knowledge. The primary metric is multiple-choice accuracy. Social. We use Social-IQA-OFT-8B as the source model and the BIG-bench Social IQA Sap et al. [2019] task for merging training and evaluation. Questions concern peopleβs intentions, reactions, and likely consequences in social situations. We report multiple-choice accuracy. Math. We use NuminaMath-OFT-8B as the source model and SVAMP Patel et al. [2021] for merging training and evaluation. The task requires generating numeric answers to arithmetic word problems. We extract the final numeric value from the generated response and compare it with the reference using relative and absolute tolerances of 10 β4 . The primary metric is numeric answer accuracy. Code. We use Qwen2.5-Coder-7B-Instruct Hui et al. [2024] as the source model and MBPP Austin et al. [2021] for merging training. We report Pass@1: a task is counted as solved only when a single generated program passes both the base and additional tests. Multilingual. We use MergeBench Llama-3.2-3B Multilingual He et al. [2026] as the source model and an eight- language subset of M-ARC Clark et al. [2018]. The task evaluates multiple-choice reasoning across languages. We report sample-averaged accuracy, pooling correct predictions over all evaluated examples rather than taking an un- weighted average of per-language accuracies. Safety. We use MergeBench Llama-3.2-3B Safety He et al. [2026] as the source model and PKU-SafeRLHF Ji et al. [2025] as the corresponding dataset. The task presents a user request and candidate responses, and asks the model to select the safer response. We report safer-response selection accuracy. This metric evaluates safety-related response selection rather than the safety of freely generated responses. Law. We use Saul-Instruct-v Colombo et al. [2024] as the source model. Merging training uses the local MMLU Law subset Hendrycks et al. [2020]. Both are formulated as multiple-choice legal questions. The primary metric is answer-selection accuracy. Finance/Business. We use Finance Llama-3.1-8B as the source model and FinQA Chen et al. [2022] for merging training and evaluation. The task requires answering questions based on financial information. Numeric answers are extracted and compared using relative and absolute tolerances of 10 β4 . For nonnumeric references, we use case- insensitive reference-answer matching in the generated response. We report the fraction of answers judged correct under these rules. Astronomy. We use AstroSage-8B de Haan et al. [2025] as the source model and arXiv astro-ph text for merging training. Evaluation uses MMLU Astronomy multiple-choice questions. The primary metric is accuracy on MMLU Astronomy; this score is distinct from language-modeling evaluation on the arXiv training corpus. 14 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Table 7: Training datasets and evaluation metrics for the 15 capability domains. We list the source models, training datasets, and corresponding evaluation metrics. All metrics are higher-is-better. DomainSource ModelTraining DatasetTask Metricβ ScienceScienceQA-OFT-8BARC-ChallengeAccuracy Common SenseCommonsenseQA-OFT-8BCommonsenseQAAccuracy SocialSocial-IQA-OFT-8BBIG-bench Social IQAAccuracy MathNuminaMath-OFT-8BSVAMPNumeric Answer Accuracy CodeQwen2.5-Coder-7B-InstructMBPPPass@1 MultilingualLlama-3.2-3B MultilingualM-ARCAveraged Accuracy SafetyLlama-3.2-3B SafetyPKU-SafeRLHFSafer-response Selection Accuracy LawSaul-Instruct-v1MMLU LawAccuracy Finance/BusinessFinance Llama-3.1-8BFinQAAnswer Accuracy AstronomyAstroSage-8BAstro-phAccuracy MedicalHuatuoGPT-o1-7BMedMCQAAccuracy ClimateClimateGPT-7BClimate-FEVER Binary Accuracy Cyber SecurityFoundation-Sec-8BSEVENLLMToken-level F1 ChemistryChemDFM-v1.5-8BMMLU ChemistryAccuracy PhysicsFeynTune-S1Hep-th ContinuationNext-sentence Selection Accuracy Medical. We use HuatuoGPT-o1-7B Chen et al. [2024] as the source model and MedMCQA Pal et al. [2022] for merging training and evaluation. The task consists of multiple-choice questions covering medical knowledge. We report accuracy based on agreement with the reference answer option. Climate. We use ClimateGPT-7B Thulke et al. [2024] as the source model and the local binary formulation of Climate- FEVER Diggelmann et al. [2021]. The task requires classifying climate-related statements according to the provided binary labels. The primary metric is classification accuracy. Cyber Security. We use Foundation-Sec-8B Kassianik et al. [2025] as the source model and SEVENLLM Ji et al. [2024a] for merging training and evaluation. The task requires generating responses to cybersecurity-related instruc- tions. We report token-level F1 between generated and reference responses, combining token precision and recall. Scores are computed for individual examples and then averaged over the evaluation set. Chemistry. We use ChemDFM-v1.5-8B Zhao et al. [2024] as the source model. Merging training uses a local mixture of the MMLU High School Chemistry and College Chemistry subsets, containing 165 and 6 training examples, respectively. Evaluation uses 100 questions from MMLU College Chemistry. The primary metric is multiple-choice accuracy. Physics. We use FeynTune-S1 Llama-3.1-8B Richmond et al. [2026] as the source model and continuation data constructed from arXiv hep-th text. The task requires selecting the true next sentence from a set of candidates. We report next-sentence selection accuracy, measuring whether the selected continuation matches the reference. Aggregate Evaluation. For multi-source fusion, we report the unweighted mean of the 15 domain scores after ex- pressing them on a common scale. Each domain receives equal weight, regardless of its evaluation-set size. This aggregate is a mean of task-specific metrics, rather than a pooled accuracy over all questions. We additionally report performance on a separate MMLU evaluation set to assess general-capability retention; this score is excluded from the 15-domain average. Generalization Evaluation. For cross-dataset generalization, we evaluate Science, Common Sense, Social, and Cli- mate capabilities on OpenBookQA Mihaylov et al. [2018], PIQA Bisk et al. [2019], CosmosQA Huang et al. [2019], and ClimateCheck Abu Ahmad et al. [2025], respectively, none of which is used for capability merging or residual- MLP training. B Notation Summary Table 8 summarizes the notation used throughout SuperAI and groups the main symbols according to the three stages of the framework: orthogonal subspace alignment, feature mapping, and residual fusion. We use h and s to denote the 15 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Table 8: Summary of notation used in SuperAI. SymbolDefinition Models and representations f h ,f s K s=1 Host model and K source models. d h ,d s Hidden dimensions of the host and source model s. A s =a i n s i=1 Shared anchor prompts for source model s and the host. β m ,h (β m ) m (a i )Layer index and last non-padding-token hidden state of model mβs,h. H (β m ) m Matrix of hidden states for prompts inA s . Subspace alignment ΞΌ m ,u m (a i ),U m Hidden-state mean, centered and normalized state, and matrix of normalized states. c m ,P m ,rPCA mean, principal-component matrix, and shared subspace dimension. Z m PCA coordinates of model m for the paired prompts. R β s , b Z sβh Orthogonal source-to-host map and aligned source coordinates. S s (β s ,β h )Held-out alignment score for a sourceβhost layer pair. w s Weight of source model s, with P K s=1 w s = 1. β β s (β h ),β β h Best source layer for a candidate host layer and selected shared host layer. Feature mapping z s (a i ),bz sβh (a i )Source PCA coordinates and their rotated host-space coordinates. b sβh (a i )Rotated coordinates reconstructed in the normalized host hidden space. e h sβh (a i )Mapped host-space representation after magnitude restoration. β s (a i )Target feature increment e h sβh (a i )β h h (a i ). Residual fusion and optimization h i,t ,h β² i,t Host hidden state before and after residual injection at token position t. b,W down ,W up MLP bottleneck width, down-projection, and up-projection. q i,t , b β i,t ,Ξ±Bottleneck activation, predicted increment, and residual scaling factor. A tr s ,s i ,NSource-s training prompts, source identity of prompta i , and total training-prompt count. p i ,Ο s Last non-padding position of prompt a i and source-specific increment normalization factor. T i ,y i,t ,p ΞΈ Answer-token positions, ground-truth token, and model output distribution. L task ,L inc Answer-token cross-entropy and increment-supervision losses. Ξ» task ,Ξ» inc ,LLoss weights and their weighted joint objective. Ξ΅Positive constant used for numerical stability. host model and a source model, respectively, while β h and β s index their corresponding layers. Shared anchor prompts are used to extract paired hidden representations from the source and host models, which are then normalized and projected into low-dimensional PCA subspaces. The resulting coordinates are aligned through an orthogonal mapping, and the alignment scores are subsequently used for sourceβhost layer matching and shared host-layer selection. After alignment, the source representation is mapped into the host hidden space to construct a source-specific target feature increment, denoted by β s (a i ). These increments provide representation-level supervision for the residual fusion module. During joint training, a lightweight residual MLP predicts input-dependent increments from the host hidden states, and the resulting update is injected into the selected host layer. The training objective combines task- output supervision with source-increment supervision, controlled by the corresponding loss weights. For each source model, the host-side normalization statistics, PCA mean, and PCA basis are fitted using the anchor prompts paired with that source model. Therefore, these quantities are technically source-dependent even when they belong to the host representation space. To simplify the notation, we omit the corresponding source index whenever it is unambiguous from the context. The same convention is used for other source-conditioned intermediate quantities where explicitly carrying the source index would unnecessarily complicate the presentation. C More Experiments C.1 Effect of PCA Subspace Dimension 16 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models Table 9: Effect of PCA subspace dimensionality. Each cell re- ports the 15-domain average score, with the absolute change from the corresponding base model shown in parentheses. PCA rank denotes the requested upper bound. PCA Rank Llama-3.2-3B Qwen2.5-3B Pythia-2.8B Base46.756.222.9 1656.3 (β 9.6)60.2 (β 4.0) 28.7 (β 5.8) 3255.3 (β 8.6)61.3 (β 5.1) 28.3 (β 5.4) 6454.5 (β 7.8)61.2 (β 5.0) 29.0 (β 6.1) 12854.2 (β 7.5)61.3 (β 5.1) 32.8 (β 9.9) 25655.2 (β 8.5)60.8 (β 4.6) 28.9 (β 6.0) 51256.9 (β 10.2)59.8 (β 3.6) 28.9 (β 6.0) To evaluate the effect of the PCA subspace dimension on multi-source capability transfer, we vary the PCA rank from 16 to 512 and compare the performance across three host models: Llama-3.2-3B, Qwen2.5-3B, and Pythia-2.8B. The results show in Table 9 that all PCA rank settings outperform their corresponding base mod- els, demonstrating that the proposed method effectively integrates multi-domain capabilities across different sub- space dimensions. Specifically, Llama-3.2-3B achieves its highest average score of 56.9 at rank 512, improv- ing by 10.2 points over the base model. Qwen2.5-3B attains a peak score of 61.3 across ranks 32 and 128, with a 5.1-point gain. Pythia-2.8B reaches its best score of 32.8 at rank 128, yielding a 9.9-point improvement. Thus, Llama-3.2-3B, Qwen2.5-3B, and Pythia-2.8B deliver optimal performance at ranks 512, 32 and 128, and 128, respectively. Overall performance does not scale monotonically with increasing rank. A plausible reason is that higher-dimensional subspaces retain more redundant and complex signals, which complicates optimization for the mapping module and hurts capability transfer. Considering the overall performance across all three host models, rank 128 achieves a solid balance between information preservation and training difficulty, and is therefore adopted as the default configuration. C.2 Effect of Training Data Scale Table 10: Effect of training data volume. Each cell reports the 15-domain average score, with the absolute change from the cor- responding base model shown in parentheses. The training cap is applied per domain; domains with fewer examples use all avail- able training samples. Examples/Domain Llama-3.2-3B Qwen2.5-3B Pythia-2.8B Base46.756.222.9 5054.1 (β 7.4)57.9 (β 1.7) 27.8 (β 4.9) 10056.3 (β 9.6)59.6 (β 3.4) 27.1 (β 4.2) 20057.6 (β 10.9)58.9 (β 2.7) 26.9 (β 4.0) 50055.6 (β 8.9)60.0 (β 3.8) 27.4 (β 4.5) 100054.2 (β 7.5)61.3 (β 5.1) 32.8 (β 9.9) To analyze the effect of training data scale on multi- source capability transfer, we vary the maximum num- ber of training samples per domain from 50 to 1,000 and compare performance across three host models. The re- sults in Table 10 indicate that even with only 50 samples per domain, all three models markedly outperform their respective base models, demonstrating high data effi- ciency. As the sample size increases, different host mod- els exhibit distinct trajectories: Llama-3.2-3B achieves its peak average score of 57.6 with 200 samples, whereas Qwen2.5-3B and Pythia-2.8B reach their highest scores of 61.3 and 32.8 at 1,000 samples, respectively. These findings show that increasing training data generally pro- motes more adequate capability mappings, yet perfor- mance does not improve monotonically with data scale. An excessive number of samples may introduce redun- dant or complex patterns, thereby complicating opti- mization for the mapping module. Overall, the proposed method yields substantial improvements in data-scarce settings, while larger training sets provide additional gains for certain host models. C.3 Multi-Source Scalability To evaluate the scalability of cross-architecture multi-source capability fusion, we incrementally integrated 1β15 source models in a fixed order and compared SuperAI with THESEUS and Transport and Merge (T&M) across four representative domains and three host architectures. As shown in Figure 9, scores generally declined as more sources were incorporated. Averaged across the 12 domainβhost combinations, SuperAI declined by 7.7 points from 1 to 15 sources, compared with 12.8 and 13.4 points for THESEUS and T&M, respectively. SuperAI achieved the highest scores at every evaluated source count in Math, Safety, and Physics. At the 15-source endpoint, the sole domainβhost combination in which it did not lead was Law on Llama-3.2-3B, where it trailed T&M by 0.4 points. At that endpoint, SuperAI exceeded the better of the two baselines in each domainβhost combination by 19.2 points on average, with particularly large margins in Safety and Physics on Pythia-2.8B and Physics on Llama-3.2-3B. These results suggest that SuperAI mitigates interference under the evaluated fixed source order, although its scores still decline as more sources are integrated. 17 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models 13581115 20 30 40 50 (a) Math Β· Llama-3.2-3B 13581115 20 30 40 50 (b) Math Β· Qwen2.5-3B 13581115 2 4 6 8 10 12 (c) Math Β· Pythia-2.8B 13581115 55 60 65 70 75 (d) Safety Β· Llama-3.2-3B 13581115 55 60 65 70 75 80 85 (e) Safety Β· Qwen2.5-3B 13581115 20 40 60 80 (f) Safety Β· Pythia-2.8B 13581115 37.5 40.0 42.5 45.0 47.5 50.0 (g) Law Β· Llama-3.2-3B 13581115 40 45 50 55 60 (h) Law Β· Qwen2.5-3B 13581115 10 20 30 40 (i) Law Β· Pythia-2.8B 13581115 60 70 80 90 (j) Physics Β· Llama-3.2-3B 13581115 70 80 90 100 110 (k) Physics Β· Qwen2.5-3B 13581115 20 40 60 80 100 (l) Physics Β· Pythia-2.8B Number of experts Capability score Host BaseTHESEUST&MSuperAI (ours) Figure 9: Multi-source scalability across capability domains. Capability scores of SuperAI, THESEUS, and T&M as the number of merged sources increases across Math, Safety, Law, and Physics. D Cross-Layer Matching and Shared Host-Layer Selection This section provides additional details on the selection of the shared host layer. Let L h denote the candidate host layers andL s denote the candidate layers of source model s. For each source model, we use prompts that are shared with the host model: A s =a i n s i=1 .(10) For every candidate layer pair (β s ,β h ), we extract the hidden state of the last non-padding token from both models. We then apply the normalization, PCA projection, and orthogonal Procrustes alignment defined in Section 3.1. The mapping is fitted onA fit s and evaluated on a disjoint matching setA val s . For a validation prompt a i , the aligned source 18 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models coordinate is bz (β s ,β h ) sβh (a i ) = z (β s ) s (a i )R (β s ,β h ) s .(11) We define the matching score of a candidate layer pair as the mean cosine similarity between the aligned source coordinates and the paired host coordinates: S s (β s ,β h ) = 1 |A val s | X a i βA val s cos bz (β s ,β h ) sβh (a i ),z (β h ,s) h (a i ) .(12) A higher score indicates that the source and host layers produce more consistent representations for the same inputs after orthogonal alignment. CKA and Retrieval@1 are reported as additional diagnostic metrics but are not used for layer selection. For each candidate host layer, we first identify the best-matching layer from every source model: β β s (β h ) = arg max β s βL s S s (β s ,β h ).(13) The corresponding score is S β s (β h ) = S s (β β s (β h ),β h ).(14) We then aggregate the best matching scores across all source models: S(β h ) = K X s=1 w s S β s (β h ), K X s=1 w s = 1.(15) We use w s = 1/K by default, so each source model contributes equally to layer selection. The shared host layer is selected as β β h = arg max β h βL h S(β h ),(16) and the final layer selected from source model s is β β s = β β s (β β h ).(17) Thus, different source models may contribute representations from different source layers, while all representations are aligned with the same host layer β β h . After layer selection, we refit the corresponding mappings using the available training anchors and use them to construct the source-specific feature increments. The layer search is performed only once before residual training and introduces no additional source-model cost during inference. E Limitation Our current evaluation focuses on offline benchmark-based capability fusion, where prompts and target tasks are pre- defined. We have not yet systematically evaluated SuperAI in interactive humanβAI settings involving multi-turn dialogue, evolving user intent, or feedback-dependent behavior. While this does not affect the validity of the reported offline fusion results, interactive deployment may introduce additional challenges such as maintaining transferred capabilities consistently across dialogue turns and adapting residual updates to changing conversational context. Fu- ture work will extend SuperAI to interactive settings by incorporating conversation-level representations, turn-aware residual routing, and human-feedback-based evaluation to study how fused capabilities behave under dynamic user interactions. F More Discussion β· Q1. The method uses ground-truth task labels to train the residual MLP. How can the authors distinguish capability transfer from ordinary supervised adaptation of the host model? We would like to clarify that SuperAI is not equivalent to standard supervised adaptation. The key distinction is that the residual MLP is jointly optimized with both the task loss and the source-derived increment supervision. As shown in Table 2, using only the task loss (Ξ» inc = 0) consistently underperforms the joint objective across all three host models, while using only the increment loss improves representation matching but does not provide the same downstream gains. This suggests that the improvement does not arise from task supervision alone; rather, the source- derived increments provide an additional transfer signal that complements the task objective. The role of the task loss is to adapt the transferred source information to the downstream computation of the host model, which is necessary because representation correspondence alone does not guarantee functional compatibility. 19 SuperAI: Merging Capabilities Across Heterogeneous Foundation Models β· Q2. The improvement over THESEUS and Transport-and-Merge is large. Could this simply be due to the additional trainable residual MLP rather than the proposed cross-model alignment mechanism? Our ablations indicate that the gain cannot be explained by the residual MLP alone. First, removing source-increment supervision and retaining the same task-trained residual module leads to substantially lower performance than the full joint objective. Second, Table 3 shows that, under the same downstream training setup, replacing PCA + orthogonal alignment with no alignment, PCA-only mapping, or unconstrained linear mapping consistently degrades performance across the three host architectures. Third, Table 4 shows that the nonlinear residual mapper further improves over a linear mapper, indicating that the alignment module and the host-adaptation module play complementary roles rather than one simply substituting for the other. Therefore, the empirical gains result from the combination of cross-model geometric alignment and host-compatible residual adaptation. β· Q3. The paper reports very high post-alignment similarity. Could these scores merely reflect overfitting of the Procrustes mapping to paired anchor samples? The reported alignment scores are not computed on the same samples used to fit the mapping. As described in the Appendix, for every candidate sourceβhost layer pair, the normalization, PCA projection, and orthogonal Procrustes mapping are fitted on A fit s , while the matching score is evaluated on a disjoint validation set A val s . CKA and Re- trieval@1 are also used as independent diagnostic metrics rather than as optimization objectives for layer selection. Moreover, Table 5 shows that downstream capability transfer improves systematically as the held-out post-alignment Match score increases from weakly matched to medium and best-matched layer pairs. This provides evidence that the alignment score captures transferable cross-model structure rather than merely fitting the anchor examples. β· Q4. Table 1 shows that some domains degrade after multi-source fusion. For example, Code decreases from 39.7 to 28.3 on Llama-3.2-3B, from 55.6 to 44.2 on Qwen2.5-3B, and from 11.1 to 1.3 on Pythia-2.8B. How should these results be interpreted? The Code results reflect the trade-off inherent in joint multi-domain fusion rather than the overall effectiveness of capability integration. SuperAI optimizes a single host model across 15 heterogeneous domains, whose objectives and output formats differ substantially. In particular, code generation requires highly structured token-level generation, making its preferred representation updates different from those of many reasoning and classification domains. Con- sequently, optimizing the shared representation for broad multi-domain performance does not necessarily preserve the optimum of every individual domain. Importantly, the goal of this setting is joint capability composition rather than independent per-domain specialization, and SuperAI consistently improves the aggregate 15-domain performance across all three hosts. Future work will investigate domain-conditioned weighting or routing to further improve the balance among heterogeneous capabilities. 20