Paper deep dive

ES-Merging: Biological MLLM Merging via Embedding Space Signals

Wonbin Lee, Dongki Kim, Sung Ju Hwang

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 83

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/22/2026, 5:08:55 AM

Summary

ES-Merging is a representation-aware framework for merging biological multimodal large language models (MLLMs) by utilizing embedding space signals rather than parameter space heuristics. It computes layer-wise and element-wise merging coefficients based on the distributional shift of modality-specific embeddings, enabling more robust cross-modal knowledge composition.

Entities (5)

ES-Merging · framework · 100%MLLM · model-architecture · 100%LoRA · technique · 95%Sliced Wasserstein Distance · metric · 95%BindingDB · dataset · 90%

Relation Signals (3)

ES-Merging → uses → Embedding Space Signals

confidence 100% · ES-Merging... estimates merging coefficients from embedding space signals.

ES-Merging → improves → Model Merging

confidence 95% · Experiments on interactive effect prediction benchmarks show that our method outperforms existing merging methods

MLLM → implements → LoRA

confidence 90% · In the MLLM setting, modality specialization is often implemented as a parameter efficient tuning with LoRA

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose a representation-aware merging framework that estimates merging coefficients from embedding space signals. We first design a probe input that consists of different modality tokens and forward it through each specialized MLLM to obtain layer-wise embedding responses that reflect modality-specific representation changes. We then estimate complementary merging coefficients at two granularities from the embedding space: layer-wise coefficients from coarse-grained signals and element-wise coefficients from fine-grained signals, which are jointly combined for robust coefficient estimation. Experiments on interactive effect prediction benchmarks show that our method outperforms existing merging methods and even surpasses task-specific fine-tuned models, establishing that embedding space signals provide a principled and effective foundation for cross-modal MLLM merging.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

82,316 characters extracted from source content.

Expand or collapse full text

ES-Merging: Biological MLLM Merging via Embedding Space Signals Wonbin Lee *1 Dongki Kim *1 Sung Ju Hwang 1,2 1 KAIST 2 DeepAuto.ai smilelwb01, cleverki, sungju.hwang@kaist.ac.kr Abstract Biological multimodal large language models (MLLMs) have emerged as powerful founda- tion models for scientific discovery. How- ever, existing models are specialized to a sin- gle modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a uni- fied MLLM, existing methods rely on input- agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose a representation-aware merging framework that estimates merging coefficients from embedding space signals. We first design a probe input that consists of different modality tokens and forward it through each specialized MLLM to obtain layer-wise embedding responses that re- flect modality-specific representation changes. We then estimate complementary merging co- efficients at two granularities from the em- bedding space: layer-wise coefficients from coarse-grained signals and element-wise coef- ficients from fine-grained signals, which are jointly combined for robust coefficient estima- tion. Experiments on interactive effect predic- tion benchmarks show that our method outper- forms existing merging methods and even sur- passes task-specific fine-tuned models, estab- lishing that embedding space signals provide a principled and effective foundation for cross- modal MLLM merging. 1 Introduction Multimodal Large Language Models (MLLMs) have been emerging as crucial foundational mod- els for scientific discovery, extending their per- ception to diverse biological modalities across molecules (Park et al., 2024; Kim et al., 2025), proteins (Abdine et al., 2024; Fei et al., 2025), and * denotes Equal Contribution cells (Fang et al., 2025b; Rizvi et al., 2025). De- spite their impressive progress within each modal- ity, many scientific problems of interest are cross- modal, requiring to understand the interactive ef- fects such as protein-ligand interactions or drug effectiveness to cell types. However, existing bio- logical MLLMs are specialized to a single modality, resulting in limited intersectional knowledge and unreliable cross-modal reasoning. Building a unified model by jointly training on different modalities is a straightforward approach to upskill cross-modal understanding. However, it is impractical and time-consuming since construct- ing curated cross-modal instruction datasets in the scientific domain typically requires intensive labor and highly specific expertise to elaborate the un- derlying principles of complex interactions. As an alternative, model merging has gained attention by efficiently combining parameters of multiple specialized models. To retain their task-specific knowledge, existing methods (Yadav et al., 2023; Huang et al., 2024; Du et al., 2024) exploit param- eter space signals, such as magnitudes, signs, and directions, heuristically assigning merging coeffi- cients. However, parameter space heuristics are input-agnostic and therefore provide only weak, indirect proxies for modality-specific adaptation. Such input-blindness makes it difficult to isolate meaningful cross-modal interactions, failing to ac- curately combine these adaptations and severely degrading cross-modal merging. Our main observation is that the input-aware embedding space contains the modality-specific in- formation. As shown in Fig. 1, when molecular to- kens are processed by different MLLMs, the hidden representations form clearly different embedding distributions. In particular, the molecule-specific LLM induces a more distinct distribution due to its modality-specific understanding. Further, we measure the embedding distribution distance be- tween the base LLM and each specialized MLLM 1 arXiv:2603.14405v1 [cs.LG] 15 Mar 2026 Molecule Token Embedding Space Base LLM Cell LLM Protein LLM Molecule LLM Figure 1: Molecule token embedding visualization of the last transformer block for the base LLM and each specialized LLM. 024681012141618202224262830 0.2 0.4 0.6 0.8 Average Normalized SWD Specialized Non-Specialized Figure 2: Layer-wise embedding distribution distance under specialized and non-specialized tokens using sliced Wasserstein distances (SWD) (Bonneel et al., 2015). under specialized and non-specialized token inputs. Fig. 2 shows that specialized inputs consistently produce larger embedding distribution distance than non-specialized inputs, suggesting that the embedding responses faithfully reflect modality- specific adaptation when the input matches the model’s specialization. This observation motivates our central design principle: rather than relying solely on heuristic parameter space signals, we esti- mate merging coefficients based on the layer-wise embedding space signals. Motivated by this observation, we propose an embedding-signal-based MLLM merging (ES- Merging), a novel framework that moves the model merging paradigm from parameter space signals to embedding space signals. Our intuition is that the modality specialization of an MLLM can be mea- sured by the differences in the embedding space between the base LLM and the MLLM as shown in Fig. 1. To this end, we first design a probe input containing multimodal tokens of different modali- ties. By forwarding this input through each MLLM and the base LLM, we obtain layer-wise embed- dings that reflect modality-specific representation changes across layers. Based on these embeddings, we propose to com- pute merging coefficients at two complementary granularities: layer-wise and element-wise. In the layer-wise manner, we compute a layer-level impor- tance score by identifying layers where the embed- ding distributional shift grows the most, capturing coarse-grained specialization. In the element-wise manner, we estimate fine-grained importance by identifying parameters that most influence the rep- resentation shift. We then combine the coarse layer importance and fine element importance to produce final merging coefficients, which are used to fuse specialized models into a single unified MLLM. This design enables more robust and calibrated co- efficient estimation by combining layer-level spe- cialization with parameter-level sensitivity. We validate the effectiveness of ES-Merging on interactive effect prediction tasks over diverse bio- logical modalities by merging three different spe- cialized MLLMs into a single unified model. Ex- perimental results show that ES-Merging outper- forms not only other model merging methods but also the task-specific fine-tuned models, empiri- cally confirming our representation-aware merging framework is crucial to obtain the salient modality- specific signals. Further analyses show the effec- tiveness of our components: combining layer- and element-wise merging coefficients best performs compared to using only one, highlighting the ne- cessity of integrating complementary specialization signals at different granularities. 2 Related Work MLLMs for Scientific Discovery Large lan- guage models (LLMs) (Touvron et al., 2023; Grattafiori et al., 2024; OpenAI, 2024a,b; Comanici et al., 2025) are increasingly being extended to scientific discovery through multimodal large lan- guage models (MLLMs) that incorporate diverse biological modalities, including molecules (Wang et al., 2025b), proteins (Xiao et al., 2025), and cells (Dip et al., 2025). In the protein domain, prior works have modeled amino-acid sequences (Xu et al., 2023; Pei et al., 2023) or jointly with struc- tures (Guo et al., 2023; Abdine et al., 2024; Wang et al., 2024a; Xiao et al., 2024; Wang et al., 2025a; Fei et al., 2025). Single cell LLMs learn the scRNA-seq representations (Fang et al., 2025a; Kharouiche et al., 2025; Fang et al., 2025b) or fur- ther incorporate histology information (Li et al., 2025; Chen et al., 2025). On the other hand, 2 molecular LLMs have been built on 1D string rep- resentations such as SMILES (Weininger, 1988) and SELFIES (Krenn et al., 2020) (Chithrananda et al., 2020; Edwards et al., 2022), 2D molecular graphs (Liu et al., 2023; Fang et al., 2023; Cao et al., 2023; Yu et al., 2024; Park et al., 2024), 3D structures (Li et al., 2024; Guo et al., 2024), and joint 2D-3D representations (Kim et al., 2025). De- spite this progress, biological MLLMs are limited to a single modality, hindering their ability to solve cross-modal scientific problems. Model Merging Model merging aims to fuse knowledge from multiple specialist models with minimal additional data or training by combining them directly in parameter space. Existing meth- ods mainly leverage the parameter space signals to guide merging. Magnitude-based methods in- clude Task Arithmetic (Ilharco et al., 2022), Con- sensus Merging (Wang et al., 2024b), and PCB- Merging (Du et al., 2024); sign-based methods in- clude TIES-Merging (Yadav et al., 2023) and EMR- Merging (Huang et al., 2024); and LS-Merge (Soro et al., 2026) performs merging in a learned la- tent space over parameters. Beyond such static merging, test-time adaptation methods dynami- cally adjust coefficients using unlabeled test data, as in AdaMerging (Yang et al., 2023) and Twin- Merging (Lu et al., 2024). However, these methods still rely on parameter space signals, which are not well suited to capturing semantic discrepan- cies across heterogeneous modalities. We instead propose an embedding space merging framework for MLLMs that derives merging coefficients from modality-aware representation signals. 3 Preliminary We begin by formally describing MLLMs, then formulating the model merging with LoRA. Multimodal Large Language Model MLLMs operate beyond the textual space by taking modal- ity token embeddings with text tokens. Concretely, letM = m 1 ,...,m K denote a set of modal- ities. For each modalitym i , a modality-specific encoderf m i projects the raw modality inputx m i to a sequence of modality tokens:H m i = f m i (x m i ). Then, an MLLM generates textual outputyby tak- ing the concatenated textual and modality tokens: y = g(H 0 ), where H 0 = [H text ; H m i ;...]. In this view, the core interface of an MLLM is the token-based embeddings: raw modality in- puts are first mapped into sequences of vectors that are compatible with the LLM’s embedding di- mension, and then processed together by a single transformer. LoRA Merging In the MLLM setting, modality specialization is often implemented as a param- eter efficient tuning with LoRA (Hu et al., 2022). Therefore, our merging methods are based on merg- ing LoRA parameters via weighted summation. Specifically, letθ m i denote the LoRA parameter of an MLLM specialized to modalitym i . We in- dex the parameters by layer and weight element: θ l,n m i denotes then-th weight in thel-th layer of the modality-specific model form i . Then, the LoRA merging is formalized as follows: θ l,n uni ← X m i ∈M λ l,n m i θ l,n m i .(1) Under this formulation, the main challenge is to estimate appropriate merging coefficientsλ l,n m i that determine how strongly each modality-specific LoRA parameter contributes to the modality under- standing. Our method addresses this coefficient es- timation problem by derivingλ l,n m i from embedding space signals rather than parameter space statistics. 4 ES-Merging We present Embedding-Signal-based MLLM Merg- ing (ES-Merging), a representation-aware frame- work for merging modality-specialized MLLMs based on embedding space signals. Specifically, ES-Merging constructs probe inputs to elicit repre- sentational differences across models, and converts these embedding space discrepancies into layer- wise and element-wise merging coefficients. 4.1 Probe Input As discussed in Section 3, we view the projected modality tokens as the core interface of the LLM backbones. In other words, once a raw modality input is mapped into the shared embedding space, the modality-specific knowledge of an MLLM is reflected in how the backbone interprets and trans- forms these token embeddings across layers. From this perspective, understanding modality special- ization reduces to analyzing the layer-wise transfor- mation of modality tokens in representation space. Motivated by this view, we design probe inputs that explicitly expose token embeddings of differ- ent modalities, so that we can compare how the 3 BaseLLM Modality 풎 풋 -specialized LLM Layer 푙 BaseLLM Modality 풎 풋 -specialized LLM Modality 풎 ퟏ Tokens Layer 푙 Base LLM Distribution Distance (SWD) (a) Layer-wise Global Merging Coefficient(b) Element-wise Local Merging Coefficient Mean Pool Mean Pool Modality 풎 ퟐ Tokens ... Coarse-grained Space (Modality 풎 ퟏ ) Modality 풎 ퟏ Tokens Modality 풎 ퟐ Tokens Layer 푙 Backprop ∆SWD L2 Distance ... Figure 3: Overview of ES-Merging. (A) Layer-wise global merging coefficients are computed from the coarse- grained embedding signals, which are the layer-wise differences of distribution distances between mean pooled representations of the base LLM and a specialized MLLM. (B) Element-wise local merging coefficients are assigned based on the fine-grained embedding signals by computing the gradients from the embedding-wise distances. base LLM and each modality-specialized MLLM process the same modality embeddings. Specifically, for each modalitym i , we collect a set of raw inputsx (k) m i K k=1 from correspond- ing datasets and transform them via the modality- specific encoderf m i into the modality token em- beddingsH (k) m i K k=1 , whereH (k) m i = f m i (x (k) m i ) . Using these embeddings, we then build the probe input by concatenating a short textual prefix with all modality token blocks: H 0,(k) probe = [H text,m 1 ; H (k) m 1 ; H text,m 2 ; H (k) m 2 ;...], whereH text,m i denotes the text-prefix embeddings associated with modalitym i (e.g., the modal- ity identifier or name).In the biological set- ting, the modality setMtypically consists of molecule, protein, cell, and the resulting probe input is illustrated in Fig. 4. By forwarding each probe input through the base LLM and each MLLM specialized to modalitym j , we obtain layer-wise embeddings by extracting the modality m i tokens, denoted asH l,(k) m i →base andH l,(k) m i →θ m j inR T m i ×d , whereldenotes the layer index andT m i denotes the number of modality-m i tokens. These layer-wise representations serve as the foundation for our subsequent merging coefficient estimation. We regard the embedding discrep- ancy between the base LLM and each modality- specialized MLLM as an embedding space signal that reflects the degree of the modality specializa- tion. We then leverage this signal at two comple- mentary levels: globally across layers to estimate which layers contribute more to specialization, and locally within each layer to identify which parame- Molecule: <molecule_tokens> Protein: <protein_tokens> Cell: <cell_tokens> Figure 4: Prompt template of the probe input. ter elements are more strongly associated with the specialized transformation. 4.2 Layer-wise Global Merging Coefficient To capture global modality specialization, we com- pute a layer-wise global merging coefficient from coarse-grained embedding signals (Fig. 3 (a)). Coarse-grained Embedding Signal We first summarize the modality-specific embeddings into a coarse-grained representation by averaging token- level embeddings, obtaining ˆ h l,(k) m i →base ∈R d for the base model and ˆ h l,(k) m i →θ m j ∈R d for a spe- cialized modelθ m j . By collecting these coarse- grained embeddings over differentKprobe in- puts, we obtain two layer-wise embedding sets, H l m i →base ∈R K×d andH l m i →θ m j ∈R K×d , which represent how the base and specialized models pro- cess modality-m i inputs at layerl. We then quan- tify their representational gap using the embedding distribution distance with sliced Wasserstein dis- tance (SWD) (Bonneel et al., 2015): SWD l m i →θ m j = SWD H l m i →base , H l m i →θ m j . A larger value indicates thatθ m j induces a stronger layer-wise shift from the base model when process- ing modality-m i tokens. 4 Layer-wise Importance Estimation While SWD l m i →θ m j measures the cumulative differences up to layerl, we are particularly interested in how much new modality-specific transformation is introduced at each layer. We therefore compute the layer-wise change: d l m i →θ m j = SWD l m i →θ m j − SWD l−1 m i →θ m j . Ifd l m i →θ m j is large, the corresponding layer of modelθ m j contributes more strongly to modality m i -specific processing. Since the magnitude of these changes can vary across models, we normal- ize them over layers using Z-score normalization, obtaining ˆ d l m i →θ m j . We aggregate the normalized changes over all input modalities to obtain the layer- wise importance score of model θ m j : s l θ m j = X m i ∈M ˆ d l m i →θ m j . Layer-wise Global Coefficient Finally, we con- vert the layer-wise importance scores into merging coefficients by applying a softmax across models: α l m j = exp(s l θ m j /τ) P m∈M exp(s l θ m /τ) .(2) As a result, a specialized MLLM receives a larger coefficient at layers where it contributes more strongly to coarse-grained representational changes over different modality inputs. 4.3 Element-wise Local Merging Coefficient While the layer-wise global coefficient captures modality importance at the transformer-layer level, it assigns a uniform merging weight to all param- eters within the same layer. To address this limi- tation, we further introduce an element-wise local merging coefficient derived from fine-grained em- bedding signals (Fig. 3 (b)). Fine-grained Embedding Signals To measure the fine-grained signals from embeddings, we first measure the distances of each embedding between the base model and a specialized modelθ m j , in- stead of using the coarse-grained representations: r l,(k) m i →θ m j = H l,(k) m i →base − H l,(k) m i →θ m j F , whereFdenotes the Frobenius norm, which is anal- ogous to the EuclideanL 2 norm for vectors. This distance measures how differently the specialized modelθ m j processes modality-m i inputs relative to the base model at layer l in a fine-grained manner. Element-wise Importance EstimationWe then estimate the importance of each parameter element by measuring how sensitive this distance is to that element. Specifically, for then-th parameter ele- ment in layerl, we accumulate the absolute gradi- ent magnitude over all modalities and probe inputs: s l,n θ m j = X m i ∈M K X k=1 ∂ r l,(k) m i →θ m j ∂ θ l,n m j . A larger score indicates that the parameter element is more sensitive to the modality-specific differ- ences between the base and specialized models. Because the raw scores can vary across layers and parameters, we normalize them within each layer using Z-score normalization, yielding ˆs l,n θ m j . Element-wise Local CoefficientWe convert the normalized scores into element-wise merging coef- ficients by applying the softmax function: β l,n m j = exp ˆs l,n θ m j /τ P m∈M exp ˆs l,n θ m /τ .(3) Resulting element-wise merging coefficients selec- tively assign each parameter element according to their fine-grained sensitivities. 4.4 Integrating Layer- and Element-wise Merging Coefficients The coarse-grained layer-wise coefficientα l m i and the fine-grained element-wise coefficientβ l,n m i each capture modality-specific importance at different levels of granularity. We combine them into a sin- gle coefficient by multiplying the layer-wise coef- ficient and the element-wise coefficient and renor- malizing across modalities: λ l,n m i = α l m i · β l,n m i P m∈M α l m · β l,n m .(4) The final parameters of the merged model are computed by Eq. 1. By integrating different co- efficient types from different granularities of em- bedding signals, ES-Merging enables calibrated merging coefficients that preserve complementary modality expertise more faithfully, enabling robust cross-modal knowledge composition. 5 Experimental Results 5.1 Experimental Setup Implementation Details For merging, we lever- age the state-of-the-art MLLMs specialized to 5 Molecule-Protein InteractionMolecule-Cell Interaction BindingDBBioSNAPHumanAvg.DrugCombGDSC2Avg. Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1 Base LLM and Specialized MLLMs LLaMA-3.1-8B-Instruct51.951.761.560.259.058.857.556.973.777.184.885.779.381.4 Mol-LLaMA (Kim et al., 2025)55.854.766.5 64.061.561.461.260.064.180.455.282.359.781.4 Prot2Text-V2 (Fei et al., 2025)59.259.255.354.147.247.253.953.565.678.459.767.062.672.7 Cell-o1 (Fang et al., 2025b)53.553.159.359.849.148.5 54.053.876.377.285.986.081.181.6 Merging Methods Avg. Merging65.364.966.466.560.960.9 64.264.172.574.685.487.978.981.2 TIES-Merging (Yadav et al., 2023)60.860.862.761.658.658.660.760.374.777.085.987.180.382.1 EMR-Merging (Huang et al., 2024)64.7 64.266.366.960.460.463.863.845.271.593.493.369.382.4 AdaMerging (Yang et al., 2023)52.351.664.362.060.060.258.957.948.036.946.538.847.237.8 PCB-Merging (Du et al., 2024)55.155.160.358.858.658.4 58.057.477.378.786.086.181.782.4 Consensus Merging (Wang et al., 2024b)59.259.362.161.059.259.160.259.876.078.484.385.880.282.1 LS-Merge (Soro et al., 2026)52.352.161.860.658.858.5 57.757.179.179.083.583.381.381.1 Avg. Merging + FT60.559.755.855.957.257.357.857.681.180.894.093.987.587.4 ES-Merging (Ours)66.065.369.168.462.061.965.765.280.780.294.194.087.487.1 Table 1: Performance comparison of ES-Merging with the base LLM with and without task-specific fine-tuning, specialized MLLMs, and merging methods on the instance-varying interaction prediction tasks. We report accuracy and macro-F1 across each subset. Bold andunderlineindicate the best and second best performances, respectively. each modality: Mol-LLaMA (Kim et al., 2025) for molecule modality, Prot2Text-V2 (Fei et al., 2025) for protein modality, and Cell-o1 (Fang et al., 2025b) for single cell modality, whose base LLMs are LLaMA-3.1-8B-Instruct (Grattafiori et al., 2024). The merging methods are applied across all modality-specific models, resulting in a unified MLLM. Since each downstream task re- quires distinct domain knowledge, we provide task- specific few-shot in-context examples to induce the appropriate task understanding. For a fair com- parison, we use the same instruction template and in-context examples for all compared methods. For more details for the implementation details, please refer to Appendix A.3 and A.4. BaselinesWe compare ES-Merging with the base LLM, MLLMs specialized to a single modality, and merging methods. As modality-specialized baselines, we consider Mol-LLaMA, Prot2Text- V2, and Cell-o1, each specialized for molecule, protein, and single-cell modalities, respectively. We also compare ES-Merging with representa- tive merging methods, including Average Merg- ing, TIES-Merging (Yadav et al., 2023), EMR- Merging (Huang et al., 2024), layer-wise AdaMerg- ing (Yang et al., 2023), PCB-Merging (Du et al., 2024), Consensus Merging (Wang et al., 2024b), LS-Merge (Soro et al., 2026), and task-specific fine- tuned model after the average merging, denoted as Avg. Merging + FT. For the base LLMs and the specialized MLLMs, unsupported modalities are represented as textual inputs. Baseline details are provided in Appendix A.2. 5.2 Instance-varying Interaction Prediction We first evaluate on instance-varying cross-modal interaction tasks, where the target counterpart changes across instances and the model should gen- eralize across diverse cross-modal pairs. Datasets We consider two instance-varying cross-modal interaction settings: molecule-protein interaction and molecule-cell interaction.For molecule-protein interaction, the task is to predict whether a given molecule interacts with a given protein, including BindingDB, BioSNAP, and Hu- man (Koh et al., 2023). For molecule-cell interac- tion, the task is to predict the effect of a molecule on a given cell, including DrugComb (Zagidullin et al., 2019) and GDSC2 (Chawla et al., 2022). In both settings, the molecule, protein, and cell counterpart changes for each instance, requiring the model to generalize across diverse cross-modal combinations rather than relying on a fixed target identity. We provide the detailed explanation of datasets in Appendix A.1. Results As shown in Table 1, ES-Merging con- sistently outperforms the merging baselines, sug- gesting that ES-Merging more effectively inte- grates complementary cross-modal knowledge from modality-specialized MLLMs and thus lead- ing to stronger generalization when the interaction counterpart varies across instances. Notably, ES- 6 CYP InhibitionCYP Substrate CYP1A2CYP2C19CYP2C9CYP2D6CYP3A4Avg.CYP2C9CYP2D6CYP3A4Avg. Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1Acc.F1 Base LLM and Specialized MLLMs LLaMA-3.1-8B-Instruct55.358.050.655.447.453.756.451.743.152.850.654.347.850.730.133.148.550.642.144.8 Mol-LLaMA68.567.165.764.363.363.364.759.264.364.065.363.662.749.761.757.455.254.459.953.8 Prot2Text-V259.455.856.852.567.460.176.355.959.650.363.954.954.548.653.452.056.054.254.651.6 Cell-o153.558.344.951.943.152.846.946.343.651.6 46.452.236.641.736.139.948.555.040.445.6 Merging Methods Avg. Merging61.260.554.153.552.753.853.849.654.655.155.254.540.339.642.441.349.345.844.042.2 TIES-Merging71.871.867.867.966.565.076.164.065.366.369.567.056.046.855.653.254.553.755.451.2 EMR-Merging70.470.564.865.466.866.374.261.866.166.068.566.060.550.655.652.951.549.455.951.0 AdaMerging52.555.353.154.244.846.045.645.750.454.349.351.133.633.643.645.749.347.442.242.2 PCB-Merging69.269.463.664.665.063.872.461.063.865.266.864.855.244.957.153.956.755.756.451.5 Consensus Merging68.568.863.664.165.764.073.560.863.864.767.064.554.546.555.652.956.757.755.652.4 LS-Merge59.158.056.456.056.254.561.051.355.553.5 57.654.751.345.744.141.948.746.348.044.6 Avg. Merging + FT68.367.767.066.562.262.167.660.967.667.666.565.065.750.659.453.055.254.660.152.7 ES-Merging (Ours)77.477.470.670.572.570.480.769.571.370.874.571.764.253.660.957.260.559.661.956.8 Table 2: Performance comparison of ES-Merging on the target-fixed functionality prediction tasks. We report accuracy and macro-F1 across each subset. Bold indicates the best and underlineindicates the second best. Merging outperforms the task-specific finetuned model (Avg. Merging + FT) on the molecule- protein interaction tasks and shows comparable performance on the molecule-cell interaction tasks, indicating that ES-merging can enhance the cross- modal reasoning without further downstream fine- tuning. This superior performance of ES-Merging comes from preserving the reasoning capabilities of specialized MLLMs as shown in Table 5 and 6 in Appendix. In contrast, task-specific fine-tuning tends to diminish these reasoning capabilities and can even degrade performance, particularly on the molecule-protein interaction tasks. On the other hand, existing merging baselines such as EMR- Merging often exhibit substantial instability across datasets, showing a degraded performance on Drug- Comb. This implies that simple parameter space heuristics are insufficient for reliably combining heterogeneous multimodal experts, supporting that ES-Merging provides a more robust integration of modality-specific knowledge. 5.3 Target-fixed Functionality Prediction We further evaluate on target-fixed cross-modal functionality prediction tasks, where each subtask is associated with a fixed target and requires target- specific biological functionality knowledge beyond simple interaction matching in Section 5.2. Datasets To this end, we evaluate on CYP en- zyme prediction. Unlike the instance-varying set- ting above, each subtask is defined with respect to a fixed enzyme target and the tasks are pre- dicting the biological functionality of the given molecule to the fixed target: inhibitory effects or substrate specificity.Specifically, we con- sider five CYP inhibition subtasks to CYP1A2, CYP2C19, CYP2C9, CYP2D6, and CYP3A4 enzymes (Weiser et al., 2023), and three CYP substrate subtasks for CYP2C9, CYP2D6, and CYP3A4 enzymes (Holmer et al., 2021), from the TDC dataset (Huang et al., 2021). These benchmarks therefore evaluate whether the merged model can capture not only detailed molecular structural variation under a shared target, but also biologically meaningful interaction types. Results As shown in Table 2, ES-Merging achieves the best average performance, suggest- ing that it more effectively preserves and integrates expert knowledge required for target-specific func- tionality prediction. In contrast, on the CYP sub- strate tasks, most merging baselines underperform Mol-LLaMA, likely because these tasks rely heav- ily on molecular structural understanding under a fixed-target setting. Notably, ES-Merging achieves performance that is comparable to or better than Mol-LLaMA, suggesting that embedding-signal- based merging can not only integrate modality- specific expertise more effectively but also enhance cross-modal understanding. 5.4 Ablation Study We further conduct an ablation study to see the effectiveness of each coefficient type. As shown in Table 3, using only one of coefficient types still outperforms other merging baselines, indicating that the embedding space signals provide salient and robust information for cross-modal merging, compared to the merging methods based on the 7 024681012141618202224262830 Molecule Protein Cell 0.320.560.810.980.030.010.870.950.900.440.880.070.420.170.570.020.040.160.080.130.610.610.010.930.690.910.000.040.020.720.000.00 0.240.080.190.010.160.960.040.020.010.470.080.660.260.450.220.980.010.330.240.710.130.030.720.020.190.040.220.090.950.010.410.70 0.440.360.000.010.810.030.080.030.090.080.040.280.320.380.220.000.950.510.680.160.260.350.270.040.120.050.780.860.030.270.590.30 0.0 0.5 1.0 0.0 0.5 1.0 0.0 0.5 1.0 Figure 5: Computed layer-wise merging coefficient visualization of each specialized MLLM derived by Eq. 2. The l-th column corresponds to the merging coefficient of the l-th layer, α l m j . Instance-varyingTarget-fixed Coefficient TypeMol-Prot Mol-Cell CYP Inhib. CYP Subs. Layer-wise63.685.273.957.1 Element-wise64.986.772.760.5 Layer×Element65.787.474.561.9 Table 3: Ablation studies on merging coefficient types. We report average accuracy on each task group. MethodTotal FLOPs↓ Avg. Merging + FT891,117 AdaMerging493,694 ES-Merging (Ours)149,807 Table 4: Total Floating Point Operations (FLOPs) as a measure of computational cost for determining the merging coefficients and LoRA module parameters parameter space signals. On the other hand, com- bining two different coefficients shows the best performance, enabling to capture different granu- larities of MLLMs’ specialization. 5.5 Merging Coefficient Analysis We visualize the layer-wise merging coefficients in Fig. 5 and the element-wise merging coefficients in Fig. 9 of Appendix. The coefficients are distinctly distributed across different MLLMs, showing that modality-specific knowledge is not incorporated uniformly throughout the model. Additionally, we observe that even when the layer-wise coefficient is high (e.g., the third layer of the molecule LLM), the element-wise merging coefficients within that layer vary substantially, as shown in Fig. 6. This re- sult indicates that modality specialization emerges at multiple levels of granularity: only a few param- eter elements within important layers are primarily salient for modality specialization. Therefore, com- bining layer-wise global coefficients with element- wise local coefficients leads to the complementary merging of different granularities, achieving an ac- curate and robust merging of different modalities. 5.6 Computational Cost Comparison We compare the computation cost with the base- lines requiring the computation of gradients and parameter updates. As shown in Table 4, the com- 0102420483072 LoRA A Q proj 0102420483072 K proj 0102420483072 V proj 0102420483072 O proj 0246 LoRA B 024602460246 0.0 0.2 0.4 0.6 0.8 1.0 Figure 6: Computed element-wise merging coefficient visualization of each module in the third layer of molecule LLM derived by Eq. 3. putational cost of ES-Merging is 3.4×and 6.1× lower than AdaMerging and Task-specific Fine- tuning respectively, since ES-Merging requires the gradient computation only one time, while other baselines iteratively compute gradients and update parameters. While ES-Merging requires less com- putational cost, it outperforms the Task-specific Finetuning and AdaMerging, indicating the effi- ciency and effectiveness of ES-Merging. 6 Conclusion We present ES-Merging, a novel MLLM merging framework that shifts model merging from rely- ing on parameter space signals to leveraging em- bedding space signals. Our key insight is that input-aware representations encode rich modality- specific specialization, providing a more faithful basis for integrating modality-specialized MLLMs. Based on this observation, we introduce two com- plementary types of merging coefficients at dif- ferent granularities: layer-wise global coefficients for capturing coarse-grained specialization and element-wise local coefficients for capturing fine- grained importance. Experiments on interactive ef- fect prediction tasks demonstrate that ES-Merging consistently improves multimodal merging perfor- mance across diverse biological tasks, highlighting embedding space signals as a principled foundation for MLLM merging. 8 References Hadi Abdine,Michail Chatzianastasis,Costas Bouyioukos, and Michalis Vazirgiannis. 2024. Prot2text: Multimodal protein’s function generation with gnns and transformers. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 10757–10765. Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. 2015. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51(1):22–45. He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. 2023. Instructmol: Multi-modal integration for build- ing a versatile and reliable molecular assistant in drug discovery. arXiv:2311.16208. Saurabh Chawla, Anja Rockstroh, Melanie Lehman, and 1 others. 2022. Gene expression based inference of cancer drug sensitivity. Nature Communications, 13(1):5680. Chi-Jane Chen, Yuhang Chen, Sukwon Yun, Natalie Stanley, and Tianlong Chen. 2025. Spatial coordi- nates as a cell language: A multi-sentence framework for imaging mass cytometry analysis. In Findings of the Association for Computational Linguistics: ACL 2025, pages 13241–13252. Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. 2020.Chemberta: large-scale self- supervised pretraining for molecular property pre- diction. arXiv:2010.09885. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Sajib Acharjee Dip, Adrika Zafor, Bikash Kumar Paul, Uddip Acharjee Shuvo, Muhit Islam Emon, Xuan Wang, and Liqing Zhang. 2025. Llm4cell: A survey of large language and agentic models for single-cell biology. arXiv preprint arXiv:2510.07793. Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim K Goh, Ho-Kin Tang, Daojing He, and 1 others. 2024. Parameter competition balancing for model merging. Advances in Neural Information Processing Systems, 37:84746– 84776. Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. 2022. Translation be- tween molecules and natural language. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 375–413. Yin Fang, Xinle Deng, Kangwei Liu, Ningyu Zhang, Jingyang Qian, Penghui Yang, Xiaohui Fan, and Huajun Chen. 2025a. A multi-modal ai copilot for single-cell analysis with instruction following. arXiv preprint arXiv:2501.08187. Yin Fang, Qiao Jin, Guangzhi Xiong, Bowen Jin, Xi- anrui Zhong, Siru Ouyang, Aidong Zhang, Jiawei Han, and Zhiyong Lu. 2025b. Cell-o1: Training llms to solve single-cell reasoning puzzles with reinforce- ment learning. arXiv preprint arXiv:2506.02911. Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Hua- jun Chen. 2023. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint arXiv:2306.08018. Xiao Fei, Michail Chatzianastasis, Sarah Almeida Carneiro, Hadi Abdine, Lawrence P Petalidis, and Michalis Vazirgiannis. 2025. Prot2text-v2: Pro- tein function prediction with multimodal contrastive alignment. Advances in Neural Information Process- ing Systems. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Han Guo, Mingjia Huo, Ruiyi Zhang, and Pengtao Xie. 2023. Proteinchat: Towards achieving chatgpt-like functionalities on protein 3d structures. Authorea Preprints. Shuhan Guo, Yatao Bian, Ruibing Wang, Nan Yin, Zhen Wang, and Quanming Yao. 2024. Unimot: Unified molecule-text language model with discrete token representation. arXiv preprint arXiv:2408.00863. Malte Holmer, Christina de Bruyn Kops, Conrad Stork, and Johannes Kirchmair. 2021. Cypstrate: A set of machine learning models for the accurate classifi- cation of cytochrome p450 enzyme substrates and non-substrates. Molecules, 26(15):4678. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xi- angyu Yue, and Wanli Ouyang. 2024. Emr-merging: Tuning-free high-performance model merging. Ad- vances in Neural Information Processing Systems, 37:122741–122769. Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. 2021. Ther- apeutics data commons: Machine learning datasets and tasks for drug discovery and development. arXiv preprint arXiv:2102.09548. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Suchin Gururangan, Ludwig Schmidt, Han- naneh Hajishirzi, and Ali Farhadi. 2022.Edit- ing models with task arithmetic. arXiv preprint arXiv:2212.04089. 9 Oussama Kharouiche, Aris Markogiannakis, Xiao Fei, Michail Chatzianastasis, and Michalis Vazirgiannis. 2025. Cell2text: Multimodal llm for generating single-cell descriptions from rna-seq data. arXiv preprint arXiv:2509.24840. Dongki Kim, Wonbin Lee, and Sung Ju Hwang. 2025. Mol-llama: Towards general understanding of molecules in large molecular language model. Ad- vances in Neural Information Processing Systems. Huan Yee Koh, Anh T.N. Nguyen, Shirui Pan, Lau- ren T. May, and Geoffrey I. Webb. 2023. Psichic: physicochemical graph neural network for learning protein-ligand interaction fingerprints from sequence data. bioRxiv. Mario Krenn, Florian Häse, AkshatKumar Nigam, Pas- cal Friederich, and Alan Aspuru-Guzik. 2020. Self- referencing embedded strings (selfies): A 100% ro- bust molecular string representation. Machine Learn- ing: Science and Technology, 1(4):045024. Longyi Li, Liyan Dong, Hao Zhang, Dong Xu, and Yongli Li. 2025.spallm: enhancing spatial do- main analysis in multi-omics data through large lan- guage model integration. Briefings in Bioinformatics, 26(4):bbaf304. Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, and Qi Tian. 2024. Towards 3d molecule-text in- terpretation in language models.arXiv preprint arXiv:2401.13923. Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, and 1 others. 2022. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022:500902. Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. 2023. MolCA: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dan- gyang Chen, and Yu Cheng. 2024. Twin-merging: Dynamic integration of modular expertise in model merging. Advances in Neural Information Process- ing Systems, 37:78905–78935. OpenAI.2024a.Gpt-4technicalreport. arXiv:2303.08774. OpenAI.2024b.Gpt-4osystemcard. arXiv:2410.21276. Jinyoung Park, Minseong Bae, Dohwan Ko, and Hyun- woo J Kim. 2024. Llamo: Large language model- based molecular graph assistant. Advances in Neural Information Processing Systems, 37:131972–132000. Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. 2023. Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language asso- ciations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1102–1123. Syed Asad Rizvi, Daniel Levine, Aakash Patel, Shiyang Zhang, Eric Wang, Curtis Jamison Perry, Nicole May- erli Constante, Sizhuang He, David Zhang, Cerise Tang, and 1 others. 2025.Scaling large lan- guage models for next-generation single-cell analysis. BioRxiv, pages 2025–04. David Rogers and Mathew Hahn. 2010. Extended- connectivity fingerprints. Journal of chemical in- formation and modeling, 50(5):742–754. Bedionita Soro, Aoxuan Silvia Zhang, Bruno Andreis, Jaehyeong Jo, Song Chong, and Sung Ju Hwang. 2026. Ls-merge: Merging language models in latent space. In The Fourteenth International Conference on Learning Representations. Hugo Touvron and 1 others. 2023.Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Chao Wang, Hehe Fan, Ruijie Quan, and Yi Yang. 2024a. Protchatgpt: Towards understanding pro- teins with large language models. arXiv preprint arXiv:2402.09649. Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz- Jimenez, François Fleuret, and Pascal Frossard. 2024b. Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813. Zhicong Wang, Zicheng Ma, Ziqiang Cao, Chang- long Zhou, Jun Zhang, and Yi Qin Gao. 2025a. Prot2chat: protein large language model with early fusion of text, sequence, and structure. Bioinformat- ics, 41(8):btaf396. Ziqing Wang, Kexin Zhang, Zihan Zhao, Yibo Wen, Abhishek Pandey, Han Liu, and Kaize Ding. 2025b. A survey of large language models for text-guided molecular discovery: from molecule generation to optimization. arXiv preprint arXiv:2505.16094. David Weininger. 1988. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical infor- mation and computer sciences, 28(1):31–36. Benjamin Weiser, Jérôme Genzling, Mihai Burai- Patrascu, Ophélie Rostaing, and Nicolas Moitessier. 2023. Machine learning-augmented docking. 1. cyp inhibition prediction. Digital Discovery, 2:1841– 1849. Peter Willett, John M Barnard, and Geoffrey M Downs. 1998.Chemical similarity searching.Journal of chemical information and computer sciences, 38(6):983–996. 10 Yijia Xiao, Edward Sun, Yiqiao Jin, Qifan Wang, and Wei Wang. 2024. Proteingpt: Multimodal llm for pro- tein property prediction and structure understanding. arXiv preprint arXiv:2408.11363. Yijia Xiao, Wanjia Zhao, Junkai Zhang, Yiqiao Jin, Han Zhang, Zhicheng Ren, Renliang Sun, Haixin Wang, Guancheng Wan, Pan Lu, and 1 others. 2025. Protein large language models: A comprehensive survey. arXiv preprint arXiv:2502.17504. Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. 2023. Protst: Multi-modality learning of pro- tein sequences and biomedical texts. In International conference on machine learning, pages 38749–38767. PMLR. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Re- solving interference when merging models. Ad- vances in neural information processing systems, 36:7093–7115. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guib- ing Guo, Xingwei Wang, and Dacheng Tao. 2023. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575. Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun. 2024. Llasmol: Advancing large language models for chemistry with a large-scale, comprehen- sive, high-quality instruction tuning dataset. arXiv preprint arXiv:2402.09391. Bulat Zagidullin, Jehad Aldahdooh, Shuyu Zheng, Wenyu Wang, Yinyin Wang, Joseph Saad, Alina Ma- lyutina, Mohieddin Jafari, Ziaurrehman Tanoli, Al- berto Pessia, and Jing Tang. 2019. Drugcomb: an integrative cancer drug combination data portal. Nu- cleic Acids Research, 47(W1):W43–W51. 11 OrganizationAppendix is organized as follows: In Section A, we provide detailed experimental settings, including datasets, baseline, implemen- tation, and prompt setting details. In Section B, we qualitatively analyze generated responses. In Section C, we provide additional analysis of our merging method. In Section D, we discuss the limitations of our work. A Experiment Settings A.1 Datasets Detail • BindingDB is a drug-target interaction dataset derived from experimentally measured small molecule-protein binding data, typically filtered to human proteins, and used as a binary clas- sification benchmark with 11,054 samples for predicting whether a drug-target pair interacts. •BioSNAP is a drug-target interaction dataset derived from known associations between US- marketed drugs and their human protein targets, and is used as a binary classification benchmark with 6,058 samples for predicting whether a drug- target pair interacts. •Human is a drug-target interaction dataset con- sisting of drug–human protein pairs with highly credible negative samples and used as a binary classification benchmark with 1,375 samples to predict whether a drug-target pair interacts. • DrugComb is a drug combination-cell line in- teraction dataset derived from standardized and harmonized drug combination screening studies across various cancer cell lines, and is used as a binary classification benchmark with 3,631 sam- ples for predicting whether a combination of two drugs produces a synergistic or antagonistic anti- cancer effect in a given cancer cell line. • GDSC2 is a drug-cell line interaction dataset de- rived from the Genomics of Drug Sensitivity in Cancer project, which screens over 1,000 geneti- cally characterized human cancer cell lines with a wide range of anti-cancer therapeutics using a newer cell screening platform introduced in 2015, and is used as a binary classification benchmark with 843 samples for predicting whether a given cancer cell line is sensitive or resistant to a spe- cific drug. •CYP1A2, CYP2C19, CYP2C9, CYP2D6, and CYP3A4 Inhibition are binary classification benchmarks for predicting whether a given drug inhibits a specific cytochrome P450 enzyme iso- form involved in drug metabolism, consisting of 2,516, 2,533, 2,418, 2,626, and 2,466 samples, respectively. •CYP2C9, CYP2D6, and CYP3A4 Substrate are three binary classification benchmarks to pre- dict whether a given drug is a substrate of a spe- cific cytochrome P450 enzyme isoform involved in drug metabolism with 134, 133, and 134 sam- ples, respectively. A.2 Baseline Detail We compare our method against three modality- specialized base models, one task-specific fine- tuned model, six existing model merging baselines, and three ablation variants of our proposed method. •Mol-LLaMA (Kim et al., 2025) is a large molec- ular language model trained via multi-modal in- struction tuning that integrates complementary 2D and 3D molecular encoders through a blend- ing module, with a Q-Former projector and a LLaMA backbone fine-tuned with LoRA, pro- viding general molecular understanding with ex- plainability and reasoning capabilities across di- verse molecular tasks. • Prot2Text-V2 (Fei et al., 2025) is a multi- modal sequence-to-text model that combines an ESM2-3B (Lin et al., 2022) protein sequence encoder with a LLaMA-3.1-8B-Instruct decoder via a nonlinear modality projector, using hybrid sequence-level contrastive alignment learning and instruction-based LoRA fine-tuning to gen- erate rich functional descriptions of proteins di- rectly from amino acid sequences. •Cell-o1 (Fang et al., 2025b) is a cell line- specialized model fine-tuned on transcriptomic omics data, capable of handling cell line- related drug response tasks but limited to single-modality inference without understanding molecule or protein. • Average Merging directly averages the parame- ter elements of all specialist MLLMs parameters without any additional computation. •Average Merging + Task-Specific Finetune constructs a base model by averaging the LoRA weights of all three specialist models, and sub- sequently fine-tunes the LoRA parameters inde- pendently on each evaluation task by sampling 2,000 examples from the training dataset for 10 epochs with a batch size of 16. 12 •TIES-Merging (Yadav et al., 2023) resolves interference among task vectors through three steps, trim, elect sign, and disjoint merge, which prune low-magnitude parameters, resolve sign conflicts by selecting the dominant direction, and merge only the parameter-aligned subset. • AdaMerging (Yang et al., 2023) learns merging coefficients for task vectors in a test-time adap- tation manner, using the minimization of model output entropy on unlabeled test samples as a surrogate objective without relying on original training data. In our experiments, we adopt the layer-wise AdaMerging variant which indepen- dently learns a merging coefficient for each layer of every task vector, showing better performance compared to the task-wise AdaMerging. •EMR-Merging (Huang et al., 2024) is a tuning- free method that first elects a unified task vector by selecting the maximum absolute value of each parameter along the dominant sign direction, and then generates lightweight task-specific masks and rescalers to align the direction and magnitude of the unified model with each original specialist model at inference time. •PCB-Merging (Du et al., 2024) is a training-free method that constructs a parameter competition balancing matrix through intra-balancing, which measures parameter significance within individ- ual tasks, and inter-balancing, which assesses parameter similarity across tasks, then drops low- scoring parameters and rescales the remaining ones to form the final merged model. • Consensus-Merging (Wang et al., 2024b) identi- fies and removes two classes of detrimental pa- rameters, selfish weights that are critical exclu- sively to a single task and catastrophic weights that are irrelevant to all tasks and retains only the consensus parameters that contribute positively to multi-task fusion. • LS-Merge (Soro et al., 2026) encodes model weights into a latent space via a transformer- based VAE, performs merging through linear in- terpolation in that space, and decodes back to parameters. • Layer-wise ES-Merging (Ours) applies only the layer-wise global coefficientα l m j , derived from SWD-based embedding distribution shifts between the base and specialized models, as the merging coefficient. •Element-wise ES-Merging (Ours) applies only the element-wise local coefficientβ l,n m j , derived from gradient-based parameter sensitivity scores with respect to fine-grained embedding distances, as the merging coefficient. •ES-Merging (Ours) integrates both the global layer coefficientα l m i and the local element co- efficientβ l,n m i into a unified merging coefficient λ l,n m i by multiplication and renormalization of el- ements in modalities. A.3 Implementation Details LoRA Configuration All specialized models used for merging share a unified LoRA (Hu et al., 2022) configuration with rankr = 8, and scaling factorα = 32, applied to all self-attention pro- jection matrices (W Q , W K , W V , W O ) and MLP projection layers (gate, up, and down projections) of every transformer block. Details of ES-MergingTo construct the probe in- puts, we randomly sample the 110 samples for each modality from a collection of test sets, construct- ing 330 samples in total. For the layer-wise merg- ing coefficient computation, we leverage SWD with slice projection dim=1,024 and distance order p=2.0, then normalized via softmax with tempera- tureτ=0.5. For the element-wise merging coeffi- cient computation, the temperatureτof the softmax is predefined as 0.5. Existing Merging Methods For all baseline methods, we follow the experimental settings re- ported in their respective original papers and offi- cial implementations. For LS-Merge (Soro et al., 2026), we train VAE with sequence length16,384, batch size512, learning rate3×10 −4 with500step warmup and cosine decay, and KL weight10 −4 with adaptive adjustment toward a target KL of50, for1,000epochs. At merge time, model parameters are split into chunks of size16,384, encoded into the latent space via the VAE encoder, merged via uniform-weighted Euclidean mean of the posterior means μ, and decoded back to parameter space. A.4 Prompt Setting for Evaluation To impart the task-specific knowledge, we pro- vide 5-shot examples from the training set with the prompt templates in Table 9 and Table 10. The retrieval strategy is designed per task as follows: 13 BindingDBBioSNAPHuman 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Density MoleculeProtein Interaction GDSC2DrugComb 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Density MoleculeCell Interaction CYP1A2 Inhib. CYP2C19 Inhib. CYP2D6 Inhib. CYP2C9 Inhib. CYP3A4 Inhib. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Density CYP Inhibition CYP2C9 Subs. CYP2D6 Subs. CYP3A4 Subs. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Density CYP Substrate Label Distribution across Datasets Figure 7: Label distribution across datasets. Each panel shows positive and negative sample counts for Molecule- Protein Interaction, CYP Inhibition, CYP Substrate, and Molecule-Cell Interaction tasks. •Molecule-Protein Interaction: Training sam- ples whose target protein sequence exactly matches the query are first collected. If more than five exact matches exist, the top-5 are selected by Tanimoto similarity (Willett et al., 1998) of Morgan fingerprints (Rogers and Hahn, 2010) between their molecules and the query molecule. If fewer than five are found, the remaining slots are filled with samples from proteins with the highest cosine similarity of protein embeddings from ESM2 to the query protein. Among can- didates from the same similar protein, the one with the highest Tanimoto similarity to the query molecule is preferred. •Drug-Cell Interaction: Training samples whose cell line shares an exact match on the top-50 ex- pressed gene set are priorly collected. If fewer than five are found, additional samples are drawn from cell lines with the highest Jaccard similarity on gene sets. When multiple candidates exist from the same cell line, the one with the high- est Tanimoto similarity to the query molecule is selected. For DrugComb, molecule similarity is computed as the average Tanimoto similarity across both drugs. • CYP Inhibition/Substrate: Since all samples share the same CYP enzyme target, protein-level filtering is unnecessary. The top-5 examples are selected solely by Tanimoto similarity between the training molecules and the query molecule. A.5 Evaluation Metric As shown in Figure 7, while some datasets exhibit a relatively balanced distribution between positive and negative samples, others such as BindingDB and CYP2D6 Inhibition display a pronounced class imbalance. In particular, the CYP Substrate task contains only approximately 130 samples per dataset, posing a critical challenge of absolute data scarcity. In such imbalanced and low resource settings, relying solely on accuracy as an evalu- ation metric can be misleading, as a model that predominantly predicts the majority class may still achieve inflated scores without genuinely captur- ing minority-class patterns. Therefore, we leverage the macro-F1 that addresses this limitation by com- puting the F1 score for each class independently and averaging them with equal weight regardless of class frequency, thereby providing a fair assess- ment of predictive performance across all classes. B Qualitative Analysis In this section, we qualitatively compare the gen- erated responses of each task-specific fine-tuned model with those of ES-Merging, as shown in Ta- ble 5 and Table 6. B.1 Molecule-Protein Interaction Prediction Table 5 presents a generated response on the molecule-protein interaction prediction task from the Human dataset, where the molecule is thymine and the target protein is thymidine phosphory- lase. Although both ES-Merging and the task- specific finetuning model correctly predict the la- bel, the two responses differ substantially in qual- ity. ES-Merging correctly identifies thymine as a pyrimidine base found in DNA and a heterocyclic aromatic compound, then progressively reasons through its biological context, such as base pair- ing with adenine and its role in DNA synthesis, to conclude that thymine is likely to interact with the protein as a substrate for enzymes involved in thymidine metabolism. This can be interpreted as evidence that the chemical structure-level knowl- edge acquired from the molecule expert model and the biological function-level knowledge acquired from the protein expert model are organically inte- grated within a single model through embedding 14 Molecule Name: Thymine SMILES: C1=CNC(=O)NC1=O Protein: Thymidine phosphorylase Protein Sequence: MAALMTPGTGAPPAPGDFSGEGSQGLPDPSPEPKQLPELIRMKRDGGRLSEADIRGFVAAVVN GSAQGAQIGAMLMAIRLRGMDLEETSVLTQALAQSGQQLEWPEAWRQQLVDKHSTGGVGDKVSLVLAPALAACGC KVPMISGRGLGHTGGTLDKLESIPGFNVIQSPEQMQVLLDQAGCCIVGQSEQLVPADGILYAARDVTATVDSLPLITAS ILSKKLVEGLSALVVDVKFGGAAVFPNQEQARELAKTLVGVGASLGLRVAAALTAMDKPLGRCVGHALEVEEALLC MDGAGPPDLRDLVTTLGGALLWLSGHAGTQAQGAARVAAALDDGSALGRFERMLAAQGVDPGLARALCSGSPAER RQLLPRAREQEELLAPADGTVELVRALPLALVLHELGAGRSRAGEPLRLGVGAELLVDVGQRLRRGTPWLRVHRDGP ALSGPQSRALQEALVLSDRAPFAAPSPFAELVLPPQQ User: Determine whether the given molecule interacts with the protein by following the example. Ground Truth: Interacts ES-Merging (Ours) The given molecule is thymine, a pyrimidine base found in DNA and RNA. It is a heterocyclic aromatic organic compound. The protein is not specified, but based on the examples, it is likely a DNA or RNA polymerase enzyme. Thymine is a nucleobase that pairs with adenine in DNA. It is also a substrate for thymidylate synthase, an enzyme that catalyzes the conversion of uracil to thymine in DNA synthesis. Given the information, it is likely that the molecule interacts with the protein, as thymine is a substrate for thymidylate synthase, an enzyme that catalyzes a critical step in DNA synthesis. Final answer: Interacts. Models with Avg. + Task-Specific Finetuning Final answer: Interacts Table 5: Case study on Molecule-Protein Interaction prediction. space signal-based merging. In contrast, the task- specific finetuned model outputs only the label “In- teracts” without any biological rationale. Because it was trained on instruction data containing labels only, its predictions are less interpretable, whereas ES-Merging integrates modality-expert knowledge to produce interpretable reasoning even for unseen cross-modal tasks. B.2 Molecule-Cell Interaction Prediction Table 6 presents a qualitative analysis of the Drug- Cell Interaction prediction task from the GDSC2 dataset, where the drug is Gefitinib, and the cell line is OVCA420. ES-Merging first identifies the given molecule as a quinazoline derivative and correctly recognizes it as a potent inhibitor of the epidermal growth factor receptor (EGFR) tyrosine kinase. It also ac- curately identifies the given cell line as a cancer cell, then focuses on genes such as RPS6, EIF1, and GNB2L1 in the ranked gene-expression list to interpret the cell as actively proliferating and potentially dependent on EGFR signaling. On the basis of this, ES-Merging correctly recognizes that the drug’s mechanism of action is effective against the given cell. This reasoning chain spanning from molecular substructure to drug class, target path- way, cell type identification, and cell-level interpre- tation demonstrates that the structural knowledge from the molecule expert model and the transcrip- tomic knowledge from the cell expert model are functionally integrated through ES-Merging, while the model with task-specific finetuning again out- puts only the label “Sensitive” without any biologi- cal rationale. C Further Analysis In this section, we provide a detailed analysis of the merging coefficients used in ES-Merging, extended results of the ablation study in the main paper, and 15 Drug Name: Gefitinib SMILES: COC1=C(C=C2C(=C1)N=CN=C2NC3=C(=C(C=C3)F)Cl)OCCCN4CCOCC4 Cell Line: OVCA420 Top 50 Genes: RPL35, FTL, MT-ND4, GAPDH, AKR1B10, UBC, RPS6, RPL24, EIF1, GNB2L1, SPP1, RPS11, AKR1C1, RPL19, GNG5, AKR1B10P1, ACTG1, MT-CO3, RPL5, FAU, RPS15A, TXN, RPL8, MIF, LDHB, KRT19, EIF4H, RPL11, ALDH1A1, MGST1, RPS12, CLIC1, SNRPD2, RPL30, H2AFX, NQO1, PRDX1, RPS5, RPLP1, H2AFZ, TUBB4B, FAUP1, NACA, RPL14, HIST1H2BK, MYL12B, SHFM1P1, HINT1P1, RPL27, HIST1H4C User: Determine whether the given cell is sensitive or resistant to the given molecule based on the ranked gene-expression list. Ground Truth: Sensitive ES-Merging (Ours) The given molecule is 4-(3-fluoro-4-methoxyphenyl)-7-methyl-6-oxo-1,4-dihydroquinazoline-3-carboximidamide, which is a quinazoline derivative. This type of molecule is known to be a potent inhibitor of the epidermal growth factor receptor (EGFR) tyrosine kinase. The cell line provided is a cancer cell, and the presence of genes such as RPS6, EIF1, and GNB2L1 in the list suggests that the cell is actively proliferating and may be dependent on EGFR signaling for growth and survival. Given the presence of these genes and the fact that the molecule is an EGFR inhibitor, it is likely that the cell is sensitive to the molecule. Final answer: Sensitive Models with Avg. + Task-Specific Finetuning Final answer: Sensitive Table 6: Case study on Molecule-Cell Interaction prediction. an analysis of embedding visualizations. C.1 Merging Coefficients C.1.1 Layer-wise Merging Coefficients SizeHuman GDSC2 CYP2C9 Inh. CYP2C9 Sub. 3260.790.668.253.7 25662.093.168.948.5 102462.094.172.564.2 Table 7: Ablation studies on projection size of SWD for layer-wise ES-Merging across Molecule-Protein and Molecule-Cell Interaction tasks. Projection Dimension Size of SWDWhen com- puting SWD for the layer-wise merging coeffi- cients, we vary projection dimension sizes on repre- sentative datasets for each task, and derived merg- ing coefficients accordingly. As shown in Table 7, performance consistently improves as the projec- tion size increases, with 1024 dimensions achiev- ing the best results. This is because a larger num- ber of projections enables a more precise approx- imation of the distributional differences in high- dimensional space, leading to more accurate com- putation of the layer-wise merging coefficients. Based on this finding, we set the SWD projection size to 1024 in our main experiments. C.1.2 Element-wise Merging Coefficients The coefficient patterns differ across q/k/v/o_proj modules even within the same layer. As shown in Figure 9, the q/k/v projection modules and the o projection module emphasize different elements even at Layer 0. Within the same module, the co- efficient distributions between LoRA A and LoRA B exhibit distinct patterns. In particular, at Layer 0 of the q projection module, LoRA A shows rel- atively balanced coefficients across all modalities, whereas LoRA B contains regions where molecule and protein are more prominent. On the other hand, 16 Molecule-Protein InteractionMolecule-Cell InteractionCYP InhibitionCYP Substrate Coefficient TypeBindingDB BioSNAP HumanAvg. DrugComb GDSC2Avg. CYP1A2 CYP2C19 CYP2C9 CYP2D6 CYP3A4Avg. CYP2C9 CYP2D6 CYP3A4Avg. Layer-wise65.568.257.063.680.190.285.276.371.472.379.270.273.961.254.955.257.1 Element-wise66.566.661.464.979.394.186.776.069.170.777.669.872.765.757.658.260.5 Layer× Element Mixed66.069.162.065.780.794.187.477.470.672.580.771.374.564.260.960.561.9 Table 8: Full results of the ablation studies on merging coefficient. We report accuracy across all datasets and their average per task group. Bold indicates the best and underlineindicates the second best. Molecule Token Embedding Space ES-Merging Cell LLM Protein LLM Molecule LLM (a) Molecule token Protein Token Embedding Space ES-Merging Cell LLM Protein LLM Molecule LLM (b) Protein token Cell Token Embedding Space ES-Merging Cell LLM Protein LLM Molecule LLM (c) Cell token Figure 8: Embedding visualization of the last transformer block for each specialized LLM and the merged model with our method for (a) molecule, (b) protein, and (c) cell tokens. coefficient distributions are far from uniform, in- dicating that layer-wise global coefficients alone cannot capture the fine-grained specialization dif- ferences and supporting the necessity of element- wise local coefficients. C.1.3 Final Coefficients of ES-Merging The layer-wise global coefficientα l m i and the element-wise local coefficientβ l,n m i focus on dif- ferent regions. The layer-wise coefficient captures coarse-grained distributional shifts in the embed- ding space, while the element-wise coefficient re- flects fine-grained importance at the individual pa- rameter level. As formulated in Eq. 4, by multiply- ing these two coefficients and normalizing, regions where both coefficients assign high importance are amplified, whereas regions where only one side is high and the other is low are suppressed. This enables merging that simultaneously incorporates global layer-level specialization signals and local element-level specialization signals. As shown in Figure 10, compared to the element-wise-only co- efficient distributions, the combined results demon- strate that the overall scale of coefficients is ad- justed according to the global importance of each layer, while the fine-grained element-level patterns are preserved. C.2 Details of Ablation Study Table 8 reports the detailed per-dataset results cor- responding to the averages of each task group in Table 3.The layer-wise ES-Merging and element-wise ES-Merging each dominate on differ- ent datasets. Despite this dataset-level variations, Layer×Element ES-Merging consistently achieves the best or comparable performance across all in- dividual datasets. These results confirm that the two coefficients capture specialization at different granularities depending on task and dataset char- acteristics, and their combination compensates for the weaknesses of either side, yielding consistent and robust merging performance at the individual dataset level as well. C.3 Embedding Visualization Figure 8 visualizes the embedding distributions of each specialist model and the model merged by ES-Merging at the last transformer block for each modality token. In each modality token space, the specialist models form distinct distributions, indicating that each model has learned modality- specific representations. The model merged by ES-Merging is positioned between the specialist distributions without being biased toward any par- ticular specialist, while being relatively close to the distribution of the specialist corresponding to each modality token. This suggests that ES-Merging integrates the modality-specific knowledge of each specialist model in a balanced manner, while pre- serving the specialization for each modality. D Limitation In this work, we present ES-Merging, a novel MLLM merging framework that derives layer-wise and element-wise merging coefficients by leverag- ing embedding space signals. Although we have 17 demonstrated its effectiveness on the integration of MLLMs specialized in biochemical domains such as molecules, proteins, and single cells, we have not explored its applicability to more general multi-modal domains such as video, image, and audio due to the lack of cross-modal benchmarks on these domains. Since the core principle of our approach to leverage embedding space signals is inherently modality-agnostic, we believe that ex- tending ES-Merging to such general multi-modal scenarios is a promising direction and leave this exploration as future work. Additionally, further in- vestigation is needed to verify whether ES-Merging can maximally preserve performance not only on cross-modal fusion tasks but also on individual single-modality tasks of each specialist model. 18 Table 9: In-context learning prompt templates for the Molecule-Protein Interaction and CYP prediction task. Angle-bracketed tokens are replaced with the corresponding encoder embeddings: <protein> and <mol> for the target pair, and<protein k>and<molecule k>for each of thekfew-shot examples. The<label>token is replaced with the ground-truth prediction string for each example pair in the few-shot context. Prompt for Molecule-Protein Interaction Prediction System You are an expert specialized in drug discovery and molecular biology. You will be given a protein and a molecule. Your task is to determine whether a given molecule interacts with a specific protein. User Determine whether the given molecule interacts with the protein by following the examples. Examples: Example 1:Protein: <protein 1>Molecule: <molecule 1>Final answer: Label . . . Examplek:Protein: <protein k>Molecule: <molecule k>Final answer: Label Protein: <protein>Molecule: <mol> Your final answer must be exactly one of: ‘Final answer: Interacts’ or ‘Final answer: Does not interact’. Prompt for CYP Inhibition Prediction System You are an expert specialized in drug discovery and molecular biology. You will be given a protein and a molecule. Your task is to determine whether a given molecule inhibits a specific protein. User Determine whether the given molecule inhibits the protein by following the examples. Examples: Example 1:Protein: <protein 1>Molecule: <molecule 1>Final answer: Label . . . Examplek:Protein: <protein k>Molecule: <molecule k>Final answer: Label Protein: <protein>Molecule: <mol> Your final answer must be exactly one of: ‘Final answer: Inhibit’ or ‘Final answer: Does not inhibit’. Prompt for CYP Substrate Prediction System You are an expert specialized in drug discovery and molecular biology. You will be given a protein and a molecule. Your task is to determine whether a given molecule is a substrate of a specific protein. User Determine whether the given molecule is a substrate of the protein by following the examples. Examples: Example 1:Protein: <protein 1>Molecule: <molecule 1>Final answer: Label . . . Examplek:Protein: <protein k>Molecule: <molecule k>Final answer: Label Protein: <protein>Molecule: <mol> Your final answer must be exactly one of: ‘Final answer: Substrate’ or ‘Final answer: Not a substrate’. 19 Table 10: In-context learning prompt template for the Molecule-Cell Interaction prediction task. Angle-bracketed tokens are replaced as follows: target molecule token <mol> and <molecule k> for each of the k few-shot examples are substituted with the corresponding encoder embedding, while the gene tokens <gene> are replaced with the corresponding cell line string. The <label> token is replaced with the ground-truth prediction string for each example pair in the few-shot context. Prompt for Molecule-Cell Interaction Prediction: GDSC2 System You are an expert specialized in cancer pharmacogenomics and drug response prediction. You will be given an anticancer molecule and a cancer cell represented as a list of gene names ordered by expression, where the most highly expressed genes appear first in descending order. Your task is to predict whether the molecule will biologically suppress the cancer cell. If the molecule is expected to suppress/inhibit cancer cell growth or viability, label it Sensitive, otherwise label it Resistant. User Determine whether the given cell is sensitive or resistant to the given molecule based on the ranked gene-expression list. Examples: Example 1:Cell: <gene 1>Molecule: <molecule 1>Final answer: Label . . . Examplek:Cell: <gene k>Molecule: <molecule k>Final answer: Label Cell: <gene>Molecule: <mol> Your final answer should be formatted as either: ‘Final answer: Sensitive’ or ‘Final answer: Resistant’. Prompt for Molecule-Cell Interaction Prediction: DrugComb System You are an expert specialized in cancer pharmacogenomics and anticancer drug-combination response prediction. You will be given two anticancer molecules and a cancer cell represented as a ranked list of gene names (highest expression first). Your task is to predict the binary interaction outcome in terms of anticancer effect in that cell line. Labels: Synergistic – the combination produces a stronger anticancer effect than expected, Antagonistic – the combination produces a weaker anticancer effect than expected. User Predict whether the drug pair is Synergistic or Antagonistic in the given cell line. Examples: Example 1:Cell:<gene 1>Molecule1:<molecule1 1>Molecule2:<molecule2 1>Final answer:Label . . . Examplek:Cell:<gene k>Molecule1:<molecule k>Molecule2:<molecule2 k>Final answer:Label Cell: <gene>Molecule1: <mol1>Molecule2: <mol2> Your final answer should be formatted as either: ‘Final answer: Synergistic’ or ‘Final answer: Antagonistic’. 20 self_attn.q_projLayer 0 0136527304095 LoRA-a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 10 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 20 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 30 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 self_attn.k_projLayer 0 0136527304095 LoRA-a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 10 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 20 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 30 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 self_attn.v_projLayer 0 0136527304095 LoRA-a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 10 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 20 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 30 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 self_attn.o_projLayer 0 0136527304095 LoRA-a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 10 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 20 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 30 0136527304095 LoRA-a 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 0.0 0.2 0.4 0.6 0.8 1.0 0246 LoRA-b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Element-wise Merging Coefficients Figure 9: Element-wise merging coefficients of the LoRA parameters forself_attn.q/k/v/o_projmodules at layers 0, 10, 20, and 30. Rows correspond to LoRA-A and LoRA-B, and columns correspond to Molecule, Protein, and Cell modalities. 21 Layer 0 0136527304095 q_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 q_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 10 0136527304095 q_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 q_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 20 0136527304095 q_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 q_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 30 0136527304095 q_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 q_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Element-wise × Layer-wise Coefficients (a) q projection Layer 0 0136527304095 k_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 k_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 10 0136527304095 k_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 k_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 20 0136527304095 k_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 k_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 30 0136527304095 k_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 k_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Element-wise × Layer-wise Coefficients (b) k projection Layer 0 0136527304095 v_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 v_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 10 0136527304095 v_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 v_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 20 0136527304095 v_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 v_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 30 0136527304095 v_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 v_proj lora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Element-wise × Layer-wise Coefficients (c) v projection Layer 0 0136527304095 o_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 o_projlora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 10 0136527304095 o_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 o_projlora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 20 0136527304095 o_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 o_projlora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Layer 30 0136527304095 o_proj lora_a Molecule 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Protein 0.0 0.2 0.4 0.6 0.8 1.0 0136527304095 Cell 0.0 0.2 0.4 0.6 0.8 1.0 0246 o_projlora_b 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 0246 0.0 0.2 0.4 0.6 0.8 1.0 Element-wise × Layer-wise Coefficients (d) o projection Figure 10: Element×Layer merging coefficients of the LoRA parameters forself_attn.q/k/v/o_projmodules at Layers 0, 10, 20, and 30, respectively. 22

Instant research discovery