Paper deep dive

Drug-Target Interaction Prediction via Hierarchical Sequential Cross-Attention over Chemical and Protein Language Models

Khadidja Henni, Hamza Abdelali, Abdelkrim Aries, Neila Mezghani, Brigitte Vannier, Sara Magdouli, Lina Abou-Abbas

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 39

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Predicting Drug-Target Interactions~(DTIs) is a central task in computational drug discovery, with direct applications in virtual screening, drug repurposing, and therapeutic candidate prioritization. Although recent deep learning methods have improved DTI prediction, many sequence-based models still process drugs and proteins independently and only combine their representations at a late prediction stage. This limits their ability to explicitly model cross-molecular dependencies between chemical substructures and protein sequence regions. In this paper, we propose a sequence-only DTI prediction architecture that combines two pre-trained language models, ChemBERTa for drug SMILES strings and ESM-2 for protein amino acid sequences, with a hierarchical interaction module. The proposed model first extracts contextual representations using pre-trained encoders, then applies 1D convolutional layers to condense local sequence patterns, followed by a sequential bidirectional cross-attention mechanism inspired by the induced-fit view of molecular recognition. Finally, attention-based pooling constructs fixed-size interaction-aware vectors for binary prediction. Experiments on BIOSNAP, Davis, and BindingDB show that the proposed model achieves the best performance on BIOSNAP, matches the best AUROC on Davis, and remains competitive on BindingDB while using only 25.2 million trainable parameters. Ablation results confirm the contribution of both the CNN and cross-attention modules, and cold-start experiments indicate promising generalization to unseen proteins and drugs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

35,526 characters extracted from source content.

Expand or collapse full text

Drug-Target Interaction Prediction via Hierarchical Sequential Cross-Attention over Chemical and Protein Language Models Khadidja Henni1, Hamza Abdelali3, Abdelkrim Aries3, Neila Mezghani12, Brigitte Vannier4, Sara Magdouli5, and Lina Abou-Abbas6 Affiliation: 1I2A institute, TÉLUQ University, Montreal, QC, Canada; 2LIO, CRCHUM, Montreal, QC, Canada 3LCSI, ESI, Algiers, Algeria; 4CoMeT UR 24344, Universite de Poitiers, Poitiers, France 5Dept. of Civil Engineering, University of Ottawa, Ottawa, ON, Canada; 6Dept. of Electrical and Computer Engineering, Lebanese American University, Byblos, Lebanon Abstract Predicting Drug-Target Interactions (DTIs) is a central task in computational drug discovery, with direct applications in virtual screening, drug repurposing, and therapeutic candidate prioritization. Although recent deep learning methods have improved DTI prediction, many sequence-based models still process drugs and proteins independently and only combine their representations at a late prediction stage. This limits their ability to explicitly model cross-molecular dependencies between chemical substructures and protein sequence regions. In this paper, we propose a sequence-only DTI prediction architecture that combines two pre-trained language models — ChemBERTa for drug SMILES strings and ESM-2 for protein amino acid sequences — with a hierarchical interaction module. The proposed model first extracts contextual representations using pre-trained encoders, then applies 1D convolutional layers to condense local sequence patterns, followed by a sequential bidirectional cross-attention mechanism inspired by the induced-fit view of molecular recognition. Finally, attention-based pooling constructs fixed-size interaction-aware vectors for binary prediction. Experiments on BIOSNAP, Davis, and BindingDB show that the proposed model achieves the best performance on BIOSNAP, matches the best AUROC on Davis, and remains competitive on BindingDB while using only 25.2 million trainable parameters. Ablation results confirm the contribution of both the CNN and cross-attention modules, and cold-start experiments indicate promising generalization to unseen proteins and drugs. Index Terms: Drug-Target Interaction, Deep Learning, Pre-trained Language Models, ChemBERTa, ESM-2, Cross-Attention, SMILES, Protein Sequences. I Introduction The development of new therapeutic compounds is a long, costly, and high-risk process. From target identification to clinical approval, drug discovery can require more than a decade of research and substantial financial investment [1]. A key step in this pipeline is the identification of Drug-Target Interactions (DTIs), which describe whether a chemical compound interacts with a biological target, usually a protein. Accurate DTI prediction supports virtual screening, drug repurposing, mechanism-of-action analysis, and early safety assessment. Traditionally, DTIs are identified through experimental screening. Although in vitro assays provide reliable evidence, they are difficult to scale to the enormous number of possible drug-protein pairs. The chemical space contains millions of possible drug-like molecules, while the human proteome contains thousands of potential protein targets. As a result, exhaustive experimental screening is impractical. Computational approaches therefore play an important role by prioritizing the most promising candidates for laboratory validation [1, 10]. From a machine learning perspective, DTI prediction is often formulated as a binary classification problem. Given a drug represented by a SMILES string and a target represented by an amino acid sequence, the model predicts whether an interaction exists. Early machine learning methods relied heavily on handcrafted descriptors, molecular fingerprints, and similarity-based features [10]. Deep learning models later reduced this dependence on manual feature engineering by learning representations directly from raw molecular and biological sequences [2]. Several neural architectures have been explored for DTI prediction, including convolutional neural networks (CNNs) [2], recurrent neural networks [3], graph neural networks [4], and Transformer-based models [5]. More recently, pre-trained language models (PLMs) have become increasingly important. ChemBERTa [6] learns chemical representations from large SMILES corpora, while ESM-2 [7] learns protein representations from large-scale protein sequence databases. These models provide rich contextual embeddings that can improve downstream DTI prediction. Despite this progress, two limitations remain common in many sequence-based models. First, drug and protein representations are often computed independently and fused only at the final prediction stage. This late-fusion strategy may fail to capture fine-grained dependencies between drug substructures and protein sequence regions. Second, global pooling or simple aggregation can discard position-specific information that may be important for identifying relevant residues or molecular fragments. To address these limitations, we propose a hierarchical sequence-based architecture for DTI prediction. The model combines ChemBERTa and ESM-2 encoders with CNN-based local feature extraction, sequential bidirectional cross-attention, and attention-based pooling. The sequential cross-attention module models interaction-aware refinement in two steps: the protein representation is first updated using drug context, and the drug representation is then updated using the refined protein context. This design is inspired by the induced-fit view of molecular recognition [9], without explicitly simulating 3D conformational changes. The main contributions of this paper are as follows: • We propose a sequence-only DTI prediction architecture combining ChemBERTa and ESM-2 with CNN-based local feature extraction, achieving competitive performance without molecular graphs or structural data. • We introduce a sequential bidirectional cross-attention mechanism that explicitly models asymmetric drug-protein dependencies and improves over its parallel counterpart on AUROC, AUPRC, and sensitivity, while achieving comparable specificity. • We use attention-based pooling to construct interaction-aware drug and protein vectors while preserving position-level importance, enabling more focused aggregation than standard global pooling. • Through systematic ablation, cold-start, and cross-dataset experiments, we demonstrate that each architectural component provides measurable improvements, and that the proposed partial fine-tuning strategy achieves competitive results with only 25.2M trainable parameters. The rest of the paper is organized as follows. Section I presents related work. Section I describes the proposed architecture. Section IV details the experimental setup. Section V presents and discusses the results. Section VI concludes the paper and outlines future work. I Background and Related Work I-A Drug and Protein Representations Drugs can be represented in several ways, including molecular descriptors, fingerprints, molecular graphs, and sequence-based notations. SMILES is one of the most widely used sequence representations because it provides a compact textual encoding of molecular structure [10]. Although SMILES strings do not explicitly describe 3D conformations, they are well suited for sequence models and chemical language models. Proteins are commonly represented by their amino acid sequences, structural graphs, contact maps, or learned embeddings. Protein sequences are widely available and can be processed by protein language models trained on large biological databases. Such models can capture contextual and evolutionary signals that are useful for downstream prediction tasks. I-B Deep Learning for DTI Prediction DeepDTA [2] introduced a CNN-based approach that processes SMILES and protein sequences directly. WideDTA [11] extended this idea by using multiple sequence channels and motif-based information. DeepAffinity [3] used recurrent architectures to model sequence dependencies, while GraphDTA [4] represented drugs as molecular graphs and applied graph neural networks. More broadly, graph-based formulations have also been used for high-dimensional representation, feature selection, and clustering, illustrating the usefulness of graph structure for organizing complex data beyond the DTI setting [24, 25]. Transformer-based methods have also been applied to DTI prediction. MolTrans [5] uses self-attention to model substructure interactions between drugs and proteins. Fine-tuning BERT-based models has further shown that pre-trained sequence encoders can improve DTI prediction performance [8]. More recent hybrid methods combine language models, graph encoders, and attention mechanisms, often improving accuracy at the cost of higher architectural complexity and larger trainable parameter counts [12, 13, 21]. I-C Attention-Based Interaction Modeling Fusion strategies in DTI prediction can be broadly divided into late fusion, self-attention over combined representations, and cross-attention. Late fusion combines drug and protein vectors after independent encoding, usually by concatenation. This strategy is simple but may not fully capture cross-modal dependencies. Cross-attention is more suitable for interaction modeling because one modality can query the other, allowing the model to learn which drug regions are relevant to which protein regions. Most existing cross-attention approaches compute the interaction in a single step or apply both attention directions simultaneously from the original representations, without exploiting any sequential dependency between the two refinement steps. This design gap motivates the sequential hierarchical interaction module proposed in this work. I Proposed Architecture I-A Problem Formulation Let D be the set of drugs and P the set of protein targets. Each drug d∈d is represented by a SMILES string, and each protein p∈p by an amino acid sequence. Given a drug-protein pair (d,p)(d,p), the objective is to predict a binary label y∈0,1y∈\0,1\, where y=1y=1 indicates an interaction and y=0y=0 indicates no known interaction. The model learns a prediction function: y^=f⁡(repr⁡(d),repr⁡(p))∈[0,1], y=f\! (repr(d),\,repr(p) )∈[0,1], (1) where repr⁡(⋅)repr(·) denotes the learned representation produced by the corresponding encoder. I-B Overall Architecture Figure 1 presents the proposed architecture, which consists of five main stages: pre-trained sequence encoding, CNN-based feature condensation, sequential bidirectional cross-attention, and attention-based pooling followed by an MLP prediction head. Fig. 1: Overview of the proposed architecture. Drug SMILES and protein sequences are encoded using ChemBERTa and ESM-2, condensed through 1D CNN layers, fused using sequential bidirectional cross-attention, and aggregated through attention-based pooling before final prediction. I-C Pre-trained Sequence Encoders Protein Encoder (ESM-2): For the protein branch, we use facebook/esm2_t30_150M_UR50D [7], a 150M-parameter protein language model pre-trained on the UniRef50 database via masked language modeling (MLM) over tens of millions of evolutionarily diverse sequences. The encoder produces contextual amino acid embeddings of dimension dp=480d_p=480. The maximum protein sequence length is set to 1,024 tokens. Drug Encoder (ChemBERTa): For the drug branch, we use seyonec/ChemBERTa-zinc-base-v1 [6, 14], a RoBERTa-based chemical language model pre-trained on SMILES strings from the ZINC database. It produces contextual drug token embeddings of dimension d=768d_d=768. The maximum drug sequence length is set to 128 tokens. Both encoders are initialized with pre-trained weights. To reduce computational cost and limit overfitting, only the top two Transformer layers of each encoder are unfrozen during fine-tuning; all remaining layers are kept frozen. This partial fine-tuning strategy reduces the number of trainable parameters from approximately 193 million to 25.2 million while preserving the knowledge acquired during pre-training. I-D CNN-Based Local Feature Extraction The contextual embeddings produced by the two encoders are passed through independent 1D CNN blocks. Each block contains: (i) a Conv1d layer with kernel size 3 and 128 output channels; (i) a ReLU activation; and (i) a MaxPool1d layer with kernel size 2, reducing sequence length by half. The role of this stage is twofold: it captures local sequence patterns such as chemical fragments or protein motifs, and it reduces sequence length to make subsequent cross-attention computation more efficient. I-E Sequential Bidirectional Cross-Attention Let p∈ℝLp×dH_p ^L_p× d and d∈ℝLd×dH_d ^L_d× d denote the CNN-condensed protein and drug representations, where LpL_p, LdL_d are the reduced sequence lengths and d=128d=128 the shared hidden dimension after CNN projection. Multi-Head Attention (MHA) with h heads is defined as: MHA⁡(Q,K,V)=Concat⁡(head1,…,headh)WO,MHA(Q,K,V)=Concat(head_1,…,head_h)\,W^O, (2) where each head applies scaled dot-product attention: Attention⁡(Q,K,V)=softmax⁡(QK⊤dk)V,Attention(Q,K,V)=softmax\! ( QK d_k )V, (3) with h=8h=8 heads and dk=d/hd_k=d/h. The proposed interaction module proceeds in two sequential steps: Drug-Guided Protein Refinement The protein representation queries the drug representation, forcing its embeddings to focus on the sub-regions most relevant to the drug’s chemical structure: p′=MHA⁡(Q=p,K=d,V=d).H _p=MHA(Q=H_p,\;K=H_d,\;V=H_d). (4) Adapted-Protein-Guided Drug Refinement The already drug-adapted protein representation then refines the drug features: d′=MHA⁡(Q=d,K=p′,V=p′).H _d=MHA(Q=H_d,\;K=H _p,\;V=H _p). (5) In a parallel design, both steps would use the original representations, making them independent. Here, Step 2 uses the refined p′H _p, creating a hierarchical dependency that provides a richer joint representation. This sequential design is inspired by the induced-fit principle [9], without claiming to simulate molecular dynamics or 3D conformational changes. I-F Attention-Based Pooling and Prediction Attention Pooling Standard global pooling discards position-level information. We replace it with a learnable scoring mechanism. For each position i, a scalar importance score is computed as: αi=2⊤tanh⁡(1i+1)+b2, _i=w_2 \! (W_1\,h_i+b_1 )+b_2, (6) where 1∈ℝda×dW_1 ^d_a× d, 2∈ℝdaw_2 ^d_a, and da=64d_a=64. The scores are normalized and used to compute a weighted sum: α¯i=exp⁡(αi)∑jexp⁡(αj),=∑i=1Lα¯ii. α_i= ( _i) _j ( _j), = _i=1^L α_i\,h_i. (7) This concentrates the fixed-size representation on pharmacologically relevant positions while suppressing uninformative regions. Prediction Head The pooled vectors pz_p and dz_d are concatenated to form a vector of dimension 2d2d and passed to an MLP with three linear layers (2d→256→128→12d→ 256→ 128→ 1), ReLU activations, and Dropout (p=0.2p=0.2) after each hidden layer. The final logit is mapped to a probability via sigmoid. Training minimizes the binary cross-entropy with logits loss: ℒ=−1N∑i=1N[yilogσ(y^i)+(1−yi)log(1−σ(y^i))],L=- 1N _i=1^N\! [y_i σ( y_i)+(1-y_i) (1-σ( y_i)) ], (8) where σ denotes the sigmoid function. IV Experimental Setup IV-A Datasets The proposed model is evaluated on three benchmark datasets: Davis, BIOSNAP, and BindingDB. Their statistics are summarized in Table I. Davis [15] contains binding affinity measurements between kinase inhibitors and kinase proteins. Following common binary DTI practice, affinities are binarized using a threshold of Kd<30K_d<30 nM, yielding a severely imbalanced dataset (≈ 1:6 ratio). BIOSNAP [16] is a balanced drug-target interaction benchmark derived from the BioSNAP biomedical network collection. It contains validated positive interactions and sampled negative pairs in equal proportion, making it suitable for evaluating discriminative performance without imbalance bias. BindingDB [17] is a large public database of experimentally measured protein-ligand binding affinities. Binary labels are derived using a standard threshold of Kd<1μMK_d<1\, . For Davis and BindingDB, the training sets are resampled to class balance to mitigate the effect of imbalance during optimization. Generative augmentation of latent drug–target representations has also been investigated as an alternative strategy for DTI class imbalance [23]. Validation and test sets retain their original distributions to reflect realistic evaluation conditions. No additional negative sampling is introduced during testing. TABLE I: Dataset statistics. Interaction pairs are reported as train / validation / test. Dataset Drugs Proteins Positive Pairs Negative Pairs Davis 68 379 1,043 / 160 / 303 1,043 / 2,846 / 5,708 BIOSNAP 4,510 2,181 9,619 / 1,374 / 2,748 9,619 / 1,374 / 2,748 BindingDB 10,665 1,413 6,334 / 927 / 1,905 6,334 / 5,717 / 11,384 IV-B Evaluation Protocol We use two evaluation settings. In the standard pair-level setting, drug-protein pairs are divided into training, validation, and test sets. A drug or protein may appear across multiple splits with different partners, measuring the model’s ability to predict new interaction pairs for known entities. In the cold-start setting, 20% of drugs (resp. proteins) are entirely held out, and all their interactions form the test set. This evaluates generalization to molecular entities not observed during training. Recent novelty-aware DTI work further emphasizes separate evaluation of ligand and protein novelty, since the two settings can induce different generalization challenges [22]. During development, all proposed-model experiments were repeated over five independent runs. The reported values correspond to the test performance of the checkpoint selected according to validation AUROC. Since not all baseline publications report standard deviations under identical splits, the comparison table reports point estimates, and small differences should be interpreted cautiously. A fully controlled benchmark with mean and standard deviation over all seeds is left for future work. IV-C Training Configuration The model is implemented using PyTorch and PyTorch Lightning. Key hyperparameters are reported in Table I. Early stopping monitors validation AUROC with a patience of three epochs. TABLE I: Main hyperparameters of the proposed model. Hyperparameter Value Optimizer AdamW Base learning rate 5×10−55× 10^-5 Learning-rate schedule Linear warmup + cosine decay Effective batch size 32 Maximum epochs 30 Unfrozen encoder layers Top 2 layers CNN output channels 128 Attention pooling dim dad_a 64 MLP hidden layers 2d→256→128→12d→ 256→ 128→ 1 MLP dropout 0.2 Protein max length 1,024 tokens Drug max length 128 tokens IV-D Metrics The model is evaluated using four complementary metrics: AUROC measures the global ability to separate positive and negative pairs across all classification thresholds; AUPRC is especially relevant for imbalanced datasets as it focuses on positive class performance; Sensitivity measures the proportion of true interactions correctly detected; and Specificity measures the proportion of non-interactions correctly rejected. V Results and Discussion V-A Comparison with State-of-the-Art Methods We compare the proposed model against representative DTI prediction methods spanning multiple architectural paradigms: CNN-based (DeepDTA [2]), Transformer-based (MolTrans [5]), PLM fine-tuning (Fine-tuning BERT [8]), dual PLM (DLM-DTI [13]), interpretable cross-attention (ICAN [19]), HyperAttentionDTI [18], and a recent multi-scale fusion approach (Yang et al. [20]). Baseline values are taken from the corresponding publications or comparable reported benchmark results when available. The comparison should therefore be interpreted as a literature-level evaluation rather than a fully controlled reimplementation of all baselines. Table I presents results on all three datasets. BIOSNAP. The proposed model obtains the best AUROC (0.920) and AUPRC (0.924) among all compared methods, demonstrating that the combination of pre-trained encoders, CNN feature extraction, and sequential cross-attention is effective on balanced DTI data. The model achieves a sensitivity of 0.854 and specificity of 0.830, slightly below Fine-tuning BERT on these two metrics, but leading overall. Davis. The proposed model reaches an AUROC of 0.920, matching the best reported result. It also obtains the highest sensitivity (0.901), which is important in virtual screening because false negatives represent missed candidate interactions. However, its AUPRC (0.356) is the lowest in the comparison on this dataset, indicating that precision under severe class imbalance remains a clear limitation of the current approach. BindingDB. On this large-scale, chemically diverse dataset, the proposed model achieves a competitive AUROC of 0.920 and AUPRC of 0.642. It does not lead on every metric — HyperAttentionDTI achieves the best AUPRC (0.876) and DLM-DTI achieves the best specificity (0.916) — but it remains competitive while relying exclusively on sequence inputs without molecular graphs or 3D structural data. TABLE I: Performance comparison on BIOSNAP, Davis, and BindingDB. Best result per metric per dataset is in bold. BIOSNAP Davis BindingDB Model AUROC AUPRC Sens. Spec. AUROC AUPRC Sens. Spec. AUROC AUPRC Sens. Spec. MolTrans [5] 0.895 0.901 0.775 0.851 0.907 0.404 0.800 0.876 0.914 0.622 0.797 0.896 DeepDTA [2] 0.834 0.849 0.726 0.813 0.792 0.272 0.688 0.778 0.902 0.604 0.757 0.904 HyperAttentionDTI [18] 0.825 0.830 0.800 0.812 0.841 0.483 0.708 0.722 0.900 0.876 0.801 0.791 Fine-tuning BERT [8] 0.914 0.900 0.862 0.847 0.920 0.395 0.824 0.802 0.922 0.623 0.814 0.793 DLM-DTI [13] 0.914 0.914 0.848 0.844 0.895 0.373 0.833 0.766 0.912 0.643 0.846 0.916 ICAN [19] 0.871 0.886 0.799 0.786 0.903 0.372 0.884 0.765 0.900 0.604 0.857 0.815 Yang et al. [20] 0.885 0.856 0.828 0.787 0.799 0.515 0.896 0.850 0.910 0.659 0.815 0.818 Proposed 0.920 0.924 0.854 0.830 0.920 0.356 0.901 0.801 0.920 0.642 0.816 0.874 V-B Integrated Dataset Evaluation To evaluate whether the model benefits from a larger and more diverse biochemical training corpus, we train it on the combined training and validation sets of Davis, BIOSNAP, and BindingDB, then evaluate separately on each original test set. As shown in Table IV, integrated training yields substantial AUPRC improvements on Davis (+0.092+0.092: 0.356→0.4480.356→ 0.448) and BindingDB (+0.122+0.122: 0.642→0.7640.642→ 0.764), confirming that the architecture scales well with data diversity. On BIOSNAP, AUPRC decreases from 0.924 to 0.840, which may reflect domain shift caused by mixing datasets with different distributions and label-generation protocols. TABLE IV: Individual training vs. integrated training. Test set Training AUROC AUPRC Sens. Spec. Davis Individual 0.920 0.356 0.901 0.801 Integrated 0.939 0.448 0.911 0.848 BIOSNAP Individual 0.920 0.924 0.854 0.830 Integrated 0.920 0.840 0.886 0.835 BindingDB Individual 0.920 0.642 0.816 0.874 Integrated 0.918 0.764 0.852 0.850 V-C Cold-Start Generalization To assess generalization to unseen entities, we conduct a cold-start evaluation using a 20% hold-out protocol applied independently to drugs and proteins. Results are reported in Table V. In the unseen-protein setting, the proposed model achieves a ROC-AUC of 0.87, outperforming MolTrans (0.77) and DeepDTA (0.85). This suggests that ESM-2 transfers useful evolutionary and structural priors that help the model reason about novel protein targets beyond its training distribution. In the unseen-drug setting, all three models reach 0.85, indicating that sequence-based approaches generalize comparably to new chemical entities in this evaluation. TABLE V: Cold-start evaluation (ROC-AUC). Model Unseen Drugs Unseen Proteins MolTrans [5] 0.85 0.77 DeepDTA [2] 0.85 0.85 Proposed 0.85 0.87 V-D Ablation Study We conduct an ablation study on BIOSNAP to evaluate the contribution of each architectural component. Results are shown in Table VI. Component importance Removing the CNN block reduces AUROC from 0.920 to 0.916, confirming that local feature extraction provides complementary information beyond the Transformer embeddings alone. Removing the cross-attention block causes a larger degradation (AUROC: 0.920→0.9110.920→ 0.911), confirming that cross-modal interaction modeling is the single most important post-encoder component. Removing both modules yields the weakest performance (AUROC: 0.905), validating their complementary and cumulative contributions. Sequential vs. parallel attention: The sequential variant outperforms the parallel variant on AUROC (0.920 vs. 0.914), AUPRC (0.924 vs. 0.920), and Sensitivity (0.854 vs. 0.833), while the parallel variant slightly improves Specificity (0.834 vs. 0.830). The gain is moderate but consistent across the main discriminative metrics, supporting the hypothesis that hierarchical refinement — where Step 2 queries an already drug-adapted protein context — provides a richer joint representation than computing both directions simultaneously. TABLE VI: Ablation study on BIOSNAP. Variant AUROC AUPRC Sens. Spec. Full model 0.920 0.924 0.854 0.830 Without CNN 0.916 0.920 0.849 0.828 Without attention 0.911 0.914 0.835 0.819 Without CNN + attention 0.905 0.907 0.828 0.812 Sequential attention 0.920 0.924 0.854 0.830 Parallel attention 0.914 0.920 0.833 0.834 V-E Model Complexity By restricting fine-tuning to the top two layers of each encoder and the newly added modules, the proposed architecture achieves only 25.2 million trainable parameters, despite a total model size of approximately 193 million. As shown in Table VII, this represents a 2.5× reduction relative to MolTrans, a 3.4× reduction relative to DLM-DTI, and a 14× reduction relative to full BERT fine-tuning, while maintaining competitive or superior predictive performance. TABLE VII: Trainable parameter comparison. Model Trainable Parameters MolTrans [5] 62.8M DLM-DTI [13] 86.7M Fine-tuning BERT [8] 353.0M Proposed 25.2M V-F Discussion and Limitations The proposed architecture performs best on BIOSNAP, where the balanced class distribution allows both AUROC and AUPRC to reflect genuine discriminative quality. On Davis, high AUROC and sensitivity are achieved, but the low AUPRC (0.356) reveals a precision limitation under severe class imbalance: the model prioritizes recall at the expense of precision, which may be acceptable in early-stage virtual screening but would require recalibration for applications demanding higher specificity. On BindingDB, competitive AUROC is obtained despite the chemical diversity of the dataset, suggesting that ChemBERTa provides sufficiently general representations. However, the gap in AUPRC relative to HyperAttentionDTI, which uses structural information, points to a ceiling for purely sequence-based approaches on chemically complex data. Several limitations should be noted. First, the method is sequence-only and does not use molecular graphs, 3D protein structures, docking poses, or binding-site geometry, which may limit performance on datasets where structural information is decisive. Second, although attention weights provide a more transparent interaction mechanism than simple concatenation, they should not be interpreted as biological explanations without validation against known binding sites or crystal structures. Third, the baseline comparison relies partly on reported literature values obtained under potentially different preprocessing and split protocols; a fully controlled reimplementation would be required for strict comparison. VI Conclusion This paper presented a hierarchical sequence-based architecture for Drug-Target Interaction prediction combining ChemBERTa and ESM-2 pre-trained language models with CNN-based local feature extraction, sequential bidirectional cross-attention, and attention-based pooling. Experiments on BIOSNAP, Davis, and BindingDB demonstrate that the proposed method achieves strong and competitive results while using only 25.2 million trainable parameters. Ablation experiments confirm that both the CNN condensation and the sequential cross-attention module provide measurable improvements over their respective ablated variants. Cold-start evaluation indicates encouraging generalization, particularly to unseen protein targets. The Discussion section explicitly identifies precision under class imbalance and the absence of structural information as current limitations. Future work will focus on four directions. First, richer chemical encoders or graph-based drug representations could be integrated to capture molecular topology more explicitly, which may reduce the AUPRC gap on imbalanced and structurally diverse datasets. Second, protein structural information or predicted binding-site features could be added to provide better biological grounding. Third, attention maps should be systematically validated against known binding residues or protein-ligand crystal structures before making strong interpretability claims. Finally, a fully controlled benchmark with reimplemented baselines, identical splits, and reported mean and standard deviation across multiple seeds would provide a more rigorous empirical foundation for future work in this line of research. Generative AI Usage Statement Generative AI tools were used only for language polishing and formatting assistance. All AI-assisted revisions were reviewed and verified by the authors. All scientific contributions, experiments, results, and conclusions are entirely produced by the authors. References [1] J. P. Hughes, S. Rees, S. B. Kalindjian, and K. L. Philpott, “Principles of early drug discovery,” British Journal of Pharmacology, vol. 162, no. 6, p. 1239–1249, 2011. [2] H. Öztürk, A. Özgür, and E. Ozkirimli, “DeepDTA: deep drug–target binding affinity prediction,” Bioinformatics, vol. 34, no. 17, p. i821–i829, 2018. [3] M. Karimi, D. Wu, Z. Wang, and Y. Shen, “DeepAffinity: interpretable deep learning of compound–protein affinity through unified recurrent and convolutional neural networks,” Bioinformatics, vol. 35, no. 18, p. 3329–3338, 2019. [4] T. Nguyen, H. Le, T. P. Quinn, T. Nguyen, T. D. Le, and S. Venkatesh, “GraphDTA: predicting drug–target binding affinity with graph neural networks,” Bioinformatics, vol. 37, no. 8, p. 1140–1147, 2021. [5] K. Huang, C. Xiao, L. M. Glass, and J. Sun, “MolTrans: Molecular Interaction Transformer for drug–target interaction prediction,” Bioinformatics, vol. 37, no. 6, p. 830–836, 2021. [6] S. Chithrananda, G. Grand, and B. Ramsundar, “ChemBERTa: Large-scale self-supervised pretraining for molecular property prediction,” arXiv preprint arXiv:2010.09885, 2020. [7] Z. Lin et al., “Evolutionary-scale prediction of atomic-level protein structure with a language model,” Science, vol. 379, no. 6637, p. 1123–1130, 2023. [8] H. Kang, S. Goo, H. Lee, J.-W. Chae, H.-Y. Yun, and S. Jung, “Fine-tuning of BERT model to accurately predict drug–target interactions,” Pharmaceutics, vol. 14, no. 8, p. 1710, 2022. [9] D. E. Koshland, Jr., “Application of a theory of enzyme specificity to protein synthesis,” Proceedings of the National Academy of Sciences of the United States of America, vol. 44, no. 2, p. 98–104, 1958. [10] S. Lim, Y. Lu, C. Y. Cho, I. Sung, J. Kim, Y. Kim, S. Park, and S. Kim, “A review on compound–protein interaction prediction methods: Data, format, representation and model,” Computational and Structural Biotechnology Journal, vol. 19, p. 1541–1556, 2021. [11] H. Öztürk, E. Ozkirimli, and A. Özgür, “WideDTA: prediction of drug–target binding affinity,” arXiv preprint arXiv:1902.04166, 2019. [12] Z. Luo, W. Wu, Q. Sun, and J. Wang, “Accurate and transferable drug–target interaction prediction with DrugLAMP,” Bioinformatics, vol. 40, no. 12, p. btae693, 2024. [13] J. Lee, D. W. Jun, I. Song, and Y. Kim, “DLM-DTI: a dual language model for the prediction of drug–target interaction with hint-based learning,” Journal of Cheminformatics, vol. 16, no. 1, p. 14, 2024. [14] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [15] M. I. Davis et al., “Comprehensive analysis of kinase inhibitor selectivity,” Nature Biotechnology, vol. 29, no. 11, p. 1046–1051, 2011. [16] M. Zitnik, R. Sosič, S. Maheshwari, and J. Leskovec, “BioSNAP Datasets: Stanford Biomedical Network Dataset Collection,” Stanford University, Aug. 2018. [17] T. Liu, Y. Lin, X. Wen, R. N. Jorissen, and M. K. Gilson, “BindingDB: a web-accessible database of experimentally determined protein–ligand binding affinities,” Nucleic Acids Research, vol. 35, no. suppl_1, p. D198–D201, 2007. [18] Q. Zhao, H. Zhao, K. Zheng, and J. Wang, “HyperAttentionDTI: improving drug–protein interaction prediction by sequence-based deep learning with attention mechanism,” Bioinformatics, vol. 38, no. 3, p. 655–662, 2022. [19] H. Kurata and S. Tsukiyama, “ICAN: Interpretable cross-attention network for identifying drug and target protein interactions,” PLOS ONE, vol. 17, no. 10, p. e0276609, 2022. [20] Z. Yang, B. Bai, J. Long, P. Wei, and J. Li, “Multi-scale Feature Fusion Neural Network for Accurate Prediction of Drug–Target Interactions,” Communications in Computer and Information Science, vol. 1964, p. 176–188, Springer, 2024. [21] K. Henni, L. Hamdad, N. Mezghani, N. E. Benazzoug, and L. Abou-Abbas, “Integrating structural graphs and cross-attention for drug–target interaction prediction,” in Proc. IEEE/ACS 22nd Int. Conf. Computer Systems and Applications (AICCSA), 2025. [22] L. Abou-Abbas and K. Henni, “Protein and ligand novelty in drug–target interaction prediction: A dual-encoder fusion strategy for more interpretable and generalizable modeling,” BMC Bioinformatics, 2026. [23] V. M. Silva, K. Henni, N. Mezghani, and Y. Abdelliche, “A matrix factorization and generative modeling framework for drug–target interaction prediction,” in Proc. BIOSTEC, vol. 2, p. 688–695, 2026. [24] K. Henni, N. Mezghani, and A. Mitiche, “Cluster density properties define a graph for effective pattern feature selection,” IEEE Access, vol. 8, p. 62841–62854, 2020, doi: 10.1109/ACCESS.2020.2981265. [25] K. Henni, P.-Y. Louis, B. Vannier, and A. Moussa, “Is-ClusterMPP: clustering algorithm through point processes and influence space towards high-dimensional data,” Advances in Data Analysis and Classification, vol. 14, no. 3, p. 543–570, 2020, doi: 10.1007/s11634-019-00379-2.

Instant research discovery