Paper deep dive
LLM-Enhanced Dual-Branch Learning for Large-Scale Multi-Label Text Classification
Hui Ye, Jing Zhang, Xiulong Yang, Rajshekhar Sunderraman
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/20/2026, 2:35:52 AM
Summary
The paper introduces DualMLC, a dual-branch framework for large-scale multi-label text classification that leverages heterogeneous language models. It processes documents through an autoregressive decoder-only model (Qwen2.5-7B) and a bidirectional encoder (BERT-base-uncased) independently. Each branch produces relevance scores over a shared label space, which are then combined via late logit fusion. This approach exploits the complementary semantic representations of the two architectures, achieving state-of-the-art results on three benchmarks: EURLex-4K, Wiki10-31K, and AmazonCat-13K.
Entities (8)
Relation Signals (7)
DualMLC → achievesstateofthearton → Wiki10-31K
confidence 95% · DualMLC achieves state-of-the-art results on three widely used large-scale multi-label text classification benchmarks.
DualMLC → achievesstateofthearton → EURLex-4K
confidence 95% · DualMLC achieves state-of-the-art results on three widely used large-scale multi-label text classification benchmarks.
DualMLC → achievesstateofthearton → AmazonCat-13K
confidence 95% · DualMLC achieves state-of-the-art results on three widely used large-scale multi-label text classification benchmarks.
DualMLC → usescomponent → Qwen2.5-7B
confidence 95% · Branch q uses Qwen2.5-7B [21]
DualMLC → usescomponent → BERT-base-uncased
confidence 95% · Branch b uses BERT-base-uncased [22]
DualMLC → usestechnique → Late Logit Fusion
confidence 95% · DualMLC combines the two score vectors through late logit fusion
Qwen2.5-7B → adaptedwith → LoRA
confidence 90% · We adapt the larger autoregressive backbone with low rank adaptation [23]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-scale multi-label text classification assigns a small subset of relevant labels to each document from a vocabulary containing thousands or tens of thousands of candidate labels. Although pretrained language models have improved semantic text representations, most representation-based approaches center their prediction pipelines on a primary encoder or combine auxiliary features within a single ranker. The complementarity between heterogeneous language models therefore remains insufficiently explored. We propose DualMLC, a dual-branch framework that processes the same document through an autoregressive decoder-only language model and a bidirectional encoder. Each branch maintains its own representation pathway and independently estimates relevance scores over the shared label space. DualMLC combines the two score vectors through late logit fusion, allowing shared evidence to reinforce relevant labels and branch-specific evidence to compensate for limitations in the other branch's representation. DualMLC achieves state-of-the-art results on three widely used large-scale multi-label text classification benchmarks. Ablation results further confirm that integrating the heterogeneous predictors produces stronger rankings than either branch alone. The source code is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.12915v1
- Canonical: https://arxiv.org/abs/2609.12915v1
Trouble viewing inline? Open PDF directly →
Full Text
35,068 characters extracted from source content.
Expand or collapse full text
LLM-Enhanced Dual-Branch Learning for Large-Scale Multi-Label Text Classification Hui Ye1, Jing Zhang2, Xiulong Yang3, Rajshekhar Sunderraman1 Affiliation: 1Department of Computer Science, Georgia State University, Atlanta, GA 30303, USA 2Amazon, San Diego, CA, USA 3School of Computer Science, Central China Normal University, Wuhan 430079, China hye2@student.gsu.edu, jingz3017@gmail.com, yangxiulong@ccnu.edu.cn, rsunderraman@gsu.edu Abstract Large-scale multi-label text classification assigns a small subset of relevant labels to each document from a vocabulary containing thousands or tens of thousands of candidate labels. Although pretrained language models have improved semantic text representations, most representation-based approaches center their prediction pipelines on a primary encoder or combine auxiliary features within a single ranker. The complementarity between heterogeneous language models therefore remains insufficiently explored. We propose DualMLC, a dual-branch framework that processes the same document through an autoregressive decoder-only language model and a bidirectional encoder. Each branch maintains its own representation pathway and independently estimates relevance scores over the shared label space. DualMLC combines the two score vectors through late logit fusion, allowing shared evidence to reinforce relevant labels and branch-specific evidence to compensate for limitations in the other branch’s representation. DualMLC achieves state-of-the-art results on three widely used large-scale multi-label text classification benchmarks. Ablation results further confirm that integrating the heterogeneous predictors produces stronger rankings than either branch alone. The source code is publicly available at https://github.com/huiyegit/DualMLC. Index Terms: large-scale multi-label text classification, large language models, heterogeneous representation learning, dual-branch learning I Introduction Multi-label text classification assigns each document a subset of relevant labels from a predefined vocabulary. In large-scale settings, the vocabulary may contain thousands or tens of thousands of labels, while each document is typically associated with only a small subset. By supporting fine-grained semantic organization and retrieval, the task is valuable in applications including web page tagging [1], product categorization, document tagging, recommendation systems [2], and semantic search. Research on large-scale multi-label text classification has improved both label prediction efficiency and semantic text representation by recent representative neural systems, including LightXML [3], XR-Transformer [2], MatchXML [4], CascadeXML [5], and QUEST [6], center their prediction pipelines on a primary text encoder. Their label scores are therefore shaped mainly by a single representation geometry. However, bidirectional encoders and autoregressive language models differ substantially in architecture, contextualization mechanism, and pretraining objective, and may therefore capture distinct yet complementary semantic evidence. This distinction is important for prediction over large label vocabularies, where many labels are semantically similar and small changes in text representation can alter their ranking positions. It raises a fundamental scientific question of whether heterogeneous encoders provide genuinely complementary representations and whether their combined evidence can improve the accuracy of the highest ranked labels. To address this issue, in this paper, we propose a Dual-branch Multi-Label Classification framework, referred to as DualMLC, that explicitly exploits complementary representations for large-scale multi-label text classification. DualMLC processes the same input text through two parallel branches constructed with different representation mechanisms, enabling the input semantics to be modeled from distinct perspectives. Each branch maintains an independent representation pathway and estimates the relevance of candidate labels using its own semantic evidence. This independent learning preserves the distinctive information captured by each representation and avoids prematurely forcing heterogeneous features into a shared space. The prediction evidence produced by the two branches is then combined to form a unified ranking over the label set, allowing shared evidence to reinforce relevant labels and complementary evidence to compensate for limitations of each individual representation. In this way, DualMLC translates representation complementarity into a direct mechanism for improving prediction over large label vocabularies. Our main contributions are summarized as follows. • We propose DualMLC, a heterogeneous dual branch framework that learns independent text representations through two distinct modeling mechanisms and integrates their prediction evidence for large-scale multi-label text classification. • We investigate representation complementarity over large label vocabularies and empirically demonstrate that the two branches capture distinct yet complementary evidence, with their integration outperforming either branch alone. • Extensive experiments demonstrate that DualMLC achieves new state-of-the-art results on three benchmark datasets, while comprehensive ablation studies verify the contribution of branch integration and the effects of key design choices. I Related Work Large-scale multi-label learning initially emphasized efficient output modeling. Babbar and Schölkopf [7] proposed DiSMEC to retain label specific discrimination with distributed optimization, whereas Yen et al. [8] developed PD-Sparse to reduce cost through primal and dual sparsity. Embedding methods instead compress local label structure. Bhatia et al. [9] learned neighborhood preserving representations in SLEEC, and Tagami [10] reconstructed a label neighborhood graph in AnnexML for approximate retrieval. Tree based methods restrict scoring to a candidate path. Jain et al. [11] introduced propensity scored trees in PfastreXML, Prabhu et al. [12] organized labels with balanced partitions in Parabel, and Khandagale et al. [13] developed shallower and more diverse trees in Bonsai. Wydmuch et al. [14] established a probabilistic foundation for label trees through eXtremeText, while Yu et al. [15] unified semantic indexing, matching, and sparse ranking in XR-Linear. These methods provide effective mechanisms for controlling label scoring, but their primary concern is output organization rather than heterogeneous semantic text encoding. Contextual encoders shifted the focus from output structure toward semantic representation learning. Liu et al. [16] proposed XML-CNN to learn convolutional document features, and You et al. [1] subsequently combined sequence encoding with label specific attention in AttentionXML. Pretrained Transformers further strengthened this direction. Chang et al. [17] introduced X-Transformer for hierarchical cluster matching, while Jiang et al. [3] developed LightXML to share one Transformer representation between label recall and ranking. Zhang et al. [2] extended fine tuning across label resolutions in XR-Transformer, and Kharbanda et al. [5] associated intermediate layers with different resolutions in CascadeXML. A parallel line enriched the prediction signal beyond the primary encoder. Xun et al. [18] modeled label correlations with CORNet, Dahiya et al. [19] aligned instance and label representations in SiameseXML, and Chien et al. [20] incorporated side information through PINA. More recently, Ye et al. [4] combined sparse, task tuned, and static sentence representations in MatchXML, while Zhou et al. [6] adapted a quantized large language model in QUEST. These advances demonstrate the value of richer semantics, but they generally refine one encoder hierarchy or combine multiple feature sources within a single ranking pipeline. DualMLC instead learns architecture distinct encoders as independently supervised predictors and integrates their evidence only after both map to the common label space. I Method I-A Task Definition Let =(xi,i)i=1ND=\(x_i,y_i)\_i=1^N denote a training set of N documents. Each document xix_i is associated with a multi-hot vector i∈0,1Ly_i∈\0,1\^L over the label set ℒ=ℓ1,…,ℓLL=\ _1,…, _L\. The entry yij=1y_ij=1 indicates that label ℓj _j is relevant to xix_i, while yij=0y_ij=0 indicates otherwise. The task is to learn a scoring function fθ(xi)∈ℝLf_θ(x_i) ^L, parameterized by the model parameter collection θ, that assigns higher scores to relevant labels. For a requested prediction depth k, the predicted label set is ^i(k)=ℓj∈ℒ∣j∈TopK(fθ(xi),k), Y_i^(k)= \ _j j \! (f_θ(x_i),k ) \, (1) where TopKTopK returns the indices of the k highest scoring entries. The central modeling requirement is therefore to construct document representations that preserve sufficient semantic evidence for reliable ranking across a large and closely related label set. I-B Overview of DualMLC DualMLC realizes the scoring function with parallel heterogeneous encoders. We use q for the autoregressive branch and b for the bidirectional branch. For r∈q,br∈\q,b\, its logit vector ir∈ℝLz_i^r ^L is ir=(r∘ℛr∘r∘r∘ℰr∘τr)(xi),z_i^r= (C_r _r _r _r _r _r )(x_i), (2) where ∘ denotes function composition, τr _r is the tokenizer, ℰrE_r is the encoder, rG_r aggregates upper layers, rP_r performs sequence pooling, ℛrR_r adjusts dimensionality, and rC_r produces L label logits. Figure 1 summarizes their flow. The branches remain independent until the common label space, which serves as a fusion interface without forcing hidden feature alignment. Each encoder retains its native contextualization and pooling convention, while its classifier maps branch specific evidence to coordinates associated with the same labels. Fig. 1: Overview of DualMLC. The same input text is processed by two heterogeneous representation branches. Each branch independently produces label logits, which are combined to obtain the final label ranking. Both branches receive direct supervision during training. I-C Heterogeneous Text Encoding The branch tokenizers τq _q and τb _b independently transform a document into two token sequences iq=τq(xi),ib=τb(xi).t_i^q= _q(x_i), _i^b= _b(x_i). (3) Separate tokenization respects the different pretrained vocabularies. For document xix_i in branch r∈q,br∈\q,b\, let ir=[ai1r,…,aiTrr]∈0,1Tra_i^r=[a_i1^r,…,a_iT_r^r]∈\0,1\^T_r denote its binary attention mask of length TrT_r, where t indexes token positions. We set aitr=1a_it^r=1 for a nonpadding token and aitr=0a_it^r=0 for padding. The mask enters every Transformer layer and excludes padded positions during document pooling. The token states are i,0r _i,0^r =Embr(ir), =Emb_r(t_i^r), (4) i,mr _i,m^r =Er,m(i,m−1r,ir),m=1,…,Mr, =E_r,m\! (H_i,m-1^r,a_i^r ), m=1,…,M_r, where EmbrEmb_r and Er,mE_r,m are the embedding layer and the mmth Transformer layer. The state i,mr∈ℝTr×drH_i,m^r ^T_r× d_r has hidden width drd_r, and MrM_r is the number of layers. Branch q uses causal attention, whereas branch b uses bidirectional attention. Causal attention summarizes a document under an autoregressive constraint, while bidirectional attention integrates evidence from both sides of every token. Their different context formation mechanisms can therefore emphasize different lexical and semantic cues for the same label, providing the representational diversity required by the dual branch design. Branch q uses Qwen2.5-7B [21], while branch b uses BERT-base-uncased [22]. We adapt the larger autoregressive backbone with low rank adaptation [23] in each attention projection indexed by p∈Q,K,V,Op∈\Q,K,V,O\, where Q, K, V, and O denote the query, key, value, and output projections. For input ∈ℝdinu ^d_in at layer m∈1,…,Mqm∈\1,…,M_q\, the adapted projection is Projm,p()=m,p0+γLrLm,pm,pL(),Proj_m,p(u)=W_m,p^0u+ _Lr_LB_m,pA_m,pD_L(u), (5) where dind_in and doutd_out are the input and output widths of the projection. The frozen pretrained weight is m,p0∈ℝdout×dinW_m,p^0 ^d_out× d_in, whose superscript 00 distinguishes it from the learned update. The factors m,p∈ℝrL×dinA_m,p ^r_L× d_in and m,p∈ℝdout×rLB_m,p ^d_out× r_L are trainable. Moreover, rLr_L is the LoRA rank, γL _L is the scaling factor, and LD_L is the LoRA dropout operator applied to u. The product m,pm,pB_m,pA_m,p constitutes a task specific update while leaving m,p0W_m,p^0 unchanged. Its rank controls the adaptation subspace, while γL/rL _L/r_L controls the update scale. This design limits the trainable parameters of branch q, whereas branch b is fully fine tuned to retain task specific flexibility. I-D Layer Aggregation and Document Representation Because useful semantics may remain distributed across upper layers, we average the last srs_r layers before pooling. Uniform averaging introduces no trainable parameters and reduces dependence on a single final layer. For a branch with MrM_r layers, the aggregated states are ¯ir=1sr∑j=0sr−1i,Mr−jr. H_i^r= 1s_r _j=0^s_r-1H_i,M_r-j^r. (6) Pooling follows the encoder conventions. With ¯i,tr H_i,t^r denoting the ttth token state, branch q uses masked mean pooling and branch b uses the first classification token iq _i^q =∑t=1Tqaitq¯i,tq∑t=1Tqaitq, = _t=1^T_qa_it^q H_i,t^q _t=1^T_qa_it^q, (7) ib _i^b =¯i,1b. = H_i,1^b. (8) This avoids imposing the same sequence summary on different encoder architectures. Since dqd_q is substantially larger than dbd_b, we use parameter free grouped mean reduction to control the label head size. For a group size g that divides dqd_q, the jjth component of ~iq∈ℝdq/g h_i^q ^d_q/g is [~iq]j=1g∑s=1g[iq]g(j−1)+s,j=1,…,dq/g. [ h_i^q ]_j= 1g _s=1^g [h_i^q ]_g(j-1)+s, j=1,…,d_q/g. (9) No reduction is applied to branch b, so ~ib=ib h_i^b=h_i^b. The grouped operation reduces the branch q classifier from LdqLd_q to Ldq/gLd_q/g weights while preserving feature scale. It does not require the two representations to have the same dimension because comparability is established only after both are mapped to L label logits. I-E Branch Specific Label Prediction Let d~q=dq/g d_q=d_q/g and d~b=db d_b=d_b denote the two classifier input dimensions. Each branch uses an independent linear classifier to map its document representation into the common label space ir=rDropout(~ir)+r,r∈q,b,z_i^r=W_rDropout\! ( h_i^r )+c_r, r∈\q,b\, (10) where r∈ℝL×d~rW_r ^L× d_r and r∈ℝLc_r ^L are classifier parameters, and DropoutDropout is classifier dropout. Thus each branch learns its own semantic mapping, while the shared label coordinates make the logit vectors directly comparable. TABLE I: P@k (%) on three multi-label text classification benchmarks. The best and second-best results are shown in bold and underlined, respectively. Method EURLex-4K Wiki10-31K AmazonCat-13K P@1 P@3 P@5 P@1 P@3 P@5 P@1 P@3 P@5 AnnexML [10] 79.66 69.64 53.52 86.46 74.28 64.20 93.54 78.36 63.30 DiSMEC [7] 83.21 70.39 58.73 84.13 74.72 65.94 93.81 79.08 64.06 PfastreXML [11] 73.14 60.16 50.54 83.57 68.61 59.10 91.75 77.97 63.68 Parabel [12] 82.12 68.91 57.89 84.19 72.46 63.37 93.02 79.14 64.51 eXtremeText [14] 79.17 66.80 56.09 83.66 73.28 64.51 92.50 78.12 63.51 Bonsai [13] 82.30 69.55 58.35 84.52 73.76 64.69 92.98 79.13 64.46 XR-Linear [15] 84.14 72.05 60.67 85.75 75.79 66.69 94.64 79.98 64.79 XML-CNN [16] 75.32 60.14 49.21 81.41 66.23 56.11 93.26 77.06 61.40 AttentionXML [1] 85.49 73.08 61.10 87.10 77.80 68.80 95.65 81.93 66.90 LightXML [3] 86.02 74.02 61.87 87.80 77.30 68.00 96.55 83.70 68.46 APLC-XLNet [24] 83.60 70.20 57.90 88.76 79.11 69.63 96.14 82.86 67.58 XR-Transformer [2] 87.22 74.39 61.69 88.00 78.70 69.10 96.25 82.72 67.01 MatchXML [4] 88.12 75.00 62.22 89.30 80.45 70.89 96.50 83.25 67.69 DualMLC (Ours) 88.82 75.80 62.74 90.78 81.11 71.94 96.90 84.39 69.14 I-F Logit Fusion and Inference The common output space enables late fusion according to i=αiq+(1−α)ib,0≤α≤1,z_i= _i^q+(1-α)z_i^b, 0≤α≤ 1, (11) where i∈ℝLz_i ^L is the fused logit vector and α controls the contribution of branch q. This preserves branch independence while allowing complementary evidence to compensate for a weak branch score. The probabilities are ^i=σ(i) p_i=σ(z_i), where σ is the elementwise sigmoid. Its monotonicity preserves the logit ranking, so inference returns the labels with the k largest entries of iz_i. I-G Training Objective Both branches receive direct supervision from the same multi-hot target. For branch r∈q,br∈\q,b\, with zijrz_ij^r denoting the jjth entry of irz_i^r, the binary cross entropy is ℒr=−1NL∑i=1N∑j=1L[yijlogσ(zijr)+(1−yij)log(1−σ(zijr))]. splitL_r=- 1NL _i=1^N _j=1^L [&y_ij σ(z_ij^r)\\ &+(1-y_ij) \! (1-σ(z_ij^r) ) ]. split (12) Let Θq _q and Θb _b be the trainable blocks of θ. The former contains the LoRA factors and branch q classifier, while the latter contains the branch b encoder and classifier. Training solves (Θq⋆,Θb⋆)=argminΘq,Θb[ℒq(Θq)+ℒb(Θb)]. ( _q , _b )= _q, _b [L_q( _q)+L_b( _b) ]. (13) The superscript ⋆ denotes the optimized parameter values. The frozen autoregressive backbone and fixed α are excluded from these sets, and no loss is applied to the fused logits. Separate losses preserve the informativeness of each predictor, allowing their logits to be evaluated separately and combined only during late fusion. TABLE I: Statistics of the benchmark datasets. Dataset Train Test Labels Avg. Labels Avg. Instances EURLex-4K 15,449 3,865 3,956 5.30 20.79 Wiki10-31K 14,146 6,616 30,938 18.64 8.52 AmazonCat-13K 1,186,239 306,782 13,330 5.04 448.57 IV Experiments IV-A Experimental Settings IV-A1 Datasets We evaluate DualMLC on three widely used multi-label text classification benchmarks that differ in domain, label cardinality, and training scale. EURLex-4K [25] contains European Union legal documents annotated with EuroVoc concepts. Wiki10-31K [26] consists of English Wikipedia articles paired with social tags collected from Delicious. AmazonCat-13K [27] contains Amazon product text associated with multiple product categories. Table I summarizes the benchmark splits used in our experiments. Following prior work, we report precision at one, three, and five, denoted by P@1, P@3, and P@5. In general, P@k is the fraction of the top k predictions that belong to the ground truth label set, averaged over all test instances. IV-A2 Baselines We compare DualMLC with thirteen representative methods and organize them into two groups. Sparse, embedding, and label tree methods include AnnexML [10], DiSMEC [7], PfastreXML [11], Parabel [12], eXtremeText [14], Bonsai [13], and XR-Linear [15], covering local output embeddings, sparse label classifiers, and hierarchical routing. Neural text encoder methods include XML-CNN [16], AttentionXML [1], LightXML [3], APLC-XLNet [24], XR-Transformer [2], and MatchXML [4], covering convolutional, recurrent, and pretrained Transformer encoders. IV-A3 Implementation Details By default, both branches truncate or pad inputs to 256 tokens. We average the last four hidden layers of both encoders, apply attention mask aware mean pooling to Qwen2.5-7B, and use the classification token, denoted by CLS, from BERT-base-uncased. A dropout rate of 0.1 is applied before each linear classifier. The 3,584 dimensional Qwen representation is reduced to 896 dimensions, while the 768 dimensional BERT representation is used directly. The Qwen backbone remains frozen and is adapted with LoRA on the query, key, value, and output projections of every attention block. We set the LoRA rank to 16, its scaling parameter to 32, and its dropout rate to 0.05. BERT is fine tuned end to end. We optimize the sum of the two branch losses using AdamW with a weight decay of 0.01. The learning rates for the Qwen LoRA parameters, Qwen classifier, BERT encoder, and BERT classifier are 5×10−55× 10^-5, 5×10−45× 10^-4, 1×10−41× 10^-4, and 2×10−32× 10^-3, respectively. The gradients are clipped to a norm of 1.0. The training and evaluation batch sizes are 2 and 32 per device. Training uses bfloat16 mixed precision and a random seed of 42. We set α=0.6α=0.6, which assigns weights of 0.6 and 0.4 to the Qwen and BERT logits. All experiments are conducted on a server equipped with eight NVIDIA GeForce RTX 4090 GPUs with 24 GB of memory each. IV-B Main Results Table I shows that DualMLC achieves the highest value for all nine dataset and metric combinations. On EURLex-4K, it improves over MatchXML by 0.70%, 0.80%, and 0.52% in P@1, P@3, and P@5. The corresponding gains over MatchXML on Wiki10-31K are 1.48%, 0.66%, and 1.05%. On AmazonCat-13K, where LightXML is the strongest baseline for all three metrics, DualMLC improves P@1, P@3, and P@5 by 0.35%, 0.69%, and 0.68%. The gains are consistent across benchmarks with different label vocabularies and training scales, which indicates that the benefit is not confined to one data regime. The distribution of improvements across ranking depths is also informative. EURLex-4K shows relatively even gains from P@1 to P@5, whereas Wiki10-31K obtains its largest improvement at P@1. This result is notable because Wiki10-31K has the largest label vocabulary among the evaluated datasets, containing 30,938 labels. On AmazonCat-13K, the gains at P@3 and P@5 exceed the gain at P@1, indicating that fusion remains useful as more labels are retained. TABLE I: Comparison of prediction branches on Wiki10-31K. Branch P@1 P@3 P@5 BERT 88.74 77.99 68.13 Qwen 90.02 79.81 70.94 DualMLC 90.78 81.11 71.94 TABLE IV: Effect of the fusion weight on Wiki10-31K. α P@1 P@3 P@5 0.5 90.80 81.00 71.91 0.6 90.78 81.11 71.94 0.7 90.25 80.59 71.68 0.8 89.75 80.39 71.53 IV-C Ablation Study Branch complementarity. Table I shows that Qwen is stronger than BERT when each branch is evaluated alone, with gains of 1.28%, 1.82%, and 2.81% in P@1, P@3, and P@5. DualMLC improves over Qwen by a further 0.76%, 1.30%, and 1.00%. The weaker individual branch therefore contributes label evidence that is not recovered by selecting only the stronger encoder. Moreover, the fusion gains of P@3 and P@5, indicate that the contribution of BERT is not limited to correcting the first prediction. It also improves the ordering of additional relevant labels. IV-D Hyper-parameter Analysis Fusion and pooling. Table IV shows that equal weighting gives the highest P@1 of 90.80%, while α=0.6α=0.6 gives the highest P@3 and P@5 of 81.11% and 71.94%. The difference between α=0.5α=0.5 and α=0.6α=0.6 remains small, whereas performance declines more clearly as the Qwen weight increases to 0.8. The method is therefore stable near balanced fusion but benefits from retaining meaningful evidence from both predictors. This behavior motivates α=0.6α=0.6 as a balanced setting. With mean pooling fixed for Qwen, Table V shows that replacing BERT mean pooling with its classification token representation improves P@1, P@3, and P@5 by 1.04%, 0.79%, and 0.94%. TABLE V: Effect of pooling strategies on Wiki10-31K. Qwen Pool BERT Pool P@1 P@3 P@5 Mean CLS 90.78 81.11 71.94 Mean Mean 89.74 80.32 71.00 TABLE VI: Effect of the number of averaged upper layers on Wiki10-31K. sqs_q sbs_b P@1 P@3 P@5 1 1 90.40 80.80 71.36 1 4 90.36 80.98 71.68 2 2 90.39 80.89 71.97 4 4 90.78 81.11 71.94 6 6 90.66 80.77 71.83 TABLE VII: Effect of input sequence length on Wiki10-31K. TqT_q TbT_b P@1 P@3 P@5 128 128 90.31 80.48 71.43 256 256 90.78 81.11 71.94 512 512 90.36 81.04 72.03 TABLE VIII: Effect of the LoRA rank setting on Wiki10-31K. rLr_L P@1 P@3 P@5 4 90.66 81.04 71.94 8 90.63 80.78 71.97 16 90.78 81.11 71.94 32 90.78 81.04 71.94 Layer aggregation and input length. Table VI shows that averaging the last four layers of both encoders improves P@1, P@3, and P@5 over using only the final layer by 0.38%, 0.31%, and 0.58%. The four layer setting gives the best P@1 and P@3, while averaging two layers gives a marginally higher P@5 by 0.03%. Extending the average to six layers produces no further gain. Table VII shows that a sequence length of 256 provides the best balance across ranking depths. Compared with length 128, it improves the three metrics by 0.47%, 0.63%, and 0.51%. Increasing the length to 512 raises P@5 by only 0.09% and lowers P@1 and P@3, so the added computation does not yield a consistent accuracy benefit. Upper layer averaging combines several semantic abstractions, whereas including too many layers can mix features that are less aligned with the downstream labels. Similarly, a longer sequence preserves more text but raises attention cost and may introduce weakly relevant tokens. The selected four layer and 256 token configuration therefore balances semantic coverage, ranking stability, and computational efficiency. LoRA configuration. Table VIII shows modest variation across the evaluated rank settings. Rank 16 gives the best P@3 and ties for the best P@1, while rank 8 exceeds it in P@5 by only 0.03%. Across ranks from 4 to 32, P@1 varies by 0.15%, P@3 by 0.33%, and P@5 by 0.03%. Since γL _L remains fixed, changing rLr_L also changes the effective scale γL/rL _L/r_L. This table therefore compares practical LoRA settings rather than isolating rank capacity alone. The limited sensitivity indicates that DualMLC does not depend on a narrowly tuned adaptation size. Rank 16 is preferred because it jointly attains the strongest P@1 and P@3 without increasing the trainable parameters to the rank 32 setting. IV-E Discussion The ablation and sensitivity results indicate that DualMLC benefits from complementary representations rather than capacity alone. Causal contextualization emphasizes progressively formed context and generative pretraining knowledge, while bidirectional contextualization evaluates tokens using evidence from both directions. Independent supervision preserves these distinct cues before fusion in a common label space. V Conclusion This work investigated whether heterogeneous language models provide complementary evidence for large-scale multi-label text classification. We introduced DualMLC, which preserves independent representation and prediction pathways and integrates their label scores through late logit fusion. Experiments on three benchmarks have showed consistent improvements across datasets and ranking depths. Branch ablations further showed that the combined predictor outperforms either branch alone, supporting the view that heterogeneous representations capture useful nonredundant evidence. These findings establish representation complementarity as a promising direction for prediction over large label vocabularies. Future work will integrate DualMLC with hierarchical or clustered label indexing to improve efficiency on substantially larger label spaces. Acknowledgments Research was sponsored by the Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-23-2-0224. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. References [1] R. You, Z. Zhang, Z. Wang, S. Dai, H. Mamitsuka, and S. Zhu, “AttentionXML: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification,” in Adv. Neural Inf. Process. Syst., vol. 32, 2019. [2] J. Zhang, W. Chang, H. Yu, and I. S. Dhillon, “Fast multi-resolution transformer fine-tuning for extreme multi-label text classification,” in Adv. Neural Inf. Process. Syst., vol. 34, 2021. [3] T. Jiang, D. Wang, L. Sun, H. Yang, Z. Zhao, and F. Zhuang, “LightXML: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification,” in Proc. AAAI Conf. Artif. Intell., vol. 35, no. 9, 2021, p. 7987–7994. [4] H. Ye, R. Sunderraman, and S. Ji, “MatchXML: An efficient text-label matching framework for extreme multi-label text classification,” IEEE Trans. Knowl. Data Eng., vol. 36, no. 9, p. 4781–4793, Sep. 2024. [5] S. Kharbanda, A. Banerjee, E. Schultheis, and R. Babbar, “CascadeXML: Rethinking Transformers for end-to-end multi-resolution training in extreme multi-label classification,” in Adv. Neural Inf. Process. Syst., vol. 35, 2022. [6] C. Zhou, J. Dong, X. Huang, Z. Liu, K. Zhou, and Z. Xu, “QUEST: Efficient extreme multi-label text classification with large language models on commodity hardware,” in Findings ACL: EMNLP, 2024, p. 3929–3940. [7] R. Babbar and B. Schölkopf, “DiSMEC: Distributed sparse machines for extreme multi-label classification,” in Proc. ACM WSDM, 2017, p. 721–729. [8] I. E.-H. Yen, X. Huang, P. Ravikumar, K. Zhong, and I. S. Dhillon, “PD-Sparse: A primal and dual sparse approach to extreme multiclass and multilabel classification,” in Proc. Int. Conf. Mach. Learn., vol. 48, 2016, p. 3069–3077. [9] K. Bhatia, H. Jain, P. Kar, M. Varma, and P. Jain, “Sparse local embeddings for extreme multi-label classification,” in Adv. Neural Inf. Process. Syst., vol. 28, 2015. [10] Y. Tagami, “AnnexML: Approximate nearest neighbor search for extreme multi-label classification,” in Proc. ACM SIGKDD Conf., 2017, p. 455–464. [11] H. Jain, Y. Prabhu, and M. Varma, “Extreme multi-label loss functions for recommendation, tagging, ranking and other missing label applications,” in Proc. ACM SIGKDD Conf., 2016, p. 935–944. [12] Y. Prabhu, A. Kag, S. Harsola, R. Agrawal, and M. Varma, “Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising,” in Proc. World Wide Web Conf., 2018, p. 993–1002. [13] S. Khandagale, H. Xiao, and R. Babbar, “Bonsai: Diverse and shallow trees for extreme multi-label classification,” Machine Learning, vol. 109, p. 2099–2119, 2020. [14] M. Wydmuch, K. Jasińska, M. Kuznetsov, R. Busa-Fekete, and K. Dembczyński, “A no-regret generalization of hierarchical softmax to extreme multi-label classification,” in Adv. Neural Inf. Process. Syst., vol. 31, 2018, p. 6355–6366. [15] H.-F. Yu, K. Zhong, J. Zhang, W.-C. Chang, and I. S. Dhillon, “PECOS: Prediction for enormous and correlated output spaces,” J. Mach. Learn. Res., vol. 23, no. 98, p. 1–32, 2022. [16] J. Liu, W. Chang, Y. Wu, and Y. Yang, “Deep learning for extreme multi-label text classification,” in Proc. ACM SIGIR Conf., 2017, p. 115–124. [17] W.-C. Chang, H.-F. Yu, K. Zhong, Y. Yang, and I. S. Dhillon, “Taming pretrained Transformers for extreme multi-label text classification,” in Proc. ACM SIGKDD Conf., 2020, p. 3163–3171. [18] G. Xun, K. Jha, J. Sun, and A. Zhang, “Correlation networks for extreme multi-label text classification,” in Proc. ACM SIGKDD Conf., 2020, p. 1074–1082. [19] K. Dahiya, A. Agarwal, D. Saini, Gururaj K, J. Jiao, A. Singh, S. Agarwal, P. Kar, and M. Varma, “SiameseXML: Siamese networks meet extreme classifiers with 100M labels,” in Proc. Int. Conf. Mach. Learn., vol. 139, 2021, p. 2330–2340. [20] E. Chien, J. Zhang, C.-J. Hsieh, J.-Y. Jiang, W.-C. Chang, O. Milenkovic, and H.-F. Yu, “PINA: Leveraging side information in extreme multi-label classification via predicted instance neighborhood aggregation,” in Proc. Int. Conf. Mach. Learn., vol. 202, 2023, p. 5616–5630. [21] Qwen Team, “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024. [22] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional Transformers for language understanding,” in Proc. NAACL, 2019, p. 4171–4186. [23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Int. Conf. Learn. Represent., 2022. [24] H. Ye, Z. Chen, D.-H. Wang, and B. D. Davison, “Pretrained generalized autoregressive model with adaptive probabilistic label clusters for extreme multi-label text classification,” in Proc. Int. Conf. Mach. Learn., vol. 119, 2020, p. 10809–10819. [25] E. Loza Mencía and J. Fürnkranz, “Efficient multilabel classification algorithms for large-scale problems in the legal domain,” in Semantic Processing of Legal Texts, vol. 6036. Springer, 2010, p. 192–215. [26] A. Zubiaga, “Enhancing navigation on Wikipedia with social tags,” in Proc. Wikimania, 2009. [27] J. McAuley and J. Leskovec, “Hidden factors and hidden topics: Understanding rating dimensions with review text,” in Proc. ACM Conf. Recommender Syst., 2013, p. 165–172.