Paper deep dive
HELM: Hierarchical and Explicit Label Modeling with Graph Learning for Multi-Label Image Classification
Marjan Stoimchev, Boshko Koloski, Jurica Levatić, Dragi Kocev, Sašo Džeroski
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/22/2026, 6:27:52 AM
Summary
HELM (Hierarchical and Explicit Label Modeling) is a novel semi-supervised framework for hierarchical multi-label classification (HMLC) in remote sensing. It integrates a Vision Transformer (ViT) with hierarchy-specific class tokens, a Graph Convolutional Network (GCN) for structural dependency modeling, and a BYOL self-supervised branch to leverage unlabeled data, achieving state-of-the-art performance on four remote sensing datasets.
Entities (8)
Relation Signals (4)
HELM → integrates → BYOL
confidence 95% · integrates a self-supervised branch to effectively leverage unlabeled imagery
HELM → utilizes → Vision Transformer
confidence 95% · HELM: (i) uses hierarchy-specific class tokens within a Vision Transformer
HELM → utilizes → Graph Convolutional Networks
confidence 95% · employs graph convolutional networks to explicitly encode the hierarchical structure
HELM → evaluatedon → UCM
confidence 90% · We perform a comprehensive evaluation on four remote sensing image (RSI) datasets (UCM, AID, DFC-15, MLRSNet).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Hierarchical multi-label classification (HMLC) is essential for modeling complex label dependencies in remote sensing. Existing methods, however, struggle with multi-path hierarchies where instances belong to multiple branches, and they rarely exploit unlabeled data. We introduce HELM (\textit{Hierarchical and Explicit Label Modeling}), a novel framework that overcomes these limitations. HELM: (i) uses hierarchy-specific class tokens within a Vision Transformer to capture nuanced label interactions; (ii) employs graph convolutional networks to explicitly encode the hierarchical structure and generate hierarchy-aware embeddings; and (iii) integrates a self-supervised branch to effectively leverage unlabeled imagery. We perform a comprehensive evaluation on four remote sensing image (RSI) datasets (UCM, AID, DFC-15, MLRSNet). HELM achieves state-of-the-art performance, consistently outperforming strong baselines in both supervised and semi-supervised settings, demonstrating particular strength in low-label scenarios.
Tags
Links
- Source: https://arxiv.org/abs/2603.11783v1
- Canonical: https://arxiv.org/abs/2603.11783v1
Trouble viewing inline? Open PDF directly →
Full Text
49,903 characters extracted from source content.
Expand or collapse full text
HELM: Hierarchical and Explicit Label Modeling with Graph Learning for Multi-Label Image Classification Marjan Stoimchev 1,2 Boshko Koloski 1,2 Jurica Levati ́ c 1 Dragi Kocev 1 Sašo Džeroski 1 1 Jožef Stefan Institute, Ljubljana, Slovenia 2 Jožef Stefan International Postgraduate School, Ljubljana, Slovenia marjan.stoimchev, boshko koloski, jurica.levatic@ijs.si dragi.kocev, saso.dzeroski@ijs.si Abstract Hierarchical multi-label classification (HMLC) is essential for modeling complex label dependencies in remote sensing. Existing methods, however, struggle with multi-path hierarchies where instances belong to multiple branches, and they rarely exploit unlabeled data. We introduce HELM (Hierarchical and Explicit Label Modeling), a novel framework that overcomes these limitations. HELM: (i) uses hierarchy-specific class tokens within a Vision Transformer to capture nuanced label interactions; (i) employs graph convolutional networks to explicitly encode the hierarchical structure and generate hierarchy-aware embeddings; and (i) integrates a self-supervised branch to effectively leverage unlabeled imagery. We perform a comprehensive evaluation on four remote sensing image (RSI) datasets (UCM, AID, DFC-15, MLRSNet). HELM achieves state-of-the-art performance, consistently outperforming strong baselines in both supervised and semi-supervised settings, demonstrating particular strength in low-label scenarios. 1 Introduction Hierarchical multi-label classification (HMLC) addresses predictive modeling problems where samples are annotated with multiple labels that are organized in a hierarchy (e.g., a tree or a directed acyclic graph). While incorporating hierarchical information has shown promise in fields like gene function prediction [1,2] and text categorization [3,4], its potential in computer vision, and specifically remote sensing, remains largely unexplored. Limitations. Modern HMLC methods typically translate the hierarchy into the network design [5,6] or embed hierarchical constraints into the loss function [7–9]. However, they face critical limitations: (i) they often assume single-path hierarchies, failing to model realistic multi-path scenarios where images contain multiple object categories across different branches of the hierarchy; (i) they underuse the hierarchy; network-based approaches are computationally heavy, and loss-based formulations often miss long-range dependencies; (i) they focus almost exclusively on supervised learning, ignoring the vast amounts of available unlabeled data. While recent methods have started using Graph Neural Networks (GNNs) [10,11], they remain limited, and semi-supervised learning (SSL) for HMLC in computer vision is practically non-existent. * Correspondence to marjan.stoimchev@ijs.si 1st Workshop on Advances in Representation Learning for Earth Observation, at the European Conference on Neural Information Processing Systems (EurIPS 2025). arXiv:2603.11783v1 [cs.CV] 12 Mar 2026 // Figure 1: High-level overview of HELM. The framework integrates a ViT encoder with hierarchy- specific tokens that feed three distinct branches: (a) classification, (b) graph learning with a GCN, and (c) a BYOL self-supervised branch. The losses from each branch are combined to optimize the model end-to-end. Our Approach. We propose HELM, a novel semi-supervised framework designed to address these limitations. HELM utilizes a multi-branch architecture with three key components: (i) hierarchy- specific class tokens integrated into a Vision Transformer (ViT) encoder to explicitly model label interactions; (i) a graph learning branch that uses Graph Convolutional Networks (GCNs) to model dependencies by propagating information through parent-child relationships; (i) a self-supervised component (BYOL) that leverages unlabeled data to learn robust representations. To our knowledge, HELM is the first semi-supervised HMLC method for images capable of handling complex multi-path hierarchies. Contributions. Our main contributions are: (1) A novel multi-token transformer architecture that integrates graph-based hierarchical reasoning and self-supervised learning for HMLC. (2) Extensive experiments on four real-world remote sensing datasets demonstrating consistent and significant improvements over baselines and state-of-the-art methods. (3) A framework that effectively leverages unlabeled data, achieving substantial performance gains (up to 37%) in low-label regimes, which are common in remote sensing applications [12]. 2 Methodology 2.1 Architecture Overview HELM consists of three branches that are jointly optimized through a composite loss function L = L s +L g +L b (Figure 1). These branches are: (i) a classification branch for discriminative learning on labeled data; (i) a graph learning branch to capture hierarchical dependencies; and (i) a self-supervised branch to leverage unlabeled data. 2.2 Encoder with Hierarchy-Specific Tokens Given an input imagex∈ R C×W×H , we follow the standard ViT process [13], dividing it intoN p patches and projecting them to obtain a sequence of patch tokensT p ∈ R N p ×d . To explicitly model the label structure, we introduceMlearnable, hierarchy-specific CLS tokensT CLS ∈ R M×d , where Mis the total number of labels (leaf and intermediate) in the hierarchy. These tokens serve a dual purpose: they provide the output dimensionality for classification and act as initial node embeddings for the graph learning branch. The tokens are concatenated,T = [T CLS ∥T p ]∈ R (M+N p )×d , and processed through the ViT encoder E(·,θ). Through the self-attention mechanism, the CLS tokens interact with the patch tokens, allowing them to evolve into semantically meaningful embeddings that represent specific labels. 2 2.3 Classification Branch This branch performs supervised learning on labeled data. It aggregates the output hierarchy-specific token embeddings ̃ z CLS ∈ R M×d via average pooling to get a unified representationf CLS . This vector is projected by a fully connected layer to the label space. The supervised loss is computed using binary cross-entropy (H) on a batch of B l labeled samples: L s = 1 B l B l X i=1 H(y i ,p s (y|x i ))(1) 2.4 Graph Learning Branch To model label dependencies, we construct a directed graphG = (V,E)from the label hierarchy. The hierarchy-specific CLS tokens ̃ z CLS serve as initial node features. A GraphSAGE [14] operatorφ(·)is applied to propagate information and generate structure-aware embeddings: ̃ z g = φ( ̃ z CLS ;G). These are then pooled and projected to get predictions. This branch processes the entire batch (labeled and unlabeled), but the loss is computed only on labeled samples, enabling a semi-supervised flow of information through the graph structure: L g = 1 B l B l X i=1 H(y i ,p g (y|x i ))(2) 2.5 Self-Supervised Branch We integrate Bootstrap Your Own Latent (BYOL) [15] to leverage unlabeled data. This branch operates on the entire batch in a label-agnostic manner. For each image, two augmented views are created. An online network (sharing the main encoder’s weightsθ) is trained to predict the representation of the same image from a target network (with weightsξupdated via an exponential moving average ofθ). The loss encourages similarity between the predictions of the online network and the projections of the target network: L b = 2− 2· ⟨q θ (g θ (F p )),g ξ (F ′ p )⟩ ∥q θ (g θ (F p ))∥·∥g ξ (F ′ p )∥ (3) whereF p andF ′ p are representations from the two views,gis a projection head, andqis a predictor network. 3 Results We evaluate HELM on four public remote sensing datasets: UCM [16], AID [17], DFC-15 [18], and MLRSNet [19]. These datasets cover a wide range of scene types and hierarchical complexities. Detailed dataset statistics, hierarchy construction, implementation settings, and baseline descriptions are provided in Appendix A.1.1–A.1.7. Supervised Results.Table 1 shows the supervised results of HELM and its variants. Incorporating hierarchical structure already improves performance, as HMLC surpasses the flat MLC baseline on all datasets. Adding graph reasoning (HELM g ) yields further gains, particularly on UCM and DFC-15, confirming the benefit of modeling label dependencies. HELM b performs best on AID and MLRSNet, indicating that utilizing unlabeled data aids generalization. The complete HELM model achieves the overall best or second-best results, with the highest AUPRCon UCM (0.904) and the lowest Ranking Loss across all datasets (0.022, 0.017, 0.006, 0.024). Semi-Supervised Results.Figure 2 shows the semi-supervised performance for different amounts of labeled data (1%, 5%, 10%, and 25%). HELM-SSL consistently surpasses both its supervised variant (HELM-SL) and the supervised HMLC baseline, with the largest AUPRCimprovements in the lowest label settings. At 1% supervision, HELM achieves gains of 25.0% on UCM, 6.6% on AID, 37.0% on DFC-15, and 18.5% on MLRSNet. Complete semi-supervised tables for all datasets and model variants are provided in Appendix A.2. 3 Table 1: Performance comparison with loss components. Best in bold, second-best underlined. Loss ComponentsAUPRC (↑)Ranking Loss (↓) MethodL s L g L b UCMAIDDFC-15MLRSNetUCMAIDDFC-15MLRSNet MLC✓0.8630.7670.9670.8380.0310.0250.0100.039 HMLC✓0.8900.8270.9710.8630.0310.0210.0080.027 HELM g ✓0.8990.8420.9790.8690.0240.0190.0070.025 HELM b ✓0.8850.8520.9690.8730.0290.0190.0120.025 HELM✓0.9040.8490.9770.8710.0220.0170.0060.024 151025 N(%) 0.256 0.453 0.651 0.848 AU PRC UCM 151025 N(%) 0.332 0.469 0.607 0.744 AID 151025 N(%) 0.436 0.614 0.791 0.969 DFC-15 151025 N(%) 0.417 0.557 0.697 0.837 MLRSNet Method:HMLC BaselineHELM-SLHELM-SSL 151025 N(%) 0.033 0.137 0.240 0.344 RL UCM 151025 N(%) 0.025 0.078 0.132 0.185 AID 151025 N(%) 0.008 0.106 0.205 0.303 DFC-15 151025 N(%) 0.027 0.044 0.062 0.079 MLRSNet Method:HMLC BaselineHELM-SLHELM-SSL Figure 2: Semi-supervised results with different labeled proportions. HELM consistently outperforms the supervised baseline, with the largest gains at 1–5% labeled data. Full results are available in Appendix A.2. Comparison with State of the Art.Table 2 compares HELM with established hierarchical multi- label classification methods: C-HMCNN [8], HiMulConE [9], and HMI [20]. HELM achieves the highest AUPRCand the lowest Ranking Loss on every dataset. It improves upon HiMulConE by 7.2% on UCM and 10.3% on AID in AUPRC, while reducing Ranking Loss by 29.0% on UCM and 15.0% on AID. These consistent improvements confirm that hierarchy-specific tokens combined with graph-based reasoning significantly enhance label consistency and predictive accuracy. Table 2: Comparison with state-of-the-art HMLC methods in the supervised setting. Best in bold, second-best underlined. AUPRC (↑)Ranking Loss (↓) MethodUCMAIDDFC-15MLRSNetUCMAIDDFC-15MLRSNet C-HMCNN [8]0.8340.7640.9620.7920.0380.0240.0120.041 HiMulConE [9]0.8430.7700.9700.8650.0310.0200.0060.035 HMI [20]0.6610.6470.9230.4370.0800.0730.0430.138 HELM (Ours)0.9040.8490.9770.8710.0220.0170.0060.025 4 Discussion We introduced HELM, a novel semi-supervised framework for HMLC that combines hierarchy- specific tokens, graph-based structure encoding, and self-supervised representation learning. The experiments show that HELM consistently improves over strong baselines and achieves state-of-the- art performance on four RSI datasets. By modeling multi-path hierarchies and leveraging unlabeled data, it provides substantial gains, particularly in low-label regimes. The effectiveness of HELM arises from three complementary design choices. First, hierarchy-specific CLS tokens allow the model to represent each label explicitly and capture fine-grained relationships through self-attention. Second, the GCN propagates information across parent-child links, enriching label embeddings with structural context. Third, the BYOL branch exploits unlabeled imagery to learn generalizable visual features that strengthen the model under limited supervision. As shown in the Appendix, the graph module adds minimal computational overhead while yield- ing clear performance benefits. The BYOL branch increases training cost but provides important improvements when labeled data are scarce, which is often the case in remote sensing. Future work will focus on automatic hierarchy discovery to remove manual construction, initialization of hierarchy-specific tokens using vision-language models, and extensions to multi-modal inputs such as SAR or multispectral imagery to improve generalization and applicability. 4 Acknowledgments We acknowledge the financial support of the Slovenian Research and Innovation Agency (ARIS) through the core research programme P2-0103 (Knowledge Technologies), projects J1-3033, J2-2505, J2-4452, J2-4660, J3-3070, J4-3095, J5-4575, J7-4636, J7-4637, and N2-0236. The work of the BK was supported by the Young Researcher grant PR-12394. References [1]Ping Fu Shou Feng and Wenbin Zheng. A hierarchical multi-label classification method based on neural networks for gene function prediction. Biotechnology & Biotechnological Equipment, 32(6):1613–1621, 2018. [2]Leander Schietgat, Celine Vens, Jan Struyf, Hendrik Blockeel, Dragi Kocev, and Sašo Džeroski. Predicting gene function using hierarchical multi-label decision tree ensembles. BMC bioinfor- matics, 11:1–14, 2010. [3]Jibing Gong, Zhiyong Teng, Qi Teng, Hekai Zhang, Linfeng Du, Shuai Chen, Md Zakirul Alam Bhuiyan, Jianhua Li, Mingsheng Liu, and Hongyuan Ma. Hierarchical graph transformer-based deep learning model for large-scale multi-label text classification. IEEE Access, 8:30885–30896, 2020. doi: 10.1109/ACCESS.2020.2972751. [4] Hao Peng, Jianxin Li, Senzhang Wang, Lihong Wang, Qiran Gong, Renyu Yang, Bo Li, Philip S. Yu, and Lifang He. Hierarchical taxonomy-aware and attentional graph capsule rcnns for large- scale multi-label text classification. IEEE Transactions on Knowledge and Data Engineering, 33(6):2505–2519, 2021. doi: 10.1109/TKDE.2019.2959991. [5]Xinqi Zhu and Michael Bain. B-cnn: branch convolutional neural network for hierarchical classification. arXiv preprint arXiv:1709.09890, 2017. [6]Brendan Kolisnik, Isaac Hogan, and Farhana Zulkernine. Condition-CNN: A hierarchical multi-label fashion image classification model. Expert Systems with Applications, 182:115195, 2021. [7] Jingzhou Chen, Peng Wang, Jian Liu, and Yuntao Qian. Label relation graphs enhanced hierarchical residual network for hierarchical multi-granularity classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4858–4867, 2022. [8] Eleonora Giunchiglia and Thomas Lukasiewicz. Coherent hierarchical multi-label classification networks. Advances in neural information processing systems, 33:9662–9673, 2020. [9]Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hierarchical multi-label contrastive learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16660–16669, June 2022. [10] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5177–5186, 2019. [11]Ziwen Lan, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama. Hierarchical multi-label attribute classification with graph convolutional networks on anime illustration. IEEE Access, 11:35447–35456, 2023. [12]Anastasiia Safonova, Gohar Ghazaryan, Stefan Stiller, Magdalena Main-Knorn, Claas Nendel, and Masahiro Ryo. Ten deep learning techniques to address small data problems with remote sensing. International Journal of Applied Earth Observation and Geoinformation, 125:103569, 2023. [13]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy. [14] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017. 5 [15]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020. [16] B. Chaudhuri, B. Demir, S. Chaudhuri, and L. Bruzzone. "Multilabel Remote Sensing Image Retrieval Using a Semisupervised Graph-Theoretic Method. IEEE Transactions on Geoscience and Remote Sensing, 56(2):1144–1158, 2018. [17]Y. Hua, L. Mou, and X.X. Zhu. Relation Network for Multi-label Aerial Image Classification. IEEE Transactions on Geoscience and Remote Sensing, 2019. [18]Y. Hua, L. Mou, and X.X. Zhu. Recurrently exploring class-wise attention in a hybrid con- volutional and bidirectional LSTM network for multi-label aerial image classification. ISPRS Journal of Photogrammetry and Remote Sensing, 149:188–199, 2019. [19]Q. Xiaoman Qi, Z. Panpan, W. Yuebin, Z. Liqiang, P. Junhuan, W. Mengfan, C. Jialong, Z. Xudong, Z. Ning, and P.M.Takis. MLRSNet: A multi-label high spatial resolution remote sensing dataset for semantic scene understanding. ISPRS Journal of Photogrammetry and Remote Sensing, 169:337–350, 2020. [20]Bo Xiong, Michael Cochez, Mojtaba Nayyeri, and Steffen Staab. Hyperbolic embedding infer- ence for structured multi-label prediction. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 33016–33028. Curran Associates, Inc., 2022. [21]Y. Yang and S. Newsam. Bag-of-visual-words and spatial extensions for land-use classification. In 18th ACM SIGSPATIAL International Symposium on Advances in Geographic Information Systems, pages 270–279, 01 2010. [22]G-S Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y. Zhong, X. Lu, and L. Zhang. AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification. IEEE Transactions on Geoscience and Remote Sensing, 55:3965 – 3981, 02 2017. [23]Referans Copernicus. Corine land cover. Copernicus Land Monitoring Service. L. Monit. Serv, 2018. [24] Peng Xia, Xingtong Yu, Ming Hu, Lie Ju, Zhiyong Wang, Peibo Duan, and Zongyuan Ge. Hgclip: Exploring vision-language models with graph representations for hierarchical under- standing. In Proceedings of the 31st International Conference on Computational Linguistics, pages 269–280, 2025. [25]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. A Appendix A.1 Experimental Settings A.1.1 Datasets We evaluate HELM across four public remote sensing image datasets: UCM [16,21] (2.1k images, 17 leaf labels), AID [17,22] (3k, 17), DFC-15 [18] (3.3k, 8), and MLRSNet [19] (109k, 60). Table 3 reports train/test splits, hierarchy sizes, and level-wise cardinality and density. UCM. The UC Merced dataset contains 2,100 aerial RGB images at256× 256with 17 object- level (leaf) labels [16]. We construct a hierarchical extension by mapping the original categories to CORINE-inspired levels, yielding 30 labels across three levels (|L| = 4, 9, 17; totalh=30, leaf ℓ=17). This introduces intermediate abstractions while preserving the original leaf space. AID. The Aerial Image Dataset comprises 3,000 images at600 × 600with 17 leaf labels [17]. We construct a hierarchical version with the same depth and level sizes as UC Merced, resulting in 30 labels across three levels (|L| = 4, 9, 17;h=30,ℓ=17). Relative to UC Merced, AID 6 Table 3: Comprehensive statistics for remote sensing datasets.Ndenotes total images;N train andN test represent training and test set sizes.|L|indicates the number of unique labels at each hierarchical level (1 to 4), whereℓcorresponds to the leaf level andhrefers to the total number of labels across all levels.Card@Level is the average number of active labels per image at that level (label cardinality), and Dens@Level is the mean per-label prevalence at that level (label density). N train N test |L|Card@LevelDens@Level Dataset 1 2 3 4 ℓ h1 2 3 4ℓh1 2 3 4ℓh UCM2,1001,6674334917-17301.74 3.03 3.19-3.197.960.44 0.34 0.19-0.19 0.27 AID3,0002,400600 4917-17302.18 4.51 4.93-4.93 11.620.54 0.50 0.29-0.29 0.39 DFC-153,3412,672669378-8171.61 2.73 2.80-2.806.830.54 0.39 0.35-0.35 0.40 MLRSNet 109,151 87,33621,8157 15 22 60 60 1042.21 3.62 4.41 6.10 6.10 15.250.32 0.24 0.20 0.10 0.10 0.16 exhibits higher label co-occurrence and per-label prevalence, providing a richer supervisory signal for hierarchical learning. DFC-15. The 2015 IEEE GRSS Data Fusion Contest dataset provides 3,341 image patches at 600× 600with 8 leaf categories [18]. From the original semantic categories, we construct a compact three-level hierarchy totaling 17 labels (|L| =3, 7, 8;h=17,ℓ=8). Although it contains fewer leaf classes than the other benchmarks, its leaf-level prevalence is highest, yielding dense supervision per image. MLRSNet. MLRSNet includes 109,151 images at256× 256annotated with 60 leaf categories [19]. We construct the deepest hierarchical extension among the four datasets, with 104 labels across four levels (|L| =7, 15, 22, 60;h=104,ℓ=60). Images typically carry many active leaf labels, while each individual leaf label is comparatively rare across the corpus, reflecting broad coverage over numerous fine-grained categories and stressing scalability. In all cases, the hierarchical versions are constructed from the original multi-label datasets to introduce intermediate semantic levels while retaining the original leaf label sets. A.1.2 Hierarchy Construction All hierarchical structures were constructed by systematically mapping classes to the CORINE Land Cover nomenclature (CLC) [23], which provides well-established relationships between land cover types across multiple levels. This enables meaningful hierarchical structures while preserving the multi-label nature at the leaf level. When direct CLC mapping was infeasible, we used ChatGPT- assisted mapping followed by manual validation [24]. Figure 3 presents a representative example of the constructed hierarchical structure for the UCM dataset. Root Artificial Surfaces Urban Fabric buildingsmobile-home Industrial Commercial and Transport Units airplanecarscourtdockship storage tanks Road and Rail Networks and Associated Land pavement Mine Dump and Construction Sites bare-soil Agricultural Areas Arable Land field Forest and Semi-Natural Areas Forests trees Shrub and/or Herbaceous Vegetation Associations chaparralgrass Water Bodies Inland Waters water Marine Waters seasand Figure 3: Example of constructed label hierarchy for the UCM dataset, derived from the CORINE Land Cover nomenclature. The hierarchy demonstrates the 3-level structure with 4 top-level cate- gories, 9 intermediate-level categories, and 17 leaf-level labels. A.1.3 Implementation Details Our HELM framework is implemented inPyTorch Lightning, leveraging the base variant of the Vision Transformer (ViT-B/16) with a patch size of16× 16pixels. The ViT backbone is initialized https://github.com/Lightning-AI/pytorch-lightning 7 with pre-trained ImageNet weights to provide strong visual priors. For the graph learning component, we employPyTorch Geometric (PyG)with a two-layer GraphSAGE network, enabling efficient message passing within the label hierarchy. The self-supervised BYOL component is integrated using LightlySSL, a specialized library for self-supervised learning in computer vision. Models are trained on RGB images resized to 224× 224 pixels using the unified learning objective. The hierarchy-specific CLS tokens produce latent representations of dimensiond = 768after average pooling. For the supervised components (classification and graph branches), we apply weak augmentations, including horizontal and vertical flips and Gaussian blur. For the BYOL component, we employ both weak and strong augmentations, with the latter incorporating color jittering, affine transformations, random cropping, and random erasing to strengthen representation learning. Training is conducted for 100 epochs using the AdamW optimizer with a base learning rate of 1× 10 −4 , cosine annealing scheduler, and a mini-batch size of 16. Mixed-precision training is utilized to accelerate computation and reduce memory footprint. All experiments are performed on four NVIDIA A100 GPUs (40 GB each). To ensure full reproducibility, all hierarchical label structures constructed for each dataset will be released as YAML configuration files, providing the exact parent–child mappings used in the graph module. These YAML files, together with the training scripts and configuration examples, will be included in the public code release accompanying the final version of this paper. A.1.4 Evaluation Strategy In the semi-supervised setting, we vary the labeled ratio in 1, 5, 10, 25%, using the remainder as unlabeled. For each configuration, we perform three independent runs with different random seeds to ensure statistical reliability. Performance is evaluated on a fixed test set using exclusively leaf labels, regardless of the training configuration. This ensures fair comparison across all methods, including those that use hierarchical information during training (HMLC, HELM variants) and those that do not (MLC baseline). We report AUPRC and ranking loss, averaged over the three independent runs. A.1.5 Computational Resources All experiments were conducted on four NVIDIA A100 GPUs equipped with 40 GB memory each. A.1.6 Evaluation Metrics We employ two performance metrics to evaluate the effectiveness of the methods: the average area under the precision-recall curve (AUPRC) and the ranking loss measure. Given that the tasks involve multi-label classification and hierarchical multi-label classification, we use a variant of AUPRC–the area under the micro-averaged Precision-Recall curve (AUPRC). These metrics are chosen because they are independent of classification thresholds and provide a reliable assessment of performance. In this context,Precrepresents the proportion of predicted labels that are correct, whileReccorre- sponds to the proportion of actual labels in the dataset that are correctly predicted. These values are calculated as follows: Prec = P i TP i P i TP i + P i FP i ,(4) Rec = P i TP i P i TP i + P i FN i ,(5) whereiiterates over all classes. By varying the decision threshold, an average Precision-Recall (PR) curve is generated. Ranking loss (RL), measuring the average fraction of incorrectly ordered label pairs for a given example, is defined as https://pytorch-geometric.readthedocs.io/en/latest/ https://github.com/lightly-ai/lightly 8 RL = 1 N N X i=1 1 |Y + i |·|Y − i | X y p ∈Y + i X y n ∈Y − i I(f i (y p )≤ f i (y n )),(6) whereNis the number of examples,Y + i andY − i are the sets of positive and negative labels for the i-th example,f i (y)is the predicted score for labely, andI(·)is the indicator function, which returns 1 if the condition is true and 0 otherwise. The loss is weighted by the sizes ofY + i andY − i , penalizing each pair of misordered labels equally. A lower ranking loss indicates better performance, as fewer label pairs are incorrectly ranked. To compute these metrics, we use the readily available implementations from scikit-learn [25]. A.1.7 Compared Methods HELM Variants and Baseline Methods. To systematically evaluate the contribution of each com- ponent, we analyze different variants of HELM by selectively including loss terms from the overall objective. Table 4 summarizes all evaluated configurations and corresponding loss components. Table 4: HELM variants and baseline methods with their loss component configurations for systematic evaluation. Method VariantNotation L s L g L b Task MLC BaselineMLC✓SL HMLC BaselineHMLC✓SL HELM (Graph-only)HELM g ✓SL, SSL HELM (BYOL-only)HELM b ✓SL, SSL HELM (Full)HELM✓SL, SSL We establish two baseline methods to isolate the impact of hierarchical information: the MLC baseline, which uses only leaf labels during training and performs standard multi-label classification, and the HMLC baseline, which incorporates the complete hierarchical label set during training but treats all labels independently without exploiting structural relationships (essentially a flat approach to hierarchical classification). Both baselines utilize only the supervised classification lossL s , omitting the graph and self-supervised components. The HELM variants systematically introduce additional components to assess their individual and combined contributions. HELM g adds graph-based dependency modeling to capture hierarchical label relationships, while HELM b incorporates self-supervised learning through BYOL to leverage unlabeled data. The complete HELM model integrates all three components, providing a unified framework that combines supervised classification, hierarchical reasoning, and self-supervised representation learning. This experimental design enables precise measurement of how each component contributes to performance, particularly in scenarios with limited labeled supervision. All methods are evaluated using exclusively leaf labels of testing examples to ensure fair comparison, regardless of training configuration. State-of-the-Art Methods. We also compare HELM against three state-of-the-art HMLC methods: C-HMCNN [8], HiMulConE [9], and HMI [20]. These methods represent a diverse range of approaches to hierarchical multi-label classification, highlighting the strengths and limitations of current techniques. C-HMCNN [8] ensures consistency between parent and child predictions through a modified binary cross-entropy (BCE) loss, providing flexibility in handling complex hierarchies. Although originally designed for tabular data, we adapted it for image-based tasks by replacing its tabular encoder with our vision transformer encoder. HiMulConE [9] adopts a two-stage approach: it first employs a hierarchy-preserving contrastive loss to learn label-aware embeddings and then trains a classifier on leaf nodes using cross-entropy and softmax. Since the original method utilizes a cross-entropy loss designed for single-label classification at the leaf level, we replaced it with a BCE loss to support scenarios where multiple labels may exist at the leaf level. HMI [20] utilizes a hyperbolic 9 Poincaré ball model to encode logical relationships, such as implication and exclusion, through geometric constraints. This approach achieves high consistency with fewer dimensions, enhancing computational efficiency. However, similar to C-HMCNN, HMI was originally designed for tabular data and we adapted it for image-based tasks by incorporating our vision transformer encoder to enable a direct comparison. For a fair evaluation, all methods are implemented using our vision encoder, and we utilize the official code repositories of each method, setting hyperparameters according to the original authors’ recommendations. A.2 Complete Semi-Supervised Learning Results Table 5 presents comprehensive semi-supervised learning results across all labeled data proportions for each dataset. The results demonstrate HELM’s effectiveness in leveraging unlabeled data, with particularly strong performance gains in low-label scenarios. Table 5: Evaluation of HELM’s individual loss components in the semi-supervised learning setting compared to the supervised HMLC baseline. Models are trained with varying proportions of labeled data and evaluated on a fixed test set. Results are reported using AUPRC(higher is better) and Ranking Loss (lower is better), averaged over three runs. The best performance for each dataset and labeled data proportion is highlighted in bold. AUPRC(↑)UCMAIDDFC-15MLRSNetAvg. ranks N (%)N (%)N (%)N (%)N (%) Methods151025151025151025151025151025 HMLC Baseline0.2680.3570.4790.6910.3660.4100.3820.5640.4810.6190.6920.7660.4700.6430.7160.7873.33.83.53.8 HELM b 0.2900.5540.6560.7700.3620.4610.5570.6710.5220.7170.8240.9090.3800.6430.7090.7883.33.03.02.8 HELM g 0.2690.5320.7140.7990.3750.4850.5970.6800.6190.7920.8750.9160.4220.6570.7070.7812.52.02.52.5 HELM0.3350.6510.7300.8070.3900.5330.6310.7010.6590.7900.8850.9340.5570.6660.7430.8161.01.31.01.0 RL(↓)UCMAIDDFC-15MLRSNetAvg. ranks Methods151025151025151025151025151025 HMLC Baseline0.2660.2320.1780.0800.1690.1220.1370.0620.2320.1640.1010.0880.0740.0480.0420.0333.03.83.83.5 HELM b 0.2470.1250.0860.0570.2160.0880.0590.0410.1790.0920.0580.0330.0980.0510.0430.0342.32.52.52.8 HELM g 0.3070.1270.0680.0500.1550.0760.0560.0390.1370.0670.0390.0310.0840.0430.0410.0333.82.32.32.0 HELM0.1370.1060.0720.0510.1330.0710.0480.0330.1320.0650.0370.0230.0610.0410.0360.0301.01.31.31.3 The complete results demonstrate that HELM consistently outperforms both the HMLC baseline and ablated variants across all datasets and labeled data proportions. The performance gains are particularly pronounced in extremely low-resource scenarios at 1-5% labeled data. At 1% labeled data, HELM achieves substantial AUPRCimprovements of 25.0% on UCM, 6.6% on AID, 37.0% on DFC-15, and 18.5% on MLRSNet compared to the HMLC baseline. As the proportion of labeled data increases, the performance gap narrows but remains consistently favorable for HELM across both metrics, demonstrating that the self-supervised component continues to provide benefits even with more abundant supervision. A.2.1 Computational Efficiency Analysis Figure 4 illustrates the computational trade-offs inherent in our framework design, evaluated on the representative UCM dataset. The graph learning component proves remarkably efficient, adding only 107K parameters while delivering substantial performance gains. In contrast, the BYOL component introduces significant computational overhead due to its dual-encoder architecture. A.3 Qualitative Evaluation This section presents a qualitative evaluation of HELM compared to state-of-the-art methods using the UCM dataset. A.3.1 Visualization of Learned Embeddings The embeddings are derived from the hierarchy-specific CLS tokens produced by the final layer of the encoder. Since our encoder serves as a common backbone architecture across all methods, the comparison between learned embeddings is both fair and direct, ensuring that differences arise solely 10 100012001400 0.86 0.87 0.88 0.89 0.90 0.91 AU PRC 86.7M 86.8M 86.9M 350040004500 176.0M 176.0M Runtime (s) Method MLC HMLC HELM ( g ) HELM ( b ) HELM Figure 4: Comparison of training times, performance, and parameter counts for the baseline methods and HELM on the UCM dataset. The size of each bubble corresponds to the number of parameters (in millions). from the design and learning mechanisms of the methods. We use Uniform Manifold Approximation and Projection (UMAP) to visualize the high-dimensional embeddings. To quantify embedding quality, we use Normalized Mutual Information (NMI), which evaluates the alignment between clusters formed by embeddings and ground-truth labels. Before computing NMI, we applyk-Nearest Neighbors clustering on the original embeddings, with the number of clusters set to match the number of labels at each hierarchical level. We report the average NMI calculated by averaging NMI values across different levels of the hierarchy. This approach provides a comprehensive evaluation of both fine-grained (leaf-level) and broader (ancestor-level) clustering quality. Figure 5 presents the UMAP analysis results across the hierarchical levels of the UCM dataset. HELM with hierarchical modeling delivers the best results, achieving NMI values of 0.411 at the first level and 0.801 at the second level of the hierarchy, with an average NMI ofNMI = 0.737across all levels. The embeddings produced are well-structured and form distinct clusters that align with the hierarchical label relationships, demonstrating the effectiveness of explicitly modeling hierarchical dependencies. This approach allows the model to capture both coarse-grained and fine-grained relationships, resulting in superior clustering quality and predictive performance. In contrast, the HMLC baseline achievesNMIof 0.612, placing it as the second-lowest performing method in the comparison. This highlights the significant limitations of excluding hierarchical relationships, which restrict the model’s ability to organize embeddings meaningfully. While HiMulConE delivers competitive results, it falls short of capturing the hierarchical structure as effectively as HELM with hierarchical modeling. A.4 Limitations While HELM outperforms state-of-the-art models, our approach has limitations. The BYOL branch is computationally intensive, significantly increasing runtime and parameter count due to its dual- encoder architecture. This computational cost is justified by the enhanced representation quality and improved generalization capabilities observed across our experimental evaluation, particularly in low-label scenarios where annotation costs are prohibitive. As shown in the UMAP analysis (Figure 5), while HELM outperforms state-of-the-art methods and generates well-defined clusters that align with the hierarchical structure of labels, it occasionally fails to consistently cluster hierarchical levels. For instance, labels such as "sand" and "water", which share the same parent label "Water Bodies" at higher levels of the hierarchy, are well-clustered individually but fail to fully reflect their shared broader grouping. This suggests that although HELM excels at modeling fine-grained relationships, there is room for improvement in capturing and representing broader hierarchical structures more consistently. 11 UMAP 2 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water C HMCNN NMI = 0.149 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water HiMulConE NMI = 0.374 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water HMI NMI = 0.267 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water HMLC baseline NMI = 0.254 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water HELM NMI = 0.411 UMAP 2 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 0.633 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 0.704 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 0.673 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 0.582 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 0.801 UMAP 1 UMAP 2 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 1.0 UMAP 1 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 1.0 UMAP 1 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 1.0 UMAP 1 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 1.0 UMAP 1 airplane bare-soil buildings cars chaparral court dock field grass mobile-home pavement sand sea ship storage tanks trees water NMI = 1.0 Level 1 Agricultural Areas Artificial Surfaces Forest and Semi-Natural Areas Water Bodies Level 2 Arable Land Urban Fabric Industrial, Commercial, and Transport Units Road and Rail Networks and Associated Land Mine, Dump, and Construction Sites Forests Shrub and/or Herbaceous Vegetation Associations Inland Waters Marine Waters Level 3 field buildings mobile-home airplane cars court dock ship storage tanks pavement bare-soil trees chaparral grass water sea sand NMI =0.594 NMI =0.693 NMI =0.647 NMI =0.612 NMI =0.737 Figure 5: Comparison of 2-D UMAP embeddings between HELM and state-of-the-art methods for the UCM dataset. The learned embeddings are colored based on different levels of the UCM label hierarchy. The visualization is based on embeddings corresponding to leaf labels, while the color coding reflects the grouping and relationships at each hierarchical level. The NMI values are reported for each method, where higher values indicate better alignment between clusters and ground truth labels, reflecting the quality of hierarchical embeddings. EurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and Introduction clearly describe the three main components of HELM: hierarchy-specific tokens, graph-based reasoning, and self-supervised learning. They also state the expected improvements in both supervised and semi-supervised settings. These claims are directly supported by the experimental results shown in Table 1 and Table 2, as well as by the learning curves in Figure 2, which confirm the accuracy and scope of the stated contributions. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper explicitly discusses both computational and methodological limi- tations. Appendix A.2.1 presents a quantitative analysis of computational efficiency, high- lighting the added cost of the BYOL branch and the minimal overhead of the graph module. The dedicated Limitations subsection further reflects on representational shortcomings in capturing high-level hierarchical groupings and the increased runtime associated with the dual-encoder design (see Fig. 5). These discussions provide a balanced view of HELM’s trade-offs in performance and scalability. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper is empirical and methodological without formal theorems or proofs. 4. Experimental result reproducibility 12 Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: All experimental details required for reproduction are provided. The datasets, hierarchy construction process, model configurations, training setup, metrics, and evaluation protocols are described in Appendix A.1.1–A.1.7, A.1.3, and A.1.4, with model variants summarized in Table 4. To ensure full reproducibility, the exact hierarchical label structures used for each dataset will be released as YAML configuration files alongside the code. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: Public datasets are cited and described (Appendix A.1.1), but an anonymized code release and scripts are not included in the current submission. We plan to release code and detailed run scripts upon acceptance. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Implementation and training details, augmentations, optimizer, schedules, batch sizes, epochs, and evaluation setup are provided in Appendix A.1.3 and A.1.4. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report averages and standard deviations over three independent runs with different random seeds in the learning curves (Fig. 2), where the variability bands make the stability of the methods clear. All tables report mean values only, while per-run results and standard deviations are omitted for brevity but are reflected in the learning curves. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide details on the computational setup (on single A100 40GB GPU) and report a dedicated computational efficiency analysis in Appendix A.2.1 and Figure 4. The analysis quantifies runtime, parameter counts, and trade-offs across model variants on the UCM dataset. For simplicity, runtime profiling is presented on a single dataset, but the relative scaling trends remain consistent across all benchmarks. Training times are comparable between runs, with the graph branch adding negligible overhead and the BYOL component introducing a predictable increase in runtime due to its dual-encoder structure. 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: The work uses public remote sensing datasets with citations, no sensitive personal data, and follows standard evaluation practices. No violations of the Code of Ethics were identified. 13 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [No] Justification: While we do not include a dedicated broader impacts section, the paper discusses potential positive implications in environmental monitoring, urban planning, and climate assessment within the Introduction and Discussion. Given the foundational and non-sensitive nature of this research, no direct negative societal impacts are anticipated. 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The work does not release high-risk foundation models or scraped web datasets. It evaluates task models on public benchmarks. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [No] Justification: All datasets and prior methods used in this work are properly cited in the main text and appendix. However, the specific dataset and code licenses are not explicitly listed in the current version for brevity. We will include the corresponding license information for each asset in the camera-ready version to ensure full compliance and transparency. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not introduce new datasets or public model checkpoints at the time of submission. Upon acceptance, all configuration files, hierarchical mappings, and trained model weights will be documented and released to support full reproducibility. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: The research does not involve crowdsourcing or human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: The research does not involve human subjects and therefore does not require IRB review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Answer: [Yes] 14 Justification: We used ChatGPT in two limited and fully disclosed ways: (i) to assist in hier- archy mapping for classes without direct CORINE Land Cover alignment (Appendix A.1.2), and (i) for light paraphrasing and linguistic refinement of manuscript text to improve clarity and readability. The model was not involved in designing methods, conducting experiments, or interpreting results. 15