Paper deep dive
ReLTEx: Reliable LLM-based Taxonomy Expansion
Zeinab Ghamlouch, Mehwish Alam
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/16/2026, 3:38:47 AM
Summary
The paper introduces ReLTEx, a framework for reliable Large Language Model (LLM)-based taxonomy expansion. It addresses issues of noise, redundancy, and hierarchical inconsistency in LLM-generated taxonomies by combining zero-shot candidate generation with structure-aware validation using a path-aware classifier and recursive expansion control. The framework is evaluated on benchmark taxonomies (SemEval-2016, Schema.org) using masked expansion settings, demonstrating improved reliability and semantic coherence compared to existing methods.
Entities (16)
Relation Signals (14)
Zeinab Ghamlouch â affiliatedwith â TĂ©lĂ©com Paris
confidence 95% · Zeinab Ghamlouch Affiliation: Télécom Paris
Mehwish Alam â affiliatedwith â TĂ©lĂ©com Paris
confidence 95% · Mehwish Alam Affiliation: Télécom Paris
ReLTEx â solves â Taxonomy Expansion
confidence 95% · we present ReLTEx, a framework for reliable LLM-based taxonomy expansion
ReLTEx â uses â Large Language Models
confidence 95% · ReLTEx combines zero-shot LLM-based candidate generation with structure-aware validation
ReLTEx â evaluatedon â SemEval-2016 Task 13
confidence 90% · We evaluate ReLTEx on two taxonomy benchmarks... SemEval-2016 Task 13 Environment taxonomy
ReLTEx â evaluatedon â Schema.org
confidence 90% · We evaluate ReLTEx on two taxonomy benchmarks... Schema.org
ReLTEx â evaluatedwithmetric â Wu & Palmer Similarity
confidence 90% · Wu & Palmer (WuP) Similarity... To evaluate the structural quality of node generation
ReLTEx â evaluatedwithmetric â Recall@k
confidence 90% · Our evaluation builds upon metrics... Recall@K... We define the set of locally recovered hidden concepts
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in Large Language Models (LLMs) have demonstrated strong capabilities in generating semantically relevant concepts and relations, making them promising tools for taxonomy enrichment. However, directly relying on LLM-generated expansions often leads to noisy, redundant, or hierarchically inconsistent structures, limiting their reliability for automated taxonomy expansion. In this paper, we present ReLTEx, a framework for reliable LLM-based taxonomy expansion. ReLTEx combines LLM-driven candidate generation with structure-aware validation and recursive expansion control to improve the consistency and quality of generated taxonomies by reducing hallucinations. We evaluate the proposed framework using benchmark taxonomies under a masked taxonomy expansion setting and compare multiple validation strategies. Experimental results, supported by both adapted evaluation metrics and human evaluation, demonstrate that ReLTEx produces more reliable and semantically coherent taxonomy expansions.
Tags
Links
- Source: https://arxiv.org/abs/2608.10970v1
- Canonical: https://arxiv.org/abs/2608.10970v1
Trouble viewing inline? Open PDF directly â
Full Text
49,976 characters extracted from source content.
Expand or collapse full text
ReLTEx: Reliable LLM-based Taxonomy Expansion Zeinab Ghamlouch Affiliation: TĂ©lĂ©com Paris, Institut Polytechnique de Paris, France zeinab.b.ghamlouch@gmail.com mehwish.alam@telecom-paris.fr Mehwish Alam Affiliation: TĂ©lĂ©com Paris, Institut Polytechnique de Paris, France zeinab.b.ghamlouch@gmail.com mehwish.alam@telecom-paris.fr Abstract Recent advances in Large Language Models (LLMs) have demonstrated strong capabilities in generating semantically relevant concepts and relations, making them promising tools for taxonomy enrichment. However, directly relying on LLM-generated expansions often leads to noisy, redundant, or hierarchically inconsistent structures, limiting their reliability for automated taxonomy expansion. In this paper, we present ReLTEx, a framework for reliable LLM-based taxonomy expansion. ReLTEx combines LLM-driven candidate generation with structure-aware validation and recursive expansion control to improve the consistency and quality of generated taxonomies by reducing hallucinations. We evaluate the proposed framework using benchmark taxonomies under a masked taxonomy expansion setting and compare multiple validation strategies. Experimental results, supported by both adapted evaluation metrics and human evaluation, demonstrate that ReLTEx produces more reliable and semantically coherent taxonomy expansions. 1 Introduction Taxonomies provide structured hierarchical representations of knowledge and play a fundamental role in numerous applications, including knowledge graphs 12, semantic search and question answering 7, etc. By organizing concepts as parentâchild relations, taxonomies enable semantic understanding, efficient navigation, and knowledge discovery. However, as domains evolve and new concepts continuously emerge, maintaining and expanding taxonomies manually becomes increasingly costly, time-consuming, and difficult to scale 7. To address these challenges, extensive research has explored automated taxonomy expansion. Early approaches relied on lexico-syntactic patterns 3, followed by distributional semantic methods 19 and graph-based techniques 9. Moreover, Pretrained Language Models (PLMs) have substantially improved taxonomy expansion by learning contextual representations of concepts and hierarchical relations 4. Despite these advances, automated taxonomy expansion remains challenging. A generated concept may be semantically ambiguous, as in âBankâ, which can refer to either a financial institution or the side of a river. Candidate generation may also introduce noise or redundancy, for example by proposing both âQuestionâ and âInquiryâ. Moreover, an otherwise valid concept may be attached to an inappropriate parent, such as placing âAnswerâ under âCreativeWorkâ rather than âCommentâ. These errors can compromise hierarchical coherence and propagate during recursive expansion. Recent advances in Large Language Models (LLMs) have created new opportunities for taxonomy enrichment which uses prompting for taxonomy construction 18 and hierarchical attachment prediction 6. Because of their extensive parametric knowledge and strong generative capabilities, LLMs can infer hierarchical relations and propose semantically relevant concepts in a zero-shot setting, making them attractive for expanding existing taxonomies, particularly in domains where curated resources are scarce or rapidly evolving. However, without explicit structural verification, LLM-generated concepts may violate the taxonomy hierarchy, and such errors can accumulate during recursive expansion. To address these challenges, we propose ReLTEx11 1 The source code and the benchmark splits are available at https://github.com/zeinabGhamlouch/ReLTEx., a framework for reliable LLM-based taxonomy expansion. ReLTEx combines zero-shot LLM-based candidate generation with structure-aware validation and recursive expansion control. Candidate parentâchild relations are verified using a path-aware classifier trained on taxonomy structures, allowing structurally inconsistent generations to be filtered before insertion into the taxonomy. Recursive stopping criteria further regulate recursive expansion and reduce the propagation of erroneous generations. We evaluate ReLTEx on benchmark taxonomies under a masked taxonomy expansion setting, where hidden concepts must be recovered from partial taxonomies. Our evaluation consists of standard and adapted evaluation metrics, human assessment, and LLM based assessment along with an ablation study. The main contributions of this work are as follows, we propose: âą A zero-shot LLM-based taxonomy expansion framework that exploits contextual information from the taxonomy hierarchy to generate candidate concepts. âą A structure-aware validation mechanism that mitigates hallucinated and structurally inconsistent parentâchild relations, improving the reliability of recursive taxonomy expansion. âą A comprehensive evaluation protocol based on adapted evaluation metrics, human evaluation, and LLM based assessment. This paper is organized as follows. Section 2 reviews related work while Section 3 defines the taxonomy expansion problem. Section 4 presents ReLTEx. Section 5 describes the datasets and evaluation metrics, and Section 6 presents the experimental results. Finally, Section 7 concludes the paper. 2 Related Work Recent methods formulate taxonomy expansion as node attachment problem, where predefined candidate concepts are inserted into an existing taxonomy by predicting their most appropriate parent node. In the following we categorize these methods into structure based and LLM-based methods. Structure based Methods. TaxoExpan 11 leverages Graph Neural Networks and self-supervised learning to insert unseen concepts through ego-graph representations. Building on the use of structural information, TEMP 4 models taxonomy paths using PLMs and optimizes a dynamic margin ranking objective over positive and negative insertion paths. STEAM 17 adopts a multi-view co-training framework that combines distributed, contextual, and lexico-syntactic representations through mini-path structures. Other approaches have similarly focused on improving hierarchical attachment through graph-based representations 10, contrastive learning strategies 8, or PLM embeddings 14. Despite their effectiveness, most existing taxonomy expansion approaches 11; 5 operate under a constrained setting in which the candidate concepts are provided in advance instead of generating entirely new concepts dynamically. LLM-based Methods. TaxoGlimpse 13 assesses the taxonomic knowledge encoded in LLMs across general and specialized taxonomies. The results show that the performance of an LLM deteriorates for specialized domains and deeper taxonomy levels. However, their work focuses on evaluating taxonomic knowledge rather than expanding an existing hierarchy. Chain-of-Layer 18 iteratively induces a taxonomy from a given set of entities through layer-wise prompting and an ensemble-based relation filtering mechanism. FLAME 6 leverages LLMs to predict the most appropriate parent for a query concept that is known in advance. Taxoria 2 is the closest work to ours, as it recursively generates novel concepts based on an existing taxonomy without relying on a predefined candidate pool. However, its validation is based primarily on embedding-based semantic similarity and does not explicitly model the structural compatibility of a generated parentâchild relation. Consequently, erroneous or weakly related generations may be accepted and propagated during recursive expansion. Unlike prior taxonomy expansion methods that either attach predefined concepts or rely primarily on semantic similarity, ReLTEx combines open-ended LLM generation with path-aware structural validation and recursive expansion control. By validating each generated parentâchild relation in its hierarchical context before insertion. 3 Problem Formulation A taxonomy can be represented as T=(V,E)T=(V,E), where V denotes the set of concepts (nodes) and E represents parentâchild relations between the concepts. Given an initial seed taxonomy T0=(V0,E0)T_0=(V_0,E_0), the goal of taxonomy expansion is to enrich the taxonomy with semantically relevant concepts while preserving hierarchical consistency. Formally, given a node vâV0vâ V_0 and its local hierarchical context, the objective is to generate a set of candidate child concepts Cv=c1,c2,âŠ,ckC_v=\c_1,c_2,âŠ,c_k\. Where k denotes the number of candidate child concepts generated for node v, and cic_i is a potential child concept of v, where i=1,âŠ,ki=1,âŠ,k. The generated candidates may be hallucinated, redundant, or structurally inconsistent. Thus, the goal is to select a curated subset suitable for safe taxonomy integration. The resulting expanded concept set can therefore be expressed as VâČ=V0âȘAvV =V_0âȘ A_v; where AvâCvA_v C_v denotes the subset of generated candidates selected for insertion into the taxonomy. 4 ReLTEx Figure 1 shows an overview of ReLTEx framework. Starting from a seed taxonomy, ReLTEx expands each taxonomy node in three stages: (A) LLM-based candidate generation, (B) structure-aware validation, and (C) recursive expansion using a stopping mechanism. The following subsections describe each stage in detail. Figure 1: Overview of the ReLTEx framework. 4.1 LLM-based Candidate Generation ReLTEx performs recursive taxonomy expansion through a depth-first traversal of the taxonomy. For each target node, the framework constructs a prompt using its local hierarchical context, consisting of the path from the taxonomy root to the target node together with its existing children. The ancestor path provides the semantic context of the current taxonomy branch, while the existing children illustrate the desired level of abstraction, guiding the LLM to generate semantically relevant child concepts at the appropriate level of granularity. The language model is instructed to generate new child concepts considering the appropriate granularity level while avoiding duplicates, synonyms, and simple rephrasing of the existing or newly generated siblings (Appendix E shows the prompt). For each parent node, the LLM is prompted to generate a fixed number k of candidate child concepts. The parameter k controls the branching factor of the expanded taxonomy and can be adjusted depending on the desired level of expansion. After generation, candidate names undergo basic lexical normalization. Exact lexical duplicates under this canonical representation, as well as candidates matching existing children or sibling nodes, are removed before the remaining candidates are passed to the validation stage. 4.2 Structure-Aware Candidate Validation We investigated three validation strategies: (i) semantic similarity filtering, which measures the semantic compatibility between the generated concept and its local taxonomy context; (i) LLM-based validation, where an LLM judges whether the generated concept represents a valid, non-redundant child of the parent; and (i) classifier-guided validation, which predicts the structural validity of the parentâchild relation using a classifier trained on taxonomy edges. In this subsection we focus on classifier guided validation. Unlike semantic similarity measures, the proposed classifier learns structural compatibility from annotated taxonomy relations by jointly considering the hierarchical context, the parent concept, and the generated child concept. The validation module is formulated as a binary classification task using a DistilRoBERTa encoder fine-tuned on labeled parentâchild relation pairs. The classifier is trained on positive and negative examples derived from the seed taxonomy, enabling it to discriminate between valid and invalid hierarchical relations. Given the hierarchical path p, the parent concept v, and a generated candidate child concept cic_i, represented as Path [SEP] Parent [SEP] Child, the classifier outputs a confidence score sâĄ(p,v,ci)â[0,1],s(p,v,c_i)â[0,1], defined as the softmax probability assigned to the positive class, representing the likelihood that the relation (v,ci)(v,c_i) corresponds to a valid taxonomy edge within the hierarchical context p. Candidate concepts with sâĄ(p,v,ci)â„Ïs(p,v,c_i)â„Ï, where Ï denotes the validation threshold, are accepted and incorporated into the taxonomy; otherwise, they are discarded. Positive examples correspond to valid taxonomy edges, whereas negative examples are generated automatically using hierarchy-aware perturbations that produce semantically plausible but structurally incorrect relations. We construct the following hard negative examples: âą Reversed edges, obtained by reversing a valid parentâchild relation, e.g., Comment â Answer becomes Answer â Comment (see Figure 1 for running example). âą Sibling confusions, where the correct parent is replaced by one of its siblings, e.g., Comment â Answer becomes Book â Answer. âą Grandparentâchild confusions, where the immediate parent is replaced by its parent, e.g., Comment â Answer becomes CreativeWork â Answer. âą Same-depth mismatches, where the child is attached to a concept at the same hierarchical level as its true parent, e.g., Article â Answer. âą Nearby hierarchy confusions, where the correct parent is replaced by a semantically related concept from a neighboring branch, e.g., Review â Answer. âą Random invalid relations, obtained by randomly pairing unrelated concepts, e.g., Organization â Answer. Each training instance is represented as Path [SEP] Parent [SEP] Child, where the path corresponds to the ancestor sequence from the taxonomy root to the parent concept. By training on both positive examples and hierarchy-aware hard negatives, the classifier learns structural compatibility rather than relying solely on semantic similarity. 4.3 Recursive Expansion Recursive generative expansion may progressively introduce semantic drift and uncontrolled taxonomy growth. For example, once an incorrect parentâchild relation is accepted, subsequent expansion may continue from that erroneous node and generate concepts that increasingly diverge from the intended taxonomy branch. To mitigate this issue, ReLTEx reuses the confidence scores assigned by the classifier during child validation to regulate recursive expansion at the branch level. Only the concepts accepted during the validation step are eligible for further recursive expansion. Let AvA_v denote the set of accepted child concepts generated for node v: Av=ciâCvâŁsâĄ(p,v,ci)â„ÏA_v=\c_iâ C_v s(p,v,c_i)â„Ï\ ReLTEx first requires a minimum number of accepted children: |Av|â„m,|A_v|â„ m, where m is the minimum number of accepted children required to continue expanding the branch. If |Av|<m|A_v|<m, expansion at node v terminates. If m=1m=1, a branch stops when none of its generated candidates passes the validation threshold. When |Av|â„m|A_v|â„ m, ReLTEx estimates the overall reliability of the current expansion step using the mean confidence of the accepted children: SâĄ(v)=1|Av|ââciâAvsâĄ(p,v,ci).S(v)= 1|A_v| _c_iâ A_vs(p,v,c_i). Averaging considers all accepted children and provides a normalized estimate of the reliability of the expansion step, without being determined by a single exceptionally high- or low-confidence prediction. If we have SâĄ(v)S(v) and SâĄ(parentâĄ(v))S(parent(v)) (mean confidence of its parent node), the expansion below v continues only if SâĄ(v)â„SâĄ(parentâĄ(v))âÎŽ,S(v)â„ S(parent(v))-ÎŽ, where ÎŽ is the maximum allowable decrease in average classifier confidence between two consecutive expansion levels. For the initial expansion step, where no parent confidence score is available, this criterion is not applied. This recursive stopping mechanism recursively expands only branches whose confidence remains sufficiently stable. 5 Experimental Setup 5.1 Datasets and LLMs We evaluate ReLTEx on two taxonomy benchmarks that differ substantially in size and structure: (i) SemEval-2016 Task 13 Environment taxonomy 1, a standard benchmark for taxonomy expansion; (i) Schema.org22 2 https://schema.org/, a large-scale real-world taxonomy that covers a broad range of domains and semantic concepts. Dataset statistics are summarized in Table 1. Table 1: Statistics of the taxonomy benchmarks. |N||N| denotes #nodes, |E||E| denotes #edges, and |D||D| the maximum depth (with the root counted as level 1). Dataset |N||N| |E||E| |D||D| SemEval-env 263 262 6 Schema.org 1,143 1,142 6 Experiments are conducted using four open-source language models from different families: Llama3.2:3B, Mistral:7B, Qwen3:8B, and DeepSeek-R1:8B. These models were selected to cover a diverse range of capabilities, including lightweight instruction following, general-purpose generation, enhanced reasoning, and explicit reasoning via a thinking model. 5.2 Evaluation Metrics Our evaluation builds upon metrics commonly adopted in taxonomy expansion literature 4; 21, including Recall@K, Mean Reciprocal Rank (MRR), and the Wu & Palmer similarity measure. Unlike traditional taxonomy expansion, ReLTEx performs generative expansion, where language models may produce multiple candidate concepts, semantically equivalent variants, or valid concepts attached under alternative parent nodes. To account for these characteristics, we adapt the classical evaluation metrics to the generative setting while preserving their original objectives. Specifically, we distinguish between concept recovery and structural attachment quality, and additionally incorporate semantic matching to account for lexical variability in generated concepts. The primary quantitative evaluation follows a masked taxonomy expansion protocol (described in Section 6), where a subset of taxonomy nodes is hidden and treated as ground truth. The enriched taxonomy is generated from the remaining seed taxonomy, and the recovered concepts are evaluated using the metrics defined below. Let Y denote the set of hidden concepts, with n=|Y|n=|Y|. For each hidden concept yâYyâ Y, let vyv_y denote the original parent node from which y was masked. Let râaânâkâ(y)rank(y) denote the ranking position of y within the accepted candidate set AvyA_v_y (see Section 4 for details). We define the set of locally recovered hidden concepts as Îlocal=yâY|yâAvy, _local= \yâ Y\; |\;yâ A_v_y \, where a hidden concept is considered recovered if it appears among the accepted generated children of its original parent node. Recall@K (R@K) measures the proportion of hidden concepts that are successfully recovered under their original parent nodes: Râ@âK=|Îlocal|n.R@K= | _local|n. SoftRecall@K (SR@K). Exact lexical matching may underestimate taxonomy expansion quality when semantically equivalent concepts are generated. To account for this, we introduce SR@K. SâRâ@âKâ(y)=1,if âmaxciâAvyâsimâĄ(y,ci)â„Ï,0,otherwise.SR@K(y)= cases1,&if _c_iâ A_v_ysim(y,c_i)â„Ï,\\ 0,&otherwise. cases where yâYyâ Y is the hidden concept, simâĄ(â ,â )sim(·,·) denotes cosine similarity between embeddings generated by the BAAI/bge-small-en-v1.5 model 15. We set Ï=0.85Ï=0.85, which provides a conservative balance between recognizing semantically equivalent concepts while avoiding matches between only loosely related concepts. The final SR@K score is computed as SâRâ@âK=1nââyâYSâRâ@âKâ(y).SR@K= 1n _yâ YSR@K(y). Mean Reciprocal Rank (MRR). Accepted candidate concepts are ranked according to decreasing classifier confidence score sâĄ(p,v,ci)s(p,v,c_i). We report two complementary variants of MRR: (i) MâRâRoverallMRR_overall and (i) MâRâRfoundMRR_found. MâRâRoverallMRR_overall measures ranking quality over all hidden concepts. Recovered concepts contribute the reciprocal of their ranking position, while hidden concepts that are not recovered receive a score of zero, thereby penalizing failure to recover hidden concepts: MâRâRoverall=1nââyâY1râaânâkâ(y),if âyâAvy,0,otherwise.MRR_overall= 1n _yâ Y cases 1rank(y),&if yâ A_v_y,\\ 0,&otherwise. cases MâRâRfoundMRR_found evaluates ranking quality only for successfully recovered hidden concepts: MâRâRfound=1|Îlocal|ââyâÎlocal1râaânâkâ(y).MRR_found= 1| _local| _yâ _local 1rank(y). Wu & Palmer (WuP) Similarity. To evaluate the structural quality of node generation, we adapt the WuP similarity to compare predicted parent placements against the original taxonomy structure. For each hidden concept yâYyâ Y, let Vypred=vâŁyâAvV_y^pred=\v yâ A_v\ denote the set of parent nodes under which a hidden concept y was generated. The attachment similarity score is then defined as wâĄ(y)=maxvâVypredâĄ2â dâeâpâtâhâ(LâCâAâ(v,vy))dâeâpâtâhâ(v)+dâeâpâtâhâ(vy).w(y)= _vâ V_y^pred 2· depth(LCA(v,v_y))depth(v)+depth(v_y). where LâCâAâ(â ,â )LCA(·,·) denotes the Lowest Common Ancestor. Higher similarity values indicate that the predicted parent node is structurally closer to the original parent node. We further define the set of globally recovered hidden concepts as Îglobal=yâYâŁVypredâ â . _global=\yâ Y V_y^predâ \. We report two variants of the WuP score. The first variant measures attachment quality over all hidden concepts. Successfully generated concepts contribute their WuP score, while hidden concepts that are not successfully generated are scored zero. WâuâPoverall=1nââyâYwâĄ(y),if âyâÎglobal,0,otherwise.WuP_overall= 1n _yâ Y casesw(y),&if yâ _global,\\ 0,&otherwise. cases The second variant evaluates attachment quality only for successfully generated concepts: WâuâPfound=1|Îglobal|ââyâÎglobalwâĄ(y).WuP_found= 1| _global| _yâ _globalw(y). 6 Evaluation Results We evaluate ReLTEx using three complementary protocols: (i) Masked Taxonomy Expansion, (i) Human Evaluation, (i) LLM based taxonomy evaluation. . 6.1 Masked Taxonomy Expansion Benchmark We evaluate concept recovery using a masked taxonomy expansion benchmark, following the standard evaluation protocol 16. We remove 20% of the nodes from the seed taxonomy and treat them as hidden test concepts. The remaining taxonomy serves as the seed taxonomy provided to ReLTEx, while the hidden concepts constitute the gold standard used for evaluation. Only leaf nodes are considered eligible for masking in order to preserve the overall taxonomy structure. Masking is performed under two constraints. (i) Hidden nodes are selected only from those parent nodes retaining at least one visible child after masking, preventing the removal of all local structural context. (i) A maximum of five hidden children is allowed under any single parent node. These constraints prevent a small number of high-degree parent nodes from dominating the evaluation process and ensures a more balanced distribution of hidden concepts across the taxonomy. The generator LLM33 3 Unless otherwise specified, candidate generation is performed using Mistral:7B. produces k=5k=5 candidate child concepts for each parent node. The root node is excluded from expansion. Generated parentâchild candidates are then validated. Candidates are accepted only if their confidence score â„ 0.90. Although ReLTEx supports recursive taxonomy expansion, benchmark experiments are performed exclusively on the masked seed taxonomy. Newly generated concepts are not recursively expanded. Since existing taxonomy expansion methods assume a predefined candidate set rather than generating new concepts, no directly comparable baseline exists for our setting. Table 2 summarizes the benchmark results under the masked taxonomy expansion setting on the SemEval Environment and Schema.org taxonomies. The higher numbers (shown in bold) indicate better performance or quality in the rest of this paper. Table 2: Results of the masked taxonomy expansion benchmark. Metric Llama3.2 Mistral Qwen3 DeepSeek SemEval Environment R@K 44.23 ± 2.11 42.31 ± 1.22 42.95 ± 0.99 28.85 ± 6.20 SR@K 53.21 ± 2.63 46.47 ± 1.89 47.76 ± 2.25 40.70 ± 4.78 MRRo 23.47 ± 3.10 23.81 ± 0.52 21.69 ± 1.08 14.53 ± 2.61 MRRf 52.92 ± 5.02 55.90 ± 1.74 50.54 ± 3.11 50.74 ± 2.60 WuPo 48.17 ± 1.99 44.51 ± 1.66 45.22 ± 1.73 31.77 ± 5.93 WuPf 92.93 ± 3.18 94.64 ± 3.40 96.63 ± 1.90 92.19 ± 6.90 Schema.org R@K 14.04 ± 2.69 23.28 ± 0.80 19.66 ± 1.06 11.06 ± 2.52 SR@K 22.99 ± 2.24 31.70 ± 2.11 31.79 ± 0.96 19.23 ± 2.62 MRRo 7.86 ± 1.18 11.97 ± 0.46 11.02 ± 1.18 6.63 ± 1.67 MRRf 52.12 ± 4.86 49.72 ± 1.62 53.94 ± 3.56 55.39 ± 4.67 WuPo 21.75 ± 3.29 27.57 ± 1.28 24.92 ± 1.61 13.75 ± 4.52 WuPf 80.27 ± 1.10 92.32 ± 3.03 90.61 ± 2.52 85.71 ± 3.03 Overall, all evaluated models demonstrate the ability to recover hidden taxonomy concepts. As expected, the larger and more diverse Schema.org taxonomy is substantially more challenging than the SemEval Environment taxonomy, leading to lower recovery and attachment scores across all models. Nevertheless, the relative ranking of the models remains largely consistent across both benchmarks. Mistral achieves the strongest overall performance across both datasets, obtaining the highest R@K, MRRo, and WuPo scores on Schema.org while remaining highly competitive on SemEval. Llama3.2 performs best on the smaller SemEval taxonomy, achieving the highest R@K, S@K, and WuPo, indicating strong concept recovery and structural placement. Qwen3 consistently produces competitive semantic recovery performance, achieving the highest S@K on Schema.org, suggesting a greater ability to generate semantically related concepts beyond exact lexical matches. DeepSeek-R1 generally obtains lower recovery scores and exhibits larger standard deviations across several metrics, indicating lower stability across different masking configurations. 6.2 Human Evaluation LLMs may generate valid novel concepts that are absent from the ground truth. To complement the automatic evaluation, we conduct a human assessment of the generated taxonomy expansions. We evaluate the expanded taxonomy obtained from the run achieving the highest R@K. For each taxonomy, we randomly sample 20% of the eligible parent nodes, providing representative coverage while keeping the manual annotation effort feasible. Eligible parents correspond to non-root nodes that retain at least one visible child in the masked seed taxonomy. The same sampled parent nodes are evaluated across all four models. To avoid model-specific bias, the four models are anonymized. We further randomize the order of both the sampled parent nodes and the generated concepts. A web-based interface44 4 https://reltex-human-eval.vercel.app/ shows the taxonomy context and allows consistent annotation. For each sampled parent node, annotators are presented with its hierarchical path, its existing children, the accepted generated children, and, when applicable, the hidden gold children removed during masking (see Appendix B for the snapshot of the annotation interface). The interface distinguishes between two evaluation settings. In recovery cases, hidden gold children exist for the current parent, allowing both recovery and enrichment to be assessed. In novel enrichment cases, no hidden gold child exists under the parent, and the evaluation focuses solely on the quality of the generated concepts. Each generated concept is independently evaluated according to the following criteria: Hierarchical Correctness (HC), Granularity Consistency (GC), Non-Redundancy (NR), Exact Recovery (ER), and Semantic Recovery (SR). Three independent annotators evaluated all eight sample sets. Final labels are obtained using majority voting across the three annotations for each generated concept. The inter-annotator agreement was measured using Fleissâ Îș, which is 0.695. Table 3 presents the final results. Expansion-quality metrics (HC, GC, and NR) are computed over all sampled parent nodes, whereas ER and SR are evaluated only on the subset of masked concepts. Despite being measured on this more restrictive evaluation set, the recovery scores demonstrate that the generated concepts frequently recover or closely match the hidden taxonomy concepts. Overall, the generated expansions receive consistently high human judgments for hierarchical correctness, granularity consistency, and non-redundancy across both datasets, indicating that the accepted concepts are generally well positioned within the taxonomy and remain consistent with the abstraction level of the surrounding branch. Table 3: Human evaluation results using majority voting across three annotators. Dataset Model Expansion Quality Recovery HC GC NR ER SR SemEval Env Llama3.2 0.978 0.978 0.978 0.200 0.300 DeepSeek-R1 0.960 0.960 0.920 0.200 0.300 Mistral 0.960 0.940 0.860 0.400 0.400 Qwen3 0.980 0.980 0.940 0.300 0.400 Schema.org Llama3.2 0.898 0.932 0.864 0.103 0.144 DeepSeek-R1 0.988 0.988 0.940 0.076 0.143 Mistral 0.978 1.000 1.000 0.101 0.202 Qwen3 1.000 1.000 0.989 0.112 0.178 On SemEval Environment, Qwen3 achieves the strongest performance in HC and GC, while Llama3.2 produces the highest non-redundancy score. Mistral obtains the best exact recovery and ties with Qwen3 for the highest SR, indicating a stronger ability to recover concepts removed during masking. On Schema.org, Qwen3 achieves perfect HC and GC. Mistral reaches perfect GC and NR together with the highest SR, whereas DeepSeek-R1 also demonstrates strong expansion quality despite lower recovery performance. 6.3 LLM-Based Taxonomy Evaluation We complement our evaluation using LITE 20, an LLM-based framework for assessing the semantic and structural quality of taxonomies. LITE defines four measures: (i) Single Concept Accuracy (SCA): evaluates the semantic clarity, validity, and coherence of individual taxonomy concepts; (i) Hierarchy Relationship Rationality (HRR): evaluates whether parentâchild relations represent logically consistent hierarchical dependencies. (i) Hierarchy Relationship Exclusivity (HRE): evaluates whether sibling concepts are semantically distinct and non-overlapping. (iv) Hierarchy Relationship Independence (HRI): evaluates the structural independence of concepts by measuring redundancy and semantic overlap among siblings. Following LITE, we assess taxonomy quality using LLM-based judgments. While the original framework reports HRE and HRI separately, our implementation returns a single score jointly reflecting both criteria, which we report as the HRE/HRI score. LITE is further applied to the fully expanded taxonomies produced by the ReLTEx (instead of masking). As recursive expansion increases the number of generated nodes at each level, using the benchmark generation setting (k=5k=5 candidates per parent) results in large taxonomies with limited benefit. Therefore, we reduce the generation to k=3k=3. We evaluate both the seed taxonomy and the enriched taxonomy, allowing direct comparison before and after enrichment. We replace the proprietary evaluator in LITE with three open-source instruction-tuned LLMs deployed locally through Ollama55 5 https://ollama.com/: Llama3.2, Qwen3:8B, and DeepSeek-R1:8B. The enriched taxonomies are generated using Mistral:7B, selected as the generator due to its strongest overall performance in the masked taxonomy expansion benchmark. The three LLMs are used exclusively as evaluators. Tables 5 and 5 report the LITE scores for the seed and the enriched taxonomy. Improvements over the seed taxonomy show that recursive enrichment preserves or enhances the quality while introducing new concepts. Table 4: LITE evaluation on the SemEval Environment. Evaluator Taxonomy SCA HRR HRE/HRI Llama3.2 Seed 8.01 8.19 7.74 ReLTEx 7.92 8.24 7.75 Qwen3 Seed 7.42 6.82 7.83 ReLTEx 7.31 6.96 8.88 DeepSeek Seed 7.71 7.43 7.03 ReLTEx 7.73 8.78 8.15 Table 5: LITE evaluation on the Schema.org. Evaluator Taxonomy SCA HRR HRE/HRI Llama3.2 Seed 8.18 8.05 7.95 ReLTEx 7.94 8.29 8.30 Qwen3 Seed 8.35 8.95 8.59 ReLTEx 8.64 8.68 8.76 DeepSeek Seed 8.19 8.67 8.05 ReLTEx 8.07 8.68 7.94 Across both taxonomies, the enriched taxonomies achieve high SCA, indicating that recursive expansion preserves semantic quality. HRR is generally maintained or improved after enrichment, suggesting that generated parentâchild relations remain logically coherent. Similar trends are observed for HRE/HRI, with most evaluators assigning equal or higher scores to the enriched taxonomies, particularly on the Environment taxonomy. The ablation study, qualitative error analysis, and expanded taxonomy statistics are provided in Appendix D, Appendix C, and Appendix E, respectively. 7 Conclusion We presented ReLTEx, a framework for taxonomy enrichment using LLMs. Unlike existing taxonomy expansion approaches that assume candidate concepts to be already available, ReLTEx jointly performs concept generation and structure-aware validation through a classifier-based validation module and recursive expansion mechanism. The proposed evaluation protocol combines automatic benchmarking, human assessment, and LLM-based taxonomy evaluation, providing complementary perspectives on taxonomy quality beyond concept recovery alone. Limitations Our experiments rely on relatively compact open-source language models to enable efficient and reproducible experimentation. Larger and more capable models may further improve the quality of generated taxonomy expansions. In addition, the proposed classifier is a learned approximation of hierarchical validity and may occasionally accept incorrect parentâchild relations. Finally, recursive LLM-based taxonomy generation remains an emerging task without a standardized benchmark or directly comparable generation baseline, making comprehensive system-level comparisons difficult. Ethical Considerations The human evaluation involved three volunteer annotators following predefined annotation guidelines. The collected annotations were used solely for research purposes, and no sensitive personal information is reported in this work. The taxonomies used are publicly available benchmark resources and contain no personal or sensitive data. Since ReLTEx relies on large language models for concept generation, it may inherit biases present in the underlying models and their training data, potentially affecting the generated concepts and their hierarchical placement. Although the proposed validation module mitigates these issues, some errors may still remain. References Bordea et al. (2016) G. Bordea, E. Lefever, and P. Buitelaar SemEval-2016 task 13: taxonomy extraction evaluation (TExEval-2). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), S. Bethard, M. Carpuat, D. Cer, D. Jurgens, P. Nakov, and T. Zesch (Eds.), San Diego, California, p. 1081â1091. External Links: Link, Document Cited by: §5.1. Ghamlouch and Alam (2025) Z. Ghamlouch and M. Alam Enriching taxonomies using large language models. In ECAI 2025-28th European Conference on Artificial Intelligence (Demo Track), Cited by: §2. Hearst (1992) M. A. Hearst Automatic acquisition of hyponyms from large text corpora. In COLING 1992 volume 2: The 14th international conference on computational linguistics, Cited by: §1. Liu et al. (2021) Z. Liu, H. Xu, Y. Wen, N. Jiang, H. Wu, and X. Yuan TEMP: taxonomy expansion with dynamic margin loss through taxonomy-paths. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 3854â3863. Cited by: §1, §2, §5.2. Margiotta et al. (2023) D. Margiotta, D. Croce, and R. Basili Taxosbert: unsupervised taxonomy expansion through expressive semantic similarity. In International Conference on Deep Learning Theory and Applications, p. 295â307. Cited by: §2. Mishra et al. (2024) S. Mishra, U. Sudev, and T. Chakraborty Flame: self-supervised low-resource taxonomy expansion using large language models. ACM Transactions on Intelligent Systems and Technology. Cited by: §1, §2. Navigli and Ponzetto (2010) R. Navigli and S. P. Ponzetto BabelNet: building a very large multilingual semantic network. In Proceedings of the 48th annual meeting of the association for computational linguistics, p. 216â225. Cited by: §1. Niu et al. (2024) Y. Niu, H. Xu, C. Liu, Y. Wen, and X. Yuan Contrastive representation learning for self-supervised taxonomy completion.. In IJCAI, Vol. 8, p. 6442â6450. Cited by: §2. Pietrasik et al. (2024) M. Pietrasik, M. Reformat, and A. Wilbik Non-parametric path based model for taxonomy induction in knowledge graphs. In Belgium netherlands conference on artificial intelligence, Cited by: §1. Shang et al. (2020) C. Shang, S. Dash, M. F. M. Chowdhury, N. Mihindukulasooriya, and A. Gliozzo Taxonomy construction of unseen domains via graph-based cross-domain knowledge transfer. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics, p. 2198â2208. Cited by: §2. Shen et al. (2020) J. Shen, Z. Shen, C. Xiong, C. Wang, K. Wang, and J. Han TaxoExpan: self-supervised taxonomy expansion with position-enhanced graph neural network. In Proceedings of the web conference 2020, p. 486â497. Cited by: §2, §2. Suchanek et al. (2024) F. M. Suchanek, M. Alam, T. Bonald, L. Chen, P. Paris, and J. Soria Yago 4.5: a large and clean knowledge base with a rich taxonomy. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, p. 131â140. Cited by: §1. Sun et al. (2024) Y. Sun, H. Xin, K. Sun, Y. E. Xu, X. Yang, X. L. Dong, N. Tang, and L. Chen Are large language models a good replacement of taxonomies?. arXiv preprint arXiv:2406.11131. Cited by: §2. Takeoka et al. (2021) K. Takeoka, K. Akimoto, and M. Oyamada Low-resource taxonomy enrichment with pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 2747â2758. Cited by: §2. Xiao et al. (2023) S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff C-pack: packaged resources to advance general chinese embedding. External Links: 2309.07597 Cited by: §5.2. Xu et al. (2024) F. Xu, S. Jiang, Z. Huang, X. Luo, S. Zhang, Y. Chen, and Y. Sun FUSE: measure-theoretic compact fuzzy set representation for taxonomy expansion. In Findings of the association for computational linguistics: ACL 2024, p. 2707â2720. Cited by: §6.1. Yu et al. (2020) Y. Yu, Y. Li, J. Shen, H. Feng, J. Sun, and C. Zhang Steam: self-supervised taxonomy expansion with mini-paths. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p. 1026â1035. Cited by: §2. Zeng et al. (2024) Q. Zeng, Y. Bai, Z. Tan, S. Feng, Z. Liang, Z. Zhang, and M. Jiang Chain-of-layer: iteratively prompting large language models for taxonomy induction from limited examples. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, p. 3093â3102. Cited by: §1, §2. Zhang et al. (2018) C. Zhang, F. Tao, X. Chen, J. Shen, M. Jiang, B. Sadler, M. Vanni, and J. Han Taxogen: constructing topical concept taxonomy by adaptive term embedding and clustering. Proc. KDDI. Cited by: §1. Zhang et al. (2025) L. Zhang, Z. Gu, S. Zheng, T. Wang, T. Li, H. Feng, and Y. Xiao LITE: llm-impelled efficient taxonomy evaluation. arXiv preprint arXiv:2504.01369. Cited by: §6.3. Zhang et al. (2024) Y. Zhang, J. Qin, and C. Feng PEB-taxo: projecting entities as boxes for taxonomy expansion. Neural Processing Letters 56 (2), p. 102. Cited by: §5.2. Appendix A Input Prompts A.1 Generation Prompt The following prompt is used to generate candidate children for each expanded taxonomy node. The placeholders enclosed in braces are dynamically replaced during execution. ⏠Current node: current_node Parent path: parent_path Existing children of this node: existing_children Task: Suggest exactly K NEW suitable children of the current node. Important: - The new children MUST be at the SAME level of specificity as the existing children. - Use the existing children as examples of the correct granularity. - Each suggestion must be a child under "current_node". Rules: - Do NOT repeat any existing child. - Do NOT suggest synonyms, rephrasings, or formatting variants. - Do NOT suggest broader or more general concepts than the existing children. Output format: - Return exactly K names. - One name per line. - No numbering, explanations, or additional text. - Do not return the parent path. A.2 LLM-as-a-Judge Prompt The following prompt is used by the LLM-based validator to assess each generated candidate independently. The placeholders are populated using the current taxonomy context. ⏠Parent path: parent_path Parent class: parent Existing children of the parent: existing_children Candidate child: candidate Task: Decide whether the candidate should be accepted as a reasonable NEW child category under the parent in this taxonomy. Guidelines: 1. Accept the candidate if it is primarily a meaningful type/category of the parent. 2. Reject the candidate if it is redundant with an existing child. 3. Reject the candidate if it is only loosely related to the parent. Return ONLY valid JSON with exactly these fields: "accept": true or false, "is_type_of_parent": true or false, "is_redundant": true or false, "confidence": number between 0 and 1, "reason": "short explanation" A candidate is accepted only if the judge marks it as acceptable, classifies it as a type of the parent, and does not identify it as redundant: Acceptâ(c)=aâĄ(c)â§tâĄ(c)â§ÂŹrâĄ(c), Accept(c)=a(c) t(c) r(c), where aâĄ(c)a(c), tâĄ(c)t(c), and râĄ(c)r(c) correspond to the accept, is_type_of_parent, and is_redundant fields, respectively. Appendix B Human Evaluation Interface (a) Recovery case (SemEval Environment). (b) Novel enrichment case (Schema.org). Figure 2: Snapshot of the annotation interface used in the human evaluation. Figure 2 illustrates the annotation interface used throughout the human evaluation. The upper panel summarizes the local taxonomy context by displaying the hierarchical path, existing children, generated accepted children, and, when applicable, the hidden gold children. The lower panel presents the evaluation form for each generated concept. Recovery-specific criteria are automatically enabled only when hidden gold children are available, while novel enrichment cases evaluate only the validity and quality of newly generated concepts. Appendix C Error Analysis To better understand the remaining limitations of ReLTEx, we manually analyzed all parentâchild relations that were accepted by our pipeline but rejected by the majority of human annotators. Rather than grouping errors according to the evaluation criteria, we categorized them based on their underlying semantic causes. Table 6 presents representative examples from each category. One category of remaining errors corresponds to incorrect hierarchical placement. In these cases, the generated concept is semantically valid but attached to an inappropriate parent within the taxonomy. For example, BankTransfer was generated under PaymentCard, although it represents an alternative payment method rather than a subtype of payment card. Similarly, overfishing was proposed as a child of marine pollution, despite representing a distinct environmental issue rather than a form of pollution. Another category corresponds to scope mismatch. Here, the generated concept belongs to the correct semantic domain but is expressed at an incompatible level of abstraction. For instance, concepts such as Climate change mitigation strategies and Climate change adaptation measures were generated as children of Climate change, mixing intervention strategies with environmental phenomena. Similar behavior was observed in Schema.org, where Gallery was generated under MediaGallery, although it is semantically more general than its parent. We also observed cases of semantic redundancy, where the generated concept was effectively a near-synonym of an existing taxonomy concept. For example, TattooStudio was generated despite the taxonomy already containing the semantically equivalent concept TattooParlor, resulting in unnecessary duplication. Table 6: Representative semantic error categories identified during qualitative analysis. Generated Concept Existing Context Category Observation BankTransfer Parent: PaymentCard Incorrect placement Not a subtype of the parent Gallery Parent: MediaGallery Scope mismatch Superclass of the parent TattooStudio Sibling: TattooParlor Semantic redundancy Near-synonymous existing concept Sho Emission Trading â Malformed concept Incomplete generated entity Finally, a small number of errors correspond to malformed concepts, including incomplete or corrupted generations such as Sho Emission Trading. These failures originate from generation artifacts rather than structural reasoning errors. Appendix D Ablation Study To assess the impact of the main design choices in ReLTEx, we conduct ablation studies on three components of the framework: the prompting context, the validation strategy, and the validator acceptance thresholds. The experiments are performed on the SemEval Environment taxonomy using Mistral:7B as the generator model. As the goal is to isolate the contribution of each design choice rather than compare datasets, the ablation is conducted on this single representative benchmark. Threshold calibration is performed through a sweep over a fixed set of generated candidates. Based on the resulting recovery performance, we select acceptance thresholds of 0.830.83 for both semantic validators and 0.900.90 for the classifier. These thresholds are used throughout all experiments reported in the paper. We compare three prompting configurations: âą Local: provides only the path to the parent node together with its existing children. âą Parent Subtree: provides the complete subtree rooted at the parent. âą Full Taxonomy: provides the entire seed taxonomy as context. For each prompting configuration, four validation strategies are evaluated: âą Semantic V1: validates candidates using cosine similarity between the parent and the generated child. âą Semantic V2: extends Semantic V1 by additionally considering the taxonomy path and similarity to the parentâs existing children. âą LLM Judge: uses Llama3.2 to determine whether a generated candidate represents a valid, non-redundant child. âą Classifier: the proposed DistilRoBERTa-based taxonomy validator. Table 7 summarizes the benchmark results. Across all validation strategies, the Local prompting configuration consistently achieves the highest R@K and SR@K. Restricting the context to the parent path and its existing children provides sufficient structural information while avoiding the additional noise introduced by larger contexts. Both the Parent Subtree and Full Taxonomy settings substantially reduce recovery performance, suggesting that exposing the language model to increasingly large portions of the taxonomy does not improve concept generation. The comparison of validation strategies further supports the proposed classifier. Across all prompting contexts, it consistently achieves the highest R@K and SR@K. Both semantic validators produce slightly lower recovery scores by rejecting a small number of valid candidates, while Semantic V2 performs nearly identically to Semantic V1, indicating that the additional taxonomy context provides little benefit in this setting. The LLM Judge is the most conservative validator, substantially reducing recovery performance by rejecting many valid candidates together with incorrect ones. Table 7: Results of the ablation study on the SemEval Environment taxonomy. Higher values indicate better performance. Context Validation Strategy R@K SR@K Local Classifier 34.62 42.31 Semantic V1 32.69 40.38 Semantic V2 32.69 40.38 LLM Judge 15.38 19.23 Parent Subtree Classifier 17.31 23.08 Semantic V1 15.38 21.15 Semantic V2 15.38 21.15 LLM Judge 3.85 3.85 Full Taxonomy Classifier 11.54 15.38 Semantic V1 9.62 13.46 Semantic V2 9.62 13.46 LLM Judge 9.62 9.62 Appendix E Expanded Taxonomy Statistics To provide a quantitative overview of the resulting enriched taxonomies, table 8 compares the original seed taxonomies with the enriched taxonomies produced by ReLTEx. To remain consistent with the LITE evaluation, we report the taxonomies generated using Mistral:7B, which was selected as the generator after achieving the strongest overall performance in the masked taxonomy expansion benchmark. Table 8: Statistics of the seed and enriched taxonomies produced by ReLTEx using Mistral:7B. |N||N| denotes the number of nodes, |E||E| the number of edges, and |D||D| the maximum depth. Dataset Taxonomy |N||N| |E||E| |D||D| SemEval Env Seed 211 210 6 ReLTEx 4883 4882 7 Schema.org Seed 912 911 6 ReLTEx 38603 38602 7 The adaptive recursive expansion substantially increases the size of both taxonomies while introducing one additional hierarchy level. This indicates that ReLTEx primarily enriches existing branches rather than generating excessively deep hierarchies. The larger increase observed for Schema.org also reflects its broader semantic coverage, providing more opportunities for recursive concept generation than the smaller SemEval Environment taxonomy.