Paper deep dive
SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction
David Števaňák, Marek Šuppa
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/22/2026, 5:23:29 AM
Summary
The paper introduces SlovKE, a large-scale dataset of 227,432 Slovak scientific abstracts with author-assigned keyphrases, designed to address the scarcity of resources for morphologically rich, low-resource languages. The authors benchmark traditional extractive methods (YAKE, TextRank, KeyBERT) against a generative LLM-based approach (KeyLLM). They identify a significant 'exact-partial' performance gap caused by morphological inflection, demonstrating that KeyLLM effectively narrows this gap by generating canonical keyphrase forms.
Entities (6)
Relation Signals (3)
SlovKE → benchmarks → KeyLLM
confidence 95% · Using this dataset, we benchmark three unsupervised baselines... and evaluate KeyLLM.
Morphological inflection → causes → Exact-partial performance gap
confidence 95% · Our analysis identifies morphological mismatch as the dominant failure mode for statistical methods.
KeyLLM → outperforms → YAKE
confidence 90% · KeyLLM’s exact-match F1@6 (~15.2) is substantially higher than the best extractive baseline (YAKE, 11.6).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Keyphrase extraction for morphologically rich, low-resource languages remains understudied, largely due to the scarcity of suitable evaluation datasets. We address this gap for Slovak by constructing a dataset of 227,432 scientific abstracts with author-assigned keyphrases -- scraped and systematically cleaned from the Slovak Central Register of Theses -- representing a 25-fold increase over the largest prior Slovak resource and approaching the scale of established English benchmarks such as KP20K. Using this dataset, we benchmark three unsupervised baselines (YAKE, TextRank, KeyBERT with SlovakBERT embeddings) and evaluate KeyLLM, an LLM-based extraction method using GPT-3.5-turbo. Unsupervised baselines achieve at most 11.6\% exact-match $F1@6$, with a large gap to partial matching (up to 51.5\%), reflecting the difficulty of matching inflected surface forms to author-assigned keyphrases. KeyLLM narrows this exact--partial gap, producing keyphrases closer to the canonical forms assigned by authors, while manual evaluation on 100 documents ($\kappa = 0.61$) confirms that KeyLLM captures relevant concepts that automated exact matching underestimates. Our analysis identifies morphological mismatch as the dominant failure mode for statistical methods -- a finding relevant to other inflected languages. The dataset (this https URL) and evaluation code (this https URL) are publicly available.
Tags
Links
- Source: https://arxiv.org/abs/2603.15523v1
- Canonical: https://arxiv.org/abs/2603.15523v1
Trouble viewing inline? Open PDF directly →
Full Text
51,593 characters extracted from source content.
Expand or collapse full text
SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction David Števa ˇ nák 1,4 Marek Šuppa 2,3,4 1 University of Vienna, Austria 2 Faculty of Mathematics, Physics and Informatics, Comenius University in Bratislava 3 Cisco Systems, Inc. 4 NaiveNeuron Correspondence: marek@suppa.sk Abstract Keyphrase extraction for morphologically rich, low-resource languages remains understudied, largely due to the scarcity of suitable evaluation datasets. We address this gap for Slovak by constructing a dataset of 227,432 scientific abstracts with author-assigned keyphrases—scraped and systematically cleaned from the Slovak Central Register of Theses—representing a 25-fold increase over the largest prior Slovak resource and approaching the scale of established English benchmarks such as KP20K. Using this dataset, we benchmark three unsupervised baselines (YAKE, TextRank, KeyBERT with SlovakBERT embeddings) and evaluate KeyLLM, an LLM-based extraction method using GPT-3.5-turbo. Unsupervised baselines achieve at most 11.6% exact-matchF1@6, with a large gap to partial matching (up to 51.5%), reflecting the difficulty of matching inflected surface forms to author-assigned keyphrases. KeyLLM narrows this exact–partial gap, producing keyphrases closer to the canonical forms assigned by authors, while manual evaluation on 100 documents (κ = 0.61) confirms that KeyLLM captures relevant concepts that automated exact matching underestimates. Our analysis identifies morphological mismatch as the dominant failure mode for statistical methods—a finding relevant to other inflected languages. The dataset (https://huggingface.co/datasets/NaiveNeuron/SlovKE) and evaluation code (https://github.com/NaiveNeuron/SlovKE) are publicly available. Keywords: keyphrase extraction, Slovak, morphologically rich languages, large language models, bench- mark dataset, evaluation metrics, Slavic NLP 1. Introduction Keyphrase extraction—identifying the words or phrases that best represent a document’s top- ics (Hasan and Ng, 2014)—is essential for sci- entific literature discovery and classification (Pa- pagiannopoulou and Tsoumakas, 2020). While recent advances have yielded strong results for En- glish and other high-resource languages (Alzaidy et al., 2019; Basaldella et al., 2018; Nguyen and Kan, 2007), morphologically rich low-resource lan- guages remain underexplored. In such languages, a single lemma can surface in dozens of inflected forms (Tsarfaty et al., 2013), creating a fundamen- tal mismatch between the surface forms models extract and the canonical forms authors assign as keyphrases. This challenge is not unique to any single language—it affects the entire family of morphologically rich languages, including most Slavic, Finno-Ugric, and Turkic languages (Beliga and Martincic-Ipsic, 2016). Figure 1 illustrates the problem: author-assigned keyphrases in nomina- tive form (e.g., Rozvojový potenciál ) appear in the abstract only in inflected variants (e.g., genitive rozvojového potenciálu). We study this problem through the lens of Slovak, a West Slavic language with limited keyphrase ex- traction research due to the lack of suitable corpora. Building on the work of Zelinka (2023), who col- lected approximately 9,000 documents but faced limitations in data quality and scale, we construct a substantially larger and cleaner dataset and con- duct a systematic evaluation of both established and LLM-based keyphrase extraction methods. We evaluate KeyLLM (Grootendorst, 2021), an LLM- based extraction approach, to determine whether generative models can better handle the morpho- logical variability that hinders traditional extractive methods. Our main contributions are: •A rigorously cleaned dataset of 227,432 Slo- vak scientific abstracts with author-assigned keyphrases—a 25-fold increase over prior work—whose size and statistics are compara- ble to established English benchmarks such as KP20K (Meng et al., 2017). •A controlled evaluation of statistical (YAKE), graph-based (TextRank), and embedding- based (KeyBERT) models on Slovak, reveal- ing that exact-matchF1@6remains below 12% while partial-matchF1@6reaches 51.5%, exposing a∼40-point gap attributable to mor- phological inflection. •Evidence that KeyLLM substantially nar- rows this exact–partial gap by generating keyphrases in canonical form rather than ex- tracting surface tokens, suggesting that gener- arXiv:2603.15523v1 [cs.CL] 16 Mar 2026 Predmetom skúmania diplomovej práce je potenciál vidieka, územný rozvoj a možnosti rozvoja vidieka. Objektom skúmania je Žilinský samosprávny kraj a jeho historický región Dolná Orava. Ciel’om práce je na základe teoretického rozpracovania problematiky a analýzy rozvojového potenciálu vymedzit’ možnosti rozvoja vidieka. V práci je sústredená pozornost’ na analýzu rozvojového potenciálu Žilinského kraja a možností využitia jeho jednotlivých zložiek na zabezpe ˇ cenie ekonomického, sociálneho a trvaloudržatel’ného rozvoja. Predpokladaný prínos práce vidíme vo formulovaní návrhov využitia rozvojového potenciálu kraja na zabezpe ˇ cenie endogénneho rozvoja vidieka. Keywords: Vidiek, Rozvoj vidieka, Rozvojový potenciál, Vidiecky región, Vidiecke sídlo, Nástroje rozvoja Figure 1: Example abstract from the Test22K dataset (Slovak). Color coding indicates keyphrase occurrences: author-assigned keyphrases appear in the text in various inflected forms (red: exact match, teal: partial overlap, green: single-word match, blue: shared fragment). Note that surface forms in the abstract (e.g., rozvojového potenciálu, genitive) differ from the canonical keyphrase form (Rozvojový potenciál, nominative), illustrating the morphological mismatch challenge. ative models are better suited to morphologi- cally rich languages. • A manual evaluation and error analysis (κ=0.61) identifying morphological mismatch as the dominant failure mode for extractive models and unmotivated adjective extraction as the primary weakness of KeyLLM. 2. Related Work Keyphrase extraction has evolved from early statis- tical and graph-based methods (Hulth, 2003; Mihal- cea and Tarau, 2004) to modern transformer-based approaches. Recent advances leveraging pre- trained language models (PLMs) have achieved state-of-the-art performance (Song et al., 2023), with BERT-based methods like KeyBERT (Groo- tendorst, 2020) and approaches utilizing domain- specific models like SciBERT (Park and Caragea, 2020) showing significant improvements. Super- vised methods formulating keyphrase extraction as sequence labeling with BiLSTM-CRF architec- tures over contextualized embeddings (Sahrawat et al., 2020) have demonstrated strong perfor- mance, though they require substantial training data. More recently, large language models have been explored for keyphrase extraction (Grooten- dorst, 2021), though primarily for high-resource languages. Wu et al. (2022) demonstrated that PLMs exhibit significant performance degradation on low-resource languages, highlighting the chal- lenges for languages like Slovak. This performance gap has motivated a grow- ing body of work on low-resource and multilingual keyphrase extraction. Gao et al. (2022) proposed retrieval-augmented multilingual keyphrase gen- eration, while multilingual datasets like MAKED (Kulkarni et al., 2022), TermEval (Terryn et al., 2020), and the recent EUROPA dataset (Salaün et al., 2024) covering 24 European languages sup- port cross-lingual research. However, most multi- lingual work focuses on Romance and Germanic languages, with limited exploration of morpholog- ically rich Slavic languages. Morphological com- plexity presents unique challenges (Tsarfaty et al., 2013), as languages like Slovak exhibit extensive inflection, with a single lemma having dozens of surface forms, complicating keyphrase matching and evaluation. Critically, no prior work has sys- tematically quantified the impact of this inflectional variation on standard evaluation metrics—a gap our study addresses by reporting both exact and partial matching and analyzing the divergence be- tween them. Among Slavic languages, Slovak, Czech, and Polish are considered under-resourced, with only partial NLP infrastructure available (e.g., corpora, taggers, morphological analyzers) (Beliga and Martincic-Ipsic, 2016). Within the Slavic family, Czech has seen more development, with Docekal and Smrz (2022) testing a query-based method on 48,879 documents, though performance wors- ened on abstracts. Polish keyphrase extraction has also been studied (Giarelis et al., 2021; P ̨ezik et al., 2022), with mixed results for deep learning approaches. Slovak keyphrase extraction is still in its early stages.Varga et al. (2022) applied unsuper- vised methods to five court decisions, with TF-IDF achieving the best results. Zelinka (2023) made an important first step by collecting approximately 9,000 documents from the Slovak thesis registry; however, the dataset inherits noise typical of large- scale web harvesting—mixed-language content, inconsistent metadata, and variable keyphrase formatting—challenges that motivate the more extensive cleaning pipeline we describe in Sec- tion 4.2. The main datasets for benchmarking keyphrase extraction in English include KPTimes (Gallina et al., 2019) (news articles) and, for scientific ar- ticles, KP20K (Meng et al., 2017), Inspec (Hulth, 2003), and SemEval (Kim et al., 2010). The KP20K dataset, with 530,000 training and 20,000 test doc- uments, is most relevant to our work. While English benefits from hundreds of thousands of annotated documents, Slavic languages rely on datasets or- ders of magnitude smaller, making it difficult to determine whether low extraction performance re- flects genuine model limitations or merely insuffi- cient data. 3. Methodology We select baselines that span three domi- nant paradigms—statistical (YAKE), graph-based (TextRank), and embedding-based (KeyBERT)— following the configuration of Zelinka (2023) to en- able direct comparison with prior Slovak work. Cru- cially, all three are extractive: they return surface tokens from the input text, making them structurally vulnerable to the morphological mismatch between in-text inflected forms and author-assigned canon- ical keyphrases. This property is not specific to Slovak; it affects every morphologically rich lan- guage and makes these baselines a meaningful diagnostic tool for quantifying the impact of inflec- tion on evaluation metrics. We contrast them with KeyLLM, a large-language-model-based approach that generates keyphrases and is therefore free to produce canonical forms regardless of the surface forms present in the text. 3.1. YAKE YAKE (Campos et al., 2020) is an unsupervised keyword extraction algorithm that uses statistical methods to analyze document words. It scores keyphrases based on factors such as casing, term position, frequency, and relatedness to context. We use theyakelibrary with Slovak stopwords and a uni- to bi-gram n-gram range, consistent with prior work. 3.2. TextRank TextRank (Mihalcea and Tarau, 2004) is another un- supervised approach that employs a graph-based structure, where keyphrases form nodes in a graph, and their relevance is computed iteratively using a variant of the PageRank algorithm (Rogers, 2002). Thepkelibrary is used with custom adjustments to favor uni- and bi-grams. 3.3. KeyBERT KeyBERT (Grootendorst, 2020) leverages BERT embeddings to extract keyphrases by measuring their cosine similarity to the entire document. The approach assumes that a word effectively describ- ing the document will have a vector representation close to that of the document itself. In our ap- proach, we utilize thekinit/slovakbert-sts-stsb (Pikuliak et al., 2022), which has been fine-tuned for Slovak, and apply it to generate uni- and bi- gram keyphrases. 3.4. KeyLLM KeyLLM builds upon KeyBERT and leverages Large Language Models (LLMs) such as GPT (Radford et al., 2018) for keyphrase extraction. Unlike KeyBERT, which relies on cosine similar- ity between BERT embeddings, KeyLLM uses prompted LLMs (GPT-3.5 (Ouyang et al., 2022) or GPT-4 (Achiam et al., 2023)) to directly generate keyphrases from the document text. To reduce computational costs, KeyLLM of- fers an optional embedding-based clustering ap- proach where documents are encoded using a sen- tence transformer (all-MiniLM-L6-v2for English orkinit/slovakbert-sts-stsbfor Slovak). A sim- ilarity threshold parameter determines clustering: higher thresholds (0.90) create smaller clusters requiring more LLM calls, while lower thresholds (0.75) reduce costs by grouping diverse documents together. We analyze both versions—with cluster- ing at various thresholds (0.75, 0.85, 0.90) and without embeddings—representing the first appli- cation to Slovak. 4. Dataset 4.1. Data Collection Our data source is the Central Register of Theses and Dissertations in Slovakia (https://opac.crzp.sk/).It is a publicly avail- able database of bachelor, diploma, rigorous, and similar types of theses from Slovak universities. Figure 2: Filtering process and percentage of rows in the original dataset The theses have been stored since 2010, which provided us with a substantial number of avail- able documents. At the time of our data collection (2024), there were a total of 794,527 available doc- uments. Each thesis record in the register includes what the system labels a primary abstract (written in the thesis’s main language, typically Slovak) and a secondary abstract (a translation, typically into English), along with author-assigned keyphrases and metadata such as university, year of comple- tion, and supervisor. The website’s session limit of 20,000 documents required strategic filtering by year and alphabetical prefixes to access nearly all records through paral- lel processing. We successfully retrieved 793,722 documents, with the remaining 805 excluded due to non-alphabetical titles or unreadable text. 4.2. Data Cleaning Starting from 793,722 scraped records, we ap- plied a multi-stage cleaning pipeline (detailed in Appendix B) consisting of: (1) duplicate removal, prioritizing records with complete abstracts and keyphrases; (2) recovery of keyphrases appended to abstract text by universities lacking a dedicated keyphrase field; (3) removal of noisy metadata prepended to abstracts (e.g., author names, thesis type, page counts); (4) language verification using thelingualibrary (Lingua, 2021), which revealed that 20% of Slovak-labeled abstracts were actu- ally in English; (5) keyphrase normalization using StanzaPOS tagging (Qi et al., 2020) to split con- catenated keyphrase lists and enforce a maximum length of four words; and (6) length-based filtering, retaining abstracts of 500–2000 characters with 4–15 keyphrases. The overall filtering process is summarized in Figure 2. 4.3. Dataset Statistics After cleaning, we obtained 227,432 records, which we release as the SlovKE (Slovak Keyphrase Ex- traction) dataset. We split them randomly (with a fixed seed and without explicit stratification) into training (80%), validation (10%), and test (10%) sets. We additionally created a larger 20% test split (Test45K) for preliminary analysis; since no significant differences were found between the two test splits, we report results only for the 10% split (Test22K). The Test45K documents were not added back to the training set. As all evaluated models are unsupervised, the training split was not used for model training; it is included to support future supervised approaches. The dataset statistics, compared to other schol- arly datasets, are presented in Table 1. As shown, our dataset aligns well in size with larger keyphrase extraction datasets in English, even in terms of sta- tistical properties. Notably, our absent-keyphrase rate (37%) is comparable to that of KP20K (42.6%) and Inspec (44.3%). This indicates that the chal- lenge of absent keyphrases in Slovak is not dis- proportionately harder than in English—contrary to what might be expected for a morphologically rich language—and that models and evaluation proto- cols developed for English absent-keyphrase gen- eration can be meaningfully applied cross-lingually. As pointed out by Meng et al. (2017), a substantial absent-keyphrase rate is not undesirable; it can enhance model training and evaluation by testing the model’s ability to generate keyphrases that are not explicitly present in the text. The keyphrases predominantly consist of unigrams and bigrams. It is worth noting that author-assigned keyphrases are inherently subjective:differ- ent authors may select different terms for similar content, and individual variation in keyphrase granularity, abstraction level, and terminology is well documented even in English datasets (Kim et al., 2010). In our dataset, this subjectivity is compounded by the fact that keyphrases are assigned by students with varying levels of academic experience, without standardized guidelines across universities. Rather than treating this as a limitation, we view it as representative of the realistic annotation conditions under which keyphrase extraction systems must operate, particularly in low-resource settings where expert re-annotation is impractical. Additional statistics on abstract length distribu- tion and token counts are provided in Appendix C. 5. Evaluation 5.1. Evaluation Metrics We compare author-assigned keyphrases with those extracted by the models, first lemmatizing both sets to account for word variations. Two com- mon matching techniques are used: Exact match- ing, where the extracted keyphrase must exactly match an author-assigned keyphrase, and Partial matching, where a match is counted if any frag- ment of the extracted keyphrase overlaps with a fragment of a golden keyphrase. Evaluations are performed usingF1score at a fixed numberkof keyphrases (denoted as F1@k), with recall capped at 1 when the num- ber of matches exceeds the golden set (ifk > |golden set|). The finalF1score is the micro- average across all test documents. We note that both metrics were designed primarily for English, where inflectional variation is limited. In morpho- logically rich languages, exact matching penalizes correct concepts expressed in a different case or number, while partial matching may over-reward coincidental token overlap. Reporting both metrics Dataset (Test)InspecKP20KSemEvalZelinkaSlovKE AnnotatorExpertAuthorAuthor+ReaderAuthorAuthor Document TypeAbstractsAbstractPapers (6-8p)AbstractsAbstracts # Documents (Train)1000530K144-182K # Documents (Validation)50020K--22K # Documents (Test)50020K1009K22K Avg. no. Words per Document134.61767,961125.63134.02 Avg. no. of Keyphrases9.85.314.76.085 Avg. len of Keyphrase2.32.62.21.681.68 % of absent Keyphrases44.3142.619.749.137 Table 1: Dataset comparison for Test sets and analyzing their divergence therefore provides a more informative picture of model quality than either metric alone—a practice we recommend for any keyphrase evaluation on morphologically rich languages. 6. Results 6.1. Comparing Baseline Models We compared models from three categories— statistical (YAKE), graph-based (TextRank), and pre-trained (KeyBERT)—using the same configura- tion as in prior work (e.g.,F1@6, uni- and bi-gram extraction). Figure 3 provides an overview of the comparison with the Zelinka dataset. Figure 3: TheF1@6score for exact (stronger color) and partial (lighter color) matches is shown for the baseline models, comparing Zelinka and SlovKE. The evaluations (Table 2; Table 3) showed slight improvements in exact matching, with YAKE po- tentially benefiting from the removal of noisy text during the data cleaning stage. This was because we cleaned the abstracts by removing words at the beginning of the text, such as ABSTRACT or Bach- elor Thesis. Due to its underlying scoring mech- anism, YAKE assigns high importance to such to- kens. For partial matching, despite the notable improvement, the ranking of top-performing mod- els remains consistent. Additional analysis across various values ofk(Figure 4) shows that YAKE Exact Matching ModelMetricSlovKE Zelinka YAKE Precision10.46.8 Recall13.28.6 F1@611.67.5 TextRank Precision7.76.5 Recall9.88.3 F1@68.67.2 KeyBERT Precision4.03.8 Recall4.74.4 F1@64.34.0 Table 2: Comparison of keyphrase extraction mod- els using Exact Matching. Partial Matching ModelMetricSlovKE Zelinka YAKE Precision36.926.6 Recall46.033.2 F1@640.629.1 TextRank Precision48.143.9 Recall56.350.6 F1@651.546.5 KeyBERT Precision46.640.8 Recall53.746.4 F1@649.442.8 Table 3: Comparison of keyphrase extraction mod- els using Partial Matching. achieves the highestF1score in exact matching, while TextRank performs best in partial matching. For allk, the gap between exact and partial matching is striking: YAKE’s exactF1@6of 11.6 vs. partialF1@6of 40.6 represents a 29-point spread, and TextRank shows an even larger gap of 43 points (8.6 vs. 51.5). We argue that this exact– partial gap is a diagnostic metric in its own right: it quantifies the degree to which morphological in- flection distorts automated evaluation. In English, where inflection is minimal, the two metrics are far closer. The magnitude of the gap we observe is therefore not merely a Slovak-specific finding—it is predictable for any language with rich nominal and verbal morphology (e.g., Czech, Polish, Finnish, Turkish, Hungarian) and suggests that standard exact-matchF1systematically underestimates ex- tractive model performance in such languages. Ide- ally, we aim not only to achieve a higherF1score for both exact and partial matching but also to nar- row the gap between them, ensuring a more reli- able evaluation. Figure 4: F1 score of exact and partial match for baseline models using differentkvalues, whereO denotes k =|golden set|. 6.2. KeyLLM Results To extend our model evaluations, we employed GPT-3.5-turbo with prompts in both Slovak and English, yielding no significant differences, thus we present only results generated using an English prompt (see Appendix A). As GPT-3.5-turbo is costly due to token pro- cessing, we initially used embeddings for cost- saving. We employed two Sentence Transform- ers:kinit/slovakbert-sts-stsbfor Slovak, and all-MiniLM-L6-v2for English. This method in- volved clustering similar documents and selecting one representative for keyphrase extraction, re- ducing document processing. The model uses a threshold parameter: if the similarity score exceeds the threshold, documents are grouped together; otherwise, they are processed separately. A higher threshold (e.g., 0.9) results in more precise group- ing with fewer documents per group, while a lower threshold (e.g., 0.7) reduces costs by clustering more diverse documents, though it may introduce noise. However, this approach was ineffective with the recommended 0.75 threshold (Figure 5). Higher thresholds (0.85) improved results, especially forall-MiniLM-L6-v2, though both models per- formed similarly at 0.90, with results comparable to keyphrase extraction without embeddings. The computational time and number of processed doc- uments were also similar for the 0.90 threshold or the model without embeddings. Figure 5: F1 score for KeyLLM with embeddings for thresholds 75, 85, 90, and without embeddings for two Sentence Transformers. The results suggest that embeddings perform best when small clusters are formed (higher thresh- old), but for optimal keyphrase extraction, it is bet- ter to avoid them. Using theall-MiniLM-L6-v2 transformer at thresholds of 0.80–0.85 yields signif- icant improvements over baseline models for exact matching, making it well-suited for large datasets where some accuracy trade-off is acceptable. Notably, KeyLLM’s exact-matchF1@6(∼15.2) is substantially higher than the best extractive base- line (YAKE, 11.6), while its partial-matchF1@6 (∼49.1) is comparable. The resulting exact–partial gap for KeyLLM (∼34 points) is narrower than that of the extractive models in relative terms: KeyLLM closes roughly 30% of the gap that YAKE exhibits atF1@6. This pattern provides direct evidence that generative models are more robust to morpho- logical inflection than extractive methods, because they can produce canonical lemma forms rather than copying inflected surface tokens. We expect this advantage to generalize to other morphologi- cally rich languages where the same extractive– generative asymmetry holds. However, partial matching showed no significant improvement over the best baselines, suggesting that KeyLLM’s gains are concentrated in form normalization rather than in identifying additional relevant concepts. We therefore conducted a further manual evaluation to disentangle these factors. 6.3. Manual Evaluation While standard keyphrase extraction evaluation re- lies on automatic matching against gold standard keyphrases (Kim et al., 2010), we additionally per- formed manual evaluation on a subset of 100 ran- domly selected documents to better understand the quality of our dataset and the practical utility of extracted keyphrases. This evaluation aimed to assess whether KeyLLM and the YAKE model extract relevant keyphrases beyond exact matches with author-assigned keyphrases. We also tested YAKE with a larger n-gram range (1,3) to examine whether this adjustment improved results. The manual evaluation was conducted to com- plement automated matching metrics and to bet- ter assess the semantic relevance of extracted keyphrases. A total of 100 documents were ran- domly sampled from the test set. Each docu- ment was evaluated with reference to the author- assigned keyphrases and the keyphrases extracted by the evaluated models. For each document, annotators reviewed the abstract and the corresponding list of extracted keyphrases. A keyphrase was considered correct if it satisfied the following criteria: 1. Topical relevance: The keyphrase referred to a central concept explicitly discussed in the abstract. 2.Conceptual adequacy: The keyphrase rep- resented a meaningful concept rather than a generic modifier (e.g., standalone adjectives were not considered sufficient). 3.Non-redundancy: When multiple surface vari- ants of the same concept were extracted (e.g., inflected forms or reordered phrases), only the most representative form was retained. 4. Linguistic validity: Morphological variants (e.g., inflected forms) were accepted as cor- rect if they clearly referred to the same concept as the author-assigned keyphrase. Two evaluation settings were defined: 1.ManualExact,whichcountedonly keyphrases that matched or were equiv- alent to the author-assigned keyphrases (including acceptable morphological variants). 2.Manual Extended, which additionally in- cluded contextually relevant keyphrases present in the abstract but omitted from the author-assigned list (e.g., methodologies, named entities, or key concepts discussed but not listed as keyphrases). The evaluation was primarily conducted by one of the authors, with a second author independently annotating a subset of 30 documents to assess inter-annotator agreement. Both annotators were native Slovak speakers with a background in NLP. Agreement was measured using Cohen’s kappa coefficient, yieldingκ = 0.61, which indicates sub- stantial agreement according to standard interpre- tation guidelines (Landis and Koch, 1977). We note that keyphrase relevance judgments are in- herently subjective—even on English benchmarks, inter-annotator agreement for keyphrase annota- tion rarely exceedsκ = 0.70(Kim et al., 2010)— and the morphological variability of Slovak intro- duces additional ambiguity in deciding whether an inflected form constitutes a match. 6.3.1.Key Observations and Failure Analysis Manual vs. Automated Evaluation: Manual evaluations, particularly for exact matches, consis- tently outperformed automated methods. While au- tomated algorithms struggled with lexical variation and word ordering, manual evaluation was able to recognize semantically related concepts expressed in different forms. For instance, in a thesis whose topic was obesity, the author-assigned keyphrases covered the core medical condition but omitted con- cepts that were explicitly discussed in the abstract, such as the psychological consequences of obe- sity or diagnostic procedures. KeyLLM extracted these as additional keyphrases (e.g., psycholog- ický problém, diagnostika), which were judged rel- evant under our Manual Extended criterion despite not appearing in the author-assigned list. These are not matches for obezita but rather complemen- tary keyphrases that improve topic coverage. KeyLLM also outperformed YAKE in root-form handling and contextual understanding. It more re- liably converted extracted keyphrases to their base forms and was able to connect words into coher- ent multi-word keyphrases (e.g., linguistic charac- teristics or social characteristics), even when the individual words did not appear contiguously in the text. This ability to produce lemmatized, canonical- form keyphrases without an explicit morphological analyzer is a practical advantage of generative LLMs that extends beyond Slovak: any language where authors assign keyphrases in citation form but text contains inflected forms would benefit sim- ilarly. Another notable strength of KeyLLM was its ability to capture distinct keyphrases without redundancy, including named entities such as com- pany names, software tools, and medical systems. In contrast, YAKE frequently extracted semanti- cally similar words or partial fragments of other keyphrases. For YAKE, exact-match performance improved under manual evaluation, as annotators could rec- ognize morphologically inflected forms as equiv- alent to author-assigned keyphrases. However, extended matching did not outperform partial auto- mated matching, since manual evaluation retained only the most representative keyphrase per con- cept and excluded repetitive surface variations. When increasing the n-gram range to (1,3), YAKE’s manual evaluation scores dropped significantly (Figure 6), as the model increasingly prioritized irrelevant trigrams over the document’s main top- ics. Figure 6: Comparison of precision, recall, and F1 score of automatically and manually evaluated YAKE model with n-gram range (1,3) 6.3.2. Failure Cases Morphological mismatch was the most fre- quent failure mode observed for YAKE. Since YAKE extracts keyphrases strictly in the form in which they appear in the text, Slovak’s rich inflec- tional morphology regularly caused mismatches with author-assigned keyphrases even when the correct concept was identified (e.g., aktívnej poli- tiky trhu instead of aktívna politika trhu). This also affected cases where a keyphrase appeared multi- ple times in different inflected forms or word orders, and YAKE selected a surface form that did not match the author’s version. This failure mode is in- herent to any extractive method operating on a mor- phologically rich language: the same mismatch pat- tern would arise in Czech, Polish, Finnish, or Turk- ish, where a single lemma can surface in dozens of case- or agreement-inflected forms. The practical implication is that extractive keyphrase methods re- quire a lemmatization or morphological normaliza- tion step to be fairly evaluated in such languages. Wrong granularity was particularly prominent for YAKE with the (1,3) n-gram range. The model frequently prioritized irrelevant trigrams that were either surface-level variants of previously extracted keyphrases or generic constructions with little dis- criminative value (e.g., analysis of trends, trends in data). This failure mode is partially obscured by au- tomated evaluation, where partial matching counts such variations as correct. Manual evaluation re- jected these phrases as redundant, explaining the gap between manual and automated scores ob- served with larger n-gram ranges (Figure 6). Semantic redundancy was another recurring is- sue for YAKE. The model often occupied multiple extraction slots with surface variants of the same concept (e.g., extracting rozvoj, rozvoj vidieka, and vidiecky rozvoj as separate keyphrases), thereby reducing the effective coverage of distinct docu- ment topics. Unmotivated adjective extraction represented the primary failure case for KeyLLM. The model fre- quently extracted standalone adjectives that lacked sufficient topical specificity when not paired with a noun. This behavior also explains the steeper decline in precision with increasingk: while highly relevant keyphrases are prioritized early, adjective- only extractions accumulate as more keyphrases are considered. 7. Conclusion This work presents SlovKE, a 227,432-document Slovak keyphrase extraction dataset – a 25-fold increase over prior work – created through system- atic data cleaning and quality control. Our evaluation of baseline models reveals that data quality has a direct impact on keyphrase ex- traction: after cleaning, YAKE’s exactF1@6im- proved from 7.5 to 11.6. More notably, the KeyLLM model (GPT-3.5-turbo) achieved an exactF1@6 of approximately 15.2, substantially narrowing the gap between exact and partial matching that char- acterizes all baseline models. This gap – with the best exactF1@6at 11.6 versus the best partial F1@6at 51.5 for baseline methods – reflects a fundamental evaluation challenge for morpholog- ically rich languages, where surface-form varia- tion makes strict matching unreliable and lenient matching overly generous. We argue that this exact–partial divergence is not merely a Slovak- specific artifact but a systematic issue that affects keyphrase evaluation in any language with produc- tive inflectional morphology, including other Slavic, Finno-Ugric, and Turkic languages. Manual evaluation on 100 documents (κ = 0.61) confirmed that KeyLLM captures semantically rele- vant keyphrases that automated metrics systemati- cally miss – including named entities, methodologi- cal terms, and contextually central concepts absent from author-assigned gold standards. These find- ings suggest that standard automated evaluation underestimates the true quality of keyphrase ex- traction in morphologically rich languages and that developing morphology-aware evaluation protocols is an important direction for the field. Beyond its immediate contribution to keyphrase extraction, SlovKE constitutes foundational in- frastructure for Slovak NLP: at 227K docu- ments with structured metadata, it is among the largest publicly available collections of Slo- vak scientific text and can support supervised keyphrase generation, document classification, cross-lingual transfer to typologically similar lan- guages (Czech, Polish, Croatian), and the devel- opment of morphology-aware evaluation metrics. The dataset is available athttps://huggingface. co/datasets/NaiveNeuron/SlovKEand the evalu- ation code athttps://github.com/NaiveNeuron/ SlovKE. 8. Limitations Dataset scope. While our dataset is the largest of its kind for Slovak, it draws exclusively from theses and dissertations registered in the Central Register, which may not capture the full diversity of Slovak academic writing. Extending coverage to journal articles, conference papers, and technical reports would broaden domain representation and is a natural next step for future dataset expansion. Residual noise. Despite extensive cleaning, some records may contain formatting inconsisten- cies or uninformative keyphrases, as neither ab- stracts nor author-assigned keyphrases underwent expert review. Developing automated quality scor- ing (e.g., keyphrase informativeness classifiers) could help identify and filter such cases, and our dataset provides a large enough training signal to support such efforts. Absence of keyphrase rankings. Our data source does not provide ranked keyphrases, pre- cluding evaluation with rank-sensitive metrics such as MRR (Voorhees and Tice, 2000) or MAP (Liu, 2009). Crowdsourced or model-assisted ranking annotation over a subset of the dataset would en- able these evaluations and constitutes a concrete direction for future work. Scale of manual evaluation. Our manual eval- uation covered 100 documents with two annota- tors (κ = 0.61). While this was sufficient to iden- tify systematic patterns in model behavior, scal- ing this evaluation – potentially through crowd- sourcing or active-learning-based annotation – would strengthen the generalizability of our find- ings across domains and document types. Model optimization. The models evaluated in this work were not fine-tuned for Slovak. Given that our dataset provides over 180,000 training documents, supervised fine-tuning of sequence labeling or generative models represents the most direct path to improved performance. In particular, fine-tuning Slovak-specific language models (e.g., SlovakBERT) for keyphrase extraction could yield substantial gains and would directly leverage the resource we provide. Unsupervised methods only. We evaluated only unsupervised and prompting-based ap- proaches. Training supervised models on this dataset – including transformer-based sequence taggers and keyphrase generation models – is an important next step that could reveal richer struc- ture in the data and substantially improve extraction accuracy. 9. Data Availability To support reproducibility and future research, we publicly release the SlovKE dataset under the Cre- ative Commons Attribution 4.0 International Li- cense (C BY 4.0) athttps://huggingface.co/ datasets/NaiveNeuron/SlovKE . The dataset com- prises 227,432 cleaned Slovak scientific abstracts with author-assigned keyphrases, split into train- ing (182,745 documents), validation (22,343), and test (22,344) sets.Each record contains the abstract text, keyphrases, and metadata (univer- sity, year, document type). All evaluation scripts and preprocessing code are available athttps: //github.com/NaiveNeuron/SlovKEunder the MIT License. We encourage the community to use this dataset not only for keyphrase extraction but also for related tasks such as document classification, cross-lingual transfer to other Slavic languages, and the development of morphology-aware evalua- tion metrics. Acknowledgements This research was partially funded by the EU NextGenerationEU through the Recovery and Re- silience Plan for Slovakia under the project No. 09I02-03-V01-00029. 10. Bibliographical References Josh Achiam, Steven Adler, Sandhini Agar- wal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Al- tenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Rabah Alzaidy, Cornelia Caragea, and C Lee Giles. 2019. Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents. In The world wide web conference, pages 2551– 2557. Marco Basaldella, Elisa Antolli, Giuseppe Serra, and Carlo Tasso. 2018. Bidirectional lstm recur- rent neural network for keyphrase extraction. In Italian Research Conference on Digital Libraries, pages 180–187. Springer. Slobodan Beliga and Sanda Martincic-Ipsic. 2016. Network-enabled keyword extraction for under- resourced languages. In Semantic Keyword- Based Search on Structured Data Sources - COST Action IC1302 Second International KEY- STONE Conference, IKC 2016, Cluj-Napoca, Romania, September 8-9, 2016, Revised Se- lected Papers, volume 10151 of Lecture Notes in Computer Science, pages 124–135. Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. 2020. Yake! keyword extraction from single documents using multiple local features. Information Sciences, 509:257–289. Martin Docekal and Pavel Smrz. 2022. Query- based keyphrase extraction from long docu- ments. arXiv preprint arXiv:2205.05391. Ygor Gallina, Florian Boudin, and Beatrice Daille. 2019.Kptimes: A large-scale dataset for keyphrase generation on news documents. arXiv preprint arXiv:1911.12559. Yifan Gao, Qingyu Yin, Zheng Li, Rui Meng, Tong Zhao, Bing Yin, Irwin King, and Michael Lyu. 2022. Retrieval-augmented multilingual keyphrase generation with retriever-generator it- erative training. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1233–1246, Seattle, United States. Association for Computational Linguistics. Nikolaos Giarelis, Nikos Kanakaris, and Nikos Karacapilidis. 2021. A comparative assessment of state-of-the-art methods for multilingual un- supervised keyphrase extraction. In IFIP In- ternational Conference on Artificial Intelligence Applications and Innovations, pages 635–645. Springer. Maarten Grootendorst. 2020. Keybert: Minimal keyword extraction with bert. Maarten Grootendorst. 2021. Keyllm: Keyphrase extraction using large language models. https://maartengr.github.io/KeyBERT/ guides/keyllm.html.Accessed:2024-03- 04. Kazi Saidul Hasan and Vincent Ng. 2014. Au- tomatic keyphrase extraction: A survey of the state of the art. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1262–1273, Baltimore, Maryland. Associ- ation for Computational Linguistics. Anette Hulth. 2003. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 conference on Empir- ical methods in natural language processing, pages 216–223. Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010. SemEval-2010 task 5 : Automatic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 21– 26, Uppsala, Sweden. Association for Computa- tional Linguistics. Pruthwik Kulkarni, Krishnandu Singh, Man- ish Kulkarni, and Manish Shrivastava. 2022. MAKED: A multi-lingual automatic keyword ex- traction dataset. In Proceedings of the Thirteenth Language Resources and Evaluation Confer- ence (LREC 2022), pages 3614–3623, Marseille, France. European Language Resources Associ- ation. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for cate- gorical data. Biometrics, 33(1):159–174. Lingua. 2021. Lingua: Language detection li- brary.https://github.com/lingua-project/ lingua. Accessed: 2024-03-04. Ling Liu. 2009. Encyclopedia of database systems. Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation.arXiv preprint arXiv:1704.06879. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411. Thuy Dung Nguyen and Min-Yen Kan. 2007. Keyphrase extraction in scientific publications. In International conference on Asian digital li- braries, pages 317–326. Springer. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feed- back. Advances in neural information processing systems, 35:27730–27744. EiriniPapagiannopoulouandGrigorios Tsoumakas. 2020.A review of keyphrase extraction. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(2):e1339. Seoyeon Park and Cornelia Caragea. 2020. Sci- entific keyphrase identification and classifica- tion by pre-trained language models interme- diate task transfer learning. In Proceedings of the 28th International Conference on Computa- tional Linguistics (COLING), pages 5409–5419, Barcelona, Spain (Online). International Commit- tee on Computational Linguistics. Piotr P ̨ezik, Agnieszka Mikołajczyk, Adam Wawrzy ́ nski, Bartłomiej Nito ́ n, and Maciej Ogrod- niczuk. 2022. Keyword extraction from short texts with a text-to-text transfer transformer. In Asian Conference on Intelligent Information and Database Systems, pages 530–542. Springer. Matúš Pikuliak, Štefan Grivalský, Martin Konôpka, Miroslav Blšták, Martin Tamajka, Viktor Bachratý, Marian Simko, Pavol Balážik, Michal Trnka, and Filip Uhlárik. 2022. SlovakBERT: Slovak masked language model. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2022, pages 7156–7168, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages.arXiv preprint arXiv:2003.07082. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Tech- nical report, OpenAI. Ian Rogers. 2002.The Google PageR- ank algorithm and how it works.https: //w.cs.princeton.edu/~chazelle/courses/ BIB/pagerank.htm. Dhruva Sahrawat, Debanjan Mahata, Haimin Zhang, Mayank Kulkarni, Agniv Sharma, Rakesh Gosangi, Amanda Stent, Yaman Kumar, Ra- jiv Shah, and Roger Zimmermann. 2020. Keyphrase extraction as sequence labeling using contextualized embeddings. In Advances in In- formation Retrieval: 42nd European Conference on IR Research, ECIR 2020, pages 328–335, Lisbon, Portugal. Springer. Olivier Salaün, Frédéric Piedboeuf, Guillaume Le Berre, David Alfonso-Hermelo, and Philippe Langlais. 2024. EUROPA: A legal multilingual keyphrase generation dataset. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 12718–12736, Bangkok, Thailand. Association for Computational Linguistics. Mingyang Song, Yi Feng, and Liping Jing. 2023. A survey on recent advances in keyphrase ex- traction from pre-trained language models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2108–2119, Dubrovnik, Croatia. Association for Computa- tional Linguistics. Ayla Rigouts Terryn, Véronique Hoste, and Els Lefever. 2020. In no uncertain terms: A dataset for monolingual and multilingual automatic term extraction from comparable corpora. Language Resources and Evaluation, 54(2):385–418. Reut Tsarfaty, Djamé Seddah, Sandra Kübler, and Joakim Nivre. 2013. Parsing morphologically rich languages: Introduction to the special issue. Computational Linguistics, 39(1):15–22. Dávid Varga, Simon Horvát, Zoltán Szoplák, L’ubomír Antoni, Stanislav Krajci, Peter Gurský, and Laura Bachnáková Rózenfeldová. 2022. Keyphrase extraction from slovak court deci- sions.In Proceedings of the 22nd Confer- ence Information Technologies - Applications and Theory (ITAT 2022), volume 3226 of CEUR Workshop Proceedings, pages 142–150. CEUR- WS.org. Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC-8 question answering track. In Proceed- ings of the Second International Conference on Language Resources and Evaluation (LREC‘00), Athens, Greece. European Language Resources Association (ELRA). Di Wu, Wasi Uddin Ahmad, and Kai-Wei Chang. 2022.Pre-trained language models for keyphrase generation: A thorough empirical study. arXiv preprint arXiv:2212.10233. Andrej Zelinka. 2023.Keyphrase extraction from slovak scientific documents.Bache- lor’s thesis, Comenius University, Bratislava, Bratislava, Slovakia.Available athttps: //opac.crzp.sk/?fn=detailBiblioForm&sid= C03CB0FBA6588971D3A2BD1B6BF. A. KeyLLM Prompt Prompt: I have the following document: [DOCUMENT] Based on the information above, extract 15 keywords that best describe the topic of the text. Make sure to only extract keywords that appear in the text. Provide the extracted keywords as a list separated by a semicolon. Sort the keywords by their relevance to the text from the most relevant to the least relevant. Do not add any additional text such as "Keywords:" etc. B. Data Cleaning Details We identified and removed duplicates, priori- tizing documents with complete abstracts and keyphrases. Our final dataset mainly consists of Bachelor and Diploma theses. We observed a significant absence of primary abstracts and keyphrases, especially in the first five years (Figure 7), while missing secondary abstracts and keyphrases persisted across later years. Figure 7: Percentage of missing primary abstracts, secondary abstracts, and keyphrases per year As shown in Table 4, some top-contributing universities exhibit a high proportion of missing keyphrases. We discovered that some universi- ties lack a designated keyphrase field, prompting students to append keyphrases at the end of their abstracts. We attempted to recover these miss- ing keyphrases by identifying and extracting lists appended to abstracts. We removed overly short abstracts and keyphrases while also cleaning noisy entries. Many abstracts contained irrelevant metadata prepended to the text, such as: UniversityMiss. PAMiss. SAMiss. K UK BA8.6515.8499.94 STU BA10.39100.000.01 SPU Nitra0.0480.717.91 KU RK56.5560.6538.29 VŠD Sládk.0.8319.3386.10 Table 4: Top contributing universities with the per- centage of missing or short primary abstracts (PA), secondary abstracts (SA), or keyphrases (K). Per- centages exceeding 50% are highlighted in bold. ABSTRACT Peter, Doe: Analysis of Various Techniques ... [Bachelor thesis]. Comenius University in Bratislava. Faculty of Philosophy. Thesis Supervisor: Mgr. John Smith, PhD. Year of Defense: 2019. Number of Pages: 80. This thesis explores various techniques for successfully writing an abstract... The extra content, typically starting with the au- thor’s name or “ABSTRACT” and ending with the publication year, was irrelevant and could obscure the abstract. We identified these patterns and trimmed the abstracts accordingly. Weidentifiedabstractswithappended keyphrases (matching ‘kl’ú ˇ cové slová’ or vari- ations), used them to fill missing values, and removed them from the abstract text to prevent model bias. We focused on Slovak keyphrase extraction and found that some abstracts labeled as Slovak were actually written in other languages. Using the lingualanguage detection library, we found that 20% of Slovak-labeled abstracts (Figure 8) were actually in English. We implemented multilanguage detection for abstracts with mixed languages, filter- ing out those with over 40% non-Slovak content. Figure 8: Languages detected in primary abstracts labeled as Slovak Keyphrases were often submitted incorrectly, either as long blocks of text or mixed-language lists.We filtered out keyphrases longer than 300 characters and removed records with insuf- ficient keyphrases or multilingual versions. For keyphrases containing multiple items, we applied multilanguage detection with a 20% non-Slovak language threshold to exclude English phrases while retaining relevant terminology. Keyphrase fields often contained lists separated by commas, dots, dashes, or semicolons. We used theStanzatokenizer to identify words and POS tags, split keyphrases into individual items, removed non-alphabetical characters, and deleted records if any keyphrase exceeded four words. This improved keyphrase length consistency, as shown in Figure 9. (a) Before(b) After Figure 9: Distribution of the number of keyphrases before and after keyphrase cleaning Finally, we filtered abstracts by length (500–2000 characters) and keyphrase count (4–15), retaining documents sufficiently descriptive for model evalu- ation. C. Additional Dataset Statistics Figure 10 shows the distribution of abstract lengths before final filtering, and Figure 11 summarizes the number of sentences and tokens per abstract. Figure 10: Distribution of abstract lengths before final filtering (a)Sen- tences (b) Tokens (all tokens (Right), excluding digits and punctua- tion (Left)) Figure 11: Box plots for the mean number of sen- tences and tokens in abstracts