Paper deep dive

KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms

Soha Lee, Soojin Lee, Heesung Yang, Hyunju Song, Hyunji Lee, Jinsan An, Jeongwan Shin, Jin Hyun Park, Jun Lee, Hyeyoung Park, Kilim Nam

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 128

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 9/18/2026, 4:54:55 AM

Summary

The paper introduces KoNeoBench, a curated benchmark dataset designed to evaluate Large Language Models' (LLMs) understanding of Korean neologisms. Comprising 1,785 neologisms attested in online news since 2020 and curated by expert lexicographers, the dataset includes usage examples, word-formation analyses, and definitions. The authors define four evaluation tasks: contextual identification, source word identification, semantic/domain categorization, and definition generation. Experiments reveal that current LLMs struggle with recovering source components, distinguishing semantic categories, and generating accurate definitions, highlighting limitations in handling recent lexical and sociocultural changes in Korean.

Entities (9)

KoNeoBench · dataset · 98%LLM · model · 95%Soha Lee · person · 95%Korean Neologisms · linguistic-concept · 92%Kyungpook National University · organization · 90%National Institute of Korean Language · organization · 90%Yonsei University · organization · 90%NeoBench · dataset · 85%KLUE · dataset · 80%

Relation Signals (9)

KoNeoBench → contains → Korean Neologisms

confidence 95% · KoNeoBench is built on 1,785 Korean neologisms attested in online news since 2020...

KoNeoBench → evaluates → LLM

confidence 95% · KoNeoBench is a benchmark for evaluating LLMs' understanding of Korean neologisms.

Soha Lee → affiliatedwith → Kyungpook National University

confidence 90% · Soha Lee 1 * ... 1 School of Computer Science and Engineering, Kyungpook National University

KoNeoBench → hastask → Definition Generation

confidence 90% · Task 4: Definition Generation

KoNeoBench → hastask → Source Word Identification

confidence 90% · Task 2: Source Word Identification

KoNeoBench → hastask → Semantic Categorization

confidence 90% · Task 3: Neologism Categorization

KoNeoBench → hastask → Contextual Identification

confidence 90% · In this paper, we designed four tasks... Task 1: Contextual Identification

KoNeoBench → curatedby → National Institute of Korean Language

confidence 85% · Korean neologism data have been compiled annually under the supervision of the National Institute of Korean Language...

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging words and meanings. Existing Korean benchmarks are centered on established vocabulary and therefore provide limited coverage of such recent lexical change, and their English-oriented design makes it difficult to assess the typological properties of Korean, in which content words combine productively with functional morphemes. In this paper, we introduce KoNeoBench, a benchmark for evaluating LLMs' understanding of Korean neologisms. KoNeoBench is built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review. Each entry provides usage examples, word-formation analyses, and dictionary-style definitions. Based on this resource, we define four tasks and report results on recent models, together with a human baseline. Our experiments show that current LLMs exhibit clear limitations in recovering source components, distinguishing semantic categories, and generating accurate definitions. These results reveal specific aspects of recent Korean lexical change that remain challenging for current LLMs. KoNeoBench is available at this https URL .

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

124,798 characters extracted from source content.

Expand or collapse full text

KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms Soha Lee 1 * , Soojin Lee 2 * , Heesung Yang 1 , Hyunju Song 3 , Hyunji Lee 1 , Jinsan An 4 , Jeongwan Shin 5 , Jin Hyun Park 6 , Jun Lee 7 , Hyeyoung Park 1† , Kilim Nam 7† 1 School of Computer Science and Engineering, Kyungpook National University 2 International Exchange Department, Kyungpook National University 3 Dept. of Korean Language and Literature Education, Kyungpook National University 4 Dept. of Korean Language and Literature, Kyungpook National University 5 Daegu Gyeongbuk Institute of Science and Technology (DGIST) 6 Dept. of Computer Science and Engineering, Texas A&M University, 7 Dept. of Korean Language and Literature, Yonsei University Abstract Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging words and meanings. Exist- ing Korean benchmarks are centered on estab- lished vocabulary and therefore provide lim- ited coverage of such recent lexical change, and their English-oriented design makes it dif- ficult to assess the typological properties of Korean, in which content words combine pro- ductively with functional morphemes. In this paper, we introduce KoNeoBench, a bench- mark for evaluating LLMs’ understanding of Korean neologisms. KoNeoBench is built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review. Each entry provides us- age examples, word-formation analyses, and dictionary-style definitions. Based on this re- source, we define four tasks and report re- sults on recent models, together with a human baseline. Our experiments show that current LLMs exhibit clear limitations in recovering source components, distinguishing semantic categories, and generating accurate definitions. These results reveal specific aspects of recent Korean lexical change that remain challenging for current LLMs. KoNeoBench is available at https://github.com/bcmilab/ko-neobench/. 1 Introduction Large language models (LLMs) achieve strong per- formance on many language tasks, but their evalua- tion still relies largely on static benchmarks built from fixed linguistic data and knowledge, which may not fully capture aspects of recent language * These authors contributed equally to this work. † Corresponding author: Kilim Nam (nki@yonsei.ac.kr), Hyeyoung Park (hypark@knu.ac.kr) (a) GPT - 5.4 SolarPro - 3 ExaOne - 40B 0.3 0.8 0.7 0.6 0.5 0.4 Accuracies of Semantic Categorization (b) General Word Neologism  “크룽지”도 원해? Do you want ’crung-ji,’ too? 크룽(Krung, Thai, capital city)+ 지도(map)? 크(Croissant)+ 룽지(nurungji, crispy rice crust)+ 도(particle attachment, too) Figure 1: Challenges in understanding Korean neol- ogisms. (a) Semantic classification performances of LLMs neologisms vs. general words. (Details are given in Appendix C.1.) (b) A neologism formed by abbrevia- tion and blending. change. Prior work has highlighted this limitation in terms of temporal generalization and temporal misalignment, showing performance degradation on future text, time-sensitive facts, and updated world knowledge (Lazaridou et al., 2021; Luu et al., 2022; Dhingra et al., 2022; Vu et al., 2024; Zhu et al., 2025). Yet recent language change is not lim- ited to factual updates; it also emerges at the lexical and semantic levels through new words and shifts in meaning. As early indicators of such change, ne- ologisms reflect not only changes in linguistic form and meaning but also the social and cultural con- text of the language community (Nam et al., 2025). Accordingly, evaluating an LLM’s understanding of neologisms can provide insight into how well it captures recent lexical and sociocultural develop- ments in language use. Recent studies have begun to evaluate LLMs’ adaptation to language change through neologisms and informal expressions. NeoBench (Zheng et al., 2024) introduces benchmark data and tasks for arXiv:2609.19916v1 [cs.CL] 17 Sep 2026 neologism processing, while Mei et al. (2024) and Sun et al. (2024) examine LLMs’ understand- ing, detection, and source identification of inter- net slang, memes, and informal expressions. How- ever, these studies focus primarily on English, and benchmarks for Korean neologisms remain lim- ited. Within Korean, our preliminary results in Fig- ure 1(a) also reveal a substantial performance gap between established words and neologisms across different LLMs, motivating a closer examination of their understanding of Korean neologisms. Beyond this performance gap, Korean neolo- gisms exhibit linguistic characteristics that war- rant explicit consideration in benchmark design. Korean is an agglutinative language in which parti- cles and inflectional endings attach to lexical stems. Figure 1(b) illustrates how a neologism formed through abbreviation and/or blending can combine with a grammatical morpheme, requiring models to identify both the neologism boundary and the func- tion of the attached morpheme. Korean also con- tains many hybrid neologisms formed from mul- tiple lexical strata, including native Korean, Sino- Korean, and loanwords. These properties make Ko- rean neologisms particularly difficult to interpret and analyze, and they should therefore be taken into account when evaluating model understanding. To address the limited coverage of Korean ne- ologisms in existing benchmarks, we introduce KoNeoBench, a benchmark for multifaceted eval- uation of LLMs’ understanding of Korean neol- ogisms. It is built from a list of Korean neolo- gisms collected and annotated through expert lexi- cographic review. It comprises four tasks: (1) Cloze task for contextual understanding of neologism us- age, (2) source word identification aligned with na- tive speakers’ interpretation, (3) semantic and spe- cialized domain categorization, and (4) dictionary- style definition generation. Using KoNeoBench, we evaluate multiple LLMs and analyze their strengths and limitations in processing Korean neologisms from various perspectives. The main contributions of this paper are as follows. • We present KoNeoBench, a Korean neolo- gism dataset compiled and annotated by lexi- cography experts. • KoNeoBench is the first benchmark to pro- vide carefully designed test sets for multi- faceted evaluation of LLMs’ understanding of Korean neologisms. •Using these Korean-specific test sets, we eval- uate a range of LLMs and analyze the major challenges and error patterns involved in pro- cessing Korean neologisms. 2 Related Works Temporal Drift and Robustness in LLMs Re- search on the robustness of LLMs under temporal change has primarily examined performance degra- dation on post-training data (Lazaridou et al., 2021; Luu et al., 2022) and time-sensitive downstream tasks such as question answering (Chen et al., 2021; Liska et al., 2022; Dhingra et al., 2022; Agarwal and Nenkova, 2022). Other studies have shown that emerging entities and shifts in word meaning or frequency can likewise affect model performance (Rijhwani and Preotiuc-Pietro, 2020; Onoe et al., 2022; Pramanick et al., 2022). Together, these find- ings motivate the study of neologisms as a lexical- level manifestation of temporal language change. Neologism Research in LLMs Research on ne- ologisms has traditionally focused on collecting and classifying new words, analyzing their forma- tion and emergence, and tracing their diffusion in online communities (Pinter et al., 2020; Ryskina et al., 2020; Zhu and Jurgens, 2021). More recently, studies have begun to evaluate how well LLMs un- derstand rapidly changing expressions such as neol- ogisms, slang, and memes (Zheng et al., 2024; Sun et al., 2024; Mei et al., 2024). Unlike traditional work on out-of-vocabulary (OOV) items, this line of research emphasizes novel forms and meanings, word-formation patterns, and community-based usage (Garneau et al., 2018; Nayak et al., 2020; Araabi et al., 2022). Earlier computational studies, for example, investigated the recovery of source words in English lexical blends and the incorpora- tion of linguistic knowledge about neologism for- mation (Cook and Stevenson, 2010; Cook, 2010), while recent work has expanded to lexical profi- ciency, scientific neologism translation, slang sense alternation, and culturally or regionally specific slang (Ciaccio et al., 2025; Lerner and Yvon, 2025; Aloraini et al., 2026; Wu and Sun, 2025; Wuraola et al., 2026). Among these studies, NEO-BENCH (Zheng et al., 2024) is the most closely related to our work. It evaluates recent English neologisms through ma- chine translation, cloze question answering, defini- tion generation, and perplexity, with an emphasis on model robustness to lexical temporal drift. Ko- NeoBench complements this perspective by focus- ing on Korean lexical change and evaluating multi- ple aspects of neologism knowledge, including ne- ologism recognition, source-component recovery, semantic and specialized-domain categorization, and definition generation. In particular, its expert- curated lexical annotations allow us to evaluate not only whether a model recognizes or interprets a recent expression in context, but also whether it captures the word-formation and semantic proper- ties that characterize Korean neologisms. Korean NLP Benchmarks and Neologism Ko- rean NLP has also seen the development of various evaluation resources. Benchmarks such as KLUE, KoBEST, HAE-RAE Bench, and Ko-H5 provide important foundations for evaluating understand- ing, reasoning, knowledge, cultural context, and LLM performance (Park et al., 2021; Jang et al., 2022; Son et al., 2024; Park et al., 2024). However, they mainly target general language understanding or relatively static linguistic knowledge. In paral- lel, linguistic research on Korean neologisms has examined not only data collection but also word- formation patterns, usage contexts, and lexicaliza- tion processes (Moon, 1999; Lee and Kim, 2014; Nam, 2015; Lee, 2006, 2024). Building on these two lines of research, KoNeoBench provides a lin- guistically grounded benchmark to evaluate how well LLMs understand the recent Korean lexical change across morphological, semantic, and con- textual dimensions. 3 KoNeoBench: A Benchmark Dataset of Korean Neologisms 3.1 Data Collection Korean neologism data have been compiled an- nually under the supervision of the National In- stitute of Korean Language since 1994. We use 1,785 neologisms listed in annual publications pro- duced since 2020 under the same methodology. The source corpus consists of articles from major gen- eral and business newspapers in Korea. Candidate neologisms were first extracted automatically as out-of-vocabulary items, and were then screened and supplemented by lexicography experts. The final list for each year was determined by verify- ing first occurrence dates in Naver News, which contains article data from around 130 media out- lets. Figure 2(a) illustrates the overall collection pipeline, and Appendix A.1 provides additional details. Data Collection Process Validated Benchmark Dataset 1,785 Neologisms Corpus Korean Major News Articles (a) 1,627,371 Candidates (2020-2024) Auto Automatic candidate extraction •Noun-centered extraction •OOV Detection against Standard Korean Language Dictionary •Removal of inflected forms •Removal of plural variants Expertscreening •NewConcept/newmeaning •Exclusioncriteria Lexicographic expert validation Manualdiscovery •Usingpragmaticmarkers (+516neologisms) First-occurrenceverification •KoreanMajorNewsArticles •ObservationYear 1,269 validated neologisms 1,785 Candidates Human Survey Year: 2024 Neologisms (Index Form): 안티알고리즘(antialgorithm) Source Form: (<영>anti-algorithm) Domain Type: terminological Specialized Domain: Information & Communication Definition:유튜브나구글등에서작동하는알고리즘의지배와 통제를거부하거나저항하는태도. UsageExample:¶안티알고리즘의경향이점짙어지면서온 라인상의행적을추적하는알고리즘에반감을느껴웹사이트를 방문할때로그인을하지않는사용하는이들이증가하고있다. Semantic Category: Human Word-Formation Type: Compounding Korean Neologism Data Sample (b) Figure 2: KoNeoBench dataset: (a) data collection pro- cess and (b) an example of a neologism and its attributes. In the automatic candidate collection stage, can- didate words are identified mainly from nouns by comparing corpus items with the headword list of the Standard Korean Language Dictionary, the largest normative dictionary of Korean, and then re- moving inflected and plural forms. Because Korean is an agglutinative language, this process inevitably produces noise and may miss genuine neologisms, making expert review essential. In the validation stage, candidates are first fil- tered using rule-based criteria and then reviewed by experts. A candidate is accepted as a neologism if it (1) is created to refer to a newly emerged object or phenomenon, (2) expresses a new nuance despite the existence of a conventional expression, or (3) functions as a phrasal unit with a unified lexical meaning. Proper nouns, group-specific expressions, nonce words, and foreign words not established in the Korean lexical system are excluded. In practice, the automatically extracted out-of-vocabulary list contained 1,627,371 candidates over five years, but most were mechanically identified errors such as incomplete fragments, failed morphological analy- ses, or misspellings. After automatic filtering and expert review, only 1,269 items were retained as Distribution by Survey Year Distribution by Domain Type General 986 (55.2%) 124 (6.9%) 309 (17.3%) 152 (8.5%) 173 (9.7%) 256 (14.3%) 325 (18.2%) 446 (25.0%) PoliticsEconomySocietyHumanLifeHealthOthers (<5%) Distribution by Semantic Category Terminological 801 (44.8%) LawPublic HealthWelfare Society AdministrationOthers (<5%) 252 (31.5%) 53 (6.6%) 70 (8.7%) 73 (9.1%) 86 (10.7%) 119 (14.9%) 148 (18.5%) Economics Specialized Domain Blending Abbreviation Suffixation Compounding Borrowing Prefixation Root Creation Multi-word Expression 1058 (58.9%) Distribution by Word-Formation (a) (b) (c) (11.3%) (11.5%) (5.7%) (6.8%) (4.4%) Count 20202021202220232024 100 300 500 400 200 Figure 3: Data Statistics in KoNeoBench. (a) Distribution by survey year. (b) Distribution by semantic and specialized domain categories. (c) Distribution by type of word-formation. neologisms, illustrating the substantial reduction required to obtain the final set. In addition, to find neologisms that could not be captured by automatic extraction, we conducted a manual addition stage by experts using pragmatic markers in the corpus, through which 516 addi- tional neologisms were obtained. Pragmatic mark- ers are contextual cues in text that signal the pres- ence of a neologism, as in expressions such as the Korean expression ‘x-라는 신어’ (‘a neologism calledx’), which are regarded as important indica- tors for assessing both the novelty and the degree of establishment of a neologism (Klosa-Kückelhaus and Wolfer, 2020; Lee, 2024). 3.2 Data Descriptions and Statistics As shown in Figure 2, KoNeoBench includes not only a list of neologisms but also their associated attributes: dictionary-style definitions and usage examples, survey year, the source form, the word- formation type, the domain type (general vs. ter- minological), the semantic category, and the spe- cialized domain for terminological neologisms. De- tailed descriptions of these attributes are provided in Appendix A.2. Figure 3 presents the distributional characteris- tics of the dataset from several perspectives. Across the years, roughly 400 neologisms were consis- tently collected each year from 2020 to 2024. In terms of word-formation patterns, multi-word (or phrasal) neologisms formed by combining two or more words, account for the largest share at 58.9%, followed by neologisms created through abbrevi- ation and blending. This pattern is closely related to the spread of English through the web. The high proportion of abbreviated and blended forms, such as ‘호캉스(ho-kangseu, ‘hotel + vacance,’ a stay- cation at a hotel), also reflects a characteristic of Korean word-formation that allows relatively free truncation and recombination at the syllable level, particularly in loanwords from English. The distribution in Figure 3(b) shows that the top six semantic categories (politics, economy, so- ciety, human, life, and health) account for 82.7% of the dataset, suggesting that these areas are socially salient domains in which new vocabulary is ac- tively created and circulated within the community. A similar skew is observed in the distribution of the 801 terminological neologisms, with particularly high proportions in health and welfare, reflecting the exceptional social conditions of the COVID-19 pandemic. In this way, neologisms reflect the socio- cultural context of the period in which they emerge. Evaluating LLMs on neologisms therefore mea- sures not only lexical knowledge but also whether models recognize lexical expressions tied to recent social and cultural phenomena. Direct evaluation of broader sociocultural reasoning is left to future work. 3.3 Evaluation Tasks KoNeoBench provides not only a list of neologisms but also associated attributes, which make it possi- ble to design a variety of evaluation tasks. In this paper, we designed four tasks for evaluating con- textual understanding, source word identification, categorical understanding, and definition genera- tion in neologism processing, and provide test sets consisting of questions, gold answers, and evalua- tion criteria. Figure 4 presents example instances for each task, and Appendix B provides the details. 3.3.1 Task 1: Contextual Identification Task 1 is a multiple-choice cloze task in which the masked neologism must be identified from its orig- inal usage context. Distractors include contextually plausible but incorrect hard negatives constructed based on lexical semantic relations, such as con- ceptual or morphological similarity (Leech, 1981), as well as randomly selected words that serve as a baseline. Additionally, because Korean case parti- cles can provide semantic and syntactic cues, we 다음 용례에서 [MASK]에 들어갈 말로 알 맞은 것은? (Which word best fits in the [MASK] in the following sentence?) “[MASK](는) 앞으로 정부의 허가를 받아 노란 번호판을 달고 영업해야 하며 일정 액수의 기 여금을 내야 한다.” (“[MASK] must, going forward, obtain government approval, operate with yellow license plates, and pay a certain amount of contribution fees.”) 다음 5개 단어 중 나머지와 의미범주가 다른 것은? (Which of the following five words belongs to a different semantic category from the others?) “안방 요금”의 의미 범주를 아래 16개 중에서 고르시오. (Choose the semantic category of “home premiere fee” from the 16 options below.) Example: 이 영화는 온라인과 오프라인에서 동시 개봉 했지만 안방 요금이 상대적으로 비싸 예상보다 많은 사람들이 극장을 찾았다. (Although this film was released simultaneously online and offline, home premiere fee was relatively expensive, so more people than expected went to theaters.) A. 줌술족(은) (zoom-drinkers) D. 역응원(은) (cheering for the opposing side) A. 썩파트B. 소세권C. 안방요금D. 창툭튀E. 직락 (rotten-apartment) (pediatrician-access area) (home premiere fee)(bulging window AC)(workjoyzone) B. 플랫폼 운수 사업자(는) C. 극폭(은) (platform-based transport operator) (extreme width) E: Hard Negative * Particles in ( ) are included in [MASK] and candidates for the type 2 questions. 주어진 표면 음절에 대해 실제 조어에 사용된 구성 성분을 찾으시오. (For the given surface syllables, identify the actual components used in forming the word.) Input: 온택트 (ontact) “온라인+콘택트” (online+contact). √ √ A. 개념B. 경제생활C. 교육D. 동식물E. 문화...(omitted) (concept)(economic life) (education) (plants & animals) (culture) Type 1: Odd-1-hot Type 2: Classification “애프터코로나”의사전적정의를생성하시오.Write the definition of “애프터코로나(aftercorona)”. Example: 관광업계에서애프터코로나를대비하기위해한기업총수가직접나섰다. (ACEOof acompanyhassteppedforwardtooverseethesituationinthetourismindustry, inorderto prepareforthepost-COVIDera.) 코로나19의 세계적 대유행이 종식된 뒤의 시기 (The period after the end of the COVID-19 global pandemic.) Task1: Contextual Neologism Identification Task2: Source Word Identification Task3: Semantic Categorization Task4: Definition Generation E. 플랫폼 이용 사용자(는) (platform user) √ Figure 4: Examples of the four evaluation tasks in KoNeoBench. construct two variants of the test set: Type 1, where the mask and answer candidates contain only the word, and Type 2, where the particle is included together with the word (See Figure 4). 3.3.2 Task 2: Source Word Identification Task 2 is motivated by Korean lexicographic prac- tice, which marks component boundaries in com- plex words, and by speakers’ tendency to infer unfa- miliar neologisms through their constituents. Even when encountering an unfamiliar word not listed in a dictionary, native speakers can segment its source words and infer its meaning by identify- ing the underlying word-formation process, such as compounding, abbreviation, or blending. This task evaluates whether LLMs can perform a similar kind of linguistic inference. The test set covers 396 neologisms whose forma- tion types are blends or abbreviations, and includes gold annotations of their constituents and corre- sponding source words. These forms combine trun- cated or omitted source words, making the original forms difficult to identify from the surface form alone. For example, in ‘danmato→dalda(sweet) + tomato’, the source words cannot be reconstructed simply by segmenting the surface syllables. To solve this task, an LLM must use the partial syllabic form of a neologism as a cue and identify seman- tically and phonologically plausible source words in the Korean lexicon. The task therefore evaluates source-component recovery and sensitivity to recur- ring formation patterns, beyond whether the model has learned the surface form and meaning of neolo- gisms. We use prompts that explicitly instruct the model to identify the source-component recovery; examples are provided in Appendix B.2. 3.3.3 Task 3: Neologism Categorization Task 3 evaluates whether a model can understand the categorical meaning of Korean neologisms. It consists of two subtasks: semantic categorization, which assigns each neologism to one of 16 gen- eral semantic categories, and specialized domain categorization, which assigns 801 terminological neologisms to one of 43 specialized domains. Semantic categorization tests whether a model can classify neologisms into semantic categories as humans do, reflecting the fact that neologisms typi- cally emerge within particular semantic domains. The specialized domain categorization task asks whether the model can identify the specialized do- main of a terminological neologism, thereby mea- suring its sensitivity to register and domain speci- ficity. Since terminological neologisms are often tied to particular knowledge communities and may be opaque outside them, this task provides a useful indicator of whether a model can capture not only its basic meaning but also its domain of use. For each category system, we design two ques- tion formats; Type 1 is an odd-one-out task in which the model selects the one item that belongs to a different category from five candidate neolo- gisms; Type 2 is a classification task in which the model directly predicts the category of a given ne- ologism (See Figure 4). By evaluating both tasks, it is possible to distinguish whether a model can merely group words into a coherent lexical class or can further link that class to an explicit category la- bel. A neologism may be grouped with other words from the same class based only on formal similarity or co-occurring context, but assigning the correct category name requires a different level of under- standing. For Type 2, we evaluate two input set- tings: one with the neologism alone, and the other with both the neologism and its usage example. 3.3.4 Task 4: Definition Generation Task 4 is a generative task that evaluates whether a model can produce a dictionary-style definition of a neologism. It goes beyond identifying a ne- ologism’s form or category and tests whether the model can explain its core meaning in natural lan- guage. We consider two input settings: a context- free setting with only the neologism, and a context- based setting with both the neologism and a usage example. Definition writing is one of the most difficult tasks in lexicography, as it requires both seman- tic analysis and concise expression in a restricted metalanguage(Hanks, 2015). This makes the task linguistically meaningful, as it evaluates whether a model can both infer the meaning of a neolo- gism and express it accurately in dictionary-style language. The prompt template is provided in Ap- pendix B.2. 4 Evaluation and Analysis 4.1 Evaluation models Using KoNeoBench, we evaluate recent LLMs and analyze the results. Our goal is not to compare the superiority of particular model families, but to examine the overall performance trends and limita- tions of LLMs with different training backgrounds, scales, release dates, and degrees of suitability for Korean language processing in handling Korean neologisms. To this end, we evaluate a total of nine models, including commercial API-based models, open-weight models, and Korean-specialized mod- els. Details of the evaluated models are summarized in Table 6 in the Appendix. 4.2 Evaluation criteria For Task 1 and Task 3, we use the standard evalu- ation metric of classification accuracy, since both are multiple-choice tasks. For Task 2 and Task 4, which require free-form responses, we design task- specific evaluation criteria. Coverage & Precision for Task 2. To evaluate source word identification in Task 2, we use cov- erage, precision, and their harmonic mean (F1) as metrics, so as to account for cases where the model identifies only part of the gold source words or gen- erates unnecessary additional ones. Coverage mea- sures the proportion of gold source words correctly identified by the model, while precision measures the proportion of generated source words that are actually included in the gold answer. Details are provided in Appendix B.3. LLM-as-a-Judge for Task 4. We evaluate LLMs’ generated definitions by comparing them with the gold definitions in KoNeoBench. Because the same meaning can be expressed in multiple ways, simple lexical matching is not appropriate. We therefore adopt an LLM-as-a-judge approach, using Gemini, a model from a different family than the evaluated LLMs, to assess semantic equiva- lence. LLM-based evaluation of generated outputs has become common and has also been adopted in prior benchmark studies (Zheng et al., 2023, 2024). To address concerns about reliability, we propose eval- uation criteria carefully designed by lexicography experts. The criteria cover three dimensions for a total of 10 points: semantic adequacy, fluency, and factuality. The full criteria and judge prompt are given in Appendix B.4 and B.2. We also conduct human validation on a subset of samples to verify the reliability of the automatic evaluation; details are provided in the results section. 4.3 Evaluation Results and Analysis Table 1 summarizes the results of all models across all tasks. The GPT series performs best overall, while earlier open models such as Solar-10.7B and LLaMA-3.1 perform worst, likely reflecting the time-sensitive nature of neologisms. In most tasks, average scores remain around 30–50 out of 100, indicating that LLMs’ understanding of Korean neologisms is still limited. Since most evaluated models were released between 2024 and 2026 (See Table 6), after the emergence of the large portion of neologisms in KoNeoBench, this low performance cannot be attributed solely to knowledge cutoff and may also reflect the inherently low-frequency nature of neologisms. Performance gaps are es- pecially large in Task 4, suggesting that definition generation is particularly discriminative. Large vari- ation is also observed in the specialized domain categorization task (Task 3), possibly due to differ- ences in training data diversity. Figure 22 demon- strates further information for each task.We also compared model performance with human perfor- mance on subsets of Tasks 1–3. Linguistics majors performed better than GPT-5.4 on Tasks 1 and 2, while GPT-5.4 performed better than the overall human group on the specialized-domain setting of Table 1: Evaluation results on four tasks in KoNeoBench. (Ave. pts. denotes the average LLM-as-a-judge score points. For each task, the best performance is shown in red and the worst performance in blue. ) ModelTask 1 (Acc. %)Task 2Task 3 Semantic (Acc. %)Task 3 Spec. Domain (Acc. %) Task 4 (Ave. pts) type 1type 2cov.prec.F1type 1type 2type 1type 2w/o ex.w/ ex. w/o part. w/ part.w/o ex. w/ ex.(diff)w/o ex. w/ ex.(diff) GPT-4.154.6252.2143.71 44.13 43.78 63.0047.0057.54(10.54↑) 80.0057.8863.38(5.50↑) 7.15 8.47 GPT-5.466.1662.9749.77 50.33 49.96 66.8053.2862.52(9.24↑)84.1264.3867.50(3.12↑) 7.49 8.69 Solar Pro3-12B44.3441.8232.48 33.86 32.90 37.1436.1349.52(13.39↑) 52.1249.3857.25(7.87↑) 6.68 7.79 Solar-10.7B37.9836.699.899.879.7632.7725.1037.98(12.88↑) 49.5023.2533.12(9.87↑) 3.5 4.41 EXAONE-3.5-7.8B47.0945.4023.48 24.93 24.00 34.1233.4548.12(14.67↑) 40.1217.0022.38(5.38↑) 6.17 7.53 EXAONE-4.0-32B52.8650.8423.80 24.69 24.08 42.6937.1453.05(15.91↑) 54.6250.3863.00(12.62↑) 6.07 7.53 Qwen2.5-7B40.4738.0018.03 18.79 18.19 36.5334.6248.80(14.18↑) 48.0042.5053.50(11.00↑) 5.56 6.81 Qwen3.5-9B47.4046.6423.42 24.04 23.62 39.9437.7651.09(13.33↑) 57.1348.2557.88(9.63↑) 5.58 7.05 LLaMA-3.1-8B34.4730.7215.51 15.80 15.50 28.9032.9447.68(14.74↑) 32.7533.2539.62(6.37↑) 5.19 6.31 Mean47.2745.0326.68 27.38 26.87 42.4337.0050.70(13.70↑) 55.3742.9250.85(7.93↑) 5.93 7.18 Max−Min31.6932.2539.88 40.46 40.20 37.9028.0024.5451.3747.3845.123.99 4.28 Task 3. Full results and statistical tests are reported in Appendix D. 4.3.1 Results on Task 1 As shown in Table 1, the best performance is achieved by GPT-5.4, a recently released API model. GPT-4.1, released about a year earlier, and EXAONE-4.0, a Korean-specialized model re- leased around the same time, show similar per- formance. By contrast, Solar Pro3, despite be- ing a recently released Korean-specialized model, performs below expectations, even falling behind EXAONE-3.5, a smaller Korean-specialized model released earlier. Even the best-performing model reaches only the 60% range in accuracy. To better understand this limitation, Figure 5 breaks down the types of answers selected by the models. Most incorrect pre- dictions are hard negatives, showing that this task measures not only general contextual understand- ing but also fine-grained discrimination among ne- ologisms. In other words, models can usually iden- tify the broad contextual area in which a neolo- gism should appear, but often fail to distinguish the correct neologism from other plausible candi- dates within that area. The poor performance of the Korean-specialized model Solar Pro3 is also largely due to these hard-negative errors (53%), suggesting that learning general Korean vocabulary alone is not sufficient for fine-grained discrimina- tion among neologisms. We also compare the results by whether the masked item includes a case particle. For all mod- els, Type 2, which includes the particle, performs worse than Type 1 (average 2.24%). This suggests that case particles may impose an additional seg- Task 1 Prediction Distribution by Answer Type 0 1 0.5 Proportion GPT - 4.1 GPT - 5.4 Solar Pro3 Solar - 10.7B EXAONE - 3.5 EXAONE - 4.0 Qwen2.5Qwen3.5 LLaMA - 3.1 Correct Hard NegativeRandom Distractor 0.59 0.66 0.44 0.38 0.47 0.53 0.40 0.47 0.34 0.40 0.33 0.53 0.54 0.49 0.44 0.54 0.47 0.48 0.18 Figure 5: Distribution of LLM responses to type 1 ques- tions in Task 1. mentation and prediction burden when models pro- cess unseen neologism forms. 4.3.2 Results on Task 2 From Table 1, no model exceeds 50 F1 on Task 2. The best-performing model, GPT-5.4, achieves only 49.96 F1, showing that current LLMs re- main far from solving this task. Neither parameter scale nor Korean-centric pretraining reliably pre- dicts performance: EXAONE-4.0 (32B) shows lit- tle improvement over the much smaller EXAONE- 3.5 (7.8B), and both Korean-specialized EXAONE models perform similarly to the general-purpose Qwen3.5. The difficulty is further increased by the irregu- larity of truncation, since both the position and unit of truncation (word, syllable, or phoneme) vary un- predictably across items. As shown by GPT-5.4’s incorrect predictions (Table 8 in the Appendix), LLMs often struggle to recover source lexemes from multiple lexical strata. Overall, the results on Task 2 indicate that current LLMs still face sub- stantial difficulty in attaining native-speaker-level understanding of neologisms that emerge dynami- cally within a language community. 4.3.3 Results on Task 3 Semantic Categorization. In the semantic cat- egorization Type 1 task, GPT-5.4 and GPT-4.1 achieve 66.80% and 63.0%, respectively, whereas EXAONE-4.0, the next best model, reaches only 42.69%, showing a large gap from the GPT models. This suggests that, regardless of Korean specializa- tion, models with stronger general language under- standing are better at comparing semantic similarity and contrast among neologisms. At the same time, even the best results remain in the 60% range, still indicating clear limitations in this categorization setting. For Type 2, the best accuracy is lower than in Type 1, which may partly reflect the larger number of target classes (16). However, as shown by the numbers in parentheses in Table 1, performance improves substantially when usage examples are provided, with gains exceeding 10% for most mod- els. This result suggests that actively leveraging usage example may improve LLMs’ understanding of neologisms. Specialized Domain Categorization. Overall performance is higher than for semantic categoriza- tion, but the gap between the best and worst mod- els exceeds 50%, making performance differences across models even more pronounced. Although the number of specialized field categories (43) is larger than that of semantic categories, their bound- aries are relatively clearer, which likely contributes to the higher performance in Type 1. Still, the gap between the GPT models and the others remains large: the GPT series scores above 80%, whereas LLaMA-3.1 stays in the 30% range, possibly re- flecting differences in the diversity and domain coverage of the training data. As with semantic categorization, Type 2 in spe- cialized domain categorization also benefits from the use of usage examples, although the improve- ment is relatively modest. This is likely because identifying a specialized field requires the model to infer not only the knowledge domain but also its area of use, which cannot be fully inferred from local context alone. (b) 10 6 9 8 7 4 5 3 20202021202220232024 20202021202220232024 Ave. Generation Score w/o usage example w/ usage example 3 6 9 8 7 4 5 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 Release DateRelease Date Ave. Generation Score w/o usage example w/ usage example Solar-10.7B LLaMA-3.1 Qwen-2.5 EXAONE-3.5 EXAONE-4.0 Qwen-3.5 GPT-4.1 Solar Pro3 GPT-5.4 GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B Not Korean Optimized Korean Optimized Model Type 32B params Model Size 16B params 8B params (a) Unknown params ? ? ? ? ? ? ? Figure 6: Analysis of LLMs’ average scores on Task 4: (a) Performance by model release dates and scales (b) Performance by survey year of neologisms. 4.3.4 Results on Task 4 The Task 4 scores in Table 1 are obtained through automatic evaluation by Gemini as the judge LLM. GPT-5.4, the best-performing model, achieves 7.49 out of 10 without usage examples and 8.69 with us- age examples, indicating very strong performance. In contrast, Solar-10.7B, the lowest-performing model, scores only 3.50 without usage examples and 4.41 even when examples are provided, show- ing a clear performance gap across models. A de- tailed breakdown by evaluation criterion is pro- vided in Figure 28 in the Appendix. Generating a dictionary-style definition and eval- uating it against a gold definition requires a model to articulate the meaning of a neologism rather than select among given options. We therefore conduct a more detailed analysis of Task 4, shown in Fig- ure 6. Figure 6(a) shows the relationship between model release date and performance, with marker size indicating model scale. In both the setting with usage examples (left) and the setting without us- age examples (right), more recently released mod- els tend to perform better. Also, the setting with usage examples shows greater separation across models, suggesting that it is a more informative evaluation setup. Additionally, Korean-specialized models (red markers) generally maintain relatively strong performance regardless of scale or release date. Figure 6 (b) shows model performance grouped by survey year in which the neologisms first ap- peared. In the setting without usage examples (left), performance declines clearly for more recent neolo- gisms, and this trend is observed fairly consistently across models. With usage examples (right), overall performance improves, and the decline for more re- cent neologisms is also reduced. The corresponding year-wise performance distributions for the other tasks are provided in the Appendix. To validate the automatic evaluation by the judge LLM, we asked a lexicography expert to assess 250 definitions generated by GPT-5.4 and compared the scores. Figure 27 in the Appendix shows the distributions of the Gemini and human evaluation scores. The average scores and linear regression results indicate that Gemini tends to assign slightly higher scores than the human evaluator, but the two still show a strong positive correlation (r = 0.708). A detailed analysis is provided in Appendix D. 5 Discussions In this section, we discuss a number of observations arising from the results and consider their impli- cations for understanding and evaluating LLMs’ ability to process Korean neologisms. Definition Generation as a Measure of Korean Neologism Understanding. Comparing the re- sults across the four tasks suggests that definition generation (Task 4) is a particularly useful measure for evaluating LLMs’ understanding of Korean ne- ologisms. It shows greater performance separation across models than the other tasks and also corre- lates well with the evaluation results of the remain- ing tasks (See Figure 26). We therefore present defi- nition generation as a complementary and compara- tively comprehensive task, since it requires models to articulate meaning rather than select an answer. In the case of Korean, this task also has a practi- cal advantage, since annual neologism dictionaries are published regularly, making it possible to ob- tain gold definitions without additional large-scale expert annotation. While the reliability of LLM-as- a-judge remains an important issue, the evaluation criteria and prompting strategy proposed in this work help address this issue. This may therefore be considered one of the main practical contributions of our study. Challenges in LLMs’ Understanding of Korean Neologisms. The results across tasks point to three main challenges for LLMs in understand- ing Korean neologisms. First, time drift and low frequency make neologisms difficult to learn, as shown by the lower performance on more recent items in Figure 6. Second, the agglutinative struc- ture of Korean introduces additional difficulty, as reflected in the weaker performance on particle- attached forms in Task 1, which is unlikely to be captured by English-centered benchmarks. Third, LLMs still struggle with the formation patterns of Korean neologisms. The below-50% results for all models in Task 2 suggest that this limitation cannot be addressed simply by scaling up training data alone. Using Examples to Improve Neologism Under- standing.Despite these challenges, the results of Task 3 and Task 4 show substantial performance gains when usage examples are provided (see Table 1). This suggests that, in real-world settings, LLMs may better understand neologisms by retrieving and leveraging usage examples through retrieval- augmented methods. 6 Conclusion In this paper, we introduced KoNeoBench, the first benchmark for multi-faceted evaluation of LLMs’ understanding of Korean neologisms. Built on a five-year collection of Korean neologisms anno- tated by lexicography experts, KoNeoBench covers tasks such as source word identification, seman- tic categorization, and definition generation, while reflecting linguistic properties specific to Korean. By treating neologisms as a distinct evaluation tar- get, it reveals challenges in real-world language use that are not fully captured by existing Korean benchmarks centered on standard vocabulary. Using KoNeoBench, we proposed an evaluation framework for Korean neologism understanding, analyzed the major sources of difficulty for cur- rent LLMs, and showed that usage examples can substantially improve performance. We hope Ko- NeoBench will serve as a useful step toward evalu- ation frameworks that better capture ongoing lan- guage change in Korean. Limitations The neologism set used in this study is limited to items observed between 2020 and 2024. Earlier neologisms were excluded in light of the fact that recent LLMs are mainly trained on data extending beyond 2023. Neologisms from 2025 are currently being curated and will be incorporated into a future release. In addition, semantic neologisms, in which existing words acquire new meanings, are excluded because their first occurrence dates are difficult to determine reliably. The collection and evaluation of semantic neologisms will be addressed in future work. KoNeoBench is based on neologisms col- lected from online news articles. While many neol- ogisms first appear in online communities or other informal settings, news media provide a practical source for large-scale collection and verification. Some recent expressions, especially those used in informal or community-specific settings, may be less well represented before they appear in news coverage. Future versions of the benchmark will extend the collection to social media and online communities, with additional procedures for verifi- cation and tracking first attestation dates. Acknowledgments This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Ko- rea government(MSIT) (RS-2026-25487705). This research was supported by the Yonsei University Research Fund of 2026 (2026-22-0116). References Oshin Agarwal and Ani Nenkova. 2022. Temporal ef- fects on pre-trained models for language processing tasks. Transactions of the Association for Computa- tional Linguistics, 10:904–921. Afnan Mohammed Aloraini, Riza Batista-Navarro, Goran Nenadic, and Viktor Schlegel. 2026. The SlangTrack dataset: Supporting the detection of words used in slang senses. In The Proceedings for the 6th International Workshop on Computational Ap- proaches to Language Change (LChange’26), pages 1–19, Rabat, Morocco. Association for Computa- tional Linguistics. Ali Araabi, Christof Monz, and Vlad Niculae. 2022. How effective is byte pair encoding for out-of- vocabulary words in neural machine translation? In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Ameri- cas (Volume 1: Research Track), pages 117–130, Or- lando, USA. Association for Machine Translation in the Americas. Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive ques- tions. In Advances in Neural Information Process- ing Systems, volume 34. Datasets and Benchmarks Track. Cristiano Ciaccio, Alessio Miaschi, and Felice Dell’Orletta. 2025. Evaluating lexical proficiency in neural language models. In Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1267–1286, Vienna, Austria. Association for Compu- tational Linguistics. Paul Cook. 2010. Exploiting Linguistic Knowledge to Infer Properties of Neologisms. Ph.D. thesis, Univer- sity of Toronto. Paul Cook and Suzanne Stevenson. 2010. Automati- cally identifying the source words of lexical blends in English. Computational Linguistics, 36(1):129– 149. Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257– 273. Nicolas Garneau, Jean-Samuel Leboeuf, and Luc Lam- ontagne. 2018. Predicting and interpreting embed- dings for out of vocabulary words in downstream tasks. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 331–333, Brussels, Bel- gium. Association for Computational Linguistics. Patrick Hanks. 2015. Definition. In Philip Durkin, editor, The Oxford Handbook of Lexicography, pages 94–122. Oxford University Press. Myeongjun Jang, Dohyung Kim, Deuk Sin Kwon, and Eric Davis. 2022. KoBEST: Korean balanced eval- uation of significant tasks. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 3697–3708, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. Annette Klosa-Kückelhaus and Sascha Wolfer. 2020. Considerations on the acceptance of german neol- ogisms from the 1990s. International Journal of Lexicography, 33(2):150–167. Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gri- bovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Infor- mation Processing Systems, volume 34, pages 29348– 29363. Kihwang Lee. 2006. Contextual characteristics of neol- ogisms in corpora. In Proceedings of the 2006 Joint Autumn Conference of The Linguistic Association of Korea and The Linguistic Society of Korea, pages 280–289. Soojin Lee. 2024. A Corpus Linguistic Study of Korean Neologisms from a Short-Term Diachronic Perspec- tive. Ph.D. thesis, Kyungpook National University. Soojin Lee and Yeni Kim. 2014. Extraction method of new-words of 2013 and its morphological and seman- tic characteristics. Journal of Korealex, (23):232– 262. Geoffrey Leech. 1981. Semantics: The Study of Mean- ing, 2nd edition. Penguin Books. Paul Lerner and François Yvon. 2025. Towards the machine translation of scientific neologisms. In Proceedings of the 31st International Conference on Computational Linguistics, pages 947–963, Abu Dhabi, UAE. Association for Computational Linguis- tics. Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tay- fun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-McMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. StreamingQA: A benchmark for adaptation to new knowledge over time in question answering models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 13604–13622. PMLR. Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Kar- ishma Mandyam, and Noah A. Smith. 2022. Time waits for no one! analysis and challenges of tem- poral misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5944–5958, Seattle, United States. Association for Computational Lin- guistics. Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, and Xueqi Cheng. 2024. SLANG: New concept com- prehension of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12558–12575, Miami, Florida, USA. Association for Computational Linguistics. Kum Hyon Moon. 1999. Types of neologism in modern korean. Journal of Korean Linguistics, 33:295–325. Kilim Nam. 2015. An analysis on the usage change of neologisms and the criteria for headwords in the dictionary: Focused on 2005-2006 neologisms. HAN- GEUL, 310:205–233. Kilim Nam, Hyunju Song, Mikyung Baek, Soojin Lee, Younghee Hyun, Beomil Kang, Jinsan An, Seol Namkung, and Jun Lee. 2025. Neologism 2024. Han- kookmunhwasa. Anmol Nayak, Hariprasad Timmapathini, Karthikeyan Ponnalagu, and Vijendran Gopalan Venkoparao. 2020. Domain adaptation challenges of BERT in tokenization and sub-word representations of out- of-vocabulary words. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 1–5, Online. Association for Computational Linguistics. Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What LMs know about unseen entities. In Findings of the Associa- tion for Computational Linguistics: NAACL 2022, pages 693–702, Seattle, United States. Association for Computational Linguistics. Chanjun Park, Hyeonwoo Kim, Dahyun Kim, SeongH- wan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, and Hwalsuk Lee. 2024. Open Ko-LLM leaderboard: Evaluating large language models in korean with Ko- H5 benchmark. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 3220–3234, Bangkok, Thailand. Association for Computational Linguistics. Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Jun- seong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, and 12 others. 2021. KLUE: Korean language understanding evaluation. In Ad- vances in Neural Information Processing Systems, volume 34. Yuval Pinter, Cassandra L. Jacobs, and Max Bittker. 2020. NYTWIT: A dataset of novel words in the new york times. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 6509–6515, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics. Aniket Pramanick, Tilman Beck, Kevin Stowe, and Iryna Gurevych. 2022. The challenges of temporal alignment on twitter during crises. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2658–2672, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020. Temporally-informed analysis of named entity recog- nition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7605–7617, Online. Association for Computational Linguistics. Maria Ryskina, Ella Rabinovich, Taylor Berg- Kirkpatrick, David Mortensen, and Yulia Tsvetkov. 2020. Where new words are born: Distributional semantic analysis of neologisms and their semantic neighborhoods. In Proceedings of the Society for Computation in Linguistics 2020, pages 367–376, New York, New York. Association for Computational Linguistics. Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, and Songseong Kim. 2024. HAE-RAE bench: Evaluation of korean knowledge in language mod- els. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7993–8007, Torino, Italia. ELRA and ICCL. Zhewei Sun, Qian Hu, Rahul Gupta, Richard Zemel, and Yang Xu. 2024. Toward informal language process- ing: Knowledge of slang in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1683–1701, Mexico City, Mexico. Association for Computational Lin- guistics. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. Fresh- LLMs: Refreshing large language models with search engine augmentation. In Findings of the Associa- tion for Computational Linguistics: ACL 2024, pages 13697–13720, Bangkok, Thailand. Association for Computational Linguistics. Siyang Wu and Zhewei Sun. 2025. How do language models generate slang: A systematic comparison be- tween human and machine-generated slang usages. In Findings of the Association for Computational Lin- guistics: EMNLP 2025, pages 6536–6559, Suzhou, China. Association for Computational Linguistics. Ifeoluwa Wuraola, Daniel Marciniak, and Nina Dethlefs. 2026. SLANG-GraphRAG: Multi-layered retrieval with domain-specific knowledge for low resource social media conversations. In Findings of the Asso- ciation for Computational Linguistics: EACL 2026, pages 3919–3931, Rabat, Morocco. Association for Computational Linguistics. Jonathan Zheng, Alan Ritter, and Wei Xu. 2024. NEO- BENCH: Evaluating robustness of large language models with neologisms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13885– 13906, Bangkok, Thailand. Association for Compu- tational Linguistics. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Sys- tems. Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. 2025. Is your LLM outdated? a deep look at temporal generaliza- tion. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7433– 7457, Albuquerque, New Mexico. Association for Computational Linguistics. Jian Zhu and David Jurgens. 2021. The structure of online social networks modulates the rate of lexical change. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2201–2218, Online. Association for Computational Linguistics. A Data Construction A.1 Data Collection Process This appendix describes the detailed criteria and yearly statistics of the Korean neologism collection procedure summarized in Section 3.1. Our neolo- gism list is based on annual collections compiled from 2020 to 2024 under a consistent methodology. The process consists of four stages: automatic can- didate collection, filtering, expert validation, and manual addition. Automatic Candidate Collection. In this stage, unregistered noun candidates were extracted from the yearly corpus using LRNounExtractorv2, an unsupervised Korean noun extractor. The tool seg- ments sentences into content-word and function- word pairs and extracts candidate single and com- pound nouns based on frequency and distributional information. Because the extracted set includes not only genuine neologisms but also incomplete frag- ments, failed analyses, and misspellings, additional filtering is required. Automatic Filtering. This stage removes candi- dates unlikely to be neologisms. We exclude (1) forms containing characters other than Hangul or Roman letters, (2) forms already attested before the target year by comparison with a large inventory of about 330 million word forms, (3) forms listed in the Standard Korean Language Dictionary, and (4) inflected or plural forms ending in particles or verbal endings. We also remove candidates that are found, through Naver News search, to have been used before the target year. Because Korean is agglutinative, nouns com- bine with many particles and predicates appear in many inflected forms. Spacing also does not always match lexical boundaries, and spacing variation is common in actual use. As a result, automatic extrac- tion of unregistered forms alone does not reliably identify neologism candidates. This is especially true for low-frequency items. This limitation is reflected in the collection statistics. As shown in Table 2, automatic extraction produced 1,627,371 candidates from 2020 to 2024, but only 1,269 were finally accepted as neologisms after expert review. Expert Validation. Final decisions on neolo- gismhood were made through expert review. In the source data, an item was accepted as a neologism if it (1) referred to a newly emerged object, con- cept, or phenomenon, (2) expressed a new nuance YearCandidatesExpert-valid.Ratio (%) 2020409,7542370.06 2021193,9051350.07 2022246,5952110.09 2023322,1772820.09 2024454,9404040.09 Total1,627,3711,2690.08 Table 2: Yearly statistics of automatically extracted can- didates and retained neologisms after expert validation. or attitude despite the existence of a conventional expression, or (3) functioned as a phrasal unit with a unified lexical meaning. Proper nouns, expres- sions restricted to individuals or specific groups, nonce words, and foreign words not established in Korean were excluded. Manual Addition. To address the limitations of automatic extraction, we also conducted manual collection. This step is particularly important for low-frequency items. We relied on pragmatic mark- ers, such as “a neologism calledx”, “so-calledx” and similar contextual cues that indicate the use of a new or unfamiliar expression. Such markers have been discussed as useful indicators of both the novelty and the establishment of neologisms (Klosa-Kückelhaus and Wolfer, 2020; Lee, 2024). A.2 Data Attributes The KoNeoBench neologism list contains Korean neologisms collected from 2020 to 2024, together with attributes such as survey year, lemma form, source form and word-formation type, semantic category, specialized field, definition, and usage example. The data is released in CSV format, and Table 3 shows sample entries. The attributes are described below. • Survey year indicates when a neologism was first identified within the collection period. Be- cause each annual survey covers news articles published from July 1 of the previous year to June 30 of the target year, a 2020 neologism refers to an item first attested between July 1, 2019 and June 30, 2020. This attribute is used to analyze yearly distributions and temporal changes in model performance. •Index form is the normalized form for index- ing and sorting. It removes spacing, hyphens, and other boundary markers from the regis- tered lemma, so that even phrasal expressions 조사 연도 색인 표제어 원어 단어 형성법 일상어/ 전문어 전문 분야 의미 범주 뜻풀이 용례빈도 구성 성분 분석 (Survey Year) (Head word) (Source form) (Word- formation type) (Domain type) (Specialized Domain) (Semantic category) (Definition)(Usage) (Freq.) (Source words identi -fication 2020 먹튜브(먹← Youtube) 혼성 일상어–사회생활 먹는방송을주제로하는 동영상공유서비스채널을 이르는말. ¶한도시에서지역음식점을 발굴하기위해당지역을 기반으로하는 먹튜브를 찾고있다. 89 먹다+ 유투브 2021 오하운(오하運) 축약 일상어–삶‘오늘하루운동’을줄여 이르는말.매일상적으로 하는운동을가리키는말이다. ¶젊은세대사이에는자존감과 성취감을높이기위한 오하운이중요한키워드로 자리를잡고있다. 310 오늘+ 하루+ 운동 2023 자서망(自署網) 합성 전문어 정보 통신 사회생활 경찰조직내부에서사용하는 통신망.주로관할지역내에서 발생한긴급상황을전파하기 위해사용한다. ¶경찰서내다양한통신망중, 112신고전화등의긴급 상황은 자서망을통해가장 먼저전파된다. 82 – 2024 스키코 어 (<영>ski- core) 차용 전문어 복식 의생활 추운날씨가지속되면서, 스키장에서뿐만아니라 일상생활에서도입을수있는 스키웨어를착용하는패션 스타일. ¶정통스키스포츠브랜드인 O가올겨울트렌드인 스키코어를반영하여어떤 기상조건에서도적응할수 있는새로운컬렉션을 출시했다고밝혔다. 74 스키+ 놈코어 Table 3: Example entries from KoNeoBench. and compounds are represented as a single continuous string. •Source form indicates the original elements of a neologism. Native Korean items are given in Hangul, Sino-Korean items in Chinese char- acters, and loanwords in their original forms with source-language labels. This attribute helps identify whether a neologism is formed from native Korean, Sino-Korean, loanwords, or a mixture of them. •Word-formation type specifies how a neolo- gism was created, including root creation, pre- fixation, suffixation, compounding, borrow- ing, abbreviation, and blending. This attribute is particularly important for abbreviation and blending, where the surface form often differs from the underlying constituents. •Domain type (general/terminological) dis- tinguishes everyday neologisms from techni- cal terms used in specific academic, industrial, or social domains. •Specialized domain specifies the concrete domain of a neologism when it is classified as terminological neologism. The categoriza- tion follows the specialized-field taxonomy of ‘Urimalsaem’, which contains 67 fields in total. The terminological neologisms in our dataset are distributed over 43 of these fields. This attribute can be used as the gold label for specialized domain categorization. •Semantic category indicates the semantic do- main of the concept denoted by a neologism. It is based on the lexical semantic category sys- tem of the Basic Korean Dictionary, with ad- ditional categories such as health & medicine and information & technology to reflect recent neologisms, resulting in a 16-category system. •Definition is a dictionary-style description of a neologism. It conveys the meaning, function, and usage of the entry, and is used as the gold definition in the definition generation task. • Usage Example shows how a neologism is used in real context. It is based on attested news usage, edited to follow language norms. The headword may be marked with braces, and proper nouns are anonymized. •Frequency indicates how often a neologism appears in the collected data or analysis cor- pus. • Source word identification is provided only for non-phrasal neologisms and describes their internal morphological structure. It iden- tifies the morphemes or meaningful con- stituents of a neologism, rather than simply segmenting the surface string. This is espe- cially important for abbreviations and blends, and is directly used in the source word identi- fication task (Task 2). A.3 Test Set Examples The benchmark dataset described in Appendix A.2 already includes the answers to Task 2 (source word identification), Task 3-Type 2 (category), and Task 4 (definition). By contrast, Task 1 and Task 3-Type 1 are multiple-choice tasks, so they require candi- date options and gold labels for testing and evalua- tion. This section describes the test sets for Task 1 and Task 3-Type 1. A.3.1 Task 1 : Contextual Identification. The test set for Task 1 is built from each neolo- gism and its attested usage sentence. We create a cloze question by replacing the target neologism with [MASK]. Each multiple-choice item includes the correct answer, one contextually plausible hard negative, and three random distractors sampled from the benchmark dataset. The hard negative may be conceptually or morphologically similar to the target, semantically contrastive, in a hyper- nym–hyponym relation, or related in sociocultural background or usage context. Although such hard negative distractors may fit the sentence grammat- ically or appear partly relevant, they do not fully match the exact referent, semantic focus, or prag- matic nuance of the target neologism. We further define two input conditions. In the first condition (Type 1), only the target neologism is replaced by [MASK], while the original case particle remains in the sentence. All answer can- didates are therefore presented in lemma form. In the second condition (Type 2), both the neologism and the attached particle are replaced by [MASK], and each candidate is presented with an appropriate particle. By comparing these two conditions, we examine whether models rely on morphosyntac- tic cues such as case particles when selecting the correct answer. A.3.2 Task 3 (Type 1) : Neologism Categorization. Task 3 (Type 1) test set is designed to evaluate whether a model can recognize categorical sim- ilarity and difference among multiple neologism candidates. Each question consists of five candidate neologisms. One is the target neologism, and the other four belong to the same category, which is Option TermSemantic category A 아동양육쿠폰 (childcare voucher) Politics and administration B 안심가림막 (protective partition) Health and medicine C 아베노마스크 (Abenomask;Abe’s mask) Politics and administration D 안보삼각형 (security triangle) Politics and administration E 약물치유법원 (drug treatment court) Politics and administration Table 4: Test example from Task 3 (semantic category), Type 1: Five candidates, including the correct answer (shown in blue) Option TermSpecialized domain A 안티알고리즘 (anti-algorithm) Information and communication B 양자비트플랫폼 (quantum bit platform) Information and communication C 전관용역 (ex-official service contract) General society D 온디바이스폰 (on-device phone) Information and communication E 온센서에이아이 (on-sensor AI) Information and communication Table 5: Test example from Task 3 (specialized domain category), Type 1: Five candidates, including the correct answer (shown in blue). different from that of the target. Thus, the four dis- tractors share a single category, while only the tar- get belongs to a different one. The model is asked to select the item whose category differs from the others, making the target the correct answer. For semantic categorization, we use the 16 se- mantic categories assigned to the full dataset. For each target, we randomly choose a category dif- ferent from its gold label, sample four neologisms from that category, and combine them with the tar- get. The candidate order is then shuffled, and the target serves as the correct answer. Table 4 shows an example. The same procedure is used for spe- cialized domain categorization, except that only ter- minological neologisms are included; Table 5 gives an example. To ensure that four same-category dis- tractors can be constructed reliably, categories with too few candidates are excluded from random sam- pling. ModelProviderAccessModel SizeRelease DateNote GPT-4.1OpenAIAPIUndisclosed2025-04-14 – GPT-5.4OpenAIAPIUndisclosed2026-03-05 – Solar Pro3UpstageAPI Undisclosed2026-01-26 Korean-optimized commercial model Solar-10.7B-InstructUpstageOpen weights10.7B2023-12-23 – EXAONE-3.5-7.8B-Instruct LG AI ResearchOpen weights7.8B2024-12-06 Korean–English bilingual instruction tuned model EXAONE-4.0-32BLG AI ResearchOpen weights32B2025-07-15 Korean–English large language model Qwen2.5-7B-InstructQwenOpen weights7B2024-09-19 – Qwen3.5-9BQwenOpen weights9B2026-03-09 – Llama-3.1-8B-InstructMetaOpen weights8B2024-07-23 – Table 6: Overview of the large language models evaluated in this study. B Evaluation Methods B.1 LLM Models for Evaluation Table 6 summarizes the large language models eval- uated in this study. To compare Korean neologism understanding across different model types, we include both commercial API-based models and instruction-tuned open-weight models. The API- based models are GPT-4.1, GPT-5.4, and Solar Pro3. The open-weight models are Solar-10.7B- Instruct, EXAONE-3.5-7.8B-Instruct, EXAONE- 4.0-32B, Qwen2.5-7B-Instruct, Qwen3.5-9B, and Llama-3.1-8B-Instruct. The models were selected based on three main considerations; (1) we include API-based models to examine how recent commercial LLMs perform on Korean neologism understanding; (2) we include the Solar and EXAONE families to analyze the performance of models specialized for Korean or Korean–English bilingual settings; (3) we include Qwen and Llama as general-purpose multilingual open-weight models for comparison. Table 6 reports each model’s provider, access type, size, release date, and notable characteris- tics. API-based models are marked as undisclosed when parameter counts are not public, whereas open-weight models are listed by their reported pa- rameter size. Release date is included only as tem- poral context and does not necessarily indicate the training-data cutoff. All models are evaluated with the same task prompts and procedure described in the main text. We used deterministic decoding for the selec- tion and classification tasks by setting the tempera- ture to 0. The answer option order was randomized with a fixed seed of 42. For Qwen3.5, the thinking mode was disabled to prevent intermediate reason- ing traces from being included in the output. The maximum output length was set according to the expected output format of each task. Task 1 used 16 output tokens, Task 2 used 2048 output tokens, and Task 3 Type 1 used 256 output tokens. For defini- tion generation (Task 4), we limited the maximum output length to 196 tokens. For the LLM-based evaluation step, we used athinking_budgetof 1024 and setmaxOutputTokensto 4096. We used temperature 0.0 and top-p0.95. The sampling op- tiondo_samplewas set to[FALSE]. To prevent generated definitions from containing Hanja, Chi- nese, or Japanese characters, we enabled a decod- ing time forbidden script token ban. In the context- based condition, the usage example was provided with the actual target neologism by replacing the [MASK]placeholder with the corresponding head- word before prompting the model. B.2 Prompts for LLM execution This appendix describes the prompt templates used for LLM execution in KoNeoBench. All prompts were designed to specify the role of the model, the input fields, the task objective, and the required output format. All prompts were written in Korean in order to match the language of the benchmark data and to minimize unintended translation ef- fects. However, English translations are provided only for readability and were not used for model inference. Placeholders enclosed in braces, such asterm,ex.,option_Aandcategory were replaced with instance-specific values. B.2.1 Prompts for Task 1 Task 1 evaluates whether a model can select the most appropriate Korean neologism for a masked position in an usage sentence. The prompt provides a Korean sentence containingMASKand five can- Task 1 Prompt Korean Prompt [System] 당신은신중한객관식문제풀이모델입니다. 한국어문장을읽고 [MASK]에가장적절한하나의 표제어를고르십시오. A, B, C, D, E중하나의대문자만출력하십시오. 선택이유를설명하지마십시오. [User] Question (용례): question Options: A. option_A B. option_B C. option_C D. option_D E. option_E A, B, C, D, E중하나의글자만답하십시오. Translation for Readability [System] You are a careful multiple-choice solver. Read the Korean sentence and choose the single best headword for [MASK]. Return only one uppercase letter among A, B, C, D, E. Do not explain your choice. [User] Question (용례): question Options: A. option_A B. option_B C. option_C D. option_D E. option_E Answer with only one letter: A, B, C, D, or E. Figure 7: Prompt template for Task 1 didate headwords labeled fromAtoE. The model is instructed to read the sentence and choose the single candidate that best fits the masked position. Figure 7 explains the prompt in both Korean and English. In both variants, the user prompt contains the same fields: the masked usage sentence and the five options. The system prompt emphasizes care- ful multiple-choice solving and explicitly prohibits explanatory text, so that the model output can be directly parsed as a single-choice answer. B.2.2 Prompts for Task 2 Task 2 evaluates whether a model can reconstruct the underlying components of a Korean neologism. Unlike ordinary morphological analysis, this task does not ask the model to segment the surface form mechanically. Instead, the model is instructed to infer the actual source components involved in the word-formation process and to restore shortened, blended, or transformed elements when appropri- ate. Figure 8 presents the prompt of the task2. It provides detailed analysis guidelines. The model should avoid arbitrary syllable-level splitting, con- vert numbers and foreign-language elements into Korean notation and recover abbreviated forms when possible. The final answer must contain only the reconstructed components connected by plus signs, without parentheses, quotation marks, com- mas, spaces, or additional explanation. B.2.3 Prompts for Task 3 Task 3 evaluates whether a model can identify the semantic category or specialized domain of Korean neologisms. The task consists of two formats: an odd-one-out format (Task 3 (Type 1)) and a direct category prediction format (Task 3 (Type 2)). Task 3(Type 1) evaluates whether a model can identify the neologism whose classification label differs from the other four candidates. Figure 10 presents the prompt used for the semantic-category setting, and Figure 11 presents the prompt used for the specialized-domain setting. Each prompt presents five Korean neologisms labeled fromAto E. Among them, four items share the same label under a given classification scheme, while one item belongs to a different label. The model is instructed to select the odd-one-out item based on the actual meaning of each neologism, rather than its surface form or option position. For the semantic-category setting, the prompt provides the complete list of 16 semantic categories. For the specialized-domain setting, the same prompt structure is used with the list of 43 domain labels. Thus, the two versions differ only in the label inventory provided to the model, while the task format, instruction structure, and output constraint remain identical. In Task 3 (Type 2), the model is given a single neologism and asked to classify it into the most ap- propriate category or domain. Figure 12–Figure 15 present the prompt templates used for Task 3 type 2 category prediction. This task evaluates whether LLMs can assign a given Korean neologism to the correct predefined category. We consider two types of labels: 16 semantic categories for general neolo- gisms and 43 specialized domains for terminolog- ical neologisms. For each label type, we use two input settings: a ‘w/o example’ setting, where the model receives only the neologism, and a ‘w/ ex- ample’ setting, where the model receives both the neologism and its usage example. In all settings, the full label inventory is provided in the prompt, and the model is instructed to output only the cor- responding category label, either a letter from A to P for semantic categories or a number from 1 to 43 for specialized domains. B.2.4 Prompts for Task 4 Task 4 evaluates whether a model can generate definition for a Korean neologism. We used two prompt variants for this task. In the ‘w/o example’ condition, the model receives only the target ne- ologism. Figure 16 present the prompt templates for this condition. In the ‘w/ example’ condition, the model receives both the target neologism and one usage example. Figure 17 present the prompt templates for this condition. Both variants include the same definition guidelines. The definition must be direct, concise, non-circular, and semantically clear. If pragmatic nuances such as praise or criti- cism to the meaning of the neologism, the model is instructed to reflect them in the definition when necessary. The required output is exactly one JSON object with two fields:termanddefinition. The termvalue must preserve the input headword with- out modification, and thedefinitionvalue must contain only one Korean sentence. Task 2 Prompt 당신은한국어신어(neologism)의구성분을분석하는언어학전문가입니다. [중요] 이작업은일반적인형태소분석이아닙니다.주어진신어를표면적으로기계분절하는것이아니라,신어를 이루는실제구성분을일정한규칙에따라복원하는작업입니다 .아래의분석지침에따라입력된신어의구성 성분을분석하십시오. [분석지침] 1. 표면음절을임의로자르지말고,실제조어에사용된구성분을찾으십시오. 2. 숫자,영어,외국어,알파벳이름등은모두한글표기로적으십시오. 3. 신어구성요소에용언(동사,형용사)이포함된경우,반드시기본형 “-다”형태로복원하십시오. 4. 축약된형태는가능한경우원래의전체형태로복원하십시오. • 다만,독립적으로널리쓰이거나사전에등재된축약형은유지할수있습니다. 5. 표준어규정에어긋나더라도실제언중이사용하는비규범적형태가구성분으로굳어져있으면, 표준형으로고치지말고그형태를유지하십시오. 6. 구성요소가구(phrase)단위인경우,내부띄어쓰기는생략하여붙여쓰십시오. 7. 복수해석이가능하더라도가장타당한분석하나만제시하십시오. [작업방식] 1. 먼저이신어가어떻게만들어졌는지 2–3문장이내로간결하게분석하십시오. 2. 각부분이어떤원래단어에서왔는지추론하십시오. 3. 분석이끝나면, “+”로연결된구성분을다시합쳤을때원래신어가유추가능한지검증하십시오. 4. 검증후,마지막줄에다음형식으로최종답을쓰십시오: 정답: 구성분1+구성분2+... [출력형식] • 분석은 2–3문장이내로간결하게작성하십시오. • 마지막줄의정답은 “정답: ”으로시작해야합니다. • 정답에는설명,괄호,따옴표,쉼표,공백을넣지마십시오. • “+”기호로만구성분을연결하십시오. [예시] 입력:관태기 분석: “관태기”는 “관계”의앞부분과 “권태기”의뒷부분을합친혼성어입니다.인간관계에권태를느끼는 시기를가리키는말입니다. “관계”+“권태기”로합치면 “관태기”로축약됩니다. 정답: 관계+권태기 입력:갑분싸 분석: “갑분싸”는 “갑자기”, “분위기”, “싸하다”의각앞글자를딴축약어입니다.용언인 “싸하다”는 기본형으로복원해야합니다.세단어를합치면원래표현이복원됩니다. 정답: 갑자기+분위기+싸하다 입력:망겜 분석: “망겜”은 “망하다”의어간 “망”과 “게임”의축약형 “겜”을합친말입니다.용언 “망하다”는기본형으로 복원합니다. “망하다”+“게임”으로분석됩니다. 정답:망하다+게임 입력:자동봉진 분석: “자동봉진”은학교생활기록부의 4대활동인 “자율활동”, “동아리활동”, “봉사활동”, “진로활동”의첫 글자를딴축약어입니다.네개의구가결합된형태입니다. 정답: 자율활동+동아리활동+봉사활동+진로활동 입력:일코노미 분석: “일코노미”는한국어 “일”(1인)과영어 economy(이코노미)를결합한혼성어입니다. 1인가구의소비 경제를뜻합니다. “일”+“이코노미”로분석됩니다. 정답: 일+이코노미 [최종지시] [중요], [분석지침], [작업방식], [출력형식], [예시]를참고하여답하시오. Figure 8: Prompt template for Task 2 Task 2 Prompt (Translation for Readability) You are a linguistic expert who analyzes the source words of Korean neologisms. [Important] This task is not ordinary morphological analysis. Rather than mechanically segmenting the given neologism at the surface level, your task is to reconstruct the actual source words that form the neologism according to a set of rules. Analyze the source words of the input neologism according to the following guidelines. [Analysis Guidelines] 1. Do not arbitrarily split surface syllables. Identify the actual source words used in word formation. 2. Numbers, English words, foreign words, alphabet names, and similar elements should all be written in Korean script. 3. If a source words includes a predicate, such as a verb or adjective, restore it to its dictionary form ending in “-다”. 4. If a component is abbreviated, restore it to its original full form whenever possible. • However, abbreviations that are widely used independently or listed in dictionaries may be retained. 5. If a non-standard form has become established as a source words in actual language use, do not correct it to the standard form; retain the form as used by speakers. 6. If a source words is a phrase, remove internal spaces and write it as a single unit. 7. Even if multiple analyses are possible, provide only the most plausible one. [Procedure] 1. First, briefly analyze how the neologism was formed in no more than 2–3 sentences. 2. Infer which original word each part comes from. 3. After the analysis, verify whether the original neologism can be inferred by combining the reconstructed source words with “+”. 4. After verification, write the final answer on the last line in the following format: Answer: source word1+source word2+... [Output Format] • Keep the analysis concise, within 2–3 sentences. • The final line must begin with “Answer: ”. • Do not include explanations, parentheses, quotation marks, commas, or spaces in the final answer. • Connect the source words only with the “+” symbol. [Examples] Input:관태기 Analysis: “관태기” is a blend formed by combining the beginning of “관계” and the latter part of “권태기”. It refers to a period of feeling tired or weary of interpersonal relationships. Combining “관계” + “권태기” yields the shortened form “관태기”. Answer: 관계+권태기 Input:갑분싸 Analysis: “갑분싸” is an abbreviation formed from the initial parts of “갑자기”, “분위기”, and “싸하다”. Since “싸하다” is a predicate, it should be restored to its dictionary form. The three words together reconstruct the original expression. Answer: 갑자기+분위기+싸하다 Input:망겜 Analysis: “망겜” is formed by combining “망”, from the verb stem of “망하다”, and “겜”, an abbreviated form of “게임”. The predicate “망하다” should be restored to its dictionary form. Therefore, it is analyzed as “망하다” + “게임”. Answer:망하다+게임 Input:자동봉진 Analysis: “자동봉진” is an abbreviation formed from the first syllables of the four major school record activities: “자율활동”, “동아리활동”, “봉사활동”, and “진로활동”. It consists of four phrase-level components. Answer: 자율활동+동아리활동+봉사활동+진로활동 Input:일코노미 Analysis: “일코노미” is a blend of the Korean word “일”, meaning one-person or single-person, and the English word economy, rendered in Korean as “이코노미”. It refers to the consumer economy of single-person households. It is analyzed as “일” + “이코노미”. Answer: 일+이코노미 [Final Instruction] Answer the input by following [Important], [Analysis Guidelines], [Procedure], [Output Format], and [Examples]. Figure 9: Prompt template for Task 2 (Translation version) Task 3 type 1 (semantic) Prompt === SYSTEM PROMPT === 당신은한국어신어의미범주를판별하는모델입니다. 이과제에서사용하는의미범주는다음 16개입니다: category 다섯개보기중네개는같은의미범주에속하고,하나만다른의미범주에속합니다. 각보기의실제의미를위의미범주체계에따라판단한뒤,나머지네개와의미범주가다른하나를고르십시오. 출력은반드시 JSON형식으로만작성하십시오. 형식: "answer": "A", "reason": "간단한 문장 근거" 규칙: • answer는반드시 A, B, C, D, E중하나여야합니다. • reason은한문장으로만작성하십시오. • 자세한추론과정은쓰지마십시오. • 보기위치가아니라의미범주차이를기준으로판단하십시오. • 주어진의미범주목록밖의임의범주를만들지마십시오. === USER PROMPT === 아래다섯개신어중네개는같은의미범주에속하고,하나만다른의미범주에속합니다. options_text 위의미범주목록을기준으로,나머지네개와의미범주가다른신어를고르십시오.반드시 JSON형식으로 답하십시오. Translation for Readability === SYSTEM PROMPT === You are a model that determines the semantic category of Korean neologisms. The semantic categories used in this task are the following 16 categories: category Among the five options, four belong to the same semantic category, and only one belongs to a different semantic category. Judge the actual meaning of each option according to the semantic category system above, and select the one whose semantic category differs from the other four. You must output only in JSON format. Format: "answer": "A", "reason": "A brief one-sentence justification" Rules: • answer must be one of A, B, C, D, or E. • reason must be written in exactly one sentence. • Do not provide a detailed reasoning process. • Make your decision based on the semantic category difference, not on the position of the options. • Do not create arbitrary categories outside the given semantic category list. === USER PROMPT === Among the following five Korean neologisms, four belong to the same semantic category, and only one belongs to a different semantic category. options_text Based on the semantic category list above, select the neologism whose semantic category differs from the other four. You must answer in JSON format. Figure 10: Prompt template for Task 3 (Type 1) (semantic categorization) Task 3 type 1 (specialized domain) Prompt === SYSTEM PROMPT === 당신은한국어신어의전문분야를판별하는모델입니다. 이과제에서사용하는전문분야는다음 43개입니다: category 다섯개보기중네개는같은전문분야에속하고,하나만다른전문분야에속합니다. 각보기의실제의미를위전문분야체계에따라판단한뒤,나머지네개와전문분야가다른하나를고르십시오. 출력은반드시 JSON형식으로만작성하십시오. 형식: "answer": "A", "reason": "간단한 문장 근거" 규칙: • answer는반드시 A, B, C, D, E중하나여야합니다. • reason은한문장으로만작성하십시오. • 자세한추론과정은쓰지마십시오. • 보기위치가아니라전문분야차이를기준으로판단하십시오. • 주어진전문분야목록밖의임의범주를만들지마십시오. === USER PROMPT === 아래다섯개신어중네개는같은전문분야에속하고,하나만다른전문분야에속합니다. options_text 위전문분야목록을기준으로,나머지네개와전문분야가다른신어를고르십시오.반드시 JSON형식으로 답하십시오. Translation for Readability === SYSTEM PROMPT === You are a model that determines the specialized domain category of Korean neologisms. The specialized domain categories used in this task are the following 43 categories: category Among the five options, four belong to the same specialized domain category, and only one belongs to a different specialized domain category. Judge the actual meaning of each option according to the specialized domain category system above, and select the one whose specialized domain category differs from the other four. You must output only in JSON format. Format: "answer": "A", "reason": "A brief one-sentence justification" Rules: • answer must be one of A, B, C, D, or E. • reason must be written in exactly one sentence. • Do not provide a detailed reasoning process. • Make your decision based on the specialized domain category difference, not on the position of the options. • Do not create arbitrary categories outside the given specialized domain list. === USER PROMPT === Among the following five Korean neologisms, four belong to the same specialized domain category, and only one belongs to a different specialized domain category. options_text Based on the specialized domain category list above, select the neologism whose specialized domain category differs from the other four. You must answer in JSON format. Figure 11: Prompt template for Task 3 (Type 1) (specialized domain categorization) Task 3 type 2 Prompt (semantic category, term only) SYSTEM 한국어신어의미범주를분류하는전문가로서, 주어진신어를가장적절한범주하나로분류하시오. USER [색인표제어] term [의미범주목록] A:〔개념〕 B:〔경제생활〕 C:〔교육〕 D:〔동식물〕 E: 〔문화〕 F:〔보건 ·의학〕 G:〔사회생활〕 H:〔삶〕 I: 〔식생활〕 J:〔의생활〕 K:〔인간〕 L:〔자연〕 M: 〔정보 ·기술〕N:〔정치와행정〕O:〔종교〕P:〔주생활〕 출력형식: • 반드시 A–P중해당하는하나의문자만 출력하세요. • 다른설명은작성하지마세요. 정답: Translation for Readability SYSTEM As an expert in classifying the semantic categories of Korean neologisms, classify the given neologism into the single most appropriate category. USER [Headword] term [Semantic category list] A:〔Concept〕 B:〔Economic Life〕 C:〔Education〕 D: 〔Animals and Plants〕 E:〔Culture〕 F:〔Health · Medicine〕 G: 〔 Social Life〕H:〔Life〕I:〔Dietary Life〕J:〔Clothing Life〕K: 〔Human〕 L:〔Nature〕 M:〔Information · Technology〕 N: 〔Politics and Administration〕 O:〔Religion〕 P:〔Housing life〕 Output format: • Output only one corresponding letter from A to P. • Do not write any explanation. Answer: Figure 12: Prompt template for Task 3 (Type 2) semantic-category with the neologism only. Task 3 type 2 Prompt (semantic category, with example) SYSTEM 한국어신어의미범주를분류하는전문가로서, 주어진신어를가장적절한범주하나로분류하시오. USER [색인표제어] term [용례] ex. [의미범주목록] A:〔개념〕 B:〔경제생활〕 C:〔교육〕 D:〔동식물〕 E: 〔문화〕 F:〔보건 ·의학〕 G:〔사회생활〕 H:〔삶〕 I: 〔식생활〕 J:〔의생활〕 K:〔인간〕 L:〔자연〕 M: 〔정보 ·기술〕 N:〔정치와행정〕O:〔종교〕P:〔주생활〕 출력형식: • 반드시 A–P중해당하는하나의문자만 출력하세요. • 다른설명은작성하지마세요. 정답: Translation for Readability SYSTEM As an expert in classifying the semantic categories of Korean neologisms, classify the given neologism into the single most appropriate category. USER [Headword] term [Usage] ex. [Semantic category list] A:〔Concept〕 B:〔Economic Life〕 C:〔Education〕 D: 〔Animals and Plants〕 E:〔Culture〕 F:〔Health · Medicine〕 G: 〔 Social Life〕H:〔Life〕I:〔Dietary Life〕J:〔Clothing Life〕K: 〔Human〕 L:〔Nature〕 M:〔Information · Technology〕 N: 〔Politics and Administration〕 O:〔Religion〕 P:〔Housing life〕 Output format: • Output only one corresponding letter from A to P. • Do not write any explanation. Answer: Figure 13: Prompt template for Task 3 (Type 2) semantic-category with the neologism and Usage. Task 3 type 2 Prompt (specialized domain, term only) SYSTEM 한국어신어의전문분야를분류하는전문가로서, 주어진신어를가장적절한분야하나로분류하시오. USER [색인표제어] term [전문분야목록] 1:『건설』 2:『경영』 3:『경제』 4:『교육』 5:『교통』 6: 『군사』 7:『기계』 8:『농업』 9:『동물』 10:『동식물』 11:『매체』 12:『물리』 13:『법률』 14:『보건일반』 15:『복식』 16:『복지』 17:『불교』 18:『사회일반』 19:『산업일반』 20:『생명』 21:『서비스업』 22: 『수의』 23:『식물』24:『심리』25:『약학』26:『언어』 27:『역사』 28:『예체능일반』 29:『음악』 30:『의학』 31:『재료』 32:『전기ᆞ전자』 33:『정보ᆞ통신』 34: 『정치』 35:『종교일반』 36:『지구』 37:『지명』 38: 『천문』39:『철학』40:『체육』41:『행정』42:『화학』 43:『환경』 출력형식: • 반드시목록에제시된 ’번호’하나만출력하세요. • 다른설명은작성하지마세요. 정답: Translation for Readability SYSTEM As an expert in classifying the specialized domain of Korean neologisms, classify the given neologism into the single most appropriate domain. USER [Headword] term [Specialized domain list] 1: [Construction] 2: [Business Management] 3: [Economics] 4: [Education] 5: [Transportation] 6: [Military] 7: [Mechanical Engineering] 8: [Agriculture] 9: [Animals] 10: [Animals and Plants] 11: [Media] 12: [Physics] 13: [Law] 14: [General Health] 15: [Clothing and Textiles] 16: [Welfare] 17: [Buddhism] 18: [General Society] 19: [General Industry] 20: [Life Science] 21: [Service Industry] 22: [Veterinary Medicine] 23: [Plants] 24: [Psychology] 25: [Pharmacy] 26: [Language] 27: [History] 28: [General Arts and Sports] 29: [Music] 30: [Medicine] 31: [Materials] 32: [Electrical and Electronic Engineering] 33: [Information and Communication] 34: [Politics] 35: [General Religion] 36: [Earth Science] 37: [Place Names] 38: [Astronomy] 39: [Philosophy] 40: [Physical Education] 41: [Public Administration] 42: [Chemistry] 43: [Environment] Output format: • Please make sure to print only one of the ’numbers’ presented in the list. • Do not write any explanation. Answer: Figure 14: Prompt template for Task 3 (Type 2) special- ized domain with the neologism only. Task 3 type 2 Prompt (specialized domain, with ex.) SYSTEM 한국어신어의전문분야를분류하는전문가로서, 주어진신어를가장적절한분야하나로분류하시오. USER [색인표제어] term [용례] ex. [전문분야목록] 1:『건설』 2:『경영』 3:『경제』 4:『교육』 5:『교통』 6: 『군사』 7:『기계』 8:『농업』 9:『동물』 10:『동식물』 11:『매체』 12:『물리』 13:『법률』 14:『보건일반』 15:『복식』 16:『복지』 17:『불교』 18:『사회일반』 19:『산업일반』 20:『생명』 21:『서비스업』 22: 『수의』 23:『식물』24:『심리』25:『약학』26:『언어』 27:『역사』 28:『예체능일반』 29:『음악』 30:『의학』 31:『재료』 32:『전기ᆞ전자』 33:『정보ᆞ통신』 34: 『정치』 35:『종교일반』 36:『지구』 37:『지명』 38: 『천문』39:『철학』40:『체육』41:『행정』42:『화학』 43:『환경』 출력형식: • 반드시목록에제시된 ’번호’하나만출력하세요. • 다른설명은작성하지마세요. 정답: Translation for Readability SYSTEM As an expert in classifying the specialized domain of Korean neologisms, classify the given neologism into the single most appropriate domain. USER [Headword] term [Usage] ex. [Specialized domain list] 1: [Construction] 2: [Business Management] 3: [Economics] 4: [Education] 5: [Transportation] 6: [Military] 7: [Mechanical Engineering] 8: [Agriculture] 9: [Animals] 10: [Animals and Plants] 11: [Media] 12: [Physics] 13: [Law] 14: [General Health] 15: [Clothing and Textiles] 16: [Welfare] 17: [Buddhism] 18: [General Society] 19: [General Industry] 20: [Life Science] 21: [Service Industry] 22: [Veterinary Medicine] 23: [Plants] 24: [Psychology] 25: [Pharmacy] 26: [Language] 27: [History] 28: [General Arts and Sports] 29: [Music] 30: [Medicine] 31: [Materials] 32: [Electrical and Electronic Engineering] 33: [Information and Communication] 34: [Politics] 35: [General Religion] 36: [Earth Science] 37: [Place Names] 38: [Astronomy] 39: [Philosophy] 40: [Physical Education] 41: [Public Administration] 42: [Chemistry] 43: [Environment] Output format: • Please make sure to print only one of the ’numbers’ presented in the list. • Do not write any explanation. Answer: Figure 15: Prompt template for Task 3 (Type 2) special- ized domain with the neologism and example. Task 4 Prompt (w/o ex.) [Role] 당신은한국어신어사전편찬전문가입니다. 입력으로주어진표제어(신어,유행어,비격식표현)의미를파악하여사전식정의를작성하십시오. [Task] 표제어만주어집니다. 표제어의형태,일반적인용법,널리알려진의미를바탕으로가장타당한정의를작성하십시오. [Definition Guidelines] 1.정의는반드시한국어한문장으로작성합니다. 2.정의는반드시사전식문체로작성할것. 3.표제어의미를직접적이고단정적으로설명하되,근거없는세부사항을임의로덧붙이지마십시오. 4.표제어자체를그대로반복하는순환정의는피하십시오. 5.불필요하게장황하게쓰지말고,핵심의미가분명하게드러나도록간결하게작성하십시오. 6.긍정,부정,풍자,비하등의뉘앙스가의미의핵심이면정의에반영하십시오. 7.예문이없으므로,특정상황으로지나치게좁히지말고보편적으로통용되는의미를중심으로정의하십시오. [Few-shot Examples] [Input] "term": "갓생" [Output] "term": "갓생", "definition": "성실하고모범적으로자기관리를하며살아가는삶이나태도" [Output Rules] –반드시 JSON객체하나만출력하십시오. –출력형식은반드시다음과같아야합니다: "term": "...", "definition": "..." – ‘term‘값은입력으로주어진표제어를수정없이그대로사용하십시오. – ‘definition‘값은한국어한문장만포함하십시오. –다른키를추가하지마십시오. –설명,주석,해설,마크다운코드블록을절대출력하지마십시오. [Final Instruction] 지금부터주어지는입력에대해,위규칙을따른단일 JSON객체만출력하십시오. Translation for Readability [Role] You are an expert lexicographer of Korean neologisms. Identify the meaning of the given headword (neologism, buzzword, or informal expression) and write a dictionary-style definition. [Task] Only the headword is provided. Based on the form of the headword, its general usage, and its widely known meaning, write the most appropriate definition. [Definition Guidelines] 1. The definition must be written as a single Korean sentence. 2. The definition must follow a dictionary-style tone. 3. Explain the meaning of the headword directly and assertively, but do not add unsupported details arbitrarily. 4. Avoid circular definitions that simply repeat the headword itself. 5. Do not write unnecessarily lengthy definitions; keep the definition concise while clearly conveying the core meaning. 6. If nuances such as positivity, negativity, satire, or derogation are central to the meaning, reflect them in the definition. 7. Since no example sentence is provided, do not narrow the meaning to a specific situation; instead, focus on the meaning that is generally used. [Few-shot Examples] [Input] "term": "갓생" [Output] "term": "갓생", "definition": "성실하고모범적으로자기관리를하며살아가는삶이나태도" [Output Rules] – Output exactly one JSON object. – The output format must be exactly as follows: "term": "...", "definition": "..." – The value of ‘term‘ must be identical to the input headword without any modification. – The value of ‘definition‘ must contain only one Korean sentence. – Do not add any other keys. – Never output explanations, comments, annotations, or Markdown code blocks. [Final Instruction] From now on, for each given input, output only a single JSON object that follows the rules above. Figure 16: Prompt template for Task 4 with the neologism only. Task 4 Prompt (w/ ex.) [Role] 당신은한국어신어사전편찬전문가입니다.입력으로주어진표제어와예문을바탕으로,해당신어의미를사전식으로 정확히정의하십시오. [Task] 입력으로다음이주어집니다. -표제어(term):한국어신어또는비격식표현 -예문(example):해당표제어가실제사용된문장 1개 예문에서확인되는용법을바탕으로표제어의핵심의미를파악하고,사전식문체의한문장정의를작성하십시오. [Definition Guidelines] 1.정의는반드시한국어한문장으로작성합니다. 2.정의는반드시사전식문체로작성할것. 3.예문에나타난의미를충실히반영하되,예문의특정장면이나사건에만묶이지말고보편적인의미로정리하십시오. 4.다만예문하나만으로확인되지않는세부의미까지임의로확장하지마십시오. 5.예문의감정적표현,과장,풍자,비유는그대로베끼지말고,그안의핵심의미를객관적으로정리하십시오. 6.표제어자체를그대로반복하는순환정의는피하십시오. 7.의미상중요한뉘앙스(비하,조롱,찬양,자조등)는필요할때정의에반영하십시오. 8.정의는장황하지않게,핵심의미가분명히드러나도록간결하게작성하십시오. [Few-shot Examples] [Input] "term": "갓생", "example": "나오늘부터갓생살거다." [Output] "term": "갓생", "definition": "성실하고모범적으로자기관리를하며살아가는삶이나태도" [Output Rules] –반드시 JSON객체하나만출력하십시오. –출력형식은반드시다음과같아야합니다: "term": "...", "definition": "..." – ‘term‘값은입력으로주어진표제어를수정없이그대로사용하십시오. – ‘definition‘값은한국어한문장만포함하십시오. –다른키를추가하지마십시오. –설명,주석,해설,마크다운코드블록을절대출력하지마십시오. [Final Instruction] 지금부터주어지는입력에대해,위규칙을따른단일 JSON객체만출력하십시오. Translation for Readability [Role] You are an expert lexicographer of Korean neologisms. Based on the given headword and example sentence, accurately define the meaning of the neologism in a dictionary-style format. [Task] The following inputs are provided. - Headword (term): a Korean neologism or informal expression - Example sentence (example): one sentence in which the headword is actually used Based on the usage observed in the example sentence, identify the core meaning of the headword and write a one-sentence definition in a dictionary-style tone. [Definition Guidelines] 1. The definition must be written as a single Korean sentence. 2. The definition must follow a dictionary-style tone. 3. Faithfully reflect the meaning shown in the example sentence, but generalize it rather than restricting it to the specific scene or event in the example. 4. However, do not arbitrarily expand the definition to include detailed meanings that cannot be confirmed from the single example sentence. 5. Do not copy emotional expressions, exaggeration, satire, or figurative wording from the example sentence as they are; instead, objectively summarize the core meaning contained in them. 6. Avoid circular definitions that simply repeat the headword itself. 7. If semantically important nuances such as derogation, mockery, praise, or self-deprecation are relevant, reflect them in the definition when necessary. 8. Do not make the definition verbose; keep it concise while clearly conveying the core meaning. [Few-shot Examples] [Input] "term": "갓생", "example": "나오늘부터갓생살거다." [Output] "term": "갓생", "definition": "성실하고모범적으로자기관리를하며살아가는삶이나태도" [Output Rules] – Output exactly one JSON object. – The output format must be exactly as follows: "term": "...", "definition": "..." – The value of ‘term‘ must be identical to the input headword without any modification. – The value of ‘definition‘ must contain only one Korean sentence. – Do not add any other keys. – Never output explanations, comments, annotations, or Markdown code blocks. [Final Instruction] From now on, for each given input, output only a single JSON object that follows the rules above. Figure 17: Prompt template for Task 4 with the neologism and example. B.3 Evaluation Criteria for Task 2 Since the order of source words is important in Task 2, evaluation is based on source words se- quences. Coverage is defined as the proportion of gold source words that are correctly recovered by the model at the correct positions. Precision is defined as the proportion of generated source words that are correctly recovered at the correct positions. Scoring examples are summarized in Table 7. Each example compares a gold source words sequence with a predicted sequence and illus- trates how scores are computed based on correctly aligned source words. These examples show how partially correct predictions, order mismatches, and predictions containing unnecessary constituents are evaluated in Task 2. Table 8 further presents a range of example calculations for coverage, precision, and F1 used in evaluating Task 2. B.4 Evaluation Criteria for Task 4 For Task 4, definitions are scored by Gemini in an LLM-as-a-judge setup. Table 9 presents the scoring criteria defined by lexicography experts. The crite- ria evaluate three dimensions: semantic adequacy, fluency, and factual reliability. Semantic adequacy includes three subcriteria. Semantic equivalence checks whether the gener- ated definition preserves the core meaning of the gold definition, even with different wording. Cov- erage measures whether the key content words of the gold definition are included, capturing possible omissions of essential meaning components. Prag- matic equivalence evaluates whether pragmatic meanings such as evaluative stance, emotion, at- titude, disparagement, or mockery are properly re- flected when needed. Fluency includes conciseness, which evaluates whether the definition is brief and free of unnec- essary repetition; non-circularity, which checks whether the headword or its synonyms are avoided; and lexicographic convention, which evaluates whether the definition follows dictionary-style met- alanguage or a genus–differentia structure. Factual reliability assesses whether the generated definition contains hallucinated content, unsup- ported details, or excessive generalization. The fi- nal score is the sum of the subcriterion scores, with a maximum of 10 points. Gemini is instructed to output only the individual subcriterion scores, not subtotals or the final total. The evaluation prompt is shown in Figure 19, the output schema in Fig- ure 18, and an English translation of the prompt in Figure 20. Expected JSON Output Schema for Gemini-based Evaluation semantic_equivalence0 to 2 (Integer). coverage0 to 2 (Integer). gold_content_words A list of three objects, each containing word and in_pred. gold_has_pragmaticBoolean value indicating whether the gold definition contains pragmatic meaning. pred_has_pragmaticBoolean value indicating whether the predicted definition contains pragmatic meaning. polarity_matchBoolean value indicating whether the pragmatic polarity matches. pragmatic_equivalenceEither 0 or 2 (Integer). concisenessEither 0 or 1 (Integer). non_circularityEither 0 or 1 (Integer). lexicographic_conventionEither 0 or 1 (Integer). factualityEither 0 or 1 (Integer). error_typesA list of error type strings. commentA brief natural-language comment. confidenceNumeric confidence score from 0 to 1. Figure 18: Expected JSON output schema for Gemini- based evaluation of Task 4 definition generation. CaseGold sequencePredictionAlignedCoveragePrecision Correct 홈 +오마카세 홈 +오마카세 홈,오마카세2 of 2 = 1.002 of 2 = 1.00 Partial 홈 +오마카세 홈 홈1 of 2 = 0.501 of 1 = 1.00 Over-generation 홈 +오마카세 홈 +오마카세 +세트 홈,오마카세2 of 2 = 1.002 of 3 = 0.67 Wrong 홈 +오마카세 홈페이지 +마케팅 +가세–0 of 2 = 0.000 of 3 = 0.00 Wrong order 홈 +오마카세 오마카세 +홈–0 of 2 = 0.000 of 2 = 0.00 Table 7: Scoring cases for source word identification (Task 2) of ‘홈마카세’ CaseGold sequencePredictionAlignedCoveragePrecisionF1 단마토 (Danmato; Sweet-Tomato) 달다(Sweet) +토마토(Tomato) 달다(Sweet) +토마토(Tomato) 달다(Sweet), 토마토(Tomato) 2 of 2 = 1.00 2 of 2 = 1.001.0 헬스닥 (Hell-sdaq) 헬(Hell) +코스닥(Kosdaq) 헬스(Health) +닥터(Doctor) –0 of 2 = 0.000 of 2 = 0.000 오하운 (Ohaun; Today’s exercise) 오늘(Today) +하루(Day) +운동(Exercise) 오늘(Today) +하체(Lower body) +운동(Exercise) 오늘(Today), 운동(Exercise) 2 of 3 = 0.672 of 3 = 0.670.67 스팔로미 (Th-fallome) 스레드(Thread) +팔로미 (Follow-Me) 스포일러(Spoiler) +팔로미 (Follow-Me) 팔로미 (Follow-Me) 1 of 2 = 0.501 of 2 = 0.500.5 Table 8: Examples of GPT-5.4 predictions and scores for Task 2 C Analysis on Evaluation Results C.1 Semantic Categorization: General vs. Neologism This section describes how the general-word results in Figure 1 were obtained. Since the performance for neologism is reported using Task 3 (Type 1), we evaluated the performance for general words un- der the same setup. We first selected 16,738 nouns from the headword list of the Basic Korean Dictio- nary that satisfied predefined conditions. From this set, we randomly sampled 1,785 items (the same number as the neologisms) and constructed test sets following the same procedure as Task 3 (Type 1). This experiment was repeated 10 times to obtain the average semantic categorization performance on general words. Results for three representative LLMs are shown in Figure 1, while Figure 21 re- ports all nine models. One caveat is that the se- mantic categories represented in the general-word and neologism sets may vary depending on random sampling. To account for this variation, Figure 21 also reports standard deviations. C.2 Overall Performances across Tasks We further examine whether the sub-tasks in KoNeoBench show common performance trends across models and what task-specific difficulties they reveal. For this analysis, we select one rep- resentative setting from each task group: Task 1 (Type 1) for contextual identification, Task 2 (F1) for source word identification, Task 3 (Semantic, Type 1) and Task 3 (Specialized Domain, Type 1) for category-based discrimination, and Task 4 with usage examples for definition generation. Fig- ure 22 and 6(a) shows performance distributions by model release date and scale. Overall, more recent and larger models tend to perform better across tasks. The GPT series is consistently among the strongest, especially in Task 1, Task 3, and Task 4, suggesting relatively strong ability in contextual identification, categorical classification, and defini- tion generation for Korean neologisms. The most notable difference in difficulty appears in Task 2. Its F1 scores are generally lower than those of the multiple-choice and classification tasks, and model gaps are also larger. This suggests that roughly understanding a neologism and identifying its source words are different challenges. While Task 1 and Task 3 allow models to rely on con- textual or categorical cues, Task 2 requires direct identification of source words that are not explicitly present in the surface form. In addition, year-wise performance distributions for Tasks 1 and 2, Task 3 Type 1, and Task 3 Type 2 are provided in Figures 23, 24, and 25, respectively. Tasks 1 and 2 show an overall decline in perfor- mance for more recent neologisms, similar to Task 4. In contrast, Task 3 exhibits relatively little year- to-year variation in both the general-language and specialized-domain settings Figure 26 shows the Pearson and Spearman cor- relations of model performance across representa- tive tasks. Most task pairs are positively correlated, and many exceed 0.7. However, the correlations are not uniformly high. In particular, Task 4 shows weaker correlations with some categorization tasks, suggesting that definition generation requires addi- tional abilities (reconstructing the core meaning of a neologism and expressing it in dictionary style) that are not fully captured by multiple-choice or classification tasks. These results support the multi-task design of KoNeoBench. High inter-task correlations indicate that the tasks measure a shared ability, namely Ko- rean neologism understanding, while lower corre- lations and task-specific differences show that each task captures distinct linguistic abilities and error patterns. Rather than reducing neologism under- standing to a single score, KoNeoBench evaluates it from complementary perspectives: contextual identification, source word identification, catego- rization, and definition generation. C.3 Performance Analysis on Task 4 C.3.1 Gemini Scores vs. Human Scores To assess the reliability of Gemini-based evalua- tion, we also conducted human validation with a lexicography expert. The validation set consisted of definitions generated by GPT-5.4 in the setting with both the neologism and its usage example. We first sampled 45 items, selecting five from each Gem- ini total-score level from 2 to 10, and then added 205 more in proportion to the overall score distri- bution, for a total of 250 definitions. The expert evaluated them using the same rubric as in Table 9, and we compared the resulting scores with the Gemini scores at both the total-score and subcrite- rion levels. Figure 27 summarizes the comparison. Fig- ure 27(a) shows a bubble plot of the total-score distributions, where each point represents a hu- man–Gemini score pair and bubble size indicates the number of items. The two evaluations are posi- tively correlated (r = 0.708), with linear regression results ofR 2 = 0.50, p < 10 −38 . This indicates that definitions scored highly by Gemini also tend to be scored highly by the expert. As shown in Fig- ure 27(b), the discrepancy is relatively small for criteria related to semantic adequacy, suggesting that Gemini-based evaluation is reasonably con- sistent in assessing semantic quality. Gemini can therefore be used as a useful auxiliary measure of overall definition quality. However, Gemini tends to assign slightly higher scores than the expert, as seen in the average sub- criterion scores in Figure 27(b)-(e). This tendency is especially clear for conciseness, lexicographic convention, and factuality, where Gemini appears more lenient. This suggests that Gemini may judge formal quality, dictionary style, and factual errors more permissively than a human expert. These results suggest that LLM-based evaluation is useful for comparing relative model differences in Task 4, but limited as an absolute measure of lex- icographic quality. The Task 4 scores in the main text should therefore be interpreted as automatic scores for comparing models under the same con- ditions, rather than as absolute quality judgments. We also note that the human validation covers only 250 definitions generated by GPT-5.4, not all mod- els or outputs. Even so, the analysis suggests that Gemini-based evaluation is reasonably consistent with expert judgment and can serve as a scalable approximate method for comparing large sets of generated definitions. C.3.2 Distribution of Scores by Subcriterion for Task 4 Figure 28 shows the average subcriterion scores of each model on Task 4. We analyze definition gener- ation from three perspectives: semantic adequacy, lexicographic fluency, and factuality. Figure 28(a) shows the results when only the neologism is given, and Figure 28(b) shows the results when a usage example is also provided. Overall, most models improve on all three di- mensions when usage examples are provided. The largest gain appears in semantic adequacy, suggest- ing that usage examples are crucial for inferring the actual meaning and context of Korean neologisms. Since many neologisms cannot be interpreted reli- ably from form alone, usage context helps models construct their core meaning more accurately. By contrast, differences in lexicographic fluency are relatively small, suggesting that LLMs can generate superficially dictionary-like definitions even with- out fully understanding the target meaning. This indicates that the main difficulty of Task 4 lies less in definitional form than in semantic accuracy. Scores on factuality also improve across all mod- els, suggesting that usage examples help reduce hallucinated or unsupported content. The subcriterion-level analysis of Task 4 clearly shows the importance of contextual information in Korean neologism definition generation. Even without usage examples, some models can produce relatively natural definitions, but performance is more stable with examples, especially in terms of semantic adequacy and factuality. In addition, the subcriterion-level analysis reveals differences that are not visible when definition generation is eval- uated only by a single total score. These results suggest that Task 4 evaluates not just generation ability, but also neologism interpretation, context- based reasoning, and lexicographic description. D Human Performance Evaluation We conducted a human evaluation on four settings: Task 1 with particle-attached forms, Task 2, and the semantic and specialized-domain categories of Task 3 (Type 1). For each setting, 50 neologisms were sampled with consideration of their frequency distribution. Twenty native Korean speakers par- ticipated, including 10 linguistics majors and 10 non-majors. The LLMs were evaluated on the same subsets. The results are reported in Table 10. The comparison shows different patterns across tasks. On Tasks 1 and 2, linguistics majors per- formed better than both GPT-5.4 and non-majors. This may reflect the greater role of Korean morpho- logical and word-formation knowledge in these tasks. The pattern was less consistent for Task 3. Linguistics majors performed better than non- majors on the semantic category setting, while GPT- 5.4 performed better than both non-majors and the overall human group on the specialized-domain set- ting. Thus, the relative difficulty for humans and models differs depending on the type of knowledge tested. We used paired Wilcoxon signed-rank tests over the 50 items to examine these differences. On Task 1, linguistics majors significantly outperformed GPT-5.4 (W = 175.0,p = .0217) and non-majors (W = 82.5,p < .001). The same pattern was observed on Task 2, where linguistics majors out- performed GPT-5.4 (W = 311.0,p = .0318) and non-majors (W = 38.5,p < .001). On the seman- tic category setting of Task 3, linguistics majors per- formed significantly better than non-majors (W = 132.5,p < .001). On the specialized-domain set- ting, GPT-5.4 performed significantly better than non-majors (W = 270.5,p = .0374) and the over- all human group (W = 285.0,p = .0370). No other comparisons were significant. Because the evaluation was conducted on a relatively small sub- set, a larger human evaluation remains for future work. DimensionSubcriterionScore Description Semantic adequacy (0–6) Semantic equivalence 0–2 •Evaluates the semantic equivalence between the gold definition and the predicted definition. (gold definition과 pred definition의미적등가성평가) • 2 points: core meaning matches • 1 point: core meaning is similar but some details differ • 0 points: core meaning does not match Coverage0–2 • Extracts three key content words from the gold definition and evalu- ates how many of them are covered by the predicted definition. (gold definition에서핵심내용어3개를추출하고, pred definition 이를얼마나포함하는지평가) • 2 points: all three are covered • 1 point: one or two are covered • 0 points: none are covered Pragmatic equivalence 0 or 2 •Evaluates whether the pragmatic meaning * of the gold definition and the predicted definition is equivalent. (gold와 pred의화용적의미가일치하는지평가) •2 points: both the gold and predicted definitions contain pragmatic meaning with consistent polarity, or neither definition contains prag- matic meaning •0 points: the presence or polarity of pragmatic meaning is inconsis- tent Fluency (0–3) Conciseness0–1 •Evaluates whether the predicted definition avoids unnecessary mod- ifiers or repetitive expressions. (pred definition에불필요한수식이나반복표현이없는지평가) • 1 point: concise • 0 points: contains unnecessary modifiers or repetition Non-circularity0–1 •Evaluates whether the predicted definition avoids circularly defining the target word using the target word itself or its close synonyms (pred definition이표제어나표제어의유의어를사용하여순환적 으로정의하지않는지평가) • 1 point: non-circular definition • 0 points: circular definition Lexicographic convention 0–1 •Evaluates whether the predicted definition appropriately follows lex- icographic conventions, such as the use of definitional metalanguage or a genus–differentia structure. (pred definition이사전 정의문 형식의 메타언어또는 유개념–종 차구조를적절히따르는지평가) • 1 point: follows lexicographic conventions • 0 points: does not follow lexicographic conventions. Factuality (0–1) Factuality0–1 •Evaluates whether the predicted definition contains hallucinations, unsupported specific claims, or overgeneralizations. (pred definition에hallucination,근거 없는 구체적 진술,과도한 일반화가있는지평가) • 1 point: no major issue • 0 points: at least one clear and significant issue is present Note. * Pragmatic meaning refers to evaluative or affective meanings associated with the defined term, including attitudes such as support, encouragement, derogation, hatred, or mockery. The total score is 10 points and is computed post hoc as the sum of the subcriterion scores. Table 9: Gemini-based evaluation rubric for Task 4 definition generation. Prompt for Task 4 Definition Judgement 당신은한국어신어정의생성결과를엄격하게채점하는평가자다. 주어진 gold definition(정답정의)와 pred definition(모델생성정의)을비교하여, example(용례)을문맥정보로 참고해채점하라.반드시아래 A–C의 3개기준만사용하고,모든점수는명시된범위내의정수(integer)여야한다. (음수불가) A) Semantic Adequacy A-1) semantic_equivalence: 0–2 points – gold definition과 predicted definition의미적등가성평가 –표현이달라도의미가일치하면높은점수부여 (패러프레이즈허용) • 2점핵심의미일치| 1점 :핵심의미는일치하나,세부사항불일치| 0점 :핵심의미불일치 A-2) coverage: 0–2 points – gold definition에서핵심적내용어(content word) 3개추출후, pred definition이를얼마나포함하는지평가 –한국어동사,형용사의활용형은동일한내용어임 –기능어(Function word)나불용어(Stopwords)제외 – gold definition에서추출한내용어와각내용어가 pred에포함되는지여부를 gold_content_words필드에출력 • 2점: 3개모두포함| 1점: 1 2개포함| 0점:미포함 –추가출력필드: gold_content_words: [“word”: “. . . ”, “in_pred”: true/false, . . . ] A-3) pragmatic_equivalence: 0점또는 2점 – gold의화용적의미와 pred의화용적의미가일치하는지평가 ※화용적의미:정의문기술대상에대한평가적의미(긍정/부정)나화자의감정혹은태도(지/응원,혐오,비하, 조롱등)등.화용적의미가일치한다는것은 gold와 pred의극성(polarity)이일치하는지를의미한다.즉, gold가 긍정이며 pred도긍정, gold가부정이면 pred도부정이어야한다. gold와 pred의극성이반대(예:긍정↔부정) 라면불일치로본다. 다음다섯가지경우를모두채점대상으로함 1) gold에화용적의미포함 + pred에화용적의미포함 +극성일치 = 2점 2) gold에화용적의미포함 + pred에화용적의미포함 +극성불일치 = 0점 3) gold에화용적의미포함 + pred에화용적의미포함 = 0점 4) gold에화용적의미포함 + pred에화용적의미포함 = 0점 5) gold에화용적의미포함 + pred에화용적의미포함 = 2점 –추가출력필드(true/false): gold_has_pragmatic, pred_has_pragmatic, polarity_match B) fluency : 0–3점 –정의문의형식적합성평가 –아래 3개하위기준을각 0점또는 1점으로채점 B-1) conciseness (간결성): 0점또는 1점 – gold definition과비교하여, pred definition에불필요한수식이나반복표현이없는지평가 • 1점:불필요한수식이나반복표현없이간결함| 0점:불필요한수식이나반복표현이포함됨 B-2) non-circularity(비순환성): 0점또는 1점 – pred definition에해당표제어나,해당표제어의유의어등을사용한순환적정의가없는지평가 • 1점:비순환적정의문| 0점:순환적정의문 B-3) lexicographic convention (사전학적관습준수): 0점또는 1점 – pred definition에사전정의문형식의메타언어(예: –을이르는말등)나유개념–종차구조(예:소확행 - [소하지만확실한](종차) + [행복](유개념))를적절하게사용하는지평가 • 1점:사전학적관습준수| 0점:사전학적관습미준수 C) factuality: 0점또는 1점 – hallucination,근거없는구체적진술,과도한일반화여부평가 • 1점:중대한 hallucination/근거없는구체화/과도한일반화없음 • 0점:위문제중하나라도크고명확하게존재함 중요규칙: 1) 모든점수는명시된범위내의정수여야한다. (음수불가) 2) 출력은각하위항목점수만포함하며,소계(semantic_adequacy, fluency)와총점(total)은출력하지마라. 합산은사후처리한다. 3) semantic_equivalence가맞으면표현이달라도감점하지마라. 4) example은의미를해석하는문맥으로만사용하라. gold를넘어서는새로운의미를추가하는근거로 사용하지마라. 5) pred가너무짧더라도핵심의미를담으면점수를줄수있다. 6) pred가장황하더라도근거없는내용이들어가면 factuality를감점하라. 7) 출력은반드시 JSON객체하나만반환하라. 8) markdown,코드블록,설명문없이아래의스키마로 JSON만출력하라. Figure 19: Prompt template for Task 4 definition generation. Prompt for Task 4 Definition Judgement(For readability) You are a strict evaluator of Korean neologism definition generation. Compare the given gold definition and predicted definition, using the example only as contextual information. Use only the following three criteria A–C, and all scores must be integers within the specified ranges. Negative scores are not allowed. A) Semantic Adequacy A-1) semantic_equivalence: 0–2 points – Evaluate the semantic equivalence between the gold definition and the predicted definition. – Assign a high score when the meanings match, even if the wording differs. Paraphrases are allowed. • 2 points: core meaning matches | 1 point: core meaning matches partially but some details differ | 0 points: core meaning does not match A-2) coverage: 0–2 points – Extract three key content words from the gold definition and evaluate how many of them are covered by the predicted definition. – Inflected forms of Korean verbs and adjectives should be treated as the same content word. – Function words and stopwords should be excluded. – Output the content words extracted from the gold definition and whether each word is included in the predicted definition in the gold_content_words field. • 2 points: all three words are included | 1 point: one or two words are included | 0 points: none are included – Additional output field: gold_content_words: ["word": "...", "in_pred": true/false, ...] A-3) pragmatic_equivalence: 0 or 2 points – Evaluate whether the pragmatic meaning of the gold definition and the predicted definition is equivalent. Pragmatic meaning refers to evaluative meaning toward the defined term, such as positive or negative evaluation, or the speaker’s emotion or attitude, including support, encouragement, hatred, derogation, or mockery. Pragmatic equivalence means that the polarity of the gold and predicted definitions is consistent. That is, if the gold definition is positive, the predicted definition should also be positive; if the gold definition is negative, the predicted definition should also be negative. If the polarity is opposite, such as positive versus negative, it should be considered inconsistent. Score all of the following five cases: 1) Gold contains pragmatic meaning + prediction contains pragmatic meaning + polarity matches = 2 points 2) Gold contains pragmatic meaning + prediction contains pragmatic meaning + polarity does not match = 0 points 3) Gold contains pragmatic meaning + prediction does not contain pragmatic meaning = 0 points 4) Gold does not contain pragmatic meaning + prediction contains pragmatic meaning = 0 points 5) Gold does not contain pragmatic meaning + prediction does not contain pragmatic meaning = 2 points – Additional output fields true/false: gold_has_pragmatic, pred_has_pragmatic, polarity_match B) Fluency: 0–3 points – Evaluate the formal appropriateness of the definition. – Score each of the following three subcriteria as either 0 or 1 point. B-1) conciseness: 0 or 1 point – Evaluate whether the predicted definition avoids unnecessary modifiers or repetitive expressions, compared with the gold definition. •1 point: concise without unnecessary modifiers or repetition | 0 points: contains unnecessary modifiers or repetitive expressions B-2) non-circularity: 0 or 1 point – Evaluate whether the predicted definition avoids circularly defining the target word using the target word itself or its synonyms. • 1 point: non-circular definition | 0 points: circular definition B-3) lexicographic convention: 0 or 1 point – Evaluate whether the predicted definition appropriately follows lexicographic conventions, such as using definitional metalanguage or a genus–differentia structure. For example, in “small but certain happiness,” “small but certain” functions as the differentia and “happiness” as the genus. • 1 point: follows lexicographic conventions | 0 points: does not follow lexicographic conventions C) Factuality: 0 or 1 point – Evaluate whether the predicted definition contains hallucinations, unsupported specific claims, or overgeneralizations. • 1 point: no major hallucination, unsupported specificity, or overgeneralization • 0 points: at least one clear and significant issue of the above types is present Important rules: 1) All scores must be integers within the specified ranges. Negative scores are not allowed. 2)The output should include only the subcriterion scores. Do not output subtotal scores such assemantic_adequacyorfluency, or the final total score. These scores will be computed post hoc. 3) If semantic_equivalence is satisfied, do not penalize differences in wording. 4) Use the example only as contextual information for interpreting the meaning. Do not use it as evidence to add new meanings beyond the gold definition. 5) Even if the predicted definition is very short, it may receive credit if it captures the core meaning. 6) Even if the predicted definition is verbose, penalize factuality if it contains unsupported content. 7) The output must be exactly one JSON object. 8) Return only JSON following the schema below, without markdown, code blocks, or explanatory text. Figure 20: Prompt template for Task 4 definition generation (Eng). General Neologism GPT - 4.1 GPT - 5.4 Solar Pro3 Solar - 10.7B EXAONE - 3.5 EXAONE - 4.0 Qwen2.5 Qwen3.5 LLaMA - 3.1 0.2 0.8 0.6 0.4 General Words vs. Neologisms (Task3 Type1) Accuracy Figure 21: Semantic categorization (Task 3-type 1) accuracy: general words vs. neologisms Not Korean Optimized Korean Optimized Model Type 32B params Model Size 16B params 8B params Unknown params ? 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 Release Date 0.3 0.5 0.7 0.6 0.4 GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B ? ? 0.25 0.45 0.65 0.55 0.35 0.05 0.35 0.55 0.45 0.25 0.15 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 2024 - 01 2024 - 04 2024 - 07 2024 - 10 2025 - 01 2025 - 04 2025 - 07 2025 - 10 2026 - 01 2026 - 04 0.3 0.7 0.9 0.2 0.45 0.55 0.5 0.35 0.45 0.65 0.1 0.7 0.5 0.3 0.2 0.5 0.7 0.6 0.4 0.3 0.4 0.35 0.3 0.25 0.25 0.55 0.7 0.4 0.5 0.55 Release DateRelease Date Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy F1 Score Task 1 (Type 1)Task 1 (Type 2)Task2 Task 3 Specialized (Type 1)Task 3 Semantic (Type 1)Task 3 Semantic (Type 2) w/o ex. Task 3 Semantic (Type 2) w/ ex.Task 3 Specialized (Type 2) w/o ex.Task 3 Specialized (Type 2) w/ ex. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B GPT-4.1 GPT-5.4 Solar Pro3 EXAONE-4.0 Qwen-3.5 EXAONE-3.5 Qwen-2.5 LLaMA-3.1 Solar-10.7B ? ? ? ? ? ? ? ? ? Figure 22: Task-wise performances by model release dates and scales Figure 23: Year-wise performance distributions for Task 1 and Task 2 Figure 24: Year-wise performance distributions for Task 3 type 1. (a) Semantic category. (b) Specialized domain Figure 25: Year-wise performance distributions for Task 3 type 2. (a) Semantic category, w/ ex. (b) Specialized domain, w/ ex. (c) Semantic category, w/o ex. (d) Specialized domain, w/o ex. Task4 w/ Example Task3 Specialized (Type1) Task3 Semantic (Type1) Task2 F1 Task1 (Type1) Task4 w/ Example Task3 Specialized (Type1) Task3 Semantic (Type1) Task2 F1 Task1 (Type1) Task4 w/ Example Task3 Specialized (Type1) Task3 Semantic (Type1) Task2 F1 Task1 (Type1) 0.6 1.0 0.8 Pearson CorrelationSpearman Correlation Figure 26: Correlation heatmap of task-wise model performance vectors Gemini Score Count 123456789100 0 1 2 3 4 5 6 7 8 9 10 푟 ! =0.50 푝<10 "#$ 푛=250 푛=1 푛=5 푛=10 푛=50 Human Score Human vs. Gemini Scores 2 0 1 Average Scor e 1 0 0.5 1 0 0.5 Total 10 0 5 Human Gemini Human Gemini Semantic Equiv. CoveragePragmatic Equiv. ConcisenessNon- circularity Lex. Convention Factuality SemanticAdequacyLexicographic Fluency Factuality (a)(b)(c)(d)(e) Figure 27: Task4. Comparison of human and gemini judge score Model / GroupTask 1Task 2Task 3 (Semantic)Task 3 (Special) GPT-4.146.00%47.05%68.00%72.00% GPT-5.460.00%51.01%70.00%82.00% Solar Pro3-12B32.00%35.90%44.00%48.00% Solar-10.7B28.00%6.80%36.00%46.00% EXAONE-3.5-7.8B48.00%33.72%38.00%40.00% EXAONE 4.0-32B48.00%24.27%50.00%50.00% Qwen2.5-7B34.00%15.20%30.00%44.00% Qwen3.5-9B46.00%19.84%44.00%58.00% LLaMA 3.1-8B30.00%13.98%32.00%44.00% Human average (n = 20)68.80%56.80%64.70%75.40% Linguistics majors (n = 10)75.00%63.40%68.40%76.20% Non-majors (n = 10)62.60%50.19%61.00%74.60% Table 10: Human and model performance on matched 50-item subsets. w/o example w/ example Evaluation Score of Generation 6 3 0 6 3 0 1 1 Score Score GPT - 4.1 GPT - 5.4 Solar Pro3 Solar - 10.7B EXAONE - 3.5 EXAONE - 4.0 Qwen2.5Qwen3.5 LLaMA - 3.1 Semantic Adequacy Lexicographic FluencyFactuality Maximum Score (a) (b) Figure 28: Average scores of generated definition.

Instant research discovery