Paper deep dive

Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages

Michele Ciletti

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 53

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/5/2026, 3:52:27 AM

Summary

The paper introduces Inspicio, an open-vocabulary, LLM-based retrieval pipeline for Word Sense Disambiguation (WSD) in historical and low-resource languages. Unlike traditional methods, Inspicio does not require a source-language sense inventory or word-to-sense mapping. Instead, it uses instruction-tuned Large Language Models to generate English translations, candidate definitions, and lemmas, which are then used to retrieve relevant synsets from the Open English WordNet (OEWN) via a hybrid dense and sparse retrieval mechanism. The system is evaluated on manually annotated datasets of Latin, Ancient Greek, and Italian, achieving high recall rates.

Entities (11)

Ancient Greek · language · 99%Italian · language · 99%Latin · language · 99%Inspicio · system · 98%Word Sense Disambiguation · task · 97%Open English WordNet · resource · 95%PREMOVE · dataset · 92%DeepSeek V3.2 · model · 90%Maximal Marginal Relevance · algorithm · 90%Qwen 3.5 · model · 90%ChromaDB · software · 85%

Relation Signals (10)

Inspicio → supportslanguage → Latin

confidence 95% · manually annotated Latin and Ancient Greek perception verbs

Inspicio → supportslanguage → Ancient Greek

confidence 95% · manually annotated Latin and Ancient Greek perception verbs

Inspicio → targets → Word Sense Disambiguation

confidence 95% · Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages

Inspicio → uses → Open English WordNet

confidence 95% · Inspicio... links tokens in context to synsets of the Open English WordNet

Inspicio → evaluatedon → PREMOVE

confidence 90% · evaluate the pipeline... on a subset of PREMOVE dataset

Inspicio → supportslanguage → Italian

confidence 90% · diachronic sample of Italian

Inspicio → usesalgorithm → Maximal Marginal Relevance

confidence 90% · Maximal Marginal Relevance re-ranking

DeepSeek V3.2 → usedin → Inspicio

confidence 85% · We pair six instruction-tuned LLMs... DeepSeek V3.2

Qwen 3.5 → usedin →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Word Sense Disambiguation has advanced rapidly for English and a handful of well-resourced modern languages, but it continues to assume the existence of a sense inventory and a word-to-sense mapping in the source language (Navigli, 2026). These assumptions break down for most historical and low-resource languages, whose dedicated WordNets are either incomplete or still under construction. We present Inspicio, an open-vocabulary retrieval pipeline that links tokens in context to synsets of the Open English WordNet (McCrae et al., 2020) without requiring any source-language inventory or mapping. For each occurrence, an instruction-tuned LLM produces two English translations of the surrounding sentence, a small set of candidate dictionary-style definitions, and a few candidate English lemmas. These outputs drive a hybrid retrieval step that combines dense definition-synset similarity, sparse lemma matching, and Maximal Marginal Relevance re-ranking. We evaluate the pipeline across a 6x6 grid of LLMs and sentence-embedding models on a new bilingual set of manually annotated Latin and Ancient Greek perception verbs, on a subset of PREMOVE dataset (Farina, 2025), and on a diachronic sample of Italian. The best configuration reaches 96% Recall@50 on the perception-verb test set, with each component contributing measurable gains, and remains competitive in the out-of-domain and cross-lingual settings.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

49,666 characters extracted from source content.

Expand or collapse full text

Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages Michele Ciletti Affiliation: Department of Humanities, University of Foggia / Via Arpi 176, 71121 Foggia, Italy Email: michele.ciletti@unifg.it Abstract Word Sense Disambiguation has advanced rapidly for English and a handful of well-resourced modern languages, but it continues to assume the existence of a sense inventory and a word-to-sense mapping in the source language (Navigli, 2026). These assumptions break down for most historical and low-resource languages, whose dedicated WordNets are either incomplete or still under construction. We present Inspicio, an open-vocabulary retrieval pipeline that links tokens in context to synsets of the Open English WordNet (McCrae et al., 2020) without requiring any source-language inventory or mapping. For each occurrence, an instruction-tuned LLM produces two English translations of the surrounding sentence, a small set of candidate dictionary-style definitions, and a few candidate English lemmas. These outputs drive a hybrid retrieval step that combines dense definition–synset similarity, sparse lemma matching, and Maximal Marginal Relevance re-ranking. We evaluate the pipeline across a 6×66× 6 grid of LLMs and sentence-embedding models on a new bilingual set of manually annotated Latin and Ancient Greek perception verbs, on the PREMOVE dataset(Farina, 2025), and on a diachronic sample of Italian. The best configuration reaches 96% Recall@50 on the perception-verb test set, with each component contributing measurable gains, and remains competitive in the out-of-domain and cross-lingual settings. 1 Introduction Word Sense Disambiguation (WSD) has made considerable progress over the past decade, in particular thanks to contextualized representations and Large Language Models (Navigli, 2009; Bevilacqua et al., 2021; Bejgu et al., 2024; Navigli, 2026). Standard formulations of these tasks share a set of strong infrastructural assumptions: (i) a sense inventory is available for the language under analysis; (i) the ambiguous spans to be disambiguated are either marked or can be reliably identified; and (i) a word-to-sense mapping lists the candidate senses for each span. Under these conditions, modern systems are able to reach high accuracy on English and a considerable number of well-resourced languages. For most of the world’s languages, and for nearly all historical ones, these assumptions hold only partially. Comprehensive WordNets remain restricted to a handful of contemporary languages, and the largest cross-lingual evaluation framework for WSD currently covers only eighteen of them, all modern (Pasini et al., 2021). Building Wordnets by hand is expensive; automatic transfer approaches, typically following the expand method (Vossen, 2004; Pianta et al., 2002), often inherit coverage gaps and noise from the source. In this paper we present Inspicio, a retrieval pipeline that performs open-vocabulary sense retrieval for historical and low-resource languages without relying on a sense inventory in the source language. Given a token in context, Inspicio prompts an LLM to translate the surrounding sentence into English, to propose a small set of candidate English definitions for the target word, and to suggest a few English lemmas that could render it in that context. These outputs are then used to query an embedding index built over the Open English WordNet (OEWN, McCrae et al., 2020). Retrieval is hybrid: dense similarity between LLM-generated definitions and synset documents is combined with sparse, lemma-based lookups, and the merged pool is diversified through Maximal Marginal Relevance to counteract the well-known fine granularity of WordNet senses. The choice of English as a pivot derives from the fact that the OEWN is the most extensively populated lexical-semantic network of its kind (McCrae et al., 2020), and it is the same pivot adopted by several bootstrapping projects that aim to create new Wordnets, including both the Latin and Ancient Greek WordNets (Minozzi, 2017; Bizzoni et al., 2014). The pipeline is otherwise language-agnostic and runs in a fully zero-shot regime, requiring no sense-annotated data in the source language and therefore being equally applicable to other historical or under-resourced languages. We evaluate Inspicio on a curated set of 150 perception-verb tokens in Latin and Ancient Greek, manually annotated with gold OEWN synsets, and additionally on PREMOVE Farina (2025), a diachronic dataset of preverbed motion verbs spanning eight centuries of Latin and Greek literature. Finally, we test it against a small diachronic dataset of old and modern Italian to test its performance in an alternative setting. Our experiments are organized around one main and three secondary research questions: RQ1. Can an LLM-driven, gloss-guided retrieval pipeline produce a high-recall candidate pool of OEWN synsets for historical language tokens without any source-language sense inventory or word-to-sense mapping? RQ2. How do the choice of generative LLM and the choice of sentence-embedding model interact in determining retrieval quality? RQ3. What is the relative contribution of dense (definition-based) and sparse (lemma-based) retrieval, and does their combination recover senses that either component would miss in isolation? RQ4. Does a pipeline tuned on a semantically focused case study generalize to a more heterogeneous, naturalistically distributed dataset and to new languages? The code, prompts, and evaluation data described in this paper are released on GitHub.11 1 https://anonymous.4open.science/r/inspicio 2 Background and Related Work 2.1 LLM-based Word-Sense Disambiguation With the rise of large instruction-tuned LLMs, many researchers have asked whether these models, pretrained on broad text corpora, retain a fine-grained understanding of word senses. Several studies have investigated this question in zero- and few-shot regimes (Kocoń et al., 2023; Yae et al., 2025; Kibria et al., 2024; Capone et al., 2024). A few consistent patterns are clear: model scale correlates strongly with disambiguation accuracy; prompt design has a non-negligible effect; and even the strongest systems are imperfect on rare or contextually subtle senses. Meconi et al. (2025) evaluated a range of open- and closed-weight LLMs and found that top systems such as GPT-4o approach the performance of specialized supervised models, while still trailing expert human annotators. Basile et al. (2025) reported analogous findings on a multilingual definition-selection benchmark derived from XL-WSD, also showing that a fine-tuned medium-sized LLM can surpass much larger zero-shot ones. Targeted prompting strategies that inject glosses and elicit chain-of-thought reasoning yield further gains (Sumanathilaka et al., 2025). 2.2 Word-Sense Disambiguation for Historical Languages WSD for historical languages has long been constrained by sparse annotated data, dictionary inventories of fine granularity, and limited cross-lingual transferability. The first systematic study for Latin is the experiment of Bamman and Burns (2020), who fine-tuned Latin BERT on a binary sense classification task derived from the Lewis and Short dictionary, restricted to the two most frequent senses of each headword. Lendvai and Wick (2022) extended this line of work with a larger sample of senses drawn from the Thesaurus Linguae Latinae. The manually annotated Latin portion of the SemEval-2020 Lexical Semantic Change dataset (Schlechtweg et al., 2020; McGillivray et al., 2022) opened the way to more elaborate strategies. Ghinassi et al. (2024) propagated WSD annotations from English to Latin through a parallel corpus, showing that automatically obtained labels can usefully complement gold data, especially on under-represented senses. Ghizzota et al. (2025) compared zero-shot and fine-tuned generative LLMs on the same resource and confirmed that medium-sized fine-tuned models can match or exceed much larger zero-shot ones. In a complementary direction, Ghizzota et al. (2026) integrated the dataset into a Linguistic Knowledge Graph and tested Graph Retrieval-Augmented Generation for sense prediction, finding that the benefit of structured authorial and lexical metadata depends substantially on model size. Dedicated studies for Ancient Greek are still few. Recent work has explored corpus-based and transformer-based methods (Mercelis et al., 2025) and proposed semi-automatic pipelines for populating the Ancient Greek WordNet through LLM-assisted annotation (Marchesi et al., 2025). Cross-linguistic experiments on Latin and Greek have addressed specific lexical classes, including preverbed motion verbs (Farina and Ciletti, 2025) and geographical common nouns (Farina et al., 2026), with the latter explicitly anchoring Latin and Greek tokens to English WordNet synsets through LLM-driven annotation. 2.3 Open-Vocabulary Word-Sense Disambiguation A separate strand of research has questioned the infrastructural assumptions of classical WSD. Bejgu et al. (2024) formalised Word Sense Linking as a task in which the spans to be disambiguated are identified directly in raw text and linked to a reference inventory, without relying on a pre-existing word-to-sense mapping. Their retriever-reader architecture shows that it is feasible for English, but its performance degrades sharply when the source-language mapping is incomplete—a situation that is the norm rather than the exception for historical languages. A few generative approaches have pushed this idea further by abandoning closed inventories altogether. Bevilacqua et al. (2020) framed WSD as a gloss generation task. Meconi et al. (2025) subsequently reported that LLMs explain word meanings substantially more accurately when allowed to produce free-form definitions than when forced to select from a fixed candidate list, reaching up to 98% accuracy in their most permissive setting. Inspicio’s pipeline fits within this landscape by assuming neither a source-language inventory nor a word-to-sense mapping; it relies on an LLM to produce English-language hypotheses about meaning, and on a sentence embedding model to align them with the Open English WordNet. The final output is therefore expressed in terms of a stable and interpretable lexical-semantic resource, while leaving the source side entirely inventory-free. 3 Methodology Inspicio processes one token at a time. Given a row consisting of a target token, its dictionary lemma, the sentence in which it occurs, and a language tag, the pipeline returns a ranked list of OEWN synsets, together with the intermediate artifacts that produced the ranking. Figure 1 summarises the architecture, which we describe in the following subsections. Hyperparameter values are reported as we introduce them; they were selected through iterative inspection of outputs (Section 4). Figure 1: Architecture of the Inspicio pipeline, showing the transition from zero-shot translation to dense and sparse OEWN synset retrieval. 3.1 Translation and Hypothesis Generation The first two stages query a generative LLM in a fully zero-shot setting. The first call asks the model to produce two English translations of the input sentence: a literal one, which stays close to source-language word order and lexical choices, and a natural one, which is fluent in modern English. We use a relatively high temperature (T=1.0T=1.0, except for when the documentation of specific models explicitly requires a different value) to encourage variation between the two outputs. This step creates a bridge to English without committing to a single translation strategy: literal renderings can preserve etymological cues that help recover compositional meanings, whereas natural ones can often surface idiomatic readings that could otherwise be lost. The second call provides the LLM with the original sentence, the target token, its lemma, and the two translations, and asks for two outputs: between one and three English dictionary-style definitions of the target word in context, ordered by likelihood, and between one and five candidate English lemmas or short multiword expressions (e.g. make up, look at) that could render the token in that occurrence. The prompt explicitly addresses well-known pitfalls of LLM-based sense generation in light of recent findings (Meconi et al., 2025): it instructs the model to define the verb itself rather than the verb-plus-negation, to include both literal and metaphorical readings when ambiguity is real, and to keep verb semantics distinct from argument semantics. We lower the temperature (T=0.8T=0.8) so that the lexicographic output remains stable while still allowing the model to surface multiple plausible senses. Definitions and lemmas are returned as JSON. 3.2 Embedding and Dense Retrieval We build four separate OEWN indexes (McCrae et al., 2020), one for each part-of-speech covered by the OEWN itself (nouns, verbs, adjectives, adverbs), under the assumption that the target token has already been PoS-tagged. Our experiments take into consideration the verb partition. Each synset is represented as a short document concatenating its lemmas, gloss, examples, hypernyms, and lexname; this representation outperformed gloss-only and lemma-only variants in preliminary tests. The documents are encoded with a sentence-embedding model (Section 5 lists the models compared) and stored in a ChromaDB collection. All embeddings are L2L_2-normalised, so dot products coincide with cosine similarities. For each LLM-generated definition d, we embed it with the same encoder used for the synsets and retrieve its top N=50N=50 nearest synsets from the index. The value of N was chosen large enough to reliably include the gold synset on a development subsample, while remaining adjustable for downstream merging. 3.3 Definition-Rank Decay Scoring A given synset may appear in the top-N of more than one definition. We aggregate its dense contributions through a weighted sum that respects the order returned by the LLM: Sbase(s)=∑d=1Dwd⋅cos⁡(d,s),S_base(s)\;=\; _d=1^Dw_d· \! (e_d,e_s ), (1) where D is the number of definitions returned (between one and three), de_d and se_s are the embeddings of definition d and synset s, and wdw_d is a rank-decay weight. We set =(1.0, 0.75, 0.5)w=(1.0,\,0.75,\,0.5), reflecting the prompt instruction that the first definition encodes the model’s most likely reading. The similarity is taken only over those definitions for which s entered the top-N pool; reusing the retrieval list, instead of re-encoding the full pool against every definition, gave comparable rankings at a fraction of the cost. 3.4 Sparse Lemma-Based Retrieval In parallel with dense retrieval, the candidate English lemmas produced in the second LLM call are used to query a precomputed inverted index that maps each English lemma to the set of synsets containing it. This index is built once from the ChromaDB metadata and cached on disk. The lemma pool augments the dense one with synsets that may be semantically distant in embedding space, but lexically anchored through translation. Such synsets are common when the source-language token has an idiomatic English counterpart that no single definition captures fully. 3.5 Merging and Final Scoring The dense and sparse pools are merged into a single candidate set. Each synset s receives a final score Sfinal(s)=Sbase(s)⋅(1+γ⋅[s∈ℒ])if Sbase(s)>0,βif Sbase(s)=0 and s∈ℒ,S_final(s)\;=\; casesS_base(s)· (1+γ·1[s ] )\\ S_base(s)>0,\\[6.0pt] β\\ S_base(s)=0 and s , cases (2) where ℒL denotes the lemma-matched set, γ=0.8γ=0.8 is a boost applied to synsets that are both densely and lexically supported, and β=0.65β=0.65 is a fixed fallback score for synsets retrieved only through the lemma index. The fallback prevents lemma-only candidates from being silently discarded, while keeping them below the dense-and-lemma combination in the ranking. The values of γ and β were tuned on the development sample. The merged pool is truncated to the top 500500 synsets by SfinalS_final before the next stage. 3.6 Diversification with Maximal Marginal Relevance OEWN exhibits well-known granularity issues, with closely related senses often distinguished by subtle nuances (Bejgu et al., 2024; Navigli, 2026). A relevance-only ranking therefore tends to fill the top-K with near-duplicate senses of the dominant reading. We re-rank the merged pool with Maximal Marginal Relevance (Carbonell and Goldstein, 1998): MMR⁡(s)=λ⋅Sfinal(s)−(1−λ)⋅maxs′∈cos(s,s′), splitMMR(s)\;=\;&λ·S_final(s)\\ &-\;(1-λ)· _s \! (e_s,e_s ), split (3) where S is the set of already-selected synsets. We set λ=0.8λ=0.8, a relevance-leaning value that introduces enough diversity to break clusters of near-identical senses while preserving the top of the ranking. The final list is truncated to K=50K=50. 3.7 Output and Auditability For every input token the pipeline writes a JSON record containing the two translations, the generated definitions and candidate lemmas, the top-K synsets with their final and base scores, the lemma-match flag, and the per-definition contributions. Every prediction is therefore inspectable, which supports downstream lexicographic reuse: a curator who disagrees with the top-1 can trace which definition or lemma drove the result and decide whether to revise the prompt, the gloss, or even the gold annotation. 4 Data 4.1 A Bilingual Perception-Verb Evaluation Set The scarcity of sense-annotated resources for Latin and Ancient Greek is a recurring obstacle in this line of work. For Latin, one of the only manually annotated datasets of any size that is mapped to a WordNet inventory is the Latin portion of the SemEval-2020 Lexical Semantic Change benchmark (Schlechtweg et al., 2020; McGillivray et al., 2022), whose mapping to LWN was added a posteriori. The PREMOVE dataset (Farina, 2025) contributes around 2,800 manually annotated tokens of Latin and Ancient Greek preverbed motion verbs, with senses encoded as OEWN synsets. No other comparably curated resources exist. To support a focused evaluation of Inspicio we built a new bilingual dataset of 150 perception-verb tokens, with 72 Latin and 78 Ancient Greek instances. The tokens were sampled from the PREMOVE Base Corpus (Farina, 2025), which is balanced across periods, genres, and authors for both languages. This balancing carries over to our sample, which therefore spans Archaic to Late Greek and Early to Post-Classical Latin. We targeted perception verbs for three reasons. First, verbs are a particularly polysemous part of speech, which makes them consistently challenging to disambiguate (Haber and Poesio, 2024). Second, perception verbs combine concrete sensory meanings with a rich layer of metaphorical and mental-state extensions: Lat. videō and AGr. horáō occur both in literal ‘see’ readings and in inferential ‘understand, realise’ ones, which makes them a natural stress test for a pipeline that must separate literal from figurative senses. Third, perception is a cognitively universal domain whose lexicalisations are well represented in the OEWN, whereas culturally specific items, often nouns (e.g. the Lat. consul), have no fitting synset and would force annotators to invent one or to refuse annotation. Lexical anchors were drawn from Levin (1993)’s class 30.1, excluding olfactory and gustatory verbs because of corpus sparsity: Lat. audiō / AGr. akoúō, Lat. videō / AGr. horáō, Lat. prefixed compounds in -spiciō / AGr. blépō, and Lat. sentiō / AGr. aisthánomai. Annotation. Each occurrence was independently annotated by two annotators, both trained linguists with prior experience in semantic annotation. For every token, the annotators selected the most appropriate OEWN synset given the original context, with access to the full sentence and, when useful, to the surrounding passage. Disagreements were resolved after a round of review to produce a single gold synset per occurrence. Inter-annotator agreement was computed on the two-annotator labels using Cohen’s κ (Cohen, 1960; Artstein and Poesio, 2008) and reached κ=0.895κ=0.895, in the upper range of values typically reported for fine-grained WordNet annotation (Passonneau et al., 2010). 4.2 Out-of-Domain Evaluation: PREMOVE To test whether the pipeline generalises beyond a curated, semantically narrow sample, we additionally evaluate on PREMOVE (Farina, 2025), a manually annotated, diachronic dataset of Latin and Ancient Greek preverbed motion verbs spanning the 8th century BCE to the 2nd century CE. Its sense annotations are encoded directly as OEWN synsets, which makes the resource immediately compatible with our setup. PREMOVE is narrow in scope, covering a single verb domain, but its tokens are naturalistically distributed: their frequency profile is Zipfian, metaphorical readings such as evenio ‘happen’ or invenio ‘find’ are well represented, and the genre and authorial range is as wide as that of our curated sample. It is therefore an informative stress test of a system tuned on a semantically focused case study. 4.3 Cross-Lingual Test: Diachronic Italian A final evaluation extends Inspicio to a new language and a different temporal range. We collected 100 Italian motion-verb tokens from the diachronic MIDIA corpus (Iacobini and D’Achille, 2022), sampled so as to match, where possible, the Latin lemmas that appear in PREMOVE (e.g. Lat. incurro → It. incorrere). The two sets are therefore comparable at their lexical entry points, but differ sharply in semantic transparency: Italian descendants of Latin preverbed motion verbs have undergone substantial lexicalisation and figurative drift, so a much smaller proportion of their occurrences retains a literal motion reading. The temporal coverage of the Italian sample, from late-medieval to contemporary, puts additional pressure on the LLM’s ability to handle diachronic variation in a language for which pretraining data is unevenly distributed across periods. Annotation followed the same protocol as for Latin and Ancient Greek, with two annotators selecting OEWN synsets in context. Cohen’s κ on the two-annotator labels reached 0.9140.914. 5 Experiments and Results We evaluate Inspicio on the three datasets introduced in Section 4. The main experiment tests the pipeline on the perception-verb test set across different models, and is designed to address RQ1 and RQ2 jointly. The PREMOVE and Italian evaluations target RQ4, and a final ablation study addresses RQ3. Models and Experimental Setup. We pair six instruction-tuned LLMs with six sentence-embedding models. The LLMs were chosen to cover the current state of the art among recent open-weights, high-parameter systems that support chain-of-thought reasoning, an ability that recent work has identified as a strong predictor of disambiguation accuracy (Farina, 2025): DeepSeek V3.222 2 https://huggingface.co/deepseek-ai/DeepSeek-V3.2, DeepSeek V4 Pro33 3 https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro, Kimi K2.644 4 https://huggingface.co/moonshotai/Kimi-K2.6, GLM 5.155 5 https://huggingface.co/zai-org/GLM-5.1, Qwen 3.5 397B A17B66 6 https://huggingface.co/Qwen/Qwen3.5-397B-A17B, and Mistral Medium 3.577 7 https://huggingface.co/mistralai/Mistral-Medium-3.5-128B. All models are queried through their official API providers with the default decoding parameters listed in their documentation; the prompts are identical across systems and are reported in Appendix A. The embedding models cover both closed and open systems, and were selected to span different training regimes and parameter counts: text-embedding-3-large88 8 https://developers.openai.com/api/docs/models/text-embedding-3-large, KaLM-Embedding-Gemma3-12B-251199 9 https://huggingface.co/tencent/KaLM-Embedding-Gemma3-12B-2511, Qwen3-Embedding-8B1010 10 https://huggingface.co/Qwen/Qwen3-Embedding-8B, Cohere Embed v41111 11 https://cohere.com/blog/embed-4, Harrier-OSS-27B1212 12 https://huggingface.co/microsoft/harrier-oss-v1-27b, and jina-embeddings-v5-text-small1313 13 https://huggingface.co/jinaai/jina-embeddings-v5-text-small. For the embedding models that supported it, we included a short query prompt (see Appendix A). All other pipeline hyperparameters (Section 3) are held fixed across runs. We report Recall@k at k∈1,10,20,50k∈\1,10,20,50\, with k=50k=50 as the headline metric. Perception Verbs. Table 1 reports Recall@50 for every ⟨ , embedding⟩ pair, and Figure 2 shows the corresponding heatmaps at k∈1,10,20,50k∈\1,10,20,50\. The best combination is DeepSeek V4 Pro with KaLM-Embedding-Gemma3-12B, which reaches 96% Recall@50. KaLM is the strongest embedding for five of the six LLMs, while DeepSeek V4 Pro is the strongest LLM in four of the six embedding columns. PREMOVE Case Study. We then apply the best combination to PREMOVE in order to test whether performance carries over to a more heterogeneous, naturalistically distributed sample. Recall@50 reaches 81.65% (Table 2), roughly 15 points below the perception-verb result. The gap is consistent with the broader semantic spread of motion verbs, the Zipfian sense distribution of the dataset, and the diachronic and metaphorical drift that affects preverbed forms (Farina and Ciletti, 2025). Diachronic Italian. The same combination is finally evaluated on the diachronic Italian set (Table 2). Recall@50 reaches 91%, higher than the PREMOVE result. The outcome is informative given the linguistic distance and diachronic shift involved: most descendants of Latin preverbed motion verbs have undergone substantial lexicalisation, and many tokens occur in figurative or grammaticalised readings that no longer involve literal motion; furthermore, the diachronic stratification of the data means that the tested tokens present widely different meanings across their occurrences in the dataset. Thus, the English-pivot strategy remains effective when the source language is itself well represented in the LLM’s pretraining. Ablation Tests. We isolate the contribution of individual pipeline components through four ablation tests, run with the best combination on the perception-verb test set (Table 2). Removing the translation stage produces the sharpest drop, to 92%, indicating that the literal/natural translation pair contributes information that the gloss generator alone does not recover. However, such a performance gap may be acceptable in certain cases considering that only a single LLM call (where the model directly identifies candidate definitions and lemmas) is needed, halving the inference cost. Removing the lemma boost causes another drop, to 94%: dense and sparse retrieval recover synsets that neither component reaches in isolation, which answers positively to RQ3. Disabling the definition-rank decay (uniform weights across the three definitions) doesn’t seem to decrease performance, as well as disabling MMR. These two parameters are the most subject to model selection and data distribution: MMR can help surface rare senses, which may not always be needed, and definition-rank decay largely depends on the tendency of the LLM to generate relatively similar or markedly different definitions. We retain MMR in the default configuration, since it noticeably diversifies the top of the ranking without harming aggregate recall, and a diverse top-K is deemed preferable for downstream lexicographic inspection. Embedding model LLM OpenAI 3-large KaLM 12B Qwen3 8B Cohere v4 Harrier 27B Jina v5 DeepSeek V3.2 91.33 95.33 90.00 88.00 89.33 89.33 DeepSeek V4 Pro 94.00 96.00 91.33 90.00 88.67 90.67 Kimi K2.6 90.67 91.33 85.33 83.33 85.33 87.33 GLM 5.1 92.67 93.33 86.00 85.33 89.33 88.67 Qwen 3.5 90.00 93.33 88.67 82.00 90.00 88.67 Mistral Medium 3.5 84.67 90.00 83.33 78.00 86.67 83.33 Table 1: Recall@50 (%) on the 150-token perception-verb test set, for every combination of LLM (rows) and sentence-embedding model (columns). The best cell is highlighted. Figure 2: Recall@k heatmaps on the perception-verb test set, for k∈1,10,20,50k∈\1,10,20,50\. Rows correspond to LLMs and columns to embedding models. Cell shading maps to relative performance within each individual metric, darker colours indicate higher recall. Configuration Recall@50 Recall@20 Case Studies PREMOVE 81.65 74.91 Diachronic Italian 91.00 84.00 Ablation Tests Full pipeline 96.00 90.67 −- translation stage 92.00 83.33 −- definition-rank decay 96.00 90.00 −- lemma boost 94.00 85.33 −- MMR re-ranking 96.67 90.67 Table 2: Recall@50 (%) of the best combination (DeepSeek V4 Pro ++ KaLM-Embedding-Gemma3-12B-2511) on the out-of-domain (PREMOVE) and cross-lingual (Italian) test sets, and ablations on the perception-verb set. 6 Discussion 6.1 Qualitative Error Analysis A close reading of the cases in which the gold synset falls below the top-K shows a few recurring patterns. The pipeline rarely fails because the LLM misunderstands the sentence; more often, it produces a semantically appropriate definition that is nonetheless slightly different from the gold synset in OEWN’s fine-grained sense space. A representative example is Lat. provideō in a context where the gold synset is oewn-01185006-v, glossed as “give what is desired or needed, especially support, food or sustenance”. The model returns the definition “make preparations to meet future needs by supplying necessary items”, which is paraphrastically close to the gold gloss but does not surface any of the synset’s lemmas (cater, ply, provide, supply). The candidate lemmas it does propose (provide for, make provision for, take care of, see to, arrange for) all match adjacent synsets, which then occupy higher positions in the ranking. This kind of failure is largely a granularity artefact of the type already noted by Bejgu et al. (2024) and Navigli (2026), and confirms that the lemma signal is complementary to the dense one rather than redundant: when both align, the gold synset surfaces more easily; when only the definition matches, the ranking is dominated by lexically similar neighbours. Two systematic biases emerged during prompt development and were addressed before the final evaluation. The first concerns a failure in interpreting co-composition Pustejovsky (2012). In preliminary runs, the model tended to incorporate the semantics of the verb’s arguments into the definition of the verb itself: Lat. iterō in iterō coniugium (literally to repeat a marriage) was glossed as “to remarry”, although iterō simply means “to do again” and the marital reading arises entirely from coniugium. The second concerns negation, which the model occasionally treated as part of the verb’s lexical content, returning the antonymic sense for negated occurrences. Explicit prompt instructions to keep verb semantics separate from argument semantics, and to define the verb independently of any negation in its context, removed both biases on our development sample. The hybrid scoring scheme makes the pipeline more tolerant to residual noise of this kind: when the definition is mildly off, lemma overlap can still pull the correct synset into the top-K, and vice versa. 6.2 Future Perspectives A natural next step is a second-stage reranker: an LLM-based judge that, given the top-K candidates and the original context, selects one or two definitive synsets. Comparable architectures have proven effective for historical languages (Ghizzota et al., 2025; Farina, 2025; Farina et al., 2026), and the high-recall pool produced by Inspicio is well suited to a reranking stage of this type. The combination would close the gap between candidate retrieval and final disambiguation, while preserving the open-vocabulary character of the upstream pipeline. Once such a reranker is in place, the pipeline can act as a generator of silver-standard sense annotations for languages that currently lack any. Annotation propagation through automatic means has already been shown to complement gold data productively for historical languages (Ghinassi et al., 2024); an open-vocabulary system would remove the requirement of an existing sense inventory in the source language. The resulting silver datasets, together with the intermediate artefacts that Inspicio already logs, could in turn feed the bootstrapping of dedicated WordNets for these languages, along the lines of existing efforts behind the Latin and Ancient Greek WordNets (Marchesi et al., 2025; Santoro et al., 2025). The pipeline would thus contribute to building the very resources whose absence motivates it in the first place. It is important to stress that the pipeline is designed to work in a zero-shot setting, addressing real-world conditions where sense-annotated data may not be available for historical languages. However, supervised fine-tuning strategies may be explored where possible to further improve performance and linguistic understanding. Again, recent advances in agentic LLMs make it an interesting possibility to test the retrieval part of our pipeline in an unconstrained, autonomous setting. Limitations Our evaluation focuses on verbs, which are a particularly polysemous part of speech (Haber and Poesio, 2024). The architecture is agnostic to part of speech, and four OEWN indexes are already in place, but additional evaluations on nouns, adjectives, and adverbs would strengthen the generality of our findings. The pipeline relies on LLM predictions at two consecutive stages, and is therefore stochastic. A misinterpretation produced at translation time can propagate to gloss generation and, from there, to retrieval. The translation stage in particular requires a relatively high sampling temperature, since the literal/natural pair is meant to expose lexical and stylistic variation. This design choice introduces a degree of run-to-run variability that more deterministic pipelines do not face. The auditability of the intermediate outputs partly mitigates this issue, in that any failure can be traced back to the specific stage at which it originated, but does not eliminate it. Finally, routing through English is a deliberate compromise. The OEWN is the most extensively populated lexical-semantic resource of its kind (McCrae et al., 2020), and serves as the pivot of several bootstrapping projects for historical and low-resource languages (Minozzi, 2017; Bizzoni et al., 2014), but its inventory still reflects the lexicalisation patterns of contemporary English. Some source-language senses have no perfect English counterpart, and our pipeline can at best return the closest available synset in such cases. As discussed above, the natural mid-term solution is the incremental construction of dedicated sense inventories for the languages of interest, a process that Inspicio itself can help support. Acknowledgments References Artstein and Poesio (2008) R. Artstein and M. Poesio Survey article: inter-coder agreement for computational linguistics. Computational linguistics 34 (4), p. 555–596. Cited by: §4.1. Bamman and Burns (2020) D. Bamman and P. J. Burns Latin bert: a contextual language model for classical philology. External Links: 2009.10053, Link Cited by: §2.2. Basile et al. (2025) P. Basile, L. Siciliani, E. Musacchio, and G. Semeraro Exploring the word sense disambiguation capabilities of large language models. arXiv preprint arXiv:2503.08662. Cited by: §2.1. Bejgu et al. (2024) A. S. Bejgu, E. Barba, L. Procopio, A. Fernández-Castro, and R. Navigli Word sense linking: disambiguating outside the sandbox. In Findings of the Association for Computational Linguistics: ACL 2024, p. 14332–14347. Cited by: §1, §2.3, §3.6, §6.1. Bevilacqua et al. (2020) M. Bevilacqua, M. Maru, and R. Navigli Generationary or “how we went beyond word sense inventories and learned to gloss”. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, p. 7207–7221. External Links: Link, Document Cited by: §2.3. Bevilacqua et al. (2021) M. Bevilacqua, T. Pasini, A. Raganato, and R. Navigli Recent trends in word sense disambiguation: a survey. In IJCAI, p. 4330–4338. Cited by: §1. Bizzoni et al. (2014) Y. Bizzoni, F. Boschetti, H. Diakoff, R. Del Gratta, M. Monachini, and G. R. Crane The making of ancient greek wordnet.. In LREC, Vol. 2014, p. 1140–1147. Cited by: §1, Limitations. Capone et al. (2024) L. Capone, S. Auriemma, M. Miliani, A. Bondielli, and A. Lenci Lost in disambiguation: how instruction-tuned LLMs master lexical ambiguity. In Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), F. Dell’Orletta, A. Lenci, S. Montemagni, and R. Sprugnoli (Eds.), Pisa, Italy, p. 148–156. External Links: Link, ISBN 979-12-210-7060-6 Cited by: §2.1. Carbonell and Goldstein (1998) J. Carbonell and J. Goldstein The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’98, New York, NY, USA, p. 335–336. External Links: ISBN 1581130155, Link, Document Cited by: §3.6. Cohen (1960) J. Cohen A coefficient of agreement for nominal scales. Educational and psychological measurement 20 (1), p. 37–46. Cited by: §4.1. Farina et al. (2026) A. Farina, M. Ciletti, B. McGillivray, and A. Ballatore Sense-based annotation of geographical nouns in ancient greek and latin: a diachronic study with llms. In Proceedings of the 10th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature 2026, p. 266–279. Cited by: §2.2, §6.2. Farina and Ciletti (2025) A. Farina and M. Ciletti Probing preverbs: evaluating large language models on latin and ancient greek preverbed motion verbs. Proceedings of Historical Languages and AI. March 5–6, 2026, Humboldt University Berlin (Germany). Cited by: §2.2, §5. Farina (2025) A. Farina PREMOVE–a diachronic dataset of ancient greek and latin annotated preverbed motion verbs. Cited by: §1, §4.1, §4.1, §4.2, §5, §6.2, Abstract. Ghinassi et al. (2024) I. Ghinassi, S. Tedeschi, P. Marongiu, R. Navigli, and B. McGillivray Language pivoting from parallel corpora for word sense disambiguation of historical languages: a case study on Latin. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), N. Calzolari, M. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue (Eds.), Torino, Italia, p. 10073–10084. External Links: Link Cited by: §2.2, §6.2. Ghizzota et al. (2025) E. Ghizzota, P. Basile, L. Siciliani, and G. Semeraro The meaning of beatus: disambiguating latin with contemporary ai models. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), p. 469–479. Cited by: §2.2, §6.2. Ghizzota et al. (2026) E. Ghizzota, P. Marongiu, P. Basile, S. Ferilli, and B. McGillivray Linguistic knowledge graphs for sense prediction: a case-study on latin. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), S. Piperidis, N. Bel, H. van den Heuvel, N. Ide, S. Krek, and A. Toral (Eds.), Palma, Mallorca, Spain, p. 10937–10952. External Links: Document Cited by: §2.2. Haber and Poesio (2024) J. Haber and M. Poesio Polysemy—evidence from linguistics, behavioral science, and contextualized language models. Computational Linguistics 50 (1), p. 351–417. External Links: ISSN 0891-2017, Document, Link, https://direct.mit.edu/coli/article-pdf/50/1/351/2367142/coli_a_00500.pdf Cited by: §4.1, Limitations. Iacobini and D’Achille (2022) C. Iacobini and P. D’Achille Il corpus midia: concezione, realizzazione, impieghi. In Corpora e Studi Linguistici, E. Cresti and M. Moneglia (Eds.), Milano, p. 207–221. External Links: ISBN 978-88-97657-55-2, Link Cited by: §4.3. Kibria et al. (2024) R. Kibria, S. I. U. Dipta, and M. A. Adnan On functional competence of LLMs for linguistic disambiguation. In Proceedings of the 28th Conference on Computational Natural Language Learning, L. Barak and M. Alikhani (Eds.), Miami, FL, USA, p. 143–160. External Links: Link, Document Cited by: §2.1. Kocoń et al. (2023) J. Kocoń, I. Cichecki, O. Kaszyca, M. Kochanek, D. Szydło, J. Baran, J. Bielaniewicz, M. Gruza, A. Janz, K. Kanclerz, A. Kocoń, B. Koptyra, W. Mieleszczenko-Kowszewicz, P. Miłkowski, M. Oleksy, M. Piasecki, Ł. Radliński, K. Wojtasik, S. Woźniak, and P. Kazienko ChatGPT: jack of all trades, master of none. Information Fusion 99, p. 101861. External Links: ISSN 1566-2535, Document, Link Cited by: §2.1. Lendvai and Wick (2022) P. Lendvai and C. Wick Finetuning Latin BERT for word sense disambiguation on the thesaurus linguae latinae. In Proceedings of the Workshop on Cognitive Aspects of the Lexicon, M. Zock, E. Chersoni, Y. Hsu, and E. Santus (Eds.), Taipei, Taiwan, p. 37–41. External Links: Link, Document Cited by: §2.2. Levin (1993) B. Levin English verb classes and alternations: a preliminary investigation. University of Chicago press. Cited by: §4.1. Marchesi et al. (2025) B. Marchesi, A. Clementelli, A. M. Mammarella, S. Zampetta, E. Biagetti, L. B. Villa, V. Mastellari, R. Ginevra, C. R. Combei, and C. Zanchi Towards the semi-automated population of the ancient greek wordnet. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), p. 647–658. Cited by: §2.2, §6.2. McCrae et al. (2020) J. P. McCrae, A. Rademaker, E. Rudnicka, and F. Bond English wordnet 2020: improving and extending a wordnet for english using an open-source methodology. In proceedings of the LREC 2020 workshop on multimodal WordNets (MMW2020), p. 14–19. Cited by: §1, §3.2, Limitations, Abstract. McGillivray et al. (2022) B. McGillivray, D. Kondakova, A. Burman, F. Dell’Oro, H. B. Sabel, P. Marongiu, and M. M. Cruz A new corpus annotation framework for latin diachronic lexical semanticsA new corpus annotation framework for latin diachronic lexical semantics. Journal of Latin Linguistics 21 (1), p. 47–105. External Links: Link, Document Cited by: §2.2, §4.1. Meconi et al. (2025) D. Meconi, S. Stirpe, F. Martelli, L. Lavalle, and R. Navigli Do large language models understand word senses?. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 33897–33916. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §2.1, §2.3, §3.1. Mercelis et al. (2025) W. Mercelis, T. V. Hal, and A. Keersmaekers Tongue, language or noise? word sense disambiguation in ancient greek with corpus-based methods. In Advances in Ancient Greek Linguistics, J. de la Villa, A. Striano, and R. Verano (Eds.), p. 813–828. External Links: Link, Document, ISBN 9783111648644 Cited by: §2.2. Minozzi (2017) S. Minozzi Latin wordnet, una rete di conoscenza semantica per il latino e alcune ipotesi di utilizzo nel campo dell’information retrieval. Strumenti digitali e collaborativi per le Scienze dell’Antichita 14, p. 123–134. Cited by: §1, Limitations. Navigli (2009) R. Navigli Word sense disambiguation: a survey. ACM computing surveys (CSUR) 41 (2), p. 1–69. Cited by: §1. Navigli (2026) R. Navigli Is word sense disambiguation dead in the llm era?. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 39753–39762. Cited by: §1, §3.6, §6.1, Abstract. Pasini et al. (2021) T. Pasini, A. Raganato, and R. Navigli XL-wsd: an extra-large and cross-lingual evaluation framework for word sense disambiguation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 13648–13656. Cited by: §1. Passonneau et al. (2010) R. Passonneau, A. Salleb-Aouissi, V. Bhardwaj, and N. Ide Word sense annotation of polysemous words by multiple annotators. Proceedings of the Language Resources and Evaluation Conference. External Links: Link Cited by: §4.1. Pianta et al. (2002) E. Pianta, L. Bentivogli, and C. Girardi MultiWordNet: developing an aligned multilingual database. In First international conference on global WordNet, p. 293–302. Cited by: §1. Pustejovsky (2012) J. Pustejovsky Co-composition ality in grammar. In The Oxford Handbook of Compositionality, W. Hinzen, E. Machery, and M. Werning (Eds.), External Links: ISBN 9780199541072, Document, Link, https://academic.oup.com/book/0/chapter/350864201/chapter-ag-pdf/44421992/book_41264_section_350864201.ag.pdf Cited by: §6.1. Santoro et al. (2025) D. Santoro, B. Marchesi, S. Zampetta, M. Del Tredici, E. Biagetti, E. Litta, C. R. Combei, S. Rocchi, T. Facchinetti, and R. Ginevra Exploring latin wordnet synset annotation with llms. In Proceedings of the 13th Global Wordnet Conference, p. 66–76. Cited by: §6.2. Schlechtweg et al. (2020) D. Schlechtweg, B. McGillivray, S. Hengchen, H. Dubossarsky, and N. Tahmasebi SemEval-2020 task 1: unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, A. Herbelot, X. Zhu, A. Palmer, N. Schneider, J. May, and E. Shutova (Eds.), Barcelona (online), p. 1–23. External Links: Link, Document Cited by: §2.2, §4.1. Sumanathilaka et al. (2025) T. G. D. K. Sumanathilaka, N. Micallef, and J. Hough Can llms assist with ambiguity? a quantitative evaluation of various large language models on word sense disambiguation. External Links: 2411.18337, Link Cited by: §2.1. Vossen (2004) P. Vossen EuroWordNet: a multilingual database of autonomous and language-specific wordnets connected via inter-lingual-index. international journal of Lexicography 17 (2), p. 161–174. Cited by: §1. Yae et al. (2025) J. H. Yae, N. C. Skelly, N. C. Ranly, and P. M. LaCasse Leveraging large language models for word sense disambiguation. Neural Computing and Applications 37 (6), p. 4093–4110. Cited by: §2.1. Appendix A Prompts This appendix presents the complete prompt templates used for all models, formatted in Markdown for clarity and reproducibility. Three prompts are presented: the translation prompt, the definition and lemma generation prompt, and the embedding prompt (only used with models that supported it). A.1 Translation Prompt You are an expert translator specializing in language. # TASK: Translate the given sentence into English in TWO ways: 1. a literal translation that stays close to the source wording 2. a natural translation that sounds fluent in modern English Focus on correctly representing the literal and metaphorical meanings of specific words. # OUTPUT: Return ONLY valid JSON with exactly these keys: "literal": "string", "natural": "string" No commentary, no markdown, no extra keys. A.2 Definition and Lemma Generation Prompt You are an expert lexicographer and linguist specializing in language semantics. # TASK: You will be given: - a target token - its dictionary lemma - the original sentence it occurs in - TWO proposed English translations of the sentence - one literal, one natural. Using all of this context, produce: 1. 1 to 3 possible English dictionary-style definitions of the target word in this context, ordered by likelihood (most likely first). Each definition must be a phrase that fully captures and explains the sense. Example: "she ran all the way home" -> "move rapidly from one place to another" 2. 1 to 5 candidate English lemmas or short expressions (1–2 words, e.g. "make up", "go out") that could translate the token in this context. # GUIDELINES: - Be specific and detailed enough to distinguish senses. - Account for negation: define the verb’s meaning, not its truth value. Example: "she didn’t run" -> "move at a speed faster than a walk", not "stand still". - Account for metaphorical meanings: when in doubt, include both literal and metaphorical definitions. Example: "she saw a risk in his plan" -> "perceive a situation mentally" works better than "perceive by sight". - Beware of distinguishing the actual meaning of a verb from those of its arguments. - Examples: "He shed a few tears" -> "let fall, emit" is the right choice, not "cry", which would absorb the meaning of "tears". - However, "She threw a party" -> "organize an event" is appropriate, just like "She caught a cold" -> "get struck by an illness", as those are actual meanings conveyed by the verbs. - Avoid unnecessary contextual information: "She ate the cake gleefully" -> "take in food", not "take in food in a joyous manner". - However, "He devoured the cake" -> "eat quickly and hungrily" is correct, because the specific manner of eating is a core semantic feature inherently lexicalized in the verb itself. - If there is genuine ambiguity, include multiple definitions; otherwise output 1. - Keep outputs precise and consistent. # OUTPUT: Return ONLY valid JSON with exactly these keys: "definitions": ["def1", "def2", "def3"], "candidate_lemmas": ["lemma1", "lemma2", "lemma3", "lemma4", "lemma5"] No commentary, no markdown, no extra keys. A.3 Embedding Prompt Given a dictionary definition, retrieve the WordNet synset that best matches its meaning.

Instant research discovery