Paper deep dive
Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text
Refael Shaked Greenfeld, Reut Tsarfaty
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/27/2026, 4:11:04 AM
Summary
The paper introduces KibutzR, the first comprehensive Coreference Resolution (CR) dataset for Modern Hebrew, designed to address the challenges of Morphologically Rich Languages (MRLs). Unlike standard English-centric CR methods that assume word-mention alignment, KibutzR accounts for complex phenomena like pronominal clitics and construct-state nouns through morpheme-level annotations. The authors also propose a three-regime evaluation protocol (Raw Text, Automatic Segmentation, and Gold Segmentation) to diagnose whether model failures stem from segmentation errors or coreference clustering. Empirical results show that contemporary LLMs perform significantly worse on Hebrew than English and that smaller encoder models often outperform large decoder models in this context.
Entities (7)
Relation Signals (4)
KibutzR → addresseschallengesof → Modern Hebrew
confidence 100% · we introduce KibutzR, the first comprehensive CR dataset for Modern Hebrew
Refael Shaked Greenfeld → affiliatedwith → Bar-Ilan University
confidence 100% · Refael Shaked Greenfeld Bar-Ilan University
Reut Tsarfaty → affiliatedwith → Bar-Ilan University
confidence 100% · Reut Tsarfaty Bar-Ilan University
KibutzR → isbasedon → Hebrew Universal Dependencies Treebank
confidence 100% · The Hebrew KibutzR dataset is based on The Hebrew Universal Dependencies Treebank
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Coreference Resolution (CR) is a fundamental NLP task critical for long-form tasks as information extraction, summarization, and many business applications. However, CR methods originally designed for English struggle with Morphologically Rich Languages (MRLs), where mention boundaries do not necessarily align with word boundaries, and a single token may consist of multiple anaphors. CR modeling and evaluation protocols standardly assume that, as in English, words and mentions mostly align. However, this assumption breaks down in MRLs, particularly in the context of LLMs' raw-text processing and end-to-end tasks. To assess and address this challenge, we introduce {\em KibutzR}, the first comprehensive CR dataset for Modern Hebrew, an MRL rich with complex words and pronominal clitics. We deliver an annotated dataset that identifies mentions at word, sub-word and multi-word levels, and propose an evaluation protocol that directly addresses word/morpheme boundary discrepancies. Our experiments show that contemporary LLMs perform significantly worse on Hebrew than on English, and that performance degrades on raw unsegmented text. Crucially, we show an inverse performance-trend in Hebrew relative to English, where smaller encoders perform far better than contemporary decoder models, leaving ample space for investigation and improvement. We deliver a new benchmark for Hebrew coreference resolution and a segmentation-aware evaluation protocol to inform future work on other MRLs.
Tags
Links
- Source: https://arxiv.org/abs/2604.17108v1
- Canonical: https://arxiv.org/abs/2604.17108v1
Trouble viewing inline? Open PDF directly →
Full Text
54,317 characters extracted from source content.
Expand or collapse full text
Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text Refael Shaked Greenfeld Bar-Ilan University shakedgreenfeld@gmail.com Reut Tsarfaty Bar-Ilan University reut.tsarfaty@biu.ac.il Abstract Coreference Resolution (CR) is a funda- mental NLP task critical for long-form tasks as information extraction, summa- rization, and many business applications. However, CR methods originally designed for English struggle with Morphologically Rich Languages (MRLs), where mention boundaries do not necessarily align with word boundaries, and a single token may consist of multiple anaphors. CR model- ing and evaluation protocols standardly as- sume that, as in English, words and men- tions mostly align. However, this assump- tion breaks down in MRLs, particularly in the context of LLMs’ raw-text processing and end-to-end tasks. To assess and ad- dress this challenge, we introduceKibutzR, the first comprehensive CR dataset for Modern Hebrew, an MRL rich with com- plex words and pronominal clitics. We deliver an annotated dataset that identi- fies mentions at word, sub-word and multi- word levels, and propose an evaluation protocol that directly addresses word/mor- pheme boundary discrepancies. Our exper- iments show that contemporary LLMs per- form significantly worse on Hebrew than on English, and that performance degrades on raw unsegmented text. Crucially, we show an inverse performance-trend in He- brew relative to English, where smaller en- coders perform far better than contempo- rary decoder models, leaving ample space for investigation and improvement. We de- liver a new benchmark for Hebrew corefer- ence resolution and a segmentation-aware evaluation protocol to inform future work on other MRLs. 1 Introduction The field of Natural Language Processing has achieved remarkable breakthroughs in recent years, driven by advanced large language mod- els (LLMs) and large-scale resources. Despite these advances, complex word struc- tures continue to pose significant challenges for automatic text processing and discourse- level understanding. In particular, the task of Coreference Resolution (CR) involves identify- ing and clustering entities across a discourse, enabling deeper text comprehension. How- ever, achieving accurate CR presents unique challenges inmorphologically rich languages (MRLs), with phenomena as pronominal clitics and morpheme-stacking that obscure mention boundaries. These linguistic intricacies make mention detection and coreference resolution particularly difficult in MRLs. Modern Hebrew, a Semitic morphologically rich language, exemplifies these challenges, where a single token in Hebrew may con- sist of multiple anaphors that designate dif- ferent entities in one and the same token, via construct-state nouns and pronominal cl- itics ( More and Tsarfaty,2016). Similarly, Arabic dialects show pronominal clitic fusion (Maamouri et al.,2004), Turkish stacks mul- tiple morphemes creating boundary detection problems ( Schüller et al.,2017), and Slavic lan- guages encode multiple grammatical relations within a single word. All these languages share the same critical challenge — that standard CR models and metrics assume that mentions essentially align with word boundaries — but this alignment either doesn’t naturally apply, or requires error-prone preprocessing. As NLP shifts towards end-to-end architec- tures and LLM processing of raw texts, evalua- tion practices designed for CR in English turn out to fail on MRL texts. Concretely, the shift to generative LLMs has created the follow- ing evaluation gap: these models process raw texts directly, but we lack frameworks to as- sess performance when morphological segmen- tation and coreference errors are intertwined. arXiv:2604.17108v1 [cs.CL] 18 Apr 2026 Exacerbating this, many lower-resourced MRLs still lack even the very basic research infrastructure for CR, including public bench- marks to assess CR and track empirical progress. Without benchmarks that isolate the error sources, we cannot diagnose where and why CR models fail on morphologically- rich texts and suggest mitigation. This work focuses on Hebrew, an MRL rich with pronominal clitics that break word-to- mention alignment. While significant progress has been made in Hebrew via pre-trained en- coders such as AlephBERT ( Seker et al.,2022) and DictaBERT (Shmidman et al.,2023), as well as Large Language Models (LLMs) like DictaLM (Shmidman et al.,2024), and while the Hebrew NLP community contributed valu- able resources for shorter-text tasks such as sentiment analysis, named entity recognition (NER) (Bareket and Tsarfaty,2021), and ques- tion answering (QA) ( Cohen et al.,2025), there remains a notable lack of resources for discourse-levelunderstanding, and in particu- larcoreference resolution(CR), limiting the development of more complex applications and long-form Hebrew tasks. In this work we address the multifaceted CR challenge in Hebrew by developing the first comprehensive Modern Hebrew CR dataset, KibutzR, accompanied by annotation guide- lines that account for morphologically complex phenomena and discrepancies in word-mention boundaries. Additionally we introduce an evaluation protocol that remains sensitive to word-mention boundary discrepancies, provid- ing a robust framework for comparing and con- trasting models, both generative decoders and encoder-based, on raw (unsegmented) texts. The contribution of this paper is thus man- ifold. First, we presentKibutzR, the first modern Hebrew coreference resolution dataset, alongside detailed annotation guidelines and a rule-based mention detector. Second, we in- troduce a segmentation-aware evaluation pro- tocol that makes word–mention boundary discrepancies explicit without assuming gold morphological analysis. Although instanti- ated for Hebrew, this protocol provides a clear blueprint for porting segmentation-aware coreference evaluation to other MRLs. Finally, we show a comprehensive empirical analysis of contemporary models tracing performance gaps back to their roots in detection and clus- tering. Together, these contributions provide an immediate resource for Hebrew NLP, and a methodological blueprint for targeting and achieving improved CR capabilities in Hebrew and other MRLs. 2 Morphological Challenges in Coreference Resolution 2.1 Referring Expressions in MRLs In morphologically rich languages, referring ex- pressions frequently occur as subtoken units. In Hebrew, for instance, the tokenדבריו(‘his words’) exemplifies this challenge: this single orthographic token requires segmentation into דבר_של_הוא(words_of_he). 1 It thus contains two separate mentions that can co refer to dis- tinct entitieswordsandhe. This phenomenon extends across typologi- cally diverse languages. Arabickitābu-hu(’his book’) similarly fuses nominal and pronominal elements that must be decomposed intokitāb+ hufor coreference resolution ( Maamouri et al., 2004). Turkish agglutination producesev- lerimizden(‘from our houses’), stacking mor- phemes asev+ler+imiz+den, where posses- sive and plural markers create overlapping mention spans ( Schüller et al.,2017). Basque etxekoak(‘those of the house’) demonstrates comparable fusion throughetxe+ko+ak, in- terleaving genitive marking with determina- tion ( Soraluze et al.,2019). Georgian verbal morphology presents the most complex case: mogvts’eren(‘they will write to us’) encodes multiple argument references through preverb mo-, object markergv, and subject agreement -en, all within a single form. These morphologi- cal patterns fundamentally challenge the word- as-unit assumption underlying most CR sys- tems. Unlike English, where mention bound- aries align with whitespaces, MRLs require models to simultaneously segment morphemes and resolve their referential relations, trans- forming a primarily semantic task into one de- manding morphosyntactic proficiency. Beyond such bounded clitics, three addi- tional phenomena increase the difficulty of CR in MRLs. First, pro-drop significantly increases ambiguity by omitting arguments that can be recovered from morphological cues 1 By Hebrew UD v2 conventions (Sade et al.,2018). (Demir and Akdağ,2024;Maamouri et al., 2009;Soraluze et al.,2019). Unlike languages with obligatory overt subjects, pro-drop lan- guages force CR systems to infer referents from verbal inflections, creating additional decision points in the resolution process. Second,construct-state nouns(CSNs) cre- ate deeply nested nominal structures that chal- lenge mention boundary detection. Unlike En- glish, which uses prepositions and determin- ers to mark possession and modification, CSN concatenates nouns directly, creating complex multi-word expressions asדו"ח ישיבת ועדת מנוי דיני בית הדין הרבני הגדול(‘report of the meeting of the committee of appointments of judges of the Great Rabbinical Court’). These construc- tions pose significant challenges for CR: they need to identify the entire construct chain as a single mention while also recognizing potential embedded mentions to (co)-refer to. Third, many MRLs show flexible word order, that erodes positional heuristics for salience and proximity that systems traditionally ex- ploit ( Al-Thubaity and Al-Dossari,2017;So- raluze et al. ,2019). Classical approaches re- lied on surface positioning patterns from fixed word order languages, where syntactic roles correlate with linear position. However, the equivalence of Hebrewדן קרא את הספרandאת הספר קרא דן(‘Dan read the book’), undermines positional features and surface-level cues. These three factors strip away surface cues that English-centric systems exploit, forcing (any kind of) models to rely more on mor- phological and semantic understanding rather than mere surface patterns. 2.2 Word Segmentation: From Pipeline Artifacts to Modern Bottlenecks A long-standing, implicit yet persistent as- sumption in evaluating CR systems is that space-delimited tokens represent single men- tions. This paradigm, while standard in the field, fundamentally misrepresents the chal- lenge of MRL CR and creates a substantial gap between reported performance in research papers and real-world deployment scenarios. In English, where referential expressions of- ten appear as standalone tokens, the distinc- tion is inconsequential — models trained on space-delimited text face no additional chal- lenge in deployment. In MRLs, the chal- lenge to identify and segment referential mate- rial is inseparable from coreference resolution, as multiple referential material routinely ap- pear within a complex word that are input- streamed to the model. To address this intrinsic duality of mention detection and coreference clustering, MRL re- search standardized the use of gold-segmented text for coreference annotation and corefer- ence evaluation as a pragmatic solution. Con- cretely, all major MRL CR corpora adopt this approach: the OntoNotes–Arabic corpus reuses the Penn ATB segmentation; Marmara– Turkish inherits METU–Sabancı morpholog- ical analysis; and EUSKOREC pre-extracts mentions with finite-state rules ( Pradhan et al.,2012;Schüller et al.,2017;Soraluze et al.,2019). While gold segmentation sim- plifies annotation and streamlines evaluation, it creates an artificial evaluation scenario that diverges from real-world text processing. In the earlier NLP-pipelines era,mention de- tectionand coreference clustering formed two distinct phases, allowing researchers to isolate error sources by comparing performance with gold versus automatic mention-boundary de- tection. This diagnostic capability has been crucial: Marmara’s baseline dropped by 31.4 CoNLL F 1 when switching from gold to au- tomatic mentions, while Basque reported 19– 21-point drops ( Schüller et al.,2017;Soraluze et al.,2015). Such comparisons revealed pre- cisely where systems failed: was the bottleneck in thedetectionorclusteringphase? In today’s neural end-to-end era, LLMs pro- cess raw text directly, fusing mention detec- tion and coreference clustering into a single phase ( Lee et al.,2017;Joshi et al.,2019). The relevant contrast is in what the model should be provided as input: should this be raw text or pre-segmented tokens? Subsequently, the question becomes: when a CR model fails on raw MRL texts, can we distinguish whether errors stem from segmentation challenges (fail- ing to identify clitics), linguistic nuances (mis- handling construct states), or genuine corefer- ence confusion (incorrect cluster)? This shift fundamentally changes how we should evaluate MRL coreference systems, yet existing CR benchmarks for MRLs continue to report results on gold-segmented text, creating an evaluation-deployment gap: models achieve strong performance on pre-segmented bench- marks but struggle with raw text in deploy- ment. This misalignment prevents us from un- derstanding — let alone improving — actual CR performance on MRLs, as we lack diagnos- tic tools to isolate error sources and quantify their relative impact. 2.3 Towards Resolving Coreference Resolution in MRLs To address the fundamental gap in MRL CR we propose a comprehensive solution that aligns research benchmarks with deployment realities while maintaining diagnostic capabil- ities, via two interconnected objectives. First, we need a comprehensive dataset that captures the full spectrum of mention spans, from subword to multiword levels, to reflect how referring expressions actually manifest in these languages. Unlike existing MRL cor- pora that rely on pre-existing segmentation schemas, we need annotations that explic- itly mark referential material within complex word-forms — pronominal clitics, construct states, and other linguistic phenomena. We address this gap in Section 3by construct- ing KibutzR, the first comprehensive Hebrew coreference dataset with morpheme-aware an- notations that systematically handle morpho- logical challenges in CR. Second, we critically need evaluation sce- narios that bridge the gap between research and deployment while preserving diagnostic capabilities. Rather than abandoning the in- sights from pipeline-era evaluations, we pro- pose a diagnostic ladder of input conditions that systematically varies the level of prepro- cessing: (1) raw text, as encountered in real applications; (2) automatic segmentation, re- vealing the impact of segmentation errors; and (3) gold segmentation, isolating pure corefer- ence challenges from morphological intricacies. This three-regime evaluation protocol allows us to quantify the relative contribution of dif- ferent error sources — segmentation versus clustering — while maintaining comparability with existing benchmarks. By testing models across all three input conditions, we can an- swer critical questions about model capabili- ties: How substantial is the performance gap between raw and segmented text? What pro- portion of errors stem from segmentation ver- sus genuine coreference confusion? Do mod- eling advantages observed in English, where LLMs now dominate, translate to MRLs when processing raw text? We define and implement this evaluation framework in Section4. With these foundations in place — compre- hensive data and diagnostic evaluation — in Section5we establish baseline performance across architectures, both state-of-the-art gen- erative LLMs and neural encoder models, providing the first comprehensive evaluation leaderboard for Hebrew coreference resolution that reflects the gap between research scenar- ios and deployment realities. 3 Building KibutzR: Construction and Annotation of the Modern Hebrew Coreference Corpus 3.1 Scope and Document Selection The Hebrew KibutzR dataset is based on The Hebrew Universal Dependencies Treebank ( Sade et al.,2018), containing 6,151 sentences, without document boundaries — a critical lim- itation for coreference resolution research. To solve this we reconstructed the original docu- ment structure. Through metadata analysis and discourse pattern recognition, we success- fully identified document boundaries and seg- mented the continuous sentence stream into 351 complete documents. This transformation — from isolated sentences to coherent docu- ments averaging 17.3 sentences (453.6 tokens) As shown in Figure 1— provides Hebrew NLP with its first document-aware corpus derived from the richly annotated UD treebank. The reconstruction process required rethink- ing train/dev/test partitioning. The original Hebrew UD splits scattered sentences from the same document across different partitions, cre- ating evaluation contamination. We therefore relocated any document appearing in multi- ple partitions exclusively to training. This principled partitioning preserves compatibility with existing Hebrew NLP tools trained on UD while ensuring clean evaluation: dependency parsers and morphological analyzers can oper- ate on the same distribution without compro- mising coreference evaluation integrity. Split Docs Sents Tokens Mentions %Docs Train 301 5,236 137,33317,50085.8 Dev26 428 10,4741,2437.4 Test24 487 12,1681,4516.8 Total 351 6,151 159,97520,194 † 100 Table 1: Corpus statistics for KibutzR. The mark- ing † excludes singletons. With singletons, the cor- pus contains 47,879 mentions. Figure 1: Distribution of sentences per document. 3.2 Annotation Guidelines Our annotation guidelines adopt OntoNotes 5.0 as their foundation while making two sub- stantive departures to accommodate concrete morphosyntactic phenomena (Sec. 2). First, OntoNotes restricts mention bound- aries to space-delimited tokens and does not annotate sub-token morphemes. This is prob- lematic for Hebrew where possessive suffixes and pronominal clitics carry independent ref- erents. We therefore permitmorpheme-level mentions within orthographic tokens whenever fused morphemes carry reference. To illus- trate, inדבריו(’his words’), we annotate the possessive suffix as a separate pronominal men- tion alongside the full token. This modifi- cation enables annotation of possessive and pronominal clitics, and proclitic preposition- s/conjunctions — all central to Hebrew refer- ence. We do not split lexical roots from tem- platic patterns. Second, OntoNotes generally annotates only single maximal NPs (no nested mentions/i- within-i), with limited exceptions (e.g., proper- noun premodifiers and appositives) — insuffi- cient for recursive construct state nouns where sub-constituents maintain independent refer- ents. We therefore treat recursivesmixut(con- struct state) as nested mention hierarchies. Forמשלת אנגליה(‘the government of Eng- Figure 2: Agreement improvement across annota- tion rounds (CoNLL and Mention F 1 ). Figure 3: The five-stage annotation pipeline land’), we allow coreference links to both the full compound and the embedded constituent אנגליה(‘England’), capturing the dual referen- tial nature of these constructions. 2 The complete annotation manual, with de- tailed examples, is available in our repository. 3 3.3 Annotation Pipeline and Quality Control Our annotation methodology follows a system- atic five-stage pipeline, illustrated in Figure3. Given the challenges outlined in Section 2, we separate mention boundary decisions from coreference clustering. This separation allows annotators to first resolve complex morpholog- ical boundaries before making referential judg- ments, reducing cognitive load and improving consistency. All stages are carried out inde- pendently by at least three annotators to se- cure reliable coverage in high agreement. For the annotation platform, we extended the TNE annotation platform ( Elazar et al., 2022) with support for morpheme-level span editing and robust right-to-left Hebrew dis- play, enabling efficient and accurate annota- tion of complex Hebrew morphological struc- 2 Other adjustments, e.g. head selection for quan- tificational/partitive NPs and Hebrew-specific tests for non-referentials, are refinements rather than concep- tual. 3 Data, guidelines, and metadata are available at https://github.com/OnlpLab/hebrew_coreference_ data/. tures. Annotators were compensated at 50 NIS/hour (60% more than the state’s mini- mum wage). The process begins withautomatic men- tion detection. We designed a custom rule- based mention detector that leverages Uni- versal Dependencies parses with rules for cl- itics, construct states, and other morpholog- ical and morphosyntactic phenomena. This recall-oriented system processes raw Hebrew text from UD-annotated documents and pro- duces text with candidate mention-spans pre- marked, including nested mentions. The high- recall design ensures comprehensive candidate coverage, allowing annotators to focus on re- finement rather than mention discovery. Next is themention refinement stage, where human annotators receive text with pre-marked mention candidates and system- atically accept or reject each candidate while adding any mentions missed by the rule-based detector. Using morpheme-level span edit- ing capabilities, annotators can precisely de- fine mention boundaries that cross traditional word boundaries. Each annotator produces their individual mention decisions for subse- quent consolidation. Later, themention consolidation stage resolves inter-annotator variation by merging mention decisions into a unified inventory. Boundary disagreements are settled by major- ity vote or expert adjudication, producing a fixed set of mentions before clustering. Duringcoreference clustering, annota- tors process the consolidated mentions sequen- tially using a single-link strategy, deciding for each mention whether it opens a new cluster or links to an existing one. Singletons are re- tained to ensure complete mention coverage. Finally, thequality control and adju- dication stagehandles systematic disagree- ment resolution through expert consolidation. An expert annotator reviews all disagreements and consolidates them cluster by cluster in se- quential order, following an approach similar to Bornstein et al.(2020). This expert makes final decisions on conflicting annotations based on linguistic criteria and annotation guidelines, ensuring consistency across the dataset. We monitoredinter-annotator agree- mentafter each batch and conducted a tar- geted revision following the first three batches. This iterative process improved macro aver- age pairwise scores from 0.63 (mention) / 0.52 (CoNLL) initially to a final agreement of 0.87 for mentions and 0.81 for CoNLL F 1 — rela- tive gains of 38% and 56% respectively (Fig- ure2). The final agreement scores for the dataset are: Mention F 1 = 0.87 (P: 0.85, R: 0.90), MUC = 0.84, B 3 = 0.81, CEAF φ = 0.79. Notably, these final agreement scores match or surpassall of those reported for the most known large-scale CR corpora: OntoNotes (85.8% MUC F 1 ;Pradhan et al. 2012), LitBank (78% B 3 F 1 ;Bamman et al. 2019), and PreCo (77% agreement;Chen et al. 2018). 4 Experimental Setup Goal and Evaluation ScenariosWe set out to evaluate coreference resolution models and isolate the impact of morphological com- plexity on downstream performance. To do so, we propose to assess CR in 3 input conditions: •Raw Text:The CR models receive un- segmented text and must discover mention boundaries and resolve coreference chains •Automatic Segmentation:The CR models operate on text segmented by a state-of-the-art linguistic segmentation model. 4 •Gold Segmentation:The CR models re- ceive gold-standard segments boundaries. For LLMs, we additionally assess CR perfor- mance in aGold Mentionscondition, where models receive pre-identified mentions, to iso- late performance on theclusteringsubtask. 5 4 For the automatic segmentation condition, we use the joint Hebrew segmentation model ofYshaayahu Levi and Tsarfaty(2024), which achieves state-of-the-art performance on Hebrew UD benchmarks and produces Universal Dependencies- conformant boundaries – 98.52 F1 on Hebrew UD benchmarks ( Yshaayahu Levi and Tsarfaty,2024). 5 Because neural encoders operate on token se- quences(i.e., as token/span classifiers rather than seg- menters), they require pre-segmented input by design and cannot process raw text directly, so neural en- coder based systems are evaluated in automatic and gold segmentation conditions only. while LLMs are evaluated across all three input conditions plus gold mentions. This asymmetry reflects fundamental archi- tectural differences: current neural encoder based sys- tems inherit pipeline assumptions where segmentation precedes coreference, while LLMs are capable of han- dling both tasks simultaneously from raw text. Metrics.We follow the standard CoNLL- 2012 shared task evaluation setup (Pradhan et al.,2012), which established three met- rics as the canonical framework for cross- linguistic coreference evaluation. 6 We report standard CR metrics: MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin,1998), and CEAF φ (Luo,2005), along with their arith- metic mean (CoNLL F 1 ). MUC measures link-based precision and recall, B 3 evaluates mention-based clustering quality, and CEAF φ computes the optimal alignment between pre- dicted and gold clusters. Neural Encoder Models.We fine-tune two state-of-the-art neural CR architectures on KibutzR’s training set. Thewl-coref system (Dobrovolskii,2021) uses word-level span representations with a coarse-to-fine an- tecedent scoring mechanism. TheLingMess system ( Otmazgin et al.,2023) incorporates multiple expert scorers that capture different linguistic signals. We adapt LingMess to He- brew (LingMess-He) by replacing the English stopword and pronoun lists with their Hebrew equivalents in the scorer modules. We exper- iment with two Hebrew neural encoders, Ale- phBERT ( Seker et al.,2022) and DictaBERT ( Shmidman et al.,2023), which constitute the strongest pretrained models available for Mod- ern Hebrew to date. 7 Generative Language Models.We eval- uate eight state-of-the-art LLMs: a Hebrew- monolingual model, DictaLM 2.0 (Shmidman et al. ,2024) , and major multilingual LLMs: GPT-4.1/4o/o1/o3, and Gemini 2.0-Flash/- Lite/2.5-Pro. Following prior work (Le and Ritter,2023), we use zero-shot prompting for the gold mentions condition to enable direct comparison with their English setup. For the end-to-end task (i.e., raw and gold segmenta- tion conditions), we employ 2-shot prompting 6 Although we annotated singleton mentions, all re- ported results exclude them, following the conventions of prior work (Cattan et al.,2021). 7 Training: 150 epochs, AdamW optimizer (encoder lr=1×10 −5 , task lr=3×10 −4 ), dropout=0.3.wl-coref: max span=64, top-k=50. LingMess-He: max span=30, top-λ=0.4. Results averaged over 5 seeds.All models use early stopping on development CoNLL F 1 with hy- perparameters following the original papers. Full hy- perparameter details are supplied in the Appendix D. All code and training scripts are available athttps: //github.com/OnlpLab/hebrew_coreference. Model Gold Segmentation MUC B 3 CEAF φ CoNLL F 1 wl-coref (+AlephBERT)47.1 41.6 44.444.4 wl-coref (+DictaBERT-base) 47.7 41.9 44.144.5 lingmess-he (+AlephBERT) 45.7 41.8 45.244.3 lingmess-he (+DictaBERT)52.6 47.7 51.050.4 Table 2: neural encoder baseline performance with gold segmentation. Results averaged across 5 seeds; standard deviations range from±0.7 to±3.7 F 1 points. This evaluation regime matches all prior coreference work on other languages. with examples from the training data, as zero- shot evaluation proved ineffective for this more complex task. We experimented with multiple prompting strategies to optimize model perfor- mance. Our final reported end-to-end prompt incorporates a Chain-of-Thought (CoT) struc- ture through a three-step pipeline that ex- plicitly handles Hebrew morphological chal- lenges. We evaluated each scenario with tem- perature 0 and report averages across five runs. See complete prompts and examples in ap- pendixB. 5 Results and Discussion Neural Encoder-Based SystemsTable2 presents our neural encoder baseline results un- der gold segmentation, the standard de facto evaluation standard in coreference resolution. Our strongest system,LingMess-Hewith the DictaBERT encoder, achieves 50.4 CoNLL F 1 . The choice of the Hebrew encoder also proves critical; DictaBERT consistently outperforms AlephBERT by 5–6 F 1 points, likely due to its larger pre-training corpus and larger vocabu- lary, optimized for Modern Hebrew. To isolate the effect of linguistic segmen- tation on downstream performance, we eval- uated our best models on gold-segmented in- put vs. predicted segmentation. Table 3re- veals a consistent performance drop of 1.4–2.6 F 1 points when replacing gold with automatic segmentation. Although modest in absolute terms, this degradation isolates segmentation as an independent bottleneck in Hebrew NLP pipelines. This finding reveals that report- ing scores only on gold-segmentation scenarios, hides a substantial portion of the error budget. For MRLs where the traditional assumption of reliable segmentation breaks down, results should be reported under both gold and auto- Model Gold Segmentation SOTA Automatic Segmentation ∆F 1 MUC B 3 CEAF φ F 1 MUC B 3 CEAF φ F 1 lingmess-he (+AlephBERT) 45.7 41.8 45.2 44.3 44.2 40.4 43.942.8−1.4 lingmess-he (+DictaBERT) 52.6 47.7 51.0 50.4 50.0 45.1 48.547.9−2.6 Table 3: Neural model performance under gold versus automatic segmentation. The∆F 1 column shows absolute performance drop when using SOTA segmentation instead of gold tokens. Both conditions use identical text; only token boundaries differ. ModelRaw Automatic Gold Gold Text Seg.Seg. Mentions Dicta 2.01.01.50.313.8 GPT-4.115.117.217.744.8 GPT-4o14.215.314.545.4 o113.416.117.937.9 o315.718.819.442.1 Gemini 2.0-Flash13.219.115.241.0 Gemini 2.0-Flash-Lite 12.114.815.238.4 Gemini 2.5-Pro22.227.426.844.7 Best neural baseline—47.950.4— Table 4: LLM performance (CoNLL F 1 ) under four regimes. Results averaged across 5 runs; closed- source LLMs exhibitσ=0.4–3.2 despite tempera- ture 0, reflecting known non-determinism in pro- duction systems. matic segmentation conditions, as well as on raw text, to provide realistic performance esti- mates. Generative LLMsTable4evaluates state- of-the-art LLMs on Hebrew CR using in- context learning under four evaluation condi- tions: raw text, automatic segmentation, gold segmentation, and gold mentions. Surprisingly, even when provided with per- fect mention boundaries (the gold-mention condition), the best-performing LLM (GPT- 4o at 45.4 F 1 ) falls 5.0 points behind the much smaller neural encoder-based baseline (50.4 F 1 ). This finding is particularly striking given the vast parameter difference—hundreds of bil- lions compared with hundreds of millions. The underperformance persists across all evaluated models; prominent LLMs like o3, Gemini 2.5- Pro, and GPT-4.1 all fail to exceed 45 F 1 . This pattern directly contradicts the English pat- tern where the same LLMs consistently outper- form neural encoder-based systems (Table 5). The inverse performance trend, from a +7.0 F 1 advantage in English to a -5.0 F 1 deficit in Hebrew, suggests that current LLMs struggle with Hebrew coreference under morphological complexity. This underperformance highlights a com- Language Neural-EncoderLLM∆ (Gold Seg.) (Gold Mentions) (LLM−NE) English81.488.4+7.0 Hebrew50.445.4−5.0 Table 5: Cross-linguistic performance inversion. English results fromLe and Ritter(2023) using the same LingMess architecture. The 12-point swing between languages reveals fundamental lim- itations of current LLMs on morphologically rich languages. pounded bottleneck: a combined failure in seg- mentation and clustering. Providing LLMs with gold mentions yields a massive perfor- mance jump (from 26.8 F 1 to 45.4 F 1 ), confirm- ing that mention boundary recovery remains a significant hurdle. At the same time, the comparison between automatic and gold seg- mentation is mixed across LLMs, suggesting that better segmentation alone does not con- sistently resolve the problem. However, the fact that LLM performance plateaus at 45.4 F 1 even with perfect mention boundaries— still 5.0 points behind the much smaller neural encoders—indicates that a substantial bottle- neck remains at the discourse-level clustering stage. Our detailed error analysis further supports this interpretation. While gold segmentation inflates pronoun share, their resolution is often easier via proximity and agreement features and therefore yields only modest gains. How- ever, neural encoders produce five times more correct clusters than LLMs (9.2/8.9 vs. 1.6– 1.9 per document) and miss seven fewer gold clusters (9.7–10.0 vs. 17.0–17.3). Thus, even when both architectures receive segmented in- put, LLMs continue to struggle with discourse- level clustering. See further details in Ap- pendix H. 6 Conclusion This paper introducesKibutzR, the first coref- erence corpus for Modern Hebrew, and uses it to re-examine modeling and evaluation prac- tices of coreference resolution under morpho- logical complexity. By evaluating both super- vised encoders and frontier LLMs, across sce- narios with raw text, automatic segmentation, gold segmentation, and gold mentions, we can isolate and characterize the segmentation and clustering challenges. We show that, first, state-of-the-art seg- mentation reduces performance of neural en- coders by 1.4–2.6 F 1 points relative to gold. Next, contemporary LLM decoders underper- form neural encoders by 5 points in Hebrew, even with gold mentions, reversing the English pattern where decoders dominate. Finally, we show that LLMs face a combined bottleneck in boundary recovery and clustering: while gold mentions yield dramatic improvements (∼20– 30 F 1 ), the comparison between automatic and gold segmentation is mixed across LLMs, sug- gesting that segmentation alone does not ex- plain the gap, and that substantial difficulty remains at the clustering stage. These findings reveal that current CR mod- eling and prompting practices struggle with Hebrew under the dual challenge of segmen- tation and coreference. We release KibutzR, its guidelines, its annotation UI, prompts, and evaluation code, to enable research toward architectures that better handle morpholog- ical segmentation and coreference resolution as interconnected components, in Hebrew and other MRLs. Limitations Our corpus is limited to newswire text; broader genres remain a subject of future work. The requirement for UD-annotated text dur- ing annotation restricted us to news. The use of automatic parses may mitigate this, but risk increasing error propagation sources. Recently released Hebrew UD parsers and analyzers en- able expansion to other domains. The corpus consists of publicly available news articles. As newswire may encode topical and gender biases, results may not generalize beyond news domains, or across news domains in vastly different (temporally spread) eras. LLM results reflect zero-shot prompting for gold mentions (following prior work) and few- shot prompting for end-to-end evaluation; al- ternative instruction curricula may improve performance, but do not eliminate the need to model segmentation uncertainty. Finally, while our automatic segmenter is strong, im- proved segmentation tools could narrow (even if not entirely erase) the observed gaps. Finally, while all of our code, guidelines, modeling, and experimental design is done in a language-agnostic manner, it is executed and evaluated only on Modern Hebrew texts. Par- allel stream of research on additional MRLs are needed to strengthen the cross-lingual manifestation of this challenge. We hope that are code, guidelines, and actual tools (UI, eval- uation setups) will greatly facilitate and ex- pand the development of such resources and analyses for multiple languages. Acknowledgments We thank Omer Goldman and Arie Cattan for their insightful comments, and three anony- mous reviewers for their valuable feedback. This research was supported by a grant from the Israeli Science Foundation (ISF grant no. 670/23) as well as a grant from the Israeli In- novation Authority (KAMIN), for which we are grateful. The computing resources for the project were kindly funded by a VATAT grant from the Planning and Budgeting Committee of the Council for Higher Education in Israel. References Aseel Al-Thubaity and Sarah Al-Dossari. 2017. A coreference resolution approach using morphological features in Arabic. In 2017 2nd International Conference on Anti- Cybercrime (ICACC), pages 123–128. IEEE. Amit Bagga and Breck Baldwin. 1998.Entity- based cross-document coreferencing using the vector space model. In36th Annual Meeting of the Association for Computa- tional Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 79–85, Montreal, Quebec, Canada. Association for Computational Lin- guistics. David Bamman, Ted Underwood, and Noah A. Smith. 2019.A large-scale corpus of coreference in literary text. InProceed- ings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 5746–5756, Hong Kong, China. Association for Computational Lin- guistics. Dan Bareket and Reut Tsarfaty. 2021. Neural modeling for named entities and morphol- ogy (NEMO2).Transactions of the Associ- ation for Computational Linguistics, 9:909– 928. Ari Bornstein, Arie Cattan, and Ido Dagan. 2020.CoRefi: A crowd sourcing suite for coreference annotation. InProceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstrations, pages 205–215, Online. As- sociation for Computational Linguistics. Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2021.Realis- tic evaluation principles for cross-document coreference resolution . InProceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 143–151, Online. Association for Computa- tional Linguistics. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018.PreCo: A large-scale dataset in preschool vocabulary for coreference resolution . InProceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 172–181, Brussels, Belgium. Association for Computational Linguistics. Amir DN Cohen, Hilla Merhav, Yoav Gold- berg, and Reut Tsarfaty. 2025. Heq: a large and diverse hebrew reading comprehension benchmark.Preprint, arXiv:2508.01812. Şeniz Demir and Hanifi İbrahim Akdağ. 2024. Mention detection in Turkish coreference resolution.Turkish Journal of Electri- cal Engineering and Computer Sciences, 32(5):682–697. Vladimir Dobrovolskii. 2021.Word-level coref- erence resolution. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7670–7675, On- line and Punta Cana, Dominican Republic. Association for Computational Linguistics. Yanai Elazar, Victoria Basmov, Yoav Gold- berg, and Reut Tsarfaty. 2022.Text-based NP enrichment.Transactions of the Associ- ation for Computational Linguistics, 10:764– 784. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019.BERT for coref- erence resolution: Baselines and analysis. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Compu- tational Linguistics. Nghia T. Le and Alan Ritter. 2023.Are large language models robust coreference re- solvers? Preprint, arXiv:2305.14489. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.End-to-end neural coref- erence resolution. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copen- hagen, Denmark. Association for Computa- tional Linguistics. Xiaoqiang Luo. 2005.On coreference resolu- tion performance metrics . InProceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. As- sociation for Computational Linguistics. Mohamed Maamouri, Ann Bies, Tim Buckwal- ter, and Wigdan Mekki. 2004. The Penn Arabic Treebank: Building a large-scale an- notated Arabic corpus. InNEMLAR confer- ence on Arabic language resources and tools . Mohamed Maamouri, Ann Bies, Seth Kulick, Nizar Habash, and Owen Obeid. 2009. A conventional orthography for dialectal Ara- bic. InProceedings of the LREC 2009 Work- shop on Semitic Language Processing. Amir More and Reut Tsarfaty. 2016. Data- driven morphological analysis and disam- biguation for morphologically rich languages in the universal dependencies framework. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pages 1481–1491. Shon Otmazgin, Arie Cattan, and Yoav Gold- berg. 2023.LingMess: Linguistically in- formed multi expert scorers for coreference resolution. InProceedings of the 17th Con- ference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 2752–2760, Dubrovnik, Croatia. Association for Computational Linguistics. Sameer Pradhan, Alessandro Moschitti, Ni- anwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted corefer- ence in OntoNotes. InJoint Conference on EMNLP and CoNLL - Shared Task, pages 1– 40, Jeju Island, Korea. Association for Com- putational Linguistics. Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The Hebrew Universal Dependency treebank: Past present and future . InPro- ceedings of the Second Workshop on Univer- sal Dependencies (UDW 2018), pages 133– 143, Brussels, Belgium. Association for Com- putational Linguistics. Peter Schüller, Kübra Cıngılı, Ferit Tunçer, Barış Gün Sürmeli, Ayşegül Pekel, Ayşe Hande Karatay, and Hacer Ezgi Karakaş. 2017. Marmara Turkish Corefer- ence Corpus and Coreference Resolution Baseline.arXiv preprint arXiv:1706.01863. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, and Reut Tsarfaty. 2022. AlephBERT: Language model pre-training and evaluation from sub- word to sentence level. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46–56, Dublin, Ireland. Asso- ciation for Computational Linguistics. Avi Shmidman, Shaltiel Tzion, and Moshe Shmidman. 2023. DictaBERT: A state- of-the-art BERT suite for modern Hebrew. arXiv preprint arXiv:2308.16687. Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, and Moshe Koppel. 2024.Adapting llms to hebrew: Unveiling dictalm 2.0 with enhanced vocabulary and instruction capa- bilities.Preprint, arXiv:2407.07080. Ander Soraluze, Xabier Arregi, Olatz Arregi, and Xabier Artola. 2015. Adapting the Stanford Coreference Resolution System to Basque. InProceedings of the 20th Interna- tional Conference on Applications of Natural Language to Information Systems (NLDB 2015), pages 289–296. Ander Soraluze, Xabier Arregi, and Xabier Ar- tola. 2019. EUSKOR: An end-to-end coref- erence resolution system for Basque.PloS one, 14(9):e0221801. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995.A model-theoretic coreference scor- ing scheme. InSixth Message Understand- ing Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. Danit Yshaayahu Levi and Reut Tsarfaty. 2024. A truly joint neural architecture for segmentation and parsing. InProceedings of the 18th Conference of the European Chap- ter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian’s, Malta. Association for Computa- tional Linguistics. A Mention-Detection Implementation A.1 Inputs and Dependencies The system requires Universal Dependencies parses of Hebrew text, including tokeniza- tion, POS tags, and dependency labels. Root candidates compriseNOUN,PROPN,PRON,NUM, and verbs when occurring insmixutcon- structions. Span expansion utilizes the fol- lowing permitted dependency labels:appos, compound:smixut,nmod:poss, andconj. A.2 Span Construction Rules (Deterministic) Spans expand bidirectionally from a root node along the permitted dependency edges, op- tionally includingamodanddetmodifiers ac- cording to Hebrew-specific rules. The sys- tem handles Hebrew-specific phenomena in- cluding pronominal clitics (possessive, object, and prepositional), nested smixut construc- tions, determiners, numerals, and conjunc- tions. When overlaps occur, the system re- solves them by selecting the maximal span, while compatible overlaps are merged. The traversal process is fully deterministic, em- ploying fixed token order and single-threaded graph walks to ensure reproducibility. A.3 Mention-Detection Pseudocode Preconditions.UD-parsed Hebrew sentences; root candidates:NOUN/PROPN/PRON/NUMand verbs-in-smixut; permitted edges:appos, compound:smixut,nmod:poss,conj. Deter- ministic traversal; overlaps resolved by maxi- mal span. Algorithm 1High-level Mention Detection for Hebrew 1:foreach sentence in a UD-parsed corpusdo 2:foreach tokentin the sentencedo 3:iftis a root candidate (NOUN/PROPN/PRON/NUM/verb-in- smixut)then 4:Initialize spanS←[t] 5:ExpandSleft/right via permitted depen- dency relations 6:Apply Hebrew-specific rules (clitics, deter- miners, numerals, conjunctions) 7:AddSto mention candidates 8:end if 9:end for 10:end for 11:Filter/merge overlaps by maximal span 12:Output all candidate mentions The complete Mention Detection system will be released upon publication. B Prompt Templates Gold-MentionClus- tering(doc_template). Task:Annotate all entity mentions in the following text with coreference clusters. Use Markdown tags to indicate clusters, e.g. [mention](#cluster_name). Do not add extra information or clusters beyond those marked. Input: [Tom](#) and [Mary](#) go to [the park](#). [It](#) was full of trees. Output: [Tom](#cluster_0) and [Mary](#cluster_1) go to [the park](#cluster_2). [It](#cluster_2) was full of trees. End-to-End Coreference (e2e_template). Cluster all entity mentions in the following He- brew text to coreference clusters. Use Markdown tags to indicate the coreference group in the output, with the format[mention](#), e.g. של]ביתהלכוכש]הם[)#( ]אנחנו[)#([)#( • First, tokenize words to expose clitics • Second, mark the mentions • Finally, cluster all mentions together Tokenization example # Input word:ביתנו # Output:בית_של_אנחנו # Input word:העסקתם # Output:העסקה_של_הם Mention-marking example # Input tokenized:אתאיבדהוא הכרה_של_הוא # Output:אתאיבד]הוא[)#( ]הכרה_של_]הוא[)#([)#( Clustering example # Input:אתאיבד]הוא[)#( ]הכרה_של_]הוא[)#([)#( # Output:]הוא[)#אגד_1( איבד את ]הכרה_של_]הוא[)#אגד_1([)#אגד_2( Examples # Input: לידס עלתה למקום החמישי אחרי שניצחה אתמול בחוץ במשחק השלמה את מנצסטר סיטי 3 2. השערים לידס: לי צפמאן )14(, קארל שאט )42(, גורדון סטראקאן )62(. לסיטי: אשלי ורד )49 מ-11 מ(, דיויד ויט )65(. # Output: ]לידס[)#אגד_0( עלתה ל ה_ מקום ה חמישי אחרי ש ניצחה אתמול ב ה_ חוץ ב משחק השלמה את ]מנצסטר סיטי[)#אגד_1( 3 2. ]]ה שערים ל]לידס[)#אגד_0([)#(:]לי צפמאן )14(, קארל שאט )42(, גורדון סטראקאן )62([[)#(. ל]סיטי[)#אגד_1(: אשלי ורד )49 מ - 11 מ(, דיויד ויט )65(. # Input: הרבה החמצות מצבים נוחים של יבנה, בגל משחק הגנתי של טבריה שהזמינה התקפות. בין חלוצי יבנה, שהרבו להחמיץ, ניצל אנריקה ורון הזדמנות אחת בלבד, כדי להעניק לקבוצתו פרס של 3 נקודות בעד נצחון שהיתה ראויה לו. שפט אריה ולף, 000,1 צופים, ביבנה. # Output: הרבה החמצות מ מצבים נוחים של ]יבנה[)#אגד_0(, בגל משחק הגנתי של טבריה ש הזמינה התקפות. בין חלוצי ]יבנה[)#אגד_0(, ש הרבו להחמיץ, ניצל ]אנריקה ורון[)#אגד_1( הזדמנות אחת בלבד, כדי להעניק ל]קבוצה_של_]הוא[)#אגד_1([)#אגד_0( פרס של 3 נקודות בעד ]נצחון ש היתה ראויה ל]הוא[)#אגד_0([)#אגד_1(. שפט אריה ולף, 000,1 צופים, ב יבנה. Nested mentions are allowed; mark every nested span and noun-phrase candidate. Keep the text exactly as it was, except for Markdown. Do not output singletons in the final cluster document. C LLM Inference Pipeline C.1 Deterministic Settings •Temperature= 0.0; default top-p; max output tokens=min(context_limit− input,4096). •Up to 3 retries on format-validation failure; fixed random seeds where applicable. Algorithm 2Inference and Output Valida- tion 1:foreach documentddo 2:p←render_template(d) 3:r←model(p,temperature= 0) 4:Validate bracket balance, sentence alignment, cluster-ID schema 5:ifvalidation failsthen 6:apply non-semantic formatting fix 7:retry (max 3) 8:end if 9:Parse clusters→C d 10:end for 11:return ∪ d C d C.2 Models and Artifacts •Models used (replication list): GPT-4o, GPT-4o-mini, GPT-4.1, o1, o3, Gemini 2.5 Pro, Gemini 2.0 Flash, Gemini 2.0 Flash Lite, DictaLM 2.0-Instruct. •The experimental framework is built on OpenAI API v0.28.1 and vLLM for local model serving. D Neural Baseline Configurations D.1 Training Regime •Optimizer: AdamW; dropout: 0.3; weight decay: 0.01; epochs: 150. •Seeds for averaging: 42, 123, 2021, 27182, 31415. •Max segment length: 512; max span width: 30 (LingMess-He) / 64 (wl-coref); rough scoringk= 50(wl-coref). HyperparameterLingMess-He wl-coref LR (encoder)1×10 −5 1×10 −5 LR (task)3×10 −4 3×10 −4 Dropout0.30.3 Max span width3064 Max segment length512512 Rough scoringk–50 Hidden size (FFNN/Scorer)20481024 Weight decay0.010.01 Table 6: Key hyperparameters for neural baselines (replication). E Annotation Guidelines (Operational) E.1 Mention Types Named entities; nominals; pronominals (in- cluding clitics); zero pronouns when recover- able from morphology. E.2 Hebrew-Specific Handling •Pronominal clitics: possessive on nouns (ספרו→]ספר_של_]הוא[[), object on verbs (ראיתיו→ראיתי_]אותו[), prepositional clitics (אליהם→אל_]הם[). •Construct state (smixut): allow nested con- structs; annotate construct and internal heads when referential. E.3 Exclusions We exclude pleonastic or dummy uses, id- ioms with non-referential components (e.g., יצא מכליו), and negated non-existent entities (e.g.,אין לי מכונית). E.4 Inter-Annotator Agreement Protocol The annotation process began with initial cali- bration on 10 documents followed by group dis- cussion. We then conducted three annotation batches with continuous agreement monitor- ing and targeted rule clarifications. Final ex- pert adjudication resolved disagreements, with IAA computed on pre-adjudication annota- tions. F Scoring Definitions F.1 Weighted Similarity For brevity letC p =C pred andC g =C gold . Sim(C p ,C g ) = ∑ m∈C p ∩C g w(m) ∑ m∈C p ∪C g w(m) .(1) We assign weights based on mention type: pro- nouns (including clitics) receivew(m) = 0.2, while content mentions (nouns and named en- tities) receivew(m) = 1.0. F.2 Outcome Categories •Correct: similarity≥0.5 •Extra: predicted cluster without a suffi- cient gold match •Missed: gold cluster without a sufficient predicted match G Environment and Artifact Bundle Our experiments support single-GPU training with CPU-only inference capability. The ar- tifact bundle includes all tool and model ver- sions, encoder checkpoints, random seeds, run scripts, prompt JSON files, validator regexes, and parsing scripts. The bundle will be dis- tributed upon publication. H Error Analysis We analyze where coreference models fail in Hebrew by comparing error patterns across segmentation conditions and model architec- tures. We examine (1) how mention type distributions — particularly pronoun ratios — shift with segmentation methods, and (2) clustering success rates. We categorize model outputs into three outcomes:Correctclus- ters (predicted clusters that match gold clus- ters),Extraclusters (spurious predictions), andMissedclusters (unresolved gold clusters). To determine matches, we compute weighted overlap between predicted and gold clusters, down-weighting pronouns (0.2) versus content mentions (1.0). 8 Experimental Setup.We compare five configurations:Neural–GoldandNeural– SOTA(our best-performing neural encoder based system on gold/automatic segmenta- tion),LLM–Gold,LLM–SOTA, andLLM– Raw(Gemini 2.5 Pro on gold/automatic/raw text). All analyses use the dev split. 8 Multiple entities can share the same pronoun form (e.g., multiple male entities all referred to as ”he”), making pronoun overlap weak evidence for cluster matching. Figure 4:Pronoun share rises under gold seg- mentation. LLM: 50.0% (LLM–SOTA)→58.9% (LLM–Gold) (+8.9p).Neural: 55.6% (Neural– SOTA)→65.1% (Neural–Gold) (+9.5p). Figure 5:Cluster outcomes by approach (per- document means).Neural models produce 5× more correct clusters than LLMs regardless of seg- mentation quality. Gold segmentation increases pronoun re- covery.By examining which mentions each segmentation method discovers, we observe that switching from automatic to gold seg- mentation increases pronoun mention share by about 9 percentage points in both archi- tectures (Figure 4). This increase occurs be- cause gold segmentation more reliably recov- ers grammatical morphemes, including bound pronouns such as possessive suffixes and object clitics, that automatic segmenters may miss or attach incorrectly. Many of these recovered pronouns are easier to resolve using local agree- ment and proximity cues, so their increased availability likely contributes to the gains un- der gold segmentation. At the same time, this pattern suggests that improved segmentation does not primarily recover the hardest content mentions. Improved segmentation alone does not close the LLM gap.Figure5reveals a striking architecture gap. Neural encoders produce about five times more correct clus- ters than LLMs (9.2/8.9 vs. 1.6–1.9 per doc- ument) and miss roughly seven fewer gold clusters (9.7–10.0 vs. 17.0–17.3). Importantly, this gap persists even when both architectures receive segmented input (Neural–SOTA vs. LLM–SOTA), indicating that better boundary information alone is insufficient to close the gap. The three LLM conditions (Raw/SOTA/- Gold) reinforce this interpretation. Segmented input improves performance relative to raw text, but correct clusters remain sparse across all LLM settings. Taken together, these analy- ses suggest that segmentation is one important source of difficulty, while a substantial resid- ual bottleneck remains at the discourse-level clustering stage.