Paper deep dive
MGAL: A Multilingual Granularity-Aware Long-Context Benchmark
Chunhan Li, Chenglin Xu, Zongyang Zhang, Jiale Liu, Zhuoxi Rao, Xudong Jia, Junxiu He, Menglin Yang, Wenjuan Gong, Zhengzhe Liu, Chengwei Qin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/24/2026, 5:30:45 AM
Summary
The paper introduces MGAL, a multilingual, granularity- and position-aware long-context benchmark for evaluating Large Language Models (LLMs). Constructed from United Nations reports across six official languages with context lengths up to 128K tokens, MGAL assesses model performance across four linguistic granularities (word, sentence, paragraph, document) and three positional strata (begin, middle, end). The study evaluates 12 LLMs, revealing that while models excel at word-level tasks, performance degrades at coarser granularities. Closed-source models maintain an advantage in lower-resource languages. Key challenges identified include local semantic crowding, where models rely on surface cues over discourse roles, and a fluency-consistency gap where outputs are fluent but factually inconsistent.
Entities (15)
Relation Signals (12)
MGAL → evaluatesgranularity → Word
confidence 99% · It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document)...
MGAL → evaluatesgranularity → Sentence
confidence 99% · It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document)...
MGAL → evaluatesgranularity → Paragraph
confidence 99% · It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document)...
MGAL → evaluatesgranularity → Document
confidence 99% · It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document)...
MGAL → usesdatasource → United Nations
confidence 98% · MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages.
Grok 4 → achievesbestperformance → Paragraph
confidence 95% · At coarser granularities, Grok-4 leads on paragraph filling...
GPT-5 → achievesbestperformance → Word
confidence 95% · On word- and sentence-level multiple-choice evaluations, GPT-5 achieves the highest average scores.
GLM-4.5 → achievesbestperformance → Document
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like ``however'' or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.
Tags
Links
- Source: https://arxiv.org/abs/2608.20853v1
- Canonical: https://arxiv.org/abs/2608.20853v1
Trouble viewing inline? Open PDF directly →
Full Text
120,728 characters extracted from source content.
Expand or collapse full text
MGAL: A Multilingual Granularity-Aware Long-Context Benchmark Chunhan Li 1 2 Chenglin Xu 2 Zongyang Zhang 2 Jiale Liu 2 Zhuoxi Rao 3 Xudong Jia 2 Junxiu He 2 Menglin Yang 1 Wenjuan Gong 2 Zhengzhe Liu 4 Chengwei Qin 1 Abstract Evaluation of long-context Large Language Mod- els (LLMs) has advanced rapidly.However, most existing benchmarks are limited to the document level and focus mainly on high- resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilin- gual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K to- kens across the six official UN languages. It cov- ers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilin- gual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear perfor- mance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sen- tences share topics and entities, models tend to fol- low surface cues (e.g., connectives like “however” or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., back- ground, outcome); and (2) A gap between fluency and consistency in generated outputs, where mod- els produce text that reads smoothly but drifts from the source facts. In addition, we observe sev- 1 The Hong Kong University of Science and Technol- ogy(Guangzhou) 2 China University of Petroleum (East China) 3 Northeastern University at Qinhuangdao 4 Lingnan Univer- sity, Hong Kong.Correspondence to:Chengwei Qin <chengweiqin@hkust-gz.edu.cn>. Proceedings of the43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s). eral patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty. 1. Introduction Large Language Models (LLMs) have achieved remarkable progress across a wide range of Natural Language Process- ing (NLP) tasks. An important frontier lies in long-context modeling, where LLMs are required to process books, re- ports, and other documents spanning from thousands to hundreds of thousands of tokens. Effective comprehension of such long-form inputs is essential for applications like summarization, knowledge-intensive QA, and policy analy- sis. However, it remains highly challenging, as models must capture fine-grained cues across multiple discourse levels while maintaining robustness to positional variation. To measure progress, several benchmarks have recently ex- tended evaluation beyond short inputs. LongBench assem- bles a multi-task suite for long contexts (Bai et al., 2024), M 4 LE expands task and language coverage (Kwan et al., 2024), LV-Eval explores long-sequence tasks (Yuan et al., 2024), and ONERULER increases multilingual coverage (Kim et al., 2025b). While each of these benchmarks pro- vides valuable advances, they also share important limi- tations: evaluations primarily focus on document-level in higher-resource languages, offer limited control over the positioning of evidence, and rarely examine fine-grained understanding across different discourse units. This gap highlights a fundamental question: how well do LLMs per- form across varying levels of granularity in long-context, particularly in lower-resource languages? To answer this question, we introduce MGAL, the first multilingual, granularity- and position-aware long-context benchmark, sourced from United Nations Digital Library reports of 8K–128K tokens across the six official languages. MGAL partitions tasks into closed-ended and open-ended groups. Closed-ended tasks admit clear, objectively scorable answers; open-ended tasks allow multiple acceptable re- sponses (e.g., paragraph filling, summarization). For every language, MGAL covers four levels of linguistic granularity with seven tasks and 420 query–response pairs each (table 1), 1 arXiv:2608.20853v1 [cs.AI] 21 Aug 2026 MGAL Table 1. Overview of MGAL task design across four levels of granularity in six UN official languages. Granularity TaskAvg LenMetric#Data Word Single-QA23,887Acc.420 Multi-QA29,048Acc.420 SentenceCloze30,838Acc.420 ParagraphFilling33,687Rouge-L420 Document Summarization28,189Rouge-L420 Translation33,294BLEU420 including word-level QA, sentence-level cloze, paragraph- level filling, and document-level summarization and trans- lation. All items are annotated from scratch to broaden coverage of domains, input lengths, languages, and task types. Beyond granularity, MGAL incorporates position awareness by stratifying examples according to evidence location (beginning, middle, end), indexed at the paragraph and document levels. Following data construction, we pri- oritize quality over quantity by manually auditing every query–response pair and removing flawed samples. MGAL uses position-aligned UN documents, where same- position sentences are semantically matched across six languages, enabling consistent cross-lingual comparison. Building on this alignment, we evaluate with precision- oriented reference metrics such as Accuracy and ROUGE-L for automatic scoring. For newly introduced open-ended generation tasks in MGAL, such as paragraph filling, we further adopt an LLM-as-a-judge method to assess quality aspects that are not fully captured by reference-based auto- matic metrics, following recent best practices (Zheng et al., 2023c; Liu et al., 2023b; Kim et al., 2025a). Upon compre- hensive evaluation of 12 long-context LLMs on MGAL, we find that: (1) models excel on word-level tasks but struggle as evaluation shifts to coarser-grained units. and (2) large open-source and closed-source models are comparable on higher-resource languages, but closed-source systems main- tain a clear advantage on lower-resource languages, with the gap more evident for smaller open-source models. Through empirical analysis, we reveal two new key chal- lenges: (1) under local semantic crowding where neigh- boring sentences contain overlapping words or repeated entities, models rely on shallow signals such as connectives (e.g., however, therefore) and repeated entities (e.g., country names, years), preferring sentences with surface overlap instead of those that fulfill the correct functional role in the paragraph (e.g., background, explanation, outcome); and (2) although outputs are often fluent and stylistically appropri- ate, they are weakly anchored to the input, leading to factual drift and unsupported claims. Additionally, we observe sev- eral patterns consistent with prior work. In sentence-cloze task, error rates are highest when blanks appear early in the text and decrease toward the end, reflecting models’ ten- dency to favor recent context due to recency-biased attention (Peysakhovich & Lerer, 2023; Hsieh et al., 2024b). We also find that models repeatedly pick earlier answer options (e.g., “A” or “B”), showing an early-position bias and reliance on option-order heuristics rather than carefully evaluating the evidence specific to each item (Pezeshkpour & Hruschka, 2024b; Zheng et al., 2023a). Overall, our findings highlight the necessity of combining multilingual, fine-grained, and position-aware evaluation. We position MGAL as a valuable benchmark for assessment across languages and granularities, informing future long- context model development. In summary, our contributions are threefold: •We present MGAL, the first multilingual long-context benchmark that is both granularity-aware (word, sentence, paragraph, document) and position-aware (begin, middle, end), covering all six official UN languages with contexts up to 128K tokens. •Through a fine-grained, multilingual, and position- controlled design, MGAL decomposes evaluation along two key dimensions, i.e., linguistic granularity and evi- dence position, across six languages, enabling more rigor- ous diagnosis than existing document-level benchmarks. • We conduct extensive evaluations of 12 long-context LLMs and use MGAL as a controlled diagnostic testbed to analyze benchmark-grounded limitations, including granularity-dependent degradation, position sensitivity, local semantic crowding, and fluency–consistency gaps. 2. Related work 2.1. Long-Context Modeling for LLMs Long-context modeling has emerged as a key challenge for LLMs, driving innovations in both training and infer- ence strategies. Building on rotary positional embeddings (RoPE) (Su et al., 2024), methods such as Position Interpola- tion (PI) (Chen et al., 2023) have demonstrated effectiveness in extending the usable context length. In parallel, sparse at- tention approaches further enhance scalability. For instance, LongLoRA (Chen et al., 2024) combines shifted sparse at- tention with LoRA, enabling models to handle up to 100k tokens at a modest computational cost. SnapKV (Li et al., 2024b) compresses the KV cache by selecting salient keys from an observation window without requiring fine-tuning, while Squeezed Attention (Hooper et al., 2025) clusters keys from the fixed portion of a prompt into centroids of- fline and subsequently filters the most relevant keys online, thereby accelerating inference when contexts overlap across requests. Collectively, these methods provide a practical 2 MGAL Table 2. Comparison to existing long-context benchmarks. “En”, “Zh”, “Es”, “Fr”, “Ru”, and “Ar” refer to tasks in English, Chinese, Spanish, French, Russian, and Arabic,“Pos-Par.” and “Pos-Doc.” indicate positional evaluation at the paragraph and document level. “Gran.” refers to Granularity. BenchmarkMax LenEnZhEsFrRuArPos-Par.Pos-Doc.Gran. LongBench(Bai et al., 2024)∼10K✓×✓× ZeroSCROLLS (Shaham et al., 2023)∼10K✓×✓× NeedleBench (Li et al., 2024a)∼10K✓×✓× RULER (Hsieh et al., 2024a) ∼10K✓×✓× LaRA (Li et al., 2025)∼10K✓×✓×✓× M 4 LE(Kwan et al., 2024)∼10K✓×✓× LV-Eval (Yuan et al., 2024)4K–60K✓×✓× ONERULER (Kim et al., 2025b)∼20K✓×✓× MGAL (ours)∼128K✓ Figure 1. Overview of MGAL. From (a) aligned multilingual UN long-context documents, we construct (b) a multi-task benchmark covering QA, cloze, paragraph filling, summarization, and translation. These tasks are organized by (c) linguistic granularity and evaluated through (d) diagnostic analyses over granularity, position, and language. toolbox for scaling LLMs to longer inputs while maintaining controllable cost and quality. 2.2. Benchmarks for Long-Context LLMs Existing benchmarks for long-context LLMs primarily eval- uate document-level comprehension and reasoning. Zero- SCROLLS (Shaham et al., 2023) targets zero-shot long- text NLU, while LongBench (Bai et al., 2023) intro- duces a bilingual, multi-task suite spanning single/multi- document QA and query-based summarization. To bet- ter control sequence length and task difficulty, synthetic benchmarks such as NeedleBench (Li et al., 2024a) and RULER (Hsieh et al., 2024a) have been proposed. NeedleBench adds the Ancestral Trace Challenge (ATC) for multi-step logical tracing, whereas RULER extends beyond vanilla NIAH to multi-hop tracing and aggre- gation.M 4 LE (Kwan et al., 2024) further expands task and language coverage, and ONERULER (Kim et al., 2025b) increases multilingual diversity. LaRA (Li et al., 2025) complements these efforts by providing a testbed for contrasting long-context LLMs with retrieval- augmented generation(RAG) pipelines. Despite these ad- vancements, current benchmarks offer limited insight into fine-grained discourse phenomena and lack systematic evaluation across languages and granularities. In partic- ular, they provide little evidence on how position sen- sitivity and discourse consistency differ between higher- and lower-resource languages at different granularities, a gap that MGAL is specifically designed to address.We compare MGAL with existing long-context benchmarks in Table 2. MGAL supports context lengths of up to 128k tokens and covers six official languages.More- over, it enables multi-granularity evaluation of LLMs’ po- sition sensitivity at both the paragraph and document lev- els. 3 MGAL Figure 2. MGAL dataset instantiation at each granularity. 3. MGAL We introduce MGAL, the first Multilingual Granularity- Aware Long-context benchmark, constructed from long- form reports in the United Nations (UN) Digital Library. MGAL covers all six official UN languages (Arabic, Chi- nese, English, French, Russian, Spanish) with context lengths ranging from 8K to 128K tokens. It spans four linguistically coherent levels of granularity (word, sentence, paragraph, document) and stratifies instances by evidence position (begin, middle, end), enabling systematic diagnosis of multilingual long-context comprehension. We illustrate representative MGAL examples at each granularity in Fig- ure 2 and show an overview of MGAL in Figure 1. 3.1. Problem Definition In MGAL, each task is defined as taking a long-document context together with task instructions as input, and produc- ing an output at the required granularity: word, sentence, paragraph, or document. We list the pipeline for MGAL generation and human verification guidelines in Figure 3. 3.2. Dataset Construction MGAL is built through a unified annotation and curation pipeline tailored to each granularity level. All data is con- structed from a large pool of approximately 70,000 UN reports, from which we identify roughly 4,000 structurally suitable reports for benchmark construction. We sample documents across the 8K–128K token range and ensure that each task contains the same number of query–response pairs for all six languages. The resulting source documents cover diverse UN policy and governance domains, including in- ternational development, human rights, peace and security, humanitarian affairs, public health, environment, education, and economics. All generated items undergo manual ver- ification by two trained annotators following task-specific guidelines, and only those approved by both are included. In UN reports, all paragraphs are explicitly numbered. Each document maintains the same paragraph count across all languages, and each corresponding paragraph contains the same number of sentences. Moreover, paragraphs and sen- tences aligned at the same positions convey equivalent se- mantics across language versions of the same report. All data is drawn from long-form UN reports and verified to ensure cross-lingual consistency across the six languages. Dataset statistics are summarized in table 1, with the de- tailed construction and human verification process provided in Appendix E. 3.2.1. WORD Word-level tasks evaluate whether LLMs can identify pre- cise words and phrases in long-context settings, probing their ability to locate and extract fine-grained evidence. We design two subtasks for evaluation: Single-paragraph Ques- tion Answering (Single-QA) and Multi-paragraph Question Answering (Multi-QA). For data construction, paragraphs are sampled from different positions in a document (begin, middle, end). We use GPT-4 (OpenAI, 2023) to generate question–answering (QA) pairs from these paragraphs, with answers annotated as word-level spans from text. All gener- ated pairs are manually verified for correctness. Single-QA The Single-QA task which constructs word- level QA pairs from individual paragraphs, with questions categorized into three types: numerical, classification, and reference. Paragraphs are sampled from different positions within the document to improve the positional diversity. Multi-QAMulti-QA includes synthesis, comparison, and retrieval subtasks.Each QA pair spans two position- controlled paragraphs from the same document, requiring models to integrate evidence across both sources. 4 MGAL Figure 3. MGAL data generation and human verification pipeline. For each granularity level, candidate instances are generated from aligned UN reports and then checked by human annotators for grounding, semantic consistency, positional correctness, and cross-lingual alignment. We provide additional details on question types for Single- QA and Multi-QA in Appendix E.1.3. 3.2.2. SENTENCE Cloze The sentence-level task is designed to assess whether models can recognize sentence roles and maintain coherence across neighboring sentences. To this end, we introduce a novel sentence-level cloze task inspired by the ‘Insert Text’ question in TOEFL iBT Reading. Specifically, the task requires LLMs to recover a masked sentence using its surrounding context, ensuring both local coherence and global consistency within the document, thereby directly probing sentence-level understanding. To construct the data, we divide each document by position at both the document and paragraph levels (beginning, middle, and end). From these segments, salient sentences are extracted using GPT-4. For each instance, the selected sentence is removed and replaced with a blank. The candidate set consists of the correct sentence along with several distractors generated by GPT-4 that are semantically related but contextually in- appropriate. The model must then identify the option that best restores the passage, maintaining local coherence with neighboring sentences while preserving alignment with the broader discourse. 3.2.3. PARAGRAPH Paragraph Filling Paragraph-level tasks probe whether models can generate coherent, contextually grounded paragraphs that extend beyond the sentence scale. At this granularity, the focus is on both the understanding and generation abilities of LLMs.We design a para- graph filling task to evaluate models’ capability to re- cover missing paragraphs using the surrounding context while maintaining the coherence of the original docu- ment. Specifically, each document is divided into position- based segments, and GPT-4 is used to identify para- graphs with clear functional roles (e.g., topic introduc- tion, contrast, or conclusion) whose content can be in- ferred from neighboring text. For each sample, the se- lected paragraph is removed and replaced with a blank. Unlike sentence-level cloze tasks, this requires free-form generation rather than selecting from candidates. The model is then tasked with generating a paragraph that restores local cohesion, preserves entity and event con- tinuity, and remains aligned with the global discourse theme. 3.2.4. DOCUMENT Our document-level tasks are designed to evaluate holis- tic document comprehension under long-context settings. Specifically, we design two tasks, summarization and trans- lation, to assess LLMs’ ability to perform comprehensive understanding and generation at the document level. Summarization Summarization evaluates whether mod- els can condense long documents into concise yet faithful summaries, testing their ability to capture salient content under extended contexts. We use human-written summaries from UN reports as reliable references. For each document, the original summary is excluded from the input, and the remaining content is given to the model, which is required to generate a faithful and informative summary under long- context conditions. Translation Translation evaluates whether models can accurately preserve meaning across languages in full- document settings, probing both cross-lingual transfer and long-context comprehension. We construct six translation datasets using multilingual counterparts of the same UN doc- uments. In each dataset, one language is fixed as the source and translated into the other five, providing a systematic and balanced setup. 5 MGAL 4. Experiments 4.1. Experimental Setup We evaluate long-context LLMs in a zero-shot setting on MGAL, without any fine-tuning. The evaluation details are provided in Appendix F. 4.1.1. MODELS To evaluate model performance across multilingual, fine- grained long-context tasks, we benchmark 12 LLMs, cov- ering both open-source and proprietary models, with con- text windows exceeding 128k tokens. The set includes GPT-5 (OpenAI, 2025), Claude Sonnet 4 (Anthropic, 2025), Gemini 2.5-Flash (Google DeepMind, 2025), Grok 4 (xAI, 2025), Doubao-Seed-1.6 (Volcengine, 2025; ByteDance Seed Team, 2025), Qwen3-235B-A22B-Instruct (Alibaba Qwen Team, 2025a), Kimi-K2-Instruct (Moonshot AI, 2025), DeepSeek V3.1 (DeepSeek, 2025), GLM-4.5 (Zhipu AI, 2025), Qwen3-30B-A3B-Instruct (Alibaba Qwen Team, 2025b), Mistral-Small-3.2-24B-Instruct (Mistral AI, 2025), and Gemma-3-27B (Google AI, 2025). 4.1.2. EVALUATION METRICS We adopt task-specific metrics aligned with each level of granularity. For Word-level QA and Sentence-level Cloze, we report average accuracy to measure the proportion of predictions that match the gold labels. For Paragraph Filling and Summarization, we use ROUGE-L (Lin, 2004). For Translation, we report BLEU (Papineni et al., 2002) com- puted with sacreBLEU(Post, 2018). The further discussion about mertrics is provided in Appendix F.1. In addition to ROUGE-L, we employ an LLM-as-a-judge for the newly introduced open-ended paragraph-filling task in MGAL to assess quality aspect that are not fully captured by reference- based automatic metrics. Under the same evaluation guide- lines used in the LLM-as-a-judge setting, we instruct our human evaluators to assess outputs in both English and Chi- nese. Details on prompts and implementation are provided in Appendix F.4. 4.2. Main Results Table 3 and Figure 4 summarize the performance (%) of all models on MGAL (see Appendix H for complete results). On word- and sentence-level multiple-choice evaluations, GPT-5 achieves the highest average scores. At coarser gran- ularities, Grok-4 leads on paragraph filling, while GLM-4.5 delivers the best summarization scores and Gemini-2.5-flash excel in translation. The results yield two key observations: (1) Fine vs. coarse performance. Across long-context LLMs, performance is consistently strong on word-level tasks but weak at coarser granularities. Detailed analyses at each granularity are provided in Appendix G.1. (2) Higher- vs. lower-resource languages. In higher- resource languages, large open-source models perform on par with closed-source systems. However, in lower-resource languages, closed-source models retain a clear advantage, with the gap more evident for smaller open-source models. The llm-as-a-judge and human evaluation results in Ap- pendix H.1.2 show that all models underperform on Topic Fidelity and Entity Consistency in paragraph filling. Im- portantly, human and LLM-as-a-judge scores show high Spearman correlation, indicating strong agreement (detailed analysis is provided in Appendix F.5). To examine whether models truly utilize the provided long context rather than relying solely on pre-training priors, we conduct a context ablation study on the Single-QA task. Detailed results are provided in Appendix G.4. 4.3. Positional Results at Different Granularities Prior work shows that long-context models often pay less attention to information placed in the middle of a sequence for question answering and retrieval (Liu et al., 2023a). Similarly, as shown in Figure 5(a) and (c), accuracy in MGAL peaks when the answer lies at the boundaries and is lowest in the middle, with clear effects at the word- and paragraph-level tasks. This boundary preference suggests that models struggle to sustain effective attention over the full sequence. By contrast, most models achieve their highest accu- racy on the sentence-cloze task when the target sentence is positioned in the middle of the document as illus- trated in Figure 5(b). To understand this pattern, we conduct both human and model-based analyses in Ap- pendix G.2.Our findings suggest that, at this gran- ularity, models tend to rely on surface overlaps and simple connectors rather than on the discourse role a sentence plays within the paragraph.Boundary sen- tences at the beginning and end often share framing or summary style and are easily confusable, whereas mid-paragraph sentences typically convey concrete facts tied to nearby entities and references, which better align with such surface cues.Therefore, the middle- position peak should not be interpreted as evidence of robust mid-context comprehension; instead, it likely reflects shallow cue-following and insufficient sensitiv- ity to sentence-level roles. This insight motivates our deeper analysis of local semantic crowding and cue re- liance in Section 5.1. We further disentangle document- level and paragraph-level positions for the sentence-cloze task and provide a model-wise error breakdown in Ap- pendix G.3. 6 MGAL Table 3. Results on word, sentence, paragraph and document tasks. Open-source and proprietary models are separated with a divider, and the top-performing LLM is highlighted in bold, second best are underlined. WordSentenceParagraphDocument ModelSingle-QAMulti-QAClozeFillingSummarizationTranslation GPT-579.7974.9930.6214.4715.4735.30 Claude Sonnet 472.7573.7621.5312.8122.0914.96 Gemini-2.5-flash77.0175.4926.0015.9723.8936.17 Grok 478.4972.6421.5819.9018.9331.05 Doubao-Seed-1.676.6373.6319.4715.2217.086.16 Qwen3-235B-A22B71.8473.6320.2914.7224.279.54 Kimi-K274.4175.17 12.0013.6918.514.46 DeepSeek-V3.175.5672.5017.3114.5426.1528.52 GLM-4.568.4469.3914.7916.15 26.8821.48 Qwen3-30B-A3B68.8671.1416.5413.7222.7410.98 Mistral-Small-3.2-24B69.1770.9132.3515.136.081.63 Gemma-3-27B66.767.8820.3714.675.911.95 Average Performance73.0972.5921.0715.0819.0016.85 4.4. Ablation Studies Effect of Instruction Placement We first investigate whether model performance in the sentence-cloze task is af- fected by the distance between the instruction and the blank. By default, instructions and candidate options are placed at the end of the document, which means blanks appearing near the beginning are far away from the task description. To test positional sensitivity, we relocate the instruction block while keeping all blanks and candidate answers unchanged. We compare three configurations: (1) placing the instruction immediately after the beginning section (Begin), (2) placing it in the middle (Middle), and (3) the default baseline place- ment at the end (End). As shown in Figure 6(a), relocating the instruction improves accuracy in the nearby document region but reduces performance at more distant positions. This indicates that long-context LLMs are sensitive to in- struction placement, with performance improving as the instruction moves closer to the blank. Effect of Option Order In this study, we aim to test whether LLMs in the sentence-cloze task are biased by the order of candidate options rather than by their actual content. Prior analyses suggest that models tend to repeatedly select earlier options, reflecting position bias rather than content evaluation (Pezeshkpour & Hruschka, 2024a; Zheng et al., 2023a). To probe this, we conduct an ablation where the same set of options is presented in different orders. Specifi- cally, we test three conditions: (1) placing high-confusion distractors at the beginning (Begin), (2) placing them in the middle (Middle), and (3) placing them at the end, which corresponds to the default baseline (End). Figure 6(b) shows that accuracy drops when distractors appear earlier (0.16 Begin,0.21Middle) and improves when pushed later (0.24 End). This pattern suggests that models exploit option- position heuristics, i.e., selecting answers based partly on their placement, rather than evaluating all candidates purely by semantic fit, we analyze more in Section 5.2. 5. More Analysis We perform a detailed analysis based on task granularity, position, and multilingual factors, highlighting several novel challenges as well as patterns consistent with prior work. 5.1. New Challenges Revealed by MGAL Preference for Surface Cues Under Local Semantic Crowding At the sentence level, we observe that models tend to overweight surface cues such as explicit connectives (e.g., however, therefore), repeated entities, or overlapping lexical patterns under local semantic crowding, where neigh- boring sentences discuss similar topics and share entities. In such cases, models often underutilize the functional role a sentence plays within the paragraph (e.g., background, ex- planation, or outcome), and instead follow shallow overlaps. For example, given an opening sentence like “The Working Group on the Universal Periodic Review, established in ac- cordance with Human Rights Council resolution 5/1 of 18 June 2007, held its first session from 7 to 18 April 2008.”, the correct next sentence should be “At its 15th meeting, the Working Group adopted the present report on Algeria.” (a development sentence). However, models often prefer 7 MGAL 20 40 60 80 100 Single-QA Multi-QA Cloze Filling Summary Translation (a) 20 40 60 80 100 EN ZH FR ES AR RU (b) Single-QA Multi-QA Cloze Filling Summary Translation Task Type 0 20 40 60 80 100 Performance (c) GPT-5 Claude Sonnet 4 Gemini-2.5-flash Grok 4 doubao-seed-1-6 Qwen3-235B-A22B Kimi-K2 Deepseek V3.1 GLM-4.5 Qwen3-30B-A3B Mistral-Small-3.2-24B Gemma 3-27B EN ZH FR ES AR RU Figure 4. Performance of our evaluated models on MGAL. (a) shows LLM performance across different tasks, (b) presents multilingual performance, and (c) reports task performance across the six official UN languages. GPT-5 achieves the top average across granularity tasks, whereas Gemini-2.5-Flash excels across languages. And models achieve strong results on word-level tasks but exhibit weaker performance at coarser granularities. BeginMiddleEnd 60 70 80 90 Accuracy (a) Word BeginMiddleEnd 0.0 0.1 0.2 0.3 0.4 Accuracy (b) Sentence BeginMiddleEnd 0.1 0.2 Rouge-L (c) Paragraph GPT5 Claude Sonnet 4 Gemini2.5-flash Grok 4 doubao-seed-1-6 Qwen3-235B-A22B Kimi-K2 Deepseek V3.1 GLM-4.5 Qwen3-30B-A3B Mistral-small-3.2-24b Gemma 3-27B Figure 5. Position Performance on MGAL. Performance dips in the middle for word and paragraph tasks but peaks there for sentence-level tasks. a summary-style sentence such as “The review of Algeria was held at the 11th meeting on 14 April 2008.”, which ap- pears plausible due to repeated years and entities but serves wrong discourse role, redundantly restating rather than ad- vancing argument. This tendency shows that models rely heavily on shallow lexical overlap rather than accurately capturing sentence-level discourse roles. Similar shortcom- ings were documented in earlier neural models (Kim et al., 2020; Maekawa et al., 2024), and our findings suggest that long-context LLMs struggle with this challenge. We further examine failure modes with LLM- and human- based annotations, identifying the most frequent error cate- gories in model predictions. We focus on Cloze cases where more than 50% of models fail, using GPT-5 and Gemini-2.5- Flash to analyze the underlying error reasons. The LLMs categorize the errors and provide supporting evidence for their judgments, which are then verified by human checkers. The final results support our finding that under local seman- tic crowding, models tend to over-rely on surface cues while under utilizing discourse-role reasoning, leading to failures in filling role-slots in patterns such as “Givenα⇒ β”, “Althoughα, stillβ”, or “βbecauseα”. More details are provided in Appendix G.2. Gap Between Fluency and Consistency in Generated Outputs In paragraph filling and summarization tasks, model outputs often exhibit high fluency and stylistic align- ment with the source document, yet they frequently neglect reliable contextual grounding. As a result, models may in- troduce unsupported entities or drift away from the intended explanation of the text. For example, models may overlook concrete cues in the surrounding paragraphs and confidently assert that a policy has already been implemented when the source text only outlines actionable recommendations. Similarly, they may hallucinate actors, dates, or institutions absent from the document. While the generated text reads smoothly and appears stylistically consistent, it is factually 8 MGAL BeginMiddleEnd Context Region Begin Middle End Instruction Position 0.380.450.31 0.360.480.33 0.350.450.34 (a) 0.0 0.5 1.0 Begin 0.0 0.5 1.0 Middle 0.0 0.5 1.0 End BeginRandomEnd Confusion Options Position 0.0 0.1 0.1 0.2 0.2 0.2 Average Performance 0.16 0.21 0.24 (b) Figure 6. Ablation studies: (a) Accuracy rises as instruction moves nearby to the blank. (b) Accuracy drops when high-confusion distractors are positioned earlier. inconsistent with the document, highlighting a persistent gap between fluency and consistency in long-context gener- ation. 5.2. Additional Observations Consistent with Previous Studies In addition to the novel findings, our evaluation confirms sev- eral patterns widely reported in earlier studies. Specifically, we observe well-documented tendencies such as reliance on nearby evidence and option reuse under uncertainty. These observations not only align with prior research but also complement our new insights, collectively offering a more comprehensive characterization of long-context model be- havior. Reliance on Nearby EvidenceIn the sentence-cloze task, blank omissions peak when the blank appears near the be- ginning of the document and decline toward the end (App- pendix G.1). This aligns with prior work showing that long-context models underutilize distant evidence due to recency-weighted attention, and that repositioning salient segments closer to the decoding point can mitigate this lim- itation (Peysakhovich & Lerer, 2023). Related studies on positional calibration and controlled placement also demon- strate that model performance is highly sensitive to the rela- tive distance between evidence and the decision anchor (i.e., the instruction and candidate options) (Hsieh et al., 2024b; Xu et al., 2024). In our setup, the anchor is appended af- ter the document, effectively anchoring the decision at the end of the input. As a result, blanks at the beginning cor- respond to the greatest anchor–evidence distance, leading to more omissions. Following the ablation in Section 4.4, we relocate instructions, and find that it reduces omissions at the corresponding position, which further suggests our explanation. Reuse of Options Under Uncertainty When uncertain, models tend to repeatedly select the same options, dispropor- tionately favoring those in earlier positions, such as A and B over C and D in Appendix G.1. This behavior reflects an early-position bias, suggesting that models rely on option- order heuristics and frequency priors rather than ground- ing their choices in item-specific evidence (Pezeshkpour & Hruschka, 2024b; Zheng et al., 2023a). The option order shuffling ablation in Section 4.4 further supports that plac- ing confusion distractors earlier degrades models’ accuracy, while placing them later improves it. 6. Conclusion We introduced MGAL, the first multilingual benchmark for evaluating long-context LLMs across multiple levels of granularity and controlled positional settings. Built from UN reports spanning 8K–128K tokens in six official lan- guages, MGAL covers four linguistic units (word, sentence, paragraph, document) and systematically varies evidence positions, enabling multilingual fine-grained and position- aware assessment. Through an extensive evaluation of 12 LLMs, we find that while models perform relatively well on fine-grained QA, their performance is weak on coarser tasks. We further identify two new challenges specific to long contexts: local semantic crowding, where models over- rely on surface cues instead of recognizing discourse roles, and a fluency–consistency gap, where generated outputs re- main stylistically fluent yet factually misaligned with the source. In addition, MGAL confirms previously observed weaknesses such as recency bias and option-order heuristics. Overall, MGAL highlights the limitations of current LLMs in multilingual long-context comprehension and establishes a rigorous testbed for guiding future work on training objec- tives and evaluation methodologies that emphasize robust, fine-grained, and position-sensitive understanding. 9 MGAL Impact Statement Our benchmark is constructed from publicly available re- ports from the United Nations (UN) Digital Library, used with respect to their public licenses and without altering their substantive meaning; we credit the UN as the original rightsholder. We screened all instances to avoid including sensitive personal data and directed human annotators to exclude content with offensive language or social biases. While the potential for encountering sensitive content from the original sources persists, this risk is mitigated as the benchmark’s primary focus is on evaluating the long-context capabilities of LLMs, not their social biases. To further pro- tect privacy, released artifacts contain only minimum text necessary for evaluation, with full documents remaining at their original sources. Furthermore, our methodology considers environmental impact by evaluating existing mod- els without additional pre-training and reporting settings. Finally, we will comply with any takedown or correction requests from rights holders or affected parties and will promptly update dataset documentation if legal interpreta- tions change. Acknowledgement This work is supported by the Youth S&&T Talent Support Programme of GDSTA (SKXRC2025462), NSFC (Category C) fund code 62506149 and Lingnan University StartUp Grant fund code: 103684. References Alibaba Qwen Team.Qwen3-235b-a22b-instruct- 2507.https://huggingface.co/Qwen/ Qwen3-235B-A22B-Instruct-2507, 2025a. AlibabaQwenTeam.Qwen3-30b-a3b-instruct- 2507.https://huggingface.co/Qwen/ Qwen3-30B-A3B-Instruct-2507, 2025b. Anthropic.Introducing claude 4 (opus 4 and son- net 4).https://w.anthropic.com/news/ claude-4, 2025. Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. Longbench: A bilingual, multi- task benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023.URLhttps:// arxiv.org/abs/2308.14508. ACL 2024 (version v2, 2024-06-19). Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 1: Long Papers), p. 3119–3137, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10.18653/v1/2024.acl-long.172. URLhttps: //aclanthology.org/2024.acl-long.172/. ByteDance Seed Team. Introduction to seed1.6 model tech- nology.https://seed.bytedance.com/en/ research, 2025. Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via position interpolation. arXiv preprint arXiv:2306.15595, 2023. Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. LongLoRA: Efficient fine-tuning of long-context large language models. In ICLR, 2024. URLhttps: //openreview.net/forum?id=6PmJoRfdaK. DeepSeek.Deepseek-v3.1 release.https:// api-docs.deepseek.com/news/news250821 , 2025. Google AI. Gemma 3 model overview.https://ai. google.dev/gemma/docs/core, 2025. Google DeepMind.Gemini 2.5 flash.https:// deepmind.google/models/gemini/flash/, 2025. Hooper, C. R. C., Kim, S., Mohammadzadeh, H., Mah- eswaran, M., Zhao, S., Paik, J., Mahoney, M. W., Keutzer, K., and Gholami, A. Squeezed attention: Accelerating long context length llm inference. In Proceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), p. 32631–32652, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1568. URLhttps:// aclanthology.org/2025.acl-long.1568/. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What’s the real context size of your long-context language mod- els?arXiv preprint arXiv:2404.06654, 2024a. doi: 10.48550/arXiv.2404.06654. URLhttps://arxiv. org/abs/2404.06654. Hsieh, C.-Y. et al.Found in the middle:Cal- ibrating positional attention bias in long-context language models.In Findings of the Associ- ation for Computational Linguistics:ACL 2024, 2024b.URLhttps://aclanthology.org/ 2024.findings-acl.890.pdf. 10 MGAL Kim, N., Feng, S., Gunasekara, C., and Lastras, L. Im- plicit discourse relation classification: We need to talk about evaluation. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 5404–5414, Online, July 2020. Association for Com- putational Linguistics. doi: 10.18653/v1/2020.acl-main. 480. URLhttps://aclanthology.org/2020. acl-main.480/. Kim, S., Suk, J., Cho, J. Y., Longpre, S., Kim, C., Yoon, D., Son, G., Cho, Y., Shafayat, S., Baek, J., Park, S. H., Hwang, H., Jo, J., Cho, H., Shin, H., Lee, S., Oh, H., Lee, N., Ho, N., Joo, S. J., Ko, M., Lee, Y., Chae, H., Shin, J., Jang, J., Ye, S., Lin, B. Y., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models. arXiv preprint arXiv:2406.05761, 2025a. doi: 10.48550/arXiv.2406. 05761. NAACL 2025 (Main Conference), version 2 (25 Mar 2025). Kim, Y., Russell, J., Karpinska, M., and Iyyer, M. One ruler to measure them all: Benchmarking multilin- gual long-context language models.arXiv preprint arXiv:2503.01996, 2025b. URLhttps://arxiv. org/abs/2503.01996. Kwan, W.-C., Zeng, X., Wang, Y., Sun, Y., Li, L., Jiang, Y., Shang, L., Liu, Q., and Wong, K.-F. M4le: A multi- ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), p. 15568–15592, Bangkok, Thailand, Au- gust 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.832.URLhttps: //aclanthology.org/2024.acl-long.832/. Li, K., Zhang, L., Jiang, Y., Xie, P., Huang, F., Wang, S., and Cheng, M.Lara: Benchmarking retrieval- augmented generation and long-context llms – no sil- ver bullet for lc or rag routing. In Proceedings of the 42nd International Conference on Machine Learning, 2025. URLhttps://icml.c/virtual/2025/ poster/46069. ICML 2025 Poster. Li, M., Zhang, S., Zhang, T., Duan, H., Liu, Y., and Chen, K. Needlebench: Evaluating llm retrieval and reasoning across varying information densities. arXiv preprint arXiv:2407.11963, 2024a. URLhttps:// arxiv.org/abs/2407.11963. Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024b. Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, p. 74–81, Barcelona, Spain, 2004. Association for Compu- tational Linguistics. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts, 2023a. URLhttps: //arxiv.org/abs/2307.03172. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C.G-eval:Nlg evaluation using gpt-4 with better human alignment.In EMNLP 2023, 2023b.URLhttps://aclanthology.org/ 2023.emnlp-main.153/. Maekawa, A., Hirao, T., Kamigaito, H., and Okumura, M. Can we obtain significant success in RST discourse parsing by using large language models? In Graham, Y. and Purver, M. (eds.), Proceedings of the 18th Con- ference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2803–2815, St. Julian’s, Malta, March 2024. Associa- tion for Computational Linguistics. doi: 10.18653/v1/ 2024.eacl-long.171. URLhttps://aclanthology. org/2024.eacl-long.171/. Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, p. 1906–1919, Online, 2020. Association for Computational Linguis- tics. doi: 10.18653/v1/2020.acl-main.173. URLhttps: //aclanthology.org/2020.acl-main.173. Mistral AI.Models overview — mistral small 3.2(2506).https://docs.mistral. ai/getting-started/models/models_ overview/, 2025. Moonshot AI. Kimi k2: Open agentic intelligence.https: //moonshotai.github.io/Kimi-K2/, 2025. Novikova, J., Du ˇ sek, O., Cercas Curry, A., and Rieser, V. Why we need new evaluation metrics for NLG. In Pro- ceedings of EMNLP, p. 2241–2252, 2017. OpenAI. GPT-4 technical report, 2023. URLhttps: //arxiv.org/abs/2303.08774. OpenAI. Introducing gpt-5.https://openai.com/ index/introducing-gpt-5/, 2025. 11 MGAL Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, p. 311–318, Philadelphia, PA, USA, 2002. Association for Computa- tional Linguistics. doi: 10.3115/1073083.1073135. Peysakhovich, A. and Lerer, A. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427, 2023.URLhttps:// arxiv.org/abs/2310.01427. Pezeshkpour, P. and Hruschka, E. Large language models sensitivity to the order of options in multiple-choice ques- tions. In Findings of the Association for Computational Linguistics: NAACL 2024, p. 2006–2017, Mexico City, Mexico, 2024a. doi: 10.18653/v1/2024.findings-naacl. 130. Pezeshkpour, P. and Hruschka, E. Large language mod- els sensitivity to the order of options in multiple- choice questions.In Findings of the Associa- tion for Computational Linguistics:NAACL 2024, 2024b.URLhttps://aclanthology.org/ 2024.findings-naacl.130/. Post, M. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, p. 186–191, Belgium, Brussels, October 2018. Association for Computational Linguistics.URLhttps://w.aclweb.org/ anthology/W18-6319. Reiter, E. A structured review of the validity of bleu. Com- putational Linguistics, 44(3):393–401, 2018. Shaham, U., Ivgi, M., Efrat, A., Berant, J., and Levy, O. Zeroscrolls: A zero-shot benchmark for long text under- standing. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, p. 7977–7989, Singa- pore, 2023. ACL. doi: 10.18653/v1/2023.findings-emnlp. 536. URLhttps://aclanthology.org/2023. findings-emnlp.536/. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/j.neucom.2023.127063. Volcengine. Doubao foundation model.https://w. volcengine.com/product/doubao, 2025. xAI. Grok 4. https://x.ai/news/grok-4, 2025. Xu, X. et al. Stress-testing long-context language models with controlled contextualized qa. In NeurIPS Datasets and Benchmarks, 2024. URLhttps://arxiv.org/ abs/2407.16695. Yuan, T., Ning, X., Zhou, D., Yang, Z., Li, S., Zhuang, M., Tan, Z., Yao, Z., Lin, D., Li, B., Dai, G., Yan, S., and Wang, Y. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k.arXiv preprint arXiv:2402.05136, 2024.URLhttps:// arxiv.org/abs/2402.05136. Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. Large language models are not robust multiple choice selectors. arXiv preprint arXiv:2309.03882, 2023a. URL https://arxiv.org/abs/2309.03882. Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM as a judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track, 2023b. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a- judge with mt-bench and chatbot arena. In NeurIPS 2023 Datasets and Benchmarks Track, 2023c. URLhttps: //arxiv.org/abs/2306.05685. Zhipu AI. Glm-4.5: Reasoning, coding, and agentic abilities. https://z.ai/blog/glm-4.5, 2025. 12 MGAL A. Reproducibility Statement We ensure reproducibility in three aspects: (1) Dataset. All source documents are drawn from the publicly accessible UN Digital Library. The data curation and annotation pipeline, including sampling strategies for each granularity, is fully documented, and the processed benchmark will be released upon publication with task splits. (2) Evaluation. We provide the exact prompts, scoring scripts, and task definitions for all evaluations. (3) Models and code. For open-source models, we will release configuration files and inference scripts upon publication; for API-based models, we document request formats and parameters. All preprocessing, annotation, and evaluation code will be released upon publication. B. The Use of Large Language Models To enhance the clarity and readability of this manuscript, we utilized Large Language Models as assistive tools. First, we employed them for language refinement, including grammatical correction, stylistic improvements, and the rephrasing of complex sentences. Second, we leveraged LLMs to support our data construction pipeline. Specifically, LLMs assisted in initial annotation and cleaning of each granularity data set derived from UN documents; these data subsequently underwent final validation by human checkers. The authors maintained full editorial control throughout this process. All substantive contributions, including ideas, methodology, analyses, and conclusions, are exclusive work. C. Limitations Although MGAL broadens the evaluation scope for fine-grained long-context understanding, several limitations remain. First, standard automatic, reference-based metrics (e.g., ROUGE-L for summarization; token-level accuracy for QA; BLEU for translation) are coarse proxies for human judgment and are sensitive to surface form, paraphrase, and length, which can underestimate or mischaracterize quality (Novikova et al., 2017; Reiter, 2018; Lin, 2004). Second, using LLM-as-a-judge improves semantic sensitivity but incurs nontrivial runtime cost and exhibits known biases (e.g., position and verbosity), requiring careful prompt design, calibration, and robustness checks (Zheng et al., 2023b). Third, while our goal is to evaluate long-context modeling independently of instruction following, real-world task formulations inevitably intertwine the two, so measured performance may partly reflect instruction adherence rather than pure context-modeling ability. D. UN Language Scope and Relative Resource Imbalance MGAL covers the six official languages of the United Nations: Arabic, Chinese, English, French, Russian, and Spanish. All six languages are high-resource languages by conventional standards, with large speaker populations: English (1.45B), Chinese/Mandarin (1.13B), Spanish (559M), French (310M), Arabic (274M), and Russian (255M). Our analysis on MGAL concerns relative resource imbalance within this high-resource subset, highlighting cross-language performance differences without conflating them with genuinely low-resource languages. E. Dataset Construction Pipeline This section describes the dataset construction pipeline for MGAL. We source documents from the United Nations Digital Library and apply manual annotations without altering substantive content. MGAL is built through a unified annotation and curation pipeline tailored to each granularity level. All data is constructed from a large pool of approximately 70,000 UN reports, from which we identify roughly 4,000 structurally suitable reports for benchmark construction. We sample documents across the 8K–128K token range and ensure that each task contains the same number of query–response pairs for all six languages. The resulting source documents cover diverse UN policy and governance domains, including international development, human rights, peace and security, humanitarian affairs, public health, environment, education, and economics. In UN reports, all paragraphs are explicitly numbered. Each document maintains the same paragraph count across all languages, and each corresponding paragraph contains the same number of sentences. Moreover, paragraphs and sentences aligned at the same positions convey equivalent semantics across language versions of the same report. 13 MGAL Each generated instance is checked by two trained annotators for correctness, contextual grounding, positional validity, and cross-lingual consistency, and only instances approved by both annotators are retained. Across tasks, the accepted/generated counts are 420/582 for Single-QA, 420/604 for Multi-QA, 420/800 for Cloze, 420/500 for Paragraph Filling, and 420/800 for Summarization. Rejected QA items mainly involve semantic ambiguity, shortcut cues that allow answering without true contextual grounding, or conflation of distinct paragraph meanings. Rejected cloze and paragraph-filling items are primarily due to unbalanced position or role selection, while rejected summarization items often arise from formatting inconsistencies in UN reports that cause extraction or cross-lingual alignment errors. E.1. Word QA For the referenced single- or multi-paragraph inputs, the LLM is prompted to generate a question, an answer, and the corresponding evidence from each specified paragraph according to the category-specific instructions. The human checkers evaluate each generated question–answer pair following a two-stage guideline. First, they verify whether the provided evidence is grounded in the input paragraph, rather than being hallucinated by the LLM. Second, the checkers assess the quality of the question–answer pairs. They begin by examining the semantic correctness and clarity of the generated question, ensuring that it is unambiguous and aligned with the predefined categories. They then evaluate whether the answer is supported by the evidence within the paragraph and whether the question–answer pair is consistent and factually correct. E.1.1. SINGLE QA We partition each document’s main body into three position-indexed regions: begin, middle, and end. From each region, we robustly extract complete paragraphs. One paragraph is uniformly sampled per region. For each sampled paragraph, we prompt an LLM to generate a single question that includes cited paragraph-bounded evidence, and requires a word or phrase answer sourced from the original paragraph. We categorize three generated question templates as: (1) Numerical for exact counts and quantitative mentions; (2) Classification for topic, sentiment, or functional category; and (3) Reference for pronoun or coreference resolution with local inference. Each question is formatted as a four-choice question with a single-answer option. The options distractors are constructed to be semantically plausible under partial reading yet inconsistent with the anchored evidence. At evaluation time, models are given the full document and the question that instructs them to answer. Figure 7. Prompt template for the Numerical type question generation. 14 MGAL Figure 8. Prompt template for the Reference type question generation. Figure 9. Prompt template for the Classification type question generation. 15 MGAL E.1.2. MULTI QA We adopt the same position segmentation as Single-QA, partitioning each document’s main body into begin, middle, and end regions. We sample the anchor paragraph pairs for one question-answer pair to test the models’ ability to integrate contextual evidence. The sampled paragraph pairs’ positions are both within regions (eg, Begin and Begin) and across regions (eg, Begin and Middle). Each question references exactly two paragraphs. Before composing the question, the LLM produces the selected paragraph evidence summaries for each of the two selected paragraphs. These summaries anchor subsequent question wording and option construction. We categorize three cross- paragraph question templates: (1) Comparison that contrasts methods, perspectives, or claims across the two paragraphs; (2) Retrieval that locates where a specific fact resides with fixed options: first paragraph, second paragraph, both, or neither; and (3) Synthesis that derives a concept or conclusion that emerges only when the two paragraphs are considered together. At evaluation time, models are given the full document, and the instructions for the model to answer with respect to those paragraphs questions. Figure 10. Prompt template for the Synthesis type question generation. 16 MGAL Figure 11. Prompt template for the Retrieval type question generation. Figure 12. Prompt template for the Comparision type question generation. 17 MGAL E.1.3. INSTANTIATIONS OF DIFFERENT QA TASK SAMPLE Task Name Single-QA Type Numerical Paragraph_id 84 Question: Which year in paragraph 84 represents the culmination of UNDP's efforts to address changing gender norms? Options: A.2025 B.2030 C. 2040 D. 2020 Finally,aschanginggendernormsandstereotypestakes considerabletime,UNDPneedslonger-term,multisectoral interventionswithpredictablefinancing.Tosignificantly contributetotheachievementofthe2030Agenda,UNDPmust workmorecloselywithpartneragenciesacrosstheUnited Nationssystemtoadvocatefornewandsustainedfundingmodels. Source_text: Figure 13. Numerical Type Sample Question. Task Name Single-QA Type Classification Paragraph_id 80 Question: What conclusion can be drawn from paragraph 80's emphasis on actions for improvement? Options: A. observations B. conclusions C. suggestions D. recommendations TheSpecialRapporteurrecommendsthattheGovernmentofEritrea: Putanimmediateendtohumanrightsviolationsdocumentedbythe SpecialRapporteurandthecommissionofinquiryonhumanrights inEritrea,includingtheongoingviolationshighlightedinthepresent report;.....Investigatetheallegationsofhumanrightsand humanitarianlawviolationsbyEritreanarmedforcesinthecontext oftheconflictinEthiopiasinceNovember2020andtakemeasures tobringperpetratorstojustice;(l)Refrainfromsubjecting Indigenouscommunitiestodiscriminatorypractices,including arbitraryarrests,andrespectandprotecttheirtraditionalwaysoflife andmeansoflivelihood; Source_text: Figure 14. Classification Type Sample Question. 18 MGAL Task Name Single-QA Type Reference Paragraph_id 22 Question: What role does paragraph 22 suggest the ACO takes in relation to the Anti-Fraud program? Options: A. directive B. adversarial C. indifferent D. supportive TheIEAScontinuedtosupportworkledbymanagementonthe Anti-FraudprogramatUN-Women.Amongothers,itprepareda lessons-learnedmemorandumonred-flags/potentialfraudrisks relatedtomanagingimplementingpartners.Italsoinitiatedalessons learned/integrityreviewofvehiclemanagement,facilitatedreporting onpotentialallegationstoOIOS,andcontinuedtosupportOIOSon itsreportsandreferrals.IASalsoassistedmanagementinpreparing thefraudassessmentandfraudpreventiontraining.TheACOis pleasedtoseetheincreasedfocusonanti-fraudawarenessraising andtrainingtoimprovethelow-levelmaturityratingofthiscapacity asreportedinthe2021ACOreporttotheEB. Source_text: Figure 15. Reference Type Sample Question. Task Name Multi-QA Type Comparison Paragraph_id 88,92 Question: Whichcross-paragraphconceptin88,92illustratesthenecessarysteps towardsenhancingchildprotection? Options: A. prevention B. implementation C. negotiation D. observation 88.GivencontinuedgraveviolationsagainstchildreninSomalia,I calluponallresponsibleUnitedNationsbodiestoensurethatthe protectionofchildrenisaddressedasapriorityintheongoingpeace processchildprotectionadvisersshouldbeincorporatedinthe UnitedNationsPoliticalOfficeforSomalia,andinanyfuture deploymentofaUnitedNationspeacekeepingoperation,toserveas interlocutorswithchildprotectionactors. 92.MySpecialRepresentativeforChildrenandArmedConflictis requestedtoundertakeamissiontoSomaliainthenearfutureto assessfirst-handthesituationforchildrenandtheimplementationof therecommendationsinmyreportsandthoseoftheSecurityCouncil WorkingGrouponChildrenandArmedConflict." Source_text: Figure 16. Comparision Type Sample Question. 19 MGAL Task Name Multi-QA Type Synthesis Paragraph_id 16,18 Question: Which synthesis element across paragraphs 16, 18 demonstrates their interconnected approach to achieving goals? Options: A. innovation B. independence C. collaboration D. competition 16.Atitssecondregularsessionof2015,CEBendorsedtheglobal initiativeon .Preparedthroughaninter- agencyconsultativeprocessundertheleadershipoftheInternational LabourOrganization...onthepromotionofsustained,inclusiveand sustainableeconomicgrowth,fullandproductiveemploymentand decentworkforall. 18.Oneinitiativethatgainedparticularmomentumin2015wasthe UnitedNationssystemdatacatalogueproject,theaimofwhichisto maximizethebenefitsofmakingUnitedNationssystemdataopen andaccessibletothepublicandotherkeystakeholders,...Asofthe endof2015,thecataloguecomprisednearly4,000datasets.An initialpubliclaunchofthedatacataloguein2016isforeseen.I. Promotingsystem-widepreparationforandfollow-uptoUnited Nationsconferencesandsummits. Source_text: Figure 17. Synthesis Type Sample Question. Task Name Multi-QA Type Retrieval Paragraph_id 4,28 Question: ComparingParagraph4andParagraph28,andnotingthatParagraph28 focusesonUNsystemclimatecoordination,inwhichparagraph (Paragraph4orParagraph28)istheexplicitstatementfoundthatthe GeneralAssemblywelcomedeffortstoincreaseeffectivenessand accountability? Options: A. Paragraph 4 B. Paragraph 28 C. Neither D. Both paragraphs 4.Inthesameresolution,theGeneralAssemblywelcomedeffortsto increasetheeffectiveness,accountabilityandcredibilityoftheUnited Nationssystem,includingbyreducingadministrativeandprocedural burdens....TheAssemblymadeadditionalrequestsrelatingtoa coordinatedapproachtomultilingualism,themainstreamingof supportforSouth-Southcooperationandthecontinuationofdialogue betweenCEBandMemberStates. 28.Atthesession,thecollectiveengagementoftheUnitedNations systemwas climate-relatedknowledge......Enhancingtheeffectiveness, efficiency,coherenceandimpactofUnitedNationsoperational activitiesfordevelopment. Source_text: Figure 18. Retrieval Type Sample Question. 20 MGAL E.2. Cloze We partition each document’s main body into three position-indexed regions, and identify paragraphs and select key sentences stratified by paragraph-level position within each region by LLM. For documents of 8K–16K tokens, we select nine target sentences to cover both document-level and paragraph-level positions; for documents over 16K tokens, we select twelve target sentences. For each selected sentence, we use an LLM to generate confusion sentences that are locally topical yet inconsistent with the exact entailment required at the blanked location. We then assemble a fixed-size option set: 10 choices (9 gold + 1 confusion) for 8K–16K-token documents and 14 choices (12 gold + 2 confusion) for documents over 16K tokens. For chapter-level inputs divided into beginning, middle, and ending sections at both the paragraph and document levels, the LLM is prompted to identify key sentences (e.g., core or turning points) at the specified positions. Based on these selected sentences, the LLM then generates distractors that alter certain details while remaining topically consistent with the original content. To verify quality, human evaluators first visualize each LLM-selected sentence in the original UN PDF. They check the positional correctness of the extracted sentences using both programmatic and manual methods. Since UN reports provide explicit paragraph numbering and sentences are segmented by periods, the evaluators can automatically confirm the sentence index and its exact location within the document. After confirming the position, the evaluators examine the semantic role of each selected sentence by inspecting the visualized sentences in the PDF to determine whether it functions as a core, transitional, or summary statement. They also verify that the selected sentences are evenly distributed across positional categories within the document to avoid selected bias. For distractor validation, the evaluators assess topic fidelity and entity consistency to ensure that each distractor remains aligned with the referenced sentence while introducing incorrect or altered details relative to the ground truth. They then compare every distractor with all correct answer options in the cloze to guarantee a clear and unambiguous distinction between the correct choice and the distractors. In the evaluation, the input is the full document with the target sentence replaced by a blank marker at its original location, the corresponding set of options, and an instruction that prompts LLM to select the sentence that best aligns with the local context. E.3. Paragraph Filling We partition each document into the Begin, Middle, and End regions. Within each region, we use LLM to extract key paragraphs. For 8K–16K-token documents, we select two target paragraphs per region; for documents over 16K tokens, we select three per region, ensuring balanced coverage across regions. Human evaluators then examine each LLM-selected paragraph by visualizing it directly in the original UN PDF. Using this visualization, the evaluators verify the semantic role of the paragraph, assessing whether it functions as a core, transitional, or summary element, and additionally confirm that the selected paragraphs are evenly distributed across positions within each document to avoid positional bias. At evaluation time, we remove the paragraph and insert a blank marker at its original location. The model receives the document replaced by blank marker and an instruction to reconstruct the missing paragraphs using surrounding context. E.4. Context For the summarization task, we programmatically extract the original summary from each document and perform cross- lingual verification by matching it against the corresponding summary section in the UN PDF across all six language versions. 21 MGAL F. Experimental Setup This section describes the experimental setup for MGAL, covering output controls, context management, tool-use constraints for agentic models, the judging protocol for generative tasks, and the evaluation prompts. We locally deploy Qwen3-30B- A3B-Instruct (Alibaba Qwen Team, 2025b), Mistral-Small-3.2-24B-Instruct (Mistral AI, 2025), and Gemma-3-27B (Google AI, 2025), while the remaining larger models are accessed via API. F.1. Interpreting Metrics Across Granularities Tasks at different granularities in MGAL adopt different evaluation metrics: accuracy for word- and sentence-level QA, ROUGE-L for paragraph filling, and ROUGE-L or BLEU for document-level summarization and translation. Since these metrics are not directly comparable in absolute value, we emphasize relative performance trends rather than raw scores when analyzing results across granularities. Specifically, we interpret higher accuracy on word- and sentence-level QA as evidence that fine-grained comprehension is easier for current LLMs, while substantially lower ROUGE-L scores on paragraph filling and summarization indicate persistent difficulty in generating coherent and factually grounded outputs at larger units. This rationale allows us to compare trends across tasks without conflating metric scales, and ensures that our conclusions about fine- versus coarse-grained performance reflect relative difficulty rather than absolute score magnitudes. F.2. Maximum Output Length Controls To prevent non-stop generation, we cap the model’s maximum output length per task/dataset. For classification-style tasks (e.g., Cloze, Single-QA, Multi-QA), we enforce single-token or single-word answers via explicit instructions and post-hoc normalization. F.3. Tool-use Control for Agent-based LLMs For models with agent capabilities (e.g., tool calls, browsing, retrieval), we disable external tools and explicitly instruct the model to rely solely on the provided text. This prevents access to outside knowledge sources or caches and yields a fair cross-model comparison under identical evidence exposure. F.4. LLM-as-a-Judge for Paragraph-filling For paragraph filling, we adopt LLM-as-a-judge for evaluation. The judge is given the previous and next paragraphs of the paragraphs that need to be generated, the gold reference, and the system output, and is instructed to ground every decision in the provided text rather than using external knowledge or chain-of-thought rationales. Motivated by (Kim et al., 2025a), we perform scoring on five dimensions: Topic Fidelity, Local Coherence, Entity Consistency, Instruction Following, and Format Compliance. Each dimension is scored as an integer from 0 to 20, and the overall score is the sum 0 to 100. We prompt LLMs’ judgment should be justified with brief, text-grounded reasons that cite evidence from the original document and include a short quote from a previous paragraph, ground truth, and the next paragraph. In Topic Fidelity, the judge verifies that the generation preserves the central topic and key claims of the reference without improperly narrowing or expanding its scope, and without introducing unsupported assertions. Local Coherence assesses logical and temporal continuity between the generation and its neighbors, including transitions, pronoun and tense alignment, and causal or contrastive links. Entity Consistency checks that subjects, events, times, locations, organizations, numbers, and coreference match the reference and the constraints implied by the surrounding context, with no invented attributes. Instruction Following evaluates adherence to the task constraints (e.g., style, perspective, length, prohibited content) as specified to the judge. Format Compliance evaluates whether the output conforms exactly to the required schema or template, including structure, headings, bulleting, tags, field ordering, and length limits. We use advanced closed-source models GPT-5, Gemini-2.5, and Grok-4 as judges and report the average evaluation score. For the LLM-as-a-judge setting in the paragraph-filling task, the judges evaluate each generated paragraph using the ground-truth paragraph and its surrounding context. The detailed inputs and prompts are provided in Appendix F.4. Each LLM judge outputs both a score and explanatory evidence. Human evaluators then verify the score and the accompanying evidence against the generated paragraph and the ground truth to ensure that the LLM’s judgment is faithful, well-grounded, 22 MGAL and free from hallucinated evidence. We report the mean score using GPT-5, Gemini-2.5-flash and Grok 4 as a judge and note robustness checks by humans. The exact prompt and field-level instructions used by the judge are shown in the Figure 19. Figure 19. The prompt for LLM-as-a-judge. F.5. Human Evaluation for Paragraph-filling Under the same evaluation guidelines used in the LLM-as-a-judge setting, we instructed our human evaluators to assess outputs in both English and Chinese. We calculated the relationships between different indicators in the Average row using the Pearson Correlation Coefficient, comparing the score sequences of all indicators across the LLM-as-a-judge and human evaluation tables. From the average scores of all models, there is a strong positive correlation (r = 0.871) between the LLM-as-a-judge indicators and the corresponding human assessment indicators. Consistently, both evaluations reveal the same fluency–consistency gap that models achieve high fluency scores but exhibit low consistency. The complete human evaluation results are provided in the Appendix H.1.2. F.6. Human Involvement and Quality Assurance Human involvement played a central role in the annotation verification and quality assurance process. All human annotators were recruited internally and enumerated through authorship. Specifically, we employed eight human checkers, each of whom independently participated in the full annotation verification pipeline. No crowdsourcing platforms or external contractors were used. Each annotator contributed approximately 90 human-hours to quality assurance, corresponding to a workload of 3 hours per day over a 30-day period. In total, this resulted in 720 human-hours dedicated exclusively to manual checking, verification, and consistency validation. Human review was applied at multiple stages of the pipeline. Annotators were responsible for validating extracted content, verifying alignment with source documents, and identifying annotation errors or inconsistencies. All instances were manually reviewed rather than sampled. When potential issues were identified, annotations were either corrected or rejected following discussion among the annotators. Disagreements among annotators occurred occasionally, primarily in ambiguous cases involving boundary decisions or semantic interpretation. Such disagreements were resolved through discussion and consensus among the eight contributors. Questions or annotations that failed to meet quality standards after review were rejected rather than force-labeled, ensuring that only high-confidence instances were retained in the final dataset. 23 MGAL F.7. Evaluation Prompts We standardize task prompts across languages and granularities; all prompts are multilingual with explicit answer-format constraints. Figure 20. An example prompt for the QA task. Figure 21. An example prompt for the Cloze task. Figure 22. An example prompt for the Paragraph filling task. 24 MGAL Figure 23. An example prompt for the Translation task. Figure 24. An example prompt for the Summarization task. 25 MGAL G. Comprehensive Analysis G.1. Granularity WordIn the Single-QA task, models exhibit uniformly high performance. Multi-Paragraph QA also achieves consistently high accuracy, for several models it even surpasses the single-paragraph setting, indicating that LLMs can aggregate cues across paragraphs to more reliably select the correct answer. Sentence Blank omissions are a major source of error. Rates peak when the blank appears near the beginning of the document and decline monotonically toward the end shown in Figure 25. In our evaluation setting, the instruction and sentence candidate options are appended after the document, placing the decision anchor at the end. Prior work shows that long-context language models underweight distant evidence due to recency-weighted attention, and that moving salient segments closer to the decoding point mitigates this effect (Peysakhovich & Lerer, 2023). Moreover, studies on positional calibration and controlled placement indicate that performance is sensitive to where the relevant evidence sits relative to the anchor (Hsieh et al., 2024b; Xu et al., 2024). Therefore, Early blanks maximize the anchor–evidence distance that makes LLMs error. Following the ablation in Section 4.4, we relocate instructions, and find that it significantly reduces omissions at the corresponding position, which further confirm our explanation. beginmiddleend begin middle end 12611267 665748 1128557 GPT-5 beginmiddleend begin middle end 155145140 168169168 14211996 Claude Sonnet 4 beginmiddleend begin middle end 1099740 543933 1098350 Gemini-2.5-flash beginmiddleend begin middle end 164167152 170160156 130114101 Grok 4 beginmiddleend begin middle end 14212385 909798 140128105 Doubao-Seed-1.6 beginmiddleend begin middle end 574522 231918 363519 Qwen3-235B-A22B beginmiddleend begin middle end 126121103 124126116 144152145 Kimi-K2 beginmiddleend begin middle end 11410247 516046 958681 DeepSeek-V3.1 beginmiddleend begin middle end 1157261 686255 858969 GLM-4.5 beginmiddleend begin middle end 464838 424334 253422 Qwen3-30B-A3B beginmiddleend begin middle end 276774 438780 74124132 Gemma 3-27B beginmiddleend begin middle end 102925 162525 101526 Mistral-Small-3.2-24B 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 0 25 50 75 100 125 150 Figure 25. Blank omissions heatmap by position across models. Most sentence-level errors in the generated options arise from local semantic crowding around the blank. Under local semantic crowding, where neighboring sentences share topics and entities, models preferentially follow surface cues(e.g., connectives or repeated entities) over the discourse role that the sentence plays in the surrounding argument, such as background, explanatory, or outcome. For example, given an opening sentence like “The Working Group on the Universal Periodic Review, established in accordance with Human Rights Council resolution 5/1 of 18 June 2007, held its first session from 7 to 18 April 2008.”, the correct next sentence should be “At its 15th meeting held on 16 April 2008, the Working Group adopted the present report on Algeria.” (a development sentence). However, models often prefer a summary-style sentence such as “The review of Algeria was held at the 11th meeting on 14 April 2008.”, which appears plausible due to repeated years and entities but serves the wrong discourse role, redundantly restating rather than advancing the argument. The position choices heatmap 26 MGAL with the ground truth and model predictions are in Figure 26. We further analyze failure modes using LLM and human annotations to identify the most common error categories in model predictions in Appendix G.2. begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 733142221503 46422723779 5958410881789 32522011654 04225520155 1881924111710 57953322842 151424626529 892014850558 GPT-5 begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 53418151112221117 23133232818202014 22484373021241520 791843028121417 451424642152014 12111718323172121 14182011171153340 914161871629733 3322249101240314 Claude Sonnet 4 begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 325243211120 39532732659 61671094522109 23721971064 01330429825 29426305111011 881427622647 46612331927 732153654444 Gemini-2.5-flash begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 730341110101844 2673631161617910 30494232019321514 412333417111813 661833241101710 12121519292132127 1317211181423145 91813968281036 19152910101147263 Grok 4 begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 521202011273 39330633738 67713181132088 03302112736 30125317394 157202346614 96922602536 461034134432 15132045746415 Doubao-Seed-1.6 begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 642372238241111 556532157191514 856832498401816 7141983838161213 181643941121415 10121544402142215 12133417161565061 10151761716521140 18183010101255539 Qwen3-235B-A22B begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 543412243131411 47737301010201515 67483291911342223 13121533630221414 10171327128231820 10201637277311626 17162910171043937 201418125941834 261635881238417 Kimi-K2 begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 739311866171110 486482874151012 65695291610311516 412112454081513 63837839191812 6121232405202825 1214237121795051 161718610933952 432921119855378 DeepSeek-V3.1 begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 2403429910141614 50643261315181516 81654241715371217 8151964633191713 791540749142513 10121340478272726 22172813142495256 14182612918441053 3427251181158498 GLM-4.5 begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 9404221412361912 52550251513251225 6164532119481916 8162155632132617 3101040840202320 11121728406242328 13183421102297160 132322181316461256 42292916101548371 Qwen3-30B-A3B begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 5644536109351413 51541261412282517 19456311325512524 6141174149272026 1162540242222827 8191523404314626 1911308101575280 2218251381440745 42172113111436382 Gemma 3-27B begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end begin_begin begin_middle begin_end middle_begin middle_middle middle_end end_begin end_middle end_end 356502086191311 4844622111120157 47477341525341419 910123463271412 514938747122415 9111626356292527 1614328121294773 1416238101245961 3612301161459375 Mistral-Small-3.2-24B 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 Figure 26. Position choices heatmap with the ground truth and model predictions. Beyond aggregate accuracy, we observe systematic option reuse illustrated in Figure 27. Within a same cloze task, some models repeatedly select the same choice, with selections concentrated on early position: A and B more than C and D, and far more than later options. This pattern indicates an early-position bias and suggests reliance on option-frequency priors (Pezeshkpour & Hruschka, 2024b; Zheng et al., 2023a) rather than item-specific evidence under uncertainty. An ablation that shuffles option order to test in Section4.4. Paragraph Across models, generated paragraphs show a systematic bias relative to the reference: they mimic the document’s surface register while underweighting diagnostic cues in adjacent paragraphs, yielding unsupported entity assertions and drift from the source’s explanatory trajectory. Faithful reconstruction relies less on global paraphrase and more on local entailment with neighboring paragraphs and selective aggregation of proximal evidence(Maynez et al., 2020). We also observe brittle local coherence: entity states and temporal anchors are not consistently propagated across adjacent context, producing continuations that read fluently yet remain locally ungrounded. For example, models may overlook concrete cues in the surrounding paragraphs and confidently assert that a policy has already been adopted when the source text only outlines actionable recommendations; they may also hallucinate on generating actors, dates, or institutions that are not present in the document. The generated text is stylistically consistent and reads smoothly, but it is factually inconsistent with the document. This illustrates the gap between fluency and consistency in long-context generation. Document Models underperform on MGAL summarization: expert summaries serve as scoping prefaces that set the mandate, framing, and outline-level topic coverage while constraining claims to verifiable source content. By contrast, model outputs drift into substantive synthesis, reorder themes, enumerate cases, and import extra textual detail. The generated summaries assert trends or policy shifts without in-context support, resulting in abstractive drift and hallucinations. These patterns explain why LLM-generated summaries, though superficially fluent, align poorly with expert references, revealing a persistent gap between fluency and consistency in generated outputs. 27 MGAL ABCDEFGHIJKLMNORTUV Choice Letters 0 50 100 150 200 250 Total Repetition Count 218 208 143 136 89 98 89 43 101 89 66 14 22 12 19 1 13 1 3 Figure 27. Repetition letter distribution. In translation, most models show a higher-resource advantage, scoring consistently higher on English and other higher- resource languages than on lower-resource ones. Moreover, the GPT-5 and Gemini-2.5-flash exhibit strong performance in the Translation task among all models evaluated, significantly outperforming the average translation score of 16.85. Cross-granularity synthesis and implications From a layered analysis across four coherence-aligned linguistic granu- larities, we find that long-context LLMs perform well in word-level, but struggle in coarser-grained tasks. Long-context LLMs capture global semantics but struggle to recognize and exploit fine-grained discourse roles. While errors at finer units accumulate and propagate to sentences, paragraphs, and documents, yielding outputs that are superficially fluent yet weakly consistent: intra-paragraph organization is loose, and sentence-level commitments are unstable, ultimately depressing overall performance. G.2. Generated Options Analysis We use GPT-5 and Gemini-2.5 to categorize sentence-level option-selection errors in Cloze with a single shared, reference- aware prompt. Each judge reads the local paragraph context and the two candidate sentences: the paragraph immediately before the target sentence; the current paragraph containing the correct sentence; and the paragraph immediately after, then returns a strict verdict. The taxonomy has three mutually exclusive classes: • Discourse-role underuse (M1). In local semantic crowding, the sentence chosen from the model does not fill the reasoning role implied by the surrounding chain (e.g., “Givenα⇒ β,” “Althoughα, stillβ,” “βbecauseα”). It functions as a parallel, rephrasing, or background statement, whereas the gold sentence uniquely completes the required role. •Surface-cue and heuristic overuse (M2). The model-picked sentence is better explained by surface signals—explicit connectives (e.g., “however,” “therefore”), repeated entities, high lexical overlap, length, or format than by context fit or entailment from the local context. • Context or knowledge deficit (M3). The model-picked sentence conflicts with constraints evident, such as topic, entities, time, causal direction, or presupposes knowledge unsupported by the given context. We also record secondary facets to aid diagnosis: S1 discourse-role misidentification (reserved for M1), S2 coreference drift, S3 connective or overlap lure, S4 temporal or ordinal misread, S5 negation or scope error, S6 locally coherent but globally incoherent, S7 world-knowledge gap, and S8 format or fluency bias. 28 MGAL Each case receives exactly one main class; secondary tags may be added. In a single pass, the judge prioritizes M1 if the gold uniquely fills the role, defaults to M2 when the choice is best explained by surface cues, and otherwise assigns M3. Ambiguity is resolved conservatively with reduced confidence. The results are illustrated in Figure 28. We categorize the Cloze errors using the LLMs judging protocol and find a highly skewed distribution in which M1 dominates, especially in the S3 subset. This distribution shows that the dominant failure pattern is not simply that models are attracted by surface overlap in isolation. Rather, models often choose a locally plausible sentence that shares topic words, entities, or connectives with the surrounding context, but fails to satisfy the discourse role required by the blank within the paragraph. In crowded local contexts, neighboring candidate sentences can be semantically close at the lexical or entity level, while differing in their discourse function, such as background, elaboration, contrast, evidence, or conclusion. This suggests our finding that under local semantic crowding—adjacent sentences sharing topics and entities—models over-follow surface cues (connectives, lexical overlap, formatting) while underusing discourse-role reasoning, failing to fill the role-slot in patterns such as “Givenα⇒ β”, “Althoughα, stillβ”, or “βbecauseα”. This cue-following induces systematic mislabeling that mid-paragraph background or result sentences are treated as openings, and post-development summaries are misread as first sentences, because neighboring sentences share topic and register, forming deceptively plausible decoys. S1S2S3S4S5S6S7S8 Secondary Tags M1 M2 M3 Main Class 6440532150805 01440901015 002001600 Co-occurrence: Main Class × Secondary Tag 0 100 200 300 400 500 Count Figure 28. Sentence analysis results heatmap. As shown in Table??, we select cases in which the majority-selected wrong sentence comes from the same paragraph and is topically or lexically close to the gold sentence, but occupies a different discourse role. These cases are therefore intended to illustrate local semantic crowding rather than arbitrary distractor errors. 29 MGAL Case 1: Regional human rights initiatives Paragraph with blank.The extent of cooperation among government departments, as well as among Governments, academia, national human rights institutions and NGOs in any given country appears particularly significant.[#]Examples mentioned include the Inter-American Institute of Human Rights, the European Union, the Organization for Security and Cooperation in Europe, the Council of Europe and the Secretariat of the Pacific Community. Gold sentence.Similarly, there seem to be an increasing number of regional initiatives, which States can both contribute to and benefit from, thereby fostering a valuable exchange of experiences and cross-fertilization of ideas. Majority-selected wrong sentence.Examples mentioned include the Inter-American Institute of Human Rights, the European Union, the Organization for Security and Cooperation in Europe, the Council of Europe and the Secretariat of the Pacific Community. Error analysis. The wrong sentence is locally plausible because it lists concrete regional organizations related to the same human-rights cooperation topic. However, the blank requires a general transition from national cooperation to regional initiatives before the examples are introduced. Models therefore confuse a concrete example sentence with the higher-level framing sentence required by the paragraph. Case 2: Gender dimension in HIV policies Paragraph with blank.[#]Member States have continued to use two approaches to address the gender dimension in HIV policies and action plans. The UNDP/World Bank/UNAIDS HIV Mainstreaming Programme supported Namibia, Botswana, Lesotho and Swaziland in strengthening their development planning processes to better integrate and implement HIV and gender priorities. Gold sentence.A joint UNAIDS/Global Fund to Fight AIDS, Tuberculosis and Malaria mission to China focused on the priority actions to be taken in the area of women and girls in China’s national HIV strategy 2011–2015. Majority-selected wrong sentence. The UNDP/World Bank/UNAIDS HIV Mainstreaming Programme supported Namibia, Botswana, Lesotho and Swaziland in strengthening their development planning processes to better integrate and implement HIV and gender priorities. Error analysis.The wrong sentence is topically close because it also concerns HIV, gender priorities, UNAIDS-related support, and national policy planning. However, the paragraph first requires an opening example before summarizing the two approaches and then introducing another programme example. The error shows that models confuse a later supporting example with the opening example needed to establish the paragraph sequence. Case 3: Belarus electoral process Paragraph with blank. The systemic shortcomings, such as the absence of a free system of registration for any movement, including political parties, the lack of equal access to the media by all political streams, the absence of transparency in turn out and vote counting and registration of voters and the ongoing harassment and discrimination of unwelcome candidates, render the entire electoral process not compatible with the concept of elections that are respectful of human rights and therefore pose the question of the purpose of such a process. He reiterates his readiness to work with the Government and continues to offer his support to civil society. The absence of real changes in the practices of the State apparatus and in the legal framework, notwithstanding a ready-to-implement list of recommendations, demonstrates a lack of political will to adhere to rights that are universally recognized. [#] The Special Rapporteur regrets that Belarus did not take into account the numerous recommendations made by the OSCE Office for Democratic Institutions and Human Rights, the United Nations human rights mechanisms and by himself on several occasions to tackle the systemic deficiencies that underpin the electoral process in Belarus and hinder the exercise by citizens of their basic freedoms. Gold sentence. He will continue to request an official visit to the country during 2017. Majority-selected wrong sentence.He reiterates his readiness to work with the Government and continues to offer his support to civil society. Error analysis. The wrong sentence shares the same subject, cooperative stance, and Special Rapporteur-centered discourse, making it locally plausible. However, it has already functioned as a general statement of readiness. The blank instead requires a forward-looking action about requesting an official visit. Models confuse a repeated stance with the next concrete action in the sequence of recommendations. Table 4. Qualitative examples of sentence selection errors in long-context modeling. 30 MGAL Table 5. Error breakdown for the sentence-cloze task. Omission denotes failing to fill the blank with any correct sentence; Position Error denotes selecting a correct sentence from the wrong document position; Distractor denotes selecting an incorrect distractor option. ModelCorrectOmissionPosition ErrorDistractor GPT-552.8917.4321.518.17 Gemini-2.5-Flash52.8714.6622.3910.08 Doubao-Seed-1.647.2524.0621.726.96 Qwen3-235B-A22B43.926.5440.379.17 Mistral-Small-3.2-24B40.574.3240.3414.77 Qwen3-30B-A3B37.767.9344.959.36 DeepSeek-V3.137.7516.2837.868.11 GLM-4.534.7916.1443.385.70 Grok-429.1731.3731.967.50 Kimi-K227.2727.6238.087.03 Claude Sonnet 425.4731.0834.508.95 Gemma-3-27B24.0916.9044.8314.18 G.3. Two-Level Position Analysis and Cloze Error Breakdown Two-level position indexing.Figure 4 in the main paper reports document-level positional results across tasks, where each instance is grouped by whether the target evidence appears at the beginning, middle, or end of the document. For word-level QA and paragraph-level filling, position is defined only at the document level. The two-level position design is introduced specifically for the sentence-cloze task, where each removed target sentence is indexed by both its document-level position and its sentence position within the paragraph. The construction details are described in Appendix F and illustrated in Figure 6. To further disentangle these two positional dimensions for cloze, Figure 26 presents a heatmap of ground-truth versus predicted positions, and Table 7 reports cloze results at both the document and paragraph levels. Cloze error breakdown. To better understand model failures in the sentence-cloze task, we further analyze the error breakdown across models. We divide predictions into four categories: Correct, Omission, Position Error, and Distractor. Omission denotes failing to fill the blank with any correct sentence; Position Error denotes selecting a correct sentence from the wrong position in the document; and Distractor denotes selecting an incorrect distractor option. As shown in Table 5, distractor error rates remain relatively stable across models, generally within 5–15%. We do not observe a pattern in which non-GPT models disproportionately select GPT-4-generated distractors. Instead, the dominant error types are position errors and omissions. For example, several open-source models, including Qwen3-30B-A3B, Gemma-3-27B, and GLM-4.5, exhibit high position-error rates, while models such as Grok-4, Claude Sonnet 4, and Kimi-K2 show relatively high omission rates. This suggests that sentence-cloze failures are mainly associated with positional sensitivity and local semantic crowding, rather than systematic bias introduced by GPT-4-generated distractors. Taken together, the two-level position analysis and the error breakdown indicate that the cloze task primarily challenges models’ ability to select a sentence that is not only semantically plausible but also positionally and discourse-functionally appropriate. These results support our interpretation that model failures are driven more by positional bias and local semantic crowding than by distractor-generation artifacts. G.4. Context Memorization Analysis To examine whether models truly use the provided long-context rather than relying solely on pre-training priors, we conduct a context ablation on the Single-QA task. Specifically, we compare two settings: (1) the standard setting with access to the full document context, and (2) a no-context setting where the document is withheld and the model must answer using only internal knowledge. We adopt the same evaluation setup as the Single-QA task and report results averaged over 12 LLMs. To investigate the model’s reliance on parametric knowledge versus contextual information, we conducted an context memorization study. The model average accuracy drops sharply from 0.73 (with context) to 0.31 (without context), demonstrating that performance is largely attributable to information drawn from the input rather than memorization. We find that the model successfully answers questions related to well-known public facts, such as the ”2030” timeline for malaria targets and the specific designation of a widely-known UN document. However, performance collapses when 31 MGAL questions require information specific to the text. For instance, the model failed to identify ”2016” as the starting point for a strategic guidance initiative, defaulting instead to the more commonly known ”2030 Agenda”. Similarly, without the text explicitly stating ”educational and financial obstacles,” the model incorrectly generalized the number of obstacles as ”many” instead of the correct answer ”two” in the original document. The findings show that questions grounded in commonsense or general policy remain moderately answerable without context, while those requiring document-specific details degrade substantially. 32 MGAL H. Full Results on MGAL Table 6. Model Performance Across All Tasks and Languages TaskModelEnglishChineseSpanishFrenchRussianArabic Single-QA GPT581.2578.3380.8382.2977.5078.54 Claude Sonnet 472.7578.9570.0074.3866.8871.88 Gemini-2.5-flash79.1675.4278.5478.5473.75 76.67 Grok 478.7576.2578.7579.1762.577.29 Doubao-Seed-1.678.5475.8378.7581.4668.7576.46 Qwen3-235B-A22B82.2953.7577.7177.7166.0473.54 Kimi-K278.9573.3378.5473.1367.0875.42 DeepSeek-V3.179.7974.7976.0479.7966.0476.88 GLM-4.576.0468.9675.4278.7561.2553.33 Qwen3-30B-A3B73.3353.5475.4278.7567.0070.63 Mistral-Small-3.2-24B74.7960.4273.3373.7561.2571.46 Gemma-3-27B72.2966.8867.9267.9259.7965.42 Multi-QA GPT582.7156.8862.560.4254.1767.29 Claude Sonnet 479.3853.7556.2557.9247.0860.21 Gemini-2.5-flash77.0856.6759.5862.2953.7559.58 Grok 478.5455.0037.7154.7939.5840.21 Doubao-Seed-1.672.2957.0856.6758.9645.8352.92 Qwen3-235B-A22B82.2944.5845.8359.5856.6750.83 Kimi-K283.3349.5853.7552.7146.0449.58 DeepSeek-V3.178.9650.2152.0845.0053.7553.75 GLM-4.572.0848.9656.6750.2161.2570.83 Qwen3-30B-A3B78.5444.5852.0852.0847.0852.5 Mistral-Small-3.2-24B80.2173.1375.4272.9253.3370.42 Gemma-3-27B78.3371.2567.0868.1359.3863.13 Cloze GPT552.8926.39 26.1426.5625.3426.38 Claude Sonnet 425.4719.9120.4820.8221.6120.88 Gemini-2.5-flash52.8720.7020.0120.1020.8021.49 Grok 429.1719.9320.2920.8917.6621.54 Doubao-Seed-1.647.2514.5914.2413.7813.8113.13 Qwen3-235B-A22B43.9216.4916.0615.8713.4715.98 Kimi-K227.278.339.329.327.789.95 DeepSeek-V3.137.7513.8314.5914.039.4014.28 GLM-4.534.7933.1812.2711.259.4810.83 Qwen3-30B-A3B37.7611.3612.8412.6911.5513.04 Mistral-Small-3.2-24B40.5721.9042.9042.576.7839.35 Gemma-3-27B24.0913.5225.4325.3310.1123.73 Filling GPT517.8414.3917.9218.793.4614.43 Claude Sonnet 420.2315.233.6320.601.6815.49 Gemini-2.5-flash20.0516.7019.6520.762.9015.76 Grok 428.8215.8825.8524.234.1020.51 Doubao-Seed-1.619.6716.3618.7519.222.7614.59 Qwen3-235B-A22B18.4315.0018.6318.983.6013.67 Kimi-K217.2613.0317.0918.403.1313.23 DeepSeek-V3.118.2315.2816.7219.403.7713.86 GLM-4.520.95 16.6119.4920.763.6415.44 Qwen3-30B-A3B17.5413.8917.3317.923.3312.28 Mistral-Small-3.2-24B18.7215.9819.1119.953.2513.76 Gemma-3-27B17.2015.0818.5719.254.1413.76 Summary GPT513.1716.7315.9216.2916.0414.64 Claude Sonnet 416.5627.3620.8221.6520.4325.72 Gemini-2.5-flash17.6433.0622.6623.5120.9525.55 Grok 416.4223.9422.9222.712.6114.97 Doubao-Seed-1.616.5129.518.4318.7411.078.21 Qwen3-235B-A22B19.1335.53 22.2423.1521.3924.17 Kimi-K216.4124.0219.3620.5314.7815.93 DeepSeek-V3.122.1436.6124.05 24.4723.1226.53 GLM-4.521.1335.3426.628.7121.1928.3 Qwen3-30B-A3B17.6330.3223.5623.4417.3324.16 Mistral-Small-3.2-24B21.8 2.694.763.371.552.28 Gemma-3-27B21.482.94.032.911.572.54 Translation GPT536.5737.1636.6532.0333.9835.44 Claude Sonnet 421.5823.079.8310.9712.4911.86 Gemini-2.5-flash38.9231.9739.6535.3234.2536.92 Grok 435.5239.3235.8532.5211.0532.04 Doubao-Seed-1.65.884.5512.825.931.646.14 Qwen3-235B-A22B24.2513.517.234.373.044.86 Kimi-K24.475.365.475.781.624.09 DeepSeek-V3.117.4636.9635.2330.6117.4033.46 GLM-4.525.3423.1925.3222.9910.4121.65 Qwen3-30B-A3B29.9814.328.216.903.942.52 Mistral-Small-3.2-24B1.120.592.632.910.142.38 Gemma-3-27B4.660.281.972.120.771.89 33 MGAL Table 7. Model Performance on Multi-QA Task at Document-Level.Rows indicate the position of the first paragraph and columns indicate the position of the second, with each cell representing the performance for the corresponding positional pair (e.g., ’Begin-Middle’). TaskModelBegin MiddleEnd Single-QA GPT580.4278.8580.21 Claude Sonnet 475.4477.4371.75 Gemini-2.5-flash77.475.9477.71 Grok 477.1978.7580.31 Doubao-Seed-1.676.8876.6276.88 Qwen3-235B-A22B72.469.2771.35 Kimi-K276.6774.1772.4 DeepSeek-V3.176.6774.2775.73 GLM-4.572.0864.3866.88 Qwen3-30B-A3B71.7567.2970.18 Mistral-Small-3.2-24B 72.1769.5170.26 Gemma-3-27B70.0463.7568.03 Table 8. Model Performance in Single-QA Task at Document-Level.Accuracy on the Single-QA task, categorized by the position (Begin, Middle, End) of the single answer-bearing paragraph. TaskPosition ModelBegin MiddleEnd Multi-QA Begin GPT576.5478.7984.18 Claude Sonnet 477.8872.8679.52 Gemini-2.5-flash75.1277.870.69 Grok 472.7966.0869.71 Doubao-Seed-1.666.9357.4474.54 Qwen3-235B-A22B70.2172.1575.02 Kimi-K276.58 68.0484.04 DeepSeek-V3.173.4370.6579.63 GLM-4.572.6770.6879.34 Qwen3-30B-A3B67.2568.5672.12 Mistral-Small-3.2-24B 54.7372.2473.02 Gemma-3-27B64.0876.5271.21 Middle GPT578.7979.6485.23 Claude Sonnet 472.8670.3582.19 Gemini-2.5-flash77.8080.2282.93 Grok 466.0866.7872.12 Doubao-Seed-1.667.4470.7569.68 Qwen3-235B-A22B72.1573.3174.12 Kimi-K268.0479.3180.41 DeepSeek-V3.170.6571.5668.79 GLM-4.570.6873.5975.68 Qwen3-30B-A3B68.5669.2364.35 Mistral-Small-3.2-24B 72.2467.2866.94 Gemma-3-27B76.5275.6171.4 End GPT584.1885.2374.19 Claude Sonnet 479.5282.1977.47 Gemini-2.5-flash70.6982.93 71.39 Grok 469.7172.1266.5 Doubao-Seed-1.674.5469.6867.83 Qwen3-235B-A22B75.0274.1265.07 Kimi-K284.04 80.4175.62 DeepSeek-V3.179.6368.7963.25 GLM-4.579.3475.6861.25 Qwen3-30B-A3B72.1264.3557.98 Mistral-Small-3.2-24B 73.0266.9465.43 Gemma-3-27B71.2171.471.15 34 MGAL Table 9. Model performance on the Cloze task, evaluated by hierarchical position. We measure performance at the beginning (Begin), middle (Middle), and end (End) of the document, and for each of these sections, we further evaluate at the beginning, middle, and end of the target paragraph. Scores are reported in accuracy.Best results are marked inbold, second best results are underlined. TaskDocument-level Model Paragraph-level Begin MiddleEnd Cloze Begin GPT530.1721.3425.79 Claude Sonnet 427.1519.7217.50 Gemini-2.5-flash26.7424.5421.73 Grok 428.5222.8320.92 Doubao-Seed-1.630.3121.3214.77 Qwen3-235B-A22B27.6921.4318.80 Kimi-K222.0114.8711.93 DeepSeek-V3.126.6518.5215.48 GLM-4.524.3718.1113.23 Qwen3-30B-A3B30.0318.1514.12 Mistral-Small-3.2-24B 38.1130.8526.30 Gemma-3-27B31.9722.1917.78 Middle GPT535.7934.5531.07 Claude Sonnet 423.9722.2820.90 Gemini-2.5-flash31.1028.3127.50 Grok 419.7918.4317.24 Doubao-Seed-1.622.7919.3617.68 Qwen3-235B-A22B22.4820.8720.38 Kimi-K212.5511.0210.09 DeepSeek-V3.121.0319.1017.11 GLM-4.517.4515.5112.22 Qwen3-30B-A3B16.1015.5714.86 Mistral-Small-3.2-24B 40.6037.9633.97 Gemma-3-27B27.1520.0117.49 End GPT534.3530.9928.84 Claude Sonnet 422.9722.7819.75 Gemini-2.5-flash27.5626.2023.21 Grok 42.3122.6620.79 Doubao-Seed-1.619.1815.9514.18 Qwen3-235B-A22B18.8816.4915.84 Kimi-K28.778.177.26 DeepSeek-V3.113.7113.0411.65 GLM-4.511.3010.899.80 Qwen3-30B-A3B13.7912.8311.72 Mistral-Small-3.2-24B 29.21 27.4323.33 Gemma-3-27B18.6513.9412.51 35 MGAL H.1. Granularity-wise Performance H.1.1. WORD-LEVEL SUBTASKS Table 11. Performance of various models on different types of word-level question-answering tasks.On word-level question-answering tasks, GPT-5 achieves the highest scores in Numerical, Reference, and Synthesis. Gemini-2.5-flash leads in Classification, Qwen3-30B- A3B excels in Comparison, and Kimi-K2 delivers the best performance in Retrieval.Best results are marked inbold, second best results are underlined. Single-QAMulti-QA Granularity-TasksNumericalClassificationReferenceComparisonRetrievalSynthesis GPT-573.4480.7385.3162.6676.5287.01 Claude Sonnet 462.8980.6481.0566.7773.8482.71 Gemini-2.5-flash67.2981.5682.1963.968.7284.64 Grok-472.580.0083.7548.2379.6667.77 Doubao-Seed-1.669.1278.7582.555.9165.7374.77 Qwen3-235B-A22B57.3876.3875.2565.3481.7982.29 Kimi-K263.8579.0680.3161.1583.6583.35 DeepSeek V3.163.2379.1784.2759.3373.2579.87 GLM-4.561.3378.2480.3560.9765.7677.08 Qwen3-30B-A3B60.1178.9377.8668.1259.6581.32 Mistral-Small-3.2-24B55.2377.2679.4564.1772.2972.29 Gemma 3-27B52.9375.8173.0956.2467.5977.48 H.1.2. PARAGRAPH FILLING USING LLM-AS-A-JUDGE AND HUMAN EVALUATION Table 12. Model performance across different metrics based on llm-as-a-judge in six UN languages. Headers are abbreviated as follows: Topic Fid. (Topic Fidelity), Local Coh. (Local Coherence), Entity Cons. (Entity Consistency), Instr. Foll. (Instruction Following), and Format Comp. (Format Compliance).Across various performance metrics, Gemini-2.5-flash achieves the highest overall score, also leading in Entity Consistency and Instruction Following. Grok-4 delivers the best Topic Fidelity, while Qwen3-235B-A22B is the strongest in Local Coherence.Best results are marked inbold, second best results are underlined. ModelTopic Fid.Local Coh.Entity Cons.Instr. Foll.Format Comp.Overall GPT-54.5810.514.395.3619.9644.81 Claude Sonnet 44.8312.035.245.7419.8147.65 Gemini-2.5-flash5.9011.276.376.9119.9350.38 Grok-45.9411.835.896.7019.8650.22 Doubao-Seed-1.65.0811.295.115.9019.9247.29 Qwen3-235B-A22B4.6712.074.915.7420.0047.39 Kimi-K24.1410.513.945.1920.0043.79 DeepSeek V3.14.469.644.625.7120.0044.43 GLM-4.54.6111.034.825.6419.9546.05 Qwen3-30B-A3B3.7910.344.595.1119.98 43.80 Mistral-Small-3.2-24B3.7910.344.595.1119.9843.80 Gemma 3-27B4.129.794.755.3819.9644.00 Average4.6810.904.935.7019.9546.08 36 MGAL Table 13. Model performance across different metrics based on human evaluation in Chinese and English. Headers are abbreviated as follows: Topic Fid. (Topic Fidelity), Local Coh. (Local Coherence), Entity Cons. (Entity Consistency), Instr. Foll. (Instruction Following), and Format Comp. (Format Compliance).Across various performance metrics, DeepSeek V3.1 achieves the highest overall score, also leading in Instruction Following. Qwen3-235B-A22B delivers the best Topic Fidelity, Local Coherence and Entity Consistency, while GPT-5 also lead in Entity Consistency and Qwen3-30B-A3B also delivers the best Local Coherence. Grok-4 and Mistral-Small-3.2-24B are the strongest in Local Coherence. Best results are marked inbold, second best results are underlined. ModelTopic Fid.Local Coh.Entity Cons.Instr. Foll.Format Comp.Overall GPT-55.3212.736.786.5119.8851.22 Claude Sonnet 45.2612.426.546.6319.8450.69 Gemini-2.5-flash5.9311.896.326.2819.8550.27 Grok-45.1411.965.875.7420.0048.71 Doubao-Seed-1.65.0711.785.215.4719.93 47.46 Qwen3-235B-A22B6.1512.866.785.8319.8351.54 Kimi-K24.7212.364.625.8619.8247.38 DeepSeek V3.16.0912.815.986.8519.8951.62 GLM-4.54.5310.734.265.1119.7944.42 Qwen3-30B-A3B4.6712.864.284.8919.8446.54 Mistral-Small-3.2-24B3.8610.393.885.1220.0043.25 Gemma 3-27B4.0710.433.945.2619.8743.57 Average5.0011.705.395.7219.9048.02 37 MGAL H.1.3. MODEL PERFORMANCE ON TRANSLATION TASKS USING BLEU Table 14. Translation tasks on English to Other languages. En-Ch means English translate to Chinese.In translation tasks, Gemini-2.5-flash delivers the best performance for English to Chinese, Spanish, French, and Russian, while Grok-4 achieves the highest score for English to Arabic.Best results are marked inbold, second best results are underlined. ModelEn-Zh En-Es En-Fr En-Ru En-Ar GPT-526.73 57.7151.44 40.146.82 Claude Sonnet 419.37 32.61 30.12 23.312.48 Gemini-2.5-flash29.36 59.58 54.05 44.636.99 Grok-428.17 53.38 51.7636.607.68 Doubao-Seed-1.612.865.805.134.201.41 Qwen3-235B-A22B16.53 40.35 38.21 21.015.12 Kimi-K212.494.002.982.230.65 DeepSeek V3.118.33 17.56 6.8237.287.34 GLM-4.526.41 35.25 37.41 22.944.68 Qwen3-30B-A3B21.83 47.30 42.03 32.706.05 Mistral-Small-3.2-24B 3.840.110.110.550.98 Gemma3-27B9.690.080.0613.330.12 Table 15. Translation tasks on Chinese to Other languages. Zh-En means Chinese translate to English.In Chinese to other language translations, Grok-4 achieves the highest scores for English (Zh-En) and Arabic (Zh-Ar). Gemini-2.5-flash leads in translations to Spanish (Zh-Es) and Russian (Zh-Ru), while GPT-5 delivers the best performance for French (Zh-Fr).Best results are marked inbold, second best results are underlined. ModelZh-En Zh-Es Zh-Fr Zh-Ru Zh-Ar GPT-557.3042.6746.9131.567.35 Claude Sonnet 444.6524.0030.1211.863.34 Gemini-2.5-flash16.9550.6845.11 39.387.73 Grok-462.4046.8645.0233.408.89 Doubao-Seed-1.613.002.344.621.830.96 Qwen3-235B-A22B14.9615.9331.303.521.81 Kimi-K210.226.658.660.920.37 DeepSeek V3.156.5946.3039.6935.37 6.87 GLM-4.528.6933.4731.4519.582.74 Qwen3-30B-A3B13.1629.4324.003.041.96 Mistral-Small-3.2-24B1.000.400.400.530.61 Gemma3-27B0.490.180.160.300.27 Table 16. Translation tasks on Spanish to Other languages. Es-En means Spanish translate to English.For Spanish to other language translations, GPT-5 is the top performer for Chinese (Es-Zh) and French (Es-Fr). Gemini-2.5-flash leads in translations to Russian (Es-Ru) and Arabic (Es-Ar), while Grok-4 achieves the best score for English (Es-En).Best results are marked inbold, second best results are underlined. ModelEs-En Es-Zh Es-Fr Es-Ru Es-Ar GPT-525.20 33.23 57.56 36.32 7.86 Claude Sonnet 416.19 15.79 11.92 10.05 0.87 Gemini-2.5-flash35.91 31.13 52.11 48.58 8.88 Grok-438.02 28.62 49.08 38.158.73 Doubao-Seed-1.611.89 11.89 4.52 11.36 0.39 Qwen3-235B-A22B8.815.103.372.971.58 Kimi-K214.08 12.32 0.931.150.39 DeepSeek V3.125.52 32.7056.4631.11 7.29 GLM-4.517.22 27.86 46.32 16.01 7.53 Qwen3-30B-A3B13.46 13.46 4.512.600.47 Mistral-Small-3.2-24B 1.08 11.27 0.540.641.01 Gemma3-27B0.14 10.13 0.040.070.22 Table 17. Translation tasks on French to Other languages. Fr-En means French translate to English.Based on the provided table* for French translation tasks, Grok-4 achieves the highest score for English (Fr-En) and GPT-5 leads for Spanish (Fr-Es). Gemini-2.5-flash is the top performer for Chinese (Fr-Zh) and Russian (Fr-Ru), while DeepSeek V3.1 delivers the best results for Arabic (Fr-Ar).Best results are marked inbold, second best results are underlined. ModelFr-En Fr-Zh Fr-Es Fr-Ru Fr-Ar GPT-556.5926.4157.2135.977.06 Claude Sonnet 421.3215.247.584.380.64 Gemini-2.5-flash56.81 35.8651.3843.9210.29 Grok-459.4425.350.0635.409.04 Doubao-Seed-1.611.846.254.6840.830.48 Qwen3-235B-A22B13.9913.444.752.921.07 Kimi-K211.2511.682.791.220.42 DeepSeek V3.156.5933.1743.0731.6111.70 GLM-4.537.7319.4721.0639.069.28 Qwen3-30B-A3B12.1420.934.412.481.10 Mistral-Small-3.2-24B0.5411.270.090.410.86 Gemma3-27B0.209.370.070.100.11 Table 18. Translation tasks on Russian to Other languages. Ru-En means Russian translate to English.For the Russian translation tasks, Grok-4 achieves the highest scores for English (Ru-En), Spanish (Ru-Es), and Arabic (Ru-Ar). Gemini-2.5-flash leads in Chinese (Ru-Zh) and French (Ru-Fr).Best results are marked inbold, second best results are underlined. ModelRu-En Ru-Zh Ru-Es Ru-Fr Ru-Ar GPT-552.65 23.70 47.94 46.126.78 Claude Sonnet 430.42 16.805.504.781.78 Gemini-2.5-flash47.46 29.23 49.3950.727.80 Grok-456.28 27.7849.93 44.489.51 Doubao-Seed-1.68.8114.363.123.640.79 Qwen3-235B-A22B11.407.341.912.641.00 Kimi-K210.069.093.121.130.18 DeepSeek V3.155.4726.64 41.32 34.739.14 GLM-4.536.70 20.25 25.70 23.472.14 Qwen3-30B-A3B9.436.486.252.440.74 Mistral-Small-3.2-24B 0.519.160.780.911.12 Gemma3-27B0.169.030.080.040.12 Table 19. Translation tasks on Arabic to Other languages. Ar-En means Arabic translate to English.On Arabic translation tasks, GPT-5 leads in translations to English (Ar-En) and Spanish (Ar-Es), while Gemini-2.5-flash achieves the highest scores for Chinese (Ar-Zh), French (Ar-Fr), and Russian (Ar-Ru).Best results are marked inbold, second best results are underlined. ModelAr-En Ar-Zh Ar-Es Ar-Fr Ar-Ru GPT-549.4119.6332.9739.0128.89 Claude Sonnet 444.045.194.713.524.97 Gemini-2.5-flash35.4222.8732.5746.9533.43 Grok-411.8312.4711.1310.829.01 Doubao-Seed-1.60.813.981.181.650.6 Qwen3-235B-A22B7.633.800.551.851.38 Kimi-K24.562.620.110.440.35 DeepSeek V3.142.1213.954.579.6516.7 GLM-4.526.4115.140.776.062.81 Qwen3-30B-A3B7.396.940.772.771.82 Mistral-Small-3.2-24B0.280.190.100.010.10 Gemma3-27B0.183.110.240.010.31 38 MGAL H.1.4. MODEL PERFORMANCE ON TRANSLATION TASKS USING CHRF++ Table 20. Translation tasks on English to Other languages. En-Ch means English trans- late to Chinese.In translation tasks, Gemini-2.5-flash delivers the best performance for English to Chinese, Spanish, French, and Russian, while Grok-4 achieves the highest score for English to Arabic.Best results are marked inbold, second best results are underlined. ModelEn-ZhEn-EsEn-FrEn-RuEn-Ar GPT-546.8382.5179.2572.273.49 Claude Sonnet 437.4759.7058.3350.362.19 Gemini-2.5-flash53.1383.2181.5874.793.43 Grok-447.3379.7380.37 62.533.92 Doubao-Seed-1.627.8023.3425.6722.831.72 Qwen3-235B-A22B45.0966.2367.4842.992.65 Kimi-K227.8822.3720.8517.541.41 DeepSeek V3.137.2740.5128.5463.673.78 GLM-4.548.9461.5564.9346.262.90 Qwen3-30B-A3B42.3470.2073.3266.722.55 Mistral-Small-3.2-24B2.547.186.925.761.43 Gemma3-27B7.666.756.095.901.33 Table 21. Translation tasks on Chinese to Other languages. Zh-En means Chinese translate to English.In Chinese to other language translations, Grok-4 achieves the highest scores for English (Zh-En) and Arabic (Zh-Ar). Gemini-2.5-flash leads in translations to Spanish (Zh-Es) and Russian (Zh-Ru), while GPT-5 delivers the best performance for French (Zh-Fr).Best results are marked inbold, second best results are underlined. ModelZh-EnZh-EsZh-FrZh-RuZh-Ar GPT-584.1471.4878.3962.443.40 Claude Sonnet 463.0561.4253.5335.086.76 Gemini-2.5-flash43.1779.8577.0072.023.98 Grok-485.6477.14 77.4258.489.31 Doubao-Seed-1.637.4722.6829.9518.231.24 Qwen3-235B-A22B39.3338.9763.3221.301.75 Kimi-K232.9823.2427.4611.391.01 DeepSeek V3.182.0773.9368.8065.48 4.44 GLM-4.551.1357.7961.4821.302.14 Qwen3-30B-A3B38.6461.0053.5720.372.61 Mistral-Small-3.2-24B9.997.847.726.141.41 Gemma3-27B9.436.695.926.721.40 Table 22. Translation tasks on Spanish to Other languages. Es-En means Spanish translate to English.For Spanish to other language translations, DeepSeek V3.1 is the top performer for French (Es-Fr). Gemini-2.5-flash leads in translations to Chineses (Es-Zh) and Russian (Es-Ru), while Grok-4 achieves the best score for English (Es-En) and Arabic (Es-AR).Best results are marked in bold, second best results are underlined. ModelEs-EnEs-ZhEs-FrEs-RuEs-Ar GPT-543.0544.5784.0465.104.24 Claude Sonnet 425.5426.1635.9530.784.22 Gemini-2.5-flash55.7247.4980.8776.973.84 Grok-461.9343.7579.3564.124.40 Doubao-Seed-1.624.8320.8839.9118.930.89 Qwen3-235B-A22B23.3620.6518.8920.011.65 Kimi-K221.6619.6810.0813.410.99 DeepSeek V3.141.3347.0184.3258.444.22 GLM-4.531.6946.3075.1436.943.75 Qwen3-30B-A3B22.0721.7725.0319.571.60 Mistral-Small-3.2-24B8.997.106.625.541.37 Gemma3-27B7.807.355.655.411.30 Table 23. Translation tasks on French to Other languages. Fr-En means French translate to English.Based on the provided table for French translation tasks, Grok-4 achieves the highest score for English (Fr-En). Gemini-2.5-flash is the top performer for Chinese (Fr-Zh), Spanish (Fr-Es) and Russian (Fr-Ru), while DeepSeek V3.1 delivers the best results for Arabic (Fr-Ar).Best results are marked in bold, second best results are underlined. ModelFr-EnFr-ZhFr-EsFr-RuFr-Ar GPT-580.6641.8272.7770.502.86 Claude Sonnet 446.8321.4630.3323.682.16 Gemini-2.5-flash82.9849.5579.3574.026.93 Grok-484.3238.8279.0164.695.60 Doubao-Seed-1.636.1720.4825.9772.941.22 Qwen3-235B-A22B38.9622.6526.4021.671.46 Kimi-K233.5822.5917.0713.561.03 DeepSeek V3.182.9645.4270.9358.547.98 GLM-4.561.8632.5745.3070.716.90 Qwen3-30B-A3B37.1623.7225.7519.831.61 Mistral-Small-3.2-24B8.987.297.155.941.41 Gemma3-27B8.057.296.765.831.28 Table 24. Translation tasks on Russian to Other languages. Ru-En means Russian translate to English.For the Russian translation tasks, Grok-4 achieves the highest scores for English (Ru-En). DeepSeek V3.1 leads in Arabic (Ru-Ar), while Gemini-2.5-flash performs best in Chinese (Ru- Zh), Spanish (Ru-Es) and French (Ru-Fr).Best results are marked in bold, second best results are underlined. ModelRu-EnRu-ZhRu-EsRu-FrRu-Ar GPT-579.7239.7377.1273.544.01 Claude Sonnet 456.2822.2424.4826.433.95 Gemini-2.5-flash73.3046.3380.8381.014.18 Grok-483.9543.75 80.1477.485.45 Doubao-Seed-1.634.6132.6721.8122.470.91 Qwen3-235B-A22B33.5721.2719.2919.790.89 Kimi-K232.0622.1917.1113.331.00 DeepSeek V3.183.54 42.3868.0164.666.84 GLM-4.563.2532.3850.3650.841.83 Qwen3-30B-A3B20.829.7713.9210.251.69 Mistral-Small-3.2-24B8.686.647.016.841.36 Gemma3-27B7.587.106.545.371.29 Table 25. Translation tasks on Arabic to Other languages. Ar-En means Arabic translate to English.On Arabic translation tasks, GPT-5 leads in translations to English (Ar-En), while Gemini- 2.5-flash achieves the highest scores for Chinese (Ar-Zh), Spanish (Ar-Es), French (Ar-Fr), and Russian (Ar-Ru).Best results are marked in bold, second best results are underlined. ModelAr-EnAr-ZhAr-EsAr-FrAr-Ru GPT-578.3740.3062.5072.3559.28 Claude Sonnet 473.2019.4018.3221.1620.78 Gemini-2.5-flash59.2043.3163.4983.2364.35 Doubao-Seed-1.612.4311.3113.5614.948.78 Grok-434.1420.1433.1733.5138.39 Qwen3-235B-A22B31.0518.701.2814.1216.67 Kimi-K223.8212.220.8711.479.33 DeepSeek V3.167.61 33.415.035.8440.26 GLM-4.557.1933.951.5242.1310.99 Qwen3-30B-A3B32.6516.051.5321.1917.06 Mistral-Small-3.2-24B6.371.682.743.882.44 Gemma3-27B3.852.980.480.312.03 39