Paper deep dive
EDRAC: Benchmarking Arabic Dialect Reading Comprehension
Noor Abo Mokh, Kirill Chirkunov, Teresa Lynn, Nizar Habash, Reham Marzouk, Malik H. Altakrori, Younes Samih, Muhammed Abu Odeh, Nour Rabih, Rahaf Alshahrani, Hamad Alshehhi, Hamdan Al-Ali, Muhra Almahri, Besher Hassan, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, Alham Fikri Aji
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/8/2026, 2:30:38 AM
Summary
The paper introduces EDRAC, the first large-scale benchmark for Dialectal Arabic (DA) Machine Reading Comprehension (MRC) and generative Question Answering (QA). It covers five major dialects (Egyptian, Moroccan, Emirati, Syrian, Saudi) using 499 passages from YouTube transcripts and 4,977 QA pairs generated via a human-LLM collaborative pipeline. The study benchmarks Arabic-centric and multilingual LLMs, revealing significant gaps in dialectal fidelity and highlighting limitations in current evaluation metrics.
Entities (13)
Relation Signals (13)
EDRAC → coversdialects → Emirati Arabic
confidence 95% · EDRAC ... covering five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic.
EDRAC → coversdialects → Egyptian Arabic
confidence 95% · EDRAC ... covering five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic.
EDRAC → coversdialects → Moroccan Arabic
confidence 95% · EDRAC ... covering five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic.
EDRAC → coversdialects → Syrian Arabic
confidence 95% · EDRAC ... covering five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic.
EDRAC → coversdialects → Saudi Arabic
confidence 95% · EDRAC ... covering five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic.
EDRAC → hassize → 4,977 QA pairs
confidence 95% · and 4,977 corresponding QA pairs generated through a human--LLM collaborative pipeline
EDRAC → hassize → 499 passages
confidence 95% · EDRAC contains 499 passages derived from naturally occurring spoken interactions
EDRAC → isbenchmarkfor → Generative QA
confidence 95% · EDRAC, the first large-scale benchmark for dialectal Arabic machine reading comprehension (MRC) and generative QA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Dialectal Arabic (DA) remains under-resourced compared to Modern Standard Arabic (MSA), particularly for machine reading comprehension (MRC) and question answering (QA). Existing Arabic QA benchmarks primarily focus on formal written MSA or multiple-choice QA, with limited coverage of naturally spoken dialects. Here, we aim to bridge this gap. We introduce EDRAC, the first large-scale benchmark for dialectal Arabic machine reading comprehension (MRC) and generative QA, covering five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic. EDRAC contains 499 passages derived from naturally occurring spoken interactions and 4,977 corresponding QA pairs generated through a human--LLM collaborative pipeline combining iterative generation, LLM-as-a-judge evaluation, and human verification. We benchmark Arabic-centric and multilingual LLMs on EDRAC using lexical and semantic metrics. Our results reveal substantial gaps between semantic answer quality and dialectal fidelity, highlighting the limitations of existing evaluation metrics for dialectal Arabic generation. EDRAC provides a realistic and challenging MRC benchmark for future research on dialectal Arabic NLP.
Tags
Links
- Source: https://arxiv.org/abs/2609.01113v1
- Canonical: https://arxiv.org/abs/2609.01113v1
Trouble viewing inline? Open PDF directly →
Full Text
108,851 characters extracted from source content.
Expand or collapse full text
EDRAC: Benchmarking Arabic Dialect Reading Comprehension Noor Abo Mokh, 1 Kirill Chirkunov, 1 Teresa Lynn, 1 Nizar Habash, 1,2 Reham Marzouk, 1 Malik H. Altakrori, 3 Younes Samih, 3 Muhammed Abu Odeh, 1 Nour Rabih, 1 Rahaf Alshahrani, 4 Hamad Alshehhi, 1 Hamdan Al-Ali, 1 Muhra Almahri, 1 Besher Hassan, 1 Mohamed Anwar, 1 Abed Alhakim Freihat, 1 Preslav Nakov, 1 Alham Fikri Aji 1 1 Mohamed bin Zayed University of Artificial Intelligence, 2 New York University Abu Dhabi, 3 IBM Research AI, 4 Qassim University noor.abomokh,alham.fikri@mbzuai.ac.ae Abstract Dialectal Arabic (DA) remains under-resourced compared to Modern Standard Arabic (MSA), particularly for machine reading comprehen- sion (MRC) and question answering (QA). Ex- isting Arabic QA benchmarks primarily focus on formal written MSA or multiple-choice QA, with limited coverage of naturally spoken di- alects. Here, we aim to bridge this gap. We introduce EDRAC, the first large-scale bench- mark for dialectal Arabic machine reading com- prehension (MRC) and generative QA, cover- ing five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic. EDRAC contains 499 passages derived from naturally occurring spoken interactions and 4,977 corre- sponding QA pairs generated through a human– LLM collaborative pipeline combining iterative generation, LLM-as-a-judge evaluation, and human verification. We benchmark Arabic- centric and multilingual LLMs on EDRAC us- ing lexical and semantic metrics. Our results reveal substantial gaps between semantic an- swer quality and dialectal fidelity, highlighting the limitations of existing evaluation metrics for dialectal Arabic generation. EDRAC provides a realistic and challenging MRC benchmark for future research on dialectal Arabic NLP. 1 Introduction Over the past decade, a plethora of resources have been created for Modern Standard Arabic (MSA), but there are fewer high-quality resources for Di- alectal Arabic (DA), mainly due to the lack of writ- ten data that reflects the natural use of spoken Ara- bic dialects. This mainly stems from the diglossic nature of Arabic (Farghaly and Shaalan, 2009), in which MSA is the primary mode of communica- tion in written texts, while DA is used for daily interactions, resulting in fewer text-based datasets (Darwish et al., 2021; Dahou et al., 2025). Moreover, manually creating and annotating such datasets is an expensive and laborious task. Traditionally, large DA datasets are either a transla- tion of existing English datasets or collected from social media resources (e.g., Bouamor et al. (2018) and Khered et al. (2025)). However, translations typically do not carry culturally relevant content, while social media sources do not reflect real-life conversations, spoken interactions, or natural use of the language, as speakers often alter their speech style while writing user-generated content (Eisen- stein, 2013; Sanguinetti et al., 2020). Consequently, constructing such a dataset requires transcribing au- dio of natural spoken conversations. For the task of dialectal machine reading compre- hension (MRC), available resources are limited to multiple-choice question answering (MCQA) tasks, (e.g., Belebele (Bandarkar et al., 2024) and Dialec- talArabicMMLU (Altakrori et al., 2026)). To date, Generative QA benchmarking for DA remains un- explored. Tasks such as MCQA rely on pre-defined options while EDRAC is an open-ended Question Answering task allowing for free-form descriptive, explanatory answers that often require reasoning. We address these gaps by presenting EDRAC (Evaluation of Dialectal Arabic Reading And Com- prehension), 1,2 a high-quality multi-dialectal MRC and Generative QA benchmark dataset for Arabic that covers Egyptian, Moroccan, Emirati, Syrian and Saudi dialects. In our curation of reading com- prehension passages, we leveraged transcripts of YouTube videos, covering a variety of genres, to capture the daily interactions and natural flow of the dialects. We initially generated 10 QA pairs per passage; after final review, the released dataset con- tains 4,977 QA pairs across 499 passages. Further- more, we outlined the process of automatically gen- erating these QA pairs using Large Language Mod- els (LLMs) with a human-in-the-loop approach. 1 EDRAC º @P X @ /idraak/ is Arabic for ‘comprehension’. 2 EDRAC is available under C BY-NC-SA 4.0 license at https://huggingface.co/datasets/MBZUAI/EDRAC arXiv:2609.01113v1 [cs.CL] 1 Sep 2026 This required a two-stage framework, involving multiple steps, each including layers of filtering and reviewing, resulting in a reliable, high-quality benchmark of fully human-validated QA pairs. Our contributions can be summarized as follows: •We introduce EDRAC, the first large-scale benchmark for Dialectal Arabic Generative QA across five major dialects: Egyptian (EGY), Emirati (UAE), Moroccan (MOR), Syrian (SYR), and Saudi (KSA), built from naturally occurring spoken interactions. •We propose a scalable human-LLM pipeline for dialectal QA generation and validation, combining self-refinement, LLM-based evalu- ation, and human verification to produce high- quality, culturally grounded QA pairs. •We present the first comprehensive evaluation of Arabic-centric and multilingual LLMs on Dialectal Arabic Generative QA, revealing major gaps in dialectal fidelity and establish- ing a benchmark for future research. 2 Related Work Existing Arabic Question Answering (QA) and Reading Comprehension (RC) benchmarks primar- ily focus on Modern Standard Arabic (MSA) and formal written text. For example, ARCD and Arabic-SQuAD (Mozannar et al., 2019) established the extractive QA setting for Arabic, while AC- QAD (Sidhoum et al., 2022), ArTrivia (Alrow- ili and Vijay-Shanker, 2023), ArabicaQA (Abdal- lah et al., 2024) expanded dataset scale through Wikipedia-derived and, in some cases, automati- cally generated QA pairs. Other QA benchmarks include BALSAM (Al-Matham et al., 2025) and the domain-specific MedAraBench (Abu-Daoud et al., 2026). These benchmarks, however, remain grounded in MSA, often providing limited coverage of the variability and structural ambiguity inherent in nat- urally spoken Arabic defined as the language used in real-world communication. Recent efforts sup- port various dialectal varieties, including the multi- dialectal Belebele (Bandarkar et al., 2024), Dialec- talArabicMMLU (Altakrori et al., 2026) and Ara- bicCulturalQA (Hassan Bhatti and Alam, 2026), MedQA-MA (Ouali and El Garouani, 2026) for Moroccan and Alyah (Alkaabi et al., 2026) for Emirati. Most, however, rely on constructed writ- ten data (such as paraphrasing, translation, or con- trolled prompting from MSA) instead of naturally occurring dialectal speech. EDRAC stands out by drawing on naturally oc- curring dialectal speech, capturing orthographic inconsistency, code-switching, and disfluent dis- course. Recent work has also explored conversational and open-domain QA. English benchmarks such as CoQA (Reddy et al., 2019) and QReCC (Anan- tha et al., 2021) introduced dialogue-aware com- prehension and conversational retrieval settings, while AraQReCC (Hassib et al., 2025) extended this paradigm to Arabic, albeit through translated conversational QA data. EDRAC departs from this paradigm by relying on authentic conversational speech rather than translated text, enabling evalua- tion under more realistic dialectal conditions. To mitigate the cost of large-scale annotation, prior efforts have explored automatic QA gener- ation using LLMs through QAmeleon (Agrawal et al., 2023), more specifically for multi-hop ques- tion generation (MQG) (Lin et al., 2024) or se- quential question rewriting (Hwang et al., 2024). Other QA benchmark construction efforts include languages such as French (Pellet et al., 2026) for multi-hop questions on historical data and Dutch (van Drie et al., 2026) for legal-domain QA. How- ever, automatic QA generation has not yet lever- aged modern LLMs from naturally occurring Ara- bic speech; existing efforts such as Elmadany et al. (2023) relied on encoder-decoder transformer ar- chitecture AraT5 (Nagoudi et al., 2022) for QA generation. EDRAC addresses these gaps through a robust human-in-the-loop framework in which LLMs gen- erate candidate QA pairs from spoken transcripts, preserving the conversational authenticity while en- suring the data quality. Table 1 provides a compari- son of EDRAC with respect to other benchmarks. 3 EDRAC EDRAC (Evaluation of Dialectal Arabic Reading And Comprehension) is a Generative QA bench- mark dataset covering five major Arabic dialects: EGY, UAE, MOR, SYR, and KSA representing a variety of dialects across the Arab World. We release a total of 499 passages and 4,977 corre- sponding QA pairs – approximately 1000 pairs per dialect. DatasetMSADialectalNaturally-occurringEvaluation ACQAD (Sidhoum et al., 2022)✓✗Extractive ArabicaQA (Abdallah et al., 2024)✓✗Extractive AraQReCC (Hassib et al., 2025)✓✗ † Conversational ARCD & Arabic-SQuAD (Mozannar et al., 2019)✓✗Extractive ArTrivia (Alrowili and Vijay-Shanker, 2023)✓✗Extractive BALSAM (Al-Matham et al., 2025)✓✗includes QA/RC MedAraBench (Abu-Daoud et al., 2026)✓✗MCQA ArabicCulturalQA (Hassan Bhatti and Alam, 2026)✓✗MCQA+Generative Belebele (Bandarkar et al., 2024)✓✗MCQA DialectalArabicMMLU (Altakrori et al., 2026)✓✗MCQA Emirati Benchmark (Alkaabi et al., 2026)✗✓✗MCQA MedQA-MA (Ouali and El Garouani, 2026)✗✓✗Generative EDRAC (Ours)✗✓Generative Table 1: Comparison of EDRAC and other related Arabic QA benchmarks. † AraQReCC introduces conversational QA, but relies on translated data rather than naturally occurring speech. (b) Transc. (a) Diar. (c) Restor. (d) Extract LLM-as-a-judge Manual Passage PASSAGE CURATION Video Collection Passage Preparation Keep Discard IAA Detok. QUESTION-ANSWER PAIRS GENERATION Initial Generation Iterate Iterative Improvement Generation Generating QA Pairs using LLMs QA Pairs Manual Review Passages + QA Pairs Human-in-the-loop EDRAC IAA Preprocessing Step Transcription Correction Manual Passage Review LLM-as-a-judge (a) Diariz. (b) Transc. (c) Restor. (d) Extract. Annotations Figure 1: The end-to-end data creation pipeline of EDRAC. The top panel outlines the passage curation workflow. The bottom panel illustrates the QA generation pipeline. A central characteristic of EDRAC is its reliance on naturally occurring speech, unscripted and, in some cases, partially scripted spoken data captured in real-world settings, thereby preserving the lin- guistic and cultural subtleties of Arabic dialects without artificial modification. To construct EDRAC, we designed a hybrid pipeline combining automated processing with hu- man verification. The process involved two main stages: passage curation and QA pairs generation. In the first stage (see Figure 1), YouTube videos in dialectal Arabic were processed through speaker diarization, transcription alignment, text restora- tion, and passage extraction. This was followed by manual correction and review by native speakers of the dialects. In the second stage, the QA pairs were generated using an iterative LLM-assisted framework based on Gemini 2.5 pro, with multiple refinement steps to improve quality and reduce hal- lucinations. Figure 1 presents the complete pipeline spanning the two stages: passage curation and QA pairs generation. The quality was ensured by an extensive man- ual annotation by a professional Language Ser- vice Provider: we provide annotator demographics, along with payment details in Appendix L. The development of comprehensive guidelines for both transcription correction and generated QAs was necessary, involving broad discussions with native speakers to ensure adequate handling of the linguis- tic nuances of Arabic dialects. The guidelines are available in Appendix J and Appendix K. 9696 74 70 43 39 34 18 15 14 Genre Distribution Personal Development Sports Other Interviews Documentary History Arts, Culture & Entertainment Science, Technology & Innovation Education Business & Economy Figure 2: Genre distribution counts for the passages across dialects. 4 Passage Curation The Arabic speech processing community has already developed extensive conversational cor- pora of dialectal transcribed speech data. Notable datasets include ArzEn (Hamed et al., 2020), STAC (Zribi et al., 2015), MASC (Al-Fetyani et al., 2023) and Mixat (Al Ali and Aldarmaki, 2024). Nev- ertheless, these corpora were designed primarily for ASR or dialect identification and were thus not suitable for the purposes of our study. Instead, we opted to curate a brand new multi- dialect transcription dataset, with post-editing fo- cusing on the curation of reading comprehension material. We chose YouTube as our primary source for representative dialectal speech as it provided a broad variety of genres of audio content across the five dialects (see Figure 2) and also allowed us to leverage pre-existing automated transcriptions. Video Collection Native speakers, who were in- structed to select content representative of their dialects across a variety of genres, recommended an initial list of 7 channels for their dialect and sub- sequent revisions and additions as necessary (see Appendix I for the final list of channels). Our collection was restricted to videos published between June 2024 and January 2026. Our target was 100 videos per dialect, but we sourced roughly 150 videos per dialect to allow for quality assur- ance filtering. We extracted the first six minutes of each video, which is equivalent to roughly 473 tokens per video passage (Table 2). Finally, we ex- cluded videos containing toxic or sensitive content. We used existing YouTube subtitles as the video transcription source. Videos lacking pre-existing transcriptions were excluded. Preprocessing stepsThe preprocessing pipeline consists of four steps: (a) Speaker diarization: we used the NVIDIA NeMo toolkit to perform speaker diarization on the extracted audio streams. This step segments the audio by speaker identity, en- suring that multi-speaker interactions—common in media—are preserved, (b) Transcription: in the transcription of speech, the tokens were times- tamped to ensure alignment between the text and the original acoustic signal, (c) Restoring text structure: to convert fragmented ASR outputs into coherent prose, we implemented a hybrid strat- egy for structure restoration that involved a review stage (See Appendix E for details). The automatic restoration step solely determines punctuation and segment boundaries; the original transcript tokens remain unchanged, and (d) Text extraction: a clean-up step discarded any passages that exceeded the passage size limit of 500 words, had missing video links, vacuous content such as promotional content and social media material such as subscrip- tion or follow requests. Finally, we extract the passages for annotation. Passage PreparationOur in-house native speak- ers performed a high-level review of the passages to ensure that there were no major issues. Transcription CorrectionWe created MS Excel- based forms for transcription correction, one pas- sage per tab, segmented by paragraph and token-by- line split to allow close scrutiny at the token level (see Appendix J for the annotation guidelines. We do not follow CODA (Habash et al., 2018)). On in- spection, the quality of YouTube’s Dialectal Arabic transcriptions was not sufficient for our benchmark. As such, we included a human transcription cor- rection stage. The annotators’ primary objective was to read the passage, listen to the corresponding video and correct any transcription issues. 27 anno- tators, representing the five dialects, collaborated in a correct-review workflow (demographics of the annotators are provided in Appendix L). 3 Detokenization and Manual Passage Review The passages were then detokenized to restore their original structure. The final (corrected) set of tran- scribed passages then underwent an additional in- ternal manual sanity check review (e.g., removing passages or QA pairs that were non-dialectal) be- fore finalizing the curated passage dataset. 3 Annotators alternated annotation and review roles, in a streamlined correct-review process. 5 Question-Answer Pairs Generation Our first attempt at QA generation involved a two- week pilot study with in-house linguists. The task required extensive creative skills to ensure ques- tions were not overly simple and could involve reasoning or deduction processes. This attempt proved so difficult and time-consuming that our next step was to explore the capability of an LLM in supporting QA generation for Arabic dialects. 5.1 Generating QA Pairs using LLMs After a number of trials, Gemini-2.5-pro 4 proved to be the most reliable LLM available for automati- cally generating QA pairs for the dialectal passages produced in Section 4. Our initial attempts at QA pairs generation, which relied solely on a single prompt, resulted in model hallucinations (such as excessive use of MSA, and irrelevant content). To enhance the quality of the generated pairs, we in- troduced an iterative refinement loop. Our QA-pair generation pipeline therefore involves a three-step generate-revise framework that we outline here. Step 1: Initial generation Given a passage, the model generates 10 QA pairs, based on a prompt de- vised over multiple iterative generate-review-revise rounds. 5 The prompt instructs the model to gen- erate reading comprehension QA pairs relevant to the provided passage, in the five dialects (see Ap- pendix B for the detailed prompt). Step 2: LLM-as-a-judge for evaluationTo mit- igate hallucination issues from Step 1, the pipeline includes an LLM-as-a-judge to evaluate the gen- erated pairs. This judge (also Gemini-2.5-pro) ap- plies a fine-grained set of quality checks to the generated pairs to evaluate their quality following an approach similar to Kim et al. (2024) and Pom- bal et al. (2025). We instruct the model to verify if the questions are objective, unbiased, relevant, an- swerable and concise. We also enforce additional stylistic rules requiring questions to be in the third person and to avoid the use of linguistic or gram- matical terms in the questions. We further impose additional criteria on the answers stipulating that they must be unambiguous and precise. For checks that are not met, the judge provides an explanation 4 https://ai.google.dev/gemini-api/docs/models/ gemini-2.5-pro 5 We initially explored several prompts to generate ques- tions with varying levels of difficulty, however, the model showed inconsistency in performance in this task, as well as a reliance on verbatim answering. for the failure in addition to recommendations for improvement on the complexity of generated pairs. The full prompt is provided in Appendix C. QA pairs that fail Step 2 of the generation framework enter into an iterative improvement loop (Step 3). Step 3: Iterative improvement generationThe same model is again provided with: the passage, the failed pairs from Step 2, and more crucially, the judge’s feedback on each pair’s quality. The model is explicitly instructed to correct any issues flagged during Step 2, while preserving the dialect charac- teristics of the passage. Steps 2 and 3 are repeated for a maximum of five times, or until the evaluated pair passes all quality checks. See Appendix D for the full prompt. Dialect Pass.Toks Sents. Par.Tok/ Sent/ Par./ Pass. Pass. Pass EGY100 46,655 1,989603 466.6 19.96.0 MOR100 50,174 2,643755 501.7 26.47.6 KSA100 47,632 2,109685 476.3 21.16.9 SYR99 42,821 3,057 1,217 432.5 30.9 12.3 UAE100 48,831 3,047 1,268 488.3 30.5 12.7 Overall499 236,113 12,845 4,528 473.17 25.74 9.07 Table 2: Post-review textual statistics and metrics across all dialects for passages, tokens, sentences and para- graphs. 5.2 Human-in-the-Loop As benchmark data cannot be created through au- tomated approaches alone, it is crucial to include a human-in-the-loop. This validation step enables assessment of Gemini 2.5 pro’s capability in gen- erating QA pairs and also flagging of LLM-related issues (Huang et al., 2025), such as hallucinations and inaccuracies. Dialectal Annotation GuidelinesManual evalu- ation of the generated QA pairs was based on de- fined criteria outlined in our annotation guidelines (see Appendix K). These guidelines were refined and finalized through an iterative process of pilot testing and revising. The first draft of the guidelines was tested in a pilot on Egyptian data, but differed significantly in terms of label categories, as it in- volved an attempt to generate QAs with various levels of difficulty - a categorization that proved difficult to define in terms of prompt instructions. In the second pilot, three native Emirati speak- ers reviewed 10 automatically generated QA pairs across five passages. <Link to Guidelines><Link to Video> Today’s conversation carries a different kind of impact. We are speaking in front of a journalist, and I consider you a truly professional journalist, highly capable, may God bless you. You possess a lifetime of experience; deserves to be recognized. I also have many questions I’d like us to discuss in this field, . Of course, we have the ordinary definition of . Journalism has changed today, Abu Mohammed. , while there is what we call the field journalist, and you have experience as a field journalist. Yes. Tell us, what’s the difference? Why did you fall in love with being a field journalist at the beginning of your career? When did the speaker’s journalism career begin, and what characterized that period? It began in the mid-1970s, and that period was marked by the early years of the country’s establishment, the founding of institutions, and the rise of journalism in the UAE. How many years of experience does the speaker Abdullah have in journalism? His experience in journalism extends for more than 45 years. According to Jamal, what is the difference between the traditional definition of a journalist and the journalists today? Traditionally, a journalist is someone who can report and write news, whereas today journalists mostly sit in their offices. Based on the speakers’ remarks, what is the main difference between a field journalist and a journalist who works from an office? A field journalist is the source of everything new because they interact directly with people and real life, unlike today’s journalists, most of whom work from their offices. 45 this long journey of more than 45 years in journalism from your beginnings in the mid-1970s until today a journalist: someone who can report and write newsJournalists are sitting in offices now Question List Question Relevancy QA Pair Naturalness Is Answer Correct? 1 RelevantNaturalCorrect 2 RelevantNaturalCorrect 45 3 RelevantNaturalCorrect 4 RelevantNaturalCorrect س نھ عن اكثر ا لمتده التجر بھ ھاي الصحا فھ مجال في ی عني عمر تج ربھ وتملك . خیر ﷲ جزاك وبا قتدار ، ی عني م حترف صحفي ا عتبرك انا ، صحفي امام نتحدث ن حنا . م ختلف وقع لھ الح دیث ا لیوم ی عني ش ویھ ، العادي الصحفي ا لتع ریف عن دنا ا حنا طبعا . ی عني المجال ھذا في فیھا نتحدث كثیره ا سئلھ عندي وا شیاء . تذكر ان ت ستحق . مید اني كصحفي تج ربھ لك و انت ال میدان صحفي في تع ریف ی عني و ھناك ، ، ا لیوم محمد بو ت غیرت الصحافھ . ؟ تج ربتك بد ایھ في ال میدان في صحفي تكون انك عشقت لیش ، ی عني الفرق شو قول لنا . نعم ال فترة؟ ھ ذیك ی میز كان الي وشو الصح فیة ال متحدث تج ربة بدت متى الإمارات. في الصح فیة وال حیاة المؤسات وتأ سیس الدولة قیام بد ایات فترة ك انت وھال فترة ال سب عینات، منتصف في تج ربتھ بدت الصحافة؟ مجال في ﷲ عبد ال متحدث خبرة مدة كم سنة. عن لأ كثر ت متد الصحافة مجال في خب رتھ ا لیوم؟ الصح فین وحال لصحفي ا لتق لیدي ا لتع ریف بین الفرق شو جمال، ال متحدث كلام حسب مك اتبھم. في یال سین فالصح فین ا لیوم أما وی كتبھا، الأ خبار ینقل یقدر الي ھو ا لتق لیدي الصحفي الم كتب؟ من ی شتغل الي والصحفي ال مید اني الصحفي بین الأساسي الفرق شو ال متح دثین، كلام على بناء من ی شتغلون أغ لبھم الي ا لیوم صح فین بعكس مباشر، بشكل وال حیاة ا لناس مع یتعامل لأنھ ی دید شي كل مصدر ھو ال مید اني الصحفي مك اتبھم. الان الى السبعینیات منتصف في انت بدایاتك من ا لمكاتب في یالسین الصحفین یكتبھا الاخبار ینقل ان یستطیع الذي ھو الصحفي Figure 3: Example of the EDRAC Google Sheets annotation interface, showing a shortened snippet of the Arabic transcript segment, its associated question–answer pairs, annotation labels, and color-coded evidence spans to guide annotators. English translations are included only for reader clarity and are not shown to annotators. The results and follow-up session revealed no- ticeable disagreement among annotators driven pri- marily by annotators’ regional linguistic variation and daily linguistic habits. Following the pilot, the guidelines were updated to improve the def- inition and categories of ‘naturalness’. We also added clearer explanation on what was acceptable as dialectal variations and how to handle code- switching. At this stage, five different sets of guide- lines were deemed more appropriate, one for each dialect. Each version differs only in the dialect- specific examples used. Annotation Set-up Dialect teams of two were assigned 100 passages and their 10 corresponding QA pairs (1000 QA pairs per dialect team). Each annotator annotated 60 passages with an overlap of 20 to allow for an IAA study. In the annotation file, each generated QA pair is cross-referenced with automatic color-coding highlighted spans within the passage to support efficiency in finding the rel- evant section(s) of the passage. We also include a link to the video relevant to each passage. See Fig- ure 3 for a visual example. The ultimate goal was to assess the LLM’s output in terms of relevancy, naturalness, and correctness, with an “ideal” result of Relevant, Natural and Correct labels. 1.Relevancy: Is the question relevant to the content of the passage? (Relevant, Irrelevant) 2.Naturalness: How well do the generated QA pairs represent the dialect as it is naturally spo- ken by native speakers? (Natural, Somewhat Awkward, Unnatural, Non-dialectal.) 3. Correctness: Does the answer meet the re- quirements of being factually accurate and supported directly by the passage, or con- cluded or inferred from information in the passage? (Correct, Incorrect, Invalid). IAA As an additional measure of quality, we conduct inter-annotator agreement using Cohen’s Kappa coefficient (Cohen, 1960) on 20 passages per dialect. While the annotations achieved high observed agreement across the categories, the re- sulting Kappa scores remained comparatively low due to the highly skewed label distribution within our dataset (also known as the “Kappa Paradox” (Feinstein and Cicchetti, 1990)). We therefore cal- culated PABAK-OS scores (a measure used for linguistic analysis in Anzovino et al. (2018)). The results showed high agreement across dialects. See Appendix M for all results. DialectRelevantNaturalCorrect EGY100%96%98% MOR98%99%96% KSA99%95%96% SYR97%98%90% UAE100%98%98% Table 3: Pre-review label distribution summary. The three dimensions were evaluated independently. Manual Review We performed a final round of manual reviews by in-house native speakers. Spot checks were carried out on the quality of annota- tions for each dialect. 209 QA pairs out of 5,000 (4.2%) were flagged. We classify the issues flagged into: (a) Fix (b) Discard (c) No change (see Table 9 in Appendix G for the statistics of the data, and Appendix N for an example of a flagged issue). Therefore the Total Flagged is a very small per- centage of the entire dialectal set of 5000 QA pairs. At this stage, 23 QA pairs were discarded, includ- ing one full passage entry, resulting in 499 passages. It is worth noting that a significantly low number of LLM-generated issues were flagged by our anno- tators. These issues generally related to incorrect answers, misinterpretation of the question, gender confusion (this affecting morphology of several to- kens), partial correctness (e.g., only retrieving part of a list) and hallucinated answers (not found in the passage). The distribution of the “ideal” re- sponse labels (Relevant, Natural and Correct) for the pre-review generated pairs are shown in Table 3. Table 4 provides a summary of the types of errors encountered during the manual review. The results of the manual review demonstrate two insights. First, Gemini 2.5 Pro can be reliably used in an iterative-improvement loop approach to generate these 5 dialectal varieties of Arabic. Second, we confirm the quality of the dataset. Error CategoryQA pairs % Incorrect Annotation1% Language naturalness<1% Model answer errors<1% Question quality issues<0.1% Cultural understanding errors<0.1% Transcription quality issues<0.1% Table 4: Pre-review percentage of QA error categories. 5.3 Data Challenges Developing a high-quality dataset for DA comes with challenges, stemming from the language’s diglossic nature, the vast number of varieties, re- gional variation that is not well-documented, and attitudes towards these varieties. One issue that influenced both the design of the guidelines and the annotation process is the orthographic variation among speakers of Arabic as there is no standardized orthography that is com- monly adopted among speakers. Speakers’ spelling choices are influenced by the phonological repre- sentation of the word or MSA orthographic rules. In our guidelines, we emphasized that spelling variants are acceptable when there is no meaning change. The definition of naturalness posed another chal- lenge. The Emirati pilot (see Section 5.2) revealed a high disagreement among speakers. An underly- ing reason for judgment variations was the anno- tators’ own regional dialectal preferences, and as- sumptions about their dialect. This was noticeable in the ‘naturalness’ category. Additionally, each Arabic dialect has its own sub-varieties that can vary lexically and phonologically. Not all speakers are familiar with such sub-varieties, and may there- fore be biased against them. From an annotation point of view, this means the annotators’ judgments can be biased and may not reflect the true nature of the dialects. The subjective judgments and regional attitudes directly impacted label distribution in the Emirati pilot. Annotators were often hesitant to la- bel unfamiliar sub-varieties as ‘natural’. To counter this, our guidelines help annotators to look past their sub-variety bias. We have also emphasized that variation is a natural linguistic phenomenon across speakers. Additionally, some of the passages contained high-register language characteristic of formal me- dia contexts (e.g., news). Prestige bias led some annotators to regard such language as unnatural. Figure 4: Model performance across five Arabic dialects under three different evaluation metrics. Blue model names indicate general multilingual models, whereas Red model names indicate Arabic-focused models. We noted in our guidelines that these forms re- flect the linguistic nuances of the native dialect and should be labeled ‘natural’. On the modeling side, we observed dialect mix- ing in an earlier attempt at generating QA pairs e.g., Syrian with Palestinian, Emirati with Saudi. This is not unusual, as models are often biased. We address this in our prompt by specifying both the dialect and the country where it is spoken. 6 Experimental Setup We use the EDRAC dataset introduced in Section 3 to establish a benchmark and evaluate a set of Arabic-centric and multilingual LLMs on their abil- ity to answer dialectal RC questions. In choosing these models, we focused on dialectal models Atlas- Chat-2B, Atlas-Chat-9B, Nile-Chat-4B, Hala-9B and Jais-13B-Chat that support Moroccan, Egyp- tian, Saudi and Emirati dialects, respectively. In ad- dition, we evaluate a number of multilingual mod- els that are both open-weight models (Qwen2.5-7B, Gemma4-31B and SILMA-9B) and closed-weight models (GPT-5.4 and Qwen3.6-Plus). Details are provided in Table 8 in Appendix F. For the QA setup, 6 we used OpenRouter (Open- Router, 2026), an online API that provides a unified access to different models using one standard inter- face. The models were prompted with: «Given the following passage, answer the accompanying question. Passage: passage Question: question Answer:» 6 Source code is available athttps://github.com/ mbzuai-nlp/EDRAC. We provide the prompt to the models in English rather than Arabic. This aligns with the standard practice of using English as the language of the prompt, especially when benchmarking a highly di- verse suite of both multilingual and Arabic-centric LLMs. For example, Aljagthami et al. (2025) and Dey et al. (2024) have demonstrated that mod- els show a more consistent performance when prompted in English. 6.1 Evaluation Metrics We report model performance on answering ques- tions using three metrics:(1) ROUGE - a token-based metric (whitespace tokenization), (2) BERTScore model (mDeBERTa-based implemen- tation), and (3) CAMeLBERTScore which is an Arabic-specific implementation. ROUGE (Lin, 2004) is typically used for the evaluation of text summarization and machine translation tasks (Chang et al., 2024). It is a token-based metric and can only capture exact lexical matches between the gold and generated texts. We also report the performance using BERTScore-F1, which is better suited to mea- suring the semantic similarity. We evaluate with BERTScore (Zhang et al., 2020) using two differ- ent encoder-only language models: mDeBERTa, a multilingual variant of DeBERTaV3 (He et al., 2023), and CAMeLBERT (Inoue et al., 2021), a model fine-tuned for Arabic dialects. We use CAMeL-Lab/bert-base-arabic-camelbert-mix. We provide results in Figure 4. The model order in the plot is based on average performance across dialects, under three metrics, sorted in descending order (1st row is the best performing model). 7 Results and Analysis Figure 5 illustrates model aggregate performance on the EDRAC dataset across various dialects, eval- uated using three distinct metrics (also see Ap- pendix A for the results). According to ROUGE-L and BERTScore-F1, the flagship GPT-5.4 achieved the highest average performance across all dialects. While the model’s proprietary nature limits our analysis, we can only speculate that this advantage stems from its massive parameter size and exten- sive training data. Among the Arabic-focused models, Atlas-Chat- 9B achieved solid overall performance. Notably, the Atlas models exhibited exceptionally strong per- formance on the Moroccan dialect, which aligns with their Moroccan-focused training data. Further- more, Nile-Chat-4B demonstrated remarkable effi- ciency, outperforming significantly larger Arabic- centric models such as Jais-13B-Chat, SILMA-9B, and Hala-9B despite its compact 4B parameter size. While Gemma4-31B also delivered strong re- sults, its massive scale makes it less practical for deployment, especially considering it was outper- formed by the much smaller Atlas-Chat-9B in most scenarios. Conversely, Qwen3.6-Plus exhibited a degradation in performance compared to its prede- cessor, suggesting a weaker Arabic performance in this evaluation setting. Lastly, we observed that standard metrics like ROUGE-L and BERTScore over-reward models for broad semantic similarity while failing to cap- ture dialectal nuances, creating a misleading leader- board that does not always accurately reflect true dialectal proficiency. For instance, when evaluated using the Arabic-specific CAMeLBERTScore-F1 metric, GPT-5.4 dropped from 1st to joint 5th with Qwen2.5-7B. This suggests that while the model generates semantically accurate responses, it strug- gles with dialectal constructions and specific lexi- cal overlap. 7.1 Human Evaluations We conducted human evaluations on two models to further investigate our findings. We selected 20 examples from GPT-5.4 output (best perform- ing model) and 20 examples from Hala-9B output (worst performing model) for each of the dialects. To ensure a representative sample across genres, we selected two questions from 10 different passages within each dialectal set. We evaluate the model outputs on two dimensions: accuracy and natural- Dialect Rouge BertScore Accuracy Naturalness EGY13%82%85%32% MOR11%85%95%50% KSA13%81%95%25% SYR8%79%82%42% UAE13%81%100%5% Table 5: Comparison of Automatic and Human Eval- uation Scores Across Five Arabic Dialects. Human evaluation scores (Accuracy and Naturalness) are based on a 40-example sample per dialect drawn equally from GPT-5.4 and Hala-9B outputs. ness (see Appendix O for annotation guidelines). The accuracy measures semantic equivalence with a reference answer. To avoid bias and lexical prim- ing during evaluations, we do not provide the pas- sages to the annotators. Table 5 shows a clear discrepancy between au- tomatic and human metrics. Standard metrics like ROUGE penalize lengthy or elaborate an- swers even when they are representative of accu- rate human responses. Conversely, while semantic- similarity metrics like BERTScore show high scores across the board, they fail to capture crit- ical issues with the "dialectness" of the generated text. For example, for some correct answers, the annotators also labeled the answer as non-dialectal. 8 Conclusion and Future Work We introduced EDRAC, the first large-scale bench- mark for dialectal Arabic machine reading compre- hension and generative question answering across five Arabic dialects.EDRAC is grounded in naturally occurring spoken interactions and con- structed through a scalable human–LLM collab- orative pipeline of iterative generation, LLM-as- a-judge evaluation, and human verification. Our experiments with Arabic-centric and multilingual LLMs reveal substantial gaps between semantic answer quality and dialectal fidelity, highlighting limitations of current evaluation metrics for dialec- tal Arabic. Our roadmap for building comparable resources can help other low-resource spoken vari- eties to address the challenge of data scarcity. In future work, we plan to expand EDRAC to additional dialects and more challenging reasoning settings, including conversational and multi-hop QA. We also aim to investigate dialect-sensitive evaluation metrics and extend our framework to other low-resource spoken languages. Limitations EDRAC covers five major Arabic dialects, but it does not capture the full linguistic diversity of the Arabic-speaking world, including finer-grained re- gional and sociolectal variation. In addition, al- though the dataset is grounded in naturally spoken interactions, some source content may still contain partial scripting or speaker self-monitoring typical of online media. Our QA generation pipeline relies on LLM- assisted generation and evaluation, which may in- troduce residual biases or stylistic artifacts despite multiple rounds of human verification and quality control. We also note that there are many high- quality Arabic-centric models available that sup- port Arabic, yet we limited our choice of models to a representative and diverse sample for the pur- poses of our study. Finally, while we evaluate a diverse set of Arabic- centric and multilingual LLMs, the rapidly evolv- ing landscape of language models means that future systems may exhibit different performance trends on the benchmark. Ethics and Broader Impact EDRAC was constructed from publicly available YouTube content using only limited transcript ex- cerpts necessary for the research objectives. The dataset is released under the C BY-NC-SA 4.0 license for non-commercial use and is intended to be solely an evaluation benchmark. To preserve benchmark integrity, it should not be included in pretraining or fine-tuning data. To reduce potential harms, we excluded videos containing sensitive, toxic, or explicit content dur- ing data collection and applied multiple layers of human review throughout the annotation pipeline. The dataset is designed to support research on di- alectal Arabic NLP, particularly for underrepre- sented spoken varieties that are often overlooked in current language technologies. At the same time, dialectal data may contain regional, social, or cultural biases that reflect natu- rally occurring speech. While we employed native- speaker annotators and quality-control procedures to improve consistency and linguistic authentic- ity, residual biases and annotation subjectivity may still remain. In addition, the use of LLM-assisted QA generation may introduce stylistic artifacts or model-specific biases despite human verification. We hope EDRAC contributes to more inclusive and representative Arabic NLP systems by encour- aging research on dialect-aware language technolo- gies and more robust evaluation methods for spo- ken language understanding. We used AI writing assistance within the scope of “Assistance purely with the language of the paper” described in the ACL Policy on Publication Ethics. Acknowledgments This work was conducted as part of the IBM–MBZUAI AI Center of Excellence. The re- search is in part with support from Google.org and the Google Cloud Research Credits program for the Gemini Academic Program through a grant to New York University Abu Dhabi’s CAMeL Lab. The au- thors gratefully acknowledge the contributions of Amr Keleg and Samar Magdy for their assistance during the data curation process. References Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. Arabi- caQA: A comprehensive dataset for Arabic question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, SIGIR ’24, pages 2049–2059, New York, NY, USA. Association for Computing Machinery. Mouath Abu-Daoud, Leen Kharouf, Omar El Hajj, Dana Samad, Mariam Al-Omari, Jihad Mallat, Khaled Saleh, Nizar Habash, and Farah Shamout. 2026. MedAraBench: Large-scale Arabic medical ques- tion answering dataset and benchmark. In Interna- tional Conference on Learning Representations, vol- ume 2026 of ICLR ’26, pages 13469–13496, Rio de Janeiro, Brazil. Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, and Mirella Lapata. 2023. QAmeleon: Multilingual QA with only 5 examples. Transac- tions of the Association for Computational Linguis- tics, 11:1754–1771. Maryam Khalifa Al Ali and Hanan Aldarmaki. 2024. Mixat: A data set of bilingual Emirati-English speech. In Proceedings of the 3rd Annual Meeting of the Spe- cial Interest Group on Under-resourced Languages @ LREC-COLING 2024, pages 222–226, Torino, Italia. ELRA and ICCL. Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. 2023. MASC: Massive arabic speech cor- pus. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1006–1013. Rawan Al-Matham, Kareem Darwish, Raghad Al- Rasheed, Waad Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, Norah Alzahrani, Eman AlBilali, Nizar Habash, Abdelrahman El-Sheikh, Muhammad Elmallah, Haonan Li, Hamdy Mubarak, Mohamed Anwar, Zaid Alyafeai, and 24 others. 2025. BALSAM: A platform for benchmarking Arabic large language models. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 258–277, Suzhou, China. Association for Com- putational Linguistics. Aamer Aljagthami, Mohammed Banabila, Musab Al- shehri, Mohammed Kabini, and Mohammad D. Alah- madi. 2025. Evaluating large language models for code translation: Effects of prompt language and prompt design. arXiv preprint arXiv:2509.12973. Omar Alkaabi, Ahmed Alzubaidi, Hamza Alobeidli, Shaikha Alsuwaidi, Mohammed Alyafeai, Leen AlQadi, Basma El Amel Boussaha, and Hakim Hacid. 2026. Alyah: An Emirati Dialect Benchmark for Evaluating Arabic Large Language Models. Sultan Alrowili and K Vijay-Shanker. 2023. ArTrivia: Harvesting Arabic Wikipedia to build a new Arabic question answering dataset. In Proceedings of Ara- bicNLP 2023, pages 191–207, Singapore (Hybrid). Association for Computational Linguistics. Malik H. Altakrori, Nizar Habash, Teresa Lynn, Younes Samih, Abed Alhakim Freihat, Kirill Chirkunov, Muhammed AbuOdeh, Radu Florian, Preslav Nakov, and Alham Fikri Aji. 2026. DialectalArabicMMLU: Benchmarking dialectal capabilities in Arabic and multilingual language models. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 3199–3219, Palma, Mallorca, Spain. European Language Resources As- sociation (ELRA). Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-Domain question answering goes conversational via question rewriting. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 520–534, Online. Association for Computational Lin- guistics. Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic identification and classification of misogynistic language on Twitter. In International Conference on Applications of Natural Language to Information Systems, pages 57–64. Springer. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele Benchmark: A parallel reading comprehension dataset in 122 lan- guage variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdul- rahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Pro- ceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources As- sociation (ELRA). Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language mod- els. ACM Trans. Intell. Syst. Technol., 15(3). Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37–46. Abdelghani Dahou, Abdelhalim Hafedh Dahou, Mo- hamed Amine Cheragui, Amin Abdedaiem, Mo- hammed A. A. Al-Qaness, Mohamed Abd Elaziz, Ahmed A. Ewees, and Zhonglong Zheng. 2025. A survey on dialect Arabic processing and analysis: Re- cent advances and future trends. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 24(8). Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jar- rar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the Arab world. Commun. ACM, 64(4):72–81. Krishno Dey, Prerona Tarannum, Md. Arid Hasan, Im- ran Razzak, and Usman Naseem. 2024. Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings. Preprint, arXiv:2410.13153. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the 2013 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 359–369, Atlanta, Geor- gia. Association for Computational Linguistics. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Octopus: A mul- titask model and toolkit for Arabic natural language generation. In Proceedings of ArabicNLP 2023, pages 232–243, Singapore (Hybrid). Association for Computational Linguistics. Ali Farghaly and Khaled Shaalan. 2009. Arabic natu- ral language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing, 8(4). Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Ram- bow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. Uni- fied guidelines and resources for Arabic dialect or- thography. In Proceedings of the Eleventh Inter- national Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). Injy Hamed, Ngoc Thang Vu, and Slim Abdennadher. 2020. ArzEn: A speech corpus for code-switched Egyptian Arabic-English. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 4237–4246, Marseille, France. European Language Resources Association. Hunzalah Hassan Bhatti and Firoj Alam. 2026. Beyond MCQ: An open-ended Arabic cultural QA bench- mark with dialect variants. In Proceedings of the Fifteenth Language Resources and Evaluation Con- ference, pages 5215–5231, Palma, Mallorca, Spain. ELRA Language Resource Association. Mariam E. Hassib, Nagwa El-Makky, and Marwan Torki. 2025. Open-domain Arabic conversational question answering with question rewriting. In Pro- ceedings of The Third Arabic Natural Language Pro- cessing Conference, pages 84–96, Suzhou, China. Association for Computational Linguistics. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large lan- guage models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2). Seonjeong Hwang, Yunsu Kim, and Gary Geunbae Lee. 2024. Explainable multi-hop question generation: An end-to-end approach without intermediate ques- tion labeling. In Proceedings of the 2024 Joint In- ternational Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC- COLING 2024), pages 6855–6866, Torino, Italia. ELRA and ICCL. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Compu- tational Linguistics. Abdullah Khered, Youcef Benkhedda, and Riza Batista- Navarro. 2025. Dial2MSA-Verified: A multi-dialect Arabic social media dataset for neural machine trans- lation to Modern Standard Arabic. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 50–62, Abu Dhabi, UAE. Associa- tion for Computational Linguistics. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing fine- grained evaluation capability in language models. Preprint, arXiv:2310.08491. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Zefeng Lin, Weidong Chen, Yan Song, and Yongdong Zhang. 2024. Prompting few-shot multi-hop ques- tion generation via comprehending type-aware se- mantics. In Findings of the Association for Computa- tional Linguistics: NAACL 2024, pages 3730–3740, Mexico City, Mexico. Association for Computational Linguistics. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic question answer- ing. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Flo- rence, Italy. Association for Computational Linguis- tics. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text- to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. OpenRouter. 2026. OpenRouter: Unified API for Large Language Models.https://openrouter.ai. Ac- cessed: 2026-05-22. Soufiyan Ouali and Said El Garouani. 2026. MedQA- MA: A Moroccan Arabic medical question- answering dataset for virtual healthcare assistants and large language models. Data in Brief, 65:112537. Aurelien Pellet, Marie Anna Puren, and Julien Perez. 2026. HistoriQA-ThirdRepublic: Multi-hop ques- tion answering corpus for historical research, par- liamentary debates from the French third republic (1870-1940). In Proceedings of the Fifteenth Lan- guage Resources and Evaluation Conference (LREC 2026), pages 207–223, Palma, Mallorca, Spain. Eu- ropean Language Resources Association (ELRA). José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and Andr’e F. T. Martins. 2025. M-prometheus: A suite of open multilingual LLM judges. In Proceed- ings of the Second Conference on Language Model- ing. John W. Ratcliff and David E. Metzener. 1988. Pattern matching: The gestalt approach. Dr. Dobb’s Journal. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249–266. Manuela Sanguinetti, Cristina Bosco, Lauren Cas- sidy, Özlem Çetino ̆ glu, Alessandra Teresa Cignarella, Teresa Lynn, Ines Rehbein, Josef Ruppenhofer, Djamé Seddah, and Amir Zeldes. 2020. Treebank- ing user-generated content: A proposal for a uni- fied representation in Universal Dependencies. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5240–5250, Marseille, France. European Language Resources Association. Abdellah Hamouda Sidhoum, M’hamed Mataoui, Faouzi Sebbak, and Kamel Smaïli. 2022. ACQAD: A Dataset for Arabic Complex Question Answering. In Atlantis Press Proceedings., Boumerdès, Algeria. Romy A.N. van Drie, Roos M. Bakker, Daan L. Di Scala, and Maaike de Boer. 2026. QuALA-NL: Ques- tion & answer with legal attribution in Dutch. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 674– 684, Palma, Mallorca, Spain. European Language Resources Association (ELRA). Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Inter- national Conference on Learning Representations, ICLR ’20. Inès Zribi, Mariem Ellouze, Bilel Hamrouni, and Lamia H. Belguith. 2015. Spoken Tunisian Ara- bic corpus "STAC": Transcription and annotation. Research in Computing Science, 90:123–135. A Results EGYMORKSASYRUAE ModelRLBF D 1 BF C 1 RL BF D 1 BF C 1 RL BF D 1 BF C 1 RL BF D 1 BF C 1 RL BF D 1 BF C 1 GPT-5.416.3 83.874.6 16.1 84.873.3 16.6 83.374.5 13.4 83.173.3 16.9 83.574.0 Qwen3.6-Plus12.0 82.271.7 13.1 82.671.8 13.1 82.573.2 9.82 81.271.2 13.5 82.373.1 Qwen2.5-7B13.4 81.673.7 12.5 80.572.4 14.8 81.975.2 12.7 80.973.6 15.2 81.974.8 Gemma4-31B12.8 81.973.0 15.0 82.273.3 16.0 82.875.6 11.6 81.073.3 16.3 82.675.2 Jais-13B-Chat12.8 80.573.9 10.9 80.772.8 13.7 80.375.5 11.7 80.574.5 14.9 80.375.7 SILMA-9B8.24 78.872.6 8.39 81.972.1 9.26 78.273.8 5.58 79.471.9 9.36 78.973.7 Command-R7B12.2 81.473.6 10.1 81.272.3 11.9 80.874.4 11.0 80.773.9 12.8 80.774.8 Nile-Chat-4B14.8 83.176.4 8.09 81.372.5 13.3 82.675.9 11.0 82.475.5 13.1 82.375.8 Atlas-Chat-2B10.6 81.572.3 15.9 83.474.8 11.3 81.073.1 9.90 81.572.3 12.1 81.073.3 Atlas-Chat-9B13.1 82.773.4 19.5 84.576.8 14.8 82.674.8 13.0 82.974.2 15.9 82.775.0 Hala-9B6.09 78.368.2 4.37 72.364.5 6.21 78.768.9 4.83 76.066.9 6.53 78.369.0 Table 6: Model Evaluation Results across Dialects. RL denotes ROUGE-L (whitespace tokenization); BF D 1 denotes BERTScore-F1 (mDeBERTa); BF C 1 denotes CAMeLBERTScore-F1. Figure 5: Performance per dialect, using the average performance for all the models. Model NameEGYMORKSASYRUAEAvg. Rank↑ GPT-5.4121211.4 Atlas-Chat-9B313132.2 Gemma4-31B542623.8 Qwen2.5-7B464444.4 Nile-Chat-4B2105355.0 Jais-13B-Chat676566.0 Qwen3.6-Plus857977.2 Command-R7B788787.6 Atlas-Chat-2B939897.6 SILMA-9B1091010109.8 Hala-9B111111111111.0 Table 7: Model rankings across five Arabic dialects, sorted by the average rank (Lower is better). B QA Generation: Initial Prompt combined_initial_prompt = ChatPromptTemplate.from_template(""" You are a linguist and passage_language native speaker. You need to generate reading comprehension non-opinionated question with precise, unambiguous answer, along with a list of exact quotes from the passage that were used to form the answer. The questions should be based on understanding and interpreting the text in the passage. A question can sometimes be derived partly through common knowledge in country that is not explicitly present in the passage. It is possible to either find the answer to this question directly in the text or to infer the answer from the overall meaning of the passage. In order to answer an inferential question, pieces of information in the passage may need to be connected, or the sequence of events may need to be tracked. The answers to the generated questions must undergo all the following cases: 1.The answer must be found directly in the passage. It should match the passage closely in meaning but does not need to be copied verbatim. Minor adjustments are allowed for pronoun consistency (e.g., change “I did” to “he did” or “she did”), as well as slight rephrasing or morphological changes. 2. The question must be phrased strictly in the third person. 3.The answer must be inferred by piecing together information from different parts of the passage. 4.The answer must be inferred by interpreting the overall meaning of the passage. You must not use a question similar to the previous questions. The answer to the question should not be the same as the answer to previous questions. Identify and extract the full quote(s) where the answer to the question is based. The quotes must be exact, and the character indices of each quote within the passage must be provided. The quotes must not include any extra details that are not explicitly in the answers Formulate each question strictly in the third person. Questions and answers must be generated using ONLY the passage_language. Your response must contain the question and answer only, or “N/A” if the question cannot be created. Keep each question short and concise. The answer must be short. IMPORTANT: The question, answer and extracted sentence must be in passage_language.Do not use MSA or other dialects, otherwise you will be penalized $1000 per word. Return your response in the following JSON format: "Question": "<your question in passage_language or N/A>", "Answer": "<your answer in passage_language or N/A>", "Quotes":[ "text": "<your quote in passage_language or N/A>", "start_char": <the starting character index of the quote in the passage as an integer, or −1 if N/A>, "end_char": <the ending character index of the quote in the passage as an integer, or −1 if N/A> ] Passage: —– passage —– Previous Questions: previous_questions """) C QA Generation: LLM-as-a-Judge Prompt combined_judgement_prompt = ChatPromptTemplate.from_template(""" You are passage_language native speaker. You are evaluating a reading comprehension question and its answer based on the passage below, which is written in passage_language. Your task is to set of critical criteria and nicetohave recommendations. Label these quality and formatting dimensions (True/False): - IsNonOpinionated: The question is objective and not based on personal opinion. - UnambiguousAnswer: The answer is clear and does not allow multiple interpretations. - IsUnbiased: The question is free from prejudice. - IsAnswerable: The question can be answered from the text with the allowed outside knowledge. - IsRelevant: The question is clearly related to the passage’s content. - IsInThirdPerson: The question is phrased strictly in the third person. - QuestionFreeFromLinguisticOrGrammarTerms: Grammar or linguistic terms are not used in the question. - IsShortQuestion: The question is 20 words or fewer. - IsPreciseAnswer: The answer does not contain unnecessary details or irrelevant information. - IsInpassage_language: The question, answer and quotes must be in passage_language. Critical: If any of the checks for boolean dimensions are false, describe how to fix it. NiceToHave: Suggest briefly how to increase the reasoning complexity of the question. Output your evaluation strictly as the following JSON schema. Return only the JSON object, nothing else. Provide evaluation strictly in JSON: "IsNonOpinionated": true|false, "IsNonOpinionated_reason": "string", "UnambiguousAnswer": true|false, "UnambiguousAnswer_reason": "string", "IsUnbiased": true|false, "IsUnbiased_reason": "string", "IsAnswerable": true|false, "IsAnswerable_reason": "string", "IsRelevant": true|false, "IsRelevant_reason": "string", "IsInThirdPerson": true|false, "IsInThirdPerson_reason": "string", "QuestionFreeFromLinguisticOrGrammarTerms": true|false, "QuestionFreeFromLinguisticOrGrammarTerms_reason": "string", "IsShortQuestion": true|false, "IsShortQuestion_reason": "string", "IsPreciseAnswer": true|false, "IsPreciseAnswer_reason": "string", "IsInpassage_language": true|false, "IsInpassage_language_reason": "string", "Recommendations": "Critical": "string", // string; empty if no critical issues "NiceToHave": "string" // string; suggestion to increase complexity Return only the JSON object, nothing else. Otherwise you will be penalized $1000 per word. Passage: —– passage —– Question: question Answer: answer Quotes: quotes """) D QA Generation: Improvement Prompt combined_improvement_prompt = ChatPromptTemplate.from_template(“You are a linguist and passage_language native speaker. Revise the original_question and original_answer based on the provided judge feedback.Your primary mandate is to resolve all issues in Recommendations.Critical. Systematically correct every dimension flagged as false in the feedback, using the associated _reason fields to guide your edits. You need to generate a new version of reading comprehension non-opinionated question with precise, unambiguous answer, along with a list of exact quotes from the passage that were used to form the answer. The questions should be based on understanding and interpreting the text in the passage. A question can sometimes be derived partly through common knowledge in country that is not explicitly present in the passage. It is possible to either find the answer to this question directly in the text or to infer the answer from the overall meaning of the passage. In order to answer an inferential question, pieces of information in the passage may need to be connected, or the sequence of events may need to be tracked. The answers to the generated questions must undergo all the following cases: 1.The answer must be found directly in the passage. It should match the passage closely in meaning but does not need to be copied verbatim. Minor adjustments are allowed for pronoun consistency (e.g., change “I did” to “he did” or “she did”), as well as slight rephrasing or morphological changes. 2. The question must be phrased strictly in the third person. 3.The answer must be inferred by piecing together information from different parts of the passage. 4.The answer must be inferred by interpreting the overall meaning of the passage. Identify and extract the full quote(s) where the answer to the question is based. The quotes must be exact, and the character indices of each quote within the passage must be provided. The quotes must not include any extra details that are not explicitly in the answers Formulate each question strictly in the third person. Questions and answers must be generated using ONLY the passage_language. Your response must contain the question and answer only, or “N/A” if the question cannot be created. Keep each question short and concise. The answer must be short. IMPORTANT: The question, answer and extracted sentence must be in passage_language.Do not use MSA or other dialects, otherwise you will be penalized $1000 per word. Return your response in the following JSON format: "Question ": "<your question in passage_language or N/A> ", "Answer": "<your answer in passage_language or N/A> ̈ , "Quotes ":[ "text ": "<your quote in passage_language or N/A> ̈ , "start_char": <the starting character index of the quote in the passage as an integer, or −1 if N/A>, "end_char": <the ending character index of the quote in the passage as an integer, or −1 if N/A> ] Passage: —– passage —– Original Question: original_question Original Answer: original_answer Original Quotes: original_quotes Judge Feedback: judge_feedback ”) E Text Structure Restoration from Dialectal Arabic Transcripts Figure 6: Three-stage pipeline for restoring punctuation and paragraph structure from raw transcripts. Given raw transcripts with utterances and timings, we restore this structure (sentence split, punctua- tion, paragraphs) in three stages, as shown in Fig- ure 6. Stage 1: Utterance-level classification The in- put transcript is represented as utterances (se- quences of tokens) with speech timings. The LLM maps each utterance to a punctuation label orNone. The list of utterances is presented to an LLM to- gether with their time intervals, and the model as- signs a punctuation label orNone. As recent re- search shows, such a representation of the task is more natural for generation models than the token- based alternative. Results of classification allow es- tablishing raw text boundaries based on non-verbal signs (pauses) and contextual information for the next steps. Stage 2: Generation of structured text The utterances, along with labels, are concatenated and passed to the LLM, which generates coher- ent, punctuated, paragraph-structured text. This step allows the model to correct ambiguities – for example, a pause-final utterance initially marked as a full stop may become a comma when the follow- ing utterance begins a continuation clause, or a new paragraph may be inserted where a shift of topic is detected. In addition, the model adds punctuation within an utterance where two clauses run together in speech without a pause. However, the generated text may differ lexically from the original (added words, dropped particles, paraphrases). Such differences are text noise that is cleaned in the next stage. Stage 3: Merge with original tokens A se- quence matching algorithm (Ratcliff and Metzener, 1988) is performed between the generated tokens and the original tokens. The alignment walks through the tokens in order, identifying the position of each punctuation mark and paragraph break rel- ative to the nearest preceding original token. Then, only punctuation and paragraph markers are ex- tracted and inserted at the corresponding positions in the original token stream. FAdditional Details on the Experimental Setup A list of the evaluated language models and their sizes is provided in Table 8. The sizes of GPT- 5.4 and Qwen3.6-Plus models are not publicly dis- closed. For most models, we used OpenRouter to perform inference, resulting in a total cost of ap- proximately $200. However, not all models were available on this platform, particularly the Arabic- centric ones. For those, we ran inference on a single NVIDIA RTX 6000. FamilyModel IDShort NameSize (B)ArEn 1MBZUAI-ParisAtlas-Chat-2BAtlas-Chat-2B2 • 2MBZUAI-ParisAtlas-Chat-9BAtlas-Chat-9B9 • 3hammh0aHala-9BHala-9B9 • 4MBZUAI-ParisNile-Chat-4BNile-Chat-4B4 • 5inceptionaiJais-13B-ChatJais-13B-Chat13 • • 6Qwenqwen3.6-plusQwen3.6-PlusN/A • • 7QwenQwen2.5-7B-InstructQwen2.5-7B7 • • 8CohereLabsc4ai-command-r7b-arabic-02-2025Command-R7B7 • • 9silma-aiSILMA-9B-Instruct-v1.0SILMA-9B9 • • 10googlegemma-4-31B-itGemma4-31B31 • • 11openaiGPT-5.4GPT-5.4N/A • • Table 8: A list of the evaluated language models and their sizes. G Final dataset stats DialectInitial PairsFlaggedFixDiscardNo ChangeFinal Pairs EGY100041 (4.1%)17123999 MOR100031 (3.1%)2038997 KSA100055 (5.5%)27127999 SYR100060 (6.0%)331611984 UAE100022 (2.2%)5215998 Total5000209 (4.2%)10223844977 Table 9: Post-review reconciliation of the total number of QA pairs before and after review per dialect. H Genre Descriptions GenreDescription Personal DevelopmentFocuses on everyday living, well-being, hobbies, and individual growth. SportsAthletic activities, organized games, and competitive event analysis. HistoryDedicated to exploring and preserving historical events and cultural backgrounds. InterviewsFirst-person accounts, direct human interaction, and conversational dialogue. DocumentaryNon-fiction, real-world observation, and deep-dive reporting. Arts, Culture & EntertainmentCreative performances, cultural trends, and media critique. Science, Tech & InnovationExplores the natural world, scientific inquiry, and technological advancements. EducationEducational texts, instructions, and publicly available information sharing. Business & EconomyCommerce, financial markets, macroeconomic theories, and business stories. OtherNews (Politics, Society), fiction (Drama, Comedy, Romance, Youth) etc. Table 10: YouTube video genres in EDRAC. 37 18 8 777 6 5 3 2 Genre Distribution - Egyptian Sports Personal Development Interviews Business & Economy Science, Technology & Innovation Arts, Culture & Entertainment Education Other Documentary History 39 23 13 10 6 5 4 Genre Distribution - Moroccan Personal Development Documentary Sports Other Arts, Culture & Entertainment History Interviews 31 27 11 10 9 4 3 22 1 Genre Distribution - Saudi Interviews Other Arts, Culture & Entertainment History Documentary Sports Personal Development Science, Technology & Innovation Business & Economy Education 25 2323 15 5 3 22 1 Genre Distribution - Syrian Personal Development Sports Other Interviews Arts, Culture & Entertainment Science, Technology & Innovation Documentary History Business & Economy 20 19 12 11 9 8 66 5 4 Genre Distribution - Emirati History Sports Interviews Personal Development Other Education Documentary Science, Technology & Innovation Arts, Culture & Entertainment Business & Economy Figure 7: Genre distribution by dialect: Egyptian, Moroccan, Saudi, Syrian and Emirati. I Channels per dialect Dialect List of Channels EGY Ahmed Waheed, Al Nahar Drama, El-Podcasters - P Q É AøXÒJ . À @, Film Gamed, Kareem Esmail, MahmoudIsmailTV, Mamdouh NasrAllah, Omar Khaled - YÀ A g Q ‘ ́ , Pharmastan - ‡ A JÉ A”P A Ø, ReadTube - @Q ÆK ...J k . , STUDIO 77, Sarah Abouelkhair,Sarah Hany, Sherif Nabil - ® A JK . · Ǫ @, Waleed - YJ À, joo sport ©K QÂÑÀ @ ̇ Ø ËP Ò∫À @, ...J À @ ë È JÉ @, ̄ X Aí J Ø B @ Q . j÷ œ @ - Mokhbir Eqtisadi, ̇Ê ÑJ . k ̄ X A É - AJ Ø @P Òª t ' P A K, ·K C K @ Ä P X, H A J K @Ò Ç ́, IÉ AøXÒK . —Î A Ø, Q m à '@ ÒK . @ ÈJ . Î Heba Abo Elkheir MOR BarbaRoss Hicham – A ÇÎ Ä P AK . QK . , Farouk Life , Foot Maroc | º P A” H Ò Ø, Hassan El Fad | Y ÆÀ @ ·Çk, IBra Traveler, OTMAN HANA / A K Ag ‡ A“ J ́, Said Naciri ̄ QÂï A JÀ @ YJ ™É, Yassin Haro, ahssan patissier, elbachir L3aouni ̇ G Ò ™À @ Q Ç . À @, soumamalak, ȔҢ Ø ·K . @ Ibnfatoma , ÈJ K . Q ™” ÈÍ∫ K KSA Alaab - H . A™À @, SBC Channel, ÈK XÒ™ÇÀ @, ≠K A Ø ̄ AÎ | Highfive SYR È K P ÒÇÀ @ È K A K . , Al Mashhad Light IK B YÓD Ñ÷ œ @, Barhom m3arawi - ̄ @Q™” – ÒÎQK . , Boshe Tv, CHEF OMARQ‘ ́ ≠J É, Donia Stories - Èí ÆÀ AÎ ̇ Õ Ò™÷ fi Ö @, Ghaith Marwan ‡ @ Q ” IJ ́, LTV ‡ ÒK Q Æ K, Majed Haidar - P Y J k Yg . A”, NewDose - Ä XÒJ K, Rano’s Home , Sama Art International | ̇ Ê Æ À @ h . A J K C À È” AÉ, Sara alwari ® P ÒÀ @ ËP AÉ, Sherin Amara, Syria TV , Syrian kitchen, Yala Story Plus, ÈJ K A JÀ @ - Althania, Qk . Ò Ø ̄ P ÒÇÀ @ ≠K QÀ @, ̇ ◊ A ÇÀ @ ≠J ÇÀ @ - ShamiChef , · ÷ fi Ö A K » Cg . | Jalal & Yasmin , – ÒÎ È J Ík . Johina Home, A“J ÇÀ @ IK X @Òk, Q“J k . ̄ P ÒÉ - Syrian Gamer, – ÒJ À @ I . g Ë A J Ø Halab Today TV UAE 4042 Studios, Abu Dhabi TV ̇ Ê . £ÒK . @ Ë A J Ø, Hamdan Al-Ali, UAE FALCONS FEDERATION, ̇ G . X ‡ ÒK Q Æ K- Dubai TV, ̄ YÀ A m à '@ YÀ A g Khaled Alkhaaldi, A KP Ranoy7, ̇ G . X A ÷ fi Ö - Sama Dubai, ËQ Æ ¢À @ Ë A J Ø, H @ Q À @ Z AJ k B Y“m ◊ ·K . ‡ @ Y‘ g QªQ” Table 11: Final list of YouTube Channels by Dialect. This is the original guidelines PDF that was given to the annotators. Transcribed Speech Annotation Guidelines To ensure consistency across all annotators, we developed Transcribed Speech Annotation Guidelines. The guidelines provide step-by-step instructions for annotations and highlight the importance of listening to the audio while reviewing the text. This is crucial to accurately capture the dialectal pronunciation of words that are orthographically similar to MSA. Annotators were instructed to skip passages with large missing segments of transcriptions and to adhere to a minimal post-editing policy with the goal of ensuring the text readability and comprehension. Prior to the annotation task, a group training session was held to present the guidelines to the annotators and explain the task in detail. Discussions were based on Egyptian examples as it is commonly understood among speakers of other dialects. Special attention was paid to the Arabic Orthographic Post-editing section of the guidelines that provides specific instructions on handling spelling variation among speakers. This task requires you to review transcriptions of videos created by an automated transcription tool. Segments of YouTube videos have been transcribed into passages. You will need to listen to the video and read the context of the passage fully before determining what minimal fixes are required to make the text readable and understandable. Resolving ambiguous cases that can be (minimally) acceptable is encouraged. 1. Preparation 1. Open the YouTube link in column A. 2. Adjust playback speed to ~60% for accurate listening. 3. For a first pass, read the corresponding automated transcription for each paragraph in column A while listening to the audio. If a passage is missing larger chunks or entire sentences, then you should skip the entire passage and flag it in the name of the tab as Skipped. 4. Column B is a version of the transcription, where each word/token is presented on a separate line 5. Column C contains a copy of column B. 6. Proceed with the annotation task of post-editing the content of column C where necessary, following the guidelines below. 7. If a word that was not spoken was erroneously inserted into the transcription then remove it and leave the field in column C empty. 8. If a word needs to be fixed/ replaced, edit it in Column C. 9. If a word or words are missing, please insert them in the same cell of the previous word (see screenshots of the annotation process at the end of this document). If the missing words are at the beginning, insert them in the cell of the first word. General Guidelines 1. Follow the Arabic Orthographic Post-editing Guidelines closely 2. At times, you will find that the automated transcription incorrectly defaults to a MSA word. In these cases, conserve the speaker’s words (e.g., “بدنا” not “نرید”). 3. Include filler words if presented in speech, for example, (ي ِ ن ْ ع َ ی = it means, ( ّ س َ ب ) = just, only, but, (ى ْ ق َ ب) = to become; or as an adverb means already, ( ً صلا َ أ ) = anyway) 4. There is no need to try to transcribe onomatopoeic sounds. Instead, insert a description of the sound in curly brackets, e.g., baby crying, man laughing, etc. 5. Orthographic variation is expected across annotators, however, it is advised to maintain consistent spelling for recurring dialectal terms within one passage. 6. Play back any uncertain segments to confirm accuracy. 7. If the passage is missing some words or parts of a phrase, you can go ahead and add them.. 8. It is strongly advised to slow down the speed of the video to help you to correctly identify potential errors in the transcription. J Transcription Guidelines Guidelines for Arabic Orthographic Post-editing of Automatic Speech Recognition Texts The annotation data we focus on here is the output of automatic speech recognition (ASR) of Dialectal Arabic (DA) audio recordings. Since DA has no official orthography standards, Arabs writing in DA tend to use a set of variant spellings that either capture the utterance phonology primarily or are inspired by Modern Standard Arabic orthographic decisions (etymological or historical spelling) (Habash et al., 2018). While proposals to conventionally normalize dialectal spelling in NLP contexts have been proposed, e.g., CODA - (Habash et al., 2018), we opt here to take a simpler route that only addresses unacceptable ASR output, and apply a minimal post-editing policy to make the post-edited text comprehensible, while mimicking what a person might actually produce naturally. We only label unacceptable ASR output as “errors” and refer to other spelling decisions that vary from MSA or CODA standards as “variants”. The two types of phenomena that will most likely capture the attention of a human post-editor performing this task are: 1. Acceptable Variants – These do not affect the understanding of the passage and do not require post-editing. 2. Unacceptable Errors – These impact readability and comprehension and must be post-edited by the annotators to minimally change into an Acceptable Variant form. 1. Acceptable Variants (Do Not Require Post-editing) The following is a motivating example from Habash et al., (2018) showing 20+ spellings of the Egyptian Arabic word /mabi’ulha:š/ ‘he does not say it’ and their frequencies from a Google Search (September 29, 2017). Since all of these variants clearly indicate the intended meaning, they are all acceptable variants for the purpose of our task at hand. Note that the variations include different decisions on how to represent the glottal stop whose MSA cognate is /q/ ق, shortened long vowels, attaching or detaching the negation proclitic mA/m, etc. We present below a list of common orthographic phenomena that we consider acceptable (unless in some context, they can lead the reader astray). Some of these are the result of letter shape similarity and other phonological similarities. Not all “similar letters” are acceptable variants: if they are visually similar but phonologically not similar, they are most likely errors. There is also the issue of commonality: some spelling variants are so common that they are acceptable because they are recognizable. Variant Set Name Variant Characters Acceptable Variants Examples Ta Marbuta /Ha ه / ة قارة أفریقیا الغنیھ / قارة أفریقیا الغنیة Alif Maqsura/Ya ي/ى إزیك یا على! / إزیك یا علي! سلملي علي الجیران! / سلم لي على الجیران! Alif Hamzas أ/إ/آ/ا اھلأ یاجدعآن / أھلا یا جدعان Hamza Forms ؤ/ئ/ء/إ/أ علماؤھم / علماءھم / علمائھم Etymological Variants ق / أ/إ/ؤ/ئ/ء أنا قلبي دلیلي / أنا ألبي دلیلي Etymological Variants ت / ط ھادا الفسطان حلو / ھادا الفستان حلو Etymological Variants ت/ث حبیتك كثیر / حبیتك كتیر Etymological Variants ز/ض الزابط واقف ھناك / الضابط واقف ھناك Etymological Variants د/ذ/ز دنب الكلب / ذنب الكلب ذنبك على جنبك / زمبك على جمبك Splits و،ب،ك، ل، ح ح نشوف و ح نعرف / حنشوف وحنعرف Merges ما ، یا، لا ماشفت / ما شفت یامحمد / یا محمد لاتحكي / لا تحكي Shortening & Lengthening و/ ، ا/ ،ي/ (long vowel spelling variant of short vowel spelling via elided diacritics) شوفت الي عملاه ؟ شفت الي عاملاه؟ It is not possible to create a full, exact list of acceptable variants for all possible contexts, as some contexts can lead the reader to a different unintended meaning. For example, while Ta Marbuta and Ta Mamduda may be interchangeable safely in contexts like جمعیة الحارة / جمعیت الحاره, they may not be in limited contexts like حركة الجدعان / حركت الجدعان. 2. Unacceptable Errors (Require Correction by Post-editing) These errors hinder readability and must be post-edited to make them Acceptable Variants. The general pattern is that the written form is so abnormal and so confusing that a reader cannot reasonably recover the intended meaning. We list below the elementary types of these errors and then exemplify their combinations, which are numerous. Error Type Description Original text Corrected text Elementary Character Omission The omission of ر in this instance makes the word unrecognisable افریقیا بتحتوي على اكث من 3000 Africa has mor than 3000 افریقیا بتحتوي على اكثر من 3000 Africa has more than 3000 Elementary Character Insertion The added Alif ا is confusing یا اریت نشوفك السنة الجایة I hope we see you next year یا ریت نشوفك السنة الجایة I hope we see you next year Elementary Character Substitution Transcription uses wrong letters which leads to different meaning في مجلس جدید منتخب في الزمالك بقیاده المستشار جلال ابراھیم. مستشار جرار ابراھیم في الاول قال لھم ھنجیب لھ دولارات منین؟ There is a newly elected board at Zamalek football Club, led by Counselor Galal Ibrahim. At first, Counselor Garar Ibrahim asked them, "Where are we going to get the dollars from?" في مجلس جدید منتخب في الزمالك بقیاده المستشار جلال ابراھیم. مستشار جلال ابراھیم في الاول قال لھم ھنجیب لھ دولارات منین؟ There is a newly elected board at Zamalek football Club, led by Counselor Galal Ibrahim. At first, Counselor Galal Ibrahim asked them, "Where are we going to get the dollars from?" Elementary Word Splitting ھیتكیف is a single word, which once split breaks the meaning. والي مش ھیتك یف مع البیئھ الي حوالیھ ھیحصل في ایھ؟ And what will happen to the one who doesn't ad apt to their surroundings? والي مش ھیتكیف مع البیئھ الي حوالیھ ھیحصل في ایھ؟ And what will happen to the one who doesn't adapt to their surroundings? Elementary Word Merging عن الكوره is two separate words and should not be joined together, even in dialect ھنتكلم عنالكوره بشكل مختلف خالص یلا بینا. We're going to talk aboutfootball in a completely different way. Let's start! ھنتكلم عن الكوره بشكل مختلف خالص یلا بینا. We're going to talk about football in a completely different way. Let's start! Elementary Incorrect Punctuation Automatic punctuation addition leading to erroneous interpretation. انا مامنش علیھ معبو، ھامن علیھ. معاك یا بعجر، I don’t even trust his own father to be with him; why should I trust. him being with you, Ba’agar انا مامنش علیھ مع ابوه، ھآمن علیھ معاك یا بعجر، I don’t even trust his own father to be with him; why should I trust, him being with you, Ba’agar Elementary Incorrect Speaker Assignment Automatic speaker diarization is incorrect. ]45 - 62[ ** 1 المتحدث ** حق طلباتھ ایھ یا حمزاوي بیھ؟ ھو عایز حقھ. انا ما عندیش مانع بس یطلق الاول. ھي **Speaker 1 **[45 - 62] What does he want, Mr. Hamzawy? He wants his rights. I have no objection, but he has to divorce first. Her [45 - 62] ** 1 المتحدث ** حق طلباتھ ایھ یا حمزاوي بیھ؟ ھو عایز حقھ. انا ما عندیش مانع بس یطلق الاول. [45 - 62] **Speaker 1 ** What does he want, Mr. Hamzawy? He wants his rights. I have no objection, but he has to divorce first. ]61 - 65[ ** 2 المتحدث ** دي بقى المشكلھ الي مكن تسب لنا وجع. **Speaker 2** [61 - 65] Is a problem that could cause us hurt. ]64 - 67[ ** 1 المتحدث ** الدماغ وجع الدماغ في ایھ یا حمزاوي بیھ؟ **Speaker 1** [64 - 67] The headache the headache. What is going on Hamzawy Bey? [61 - 65] ** 2 المتحدث ** ھي دي بقى المشكلھ الي مكن تسب لنا وجع.الدماغ **Speaker 2** [61 - 65] It is a problem that could give us a headache. [64 - 67] ** 1 المتحدث ** وجع الدماغ في ایھ یا حمزاوي بیھ؟ **Speaker 1** [64 - 67] What headache would it give you, Mr. Hamzawy? Complex Error Character insertion and Character substitution Invalid/ jumbled spellings, as well as word splitting. This spelling corruption can often occur when words are incorrectly split. وجیل لاعیبھ بعد كده بقت لاعیبھ كبیره جدا زي والد صلاح الدین And a generation of players later emerged, becoming top-level stars, like Walld Salah El-Din. وجیل لاعیبھ بعد كده بقت لاعیبھ كبیره جدا زي ولید صلاح الدین And a generation of players later emerged, becoming top-level stars, like Walid Salah El-Din. Complex Error Word Splitting and Character insertion ھیتكیف is a single word that cannot be split, even in dialect والي مش ھیتك فیف مع البیئھ الي حوالیھ ھیحصل في ایھ؟ And what will happen to the ones who don’t ad appt to their surroundings? والي مش ھیتكیف مع البیئھ الي حوالیھ ھیحصل في ایھ؟ And what will happen to the ones who don’t adapt to their surroundings? Complex Error Word Splitting and Character substitution Invalid/ jumbled spellings, as well as word splitting. This spelling corruption can often occur when words are incorrectly split. فھس اناى لما اشتري كامیرا كویسھ So A will wa ait until I buy a good camera فھستنى لما اشتري كامیرا كویسھ So I will wait until I buy a good camera Complex Error Resolvable Meaningless or Illogical Sentences Some sentences lack coherence due to missing words and should be corrected. In this case, annotators should relisten to the original video carefully and fill in the missing words to convey the correct meaning of the sentence. ! لھوي یا لھوي یا مسرتي یا ستك ستك كر یا! لھوي الحقوني الحقوني! Oh mu Gud! Oh mu Gud! Oh mu Gud! Oh mu Gud! My lady Karaya, somebody help me, somebody help me! ! لھوي یا لھوي یا مصبتي یا مصبتي ست كرام ة! لھوي الحقوني الحقوني! Oh my God! Oh my God! Oh My God! Oh my God! My lady Karama, somebody help me, somebody help me! Complex Error Unresolvable, Meaningless or Illogical Sentences Too many errors resulting from multiple speakers overlapping in the original audio. It would be quicker to transcribe this audio from scratch rather than یلا، امي ھات وحش نومي، ي خلي بالك منھ والنبي یا حاضر. حاضر بقى انا مامنش علیھ معبو، ھامن علیھ. معاك یا بعجر، انت بعجر كده، طب الحمد له. Passage should be flagged for removal. post-edit this automated transcription. Screenshots of the Annotation Process 1. Starting point 2. Check out the video 3. Edits are marked automatically 4. Edit types: Do not add any type names. This annotated screenshot is for explaining how different types are handled. This is the original guidelines PDF that was given to the annotators. Reading Comprehension Q&A Pair Evaluations Annotation Guidelines (Emirati Arabic) Introduction This task aims to evaluate the performance of LLMs in generating question-answer pairs in Emirati Arabic based on accompanying passages. You are provided with a passage written in Emirati Arabic along with 10 associated, automatically generated question-answer (QA) pairs. The passages are transcriptions of YouTube videos in Emirati Arabic. The segments provided for annotation correspond to the first six minutes of each video. The video link and the exact time intervals are included at the top of each passage. The ultimate purpose of the dataset we are creating is to evaluate LLM capabilities in handling dialectal varieties of Arabic. Figure 1. illustrates the content of the annotation sheet. A. A link to the annotation guidelines (i.e. this document) B. Video link C. Question Relevancy label D. QA Pair Naturalness label E. Answer Correctness label K Annotation Guidelines F. Corrected Answer field G. Comment field QA Pair Review Start by reading the passage while listening to the video. Next, read the question-answer (QA) pairs carefully. Each QA-pair has an assigned colour that corresponds to a highlighted span in the passage. The span is intended to help you find the answer in the passage. (Note, however, that the highlighted spans in the passage were automatically generated. As such, the spans are not always accurate, and the answer may lie elsewhere.) Bear in mind that the answer may be found in different ways: • The answer can be found in one of the sentences that has been automatically highlighted. • The answer can be found by piecing together information from multiple sentences (highlighted in the same colour). • The answer does not lie in the corresponding highlighted sentence(s), but instead elsewhere in the passage. The following passage presents the transcription of a video spoken in Emirati Arabic. The passage is predominantly in Emirati Arabic. You are required to assess the quality of the QA-pairs through (1) Question Relevancy, (2) QA Pair Naturalness and (3) Answer Correctness: 1. Question Relevancy 1.1. Relevant: The question is relevant to the content of the passage. Example (i) Q منو الجهات الي شكرها المتحدث على تنظيم وتقديم فعاليات مهرجان شتا حتا؟ EN. Which entities did the speaker thank for organizing and presenting the activities of the Hatta Winter Festival? 1.2. Irrelevant: The question is irrelevant in terms of the subject matter of the passage. Example (i) Q اشو أبرز الأشياء الي سيدي سمو الشيخ حمدان سواها عشان يدعم الشباب في دبي ؟ EN. What are the most notable things His Highness Sheikh Hamdan has done to support young people in Dubai? 2. QA Pair Naturalness QA pair naturalness reflects how well the generated QA pairs represent Emirati Arabic as it is naturally spoken by native speakers. Here, Emirati Arabic refers to all sub-varieties of the dialect. Naturalness should be evaluated with respect to both the question and the answer. If either of them sounds odd to Emirati speaker’s ear, such as being awkward, unnatural, or non-dialectal—then the entire question–answer pair should be marked accordingly. Variations Note that QA pairs may contain spelling variations that either capture the utterance phonology primarily or are inspired by Modern Standard Arabic orthographic decisions (etymological or historical spelling) (Habash et al., 2018). Acceptable variations are those that do not cause the meaning to change. Table 1 presents a list of common orthographic phenomena that we consider acceptable. Some of these are the result of letter shape similarity and other phonological similarities. Not all “similar letters” are considered acceptable variants: if they are visually similar but phonologically not similar, they are most likely errors. There is also the feature of commonality: some spelling variants are so common that they are acceptable because they are recognizable. For example, both الغنيه and الغنية are common spelling variants that are acceptable because they are recognizable. Note that it is not possible to create a full, exact list of acceptable variants for all possible contexts, as some contexts can lead the reader to a different unintended meaning. For example, while Ta Marbuta and Ta Mamduda may be interchangeable safely in contexts like جمعية الحارة / جمعيت الحاره (`neighbourhood association’) , they may not be in limited contexts like حركة الشباب/ حركت الشباب (`the guys’ movement’ vs `I moved the guys’). Variant Set Name Variant Characters Dialectal Variants Examples Ta Marbuta /Ha ه / ة قارة أفريقيا الغنيه / قارة أفريقيا الغنية Alif Maqsura/Ya ي/ى إزيك يا على! / إزيك يا علي! سلملي علي الجيران! / سلم لي على الجيران! Alif Hamzas أ/إ/آ/ا اهلأ ياجدعآن / أهلا يا جدعان Hamza Forms ؤ/ئ/ء/إ/أ علماؤهم / علماءهم / علمائهم Etymological Variants ق / أ/إ/ؤ/ئ/ء/ج أنا قلبي دليلي / أنا ألبي دليلي/ انا جلبي دليلي Etymological Variants ت / ط هادا الفسطان/البنتلون حلو / هادا الفستان/البنطلون حلو Etymological Variants ش/ ي شجر / شير Etymological Variants ت/ث/س حبيتك كثير / حبيتك كتير Etymological Variants ز/ض/ظ/ذ الزابط واقف هناك / الضابط واقف هناك Etymological Variants د/ذ/ز دنب الكلب / ذنب الكلب ذنبك على جنبك / زمبك على جمبك Splits و،ب،ك، ل، ح ح نشوف و ح نعرف / حنشوف وحنعرف Merges ما ، يا، ل ماشفت / ما شفت /مشفتش يامحمد / يا محمد لتحكي / ل تحكي Shortening & Lengthening /ا/ ،ي ، /و (long vowel spelling variant of short vowel spelling via elided diacritics) شوفت الي عملاه ؟ شفت الي عاملاه؟ Table 1: a list of acceptable orthographic variants ‘Naturalness’ is classified into the following labels and each QA-pair should be labelled accordingly. 2.1. Natural: The text reflects a written representation of typical speech spoken by a native speaker of Emirati Arabic without sounding foreign or disfluent. It can reflect a range of form Example (i) Q شو كانت الأمانة الي كلفوا فيها أهل حتا المتحدث؟ EN. What was the responsibility entrustment to the people of Hatta, the speaker? A كانت الأمانة هي توصيل أسمى آيات التهاني والتبريكات لصاحب السمو الشيخ محمد بن راشد آل مكتوم والقيادة باسم كل بيت إماراتي EN. The entrustment was to convey the highest congratulations and blessings to His Highness Sheikh Mohammed bin Rashid Al Maktoum and the leadership on behalf of every Emirati household. Natural code-switching between MSA and Emirati Arabic due to the lexical similarities is considered “natural”, as long as the text can be read fully in a dialectal voice or fully in an MSA voice. Common code-mixing where frequent MSA phrases or occasional borrowings from MSA is used for formality or terminology is also natural. Examples of MSA mixing - ٫الصداع النصفي الأغلبية الساحقة٫ م ُ كيلع ُ ملاسلا.Example (iv) illustrates a QA pair that is acceptable in both Emirati Arabic and MSA. The main difference lies in how the words are pronounced. Example (iv) Q كيف وصف المتحدث الأجواء في مهرجان شتا حتا؟ EN How did the speaker describe the atmosphere at the Hatta Winter Festival? A وصفها بأنها كلها سعادة وفرحة وراحة ومسرات، وفيها ألعاب نارية وأضواء وفعاليات ومطاعم ومحلات EN He described it as a place of pure happiness, joy, comfort, and pleasure, featuring fireworks, lights, events, restaurants, and shops. Additionally, you may come across QA pairs where a high register of Emirati Arabic is used. Although this form is more common in news and interviews and is not typically used in everyday conversation, it still reflects the linguistic nuances of the native dialect and should be labeled ‘natural’. Example (v) represents a QA pair that contains a form of Emirati Arabic that is commonly used in media to be understood by a wide range of people. Example (v) Q كيف وصف المتحدث شعور كل بيت إماراتي بالإنجاز الي صار؟ EN How did the speaker describe the feeling of every Emirati household regarding the achievement that took place? A وصفهم بأنهم متنومسين وفخورين وسعيدين EN He described them as proud, happy, and content. Note that a “natural” sentence may not always reflect how you may speak yourself, but if you were to hear the sentence, you would understand it fully and perceive it as Emirati Arabic. 2.2. Somewhat awkward: It sounds a bit awkward, but native speakers of Emirati Arabic could still say it. Example (vi) Q شو يقصد المتحدث بلايسحبون فلوسك هو برنامج مسابقات ول هوما EN What does the speaker mean by ‘they don’t withdraw your money’? Is it a competition show, or what? A برنامج مسابقات بس يسولف يوم يقول المندوس مندوسك ل يسحبون فلوسك EN It’s a competition program, he is just chatting casually when he says ‘Al-Mandous is yours, they don’t withdraw your money. 2.3. Unnatural: The text contains disfluencies that would not be used by native speakers. It sounds like artificially created text or something a non-native speaker would say. Example (vii) Q كيف وصفت المتحدث الأجواء داخل مهرجان شتا حتا؟ EN How did the speaker describe the atmosphere inside the Hatta Winter Festival? A وصفوها بأنها كلها سعادة وفرحة وراحة ومسيرات وسيارات، وفيها ألعاب نارية وأضواء ودرونات وفعاليات ومطاعم ومحلات EN They described it as full of happiness, joy, and relaxation, with parades and cars. It also features fireworks, lights, drones, events, restaurants, and shops 2.4. Non-dialectical: The text is fully or predominantly written in MSA or in any dialect other than Emirati Arabic and there is no circumstance in which the text can be read in Emirati Arabic voice without sounding awkward. Example (viii) Q كيف وصف المتحدث شعور كل بيت إماراتي بالإنجاز الذي حدث؟ EN How did e speaker describe the feeling of every Emirati household regarding the achievement? A وصفهم بأنهم متنومسين وفخورين وسعيدين. EN He described them as proud, happy, and content. 3. Answer Correctness 3.1. Correct: a. The answer is factually accurate and supported directly by the passage. Example (ix) Q كم برنامج تلفزيوني قدمته مؤسة دبي لإعلام من حتا؟ EN How many television programs did the Dubai Media Incorporated present from Hatta? A قدمت خمس برامج تلفزيونية من حتا . EN Five television programs were presented from Hatta. OR b. The answer is concluded or inferred from information in the passage. Example (x) Q إشو الي يخلي أهل حتى تحس بهالسعادة؟ EN What makes the people of Hatta feel such pride and happiness? A لفوز الشيخ محمد بن راشد بسيف الشرف ،وفخرهم بكل الي حقه EN Due to Sheikh Mohammed bin Rashid’s winning of the Sword of Honor, and their pride in all that he has accomplished 3.2. Incorrect: a. The answer is factually wrong or incomplete. Example (xi) Q منو الجهات الي شكرها المتحدث على تنظيم وتقديم فعاليات مهرجان شتا حتا ؟ EN Who did the speaker thank for organizing and delivering the Hatta Winter Festival events? A صاحب السمو الشيخ محمد بن راشد آل مكتوم ،وسمو الشيخ حمدان بن محمد بن راشد آل مكتوم ،والشيخة هند بنت جمعة آل مكتوم ،والشيخ محمد بن راشد بن محمد بن راشد بن أل مكتوم EN His Highness Sheikh Mohammed bin Rashid Al Maktoum; His Highness Sheikh Hamdan bin Mohammed bin Rashid Al Maktoum; Sheikha Hind bint Juma Al Maktoum; and Sheikh Mohammed bin Rashid bin Mohammed bin Rashid Al Maktoum If the answer is incorrect, and available in the passage, please provide the correct answer in the Corrected Answer column. Example (xii) Q منو الجهات الي شكرها المتحدث على تنظيم وتقديم فعاليات مهرجان شتا حتا ؟ EN. Who did the speaker thank for organizing and delivering the Hatta Winter Festival events? A شكر براند دبي، الذراع الإبداعي لمكتب الإعلامي لحكومة دبي، ومؤسة دبي لإعلام وقنواتها الي قدموا برامج تلفزيونية من حتا EN He thanked Brand Dubai, the creative arm of the Dubai Government Media Office, and Dubai Media Incorporated and its channels that presented television programs from Hatta. 3.3. Invalid: a. The answer fails to address the question. Example (xiii) Q شو علاقة "براند دبي "بالمكتب الإعلامي لحكومة دبي ؟ EN What is the relationship between ‘Brand Dubai’ and the Dubai Government Media Office? A مهرجان التسوق في دبي EN Shopping festival at Dubai OR b. The question requires additional information (not in the passage) that is not common knowledge among the majority of native speakers of this dialect. Example (xiv) Q تاراملإا ديحوت لبق اتح ةقطنم نكست تناك لئابقلا ّ يأ؟ EN Which tribes lived in the Hatta area before the unification of the UAE? A قبل توحيد الإمارات ،كان يسكن حتا ضعبو ساي ينبو ةيلا ُ علا ةليبق العوائل المحلية EN Comments Section Please add notes here if you need to highlight unclear cases or your motivation for a particular labelling decision. L Annotators’ Demographics IDDatasetNativeResidenceAgeGenderDegreeTaskBackground A1EGYEGYEGY30sFBAA, RLA, PT A2EGYSYRSYR40sFPhDA, RCT A3EGYEGYEGY30sFBAALA, RA, CT A4EGYEGYEGY30sFPhDA,RL, CL A5EGYEGYEGY40sFMAA,RL, AL A6MORMORMOR30sFBAARA, LA A7MORMORMOR30sFBAA, RLA A8MORMORMOR40sMPhDALA, RA, CT A9MORMORMOR30sMMBAA, RCT, PT A10MORMORFR30sFPhDA,RCS A11MORMORMOR30sFBAA,REL A12KSASYRSYR40sMBAA, RLA, RA, CT A13KSASYRSYR40sFBAA, RLA A14KSASYRSYR50sFBAALA, PT A15KSAKSAKSA30sFPhDA,RTR A16KSAKSAKSA30sFPhDA,RCL A17SYRSYRSYR40sMBAALA, RA, CT A18SYRSYRSYR20sFBAA, RCT, RA A19SYRSYRSYR30sFBAA, RCT, PT A20SYRSYRSYR40sFBAALA, RA, PT A21SYRSYRUK40sMMAA,RLA A22SYRSYRSYR20sMMSA,RCS A23UAEPALUAE20sMBAA, RLA A24UAEPALUAE40sMBAALA A25UAEPALUAE40sFBAA, RLA A26UAEUAEUSA40sFBAA,RLA A27UAEUAEUAE20sMBAA,RCS Table 12: Annotators’ demographics and roles for the transcription correction task. There are two roles: Annotate (A), and Review (R). The annotators’ background experience includes: Certified Teacher (CT), Private Tutor (PT), Linguistic Annotator (LA), and Research Assistant (RA). All annotators are native speakers of Arabic, and worked on their native dialect or dialects of a country where they have resided for more than 15 years (if they worked on a different dialect). IDDatasetNativeResidenceAgeGenderDegreeTaskBackground A1EGYEGYEGY30sFPhDA,RL, CL A2EGYEGYEGY40sFMAA,RL, AL A3MORMORFR30sFPhDA,RCS A4MORMORMOR30sFBAA,REL A5KSAKSAKSA30sFPhDA,RCL A6KSAKSAKSA20sFBAA,RCS A7SYRSYRUK40sMMAA,RLA A8SYRSYRSYR20sMMSA,RCS A9UAEUAEUSA40sFBAA,RLA A10UAEUAEUAE20sMBAA,RCS Table 13: Annotators’ demographics and roles for the QA annotation task. There are two roles: Annotate (A), and Review (R). The annotators’ educational background includes: Linguistics (L), Computational Linguistics (CL), Applied Linguistics (AL), English Linguistics (EL), Translation (TR) and Computer Science (CS). All annotators are native speakers of Arabic, and worked on their native dialect or dialects of a country where they have resided for more than 15 years (if they worked on a different dialect). TaskPayment in USD Transcription Correction$3,540.0 QA Annotation$4,330.0 Table 14: Annotators were paid $15 an hour for each one of the tasks. In the table above, we provide the total amount of money in USD paid to the annotators per task. M IAA For QA Cohen KappaSimple Agreement P o P e Kappa(x)Majority DisagreementMajority AgreementPABAK-OS EGYRelevancy1.001.00undefined0.001.001.00 Naturalness0.930.930.000.080.930.93 Correctness0.970.970.000.030.970.97 MORRelevancy1.001.00undefined0.001.001.00 Naturalness0.970.970.000.030.970.97 Correctness0.980.980.000.010.980.99 KSARelevancy1.001.00undefined0.001.001.00 Naturalness0.920.900.150.080.910.94 Correctness0.990.990.000.020.980.99 SYRRelevancy1.001.00undefined0.001.001.00 Naturalness0.990.990.000.010.990.99 Correctness0.800.730.260.190.800.97 UAERelevancy1.001.00undefined0.001.001.00 Naturalness0.810.82-0.060.170.820.65 Correctness0.980.970.330.010.980.99 Table 15: Inter-Annotator Agreement for QA Annotations. K is ‘undefined’ when P o = P e =1. N Example of a flagged issue DialectQuestionAnswer QA Pair Naturalness Comments on the label by annotators Suggested Correction EmiratiSomewhat Awkward theword' (Because) is not Emirati What was the first thing the speaker did to start collecting items for his museum? He asked his mother about his deceased father's belongings, such as his weapon and dagger, so that she could give them to him. He asked his mother about his deceased father's belongings, such as his weapon and dagger, because he wanted them. . ' . سواه شي أول شو يبدأ عشان المتحدث متحفه؟ مقتنيات يجمع ضأغرا عن أمه سأل سلاحه مثل المتوفى، أبوه تعطيه عشان وخنجره، إ ياها عشان ضأغرا عن أمه سأل سلاحه مثل المتوفى، أبوه يباهم لانه وخنجره، Figure 8: Example of a flagged issue in Emirati Arabic by our annotators. This is the original guidelines PDF that was given to the annotators for Egyptian Arabic. Human Evaluations Guidelines – Egyptian Arabic Task Overview This task aims to evaluate the performance of LLMs in answering open-ended questions in Egyptian Arabic. At this stage we only provide the automatically generated answers (without the questions). The ultimate purpose of the dataset we are creating is to evaluate LLM capabilities in handling dialectal varieties of Arabic. For each answer, you will assess the automatically generated answers on two dimensions: (1) accuracy and (2) naturalness. You will be provided with a spreadsheet that contains two tabs. The first is labeled as Model_1, and the second tab is labeled Model_2. The IDs and the reference answers are the same in both tabs, while the generated answers differ because they were generated by two different models (Model_1 and Model_2). In each tab, you’l find five columns: A. ID: The passage ID (you can disregard this) B. Reference Answer (gold/correct answer) C. Generated Answer (LLM generated answer by Model_1 or Model_2) D. Accuracy (correct/incorrect answer) E. Naturalness (natural, somewhat awkward, unnatural, or non-dialectal) You’l be evaluating 40 answers in total (20 per model). See detailed definitions of Accuracy and Naturalness below. You are going to evaluate the generated answers on the dimensions: (1) Accuracy (col. D) (2) Naturalness (col. E). As defined below: 1. Accuracy Each row includes two answers. First, read the reference answer (col. B) and the generated answer (col. C). Then for each generated answer, determine the correct label correct or incorrect or unsure: An answer can be labeled as 'Correct’ - if it is: a. factually accurate based on the reference answer Example (i) A يعتبره شعور طبيعي لأنه فطرة، وبشرط إنه يكون معتدل ومؤقت وي على فترات. B يعتبر المتحدث الغضب ا ً روعش ا ً يعيبط وغير مذموم **إذا كان ً لدتعم ا ً تقؤمو ويحدث على فترات** ، مثل أن يرى الإنسان ا ً فقوم يزعجه فيغضب بشكل طبيعي ثم يزول هذا الغضب. EN for A. He considers it a natural feeling because it is instinctive, provided that it is mild, temporary, and y at intervals. EN for B. The speaker considers anger to be a natural and non-reprehensible emotion **if it is mild, temporary, and occurs at intervals**, such as when O Human Evaluation Guidelines a person sees a situation that bothers him, and he gets angry naturally, and then this anger disappears. b. a paraphrasing of the reference answer An answer can be marked ‘Incorrect’, if it is: a. factually wrong given the reference answer b. has missing information or is only partially correct (sometimes cut-off parts of the answers) Unsure – not clear which option to choose from, below are examples: a. mentions steps how it got to the answer b. Added information that is not clear 2. Naturalness In this part mark the generated answer as: natural, somewhat awkward, unnatural, non-dialectal, following the definitions below: a. Natural: The text reflects a written representation of typical speech as spoken by a native speaker of Egyptian Arabic without sounding foreign or disfluent. Natural Egyptian Arabic can reflect a range of formality levels (from ‘chatting to friends’ to ‘talking to a senior manager’). Example (i) A مستقبلات الدينوسين في الدماغ بتزيد عندهم EN. Adenosine receptors in the brain increase in number for them. Due to the lexical similarities, code-switching between MSA and Egyptian Arabic is considered “natural”. While a text can be read fully in an MSA voice, it is considered “natural” as long as it can be read fully in a dialectal Egyptian Arabic voice. Common code-mixing where frequent MSA phrases or occasional borrowings from MSA is used for formality or terminology is also natural. Examples of MSA mixing - الصداع النصفي٫ الأغلبية الساحقة٫ م ُ كيلع ُ ملاسلا. Example (i) illustrates an answer that is acceptable in both Egyptian Arabic and MSA. The main difference lies in how the words are pronounced. Example (i) A من 110 مل غرام ل 300 مل غرام EN From 110 milligrams to 300 milligrams. Additionally, you may come across an Answer where a high register of Egyptian Arabic is used. Although this form is more common in news and interviews and is not typically used in everyday conversation, it still reflects the linguistic nuances of the native dialect and should be labelled ‘natural’. Example (iv) represents an Answer that contains a form of Egyptian Arabic that is commonly used in media to be understood by a wide range of people. Example (iv) A فاعليتها بتقل EN Its effectiveness decreases Note that a “natural” sentence may not always reflect how you may speak yourself, but if you were to hear the sentence, you would understand it fully and perceive it as Egyptian Arabic. b. Somewhat awkward: It sounds a bit awkward, but native speakers of Egyptian Arabic could still say it. Example (v) A الكافين يازميلي هي المادة صاحبة اكتر مفعول نفسي بيتعاملوا معاها في العالم بحاله EN Caffeine, my friend, is the most widely used psychoactive substance in the whole world. c. Unnatural: The text contains disfluencies that would not be used by native speakers. It sounds like artificially created text or something a non-native speaker would say. Example (vi) A مادة الكافين ليه تأثيرات نفسي بيستخدمها العالم كله EN Caffeine substance is the most widely used psychoactive substance in the whole world. d. Non-dialectal: The text is fully or predominantly written in MSA or in any dialect other than Egyptian Arabic and there is no circumstance in which the text can be read in Egyptian Arabic voice without sounding awkward. Example (vii) A مادة الكافين هي المادة ذات التاثير النفسي الاكثر استخداما في العالم كله . EN Caffeine substance is the most widely used psychoactive substance in the whole world We provide a drop-down menu for each one of the categories. For each one of the dimensions, choose one of the available options. Here is an example of a spreadsheet: