Paper deep dive
Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urdu
Farah Adeeba, Abdul Rafae Khan, Rajesh Bhatt, Hassan Sajjad
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/11/2026, 3:55:36 AM
Summary
This paper evaluates the linguistic and cultural capabilities of three multilingual Large Language Models (GPT-5.1, Qwen-3-Max, DeepSeek-3.1) in generating Urdu stories. Using a novel dataset of 93 stories (Urdu-Stories), the authors find that while models produce surface-level fluent text, they suffer from significant grammatical errors, semantic incoherence, and cultural shallowness. Key issues include memorization of source texts, unnatural title repetition, gender agreement failures, and the imposition of Western cultural norms on Pakistani contexts. The study highlights that current LLMs lack deep cultural grounding and robust morphosyntactic understanding in low-resource languages like Urdu.
Entities (13)
Relation Signals (9)
Abdul Rafae Khan → affiliatedwith → Monash University Malaysia
confidence 95% · Abdul Rafae Khan Affiliation: Malaysia School of Information Technology, Monash University Malaysia
Hassan Sajjad → affiliatedwith → Dalhousie University
confidence 95% · Hassan Sajjad Affiliation: Dalhousie University, Canada
Rajesh Bhatt → affiliatedwith → University of Massachusetts Amherst
confidence 95% · Rajesh Bhatt Affiliation: University of Massachusetts Amherst, USA
Farah Adeeba → affiliatedwith → University of Konstanz
confidence 95% · Farah Adeeba Affiliation: University of Konstanz, Germany
LLMs → exhibitsweaknessin → Urdu
confidence 95% · Our notable findings suggest that LLMs often make basic errors of grammar and semantics. The stories lack coherence... and show pervasive cultural shallowness.
DeepSeek-3.1 → usedinstudy → Urdu-Stories
confidence 95% · We generate Urdu-Stories, a corpus of 93 stories generated using three contemporary LLMs (GPT-5.1, Qwen-3-Max, DeepSeek-3.1).
Qwen-3-Max → usedinstudy → Urdu-Stories
confidence 95% · We generate Urdu-Stories, a corpus of 93 stories generated using three contemporary LLMs (GPT-5.1, Qwen-3-Max, DeepSeek-3.1).
GPT-5.1 → usedinstudy →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multilingual large language models (LLMs) are increasingly used for open-ended text generation, yet their behaviour in low-resource languages remains poorly understood. In this work, we question how correct and reliable is the generation of multilingual LLMs when used for the task of story generation. We consider Urdu language as a representative low-resource language. We generate Urdu-Stories, a corpus of 93 stories generated using three contemporary LLMs (GPT-5.1, Qwen-3-Max, DeepSeek-3.1). We manually annotate the errors present in them under a nine-label linguistic, semantic, and cultural taxonomy. Our notable findings suggest that LLMs often make basic errors of grammar and semantics. The stories lack coherence, have unnatural repetition and show pervasive cultural shallowness. We further show using few-shot prompting that the cultural and context errors largely remain unresolved. Our findings highlight the limitations of current LLMs as a reliable source of content generation and information retrieval for low-resource languages.
Tags
Links
- Source: https://arxiv.org/abs/2609.10758v1
- Canonical: https://arxiv.org/abs/2609.10758v1
Trouble viewing inline? Open PDF directly →
Full Text
87,862 characters extracted from source content.
Expand or collapse full text
rm[ BoldFont=TeXGyreTermesX-Bold.otf, ItalicFont=TeXGyreTermesX-Italic.otf, BoldItalicFont=TeXGyreTermesX-BoldItalic.otf ]TeXGyreTermesX-Regular.otf [*arabic]rm[ BoldFont=Amiri-Bold.ttf, ItalicFont=Amiri-Italic.ttf, BoldItalicFont=Amiri-BoldItalic.ttf ]Amiri-Regular.ttf Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urdu Farah Adeeba Affiliation: University of Konstanz, Germany Affiliation: University of Massachusetts Amherst, USA Email: farah.adeeba@uni-konstanz.de Abdul Rafae Khan Affiliation: Malaysia School of Information Technology, Monash University Malaysia Email: Abdul.Rafae@monash.edu Rajesh Bhatt Affiliation: University of Massachusetts Amherst, USA Email: bhatt@umass.edu Hassan Sajjad Affiliation: Dalhousie University, Canada Email: HSajjad@dal.ca Abstract Multilingual large language models (LLMs) are increasingly used for open-ended text generation, yet their behaviour in low-resource languages remains poorly understood. In this work, we question how correct and reliable is the generation of multilingual LLMs when used for the task of story generation. We consider Urdu language as a representative low-resource language. We generate Urdu-Stories, a corpus of 93 stories generated using three contemporary LLMs (GPT-5.1, Qwen-3-Max, DeepSeek-3.1). We manually annotate the errors present in them under a nine-label linguistic, semantic, and cultural taxonomy. Our notable findings suggest that LLMs often make basic errors of grammar and semantics. The stories lack coherence, have unnatural repetition and show pervasive cultural shallowness. We further show using few-shot prompting that the cultural and context errors largely remain unresolved. Our findings highlight the limitations of current LLMs as a reliable source of content generation and information retrieval for low-resource languages. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of text generation tasks, yet this success remains concentrated predominantly in English and a few other handful of languages Joshi et al. (2020); Blasi et al. (2022). While many of these models are described as multilingual and are capable of generating text in numerous languages, their performance in most of these languages is insufficiently evaluated Hu et al. (2025); Shani et al. (2026). This gap is especially pronounced for low-resource languages, which collectively represent a substantial portion of the world’s linguistic diversity. Evaluating language competence demands more than measuring syntax, semantics and morphology or fluency of the generated text. Language is inseparable from the cultural values, social norms and shared historical experiences (Hershcovich et al., 2022). Consequently, a reliable language model must internalize these nuances to serve reliably as a tool for information retrieval and content generation. Without such grounding, generated content may appear superficially fluent while remaining semantically unstable, culturally shallow, or socially misaligned. The few existing evaluations of LLM creative writing have focused almost exclusively on English. Gómez-Rodríguez and Williams (2023) show that top commercial LLMs match or exceed human writers on fluency, coherence, and style in English short fiction, while humans retain an edge in originality and humour. Whether these conclusions transfer to a low-resource, culturally distant language is, to our knowledge, untested. Beyond English-centric creative writing, prior work on multilingual evaluation has primarily focused on benchmarking general language understanding Ahuja et al. (2023); Xuan et al. (2025) and translation quality Kreutzer et al. (2025); Zhu et al. (2024), mostly using English benchmarks as foundation Thellmann et al. (2024). Researchers have reported substantial performance gaps in low-resource languages; however, these studies have largely not examined long-form generative capabilities in depth Unjum et al. (2025); Anikina et al. (2025). Similarly, research on cultural and value alignment has shown that language models tend to reflect dominant cultural priors, but these studies have largely been limited to classification or preference tasks rather than open-ended narrative generation Tao et al. (2024); Santurkar et al. (2023). While recent work has explored automated evaluation and LLM-as-judge frameworks Gu et al. (2024), these methods have been shown to exhibit reliability issues and sensitivity to evaluation settings, particularly in more complex generation scenarios Schroeder and Wood-Doughty (2024); Thakur et al. (2025). In this work, we question how well contemporary multilingual LLMs are grounded in the linguistic and cultural dimensions of low-resource languages. We take Urdu as our case study and investigate the capacity of LLMs to produce stories that are linguistically correct, semantically meaningful, and culturally situated within the Pakistani context. Story generation is a deliberate design choice, unlike factual or short-form tasks, narrative generation demands contextual awareness, structural coherence, and cultural sensitivity sustained over longer text, making it a particularly revealing testbed for model capability. Concretely, we examine whether LLM-generated Urdu stories preserve contextual relevance, avoid formulaic narrative patterns, and handle culturally sensitive themes appropriately, whether those themes originate from Urdu literary traditions or are adapted from Western sources. This leads us to the following research questions: • Originality. Do LLMs generate original Urdu narratives, or reproduce remembered source texts, and how does this reproduction degrade as source-text visibility in training data decreases? • Linguistic and semantic accuracy. How accurately do LLMs handle Urdu-specific grammar and do their stories remain semantically coherent at the sentence and discourse level? • Cultural fidelity. To what extent do generated stories preserve Pakistani cultural context? • Narrative structure. What structural patterns characterise LLM-generated Urdu stories, and how do they differ across models? • Tractability. Which of the above failures can be remediated by targeted prompting? To study these questions, we introduce Urdu-Stories, a dataset of 93 stories generated using three contemporary LLMs: GPT-5.1, Qwen-3-Max, and DeepSeek-3.1. All stories are annotated by a native Urdu speaker using a thirteen-label taxonomy covering linguistic, semantic, and entity dimensions, with inter-annotator agreement verified by four additional native-speaker annotators on a stratified 60-story subset (κ=0.81κ=0.81). Our analysis reveals five consistent findings. First, models exhibit memorisation over creativity: when prompted with well-known literary titles, they frequently reproduce the original story rather than generating a novel narrative. Second, we observe unnatural title anchoring: models repeatedly insert the provided title into the story body to signal topical relevance. To quantify this behaviour, we introduce the Title Repetition Rate (TRR), which substantially exceeds the repetition patterns found in human-written stories. Third, we identify linguistic fragility, including spelling inconsistencies and grammatical instability. Fourth, models demonstrate semantic and register instability, producing factual contradictions, grammatically valid but meaningless sentences, and unprompted switching into Punjabi. Finally, we observe cultural shallowness, reflected in pervasive cross-script contamination and a 3.7:1 male-default protagonist bias. Together, these findings demonstrate that current multilingual LLMs remain fluent in Urdu at the surface level, yet fail to achieve deeper creativity, semantic robustness, and cultural grounding and, we expect, generalise to many other low-resource languages , underscoring the need for dedicated development effort beyond surface-level multilinguality.11 1 We release the dataset, annotations, and evaluation scripts to support future research on open-ended generation in low-resource languages. 2 Story Corpus Development 2.1 Story Title Selection We constructed a corpus of 31 story titles drawn from four different sources (Table 10): Translated Short Story Titles (TSST) from canonical English short fiction; Urdu Short Story Titles (UST) from canonical twentieth-century Urdu writers used verbatim ; Fables, short Indo-Pakistani moral tales used as a memorisation control; and Neutral Urdu Titles (NUT), author-original Urdu titles composed by the research team as an unseen-prompt control. The complete title list with authors, sources, and ALA-LC transliterations is given in Appendix C (Table 11). For TSST, we obtained an initial Urdu rendering of each English title via the GPT-5.1 translation API and then manually verified every translation with a native Urdu speaker; published Urdu renderings were retained verbatim where they exist. The four-way split is intentional: TSST tests cross-cultural transfer, UST tests reproduction vs. reinterpretation of canonical Urdu literature, Fable controls for plot memorisation, and NUT controls for memorisation in absence of any prior text. Following the leakage-control principle of Gómez-Rodríguez and Williams (2023), who tested LLMs on an epic combat between Ignatius J. Reilly and a pterodactyl, a scenario invented specifically so that no matching text could exist in any model’s training data, all six NUT titles were composed by the authors of this paper and do not correspond to any published Urdu literary work or publicly available source. Titles were designed to be contextually plausible within Pakistani settings (e.g. سوات کی شام, “Evening in Swat”) without being derivable from any known text, and no information about them was shared publicly before story generation. This ensures that stories generated under NUT prompts reflect the model’s generative behavior rather than reproduction of a memorized source. 2.2 Generation Protocol Three LLMs were evaluated: GPT (5.1), QWEN (3-Max), and DeepSeek (3.1), all accessed in November, 2025. Each title was issued to every model exactly once using a fixed zero-shot prompt: “Write a 2000-word Urdu story in a Pakistani cultural context. The title is دوسری عورت.” 2.3 Corpus Statistics The protocol produced 93 stories, totalling 151,965 words and 9,460 sentences (Table 1). Table 1: Per-model corpus statistics Statistic GPT QW DS Overall Stories 31 31 31 93 Avg words / story 1,587 1,382 1,934 1,634 Min 726 946 1,461 1,044 Max words 1,997 1,856 3,030 2,294 Std. dev. (words) 290 199 317 351 Total words 49,199 42,827 59,939 151,965 Avg. sentences / story 122.9 101.6 149.6 101.7 Total sentences 3,809 3,149 4,639 11,597 Length compliance (%) 58.1 12.9 80.6 50.5 2.4 Annotation and Inter-Annotator Agreement A native Urdu speaker annotated all 93 stories in Label Studio using thirteen labels across three categories (Table 2). Most labels are applied at the sentence level; entity labels (Character, City, Protagonist, ) are applied once per story. Agreement Case (gender, oblique-case agreement) is treated as a sub-type of Grammar Error; Semantic Anomaly marks sentences that are syntactically well-formed but semantically empty; Context Mismatch marks factual contradictions. A stratified subset of 60 stories (20 per model, balanced across title categories) was independently re-annotated by four additional native Urdu speakers (15 stories per annotator) . Cohen’s κ was computed using exact text+label match for PER/City, substring-tolerant match for protagonist labels , and same-label-in-same-sentence match for all other labels. Overall agreement was κ = 0.81 (substantial reliability); per-model and per-label scores are reported in Appendix H. The thirteen-label taxonomy itself was developed through an iterative inductive process: the annotation team first read a sample of generated stories, cataloged recurring error patterns, and progressively refined and merged categories and labels based on what was actually observed in the data, rather than adopting a taxonomy from prior theoretical work. Table 2: Annotation labels. G = granularity (U story / S Sentence). Label G Definition / Example Linguistic Grammar Error S Syntax احمد اور ثنا نے سب کے لیے تحفے لائے Agreement Error S Obliqueness . چھوٹی سی لنگر خانہ → چھوٹا سا لنگر خانہ Spelling Error S Misspelled Urdu word. مسئیبت → مصیبت Space Deletion S Two words fused, no space. چاہتیہوں → چاہتی ہوں Em-dash S Semantic Context Mismatch S Factual contradiction within story. “had no children …her son went to school” Semantic Anomaly S Grammatical but meaningless. آواز میں عجیب کانپ تھی Cultural Incons. S Non-Pakistani reference. چند اسٹیشن کے بعد، ٹرین کا ریفریشمنٹ وین کھلا Script Contamination S Non-Urdu script token (whole/partial). hospital; حajی Entities Character U Character name. حمزہ City U Place name. لاہور MaleProtag. U Main male character. حمزہ FemaleProtag. U Main female character. ثنا 3 Qualitative Analysis Several models generated grammatically correct Urdu but failed to preserve culturally authentic Pakistani dialogues and naming conventions. 3.1 Linguistic and Semantic Accuracy Manual annotation of all 93 generated stories identified quality issues across three dimensions: linguistic errors , semantic coherence failures, and Western-influenced cultural mismatches . Table 7 presents the complete breakdown. 3.1.1 Linguistic Errors Linguistic errors constituted the largest share of identified issues. Four subtypes were annotated: grammatical mistakes, spelling errors, space deletion, and em-dash. Grammatical errors were the most frequent category overall. QWEN exhibited the highest rate , followed by DeepSeek and GPT . Recurring patterns included incorrect postposition usage (نے، کو، سے), broken verb-subject agreement, and malformed compound sentences. To characterise the kinds of grammatical failures observed, we discuss three representative examples, each highlighting a different fault line in Urdu syntax. (i) Ergative case marker omission. In QWEN’s زرین خاموشی سے بندوق نیچے رکھ دی (Zarīn khāmōshī sē bandūq nīchē rakh dī, “Zarin quietly put the gun down”), the agent زرین requires the ergative postposition نے (nē) because the verb رکھ دی (rakh dī) is transitive and perfective. The same omission recurs in GPT’s output . Ergative نے marking is a hallmark of Urdu morphosyntax with no English analogue, and its absence suggests cross-lingual transfer from a nominative-only template. (i) Gender agreement violation. In GPT’s اس کے منہ سے آج کوئی غرور بھرا بھونک نہیں نکل رہا تھا (Us kē munh sē āj kōʾī ghurūr bharā bhaunk nahīṅ nikal rahā thā, “no proud bark came out of his mouth today”), the noun بھونک (bhaunk, bark) is feminine in Urdu and triggers feminine agreement on both the modifier and the verb. The model nevertheless propagates masculine agreement through the entire clause, indicating that grammatical gender is being assigned by surface frequency rather than by lexical entry. (i) Aspect mismatch in sequential narrative. QWEN’s اگلے دن، وہ سویرے اٹھا، نہاتا دھوتا، اور کوٹ پہنا اور دفتر پہنچ گیا (Aglē din, wuh sawērē uṭhā, nahātā dhōtā, ōvar kōṭ pahnā aur daftar pahuṅch gayā, “the next day, he got up early, bathed, put on the overcoat, and reached the office”) chains four events in a single sentence using past-perfective forms, but inserts نہاتا دھوتا (nahātā dhōtā, habitual imperfective). This aspectual slip is particularly diagnostic because it occurs within an otherwise well-formed sequence, suggesting that the model can produce isolated verb forms but struggles to maintain aspectual consistency across a narrative clause. Their recurrence across all three models suggests that these are not random mistakes but reflect a deeper gap in how Urdu morphosyntax is represented during generation. Agreement Error A substantial share of grammatical issues (cf. Table 7) involved local agreement failures within a 2–3 word window: gender agreement(adjective/verb mismatched to the head noun’s gender), the most common sub-type; cf. example (i) above) and oblique-case agreement ( a masculine singular noun or adjective failing to take its oblique form before a postposition e.g. سر کٹا سوار نے → سر کٹے سوار نے). Both are mechanical and strictly local, suggesting a lightweight post-editing pass over noun-phrase windows could correct a large share of them without retraining the model. Spelling errors were distributed across all three models, suggesting that spelling degradation in Nastaliq is model-agnostic and likely reflects tokenisation artefacts rather than model-specific weaknesses. Four recurring patterns were identified: hamza misplacement, vowel and diacritic confusion, visually-similar character substitution, and English-mediated retroflex confusion. QWEN dropped the hamza-on-yē in adjectival derivations (دریاوی for دریائی, “riverine”), while DeepSeek inserted a hamza into an Arabic loanword that does not take one (مسئبت for مصیبت, “calamity”). Vowel-marking errors such as DeepSeek’s میچس for ماچس (“matchbox”) and GPT’s جھلے ہوئے ہاتھ for جلے ہوئے ہاتھ (“burnt hands”) resemble the mistakes of a learner who has heard a word but not internalised its written form. The most diagnostic pattern is the retroflex series (ٹ, ڈ, ڑ), one of the few phonemic features of Urdu absent from both Arabic and Persian. DeepSeek produced ربر (rabar) for ربڑ (rabaṛ, “rubber”), erasing a retroflex flap that English “rubber” lacks; conversely it rendered the Punjabi-Urdu name الہ دتّہ as الہ ڈتہ, inserting a retroflex where dental د is correct, overgeneralising the English romanisation convention that uses retroflex symbols for South Asian dentals. The fact that these errors move in opposite directions, both governed by the model’s English representation of the same Urdu phoneme, suggests that retroflexes are resolved through an English-mediated phonetic intermediary rather than from a native Urdu inventory. Space deletion. Each model fuses a distinct, tokenizer-specific set of Urdu words into single orthographic units, an error invisible to readers but consequential for whitespace-based NLP pipelines (See Appendix N for details). Em-dash Urdu prose conventionally does not employ em-dashes (—) as structural markers; their presence indicates stylistic transfer from English writing conventions. Automatic detection across all 93 stories identified a total of 623 dash occurrences (avg = 6.6 per story), with 21 showing the highest frequency. Per-model distribution and per-category breakdown are reported in Table 8 (Appendix). QWEN produced the most em-dashes both in total and per story, consistent with its higher overall rate of English-stylistic transfer. 3.1.2 Semantic Coherence Semantic coherence failures constituted the second largest error category. Semantic Anomaly. Sentences that are syntactically well-formed but semantically impossible or meaningless. GPT’s آگ کے چراغ سے اُبلتی چائے پی رہا تھا (“he was drinking tea boiling from a lamp of fire”) is grammatically perfect but violates real-world plausibility. A subtler pattern is near-synonym confusion: DeepSeek substitutes بڑھاوا incitement for the intended ترقی (promotion), yielding a syntactically intact but semantically incoherent sentence, a failure fluent Urdu speakers would not produce (further examples in Appendix M). Context mismatch.ِ Context-mismatch errors arise when a generated sentence is locally well-formed but conflicts with earlier narrative content, a real-world fact, or a culturally established script. The annotated instances fall into five recurring failure modes, each of which reveals a different limitation of current Urdu LLMs. Kinship-term misuse is the dominant sub-category. Urdu encodes a much finer-grained kinship lexicon than English, distinguishing paternal from maternal relations (دادی dādī “father’s mother” vs نانی nānī “mother’s mother”), nephews by sibling gender (بھتیجا bhatījā “brother’s son” vs بھانجا bhānjā “sister’s son”), and address forms by social relation. Models routinely collapse these distinctions: in one QWEN story, a brother writes to his sister calling her son بھانجا (bhānjā, “sister’s son”) where بھتیجا (bhatījā, “brother’s son”) is required; a DeepSeek story introduces a maternal-side visit but uses دادی اماں (dādī ammā, “paternal grandmother”) for the maternal grandmother encountered there. Address forms are similarly over-generalised: QWEN uses ابا جان (abbā jān, “father, respectful”) for an elderly stranger across multiple stories, and a DeepSeek wife addresses her husband as بیٹا (beṭā, “son”). These errors are consistent with an English-mediated representation in which “uncle”, “nephew”, “grandmother”, and “dear” map onto whichever Urdu term is statistically most frequent in the training data, regardless of the discourse context. Long-range gender breakdown forms a second cluster, mostly observed in DeepSeek : once a female protagonist is established, the model later produces verbs and adjectives in the masculine (e.g جانتا instead of feminine جانتی(jāntī, fem.); ), or describes her with masculine-coded items (a girl wearing عمامہ , a male turban. These are not local agreement slips but long-range failures in which discourse-established gender is lost over several sentences (further examples in Appendix M). Cultural and religious anachronism also appeared: a story has the remains of a Muslim character cremated and given a funeral with an empty coffin, neither of which conforms to Islamic burial practice , and a QWEN story conflates the Eid crescent (ہلال) with a full moon (بدر . The models have memorized Urdu vocabulary but not the cultural scripts that constrain its use (further examples in Appendix M) . Intra-story contradictions include references to a daughter “lost five years ago” who is nonetheless still being educated in the same paragraph (QWEN),and a mother said to have sewn a blanket after her son had already left (GPT) . These reveal a lack of persistent state across sentences in medium-length narratives. Physical and commonsense impossibilities round out the category: a chandelier inside a bus, books packed inside a thermos, and a wrist-watch from 1947 displaying the date 14 November 1971. The surface fluency of these sentences masks the absence of a world model. Although Context Mismatch contributes the small percentage of any linguistic-error category, its qualitative profile is the most diagnostic of how multilingual LLMs fail. The kinship and cultural sub-types in particular indicate that current models reproduce Urdu surface forms while operating with an English-anchored semantic representation. 3.2 Cultural and Script Fidelity A small but qualitatively striking sub-set of context-mismatch errors involves the substitution of Pakistani objects, institutions, and practices with their nearest Western analogues, while the surrounding Urdu prose remains fluent. Although these instances are too few to dominate any single model’s overall error rate, they form a coherent pattern that reinforces the cross-lingual contamination hypothesis raised in our spelling and punctuation sections. Three sub-types recur. Western material culture in Pakistani settings. A GPT story set in a Punjabi village describes the bedroom wall as covered with پیلا وال پیپر (pīlā wāl pepar, “yellow wallpaper”), an interior-decor convention essentially absent from rural Pakistani homes, which use lime-wash or distemper paint. Another GPT story has a rickshaw driver refer to a گیس اسٹیشن (gais isṭeshan, “gas station”), an American term, where Pakistani usage is uniformly پٹرول پمپ (paṭrol pamp, “petrol pump”). Western institutional and role labels. DeepSeek refers to a twelve-year-old grandson as a بارہ سالہ جونیئر (bārah sālah jūniyar, “twelve-year-old junior”), importing the American junior–senior school-grade classification into a Pakistani village imam’s household, where such labels are not used. The same model also introduces a والدین میٹنگ (wāldain mīṭing, “parents meeting”) as a literal translation of the English “parent–teacher meeting”, whereas the established Urdu term is پی ٹی ایم (pī ṭī em). Language Register Violations. A qualitatively distinct failure model, observed exclusively in DeepSeek, is the substitution of Urdu with Punjabi at the phrase and sentence level: characters switch to Punjabi dialogue within an otherwise Urdu narrative, a discourse-level failure and a direct violation of the prompt’s explicit Urdu instruction . The pattern suggests DeepSeek conflates Pakistani linguistic identity with Punjabi, the majority regional language, rather than maintaining the prompted language boundary. Script Contamination in LLM-Generated Urdu Text . Beyond lexical foreign-word intrusion, automated Unicode-level analysis revealed a more fundamental failure mode: the embedding of non-Urdu script characters (Latin, Devanagari, CJK and Cyrillic) within generated Urdu text. A word was flagged as contaminated when any character in it fell outside the Urdu Unicode range (U+0600 to U+06F) and its standard presentation forms, excluding digits and punctuation. DeepSeek exhibited the most severe contamination , followed by QWEN and GPT. Contamination took two structural forms: pure intrusion, in which an entire word belongs to a foreign script, and partial contamination, in which a single word mixes Urdu with foreign characters and so corrupts an otherwise valid Urdu token. Latin was the most frequent contaminant across all three models, followed by Devanagari, with CJK and Cyrillic appearing only in DeepSeek. Per-script and per-model breakdowns, full token lists, and Unicode-level analysis are in Appendix D. 3.3 Gender Representation in Protagonist Roles Across all 93 stories, male protagonists outnumbered female protagonists 70 to 19 with an overall 3.7:1 ratio (Table 6). No model approached gender parity. QWEN was the most male-skewed (5.2:1), while GPT and DeepSeek shared a 3.1:1 ratio. This disparity is particularly noteworthy given that generation prompts were gender-neutral and contained no explicit instruction regarding protagonist gender. The consistent male-default pattern across all three models suggests that gender bias is encoded in the underlying generative behaviour rather than being prompt-induced. This finding aligns with prior work on gender bias in neural text generation (Masoudian et al., 2025; van Blerck et al., 2025). 3.4 Originality: Source Engagement and Memorisation 3.4.1 Title Repetition and Three Modes of Source Engagement A natural question raised by the linguistic and cultural errors of the previous sections is whether the underlying problem is one of language coverage or text coverage: does the model fail because it cannot speak Urdu well, or because it has never read the specific Urdu texts that our titles reference? To disentangle these, we introduce a simple, length-invariant lexical diagnostic, the Title Repetition Rate (TRR), defined as the number of exact title-phrase occurrences in the body of a generated story (excluding the opening line) per 100 words of body text. Both title and body are normalised before matching by removing diacritics, applying character-variant normalisation, stripping punctuation and normalising whitespace. The metric requires no LLM judge and no content-level annotation, making it robust to the methodological concerns that affect more semantic evaluation pipelines. Table 3 shows mean TRR by category and model. The ordering is identical for every model and spans a tenfold range, revealing three distinct modes of source engagement: Plot Reproduction (PR), Fables. For globally circulated fables present in every model’s training data, TRR is near zero [≈ 0.06]. The model retrieves a complete memorised plot and does not need to re-anchor on the title. Thematic Reproduction (TR), English Classic For Western literary classics translated into Urdu, TRR is intermediate (TRR ≈ 0.45). The model invokes a remembered theme (mental-health breakdown, a wife’s brief liberation, sudden inversion of fortune) without retrieving a full plot, and uses the title as a partial anchor. Lexical Title Unpacking (LT), Urdu canon For titles drawn from the Urdu literary canon e.g. (لحاف, اور کوٹ), texts largely absent from web pretraining corpora, TRR rises sharply( ≈ 0.58, peak 2.79) . With neither plot nor theme to retrieve, the only object the model can anchor on is the surface meaning of the title itself. We call this the Title Lexicalisation Effect: in the absence of source-text knowledge, an LLM treats a metaphorical title as a literal writing prompt. The most striking instance is Chughtai’s لحاف, a 1942 short story metaphorising hidden lesbian desire, rendered by QWEN as a literal narrative about a quilt with the word لحاف (blanket) repeated 35 times (TRR = 2.79). Bedi’s گرم کوٹ(warm coat) becomes a story about how to obtain a warm garment, and Manto’s ٹھنڈا گوشت(cold meat), a story of partition-era trauma, becomes a story about cold meat. Because TRR is computed on raw text rather than plot content, the same metric applies to the original Urdu stories. Source texts of لحاف(blanket), اور کوٹ(over coat), گرم کوٹ (warm coat)and ٹھنڈا گوشت (cold meat) yield a human-author TRR baseline of 0.04 to 0.52: the original authors use their title words sparingly, as thematic anchors rather than literal subjects. LLM TRR exceeds the human baseline in 8 of 9 comparisons (88.9%), with ratios from 1.4× to 12.7×. The two extremes (QWEN’s لحاف at 5.3× Chughtai’s original and DeepSeek’s ٹھنڈا گوشت at 12.7× Manto’s) are precisely the canonical Urdu texts most likely to be absent from training data. Per-title breakdowns and top repetition cases are in Appendix B. Table 3: Mean TRR by category and model. The tenfold spread between Fable (0.06) and Urdu Literature (0.58) is the quantitative signature of the three modes. Category DS GPT QWEN Avg Fable (PR) 0.11 0.01 0.04 0.06 Neutral Urdu (OR) 0.29 0.36 0.24 0.30 English Literature (TR) 0.51 0.46 0.39 0.45 Urdu Literature (LT) 0.57 0.51 0.67 0.58 Overall 0.43 0.40 0.40 0.41 3.4.2 Memorisation on Known Source Texts The Lexical Title Unpacking mode of Section 3.4.1 has no source plot to memorise. This subsection asks the complementary question: when a source text is plausibly in training data (fables and Western classics), how much of its plot, theme and resolution does the model reproduce? For every (original, generated) pair in our corpus (n=12n=12 titles per model, comprising 4 fables, 3 English classics and 5 Urdu-canon controls), GPT-5.4 scores plot, theme, character and resolution overlap on [0,1][0,1]. A story is classified as memorisation when at least two of plot, theme and resolution pass fixed thresholds; crossed with character overlap, this yields the 2×2 typology of Faithful Retelling, Disguised Memorisation, Spin-off, Original. Method, thresholds and the full per-story table are in Appendix F. Table 4 reports per-model outcomes. GPT and DeepSeek produce Faithful Retellings in 25% of cases; QWEN in only 8%. The composite score (mean of all four overlap axes) gives the same ordering: GPT 0.50 >> DeepSeek 0.43 ≫ QWEN 0.27. All 15 Urdu-canon rows receive the Original label across all three models with plot overlap effectively zero, quantitatively confirming the TRR-section prediction that the Urdu canon is absent from training data. Fables almost always trigger Faithful Retelling, with GPT at 3/4, DeepSeek at 2/4 and QWEN at 1/4. The one fable all three models reproduce identically is The Tortoise and the Hare (plot overlap 1.00 for every model). The single fable nobody reproduces is The Fool Donkey (بیوقوف گدھا), which exists in multiple narrative variants across South Asian and global oral tradition. We read its absence as evidence that coverage in training data is title-specific: when a fable circulates with a stable canonical plot it is memorised, but when it circulates as a family of variants no single version dominates the training signal. English classics produce a third pattern of high theme overlap, moderate plot overlap, and low character overlap. This is the structural signature of a model that has read about the story (summaries, critical commentary) without memorising the story itself. GPT’s Monkey’s Paw (theme 0.90, plot 0.29) and DeepSeek’s Story of an Hour (plot 0.86, theme 0.95, character 0.75, the only English-classic PR in our corpus) illustrate the two extremes of this regime; QWEN on the same titles scores theme 0.20 and 0.60 respectively, indicating it has neither read nor read-about these works. Together, the two analyses describe a single gradient: as source-text presence diminishes, LLM Urdu generation degrades gracefully from faithful reproduction ( low TRR, high overlap), through thematic paraphrase (moderate TRR, theme-only overlap), to lexical title-unpacking (high TRR, near-zero overlap). This behaviour is also consistent with recent findings on the “Artificial Hivemind” effect in open-ended LLM generation, where models converge toward retrieval-driven and structurally similar outputs despite the absence of a single correct response (Jiang et al., 2026) Table 4: Per-model memorisation outcomes. Composite = mean of plot, theme, resolution and character overlap. Outcome (%) GPT DS QWEN Faithful Retelling 25.0 25.0 08.3 Original 58.3 66.7 83.3 Spin-off 16.7 08.3 08.3 Composite score 0.50 0.43 0.27 3.5 Narrative Structure: Character Networks During annotation it was observed that the generated stories felt unusually warm, with QWEN’s stories feeling the warmest. To test whether this impression reflected a measurable structural signal, the signed character network methodology of (Nonaka and Perry, 2025) was applied to our corpus. For every story, GPT-5.4 extracted the characters and their pairwise relationships, each labelled as positive, negative or neutral with intensity 1 to 3. A signed weighted graph was built per story. Extraction prompt, per-story distributions, and the rationale for excluding the human corpus are in Appendix G. For a story’s signed graph G=(V,E)G=(V,E) with edge weight we∈−3,…,+3w_e∈\-3,…,+3\: density =2|E|/(|V|(|V|−1))=2|E|/(|V|(|V|-1)); clustering is the mean local clustering coefficient; positivity ratio is the share of non-neutral edges that are positive; and mean signed weight rescales 13|E|∑ewe 13|E| _ew_e to [−1,+1][-1,+1]. Small, tightly bound social worlds. Ignoring neutral acquaintances, the positivity ratio is 0.78 for GPT and 0.82 for both DeepSeek and QWEN. The median is 1.00 for GPT and QWEN and 0.92 for DeepSeek, meaning that the majority of generated stories contain no negative relationship at all (Table 5). One of the GPT story illustrates the pattern: of eleven characters and ten relationships, nine are positive (a doctor mentors the protagonist, a friend invites him to a gathering, a religious teacher offers spiritual guidance, his father loves him, his mother prays for him, even a passing elderly woman blesses him), and the single negative edge is confined to the explicitly antagonistic role. This protagonist-cushioning structure recurs across all three models. The annotators’ QWEN >> GPT >> DeepSeek ranking is only partially supported: QWEN sits at the top of the warmth scale, but DeepSeek is statistically indistinguishable, and the ranking collapses into a coarser QWEN, DeepSeek >> GPT split. Intensity separates the models more cleanly than positivity does. The mean signed edge weight on [−1,+1][-1,+1] is 0.29 for GPT against 0.37 for DeepSeek and 0.38 for QWEN: QWEN and DeepSeek dial relationships up to maximum intensity (mentor-as-spiritual-guide, mother-as-saint), while GPT distributes more weight at intensities 1 and 2, producing supportive but ordinary relationships. This is consistent with our earlier finding (Section 3.4.2) that GPT and DeepSeek anchor most strongly to remembered source plots: memorisation and emotional amplification are correlated tendencies, and DeepSeek ranks high on both. Same prompt, different casts. Despite identical Urdu titles, the three models populate their stories with near-disjoint character sets. Pairwise Jaccard overlap on character names is 0.013 to 0.018 across model pairs, with median exactly zero: for more than half of all prompts, no two models share a single character name. The structural patterns above are therefore not driven by a shared casting template; each model explores a distinct region of the character space. Table 5: Network metrics per model (mean over 31 stories; GPT-5.4 extraction). Signed-only rows restrict to stories with ≥1≥ 1 non-neutral edge. Metric GPT DS QWEN Characters / story 6.3 6.0 5.7 Edges / story 6.8 7.6 6.6 Density 0.53 0.56 0.55 Clustering 0.40 0.54 0.46 Positivity (all) 0.78 0.82 0.82 Mean signed weight (all) 0.29 0.37 0.38 Positivity (signed only) 0.80 0.82 0.82 Mean signed weight (signed) 0.30 0.37 0.38 Pairwise character Jaccard (mean / median) GPT & DS 0.013 / 0.000 GPT & QWEN 0.018 / 0.000 DS & QWEN 0.013 / 0.000 3.6 Robustness and Validation Four additional checks probe the robustness of the findings above; full tables are in Appendices I–L. Repeated generations. Five titles (one per category) were regenerated twice more per model (30 additional stories) to estimate within-condition variance. TRR’s SD is modest (0.027–0.055) and its direction is stable across all nine model×run comparisons; positivity ratio is the most stable metric (SD 0.032–0.048, all nine values >>0.71); the male-protagonist skew and the linguistic/semantic error patterns both persist in every run. English-language control. The same three models generated the same seven TSST titles in English (21 stories). Positivity ratio is 17 points higher in Urdu than English (0.807 vs. 0.637) and TRR is 3.5–5.8× higher, while faithful retelling is lower in Urdu (19.4% vs. 38.1%) – consistent with the Urdu memorization gap reflecting thinner training coverage of the Urdu canon rather than a general model property. Cross-family judge validation. Re-scoring memorization and character networks with Gemini-3.5-Flash as an out-of-family judge preserves the direction and model ordering of every headline finding; categorical faithful-retelling percentages are identical across judges. Human validation of memorisation labels. A native Urdu speaker manually re-checked the GPT-4o-mini judge’s label on 13 story pairs, agreeing on 10/13 (76.9%); every disagreement involved over-estimated plot reproduction on UST titles. 4 Tractability via LLM Remediation To test which of the failures documented above can be corrected without retraining, a targeted post-editing experiment was conducted on a stratified subset of 15 stories (5 per model, balanced across title categories). Each story was passed through GPT-5.5 with a JSON-response prompt instructing it to detect and minimally fix five error types: grammar errors, agreement, agreement errors, spelling errors, space deletion, and semantic anomaly. The prompt enforced single-token edits where possible (e.g. inserting one missing space, replacing one misspelled word) and forbade sentence-level rewrites or reformatting. The full system prompt is given in Appendix O. GPT-5.5 output was compared against the human annotations on the same 15 stories. Alignment with human-flagged errors reached approximately 90% across the four linguistic categories: grammar, agreement, spelling and space deletion were detected and corrected at high precision, with single-character or single-word fixes that preserved the surrounding text. Semantic anomaly (syntactically well-formed but semantically empty sentences) was detected at lower but useful rates, with GPT-5.5 successfully repairing cases such as near-synonym confusion (بڑھاوا for ترقی) where the lexical substitution was local. A small number of kinship-term errors (e.g. خالہ for پھوپھی) were also caught, indicating partial sensitivity to Urdu’s relational lexicon. The remediation boundary is sharp. Errors that GPT-5.5 did not catch are precisely the failure modes identified in Sections 3.1.2 and 3.2: long-range gender breakdown across discourse, cultural and religious anachronisms (a Muslim character cremated, a full moon on Eid, Western-context substitutions e.g. a gas station, refreshment van ) and Punjabi register switching in DeepSeek output. These errors require background knowledge of Pakistani cultural scripts, persistent discourse state, or commitment to a prompted language register, none of which a sentence-level proofreading pass can supply. 5 Conclusion This study evaluates whether contemporary multilingual LLMs are genuinely competent in Urdu or only fluent on its surface. A corpus of LLM generated Urdu-Stories annotated by native speakers, give a consistent answer. When a story’s source text is well represented online, models reproduce it; when it is not, they fall back on unpacking the title itself. The surface language is fluent but not native, with errors clustering on features English does not share: ergative case, gender agreement, retroflex consonants spelled through an English intermediary. Father’s sister become Mother’s sister, Muslim characters are cremated, gas stations appear in Punjabi villages. The social worlds are oddly warm, with the median story containing no negative relationship at all and male protagonists outnumbering female ones almost four to one. A modern model can proofread away most of the grammar and spelling errors, but it cannot fix the cremation or the gas station. The models can spell and form sentences, mostly, but they cannot yet write a Pakistani story. Limitations The Urdu-Stories corpus, comprising 93 stories generated across three commercial models for a single language, remains modest in scale. Extending the analysis to additional commercial and open-weight systems (e.g., Llama, Aya) and to other low-resource languages would strengthen the generality of the cross-model findings. Our comparison against human-authored Urdu literature relies on published canonical short fiction rather than prompt-matched human generations. We intentionally treat these literary works as culturally validated reference distributions for Urdu narrative structure, thematic development, and stylistic behaviour. Metrics such as Title Repetition Rate (TRR) are length-normalised specifically to permit comparison between LLM-generated and human-authored texts despite differences in story length and authorship conditions. While prompt-controlled human generations could provide a more experimentally matched baseline, published Urdu literature constitutes a stronger benchmark for evaluating whether LLMs reproduce the linguistic, cultural, and stylistic properties of native literary writing. Exact-phrase TRR may underestimate title anchoring when a model decomposes a multi-word title into its constituent lexical items. GPT’s گرم کوٹ (TRR = 0.16) illustrates this behaviour: although the full phrase appears only three times, the head noun کوٹ recurs throughout the story, with the protagonist—wrapped in a شال—repeatedly longing for a گرم کوٹ. The title therefore remains thematically central even when the exact surface form is infrequent. Partial-word, lemma-level, and synonym-aware matching are natural extensions that we leave to future work. Moreover, commercial API outputs are non-deterministic and model versions evolve over time, limiting exact reproducibility. We attempted to include an open-weight multilingual model (LLaMA-70B) for comparison, but in this long-form Urdu story-generation setting it entered repetitive generation loops after approximately 500 words, producing outputs unsuitable for analysis; extending the comparison to a broader range of open-weight multilingual models is left to future work. References Ahuja et al. (2023) K. Ahuja, H. Diddee, R. Hada, M. Ochieng, K. Ramesh, P. Jain, A. Nambi, T. Ganu, S. Segal, M. Ahmed, et al. Mega: multilingual evaluation of generative ai. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 4232–4267. Cited by: §1. Anikina et al. (2025) T. Anikina, J. Cegin, J. Simko, and S. Ostermann A rigorous evaluation of LLM data generation strategies for low-resource languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 8293–8314. Cited by: §1. Blasi et al. (2022) D. Blasi, A. Anastasopoulos, and G. Neubig Systematic inequalities in language technology performance across the world’s languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5486–5505. Cited by: §1. Gómez-Rodríguez and Williams (2023) C. Gómez-Rodríguez and P. Williams A confederacy of models: a comprehensive evaluation of LLMs on creative writing. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, p. 14504–14528. External Links: Link Cited by: §1, §2.1. Gu et al. (2024) J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, et al. A survey on LLM-as-a-judge. The Innovation. Cited by: §1. Hershcovich et al. (2022) D. Hershcovich, S. Frank, H. Lent, M. de Lhoneux, M. Abdou, S. Brandl, E. Bugliarello, L. Cabello Piqueras, I. Chalkidis, R. Cui, C. Fierro, K. Margatina, P. Rust, and A. Søgaard Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), External Links: Link Cited by: §1. Hu et al. (2025) S. Hu, I. Vulić, and A. Korhonen Quantifying language disparities in multilingual large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 4003–4018. Cited by: §1. Jiang et al. (2026) L. Jiang, Y. Chai, M. Li, M. Liu, R. Fok, N. Dziri, Y. Tsvetkov, M. Sap, and Y. Choi Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link Cited by: §3.4.2. Joshi et al. (2020) P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 6282–6293. Cited by: §1. Kreutzer et al. (2025) J. Kreutzer, E. Briakou, S. Agrawal, M. Fadaee, and K. Tom Multilingual LLM evaluation through the lens of machine translation evaluation. arXiv preprint arXiv:2504.11829. Cited by: §1. Masoudian et al. (2025) S. Masoudian, G. Escobedo, H. Strauss, and M. Schedl Investigating gender bias in LLM-Generated stories via psychological stereotypes. External Links: 2508.03292, Link Cited by: §3.3. Nonaka and Perry (2025) H. Nonaka and K. E. Perry Evaluating LLM story generation through large-scale network analysis on social structures. In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling, External Links: Link Cited by: §3.5. Santurkar et al. (2023) S. Santurkar, E. Durmus, F. Ladhak, C. Lee, P. Liang, and T. Hashimoto Whose opinions do language models reflect?. In International conference on machine learning, p. 29971–30004. Cited by: §1. Schroeder and Wood-Doughty (2024) K. Schroeder and Z. Wood-Doughty Can you trust llm judgments? reliability of LLM-as-a-judge. arXiv preprint arXiv:2412.12509. Cited by: §1. Shani et al. (2026) C. Shani, Y. Reif, N. Roll, D. Jurafsky, and E. Shutova The roots of performance disparity in multilingual language models: intrinsic modeling difficulty or design choices?. arXiv preprint arXiv:2601.07220. Cited by: §1. Tao et al. (2024) Y. Tao, O. Viberg, R. S. Baker, and R. F. Kizilcec Cultural bias and cultural alignment of large language models. PNAS nexus 3 (9), p. pgae346. Cited by: §1. Thakur et al. (2025) A. S. Thakur, K. Choudhary, V. S. Ramayapally, S. Vaidyanathan, and D. Hupkes Judging the judges: evaluating alignment and vulnerabilities in LLM-as-judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM2^2), p. 404–430. Cited by: §1. Thellmann et al. (2024) K. Thellmann, B. Stadler, M. Fromm, J. S. Buschhoff, A. Jude, F. Barth, J. Leveling, N. Flores-Herr, J. Köhler, R. Jäkel, et al. Towards multilingual LLm evaluation for european languages. arXiv preprint arXiv:2410.08928. Cited by: §1. Unjum et al. (2025) N. Unjum, S. Evert, K. Sarveswaran, R. Weerasinghe, and N. Jayatilleke LLMs for low-resource languages: a survey. Available at SSRN 6122026. Cited by: §1. van Blerck et al. (2025) I. C.E. van Blerck, E. S. de Lima, M. M.E. Neggers, and T. Calders Unveiling gender bias in LLM-generated hero and heroine narratives. Entertainment Computing 55, p. 100972. External Links: ISSN 1875-9521, Document, Link Cited by: §3.3. Xuan et al. (2025) W. Xuan, R. Yang, H. Qi, Q. Zeng, Y. Xiao, A. Feng, D. Liu, Y. Xing, J. Wang, F. Gao, et al. MMLU-prox: a multilingual benchmark for advanced large language model evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 1513–1532. Cited by: §1. Zhu et al. (2024) W. Zhu, H. Liu, Q. Dong, J. Xu, S. Huang, L. Kong, J. Chen, and L. Li Multilingual machine translation with large language models: empirical results and analysis. In Findings of the association for computational linguistics: NAACL 2024, p. 2765–2781. Cited by: §1. Appendix A Appendix Table 6: Protagonist Gender Distribution by Model Model Male Female Total M:F Ratio DeepSeek 22 7 29 3.1:1 GPT 22 7 29 3.1:1 QWEN 26 5 31 5.2:1 Overall 70 19 89 3.7:1 Table 7: Manual Annotation Counts by Label and Model Category Label GPT QWEN DS Total Linguistic Grammar Error 72 138 106 316 Agreement Error 41 92 39 172 Spelling Error 16 40 52 108 Space Deletion 79 11 50 140 Em-Dash 261 320 42 623 Subtotal 469 601 289 1359 Semantic Context Mismatch 8 21 21 50 Semantic Anomaly 41 92 39 172 Western Influenced Cultural Mismatch 3 0 5 8 Script Contamination 11 14 61 86 Subtotal 63 127 126 316 Total Errors 532 728 415 1675 Entities Character Name 94 80 99 273 City 40 49 37 126 MaleProtag 22 26 22 70 FemaleProtag 7 5 7 19 Table 8: Em-dash usage per model. Model Total Avg Min Max GPT 261 8.4 0 18 QWEN 320 10.32 3 21 DeepSeek 42 1.35 0 17 Overall 623 6.7 0 21 Appendix B Title Repetition: Method and Extended Results B.1 Metric Definition The Title Repetition Rate is defined as TRR=count(title_phrase∈story_body)total_words×100,TRR= count(title\_phrase \_body)total\_words× 100, (1) where the title phrase is matched exactly against the normalised body of the generated story, excluding the opening line. Normalisation removes diacritics, applies standard Urdu character-variant folding (e.g. different forms of ی and ہ), strips punctuation, and collapses internal whitespace. Both title and body are normalised identically before matching. B.2 Per-Model Aggregate Statistics Table 9 reports per-model aggregates across all 93 stories. DeepSeek produces the longest stories on average and also the highest absolute count of title repetitions per story; QWEN and GPT are close on TRR despite QWEN producing the shortest stories and GPT the most TRR = 0 stories. Table 9: Per-model TRR aggregates across all 93 stories. Avg Count = mean exact title occurrences per story; Avg Length = mean story length in words; TRR=0 = number of stories with no title mention. Model Avg TRR Avg Count Avg Length TRR=0 GPT 0.40 6.06 1,584 5 QWEN 0.40 5.19 1,379 6 DeepSeek 0.43 8.12 1,908 2 Overall 0.41 6.46 1,624 13 Figure 1: Average TRR by story category and model. Urdu Literature titles show consistently highest TRR, reflecting systematic literal interpretation of metaphorical titles. Appendix C Full Story Title Corpus Table 10: Distribution of story titles across the four source categories. Source Abbrev. # Translated Short Story Titles TSST 10 Urdu Short Story Titles UST 11 Fables Fable 4 Neutral Urdu Titles NUT 6 Total 31 Table 11 lists all 31 titles with their original authors (where applicable), source category, and ALA-LC transliteration. Per-title generation statistics appear in Table 12. Table 11: Complete list of 31 story titles used in this study. TSST = Translated Short Story Title (English origin); UST = Urdu Short Story Title (canonical Urdu/South-Asian literature); Fable = traditional moral tale; NUT = Neutral Urdu Title (composed by authors). ID Title (Urdu) Transliteration / English Original Author / Source Cat. 1 ایمان کا سفر Īmān kā Safar / Journey of Faith — (composed) NUT 2 آخری ٹرین Ākhrī Ṭrēn / The Last Train — (composed) NUT 3 سوات کی شام Swāt kī Shām / Evening in Swat — (composed) NUT 4 آخری بس Ākhrī Bas / The Last Bus — (composed) NUT 5 عید کا نیا چاند ʿĪd kā Nayā Chānd / The New Moon of Eid — (composed) NUT 6 ایمان کی تلاش Īmān kī Talāsh / Search for Faith — (composed) NUT 7 پیلا وال پیپر Pīlā Vāl Pēpar / The Yellow Wallpaper Charlotte P. Gilman TSST 8 لاٹری Lāṭrī / The Lottery Shirley Jackson TSST 9 بندر کا پنجہ Bandar kā Panjah / The Monkey’s Paw W. W. Jacobs TSST 10 دوسری عورت Dūsrī ʿAurat / The Second Woman (Urdu literary tradition) UST 11 ٹھنڈا گوشت Ṭhanḍā Gōsht / Cold Meat Saadat Hasan Manto UST 12 خاموش اذان Khāmōsh Az̤ān / Silent Call to Prayer –(composed) UST 13 خرگوش اور کچھوا Khargōsh aur Kachhwā / Hare and Tortoise Aesop / Folk Fable 14 پیاسا کوا Piyāsā Kawwā / The Thirsty Crow Aesop / Folk Fable 15 بیوقوف گدھا Bēwaqūf Gadhā / The Foolish Donkey Folk Fable 16 اندھا آدمی اور ہاتھی Andhā Ādmī aur Hāthī / Blind Men & Elephant John G. Saxe TSST 17 لحاف Liḥāf / The Quilt Ismat Chughtai UST 18 جلا وطن Jalā Waṭan / Exiled Ghulam Abbas UST 19 آگ کا دریا Āg kā Daryā / River of Fire Qurratulain Hyder UST 20 گرم کوٹ Garm Koṭ / Warm Coat Rajinder Singh Bedi UST 21 اور کوٹ Ōvar Koṭ / The Overcoat Ghulam Abbas UST 22 لالچی کتا Lālchī Kuttā / The Greedy Dog Aesop / Folk Fable 23 پچھتاوا Pachtāwā / Regret (Urdu literary tradition) UST 24 سر کٹا سوار Sar Kaṭā Sawār / The legend of Sleepy Hollow Washington Irving TSST 25 ایک گھنٹے کی کہانی Ek Ghanṭē kī Kahānī / Story of an Hour Kate Chopin TSST 26 پہاڑی کی برف Pahāṛī kī Barf / The Snows of Kilimanjaro Ernest Hemingway TSST 27 لڑکے اور لڑکیاں Laṛkē aur Laṛkiyāṅ / Boys and Girls Alice Munro TSST 28 سلطانہ کا خواب Sulṭānah kā Khwāb / Sultana’s Dream Rokeya S. Hossain UST 29 خاتون یا شیر Khātūn yā Shēr / The Lady or the Tiger Frank R. Stockton TSST 30 خط Khaṭ / The Letter Anton Chekhov TSST 31 پت جھڑ کی آواز Pat Jhaṛ kī Āwāz / Sound of Autumn Leaves Qurratulain Hyder UST Table 12: Average Title Repetition Rates Across Stories. TSST = Translated Short Story Title, UST = Urdu Short Story Title, NUT = Neutral Urdu Title. Story ID Title (ALA-LC Transliteration / English Translation) Source Avg. TRR Avg. Count 1 ایمان کا سفر (Īmān kā Safar / Journey of Faith) NUT 0.2459 3.6667 2 آخری ٹرین (Ākhrī Ṭrēn / The Last Train) NUT 0.6595 8.6667 3 سوات کی شام (Swāt kī Shām / Evening in Swat) NUT 0.2165 3.0000 4 آخری بس (Ākhrī Bas / The Last Bus) NUT 0.3025 5.0000 5 عید کا نیا چاند (ʿĪd kā Nayā Chānd / The New Moon of Eid) NUT 0.1254 2.0000 6 ایمان کی تلاش (Īmān kī Talāsh / Search for Faith) NUT 0.2215 3.3333 7 پیلا وال پیپر (Pīlā Vāl Pēpar / The Yellow Wallpaper) TSST 0.3524 4.0000 8 لاٹری (Lāṭrī / The Lottery) TSST 0.9110 15.0000 9 بندر کا پنجہ (Bandar kā Panjah / The Monkey’s Paw) TSST 0.3188 5.6667 10 دوسری عورت (Dūsrī ʿAurat / The Second Woman) UST 0.2804 5.6667 11 ٹھنڈا گوشت (Ṭhanḍā Gōsht / Cold Meat) UST 0.3720 5.6667 12 خاموش اذان (Khāmōsh Az̤ān / Silent Call to Prayer) UST 0.4921 7.3333 13 خرگوش اور کچھوا (Khargōsh aur Kachhwā / The Hare and the Tortoise) Fable 0.0219 0.3333 14 پیاسا کوا (Piyāsā Kawwā / The Thirsty Crow) Fable 0.0228 0.3333 15 بیوقوف گدھا (Bēwaqūf Gadhā / The Foolish Donkey) Fable 0.1589 3.0000 16 اندھا آدمی اور ہاتھی (Andhā Ādmī aur Hāthī / The Blind Man and the Elephant) TSST 0.0000 0.0000 17 لحاف (Liḥāf / The Quilt) UST 1.7525 26.6667 18 جلا وطن (Jalā Waṭan / Exiled) UST 0.2507 4.3333 19 آگ کا دریا (Āg kā Daryā / River of Fire) UST 0.3410 5.0000 20 گرم کوٹ (Garm Koṭ / Warm Coat) UST 0.7702 10.6667 21 اور کوٹ (Ōvar Koṭ / The Overcoat) UST 1.2524 21.3333 22 لالچی کتا (Lālchī Kuttā / The Greedy Dog) Fable 0.0184 0.3333 23 پچھتاوا (Pachtāwā / Regret) TSST 0.2198 4.0000 24 سر کٹا سوار (Sar Kaṭā Sawār / The Headless Horseman) TSST 0.3038 5.3333 25 ایک گھنٹے کی کہانی (Ek Ghanṭē kī Kahānī / The Story of an Hour) TSST 0.0791 1.3333 26 پہاڑی کی برف (Pahāṛī kī Barf / Snow of the Mountain) TSST 0.1813 3.0000 27 لڑکے اور لڑکیاں (Laṛkē aur Laṛkiyāṅ / Boys and Girls) TSST 0.1513 2.3333 28 سلطانہ کا خواب (Sulṭānah kā Khwāb / Sultana’s Dream) UST 0.0156 0.3333 29 خاتون یا شیر (Khātūn yā Shēr / The Lady or the Tiger) TSST 0.0755 1.3333 30 خط (Khaṭ / The Letter) TSST 2.3459 36.6667 31 پت جھڑ کی آواز (Pat Jhaṛ kī Āwāz / Sound of Autumn Leaves) UST 0.2860 5.0000 Appendix D Script Contamination: Full Analysis D.1 Detection Method A word was flagged as script-contaminated when any character in it fell outside the Urdu Unicode range (U+0600 to U+06F) and the standard Arabic presentation forms (U+FB50 to U+FDFF, U+FE70 to U+FEFF), excluding digits, punctuation and whitespace. We classify each contaminated word into two structural patterns: • Pure intrusion. The entire word belongs to a foreign script (e.g. hospital, career, water). • Partial contamination. The word mixes characters from Urdu and at least one foreign script (e.g. حajی for حاجی). Aggregate counts per script and per model are given in Table 13. D.2 Latin Script Intrusion Latin was the most frequent contaminant across all three models. Pure English-word substitution included everyday vocabulary for which standard Urdu equivalents exist: hospital, career, water, warmth, tight. Two partial-contamination cases warrant additional discussion beyond what the main text reports. First, QWEN’s substitution of حاجی with حajی replaces the vowel graphemes ا and ی with their Latin phonetic equivalents a and j. The form recurs 10 times in the same 1,343-word story and never spaces out the Latin characters, indicating that the token boundary itself is broken in the model’s vocabulary. Second, DeepSeek produced کہa, replacing the final ا of کہا (said) with Latin a: phonetically equivalent but orthographically invalid. The substitution suggests that the decoder resolved an ambiguous token boundary by falling back to Latin. DeepSeek produced رسمیities, attaching the English plural morpheme -ities to the Urdu adjective رسمی (formal), a previously undocumented form of cross-lingual morphological contamination in which English derivational morphology is applied to an Urdu stem. D.3 Devanagari (Hindi) Intrusion Devanagari intrusion was most prevalent in DeepSeek and also appeared in GPT and QWEN. Because Urdu and Hindi share extensive phonological overlap, the proximity produces character-level cross-script leakage in which phonetically equivalent Devanagari diacritics substitute for their Urdu counterparts. For example, ٹھ ंڈی (cold) carries an internal Devanagari nasaliser. More severe is کुत्तوں, where the Urdu word for dogs is partially rendered in Devanagari mid-token: a morpheme-level script switch rather than a diacritic substitution. D.4 Chinese and Cyrillic Intrusion The most striking contamination instances involve Chinese (CJK) and Cyrillic characters embedded within Urdu tokens. These scripts have low or no phonological or typological relationship to Urdu, making their presence entirely anomalous. In every observed case the foreign token is semantically appropriate in context: yánsù (serious), de shēnghuó (of life), huàtí (topic), iskrennost’ (sincerity). The failure is orthographic rather than semantic: DeepSeek selected a valid lexical item from its Chinese or Cyrillic vocabulary where an Urdu equivalent was required, and omitted the delimiting space, producing undelimited bilingual compounds. The exception is پراج екٹ, where Cyrillic characters are substituted for similar Urdu phonemes within a single word, a pattern structurally analogous to the Devanagari substitutions above. Table 13: Script Contamination Summary by Model Metric GPT QWEN DeepSeek Total Affected stories 6 3 15 24 Total contaminated words 11 14 61 86 Pure intrusion 8 3 42 53 Partial mixed 3 11 19 33 By Script Latin (English) 7 13 47 67 Devanagari (Hindi) 3 1 9 13 Chinese (CJK) 0 0 3 3 Cyrillic 1 0 2 3 Appendix E Top Repetition Cases Table 14 lists individual (title, model) pairs with the highest absolute repetition counts. UST titles dominate the list, consistent with the Lexical Title Unpacking interpretation. Table 14: Top repetition cases across literary title sources and LLMs. TSST = Translated Short Story Title; UST = Urdu Short Story Title. Title (ALA-LC / English) Source Model Count (Rate per 100) خط (Khaṭ / The Letter) TSST DeepSeek 61 (3.27) GPT 33 (2.08) لحاف (Liḥāf / The Quilt) UST QWEN 35 (2.79) DeepSeek 28 (1.30) GPT 17 (1.17) اور کوٹ (Ōvar Koṭ / The Overcoat) UST DeepSeek 35 (1.76) QWEN 19 (1.41) لاٹری (Lāṭrī / The Lottery) TSST GPT 20 (1.16) گرم کوٹ (Garm Koṭ / Warm Coat) UST DeepSeek 17 (1.03) خاموش اذان (Khāmōsh Az̤ān / Silent Call to Prayer) UST GPT 16 (1.14) E.1 Original-Author vs LLM TRR Table 15 compares LLM TRR against the original-author baseline for the four canonical Urdu titles whose source texts were obtained from Rekhta. Table 15: TRR: original authors vs LLM-generated stories. Title Author Orig TRR GPT QWEN DS Avg Ratio لحاف Ismat Chughtai 0.52 1.17 (2.2×) 2.79 (5.3×) 1.30 (2.5×) 3.3× اور کوٹ Ghulam Abbas 0.41 0.59 (1.4×) 1.41 (3.4×) 1.76 (4.3×) 3.0× گرم کوٹ Rajinder Singh Bedi 0.33 0.16 (0.5×)† 1.13 (3.4×) 1.03 (3.1×) 2.3× ٹھنڈا گوشت Saadat Hasan Manto 0.04 0.48 (10.6×) 0.05 (1.3×) 0.57 (12.7×) 8.2× † GPT decomposed the title, using کوٹ alone extensively (see Section Limitations). Appendix F Memorisation Analysis: Method and Full Results F.1 Method For each (original, generated) story pair (n=36n=36 pairs in total: 12 titles × 3 models) we asked GPT-4o-mini at temperature 0 with JSON response format to produce four overlap scores in [0,1][0,1] and a categorical label. Axes. • Plot overlap. The judge first extracts 4 to 8 character-agnostic plot beats from the original (e.g. “a thirsty animal finds water it cannot reach”), then reports the fraction of those beats that recur in the generated story in the same causal order. • Theme overlap. Semantic similarity of the central moral or message of the two stories. • Character overlap. Set-style overlap of the principal characters and their narrative functions, insensitive to renaming. • Resolution match. Semantic similarity of the one-sentence narrative endings, compared in isolation. The judge also returns a categorical verdict ∈same,partial,different∈\ same, partial, different\. Thresholds. Each axis is converted to a boolean using a fixed threshold: plot ≥ 0.60, theme ≥ 0.70, resolution ≥ 0.80. A generated story is labelled memorisation when at least two of these three booleans are true. Crossed with character overlap (threshold 0.50), this yields the 2×2 typology of Table 16. These thresholds are corpus-specific, single-annotator operating points rather than values drawn from a prior theoretical framework. They were set by manual calibration: a native Urdu-speaking researcher compared each generated story with its original and judged whether a Faithful Retelling label was appropriate; the cut-offs above are the values that best matched those judgements. Plot received a lower threshold than theme or resolution because it is evaluated across multiple narrative beats, whereas theme and resolution each require a single, stronger judgement of semantic equivalence. end Categorical label. Independent of the threshold rule, the judge also assigns one of four discrete labels: PR (Plot Reproduction: plot, theme, characters and setting all substantially preserved); PI (Partial Inheritance: two or three axes partially overlap); OR (Original: only the title or a high-level topic is shared); LI (Lexical Inheritance: title and surface motifs reused but no narratological axis aligns; no story in our corpus received this label). Table 16: Memorisation typology (definitions). Char. LOW Char. HIGH ≥2≥ 2 axes agree Disguised Mem. Faithful Retelling <2<2 axes agree Original Spin-off F.2 Per-Category Breakdown Table 17 disaggregates outcomes by category. Fables dominate the Faithful Retelling cell for GPT and DeepSeek; English classics produce the mixed pattern (one Chopin PR for DeepSeek, all others Original); every Urdu-canon row is Original for every model. Table 17: Memorisation label counts by category and model. PR = Plot Reproduction; PI = Partial Inheritance; OR = Original. Fable (n=4n=4) English (n=3n=3) Urdu canon (n=5n=5) Model PR PI OR PR PI OR PR PI OR GPT 3 0 1 0 0 3 0 1 4 DeepSeek 2 1 1 1 0 2 0 0 5 QWEN 1 0 3 0 1 2 0 0 5 F.3 Cross-Model Plot Overlap For each title we also computed pairwise overlap between the three generated versions (Table 18). GPT and DeepSeek overlap most often (mean 0.45), consistent with both models tending toward Faithful Retelling on the same fables. QWEN is roughly equidistant from both (0.33 to 0.34) and contributes new plots more often than it mirrors either. Table 18: Mean pairwise overlap between the three generated versions of the same source title (n=12n=12 titles). Model pair Mean Median GPT & DeepSeek 0.45 0.35 DeepSeek & QWEN 0.34 0.28 GPT & QWEN 0.33 0.35 Appendix G Cross-Model Structural Analysis: Details G.1 Extraction Protocol For each of the 93 LLM-generated stories, the following instruction was issued to GPT-5.4 at temperature 0 with JSON response format enforced. The judge was given the story title and the full story text (truncated at 18,000 characters, a limit no generated story exceeded). You are a literary analyst fluent in Urdu. Extract: (1) the list of distinct characters who actually appear or speak in the story (skip purely mentioned-by-name characters with no role); (2) the signed pairwise relationships between those characters as evidenced by the story. For each character record a name and a role (protagonist, antagonist, supporting, mentor, love-interest, antagonised-victim, narrator, other). For each relationship record source, target, sign (positive = affection / support / alliance / love / friendship; negative = conflict / hostility / betrayal / abuse / rivalry; neutral = acquaintance / professional / functional), intensity ∈1,2,3∈\1,2,3\, and a ≤ 20-word Urdu evidence paraphrase. Only include relationships with textual support; use the same spelling of a name across all of that character’s edges; do not invent characters; list symmetric relationships once. G.2 Metric Definitions Given a story-level signed weighted graph G=(V,E)G=(V,E) with edge weight we=sign(e)⋅intensity(e)∈−3,…,+3w_e=sign(e)·intensity(e)∈\-3,…,+3\: • Density =2|E|/(|V|(|V|−1))=2|E|/(|V|(|V|-1)). • Average clustering: mean unweighted local clustering coefficient over all nodes. • Positivity ratio =|e:we>0|/|e:we≠0|=|\e:w_e>0\|/|\e:w_e≠ 0\| (positive share of signed edges; neutral edges excluded). • Mean signed edge weight =13|E|∑e∈Ewe= 13|E| _e∈ Ew_e, rescaled to [−1,+1][-1,+1]. • Jaccard character overlap between models A and B on the same prompt =|CA∩CB|/|CA∪CB|=|C_A∩ C_B|/|C_A∪ C_B|, where CXC_X is the set of normalised character names produced by model X. Duplicate edges between the same character pair are averaged before any metric is computed. G.3 Per-Story Distributions Figures 2 and 3 show per-story distributions of density and mean signed edge weight. Figure 2: Density per story. White diamonds mark means. Figure 3: Mean signed edge weight per story on [−1,+1][-1,+1]. G.4 Why Human Stories Are Excluded The human reference corpus consists of published Urdu short fiction by canonical authors (Manto, Chughtai, Bedi, Krishan Chander). These stories run 5,000 to 15,000 words against an LLM target of 2,000. A longer story has room for incidental and functional characters (shopkeepers, neighbours, passing acquaintances) that appear in the network as additional neutral edges, inflating character counts and the neutral edge share. A direct human-vs-LLM comparison on these metrics would therefore be confounded by length. Truncating the human stories to 2,000 words was considered and rejected, as it would discard the resolution and closing character introductions of most published short stories and replace a length confound with a worse truncation artefact. The network analysis is therefore restricted to the three LLM corpora, with a length-matched human comparison left to future work. begin Appendix H Annotation: Per-Model and Per-Label Agreement Table 19 reports Cohen’s κ computed separately for the stories generated by each model, on the 60-story inter-annotator-agreement subset described in Section 2.4. Agreement is consistently substantial and does not vary meaningfully by source model. Table 19: Inter-annotator agreement by model (60-story IAA subset). Model Cohen’s κ Interpretation GPT 0.8072 Substantial DeepSeek 0.8084 Substantial QWEN 0.8058 Substantial Overall 0.81 Substantial We do not report agreement broken down by individual label, because the number of instances of several labels (e.g. Cultural Inconsistency) in the 60-story subset is too small to support a meaningful per-label estimate. Appendix I Repeated Generations: Full Results To estimate within-condition variance, five titles (IDs 6, 8, 10, 13 and 18, one per title category, selected with a fixed random seed) were regenerated twice more per model, using the same prompt as the main corpus, yielding 30 additional Urdu stories. Table 20: Title Repetition Rate across three independent runs (5 titles). Model R1 R2 R3 Mean SD GPT 0.100 0.140 0.218 0.153 0.048 QWEN 0.258 0.219 0.193 0.223 0.027 DeepSeek 0.170 0.150 0.276 0.199 0.055 Table 21: Raw title-occurrence count per story across three runs (5 titles). Model R1 R2 R3 Mean SD GPT 1.80 2.80 4.40 3.00 1.06 QWEN 4.20 3.00 2.80 3.33 0.60 DeepSeek 2.80 3.00 5.20 3.67 1.08 TRR’s SD ranges from 0.027 to 0.055 across models, indicating moderate run-to-run variation in magnitude. The direction of the Lexical Title Unpacking effect is nonetheless stable: full-corpus Urdu-canon TRR (0.40–0.43 across models) exceeds the re-generated TRR in all nine model×run comparisons. Raw counts show the same pattern independently of story length, with Urdu mean counts (5.19–8.12) exceeding re-generated means (3.00–3.67) across all models and runs. Table 22: Positivity ratio in character networks across three runs (5 titles). Model R1 R2 R3 Mean SD GPT 0.780 0.803 0.858 0.814 0.032 DeepSeek 0.820 0.880 0.778 0.826 0.042 QWEN 0.820 0.719 0.820 0.786 0.048 Positivity ratio is the most stable finding of the three: SD ranges from 0.032 to 0.048, and all nine values (3 models × 3 runs) exceed 0.71. The positivity bias documented in Section 3.5 is therefore not an artefact of a single stochastic generation. Table 23: Protagonist gender across three runs (5 titles). Model R1 M:F R2 M:F R3 M:F Pooled M:F GPT 4:1 3:1 3:1 10:3 (3.3:1) DeepSeek 4:0 3:1 2:1 9:2 (4.5:1) QWEN 5:0 3:1 2:1 10:2 (5.0:1) Pooled overall 29:7 (4.1:1) Protagonist gender distribution is stable across all three runs: male protagonists outnumber female protagonists in every run and every model, with no run producing female-majority output. The pooled ratio (4.1:1) is consistent with the full-corpus finding (3.7:1). Table 24: Manual annotation counts on the two additional runs (R1, R2), by error type and model. Model Ling. R1 Ling. R2 Sem. R1 Sem. R2 Tot. R1 Tot. R2 GPT 15 35 4 12 19 47 DeepSeek 35 15 11 2 46 17 QWEN 18 11 6 8 24 19 Total 68 61 21 22 89 83 Manual annotation of the two additional runs confirms that linguistic and semantic errors persist across independent generations: errors are present in every run for every model, and no model produces error-free output in either run. Recurring error types mirror those documented in the main corpus. For agreement errors, QWEN produced موبائل فونوں کا چھوٹا سا شاپ (“a small mobile phone shop”), where the masculine noun “shop” requires a masculine modifier but the broader noun phrase is malformed. For semantic anomaly, a DeepSeek sentence introduced a character as “He was Haji Fazl, the one with soggy bread”, syntactically well-formed but semantically incoherent as an identifying description. For context mismatch, a mother addressing her son said تو تو میرے سامنے بیٹھا ہے، میرے ابّا کی دعا قبول ہو گئی (“You are sitting before me; my father’s prayer was answered”), where “my father” refers to the mother’s own father rather than the son’s, a kinship error. Cultural and script-level errors also recurred: GPT mixed Punjabi phrases into an otherwise Urdu narrative and embedded Latin-script English words within Urdu text; QWEN’s retelling of The Hare and the Tortoise introduced a Christmas card-making competition, a Western cultural substitution with no grounding in Pakistani context; and em-dash usage recurred in GPT stories, consistent with the stylistic-transfer finding of Section 3.1.1. Appendix J English-Language Control: Full Results To separate what is specific to Urdu generation from what is a general property of these three models, the same three models were prompted with the same seven TSST titles in English (21 stories), evaluated on the three dimensions directly comparable across languages: positivity bias, memorisation behaviour, and title repetition rate. Script contamination, space deletion, and em-dash usage were not carried over, as they are either specific to Arabic-script generation or normatively acceptable in English prose. Table 25: Character-network sentiment: Urdu vs. English. Model Ur. Pos. En. Pos. Ur. Edge En. Edge GPT 0.78 0.74 0.29 0.18 QWEN 0.82 0.60 0.38 0.15 DeepSeek 0.82 0.57 0.37 0.00 Overall 0.807 0.637 0.347 0.110 The positivity ratio is 17 points higher in Urdu (0.807) than in English (0.637); the gap is 22 and 25 points for QWEN and DeepSeek respectively, while GPT shows the smallest difference (0.78 vs. 0.74), suggesting its positivity bias is less language-dependent. Mean signed edge weight follows the same pattern: DeepSeek scores 0.00 in English versus 0.37 in Urdu, indicating that emotional amplification in Urdu stories is more intense as well as more positive. Table 26: Memorisation outcomes: Urdu vs. English (7 TSST titles). Model Ur. FR% En. FR% Ur. Comp. En. Comp. GPT 25.0 14.3 0.50 0.386 QWEN 08.3 42.9 0.27 0.599 DeepSeek 25.0 57.1 0.43 0.664 Overall 19.4 38.1 0.40 0.550 Table 27: Cross-model plot overlap (mean Jaccard): Urdu vs. English. Model pair Urdu English GPT – DeepSeek 0.45 0.393 GPT – QWEN 0.33 0.486 DeepSeek – QWEN 0.34 0.671 Overall 0.37 0.517 English stories show nearly double the faithful-retelling rate of Urdu (38.1% vs. 19.4%), and cross-model plot overlap is higher in English (0.517 vs. 0.373). Both findings are consistent with stronger training-data coverage of English source texts. Model rankings reverse across languages: QWEN is the weakest memoriser in Urdu (8.3%) but second-highest in English (42.9%), and DeepSeek is strongest in English (57.1%) but middle-ranked in Urdu (25.0%) – each model’s memorisation behaviour appears to reflect its own language-specific training coverage rather than a fixed model property. The higher Original label rate in Urdu does not reflect creativity; as the TRR analysis shows, it reflects the absence of source-text knowledge. Table 28: Title Repetition Rate: Urdu vs. English. Model Ur. TRR En. TRR (full) En. TRR (2k w) Ur. Cnt En. Cnt GPT 0.40 0.069 0.049 6.06 4.00 QWEN 0.40 0.110 0.070 5.19 2.86 DeepSeek 0.43 0.124 0.105 8.12 4.00 Overall 0.41 0.101 0.075 6.46 3.62 Urdu TRR is 3.5× to 5.8× higher than English TRR across all three models, aligning directly with the Lexical Title Unpacking finding. Raw counts confirm this independently of story length (Urdu mean 6.46 vs. English mean 3.62), and the pattern holds after truncating English stories to 2,000 words, so it is not an artefact of story-length variation. Table 29: Protagonist gender: Urdu vs. English. Model Ur. M Ur. F Ur. M:F En. M En. F En. M:F GPT 22 7 3.1:1 2 5 0.4:1 QWEN 26 5 5.2:1 3 4 0.75:1 DeepSeek 22 7 3.1:1 1 6 0.17:1 Overall 70 19 3.7:1 6 15 0.4:1 The Urdu male-to-female ratio (3.7:1) reverses to 0.4:1 in English, suggesting the male-default pattern in Urdu is language-conditioned rather than universal. We note a title-selection confound: several TSST titles have canonical female protagonists (The Yellow Wallpaper, The Story of an Hour, Boys and Girls), which partially accounts for the English female majority; the comparison should therefore be read as indicative rather than a clean cross-lingual experiment. On name diversity, QWEN used “Ayesha” in 5 of 7 English stories and DeepSeek used “Zara” in 4 of 7: despite the language change, models continue to draw from a limited name inventory. Pakistani vocabulary in English stories. The generation prompt specified a Pakistani cultural context. In the English stories, models signalled this setting through Urdu loanwords embedded in English prose, including “haveli”, “mohalla”, “rishta”, “nihari”, “Allah ka shukar”, and a dupatta’s pallu. This is the inverse of the pattern documented in Urdu (Section 3.2), where Pakistani surface vocabulary was present but cultural grounding remained shallow: when writing in English about a Pakistani setting, models appear to rely on lexical markers as a proxy for cultural knowledge rather than producing contextually situated narrative content. Appendix K Cross-Family Judge Validation: Full Results To address same-family-judge concerns, the memorisation-scoring and character-network-extraction pipelines were re-run with Gemini-3.5-Flash as an out-of-family judge on all 36 story pairs and all 93 stories respectively. Table 30: Memorisation results: GPT-5.4 vs. Gemini-3.5-Flash. Metric GPT-5.4 Gemini GPT faithful retelling (%) 25.0 25.0 DeepSeek faithful retelling (%) 25.0 25.0 QWEN faithful retelling (%) 08.3 08.3 GPT composite score 0.500 0.364 DeepSeek composite score 0.430 0.347 QWEN composite score 0.270 0.162 Cross overlap (GPT–DeepSeek) 0.450 0.388 Cross overlap (GPT–QWEN) 0.330 0.217 Cross overlap (DeepSeek–QWEN) 0.340 0.271 Faithful-retelling percentages are identical across both judges for all three models. Composite scores and cross-model overlap are somewhat lower under Gemini, but direction and model ordering are preserved (GPT >> DeepSeek >> QWEN). Categorical memorisation findings are therefore robust to judge identity, while continuous scores should be read as approximate. Table 31: Character-network metrics: GPT-5.4 vs. Gemini-3.5-Flash. Metric GPT-5.4 Gemini GPT positivity ratio 0.780 0.835 DeepSeek positivity ratio 0.820 0.806 QWEN positivity ratio 0.820 0.877 GPT mean signed weight 0.290 0.442 DeepSeek mean signed weight 0.370 0.458 QWEN mean signed weight 0.380 0.572 Jaccard (GPT–DeepSeek) 0.013 0.008 Jaccard (GPT–QWEN) 0.018 0.022 Jaccard (DeepSeek–QWEN) 0.013 0.012 Positivity ratios under Gemini are consistent with the GPT-5.4 results in both direction and model ordering, and all values remain well above 0.80. Cross-model Jaccard overlap remains near zero under both judges (median 0.0 across all pairs), confirming that the character-diversity finding is robust to judge identity. Mean signed edge weight is somewhat higher under Gemini, indicating that intensity estimates are more sensitive to judge choice than directional findings are. GPT-story error detection. To cross-validate linguistic/semantic error detection against an out-of-family judge, Gemini-3.5-Flash was run on the 31 GPT-generated stories. Gemini flagged 176 errors, of which 149 were confirmed correct by a native Urdu speaker on manual review (84.7% precision); the remaining 27 were false positives arising from legitimate spelling variants and acceptable space-deletion patterns in Urdu orthography. This validation was conducted only for GPT-generated stories due to budget constraints (see Limitations); the agreement between Gemini and human annotation nonetheless provides evidence that the error patterns identified in Section 3.5 are not artefacts of same-family judge preference. Appendix L Human Validation of Memorisation Labels A native Urdu speaker manually reviewed 13 story pairs, covering all three memorisation-label categories (PR, PI, OR) and all four title categories, and compared their own judgement against the GPT-4o-mini judge’s categorical label. Table 32: Human validation of memorisation labels (13 pairs). PR = Plot Reproduction, PI = Partial Inheritance, OR = Original. # ID Model Cat. Judge Human Match 1 13 GPT Fable PR PR Yes 2 13 QWEN Fable PR PR Yes 3 14 DeepSeek Fable PR PR Yes 4 22 GPT Fable PR PR Yes 5 25 DeepSeek TSST PR PR Yes 6 9 GPT TSST OR OR Yes 7 11 GPT UST OR OR Yes 8 17 DeepSeek UST OR OR Yes 9 20 GPT UST PI PI Yes 10 21 DeepSeek UST PI PI Yes 11 25 QWEN TSST PI OR No 12 31 DeepSeek UST PR OR No 13 17 GPT UST PI OR No Agreement is 10/13 (76.9%). All four Fable cases and all OR cases assigned by the judge agree exactly with human judgement (3/3). All three mismatches occur in UST titles, where the judge over-estimates plot reproduction (assigning PR or PI where the human judged OR), consistent with the Section 4 observation that the automatic judge is most prone to error on titles it has never seen a source text for. Appendix M Extended Examples: Context Mismatch and Semantic Errors This appendix collects additional illustrative examples for the error sub-categories summarised in Section 3.1.2, beyond the one or two examples given in the main text for space. Kinship-term misuse (additional examples). A DeepSeek story introduces a maternal-side visit but uses دادی اماں (dādī ammā, “paternal grandmother”) for the grandmother encountered there. QWEN uses ابا جان (abbā jān, “father, respectful”) for an elderly stranger across multiple stories, an over-generalised address form. Long-range gender breakdown (additional example). In one DeepSeek story, two figures including a girl are jointly referred to as دو نوجوان (do nau-jawān, “two young men”), a masculine plural noun applied across a mixed-gender pair. Cultural and religious anachronism (additional examples). Another QWEN story describes a moon-sighting occurring in the morning, inconsistent with the lunar-sighting practice it is meant to depict. A GPT story promotes a chef to کیپٹن (kepṭan, “captain”), a term reserved for restaurant front-of-house staff in Pakistani usage rather than kitchen roles. Appendix N Space Deletion Errors GPT fuses ہو (ho, “be”) constructions at substantial rates (ہوگیا 41%, ہوجاتا 40%, ہوگا 67%), while DeepSeek and QWEN write the same forms with a space in nearly every instance. The pattern inverts for آ (ā, “come”) auxiliaries: DeepSeek fuses آگیا 44% and آگئے 67% of the time, while GPT and QWEN remain near zero. QWEN shows almost no fusion of either family. Each model therefore mishandles a different closed inventory of compounds while writing the rest correctly, indicating that the errors reflect sub-word vocabulary choices in each model’s tokeniser rather than a shared weakness in Urdu generation. Crucially, this distinction matters downstream: whitespace-based Urdu NLP pipelines treat ہوگیا (fused) and ہو گیا (spaced) as different tokens, so if LLM output is reused as synthetic training data, a model’s fusion preferences gain additional statistical support and risk self-reinforcing contamination of the language’s digital record. Appendix O Tractability: GPT-5.5 Remediation Prompt The post-editing pass described in Section 4 used the system prompt reproduced below. The model was issued one call per story with a JSON list of numbered sentences; the response was constrained to a JSON object with a single results array, with one entry per detected issue. The prompt enforces minimal edits (a single inserted space or a single replaced word) and includes a self-check for SpaceDeletion to suppress punctuation-adjacent false positives. Sentence splitting, JSON parsing, and a post-hoc validator that rejected any SpaceDeletion whose inserted space did not occur between two Urdu letters were applied around this prompt. You are an expert Urdu linguist and proofreader. You will receive a JSON list of numbered Urdu sentences from one story. Detect the following issue types and fix them with MINIMAL changes. Error Types 1. SpaceDeletion (a.k.a. Word Segmentation error). Two adjacent Urdu WORDS are written as a single token because the space BETWEEN THE TWO WORDS is missing. This is ONLY about word-to-word boundaries. Flag examples (word-word fusion): • چاہتیہوں → چاہتی ہوں • گھرگیا → گھر گیا • ہوگیاہے → ہو گیا ہے • میںنے → میں نے • خوبصورتلڑکی → خوبصورت لڑکی Do NOT flag any of these: • Missing space AROUND punctuation (، ۔ ? ! . :) e.g. ہوگا،ہوچکا is NOT a SpaceDeletion issue. • Missing space around quotes (", ”, “, ’, ’, ‘), brackets, parentheses, dashes, or numbers. • Missing line breaks, paragraph breaks, or dialogue formatting. • Long or multi-clause sentences. Fix rule: insert ONLY the single missing space between the two fused WORDS. Do not touch any punctuation, quotes, or formatting. correct_sentence must differ from incorrect_sentence by exactly one (or a few) inserted SPACE characters between word characters. 2. SpellingIssue. A single word is misspelled. ہیریو → ہیرو; مسئیبت → مصیبت. Fix: change ONLY the misspelled word. 3. GrammarIssue. Wrong case marking, missing or wrong نے/کو/سے, verb agreement, etc. اُنہوں نے مجھے گرمجوشی سے خیرمقدم کیا → اُنہوں نے میرا گرمجوشی سے خیرمقدم کیا. Fix: minimal grammatical correction. 4. Agreement Error. Gender, number, or oblique-case agreement error between a noun and its modifier (adjective, demonstrative, quantifier, or possessive). ایک چھوٹی سی لینگر خانہ → ایک چھوٹا سا لنگر خانہ; اتنی خراب کر دیتی ہے → اتنا خراب کر دیتی ہے. 5. Semantic Anomaly. Sentence is grammatically OK but a word choice makes it meaningless or wrong in context. آواز میں ایک عجیب سی کانپ تھی → آواز میں ایک عجیب سی کپکپاہٹ تھی. Fix: replace ONLY the offending word. ======= GENERAL RULES ===== • One sentence may contain MULTIPLE issues → emit one entry per issue. • If a sentence has NO issue, emit nothing for it. • incorrect_sentence MUST equal the original sentence text exactly. • correct_sentence MUST contain ONLY the minimal fix for THAT issue. • Never split or merge sentences. Never reformat dialogue or quotes. • Self-check before emitting SpaceDeletion: are the two characters around the inserted space BOTH Urdu letters, and was there NO whitespace and NO punctuation between them in the original? If NO, do NOT emit SpaceDeletion. Respond ONLY with valid JSON, no markdown, no commentary: "results": [ "sentence_id": <int>, "error_tag": "<SpaceDeletion|SemanticAnomaly|SpellingIssue|GrammarIssue|AgreementError>", "incorrect_sentence": "<original sentence>", "correct_sentence": "<sentence with ONLY the minimal fix>" ] “‘ end