Paper deep dive
VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages
Usneek Singh, Poorvaja Veera Balaji Kumar, Parth Nanda, Anand Madhusoodanan, Geyang Guo, Wei Xu, Junyi Jessy L
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 9/3/2026, 3:18:33 AM
Summary
The paper introduces VakyArth, the first pragmatic benchmark for Indic languages (Hindi, Punjabi, Tamil, Malayalam), designed to evaluate Large Language Models (LLMs) on pragmatic competence. The benchmark covers five phenomena: deixis, speech acts, implicature, social pragmatics, and coherence, using multiple-choice questions, natural language inference, and translation tasks. Results indicate that current multilingual LLMs consistently fail to interpret meanings rooted in Indic linguistic and cultural conventions. Key findings include higher accuracy in MCQs compared to NLI, a translation advantage for Indo-Aryan languages over Dravidian languages, and the inability of automatic metrics like COMET to detect pragmatically unfaithful outputs.
Entities (19)
Relation Signals (16)
Tamil → belongstofamily → Dravidian
confidence 100% · Tamil... Family Dravidian
Malayalam → belongstofamily → Dravidian
confidence 100% · Malayalam... Family Dravidian
Hindi → belongstofamily → Indo-Aryan
confidence 100% · Hindi... Family Indo-Aryan
Punjabi → belongstofamily → Indo-Aryan
confidence 100% · Punjabi... Family Indo-Aryan
VakyArth → coverslanguage → Punjabi
confidence 100% · VakyArth is a diagnostic benchmark that covers Hindi, Punjabi, Tamil, and Malayalam
VakyArth → coverslanguage → Tamil
confidence 100% · VakyArth is a diagnostic benchmark that covers Hindi, Punjabi, Tamil, and Malayalam
VakyArth → coverslanguage → Malayalam
confidence 100% · VakyArth is a diagnostic benchmark that covers Hindi, Punjabi, Tamil, and Malayalam
VakyArth → coverslanguage → Hindi
confidence 100% · VakyArth is a diagnostic benchmark that covers Hindi, Punjabi, Tamil, and Malayalam
VakyArth → evaluatesphenomenon → Coherence
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Real-world communication often requires pragmatic reasoning: interpreting meanings implied through context and cultural convention rather than stated literally. Existing pragmatic evaluation remains largely limited to English and high-resource languages, leaving Indic languages unexplored despite their linguistic and cultural diversity. We introduce VakyArth, the first pragmatic benchmark for Indic languages, designed as a diagnostic evaluation covering Hindi, Punjabi, Tamil, and Malayalam. VakyArth evaluates models across five phenomena: deixis, speech acts, implicature, social pragmatics, and coherence; through multiple-choice questions, natural language inference, and translation, with all items authored by native speakers. Across multilingual large language models (LLMs) of varying families and sizes, we find consistent failures on pragmatic meanings rooted in Indic linguistic and cultural conventions. Our analysis shows systematic differences across languages and tasks: MCQ accuracy exceeds NLI accuracy in all model-language combinations, translation performance does not reliably track pragmatic understanding, and Indo-Aryan languages show a translation advantage over Dravidian languages. We further show that automatic translation metrics can miss fluent but pragmatically unfaithful outputs, especially for implicature and deixis.
Tags
Links
- Source: https://arxiv.org/abs/2609.01788v1
- Canonical: https://arxiv.org/abs/2609.01788v1
Trouble viewing inline? Open PDF directly →
Full Text
59,982 characters extracted from source content.
Expand or collapse full text
VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages Usneek Singh 1∗ Poorvaja Veera Balaji Kumar 1 Parth Nanda 1 Anand Madhusoodanan 1 Geyang Guo 1 Wei Xu 1 Junyi Jessy Li 2 1 Georgia Institute of Technology, 2 University of Texas at Austin ∗ usingh68@gatech.edu Abstract Real-world communication often requires prag- matic reasoning: interpreting meanings im- plied through context and cultural convention rather than stated literally. Existing pragmatic evaluation remains largely limited to English and high-resource languages, leaving Indic lan- guages unexplored despite their linguistic and cultural diversity. We introduce VakyArth, the first pragmatic benchmark for Indic languages, designed as a diagnostic evaluation covering Hindi, Punjabi, Tamil, and Malayalam. Vak- yArth evaluates models across five phenom- ena: deixis, speech acts, implicature, social pragmatics, and coherence; through multiple- choice questions, natural language inference, and translation, with all items authored by na- tive speakers. Across multilingual large lan- guage models (LLMs) of varying families and sizes, we find consistent failures on pragmatic meanings rooted in Indic linguistic and cul- tural conventions. Our analysis shows sys- tematic differences across languages and tasks: MCQ accuracy exceeds NLI accuracy in all model-language combinations, translation per- formance does not reliably track pragmatic un- derstanding, and Indo-Aryan languages show a translation advantage over Dravidian languages. We further show that automatic translation met- rics can miss fluent but pragmatically unfaithful outputs, especially for implicature and deixis. 1 Introduction As large language models (LLMs) become widely used, their impact depends not only on supporting professional tasks but also on helping users navi- gate everyday communication across diverse lin- guistic and cultural contexts. India offers such a set- ting: it has the second-largest AI user base globally, yet usage remains concentrated in professional con- texts such as software development, with much less adoption for everyday and personal tasks (Appel, 2026). Broadening AI’s impact in India, therefore, Ek kapde ki dukaan mein salesman keh raha hai, 'Yeh suit 5000 ka hai, voh 3000 ka hai, aur voh saamne wala shelf ka 8000 ka hai.' Grahak ne kaha, 'Mujhe beech wala dikhao". In a clothing store, the salesman says: 'This suit is ₹5000, that one is ₹3000, and the one on the shelf over there is ₹8000.' The customer says: 'Show me the middle one.' Hindi Ordinal vs Spatial Deixis Grahak kaunsa suit dekhna chahata hai? Which suit does the customer want? A) Rs 3000 one. B) Rs 5000 one. C) Rs 8000 one. "Beech wala" refers to the middle price (₹5000), not the spatially middle suit. Models conflate ordinal and spatial deixis. 3000 wala suit Rs 3000 one 5000 waala suit Rs 5000 suit. Correct. The customer wants the middle priced suit. Figure 1: A Hindi deixis item from VAKYARTH. The customer asks for beech wala (the middle one), referring to the middle-priced suit (Rs 5000). Any Hindi speaker resolves this by price ranking, not by the order of men- tion. Evaluated models pick Rs 3000 missing that beech wala here anchors to price, not utterance order. requires moving beyond work-related applications toward systems that can support daily usage. Everyday communication in Indic contexts of- ten depends on culturally grounded meanings that go beyond literal form. For example, as shown in Figure 1, when a Hindi-speaking customer asks for “beech wala” in a clothing-store dialogue, the phrase literally means “the middle one”. In context, however, the phrase refers to the middle-priced suit rather than the spatially middle item, a distinction missed by multiple existing LLMs (e.g., Gemma- 4-31B). Pragmatic reasoning is needed here: recov- ering meaning from context, cultural convention, and speaker intent (Ma et al., 2025). 1 arXiv:2609.01788v1 [cs.CL] 1 Sep 2026 Existing pragmatic evaluation mainly focuses on English or high-resource languages (Ma et al., 2025; Park et al., 2024b). To the best of our knowl- edge, no pragmatic evaluation exists for Indic lan- guages. Furthermore, Indic pragmatics is highly diverse: languages across India span multiple fami- lies, scripts, and sociolinguistic communities, with pragmatic cues encoded differently. We introduce VakyArth 1 , the first pragmatic benchmark for Indic languages. VakyArth is a diagnostic benchmark that covers Hindi, Punjabi, Tamil, and Malayalam, which represent diverse linguistic and regional contexts across India. It evaluates multilingual LLMs along five pragmatic phenomena: deixis, speech acts, implicature, so- cial pragmatics, and coherence (see examples in Figure 2). Each phenomenon is constructed in the corresponding language and evaluated under three task formats: multiple-choice questions, nat- ural language inference, and translation, allowing us to assess pragmatic understanding at different levels of explicitness. All items are authored by native speakers and include naturally code-mixed instances to reflect real usage patterns. We evaluate five multilingual LLMs across model families and sizes (ranging from 8B to 111B), and find consistent failure patterns rooted in Indic linguistic and cultural conventions. Models struggle with culturally grounded sarcasm and hy- perbole, indirect refusals and requests, shifting de- ictic anchors such as “kal” and “parso”, and social cues involving kinship address and gender norms. Our quantitative analysis further reveals system- atic differences across tasks, phenomena, and lan- guages. Indo-Aryan languages show a translation advantage over Dravidian languages, likely reflect- ing differences in pretraining coverage (Naous and Xu, 2025). MCQ accuracy exceeds NLI accu- racy in all model-language combinations, suggest- ing pragmatic interpretation benefits from answer- choice scaffolding. Translation performance does not track MCQ or NLI scores, indicating that recog- nizing pragmatic meaning and expressing it faith- fully across languages are distinct capabilities. In- terestingly, Gemma-4-31B achieves the highest per- formance despite being smaller and lacking Indic specialization. Automatic metrics also fail to pe- nalize fluent but pragmatically unfaithful outputs for implicature and deixis. 1 VakyArth combines Vakya (utterance) and Arth (mean- ing) from Sanskrit, reflecting the benchmark’s focus on the meaning beneath the utterance. We release our dataset and code athttps:// github.com/Usneek1/VakyArth. 2 Related Work Pragmatic evaluation of LLMs.The evaluation of pragmatic competence in LLMs has attracted growing attention. Ma et al. (2025) provide a comprehensive survey, noting that most resources cover a narrow set of phenomena and are heavily skewed toward high-resource languages. Early di- agnostic work includes IMPPRES (Jeretic et al., 2020), a targeted NLI dataset for implicature and presupposition, and Ruis et al. (2023), who showed instruction-tuned models substantially outperform base models on implicature resolution. More com- prehensive benchmarks have since emerged: PUB (Sravanthi et al., 2024) covers implicature, pre- supposition, reference, and deixis across fourteen multiple-choice tasks, and Yu et al. (2026) focus on alternative-based pragmatic reasoning. All of these benchmarks are in English. Multilingual and cross-cultural pragmatic benchmarks. Extending pragmatic evaluation beyond English has been limited. MultiPragEval (Park et al., 2024b) evaluates models across En- glish, German, Korean, and Chinese using Gricean maxims, but covers only European and East Asian languages. Park et al. (2024a) and Koo et al. (2025) introduce pragmatic benchmarks for Korean, with latter reporting substantial gaps between human and model performance on indirect speech acts. A related line of work evaluates cultural knowl- edge rather than pragmatic competence (Chiu et al., 2024; Shaikh et al., 2023). Unlike these studies, which test whether models know cultural facts, our work evaluates whether models can reason about implied meaning in context. No prior work evalu- ates pragmatic competence for any Indic language. Evaluation of LLMs on Indic languages.Indic NLP evaluation has progressed from foundational benchmarks to increasingly culturally grounded ones, yet pragmatic competence remains unad- dressed. Early work such as IndicNLPSuite (Kak- wani et al., 2020) and IndicXTREME (Doddapa- neni et al., 2023) established baselines for clas- sification and named entity recognition. More recent work covers generation: IndicGenBench (Singh et al., 2024) evaluates translation and sum- marization across 29 languages, and MILU (Verma et al., 2025) introduces culturally specific knowl- 2 edge tasks. The most culturally grounded effort is PARIKSHA (Watts et al., 2024), which evalu- ates 30 models across 10 Indic languages and finds that LLM evaluators align poorly with humans on culturally nuanced responses — a finding that res- onates with our own human-COMET divergence on pragmatic translation. Despite this progression, none of these benchmarks require pragmatic infer- ence: understanding implied meaning, resolving deictic reference, or interpreting indirect speech acts. VakyArth is the first to address this gap. 3 Evaluation Design We design our evaluation around three guiding prin- ciples: phenomena coverage (the testing should span the core dimensions of pragmatic compe- tence, not a single category), task diversity (evalu- ation should probe understanding at multiple levels of explicitness), and cultural authenticity (items should reflect how Indic languages are actually used, including code-mixing and culturally specific pragmatic conventions). 3.1 Pragmatic Phenomena We evaluate five phenomena that collectively cap- ture the range of non-literal, context-dependent meaning central to everyday communication (see Figure 2). These categories follow the taxonomy of Ma et al. (2025), but are instantiated through distinctively Indic examples drawn from cultural knowledge of native speakers. (1) Deixisrefers to words whose reference de- pends entirely on the discourse context, who is speaking, where, and when (Webber, 1988). In- dic languages present particular deictic challenges: temporal words like kal (yesterday/tomorrow) and parso (day before yesterday/day after tomorrow) are bidirectional, resolving to past or future only through context; proximal-distal contrasts like intha-antha carry meaning that translation flattens; and cultural calendar references. For example, A Tamil item presents a speaker comparing two pumpkins: “intha poosani paaka pinjaa irukku, antha poosani nallaa irukku” (“this pumpkin looks less ripe, that one looks better”) illustrates how the proximal–distal contrast (intha vs. an- tha) carries the entire communicative point. A translation that flattens both to “the pumpkin” is grammatically fine but loses the comparison entirely. We also add multi-party dialogues for more com- plexity. For example, in a Punjabi station exchange, speaker B says “tussi edher a jao” (“you come over here”) while speaker A is already walking to- ward the wrong gate. Correctly resolving “edher” (“here”) requires tracking which gate each speaker is at - a reference chain that models often collapse. (2) Speech Actsare utterances that perform ac- tions such as requests, offers, refusals, and warn- ings, whose illocutionary force often diverges sub- stantially from their surface form (Austin, 1975). South Asian communicative norms make particu- larly heavy use of indirect speech acts as politeness strategies. For example, A mother who enters her son’s room at 11 AM and asks (in Hindi) "Beta, kya aaj chutti hai kya?" ("Son, is it a holiday today?") is not seek- ing calendar information. She is delivering an indirect command to wake up and get to work. Likewise, a Punjabi host who says “kuch khaas nahi baneya” (“nothing special was made”) while placing food in front of guests is performing a ritual offer, not a sincere self-criticism. (3) Implicaturerefers to meaning conveyed but not explicitly stated, requiring the listener to rea- son beyond the literal content (Bach, 2006). In In- dic languages, implicature is frequently expressed through culturally embedded expressions that are opaque without cultural knowledge. For example, A teacher who tells Rahul "Tumne toh iss pareek- sha mein pass hokar kila jeet liya" ("You have conquered a fort by passing this exam") is using a common Hindi hyperbole to express praise. A model that processes the surface meaning pre- dicts that Rahul won a real fort; the correct infer- ence is that the teacher is simply congratulating him on passing. Similarly, in Tamil, a mother telling departing guests “kai nenachu po” (“wet your hands and go”) is not issuing a hygiene instruction but ex- tending a warm, face-saving invitation to stay and eat. The ritual indirectness is the social point of the expression. (4) Social Pragmaticscaptures how social fac- tors such as kinship, gender norms, power dynam- ics, and cultural conventions around face shape language use (Ma et al., 2025). These items require models to bring cultural world knowledge to bear 3 IMPLICATUREDEIXIS SPEECH ACTSSOCIAL PRAGMATICS COHERENCE Veetukku vantha virunthaaligal purapidrapo kai nenachittu ponganu amma sonnanga. Mother told departing guests, "wet your hands and go." Premise Amma virunthaaligalai kai kaluva sonnaargal. Mother is asking the guests to wash their hands. Hypothesis LABEL: CONTRADICTION "Kai nenachittu po" literally means "wet your hands and go" but is a traditional Tamil farewell invitation to stay and eat. Parso ki meeting hum zyada opinions nhi de paye the, lekin parso ki meeting mein zaroor denge. Source We couldn't share many opinions in the day before yesterday's meeting, but we will definitely do so in the day after tomorrow's meeting. Target "Parso" appears twice, once meaning day before yesterday, once meaning day after tomorrow. Both senses must be inferred from context. Jado Priya mehmanan nu khana serve kar rahi si, usne keha, "Kuch khaas nahi baneya ajj, par try karo." While serving food, Priya said, "nothing special today, but please try." Priya mehmanan nu khaane di quality baare warning de rahi hai. Priya is warning guests about the quality of food. Premise Hypothesis LABEL: CONTRADICTION Downplaying while serving is a ritual politeness in Punjabi culture, the actual speech act is a warm offer, not a warning. Nee ninte suhruthinte veetil poyappol avante valiya sahodhari Anjana-ye kandu. Nee avale engane vinayathode vilikkum? You visit your friend's house and meet his older sister Anjana. How do you address her politely? Question ANSWER: B In Malayalam, addressing an elder requires combining their name with the kinship honorific, using the name or honorific alone is considered impolite. Options Question Doctor banan lai bahut mehnat karni pendi hai. Pehla kadam hunda hai NEET ........... Doctory vich speciality vaaste tan hor kai saal lagg jande ne. Kehda sentence paragraph vich fit nahi ho reha? Becoming a doctor requires a lot of hard work. The first step is ..... Specialization takes several more years. Which sentence does NOT fit in the paragraph? Options ANSWER: C This sentence breaks the logical chain. A. Anjana B. Anjana chechi C. Chechi D. Aunty TAMIL PUNJABIMALAYALAM HINDI ENGLISH A. Doctor banan lai bahut mehnat karni pendi hai. Becoming a doctor requires a lot of hard work. B. Uthe padhayi kaafi aukhi hundi hai. The studies there are quite difficult. C. Par doctory ek bahut nek seva hai. But medicine is a noble profession. D. Doctory vich speciality vaaste tan ........ jande Specialization takes several more years. TAMIL PUNJABI PUNJABI VakyArth . Pragmatic Evaluation on Indic Languages MCQNLITranslation Figure 2: VakyArth benchmark overview. Items span five pragmatic phenomena : implicature, deixis, speech acts, social pragmatics, and coherence across Hindi, Punjabi, Tamil, and Malayalam, evaluated under NLI, translation, and MCQ formats. In each case, the correct label or translation cannot be recovered from surface form alone. in ways that go beyond linguistic competence. For example, In Punjabi, greeting a male friend with a hug while greeting his wife with folded hands (hath jodna) and Sat Sri Akal is the culturally appropri- ate norm for mixed-gender visits. A model with- out cultural grounding might incorrectly read this a sign of differential regard toward the wife. In Malayalam, addressing a friend’s older sister requires combining her name with the kinship honorific chechi: “Anjana chechi” is the polite form, while “Anjana” alone signals disrespect and “Chechi” alone is too impersonal. (5) Coherencerequires reasoning about how ut- terances relate within a larger discourse unit, be- yond what any single sentence conveys (Van Dijk et al., 1997). Existing work draws extensively on sentence-ordering or narrative cloze tasks (Barzilay and Lapata, 2008; Mostafazadeh et al., 2016; Beyer et al., 2021). In contrast, our items test discourse- level reasoning within Indic language paragraphs, where topically plausible but discourse-incoherent distractors require models to track argumentative structure rather rely on keyword overlap. For ex- ample, A Tamil paragraph describes a traditional ba- nana leaf meal in sequence.. hot rice, lentils, ghee, kootu, poriyal ending with payasam. The most appropriate response the accompaniments like vadai and appalam that complete the feast. Options like "banana leaf meals are no longer common" or "it is difficult to serve on a leaf" are topically related to banana leaf meals but break the sequence the paragraph has built up. 4 3.2 Task Types Each phenomenon is evaluated across three task formats, chosen to probe pragmatic understanding at different levels of explicitness: Multiple-Choice Question (MCQ). The model picks the most appropriate interpretation from four options. We deliberately construct distractors that reflect the literal meaning of the utterance, so mod- els that do not reason pragmatically will be misled. Natural Language Inference (NLI). The model labels the relationship between a premise and a hypothesis as entailment, contradiction, or neutral. This is intended to be harder than MCQ since no options to guide the model. We include more con- tradiction cases, especially for implicature, since models tend to treat figurative premises as entailing hypotheses that they actually contradict. Translation.The model translates an Indic- language utterance into English. Translating prag- matic content is challenging because implied mean- ings, cultural references, and speaker intent do not always have a direct equivalent in English. A model that does not understand the pragmatic meaning will produce a translation that is fluent but wrong. 3.3 Dataset Construction All items were written from scratch by a team of four native speakers, one per language, each with graduate-level NLP training. Annotators drew on personal linguistic knowledge, cultural familiarity, and grammar references to write items. Each item was reviewed by the full team to check for ambi- guity, label correctness, and translation accuracy. Items where the correct answer was recoverable from surface cues alone were removed. Most items are code-mixed with English. This reflects how people actually use these languages in daily life, especially in Hindi and Punjabi. Tamil and Malay- alam items have less code-mixing, consistent with real usage patterns in those communities. Table 1 summarizes the language statistics. Vak- yArth contains 578 items across four languages: Hindi (185), Punjabi (125), Tamil (134), Malay- alam (134), spanning a combined speaker pop- ulation exceeding 800 million. The corpus to- tals 14,921 words of source text and 424 human- authored English reference sentences in the trans- lation subcorpus. The token/word figures reflect a fundamental typological difference between the two language families: Tamil and Malayalam are agglutinative, requiring significantly more tokens HindiPunjabiTamilMalay. FamilyIndo-Aryan Indo-Aryan Dravidian Dravidian ScriptDevanagari GurmukhiTamilMalayalam Items185125134134 TR sen.1474886143 T/W (Llama)1.951.903.154.44 T/W (Gemma)1.761.732.843.82 T/W (Qwen)1.961.873.134.32 Table 1: Language statistics for VakyArth. T/W (To- kens/Words) is the average tokens per word under each model’s tokenizer for Latin-transliterated source text. Dravidian languages consistently require 1.6–2.5× more tokens per word than Indo-Aryan languages across all three tokenizers. TR sentences = total sentences in the translation subcorpus. per word than the morphologically simpler Hindi & Punjabi, consistently across all three tokenizers. 4 Evaluation Setup Models.We evaluate five instruction-tuned mod- els ranging from 8B to 111B parameters. Three are open-weight models run locally on an NVIDIA H100 GPU via the HuggingFace Transformers li- brary (Wolf et al., 2020):Llama-3-8B-Instruct, Gemma-4-31B, andQwen3-32B. The remaining two are open-weight multilingual models accessed via their respective APIs rather than run locally: Sarvam-105B, which has dedicated Indic language pretraining, andCommand-A-03-2025(111B pa- rameters), Cohere’s flagship multilingual model. All models are run with a temperature of 0.2 and a maximum of 128 new tokens. Prompting.We use a three-shot prompting setup for all models and tasks. Each prompt contains three fixed examples written in the same language and script as the test item. Examples are shared across phenomena within a language but are task- specific. We use few-shot prompting rather than zero-shot because pilot runs showed that models frequently failed to follow the required output for- mat under zero-shot conditions, making automatic parsing unreliable. More details on prompts can be found in Appendix A. Input format. All items are presented in Latin transliteration. This choice reflects everyday usage: Roark et al. (2020) documents informal romaniza- tion as the standard way South Asian languages are typed in digital communication, which is the 5 setting our benchmark targets. Methodologically, Jaavid et al. (2024) show that romanized Indic text reduces token fertility by 2–4×and matches or out- performs native-script representation across NLU, NLG, and MT tasks, supporting our choice to min- imize tokenization artifacts. We additionally show the effect of using native script vs. Latin script on pragmatic understanding (see §5.5). Evaluation metrics. For MCQ and NLI, we re- port accuracy, computed by exact match between the model’s predicted label and the gold answer. For translation, we report two scores: (1) automatic evaluation using COMET (Rei et al., 2020) with thewmt22-comet-dacheckpoint, and (2) human evaluation scores on a 1–5 scale covering adequacy, fluency, and pragmatic adequacy, collected from four native speaker raters, one per language. We chosewmt22-comet-daover newer checkpoints af- ter empirically validating that it achieves the high- est correlation with human judgements in our set- ting compared to XCOMET (Guerreiro et al., 2024) and MetricX (Juraska et al., 2023); we present a detailed comparison of all three metrics in §5.4. 5 Results and Analysis Table 2 presents results across all four models, four languages, and five phenomena for MCQ accuracy, NLI accuracy, and COMET score. We organize our analysis around the pragmatic failure patterns that VakyArth uncovers. We discuss what is specif- ically challenging about Indic pragmatics for cur- rent models. We also present some quantitative patterns and insights in §5.2. 5.1 Pragmatic Failure Patterns Deictic Reference Failure. Models frequently struggle to resolve deictic expressions that shift flu- idly across discourse contexts. In particular, mod- els make mistakes on bidirectional temporal ex- pressions in Hindi and Punjabi. For instance, if a speaker says on Monday, “kal mera interview hai” (“My interview is tomorrow”), and is asked on Tuesday how it went, Llama, Command-A and Gemma fail to correctly anchor the event timeline. Beyond absolute shifts, models also fail on relative calendar references conventionalized among native speakers: when presented with a Friday utterance like “agle hafte Somvar ko exam hai” (“The exam is next week on Monday”), models compute the target date as seven days from the utterance rather than three. Culturally Embedded Expressions. Models make mistakes in understanding idiomatic expres- sions rooted in regional socio-religious traditions. For instance, when a mother in Tamil tells depart- ing guests “kai nenachittu ponganu” (“wet your hands and go”), Llama, Gemma, and Qwen predict an instruction to return home and eat, while Sarvam and Command-A classify it as a hygiene command. No model recovers the correct pragmatic mean- ing which is a conventionalized invitation to stay for a meal. Similarly, when someone in Punjabi states “bas meri kudi da viyah karke mai ganga nha lavan” (“once my daughter is married, I will bathe in the Ganga”), all models predict a literal travel intention, missing the culturally embedded vow signifying relief at fulfilling a major parental duty. Indirect Speech Acts and Soft Refusals.South Asian communicative norms strongly disprefer di- rect refusals and favor indirect speech acts that models misinterpret at face value. For instance, when a speaker declines an invitation with the standard Hindi hedge “dekhte hain, try karungi” (“Let’s see, I’l try”), models predict she intends to attend. Native speakers recognize this as a po- lite soft declination. When a Tamil host asks a departing guest “enna avasaram” (“What is the rush?”), models interpret it as an inquiry into an actual emergency rather than a standard hospitality request, urging the guest to stay longer. Social and Kinship Norms.Models fail to inter- pret utterances whose meaning depends on regional social conventions, or ritual politeness scripts. For instance, in Punjabi culture, an elder telling a younger relative "tu taan bada kamzor lag reha hai" (You look quite thin) is a conventionalized expression of care; all five models misread it as mockery. Models also fail to parse inverted polite- ness scripts: when a young relative accepts cash at a Malayalam wedding saying "ithonnum ven- daayirunnu, chetta/chechi" (This wasn’t necessary, brother/sister), models predict a genuine rejection, failing to recognize the socially obligatory script of verbal refusal during physical acceptance. The ritual farewell "poyittu varam" (I will go and come back) is read as a literal promise of immediate re- turn by the evaluated models. Sarcasm, Hyperbole, and Implied Criticism. Models misinterpret non-literal assertions, such as detecting hyperbole or decoding sarcasm used for 6 Tasks:Á MCQ (Accuracy)Ü NLI (Accuracy)L Translation (COMET) DeixisSpeech ActsImplicatureSocial Prag.Coherence Lang.ModelÁÜLÁÜLÁÜLÁÜLÁÜL Hindi Llama-3-8B0.64 0.38 0.80 1.00 0.45 0.78 0.73 0.20 0.76 0.90 0.22 0.71 0.79 0.50 0.80 Gemma-4-31B0.79 0.46 0.85 1.00 0.45 0.84 0.91 0.300.86 0.90 0.22 0.820.930.50 0.83 Qwen3-32B0.71 0.850.73 0.900.73 0.54 0.730.80 0.58 0.90 0.44 0.77 0.930.92 0.76 Sarvam-105B0.93 0.69 0.81 1.00 0.55 0.67 0.91 0.300.71 0.90 0.330.75 1.00 0.750.77 Command-A-111B 0.860.92 0.841.00 0.73 0.840.91 0.80 0.830.90 0.44 0.83 0.930.92 0.83 Punjabi Llama-3-8B0.22 0.22 0.78 0.33 0.44 0.75 0.10 0.330.67 0.67 0.110.61 0.40 0.50 0.86 Gemma-4-31B0.78 0.330.85 1.00 0.56 0.82 0.60 0.330.78 1.00 0.22 0.79 0.600.330.89 Qwen3-32B0.56 0.11 0.58 0.780.11 0.49 0.500.330.35 0.830.22 0.43 0.40 0.330.52 Sarvam-105B0.67 0.44 0.84 0.67 0.33 0.800.500.330.55 0.830.22 0.710.80 0.50 0.89 Command-A-111B 0.670.44 0.850.780.22 0.80 0.40 0.44 0.730.830.22 0.68 0.80 0.50 0.89 Tamil Llama-3-8B0.33 0.380.58 0.440.33 0.54 0.670.33 0.53 0.670.330.50 0.11 0.560.53 Gemma-4-31B0.89 0.50 0.79 0.89 0.67 0.820.78 0.440.710.78 0.56 0.77 0.560.78 0.76 Qwen3-32B0.670.25 0.59 0.33 0.440.57 0.670.56 0.47 0.56 0.22 0.52 0.67 0.78 0.51 Sarvam-105B0.670.25 0.790.440.440.75 0.78 0.00 0.82 0.78 0.330.750.560.560.66 Command-A-111B 0.33 0.12 0.78 0.440.440.82 0.670.22 0.69 0.44 0.22 0.67 0.560.78 0.76 Malayalam Llama-3-8B0.22 0.22 0.58 0.56 0.56 0.61 0.50 0.22 0.55 0.44 0.22 0.58 0.56 0.44 0.59 Gemma-4-31B0.89 0.44 0.83 1.00 0.56 0.80 1.00 0.22 0.78 1.00 0.22 0.78 1.00 0.56 0.75 Qwen3-32B0.33 0.44 0.55 0.78 0.56 0.61 0.62 0.22 0.56 0.780.330.61 0.78 0.670.65 Sarvam-105B0.670.67 0.37 0.78 0.56 0.38 0.880.22 0.35 0.780.44 0.36 0.890.78 0.36 Command-A-111B 0.56 0.440.770.890.56 0.770.880.110.751.00 0.22 0.740.890.44 0.74 Best score per column in bold; second best is underlined. Table 2: MCQ accuracy, NLI accuracy, and translation COMET scores for all five models across four languages and five pragmatic phenomena. MCQ accuracy exceeds NLI accuracy in every model-language combination. Best score per column in bold; second-best underlined. implied criticism. For example, when a character arrives late and their aunt remarks in Punjabi, “VIP a gaye, hun program shuru kar dinde a” (“The VIP has arrived, now we can start the program”), all five models interpret the utterance as a genuine warm welcome. A native speaker instantly recog- nizes the pointed sarcasm about the lateness. Sim- ilarly, when a father in Malayalam exclaims “e veettil enikku shwaasam polum kittunnilla” (“I can- not even breathe in this house”), all models predict literal respiratory distress, missing the culturally transparent hyperbolic expression of frustration. 5.2 Model and Task comparisons Which model performs best? Gemma-4-31B is the strongest model overall in the benchmark (MCQ 0.86 NLI 0.43 COMET 0.81). Command- A-111B is the next strongest model, close behind Gemma on MCQ (0.74) and COMET (0.78), and edging ahead of it on NLI (0.46 vs. 0.43). Sarvam- 105B is comparable to Command-A on MCQ (0.77) but noticeably weaker on translation (COMET 0.66). Despite being smaller than Sarvam-105B and Command-A-111B and lacking dedicated In- dic pretraining, Gemma-4-31B is the most reliable model in our evaluation. Do all tasks have equal performance? MCQ scores are higher than NLI scores in all 20 of 20 model-language aggregates. This mirrors the dis- tinction between receptive and productive linguis- tic competence in psycholinguistics: recognizing a correct interpretation among options is cognitively easier than generating or inferring it without scaf- folding (Ruis et al., 2023). Do all languages have the same performance? The Dravidian languages, Tamil and Malayalam, lag behind on translation (though this gap is not consistent across MCQ and NLI) than the Indo- Aryan languages, Hindi and Punjabi. This is likely because of lower Dravidian language representa- tion in model pretraining data rather than any struc- tural difference in pragmatic difficulty. The trans- lation gap is most pronounced for Coherence and Deixis, and smallest for Implicature. 7 5.3 Error patterns In this section we look inside the errors, examin- ing whether models fail in predictable directions, whether failures are shared across model families, and whether NLI errors reveal systematic biases. Do models default to literal meaning? A de- sign choice in VakyArth was to construct MCQ distractors that reflect the literal, surface reading of each utterance. If models are failing because they cannot reason pragmatically, they should fall into these traps. We test this directly on the MCQ subset where a literal-meaning distractor is unam- biguously identifiable, and look at what models pick when they get the item wrong. Across all wrong-answer events, models chose the literal dis- tractor 59.1% of the time, nearly twice the 33.3% rate expected by chance(p < 0.001). This shows models are failing in a specific and predictable di- rection. For downstream applications, this means pragmatic failures are systematic and predictable rather than noisy, and can potentially be mitigated through targeted prompting or training. Are failures model-specific or shared across models? For each item, we counted how many of the four models failed. If failures were driven by model-specific weaknesses, we would expect them scattered across different items. Instead, models fail on the same items. On NLI, 24.7% of items are failed by all four models, rising to 41.7% for Social Pragmatics. This suggests that the items where all models fail together isolate cases where pragmatic reasoning breaks down regardless of the model’s overall capability. Do models show systematic label bias on NLI? We construct our NLI dataset more as contradiction labels (57.5%) because pragmatic reasoning often involves recognizing that a figurative or indirect utterance contradicts its literal reading. If models default to the literal reading, they should systemat- ically mislabel these items as Entailment. Figure 3 shows this is exactly what happens. When the gold label is Entailment, models get it right 78.8% of the time. But when the gold label is Contra- diction, models predict Entailment 53.8% of the time — more often than they predict Contradic- tion (23.1%). The bias also compounds our earlier finding on literal distractors: when scaffolding is removed, models do not merely guess randomly among labels — they systematically default to the reading where the utterance means what it literally EntailmentContradictionNeutral Model prediction Entailment Contradiction Neutral Gold label 78.8%4.0%17.1% 53.8%23.1%23.1% 47.1%2.9%50.0% 0 10 20 30 40 50 60 70 80 % Figure 3: NLI confusion matrix pooled across all four models. Rows are gold labels, columns are model pre- dictions. The highlighted cell marks the systematic bias toward Entailment (literal meaning in our case) when correct label is Contradiction. says. 5.4 Human Evaluation and COMET Correlation We collect human translation scores on a 1–5 scale covering adequacy, fluency, and pragmatic ade- quacy from four native raters, one per language. Given the cost of native-speaker annotation, we sample a subset of 104 items (Hindi 31, Pun- jabi 26, Tamil 23, Malayalam 24) out of 204 to- tal items, translated by four of the five evaluated models (Llama-3-8B, Gemma-4-31B, Qwen3-32B, Sarvam-105B), producing4× 104 = 416scored outputs across all four languages and five phe- nomena. We compare them against three auto- matic metrics: COMET (Rei et al., 2020) with the wmt22-comet-dacheckpoint, XCOMET (Guer- reiro et al., 2024), and MetricX (Juraska et al., 2023). As shown in Table 3, COMET achieves the highest overall correlation with human judgements (Pearsonr = 0.813, Spearmanρ = 0.860), outper- forming both XCOMET (ρ = 0.792) and MetricX (ρ = 0.788) despite being an older checkpoint. We hypothesize that XCOMET and MetricX, op- timized on formal translation benchmarks, gener- alize less well to the colloquial, code-mixed, and culturally embedded text in our benchmark. Corre- lation varies across languages and phenomena. All three metrics agree most with human scores on Co- herence and Social Pragmatics, where translation errors tend to be large and obvious. Agreement is weakest on Implicature and Deixis, where models produce fluent translations that sound correct but 8 miss the pragmatic meaning, the kind of subtle er- ror that automatic metrics struggle to detect. This finding has a practical implication: automatic eval- uation alone is insufficient for benchmarks where pragmatic faithfulness and surface fluency can di- verge, and human judgment remains necessary for phenomena like Implicature and Deixis. We show a few such examples in §A.3. COMETXCOMETMetricX r ρ r ρ r ρ All0.81 0.86 0.80 0.79 0.73 0.79 By language Hindi0.85 0.77 0.65 0.54 0.60 0.60 Punjabi0.77 0.80 0.78 0.77 0.74 0.76 Tamil0.82 0.80 0.72 0.61 0.62 0.62 Malayalam0.68 0.81 0.85 0.76 0.71 0.73 By phenomenon Implicature0.81 0.83 0.80 0.82 0.69 0.75 Deixis0.80 0.81 0.85 0.81 0.72 0.82 Speech Acts 0.76 0.87 0.77 0.79 0.74 0.80 Social Prag.0.86 0.87 0.85 0.86 0.77 0.83 Coherence0.87 0.88 0.85 0.82 0.78 0.82 Table 3: Pearson (r) and Spearman (ρ) correlation of three automatic metrics with human judgements across all four languages and five phenomena (n = 416). COMET achieves the highest overall correlation (ρ = 0.86) despite being an older checkpoint compared to MetricX. XCOMET is weaker on Hindi and Tamil. 5.5 Script Experiment We evaluate all MCQ items across all five phenom- ena, in both native script (Devanagari, Gurmukhi, Tamil, and Malayalam) and Latin transliteration, to study whether script choice affects pragmatic understanding. Pooled across all languages, phe- nomena, and models, native script yields a mod- est overall advantage (76.5% vs. 70.6% accuracy). However, the effect is neither universal nor uni- form in size: Punjabi (Qwen3-32B) and Malay- alam (Command-A) both perform slightly worse in native script, and the size of the advantage where it does appear varies widely, from negligible (Hindi, Command-A: no difference) to substantial (Malay- alam, Qwen3-32B: +23.0 points; Tamil, Command- A: +17.8 points). No single model is uniformly script-invariant or uniformly script-sensitive across languages, and no consistent language-level pattern emerges either. We leave a deeper investigation of this effect to future work. Since VakyArth targets everyday communication, we adopt Latin translit- eration as the standard script in our evaluation. Llama-3-8B Gemma-4-31B Qwen3-32B Command-A 0.0 0.2 0.4 0.6 0.8 1.0 MCQ Accuracy 0.80 0.90 0.83 0.92 0.86 0.95 0.85 0.92 Hindi LatinNative Llama-3-8B Gemma-4-31B Qwen3-32B Command-A 0.0 0.2 0.4 0.6 0.8 1.0 MCQ Accuracy 0.31 0.80 0.61 0.67 0.31 0.82 0.56 0.77 Punjabi LatinNative Llama-3-8B Gemma-4-31B Qwen3-32B Command-A 0.0 0.2 0.4 0.6 0.8 1.0 MCQ Accuracy 0.46 0.98 0.66 0.84 0.58 0.98 0.89 0.80 Malayalam LatinNative Llama-3-8B Gemma-4-31B Qwen3-32B Command-A 0.0 0.2 0.4 0.6 0.8 1.0 MCQ Accuracy 0.44 0.78 0.58 0.49 0.56 0.80 0.69 0.67 Tamil LatinNative Figure 4: Script experiment: native script vs. Latin transliteration MCQ accuracy for all four languages, pooled across all five phenomena. No single model is uniformly script-invariant or uniformly script-sensitive across languages 6 Conclusion We present VakyArth, the first diagnostic bench- mark for pragmatic competence in Indic languages, covering Hindi, Punjabi, Tamil, and Malayalam across five phenomena and three task types. Eval- uating five models ranging from 8B to 111B pa- rameters, we find consistent, phenomenon-specific failure patterns that existing benchmarks cannot de- tect. Gemma-4-31B is the strongest model overall despite lacking Indic specialization, and a persis- tent MCQ-NLI gap suggests that recognizing and reasoning about pragmatic meaning are distinct competencies. When models fail, they do so in a systematic and predictable direction defaulting to literal readings rather than guessing randomly among labels. Standard automatic metrics, while broadly reliable, fail to penalize fluent but prag- matically unfaithful outputs for implicature and deixis. We hope VakyArth serves as a foundation for building more pragmatically capable and cul- turally aware language technologies for Indic lan- guage users. 7 Limitations VakyArth has three main limitations. First, the script experiment covers only MCQ items; a fuller ablation and detailed investigation across all mod- els, tasks, and languages remains future work. Sec- ond, all items are single-turn utterances or short 9 passages and do not cover multi-turn pragmatics, which is an important dimension of real-world lan- guage use. Third, while our four languages repre- sent a diverse sample of Indic language families, hundreds of other Indic languages remain uncov- ered. Acknowledgments This work was partially funded by NSF Awards 2107524 and 2145479, as well as Good Systems, 2 a UT Austin Grand Challenge to develop responsible AI technologies. We thank Kelly Marchisio for their valuable feedback. We also thank Cohere for providing API credits to support this work. References Ruth Appel. 2026. India country brief: The anthropic economic index. John Langshaw Austin. 1975. How to do things with words. Harvard university press. Kent Bach. 2006. Speech acts and pragmatics. The Blackwell guide to the philosophy of language, pages 147–167. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Compu- tational Linguistics, 34(1):1–34. Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021. Is incoherence surprising? targeted evalua- tion of coherence prediction from language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4164–4173. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and 1 others. 2024. Culturalbench: a robust, diverse and challenging benchmark on mea- suring (the lack of) cultural knowledge of llms. Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no indic language behind: Building mono- lingual corpora, benchmark and models for indic languages. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12402–12426. Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins. 2 https://goodsystems.utexas.edu/ 2024. xcomet: Transparent machine translation eval- uation through fine-grained error detection. Transac- tions of the Association for Computational Linguis- tics, 12:979–995. J Jaavid, Raj Dabre, M Aswanth, Jay Gala, Than- may Jayakumar, Ratish Puduppully, and Anoop Kunchukuttan. 2024. Romansetu: Efficiently un- locking multilingual capabilities of large language models via romanization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15593– 15615. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language infer- ence models imppressive? learning implicature and presupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 8690–8705. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. Metricx-23: The google submission to the wmt 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul NC, Avik Bhattacharyya, Mitesh M Khapra, and Pratyush Kumar. 2020. Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages. In Findings of the association for compu- tational linguistics: EMNLP 2020, pages 4948–4961. Youngeun Koo, Jiwoo Lee, Dojun Park, Seohyun Park, and Sungeun Lee. 2025. Evaluating large language models on understanding korean indirect speech acts. arXiv preprint arXiv:2502.10995. Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, and Barbara Plank. 2025. Pragmatics in the era of large language models: A survey on datasets, evaluation, opportunities and chal- lenges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 8679–8696. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, San Diego, California. Association for Computational Linguis- tics. Tarek Naous and Wei Xu. 2025. On the origin of cul- tural biases in language models: From pre-training data to linguistic phenomena. In Proceedings of the 2025 Conference of the Nations of the Ameri- cas Chapter of the Association for Computational 10 Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6423–6443. Dojun Park, Jiwoo Lee, Hyeyun Jeong, Seohyun Park, and Sungeun Lee. 2024a. Pragmatic competence evaluation of large language models for the korean language. In Proceedings of the 38th Pacific Asia Conference on Language, Information and Computa- tion, pages 256–266. Dojun Park, Jiwoo Lee, Seohyun Park, Hyeyun Jeong, Youngeun Koo, Soonha Hwang, Seonwoo Park, and Sungeun Lee. 2024b. Multiprageval: Multilingual pragmatic evaluation of large language models. In Proceedings of the 2nd GenBench Workshop on Gen- eralisation (Benchmarking) in NLP, pages 96–119. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. In Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 2685–2702. Brian Roark, Lawrence Wolf-Sonkin, Christo Kirov, Sabrina J Mielke, Cibu Johny, Isin Demirsahin, and Keith Hall. 2020. Processing south asian languages written in the latin script: the dakshina dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2413–2423. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2023. The goldilocks of pragmatic understanding: Fine- tuning strategy matters for implicature resolution by llms. Advances in Neural Information Processing Systems, 36:20827–20905. Omar Shaikh, Caleb Ziems, William Held, Aryan Pari- ani, Fred Morstatter, and Diyi Yang. 2023. Modeling cross-cultural pragmatic inference with codenames duet. In Findings of the Association for Computa- tional Linguistics: ACL 2023, pages 6550–6569. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Di- nesh Tewari, and Partha Talukdar. 2024. Indicgen- bench: A multilingual benchmark to evaluate gen- eration capabilities of llms on indic languages. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 11047–11073. Settaluri Sravanthi, Meet Doshi, Pavan Tankala, Rudra Murthy, Raj Dabre, and Pushpak Bhattacharyya. 2024. Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities. In Find- ings of the Association for Computational Linguistics: ACL 2024, pages 12075–12097. Teun A Van Dijk and 1 others. 1997.The study of discourse. Discourse as structure and process, 1(34):703–52. Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen. 2025. Milu: A multi-task indic language understand- ing benchmark. In Proceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10076–10132. Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, and Sunayana Sitaram. 2024. Pariksha: A large-scale investigation of human-llm evaluator agreement on multilingual and multi-cultural data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7900–7932. Bonnie Webber. 1988. Discourse deixis: Reference to discourse segments. In 26th Annual Meeting of the Association for Computational Linguistics, pages 113–122. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45. Kefan Yu, Qingcheng Zeng, Weihao Xuan, Wanxin Li, Jingyi Wu, and Rob Voigt. 2026. The pragmatic mind of machines: Tracing the emergence of pragmatic competence in large language models. In Proceed- ings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 192–213. 11 A Appendix A.1 Prompt Templates We use 3-shot prompts for all models and tasks. Instructions and few-shot examples are written in the source language to avoid introducing English- language bias into the evaluation. The prompt struc- ture is identical across all four languages, only the instruction text and examples are adapted to each language by a native speaker. Below we show the templates for each task type using Hindi as a repre- sentative example. Translation Prompt Neeche diye gaye Hindi text ko English mein anuvad karo. Sirf translated text hi output mein dikhayein. Source: Unhone milkar kaam kiya. Translation: They worked together. Source: Bazaar mein bahut saamaan milta hai. Translation: A wide variety of goods is available in the market. Source: Usne apna vaada nibhaya aur samay par pahunch gaya. Translation: He kept his promise and arrived on time. Source: [source utterance] Translation: MCQ Prompt Is prashan ka sahi uttar kya hai? Jabab mein sirf option ka akshar (A/B/C...) aur us option ka text likho. Context: Priya ne kaha, “Aaj khana bahut achha bana hai.” Options: A. Priya ko khana pasand aaya B. Priya ne khana nahi khaya C. Priya ne khana banaya D. Priya ko bhook nahi thi Answer: A. Priya ko khana pasand aaya Context: Ramesh ne kaha, “Bahar bahut thand hai.” Lekin use bahar jaana tha. Options: A. Ramesh bahar nahi jayega B. Ramesh bahar jayega C. Ramesh ko thand nahi lagti D. Ramesh ghar mein rehna chahta hai Answer: B. Ramesh bahar jayega Context: Maa ne kaha, “Koi baat nahi, agli baar dhyan rakhna.” Options: A. Maa gusse mein hai B. Maa ne maafi maangi C. Maa ne tasalli di D. Maa ne saza di Answer: C. Maa ne tasalli di Context: [source utterance] Question: [question] Options: [A. option 1] [B. option 2] [C. option 3] [D. option 4] Answer: 12 NLI Prompt Neeche diye gaye do vaakyon ke beech sambandh batao. Jabab mein sirf ‘Entailment’, ‘Contradiction’, ya ‘Neutral’ likho. Premise: A: Khana taiyaar hai? B: Main abhi bazaar gaya tha. Hypothesis: Khana taiyaar hai. Label: Contradiction Premise: A: Kya tujhe pata hai kal chutti hai? B: Haan, main subah se plan kar raha hoon. Hypothesis: B kal kuch karna chahta hai. Label: Entailment Premise: A: Bahar baarish ho rahi hai. B: Theek hai. Hypothesis: B bahar jayega. Label: Neutral Premise: [source premise] Hypothesis: [hypothesis] Label: A.2 Annotation Guidelines for Translation Rating Human evaluation scores for translation outputs were collected from four native speaker annotators, one per language, each with graduate-level NLP training. Annotators rated each translation on a 1–5 scale assessing how well the model preserved the pragmatic intent of the source utterance including implied meaning, tone, cultural nuance, and illo- cutionary force rather than surface-level adequacy or fluency alone. Annotators were instructed to read the source utterance, its English gloss, and the model output, and assign a score based on the following rubric. Annotators were explicitly instructed not to ma- jorly penalise grammatical imperfections if the pragmatic meaning was preserved, and not to re- ward fluent outputs that missed the implied mean- ing. Each translation rated independently without access to scores from other models or annotators. ScoreCriterion 5Pragmatic intent fully preserved, including tone, implicature, and cultural nuance. 4Pragmatic intent mostly preserved; minor loss of nuance that does not affect overall meaning. 3Core pragmatic intent present but cultural or tonal nuance is lost. 2Pragmatic intent partially lost; translation feels overly literal. 1Pragmatic intent completely lost; translation is misleading or pragmatically vacuous. Table 4: Rubric used for human evaluation of translation outputs. Annotators assessed pragmatic fidelity rather than surface fluency. A.3 COMET–Human Score Divergence Examples We present four translation outputs where COMET assigns a high score but human raters assign 1 or 2 out of 5. These cases illustrate why automatic metrics are sometimes insufficient for evaluating pragmatic translation quality. COMET rewards lexical and structural similarity to the reference but cannot detect pragmatically inverted or temporally misanchored outputs. Example 1:Deictic inversion (Malayalam, Deixis) Source: “Njan ippozhum stationinte pinnile gate-il aanu. Ivide varenda; nee avide thanne nilkku, njan athu vannu cherkam.” Gloss: “I am still at the back gate of the station. Don’t come here; stay where you are, I’l come there.” Model output (Qwen3-32B): “I am still at the station. Come here; you stand there, I will come there.” COMET: 0.78Human: 1/5 The model inverts ivide varenda (don’t come here) to “Come here”, producing a navigation instruction that is the exact opposite of the source. COMET scores the output highly because most content words are present; a human immediately recognises the instruction is backwards. Example 2: Subject-object inversion in implica- ture (Punjabi, Implicature) Source: “Surinder di salah leh ke na baithna, uh tah kath da ullu hai.” Gloss: “Don’t take Surinder’s advice; he is a complete fool.” Model output (Sarvam-105B): “He made a fool of him- self by not listening to Surinder’s advice.” COMET: 0.76Human: 1/5 Kath da ullu is a common Punjabi meaning “a complete fool.” Sarvam mistranslates the subject of the insult — the output implies the speaker is foolish for ignoring Surinder, whereas the source calls Surinder the fool. The pragmatic meaning is exactly inverted. 13 Example 3: Temporal deixis chain failure (Pun- jabi, Deixis) Source: “Seema ne Mangalvaar nu mainu dassya ki Priya ne pichhle Somvaar nu unhu keha si, ‘pehla mera interview parso si...’ ” Gloss: “On Tuesday, Seema told me that Priya had told her last Monday, ‘My interview was originally on Wednesday [the day after tomorrow relative to that Mon- day]...’ ” Model output (Sarvam-105B): “My interview was the day after tomorrow...” COMET: 0.86Human: 2/5 The word parso means the day two steps from today— past or future depending on context. The model fails to anchor it to the correct speech time (last Monday), producing a future reference where a past reference is required. COMET scores the output 0.86 because the words are nearly identical to the reference. Example 4: Completed vs ongoing action deixis (Tamil, Deixis) Source: “Shanthi, unnoda friends ellaarum vanthutanga, veliya saaptutu irukkanga, neeyum poi senthum saap- pudu.” Gloss: “Shanthi, all your friends have come [already], they are eating outside, you also go and join them.” Model output (Qwen3-32B): “Shanthi, all your friends are coming, you are also going to join them.” COMET: 0.65Human: 1/5 Vanthutanga is a perfective form indicating completed action — the friends have already arrived. The model translates it as an ongoing future event, losing the ur- gency that the friends are already there waiting. The deictic anchor shifts from present to future. A.4 Error Analysis We present five representative failure cases, one per phenomenon. Each case illustrates a distinct failure mode that recurs across the benchmark. Example 1: Face-saving invitation read literally (Tamil, Implicature, MCQ) Context: “Veetukku vantha virunthaaligal purapidrapo kai nenachittu ponganu amma sonnanga.” Gloss: “As the guests who had come home were leaving, mother told them: wet your hands and go.” Question: What did mother tell the guests? A. Kai kaluva sonnanga (Told them to wash their hands) B. Inga saapida sonnanga (Invited them to eat here) C. Veetukku poi saapida sonnanga (Told them to go home and eat) D. Kulichutu poga sonnanga (Told them to bathe before leaving) ModelPredictionGold Llama-3.1-8BC✗B Gemma-4-31BC✗B Qwen3-32BC✗B Sarvam-105BA✗B Command-A✗B Kai nenachittu po (“wet your hands and go”) is a traditional Tamil farewell invitation to stay and eat — a face-saving formula that allows guests to decline gracefully. Llama, Gemma, and Qwen interpret it as telling guests to go home and eat (C); Sarvam reads it literally as a hygiene instruction (A). No model recovers the implied invitation to stay (B). Example 2: Referent confusion in family dia- logue (Tamil, Deixis, MCQ) Context: “Unakku thevayaana thuni mani ellam eduthu vechachanu amma unkitta kekka sonnanga. Ne eduthu- vechuttiyanu en thangachi kitta na ketten.” Gloss: “Amma asked me to check with you [unkitta] whether you had taken all the cloth pieces you needed. I asked my younger sister [thangachi] whether she had taken them.” Question: Who does the speaker refer to with ‘unakku’? A. AppaB. Amma C. ThangachiD. Akka ModelPredictionGold Llama-3.1-8B (Amma)✗C (Thangachi) Gemma-4-31BC✔C (Thangachi) Qwen3-32B (Amma)✗C (Thangachi) Sarvam-105BC✔C (Thangachi) Command-AB (Amma)✗C (Thangachi) The utterance involves three participants (amma, speaker, thangachi) across two turns. Llama and Qwen conflate amma and thangachi, failing to track the referential chain across the two embedded clauses. Example 3: Hyperbolic repetition read literally (Punjabi, Speech Acts, NLI) Premise: “Papa ne naraaz hoke Raj nu keha, ‘Main tenu 100 vari keha, ghar vich ball na kheliya kar.’ ” Gloss: “Papa angrily told Raj, ‘I have told you 100 times, don’t play ball inside the house.’ ” Hypothesis: Papa has already told Raj this same thing 99 times before. ModelPredictionGold Llama-3.1-8BEntailment✗Contradiction Gemma-4-31BContradiction✔Contradiction Qwen3-32BEntailment✗Contradiction Sarvam-105BNeutral✗Contradiction Command-AEntailment✗Contradiction “100 vari” is a hyperbolic speech act expressing frus- tration, not a literal count. The hypothesis that papa said it exactly 99 times before contradicts the pragmatic meaning. Llama and Qwen treat it as a factual claim and predict Entailment. 14 Example 4: Veiled criticism read as neutral ob- servation (Punjabi, Social Pragmatics, NLI) Premise: “Kal jado mainu meri maasi mili, unha ne keha, ‘Tu taan bada kamzor lag reha hai.’ ” Gloss: “Yesterday when I met my aunt, she said, ‘You look quite thin/weak.’ ” Hypothesis: The aunt was mocking the speaker’s health. ModelPredictionGold Llama-3.1-8BEntailment✗Contradiction Gemma-4-31BNeutral✗Contradiction Qwen3-32BNeutral✗Contradiction Sarvam-105BNeutral✗Contradiction Command-ANeutral✗Contradiction In Punjabi culture, an elder commenting on a younger person’s appearance (“you look thin”) is a conventional expression of concern and care, not mockery. The correct label is Contradiction — the aunt is being caring, not sarcastic. All four models miss this cultural norm. Example 5: Topically related but discourse- incoherent (Tamil, Coherence, MCQ) Context: “Tamilnadu, India la irukurathulaaye south la irukkura state. Idhu areavula 10-avathu periya state, janathila 6-avathu periyadhu. Tamilnadula la iruka makkal tamil pesuvaanga...” Gloss: “Tamil Nadu is a southern state of India. It is the 10th largest by area and 6th most populous. People in Tamil Nadu speak Tamil...” Question: What is the most appropriate next sentence? A.Tamilnadukku naraya sutrula payanigal varuvaanga (Many tourists visit) B.Chennai dhan thalainagaram (Chen- nai is the capital) C.Tamil classica solappadum Thirukku- ral... (literary reference) D.Tamil cinema industryah kollywoodnu solluvaanga (Kollywood) ModelPredictionGold Llama-3.1-8BA✗B Gemma-4-31B✔B Qwen3-32BC✗B Sarvam-105BD✗B Command-AB✔B The paragraph introduces factual information about Tamil Nadu systematically (area, population, language). The next logical fact is the capital city. Options A, C, D are all topically related to Tamil Nadu but break the structured informational progression. Three models pick topically plausible but discourse-incoherent continua- tions. 15