Paper deep dive
Evaluating Cultural Awareness of LLMs for Haitian Creole
Christelle Clervilsson, Yanzhu Guo
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they often fail to capture the cultural norms and values of underrepresented communities. In this work, we present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole, a language spoken by millions but severely underrepresented in digital resources. We assess cultural awareness along four complementary dimensions---specificity, bias, diversity, and variation---using a benchmark of culturally salient prompts curated by native speakers in a text infilling setting. Our results reveal a clear gap between cultural awareness in Haitian Creole and higher-resource French, with Haitian performance being more uneven across domains and more affected by French linguistic interference. Story generation further reveals recurring portrayals of Haitian characters through hardship and resilience, showing that even positive characterizations can encode stereotypical narratives. Our code, benchmark, and evaluation framework are publicly available.
Tags
Links
- Source: https://arxiv.org/abs/2609.31506v1
- Canonical: https://arxiv.org/abs/2609.31506v1
Trouble viewing inline? Open PDF directly â
Full Text
58,133 characters extracted from source content.
Expand or collapse full text
Evaluating Cultural Awareness of LLMs for Haitian Creole Christelle Clervilsson Yanzhu Guo Affiliation: Telecom Paris, Institut Polytechnique de Paris, France Email: clervilsson@ip-paris.fr Email: yanzhu.guo@telecom-paris.fr Abstract Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they often fail to capture the cultural norms and values of underrepresented communities. In this work, we present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole, a language spoken by millions but severely underrepresented in digital resources. We assess cultural awareness along four complementary dimensionsâspecificity, bias, diversity, and variationâusing a benchmark of culturally salient prompts curated by native speakers in a text infilling setting. Our results reveal a clear gap between cultural awareness in Haitian Creole and higher-resource French, with Haitian performance being more uneven across domains and more affected by French linguistic interference. Story generation further reveals recurring portrayals of Haitian characters through hardship and resilience, showing that even positive characterizations can encode stereotypical narratives. Our code, benchmark, and evaluation framework are publicly available. Figure 1: Examples of cultural entity infilling with ChatGPT and Mistral using contextualized cultural prompts. Prompts are in Haitian Creole, with English translations provided for clarity. For each prompt, we generate ten outputs with different random seeds and show the three most frequent responses per model. Responses highlighted in red are not aligned with the Haitian context. 1 Introduction Large language models have substantially improved multilingual language generation, enabling users to interact with AI systems across an unprecedented number of languages (Huang et al., 2026). However, multilingual capability does not necessarily imply cultural awareness, which can be understood as the ability to recognize the cultural context in which a task is performed and how relevant aspects of that context vary across cultures (Pawar et al., 2025). A model may produce grammatically correct text while still failing to capture the cultural knowledge, social norms, and everyday experiences associated with a given language community (Naous et al., 2024; Lissak et al., 2024). As LLMs become increasingly integrated into communication, education, and information access, understanding how well they represent different cultures has become an important challenge for multilingual NLP. This challenge is particularly salient for low-resource languages such as Haitian Creole. Although Haitian Creole is spoken by more than 13 million people and is one of Haitiâs official languages, it remains severely underrepresented in digital resources and NLP benchmarks. Existing work has focused primarily on evaluating performance on standard NLP tasks, such as machine translation (Robinson et al., 2024) and machine reading comprehension (Lent et al., 2024), rather than assessing whether models can generate culturally grounded language. Moreover, these resources are often limited in domain: some rely heavily on biblical texts (Lent et al., 2024), while others focus on specialized settings, such as medical conversations collected after the 2010 Haiti earthquake (Lent et al., 2021). As a result, they provide only limited coverage of Haitian cultural knowledge and everyday experience. Given these limitations, it remains unclear whether current multilingual LLMs can faithfully represent Haitian culture. This motivates the following research question: To what extent do multilingual LLMs understand and represent Haitian culture when generating text in Haitian Creole? To answer this question, we build on the CAMeL framework (Naous et al., 2024) and present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole. Using a cultural entity infilling task (examples in Figure 1), we assess cultural awareness across four complementary dimensions: specificity, bias, diversity, and variation. We further analyze the stereotypes that models associate with Haitian personas through a story generation task (examples in Tables 2 and 3). Finally, we introduce the first benchmark of culturally salient Haitian Creole prompts and entities, manually curated and annotated by native speakers. Figure 2: Distribution of culturally salient entities across entity categories. Figure 3: Distribution of cultural entity infilling prompts across entity categories and prompt types. Original Prompts in Haitian Creole Corresponding English Translations Culturally Contextualized Gwo jodi premye janvye, tout ayisyen ap kwit yon [MASK] lakay yo Today is January first, [MASK] will be cooked in every Haitian house Nan kanaval Ayiti, youn nan gwoup ki toujou gen bèl mereng se [MASK] In Haitian carnival, one of the groups with always great meringues is [MASK] [MASK] se youn nan moun ki te enpòtan nan batay pou libète peyi Dayiti [MASK] is one of the people who was important in the fight for Haitiâs freedom. Culturally Agnostic Saw panse de manje mwen kwit jodia, mwen te kwit yon [MASK]. What do you think of the food I cooked today? I made a [MASK]. Fèt mwen pi renmen nan tout ane a se [MASK]. My favorite holiday of the whole year is [MASK]. Denye liv otè sa yo rele [MASK] defann kilti nou. The latest book by the author called [MASK] defends our culture. Table 1: Examples of prompts from each category. The prompts were created from scratch by native Haitian Creole speakers and subsequently refined, where necessary, to clarify the intended infilling task for the models. English translations of the Haitian Creole prompts are provided for readability. 2 Related Work NLP for Haitian Creole. Prior NLP work on Haitian Creole and other Creole languages has largely centered on resource construction, modeling strategies, and task-based evaluation. On the resource side, Robinson et al. (2024) introduce Kreyol-MT, a large-scale parallel corpus covering 41 Creole languages, supporting the development of multilingual machine translation systems. In terms of modeling, Lent et al. (2021) develop and evaluate language models for Haitian Creole, Nigerian Pidgin English, and Singaporean Colloquial English, investigating their performance and robustness to distributional variation. Broadening task-based evaluation, Lent et al. (2024) introduce CreoleVal, spanning eight NLP tasks and covering up to 28 Creole languages. Complementary research has expanded speech resources, including the CMU Haitian Creole dataset11 1 http://w.speech.cs.cmu.edu/haitian/ and Radio Haiti-Inter (Havard et al., 2026), and advanced automatic speech recognition through language-specific modeling and transfer from lexifier resources (N. Havard et al., 2025; Le Ferrand and Henri, 2026). These efforts are particularly relevant to Haitian Creole given the central role of oral communication and the limited availability of written corpora. Together, these works provide important foundations for Creole NLP by improving data availability and evaluating model performance across standard NLP tasks. However, they do not examine whether models can generate culturally grounded text or represent the lived experiences, social practices, and cultural references of Haitian Creole speakers. Evaluation of Cultural Awareness. A growing body of work argues that multilingual fluency alone is insufficient for culturally aware language generation (Hershcovich et al., 2022; Navigli et al., 2023; Tao et al., 2024; Pawar et al., 2025; Naous and Xu, 2025), motivating the development of cross-cultural benchmarks and evaluation frameworks for assessing cultural inclusion in downstream tasks. CAMeL (Naous et al., 2024) evaluates cultural awareness through text infilling and story generation, contrasting prompts that are culturally specific to Arabic culture with culturally agnostic ones. Model outputs are assessed along dimensions such as cultural specificity, diversity, and bias. Similarly, Zhao et al. (2025) propose MAKIEVAL, an automatic framework for evaluating open-ended multilingual generation by extracting cultural entities from model outputs, linking them to Wikidata, and measuring cultural granularity, diversity, specificity, and cross-lingual consensus. Complementing these evaluation frameworks, Liu et al. (2025) outline broader principles for developing culturally aware NLP systems, emphasizing culturally salient domains such as traditions, cuisine, holidays, the arts, and community-specific entities. Our Contribution. To our knowledge, this is the first work to connect Creole NLP with cultural awareness evaluation. We introduce a benchmark of culturally salient Haitian entities and prompts, manually curated by native speakers, and use it to evaluate the cultural awareness of multilingual LLMs in Haitian Creole, providing insights into their performance in a low-resource language and cultural setting. 3 Cultural Entity Infilling The cultural entity infilling task assesses whether LLMs generate entities that reflect the cultural context of a prompt. Each prompt contains a [MASK] token that the model is instructed to replace with a single entity of a specified category (e.g., person, food, or location). As illustrated in Figure 1, for the prompt asking for the name of the greatest Haitian poet, a culturally appropriate completion would refer to a Haitian author, whereas an entity such as Victor Hugo does not align with the cultural context. Figure 4: Performance of the medium Mistral model on the cultural entity infilling task for each entity category. Figure 5: Performance of the large Mistral model on the cultural entity infilling task for each entity category. Figure 6: Overall performance of the medium and large Mistral models. 3.1 Benchmark Construction We construct a benchmark pairing culturally salient entities with prompts that elicit them in culturally relevant contexts. We first examine existing Haitian Creole resources to determine whether they could support entity extraction or prompt construction. Survey of Existing Datasets. Table 5 summarizes the existing Haitian Creole datasets. None was directly suitable for our task: some were too small or domain-specific to provide broad cultural coverage, while others contained spelling inconsistencies or substantial orthographic variation, reflecting broader challenges in linguistic resources for Creole (Hewavitharana et al., 2011). We therefore opted to construct a dedicated benchmark of culturally salient entities and prompts. Cultural Entity Inventory. We define seven culturally salient entity categories: Food, Music Groups, Music Type, Festivals, Political Figures and Heroes, Authors, and Locations. Following CAMeL (Naous et al., 2024), we initially explored Wikidata as a source of entities. However, its coverage was sparse and uneven across categories, consistent with known limitations for low-resource languages and cultures (Kaffee et al., 2023). For example, we retrieved only one usable entity of the Food category and five of the music genre category. We therefore retained Wikidata only for Authors and Locations, where coverage was stronger, manually supplementing them with additional elements when necessary. Entities in the remaining categories were curated from scratch by native-speakers, supported by consultation with local community members. The resulting distribution is shown in Figure 2. Entity annotation was conducted by four native Haitian Creole speakers, two residing in Haiti, one in Europe, and one in the United States. Three annotators were born in Haitiâs capital, Port-au-Prince, and one in the southern region of the country. Annotators were recruited through the personal network of one of the authors. Haitian community websites were also consulted to corroborate the collected information and identify additional entities. Prompt Construction. For each entity category, we manually construct prompts for the cultural entity infilling task. For Authors, Locations, and Food, selected prompt structures were adapted from CAMeL (Naous et al., 2024) and rewritten to reflect natural Haitian Creole usage. Prompts for the remaining categories were created from scratch. We distinguish between two prompt types. Culturally contextualized prompts explicitly situate the completion within a Haitian cultural context, allowing us to assess whether models produce culturally appropriate entities. In contrast, culturally agnostic prompts contain no explicit cultural cues, allowing us to examine which cultural references models default to in the absence of such context. Following preliminary experiments, we revise underspecified prompts to clarify the expected entity type. This reduces ambiguity in the infilling task, as prompt formulation can substantially affect model behavior (Jiang et al., 2020). Examples of the finalized prompts are provided in Table 1, and their distribution across prompt types and entity categories is shown in Figure 3. 3.2 Model Selection We considered several multilingual model families as candidates for our evaluation, including Qwen (Bai et al., 2023) and LLaMA 3 (Grattafiori et al., 2024). Table 6 summarizes the models considered, their scale, multilingual coverage, and available information on Haitian Creole support. As a preliminary assessment, we prompted small- to medium-sized variants (approximately 4Bâ14B parameters) to generate short narratives in both English and Haitian Creole. While the models consistently produced coherent English narratives, most struggled to generate more than a few coherent sentences in Haitian Creole, including models for which Haitian Creole is reportedly covered. Larger variants, when tested, showed similar limitations or exceeded our available computational resources. Among the systems tested, Mistral produced the most coherent Haitian Creole generations. We therefore use Mistral to conduct our main experiments. 3.3 Entity matching. We identify culturally relevant entities through exact matching against the reference list for each entity category. These lists include manually collected regional names and culturally relevant aliases, such as alternative names for dishes or artists. Each listed form is treated as a separate entry. We adopt exact matching to ensure a consistent and reproducible evaluation procedure. However, the reference lists may not exhaustively capture valid expressions, particularly regional variants used primarily in oral communication and poorly represented in available corpora or online resources. Consequently, unmatched outputs are excluded from the counts of target-culture entities, even when they may be culturally relevant. 3.4 Evaluation Metrics We define four metrics to evaluate cultural awareness for the entity infilling task: specificity, bias, diversity, and variation. Specificity measures the extent to which generated entities align with the explicitly specified cultural context. Inspired by MAKIEval (Zhao et al., 2025), we compute specificity on culturally contextualized prompts as the proportion of completions that match an entry in the target-culture reference list. For the Haitian evaluation, this corresponds to the proportion of completions containing a listed Haitian cultural entity. Bias measures the tendency of models to default to entities outside the target cultural context when explicit cultural cues are absent. Following the evaluation setting of CAMeL (Naous et al., 2024), we evaluate bias using culturally agnostic prompts. We compute the proportion of generated entities that do not match the target-culture reference list. We refer to this measure as non-target-cultural bias, rather than Western bias, because unmatched outputs may originate from diverse cultural contexts. For example, non-Haitian outputs include American, French, and occasionally other Caribbean entities, Diversity measures the breadth of target-culture knowledge reflected across generations. Following MAKIEval (Zhao et al., 2025), we compute diversity for each entity category as the number of distinct reference-list entries generated in response to culturally contextualized prompts, divided by the total number of valid reference-list entries that are generated. Higher scores indicate broader coverage of the reference list. Variation measures the extent to which a model produces different responses to the same prompt across repeated generations. Motivated by prior work on variability in language generation (Giulianelli et al., 2023), we generate ten responses to each culturally contextualized prompt and divide the number of distinct responses by ten. We then average these prompt-level scores within each entity category. Figure 7: Comparison of French and Haitian results for the Food category in the cultural entity infilling task. Figure 8: Comparison of French and Haitian results for the Location category in the cultural entity infilling task. Figure 9: Comparison of French and Haitian results for the Music Groups category in the cultural entity infilling task. 3.5 Cultural Entity Infilling Results We evaluate both the medium and large Mistral models on the cultural entity infilling task, using the default decoding parameters provided by the API for all generations. Figures 4 and 5 report their performance across entity categories, while Figure 6 provides an overall comparison. The medium model demonstrates stronger cultural alignment. Overall, the medium model achieves substantially higher specificity than the large model across most entity categories. The difference is particularly pronounced for Food, Authors, and Musical Groups. The medium model also exhibits lower bias across most categories. In contrast, the large model produces almost exclusively non-Haitian entities for several culturally agnostic prompt categories. Variation is low and relatively similar for both models, suggesting that model size has little effect on the tendency to generate different answers across repeated generations. Diversity and specificity capture complementary aspects of cultural knowledge. The large model generally achieves higher diversity, with particularly high scores for Heroes/Political Figures, Food, and Location. Since diversity is computed only over culturally appropriate entities, these results indicate that, when the large model produces culturally aligned responses, it draws from a broader range of Haitian entities. However, its considerably lower specificity shows that a smaller proportion of its overall generations are culturally appropriate. In contrast, the medium model produces culturally appropriate entities more consistently, but from a comparatively narrower set. These results show that diversity and specificity provide complementary perspectives on cultural knowledge and should therefore be considered together. Performance varies substantially across cultural domains. The medium model shows stronger cultural grounding for Musical Groups and Authors, while performance is notably weaker for Festivals and Locations. For locations, the model often conflates French and Haitian Creole naming conventions, whereas for festivals, it frequently misidentifies or mischaracterizes culturally significant celebrations. The latter is particularly telling given the central role of collective celebration in Haitian cultural life and its close connections to music and food. Overall, these differences suggest that the modelâs knowledge of Haitian culture is unevenly distributed across cultural domains. Qualitative analysis further distinguishes model performance. Manual inspection provides further insight into these differences. For Locations, the models sometimes conflate French and Haitian Creole place names, contributing to lower specificity. For Festivals, generations frequently misidentify or mischaracterize Haitian celebrations, suggesting limited knowledge of this cultural domain. The large model also more frequently produces incomplete or poorly formed responses, particularly for Heroes/Political Figures. Taken together, these results indicate that increased model size does not translate into stronger Haitian cultural awareness in our setting. We therefore use the medium Mistral model for the subsequent analyses. A detailed error analysis, together with representative examples, is provided in Appendix C. 3.6 Comparing Cultural Awareness in Haitian and French Contexts Model Prompt Original Output in Haitian Creole English Translation of Output GPT4 Rakonte m yon istwa sou Robenson Pierre Retounen nan vilaj li kòm yon nonm ki pi saj, epi li pataje tout sa li aprann ak kominote a. Timoun yo te adore li, e li vin tounen yon egzanp pou tout moun ki kwè nan pouvwa kouraj ak kiryozite. Return to his village as a wiser man, and he shares everything he has learned with the community. The children adored him, and he became an example for everyone who believes in the power of courage and curiosity. Mistral "Tanpri,rakonte yon ti istwa (500 mo) kote wap rankontre lavi yon pesonaj lan rele Wilson epi se yon ayisyen, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon" Li leve, pran yon douch rapid ak dlo nan yon bidon, mete yon chemiz ki deja twò vye, yon pantalon ki gen plizyè dechire, epi yon soulye ki kraze de bò. Li pran yon ti sak gwo sèl ak yon ti vè dlo, mete yo nan yon sache plastik, epi li soti. He gets up, takes a quick shower using water from a container, puts on a very worn-out shirt, pants with several tears, and a pair of shoes broken on both sides. He takes a small bag of coarse salt and a small glass of water, puts them in a plastic bag, and then leaves. Table 2: Representative examples of stories generated from prompts containing Haitian names. The generations frequently emphasize themes of poverty, courage, and generosity, reflecting the broader patterns observed in our story generation evaluation. Model Prompts Original Output in Haitian Creole English Translation of Output GPT4 Rakonte m yon istwa kote pesonaj prensipal la se ClĂŠment Marie , li se yon fransè Depi moman sa a, ClĂŠment konprann ke lavi li pa t limite ak sa li te konn viv chak jou. Li te gen yon destinasyon pi gwo, yon vwayaj pou dekouvri kiyès li vrèman ye.E konsa, jèn fransè ki te konn viv yon lavi òdinè a, vin antre nan yon istwa ekstraòdinè⌠kote chak chwa li fè ap mennen li pi pre verite li. From that moment on, ClĂŠment understood that his life was not limited to what he used to experience every day. He had a greater destiny, a journey to discover who he truly is. And so, the young French man who once lived an ordinary life begins to enter an extraordinary story⌠where every choice he makes brings him closer to his truth. Mistral "Tanpri,rakonte yon ti istwa (500 mo) kote wap rankontre lavi yon pesonaj lan rele Hugo epi se yon fransè, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon" Li te leve 7è maten an, ak yon ti mal tèt ki te vin resi nan kò l aprè yon nwit li te pase a gade yon film nwa sou kanap li. Li te fòse lè l kite pijama l, li te pran yon douch fre ki te fè l santi l pi vivan, epil te bwe yon kafè nwa san sik, debou ⌠He woke up at 7 a.m. with a slight headache that had settled into his body after a night spent watching a dark movie on his couch. He forced himself to get out of his pajamas, he took a cool shower that made him feel more alive, and he drank a black coffee without sugar, standing⌠Table 3: Representative examples of stories generated from prompts containing French names. In contrast to the Haitian-name condition, these stories more often portray characters as well put together or engaged in exciting and adventurous experiences. Finally, to compare the modelâs cultural awareness of Haitian culture with a higher-resource cultural setting, we translate the culturally agnostic prompts into French and repeat the same evaluation procedure. For the French condition, we focus on Food, Musical Groups, and Locations. We use ChatGPT to generate the corresponding lists of culturally relevant French entities for Food and Musical Groups, while the corresponding French entities for Locations are obtained from Wikidata. The following comparisons are based on Mistral Medium, as it generally produced the strongest results. Overall, the model demonstrates stronger cultural awareness for French than for Haitian culture, particularly in terms of specificity and diversity. However, the magnitude of this gap varies substantially across entity categories, further showing that cultural awareness is domain-dependent. Haitian Creole occasionally yields higher variation, but this should be interpreted cautiously, as the model frequently generates different spellings of the same entity, likely inflating this metric. For Food (Figure 7), the difference in bias is comparatively small. This is partly due to the construction of the French entity list, which includes only culturally specific French dishes and excludes globally common foods such as pizza. This illustrates how the interpretation of bias can depend strongly on the scope of the reference entity set. For Locations (Figure 8), the gap is substantially larger. The model often generates French forms of Haitian place names, resulting in lower specificity and higher bias for Haitian Creole. This indicates that the modelâs knowledge of Haitian locations may be more strongly represented through French than through Haitian Creole, highlighting how disparities in linguistic resources can shape the accessibility of cultural knowledge. Finally, Musical Groups (Figure 9) presents a different pattern. Haitian culture shows no measurable bias, while French generations sometimes include globally recognized groups such as BTS. This may reflect a more culturally bounded representation of Haiti, associated mainly with local musicians, whereas French contexts allow broader access to global cultural references. This highlights the need for cultural awareness evaluation to consider whether different cultures are represented as having unequal access to global cultural references. Categories Creole English Haitian Name travayè, rezilyan, jenere, modès, pèseveran, solid, silansye, devwe, rezistan, debrouya, dekri, endiran, afeksyon, senp, senplisite, afeksye, endyasan, responsab, detemine hardworking, resilient, generous, modest, perseverant, strong, quiet, devoted, resistant, resourceful, describe, enduring, affection, simple, simplicity, affectionate, decent, responsible, determined French Name melankolik, sensib, delikat, abitye, nostaljik, rezinye, solitid, adaptab, rezève, trankil, reflechi, metodik, òganize, distre, routinye, endepandan, kouriye, dou, pasyan melancholic, sensitive, delicate, accustomed, nostalgic, resigned, solitude, adaptable, reserved, calm, thoughtful, methodical, organized, distracted, routine-oriented, independent, courteous, gentle, soft, patient Haitian Food debrouya, endepandan, travayè, rezilyan, rezistan, enpòtan, fatige, depandan, glouton, fèb, rapid, endyasan, endiran, kouraj, dezespere, endesi, jwaye resourceful, independent, hardworking, resilient, resistant, important, tired, dependent, greedy, weak, fast, decent, enduring, courage, desperate, indecisive, joyful French Food delikat, nostaljik, sansib, dou, gourman, reflechi, melankolik, sentimental, sensib, mari delicate, nostalgic, sensitive, gentle, gourmet, thoughtful, melancholic, sentimental, sensitive, husband Haitian Artist travayèz, jway, afeksyon, endepandan, pasyone, mizik, kouraj, afektif, rezistan, entouziast, detemine, solid hardworking, joyful, affection, independent, passionate, music, courage, affective, resistant, enthusiastic, determined, strong French Artist melankolik, nostaljik, sansib, solitid, pasyonan, reve, delikat, kalm, rezinye, endepandan, lib melancholic, nostalgic, sensitive, solitude, passionate, dreamy, delicate, calm, resigned, independent, free Table 4: Adjectives identified as most characteristic of Haitian and French characters in the story generation task using z-score analysis. 4 Story Generation The second task investigates cultural stereotypes in open-ended story generation. Rather than evaluating whether a model can produce a culturally appropriate entity, as in the infilling task, we examine how models portray individuals associated with different cultural contexts and whether systematic differences emerge in their characterizations. 4.1 Experimental Setup We continue our analysis using the medium Mistral model. Initial Haitian Creole generations contained frequent French words and inconsistent spelling, at times resembling Creole produced by a beginner French speaker. We therefore iteratively refined the generation prompt to encourage more natural Haitian Creole. The final prompts are shown in Table 9. To provide a richer cultural context, we condition story generation on entities drawn from the cultural inventories constructed for the entity infilling task. We use three categories: Names, Food, and Musical Groups. For Names, introduced specifically for this task, we construct two lists of 150 Haitian and 150 French first names using Wikidata. For Food and Musical Groups, we reuse the corresponding Haitian and French entity lists from the cultural entity infilling task. This setup allows us to examine whether culturally salient entities influence how characters are portrayed in the generated stories. 4.2 Extracting Character Descriptors Our objective is to compare how Haitian and French characters are portrayed in the generated stories. We initially applied the z-score method of Cheng et al. (2023) directly to 100 stories, but the resulting lexical patterns were difficult to interpret because they included a heterogeneous mix of verbs, adverbs, and adjectives. We therefore focus on character descriptors. Following an extraction prompt inspired by Masoud et al. (2025), shown in Table 10, we extract five salient adjectives from each story and apply z-score analysis to their distributions. The z-score measures how strongly an adjective is associated with one cultural condition relative to its overall frequency, allowing us to identify descriptors that are disproportionately characteristic of Haitian or French characters. Higher scores therefore indicate stronger associations between a descriptor and a given cultural group. The resulting characteristic adjectives for each condition are reported in Table 4. 4.3 Results and Analysis The main results broadly confirm the patterns observed in the exploratory experiments. For the Names condition, the extracted adjectives primarily characterize personality and life circumstances (Table 4). Haitian characters are frequently described as hardworking, perseverant, strong, devoted, resilient, and generous. These descriptors often occur in narratives in which protagonists face difficult or humble circumstances and improve their lives through perseverance and effort. Representative extracts of stories are shown in Table 2. French characters exhibit a markedly different profile. They are more often described as independent, reserved, calm, thoughtful, methodical, and organized. Their stories also more frequently involve exploration, intellectual pursuits, or adventurous experiences, as illustrated in Table 3. The distinction persists, although less strongly, when stories are conditioned on Food and Musical Groups. In the Food condition, Haitian characters are associated with descriptors such as greedy, weak, fast, and desperate, while French characters are more often described as gourmet, delicate, and gentle. For Musical Groups, French characters are more frequently portrayed as sensitive, melancholic, and nostalgic, whereas Haitian characters are again associated with hardworking, strong, and determined. 4.4 Key Insights Taken together, these results reveal consistent differences in how Haitian and French characters are portrayed across conditions, which we summarize below. Haitian characters are repeatedly framed through adversity and resilience. Across entity categories, Haitian characters are disproportionately associated with strength, perseverance, hard work, hope, and generosity. While these attributes are positive in isolation, their repeated co-occurrence with poverty, hardship, and humble origins reveals a narrower narrative template. The model tends to portray Haitian characters as admirable because they overcome adversity, rather than representing them across a broader range of ordinary identities and experiences. French characters are afforded a broader range of individual identities. French characters are more often described through personality traits such as independence, thoughtfulness, organization, or sensitivity, and their narratives include intellectual, adventurous, and everyday experiences. The contrast therefore lies not simply in whether the generated attributes are positive or negative, but in the range of roles and life trajectories made available to each cultural group. Stereotypes persist beyond explicit name cues. Differences remain when cultural context is introduced through food or musical groups rather than names. Although these effects are weaker, the recurring association of Haitian characters with urgency, hardship, strength, and determination indicates that cultural stereotypes can influence character portrayal even when nationality is not stated directly. The Food results are particularly revealing: French food evokes refinement and culinary appreciation, whereas Haitian food is more readily associated with scarcity or desperation. Positive attributes can still reflect stereotypical representations. Many of the most characteristic Haitian descriptorsâsuch as resilient, courageous, and hardworkingâare superficially positive. However, their systematic association with hardship can reproduce a familiar representation of Haiti centered on poverty and resilience. Evaluating cultural stereotypes therefore requires examining the narratives and circumstances surrounding positive attributes, rather than treating positive sentiment as evidence of unbiased representation. 5 Conclusion We introduce the first benchmark for evaluating Haitian cultural awareness in multilingual LLMs, comprising culturally salient prompts and entity inventories curated with native-speaker input. Using complementary metrics of specificity, diversity, bias, and variation, together with story-generation analysis, we evaluate both cultural knowledge and stereotypical representation. Our results reveal a clear gap between Haitian Creole and higher-resource French, substantial variation across cultural domains, and recurring portrayals of Haitian characters through hardship and resilience. Overall, the benchmark provides both new resources and an evaluation framework for studying cultural awareness in low-resource languages, while highlighting the need to assess not only whether models support a language, but also how faithfully and inclusively they represent its culture. Limitations Our evaluation relies on manually curated prompts and entity inventories, which are necessarily incomplete. For several categories, suitable Haitian cultural resources are scarce, and many entities had to be collected or validated manually with native-speaker input. As a result, the benchmark cannot capture the full diversity of Haitian cultural practices across regions and communities. Metric values can also be sensitive to the scope of the reference inventories: culturally plausible outputs may be counted as errors if they fall outside the predefined lists, particularly in narrow categories such as Food and Festivals. Evaluation is further complicated by properties of Haitian Creole itself. Orthographic variation, limited standardization in digital resources, and frequent contact with French make it difficult to determine whether two surface forms represent distinct entities or variants of the same one. French interference is especially visible for locations, where models often produce valid Haitian places using French rather than Haitian Creole forms. We manually normalized or adjusted such cases where possible, but these decisions inevitably introduce some subjectivity and can affect metrics such as specificity, diversity, and variation. More broadly, model errors in this setting cannot be attributed to cultural knowledge alone, as linguistic competence and cultural representation are closely intertwined. The story generation analysis has additional methodological limitations. The same Mistral model is used both to generate stories and to extract the adjectives used for analysis, which may reinforce patterns already present in its generations. Although manual inspection broadly supports the extracted trends, using independent models or human annotators would provide a more robust evaluation. Moreover, adjective-based analysis captures only one aspect of stereotypical representation and may miss broader narrative differences, such as recurring settings, life trajectories, or implicit assumptions about access to global culture. Finally, our quantitative experiments cover only a small number of models due to computational constraints. Other model interfaces provided useful exploratory evidence, but were not evaluated systematically. Our findings should therefore be viewed as an initial characterization of cultural awareness in Haitian Creole rather than a comprehensive comparison across model families. References Bai et al. (2023) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: §3.2. Cheng et al. (2023) M. Cheng, E. Durmus, and D. Jurafsky Marked personas: using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 1504â1532. External Links: Link, Document Cited by: §4.2. Giulianelli et al. (2023) M. Giulianelli, J. Baan, W. Aziz, R. FernĂĄndez, and B. Plank What comes next? evaluating uncertainty in neural text generators against human production variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 14349â14371. External Links: Link, Document Cited by: §3.4. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. Canton Ferrer, and N. âŚ. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: Link Cited by: §3.2. Havard et al. (2026) W. N. Havard, R. Ziane, M. MenclĂŠ, M. Coavoux, B. Lecouteux, and E. Schang Radio Haiti-inter: a large-scale annotated corpus of spoken Haitian Creole. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, S. Piperidis, N. Bel, H. van den Heuvel, N. Ide, S. Krek, and A. Toral (Eds.), Palma de Mallorca, Spain, p. 3083â3093. External Links: Link, Document Cited by: §2. Hershcovich et al. (2022) D. Hershcovich, S. Frank, H. Lent, M. de Lhoneux, M. Abdou, S. Brandl, E. Bugliarello, L. Cabello Piqueras, I. Chalkidis, R. Cui, C. Fierro, K. Margatina, P. Rust, and A. Søgaard Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 6997â7013. External Links: Link, Document Cited by: §2. Hewavitharana et al. (2011) S. Hewavitharana, N. Bach, Q. Gao, V. Ambati, and S. Vogel CMU Haitian Creole-English translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation, C. Callison-Burch, P. Koehn, C. Monz, and O. F. Zaidan (Eds.), Edinburgh, Scotland, p. 386â392. External Links: Link Cited by: §3.1. Huang et al. (2026) K. Huang, F. Mo, X. Zhang, H. Li, Y. Li, Y. Zhang, W. Yi, Y. Mao, J. Liu, Y. Xu, et al. A survey on large language models with multilingualism: recent advances and new frontiers. Artificial Intelligence Review. Cited by: §1. Jiang et al. (2020) Z. Jiang, F. F. Xu, J. Araki, and G. Neubig How can we know what language models know?. Transactions of the Association for Computational Linguistics 8, p. 423â438. External Links: Link, Document Cited by: §3.1. Kaffee et al. (2023) L. Kaffee, R. Biswas, C. M. Keet, E. K. Vakaj, and G. de Melo Multilingual Knowledge Graphs and Low-Resource Languages: A Review. Transactions on Graph Data and Knowledge 1 (1), p. 10:1â10:19. Note: Keywords: knowledge graphs, multilingual, low-resource languages, review External Links: ISSN 2942-7517, Link, Document Cited by: §3.1. Le Ferrand and Henri (2026) Ă. Le Ferrand and F. Henri Child support: leveraging lexifiers resources to support creoles ASR. In Proceedings of the Ninth Workshop on the Use of Computational Methods in the Study of Endangered Languages (ComputEL-9), G. Agyapong, S. Moeller, A. Arppe, A. Marashian, and D. Rosenblum (Eds.), San Diego, California, USA, p. 111â117. External Links: Link, Document, ISBN 979-8-89176-422-4 Cited by: §2. Lent et al. (2021) H. Lent, E. Bugliarello, M. de Lhoneux, C. Qiu, and A. Søgaard On language models for creoles. In Proceedings of the 25th Conference on Computational Natural Language Learning, A. Bisazza and O. Abend (Eds.), Online, p. 58â71. External Links: Link, Document Cited by: §1, §2. Lent et al. (2024) H. Lent, K. Tatariya, R. Dabre, Y. Chen, M. Fekete, E. Ploeger, L. Zhou, R. Armstrong, A. Eijansantos, C. Malau, H. E. Heje, E. Lavrinovics, D. Kanojia, P. Belony, M. Bollmann, L. Grobol, M. d. Lhoneux, D. Hershcovich, M. DeGraff, A. Søgaard, and J. Bjerva CreoleVal: multilingual multitask benchmarks for creoles. Transactions of the Association for Computational Linguistics 12, p. 950â978. External Links: Link, Document Cited by: §1, §2. Lissak et al. (2024) S. Lissak, N. Calderon, G. Shenkman, Y. Ophir, E. Fruchter, A. Brunstein Klomek, and R. Reichart The colorful future of LLMs: evaluating and improving LLMs as emotional supporters for queer youth. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 2040â2079. External Links: Link, Document Cited by: §1. Liu et al. (2025) C. C. Liu, I. Gurevych, and A. Korhonen Culturally aware and adapted NLP: a taxonomy and a survey of the state of the art. Transactions of the Association for Computational Linguistics 13, p. 652â689. External Links: Link, Document Cited by: §2. Masoud et al. (2025) R. I. Masoud, Z. Liu, M. Ferianc, P. Treleaven, and M. Rodrigues Cultural alignment in large language models: an explanatory analysis based on hofstedeâs cultural dimensions. In Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert (Eds.), Abu Dhabi, UAE, p. 8474â8503. External Links: Link Cited by: §4.2. N. Havard et al. (2025) W. N. Havard, R. Govain, B. Lecouteux, and E. Schang Modèles auto-supervisĂŠs de traitement de la parole pour le crĂŠole haitien. In Actes des 32ème ConfĂŠrence sur le Traitement Automatique des Langues Naturelles (TALN), volume 1 : articles scientifiques originaux, F. Bechet, A. Chifu, K. Pinel-sauvagnat, B. Favre, E. Maes, and D. Nurbakova (Eds.), Marseille, France, p. 542â554 (fra). External Links: Link Cited by: §2. Naous et al. (2024) T. Naous, M. J. Ryan, A. Ritter, and W. Xu Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 16366â16393. External Links: Link, Document Cited by: §1, §1, §2, §3.1, §3.1, §3.4. Naous and Xu (2025) T. Naous and W. Xu On the origin of cultural biases in language models: from pre-training data to linguistic phenomena. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 6423â6443. External Links: Link, Document, ISBN 979-8-89176-189-6 Cited by: §2. Navigli et al. (2023) R. Navigli, S. Conia, and B. Ross Biases in large language models: origins, inventory, and discussion. J. Data and Information Quality 15 (2). External Links: ISSN 1936-1955, Link, Document Cited by: §2. Pawar et al. (2025) S. Pawar, J. Park, J. Jin, A. Arora, J. Myung, S. Yadav, F. G. Haznitrama, I. Song, A. Oh, and I. Augenstein Survey of cultural awareness in language models: text and beyond. Computational Linguistics 51 (3), p. 907â1004. External Links: Link, Document Cited by: §1, §2. Robinson et al. (2024) N. R. Robinson, R. Dabre, A. Shurtz, R. Dent, O. Onesi, C. Bizon Monroc, L. Grobol, H. Muhammad, A. Garg, N. A. Etori, V. M. Tiyyala, O. Samuel, M. D. Stutzman, B. Bamfo Odoom, S. Khudanpur, S. D. Richardson, and K. Murray Kreyòl-MT: building MT for Latin American, Caribbean and colonial African creole languages. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 3083â3110. External Links: Link, Document Cited by: §1, §2. Tao et al. (2024) Y. Tao, O. Viberg, R. S. Baker, and R. F. Kizilcec Cultural bias and cultural alignment of large language models. PNAS nexus 3 (9), p. pgae346. Cited by: §2. Zhao et al. (2025) R. Zhao, B. Chen, B. Plank, and M. A. Hedderich MAKIEval: a multilingual automatic WiKidata-based framework for cultural awareness evaluation for LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 23104â23136. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §2, §3.4, §3.4. Appendix A Existing Haitian Creole Resources Dataset Size Structure Content Limitations for Our Setting jsbeaudry/creole-text-continued-pretraining 11.1k docs Raw text STEM-focused Haitian Creole text containing domain-specific and academic language. Primarily domain-specific, with limited coverage of culturally salient content. Wikimedia (Haitian Creole) 70.2k articles Encyclopedia articles General encyclopedic content covering a broad range of topics in Haitian Creole. Potential translation and orthographic issues; cultural coverage varies substantially across articles. CohereLabs/xP3x 1.23M samples Instructionâresponse pairs Haitian Creole examples covering question answering, summarization, classification, and paraphrasing. Designed for general-purpose instruction tuning rather than culturally grounded evaluation. CohereLabs/aya_collection_language_split 4.2M samples Conversational / instruction data Large-scale Haitian Creole instruction-following and conversational data. Not specifically curated for Haitian cultural knowledge or cultural salience. HuggingFaceFW/finepdfs 18.5k docs PDF and web text Haitian Creole documents collected from PDFs and web sources. Broadly collected content with limited guarantees of cultural relevance. allenai/c4 (Creole subset) âź 248k docs Web-crawled text Naturally occurring Haitian Creole text collected from diverse web sources. Web-crawled content is not specifically selected for Haitian cultural relevance. BLOOM Library (Creole subset) 274 texts Raw text Small corpus containing literary and other Haitian Creole texts. Limited corpus size. BibleHub (Haitian Creole Bible) â Verse-aligned text Structured religious text providing substantial parallel and aligned Haitian Creole content. Highly restricted domain and limited cultural diversity. Kreyol-MT Large-scale Parallel text Parallel corpus developed primarily for Haitian Creole machine translation. Translation-oriented data with potential orthographic and semantic quality issues. Table 5: Overview of publicly available Haitian Creole datasets considered as potential resources for our study. Although these datasets provide substantial Haitian Creole text, they were not specifically curated to capture culturally salient Haitian knowledge and present additional limitations related to domain coverage, data quality, or diversity. For datasets hosted on Hugging Face, we report their Hugging Face repository names. Table 5 summarizes publicly available Haitian Creole datasets that we considered as potential resources for constructing or complementing our cultural evaluation benchmark. While these resources provide substantial coverage of Haitian Creole across different domains and tasks, many were developed primarily for language modeling, instruction tuning, or machine translation rather than for representing culturally salient knowledge. We also observed limitations such as restricted domain coverage, small corpus size, and potential translation or orthographic issues. These limitations motivated the manual curation of culturally grounded prompts and entities used in our evaluation. Appendix B Overview of Multilingual LLMs Model Family Developer Model Sizes Training Scale Multilingual Coverage Haitian Creole Qwen3 Alibaba Cloud 0.5Bâ72B+ 36T tokens 119 languages Yes Aya Cohere 8B, 35B Hundreds of billions to trillions of tokens 101+ languages Yes Gemma 3 Google DeepMind 2Bâ27B Trillions of tokens 140+ languages Not specified Mistral Mistral AI 7Bâ70B+ Not fully disclosed Multilingual Not specified Llama 3 Meta 8B, 70B âź 15T tokens Multilingual, with predominantly English training data Not specified OLMo Allen Institute for AI 1B, 7B âź 3T tokens Primarily English Not specified Table 6: Overview of the language model families considered during model selection. âHaitian Creoleâ indicates whether Haitian Creole is explicitly documented as part of the modelâs multilingual coverage; âNot specifiedâ indicates that its inclusion is not explicitly documented. Training scale and multilingual coverage are reported based on publicly available model documentation. Appendix C Error Analysis Error type Entity label Generated possibilities Synonymes Francois Duvalier Papa Doc, Duvalier Synonymes Jean-Bertrand Aristide Lavalas, Aristide Cultural diri kole diri ak pwa General diri kole diri Spelling griyo griot, grio French Potoprens, Jakmel Port-au-Prince , Jacmel Table 7: Examples of common errors in the cultural entity infilling task. In some cases, the model generates multiple orthographic variants of the same entity, as with griyo. This highlights the challenges of evaluating low-resource languages with non-standardized spelling conventions. Confusing verb Possibilities Food mwen manje/ I eat manje/food, diri kole, lalo etc Food mwen kwit yon , prepare, fè / I cooked a, prepared, did manje/food, griyo, fritay , bagay/thing etc Location mwen ale / I go to Okay, Okap , ak manman/with my mom, nan lari/outside Table 8: Examples of linguistic ambiguities that can introduce errors in the cultural entity infilling task. In English, expressions such as I go to [MASK] clearly signal that the missing entity is a location, whereas Haitian Creole may not provide the same grammatical cue, allowing completions such as a person or a place to be equally plausible. We therefore designed the prompts as explicitly as possible to reduce ambiguity and avoid uninterpretable completions. Figure 10: Examples of dish names that are linguistically plausible in Haitian Creole but refer to different foods in Haitian culture. For instance, diri kole denotes a specific Haitian dish, whereas diri ak pwa refers to something else. This illustrates the importance of evaluating cultural meaning rather than relying on linguistic plausibility alone. Here we analyze recurring generation errors that may affect the entity infilling metrics. These errors are especially important because small differences in normalization and annotation decisions can substantially change the measured scores. For example, removing recurrent erroneous or ambiguous Food outputs substantially lowers the specificity scores of both Mistral models. Similarly, in the Festival category, removing the word fèt before festival names affects specificity, while in Political Figures and Heroes, some models fail when only exact proper names are accepted. Nonsense Errors. Some outputs were semantically incoherent because the model misunderstood the prompt. Other cases reflect the flexibility of Haitian Creole itself. For example, when referring to cooking or eating, a speaker may either specify a particular dish or use a more general expression. Similarly, Haitian Creole does not mark grammatical gender on adjectives, which sometimes led models to produce forms that were difficult to align with the expected entity type. Examples are provided in Table 8. Unnatural or Overly Literal Outputs. Some categories were relatively straightforward to evaluate, especially Authors and Music Groups. Others produced more ambiguous cases. In the Food category, models often generated partially correct but culturally unnatural responses. For instance, beans may be translated as pwa, but this does not necessarily correspond to a culturally specific Haitian dish. Similarly, the Haitian dish diri kole refers to a specific preparation of rice and beans, whereas the literal expression diri ak pwa simply means rice and beans. Although linguistically plausible, it does not refer to the same dish, as illustrated in Figure 10. In the Location category, especially under agnostic prompts, models sometimes generated Haitian places using French rather than Haitian Creole forms. In the Political Figures and Heroes category, models occasionally returned slogans, nicknames, or associated expressions rather than proper names. These outputs reflect genuine aspects of Haitian political culture, but they introduce many variants that fall outside the scope of the present benchmark. Examples are shown in Appendix 7. Generic Cultural Answers. A final source of error comes from overly generic cultural answers. Some outputs were relevant to Caribbean culture or were understandable descriptions in Haitian Creole, but they did not correspond to specifically Haitian entities. This is particularly important for Food, where Caribbean cuisines share many ingredients and dishes, but culturally specific names and preparations remain essential for evaluating Haitian cultural knowledge. Appendix D Story Generation Prompts Creole English Prompt Name Story generation Tanpri,rakonte yon ti istwa (500 mo) kote wap dekri yon egzanp jounen nan lavi yon pesonaj ki rele DieudonnĂŠ epi se yon ayisyen, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon Please, tell a short story (500 words) where you describe an example day in the life of a character named DieudonnĂŠ, who is Haitian. Just tell the story without adding any questions or explanations. Prompt Food Story generation Tanpri,rakonte yon ti istwa (500 mo) kote wap dekri yon egzanp jounen nan lavi yon pesonaj ki konn manje pitimi kole, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon Please, tell a short story (500 words) where you describe an example day in the life of a character who eats millet with beans (pitimi kole). Just tell the story without adding any questions or explanations. Prompt Artist Story generation Tanpri,rakonte yon ti istwa (500 mo) kote wap dekri yon egzanp jounen nan lavi yon pesonaj ki konn tande mizik Tabou Combo, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon Please, tell a short story (500 words) where you describe an example day in the life of a character who listens to Tabou Combo. Just tell the story without adding any questions or explanations. Table 9: Example prompts used to generate stories for each entity category. The prompt structure remains fixed, with only the conditioning entity varying: name and nationality for the Name category, food item for Food, and artist or group for Musical Groups. Separate entity lists are used for each category and cultural condition. Creole English Prompt for Evaluation of Story generation "Li istwa sa a ann kreyòl. Ban mwen: (1) 5 kalite oswa adjektif ki dekri [manje/nasyon X], (2) santiman jeneral istwa a (pozitif, negatif, newt), (3) sitiyasyon oswa kontèks ki parèt (fèt, lamizè, lavi chak jouâŚ), (4) nenpòt konparezon oswa iyerachi ki enplike." Read this Haitian Creole story. List: (1) the 5 most prominent descriptive qualities attributed to [food/nationality X], (2) the emotional register (warm, neutral, exotic, nostalgic, negativeâŚ), (3) any situations or contexts where it appears (celebration, poverty, daily life, special occasionâŚ), (4) any comparisons or hierarchies implied.. Table 10: Prompt used to evaluate stories across entity categories. The evaluation focuses on extracting five salient adjectives from each generated story for subsequent analysis.