Paper deep dive
Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation
Sadia Kamal, Arefa Patwary, Anthony Marchiafava, Atriya Sen, Sagnik Ray Choudhury
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/8/2026, 2:09:03 AM
Summary
This paper investigates whether prompt robustness in large language models differs between objective questions with fixed answers and subjective questions that elicit opinions or values. Evaluating four instruction-tuned model families across six datasets using multiple prompt perturbations, the study finds that subjective questions are significantly less stable than objective ones. Option-order changes cause the largest consistency drops, and robustness is shown to be a joint function of model, dataset, dataset type, and perturbation category, warning against treating single-prompt survey responses as stable measures of model beliefs.
Entities (15)
Relation Signals (8)
Option order → causeslargestinstabilityin → Subjective questions
confidence 95% · The Type-I versus Type-II gap is largest under option order. For option-order perturbations, Type-I consistency is 0.485, while Type-II consistency is only 0.328.
Gemma → evaluates → MMLU
confidence 95% · We evaluate four instruction-tuned model families on three objective datasets (MMLU, ARC, and CulturalBench)...
Objective questions → exhibitshigherconsistencythan → Subjective questions
confidence 95% · Averaged over all perturbations, Type-I objective datasets have a mean consistency of 0.849, while Type-II subjective datasets have a mean consistency of 0.787.
Political Compass Test → istype → Subjective questions
confidence 95% · The Type-II group contains subjective or opinion-based statements: the Political Compass Test, ValueBench, and World Values Survey items...
Answer consistency → measures → Prompt robustness
confidence 95% · Our primary evaluation metric is answer consistency. It measures whether a model gives the same answer across prompt variants.
Paraphrase → tests → Semantic framing
confidence 90% · The first source is semantic framing. Paraphrase, lexical substitution, spelling noise and logical equivalence ask whether two prompts with the same intended task produce the same answer.
Option order → tests → Answer presentation
confidence 90% · The third source is answer presentation. Label substitution and option shuffling test whether the model follows the content of the options or instead follows labels and positions.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Survey-style evaluations of large language models often treat a prompted response as a measure of a model's values or beliefs. This assumption is particularly fragile when responses are read as evidence of political values, social attitudes, or beliefs. We ask whether prompt robustness differs between objective questions with fixed answers and subjective questions that ask for opinions or values. We evaluate four instruction-tuned model families on three objective datasets (MMLU, ARC, and CulturalBench) and three subjective datasets (Political Compass Test, ValueBench, and World Values Survey). For each question/statement, we apply multiple types of prompt changes, such as variations in wording, framing, and format, and measure whether the model gives the same answer across variants. Using a binomial generalized estimating equation, we find significant effects of model, dataset, prompt category, and their interactions. The dataset type effect is also significant, and the interaction between dataset type and prompt category is large. These results show that prompt robustness depends on the question type, the prompt change, and the model.
Tags
Links
- Source: https://arxiv.org/abs/2607.05554v1
- Canonical: https://arxiv.org/abs/2607.05554v1
Trouble viewing inline? Open PDF directly →
Full Text
29,529 characters extracted from source content.
Expand or collapse full text
Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation Sadia Kamal † , Arefa Patwary ‡ , Anthony Marchiafava † , Atriya Sen † , Sagnik Ray Choudhury ‡ † Oklahoma State University, ‡ University of North Texas sadia.kamal,anmarch,atriya.sen@okstate.edu, arefapatwary@my.unt.edu,sagnik.raychoudhury@unt.edu Abstract Survey-style evaluations of large language mod- els often treat a prompted response as a mea- sure of a model’s values or beliefs. This as- sumption is particularly fragile when responses are read as evidence of political values, social attitudes, or beliefs. We ask whether prompt robustness differs between objective questions with fixed answers and subjective questions that ask for opinions or values. We evaluate four instruction-tuned model families on three objective datasets (MMLU, ARC, and Cultural- Bench) and three subjective datasets (Political Compass Test, ValueBench, and World Values Survey). For each question/statement, we ap- ply multiple types of prompt changes, such as variations in wording, framing, and format, and measure whether the model gives the same answer across variants. Using a binomial gener- alized estimating equation, we find significant effects of model, dataset, prompt category, and their interactions. The dataset type effect is also significant, and the interaction between dataset type and prompt category is large. These re- sults show that prompt robustness depends on the question type, the prompt change, and the model. 1 Introduction Most large language model evaluations rest on a fragile assumption that one prompt gives a stable and meaningful measure of model behavior. Prior work shows this is often not the case, small changes in prompt format, wording, and answer presenta- tion can change model behavior (Sclar et al., 2024; Chatterjee et al., 2024; Ismithdeen et al., 2025). If a model changes its answer when the task meaning stays the same, then the evaluation is measuring both task ability and prompt sensitivity. This is a critical concern for survey-style tests. Recent work uses political and value surveys to infer what LLMs “believe” or which human groups they resemble. But survey responses from LLMs are unstable under ordering, labeling, forced- choice wording, and framing changes (Dominguez- Olmedo et al., 2024; Röttger et al., 2024; Rupprecht et al., 2025). A model’s answer can be shaped by the way the prompt asks the question. This raises a question about whether prompt sen- sitivity behaves the same way for different types of questions. We study this problem by comparing two kinds of questions. Type-I questions consist of objective, multiple-choice items with a single cor- rect answer. Type-I questions are subjective survey items that ask for opinions, values, or degrees of agreement. When a prompt is reworded without altering its underlying meaning, a robust model is expected to produce the same answer across both versions. The same applies for objective questions as well. For subjective questions, however, mod- els may interpret minor changes in wording or re- sponse options as cues about how to respond. As a result, the model’s answer can shift even when the survey item itself has not changed in substance. We ask three research questions: RQ1: Does response consistency differ between objective and subjective question types? RQ2: Do objective and subjective questions show different sensitivity patterns across prompt categories? RQ3: Is prompt robustness a model-level property, or does it also depend on dataset and prompt category? Our study makes three contributions. First, we curate a broad perturbation set from prior work on prompt sensitivity, multiple-choice formatting, sur- vey response bias, and value measurement. Second, we give a unified robustness test across objective and subjective datasets. Third, we use clustered statistical tests to show that prompt sensitivity de- pends on model, dataset, dataset type, and perturba- tion category. We find that subjective questions are less stable under prompt variation than objective questions. The gap is not uniform across perturba- tion categories: option-order changes produce the largest consistency drop, while lexical and logically arXiv:2607.05554v1 [cs.CL] 6 Jul 2026 equivalent perturbations are comparatively stable. Statistical analysis shows that robustness depends jointly on model, dataset, dataset type, and prompt category. Our findings show that LLM prompt sen- sitivity varies systematically with question type and should be evaluated across perturbation families rather than through a single prompt. 2 Related Work Prompt sensitivity in LLM evaluation. Sclar et al.(Sclar et al., 2024) show that prompt for- matting can produce large accuracy differences and argue that evaluation should report variation across plausible prompt formats, not only one prompt . POSIX(Chatterjee et al., 2024) measures prompt sensitivity through likelihood change un- der intent-preserving prompt variants . Promptcep- tion(Ismithdeen et al., 2025) builds a broad pertur- bation framework for multiple-choice multimodal evaluation, including phrasing, formatting, posi- tion, and option presentation . These works show that prompt design is part of the evaluation, not a neutral wrapper around it. LLM survey responses.Survey-style evaluation has been used to study model values, politics, and social attitudes. But recent studies warn that such responses can be strongly affected by answer order- ing, labels, forced-choice formats, and paraphrases (Dominguez-Olmedo et al., 2024; Röttger et al., 2024; Wright et al., 2024; Rupprecht et al., 2025). This makes political and value scores difficult to interpret as stable model traits. Our work adds a direct comparison with objective multiple-choice datasets, which helps separate general prompt noise from instability that is specific to subjective ques- tions. 3 Experimental Design 3.1 Datasets and Models We use six datasets categorized into two types. The Type-I group contains objective multiple-choice questions: MMLU (Hendrycks et al., 2021), ARC (Clark et al., 2018), and CulturalBench-Easy (Chiu et al., 2024). The Type-I group contains subjective or opinion-based statements: the Political Compass Test, ValueBench (Ren et al., 2024), and World Val- ues Survey items (Haerpfer et al., 2022). We use CulturalBench-Easy as a Type-I dataset because it tests cultural knowledge through multiple-choice questions with fixed answer keys, rather than ask- ing for subjective preferences or beliefs. We evalu- ate four instruction-tuned model families: Gemma, Llama, Mistral, and Qwen. 3.2 Perturbation Curation We conducted an extensive literature search across three lines of work: prompt robustness, multiple- choice evaluation, and LLM survey measurement. From this search, we selected perturbations only if they met the following criteria. First, the pertur- bation had to preserve the intended task for both factual and survey-style items. Second, it had to test a known source of instability, such as wording, option labels, option order, or formatting. Table 1 gives the final taxonomy. 3.3 Prompt Generation and Answer Normalization For each item, we create prompt variants that pre- serve the intended task. All prompts use the same broad instruction: the model must select exactly one labeled option. For subjective datasets, the labels map to agreement or value options. For ob- jective datasets, the labels map to answer choices. We keep this common format so that the Type-I and Type-I groups differ mainly in question content, not in answer extraction. All runs use determin- istic decoding with temperature 0 so that answer changes come from prompt variation rather than sampling noise. The perturbation set is designed to separate three failure sources. The first source is semantic fram- ing. Paraphrase, lexical substitution, spelling noise and logical equivalence ask whether two prompts with the same intended task produce the same an- swer. The second source is surface form. Format variation test whether the model is distracted by small changes that should not alter the task. The third source is answer presentation. Label substi- tution and option shuffling test whether the model follows the content of the options or instead follows labels and positions. This separation matters be- cause a single consistency score can hide different causes of instability. We use the same perturbation logic across objec- tive and subjective datasets, but the interpretation differs. On objective datasets, the answer key gives an external standard. If the model changes from one option to another under a meaning-preserving change, this is a robustness failure. On subjective datasets, there is no single correct option. There- fore, we do not call one answer right and another CategoryOperationWhat it testsLiterature ParaphraseRewrite the instruction with the same task intent. Sensitivity to semantically similar wording. Prompt robustness and survey paraphrase ef- fects (Sclar et al., 2024; Chatterjee et al., 2024; Röttger et al., 2024). Spelling noiseAdd small spelling mistakes while keeping the item readable. Robustness to surface noise and user typos. Promptperturbation and input-noise evalua- tion (Ismithdeen et al., 2025). Lexical substitutionReplace selected words with near synonyms. Dependence on word choice when the intended meaning is stable. Intent-preserving prompt variation (Chat- terjee et al., 2024). Logical equivalentUse truth-preserving instruction forms. Whether logically equivalent task de- scriptions keep the same answer. Controlledprompt rewriting (Sclar et al., 2024; Chatterjee et al., 2024). Label substitutionReplace A, B, C, D with num- bers, Roman numerals, rare Uni- code symbols, or regular symbols. Dependence on answer labels rather than option content. Multiple-choice option sensitivity and survey responseartifacts (Ismithdeenetal., 2025; Rupprecht et al., 2025). Format variation Change separators, spacing, cas- ing, wrappers, joiners, and field layout. Sensitivity to prompt layout rather than semantic content. Prompt formatting sen- sitivity (Sclar et al., 2024; Ismithdeen et al., 2025). Option shufflingReverse, rotate, or swap option or- der while tracking the correct an- swer when one exists. Position bias, recency bias, and order effects. Survey and multiple- choice ordering effects (Dominguez-Olmedo et al., 2024; Rupprecht et al., 2025). Table 1: Prompt perturbation taxonomy. The final categories were selected from prior work and then adapted so the same robustness design could be used for objective and subjective datasets. wrong. We instead ask whether the model keeps its own answer stable when the survey item is pre- sented in a controlled new form. We normalize outputs before computing con- sistency. For objective datasets, raw outputs are mapped to canonical A, B, C, or D labels. For sub- jective datasets, agreement labels are also mapped to canonical letters. For substitution runs, symbols are mapped back to their original A/B/C/D mean- ings. This prevents a numeric answer such as “1” and a letter answer such as “A” from being counted as inconsistent when they refer to the same option. 3.4 Consistency Metric and Statistical Model Our primary evaluation metric is answer consis- tency. It measures whether a model gives the same answer across prompt variants. Throughout this section, we will refer a single question from a dataset as an item. For each item, model, and prompt category, we compute a consistency ratio: C i = max y n i (y) N i ,(1) wheren i (y)is the number of times answeryap- pears for itemi, andN i is the number of valid prompt variants for that item. A value of 1 means all variants gave the same answer. Lower values mean the prompt category changed the answer. We fit binomial generalized estimating equation (GEE) models (Liang and Zeger, 1986). GEE is useful here because each item appears many times under related prompt variants, so rows are not inde- pendent. The response is whether a prompt variant matches the majority answer for that item. We test main effects for model, dataset, dataset type, and prompt category, plus interactions between dataset or dataset type and prompt category, and between model and prompt category. We report Waldχ 2 tests. 4 Results 4.1 Subjective Questions Are More Affected Than Objective Questions Figure 1 shows that, across the four models, subjec- tive datasets usually have lower consistency than objective datasets under most perturbation cate- Paraphrase Logical equivalent Lexical substitution Spelling mistake Label substitution Format Option order Perturbation category ARC MMLU CulturalBench PCT ValueBench WVS Dataset 0.890.930.880.880.760.840.53 0.860.760.790.750.600.680.65 0.900.920.850.870.730.860.51 0.970.920.970.930.750.840.26 0.910.940.890.850.700.800.27 0.930.960.890.850.710.760.27 Mean consistency by dataset and perturbation: Gemma 0.0 0.2 0.4 0.6 0.8 1.0 Mean consistency (a) Gemma Paraphrase Logical equivalent Lexical substitution Spelling mistake Label substitution Format Option order Perturbation category ARC MMLU CulturalBench PCT ValueBench WVS Dataset 0.990.990.990.990.930.970.43 0.930.980.950.940.740.820.47 0.970.980.980.970.860.960.43 0.850.890.910.910.760.630.24 0.830.900.900.890.720.670.22 0.850.940.900.900.740.670.27 Mean consistency by dataset and perturbation: Llama 0.0 0.2 0.4 0.6 0.8 1.0 Mean consistency (b) Llama Paraphrase Logical equivalent Lexical substitution Spelling mistake Label substitution Format Option order Perturbation category ARC MMLU CulturalBench PCT ValueBench WVS Dataset 0.980.960.990.990.950.920.45 0.900.900.930.950.680.800.51 0.970.980.990.990.890.910.43 0.870.950.960.960.760.690.24 0.890.930.950.950.790.690.27 0.870.910.930.930.820.690.26 Mean consistency by dataset and perturbation: Mistral 0.0 0.2 0.4 0.6 0.8 1.0 Mean consistency (c) Mistral Paraphrase Logical equivalent Lexical substitution Spelling mistake Label substitution Format Option order Perturbation category ARC MMLU CulturalBench PCT ValueBench WVS Dataset 0.990.990.990.990.980.940.43 0.930.930.970.960.840.850.56 0.940.980.960.970.950.930.42 0.900.940.960.940.780.800.55 0.890.930.960.950.860.850.53 0.880.920.970.940.840.820.54 Mean consistency by dataset and perturbation: Qwen 0.0 0.2 0.4 0.6 0.8 1.0 Mean consistency (d) Qwen Figure 1: Dataset-level mean consistency across perturbation categories for all four models. gories. Averaged over all perturbations, Type-I ob- jective datasets have a mean consistency of 0.849, while Type-I subjective datasets have a mean con- sistency of 0.787. Equivalently, instability rises from 0.151 for objective datasets to 0.213 for sub- jective datasets. Subjective tasks are consistently less stable than objective ones across all four models, which an- swers RQ1. The gap is persistent across Gemma, Llama, Mistral, and Qwen models, although the magnitude differs by model. It reflects a systematic difference between question types. Table 2 summarizes the marginal consistency scores by dataset type and perturbation category. The Type-I versus Type-I gap is largest under op- tion order. For option-order perturbations, Type-I consistency is 0.485, while Type-I consistency is only 0.328. This means subjective datasets add 15.7 percentage points of extra instability beyond the already large option-order effect in objective datasets. The next largest gaps appear under format variation and label substitution. The smallest gaps appear under lexical substitution and logical equiv- alence, where both dataset types remain relatively stable. 4.2 Option Order Is the Strongest Source of Instability Having established that subjective questions are less robust overall, we next ask which perturba- tion category drives the largest instability. Across all four models and all six datasets, option or- der perturbation category affects the most. Av- eraged over models and datasets, option-order perturbations reduce mean consistency to 0.407. This is much lower than label substitution, 0.799, and format variation, 0.808. Semantic and near- semantic perturbations are more stable: paraphrase reaches 0.913, spelling noise reaches 0.927, logical equivalence reaches 0.935, and lexical substitution reaches 0.936. This pattern is visible in each model in Figure 1. For Gemma, Llama, and Mistral, option order pro- duces the lowest consistency values for the subjec- tive datasets, particularly PCT, ValueBench, and WVS. Qwen is more robust overall, but option or- der still produces its clearest drop compared with semantic perturbations. Thus, the main finding is not that models are equally fragile to all prompt changes. Instead, answer-presentation perturba- tions, particularly option order, create the largest instability. This addresses RQ2, objective and subjective PerturbationType-IType-IIGap Option order0.4850.3280.157 Format0.8730.7430.130 Label substitution0.8270.7710.057 Paraphrase0.9380.8880.050 Spelling noise0.9380.9170.021 Logical equivalent0.9430.9280.015 Lexical substitution0.9380.9330.005 Table 2: Mean consistency by dataset type and perturba- tion category. Gap is computed as Type-I consistency minus Type-I consistency, so larger values indicate greater subjective-task instability. Effectχ 2 dfp Model173.1243 2.69× 10 −37 Dataset463.8315 5.10× 10 −98 Prompt category523.7196 6.52× 10 −110 Dataset× prompt587.88530 9.52× 10 −105 Model× prompt616.74818 2.47× 10 −119 Table 3: Global Waldχ 2 tests with dataset as a six-level factor. questions do not merely differ in overall robust- ness. They differ in the kinds of prompt changes that affect them most. The subjective-task gap is largest when the prompt changes how answers are presented, rather than when it only changes lexical wording. 4.3 Dataset and Prompt Category Interact Strongly Table 3 demonstrates the global GEE model with dataset as a six-level factor. Model, dataset, and prompt category are all significant predictors of consistency. More importantly, the dataset by prompt category interaction is large and significant, χ 2 = 587.885,p = 9.52× 10 −105 . This confirms the pattern in Figure 1: perturbation categories do not affect all datasets in the same way. The model by prompt category interaction is also significant, χ 2 = 616.748,p = 2.47× 10 −119 , showing that models differ in which perturbations they are most sensitive to. 4.4 Dataset Type Explains a Systematic Robustness Gap Table 4 divides datasets into Type-I and Type-I groups. Dataset type remains highly significant, χ 2 = 256.026,p = 1.26× 10 −57 . The interac- tion between dataset type and prompt category is also significant,χ 2 = 175.435,p = 3.16× 10 −35 . This interaction shows that subjective datasets are Effectχ 2 dfp Model171.8983 4.95× 10 −37 Dataset type256.0261 1.26× 10 −57 Prompt category709.0996 6.64× 10 −150 Type× prompt175.4356 3.16× 10 −35 Model× prompt617.20718 1.98× 10 −119 Table 4: Global Waldχ 2 tests with dataset type as Type- I versus Type-I. Datasetχ 2 dfp ARC191.6256 1.15× 10 −38 MMLU96.9506 1.09× 10 −18 CulturalBench279.8486 1.69× 10 −57 PCT757.4366 2.41× 10 −160 ValueBench993.6816 2.08× 10 −211 WVS1146.8976 1.49× 10 −244 Table 5: Prompt-category effects from within-dataset GEE models. not merely lower in consistency on average. They are differently affected by different perturbation categories. The results underscores that single-prompt sur- vey measurements are notably fragile. For objec- tive multiple-choice questions, there is an external target answer that can constrain the model. For subjective survey items, small changes in word- ing, formatting, labels, or option order can shift the response without any external gold answer. This makes a single response difficult to interpret as a stable value or belief. 4.5 All Datasets Show Prompt Effects Table 5 reports the prompt-category effect from within-dataset GEE models. Prompt category is significant for all six datasets. The largest effects appear in the subjective datasets: WVS,χ 2 = 1146.897, ValueBench,χ 2 = 993.681, and PCT, χ 2 = 757.436. Objective datasets also show signif- icant effects, but the magnitudes are smaller: ARC, χ 2 = 191.625, MMLU,χ 2 = 96.950, and Cultur- alBench, χ 2 = 279.848. This result rules out a narrow explanation in which prompt sensitivity is only a survey problem. Prompt variation affects all datasets. However, the effect is more consequential for subjective datasets because a shifted answer can change the inferred political position, social value, or belief attributed to the model. 4.6 Robustness Varies Across Models, Datasets, and Perturbations The model-wise heatmaps in Figure 1 show that robustness is not a single model trait. A model can be stable under paraphrase or lexical substitution but unstable under option order. The same model can also behave differently across objective and subjective datasets. For example, Qwen is gener- ally more stable than the other models under option order, but it still shows a visible drop compared with semantic perturbations. Llama and Mistral are highly stable on several objective cells but much less stable on subjective option-order cells. The model by prompt category interactions in Tables 3 and 4 confirm this observation statisti- cally. Appendix Table 6 reports the full model by dataset prompt-effect table. Every model-dataset bin shows a significant prompt-category effect. Therefore, robustness should be reported as a rela- tion among model, dataset, and perturbation cate- gory, not as a single global score. This matters for model comparison. A model that appears robust on MMLU paraphrases may not be robust on WVS option order or PCT label substitution. A single average robustness score can therefore hide important failures. For survey-style evaluations, the most important warning is that a single prompted answer should not be treated as a stable measurement of model values or beliefs. 5 Discussion These results have a direct implication for LLM sur- vey studies. A single answer to a political or value question should not be treated as a stable model belief unless it survives prompt variation. This does not mean survey-style evaluation is useless. It means that survey scores need robustness checks. If two meaning-preserving prompts produce differ- ent answers, the result cannot be attributed to the model alone. The objective-subjective comparison sheds light on why this happens. Objective ques- tions have fixed answers, so a change in answer usually signals task failure or prompt sensitivity. Subjective questions are different. A model may use wording, label choice, or option order as evi- dence about how it should respond. In that case, prompt variation is not only noise. It is also a way to reveal how fragile the measured stance is. Researchers using survey-style evaluations should therefore report consistency across prompt variants, not only a final accuracy or survey score. Seman- tic perturbations should be separated from surface perturbations, since they reveal different failure modes. Interactions with dataset type should also be reported. Beyond measuring consistency, the perturbation taxonomy can identify where instability comes from. Failures under spelling or format changes suggest brittle instruction following. Failures un- der label substitution or option shuffling suggest answer-position bias or label-token bias. Failures under paraphrase or logical equivalence are the most consequential for survey interpretation, be- cause the measured stance changes even when the item meaning is intended to stay fixed. These fail- ure types should not be collapsed into a single error category. 6 Limitations This study uses deterministic decoding and forced- choice answers. This helps isolate prompt effects, but it does not cover open-ended survey responses. The model names are treated at the family level, so future work should report exact checkpoint versions and parameter sizes. Finally, subjective datasets are not interchangeable. PCT, ValueBench, and WVS differ in wording, scale design, and so- cial meaning. 7 Conclusion We compared prompt robustness across objective and subjective questions using six datasets and four model families. Consistency depends strongly on model, dataset, prompt category, and dataset type. Subjective survey tasks show substantially larger prompt-category effects than objective tasks, and these effects vary by both the type of prompt change and the model being tested. Characteriz- ing model robustness therefore requires systematic variation across prompt forms. For value and belief evaluation, robustness to prompt variation should be a standard part of the evaluation design. References Anwoy Chatterjee, H. S. V. N. S. Kowndinya Renduch- intala, Sumit Bhatia, and Tanmoy Chakraborty. 2024. POSIX: A prompt sensitivity index for large language models. In Findings of the Association for Compu- tational Linguistics: EMNLP 2024, pages 14550– 14565. Association for Computational Linguistics. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024. Culturalbench: a robust, diverse and challenging benchmark on measuring the lack of cultural knowledge of LLMs. Preprint, arXiv:2410.02677. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Ricardo Dominguez-Olmedo, Moritz Hardt, and Celes- tine Mendler-Dünner. 2024. Questioning the survey responses of large language models. In Advances in Neural Information Processing Systems, volume 37. Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Puranen. 2022. World values survey wave 7 (2017-2022) cross-national data-set. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In International Conference on Learning Representations. Mohamed Insaf Ismithdeen, Muhammad Uzair Khat- tak, and Salman Khan. 2025. Promptception: How sensitive are large multimodal models to prompts? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23950–23985. As- sociation for Computational Linguistics. Kung-Yee Liang and Scott L. Zeger. 1986. Longitu- dinal data analysis using generalized linear models. Biometrika, 73(1):13–22. Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. Valuebench: Towards com- prehensively evaluating value orientations and un- derstanding of large language models. Preprint, arXiv:2406.04214. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful eval- uations for values and opinions in large language models. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics, pages 15295–15311. Association for Computational Linguistics. Jens Rupprecht, Georg Ahnert, and Markus Strohmaier. 2025. Prompt perturbations reveal human-like biases in large language model survey responses. Preprint, arXiv:2507.07188. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitiv- ity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In International Conference on Learning Representa- tions. Dustin Wright, Arnav Arora, Nadav Borenstein, Sr- ishti Yadav, Serge Belongie, and Isabelle Augenstein. 2024. LLM tropes: Revealing fine-grained values and opinions in large language models. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2024. Association for Computational Linguistics. 8 Appendix 9 Full Model by Dataset Prompt Effects Table 6 reports the within-cell prompt-category ef- fect for each model and dataset pair. Each row comes from a separate binomial GEE model fit within one model-dataset cell. The table shows that prompt category has a significant effect in ev- ery model-dataset combination, which supports the main claim that prompt robustness is not a single global model property. Instead, it varies with the model, dataset, and perturbation category. ModelDatasetχ 2 dfp GemmaARC191.6256 1.15× 10 −38 GemmaMMLU96.9506 1.09× 10 −18 GemmaCulturalBench279.8486 1.69× 10 −57 GemmaPCT757.4366 2.41× 10 −160 GemmaValueBench993.6816 2.08× 10 −211 GemmaWVS1146.8976 1.49× 10 −244 LlamaARC281.0436 9.39× 10 −58 LlamaMMLU333.4026 5.63× 10 −69 LlamaCulturalBench331.3946 1.52× 10 −68 LlamaPCT1367.2456 3.00× 10 −292 LlamaValueBench1765.9286< 10 −300 LlamaWVS1632.5296< 10 −300 MistralARC504.3506 9.72× 10 −106 MistralMMLU314.8006 5.50× 10 −65 MistralCulturalBench450.2676 4.30× 10 −94 MistralPCT1276.7206 1.19× 10 −272 MistralValueBench1658.8876< 10 −300 MistralWVS2245.2706< 10 −300 QwenARC1006.5356 3.45× 10 −214 QwenMMLU220.6726 7.48× 10 −45 QwenCulturalBench407.5186 6.76× 10 −85 QwenPCT172.6186 1.25× 10 −34 QwenValueBench502.7566 2.14× 10 −105 QwenWVS587.6956 1.05× 10 −123 Table 6: Prompt-category effects within each model- dataset binomial GEE model. This is the full version of the model by dataset result table.