Paper deep dive
Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions
Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/20/2026, 4:41:59 AM
Summary
This paper introduces Interaction-based Prompt Sensitivity (IPS), a fine-grained metric to evaluate and explain the instability of Large Language Models (LLMs) caused by subtle prompt changes. By decomposing LLM outputs into nonlinear interactions between input variables, the authors demonstrate that prompt sensitivity arises from unstable interaction patterns, even when final outputs remain consistent. The study evaluates 50 open-source LLMs and identifies four factors that reduce prompt sensitivity: supervised fine-tuning, increased model scale, dense architectures, and few-shot learning. A key finding is that these factors primarily stabilize low-order interactions.
Entities (12)
Relation Signals (9)
Interaction-based Prompt Sensitivity → measures → Prompt Sensitivity
confidence 95% · we propose an Interaction-based Prompt Sensitivity (IPS) metric by quantifying changes in interactions when we introduce subtle changes to prompts.
Supervised Fine-Tuning → reduces → Prompt Sensitivity
confidence 94% · Supervised fine-tuning reduces the prompt sensitivity. Instruct/chat models (with supervised fine-tuning) exhibit lower prompt sensitivity than base models.
Increased Model Scale → reduces → Prompt Sensitivity
confidence 94% · LLMs with larger parameter numbers exhibit lower prompt sensitivity.
Dense Architectures → reduces → Prompt Sensitivity
confidence 94% · Dense models are generally less sensitive than mixture-of-experts (MoE) models.
Few-Shot Learning → reduces → Prompt Sensitivity
confidence 94% · Few-shot learning considerably reduces prompt sensitivity compared to 0-shot learning.
interactions → usedforanalysis → Prompt Sensitivity
confidence 93% · we introduce interactions as a fine-grained tool to analyze prompt sensitivity of LLMs.
Four Factors → stabilize → Low-Order Interactions
confidence 92% · all four factors tend to reduce the prompt sensitivity of low-order interactions (i.e., interactions involving few input variables).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The remarkable capabilities of large language models (LLMs) are often undermined by their instability. Even subtle and semantically irrelevant changes in prompts can cause dramatic fluctuations in performance, a phenomenon known as prompt sensitivity. Previous studies typically evaluate prompt sensitivity by comparing the LLM's final outputs when prompts change. However, such coarse-grained metrics fail to explain the internal reasons for prompt sensitivity. In this paper, we introduce interactions as a fine-grained tool to analyze prompt sensitivity of LLMs. Specifically, we decompose the output score of the LLM into a set of interactions. Each interaction represents a nonlinear relationship involving a set of input variables. We discover that subtle changes to prompts can trigger severe instability in interactions, even when the outputs of the LLM remain the same. To this end, we propose an Interaction-based Prompt Sensitivity (IPS) metric by quantifying changes in interactions when we introduce subtle changes to prompts. We apply the IPS metric to 50 open-source LLMs and uncover four factors that reduce the prompt sensitivity of LLMs, including supervised fine-tuning, increased model scales, dense architectures, and few-shot learning. More crucially, we discover a common mechanism by which these four factors reduce prompt sensitivity: all four factors tend to reduce the prompt sensitivity of low-order interactions (i.e., interactions involving few input variables).
Tags
Links
- Source: https://arxiv.org/abs/2608.18539v1
- Canonical: https://arxiv.org/abs/2608.18539v1
Trouble viewing inline? Open PDF directly →
Full Text
144,014 characters extracted from source content.
Expand or collapse full text
Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Ruiyang Qin 1 Qingzhuo Wang 1 Tian Wang 1 Zhihua Wei 1 Wen Shen 1† Abstract The remarkable capabilities of large language models (LLMs) are often undermined by their instability. Even subtle and semantically irrele- vant changes in prompts can cause dramatic fluc- tuations in performance, a phenomenon known as prompt sensitivity. Previous studies typically eval- uate prompt sensitivity by comparing the LLM’s final outputs when prompts change. However, such coarse-grained metrics fail to explain the in- ternal reasons for prompt sensitivity. In this paper, we introduce interactions as a fine-grained tool to analyze prompt sensitivity of LLMs. Specifically, we decompose the output score of the LLM into a set of interactions. Each interaction represents a nonlinear relationship involving a set of input vari- ables. We discover that subtle changes to prompts can trigger severe instability in interactions, even when the outputs of the LLM remain the same. To this end, we propose an Interaction-based Prompt Sensitivity (IPS) metric by quantifying changes in interactions when we introduce subtle changes to prompts. We apply the IPS metric to 50 open- source LLMs and uncover four factors that re- duce the prompt sensitivity of LLMs, including supervised fine-tuning, increased model scales, dense architectures, and few-shot learning. More crucially, we discover a common mechanism by which these four factors reduce prompt sensitivity: all four factors tend to reduce the prompt sensi- tivity of low-order interactions (i.e., interactions involving few input variables). 1. Introduction LLMs have demonstrated exceptional proficiency in nu- merous natural language processing tasks (Srivastava et al., 2023; Zhou et al., 2023; Annepaka & Pakray, 2024; Chang 1 School of Computer Science and Technology, Tongji Uni- versity, Shanghai, China. † Correspondence to: Wen Shen <wenshen@tongji.edu.cn>. Proceedings of the43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s). et al., 2024), a success largely driven by the effectiveness of prompting. However, this power is undermined by prompt sensitivity. That is, semantically unimportant changes to the prompt can result in divergent outputs (Sclar et al., 2024). Current research (Sclar et al., 2024; Chatterjee et al., 2024; Lu et al., 2024; Alzahrani et al., 2024; Errica et al., 2025; Razavi et al., 2025) on evaluating prompt sensitivity only focuses on the LLM’s final output. These output-based met- rics typically measure changes in performance, such as task accuracy or output consistency. As coarse-grained measures, these metrics only reveal the consequences of prompt sensi- tivity (e.g., the LLM’s prediction changes from one answer to another one) but fail to explain its underlying reasons. In this paper, we aim to evaluate the prompt sensitivity of LLMs from a fine-grained perspective and investigate the underlying reasons why certain factors can decrease the prompt sensitivity of LLMs. Recent research (Chen et al., 2024; Zhou et al., 2024; Ren et al., 2025; 2024b) has utilized interactions to explain the fine-grained inference logic of deep neural networks (DNNs). Inspired by these studies, we introduce the interactions framework to fine-grainedly analyze the prompt sensitivity of LLMs. Specifically, given a sentencexwithninput variables (e.g., words or tokens) indexed byN = 1, 2,...,n, an inter- action represents an intricate nonlinear relationship associ- ated with a specific combination of input variables. Con- sider the sentencex = “He is a green hand.”In this con- text, the idiom “green hand” carries the meaning of “be- ginner”. The joint presence of the input variables in the set S = green, hand⊆Ntriggers a special interaction effect. This interaction effect, denoted asI S , pushes the network’s inference towards the semantic meaning of“beginner.”Li & Zhang (2023) have mathematically demonstrated that the scalar outputv(x)of a DNN is always equivalent to the out- put of an interaction-based logical modelφ(x) = P S⊆N I S . That is,v(x) = φ(x) = P S⊆N I S . Thus, the inference logic of a DNN can be explained by a set of interactions. Coarse-grained analysis vs. fine-grained analysis. Tradi- tional analysis of prompt sensitivity can only coarsely re- flect whether the LLM’s prediction alters when the prompt changes. Beyond this, we introduce an interaction-based log- ical modelφ(x)as a fine-grained analytical tool to analyze the stable and unstable interactions encoded by the LLM for 1 arXiv:2608.18539v1 [cs.LG] 19 Aug 2026 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Question Answers: A.Op A B.Op B C.Op C D.Op D Answer: 푷풓풐풎풑풕 푻풆풎풑풍풂풕풆 푻: Equivalently modeling Interaction-based Logical Model 흓풙= σ 푺⊆푵 푰 푺 (풙|푻) 푣“퐵” 푥,푇) Template Formatting (True Answer: B) 푰풏풑풖풕 풙∶ Question:Which one of these animals does NOT lay eggs Op A: chickens Op B: dogs Op C: frogs Op D: turtles LLM Figure 1. Using interactions for fine-grained analysis of prompt sensitivity. Given an inputxand a prompt templateT, the LLM’s output scorev(“B”|x,T )is equivalent to the output of an interaction-based logical modelφ(x), i.e.,v(“B”|x,T ) = φ(x) = P S⊆N I S . In this way, we can uncover the underlying reasons for the prompt sensitivity of LLMs by analyzing detailed interaction patterns. Specifically, we divide interactions into those that are stable to prompt changes (i.e. stable interaction) and those that are not (i.e. unstable interaction). each specific sample. As shown in Figure 1, when we keep the same inputxbut introduce semantically equivalent alter- ations to the prompt templateT(e.g., change “Answers” to “ANSWERS”), traditional coarse-grained analysis can only reveal that the LLM’s prediction changes from the correct answer “B” to the incorrect answer “A” but fails to explain why or how this change occurs. In contrast, our interaction- based analysis precisely identifies unstable interactions that may contribute to the prompt sensitivity of the LLM. An in- teraction is considered stable if its effect changes minimally (e.g.,I lay, eggs shifts from 0.6242 to 0.6076). Conversely, an interaction is deemed unstable if its effect fluctuates dramat- ically (e.g.,I NOT, lay, eggs, dogs shifts from 0.5175 to -0.0594). Strikingly, we find that unstable interactions exist even when the LLM’s final output remains the same. This indicates that our fine-grained analysis reveals potential instability that is entirely invisible to traditional, output-level metrics. Building on these findings, we leverage the interaction framework to propose a fine-grained metric to evaluate the prompt sensitivity of LLMs, which is termed Interaction- based Prompt Sensitivity (IPS). This metric quantifies changes in interaction patterns when LLMs process differ- ent prompts. We apply the proposed IPS metric to evaluate the prompt sensitivity of 50 open-source LLMs. However, drawing conclusions about which LLM families are more or less sensitive from this ranking is challenging, as an LLM’s prompt sensitivity stems from multiple, intertwined factors. To this end, we conduct a series of comparative experiments to disentangle these factors, discovering four factors that reduce the prompt sensitivity of LLMs: (1) Supervised fine- tuning reduces the prompt sensitivity. Instruct/chat models (with supervised fine-tuning) exhibit lower prompt sensi- tivity than base models. (2) LLMs with larger parameter numbers exhibit lower prompt sensitivity. (3) Dense models are generally less sensitive than mixture-of-experts (MoE) models. (4) Few-shot learning considerably reduces prompt sensitivity compared to 0-shot learning. More crucially, we explore and uncover a common underly- ing mechanism that explains how the four aforementioned factors reduce prompt sensitivity: they primarily reduce the instability of low-order interactions (i.e., interactions involving a small number of input variables). This find- ing is counterintuitive, as our experiments demonstrate that high-order interactions (i.e., interactions involving many input variables) tend to exhibit the highest sensitivity, while low-order interactions are inherently less sensitive. Unex- pectedly, these factors further stabilize the already stable low-order interactions, yet remain ineffective in addressing the more pronounced sensitivity of high-order interactions. 2. Related Work Prompt sensitivity of LLMs. Previous studies (Sun et al., 2024; Zhu et al., 2024a; Sclar et al., 2024; Alzahrani et al., 2024) demonstrated that LLMs are highly sensitive to mi- nor perturbations or semantically unimportant alterations to prompts, which can lead to significant performance varia- tion. Such prompt sensitivity presents a considerable risk to the reliability of LLMs. Existing metrics (Sclar et al., 2024; Chatterjee et al., 2024; Lu et al., 2024; Zhuo et al., 2024; Cao et al., 2024; Errica et al., 2025; Razavi et al., 2025) for evaluating the prompt sensitivity of LLMs typically measure shifts in final outputs, such as task accuracy or output con- sistency. However, these metrics are coarse-grained and fail to probe the LLM’s internal logic. In this paper, we propose 2 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions a fine-grained metric that evaluates prompt sensitivity based on interactions. This framework enables us to uncover the underlying mechanisms of prompt sensitivity in LLMs. Using game-theoretic interactions to explain DNNs. Tra- ditional methods for explanations (Tenney et al., 2020; Zhang et al., 2020) often lack mathematical guarantees of faithfulness, meaning their outputs may not accurately rep- resent the internal logic employed by the DNN. To this end, Ren et al. (2023a) proposed to use interactions between input variables to explain DNNs and provided a series of the- oretical guarantees for the method’s validity. Furthermore, it has been empirically discovered (Li & Zhang, 2023) and theoretically proven (Ren et al., 2024a) that a DNN typically encodes only a sparse set of interactions. At the application level, the interaction framework has proven effective in a wide range of complex tasks, including adversarial trans- ferability (Deng et al., 2024), model generalization (Chen et al., 2024; Zhou et al., 2024), model training process (Ren et al., 2025; 2024b), overfitting (Ren et al., 2023b) and other tasks (Shen et al., 2024; Li et al., 2025; Wen et al., 2026). In this paper, we use the interaction framework to analyze the prompt sensitivity of LLMs. 3. Interaction-Based Analysis of the Prompt Sensitivity of LLMs 3.1. Preliminaries: Interactions This subsection introduces the definition of interactions, as well as the mathematical guarantees of interaction-based explanation. Given a DNNvand an input sentencexwithn input variables (e.g., words) indexed byN =1, 2,...,n, letv(x)∈ Rdenote the scalar output of the DNN. Here we setv(x) = log p(y=y * |x) 1−p(y=y * |x) ∈ R , wherep(y = y * |x)repre- sents the probability of generating the ground truth token y * given the inputx. We define a surrogate logical model φ(x)to match the scalar outputv(x)of the DNN. Recent studies (Li & Zhang, 2023; Ren et al., 2023a) have proven Theorem 3.1, which shows that the output scorev(x)on any randomly masked 1 inputx T can be accurately calculated by the following surrogate logical model φ(·). φ(x T )≜ φ(x ∅ ) + X S⊆N 1(S | x T )· I S ,(1) where the AND trigger function1(S | x T ) ∈ 0, 1rep- resents an AND relationship between input variables in S, which can also be termed AND interaction pattern. The scalar weightI S quantifies the effect of an AND rela- tionship, which can also be termed interaction effect. An AND relationship is activated only by the joint presence of all input variables in the setS, i.e., all input variables in 1 It is common to use a specific token or embedding to mask input variables of a DNN, e.g., replacing the target token with a specific [MASK] token. Please see Appendix A for details. Sare not masked. For instance, given the input sentence x = “He is a green hand,”the co-occurrence of the input vari- ables in the setS = green, handcontributes a numerical effectI S that pushes the surrogate logical model’s inference towards the semantic meaning of“beginner.”If an AND interactionSis triggered, i.e.,1(S | x T ) = 1, the corre- sponding interaction effectI S is added to the output of the logical model. Otherwise, if any word inSis masked and the AND interaction is not triggered, i.e.,1(S | x T ) = 0, its corresponding interaction effectI S is not added to the output of the logical model.x ∅ represents that all input variables in N are masked. Theorem 3.1 (Universal matching property, proven in Ap- pendix B). Given an inputxwithninput variables, we randomly mask any combinations of input variables to gen- erate2 n masked inputsx T | T ⊆ N. For every masked inputx T , when the scalar weightI S in the logical modelφ(·) are set toI S = P S ′ ⊆S (−1) |S|−|S ′ | · v(x S ′ ),φ(x ∅ ) = v(x ∅ ), the output of the logical modelφ(·)can always match the DNN’s output score v(·). ∀ T ⊆ N, v(x T ) = φ(x T )(2) In addition, the sparsity property also provides a theoret- ical guarantee for the faithfulness of interaction-based explanation. The sparsity property shows that a DNN only encodes a sparse set of interactions with salient effects. That is, only a small subset of all2 n interactions in Theorem 3.1, termed salient interactions, have a significant impact on the logical model’s output. In contrast, the majority of interac- tions have negligible effects and are considered noise pat- terns. The sparsity property of AND interactions has been proven by Ren et al. (2024a). We follow Li & Zhang (2023) to extract AND-OR interactions 2 from input variables. Such a technique has proven effective in pursuing higher sparsity of interactions, supported by both theoretical proofs (Li & Zhang, 2023) and extensive empirical validation (Ren et al., 2023a; Zhou et al., 2024). 3.2. Verifying the Faithfulness of Considering Interactions as Inference Patterns Used by LLMs Before utilizing the interaction framework to analyze the prompt sensitivity of LLMs, we need to theoretically prove and experimentally validate the faithfulness of using in- teractions to explain LLMs. In theory, Theorem 3.1 guar- antees that the surrogate logical model’s outputφ(·)can always match the LLM’s outputv(·)for all2 n masked sam- ples. Since the logical model’s outputφ(·)is composed entirely of the sum of all interaction effects as defined in Eq. (1), we can consider interactions as the detailed in- ference patterns that constitute the LLM’s internal logic. 2 The OR interaction is proved to be a specific AND interaction. Please see Appendix C for proof and Appendix D for how to extract OR interactions. 3 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Zoom inZoom in (a) (b) LLM ： Llama - 2 - 7b LLM’s outputmatching error 푣 푥 푇 Figure 2. (a) Verifying the sparsity of interactions. We show absolute values of normalized interactions in a descending order. LLMs all encode a small number of salient interactions, while most of the interaction effects are negligible. (b) Verifying the quality of universal matching for any 2 n masked inputs. The red line plots outputs of the LLM in an ascending order. Same Output Different Outputs Interaction Interaction Effect 푰 푺 풙푻 푰 푺 풙 푻 (a) (b) Interaction Interaction Effect Interaction Interaction Effect Interaction Interaction Effect Figure 3. (a) Case studies of interaction-level analysis. The plots compare the interaction effects encoded by the LLM for the same input xunder two semantically equivalent prompt templates,T(blue) and ˆ T(red). The bottom row shows examples where the LLM’s output remains the same; the top row shows examples where it changes. Results show that interaction effects are highly unstable, even when the output of the LLM remains the same. (b) The distribution of salient interactions types of various LLMs. Therefore, we can evaluate the prompt sensitivity of LLMs by measuring the instability of interaction patterns. In practice, we conduct experiments to verify whether LLMs encode sparse interactions. Consider the multiple choice question (MCQ) in Figure 1 as an example. LetQ denote the set of all words in the question, e.g.,Q=Which, one, of, these, animals, does, NOT, lay, eggs,Mdenote the set of all words in the options, e.g.,M=chickens, dogs, frogs, turtles, andTdenote the set of all words in the prompt tem- plate, e.g.,T=Answers:, A., B., C., D., Answer:. Specifically, we use words 3 inQ∪ Mas input variables and compute the interaction effectI S of all interactionsS ⊆ Q∪ M. Mean- while, we treat words in prompt templateTas background context. We follow Li & Zhang (2023) to extract AND and OR interactions. Thus, given an inputx = Q∪ Mwithn words, we can obtain2 n+1 interactions, including2 n AND interactions and2 n OR interactions. For each interaction ef- 3 We use words instead of tokens as input variables because different LLMs may divide the same word into different tokens. For example, Llama-2-7B tokenizes the word “Elements” into two tokens “Element” and “s”, while Qwen3-8B treats it as one token. fectI S , we apply min-max normalization to it. Specifically, e I S ≜ sgn(I S )· |I S |−Min Max−Min , whereMinandMaxare the minimum and maximum absolute values of all2 n+1 interaction effects; sgn(I S ) = I S |I S | represents the sign ofI S . Figure 2 (a) shows the distribution of| e I S |. Results 4 verify that only a small set of interactions have salient effects, while most of the interactions have negligible effects and can be considered as noise patterns. Figure 2 (b) compares the LLM’s true outputv(x T )for all2 n masked inputs against the logical model using only the most salient interactions. Even when using the top 3% or top 5% of all interactions, the matching error is minimal. This empirically demonstrates that the LLM’s output can be faithfully approximated by a small set of salient interactions. 3.3. Using Interactions as a Fine-Grained Tool to Analyze the Prompt Sensitivity of LLMs Based on the above verification, we deploy interactions as a fine-grained analytical tool. Specifically, for an LLM, we 4 Results on more LLMs in Appendix F support the conclusion. 4 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions (a)(b) Figure 4. Prompt sensitivity of all the 50 open-source LLMs in an ascending order. visualize the changes in salient interactions for the same in- put under a pair of prompt templates. As Figure 3 (a) shows, when the LLM’s outputs are different, nearly half of the salient interactions reverse their sign (from positive to nega- tive, or vice versa), another 30%-40% change significantly in magnitude, and only a small fraction (10%-20%) remain stable. Strikingly, even when the LLM’s output remains the same, the majority (60%-80%) of the salient interactions are still unstable. This offers preliminary evidence that semanti- cally irrelevant alterations to the prompt template can lead to significant changes in the salient interaction patterns. To systematically analyze this instability beyond individual cases, we classify interactions into three distinct types: 1) Opposite sign: The interaction effect reverses its sign (e.g., from positive to negative). The sign change represents the most severe form of instability, as it can completely reverse the LLM’s internal logic. 2) Same sign & different effect: The interaction maintains its positive or negative influence, but its magnitude changes substantially (i.e., its effect is more than doubled or less than halved). 3) Same sign & similar effect: The interaction’s sign and magnitude both remain stable, which represents robust, stable interactions. Figure 3 (b) plots the distribution of three interaction types over all samples, conditioned on whether the LLM’s final output changes or remains the same across a pair of prompts. When the LLM generates different outputs, the interaction patterns are highly unstable. Opposite sign interactions ac- count for approximately 50%, meaning nearly half of salient interactions reverse the sign of their effect. The truly stable Same sign & similar effect interactions account for a mere 2.7%-6.8%, while the remaining 42.6%-47.4% of interac- tions, though maintaining their sign, change significantly in magnitude. More alarmingly, even when the final output of the LLM remains the same, the instability of interactions still exists. While the situation improves, the majority of interactions still fall into the two unstable categories.. The results demonstrate that output-level analysis is insuf- ficient to capture the unreliable internal patterns of LLMs. Conversely, our interaction-based analysis offers a fine- grained lens to uncover latent instability of LLMs, offering a new analytical tool for quantifying the ratio of stable and unstable interactions on a per-sample basis for any LLM. 4. Evaluating and Analyzing the Prompt Sensitivity of LLMs 4.1. Evaluating Interaction-Based Prompt Sensitivity We propose an interaction-based metric to measure the prompt sensitivity of LLMs. Given an MCQ datasetD, for any inputx∈D, it is composed of the questionQand the optionsM, i.e.,x = Q∪M. Given a prompt templateT, we make minor changes toTand obtain a modified prompt template ˆ T. By applying the pair of prompt templatesTand ˆ Tto the same inputQ∪ M, we can construct two similar prompts. The LLM is supposed to extract similar salient interactions fromQ∪Mwhen processing these two prompts becauseTand ˆ Thave the same semantic meaning. Thus, letΩ salient (x|T ) = S ∈ Ω(x) | | e I S (x|T )| > τrepresent the set of salient interactions extracted fromxgiven the prompt templateT. Hereτis a threshold used to distin- guish salient interactions from noise patterns. In the main paper, we setτas 0.1 for all experiments. Please see Ap- pendix F.8 for hyperparameter experiments ofτ. Similarly, letΩ salient (x| ˆ T ) = S ∈ Ω(x) | | e I S (x| ˆ T )| > τrepresent the set of salient interactions extracted fromxgiven the prompt template ˆ T. Therefore, on the datasetD, we define the LLM’s Interaction-based Prompt Sensitivity as IPS. IPS≜ E x E T, ˆ T 1 |Ω union | X S∈Ω union | e I S (x|T )− e I S (x| ˆ T )| | e I S (x|T )|+| e I S (x| ˆ T )| 2 , (3) whereΩ union = Ω salient (x|T ) ∪ Ω salient (x| ˆ T )is a unified set by taking the union of the two salient setsΩ salient (x|T )and Ω salient (x| ˆ T ); the outer expectation,E x , represents an averag- ing over all inputsx∈D, and the inner expectation,E T, ˆ T , represents an averaging over all pairs of prompt templates (T, ˆ T ). This metric evaluates prompt sensitivity of LLMs by calculating the symmetric mean absolute percentage error of salient interactions over all samples in the datasetD. Models and Datasets. We conduct experiments on 50 open- source LLMs from 6 model families. This diverse set includes 10 LLMs from the Llama family: Llama-2, Llama- 3, Llama-MoE (Touvron et al., 2023; Grattafiori et al., 2024; Zhu et al., 2024b; Qu et al., 2024); 4 LLMs from the Mistral family: Mistral, Mixtral (Jiang et al., 2023; Jiang 5 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions et al., 2024); 25 LLMs from the Qwen family: Qwen2, Qwen2.5, Qwen3, Qwen1.5-MoE, Qwen3-30B-A3B (Yang et al., 2024; 2025); 9 LLMs from the OLMo family: OLMo, OLMo-2, OLMoE (Groeneveld et al., 2024; OLMo et al., 2025; Muennighoff et al., 2024); 2 LLMs from the InternLM family: InternLM2 (Cai et al., 2024). We evaluate all LLMs on two widely used MCQ benchmarks: ARC (Clark et al., 2018) and MMLU (Hendrycks et al., 2021). Experiments on open-ended tasks are shown in Section 4.3. For each input, we apply five distinct prompt templates to it. These templates maintain the same core content and differ only in minor formatting details, such as letter case and separators. For masking words in the input sentences, we follow the ap- proach of Cheng et al. (2025) and utilize a certain[MASK] token for each LLM. A comprehensive list of all LLMs, along with the specific prompt templates, mask tokens used in experiments is provided in Appendix E. We apply the IPS metric to evaluate the prompt sensitivity of 50 open-source LLMs. As shown in Figure 4 (a), we observe a wide variance in IPS, ranging from 1.268 (Qwen2.5-72B- Instruct) to 1.752 (Mistral-7B-v0.3). However, no LLM series or families achieve a complete victory. The distribu- tion indicates that an LLM’s prompt sensitivity is influenced by multiple underlying factors. Identifying these factors is of great significance for the robustness research of LLMs. To validate the reliability of the IPS metric, we investigate its correlation with the traditional metrics output consis- tency, which is defined as the proportion of samples where the LLM generates identical predictions across different prompt templates. Figure 4 (b) shows that the IPS score exhibits a negative correlation with output consistency. This result confirms that IPS aligns well with the output con- sistency while offering a more fine-grained perspective to quantify prompt sensitivity beyond output matching. Robustness to the thresholdτ. In the main paper, we set the thresholdτas 0.1 for all experiments. To verify that our findings are robust to the selection ofτ, we compute interac- tions across 16 distinct threshold values, ranging from 0.05 to 0.20 with a step size of 0.01. We evaluate the consistency of IPS rankings across these thresholds, observing an aver- age Spearman’s rank correlation of 0.9905 and an average Pearson correlation of 0.9957 (Appendix F.8.1). These near- perfect correlations indicate that the relative ranking among LLMs remains highly stable. More crucially, we conduct all the experiment withτ = 0.05andτ = 0.15. As detailed in Appendix F.8.2, the main conclusions remain same as τ = 0.1, proving that our findings are robust to different τ . 4.2.Analyzing the Factors Impacting Prompt Sensitivity In this section, we investigate four factors that might influ- ence the prompt sensitivity of LLMs, including (1) super- vised fine-tuning, (2) model scales, (3) model architectures, Llama-2-7B Pair Llama-2-13B PairLlama-2-70B Pair Llama-3-8B Pair Mistral-7B-V0.3 Pair Mixtral-8X7B Pair Internlm2-7B Pair Olmo-7B Pair Olmoe-7B Pair Qwen2-7B PairQwen3-4B PairQwen3-8B Pair Qwen3-14B Pair Qwen1.5-Moe-A2.7B Pair Qwen3-30B-A3B Pair 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS Base ModelsInstruct / Chat Models Figure 5. A comparison of the prompt sensitivity between instruc- t/chat models and base models. Results show that instruct/chat models are less sensitive than corresponding base models. Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Tip: within each model series, lighter color represents smaller model scale Figure 6. A comparison of prompt sensitivity across different model scales. As the model scale increases, the prompt sensitivity within a model series systematically decreases. and (4) prompting methods. Factor 1: instruct/chat models vs. base models. Super- vised fine-tuning (e.g., instruction tuning) is now a common practice to align base models with human preferences for certain tasks, yielding models often referred to as instruct or chat models (for different tasks or purposes). Therefore, we investigate the impact of supervised fine-tuning by compar- ing the prompt sensitivity of instruct/chat models with base models. Results 5 on the ARC dataset in Figure 5 show that almost all instruct/chat models exhibit lower prompt sensitiv- ity than their corresponding base models. This demonstrates that supervised fine-tuning enables the LLM to encode more stable interactions. A valid explanation is that base models are pre-trained on unstructured and raw texts, but instruc- t/chat models are further fine-tuned on instruction-response datasets or dialogue datasets. Thus, instruct/chat models can precisely understand the function of prompt templates and focus on the task-relevant inputs. Factor 2: model scales. We investigate the relationship between the model scale (i.e., the number of parameters) and the prompt sensitivity. Results 5 on the ARC dataset in Figure 6 show that within the same model series, as the model scale increases, the overall prompt sensitivity sys- tematically decreases. This demonstrates that larger LLMs 5 Results on the MMLU dataset in Appendix F exhibit the same conclusion. 6 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Llama 2 FamilyLlama 3 Family Mistral Family Qwen Family Olmo Family 1.3 1.4 1.5 1.6 1.7 Prompt Sensitivity IPS Llama-2-70B-Chat Llama-2-70B Llama-2-13B-Chat Llama-2-13B Llama-2-7B-Chat Llama-2-7B Llama-Moe-V1-3_5B-2_8-Sft Llama-3-8B-Instruct Llama-3-8B Llama-Moe-V2-3_8B-2_8-Sft Mistral-7B-V0.3-Instruct Mixtral-8X7B-Instruct Mixtral-8X7B Mistral-7B-V0.3 Qwen2.5-72B-Instruct Qwen2-72B-Instruct Qwen2-7B-Instruct Qwen3-14B-Instruct Qwen2.5-32B-Instruct Qwen2.5-14B-Instruct Qwen3-4B-Instruct Qwen3-8B-Instruct Qwen3-8B Qwen3-14B Qwen2.5-7B-Instruct Qwen3-4B Qwen2-7B Qwen3-30B-A3B-Instruct Qwen3-30B-A3B Qwen1.5-Moe-A2.7B-Chat Qwen1.5-Moe-A2.7B Olmo-7B-Instruct Olmo-2-32B-Instruct Olmo-2-7B-Instruct Olmo-2-13B-Instruct Olmo-7B Olmoe-7B Olmoe-7B-Instruct MoE Model Dense Model MoE Model Dense Model Figure 7. A Comparison of prompt sensitivity between MoE mod- els and dense models. Generally, MoE models tend to be more sensitive than dense models in the same model family. Llama-2-7B Llama-2-7B-Chat Llama-3-8B Llama-3-8B-Instruct Mistral-7B-V0.3 Mistral-7B-V0.3-Instruct Qwen2-7B Qwen2-7B-Instruct Qwen2.5-7B-Instruct Qwen3-8B-Instruct Olmo-2-7B-Instruct Olmoe-7B-Instruct Qwen1.5-Moe-A2.7B-Chat 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 Prompt Sensitivity IPS 0-shot1-shot2-shot3-shot Figure 8. A comparison of prompt sensitivity between 0-shot learning and few-shot learning. The drop in prompt sensitivity is substantial from 0-shot to 1-shot. encode more stable interactions, making them less suscepti- ble to superficial changes in the prompt template. Factor 3: dense models vs. MoE models. MoE models scale up model capacity with minimal computational cost by dynamically activating different subsets of “expert” sub- networks (Cai et al., 2025). In contrast, all parameters in dense models participate in every computation. We aim to investigate the impact of model architectures by comparing the prompt sensitivity of dense models with MoE models. Results 5 on the ARC dataset in Figure 7 show that in model families including Llama-2, Llama-3, Qwen and Olmo, all MoE models exhibit higher prompt sensitivity than dense models. This suggests that MoE models encode more un- stable interactions. A potential confounding factor is the number of active parameters. Given that smaller models are more sensitive (Factor 2), one might attribute MoE instabil- ity to lower active parameters. To mitigate the influence of active parameters, we conduct controlled variable analysis. Results (Table 7, Appendix I) confirms that MoE models remain more sensitive than dense models even with similar active parameters. We attribute this to the dynamic rout- ing mechanism. For different prompt templates, the gating network may route the input to different experts so that it is processed by different sub-networks, leading to different interactions encoded and thus higher sensitivity. In conclu- sion, while MoE models achieve impressive performance with reduced computational overhead, this benefit comes at the cost of weaker stability. Factor 4: few-shot learning vs. 0-shot learning. Few- Llama-2-7B-Chat Llama-3-8B-Instruct Mistral-7B-V0.3-Instruct Mixtral-8X7B-Instruct Qwen2-7B-Instruct Qwen2.5-7B-Instruct Qwen3-8B-Instruct Qwen1.5-Moe-A2.7B-Chat Qwen3-30B-A3B-Instruct Olmo-7B-Instruct Olmoe-7B-Instruct Internlm2-Chat-7B 0.75 1.00 1.25 1.50 1.75 2.00 Prompt Sensitivity of Different Orders IPS type Low OrderMid OrderHigh Order Figure 9. Prompt sensitivity of LLMs for different order types. shot learning is utilized to improve LLMs’ performance by providing in-context examples to better specify the task (Wang et al., 2020). We investigate the impact of prompting methods on prompt sensitivity by comparing 0-shot learning with few-shot learning 6 . Results 5 on the ARC dataset in Fig- ure 8 show that incorporating in-context examples leads to a significant reduction in prompt sensitivity across all tested LLMs. For most of the LLMs, the most substantial drop occurs when moving from 0-shot to 1-shot, while adding more in-context examples yields slower reductions. This suggests that even a single example is sufficient to establish the LLM’s understanding of the task, leading it to ignore superficial template variations and focus on the core input. 4.3. Explore the Underlying Mechanisms of Improved Stability for All Factors In this section, we aim to explore whether there exists a common reason to explain the underlying mechanisms by which the four aforementioned factors reduce the prompt sensitivity of LLMs. Specifically, we analyze the prompt sensitivity of different types of interactions, so as to reveal the source of the LLM’s prompt sensitivity. To this end, we analyze the sensitivity of interactions with different complexities, which are defined as the orders of interac- tions. The order of an interactionSis defined as the number of input variables involved, i.e.,|S|. An interaction with high order indicates an intricate relationship including many input variables, while an interaction with low order repre- sents a simple relationship including few input variables. We further define three types of prompt sensitivity met- rics corresponding to different types of interaction orders. Specifically, we partition interactions inΩ union in Eq. (3) into three distinct groups based on their orders: low-order, mid-order, and high-order. Given an inputxwithnwords, Ω low union ≜S ∈ Ω union | 1≤|S|≤⌊ 1 3 n⌋, Ω mid union ≜S ∈ Ω union | ⌊ 1 3 n⌋ <|S|≤⌊ 2 3 n⌋, Ω high union ≜S ∈ Ω union |⌊ 2 3 n⌋ <|S|≤ n . Then we calculate prompt sensitivity for low-order, mid- order, and high-order interactions, as IPS low , IPS mid , IPS high . Figure 9 presents the prompt sensitivity of different order 6 See Appendix E for the prompt templates and settings of few-shot learning. 7 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions (a) The relative change of Instruct/Chat Model compared to Base Model (b) The relative change of Dense Model compared to MoEModel (c) The relative change of Few-shot learning compared to 0-shot learning Figure 10. Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors. Llama-2 (Chat) Qwen2.5 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity of Different Orders IPS type 7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B Low Order Llama-2 (Chat) Qwen2.5 (Instruct) Olmo2 (Instruct) 7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B Mid Order Llama-2 (Chat) Qwen2.5 (Instruct) Olmo2 (Instruct) 7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B High Order Tip: within each model series, lighter color represents smaller model scale Figure 11. A comparison of prompt sensitivity of different order types across different model scales. types on the ARC dataset. Results 4 show that the prompt sensitivity of low-order interactions is the lowest, followed by mid-order, while high-order interactions exhibit the high- est prompt sensitivity. This indicates that low-order interac- tions encoded by LLMs are relatively stable when faced with subtle changes to prompt templates, i.e., simple interaction patterns are more robust. Conversely, the high sensitivity of high-order interactions reveals that the LLMs’ internal representation of complex patterns is highly unstable. Explaining why the four factors can reduce prompt sensi- tivity. Inspired by the results in Figure 9, we now investigate how the four aforementioned factors influence the prompt sensitivity of low-, mid-, and high-order interactions. This order-level analysis aims to reveal the common mechanism by which these factors reduce the LLM’s prompt sensitivity. For each factor, we quantify its effect on low-, mid-, and high-order interactions by computing the relative change in IPS between the LLM with the factor and its counterpart without it. Specifically, giventype∈low, mid, high, the rel- ative change is defined as∆IPS type = (IPS type A −IPS type B )/IPS type B . For Factor 1 (fine-tuned vs. base), A and B are fine-tuned and base models, respectively. For Factor 3 (dense vs. MoE), A and B are dense and MoE models, with the final ∆IPS type being the average over all pairs of a specific dense model and a specific MoE model within a model family. For Factor 4 (few-shot vs. 0-shot), A and B are x-shot (x∈1, 2, 3) and 0-shot learning. The relative change met- ric is unsuitable for Factor 2 (model scales), as scale is a continuous variable. Instead, we directly analyze the trend of IPS values as model parameters increase. Results 4 shown in Figure 10 and Figure 11 converge on a common explanation for how the four factors reduce prompt sensitivity. The most significant reduction in prompt sen- sitivity is consistently observed in low-order interactions. An obvious, though less pronounced, decrease is also seen at the mid-order level. In contrast, the change of the sensitivity of high-order interactions is relatively minimal, remaining at a high level. It indicates that the stability of low-order interactions is critical to the overall robustness of LLMs. This phenomenon is unexpected. Although results in Fig- ure 9 show that low-order interactions are naturally more robust than other types of interactions, the four factors above still significantly reduce the sensitivity of low-order inter- actions. Instead, they fail to reduce the sensitivity of high- order interactions, which are inherently the most sensitive. This phenomenon indicates that stable low-order interac- tions are much easier for LLMs to learn, while it is difficult for LLMs to make high-order interactions more stable. Robustness on open-ended tasks. To verify the robustness of our findings, we conduct additional experiments on open- ended generation tasks. We utilize the Dolly-15k dataset (Conover et al., 2023), which contains a diverse range of non-MCQ tasks, including open Q&A, classification, and others. The results of this analysis, detailed in Appendix K, consistently verify our main conclusions drawn from the MCQ experiments. This strongly suggests that the four factors and the underlying mechanisms of prompt sensitiv- ity we have uncovered can be generalized to open-ended questions. Robustness to more complex prompt perturbations. In this experimental setup, we use the Dolly-15k dataset and introduce more complex prompt perturbations, specifically semantic paraphrases and instruction reordering (detailed in Appendix L.1), to test the robustness of our conclusions. The results in Figures 12, 13, and 14 consistently affirm our conclusions drawn from the template-based experiments. This indicates that our conclusions are robust to more com- 8 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions (a) Instruct/Chat Models vs. Base Models (b) Model Scales(c) Dense Models vs. MoEModels (d) Few-Shot Learning vs. 0-Shot Learning Figure 12. (a) A comparison of the prompt sensitivity between instruct/chat models and base models. (b) A comparison of prompt sensitivity across different model scales. (c) A comparison of prompt sensitivity between MoE models and dense models. (d) A comparison of prompt sensitivity between 0-shot learning and few-shot learning. (a)The relative change of Instruct/Chat Model compared to Base Model (b) The relative change of Dense Model compared to MoE Model (c) The relative change of Few-shot learning compared to 0-shot learning Figure 13. Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors. Llama-2 (Base) Llama-2 (Chat) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity of Different Orders IPS type 7B13B70B7B13B70B Low Order Llama-2 (Base) Llama-2 (Chat) 7B13B70B7B13B70B Mid Order Llama-2 (Base) Llama-2 (Chat) 7B13B70B7B13B70B High Order Figure 14. A comparison of prompt sensitivity at the order-level across different model scales. plex prompt perturbations. Strategies for reducing the computational cost. To re- duce the computational cost of the interaction framework, recent studies (Chen et al., 2024; Cheng et al., 2025) used the following two strategies: (1) Select informative words as input variables while treating uninformative ones (e.g., stop words) as fixed background context. (2) Merge related words into combined phrases as input variables. The specific se- lection strategies are detailed in Appendix K.2. Results in Appendix K.3 show that using the above two strategies on open-ended tasks yields the same conclusions. In addi- tion to selection strategies, techniques specifically designed for efficiently computing sparse interactions to bypass ex- haustiveO(2 n )evaluations (Kang et al., 2025; Butler et al., 2026) represent a clear path for reducing computational cost. Further details are provided in Appendix J. 5. Conclusion In this paper, we propose an interaction-based metric to evaluate the prompt sensitivity of LLMs. We discover that employing supervised fine-tuning, increasing model scale, using dense over MoE architectures, and applying few-shot learning all serve to reduce the prompt sensitivity of LLMs. Our findings offer novel insights into both model designs and prompting methods for improving the robustness of LLMs. More crucially, we find that these factors achieve lower sensitivity primarily by reducing the sensitivity of low-order interactions, while the prompt sensitivity of high- order interactions remains at a relatively high level. In future studies, new training methods could be designed to increase the LLM’s reliance on stable low-order interactions or, alter- natively, to reduce the instability of high-order interactions. 9 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Acknowledgements This work is partially supported by the Shanghai Sci- ence and Technology Commission (No. 25511102900), the National Nature Science Foundation of China (No.62376199,62576249), and the Shanghai Municipal Ed- ucation Commission (No. 24CGA20). Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. References Alzahrani, N., Alyahya, H., Alnumay, Y., AlRashed, S., Alsubaie, S., Almushayqih, Y., Mirza, F., Alotaibi, N., Al-Twairesh, N., Alowisheq, A., Bari, M. S., and Khan, H. When benchmarks are targets: Revealing the sen- sitivity of large language model leaderboards. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), p. 13787–13805, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1 0 . 1 8 6 5 3 / v 1 / 2 0 2 4 . a c l - l o n g . 7 4 4.URLhttps: //aclanthology.org/2024.acl-long.744/. Ancona, M., Oztireli, C., and Gross, M. Explaining deep neural networks with a polynomial time algorithm for Shapley value approximation. In International conference on machine learning, p. 272–281. PMLR, 2019. Annepaka, Y. and Pakray, P. Large language models: A sur- vey of their development, capabilities, and applications. Knowledge and Information Systems, p. 1–56, 2024. Bird, S. NLTK: the natural language toolkit. In Proceed- ings of the COLING/ACL 2006 interactive presentation sessions, p. 69–72, 2006. Butler, L., Agarwal, A., Kang, J. S., Erginbas, Y. E., Yu, B., and Ramchandran, K. ProxySPEX: Inference-efficient interpretability via sparse feature interactions in LLMs. In The Thirty-ninth Annual Conference on Neural In- formation Processing Systems, 2026. URLhttps: //openreview.net/forum?id=KI8qan2EA7. Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 2025. Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. InternLM2 technical report. arXiv preprint arXiv:2403.17297, 2024. Cao, B., Cai, D., Zhang, Z., Zou, Y., and Lam, W. On the worst prompt performance of large language models. Advances in Neural Information Processing Systems, 37: 69022–69042, 2024. Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45, 2024. Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., and Chakraborty, T. POSIX: A prompt sensitiv- ity index for large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, p. 14550–14565, Miami, Florida, USA, Novem- ber 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings- emnlp.852. URL https://aclanthology.org/2024.findin gs-emnlp.852/. Chen, L., Lou, S., Huang, B., and Zhang, Q.Defin- ing and extracting generalizable interaction primitives from DNNs. In The Twelfth International Conference on Learning Representations, 2024.URLhttps: //openreview.net/forum?id=OCqyFVFNeF. Cheng, L., Zhang, J., Ren, Q., and Zhang, Q. Revisiting generalization power of a DNN in terms of symbolic interactions. arXiv preprint arXiv:2502.10162, 2025. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try ARC, the AI2 reasoning chal- lenge. arXiv preprint arXiv:1803.05457, 2018. Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world’s first truly open instruction- tuned llm, 2023. URLhttps://w.databricks .com/blog/2023/04/12/dolly-first-ope n-commercially-viable-instruction-tun ed-llm. Covert, I., Lundberg, S., and Lee, S.-I. Explaining by remov- ing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1–90, 2021. Dabkowski, P. and Gal, Y. Real time image saliency for black box classifiers. Advances in neural information processing systems, 30, 2017. Deng, H., Zou, N., Du, M., Chen, W., Feng, G., Yang, Z., Li, Z., and Zhang, Q. Unifying fourteen post-hoc attribution methods with Taylor interactions. IEEE Transactions on 10 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Pattern Analysis and Machine Intelligence, 46(7):4625– 4640, 2024. Errica, F., Sanvito, D., Siracusano, G., and Bifulco, R. What did I do wrong? quantifying LLMs’ sensitivity and consis- tency to prompt engineering. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 1543–1558, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.73. URL https:// aclanthology.org/2025.naacl-long.73/. Fong, R., Patrick, M., and Vedaldi, A. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF international conference on computer vision, p. 2950–2958, 2019. Fong, R. C. and Vedaldi, A. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, p. 3429–3437, 2017. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Groeneveld, D., Beltagy, I., Walsh, E., Bhagia, A., Kinney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., et al. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15789–15809, 2024. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URLhttps://open review.net/forum?id=d7KBjmI3GmQ. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Singh Chaplot, D., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Renard Lavaud, L., Lachaux, M.-A., Stock, P., Le Scao, T., Lavril, T., Wang, T., Lacroix, T., and El Sayed, W. Mistral 7B. arXiv e-prints, art. arXiv:2310.06825, October 2023. doi: 10.48550/arXiv.2310.06825. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. Kang, J. S., Butler, L., Agarwal, A., Erginbas, Y. E., Pedarsani, R., Yu, B., and Ramchandran, K. SPEX: Scal- ing feature interaction explanations for LLMs. In Forty- second International Conference on Machine Learning, 2025. URLhttps://openreview.net/forum ?id=pRlKbAwczl. Li, M. and Zhang, Q. Does a neural network really en- code symbolic concepts? In International conference on machine learning, p. 20452–20469. PMLR, 2023. Li, Q., Ruan, J., Wu, F., Chen, Y., Wei, Z., and Shen, W. A unified approach to interpreting self-supervised pre- training methods for 3D point clouds via interactions. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 27315–27324, 2025. doi: 10.1109/CVPR52734.2025.02544. Lu, S., Schuff, H., and Gurevych, I. How are prompts differ- ent in terms of sensitivity? In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), p. 5833–5856, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.325. URL https://aclanthology.org/2024.naacl-l ong.325/. Lundberg, S. M. and Lee, S.-I. A unified approach to inter- preting model predictions. Advances in neural informa- tion processing systems, 30, 2017. Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Mor- rison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., et al. OLMoE: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024. OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D., Brahman, F., Clark, C., Dasigi, P., Dziri, N., Ettinger, A., Guerquin, M., Heineman, D., Ivison, H., Koh, P. W., Liu, J., Malik, S., Merrill, W., Miranda, L. J. V., Morrison, J., Murray, T., Nam, C., Poznanski, J., Pyatkin, V., Rangapur, A., Schmitz, M., Skjonsberg, S., Wadden, D., Wilhelm, C., Wilson, M., Zettlemoyer, L., Farhadi, A., Smith, N. A., and Hajishirzi, H. 2 olmo 2 furious, 2025. URLhttps://arxiv.org/abs/25 01.00656. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. 11 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Qu, X., Dong, D., Hu, X., Zhu, T., Sun, W., and Cheng, Y. Llama-MoE v2: Exploring sparsity of Llama from per- spective of mixture-of-experts with post-training. arXiv preprint arXiv:2411.15708, 2024. Razavi, A., Soltangheis, M., Arabzadeh, N., Salamat, S., Zihayat, M., and Bagheri, E. Benchmarking prompt sensi- tivity in large language models. In European Conference on Information Retrieval, p. 303–313. Springer, 2025. Ren, J., Li, M., Chen, Q., Deng, H., and Zhang, Q. Defin- ing and quantifying the emergence of sparse concepts in DNNs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 20280– 20289, 2023a. Ren, J., Zheng, X., Liu, J., Lizarraga, A., Wu, Y. N., Lin, L., and Zhang, Q. Monitoring primitive interactions during the training of DNNs. Proceedings of the AAAI Con- ference on Artificial Intelligence, 39(19):20183–20191, Apr. 2025. doi: 10.1609/aaai.v39i19.34223. URL https://ojs.aaai.org/index.php/AAAI/ article/view/34223. Ren, Q., Deng, H., Chen, Y., Lou, S., and Zhang, Q. Bayesian neural networks avoid encoding complex and perturbation-sensitive concepts. In International Confer- ence on Machine Learning, p. 28889–28913. PMLR, 2023b. Ren, Q., Gao, J., Shen, W., and Zhang, Q. Where we have arrived in proving the emergence of sparse interaction primitives in DNNs. In The Twelfth International Confer- ence on Learning Representations, 2024a. URLhttps: //openreview.net/forum?id=3pWSL8My6B. Ren, Q., Zhang, J., Xu, Y., Xin, Y., Liu, D., and Zhang, Q. Towards the dynamics of a DNN learning symbolic interactions. Advances in Neural Information Processing Systems, 37:50653–50688, 2024b. Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. Quantify- ing language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Confer- ence on Learning Representations, 2024. URLhttps: //openreview.net/forum?id=RIu5lyNXjT. Shen, W., Wei, Z., Ren, Q., Zhang, B., Huang, S., Fan, J., and Zhang, Q. Interpretable rotation-equivariant quater- nion neural networks for 3D point cloud processing. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 46(5):3290–3304, 2024. doi: 10.1109/TPAMI.20 23.3346383. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023. Sturmfels, P., Lundberg, S., and Lee, S.-I. Visualizing the impact of feature attribution baselines. Distill, 5(1):e22, 2020. Sun, J., Shaib, C., and Wallace, B. C. Evaluating the zero- shot robustness of instruction-tuned language models. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview .net/forum?id=g9diuvxN6D. Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribu- tion for deep networks. In International conference on machine learning, p. 3319–3328. PMLR, 2017. Tenney, I., Wexler, J., Bastings, J., Bolukbasi, T., Coenen, A., Gehrmann, S., Jiang, E., Pushkarna, M., Radebaugh, C., Reif, E., et al. The language interpretability tool: Ex- tensible, interactive visualizations and analysis for nlp models. In Proceedings of the 2020 conference on em- pirical methods in natural language processing: system demonstrations, p. 107–118, 2020. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Wang, Y., Yao, Q., Kwok, J. T., and Ni, L. M. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1–34, 2020. Wen, L., Zheng, L., Li, H., Sun, L., Wei, Z., and Shen, W. Interpreting arithmetic reasoning in large language mod- els using game-theoretic interactions. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URLhttps://openreview.net /forum?id=tRvzEL64dY. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo, Z., and Fan, Z. Qwen2 technical report, 2024. URL https://arxiv.org/abs/2407.10671. 12 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Zhang, Q., Wang, X., Cao, R., Wu, Y. N., Shi, F., and Zhu, S.-C. Extraction of an explanatory graph to interpret a CNN. IEEE transactions on pattern analysis and machine intelligence, 43(11):3863–3877, 2020. Zhou, H., Zhang, H., Deng, H., Liu, D., Shen, W., Chan, S.-H., and Zhang, Q. Explaining generalization power of a DNN using interactive concepts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p. 17105–17113, 2024. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., and Xie, X. Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts. In Pro- ceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis, LAMPS ’24, p. 57–68, New York, NY, USA, 2024a. Associa- tion for Computing Machinery. ISBN 9798400712098. doi: 1 0 . 1 1 4 5 / 3 6 8 9 2 1 7 . 3 6 9 0 6 2 1.URLhttps: //doi.org/10.1145/3689217.3690621. Zhu, T., Qu, X., Dong, D., Ruan, J., Tong, J., He, C., and Cheng, Y. Llama-MoE: Building mixture-of-experts from Llama with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 15913–15923, 2024b. Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., and Chen, K. ProSA: Assessing and understanding the prompt sen- sitivity of LLMs. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2024, p. 1950–1976, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findi ngs-emnlp.108. URLhttps://aclanthology.o rg/2024.findings-emnlp.108/. 13 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions A. Masking Strategies of Input Variables In attribution method research, it is common to employ a specific token or embedding to mask the input variables of a deep neural network (DNN) (Lundberg & Lee, 2017; Ancona et al., 2019; Fong et al., 2019) and use changes in network outputs on the masked samples to estimate attributions of different input variables. The selection of a masking approach is complex, as each method has its weakness. For example, replacing input variables with the mean baseline value (the average of all samples) or the zero baseline value can introduce out-of-distribution signals, thereby providing the model with artificial information, such as uniform grey or black dots in an image (Dabkowski & Gal, 2017; Ancona et al., 2019; Sundararajan et al., 2017). Additionally, blurring image pixels using a Gaussian kernel (Fong & Vedaldi, 2017; Fong et al., 2019) as the masked state removes high-frequency signals but fails to eliminate low-frequency signals (Covert et al., 2021; Sturmfels et al., 2020). Given these challenges, we adopt a token replacement strategy, which is standard for the text domain. This involves substituting the target input word with a dedicated[MASK]token at the embedding level. For example, to mask the word ”green” in the input ”He is a green hand,” we would provide the LLM with the modified input ”He is a [MASK] hand.” This approach effectively nullifies the specific semantic contribution of the target word without introducing out-of-distribution artifacts, ensuring a clean and consistent baseline for our interaction analysis. For the specific[MASK]token for each LLM, please refer to Section 5 for details. B. Proof of Theorem B.1. Proof of Universal Matching Property In the main body of the paper, for the sake of simplicity and clarity, we introduced the Universal Matching Property (Theorem 1) primarily through the lens of AND interactions. However, our empirical analysis and the underlying theoretical framework are built upon a more comprehensive AND-OR interaction framework. This extended framework, which incorporates both AND and OR interaction patterns, also adheres to the Universal Matching Property. In this section, we provide the formal proof for the Universal Matching Property of the complete AND-OR interaction framework. This proof is more general and naturally subsumes the proof for the AND interaction framework presented as Theorem 1 in the main text. We will demonstrate that the output of the surrogate logical model, which is the sum of all AND-OR interaction effects, can perfectly match the output of the Deep Neural Network (DNN) for any masked sample. The surrogate logical model φ(·) is defined as follows: φ(x T )≜ φ(x ∅ ) + X S⊆N,S̸=∅ 1 AND (S |x T )· I AND S + X S⊆N,S̸=∅ 1 OR (S |x T )· I OR S , (4) where the AND trigger function1 AND (S |x T ) ∈ 0, 1represents an AND relationship between input variables inS, which can also be termed AND interaction pattern; the OR trigger function1 OR (S |x T ) ∈ 0, 1represents an OR relationship between input variables inS, which can also be termed OR interaction pattern. The scalar weightI AND S quantifies the effect of an AND relationship, which can also be termed AND interaction effect; the scalar weightI OR S quantifies the effect of an OR relationship, which can also be termed OR interaction effect. An AND relationship is activated only by the joint presence of all input variables in the setS, i.e., all input variables inSare not masked. For instance, given the input sentencex = “He is a green hand,”the co-occurrence of the input variables in the setS =green, hand contributes a numerical effectI AND S that pushes the surrogate logical model’s inference towards the semantic meaning of“beginner.”If an AND interactionSis triggered, i.e.,1 AND (S |x T ) = 1, the corresponding interaction effectI AND S is added to the output of the logical model. Otherwise, if any word inSis masked and the AND interaction is not triggered, i.e.,1 AND (S |x T ) = 0, the interaction effectI AND S is not added to the output of the logical model. An OR relationship is activated by the presence of any of all input variables in the setS, i.e., any input variables inSare not masked. For instance, given the input sentencex = “The service was terrible and the food was awful,”the presence of any input variables in the setS =terrible, awfulcontributes a numerical effectI OR S that pushes the surrogate logical model’s inference towards a negative sentiment classification. If an OR interactionSis triggered, i.e.,1 OR (S |x T ) = 1, the corresponding interaction effectI OR S is added to the output of the logical model. Otherwise, if all words inSare masked and the OR interaction is not triggered, i.e.,1 OR (S |x T ) = 0, the interaction effectI OR S is not added to the output of the logical model.x ∅ represents that all input variables in N are masked. Definition of universal matching property for AND-OR interactions. When the scalar weights in the surrogate logical 14 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions modelφ(·)are set toI AND S = P T⊆S (−1) |S|−|T| v and (x T )andI OR S = − P T⊆S (−1) |S|−|T| v or (x N ), the output ofφ(·) can always match the output score of the DNN v(·), i.e.,∀T ⊆ N,v(x T ) = φ(x T ). Here v and (x T ) + v or (x T ) = v(x T ). We need to prove that given an input samplex, for each masked samplex T |T ⊆ N, the network output scorev(x T )∈ R can be well matched by the surrogate logical modelφ(x T ). The surrogate logical modelφ(x T )uses the sum of AND interactions and OR interactions to accurately explain/match the network output score v(x T ). ∀T ⊆ N,v(x T ) = φ(x T ). φ(x T ) = φ(x ∅ ) + X S⊆N,S̸=∅ 1 AND (S |x T )· I AND S + X S⊆N,S̸=∅ 1 OR (S |x T )· I OR S , = v(x ∅ ) + X S⊆T,S̸=∅ I AND S |z v and (x T ) + X S⊆N,S∩T̸=∅ I OR S | z v or (x T ) (5) Proof.(1) Universal matching property of AND interactions. For all2 n masked samplesx T | T ⊆ N, what we need to prove is that the outputv and (x T )of a DNN can be universally explained by all the interactions inT ⊆ N, i.e., ∀S ⊆ T,S ̸=∅,v and (x T ) = P S⊆T,S̸=∅ I AND S (x) = v(x ∅ ) + P S⊆T,S̸=∅ I AND S . Here, v(x ∅ ) = v and (x ∅ ). According to the definition of the AND interaction,I AND S (x) = P L⊆S (−1) |S|−|L| v and (x L ) . To simplify the computation of the sum of AND interactions P S⊆T,S̸=∅ I AND S (x) = P S⊆T,S̸=∅ P L⊆S (−1) |S|−|L| v and (x L ) , we exchange the order of summation of the setL⊆ S ⊆ Tand the setS ⊇ L. Given a set of input variablesL, we compute all linear combinations of all setsScontainingLwith respect to the model outputsv and (x S ), i.e., P S:L⊆S⊆T (−1) |S|−|L| v and (x L ) . Then, we compute all summations over the setL⊆ Tas P S⊆T,S̸=∅ I AND S (x) = P L⊆T P S:L⊆S⊆T (−1) |S|−|L| v and (x L ) . Then, we can compute different cases of L⊆ S ⊆ T as follows: (1) When L = T = S, P S:L⊆S⊆T (−1) |S|−|L| v and (x L ) = (−1) |T|−|T| v and (x L ) = v and (x L ). (2) WhenL ⊆ S ⊆ T,L ̸= T, let us consider the linear combinations of all setsSwith number|S|for the model outputv and (x L ), respectively. Letm :=|S|−|L|, (0≤ m≤|T|−|L|), then there are a total ofC m |T|−|L| combinations of all setsSof order|S|. GivenL, accumulating the model outputsv and (x L )corresponding to allS ⊇ L, we can get P S:L⊆S⊆T (−1) |S|−|L| v and (x L ) = v and (x L )· X |T|−|L| m=0 C m |T|−|L| (−1) m | z =0 = 0. Considering all the cases, the complete derivation of the sum of AND interactions is as follows. X S⊆T,S̸=∅ I AND S = X S⊆T,S̸=∅ X L⊆S (−1) |S|−|L| v and (x L ) = X L⊆T X S:L⊆S⊆T (−1) |S|−|L| v and (x L )− v and (x ∅ ) =v and (x T ) | z L=T + X L⊆T,L̸=T v and (x L )· X |T|−|L| m=0 C m |T|−|L| (−1) m | z =0 −v and (x ∅ ) =v and (x T )− v(x ∅ ) (6) Therefore, we have proven that∀∅̸= T ⊆ N,v and (x T ) = v(x ∅ ) + P S⊆T,S̸=∅ I AND S . (2) Universal matching theorem of OR interactions.What we need to prove is that∀T ⊆ N,v or (x T ) = P S∈S:S∩T̸=∅∪∅ I OR S = P S:S∩T̸=∅ I OR S . Here I OR ∅ = v or (x ∅ ) = 0. According to the definition of the OR interaction,I OR S := − P L⊆S (−1) |S|−|L| v or (x N ). To simplify the computa- tion of the sum of OR interactions P S:S∩T̸=∅ I OR S = P S:S∩T̸=∅ h − P L⊆S (−1) |S|−|L| v or (x N ) i , we also exchange the order of summation of the setL ⊆ S ⊆ Nand the setS : S ∩ T = ∅. Given a set of input variablesL, we compute all linear combinations of all setsScontainingLwith respect to the model outputsv or (x N ), i.e., 15 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions P S:S∩T̸=∅,N⊇S⊇L (−1) |S|−|L| v or (x N ). Then, we compute all summations over the setL ⊆ Nas P S:S∩T̸=∅ I OR S = − P L⊆N P S:S∩T̸=∅,N⊇S⊇L (−1) |S|−|L| v or (x N ). Then, we can compute different cases ofL⊆ S ⊆ N,S∩ T ̸=∅as follows: (1) WhenL = N(thenS = N), P S:S∩T̸=∅,S⊇L (−1) |S|−|L| v or (x N ) = (−1) |N|−|N| v or (x ∅ ) = v or (x ∅ ) = 0, Here I OR ∅ = v or (x ∅ ) = 0. (2) WhenL = N \ T, for all setsS : S ⊇ L,S∩ T ̸=∅(thenS ̸= N \ T,S ̸= L), let us consider the linear combinations of all setsSwith number|S|for the model outputv or (x T ), respectively. Let|S ′ | := |S| − |L|, (1 ≤ |S ′ | ≤ |T|), then there are a total ofC |S ′ | |T| combinations of all setsSof order|S|. Thus, P S:S∩T̸=∅,S⊇L (−1) |S|−|L| v or (x N ) = v or (x T )· X |T| |S ′ |=1 C |S ′ | |T| (−1) |S ′ | |z =−1 =−v or (x T ). (3) WhenL∩ T ̸= ∅,L ̸= N, for all setsS : S ⊇ L,S ∩ T ̸= ∅, let us consider the linear combinations of all sets Swith number|S|for the model outputv or (x T ), respectively. Let us split|S|−|L|into|S ′ |and|S ′ |, i.e.,|S|−|L| = |S ′ | +|S ′ | , whereS ′ = i|i ∈ S,i /∈ L,i ∈ N \ T,S ′ = i|i ∈ S,i /∈ L,i ∈ T(then0 ≤ |S ′ | ≤ |T|−|T ∩ L|) andS ′ + S ′ + L = S. Thus, there are a total ofC |S ′ | |T|−|T∩L| combinations of all setsS ′ of order|S ′ |. Thus, P S:S∩T̸=∅,S⊇L (−1) |S|−|L| v or (x N ) = v or (x N )· P S ′ ⊆N X |T|−|T∩L| |S ′ |=0 C |S ′ | |T|−|T∩L| (−1) |S ′ |+|S ′ | | z =0 = 0. (4) WhenL∩ T = ∅,L ̸= N \ T, let us split|S|−|L|into|S ′ |and|S ′ |, i.e.,|S|−|L| = |S ′ | +|S ′ |, whereS ′ = i|i ∈ S,i /∈ L,i ∈ N \ T,S ′ = i|i ∈ S,i ∈ T(then0 ≤ |S ′ | ≤ |T|) andS ′ + S ′ + L = S. Thus, there are a total ofC |S ′ | |T| combinations of all setsS ′ of order|S ′ |. Thus, P S:S∩T̸=∅,S⊇L (−1) |S|−|L| v or (x N ) = v or (x N )· P S ′ ⊆N X |T| |S ′ |=0 C |S ′ | |T| (−1) |S ′ |+|S ′ | |z =0 = 0. Considering all the cases, the complete derivation of the sum of OR interactions is as follows. X S:S∩T̸=∅ I OR S = X S:S∩T̸=∅ h − X L⊆S (−1) |S|−|L| v or (x N ) i =− X L⊆N X S:S∩T̸=∅,N⊇S⊇L (−1) |S|−|L| v or (x N ) =− |T| X |S ′ |=1 C |S ′ | |T| (−1) |S ′ | · v or (x T ) | z L=N −v or (x ∅ ) |z L=N − X L∩T̸=∅,L̸=N X S ′ ⊆N |T|−|T∩L| X |S ′ |=0 C |S ′ | |T|−|T∩L| (−1) |S ′ |+|S ′ | · v or (x N ) − X L∩T =∅,L̸=N X S ′ ⊆N |T| X |S ′ |=0 C |S ′ | |T| (−1) |S ′ |+|S ′ | · v or (x N ) =−(−1)· v or (x T )− v or (x ∅ )− X L∩T̸=∅,L̸=N X S ′ ⊆N 0 · v or (x N ) − X L∩T =∅,L̸=N X S ′ ⊆N 0 · v or (x N ) = v or (x T )− v or (x ∅ ) = v or (x T ) (7) 16 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Therefore, we have proven that∀T ⊆ N,v or (x T ) = P S:S∩T̸=∅ I OR S . (3) Universal matching theorem of AND-OR interactions. With the universal matching property of AND interactions and the universal matching property of OR interactions, we can easily getv(x T ) = φ(x T ) = v and (x T ) + v or (x T ) = v(x ∅ ) + P S⊆T,S̸=∅ I AND S + P S⊆N,S∩T̸=∅ I OR S , thus, we obtain the universal matching property of AND-OR interactions. B.2. Proof of Sparsity Property Given all the masked samplesx T | T ⊆ N, the surrogate logical modelφ(x T )only utilizes a small set of salient AND interactions inΩ AND and salient OR interactions inΩ OR to approximate the network output scorev(x T ). That is, the network’s output can be well approximated by a small set of AND-OR interactions. v(x T ) =φ(x T )≈v(x ∅ ) + X S⊆T,S̸=∅,S∈Ω AND I AND S + X S⊆T,S̸=∅,S∈Ω OR I OR S (8) Proof. It has been proven by Ren et al. (2024a) that under three common conditions 7 , the output scorev and (x T )of a well- trained DNN on all2 n masked samplesx T |T ⊆ Ncould be universally estimated by a small number of AND interactions T ∈ Ω AND with salient interaction effectsI AND S , s.t.,|Ω AND | ≪ 2 n , i.e.,∀T ⊆ N,v and (x T ) = P S⊆T,S̸=∅ I AND S ≈ P S⊆T,S̸=∅,S∈Ω AND I AND S . According to Eq. (6),v and (x T ) = v(x ∅ ) + P S⊆T,S̸=∅ I AND S . Therefore,v and (x T ) ≈ v(x ∅ ) + P S⊆T,S̸=∅,S∈Ω AND I AND S . Besides, as proven in Section C, the OR interaction can be considered as a special AND interaction. Thus, the confidence scorev or (x T )of a well-trained DNN on all2 n masked samplesx T |T ⊆ Ncould be universally estimated by a small number of OR interactionsT ∈ Ω OR with salient interaction effectsI OR S , s.t.,|Ω OR | ≪ 2 n . Similarly,v or (x T ) = P S⊆T,S̸=∅ I OR S ≈ P S⊆T,S̸=∅,S∈Ω OR I OR S Thus, for each randomly masked samplex T ,T ⊆ N, the surrogate logical modelφ(x T )can use a small number of salient AND-OR interactions to approximate the network output scorev(x T ), i.e.,v(x T ) = φ(x T ) = v and (x T ) + v or (x T )≈ (x ∅ ) + P S⊆T,S̸=∅,S∈Ω AND I AND S + P S⊆T,S̸=∅,S∈Ω OR I OR S . C. OR Interactions Can Be Considered as Special AND Interactions If we reverse the definition of the masked state and the unmasked state of the input variable, the OR interactionI OR S can be considered as a special kind of AND interaction I AND S . Given an input samplex∈ R n and the output score of a DNN asv(·), if we randomly mask input variables inx, we can get all2 n masked samples. Letx S denote the certain masked input sample when input variables inN \ Sare all masked and input variables in S are kept unchanged. (x S ) i = ( x i , i∈ S b i , i∈ N \ S (9) whereb∈ R n are baseline values to represent the masked state of input variables. If we reverse the definition of the masked state and the unmasked state of an input variable, i.e., we considerbas the input sample and considerx as the masked state, then the masked sample e x S can be defined as follows. ( e x S ) i = ( b i , i∈ S x i , i∈ N \ S (10) Thus, we can getx N = e x S . To simplify the analysis, let us assumev and (x S ) = v or (x S ) = 0.5v(x S ), then the OR 7 Here are the three conditions: (1) The DNN doesn’t encode extremely high-order AND interactions. (2) The DNN performs effectively on masked samples and exhibits greater confidence as the input sample is less masked. (3) When we increase the number of masked input variables, the confidence of the DNN does not drop significantly. 17 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions interaction I OR S can be regarded as a specific AND interaction I AND S ( e x) as follows. I OR S (x) =− X T⊆S (−1) |S|−|T| v or (x N ), =− X T⊆S (−1) |S|−|T| v or ( e x T ), =− X T⊆S (−1) |S|−|T| v and ( e x T ), =−I AND S ( e x). (11) Now we have proven that OR interactions can be considered as special AND interactions. D. Details of Extracting the Sparsest AND-OR Interactions We follow Li & Zhang (2023) to extract AND-OR interactions. Given a masked samplex T , the output score of the network v(x T )can be decomposed into a combination of AND interaction and OR interaction, i.e.,v(x T ) = v and (x T ) + v or (x T ). Specifically,v and (x T ) = 0.5· v(x T ) + γ T andv or (x T ) = 0.5· v(x T )− γ T , whereγ T | T ⊆ Nis a set of learnable parameters. The parametersγ T were trained through minimizing the following LASSO-like loss to obtain sparse interactions: min γ T X S⊆N |I AND S (x)| +|I OR S (x)|,(12) whereI AND S (x)= P T⊆S (−1) |S|−|T| v and (x T )= P T⊆S (−1) |S|−|T| (0.5 · v(x T ) + γ T ) andI OR S (x)= − P T⊆S (−1) |S|−|T| v or (x N ) = − P T⊆S (−1) |S|−|T| (0.5 · v(x T ) − γ T ) . Thus, we can extract the sparsest set of AND-OR interactions. E. Experimental Details E.1. Computing Infrastructure We conducted all our experiments on four NVIDIA Tesla V100-DGXS GPUs, each with 32 GB of VRAM. The software environment consisted of NVIDIA Driver version 570.133.07 and CUDA 12.8. For all of the evaluated LLMs, we used a torch.float16 data type, which provides a standard level of precision for inference tasks. E.2. Model Details We conduct experiments on 50 open-source LLMs from 6 major model families. A comprehensive list of all evaluated models is provided in Table 1. To facilitate a controlled analysis of the factors influencing prompt sensitivity, we group these models into specific subsets for each comparison, as detailed below. (1) Instruct/Chat vs. Base Models. To investigate the impact of the alignment process, we form pairs of instruct/chat models and their corresponding base models. This comparison includes models from the Llama, Mistral, Qwen, InternLM, and Olmo families. The main LLMs used for this comparison are: • Llama Family: – llama-2-7b-chat vs. llama-2-7b – llama-2-13b-chat vs. llama-2-13b – llama-2-70b-chat vs. llama-2-70b – llama-3-8b-instruct vs. llama-3-8b • Mistral Family: – mistral-7b-v0.3-instruct vs. mistral-7b-v0.3 – mixtral-8x7b-instruct vs. mixtral-8x7b • Qwen Family: 18 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Table 1. A comprehensive list and characteristics of LLMs, grouped by model family and model series. Model FamilyModel SeriesModelTypeArchitectureScale Llama (10 models) Llama 2 (6 models) Llama-2-7bBaseDense7B Llama-2-7b-chatChatDense7B Llama-2-13bBaseDense13B Llama-2-13b-chatChatDense13B Llama-2-70bBaseDense70B Llama-2-70b-chatChatDense70B Llama 3 (2 models) Llama-3-8bBaseDense8B Llama-3-8b-instructInstructDense8B Llama MoE (2 models) Llama-moe-v1-35b-28-sft InstructMoE3.5B (Activated) Llama-moe-v2-38b-28-sft InstructMoE3.8B (Activated) Mistral (4 models) Mistral (2 models) Mistral-7b-v0.3BaseDense7B Mistral-7b-v0.3-instructInstructDense7B Mixtral (2 models) Mixtral-8x7bBaseMoE13B (Activated) Mixtral-8x7b-instructInstructMoE13B (Activated) Qwen (25 models) Qwen 1.5 MoE (2 models) Qwen1.5-moe-a2.7b-chatChatMoE2.7B (Activated) Qwen1.5-moe-a2.7bBaseMoE2.7B (Activated) Qwen 2 (5 models) Qwen2-7bBaseDense7B Qwen2-0.5b-instructInstructDense0.5B Qwen2-1.5b-instructInstructDense1.5B Qwen2-7b-instructInstructDense7B Qwen2-72b-instructInstructDense72B Qwen 2.5 (7 models) Qwen2.5-0.5b-instructInstructDense0.5B Qwen2.5-1.5b-instructInstructDense1.5B Qwen2.5-3b-instructInstructDense3B Qwen2.5-7b-instructInstructDense7B Qwen2.5-14b-instructInstructDense14B Qwen2.5-32b-instructInstructDense32B Qwen2.5-72b-instructInstructDense72B Qwen 3 (9 models) Qwen3-0.6b-instructInstructDense0.6B Qwen3-1.7b-instructInstructDense1.7B Qwen3-4bBaseDense4B Qwen3-4b-instructInstructDense4B Qwen3-8bBaseDense8B Qwen3-8b-instructInstructDense8B Qwen3-14bBaseDense14B Qwen3-14b-instructInstructDense14B Qwen3-32b-instructInstructDense32B Qwen 3 A3B (2 models) Qwen3-30b-a3b-instructInstructMoE3B (Activated) Qwen3-30b-a3bBaseMoE3B (Activated) Olmo (9 models) Olmo v1 (3 models) Olmo-1bBaseDense1B Olmo-7bBaseDense7B Olmo-7b-instructInstructDense7B Olmo v2 (4 models) Olmo-2-1b-instructInstructDense1B Olmo-2-7b-instructInstructDense7B Olmo-2-13b-instructInstructDense13B Olmo-2-32b-instructInstructDense32B OlmoE (2 models) Olmoe-7bBaseMoE1B (Activated) Olmoe-7b-instructInstructMoE1B (Activated) InternLM (2 models) InternLM 2 (2 models) Internlm2-7bBaseDense7B Internlm2-chat-7bChatDense7B 19 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions – qwen3-4b-instruct vs. qwen3-4b – qwen3-8b-instruct vs. qwen3-8b – qwen3-14b-instruct vs. qwen3-14b – qwen1.5-moe-a2.7b-chat vs. qwen1.5-moe-a2.7b – qwen3-30b-a3b-instruct vs. qwen3-30b-a3b • Olmo Family: – olmo-7b-instruct vs. olmo-7b – olmoe-7b-instruct vs. olmoe-7b • InternLM Family: – internlm2-chat-7b vs. internlm2-7b (2) Dense vs. MoE Models. To analyze the effect of architecture, we compare dense and Mixture-of-Experts (MoE) models, primarily within the same model family to control for other variables. The main LLMs used for this comparison are: • Llama Family: – Dense:llama-2-7b,llama-2-7b-chat,llama-2-13b,llama-2-13b-chat,llama-2-70b, llama-2-70b-chat, llama-3-8b, llama-3-8b-instruct. – MoE: llama-moe-v1-35b-28-sft, llama-moe-v2-38b-28-sft. • Mistral Family: – Dense: mistral-7b-v0.3, mistral-7b-v0.3-instruct. – MoE: mixtral-8x7b, mixtral-8x7b-instruct. • Qwen Family: – Dense:qwen2-7b,qwen2-7b-instruct,qwen2-72b-instruct,qwen2.5-7b-instruct, qwen2.5-14b-instruct,qwen2.5-32b-instruct,qwen2.5-72b-instruct,qwen3-4b, qwen3-4b-instruct,qwen3-8b,qwen3-8b-instruct,qwen3-14b,qwen3-14b-instruct, qwen3-32b-instruct. –MoE:qwen1.5-moe-a2.7b,qwen1.5-moe-a2.7b-chat,qwen3-30b-a3b, qwen3-30b-a3b-instruct. • Olmo Family: – Dense:olmo-7b,olmo-7b-instruct,olmo-2-7b-instruct,olmo-2-13b-instruct, olmo-2-32b-instruct. – MoE: olmoe-7b, olmoe-7b-instruct. (3) Model Scale. To study the impact of model scale, we analyze a series of LLMs from the same family and with the same training paradigm but with varying parameter counts. The main LLMs used for this comparison are: • Llama-2 (Base): 7b, 13b, 70b. • Llama-2 (Chat): 7b, 13b, 70b. • Qwen2 (Instruct): 0.5b, 1.5b, 7b, 72b. • Qwen2.5 (Instruct): 0.5b, 1.5b, 3b, 7b, 14b, 32b, 72b. • Qwen3 (Instruct): 0.6b, 1.7b, 4b, 8b, 14b, 32b. • Olmo-2 (Instruct): 1b, 7b, 13b, 32b. 20 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions E.3. Generation Configuration of LLMs To ensure reproducible results, we employ a greedy search strategy for all LLMs. This is achieved by setting the “dosample” parameter to “False” in our generation configuration. When “dosample=False”, the LLM selects the token with the highest probability as the next token in the sequence. By adopting this greedy approach, we eliminate the randomness inherent in sampling-based methods. The configuration ensures that for a given input, the same LLM will generate the exact same output every time, which is a critical requirement for the replicability of our experiments. E.4. How to Mask Input Words For Different LLMs To compute interactions, we follow the approach of Cheng et al. (2025) and mask the words inN \ Sby replacing them with a LLM-specific[MASK]token. Our selection of this token follows a prioritized strategy: (1) We preferentially use the LLM’s designated unknown (<unk>) token. (2) If an unknown token is not available or suitable, we use the padding (<pad>) token as a fallback. Since the specific token strings and their corresponding IDs vary across different LLMs, the exact mask token used for each LLM is detailed below: •Forllama-2-7b,llama-2-7b-chat,llama-moe-v1-3.5b-2.8-sft,mistral-7b-v0.3, mistral-7b-v0.3-instruct,mixtral-8x7b,mixtral-8x7b-instruct,internlm2-7b, internlm2-chat-7b, we use the <unk> token (ID: 0) to mask words. • Forllama-2-13b,llama-2-13b-chat,llama-2-70b,llama-2-70b-chat,weusethe <|padtoken|> token (ID: 0) to mask words. •Forllama-3-8b,llama-3-8b-instruct, we use the<|padtoken|>/<|reservedspecialtoken250|> token (ID: 128255) to mask words. •Forllama-moe-v2-3.8b-2.8-sft, we use the<|padtoken|>/<|eotid|>token (ID:128009) to mask words. • For qwen2-7b, we use the <|PADTOKEN|> token (ID: 151646) to mask words. •For a large group of Qwen models, includingqwen2-0.5b-instruct,qwen2-1.5b-instruct, qwen2-7b-instruct,qwen2-72b-instruct,qwen2.5series,qwen3-0.6b-instruct, qwen3-1.7b-instruct,qwen3-4bseries,qwen3-32b-instruct,qwen1.5-moeseries,and qwen3-30b-a3bseries, we use the<|padtoken|>/<|endoftext|>token (ID:151643) to mask words. •Forqwen3-8b,qwen3-8b-instruct,qwen3-14b,qwen3-14b-instruct,weusethe <|padtoken|>/<|visionpad|> token (ID: 151654) to mask words. •For the Olmo V1 series models, includingolmo-1b,olmo-7b,olmo-7b-instruct,olmoe-7b, and olmoe-7b-instruct, we use the <|padding|> token (ID: 1) to mask words. •FortheOlmoV2seriesmodels,includingolmo-2-1b-instruct,olmo-2-7b-instruct, olmo-2-13b-instruct, andolmo-2-32b-instruct, we use the<|padtoken|>/<|endoftext|> token (ID: 100257) to mask words. E.5. Prompt Templates To systematically evaluate the prompt sensitivity of LLMs, we designed a set of five distinct prompt templates. As illustrated in Figure 15, these templates are derived from a base prompt template (i.e., Prompt Template 1) through a series of subtle, semantically irrelevant modifications. These variations include changes in letter case,e.g., “Answers” vs. “ANSWERS” and alterations to separators,e.g., “:” vs. “::” or the format of option markers,e.g., “A.” vs. “A)”. Crucially, these changes only affect the superficial formatting while preserving the core semantic meaning of the prompt template. In our experimental procedure, for a given input, which consists of a question and options, we apply each of the five prompt templates to generate five prompts. For every unique pair of these five prompts, we then calculate the prompt sensitivity by quantifying the change in the interactions among the input variables (i.e., words within the question and options). This procedure allows us to precisely measure how much the LLM’s interaction patterns of the core input are perturbed by superficial changes in the prompt template, thus evaluating the prompt sensitivity of LLMs. 21 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Question Answers: A.Option A B.Option B C.Option C D.Option D Answer: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟏ: Question ANSWERS: A.Option A B.Option B C.Option C D.Option D ANSWER: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟐ: Question Answers:: A.Option A B.Option B C.Option C D.Option D Answer:: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟑ: Question ANSWERS:: A.Option A B.Option B C.Option C D.Option D ANSWER:: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟒ: Question Answers: A)Option A B)Option B C)Option C D)Option D Answer: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟓ: Figure 15. Different prompt templates. Red parts show the difference between the current prompt template with the first prompt template, i.e., Prompt Template 1. E.6. Few-shot Learning Templates For this experiment, we selected the pair of prompt templates that exhibited the highest average prompt sensitivity in the 0-shot setting, aiming to test if few-shot learning could help the most severe situation. To investigate whether few-shot learning can mitigate high prompt sensitivity, we conducted a follow-up experiment. We selected Prompt Template 1 and Prompt Template 4 from Figure 15 for this analysis, as this pair exhibited the highest average prompt sensitivity in our 0-shot setting. This allowed us to test the efficacy of few-shot learning in the most challenging scenario. Based on these two base templates, we constructed few-shot learning prompts with one, two, and three in-context examples (i.e., 1-shot, 2-shot, and 3-shot learning), as illustrated in Figure 16. The examples were formulated using certain questions and their corresponding answers, randomly selected from a set of datasets that are not included in the test set. The structure of each example is related to its corresponding prompt template. For instance, the first example (Example 1) is formatted differently for each template: • Example 1 For Prompt Template 1: Question: Which type of precipitation consists of frozen rain drops? Answers: A.sleet B.hail C.snow D.fog Answer: A • Example 1 For Prompt Template 4 (Note the different format): Question: Which type of precipitation consists of frozen rain drops? ANSWERS:: A.sleet B.hail C.snow D.fog ANSWER:: A The other two examples (Example 2 and Example 3) are presented below: • Example 2 For Prompt Template 1: Question: Decayed prehistoric plants have helped in the formation of Answers: A.coal, shale, and quartz. B.coal, oil, and gas. C.shale, quartz, and coal. D.oil, shale, and granite. Answer: B • Example 2 For Prompt Template 4: 22 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Question: Decayed prehistoric plants have helped in the formation of ANSWERS:: A.coal, shale, and quartz. B.coal, oil, and gas. C.shale, quartz, and coal. D.oil, shale, and granite. ANSWER:: B • Example 3 For Prompt Template 1: Question: Which describes a material that is not a food? Answers: A.It stores energy but not nutrients. B.It does not store energy or nutrients. C.It stores energy and nutrients. D.It does not store energy but stores nutrients. Answer: B • Example 3 For Prompt Template 4: Question: Which describes a material that is not a food? ANSWERS:: A.It stores energy but not nutrients. B.It does not store energy or nutrients. C.It stores energy and nutrients. D.It does not store energy but stores nutrients. ANSWER:: B 23 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Instruction: Read the following question and the four options provided. Choose the single best answer and provide only its corresponding letter. Example: Example 1 Question: Question Answers: A.Option A B.Option B C.Option C D.Option D Answer: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟏ: Instruction: Read the following question and the four options provided. Choose the single best answer and provide only its corresponding letter. Example: Example 1 Question: Question ANSWERS:: A.Option A B.Option B C.Option C D.Option D ANSWER:: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟐ: 1-shot Learning Instruction: Read the following question and the four options provided. Choose the single best answer and provide only its corresponding letter. Example: Example 1 Example 2 Question: Question Answers: A.Option A B.Option B C.Option C D.Option D Answer: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟏ: Instruction: Read the following question and the four options provided. Choose the single best answer and provide only its corresponding letter. Example: Example 1 Example 2 Question: Question ANSWERS:: A.Option A B.Option B C.Option C D.Option D ANSWER:: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟐ: 2-shot Learning Instruction: Read the following question and the four options provided. Choose the single best answer and provide only its corresponding letter. Example: Example 1 Example 2 Example 3 Question: Question Answers: A.Option A B.Option B C.Option C D.Option D Answer: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟏ: Instruction: Read the following question and the four options provided. Choose the single best answer and provide only its corresponding letter. Example: Example 1 Example 2 Example 3 Question: Question ANSWERS:: A.Option A B.Option B C.Option C D.Option D ANSWER:: 푷풓풐풎풑풕푻풆풎풑풍풂풕풆ퟐ: 3-shot Learning Figure 16. Prompt templates of few-shot learning. 24 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions F. More Experimental Results F.1. More Results on the Verification of the Sparsity of Interactions Here are more results on the verification of the sparsity of interactions. As illustrated in Figure 17, the results verify that only a small set of interactions have salient effects, while most of the interactions have negligible effects and can be considered as noise patterns. Figure 17. Verifying the sparsity of interactions. We show absolute values of normalized interactions in a descending order. LLMs all encode a small number of salient interactions, while most of the interaction effects are negligible. F.2. More Results on the Verification of the Sparsity of Interactions Here are more results on the verification of quality of universal matching. Figure 18 compares the LLM’s true outputv(x T ) for all masked inputs against the logical model using only the most salient interactions. Even when using just the top 3% or top 5% of all interactions, the matching error is minimal. This empirically demonstrates that the LLM’s output can be faithfully approximated by a small, sparse set of salient interactions. 25 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions internlm2 - chat - 7b LLM’s outputmatching error 푣 푥 푇 llama - 2 - 7b LLM’s outputmatching error 푣 푥 푇 llama - 2 - 7b - chat LLM’s outputmatching error 푣 푥 푇 llama - 3 - 8b - instruct LLM’s outputmatching error 푣 푥 푇 mistral - 7b - v0.3 LLM’s outputmatching error 푣 푥 푇 olmo - 2 - 7b - instruct LLM’s outputmatching error 푣 푥 푇 olmoe - 7b LLM’s outputmatching error 푣 푥 푇 qwen2.5 - 7b - instruct LLM’s outputmatching error 푣 푥 푇 qwen2 - 7b - instruct LLM’s outputmatching error 푣 푥 푇 qwen3 - 8b - instruct LLM’s outputmatching error 푣 푥 푇 Figure 18. Verifying the quality of universal matching for any 2 n masked inputs. The red line plots outputs of the LLM in an ascending order. 26 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions F.3. Detailed Case Study Figure 19 is the detailed case study of how to use our interaction-based analytical tool. It offers preliminary evidence that semantically irrelevant alterations to the prompt template can lead to significant changes in the salient interaction patterns, even when the input and output remains unchanged. This reveals the existence of unstable interactions, which we propose as the underlying cause of prompt sensitivity. Which characteristic describes the texture of a kitten's fur ANSWERS: A.gray B.warm C.long D.soft ANSWER: 푰풏풑풖풕풙+ 푷풓풐풎풑풕푻풆풎풑풍풂풕풆푻→ LLM Generated Output: D LLM Which characteristic describes the texture of a kitten's fur Answers: A.gray B.warm C.long D.soft Answer: Equivalently modeling 흓 ퟏ 풙 푣퐷푃) 푷=푭풐풓풎풂풕(풙,푻): 흓 ퟏ 풙= 푺⊆푵 푰 푺 풙푻 푰풏풑풖풕풙+ 푷풓풐풎풑풕푻풆풎풑풍풂풕풆 푻→ 푷=푭풐풓풎풂풕(풙, 푻): Generated Output: D Equivalently modeling 흓 ퟐ 풙 푣퐷 푃) 흓 ퟐ 풙= 푺⊆푵 푰 푺 풙 푻 LLM：Llama-2-7b-chat 푰 푺 풙푻 푰 푺 풙 푻 S =long S =gray S =gray, warm S =characteristic, describes, the, texture, a, kitten's, fur, gray, long S =fur, soft S =fur, texture, warm S =describes, the, texture, of, a, fur S =characteristic, describes, the, fur, gray, long, soft S =kitten’s, fur, gray S =characteristic, describes, the, texture, kitten's, fur, warm, long, soft Which of these traits is most influenced by environment ANSWERS: A.weight B.haircolor C.bloodtype D.handedness ANSWER: 푰풏풑풖풕풙+ 푷풓풐풎풑풕푻풆풎풑풍풂풕풆푻→ LLM Generated Output: C LLM Which of these traits is most influenced by environment Answers: A.weight B.haircolor C.bloodtype D.handedness Answer: Equivalently modeling 흓 ퟏ 풙 푣퐶푃) 푷=푭풐풓풎풂풕(풙,푻): 흓 ퟏ 풙= 푺⊆푵 푰 푺 풙푻 푰풏풑풖풕풙+ 푷풓풐풎풑풕푻풆풎풑풍풂풕풆 푻→ 푷=푭풐풓풎풂풕(풙, 푻): Generated Output: D Equivalently modeling 흓 ퟐ 풙 푣퐶 푃) 흓 ퟐ 풙= 푺⊆푵 푰 푺 풙 푻 LLM：Llama-2-7b-chat 푰 푺 풙푻 푰 푺 풙 푻 S =by, environment, influenced, is, most, of, these, traits, weight S =Which, by, environment, influenced, is, most, of, these, traits, weight S =by, influenced, is, most, of, these, traits, weight S =by, hair color, influenced, is, most, of, these, traits, weight S =Which, by, hair color, influenced, is, most, of, these, traits, weight S =Which, of, these, traits, is, most, environment, weight, hair color S =Which, of, these, is, most, influenced, environment, hair color S =weight, hair color S =hair color S =Which, of, these, traits, is, most, influenced, environment, weight, hair color Case 1 Case 2 Figure 19. A case study of interaction-level analysis revealing latent instability. The same inputxis formatted with two semantically identical templates,Tand ˆ T, differing only in letter case (e.g.,“Answer” vs. “ANSWER”). Although the LLM generates the same correct output (“D”) in both cases, the composition of the interaction-based logical modelφ(x)reveals significant internal divergence. Many interaction effects are highly unstable, changing in either sign or magnitude. This highlights a critical risk of prompt sensitivity that is invisible to output-level 27 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions F.4. More Results on the Prompt Sensitivity of Different Orders Here are more results on the prompt sensitivity of different orders on the ARC dataset. As illustrated in Figure 20, it shows that the prompt sensitivity of low-order interactions is the lowest, followed by mid-order, while high-order interactions exhibit the highest prompt sensitivity. This indicates that low-order interactions encoded by LLMs are highly stable when faced with subtle changes to prompt templates, i.e., simple interaction patterns are more robust. Conversely, the high sensitivity of high-order interactions reveals that the LLMs’ internal representation of complex patterns is highly unstable. Qwen2.5-14B-InstructQwen2.5-32B-InstructQwen2.5-72B-Instruct Qwen3-0.6B-InstructQwen3-1.7B-Instruct Qwen3-4B Qwen3-4B-Instruct Qwen3-8B Qwen3-8B-Instruct Qwen3-14B Qwen3-14B-InstructQwen3-32B-Instruct Qwen3-30B-A3B Qwen3-30B-A3B-Instruct Olmo-1BOlmo-7B Olmo-7B-Instruct Olmo-2-1B-InstructOlmo-2-7B-Instruct Olmo-2-13B-InstructOlmo-2-32B-Instruct Olmoe-7B Olmoe-7B-Instruct Internlm2-7B Internlm2-Chat-7B 0.50 0.75 1.00 1.25 1.50 1.75 2.00 2.25 Prompt Sensitivity of Differents Orders IPS type Model Set 2 Low OrderMid OrderHigh Order Llama-2-7B Llama-2-7B-Chat Llama-2-13B Llama-2-13B-Chat Llama-2-70B Llama-2-70B-Chat Llama-3-8B Llama-3-8B-Instruct Llama-Moe-V1-3_5B-2_8-SftLlama-Moe-V2-3_8B-2_8-Sft Mistral-7B-V0.3 Mistral-7B-V0.3-Instruct Mixtral-8X7B Mixtral-8X7B-Instruct Qwen1.5-Moe-A2.7B Qwen1.5-Moe-A2.7B-Chat Qwen2-7B Qwen2-0.5B-InstructQwen2-1.5B-Instruct Qwen2-7B-Instruct Qwen2-72B-Instruct Qwen2.5-0.5B-InstructQwen2.5-1.5B-Instruct Qwen2.5-3B-InstructQwen2.5-7B-Instruct 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 Prompt Sensitivity of Differents Orders IPS type Model Set 1 Low OrderMid OrderHigh Order Figure 20. A comparison of the prompt sensitivity of three order types. Results show that low-order interactions are the least sensitive, while high-order interactions are the most sensitive. 28 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions F.5. More Results on Relative Change in the Prompt Sensitivity of Low-, Mid-, and High-Order Interactions for Different Factors. (a) The relative change of Instruct/Chat Model compared to Base Model(b) The relative change of Dense Model compared to MoE Model (c) The relative change of Few-shot learning compared to 0-shot learning Figure 21. Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors. F.6. More Results on the Prompt Sensitivity of Different Order Types across Different Model Scales. Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity of Different Orders IPS type 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Low Order Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Mid Order Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B High Order Tip: within each model series, lighter color represents smaller model scale Figure 22. A comparison of prompt sensitivity of different order types across different model scales. F.7.More Results on the Prompt Sensitivity of Different Orders for Each Individual LLM when Applying Few-Shot Learning Llama-3-8B Mistral-7B-V0.3 Qwen2-7B Olmoe-7B-Instruct 125 100 75 50 25 0 25 Relative Change of Different Orders IPS type (%) 1-shot2-shot3-shot1-shot2-shot3-shot1-shot2-shot3-shot1-shot2-shot3-shot Low OrderMid OrderHigh Order Figure 23. A comparison of prompt sensitivity of low-, mid-, and high-order interactions between 0-shot learning and few-shot learning. Prompt sensitivity at all three order levels shows an clear drop when applying few-shot learning. 29 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions F.8. Hyperparameter Experiments of the threshold τ To rigorously evaluate the robustness of our Interaction-based Prompt Sensitivity (IPS) metric, we conducted a hyperparam- eter sweep on the threshold τ . F.8.1. DETAILED MODEL RANKINGS UNDER VARYING THRESHOLDS We aggregated the ranking and scoring consistency across all 16 thresholds using five metrics. As shown in Table 2, the high correlation coefficients and low error rates demonstrate that the IPS metric is highly robust to the choice of τ . Table 2. Summary of consistency metrics across 16 differentτthresholds (0.05–0.20). The high values in correlation metrics and low RMSE indicate that the relative ranking of model sensitivity remains stable regardless of the specific threshold used. MetricValueInterpretation Spearman’s ρ0.9905Rank Correlation: Measures the average similarity of the overall ranking trends. A value close to 1.0 indicates near-perfect monotonic consistency. Pearson’s r0.9957Linearity: Measures the linear correlation of the raw IPS scores, indicating that the scale of sensitivity shifts linearly across thresholds. Kendall’s τ0.9465 Pairwise Consistency: Indicates the probability that any pair of models maintains their relative order (better/worse) across different thresholds. RMSE1.72Ranking Stability: On average, a model’s rank fluctu- ates by only±1.72positions across different threshold settings. Top-10 Overlap96.7%SOTA Stability: The set of the top-10 most stable models remains 96.7% identical, ensuring reliable identification of the best-performing models. To provide a granular view of robustness, Table 3 details the IPS scores across 10 distinct thresholds ranging fromτ = 0.05 toτ = 0.20. Models are sorted based on their stability at the baseline thresholdτ = 0.05. The data reveals that while absolute scores fluctuate, the relative ranking of model stability remains highly consistent. Table 3. Detailed IPS scores for 50 LLMs across 10 different thresholds. The consistency in color gradients (implied by values) across rows confirms the robustness of the metric. IPS Score (↓) at Threshold τ Model Name0.050.060.070.080.090.100.120.150.180.20 Qwen2.5-72B-Instruct1.328 1.312 1.297 1.286 1.276 1.268 1.255 1.240 1.228 1.222 Qwen2-72B-Instruct1.362 1.346 1.333 1.322 1.312 1.304 1.290 1.278 1.271 1.265 Qwen2-7B-Instruct1.411 1.400 1.391 1.383 1.376 1.368 1.357 1.341 1.327 1.318 Qwen3-32B-Instruct1.424 1.419 1.415 1.411 1.408 1.406 1.403 1.401 1.396 1.395 Qwen3-14B-Instruct1.443 1.438 1.434 1.429 1.427 1.425 1.419 1.415 1.412 1.407 Llama-2-70B-Chat1.450 1.446 1.441 1.438 1.436 1.433 1.428 1.420 1.408 1.398 Qwen2.5-32B-Instruct1.463 1.454 1.446 1.440 1.434 1.430 1.423 1.415 1.411 1.409 Llama-3-8B-Instruct1.486 1.483 1.480 1.477 1.475 1.473 1.467 1.460 1.447 1.436 Qwen2.5-14B-Instruct1.490 1.481 1.473 1.465 1.459 1.454 1.446 1.436 1.430 1.425 Qwen3-4B-Instruct1.496 1.493 1.490 1.488 1.485 1.484 1.480 1.473 1.470 1.464 Qwen2-1.5B-Instruct1.497 1.494 1.492 1.487 1.483 1.477 1.465 1.447 1.429 1.416 Llama-2-70B1.505 1.504 1.504 1.505 1.505 1.507 1.509 1.516 1.523 1.527 Qwen3-8B-Instruct1.515 1.511 1.508 1.505 1.502 1.499 1.494 1.491 1.491 1.490 Continued on next page 30 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Table 3 – continued from previous page IPS Score (↓) at Threshold τ Model Name0.050.060.070.080.090.100.120.150.180.20 Qwen3-8B1.553 1.554 1.554 1.554 1.554 1.555 1.557 1.558 1.557 1.555 Qwen3-4B1.555 1.562 1.566 1.570 1.573 1.575 1.578 1.579 1.574 1.572 Llama-2-13B-Chat1.559 1.555 1.551 1.547 1.544 1.541 1.535 1.528 1.520 1.514 Qwen3-14B1.559 1.558 1.557 1.556 1.556 1.555 1.553 1.552 1.549 1.548 Qwen2-7B1.570 1.574 1.576 1.577 1.578 1.578 1.579 1.580 1.574 1.569 Qwen3-30B-A3B-Instruct1.575 1.575 1.576 1.577 1.579 1.580 1.582 1.583 1.583 1.582 Qwen2.5-7B-Instruct1.580 1.575 1.572 1.569 1.567 1.565 1.564 1.562 1.563 1.563 Llama-2-13B1.592 1.602 1.610 1.617 1.623 1.628 1.636 1.645 1.653 1.655 Olmo-1B1.601 1.610 1.617 1.623 1.628 1.632 1.639 1.647 1.652 1.655 Olmo-7B-Instruct1.604 1.599 1.595 1.592 1.589 1.587 1.583 1.579 1.578 1.576 Llama-3-8B1.618 1.628 1.637 1.644 1.650 1.655 1.664 1.674 1.679 1.682 Qwen2.5-3B-Instruct1.622 1.618 1.615 1.611 1.608 1.606 1.601 1.595 1.590 1.585 Olmo-2-13B-Instruct1.625 1.627 1.628 1.629 1.630 1.631 1.632 1.633 1.632 1.631 Olmo-2-7B-Instruct1.626 1.627 1.627 1.628 1.629 1.629 1.629 1.627 1.628 1.626 Olmo-2-32B-Instruct1.626 1.626 1.625 1.624 1.623 1.623 1.622 1.620 1.619 1.618 Qwen3-30B-A3B1.629 1.631 1.634 1.637 1.639 1.642 1.646 1.651 1.655 1.656 Mistral-7B-v0.3-Instruct1.630 1.629 1.629 1.629 1.629 1.631 1.633 1.638 1.641 1.644 InternLM2-Chat-7B1.634 1.639 1.641 1.644 1.647 1.650 1.656 1.660 1.664 1.667 Qwen2-0.5B-Instruct1.639 1.650 1.658 1.663 1.666 1.669 1.669 1.665 1.658 1.651 Mixtral-8x7B-Instruct1.641 1.638 1.636 1.633 1.632 1.631 1.629 1.628 1.629 1.630 Qwen1.5-MoE-A2.7B1.645 1.658 1.668 1.675 1.682 1.687 1.696 1.705 1.712 1.717 Qwen1.5-MoE-A2.7B-Chat1.648 1.654 1.657 1.660 1.663 1.665 1.667 1.668 1.664 1.660 Qwen3-1.7B-Instruct1.648 1.645 1.642 1.641 1.640 1.639 1.639 1.637 1.638 1.637 Qwen2.5-0.5B-Instruct1.652 1.663 1.671 1.677 1.681 1.684 1.688 1.689 1.687 1.685 Qwen2.5-1.5B-Instruct1.658 1.663 1.667 1.671 1.674 1.676 1.679 1.684 1.686 1.689 Llama-MoE-v2-3 8B-28-SFT 1.665 1.669 1.673 1.676 1.678 1.681 1.685 1.691 1.696 1.698 Llama-2-7B-Chat1.670 1.677 1.683 1.687 1.690 1.691 1.692 1.693 1.691 1.689 Olmo-7B1.671 1.679 1.685 1.690 1.695 1.699 1.706 1.714 1.718 1.721 Mixtral-8x7B1.677 1.682 1.686 1.689 1.692 1.695 1.701 1.709 1.715 1.720 Olmoe-7B1.680 1.689 1.696 1.702 1.707 1.711 1.720 1.733 1.742 1.746 Qwen3-0.6B-Instruct1.682 1.678 1.675 1.673 1.671 1.668 1.665 1.659 1.654 1.653 Llama-2-7B1.682 1.692 1.699 1.705 1.711 1.716 1.724 1.733 1.740 1.745 Olmo-2-1B-Instruct1.698 1.710 1.719 1.727 1.734 1.739 1.747 1.756 1.760 1.760 InternLM2-7B1.706 1.711 1.716 1.720 1.724 1.728 1.734 1.743 1.751 1.758 Llama-MoE-v1-3 5B-28-SFT 1.715 1.720 1.724 1.728 1.732 1.735 1.743 1.753 1.761 1.765 Olmoe-7B-Instruct1.716 1.717 1.719 1.721 1.722 1.723 1.724 1.726 1.730 1.732 Mistral-7B-v0.31.720 1.729 1.737 1.743 1.748 1.752 1.760 1.768 1.775 1.779 F.8.2. VERIFYING THE GENERALIZABILITY OF THE METHODS AND CONCLUSIONS ON DIFFERENT THRESHOLD τ . In Sections 4.2 and 4.3, we set the thresholdτto 0.1 to distinguish salient interactions from noise. This threshold directly influences the proportion of interactions classified as salient interactions. A higherτvalue usually generates a smaller set of salient interactions with more significant effects. Li & Zhang (2023) conducted experiments which show that conclusions are not sensitive to the choice ofτ. Our choice ofτis guided by the empirical sparsity of interactions. Figure 2 (a) shows a sharp “elbow” in the distribution of interaction effects, clearly separating a small set of high-magnitude salient interactions from a long tail of near-zero noise interactions. A thresholdτchosen from the range of 0.05 to 0.15 effectively captures this salient set, satisfying the sparsity assumption without being overly restrictive. To ensure the robustness of our findings, we conducted hyperparameter experiments withτ = 0.05andτ = 0.15. Our main conclusions remain consistent across different threshold values. 31 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Here are the results for τ = 0.15. Llama-2-7B Pair Llama-2-13B PairLlama-2-70B Pair Llama-3-8B Pair Mistral-7B-V0.3 Pair Mixtral-8X7B Pair Internlm2-7B Pair Olmo-7B Pair Olmoe-7B Pair Qwen2-7B PairQwen3-4B PairQwen3-8B Pair Qwen3-14B Pair Qwen1.5-Moe-A2.7B Pair Qwen3-30B-A3B Pair 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS Base ModelsInstruct / Chat Models Figure 24. A comparison of the prompt sensitivity between instruc- t/chat models and base models. Results show that instruct/chat models are less sensitive than corresponding base models. Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Tip: within each model series, lighter color represents smaller model scale Figure 25. A comparison of prompt sensitivity across different model scales. As the model scale increases, the prompt sensitivity within a model series systematically decreases. Llama 2 FamilyLlama 3 Family Mistral Family Qwen Family Olmo Family 1.3 1.4 1.5 1.6 1.7 Prompt Sensitivity IPS Llama-2-70B-Chat Llama-2-70B Llama-2-13B-Chat Llama-2-13B Llama-2-7B-Chat Llama-2-7B Llama-Moe-V1-3_5B-2_8-Sft Llama-3-8B-Instruct Llama-3-8B Llama-Moe-V2-3_8B-2_8-Sft Mixtral-8X7B-Instruct Mistral-7B-V0.3-Instruct Mixtral-8X7B Mistral-7B-V0.3 Qwen2.5-72B-Instruct Qwen2-72B-Instruct Qwen2-7B-Instruct Qwen2.5-32B-Instruct Qwen3-14B-Instruct Qwen2.5-14B-Instruct Qwen3-4B-Instruct Qwen3-8B-Instruct Qwen3-14B Qwen3-8B Qwen2.5-7B-Instruct Qwen3-4B Qwen2-7B Qwen3-30B-A3B-Instruct Qwen3-30B-A3B Qwen1.5-Moe-A2.7B-Chat Qwen1.5-Moe-A2.7B Olmo-7B-Instruct Olmo-2-32B-Instruct Olmo-2-7B-Instruct Olmo-2-13B-Instruct Olmo-7B Olmoe-7B-Instruct Olmoe-7B MoE Model Dense Model MoE Model Dense Model Figure 26. A Comparison of prompt sensitivity between MoE models and dense models. Generally, MoE models tend to be more sensitive than dense models in the same model family. Llama-2-7B Llama-2-7B-Chat Llama-3-8B Llama-3-8B-Instruct Mistral-7B-V0.3 Mistral-7B-V0.3-Instruct Qwen2-7B Qwen2-7B-Instruct Qwen2.5-7B-Instruct Qwen3-8B-Instruct Olmo-2-7B-Instruct Olmoe-7B-Instruct Qwen1.5-Moe-A2.7B-Chat 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 Prompt Sensitivity IPS 0-shot1-shot2-shot3-shot Figure 27. A comparison of prompt sensitivity between 0-shot learning and few-shot learning. The drop in prompt sensitivity is substantial from 0-shot to 1-shot. (a) The relative change of Instruct/Chat Model compared to Base Model(b) The relative change of Dense Model compared to MoE Model (c) The relative change of Few-shot learning compared to 0-shot learning Figure 28. Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors. 32 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Qwen2.5-14B-InstructQwen2.5-32B-InstructQwen2.5-72B-Instruct Qwen3-0.6B-InstructQwen3-1.7B-Instruct Qwen3-4B Qwen3-4B-Instruct Qwen3-8B Qwen3-8B-Instruct Qwen3-14B Qwen3-14B-InstructQwen3-32B-Instruct Qwen3-30B-A3B Qwen3-30B-A3B-Instruct Olmo-1BOlmo-7B Olmo-7B-Instruct Olmo-2-1B-InstructOlmo-2-7B-Instruct Olmo-2-13B-InstructOlmo-2-32B-Instruct Olmoe-7B Olmoe-7B-Instruct Internlm2-7B Internlm2-Chat-7B 0.50 0.75 1.00 1.25 1.50 1.75 2.00 2.25 Prompt Sensitivity of Differents Orders IPS type Model Set 2 Low OrderMid OrderHigh Order Llama-2-7B Llama-2-7B-Chat Llama-2-13B Llama-2-13B-Chat Llama-2-70B Llama-2-70B-Chat Llama-3-8B Llama-3-8B-Instruct Llama-Moe-V1-3_5B-2_8-SftLlama-Moe-V2-3_8B-2_8-Sft Mistral-7B-V0.3 Mistral-7B-V0.3-Instruct Mixtral-8X7B Mixtral-8X7B-Instruct Qwen1.5-Moe-A2.7B Qwen1.5-Moe-A2.7B-Chat Qwen2-7B Qwen2-0.5B-InstructQwen2-1.5B-Instruct Qwen2-7B-Instruct Qwen2-72B-Instruct Qwen2.5-0.5B-InstructQwen2.5-1.5B-Instruct Qwen2.5-3B-InstructQwen2.5-7B-Instruct 0.50 0.75 1.00 1.25 1.50 1.75 2.00 2.25 Prompt Sensitivity of Differents Orders IPS type Model Set 1 Low OrderMid OrderHigh Order Figure 29. A comparison of the prompt sensitivity of three order types. Results show that low-order interactions are the least sensitive, while high-order interactions are the most sensitive. Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity of Different Orders IPS type 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Low Order Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Mid Order Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B High Order Tip: within each model series, lighter color represents smaller model scale Figure 30. A comparison of prompt sensitivity at the order-level across different model scales. 33 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Here are the results for τ = 0.05. Llama-2-7B Pair Llama-2-13B PairLlama-2-70B Pair Llama-3-8B Pair Mistral-7B-V0.3 Pair Mixtral-8X7B Pair Internlm2-7B Pair Olmo-7B Pair Olmoe-7B Pair Qwen2-7B PairQwen3-4B PairQwen3-8B Pair Qwen3-14B Pair Qwen1.5-Moe-A2.7B Pair Qwen3-30B-A3B Pair 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS Base ModelsInstruct / Chat Models Figure 31. A comparison of the prompt sensitivity between instruc- t/chat models and base models. Results show that instruct/chat models are less sensitive than corresponding base models. Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity IPS 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Tip: within each model series, lighter color represents smaller model scale Figure 32. A comparison of prompt sensitivity across different model scales. As the model scale increases, the prompt sensitivity within a model series systematically decreases. Llama 2 FamilyLlama 3 Family Mistral Family Qwen Family Olmo Family 1.35 1.40 1.45 1.50 1.55 1.60 1.65 1.70 Prompt Sensitivity IPS Llama-2-70B-Chat Llama-2-70B Llama-2-13B-Chat Llama-2-13B Llama-2-7B-Chat Llama-2-7B Llama-Moe-V1-3_5B-2_8-Sft Llama-3-8B-Instruct Llama-3-8B Llama-Moe-V2-3_8B-2_8-Sft Mistral-7B-V0.3-Instruct Mixtral-8X7B-Instruct Mixtral-8X7B Mistral-7B-V0.3 Qwen2.5-72B-Instruct Qwen2-72B-Instruct Qwen2-7B-Instruct Qwen3-14B-Instruct Qwen2.5-32B-Instruct Qwen2.5-14B-Instruct Qwen3-4B-Instruct Qwen3-8B-Instruct Qwen3-8B Qwen3-4B Qwen3-14B Qwen2-7B Qwen3-30B-A3B-Instruct Qwen2.5-7B-Instruct Qwen3-30B-A3B Qwen1.5-Moe-A2.7B Qwen1.5-Moe-A2.7B-Chat Olmo-7B-Instruct Olmo-2-13B-Instruct Olmo-2-7B-Instruct Olmo-2-32B-Instruct Olmo-7B Olmoe-7B Olmoe-7B-Instruct MoE Model Dense Model MoE Model Dense Model Figure 33. A Comparison of prompt sensitivity between MoE models and dense models. Generally, MoE models tend to be more sensitive than dense models in the same model family. Llama-2-7B Llama-2-7B-Chat Llama-3-8B Llama-3-8B-Instruct Mistral-7B-V0.3 Mistral-7B-V0.3-Instruct Qwen2-7B Qwen2-7B-Instruct Qwen2.5-7B-Instruct Qwen3-8B-Instruct Olmo-2-7B-Instruct Olmoe-7B-Instruct Qwen1.5-Moe-A2.7B-Chat 0.8 1.0 1.2 1.4 1.6 1.8 2.0 Prompt Sensitivity IPS 0-shot1-shot2-shot3-shot Figure 34. A comparison of prompt sensitivity between 0-shot learning and few-shot learning. The drop in prompt sensitivity is substantial from 0-shot to 1-shot. (a) The relative change of Instruct/Chat Model compared to Base Model(b) The relative change of Dense Model compared to MoE Model (c) The relative change of Few-shot learning compared to 0-shot learning Figure 35. Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors. 34 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Qwen2.5-14B-InstructQwen2.5-32B-InstructQwen2.5-72B-Instruct Qwen3-0.6B-InstructQwen3-1.7B-Instruct Qwen3-4B Qwen3-4B-Instruct Qwen3-8B Qwen3-8B-Instruct Qwen3-14B Qwen3-14B-InstructQwen3-32B-Instruct Qwen3-30B-A3B Qwen3-30B-A3B-Instruct Olmo-1BOlmo-7B Olmo-7B-Instruct Olmo-2-1B-InstructOlmo-2-7B-Instruct Olmo-2-13B-InstructOlmo-2-32B-Instruct Olmoe-7B Olmoe-7B-Instruct Internlm2-7B Internlm2-Chat-7B 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 Prompt Sensitivity of Differents Orders IPS type Model Set 2 Low OrderMid OrderHigh Order Llama-2-7B Llama-2-7B-Chat Llama-2-13B Llama-2-13B-Chat Llama-2-70B Llama-2-70B-Chat Llama-3-8B Llama-3-8B-Instruct Llama-Moe-V1-3_5B-2_8-SftLlama-Moe-V2-3_8B-2_8-Sft Mistral-7B-V0.3 Mistral-7B-V0.3-Instruct Mixtral-8X7B Mixtral-8X7B-Instruct Qwen1.5-Moe-A2.7B Qwen1.5-Moe-A2.7B-Chat Qwen2-7B Qwen2-0.5B-InstructQwen2-1.5B-Instruct Qwen2-7B-Instruct Qwen2-72B-Instruct Qwen2.5-0.5B-InstructQwen2.5-1.5B-Instruct Qwen2.5-3B-InstructQwen2.5-7B-Instruct 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 Prompt Sensitivity of Differents Orders IPS type Model Set 1 Low OrderMid OrderHigh Order Figure 36. A comparison of the prompt sensitivity of three order types. Results show that low-order interactions are the least sensitive, while high-order interactions are the most sensitive. Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity of Different Orders IPS type 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Low Order Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Mid Order Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B High Order Tip: within each model series, lighter color represents smaller model scale Figure 37. A comparison of prompt sensitivity at the order-level across different model scales. 35 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions F.9. Results on MMLU Dataset Here are the results on the MMLU dataset and τ is set to 0.1, we can observe the same conclusions on this dataset. Llama-2-7B Pair Llama-2-13B PairLlama-2-70B Pair Llama-3-8B Pair Mistral-7B-V0.3 Pair Mixtral-8X7B Pair Internlm2-7B Pair Olmo-7B Pair Olmoe-7B Pair Qwen2-7B PairQwen3-4B PairQwen3-8B Pair Qwen3-14B Pair Qwen1.5-Moe-A2.7B Pair Qwen3-30B-A3B Pair 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS Base ModelsInstruct / Chat Models Figure 38. A comparison of the prompt sensitivity between instruc- t/chat models and base models. Results show that instruct/chat models are less sensitive than corresponding base models. Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Tip: within each model series, lighter color represents smaller model scale Figure 39. A comparison of prompt sensitivity across different model scales. As the model scale increases, the prompt sensitivity within a model series systematically decreases. Llama 2 FamilyLlama 3 Family Mistral Family Qwen Family Olmo Family 1.40 1.45 1.50 1.55 1.60 1.65 1.70 1.75 Prompt Sensitivity IPS Llama-2-70B-Chat Llama-2-70B Llama-2-13B-Chat Llama-2-13B Llama-2-7B-Chat Llama-2-7B Llama-Moe-V1-3_5B-2_8-Sft Llama-3-8B-Instruct Llama-3-8B Llama-Moe-V2-3_8B-2_8-Sft Mistral-7B-V0.3-Instruct Mixtral-8X7B-Instruct Mixtral-8X7B Mistral-7B-V0.3 Qwen2-72B-Instruct Qwen2-7B-Instruct Qwen2.5-72B-Instruct Qwen2.5-32B-Instruct Qwen3-14B-Instruct Qwen2.5-14B-Instruct Qwen3-8B-Instruct Qwen3-4B-Instruct Qwen3-8B Qwen3-14B Qwen1.5-Moe-A2.7B-Chat Qwen3-4B Qwen2.5-7B-Instruct Qwen3-30B-A3B-Instruct Qwen2-7B Qwen3-30B-A3B Qwen1.5-Moe-A2.7B Olmo-7B-Instruct Olmo-2-32B-Instruct Olmo-2-13B-Instruct Olmo-2-7B-Instruct Olmo-7B Olmoe-7B Olmoe-7B-Instruct MoE Model Dense Model MoE Model Dense Model Figure 40. A Comparison of prompt sensitivity between MoE models and dense models. Generally, MoE models tend to be more sensitive than dense models in the same model family. Llama-2-7B Llama-2-7B-Chat Llama-3-8B Llama-3-8B-Instruct Mistral-7B-V0.3 Mistral-7B-V0.3-Instruct Qwen2-7B Qwen2-7B-Instruct Qwen2.5-7B-Instruct Qwen3-8B-Instruct Olmo-2-7B-Instruct Olmoe-7B-Instruct Qwen1.5-Moe-A2.7B-Chat 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 Prompt Sensitivity IPS 0-shot1-shot2-shot3-shot Figure 41. A comparison of prompt sensitivity between 0-shot learning and few-shot learning. The drop in prompt sensitivity is substantial from 0-shot to 1-shot. (a) The relative change of Instruct/Chat Model compared to Base Model(b) The relative change of Dense Model compared to MoEModel (c) The relative change of Few-shot learning compared to 0-shot learning Figure 42. Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors. 36 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Qwen2.5-14B-InstructQwen2.5-32B-InstructQwen2.5-72B-Instruct Qwen3-0.6B-InstructQwen3-1.7B-Instruct Qwen3-4B Qwen3-4B-Instruct Qwen3-8B Qwen3-8B-Instruct Qwen3-14B Qwen3-14B-InstructQwen3-32B-Instruct Qwen3-30B-A3B Qwen3-30B-A3B-Instruct Olmo-1BOlmo-7B Olmo-7B-Instruct Olmo-2-1B-InstructOlmo-2-7B-Instruct Olmo-2-13B-InstructOlmo-2-32B-Instruct Olmoe-7B Olmoe-7B-Instruct Internlm2-7B Internlm2-Chat-7B 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 Prompt Sensitivity of Different Orders IPS type Model Set 2 Low OrderMid OrderHigh Order Llama-2-7B Llama-2-7B-Chat Llama-2-13B Llama-2-13B-Chat Llama-3-8B Llama-3-8B-Instruct Llama-Moe-V1-3_5B-2_8-SftLlama-Moe-V2-3_8B-2_8-Sft Mistral-7B-V0.3 Mistral-7B-V0.3-Instruct Mixtral-8X7B Mixtral-8X7B-Instruct Qwen1.5-Moe-A2.7B Qwen1.5-Moe-A2.7B-Chat Qwen2-7B Qwen2-0.5B-InstructQwen2-1.5B-Instruct Qwen2-7B-Instruct Qwen2-72B-Instruct Qwen2.5-0.5B-InstructQwen2.5-1.5B-Instruct Qwen2.5-3B-InstructQwen2.5-7B-Instruct 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 Prompt Sensitivity of Different Orders IPS type Model Set 1 Low OrderMid OrderHigh Order Figure 43. A comparison of the prompt sensitivity of three order types. Results show that low-order interactions are the least sensitive, while high-order interactions are the most sensitive. Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity of Different Orders IPS type 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Low Order Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B Mid Order Llama-2 (Base) Llama-2 (Chat) Qwen2 (Instruct) Qwen2.5 (Instruct) Qwen3 (Instruct) Olmo2 (Instruct) 7B13B70B7B13B70B0.5B1.5B7B72B0.5B1.5B3B7B14B32B72B0.6B1.7B4B8B14B32B1B7B13B32B High Order Tip: within each model series, lighter color represents smaller model scale Figure 44. A comparison of prompt sensitivity at the order-level across different model scales. 37 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions G. Prompt Sensitivity of altering tokens vs. adding tokens Disaggregating prompt sensitivity by the type of perturbation offers deeper insights into model behavior. Following this direction, we conduct a fine-grained decomposition of our experimental results, comparing prompt sensitivity of two distinct categories to prompt alterations: 1. Altering tokens: This involves modifying the capitalization of words, such as from "Answer" to "ANSWER". 2. Adding tokens: This involves adding symbolic components, for instance, changing a colon from ":" to "::". Dense ModelAltering TokensAdding TokensDifference internlm2-chat-7b1.6731.676+0.003 llama-2-13b-chat1.5201.433-0.087 llama-2-7b-chat 1.6331.710+0.077 llama-3-8b-instruct1.4501.467+0.017 mistral-7b-v0.3-instruct1.5381.635+0.097 olmo-2-7b-instruct1.6181.655+0.037 olmo-7b-instruct 1.4621.496+0.034 qwen2-7b-instruct1.3661.404+0.038 qwen2.5-7b-instruct 1.5581.567+0.009 qwen3-8b-instruct 1.5051.550+0.045 Table 4. IPS Scores of Dense Models on Different Perturbation Types. Higher scores indicate greater sensitivity. The more sensitive perturbation type for each model is highlighted in bold. MoE ModelAltering TokensAdding TokensDifference llama-moe-v1-3.5b-sft1.7361.724-0.012 llama-moe-v2-3.8b-sft 1.6981.655-0.043 mixtral-8x7b-instruct1.6581.617-0.041 olmoe-7b-instruct1.7251.699-0.026 qwen1.5-moe-a2.7b-chat1.6691.634-0.035 qwen3-30b-a3b-instruct 1.6081.638+0.030 Table 5. IPS Scores of MoE Models on Different Perturbation Types. The more sensitive perturbation type for each model is highlighted in bold. Our results highlight a clear architectural divide: (1) Dense models are more sensitive to adding tokens. As shown in Table 4, 9 out of the 10 analyzed dense models exhibit greater sensitivity to the ”adding tokens” category. This suggests a strong, consistent trend where changes to the template’s structure have a more pronounced impact on the internal interactions of dense architectures. (2) MoE models are more sensitive to altering tokens. In stark contrast, Table 5 shows that 5 out of the 6 MoE models are more sensitive to ”altering tokens”. This consistent pattern suggests that MoE architectures are more susceptible to variations in the change of word capitalization. H. Comparison between IPS and Other Metrics To demonstrate the unique value of our interaction-based approach, we compare the Interaction-based Prompt Sensitivity (IPS) against standard coarse-grained metrics derived from internal representations. Specifically, we measure the Cosine Similarity andL 2 Distance of the final-layer hidden states under prompt perturbations. While these metrics are commonly used to assess representation robustness, our analysis reveals that they fail to capture the nuanced mechanisms of prompt sensitivity in LLMs. H.1. Empirical Inconsistency of Representation-based Metrics We re-evaluate Factor 1 (Base vs. Instruct/Chat models) using these representation-based metrics. The results, summarized in Table 6, demonstrate a significant lack of consistency compared to the robust trends observed via IPS. As shown in Table 6, neither Cosine Similarity nor L 2 Distance provides a reliable proxy for prompt sensitivity: 38 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Table 6. Comparison of stability metrics (Cosine Similarity andL 2 Distance) for Base vs. Instruct/Chat models. Unlike IPS, these metrics fail to show a consistent trend regarding the impact of Supervised Fine-Tuning. Base ModelInstruct/Chat Model Model FamilyCosine (↑) L 2 Dist. (↓) Cosine (↑) L 2 Dist. (↓) Llama-2-7B0.770 75.12 0.78166.51 Llama-2-13B 0.942 17.350.92719.69 Llama-2-70B 0.911 33.650.79543.14 Llama-3-8B0.85679.77 0.880 73.56 Mistral-7B-V0.30.811207.89 0.843 154.43 Mixtral-8x7B 0.965138.710.875 136.03 InternLM2-7B0.898142.51 0.987 75.83 Olmo-7B 0.793 36.880.66148.33 Olmoe-7B 0.672 56.670.51085.73 Qwen2-7B0.864138.73 0.888 136.49 Qwen3-4B 0.914 60.130.77973.82 Qwen3-8B 0.995 33.950.96241.42 Qwen3-14B 0.94767.850.932 51.97 Qwen1.5-MoE-A2.7B0.765160.31 0.911 86.98 Qwen3-30B-A3B0.86481.13 0.932 39.04 •Contradictions between metrics: For models like Mixtral-8x7B, the two metrics contradict each other—Cosine Similarity suggests the Base model is more stable, while L 2 Distance favors the Instruct model. •Inconsistency with established trends: While our IPS analysis (and general consensus) identifies Instruct/Chat models as more robust to prompt variations, representation metrics frequently suggest the opposite. For instance, in the Llama-2-13B and Qwen3-8B pairs, the Base models exhibit higher cosine similarity and lowerL 2 distance than their Instruct counterparts. •Random fluctuations: There is no discernible pattern across model families. For Llama-2-7B, the Chat version appears more stable via Cosine Similarity but less stable via L 2 Distance. These contradictions indicate that global measures of hidden state changes are too coarse to serve as accurate indicators of the model’s functional sensitivity. H.2. Superiority of the Interaction-based Framework The empirical limitations of representation-based metrics highlight the theoretical advantages of our proposed framework. The superiority of IPS stems from two fundamental differences: 1. Explanability (“Why” vs. “What”): Hidden state similarity merely measures what has changed—the magnitude or direction of the aggregate internal representation vector. It treats the model as a black box regarding the reasoning process. In contrast, our interaction framework explains why the output fluctuates. By decomposing predictions into interactions, we can pinpoint specific combinations of input tokens (inference patterns) that become unstable. This fine-grained insight allows us to distinguish between benign representation shifts and those that disrupt the model’s logical coherence. 2. Faithfulness to the Output: Representation metrics lack a direct mathematical link to the final prediction. A small shift in Euclidean distance can sometimes lead to a flipped prediction, while a large shift might not. Conversely, our method is grounded in the Universal Matching Property (Theorem 1). This theorem guarantees that the sum of all interactions perfectly reconstructs the LLM’s output score. Consequently, IPS provides a faithful evaluation of the decision-making logic, ensuring that the measured sensitivity directly reflects the instability in the model’s actual predictive mechanism. 39 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions I. Detailed Discussion on the Impact of Model Architecture We notice that in Figure 7, the behavior of Mistral family is different from the other family: the dense models are more sensitive than the MoE models at low-order and mid-order level. We attribute this to the number of activated parameters, extending our finding from Factor 2 that larger models are less sensitive. While most MoE models (e.g., in Llama and Qwen families) are more sensitive due to fewer activated parameters, the Mistral case is reversed: Mixtral-8x7B activates more parameters than its dense counterpart Mistral-7B-V0.3 (13B vs. 7B), resulting in lower prompt sensitivity. To more rigorously isolate the influence of model architecture on the prompt sensitivity from model size or other factors, we conduct a controlled variable analysis. We select specific pairs from the Qwen and OLMo families that share the most similar model scales (i.e., activated parameters) and analogous training paradigms (i.e., base vs. base, instruct/chat vs. instruct/chat). This targeted comparison enables us to minimize confounding factors and focus directly on the architectural impact. Results in Table 7 consistently show that MoE models exhibit higher prompt sensitivity than dense models. This suggests that the increased prompt sensitivity of MoE architectures is not merely a consequence of smaller number of activated parameters. Instead, it further strengthens our conclusion that the MoE architecture inherently increases the prompt sensitivity of LLMs. Table 7. Controlled variable analysis of prompt sensitivity between MoE and dense models. Model NameArchitectureAct. Params.TypeIPS (ARC)↓IPS (MMLU)↓ Qwen1.5-moe-a2.7b-chatMoE2.7BChat1.6651.640 Qwen2.5-3b-instructDense3BInstruct1.6061.625 Olmoe-7bMoE1BBase1.7141.717 Olmo-1bDense1BBase1.6321.658 Qwen3-30b-a3bMoE3BBase1.6421.680 Qwen3-4bDense4BBase1.5681.599 Qwen3-30b-a3b-instructMoE3BInstruct1.5801.617 Qwen3-4b-instructDense4BInstruct1.4831.551 J. Solutions for reducing the computational cost of the method The limitation of our current study lies in the computational cost of the interaction framework. The method’s complexity scales exponentially with the number of input variablesn, as it requires evaluating2 n masked inputs. However, applying this method to very long text inputs would demand a high computational load. Future work can address this scalability challenge through several promising avenues. These strategies aim to reduce the effective number of input variables without fundamentally changing the faithfulness of the analysis: (1) Selective Input Variable Analysis. One approach is to analyze only a subset of informative input variables (i.e., words) while treating uninformative ones (e.g., stop words) as fixed background context. Previous research has demonstrated that this selection does not significantly impair the faithfulness of the interaction framework (Chen et al., 2024). (2) Phrase-level Aggregation. Instead of analyzing individual words, we can operate at a coarser perspective by merging related words into combined phrasal units. This reduces the total number of input variables while preserving key semantic meaning. For methods (1) and (2), we have put them into practice. In our experiments on long-form open-ended questions, we apply these two methods to effectively control the number of input variables. The specific selection strategies are detailed in Appendix K and Appendix L. It demonstrates that these techniques can substantially reduce computational complexity without affecting the key conclusions. (3) Approximation Methods. In addition to selection strategies, techniques specifically designed for efficiently computing sparse interactions to bypass exhaustiveO(2 n )evaluations (Kang et al., 2025; Butler et al., 2026), represent a clear path for reducing computational cost. These strategies represent promising directions for extending the powerful capabilities of interaction-based analysis to a wider range of long-text NLP tasks. 40 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions K. Details of the Experiments on Open-Ended Generation Tasks K.1. Experimental Setup This section illustrates the detailed setup of open-ended question-answering tasks, which more closely resemble real- world user scenarios. We employed the Databricks Dolly-15k dataset, an open-source collection of instruction-following records. This dataset spans multiple behavioral categories as defined in the InstructGPT paper (Ouyang et al., 2022), including brainstorming, classification, closed QA, open QA, and summarization, providing a diverse and realistic dataset for evaluating model robustness. We test the results mainly on the Llama-2 family. K.2. Selection of Input Variables Given that the text length in open-ended instructions far exceeds that of MCQ tasks, a direct analysis of all words would lead to exponential computational costs. To address this challenge, we applied the optimization strategies discussed in our limitations section: (1) Selective Input Variable Analysis and (2) Phrase-level Aggregation. Our approach is guided by a systematic procedure to choose a fixed number of key input variables from the full input. Specifically, for each input sentence, we select meaningful words or phrases to construct the set of input variablesN. A word is considered “meaningful” if it is not an NLTK (Bird, 2006) stop word or a punctuation mark. The remaining parts of the text, such as generic instruction templates (e.g., ”Below is an instruction...”) and stop words, are treated as fixed background context. During the interaction analysis, only the variables within the set N are masked. For a concrete example, consider the following prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Identify from the following list characters from The X-Files who are bald or balding: Walter Skinner, John Fitzgerald Byers, Dana Scully, Melvin Frohike, Darius Michaud, Peter Watts, Conrad Strughold, Queequeg ### Response: Selection Process for Input Variables: 1. Method (1) Application: We designate the generic instruction template and functional stop words (e.g., ”from”, ”the”, ”who”, ”are”) as background context, excluding them from the input variable set. 2.Method (2) Application: We aggregate words forming core semantic concepts into single phrasal units, such as the key entity "Walter Skinner" and the critical condition "bald or balding". Final Input Variables: [ "Identify", "following list", "characters","X-Files", "bald or balding","Walter Skinner", "John Fitzgerald Byers", "Dana Scully","Melvin Frohike", "Darius Michaud", "Peter Watts","Conrad Strughold", "Queequeg" ] K.3. Experimental Results By applying the aforementioned input variable selection strategies (Methods 1 and 2) in our experiments on the Dolly dataset, we successfully managed the analytical complexity for each long-text input, leading to a substantial reduction in computational cost. Crucially, the experimental outcomes derived from open-ended questions and the optimized setup remained highly consistent with the main conclusions drawn from our MCQ-based experiments. 41 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions Here are the results and τ is set to 0.1, we can observe the same conclusions in this experiment. Llama-2-7B Pair Llama-2-13B PairLlama-2-70B Pair 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity IPS Base ModelsInstruct / Chat Models Figure 45. A comparison of the prompt sensitivity between instruc- t/chat models and base models. Results show that instruct/chat models are less sensitive than corresponding base models. Llama-2 (Base) Llama-2 (Chat) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity IPS 7B13B70B7B13B70B Tip: within each model series, lighter color represents smaller model scale Figure 46. A comparison of prompt sensitivity across different model scales. As the model scale increases, the prompt sensitivity within a model series systematically decreases. Llama 2 Family 1.30 1.35 1.40 1.45 1.50 1.55 1.60 1.65 Prompt Sensitivity IPS Llama-2-70B-Chat Llama-2-13B-Chat Llama-2-7B-Chat Llama-2-70B Llama-2-13B Llama-2-7B Llama-Moe-V1-3_5B-2_8-Sft MoE Model Dense Model MoE Model Dense Model Figure 47. A Comparison of prompt sensitivity between MoE models and dense models. Generally, MoE models tend to be more sensitive than dense models in the same model family. Llama-2-7B-Chat Llama-2-13B-Chat Llama-Moe-V1-3_5B-2_8-Sft 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity IPS 0-shot1-shot2-shot3-shot Figure 48. A comparison of prompt sensitivity between 0-shot learning and few-shot learning. The drop in prompt sensitivity is substantial from 0-shot to 1-shot. Llama-2-7B Llama-2-7B-Chat Llama-2-13B Llama-2-13B-Chat Llama-2-70B Llama-2-70B-Chat Llama-Moe-V1-3_5B-2_8-Sft 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Prompt Sensitivity of Differents Orders IPS type Low OrderMid OrderHigh Order Figure 49. A comparison of the prompt sensitivity of three order types. Results show that low-order interactions are the least sensitive, while high-order interactions are the most sensitive. 42 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions (a)The relative change of Instruct/Chat Model compared to Base Model (b) The relative change of Dense Model compared to MoE Model (c) The relative change of Few-shot learning compared to 0-shot learning Figure 50. Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors. Llama-2 (Base)Llama-2 (Chat) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity of Different Orders IPS type 7B13B70B7B13B70B Low Order Llama-2 (Base)Llama-2 (Chat) 7B13B70B7B13B70B Mid Order Llama-2 (Base)Llama-2 (Chat) 7B13B70B7B13B70B High Order Tip: within each model series, lighter color represents smaller model scale Figure 51. A comparison of prompt sensitivity at the order-level across different model scales. 43 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions L. Details of the Experiments Beyond Prompt Template Modifications L.1. Experimental Setup To assess the generalizability of our findings beyond superficial template modifications, we conducted an additional experiment on the Dolly-15k dataset. This setup introduces more realistic and complex prompt perturbations that better reflect authentic user interactions, specifically semantic paraphrasing and instruction reordering. We employed these techniques to test the robustness of our conclusions under more challenging conditions. An illustrative example of a semantic paraphrase used in our experiment is shown below: An illustrative example of the full prompt structure and the applied perturbations is shown below. Original Prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Identify from the following list characters from The X-Files who are bald or balding: Walter Skinner, John Fitzgerald Byers, Dana Scully, Melvin Frohike, Darius Michaud, Peter Watts, Conrad Strughold, Queequeg ### Response: Perturbation 1: Semantic Paraphrase: Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Select from the list below figures from The X-Files that are losing their hair or bald: Walter Skinner, John Fitzgerald Byers, Dana Scully, Melvin Frohike, Darius Michaud, Peter Watts, Conrad Strughold, Queequeg ### Response: Perturbation 2: Instruction Reordering: Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: From The X-Files, identify characters who are bald or balding from the following list: Walter Skinner, John Fitzgerald Byers, Dana Scully, Melvin Frohike, Darius Michaud, Peter Watts, Conrad Strughold, Queequeg ### Response: L.2. Selection of Input Variables For this experiment, we employed the same input variable selection strategy as detailed in Appendix K.2. This approach combines Selective Input Variable Analysis and Phrase-level Aggregation to manage the computational complexity associated with longer, open-ended instructions while preserving the core semantic elements for interaction analysis. L.3. Experimental Results The experimental outcomes from this more challenging setup verify the main conclusions of our paper. Despite the increased complexity of the perturbations, we consistently observed that the four identified factors (supervised fine-tuning, increased 44 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions model scale, dense architectures, and few-shot learning) reduce prompt sensitivity, primarily by stabilizing low-order interactions. This replication demonstrates that our conclusions are not confined to simple template variations but hold true in scenarios that more closely mirror authentic user interactions, thereby strengthening the generalizability of our work. Here are the results and τ is set to 0.1, we can observe the same conclusions in this experiment. Llama-2-7B Pair Llama-2-13B PairLlama-2-70B Pair 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS Base ModelsInstruct / Chat Models Figure 52. A comparison of the prompt sensitivity between instruc- t/chat models and base models. Results show that instruct/chat models are less sensitive than corresponding base models. Llama-2 (Base) Llama-2 (Chat) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity IPS 7B13B70B7B13B70B Tip: within each model series, lighter color represents smaller model scale Figure 53. A comparison of prompt sensitivity across different model scales. As the model scale increases, the prompt sensitivity within a model series systematically decreases. Llama 2 Family 1.40 1.45 1.50 1.55 1.60 1.65 1.70 Prompt Sensitivity IPS Llama-2-70B-Chat Llama-2-13B-Chat Llama-2-70B Llama-2-7B-Chat Llama-2-13B Llama-2-7B Llama-Moe-V1-3_5B-2_8-Sft MoE Model Dense Model MoE Model Dense Model Figure 54. A Comparison of prompt sensitivity between MoE models and dense models. Generally, MoE models tend to be more sensitive than dense models in the same model family. Llama-2-7B-Chat Llama-2-13B-Chat Llama-Moe-V1-3_5B-2_8-Sft 0.8 1.0 1.2 1.4 1.6 1.8 Prompt Sensitivity IPS 0-shot1-shot2-shot3-shot Figure 55. A comparison of prompt sensitivity between 0-shot learning and few-shot learning. The drop in prompt sensitivity is substantial from 0-shot to 1-shot. Llama-2-7B Llama-2-7B-Chat Llama-2-13B Llama-2-13B-Chat Llama-2-70B Llama-2-70B-Chat Llama-Moe-V1-3_5B-2_8-Sft 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 Prompt Sensitivity of Differents Orders IPS type Low OrderMid OrderHigh Order Figure 56. A comparison of the prompt sensitivity of three order types. Results show that low-order interactions are the least sensitive, while high-order interactions are the most sensitive. 45 Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions (a)The relative change of Instruct/Chat Model compared to Base Model (b) The relative change of Dense Model compared to MoE Model (c) The relative change of Few-shot learning compared to 0-shot learning Figure 57. Comparing the relative change in the prompt sensitivity of low-, mid-, and high-order interactions for different factors. Llama-2 (Base)Llama-2 (Chat) 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Prompt Sensitivity of Different Orders IPS type 7B13B70B7B13B70B Low Order Llama-2 (Base)Llama-2 (Chat) 7B13B70B7B13B70B Mid Order Llama-2 (Base)Llama-2 (Chat) 7B13B70B7B13B70B High Order Tip: within each model series, lighter color represents smaller model scale Figure 58. A comparison of prompt sensitivity at the order-level across different model scales. 46