Paper deep dive
COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering
Kavinda Kehelella, Sanghamitra Dutta
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/12/2026, 6:24:28 AM
Summary
The paper introduces COFFEQA, a Counterfactual-guided Few-Shot Distillation framework for open-book Question Answering. It addresses the challenge of data-intensive knowledge distillation by using counterfactual examples—minimally modified contexts that change the teacher's answer—to provide additional supervision for challenging samples near the teacher's decision boundary. The method employs a new metric, Context Following Difficulty (CFD), to validate counterfactuals. Experiments show COFFEQA outperforms factual-only baselines on benchmarks like TriviaQA using T5 and Qwen2.5 models.
Entities (10)
Relation Signals (8)
COFFEQA → evaluatedon → TriviaQA
confidence 95% · For instance, on TriviaQA, COFFEQA improves over the factual-only baseline
COFFEQA → outperforms → factual-only distillation baselines
confidence 95% · COFFEQA outperforms factual-only distillation baselines under the same few-shot budget
COFFEQA → uses → Counterfactual
confidence 95% · CofFeQA ... aptly uses counterfactuals to provide additional supervision around challenging factual samples
COFFEQA → usesmodel → T5
confidence 95% · Experiments ... using T5 and Qwen2.5 families show that COFFEQA outperforms
COFFEQA → usesmodel → Qwen2.5
confidence 95% · Experiments ... using T5 and Qwen2.5 families show that COFFEQA outperforms
COFFEQA → validatesusing → Context Following Difficulty
confidence 95% · CofFeQA uses CFD-validated factual-counterfactual pairs for few-shot distillation
Counterfactual → definedas → minimally modified context
confidence 90% · we define a counterfactual as a minimally modified context that induces the teacher to change its response
Context Following Difficulty → measures → agreement with intended alternative answer
confidence 90% · CFD that further validates counterfactuals ... based on agreement with the intended alternative answer
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily dependent on the choice of these training samples. In fact, we demonstrate that using more uncertain samples near the teacher's decision boundary can significantly deteriorate distillation performance. Motivated by this observation, we propose Counterfactual-guided Few-shot Distillation in Open-book Question Answering (COFFEQA), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer. We provide geometric insights on how factual-counterfactual pairs can constrain the deviation between the teacher's and student's decision boundaries. We further introduce a new metric, Context Following Difficulty (CFD) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context. COFFEQA uses CFD-validated factual-counterfactual pairs for few-shot distillation. Experiments across five in-domain and three out-of-domain QA benchmarks using T5 and Qwen2.5 families show that COFFEQA outperforms factual-only distillation baselines under the same few-shot budget. For instance, on TriviaQA, COFFEQA improves over the factual-only baseline by 8.77 average EM points and 9.98 average F1 points under the T5 setting with 32 few-shot pairs.
Tags
Links
- Source: https://arxiv.org/abs/2610.12837v1
- Canonical: https://arxiv.org/abs/2610.12837v1
Trouble viewing inline? Open PDF directly →
Full Text
187,778 characters extracted from source content.
Expand or collapse full text
CofFeQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering Kavinda Kehelella Affiliation: University of Maryland, College Park Email: kkehelel@umd.edu Sanghamitra Dutta Affiliation: University of Maryland, College Park Email: sanghamd@umd.edu Abstract Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily dependent on the choice of these training samples. In fact, we demonstrate that using more uncertain samples near the teacher’s decision boundary can significantly deteriorate distillation performance. Motivated by this observation, we propose Counterfactual-guided Few-shot Distillation in open-book Question Answering (CofFeQA), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer. We provide geometric insights on how factual-counterfactual pairs can constrain the deviation between the teacher’s and student’s decision boundaries. We further introduce a new metric, Context Following Difficulty (CFD) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context. CofFeQA uses CFD-validated factual-counterfactual pairs for few-shot distillation. Experiments across five in-domain and three out-of-domain QA benchmarks using T5 and Qwen2.5 families show that CofFeQA outperforms factual-only distillation baselines under the same few-shot budget. For instance, on TriviaQA, CofFeQA improves over the factual-only baseline by 8.778.77 average EM points and 9.989.98 average F1 points under the T5 setting with 32 few-shot pairs. 1 Introduction Large language models (LLMs) have achieved remarkable success across a wide range of natural language processing tasks (Achiam et al., 2023; Chung et al., 2024; Gou et al., 2021; Raffel et al., 2020; Fang et al., 2026). However, their increasing size and computational requirements make deployment in resource-constrained environments particularly challenging (Kaplan et al., 2020; Brown et al., 2020; Strubell et al., 2019; Hoffmann et al., 2022). This motivates the need for compact models with low training and inference costs that preserve task-relevant capabilities of larger models (Liang et al., 2023; Saadi and Wang, 2025; Liu et al., 2024; Dissanayake et al., 2024). Knowledge distillation (Hinton et al., 2015) (see surveys Gou et al. (2021); Sanh et al. (2019); Xu et al. (2024b)) seeks to address this challenge by transferring knowledge from a large teacher model to a smaller student model. However, most distillation approaches typically rely on large-scale datasets or extensive supervision from the teacher, thus incurring prohibitive training costs for already resource-intensive language models (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2020; Wang et al., 2020; Dissanayake and Dutta, 2025; Hamman et al., 2025). This leads to our primary question: How can we perform data-efficient distillation in LLMs? Towards addressing this question, our work studies few-shot distillation for generative tasks, with a focus on open-book question answering. We find that the performance of few-shot distillation is heavily dependent on the choice of the distillation dataset, consistent with the broader importance of data selection for efficient LLM training (Fu et al., 2026; Zhou et al., 2023). In particular, samples near the teacher’s decision boundary can significantly deteriorate distillation performance (see Fig. 6 in Section C), also consistent with prior findings on knowledge transfer around decision boundaries (Chen et al., 2025). This motivates us to provide additional supervision around these challenging regions while maintaining a limited few-shot budget. To achieve this, we leverage counterfactual examples. Counterfactual-based data augmentation has improved classification performance (Kaushik et al., 2019; Wu et al., 2021; Ross et al., 2021; Gardner et al., 2020; McAleese and Keane, 2024; Bhattacharjee et al., 2024; Egea et al., 2025) by encouraging models to capture causal dependencies and avoid shortcut learning (Guo et al., 2023; Chen et al., 2023b). However, its applicability to few-shot distillation for generative tasks remains largely unexplored. Unlike classification, generative tasks produce free-form outputs, making counterfactuals non-trivial to define. In generative tasks, we define a counterfactual to be a minimally modified input that changes the expected output while preserving overall semantic consistency, such as modifying supporting context in a question to induce a different answer (Guo et al., 2023; Longpre et al., 2021; Paranjape et al., 2022; Köksal et al., 2023; Yu et al., 2023). In this work, we propose Counterfactual-guided Few-shot distillation in open-book Question Answering (CofFeQA), a novel strategy for few-shot distillation that aptly uses counterfactuals to provide additional supervision around challenging factual samples near the teacher’s decision boundary. We demonstrate how selecting high-quality counterfactuals can guide knowledge transfer more effectively using extremely limited few-shot budget. Our main contributions are as follows: Figure 1: Counterfactual-guided few-shot distillation framework. We begin with (1) a few-shot factual QA dataset fD_f and (2) generate a counterfactual candidate for a factual sample. (3) The generated counterfactual is evaluated using the teacher model with (4) our proposed Context Following Difficulty (CFD) metric and accepted if it satisfies the CFD threshold τ. (5) The CFD-validated counterfactuals are then paired with their corresponding factuals and used to distill the student model. Counterfactual-guided few-shot distillation for generative tasks. We show that few-shot distillation becomes particularly challenging for factuals near the teacher’s decision boundary, where the teacher has a smaller margin between its preferred answer and competing alternatives. Motivated by this observation, we propose CofFeQA, a novel framework which leverages counterfactuals for few-shot distillation in open-book QA. Rather than relying solely on factual supervision, CofFeQA pairs challenging factuals with their counterfactuals that expose changes in the teacher’s response under minimal modifications to the input context, providing additional supervision around the teacher’s decision boundary. This reduces dependence on large factual training corpora and improves training efficiency under limited supervision (see Figure 1 for an overview). Geometric analysis of counterfactual-guided knowledge transfer. We first introduce a notion of decision boundaries for the teacher and student in the context-embedding space, motivated by the fact that variations in the input context can lead to different answers. Our formulation enables us to analyze distillation in open-book QA from a geometric perspective. Specifically, we show that factual-counterfactual pairs can act as probes for the student to approximate the local geometry of the teacher’s decision boundary. To quantify the deviation between the teacher and student, we use the Hausdorff distance (Rockafellar and Wets, 1998), which measures the worst-case deviation between their decision boundaries. We show in Proposition 1 that this deviation is bounded by the magnitude of the counterfactual perturbations and their coverage of the teacher’s decision boundary. Importantly, factuals near the teacher’s decision boundary can require smaller perturbations to induce a change in the teacher’s response, which can lead to tighter teacher-student boundary alignment. Context Following Difficulty (CFD) for counterfactual validation. Although we generate counterfactuals by perturbing the original input context to induce a targeted alternative answer, not all generated examples are truly valid with respect to the teacher. The teacher may not produce same target answers using the modified context or may still be able to infer them without actually relying on the context. Such cases weaken the intended causal relationship that the student was intended to learn from the teacher and may encourage shortcut learning. To address this issue, we introduce a new metric Context Following Difficulty (CFD) to evaluate generated counterfactuals before distillation. CFD jointly considers whether the teacher produces the intended alternative answer and whether that answer becomes more predictable when conditioned on the modified context. CofFeQA uses CFD-validated counterfactuals, ensuring that the resulting supervision better captures the teacher’s context-dependent behavior. Empirical evaluation. We conduct comprehensive experiments across five in-domain and three out-of-domain benchmarks using the T5 (Raffel et al., 2020) and Qwen2.5 (Bai et al., 2023) model families. The in-domain benchmarks include NaturalQuestions, SQuAD, NewsQA, TriviaQA, and HotpotQA, while the out-of-domain benchmarks include DROP, DuoRC, and RelationExtraction (Fisch et al., 2019). We evaluate CofFeQA with standard knowledge distillation (KD), sequence-level knowledge distillation (SeqKD), and layer-wise distillation (LWD) baselines, under the same few-shot training budget. Across these settings, CofFeQA outperforms factual-only baselines. For instance, on TriviaQA, CofFeQA improves over the KD baseline by 8.778.77 average EM points and 9.989.98 average F1 points under the T5 setting with 32 few-shot pairs. 1.1 Related Work Counterfactual Data Augmentation. Counterfactual data augmentation has been widely studied for improving robustness in neural networks (Kaushik et al., 2020; Sachdeva et al., 2024; Calderon et al., 2022; Egea et al., 2025; Mishra et al., 2021). Early approaches relied on human-generated counterfactuals (Kaushik et al., 2019; Gardner et al., 2020), while later works automated counterfactual generation through trained generators and editors (Wu et al., 2021; Ross et al., 2021; Chen et al., 2021; Tokpo and Calders, 2024) and more recently through prompting LLMs (Bhattacharjee et al., 2024), alongside efforts to evaluate the quality of generated counterfactuals (Youssef et al., 2024; McAleese and Keane, 2024). For example, VISION (Egea et al., 2025) uses LLM-generated counterfactuals to train graph neural networks for vulnerability classification. Other works study counterfactual augmentation through active learning (Balashankar et al., 2023) and fairness-aware pipelines (Tokpo and Calders, 2024). Counterfactual explanations have been studied for model extraction with theoretical guarantees (Dissanayake and Dutta, 2024; Dissanayake and Dutta, 2026) and for robustness to model changes (Noorani et al., 2025; Hamman et al., 2024). Counterfactual applications also extend to generative tasks, including machine translation (Liu et al., 2021) and dialogue (Ou et al., 2022). However, most of these works focus on neural network models. Counterfactual augmentation to fine-tune LLMs has received limited attention. CATfOOD (Sachdeva et al., 2024) uses LLM-generated counterfactuals to train smaller language models, while Rajagopal et al. (2022) use counterfactually augmented training data for pretrained summarization models. In contrast, we use factual-counterfactual pairs as structured supervision for few-shot distillation in generative open-book QA, where the teacher produces free-form answers rather than predictions over a predefined labels. Counterfactual Question Answering. Counterfactual reasoning has recently gained traction in question answering (QA) (Ross et al., 2022; Khashabi et al., 2020; Huang et al., 2025). DisentQA (Neeman et al., 2023) uses counterfactual supervision to separate parametric and contextual knowledge (Xu et al., 2024a; Wang et al., 2023a), while IfQA (Yu et al., 2023) provides a benchmark for QA under hypothetical scenarios that require reasoning beyond factual retrieval. Counterfactual reasoning has also been studied in multi-hop QA (Guo et al., 2023) and CofCA (Wu et al., 2024), where counterfactual inference is used to mitigate shortcut reasoning. Retrieval-guided approaches further construct counterfactuals through context and answer modifications (Paranjape et al., 2022; Dixit et al., 2022). While these works demonstrate the value of counterfactuals for improving robustness and reasoning in QA, they do not study how they can facilitate knowledge distillation. Distillation in LLMs. Knowledge Distillation (Hinton et al., 2015) aims to train smaller student models to mimic larger teacher models, with extensions to sequence-level distillation (Xu et al., 2024b; Kim and Rush, 2016; Gu et al., 2024; Yang et al., 2025), task-specific distillation (Dissanayake et al., 2024; Sanh et al., 2019; Team et al., 2024; Wang et al., 2025), and layer-wise distillation (Jiao et al., 2020). Few-shot distillation has been studied for transferring pretrained transformer knowledge in tabular classification (Dissanayake and Dutta, 2025) and multiple-choice question answering (Sutanto et al., 2025). In particular, Sutanto et al. (2025) use an LLM to generate multiple-choice questions and answer options, together with probability scores that are distilled into an encoder-only student model. However, their approach relies on a predefined answer-choice space, making it less directly applicable to generative open-book QA, where the student produces free-form answers. While generative and instruction-level distillation often rely on direct supervision or synthetic data (Hsieh et al., 2023; Wang et al., 2023b; Gu et al., 2024), only a few recent works have investigated counterfactuals for knowledge transfer. DISCO (Chen et al., 2023a) generates counterfactual examples using an LLM and employs a task-specific teacher to filter the generated examples, which are then used as an augmented dataset to train a smaller student model. Notably, DISCO is different from standard distillation via aligning representations since the teacher primarily serves as a data-filtering mechanism rather than directly transferring its predictive behavior to the student through a distillation loss. Moreover, its empirical evaluation focuses on natural language inference, a classification task with a predefined label space. Another related work Hamman et al. (2025) study counterfactual supervision for few-shot distillation, but also focus on classification settings. These approaches do not address generative tasks, where the outputs are open-ended. In contrast, we specifically focus on few-shot knowledge distillation for generative open-book QA. Our approach is motivated by the difficulty of distilling with factuals near the teacher’s decision boundary and uses counterfactuals to provide additional supervision around these challenging regions. We also provide new geometric insights on counterfactual-guided knowledge transfer and a new metric CFD to further check counterfactual validity in open-book QA. 2 CofFeQA: Counterfactual-Guided Few-Shot Distillation Our strategy is motivated by the observation that factual samples near the teacher’s decision boundary are particularly challenging to distill under limited supervision. We use margin sampling (Scheffer et al., 2001) as an operational proxy for boundary proximity, where smaller teacher prediction margins indicate boundary near samples. Empirically, we observe that distillation performance degrades as this margin decreases; a detailed analysis is provided in Section C. Motivated by this observation, CofFeQA pairs boundary-near factuals with their counterfactuals that expose how the teacher’s response changes under minimal variations of the input context. We first define and describe the counterfactual generation process, and then geometrically analyze how factual-counterfactual pairs facilitate teacher-student boundary alignment. Finally, we introduce Context Following Difficulty (CFD) to validate generated counterfactuals and present the overall CofFeQA distillation framework. 2.1 Counterfactual Definition and Generation We define counterfactual generation for open-book QA as constructing minimally modified training examples that preserve the original question while changing the evidence to support a different answer. Given a triplet (Q,C,A)(Q,C,A), where Q is the question, C is the context, and A is the answer, the counterfactual generation framework constructs a counterfactual by modifying the context using an alternative answer. The resulting counterfactual triplet is denoted as (Q,C′,A′)(Q,C ,A ). We generate counterfactuals using GPT-4o-mini (OpenAI, 2024) in three stages. First, the answer A is classified into one of five entity types: person, location, organization, date, or numeric; samples outside these categories are excluded. Second, GPT-4o-mini generates an alternative answer A′A of the same entity type that is similar in category but distinct from A. Previously generated alternatives are excluded to encourage diversity across generation attempts. Finally, GPT-4o-mini is provided with the original triplet (Q,C,A)(Q,C,A) and A′A to generate the counterfactual context C′C by minimally editing C. Specifically, occurrences of A are replaced with A′A , together with any related references that must be changed to keep the passage grammatically consistent. As a result, the modified context C′C supports A′A as the answer to the same question Q, yielding the counterfactual triplet (Q,C′,A′)(Q,C ,A ). The system and user prompts used are shown in Figure 2. Additional prompt details are provided in Appendix D. Counterfactual Context Generation: System Prompt You are an expert at creating counterfactual reading-comprehension passages for QA training data. Given an original context that supports a specific answer, you rewrite it MINIMALLY so that it instead unambiguously supports a different target answer, by replacing occurrences of the original answer (and grammatically-dependent references to it) with the target answer, while leaving everything else in the passage unchanged. Do not add explanations or extra content. Counterfactual Context Generation: User Prompt Question: question, Original context (supports answer “original_answer”): original_context Target answer (the new counterfactual answer): “target_answer” Rewrite the context above so that it minimally changes to unambiguously support “target_answer” as the answer to the question, instead of “original_answer”. Replace occurrences of “original_answer” (and any grammatically-dependent references to it) with “target_answer”, keeping everything else in the passage exactly the same. Do not add commentary. Respond with ONLY: "reasoning": "<1-2 sentences on what you changed>", "counterfactual_context": "<the full rewritten context>" Figure 2: Counterfactual context generation prompts. System and user prompts used to minimally modify the factual context C into a counterfactual context C′C that supports the target answer A′A . 2.2 Geometric analysis of counterfactual-guided knowledge transfer We provide a geometric perspective on how factual-counterfactual pairs can facilitate knowledge transfer. We first introduce a notion of a decision boundary for open-book QA. While classification tasks typically consider a label-flipping boundary, we define a context-response boundary in the context-embedding space. In classification, the boundary separates regions where two classes have equal probabilities. In contrast, in open-book QA, it represents the transition region in context embedding space where the model shifts from generating one answer to another answer. In open-book QA, the context space may contain regions corresponding to the factual answer A, the intended counterfactual answer A′A , and other answer (A′A ) regions, different from both A and A′A corresponding to out-of-context or unrelated outputs. Our analysis focuses specifically on the boundary between the A and A′A response regions, as this is the boundary relevant to factual-counterfactual supervision. Our CFD-based validation helps identify and reject counterfactual candidates that induce responses corresponding to the other answer (A′A ) regions before distillation. We therefore restrict the geometric analysis to the boundary between factual answer space (A) and intended counterfactual answer (A′A ) space. We use the Hausdorff distance (Rockafellar and Wets, 1998) to quantify the worst-case deviation between the teacher context-response boundary and the student context-response boundary and derive an upper bound for this deviation in Proposition 1 (The proof can be found in Appendix E). Figure 3: Visualizing the Hausdorff distance between ℳtM_t and ℳsM_s. Definition of Hausdorff Distance (Rockafellar and Wets, 1998). Let (ℳ,d)(M,d) be a metric space. For each pair of non-empty subsets ℳt⊂ℳM_t and ℳs⊂ℳM_s , the Hausdorff distance between ℳtM_t and ℳsM_s is defined as: dH(ℳt,ℳs):=maxsupct∈ℳtinfcs∈ℳsd(ct,cs),supcs∈ℳsinfct∈ℳtd(ct,cs).d_H(M_t,M_s):= \ _c_t _t _c_s _sd(c_t,c_s), _c_s _s _c_t _td(c_t,c_s) \. This measures the maximum of the minimum distances between two sets, capturing the worst-case geometric deviation. (see Figure 3). Proposition 1 Let C be the space of context embeddings, c∈c arbitrary input context, Q∈Q input question, A∈A factual answer, and A′∈′A counterfactual answer. Consider a teacher and a student with response functions Gt(Q,c)=argmaxAPt(A|Q,c)G_t(Q,c)= _AP_t(A|Q,c) and Gs(Q,c)=argmaxAPs(A|Q,c)G_s(Q,c)= _AP_s(A|Q,c). The respective context-flipping boundaries can be defined as the sets: ℳt=c∈∣Pt(A|Q,c)=Pt(A′|Q,c),ℳs=c∈∣Ps(A|Q,c)=Ps(A′|Q,c).M_t=\c P_t(A|Q,c)=P_t(A |Q,c)\, _s=\c P_s(A|Q,c)=P_s(A |Q,c)\. Suppose there exists a factual-counterfactual paired dataset f+cf=(ci,ci′)i=1kD_f+cf=\(c_i,c_i )\_i=1^k for the above ℳtM_t that satisfies three conditions: (1)(1) Minimal context perturbation in context embedding space is bounded by α (i.e., ‖ci−ci′‖≤α||c_i-c_i ||≤α), (2)(2) Exact answer generation: Gt(Q,c)=Gs(Q,c)G_t(Q,c)=G_s(Q,c) where c=ci,ci′c=\c_i,c _i\, (3)(3) ϵε-spread boundary coverage: Counterfactuals are well-distributed across the context-response boundaries with spread ϵε, i.e., if t∈ℳt _t, then ∃ti∗such that‖t−ti∗‖≤ϵ∃ t_i^*\;such that\;\|t-t_i^*\|≤ε. This applies for ℳsM_s as well. i.e., if s∈ℳs _s, then ∃si∗such that‖s−si∗‖≤ϵ∃ s_i^*\;such that\;\|s-s_i^*\|≤ε. Then, the Hausdorff distance between the teacher boundary and student boundary satisfies: dH(ℳt,ℳs)≤α+ϵ.d_H(M_t,M_s)≤α+ε. Intuition behind proposition and assumptions. Proposition 1 highlights two properties of factual-counterfactual supervision that govern teacher-student boundary alignment: the magnitude of the counterfactual perturbations (α) and the coverage of the context-response boundary (ϵε). First, each factual-counterfactual pair provides supervision across a local change in the teacher’s response. Importantly, when the factual sample is already close to the teacher’s decision boundary, a smaller contextual perturbation is sufficient to cross the boundary and induce the alternative response A′A . Therefore, starting from boundary-near factual samples leads to a smaller upper bound on the perturbation α, tightening the bound. Although those samples are difficult under factual-only supervision, their proximity to the boundary makes them useful starting points for constructing local counterfactual supervision. Second, the boundary coverage condition captures how well the induced crossing points cover both the context-response boundaries. Better coverage corresponds to a smaller ϵε, tightening the bound. Thus, we provide a geometric characterization of how local counterfactual perturbations and boundary coverage relate to teacher-student boundary deviation. The analysis above also assumes that the student model agrees with the teacher model on the paired factual and counterfactual samples (Assumption (2) Exact answer generation). This idealized condition captures the objective of distillation, where the student is trained to reproduce the teacher’s behavior on the available supervision. More importantly, the analysis also relies on the generated counterfactual context C′C to induce the intended alternative answer A′A . In practice, this may not be guaranteed due to inherently stochastic and unpredictable generative behavior of LLMs Holtzman et al. (2019). The teacher may fail to produce A′A , or conditioning on C′C may provide only limited additional predictive benefit for A′A . For realization of this objective in our implementation, we therefore introduce a new metric Context Following Difficulty (CFD) in the following section to validate generated counterfactuals before they are used for knowledge distillation. Figure 4: Illustrative examples of CFD-based counterfactual evaluation from the NQ dataset. The four cases demonstrate how Answer Agreement (A) and Contextual Loss Ratio (CLR) complement each other in evaluating generated counterfactual candidates. 2.3 Context Following Difficulty (CFD) for counterfactual validation Generated counterfactual samples should be evaluated using the teacher to determine whether they induce the intended alternative answer A′A rather than another response. Since exact matching can be unreliable for generative tasks (Holtzman et al., 2019), we use token-level F1 (Rajpurkar et al., 2016) to measure agreement between the teacher output and A′A . However, answer agreement alone is insufficient, as generating the intended answer does not necessarily indicate that the C′C provides strong evidence for that prediction. A high-quality counterfactual should therefore not only induce the intended answer A′A , but also make A′A substantially more predictable when conditioned on C′C than from the Q alone. To capture these two properties, we introduce a new metric Context Following Difficulty (CFD), which combines Answer Agreement (A) with the Contextual Loss Ratio (CLR). First, AθA_θ measures whether the teacher generates the intended alternative answer A′A under the counterfactual context C′C : Aθ(Q,C′,A′)=F1(A^θ(Q,C′),A′),A_θ(Q,C ,A )=F1 ( A_θ(Q,C ),A ), (1) where A^θ(Q,C′) A_θ(Q,C ) is the answer generated by the teacher given Q and C′C . A higher AθA_θ indicates stronger agreement with the intended answer A′A . However, high answer agreement alone does not establish that the counterfactual context plays a role in predicting A′A . Inspired by the Instruction Following Difficulty (IFD) metric (Li et al., 2024), we define the Contextual Loss Ratio (CLR) as: CLRθ(Q,C′,A′)=Lθ(A′∣Q,C′)Lθ(A′∣Q).CLR_θ(Q,C ,A )= L_θ(A Q,C )L_θ(A Q). (2) A low CLRθCLR_θ indicates that conditioning on C′C substantially reduces the teacher’s loss for A′A relative to conditioning on Q alone, i.e., C′C plays a strong role in predicting the intended alternative answer. Conversely, a higher CLRθCLR_θ indicates that providing C′C yields only a limited reduction in the loss of A′A , suggesting weaker dependence of the target prediction on C′C . Thus, CLRθCLR_θ measures how strongly C′C contributes to the prediction of the teacher model A′A . Combining these two measures, we finally define our Context Following Difficulty (CFD) score as: CFDθ(Q,C′,A′)=CLRθ(Q,C′,A′)Aθ(Q,C′,A′)+δ,CFD_θ(Q,C ,A )= CLR_θ(Q,C ,A )A_θ(Q,C ,A )+δ, (3) Algorithm 1 CofFeQA 0: D, teacher θt _t, student θs _s, epochs E, learning rate η, weight α, CFD threshold τ, few-shot budget k, cf=ϕ,f=ϕD_cf=φ,D_f=φ 1: while |cf|<k/2|D_cf|<k/2 do 2: Pick (Q,C,A)∈(Q,C,A) 3: (Q,C′,A′)←GenCF(Q,C,A)(Q,C ,A ) (Q,C,A) 4: if CFDθt(Q,C′,A′)≤τCFD_ _t(Q,C ,A )≤τ then 5: cf←cf∪(Q,C′,A′)D_cf _cf∪\(Q,C ,A )\ 6: f←f∪(Q,C,A)D_f _f∪\(Q,C,A)\ 7: end if 8: end while 9: for e=1e=1 to E do 10: for all (Q,C,A)∈f∪cf(Q,C,A) _f _cf do 11: ℒCE←CE(θs(Q,C),A)L_CE ( _s(Q,C),A) 12: ℒKD←KL(θt(Q,C)∥θs(Q,C))L_KD ( _t(Q,C) _s(Q,C)) 13: ℒ←αℒCE+(1−α)ℒKDL← _CE+(1-α)L_KD 14: Update θs←θs−η∇θsℒ _s← _s-η _ _sL 15: end for 16: end for 17: return distilled student θs _s where δ>0δ>0 is a small constant for numerical stability. We prefer low CLRθCLR_θ and high AθA_θ. A lower CFDθCFD_θ score favors counterfactuals for which the teacher both generates an answer close to the intended target A′A and relies more strongly on the counterfactual context when predicting A′A . Figure 4 illustrates the complementary roles of AθA_θ and CLRθCLR_θ in four representative cases. Cases (a) and (b) show that a counterfactual may fail to induce the intended answer even when the modified context contains or supports it. Case (c) has high AθA_θ but high CLRθCLR_θ, indicating limited additional predictive benefit from C′C . In contrast, case (c) represents a valid counterfactual with a low CFDθCFD_θ. Together, these examples show why both AθA_θ and CLRθCLR_θ are necessary for validation. The two components of CFDθCFD_θ therefore capture complementary properties of a valid and informative counterfactual. AθA_θ verifies the intended answer A′A , while CLRθCLR_θ measures how strongly the modified context contributes to predicting A′A . Our algorithm is provided in Algorithm 1 (also in Figure 1). 3 Experiments Datasets. We evaluate CofFeQA on eight open-domain question answering benchmarks. The five in-domain benchmarks are Natural Questions (NQ) (Kwiatkowski et al., 2019), SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al., 2017), and HotpotQA (Yang et al., 2018). The three out-of-domain benchmarks are DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), and RelationExtraction (Levy et al., 2017). Additional details about the datasets used in our experimental evaluations are provided in Appendix F.1 Models. We evaluate our method using two model families with different architectures: the encoder-decoder model T5 (Raffel et al., 2020) and the decoder-only model Qwen2.5 (Bai et al., 2023). In the T5 setting, T5-Large (770M parameters) serves as the teacher, while T5-Base (220M parameters) serves as the student. In the Qwen2.5 setting, Qwen2.5-7B-IT serves as the teacher, while Qwen2.5-1.5B-IT serves as the student. Full-training details are in Appendix F.2. Table 1: In-domain F1 score comparison across k-shots for T5 and Qwen2.5 models using baselines. Mean F1 scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. baseline). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg NQ KD 61.830.49_ 0.60.49 61.442.08_ 0.62.08 60.341.91_ 0.61.91 56.913.20_ 0.63.20 58.071.33_ 0.61.33 59.722.15_ 0.62.15 66.910.44_ 0.60.44 66.010.59_ 0.60.59 65.710.39_ 0.60.39 64.680.61_ 0.60.61 65.660.69_ 0.60.69 65.790.80_ 0.60.80 + CofFeQA 62.160.27_ 0.60.27 62.091.30_ 0.61.30 60.851.37_ 0.61.37 59.082.01_ 0.62.01 58.632.30_ 0.62.30 60.561.65_ 0.61.65 66.270.27_ 0.60.27 68.100.20_ 0.60.20 67.230.32_ 0.60.32 67.250.73_ 0.60.73 67.190.99_ 0.60.99 67.210.65_ 0.60.65 LWD 61.810.56_ 0.60.56 61.232.49_ 0.62.49 60.371.56_ 0.61.56 58.161.81_ 0.61.81 58.722.29_ 0.62.29 60.061.57_ 0.61.57 65.625.03_ 0.65.03 66.320.94_ 0.60.94 65.901.18_ 0.61.18 65.610.85_ 0.60.85 66.620.86_ 0.60.86 66.010.45_ 0.60.45 + CofFeQA 62.120.22_ 0.60.22 61.462.57_ 0.62.57 60.042.09_ 0.62.09 60.031.54_ 0.61.54 58.932.71_ 0.62.71 60.521.27_ 0.61.27 63.984.96_ 0.64.96 68.160.35_ 0.60.35 67.780.64_ 0.60.64 68.070.56_ 0.60.56 67.960.77_ 0.60.77 67.191.80_ 0.61.80 SeqKD 60.681.24_ 0.61.24 59.191.07_ 0.61.07 60.801.12_ 0.61.12 59.781.57_ 0.61.57 62.091.54_ 0.61.54 60.511.11_ 0.61.11 65.890.34_ 0.60.34 67.550.55_ 0.60.55 66.840.31_ 0.60.31 67.330.35_ 0.60.35 67.730.76_ 0.60.76 67.070.74_ 0.60.74 + CofFeQA 61.510.69_ 0.60.69 61.880.64_ 0.60.64 61.520.92_ 0.60.92 61.870.81_ 0.60.81 61.361.41_ 0.61.41 61.620.23_ 0.60.23 66.560.42_ 0.60.42 66.920.74_ 0.60.74 68.480.89_ 0.60.89 68.040.58_ 0.60.58 68.220.39_ 0.60.39 67.640.85_ 0.60.85 SQuAD KD 86.450.54_ 0.60.54 84.181.39_ 0.61.39 83.631.71_ 0.61.71 84.051.14_ 0.61.14 83.101.19_ 0.61.19 84.281.28_ 0.61.28 77.750.13_ 0.60.13 76.780.24_ 0.60.24 76.400.47_ 0.60.47 75.870.41_ 0.60.41 76.450.39_ 0.60.39 76.650.70_ 0.60.70 + CofFeQA 86.560.16_ 0.60.16 86.130.72_ 0.60.72 85.361.03_ 0.61.03 85.460.48_ 0.60.48 84.231.58_ 0.61.58 85.550.89_ 0.60.89 76.950.46_ 0.60.46 77.280.59_ 0.60.59 77.420.42_ 0.60.42 76.640.85_ 0.60.85 75.950.34_ 0.60.34 76.850.59_ 0.60.59 LWD 86.470.50_ 0.60.50 84.131.13_ 0.61.13 83.861.78_ 0.61.78 83.450.83_ 0.60.83 82.921.18_ 0.61.18 84.171.37_ 0.61.37 76.700.65_ 0.60.65 76.960.30_ 0.60.30 76.710.41_ 0.60.41 76.151.00_ 0.61.00 77.201.07_ 0.61.07 76.740.39_ 0.60.39 + CofFeQA 86.560.23_ 0.60.23 86.090.78_ 0.60.78 84.850.81_ 0.60.81 85.570.49_ 0.60.49 84.531.65_ 0.61.65 85.520.84_ 0.60.84 76.360.06_ 0.60.06 77.190.24_ 0.60.24 77.750.41_ 0.60.41 77.220.33_ 0.60.33 77.821.50_ 0.61.50 77.270.58_ 0.60.58 SeqKD 85.430.69_ 0.60.69 86.250.67_ 0.60.67 85.570.28_ 0.60.28 85.450.32_ 0.60.32 84.960.52_ 0.60.52 85.530.46_ 0.60.46 78.220.46_ 0.60.46 77.790.17_ 0.60.17 77.900.51_ 0.60.51 77.350.49_ 0.60.49 78.260.96_ 0.60.96 77.900.37_ 0.60.37 + CofFeQA 86.410.45_ 0.60.45 85.470.51_ 0.60.51 85.730.73_ 0.60.73 85.650.36_ 0.60.36 85.020.37_ 0.60.37 85.660.50_ 0.60.50 77.250.37_ 0.60.37 77.890.33_ 0.60.33 77.800.65_ 0.60.65 77.850.53_ 0.60.53 78.630.67_ 0.60.67 77.920.49_ 0.60.49 NewsQA KD 72.722.33_ 0.62.33 73.431.92_ 0.61.92 70.562.16_ 0.62.16 69.952.11_ 0.62.11 65.352.07_ 0.62.07 70.403.17_ 0.63.17 57.320.77_ 0.60.77 54.141.48_ 0.61.48 54.321.11_ 0.61.11 55.101.69_ 0.61.69 53.641.71_ 0.61.71 54.901.45_ 0.61.45 + CofFeQA 73.663.02_ 0.63.02 74.261.18_ 0.61.18 75.431.71_ 0.61.71 74.191.38_ 0.61.38 72.852.24_ 0.62.24 74.080.94_ 0.60.94 60.770.73_ 0.60.73 57.950.82_ 0.60.82 56.880.60_ 0.60.60 57.481.17_ 0.61.17 59.412.54_ 0.62.54 58.501.58_ 0.61.58 LWD 72.672.30_ 0.62.30 73.002.05_ 0.62.05 70.422.59_ 0.62.59 69.292.49_ 0.62.49 65.250.79_ 0.60.79 70.123.14_ 0.63.14 58.761.01_ 0.61.01 58.620.68_ 0.60.68 56.900.93_ 0.60.93 57.511.55_ 0.61.55 55.722.27_ 0.62.27 57.501.26_ 0.61.26 + CofFeQA 73.433.03_ 0.63.03 74.980.66_ 0.60.66 74.851.56_ 0.61.56 73.771.18_ 0.61.18 73.062.36_ 0.62.36 74.020.86_ 0.60.86 60.490.88_ 0.60.88 58.181.05_ 0.61.05 58.051.13_ 0.61.13 59.890.48_ 0.60.48 61.234.39_ 0.64.39 59.571.41_ 0.61.41 SeqKD 66.771.63_ 0.61.63 70.091.72_ 0.61.72 70.213.11_ 0.63.11 69.451.83_ 0.61.83 72.160.81_ 0.60.81 69.741.94_ 0.61.94 49.101.20_ 0.61.20 51.491.04_ 0.61.04 57.600.49_ 0.60.49 58.971.25_ 0.61.25 61.611.31_ 0.61.31 55.755.25_ 0.65.25 + CofFeQA 72.721.51_ 0.61.51 74.651.26_ 0.61.26 75.991.48_ 0.61.48 76.541.88_ 0.61.88 75.771.46_ 0.61.46 75.131.51_ 0.61.51 51.221.13_ 0.61.13 55.650.88_ 0.60.88 56.900.95_ 0.60.95 60.141.62_ 0.61.62 63.381.95_ 0.61.95 57.464.60_ 0.64.60 TriviaQA KD 53.704.24_ 0.64.24 50.403.27_ 0.63.27 46.492.23_ 0.62.23 45.802.86_ 0.62.86 46.931.44_ 0.61.44 48.663.33_ 0.63.33 61.571.09_ 0.61.09 61.980.79_ 0.60.79 59.921.06_ 0.61.06 57.501.14_ 0.61.14 59.261.01_ 0.61.01 60.051.82_ 0.61.82 + CofFeQA 60.061.45_ 0.61.45 56.322.19_ 0.62.19 56.477.04_ 0.67.04 50.541.92_ 0.61.92 49.521.77_ 0.61.77 54.584.43_ 0.64.43 62.490.42_ 0.60.42 60.580.69_ 0.60.69 62.320.67_ 0.60.67 60.530.74_ 0.60.74 57.961.49_ 0.61.49 60.771.83_ 0.61.83 LWD 53.733.89_ 0.63.89 51.862.61_ 0.62.61 44.612.48_ 0.62.48 45.081.90_ 0.61.90 47.191.42_ 0.61.42 48.504.10_ 0.64.10 61.100.41_ 0.60.41 61.201.11_ 0.61.11 59.260.40_ 0.60.40 58.070.42_ 0.60.42 58.861.29_ 0.61.29 59.701.40_ 0.61.40 + CofFeQA 59.701.51_ 0.61.51 56.492.51_ 0.62.51 56.357.32_ 0.67.32 50.461.88_ 0.61.88 49.671.09_ 0.61.09 54.534.30_ 0.64.30 62.250.68_ 0.60.68 61.490.79_ 0.60.79 61.760.50_ 0.60.50 60.211.16_ 0.61.16 57.871.39_ 0.61.39 60.721.76_ 0.61.76 SeqKD 59.013.46_ 0.63.46 52.991.83_ 0.61.83 50.192.62_ 0.62.62 50.532.14_ 0.62.14 48.611.73_ 0.61.73 52.274.08_ 0.64.08 61.361.02_ 0.61.02 59.990.54_ 0.60.54 60.520.81_ 0.60.81 59.851.07_ 0.61.07 60.581.30_ 0.61.30 60.460.60_ 0.60.60 + CofFeQA 60.702.14_ 0.62.14 55.111.75_ 0.61.75 54.791.59_ 0.61.59 51.053.29_ 0.63.29 53.401.24_ 0.61.24 55.013.56_ 0.63.56 63.130.84_ 0.60.84 62.051.05_ 0.61.05 61.760.76_ 0.60.76 60.170.64_ 0.60.64 59.860.66_ 0.60.66 61.391.36_ 0.61.36 HotpotQA KD 70.372.43_ 0.62.43 70.963.32_ 0.63.32 64.273.26_ 0.63.26 70.231.10_ 0.61.10 69.423.49_ 0.63.49 69.052.73_ 0.62.73 74.090.48_ 0.60.48 75.350.53_ 0.60.53 72.731.19_ 0.61.19 73.150.57_ 0.60.57 73.100.97_ 0.60.97 73.681.06_ 0.61.06 + CofFeQA 69.873.08_ 0.63.08 73.392.01_ 0.62.01 71.311.26_ 0.61.26 70.651.90_ 0.61.90 72.612.43_ 0.62.43 71.561.43_ 0.61.43 73.380.92_ 0.60.92 74.790.50_ 0.60.50 76.440.65_ 0.60.65 75.690.61_ 0.60.61 76.171.07_ 0.61.07 75.291.24_ 0.61.24 LWD 69.892.41_ 0.62.41 70.773.76_ 0.63.76 64.113.08_ 0.63.08 69.881.68_ 0.61.68 70.442.77_ 0.62.77 69.022.77_ 0.62.77 74.200.49_ 0.60.49 74.450.78_ 0.60.78 72.400.86_ 0.60.86 74.360.54_ 0.60.54 74.071.84_ 0.61.84 73.900.85_ 0.60.85 + CofFeQA 69.213.32_ 0.63.32 74.091.82_ 0.61.82 72.041.56_ 0.61.56 70.502.17_ 0.62.17 73.282.23_ 0.62.23 71.821.99_ 0.61.99 73.870.91_ 0.60.91 74.770.41_ 0.60.41 74.700.53_ 0.60.53 75.750.89_ 0.60.89 77.320.82_ 0.60.82 75.281.32_ 0.61.32 SeqKD 75.810.84_ 0.60.84 75.650.88_ 0.60.88 75.791.16_ 0.61.16 74.162.96_ 0.62.96 74.223.15_ 0.63.15 75.130.86_ 0.60.86 75.370.66_ 0.60.66 76.490.78_ 0.60.78 75.260.50_ 0.60.50 72.290.71_ 0.60.71 73.591.65_ 0.61.65 74.601.66_ 0.61.66 + CofFeQA 74.840.98_ 0.60.98 76.990.54_ 0.60.54 75.841.63_ 0.61.63 75.032.36_ 0.62.36 73.143.32_ 0.63.32 75.171.41_ 0.61.41 73.670.66_ 0.60.66 74.071.30_ 0.61.30 76.230.70_ 0.60.70 76.040.56_ 0.60.56 74.901.46_ 0.61.46 74.981.14_ 0.61.14 KD 69.0112.31_ 0.612.31 68.0812.77_ 0.612.77 65.0613.63_ 0.613.63 65.3914.56_ 0.614.56 64.5813.42_ 0.613.42 66.4213.25_ 0.613.25 67.538.48_ 0.68.48 66.859.44_ 0.69.44 65.819.04_ 0.69.04 65.269.20_ 0.69.20 65.629.45_ 0.69.45 66.219.09_ 0.69.09 + CofFeQA 70.4610.57_ 0.610.57 70.4411.60_ 0.611.60 69.8811.55_ 0.611.55 67.9813.55_ 0.613.55 67.5713.57_ 0.613.57 69.2712.09_ 0.612.09 67.976.97_ 0.66.97 67.748.48_ 0.68.48 68.068.89_ 0.68.89 67.528.66_ 0.68.66 67.348.70_ 0.68.70 67.728.28_ 0.68.28 LWD 68.9112.29_ 0.612.29 68.2012.24_ 0.612.24 64.6714.34_ 0.614.34 65.1714.37_ 0.614.37 64.9013.30_ 0.613.30 66.3713.20_ 0.613.20 67.287.91_ 0.67.91 67.518.03_ 0.68.03 66.248.42_ 0.68.42 66.348.77_ 0.68.77 66.499.31_ 0.69.31 66.778.47_ 0.68.47 AVG + CofFeQA 70.2110.66_ 0.610.66 70.6211.77_ 0.611.77 69.6311.55_ 0.611.55 68.0713.41_ 0.613.41 67.8913.65_ 0.613.65 69.2812.11_ 0.612.11 67.397.21_ 0.67.21 67.968.20_ 0.68.20 68.018.34_ 0.68.34 68.238.24_ 0.68.24 68.449.09_ 0.69.09 68.008.12_ 0.68.12 SeqKD 69.5411.05_ 0.611.05 68.8413.18_ 0.613.18 68.5113.62_ 0.613.62 67.8713.39_ 0.613.39 68.4113.73_ 0.613.73 68.6312.87_ 0.612.87 65.9911.67_ 0.611.67 66.6611.13_ 0.611.13 67.628.88_ 0.68.88 67.167.92_ 0.67.92 68.357.61_ 0.67.61 67.169.30_ 0.69.30 + CofFeQA 71.2410.62_ 0.610.62 70.8212.19_ 0.612.19 70.7712.42_ 0.612.42 70.0313.59_ 0.613.59 69.7412.44_ 0.612.44 70.5212.16_ 0.612.16 66.3710.14_ 0.610.14 67.328.96_ 0.68.96 68.279.08_ 0.69.08 68.458.42_ 0.68.42 69.007.80_ 0.67.80 67.888.69_ 0.68.69 Baselines. We compare CofFeQA against three distillation baselines: (i) Standard Knowledge Distillation (KD) (Hinton et al., 2015), (i) Sequence-Level Knowledge Distillation (seqKD) (Kim and Rush, 2016), and (i) Layer-Wise Distillation (LWD) (Jiao et al., 2020). All methods use the same few-shot splits for fair comparison. Additional details of the baselines and implementation details are provided in Appendix F.3. Setup. Following prior work (Hamman et al., 2025), we first fine-tune the teacher model on the full training dataset. For each few-shot budget k∈16,32,64,128,256k∈\16,32,64,128,256\, we rank factual samples by their teacher prediction margins, using margin sampling (Scheffer et al., 2001). The factual-only distillation baseline uses the k samples with the smallest margins. CofFeQA instead uses the k/2k/2 factual samples with the smallest margins together with their corresponding k/2k/2 counterfactuals, maintaining the same total few-shot budget k. We evaluate accuracy using EM and F1 against the gold answers and teacher-student agreement using EM and F1 between their generated answers. All results are reported as the mean and standard deviation over five random seeds. 3.1 Main Experimental Results CofFeQA outperforms the baselines. As shown in Table 1, CofFeQA improves performance across all three distillation baselines under the T5 setting. When averaged across all datasets and k-shot settings, CofFeQA improves the F1 score from 66.4266.42 to 69.2769.27 with standard KD, from 66.3766.37 to 69.2869.28 with LWD, and from 68.6368.63 to 70.5270.52 with SeqKD. Moreover, CofFeQA exhibits lower standard deviations than the corresponding baselines in many settings, indicating more stable performance across seeds. These results demonstrate that counterfactual supervision provides consistent benefits over the baselines. Detailed EM scores are provided in Table 8 in Appendix H.1. CofFeQA generalizes across different model families. CofFeQA improves standard KD across both the encoder-decoder T5 and decoder-only Qwen2.5 model families, where the teachers are approximately 3.5×3.5× and 4.7×4.7× larger than their respective students. Under Qwen2.5, CofFeQA improves the average F1 score from 66.2166.21 to 67.7267.72 with standard KD, from 66.7766.77 to 68.0068.00 with LWD, and from 67.16 to 67.88 with SeqKD. These results indicate that the benefits of CofFeQA extend across different model architectures and scales. Students mimic the teacher more closely under CofFeQA. To examine how closely the student follows the teacher, we measure teacher-student agreement using EM and F1. As shown in Figure 5, CofFeQA generally achieves higher teacher-student agreement than standard KD across datasets and k-shot settings. Averaged across all benchmarks and settings, CofFeQA improves F1 agreement under standard KD from 68.5168.51 to 71.5071.50 under T5, with a similar trend for Qwen2.5. These results suggest that counterfactual supervision helps the student more closely reproduce the teacher’s output behavior. Detailed results and additional radar plots are provided in Appendix H.2. Figure 5: Teacher-student F1 agreement of standard KD and CofFeQA across datasets and k-shot settings using T5. CofFeQA generally achieves higher agreement than the baseline. Spokes denote k-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization. CofFeQA generalizes well to out-of-domain datasets. To evaluate out-of-domain generalization, we use Natural Questions for distillation and evaluate the resulting students on three out-of-domain test sets: DROP, DuoRC, and RelationExtraction. As shown in Table 2, CofFeQA improves over standard KD when averaged across datasets and k-shot settings for both model families. For T5 and Qwen2.5, CofFeQA improves the overall F1 score over standard KD from 53.3553.35 to 54.5254.52 and from 57.0957.09 to 57.7757.77, respectively. These results indicate that the benefits of counterfactual supervision extend beyond the in-domain datasets used for distillation. Detailed EM scores are in Appendix H.3. Table 2: Out-of-domain F1 score comparison across k-shots for T5 and Qwen2.5 models using KD. Mean F1 scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. KD). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg DROP KD 38.561.72_ 0.61.72 35.813.76_ 0.63.76 32.482.47_ 0.62.47 28.063.65_ 0.63.65 27.781.41_ 0.61.41 32.544.73_ 0.64.73 49.581.13_ 0.61.13 44.921.31_ 0.61.31 42.070.45_ 0.60.45 41.280.67_ 0.60.67 42.570.94_ 0.60.94 44.083.36_ 0.63.36 + CofFeQA 39.241.14_ 0.61.14 37.541.20_ 0.61.20 32.412.71_ 0.62.71 32.292.00_ 0.62.00 29.441.53_ 0.61.53 34.194.06_ 0.64.06 48.580.79_ 0.60.79 49.531.23_ 0.61.23 45.201.31_ 0.61.31 42.651.43_ 0.61.43 41.360.60_ 0.60.60 45.463.58_ 0.63.58 DuoRC KD 50.381.23_ 0.61.23 48.493.56_ 0.63.56 46.982.07_ 0.62.07 42.521.73_ 0.61.73 44.461.80_ 0.61.80 46.573.13_ 0.63.13 41.730.77_ 0.60.77 44.841.29_ 0.61.29 43.381.30_ 0.61.30 42.771.16_ 0.61.16 42.711.49_ 0.61.49 43.081.15_ 0.61.15 + CofFeQA 51.660.56_ 0.60.56 50.581.02_ 0.61.02 44.552.64_ 0.62.64 46.141.23_ 0.61.23 44.721.51_ 0.61.51 47.533.36_ 0.63.36 41.041.11_ 0.61.11 45.010.92_ 0.60.92 43.110.98_ 0.60.98 43.251.00_ 0.61.00 44.391.42_ 0.61.42 43.361.52_ 0.61.52 RelationExtraction KD 82.960.99_ 0.60.99 83.300.46_ 0.60.46 82.650.61_ 0.60.61 78.302.66_ 0.62.66 77.531.47_ 0.61.47 80.952.79_ 0.62.79 85.320.45_ 0.60.45 85.340.49_ 0.60.49 84.360.25_ 0.60.25 82.770.71_ 0.60.71 82.740.19_ 0.60.19 84.111.30_ 0.61.30 + CofFeQA 82.980.39_ 0.60.39 82.990.65_ 0.60.65 83.221.22_ 0.61.22 80.451.51_ 0.61.51 79.551.50_ 0.61.50 81.841.71_ 0.61.71 84.710.55_ 0.60.55 85.660.33_ 0.60.33 85.620.18_ 0.60.18 83.610.77_ 0.60.77 82.880.80_ 0.60.80 84.501.23_ 0.61.23 KD 57.3023.00_ 0.623.00 55.8724.59_ 0.624.59 54.0425.82_ 0.625.82 49.6325.86_ 0.625.86 49.9225.32_ 0.625.32 53.3524.91_ 0.624.91 58.8823.24_ 0.623.24 58.3723.36_ 0.623.36 56.6024.05_ 0.624.05 55.6123.54_ 0.623.54 56.0023.15_ 0.623.15 57.0923.40_ 0.623.40 AVG + CofFeQA 57.9622.54_ 0.622.54 57.0323.40_ 0.623.40 53.3926.53_ 0.626.53 52.9624.79_ 0.624.79 51.2425.69_ 0.625.69 54.5224.58_ 0.624.58 58.1123.34_ 0.623.34 60.0722.28_ 0.622.28 57.9723.96_ 0.623.96 56.5023.48_ 0.623.48 56.2123.15_ 0.623.15 57.7723.17_ 0.623.17 CofFeQA improves performance under modified contextual evidence. Following Longpre et al. (2021), we replace each test answer in the context with a randomly selected answer of the same entity type and evaluate against the modified target. As shown in Table 13, CofFeQA improves average F1 agreement over standard KD from 66.7266.72 to 69.5569.55 for T5 and from 49.2349.23 to 53.5553.55 for Qwen2.5. This suggests that CofFeQA better adapts predictions to modified contextual evidence when it differs from pre-trained knowledge. Corresponding EM results are provided in Appendix H.4. 3.2 Ablation Studies We conduct three ablation studies to examine different components of CofFeQA: (i) We compare CofFeQA with a variant that uses generated counterfactuals without CFD verification to isolate the contribution of CFD; (i) We vary the CFD threshold τ to analyze its effect on downstream performance and teacher-student agreement; and (i) We measure counterfactual generation costs, training time and energy consumption across baselines to analyze the computational cost of CofFeQA. Full ablation results and experimental details are in Appendix G. 4 Conclusion In this paper, we present a counterfactual-guided knowledge distillation method for few-shot open-book question answering. Motivated by the difficulty of distilling with factual samples near the teacher’s decision boundary, our approach uses factual-counterfactual pairs to provide additional supervision around these challenging regions. We further introduce CFD to validate counterfactuals before distillation. Experiments across multiple datasets and LLM architectures demonstrate improved knowledge transfer under limited supervision. Limitations and future directions are discussed in Appendix B. Overall, our findings highlight the potential of counterfactual supervision for improving knowledge transfer in generative QA under limited supervision in the few-shot setting. Reproducibility statement To support reproducibility, we provide detailed descriptions of the datasets, training procedures, evaluation settings, and baseline implementations in Appendix F. Specifically, Appendix F.1 describes the datasets and experimental splits, Appendix F.2 provides details of teacher model fine-tuning and evaluation, and Appendix F.3 describes the baseline formulations and implementation details. We also provide the assumptions and complete proof of our geometric analysis in Appendix E. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. Bai et al. (2023) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: §1, §3. Balashankar et al. (2023) A. Balashankar, X. Wang, Y. Qin, B. Packer, N. Thain, E. H. Chi, J. Chen, and A. Beutel Improving classifier robustness through active generative counterfactual data augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 127–139. Cited by: §1.1. Bhattacharjee et al. (2024) A. Bhattacharjee, R. Moraffah, J. Garland, and H. Liu Zero-shot llm-guided counterfactual generation: a case study on nlp model evaluation. In 2024 IEEE International Conference on Big Data (BigData), p. 1243–1248. Cited by: §1.1, §1. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §1. Calderon et al. (2022) N. Calderon, E. Ben-David, A. Feder, and R. Reichart DoCoGen: domain counterfactual generation for low resource domain adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7727–7746. Cited by: §1.1. Chen et al. (2021) H. Chen, R. Xia, and J. Yu Reinforced counterfactual data augmentation for dual sentiment classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 269–278. Cited by: §1.1. Chen et al. (2025) Y. Chen, X. Xu, F. de Hoog, J. Liu, and S. Wang Medium-difficulty samples constitute smoothed decision boundary for knowledge distillation on pruned datasets. In International Conference on Learning Representations, Vol. 2025, p. 54106–54123. Cited by: §1. Chen et al. (2023a) Z. Chen, Q. Gao, A. Bosselut, A. Sabharwal, and K. Richardson DISCO: distilling counterfactuals with large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5514–5528. Cited by: §1.1. Chen et al. (2023b) Z. Chen, L. Hu, W. Li, Y. Shao, and L. Nie Causal intervention and counterfactual reasoning for multi-modal fake news detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 627–638. Cited by: §1. Chung et al. (2024) H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research 25 (70), p. 1–53. Cited by: §1. Dissanayake and Dutta (2024) P. Dissanayake and S. Dutta Model reconstruction using counterfactual explanations: a perspective from polytope theory. Advances in Neural Information Processing Systems 37, p. 83397–83429. Cited by: §1.1. Dissanayake and Dutta (2025) P. Dissanayake and S. Dutta Tabdistill: distilling transformers into neural nets for few-shot tabular classification. arXiv preprint arXiv:2511.05704. Cited by: §1.1, §1. Dissanayake and Dutta (2026) P. Dissanayake and S. Dutta Foundations of efficient model reconstruction using counterfactual explanations. IEEE Journal on Selected Areas in Information Theory. Cited by: §1.1. Dissanayake et al. (2024) P. Dissanayake, F. Hamman, B. Halder, I. Sucholutsky, Q. Zhang, and S. Dutta Quantifying knowledge distillation using partial information decomposition. arXiv preprint arXiv:2411.07483. Cited by: §1.1, §1. Dixit et al. (2022) T. Dixit, B. Paranjape, H. Hajishirzi, and L. Zettlemoyer Core: a retrieve-then-edit framework for counterfactual data generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, p. 2964–2984. Cited by: §1.1. Dua et al. (2019) D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 2368–2378. Cited by: §F.1, §3. Egea et al. (2025) D. Egea, B. Halder, and S. Dutta Vision: robust and interpretable code vulnerability detection leveraging counterfactual augmentation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 8, p. 812–823. Cited by: §1.1, §1. Fang et al. (2026) L. Fang, X. Yu, J. Cai, Y. Chen, S. Wu, Z. Liu, Z. Yang, H. Lu, X. Gong, Y. Liu, et al. Knowledge distillation and dataset distillation of large language models: emerging trends, challenges, and future directions. Artificial Intelligence Review 59 (1), p. 17. Cited by: §1. Fisch et al. (2019) A. Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, and D. Chen MRQA 2019 shared task: evaluating generalization in reading comprehension. In Proceedings of the 2nd workshop on machine reading for question answering, p. 1–13. Cited by: §1. Fu et al. (2026) Y. Fu, F. Hamman, and S. Dutta T-shirt: token-selective hierarchical data selection for instruction tuning. Advances in Neural Information Processing Systems 38, p. 113932–113958. Cited by: §1. Gardner et al. (2020) M. Gardner, Y. Artzi, V. Basmov, J. Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y. Elazar, A. Gottumukkala, et al. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, p. 1307–1323. Cited by: §1.1, §1. Gou et al. (2021) J. Gou, B. Yu, S. J. Maybank, and D. Tao Knowledge distillation: a survey. International Journal of Computer Vision 129 (6), p. 1789–1819. Cited by: §1. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang Minillm: knowledge distillation of large language models. In The twelfth international conference on learning representations, Cited by: §1.1. Guo et al. (2023) W. Guo, Q. Gong, Y. Rao, and H. Lai Counterfactual multihop qa: a cause-effect approach for reducing disconnected reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4214–4226. Cited by: §1.1, §1. Hamman et al. (2025) F. Hamman, P. Dissanayake, Y. Fu, and S. Dutta Few-shot knowledge distillation of llms with counterfactual explanations. arXiv preprint arXiv:2510.21631. Cited by: Appendix E, §1.1, §1, §3. Hamman et al. (2024) F. Hamman, E. Noorani, S. Mishra, D. Magazzeni, and S. Dutta Robust algorithmic recourse under model multiplicity with probabilistic guarantees. IEEE Journal on Selected Areas in Information Theory 5, p. 357–368. Cited by: §1.1. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: 1st item, §1.1, §1, §3. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 10. Cited by: §1. Holtzman et al. (2019) A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Cited by: §2.2, §2.3. Hsieh et al. (2023) C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, p. 8003–8017. Cited by: §1.1. Huang et al. (2025) L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), p. 1–55. Cited by: §1.1. Jiao et al. (2020) X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu Tinybert: distilling bert for natural language understanding. In Findings of the association for computational linguistics: EMNLP 2020, p. 4163–4174. Cited by: 2nd item, §1.1, §1, §3. Joshi et al. (2017) M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1601–1611. Cited by: §F.1, §3. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §1. Kaushik et al. (2019) D. Kaushik, E. Hovy, and Z. C. Lipton Learning the difference that makes a difference with counterfactually-augmented data. arXiv preprint arXiv:1909.12434. Cited by: §1.1, §1. Kaushik et al. (2020) D. Kaushik, A. Setlur, E. Hovy, and Z. C. Lipton Explaining the efficacy of counterfactually augmented data. arXiv preprint arXiv:2010.02114. Cited by: §1.1. Khashabi et al. (2020) D. Khashabi, T. Khot, and A. Sabharwal More bang for your buck: natural perturbation for robust question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 163–170. Cited by: §1.1. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, p. 1317–1327. Cited by: 3rd item, §F.3, §1.1, §3. Köksal et al. (2023) A. Köksal, R. Aksitov, and C. Chang Hallucination augmented recitations for language models. arXiv preprint arXiv:2311.07424. Cited by: §1. Kwiatkowski et al. (2019) T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7, p. 453–466. Cited by: §F.1, §3. Levy et al. (2017) O. Levy, M. Seo, E. Choi, and L. Zettlemoyer Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st conference on computational natural language learning (CoNLL 2017), p. 333–342. Cited by: §F.1, §3. Li et al. (2024) M. Li, Y. Zhang, Z. Li, J. Chen, L. Chen, N. Cheng, J. Wang, T. Zhou, and J. Xiao From quantity to quality: boosting llm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 7602–7635. Cited by: §2.3. Liang et al. (2023) C. Liang, S. Zuo, Q. Zhang, P. He, W. Chen, and T. Zhao Less is more: task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, p. 20852–20867. Cited by: §1. Liu et al. (2024) J. Liu, C. Zhang, J. Guo, Y. Zhang, H. Que, K. Deng, Z. Bai, J. Liu, G. Zhang, J. Wang, et al. Ddk: distilling domain knowledge for efficient large language models. Advances in Neural Information Processing Systems 37, p. 98297–98319. Cited by: §1. Liu et al. (2021) Q. Liu, M. Kusner, and P. Blunsom Counterfactual data augmentation for neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 187–197. Cited by: §1.1. Longpre et al. (2021) S. Longpre, K. Perisetla, A. Chen, N. Ramesh, C. DuBois, and S. Singh Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7052–7063. Cited by: §1, §3.1. McAleese and Keane (2024) S. McAleese and M. Keane A comparative analysis of counterfactual explanation methods for text classifiers. arXiv preprint arXiv:2411.02643. Cited by: §1.1, §1. Mishra et al. (2021) S. Mishra, S. Dutta, J. Long, and D. Magazzeni A survey on the robustness of feature importance and counterfactual explanations. arXiv preprint arXiv:2111.00358. Cited by: §1.1. Neeman et al. (2023) E. Neeman, R. Aharoni, O. Honovich, L. Choshen, I. Szpektor, and O. Abend Disentqa: disentangling parametric and contextual knowledge with counterfactual question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 10056–10070. Cited by: §1.1. Noorani et al. (2025) E. Noorani, P. Dissanayake, F. Hamman, and S. Dutta Counterfactual explanations for model ensembles using entropic risk measures. arXiv preprint arXiv:2503.07934. Cited by: §1.1. OpenAI (2024) OpenAI GPT-4o mini: Advancing Cost-Efficient Intelligence. Note: https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/Accessed: 2026-09-20 Cited by: Appendix D, §2.1. Ou et al. (2022) J. Ou, J. Zhang, Y. Feng, and J. Zhou Counterfactual data augmentation via perspective transition for open-domain dialogues. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 1635–1648. Cited by: §1.1. Paranjape et al. (2022) B. Paranjape, M. Lamm, and I. Tenney Retrieval-guided counterfactual generation for qa. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1670–1686. Cited by: §1.1, §1. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 (140), p. 1–67. Cited by: §1, §1, §3. Rajagopal et al. (2022) D. Rajagopal, S. Shakeri, C. N. d. Santos, E. Hovy, and C. Chang Counterfactual data augmentation improves factuality of abstractive summarization. arXiv preprint arXiv:2205.12416. Cited by: §1.1. Rajpurkar et al. (2016) P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing, p. 2383–2392. Cited by: §F.1, §2.3, §3. Rockafellar and Wets (1998) R. T. Rockafellar and R. J. Wets Variational analysis. Springer. Cited by: §1, §2.2, §2.2. Ross et al. (2021) A. Ross, A. Marasović, and M. E. Peters Explaining nlp models via minimal contrastive editing (mice). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, p. 3840–3852. Cited by: §1.1, §1. Ross et al. (2022) A. Ross, T. Wu, H. Peng, M. E. Peters, and M. Gardner Tailor: generating and perturbing text with semantic controls. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3194–3213. Cited by: §1.1. Saadi and Wang (2025) K. Saadi and D. Wang TASKD-llm: task-aware selective knowledge distillation for llms. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, Cited by: §1. Sachdeva et al. (2024) R. Sachdeva, M. Tutek, and I. Gurevych CATfOOD: counterfactual augmented training for improving out-of-domain performance and calibration. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1876–1898. Cited by: §1.1. Saha et al. (2018) A. Saha, R. Aralikatte, M. M. Khapra, and K. Sankaranarayanan DuoRC: towards complex language understanding with paraphrased reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1683–1693. Cited by: §F.1, §3. Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, and T. Wolf DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. Cited by: §1.1, §1. Scheffer et al. (2001) T. Scheffer, C. Decomain, and S. Wrobel Active hidden markov models for information extraction. In International symposium on intelligent data analysis, p. 309–318. Cited by: Appendix C, §2, §3. Strubell et al. (2019) E. Strubell, A. Ganesh, and A. McCallum Energy and policy considerations for deep learning in nlp. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 3645–3650. Cited by: §1. Sun et al. (2019) S. Sun, Y. Cheng, Z. Gan, and J. Liu Patient knowledge distillation for bert model compression. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 4323–4332. Cited by: §1. Sutanto et al. (2025) P. Sutanto, J. Santoso, E. I. Setiawan, and A. P. Wibawa LLM distillation for efficient few-shot multiple choice question answering.. In EMNLP (Findings), p. 8502–8530. Cited by: §1.1. Team et al. (2024) G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: improving open language models at a practical size. arXiv preprint arXiv:2408.00118. Cited by: §1.1. Tokpo and Calders (2024) E. K. Tokpo and T. Calders Fairflow: an automated approach to model-based counterfactual data augmentation for nlp. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, p. 160–176. Cited by: §1.1. Trischler et al. (2017) A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman Newsqa: a machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, p. 191–200. Cited by: §F.1, §3. Wang et al. (2025) F. Wang, Z. Zhang, X. Zhang, Z. Wu, T. Mo, Q. Lu, W. Wang, R. Li, J. Xu, X. Tang, et al. A comprehensive survey of small language models in the era of large language models: techniques, enhancements, applications, collaboration with llms, and trustworthiness. ACM Transactions on Intelligent Systems and Technology 16 (6), p. 1–87. Cited by: §1.1. Wang et al. (2020) W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou Minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems 33, p. 5776–5788. Cited by: §1. Wang et al. (2023a) Y. Wang, S. Feng, H. Wang, W. Shi, V. Balachandran, T. He, and Y. Tsvetkov Resolving knowledge conflicts in large language models. arXiv preprint arXiv:2310.00935. Cited by: §1.1. Wang et al. (2023b) Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi Self-instruct: aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 13484–13508. Cited by: §1.1. Wu et al. (2024) J. Wu, L. Yang, Z. Wang, M. Okumura, and Y. Zhang Cofca: a step-wise counterfactual multi-hop qa benchmark. arXiv preprint arXiv:2402.11924. Cited by: §1.1. Wu et al. (2021) T. Wu, M. T. Ribeiro, J. Heer, and D. S. Weld Polyjuice: generating counterfactuals for explaining, evaluating, and improving models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 6707–6723. Cited by: §1.1, §1. Xu et al. (2024a) R. Xu, Z. Qi, Z. Guo, C. Wang, H. Wang, Y. Zhang, and W. Xu Knowledge conflicts for llms: a survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 8541–8565. Cited by: §1.1. Xu et al. (2024b) X. Xu, M. Li, C. Tao, T. Shen, R. Cheng, J. Li, C. Xu, D. Tao, and T. Zhou A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116. Cited by: §1.1, §1. Yang et al. (2025) C. Yang, Y. Zhu, W. Lu, Y. Wang, Q. Chen, C. Gao, B. Yan, and Y. Chen Survey on knowledge distillation for large language models: methods, evaluation, and application. ACM Transactions on Intelligent Systems and Technology 16 (6), p. 1–27. Cited by: §1.1. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, p. 2369–2380. Cited by: §F.1, §3. Youssef et al. (2024) P. Youssef, C. Seifert, J. Schlötterer, et al. LLMs for generating and evaluating counterfactuals: a comprehensive study. In Findings of the association for computational linguistics: EMNLP 2024, p. 14809–14824. Cited by: §1.1. Yu et al. (2023) W. Yu, M. Jiang, P. Clark, and A. Sabharwal Ifqa: a dataset for open-domain question answering under counterfactual presuppositions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 8276–8288. Cited by: §1.1, §1. Zhou et al. (2023) C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: less is more for alignment. Advances in Neural Information Processing Systems 36, p. 55006–55021. Cited by: §1. Appendix A Appendix Contents • Section B Limitations and Future Work • Section C KD Performance on Boundary-Near Factual Samples • Section D Counterfactual Generation Prompts • Section E Proof of Proposition 1 • Section F Experimental Setup Details – Section F.1 Datasets – Section F.2 Teacher Model Fine-Tuning and Evaluation – Section F.3 Additional Details of the Baselines • Section G Ablation Studies – Section G.1 Effect of CFD Validation – Section G.2 CFD Threshold Ablation – Section G.3 Cost Analysis • Section H Additional Experimental Results – Section H.1 In-Domain EM Results – Section H.2 In-Domain Agreement Results – Section H.3 Out-of-Domain EM Results – Section H.4 Performance under Modified Contextual Evidence Appendix B Limitations and Future Work Our current counterfactual generation framework is limited to QA samples with entity-based answers. Extending counterfactual generation beyond these could broaden the applicability of CofFeQA. Future work could explore more general counterfactual generation strategies that can accommodate diverse answer types while preserving the intended contextual changes. The current formulation of CofFeQA is also designed specifically for open-book question answering, where counterfactuals can be constructed by modifying the supporting context while preserving the question. Extending the definition, generation, and validation of counterfactuals to other generative tasks is an important direction for investigating whether counterfactual-guided few-shot distillation generalizes beyond open-book QA. Appendix C KD Performance on Boundary-Near Factual Samples To investigate how proximity to the teacher’s decision boundary affects knowledge distillation, we group factual samples according to their prediction margins under the teacher model. Intuitively, a small margin indicates that the teacher assigns similar probabilities to its two most likely token predictions, reflecting greater uncertainty in its output. We use margin sampling (Scheffer et al., 2001) as a proxy for proximity to the teacher’s decision boundary. Measuring proximity to the decision boundary. For a factual sample x=(Q,C,A)x=(Q,C,A) with target answer sequence y=(y1,…,yT)y=(y_1,…,y_T), the teacher produces a probability distribution over the vocabulary at each answer position t. Let v(1)tv_(1)^t and v(2)tv_(2)^t denote the teacher’s most likely and next most likely tokens, respectively, at position t, conditioned on the input x and the preceding target tokens y<ty_<t. We define the token-level prediction margin as mt(x,y)=logPθ(v(1)t∣x,y<t)−logPθ(v(2)t∣x,y<t).m_t(x,y)= P_θ\! (v_(1)^t x,y_<t )- P_θ\! (v_(2)^t x,y_<t ). (4) The prediction margin for the complete answer sequence is then computed as the mean of the token-level margins: margin(x,y)=1T∑t=1Tmt(x,y).margin(x,y)= 1T _t=1^Tm_t(x,y). (5) Here, v(1)tv_(1)^t and v(2)tv_(2)^t are the teacher’s own top-1 and top-2 predictions at each position and are not defined according to whether they match the ground-truth token. A smaller average margin indicates greater ambiguity between the teacher’s two most likely token predictions. We therefore use this quantity as a proxy for proximity to the teacher’s decision boundary. For each dataset, we sort the factual samples in ascending order of their teacher prediction margins and divide them into three margin bins, where Bin 0 contains samples with the smallest margins, Bin 1 contains samples with intermediate margins, and Bin 2 contains samples with the largest margins. To evaluate the effect of teacher prediction margin on knowledge distillation under a controlled few-shot budget, we sample 100 factual examples from each bin and perform knowledge distillation separately using the samples from each bin. Thus, each setting uses the same distillation budget of 100 factual samples, with the primary difference being their teacher prediction margins. Figure 6 reports the resulting KD performance when distillation is performed using 100 factual samples from each of the three margin bins. Figures 6(a) and 6(b) show EM and F1 averaged across the five in-domain datasets, respectively. For both metrics, distillation using the smallest-margin samples in Bin 0 results in substantially lower performance than using samples from Bins 1 and 2. The mean EM increases from 56.94%56.94\% in Bin 0 to 67%67\% in Bins 1–2, while the mean F1 increases from 65.99%65.99\% to 72–73%72–73\% in Bins 1–2. In contrast, the difference between Bins 1 and 2 is relatively small for both metrics. This suggests that the main degradation in distillation performance occurs when the few-shot factual supervision consists of samples with the smallest teacher prediction margins. (a) Aggregated EM (b) Aggregated F1 (c) Dataset-wise EM (d) Dataset-wise F1 Figure 6: KD performance as a function of the teacher prediction margin. Factual samples are grouped into three bins according to their average token-level prediction margins, with Bin 0 containing the smallest margin samples and Bin 2 containing the largest margin samples. Smaller margins are used as a proxy for greater proximity to the teacher’s decision boundary. Knowledge distillation is performed separately using 100 factual samples from each bin, maintaining the same distillation budget across all three settings. Panels (a) and (b) report EM and F1 averaged across the five in-domain datasets, respectively, while panels (c) and (d) report the corresponding dataset-wise results. Distillation using samples from the smallest margin bin generally results in lower performance than using samples from the larger margin bins. Figures 6(c) and 6(d) show the corresponding dataset-wise EM and F1 results for HotpotQA, NewsQA, Natural Questions (NQ), SQuAD, and TriviaQA. The same general pattern is observed across the five datasets: distillation using samples from Bin 0 results in lower EM and F1 than using samples from the larger-margin bins. The difference is particularly pronounced for HotpotQA, NewsQA, TriviaQA and NQ, whereas SQuAD exhibits a smaller difference across the three bins. Thus, the lower distillation performance associated with the smallest-margin samples is observed across datasets rather than being driven by a single benchmark. Overall, these results support our motivating observation that factual samples with small teacher prediction margins are more challenging to distill under limited factual supervision. Since a small margin reflects greater ambiguity between the teacher’s most likely token predictions, we use these samples as an operational approximation of boundary-near factual examples. Importantly, this degradation is observed while maintaining the same 100 sample distillation budget across all three margin bins. This observation motivates CofFeQA to complement boundary-near factual samples with counterfactual supervision, providing additional information about how the teacher’s response changes under modified contextual evidence. Appendix D Counterfactual Generation Prompts We use GPT-4o-mini (OpenAI, 2024) for counterfactual generation. The same prompts are used for both the T5 and Qwen2.5 experiments. Counterfactual generation consists of three stages: (i) identifying the entity type of the original answer, (i) generating a plausible alternative entity of the same type, and (i) minimally modifying the original context to support the alternative answer. The prompts used in these stages are shown in Figure 7. Entity Type Classification What type of named entity is this answer: “answer_text”? Classify as PERSON, LOCATION, ORGANIZATION, DATE, NUMERIC, or OTHER if none apply. Alternative Entity Generation The original answer is “original_answer”, which is a entity_type. Propose ONE plausible alternative entity_type that is CLOSE to “original_answer” (similar in scale, category, or context) but clearly DIFFERENT from it. Do NOT propose any of these already-tried alternatives: avoid_list. Respond with ONLY a JSON object: "alternative_entity": "<your proposed alternative>" Counterfactual Context Generation: System Prompt You are an expert at creating counterfactual reading-comprehension passages for QA training data. Given an original context that supports a specific answer, you rewrite it MINIMALLY so that it instead unambiguously supports a different target answer, by replacing occurrences of the original answer (and grammatically-dependent references to it) with the target answer, while leaving everything else in the passage unchanged. Do not add explanations or extra content. Counterfactual Context Generation: User Prompt Question: question Original context (supports answer “original_answer”): original_context Target answer (the new counterfactual answer): “target_answer” Rewrite the context above so that it minimally changes to unambiguously support “target_answer” as the answer to the question, instead of “original_answer”. Replace occurrences of “original_answer” (and any grammatically-dependent references to it) with “target_answer”, keeping everything else in the passage exactly the same. Do not add commentary. Respond with ONLY a JSON object: "reasoning": "<1-2 sentences on what you changed>", "counterfactual_context": "<the full rewritten context>" Figure 7: Prompts used for counterfactual generation. The generation process consists of entity type classification, alternative entity generation, and counterfactual context generation. The final stage uses a system prompt and a user prompt to minimally modify the original context to support the alternative answer. Generated counterfactuals are subsequently evaluated using CFD before being used for distillation. Appendix E Proof of Proposition 1 In this section, we provide the proof of the upper bound on the Hausdorff distance between the teacher’s context-response boundary ℳtM_t and the student’s context-response boundary ℳsM_s. By definition, the Hausdorff distance between ℳtM_t and ℳsM_s is dH(ℳt,ℳs)=maxsupt∈ℳtinfs∈ℳs‖t−s‖,sups∈ℳsinft∈ℳt‖s−t‖.d_H(M_t,M_s)= \ _t _t _s _s\|t-s\|,\; _s _s _t _t\|s-t\| \. (6) We bound the two directed distances separately (Hamman et al., 2025). Part 1: Bounding the deviation from ℳtM_t to ℳsM_s Let t∈ℳt _t be an arbitrary point on the teacher’s boundary. By the boundary-coverage assumption (Assumption 3), there exists a teacher boundary crossing point ti∗∈ℳt_i^* _t associated with a factual-counterfactual pair (ci,ci′)(c_i,c_i ) such that (see Figure 8): ‖t−ti∗‖≤ϵ.\|t-t_i^*\|≤ε. (7) Figure 8: Visualization of the distances between ℳtM_t and ℳsM_s. cic_i and ci′c _i are factual and counterfactual pairs with ‖ci−ci′‖=α′≤α\|c_i-c_i \|=α ≤α). t is an arbitrary point in ℳtM_t that finds some ti∗t_i^* cut point in ℳtM_t within ϵ′ε distance (ϵ′≤ϵε ≤ε). Under Assumption 1, the factual-counterfactual perturbation is bounded as ‖ci−ci′‖≤α.\|c_i-c_i \|≤α. (8) Under the teacher-student agreement assumption (Assumption 2), the teacher and student agree on the responses associated with both endpoints cic_i and ci′c_i . Since the factual and counterfactual contexts correspond to different responses, both the teacher and student boundaries cross the segment [ci,ci′][c_i,c_i ]. Let ti∗∈ℳt_i^* _t and si∗∈ℳs_i^* _s denote their respective boundary crossing points on this segment. Because both ti∗t_i^* and si∗s_i^* lie on the same segment [ci,ci′][c_i,c_i ], ‖ti∗−si∗‖≤‖ci−ci′‖≤α.\|t_i^*-s_i^*\|≤\|c_i-c_i \|≤α. (9) Combining Equations equation 7 and equation 9 and applying the triangle inequality, we obtain ‖t−si∗‖≤‖t−ti∗‖+‖ti∗−si∗‖≤ϵ+α.\|t-s_i^*\|≤\|t-t_i^*\|+\|t_i^*-s_i^*\|≤ε+α. (10) Since there exists at least one point si∗∈ℳs_i^* _s within distance ϵ+αε+α of t, the infimum distance satisfies: infs∈ℳs‖t−s‖≤ϵ+α. _s _s\|t-s\|≤ε+α. (11) Taking the supremum over all points t∈ℳt _t yields: supt∈ℳtinfs∈ℳs‖t−s‖≤ϵ+α. _t _t _s _s\|t-s\|≤ε+α. (12) Part 2: Bounding the deviation from ℳsM_s to ℳtM_t We follow a symmetric logic for the student boundary. For any point s∈ℳs _s, Assumption 3 guarantees the existence of a student boundary cutpoint si∗s_i^* such that ‖s−si∗‖≤ϵ\|s-s_i^*\|≤ε. From the relationship established in Part 1, the distance between this student cutpoint and the corresponding teacher cutpoint ti∗t_i^* is ‖si∗−ti∗‖≤α\|s_i^*-t_i^*\|≤α. Applying the triangle inequality: ‖s−ti∗‖≤‖s−si∗‖+‖si∗−ti∗‖≤ϵ+α\|s-t_i^*\|≤\|s-s_i^*\|+\|s_i^*-t_i^*\|≤ε+α (13) Since ti∗∈ℳt_i^* _t, it follows that inft∈ℳt‖s−t‖≤ϵ+α _t _t\|s-t\|≤ε+α. Taking the supremum over all s∈ℳs _s: sups∈ℳsinft∈ℳt‖s−t‖≤ϵ+α _s _s _t _t\|s-t\|≤ε+α (14) Final Result Combining Equations equation 12 and equation 14, we obtain dH(ℳt,ℳs)≤maxϵ+α,ϵ+α=α+ϵ.d_H(M_t,M_s)≤ \ε+α,ε+α\=α+ε. (15) This confirms that the student boundary remains within an (α+ϵ)(α+ε)-tube of the teacher boundary. Appendix F Experimental Setup Details F.1 Datasets We evaluate CofFeQA on eight open-domain question answering benchmarks, comprising five in-domain and three out-of-domain datasets. The in-domain benchmarks include Natural Questions (NQ), SQuAD, NewsQA, TriviaQA, and HotpotQA. NQ consists of real user queries collected from Google search logs (Kwiatkowski et al., 2019). SQuAD (Stanford Question Answering Dataset) contains questions authored by crowdworkers based on Wikipedia articles (Rajpurkar et al., 2016). NewsQA is constructed from CNN news articles with questions generated crowdworkers (Trischler et al., 2017). TriviaQA consists QA pairs sourced from trivia and quiz-league websites (Joshi et al., 2017). HotpotQA focuses on multi-hop reasoning, requiring models to combine information from multiple Wikipedia articles (Yang et al., 2018). The out-of-domain benchmarks include DROP, DuoRC, and RelationExtraction. DROP contains quantitative QA pairs which has numerical and text answers (Dua et al., 2019). DuoRC contains QA pairs derived from movie plots, constructed by crowdworkers using IMDb and Wikipedia (Saha et al., 2018). RelationExtraction reformulates relation extraction as a reading comprehension task using Wikipedia text (Levy et al., 2017). The datasets are designed for generalization in open-book question answering, where samples are drawn from diverse distributions. The datasets have following properties in common. • Question distribution: The test sets contain different styles of questions, such as entity-based and relational questions, collected from diverse sources, including crowdworkers, domain experts, and exam writers. • Context distribution: Passages are drawn from heterogeneous sources such as science articles, novels, medical abstracts, news articles. • Answerability: The answer can be generated using the provided context, and the answer span appears within the first 800 tokens of the passage. The training and validation sets for the teacher are drawn from the original training splits after removing duplicate entries. Dataset statistics are reported in Table 3. We generate counterfactuals only for the samples with entity-based answers. Statistics of the knowledge distillation datasets are also shown in Table 3, where student train and val denote available factual training samples with entity-based answers for distillation. The in-domain test sets for knowledge distillation are taken from the original development splits after filtering for entity-based answer samples. The out-of-domain test set counts are as follows: DROP (496), and DuoRC (450), RelationExtraction (912). Table 3: Statistics of teacher and student datasets used in our experiments. Teacher datasets contain the original training/validation splits, while student datasets contain available factual instances for distillation and evaluation. Teacher Dataset Student Dataset Dataset Train Val. Train Val. Test NQ 93,663 10,408 34,383 3,821 2,283 SQuAD 77,847 8,651 16,341 1,816 2,790 NewsQA 66,655 7,407 10,956 1,218 690 TriviaQA 55,519 6,170 4,415 491 607 HotpotQA 65,630 7,294 28,956 3,218 2,052 F.2 Teacher Model Fine-Tuning and Evaluation All fine-tuning experiments were conducted on a machine equipped with four NVIDIA RTX 6000 Ada GPUs. For T5-Large fine-tuning experiments, we used a batch size of 4, a learning rate of 1×10−41× 10^-4, and a maximum of 5 training epochs. Training was stopped early based on validation loss using an early stopping criterion with a patience of 3 validation checks. For Qwen2.5-7B-IT fine-tuning experiments, we used a batch size of 1, a learning rate of 2×10−52× 10^-5, and a maximum of 5 training epochs. Training was stopped early based on validation loss using an early stopping criterion with a patience of 3 validation checks. On average, all models converged within 1 to 2 epochs. We evaluated the fine-tuned teacher models using the same test sets for each dataset. The results can be found in Table 4 Table 4: Teacher model performance (EM and F1) on factual and counterfactual test sets across five in-domain QA benchmarks for T5 and Qwen2.5 models. Test Set Type Model Score NQ SQuAD NewsQA TriviaQA HotpotQA Factual T5 EM 74.90 79.64 73.19 67.38 77.19 F1 77.76 87.78 78.97 69.27 85.49 Qwen2.5 EM 79.46 83.91 75.65 73.48 79.34 F1 82.83 91.04 82.50 76.37 88.52 Counterfactual T5 EM 67.94 79.75 66.38 58.32 65.84 F1 70.79 87.04 71.35 60.58 71.37 Qwen2.5 EM 69.78 79.35 66.23 31.47 64.13 F1 73.61 86.46 70.55 32.22 70.74 F.3 Additional details of the baselines We compare CofFeQA against three knowledge distillation baselines. To ensure a fair comparison, all methods use the same k-shot data splits, and evaluation settings. • Standard Knowledge Distillation (KD). Standard knowledge distillation (KD) (Hinton et al., 2015) transfers knowledge from a larger teacher model to a smaller student model by aligning their output distributions. Let x denote the input, y=(y1,…,yT)y=(y_1,…,y_T) the ground-truth output sequence, and psp_s and ptp_t the student and teacher model distributions, respectively. The student is first trained using the standard cross-entropy objective ℒCE=−∑t=1Tlogps(yt∣x,y<t).L_CE=- _t=1^T p_s(y_t x,y_<t). (16) In addition, KD encourages the student to match the teacher’s token-level output distribution at each decoding step using ℒKD=∑t=1TKL(pt(⋅∣x,y<t)∥ps(⋅∣x,y<t)).L_KD= _t=1^TKL (p_t(· x,y_<t)\,\|\,p_s(· x,y_<t) ). (17) The final training objective combines the supervised and distillation losses: ℒ=αℒCE+(1−α)ℒKD,L= _CE+(1-α)L_KD, (18) where α∈[0,1]α∈[0,1] controls the trade-off between fitting the ground-truth sequence and matching the teacher’s output distribution. Thus, standard KD transfers knowledge by aligning the teacher and student token-level distributions while retaining supervision from the ground-truth sequence. • Layer-wise Distillation (LWD). Layer-wise distillation (LWD) (Jiao et al., 2020) extends output-level knowledge distillation by additionally transferring information from the intermediate representations of the teacher. Representations from selected teacher layers are aligned with the corresponding representations of the student through an auxiliary layer-wise distillation objective, providing supervision at both the intermediate representation and output levels. Let t(lt)h_t^(l_t) and s(ls)h_s^(l_s) denote the hidden representations from a paired teacher layer ltl_t and student layer lsl_s, respectively. Since the teacher and student may have different hidden dimensions, a learnable projection WlsW_l_s is applied to the student representation before alignment. The layer-wise distillation loss is defined as ℒLWD=1||∑(ls,lt)∈‖Wlss(ls)−t(lt)‖22,L_LWD= 1|P| _(l_s,l_t) \|W_l_sh_s^(l_s)-h_t^(l_t) \|_2^2, (19) where P denotes the set of paired student and teacher layers. The final objective combines the standard KD objective with the layer-wise representation loss: ℒ=αℒCE+(1−α)ℒKD+βℒLWD,L= _CE+(1-α)L_KD+ _LWD, (20) where β controls the contribution of intermediate representation alignment. Thus, unlike standard KD, which transfers knowledge through the output distributions, LWD additionally encourages the student to reproduce the teacher’s intermediate representations. • Sequence-Level Knowledge Distillation (SeqKD). Unlike standard KD, which aligns the teacher and student token-level distributions at each decoding step, sequence-level knowledge distillation (seqKD) transfers knowledge through complete sequences generated by the teacher Kim and Rush (2016). Let x denote the input and y=(y1,…,yT)y=(y_1,…,y_T) denote the output sequence, and let psp_s and ptp_t denote the student and teacher model distributions, respectively. The sequence-level distillation objective is defined as ℒSeqKD=−∑y∈pt(y∣x)logps(y∣x),L_SeqKD=- _y p_t(y x) p_s(y x), (21) where Y denotes the set of all possible output sequences. Since directly optimizing over all possible output sequences is intractable, seqKD approximates the teacher’s sequence-level distribution using its highest-probability output sequence. Specifically, the teacher-generated sequence is obtained as y^≈argmaxy∈pt(y∣x), y≈ _y p_t(y x), (22) where beam search is used to approximate the highest probability sequence under the teacher distribution. The resulting sequence-level distillation objective is then ℒSeqKD≈−logps(y^∣x)=−∑t=1|y^|logps(y^t∣x,y^<t).L_SeqKD≈- p_s( y x)=- _t=1^| y| p_s ( y_t x, y_<t ). (23) Thus, while standard KD transfers knowledge by matching the teacher’s token-level probability distributions, seqKD uses the complete sequence generated by the teacher as the training target for the student. For standard KD, the student was trained using a weighted combination of the MLE loss and the token-level distillation loss. The temperature was set to 1, and the α was set to 0.50.5. For LWD, student decoder layers were uniformly mapped to corresponding teacher decoder layers across the model depth. Layer-wise knowledge transfer was performed by aligning the decoder self-attention distributions of the mapped teacher and student layers using KL divergence. Attention distributions were averaged across attention heads before computing the layer-wise loss. The layer-wise distillation loss was added to the standard KD objective with a weight of 5.05.0. For seqKD, the teacher generated a complete output sequence for each training input using beam search with a beam size of 4. The original ground-truth answer was replaced by the corresponding teacher-generated sequence, which was then used as the training target for the student. Following the original seqKD formulation (Kim and Rush, 2016), the student was trained using cross-entropy loss on the teacher-generated sequences without an additional token-level distillation loss. For all three baselines, we used a batch size of 4 and a learning rate of 1×10−41× 10^-4. The maximum number of training epochs was set to 100, with early stopping based on validation loss using a patience of 3 validation checks. Appendix G Ablation Study Results G.1 Effect of CFD Validation To isolate the contribution of CFD verification, we compare CofFeQA with a variant that generates counterfactuals without CFD verification. Both variants use identical factual data splits and training settings, with the only difference being whether the generated counterfactuals are verified using CFD. For CofFeQA with CFD verification, we use the selected threshold τ=0.3τ=0.3. Experiments are conducted using T5 across all datasets with five random seeds for each k-shot setting, and the results are averaged across datasets. Figure 9 reports the improvement of each variant over standard KD. CFD verification generally provides larger gains in both downstream QA performance and teacher-student agreement, with particularly clear improvements at k=32k=32, 6464, and 128128. These results indicate that the improvements of CofFeQA are not solely due to the inclusion of counterfactual examples; verifying the generated counterfactuals using CFD provides additional benefits for knowledge transfer. (a) Accuracy EM (b) Accuracy F1 (c) Agreement EM (d) Agreement F1 Figure 9: Effect of CFD verification on CofFeQA. Performance gains over standard KD (Δ=CofFeQA−KD = CofFeQA-KD) are shown across different few-shot budgets k. We compare CofFeQA without CFD verification against the full CofFeQA framework with CFD verification (τ=0.3τ=0.3). Panels (a) and (b) report accuracy EM and F1 gains, while panels (c) and (d) report teacher-student agreement EM and F1 gains. CFD verification generally yields larger improvements, demonstrating the benefit of validating generated counterfactuals before distillation. G.2 CFD Threshold Ablation We analyze the sensitivity of CofFeQA to the CFD threshold τ using T5 on TriviaQA. We evaluate τ∈0.1,0.2,0.3,0.4,0.5τ∈\0.1,0.2,0.3,0.4,0.5\ across k, with five random seeds for each k-shot setting. For each threshold, we report the averaged improvement of CofFeQA over standard KD (Δ=CofFeQA−KD = CofFeQA-KD). As shown in Figure 10, performance is sensitive to the CFD threshold. The threshold τ=0.1τ=0.1 yields negligible improvements, whereas moderate thresholds produce substantially larger gains in both accuracy EM/F1 and teacher-student agreement. The gains peak around τ=0.2τ=0.2–0.30.3 and gradually decrease as the threshold becomes more permissive. Overall, these results demonstrate the importance of controlling counterfactual quality through the CFD threshold. Figure 10: CFD threshold ablation. Effect of the CFD threshold τ on CofFeQA using T5 on TriviaQA. The bars show the improvement of CofFeQA over standard KD (Δ=CofFeQA−KD = CofFeQA-KD), averaged across k∈16,32,64,128,256k∈\16,32,64,128,256\ and five random seeds. The left plot shows the improvement in accuracy measured by EM and F1, while the right plot shows the improvement in teacher-student agreement measured by EM and F1. G.3 Cost Analysis We analyze the computational cost of CofFeQA during both counterfactual generation and distillation. Table 5 reports the per-sample cost of counterfactual generation, averaged over all generation attempts. Counterfactual generation requires approximately 2.882.88 seconds per candidate, with relatively small CPU and GPU energy consumption for both model settings. Table 6 and Table 7 reports the training duration and energy consumption during distillation under T5 and Qwen2.5, respectively, measured using CodeCarbon. In both models, for standard KD and LWD, CofFeQA achieves higher accuracy while requiring less training time and energy across all k-shot settings. For SeqKD, CofFeQA also improves accuracy across all settings, but incurs higher computational costs at the smaller budgets (k=16k=16 and 3232). From k=64k=64 onward, its computational cost becomes comparable to the baseline. Overall, these results show that CofFeQA introduces an additional counterfactual-generation cost, while maintaining favorable computational efficiency during the subsequent distillation stage. Table 5: Average duration and CPU/GPU energy consumption per counterfactual generation candidate under the T5 and Qwen2.5 settings. Results are averaged across all generation attempts. Model Duration/candidate (s) CPU/candidate (kWh) GPU/candidate (kWh) T5 2.884 0.000083 0.000055 Qwen2.5 2.877 0.000071 0.000067 Table 6: Comparison of EM score, distillation duration, and CPU/GPU energy consumption between baseline methods and COFFEQA across different few-shot budgets under the T5 setting. Objective Method k EM (%) Duration (s) CPU (kWh) GPU (kWh) KD Baseline 16 61.88 62.86 0.00201 0.01342 32 60.82 50.34 0.00153 0.00980 64 54.83 56.15 0.00170 0.01259 128 56.07 75.90 0.00224 0.01662 256 55.56 128.67 0.00378 0.02777 CofFeQA 16 63.36 45.19 0.00143 0.01016 32 64.20 42.58 0.00133 0.00879 64 62.93 49.46 0.00149 0.01027 128 59.67 68.09 0.00203 0.01428 256 59.49 114.55 0.00336 0.02487 LWD Baseline 16 61.75 150.44 0.00474 0.02235 32 60.94 127.87 0.00405 0.01862 64 54.52 137.14 0.00425 0.02128 128 55.49 161.15 0.00488 0.02327 256 55.88 221.42 0.00694 0.03796 CofFeQA 16 62.94 121.86 0.00383 0.01784 32 64.38 109.26 0.00344 0.01799 64 62.72 127.24 0.00389 0.01743 128 59.94 131.83 0.00400 0.01907 256 59.78 180.71 0.00569 0.03242 SeqKD Baseline 16 61.77 59.14 0.00186 0.01021 32 61.19 39.74 0.00127 0.00801 64 61.37 36.16 0.00115 0.00774 128 61.15 44.13 0.00136 0.00988 256 62.06 71.46 0.00215 0.01550 CofFeQA 16 64.81 91.77 0.00287 0.01409 32 64.70 53.12 0.00169 0.01120 64 64.50 37.99 0.00121 0.00832 128 64.54 47.76 0.00149 0.00999 256 64.06 82.79 0.00248 0.01780 Table 7: Comparison of EM score, distillation duration, and CPU/GPU energy consumption between baseline methods and COFFEQA across different few-shot budgets under the Qwen2.5. Objective Method k EM (%) Duration (s) CPU (kWh) GPU (kWh) KD Baseline 16 61.22 748.41 0.02378 0.11453 32 60.24 636.82 0.01930 0.11645 64 59.13 774.16 0.02278 0.15832 128 58.51 1202.28 0.03660 0.27748 256 59.07 1793.38 0.05119 0.35929 CofFeQA 16 61.73 704.12 0.02244 0.11386 32 61.59 698.58 0.02124 0.13367 64 61.78 693.77 0.02043 0.14167 128 60.93 1066.53 0.03231 0.24643 256 60.59 1728.09 0.04949 0.33793 LWD Baseline 16 60.77 892.88 0.02964 0.14427 32 60.94 614.12 0.01998 0.11107 64 59.71 560.93 0.01830 0.10488 128 59.52 641.79 0.02027 0.11110 256 59.70 840.55 0.02688 0.16956 CofFeQA 16 61.10 852.61 0.02920 0.13976 32 61.51 727.77 0.02350 0.12773 64 61.49 477.78 0.01546 0.08719 128 61.80 526.55 0.01641 0.09615 256 62.03 759.89 0.02396 0.14949 SeqKD Baseline 16 59.72 584.15 0.01826 0.11342 32 60.46 559.89 0.01701 0.10684 64 60.96 642.25 0.01894 0.14049 128 60.79 970.14 0.02825 0.21411 256 62.23 1683.50 0.04842 0.38788 CofFeQA 16 59.98 648.94 0.01986 0.12576 32 60.72 557.76 0.01663 0.11214 64 62.16 641.84 0.01882 0.13954 128 62.21 925.38 0.02703 0.18591 256 62.88 1527.88 0.04422 0.34185 Appendix H Additional Experimental Results H.1 In-Domain EM Results Table 8 reports the in-domain EM results across all the baselines. Table 8: In-Domain EM score comparison across k-shots for T5 and Qwen2.5 models using baselines. Mean EM scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. baseline). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg NQ KD 58.100.44_ 0.60.44 57.733.52_ 0.63.52 54.733.79_ 0.63.79 47.846.31_ 0.66.31 50.323.61_ 0.63.61 53.754.54_ 0.64.54 61.950.36_ 0.60.36 61.240.61_ 0.60.61 60.890.44_ 0.60.44 60.180.53_ 0.60.53 62.070.47_ 0.60.47 61.270.78_ 0.60.78 + CofFeQA 58.890.33_ 0.60.33 58.491.50_ 0.61.50 56.922.34_ 0.62.34 49.866.34_ 0.66.34 51.205.30_ 0.65.30 55.074.24_ 0.64.24 61.790.27_ 0.60.27 63.120.35_ 0.60.35 63.060.47_ 0.60.47 62.660.82_ 0.60.82 62.841.32_ 0.61.32 62.690.54_ 0.60.54 LWD 58.200.48_ 0.60.48 57.473.89_ 0.63.89 54.803.04_ 0.63.04 50.153.73_ 0.63.73 50.905.43_ 0.65.43 54.303.68_ 0.63.68 60.196.36_ 0.66.36 61.910.79_ 0.60.79 61.101.33_ 0.61.33 60.710.44_ 0.60.44 61.821.03_ 0.61.03 61.150.73_ 0.60.73 + CofFeQA 58.840.27_ 0.60.27 57.453.65_ 0.63.65 55.004.16_ 0.64.16 52.584.70_ 0.64.70 51.166.81_ 0.66.81 55.013.22_ 0.63.22 58.956.05_ 0.66.05 62.530.63_ 0.60.63 63.490.56_ 0.60.56 63.480.69_ 0.60.69 63.391.46_ 0.61.46 62.371.95_ 0.61.95 SeqKD 54.751.75_ 0.61.75 54.471.07_ 0.61.07 56.441.11_ 0.61.11 55.851.57_ 0.61.57 58.591.65_ 0.61.65 56.021.64_ 0.61.64 59.850.41_ 0.60.41 61.600.61_ 0.60.61 61.960.40_ 0.60.40 62.370.58_ 0.60.58 62.941.21_ 0.61.21 61.741.17_ 0.61.17 + CofFeQA 56.970.92_ 0.60.92 57.661.14_ 0.61.14 57.781.20_ 0.61.20 58.740.76_ 0.60.76 58.291.49_ 0.61.49 57.890.67_ 0.60.67 61.720.49_ 0.60.49 61.230.77_ 0.60.77 63.980.80_ 0.60.80 63.720.76_ 0.60.76 64.060.83_ 0.60.83 62.941.36_ 0.61.36 SQuAD KD 78.360.77_ 0.60.77 75.462.06_ 0.62.06 73.683.48_ 0.63.48 74.971.75_ 0.61.75 74.371.95_ 0.61.95 75.371.80_ 0.61.80 70.270.18_ 0.60.18 69.400.28_ 0.60.28 69.000.40_ 0.60.40 66.880.58_ 0.60.58 67.910.25_ 0.60.25 68.691.32_ 0.61.32 + CofFeQA 78.470.26_ 0.60.26 77.861.15_ 0.61.15 77.231.18_ 0.61.18 77.110.46_ 0.60.46 75.841.68_ 0.61.68 77.300.99_ 0.60.99 69.390.57_ 0.60.57 70.130.69_ 0.60.69 69.930.55_ 0.60.55 68.920.92_ 0.60.92 67.210.67_ 0.60.67 69.121.17_ 0.61.17 LWD 78.360.68_ 0.60.68 75.431.58_ 0.61.58 74.063.53_ 0.63.53 73.842.17_ 0.62.17 74.441.54_ 0.61.54 75.231.85_ 0.61.85 69.400.82_ 0.60.82 69.510.30_ 0.60.30 68.950.31_ 0.60.31 67.550.90_ 0.60.90 68.921.15_ 0.61.15 68.870.78_ 0.60.78 + CofFeQA 78.370.33_ 0.60.33 77.831.14_ 0.61.14 76.541.06_ 0.61.06 77.230.60_ 0.60.60 76.131.79_ 0.61.79 77.220.92_ 0.60.92 68.950.17_ 0.60.17 69.850.24_ 0.60.24 70.070.75_ 0.60.75 69.380.31_ 0.60.31 69.741.37_ 0.61.37 69.600.44_ 0.60.44 SeqKD 76.450.95_ 0.60.95 77.940.93_ 0.60.93 76.870.67_ 0.60.67 76.950.18_ 0.60.18 76.380.64_ 0.60.64 76.920.62_ 0.60.62 70.770.46_ 0.60.46 70.960.28_ 0.60.28 70.780.47_ 0.60.47 70.320.39_ 0.60.39 71.380.74_ 0.60.74 70.840.38_ 0.60.38 + CofFeQA 78.150.48_ 0.60.48 77.300.46_ 0.60.46 77.690.71_ 0.60.71 77.270.68_ 0.60.68 76.560.15_ 0.60.15 77.390.59_ 0.60.59 70.040.44_ 0.60.44 71.000.43_ 0.60.43 71.120.88_ 0.60.88 70.780.62_ 0.60.62 71.461.00_ 0.61.00 70.880.53_ 0.60.53 NewsQA KD 64.992.59_ 0.62.59 64.202.93_ 0.62.93 59.014.29_ 0.64.29 55.832.73_ 0.62.73 50.141.27_ 0.61.27 58.836.15_ 0.66.15 48.201.05_ 0.61.05 43.741.28_ 0.61.28 42.991.22_ 0.61.22 45.191.65_ 0.61.65 42.931.48_ 0.61.48 44.612.21_ 0.62.21 + CofFeQA 64.783.94_ 0.63.94 67.591.30_ 0.61.30 67.392.56_ 0.62.56 64.352.37_ 0.62.37 59.803.90_ 0.63.90 64.783.15_ 0.63.15 52.290.50_ 0.60.50 49.331.11_ 0.61.11 47.281.13_ 0.61.13 46.671.56_ 0.61.56 48.932.76_ 0.62.76 48.902.20_ 0.62.20 LWD 64.752.54_ 0.62.54 63.803.28_ 0.63.28 58.874.49_ 0.64.49 54.144.04_ 0.64.04 50.140.74_ 0.60.74 58.346.24_ 0.66.24 49.160.55_ 0.60.55 48.291.11_ 0.61.11 46.811.47_ 0.61.47 47.331.21_ 0.61.21 44.932.77_ 0.62.77 47.301.60_ 0.61.60 + CofFeQA 64.434.04_ 0.64.04 68.060.73_ 0.60.73 66.931.52_ 0.61.52 63.622.63_ 0.62.63 60.414.62_ 0.64.62 64.693.00_ 0.63.00 52.260.82_ 0.60.82 49.191.21_ 0.61.21 48.121.39_ 0.61.39 50.230.52_ 0.60.52 51.684.43_ 0.64.43 50.301.71_ 0.61.71 SeqKD 54.582.52_ 0.62.52 55.682.89_ 0.62.89 58.233.72_ 0.63.72 60.673.24_ 0.63.24 62.231.37_ 0.61.37 58.283.23_ 0.63.23 41.541.12_ 0.61.12 43.740.88_ 0.60.88 45.770.55_ 0.60.55 48.411.32_ 0.61.32 52.001.08_ 0.61.08 46.294.08_ 0.64.08 + CofFeQA 63.802.04_ 0.62.04 66.352.23_ 0.62.23 66.201.23_ 0.61.23 70.091.82_ 0.61.82 68.172.03_ 0.62.03 66.922.36_ 0.62.36 42.581.04_ 0.61.04 45.711.17_ 0.61.17 48.170.85_ 0.60.85 50.001.99_ 0.61.99 53.622.15_ 0.62.15 48.024.19_ 0.64.19 TriviaQA KD 50.543.88_ 0.63.88 47.713.13_ 0.63.13 44.382.08_ 0.62.08 44.122.82_ 0.62.82 45.301.17_ 0.61.17 46.412.71_ 0.62.71 58.391.08_ 0.61.08 58.680.90_ 0.60.90 56.901.23_ 0.61.23 54.431.23_ 0.61.23 56.050.79_ 0.60.79 56.891.75_ 0.61.75 + CofFeQA 56.471.45_ 0.61.45 53.052.10_ 0.62.10 53.156.24_ 0.66.24 48.201.94_ 0.61.94 47.451.73_ 0.61.73 51.663.78_ 0.63.78 59.040.48_ 0.60.48 57.460.65_ 0.60.65 59.240.69_ 0.60.69 57.660.92_ 0.60.92 55.191.49_ 0.61.49 57.721.62_ 0.61.62 LWD 50.513.54_ 0.63.54 49.002.51_ 0.62.51 42.702.28_ 0.62.28 43.261.75_ 0.61.75 45.501.07_ 0.61.07 46.193.46_ 0.63.46 57.790.32_ 0.60.32 57.960.83_ 0.60.83 56.240.30_ 0.60.30 54.990.59_ 0.60.59 55.751.27_ 0.61.27 56.551.29_ 0.61.29 + CofFeQA 56.181.51_ 0.61.51 52.982.33_ 0.62.33 53.116.50_ 0.66.50 48.011.59_ 0.61.59 47.640.98_ 0.60.98 51.583.66_ 0.63.66 58.950.59_ 0.60.59 58.420.99_ 0.60.99 58.480.35_ 0.60.35 57.271.17_ 0.61.17 54.831.27_ 0.61.27 57.591.66_ 0.61.66 SeqKD 55.353.42_ 0.63.42 50.051.85_ 0.61.85 47.512.53_ 0.62.53 48.141.96_ 0.61.96 46.561.92_ 0.61.92 49.523.50_ 0.63.50 58.290.90_ 0.60.90 56.970.38_ 0.60.38 57.660.53_ 0.60.53 56.841.21_ 0.61.21 57.531.28_ 0.61.28 57.460.58_ 0.60.58 + CofFeQA 56.801.98_ 0.61.98 51.861.82_ 0.61.82 51.831.36_ 0.61.36 48.532.97_ 0.62.97 51.141.35_ 0.61.35 52.032.99_ 0.62.99 59.700.85_ 0.60.85 59.111.01_ 0.61.01 58.850.75_ 0.60.75 57.230.69_ 0.60.69 56.970.76_ 0.60.76 58.371.21_ 0.61.21 HotpotQA KD 57.425.77_ 0.65.77 58.987.55_ 0.67.55 42.356.57_ 0.66.57 57.572.64_ 0.62.64 57.645.67_ 0.65.67 54.796.98_ 0.66.98 67.310.57_ 0.60.57 68.160.42_ 0.60.42 65.861.18_ 0.61.18 65.880.37_ 0.60.37 66.401.26_ 0.61.26 66.721.00_ 0.61.00 + CofFeQA 58.168.44_ 0.68.44 64.034.40_ 0.64.40 59.962.24_ 0.62.24 58.814.43_ 0.64.43 63.193.02_ 0.63.02 60.832.63_ 0.62.63 66.121.10_ 0.61.10 67.910.48_ 0.60.48 69.400.74_ 0.60.74 68.750.75_ 0.60.75 68.791.00_ 0.61.00 68.191.27_ 0.61.27 LWD 56.935.65_ 0.65.65 59.038.17_ 0.68.17 42.166.37_ 0.66.37 56.035.04_ 0.65.04 58.395.61_ 0.65.61 54.517.00_ 0.67.00 67.320.58_ 0.60.58 67.030.72_ 0.60.72 65.460.93_ 0.60.93 67.030.45_ 0.60.45 67.102.22_ 0.62.22 66.790.75_ 0.60.75 + CofFeQA 56.879.31_ 0.69.31 65.583.89_ 0.63.89 62.011.86_ 0.61.86 58.254.73_ 0.64.73 63.553.92_ 0.63.92 61.253.63_ 0.63.63 66.411.07_ 0.61.07 67.550.54_ 0.60.54 67.300.79_ 0.60.79 68.630.97_ 0.60.97 70.521.08_ 0.61.08 68.081.57_ 0.61.57 SeqKD 67.691.02_ 0.61.02 67.820.94_ 0.60.94 67.812.05_ 0.62.05 64.145.13_ 0.65.13 66.563.21_ 0.63.21 66.801.58_ 0.61.58 68.140.77_ 0.60.77 69.020.94_ 0.60.94 68.650.43_ 0.60.43 66.000.72_ 0.60.72 67.301.53_ 0.61.53 67.821.20_ 0.61.20 + CofFeQA 68.321.38_ 0.61.38 70.320.59_ 0.60.59 69.002.03_ 0.62.03 68.082.52_ 0.62.52 66.134.35_ 0.64.35 68.371.52_ 0.61.52 65.880.68_ 0.60.68 66.571.26_ 0.61.26 68.680.60_ 0.60.60 69.320.64_ 0.60.64 68.281.59_ 0.61.59 67.741.46_ 0.61.46 KD 61.8810.53_ 0.610.53 60.8210.13_ 0.610.13 54.8312.63_ 0.612.63 56.0711.94_ 0.611.94 55.5611.40_ 0.611.40 57.8310.78_ 0.610.78 61.228.62_ 0.68.62 60.2410.27_ 0.610.27 59.1310.14_ 0.610.14 58.518.96_ 0.68.96 59.0710.13_ 0.610.13 59.649.59_ 0.69.59 + CofFeQA 63.369.01_ 0.69.01 64.209.42_ 0.69.42 62.939.55_ 0.69.55 59.6711.77_ 0.611.77 59.4911.12_ 0.611.12 61.939.98_ 0.69.98 61.736.60_ 0.66.60 61.598.40_ 0.68.40 61.789.26_ 0.69.26 60.939.25_ 0.69.25 60.598.39_ 0.68.39 61.328.32_ 0.68.32 LWD 61.7510.57_ 0.610.57 60.949.70_ 0.69.70 54.5213.16_ 0.613.16 55.4911.37_ 0.611.37 55.8811.36_ 0.611.36 57.7210.74_ 0.610.74 60.778.08_ 0.68.08 60.948.37_ 0.68.37 59.718.64_ 0.68.64 59.528.53_ 0.68.53 59.709.72_ 0.69.72 60.138.65_ 0.68.65 AVG + CofFeQA 62.949.22_ 0.69.22 64.389.67_ 0.69.67 62.729.50_ 0.69.50 59.9411.31_ 0.611.31 59.7811.22_ 0.611.22 61.959.96_ 0.69.96 61.106.66_ 0.66.66 61.518.19_ 0.68.19 61.498.65_ 0.68.65 61.808.08_ 0.68.08 62.038.55_ 0.68.55 61.597.91_ 0.67.91 SeqKD 61.779.91_ 0.69.91 61.1911.44_ 0.611.44 61.3711.27_ 0.611.27 61.1510.68_ 0.610.68 62.0610.93_ 0.610.93 61.5110.60_ 0.610.60 59.7211.46_ 0.611.46 60.4610.92_ 0.610.92 60.969.98_ 0.69.98 60.798.51_ 0.68.51 62.237.69_ 0.67.69 60.839.65_ 0.69.65 + CofFeQA 64.818.90_ 0.68.90 64.7010.09_ 0.610.09 64.5010.03_ 0.610.03 64.5411.13_ 0.611.13 64.069.72_ 0.69.72 64.529.83_ 0.69.83 59.9810.51_ 0.610.51 60.729.59_ 0.69.59 62.169.12_ 0.69.12 62.218.66_ 0.68.66 62.887.50_ 0.67.50 61.598.95_ 0.68.95 H.2 In-Domain Agreement Results Tables 9 and 10 report the in-domain teacher-student agreement results using EM and F1, respectively. For better visualization, Figures 11 and 12 illustrate the F1 agreement across datasets and k-shot settings for all baselines using T5 and Qwen2.5, respectively. Table 9: In-domain teacher-student agreement EM score comparison across k-shots for T5 and Qwen2.5 models using baselines. Mean agreement EM scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. baseline). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg NQ KD 56.234.87_ 0.64.87 60.202.87_ 0.62.87 54.476.56_ 0.66.56 50.267.34_ 0.67.34 52.625.00_ 0.65.00 54.763.76_ 0.63.76 61.880.68_ 0.60.68 62.930.40_ 0.60.40 62.840.62_ 0.60.62 62.530.22_ 0.60.22 63.730.83_ 0.60.83 62.780.67_ 0.60.67 + CofFeQA 61.340.56_ 0.60.56 60.640.49_ 0.60.49 57.971.36_ 0.61.36 52.946.72_ 0.66.72 53.464.92_ 0.64.92 57.273.93_ 0.63.93 61.460.53_ 0.60.53 62.970.32_ 0.60.32 64.520.13_ 0.60.13 64.940.11_ 0.60.11 64.721.17_ 0.61.17 63.721.48_ 0.61.48 LWD 59.860.63_ 0.60.63 58.944.18_ 0.64.18 56.423.20_ 0.63.20 51.354.46_ 0.64.46 52.225.87_ 0.65.87 55.763.85_ 0.63.85 60.080.64_ 0.60.64 63.050.76_ 0.60.76 62.791.26_ 0.61.26 62.500.77_ 0.60.77 62.661.13_ 0.61.13 62.211.21_ 0.61.21 + CofFeQA 61.310.40_ 0.60.40 58.953.37_ 0.63.37 56.574.41_ 0.64.41 53.984.65_ 0.64.65 52.747.23_ 0.67.23 56.713.52_ 0.63.52 58.920.62_ 0.60.62 63.870.70_ 0.60.70 64.870.51_ 0.60.51 65.510.56_ 0.60.56 65.471.02_ 0.61.02 63.732.77_ 0.62.77 SeqKD 57.521.96_ 0.61.96 57.191.23_ 0.61.23 59.241.76_ 0.61.76 58.921.83_ 0.61.83 62.411.97_ 0.61.97 59.062.07_ 0.62.07 61.450.52_ 0.60.52 64.810.70_ 0.60.70 65.010.38_ 0.60.38 65.860.44_ 0.60.44 66.301.21_ 0.61.21 64.691.91_ 0.61.91 + CofFeQA 60.220.90_ 0.60.90 59.691.20_ 0.61.20 60.211.00_ 0.61.00 60.261.37_ 0.61.37 60.881.53_ 0.61.53 60.250.42_ 0.60.42 62.450.47_ 0.60.47 63.090.32_ 0.60.32 66.360.48_ 0.60.48 66.860.45_ 0.60.45 67.150.35_ 0.60.35 65.182.23_ 0.62.23 SQuAD KD 85.350.75_ 0.60.75 81.941.37_ 0.61.37 80.523.17_ 0.63.17 82.172.32_ 0.62.32 78.676.21_ 0.66.21 81.732.46_ 0.62.46 73.300.35_ 0.60.35 72.700.26_ 0.60.26 71.990.47_ 0.60.47 69.980.81_ 0.60.81 71.030.69_ 0.60.69 71.801.32_ 0.61.32 + CofFeQA 85.730.39_ 0.60.39 85.550.58_ 0.60.58 83.331.33_ 0.61.33 83.960.45_ 0.60.45 81.282.43_ 0.62.43 83.971.82_ 0.61.82 72.180.76_ 0.60.76 73.000.50_ 0.60.50 72.550.76_ 0.60.76 71.630.86_ 0.60.86 70.300.67_ 0.60.67 71.931.04_ 0.61.04 LWD 85.161.09_ 0.61.09 81.722.03_ 0.62.03 79.944.61_ 0.64.61 78.972.73_ 0.62.73 79.912.29_ 0.62.29 81.142.46_ 0.62.46 72.520.60_ 0.60.60 72.510.28_ 0.60.28 72.190.52_ 0.60.52 70.531.14_ 0.61.14 72.281.10_ 0.61.10 72.010.84_ 0.60.84 + CofFeQA 85.980.40_ 0.60.40 84.371.68_ 0.61.68 83.131.30_ 0.61.30 84.210.77_ 0.60.77 82.192.26_ 0.62.26 83.971.43_ 0.61.43 72.240.63_ 0.60.63 73.160.29_ 0.60.29 73.710.73_ 0.60.73 72.370.18_ 0.60.18 72.941.32_ 0.61.32 72.880.60_ 0.60.60 SeqKD 83.481.46_ 0.61.46 84.991.17_ 0.61.17 83.981.52_ 0.61.52 84.570.25_ 0.60.25 84.590.35_ 0.60.35 84.320.59_ 0.60.59 74.850.45_ 0.60.45 74.620.53_ 0.60.53 74.090.78_ 0.60.78 73.900.72_ 0.60.72 74.881.16_ 0.61.16 74.470.45_ 0.60.45 + CofFeQA 85.560.46_ 0.60.46 84.210.49_ 0.60.49 84.520.70_ 0.60.70 84.470.88_ 0.60.88 83.250.72_ 0.60.72 84.400.82_ 0.60.82 73.810.65_ 0.60.65 75.000.45_ 0.60.45 74.650.60_ 0.60.60 74.400.43_ 0.60.43 75.250.70_ 0.60.70 74.620.56_ 0.60.56 NewsQA KD 63.282.18_ 0.62.18 63.103.40_ 0.63.40 60.612.80_ 0.62.80 51.194.13_ 0.64.13 46.581.56_ 0.61.56 56.957.62_ 0.67.62 47.741.38_ 0.61.38 44.121.68_ 0.61.68 43.651.56_ 0.61.56 46.441.30_ 0.61.30 42.521.92_ 0.61.92 44.892.14_ 0.62.14 + CofFeQA 63.832.50_ 0.62.50 66.090.66_ 0.60.66 67.131.61_ 0.61.61 61.714.24_ 0.64.24 58.003.91_ 0.63.91 63.353.65_ 0.63.65 51.010.64_ 0.60.64 48.441.27_ 0.61.27 48.171.06_ 0.61.06 47.421.02_ 0.61.02 50.121.82_ 0.61.82 49.031.48_ 0.61.48 LWD 63.623.09_ 0.63.09 62.813.88_ 0.63.88 55.833.91_ 0.63.91 49.774.38_ 0.64.38 45.941.27_ 0.61.27 55.597.81_ 0.67.81 48.491.44_ 0.61.44 49.331.12_ 0.61.12 48.201.65_ 0.61.65 47.970.52_ 0.60.52 45.133.12_ 0.63.12 47.831.59_ 0.61.59 + CofFeQA 64.672.80_ 0.62.80 66.900.42_ 0.60.42 66.381.74_ 0.61.74 60.753.06_ 0.63.06 56.524.03_ 0.64.03 63.044.37_ 0.64.37 51.280.76_ 0.60.76 48.031.76_ 0.61.76 48.611.40_ 0.61.40 52.031.01_ 0.61.01 52.784.17_ 0.64.17 50.552.11_ 0.62.11 SeqKD 56.493.01_ 0.63.01 57.162.91_ 0.62.91 59.513.20_ 0.63.20 60.462.70_ 0.62.70 62.091.90_ 0.61.90 59.142.32_ 0.62.32 40.231.47_ 0.61.47 41.971.21_ 0.61.21 47.300.72_ 0.60.72 48.781.35_ 0.61.35 53.301.05_ 0.61.05 46.325.29_ 0.65.29 + CofFeQA 64.781.38_ 0.61.38 66.291.75_ 0.61.75 66.291.35_ 0.61.35 68.871.16_ 0.61.16 66.032.30_ 0.62.30 66.451.49_ 0.61.49 41.281.60_ 0.61.60 46.491.11_ 0.61.11 48.350.85_ 0.60.85 51.101.83_ 0.61.83 53.941.74_ 0.61.74 48.234.80_ 0.64.80 TriviaQA KD 56.443.22_ 0.63.22 53.712.02_ 0.62.02 49.563.44_ 0.63.44 49.462.68_ 0.62.68 50.351.12_ 0.61.12 51.903.07_ 0.63.07 63.261.05_ 0.61.05 64.151.19_ 0.61.19 62.570.60_ 0.60.60 61.251.79_ 0.61.79 63.391.28_ 0.61.28 62.931.09_ 0.61.09 + CofFeQA 62.240.99_ 0.60.99 59.900.68_ 0.60.68 59.316.57_ 0.66.57 52.622.03_ 0.62.03 53.152.57_ 0.62.57 57.444.31_ 0.64.31 62.930.78_ 0.60.78 61.910.41_ 0.60.41 64.750.79_ 0.60.79 63.761.02_ 0.61.02 60.762.52_ 0.62.52 62.821.56_ 0.61.56 LWD 55.293.79_ 0.63.79 54.202.68_ 0.62.68 47.641.98_ 0.61.98 48.601.94_ 0.61.94 49.751.24_ 0.61.24 51.103.43_ 0.63.43 62.111.30_ 0.61.30 64.380.96_ 0.60.96 63.301.03_ 0.61.03 61.480.73_ 0.60.73 62.312.00_ 0.62.00 62.721.14_ 0.61.14 + CofFeQA 61.051.30_ 0.61.30 58.422.54_ 0.62.54 58.257.48_ 0.67.48 52.781.70_ 0.61.70 52.881.59_ 0.61.59 56.683.68_ 0.63.68 62.801.00_ 0.61.00 62.670.61_ 0.60.61 64.250.69_ 0.60.69 63.531.05_ 0.61.05 61.291.53_ 0.61.53 62.911.11_ 0.61.11 SeqKD 57.833.70_ 0.63.70 51.832.61_ 0.62.61 46.922.27_ 0.62.27 46.852.66_ 0.62.66 44.481.74_ 0.61.74 49.585.33_ 0.65.33 63.301.11_ 0.61.11 61.781.17_ 0.61.17 63.261.49_ 0.61.49 62.470.97_ 0.60.97 65.310.86_ 0.60.86 63.221.32_ 0.61.32 + CofFeQA 60.461.49_ 0.61.49 53.082.02_ 0.62.02 52.322.40_ 0.62.40 47.713.82_ 0.63.82 48.901.86_ 0.61.86 52.494.99_ 0.64.99 63.720.50_ 0.60.50 64.450.86_ 0.60.86 64.550.80_ 0.60.80 63.761.30_ 0.61.30 63.061.45_ 0.61.45 63.910.61_ 0.60.61 HotpotQA KD 56.974.68_ 0.64.68 58.657.33_ 0.67.33 42.357.99_ 0.67.99 53.014.67_ 0.64.67 54.896.91_ 0.66.91 53.176.41_ 0.66.41 64.740.39_ 0.60.39 66.060.32_ 0.60.32 62.801.31_ 0.61.31 63.130.77_ 0.60.77 63.701.31_ 0.61.31 64.091.33_ 0.61.33 + CofFeQA 60.364.46_ 0.64.46 58.955.76_ 0.65.76 57.541.95_ 0.61.95 53.814.75_ 0.64.75 61.993.86_ 0.63.86 58.533.11_ 0.63.11 64.051.24_ 0.61.24 65.770.39_ 0.60.39 66.670.79_ 0.60.79 66.180.63_ 0.60.63 66.340.69_ 0.60.69 65.801.03_ 0.61.03 LWD 53.635.91_ 0.65.91 55.498.74_ 0.68.74 38.306.33_ 0.66.33 52.325.27_ 0.65.27 54.965.91_ 0.65.91 50.947.17_ 0.67.17 65.570.56_ 0.60.56 65.700.78_ 0.60.78 64.050.46_ 0.60.46 65.050.58_ 0.60.58 65.602.06_ 0.62.06 65.190.69_ 0.60.69 + CofFeQA 54.049.44_ 0.69.44 62.834.28_ 0.64.28 58.022.00_ 0.62.00 54.384.91_ 0.64.91 60.444.15_ 0.64.15 57.943.81_ 0.63.81 65.491.12_ 0.61.12 65.990.60_ 0.60.60 66.260.61_ 0.60.61 66.711.14_ 0.61.14 67.860.48_ 0.60.48 66.460.90_ 0.60.90 SeqKD 67.390.69_ 0.60.69 66.150.80_ 0.60.80 66.402.12_ 0.62.12 61.395.83_ 0.65.83 65.403.45_ 0.63.45 65.352.32_ 0.62.32 68.091.04_ 0.61.04 68.430.41_ 0.60.41 66.760.83_ 0.60.83 63.960.46_ 0.60.46 65.521.40_ 0.61.40 66.551.85_ 0.61.85 + CofFeQA 65.511.38_ 0.61.38 67.131.28_ 0.61.28 67.131.28_ 0.61.28 65.782.88_ 0.62.88 63.594.39_ 0.64.39 65.821.46_ 0.61.46 67.040.65_ 0.60.65 67.071.30_ 0.61.30 68.360.90_ 0.60.90 67.780.63_ 0.60.63 66.810.96_ 0.60.96 67.410.64_ 0.60.64 KD 63.6512.48_ 0.612.48 63.5210.85_ 0.610.85 57.5014.50_ 0.614.50 57.2214.01_ 0.614.01 56.6212.70_ 0.612.70 59.7012.46_ 0.612.46 62.189.22_ 0.69.22 61.9910.68_ 0.610.68 60.7710.37_ 0.610.37 60.678.65_ 0.68.65 60.8810.75_ 0.610.75 61.309.90_ 0.69.90 + CofFeQA 66.7010.72_ 0.610.72 66.2211.15_ 0.611.15 65.0610.93_ 0.610.93 61.0113.36_ 0.613.36 61.5711.60_ 0.611.60 64.1111.37_ 0.611.37 62.337.57_ 0.67.57 62.428.93_ 0.68.93 63.339.07_ 0.69.07 62.789.10_ 0.69.10 62.457.70_ 0.67.70 62.668.41_ 0.68.41 LWD 63.5112.72_ 0.612.72 62.6311.18_ 0.611.18 55.6315.46_ 0.615.46 56.2012.81_ 0.612.81 56.5613.47_ 0.613.47 58.9112.65_ 0.612.65 61.758.79_ 0.68.79 63.008.46_ 0.68.46 62.108.67_ 0.68.67 61.518.34_ 0.68.34 61.5910.04_ 0.610.04 61.998.83_ 0.68.83 + CofFeQA 65.4112.13_ 0.612.13 66.2910.66_ 0.610.66 64.4711.12_ 0.611.12 61.2213.22_ 0.613.22 60.9512.28_ 0.612.28 63.6711.65_ 0.611.65 62.147.78_ 0.67.78 62.759.18_ 0.69.18 63.549.17_ 0.69.17 64.037.47_ 0.67.47 64.077.58_ 0.67.58 63.308.14_ 0.68.14 SeqKD 64.5411.47_ 0.611.47 63.4613.09_ 0.613.09 63.2113.57_ 0.613.57 62.4413.69_ 0.613.69 63.7914.25_ 0.614.25 63.4912.94_ 0.612.94 61.5813.01_ 0.613.01 62.3212.34_ 0.612.34 63.299.84_ 0.69.84 62.999.09_ 0.69.09 65.067.69_ 0.67.69 63.0510.31_ 0.610.31 AVG + CofFeQA 67.3110.49_ 0.610.49 66.0811.61_ 0.611.61 66.0911.88_ 0.611.88 65.4213.37_ 0.613.37 64.5312.36_ 0.612.36 65.8911.77_ 0.611.77 61.6612.22_ 0.612.22 63.2210.43_ 0.610.43 64.459.77_ 0.69.77 64.788.57_ 0.68.57 65.247.72_ 0.67.72 63.879.67_ 0.69.67 Table 10: In-domain teacher-student agreement F1 score comparison across k-shots for T5 and Qwen2.5 models using baselines. Mean agreement F1 scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. baseline). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg NQ KD 62.313.15_ 0.63.15 64.501.70_ 0.61.70 61.752.79_ 0.62.79 60.113.06_ 0.63.06 60.982.11_ 0.62.11 61.931.66_ 0.61.66 68.950.66_ 0.60.66 69.190.31_ 0.60.31 68.870.59_ 0.60.59 68.300.55_ 0.60.55 68.940.85_ 0.60.85 68.850.33_ 0.60.33 + CofFeQA 65.080.40_ 0.60.40 64.910.42_ 0.60.42 62.650.70_ 0.60.70 61.962.63_ 0.62.63 61.362.37_ 0.62.37 63.191.71_ 0.61.71 68.180.32_ 0.60.32 70.290.31_ 0.60.31 70.470.34_ 0.60.34 70.630.28_ 0.60.28 70.450.76_ 0.60.76 70.011.03_ 0.61.03 LWD 64.220.62_ 0.60.62 63.552.69_ 0.62.69 62.551.45_ 0.61.45 60.322.31_ 0.62.31 61.122.68_ 0.62.68 62.351.63_ 0.61.63 67.540.52_ 0.60.52 69.091.01_ 0.61.01 69.071.11_ 0.61.11 68.780.64_ 0.60.64 69.190.52_ 0.60.52 68.730.68_ 0.60.68 + CofFeQA 65.110.37_ 0.60.37 63.842.50_ 0.62.50 62.652.03_ 0.62.03 62.311.45_ 0.61.45 61.583.01_ 0.63.01 63.101.39_ 0.61.39 66.100.51_ 0.60.51 71.110.54_ 0.60.54 70.930.65_ 0.60.65 71.540.24_ 0.60.24 71.350.52_ 0.60.52 70.212.30_ 0.62.30 SeqKD 63.331.54_ 0.61.54 62.021.16_ 0.61.16 63.641.50_ 0.61.50 62.601.83_ 0.61.83 65.661.78_ 0.61.78 63.451.39_ 0.61.39 68.480.35_ 0.60.35 71.630.55_ 0.60.55 70.870.41_ 0.60.41 71.580.11_ 0.60.11 71.850.78_ 0.60.78 70.881.39_ 0.61.39 + CofFeQA 64.850.67_ 0.60.67 64.230.93_ 0.60.93 64.230.78_ 0.60.78 63.711.28_ 0.61.28 64.051.47_ 0.61.47 64.210.42_ 0.60.42 69.590.42_ 0.60.42 70.190.45_ 0.60.45 72.230.57_ 0.60.57 72.150.50_ 0.60.50 72.300.54_ 0.60.54 71.291.30_ 0.61.30 SQuAD KD 90.290.71_ 0.60.71 87.730.93_ 0.60.93 87.201.46_ 0.61.46 88.171.55_ 0.61.55 85.903.23_ 0.63.23 87.861.60_ 0.61.60 79.820.28_ 0.60.28 79.140.08_ 0.60.08 78.580.53_ 0.60.53 78.140.70_ 0.60.70 78.640.43_ 0.60.43 78.860.64_ 0.60.64 + CofFeQA 90.450.25_ 0.60.25 90.430.40_ 0.60.40 88.880.85_ 0.60.85 89.130.39_ 0.60.39 87.341.48_ 0.61.48 89.251.29_ 0.61.29 79.140.46_ 0.60.46 79.470.57_ 0.60.57 79.450.58_ 0.60.58 78.700.69_ 0.60.69 78.220.45_ 0.60.45 79.000.53_ 0.60.53 LWD 90.160.61_ 0.60.61 87.721.35_ 0.61.35 87.322.34_ 0.62.34 86.301.00_ 0.61.00 85.981.50_ 0.61.50 87.501.65_ 0.61.65 79.020.53_ 0.60.53 79.180.26_ 0.60.26 78.920.30_ 0.60.30 78.330.99_ 0.60.99 79.561.05_ 0.61.05 79.000.45_ 0.60.45 + CofFeQA 90.770.30_ 0.60.30 89.780.79_ 0.60.79 88.640.83_ 0.60.83 89.370.57_ 0.60.57 87.921.87_ 0.61.87 89.301.09_ 0.61.09 78.790.21_ 0.60.21 79.720.27_ 0.60.27 80.290.39_ 0.60.39 79.430.28_ 0.60.28 80.141.43_ 0.61.43 79.670.60_ 0.60.60 SeqKD 89.300.97_ 0.60.97 90.120.85_ 0.60.85 89.540.67_ 0.60.67 89.510.20_ 0.60.20 89.330.28_ 0.60.28 89.560.33_ 0.60.33 80.780.43_ 0.60.43 80.210.32_ 0.60.32 80.050.73_ 0.60.73 79.790.50_ 0.60.50 80.671.04_ 0.61.04 80.300.42_ 0.60.42 + CofFeQA 90.620.50_ 0.60.50 89.300.51_ 0.60.51 89.430.66_ 0.60.66 89.630.45_ 0.60.45 88.750.71_ 0.60.71 89.550.68_ 0.60.68 79.770.44_ 0.60.44 80.530.29_ 0.60.29 80.450.29_ 0.60.29 80.310.42_ 0.60.42 81.220.64_ 0.60.64 80.460.52_ 0.60.52 NewsQA KD 72.061.88_ 0.61.88 72.842.12_ 0.62.12 71.261.39_ 0.61.39 67.682.77_ 0.62.77 64.941.45_ 0.61.45 69.763.34_ 0.63.34 58.691.01_ 0.61.01 56.241.74_ 0.61.74 56.721.78_ 0.61.78 57.301.50_ 0.61.50 54.541.80_ 0.61.80 56.701.52_ 0.61.52 + CofFeQA 73.002.11_ 0.62.11 73.970.76_ 0.60.76 75.481.49_ 0.61.49 73.082.41_ 0.62.41 72.801.84_ 0.61.84 73.671.11_ 0.61.11 61.020.67_ 0.60.67 58.951.08_ 0.61.08 59.330.52_ 0.60.52 59.990.98_ 0.60.98 61.682.02_ 0.62.02 60.191.14_ 0.61.14 LWD 72.442.55_ 0.62.55 72.602.24_ 0.62.24 69.241.92_ 0.61.92 67.642.49_ 0.62.49 63.581.11_ 0.61.11 69.103.74_ 0.63.74 60.001.41_ 0.61.41 61.230.59_ 0.60.59 60.021.51_ 0.61.51 59.981.16_ 0.61.16 56.852.27_ 0.62.27 59.611.64_ 0.61.64 + CofFeQA 73.942.16_ 0.62.16 74.730.65_ 0.60.65 74.811.57_ 0.61.57 72.801.39_ 0.61.39 71.651.83_ 0.61.83 73.591.35_ 0.61.35 61.540.86_ 0.60.86 58.891.36_ 0.61.36 59.971.08_ 0.61.08 62.521.07_ 0.61.07 63.614.34_ 0.64.34 61.311.90_ 0.61.90 SeqKD 68.571.99_ 0.61.99 70.941.79_ 0.61.79 71.072.51_ 0.62.51 69.371.92_ 0.61.92 72.491.46_ 0.61.46 70.491.54_ 0.61.54 48.961.45_ 0.61.45 51.490.86_ 0.60.86 59.390.35_ 0.60.35 60.591.39_ 0.61.39 63.640.99_ 0.60.99 56.816.28_ 0.66.28 + CofFeQA 74.010.97_ 0.60.97 75.271.16_ 0.61.16 76.351.83_ 0.61.83 76.271.39_ 0.61.39 74.781.61_ 0.61.61 75.331.00_ 0.61.00 51.151.39_ 0.61.39 57.400.32_ 0.60.32 58.421.31_ 0.61.31 61.761.65_ 0.61.65 64.651.69_ 0.61.69 58.675.09_ 0.65.09 TriviaQA KD 59.243.43_ 0.63.43 55.962.19_ 0.62.19 51.303.58_ 0.63.58 51.042.70_ 0.62.70 51.791.05_ 0.61.05 53.873.61_ 0.63.61 65.521.01_ 0.61.01 66.501.00_ 0.61.00 64.890.54_ 0.60.54 63.971.56_ 0.61.56 66.341.09_ 0.61.09 65.441.05_ 0.61.05 + CofFeQA 65.440.86_ 0.60.86 62.680.75_ 0.60.75 61.916.87_ 0.66.87 54.531.97_ 0.61.97 54.982.30_ 0.62.30 59.914.89_ 0.64.89 65.540.55_ 0.60.55 64.650.42_ 0.60.42 67.330.66_ 0.60.66 66.120.89_ 0.60.89 63.442.38_ 0.62.38 65.411.47_ 0.61.47 LWD 58.134.26_ 0.64.26 56.472.80_ 0.62.80 49.352.13_ 0.62.13 50.192.02_ 0.62.02 51.131.31_ 0.61.31 53.053.97_ 0.63.97 64.931.23_ 0.61.23 66.710.93_ 0.60.93 65.660.93_ 0.60.93 64.230.69_ 0.60.69 65.401.85_ 0.61.85 65.390.92_ 0.60.92 + CofFeQA 64.151.26_ 0.61.26 61.392.69_ 0.62.69 60.847.86_ 0.67.86 54.741.74_ 0.61.74 54.691.53_ 0.61.53 59.164.25_ 0.64.25 65.291.07_ 0.61.07 65.220.76_ 0.60.76 66.850.73_ 0.60.73 65.930.86_ 0.60.86 64.031.46_ 0.61.46 65.471.04_ 0.61.04 SeqKD 64.783.94_ 0.63.94 57.822.78_ 0.62.78 52.952.61_ 0.62.61 53.122.81_ 0.62.81 50.101.85_ 0.61.85 55.755.76_ 0.65.76 65.541.08_ 0.61.08 64.281.20_ 0.61.20 65.491.09_ 0.61.09 65.371.11_ 0.61.11 68.030.88_ 0.60.88 65.741.38_ 0.61.38 + CofFeQA 66.541.86_ 0.61.86 59.622.26_ 0.62.26 58.502.34_ 0.62.34 54.023.93_ 0.63.93 54.051.96_ 0.61.96 58.555.14_ 0.65.14 66.130.46_ 0.60.46 66.650.95_ 0.60.95 66.840.68_ 0.60.68 66.081.02_ 0.61.02 65.891.23_ 0.61.23 66.320.41_ 0.60.41 HotpotQA KD 71.012.73_ 0.62.73 71.114.62_ 0.64.62 65.093.52_ 0.63.52 69.611.64_ 0.61.64 68.773.88_ 0.63.88 69.122.46_ 0.62.46 74.130.47_ 0.60.47 75.680.37_ 0.60.37 72.551.07_ 0.61.07 73.180.70_ 0.60.70 73.361.14_ 0.61.14 73.781.20_ 0.61.20 + CofFeQA 71.542.28_ 0.62.28 72.142.83_ 0.62.83 71.250.72_ 0.60.72 69.382.28_ 0.62.28 73.262.65_ 0.62.65 71.511.42_ 0.61.42 73.660.94_ 0.60.94 75.040.38_ 0.60.38 76.430.66_ 0.60.66 75.880.54_ 0.60.54 76.241.15_ 0.61.15 75.451.13_ 0.61.13 LWD 69.132.52_ 0.62.52 69.974.08_ 0.64.08 62.943.25_ 0.63.25 69.411.96_ 0.61.96 70.023.04_ 0.63.04 68.293.02_ 0.63.02 75.340.48_ 0.60.48 75.910.89_ 0.60.89 73.950.52_ 0.60.52 75.460.68_ 0.60.68 75.561.81_ 0.61.81 75.240.76_ 0.60.76 + CofFeQA 68.893.68_ 0.63.68 73.802.02_ 0.62.02 71.171.64_ 0.61.64 69.832.23_ 0.62.23 73.232.24_ 0.62.24 71.382.12_ 0.62.12 75.371.05_ 0.61.05 76.110.59_ 0.60.59 76.210.50_ 0.60.50 77.180.96_ 0.60.96 77.990.62_ 0.60.62 76.571.02_ 0.61.02 SeqKD 76.560.65_ 0.60.65 76.061.09_ 0.61.09 75.411.83_ 0.61.83 73.753.42_ 0.63.42 74.703.20_ 0.63.20 75.291.11_ 0.61.11 76.230.70_ 0.60.70 77.170.47_ 0.60.47 75.420.59_ 0.60.59 72.530.26_ 0.60.26 74.291.49_ 0.61.49 75.131.80_ 0.61.80 + CofFeQA 74.791.16_ 0.61.16 76.270.53_ 0.60.53 76.040.88_ 0.60.88 75.072.49_ 0.62.49 73.293.56_ 0.63.56 75.091.19_ 0.61.19 75.240.56_ 0.60.56 75.291.52_ 0.61.52 77.080.83_ 0.60.83 76.610.52_ 0.60.52 75.591.35_ 0.61.35 75.960.83_ 0.60.83 KD 70.9812.11_ 0.612.11 70.4311.73_ 0.611.73 67.3213.26_ 0.613.26 67.3213.76_ 0.613.76 66.4812.56_ 0.612.56 68.5112.59_ 0.612.59 69.428.07_ 0.68.07 69.358.89_ 0.68.89 68.328.21_ 0.68.21 68.188.07_ 0.68.07 68.369.03_ 0.69.03 68.738.42_ 0.68.42 + CofFeQA 73.1010.33_ 0.610.33 72.8310.92_ 0.610.92 72.0311.03_ 0.611.03 69.6213.03_ 0.613.03 69.9512.44_ 0.612.44 71.5011.44_ 0.611.44 69.517.06_ 0.67.06 69.688.15_ 0.68.15 70.607.91_ 0.67.91 70.267.51_ 0.67.51 70.017.40_ 0.67.40 70.017.55_ 0.67.55 LWD 70.8212.09_ 0.612.09 70.0611.68_ 0.611.68 66.2813.81_ 0.613.81 66.7713.28_ 0.613.28 66.3612.90_ 0.612.90 68.0612.62_ 0.612.62 69.367.74_ 0.67.74 70.427.19_ 0.67.19 69.527.30_ 0.67.30 69.357.62_ 0.67.62 69.318.87_ 0.68.87 69.607.72_ 0.67.72 + CofFeQA 72.5710.88_ 0.610.88 72.7111.22_ 0.611.22 71.6211.15_ 0.611.15 69.8112.99_ 0.612.99 69.8112.64_ 0.612.64 71.3111.66_ 0.611.66 69.427.30_ 0.67.30 70.218.35_ 0.68.35 70.857.94_ 0.67.94 71.327.18_ 0.67.18 71.427.66_ 0.67.66 70.647.60_ 0.67.60 SeqKD 72.5110.70_ 0.610.70 71.3912.69_ 0.612.69 70.5213.62_ 0.613.62 69.6713.55_ 0.613.55 70.4614.28_ 0.614.28 70.9112.77_ 0.612.77 68.0012.24_ 0.612.24 68.9611.49_ 0.611.49 70.248.12_ 0.68.12 69.977.33_ 0.67.33 71.706.44_ 0.66.44 69.779.01_ 0.69.01 AVG + CofFeQA 74.1610.20_ 0.610.20 72.9411.59_ 0.611.59 72.9112.02_ 0.612.02 71.7413.51_ 0.613.51 70.9812.94_ 0.612.94 72.5511.92_ 0.611.92 68.3810.96_ 0.610.96 70.018.79_ 0.68.79 71.008.71_ 0.68.71 71.387.55_ 0.67.55 71.936.88_ 0.66.88 70.548.47_ 0.68.47 Figure 11: Teacher-student F1 agreement of KD, LWD, and SeqKD with and without CofFeQA across datasets and k-shot settings using T5. CofFeQA generally achieves higher agreement across the distillation baselines. Spokes denote k-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization. Figure 12: Teacher-student F1 agreement of KD, LWD, and SeqKD with and without CofFeQA across datasets and k-shot settings using Qwen2.5. CofFeQA generally achieves higher agreement across the distillation baselines. Spokes denote k-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization. H.3 Out-of-Domain EM Results Table 11 reports the out-of-domain EM results for T5 and Qwen2.5. Table 11: Out-of-domain EM score comparison across k-shots for T5 and Qwen2.5 models using KD. Mean EM scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. KD). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg DROP KD 34.751.78_ 0.61.78 32.004.64_ 0.64.64 27.563.50_ 0.63.50 22.634.29_ 0.64.29 23.801.97_ 0.61.97 28.155.20_ 0.65.20 44.730.78_ 0.60.78 39.191.71_ 0.61.71 36.930.33_ 0.60.33 35.030.90_ 0.60.90 36.571.04_ 0.61.04 38.493.79_ 0.63.79 + CofFeQA 35.681.01_ 0.61.01 33.861.47_ 0.61.47 28.363.54_ 0.63.54 25.983.56_ 0.63.56 24.652.56_ 0.62.56 29.714.85_ 0.64.85 44.040.73_ 0.60.73 44.321.35_ 0.61.35 39.561.51_ 0.61.51 37.581.03_ 0.61.03 35.191.10_ 0.61.10 40.144.00_ 0.64.00 DuoRC KD 44.631.56_ 0.61.56 43.123.14_ 0.63.14 41.471.91_ 0.61.91 38.171.43_ 0.61.43 39.331.84_ 0.61.84 41.352.65_ 0.62.65 33.990.80_ 0.60.80 38.531.06_ 0.61.06 36.661.60_ 0.61.60 36.621.10_ 0.61.10 36.621.80_ 0.61.80 36.481.62_ 0.61.62 + CofFeQA 46.010.84_ 0.60.84 44.630.87_ 0.60.87 39.022.37_ 0.62.37 40.941.17_ 0.61.17 40.041.08_ 0.61.08 42.133.03_ 0.63.03 33.680.84_ 0.60.84 37.551.39_ 0.61.39 36.880.78_ 0.60.78 36.700.87_ 0.60.87 38.581.48_ 0.61.48 36.681.83_ 0.61.83 RelationExtraction KD 74.711.74_ 0.61.74 75.130.68_ 0.60.68 74.491.19_ 0.61.19 66.705.61_ 0.65.61 66.653.71_ 0.63.71 71.534.44_ 0.64.44 78.380.36_ 0.60.36 78.950.41_ 0.60.41 78.000.17_ 0.60.17 76.381.13_ 0.61.13 76.880.52_ 0.60.52 77.721.06_ 0.61.06 + CofFeQA 74.620.58_ 0.60.58 74.341.86_ 0.61.86 75.321.82_ 0.61.82 69.844.12_ 0.64.12 69.222.65_ 0.62.65 72.672.90_ 0.62.90 78.290.87_ 0.60.87 78.730.49_ 0.60.49 79.030.38_ 0.60.38 77.060.89_ 0.60.89 75.901.59_ 0.61.59 77.801.30_ 0.61.30 KD 51.3620.81_ 0.620.81 50.0822.39_ 0.622.39 47.8424.11_ 0.624.11 42.5022.35_ 0.622.35 43.2621.70_ 0.621.70 47.0122.24_ 0.622.24 52.3623.16_ 0.623.16 52.2223.15_ 0.623.15 50.5323.79_ 0.623.79 49.3423.43_ 0.623.43 50.0223.26_ 0.623.26 50.9023.25_ 0.623.25 AVG + CofFeQA 52.1020.17_ 0.620.17 50.9420.96_ 0.620.96 47.5724.62_ 0.624.62 45.5822.29_ 0.622.29 44.6422.64_ 0.622.64 48.1722.11_ 0.622.11 52.0023.35_ 0.623.35 53.5322.08_ 0.622.08 51.8223.60_ 0.623.60 50.4523.05_ 0.623.05 49.8922.59_ 0.622.59 51.5422.81_ 0.622.81 H.4 Performance under Modified Contextual Evidence Tables 12 and 13 report the EM and F1 scores on modified-context test sets, respectively. Tables 14 and 15 report the corresponding teacher-student agreement scores using EM and F1, respectively. Table 12: EM score comparison on modified-context test sets across k-shot settings for T5 and Qwen2.5 models using KD. Mean EM scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. KD). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg NQ KD 54.904.68_ 0.64.68 58.162.79_ 0.62.79 53.185.88_ 0.65.88 49.006.84_ 0.66.84 51.184.85_ 0.64.85 53.283.51_ 0.63.51 52.100.64_ 0.60.64 52.980.66_ 0.60.66 52.900.36_ 0.60.36 51.530.57_ 0.60.57 53.040.68_ 0.60.68 52.510.67_ 0.60.67 + CofFeQA 58.700.78_ 0.60.78 59.330.62_ 0.60.62 56.201.23_ 0.61.23 51.836.47_ 0.66.47 52.344.46_ 0.64.46 55.683.49_ 0.63.49 51.650.51_ 0.60.51 53.901.04_ 0.61.04 56.100.50_ 0.60.50 56.730.33_ 0.60.33 56.931.20_ 0.61.20 55.062.26_ 0.62.26 SQuAD KD 78.140.84_ 0.60.84 75.710.88_ 0.60.88 74.522.49_ 0.62.49 75.631.17_ 0.61.17 73.435.21_ 0.65.21 75.491.75_ 0.61.75 60.370.35_ 0.60.35 59.740.66_ 0.60.66 59.180.35_ 0.60.35 60.340.78_ 0.60.78 60.980.41_ 0.60.41 60.120.69_ 0.60.69 + CofFeQA 78.420.30_ 0.60.30 78.490.37_ 0.60.37 77.051.20_ 0.61.20 76.980.61_ 0.60.61 75.771.62_ 0.61.62 77.341.14_ 0.61.14 58.480.40_ 0.60.40 60.800.38_ 0.60.38 61.080.74_ 0.60.74 62.571.35_ 0.61.35 64.771.63_ 0.61.63 61.542.32_ 0.62.32 NewsQA KD 64.292.04_ 0.62.04 64.583.03_ 0.63.03 63.192.52_ 0.62.52 54.843.97_ 0.63.97 50.490.94_ 0.60.94 59.486.43_ 0.66.43 39.541.29_ 0.61.29 34.991.04_ 0.61.04 34.991.38_ 0.61.38 33.421.22_ 0.61.22 32.902.06_ 0.62.06 35.172.61_ 0.62.61 + CofFeQA 64.093.66_ 0.63.66 67.251.36_ 0.61.36 67.971.77_ 0.61.77 64.203.27_ 0.63.27 61.913.52_ 0.63.52 65.082.49_ 0.62.49 46.090.98_ 0.60.98 43.251.35_ 0.61.35 39.421.14_ 0.61.14 40.122.49_ 0.62.49 40.841.85_ 0.61.85 41.942.73_ 0.62.73 TriviaQA KD 51.603.07_ 0.63.07 48.432.15_ 0.62.15 44.683.20_ 0.63.20 44.382.50_ 0.62.50 45.501.21_ 0.61.21 46.923.07_ 0.63.07 38.250.63_ 0.60.63 32.950.74_ 0.60.74 34.331.89_ 0.61.89 33.382.00_ 0.62.00 34.302.85_ 0.62.85 34.642.11_ 0.62.11 + CofFeQA 57.031.10_ 0.61.10 54.600.90_ 0.60.90 53.716.19_ 0.66.19 48.111.60_ 0.61.60 48.142.06_ 0.62.06 52.324.02_ 0.64.02 38.062.03_ 0.62.03 43.460.73_ 0.60.73 39.511.46_ 0.61.46 45.441.55_ 0.61.55 44.712.20_ 0.62.20 42.233.27_ 0.63.27 HotpotQA KD 60.634.26_ 0.64.26 62.125.92_ 0.65.92 46.318.29_ 0.68.29 56.834.49_ 0.64.49 58.266.67_ 0.66.67 56.836.23_ 0.66.23 43.390.55_ 0.60.55 44.140.88_ 0.60.88 40.871.19_ 0.61.19 45.360.93_ 0.60.93 47.451.66_ 0.61.66 44.242.43_ 0.62.43 + CofFeQA 63.174.19_ 0.64.19 62.565.51_ 0.65.51 61.812.05_ 0.62.05 57.244.83_ 0.64.83 64.813.95_ 0.63.95 61.922.84_ 0.62.84 45.220.40_ 0.60.40 45.850.49_ 0.60.49 46.250.62_ 0.60.62 46.410.65_ 0.60.65 54.051.30_ 0.61.30 47.563.66_ 0.63.66 KD 61.9110.32_ 0.610.32 61.809.92_ 0.69.92 56.3712.49_ 0.612.49 56.1411.95_ 0.611.95 55.7710.87_ 0.610.87 58.4010.65_ 0.610.65 46.739.35_ 0.69.35 44.9611.48_ 0.611.48 44.4511.10_ 0.611.10 44.8011.69_ 0.611.69 45.7312.09_ 0.612.09 45.3411.06_ 0.611.06 AVG + CofFeQA 64.288.44_ 0.68.44 64.449.11_ 0.69.11 63.359.43_ 0.69.43 59.6711.42_ 0.611.42 60.5910.87_ 0.610.87 62.479.71_ 0.69.71 47.907.64_ 0.67.64 49.457.68_ 0.67.68 48.479.80_ 0.69.80 50.259.14_ 0.69.14 52.269.60_ 0.69.60 49.678.51_ 0.68.51 Table 13: F1 score comparison on modified-context test sets across k-shot settings for T5 and Qwen2.5 models using KD. Mean F1 scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. KD). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg NQ KD 60.033.26_ 0.63.26 61.701.69_ 0.61.69 59.732.45_ 0.62.45 57.842.95_ 0.62.95 58.492.03_ 0.62.03 59.561.49_ 0.61.49 54.520.43_ 0.60.43 55.360.65_ 0.60.65 55.400.54_ 0.60.54 54.070.58_ 0.60.58 55.290.50_ 0.60.50 54.930.60_ 0.60.60 + CofFeQA 62.050.53_ 0.60.53 62.650.65_ 0.60.65 60.040.70_ 0.60.70 59.792.52_ 0.62.52 59.301.90_ 0.61.90 60.761.49_ 0.61.49 53.740.50_ 0.60.50 56.260.96_ 0.60.96 58.430.45_ 0.60.45 59.070.30_ 0.60.30 59.511.25_ 0.61.25 57.402.40_ 0.62.40 SQuAD KD 86.280.78_ 0.60.78 84.150.74_ 0.60.74 83.721.19_ 0.61.19 84.551.04_ 0.61.04 82.902.76_ 0.62.76 84.321.26_ 0.61.26 65.260.33_ 0.60.33 64.490.59_ 0.60.59 64.190.38_ 0.60.38 65.670.74_ 0.60.74 66.220.60_ 0.60.60 65.170.83_ 0.60.83 + CofFeQA 86.520.19_ 0.60.19 86.510.35_ 0.60.35 85.280.98_ 0.60.98 85.340.62_ 0.60.62 84.311.31_ 0.61.31 85.590.94_ 0.60.94 63.820.44_ 0.60.44 65.570.28_ 0.60.28 66.120.75_ 0.60.75 67.461.36_ 0.61.36 69.791.74_ 0.61.74 66.552.23_ 0.62.23 NewsQA KD 71.992.01_ 0.62.01 73.561.97_ 0.61.97 72.391.33_ 0.61.33 69.033.06_ 0.63.06 66.170.96_ 0.60.96 70.633.00_ 0.63.00 46.191.28_ 0.61.28 42.101.00_ 0.61.00 42.911.56_ 0.61.56 39.781.20_ 0.61.20 39.192.34_ 0.62.34 42.032.80_ 0.62.80 + CofFeQA 72.813.00_ 0.63.00 74.101.22_ 0.61.22 75.661.73_ 0.61.73 73.921.69_ 0.61.69 74.231.75_ 0.61.75 74.151.02_ 0.61.02 52.160.74_ 0.60.74 49.461.12_ 0.61.12 46.001.24_ 0.61.24 47.371.99_ 0.61.99 47.491.25_ 0.61.25 48.502.39_ 0.62.39 TriviaQA KD 54.793.30_ 0.63.30 51.312.29_ 0.62.29 46.773.37_ 0.63.37 46.182.44_ 0.62.44 47.351.26_ 0.61.26 49.283.68_ 0.63.68 40.220.69_ 0.60.69 34.680.70_ 0.60.70 35.751.90_ 0.61.90 34.982.06_ 0.62.06 36.233.00_ 0.63.00 36.372.24_ 0.62.24 + CofFeQA 60.721.02_ 0.61.02 57.890.89_ 0.60.89 57.056.86_ 0.66.86 50.471.55_ 0.61.55 50.222.09_ 0.62.09 55.274.70_ 0.64.70 39.772.14_ 0.62.14 45.270.70_ 0.60.70 41.431.40_ 0.61.40 46.841.40_ 0.61.40 46.572.29_ 0.62.29 43.983.19_ 0.63.19 HotpotQA KD 71.872.37_ 0.62.37 71.753.96_ 0.63.96 66.223.20_ 0.63.20 70.061.28_ 0.61.28 69.103.78_ 0.63.78 69.802.32_ 0.62.32 47.080.52_ 0.60.52 47.940.70_ 0.60.70 44.261.25_ 0.61.25 48.551.05_ 0.61.05 50.531.57_ 0.61.57 47.672.29_ 0.62.29 + CofFeQA 71.731.89_ 0.61.89 72.792.70_ 0.62.70 72.240.81_ 0.60.81 69.882.26_ 0.62.26 73.222.78_ 0.62.78 71.971.30_ 0.61.30 49.560.43_ 0.60.43 49.780.31_ 0.60.31 49.920.57_ 0.60.57 50.010.54_ 0.60.54 57.381.43_ 0.61.43 51.333.39_ 0.63.39 KD 68.9912.23_ 0.612.23 68.4912.48_ 0.612.48 65.7613.82_ 0.613.82 65.5314.39_ 0.614.39 64.8013.15_ 0.613.15 66.7213.13_ 0.613.13 50.669.62_ 0.69.62 48.9211.56_ 0.611.56 48.5011.24_ 0.611.24 48.6112.09_ 0.612.09 49.4912.21_ 0.612.21 49.2311.25_ 0.611.25 AVG + CofFeQA 70.7610.37_ 0.610.37 70.7911.12_ 0.611.12 70.0611.59_ 0.611.59 67.8813.36_ 0.613.36 68.2613.45_ 0.613.45 69.5511.89_ 0.611.89 51.818.63_ 0.68.63 53.277.92_ 0.67.92 52.389.90_ 0.69.90 54.158.91_ 0.68.91 56.159.56_ 0.69.56 53.558.75_ 0.68.75 Table 14: Teacher-student agreement EM score comparison on modified-context test sets across k-shot settings for T5 and Qwen2.5 models using KD. Mean agreement EM scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. KD). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg NQ KD 56.234.87_ 0.64.87 60.202.87_ 0.62.87 54.476.56_ 0.66.56 50.267.34_ 0.67.34 52.625.00_ 0.65.00 54.763.76_ 0.63.76 53.690.61_ 0.60.61 55.750.87_ 0.60.87 55.440.53_ 0.60.53 54.310.82_ 0.60.82 55.250.27_ 0.60.27 54.890.86_ 0.60.86 + CofFeQA 61.340.56_ 0.60.56 60.640.49_ 0.60.49 57.971.36_ 0.61.36 52.946.72_ 0.66.72 53.464.92_ 0.64.92 57.273.93_ 0.63.93 52.620.58_ 0.60.58 56.011.03_ 0.61.03 57.340.62_ 0.60.62 58.570.68_ 0.60.68 58.500.62_ 0.60.62 56.612.46_ 0.62.46 SQuAD KD 85.350.75_ 0.60.75 81.941.37_ 0.61.37 80.523.17_ 0.63.17 82.172.32_ 0.62.32 78.676.21_ 0.66.21 81.732.46_ 0.62.46 61.860.51_ 0.60.51 61.420.73_ 0.60.73 61.100.59_ 0.60.59 61.371.08_ 0.61.08 61.690.77_ 0.60.77 61.490.29_ 0.60.29 + CofFeQA 85.730.39_ 0.60.39 85.550.58_ 0.60.58 83.331.33_ 0.61.33 83.960.45_ 0.60.45 81.282.43_ 0.62.43 83.971.82_ 0.61.82 60.450.43_ 0.60.43 62.220.32_ 0.60.32 62.221.01_ 0.61.01 63.670.99_ 0.60.99 64.480.86_ 0.60.86 62.611.55_ 0.61.55 NewsQA KD 63.282.18_ 0.62.18 63.103.40_ 0.63.40 60.612.80_ 0.62.80 51.194.13_ 0.64.13 46.581.56_ 0.61.56 56.957.62_ 0.67.62 41.801.25_ 0.61.25 37.860.82_ 0.60.82 37.510.50_ 0.60.50 36.871.29_ 0.61.29 35.971.75_ 0.61.75 38.002.24_ 0.62.24 + CofFeQA 63.832.50_ 0.62.50 66.090.66_ 0.60.66 67.131.61_ 0.61.61 61.714.24_ 0.64.24 58.003.91_ 0.63.91 63.353.65_ 0.63.65 46.121.78_ 0.61.78 43.590.93_ 0.60.93 40.840.80_ 0.60.80 41.911.62_ 0.61.62 43.301.75_ 0.61.75 43.151.99_ 0.61.99 TriviaQA KD 56.443.22_ 0.63.22 53.712.02_ 0.62.02 49.563.44_ 0.63.44 49.462.68_ 0.62.68 50.351.12_ 0.61.12 51.903.07_ 0.63.07 39.041.14_ 0.61.14 40.131.66_ 0.61.66 38.250.98_ 0.60.98 38.160.98_ 0.60.98 38.061.68_ 0.61.68 38.730.88_ 0.60.88 + CofFeQA 62.240.99_ 0.60.99 59.900.68_ 0.60.68 59.316.57_ 0.66.57 52.622.03_ 0.62.03 53.152.57_ 0.62.57 57.444.31_ 0.64.31 37.461.34_ 0.61.34 37.690.63_ 0.60.63 39.671.35_ 0.61.35 35.420.26_ 0.60.26 34.531.08_ 0.61.08 36.962.03_ 0.62.03 HotpotQA KD 56.974.68_ 0.64.68 58.657.33_ 0.67.33 42.357.99_ 0.67.99 53.014.67_ 0.64.67 54.896.91_ 0.66.91 53.176.41_ 0.66.41 50.380.70_ 0.60.70 50.920.64_ 0.60.64 46.891.09_ 0.61.09 50.651.52_ 0.61.52 51.491.81_ 0.61.81 50.071.82_ 0.61.82 + CofFeQA 60.364.46_ 0.64.46 58.955.76_ 0.65.76 57.541.95_ 0.61.95 53.814.75_ 0.64.75 61.993.86_ 0.63.86 58.533.11_ 0.63.11 51.860.89_ 0.60.89 52.230.58_ 0.60.58 52.810.67_ 0.60.67 52.180.93_ 0.60.93 56.611.45_ 0.61.45 53.141.97_ 0.61.97 KD 63.6512.48_ 0.612.48 63.5210.85_ 0.610.85 57.5014.50_ 0.614.50 57.2214.01_ 0.614.01 56.6212.70_ 0.612.70 59.7012.46_ 0.612.46 49.359.21_ 0.69.21 49.2110.08_ 0.610.08 47.8410.41_ 0.610.41 48.2710.56_ 0.610.56 48.4911.12_ 0.611.12 48.6310.22_ 0.610.22 AVG + CofFeQA 66.7010.72_ 0.610.72 66.2211.15_ 0.611.15 65.0610.93_ 0.610.93 61.0113.36_ 0.613.36 61.5711.60_ 0.611.60 64.1111.37_ 0.611.37 49.708.54_ 0.68.54 50.359.78_ 0.69.78 50.5710.00_ 0.610.00 50.3511.65_ 0.611.65 51.4812.23_ 0.612.23 50.4910.35_ 0.610.35 Table 15: Teacher-student agreement F1 score comparison on modified-context test sets across k-shot settings for T5 and Qwen2.5 models using KD. Mean agreement F1 scores are reported with standard deviations as subscripts (MeanstdMean_std). Bold values indicate the better-performing method (CofFeQA vs. KD). Blue shading highlights aggregate results. T5 Qwen2.5 Dataset Method 16 32 64 128 256 Avg 16 32 64 128 256 Avg NQ KD 62.313.15_ 0.63.15 64.501.70_ 0.61.70 61.752.79_ 0.62.79 60.113.06_ 0.63.06 60.982.11_ 0.62.11 61.931.66_ 0.61.66 59.300.46_ 0.60.46 61.210.76_ 0.60.76 60.870.35_ 0.60.35 59.530.70_ 0.60.70 60.450.37_ 0.60.37 60.270.83_ 0.60.83 + CofFeQA 65.080.40_ 0.60.40 64.910.42_ 0.60.42 62.650.70_ 0.60.70 61.962.63_ 0.62.63 61.362.37_ 0.62.37 63.191.71_ 0.61.71 58.040.48_ 0.60.48 61.551.04_ 0.61.04 62.830.51_ 0.60.51 63.710.58_ 0.60.58 64.030.59_ 0.60.59 62.032.43_ 0.62.43 SQuAD KD 90.290.71_ 0.60.71 87.730.93_ 0.60.93 87.201.46_ 0.61.46 88.171.55_ 0.61.55 85.903.23_ 0.63.23 87.861.60_ 0.61.60 67.690.47_ 0.60.47 67.240.39_ 0.60.39 66.980.45_ 0.60.45 68.040.86_ 0.60.86 68.340.57_ 0.60.57 67.660.56_ 0.60.56 + CofFeQA 90.450.25_ 0.60.25 90.430.40_ 0.60.40 88.880.85_ 0.60.85 89.130.39_ 0.60.39 87.341.48_ 0.61.48 89.251.29_ 0.61.29 66.490.57_ 0.60.57 67.980.16_ 0.60.16 68.370.80_ 0.60.80 69.490.87_ 0.60.87 71.001.13_ 0.61.13 68.661.69_ 0.61.69 NewsQA KD 72.061.88_ 0.61.88 72.842.12_ 0.62.12 71.261.39_ 0.61.39 67.682.77_ 0.62.77 64.941.45_ 0.61.45 69.763.34_ 0.63.34 50.551.41_ 0.61.41 47.131.02_ 0.61.02 47.651.17_ 0.61.17 45.171.52_ 0.61.52 44.412.23_ 0.62.23 46.982.40_ 0.62.40 + CofFeQA 73.002.11_ 0.62.11 73.970.76_ 0.60.76 75.481.49_ 0.61.49 73.082.41_ 0.62.41 72.801.84_ 0.61.84 73.671.11_ 0.61.11 54.761.54_ 0.61.54 52.541.06_ 0.61.06 50.421.16_ 0.61.16 51.801.19_ 0.61.19 51.951.57_ 0.61.57 52.291.58_ 0.61.58 TriviaQA KD 59.243.43_ 0.63.43 55.962.19_ 0.62.19 51.303.58_ 0.63.58 51.042.70_ 0.62.70 51.791.05_ 0.61.05 53.873.61_ 0.63.61 41.481.15_ 0.61.15 42.621.27_ 0.61.27 40.530.74_ 0.60.74 40.391.21_ 0.61.21 40.481.58_ 0.61.58 41.100.96_ 0.60.96 + CofFeQA 65.440.86_ 0.60.86 62.680.75_ 0.60.75 61.916.87_ 0.66.87 54.531.97_ 0.61.97 54.982.30_ 0.62.30 59.914.89_ 0.64.89 39.751.33_ 0.61.33 40.070.55_ 0.60.55 42.011.32_ 0.61.32 37.770.19_ 0.60.19 37.381.09_ 0.61.09 39.401.88_ 0.61.88 HotpotQA KD 71.012.73_ 0.62.73 71.114.62_ 0.64.62 65.093.52_ 0.63.52 69.611.64_ 0.61.64 68.773.88_ 0.63.88 69.122.46_ 0.62.46 56.900.71_ 0.60.71 57.620.58_ 0.60.58 53.651.08_ 0.61.08 57.201.38_ 0.61.38 58.011.66_ 0.61.66 56.681.74_ 0.61.74 + CofFeQA 71.542.28_ 0.62.28 72.142.83_ 0.62.83 71.250.72_ 0.60.72 69.382.28_ 0.62.28 73.262.65_ 0.62.65 71.511.42_ 0.61.42 58.740.70_ 0.60.70 59.120.62_ 0.60.62 59.700.64_ 0.60.64 58.930.87_ 0.60.87 63.021.12_ 0.61.12 59.901.78_ 0.61.78 KD 70.9812.11_ 0.612.11 70.4311.73_ 0.611.73 67.3213.26_ 0.613.26 67.3213.76_ 0.613.76 66.4812.56_ 0.612.56 68.5112.59_ 0.612.59 55.189.82_ 0.69.82 55.1610.13_ 0.610.13 53.9410.46_ 0.610.46 54.0711.19_ 0.611.19 54.3411.59_ 0.611.59 54.5410.58_ 0.610.58 AVG + CofFeQA 73.1010.33_ 0.610.33 72.8310.92_ 0.610.92 72.0311.03_ 0.611.03 69.6213.03_ 0.613.03 69.9512.44_ 0.612.44 71.5011.44_ 0.611.44 55.569.83_ 0.69.83 56.2510.60_ 0.610.60 56.6710.46_ 0.610.46 56.3412.24_ 0.612.24 57.4713.14_ 0.613.14 56.4611.18_ 0.611.18