Paper deep dive
Where Should a Document Live: Context, Representations, or Parameters?
Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias, Adrià de Gispert
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.
Tags
Links
- Source: https://arxiv.org/abs/2609.17346v1
- Canonical: https://arxiv.org/abs/2609.17346v1
Trouble viewing inline? Open PDF directly →
Full Text
67,687 characters extracted from source content.
Expand or collapse full text
Where Should a Document Live: Context, Representations, or Parameters? Nathanaël Carraz Rakotonirina Momchil Hardalov Affiliation: Gonzalo Iglesias Adrià de Gispert Affiliation: Amazon AGI Affiliation: ncarraz, momchilh, gjii, agispert@amazon.com Abstract To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model’s parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than 50×50×. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding. 1 Introduction Large language models (LLMs) often need additional information, such as enterprise knowledge bases, patient records, or legal corpora to answer questions that were not part of their training data. Techniques such as in-context learning (Brown et al., 2020, ICL) and retrieval-augmented generation (Lewis et al., 2020, RAG) offer a straightforward solution by placing the document in the context window. However, they are bounded by the context length and require re-processing the entire document for every query. Knowledge injection methods, on the other hand, encode the documents once into compact adapters, replacing the textual context during inference (Eyuboglu et al., 2025; Su et al., 2025; Caccia et al., 2025). These methods fall into two families: Representation-based methods, which incorporate the document into a compressed key–value (KV) cache prefix (e.g. Cartridges (Eyuboglu et al., 2025) and Compaction (Zweiger et al., 2026)); and Parametric methods, which encode the knowledge into the model parameters (e.g. full fine-tuning, LoRA (Hu et al., 2022), or MLP adapters (Houlsby et al., 2019)). Throughout the paper we use adapters as an umbrella term for both KV caches and model weights. Even though these methods are widely used in practice, prior work mainly evaluates question-answering (QA) tasks over Wikipedia documents that the model has likely already seen in pre-training, therefore testing recall rather than genuinely encoding new knowledge (Ovadia et al., 2024; Su et al., 2025). Eyuboglu et al. (2025) takes a step towards a fairer comparison between parameter-based and representation-based methods, but it covers only a limited set of methods and in a single-document setting. In contrast, in our work we compare head-to-head five state-of-the-art adaptation approaches on five knowledge-intensive tasks, both in the single-document and in the more realistic multi-document retrieval setting. Moreover, we quantify catastrophic forgetting on representative control benchmarks (math, instruction following, general knowledge, coding) and perform a storage-matched and runtime analysis of each method. Our analysis reveals that there is no single best method, and which method to use will depend on the use case. Our main contributions are the following: • We present a holistic, controlled comparison of different knowledge injection methods: (i) representation-based (Cartridges, Compaction); and (i) parametric (LoRA, MLP adapters, full fine-tuning) on knowledge-intensive tasks in both single- and multiple-document settings. • We study how adapter (KV caches or weights) size influences model performance and catastrophic forgetting, quantifying the trade-off for each knowledge injection method. • We show that adapter composition in a multi-document retrieval setting is possible under certain conditions, but it remains challenging. 2 Related work Parameter-Efficient Fine-Tuning. PEFT methods adapt a model by training only a small subset of its parameters or representations. They can be broadly categorized into two groups. Representation-based methods prepend a sequence of trainable vectors to the input and optimize only those vectors, and sometimes their activations across layers (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2022; Eyuboglu et al., 2025). Parametric methods train a small number of parameters. Representative examples include adapters (Houlsby et al., 2019), small trainable modules inserted between layers, and LoRA (Hu et al., 2022), which learns a low-rank update to the weight matrices. Compared with full fine-tuning, PEFT offers several advantages: it requires less memory and compute to train, it can be stored and served efficiently (Chen et al., 2023), and multiple modules or adapters can be composed (Yadav et al., 2023; Zhong et al., 2024; Prabhakar et al., 2025; Hardalov et al., 2026). Finally, it better preserves the model’s general capabilities. In particular, LoRA matches the accuracy and sample efficiency of full fine-tuning while being less prone to catastrophic forgetting (Schulman and Lab, 2025). Knowledge injection. Once Large Language Models (LLMs) are trained, they encode only the knowledge present in their training data. To answer questions about specific documents, a knowledge base, or any other source of knowledge, they must go through an adaptation phase. The most straightforward way is to add the relevant documents to the model’s context, possibly preceded by a retrieval stage that selects and filters the documents to be inserted (Lewis et al., 2020; Guu et al., 2020; Gao et al., 2023). An alternative to in-context injection is to encode the documents directly in the representations (Kujanpä et al., 2024; Rakotonirina and Baroni, 2024; Eyuboglu et al., 2025) or parameters (Xiao et al., 2023; Su et al., 2025) of the model. Representation-based knowledge injection is akin to prompt compression methods (Mu et al., 2023; Chevalier et al., 2023; Qin et al., 2024; Łajewska et al., 2025) that encode the context into shorter sequences of vectors or soft tokens. Recent work (Eyuboglu et al., 2025; Su et al., 2025; Caccia et al., 2025) has demonstrated that the best way to inject new knowledge is to train on synthetic data derived from the corpus rather than on the corpus itself, and to use a distillation objective rather than next-token prediction. Comparing context, representations, and parameters. Several lines of work compare context-based and parametric knowledge injection, but reach mixed and narrowly scoped conclusions. Ovadia et al. (2024) report that retrieval-augmented generation (RAG), which relies on in-context learning, consistently beats unsupervised fine-tuning for fact injection. However, they do not cover supervised fine-tuning or parameter-efficient adaptation. Parametric RAG (Su et al., 2025) encodes each document into a separate LoRA adapter and merges the adapters of the retrieved documents at inference time. It reports higher accuracy than in-context RAG in addition to lower latency. Subsequent work (Tang et al., 2025) shows that parametric methods do not consistently outperform text-based RAG, and that combining the two performs best. Both these works train and evaluate on QA datasets whose documents are Wikipedia paragraphs that were presumably already seen during pretraining, so the model is tested on recalling facts already encoded in its weights rather than on encoding new ones. In contrast, we evaluate on knowledge-intensive tasks with uncontaminated documents, as confirmed by our No Context baseline, which queries the LLM without access to external documents. Closest to our setting, Eyuboglu et al. (2025) compare Cartridges against in-context learning and LoRA, but report only limited results for the multi-document case where adapters are composed. We instead run an extensive, controlled comparison that spans all adaptation families and knowledge-intensive datasets varying widely in document length and task type, covering both the single-document setting (only the gold document is encoded) and the more realistic multi-document setting (several documents are retrieved and composed). 3 Methodology Adaptation methods. We conduct a holistic evaluation on a variety of state-of-the-art adaptation methods from different families: context-based, representation-based, and parametric. We evaluate the following methods: • No context: only the question is given to the model. This baseline quantifies how much can be answered from pretraining knowledge alone, with no access to external documents. • ICL: the document is placed in the model’s context. This is the standard in-context baseline. In the multiple document setting, this is standard text-based RAG. • Cartridges (Eyuboglu et al., 2025): a representation-based method that trains a small KV-cache prefix that is prepended to the question during inference. For the multi-document setting, Cartridges are trained following Hardalov et al. (2026). • Compaction (Zweiger et al., 2026): a representation-based method that compresses the document into a compact set of key–value pairs by matching the attention the model would place on the full document. • LoRA (Hu et al., 2022): a parametric method that adds trainable low-rank updates to the feed-forward projections of every layer, keeping the base model frozen. • MLP adapters (Houlsby et al., 2019): a parametric method that inserts a small MLP after the attention block of every layer, training only these modules while keeping the base model frozen. • Full fine-tuning: a parametric method that updates all model parameters, serving as an upper bound on capacity. Knowledge injection settings. The task we are focusing on is what is the best approach for injecting new knowledge into a model. Importantly, we do not limit the QA task to a single-document scenario, where only the gold document is passed along with the questions in the model context, but also consider the more realistic scenario where the model answers based on documents retrieved from the corpus. In our experiments, we learn one adapter per document and pass only the question and the system instruction into the model prompt. These two settings can be formalized as follows: • Single (gold) document: This is the oracle setting, i.e., we encode only the gold document in the model prompt; the goal is to isolate how knowledge in a single document is injected, independent of retrieval. • Multiple documents: This follows the RAG setting, i.e., for each question we retrieve the top-k chunks and compose their corresponding adapters. For the representation-based methods we concatenate the k retrieved KV caches, and for the parametric methods we merge the k retrieved weights into a single set of weights. We have compared different merging approaches in Appendix B, based on which we selected the simple model merge by averaging. We also jointly train on all documents for parametric methods. 4 Experimental Settings Dataset Docs Questions Qs/Doc Avg. Tok. Total Tok. Task Type Domain LongHealth 20 400 20.0 11,700 236K Multiple-choice (5-way) Clinical patient records QASPER 407 1,451 3.5 4,751 665K Extract./free-form/yes-no Full research papers QuALITY 115 2,086 18.1 5,713 1.9M Multiple-choice (4-way) Fiction & non-fiction narratives T2-RB/FinQA 380 1,147 3.0 1,026 392K Math. Calculation Corporate earnings reports w/ tables TechQA 496 610 1.8 1,509 748K Extractive IBM IT support technotes Table 1: Dataset statistics where Docs is the number of unique documents (one cartridge per document), Qs/Doc is the average number of questions per document, and Avg. Tok. is the average document length in tokens (Qwen3-8B tokenizer). Models. We use Qwen3-8B (Yang et al., 2025) as our main model. We also validate our results on Gemma-3-12B (Team, 2025) in Appendix C. Dataset. We evaluate on the following datasets (see Table 1 for dataset statistics) using their corresponding metric:11 1 We adopted the datasets and metrics as described in Hardalov et al. (2026). • LongHealth Adams et al. (2024): A clinical QA benchmark consisting of detailed fictional patient records with multiple-choice questions that test information extraction, negation understanding, and temporal sorting over long medical notes. We extract the model’s answer by fuzzy option matching, then compute exact match against the gold option. • QASPER Dasigi et al. (2021): An info-seeking QA dataset over full NLP research papers, where questions are written by NLP practitioners who saw only the title and abstract. Answers include extractive spans, free-form text, and yes/no responses. The metric is token-level F1 with the reference answers. • QuALITY Pang et al. (2022): A multiple-choice reading comprehension benchmark over long-form fiction and non-fiction narratives, where questions require careful reading and reasoning rather than surface-level pattern matching. The metric is exact match on the extracted answer letter (A, B, C, or D). • T2-RAGBench/FinQA Strich et al. (2026); Chen et al. (2021b): A decontextualized variant of FinQA designed for RAG evaluation, where questions about corporate earnings reports have been rewritten to be context-independent. The task requires numerical reasoning by constructing mathematical formulas over tables and text extracted from financial filings. The model is expected to produce a formula, which we evaluate to a numerical value and compare against the ground-truth answer with a relative tolerance of 1%. • TechQA Castelli et al. (2020): A technical support QA dataset drawn from IBM’s Technote corpus, where questions require extracting precise solutions from IT documentation covering enterprise software and infrastructure issues. We use DeepSeek-Distilled-Qwen-32B DeepSeek-AI (2025) as a judge to evaluate the model’s prediction against the reference answer (see Appendix F for more details). To assess whether knowledge injection degrades the model’s general abilities, we additionally evaluate on four control benchmarks: GSM8K Cobbe et al. (2021) for grade-school math, HumanEval Chen et al. (2021a) for code generation, IFEval Zhou et al. (2023) for instruction following, and MMLU Hendrycks et al. (2021) for broad knowledge. We evaluate adapted models and compare against the original model to measure catastrophic forgetting (Section 6). Training data. Following Eyuboglu et al. (2025) and Hardalov et al. (2026), we train on LLM-generated synthetic data called Self-Study data. More concretely, a question-generator LLM receives a part of the original document and a seed prompt, and produces a set of questions. An answer-generator LLM, given the same document, but not the seed prompt, answers each question providing tokens and their log probability distributions. Same as Hardalov et al. (2026) we use a GPT-OSS 120B (Agarwal et al., 2025) as the question generator and our target model as the answer generator. We generate n=20n=20 questions per chunk (more details in Appendix G). We train all methods using the same dataset. Method LongHealth QuALITY QASPER FinQA TechQA Avg. (Acc.) (Acc.) (F1) (EM) (Judge) No context 37.5 ± 1.1 43.6 ± 0.4 19.2 ± 0.6 2.9 ± 0.0 21.1 ± 1.1 24.9 ICL 87.4 ± 0.8 82.5 ± 0.3 56.7 ± 0.4 66.8 ± 2.7 74.7 ± 0.9 73.6 Cartridge 81.1 ± 1.1 78.6 ± 0.9 54.9 ± 0.3 62.7 ± 0.2 75.8 ± 0.7 70.6 Compaction 87.7 ± 0.9 82.1 ± 0.3 54.8 ± 0.1 66.4 ± 0.2 76.0 ± 2.4 73.4 LoRA 75.3 ± 0.9 73.7 ± 1.2 50.3 ± 0.6 49.0 ± 0.9 74.3 ± 2.6 64.5 MLP adapters 74.8 ± 0.9 72.3 ± 1.2 47.5 ± 0.9 44.1 ± 0.3 72.5 ± 1.8 62.2 Full fine-tuning 69.0 ± 2.4 72.8 ± 0.3 39.8 ± 0.3 41.6 ± 2.8 76.0 ± 3.4 59.8 Table 2: Single-document scores at a fixed adapter size (Qwen3-8B). Cartridge and Compaction use 2×2× compression, LoRA rank 6464, and MLP a bottleneck of 512512. Scores are averaged across 3 runs (± standard deviation); bold marks the best method per dataset and underline the second best. Objective function. We use the distillation objective that minimizes the KL divergence between a teacher, which is a model with the document in context, and a student, which is the same model to be adapted. We select this distillation objective since it achieves a better performance compared to next-token-prediction for knowledge injection (Eyuboglu et al., 2025). We corroborate these findings for LoRA in Appendix D. Retrieval. For the multi-document setting, we index the documents into chunks and retrieve the relevant chunks for each question, without query reformulation. For ICL and representation-based methods, the documents or KV caches are concatenated following their retrieval order. With 1,024-token chunks, LongHealth and QuALITY average ∼2 2 chunks per unique adapter, while for TechQA and FinQA the ratio is closer to 1:1 (see Table 9). We provide more details about retrieval and the RAG baseline in Appendix H. Hyperparameters. For the fixed adapter size experiments, we use a compression rate of 2×2× for Cartridge and Compaction, a rank of 6464 (with α=128α=128) for LoRA, and a bottleneck dimension of 512512 for MLP adapters. LoRA is applied to the feed-forward projections of every layer, and the MLP adapters are added as a residual bottleneck on the output of every transformer layer. When comparing the methods across a range of adapter sizes, we vary the compression rate over 2×2×, 10×10×, 20×20×, 50×50×, and 100×100×. We then choose the LoRA rank and the MLP bottleneck dimension so that each adapter matches the per-document memory footprint of the KV-cache methods at that rate. Because storage is tied to document length, these matched sizes are dataset-dependent. We do not report a storage-matched adapter at 100×100× because the target footprint falls below the smallest trainable adapter (a rank of at least one). During inference, we use the decoding hyperparameters recommended for Qwen3 (Yang et al., 2025): temperature 0.60.6, top-p 0.950.95, and top-k 2020, with a maximum of 4,0964,096 generated tokens. We list full training hyperparameters in Appendix A. 5 Experimental Results Figure 1: Single-document score versus adapter size (MiB, log scale). Cartridges and Compaction are swept over compression rates 2×2×, 10×10×, 20×20×, 50×50×, and 100×100×. LoRA and the MLP adapter use the rank and bottleneck dimension that match the KV-cache memory footprint at each rate. 5.1 Single Oracle document Fixed-size adapter. Table 2 shows the model performance in oracle setting, i.e., providing only the encoded gold document. No Context (24.9) performs far below the other methods, confirming that the knowledge is genuinely injected rather than recalled. This gap is largest on FinQA, where answering a question requires extracting specific numbers from the text and tables of the document. In this setting, representation-based methods have a clear edge and outperform the parametric ones on nearly all datasets. Compaction is strongest overall with an average score of 73.4, matching the ICL upper bound of 73.6, followed by Cartridges (70.6). On the parametric side, LoRA is the best method (64.5 on average), ahead of MLP adapters (62.2) and full fine-tuning (59.8). Full fine-tuning is only competitive on TechQA and weakest on datasets whose metric penalizes malformed output (QASPER F1, FinQA formulas), which we relate to catastrophic forgetting (studied in § Catastrophic forgetting). Comparing adapter sizes. We also compare the methods across a range of adapter sizes in the single-document setting in Figure 1. The two representation-based methods behave very differently as the compression rate increases. Compaction performs best at low compression but degrades steeply, overtaken by the parametric methods at high compression: on FinQA it falls from 66.466.4 at 2×2× to 19.319.3 at 20×20×, and on LongHealth from 87.787.7 to 46.546.5 at 100×100×. Cartridges instead stay nearly flat (e.g. LongHealth 81.1→77.381.1→ 77.3, QuALITY 78.6→76.478.6→ 76.4, TechQA 75.8→76.975.8→ 76.9 from 2×2× to 100×100×) and dominate the storage-matched parametric methods on every dataset. LoRA and MLP adapter improve with size up to a point and then plateau. We confirm this by training wider adapters in Appendix A. Figure 2: Multi-document results (Qwen3-8B). Score versus the number of retrieved documents k, composing the top-k retrieved per-document artifacts. (Top) Fixed adapter size. ICL refers to RAG in this setting, ICL (Oracle) is the in-context upper bound with the gold document in context, and Joint is a single adapter trained on all documents. (Bottom) Varying adapter size. Each method is swept over the storage-matched compression ratios (where available), with method encoded by color and compression rate by shade (dark = larger adapter / lower compression). 5.2 Multiple retrieved documents Fixed-size adapter. We retrieve the top-k chunks for each question and compose their adapters (mapping each chunk to its source document and removing duplicates): KV caches are concatenated for the representation-based methods, and weights are averaged for the parametric methods. Figure 2 reports performance against the number of retrieved documents (k=1,3,5,10k=1,3,5,10). In the comparison, we include a single adapter, trained jointly on all documents only for the parametric methods, as Hardalov et al. (2026) shows that Cartridges perform worse when trained jointly, while Compaction shifts the rope embedding to the original size of the compacted caches, which goes beyond the context window of the model. ICL and Cartridges maintain or improve their performance as k increases: from k=1k=1 to k=10k=10, Cartridges rise on LongHealth (70.8→83.270.8→ 83.2) and hold on TechQA (70.7→70.970.7→ 70.9) and QuALITY (72.5→74.672.5→ 74.6). In contrast, both Compaction and the merged parametric adapters degrade monotonically. Compaction drops from k=1k=1 to k=10k=10 on every dataset (LongHealth 72.9→56.372.9→ 56.3, TechQA 65.4→26.465.4→ 26.4), and the merged adapters fall even faster: LoRA on TechQA collapses from 57.457.4 at k=1k=1 to 23.523.5 at k=3k=3, and on QuALITY from 70.870.8 to 44.244.2 at k=10k=10. The degradation is most severe on FinQA, where merging collapses accuracy from 34.534.5 at k=1k=1 to 4.94.9 at k=3k=3. Joint training outperforms merging as soon as a single distractor is added (e.g. FinQA 13.013.0 vs. 4.94.9 and TechQA 40.040.0 vs. 23.523.5 at k=3k=3), except on QuALITY, where it takes the lead only at k=5k=5. Compaction and parametric injection are thus effective in the single-document setting but do not support multi-document composition. Figure 3: Forgetting on control benchmarks (Qwen3-8B). The score is averaged across the five source datasets, and error bars are the standard error across those datasets. (Top) The representation-based methods are swept over compression rate; the lowest-compression column merges the 2×2× and 3×3× points (the two longest datasets use 3×3×). (Bottom) The parametric methods are swept over adapter size. Varying adapter size. We repeat the multi-document analysis across adapter sizes in Figure 2. Cartridges are mostly stable across compression rates; on LongHealth, for example, the k=10k=10 score is 83.283.2 at 2×2× and still 76.576.5 at 100×100×. The only exception is FinQA, where heavier compression degrades performance (50.750.7 down to 22.122.1 from 2×2× to 20×20×), as in the single-document case. For LoRA and MLP, adapter size has essentially no effect on the composed result: by k=10k=10 their scores converge regardless of the rate, to around 55 points on FinQA and 4040–4949 on LongHealth at both 2×2× and 50×50×. Compaction behaves similarly under composition but is further hurt by compression, dropping on LongHealth at k=10k=10 from 56.356.3 at 2×2× to 37.237.2 at 50×50×. In short, using a larger rank, a wider bottleneck, or a lower compression rate does not fix the composition problem. 6 Analysis Catastrophic forgetting. We assess how knowledge injection affects the general capabilities of the model by evaluating the adapted models on four control benchmarks, sampling 3 adapters (an adapter encodes a single document) per training dataset. We plot performance as a function of compression rate for the representation-based methods (Figure 3, top) and as a function of adapter size for the parametric methods (Figure 3, bottom). Among the representation-based methods, Compaction stays at the base model at every compression rate, while Cartridges degrade by 6% on average, mainly on code generation, where HumanEval drops by 16 points at high compression. In the parametric methods, LoRA retains the base model’s capabilities at every rank, whereas the MLP adapter is stable up to a moderate bottleneck (d≤384d≤384) and then degrades as it widens: on GSM8K it falls from ∼92 92 at the smallest bottleneck to ∼60 60 at the largest, with parallel drops on every benchmark. The large MLP adapters are trained at a lower learning rate (2×10−52× 10^-5 instead of 10−410^-4), which we found necessary to avoid a much steeper collapse (e.g., GSM8K down to 2424 at 10−410^-4). LoRA is more robust at matched parameter count, pointing to the low-rank constraint, rather than the number of parameters alone, as what accounts for preserving general capability. Cost analysis. At inference time, ICL processes all the retrieved documents for each query during the prefill phase22 2 Prefilling cost can be attenuated over time with prompt caching, which is also applicable to the representation-based methods. and then decodes while attending over all of them. This can be a large recurring cost, from about 11k tokens on FinQA to 1111k on LongHealth for the document alone, paid on every query. Representation-based and parametric knowledge injection methods remove the document from the prompt, prefilling only the question. The representation-based methods still attend over a stored prefix, but a compressed one, so their per-query cost shrinks with the compression rate. With representation-based methods a 10×10× token reduction yields ∼100× 100× fewer prefill FLOPs due to quadratic attention. At inference, adapters are often pre-computed BF16 tensors, so loading requires only a GPU memory transfer, not a forward pass. On Qwen3-8B with H200 GPUs, loading 10 cartridges at 20×20× compression (6K KV tokens, 850 MiB) takes 30–50 ms, compared with 400–800 ms to prefill the equivalent 120K raw tokens. For the parametric methods the model attends to only the question, as the document lives in the weights. Overall, every injection method is cheaper per query than ICL, with the parametric methods the cheapest at inference and independent of document length. 7 Discussion Which method to use. We find that no single method dominates across all considered settings. Cartridges are the most accurate knowledge injection method, including in the more realistic multi-document setting, but they degrade the model’s general capabilities at high compression rates. Compaction does not suffer from any forgetting but only matches Cartridges in the single-document setting and at low compression. The parametric methods lag behind Cartridges irrespective of the adapter size. Their advantage is instead that the adapter size does not depend on the document and that they are less prone to forgetting, except for large MLP adapters, which are never chosen in practice as they do not bring any performance boost. Jointly fine-tuning on all documents outperforms merging the parametric adapters when the number of retrieved documents is high. Full fine-tuning is dominated throughout: it is the most expensive to train and store yet the least accurate, and it forgets more than any low-rank adapter. Composition is not straightforward. Combining per-document artifacts does not always work out of the box. This holds across families except for Cartridges: merging low-rank or full-rank adapters collapses with even a single distractor, and concatenating several independently compressed caches degrades in the same way. Although Su et al. (2025) showed promising results with merging LoRA adapters as an alternative to RAG, their findings were limited to small Wikipedia snippets, and the approach does not hold for the more complex tasks we consider. The composition of Cartridges in (Eyuboglu et al., 2025) was also limited, but the mixed training proposed in Hardalov et al. (2026) allows cartridges to be concatenated and perform well in the multi-document setting. A class of methods called KV-cache reuse addresses the problem of concatenating independently compressed caches, and could be applied to Compaction. KV Packet (Chen et al., 2026) and KVLink (Yang et al., 2026) add trainable soft tokens to bridge the discontinuities between the caches. C2KV (Du et al., 2026) instead compresses and composes jointly, learning a compression objective under which the caches remain mergeable rather than optimizing each cache alone. For the parametric adapters, we are not aware of any competitive approach to combine per-document weights: more advanced weight-merging techniques do not lead to any significant improvement (Appendix B). Joint fine-tuning is the best available option but still lags composed cartridges, especially on information-dense datasets like FinQA. On catastrophic forgetting. Among the representation-based methods, only Cartridges suffer from catastrophic forgetting on HumanEval. This suggests that the forgetting does not inherently come from the compressed KV caches. A possible fix is to initialize the KV caches using Compaction during Cartridges training. Regarding the parametric methods, only the full-rank MLP adapter forgets despite using the same number of parameters for all methods. Full fine-tuning also severely degrades the model’s general capabilities, suggesting that it is the full-rank nature of the update, rather than the parameter budget, that drives catastrophic forgetting. The low-rank constraint thus acts as a regularizer, suggesting that any parametric injection method intended to preserve general ability should be built around a subspace constraint. 8 Conclusion We present a controlled comparison of representation-based and parametric knowledge injection methods across five knowledge-intensive benchmarks, in both single- and multi-document settings. We find that no method wins on every axis, but the trade-offs are clear. Cartridges are the most accurate injection method at nearly every storage budget and the only one that extends to multiple retrieved documents, at the cost of mild forgetting on the control benchmarks. Compaction matches the in-context oracle at low compression but degrades quickly as compression increases. The parametric adapters trail Cartridges at matched storage and cannot be composed across documents, but they are fixed-size and cheap to serve. Composing independently trained per-document artifacts remains the main open problem; for the representation-based family, compressed-cache reuse is a promising route. Limitations Our study focuses on two mid-sized instruction-tuned models, Qwen3-8B and Gemma-3-12B, and we do not test whether the same trade-offs hold for larger models. We also only investigate knowledge-intensive tasks and do not cover acquiring or improving skills, such as reasoning or coding, which may interact differently with each injection method. Our multi-document analysis relies on a single retrieve-then-compose pipeline: mean-merging for the parametric adapters, and prompt concatenation for the representation-based methods. The composition findings should therefore be read as the behavior of these standard merging operators, which more sophisticated reranking, filtering (Asai et al., 2024), routing, retrieval, or merging schemes may improve. All methods are trained on the same self-study synthetic data, so their relative performance is tied to the quality of that data and of the generators that produce it. Ethical Considerations Our work is a controlled comparison rather than a deployed system, but the knowledge injection methods we study carry the usual risks of language-model applications. Because the injected knowledge is stored in the weights or KV caches, it is harder to inspect or update than an explicitly retrieved passage, and an adapted model can still hallucinate. All datasets we use are publicly available and, to the best of our knowledge, free of personal data, and we use each of them in accordance with its license (see Appendix I). Our training data is generated by an LLM (self-study) and is not additionally filtered. Finally, the catastrophic forgetting we measure is itself a safety-relevant failure mode, as injecting new knowledge can silently erode the general capabilities of the model. References Adams et al. (2024) L. Adams, F. Busch, D. Truhn, and K. K. Bressem LongHealth: a question answering benchmark with long clinical documents. arXiv preprint arXiv:2401.14490. Cited by: 1st item, 1st item. Agarwal et al. (2025) S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, et al. Gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Cited by: Appendix G, §4. Asai et al. (2024) A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi Self-rag: learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, Vol. 2024, p. 9112–9141. Cited by: Limitations. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, p. 1877–1901. External Links: Link Cited by: §1. Caccia et al. (2025) L. Caccia, A. Ansell, E. Ponti, I. Vulić, and A. Sordoni Training plug-n-play knowledge modules with deep context distillation. arXiv preprint arXiv:2503.08727. Cited by: §1, §2. Castelli et al. (2020) V. Castelli, R. Chakravarti, S. Dana, A. Ferritto, R. Florian, M. Franz, D. Garg, D. Khandelwal, S. McCarley, M. McCawley, M. Nasr, L. Pan, C. Pendus, J. Pitrelli, S. Pujar, S. Roukos, A. Sakrajda, A. Sil, R. Uceda-Sosa, T. Ward, and R. Zhang The TechQA dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, p. 1269–1278. External Links: Link, Document Cited by: 5th item, 5th item. Chen et al. (2026) C. Chen, G. L. Zhang, X. Yin, C. Zhuo, B. Li, and U. Schlichtmann KV packet: recomputation-free context-independent kv caching for llms. arXiv preprint arXiv:2604.13226. Cited by: §7. Chen et al. (2023) L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy Punica: multi-tenant lora serving. arxiv. arXiv preprint arXiv:2310.18547. Cited by: §2. Chen et al. (2021a) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: 2nd item, §4. Chen et al. (2021b) Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T. Huang, B. Routledge, and W. Y. Wang FinQA: a dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, p. 3697–3711. External Links: Link, Document Cited by: 4th item, 4th item. Chevalier et al. (2023) A. Chevalier, A. Wettig, A. Ajith, and D. Chen Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 3829–3846. Cited by: §2. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: 1st item, §4. Dasigi et al. (2021) P. Dasigi, K. Lo, I. Beltagy, A. Cohan, N. A. Smith, and M. Gardner A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, p. 4599–4610. Cited by: 2nd item, 2nd item. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948. External Links: Link Cited by: 5th item. Du et al. (2026) C. Du, J. Chen, H. Tang, K. Liu, T. Lan, L. Qu, C. Niu, S. Liu, G. Chen, and F. Wu C2KV: compressed and composable kv cache reuse for efficient llm inference. arXiv preprint arXiv:2607.17715. Cited by: §7. Eyuboglu et al. (2025) S. Eyuboglu, R. S. Ehrlich, S. Arora, N. Guha, D. Zinsley, E. R. Liu, A. Rudra, J. Y. Zou, A. Mirhoseini, and C. Re Cartridges: lightweight and general-purpose long context representations via self-study. In ES-FoMo I: 3rd Workshop on Efficient Systems for Foundation Models, External Links: Link Cited by: Appendix D, Appendix E, Appendix G, §1, §1, §2, §2, §2, 3rd item, §4, §4, §7. Gao et al. (2024) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou The language model evaluation harness. External Links: Document, Link Cited by: Appendix E. Gao et al. (2023) Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, H. Wang, et al. Retrieval-augmented generation for large language models: a survey. arXiv preprint arXiv:2312.10997 2 (1), p. 32. Cited by: §2. Guu et al. (2020) K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang Retrieval augmented language model pre-training. In International conference on machine learning, p. 3929–3938. Cited by: §2. Hardalov et al. (2026) M. Hardalov, G. Iglesias, and A. de Gispert Cartridges at scale: training modular kv caches over large document collections. arXiv preprint arXiv:2606.04557. Cited by: Appendix A, Appendix G, Appendix H, §2, 3rd item, §4, §5.2, §7, footnote 1. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), Cited by: 4th item, §4. Houlsby et al. (2019) N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly Parameter-efficient transfer learning for nlp. In International conference on machine learning, p. 2790–2799. Cited by: §1, §2, 6th item. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: low-rank adaptation of large language models.. Iclr 1 (2), p. 3. Cited by: §1, §2, 5th item. Kujanpä et al. (2024) K. Kujanpä, H. Valpola, and A. Ilin Knowledge injection via prompt distillation. arXiv e-prints, p. arXiv–2412. Cited by: §2. Lester et al. (2021) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, p. 3045–3059. External Links: Link, Document Cited by: §2. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, p. 9459–9474. External Links: Link Cited by: §1, §2. Li and Liang (2021) X. L. Li and P. Liang Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, p. 4582–4597. Cited by: §2. Liu et al. (2022) X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang P-tuning: prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 61–68. Cited by: §2. Mu et al. (2023) J. Mu, X. Li, and N. Goodman Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems 36, p. 19327–19352. Cited by: §2. Ovadia et al. (2024) O. Ovadia, M. Brief, M. Mishaeli, and O. Elisha Fine-tuning or retrieval? comparing knowledge injection in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1, §2. Pang et al. (2022) R. Y. Pang, A. Parrish, N. Joshi, N. Nangia, J. Phang, A. Chen, V. Padmakumar, J. Ma, J. Thompson, H. He, and S. R. Bowman QuALITY: question answering with long input texts, yes!. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics, p. 5336–5358. Cited by: 3rd item, 3rd item. Prabhakar et al. (2025) A. Prabhakar, Y. Li, K. Narasimhan, S. Kakade, E. Malach, and S. Jelassi Lora soups: merging loras for practical skill composition tasks. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, p. 644–655. Cited by: §2. Qin et al. (2024) G. Qin, C. Rosset, E. Chau, N. Rao, and B. Van Durme Dodo: dynamic contextual compression for decoder-only lms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9961–9975. Cited by: §2. Rakotonirina and Baroni (2024) N. C. Rakotonirina and M. Baroni MemoryPrompt: a light wrapper to improve context tracking in pre-trained language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), p. 11187–11195. Cited by: §2. Schulman and Lab (2025) J. Schulman and T. M. Lab LoRA without regret. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/lora/ External Links: Document Cited by: §2. Strich et al. (2026) J. Strich, E. K. Isgorur, M. Trescher, C. Biemann, and M. Semmann T2-RAGBench: text-and-table benchmark for evaluating retrieval-augmented generation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 165–191. External Links: Link Cited by: Appendix E, 4th item, 4th item. Su et al. (2025) W. Su, Y. Tang, Q. Ai, J. Yan, C. Wang, H. Wang, Z. Ye, Y. Zhou, and Y. Liu Parametric retrieval augmented generation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 1240–1250. Cited by: §1, §1, §2, §2, §7. Tang et al. (2025) Y. Tang, W. Su, Q. Ai, and Y. Liu Understanding parametric knowledge injection in retrieval-augmented generation. arXiv preprint arXiv:2510.12668. Cited by: §2. Team (2025) G. Team Gemma 3 technical report. arXiv preprint arXiv:2503.19786. External Links: Link Cited by: Appendix C, Appendix I, §4. Xiao et al. (2023) C. Xiao, Z. Zhang, X. Han, C. Chan, Y. Lin, Z. Liu, X. Li, Z. Li, Z. Cao, and M. Sun Plug-and-play document modules for pre-trained models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15713–15729. Cited by: §2. Yadav et al. (2023) P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal TIES-merging: resolving interference when merging models. In Advances in Neural Information Processing Systems, Cited by: Appendix B, §2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. External Links: Link Cited by: Appendix I, §4, §4. Yang et al. (2026) J. Yang, B. Hou, W. Wei, Y. Bao, and S. Chang Kvlink: accelerating large language models via efficient kv cache reuse. Advances in Neural Information Processing Systems 38, p. 133797–133824. Cited by: §7. Yu et al. (2024) L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li Language models are super mario: absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning, Cited by: Appendix B. Zhong et al. (2024) M. Zhong, Y. Shen, S. Wang, Y. Lu, Y. Jiao, S. Ouyang, D. Yu, J. Han, and W. Chen Multi-lora composition for image generation. arXiv preprint arXiv:2402.16843. Cited by: §2. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Cited by: 3rd item, §4. Zweiger et al. (2026) A. Zweiger, X. Fu, H. Guo, and Y. Kim Fast KV compaction via attention matching. arXiv preprint arXiv:2602.16284. Cited by: Appendix E, §1, 4th item. Łajewska et al. (2025) W. Łajewska, M. Hardalov, L. Aina, N. Anna John, H. Su, and L. Marquez Understanding and improving information preservation in prompt compression for LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, p. 17520–17541. External Links: Document, ISBN 979-8-89176-335-7, Link Cited by: §2. Appendix A Hyperparameters We follow the hyperparameters in (Hardalov et al., 2026) for Cartridges. We train for 20 epochs on QASPER, FinQA, and TechQA, and 5 epochs on LongHealth and QuALITY. The longer-document datasets require more epochs to converge. All parametric methods use the AdamW optimizer with gradient clipping at 1.01.0, and an effective batch size of 6464 (via gradient accumulation). We found that using learning-rate schedule or warmup did not result in any performance boost. LoRA uses rank r=64r=64 and α=128α=128, applied to the feed-forward projections (gate, up, down) of every layer, at learning rate 1×10−41× 10^-4. The MLP adapter has a bottleneck dimension of d=512d=512 inserted as a residual on every layer’s output, also at 1×10−41× 10^-4. The two largest variants (d∈1536,3072d∈\1536,3072\) instead use 2×10−52× 10^-5, as 1×10−41× 10^-4 was training-unstable at that size. We set the learning rate to 2×10−52× 10^-5 for full fine-tuning. Appendix B LoRA Merging Approaches In the multi-document setting we compose per-document LoRA adapters by merging the top-k retrieved adapters per question. The main results use a linear average (mean). Here we compare it against three alternatives from the model-merging literature: cat, which concatenates the low-rank factors; TIES (Yadav et al., 2023), which trims each adapter to its highest-magnitude entries (density 0.50.5), resolves sign conflicts, and averages the agreeing weights; and DARE (Yu et al., 2024), which randomly drops and rescales weights before a linear merge. We compare the merging methods with top-55 retrieved documents in Table 3. The four merge strategies are within roughly one point of one another on every dataset. The failure is therefore in composing independently trained adapters at all, not in the choice of merge operator. Appendix C Gemma-3-12B Results To check that our findings are not specific to Qwen3-8B, we replicate the single-document setting on Gemma-3-12B Team (2025) (Table 4). Similar to Qwen3-8B, No context is far below all knowledge injection methods, and Compaction at 2×2× nearly matches the in-context oracle (65.265.2 vs. 68.968.9) while the parametric LoRA trails behind (50.850.8). Unlike on Qwen3-8B, the trained Cartridges underperform here (37.337.3 on average): on Gemma-3-12B they stay below the training-free Compaction on every dataset, which we attribute to their catastrophic forgetting as presented in Table 5. Merge LongH. QuAL. FinQA TechQA Mean 51.2 ± 1.9 56.0 ± 0.7 5.1 ± 0.1 21.7 ± 0.4 Cat 51.0 ± 2.2 56.0 ± 0.7 5.0 ± 0.1 22.1 ± 1.0 TIES 51.2 ± 2.3 56.0 ± 0.6 5.1 ± 0.1 21.9 ± 0.9 DARE 50.9 ± 2.0 56.0 ± 0.7 5.0 ± 0.1 22.5 ± 0.8 Table 3: LoRA merging approaches (Qwen3-8B, rank 6464). We merge the adapters of the five retrieved documents per question. Scores are the mean ± standard deviation over 3 seeds. Method LongHealth QuALITY QASPER FinQA TechQA Avg. (Acc.) (Acc.) (F1) (EM) (Judge) No context 40.6 ± 1.4 28.0 ± 0.7 19.3 ± 0.3 5.2 ± 0.0 14.8 ± 1.2 21.6 ICL (oracle) 78.5 ± 0.4 77.4 ± 0.2 51.3 ± 0.3 61.9 ± 0.0 75.4 ± 1.9 68.9 Cartridge 49.8 ± 2.1 43.0 ± 1.4 23.4 ± 1.2 9.5 ± 1.7 60.6 ± 0.3 37.3 Compaction 81.4 ± 0.3 74.4 ± 0.5 48.0 ± 0.2 61.2 ± 0.9 61.0 ± 2.6 65.2 LoRA 56.5 ± 1.2 62.4 ± 0.5 34.8 ± 1.0 34.7 ± 0.2 65.6 ± 1.0 50.8 Table 4: Single-document results on Gemma-3-12B. Same protocol as Table 2 (Compaction 2×2×, LoRA rank 6464. Scores averaged over 3 runs ± standard deviation where available). Method GSM8K HumanEval IFEval MMLU Base model 88.7 83.5 78.4 72.6 Cartridge 36.1 ± 23.9 54.4 ± 14.6 39.2 ± 15.7 56.9 ± 8.2 Table 5: Forgetting on control benchmarks (Gemma-3-12B). Base-model accuracy followed by the 2×2× trained Cartridges. Appendix D Distillation vs. Next-token Prediction Our training objective distills the teacher’s next-token distribution rather than training on the teacher’s sampled tokens with the standard next-token-prediction cross-entropy. Table 6 compares the two objectives for LoRA in the single-document setting, holding everything else fixed. Distillation outperforms next-token-prediction on every dataset and is close to 44 points higher on average, with the largest gains on the reasoning-heavy datasets. This corroborates the objective choice of Eyuboglu et al. (2025) and motivates its use throughout our experiments. Dataset Distillation Next-token prediction LongHealth 75.2 ± 0.9 72.6 ± 1.6 QuALITY 73.7 ± 1.2 70.7 ± 0.7 QASPER 50.3 ± 0.6 46.9 ± 1.0 FinQA 49.0 ± 0.9 41.3 ± 0.6 TechQA 74.8 ± 3.3 73.2 ± 1.2 Avg. 64.6 60.9 Table 6: Distillation vs. next-token prediction for per-document LoRA (Qwen3-8B, rank 6464), single-document setting. Scores are the mean ± standard deviation over 3 seeds. Appendix E Evaluation Prompts We use task-specific system prompts during evaluation. All prompts instruct the model to wrap its final answer in <answer> tags. In the Cartridge setting, the document context is encoded in the trained KV cache prefix and is not included in the text prompt, so the model receives only the system prompt and the user question. In the Oracle (Text) baseline, the full document text is prepended to the user message. Table 7 lists the exact prompts used. For FinQA, we use the evaluation prompt recommended in the T2-RAGBench paper Strich et al. (2026).33 3 https://github.com/uhh-hcds/g4kmu-paper/tree/main/src/g4k/prompts/sys_prompts For QASPER44 4 https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/qasper and QuALITY, we adopt the prompts from previous work Gao et al. (2024); Eyuboglu et al. (2025); Zweiger et al. (2026). For LongHealth, we use the prompt format from the original benchmark55 5 https://github.com/HazyResearch/cartridges/blob/main/examples/benchmarks/longhealth/baseline_longhealth.py. For TechQA, since no standard evaluation prompt exists, we conducted a prompt search over several variants, including a minimal instruction (“Answer concisely”), a domain-specific expert framing, a chain-of-thought variant, and a structured extraction format, and selected the prompt that yielded the highest Oracle (Text) accuracy on the development set. Benchmark System Prompt + User Message Format T2-RAGBench / FinQA System: You are an expert in answering financial questions by constructing mathematical formulas based on a simple syntax. - Task: Provide a FORMULA ANSWER to the question based on the given context. Guidelines: 1. Answer Type: A formula is either a number or one of: add(f1, f2), subtract(f1, f2), multiply(f1, f2), divide(f1, f2), exp(f1, f2), greater(f1, f2) 2. Reasoning: Carefully analyze the context. Pay special attention to the table. 3. Final Answer: "<answer> FORMULA </answer>" User: context Question: question LongHealth System: Please reference the patient medical records to answer the user’s questions. Choose the single best option and provide your answer exactly as it appears in the options. Wrap your answer in: <answer> The correct option text here </answer> User: question A. option_a B. option_b C. option_c D. option_d E. option_e QASPER System: You are a research assistant answering questions about a scientific paper. Answer as briefly as possible. Give only the answer, no explanation. If the question cannot be answered from the paper, say "Unanswerable". For yes/no questions, answer "Yes" or "No". Wrap your answer in: <answer> ... </answer> User: question QuALITY System: You are a careful reader answering multiple-choice questions about a long article. Read the article and choose the single best answer option. Provide your answer as the letter (A, B, C, or D) wrapped in answer tags. Example: <answer> B </answer> User: question A. option_a B. option_b C. option_c D. option_d Answer with the letter of the correct option (A, B, C, or D). TechQA System: You are an expert technical support assistant specializing in IT infrastructure, software products, and enterprise systems. Answer the technical question to the best of your ability. Be concise and factual. Wrap your answer in: <answer> YOUR_ANSWER </answer> User: question Table 7: Evaluation prompts used for each benchmark. Variables in braces are replaced with benchmark data at evaluation time. Appendix F LLM-as-a-Judge evaluation A separate judge model (DeepSeek-Distilled-Qwen-32B) evaluates whether the TechQA generated answer is factually equivalent to the reference answer. The judge outputs a JSON object with a justification and a binary grade. A score of 1 is assigned when the judge returns "grade": "correct", and 0 otherwise. Table 8 shows the exact judge prompt. Role Prompt System You are to act as an impartial judge, evaluating whether an answer to a question matches a provided reference answer. Your task is to determine if the given answer is correct based on its factual equivalence to the reference answer, ignoring differences in punctuation and phrasing. When evaluating the answer, consider the following criteria: 1. Factual equivalence: Does the answer convey the same information as the reference answer? 2. Completeness: Does the answer address all parts of the question that the reference answer addresses? 3. Accuracy: Is the information in the answer consistent with the reference answer? 4. Additional information: If the answer contains more information than the reference answer, does it remain consistent and not contradict itself? Your response should be structured as follows: 1. A justification for your decision, explaining your reasoning based on the evaluation criteria. 2. A grade of either "correct" or "incorrect". Important note on deflections and invalid questions: - If the answer is a deflection or does not attempt to answer the question, grade it as "incorrect" unless the reference answer is also a deflection. - If both the answer and the reference answer indicate that the question is invalid or cannot be answered, grade it as "correct". - If the answer is a placeholder like YOUR_ANSWER, any generic phrase that does NOT address the question, or an empty answer, then count it as "incorrect". Your response should be in json format as follows: "justification": "...", "grade": "correct" or "incorrect" User Here is the question: query Here is the answer to be judged: answer Here is the reference answer: expected_answer Table 8: LLM-as-a-Judge prompt used for scoring free-form answers on TechQA. The judge model (DeepSeek-Distilled-Qwen-32B) receives the system prompt defining evaluation criteria and a user message containing the question, generated answer, and reference answer. Appendix G Self-Study Data Synthesis Details We generate the self-study data following Eyuboglu et al. (2025); Hardalov et al. (2026). Question generation model. We use GPT-OSS 120B Agarwal et al. (2025) as the question generator (MQM_Q). This model receives the full document context in its system prompt and generates batches of 20 diverse questions per call. The answer generator (MAM_A) is the target model itself (Qwen3-8B), ensuring that the distillation signal reflects the student’s own output distribution. Seed prompt types. Each synthesis call randomly selects a seed prompt type from five categories: structuring (requests to organize information into JSON, YAML, or other formats), summarization (requests to summarize specific sections), question (factual recall and reasoning questions), use_case (practical downstream application tasks), and creative (open-ended discussion prompts). This diversity ensures the cartridge is trained on varied interaction patterns rather than only factoid QA. Multi-question prompt format. In batched mode, the question generator receives the following instruction (shown for the question type): Generate n diverse questions that test knowledge of the information in the corpus above. Each question should cover a different fact, detail, or aspect of the corpus. Vary the style: mix factual recall, comparison, reasoning, and detail-oriented questions. Include specific details (ids, names, titles, dates, numerical values, etc.) in each question so it is clear what you are asking about. Output ONLY a JSON array of strings, e.g., ["question 1", "question 2", ...]. No other text, no markdown fences, no explanation. The structured JSON output format enables reliable parsing; responses that fail to parse are discarded (typically <5%<5\% of calls). Proportional-to-length sampling. We replace uniform random chunk sampling with a proportional-to-length strategy ensuring balanced document coverage. Each document did_i in the collection is assigned a sampling weight wi=|di|minj|dj|w_i= |d_i| _j|d_j|, so that longer documents, which are assumed to contain more facts, are sampled proportionally more often during synthesis. Sampling rounds and temperatures. For each dataset, we run 4 independent sampling rounds, one per seed prompt type (question, structuring, summarization, use_case), each generating 10,000 samples for a total of 40,000 training examples per dataset.66 6 We exclude the creative seed type from training data as it produces open-ended prompts less suited for distillation. MQM_Q (GPT-OSS 120B, the question generator) uses temperature 0.60.6 with top-p=0.95p=0.95, top-k=20k=20, and a maximum of 4,096 completion tokens per call. MAM_A (Qwen3-8B, the teacher/answer generator) uses temperature 0.00.0 (greedy decoding) with a maximum of 2,048 completion tokens, ensuring deterministic distillation targets. Both models operate with thinking mode enabled. Sampling configuration. Within each round, we generate 10,000 synthesis samples with a batch size of 4 contexts and up to 128 parallel API calls. Documents are sampled with a fixed random seed for reproducibility. For multi-note documents (e.g., LongHealth patient records), we sample one note per prompt to ensure fine-grained coverage of individual clinical notes. Appendix H RAG Baseline: Indexing and Retrieval Details We describe the full RAG pipeline below. Document indexing. Each dataset’s documents are indexed with chunk size C=512C=512 tokens. Documents are chunked using a fixed-size strategy with a 10% token overlap between adjacent chunks. Chunks are embedded using Amazon Titan Embed Text v2 (amazon.titan-embed-text-v2:0) via Amazon Bedrock77 7 https://aws.amazon.com/bedrock with 1024-dimensional embeddings and retrieved via cosine similarity. Retrieval. At evaluation time, each question is issued as a retrieval query against the Knowledge Base. We retrieve the top-K chunks for K∈1,3,5,10K∈\1,3,5,10\. To avoid redundant API calls, we retrieve K=10K=10 once per chunk size and slice for smaller K values. Retrieved chunks are concatenated in score order and prepended to the question as the context for the reader model. Table 9 reports how many unique source documents the top-k retrieved chunks cover. Dataset k=1k=1 k=3k=3 k=5k=5 k=10k=10 LongHealth 1.0 1.8 2.5 4.4 QuALITY 1.0 1.4 2.2 4.8 FinQA 1.0 3.0 5.0 8.7 TechQA 1.0 3.0 4.9 8.1 Table 9: Average number of unique source documents covered by the top-k retrieved chunks (chunk size 1024). Datasets with shorter documents (FinQA, TechQA) have higher chunk diversity, while longer-document datasets (LongHealth, QuALITY) exhibit within-document clustering. Effective token counts. Table 10 reports the average number of context tokens consumed per query for Text RAG at chunk size 1024 across different k values. Due to the 10% overlap between adjacent chunks and shorter final chunks in some documents, the effective token count is slightly below the nominal k×1024k× 1024. Dataset k=1k=1 k=3k=3 k=5k=5 k=10k=10 LongHealth 986 2,960 4,944 9,860 QuALITY 941 2,856 4,733 9,412 FinQA ∼ 960 ∼ 2,880 ∼ 4,800 ∼ 9,600 TechQA ∼ 960 ∼ 2,880 ∼ 4,800 ∼ 9,600 Table 10: Average context tokens per query for Text RAG (chunk size 1024). Retrieval Quality: Recall and MRR. We use the same document-level retrieval as Hardalov et al. (2026). Across all four datasets (LongHealth, QuALITY, FinQA, and TechQA), the dense retriever achieves consistently high retrieval quality, with Recall@10 reaching approximately 95% or higher across chunk sizes, indicating that the relevant documents are almost always retrieved. This suggests that remaining performance differences are primarily due to information loss from chunking rather than retrieval failures. For the complete retrieval methodology, detailed Recall@K and MRR@K results, and further analysis, can be found in Appendix G Hardalov et al. (2026). Appendix I Artifact Licenses and Terms of Use Licenses and terms for use/distribution of artifacts. The five knowledge-intensive benchmarks used in this work are publicly available under the following licenses: • LongHealth Adams et al. (2024): Released under the Apache-2.0 License.88 8 https://github.com/kbressem/LongHealth/blob/main/LICENSE The dataset consists of entirely fictional patient records created by the authors; no real patient data is included. • QASPER Dasigi et al. (2021): Released under the C BY 4.0 License.99 9 https://huggingface.co/datasets/allenai/qasper The dataset consists of questions and answers over NLP research papers; the dataset itself is distributed under C BY 4.0. • QuALITY Pang et al. (2022): Released under the C BY 4.0 License.1010 10 https://nyu-mll.github.io/quality/ Source texts are drawn from Project Gutenberg (public domain) and other permissively licensed collections, including nonfiction and fiction sources such as Slate articles from the Open American National Corpus, The Long+Short, Freesouls, and Open Access books. These texts are published works and do not contain personal data, though some may include mature themes typical of literary and journalistic content. • FinQA Chen et al. (2021b): Released under the MIT License.1111 11 https://github.com/czyssrs/FinQA/blob/main/LICENSE The underlying financial reports are sourced from corporate filings made available via the SEC EDGAR system, including earnings reports and annual/quarterly reports (e.g., 10-K and 10-Q documents). These are publicly accessible financial disclosures prepared by reporting companies. T2-RAGBench Strich et al. (2026) is also released under the MIT License. • TechQA Castelli et al. (2020): The NVIDIA TechQA-RAG-Eval variant used in this work is released under the Apache-2.0 License and is explicitly cleared for commercial and non-commercial use.1212 12 https://huggingface.co/datasets/nvidia/TechQA-RAG-Eval It is derived from the original IBM TechQA dataset, whose code repository is also Apache-2.0.1313 13 https://github.com/IBM/techqa/blob/master/LICENSE.md We additionally use four control benchmarks to measure catastrophic forgetting, all publicly available: • GSM8K Cobbe et al. (2021): grade-school math word problems, released under the MIT License. • HumanEval Chen et al. (2021a): Python code-generation problems, released under the MIT License. • IFEval Zhou et al. (2023): instruction-following prompts, released under the Apache-2.0 License. • MMLU Hendrycks et al. (2021): multiple-choice questions across 57 subjects, released under the MIT License. The models used in our experiments, Qwen3-8B Yang et al. (2025) and Gemma-3-12B Team (2025), are released under the Apache 2.0 License and the Gemma Terms of Use respectively, both of which permit research use. We do not release new datasets in this work. The trained cartridge artifacts (KV cache parameters) are derivatives of the above datasets and the model weights; any release of such artifacts would be subject to the intersection of the applicable licenses above. Offensive content and personal data. • LongHealth: The dataset consists of entirely fictional patient records with no real individuals. No anonymization was required. We verified that no real names, addresses, or identifying information appear in the data. • QASPER: Source texts are NLP research papers. No personal data or offensive content is expected; the domain is scientific writing. • QuALITY: Source texts are fiction and non-fiction narratives, these are published literary works; no personal data is present. • FinQA / T2-RAGBench: Source texts are corporate earnings reports from SEC EDGAR. These are formal financial documents; no personal data or offensive content is present. • TechQA: Source texts are IBM Technote IT support documents. These are technical documentation; no personal data or offensive content is expected. • Control benchmarks (GSM8K, HumanEval, IFEval, MMLU): These are widely used, publicly released evaluation suites of math word problems, programming problems, instruction-following prompts, and academic multiple-choice questions. They contain no personal data, and no offensive content is expected. No additional anonymization steps were taken beyond those applied by the original dataset creators, as none of the datasets contain personal data about private individuals. Appendix J AI use disclosure We used AI to assist with code writing and manuscript typesetting.