Paper deep dive
KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving
Minsoo Cheong, Woosang Lim, Vincent-Daniel Yun, Sungjoo Yoo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/22/2026, 3:04:39 AM
Summary
The paper introduces KV-Rescue, a training-free inference framework that mitigates accuracy loss and runaway degeneration caused by KV-cache eviction in large language models. By interleaving reasoning steps from an evicted base model (e.g., Qwen2.5-Math 7B/72B) and a lightweight full-context helper (e.g., Qwen2.5-Math 1.5B), guided by a process reward model, KV-Rescue recovers 87% of the accuracy lost to eviction at budget B=64. An online detector using entropy and compressibility terminates incoherent or repetitive base-model candidates early, reducing token generation by 43%.
Entities (14)
Relation Signals (15)
KV-Rescue → achievesaccuracyrecovery → 8.7%
confidence 95% · KV-Rescue recovers an average of 87% of the accuracy lost to eviction at eviction budget B=64
KV-Rescue → evaluatedon → GSM8K
confidence 95% · Across five math benchmarks... GSM8K
KV-Rescue → evaluatedon → OlympiadBench
confidence 95% · Across five math benchmarks... OlympiadBench
KV-Rescue → evaluatedon → MATH500
confidence 95% · Across five math benchmarks... MATH500
KV-Rescue → evaluatedon → AIME24
confidence 95% · Across five math benchmarks... AIME 2024
KV-Rescue → evaluatedon → AMC23
confidence 95% · Across five math benchmarks... AMC 2023
KV-Rescue → reducestokengeneration → 43%
confidence 95% · preventing runaway degeneration cuts base-model token generation by 43% on average
KV-Rescue → uses → Process Reward Model
confidence 95% · a process reward model selects the step that extends the shared trajectory
KV-Rescue → uses → KV Cache Eviction
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:KV-cache eviction caps the memory cost of long reasoning traces but is inherently lossy because the model decodes from a partial view of its history. Under aggressive budgets, this not only lowers accuracy but can also cause runaway degeneration, where the model produces incoherent or repetitive tokens until reaching the length limit. We characterize much of this loss as an information gapf caused by missing context, rather than a capability gap caused by limited model capacity. An evicted 7B model and a full-context 1.5B model make complementary errors, and an oracle choice between their answers recovers 79% of the accuracy gap to the full-KV 7B model. Based on this observation, we propose KV-Rescue, a training-free inference framework that bridges the information gap introduced by KV eviction using a lightweight full-context helper. KV-Rescue interleaves reasoning steps from the two models into a shared trajectory. An online detector uses entropy and compressibility to terminate the generation of incoherent or repetitive base-model candidates early. Across five math benchmarks with Qwen2.5-Math 7B and 72B, KV-Rescue recovers an average of 87% of the accuracy lost to eviction at eviction budget B=64. A decode-cost analysis further shows that preventing runaway degeneration cuts base-model token generation by 43% on average.
Tags
Links
- Source: https://arxiv.org/abs/2608.15797v1
- Canonical: https://arxiv.org/abs/2608.15797v1
Trouble viewing inline? Open PDF directly →
Full Text
53,940 characters extracted from source content.
Expand or collapse full text
KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving Minsoo Cheong Woosang Lim Vincent-Daniel Yun Sungjoo Yoo Abstract KV-cache eviction caps the memory cost of long reasoning traces but is inherently lossy because the model decodes from a partial view of its history. Under aggressive budgets, this not only lowers accuracy but can also cause runaway degeneration, where the model produces incoherent or repetitive tokens until reaching the length limit. We characterize much of this loss as an information gap caused by missing context, rather than a capability gap caused by limited model capacity. An evicted 7B model and a full-context 1.5B model make complementary errors, and an oracle choice between their answers recovers 79% of the accuracy gap to the full-KV 7B model. Based on this observation, we propose KV-Rescue, a training-free inference framework that bridges the information gap introduced by KV eviction using a lightweight full-context helper. KV-Rescue interleaves reasoning steps from the two models into a shared trajectory. An online detector uses entropy and compressibility to terminate the generation of incoherent or repetitive base-model candidates early. Across five math benchmarks with Qwen2.5-Math 7B and 72B, KV-Rescue recovers an average of 87% of the accuracy lost to eviction at eviction budget B=64B=64. A decode-cost analysis further shows that preventing runaway degeneration cuts base-model token generation by 43% on average. Introduction The growing use of long-context reasoning and agentic systems makes the KV cache a major bottleneck in LLM inference. KV-cache compression methods address this bottleneck, including quantization (7; 13; 17) and merging (20); among them, KV eviction (25; 11) is one of the most actively studied. Recent work tailors eviction to reasoning models by exploiting their structure (1) and reports that aggressive eviction can induce degeneration: the model falls into repetition loops or, in severe cases, emits incoherent tokens until reaching the length limit. Eviction is still inherently lossy, since the model sees only a selected part of the context rather than its full history, and the problem grows more acute at low budgets. Figure 1: (a) An evicted 7B and a full-KV 1.5B make complementary errors; an oracle pick between them recovers 79%79\% of the eviction loss. (b) Best-of-N plateaus below full-KV, while KV-Rescue (ours) recovers most of the gap and wins at every N. (MATH500, Qwen2.5-Math-7B, budget 128.) Eviction loss is an information gap, not a capability gap. The accuracy lost to eviction is an information gap: the model cannot see parts of its past context. A small model underperforming a large one instead reflects a capability gap: it sees the same information but has less capacity to solve the problem. These gaps arise from different limitations, so a small full-context model and an evicted large model can make complementary errors. Figure 1(a) supports this view: an oracle choice between an evicted 7B model and a full-context 1.5B model recovers 79%79\% of the accuracy gap to the full-KV 7B model. KV Eviction and Test-Time Scaling. It remains unclear whether test-time scaling alone can recover the loss caused by eviction. We first apply stepwise best-of-N to the evicted model. As Figure 1(b) shows, increasing N improves accuracy but eventually plateaus below full-KV. All candidates are sampled from the same partial cache state, so additional sampling does not restore the missing context. Motivated by these observations, we propose KV-Rescue, a training-free inference framework that bridges the information gap introduced by KV eviction with a lightweight full-context helper. KV-Rescue interleaves reasoning steps from the evicted base model and the helper. At each step, both models generate candidates from their respective cache states, and a process reward model selects the step that extends the shared trajectory. We generalize this procedure to a best-of-N setting that draws several candidates from each model. On MATH500 at budget 128 (Figure 1(b)), KV-Rescue surpasses full-KV pass@1 at N=4N=4 and continues to improve as N increases. Across five math benchmarks with Qwen2.5-Math 7B and 72B, KV-Rescue recovers an average of 87%87\% of the accuracy lost to eviction at eviction budget B=64B=64. It also suppresses runaway degeneration; in our decode-cost analysis, this reduces base-model token generation by 43%43\% on average. Our contributions are as follows: • We identify KV eviction loss as an information gap distinct from a model’s capability gap, and show that an evicted large model and a small full-context model make complementary errors. • We introduce KV-Rescue, a training-free inference framework for context-complementary reasoning between an evicted base model and a lightweight full-context helper, with an online detector that terminates degenerate base candidates during generation. • We evaluate KV-Rescue across five math benchmarks, two base-model sizes, and two eviction methods, showing that its recovery persists across model scales and eviction policies while controlling runaway degeneration. Preliminaries KV Eviction At decoding step t, a Transformer caches the key–value pairs of all preceding tokens, t=(ki,vi)i<tC_t=\(k_i,v_i)\_i<t, where we suppress layer and attention-head indices for simplicity. The cache grows linearly with the sequence length and becomes a major memory cost for long reasoning traces. KV eviction limits the number of retained entries to a budget B by assigning each entry an importance score sis_i, such as accumulated attention, and retaining the highest-scoring entries (1; 11; 25): ℐtB _t^B =TopB(sii<t), =TopB\! (\s_i\_i<t ), (1) tB _t^B =(ki,vi):i∈ℐtB, =\(k_i,v_i):i _t^B\, a~t a_t =Attn(qt,tB), =Attn\! (q_t,C_t^B ), where TopBTopB returns the indices of the B largest scores. Attention is computed over tBC_t^B rather than the full cache tC_t. Once an entry is evicted, subsequent decoding steps cannot access it without recomputation or retrieval. Reducing B thus restricts the historical information available to future predictions. PRM-Guided Stepwise Best-of-N Best-of-N (4) samples N candidates and selects the one assigned the highest score. Stepwise best-of-N (16) applies this selection at each reasoning step rather than over complete trajectories, allowing the scorer to guide generation as it unfolds. A solution to a question q is represented as a sequence of reasoning steps =(s1,…,sT)s=(s_1,…,s_T). At step t, the committed context is ct=(q,s<t)c_t=(q,s_<t). A process reward model (PRM) (18; 12; 19; 22) assigns a score R(ct,s)∈ℝR(c_t,s) to a candidate step s. Given a policy π, stepwise best-of-N samples and selects a step as st(j) s_t^(j) ∼π(⋅∣ct),j=1,…,N, π\! (· c_t ), j=1,…,N, (2) jt⋆ j_t =argmaxj∈1,…,NR(ct,st(j)), = *arg\,max_j∈\1,…,N\R\! (c_t,s_t^(j) ), st s_t =st(jt⋆). =s_t^(j_t ). The context is then updated to ct+1=(q,s≤t)c_t+1=(q,s_≤ t), and decoding continues. Because the candidates share the same committed prefix, they can be scored in a single batched forward pass. Reasoning Model Degeneration Figure 2: Two degeneration modes observed under aggressive KV eviction in Qwen2.5-Math-7B stepwise best-of-N decoding traces. Coherent reasoning (gray) transitions into (a) incoherent generation or (b) a repetition loop, and continues to the length limit without producing an answer. Autoregressive generation can degenerate into off-task or repetitive text instead of completing the response (21). Recent work reports that aggressive KV eviction can make such failures more frequent in reasoning models (1). We observe two recurring modes in our setting (Figure 2). Incoherent generation produces high-entropy sequences of unrelated punctuation, symbols, and mixed-script tokens. Repetition loops repeatedly emit the same token or phrase until reaching the length limit. Both modes reduce answer accuracy and waste decoding computation. We target these two discrete failure modes; more gradual failures, such as unnecessarily prolonged but coherent reasoning (3; 14), are outside the scope of this work. Figure 3: Online degeneration detection (Qwen2.5-Math-7B, R-KV eviction). Both signals are computed from statistics already available during decoding. (a,b) Incoherent generation has high token-level entropy, so coherent and incoherent candidates split into two modes that a flat τent _ent separates without labels. (c,d) Repetition loops are confident (low entropy) but highly redundant: the gzip ratio (compressed size of a sliding text window over its raw size; lower means more repetitive) collapses below τcmp _cmp, leaving a small loop population cleanly below the coherent bulk. Figure 4: KV-Rescue. At each step, candidates are sampled from an evicted base πb _b and a full-context helper πh _h; an online degeneration detector prunes degenerate base steps, the PRM commits the best step sts_t, and base and helper steps interleave. The best-of-N baseline for comparison is in the Appendix Method Stepwise Interleaved Reasoning KV-Rescue extends the answer-level complementarity in Figure 1 to stepwise reasoning. At each step, it selects between candidates from a KV-evicted base model and a lightweight full-context helper using a process reward model (PRM). Setup. Given a question q, a solution is a sequence of steps =(s1,…,sT)s=(s_1,…,s_T) delimited by “ ”. We write ct=(q,s<t)c_t=(q,s_<t) for the committed context and R(ct,s)∈[0,1]R(c_t,s)∈[0,1] for the PRM score of a candidate s. The base model πb _b conditions on an evicted cache state ℰB(ct)E_B(c_t) with budget B, while the helper πh _h conditions on the full context ctc_t. The models thus share the same trajectory but access different views of its history. Stepwise interleaving. At step t, we sample NbN_b base candidates and NhN_h helper candidates, then commit the candidate with the highest PRM score: stb,i s_t^b,i ∼πb(⋅∣ℰB(ct)),i=1,…,Nb, _b\! (· _B(c_t) ), i=1,…,N_b, (3) sth,j s_t^h,j ∼πh(⋅∣ct),j=1,…,Nh, _h\! (· c_t ), j=1,…,N_h, tb _t^b =stb,ii=1Nb,th=sth,jj=1Nh, =\s_t^b,i\_i=1^N_b, _t^h=\s_t^h,j\_j=1^N_h, st s_t =argmaxs∈tb∪thR(ct,s). = *arg\,max_s _t^b _t^hR(c_t,s). The context is then extended to ct+1=(ct,st)c_t+1=(c_t,s_t). If a base candidate is selected, its fork becomes the next persistent base state. If a helper candidate is selected, the base model processes the selected tokens and then applies eviction. At the next step, the helper again receives the full committed context. Both models continue from the same trajectory while preserving their distinct context views. Best-of-(Nb+Nh)(N_b+N_h). The one-candidate case uses Nb=Nh=1N_b=N_h=1. Our experiments use Nb=Nh=N_b=N_h=N, denoted best-of-N+N+N, and compare against eviction-only best-of-N with Nb=N_b=N and Nh=0N_h=0. This matches the base-model candidate count and eviction budget; the helper cost is analyzed in the efficiency analysis below. Online Degeneration Detection Under aggressive eviction, base candidates exhibit two recurring failure modes: incoherent generation and repetition loops. Figure 3 shows that they have distinct online signals, which we use to terminate degenerate base candidates before PRM scoring. Incoherent generation. Incoherent candidates exhibit sustained high predictive entropy. At generated position e, we compute the entropy and its moving average over a recent window WeW_e: He=−∑vpe(v)logpe(v),H¯e=1|We|∑r∈WeHr.H_e=-\! _vp_e(v) p_e(v), H_e= 1|W_e| _r∈ W_eH_r. (4) A candidate is terminated when H¯e>τent H_e> _ent. Repetition loops. Loops are typically low-entropy but highly compressible. Let zez_e denote a fixed-width window over the most recent candidate text. Once the window is available, we compute ρe=|gzip(ze)||ze| _e= |gzip(z_e)||z_e| (5) and terminate the candidate when ρe<τcmp _e< _cmp. Figure 3(d) reports the per-candidate minimum ρ⋆(s)=mineρeρ (s)= _e _e for analysis, while online detection uses the current ρe _e. The detector fires at the first eligible position satisfying either test: dege(s)=[H¯e>τent]∨[ρe<τcmp].deg_e(s)=1[ H_e> _ent] 1[ _e< _cmp]. (6) Only base candidates are screened. With deg(s)=⋁edege(s)deg(s)= _edeg_e(s), selection becomes ^t C_t =s∈tb:¬deg(s)∪th, =\s _t^b: (s)\ _t^h, (7) st s_t =argmaxs∈^tR(ct,s). = *arg\,max_s∈ C_tR(c_t,s). Helper candidates keep the pool nonempty even if all base candidates are aborted. Both signals use quantities already available during decoding and require no additional model forward pass. KV-Rescue KV-Rescue combines stepwise interleaving with online degeneration detection (Figure 4), committing the highest-scoring candidate among the surviving base candidates and the full-context helper candidates. Table 1: Decode cost of best-of-N (BoN) vs. KV-Rescue (KVR) on Qwen2.5-Math-7B (N=8N=8): per-problem mean steps, output length, and generated 7B-base / 1.5B-small tokens. Cells are heat-shaded per dataset and metric (green = low, red = high); as the KV budget shrinks the evicted-only BoN runs away (red) while KVR stays controlled (green). full = full-KV; Steps Out. len Base 7B tok 1.5B tok Dataset Bud. BoN KVR BoN KVR BoN KVR KVR MATH500 full 27.0 – 673 – 6,106 – – 512 31.8 15.4 670 687 6,189 5,841 6,102 256 46.7 14.8 701 653 6,704 5,855 5,931 128 63.8 15.6 754 666 7,607 6,068 6,142 64 114 21.1 960 671 10,587 6,327 6,560 AIME24 full 171 – 1,566 – 16,803 – – 512 148 43.8 1,510 1,369 16,457 12,775 13,591 256 222 41.9 1,663 1,295 18,943 12,842 13,620 128 275 51.1 1,768 1,362 20,660 13,159 14,307 64 229 64.9 1,600 1,388 20,573 13,584 15,812 AMC23 full 40.4 – 966 – 9,377 – – 512 35.2 25.0 959 1,011 9,695 9,574 9,476 256 73.6 23.4 1,000 988 10,440 9,159 10,350 128 121 30.7 1,166 1,015 12,706 9,654 10,451 64 152 32.1 1,295 944 14,947 9,209 9,978 GSM8K full 8.3 – 307 – 2,477 – – 512 8.8 6.6 307 303 2,500 2,448 2,317 256 9.9 6.8 314 306 2,577 2,484 2,380 128 28.9 7.0 376 307 3,422 2,537 2,446 64 96.2 8.5 619 309 7,056 2,607 2,590 OlympiadBench full 57.5 – 1,016 – 9,499 – – 512 84.9 26.8 1,092 1,049 11,040 9,550 10,189 256 98.8 29.1 1,113 1,026 11,673 9,458 10,242 128 139 36.2 1,233 1,053 13,793 9,819 13,132 64 187 36.4 1,425 1,040 17,327 10,171 10,920 Figure 5: Main results. pass@1 vs. base KV budget across the 4×54× 5 grid (Qwen2.5-Math 7B/72B × N=1,8N=1,8; MATH500, AIME24, AMC23, GSM8K, OlympiadBench). Blue is KV-Rescue (ours), red is eviction-only best-of-N; dashed and dotted lines mark the base full-KV and helper full-KV references. Best-of-N falls off sharply as the budget shrinks, while KV-Rescue stays close to base full-KV across budgets. Experiments Experiment Setup Models. We use the Qwen2.5-Math family (22), which provides reasoning models at multiple scales together with a matching PRM, reducing distribution mismatch in step scoring (8; 27). The base is Qwen2.5-Math-7B-Instruct or Qwen2.5-Math-72B-Instruct (hereafter 7B and 72B), the full-context helper is Qwen2.5-Math-1.5B-Instruct, and the PRM is Qwen2.5-Math-PRM-7B. We retain the 7B PRM for the 72B base to test whether a smaller verifier can guide the larger model. All models run in bfloat16, with context capped at 40964096 tokens. Benchmarks and decoding. We report pass@1 on MATH500 (6), AIME 2024 (23), AMC 2023 (15), GSM8K (4), and OlympiadBench (5), using the official Qwen2.5-Math grader. AIME and AMC are averaged over five seeds, and the remaining benchmarks use one run. We sample at temperature 0.80.8, top-p 0.950.95, and top-k 2020. Main results use N∈1,8N∈\1,8\. For both methods, N denotes the number of candidates drawn from the base model; KV-Rescue additionally draws N candidates from the 1.5B helper. The comparison matches the base-model candidate budget and eviction budget, rather than the total number of policy candidates. The helper requires approximately 24% and 2.5% of the per-token decode compute of the 7B and 72B bases, respectively. KV eviction. We use R-KV (1) with λ=0.1λ=0.1 and a protected recent window of 128 tokens. The reported budget B∈64,128,256,512B∈\64,128,256,512\ controls the score-selected entries, giving a physical cache bound of B′=B+128B =B+128. We also evaluate a no-eviction full-KV reference. Implementation and hardware. We build on the v1 engine of a modified vLLM (9). Efficiency and latency are measured on A100 SXM GPUs with 40GB memory. The appendix provides full implementation details and repeats the 7B evaluation with SnapKV (11), showing that the recovery is not specific to R-KV. Experiment Results Accuracy versus KV Budget. Figure 5 compares accuracy across the 4×54× 5 grid. Eviction-only best-of-N degrades sharply as B decreases and often approaches or falls below the full-context helper at the smallest budget. KV-Rescue remains substantially more stable and closer to the full-KV reference. At B=64B=64, it recovers an average of 87%87\% of the full-KV accuracy loss across the grid, using (AKVR−Aevict)/(Afull−Aevict)(A_KVR-A_evict)/(A_full-A_evict) for each setting. Increasing N generally improves both methods but does not eliminate the gap under aggressive eviction. The same recovery pattern persists with the 72B base and 7B PRM. Generation Length. Table 1 compares reasoning-step count, output length, and base-model token generation. As B decreases, eviction-only best-of-N exhibits severe length inflation as degenerate generations approach the length cap. KV-Rescue prevents these runaways and remains closer to the full-KV reference. At B=64B=64 in the 7B, N=8N=8 analysis, it reduces base-model token generation by 43%43\% on average across benchmarks. Discussion Recovering KV-Eviction-Induced Information Loss Figure 6: Share of committed steps selected from the evicted 7B base and the full-context 1.5B helper on MATH500 with best-of-N+N+N decoding: (a) as N varies at budget 128 and (b) as the KV budget varies at N=8N=8. Figure 7: A representative problem solved by KV-Rescue but not by either model’s independent trajectory: (a) the evicted base, (b) KV-Rescue, and (c) the full-context helper. Blue and orange mark committed base and helper steps, respectively. We examine how the two models contribute to the recovered trajectory. Figure 6 shows that the helper is not merely a rare fallback: it supplies 1717–34%34\% of committed steps across the evaluated settings. Its share increases from 17%17\% at budget 512 to 33%33\% at budget 64, consistent with full-context candidates becoming more useful as the base loses more history. The base nevertheless contributes most steps, and the source ratio changes little as N grows, indicating that KV-Rescue combines the two models rather than replacing the base with the helper. Figure 7 illustrates this complementarity. The evicted base loses track of the requested quantity, while the helper retains the target but makes an error in the derivation. By interleaving steps from the two models, KV-Rescue avoids both failures and reaches the correct answer. Together, the selection trend and the example support the intended mechanism: the helper contributes selectively, and its role grows as eviction becomes more severe. Degeneration Analysis Table 2: Ablation of interleaving and early exit on MATH500 (N=8N=8, Qwen2.5-Math-7B). Rows are cumulative. Degen. is the fraction of incorrect trajectories with at least 256 committed steps, used to identify context-length runaways. Base tok. is the mean number of tokens generated by base-model candidates per problem; Steps is the mean number of committed steps; and E is the fraction of base candidates terminated by early exit. Budget Method Acc. Degen. Base tok. Steps E (%) (%) (%) full full-KV 84.0 2.4 6,106 27 – 64 best-of-N 52.8 13.4 10,587 114 – + interleaving 81.8 2.8 7,898 36 – + early exit 82.2 0.2 6,327 21 8.0 128 best-of-N 69.2 6.0 7,607 64 – + interleaving 81.2 1.4 6,673 25 – + early exit 83.2 0.0 6,068 16 6.4 Table 2 isolates the contributions of interleaving and early exit. At budget 64, eviction-only best-of-N raises the degeneration rate from 2.4%2.4\% under full-KV to 13.4%13.4\%. Sampling alone does not prevent these runaways because every candidate is generated from the same evicted cache. Interleaving substantially reduces trajectory degeneration. Adding full-context helper candidates gives the PRM alternatives that do not share the base model’s missing context. At budget 64, interleaving reduces degeneration from 13.4%13.4\% to 2.8%2.8\%, lowers the mean trajectory from 114 to 36 steps, and raises accuracy from 52.8%52.8\% to 81.8%81.8\%. This accounts for most of the recovery. Early exit removes candidate-level token waste. Interleaving can avoid committing a degenerate base candidate while still decoding that candidate to the step limit. Early exit terminates such branches during generation. At budget 64, it fires on 8.0%8.0\% of base candidates and reduces base-model token generation by a further 20%20\% (7.97.9k→6.3→6.3k) without reducing accuracy. It also lowers degeneration from 2.8%2.8\% to 0.2%0.2\%, while eliminating it under the table’s criterion at budget 128. Figure 8: Per-sequence KV memory (a) and per-token decode compute (b) of KV-Rescue vs. context length, for a 7B and a 72B base (bf16, R-KV budget 512512, B′=640B =640). Bars are KV-Rescue: the evicted base (blue, bounded by the budget) plus the full-KV helper 1.5B (orange). Lines are the full-KV base baseline. Table 3: MATH500 wall-clock efficiency (Qwen2.5-Math-72B, N=8N=8, KV budget 64; optimized engine with batched eviction and dynamic KV reserve). prob/min: steady-state throughput (excludes one-time load); tgent_gen: per-problem generation time (base πb _b and full-context helper πh _h run concurrently); tscoret_score: PRM time; all amortized over batch concurrency. Max length 4096; full-KV OOMs at batch 64. Method Batch prob/min tgent_gen (s) tscoret_score (s) 72B tok full-KV 8 6.7 8.41 0.375 5,392 64 OOM BoN 8 1.2 44.45 1.619 18,144 64 3.2 12.84 0.193 18,371 KV-Rescue 8 4.2 12.67 0.410 5,951 64 10.6 4.58 0.034 5,460 Efficiency Analysis We analyze the cost of KV-Rescue theoretically and report wall-clock measurements below; the full derivation is in the appendix. Figure 8 shows KV memory and decode compute against context length at a fixed generation length, allowing a per-token comparison. Eviction caps the base model’s KV, while the full-KV baseline grows linearly (0.310.31 vs. 1.251.25 GiB at 44k for the 72B base). The helper’s KV grows with context but remains small, and a full-context 1.5B forward adds only ∼2.5% 2.5\% of the 72B base’s per-token compute (∼24% 24\% for the 7B base). The added model-level cost is modest, particularly as the base model scales. This fixed-length comparison isolates the per-token overhead. In practice, KV-Rescue also generates fewer tokens by preventing the runaway degeneration that inflates the evicted baseline’s output length (Table 1). It combines a lightweight helper with substantially fewer expensive base-model tokens. The wall-clock measurements below account for the complete inference pipeline, including helper generation, PRM scoring, and system overheads. Table 3 reports MATH500 wall-clock throughput on a single node of eight A100 40GB GPUs. To keep the total accelerator allocation fixed across methods, the 72B base is tensor-parallel across all eight GPUs, with the PRM and 1.5B helper co-located on the same GPU pool. The reported throughput thus includes resource contention and scheduling overhead from co-location rather than assuming isolated model execution. Even under this shared-resource configuration, at batch 64, KV-Rescue achieves 3.3×3.3× the throughput of eviction-only best-of-N at the same base KV budget, while reducing base-model generation by 70%70\% (18.4k→5.5k18.4k→5.5k tokens per problem). Capping the base KV cache also allows KV-Rescue to serve batch 64, which full-KV cannot fit. PRM scoring remains a small fraction of generation time, and the helper runs concurrently with the base. Since the helper has a small compute and memory footprint, it is also amenable to disaggregated execution on a lower-cost accelerator, offering a path to further isolate helper execution and improve concurrency. These results show that KV-Rescue makes aggressive eviction practical by preventing the degeneration that otherwise inflates its decode cost. Related Work KV cache compression. A large body of work reduces KV-cache memory through quantization (7; 13; 17), merging (20), and eviction (25; 11); recent work further adapts eviction to reasoning models (1). These methods operate within a single model, reducing the precision, redundancy, or number of entries in its cache. Closest to our setting is SmallKV (26), which maintains a full-cache small model and uses its attention to guide large-model eviction and approximate the contribution of marginal tokens. The large model nevertheless remains the sole generator. KV-Rescue instead treats the small full-context model as an independent reasoning policy and interleaves its steps with those of the evicted base. It operates above the eviction mechanism and can be orthogonally combined with different eviction policies. Test-time and multi-model inference. Test-time scaling improves reasoning by sampling and selecting candidates through best-of-N (4) and PRM-guided step or beam search (16; 12). These methods typically sample from a single policy conditioned on the same context state. KV-Rescue instead draws step candidates from two policies with asymmetric context access, using selection to recover information unavailable to the evicted base. Speculative decoding similarly combines small and large models, but uses the small model to draft tokens while preserving the large model’s output distribution (10; 2). In KV-Rescue, either model may contribute a committed reasoning step, and the goal is to recover reasoning quality under KV eviction rather than accelerate an unchanged target policy. Conclusion KV eviction bounds the memory cost of long reasoning but can remove information needed by later steps. We showed that much of the resulting loss is an information gap rather than a capability gap: an evicted large model and a lightweight full-context model make complementary errors. Based on this observation, we introduced KV-Rescue, a training-free inference framework that interleaves reasoning steps from the two models and terminates degenerate base candidates online. Across five math benchmarks with Qwen2.5-Math 7B and 72B, KV-Rescue recovers an average of 87%87\% of the accuracy lost at an eviction budget of 64, while reducing base-model token generation by 43%43\% in our decode-cost analysis. At batch 64, KV-Rescue achieves 3.3×3.3× the throughput of eviction-only best-of-N, while full-KV runs out of memory. These results show that complementary context views can preserve reasoning quality under severe KV memory constraints. References Cai et al. (2025) Z. Cai, W. Xiao, H. Sun, C. Luo, Y. Zhang, K. Wan, Y. Li, Y. Zhou, L. Chang, J. Gu, Z. Dong, A. Anandkumar, A. Asi, and J. Hu R-kv: redundancy-aware kv cache compression for training-free reasoning models acceleration. arXiv preprint arXiv:2505.24133. Cited by: Appendix C, Introduction, KV Eviction, Reasoning Model Degeneration, KV eviction., KV cache compression.. Chen et al. (2023) C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: Test-time and multi-model inference.. Chen et al. (2024) X. Chen, J. Xu, T. Liang, Z. He, J. Pang, D. Yu, L. Song, Q. Liu, M. Zhou, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu Do not think that much for 2+3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187. Cited by: Reasoning Model Degeneration. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: PRM-Guided Stepwise Best-of-N, Benchmarks and decoding., Test-time and multi-model inference.. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828–3850. Cited by: Benchmarks and decoding.. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Krispin, A. Jain, N. Chen, S. Toyer, L. Song, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874. Cited by: Benchmarks and decoding.. Hooper et al. (2024) C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami KVQuant: towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079. Cited by: Introduction, KV cache compression.. Khalifa et al. (2025) M. Khalifa, R. Agarwal, L. Logeswaran, J. Kim, H. Peng, M. Lee, H. Lee, and L. Wang Process reward models that think. arXiv preprint arXiv:2504.16828. Cited by: Models.. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611–626. Cited by: Appendix C, Implementation and hardware.. Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, p. 19274–19286. Cited by: Test-time and multi-model inference.. Li et al. (2024) Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen Snapkv: llm knows what you are looking for before generation. Advances in Neural Information Processing Systems 37, p. 22947–22970. Cited by: Appendix E, Introduction, KV Eviction, Implementation and hardware., KV cache compression.. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations, Vol. 2024, p. 39578–39601. Cited by: PRM-Guided Stepwise Best-of-N, Test-time and multi-model inference.. Liu et al. (2024) Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu KIVI: a tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750. Cited by: Introduction, KV cache compression.. Lotfi et al. (2026) S. Lotfi, P. Kirichenko, S. Li, and Z. Liu Quantized reasoning models think they need to think longer, but they do not. arXiv preprint arXiv:2606.00206. Cited by: Reasoning Model Degeneration. math-ai (2024) math-ai Amc23: a benchmark dataset for american mathematics competitions 2023. Hugging Face. Note: https://huggingface.co/datasets/math-ai/amc23 Cited by: Benchmarks and decoding.. Snell et al. (2024) C. Snell, J. Lee, K. Xu, and A. Kumar Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314. Cited by: PRM-Guided Stepwise Best-of-N, Test-time and multi-model inference.. Son et al. (2026) D. Son, E. Choi, and S. Yoo NSNQuant: a double normalization approach for calibration-free low-bit vector quantization of kv cache. Advances in Neural Information Processing Systems 38, p. 43124–43159. Cited by: Introduction, KV cache compression.. Uesato et al. (2022) J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. Cited by: PRM-Guided Stepwise Best-of-N. Wang et al. (2024a) P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui Math-shepherd: verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9426–9439. Cited by: PRM-Guided Stepwise Best-of-N. Wang et al. (2024b) Z. Wang, B. Jin, Z. Yu, and M. Zhang Model tells you where to merge: adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454. Cited by: Introduction, KV cache compression.. Xu et al. (2022) J. Xu, X. Liu, J. Yan, D. Cai, H. Li, and J. Li Learning to break the loop: analyzing and mitigating repetitions for neural text generation. Advances in Neural Information Processing Systems 35, p. 3082–3095. Cited by: Reasoning Model Degeneration. Yang et al. (2024) A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, and Z. Zhang Qwen2.5-math technical report: toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. Cited by: Appendix C, PRM-Guided Stepwise Best-of-N, Models.. Zhang and Math-AI (2024) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2024. External Links: Link Cited by: Benchmarks and decoding.. Zhang et al. (2025) Z. Zhang, C. Zheng, Y. Wu, B. Zhang, R. Lin, B. Yu, D. Liu, J. Zhou, and J. Lin The lessons of developing process reward models in mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, p. 10495–10516. Cited by: Appendix C. Zhang et al. (2023) Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen H2O: heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2306.14048 Cited by: Introduction, KV Eviction, KV cache compression.. Zhao et al. (2026) Y. Zhao, Y. Peng, N. Cam-Tu, Z. Li, X. Wang, H. Zhao, and X. Fu Smallkv: small model assisted compensation of kv cache compression for efficient llm inference. Advances in Neural Information Processing Systems 38, p. 48637–48665. Cited by: KV cache compression.. Zhu et al. (2025) J. Zhu, C. Zheng, J. Lin, K. Du, Y. Wen, Y. Yu, J. Wang, and W. Zhang Retrieval-augmented process reward model for generalizable mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, p. 8453–8468. Cited by: Models.. Appendix A KV-Rescue vs KV Evicted Best-of-N Figure 9: KV-Rescue. (a) Best-of-N under KV eviction draws every candidate from the same lossy cache, so the information lost to eviction leads the base to produce wrong or degenerate steps. (b) KV-Rescue samples steps from an evicted base πb _b and a full-context helper πh _h; an online degeneration detector prunes degenerate base steps, the PRM commits the best step sts_t, and base and helper steps interleave. Figure 9 contrasts the two settings. Under KV eviction, best-of-N draws every candidate step from the same lossy cache, so all candidates share the base model’s missing context: the PRM can only select among them, and sampling cannot recover the evicted information. As a result the base answers incorrectly or degenerates, emitting incoherent or repetitive steps until the length cap (Fig. 9a). KV-Rescue instead samples candidate steps from two policies at each step, the evicted base πb _b and a full-context helper πh _h. An online degeneration detector prunes degenerate base candidates before selection, the PRM commits the best surviving step sts_t, and both caches re-base onto ct+1c_t+1. Committed base and helper steps interleave over the solution, each covering the other’s failure and recovering the correct trajectory (Fig. 9b). Appendix B KV-Rescue Pseudocode Algorithm 1 summarizes one decoding pass. Each step samples NbN_b candidates from the evicted base πb _b (on the compressed cache ℰB(c)E_B(c)) and NhN_h from the full-context helper πh _h, aborting any that fire the online detector deg(⋅)deg(·) mid-generation. The PRM commits the best non-degenerate step sts_t, which extends the context c; both caches then re-base onto the shared prefix. The loop repeats until an <eos> step or the length cap L. Algorithm 1 KV-Rescue decoding for one question 0: question q; evicted base policy πb _b with budget B; full-context helper policy πh _h; PRM R; candidate counts Nb,NhN_b,N_h with Nh≥1N_h≥ 1; detector thresholds τent,τcmp _ent, _cmp; maximum output-token length L 0: solution s 1: c←qc← q; ←()s←(); t←1t← 1 2: initialize the compressed base cache Kb←ℰB(q)K_b _B(q) and full-context helper cache Kh←qK_h← q 3: while lentok()<Llen_tok(s)<L do 4: generate base candidates stb,ii=1Nb∼πb(⋅∣Kb)\s_t^b,i\_i=1^N_b _b(· K_b), aborting a candidate as soon as it satisfies deg(stb,i,τent,τcmp)deg(s_t^b,i; _ent, _cmp) 5: generate helper candidates sth,jj=1Nh∼πh(⋅∣Kh)\s_t^h,j\_j=1^N_h _h(· K_h) without degeneration screening 6: ^tb←stb,i:¬deg(stb,i,τent,τcmp) C_t^b←\\,s_t^b,i: (s_t^b,i; _ent, _cmp)\,\ 7: t←^tb∪sth,jj=1NhC_t← C_t^b∪\s_t^h,j\_j=1^N_h 8: st←argmaxs∈tR(c,s)s_t← *arg\,max_s _tR(c,s) PRM selects the next step 9: ←|sts \|s_t; c←(c,st)c←(c,s_t) 10: if st∈^tbs_t∈ C_t^b then 11: retain the selected base fork as KbK_b 12: append sts_t to KhK_h 13: else 14: retain the selected helper fork as KhK_h 15: append sts_t to KbK_b and apply ℰBE_B 16: end if 17: if sts_t ends with <eos> then 18: break 19: end if 20: t←t+1t← t+1 21: end while 22: return s Appendix C Implementation Details Models. We instantiate KV-Rescue on the Qwen2.5-Math family (22). The base reasoner is Qwen2.5-Math-7B-Instruct (and Qwen2.5-Math-72B-Instruct for the scaling study); the full-KV helper is Qwen2.5-Math-1.5B-Instruct. Step selection uses the process reward model Qwen2.5-Math-PRM-7B (24), a token-classification reward model: we form prompt⊕ ⊕s1⊕⋯⊕sk⊕ n n s_1 … s_k n n and take the softmax probability of the positive label at the final token as the step score R(⋅)R(·) (the step separator is , matching the PRM’s training-time separator). All models run in bfloat16. KV eviction. Eviction is applied to the base model only, via R-KV (1), which compresses the base’s KV to a fixed budget B at every step by scoring cached tokens with a mix of query-attention importance and key redundancy (cosine similarity), keeping the top-B plus a small recent window. We use budgets B∈64,128,256,512B∈\64,128,256,512\, a headroom buffer w=128w=128 (so the physical KV is bounded by B′=B+wB =B+w), and mix weight λ=0.1λ=0.1. The helper and PRM keep their full KV cache. Best-of-(N+N)(N+N) trajectory. KV-Rescue grows a single accepted trajectory with a fork-based engine on top of vLLM v1 (9) that avoids re-prefilling history: one persistent sequence holds the accepted trajectory’s (evicted) base KV, and each step (i) forks N base candidates by physically copying that sequence’s live (post-eviction) KV slots into fresh blocks, (i) requests N full-KV helper candidates in parallel, (i) scores all 2N2N candidates with the PRM and accepts the best. If a base candidate is accepted it becomes the trajectory’s new state (its KV already contains the step, evicted correctly); if a helper step is accepted, the base model encodes its tokens in a single forward pass to extend the trajectory, so the base KV absorbs the step and eviction stays incremental. The whole procedure is forward-passes only, with no backtracking and no history re-prefill. We use N=8N=8. The engine is continuous-batched and event-driven: rather than lock-stepping one problem at a time, it keeps many problems in flight in a single shared running set, and a single step() advances all their live candidates together (vLLM continuous-batches at the token level). Each problem proceeds to scoring the instant its own N base candidates have reached the step separator, and retires and is refilled independently, so fast problems do not wait on the slowest and the base GPU stays saturated across problems. The per-step PRM and helper calls (below) run on background threads, off the engine’s critical path. Online degeneration detection. Each base candidate is screened before scoring: it is dropped if its mean per-token predictive entropy exceeds τent=7.0 _ent=7.0 (incoherent generation) or the minimum gzip ratio of a sliding window of width wcmp=800w_cmp=800 falls below τcmp=0.18 _cmp=0.18 (repetition loop); the entropy check uses a guard of 2424 tokens and a window of 1212. Both signals are by-products of the existing decode and add negligible cost. Decoding and evaluation. Following the Qwen2.5-Math non-thinking regime we sample at temperature 0.80.8, top-p 0.950.95, top-k 2020, with maximum context 40964096, up to 10241024 reasoning steps, and at most 512512 tokens per step. We evaluate pass@1 accuracy on MATH500, AIME24, AMC23, GSM8K, and OlympiadBench, extracting the final answer with the official Qwen2.5-Math grader; AIME24 and AMC23 are averaged over five seeds and the rest use a single seed. Serving. The base engine, PRM, and helper run as separate vLLM instances; the PRM and helper are queried over asynchronous HTTP and overlap with base decoding, so their latency is hidden behind the base forward. For the 7B base we co-locate all three on each GPU and shard the problem set across GPUs with work-stealing; for the 72B base we shard the base model with tensor parallelism (TP=8TP=8) and co-locate the PRM (TP=4TP=4) and helper alongside it. Appendix D Overhead Derivation We derive the per-sequence KV memory and per-token decode compute shown in Fig.8, and the context length at which evicted decode becomes cheaper than full-KV, and evaluate them with the Qwen2.5-Math constants. Notation For a decoder-only transformer let L be the number of layers, d the model dimension, hkvh_kv the number of key/value heads, dhd_h the head dimension, P the parameter count, and b the bytes per element (b=2b=2 for bfloat16). Let ℓ be the current context length (tokens), B the R-KV budget, w the eviction buffer, and B′=B+wB =B+w the resulting physical KV bound. Table 4 lists the values we use. Table 4: Model constants used in the derivation (from each model’s config.json). model L d hkvh_kv dhd_h dffnd_ffn P base 7B 28 3584 4 128 18944 7.62×1097.62× 10^9 base 72B 80 8192 8 128 29568 72.7×10972.7× 10^9 helper 1.5B 28 1536 2 128 8960 1.54×1091.54× 10^9 KV memory Each cached token stores a key and a value vector of size dhd_h in every KV head of every layer, so the per-token, per-sequence KV footprint is mkv= 2Lhkvdhb,m_kv\;=\;2\,L\,h_kv\,d_h\,b, (8) the leading 22 counting keys and values. Substituting Table 4 gives mkv7B=56KiBm_kv^7B=56\,KiB, mkv72B=320KiBm_kv^72B=320\,KiB, and mkv1.5B=28KiBm_kv^1.5B=28\,KiB per token. Full-KV memory grows linearly in context, whereas eviction bounds it at the budget: Mfull(ℓ)=mkvℓ,Mevict=mkvB′(constant in ℓ).M_full( )=m_kv\, , M_evict=m_kv\,B (constant in ). (9) The eviction saving on the base model is therefore model-size independent, Mfull(ℓ)Mevict=ℓB′, M_full( )M_evict= B , (10) e.g. ℓ/B′=8192/640≈12.8× /B =8192/640≈ 12.8× at ℓ=8k =8k, B=512B=512. For the full method a problem generates N candidate sequences; the base is evicted and the helper keeps full KV, so MKVR M_KVR =N(mkvbaseB′+mkvhelpℓ), =N (m_kv^baseB +m_kv^help\, ), (11) Mfull-BoN M_full -BoN =Nmkvbaseℓ. =N\,m_kv^base\, . The helper term mkvhelpℓm_kv^help does not depend on the base model, so it dominates MKVRM_KVR when the base is small and is negligible when the base is large. At ℓ=4k =4k, B=512B=512, N=8N=8, the ratio Mfull-BoN/MKVRM_full -BoN/M_KVR is 1.75/1.15≈1.5×1.75/1.15≈ 1.5× for the 7B base but 10.0/2.44≈4.1×10.0/2.44≈ 4.1× for the 72B base. Decode compute For one decoded token the forward cost splits into a context-independent dense part (all linear projections and the FFN, ≈2P≈ 2P FLOP by the standard multiply–add count) and an attention part that is linear in the number of cached tokens attended to: C(ℓ)=2P⏟Cdense+4Ldℓ⏟Cattn(ℓ).C( )= 2P_C_dense\;+\; 4\,L\,d\, _C_attn( ). (12) The attention term is the two matmuls per layer, QK⊤QK and the value aggregation; each runs over the ℓ cached positions across the full query width d, costing dℓd multiply–adds per token, so at two FLOP per multiply–add the two matmuls give 4dℓ4d FLOP, summed over L layers. (Under grouped-query attention the KV memory of Eq. (8) scales with the hkvh_kv cached KV heads, whereas attention compute scales with the full query width d.) With Table 4, CdenseC_dense is 15.215.2 and 145.4145.4 GFLOP/token for 7B and 72B, while Cattn(ℓ)=0.401M⋅ℓC_attn( )=0.401M· and 2.62M⋅ℓ2.62M· FLOP respectively; at ℓ=4k =4k attention is only 11%11\% (7B) / 7%7\% (72B) of the forward, so bounded-KV attention saves little compute at short context. Eviction is not free, however. When R-KV fires (every step once ℓ>B′ >B ), it makes, per layer, one attention-score pass plus a key-similarity, pooling and top-k selection over the B′B physical tokens; to first order this is αCattn(B′)α\,C_attn(B ) with α∈[1.5,3]α∈[1.5,3], so the evicted per-token cost Cevict=Cdense+(1+α)Cattn(B′)=2P+(1+α) 4LdB′C_evict=C_dense+(1+α)\,C_attn(B )=2P+(1+α)\,4LdB (13) depends only on the bounded B′B and is thus flat in ℓ . Evicted decode is cheaper per token once the full-context attention outgrows this bounded cost, Cattn(ℓ)≥(1+α)Cattn(B′)C_attn( )≥(1+α)\,C_attn(B ). As CattnC_attn is linear in ℓ , this happens beyond a crossover that is independent of model size, ℓ⋆=(1+α)B′. =(1+α)\,B . (14) We measure this crossover at ℓ⋆≈2500 ≈ 2500; for B′=640B =640 that corresponds to α≈2.9α≈ 2.9, near the top of the [1.5,3][1.5,3] range and consistent with R-KV doing more than a single attention pass (it also computes the key similarity, pooling and top-k). Below ℓ⋆ eviction is slightly slower, as the scoring outweighs the attention it saves; above it eviction is faster, and the gap grows with ℓ−ℓ⋆ - , reaching about 2%2\% at 4k4k and 6%6\% at 12k12k. The N helper candidates add, per token, a fraction ρ=Chelp(ℓ)/Cevictbaseρ=C^help( )/C^base_evict of a base candidate, ρ=2Phelp+4Lhelpdhelpℓ2Pbase+(1+α) 4LbasedbaseB′;ρ= 2P^help+4L^helpd^help 2P^base+(1+α)\,4L^based^baseB ; (15) the helper is a fixed 1.51.5B model, so this fraction falls as the base grows, from about 24%24\% for the 7B base to 2.5%2.5\% for the 72B base at ℓ=4k =4k. These are the quantities in Fig.8: eviction caps the base KV cache (an ℓ/B′ /B saving, Eq. (10)) and its attention cost, at the price of a per-token overhead that is only repaid past ℓ⋆≈2500 ≈ 2500 (Eq. (14)), plus the small helper cost ρ (Eq. (15)) of roughly 24%24\% for the 7B base and 2.5%2.5\% for the 72B base. Figure 10: KV-Rescue with SnapKV eviction (Qwen2.5-Math-7B). pass@1 vs. base KV budget over the five benchmarks (MATH500, AIME24, AMC23, GSM8K, OlympiadBench), for N=1N=1 (top) and N=8N=8 (bottom). Blue is KV-Rescue (ours), red is eviction-only best-of-N; dashed and dotted lines mark the base full-KV and helper (small) full-KV references. Same setup as main results with R-KV replaced by SnapKV. Table 5: Accuracy (%, boxed+correct; mean± over seeds) vs. base KV budget, for the 4×54× 5 Pareto grid (Qwen2.5-Math 7B/72B, N=1,8N=1,8). bon = evicted-main best-of-N; b2n = ours (+full-KV 1.5B safety net); full-KV / small full-KV are no-eviction references. Metric identical to the Pareto figure (is_correct ∧ ; PRM/small-degraded rows excluded). std shown only where >>1 seed (AIME, AMC23). model N budget MATH500 AIME24 AMC23 GSM8K Olympiad 7B 1 bon 64 46.8 0.7 ± 1.3 16.5 ± 3.7 54.4 13.8 bon 128 64.4 1.3 ± 1.6 28.0 ± 2.9 81.0 25.8 bon 256 73.6 4.7 ± 1.6 46.0 ± 9.7 92.0 32.9 bon 512 77.0 4.7 ± 2.7 50.0 ± 6.5 95.3 36.1 b2n 64 77.8 12.7 ± 3.9 58.0 ± 5.6 91.8 38.4 b2n 128 78.6 12.7 ± 1.3 60.0 ± 3.5 94.2 39.4 b2n 256 80.8 14.7 ± 2.7 62.5 ± 1.6 95.0 41.2 b2n 512 83.0 16.7 ± 4.7 64.0 ± 2.5 95.9 42.5 full-KV 80.6 12.0 ± 5.4 56.0 ± 3.0 95.5 40.0 small full-KV 73.6 9.3 ± 2.5 53.5 ± 4.1 86.6 37.0 7B 8 bon 64 52.8 1.3 ± 1.6 19.0 ± 3.0 61.0 16.7 bon 128 69.2 4.7 ± 1.6 44.5 ± 5.6 82.9 29.6 bon 256 79.4 8.7 ± 2.7 54.0 ± 8.5 94.7 38.1 bon 512 85.2 14.0 ± 4.9 60.0 ± 4.5 96.0 41.6 b2n 64 82.2 12.0 ± 2.7 60.5 ± 6.2 94.8 40.6 b2n 128 83.2 14.0 ± 4.9 62.0 ± 3.7 95.9 40.9 b2n 256 84.2 14.7 ± 1.6 63.0 ± 4.0 96.6 42.8 b2n 512 85.4 16.0 ± 2.5 62.0 ± 5.8 96.8 44.0 full-KV 84.0 19.3 ± 2.5 64.0 ± 2.5 96.4 41.8 small full-KV 73.6 12.0 ± 2.7 54.0 ± 3.7 87.2 39.4 72B 1 bon 64 36.0 2.0 ± 2.7 10.5 ± 1.9 42.5 10.2 bon 128 53.8 4.0 ± 3.3 23.5 ± 3.7 67.2 19.1 bon 256 68.6 5.3 ± 2.7 38.5 ± 5.1 88.5 27.6 bon 512 78.6 12.0 ± 4.0 48.5 ± 6.4 93.0 38.8 b2n 64 76.4 12.0 ± 1.6 55.0 ± 7.7 91.3 39.3 b2n 128 79.2 12.0 ± 4.5 58.5 ± 3.7 93.6 39.7 b2n 256 80.6 14.7 ± 5.0 63.5 ± 6.6 95.1 41.0 b2n 512 83.4 16.0 ± 4.4 63.5 ± 3.4 96.1 43.0 full-KV 82.6 15.3 ± 2.7 56.5 ± 4.1 93.3 43.3 small full-KV 73.6 9.3 ± 2.5 53.5 ± 4.1 86.6 37.0 72B 8 bon 64 60.6 4.0 ± 1.3 31.5 ± 6.0 74.1 28.6 bon 128 74.8 9.3 ± 3.3 52.0 ± 5.3 90.5 35.7 bon 256 83.4 14.0 ± 1.3 62.0 ± 3.3 95.0 43.3 bon 512 85.6 19.3 ± 6.5 69.5 ± 5.1 96.2 46.8 b2n 64 82.6 15.3 ± 3.4 61.0 ± 5.1 95.2 43.4 b2n 128 82.4 18.0 ± 2.7 61.5 ± 4.6 96.0 43.7 b2n 256 86.2 17.3 ± 2.5 67.0 ± 4.3 96.3 46.7 b2n 512 86.6 19.3 ± 3.3 72.5 ± 8.5 97.0 48.4 full-KV 86.4 23.3 ± 3.7 71.5 ± 3.4 96.5 50.2 small full-KV 73.6 12.0 ± 2.7 54.0 ± 3.7 87.2 39.4 Appendix E Experiment Results Main Result Table 5 provides the exact pass@1 numbers behind the main results figure across the 4×54× 5 grid. KV-Rescue with SnapKV KV-Rescue is agnostic to the underlying evictor: it treats the base as a black-box policy conditioned on whatever cache the evictor keeps. To verify this, Figure 10 repeats the 7B study with R-KV replaced by SnapKV (11), and the same pattern holds, showing the recovery does not depend on the specific eviction method.