Paper deep dive
KV-streams for Efficient Compaction in Agentic Reinforcement Learning
Emiliano Penaloza, Dane Malenfant, Dheeraj Vattikonda, Roger Creus Castanyer, Siddarth Venkatraman, Abhay Puri, Jonathan Light, Matthew James Sargent, Augustine N. Mavor-Parker, Massimo Caccia, Lucas Caccia, Glen Berseth, Esmeralda S. Whitammer, Alessandro Sordoni, Minseon Kim, Marc-Alexandre CÎté, Laurent Charlin, Guillaume Lajoie
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/30/2026, 7:38:47 AM
Summary
The paper introduces KV-streams, a plug-and-play strategy for efficient context compaction in agentic Reinforcement Learning (RL) with Large Language Models (LLMs). It addresses the bottleneck of GPU memory and training throughput caused by repeated prefilling of retained context tokens during compaction. KV-streams achieves this by streaming the Key-Value (KV) cache forward rather than flushing it, eliminating redundant computation. Experiments on TextWorld, ALFWorld, and SWE-bench demonstrate significant wall-clock speedups (2.6x to 5x) and comparable or superior performance to re-prefill compaction and full context baselines. Additionally, the study finds that the streamed KV cache can act as a recurrent state, allowing the model to retain information from evicted context, a behavior that emerges through RL alone without requiring preliminary Supervised Fine-Tuning (SFT).
Entities (17)
Relation Signals (17)
KV-streams â enables â streaming KV cache
confidence 95% · KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction.
KV-streams â evaluatedon â TextWorld
confidence 95% · We evaluate KV-streams... on TextWorld... obtaining a 5x speedup
KV-streams â evaluatedon â SWE-bench Verified
confidence 95% · We carry these results over to agentic software-engineering tasks... SWE-bench Verified
KV-streams â evaluatedon â ALFWorld
confidence 95% · We evaluate KV-streams... on ALFWorld... obtaining a 1.3x speedup
KV-streams â improves â Training Throughput
confidence 95% · KV-streams... substantially increases throughput while showing no evidence of hindering performance.
KV-streams â reduces â Wall-clock Time
confidence 95% · achieving a 2.6 to 5x wall-clock speedup in training.
KV-streams â compatiblewith â Markovian Thinker
confidence 90% · We show that KV-streams enable three different compaction strategies... Markovian Thinker
KV-streams â compatiblewith â Sliding Window
confidence 90% · We show that KV-streams enable three different compaction strategies... Sliding Window
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.
Tags
Links
- Source: https://arxiv.org/abs/2609.35750v2
- Canonical: https://arxiv.org/abs/2609.35750v2
Trouble viewing inline? Open PDF directly â
Full Text
67,605 characters extracted from source content.
Expand or collapse full text
KV-streams for Efficient Compaction in Agentic Reinforcement Learning Emiliano Penaloza ââ 1,2,5 , Dane Malenfant â 1,3 , Dheeraj Vattikonda 1,4 , Roger Creus Castanyer 1,2 , Siddarth Venkatraman 1,5 , Abhay Puri 6 , Jonathan Light 2,7 , Matthew James Sargent 8,9 , Augustine N. Mavor-Parker 9 , Massimo Caccia 12 , Lucas Caccia 12 , Glen Berseth 1,5 , Esmeralda S. Whitammer 10 , Alessandro Sordoni 1 , Minseon Kim 2 , Marc-Alexandre CĂŽtĂ© 2 , Laurent Charlin â 1,11 , Guillaume Lajoie â 1,5 1 Mila 2 Microsoft 3 McGill University 4 Polytechnique MontrĂ©al 5 UniversitĂ© de MontrĂ©al 6 ServiceNow Inc 7 Rensselaer Polytechnic Institute 8 University College London, University of London 9 Vmax 10 Edinburgh University 11 HEC MontrĂ©al 12 Cohere â Core contributors â Equal advising â Corresponding author: emilianopp550@gmail.com Abstract Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a2.6to5Ăwall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline. Code Repository: github.com/Emilianopp/KV-streams SGLang Fork: github.com/Emilianopp/sglang 0/0800/1001600/200 GPU hours / wall-clock hours 0.5 0.6 0.7 0.8 0.9 1.0 final success rate TextWorld 0/040/1080/20120/30 GPU hours / wall-clock hours 0.7 0.8 0.9 1.0 ALFWorld 0/0200/25400/50600/75 GPU hours / wall-clock hours 0.40 0.45 0.50 0.55 0.60 SWE-bench Verified SummarySliding-WindowMarkovian ThinkerMarkovian PickerFull context Re-prefill compactionKV-streams Figure 1: Final evaluation score against total training compute for every run on TextWorld, ALFWorld and SWE-bench Verified. Axis ticks give GPU hours and wall-clock hours. Colour denotes the compaction strategy, circles are KV- streams and squares re-prefill compaction, with full context as the reference. Error bars give the standard deviation over three seeds on SWE-bench Verified, and the standard error over seeds, or over evaluation episodes for single-seed runs, on the text-based games. KV-streams reaches the same final score as re-prefill compaction at a fraction of the compute. arXiv:2609.35750v2 [cs.LG] 29 Sep 2026 KV-streams for Efficient Compaction in Agentic Reinforcement Learning A Prefill-Compaction Inference P P P P Training KV-streams Inference P P P P Training B Compaction strategies Summary P123456 ()Îșí PS7891011 Markovian Thinker P123456 ()Îșí P456789 Sliding Window P123456 ()Îșí P234567 Markovian Pick P123456 ()Îșí P134789 prompt cached prefilled new tokens summary training tokens Figure 2: Compaction strategies and their training costs. (A) re-prefill compaction recomputes retained tokens and creates separate training traces. KV-streams preserves retained KVs and trains on a continuous trace with an attention mask that reproduces eviction. (B) context transformations under each strategy (P: prompt; S: summary). 1 Introduction Reinforcement learning on long-horizon agentic tasks requires generating many concurrent rollouts, each having key-value (KV) caches growing linearly with length. Under a fixed GPU memory budget, the longest-running rollouts quickly become the bottleneck, monopolizing memory that shorter rollouts could otherwise use. Reducing the memory footprint of long traces can therefore substantially increase training throughput, enabling more optimizer steps and larger batch sizes (Kwon et al., 2023; Chen et al., 2026). To stop memory from growing, a variety of compaction techniques can be used. They remove a portion of the context once a fixed token budget is exceeded (e.g. replacing older context with a summary) (Wu et al., 2026b; Li et al., 2026b). Compaction lets a trace extend indefinitely (in principle) without growing its memory footprint, but increases training compute. This is due to the retained context being prefilled each time following a compaction, so the model performs repeated forward passes over the same tokens (Cim et al., 2026). In this work, we propose KV-streams to alleviate the computational bottleneck of compaction. Specifically, KV-streams evicts KV entries directly in the inference engine yielding the memory benefits of compaction while maintaining a single continuous stream of KVs, thus avoiding any repeated prefill cost. On the trainer side, KV-streams reproduce the eviction pattern with a custom attention mask, so no tokens are re-processed. Through this simple plugin, KV-streams substantially increase both training and inference throughput on the four compaction strategies we tested, and are compatible with others. Figure 2 summarizes those four strategies and the mechanism behind the speedup. We evaluate KV-streams combined with various compaction strategies on long-horizon agentic tasks, using Qwen3-4B-Instruct-2507andQwen3.5-4B, obtaining a5Ăspeedup on TextWorld (CĂŽtĂ© et al., 2019) and a1.3Ă speedup on ALFWorld (Shridhar et al., 2021). Interestingly, while early in training both KV-streams and re-prefill compaction can hinder performance compared to the use of full context, with sufficient training, KV-streams can close this gap and even surpass the performance of full context. Further, we find that models trained with KV-streams also generalize better to unseen text-based games (Cui et al., 2025; Hausknecht et al., 2020; Wang et al., 2022; Jansen et al., 2024). We then carry these results over to agentic software-engineering tasks, where KV-streams reach peak performance 3Ă faster than full context while maintaining comparable performance. Prior work (Kontonis et al., 2026) showed that preserving KVs across compactions lets the KV cache act as a "pseudo- recurrent" state, but found that this behaviour requires a preliminary SFT stage in their training setup. We test whether that requirement holds in our approach. Empirically, the additional wall-clock time spent on SFT does not improve performance, and synthetic experiments show that pseudo-recurrence can emerge in the KV cache without this preliminary SFT stage. This is a significant improvement for RL post training pipelines, with up to a 5Ă speedup. Overall, KV-streams is a simple plug-and-play addition to any compaction strategy that greatly reduces training wall- clock, matches or exceeds the performance of re-prefill compaction, generalizes effectively, and requires no expensive 2 KV-streams for Efficient Compaction in Agentic Reinforcement Learning SFT stage for initialization. We make our inference and training code public athttps://github.com/Emilianopp/ KV-streams for a subset of the experiments. 2 Compaction Long-running LLM agents commonly manage context growth by summarizing earlier interactions or offloading information to external memory, keeping the active context within a fixed budget (Packer et al., 2024; Wu et al., 2026b; Lu et al., 2025). It is necessary to scale LLM context windows by preventing âcontext rotâ and limiting growing KV-memory. During training, limiting these effects is quite important. For instance, a single long rollout can monopolize the inference serverâs memory, throttling throughput. On top of that, agentic settings are specially affected by this hindrance as the environmentâs feedback adds to the growing context. For instance, in software engineering, reading or writing a file can consume a substantial amount of context that may not be necessary at later stages of the task. Reducing this context is therefore important for scaling long-context agentic RL. While prior works have attempted to alleviate these issues Aghajohari et al., 2026; Wu et al., 2026a they either focus on non-agentic settings, which simplify the problem, they do not study its interaction with RL, or most detrimentally, they re-prefill the context incurring a high compute cost. Our goal is to fill these gaps and eliminate any redundant compute cost. To illustrate where this added cost comes from, we formalize compaction and show how KV-streams can be applied to any strategy that meets this definition. We note that our experiments are in agentic settings, which require turn-wise compaction (see Appendix B.4), but we use a generalized entry-level view to describe the process, which generalizes to tokens/turns. KV-streams is readily amenable to other long-context use cases as well. A compaction operatorÎșmaps a contextCto a shorter oneC âČ = Îș(C). It fires whenever a triggerTis met, either a fixed entry budgetBthat caps|C|or the modelâs own decision to compact (Li et al., 2026a). When triggered,Îșcomposes two operators,Îș = gen⊠del. The delete operator removes a span of at leastbâ„ 1tokens, so the context always shrinks, and generate emits a bridge s in its place, possibly empty, such as a summary (Li et al., 2026b; Wu et al., 2026b). 16128256512 tokens kept per compaction (o + s) 0k 250k 500k 750k 1000k extra trainer tokens B = 128 B = 256 B = 512 KV-streams (0) Figure 3: Extra tokens prefilled over a32k-token rollout vs. tokens kept per compactiono +s(overlap oplus summarys), for memory budgetsB. As the retained tokens approach the budget, the repeated prefill grows by a factor of N Bâ(o+s) , the number of times each kept token is re-processed by the trainer. KV-streams prefills nothing again (green, at 0). Prefill-compaction. Typical compaction pipelines continue the compacted context in a fresh trace, which resets the kept tokensâ cached KVs along with their positional encodings. Con- cretely, a retained token that held positioniand cache(k i ,v i )in Cis re-emitted into the new traceC âČ with reset positional encod- ings. Recomputing it under this new position and context gives a different cache entry,(k âČ i ,v âČ i )Ìž= (k i ,v i ), so both its position and its cached KV change. This reset is what forces the kept entries to be prefilled again rather than reused. This strategy incurs a heavy cost. Letrbe the number of entries a compaction keeps, the retained overlap plus any generated summary. Allrof them are prefilled into the fresh trace even though their KVs were already computed once (Cim et al., 2026). Under a token/turn budgetB, the context holdsrentries right after a compaction, so the model can generateb = Bâ rnew entries before it hits the budget and compacts again. A rollout ofNentries therefore compactsN/btimes, and each of those compactions prefillsrentries a second time. Over the whole rollout the extra prefilled entries add up to â re-prefill = rollout entries z| N · kept per compaction z| r Bâ r |z new entries between compactions , which diverges as the retained context approaches the budget, derived in App. E). In practice this can be mitigated by producing fewer compactions or reducing the number of entries kept per compaction (e.g. replacing all turns with a summary), yet this only mitigates the cost and does not remove it. 3 KV-streams for Efficient Compaction in Agentic Reinforcement Learning Figure 3 illustrates this cost at a granular token level, showing how compacting more often makes the training grow exponentially with increased number of compactions. KV-streams.To alleviate these issues we introduce KV-streams, a drop-in modification that skips the repeated prefill entirely. Rather than prefilling the kept tokens again, KV-streams carries the retained cache forward: the retained overlap and the summaryskeep their existing entries(k i ,v i )instead of being recomputed as(k âČ i ,v âČ i ), soâ re-prefill = 0. In the trainer, a custom attention mask blocks attention to the deleted span, replicating this eviction in one forward pass. Figure 2 visualizes the mechanisms for KV-streams, showing how by directly evicting the KV-cache a compacted rollout avoids re-prefilling and does not require to be broken into multiple traces. 3 Compaction Strategies KV-streams alleviates the cost of prefilling context, but different compaction strategies retain different amounts of contextr = o + s. We study four that span this range, from keeping almost none of it to keeping nearly the full budget. Summary. The standard strategy asks the model to summarize its prior context and then drops it (Wu et al., 2026b; Li et al., 2026b). It keeps no entries (o = 0) and replaces them with a summary of variable lengthsup to a preset limit. Markovian Thinker.The Markovian thinker (Aghajohari et al., 2026) keeps the prefill cost down by deleting half the context at each compaction and replacing nothing, so o = B/2 and s = 0. Markovian Pick. Itâs possible that keeping the most recent half is not the most efficient strategy, rather a more efficient mechanism is to let the model pick what turns to remove and which to keep. Thus, at each compaction the model chooses b entries to retain deleting all others. Sliding Window.To probe the high-retention regime where the repeated prefill is most expensive (Figure 3), we use sliding-window compaction (Xiao et al., 2024), which keeps almost the full window (oclose toB,s = 0). It retains the most context and so incurs the largest prefill cost, exactly where KV-streams helps most. 4 Experimental Setup Benchmarks and protocol. We evaluate KV-streams on two text-based games and on software-engineering tasks. Both settings test whether an agent can carry information forward once older context is compacted. Text-based games often require recalling details received early in the episode, so compaction can hurt performance if the agent fails to retain them. Software-engineering tasks involve reading and writing long files, and removing these files from context can discard information the agent later relies on. The text-based games useQwen3-4B-Instruct-2507and the software-engineering tasks useQwen3.5-4B. Across benchmarks we sample8rollouts per prompt and use a learning rate of10 â6 . Within each benchmark, all runs share the same optimization settings and differ only in the compaction strategy. We report evaluation score against cumulative GPU-hours, and we stop a run early only when its wall-clock time exceeds that of the full-context run. TextWorld (CĂŽtĂ© et al., 2019) provides procedurally generated games across four task families (coin_collector, cooking, simple, treasure_hunter) at three difficulty levels. We generate 60,000 synthetic games split uniformly across task families and difficulties, and retain 256 representative tasks for evaluation. Compaction triggers every10turns, and episodes run for up to200turns within a32k-token budget. We train for500gradient steps with a batch size of512on an 8-GPU H100 node (4 inference, 4 trainer), and report mean avg@1 on the evaluation tasks. ALFWorld (Shridhar et al., 2021) is a household environment in which the agent completes a language-specified task by issuing text commands and receives the admissible commands at each step. Compaction triggers every10turns, and episodes run for up to 100 turns within a 16k-token budget. We train on the standard split for up to 200 gradient steps with a batch size of128on4A100 GPUs (1inference,3trainer). Every20gradient steps we evaluate one rollout per game on the 70 seen and 67 unseen validation games, reporting the mean success rate over the two splits. For software engineering we train on1,028SWE-rebench (Badertdinov et al., 2026) and443ScaleSWE (Zhao et al., 2026) tasks. We remove tasks that the base model solves on every attempt and tasks it never solves, measured by 4 KV-streams for Efficient Compaction in Agentic Reinforcement Learning 0/0400/50800/1001200/150 GPU hours / wall-clock hours 0.2 0.4 0.6 0.8 1.0 success rate TextWorld 0/030/7.560/1590/22.5 GPU hours / wall-clock hours 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ALFWorld SummarySliding-WindowMarkovian ThinkerMarkovian PickerFull context Re-prefill compactionKV-streams Figure 4: KV-streams reaches comparable or better success with fewer GPU hours on TextWorld (left) and ALFWorld (right). Axis ticks give GPU hours and wall-clock hours (8GPUs on TextWorld,4on ALFWorld). Solid and dashed curves denote re-prefill compaction and KV-streams, with full context as a reference. Arrowheads mark truncation after exceeding the runtime of full-context completion. We provide complete curves appear in Figure 10. pass@3, so that every remaining task carries a training signal. Compaction triggers every30turns, and episodes run for up to100turns within a64k-token budget. We compare the two strongest KV-streams configurations, Markovian Thinker and Sliding Window, against full context. All methods train with a batch size of256on an8-GPU GB200 node, and we evaluate on SWE-bench Verified (OpenAI, 2024). Due to limited compute (as some experiments can take over a week to execute), we only run multi-seed experiments for AlfWorld, running single seed experiments for both software-engineering tasks and TextWorld. Appendix B lists the concurrency settings and hyperparameters for each benchmark, and Appendix B.2 describes the concurrency sweep used to choose them. Inference engine implementation.Our aim is to show the speedup at as close to production scale as possible, so we build on asynchronous RL frameworks that decouple the trainer and inference GPUs. For the text-based games, we run a forked vLLM inference engine and a forked prime-RL (Team et al., 2025) trainer 1 , both modified to support KV-streams. Our vLLM configuration uses 16-token KV-cache blocks (Kwon et al., 2023). Only complete blocks are added to the prefix cache. 2 When a request does not end on a multiple of16, the trailing tokens stay uncommitted until a later request completes the group. This breaks the stream, since the next request may evict tokens that the uncommitted tokens would have attended to, and we initially found it to substantially hinder performance. We fix it by padding each completion to a multiple of16tokens. The padding must be prefilled by both the inference engine and the trainer, which inflates the total token count. We account for this in Figure 4 and describe it in Appendix B.3. Despite this inefficiency, KV-streams still improves throughput. On the trainer side we use asynchronous RL with a maximum off-policy lag of3, and we modify prime-RL to apply in-flight weight updates without flushing the KV cache (PichĂ© et al., 2026). This change is important for scaling KV-streams, since flushing on every weight update would force a prefill of entire, possibly long, traces and undo the memory savings. For the software-engineering experiments, we instead use a forked Slime trainer with a forked SGLang inference engine 3 , which commits individual tokens with a block size of1and therefore needs no padding. 1 Forked fromhttps://github.com/PrimeIntellect-ai/prime-rl. Our code:https://github.com/Emilianopp/ KV-streams. 2 https://docs.vllm.ai/en/v0.9.1/design/v1/prefix_caching.html 3 https://github.com/Emilianopp/sglang 5 KV-streams for Efficient Compaction in Agentic Reinforcement Learning 0200400 step 0 1.5k 3k 4.5k Time per step (s) 0200400 step 0 1k 2k 3k Generate completions (s) 0200400 step 0 800 1.6k 2.4k Forward/ backward (s) 0200400 step 0 40 80 120 Turns per episode 0200400 step 0 6k 12k 18k Seq. length (non-padding) SummarySliding-WindowMarkovian ThinkerMarkovian PickerFull context Re-prefill compactionKV-streams Figure 5: Smoothed per-step timing and rollout statistics on TextWorld for runs that do not exceed the full-context wall-time. KV-streams (dashed) reduces generation and forward/backward time relative to re-prefill compaction (solid). Rollout statistics show mean turns per episode and sequence length. We find KV-streams to be substantially faster during forward/backward and generation. 5 Experimental Results Here we describe experimental results for both text-based games and software-engineering tasks. For text-based games, we outline all compaction strategies. For software engineering, we only evaluate a subset of the compaction strategies due to compute limits. 5.1 Text-Based Games KV-streams reduces experiment wall-time. Figure 4 shows success rate against GPU-hours on TextWorld and ALFWorld for every compaction strategy under re-prefill compaction and under KV-streams. On TextWorld, KV- streams reaches the final performance of re-prefill compaction with between3.7Ăand11.3Ăfewer GPU-hours, while performing better or similarly. Re-prefill compaction is also expensive in absolute terms. Summary and Sliding-Window with re-prefill compaction consume more GPU-hours than training with full context. We find that all variants of KV-streams substantially reduce wall-clock time compared to both full context and their re-prefill counterparts. On ALFWorld, due to shorter traces (the maximum sequence length is 16k), the re-prefilling overhead is smaller, which reduces the gap. Regardless, KV-streams still achieves an increased throughput between1.3Ăand3.0Ă. KV-streams again reaches the final success rate of re-prefill compaction without incurring a performance loss across the four strategies. The full training curves for these runs, including the ones truncated in Figure 4, are in Appendix C.1. Where does the speedup come from?Figure 5 breaks the time per step into generation and forward/backward. As expected, the bottleneck for re-prefill compaction is the forward and backward pass, where a large retained contextr forces repeated recomputation. KV-streams instead reduces time in both the forward/backward pass and generation. While we find that initially re-prefill methods in TextWorld lead to shorter episodes and thus faster sampling, this gain diminishes as training progresses and is erased by the added training time. Performance transfer.We aim to evaluate whether the speedups from KV-streams come at the cost of worse transfer to other tasks. We evaluate on unseen text-based games (Cui et al., 2025; Hausknecht et al., 2020; Wang et al., 2022; Jansen et al., 2024) using the final checkpoint from the TextWorld runs, with the same budget and context settings as training (B = 10turns and a32k-token context limit). Figure 7A reports the score for each game averaged over three seeds. For every game, a version of KV-streams performs on par with the best strategy. A version of KV-streams also improves upon the base model in every game, showing that learning on TextWorld transfers to other text-based games. No single compaction strategy performs best across all games (Figure 7A), consistent with broader findings that memory-method performance depends on the task (Wang et al., 2026). Regardless, we find no loss in transfer for KV-streams compared to full context and re-prefill compaction. 6 KV-streams for Efficient Compaction in Agentic Reinforcement Learning 0/0200/25400/50600/75 GPU hours / wall-clock hours 0.40 0.45 0.50 0.55 0.60 success rate SWE-bench Verified 0/0200/25400/50600/75 GPU hours / wall-clock hours 0.4 0.6 0.8 success rate Training reward Sliding-WindowMarkovian ThinkerFull context Re-prefill compactionKV-streams Figure 6: Comparison of KV-streams against full context and re-prefill Markovian Thinker on software-engineering tasks. Left: SWE-bench Verified success rate, smoothed with a centred three-checkpoint average. The final point is the mean and standard deviation over three evaluation seeds of the last checkpoint, as in Figure 1. Right: training reward, smoothed with an exponential moving average over8gradient steps for every run (raw trace faint). Axis ticks give GPU hours and wall-clock hours on8GPUs, and every run is shown up to200gradient steps. KV-streams reaches peak performance about 3Ă faster than full context and about 2Ă faster than re-prefill compaction. 5.2 Software Engineering Agents KV-streams scales to software-engineering tasks.Using the training set described in Section 4, we train for200 gradient steps and evaluate on SWE-bench Verified, a standard software-engineering benchmark of500Python bug fixes. Figure 6 shows the evaluation results (left) and the training reward (right). To measure final performance, we evaluate the last checkpoint of each run over three seeds. Sliding-Window with KV-streams reaches52.7± 2.1%, statistically within the margin of error of full context (51.5± 1.2%) and re-prefill Markovian Thinker (53.4± 0.2%). Our results therefore remain consistent with the text-based games, finding no significant degradation in performance compared to full context. As before, KV-streams substantially reduces wall-clock time. Sliding-Window with KV-streams reaches its peak in about20hours compared to more than65hours for full context, a3Ăspeedup. The gap persists when comparing Markovian Thinker under re-prefill compaction and under KV-streams, where KV-streams reaches its peak in about 32 hours versus about 60 hours for re-prefill. 6 Is SFT Necessary for Recurrent KVs? Since retained KVs attended to tokens that are no longer in the context, prior work has shown they can act as a psudo-recurrent state (Kontonis et al., 2026), carrying forward information that has since been removed from the context. Because the model was not pretrained with this in mind, prior work relies on supervised fine-tuning to make the behaviour emerge as a first post training stage, before RL fine tuning can be done. We ask if this two-stage post training is necessary, and test whether KV-streams can enable direct RL fine tuning without an SFT stage. We test this by comparing cost savings of skipping an SFT warm start on TextWorld, and we analyze the mechanisms giving rise to this pseudo recurrent state in a controlled synthetic experiment 6.1 Prior SFT on TextWorld If SFT were a necessary precursor for this recurrent behaviour, we would expect performance to be better with prior SFT. To analyze this and remove the confounder of SFT from a better model, we collect 10k trajectories from Qwen3-4B-Instruct-2507with a 32k context limit (matching RL training), then perform SFT on those traces using 7 KV-streams for Efficient Compaction in Agentic Reinforcement Learning A 0.00 0.20 0.40 0.60 0.80 ALFWorld 0.00 0.20 0.40 0.60 ScienceWorld 0.00 0.01 0.02 0.03 Jericho 0.00 0.10 0.20 0.30 DiscoveryWorld Mean reward Full ContextMarkovian ThinkerSliding WindowSummaryBase Model Re-Prefill CompactionKV-streamsBest mean B 0/0120/15240/30360/45 GPU hours / wall-clock hours 0.2 0.4 0.6 0.8 1.0 success rate TextWorld SFT-Time Sliding-WindowMarkovian Thinker SFT + RLRL only Figure 7: (A) Transfer of TextWorld-trained checkpoints to other environments. Bars show mean reward and whiskers show the standard deviation across evaluation seeds. (B) An SFT warm-start does not pay for itself once its cost is counted. TextWorld success rate against GPU hours, as in Figure 4, for Sliding-Window and Markovian Thinker with KV-streams, with an SFT warm-start followed by RL (SFT + RL, solid) and RL only (dashed). SFT + RL curves start after the GPU hours spent on SFT (SFT-Time arrow). The SFT loss is in Appendix C.2. randomly sampled eviction attention masks. For instance, sometimes we do sliding window with B = 10 and a stride equal to 1, other times we use Markovian Thinker truncation masking half the context (B = N/2). Figure 7 B compares RL from the base model against an SFT warm-start followed by RL (Vattikonda et al., 2025). The model fits the full-context loss (Appendix C.2 shows the SFT training curve), yet once SFT compute is counted, RL alone is more compute efficient and shows no observable drop in performance. We generally find that SFT does not justify its cost heavily slowing down Markovian Thinker while when using sliding window both strategies reach peak performance in the same amount of time. These results suggest that prior SFT is not an important pre-requesite for enabling KV-streams, substantially simplifying the effort needed to enable it. 6.2 Recalling Evicted Content The prior experiment shows empirically that KV-streams does not depend on prior SFT. Here we test this systematically with a controlled synthetic task. The model is first assigned an object (a fruit) and then asked to decode a sequence of numbers, with a budget ofktokens. Next we evict the turn containing the assignment and ask the model which object it was assigned, choosing from a list of candidates. After eviction the assignment no longer appears in the modelâs input, so the model cannot reference it and instructing the model to count numbers ensures it does not leak the object in its own context. Thus, it can recover the object only if the tokens it decoded while the assignment was still visible encoded that information in their KV entries, which KV-streams carries forward. We vary the number of decoded tokensk â 16, 32, 64, 128, 256, 512to test whether recall scales with the capacity of the KV-cache. Two further evaluations modify this base setting. The first asks whether a model trained to recall fruits generalizes to a new class of objects (actors). The second asks whether training can extend recall to an object the base model assigns zero probability to, by asking for a TV when the prompt lists only fruits as candidates. RL enables information leakage through KVs. In the first task the model must retrieve the evicted assignment from a list of five candidate fruits, one of which is the assignment. Because the list always contains the answer, random sampling under RL produces some successes from the start, and the model can bootstrap from them. Figure 8A illustrates the setup and reports the results. As the KV-cache budget grows, recall rises to100%and matches SFT. It becomes unreliable only at the smallest budgets of16and32tokens. KV-streams therefore learns to carry the assignment forward through RL alone, without an SFT stage. While this result is sufficient to show that information can be leaked through KVs alone, it is restricted in the setting that it could overfit to the specific use case making things less reliable, as well as the support between the model with evicted context and unevicted context always had overlap, i.e., the both full context and evicted had sufficient probability over the assigned fruit. 8 KV-streams for Efficient Compaction in Agentic Reinforcement Learning AIn-distribution evict ? BGeneralization evict ? C Zero-support target evict ? UserAssistant Evaluation: held-out fruit assignments P 1, 2, ... k tokens P 1, 2, ... k tokens Evaluation: Heldout actor names P 1, 2, ... k tokens P 1, 2, ... k tokens Evaluation: all six trained targets P 1, 2, ... k tokens P 1, 2, ... k tokens Base model: p(TV)=0.00RL-trained: p(TV)=1.00 RLSFT 512 256 128 64 32 16 KV-cache token capacity (k) 100± 0100± 0 74± 44100± 0 100± 086± 23 100± 0100± 0 73± 46100± 0 49± 44100± 0 RLSFT 512 256 128 64 32 16 KV-cache token capacity (k) 94± 1100± 0 81± 399± 1 49± 4564± 5 50± 4384± 13 61± 1095± 8 37± 2036± 45 RLSFT 512 256 128 64 32 16 KV-cache token capacity (k) 100± 0100± 0 72± 48100± 0 61± 38100± 0 56± 35100± 0 61± 38100± 0 37± 299± 0 050100 Mean accuracy (%) Figure 8: Post-eviction retrieval in-distribution (A), on new names (B), and with TV trained but unlisted (C). Heatmaps show RL and SFT accuracy (mean±SD over three seeds) at the best learning rate per condition. Panel C covers all trained targets; its inset compares empirical TV-answer probabilities in the initial and final training batches of an RL run. Icons represent text targets. KV-streams generalizes across contexts. To test whether the information carried in the KV-cache is overfit to the training context, we modify the task. We take the model trained to recall fruits and ask it to recall an actor instead. Figure 8B shows the results. RL and SFT perform similarly at larger token budgets, and RL trails at smaller ones. Given a sufficient KV-token budget, KV-streams trained with SFT and with RL generalizes beyond the training domain. KV-streams enables recall of objects with zero initial support.Finally, we test whether RL can teach the model to recall an object it would never sample once the assignment turn is evicted. We assign one of six objects at random, five fruits and a television, but at recall we list only the five fruits as options. The base model therefore places almost no probability on the television and would not sample it on its own. A model with full context, by contrast, can always recall that it was assigned the television. This lets us test whether information carried in the KV-cache can align the compacted modelâs distribution with the full-context one, bringing in a token that would otherwise never be sampled. SFT has no difficulty here by construction, since it forces the target response. Figure 8C shows that RL learns this behaviour as well, and as the KV-token budget grows it recalls the television100%of the time. Even in this hardest case, RL alone aligns the compacted distribution with that of full context. Overall, our empirical and synthetic experiments show that RL alone is enough for the model to carry information forward through the KV-cache, given sufficient KV-cache capacity, and that no expensive SFT stage is needed. This broadens the applicability of KV-streams. It becomes a plug-and-play mechanism that can be added at post-training without any additional pipeline stage. 7 Related Work Many methods keep long-context generation affordable by capping how much each new token can cost. Some rewrite the model itself, as in recurrent and state-space architectures, which hold cost flat but only after the model is retrained to 9 KV-streams for Efficient Compaction in Agentic Reinforcement Learning use them (Gu & Dao, 2024; Yang et al., 2025; Fu et al., 2025). Transformer-XL carries hidden states between segments (Dai et al., 2019), while Compressive Transformer compresses older memories for subsequent attention (Rae et al., 2020). GTrXL adapts Transformer-XL for stable reinforcement learning in memory-dependent environments (Parisotto et al., 2020). Others leave the pretrained model untouched and simply hold the context to a fixed size, most often by summarizing old turns or keeping the most recent ones (Wu et al., 2026b; Li et al., 2026b). The second family is convenient but pays a hidden tax, since every time the budget is hit the window is torn down and rebuilt, and the same tokens are pushed through the model again (Cim et al., 2026; Aghajohari et al., 2026; Kontonis et al., 2026; Wu et al., 2026a). That repeated work is the prefill cost we study in Section 2, and it is what makes these runs jagged and starves the GPU. KV-streams targets this tax directly. We retain whatever the strategy chooses to keep, but rather than rebuild the window we drop the evicted keys and values inside the inference engine and mirror that drop in the trainer with an attention mask. The upshot is a smooth, steady cost per token that does not depend on which compaction rule is in play, close in spirit to schemes like attention sinks (Xiao et al., 2024) that sidestep rebuilding but without committing to one fixed retention pattern. In concurrent work, sliding windows over the KV cache have also been shown to be an efficient alternative for long generations, for test-time scaling (Muennighoff et al., 2026) and against linear attention (Jolicoeur-Martineau et al., 2026). KV-streams is complementary, since it applies to any compaction strategy and removes the re-prefill cost of training it with RL. Inference-time cache eviction is also studied by H 2 O, which retains recent and high-attention tokens (Zhang et al., 2023), and TOVA, which interprets Transformers as multi-state RNNs and bounds their state through KV-cache compression (Oren et al., 2024). Our focus is on efficient agentic RL training that reproduces the inference-time eviction pattern. A related line of work trains the model to manage its own context, so that the content surviving a compaction is itself learned, as a generated summary in some methods and as a compact memory state in others (Yan et al., 2026b,a; Zhou et al., 2026; Li et al., 2026b; Lu et al., 2025). These methods decide what to keep, and they all rebuild the window after each compaction, so the retained content is prefilled again and the training cost we study applies to them unchanged. KV-streams is orthogonal to this choice. It does not propose a new compaction rule and instead makes any rule cheaper to train, so these learned strategies could run on top of it. AutoCompressors adapt pretrained language models to compress context into learned summary vectors used as soft prompts (Chevalier et al., 2023). For multi-turn agents, summarization-based context management can also be optimized jointly with tool use through RL (Lu et al., 2025). 8 Conclusion We introduce KV-streams, a plug-and-play mechanism that is compatible with any compaction strategy. We show that by avoiding the repeated prefill cost, KV-streams substantially reduces experiment wall-time, providing up to a3Ă speedup while often improving performance and at worst not hindering it. Further, we show that streamed KVs can act as a pseudo-recurrent state, carrying forward information that has long been removed from the context. Unlike prior work, we show that prior SFT is not a prerequisite for this effect to occur, which greatly simplifies the adoption of KV-streams. Overall, we show that KV-streams is a simple and effective way to substantially increase experiment throughput. AI Use Statement In this work, we used generative AI tools to assist with manuscript drafting and editing, data analysis and plotting code, and the creation and refinement of figures, diagrams, and illustrative icons. We have reviewed all AI-assisted work and take responsibility for the final content, including text, claims, code, and visual artifacts. Acknowledgments We thank the Mila IDT team for their support with the compute infrastructure used in this work. We thank Amirhossein Kazemnejad for their constructive feedback on the project. EP acknowledges the support of the NSERC PGS-D grant and the Bourse en intelligence artificielle provided by UniversitĂ© de MontrĂ©al. LC recognizes the support of NSERC, the Canada CIFAR AI Chair Program, the Canada First Research Excellence Fund and IVADO. GL and GB acknowledge the support of the Canada CIFAR AI Chair Program. 10 KV-streams for Efficient Compaction in Agentic Reinforcement Learning References Milad Aghajohari, Kamran Chitsaz, Amirhossein Kazemnejad, Sarath Chandar, Alessandro Sordoni, Aaron Courville, and Siva Reddy. The markovian thinker: Architecture-agnostic linear scaling of reasoning. In The Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview.net/forum?id= 3As6AQ9ELI. Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei An- driushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents. Advances in Neural Information Processing Systems, 38, 2026. Kaiwen Chen, Xin Tan, Jingzong Li, and Hong Xu. Libra: Efficient resource management for agentic rl post-training, 2026. URL https://arxiv.org/abs/2606.03077. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 3829â3846, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.232. URL https://aclanthology.org/2023.emnlp-main.232/. Musa Cim, Burak Topcu, Chita Das, and Mahmut Taylan Kandemir. Parallel context compaction for long-horizon llm agent serving, 2026. URL https://arxiv.org/abs/2605.23296. Marc-Alexandre CĂŽtĂ©, Ăkos KĂĄdĂĄr, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworld: A learning environment for text-based games. In Tristan Cazenave, Abdallah Saffidine, and Nathan Sturtevant (eds.), Computer Games, p. 41â75, Cham, 2019. Springer International Publishing. ISBN 978-3-030-24337-1. Christopher Zhang Cui, Xingdi Yuan, Ziang Xiao, Prithviraj Ammanabrolu, and Marc-Alexandre CĂŽtĂ©. Tales: Text adventure learning environment suite, 2025. URL https://arxiv.org/abs/2504.14128. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum, and LluĂs MĂ rquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 2978â2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1285. URLhttps: //aclanthology.org/P19-1285/. Zichuan Fu, Wentao Song, Yejing Wang, Xian Wu, Yefeng Zheng, Yingying Zhang, Derong Xu, Xuetao Wei, Tong Xu, and Xiangyu Zhao. Sliding window attention training for efficient large language models, 2025. URL https://arxiv.org/abs/2502.18845. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=tEYskw1VY2. Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre CĂŽtĂ©, and Xingdi Yuan. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 2020. Peter Jansen, Marc-Alexandre CĂŽtĂ©, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, p. 10088â10116. Curran Associates, Inc., 2024. doi: 10.52202/079017-0324. URLhttps://proceedings.neurips.c/paper_files/paper/2024/ file/13836f251823945316ae067350a5c366-Paper-Datasets_and_Benchmarks_Track.pdf. Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, and Emy Gervais. Sliding-window beats linear attention, 2026. URL https://arxiv.org/abs/2608.28444. Vasilis Kontonis, Yuchen Zeng, Shivam Garg, Lingjiao Chen, Hao Tang, Ziyan Wang, Ahmed Hassan Awadallah, Eric Horvitz, John Langford, and Dimitris Papailiopoulos. MEMENTO: Teaching LLMs to manage their context. In Third Conference on Language Modeling, 2026. URL https://openreview.net/forum?id=YaYiQDVsi0. 11 KV-streams for Efficient Compaction in Agentic Reinforcement Learning Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP â23, p. 611â626, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702297. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165. Tianjian Li, Jingyu Zhang, William Jurayj, Xi Wang, Chuanyang Jin, Mehrdad Farajtabar, Eric Nalisnick, and Daniel Khashabi. Self-compacting language model agents, 2026a. URL https://arxiv.org/abs/2606.23525. Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, and Yuxiao Dong. Compactionrl: Reinforcement learning with context compaction for long-horizon agents, 2026b. URL https://arxiv.org/abs/2607.05378. Miao Lu, Weiwei Sun, Weihua Du, Zhan Ling, Xuesong Yao, Kang Liu, and Jiecao Chen. Scaling llm multi-turn rl with end-to-end summarization-based context management, 2025. URL https://arxiv.org/abs/2510.06727. Niklas Muennighoff, Zhengyang Wang, Zeyi Chen, Weijia Shi, Binyuan Hui, John Yang, Dapeng Jiang, Mika Senghaas, Fares Obeid, Johannes Hagemann, Sami Jaghouar, Ludwig Schmidt, Percy Liang, Jason Wei, Andrew Y. Ng, Luke Zettlemoyer, Yejin Choi, and Mike Lewis. Prefix sliding for efficient test-time scaling, 2026. URL https://arxiv.org/abs/2608.26070. OpenAI. Introducing SWE-bench verified. OpenAI announcement, August 2024. URLhttps://openai.com/ index/introducing-swe-bench-verified/. Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state RNNs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 18724â18741, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.1043. URLhttps://aclanthology.org/2024. emnlp-main.1043/. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024. URL https://arxiv.org/abs/2310.08560. Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, RaphaĂ«l Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Hal DaumĂ© I and Aarti Singh (eds.), Proceedings of the 37th Interna- tional Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, p. 7487â7498. PMLR, 13â18 Jul 2020. URL https://proceedings.mlr.press/v119/parisotto20a.html. Alexandre PichĂ©, Ehsan Kamalloo, Rafael Pardinas, Xiaoyin Chen, and Dzmitry Bahdanau. PipelineRL: Faster on-policy reinforcement learning for long sequence generation. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=A35ak14Cyp. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SylKikSYDH. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=0IOX0YcCdTn. Prime Intellect Team, Mika Senghaas, Fares Obeid, Sami Jaghouar, William Brown, Jack Min Ong, Daniel Auras, Matej Sirovatka, Jannik Straube, Andrew Baker, Sebastian MĂŒller, Justus Mattern, Manveer Basra, Aiman Ismail, Dominik Scherm, Cooper Miller, Ameen Patel, Simon Kirsten, Mario Sieg, Christian Reetz, Kemal Erdem, Vincent Weisser, and Johannes Hagemann. Intellect-3: Technical report, 2025. URL https://arxiv.org/abs/2512.16144. Dheeraj Vattikonda, Santhoshi Ravichandran, Emiliano Penaloza, Hadi Nekoei, Thibault de Chezelles, Megh Thakkar, Nicolas Gontier, Miguel Muñoz MĂĄrmol, Sahar Omidi Shayegan, Stefania Raimondo, Steve (Xue) Liu, Alexandre Drouin, Alexandre Piche, Alexandre Lacoste, and Massimo Caccia. How to train your llm web agent: A statistical diagnosis. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.), Advances in 12 KV-streams for Efficient Compaction in Agentic Reinforcement Learning Neural Information Processing Systems, volume 38, Main Conference, p. 53256â53282. Curran Associates, Inc., 2025. doi: 10.52202/085713-1774. URLhttps://proceedings.neurips.c/paper_files/paper/2025/ file/4d239452d41ded07139801d7c966c629-Paper-Conference.pdf. Ruoyao Wang, Peter Jansen, Marc-Alexandre CĂŽtĂ©, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader?, 2022. URL https://arxiv.org/abs/2203.07540. Yuyao Wang, Zhongjian Zhang, Mo Chi, Kaichi Yu, Yuhan Li, Miao Peng, Bing Tong, Chen Zhang, Yan Zhou, and Jia Li. Evomembench: Benchmarking agent memory from a self-evolving perspective. arXiv preprint arXiv:2605.18421, 2026. Ian Wu, Yuxiao Qu, Amrith Setlur, and Aviral Kumar. Reasoning cache: Continual improvement over long horizons via short-horizon RL. In Forty-third International Conference on Machine Learning, 2026a. URLhttps:// openreview.net/forum?id=IpeH08AJ9P. Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Xinmiao Yu, Dingchu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Minhao Cheng, Shuai Wang, Hong Cheng, and Jingren Zhou. Resum: Unlocking long-horizon search intelligence via context summarization, 2026b. URLhttps://arxiv.org/abs/2509.13313. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL https://arxiv.org/abs/2309.17453. Yuchen Yan, Liang Jiang, Jin Jiang, Shuaicheng Li, Zujie Wen, Zhiqiang Zhang, Jun Zhou, Jian Shao, Yueting Zhuang, and Yongliang Shen. Inftythink+: Effective and efficient infinite-horizon reasoning via reinforcement learning, 2026a. URL https://arxiv.org/abs/2602.06960. Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang, Jian Shao, and Yueting Zhuang. Inftythink: Breaking the length limits of long-context reasoning in large language models. In The Fourteenth International Conference on Learning Representations, 2026b. URL https://openreview.net/forum?id=T1h5em349L. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? id=r8H7xhYPwz. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher RĂ©, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: heavy-hitter oracle for efficient generative inference of large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS â23, Red Hook, NY, USA, 2023. Curran Associates Inc. Jiale Zhao, Guoxin Chen, Fanzhe Meng, Minghao Li, Jie Chen, Hui Xu, Yongshuai Sun, Wayne Xin Zhao, Ruihua Song, Yuan Zhang, et al. Immersion in the github universe: Scaling coding agents to mastery. arXiv preprint arXiv:2602.09892, 2026. Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Bryan Kian Hsiang Low, and Paul Pu Liang. MEM1: Learning to synergize memory and reasoning for efficient long-horizon agents. In The Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview.net/forum? id=XY8AaxDSLb. A Broader Impacts KV-streams lowers the cost of training long-running agents by reducing repeated prefill. These savings may also make harmful or unauthorized automation cheaper. Agents handling sensitive information still need access controls, monitoring, and secure data handling. We release no new pretrained models or datasets. KV-streams does not replace the safety policies, permissions, or privacy protections of the underlying models and environments. 13 KV-streams for Efficient Compaction in Agentic Reinforcement Learning B Training and Implementation Details B.1 Hyperparameters and Rollout Settings Table 1 lists the shared hyperparameters for each benchmark. Text-based games use prime-RL with vLLM; software- engineering tasks use Slime with SGLang. For each strategy, rollout concurrency is the same under re-prefill com- paction and KV-streams (Table 2). Full context uses lower concurrency because long traces occupy more memory (Appendix B.2). The async level limits how many policy versions a rollout can lag behind the trainer; we drop rollouts that exceed the off-policy step limit. TextWorldALFWorldSWE Model Qwen3-4B-Instruct-2507 Qwen3-4B-Instruct-2507 Qwen3.5-4B GPUs (inference / trainer)4 / 4 H1001 / 3 A1004 / 4 GB200 Rollouts per prompt888 Batch size (rollouts per step)512128256 Gradient steps500200200 Learning rate10 â6 10 â6 10 â6 Weight decay0.010.010.1 Adam (ÎČ 1 ,ÎČ 2 )(0.9, 0.9)(0.9, 0.9)(0.9, 0.98) Gradient norm clipping (max norm)1.01.01.0 KL coefficient0.0010.0010.001 Sampling temperature1.01.01.0 Max tokens per turn1,0241,0248,192 Sequence budget (tokens)32k16k64k Max turns per episode200100100 Compaction interval (turns)101030 KV block size (tokens)16161 Async level / off-policy steps1 / 31 / 31 / 2 Evaluation interval (steps)152020 Evaluation set256 games70 seen + 67 unseenSWE-bench Verified Table 1: Hyperparameters shared by all runs within each benchmark. Compaction strategy and rollout concurrency vary between runs (Table 2). TextWorldALFWorldSWE Summary48 (192)64 (64)â Sliding-Window48 (192)64 (64) 70 (280) Markovian Thinker 48 (192)64 (64) 68 (272) Markovian Picker48 (192)64 (64)â Full context12 (48)16 (16) 28 (112) Table 2: Concurrent rollouts per inference GPU for each strategy, with the total across GPUs in parentheses. Settings are the same for re-prefill compaction and KV-streams. 14 KV-streams for Efficient Compaction in Agentic Reinforcement Learning B.2 Concurrency Sweep We sweep rollout concurrency in vLLM on a node with four A100 (80GB) GPUs, comparing full context with sliding- window KV-streams (Figure 9). Full-context throughput falls above12concurrent rollouts per GPU, while KV-streams sustains higher concurrency. This motivates the lower concurrency used for full context (Table 2). 12243648 concurrent rollouts per GPU 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 rollouts per second Rollout throughput 12243648 concurrent rollouts per GPU 0.0 0.5 1.0 1.5 2.0 generated tokens per second (k) Token throughput Full contextKV-streams (10-turn window)KV cache full (preemptions) Figure 9: Sampling throughput against the number of concurrent rollouts per GPU on TextWorld, measured once512 rollouts have completed. Left: rollouts per second. Right: generated tokens per second. KV-streams uses a sliding window of 10 turns. Hollow markers denote settings where the KV cache fills and vLLM preempts requests. B.3 vLLM Block Padding Our vLLM configuration commits only complete16-token blocks to the prefix cache. A partial block must be recomputed on the next request. We pad each turn to a multiple of16tokens so that KV-streams can reuse all of its KVs. The token counts in Figure 4 include this padding. SGLang uses a block size of 1 and needs no padding. B.4 Turn-Wise Compaction In the agentic experiments, compaction removes whole turns, including their<|im_start|>and<|im_end|>tags. This preserves the boundaries of the remaining messages. The retained lengthrcounts tokens in the retained turns and any generated summary (Section 2). Token-level eviction produced degenerate text in our initial experiments. The model required substantial SFT before its generations began to improve. We suspect that removing message-boundary tokens disrupts the format learned during pre- and mid-training. Evicting complete turns caused no such instability in the initial responses, so we use turn-wise compaction in the agentic experiments. It gives less precise control over context length and per-trace memory use. Training with token-level eviction masks during mid-training may allow finer-grained eviction. 15 KV-streams for Efficient Compaction in Agentic Reinforcement Learning C Additional Training Results C.1 Full Training Curves Figures 10 and 11 extend the training and timing curves to each runâs last checkpoint, including runs stopped after exceeding the full-context wall-clock time. Summary and Sliding-Window with re-prefill compaction exceed this budget and have longer step times. Figure 12 plots mean training reward and evaluation success rate by gradient step. 0.2 0.4 0.6 0.8 1.0 success rate TextWorld 0.4 0.6 0.8 1.0 ALFWorld 0 25 50 75 100 125 turns per episode 0 10 20 30 40 0/01000/125 GPU hours / wall-clock hours 0 5000 10000 15000 20000 25000 seq. length (non-padding) 0/050/12.5100/25 GPU hours / wall-clock hours 0 2500 5000 7500 10000 12500 SummarySliding-WindowMarkovian ThinkerMarkovian PickerFull context Re-prefill compactionKV-streams Figure 10: Training curves through each runâs last checkpoint (dot). Summary and Sliding-Window with re-prefill compaction exceed the full-context compute budget. The lower rows show smoothed mean turns per episode and rollout length, excluding padding. 16 KV-streams for Efficient Compaction in Agentic Reinforcement Learning 0200400 step 0 1.5k 3k 4.5k Time per step (s) 0200400 step 0 1k 2k 3k Generate completions (s) 0200400 step 0 1.5k 3k 4.5k Forward/ backward (s) 0200400 step 0 40 80 120 Turns per episode 0200400 step 0 8k 16k 24k Seq. length (non-padding) SummarySliding-WindowMarkovian ThinkerMarkovian PickerFull context Re-prefill compactionKV-streams Figure 11: Per-step timing and rollout statistics, including Summary and Sliding-Window with re-prefill compaction. 0.4 0.6 0.8 1.0 training reward TextWorld 0.2 0.4 0.6 0.8 1.0 ALFWorld 0200400 step 0.2 0.4 0.6 0.8 1.0 eval success rate 080160 step 0.2 0.4 0.6 0.8 1.0 SummarySliding-WindowMarkovian ThinkerMarkovian PickerFull context Re-prefill compactionKV-streams Figure 12: Training curves by gradient step on TextWorld (left) and ALFWorld (right). Top: smoothed mean training reward. Bottom: evaluation success rate, with faint points for individual evaluations. Multi-seed curves show the per-step mean and shaded seed range. Solid: re-prefill compaction; dashed: KV-streams. 17 KV-streams for Efficient Compaction in Agentic Reinforcement Learning C.2 SFT Warm-Start Training Loss The SFT warm-start in Section 6.1 uses10k full-context trajectories fromQwen3-4B-Instruct-2507and randomly sampled eviction attention masks. We train for500steps. Loss falls quickly over the first50steps, then decreases more slowly (Figure 13). 0100200300400 step 0.2 0.3 0.4 0.5 SFT loss Figure 13: SFT warm-start loss by optimizer step. The faint line shows per-step loss; the bold line shows a centered 9-step rolling mean. D Controlled Retrieval Protocol D.1 Task Setup Assignment. We assign the model one of five fruits: apple, banana, mango, orange, or pineapple. The assignment prompt lists all five candidates and asks the model to reply âAssignment acknowledged.â without repeating the fruit. We evaluate on 200 held-out trials, balanced at 40 trials per fruit. KV-cache token capacity. The model then decodes space-separated ascending integers, such as â1 2 3 4 5...â, without naming or paraphrasing a fruit or answering the retrieval question. We varykover16, 32, 64, 128, 256, 512 decoded tokens, not integers. This capacity counts only the sequence-token KVs; prompt and other token KVs are additional. We check that the retained sequence contains no candidate answer. All reported held-out trials pass this check. Eviction and retrieval.We evict the assignment and acknowledgment as one turn and retain the decoded sequence and its KVs. The model then recalls its assigned fruit from the five candidates in the format âRecall: <fruit>â. We score the parsed answer by exact match and verify that eviction preceded the first answer token. With no assignment or candidate answer in the retained sequence, correct recall requires information carried forward in the KVs. D.2 Evaluation Conditions Prompt generalization. We evaluate the five-fruit checkpoints with a different retrieval prompt and no further training (Figure 14B). The assignments, candidates, KV-cache token capacity, and eviction procedure are unchanged. In both prompts, [choices] is a shuffled, comma-separated list of the five fruits. Training and in-distribution evaluation: âWhich fruit were you assigned from these five fruits: [choices]? Respond with exactly âRecall: <fruit>â, replacing <fruit> with the assigned fruit. Do not explain.â Prompt-generalization evaluation: âWhat is your favourite object? Choose from these five options: [choices]. Respond with exactly âRecall: <object>â, replacing <object> with one option. Do not explain.â 18 KV-streams for Efficient Compaction in Agentic Reinforcement Learning Zero-support target (TV).We train on six objects: apple, banana, mango, orange, pear, and television (Figure 14C). The assignment and retrieval prompts list only the five fruits, but TV assignments explicitly name television. Sequence generation and eviction are unchanged. Correct recall must recover the assigned object even when it is not listed. Evaluation uses 240 trials, balanced at 40 per object. Generalization to actor names. We evaluate the five-fruit checkpoints on actor names without further training: Kathryn Erbe, Kevin Bacon, Naomi Watts, Nicole Kidman, and Toni Collette. The model decodes the sequence, then recalls the name after we evict the assignment turn. We evaluate with and without the five names listed as candidates, using200trials per condition, balanced at40per name. As in the fruit task, we reject sequences containing a candidate answer and score recall by exact match. D.3 Training Objectives Training data and checkpoint selection. We train one model per objective and KV-cache token capacityk â 16, 32, 64, 128, 256, 512. The five-fruit task has1,000training and200held-out assignments; the six-target TV task has1,200and240. Both use200training and40held-out examples per target. RL, RFT, and SFT start from Qwen3-4B-Instructand run for100updates. Self-distillation starts from the corresponding SFT checkpoint and runs for another 100 updates. We evaluate final checkpoints (Figure 14). RL. We sample on-policy batches of128trajectories, with four rollouts per assignment, at temperature1.0and top-p = 0.95. Reward is one for an exact post-eviction match with no candidate answer in the decoded sequence, and zero otherwise. The loss weights each generated response tokenâs negative log-probability by its trajectoryâs advantage. The advantage uses a batch-wide leave-one-out baseline: the mean reward of the other127trajectories. If every answer in the batch is incorrect, the baseline is chance accuracy (1/5for five fruits and1/6for the six-target TV task). We add no supervised target loss. RFT. Rejection-sampling fine-tuning (RFT) uses the same sampling and reward as RL. It minimizes negative log- likelihood on generated response tokens from successful trajectories. Incorrect trajectories and those containing a candidate answer in the decoded sequence contribute no loss. RFT uses no leave-one-out advantage weighting. Both RL and RFT use AdamW with a constant learning rate of5Ă 10 â6 , weight decay0.01,(ÎČ 1 ,ÎČ 2 ) = (0.9, 0.95), and gradient-norm clipping at 1.0. SFT. SFT demonstrations contain the assignment acknowledgment, a deterministic sequence of ascending integers truncated toktokens, and the correct responseRecall: <target>. We minimize teacher-forced cross-entropy on assistant tokens only, with the eviction mask blocking attention from the answer to the assignment turn. We use batches of128, AdamW with zero weight decay,(ÎČ 1 ,ÎČ 2 ) = (0.9, 0.95), and gradient-norm clipping at1.0. The learning rate warms up for 20 updates to 10 â5 , then follows a cosine schedule to 10 â6 . Self-distillation (SD).SD reuses the SFT demonstrations and adds a KL loss to supervised cross-entropy, each with coefficient1.0. The student uses the eviction mask. The teacher is a full-context pass of the same current model, without gradients, and can attend to the assignment. We recompute the teacher distribution as training progresses. The KL loss approximatesD KL (p student â„p teacher )over the studentâs top100vocabulary tokens at each assistant position, without renormalizing their probabilities. Optimizer and batch settings match SFT, except that the learning rate warms up for five updates to 5Ă 10 â7 and follows a cosine schedule to 10 â7 . All four objectives use the same eviction mask: answer tokens can attend to the decoded sequenceâs KVs but not to the assignment turn. 19 KV-streams for Efficient Compaction in Agentic Reinforcement Learning D.4 Additional Retrieval Results Figures 14 and 15 show mean accuracy over three seeds. For each condition, we select the learning rate with the highest mean among configurations with three completed seeds. All three seeds use that rate. Figure 8 uses the same selection rule for RL and SFT. For the TV task, accuracy is averaged over all six objects, not just TV trials. RLRFTSFTSD 100.0% 512 100.0% 512 100.0% 512 100.0% 512 74.3% 256 73.3% 256 100.0% 256 100.0% 256 100.0% 128 74.2% 128 86.5% 128 98.5% 128 100.0% 64 74.3% 64 100.0% 64 100.0% 64 73.3% 32 73.3% 32 100.0% 32 100.0% 32 48.7% 16 22.0% 16 99.7% 16 99.8% 16 RLRFTSFTSD 100.0% 512 100.0% 512 89.0% 512 90.2% 512 49.5% 256 64.5% 256 83.8% 256 84.2% 256 57.8% 128 67.7% 128 64.3% 128 76.0% 128 97.2% 64 74.3% 64 97.8% 64 75.0% 64 58.3% 32 63.5% 32 100.0% 32 100.0% 32 42.3% 16 23.7% 16 89.0% 16 88.5% 16 RLRFTSFTSD 100.0% 512 100.0% 512 100.0% 512 100.0% 512 72.2% 256 72.6% 256 100.0% 256 100.0% 256 61.1% 128 16.7% 128 100.0% 128 100.0% 128 55.7% 64 83.3% 64 100.0% 64 100.0% 64 61.1% 32 61.1% 32 100.0% 32 100.0% 32 37.2% 16 38.9% 16 98.9% 16 98.6% 16 0.020.040.060.080.0100.0 Mean accuracy (%) AIn-distributionBPrompt generalization CZero-support target Figure 14: Mean post-eviction accuracy over three seeds: (A) in-distribution, (B) prompt generalization, and (C) a zero-support target (TV trained but unlisted). Panel C averages all six targets. Smaller cell labels give KV-cache token capacity k. RLRFTSFTSD 94.2% 512 100.0% 512 100.0% 512 100.0% 512 81.2% 256 52.0% 256 99.2% 256 99.0% 256 48.7% 128 65.7% 128 63.7% 128 69.2% 128 50.2% 64 44.8% 64 83.8% 64 83.7% 64 60.7% 32 48.3% 32 95.2% 32 95.2% 32 36.7% 16 52.2% 16 35.7% 16 35.3% 16 RLRFTSFTSD 27.2% 512 28.8% 512 31.7% 512 30.3% 512 16.5% 256 4.3% 256 27.0% 256 27.8% 256 7.0% 128 1.2% 128 14.7% 128 29.5% 128 8.2% 64 0.3% 64 29.5% 64 28.8% 64 1.2% 32 1.8% 32 63.2% 32 65.7% 32 0.0% 16 1.3% 16 13.5% 16 12.0% 16 0.020.040.060.080.0100.0 Mean accuracy (%) AGeneralization with options BGeneralization without options Figure 15: Generalization to new actor names with (A) listed options and (B) no listed options. Cells show mean retrieval accuracy over three seeds; smaller labels give KV-cache token capacity k. 20 KV-streams for Efficient Compaction in Agentic Reinforcement Learning E Prefill Token Cost A rollout generatesNtokens with compaction operatorÎșand token budgetB. Each compaction retainsr = o + s tokens: overlapoand any generated summarys. Re-prefill compaction copies them into a new trace and prefills them again. Number of compactions. After compaction, the context holdsrtokens, leaving room forb = Bâ rnew tokens. Generating N tokens takes m = N b = N Bâ r compactions. Prefill cost. Each compaction prefills the r retained tokens, adding a total token cost of â re-prefill = mr = N r Bâ r = N (o + s) Bâ (o + s) . The cost diverges asrapproachesB. It depends on the total retained length, not the split between overlap and summary. Retaining only a summary of lengths = B/4givesr = B/4andâ re-prefill = N/3, regardless ofB. KV-streams reuses the retained KVs, so â re-prefill = 0. 21