Paper deep dive
PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding
Siru Zhong, Qiongyan Wang, Xiaohui Lv, Yuzheng Zhuang, Shuai Tao, Wulong Liu, Haohuan Fu, Yuxuan Liang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/27/2026, 2:56:07 AM
Summary
The paper introduces Prefix-Steered Recurrent Memory (PReM), a memory-token-free framework for long-video understanding using frozen Vision-Language Models (VLMs). PReM decouples video ingestion from query answering by using a recurrent writer to distill visual streams into a compact associative state and a question-conditioned readout to steer model prefixes during prefill. It outperforms baselines across six benchmarks with minimal parameter tuning and memory overhead.
Entities (15)
Relation Signals (17)
PReM → usesbackbone → LLaVA-Video-7B
confidence 95% · We instantiate PReM on three frozen backbones: ... and LLaVA-Video-7B
PReM → usesbackbone → Qwen2.5-VL 3B
confidence 95% · Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B
PReM → usesbackbone → Qwen3-VL-8B
confidence 95% · We instantiate PReM on three frozen backbones: ... Qwen3-VL-8B
PReM → outperforms → frozen baselines
confidence 92% · PReM consistently outperforms frozen baselines at every evaluated visual budget.
PReM → evaluatedon → EgoSchema
confidence 90% · Evaluation spans six benchmarks: ... EgoSchema
PReM → evaluatedon → LVBench
confidence 90% · Evaluation spans six benchmarks: ... and LVBench
PReM → evaluatedon → MVBench
confidence 90% · Evaluation spans six benchmarks: ... MVBench
PReM → evaluatedon → LongVideoBench
confidence 90% · Evaluation spans six benchmarks: LongVideoBench (LVB)
PReM → evaluatedon →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent Memory (PREM), a memory-token-free framework for frozen vision-language models (VLMs). PREM separates video ingestion from query answering: a recurrent writer distills visual streams into a compact 256 KiB multi-slot associative state, while a question-conditioned readout adds memory-derived key/value (K/V) steering modulations to existing non-visual prompt prefixes during prefill. This enables write-once, query-many inference without extra prompt tokens or decoding recurrence. Across six long-video benchmarks in offline and streaming end-of-stream settings, PREM consistently outperforms frozen baselines at every evaluated visual budget. Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle retrieval. These gains require tuning 0.24% of backbone parameters at 0.03 GiB of peak GPU memory overhead.
Tags
Links
- Source: https://arxiv.org/abs/2609.23601v1
- Canonical: https://arxiv.org/abs/2609.23601v1
Trouble viewing inline? Open PDF directly →
Full Text
86,144 characters extracted from source content.
Expand or collapse full text
PReM: Prefix-Steered Recurrent Memory for Long-Video Understanding Siru Zhong Affiliation: The Hong Kong University of Science and Technology (Guangzhou) Affiliation: Beta Infinity Qiongyan Wang Affiliation: The Hong Kong University of Science and Technology (Guangzhou) Xiaohui Lv Affiliation: Beta Infinity Yuzheng Zhuang Affiliation: Beta Infinity Shuai Tao Affiliation: Beta Infinity Wulong Liu Affiliation: Beta Infinity Haohuan Fu Affiliation: Tsinghua University Yuxuan Liang Affiliation: The Hong Kong University of Science and Technology (Guangzhou) Abstract Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent Memory (PReM), a memory-token-free framework for frozen vision-language models (VLMs). PReM separates video ingestion from query answering: a recurrent writer distills visual streams into a compact ≈256≈256 KiB multi-slot associative state, while a question-conditioned readout adds memory-derived key/value (K/V) steering modulations to existing non-visual prompt prefixes during prefill. This enables write-once, query-many inference without extra prompt tokens or decoding recurrence. Across six long-video benchmarks in offline and streaming end-of-stream settings, PReM consistently outperforms frozen baselines at every evaluated visual budget. Under a constrained budget of 16 frames, PReM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle retrieval. These gains require tuning ≈0.24%≈0.24\% of backbone parameters at 0.03 GiB of peak GPU memory overhead. Code is available at https://github.com/siruzhong/PReM. 1 Introduction Long-video understanding requires multimodal models to reason over visual evidence scattered across extended temporal horizons (Wu et al., 2024; Fu et al., 2025; Mangalam et al., 2023). In practical domains such as continuous video streams, egocentric perception (Grauman et al., 2022), instructional procedures (Tang et al., 2019), and robotic manipulation and navigation (Brohan et al., 2023; Anwar et al., 2024), decisive evidence is often brief and surrounded by uninformative content. Effective reasoning thus demands both comprehensive temporal coverage and high sensitivity to subtle cues. However, modern vision-language models (VLMs) are constrained by strict visual token budgets due to quadratic attention complexity and GPU memory limits. This tension creates an inevitable trade-off between temporal breadth and sequence cost. Aggressive frame downsampling risks discarding transient events entirely, while admitting more visual tokens incurs steep computational overhead and dilutes attention, frequently inducing “lost-in-the-middle” failures (Liu et al., 2024). As illustrated in Figure 1, existing approaches navigate this trade-off through three main paradigms: (1) Visual context compression reduces spatiotemporal redundancy via pooling, frame selection, or multi-rate sampling (Cheng et al., 2024; Shen et al., 2025; Li et al., 2026; Xu et al., 2024), yet information discarded during compression cannot be recovered for subsequent reasoning; (2) Video memory banks maintain episodic feature pools or explicit token banks (He et al., 2024; Song et al., 2024; Wang et al., 2025; Zhang et al., 2025; Diko et al., 2025; Liu et al., 2026), but their token-based readouts directly occupy prompt sequence capacity; and (3) KV-cache management dynamically prunes, merges, or evicts intermediate attention caches (Qin et al., 2025; Kim et al., 2025; Di et al., 2025; Ning et al., 2025; Yang et al., 2025; Chen et al., 2026), which avoids prompt token overhead but couples directly to internal attention history. Figure 1: Comparison of long-video VLM paradigms. (a) Compression reduces visual context. (b) Memory banks consume prompt capacity. (c) KV management operates on internal attention states. (d) PReM steers the backbone via prefix modulations (ΔK,ΔV)( K, V) without sequence tokens. Consequently, historical visual evidence in existing paradigms either competes for prompt sequence budget or couples directly to internal attention caches. To address this challenge, we propose Prefix-Steered Recurrent Memory (PReM), a dual-pathway framework that separates video ingestion from query answering. During a one-time ingestion pass, a query-agnostic recurrent writer distills incoming visual features into a compact, multi-slot associative state (≈256≈256 KiB) using selective gating to filter redundancy and protect salient cues. At query time, a question-conditioned readout retrieves memory content and injects it as additive key/value (K/V) steering modulations into non-visual prompt prefixes exclusively during prefill. This design steers the frozen VLM’s attention over its visual buffer without appending auxiliary prompt tokens or altering historical visual caches, supporting write-once, query-many inference. Our main contributions are summarized as follows: • Decoupled Memory Architecture. We introduce a memory-token-free framework for frozen VLMs that separates continuous video ingestion from query answering, retaining historical visual evidence in an external, constant-footprint recurrent state without consuming prompt token capacity. • Gated Recurrence & Prefill-Only Steering. We design a multi-factor gated write update to selectively filter temporal redundancy, coupled with a question-conditioned readout that steers prefix K/V representations during prefill, eliminating recurrent updates during generation. • Comprehensive Empirical Validation. Across six long-video benchmarks and three VLM families, PReM improves macro-average accuracy at every evaluated budget in both offline and streaming end-of-stream settings. Under a constrained budget of 16 frames, it achieves a 3.06% macro-average gain on Qwen2.5-VL-3B (with task-specific improvements up to 11.0%) while tuning ≈0.24%≈0.24\% parameters at 0.03 GiB peak GPU memory overhead. 2 Related Work Visual context compression. Visual context compression alleviates quadratic attention costs by pruning tokens along different dimensions. VideoLLaMA 2 (Cheng et al., 2024) aggregates spatiotemporal features via convolutional connectors, while LLaMA-VID (Li et al., 2024b) distills each frame into dual context-content tokens. To balance spatial detail and motion, SlowFast-LLaVA (Xu et al., 2024) adopts asymmetric frame-rate pathways. Complementarily, LongVU (Shen et al., 2025) and VideoChat-Flash (Li et al., 2026) employ text-guided frame selection and hierarchical token compression, respectively, whereas LongVILA (Chen et al., 2024) scales context via multimodal sequence parallelism. However, information discarded during compression cannot be recovered for subsequent generation. In contrast, PReM maintains a compact visual buffer for direct grounding while accumulating extended temporal context into an external recurrent state. Video memory banks. To maintain historical context beyond the immediate window, token-based memory banks accumulate and cluster visual features across extended horizons. MA-LMM (He et al., 2024) accumulates online features in memory banks, while MovieChat (Song et al., 2024) separates tokens into short- and long-term stores inspired by the Atkinson–Shiffrin model. VideoLLaMB (Wang et al., 2025) propagates recurrent memory tokens across segmented scenes, while ReWind (Diko et al., 2025) leverages instruction-conditioned memory to retrieve salient frames for reasoning. To preserve streaming details, Flash-VStream (Zhang et al., 2025) and VideoChat-Online (Huang et al., 2025) incorporate multi-resolution augmentation buffers. However, these methods expose memory by appending auxiliary tokens directly to the prompt or relying on backbone-coupled bridges, which exhaust sequence capacity. PReM avoids sequence overhead by keeping its recurrent state outside the prompt and accessing it via additive prefix steering. KV-cache management. Managing intermediate attention key-value (KV) states offers a token-free alternative. Video-XL-2 (Qin et al., 2025) and InfiniPot-V (Kim et al., 2025) compress caches via task-aware sparsification or value-norm pruning. For long-term retrieval, ReKV (Di et al., 2025) offloads chunked KV states to host memory, while StreamMem (Yang et al., 2025) aggregates incoming caches into query-agnostic fixed buffers. While effective, these methods operate directly on high-dimensional model-internal activations, tying memory management to backbone-specific attention representations and cache formats. PReM constructs an external recurrent state directly from visual representations and modulates prefix K/V states exclusively during prefill. Fast weights and associative memory. Fast-weight models and linear Transformers maintain context within dynamically updated parameters (Schmidhuber, 1992; Katharopoulos et al., 2020; Schlag et al., 2021; Ramsauer et al., 2021). Recent systems adapt this for long sequences: Titans (Behrouz et al., 2025) and StreamTTT (Chen et al., 2026) employ test-time training to optimize parameters online, introducing non-trivial latency. Closest in interface, δ-mem (Lei et al., 2026) generates low-rank attention corrections during autoregressive generation. PReM departs from these designs by employing closed-form, multi-factor gated outer-product writes to bypass online optimization, and applying modulations exclusively during prefill to eliminate recurrent computation during generation. 3 Prefix-Steered Recurrent Memory Problem formulation and overview. Given an extended video =vtt=1TV=\v_t\_t=1^T of T frames and a textual query Q, long-video understanding aims to generate an answer A conditioned on both modalities. PReM addresses this by decoupling temporal retention from sequence capacity via a dual-pathway interface (Figure 2). The primary pathway feeds a bounded visual buffer B⊂V_B containing B frames into a frozen VLM backbone for direct visual grounding. Concurrently, a recurrent writer ingests the full frame sequence directly from visual encoder representations, updating an external, constant-footprint multi-slot associative state S while bypassing the LLM backbone during ingestion. At query time, a question-conditioned readout projects S into additive steering vectors dik,(ℓ),div,(ℓ)d_i^k,( ),d_i^v,( ) to modulate non-visual prefix K/V representations during prefill. This mechanism steers backbone attention over the visual buffer without adding sequence tokens or fine-tuning backbone weights. Figure 2: PReM architecture and supervision. A query-agnostic gated residual rule updates a K-slot recurrent state S. During QA, prefix representations route memory reads into additive K/V steering. The visual buffer remains standard input; state S operates outside the prompt with zero tokens. 3.1 Query-Agnostic Recurrent Write Let E(vt)p∈ℝdencE(v_t)_p ^d_enc denote the feature from the frozen vision encoder for frame vt∈v_t at spatial patch p∈1,…,Ptp∈\1,…,P_t\, where PtP_t is the number of patch tokens at step t. We obtain the frame-level visual feature xtx_t through spatial mean-pooling and project it into address and content spaces: xt=1Pt∑p=1PtE(vt)p,kt=WkLN(xt),ut=WvLN(xt),x_t= 1P_t _p=1^P_tE(v_t)_p, k_t=W_kLN(x_t), u_t=W_vLN(x_t), (1) where LN(⋅)LN(·) denotes layer normalization. The learnable matrices Wk∈ℝdk×dencW_k ^d_k× d_enc and Wv∈ℝdv×dencW_v ^d_v× d_enc project xtx_t into address and content spaces, respectively. Writer features follow the visual-to-text projection (denc=dmodeld_enc=d_model); spatial patch tokens remain available in the visual buffer. Given the sequence xtt=1T\x_t\_t=1^T, the recurrent state maintains K matrix slots Sm∈ℝdv×dkS_m ^d_v× d_k and slot confidences cm∈[0,1]c_m∈[0,1], where dkd_k and dvd_v denote key and value dimensions. A learnable address transform Am∈ℝdk×dkA_m ^d_k× d_k, initialized near identity, parameterizes each slot: κt,m=norm(Amkt),et,m=ut−Smκt,m, _t,m=norm(A_mk_t), e_t,m=u_t-S_m _t,m, (2) where norm(v)=v/max(∥v∥2,ϵnorm)norm(v)=v/ ( v _2, _norm) denotes ℓ2 _2 normalization with numerical stabilizer ϵnorm=10−12 _norm=10^-12. The vector κt,m∈ℝdk _t,m ^d_k is the unit-norm write address, and residual et,m∈ℝdve_t,m ^d_v measures the discrepancy between projected value utu_t and slot prediction Smκt,mS_m _t,m. As illustrated in Figure 3, five bounded gates regulate memory updates over extended horizons: πtw π^w_t =softmax(rwkt), =softmax(r_wk_t), bt,m b_t,m =σ((Wbkt)m), =σ((W_bk_t)_m), nt,m n_t,m =clip(∥et,m∥2max(∥ut∥2,ϵ),0,1), =clip\! ( e_t,m _2 ( u_t _2,ε),0,1 ), zt z_t =σ(ϕ(LN(xt))), =σ(φ(LN(x_t))), at,m a_t,m =[1−cm(1−zt)]+, =[1-c_m(1-z_t)]_+, gt,m g_t,m =πt,mwbt,mnt,mztat,m, =π^w_t,mb_t,mn_t,mz_ta_t,m, (3) where σ(⋅)σ(·) is sigmoid, [⋅]+=max(0,⋅)[·]_+= (0,·) is ReLU, and ϵ=10−6ε=10^-6 is a numerical stabilizer. Affine gate biases are implicit. The five gates regulate updates as follows: (1) Slot routing (πtw∈ΔK _t^w∈ ^K) maps ktk_t via rw∈ℝK×dkr_w ^K× d_k for semantic slot specialization; (2) Update stability (bt,m∈(0,1)b_t,m∈(0,1)) uses Wb∈ℝK×dkW_b ^K× d_k to scale steps and suppress perturbations; (3) Residual novelty (nt,m∈[0,1]n_t,m∈[0,1]) suppresses redundant writes when et,m≈0e_t,m≈ 0; (4) Visual salience (zt∈(0,1)z_t∈(0,1)) downweights background frames via ϕφ, a dmodel→⌊dmodel/4⌋→1d_model\!→\! d_model/4 \!→\!1 GELU MLP; and (5) Confidence protection (at,m∈[0,1]a_t,m∈[0,1]) protects confident slots (cm→1c_m→ 1) from low-salience writes (zt→0z_t→ 0). The gate gt,mg_t,m modulates the outer-product update for each slot matrix SmS_m and its confidence cmc_m: Sm←σ(fm)Sm+ηwgt,met,mκt,m⊤,cm←maxγccm,sg(gt,mbt,m),S_m←σ(f_m)S_m+ _wg_t,me_t,m _t,m , c_m← \ _cc_m,sg(g_t,mb_t,m)\, (4) where fm∈ℝf_m is a learnable slot-wise forget logit, and σ(fm)σ(f_m) controls gradual memory decay. The fixed factor ηw>0 _w>0 sets the write step size, while γc∈(0,1) _c∈(0,1) sets the exponential confidence decay. The stop-gradient operator sg(⋅)sg(·) blocks confidence gradients. The recurrent state occupies (Kdvdk)O(Kd_vd_k) memory (≈256≈256 KiB for K=4K=4 and dv=dk=128d_v=d_k=128), remaining constant regardless of video duration. Figure 3: Recurrent write update. Subtracting slot predictions from visual projections yields associative residuals, modulated by multi-factor controls to update decayed memory and slot confidences. 3.2 Question-Conditioned Read and Prefix Steering Because the recurrent write update is query-agnostic (Eqs. 1–4), the multi-slot state S=Smm=1KS=\S_m\_m=1^K supports repeated queries after a single video ingestion pass. At query time, we average embeddings of existing non-visual prefix tokens (instructions, query Q, options, and prompt boundaries) into a global question representation q∈ℝdmodelq ^d_model. To accommodate different abstraction levels across depth, we partition the L backbone layers into G contiguous groups, where layer ℓ∈1,…,L ∈\1,…,L\ maps to group g(ℓ)=⌊(ℓ−1)G/L⌋∈0,…,G−1g( )= ( -1)G/L ∈\0,…,G-1\. At layer ℓ and prefix position i, the hidden state hi(ℓ)∈ℝdmodelh_i^( ) ^d_model is conditioned on q and projected to the normalized read address ξi,m(ℓ)∈ℝdk _i,m^( ) ^d_k, retrieving value ri,m(ℓ)∈ℝdvr_i,m^( ) ^d_v from slot m: h~i(ℓ) h_i^( ) =LN(hi(ℓ))+WcLN(q), =LN(h_i^( ))+W_cLN(q), qi(ℓ) q_i^( ) =WrLN(h~i(ℓ)), =W_rLN( h_i^( )), (5) ξi,m(ℓ) _i,m^( ) =norm(Amqi(ℓ)), =norm(A_mq_i^( )), ri,m(ℓ) r_i,m^( ) =Smξi,m(ℓ), =S_m _i,m^( ), where Wc∈ℝdmodel×dmodelW_c ^d_model× d_model is a conditioning projection, Wr∈ℝdk×dmodelW_r ^d_k× d_model maps conditioned features to query space, and AmA_m is the slot address transform from Eq. 2. A router R:ℝ3dv→ℝR:R^3d_v evaluates conditioned query qi(ℓ)q_i^( ), retrieved value ri,m(ℓ)r_i,m^( ), and their Hadamard product, producing relevance logits si,m(ℓ)s_i,m^( ) that normalize into routing weights ρr,(ℓ)i,:∈ΔKρ^r,( )_i,:∈ ^K: si,m(ℓ)=R([Wqvqi(ℓ);ri,m(ℓ);Wqvqi(ℓ)⊙ri,m(ℓ)]),ρi,:r,(ℓ)=softmaxm(si,m(ℓ)/τ),s_i,m^( )=R([W_qvq_i^( );r_i,m^( );W_qvq_i^( ) r_i,m^( )]), ρ^r,( )_i,:=softmax_m(s_i,m^( )/τ), (6) where [⋅;⋅][·;·] denotes concatenation, ⊙ the Hadamard product, and τ>0τ>0 the routing temperature. Wqv∈ℝdv×dkW_qv ^d_v× d_k aligns dimensions; R is a two-layer GELU router (Table 4). Group-specific heads Hg(ℓ),ma∈ℝdmodel×dvH_g( ),m^a ^d_model× d_v (a∈k,va∈\k,v\) map weighted reads to steering vectors dia,(ℓ)∈ℝdmodeld_i^a,( ) ^d_model: dia,(ℓ)=αeffK∑m=1Kρi,mr,(ℓ)Hg(ℓ),mari,m(ℓ),αeff=exp(α^)αrun,d_i^a,( )= _effK _m=1^Kρ^r,( )_i,mH_g( ),m^ar_i,m^( ), _eff= ( α) _run, (7) where α^∈ℝ α is a learnable scale logit and αrun _run is an inference-time scale. The steering vectors define a hidden-state perturbation before the frozen K/V projections: Ki(ℓ)=WK(ℓ)(hi(ℓ)+dik,(ℓ)),Vi(ℓ)=WV(ℓ)(hi(ℓ)+div,(ℓ)),K_i^( )=W_K^( )(h_i^( )+d_i^k,( )), V_i^( )=W_V^( )(h_i^( )+d_i^v,( )), (8) where WK(ℓ),WV(ℓ)W_K^( ),W_V^( ) are frozen projections (biases omitted). The perturbations induce additive shifts ΔK=WKdk K=W_Kd^k and ΔV=WVdv V=W_Vd^v at the projection outputs, followed by the backbone’s native attention transforms (Appendix A.5). Prefill-only steering. Modulation applies exclusively to non-visual prompt prefixes (e.g., instructions, questions Q) during prefill. Modulating prefix keys (ΔK K) reshapes prompt-level attention addressing, while values (ΔV V) inject distilled long-horizon evidence that causally guides subsequent layers. Once prefill completes, these representations persist in the standard KV cache, so autoregressive generation inherits full-horizon guidance without recurrent operations. Appendices A.5 and A.6 detail the attention interface and initialization. 3.3 Training and Inference Protocols During training, videos are sampled uniformly at a fixed frame rate. The writer ingests an incoming sequence of T frames to update the recurrent state S, while the decoder prompt draws a visual buffer BV_B of B frames from the same stream. We optimize solely PReM parameters by minimizing: ℒ=ℒCE+λbalℒbal+λpredℒpred,L=L_CE+ _balL_bal+ _predL_pred, (9) where ℒCEL_CE is the cross-entropy loss, and λbal,λpred _bal, _pred weight auxiliary objectives. The load-balancing term ℒbal=∑m(ρ¯m−1/K)2L_bal= _m( ρ_m-1/K)^2 averages read-routing weights ρ¯m ρ_m over valid prefixes. The prediction loss masks a high-salience temporal span during a second pass and reconstructs its normalized value features from the held-out state, conditioned on the question and temporal position. Appendix A.7 details the head, shapes, target, and loss. During inference, PReM applies the same update and modulation interfaces (Eqs. 4 and 8) in offline and streaming end-of-stream modes: offline ingestion processes the full stream once, whereas streaming carries state (S,c)(S,c) across chunks and answers after the video ends, both predicting A via one-token constrained decoding. 4 Experiments We evaluate PReM across four questions: • RQ1 (Performance): How does PReM compare with long-video and streaming methods? (§4.2) • RQ2 (Scaling): Do the gains hold across visual budgets, durations, and backbones? (§4.3) • RQ3 (Ablations): What are the contributions of prefix steering and write gating, and how stable is the design under training horizon and hyperparameters? (§4.4) • RQ4 (Efficiency): Where do efficiency and task gains originate under matched settings? (§4.5) 4.1 Experimental Setup Models and baselines. We instantiate PReM on three frozen backbones: Qwen2.5-VL-3B, Qwen3-VL-8B, and LLaVA-Video-7B (Bai et al., 2025b; Bai et al., 2025a; Zhang et al., 2024). Per Table 1, baselines include: (1) specialized architectures spanning compression (LLaMA-VID, VideoLLaMA2, LLaVA-Video, LongVILA, LongVU) and streaming memory (MovieChat, StreamChat, ReKV, Flash-VStream) under native budgets; and (2) generalist base VLMs (Qwen2.5-VL-3B, Qwen3-VL-8B). We train only PReM for one epoch on a LLaVA-Video-178K subset (6,430 videos, 17,849 QA pairs, <1.4%<1.4\% of its QA pairs; Table 3). Trainable parameters total 9.07M (Qwen2.5) and 27.48M (Qwen3) with dv=dk=128d_v=d_k=128 (≈0.24%≈0.24\%–0.30%0.30\%). The standard configuration uses K=4K=4; Table 2 uses K=1K=1 for the controlled comparison. Training samples 1 FPS with write cap T=64T=64 and buffer B. Evaluation protocol. Evaluation spans six benchmarks: LongVideoBench (LVB) (Wu et al., 2024), MLVU (Zhou et al., 2025), Video-MME (VMME) (Fu et al., 2025), EgoSchema (EgoS) (Mangalam et al., 2023), MVBench (MVB) (Li et al., 2024a), and LVBench (LVBN) (Wang et al., 2024). Models predict via one-token constrained decoding; we report per-benchmark accuracy and macro Avg. Per §3.3, evaluation ingests a question-agnostic 1 FPS stream (T≤240T≤240). Matched controls fix the backbone, prompt template, decoder budget B∈16,32,64,90B∈\16,32,64,90\, resolution cap, and decoding rule; Base (B=240B=240) is a high-budget reference. Offline mode writes the stream once; streaming mode ingests two-frame chunks carrying state (S,c)(S,c) and answers at stream end (detailed in Appendix A). 4.2 RQ1: Performance Comparison Table 1: Public-benchmark comparison (%). External systems follow official configurations (Appendix A). Base VLMs and PReM use matched B=90B=90 on frozen backbones. Subscripts denote paired gains over Base. Bold marks top performance within each regime. Method (Backbone) Tuning / Budget LVB MLVU VMME EgoS MVB LVBN Avg Offline Ingestion (Specialized long-video architectures) LLaMA-VID (Vicuna-7B) Full-FT (1 FPS) 28.27 35.26 30.11 27.20 32.78 25.89 29.92 VideoLLaMA2 (Qwen2-7B) Full-FT (B=16B=16) 48.54 56.92 54.59 61.20 55.43 36.73 52.24 LLaVA-Video (Qwen2-7B) SFT (B=64B=64) 59.39 70.02 63.33 58.20 59.60 42.41 58.82 LongVILA (Qwen2-7B) Long-FT (B=32/256B=32/256) 55.27 65.33 60.11 72.20 64.65 40.99 59.76 LongVU (Qwen2-7B) SFT (0.5 FPS) 52.80 65.06 56.30 70.00 64.43 37.70 57.72 StreamChat (LongVA-7B) Train-free (20%) 46.00 54.30 46.15 47.20 45.73 35.31 45.78 ReKV (LLaVA-OV-7B) Train-free (0.5 FPS) 56.17 67.03 57.67 60.80 55.43 39.77 56.15 Flash-VStream (Qwen2-VL-7B) LoRA (B=240B=240) 52.06 62.71 58.19 69.60 60.42 38.22 56.87 Generalist base VLMs & PReM (matched B=90B=90) Qwen2.5-VL-3B Frozen (B=90B=90) 52.13 63.45 61.26 61.20 57.92 40.41 56.06 PReM (Qwen2.5-VL-3B) Plug-in (B=90B=90) 55.27+3.14 65.47+2.02 61.30+0.04 61.20 61.35+3.43 43.12+2.71 57.95+1.89 Qwen3-VL-8B Frozen (B=90B=90) 60.88 70.94 67.56 72.60 65.90 45.32 63.87 PReM (Qwen3-VL-8B) Plug-in (B=90B=90) 63.05+2.17 71.22+0.28 67.87+0.31 74.20+1.60 67.10+1.20 45.76+0.44 64.87+1.00 Streaming End-of-Stream (Chronological Ingestion, native budgets) MovieChat (Vicuna-7B) Train-free (128×8128×8) 21.17 28.55 26.44 20.40 29.38 23.56 24.92 StreamChat (LongVA-7B) Train-free (20%) 46.00 53.20 46.63 49.00 45.68 34.41 45.82 ReKV (LLaVA-OV-7B) Train-free (0.5 FPS) 56.17 67.03 57.67 60.80 55.43 39.83 56.16 Flash-VStream (Qwen2-VL-7B) LoRA (B=240B=240) 52.28 60.55 55.89 70.40 57.62 37.19 55.66 Generalist base VLMs & PReM (matched B=90B=90) Qwen2.5-VL-3B Frozen (B=90B=90) 54.75 63.59 59.63 61.40 61.20 41.77 57.06 PReM (Qwen2.5-VL-3B) Plug-in (B=90B=90) 56.54+1.79 65.52+1.93 61.30+1.67 61.20-0.20 61.40+0.20 42.93+1.16 58.15+1.09 Qwen3-VL-8B Frozen (B=90B=90) 61.71 69.75 67.22 72.40 66.05 42.61 63.29 PReM (Qwen3-VL-8B) Plug-in (B=90B=90) 62.15+0.44 71.22+1.47 67.87+0.65 74.20+1.80 67.13+1.08 45.76+3.15 64.72+1.43 Table 1 provides system-level context from specialized long-video systems at native budgets and measures gains against matched-budget frozen base VLMs (B=90B=90). PReM remains competitive with specialized long-video systems under their native configurations. Across both backbones, PReM provides consistent gains over paired frozen bases: offline Avg rises by +1.89%+1.89\% (3B) and +1.00%+1.00\% (8B); streaming Avg rises by +1.09%+1.09\% (3B) and +1.43%+1.43\% (8B). In streaming, PReM (Qwen3-VL-8B) reaches 64.72% Avg while training only 27.5M parameters (<0.30%<0.30\%). 4.3 RQ2: Budget Scaling and Generalization Budget scaling and backbone transfer. Figure 4 shows PReM outperforming frozen baselines across budgets B∈16,32,64,90B∈\16,32,64,90\. On Qwen2.5-VL-3B, gains peak under temporal sparsity (B=16B=16): offline Avg rises by +3.06% (50.39% to 53.45%) and streaming Avg by +1.45%. On Qwen3-VL-8B, steady gains reach +2.00% offline (B=64B=64) and +1.58% in streaming (B=16B=16). At B=90B=90, margins persist (+1.89%+1.89\% and +1.00%+1.00\%). Gains transfer to LLaVA-Video-7B (+0.90%+0.90\% offline at B=16B=16; +0.20+0.20–+1.29%+1.29\% streaming; Table 6), confirming generality. Figure 4: Horizontal axis: PReM−Base PReM-Base Avg gain; vertical axis: visual budget B∈16,32,64,90B∈\16,32,64,90\. Open/filled markers denote Base/PReM. Full breakdowns appear in Appendix Tables 5 and 6. Figure 5: Duration-stratified PReM−Base PReM-Base gains on Qwen2.5-VL-3B (B=16B=16, K=4K=4, Tmem=16T_mem=16; Appendix B.1). Error bars: 95% paired bootstrap CIs (20k); labels: Base→PReMBase→ PReM accuracy. Generalization across native video durations. Figure 5 stratifies the four-slot diagnostic run by native video duration. On Video-MME, gains increase monotonically with video length: +0.67%+0.67\% on Short, +1.78%+1.78\% on Medium, and +1.89%+1.89\% on Long videos. On LVBench, the advantage widens to +5.61% on videos of at least 60 minutes, with the 95% bootstrap CI strictly above zero. This diagnostic run shows its largest gain in the longest LVBench group. 4.4 RQ3: Ablations and Training Robustness Diagnostic ablations. Figure 6 reports component ablations on Qwen2.5-VL-3B (B=16B=16) under a shared experimental setup. Full PReM reaches 53.45% Avg; K-only and V-only variants reach 51.66% and 51.92%, while write-gate variants span 52.03%–52.66%. The zero-buffer probe (B=0B=0) reaches 38.33%, +3.51+3.51 above the vision-free Base. Figure 6: Diagnostic ablations on Qwen2.5-VL-3B (offline B=16B=16). Horizontal axis: Avg accuracy (%). (a) Steering target interfaces. (b) Write gating terms against Full PReM (dashed line). (c) Zero-buffer probe (B=0B=0) isolating recurrent retention against B=16B=16 Base reference. Training efficiency and horizon generalization. Figure 9a–b sweeps writer cap T against full-horizon evaluation (T=240,B=16T=240,B=16). Even at T=16T=16, enabling S boosts offline Avg by +2.17+2.17 over the clean Base, reaching 52.56%. The selected T=64T=64 checkpoint reaches 53.45% offline and 53.46% streaming at B=16B=16 (Table 5), while using 73.3% fewer writer steps than T=240T=240. Thus PReM learns effective recurrent dynamics from short horizons. Sensitivity. We evaluate 36 configurations and a clean Base reference across eight hyperparameters (Table 10; Figure 11). Training and capacity sweeps yield 51.29%–52.76% Avg, with the slot sweep peaking at K=4K=4. Inference scales αrun=0.5/1.0 _run=0.5/1.0 yield 52.26%/51.74%; larger scales 1.5/2.01.5/2.0 reduce Avg to 48.79%/43.56%. Multi-seed results appear in Appendix B.2. 4.5 RQ4: Efficiency and Task Analysis Controlled streaming and task advantages. Table 2 compares the three memory paradigms from Sec. 1 on Qwen2.5-VL-3B at B=16B=16 (details in Appendix A.4). Parameter-matched PReM (K=1K=1) achieves top Avg (53.46%), outperforming Base (52.01%), LoRA (51.21%), Token-Readout (K=1K=1, 51.33%), and InfiniPot-V (51.73%). InfiniPot-V reaches 61.08% on MVB and 54.48% on VMME, while its EgoSchema accuracy is 50.20% (4.204.20 points below Base), with an 84.24 MiB footprint and 17.44 s ingest latency. In contrast, PReM retains long-horizon evidence (EgoS 54.60%, LVBN 38.48%) with a 0.063 MiB state and zero extra prompt tokens, cutting ingestion time to 7.81 s. Figure 7: Highest-gain task categories on Qwen2.5-VL-3B (offline B=16B=16). Horizontal axis: paired PReM−Base PReM-Base accuracy difference (%) with 95% bootstrap CIs. The complete category breakdown is provided in Appendix Figure 10. Table 2: Controlled streaming mechanism comparison on Qwen2.5-VL-3B. Evaluated under matched input streams (T=240,B=16T=240,B=16) and parameter budgets. Bold and underline mark top and second-best results per benchmark. Benchmark Accuracy (%) Resource Accounting Method Params (M) LVB MLVU VMME EgoS MVB LVBN Avg State (MiB / Tok.) VRAM (GiB) Ingest (s) Answer (s) Base (Frozen, B=16B=16) 0 52.73 55.68 53.89 54.40 59.98 35.38 52.01 0.000 / 0 7.43 5.64 0.30 + Parameter Fine-tuning (LoRA, r=20r=20) 9.22 49.36 55.45 53.04 54.60 57.98 36.86 51.21 0.000 / 0 7.46 5.64 0.30 + Token-Readout (Memory Bank) 9.07 51.23 55.59 53.48 52.80 58.70 36.15 51.33 0.063 / +10 7.48 9.20 0.35 + InfiniPot-V (KV-Cache Management) 0 52.06 57.10 54.48 50.20 61.08 35.44 51.73 84.236 / 0 7.99 17.44 0.32 + PReM (Prefix Recurrent Memory) 9.07 53.55 59.68 54.11 54.60 60.33 38.48 53.46 0.063 / 0 7.46 7.81 0.38 Figure 8: Qualitative write-read dynamics on EgoSchema. (a) Evidence frames. (b) Write intensity and gating. (c) Margin drop mfull−mablatedm_full-m_ablated under temporal/slot removal, where m is the gold-answer logit minus the largest alternative logit. (d) Readout gain mon−moffm_on-m_off. Single-video case study. Figure 8 examines a four-slot diagnostic checkpoint on an EgoSchema example. For Q1 (querying banner text at 139 s), ablating S3S_3 or S4S_4 reduces the gold margin by 0.750.75, whereas ablating S1S_1 or S2S_2 causes no degradation. Enabling prefix steering shifts the prediction from D (incorrect) to B (gold), illustrating slot-specific contributions. Runtime efficiency. Figure 9c–d summarizes efficiency: peak GPU memory increases by only 0.03 GiB across budgets. Ingestion adds ≈2≈2 s, while answering latency matches Base within 0.08 s (B=16B=16) and 0.24 s (B=90B=90). Across benchmarks, 41.4% of questions share cached (S,c)(S,c) states, amortizing ingestion across repeated queries. Figure 9: Training horizon and cost on Qwen2.5-VL-3B. (a–b) Training writer cap T (evaluation T=240T=240, B=16B=16; Table 9); dashed: zero gain; filled: selected T=64T=64. (c) Peak memory tracks Base: B (16→9016→90) costs +1.86+1.86 GiB, PReM a constant +0.03+0.03 GiB. (d) Answering latency closely tracks Base across visual budgets (≤0.24≤0.24 s difference), while ingestion adds ≈2≈2 s per QA (Table 7). 5 Conclusion PReM shows that a compact recurrent state, steered into a frozen backbone via prefix keys and values, achieves long-horizon video understanding without prompt tokens while tuning only a lightweight module. It accumulates evidence under visual sparsity, enabling write-once, query-many reuse without decoding recurrence. Future work includes extending PReM with spatially grounded memory for embodied agents (Anwar et al., 2024). AI Use Statement Generative AI tools assisted with manuscript critique, methodology and experiment feedback, language editing, LaTeX grammar checking, and text formatting. The authors independently made all methodological, implementation, experimental, and analytical decisions. All AI-assisted text and empirical claims were reviewed against source code, logs, and checkpoints, and the authors take full responsibility for the final content. Reproducibility Statement The main text and appendix provide all details needed to reproduce the reported findings. Section 3 specifies the architecture, read/write interfaces, objectives, and ingestion protocols; Section 4.1 details training configurations, backbones, budgets, and metrics. Further implementation details, sensitivity sweeps, and multi-seed runs appear in Appendix A–B. Code, configs, and checkpoints are available at https://github.com/siruzhong/PReM. Ethics Statement This work studies long-video understanding on public academic benchmarks, training lightweight PReM modules on frozen VLMs without collecting human-participant data or private video assets. Dual-use applications such as surveillance fall outside our scope. Like its backbones, PReM may inherit web-scale biases; deployment should apply appropriate moderation and fairness safeguards. All authors adhere to the ICLR Code of Ethics. References Anwar et al. (2024) A. Anwar, J. Welsh, J. Biswas, S. Pouya, and Y. Chang ReMEmbR: building and reasoning over long-horizon spatio-temporal memory for robot navigation. arXiv preprint arXiv:2409.13682. Cited by: §1, §5. Bai et al. (2025a) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. Cited by: §4.1. Bai et al. (2025b) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. Cited by: §4.1. Behrouz et al. (2025) A. Behrouz, P. Zhong, and V. Mirrokni Titans: learning to memorize at test time. In Advances in Neural Information Processing Systems, Cited by: §2. Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. Gonzalez Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich RT-2: vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. Cited by: §1. Caba Heilbron et al. (2015) F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles ActivityNet: a large-scale video benchmark for human activity understanding. In IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §A.2. Chen et al. (2026) J. Chen, Z. Zhong, and M. Z. Shou StreamTTT: reconciling real-time perception and long-term memory in streaming vlms. arXiv preprint arXiv:2608.13416. Cited by: §1, §2. Chen et al. (2024) Y. Chen, F. Xue, D. Li, Q. Hu, L. Zhu, X. Li, Y. Fang, H. Tang, S. Yang, Z. Liu, E. He, H. Yin, P. Molchanov, J. Kautz, L. Fan, Y. Zhu, Y. Lu, and S. Han LongVILA: scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188. Cited by: 4th item, §2. Cheng et al. (2024) Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo, D. Zhao, and L. Bing VideoLLaMA 2: advancing spatial-temporal modeling and audio understanding in video-LLMs. arXiv preprint arXiv:2406.07476. Cited by: 2nd item, §1, §2. Di et al. (2025) S. Di, Z. Yu, G. Zhang, H. Li, T. Zhong, H. Cheng, B. Li, W. He, F. Shu, and H. Jiang Streaming video question-answering with in-context video KV-cache retrieval. In International Conference on Learning Representations, Cited by: 7th item, §1, §2. Diko et al. (2025) A. Diko, T. Wang, W. Swaileh, S. Sun, and I. Patras ReWind: understanding long videos with instructed learnable memory. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1, §2. Fu et al. (2025) C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, C. Shan, R. He, and X. Sun Video-MME: the first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1, §4.1. Grauman et al. (2022) K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, A. Gebreselasie, C. González, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolář, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. Ruiz, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Z. Zhao, Y. Zhu, P. Arbeláez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik Ego4D: around the world in 3,000 hours of egocentric video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1. He et al. (2024) B. He, H. Li, Y. K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S. Lim MA-LMM: memory-augmented large multimodal model for long-term video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1, §2. Huang et al. (2025) Z. Huang, X. Li, J. Li, J. Wang, X. Zeng, C. Liang, T. Wu, X. Chen, L. Li, and L. Wang Online video understanding: OVBench and VideoChat-Online. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §2. Katharopoulos et al. (2020) A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret Transformers are RNNs: fast autoregressive transformers with linear attention. In International Conference on Machine Learning, Cited by: §2. Kim et al. (2025) M. Kim, K. Shim, J. Choi, and S. Chang InfiniPot-V: memory-constrained KV cache compression for streaming video understanding. In Advances in Neural Information Processing Systems, Cited by: §1, §2. Lei et al. (2026) J. Lei, D. Zhang, J. Li, W. Wang, K. Fan, X. Liu, Q. Liu, X. Ma, B. Chen, and S. Poria δ-mem: efficient online memory for large language models. arXiv preprint arXiv:2605.12357. Cited by: §2. Li et al. (2024a) K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, L. Wang, and Y. Qiao MVBench: a comprehensive multi-modal video understanding benchmark. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §4.1. Li et al. (2026) X. Li, Y. Wang, J. Yu, X. Zeng, Y. Zhu, H. Huang, J. Gao, K. Li, Y. He, C. Wang, Y. Qiao, Y. Wang, and L. Wang VideoChat-Flash: hierarchical compression for long-context video modeling. In International Conference on Learning Representations, Cited by: §1, §2. Li et al. (2024b) Y. Li, C. Wang, and J. Jia LLaMA-VID: an image is worth 2 tokens in large language models. In European Conference on Computer Vision, Cited by: 1st item, §2. Liu et al. (2026) B. Liu, S. Wang, X. Ding, Z. Wu, B. Du, and X. Li Beyond frame selection: generative latent evidence aggregation for long-video understanding. arXiv preprint arXiv:2607.28516. Cited by: §1. Liu et al. (2024) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12. Cited by: §1. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled Weight Decay Regularization. In International Conference on Learning Representations, Cited by: §A.1. Mangalam et al. (2023) K. Mangalam, R. Akshulakov, and J. Malik EgoSchema: a diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems, Cited by: §1, §4.1. Ning et al. (2025) Z. Ning, G. Liu, Q. Jin, C. Li, W. Ding, M. Guo, and J. Zhao LiveVLM: efficient online video understanding via streaming-oriented KV cache and retrieval. arXiv preprint arXiv:2505.15269. Cited by: §1. Pătrăucean et al. (2023) V. Pătrăucean, L. Smaira, A. Gupta, A. Recasens Continente, L. Markeeva, D. Banarse, S. Koppula, J. Heyward, M. Malinowski, Y. Yang, C. Doersch, T. Matejovicova, Y. Sulsky, A. Miech, A. Frechette, H. Klimczak, R. Koster, J. Zhang, S. Winkler, Y. Aytar, S. Osindero, D. Damen, A. Zisserman, and J. Carreira Perception Test: a diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems, Cited by: §A.2. Qin et al. (2025) M. Qin, X. Liu, Z. Liang, Y. Shu, H. Yuan, J. Zhou, S. Xiao, B. Zhao, and Z. Liu Video-XL-2: towards very long-video understanding through task-aware KV sparsification. arXiv preprint arXiv:2506.19225. Cited by: §1, §2. Ramsauer et al. (2021) H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter Hopfield networks is all you need. In International Conference on Learning Representations, Cited by: §2. Schlag et al. (2021) I. Schlag, K. Irie, and J. Schmidhuber Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, Cited by: §2. Schmidhuber (1992) J. Schmidhuber Learning to control fast-weight memories: an alternative to dynamic recurrent networks. Neural Computation 4. Cited by: §2. Shen et al. (2025) X. Shen, Y. Xiong, C. Zhao, L. Wu, J. Chen, C. Zhu, Z. Liu, F. Xiao, B. Varadarajan, F. Bordes, Z. Liu, H. Xu, H. J. Kim, B. Soran, R. Krishnamoorthi, M. Elhoseiny, and V. Chandra LongVU: spatiotemporal adaptive compression for long video-language understanding. In International Conference on Machine Learning, Cited by: 5th item, §1, §2. Sigurdsson et al. (2016) G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta Hollywood in homes: crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, Cited by: §A.2. Song et al. (2024) E. Song, W. Chai, G. Wang, Y. Zhang, H. Zhou, F. Wu, H. Chi, X. Guo, T. Ye, Y. Zhang, Y. Lu, J. Hwang, and G. Wang MovieChat: from dense token to sparse memory for long video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: 9th item, §1, §2. Tang et al. (2019) Y. Tang, D. Ding, Y. Rao, Y. Zheng, D. Zhang, L. Zhao, J. Lu, and J. Zhou COIN: a large-scale dataset for comprehensive instructional video analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1. Wang et al. (2024) W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, X. Gu, S. Huang, B. Xu, Y. Dong, M. Ding, and J. Tang LVBench: an extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035. Cited by: §4.1. Wang et al. (2025) Y. Wang, Y. Song, C. Xie, Y. Liu, and Z. Zheng VideoLLaMB: long streaming video understanding with recurrent memory bridges. In IEEE/CVF International Conference on Computer Vision, Cited by: §1, §2. Wu et al. (2024) H. Wu, D. Li, B. Chen, and J. Li LongVideoBench: a benchmark for long-context interleaved video-language understanding. In Advances in Neural Information Processing Systems, Cited by: §1, §4.1. Xiao et al. (2021) J. Xiao, X. Shang, A. Yao, and T. Chua NExT-QA: next phase of question-answering to explaining temporal actions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §A.2. Xiong et al. (2025) H. Xiong, Z. Yang, J. Yu, Y. Zhuge, L. Zhang, J. Zhu, and H. Lu Streaming video understanding and multi-round interaction with memory-enhanced knowledge. In International Conference on Learning Representations, Cited by: 6th item. Xu et al. (2024) M. Xu, M. Gao, Z. Gan, H. Chen, Z. Lai, H. Gang, K. Kang, and A. Dehghan SlowFast-LLaVA: a strong training-free baseline for video large language models. arXiv preprint arXiv:2407.15841. Cited by: §1, §2. Yang et al. (2025) Y. Yang, Z. Zhao, S. N. Shukla, A. Singh, S. K. Mishra, L. Zhang, and M. Ren StreamMem: query-agnostic KV cache memory for streaming video understanding. arXiv preprint arXiv:2508.15717. Cited by: §1, §2. Yu et al. (2019) Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao ActivityNet-QA: a dataset for understanding complex web videos via question answering. In AAAI Conference on Artificial Intelligence, Cited by: §A.2. Zhang et al. (2025) H. Zhang, Y. Wang, Y. Tang, Y. Liu, J. Feng, and X. Jin Flash-VStream: efficient real-time understanding for long video streams. In IEEE/CVF International Conference on Computer Vision, Cited by: 8th item, §1, §2. Zhang et al. (2024) Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li LLaVA-Video: video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. Cited by: 3rd item, §A.2, Table 3, §4.1. Zhou et al. (2025) J. Zhou, Y. Shu, B. Zhao, B. Wu, Z. Liang, S. Xiao, M. Qin, X. Yang, Y. Xiong, B. Zhang, T. Huang, and Z. Liu MLVU: benchmarking multi-task long video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §4.1. Zhou et al. (2018) L. Zhou, C. Xu, and J. J. Corso Towards automatic learning of procedures from web instructional videos. In AAAI Conference on Artificial Intelligence, Cited by: §A.2. Appendix Contents A Reproducibility and Implementation Details p. A B Complete Experimental Results and Analyses p. B C Mathematical Notation Reference p. C D Limitations and Future Work p. D E LLM Usage p. E Appendix A Reproducibility and Implementation Details A.1 Training Configuration and Hyperparameters All PReM variants utilize the bounded visual buffer formulation and spatially mean-pooled temporal feature projections (§3.1). The vision encoder and language model backbone remain frozen throughout training. We optimize solely PReM modules for one epoch using AdamW (Loshchilov and Hutter, 2019) (β1=0.9 _1=0.9, β2=0.95 _2=0.95, weight decay 0.010.01) with gradient norm clipping at 1.01.0. The peak learning rate is set to 2×10−42× 10^-4 for Qwen2.5-VL-3B and 10−410^-4 for Qwen3-VL-8B, scheduled with a linear warmup for the first 3%3\% of steps followed by cosine annealing to zero in bfloat16 mixed precision. Each training step processes an individual video-QA pair (per-GPU batch size of 1 without gradient accumulation). Distributed Data Parallelism (DDP) scales training across 8 NVIDIA A100 (80GB) GPUs, completing in ≈3.5≈3.5 hours for Qwen2.5-VL-3B and ≈6.5≈6.5 hours for Qwen3-VL-8B. The recurrent state (S,c)(S,c) is initialized to zero and rebuilt independently per video instance. Videos are sampled uniformly at 1 FPS with frame resolutions dynamically constrained to at most 200,704 pixels (≈448×448≈448× 448). The training write cap is fixed to T=64T=64, while the decoder prompt draws a visual buffer of B frames from the identical temporal sequence. Trainable parameter counts are 9.07M for Qwen2.5-VL-3B and 27.48M for Qwen3-VL-8B, corresponding to ≈0.24%≈0.24\% and ≈0.30%≈0.30\% of the base architectures. The fp32 recurrent state occupies 4K(dvdk+1)4K(d_vd_k+1) bytes: 262,160 bytes (≈256≈256 KiB) for the standard K=4K=4 configuration, and 65,540 bytes (≈64≈64 KiB) for the K=1K=1 controls in Table 2. A.2 Training Data and Decontamination Protocol The training corpus is a curated compact subset of LLaVA-Video-178K (Zhang et al., 2024), sourced from NextQA (Xiao et al., 2021), ActivityNet (Caba Heilbron et al., 2015; Yu et al., 2019), Charades (Sigurdsson et al., 2016), YouCook2 (Zhou et al., 2018), the Perception Test (Pătrăucean et al., 2023), and public YouTube video clips. Training uses 6,430 unique videos and 17,849 question-answer pairs from LLaVA-Video-178K (≈3.6%≈3.6\% of videos and ≈1.4%≈1.4\% of QA pairs). The full repository comprises 178K videos and 1.3M instruction pairs. Decontamination. We filter training IDs that match the six evaluation benchmarks: LongVideoBench, MLVU, Video-MME, EgoSchema, MVBench, and LVBench. Decontamination filters out any training sample matching canonical YouTube video IDs, Ego4D universally unique identifiers (UUIDs), native benchmark keys, and Charades clip IDs reused by MVBench. Filename collisions across verified independent sources are treated as distinct videos. Table 3 details the resulting source composition, duration distribution, and sample counts. Table 3: Training split composition after sampling and video-level decontamination. Filtered from LLaVA-Video-178K (Zhang et al., 2024). Source and duration blocks summarize the retained split (6,430 unique videos and 17,849 QA pairs) with zero evaluation benchmark overlap. Split Category Primary Task / Domain Videos Share (%) QA Pairs Share (%) QA / Vid (a) Source Composition NextQA Causal & temporal reasoning 2,886 44.9 5,785 32.4 2.0 ActivityNet Long human action understanding 1,325 20.6 3,207 18.0 2.4 YouTube Open-domain video reasoning 804 12.5 3,361 18.8 4.2 Charades Indoor daily activity interaction 543 8.4 2,371 13.3 4.4 Perception Test Fine-grained spatio-temporal perception 557 8.7 1,787 10.0 3.2 YouCook2 Procedural recipe & step grounding 315 4.9 1,338 7.5 4.2 (b) Video Duration Distribution 0–30 s Short clips (rapid actions) 2,135 33.2 6,013 33.7 2.8 30–60 s Medium clips (event transitions) 2,267 35.3 6,468 36.2 2.9 1–2 min Extended sequences (multi-stage) 1,382 21.5 3,379 18.9 2.4 2–3 min Long horizon (narrative/procedural) 646 10.0 1,989 11.1 3.1 Total Retained Multi-domain multimodal instruction 6,430 100.0 17,849 100.0 2.8 A.3 Baseline Implementation and Reproduction Protocol To ensure rigorous system comparisons in Table 1, we reproduce or benchmark external long-video architectures using their official open-source repositories under their recommended inference environments and official checkpoints: • LLaMA-VID (Li et al., 2024b): Evaluated using the official Vicuna-7B checkpoint with 1 FPS uniform sampling and dual-token frame representation (generating 2 tokens per frame). • VideoLLaMA2 (Cheng et al., 2024): Evaluated using the official Qwen2-7B base checkpoint at B=16B=16 uniformly sampled frames. • LLaVA-Video (Zhang et al., 2024): Evaluated using the official 7B SFT checkpoint with B=64B=64 frames under official prompt templates. • LongVILA (Chen et al., 2024): Evaluated using the official long-context adapted Qwen2-7B checkpoint under official budget configs (B=32B=32 or B=256B=256 according to benchmark specifications). • LongVU (Shen et al., 2025): Evaluated using the official Qwen2-7B SFT checkpoint with cross-frame spatio-temporal reduction at 0.5 FPS (and 2 FPS on short-video MVBench). • StreamChat (Xiong et al., 2025): Evaluated training-free atop LongVA-7B, applying dynamic token pruning to retain 20% of source visual frames. • ReKV (Di et al., 2025): Evaluated training-free on LLaVA-OneVision-7B at 0.5 FPS, using offline KV-cache construction with dynamic top-64 KV-chunk retrieval. • Flash-VStream (Zhang et al., 2025): Evaluated using the official Qwen2-VL-7B LoRA checkpoint with a multi-resolution streaming memory buffer over B=240B=240 ingested frames. • MovieChat (Song et al., 2024): Evaluated training-free on Vicuna-7B with sliding-window temporal consolidation (128 video fragments × 8 frames). • Generalist Base VLMs: Qwen2.5-VL-3B, and Qwen3-VL-8B are evaluated zero-shot via official implementations under greedy one-token constrained decoding. A.4 Controlled Same-Backbone Mechanisms Table 2 uses Qwen2.5-VL-3B with the registered T=240T=240, B=16B=16 streaming protocol, bfloat16 eager attention, and one-token constrained decoding. The backbone, resolution cap, frame sampling, evaluation manifests, and answer interface are fixed across rows. Base and LoRA consume the same deterministic B-frame reservoir; Token-Readout and PReM additionally build a recurrent state from the sampled stream. Token-Readout and PReM both use K=1K=1, matching memory capacity in this controlled comparison. The adapted controls use the same de-duplicated 6,430-video, 17,849-QA training split as the main experiments (seed 13) with no development holdout. • LoRA: Freezes the vision encoder and language backbone except for self-attention projections qprojq_proj, kprojk_proj, vprojv_proj, oprojo_proj across all 36 transformer layers (rank r=20r=20, α=40α=40, dropout 0, no bias adaptation; 9,216,000 trainable parameters). Trained for one epoch with AdamW at learning rate 2×10−42× 10^-4, 3% warmup, global batch size 8. LoRA adds no prompt tokens and has no recurrent writer. • Token-Readout: Reuses the exact same recurrent writer and trainable capacity as PReM (K=1,dv=dk=128K=1,d_v=d_k=128, 4 layer groups, writer cap 128 tokens, λpred=0.2 _pred=0.2, 4 prediction targets; 9,070,982 trainable parameters), trained with the same optimizer schedule and cross-entropy plus prediction MSE. Instead of prefix K/V steering, the slot readout projects into 2K(1+4)=102K(1+4)=10 continuous tokens via global and group-specific heads, RMS-normalized to mean prompt norm and inserted at the prompt/answer boundary (labels masked, RoPE positions shifted). • InfiniPot-V: Evaluated using the official InfiniPot-V algorithm adapted to Qwen2.5-VL-3B eager attention without fine-tuning, dynamically maintaining top-k informative KV caches across ingested chunks up to the target memory budget while evicting redundant spatiotemporal KV entries. Cross-Family Training Details for LLaVA-Video-7B. For LLaVA-Video-7B transfer (Table 6), PReM modules (K=4,dv=dk=128K=4,d_v=d_k=128, grouping G=1G=1) are trained for one epoch on the identical decontaminated training split using AdamW with peak learning rate 1×10−41× 10^-4 (3% linear warmup, cosine decay), global batch size 8, and bfloat16 precision on 8×A100 GPUs (≈5.5≈5.5 hours). Trainable parameters total 18.34M (≈0.26%≈0.26\% of the 7B backbone). The vision backbone and language layers remain completely frozen. A.5 Decoder Token Order and Attention Interface The decoder keeps the frozen backbone chat template. For Qwen2.5-VL and Qwen3-VL, the user turn places the video sequence immediately prior to the question and options; the processor expands video placeholders into the B-frame visual token sequence. LLaVA-Video-7B employs an equivalent image-token expansion. The resulting sequence structure is: Instruction / System ⟶ B-Frame Visual Tokens ⟶ Question & Options ⟶ Answer Token Text Tokens ⟶ Visual Patch Tokens ⟶ Text Prefix Tokens ⟶ Constrained Decoding Visual token spans are identified via backbone video/image placeholder indices. The steering mask ℳprefixM_prefix is the complement of this visual span, intersected with the valid attention mask and restricted to prompt prefix positions preceding the answer generation token. Writer inputs xtx_t are spatially mean-pooled visual features extracted from the frozen vision encoder, outside the decoder sequence. PReM executes natively within the backbone causal self-attention kernel. The implementation adds the computed prefix modulations ΔKi(ℓ) K_i^( ) and ΔVi(ℓ) V_i^( ) to the outputs of the frozen K/V projection hooks: K~i(ℓ)=WK(ℓ)hi(ℓ)+ΔKi(ℓ),V~i(ℓ)=WV(ℓ)hi(ℓ)+ΔVi(ℓ), K_i^( )=W_K^( )h_i^( )+ K_i^( ), V_i^( )=W_V^( )h_i^( )+ V_i^( ), For Qwen3-VL, Q/K RMSNorm follows the projection hooks; RoPE then rotates Q/K before attention and cache storage. Qwen2.5-VL omits Q/K RMSNorm. Consequently, Qwen3-VL’s effective key perturbation is nonlinear, while its value perturbation remains additive. Visual tokens, padding tokens, and answer positions are unsteered (di=0d_i=0). During autoregressive generation, modulation operates exclusively during prefill; the decoding cache directly serves the steered prefix representations without recomputation. A.6 State Initialization, Constants, and Training Batch Table 4 documents the initialization values and fixed hyperparameters for Equations 4–6. For every video, (S,c)(S,c) starts at zero. Streaming end-of-stream evaluation carries (S,c)(S,c) across consecutive two-frame chunks and answers after video completion. Over n steps without writes, each slot retains a factor σ(fm)nσ(f_m)^n of its state, while confidence decays by γcn _c^n. At initialization (fm=4f_m=4), state retention after 240 steps is approximately 1.3%; training learns fmf_m. The extra bt,mb_t,m factor discounts unstable writes in the confidence update; the 1/K1/K factor in Eq. 7 damps the routed steering amplitude as slot count increases. The forget logit fmf_m is learned via backpropagation, whereas the write step size ηw _w, confidence decay γc _c, and read-router temperature τ are held constant. The write router operates with an untempered softmax, while temperature τ=1.0τ=1.0 governs the read router in Equation 6. The auxiliary load-balancing weight is λbal=0.10 _bal=0.10 by default (swept in Table 10). Main checkpoints use K=4K=4 and dv=dk=128d_v=d_k=128; the slot and layer-group sweeps vary K and G, respectively. Table 4: Initialization and operational constants for PReM. The standard configuration uses K=4K=4 and dv=dk=128d_v=d_k=128. Recurrent states reset independently per video instance; parameters are optimized via AdamW or maintained as fixed constants. Default linear weights and biases use (−1/din,1/din)U(-1/ d_in,1/ d_in). LayerNorm uses unit scale, zero bias, and ϵLN=10−5 _LN=10^-5. Quantity / Symbol Initialization / Value Operational Status Recurrent Memory State Variables Matrix state S∈ℝK×dv×dkS ^K× d_v× d_k Zero tensor (0) Update (Eq. 4) Confidence vector c∈[0,1]Kc∈[0,1]^K Zero vector (0) Update (Eq. 4) Trainable Model Parameters (AdamW Optimization) Forget logit fmf_m 4.04.0 (σ(fm)≈0.982σ(f_m)≈ 0.982) Trainable Slot address transform AmA_m I+0.02(0,1)I+0.02\,N(0,1) Trainable Visual projections Wk,WvW_k,W_v dk×dmodeld_k\!×\!d_model and dv×dmodeld_v\!×\!d_model; rectangular near-identity +0.01(0,1)+0.01\,N(0,1) Trainable Write routing rwr_w Linear(dk,Kd_k,K), default initialization Trainable Update stability WbW_b Linear(dk,Kd_k,K), default initialization Trainable Visual salience ϕφ Linear(dmodel,⌊dmodel/4⌋d_model, d_model/4 )–GELU–Linear(⌊dmodel/4⌋,1 d_model/4 ,1), default initialization Trainable Read conditioning WcW_c dmodel×dmodeld_model\!×\!d_model, bias-free; default initialization Trainable Read projection WrW_r dk×dmodeld_k\!×\!d_model, copied from the initialized WkW_k at construction Trainable Query–value alignment WqvW_qv dv×dkd_v\!×\!d_k, bias-free; default initialization Trainable Read router R Linear(3dv,dv3d_v,d_v)–GELU–Linear(dv,1d_v,1); 3dv2+2dv+13d_v^2+2d_v+1 parameters (49,409 at dv=128d_v=128), default initialization Trainable Steering projection heads Hg,mk,vH_g,m^k,v 10−4(0,1)10^-4N(0,1) Trainable Base steering scale α α logαtrain _train Trainable Prediction branch (Pq,Pp,Fpred)(P_q,P_p,F_pred) dvdmodel+4dv2+7dvd_vd_model+4d_v^2+7d_v; PqP_q near-identity +0.01(0,1)+0.01\,N(0,1), Pp/FpredP_p/F_pred default Linear/LayerNorm init; 328,576 for Qwen2.5-VL-3B Learned only when λpred>0 _pred>0 Fixed Hyperparameters (Frozen Architectural Constants) Base write step size ηw _w 1.01.0 Frozen constant Confidence momentum γc _c 0.950.95 Frozen constant Read-router temperature τ 1.01.0 Frozen constant A.7 Feature Prediction Head For prediction, we select a contiguous min(Tpred,Tw−1) (T_pred,T_w-1)-step span with the highest detached mean salience, block its writes, and run a second writer pass to obtain a held-out state S−S^-. Let TwT_w be the pooled writer sequence length, x~t=LN(xt)∈ℝdmodel x_t=LN(x_t) ^d_model denote the video-only writer input, q∈ℝdmodelq ^d_model the mean non-visual prefix embedding, and t¯=(t−1)/(Tw−1)∈[0,1] t=(t-1)/(T_w-1)∈[0,1] the normalized temporal position. The prediction read reuses WrW_r, AmA_m, WqvW_qv, and R from Eqs. 5–6, with WrLN(q)W_rLN(q) as its query. The prediction branch reads the held-out state at every writer step and computes rt− r_t^- =Read(S−,qTw)t∈ℝdv, =Read(S^-,q1_T_w)_t ^d_v, (10) ht h_t =rt−+PqLN(q)+Ppt¯+bp, =r_t^-+P_qLN(q)+P_p t+b_p, y^t y_t =Fpred(ht), =F_pred(h_t), Here Pq∈ℝdv×dmodelP_q ^d_v× d_model is bias-free, Pp∈ℝdv×1P_p ^d_v× 1 has bias bp∈ℝdvb_p ^d_v, and FpredF_pred is LNdv→Linear(dv,2dv)→GELU→Linear(2dv,dv)LN_d_v\!→\!Linear(d_v,2d_v)\!→\!GELU\!→\!Linear(2d_v,d_v). The target is the stop-gradient, normalized value projection yt=sg(norm(Wvx~t))y_t=sg(norm(W_v x_t)), and the masked objective is ℒpred=1∑tmt∑t=1Twmt∥norm(y^t)−yt∥22,L_pred= 1 _tm_t _t=1^T_wm_t ( y_t)-y_t _2^2, (11) where mtm_t is the selected span mask. The loss is evaluated per video and averaged across the batch; sequences with Tw≤1T_w≤ 1 contribute zero. Thus the head has input and output shape [N,Tw,dv][N,T_w,d_v] for batch size N; its only temporal conditioning is the scalar normalized position t¯ t, while query conditioning is broadcast across the TwT_w writer steps. The branch contains dvdmodel+4dv2+7dvd_vd_model+4d_v^2+7d_v trainable parameters (328,576 for Qwen2.5-VL-3B with dmodel=2048,dv=128d_model=2048,d_v=128). It is used only during training and is disabled at test time. A.8 Evaluation Protocols and Benchmark Specifications Evaluation is conducted at 1 FPS with a maximum resolution cap of 200,704 pixels per frame. For videos up to 240 seconds, frames are sampled at 1 FPS; for longer videos, the stream is uniformly sampled across the full temporal horizon up to the maximum ingestion cap of T=240T=240 frames. Both the writer and the decoder share the same frame sample under identical visual resolution caps. Testing evaluates decoder visual budgets B∈16,32,64,90B∈\16,32,64,90\; same-backbone comparisons match B identically. In offline mode, the visual prompt buffer BV_B samples B frames uniformly across the stream, and the writer processes the sequence in a single pass. In streaming end-of-stream mode, frames arrive chronologically in sequential two-frame chunks carrying state (S,c)(S,c) forward while a deterministic uniform B-frame reservoir is maintained; the reservoir contents form the prompt buffer when the stream ends. All benchmarks evaluate via one-token constrained decoding over option letters A,B,C,D,…\A,B,C,D,…\ using official evaluation scripts across 12,261 total questions: LongVideoBench (1,337), MLVU (2,175), Video-MME (2,700), EgoSchema (500), MVBench (4,000), and LVBench (1,549). Appendix B Complete Experimental Results and Analyses This section provides the exhaustive tabular results and fine-grained breakdowns supporting the analyses in Section 4. Tables 5 and 6 present the complete per-benchmark results across visual budgets B∈16,32,64,90B∈\16,32,64,90\ for Qwen and LLaVA-Video backbones (supporting Figure 4). Table 8 documents the individual benchmark accuracies across steering targets (supporting Figure 6a). Table 9 reports per-benchmark accuracies across writer training horizons. Table 7 reports streaming end-of-stream latency and memory. Furthermore, Figure 11 and Table 10 document exhaustive sensitivity sweeps across 36 parameter configurations and a clean Base reference. B.1 Diagnostic Configurations The duration analysis in Figure 5 uses the Tmem=16T_mem=16 checkpoint from Table 10: K=4K=4, G=1G=1, training frame cap 6464, αtrain=αrun=1 _train= _run=1, λbal=0.05 _bal=0.05, λpred=0.1 _pred=0.1, and Tpred=8T_pred=8. Its aggregate accuracies are 54.56% on Video-MME and 38.09% on LVBench. The diagnostic compares the uniform-frame Base with a reservoir-buffer PReM run at B=16B=16 and ingestion cap 240240. The component ablations in Table 8 share the Full PReM setup except for the stated component change. Table 10 varies its stated factor within each sweep. Table 7 reports runtime under T=240T=240 evaluation ingestion from separate profiling runs; accuracies follow Table 5. Latency columns are rounded independently. Figure 10: Exhaustive native task-category accuracy differences (PReM−Base PReM-Base, %) on Qwen2.5-VL-3B at offline B=16B=16. Error bars denote 95% paired bootstrap confidence intervals across question resamples. Benchmarks lacking native task categories (LongVideoBench, EgoSchema) are omitted. Table 5: Matched-budget results on Qwen backbones. Accuracy (%). Each cell shows Base → PReM performance across budgets B. Bold highlights the superior score between paired settings; Δ denotes absolute Avg gain. B=240B=240 is a high-budget Base reference. Budget LVB MLVU VMME EgoS MVB LVBN Avg Δ (a) Qwen2.5-VL-3B (Offline Ingestion) B=16B=16 49.06 → 53.48 54.90 → 59.82 53.11 → 54.19 54.40 → 54.60 56.50 → 60.13 34.34 → 38.48 50.39 → 53.45 ↑ 3.06 B=32B=32 50.86 → 53.33 58.53 → 62.48 57.26 → 57.44 56.80 → 58.60 57.77 → 61.50 37.70 → 40.03 53.15 → 55.56 ↑ 2.41 B=64B=64 51.68 → 55.65 61.47 → 64.23 59.56 → 59.81 60.60 → 61.40 57.88 → 61.30 38.99 → 41.06 55.03 → 57.24 ↑ 2.21 B=90B=90 52.13 → 55.27 63.45 → 65.47 61.26 → 61.30 61.20 → 61.20 57.92 → 61.35 40.41 → 43.12 56.06 → 57.95 ↑ 1.89 Ref (B=240B=240) 53.78 65.61 62.15 62.20 60.50 43.12 57.89 – (b) Qwen2.5-VL-3B (Streaming End-of-Stream) B=16B=16 52.73 → 53.55 55.68 → 59.68 53.89 → 54.11 54.40 → 54.60 59.98 → 60.33 35.38 → 38.48 52.01 → 53.46 ↑ 1.45 B=32B=32 53.40 → 54.53 60.00 → 62.34 56.26 → 57.70 58.40 → 58.60 61.38 → 61.50 37.90 → 40.09 54.56 → 55.79 ↑ 1.23 B=64B=64 54.90 → 56.02 61.66 → 64.18 58.93 → 59.70 60.60 → 60.80 61.08 → 61.33 41.12 → 41.45 56.38 → 57.25 ↑ 0.87 B=90B=90 54.75 → 56.54 63.59 → 65.52 59.63 → 61.30 61.40 → 61.20 61.20 → 61.40 41.77 → 42.93 57.06 → 58.15 ↑ 1.09 (c) Qwen3-VL-8B (Offline Ingestion) B=16B=16 56.84 → 57.52 60.55 → 62.30 60.00 → 60.41 69.00 → 70.00 64.60 → 66.15 38.00 → 38.28 58.16 → 59.11 ↑ 0.95 B=32B=32 58.94 → 59.69 65.15 → 65.29 63.33 → 64.04 70.00 → 73.00 65.38 → 67.33 39.83 → 40.67 60.44 → 61.67 ↑ 1.23 B=64B=64 60.36 → 61.85 68.18 → 70.30 66.11 → 66.96 70.60 → 74.60 65.83 → 67.03 41.83 → 44.16 62.15 → 64.15 ↑ 2.00 B=90B=90 60.88 → 63.05 70.94 → 71.22 67.56 → 67.87 72.60 → 74.20 65.90 → 67.10 45.32 → 45.76 63.87 → 64.87 ↑ 1.00 Ref (B=240B=240) 64.25 75.59 70.19 73.00 65.95 47.39 66.06 – (d) Qwen3-VL-8B (Streaming End-of-Stream) B=16B=16 56.92 → 57.74 59.17 → 62.53 58.41 → 60.52 69.00 → 69.40 64.55 → 66.13 36.99 → 38.22 57.51 → 59.09 ↑ 1.58 B=32B=32 58.56 → 60.58 62.90 → 65.29 62.52 → 64.00 71.40 → 72.40 65.83 → 67.25 40.48 → 40.67 60.28 → 61.70 ↑ 1.42 B=64B=64 62.00 → 61.78 67.26 → 70.34 65.67 → 66.96 72.20 → 74.60 66.03 → 67.00 42.87 → 43.71 62.67 → 64.07 ↑ 1.40 B=90B=90 61.71 → 62.15 69.75 → 71.22 67.22 → 67.87 72.40 → 74.20 66.05 → 67.13 42.61 → 45.76 63.29 → 64.72 ↑ 1.43 Table 6: Cross-family transfer to LLaVA-Video-7B. Accuracy (%). Each cell shows Base → PReM performance across visual budgets B. Bold highlights the superior score between paired settings; Δ denotes absolute Avg gain over Base. Evaluated through budget B=90B=90. Budget LVB MLVU VMME EgoS MVB LVBN Avg Δ (a) LLaVA-Video-7B (Offline Ingestion) B=16B=16 56.32 → 57.52 61.29 → 62.76 59.96 → 60.07 54.20 → 56.00 58.70 → 59.03 38.35 → 38.80 54.80 → 55.70 ↑ 0.90 B=32B=32 58.12 → 58.64 66.57 → 66.76 62.15 → 62.37 58.00 → 58.20 59.90 → 60.48 40.87 → 42.29 57.60 → 58.12 ↑ 0.52 B=64B=64 59.39 → 59.99 70.02 → 70.39 63.33 → 63.48 58.20 → 58.00 59.60 → 60.15 42.41 → 43.58 58.82 → 59.27 ↑ 0.45 B=90B=90 59.69 → 60.66 70.39 → 70.57 63.74 → 63.52 58.40 → 59.00 60.12 → 60.18 45.13 → 45.47 59.58 → 59.90 ↑ 0.32 (b) LLaVA-Video-7B (Streaming End-of-Stream) B=16B=16 55.87 → 56.47 60.46 → 61.61 59.41 → 60.07 53.80 → 54.00 57.43 → 58.83 39.57 → 40.28 54.42 → 55.21 ↑ 0.79 B=32B=32 57.29 → 58.12 65.75 → 66.11 61.00 → 60.96 57.60 → 57.00 58.03 → 58.25 40.48 → 40.87 56.69 → 56.89 ↑ 0.20 B=64B=64 57.29 → 58.79 68.18 → 69.66 62.59 → 63.00 58.20 → 58.40 58.03 → 58.18 41.83 → 43.58 57.69 → 58.60 ↑ 0.91 B=90B=90 58.04 → 59.61 68.55 → 70.57 63.85 → 63.98 59.00 → 59.60 58.05 → 58.90 42.16 → 44.74 58.28 → 59.57 ↑ 1.29 Table 7: Streaming end-of-stream efficiency on Qwen2.5-VL-3B. Evaluated on NVIDIA H20 (1 FPS, batch size 1), supporting Figure 9c–d. Base and PReM share budget B. Ingestion and answering are averaged over all QA pairs, including cache hits with negligible ingestion time; total latency reports mean / p95. Subscripts indicate peak memory overhead; Δ denotes absolute Avg gain. Method B Backbone / Train. Params (M) Peak Mem. (GiB) Ingest / Ans. (s) Mean / p95 (s) Online Avg Δ Base 16 3754.62 / 0.00 7.43 5.64 / 0.30 5.94 / 29.78 52.01 – PReM 16 3754.62 / 9.07 7.46+0.03 7.81 / 0.38 8.19 / 38.11 53.46 ↑ +1.45 Base 32 3754.62 / 0.00 7.83 5.71 / 0.53 6.24 / 30.31 54.56 – PReM 32 3754.62 / 9.07 7.86+0.03 7.83 / 0.64 8.48 / 38.76 55.79 ↑ +1.23 Base 64 3754.62 / 0.00 8.64 5.81 / 0.96 6.77 / 31.44 56.38 – PReM 64 3754.62 / 9.07 8.67+0.03 7.88 / 1.15 9.02 / 40.02 57.25 ↑ +0.87 Base 90 3754.62 / 0.00 9.29 5.89 / 1.31 7.19 / 32.48 57.06 – PReM 90 3754.62 / 9.07 9.32+0.03 7.90 / 1.55 9.45 / 40.95 58.15 ↑ +1.09 Table 8: Diagnostic component ablations on Qwen2.5-VL-3B (offline B=16B=16, %). Steering targets evaluate separately trained heads. All variants share the same experimental setup except for the ablated component. Δ denotes the score drop relative to Full PReM. Zero-buffer probes (B=0B=0) isolate recurrent retention without visual prompts. Variant LVB MLVU VMME EgoS MVB LVBN Avg Δ vs. Full Base (B=16B=16, no memory) 49.06 54.90 53.11 54.40 56.50 34.34 50.39 – Full PReM (K/V modulation) 53.48 59.82 54.19 54.60 60.13 38.48 53.45 – Modulation Interface Q/O modulation 52.66 59.49 53.81 47.00 59.98 36.02 51.49 ↓ 1.96 Key-only (K) modulation 52.28 56.14 53.33 52.00 59.78 36.41 51.66 ↓ 1.79 Value-only (V) modulation 51.91 59.86 53.59 49.80 59.12 37.25 51.92 ↓ 1.53 Write Gating Terms Uniform write routing (πwπ^w) 52.73 59.36 53.96 52.40 59.58 37.96 52.66 ↓ 0.79 w/o Update stability (b) 52.06 57.70 53.48 53.00 59.60 37.70 52.26 ↓ 1.19 w/o Visual salience (z) 51.91 58.53 54.00 50.80 59.73 37.83 52.13 ↓ 1.32 w/o Confidence protection (a) 52.43 60.23 53.70 49.40 59.45 37.44 52.11 ↓ 1.34 w/o Residual novelty (n) 51.98 59.17 53.30 49.60 59.38 38.73 52.03 ↓ 1.42 Zero-Buffer Retention Probes (B=0B=0) PReM (B=0B=0, memory only) 43.40 47.73 40.65 27.80 40.88 29.50 38.33 – Base (B=0B=0, vision-free) 39.27 42.02 37.30 26.80 36.70 26.86 34.82 – Table 9: Writer training budget. Per-benchmark accuracy (%) on Qwen2.5-VL-3B (B=16B=16, evaluated at T=240T=240), supporting Figure 9a–b. Checkpoints use write cap T (αrun=0.75 _run=0.75); clean Base rows provide the protocol-matched reference. Shading and bold highlight the highest Avg per protocol. Training Horizon LVB MLVU VMME EgoS MVB LVBN Avg Δ Offline Ingestion Clean Base 49.06 54.90 53.11 54.40 56.50 34.34 50.39 – Writer T=16T=16 52.58 58.25 53.85 52.80 59.92 37.96 52.56 ↑ +2.17 Writer T=32T=32 52.58 57.75 53.78 52.20 59.98 38.22 52.42 ↑ +2.03 Writer T=64T=64 53.48 59.82 54.19 54.60 60.13 38.48 53.45 ↑ +3.06 Writer T=90T=90 53.18 59.82 54.04 49.80 60.12 37.44 52.40 ↑ +2.01 Writer T=240T=240 52.36 58.99 54.07 52.40 59.52 38.48 52.64 ↑ +2.25 Streaming End-of-Stream Clean Base 52.73 55.68 53.89 54.40 59.98 35.38 52.01 – Writer T=16T=16 51.91 58.11 53.85 52.60 60.15 37.96 52.43 ↑ +0.42 Writer T=32T=32 52.36 58.02 53.52 52.40 60.33 38.09 52.45 ↑ +0.44 Writer T=64T=64 53.55 59.68 54.11 54.60 60.33 38.48 53.46 ↑ +1.45 Writer T=90T=90 53.25 59.68 53.85 50.40 60.23 37.38 52.46 ↑ +0.45 Writer T=240T=240 51.38 59.17 54.00 51.80 59.70 37.96 52.34 ↑ +0.33 Figure 11: One-factor sensitivity sweeps on Qwen2.5-VL-3B (offline B=16B=16, Avg). Markers denote checkpoints or inference configurations; teal and orange highlight extreme values within each sweep. Corresponding per-benchmark numeric breakdowns are tabulated in Table 10. Table 10: Hyperparameter sensitivity on Qwen2.5-VL-3B (offline B=16B=16, %; configurations in Appendix B.1). (a) Training sweeps: αtrain _train, λbal _bal, λpred _pred, and TpredT_pred (writer steps); read-router temperature τ=1τ=1 is fixed. (b) Inference scale αrun _run changes steering without retraining; Base is the clean frozen reference. (c) Capacity sweeps: K slots, training-time pooled-writer cap TmemT_mem (no prompt positions), and G layer-head groups. Shading and bold mark the highest Avg within each sweep. Parameter Setting LVB MLVU VMME EgoS MVB LVBN Avg (a) Training Training steering scale αtrain _train 0.25 52.06 56.87 53.30 53.20 60.00 37.19 52.10 0.50 52.95 57.52 54.07 51.80 59.68 37.44 52.24 0.75 52.95 59.54 54.04 51.40 59.65 38.41 52.67 1.00 51.98 60.18 53.74 48.60 58.73 37.70 51.82 Load-balancing weight λbal _bal 0.00 52.36 60.55 54.15 49.40 59.30 38.28 52.34 0.05 50.93 59.86 54.07 48.60 59.40 37.38 51.71 0.10 53.03 59.54 53.81 51.20 59.20 37.38 52.36 0.20 51.68 59.26 53.85 50.80 59.40 37.31 52.05 Prediction loss weight λpred _pred 0.00 53.33 58.21 54.00 51.80 59.55 38.02 52.48 0.05 53.03 58.16 54.00 52.40 59.88 38.61 52.68 0.10 52.88 57.20 53.89 53.00 59.95 37.64 52.43 0.20 52.21 56.97 53.67 53.00 59.95 37.70 52.25 Masked prediction span TpredT_pred (writer steps) 2 52.80 59.13 53.26 49.60 59.63 37.44 51.98 4 52.95 59.86 53.93 48.80 59.30 37.31 52.03 8 52.13 59.72 53.04 46.20 59.65 36.99 51.29 16 51.68 60.00 52.89 47.20 59.23 36.73 51.29 32 51.91 59.68 52.78 49.40 59.38 37.38 51.75 Parameter Setting LVB MLVU VMME EgoS MVB LVBN Avg (b) Inference Inference steering scale αrun _run Base 49.06 54.90 53.11 54.40 56.50 34.34 50.39 0.5 52.73 59.22 53.78 50.00 59.88 37.96 52.26 1.0 52.51 60.32 53.07 47.20 59.63 37.70 51.74 1.5 49.21 56.87 50.89 39.80 58.73 37.25 48.79 2.0 44.80 47.26 46.19 35.40 55.63 32.09 43.56 (c) Memory capacity Memory slot count K 1 51.76 58.90 54.11 52.60 59.33 37.77 52.41 2 52.43 59.63 54.37 50.20 59.33 37.96 52.32 4 52.80 58.67 54.26 52.20 59.78 38.22 52.65 8 53.25 58.57 53.59 50.00 59.23 38.22 52.14 16 52.13 57.89 53.48 54.20 59.78 37.06 52.42 Pooled-writer cap TmemT_mem 16 52.73 59.31 54.56 49.20 59.78 38.09 52.28 32 52.88 59.36 53.89 52.40 59.88 36.86 52.55 64 53.03 59.68 54.19 49.80 59.85 37.06 52.27 128 52.95 59.26 54.11 51.80 59.53 37.06 52.45 256 52.06 59.40 53.74 50.40 59.03 37.57 52.03 Layer-head groups G 1 53.03 58.99 54.19 49.20 59.50 38.09 52.17 2 52.95 58.48 53.67 50.00 59.33 37.83 52.04 4 53.78 59.22 53.96 52.60 59.68 37.31 52.76 8 52.73 58.99 53.11 51.80 59.95 37.06 52.27 16 52.13 58.62 53.00 51.40 59.90 36.28 51.89 B.2 Multi-seed Training Stability To verify optimization robustness, Table 11 reports fine-grained per-benchmark accuracy alongside aggregate statistics across three independent training runs (seeds 13, 17, 23) at offline B=16B=16. Seed 13 reuses the Table 5 main-run checkpoint. On Qwen2.5-VL-3B, mean accuracy is 53.38%53.38\% with sample standard deviation (SD) 0.06%0.06\%, a +2.99±0.06%+2.99± 0.06\% gain over Base (50.39%); on Qwen3-VL-8B, the mean is 59.11%59.11\% (SD=0.03%SD=0.03\%), a +0.95±0.03%+0.95± 0.03\% gain over Base (58.16%). Every seed exceeds its paired Base Avg; Table 11 reports the per-benchmark variation. Table 11: Multi-seed evaluation across benchmarks (offline B=16B=16, %). Accuracy and standard deviations across three independent runs (seeds 13, 17, 23) alongside deterministic Base. Method Seed LVB MLVU VMME EgoS MVB LVBN Avg (a) Qwen2.5-VL-3B Base – 49.06 54.90 53.11 54.40 56.50 34.34 50.39 PReM 13 53.48 59.82 54.19 54.60 60.13 38.48 53.45 PReM 17 53.33 59.54 54.15 54.20 60.15 38.67 53.34 PReM 23 53.10 60.64 53.81 54.00 59.90 38.67 53.35 Mean 53.30 60.00 54.05 54.27 60.06 38.61 53.38 SD (±) 0.19 0.57 0.21 0.31 0.14 0.11 0.06 Method Seed LVB MLVU VMME EgoS MVB LVBN Avg (b) Qwen3-VL-8B Base – 56.84 60.55 60.00 69.00 64.60 38.00 58.16 PReM 13 57.52 62.30 60.41 70.00 66.15 38.28 59.11 PReM 17 57.67 62.34 60.44 69.80 66.25 38.35 59.14 PReM 23 57.52 62.21 60.33 69.80 66.12 38.54 59.09 Mean 57.57 62.28 60.39 69.87 66.17 38.39 59.11 SD (±) 0.09 0.07 0.06 0.12 0.07 0.13 0.03 Appendix C Mathematical Notation Reference Table 12 summarizes the mathematical symbols, dimensions, and operational definitions. Table 12: Mathematical notation reference for Prefix-Steered Recurrent Memory. Summarizes the core symbols, tensor dimensions, and operational definitions of PReM across memory updates, prefix steering, and optimization objectives. Symbol Meaning and Operational Scope Visual Input and Recurrent Memory State E(vt)p,xtE(v_t)_p,\ x_t Encoder patch token at position p and spatially pooled visual frame feature at time t S∈ℝK×dv×dkS ^K× d_v× d_k Recurrent associative memory state tensor across K memory slots c∈[0,1]Kc∈[0,1]^K Per-slot write-confidence vector tracking temporal update history B,TB,\ T Decoder prompt visual frame budget and total writer ingestion cap (train T=64T=64; eval T=240T=240) Memory Write Dynamics and Gating System Wk,Wv,AmW_k,W_v,A_m Learnable key/value projection matrices and slot-specific addressing transforms κt,m _t,m Normalized write addressing query for memory slot m at time t et,me_t,m Associative reconstruction residual ut−Smκt,mu_t-S_m _t,m bt,m,nt,mb_t,m,\ n_t,m Update-stability gate and residual-novelty gate activations (Eq. 3) zt,at,mz_t,\ a_t,m Visual-salience gate and confidence-protection gate activations (Eq. 3) gt,mg_t,m Composite write coefficient: πt,mwbt,mnt,mztat,mπ^w_t,mb_t,mn_t,mz_ta_t,m σ(fm),ηw,γcσ(f_m),\ _w,\ _c Learnable slot retention probability, base write step size, and confidence decay rate πwπ^w Softmax write-routing distribution across associative slots Prefix Steering and Read Interface ξi,m(ℓ) _i,m^( ) Layer-wise normalized read query for slot m at prompt token index i ρr,τρ^r,\ τ Normalized read relevance scores and read-router temperature parameter Hg,mk,Hg,mvH_g,m^k,\ H_g,m^v Group- and slot-specific key and value prefix steering projection heads dik,(ℓ),div,(ℓ)d_i^k,( ),\ d_i^v,( ) Additive prefix modulation vectors injected into backbone Key and Value paths WK(ℓ),WV(ℓ)W_K^( ),\ W_V^( ) Frozen language model self-attention Key and Value projection weights α^,αrun,αeff α,\ _run,\ _eff Learnable scale logit, inference-time steering multiplier, and effective steering scale G,TmemG,\ T_mem Attention layer group count and training-time pooled-writer cap Training Objectives and Hyperparameters ℒCEL_CE Primary cross-entropy loss over target response tokens ℒbal,λbalL_bal,\ _bal Router load-balancing loss (uniformity penalty) and auxiliary loss weight ℒpred,λpredL_pred,\ _pred Salient feature reconstruction loss and corresponding loss weight TpredT_pred Masked prediction span measured in pooled writer steps Appendix D Limitations and Future Work Three design choices define the scope of this work: • Fixed Slot Granularity: The recurrent memory uses a fixed number K of associative slots with fixed hidden dimensionality. Dynamic slot allocation or hierarchical multi-scale abstractions would capture complex event boundaries in continuous streams. • Frame Sampling Rate: The architecture samples at 1 FPS, uniformly subsampling longer videos to T=240T=240 frames. Spatial mean-pooling retains frame summaries while discarding intra-frame layout from the recurrent state. Dense action streams at higher native frame rates call for adaptive write filtering. • Evaluation Paradigms: Evaluation targets offline and streaming end-of-stream multiple-choice QA; future work will extend PReM to open-ended dialogue and lifelong streaming agent environments. Appendix E LLM Usage Large language models assisted with manuscript critique, methodology and experiment feedback, language editing, LaTeX grammar checking, and text formatting. The conceptual methodology, mathematical formulation, model implementation, and experimental evaluations were conducted independently by the authors. All AI-assisted text, empirical results, derivations, and factual claims were verified against the source code, training logs, and evaluation checkpoints. The authors assume full responsibility for the contents of this paper.