Paper deep dive
PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits
Namkyung Yoon, Sanghong Kim, Hwangnam Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/3/2026, 2:27:43 AM
Summary
The paper introduces PARALLEL, a reinforcement learning-inspired approach for efficient language model adaptation. It utilizes separate goal-related and uncertainty-related controller signals to dynamically determine the intensity of parameter updates (Skip, Light, or Strong) for each sample in a streaming dataset. This method aims to maximize performance while minimizing adaptation time and energy consumption by avoiding unnecessary updates on low-utility samples, constrained by a cumulative update-mass budget.
Entities (13)
Relation Signals (12)
PARALLEL → comparedwith → Full adaptation
confidence 95% · The main experiments compare PARALLEL with Frozen and Full adaptation.
PARALLEL → evaluatedon → XSUM
confidence 95% · experiments on XSum and CNN/DailyMail show that PARALLEL retains
PARALLEL → evaluatedon → CNN/DailyMail
confidence 95% · experiments on XSum and CNN/DailyMail show that PARALLEL retains
PARALLEL → uses → LoRA
confidence 95% · PARALLEL controls whether and how strongly the adapter is updated... only the LoRA adapter is updated.
PARALLEL → uses → REINFORCE
confidence 92% · A controller trained with one-step REINFORCE uses these signals to determine whether and how strongly to update the adapter
PARALLEL → achievesmetricon → ROUGE-2
confidence 90% · PARALLEL retains 96.9–98.6% of the ROUGE-1 and ROUGE-2 scores
PARALLEL → achievesmetricon → ROUGE-1
confidence 90% · PARALLEL retains 96.9–98.6% of the ROUGE-1... scores
PARALLEL → achievesmetricon → ROUGE-L
confidence 90% · 98.8–98.9% of the corresponding ROUGE-L scores
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent language models achieve strong performance across a variety of tasks, but conventional adaptation applies updates uniformly across training samples regardless of their local update benefit. We propose PARALLEL, a prefrontal-aligned reinforcement inspired approach for language-model learning. Inspired by the complementary roles of goal-related and uncertainty-related control, PARALLEL represents these forms of information as separate controller signals and combines them with the current model representation. A reinforcement-inspired controller assigns sample-dependent update intensity using immediate utility-cost feedback. PARALLEL therefore learns when and how strongly to adapt to each sample, prioritizing beneficial updates while limiting unnecessary parameter changes. PARALLEL uses available updates more efficiently than selective baselines while retaining 94.1--99.2\% of Full-adaptation performance. Beyond multiple-choice reasoning, experiments on XSum and CNN/DailyMail show that PARALLEL retains 96.9--98.6\% of the ROUGE-1 and ROUGE-2 scores achieved by Full adaptation and 98.8--98.9\% of the corresponding ROUGE-L scores. When compared at the same cumulative adaptation time or GPU energy, PARALLEL achieves higher ARC accuracy and exhibits a more stable late-stage adaptation trajectory than Full adaptation in the representative run. These results show that learning when and how strongly to update each sample supports stable and efficient post-deployment stream adaptation while avoiding unnecessary updates.
Tags
Links
- Source: https://arxiv.org/abs/2607.28982v1
- Canonical: https://arxiv.org/abs/2607.28982v1
Trouble viewing inline? Open PDF directly →
Full Text
42,662 characters extracted from source content.
Expand or collapse full text
PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits Namkyung Yoon1 , Sanghong Kim1 , Hwangnam Kim1 Abstract Recent language models achieve strong performance across a variety of tasks, but conventional adaptation applies updates uniformly across training samples regardless of their local update benefit. We propose PARALLEL, a prefrontal-aligned reinforcement inspired approach for language-model learning. Inspired by the complementary roles of goal-related and uncertainty-related control, PARALLEL represents these forms of information as separate controller signals and combines them with the current model representation. A reinforcement-inspired controller assigns sample-dependent update intensity using immediate utility–cost feedback. PARALLEL therefore learns when and how strongly to adapt to each sample, prioritizing beneficial updates while limiting unnecessary parameter changes. PARALLEL uses available updates more efficiently than selective baselines while retaining 94.194.1–99.2%99.2\% of Full-adaptation performance. Beyond multiple-choice reasoning, experiments on XSum and CNN/DailyMail show that PARALLEL retains 96.996.9–98.6%98.6\% of the ROUGE-1 and ROUGE-2 scores achieved by Full adaptation and 98.898.8–98.9%98.9\% of the corresponding ROUGE-L scores. When compared at the same cumulative adaptation time or GPU energy, PARALLEL achieves higher ARC accuracy and exhibits a more stable late-stage adaptation trajectory than Full adaptation in the representative run. These results show that learning when and how strongly to update each sample supports stable and efficient post-deployment stream adaptation while avoiding unnecessary updates. Introduction Recent language models achieve strong performance across a wide range of tasks, including natural language understanding, reasoning, and knowledge-intensive question answering (Zhao et al. 2026; Chang et al. 2024). However, adapting these models to task-specific data still requires repeated parameter updates. Conventional learning methods generally apply updates uniformly across training samples, even though the local benefit of an update can vary across samples. Consequently, repeatedly updating low-utility samples may consume adaptation time and energy without proportional performance improvement (Yoon et al. 2026). This limitation is particularly relevant to post-deployment stream adaptation: a general-purpose language model may require supervised adaptation to a target-task distribution, yet applying full-intensity updates to every incoming labeled sample can accumulate substantial time and energy without proportional performance gains (Hoi et al. 2021). Our objective is therefore to allocate adapter updates across the stream according to their estimated utility, supporting effective target-task adaptation while avoiding unnecessary updates. Existing approaches improve adaptation efficiency through different strategies. Parameter-efficient methods reduce the number of trainable parameters (Hu et al. 2022). Data-selection methods instead prioritize samples based on uncertainty or difficulty (Settles 2009; Paul et al. 2021). However, parameter-efficient adaptation does not control how updates are allocated across samples, while data-selection methods typically apply the same update strength to every selected sample. Consequently, neither approach jointly determines whether an update is needed and how strongly each incoming sample should influence adaptation. To address this problem, we draw inspiration from an organizational principle reported in the human lateral prefrontal cortex. Recent neuroimaging evidence indicates that goal-related and uncertainty-related information is represented in a factorized manner, supporting stable yet flexible learning (Sung et al. 2025). This finding motivates a control structure in which goal-related and uncertainty-related signals are modeled separately and then combined to regulate language-model adaptation. Figure 1: Overview of the proposed PARALLEL framework. Based on this principle, we propose PARALLEL, a Prefrontal-Aligned Reinforcement Approach for Language-Model Learning under Explicit Limits. Rather than applying uniform updates throughout the stream, PARALLEL learns whether and how strongly the model should adapt to each incoming sample, thereby supporting efficient post-deployment adaptation while avoiding unnecessary updates. PARALLEL represents goal-related and uncertainty-related information as separate controller signals and combines them with the current model representation. A controller trained with one-step REINFORCE uses these signals to determine whether and how strongly to update the adapter on each labeled sample. It learns from immediate utility–cost feedback, while a hard cumulative update-mass constraint bounds the total update intensity allocated across the stream. We evaluate PARALLEL on three reasoning and two summarization benchmarks. The main experiments compare PARALLEL with Frozen and Full adaptation. Additional budget-matched comparisons are conducted across all five benchmarks, while cross-backbone evaluation, controller-input ablations, and stream-level measurements of adaptation time and GPU energy are conducted in the benchmark setting. The results show that PARALLEL effectively allocates a bounded update budget and retains most of the performance under Full adaptation across three compact backbones. Accordingly, PARALLEL should be viewed as a mechanism for controlling update allocation rather than accelerating offline fine-tuning. The main contributions are as follows: • We propose PARALLEL, which uses separate goal-related and uncertainty-related signals to perform online LoRA update control through three actions, determining whether and how strongly to update each incoming sample under a cumulative update-mass constraint. • We demonstrate effective budget-constrained adaptation across three reasoning and two summarization benchmarks using multiple language models. The remainder of this paper is organized as follows. Section 2 reviews related work, Section 3 presents PARALLEL, Section 4 reports the experimental evaluation, and Section 5 concludes the paper. Related Work Efficient Adaptation Methods Full-parameter fine-tuning updates all parameters of a pretrained language model and therefore requires substantial memory and computation (Devlin et al. 2019). Parameter-efficient methods instead reduce the number of trainable parameters while preserving the pretrained backbone. LoRA introduces trainable low-rank components while keeping the original model parameters frozen (Hu et al. 2022). In our evaluation, Full adaptation applies the same LoRA configuration to every sample in the adaptation stream. Data-selection strategies determine which samples are used for adaptation under a limited sample budget. Random selection draws a fixed-size subset uniformly at random (Citovsky et al. 2021). Uncertainty-based active learning prioritizes samples on which the current model is least confident (Lewis and Gale 1994; Settles 2009). Its entropy-based variant ranks samples by predictive entropy and selects those with the highest values (Settles 2009). Parameter-efficient methods control which parameters are trained, whereas data-selection methods control which samples are selected. PARALLEL instead processes the labeled adaptation stream sequentially and learns sample-wise update intensity, while a cumulative constraint prevents excessive updates. Reinforcement Learning for Adaptation Control Reinforcement learning has been widely applied to language models to align generated responses with task objectives, human preferences, and instruction-following requirements (Ouyang et al. 2022). In conventional reinforcement learning from human feedback, the language model defines a policy over generated tokens or responses, and reward feedback directly optimizes its generation behavior. PARALLEL instead adopts a reinforcement-inspired controller for adaptation control. For each labeled sample, the controller selects the intensity of the supervised adapter update and learns from immediate utility–cost feedback. The adapter itself is optimized only with the supervised task loss. Prefrontal-Aligned Learning Control Recent neuroimaging evidence suggests that goal-related and uncertainty-related information is represented in a factorized manner within the lateral prefrontal cortex, supporting stable yet flexible learning (Sung et al. 2025). This organization motivates treating information about the current task objective and predictive uncertainty as distinct but complementary signals for learning control. PARALLEL adopts this principle as a computational abstraction. It represents goal-related and uncertainty-related information as separate controller signals and combines them with the current model representation to allocate update intensity across samples. System Design As shown in Figure 1, PARALLEL controls whether and how strongly the adapter is updated for each incoming labeled sample. For each sample, it computes a goal-related signal representing consistency with the target response and an uncertainty-related signal reflecting the reliability of the current prediction. Our PARALLEL controller combines these signals with the current model representation and selects the update intensity. The pretrained language-model backbone remains frozen, while the controller uses the goal-related and uncertainty-related signals to determine whether and how strongly to update the LoRA adapter for each sample. Control Signals in PARALLEL Let =(xt,yt⋆)t=1TD=\(x_t,y_t )\_t=1^T denote the labeled adaptation stream, where T is the total number of adaptation samples, xtx_t is the question or prompt of the t-th sample, and yt⋆y_t is its target response. The language-model backbone with parameters θ remains frozen, while task adaptation is performed through the model with parameters θa _a. PARALLEL observes the labeled samples sequentially and determines whether and how strongly θa _a should be updated for each sample. Before choosing an update action, PARALLEL assesses how well the current model has already learned the labeled sample (xt,yt⋆)(x_t,y_t ). It performs a diagnostic forward pass using the current model parameters without applying an update. The resulting prediction quality serves as goal-related evidence for the controller: poor prediction supports a stronger update, whereas accurate prediction supports a light update or skipping the sample. Since the target response yt⋆y_t is available during supervised adaptation, PARALLEL evaluates it using the standard autoregressive objective adopted in supervised fine-tuning of language models (Chatterjee et al. 2025). Specifically, each target token is evaluated using the input and the preceding ground-truth target tokens. The diagnostic forward pass produces the following length-normalized pre-update target loss: ℓt−=−1Nt∑j=1Ntlogpθ,θa(yt,j⋆∣xt,yt,<j⋆), _t^-=- 1N_t _j=1^N_t p_θ, _a (y_t,j x_t,y_t,<j ), (1) where NtN_t is the number of target tokens and the superscript −- denotes measurement before the update action is selected and applied. Length normalization makes the loss comparable across responses of different lengths. A small ℓt− _t^- means that the model already assigns high probability to the target response, while a large ℓt− _t^- indicates a greater need for adaptation. This loss is used only to guide action selection during supervised adaptation and is not required during inference. As a complementary discrete measure, the same diagnostic forward pass also computes the pre-update target-token accuracy: At−=1Nt∑j=1Nt[argmaxypθ,θa(y∣xt,yt,<j⋆)=yt,j⋆],A_t^-= 1N_t _j=1^N_tI [ _yp_θ, _a (y x_t,y_t,<j )=y_t,j ], (2) where [⋅]I[·] is the indicator function. Thus, At−A_t^- is the proportion of target positions at which the most probable token matches the ground-truth token. The loss is a continuous measure that reflects the probability assigned to each target token, whereas target-token accuracy is an indicator-based measure of whether the top-1 prediction is correct. PARALLEL constructs the goal-related signal as sg,t=[At−,exp(−ℓt−)],s_g,t= [A_t^-, (- _t^-) ], (3) where At−A_t^- denotes the target-token accuracy and ℓt− _t^- is the length-normalized target loss before adaptation. Because ℓt− _t^- is length-normalized, exp(−ℓt−) (- _t^-) corresponds to the geometric mean of the probabilities assigned to the target tokens and therefore serves as a probability-based target-confidence measure. Together, these two components summarize how closely the current model behavior agrees with the labeled target. Target agreement alone does not describe how confidently the model forms its prediction. PARALLEL therefore constructs an uncertainty-related signal containing five complementary statistics: su,t=[Htchoice,Httok,ct,mt,vt].s_u,t= [H_t^choice,H_t^tok,c_t,m_t,v_t ]. (4) The five components are defined as follows: • HtchoiceH_t^choice is the normalized entropy over candidate responses. A high value indicates that probability is distributed across multiple choices. • HttokH_t^tok is the average normalized entropy of the token-level predictive distributions. A high value indicates diffuse token predictions. • ctc_t is the mean top-1 confidence across target-token positions. A high value indicates strong confidence in the most probable tokens. • mtm_t is the mean probability margin between the top-1 and top-2 tokens. A large margin indicates a more decisive prediction. • vtv_t is the variance of the probabilities assigned to the target tokens across positions. It represents how consistently the model assigns confidence throughout the target response. The normalized token entropy HttokH_t^tok is computed by dividing the predictive entropy at each target-token position by log|| |V|, where V is the model vocabulary, and then averaging over the target positions. For a multiple-choice sample with KtK_t candidate responses, let ℓt,kcand _t,k^cand denote the length-normalized loss of candidate k. Its choice probability is defined as follows: qt,k q_t,k =exp(−ℓt,kcand)∑r=1Ktexp(−ℓt,rcand), = (- _t,k^cand) _r=1^K_t (- _t,r^cand), (5) Htchoice H_t^choice =−1logKt∑k=1Ktqt,klogqt,k. =- 1 K_t _k=1^K_tq_t,k q_t,k. For XSum and CNN/DailyMail, which do not provide discrete answer choices, we set Htchoice=0H_t^choice=0, and the controller relies on the remaining goal-related, uncertainty-related, and representation signals. Lower-loss candidates receive higher probabilities, while HtchoiceH_t^choice measures the ambiguity among the candidates. The normalization by logKt K_t makes the entropy comparable across samples with different numbers of choices. HtchoiceH_t^choice is high when probability is distributed across several candidates and low when the prediction is concentrated on a particular candidate. The goal-related signal therefore describes how closely the prediction agrees with the target, whereas the uncertainty-related signal describes how confidently the prediction is formed. The two signals contain numerical summaries of model behavior but do not fully represent the meaning or context of the sample. To provide this information, PARALLEL concatenates them with the final prompt-token representation hth_t: zt=[sg,t;su,t;ht],z_t=[s_g,t;s_u,t;h_t], (6) where the semicolon denotes concatenation. The resulting state combines target agreement, predictive uncertainty, and the current model representation. All components of ztz_t are detached from the language-model computation graph before being passed to the controller. Budget-Constrained Controller Learning PARALLEL uses the controller state ztz_t to produce an action policy as follows: πθc(at∣zt)=[softmax(fθc(zt))]at. _ _c(a_t z_t)= [softmax (f_ _c(z_t) ) ]_a_t. (7) where fθcf_ _c is a lightweight controller network with parameters θc _c, and ata_t denotes the selected action for sample t. The goal-related and uncertainty-related signals are not independently mapped to separate actions. Instead, the controller interprets their joint evidence to determine the required degree of adaptation. When the current prediction is sufficiently aligned with the target and is considered reliable, the update can be inhibited. When the evidence is intermediate or the two signals provide conflicting information, a moderated adjustment can be applied. When the current behavior indicates a clear need for correction, the adapter can receive a full-strength update. From a cognitive-control perspective, these cases correspond to inhibition, moderated adjustment, and full correction. PARALLEL represents these three functional regimes as =Skip,Light,StrongA=\ Skip, Light, Strong\. The update weight associated with the selected action is wt=w(at)=0,at=Skip,λt,at=Light,1,at=Strong,0<λt<1.w_t=w(a_t)= cases0,&a_t= Skip,\\ _t,&a_t= Light,\\ 1,&a_t= Strong, cases 0< _t<1. (8) Where Skip inhibits the adapter update, Light applies an intermediate update scale, and Strong applies the reference full-strength update. The controller learns how the combined state ztz_t maps to these regimes without relying on manually defined thresholds. rtr_t denotes the controller reward, which is formally defined in Eq. 16. The Light scale λt _t is adjusted using the observed rewards of the Light and Strong actions. For k∈Light,Strongk∈\ Light, Strong\, the action-wise reward estimate is updated as r¯k,t=βλr¯k,t−1+(1−βλ)rt,at=k,r¯k,t−1,at≠k. r_k,t= cases _λ r_k,t-1+(1- _λ)r_t,&a_t=k,\\ r_k,t-1,&a_t≠ k. cases (9) When both reward estimates are available, PARALLEL computes a target scale by comparing their positive reward per unit update mass: λ^t=clip([r¯Light,t]+λtmax([r¯Strong,t]+,ϵ),λmin,λmax). λ_t=clip ( [ r_Light,t]_+ _t ([ r_Strong,t]_+,ε), _ , _ ). (10) Because λt _t is bounded below by λmin>ϵ _ >ε, the stabilizer is required only for the Strong return. When the estimated Strong return is non-positive while the Light return is positive, the small stabilized denominator drives λ^t λ_t toward λmax _ . When both returns are non-positive, the ratio becomes zero and λ^t λ_t is set to λmin _ . The Light scale is then updated as λt+1=clip(βλλt+(1−βλ)λ^t,λmin,λmax). _t+1=clip ( _λ _t+(1- _λ) λ_t, _ , _ ). (11) The update of λt _t begins only after both Light and Strong have been selected at least once; until then, λt _t remains unchanged. The reported experiments use λ1=0.5 _1=0.5, βλ=0.95 _λ=0.95, and [λmin,λmax]=[0.05,0.95][ _ , _ ]=[0.05,0.95]. The cumulative amount of allocated updating after processing t samples is Mt=∑i=1tw(ai).M_t= _i=1^tw(a_i). (12) The cumulative update mass and its budget constraint are defined as Mt=∑i=1twi,MTT≤ρ.M_t= _i=1^tw_i, M_TT≤ρ. (13) Update mass is the cumulative sum of the action weights w(at)w(a_t), and a budget ratio ρ permits at most ρTρ T mass over a T-sample stream. Before sampling, actions that would exceed the remaining mass are masked and the remaining probabilities are renormalized, ensuring MT≤ρTM_T≤ρ T. This budget limits the cumulative update intensity rather than the number of labeled samples processed, the number of LoRA parameters, or the measured parameter displacement. When wt>0w_t>0, PARALLEL performs one AdamW step on the supervised target loss with the learning rate scaled by wtw_t. When wt=0w_t=0, the optimizer step is omitted. The language-model backbone remains frozen, and only the LoRA adapter is updated. After applying the selected action, PARALLEL reevaluates the same sample to obtain the post-action target loss ℓt+ _t^+ and target-token accuracy At+A_t^+. The immediate improvement is defined as Δt=(ℓt−ℓt+)+γacc(At+−At−), _t=( _t^-- _t^+)+ _acc(A_t^+-A_t^-), (14) where γacc _acc controls the contribution of the accuracy change. For Skip, no adapter update occurs, so ℓt+=ℓt− _t^+= _t^-, At+=At−A_t^+=A_t^-, and Δt=0 _t=0. PARALLEL also considers how quickly the available budget is consumed. The budget-pressure term is Pt=[MtρT−tT]+.P_t= [ M_tρ T- tT ]_+. (15) The first fraction represents the consumed portion of the update budget, while the second represents the processed portion of the adaptation stream. Thus, PtP_t becomes positive when budget consumption advances faster than stream progression. The controller reward is rt=Δt−αcwt−αbPt,r_t= _t- _cw_t- _bP_t, (16) where αc _c penalizes update intensity and αb _b penalizes excessive early budget consumption. The hard constraint in Eq. (13) prevents budget violation, whereas PtP_t encourages the controller to distribute the available budget across the stream. The controller is optimized using one-step REINFORCE (Williams 1992): ℒctrl=−(rt−bt−1)logπ~θc,t(at∣zt)−αentℋ(π~θc,t),L_ctrl=-(r_t-b_t-1) π_ _c,t(a_t z_t)- _entH ( π_ _c,t ), (17) where π~θc,t π_ _c,t is the budget-masked policy and αent _ent controls entropy regularization. After each controller step, the reward baseline is updated using an exponential moving average, bt=βbbt−1+(1−βb)rtb_t= _bb_t-1+(1- _b)r_t. If rt>bt−1r_t>b_t-1, the selected action is reinforced under similar controller states; otherwise, its probability is reduced. Entropy regularization prevents premature concentration on a single action. Because rtr_t compares the same sample before and after the selected action, the controller learns from immediate utility–cost feedback rather than optimizing a long-horizon episodic return. Algorithm 1 PARALLEL adaptation 0: Labeled adaptation stream =(xt,yt⋆)t=1TD=\(x_t,y_t )\_t=1^T, frozen backbone θ, LoRA adapter θa _a, controller θc _c, budget ratio ρ 1: Initialize M0←0M_0← 0, reward baseline b0b_0, Light scale λ1 _1, and action-reward estimates 2: for t=1,…,Tt=1,…,T do 3: Compute ℓt− _t^-, At−A_t^-, sg,ts_g,t, su,ts_u,t, and hth_t 4: Form zt=[sg,t;su,t;ht]z_t=[s_g,t;s_u,t;h_t] and compute πθc(⋅∣zt) _ _c(· z_t) 5: Mask actions satisfying Mt−1+wλt(a)>ρTM_t-1+w_ _t(a)>ρ T and renormalize to π~θc,t π_ _c,t 6: Sample at∼π~θc,ta_t π_ _c,t and set wt←wλt(at)w_t← w_ _t(a_t) 7: if wt>0w_t>0 then 8: Update θa _a by one supervised AdamW step with the learning rate scaled by wtw_t 9: end if 10: Set Mt←Mt−1+wtM_t← M_t-1+w_t 11: Obtain ℓt+ _t^+ and At+A_t^+, and compute reward rtr_t using MtM_t 12: Update θc _c using Eq. (17) with baseline bt−1b_t-1 13: Update the action-reward estimates and Light scale λt+1 _t+1 14: Update btb_t 15: end for 16: return Adapted LoRA parameters θa _a Dataset Metric Frozen PARALLEL Full LoRA ARC-Challenge acc_norm 0.481 0.569 0.584 OpenBookQA acc_norm 0.418 0.540 0.574 CommonsenseQA Accuracy 0.789 0.842 0.849 XSum R-1 0.2000 0.3650 0.3723 R-2 0.0506 0.1409 0.1454 R-L 0.1392 0.2915 0.2951 CNN/DM R-1 0.3067 0.3937 0.3991 R-2 0.1034 0.1694 0.1742 R-L 0.1870 0.2685 0.2715 Table 1: Overall benchmark results. Experiments Experimental Setup We evaluate PARALLEL on ARC-Challenge (Clark et al. 2018), OpenBookQA (Mihaylov et al. 2018), CommonsenseQA (Talmor et al. 2019), XSum (Narayan et al. 2018), and CNN/DailyMail (Hermann et al. 2015). The first three benchmarks evaluate science and commonsense reasoning, while XSum and CNN/DailyMail evaluate abstractive news summarization in separate experiments. The primary evaluation uses Qwen2.5-3B-Instruct on all five benchmarks. For the reasoning experiments, fixed training streams contain 90% of each official training split: 1,007 examples for ARC-Challenge, 4,461 for OpenBookQA, and 8,767 for CommonsenseQA. We additionally evaluate Qwen2.5-0.5B-Instruct, SmolLM2-360M-Instruct (Ben Allal et al. 2025), and TinyLlama-1.1B-Chat (Zhang et al. 2024) on ARC-Challenge with the official training split and seeds 42, 43, and 44. All adaptation methods use a frozen language-model backbone and a trainable LoRA adapter (Hu et al. 2022). Training uses one pass, batch size 1, bfloat16, and AdamW with a learning rate of 2×10−52× 10^-5, linear scheduling, and a warmup ratio of 0.03. The LoRA configuration uses rank 16, scaling 32, and dropout 0.05. Within each model–dataset condition, all methods use the same tokenizer, prompt format, adapter configuration, and optimizer settings. The PARALLEL controller is a one-hidden-layer multilayer perceptron with hidden dimension 64 and a GELU activation. It is optimized using AdamW with a learning rate of 8×10−48× 10^-4. The controller objective uses γacc=0.30 _acc=0.30, αc=0.14 _c=0.14, αb=0.40 _b=0.40, αent=0.01 _ent=0.01, and reward-baseline decay βb=0.95 _b=0.95. The LoRA adapter is optimized using the supervised target loss, whereas the controller is optimized using reward feedback. Final adapters are evaluated without in-context demonstrations. For the reasoning benchmarks, we use lm-eval-harness (Biderman et al. 2024) and report acc_norm for ARC-Challenge and OpenBookQA and acc for CommonsenseQA. For XSum and CNN/DailyMail, we report ROUGE-1, ROUGE-2, and ROUGE-L. Target ρ ARC-C OBQA CSQA 0.150.15 0.51620.5162 0.49600.4960 0.83780.8378 0.300.30 0.56910.5691 0.54000.5400 0.84190.8419 0.500.50 0.55380.5538 0.56000.5600 0.84030.8403 Table 2: Budget sensitivity of PARALLEL. Compared Methods and Metrics Frozen performs no adaptation, while Full adaptation applies the same LoRA update to every sample. Across all five benchmarks, we additionally compare two budget-matched subset baselines: Random selection (Citovsky et al. 2021), entropy-based Active learning (Lewis and Gale 1994; Settles 2009). The selection scores are computed once before adaptation, and each selected sample receives a full-strength update. PARALLEL instead uses the goal-related and uncertainty-related signals to assign action-dependent update weights while processing the complete labeled stream. Following Eq. (13), normalized update mass is reported as MT/TM_T/T. Frozen and Full adaptation have masses of 0 and 11, respectively, and the main budget-matched comparison uses ρ=0.30ρ=0.30. For a task score S, we report Rscore=100×SSFull,Rgain=100×S−SFrozenSFull−SFrozen.R_score=100× SS_Full, R_gain=100× S-S_FrozenS_Full-S_Frozen. (18) These metrics denote Full-score retention and adaptation-gain recovery, respectively. For XSum and CNN/DailyMail, benchmark-level RgainR_gain is computed by averaging the values for ROUGE-1, ROUGE-2, and ROUGE-L. Budget Selection Before conducting the full evaluation, we perform a preliminary search to select an update budget for PARALLEL. Rather than tuning the budget separately for each dataset, we evaluate three representative ratios, ρ∈0.15,0.30,0.50ρ∈\0.15,0.30,0.50\, on a subset of the benchmarks. The aim is to identify a practical operating point rather than a globally optimal ratio. As shown in Table 2, increasing ρ from 0.150.15 to 0.300.30 improves performance across the evaluated benchmarks, whereas further gains at 0.500.50 are task dependent despite the additional updates. Among the evaluated ratios, ρ=0.30ρ=0.30 therefore provides the most favorable balance between performance and update cost. Experimental Results Table 1 summarizes the results across reasoning and summarization benchmarks. On ARC-Challenge, OpenBookQA, and CommonsenseQA, PARALLEL retains 97.4%97.4\%, 94.1%94.1\%, and 99.2%99.2\% of the performance under Full adaptation, respectively. It also recovers 85.4%85.4\%, 78.2%78.2\%, and 88.3%88.3\% of the corresponding gains over Frozen. On XSum, performance retention ranges from 96.9%96.9\% to 98.8%98.8\%, while gain recovery ranges from 95.3%95.3\% to 97.7%97.7\%. On CNN/DailyMail, performance retention and gain recovery across the three ROUGE metrics range from 97.2% to 98.9% and from 93.2% to 96.4%, respectively. Task Metric Random selection Active learning PARALLEL ARC-C Acc. 0.54520.5452 0.52130.5213 0.56910.5691 OBQA Acc. 0.54400.5440 0.51800.5180 0.54000.5400 CSQA Acc. 0.83130.8313 0.84440.8444 0.84190.8419 XSum R-1 0.36520.3652 0.35950.3595 0.36500.3650 R-2 0.14140.1414 0.13640.1364 0.14090.1409 R-L 0.29140.2914 0.28630.2863 0.29150.2915 CNN/DM R-1 0.39500.3950 0.39260.3926 0.39370.3937 R-2 0.17050.1705 0.16470.1647 0.16940.1694 R-L 0.26790.2679 0.23430.2343 0.26850.2685 Table 3: Comparison of various adaptation strategies. Figure 2: Gain recovery of budget-matched methods across five benchmarks. Table 3 evaluates update-allocation efficiency by comparing PARALLEL with fixed subset-selection methods under the same target update-mass ratio. Among the budget-matched methods, PARALLEL achieves the highest ARC-Challenge score, outperforming the next-best method by 0.0239. Overall, PARALLEL remains consistently at or near the top. PARALLEL achieves the highest ROUGE-L on both XSum and CNN/DailyMail, with scores of 0.29150.2915 and 0.26850.2685. This consistent ROUGE-L advantage indicates better preservation of long-sequence content overlap under the same update budget. Consequently, Figure 2 shows that PARALLEL achieves the highest average RgainR_gain recovery of 88.6% across the five benchmarks, with a minimum benchmark-level recovery of 78.2%. Backbone Frozen Full LoRA PARALLEL Mass Qwen-0.5B 0.3380.338 0.388±0.0040.388±0.004 0.371±0.0030.371±0.003 0.2970.297 Smol-360M 0.3440.344 0.401±0.0010.401±0.001 0.386±0.0040.386±0.004 0.3000.300 Tiny-1.1B 0.3270.327 0.370±0.0040.370±0.004 0.358±0.0040.358±0.004 0.3000.300 Macro 0.3360.336 0.3860.386 0.3720.372 0.2990.299 Table 4: ARC-Challenge results for different language models. RscoreR_score are three-seed mean± . Mass is normalized update mass. Cross-Backbone Evaluation We evaluate PARALLEL on Qwen2.5-0.5B-Instruct, SmolLM2-360M-Instruct, and TinyLlama-1.1B-Chat using ARC-Challenge, ρ=0.30ρ=0.30, and three seeds. As shown in Table 4, PARALLEL improves over Frozen across all backbones with observed update masses of 0.2970.297–0.3000.300. It retains 95.695.6–96.8%96.8\% of Full performance and recovers 66.166.1–74.5%74.5\% of its adaptation gain, with macro averages of 96.2%96.2\% and 71.1%71.1\%, respectively. These results demonstrate consistent performance–update behavior across the three compact backbones. Ablation of Controller Inputs We examine the contribution of each controller input by removing the goal-related signal sgs_g, the uncertainty-related signal sus_u, or the model representation hth_t, while keeping the training and budget settings unchanged. Variant ARC-C OBQA CSQA PARALLEL 0.56910.5691 0.54000.5400 0.84190.8419 w/o goal signal sgs_g 0.55290.5529 0.53400.5340 0.82880.8288 w/o uncertainty signal sus_u 0.56060.5606 0.53920.5392 0.84110.8411 w/o representation hth_t 0.54690.5469 0.52400.5240 0.83870.8387 Table 5: Ablation of the controller inputs. As shown in Table 5, the complete PARALLEL configuration achieves the highest accuracy on all three benchmarks. Removing sgs_g causes clear degradation on ARC-Challenge and CommonsenseQA, while removing hth_t produces the largest reductions on ARC-Challenge and OpenBookQA. Overall, no ablated variant outperforms the complete configuration, supporting the complementary use of the controller inputs. Figure 3: Validation accuracy–cost trade-offs under sequential sample updates, measured by cumulative adaptation time and GPU energy. Practical Efficiency of PARALLEL We evaluate whether sample-wise update allocation yields higher target-task accuracy when post-deployment adaptation is constrained by time or GPU energy. Unlike the preceding endpoint comparisons, this experiment compares validation accuracy at the same measured adaptation cost. The current adapter is evaluated every 64 samples on all 299 validation examples. Adaptation time includes method-specific stream processing and updates, while GPU energy is obtained by integrating NVIDIA Management Library (NVML) power measurements sampled every 50 ms (Kasichayanula et al. 2012). Figure 3 presents a representative seed-42 trajectory on a sequential 512-sample ARC-Challenge stream. Both Full LoRA and PARALLEL process every incoming sample. Full LoRA updates the adapter on every sample, whereas PARALLEL uses the controller to determine whether and how strongly the adapter is updated. We measure acc_norm and cumulative adaptation costs at fixed intervals. Cumulative cost denotes the running time or energy from the beginning of the stream and includes stream processing, controller computation, and adapter updates, while excluding checkpoint evaluation. The top and bottom panels report accuracy according to cumulative adaptation time and GPU energy, respectively. A higher curve at the same cumulative cost indicates greater adaptation accuracy under the same resource limit. At approximately 7.87.8 seconds, PARALLEL achieves 0.3630.363, compared with 0.3440.344 for Full LoRA. At approximately 0.155Wh0.155~Wh, the corresponding scores are 0.3590.359 and 0.3440.344. These results demonstrate a more favorable accuracy–cost balance for resource-constrained post-deployment adaptation. Conclusion We introduced PARALLEL, a prefrontal-aligned framework for sample-wise LoRA update allocation under a cumulative update-mass constraint. Its controller combines goal-related, uncertainty-related, and representation signals to determine whether and how strongly to update each labeled sample and is trained with one-step REINFORCE using immediate utility–cost feedback. Experiments across three reasoning benchmarks and two summarization tasks show that PARALLEL retains most of the performance under Full adaptation while operating under a limited cumulative update mass. It also achieves higher accuracy than Full LoRA at matched cumulative adaptation time and GPU energy in the stream setting. Controller-input ablations further support the complementary use of all three controller inputs for sample-wise update allocation. These results demonstrate the potential of sample-wise update allocation for language-model adaptation under a limited cumulative update mass. Future work will extend PARALLEL to larger models, continual and generative adaptation, and broader deployment conditions while directly optimizing model stability and measured hardware costs. References L. Ben Allal, A. Lozhkov, E. Bakouch, et al. (2025) SmolLM2: when smol goes big—data-centric training of a small language model. arXiv preprint arXiv:2502.02737. Cited by: Experimental Setup. S. Biderman, H. Schoelkopf, L. Sutawika, L. Gao, J. Tow, B. Abbasi, A. F. Aji, P. S. Ammanamanchi, S. Black, J. Clive, et al. (2024) Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782. Cited by: Experimental Setup. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. (2024) A survey on evaluation of large language models. ACM transactions on intelligent systems and technology 15 (3), p. 1–45. Cited by: Introduction. A. Chatterjee, H. S. V. N. S. Kowndinya Renduchintala, S. Bhatia, and T. Chakraborty (2025) On the effect of instruction tuning loss on generalization. Transactions of the Association for Computational Linguistics 13, p. 1360–1380. External Links: Document, Link Cited by: Control Signals in PARALLEL. G. Citovsky, G. DeSalvo, C. Gentile, L. Karydas, A. Rajagopalan, A. Rostamizadeh, and S. Kumar (2021) Batch active learning at scale. In Advances in Neural Information Processing Systems, Vol. 34, p. 11933–11944. External Links: Link Cited by: Efficient Adaptation Methods, Compared Methods and Metrics. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: Experimental Setup. J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, Minneapolis, Minnesota, p. 4171–4186. External Links: Document, Link Cited by: Efficient Adaptation Methods. K. M. Hermann, T. Kočiský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom (2015) Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, Vol. 28, p. 1693–1701. External Links: Link Cited by: Experimental Setup. S. C. H. Hoi, D. Sahoo, J. Lu, and P. Zhao (2021) Online learning: a comprehensive survey. Neurocomputing 459, p. 249–289. External Links: Document Cited by: Introduction. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: Introduction, Efficient Adaptation Methods, Experimental Setup. K. Kasichayanula, D. Terpstra, P. Luszczek, S. Tomov, S. Moore, and G. D. Peterson (2012) Power aware computing on gpus. In 2012 Symposium on Application Accelerators in High Performance Computing, p. 64–73. Cited by: Practical Efficiency of PARALLEL. D. D. Lewis and W. A. Gale (1994) A sequential algorithm for training text classifiers. In SIGIR ’94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, W. B. Croft and C. J. van Rijsbergen (Eds.), London, p. 3–12. External Links: Document Cited by: Efficient Adaptation Methods, Compared Methods and Metrics. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal (2018) Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, p. 2381–2391. Cited by: Experimental Setup. S. Narayan, S. B. Cohen, and M. Lapata (2018) Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, p. 1797–1807. External Links: Document, Link Cited by: Experimental Setup. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730–27744. External Links: Link Cited by: Reinforcement Learning for Adaptation Control. M. Paul, S. Ganguli, and G. K. Dziugaite (2021) Deep learning on a data diet: finding important examples early in training. In Advances in Neural Information Processing Systems, Vol. 34, p. 20596–20607. Cited by: Introduction. B. Settles (2009) Active learning literature survey. Technical report Technical Report 1648, University of Wisconsin–Madison. Cited by: Introduction, Efficient Adaptation Methods, Compared Methods and Metrics. Y. Sung, M. Rigotti, and S. W. Lee (2025) Factorized embedding of goal and uncertainty in the lateral prefrontal cortex guides stably flexible learning. Nature Communications. Cited by: Introduction, Prefrontal-Aligned Learning Control. A. Talmor, J. Herzig, N. Lourie, and J. Berant (2019) Commonsenseqa: a question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 4149–4158. Cited by: Experimental Setup. R. J. Williams (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (3–4), p. 229–256. External Links: Document Cited by: Budget-Constrained Controller Learning. N. Yoon, K. Yoo, W. Jung, S. Kim, and H. Kim (2026) Beyond learning: a training-free alternative to model adaptation. arXiv preprint arXiv:2602.16189. Cited by: Introduction. P. Zhang, G. Zeng, T. Wang, and W. Lu (2024) Tinyllama: an open-source small language model. arXiv preprint arXiv:2401.02385. Cited by: Experimental Setup. W. X. Zhao, K. Zhou, J. Li, T. Tang, Z. Dong, Y. Hou, B. Zhang, Y. Min, J. Zhang, P. Liu, et al. (2026) A survey of large language models. Frontiers of Computer Science 20 (12), p. 2012627. Cited by: Introduction.