Paper deep dive

TextualRL: Textual Reinforcement Learning for Context Management

Tianyu Huai, Jie Zhou, Bo Mao, Qin Chen, Liang He, Xipeng Qiu

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 81

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:LLM agents can adapt to task feedback by revising the textual context that guides their behavior, without changing their model parameters. However, turning execution rewards into reusable context updates is challenging: individual trajectories offer limited comparative evidence about what to change, while task-specific fixes may overgeneralize when incorporated into a shared context. We introduce TextualRL, a gradient-free reinforcement learning framework that treats a group of rollouts from the same task under the same context, rather than an isolated trajectory, as the unit of policy critique. Groupwise Policy Critique contrasts successful and failed rollouts within a task; when outcomes are uniform, it draws on cross-task evidence to identify behaviors worth preserving or formulate repair hypotheses. Cross-Group Policy Update reviews the resulting proposals against evidence pooled across groups, including observations that yield no proposed edit, to refine their content and applicability. Reviewed proposals form a bounded textual patch, with adoption of the candidate context determined by held-out validation rewards. Together, these components establish a reward-driven policy optimization loop in which multiple attempts jointly inform reusable context updates. We evaluate TextualRL with five target agents on six benchmarks. Averaged over the six benchmarks, TextualRL improves test accuracy over the base model by 24.4 and 15.3 percentage points for GPT-5.5 and Qwen3.8-27B, respectively.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

77,271 characters extracted from source content.

Expand or collapse full text

TEXTUALRL: TEXTUAL REINFORCEMENT LEARNING FOR CONTEXT MANAGEMENT Tianyu Huai 1,2 , Jie Zhou 1∗ , Bo Mao 1 , Qin Chen 1 , Liang He 1 , Xipeng Qiu 2,3 1 School of Computer Science and Technology, East China Normal University, Shanghai 2 Shanghai Innovation Institute, 3 Fudan University tyhuai@stu.ecnu.edu.cn, jzhou@cs.ecnu.edu.cn https://github.com/ECNU-ICALK/TextualRL ABSTRACT LLM agents can adapt to task feedback by revising the textual context that guides their behavior, without changing their model parameters. However, turning execution rewards into reusable context updates is challenging: individual trajectories offer limited comparative evidence about what to change, while task-specific fixes may overgeneralize when incorporated into a shared context. We introduce TextualRL, a gradient-free reinforcement learning framework that treats a group of rollouts from the same task under the same context, rather than an isolated trajectory, as the unit of policy critique. Groupwise Policy Critique contrasts successful and failed rollouts within a task; when outcomes are uniform, it draws on cross-task evidence to identify behaviors worth preserving or formulate repair hypotheses. Cross-Group Policy Update reviews the resulting proposals against evidence pooled across groups, including observations that yield no proposed edit, to refine their content and applicability. Reviewed proposals form a bounded textual patch, with adoption of the candidate context determined by held-out validation rewards. Together, these components establish a reward-driven policy optimization loop in which multiple attempts jointly inform reusable context updates. We evaluate TextualRL with five target agents on six benchmarks. Averaged over the six benchmarks, TextualRL improves test accuracy over the base model by 24.4 and 15.3 percentage points for GPT-5.5 and Qwen3.8-27B, respectively. 1 Introduction Reinforcement learning (RL) improves an agent’s behavior through interaction and reward [1]. For LLM agents whose parameters are unavailable for optimization, textual context offers an alternative interface for adaptation: instructions and procedures condition how a fixed model interprets tasks, uses tools, and produces outputs. Reflexion [2] incorporates verbal feedback into subsequent attempts, while Agentic Context Engineering (ACE) [3] evolves contextual playbooks from execution experience. These developments motivate policy optimization in context space: the objective remains higher task reward, but the optimization variable is the context rather than the model weights. Existing work provides several mechanisms for optimizing textual artifacts. ProTeGi [4] and TextGrad [5] use natural- language feedback to guide updates, while GEPA [6] combines reflection on execution traces with evolutionary prompt search. For agent skills, Trace2Skill [7] extracts trajectory-local lessons and hierarchically consolidates them into reusable instructions. SkillOpt [8] instead analyzes successful and failed trajectory minibatches separately, merges their proposed edits, and validates bounded skill updates. These methods establish the value of execution-driven adaptation, while highlighting an important design choice: how should trajectories be organized before their feedback is converted into a shared context update? Group-based RL provides a useful organizing principle. Group Relative Policy Optimization (GRPO) [9] samples multiple responses to the same question and uses their relative rewards to guide parameter updates. For textual adaptation, a group of same-task rollouts likewise provides a comparison unavailable from an isolated execution: under ∗ Corresponding author. arXiv:2610.18216v1 [cs.CL] 13 Oct 2026 ICALK@ECNU Policy policy � � Environment environment dynamics Trajectories rollouts D Reward&Advantage R (τ) & � Policy Update parameter update Δ� Context C Editable context Agent Rollout Frozen LLM agent Rollout Outcomes D = � � �=1 � · Task trajectories and outcomes Outcome Analysis Mixed outcomes Within-task contrast All-success / -failure Cross-task analysis Context Update C′= C +ΔC Bounded textual edit ΔC e.g., insert / delete / replace Condition agent K same-task rollouts improved context C′ improved policy � �’ r =(R(τ 1 ),...,R(τ K )) Figure 1: Conceptual comparison of parameter-space and context-space policy optimization. TextualRL uses groups of same-task rollouts and their rewards to propose textual edits ∆C to context C, while the target model remains frozen. the same task and context, which behavioral differences accompany different outcomes? Comparative experience learning is also explored by ExpeL [10] and ReasoningBank’s parallel MaTTS [11], which extract reusable insights or memories from alternative executions. Building on this perspective, we ask: how can groups of rewarded executions drive policy optimization when updates are made to shared textual context rather than model parameters? The goal is not merely to select a successful attempt, but to use multiple attempts to improve the policy governing future executions. Turning grouped experience into a context update presents two challenges. First, rewards identify outcomes, not the instructions that should change. Within-task success–failure comparisons provide evidence about alternative behaviors, but this contrast disappears when every sampled rollout succeeds or every rollout fails. An analysis procedure must therefore distinguish evidence supporting preservation from evidence suggesting an unverified repair. Second, a context is shared across tasks, whereas a proposed modification may be supported only under particular conditions. Merging proposals alone cannot reveal every exception: a successful execution may require no edit while still contradicting another group’s proposed rule. Effective updates therefore require both outcome-aware policy critique and a review of the evidence that supports or limits each modification. We introduce TextualRL, a gradient-free reinforcement learning algorithm for context management that treats same-task rollout groups as the basic unit of policy critique. As illustrated in Figure 1, each iteration collects multiple executions per task under the current context while keeping the target model frozen. Groupwise Policy Critique compares successful and failed rollouts within mixed-outcome tasks; for all-success or all-failure tasks, it uses cross-task evidence to identify behaviors worth preserving or formulate repair hypotheses, respectively. The critics return edit proposals and trajectory evidence separately, retaining observations even when no edit is proposed. Cross-Group Policy Update reviews consolidated proposals against evidence pooled across analysis groups, retaining, narrowing, or revising instructions according to their support and applicability. The reviewed edits form a bounded textual patch. A step-level validation gate accepts the candidate context only if its reward on held-out tasks exceeds the recorded reference score. The selected context then conditions subsequent rollouts, closing a reward-driven policy optimization loop without model-parameter updates. We evaluate TextualRL with five target agents across six benchmarks. Averaged over the six benchmarks, TextualRL improves test accuracy over the no-skill baseline by 24.4 and 15.3 percentage points for GPT-5.5 and Qwen3.8-27B, respectively. The corresponding gains over the best baseline are 0.9 and 3.2 percentage points. Our contributions are threefold: •We introduce TextualRL, a group-based algorithm for gradient-free policy optimization over the shared textual context of a frozen LLM agent. • We develop Groupwise Policy Critique, which uses the available reward structure to combine within-task comparisons with cross-task preservation and repair analysis. • We develop Cross-Group Policy Update, which retains evidence independently of edit proposals to review their content and applicability before bounded, validation-gated updates. 2 ICALK@ECNU 2 Related Work Prompt and Context Optimization.Automatic prompt optimization searches for text that improves task performance. AutoPrompt [12] uses gradient-guided search over discrete trigger tokens, whereas APE [13] generates and evaluates instructions from demonstrations. DSPy [14] compiles modular language-model pipelines through demonstration bootstrapping and selection, while MIPRO [15] jointly optimizes instructions and demonstrations across modules. Context and skill optimization extends this scope to reusable guidance for future executions. Dynamic Cheatsheet [16] maintains adaptive memory across test queries. Trace2Skill [7] independently extracts trajectory-local patches and hierarchically consolidates them into agent skills. EvoSkill [17] proposes skills from failed trajectories and evaluates candidate skill repositories through validation. SkillOpt [8] separately analyzes successful and failed trajectory minibatches, merges their proposals, and selects bounded skill updates using held-out validation. TextualRL shares the goal of feedback-driven textual adaptation but changes how executions inform an update: same-task rollouts under the same context are preserved for joint critique before proposal consolidation. Moreover, trajectory evidence is retained independently of proposed edits. This allows observations that warrant no new instruction to constrain other proposals, making evidence about a rule’s applicability available beyond the editing suggestions themselves. Learning from Trajectory Feedback.Trajectory feedback supports both task execution and learning across attempts. ReAct [18] interleaves reasoning and actions with environment observations, and Self-Refine [19] iteratively critiques and revises an output. LATS [20] combines tree search, self-reflection, and environment feedback to explore alter- native trajectories. Voyager [21] uses execution feedback to improve programs and accumulate reusable code skills. Comparative experience learning is also established: ExpeL [10] extracts insights from same-task success–failure pairs and shared patterns across successful tasks, while ReasoningBank [11] uses parallel MaTTS to curate reusable memory by comparing multiple executions of the same query. In parameter-space RL, GRPO [9] uses relative rewards within groups of same-question responses to guide policy updates. TextualRL instead performs gradient-free policy optimization over a shared context, coupling comparative experience with an explicit proposal–review–validation loop. Groupwise Policy Critique adapts its analysis to the available reward structure: mixed outcomes support within-task contrast, whereas uniform outcomes require cross-task evidence for preservation or repair hypotheses. Cross-Group Policy Update then reviews both the content and applicability of proposed instructions using independently pooled evidence, before edits undergo held-out validation. Together, these mechanisms connect outcome-aware comparison to cross-task evidence review, turning groups of attempts into reward-selected updates to a frozen agent’s shared policy context. 3 TextualRL 3.1 Context-Space Policy Optimization LetC ∈Cdenote the editable textual context containing instructions and procedures shared across tasks. For a task x, the context induces a trajectory distributionπ θ (τ | x,C)through the target agent, where a trajectoryτcomprises actions, available observations, and the final output. The target parametersθremain fixed, and TextualRL optimizes the context to maximize expected task reward: max C∈C J (C) =E x∼p task E τ∼π θ (·|x,C) [R(τ )],(1) wherep task is the task distribution andR(τ )∈0, 1indicates task success. Policy optimization thus operates through textual changes to C rather than updates to θ. At stept, we sample a training batchB t and collectKrollouts per task under the current contextC t . For eachx∈ B t , we retain its indexed rollout group and reward vector: D x =τ x,i K i=1 ,r x = (r x,1 ,...,r x,K ),(2) wherer x,i = R(τ x,i ). We omit the step index fromD x andr x for clarity. Keeping same-task rollouts together preserves the behavioral and outcome comparisons used to propose an update. As shown in Figure 2, each step comprises two modules. Groupwise Policy Critique uses the available outcome structure to produce edit proposals and trajectory evidence. Cross-Group Policy Update reviews these proposals against pooled evidence, constructs a bounded textual patch, and uses held-out validation rewards to select the next context. Subsequent rollouts are conditioned on the selected context, forming a gradient-free policy optimization loop. 3 ICALK@ECNU (a) Groupwise Policy Critique (b) Cross-Group Policy Update Context C Frozen Agent Training Set K rollouts per task Task batch 1ijK Task 1 (mixed) ... × ... ... × Task 2 (all-success) ... ... ... Task 3 (all-success) ... ... ... Task 4 (all-failure) × ... × ... × ... × Task 5 (all-failure) × ... × ... × ... × Analyst outputs (in parallel) Proposed edits Evidence cards Mixed outcomes (From Task 1) Within-task contrast Illustrative divergent trajectories τᵢ (failure) ... (Action A) ... × τⱼ (success) ... (Action B) ... Optimizer Model Edit 1 Constraint scope Evidence 1 Divergent behavior All-success (From Tasks 2–3) Cross-task preservation Common successful behavior Task 2 (success) ... [A] ... Task 3 (success) ... [A] ... Optimizer Model Edit 2 Keep behavior Evidence 2 Consistent pattern All-failure (From Tasks 4–5) Cross-task repair Common failure pattern across tasks Task 4 (failure) ... [B] ... × Task 5 (failure) ... [C] ... × Optimizer Model Edit 3 Revise format Evidence 3 Repeated errors Relevant evidence cards Proposed edits E2 Condition: Answer field specified. Behavior: Concise answer inside tags. Observed effect: Brief explanation outside tags. Task / trajectory reference: Task 2 / τ₁ E3 Condition: Answer tags required. Behavior: Returned an answer without tags. Observed effect: Required tags absent. Task / trajectory reference: Task 4 / τ₃ Edit 1 Purpose: Constrain answer scope. Content: Return only the final answer. Omit explanations and details from the response. Edit 3 Purpose: Enforce output format. Content: When answer tags are required, place the final answer inside <answer>...</answer>. limits supports Optimizer Model Evidence-based proposal review Retain edit Well supported Narrow scope Limit applicability Revise edit Modify text Reviewed edit 1 Purpose: Constrain answer scope. Content: Follow the task’s requested output format. Details outside the field only when permitted by the task. Reviewed edit 2Reviewed edit 3 merge Textual patch ΔC Candidate context C′ = C + ΔC Validation Set AcceptReject (retain C) Context C′ Step Buffer Shared history for all analysts Reward-based selection Within-epoch feedback Figure 2: Overview of TextualRL. (a) Groupwise Policy Critique preserves same-task rollout groups and uses within- task contrast for mixed outcomes, cross-task preservation for all-success outcomes, and cross-task repair for all-failure outcomes. Parallel critics return edit proposals and evidence cards separately. (b) Cross-Group Policy Update reviews proposals against pooled evidence, constructs a bounded patch, and selects the candidate context through held-out validation. Within-epoch feedback informs subsequent critique calls. 3.2 Groupwise Policy Critique The comparison available within a rollout group depends on its reward structure. Letn x = P K i=1 r x,i be the number of successful rollouts for task x. We route each group according to route(r x ) =    mixed,0 < n x < K, all-success, n x = K, all-failure, n x = 0. (3) Before critique, we consolidate duplicate trajectories while preserving their outcome labels, original rollout references, and multiplicities. All analysis groups retain complete task blocks; multiple rollouts of one task do not count as support from multiple distinct tasks. Within-task contrast.Each mixed-outcome task forms a separate analysis group. The critic compares successful and failed rollouts to identify the earliest observable behavioral divergence, such as a different action or use of available evidence. It proposes at most one atomic repair or preservation rule supported by this contrast, and returns no edit when no supported distinction is observable. Cross-task preservation. When all rollouts succeed, there is no within-task success–failure contrast. We therefore group all-success task blocks into analysis minibatches containing at least two distinct tasks. The critic identifies a shared, observable behavior worth preserving and proposes an instruction only when that behavior is not already covered by C t . Cross-task repair.All-failure task blocks are grouped in the same manner. The critic identifies a shared, observable failure mechanism across at least two distinct tasks and proposes a corresponding repair hypothesis. Failure alone neither supplies a correct answer nor verifies the proposed repair. Both homogeneous routes return no edit when the required common behavior or mechanism is unsupported, and a route with fewer than two task blocks is skipped. Edit proposals and trajectory evidence.Letg ⊆ B t be the tasks assigned to one critique call andG t the collection of these analysis groups. A mixed group contains one task, whereas a homogeneous group contains at least two tasks 4 ICALK@ECNU from the same outcome category. Each call returns proposed edits P g and evidence cards E g : (P g ,E g ) = Critique C t ,H t ,(x,D x ,r x ) : x∈ g .(4) Here,Critiqueinvokes the optimizer model with route-specific instructions. Calls run in parallel and share the current context C t and within-epoch history H t . An edit proposal specifies a modification toC t . An evidence card records an observable condition, behavior, and effect, together with a short supporting quote and references to the corresponding task and rollout. Unobserved conditions or effects are marked as unknown. Cards may be returned even whenP g is empty: behavior already handled correctly by the current context can still constrain a proposal from another group. We therefore pool proposals and evidence independently: P t = [ g∈G t P g , E t = [ g∈G t E g .(5) 3.3 Cross-Group Policy Update A modification supported by one group may restrict behavior that succeeds elsewhere. Cross-Group Policy Update therefore reviews the content and applicability of local proposals before evaluating their combined effect on the shared context. Evidence-based proposal review.The optimizer first consolidatesP t through hierarchical merging to remove redun- dancy and resolve conflicting instructions. It then reviews the consolidated candidates usingC t and the independently pooled evidenceE t . For each edit, it compares the prescribed behavior and scope with relevant observations, including the behavior being regulated, the failure mechanism being addressed, or an exception within the proposed scope. The review records whether these observations support, limit, or contradict the instruction. Based on these relations, the optimizer retains the edit, narrows its applicability using an observable condition, or revises its content. The resulting edits form e P t , with each decision accompanied by a rationale and references to the evidence used. Insufficient relevant evidence leaves the proposal’s original scope and uncertainty unchanged. The review neither requires evidence from every outcome category nor introduces unrelated edits to account for unused cards. Figure 2(b) illustrates this distinction. Card E2 records a successful response with the answer inside the required field and an explanation outside it, limiting Edit 1’s blanket answer-only restriction. The revised instruction follows the requested format and allows details outside the field only when permitted by the task. Card E3 instead supports Edit 3 by documenting missing required answer tags. The poolE t contains only current-step evidence, so this review requires no additional target-agent rollouts. Textual patch.Each reviewed edit specifies an operation—append,insert_after,replace, ordelete—together with target text when required and replacement or insertion content. We select at mostL t edits, ranking candidates with the optimizer when they exceed this budget. We also remove edits with explicit dependencies on information unavailable during target execution. The remaining operations form a textual patch ∆C t , yielding C ′ t = C t + ∆C t ≡ Apply(C t , ∆C t ),(6) where + denotes applying textual operations rather than numerical addition. Reward-based selection.We evaluateC ′ t on a held-out validation setVusing the benchmark’s binary success metric. Let b V (C)denote mean validation success under the evaluation protocol ands t the trainer’s recorded reference score. The step-level selection rule is (C t+1 ,s t+1 ) = ( (C ′ t , b V (C ′ t )), b V (C ′ t ) > s t , (C t ,s t ),otherwise. (7) Thus, an accepted candidate updates both the active context and the reference score; a rejected candidate leaves both unchanged. Validation evaluates the complete candidate context, not the individual effects of its constituent edits. Within-epoch feedback.After validation, the shared step buffer augmentsH t with observed rollout failure patterns. For rejected candidates, it also records the attempted edit combination and its validation score. Subsequent critique calls in the same epoch receive this history to reduce repeated ineffective proposals. Rejection concerns the evaluated combination and does not imply that every constituent edit is harmful. This history is distinct fromE t , which contains only the current step’s evidence. Algorithm 1 summarizes the step-level procedure. Appendix C.3 provides the optimizer prompt blocks. 5 ICALK@ECNU Table 1: Main test results (%). Bold and underlining mark the highest and second-highest distinct scores in each column within each target model. Subscripts on TextualRL scores show percentage-point gains over the corresponding Base Model. ModelSkill SourceSearchQASpreadsheetOfficeQADocVQALiveMathALFWorld GPT-5.5 Base Model77.741.833.178.837.683.6 Human skill81.872.966.990.138.491.8 LLM skill80.943.251.789.640.093.3 Trace2Skill82.449.665.790.652.087.3 TextGrad81.441.142.087.249.282.8 GEPA84.873.663.989.143.285.8 SkillOpt87.380.772.191.266.995.5 TextualRL88.9 +11.2 82.5 +40.7 72.7 +39.6 92.5 +13.7 67.2 +29.6 95.5 +11.9 GPT-5.4 Base Model76.941.450.077.636.875.4 Human skill80.159.359.388.541.674.6 LLM skill79.446.120.490.436.883.6 Trace2Skill81.253.251.289.340.073.1 TextGrad82.538.654.785.336.678.4 GEPA82.461.160.488.341.674.6 SkillOpt83.1 62.562.891.244.091.0 TextualRL85.5 +8.6 63.6 +22.2 64.5 +14.5 91.7 +14.1 47.2 +10.4 91.8 +16.4 Qwen3.5- 4B Base Model68.19.314.586.922.430.6 Human skill66.316.422.787.818.428.4 LLM skill65.010.420.988.028.855.2 Trace2Skill68.519.316.388.027.264.9 TextGrad60.713.920.985.610.653.7 GEPA68.616.922.785.128.860.4 SkillOpt71.2 23.929.789.052.081.3 TextualRL73.9 +5.8 27.1 +17.8 32.0 +17.5 89.3 +2.4 56.0 +33.6 82.1 +51.5 Qwen3.6- 35B-A3B Base Model72.738.245.987.631.259.7 Human skill74.144.341.988.429.644.8 LLM skill72.642.946.588.424.860.4 Trace2Skill75.433.232.090.429.670.9 TextGrad76.422.933.784.57.267.9 GEPA75.845.443.688.031.273.9 SkillOpt80.3 47.547.191.441.682.1 TextualRL81.2 +8.5 52.1 +13.9 49.4 +3.5 92.8 +5.2 44.8 +13.6 83.6 +23.9 Qwen3.8- 27B Base Model73.643.643.088.534.450.8 Human skill74.948.643.689.032.847.8 LLM skill74.546.442.488.832.059.7 Trace2Skill76.945.043.090.437.672.4 TextGrad77.639.336.188.235.256.7 GEPA78.350.744.289.040.075.4 SkillOpt80.456.446.593.348.082.1 TextualRL82.7 +9.1 60.7 +17.1 47.7 +4.7 94.4 +5.9 56.0 +21.6 84.3 +33.5 4 Experiments 4.1 Experimental Setups Benchmarks. We evaluate TextualRL on six benchmarks spanning question answering, document understanding, mathematical reasoning, and interactive task execution. SearchQA [22] requires answering questions from pro- vided search snippets. Spreadsheet [23] evaluates workbook manipulation through executable Python programs. OfficeQA [24] requires reasoning over enterprise documents using local file tools and supplied source-page evidence. DocVQA [25] pairs questions with document images. LiveMathematicianBench [26], abbreviated as LiveMath, tests mathematical reasoning through multiple-choice questions. ALFWorld [27] evaluates sequences of textual actions for completing household tasks. We use the released benchmark splits from SkillOpt [8], with dataset details, split sizes, and evaluation metrics provided in Appendix B.1. The training split supplies trajectories for context optimization, while validation scores determine step-level update acceptance and checkpoint selection. The test split is used only to evaluate the selected contexts. 6 ICALK@ECNU Table 2: Ablations on Qwen3.8-27B (%). Full TextualRL is the shared reference for both groups. SettingSearchQA Spreadsheet OfficeQA DocVQA LiveMath ALFWorld TextualRL (full)82.760.747.794.456.084.3 Groupwise Policy Critique w/o outcome routing81.858.946.592.553.683.6 Single rollout (K = 1)80.757.546.593.350.482.1 Cross-Group Policy Update Intra-group cards only82.159.347.193.053.683.6 Remove cards without edits80.958.245.993.654.482.8 w/o proposal review81.457.145.492.850.482.8 Baselines.We compare TextualRL with seven baselines in Table 1. Base Model evaluates the frozen target using the original LLM. Human skill uses manually written task guidance, while LLM skill uses a skill generated in a single pass. These baselines measure the benefit of adding static instructions without iterative optimization. Trace2Skill [7] extracts lessons from individual training trajectories and consolidates them into skills. TextGrad [5] updates text using language-model feedback as textual gradients. GEPA [6] uses reflective evolutionary search to improve prompts from execution feedback. SkillOpt [8] aggregates edit proposals from minibatches and selects bounded skill updates through validation. These main comparisons use the benchmark-specific direct-chat setting, including the tools or environment interactions required by each task. Implementation Details.We use five frozen target models: GPT-5.5, GPT-5.4, Qwen3.5-4B, Qwen3.6-35B-A3B, and Qwen3.8-27B. A separate context is optimized for every target–benchmark pair. Unless otherwise specified, GPT-5.5 serves as the optimizer. The target model’s parameters remain unchanged during both optimization and evaluation. Starting from the benchmark-specific initial context, we optimize for four epochs with up to 40 tasks per batch and K = 4rollouts per task. A full batch therefore contains 160 trajectories. Analysis and merge minibatches have size eight, subject to the task-group constraints in Section 3.2. The edit budget follows a cosine schedule from four to two edits. The within-epoch buffer provides feedback to subsequent critique calls. Additional implementation details appear in Appendix B. 4.2 Main Results Table 1 compares TextualRL with the seven baselines across all five target models. TextualRL achieves the highest six-benchmark average for every target, with gains of 11.4–24.4 percentage points over Base Model and 0.9–3.2 points over the strongest baseline SkillOpt. It outperforms all baselines in 29 of the 30 target–benchmark settings and ties SkillOpt on GPT-5.5 with ALFWorld. For GPT-5.5, the average score increases from 58.8% without a skill to 83.2% with TextualRL. For Qwen3.8-27B, the corresponding scores are 55.6% and 71.0%. The improvements therefore extend across both GPT and Qwen targets, with a separately optimized context for each target and benchmark. Specifically, TextualRL raises Qwen3.5-4B from 30.6% without a skill to 82.1% on ALFWorld, a gain of 51.5 percentage points and the largest improvement over Base Model. This result shows that optimizing context can substantially improve task completion even for the smallest frozen target. On Qwen3.8-27B, TextualRL reaches 56.0% on LiveMath and 60.7% on Spreadsheet, exceeding the strongest baseline by 8.0 and 4.3 points, respectively. The advantage is more modest on DocVQA, ranging from 0.3 to 1.4 points over the strongest baseline across the five targets. 4.3 Ablations To evaluate the contribution of each module, we conduct ablation studies on Qwen3.8-27B and report the results in Table 2. For Groupwise Policy Critique, w/o outcome routing pools all trajectories collected in a training step and partitions them directly into analyst minibatches of eight trajectories, without first separating them into mixed-outcome, all-success, and all-failure groups. This compares outcome-aware critique with direct minibatch analysis. We also evaluate a single rollout per task (K = 1). For Cross-Group Policy Update, intra-group cards only restricts the review of each merged edit to evidence cards from its originating critique groups. We additionally remove cards produced without edit proposals or disable cross-group proposal review. The results show that removing outcome routing reduces the six-benchmark average by 1.5 percentage points, while usingK = 1reduces it by 2.5 points. These results support the value of both organizing critique by outcome patterns and collecting multiple rollouts per task in Groupwise Policy Critique. For Cross-Group Policy Update, restricting review to cards from the originating critique groups reduces the average by 1.2 points, while removing cards without 7 ICALK@ECNU Table 3: Mean± sample standard deviation over three independent runs on Qwen3.8-27B (%). MethodSearchQASpreadsheetOfficeQADocVQALiveMathALFWorldOverall SkillOpt 79.93±0.47 56.55±0.54 45.93±0.59 92.87±0.41 47.73±1.22 81.59±0.86 67.43±0.33 TextualRL82.62±0.1660.24±0.5547.09±0.5894.03±0.4156.00±0.8084.08±1.1470.68±0.26 Table 4: Effect of training rollout count K on Qwen3.8-27B (%). Method KSearchQA Spreadsheet OfficeQA DocVQA LiveMath ALFWorld Overall TextualRL 180.757.546.593.350.482.168.4 281.458.947.793.653.682.869.7 482.760.747.794.456.084.371.0 882.960.048.894.156.085.171.1 Table 5: Test results (%) with GPT-5.5 under different agent harnesses. HarnessSkill SourceSearchQASpreadsheetOfficeQADocVQALiveMath Codex Base Model81.827.538.387.235.2 Human skill84.150.740.088.848.8 LLM skill83.425.034.389.845.6 EvoSkill61.467.542.489.363.2 SkillOpt87.385.051.192.278.4 TextualRL88.6 +6.8 85.7 +58.2 56.4 +18.1 92.8 +5.6 76.8 +41.6 Claude Code Base Model81.922.157.686.640.8 Human skill83.737.166.388.044.0 LLM skill82.437.156.489.644.8 EvoSkill84.075.070.387.252.0 SkillOpt85.9 80.471.590.156.5 TextualRL87.1 +5.2 83.9 +61.8 73.3 +15.7 90.9 +4.3 57.6 +16.8 edits reduces it by 1.7 points. These results support using evidence beyond a proposal’s originating groups and retaining observations that do not directly produce an edit. Disabling cross-group proposal review reduces the average by 2.7 points, highlighting the value of using evidence to assess and refine proposals before updating the context. Full TextualRL achieves the highest score on every benchmark in both groups. 4.4 Additional Experiments Run stability.To assess variation across runs, we repeat context optimization three times for TextualRL and SkillOpt on Qwen3.8-27B, keeping the data splits and evaluation protocol fixed. Each run independently selects its checkpoint by validation score before test evaluation. Table 3 reports the mean and sample standard deviation across runs. TextualRL achieves an overall mean of 70.68%, compared with 67.43% for SkillOpt, and has a higher mean score on all six benchmarks. The largest mean gain is on LiveMath, at 8.27 points. The overall advantage ranges from 3.18 to 3.36 points across the three runs, showing that the improvement persists across repeated optimization runs. Rollout count. To examine the effect of trajectory sampling, we vary the number of rollouts per training task over K ∈1, 2, 4, 8on Qwen3.8-27B. Table 4 shows that the overall score increases from 68.4% atK = 1to 71.0% at K = 4. IncreasingKfrom four to eight doubles the rollout count per task but adds only 0.1 percentage points to the overall score. The additional rollouts do not improve every benchmark: Spreadsheet and DocVQA decrease by 0.7 and 0.3 points, respectively, while LiveMath remains unchanged. In this setting,K = 4captures most of the observed improvement with half the rollouts per task required by K = 8, so we use it as the default setting. Agent harnesses. To evaluate TextualRL beyond direct chat, we use GPT-5.5 within the Codex and Claude Code harnesses on five benchmarks, excluding ALFWorld. Table 5 includes the harness baselines reported by SkillOpt, with EvoSkill [17] as an additional skill evolution baseline. TextualRL achieves five-benchmark averages of 80.1% and 78.6% under Codex and Claude Code, respectively, improving over Base Model by 26.1 and 20.8 percentage points. Relative to SkillOpt, the average gains are 1.3 and 1.7 points. TextualRL obtains the highest score in nine of the ten harness–benchmark settings. The exception is Codex LiveMath, where TextualRL scores 76.8% versus SkillOpt’s 78.4%. These results support the effectiveness of context optimization across both agent harnesses. 8 ICALK@ECNU Table 6: Skill transfer from Qwen3.8-27B to two target models (%). Target modelSkillSearchQA Spreadsheet OfficeQA DocVQA LiveMath ALFWorld Overall Qwen3.6- 35B-A3B Origin81.252.149.492.844.883.667.3 Transferred81.453.248.393.347.283.667.8 Qwen3.5-4B Origin73.927.132.089.356.082.160.1 Transferred74.327.932.090.155.282.160.3 Table 7: Optimizer comparison with Qwen3.8-27B as the fixed target (%). OptimizerSearchQA Spreadsheet OfficeQA DocVQA LiveMath ALFWorld Overall GPT-5.582.760.747.794.456.084.371.0 Qwen3.8-27B81.558.646.593.954.483.669.7 01234 Epoch 0.40 0.45 0.50 0.55 0.60 0.65 Score (a) OfficeQA 01234 Epoch 0.84 0.86 0.88 0.90 0.92 0.94 0.96 0.98 1.00 (b) DocVQA 01234 Epoch 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 (c) LiveMath Train rolloutValidation bestUnseen test Figure 3: Epoch curves for TextualRL on Qwen3.8-27B. Skill transfer.To test whether learned skills remain effective beyond the model used during optimization, we transfer the benchmark-specific skills learned with Qwen3.8-27B as the target and GPT-5.5 as the optimizer to Qwen3.6- 35B-A3B and Qwen3.5-4B. Table 6 compares skills optimized for each receiving model (Origin) with the transferred skills, which are used without further context optimization. Transfer yields a slightly higher overall score on both receiving models and matches the original skill on ALFWorld. The gains are not uniform: OfficeQA decreases by 1.1 percentage points on Qwen3.6-35B-A3B, and LiveMath decreases by 0.8 points on Qwen3.5-4B. These results suggest that the learned guidance can be applied across the tested Qwen models while retaining its overall effectiveness without additional context optimization. Optimizer choice. To examine whether the observed gains depend on using GPT-5.5 as the optimizer, we also optimize skills with Qwen3.8-27B while keeping the target fixed as Qwen3.8-27B. Table 7 reports an overall score of 69.7% with the Qwen optimizer, compared with 71.0% with GPT-5.5. GPT-5.5 produces higher scores on all six benchmarks, with an overall advantage of 1.3 percentage points. Nevertheless, the Qwen optimizer improves over the Base Model in Table 1 on every benchmark, with an average gain of 14.1 points. TextualRL therefore remains effective when the same model serves as both target and optimizer in this setting, although the optimizer choice affects the final task performance. Training dynamics. To examine when context optimization yields gains and whether additional epochs continue to improve unseen performance, we track training, validation, and test scores across four epochs on Qwen3.8-27B. Figure 3 shows that most test improvement occurs early, with different gains across benchmarks. By epoch 2, DocVQA improves from 88.5% to 93.9%, and LiveMath from 34.4% to 54.4%. Their validation scores then plateau, while test scores fluctuate within 93.6–94.4% and 54.4–56.0%, respectively. OfficeQA reaches 47.1% on test after the first epoch and shows no sustained improvement thereafter, despite a validation score of 62.5%. Thus, further optimization does 9 ICALK@ECNU not consistently improve unseen performance. On OfficeQA and LiveMath, validation scores improve much more than test scores. This highlights a generalization challenge: instructions favored by a fixed validation set may be less effective on unseen tasks. 5 Conclusion We introduce TextualRL, a gradient-free reinforcement learning framework for optimizing the textual context of a frozen LLM agent through task rewards. Groupwise Policy Critique uses same-task rollout comparisons and cross-task analysis to propose context edits according to the available outcome patterns. Cross-Group Policy Update reviews these proposals against pooled trajectory evidence, including observations that produce no edit, to refine their content and applicability before validation-based selection. Across five target models and six benchmarks, TextualRL achieves the highest six-benchmark average for every target. Ablation studies and additional experiments support the effectiveness of both modules and the consistency of the observed improvements. References [1] Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237–285, 1996. doi: 10.1613/jair.301. URLhttps://doi.org/10.1613/ jair.301. [2] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 8634–8652. Curran Associates, Inc., 2023. doi: 10.52202/075280-0377. URLhttps://proceedings.neurips.c/paper_ files/paper/2023/file/1b44b878b782e6954cd888628510e90-Paper-Conference.pdf. [3]Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Y. Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. In The Fourteenth International Conference on Learning Representations, 2026. URLhttps://proceedings.iclr.c/paper_files/paper/2026/ hash/8a94f6f922d995d7d3f4ebf4143e442-Abstract-Conference.html. [4]Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957–7968. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.494. URL https://aclanthology.org/2023.emnlp-main.494/. [5]Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative AI by backpropagating language model feedback. Nature, 639(8055):609–616, 2025. doi: 10.1038/s41586-025-08661-4. URL https://w.nature.com/articles/s41586-025-08661-4. [6]Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Ar- nav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab.GEPA: Reflective prompt evo- lution can outperform reinforcement learning.In International Conference on Learning Representa- tions, pages 8479–8565, 2026. URLhttps://proceedings.iclr.c/paper_files/paper/2026/hash/ 0e9e708b6f48e14fd0ac29e167413f76-Abstract-Conference.html. [7]Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2Skill: Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158, 2026. URL https://arxiv.org/abs/2603.25158. [8]Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. SkillOpt: Executive strategy for self- evolving agent skills. arXiv preprint arXiv:2605.23904, 2026. URL https://arxiv.org/abs/2605.23904. [9]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. doi: 10.48550/arXiv.2402.03300. URLhttps://arxiv.org/ abs/2402.03300. [10]Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642. AAAI Press, March 2024. doi: 10.1609/aaai.v38i17.29936. URLhttps://ojs.aaai.org/ index.php/AAAI/article/view/29936. 10 ICALK@ECNU [11]Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. ReasoningBank: Scaling agent self-evolving with reasoning memory. In International Conference on Learning Representations, pages 94327–94354, 2026. URLhttps://proceedings.iclr.c/paper_ files/paper/2026/hash/980ea04d23d1f6908964eba2a74afe45-Abstract-Conference.html. [12]Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.346. URL https://aclanthology.org/2020.emnlp-main.346/. [13]Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=92gvk82DE-. [14]Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Sai- ful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, volume 2024, pages 54928–54958, 2024. URLhttps://proceedings.iclr.c/paper_files/paper/2024/file/ f1cf02ce09757f57c3b93c0db83181e0-Paper-Conference.pdf. [15]Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9340–9366, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.525. URLhttps://aclanthology.org/2024. emnlp-main.525/. [16]Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic Cheatsheet: Test-time learning with adaptive memory. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7080–7106. Association for Computational Linguistics, March 2026. doi: 10.18653/v1/2026.eacl-long.333. URLhttps://aclanthology. org/2026.eacl-long.333/. [17]Salaheddin Alzubi, Noah Provenzano, Jaydon Bingham, Weiyuan Chen, and Tu Vu. EvoSkill: Automated skill discovery for multi-agent systems. arXiv preprint arXiv:2603.02766, 2026. URLhttps://arxiv.org/abs/ 2603.02766. [18]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2210.03629. [19]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Her- mann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative refinement with self- feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 46534–46594. Curran Associates, Inc., 2023. doi: 10.52202/075280-2019. URLhttps://proceedings.neurips.c/paper_files/paper/2023/file/ 91edff07232fb1b55a505a9e9f6c0f3-Paper-Conference.pdf. [20] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 62138–62160. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/zhou24r.html. [21]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=ehfRiF0R3a. [22]Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. SearchQA: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179, 2017. URL https://arxiv.org/abs/1704.05179. 11 ICALK@ECNU [23]Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. SpreadsheetBench: Towards challenging real world spreadsheet manipulation. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 94871–94908. Curran Associates, Inc., 2024. doi: 10.52202/079017-3007. URLhttps://proceedings.neurips.c/paper_files/paper/2024/file/ ac840df270ac537d74530a15c332684-Paper-Datasets_and_Benchmarks_Track.pdf. [24]Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins, Ivan Zhou, Cindy Wang, Ashutosh Baheti, Owen Oertell, Jacob Portes, Sam Havens, Erich Elsen, Michael Bendersky, Matei Zaharia, and Xing Chen. OfficeQA Pro: An enterprise benchmark for end-to-end grounded reasoning. arXiv preprint arXiv:2603.08655, 2026. URL https://arxiv.org/abs/2603.08655. [25]Minesh Mathew, Dimosthenis Karatzas, and C.V. Jawahar. DocVQA: A dataset for VQA on document im- ages. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2200–2209, January 2021. URLhttps://openaccess.thecvf.com/content/WACV2021/html/Mathew_ DocVQA_A_Dataset_for_VQA_on_Document_Images_WACV_2021_paper.html. [26]Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, and Nima Mesgarani. LiveMathematicianBench: A live benchmark for research-level mathematical reasoning with proof sketches. arXiv preprint arXiv:2604.01754, 2026. URL https://arxiv.org/abs/2604.01754. [27] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2010.03768. 12 ICALK@ECNU Algorithm 1 A step of TextualRL with fixed target parameters Require:ContextC, recorded validation scores, training batchB, trajectories per taskK, validation setV, edit budget L, shared history H 1: For each x∈ B, collect D x under C and computer x (Equation 2) 2: Route each D x usingr x , then consolidate duplicate trajectories within the task 3: Form groupsG: one per mixed task and multiple tasks per homogeneous group 4: In parallel, obtain (P g ,E g ) for each g ∈G from(x,D x ,r x ) : x∈ g, C, and H 5: Pool proposals P and valid current-step cards E independently 6: Consolidate P hierarchically and review the candidates against E to obtain e P 7: Select at most L edits from e P and filter dependencies on unavailable runtime information 8: if no edits remain then 9: return C,s▷ No candidate update 10: end if 11: Assemble ∆C and apply it to obtain C ′ ← C + ∆C 12: Evaluate v ′ ← b V (C ′ ) onV 13: if v ′ > s then 14:Adopt C ′ ,v ′ and update the best validation-scored snapshot when improved 15: else 16:Retain C,s and record the rejected edit combination and its score 17: end if 18: Add rollout and rejection feedback to H for subsequent proposals A Method Implementation Details Proposal consolidation.Mixed-task and all-failure proposals enter the corrective branch, while all-success proposals enter the preservation branch. The optimizer recursively merges proposal batches within each branch before combining the two branches. The final merge prioritizes corrective edits unless they conflict with a supported success pattern. Edit application. Patch operations are applied deterministically while preserving protected scaffold regions. The per-edit dependency filter uses explicit patterns with limited negation handling to detect dependencies on information unavailable during target execution, such as gold answers, grader feedback, or validation scores. It removes an affected edit as a whole and retains other valid edits from the same proposal. Evidence provenance.We retain evidence cards whose task and rollout references resolve to the displayed trajectories and whose supporting quotes occur in the referenced trace. Repeated cards for the same task and trajectory are consolidated. References attached to reviewed edits are checked against the pooled cards. If no valid cards remain, the optimizer uses ordinary proposal merging. Repeated proposals.The step buffer screens exact or near-equivalent repetitions of a rejected edit combination at the combination level. An individual edit family is blocked only after it appears in at least three distinct rejected candidates and no accepted candidate. B Implementation and Data Notes B.1 Data Splits and Metrics Table 8 summarizes the data splits and evaluation metrics for all six benchmarks, with counts referring to the experi- mental subsets. We report binary task accuracy or success rate in percentages. SpreadsheetBench requires all test cases for a task to pass. DocVQA uses hard accuracy, with ANLS≥ 0.999counted as correct. Overall performance is the unweighted mean across the six benchmarks. For ALFWorld, we use the official SkillOpt release with 39 training, 18 validation, and 134 test environments. The released validation manifest selects 18 environments from the fullvalid_seenpool of 140, whereas the SkillOpt pa- per [8] describes 140 validation environments. Our update acceptance and checkpoint selection use the 18 environments in the released manifest. 13 ICALK@ECNU Table 8: Benchmark splits and evaluation metrics. BenchmarkTrainValidationTestMetric SearchQA4002001,400Normalized exact match SpreadsheetBench8040280All-case success OfficeQA5024172Normalized exact match DocVQA10753374Hard accuracy LiveMath3517125Choice accuracy ALFWorld3918134Task success B.2 Critique and Merge Batches In full TextualRL, the critique minibatch size of eight counts distinct task blocks, whereas the merge minibatch size counts proposal bundles. Each mixed task is analyzed separately with all of its unique sibling trajectories. Within each homogeneous route, task blocks are grouped into batches of eight. A final batch containing two to seven tasks is processed as is. If the final batch contains only one task, it is combined with the preceding batch, producing a batch of nine. A homogeneous route containing only one task is skipped. Outcome routing uses the originalKrollout outcomes before deduplication. Duplicate trajectories are consolidated only within the same task and outcome label, using the recorded conversation with whitespace normalized and timing and request metadata removed. Each retained trajectory keeps its multiplicity and original rollout references. Deduplication reduces the trajectories shown to the critic but does not changeK, the task’s outcome category, or the number of distinct tasks supporting a proposal. Hierarchical merging processes up to eight proposal bundles per call, and each bundle may contain multiple edits. A final batch of two to seven bundles is merged normally. A singleton bundle passes directly to the next level without a merge call. These batch sizes are separate from the edit budget, which bounds the number of edits applied to form the candidate context. For w/o outcome routing, we pool the step’s trajectories and form analyst minibatches of eight trajectories. We adapt the mixed-outcome analyst prompt by removing the requirements that trajectories belong to a single task and include both successes and failures. The resulting unified prompt asks the analyst to produce edit proposals and evidence cards from the available trajectories and outcomes. In intra-group cards only, each critique groupg ∈G t corresponds to one analyst call. Separate calls within the same outcome route remain distinct groups. The ablation retains hierarchical consolidation followed by evidence review during the final merge. For each merged edit, review is restricted to the union of evidence cards from the critique groups that contributed its source proposals. For example, an edit formed from proposals by groupsAandBcan useE A ∪E B , while cards from other groups are excluded. B.3 Generation and Evaluation Training rollouts use a temperature of 0.7, while validation and test inference use a temperature of 0. Top-p is left at the serving backend’s default. The random seed is set to 42. GPT-5.5 and GPT-5.4 use medium reasoning effort. For each target–benchmark pair, context optimization collectsK = 4rollouts per training task under the same current context, except in the rollout-count ablation. Validation selects the context used for test evaluation. The TextualRL scores in Table 1 are from individual optimization runs. The run-stability experiment separately repeats the complete optimization and validation-based checkpoint selection three times for TextualRL and SkillOpt on Qwen3.8-27B, with fixed data splits. Its reported variation is across independently optimized contexts, rather than repeated evaluations of a single context. Each rollout starts with a fresh interaction history and the same task inputs, with separate sample identifiers for its saved outputs. Spreadsheet rollouts operate on independent copies of the input workbook. OfficeQA rollouts start new conversations over the supplied documents. ALFWorld rollouts initialize and reset separate environments using the task’s game file, with environment seeds derived from the batch seed and environment index. 14 ICALK@ECNU Table 9: Token usage with Qwen3.8-27B as the target model with GPT-5.5 as optimizer. Iteration usage includes candidate validation. Inference usage sums input and visible output text per task. BenchmarkMethod Final tokens Recorded iteration tokens (M) Inference text tokens / task TargetOptimizerTotal SearchQA SkillOpt93731.337.6739.002,698.2 TextualRL2,42266.2613.5779.835,035.4 Spreadsheet SkillOpt2,90849.624.9954.609,044.2 TextualRL1,39732.998.3541.344,145.6 OfficeQA SkillOpt1,58944.184.7048.88105,784.5 TextualRL974142.877.21150.08137,714.1 DocVQA SkillOpt4154.600.635.22741.0 TextualRL65910.760.7911.561,132.2 LiveMath SkillOpt1,5341.591.292.887,889.8 TextualRL1,9134.984.349.328,410.2 ALFWorld SkillOpt5168.550.288.8324,929.9 TextualRL1,60025.210.8326.0543,786.2 B.4 Checkpoint Selection For each run, we select the context with the highest recorded validation score for test evaluation. We track this checkpoint separately from the active context after the final update. Checkpoint selection uses validation scores exclusively, and the selected context is then supplied to the frozen LLM agent for test tasks. B.5 Token Accounting The settings and selected skills for Table 9 follow the corresponding Qwen3.8-27B experiments in Table 1, with GPT-5.5 as optimizer. Final refers to the skill selected for inference. Table 9 reports recorded iteration subtotals. Target counts sum input and output tokens for training rollouts and candidate validation within saved steps. Optimizer counts cover optimization calls in those same steps. Totals include all saved steps, including those after the selected checkpoint. The accounting scope excludes subsequent unseen evaluation, initial validation outside these steps, and unrecorded recovery or retry usage. M denotes one million tokens. Totals are calculated from the original integer counts, and each displayed column is rounded independently. Inference counts use the Qwen3.8-27B tokenizer withadd_special_tokens=False. Input message text is accumu- lated across all saved decisions for each task, with repeated history counted at each decision. Output counts include the saved visible model text. Their sum is calculated before rounding and averaged over tasks with saved trajectories. Each token contributes equally to this measure of recorded text volume, independent of token prices, caching, or computational cost. TextualRL records 2.05–3.24 times SkillOpt’s iteration tokens on five benchmarks. CollectingK = 4trajectories per task incurs sampling cost to obtain comparative feedback, analogous to group-based RL such as GRPO [9]. Trajectory analysis, evidence review, and candidate validation also consume optimization tokens. Review reuses existing rollouts. At inference, these optimization procedures are absent, and token usage depends on the learned context and execution history. The corresponding increases are smaller, at 6.6–86.6%. Spreadsheet instead uses 24.3% fewer recorded iteration tokens and 54.2% fewer inference text tokens. B.6 Evidence Review Audit To examine how review uses recorded evidence, we audit runs with Qwen3.8-27B as target and GPT-5.5 as optimizer across all six benchmarks. We inspect all 76 saved steps, containing 464 merged edits and 578 raw evidence references. We randomly sample up to 20 reviewed edits per benchmark with seed 42, including all 13 available LiveMath edits, yielding 113 edits and 153 references. Two separate GPT-6 Astra sessions independently assess the same sample using an identical rubric, without seeing the original review labels, validation outcomes, or each other’s judgments. Each assesses card faithfulness to the available runtime text, the card’s relation to the final reviewed edit, and the grounding of the edit’s scope. 15 ICALK@ECNU Table 10: Evidence review audit on Qwen3.8-27B runs. Audit itemResultMeasurement Reference traceability99.13% (573/578)Card resolves and its quote matches the source trajec- tory. LLM relation agreement94.77% (145/153)Agreement between two GPT-6 Astra assessments,κ = 0.715. Reported review changes8.48% (37/436)Optimizer reports narrow or revise. Table 11: Unseen-test performance (%) of three context variants for selected updates. BenchmarkOld skillNew skill w/o reviewNew skill SearchQA81.5782.0782.71 OfficeQA45.3545.9347.09 DocVQA89.5790.3791.44 Table 10 separates source traceability from model assessment. References are checked against the current-step card pool after the existing card validation procedure, and all raw review references remain in the denominator. Relation labels are supports, limits, contradicts, irrelevant, and unclear, always evaluated against the final reviewed edit. The two assessments label 134/153 card statements as supporting the rule. Agreement measures consistency between two evaluations by the same model, rather than correctness against human judgments. These findings support the auditability of Cross-Group Policy Update. Traceable references allow readers to inspect the observations cited for individual rules, while the two assessments agree on a supporting relation for most sampled references (87.58%). The review therefore exposes links between evidence and edits that can be examined beyond the final task score. The optimizer reports narrowing or revision for 37 of the 436 edits with review metadata. This pattern is consistent with review serving as a selective refinement stage, where proposals can be retained after evidence comparison and changes can target their content or applicability. Effect of reviewed updates. To examine whether evidence review improves task performance beyond proposal consolidation, we evaluate three context variants for one selected update each on SearchQA, OfficeQA, and DocVQA, using Qwen3.8-27B as the target and GPT-5.5 as optimizer. Old skill is the context before the selected update. New skill w/o review applies the consolidated proposal without evidence review, while New skill applies the corresponding reviewed edit. Each comparison uses the same historical context and unseen-test split. Figure 4 illustrates how review refines the applicability of consolidated proposals. SearchQA protects proper names and titles during category extraction and removes the surname-only rule. OfficeQA removes the ambiguous-constant fallback while retaining a qualified exception for external constants. DocVQA preserves the Arabic-digit default, specifies the applicable field labels, and permits document-specific Roman numerals. In all three selected updates, review improves on an already beneficial proposal (Table 11) while clarifying the conditions and exceptions in the resulting guidance. 16 ICALK@ECNU (a) SearchQA Preserve exceptions to shortening New skill w/o review Categories. Do not include the generic noun if it is redundant with the question's scope. Surnames. If the question asks for a person's name and provides the first name, output only the surname ... New skill Category exception. Do not apply this to proper names or unique titles where the full phrase is the identifier (e.g., 'Golden Boy' remains 'Golden Boy', not just 'Golden'). Cross-group evidence. A success group preserves the complete title 'Dalai Lama', limiting category-based shortening. Adds a category exception for proper names and titles. Removes the surname-only rule. Reported unseen test (%) Old skill: 81.57 New skill w/o review 82.07 New skill 82.71 +0.64 p (b) OfficeQA Restrict the external-constant exception New skill w/o review If the constant is ambiguous or multiple values are plausible, state the assumption made. New skill If a required conversion factor or constant (e.g., historical exchange rates, physical constants) is not found in the corpus but is a well-established, unambiguous general knowledge fact for the specific period/entity, you may use it. Cross-group evidence. A failure case from another group cites an unverified, memorized exchange rate, highlighting the need to check whether an external value qualifies for the exception. The ambiguous-constant fallback is removed. Only well-established, unambiguous constants remain eligible under the exception. Reported unseen test (%) Old skill: 45.35 New skill w/o review 45.93 New skill 47.09 +1.16 p (c) DocVQA Respect document-specific numerals New skill w/o review When asked for a specific number, page, or ID, output the specific numeric value or identifier in Arabic digits. New skill For fields labeled 'no.', 'number', or 'ID', output the specific numeric value or identifier in Arabic digits (unless the document explicitly uses Roman numerals for that specific identifier). Cross-group evidence. A failure group records 'I' instead of '1', supporting the shared Arabic-digit default. Retains the Arabic-digit default, scopes the rule to specified field labels, and adds a document-specific Roman-numeral exception. Reported unseen test (%) Old skill: 89.57 New skill w/o review 90.37 New skill 91.44 +1.07 p Key excerpts; unchanged text omitted. Scores evaluate each complete skill on the same unseen test. Figure 4: Key rule changes before and after evidence review for the updates in Table 11. 17 ICALK@ECNU 01234 Epoch 0.72 0.74 0.76 0.78 0.80 0.82 0.84 Score (a) SearchQA 01234 Epoch 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 (b) Spreadsheet 01234 Epoch 0.40 0.50 0.60 0.70 0.80 0.90 1.00 (c) ALFWorld Train rolloutValidation bestUnseen test Figure 5: Epoch curves for TextualRL on SearchQA, Spreadsheet, and ALFWorld with Qwen3.8-27B. Epoch and score definitions follow Figure 3. C Experimental results C.1 Training Dynamics Epoch 0 denotes the Base Model. The training curve reports the mean rollout score within each epoch. Validation and test scores correspond to the best checkpoint selected by validation up to that epoch. Figure 5 shows that SearchQA and Spreadsheet also obtain most of their test gains in the first epoch, increasing from 73.6% to 81.4% and from 43.6% to 58.9%, respectively. Subsequent changes are smaller, with final scores of 82.5% and 60.7%. ALFWorld’s reported test score increases across all four epochs, from 50.8% to 84.3%, while validation reaches 100% at epoch 3. Together with Figure 3, these curves show that the timing and magnitude of improvements vary by benchmark. A validation-selected checkpoint can remain unchanged across epochs, with repeated test evaluations yielding different scores. 18 ICALK@ECNU (a) SearchQA Exact answer extraction Content: If a formatted document provides a Clue | Answer pair, extract the Answer span exactly. (b) Spreadsheet Formula handling Content: Unless the instruction explicitly states the cell must contain a live formula that recalculates, compute the result in Python and write the static value (number, string, date) to the cell. (c) OfficeQA Operand verification Content: For multi-step computations, first write out a complete operand ledger listing every value with its period and source, verify the ledger against the retrieved table, and only then perform the arithmetic. (d) DocVQA Table grounding Content: For tables, identify the specific row label and column header mentioned in the question, then extract the value at their intersection. (e) LiveMath Theorem scope Content: Be cautious about asserting uniform-in-time or global bounds (e.g., constant C independent of T) unless the theorem explicitly provides them. (f) ALFWorld State recovery Content: If a take or move action returns "Nothing happens," immediately issue an inventory action to confirm what you are currently carrying. Figure 6: Representative instructions from the optimized contexts for six benchmarks. Original wording is preserved, with formatting standardized for readability. C.2 Optimized Contexts To illustrate what TextualRL encodes in the optimized context, Figure 6 presents one instruction excerpt from each of the six benchmarks for Qwen3.8-27B. These examples specify concrete operations, such as verifying source values before arithmetic and checking inventory after a failed action. They also define when a rule applies, including when to write static spreadsheet values and when a theorem supports a claimed bound. 19 ICALK@ECNU C.3 Optimizer Prompts The boxes below reproduce the method-specific system-prompt blocks used by TextualRL, with line wrapping and formatting adjusted for readability. In the implementation, skill denotes the editable contextC. Each outcome-specific block defines the corresponding critique task, and all three routes share the evidence-card instruction. The final block guides proposal review against the pooled evidence. C.3.1 Outcome Analysis The user message supplies the current contextC t , the edit budget, within-epoch historyH t , and the assigned task trajec- tories with their outcomes. The analyst returns a JSON object containing a proposedpatchandcross_group_cards. Mixed outcomes: within-task contrast This call contains all unique trajectories from exactly one task whose K sibling rollouts include both successes and failures. Compare complete trajectories, locate the first observable behavioral divergence, and emit at most one atomic repair or preservation rule. The rule must be supported by the within-task contrast rather than by task literals or final evaluator labels. If no plausible causal divergence is visible, return an empty patch. All-success outcomes: cross-task preservation Each Task Block is one independent task whose sibling rollouts all succeeded. Multiple unique trajectories inside one Task Block are correlated alternatives, not independent votes. First compare Task Blocks semantically. Emit a rule only for an inference-time behavior shared by at least two distinct task_ids. Treat the result as preservation evidence, not proof that the behavior caused success. If no cross-task invariant is visible, return an empty patch. Set evidence_kind to consensus and generality_scope to cross_task. All-failure outcomes: cross-task repair Each Task Block is one independent task whose sibling rollouts all failed. Multiple unique trajectories inside one Task Block are correlated attempts, not independent votes. First cluster Task Blocks by a shared, observable failure mechanism. Emit a cautious repair hypothesis only when at least two distinct task_ids expose the same mechanism. Do not infer a correct answer or task-specific procedure from failure alone. If no cross-task mechanism is visible, return an empty patch. Set evidence_kind to consensus and generality_scope to cross_task. 20 ICALK@ECNU C.3.2 Evidence and Review The shared evidence instruction is included in all three critique routes. For cross-group review, the user mes- sage suppliesC t , the consolidated candidate patches, and the pooled current-step cardsE t . The review status insufficient_evidence retains the proposal’s original scope and uncertainty, as described in Section 3.3. Shared instruction: trajectory evidence cards Keep the current route’s proposal rules and edit budget. In the same JSON response, alongside patch, also return cross_group_cards, a list of objects: task_id, rollout_id, quote, condition, behavior, observed_effect. For each displayed unique trajectory, report at most ONE salient behavioral observation, even when no edit is warranted (return patch with an empty edits list in that case). quote must be a short verbatim runtime action/observation span, up to 800 characters. Describe observable preconditions, the action, and its visible effect in at most 300 characters each. Use "unknown" when the trace does not establish a condition or effect. Do not invent a successful repair. Choose behavior relevant to a proposed repair or worth preserving. For mixed groups expose both sides of the divergence; for homogeneous groups describe the actual observed behavior, not a presumed cause of the final score. Only cite Task Block IDs and displayed trajectory IDs in this call. Do not cite gold answers, evaluator feedback, final grading labels, hidden siblings, or previous-step summaries as runtime evidence. Cards are evidence for later semantic comparison, not extra edits or independent votes. Do not put any card metadata or task IDs into executable skill content. Cross-Group Policy Update: evidence-based proposal review Alongside the candidate edits, you receive behavioral evidence from the current training step’s mixed, all-success, and all-failure task groups. Associate each candidate ONLY with semantically relevant actions and observable preconditions. Benchmark names, task-ID similarity, and matching outcome labels are not evidence of relevance. Do not force all three group types to support every candidate. Mixed evidence can suggest an improvement direction. Independent failure tasks can corroborate an observed failure mechanism, not prove a repair. Successful behavior can support preservation or LIMIT a repair’s scope: do not prohibit a behavior merely because it also appears in a failed task. Distinguish a genuine contradiction under comparable conditions from different valid preconditions. Use an observable condition in executable content when narrowing a rule; keep supported exceptions and unrelated successful behavior intact. Only revise an existing proposed behavior; do not invent unrelated edits to use every card. If no relevant cross-group evidence exists, keep the original evidence scope and uncertainty rather than inventing support or discarding the edit solely for missing evidence. All-success and all-failure proposals remain eligible. Maintain the normal output patch format and edit budget. For EACH output edit, add training-only cross_group_review with decision (retain, narrow, revise, or insufficient_evidence), reason, and evidence (a list of task_id, rollout_id, relation [supports, limits, contradicts], reason). Cite only cards actually used. Correlated siblings are not independent task votes. These records do not automatically change support_count, ranking, acceptance, or causal confidence. Keep metadata out of skill content. Existing protected sections and runtime observability restrictions still apply. C.4 Complete Update Example Figure 7 traces an accepted SearchQA update using GPT-5.5 as both the frozen target and the optimizer. For the same declarative clue, three trajectories identify Henrik Ibsen, while one returnsTrue. The mixed analyst proposes resolving the implied entity and reserving Boolean answers for explicit verification questions. The optimizer reviews the consolidated proposal against current-step evidence from all three outcome groups and retains this rule alongside complementary extraction guidance. The resulting patch is appended to a context with no learned rules and accepted after validation accuracy rises from 69.0% to 77.0%. This improvement is measured for the complete patch. 21 ICALK@ECNU 1 Task and mixed trajectories SearchQA / K = 4 Question. "En Folkefiende" is the Norwegian title of his 1882 drama "An Enemy of the People" SUCCESS / 3 OF 4 Henrik Ibsen Resolves the person referred to by "his". FAILURE / 1 OF 4 True Treats the clue as a truth judgment. 2 Proposed edit Mixed analyst / append When a question is phrased as a declarative clue or sentence fragment rather than an explicit interrogative, identify the entity that completes or resolves the clue. In particular, resolve pronouns such as “his,” “her,” or “their” to the named person or entity supported by the context. Do not answer True or False unless the prompt explicitly asks for verification or a yes/no judgment. 3 Evidence-based proposal review Mixed Observed: Ibsen vs. True. Supports: resolve the entity implied by a declarative clue. All-success Observed: John Bonham from a direct clue match. Supports: prioritize the direct clue-answer match. All-failure Observed: France where an adjective was required. Supports: match the requested answer type. Retain the consolidated proposal, including the rule from the mixed analyst. Selected card summaries above. Final review cites 11 cards across all three groups. 4 Textual patch and selection One append operation Precise Answer Extraction Highlight: rule carried from the mixed proposal • First determine the requested answer type, such as a person, entity, affiliation, location, category, attribute, or phrase completion. • Match the question's most distinctive combination of constraints, such as dates, roles, definitions, relationships, or quantities, to the provided context. Account for supported paraphrases, translations, aliases, and epithets. • If the prompt is a declarative clue, quotation, or sentence fragment, identify the entity or phrase that completes it. Resolve pronouns such as "his," "her," and "their" from the provided context. Do not answer True or False unless the prompt explicitly requests verification or a yes/no judgment. • Prefer an exact completion or directly stated answer that links the full clue to one candidate over passages that merely share isolated keywords. When useful, check additional passages for corroboration, while rejecting tangential or merely prominent related details. • Return the shortest complete span that unambiguously answers the question. Omit unnecessary surnames, titles, articles, categories, explanations, and parenthetical qualifiers unless they are required to distinguish the answer or are part of the requested form. • When the answer is explicitly stated, preserve its wording, spacing, punctuation, capitalization, and hyphenation rather than adding inferred words or rewriting it. Put only the final answer inside the requested <answer>...</answer> tags when that format is specified. VALIDATION ACCURACY 69.0% → 77.0% 138/200 → 154/200 ACCEPTED Accuracy change is measured for the complete patch. Figure 7: A complete context update example on SearchQA. 22

Instant research discovery