Paper deep dive

RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation

Hengbo Xiao, Boyao Zhang, Purui Liu, Yuxuan Zheng, Haoran Yin, Haibo Liu, Fan Zhang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 84

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 10/11/2026, 3:48:43 AM

Summary

The paper introduces RewardWeaver, a self-evolving reward adaptation framework for language agents in long-horizon interactions. It addresses the challenge of sparse terminal feedback and difficult credit assignment by maintaining a validated capability space with fixed semantics. The framework performs outcome-grounded backward attribution on low-outcome trajectories to identify recurrent, policy-controlled capability bottlenecks. It then dynamically selects process rewards (Rubrics) for the next training stage, reallocating training credit to current bottlenecks. Recurrent failures outside the existing capability space trigger a controlled expansion procedure. RewardWeaver achieves state-of-the-art results on SOTOPIA, AmazonHistoryPrice, and a Sales Benchmark.

Entities (10)

AmazonHistoryPrice · benchmark · 100%RewardWeaver · methodframework · 100%SOTOPIA · benchmark · 100%Fan Zhang · person · 90%Hengbo Xiao · person · 90%Process Rewards · concept · 90%RLVR · technique · 90%Rubrics · concept · 90%Sales Benchmark · benchmark · 90%AMPO · baseline-method · 80%

Relation Signals (8)

RewardWeaver → evaluateson → SOTOPIA

confidence 100% · We evaluate RewardWeaver on SOTOPIA

RewardWeaver → evaluateson → AmazonHistoryPrice

confidence 100% · We evaluate RewardWeaver on ... AmazonHistoryPrice

RewardWeaver → achieves → State-of-the-art results

confidence 95% · Across social interaction, bilateral bargaining, and domain-specific sales, REWARDWEAVER establishes new state-of-the-art (SOTA) results.

RewardWeaver → dynamicallyselects → Process Rewards

confidence 95% · and dynamically selects the corresponding process rewards for the next stage.

RewardWeaver → employs → Outcome-grounded backward attribution

confidence 95% · After each training stage, it performs outcome-grounded backward attribution on low-outcome trajectories

RewardWeaver → uses → Validated Capability Space

confidence 95% · RewardWeaver maintains a validated capability space in which the semantics of admitted Rubrics remain fixed

RewardWeaver → evaluateson → Sales Benchmark

confidence 90% · We evaluate RewardWeaver on ... a newly constructed Sales Benchmark.

RewardWeaver → outperforms → AMPO

confidence 90% · it outperforms AMPO, the strongest prior method, on every metric under the same partner protocol

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in domains where task outcomes can be reliably evaluated, but long-horizon interaction remains challenging due to sparse terminal feedback and difficult credit assignment. Process rewards provide denser supervision, yet the capabilities most relevant for training can change as the policy evolves: a behavior that is easy to evaluate or frequently deficient need not be the bottleneck currently limiting task success. We introduce RewardWeaver, a self-evolving reward adaptation framework for language agents in long-horizon interaction. RewardWeaver maintains a validated capability space in which the semantics of admitted Rubrics remain fixed, and closes the loop between policy optimization, task evaluation, failure attribution, and reward adaptation. After each training stage, it performs outcome-grounded backward attribution on low-outcome trajectories, aggregates recurrent and policy-controlled capability bottlenecks, and dynamically selects the corresponding process rewards for the next stage. Recurrent failures not covered by the existing capability space trigger a separate, controlled expansion procedure. We evaluate REWARDWEAVER on SOTOPIA, Amazon?HistoryPrice, and a newly constructed Sales Benchmark. Across social interaction, bilateral bargaining, and domain-specific sales, REWARDWEAVER establishes new state-of-the-art (SOTA) results. Ablations further demonstrate the importance of dynamic reward allocation, failure-grounded attribution, and stable semantics for admitted capabilities.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

80,276 characters extracted from source content.

Expand or collapse full text

RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation Hengbo XiaoYoolee.ai Boyao ZhangUniversity of Science and Technology of China Purui LiuPeking University Yuxuan ZhengPeking University Haoran YinTianjin University Haibo LiuYoolee.ai Fan ZhangYoolee.ai fan.zhang@yoolee.ai Abstract Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in domains where task outcomes can be reliably evaluated, but long-horizon interaction remains challenging due to sparse terminal feedback and difficult credit assignment. Process rewards provide denser supervision, yet the capabilities most relevant for training can change as the policy evolves: a behavior that is easy to evaluate or frequently deficient need not be the bottleneck currently limiting task success. We introduce RewardWeaver, a self-evolving reward adaptation framework for language agents in long-horizon interaction. RewardWeaver maintains a validated capability space in which the semantics of admitted Rubrics remain fixed, and closes the loop between policy optimization, task evaluation, failure attribution, and reward adaptation. After each training stage, it performs outcome-grounded backward attribution on low-outcome trajectories, aggregates recurrent and policy-controlled capability bottlenecks, and dynamically selects the corresponding process rewards for the next stage. Recurrent failures not covered by the existing capability space trigger a separate, controlled expansion procedure. We evaluate RewardWeaver on SOTOPIA, AmazonHistoryPrice, and a newly constructed Sales Benchmark. Across social interaction, bilateral bargaining, and domain-specific sales, RewardWeaver establishes new state-of-the-art (SOTA) results. Ablations further demonstrate the importance of dynamic reward allocation, failure-grounded attribution, and stable semantics for admitted capabilities. A Preprint †footnotetext: ∗ Equal contribution. ♠ Project leader. † Corresponding author. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has enabled substantial progress in mathematics and code, where task outcomes can be reliably checked by rules, tests, or execution (Shao et al., 2024; Le et al., 2022; Guo et al., 2025). Long-horizon interaction, however, poses a fundamentally different optimization problem. In settings such as negotiation, social coordination, and collaborative decision making (Zhou et al., 2024a; Wang et al., 2026; Liu et al., 2025), each action changes the partner’s subsequent behavior and reshapes the interaction trajectory. Final outcomes emerge from sequences of interdependent decisions across many turns, making rewards sparse, delayed, and difficult to assign to specific behaviors (Arjona-Medina et al., 2019; Hung et al., 2019; Meulemans et al., 2023). A terminal score indicates whether an interaction succeeded, but certifies failure without localizing its cause and cannot by itself direct training. Rubric-based rewards and evaluators provide denser supervision by assessing intermediate behaviors against explicit, fine-grained criteria (Gunjal et al., 2026; Wei et al., 2026; Hashemi et al., 2024; Lee et al., 2025), and rubric construction now scales from instance-specific generation to reusable repositories (He et al., 2025; Li et al., 2026). Recent approaches further revise or evolve these criteria as the policy changes, tracking its current response distribution (Guan et al., 2026). However, tracking the response distribution is not tracking the failure distribution: a behavior may be frequent or easy to discriminate without being a recurrent cause of task failure, while a less frequent capability deficiency may constitute the actual performance bottleneck. The key question is not only how evaluation criteria should adapt, but what evidence should drive that adaptation. We argue that this evidence should be recurrent failures in low-outcome trajectories, rather than response characteristics alone. Figure 1 contrasts these reward-adaptation paradigms. The central challenge is not merely how to design better process rewards, but how to decide what the current policy should optimize next. This introduces an outer-loop optimization problem complementary to standard RL: the inner-loop optimizer determines how to improve the policy under a given reward, while the outer loop must determine which capabilities deserve training credit. Diagnostic evidence must therefore satisfy three conditions: it must be outcome-grounded, drawn from trajectories that failed rather than sampled uniformly; recurrent, reappearing across trajectories rather than reflecting incidental variation; and policy-controlled, attributable to the agent’s own behavior (Zhang et al., 2025a) rather than to the partner or the scenario. Isolated low scores satisfy none of these conditions. In this view, the task-level objective should remain stable, while process-level training credit shifts toward the capabilities that currently constrain performance. Figure 1: Reward Adaptation under Policy Evolution. (a) Task-specific Rubrics remain fixed throughout training. (b) External-LLM methods revise Rubrics as policy evolves. (c) Co-evolution methods jointly update Rubrics and the policy. (d) RewardWeaver grounds reward adaptation in recurrent task failures and prioritizes capability Rubrics according to current policy bottlenecks. We introduce RewardWeaver, a self-evolving reward adaptation framework for language agents in long-horizon interaction. RewardWeaver closes the loop between policy training, automatic evaluation, failure attribution, and reward adaptation. At each stage, it identifies recurrent capability bottlenecks from on-policy interaction failures and dynamically selects the corresponding Rubrics to construct the next process reward. Unlike prior self-evolving rubric methods that evolve what is evaluated, RewardWeaver evolves what is optimized: it preserves stable semantics for admitted capabilities while dynamically reallocating training credit across them. Recurrent failures that fall outside the existing Rubric space are incorporated only through controlled validation, allowing the reward to evolve without uncontrolled semantic drift. We evaluate RewardWeaver on SOTOPIA (Zhou et al., 2024a) and related social intelligence benchmarks against outcome-only reinforcement learning, static capability prioritization, and alternative attribution and rubric-design strategies. RewardWeaver improves the primary evaluation metrics and achieves new state-of-the-art (SOTA) performance under matched training and evaluation protocols. Further ablation, sensitivity, and qualitative analyses show that failure-grounded attribution, adaptive credit reallocation, and stable semantics for admitted capabilities are key contributors to the gains. Our contributions are summarized as follows: (1) We formulate reward adaptation in long-horizon interaction as an outer-loop optimization problem that determines which process capabilities should be prioritized next. (2) We propose RewardWeaver, which dynamically reallocates training credit through failure-driven capability-bottleneck identification while preserving stable task objectives and evaluation semantics. (3) We conduct systematic experiments demonstrating the advantages of self-evolving reward adaptation over outcome-only and static process-reward baselines, and isolate the contributions of RewardWeaver’s key components. 2 RewardWeaver Methodology Figure 2: The Overall Architecture of RewardWeaver. (a) The policy is optimized under the current reward; its on-policy low-outcome trajectories reveal recurrent capability bottlenecks that determine the next-stage allocation of training credit. (b) Failure attribution grounds bottleneck identification in concrete behavioral evidence. (c) A validated capability space provides stable semantics for admitted capabilities, while recurrent uncovered failures trigger controlled expansion. RewardWeaver separates reward semantics from reward relevance: a validated capability space determines what can be rewarded, while a stage-wise outer loop determines what should be rewarded. Training priorities can evolve with the policy without the semantics of admitted capabilities drifting, while the capability library itself expands only through a separate validated procedure. 2.1 Task Formulation At stage k, policy πk _k interacts with partner πpartner _partner: given interaction history hth_t, the two parties act alternately with at∼πk(⋅∣ht)a_t _k(· h_t) and bt∼πpartner(⋅∣ht,at)b_t _partner(· h_t,a_t). A completed trajectory τ receives a fixed task-level outcome Ro(τ)R_o(τ) and a stage-dependent process reward, Rk(τ)=Ro(τ)+1|Ak|∑d∈AkJ¯d(τ),R_k(τ)=R_o(τ)+ 1|A_k| _d∈ A_k J_d(τ), (1) where kD_k is the capability space available at stage k, Ak⊆kA_k _k the active subset, and J¯d(τ) J_d(τ) the trajectory-level score of capability evaluator JdJ_d; all active Rubrics enter with equal weight, so adaptation determines which capabilities receive credit rather than how much each receives. The terminal objective remains fixed; only process-level credit changes: standard RL determines how to optimize under RkR_k, i.e., (πk,Rk)→πk+1( _k,R_k)→ _k+1, whereas RewardWeaver determines which capabilities should receive training credit next. Throughout, capability denotes the behavioral concept, Rubric its frozen operational definition, and evaluator the scoring function JdJ_d; the active set is the subset of capabilities currently receiving process-level training credit, and a reward specification couples the active set with how its evaluators are composed. 2.2 Validated Capability Space Reward adaptation requires stable semantics for admitted capabilities. RewardWeaver operates over a validated space D of observable, policy-controllable behaviors, where each d∈d couples a frozen Rubric definition with an evaluator Jd(x)=Meval(pd,x)J_d(x)=M_eval(p_d,x), and pdp_d operationalizes the definition over interaction context x. Construction has two stages: candidates are first induced from interaction evidence, consolidated across runs, and filtered for generality, operationality, and semantic distinctiveness; each is then operationalized by a dedicated evaluator and subjected to source-disjoint acceptance testing (Algorithm 1). Candidates may be revised before admission; only validated pairs are admitted into D, so discovery remains flexible while admitted semantics remain fixed. Admission testing certifies each evaluator on its own source-disjoint acceptance set, so every admitted capability satisfies the same validity criteria. The SOTOPIA capability space, representative entries, and their operationalization into executable criteria are provided in Appendix A. Algorithm 1 Validated Capability-Space Construction 1: Raw interaction corpus rawT_raw, induction models ℳM, validity criteria V, evaluator configuration κ, repeat count R, construction budget B 2: Validated capability space =(d,Jd)D=\(d,J_d)\ 3: Stage I: Capability Discovery and Consolidation 4: ←NormalizeAndFilter(raw)T← NormalizeAndFilter(T_raw) 5: ℱ←OpenInductionWithEvidence(,ℳ)F← OpenInductionWithEvidence(T,M) 6: for r=1,…,Rr=1,…,R do 7: r←Consolidate(EmbeddingNeighbors(ℱ))S_r← Consolidate( EmbeddingNeighbors(F)) 8: r←Filter(r,generality,operationality,distinctiveness)S_r← Filter(S_r;generality,operationality,distinctiveness) 9: end for 10: ←AlignAndSelect(rr=1R,recurrence,coverage,support)S← AlignAndSelect(\S_r\_r=1^R;recurrence,coverage,support) 11: ←DefineCandidates(,ℱ)S← DefineCandidates(S,F) 12: Stage I: Operationalization and Admission 13: ←∅D← 14: while ≠∅S≠ and budget B remains do 15: remove candidate d from S 16: d←RetrieveRelevantCases(,d)X_d← RetrieveRelevantCases(T,d) 17: ℒd←AnnotateAndVerify(d,d)L_d← AnnotateAndVerify(X_d,d) 18: (ℒddev,ℒdacc)←SourceDisjointSplit(ℒd)(L_d^dev,L_d^acc)← SourceDisjointSplit(L_d) 19: (ℒddev,ℒdacc)←EnsureCoverage(ℒddev,ℒdacc,d)(L_d^dev,L_d^acc)← EnsureCoverage(L_d^dev,L_d^acc,d) 20: pd⋆←DevelopEvaluator(d,ℒddev,κ)p_d ← DevelopEvaluator(d,L_d^dev,κ) 21: Jd←FreezeEvaluator(pd⋆,d,κ)J_d← FreezeEvaluator(p_d ,d,κ) 22: vd←ValidateCandidate(d,Jd,ℒdacc,,)v_d← ValidateCandidate(d,J_d,L_d^acc,D;V) 23: if vd=Acceptv_d= Accept then 24: ←∪(FreezeDefinition(d),Jd)D ∪\( FreezeDefinition(d),J_d)\ 25: else if vd=Revisev_d= Revise then 26: d′←ReviseCandidate(d,vd)d ← ReviseCandidate(d,v_d) 27: ←∪d′S ∪\d \ 28: else 29: discard d 30: end if 31: end while 32: return D 2.3 Outcome-Grounded Failure Attribution Terminal outcomes identify low-outcome trajectories that warrant diagnosis, but do not by themselves establish an agent-side failure or determine what should receive training credit. After stage k, πk+1 _k+1 generates on-policy trajectories k+1T_k+1, from which the low-outcome cohort ℱk+1=τi∈k+1∣qo(Ro(τi),xi)=1F_k+1=\ _i _k+1 q_o(R_o( _i),x_i)=1\ is selected, where xix_i denotes the task instance of trajectory τi _i. For each τi∈ℱk+1 _i _k+1, RewardWeaver localizes failure-relevant evidence, retrieves candidate capabilities Ci⊆kC_i _k, and produces attribution records ai,d=(si,d,ci,d,ei,d)a_i,d=(s_i,d,c_i,d,e_i,d) for d∈Cid∈ C_i, where si,ds_i,d is the deficit judgment, ci,dc_i,d its confidence, and ei,de_i,d the supporting evidence; unsupported assignments are rejected. Trajectory-level records are aggregated into Gk+1=Aggregate⁡(ai,d)G_k+1=Aggregate(\a_i,d\), which retains recurrent, supported, and policy-controlled failure modes rather than isolated low scores. 2.4 Adaptive Credit Allocation A capability can be valid yet irrelevant to the current policy; RewardWeaver treats training relevance as stage-dependent. Given Gk+1G_k+1, it selects a budgeted active set Ak+1=Select⁡(Gk+1,k)⊆kA_k+1=Select(G_k+1,D_k) _k. All active Rubrics contribute equally to the process reward, yielding Rk+1(τ)=Ro(τ)+1|Ak+1|∑d∈Ak+1J¯d(τ)R_k+1(τ)=R_o(τ)+ 1|A_k+1| _d∈ A_k+1 J_d(τ). As a bottleneck recedes, its Rubric may leave the active set and capacity is released for newly dominant failures, so the process reward is selective and stage-dependent rather than cumulative. Algorithm 2 Failure Attribution and Adaptive Credit Allocation 1: On-policy trajectories k+1T_k+1, capability space kD_k, current reward specification PkP_k, adaptation history kS_k, task-performance summary yk+1y_k+1, calibration set ℋk+1H_k+1, parameters Θ with active budget K and calibration limit M 2: Next-stage reward specification Pk+1P_k+1, or calibration failure ⊥ 3: ℱk+1←LowOutcomes(k+1)F_k+1← LowOutcomes(T_k+1); k+1←∅Z_k+1← 4: for τ∈ℱk+1τ _k+1 do 5: eτ←LocalizeEvidence(τ)e_τ← LocalizeEvidence(τ) 6: Cτ←Retrieve(eτ,k)C_τ← Retrieve(e_τ,D_k) 7: τ←Attribute(eτ,Cτ)Z_τ← Attribute(e_τ,C_τ) ⊳ Abstention allowed 8: k+1←k+1∪τZ_k+1 _k+1 _τ 9: end for 10: (Gk+1,rd,nd,cd,fd)←Aggregate(k+1)(G_k+1,\r_d,n_d,c_d,f_d\)← Aggregate(Z_k+1) 11: pd←αrd+βcd+γ⁡(1−fd)p_d←α r_d+β c_d+γ(1-f_d), d∈kd _k 12: (ℬhigh,ℬweak)←Eligible(Gk+1,k,Θ)(B^high,B^weak)← Eligible(G_k+1,S_k; ) 13: Ak+1←TopK(ℬhigh,p,K)A_k+1← TopK(B^high,p,K) 14: if |Ak+1|<K|A_k+1|<K and AllowWeak(ℬhigh,k,Θ) AllowWeak(B^high,S_k; ) then 15: Ak+1←Ak+1∪TopK(ℬweak∖Ak+1,p,K−|Ak+1|)A_k+1← A_k+1∪ TopK(B^weak A_k+1,p,K-|A_k+1|) 16: end if 17: (Ak+1,k+1)←Schedule(Ak+1,Gk+1,k,yk+1,Θ)(A_k+1,S_k+1)← Schedule(A_k+1,G_k+1,S_k,y_k+1; ) 18: λk+1,d←1/|Ak+1| _k+1,d← 1/|A_k+1|, d∈Ak+1d∈ A_k+1 ⊳ Equal weight over active Rubrics 19: Qk+1←CompileRubrics(Ak+1,Pk)Q_k+1← CompileRubrics(A_k+1,P_k) ⊳ Assemble a reward specification 20: for m=1,…,Mm=1,…,M do 21: (s,δ)←Calibrate(Qk+1,ℋk+1)(s,δ)← Calibrate(Q_k+1,H_k+1) 22: if Accept(s) Accept(s) then ⊳ Rubric-set publication gate 23: Pk+1←Compose(Pk,Ak+1,λk+1,Qk+1,s,δ,k+1)P_k+1← Compose(P_k,A_k+1, _k+1,Q_k+1,s,δ,S_k+1) 24: Publish(Pk+1) Publish(P_k+1) 25: return Pk+1P_k+1 26: else if m<Mm<M then 27: Qk+1←Revise(Qk+1,s,δ)Q_k+1← Revise(Q_k+1,s,δ) ⊳ Revise composition; keep (d,Jd)(d,J_d) frozen 28: end if 29: end for 30: return ⊥ ⊳ Calibration failure; no new reward is published Letting PkP_k denote the deployed process-reward specification at stage k, kS_k its adaptation history, and yk+1y_k+1 the stage-level task-performance summary, Algorithm 2 specifies the complete attribution-to-reward update. Importantly, the update changes which Rubrics are active and how they are composed into the next-stage reward, but never modifies the frozen semantics or evaluators of the capabilities admitted to kD_k. 2.5 Controlled Capability Expansion Adaptive credit allocation operates only over the capability space already validated at the beginning of the update, so the transition from stage k to k+1k+1 always satisfies Ak+1⊆kA_k+1 _k. When recurrent failures cannot be represented by kD_k, RewardWeaver invokes the same admission procedure as Algorithm 1: validated additions form the expanded space k+1D_k+1, but newly admitted capabilities do not alter the reward update that triggered their discovery—they become eligible only in the subsequent adaptation step, i.e., Ak+2⊆k+1A_k+2 _k+1. The fast loop therefore reallocates training credit within fixed semantics, (πk,Rk)→πk+1→k+1→ℱk+1→Gk+1→Ak+1⊆k→Rk+1,( _k,R_k)→ _k+1 _k+1 _k+1→ G_k+1→ A_k+1 _k→ R_k+1, (2) while the slower semantic loop updates (ℱk+1,k)→k+1(F_k+1,D_k) _k+1, with any new dimensions becoming available only to later reward updates. This separation defines the methodological boundary of RewardWeaver: training priorities evolve rapidly; reward semantics evolve conservatively. Table 1: Main results on SOTOPIA. The best result in each setting is shown in bold. In the Partner setting, the counterpart is GPT-4o for literature results and Qwen2.5-72B for AMPO and RewardWeaver in our experiments. Self-Play Partner Method SOTOPIA Goal / Overall ↑ SOTOPIA-Hard Goal / Overall ↑ SOTOPIA Goal / Overall ↑ SOTOPIA-Hard Goal / Overall ↑ Proprietary LLMs GPT-4o 8.19 / 3.76 6.97 / 3.46 8.19 / 3.76 6.97 / 3.46 Claude-3.5-Sonnet 8.29 / 3.71 6.33 / 3.09 8.42 / 3.77 6.64 / 3.30 DeepSeek-V3 8.15 / 3.62 6.34 / 3.09 8.14 / 3.72 6.69 / 3.31 Large Reasoning Models OpenAI-o1 7.93 / 3.58 5.69 / 2.71 8.09 / 3.69 6.65 / 3.20 OpenAI-o3-mini 7.38 / 3.30 5.14 / 2.36 7.96 / 3.61 6.33 / 2.98 Gemini-2.5-Pro 7.85 / 3.43 5.67 / 2.55 8.12 / 3.59 6.70 / 3.09 DeepSeek-R1 7.97 / 3.40 5.86 / 2.73 7.92 / 3.49 6.20 / 2.95 QwQ-32B 7.70 / 3.30 5.35 / 2.41 7.80 / 3.47 6.19 / 2.91 Post-training on Qwen2.5-7B-Instruct Qwen2.5-7B-Instruct 7.91 / 3.55 6.21 / 3.01 6.71 / 3.13 5.90 / 2.90 w/ PPDPP (Deng et al., 2024) 7.97 / 3.65 6.63 / 3.31 8.07 / 3.71 6.76 / 3.35 w/ EPO (Liu et al., 2025) 8.09 / 3.51 6.82 / 3.12 8.41 / 3.86 6.81 / 3.51 w/ DAT (Li et al., 2024) 7.97 / 3.59 6.39 / 3.10 8.11 / 3.70 6.78 / 3.36 w/ DSI (Zhang et al., 2025b) 8.35 / 3.75 7.31 / 3.51 8.15 / 3.70 6.87 / 3.42 AMPO (Wang et al., 2026) 8.45 / 4.20 7.67 / 3.94 8.30 / 4.09 7.30 / 3.80 RewardWeaver (ours) 8.72 / 4.47 7.90 / 4.38 8.49 / 4.47 7.66 / 4.22 3 Experiments 3.1 Experimental Settings Benchmarks. We evaluate RewardWeaver on three long-horizon benchmarks that together span general social interaction, bilateral bargaining, and domain-specific sales. SOTOPIA (Zhou et al., 2024a) contains 90 social scenarios; we report on the full benchmark and SOTOPIA-Hard. AmazonHistoryPrice (Xia et al., 2024) contains 930 products across 18 categories and evaluates bargaining under asymmetric information, with Mutual Interest (MI) instances admitting feasible agreements and Conflicting Interest (CI) instances admitting none. Following SOTOPIA-π (Wang et al., 2024b), we construct a health-service Sales Benchmark of 400 tasks across 20 scenario families; evaluation uses held-out instances within training families (see Appendix B). Evaluation. SOTOPIA and the Sales Benchmark use the SOTOPIA-EVAL protocol (Zhou et al., 2024a), reporting Goal Completion and Overall under Self-Play and fixed-Partner settings; SOTOPIA-Hard is the challenging subset, and Sales-Hard the 50 constraint-conflict tasks. For AmazonHistoryPrice, we report MI Deal Rate, CI Correct No-Deal Rate, Task Success Rate, Normalized Seller Utility, and Expected MI Seller Return. Task Success counts a legal MI agreement or correct CI no-deal; Expected MI Seller Return combines agreement probability with the resulting seller utility. For SOTOPIA, RewardWeaver uses a validated 40-dimensional capability space (Appendix A), activates at most B=6B=6 Rubrics per stage, and updates allocation every 30 steps. 3.2 Experimental Results and Analysis RQ1: Does RewardWeaver improve long-horizon interactive learning? Across the three benchmarks, RewardWeaver improves most primary metrics, with particularly strong gains in the harder SOTOPIA settings. On SOTOPIA (Table 1), it outperforms AMPO, the strongest prior method, on every metric under the same partner protocol, most pronounced on SOTOPIA-Hard (Goal/Overall: 7.67/3.94 to 7.90/4.38 Self-Play, 7.30/3.80 to 7.66/4.22 Partner), suggesting adaptive reward allocation remains effective under harder scenarios and external partners. On AmazonHistoryPrice (Figure 3), RewardWeaver jointly improves MI Deal Rate and seller-side utility, reaching the upper-right performance frontier; complete numerical results are reported in Appendix Table C1. On the Sales Benchmark (Table C2), it achieves the best result on six of eight metrics. Figure 3: Main results on AmazonHistoryPrice. Under the DeepSeek-V4-Pro buyer, RewardWeaver reaches the upper-right frontier, achieving a 95.2% MI Deal Rate and the highest Normalized Seller Utility (0.9737) among all evaluated methods. RQ2: Does dynamic reward adaptation matter, and does it need to target the correct bottlenecks? Table 2 compares RewardWeaver with outcome-only GRPO, Static Top-K, and Random Active Set. Outcome-only GRPO is weaker overall, supporting the usefulness of process-level supervision in this setting, while Static Top-K remains below RewardWeaver, suggesting that a fixed capability set becomes insufficient as training proceeds. Random Active Set preserves stage-wise reward changes but breaks the mapping from diagnosed bottlenecks to rewarded capabilities, providing a controlled test of bottleneck alignment; it underperforms RewardWeaver across all eight metrics. This comparison supports the importance of aligning reward allocation with the diagnosed bottlenecks, beyond the benefit of reward changes alone. Table 2: Ablation study on SOTOPIA. The counterpart is Qwen2.5-72B. The best result in each column is bold. We isolate the effects of dynamic reward adaptation, failure-grounded bottleneck identification, and stable semantics for admitted capabilities under matched training budgets. Self-Play Partner Method SOTOPIA Goal / Overall ↑ SOTOPIA-Hard Goal / Overall ↑ SOTOPIA Goal / Overall ↑ SOTOPIA-Hard Goal / Overall ↑ RewardWeaver (ours) 8.72 / 4.47 7.90 / 4.38 8.49 / 4.47 7.66 / 4.22 Dynamic Reward Adaptation Outcome-only GRPO 8.43 / 4.31 7.53 / 4.05 8.35 / 4.26 7.71 / 4.12 Static Top-K Dimensions 8.44 / 4.31 7.77 / 4.14 8.36 / 4.23 7.49 / 3.83 Random Active Set 8.47 / 4.41 7.56 / 4.18 8.33 / 4.27 7.31 / 3.97 Failure-Grounded Bottleneck Identification Dynamic Top-K by Failure Frequency 8.21 / 4.23 7.36 / 4.03 8.11 / 3.40 7.13 / 3.25 Random-Trajectory Attribution 8.38 / 4.37 7.67 / 4.25 8.24 / 4.31 7.60 / 4.17 Success–Failure Trajectory Attribution 8.55 / 4.52 7.93 / 4.32 8.34 / 4.33 7.60 / 4.14 Stable Semantics for Admitted Capabilities Open-Vocabulary Dimensions 8.50 / 4.37 7.69 / 4.19 8.35 / 4.18 7.44 / 3.95 Instance-Specific Rubrics 8.61 / 4.43 7.90 / 4.29 8.47 / 4.33 7.83 / 4.21 RQ3: Is failure-grounded attribution necessary for identifying policy bottlenecks? Table 2 separates failure attribution from simpler deficiency-based selection. Dynamic Top-K by Failure Frequency reselects capabilities each stage by deficiency prevalence on low-outcome trajectories, but performs no backward attribution or evidence localization. It underperforms RewardWeaver across all eight metrics, suggesting deficiency frequency alone is insufficient to reliably identify failure-relevant capabilities in these settings. Random-Trajectory Attribution degrades performance, suggesting attribution needs to be grounded in outcome-relevant failures rather than generic behavior. Success–Failure Trajectory Attribution remains competitive but does not consistently improve over RewardWeaver. These results suggest training relevance is better identified by evidence-grounded failure attribution than by deficiency prevalence or outcome-agnostic behavioral analysis. Appendix D provides trajectory-level cases of contrastive validation and selective attribution. Figure 4: Evolution of capability priorities on SOTOPIA. Normalized priorities of the 15 most prominent Rubrics across training stages. Reward emphasis shifts as different capability bottlenecks become training-relevant, illustrating stage-dependent reallocation of process-level credit in RewardWeaver. RQ4: Does semantic stability of admitted capabilities matter? As shown in Table 2, relaxing semantic stability also reduces performance. Open-Vocabulary Dimensions is consistently weaker than RewardWeaver across all four settings, suggesting an advantage of maintaining stable, validated capability definitions. Instance-Specific Rubrics is competitive, but remains inconsistent across settings despite occasionally matching or exceeding individual metrics. 3.3 Reward Adaptation Analysis Reward Evolution. Figure 4 shows reward emphasis shifting across training stages. Maintain Goal Commitment dominates early but later recedes, while Switch Strategies, State Benefits, and Respond with Empathy become more prominent; other capabilities, such as Conditional Concessions, re-emerge after decline. These patterns show RewardWeaver reallocates process-level credit as training-relevant bottlenecks change, rather than accumulating reward Rubrics. Bottleneck Attribution and Subsequent Failure Reduction. Complementing the controlled RQ2 ablations, we examine whether the attribution signal is prospectively associated with bottleneck reduction. We define bottleneck importance Ik,d=Fk,dfail−Fk,dsuccI_k,d=F^fail_k,d-F^succ_k,d and next-stage failure reduction ΔFk,d=Fk,dfail−Fk+1,dfail F_k,d=F^fail_k,d-F^fail_k+1,d. Bottleneck importance positively correlates with subsequent reduction (Spearman ρ=0.28ρ=0.28), remains positive after controlling for failure rate (partial ρ=0.22ρ=0.22), and strengthens within stage (ρ=0.31ρ=0.31), while current failure rate alone yields ρ=0.21ρ=0.21. The attribution ranking carries prospective information about which bottlenecks recede next. This analysis is not a causal identification of capability-level improvement; causal evidence comes from the controlled ablations in RQ2 and RQ3. Full results are reported in Appendix E. Sensitivity Analysis. We vary both the active Rubric budget B and the adaptation interval. Performance is non-monotonic in both: B=6B=6 achieves the strongest overall results among the tested budgets, while the 30-step interval provides the most consistently strong performance across Self-Play and Partner evaluation. Effective reward adaptation therefore depends on both selective credit allocation and an appropriate adaptation timescale. Full results are reported in Appendix F. Capability Retention. We examine whether capabilities regress after leaving the active reward set, tracking semantically aligned capabilities with fixed evaluation criteria across stages. Across the six capabilities deactivated after the initial stage, all 36 post-deactivation observations remain within 0.02 of or above their final active scores, with a mean change of +0.10+0.10. Adapt to New Information, deactivated later, remains stable. On the fixed diagnostic set, we observe no systematic regression in the tracked capabilities after deactivation, providing evidence that reward reallocation need not immediately erase previously emphasized behaviors. Full results are reported in Appendix G. 4 Related Work Outcome and Preference-Based Rewards. RLVR grounds rewards in verifiable task outcomes (Shao et al., 2024; Le et al., 2022; Guo et al., 2025), and process verifiers extend it to intermediate solution steps (Wang et al., 2024a; Lightman et al., 2024; Lu et al., 2024; Zeng et al., 2025; Setlur et al., 2025; Zheng et al., 2025). RLHF learns reward models from human preference data (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022); subsequent work diversifies feedback types (Wu et al., 2023; Yuan et al., 2024; Wang et al., 2024c), develops ranking objectives that bypass explicit reward modeling (Yuan et al., 2023; Hwang et al., 2023; Rafailov et al., 2023), and automates evaluation with AI feedback and pretrained LLM judges (Lee et al., 2023; Zheng et al., 2023; Liu et al., 2023; Kim et al., 2024a; Kim et al., 2024b; Ke et al., 2024). In many of these approaches, however, the reward specification is fixed or largely determined before policy training; which capabilities the current policy should optimize next remains open. Rubric-Based Process Rewards. Rubrics structure process quality: instance-specific rubrics are auto-generated (Gunjal et al., 2026; Wei et al., 2026; He et al., 2025), multidimensional evaluators induced (Hashemi et al., 2024), checklist scoring improves reliability (Lee et al., 2025), construction scales into reusable repositories (Li et al., 2026), and EvoRubric adapts criteria to the response distribution (Guan et al., 2026). These methods sharpen response-quality representation, yet discriminability does not determine training priority: a criterion may separate good from bad responses yet bear weakly on success, whereas a rarely violated capability may bind. Which criteria recurrent low-outcome failures implicate is underexplored. Credit Assignment in Multi-Turn Interaction. Assigning feedback in long-horizon interaction is long-standing: methods decompose delayed returns (Arjona-Medina et al., 2019), transport value across time (Hung et al., 2019), or estimate counterfactuals (Meulemans et al., 2023). For language agents, feedback spans episode-level scoring (Zhou et al., 2024a), utterance-level rewards from episode feedback or Shapley decomposition (Yu et al., 2025; Feng et al., 2026), segment-level preferences (Kong et al., 2025), and process rewards over strategic reasoning (Liu et al., 2025); hierarchical and retrospective agents optimize behavior (Zhou et al., 2024b; Yao et al., 2024), and automated attribution localizes failures (Zhang et al., 2025a). Orthogonally, AMPO (Wang et al., 2026) adapts reasoning depth to interaction modes, WebRL (Qi et al., 2025) evolves curricula, and PPDPP (Deng et al., 2024), DAT (Li et al., 2024), and DSI (Zhang et al., 2025b) steer dialogue through planners or strategy-injected corpora. These methods determine where an objective applies within an interaction; reward adaptation asks what the process reward should contain; RewardWeaver selects capability-level objectives from outcome-grounded failures. 5 Discussion The results of RewardWeaver suggest that reward relevance is stage-dependent and should be grounded in recurrent failures linked to task outcomes. This casts reward adaptation as an outer-loop problem: standard RL determines how to optimize under a given reward, while the outer loop determines what should receive training credit next. Reward adaptation should therefore track what currently limits performance, not merely what can be evaluated well. RewardWeaver nevertheless depends on LLM-based evaluators and informative terminal outcomes, so errors in either can propagate into reward adaptation. Its active budget and adaptation interval may also require re-tuning across domains. AI Use Statement In this work, we used generative AI tools to support discussions of research methodology and experimental design, qualitative analysis and interpretation of results, and the drafting, restructuring, and language polishing of the manuscript. Generative AI was also used to improve the presentation and organization of selected algorithm descriptions, tables, figure captions, and appendix materials. All AI-assisted content was carefully reviewed by the authors. The research methodology, experimental settings, data, and empirical results were verified against the actual implementation and experimental records, and the authors independently determined whether to adopt any AI-generated suggestions. The authors take full responsibility for the final content of this work, including any text, analysis, or other research artifacts generated or modified with the assistance of generative AI. References Arjona-Medina et al. (2019) J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter RUDDER: return decomposition for delayed rewards. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §4. Christiano et al. (2017) P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Deng et al. (2024) Y. Deng, W. Zhang, W. Lam, S. Ng, and T. Chua Plug-and-play policy planner for large language model powered dialogue agents. In International Conference on Learning Representations (ICLR), Cited by: Table 1, §4. Feng et al. (2026) X. Feng, Y. Jiang, X. Feng, D. Yin, L. Qin, Y. Ye, L. Huang, W. Ma, Y. Gu, C. Qin, B. Qin, and L. Kong SAVOIR: learning social savoir-faire via shapley-based reward attribution. arXiv preprint arXiv:2604.18982. Cited by: §4. Guan et al. (2026) X. Guan, X. Hu, S. Huang, Z. Wang, B. Zhang, Z. Li, P. Xie, B. Liu, and J. Cao EvoRubric: self-evolving rubric-driven RL for open-ended generation. arXiv preprint arXiv:2605.29847. Cited by: §1, §4. Gunjal et al. (2026) A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. M. Hendryx Rubrics as rewards: reinforcement learning beyond verifiable domains. In The Fourteenth International Conference on Learning Representations, Cited by: §1, §4. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, p. 633–638. Cited by: §1, §4. Hashemi et al. (2024) H. Hashemi, J. Eisner, C. Rosset, B. Van Durme, and C. Kedzie LLM-Rubric: a multidimensional, calibrated approach to automated evaluation of natural language texts. In Proceedings of ACL, Cited by: §1, §4. He et al. (2025) Y. He, W. Li, H. Zhang, S. Li, K. Mandyam, S. Khosla, Y. Xiong, N. Wang, X. Peng, B. Li, S. Bi, S. G. Patil, Q. Qi, S. Feng, J. Katz-Samuels, R. Y. Pang, S. Gonugondla, H. Lang, Y. Yu, Y. Qian, M. Fazel-Zarandi, L. Yu, A. Benhalloum, H. Awadalla, and M. Faruqui AdvancedIF: rubric-based benchmarking and reinforcement learning for advancing LLM instruction following. arXiv preprint arXiv:2511.10507. Cited by: §1, §4. Hung et al. (2019) C. Hung, T. Lillicrap, J. Abramson, Y. Wu, M. Mirza, F. Carnevale, A. Ahuja, and G. Wayne Optimizing agent behavior over long time scales by transporting value. Nature Communications. Cited by: §1, §4. Hwang et al. (2023) M. Hwang, G. Lee, H. Kee, C. Kim, K. Lee, and S. Oh Sequential preference ranking for efficient reinforcement learning from human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Ke et al. (2024) P. Ke, B. Wen, Z. Feng, X. Liu, X. Lei, J. Cheng, S. Wang, A. Zeng, Y. Dong, H. Wang, J. Tang, and M. Huang CritiqueLLM: towards an informative critique generation model for evaluation of large language model generation. In Proceedings of ACL, Cited by: §4. Kim et al. (2024a) S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, and M. Seo Prometheus: inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations (ICLR), Cited by: §4. Kim et al. (2024b) S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo Prometheus 2: an open source language model specialized in evaluating other language models. In Proceedings of EMNLP, Cited by: §4. Kong et al. (2025) A. Kong, W. Ma, S. Zhao, Y. Li, Y. Wu, K. Wang, X. Liu, Q. Li, Y. Qin, and F. Huang SDPO: segment-level direct preference optimization for social agents. In Proceedings of ACL, p. 12409–12423. Cited by: §4. Le et al. (2022) H. Le, Y. Wang, S. Gotmare, S. Savarese, and S. Hoi CodeRL: mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §4. Lee et al. (2023) H. Lee, S. Phatale, H. Mansoor, K. Lu, T. Mesnard, C. Bishop, E. Ferret, and P. Riedel RLAIF: scaling reinforcement learning from human feedback with AI feedback. arXiv preprint arXiv:2309.00267. Cited by: §4. Lee et al. (2025) Y. Lee, J. Kim, J. Kim, H. Cho, P. Kang, and N. Kim CheckEval: a reliable LLM-as-a-judge framework for evaluating text generation using checklists. In Proceedings of EMNLP, Cited by: §1, §4. Li et al. (2024) K. Li, Y. Wang, F. Viégas, and M. Wattenberg Dialogue action tokens: steering language models in goal-directed dialogue with a multi-turn planner. arXiv preprint arXiv:2406.11978. Cited by: Table 1, §4. Li et al. (2026) S. Li, J. Zhao, M. Wei, H. Ren, Y. Zhou, J. Yang, S. Liu, K. Zhang, and W. Chen RubricHub: a comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. arXiv preprint arXiv:2601.08430. Cited by: §1, §4. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations (ICLR), Cited by: §4. Liu et al. (2025) X. Liu, K. Wang, Y. Li, Y. Wu, W. Ma, A. Kong, F. Huang, J. Jiao, and J. Zhang EPO: explicit policy optimization for strategic reasoning in LLMs via reinforcement learning. In Proceedings of ACL, p. 15371–15396. Cited by: §1, Table 1, §4. Liu et al. (2023) Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of EMNLP, Cited by: §4. Lu et al. (2024) J. Lu, Z. Dou, H. Wang, Z. Cao, J. Dai, Y. Wan, Y. Feng, and Z. Guo AutoPSV: automated process-supervised verifier. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Meulemans et al. (2023) A. Meulemans, S. Schug, S. Kobayashi, N. D. Daw, and G. Wayne Would I have gotten that reward? Long-Term credit assignment by counterfactual contribution analysis. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §4. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Qi et al. (2025) Z. Qi, X. Liu, I. L. Iong, H. Lai, X. Sun, J. Sun, X. Yang, Y. Yang, S. Yao, W. Xu, J. Tang, and Y. Dong WebRL: training LLM web agents via self-evolving online curriculum reinforcement learning. In International Conference on Learning Representations (ICLR), Cited by: §4. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Setlur et al. (2025) A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar Rewarding progress: scaling automated process verifiers for LLM reasoning. In International Conference on Learning Representations (ICLR), Cited by: §4. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §4. Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano Learning to summarize from human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Wang et al. (2026) M. Wang, Y. Li, H. Wang, X. Zhang, N. Xu, B. Wu, F. Huang, H. Yu, and W. Mao Adaptive social learning via mode policy optimization for language agents. In International Conference on Learning Representations (ICLR), Cited by: §1, Table 1, §4. Wang et al. (2024a) P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui Math-shepherd: verify and reinforce LLMs step-by-step without human annotations. In Proceedings of ACL, Cited by: §4. Wang et al. (2024b) R. Wang, H. Yu, W. Zhang, Z. Qi, M. Sap, G. Neubig, Y. Bisk, and H. Zhu SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents. arXiv preprint arXiv:2403.08715. Cited by: §3.1. Wang et al. (2024c) Z. Wang, Y. Dong, O. Delalleau, J. Zeng, G. Shen, D. Egert, J. J. Zhang, M. N. Sreedhar, and O. Kuchaiev HelpSteer2: open-source dataset for training top-performing reward models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Wei et al. (2026) X. Wei, Q. Zong, X. Li, E. J. Yu, and S. Li QuRL: rubrics as judge for open-ended question answering. In International Conference on Learning Representations (ICLR), Cited by: §1, §4. Wu et al. (2023) Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi Fine-grained human feedback gives better rewards for language model training. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Xia et al. (2024) T. Xia, Z. He, T. Ren, Y. Miao, Z. Zhang, Y. Yang, and R. Wang Measuring bargaining abilities of LLMs: a benchmark and a buyer-enhancement method. In Findings of the Association for Computational Linguistics: ACL 2024, Cited by: §3.1. Yao et al. (2024) W. Yao, S. Heinecke, J. C. Niebles, Z. Liu, Y. Feng, L. Xue, R. Murthy, Z. Chen, J. Zhang, D. Arpit, R. Xu, P. Mui, H. Wang, C. Xiong, and S. Savarese Retroformer: retrospective large language agents with policy gradient optimization. In International Conference on Learning Representations (ICLR), Cited by: §4. Yu et al. (2025) H. Yu, Z. Qi, Y. Zhao, K. Nottingham, K. Xuan, B. P. Majumder, H. Zhu, P. P. Liang, and J. You SOTOPIA-RL: reward design for social intelligence. arXiv preprint arXiv:2508.03905. Cited by: §4. Yuan et al. (2023) H. Yuan, Z. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang RRHF: rank responses to align language models with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Yuan et al. (2024) Y. Yuan, J. Hao, Y. Ma, Z. Dong, H. Liang, J. Liu, Z. Feng, K. Zhao, and Y. Zheng Uni-RLHF: universal platform and benchmark suite for reinforcement learning with diverse human feedback. In International Conference on Learning Representations (ICLR), Cited by: §4. Zeng et al. (2025) T. Zeng, S. Zhang, S. Wu, C. Classen, D. Chae, E. Ewer, M. Lee, H. Kim, W. Kang, J. Kunde, Y. Fan, J. Kim, H. I. Koo, K. Ramchandran, D. Papailiopoulos, and K. Lee VersaPRM: multi-domain process reward model via synthetic reasoning data. In International Conference on Machine Learning (ICML), Cited by: §4. Zhang et al. (2025a) S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y. Chen, and Q. Wu Which agent causes task failures and when? On automated failure attribution of LLM multi-agent systems. In International Conference on Machine Learning (ICML), Cited by: §1, §4. Zhang et al. (2025b) W. Zhang, T. Liu, M. Song, X. Li, and T. Liu SOTOPIA-Ω : dynamic strategy injection learning and social instruction following evaluation for social agents. In Proceedings of ACL, p. 24669–24697. Cited by: Table 1, §4. Zheng et al. (2025) C. Zheng, Z. Zhang, B. Zhang, R. Lin, K. Lu, B. Yu, D. Liu, J. Zhou, and J. Lin ProcessBench: identifying process errors in mathematical reasoning. In Proceedings of ACL, Cited by: §4. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Cited by: §4. Zhou et al. (2024a) X. Zhou, H. Zhu, L. Mathur, R. Zhang, Z. Qi, H. Yu, L. Morency, Y. Bisk, D. Fried, G. Neubig, and M. Sap SOTOPIA: interactive evaluation for social intelligence in language agents. In International Conference on Learning Representations (ICLR), Cited by: §1, §1, §3.1, §3.1, §4. Zhou et al. (2024b) Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar ArCHer: training language model agents via hierarchical multi-turn RL. In International Conference on Machine Learning (ICML), Cited by: §4. Appendix A Validated Capability-Space Construction for SOTOPIA in RewardWeaver The SOTOPIA capability space contains 40 validated Rubrics organized into seven behavioral families. We first summarize the structure of the space (Table A1), then present representative semantic entries (Table A2), and finally show how a frozen Rubric is operationalized into an executable evaluator (Section A.3). Table A1: Behavioral families in the SOTOPIA capability Rubric library. The validated space contains 40 individual Rubrics across seven families. Family # Rubrics Behavioral Scope Goal Expression and Persistence 5 Goal articulation and retention Information Acquisition 8 Information elicitation and clarification Information Use 5 Evidence use and adaptation Concern Handling and Persuasion 6 Objection handling and negotiation Goal Progression and Closure 7 Progress, strategy, and completion Relationship Management 5 Empathy and interaction quality Boundaries and Safety 4 Privacy, truthfulness, and social boundaries Total 40 A.1 Capability-Space Overview Table A1 summarizes the seven behavioral families spanned by the 40-dimensional SOTOPIA capability space. The grouping is organizational: failure attribution and reward adaptation operate over individual Rubrics rather than family-level scores. A.2 Representative Capability Rubrics Table A2 presents representative entries covering the major regions of the capability space. Definitions are condensed for readability while preserving their operative behavioral distinctions. Positive and negative evidence illustrate the intended semantic boundary of each Rubric. Table A2: Representative entries from the 40-dimensional SOTOPIA capability Rubric library. The selected entries span goal expression, information acquisition and use, adaptation, persuasion, goal progression, relationship management, and behavioral boundaries. Use of Key Information is further operationalized in Section A.3. ID Capability Definition Positive Evidence Negative Evidence explicit-goal Explicit Goal Expression Explicitly state a task-required request, position, information, feeling, boundary, or correction that has not yet been expressed. Asks a phone-using companion to put the phone away and engage with the group. Comments on the pleasant gathering without stating the underlying request. elicit-key-info Targeted Information Elicitation Directly ask for task-relevant information that is required but not yet available. Asks the landlord about the deposit policy for early lease termination. Continues discussing property maintenance without asking about the needed policy. use-key-information Use of Key Information Accurately use at least one known, relevant, and safely usable fact to support the current response. Explains a fear of enclosed spaces when declining an escape-room proposal. Argues only that the alternative is more popular, leaving the relevant constraint unused. adapt-to-new-info Adapt to New Information Adjust a request or plan when newly disclosed information changes its feasibility. Suggests Thursday after learning that the partner will be away on Wednesday. Repeats the Wednesday request despite the disclosed trip. address-concern Address Concerns with New Options After the partner raises a specific concern, propose an actionable adjustment that directly reduces that concern. Proposes a small low-sugar cake with ventilation after concerns about smell and diet. Repeats that baking is enjoyable without modifying the proposal. strategy-pivot Switch Strategies After repeated rejection or deferral, use a new argument, angle, or plan rather than repeating the same request. Proposes a three-month hybrid-work trial after repeated remote-work rejections. Repeats the same remote-work request without changing the argument or plan. empathic-acknowledgment Respond with Empathy Acknowledge the partner’s expressed negative emotion or difficulty before continuing the task interaction. Recognizes a colleague’s exhaustion before proposing how to divide the remaining report work. Ignores the exhaustion and immediately demands additional overtime. secret-keeping Confidentiality Preservation Avoid disclosing information explicitly marked as confidential when the related topic arises. Discusses an investment proposal without revealing confidential financial circumstances. Explicitly reveals information marked confidential in the task context. reject-improper-request Rejection of Improper Requests Explicitly reject a request that violates social norms, law, or the agent’s stated boundaries. Refuses to make a false statement to cover for a roommate. Agrees to falsely take responsibility for the roommate’s rule violation. privacy-boundary Privacy Boundary Respect Avoid probing task-irrelevant sensitive personal information when such information enters the conversation. Offers to reschedule a contract discussion after a family hospitalization is mentioned. Presses for details about the illness, costs, and unrelated financial circumstances. A.3 Rubric-Specific Evaluator Operationalization We use Use of Key Information, introduced in Table A2, as a concrete example of Rubric-Specific Evaluator construction. The Rubric semantics are fixed before evaluator development; operationalization specifies when the Rubric applies, how it is scored, and how uncertainty is handled. Table A3 summarizes the resulting decision structure. Table A3: Operationalization of Use of Key Information. Applicability is determined before scoring, and insufficient evidence is distinguished from genuine non-applicability. Field Operationalization Rubric ID fixed-use-key-information-v1 Applicability The Rubric applies when at least one safe, directly relevant fact, preference, constraint, or confirmed arrangement is available in the visible context before the target response. Score =1=1 The target response accurately uses at least one currently relevant item of known information to explain, request, confirm, adapt, or respond. Score =0=0 The Rubric applies, but the target response uses none of the relevant known information, or misstates a relevant fact, quantity, person, relation, or constraint. Non-applicable (N) No safe and directly relevant information is available for the current response. Failure to use available information is scored as 00, not N. Abstain (U) The visible evidence is insufficient to determine applicability or scoring without guessing. Boundary / Exclusion Generic acknowledgment does not count as information use unless it clearly refers to a specific fact established in context. Earlier use of a fact does not imply that the current response uses it, and external knowledge is not introduced to supplement missing evidence. Evaluator Operationalization. Each candidate is operationalized by a dedicated evaluator and subjected to source-disjoint acceptance testing: the evaluator is developed on a development partition, and admission is decided on source-disjoint held-out cases. Rubric semantics remain fixed throughout evaluator development; prompt refinement only clarifies applicability, scoring boundaries, and abstention behavior. Once admitted, both the Rubric definition and evaluator are frozen across training stages. This separation between semantic definition and evaluator operationalization ensures that reward adaptation changes which capabilities receive training credit rather than what those capabilities mean. Appendix B Sales Benchmark Construction and Evaluation We construct the Sales Benchmark to evaluate long-horizon interaction under structured business rules, asymmetric information, and conflicting objectives. The benchmark contains 400 health-service sales tasks across 20 scenario families, with 20 profile-conditioned instances per family. Among them, 350 are standard tasks and 50 are constraint-conflict tasks. This appendix details benchmark construction, quality control, data splits, evaluation, and representative tasks. B.1 Construction Protocol Each scenario family is instantiated with 20 buyer profiles. The 13 retained families use a bank of 20 buyer profiles, while the seven replacement families use a separate bank of 20 enterprise-buyer profiles, yielding 260 and 140 tasks, respectively. Model-assisted authoring. We used GPT-5.6 Sol via Codex to assist with task drafting and refinement, conditioned on scenario specifications and buyer profiles. Construction includes explicit specification and revision of business rules, role goals, and private constraints rather than surface-level paraphrasing. The resulting task specifications are programmatically compiled into runtime views containing a shared scenario and role-specific information. Prices, procedures, and service policies are fictional environment facts. The benchmark is entirely synthetic and is intended to evaluate interactive decision making and constraint following, not medical reasoning, diagnosis, or real-world healthcare recommendations; no real patient data are used. Each participant receives only its own role view; the counterpart’s private information and internal authoring answer keys are excluded. B.2 Authoring Template and Quality Control The following compact template summarizes the constraints used to specify and revise tasks. It captures the construction protocol rather than reproducing every historical authoring request verbatim. Construct or revise a health-service sales task from: scenario specification, buyer profile, business rules, and standard / constraint-conflict designation. Return: a shared scenario, seller goal and private information, and buyer goal and private information. Specify decision-relevant options, prices, timing, and authority. Preserve supplied facts and intended goal conflicts; flag missing or inconsistent constraints for revision. Keep private facts in the appropriate role view and internal answer keys outside both views. Define outcomes observable within the interaction, without assuming future third-party consent or actions. Do not prescribe an ideal dialogue or make unsupported medical claims. Quality control and deduplication. Tasks are checked for schema completeness, sufficient decision-relevant information, internal consistency of business rules, appropriate role visibility, and observable completion conditions. Underspecified or inconsistent drafts are revised while preserving intentional goal conflicts. Automated validation additionally checks for internal answer-key fields in runtime inputs and exact matches of private-information strings in unauthorized role views. Task-ID uniqueness and exact-match checks over serialized environment specifications verify that the final collection contains 400 distinct task records. B.3 Scenario Families Table B1 summarizes the 20 scenario families. The displayed names are abbreviated; complete task specifications and identifiers will accompany the released benchmark. Table B1: Scenario families in the Sales Benchmark. Family Scenario R01 Staged corporate procurement R02 Data-use purpose separation R03 Cross-city service-network due diligence R04 Multi-site acceptance and objection terms R05 Corporate lead qualification R06 Sign-language and bilingual support R07 Contract, payment, and invoice entities S01 Budget-constrained add-ons S02 Final price confirmation S04 Retention against a cheaper competitor S05 Package reduction S06 Deposits and slot reservation S10 Time-constrained service selection S13 Paid report interpretation S14 Package-tier selection S15 Renewal timing S16 Referral rewards S17 Gift-package transfer S19 Follow-up examination conversion S20 Refund and replacement purchase B.4 Split and Evaluation Split and overlap checks. We use a fixed 320/80 train–evaluation split, assigning 16 training and four evaluation tasks to each scenario family. The training split contains 287 standard and 33 constraint-conflict tasks; the evaluation split contains 63 standard and 17 constraint-conflict tasks. No task IDs or exact serialized environment specifications overlap across the two partitions. Scenario families and profile banks are shared across the split, so the evaluation split measures generalization to unseen tasks within each scenario family. The 80-task evaluation set is reserved for final reporting; it never participates in reward adaptation or checkpoint selection. Outcome evaluation. The outcome judge is Qwen2.5-72B. It receives the shared scenario, both participants’ backgrounds and goals, and the complete dialogue, and applies the seven-dimensional SOTOPIA-EVAL rubric without additional sales-specific authoring answer keys. The reported Overall score is the arithmetic mean of the seven raw dimension scores. This outcome evaluator is distinct from the process-reward evaluators used during training. Release. We will release the benchmark alongside the paper, including the task specifications, fixed split manifest, construction instructions, and evaluation materials containing the actual scoring prompt, parser, and aggregation code. B.5 Representative Tasks We provide two representative tasks below. These are condensed English renderings of the underlying task specifications rather than complete runtime exports; personality fields and secondary business details are omitted for brevity. Standard: Refund and replacement purchase. sales-crafted-S20-P01. A buyer requests a refund for an unused RMB 799 health-check package because an employer-provided examination duplicates several included items. The seller may offer fixed RMB 599 alternatives with a RMB 200 difference refund, but must preserve the buyer’s unconditional full-refund option. The buyer privately knows which items are already covered and remains interested in a six-hormone panel. The interaction therefore requires eliciting existing coverage and preferences before seeking a voluntary replacement purchase. Constraint conflict: Gift-package transfer. sales-crafted-S17-P03. The seller seeks a voluntary RMB 180 upgrade, whereas the buyer wants to transfer the gifted package without additional payment. Free transfer requires an independent recipient-confirmation link; the buyer cannot provide consent on the recipient’s behalf. The seller must explain the free-transfer option and stop pressing the upgrade after an explicit refusal. Initiating a compliant free transfer does not satisfy the seller’s paid-upgrade objective, and issuing the link does not by itself establish a completed transfer. Appendix C Additional Benchmark Results of RewardWeaver Table C1 reports the full AmazonHistoryPrice results corresponding to Figure 3 in the main text. Table C2 presents the full results on the Sales Benchmark under Self-Play and Partner evaluation. Table C1: Main results on AmazonHistoryPrice. All sellers are evaluated with deepseek-v4-pro as the buyer. The best result in each column is shown in bold. Method MI Deal Rate ↑ CI Correct No-Deal Rate ↑ Task Success Rate ↑ Normalized Seller Utility ↑ Expected MI Seller Return ↑ Proprietary LLMs doubao-seed-2-0-pro 94.40 100.00 94.50 0.85 0.80 glm-5.3-flash 95.20 100.00 95.30 0.91 0.86 glm-5.2 92.00 100.00 92.20 0.93 0.86 qwen3.8-max 80.80 100.00 81.30 0.95 0.76 qwen3.8-flash 83.20 100.00 83.60 0.93 0.77 qwen3.7-plus 51.20 100.00 52.30 0.92 0.47 deepseek-v4-pro 83.20 100.00 83.60 0.87 0.73 Post-training on Qwen2.5-7B-Instruct Qwen2.5-7B-Instruct w/ GRPO 84.00 66.70 83.60 0.56 0.47 RewardWeaver (ours) 95.20 100.00 95.30 0.97 0.93 Table C2: Main results on the Sales Benchmark. In the Partner setting, the counterpart is Qwen2.5-72B-Instruct. The best result in each column is shown in bold. Self-Play Partner Method Sales Goal / Overall ↑ Sales-Hard Goal / Overall ↑ Sales Goal / Overall ↑ Sales-Hard Goal / Overall ↑ Proprietary LLMs doubao-seed-2-1-pro 8.55 / 4.36 7.29 / 3.99 8.81 / 4.43 7.47 / 4.10 qwen3.8-max 8.65 / 4.38 7.76 / 4.09 8.76 / 4.43 7.76 / 4.17 gemini-3.1-pro-preview 8.63 / 4.37 7.29 / 4.11 8.83 / 4.48 7.59 / 4.08 deepseek-v4-pro 8.68 / 4.30 7.24 / 3.97 8.67 / 4.39 7.82 / 4.17 glm-5.2 8.38 / 4.30 7.59 / 4.12 8.73 / 4.40 7.88 / 4.19 Post-training on Qwen2.5-7B-Instruct Qwen2.5-7B-Instruct 8.33 / 4.30 7.18 / 3.97 8.41 / 4.33 7.12 / 3.99 Qwen2.5-7B-Instruct w/ GRPO 8.38 / 4.41 7.71 / 4.28 8.59 / 4.47 7.65 / 4.29 RewardWeaver (ours) 8.73 / 4.43 7.88 / 4.31 8.90 / 4.53 7.71 / 4.24 Appendix D Qualitative Case Studies of RewardWeaver To complement the aggregate experiments, we examine how reward adaptation operates at the trajectory level through three representative cases from a complete SOTOPIA training run. Together, they illustrate three properties of the mechanism: stage-dependent bottleneck transition, contrastive validation of failure-derived Rubrics, and selective attribution under uncertainty. D.1 Bottleneck Transition Multiple low-outcome trajectories exhibit a recurrent pattern of verbose, abstract, and difficult-to-interpret language. In a representative right-of-way negotiation at a four-way stop, the agent receives Goal Completion 2/102/10, and the task evaluator explicitly attributes the failure to unclear communication. The same pattern recurs across other trajectories in the cohort, indicating a stage-level bottleneck rather than an isolated error. The subsequent attribution step identifies Clear Expression as a candidate capability. Following rubric generation and calibration, the next reward specification incorporates a criterion targeting the observed communication failures. In the following stage, repeated rollouts of the same scenario exhibit improved behavior, reaching Goal Completion scores of 7 and 9; the previous language degradation was not observed in the sampled follow-up rollouts. The failure distribution then shifts again: the new low-outcome cohort exhibits a different recurrent pattern, and the next adaptation step redirects emphasis toward Concrete Solutions. The case illustrates the stage dependence of training relevance: recurrent failure→bottleneck attribution→reward adaptation→new failure distribution.recurrent failure attribution adaptation failure distribution. A capability need not remain a training priority once its associated bottleneck recedes, and a capability that was previously non-limiting may become dominant as the policy and interaction distribution evolve. Thus, RewardWeaver allocates process-level credit according to the current bottleneck rather than a fixed capability set. As scorecards are regenerated between adaptation stages, the qualitative comparison is based on repeated evaluation of the same scenario. D.2 Contrastive Validation of Failure-Derived Rubrics A Rubric proposed from failure-side evidence is not necessarily specific to failure. A candidate evaluator for Empathic Support is calibrated on a matched set containing both high- and low-outcome trajectories. The evaluator is applicable to 23 trajectories, yet all 23 pass, yielding zero observed failures and very low score variation. It is flagged as collapsed and targeted for revision. The candidate rubric set fails the publication gate because its validity rate is below the required threshold. Rubric-set acceptance conditions govern publication, while Rubric-level diagnostics guide local revision. This distinction is central: failure-derived⇏failure-specific.failure-derived\; \;failure-specific. A Rubric may be motivated by failed trajectories yet fail to distinguish failure-relevant behavior on held-out contrastive examples. Such a signal provides little useful discrimination for process supervision. RewardWeaver therefore separates bottleneck discovery from reward-rubric calibration: failure evidence proposes candidate capabilities, while calibration determines whether the generated rubric set satisfies publication conditions and uses Rubric-level diagnostics to guide revision. This mechanism differs from the Success–Failure Trajectory Attribution variant studied in RQ3, which introduces success trajectories during bottleneck attribution. Here, calibration occurs after rubric generation and governs publication of the candidate reward specification. Success trajectories are not required for the failure-attribution step; calibration remains required before publishing an updated reward specification. D.3 Selective Attribution under Uncertainty Failure-grounded attribution does not require every low-outcome trajectory to be mapped to an agent-side capability deficit. In one representative trajectory, the agent fails to achieve its task goal because the partner’s private objective is to decline the interaction. The task evaluator explicitly attributes the outcome to the partner’s preference, and no agent-side capability is activated. Similar cases occur elsewhere in the run, including trajectories with low terminal outcomes despite all currently active capability checks being satisfied. These observations establish that terminal failure alone is insufficient evidence for capability-level credit assignment. The attribution pipeline therefore admits three outcomes: supported capability bottleneck,interaction-driven failure,insufficient evidence.supported capability bottleneck, -driven failure, evidence. Capability-level training credit is assigned only when the failure admits an evidence-supported, policy-controlled bottleneck. This selectivity prevents partner behavior, scenario constraints, or ambiguous evidence from being silently converted into agent-side learning objectives. Accordingly, failure-grounded attribution is not forced attribution: low outcome determines which trajectories warrant diagnosis, whereas capability-level credit is assigned only when the failure is supported by observable, policy-controlled evidence. Summary. The three cases illustrate complementary aspects of adaptive process rewards. First, training relevance is stage-dependent and should track the current failure bottleneck. Second, failure-derived Rubrics require calibration before publication: rubric-set acceptance checks govern publication, while Rubric-level diagnostics guide revision. Third, attribution remains selective under uncertainty: low outcome is a trigger for diagnosis, not a guarantee of capability-level blame. Together, the cases show that effective reward adaptation requires current relevance, discriminative criteria, and evidence-supported attribution; RewardWeaver therefore combines evidence-grounded candidate selection with rubric-set calibration. Appendix E Association Between Bottleneck Attribution and Subsequent Failure Reduction We further analyze whether the bottleneck importance estimated by RewardWeaver is informative about subsequent changes in capability-level failure patterns. E.1 Analysis Setup For each capability d at training stage k, we define bottleneck importance as Ik,d=Fk,dfail−Fk,dsucc,I_k,d=F^fail_k,d-F^succ_k,d, where Fk,dfailF^fail_k,d is the fraction of low-outcome trajectories attributed to a deficiency in capability d, and Fk,dsuccF^succ_k,d is the corresponding fraction among high-outcome trajectories. A larger Ik,dI_k,d therefore indicates that the capability deficiency is more strongly associated with low-outcome trajectories. This contrast is constructed for analysis: the attribution mechanism itself (Section 2.3) operates on low-outcome trajectories only, and the success-side contrast is introduced here to characterize how informative the attribution ranking is. We quantify the subsequent reduction of this failure mode as ΔFk,d=Fk,dfail−Fk+1,dfail, F_k,d=F^fail_k,d-F^fail_k+1,d, such that a larger positive value corresponds to a larger decrease in the attributed failure rate in the following stage. We evaluate this relationship over 240 capability–stage pairs spanning 40 capabilities and six adjacent training-stage transitions. We use Spearman rank correlation because the attribution mechanism primarily operates through relative capability priorities. E.2 Results Table E1 summarizes the results. Bottleneck importance exhibits a positive association with next-stage failure reduction, with Spearman ρ=0.28ρ=0.28. In comparison, the current failure rate alone shows a weaker association with subsequent reduction (ρ=0.21ρ=0.21). To determine whether the attribution signal provides information beyond the prevalence of the current failure mode, we additionally control for Fk,dfailF^fail_k,d. The association remains positive, with partial Spearman ρ=0.22ρ=0.22. We further compare capabilities within the same training stage, obtaining a stronger association of ρ=0.31ρ=0.31. Table E1: Association between bottleneck attribution and subsequent failure reduction. We measure the association between capability-level signals at stage k and the reduction in attributed failures at stage k+1k+1. Analysis Spearman ρ Bottleneck importance Ik,dI_k,d vs. ΔFk,d F_k,d 0.28 Current failure rate Fk,dfailF^fail_k,d vs. ΔFk,d F_k,d 0.21 Ik,dI_k,d vs. ΔFk,d F_k,d, controlling for Fk,dfailF^fail_k,d 0.22 Within-stage association 0.31 The comparison reveals two complementary properties of the attribution signal. First, its association is stronger than that of current failure prevalence alone. Second, the association remains positive after accounting for current failure prevalence and when capabilities are compared within the same training stage. Thus, the attribution signal provides a stage-dependent ranking whose higher values are associated with larger subsequent reductions in capability-level failure rates. Appendix F Sensitivity Analysis of RewardWeaver We examine two design choices governing reward adaptation in RewardWeaver: the number of capabilities that receive concurrent training credit and the frequency with which this allocation is updated. Unless otherwise specified, the main configuration uses an active Rubric budget of B=6B=6 and updates reward allocation every 30 optimization steps. F.1 Active Rubric Budget Table F1 varies the number of concurrently active Rubrics under both Self-Play and Partner evaluation. Performance is non-monotonic in B: neither reducing the active set nor activating more Rubrics consistently improves performance. The main setting B=6B=6 achieves the strongest results across all reported settings, suggesting that effective process supervision depends on concentrating training credit on a selective set of current bottlenecks rather than accumulating additional Rubrics. Table F1: Sensitivity to the active Rubric budget B. Results are reported as Goal Completion / Overall under both Self-Play and Partner evaluation. The main configuration uses B=6B=6. Increasing the number of simultaneously active Rubrics does not monotonically improve performance, supporting selective rather than cumulative credit allocation. Self-Play Partner B SOTOPIA SOTOPIA-Hard SOTOPIA SOTOPIA-Hard 2 8.46 / 4.32 7.50 / 3.97 8.38 / 4.17 7.46 / 3.92 4 8.31 / 4.29 7.79 / 4.16 8.21 / 4.19 7.37 / 4.03 6 8.72 / 4.47 7.90 / 4.38 8.49 / 4.47 7.66 / 4.22 8 8.49 / 4.44 7.67 / 4.25 8.31 / 4.25 7.59 / 4.15 10 8.50 / 4.46 7.63 / 4.36 8.34 / 4.37 7.57 / 4.21 F.2 Adaptation Interval Table F2 varies the reward-adaptation interval under both Self-Play and Partner evaluation. Performance is again non-monotonic. The 30-step configuration provides the most consistently strong results across interaction modes and difficulty levels: it achieves the strongest Goal Completion scores under Self-Play and the strongest results across all Partner settings, while the 20-step setting is marginally higher on selected Self-Play Overall scores. These results indicate that reward adaptation is sensitive to its update timescale rather than benefiting monotonically from either faster or slower updates. Table F2: Sensitivity to the reward-adaptation interval. The main configuration updates reward allocation every 30 optimization steps. Results are reported as Goal Completion / Overall. Self-Play Partner Interval SOTOPIA SOTOPIA-Hard SOTOPIA SOTOPIA-Hard 10 8.28 / 4.41 7.73 / 4.36 8.32 / 4.33 7.59 / 4.20 20 8.50 / 4.47 7.81 / 4.39 8.27 / 4.19 7.47 / 4.03 30 8.72 / 4.47 7.90 / 4.38 8.49 / 4.47 7.66 / 4.22 60 8.21 / 4.26 7.44 / 3.95 8.34 / 4.28 7.49 / 4.04 Taken together, the two analyses highlight complementary requirements of adaptive process rewards: credit allocation should remain selective in capability space and operate at an appropriate adaptation timescale. Appendix G Capability Retention under Reward Reallocation We further examine whether reallocating process-level credit causes previously emphasized capabilities to deteriorate once they leave the active reward set. G.1 Evaluation Protocol We evaluate the policy at each stage on the same fixed diagnostic scenarios and score each tracked capability using a fixed evaluation criterion across all stages. For capability d at stage k, we denote the resulting diagnostic score by Sk,dS_k,d, computed as the mean normalized process score over applicable episodes. Six capabilities receive process-level credit in the initial stage and are subsequently deactivated: Address Concerns with New Options, State Benefits, Conditional Concessions, Maintain Goal Commitment, Regulate Pressure, and Switch Strategies. Adapt to New Information is deactivated at a later stage. We additionally report three continuously active capability families, namely Respond with Empathy, Early Disclosure, and Concrete Next Step, as reference trajectories. For a capability deactivated after stage k, we measure retention by its subsequent change ΔSt,d=Sk+t,d−Sk,d S_t,d=S_k+t,d-S_k,d relative to the final active stage. G.2 Retention Results Table G1 reports the capability trajectories. The six capabilities deactivated after the initial stage show no systematic regression: across 36 post-deactivation capability–stage observations, the mean change is +0.10+0.10, the median change is +0.06+0.06, and the minimum change is −0.01-0.01. No observation decreases by more than 0.02 from its final active score. Adapt to New Information, which is deactivated later, remains at 1.00 throughout all subsequent stages. Table G1: Capability retention across training stages. † denotes capabilities deactivated after the initial stage, ‡ denotes a capability deactivated later, and unmarked dimensions remain active during the evaluated period. Capability Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Stage 6 Stage 7 Address Concerns with New Options† 0.80 0.85 0.92 0.95 0.86 0.89 0.90 State Benefits† 0.84 0.88 0.97 0.98 0.95 0.94 0.98 Conditional Concessions† 0.85 0.90 0.98 0.93 0.86 0.90 0.96 Maintain Goal Commitment† 0.96 0.95 1.00 0.98 0.96 1.00 1.00 Regulate Pressure† 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Switch Strategies† 0.62 0.91 1.00 1.00 0.88 1.00 1.00 Adapt to New Information‡ 1.00 0.99 1.00 1.00 1.00 1.00 1.00 Respond with Empathy 0.99 1.00 1.00 1.00 1.00 1.00 1.00 Early Disclosure 0.92 0.90 0.94 0.94 0.91 0.92 0.99 Concrete Next Step 0.99 1.00 1.00 1.00 0.99 0.99 0.99 The event-aligned trajectories make the retention pattern particularly clear. Among the six early-deactivated dimensions, every post-deactivation observation remains close to or above its final active score. Several capabilities continue to improve despite receiving no further direct process-level credit: Switch Strategies rises from 0.62 to 1.00, State Benefits from 0.84 to 0.98, and Address Concerns with New Options from 0.80 to 0.90. The later-deactivated Adapt to New Information is similarly stable. These results show that, on the fixed diagnostic set, shifting reward emphasis toward newly relevant capabilities does not systematically erase capabilities emphasized at earlier stages. The adaptive reward therefore behaves as a reallocation of training credit rather than requiring previously learned capabilities to be continually reinforced.

Instant research discovery