Paper deep dive
Fibration Policy Optimization
Chang Li, Tshihao Tsu, Yaren Zhang, Chao Xue, Xiaodong He
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 12:42:00 AM
Summary
The paper introduces Fibration Policy Optimization (FiberPO), a novel framework for LLM policy optimization that addresses multi-scale stability control. It derives the Aggregational Policy Censoring Objective (APC-Obj) as an unconstrained reformulation of TV-TRPO, and develops Fiber Bundle Gating (FBG) to decompose ratio gating into base-level (trajectory) and fiber-level (token) components. This approach enables hierarchical stability control (Fibration Gating Hierarchy) and improves token efficiency by providing a restorative gradient structure.
Entities (5)
Relation Signals (4)
FiberPO → derivedfrom → APC-Obj
confidence 99% · From APC-Obj and FBG we derive Fibration Policy Optimization
FiberPO → derivedfrom → FBG
confidence 99% · From APC-Obj and FBG we derive Fibration Policy Optimization
APC-Obj → isreformulationof → TV-TRPO
confidence 98% · APC-Obj, the first exact unconstrained reformulation of sample-based TV-TRPO
FGH → generalizes → FBG
confidence 95% · fibrations compose algebraically into a Fibration Gating Hierarchy (FGH) that scales the same gating mechanism
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models are increasingly trained as heterogeneous systems spanning multiple domains, expert partitions, and agentic pipelines, yet prevalent proximal objectives operate at a single scale and lack a principled mechanism for coupling token-level, trajectory-level, and higher-level hierarchical stability control. To bridge this gap, we derive the Aggregational Policy Censoring Objective (APC-Obj), the first exact unconstrained reformulation of sample-based TV-TRPO, establishing that clipping-based surrogate design and trust-region optimization are dual formulations of the same problem. Building on this foundation, we develop Fiber Bundle Gating (FBG), an algebraic framework that organizes sampled RL data as a fiber bundle and decomposes ratio gating into a base-level gate on trajectory aggregates and a fiber-level gate on per-token residuals, with provable first-order agreement with the true RL objective near on-policy. From APC-Obj and FBG we derive Fibration Policy Optimization (or simply, FiberPO), a concrete objective whose Jacobian is block-diagonal over trajectories, reduces to identity at on-policy, and provides better update direction thus improving token efficiency. The compositional nature of the framework extends beyond the trajectory-token case: fibrations compose algebraically into a Fibration Gating Hierarchy (FGH) that scales the same gating mechanism to arbitrary hierarchical depth without new primitives, as demonstrated by FiberPO-Domain, a four-level instantiation with independent trust-region budgets at the domain, prompt group, trajectory, and token levels. Together, these results connect the trust-region theory, a compositional algebraic structure, and practical multi-scale stability control into a unified framework for LLM policy optimization.
Tags
Links
- Source: https://arxiv.org/abs/2603.08239v1
- Canonical: https://arxiv.org/abs/2603.08239v1
Trouble viewing inline? Open PDF directly →
Full Text
258,280 characters extracted from source content.
Expand or collapse full text
Fibration Policy Optimization Chang Li1 Tshihao Tsu2 Yaren Zhang2 Chao Xue1 Xiaodong He1 1JD Explore Academy 2Carleton University lichang93, xuechao19, xiaodong.he@jd.com zhihaoxu, yarenzhang@cmail.carleton.ca Abstract Large language models are increasingly trained as heterogeneous systems spanning multiple domains, expert partitions, and agentic pipelines, yet prevalent proximal objectives operate at a single scale and lack a principled mechanism for coupling token-level, trajectory-level, and higher-level hierarchical stability control. To bridge this gap, we derive the Aggregational Policy Censoring Objective (APC-Obj), the first exact unconstrained reformulation of sample-based TV-TRPO, establishing that clipping-based surrogate design and trust-region optimization are dual formulations of the same problem. Building on this foundation, we develop Fiber Bundle Gating (FBG), an algebraic framework that organizes sampled RL data as a fiber bundle and decomposes ratio gating into a base-level gate on trajectory aggregates and a fiber-level gate on per-token residuals, with provable first-order agreement with the true RL objective near on-policy. From APC-Obj and FBG we derive Fibration Policy Optimization (or simply, FiberPO), a concrete objective whose Jacobian is block-diagonal over trajectories, reduces to identity at on-policy, and provides better update direction thus improving token efficiency. The compositional nature of the framework extends beyond the trajectory-token case: fibrations compose algebraically into a Fibration Gating Hierarchy (FGH) that scales the same gating mechanism to arbitrary hierarchical depth without new primitives, as demonstrated by FiberPO-Domain, a four-level instantiation with independent trust-region budgets at the domain, prompt group, trajectory, and token levels. Together, these results connect the trust-region theory, a compositional algebraic structure, and practical multi-scale stability control into a unified framework for LLM policy optimization. 1 Introduction Large language models are no longer single, monolithic policies: they are increasingly deployed and trained as heterogeneous systems—agentic pipelines spanning domains and tools, mixture-of-experts (MoE) architectures with conditional routing, and distributed/asynchronous training stacks where optimization noise and data nonstationarity are structural rather than incidental. In this regime, alignment via RLHF [3] must simultaneously handle multi-scale instability: token-level stochasticity, trajectory-level drift, and system-level heterogeneity (domains/experts/agents) interacting in the same update. Existing PPO-style “proximal” objectives [5, 6, 8] provide only coarse local controls (mostly per-token clipping) and limited diagnostics when failures arise from global structure (e.g., a drifting subset of trajectories, an expert partition, or a domain slice). This motivates importing more expressive mathematical structure, beyond new loss heuristics, to build controllers that can allocate stability budgets across the relevant global contexts, composably across arbitrarily many hierarchical levels (tokens, trajectories, prompt groups, domains etc). In this work, we develop Fiber Bundle Gating (FBG), an algebraic framework grounded in fiber bundle theory, and derive FiberPO-Trajectory (or simply, FiberPO) from it, a concrete policy optimization objective that decomposes trust-region maintenance into compositional global and local components, providing multi-scale stability control with first-order fidelity to the true RL objective near on-policy and a restorative gradient structure rarely explored in existing methods (see also [7]). The development proceeds through four stages. We begin from TRPO [4], whose trust-region radius depends on the discount factor γ. LLM RL effectively requires γ=1γ=1 (rewards are sparse and determined only at completion), and we prove that both TV- and KL-based TRPO trust regions then collapse to the reference policy, permitting only trivial updates (Theorem 2.1). This does not preclude trust-region-style stabilization in the γ=1γ=1 setting, but it shows that the classical radius from TRPO cannot be used as-is, and a useful trust-region design must decouple the trust-region maintaining mechanism from the particular radius prescribed by the monotonic improvement guarantee. To make this decoupling precise, we observe that a more fundamental gap must first be addressed. Clipping-based surrogates such as PPO [5] were originally motivated as tractable approximations to TRPO’s constraint, yet the precise relationship between ratio clipping and trust-region maintenance remains insufficiently understood, leaving open whether clipping merely imitates a trust region or can exactly reproduce one. We attempt to close this gap by deriving Aggregational Policy Censoring Objective (APC-Obj), an unconstrained clipping-based surrogate whose per-action clip bound Bs,aB_s,a decomposes the TV trust-region constraint into an explicit, cross-action-coupled form. We prove a formal equivalence (Theorem D.10): under standard function approximation assumptions, maximizing APC-Obj yields the same policy update as sample-based TV-TRPO, establishing that clipping-based surrogate design and trust-region policy optimization are dual formulations of the same optimization problem. Although APC-Obj itself yields trivial updates at γ=1γ=1 (the same vanishing as TRPO), its contribution is structural rather than algorithmic: it separates the trust-region maintenance mechanism (cross-action-coupled clipping) from the specific radius prescribed by the classical bound, so that the mechanism remains well-defined at any δ>0δ>0. This separation provides a reliable analytical anchor from which PPO [5], GRPO [6], and GSPO [9] can each be formally derived via identified relaxation steps, using a shared Ratio Gating Formalism (RGF), making each method’s departure from the trust-region optimum explicit and traceable. The APC-Obj-based taxonomy reveals a structural gap shared by all existing methods: token-wise objectives (PPO, GRPO) gate each ratio independently without directly bounding trajectory-level drift, while sequence-wise objectives (GSPO) collapse each trajectory to a single aggregate, suppressing within-trajectory variation. The relevant stability criterion, trajectory-level TV divergence, is inherently global, yet gradient information is inherently local. Few existing methods couple the two scales. To close this gap, we propose Fiber Bundle Gating (FBG), an algebraic framework that organizes sampled RLHF data as a fiber bundle, with tokens as the total space and global contexts (trajectories, partitioned into positive and negative drift channels) as the base space. FBG decomposes ratio gating into several operations: pushing token-level information to the base to form global aggregates, applying a base-level gate to maintain a trust-region budget at the context level, reflecting the gated signal back to tokens via a Markov kernel, and gating fiber-level residuals to preserve token-level gradients. We prove that this decomposition preserves first-order agreement with the true RL objective near on-policy whenever the atomic gates reduce to identity at the reference point (Theorem 4.5). We note that the fibration structure here is not an external import but is inherent in the data: methods that aggregate per-token quantities into context-level statistics implicitly invokes a pushforward πE∗ _E*, and methods that distribute a context-level signal back to individual tokens implicitly constructs a Markov kernel K. Making these operations explicit exposes the reflecting condition πE∗∘K=id _E* K=id_B, a structural constraint ensuring that global and local gates operate on orthogonal components without double-counting (Appendix E.3). Any gating map that avoids this double-counting while preserving first-order agreement necessarily satisfies the reflecting condition, and therefore implicitly re-derives the fibration decomposition. Building on APC-Obj and FBG, we derive FiberPO by starting from a δ-relaxed APC-Obj formulation and decomposing its coupled clipping constraint into two FBG components: a base-level aggregate gate gaggg agg that allocates a global trust-region budget δ across trajectory-level drift, and a fiber-level logcliplogclip that bounds each token’s residual deviation by ε . This decomposition inherits the trust-region mechanism of APC-Obj while providing independent control at two scales, an explicit budget separation is rarely explored in prior methods. The resulting Jacobian is block-diagonal over trajectories, reduces to identity at on-policy, and exhibits a restorative gradient in the rollback regime that actively corrects trajectory drift. This restorative property is absent in PPO, GRPO, and GSPO, which either zero the gradient (PPO/GRPO clipping) or suppress it uniformly (GSPO gating) when a trajectory drifts beyond the clipping boundary. The hierarchy of domains, prompt groups, trajectories, and tokens motivates a framework that scales beyond two levels. The fiber bundle formalism naturally supports this extension. Because fibrations compose algebraically, the same FBG gating mechanism extends to deeper hierarchies by chaining fibrations into a Fibration Gating Hierarchy (FGH). We demonstrate this by deriving FiberPO-Domain, a four-level instantiation (domain, prompt group, trajectory, token) that applies gaggg agg independently at each level of the hierarchy, with the gated residual at each level capturing only the deviation from the next-coarser aggregate. This construction requires no new gating primitives: the same algebraic decomposition that produces FiberPO at two levels produces FiberPO-Domain at four, providing per-domain and per-prompt-group trust-region control that is absent in all existing methods. The compositionality of fibrations is what allows the framework to scale to the hierarchical, multi-domain training regimes that modern LLM systems increasingly demand. In summary, this paper contributes: • Aggregational Policy Censoring Objective (APC-Obj), the first exact unconstrained reformulation of sample-based TV-TRPO, with a formal equivalence proof (Theorem D.10). APC-Obj establishes clipping-based surrogate design and trust-region policy optimization as dual formulations of the same problem, and provides a reliable analytical anchor from which PPO, GRPO, and GSPO are each derived via identified relaxations. • Fiber Bundle Gating (FBG) and its hierarchical generalization, Fibration Gating Hierarchy (FGH), a compositional algebraic framework that couples global (base) and local (fiber) stability control through density-based gating on a fiber bundle, with a first-order agreement guarantee near on-policy (Theorem 4.5). The algebraic compositionality of fibrations allows the same construction to extend to arbitrary hierarchical depth without introducing new gating primitives. • FiberPO-Trajectory, a concrete FBG instantiation (trajectory, token) derived from APC-Obj that decomposes trust-region control into a base-level aggregate gate (budget δ) and a fiber-level residual gate (budget ε ), with a block-diagonal, restorative Jacobian structure. • FiberPO-Domain, a four-level FGH instantiation (domain, prompt group, trajectory, token) that extends FiberPO-Trajectory to multi-domain training, providing independent trust-region budgets at each hierarchical level. 2 TRPO and the Discount Factor Obstruction The monotonic-improvement framework of Kakade and Langford [1], refined by TRPO [4], remains the primary theoretical foundation for trust-region policy optimization. TRPO guarantees monotonic improvement by constraining each policy update to a trust region defined via TV or KL divergence, with a trust-region radius governed by the discount factor γ through a surrogate-gap bound. In this section, we show that this γ-dependence creates a fundamental obstruction for episodic LLM RL, where the effective discount factor is γ=1γ=1. The classical surrogate-gap bound (Appendix B.1) shows that the gap between the true RL objective J(θ)J(θ) and its linear surrogate J(1)(θ|θold)J^(1)(θ\,|\, _ old) satisfies J(θ)−J(1)(θ|θold)≥−4γ‖A∙(θold)‖∞(1−γ)2DTVmax(θ∥θold)2,J(θ)-J^(1)(θ\,|\, _ old)\;≥\;-\, 4γ\,\|A^( _ old)_ \|_∞(1-γ)^2\;D_ TV (θ\,\|\, _ old)^2, (1) which confines TRPO updates to a trust region of radius δ(TRPO)=(1−γ)/(8γ)δ^( TRPO)=(1-γ)/(8γ) (Lemma B.2). In LLM RL, later tokens must not be discounted, since the correctness of a response is often determined only at completion, so the training objective effectively requires γ=1γ=1. Inspecting the bound (1), the penalty coefficient γ/(1−γ)2γ/(1-γ)^2 diverges as γ→1γ→ 1, forcing δ(TRPO)→0δ^( TRPO)→ 0 to maintain the guarantee. We formalize this observation as a vanishing theorem: Theorem 2.1 (TRPO vanishing theorem). When γ=1γ=1, both the TV-based and KL-based TRPO trust regions collapse to the reference policy (Definition B.1): ℬδ(TRPO)TV-TR(θold)=ℬδ(TRPO)KL-TR(θold)=πθold.B TV-TR_δ^( TRPO)( _ old)\;=\;B KL-TR_δ^( TRPO)( _ old)\;=\;\\, _ _ old\,\. Consequently, the only policy update permitted by TV-based TRPO’s trust region at γ=1γ=1 is the trivial one: πθnew=πθold _ _new= _ _old. Proof: see Appendix B.2. Theorem 2.1 does not imply that trust-region-style stabilization is impossible in the γ=1γ=1 setting. It shows that the specific radius from TRPO’s discounted analysis cannot be used as-is. A natural workaround, annealing γ<1γ<1 toward 11 during training [2], is impractical for LLMs, whose rewards are typically sparse and all-or-nothing, making discounted reward estimation unstable (see Appendix B.3 for discussion). Importantly, δ(TRPO)=0δ^( TRPO)=0 at γ=1γ=1 does not preclude the existence of a useful TV bound δ>0δ>0 that controls update stability in the γ=1γ=1 case. In Section 3, we provide both theoretical and empirical evidence for such a bound through the lens of GSPO [9]. The vanishing theorem thus motivates relaxing δ into a tunable hyperparameter while retaining the trust-region structure as a design guide for objective functions. This relaxed formulation is the common starting point for PPO [5], GRPO [6], GSPO [9], and our FiberPO, as we shall show. To make the connection between TRPO’s trust-region mechanism and these practical methods precise, we seek an unconstrained reformulation of sample-based TV-TRPO (Appendix B.1) that admits an explicit Ratio Gating Formalism (RGF, Section 3.1). Because PPO, GRPO, and GSPO each also admit RGF representations, casting TRPO into the same language will let us identify the exact relaxation steps that transform a trust-region-enforced source objective into each practical method. 3 Ratio Gating Formalism and APC-Obj The vanishing theorem (Theorem 2.1) shows that TRPO’s trust-region radius collapses at γ=1γ=1, but the trust-region structure itself remains a valuable design guide. To retain this structure, we first introduce the Ratio Gating Formalism (RGF), a unified framework that captures a broad family of proximal objectives by concentrating algorithmic choices into a single ratio gating map. We then use RGF to reformulate sample-based TV-TRPO (Appendix B.1) as an unconstrained clipping-based surrogate, Aggregational Censoring Policy Optimization (APC-Obj), which enforces the same policy update as the sample-based TV-TRPO objective (Theorem D.10; see Appendix D for the full proof). The motivation for casting this TRPO-equivalent objective specifically in RGF form is that PPO, GRPO, and GSPO each also admit explicit RGF representations (Appendix C.4). Expressing all methods in this shared language makes the relationship between the trust-region-enforced source (APC-Obj) and each practical method more transparent. Starting from this trust-region-enforced source objective, we show that PPO, GRPO, and GSPO each arise through identifiable relaxations that preserve the clipping structure and thereby inherit (partially) APC-Obj’s TV-penalty mechanism. We further establish that GSPO’s clipping implicitly maintains a TV-like trust region, providing empirical evidence that a useful bound of TV distance exists even at γ=1γ=1. 3.1 The Ratio Gating Formalism Definition 3.1 (Ratio Gating Formalism (RGF)). An RGF surrogate objective takes the form J^(θ|θold)=∑(s,a,ℐ)∈ℰμs,a,ℐ(r∙(θ|θold))s,a,ℐA^s,aθold, J(θ\,|\, _ old)\;=\; _(s,a,I)\,∈\,E _s,a,I\;G\! (r_ (θ\,|\, _ old) )_s,a,I\; A _ old_s,a, (2) where: • ℰE is the sampled index set of state–action pairs augmented with extra information ℐI (e.g., trajectory membership, timestep); • μs,a,ℐ _s,a,I are predetermined aggregation weights; • r∙(θ|θold)=rs,ar_ (θ\,|\, _ old)=\r_s,a\ with rs,a:=πθ(a|s)/πθold(a|s)r_s,a:= _θ(a\,|\,s)\,/\, _ _ old(a\,|\,s) are the importance-sampling ratios; • :ℝ>0ℰ→ℝ>0ℰG:R_>0^E _>0^E is the ratio gating map, the central design object that transforms ratio tuples into gated ratio tuples; • A^s,aθold A _ old_s,a is an estimate of the advantage function under πθold _ _ old. RGF organizes a wide family of surrogate objectives, including PPO [5], GRPO [6], GSPO [9], APC-Obj, and FiberPO. Because all these methods share the RGF form, the differences between them reduce entirely to their choice of (ℰ,μ,)(E,\,μ,\,G). In particular, deriving a trust-region-equivalent surrogate (APC-Obj) in RGF form lets us identify, for each practical method, the precise relaxation steps that transform the trust-region-enforced APC-Obj gating into the method’s own gating map. Appendix C.4 gives the explicit (ℰ,μ,)(E,\,μ,\,G) specifications for each method. 3.2 Aggregational Policy Censoring Objective (APC-Obj) We now construct a concrete RGF instance that is provably equivalent to sample-based TV-TRPO (Appendix B.1). Aggregational Policy Censoring Objective (APC-Obj) is obtained by solving the sample-based surrogate-gap objective (1) in closed form, yielding an unconstrained clipping-based surrogate whose clip bounds explicitly allocate the TV trust-region budget Tsδ(APC-Obj)T_sδ^( APC-Obj) in a cross-action-coupled form. Definition 3.2 (Aggregational Policy Censoring Objective, APC-Obj policy iteration111The APC-Obj objective can equivalently be written in the RGF form; see Appendix C.1.). The APC-Obj update selects θnew=argmaxθJ^APC-Obj(θ|θold) _ new= _θ\, J APC-Obj(θ| _ old), where J^APC-Obj(θ|θold)=1T∑(s,a,τ,t)∈¯[clip(rs,a−1,Tsδ(APC-Obj)−∑(s,a′,τ′,t′)∈¯s(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|)A^s,aθold+A^s,aθold]. J APC-Obj(θ| _ old)\;=\; 1T _(s,a,τ,t)∈ X [clip\! (r_s,a-1,\;T_sδ^( APC-Obj)-\!\!\!\!\!\! _ subarrayc(s,a ,τ ,t )∈ X_s\\ (a ,τ ,t )≠(a,τ,t) subarray\!\!\!\!\!\!|r_s,a -1| )\; A _ old_s,a\;+\; A _ old_s,a ]. (3) X denotes the set of all state–action pairs in the sampled trajectories, with ns,an_s,a the sample multiplicity of pair (s,a)(s,a). For each state s∈[]s [X], let s:=a∈∣(s,a)∈X_s:=\a (s,a) \ be the set of actions co-occurring with s. Define T:=∑(s,a)∈ns,aT:= _(s,a) n_s,a (total sample count) and Ts:=∑a∈sns,aT_s:= _a _sn_s,a (per-state count). We also denote by ¯ X the augmented space (s,a,τ,t)\(s,a,τ,t)\ of all sampled state–action pairs augmented with trajectory membership τ and time step t. Each element of ¯ X is a distinct sampled token, so |¯|=T| X|=T and |¯s|=Ts| X_s|=T_s. ¯ X is an example of an augmented index set ℰE used in Definition 3.1. δ(APC-Obj):=(1−γ)28γ‖A^∙θold‖∞M(A^∙θold)Tδ^( APC-Obj):= (1-γ)^28γ\,\| A _ old_ \|_∞\, M( A _ old_ )T (the argument A^∙θold A _ old_ makes the dependence on the advantage explicit), the per-entry clip bound is Bs,a,τ,t:=Tsδ(APC-Obj)−∑(s,a′,τ′,t′)∈¯s,(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|B_s,a,τ,t:=T_sδ^( APC-Obj)- _ subarrayc(s,a ,τ ,t )∈ X_s,\,(a ,τ ,t )≠(a,τ,t) subarray|r_s,a -1|, and clip(a,B):=clip(a,−B+,B+)clip(a,B):=clip(a,-B^+,B^+) with B+:=max(B,0)B^+:= (B,0). M depends on A^θold A _ old and is bounded by ‖A^∙θold‖∞\| A _ old_ \|_∞, see Lemma D.2. See Appendix C.1 and Appendix D for more details. The per-entry clip bound Bs,a,τ,tB_s,a,τ,t can be interpreted as: the per-state trust-region budget Tsδ(APC-Obj)T_sδ^( APC-Obj) minus the TV contribution already consumed by all other entries at the same state. This cross-entry coupling is what enforces a per-state TV constraint in aggregate, rather than merely bounding each token independently. When the remaining budget is exhausted (i.e. Bs,a,τ,t≤0B_s,a,τ,t≤ 0), the deviation rs,a−1r_s,a-1 is clipped to zero, censoring that entry’s contribution to the surrogate. Despite their superficially different structures (APC-Obj applies per-token clipping with cross-action coupling, while sample-based TV-TRPO maximizes a globally penalized objective), the two algorithms produce the same policy update. Theorem 3.3 (Sample-based TV-TRPO and APC-Obj equivalence; restatement of Theorem D.10). Suppose θ satisfies standard function approximation assumptions, APC-Obj and sample-based TV-TRPO (Definition D.1) produce the same policy update: πθnew(APC-Obj)=πθnew(TV-TRPO). _ _ new^( APC-Obj)\;=\; _ _ new^( TV-TRPO). The proof (Appendix D) proceeds in three stages: (i) we derive the maximizer of sample-based TV-TRPO (Theorem D.3), showing that the optimal ratio deviation is a scalar multiple of the per-state unit-TV maximizer, (i) we establish a retraction property (Lemma D.9), showing that APC-Obj’s clipping mechanism projects any trust-region violation strictly back inside the feasible set, and (i) we combine these to show that the APC-Obj maximizer must lie inside the trust region, at which point its objective reduces to the same linear program solved by sample-based TV-TRPO. ∎ Remark 3.4 (APC-Obj as the structural bridge for δ-relaxation). The vanishing theorem (Theorem 2.1) establishes that δ(TRPO)=δ(APC-Obj)=0δ^( TRPO)=δ^( APC-Obj)=0 at γ=1γ=1, rendering both TRPO and APC-Obj trivial as deployed algorithms. However, APC-Obj’s significance is structural rather than algorithmic: its RGF form cleanly separates the clipping mechanism (cross-action-coupled clipping, which remains well-defined for any δ>0δ>0) from the specific trust-region enforcing radius δ(APC-Obj)δ^( APC-Obj) inherited from the TRPO’s strict stability requirement. This separation is what makes δ-relaxation—replacing the vanishing δ(APC-Obj)δ^( APC-Obj) with a positive tunable hyperparameter—a precise, traceable operation rather than an ad hoc modification. The GSPO analysis below (Claim 3.5, Remark 3.6) then provides the evidence that a specific positive δ controls policy drift at γ=1γ=1, supporting the motivation for relaxation. APC-Obj therefore serves as a design guide for surrogate construction: PPO [5], GRPO [6], and GSPO [9] can each be recovered from APC-Obj through identifiable relaxation steps (Appendix C.5). Among these relaxations, GSPO is of particular interest because it directly addresses the open question raised by the vanishing theorem: does a positive TV trust-region radius δ>0δ>0 effectively maintain stability in training at γ=1γ=1? GSPO replaces δ(APC−Obj)δ (APC-Obj) with a positive tunable hyperparameter ϵ(GSPO)ε (GSPO) and introduces a direct connection between its clipping threshold and the TV trust-region radius, providing both theoretical and empirical evidence for an affirmative answer. Claim 3.5. GSPO (without minimization function) maintains approximately an average total-variation distance bound at the trajectory level with D¯TV(tj)≤δ∼ϵ(GSPO)/2 D_ TV (tj)≤δ ε (GSPO)/2. For definition of D¯TV(tj) D_ TV (tj) see Appendix A.3. Proof: see Appendix C.6. ∎ Remark 3.6. In practice, GSPO uses ϵ(GSPO)≈4×10−4ε (GSPO)≈ 4× 10^-4 [9], which translates to δ≈2×10−4≈0.014δ≈ 2× 10^-4≈ 0.014. GSPO runs stably at this scale, providing empirical evidence that a TV trust region with δ∼10−2δ 10^-2 remains effective for maintaining stability in LLM RL at γ=1γ=1. This confirms that ℬδTV-TRB TV-TR_δ is a meaningful design criterion even when TRPO’s own radius has vanished: the vanishing theorem (Theorem 2.1) shows that relaxation is necessary to have nontrivial policy improvement, while in practice the relaxation is totally reasonable. However, all methods derived from APC-Obj via these relaxations operate at a single scale: token-wise objectives (PPO, GRPO) gate each ratio independently without directly bounding trajectory-level drift, while sequence-wise objectives (GSPO) collapse each trajectory to a single aggregate and suppress within-trajectory variation. The relevant stability criterion, the trajectory-level average TV divergence D¯TV(tj) D_ TV (tj) (Appendix A.3), is defined by aggregating per-token ratios over a single trajectory, so neither independent token-wise clipping nor trajectory-level gating alone provides a principled coupling between the two scales. Because APC-Obj supplies a trust-region-enforced source objective, it can be composed with more structured gating mechanisms that operate across scales while retaining the underlying TV-based stability guarantee. In the next section, we introduce Fiber Bundle Gating (FBG) to address this gap. 4 Fiber Bundle Gating Framework The APC-Obj derivations in Section 3 provide a trust-region-enforced source objective and reveal that, token-wise proximal objectives (PPO, GRPO) gate each token’s importance ratio independently without directly bounding trajectory-level drift, while trajectory-wise objectives (GSPO, Claim 3.5) collapse each trajectory to a single aggregate ratio, reducing trajectory-level variation. Neither paradigm offers a direct coupling between these two scales. In this section, we introduce an algebraic framework that addresses this limitation by exploiting the natural two-level structure of sampled RLHF data. We first organize sampled token data as a fiber bundle (Section 4.1), and then define Fiber Bundle Gating (FBG), a density-based gating operator that decomposes stability control into explicit global and local components while preserving first-order agreement with the true RL objective near on-policy. (Section 4.2). 4.1 Fiber Bundle Model Sampled RLHF data has a natural two-level structure: every token belongs to a trajectory or global domain context. Global information arises by aggregating local quantities over a chosen domain or context; for example, the trajectory-level average total variation distance D¯TV(tj) D_ TV (tj) (Appendix A.3) aggregates per-token ratios within each trajectory. The notions of “local information” (per-token ratios) and “global information” (trajectory-level aggregates) are intuitive but imprecise; to formalize them and, crucially, to describe how they interact under gating, we represent both as densities on a manifold. Fiber bundle theory is an expressive mathematical language for this type of hierarchical decomposition. A fiber bundle πE:E→B _E:E→ B separates a total space E of local data points from a base space B of global contexts; for each context b∈Bb∈ B, the fiber πE−1[b] _E^-1[\b\] collects exactly the local data belonging to b. We organize sampled RLHF data as a fiber bundle of this form: policy ratios are first converted to densities on the total space E via a fibration decomposition map ℱF; all gating operations (global and local) are performed directly on these densities; and a recovery map ℛR converts the gated densities back to policy ratios for use in the surrogate objective. (see Appendix E.1, E.2, and E.3 for a formal definition). To this end, we first define the augmented sampled space (Appendix A.2): ¯:=(st(τ),at(τ),τ,t)⊆×Tjθold×ℕ. X:=\(s_t(τ),a_t(τ),τ,t)\ ×Tj _ old×N. Each element of ¯ X is a sampled state-action pair together with its trajectory membership and timestep. The space ¯ X serves as the index set ℰE in the RGF form (Definition 3.1). Remark 4.1. For mathematical completeness, even if a trajectory terminates at a finite step, we include a null state-action pair (∅,∅,τ,t)( , ,τ,t) after termination. The fiber bundle in our work consists of the following objects: Base space (B). The base space collects all classes or contexts. In this work, we define B:=Tjθold×−1,+1,B:=Tj _ old×\-1,+1\, so a typical element of the base is a tuple (τ,+1)(τ,+1) or (τ,−1)(τ,-1). Total space (E). The total (bundle) space consists of all individual token-level data points together with a sign label. We define E:=¯×−1,+1.E:= X×\-1,+1\. The sign channel −1,+1\-1,+1\ means positive and negative log-ratio contributions. Bundle projection (πE _E). The projection πE:E→B _E:E→ B maps each augmented token to its corresponding context: πE:¯×−1,+1→Tjθold×−1,+1πE(s,a,τ,t;l):=(τ,l) _E: X×\-1,+1\→ Tj _ old×\-1,+1\ _E(s,a,τ,t;\,l):=(τ,l) sending an augmented state-action pair to its trajectory while preserving the sign channel, which means a point in the base B (e.g., (τ,1)(τ,1)) will either correspond to the tokens sampled in τ with positive log-ratio contribution or the tokens sampled in τ with negative log-ratio contribution, depending on the sign label l (l is either +1+1 or −1-1). Fiber. For a given (trajectory, sign) tuple b∈Bb∈ B, the fiber Eb:=πE−1[b]E_b:= _E^-1[\b\] contains all the local points belonging to the class/context b. In our instantiation, πE−1[(τ,l)] _E^-1[\(τ,l)\] collects all sampled tokens from the trajectory τ with sign label l. Remark 4.2 (Choice of base space and extensibility). The choice of base space B determines the granularity of global control. Here we instantiate FBG on a topologically trivial (product) fiber bundle with B=Tjθold×−1,+1B=Tj _ old×\-1,+1\, where the base gate controls trajectory-level drift. The framework extends to richer global structure by composing fibrations into a hierarchy: enriching B with domain and prompt group indices produces per-domain and per-group aggregates, as realized concretely in FiberPO-Domain (Section 6.2). In each case, the fiber πE−1[b] _E^-1[\b\] collects exactly those tokens belonging to class b, and the local-global decomposition separates class-level drift from within-class variation. See Remark E.2 for further discussion. Figure 1: Fiber bundle model for sampled RLHF data. The base space B=Tjθold×−1,+1B=Tj _ old\!×\!\-1,+1\ consists of two separate lines indexed by the sign label l∈−1,+1l∈\-1,+1\, encoding trajectory-level global information as densities on B. Each fiber πE−1[(τ,l)] _E^-1[\(τ,l)\] collects the per-token data points belonging to trajectory τ with sign l, encoding local information as densities on the total space E. The maps ℱF and ℛR convert between policy ratios and densities. Figure 1 illustrates this fiber bundle structure: the projection πE _E maps each token to its trajectory context. The densities on the base space B and bundle space E will be introduced in the next section, where we define the FBG operator that performs stability control by gating these densities. 4.2 Fiber Bundle Gating Building on the fiber bundle model (Section 4.1) and the RGF language (Section 3.1), we now introduce Fiber Bundle Gating (FBG), a framework that formally couples global and local information by operating on densities over the bundle and base space. The FBG operator acts in four stages: it pushes the token-level density forward to the base to form the trajectory-level base density; applies a base-level atomic gate to enforce a trust-region budget at the trajectory level; reflects the gated global signal back to the total space via a Markov kernel K; and gates the residual density on each fiber, capturing each token’s deviation from the trajectory aggregate, to prevent individual token spikes. The reflecting condition on K ensures the two components decouple cleanly: the residual carries no base-level information, so global and local gating operate independently and compositionally. Once an FBG instance is specified, it induces a surrogate objective in RGF form (Definition 3.1), with the ratio gating map =ℛ∘G∘ℱG=R G determined by the bundle and gating functions (Definition 4.4). All four constituent objects are defined relative to the chosen base space B and therefore extend naturally to richer global structures such as domain or temporal groupings (see Appendix E.3 for detailed exposition). Definition 4.3 (Fiber Bundle Gating). A Fiber Bundle Gating instance is specified by the tuple (E,B,πE,K,gBase,gFiber,−;ℱ,ℛ)(E,B, _E,K,g_ Base,g_ Fiber,-;\,F,R), where πE:E→B _E:E→ B is a fiber bundle. Let E and B denote the spaces of densities on E and B, respectively. The FBG gating function G(πE,K,gBase,gFiber,−):→G_( _E,K,g_ Base,g_ Fiber,-):E is defined as: G(σ):=K((⨁b∈BgBase,b)(πE∗σ))+((⨁i∈EgFiber,i,πE∗(σ))(σ−K(πE∗σ))),G(σ):=K\! ( ( _b∈ Bg_ Base,b )( _E*σ) )+ ( ( _i∈ Eg_ Fiber,i, _E*(σ) ) (σ-K( _E*σ) ) ), (4) where the constituent objects are: 1. Fiber bundle and density spaces. πE:E→B _E:E→ B is a fiber bundle. Associated to E and B are their density spaces E and B, consisting of all densities on the total space E and the base space B, respectively (see Appendix E.2 for the formal definition of densities on manifolds). 2. Fibration decomposition and recovery maps. The fibration decomposition map ℱ:ℝ>0ℰ→F:R_>0^E converts policy ratios to densities on the total space, and the recovery map ℛ:→ℝ>0ℰR:E _>0^E is a left-inverse of ℱF that converts densities back to policy ratios. 3. Reflecting Markov kernel. K:→K:B is a Markov kernel satisfying the reflecting condition πE∗∘K=id _E* K=id_B. The subtraction σ−K(πE∗σ)σ-K( _E*σ) decouples the global information encoded in the base density πE∗σ _E*σ from σ, through the Markov kernel K. We need to ensure that, after decoupling, the residual density should no longer contribute to any global information, so we must have πE∗(σ−K(πE∗σ))=0 _E*(σ-K( _E*σ))=0, which directly implies our requirement for K: πE∗∘K=id _E* K= id_B. 4. Atomic gating functions. gBase,b:b1B→b1Bg_ Base,b:D^1_bB ^1_bB and gFiber,i,pB:i1E→i1Eg_ Fiber,i,p_B:D^1_iE ^1_iE are pointwise gating functions on the localized base and fiber densities, respectively, where b∈Bb∈ B, i∈Ei∈ E, and pB∈p_B . The fiber gating function gFiber,i,pBg_ Fiber,i,p_B depends on the base density pBp_B, reflecting the fact that local gating may be conditioned on global context. The FBG gating function (4) operates in two steps. First, the base component: the density σ is pushed forward to the base via πE∗ _E* to form a base density, gated by gBaseg_ Base, and reflected back to the total space via the Markov kernel K. Second, the fiber component: the residual σ−K(πE∗σ)σ-K( _E*σ), which captures local information after removing the global contribution, is gated by gFiberg_ Fiber. The reflecting condition on K ensures that these two components are orthogonal in the sense that the residual carries no base-level information. Definition 4.4 (FBG gating form). Given an FBG instance (E,B,πE,K,gBase,gFiber,−;ℱ,ℛ)(E,B, _E,K,g_ Base,g_ Fiber,-;\,F,R), the associated surrogate objective takes the RGF form: J^(G;ℱ,ℛ)(θ|θold):=∑τ∈Tjπθold[∑t=0Tτ−11Tτℛ∘G∘ℱ(r∙)st,atA^st,at], J_(G;F,R)(θ| _ old):= _τ _ _ old [ _t=0^T_τ-1 1T_τ\,R G (r_ )_s_t,a_t\; A_s_t,a_t ], (5) where the induced ratio gating map is the composition =ℛ∘G∘ℱ:ℝ>0¯→ℝ>0¯G=R G :R_>0 X _>0 X. Fiber Bundle Gating guarantees the first order approximation to the true RL objective near on-policy: Theorem 4.5 (First-order agreement of FBG). Suppose the atomic gating functions satisfy the identity conditions at the on-policy density, i.e., for all i∈Ei∈ E and b∈Bb∈ B: gFiber,i,pB(ℱ()i−K(πE∗ℱ())i)=ℱ()i−K(πE∗ℱ())ig_ Fiber,i,p_B(F(1)_i-K( _E*F(1))_i)=F(1)_i-K( _E*F(1))_i gFiber,i,pB′(ℱ()i−K(πE∗ℱ())i)=1,gBase,b(πE∗ℱ())=πE∗ℱ(),gBase,b′(πE∗ℱ())=1.g _ Fiber,i,p_B(F(1)_i-K( _E*F(1))_i)=1, g_ Base,b( _E*F(1))= _E*F(1), g_ Base,b ( _E*F(1))=1. Then the associated FBG objective agrees with the true RL objective to first order at θ=θoldθ= _ old: J^(G;ℱ,ℛ)(θold|θold)=J(θold),∇θJ^(G;ℱ,ℛ)(θ|θold)|θ=θold=∇θJ(θ)|θ=θold. J_(G;F,R)( _ old| _ old)=J( _ old), _θ J_(G;F,R)(θ| _ old) |_θ= _ old= _θJ(θ) |_θ= _ old. Proof See Appendix E.4. ∎ Remark 4.6 (Global-to-local information exchange). Beyond first-order agreement, FBG’s decoupling mechanism is essential for exchanging global gating effects with individual token ratios control. General RGF gating maps only operate on ratio tuples r∙r_ , without the structured decomposition, naively combining global and local gating. This may produce undesirable gradients, or entangle the two scales so that neither gate operates at its intended granularity. FBG resolves this: gBaseg_ Base gates the trajectory-level base density in isolation, while gFiberg_ Fiber gates local residual σ−K(πE∗σ)σ-K( _E*σ)—the per-token variation from which all global influence has been removed. The reflecting condition guarantees that these two operations are orthogonal and non-interfering (see Appendix E.3 for a detailed discussion). Remark 4.7 (Extensibility via the base space). All four constituent objects of FBG are defined relative to the chosen base space B. Extending B to include additional classifying factors (e.g., domain, temporal grouping) naturally extends the FBG instance: the pushforward πE∗ _E* produces finer aggregates, the Markov kernel K reflects finer base densities back to the bundle, and the atomic gates gBaseg_ Base, gFiberg_ Fiber operate per class and locally, respectively. Theorem 4.5 guarantees first-order agreement for any such extension, provided the atomic gates satisfy the identity conditions at the on-policy point. The FiberPO-Trajectory objective derived in Section 5 is the simplest nontrivial instantiation of this framework. In Section 6, we generalize FBG to the Fibration Gating Hierarchy (FGH) and derive FiberPO-Domain, a four-level instantiation that incorporates domain and prompt group structure (see also Remark E.2). 5 FiberPO Starting from the APC-Obj objective (Section 3.2), we derive the FiberPO surrogate by applying a sequence of controlled transformations yielding a Fiber Bundle Gating form (Definition 4.3). The derivation proceeds through four derivation steps (detailed in Appendix E.5): 1. δ-relaxation: replace the vanishing TRPO/APC-Obj threshold δ(APC-Obj)δ^( APC-Obj) with a positive tunable hyperparameter δ. This step is necessitated by the vanishing theorem (Theorem 2.1), which shows the classical radius is zero at γ=1γ=1, and justified by the GSPO analysis (Claim 3.5), which demonstrates that a positive δ∼10−2δ 10^-2 effectively controls policy drift in the undiscounted setting. The clipping mechanism in APC-Obj’s RGF form remains well-defined at the relaxed radius (Remark 3.4). 2. Logarithmic approximation: move to log-ratio coordinates via logr≈r−1 r≈ r-1 and 1+clip(⋯)≈exp(clip(⋯))1+clip(·s)≈ (clip(·s)). Working in logr∈ℝ r and exponentiating the result guarantees r>0r>0, so all gating operations (clipping, piecewise-linear gates) can be designed with unconstrained linear algebra on ℝR rather than constrained to preserve positivity. 3. Sequence-level aggregation: restrict the cross-token coupling to tokens within the same trajectory (replacing the all-token sum with a per-trajectory sum) and replace the per-state budget TsδT_sδ with a per-trajectory budget TτδT_τδ. 4. Clipping decomposition: decompose the coupled clipping term into an explicit base-level gate gaggg agg on the trajectory aggregate ratios sτ+,sτ−s_τ^+,s_τ^- and a fiber-level logcliplogclip on per-token residuals, with per-channel trust-region budgets C+,C−C^+,C^- (C++C−=δC^++C^-=δ) that accommodate the intrinsic asymmetry logsτ−≥logsτ+ s^-_τ≥ s^+_τ. Derivation 4 is the central and distinctive step: it transforms a single coupled clipping bound into two explicit, complementary gating components that map directly onto the FBG architecture (base gate + fiber gate). The resulting objective takes the following form. Definition 5.1 (Fiber-Aware Clipping Policy Optimization (FiberPO)). The FiberPO objective in RGF form is: J^FiberPO(θ|θold)=∑(s,a,τ,t)∈¯1|Tjθold|1Tτ⋅(r∙)s,a,τ,t⋅A^s,a, J^FiberPO(θ| _ old)= _(s,a,τ,t)∈ X 1|Tj _ old| 1T_τ·G(r_ )_s,a,τ,t\;· A_s,a, (6) where ℰ=¯:=(st(τ),at(τ),τ,t)⊆×Tjπθold×ℕE= X:=\(s_t(τ),a_t(τ),τ,t)\ ×Tj _ _ old×N is the augmented state-action pair space222In the RGF (Definition 3.1), each index i∈ℰi carries extra information ℐI; here the extra information is the trajectory–timestep pair (τ,t)(τ,t), so each element of ℰE is an augmented state-action pair (s,a,τ,t)(s,a,τ,t). See Appendix A.2 for details., TτT_τ is the length of trajectory τ, and the ratio gating map =ℛ∘G∘ℱG=R G (Definition 4.4) acts on the full tuple of importance ratios r∙r_ and produces a gated ratio for each augmented token (s,a,τ,t)∈¯(s,a,τ,t)∈ X. The fully expanded objective and its equivalent expression in LLM notation (g,j,i)(g,j,i) are given in Definition E.9 and Appendix E.6, respectively. The FBG gating map G for each token i≡(s,a,τ,t)i≡(s,a,τ,t) is a multiplicative decomposition into a trajectory-level base weight and a token-level gated residual: (r∙)i=exp∘gagg(logsτ+,C+,Tτ)exp∘gagg(logsτ−,C−,Tτ)⏟wτbase:base weight (after base gating)⋅logclip((sτ(li))−liri,ϵ)logclip((sτ(−li))−li,ϵ)⏟r~ifiber:gated residual (after fiber gating),G(r_ )_i\;=\; \;g agg( s_τ^+,\;C^+,\;T_τ) \;g agg( s_τ^-,\;C^-,\;T_τ)_w base_τ\;:\;base weight (after base gating)\;·\; logclip\! ((s_τ^(l_i))^-l_i\,r_i,\;ε ) logclip\! ((s_τ^(-l_i))^-l_i,\;ε )_ r_i fiber\;:\;gated residual (after fiber gating), (7) Each term will be explained carefully in the subsections below. Overall, we call wτbasew base_τ the base weight and r~ifiber r_i fiber the gated residual. This decomposition directly reflects the FBG architecture (Definition 4.3): the base gate gBaseg_ Base operates on pushed-forward densities on B, and the fiber gate gFiberg_ Fiber operates on the residual after reflecting back to E via the Markov kernel K. The base weight wτbasew base_τ depends only on trajectory-level aggregates and is shared by all tokens in trajectory τ; it controls how much gradient signal the trajectory as a whole is permitted to contribute. The gated residual r~ifiber r_i fiber captures each token’s deviation from the trajectory aggregate on its respective sign channel, gated by logcliplogclip to prevent individual token spikes. The constituents of each component are defined in the following two subsections. 5.1 Base Weight (Base Gate) The base weight is wτbase:=exp∘gagg(logsτ+,C+,Tτ)exp∘gagg(logsτ−,C−,Tτ),w base_τ\;:=\; \;g agg( s_τ^+,\;C^+,\;T_τ) \;g agg( s_τ^-,\;C^-,\;T_τ), (8) where the constituent quantities are: Trajectory aggregate ratios. The positive and negative aggregate ratios decompose the trajectory-level drift by sign: logsτ+:=1Tτ∑t=0Tτ−1max(logrst(τ),at(τ), 0),logsτ−:=1Tτ∑t=0Tτ−1max(−logrst(τ),at(τ), 0). s^+_τ:= 1T_τ _t=0^T_τ-1 ( r_s_t(τ),a_t(τ),\,0), s^-_τ:= 1T_τ _t=0^T_τ-1 (- r_s_t(τ),a_t(τ),\,0). (9) Here logsτ+ s^+_τ averages the positive log-ratio contributions (tokens whose likelihood increased under πθ _θ) and logsτ− s^-_τ averages the negative ones (tokens whose likelihood decreased). We refer to logsτ+ s^+_τ and logsτ− s^-_τ as the positive and negative channels’ drift, or the signed-trajectory mean, respectively. One of the merits of splitting by sign (for more theoretical reasoning, see Appendix E.5.4) rather than averaging all log-ratios is that the total trajectory drift logr¯τ=logs+−logs− r_τ= s^+- s^- can be small even when both logs+ s^+ and logs− s^- are individually large. In this case, the trajectory contains many token-ratios that have shifted substantially in both positive and negative directions. (i) If one simply performs logr¯τ r_τ mean-subtraction, the mean-subtracted individual token will be large and easily clipped by token-level logcliplogclip (for details, see ratio decomposition in 5.2), and (i) in this case, the trajectory TV distance D¯^TV(τ)≈(1/Tτ)∑t|logrst,at| D_ TV^(τ)≈(1/T_τ) _t| r_s_t,a_t| is large and may need control, however keeping only logr¯τ r_τ would mask this need for control. By tracking each sign channel independently, gaggg agg detects high total variation (in i) in the importance weights even when the signed average nearly cancels, can apply rollback on the offending channel’s aggregate ratio without suppressing the well-behaved aggregate ratio, while also retaining many well-behaved local tokens that are located near the possibly large logs± s^± unclipped (from i), providing more nuanced control over the optimization process. Aggregate gating function gaggg agg. The function gaggg agg is a piecewise-linear gate that transforms the aggregate log-ratio on each sign channel: gagg(x,C,Tτ):=xif |x|≤Csign(x)(Tτ+1)C−Tτxif C<|x|<(1+Tτ−1)C0otherwiseg agg(x,C,T_τ):= \ array[]lx&if |x|≤ C\\[5.0pt] sign(x)(T_τ+1)C-T_τx&if C<|x|<(1+T_τ^-1)C\\[5.0pt] 0&otherwise array . (10) where TτT_τ is the sequence length and C∈C+,C−C∈\C^+,C^-\ is the per-channel trust-region budget. The three regimes correspond directly to the three APC-Obj clipping regimes (P), (R), (Z) (Appendix C.2), lifted to the aggregate level via the clipping decomposition (Appendix E.5, Eq. 77): • Pass-through (P) (|x|≤C|x|≤ C): the aggregate drift is within the trust-region budget. The gate outputs x unchanged, preserving the full importance-sampling correction on this channel. This corresponds to the aggregate-level counterpart of APC-Obj regime (P) where D^TV(s)≤δ(APC-Obj) D_ TV(s)≤δ^( APC-Obj) (Eq. 61): the total budget consumption (C1) + (C2) does not exceed the budget (B), and the clipping is inactive. • Rollback (R) (C<|x|<C∗:=(1+Tτ−1)C<|x|<C^*:=(1+T_τ^-1)C): the aggregate drift has exceeded the budget. The gate reverses slope to −Tτ-T_τ, producing a restorative gradient that actively opposes the drift direction and pushes the aggregate back toward on-policy. The term “rollback” follows the terminology of [7], who introduced a similar slope-reversing mechanism at the per-token level. This is the aggregate-level counterpart of APC-Obj regime (R) (Eq. 62), where D^TV(s)>δ(APC-Obj) D_ TV(s)>δ^( APC-Obj) but the current token’s deviation (C2) has not individually exhausted the budget overshoot, so the clip suppresses |rs,a−1||r_s,a-1| without zeroing it333In the per-state APC-Obj objective (Eq. 60), regime (R) requires |rs,a−1|/Ts>D^TV(s)−δ|r_s,a-1|/T_s> D_ TV(s)-δ, equivalently 0<|clip(rs,a−1,(B)−(C1))|<|rs,a−1|0<|clip(r_s,a-1,(B)-(C1))|<|r_s,a-1|. After trajectory-level aggregation (Appendix E.5.3), logarithmic approximation (Appendix E.5.2), and clipping decomposition (Eq. 77), this per-token condition on |rs,a−1||r_s,a-1| lifts to a condition on the aggregate log-ratio logsτ(l) s_τ^(l) that still depends on individual logri r_i (i.e., |logri|/Tτ>logsτ(li)−δ| r_i|/T_τ> s^(l_i)_τ-δ). Since TτT_τ is typically large in LLM settings, the per-token dependence becomes negligible and the condition reduces to the uniform cross-token condition C(l)<logsτ(l)<C∗(l)C^(l)< s_τ^(l)<C^*(l) on the aggregate ratio alone.. The remaining budget induces a linear penalty whose gradient opposes further drift. The rollback regime is the unique continuous piecewise-linear interpolation between pass-through and zeroing, with slope fixed by the APC-Obj decomposition structure. • Zeroed (Z) (|x|≥C∗|x|≥ C^*): the budget is fully consumed. The gate outputs 0, completely blocking gradient signal for this channel. This is the aggregate-level counterpart of APC-Obj regime (Z) (Eq. 63), where the cross-token consumption (C1) alone exceeds the total budget (B), making the residual clip bound nonpositive and forcing the clip output to zero. The rollback regime has width C/TτC/T_τ and shrinks for longer trajectories, so gaggg agg approaches a hard clip at ±C± C as Tτ→∞T_τ→∞. Per-channel budgets. C+C^+ and C−C^- are the per-channel trust-region budgets satisfying C++C−=δC^++C^-=δ, with the recommendation C−<C+C^-<C^+ to compensate for the intrinsic KL bias logsτ−≥logsτ+ s^-_τ≥ s^+_τ (Appendix E.5). The symmetric case C+=C−=δ/2C^+=C^-=δ/2 is a special case. Figure 2: Base weight visualization. Panel (a) shows the aggregate gate gaggg agg (Eq. 10) with three regimes: pass-through (|x|≤C|x|≤ C, slope 11), rollback (C<|x|<C∗C<|x|<C^*, slope −k-k), and zeroed (|x|≥C∗|x|≥ C^*, output 0). As k=Tτk=T_τ increases, the rollback regime narrows (width C/TτC/T_τ) and gaggg agg approaches a hard clip at ±C± C. Panel (b) shows the base weight logwτbase w_τ base (Eq. 8) in (logs+,logs−)( s^+, s^-)-space with asymmetric thresholds C+=0.20C^+=0.20, C−=0.1C^-=0.1 (k=2k=2). Dashed lines mark the budget boundaries C±C^± (onset of rollback); dotted lines mark the full-gating thresholds C∗±C^*± (onset of zeroing). The five global regimes (G-I through G-I) follow a non-monotonic pattern: |logw|| w| rises through the rollback onset (G-I,r), peaks when one channel is fully gated (G-I), declines under mutual rollback (G-I,r), and collapses to zero when both channels are fully gated (G-I, wτbase=1w base_τ=1). Regime analysis of the base weight. Figure 2 visualizes gaggg agg and the resulting base weight. Since wτbasew base_τ is a ratio of two independently gated channels (Eq. 8), the combined behavior depends on which regime each channel occupies. The key structural feature is that wτbasew base_τ depends on logsτ+ s_τ^+ and logsτ− s_τ^- independently: two trajectories with identical mean log-ratio 1Tτ∑logrt=logs+−logs− 1T_τΣ r_t= s^+- s^- but different D¯^TV(τ) D_ TV^(τ) may produce same base weights, depending on whether (logs+,logs−)( s^+, s^-) lies in the G-I region or not. A trajectory with large positive and negative shifts falls outside G-I, triggering rollback on both channels even when the shifts nearly cancel; this produces a gated base weight. In contrast, a trajectory in G-I has base weight exactly exp(logs+−logs−) ( s^+- s^-), since gaggg^agg acts as the identity. The five global regimes are: • G-I (both pass-through): wτbase=exp(logs+−logs−)w base_τ= ( s^+- s^-), the unmodified importance-sampling ratio. The base gate is transparent. • G-I,r (one channel rolling back): the active rollback produces a restorative gradient that opposes the drifting channel; |logwτbase|| w base_τ| is rising. • G-I (one channel fully gated): |logwτbase|| w base_τ| peaks at approximately C(l)C^(l) (the surviving channel l’s budget), delivering maximum one-sided correction. • G-I,r (mutual rollback): both channels are past their thresholds; |logwτbase|| w base_τ| is declining. • G-I (both fully gated): wτbase=1w base_τ=1 and the trajectory-level gradient contribution vanishes entirely. 5.2 Gated Residual (Fiber Gate) The gated residual captures each token’s deviation from the trajectory aggregate, gated to prevent individual token spikes: r~ifiber:=logclip((sτ(li))−liri,ϵ)logclip((sτ(−li))−li,ϵ). r_i fiber\;:=\; logclip\! ((s_τ^(l_i))^-l_i\,r_i,\;ε ) logclip\! ((s_τ^(-l_i))^-l_i,\;ε ). (11) We now define its constituent elements. Log-clipping function. The log-clipping function is: logclip(x,ϵ):=exp(clip(logx,±ϵ)).logclip(x,ε)\;:=\; \! (clip( x,\;±ε) ). (12) We explain the specific choice of log space and thus log clip in Appendix E.5.2. Intuitively, in non-log space, this is equivalently clip(x,e−ϵ,e+ϵ)clip(x,\;e^-ε,\;e^+ε): the argument is clamped to the interval [e−ϵ,e+ϵ][e^-ε,e^+ε]. The asymmetry is inherent: the upper bound e+ϵ≈1+ϵ+ϵ2/2e^+ε≈ 1+ε+ε^2/2 is further from 11 than the lower bound e−ϵ≈1−ϵ+ϵ2/2e^-ε≈ 1-ε+ε^2/2, so a symmetric ϵε in log-space naturally produces an asymmetric clip in ratio space. This subsumes the effect of asymmetric clip bounds ϵ+/ϵ−ε^+/ε^- used in PPO/GRPO without introducing an additional hyperparameter. Sign label and channel decomposition. For each token i, the sign assignment li:=sign(logri)∈+1,−1l_i\;:=\;sign( r_i)\;∈\;\+1,\,-1\ (13) indicates whether the new policy has increased (li=+1l_i=+1) or decreased (li=−1l_i=-1) the likelihood of token i relative to the reference policy. A parenthesized sign-label superscript selects the sign channel: sτ(li)=sτ+s_τ^(l_i)=s_τ^+ when li=+1l_i=+1 and sτ−s_τ^- when li=−1l_i=-1; the negated form sτ(−li)s_τ^(-l_i) selects the opposite channel (see Appendix A.4 for notation conventions). This sign partitions the tokens within each trajectory into two channels (positive-ratio and negative-ratio, reflecting the FBG fiber bundle structure of Section 4.1, see Appendix E.5.4 for the instantiation with signed channels). The gated residual (11) decomposes into two sign-channel cases: r~ifiber=logclip((sτ+)−1ri,ϵ)logclip((sτ−)−1,ϵ)if li=+1,logclip((sτ−)+1ri,ϵ)logclip((sτ+)+1,ϵ)if li=−1. r_i fiber= cases logclip\! ((s_τ^+)^-1\,r_i,\;ε )logclip\! ((s_τ^-)^-1,\;ε )&if l_i=+1,\\[11.99998pt] logclip\! ((s_τ^-)^+1\,r_i,\;ε )logclip\! ((s_τ^+)^+1,\;ε )&if l_i=-1. cases In each case, the numerator involves only the same-sign aggregate sτ(li)s_τ^(l_i), while the denominator involves the opposite-sign aggregate sτ(−li)s_τ^(-l_i). Fiber residual and the gated residual in log-space. Define the fiber residual ui:=lilogri−logsτ(li)u_i\;:=\;l_i r_i- s_τ^(l_i) (14) and the opposite-sign aggregate vi:=−logsτ(−li).v_i\;:=\;- s_τ^(-l_i). (15) The gated residual (11) can be written equivalently in terms of uiu_i and viv_i as: r~ifiber=logclip(eliui,ϵ)logclip(elivi,ϵ)=exp(clip(liui,ϵ)−clip(livi,ϵ)). r_i fiber\;=\; logclip\! (e^l_i\,u_i,\;ε )logclip\! (e^l_i\,v_i,\;ε )= (clip\! (l_i\,u_i,\;ε )-clip\! (l_i\,v_i,\;ε )). (16) The fiber residual uiu_i (14) subtracts the same-sign trajectory mean (logsτ(li) s_τ^(l_i)) from each token’s log-ratio. The numerator’s logcliplogclip acts on eliuie^l_iu_i, which involves only the same-sign channel aggregate sτ(li)s_τ^(l_i); this prevents the opposite channel’s aggregate ratio from directly entering the numerator clip, avoiding opposite-channel contamination. The denominator’s logcliplogclip acts on elivie^l_iv_i, incorporating the opposite-sign aggregate sτ(−li)s_τ^(-l_i) to complete the subtraction by trajectory-mean log-ratio (logr¯τ:=(1/Tτ)∑tlogrst,at r_τ:=(1/T_τ) _t r_s_t,a_t). Ratio decoupling. When neither the numerator nor denominator logcliplogclip is saturated (i.e., the internal clipclip is inactive: |ui|≤ϵ|u_i|≤ε and |vi|≤ϵ|v_i|≤ε), the exp and log cancel and we obtain: logr~ifiber=li(ui−vi)=logri−logr¯τ, r_i fiber=l_i(u_i-v_i)= r_i- r_τ, where, recalling the log-aggregate ratio logsτ:=1Tτ∑t=0Tτ−1logrst(τ),at(τ) s_τ:= 1T_τ _t=0^T_τ-1 r_s_t(τ),a_t(τ) (see also Appendix C.6 for definition), logr¯τ:=logsτ=1Tτ∑t=0Tτ−1logrst(τ),at(τ) r_τ\;:=\; s_τ\;=\; 1T_τ _t=0^T_τ-1 r_s_t(τ),a_t(τ) is the trajectory mean log-ratio. Thus the gated residual is a trajectory-mean-centered quantity: it measures each token’s deviation from the trajectory average, with the same-sign/opposite-sign channel split ensuring that the ϵε-clipping acts within each sign channel independently. First, the total mean-centering logr¯τ r_τ has the advantage of recovering the true linear surrogate and thus true RL objective near on-policy, when combined with the base weight’s pass-through behavior in G-I. Second, the signed-trajectory-mean centering quantity liui=logri−lilogsτ(li)l_iu_i= r_i-l_i s^(l_i)_τ (from the sign channel decomposition also (13)) that passed into the numerator (equivalently may be expressed as exp∘clip(liui,ϵ) (l_iu_i,ε)) provides the following additional advantages: Fiber residual clip isolates signed trajectory-level drift: even when the trajectory aggregate logsτ(li) s^(l_i)_τ is large (indicating substantial overall policy shift), individual tokens are regulated only by how much they deviate from the trajectory norm. Tokens that move relatively closely with the same-sign trajectory-mean (trajectory-level drift) always pass through the FiberPO gating function (G of FiberPO) unattenuated, thus preserving well-behaved tokens their full gradient signal, allowing them to provide finer update direction, leading to higher token-efficiency, even when (signed-)trajectory-level drift are large (logsτ±>ϵ s^±_τ>ε). And in this case, methods like PPO/GRPO etc. that clip logri r_i directly would have more than half tokens to exceed the clipclip bound simultaneously and destroy token-level discriminated signal. Fiber residual isolates inter-trajectory and intra-trajectory drift: To demonstrate this point, consider two trajectories answering “Name a famous landmark”: “I love Paris and the Eiffel Tower” vs. “I love Rome and the Colosseum.” Globally, the policy may strongly prefer the Paris response, perhaps it scores higher overall, so the aggregate ratio sτ+s_τ^+ for that trajectory is large. Without decoupling, this global preference bias leaks into every token’s gradient: the token Colosseum in the Rome trajectory receives a weaker learning signal not because the token-level association “Rome → Colosseum” is poor, but simply because its trajectory is globally less preferred. The residual decomposition in Eq. 7 prevents this contamination. By subtracting the trajectory aggregate from each token’s log-ratio, the fiber gate r~ifiber r_i fiber isolates the pure local association, how much “Colosseum” co-varies with “Rome” relative to what the trajectory drift alone would predict, and gates it independently via logcliplogclip. Within each trajectory, token-level learning thus operates at a uniform, unbiased scale: P(Colosseum∣Rome)P(Colosseum ) and P(Eiffel Tower∣Paris)P(Eiffel Tower ) are each refined on their own statistical merits, free from the global preference P(Paris trajectory)≫P(Rome trajectory)P(Paris trajectory) P(Rome trajectory). The base weight wτbasew base_τ then re-couples the trajectory-level preference when the two scales are composed, so that global significance is preserved without polluting local precision. This is the orthogonal, non-interfering decomposition guaranteed by the reflecting condition πE∗∘K=id _E* K=id_B (Appendix E.3). Regime analysis of the gated residual. To make the behavior of the ϵε-clip explicit, we partition trajectories into three local regimes according to how many tokens’ fiber residuals exceed the clip threshold: • L-I (unclipped): maxi|ui|<ϵ _i|u_i|<ε. Every token retains its full per-token self-gating; the fiber gate(gated residual r~ifiber r fiber_i) imposes no restriction and faithfully tracks each token’s deviation from the trajectory mean log-ratio. • L-I (selective clipping): 0<nϵ<Tτ0<n_ε<T_τ, where nϵn_ε counts tokens with |ui|≥ϵ|u_i|≥ε. Outlier tokens lose their direct per-token gradient while well-behaved tokens retain full signal—the regime where most of the meaningful per-token signals are retained. • L-I (all logcliplogclip’s saturated): mini|ui|≥ϵ _i|u_i|≥ε. Individual token identity is lost; the Jacobian (20) reduces to (r)jrj1Tτγτ(lj) G(r)_jr_j 1T_τ _τ^(l_j), the gradient is governed entirely by the trajectory-level controlled base weight wτbasew_τ base. Figure 3 visualizes the joint local–global regime map on the probability simplex. It also shows that many parts of L-I/I/I regime cascade fits inside G-I. This is the outcome of encouraged parameter choice: That local token-level regulation engages before global trajectory-level regulation, allowing the fiber gate to preserve the most meaningful(non-outlier) per-token gradient signal for as long as possible, while the base gate remains transparent until the trajectory as a whole has drifted substantially where regulation on global quantities becomes necessary. Figure 3: Regime map on the probability simplex given by all possible configurations of a trajectory’s policy ratios (subject to probabilistic constraint (1Ttrajectory∑t=0Ttrajectory−1rt=1 1T_ trajectory _t=0^T_ trajectory-1r_t=1) where T=3T=3, ε=0.025 =0.025, C+=0.15C^+=0.15, C−=0.09C^-=0.09, K=3K=3, C∗±=(1+1/K)C±C^*±=(1+1/K)\,C^±). (a) Local branch (zoomed ±0.18± 0.18 around centroid): L-I (no clip, central polytope inside teal boundary), L-I (partial clip, between teal solid and purple solid boundaries), and L-I (full clip, outside purple boundary). Red and blue dashed contours show the rollback thresholds N=C−N=C^- and P=C+P=C^+. (b) Global branch (full simplex): five concentric regimes—G-I (inside red solid N=C−N=C^-), G-I,r (rollback transition, between red solid N=C−N=C^- and red dotted N=C∗−N=C^*-), G-I (between red dotted N=C∗−N=C^*- and blue solid P=C+P=C^+), G-I,r (rollback transition, between blue solid P=C+P=C^+ and blue dotted P=C∗+P=C^*+), and G-I (outside blue dotted P=C∗+P=C^*+). 5.3 Per-Token and Total Objective Substituting the gated ratio decomposition (7) into the FiberPO objective (6), the per-token objective contribution is: 1|Tjθold|1Tτwτbase⋅r~ifiber⋅A^i, 1|Tj _ old|\, 1T_τ\;w base_τ· r_i fiber· A_i, (17) and the total objective is J^FiberPO(θ|θold)=∑(s,a,τ,t)∈¯1|Tjθold|1Tτwτbase⋅r~ifiber⋅A^s,a. J FiberPO(θ| _ old)= _(s,a,τ,t)∈ X 1|Tj _ old|\, 1T_τ\;w base_τ· r_i fiber· A_s,a. (18) In terms of the fiber residual uiu_i, opposite-sign aggregate viv_i, and aggregate ratios logsτ± s^±_τ, this can be expressed as: J^FiberPO=∑i∈¯1|Tjθold|1Tτexp∘gagg(logsτ+,C+,Tτ)exp∘gagg(logsτ−,C−,Tτ)⋅logclip(eliui,ϵ)logclip(elivi,ϵ)⋅A^i. J FiberPO= _i∈ X 1|Tj _ old|\, 1T_τ\; \,g agg( s^+_τ,C^+,T_τ) \,g agg( s^-_τ,C^-,T_τ)· logclip(e^l_iu_i,ε)logclip(e^l_iv_i,ε)· A_i. Figure 4: Per-token and total FiberPO objective under parameterized drift ri=1+tΔ^ir_i=1+t _i (T=10T=10, k=32k=32, ϵ=0.04ε=0.04, C+=0.12C^+=0.12, C−=0.05C^-=0.05), with sign(Δ^i)=sign(A^i)sign( _i)=sign( A_i), reflecting the typical policy update in which tokens with positive advantage increase their probability (Δ^i>0 _i>0) and tokens with negative advantage decrease their probability (Δ^i<0 _i<0). GRPO in the graph uses the same ϵε-clip as FiberPO. (a) Trajectory in (logs+,logs−)( s^+, s^-)-space overlaid on the logw w heatmap; as drift increases, the trajectory moves through the global regimes of Figure 2(b). (b) Per-token fiber correction at t=1t=1: fiber residual (red) and base-weight contribution (orange) stack to form the full gated log-ratio. (c) Per-token objective for two representative tokens. (d) Total objective: FiberPO’s gaggg agg and logcliplogclip saturate, causing the objective to return to baseline. Vertical markers: ϵε-clip transitions (gray, L-I→ -I), C−C^- rollback (red, G-I,r onset), C+C^+ rollback (blue, G-I,r onset). Figure 4 traces the per-token and total objective under varying policy drift. The key features are: • Local regulation engages first (gray vertical lines, panels c, d): individual tokens hit the ϵε-clip boundary (|ui|=ϵ|u_i|=ε, L-I→ -I transition) well before the trajectory-level aggregates reach the budget threshold. At this stage wbasew base is in G-I, so only per-token spikes are capped while the trajectory-level importance-sampling correction remains intact. • Rollback activates on aggregate drift (between solid and dotted red/blue vertical lines): once logsτ− s^-_τ exceeds C−C^- (or logsτ+ s^+_τ exceeds C+C^+), gaggg agg enters rollback regime. The base weight wτbasew base_τ begins to oppose the drift direction, producing a restorative gradient (G-I,r). • Non-monotonic pattern (panel d): the total objective rises, peaks as |logwτbase|| w base_τ| reaches its maximum (G-I,r), then declines through mutual rollback (G-I,r) and collapses to the no-reweighting baseline when both channels are fully gated (G-I, wτbase=1w base_τ=1). 5.4 Theoretical Analysis Having defined the FiberPO objective and its two-scale gating structure, we now analyze the gradient-level properties that distinguish it from existing proximal methods. Gradient Decomposition via the Ratio Gating Jacobian. The gradient of a generic RGF objective (Definition 3.1), and thus of FiberPO, with respect to the model parameters θ admits the decomposition: ∂J^(θ|θold)∂θ=∑k∈ℰ∂zk∂θ∂J^∂zk=∑i,j,k∈ℰ∂zk∂θμi⏟parametrization-relatedprior weights∂(r∙)i∂rj⏟()ij(rj(δjk−(πθ)k))⏟jk=∂rj∂zk⏟ratio gating factorA^ψ(i)⏟advantage force, ∂ J(θ| _ old)∂θ= _k∈ E ∂ z_k∂θ ∂ J∂ z_k= _i,j,k∈ E ∂ z_k∂θ\, _i_ subarraycparametrization-related\\ prior weights subarray (r_ )_i∂ r_j_ J(G)_ij\; (r_j( _jk-( _θ)_k) )_ _jk\,=\, ∂ r_j∂ z_k_ratio gating factor A_ψ(i)_advantage force, (19) where ziz_i is the logit parameter of the policy, i.e., πθ(⋅|s)=softmax((θ,s)) _θ(·|s)=softmax(z(θ,s)) (see Appendix A.4), δjk _jk is the Kronecker delta, (πθ)k≡πθ(ak|sk)( _θ)_k≡ _θ(a_k|s_k) is the policy probability at the state-action pair indexed by k∈ℰk , and ψ(i)ψ(i) maps an augmented index i∈ℰi to its underlying state-action pair (s,a)(s,a). The gradient decomposes into three factors: the parametrization-related prior weights ∂zk∂θμi ∂ z_k∂θ _i, which are fixed a priori; the advantage force A^ψ(i) A_ψ(i), which comes from the reward model and is also given; and the ratio gating factor, which is the only component determined by the choice of surrogate objective. Within the ratio gating factor, the matrix ≡[rj(δjk−(πθ)k)]j,k∈ℰ ≡ [r_j( _jk-( _θ)_k) ]_j,k∈ E is simply the transformation matrix from logit space to ratio space [∂rj∂zk][ ∂ r_j∂ z_k]. Therefore, the Jacobian of the ratio gating map, ()ij≡∂(r∙)i∂rj J(G)_ij≡ (r_ )_i∂ r_j, is the central object that determines how different surrogate objectives shape the policy update. The FiberPO Ratio Gating Jacobian. To characterize how this two-scale gating shapes gradient flow, we compute the Jacobian of the ratio transform G. Since i=exp(logi)G_i= ( _i), the Jacobian ()ij:=∂(r∙)i∂rj J(G)_ij:= (r_ )_i∂ r_j factors as irj⋅∂logi∂logrj G_ir_j· ∂ _i∂ r_j. Using the gating decomposition (7) and differentiating through gaggg agg, logcliplogclip, and the aggregate ratios sτ±s_τ^±, one obtains: Proposition 5.2 (Jacobian of FiberPO ratio transform). The Jacobian of the FiberPO ratio gating map is: ()ij=(r∙)irjtji=tjj[|ui|≤ϵi=j⏟(a) local self-gating+1Ttji(γtji(lj)−|ui|≤ϵli=lj−|vi|≤ϵ(1−li=lj))⏟(b) trajectory-mediated coupling], J(G)_ij= G(r_ )_ir_j\;I_tj_i=tj_j [\; I_|u_i|≤ε\;I_i=j_(a) local self-gating\;+\; 1T_tj_i ( _tj_i^(l_j)-I_|u_i|≤ε\;I_l_i=l_j-I_|v_i|≤ε\,(1-I_l_i=l_j) )_(b) trajectory-mediated coupling\; ], (20) where: • tjitj_i represents the trajectory which state-action pair in i belongs to, • ui:=lilogri−logstji(li)u_i:=l_i r_i- s_tj_i^(l_i) is the fiber residual of token i (Eq. 14), • vi:=−logstji(−li)v_i:=- s_tj_i^(-l_i) is the opposite-sign aggregate (Eq. 15), • γτ(l):=(gagg)′(logsτ(l),C(l),Tτ) _τ^(l):=(g agg) ( s_τ^(l),\;C^(l),\;T_τ) is the derivative of the base gate (Eq. 10), taking the value 11 in the pass-through regime (|logsτ(l)|≤C(l)| s_τ^(l)|≤ C^(l)), −Tτ-T_τ in the rollback regime (C(l)<|logsτ(l)|<(1+Tτ−1)C(l)C^(l)<| s_τ^(l)|<(1+T_τ^-1)C^(l)), and 0 in the zeroed regime (|logsτ(l)|≥(1+Tτ−1)C(l)| s_τ^(l)|≥(1+T_τ^-1)C^(l)). Three structural properties follow from this expression. Property 5.3 (Trajectory independence). The factor tji=tjjI_tj_i=tj_j ensures that tokens in different trajectories are fully decoupled: ()ij=0 J(G)_ij=0 whenever tji≠tjjtj_i _j. Equivalently, the Jacobian is block-diagonal over trajectories, so each trajectory’s gradient is self-contained.444It is still worth mention that there might still be cross-trajectory coupling via the parametrization-related prior weights ∂zk∂θ ∂ z_k∂θ in the full gradient decomposition (Eq. 19), however, (i) a canonical choice of parametrization should have different parameters controlling for unrelated trajectories, thus typically leading to sparse ∂zk∂θ ∂ z_k∂θ and thus minimal unrelated cross-trajectory coupling. In contrast, the related trajectory-level coupling of parametrization is in fact desirable, as it allows more stable and robust gradient direction provided by correlated gradients of related trajectories. (i) The parametrization-related prior weights ∂zk∂θ ∂ z_k∂θ are independent of the choice of surrogate objective and thus is not possible to further control with surrogate formulations. The block-diagonal Jacobian already provides maximal decoupling that can be achieved at the surrogate level. Property 5.4 (First-order agreement). At the on-policy point (r∙=r_ =1), the Jacobian reduces to ()ij=i=j/rj J(G)_ij=I_i=j/r_j, the identity gating. The FiberPO surrogate therefore recovers the linear surrogate J(1)(θ|θold)J^(1)(θ| _ old) and the true RL objective to first order near on-policy (Theorem 4.5). Property 5.5 (Scale separation). The local self-gating term (a) in Eq. 20 has O(1)O(1) magnitude. The coupling term (b) is weighted by 1/Tτ1/T_τ, so trajectory-mediated effects scale inversely with sequence length. In the near-on-policy regime, the local gradient dominates; trajectory-level corrections become significant only as aggregate drift grows. 6 Fibration Hierarchy and FiberPO-Domain The fiber bundle model of Section 4.1 captures a single level of abstraction: tokens are grouped into trajectories, and Fiber Bundle Gating decomposes stability control into a base gate (trajectory level) and a fiber gate (token level). In practice, however, dependency and classification in LLM RL training are inherently hierarchical. Tokens belong to trajectories, trajectories belong to prompt groups, and prompt groups belong to domains such as mathematics, code generation, and instruction following. There is no reason to restrict the fibration to a single level, and the algebraic structure of FBG extends naturally to a chain of fibrations. This generalization is not merely a matter of symbolic generality. Different levels of the hierarchy demand different trust-region radii for substantive reasons. First, different domains may be trained at different stages and therefore have different proximity to the optimum, requiring appropriately sized per-domain trust regions (a radius that is too large causes instability, while one that is too small leads to premature convergence to a local optimum). Second, parameters associated with each hierarchical level carry different degrees of importance in the network architecture and should accordingly be given different amounts of flexibility. Third, even the positive and negative drift channels within each level demand different budgets, to avoid overly optimistic updates from accidental rewards or excessive punishment of exploratory behavior. Existing methods provide only coarse control over these demands: GRPO applies a uniform per-token clip, GSPO operates at a single trajectory level, and APC-Obj and TRPO enforce a global aggregate trust region that does not discriminate across finer structures. FiberPO-Trajectory (Section 5) improves on these by coupling trajectory-level and token-level control, but it operates at only two levels of the hierarchy. In this section, we show that the fibration formalism generalizes naturally to arbitrarily many hierarchical levels, providing a principled compositional framework for multi-scale trust-region control. We formalize this generalization as the Uniform Fibration Hierarchy and its associated Fibration Gating Hierarchy, and demonstrate its utility by deriving FiberPO-Domain, a concrete four-level instantiation (domain, prompt group, trajectory, token). The same construction extends readily to any level of hierarchical complexity that a given training setup requires. 6.1 Uniform Fibration Hierarchy and Fibration Gating Hierarchy We begin by extending the fiber bundle model to a chain of fibrations, each representing one level of abstraction. Definition 6.1 (Uniform Fibration Hierarchy). A uniform fibration hierarchy of depth n is a chain of spaces Bkk∈0,…,n\B_k\_k∈\0,…,n\ (called strata) together with fibrations πkk∈1,…,n\ _k\_k∈\1,…,n\, where each πk:Bk→Bk−1 _k:B_k→ B_k-1 is a fibration projection: B0⟵π1B1⟵π2B2⟵π3⋯⟵πnBn≡E.B_0 _1 B_1 _2 B_2 _3 ·s _n B_n≡ E. (21) The chain of fibrations naturally induces a chain of pushforward maps on density spaces: 0⟵π1∗1⟵π2∗2⟵π3∗⋯⟵πn∗n≡,B_0 _1* B_1 _2* B_2 _3* ·s _n* B_n , (22) where kB_k denotes the space of densities on BkB_k. Building on the uniform fibration hierarchy, we equip each stratum with gating infrastructure analogous to Fiber Bundle Gating (Definition 4.3). Definition 6.2 (Fibration Gating Hierarchy (FGH)). A Fibration Gating Hierarchy consists of the following objects: Object 1. Uniform fibration hierarchy Bk,πk\B_k, _k\ (Definition 6.1). Object 2. Reflecting Markov kernels Kkk∈0,…,n−1\K_k\_k∈\0,…,n-1\, Kk:k→k+1K_k:B_k _k+1: 0⟶K01⟶K12⟶K2⋯⟶Kn−1n≡.B_0 K_0 B_1 K_1 B_2 K_2 ·s K_n-1 B_n . Object 3. Atomic gating functions on each stratum, gk,p<kk∈0,…,n\g_k,p_<k\_k∈\0,…,n\: gk,p<k:k→k,g_k,p_<k:B_k _k, where gk,p<kg_k,p_<k may depend on the densities on all lower strata p<k≡(p0,p1,…,pk−1)p_<k≡(p_0,p_1,…,p_k-1). Object 4. Fibration decomposition and recovery maps, the same as in FBG (Definition 4.3): ℱ:ℝℰ→F:R^E , ℛ:→ℝℰR:E ^E. These objects are subject to the following conditions: (i) Reflecting condition: πk∗∘Kk−1=idk−1 _k* K_k-1=id_B_k-1 for 0<k≤n0<k≤ n, equivalently πk∗(σ−Kk−1(πk∗σ))=0 _k*(σ-K_k-1( _k*σ))=0, ensuring that the residual at each level carries no lower-stratum information. (i) Recovery condition: ℛ∘ℱ=idℝℰR =id_R^E. (i) Locality condition: gk,p<k=⨁b∈Bkgk,p<k,bg_k,p_<k= _b∈ B_kg_k,p_<k,b, i.e., gk,p<k(σk)b=gk,p<k,b((σk)b)g_k,p_<k( _k)_b=g_k,p_<k,b(( _k)_b) for all σk∈k _k _k and b∈Bkb∈ B_k. Each atomic gating function acts pointwise over the elements of its stratum. For notational convenience, we define sequential compositions of the fibration and kernel maps. Let Km←k K_m← k :=Km−1∘Km−2∘⋯∘Kk+1∘Kkfor k<m,Kk←k:=idk, :=K_m-1 K_m-2 ·s K_k+1 K_k k<m, K_k← k:=id_B_k, (23) πk←m _k← m :=πk+1∘πk+2∘⋯∘πm−1∘πmfor k<m,πk←k:=idBk, := _k+1 _k+2 ·s _m-1 _m k<m, _k← k:=id_B_k, (24) πk←m∗ _k← m\,* :=πk+1∗∘πk+2∗∘⋯∘πm−1∗∘πm∗for k<m,πk←k∗:=idk. := _k+1\,* _k+2\,* ·s _m-1\,* _m* k<m, _k← k\,*:=id_B_k. (25) We also write π<k∗σ≡(π0←n∗σ,π1←n∗σ,…,πk−1←n∗σ) _<k\,*σ≡( _0← n\,*σ,\; _1← n\,*σ,\;…,\; _k-1← n\,*σ) for σ∈≡nσ _n, collecting the pushed-forward densities on all strata below level k. The FGH gating function generalizes the FBG operator (4) by gating the residual at each stratum and reflecting all contributions back to the total space: G(σ):=Kn←0∘g0∘π0←n∗(σ)+∑0<k≤nKn←k∘gk,π<k∗σ(πk←n∗σ−Kk−1(πk−1←n∗σ)),G(σ)\;:=\;K_n← 0 g_0 _0← n\,*(σ)\;+\; _0<k≤ nK_n← k g_k, _<k\,*σ\! ( _k← n\,*σ-K_k-1\! ( _k-1← n\,*σ ) ), (26) which admits the compact form G(σ)=∑0≤k≤nKn←k∘Gk∘πk←n∗(σ),G(σ)\;=\; _0≤ k≤ nK_n← k G_k _k← n\,*(σ), (27) where the per-stratum operator GkG_k is defined by Gk(σk):=g0(σk)if k=0,gk,π<k∗σ(σk−Kk−1∘πk∗(σk))otherwise.G_k( _k):= casesg_0( _k)&if k=0,\\[3.00003pt] g_k, _<k\,*σ\! ( _k-K_k-1 _k*( _k) )&otherwise. cases (28) The structure mirrors that of Fiber Bundle Gating: the density σ is pushed forward to each stratum kB_k via πk←n∗ _k← n\,*, the contribution already accounted for at stratum k−1k-1 is subtracted through σk−Kk−1∘πk∗(σk) _k-K_k-1 _k*( _k), the residual is gated by gk,π<k∗σg_k, _<k\,*σ, and all gated components are reflected back to E via Kn←kK_n← k and summed. The associated FGH surrogate objective is defined analogously to the FBG gating form (5): J^FGH(θ|θold)=∑i∈ℰμiℛ∘G∘ℱ(r∙(θ|θold))iA^i. J FGH(θ| _ old)= _i _i\;R G (r_ (θ| _ old))_i\; A_i. (29) Remark 6.3 (Recovery of Fiber Bundle Gating). When n=1n=1, the fibration hierarchy reduces to a single fiber bundle π1:B1≡E→B0≡B _1:B_1≡ E→ B_0≡ B, and the FGH definitions recover exactly the FBG framework of Definition 4.3. In this case, the gating function (26) reduces to G(σ)=K0(g0(π1∗σ))+g1,π1∗σ(σ−K0(π1∗σ))G(σ)=K_0(g_0( _1*σ))+g_1, _1*σ(σ-K_0( _1*σ)), which is precisely (4). The FiberPO objective (Definition 5.1), which operates at the trajectory and token levels, corresponds to n=1n=1. 6.2 FiberPO-Domain We now instantiate the Fibration Gating Hierarchy with n=3n=3 to obtain FiberPO-Domain, which extends FiberPO’s two-level gating (trajectory, token) to a four-level hierarchy: domain, prompt group, trajectory, and token. We first introduce the relevant notation. Let D∈DomainD denote a domain index (e.g., mathematics, code, instruction following), and let DgD_g denote the domain of prompt group g. For simplicity, we assume that training prompts are not composite domain tasks, so that we may also write DτD_τ for the domain of trajectory τ and Di:=DτD_i:=D_τ for the domain of an augmented state-action pair i≡(s,a,τ,t)∈¯i≡(s,a,τ,t)∈ X. The prompt group index of trajectory τ is denoted gτg_τ, and gig_i denotes the prompt group of augmented pair i. Let τi _i denote the trajectory to which i belongs, TτT_τ the length of trajectory τ, TD:=∑τ∈DTτT_D:= _τ∈ DT_τ the total number of state-action pairs classified to domain D, and Tg:=∑τ∈gTτT_g:= _τ∈ gT_τ the total number of state-action pairs within prompt group g. We write Tjθold(D)Tj _ old(D) for the set of trajectories in domain D and Tjθold(g)Tj _ old(g) for those in prompt group g. Definition 6.4 (FiberPO-Domain). The FiberPO-Domain objective is: J^FiberPO-Domain(θ|θold)=∑i∈¯1|Tjθold|1TτwiBase⋅r~iFiber⋅A^i. J^FiberPO-Domain(θ| _ old)= _i∈ X 1|Tj _ old|\, 1T_τ\;w_i Base· r_i Fiber· A_i. (30) The base weight decomposes into six gated aggregate terms, three per sign channel, corresponding to the domain, prompt group, and trajectory levels: wiBase:=exp(gagg(logsDi+,C+,TDi)+gagg(logsgi+−logsDi+,C+,Tgi)+gagg(logsτi+−logsgi+,C+,Tτi)−gagg(logsDi−,C−,TDi)−gagg(logsgi−logsDi−,C−,Tgi)−gagg(logsτi−logsgi−,C−,Tτi)),w_i Base:= \! ( array[]rl&g agg( s^+_D_i,\,C^+,\,T_D_i)\;+\;g agg( s^+_g_i- s^+_D_i,\,C^+,\,T_g_i)\;+\;g agg( s^+_ _i- s^+_g_i,\,C^+,\,T_ _i)\\[5.0pt] -&g agg( s^-_D_i,\,C^-,\,T_D_i)\;-\;g agg( s^-_g_i- s^-_D_i,\,C^-,\,T_g_i)\;-\;g agg( s^-_ _i- s^-_g_i,\,C^-,\,T_ _i) array ), (31) where the aggregate gate gaggg agg is defined in Eq. 10. The gated residual is identical to FiberPO’s fiber gate (Eq. 11): r~iFiber:=logclip((sτi(li))−liri,ϵ)logclip((sτi(−li))−li,ϵ), r_i Fiber:= logclip\! ((s_ _i^(l_i))^-l_i\,r_i,\;ε )logclip\! ((s_ _i^(-l_i))^-l_i,\;ε ), (32) where li:=sign(logri)l_i:=sign( r_i). The prompt-group-level and domain-level aggregate log-ratios appearing in the base weight are defined as: logsg(l):=1|Tjθold(g)|∑τ∈glogsτ(l),logsD(l):=∑g∈D|Tjθold(g)||Tjθold(D)|logsg(l). s_g^(l):= 1|Tj _ old(g)| _τ∈ g s_τ^(l), s_D^(l):= _g∈ D |Tj _ old(g)||Tj _ old(D)| s_g^(l). (33) The associated FGH objects (strata, reflecting kernels, atomic gates, and decomposition/recovery maps) that realize this objective as an n=3n=3 FGH instance are specified in Appendix E.7. The base weight (31) has a transparent hierarchical structure. Each of the three positive terms gates the residual drift at one level of the hierarchy: logsDi+ s^+_D_i captures domain-level drift, logsgi+−logsDi+ s^+_g_i- s^+_D_i captures the prompt group’s deviation from its domain aggregate, and logsτi+−logsgi+ s^+_ _i- s^+_g_i captures the trajectory’s deviation from its prompt group aggregate. The negative channel is treated symmetrically. At each level, gaggg agg independently maintains a trust-region budget with the same three-regime (pass-through, rollback, zeroed) behavior as in FiberPO-Trajectory (Section 5.1), scaled by the appropriate count (TDiT_D_i, TgiT_g_i, or TτiT_ _i). FiberPO-Domain inherits the structural merits that FiberPO-Trajectory demonstrates at the trajectory and token levels. The first-order agreement with the true RL objective near on-policy extends to the full hierarchy under the same identity conditions on the atomic gates (Theorem 4.5 applied to the FGH framework). The ratio decoupling property (Section 5.2), which ensures that tokens are regulated by their deviation from the trajectory mean rather than by their absolute log-ratio, now extends upward: each trajectory is regulated by its deviation from the prompt group aggregate, and each prompt group is regulated by its deviation from the domain aggregate. The rollback mechanism operates independently at each level, so that budget violation at one level of the hierarchy does not contaminate the gating at other levels. This information decoupling between hierarchical levels provides substantially more versatile optimization control for multi-domain LLM RL training than any single-scale or two-scale method. The FiberPO-Domain objective (30) admits an equivalent FGH gating form (r∙)i=wiBase⋅r~iFiber=ℛ∘G∘ℱ(r∙)iG(r_ )_i=w_i Base· r_i Fiber=R G (r_ )_i corresponding to the n=3n=3 case of Definition 6.2. The detailed specification of the constituent FGH objects (fibration hierarchy, reflecting Markov kernels, atomic gating functions, and decomposition/recovery maps) is given in Appendix E.7. Remark 6.5 (Heterogeneous fibration systems). The uniform fibration hierarchy assumes a linear chain of strata B0←B1←⋯←BnB_0← B_1←·s← B_n, which suffices for the domain–prompt-group–trajectory–token hierarchy considered here. A more general construction replaces the linear chain with a rooted tree (or more generally, a quiver), where each edge carries a fibration and each vertex carries a gating function. Given a rooted tree with source and target maps s,t:Edge→Vertexs,t:Edge (edges directed toward the root), one assigns a space FvF_v to each vertex and a fibration F1(e):Fs(e)→Ft(e)F_1(e):F_s(e)→ F_t(e) to each edge, together with reflecting kernels and atomic gates at each vertex. The resulting gating operator aggregates gated residuals from all vertices back to the total leaf space, generalizing the linear sum (26) to a tree-indexed sum. This heterogeneous fibration system accommodates branching hierarchies (e.g., separate domain and modality classifications at the same level) and provides a framework for expressing gating algebras over arbitrary dependency structures. 7 Conclusion This paper develops an algebraic approach to multi-scale stability control in reinforcement learning for large language models. We begin by proving that classical TRPO trust-region guarantees collapse at discount factor γ=1γ=1, the regime required by sparse-reward LLM tasks, motivating a separation of the trust-region maintenance mechanism from the specific radius prescribed by the classical bound. We then derive Aggregational Policy Censoring Objective (APC-Obj), an unconstrained clipping-based surrogate that is provably equivalent to sample-based TV-TRPO under complete parametrization, establishing that clipping-based surrogate design and trust-region policy optimization are dual formulations of the same problem. Through the Ratio Gating Formalism (RGF), we show that PPO, GRPO, and GSPO each arise from APC-Obj via identifiable relaxation steps, making each method’s departure from the trust-region optimum explicit. Building on this analytical foundation, we introduce Fiber Bundle Gating (FBG), a framework that organizes sampled RLHF data as a fiber bundle and decomposes ratio gating into coordinated global and local components operating on densities over the base and total spaces. FBG preserves first-order agreement with the true RL objective near on-policy whenever the atomic gates reduce to identity at the reference point. From a relaxed APC-Obj formulation, we derive FiberPO, a concrete FBG instantiation whose trajectory-level aggregate gate and token-level residual gate provide independent trust-region budgets at two scales. The FiberPO Jacobian is block-diagonal over trajectories, reduces to identity at on-policy, and exhibits a restorative gradient structure in the rollback regime that is absent in all prior methods. To our knowledge, this is the first work to provide a principled algebraic framework that unifies trust-region maintenance, multi-scale gating, and gradient-level guarantees within a single coherent formalism. A natural question is whether the fiber bundle apparatus is necessary, or whether the same multi-scale gating could be achieved with a carefully hand-crafted structured clipping rule. We observe that methods coupling global and local stability control necessarily involves a dependency structure in which many local quantities depend on a shared global context. This dependency defines a fibration πE:E→B _E:E→ B, and the three objects of a fiber bundle, the total space E, the base space B, and the projection πE _E, correspond directly to the components already present in the data. Methods that aggregate local quantities into context-level statistics implicitly invoke the pushforward πE∗ _E*, and methods that distribute context-level information back to local entries implicitly construct a Markov kernel K. The formalism makes these operations explicit and thereby exposes the reflecting condition πE∗∘K=id _E* K=id_B (for methods coupling global and local information to satisfy), a structural constraint ensuring that the residual σ−K(πE∗σ)σ-K( _E*σ) carries no base-level information (Appendix E.3). Without this condition, the fiber gate re-gates global information already handled by the base gate, or the base gate inadvertently constrains genuinely local variation. Designing a gating map that avoids this double-counting while preserving first-order agreement (Theorem 4.5) amounts to satisfying exactly the reflecting condition, i.e., implicitly re-deriving the fiber bundle decomposition. The formalism identifies and makes explicit an underlying decomposition structure that multi-scale gating has, and provides a principled framework for reasoning about its correctness. Because fibrations compose algebraically, the same three-object template chains into the Fibration Gating Hierarchy (FGH) without introducing new primitives, with each stratum’s residual capturing the purest local variation at that level after all coarser influence has been removed. FiberPO-Domain (domain, prompt group, trajectory, token) is one such instantiation, and deeper hierarchies follow by appending another fibration layer. Achieving the same compositional guarantee with ad-hoc notation would require re-proving orthogonality and first-order agreement at every new level. The algebraic compositionality of fibrations provides this automatically. This positions FBG as a foundation for the stability challenges that arise in large-scale agentic systems and heterogeneous multi-domain training. Appendix A Notation and Preliminaries A.1 On MDP For a given MDP, we denote its action space A, state space S, state transition probability (s′|s,a)T(s |s,a), its θ-parameter dependent policy πθ(a|s) _θ(a|s), and reward function R:→ℝR:S . We denote the space of all signed measures over a space X (equipped with a sigma algebra on it) as (X)M(X). The space of probability measures is denoted ℙ(X)P(X), and the space of smooth probability measures is denoted C∞ℙ(X)C^∞P(X) when X is equipped with a smooth structure. We denote p0,πθ,P_p_0, _θ,T (or simply θP_θ if T and p0p_0 are clear or assumed from context) as the probabilistic distribution of RL trajectories given the prior distribution p0p_0 for initial states, the πθ _θ action policy, and the state transition distribution T. The discounted RL objective is denoted J(θ):=τ∼θ[∑t=0∞γtR(st(τ))].J(θ):=E_τ _θ [ _t=0^∞γ^tR(s_t(τ)) ]. Given a reference policy parameter θold _ old, we define: Policy-likelihood ratio. The policy-likelihood ratio (or policy ratio) is defined to be rs,a(θ|θold):=πθ(a|s)πθold(a|s),a∈,s∈.r_s,a(θ| _ old):= _θ(a|s) _ _ old(a|s), a ,~s . If θ,θoldθ, _ old are clear from context, we abbreviate this as rs,ar_s,a. Linear surrogate RL objective. The linear surrogate RL objective (with respect to θold _ old) is denoted J(1)(θ|θold):=J(θold)+τ∼θold[∑t=0∞γtrst(τ),at(τ)(θ|θold)Ast(τ),at(τ)(θold)],J^(1)(θ| _ old):=J( _ old)+E_τ _ _ old [ _t=0^∞γ^tr_s_t(τ),a_t(τ)(θ| _ old)\;A^( _ old)_s_t(τ),a_t(τ) ], where As,a(θold)A^( _ old)_s,a is the advantage value for the state-action pair (s,a)(s,a) given the reference policy πθold _ _ old. The linear surrogate RL objective is the first-order approximation to the true RL objective near on-policy (θ≈θoldθ≈ _ old), and thus has the same gradient as the true RL objective at θ=θoldθ= _ old [1]: ∇θJ(1)(θ|θold)|θ=θold=∇θJ(θ)|θ=θold. _θJ^(1)(θ| _ old) |_θ= _ old= _θJ(θ) |_θ= _ old. Advantage norm. We further denote max(s,a)∈×|As,a(θ)|≡‖A∙(θ)‖∞ _(s,a) ×A|A^(θ)_s,a|≡\|A^(θ)_ \|_∞, the L∞L^∞ norm of the advantage A∙(θ)A^(θ)_ (for placeholder ∙ running over all state-action pairs), a notation agreeing with our later convention. Discounted state distribution. We denote the discounted state distribution ρ(γ)πθρ _θ_(γ), defined as ρ(γ)πθ(U):=τ∼θ[∑t=0∞γtU(st(τ))],ρ _θ_(γ)(U_S):=E_τ _θ [ _t=0^∞γ^tI_U_S(\,s_t(τ)\,) ], where U⊆U_S is a measurable set in state space S, and X(x):=1if x∈X0otherwiseI_X(x):= \ array[]l1&if x∈ X\\ 0&otherwise array . is the indicator function. Note that for LLM γ=1γ=1, this infinite sum does not necessarily converge. However, typical LLMs have finite response length (or at least clip overly long responses), so the discounted state distribution can still be defined. Normalized state distribution. We define the normalized state distribution ρ¯πθ:=(1−γ)ρ(γ)πθif γ<1limγ→1−(1−γ)ρ(γ)πθif γ=1 ρ _θ:= \ array[]l (1-γ)ρ _θ_(γ)&if γ<1\\[10.00002pt] _γ→ 1^-(1-γ)ρ _θ_(γ)&if γ=1 array . and the trajectory-length-normalized state distribution ρ¯(tj)πθ(U):=τ∼θ[∑t=0Tτ−11TτU(st(τ))], ρ _θ_ (tj)(U_ S):=E_τ _θ [ _t=0^T_τ-1 1T_τI_U_S(\,s_t(τ)\,) ], where TτT_τ is the length of trajectory τ. A.2 On Estimation for MDP In the context of estimation, we denote ^X∼p[X] E_X p[X] the Monte Carlo estimation of X∼p[X]E_X p[X]. For trajectory distributions θP_θ, we denote TjθTj^θ the collection of all sampled trajectories, Tjθ(s0)Tj^θ(s_0) the set of all sampled trajectories from initial state s0s_0, and X the collection of all state-action pairs that appeared in sampled trajectories, i.e., :=(st(τ),at(τ))∣τ∈Tjθ,X:=\(s_t(τ),a_t(τ)) τ ^θ\, in this given context of estimation. For later convenience, we also denote s0[Tjθ]s_0[Tj^θ] the set of all initial states s0s_0 that appeared in the sampled trajectories. We denote the augmented (sampled) state-action pair space ¯:=(st(τ),at(τ),τ,t)⊆×Tjπθold×ℕ X:=\(s_t(τ),a_t(τ),τ,t)\ ×Tj _ _ old×N (for mathematical completeness, even if a trajectory terminates at a finite step, we still include the null state-action pair (∅,∅,τ,t)( , ,τ,t) after termination). For any state-action pair (s,a)(s,a), we define ns,a∈ℕn_s,a as the number of times (s,a)(s,a) appears in all the sampled trajectories TjθTj^θ (counting multiplicity inside the same trajectory as well). We further denote the normalized counting measure on space X as μ _X: μ((s,a))≡μs,a=ns,a∑(s′,a′)∈ns′,a′. _X(\(s,a)\)≡ _s,a= n_s,a _(s ,a ) n_s ,a . We also introduce, for a trajectory τ, the quantity ns,a(τ)n^(τ)_s,a representing the number of times state-action pair (s,a)(s,a) appears in trajectory τ. (For LLM, it never goes beyond 1. However, for rigorousness and generality to RL, we still employ this notation; furthermore, we always assign n∅,∅=0n_ , =0 for the null state-action pair.) The normalized counting measure μ(τ)μ^(τ)_X is associated to it: μ(τ)((s,a))≡μs,a(τ)=ns,a(τ)∑(s′,a′)∈(τ)ns′,a′(τ), _X^(τ)(\(s,a)\)≡ _s,a^(τ)= n^(τ)_s,a _(s ,a ) ^(τ)n^(τ)_s ,a , where (τ)X^(τ) is the set of all state-action pairs that appear in trajectory τ. We also denote μ¯(τ):=1Tτ∈τ _ X^(τ):= 1T_τI_∈τ the normalized counting measure on trajectory τ over the augmented state-action space ¯ X, where ∈τ((s,a,τ′,t))=1if τ=τ′0otherwise,I_∈τ((s,a,τ ,t))= \ array[]l1&if τ=τ \\ 0&otherwise array ., and we conveniently denote μ¯s,a,t(τ)≡μ¯(s,a,τ,t)≡μ¯(τ)((s,a,τ,t)) μ^(τ)_s,a,t≡ μ_(s,a,τ,t)≡μ^(τ)_ X(\(s,a,τ,t)\). For convenience, we also define the normalized counting measure on natural numbers ℕN (and subsets thereof) with respect to a trajectory as μ(τ)∈ℙ(ℕ)μ^(τ) (N), μ(τ)(n):=1Tτ[0,Tτ)(n)μ^(τ)(\n\):= 1T_τI_[0,T_τ)(n). A.3 On Divergences and Their Monte-Carlo Estimation We define the total variation (TV) distance and Kullback–Leibler (KL) divergence at three levels—per-state, average/max over states, and trajectory-normalized—and give the distributional definition, the equivalent ratio form, and the Monte-Carlo estimator for each. All divergences are measured from πθold _ _ old to πθ _θ using the policy ratio rs,a:=πθ(a|s)/πθold(a|s)r_s,a:= _θ(a|s)/ _ _ old(a|s); we abbreviate D(θold∥θ)D( _ old\|θ) when the policies are clear from context. A.3.1 Per-State Divergences Definitions. For a fixed state s∈s , the per-state TV distance and KL divergence between πθold(⋅|s) _ _ old(·|s) and πθ(⋅|s) _θ(·|s) are DTV(s):=DTV(πθold(⋅|s)∥πθ(⋅|s))=12∑a∈|πθold(a|s)−πθ(a|s)|,D_ TV(s)\;:=\;D_ TV\! ( _ _ old(·|s)\, \|\, _θ(·|s) )\;=\; 12 _a | _ _ old(a|s)- _θ(a|s) |, (34) DKL(s):=DKL(πθold(⋅|s)∥πθ(⋅|s))=∑a∈πθold(a|s)logπθold(a|s)πθ(a|s).D_ KL(s)\;:=\;D_ KL\! ( _ _ old(·|s)\, \|\, _θ(·|s) )\;=\; _a _ _ old(a|s) _ _ old(a|s) _θ(a|s). (35) They are related by Pinsker’s inequality: DTV(s)2≤12DKL(s),D_ TV(s)^2\;≤\; 12\,D_ KL(s), (36) which is used throughout this paper to convert between KL-based and TV-based bounds. Ratio forms. Using rs,a=πθ(a|s)/πθold(a|s)r_s,a= _θ(a|s)/ _ _ old(a|s), the per-state divergences can be rewritten as expectations under πθold(⋅|s) _ _ old(·|s): DTV(s)=12a∼πθold(⋅|s)[|rs,a−1|],D_ TV(s)\;=\; 12\,E_a _ _ old(·|s)\! [\,|r_s,a-1|\, ], (37) DKL(s)=−a∼πθold(⋅|s)[logrs,a].D_ KL(s)\;=\;-\,E_a _ _ old(·|s)\! [ r_s,a ]. (38) The TV ratio form (37) follows from factoring out πθold(a|s) _ _ old(a|s) in (34): DTV(s) D_ TV(s) =12∑a∈|πθold(a|s)−πθ(a|s)|=12∑a∈πθold(a|s)|πθold(a|s)−πθ(a|s)πθold(a|s)| = 12 _a | _ _ old(a|s)- _θ(a|s) |= 12 _a _ _ old(a|s) | _ _ old(a|s)- _θ(a|s) _ _ old(a|s) | =12∑a∈πθold(a|s)|1−rs,a|=12a∼πθold(⋅|s)[|rs,a−1|]. = 12 _a _ _ old(a|s)\, |1-r_s,a |= 12\,E_a _ _ old(·|s)\! [\,|r_s,a-1|\, ]. The KL ratio form (38) follows from log(πθold/πθ)=−logrs,a ( _ _ old/ _θ)=- r_s,a in (35). Monte-Carlo estimation. Let s:=a:(s,a)∈X_s:=\a:(s,a) \ be the set of actions sampled at state s, with multiplicities ns,an_s,a per-state sample count Ts:=∑a∈sns,aT_s:= _a _sn_s,a, total sample count T=∑sTsT= _sT_s is the total sample count, and the augmented state-action space ¯ X (Appendix A.2). The Monte-Carlo estimators of the per-state divergences are: D^TV(s):=1Ts∑a∈sns,a|rs,a−1|, D_ TV(s)\;:=\; 1T_s _a _sn_s,a\,|r_s,a-1|, (39) D^KL(s):=−1Ts∑a∈sns,alogrs,a, D_ KL(s)\;:=\;- 1T_s _a _sn_s,a\, r_s,a, (40) which are the empirical means of the ratio-form integrands (37)–(38) over the samples at state s. A.3.2 Average and Max Divergences Average divergence. The average TV and KL divergences weight the per-state quantities by the normalized state distribution ρ¯πθold ρ _ _ old (Appendix A.1): D¯TV/KL(θold∥θ):=s∼ρ¯πθold[DTV/KL(s)]. D_ TV/KL( _ old\|θ)\;:=\;E_s ρ _ _ old\! [D_ TV/KL(s) ]. (41) Estimation of the average divergence. The augmented state-action space ¯ X (Appendix A.2) pools tokens across all trajectories, so that the empirical state frequencies in ¯ X are proportional to ρ¯πθold ρ _ _ old, i.e., TsT∼ρ¯πθold(s) T_sT ρ _ _ old(s) as T→∞T→∞. The average divergence is therefore estimated by weighting the per-state estimators (39)–(40) by their sample counts, which can be expanded in two equivalent ways—over the states []S[X] with multiplicities, or directly over the augmented space ¯ X: D¯^TV/KL(θold∥θ)=1T∑s∈[]TsD^TV/KL(s)=1T∑s∈[]∑a∈sns,af(rs,a)=1T∑(s,a,τ,t)∈¯f(rs,a), D_ TV/KL( _ old\|θ)\;=\; 1T _s [X]T_s\; D_ TV/KL(s)\;=\; 1T _s [X] _a _sn_s,a\,f(r_s,a)\;=\; 1T _(s,a,τ,t)∈ Xf(r_s,a), (42) where f(rs,a)=|rs,a−1|f(r_s,a)=|r_s,a-1| for TV, and respectively f(rs,a)=−logrs,af(r_s,a)=- r_s,a for KL, and []S[X] is the set of states appearing in X. The second equality substitutes the definition of D^TV/KL(s) D_ TV/KL(s) (Eqs. 39–40); the third unfolds the state-action multiplicities into the augmented index (s,a,τ,t)(s,a,τ,t), since each occurrence of (s,a)(s,a) in ¯ X contributes exactly one term. Note that the estimation here is for the undiscounted case. We do not use discounted averages of TV or KL(or their estimation) elsewhere in this paper; for the generic form of discounted estimators, see Appendix D.1. Max TV divergence. The max TV divergence takes the worst-case per-state TV over the entire state space: DTVmax(θold∥θ):=maxs∈DTV(s).D _ TV( _ old\|θ)\;:=\; _s D_ TV(s). (43) This quantity appears in the TRPO surrogate-gap bound (Eq. 1). Its sample-based estimator restricts the maximum to states observed in the batch: D^TVmax(θold∥θ):=maxs∈[]D^TV(s), D _ TV( _ old\|θ)\;:=\; _s [X] D_ TV(s), (44) which is used in the sample-based TV-TRPO formulation (Appendix D.2). A.3.3 Trajectory-Normalized Divergences Definition. The trajectory-normalized TV and KL divergences average the per-state quantities over the trajectory-level normalized state distribution ρ¯(tj)πθold ρ _ _ old_ (tj), which weights each state by its frequency within individual trajectories (Appendix A.1): D¯TV/KL(tj)(θold∥θ):=s∼ρ¯(tj)πθold[DTV/KL(s)]. D_ TV/KL (tj)( _ old\|θ)\;:=\;E_s ρ _ _ old_ (tj)\! [D_ TV/KL(s) ]. (45) The key distinction from D¯TV/KL D_ TV/KL (41) is the state distribution: ρ¯πθold ρ _ _ old weights states by their overall frequency across the sample, whereas ρ¯(tj)πθold ρ _ _ old_ (tj) gives each trajectory equal weight and distributes that weight uniformly among the trajectory’s states. Ratio forms. Substituting the per-state ratio forms (37)–(38) into (45) and using the law of total expectation yields: D¯TV(tj)(θold∥θ)=12s∼ρ¯(tj)πθold,a∼πθold(⋅|s)[|rs,a−1|],D¯KL(tj)(θold∥θ)=−s∼ρ¯(tj)πθold,a∼πθold(⋅|s)[logrs,a]. D_ TV (tj)( _ old\|θ)= 12\,E_ subarraycs ρ _ _ old_ (tj),\\ a _ _ old(·|s) subarray\! [|r_s,a-1| ], D_ KL (tj)( _ old\|θ)=-\,E_ subarraycs ρ _ _ old_ (tj),\\ a _ _ old(·|s) subarray\! [ r_s,a ]. (46) Estimation. We describe three related estimations for D¯TV/KL(tj) D (tj)_ TV/KL, and D¯TV/KL D_ TV/KL. (i) Single-trajectory estimation via ^(τ) E^(τ). For a fixed trajectory τ∼θoldτ _ old, the sequence of states st(τ)t=0Tτ−1\s_t(τ)\_t=0^T_τ-1 constitutes samples from ρ¯(tj)πθold ρ _ _ old_ (tj), with actions drawn from πθold(⋅|st(τ)) _ _ old(·|s_t(τ)). We define the single-trajectory Monte-Carlo estimator ^(τ) E^(τ) as the empirical average over the state-action pairs visited by trajectory τ: ^(τ)[f(s,a)]:=1Tτ∑t=0Tτ−1f(rst(τ),at(τ)), E^(τ)[f(s,a)]\;:=\; 1T_τ _t=0^T_τ-1f\! (r_s_t(τ),~a_t(τ) ), (47) where all samples come from the single trajectory τ. Applying this to the per-state divergences: D¯^TV(τ)(θold∥θ):=^(τ)[DTV(s)]=1Tτ∑t=0Tτ−1|rst(τ),at(τ)−1|, D^(τ)_ TV( _ old\|θ)\;:=\; E^(τ)\! [D_ TV(s) ]\;=\; 1T_τ _t=0^T_τ-1|r_s_t(τ),a_t(τ)-1|, (48) D¯^KL(τ)(θold∥θ):=^(τ)[DKL(s)]=−1Tτ∑t=0Tτ−1logrst(τ),at(τ). D^(τ)_ KL( _ old\|θ)\;:=\; E^(τ)\! [D_ KL(s) ]\;=\;- 1T_τ _t=0^T_τ-1 r_s_t(τ),a_t(τ). (49) Averaging the single-trajectory estimators over trajectories recovers the population quantity: D¯TV/KL(tj)(θold∥θ)=τ∼θold[D¯^TV/KL(τ)(θold∥θ)]=(s,a)∼ρ¯(tj)πθold⋅πθold[f(rs,a)]. D_ TV/KL (tj)( _ old\|θ)\;=\;E_τ _ old\! [ D^(τ)_ TV/KL( _ old\|θ) ]=E_(s,a) ρ _ _ old_ (tj)· _ _ old\! [f(r_s,a) ]. (50) It is worth mentioning that the single-trajectory estimator D¯^TV/KL(τ) D^(τ)_ TV/KL is not a single-sample estimator of the trajectory-normalized divergence D¯TV/KL(tj) D_ TV/KL (tj). We have mentioned the sequence of states(resp. state-action pairs) from a single trajectory τ are many samples from the trajectory-length-normalized state distribution ρ¯(tj)πθold ρ _ _ old_ (tj) (resp. ρ¯(tj)πθold⋅πθold ρ _ _ old_ (tj)· _ _ old), so the single-trajectory estimator D¯^TV/KL(τ) D^(τ)_ TV/KL is an empirical average over those samples, and thus is a more accurate many sample(TτT_τ many samples) Monte-Carlo estimator of D¯TV/KL(tj) D_ TV/KL (tj) according to the second equality. (i) Estimation over the augmented state-action space. Equivalently, D¯TV/KL(tj) D (tj)_ TV/KL can be estimated by averaging over the full augmented state-action space ¯ X(which contains not just one, but many trajectories), but weighting each sample by the inverse length of the trajectory it belongs to: D¯^TV/KL(tj)(θold∥θ)=1|Tjθold|∑(s,a,τ,t)∈¯1Tτf(rs,a), D (tj)_ TV/KL( _ old\|θ)\;=\; 1|Tj _ old| _(s,a,τ,t)∈ X 1T_τ\,f(r_s,a), (51) where f(rs,a)=|rs,a−1|f(r_s,a)=|r_s,a-1| for TV (resp. −logrs,a- r_s,a for KL), and |Tjθold||Tj _ old| is the number of sampled trajectories. The factor 1/Tτ1/T_τ ensures that each trajectory contributes equally regardless of its length, matching the trajectory-length-normalized state distribution ρ¯(tj)πθold ρ _ _ old_ (tj). (i) Recovering the average divergence from trajectory estimates. Given a set of sampled trajectories TjθoldTj _ old, weighting each single-trajectory estimate by its trajectory length recovers the (non-trajectory-normalized) average divergence (41): D¯^TV/KL(θold∥θ)=1T∑τ∈TjθoldTτD¯^TV/KL(τ)(θold∥θ)=1T∑(s,a,τ,t)∈¯f(rs,a), D_ TV/KL( _ old\|θ)\;=\; 1T _τ _ oldT_τ\; D^(τ)_ TV/KL( _ old\|θ)\;=\; 1T _(s,a,τ,t)∈ Xf(r_s,a), (52) where T=∑τTτT= _τT_τ is the total sample count. This follows because the trajectory-level estimator D¯^TV/KL(τ) D^(τ)_ TV/KL averages uniformly over TτT_τ tokens within trajectory τ, so multiplying by Tτ/T_τ/T reweights each token equally across all trajectories, which is exactly the empirical average over ¯ X and thus estimates D¯TV/KL D_ TV/KL under the normalized state distribution ρ¯πθold ρ _ _ old. A.4 On Ratio Space For sets X,YX,Y, we denote YXY^X the collection of all functions from X to Y. Definition A.1 (Policy-ratio space). Given the new policy πθ _θ and old policy πθold _ _ old, we can sample a set of state-action pairs X. The space ℝ≥0≡(ℝ≥0)R_≥ 0^X≡(R_≥ 0)^X assigns a non-negative real number to every (s,a)∈(s,a) . We define ℝ>0R^X_>0 as the policy ratio space of all possible policy ratios over the sampled state-action space X. We typically denote its elements (the tuples of policy ratios) as r∙(θ|θold)≡r∙r_ (θ| _ old)≡ r_ (for placeholder ∙ running over all X). A tuple consists of components: rst,at(θ|θold)≡rst,at:=πθ(at|st)πθold(at|st).r_s_t,a_t(θ| _ old)≡r_s_t,a_t:= _θ(a_t|s_t) _ _ old(a_t|s_t). Remark A.2. To incorporate more information, we sometimes use the space of state-action pairs with extra information instead of X. For example, in our fiber bundle and FiberPO, we use ¯ X and ℝ>0¯R X_>0. In general RGF forms, we use ℰE and ℝ>0ℰR E_>0. Subscript conventions. Similarly, for elements in Y^X, we use the bullet subscript (−)∙(-)_ to denote its placeholder index, which runs over X, and (−)s,a∈Y(-)_s,a∈ Y is the element evaluated at state-action pair (s,a)(s,a). We denote |q∙|p|q_ |_p the LpL^p norm of quantity q∙∈ℂq_ ^X in the measure space (,μ)(X, _X). Parity superscript convention. In FiberPO, the sign assignment li:=sign(logri)∈+1,−1l_i:=sign( r_i)∈\+1,-1\ partitions tokens into positive and negative channels. When l (or lil_i, ls,al_s,a) appears as a superscript in parentheses, it selects the corresponding sign channel: sτ(l)s_τ^(l) denotes sτ+s_τ^+ when l=+1l=+1 and sτ−s_τ^- when l=−1l=-1, and similarly C(l)C^(l), γτ(l) _τ^(l), etc. The negated form sτ(−l)s_τ^(-l) denotes the opposite channel. Logit parametrization. We also introduce the quantity z∙∈ℝz_ ^X or z∙,∙∈ℝ×z_ , S× A, called the logit parameter, the output of the second-to-last layer of the neural network before the softmax activation. That is, πθ(a|s)=exp(zs,a(θ))∑a′:(s,a′)∈exp(zs,a′(θ)). _θ(a|s)= (z_s,a(θ)) _a :\,(s,a ) (z_s,a (θ)). This is a natural parametrization of probability measures, with one redundant dimension given by adding a constant to all logits (which maps to the same probability measure). Appendix B TRPO: Background, Proof, and Discussion B.1 TV-Based TRPO (TV-TRPO) Consider the discounted RL objective J(θ):=τ∼θ[∑t=0∞γtRt(τ)],J(θ)\;:=\;E_τ _θ\! [\, _t=0^∞γ^t\,R_t(τ) ], (53) and its linear surrogate with respect to a reference policy πθold _ _ old: J(1)(θ|θold):=J(θold)+τ∼θold[∑t=0∞γtrst(τ),at(τ)(θ|θold)Ast(τ),at(τ)(θold)],J^(1)(θ\,|\, _ old)\;:=\;J( _ old)+E_τ _ _ old\! [\, _t=0^∞γ^t\,r_s_t(τ),a_t(τ)(θ\,|\, _ old)\;A^( _ old)_s_t(τ),a_t(τ) ], (54) where rs,a(θ|θold):=πθ(a|s)/πθold(a|s)r_s,a(θ\,|\, _ old):= _θ(a\,|\,s)\,/\, _ _ old(a\,|\,s) is the importance-sampling ratio and As,a(θold)A^( _ old)_s,a is the advantage under πθold _ _ old. The linear surrogate is the first-order approximation of the true objective near on-policy (θ≈θoldθ≈ _ old); in particular, ∇θJ(1)(θ|θold)|θ=θold=∇θJ(θ)|θ=θold _θJ^(1)(θ\,|\, _ old) |_θ= _ old= _θJ(θ) |_θ= _ old. Trust region policy optimization (TRPO) [4] bounds the surrogate gap: J(θ)−J(1)(θ|θold)≥−4γ‖A∙(θold)‖∞(1−γ)2DTVmax(θ∥θold)2,J(θ)-J^(1)(θ\,|\, _ old)\;≥\;-\, 4γ\,\|A^( _ old)_ \|_∞(1-γ)^2\;D_ TV (θ\,\|\, _ old)^2, (55) where DTVmax(θ∥θold):=maxsDTV(πθold(⋅|s)∥πθ(⋅|s))D_ TV (θ\,\|\, _ old):= _sD_ TV\! ( _ _ old(·\,|\,s)\,\|\, _θ(·\,|\,s) ). Note that, unlike the original TRPO formulation of [4] which uses a KL-divergence penalty, here we work with the total-variation (TV) form throughout, since TV distance is the divergence that arises naturally from the performance-difference bound above. TRPO’s monotonic improvement guarantee. Based on this bound, TRPO constructs a minorization–maximization (M) algorithm: at each step, (55) provides a minorizer of the true objective J(θ)J(θ), and the update maximizes this minorizer, i.e. θnew _ new is obtained by maximizing the penalized surrogate J(1)(θ|θold)−4γ‖A∙(θold)‖∞(1−γ)2DTVmax(θ∥θold)2.J^(1)(θ\,|\, _ old)\;-\; 4γ\,\|A^( _ old)_ \|_∞(1-γ)^2\;D_ TV (θ\,\|\, _ old)^2. (56) Since the true objective is bounded below by (56) and the optimum of (56) is at least J(θold)J( _ old) (attained at θ=θoldθ= _ old), the true objective is guaranteed to be non-decreasing at every update, making the policy optimization stable. We refer to this M algorithm as TV-based TRPO or TV-TRPO. Sample-based TV-TRPO. In practice, DTVmaxD_ TV is intractable to compute over large state spaces, so it is replaced by the sample maximum max to estimate DTVmaxD_ TV , this will be termed as sample-based TV-TRPO objective that serves as the theoretical foundation for the remainder of this paper. B.2 Proof of the TRPO Vanishing Theorem We first introduce the necessary definitions, then state and prove the theorem. Definition B.1 (Trust Region). For δ≥0δ≥ 0, define the TV trust region as ℬδTV-TR(θold):=π∈(→ℙ())|DTVmax(πθold∥π)≤δ,B TV-TR_δ( _ old)\;:=\; \\,π∈(S (A))\; |\;D_ TV ( _ _ old\,\|\,π)≤δ\, \, and the KL trust region as ℬδKL-TR(θold):=π∈(→ℙ())|DKLmax(πθold∥π)≤δ,B KL-TR_δ( _ old)\;:=\; \\,π∈(S (A))\; |\;D_ KL ( _ _ old\,\|\,π)≤δ\, \, where ℙ()P(A) denotes the space of probability measures over the action space. The surrogate-gap bound (55) implies that TRPO updates are confined to a trust region whose radius depends on γ: Lemma B.2 (TV-TRPO update lies in the trust region). Define the TRPO trust-region radius δ(TRPO):=1−γ8γδ^( TRPO):= 1-γ8γ (see also (76)). For any θnew _ new produced by sample-based TV-TRPO, πθnew∈ℬδ(TRPO)TV-TR(θold) _ _ new TV-TR_δ^( TRPO)( _ old) (This is a restatement of Lemma D.12). Theorem B.3 (TRPO vanishing theorem; restatement of Theorem 2.1). When γ=1γ=1, both the TV-based and KL-based TRPO trust regions collapse to the reference policy: ℬδ(TRPO)TV-TR(θold)=ℬδ(TRPO)KL-TR(θold)=πθoldB TV-TR_δ^( TRPO)( _ old)\;=\;B KL-TR_δ^( TRPO)( _ old)\;=\;\\, _ _ old\,\ Proof. By Lemma B.2, taking γ→1−γ→ 1^- in δ(TRPO)=1−γ8γδ^( TRPO)= 1-γ8γ yields δ(TRPO)→0+δ^( TRPO)→ 0^+; at γ=1γ=1 we have δ(TRPO)=0δ^( TRPO)=0. Any policy πθ _θ inside the trust region must therefore satisfy DTVmax(πθold∥π)=0D_ TV ( _ _ old\,\|\,π)=0 (and likewise for KL). This implies πθold(⋅|s)=πθ(⋅|s) _ _ old(·\,|\,s)= _θ(·\,|\,s) for every s, i.e., πθ _θ is identical to πθold _ _ old. ∎ B.3 On γ-Annealing as a Workaround One might attempt to circumvent the vanishing theorem by introducing a variable discount factor γ<1γ<1 into LLM RL, applying TRPO during training, and gradually annealing γ→1γ→ 1. Similar discount-scheduling strategies have been explored in other RL settings [2]. However, this approach faces a fundamental practical obstacle in LLM RL. Language model reward functions are typically sparse and discontinuous: they yield a meaningful signal only at the end of a response, and often in an all-or-nothing fashion—the response is either fully correct or receives no reward. Some approaches incorporate a partial, continuous reward signal via a KL divergence penalty term [3], but this remains the exception rather than the rule in RLVR settings. Introducing a variable discount factor γ<1γ<1 into such reward landscapes would exponentially suppress later tokens in the training objective J(θ)=[∑tγtRt]J(θ)=E[ _tγ^tR_t], systematically downweighting the very signals (end-of-response correctness) that RL is meant to optimize. Moreover, the interaction between a changing γ and the already-discontinuous reward surface can further destabilize reward estimation, as the effective reward scale shifts across training. These difficulties make a γ-annealing strategy impractical for LLM RL, reinforcing the need for a direct approach to trust-region-style stabilization in the γ=1γ=1 case. Appendix C RGF and APC-Obj: Derivations C.1 APC-Obj in RGF form Definition C.1 (APC-Obj policy iteration). The APC-Obj update selects θnew=argmaxθJ^APC-Obj(θ|θold) _ new= _θ\, J APC-Obj(θ| _ old), where J^APC-Obj(θ|θold)=1T∑(s,a,τ,t)∈¯[clip(rs,a−1,Tsδ(APC-Obj)−∑(s,a′,τ′,t′)∈¯(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|)A^s,aθold+A^s,aθold], J APC-Obj(θ| _ old)\;=\; 1T _(s,a,τ,t)∈ X [clip\! (r_s,a-1,\;T_sδ^( APC-Obj)-\!\!\!\!\!\! _ subarrayc(s,a ,τ ,t )∈ X\\ (a ,τ ,t )≠(a,τ,t) subarray\!\!\!\!\!\!|r_s,a -1| )\, A _ old_s,a+ A _ old_s,a ], (57) where δ(APC-Obj):=(1−γ)28γ‖A^∙θold‖∞M(A^∙θold)Tδ^( APC-Obj):= (1-γ)^28γ\,\| A _ old_ \|_∞\, M( A _ old_ )T (the argument A^∙θold A _ old_ makes the dependence on the advantage explicit), the per-entry clip bound is Bs,a,τ,t:=Tsδ(APC-Obj)−∑(s,a′,τ′,t′)∈¯s,(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|B_s,a,τ,t:=T_sδ^( APC-Obj)- _ subarrayc(s,a ,τ ,t )∈ X_s,\,(a ,τ ,t )≠(a,τ,t) subarray|r_s,a -1|, and clip(a,B):=clip(a,−B+,B+)clip(a,B):=clip(a,-B^+,B^+) with B+:=max(B,0)B^+:= (B,0). We point out that the APC-Obj objective in Definition C.1 removes the (1−γ)−1(1-γ)^-1 normalization constant in D.4. As the only factor (Sitting aside δ(APC-Obj)δ^( APC-Obj)’s dependence on γ, as we eventually will relax it to a constant hyperparameter. Even if δ(APC−Obj)=0δ (APC-Obj)=0, it will not make APC-Obj objective diverge) in the discounted APC-Obj objective that prevents convergence as γ→1γ→ 1 is the multiplier 11−γ 11-γ. This factor is a constant that does not depend on the policy ratios rs,ar_s,a, thus removing it does not change the argmax over θ and the APC-Obj policy update, for any value of γ. This removal corresponds to estimating the discounted objective with respect to the normalized discounted state distribution ρ¯(γ)θold ρ_(γ) _ old than the unnormalized one ρ(γ)θold=11−γρ¯(γ)θold _(γ) _ old= 11-γ ρ_(γ) _ old. With this convention, the APC-Obj objective converges and is well-defined at γ=1γ=1. 555We also mention that the APC-Obj objective varies smoothly with γ, as the per-state discount weight Γ(s)Γ (s) (Γ(s)=∑(s′,a,τ,t)∈¯,s′=sγt (s)= _(s ,a,τ,t)∈ X,\,s =sγ^t and Γ=∑sΓ(s) = _s (s)) in APC-Obj algorithm varies smoothly with γ. When γ=1γ=1, γt=1γ^t=1 for the summand, so Γ(s)=Ts (s)=T_s and Γ=T =T, giving Γ(s)Γ=TsT (s) = T_sT for every state. Adding 1Ts 1T_s weight for each token given by the APC-Obj in D.4, one recovering the normalized counting measure over all sampled tokens, the form displayed in Definition C.1 here. Thus removing of the (1−γ)−1(1-γ)^-1 factor smoothly and analytically extends the APC-Obj objective to γ=1γ=1 without any discontinuity or singularity, resulting the exact form in Definition C.1. The APC-Obj objective can equivalently be written in the RGF form of Definition 3.1: Definition C.2 (APC-Obj surrogate objective, in RGF form). J^APC-Obj(θ|θold)=1T∑(s,a,ℐ)∈ℰ[clip(rs,a−1,(Tsδ(APC-Obj)−∑(s,a′,ℐ′)∈ℰ(a′,ℐ′)≠(a,ℐ),|rs,a′−1|))A^s,aθold+A^s,aθold], J APC-Obj(θ| _ old)\;=\; 1T _(s,a,I) [clip\! (r_s,a-1,\;\, (T_sδ^( APC-Obj)-\!\!\!\!\!\! _ subarrayc(s,a ,I ) \\ (a ,I )≠(a,I), subarray\!\!\!\!\!\!|r_s,a -1| ) )\; A _ old_s,a\;+\; A _ old_s,a ], (58) where ℐ=(τ,t)I=(τ,t) carries the trajectory membership and time step associated with state–action pair (s,a)(s,a) during sampling, ℰ⊆×Tjθold×ℕE ×Tj _ old×N is the augmented index set in which each sampled token appears as a separate entry, and μs,a,ℐ=1T _s,a,I= 1T. Reading off the ratio gating map from (58): Definition C.3 (APC-Obj ratio gating map). The APC-Obj ratio gating map APC-Obj:ℝ>0ℰ→ℝ>0ℰG APC-Obj:R_>0^E _>0^E is APC-Obj(r∙)s,a,ℐ=clip(rs,a−1,(Tsδ(APC-Obj)−∑(s,a′,ℐ′)∈ℰ(a′,ℐ′)≠(a,ℐ)|rs,a′−1|))+ 1.G APC-Obj(r_ )_s,a,I\;=\;clip\! (r_s,a-1,\; (T_sδ^( APC-Obj)-\!\!\!\!\!\! _ subarrayc(s,a ,I ) \\ (a ,I )≠(a,I) subarray\!\!\!\!\!\!|r_s,a -1| ) )\;+\;1. (59) Unlike PPO and GRPO, the APC-Obj gating map depends on the entire ratio tuple r∙r_ through the cross-action coupling in the clip bound, which is what enforces the per-state total-variation constraint in aggregate. C.2 APC-Obj Budget Consumption and Clipping Regimes The APC-Obj gating map (Definition C.3) contains a policy-ratio-dependent budget inside the clipping term that enforces the per-state trust-region constraint. To understand how this budget shapes the gated ratios, we analyze the per-token clipping behavior by decomposing it into three regimes according to budget consumption. Budget and consumption terms. Consider a single per-token objective term from (58): (clip(rs,a−1,Tsδ(APC-Obj)⏟(B): budget−∑(s,a′,ℐ′)∈ℰ(a′,ℐ′)≠(a,ℐ)|rs,a′−1|)+1)A^s,aθold. (clip\! (r_s,a-1,\; T_sδ^( APC-Obj)_(B): budget-\!\!\!\!\!\! _ subarrayc(s,a ,I ) \\ (a ,I )≠(a,I) subarray\!\!\!\!\!\!|r_s,a -1| )+1 )\; A _ old_s,a. (60) We identify three components that govern the clipping behavior: • (B) The total budget: Tsδ(APC-Obj)T_s\,δ^( APC-Obj). • (C1) The cross-token consumption: ∑(s,a′,ℐ′)∈ℰ,(a′,ℐ′)≠(a,ℐ)|rs,a′−1|=TsD^TV(s)−|rs,a−1| _ subarrayc(s,a ,I ) ,\;(a ,I )≠(a,I) subarray|r_s,a -1|\;=\;T_s\, D_ TV(s)-|r_s,a-1|, the total budget already consumed by all other tokens (except a) at state s. • (C2) The current-token deviation: |rs,a−1||r_s,a-1|, the amount consumed by the token under consideration. The sum (C1)+(C2)=TsD^TV(s)(C1)+(C2)=T_s\, D_ TV(s) is the total per-state TV consumption. The effective clip bound for the current token is (B)−(C1)(B)-(C1), and the clipping activates when (C2) exceeds this residual budget. Three clipping regimes. The clipping behavior at each token (s,a,ℐ)(s,a,I) falls into exactly one of three regimes, determined by the relationship between the budget (B) and the consumption terms (C1), (C2): (P) Pass-through regime (clip inactive). The clip acts as the identity, so the gated ratio equals the ungated ratio. This occurs when |rs,a−1|≤Tsδ(APC-Obj)−(C1),|r_s,a-1|\;≤\;T_s\,δ^( APC-Obj)-(C1), (61) which, after substituting (C1)+(C2)=TsD^TV(s)(C1)+(C2)=T_s\, D_ TV(s), reduces to a uniform condition across tokens: D^TV(s)≤δ(APC-Obj). D_ TV(s)\;≤\;δ^( APC-Obj). In this regime, the aggregate consumption (C1) + (C2) does not exceed the total budget (B), and the per-token objective coincides with the linear surrogate. (R) Rollback regime (clip active, output nonzero). The clip is active and suppresses |rs,a−1||r_s,a-1| to a smaller value, but does not zero it out. This requires both D^TV(s)>δ(APC-Obj)and|rs,a−1|Ts>D^TV(s)−δ(APC-Obj). D_ TV(s)\;>\;δ^( APC-Obj) |r_s,a-1|T_s\;>\; D_ TV(s)-δ^( APC-Obj). (62) The first condition is the same uniform budget violation as in (P) (with reversed inequality). The second condition is token-specific: the cross-token consumption (C1) alone has not exhausted the entire budget, so the residual clip bound (B)−(C1)>0(B)-(C1)>0 provides a nonzero but reduced clip output. In terms of the clip operator: 0<|clip(rs,a−1,(B)−(C1))|<|rs,a−1|0<|clip(r_s,a-1,\,(B)-(C1))|<|r_s,a-1|. (Z) Zeroed regime (clip output is zero). The clip output is exactly zero, and the token no longer contributes to the surrogate objective. This occurs when D^TV(s)>δ(APC-Obj)and|rs,a−1|Ts≤D^TV(s)−δ(APC-Obj). D_ TV(s)\;>\;δ^( APC-Obj) |r_s,a-1|T_s\;≤\; D_ TV(s)-δ^( APC-Obj). (63) Here the cross-token consumption (C1) alone already exceeds the total budget (B), making the residual clip bound (B)−(C1)≤0(B)-(C1)≤ 0, which forces the clip output to zero and eliminates the token’s contribution entirely. Connection to the retraction property. By Lemma D.9, whenever the policy ratios enter either the rollback regime (R) or the zeroed regime (Z)—that is, whenever the per-state TV divergence exceeds the trust-region radius δ(APC-Obj)δ^( APC-Obj)—the APC-Obj clipping mechanism suppresses the gated ratios’ per-state TV divergence strictly below δ(APC-Obj)δ^( APC-Obj). This ensures that the argmax solution, i.e. the policy update, does not lie outside the trust region ℬδ(APC-Obj)TV-TRPOB TV-TRPO_δ^( APC-Obj). Remark C.4 (From per-state regimes to aggregate regimes in FiberPO). The three APC-Obj clipping regimes (P), (R), (Z) are defined at the per-state level with total budget Tsδ(APC-Obj)T_s\,δ^( APC-Obj). After trajectory-level aggregation (replacing the per-state count TsT_s with the per-trajectory length TτT_τ; see Appendix E.5.3), logarithmic approximation (rs,a−1→logrs,ar_s,a-1→ r_s,a; see Appendix E.5.2), and clipping decomposition into the fiber bundle gating form (Eq. 77), these three per-state regimes lift to the three regimes of the aggregate gating function gaggg agg (Eq. 10): pass-through, rollback, and zeroed, respectively. C.3 LLM-to-RL Notation Translation To express PPO, GRPO, and GSPO in RGF form, we first establish the correspondence between the LLM indexing convention (g,j,i)(g,j,i) and the RL notation (τ,t)(τ,t) used throughout this paper. Index correspondence. In LLM notation, g indexes the query (prompt), j indexes the response (completion) generated for that query, and i indexes the token position within the response. The RL counterparts are the initial state s0(τ)s_0(τ), the trajectory τ, and the time step t: g:=s0(τ),j:=τ,i:=t.g\,:=\,s_0(τ), j\,:=\,τ, i\,:=\,t. Consequently, any per-token quantity qst(τ),at(τ)q_s_t(τ),a_t(τ) in RL notation corresponds to qj,i(g)q_j,i^(g) in LLM notation. Aggregate ratios. The positive/negative sequence aggregate ratios sτ±s^±_τ in RL notation are denoted sj(g)±s^(g)±_j in LLM notation. C.4 PPO, GRPO, and GSPO in RGF Form We give the explicit RGF specifications (ℰ,μ,)(E,\,μ,\,G) for each method. Recall the general RGF form (Definition 3.1): J^(θ|θold)=∑(s,a,ℐ)∈ℰμs,a,ℐ(r∙)s,a,ℐA^s,a. J(θ| _ old)= _(s,a,I)∈ E _s,a,I\;G(r_ )_s,a,I\; A_s,a. PPO [5]. ℐ=∅I= (no extra information), ℰ≃E , μs,a=μ((s,a))=ns,aT _s,a= _X(\(s,a)\)= n_s,aT (normalized counting measure of state-action pair (s,a)(s,a), counting appearance multiplicity from all sampled trajectories. T is number of sampled state-action pairs(counting multiplicity) and ns,an_s,a is the multiplicity for state-action pair (s,a)(s,a). Definition is also given in Appendix A.2), and PPO(r∙)s,a=sign(A^s,a)⋅min(clip(rs,a, 1±ϵ)sign(A^s,a),rs,asign(A^s,a)).G PPO(r_ )_s,a\;=\;sign( A_s,a)· \! (clip(r_s,a,\,1±ε)\,sign( A_s,a),\;r_s,a\,sign( A_s,a) ). GRPO [6]. ℐ=τI=τ (trajectory membership), ℰ⊆×TjθoldE ×Tj _ old, μs,a,τ=1|Tjθold|⋅1Tτ _s,a,τ= 1|Tj _ old|· 1T_τ (where TτT_τ is the total number of steps within trajectory τ), and GRPO=PPOG GRPO=G PPO (the same token-wise clipping, but with trajectory-normalized weights). GSPO [9]. ℐ=τI=τ (trajectory membership), ℰ⊆×TjθoldE ×Tj _ old, μs,a,τ=1|Tjθold|⋅1Tτ _s,a,τ= 1|Tj _ old|· 1T_τ, and GSPO(r∙)s,a,τ=lτ⋅min(clip(sτ, 1±ϵ)lτ,sτlτ),G GSPO(r_ )_s,a,τ\;=\;l_τ· \! (clip(s_τ,\,1±ε)\,l_τ,\;s_τ\,l_τ ), where TτT_τ is the total number of steps within trajectory τ, lτ:=sign(∑t=0Tτ−1A^st(τ),at(τ)θold)l_τ:=sign ( _t=0^T_τ-1 A _ old_s_t(τ),a_t(τ) ), and sτ:=(∏t=0Tτ−1rst(τ),at(τ))1/Tτs_τ:= ( _t=0^T_τ-1r_s_t(τ),a_t(τ) )^1/T_τ is the geometric-mean aggregate ratio over trajectory τ. Note that GSPO gates all tokens in a trajectory by the same aggregate sτs_τ, moving the clipping from the token level to the trajectory level. Remark C.5. If we remove the minimize function from the clippings (i.e., let (r∙)s,a,ℐ=clip(rs,a,1±ϵ)G(r_ )_s,a,I=clip(r_s,a,1±ε) for PPO/GRPO, and (r∙)s,a,ℐ=clip(sτ,1±ϵ)G(r_ )_s,a,I=clip(s_τ,1±ε) for GSPO), then the gating functions contain no sign functions (signsign and lτl_τ). C.5 Detailed Derivation: From APC-Obj to PPO, GRPO, and GSPO We now trace the precise steps that transform the APC-Obj objective into each of PPO, GRPO, and GSPO, identifying the relaxation or approximation heuristic at each step. Throughout, rs,ar_s,a implicitly represents rs,a(θ|θold)r_s,a(θ| _ old). We present the derivations without the minimize function, as it is not derived from trust-region theory but from other considerations (e.g., preventing excessive reward for actions that happen to have high ratios). Recall the APC-Obj RGF form (58): J^APC-Obj(θ|θold)=1T∑(s,a,ℐ)∈ℰ[clip(rs,a−1,(Tsδ(APC-Obj)−∑(s,a′,ℐ′)∈ℰ(a′,ℐ′)≠(a,ℐ),|rs,a′−1|))A^s,aθold+A^s,aθold]. J APC-Obj(θ| _ old)\;=\; 1T _(s,a,I) [clip\! (r_s,a-1,\; (T_sδ^( APC-Obj)-\!\!\!\!\!\! _ subarrayc(s,a ,I ) \\ (a ,I )≠(a,I), subarray\!\!\!\!\!\!|r_s,a -1| ) )\; A _ old_s,a\;+\; A _ old_s,a ]. (64) C.5.1 APC-Obj to PPO The RGF form of PPO (without the minimize function) is: J^PPO(θ|θold)=∑(s,a)∈μs,aclip(rs,a, 1±ϵ(PPO))A^s,a. J PPO(θ| _ old)= _(s,a) _s,a\;clip(r_s,a,\,1±ε^(PPO))\; A_s,a. Step 1 (δ-relaxation). Replace the vanishing δ(APC-Obj)δ^( APC-Obj) (which is zero at γ=1γ=1) with a positive tunable hyperparameter δ(PPO)δ^( PPO); the exact trust-region guarantee is traded for a controllable approximation whose departure is quantified by δ(PPO)δ^( PPO) itself. The relaxed objective becomes: J^(θ|θold)=1T∑(s,a,τ,t)∈¯[clip(rs,a−1,±(Tsδ(PPO)−∑(s,a′,τ′,t′)∈¯s(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|))+1]A^s,aθold. J(θ| _ old)\;=\; 1T _(s,a,τ,t)∈ X [clip\! (r_s,a-1,\;±\, (T_sδ^( PPO)-\!\!\!\!\!\! _ subarrayc(s,a ,τ ,t )∈ X_s\\ (a ,τ ,t )≠(a,τ,t) subarray\!\!\!\!\!\!|r_s,a -1| ) )+1 ]\, A _ old_s,a. Step 2 (decoupling). Decouple the clipping of individual ratios from other ratios by dropping the cross-entry sum from the clip bound and replacing it with a per-token constant ϵ(PPO)ε^( PPO). This yields the PPO objective: J^PPO(θ|θold)=∑(s,a)∈μs,aclip(rs,a, 1±ϵ(PPO))A^s,aθold. J PPO(θ| _ old)\;=\; _(s,a) _s,a\;clip(r_s,a,\,1±ε^( PPO))\; A _ old_s,a. This step eliminates the gradient coupling between ratios of different state-action pairs, improving stability and robustness. C.5.2 APC-Obj to GRPO The RGF form of GRPO (without the minimize function) is: J^GRPO(θ|θold)=∑τ∈Tjθold∑t=0Tτ−11|Tjθold|⋅1Tτclip(rs,a, 1±ϵ(GRPO))A^s,a, J GRPO(θ| _ old)= _τ _ old _t=0^T_τ-1 1|Tj _ old|· 1T_τ\;clip(r_s,a,\,1±ε^(GRPO))\; A_s,a, where s≡st(τ),a≡at(τ)s≡ s_t(τ),\,a≡ a_t(τ). The derivation uses the same first two steps as PPO: Step 1 (δ-relaxation). δ(APC-Obj)↦δ(GRPO)δ^(APC-Obj) δ^(GRPO). Step 2 (decoupling). Replace the per-entry clip bound Bs,a,τ,t:=Tsδ(APC-Obj)−∑(s,a′,τ′,t′)∈¯s,(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|B_s,a,τ,t:=T_sδ^( APC-Obj)- _ subarrayc(s,a ,τ ,t )∈ X_s,\\ (a ,τ ,t )≠(a,τ,t) subarray|r_s,a -1| with a per-token constant ϵ(GRPO)ε^( GRPO). Step 3 (sequence length weighting). Change the sampling weight from the normalized counting measure μs,a _s,a to the trajectory-normalized measure μ(GRPO)((st(τ),at(τ),τ)):=1|Tjθold|⋅1Tτ _(GRPO)(\(s_t(τ),a_t(τ),τ)\):= 1|Tj _ old|· 1T_τ. This yields the final GRPO objective: J^GRPO(θ|θold)=∑τ∈Tjθold∑t=0Tτ−11|Tjθold|⋅1Tτclip(rs,a−1,±ϵ(GRPO))A^s,a+A^s,a. J GRPO(θ| _ old)= _τ _ old _t=0^T_τ-1 1|Tj _ old|· 1T_τ\;clip(r_s,a-1,\,±ε^(GRPO))\; A_s,a+ A_s,a. Compared with PPO, GRPO penalizes extremely long sequences/trajectories via the 1/Tτ1/T_τ normalization. C.5.3 APC-Obj to GSPO The RGF form of GSPO (without the minimize function) is: J^GSPO(θ|θold)=∑τ∈Tjθold∑t=0Tτ−11|Tjθold|⋅1Tτclip(sτ, 1±ϵ(GSPO))A^s,a, J GSPO(θ| _ old)= _τ _ old _t=0^T_τ-1 1|Tj _ old|· 1T_τ\;clip(s_τ,\,1±ε^(GSPO))\; A_s,a, where s≡st(τ)s≡ s_t(τ), a≡at(τ)a≡ a_t(τ), and sτ:=(∏t=0Tτ−1rst(τ),at(τ))1/Tτ=exp(1Tτ∑t=0Tτ−1logrs,a)s_τ:= (Π^T_τ-1_t=0r_s_t(τ),a_t(τ) )^1/T_τ= ( 1T_τ _t=0^T_τ-1 r_s,a ) is the geometric mean of the ratios. The derivation proceeds as follows: Step 1 (δ-relaxation). δ(APC-Obj)↦δ(GSPO)δ^(APC-Obj) δ^(GSPO). Step 2 (trajectory-level decoupling approximation). Decouple the clipping of individual ratios at the trajectory level while performing trajectory-level reweighting. The per-entry clip bound Bs,a,τ,t:=Tsδ(GSPO)−∑(s,a′,τ′,t′)∈¯s,(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|B_s,a,τ,t:=T_sδ^( GSPO)- _ subarrayc(s,a ,τ ,t )∈ X_s,\\ (a ,τ ,t )≠(a,τ,t) subarray|r_s,a -1| (which couples all entries at the same state) is replaced by a per-trajectory bound Tτδseq(GSPO)−∑t′≠t|rst′(τ),at′(τ)−1|T_τδ^( GSPO)_seq- _t ≠ t|r_s_t (τ),a_t (τ)-1|. This yields: J^(θ|θold)=∑τ∈Tjθold∑t=0Tτ−11|Tjθold|⋅1Tτ[clip(rs,a−1,±(Tτδseq(GSPO)−∑t′=0t′≠tTτ−1|rst′(τ),at′(τ)−1|))+1]A^s,a. J(θ| _ old)= _τ _ old _t=0^T_τ-1 1|Tj _ old|· 1T_τ\; [clip (r_s,a-1,\;± (T_τδ^(GSPO)_seq-\!\!\! _ subarrayct =0\\ t ≠ t subarray^T_τ-1\!\!\!|r_s_t (τ),a_t (τ)-1| ) )+1 ]\; A_s,a. It is also worth mentioning that in GSPO, the advantage estimation is chosen so that A^s,a A_s,a agrees across all tokens within the same trajectory, so we can denote A^τ A_τ as the common advantage for trajectory τ. If the advantage estimation is not chosen in this way simply denote A^τ A_τ as the average advantage across all tokens within trajectory τ. Step 3 (log-ratio approximation). Replace r−1r-1 by logr r, which provides better algebraic properties. Recall D¯^TV(τ):=1Tτ∑t′=0Tτ−1|rst(τ),at(τ)−1| D^(τ)_ TV:= 1T_τ _t =0^T_τ-1|r_s_t(τ),a_t(τ)-1|, the Monte Carlo estimation of the average total variation distance given a sampled trajectory τ, and −logsτ:=D¯^KL(τ):=1Tτ∑t′=0Tτ−1logrst(τ),at(τ)- s_τ:= D^(τ)_ KL:= 1T_τ _t =0^T_τ-1 r_s_t(τ),a_t(τ), the sequence aggregate policy ratio (recall from Claim 3.5 that logsτ s_τ is the Monte Carlo estimation for the negative average KL divergence). The objective becomes: J^(θ|θold)=∑τ∈Tjθold∑t=0Tτ−11|Tjθold|⋅1Tτ[clip(logrs,a,±(Tτδseq(GSPO)−TτD¯^TV(τ)+|logrs,a|))+1]A^s,a. J(θ| _ old)= _τ _ old _t=0^T_τ-1 1|Tj _ old|· 1T_τ\; [clip ( r_s,a,\;± (T_τδ^(GSPO)_seq-T_τ D^(τ)_ TV+| r_s,a| ) )+1 ]\; A_s,a. Inspecting the clipping term, one sees that it activates only when TτD¯^TV(τ)≥Tτδseq(GSPO)T_τ D^(τ)_ TV≥ T_τδ^(GSPO)_seq, regardless of logrs,a r_s,a. We therefore consider two cases: 1. TτD¯^TV(τ)≥Tτδseq(GSPO)T_τ D^(τ)_ TV≥ T_τδ^(GSPO)_seq: clipping applies to all individual log policy ratios within the trajectory. 2. TτD¯^TV(τ)<Tτδseq(GSPO)T_τ D^(τ)_ TV<T_τδ^(GSPO)_seq: all clipping functions reduce to the identity. In case (2), the objective simplifies to: J^Case(2)(θ|θold)=∑τ∈Tjθold1|Tjθold|⋅(logsτ+1)A^τ. J Case\,(2)(θ| _ old)= _τ _ old 1|Tj _ old|·( s_τ+1)\; A_τ. Step 4 (clip-bound approximation). The unclipped case (2) holds if and only if Tτδseq(GSPO)≥TτD¯^TV(τ)T_τδ^(GSPO)_seq≥ T_τ D^(τ)_ TV. By Pinsker’s inequality 12DKL≥DTV2 12D_ KL≥ D_ TV^2, we approximately have −12logsτ≡12D¯^KL(τ)≥˙(D¯^TV(τ))2- 12 s_τ≡ 12 D^(τ)_ KL\; ≥\;( D^(τ)_ TV)^2. Thus when −logsτ≤2(δseq(GSPO))2- s_τ≤ 2(δ (GSPO)_seq)^2, we obtain Tτδseq(GSPO)≥Tτ−12logsτ≥˙TτD¯^TV(τ)T_τδ (GSPO)_seq≥ T_τ - 12 s_τ\; ≥\;T_τ D^(τ)_ TV, a condition for case (2) to be likely satisfied. Furthermore, GSPO implicitly requires the gradient to vanish when case (1) holds. Using this, we can combine both cases using the bound we just obtained via a simple clipping function on the aggregated ratio: J^(θ|θold)=1|Tjθold|∑τ∈Tjθold(clip(logsτ,±2(δseq(GSPO))2)+1)A^τ. J(θ| _ old)= 1|Tj _ old| _τ _ old (clip( s_τ,\,± 2(δ (GSPO)_seq)^2)+1 )\; A_τ. Applying the log approximation logsτ≈sτ−1 s_τ≈ s_τ-1 once more: J^(θ|θold)=1|Tjθold|∑τ∈Tjθold(clip(sτ−1,±2(δseq(GSPO))2)+1)A^τ. J(θ| _ old)= 1|Tj _ old| _τ _ old (clip(s_τ-1,\,± 2(δ (GSPO)_seq)^2)+1 )\; A_τ. Replacing the constant 2(δseq(GSPO))22(δ (GSPO)_seq)^2 with ϵ(GSPO)ε (GSPO), we arrive at the GSPO objective (without the minimize function): J^GSPO(θ|θold) J GSPO(θ| _ old) =1|Tjθold|∑τ∈Tjθoldclip(sτ, 1±ϵ(GSPO))A^τ = 1|Tj _ old| _τ _ oldclip(s_τ,\,1±ε (GSPO))\; A_τ =J^GSPO(θ|θold)=∑τ∈Tjθold∑t=0Tτ−1(s≡st(τ),a≡at(τ))1|Tjθold|⋅1Tτclip(sτ, 1±ϵ(GSPO))A^s,a. = J GSPO(θ| _ old)\;=\; (s≡ s_t(τ),\,a≡ a_t(τ)) _τ _ old _t=0^T_τ-1 1|Tj _ old|· 1T_τ\;clip(s_τ,\,1±ε^( GSPO))\; A_s,a. C.6 Proof of Claim 3.5: GSPO Maintains a trajectory-level TV distance bound Proof. For notation, see: Appendix A.1. Recall the GSPO objective function in RGF form (without minimization clipping): J^GSPO(θ|θold)=∑τ∈Tjθold∑t=0Tτ−1(s≡st(τ),a≡at(τ))1|Tjθold|⋅1Tτclip(sτ, 1±ϵ(GSPO))A^s,a. J GSPO(θ| _ old)\;=\; (s≡ s_t(τ),\,a≡ a_t(τ)) _τ _ old _t=0^T_τ-1 1|Tj _ old|· 1T_τ\;clip(s_τ,\,1±ε^( GSPO))\; A_s,a. We write ϵε instead of ϵ(GSPO)ε^( GSPO) for convenience. Given a sampled trajectory τ, the log-aggregate ratio is defined to be logsτ:=1Tτ∑t=0Tτ−1logrst(τ),at(τ) s_τ:= 1T_τ _t=0^T_τ-1 r_s_t(τ),a_t(τ). This is a Monte Carlo estimation for the negative trajectory-normalized average KL divergence (at the trajectory level), see (49). Thus, the clipping of the aggregate ratio in the objective function maintains D¯KL(tj)≈(50)D¯^KL(τ)≤−log(1−ϵ)≈ϵ(when ϵ∼0). D_ KL^( tj) eq:dbar_tj_as_expectation≈ D_ KL^(τ)\;≤\;- (1-ε)\;≈\;ε (when ε 0). Furthermore, based on 12DKL≥(DTV)2 12D_ KL≥(D_ TV)^2 (Pinsker’s inequality) and p[X2]≥p[X]2E_p[X^2] _p[X]^2 for probability measures, we have: (D¯TV(tj))2=s∼ρ¯(tj)θold[DTV(s)]2≤s∼ρ¯(tj)θold[DTV(s)2]≤s∼ρ¯(tj)θold[12DKL(s)]=12D¯KL(tj)≲ϵ2. ( D_ TV^( tj) )^2\;=\;E_s ρ_ (tj) _ old[D_ TV(s)]^2\;≤\;E_s ρ_ (tj) _ old[D_ TV(s)^2]\;≤\;E_s ρ_ (tj) _ old\! [ 12D_ KL(s) ]\;=\; 12\, D_ KL^( tj)\; \; ε2. Therefore D¯TV(tj)≲ϵ/2 D_ TV^( tj) ε/2, which provides a rough estimation for δ which also bounds the trajectory-normalized average TV divergence. We roughly conclude that δ∼ϵ/2δ ε/2. ∎ Appendix D Equivalence of APC-Obj and sample-based TV-TRPO This appendix establishes that the Aggregational Policy Censoring Objective (APC-Obj) algorithm and sample-based Total Variation Trust Region Policy Optimization (TV-TRPO) produce identical policy updates under standard function approximation assumptions. The equivalence is nontrivial because APC-Obj and sample-based TV-TRPO have superficially different structures: APC-Obj applies per-token clipping with cross-action coupling, while sample-based TV-TRPO maximizes a globally penalized objective. The proof proceeds in three stages: (i) we characterize the maximizer of sample-based TV-TRPO (Theorem D.3), (i) we show that APC-Obj’s clipping mechanism implicitly enforces the same trust-region constraint (Lemma D.9), and (i) we unite the two by showing both produce the same policy ratio deviation (Theorem D.10). D.1 Sampling Notation and Standing Assumptions Let X denote the set of all state–action pairs encountered in the sampled trajectories, and let ns,an_s,a denote the sample multiplicity of pair (s,a)(s,a) in the sampled set. Write []S[X] for the set of all states appearing in X, and for each state s∈[]s [X], let s:=a∈∣(s,a)∈X_s:=\a (s,a) \ be the set of actions co-occurring with s. Define T:=∑(s,a)∈ns,aT:= _(s,a) n_s,a as the total sample count (with multiplicity), and Ts:=∑a∈sns,aT_s:= _a _sn_s,a as the per-state sample count. We also denote the augmented state–action pair space ¯:=(s,a,τ,t)∣(s,a)∈,τ∈Tjθold,t∈0,…,Tτ−1 X:=\(s,a,τ,t) (s,a) ,\;τ _ old,\;t∈\0,…,T_τ-1\\, the set of all sampled state–action pairs augmented with their trajectory membership τ and trajectory time step t. Each element of ¯ X is a distinct sampled token, so |¯|=T| X|=T. The multiplicity ns,an_s,a counts how many augmented entries in ¯ X share the pair (s,a)(s,a): ns,a=|(τ,t)∣(s,a,τ,t)∈¯|n_s,a=|\(τ,t) (s,a,τ,t)∈ X\|. The policy importance-sampling ratio is rs,a(θ|θold):=πθ(a|s)/πθold(a|s)r_s,a(θ| _ old):= _θ(a|s)\,/\, _ _ old(a|s), and we write the ratio deviation as Δs,a:=rs,a−1 _s,a:=r_s,a-1. We adopt two standing identities. Both hold exactly for any policy πθ _θ; their sample-based counterparts are approximations that we assume hold exactly throughout this section. 1. Probability constraint. Since ∑a∈πθ(a|s)=1 _a _θ(a|s)=1 for any policy πθ _θ (not only πθold _ _ old), the identity a∼πθold(⋅|s)[rs,a]=∑a∈πθold(a|s)πθ(a|s)πθold(a|s)=∑a∈πθ(a|s)= 1E_a _ _ old(·|s)[r_s,a]\;=\; _a _ _ old(a|s)\, _θ(a|s) _ _ old(a|s)\;=\; _a _θ(a|s)\;=\;1 holds for every θ; the πθold _ _ old factors cancel algebraically, so the identity does not require πθ=πθold _θ= _ _ old. The sample-based counterpart replaces the expectation with an average over the observed action set sX_s: 1Ts∑a∈sns,ars,a= 1⟺∑a∈sns,aΔs,a= 0,∀s∈[]. 1T_s _a _sn_s,a\,r_s,a\;=\;1 _a _sn_s,a\, _s,a\;=\;0, ∀\,s [X]. (65) This approximation incurs error from two sources: (i) actions in ∖sA _s where πθ _θ may place mass but no samples exist, and (i) finite-sample noise in the empirical frequencies ns,a/Tsn_s,a/T_s. We assume both errors are negligible for the purposes of this analysis. 2. Advantage centering. By definition of the advantage, a∼πθold(⋅|s)[As,aθold]=0E_a _ _ old(·|s)[A _ old_s,a]=0, so under sampling: 1Ts∑a∈sns,aA^s,aθold= 0,∀s∈[]. 1T_s _a _sn_s,a\, A _ old_s,a\;=\;0, ∀\,s [X]. (66) We assume both sample-based identities hold exactly throughout this section. Discounted estimators. In the discounted case, there are a few consistent estimators for discounted quantities. For any X:×→ℝX:S×A , τ∼θ[∑t=0Tτ−1γtXst(τ),at(τ)]=(s,a)∼ρ(γ)θ⋅πθ[Xs,a]=11−γ(s,a)∼ρ¯(γ)θ⋅πθ[Xs,a]=11−γs∼ρ¯(γ)θ[a∼πθ(⋅|s)[Xs,a]]E_τ _θ [ _t=0^T_τ-1γ^tX_s_t(τ),a_t(τ) ]=E_(s,a) _(γ)^θ· _θ[X_s,a]= 11-γE_(s,a) ρ_(γ)^θ· _θ[X_s,a]= 11-γE_s ρ_(γ)^θ[E_a _θ(·|s)[X_s,a]] (the second from the definition of the normalized discounted distribution). The discounted naive Monte Carlo estimator is ^τ∼θ[∑t=0Tτ−1γtXst(τ),at(τ)]=1|Tjθ|∑τ∈Tjθ∑t=0Tτ−1γtXst(τ),at(τ). E_τ _θ [ _t=0^T_τ-1γ^tX_s_t(τ),a_t(τ) ]= 1|Tj^θ| _τ ^θ _t=0^T_τ-1γ^tX_s_t(τ),a_t(τ). An alternative estimator leverages the time independence of the policy πθ(a|s) _θ(a|s): 11−γ^(s,a)∼ρ¯(γ)θ⋅πθ[Xs,a]=11−γ∑s∈[]Γ(s)Γ(1Ts∑a∈sns,aXs,a), 11-γ E_(s,a) ρ_(γ)^θ· _θ[X_s,a]= 11-γ _s [X] (s) ( 1T_s _a _sn_s,aX_s,a ), where Γ(s):=∑(s′,a,τ,t)∈¯,s′=sγt (s):= _(s ,a,τ,t)∈ X,\;s =sγ^t and Γ=∑s∈[]Γ(s) = _s [X] (s). It is worth noting that Γ(s)/Γ∼ρ¯(γ)θ(s) (s)/ ρ^θ_(γ)(s) when number of samples T→∞T→∞. We choose this estimator as our discounted-case sampling estimator, for the following reason: no matter at which step a state st=s_t=s is encountered, the action at∼πθ(⋅|st)a_t _θ(·|s_t) is always a valid sample from πθ(⋅|s) _θ(·|s) and contributes equally to the estimate of a∼πθ(⋅|st)[Xst,a]E_a _θ(·|s_t)[X_s_t,a], because the policy is always time-independent. However, the naive discounted Monte Carlo estimator always weights this (st,at)(s_t,a_t) by γtγ^t, reducing the contribution of valid policy samples drawn at large time steps and thereby wasting their potential for better estimation. D.2 The Sample-Based TV-TRPO Objective The sample-based TV-TRPO objective is the Monte Carlo counterpart of the penalized surrogate (56). Definition D.1 (Sample-based TV-TRPO policy iteration). The updated parameter θnext _ next at each iteration is the argmax of the sample-based TV-TRPO objective, obtained by replacing the expectations with sample averages: J^(TV-TRPO)(θ|θold) J^( TV-TRPO)(θ| _ old) =J^(1)(θ|θold)−4γ‖A^∙θold‖∞(1−γ)2maxs∼ρ(γ)θold^D^TV(s,θ|θold)2 \;=\; J^(1)(θ| _ old)\;-\; 4γ\,\| A _ old_ \|_∞(1-γ)^2 _s _(γ) _ old D_ TV(s,θ| _ old)^2 =J(θold)+^(s,a)∼ρ(γ)θold⋅πθold[rs,aA^s,a] \;=\;J( _ old)+ E_(s,a) _(γ) _ old· _ _ old [r_s,a\, A_s,a ] −4γ‖A^∙θold‖∞(1−γ)2(maxs∼ρ(γ)θold^^a∼πθold(⋅|s)[|rs,a−1|])2 \;-\; 4γ\,\| A _ old_ \|_∞(1-γ)^2 ( _s _(γ) _ old E_a _ _ old(·|s) [|r_s,a-1| ] )^\!2 =J(θold)+1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,ars,aA^s,aθold \;=\;J( _ old)+ 1(1-γ) _s [X] (s)T_s _a _sn_s,a\,r_s,a\, A _ old_s,a −4γ‖A^∙θold‖∞(1−γ)2(maxs∈[]1Ts∑a∈sns,a|rs,a−1|⏟D^TV(s): per-state TV divergence)2. -\, 4γ\,\| A _ old_ \|_∞(1-γ)^2\! ( _s [X] 1T_s _a _sn_s,a\,|r_s,a-1|_ D_ TV(s): per-state TV divergence )^\!2\!. (67) When multiple maximizers exist, sample-based TV-TRPO selects the one with smallest ratio-deviation L2L^2 norm: min∑s∈[]1Ts∑a∈s|rs,a−1|2 _s [X] 1T_s _a _s|r_s,a-1|^2. Note the second term contains the per-state estimated TV divergence D^TV(s) D_ TV(s) (Eq. 39), the Monte-Carlo estimator of the per-state TV distance DTV(s)D_ TV(s) (Eq. 34); see Appendix A.3.1 for the full definitions and ratio forms. The max-TV penalty couples all actions at every state, making the penalized objective a quadratic function of max per-state TV divergence D^TVmax=maxs∈[]D^TV(s) D _ TV= _s [X] D_ TV(s). Reformulation in ratio deviations. Denoting the policy ratio deviation Δs,a=rs,a−1 _s,a=r_s,a-1, the max-TV divergence becomes D^TVmax=maxs∈[]D^TV(s)=maxs∈[]1Ts∑a∈sns,a|Δs,a|. D _ TV\;=\; _s [X] D_ TV(s)\;=\; _s [X] 1T_s _a _sn_s,a\,| _s,a|. Since rs,a=1+Δs,ar_s,a=1+ _s,a, the linear surrogate rewrites as J(θold)+1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,ars,aA^s,aθold=1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aΔs,aA^s,aθold+1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aA^s,aθold+J(θold)⏟constant in Δ.J( _ old)+ 1(1-γ) _s [X] (s)T_s _a _sn_s,a\,r_s,a\, A _ old_s,a\;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, _s,a\, A _ old_s,a\\ +\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, A _ old_s,a~+~J( _ old)_constant in . The constant term is independent of Δ∙ _ and therefore irrelevant for maximization. We may thus equivalently maximize the reduced objective j(Δ):=1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aΔs,aA^s,aθold−4γ‖A^∙θold‖∞(1−γ)2(maxs∈[]1Ts∑a∈sns,a|Δs,a|⏟D^TV(s,Δ): per-state TV divergence)2,j( )\;:=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, _s,a\, A _ old_s,a\;-\; 4γ\,\| A _ old_ \|_∞(1-γ)^2 ( _s [X] 1T_s _a _sn_s,a\,| _s,a|_ D_ TV(s, ): per-state TV divergence )^\!2, (68) That is, j(Δ)j( ) differs from J^(TV-TRPO) J^( TV-TRPO) only by the additive constant J(θold)+1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aA^s,aθoldJ( _ old)+ 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, A _ old_s,a, which is independent of rs,ar_s,a and Δs,a _s,a, so j(Δ)j( ) and J^(TV-TRPO)(Δ|θold) J^( TV-TRPO)( | _ old) share the same maximizer. D.3 Characterizing the Sample-Based TV-TRPO Maximizer The key insight is that TV-TRPO’s quadratic penalty on max TV divergence forces the optimal solution to equalize the TV divergence across all states. This section makes this statement precise. Per-state normalized-TV maximizers. For each state s, consider maximizing the following auxiliary objective over the ratio deviations: Δ^s,∙∗:=argmaxΔs,∙:D^TV(s,Δ)=1,∑a∈sns,aΔs,a=0∑a∈sns,aΔs,aA^s,aθold, ^*_s, \;:=\; _ subarrayc _s, :\; D_ TV(s, )=1,\\ _a _sn_s,a\, _s,a=0 subarray _a _sn_s,a\, _s,a\, A _ old_s,a, (69) with two constraints: (i) the estimated per-state TV divergence D^TV(s,Δ)=1Ts∑a∈sns,a|Δs,a|=1Ts∑a∈sns,a|rs,a−1| D_ TV(s, )= 1T_s _a _sn_s,a\,| _s,a|= 1T_s _a _sn_s,a\,|r_s,a-1| is fixed to 11, and (i) the probability constraint ∑a∈sns,aΔs,a=0 _a _sn_s,a\, _s,a=0. Let M(s):=∑a∈sns,aΔ^s,a∗A^s,aθoldM(s):= _a _sn_s,a\, ^*_s,a\, A _ old_s,a denote the optimal objective value of this per-state problem. Linearity of TV maximizer. Because the objective is linear in Δs,∙ _s, , the TV divergence constraint D^TV(s,Δ)=c D_ TV(s, )=c is also linear in |Δs,∙|| _s, |, and the probability constraint ∑ans,aΔs,a=0 _an_s,a\, _s,a=0 is linear, the maximizer at radius c is simply cΔ^s,∙∗c\, ^*_s, , with optimal value cM(s)c\,M(s): cΔ^s,∙∗=argmaxΔs,∙:D^TV(s,Δ)=c,∑ans,aΔs,a=0∑a∈sns,aΔs,aA^s,aθold=cM(s),c≥0.c\, ^*_s, \;=\; _ subarrayc _s, :\; D_ TV(s, )=c,\\ _an_s,a _s,a=0 subarray _a _sn_s,a\, _s,a\, A _ old_s,a\;=\;c\,M(s), c≥ 0. (70) The same conclusion holds when the equality constraint is relaxed to D^TV(s,Δ)≤c D_ TV(s, )≤ c: Writing Δs,∙=c∗Δ^s,∙ _s, =c^* _s, with c∗≤c^*≤ c and D^TV(s,Δ^)=1 D_ TV(s, )=1, one can reduce the problem to maxc∗≤cc∗M(s) _c^*≤ cc^*M(s), which is attained at c∗=c^*=c: argmaxΔs,∙:D^TV(s,Δ)≤c,∑a∈sns,aΔs,a=0∑a∈sns,aΔs,aA^s,aθold _ subarrayc _s, :\; D_ TV(s, )≤ c,\\ _a _sn_s,a\, _s,a=0 subarray _a _sn_s,a\, _s,a\, A _ old_s,a =argmaxc∗Δ^s,∙:c∗≤c,D^TV(s,Δ^)=1,∑a∈sns,aΔ^s,a=0∑a∈sns,ac∗Δ^s,aA^s,aθold \;=\; _ subarraycc^*\, _s, :\;c^*≤ c,\; D_ TV(s, )=1,\\ _a _sn_s,a\, _s,a=0 subarray _a _sn_s,a\,c^*\, _s,a\, A _ old_s,a =(argmaxc∗≤cc∗M(s))Δ^s,∙∗=cΔ^s,∙∗. \;=\; ( _c^*≤ c\;c^*\,M(s) ) ^*_s, \;=\;c\, ^*_s, . (71) Aggregate notation. Define the aggregate optimal value M:=1Γ∑s∈[]Γ(s)TsM(s)=1Γ∑s∈[]Γ(s)Ts∑a∈sns,aΔ^s,a∗A^s,aθoldM:= 1 _s [X] (s)T_s\,M(s)= 1 _s [X] (s)T_s _a _sn_s,a\, ^*_s,a\, A _ old_s,a. This quantity is the expectation of M(s)Ts M(s)T_s under the probability distribution Γ(s)Γ (s) over []⊆S[X] . This distribution varies smoothly with γ; when γ=1γ=1, Γ(s)Γ (s) becomes TsT T_sT, the normalized counting distribution of ¯ X over []S[X]. Lemma D.2 (M is bounded by the advantage norm). M≤‖A^∙θold‖∞M≤\| A _ old_ \|_∞. Proof. Since 1=D^TV(s,Δ^∗)=1Ts∑a∈sns,a|Δ^s,a∗|1= D_ TV(s, ^*)= 1T_s _a _sn_s,a\,| ^*_s,a|, M(s)Ts=1Ts∑a∈sns,aΔ^s,a∗A^s,aθold≤1Ts∑a∈sns,a|Δ^s,a∗||A^∙θold|∞=|A^∙θold|∞⋅1Ts∑a∈sns,a|Δ^s,a∗|=|A^∙θold|∞. M(s)T_s= 1T_s _a _sn_s,a\, ^*_s,a\, A _ old_s,a≤ 1T_s _a _sn_s,a\,| ^*_s,a|\,| A _ old_ |_∞=| A _ old_ |_∞· 1T_s _a _sn_s,a\,| ^*_s,a|=| A _ old_ |_∞. Therefore, M=s∼Γ(⋅)Γ[M(s)Ts]≤s∼Γ(⋅)Γ[|A^∙θold|∞]=|A^∙θold|∞M=E_s (·) \! [ M(s)T_s ] _s (·) \! [| A _ old_ |_∞ ]=| A _ old_ |_∞. ∎ Theorem D.3 (Sample-based TV-TRPO maximizer). Suppose Δ∙∗ ^*_ maximizes j(Δ)j( ) (equivalently, J^(TV-TRPO) J^( TV-TRPO)) subject to the probability constraint (65). Then Δs,a∗=t∗Δ^s,a∗,t∗=1−γ8γ‖A^∙θold‖∞M. ^*_s,a\;=\;t^*\, ^*_s,a, t^*\;=\; 1-γ8γ\,\| A _ old_ \|_∞\,M. Proof. Let t:=D^TVmax(Δ∗)=maxs∈[]D^TV(s,Δ∗)t:= D _ TV( ^*)= _s [X] D_ TV(s, ^*). We first establish that D^TV(s,Δ∗)=t D_ TV(s, ^*)=t for every s∈[]s [X]. Note that if M(s)=0M(s)=0 for every state s, then M=0M=0, t∗=0t^*=0, and maxj(Δ)=0 j( )=0, so the theorem holds trivially with Δ∗=0 ^*=0. We may therefore assume M(s)>0M(s)>0 for at least one state. Suppose for contradiction that some state s with M(s)>0M(s)>0 has D^TV(s,Δ∗)=t(s)<t D_ TV(s, ^*)=t(s)<t. Since Δ∗ ^* maximizes j, the per-state component Δs,∙∗ ^*_s, must maximize ∑ans,aΔs,aA^s,aθold _an_s,a\, _s,a\, A _ old_s,a subject to D^TV(s,Δ)=t(s) D_ TV(s, )=t(s) and the probability constraint. By the maximizer linearity (70), Δs,∙∗=t(s)Δ^s,∙∗ ^*_s, =t(s)\, ^*_s, , achieving value t(s)M(s)t(s)\,M(s). Now construct Δ∗ ^** by setting Δs,∙∗:=t(s)+t2Δ^s,∙∗ ^**_s, := t(s)+t2\, ^*_s, while keeping Δs′,∙∗=Δs′,∙∗ ^**_s , = ^*_s , for all s′≠s ≠ s. The per-state advantage contribution at s increases to t(s)+t2M(s)>t(s)M(s) t(s)+t2\,M(s)>t(s)\,M(s), while D^TVmax(Δ∗)=t D _ TV( ^**)=t since D^TV(s,Δ∗)=t(s)+t2<t D_ TV(s, ^**)= t(s)+t2<t. Therefore: j(Δ∗) j( ^**) =1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aΔs,a∗A^s,aθold−4γ‖A^∙θold‖∞(1−γ)2t2 \;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, ^**_s,a\, A _ old_s,a\;-\; 4γ\,\| A _ old_ \|_∞(1-γ)^2\,t^2 =Γ(s)(1−γ)Γt−t(s)2TsM(s)+1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aΔs,a∗A^s,aθold−4γ‖A^∙θold‖∞(1−γ)2t2 \;=\; (s)(1-γ) \, t-t(s)2T_s\,M(s)\;+\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, ^*_s,a\, A _ old_s,a\;-\; 4γ\,\| A _ old_ \|_∞(1-γ)^2\,t^2 =Γ(s)(1−γ)Γt−t(s)2TsM(s)+j(Δ∗)>j(Δ∗), \;=\; (s)(1-γ) \, t-t(s)2T_s\,M(s)\;+\;j( ^*)\;>\;j( ^*), where the strict inequality holds since M(s)>0M(s)>0 and t>t(s)t>t(s). This contradicts the optimality of Δ∗ ^*. For states with M(s)=0M(s)=0, the per-state advantage contribution is zero(by TV-TRPO algorithm, when multiple argmax exists, the smallest L2 norm solution is chosen, thus forces D^TV(s)=0 D_ TV(s)=0), thus we can simply not consider them without loss of generality. Hence D^TV(s,Δ∗)=t D_ TV(s, ^*)=t for all s (except those with M(s)=0M(s)=0). Since D^TV(s,Δ∗)=t D_ TV(s, ^*)=t for every s, and each Δs,∙∗ ^*_s, must maximize its per-state advantage contribution subject to this constraint, the maximizer linearity (70) gives Δs,∙∗=tΔ^s,∙∗ ^*_s, =t\, ^*_s, for all s. The discount-weighted total advantage contribution is then: 1Γ∑s∈[]Γ(s)Ts∑a∈sns,aΔs,a∗A^s,aθold=tΓ∑s∈[]Γ(s)TsM(s)=tM. 1 _s [X] (s)T_s _a _sn_s,a\, ^*_s,a\, A _ old_s,a\;=\; t _s [X] (s)T_s\,M(s)\;=\;t\,M. Substituting into j: j(Δ∗)=tM1−γ−4γ‖A^∙θold‖∞(1−γ)2t2.j( ^*)\;=\; t\,M1-γ\;-\; 4γ\,\| A _ old_ \|_∞(1-γ)^2\,t^2. This is a concave quadratic in t with maximum at t∗=1−γ8γ‖A^∙θold‖∞M.∎t^*\;=\; 1-γ8γ\,\| A _ old_ \|_∞\,M. D.4 The APC-Obj Objective (In discounted case) Definition D.4 (APC-Obj policy iteration, in discounted case). The APC-Obj update selects θnew=argmaxθJ^APC-Obj(θ|θold) _ new= _θ\, J APC-Obj(θ| _ old), where J^APC-Obj(θ|θold)=1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,a[clip(rs,a−1,Tsδ(APC-Obj)−∑(s,a′,τ′,t′)∈¯(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|)A^s,aθold+A^s,aθold], J APC-Obj(θ| _ old)\;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a [clip\! (r_s,a-1,\;T_sδ^( APC-Obj)\\ -\!\!\!\!\!\! _ subarrayc(s,a ,τ ,t )∈ X\\ (a ,τ ,t )≠(a,τ,t) subarray\!\!\!\!\!\!|r_s,a -1| )\, A _ old_s,a+ A _ old_s,a ], (72) where δ(APC-Obj):=1−γ8γ‖A^∙θold‖∞Mδ^( APC-Obj):= 1-γ8γ\,\| A _ old_ \|_∞\,M, and M is also used in the sample-based TV-TRPO maximizer (Theorem D.3). The per-entry clip bound is Bs,a,τ,t:=Tsδ(APC-Obj)−∑(s,a′,τ′,t′)∈¯,(a′,τ′,t′)≠(a,τ,t)|rs,a′−1|,B_s,a,τ,t:=T_sδ^( APC-Obj)-\!\!\!\! _ subarrayc(s,a ,τ ,t )∈ X,\,(a ,τ ,t )≠(a,τ,t) subarray\!\!\!\!|r_s,a -1|, and clip(a,B):=clip(a,−B+,B+)clip(a,B):=clip(a,-B^+,B^+) with B+:=max(B,0)B^+:= (B,0). When multiple argmax exists, we assume APC-Obj chooses the one with smallest ratio-deviation L2L^2 norm. D.5 Surrogated APC-Obj Objective and Objective Equivalence To analyze APC-Obj, we introduce a surrogate formulation that replaces the policy parameter θ with the ratio deviation Δ directly. Definition D.5 (Surrogated APC-Obj objective). For ratio deviations Δ∙ _ and threshold δ, define j(APC-Obj)(Δ,δ):=1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈s[ns,aclip(Δs,a,Tsδ−∑a′∈s(ns,a′−δa,a′)|Δs,a′|)A^s,aθold+ns,a(1−(Δ)s)A^s,aθold],j^( APC-Obj)( ,δ)\;:=\; 1(1-γ) _s [X] (s)T_s _a _s [n_s,aclip\! ( _s,a,\;T_sδ-\!\!\! _a _s\!\!(n_s,a - _a,a )\,| _s,a | )\, A _ old_s,a\\ +n_s,a(1-C( )_s)\, A _ old_s,a ], (73) where δa,a′ _a,a is the Kronecker delta, and the correction term is (Δ)s:=1Ts∑a∈sns,aclip(Δs,a,Tsδ(APC-Obj)−∑a′∈s(ns,a′−δa,a′)|Δs,a′|).C( )_s\;:=\; 1T_s _a _sn_s,aclip\! ( _s,a,\;T_sδ^( APC-Obj)-\!\!\! _a _s\!\!(n_s,a - _a,a )\,| _s,a | ). (74) The correction term (Δ)sC( )_s vanishes whenever D^TV(s,Δ)<δ(APC-Obj) D_ TV(s, )<δ^( APC-Obj): in that regime, the clipping is inactive (all clip terms are identities), and the probability constraint (65) gives (Δ)s=1Ts∑ans,aΔs,a=0C( )_s= 1T_s _an_s,a\, _s,a=0. Lemma D.6 (Objective equivalence). Let Δ∙(θ):=r∙(θ|θold)−1 _ (θ):=r_ (θ| _ old)-1. Then j(APC-Obj)(Δ(θ),δ(APC-Obj))=J^APC-Obj(θ|θold).j^( APC-Obj)( (θ),\,δ^( APC-Obj))\;=\; J APC-Obj(θ| _ old). Proof. Expanding the surrogated objective and collecting terms: j(APC-Obj)(Δ(θ),δ(APC-Obj)) j^( APC-Obj)( (θ),δ^( APC-Obj)) =J^APC-Obj(θ|θold)−1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,a(Δ(θ))sA^s,aθold \;=\; J APC-Obj(θ| _ old)\;-\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a\,C( (θ))_s\, A _ old_s,a =J^APC-Obj(θ|θold)−1(1−γ)Γ∑s∈[]Γ(s)(Δ(θ))s⋅1Ts∑a∈sns,aA^s,aθold. \;=\; J APC-Obj(θ| _ old)\;-\; 1(1-γ) _s [X] (s)\,C( (θ))_s· 1T_s _a _sn_s,a\, A _ old_s,a. The advantage centering identity (66) ensures 1Ts∑a∈sns,aA^s,aθold=0 1T_s _a _sn_s,a\, A _ old_s,a=0 for each s, so the correction vanishes. ∎ Remark D.7 (Equivalence with the original form using X¯ X). The surrogated form is obtained from the original APC-Obj (Definition D.4) by grouping entries that share the same pair (s,a)(s,a) and this extra constant 1(1−γ)Γ∑s∈[]Γ(s)C(Δ)sTs∑a∈sns,aΔs,a 1(1-γ) _s∈ S[X] (s)C( )_sT_s _a _sn_s,a\, _s,a, term that evaluates to 0. The augmented state-action pairs (s,a,τ,t)(s,a,τ,t) in the space ¯ X with the same (s,a)(s,a) has the same ratio rs,ar_s,a and the same clip bound. Collecting the ns,an_s,a identical terms produces ns,aclip(Δs,a,Tsδ−∑a′(ns,a′−δa,a′)|Δs,a′|)n_s,aclip( _s,a,\;T_sδ- _a (n_s,a - _a,a )| _s,a |). D.6 Retraction Property of APC-Obj Clipping The following lemma establishes that the APC-Obj clipping mechanism acts as a retraction: when a candidate solution violates the TV constraint, the clipping projects the gated ratios back strictly inside the trust region. Combining this retraction property with the property that smaller D^TV(s) D_ TV(s) always produce argmax solutions with less objective value, the next theorem D.10 used these two properties to show APC-Obj implicitly enforces the update policy, given by the argmax, lies within TV trust-region constraint. Thus, this property can be seen as the mechanism by which APC-Obj enforces the trust-region constraint, without explicitly constraining the optimization problem. Definition D.8 (Clipping operator and centered projection). For any vector us,a∈ℝu_s,a ^X satisfying ∑a∈sns,aus,a=0 _a _sn_s,a\,u_s,a=0 for all s∈[]s [X], define the clipping operator (u∙)s,a:=clip(us,a,Ts(δ−D^TV(s,u))+|us,a|),G(u_ )_s,a\;:=\;clip\! (u_s,a,\;T_s(δ- D_ TV(s,u))+|u_s,a| ), and the centered projection proj((u))s,a:=(u)s,a−((u))s.proj(G(u))_s,a\;:=\;G(u)_s,a-C(G(u))_s. Lemma D.9 (Retraction property). For any u∙u_ satisfying the probability constraint (65): 1. D^TV(s,proj((u)))≤δ D_ TV\! (s,\,proj(G(u)) )≤δ for all s. 2. If D^TV(s,u)>δ D_ TV(s,u)>δ, then D^TV(s,proj((u)))<δ D_ TV\! (s,\,proj(G(u)) )<δ (strict inequality). Proof. When D^TV(s,u)≤δ D_ TV(s,u)≤δ, the clipping is inactive, proj((u))=uproj(G(u))=u, and the claim holds trivially. It remains to prove the case for D^TV(s,u)>δ D_ TV(s,u)>δ. Since all expressions aggregate only over s⊆X_s , we fix a state s without loss of generality. For any vector wa∈ℝsw_a ^X_s, we have TsD^TV(s,w)=∑a∈sns,a|wa|T_s\, D_ TV(s,w)= _a _sn_s,a\,|w_a|. Write va:=(u)s,av_a:=G(u)_s,a and v¯:=1Ts∑a∈sns,ava v:= 1T_s _a _sn_s,a\,v_a; by definition, v¯=(u)s v=C(u)_s. Observe that (−w)=−(w)G(-w)=-G(w), (−w)=−(w)C(-w)=-C(w), and D^TV(s,−w)=D^TV(s,w) D_ TV(s,-w)= D_ TV(s,w); hence proving the result for u implies it holding for −u-u as well. Thus we may therefore proceed to prove only the v¯≥0 v≥ 0 case without loss of generality: if v¯<0 v<0, prove with −u-u, so that (−u)¯=(−u)=−v¯>0 (-u)=C(-u)=- v>0, and by proving the result for −u-u, the above symmetric property implies it holding for u that has v¯<0 v<0 as well. Let s+(w):=a∈s∣wa>0X_s^+(w):=\a _s w_a>0\. By definition of v¯ v, ∑a∈sns,a(va−v¯)=0 _a _sn_s,a(v_a- v)=0. Splitting the absolute value sum into its positive and negative parts: TsD^TV(s,v−v¯) T_s\, D_ TV(s,v- v) =∑a∈sns,a|va−v¯| \;=\; _a _sn_s,a\,|v_a- v| =∑a∈s+(v−v¯)ns,a(va−v¯)−∑a∈s+(v¯−v)ns,a(va−v¯). \;=\; _a _s^+(v- v)n_s,a(v_a- v)\;-\; _a _s^+( v-v)n_s,a(v_a- v). Since (∑a∈s+(v−v¯)ns,a(va−v¯))+(∑a∈s+(v¯−v)ns,a(va−v¯))=∑a∈sns,a(va−v¯)=0 ( _a _s^+(v- v)n_s,a(v_a- v) )+ ( _a _s^+( v-v)n_s,a(v_a- v) )= _a _sn_s,a(v_a- v)=0, the two partial sums are equal in magnitude, giving: TsD^TV(s,v−v¯)= 2∑a∈s+(v−v¯)ns,a(va−v¯).T_s\, D_ TV(s,v- v)\;=\;2\! _a _s^+(v- v)n_s,a(v_a- v). Since v¯≥0 v≥ 0, every a∈s+(v−v¯)a _s^+(v- v) satisfies va>0v_a>0, which implies us,a>0u_s,a>0 (the clipping va=(u)s,av_a=G(u)_s,a preserves the sign of us,au_s,a whenever va≠0v_a≠ 0). When D^TV(s,u)>δ D_ TV(s,u)>δ, the clipping reduces the absolute value of us,au_s,a to vav_a by exactly Ts(D^TV(s,u)−δ)T_s( D_ TV(s,u)-δ) for va>0v_a>0, i.e.: us,a=va+Ts(D^TV(s,u)−δ)>va.u_s,a\;=\;v_a+T_s( D_ TV(s,u)-δ)\;>\;v_a. Substituting va=us,a−Ts(D^TV(s,u)−δ)v_a=u_s,a-T_s( D_ TV(s,u)-δ) for each a∈s+(v−v¯)a _s^+(v- v): TsD^TV(s,v−v¯) T_s\, D_ TV(s,v- v) = 2∑a∈s+(v−v¯)ns,a(va−v¯) \;=\;2\! _a _s^+(v- v)n_s,a(v_a- v) = 2∑a∈s+(v−v¯)ns,a(us,a−Ts(D^TV(s,u)−δ)−v¯) \;=\;2\! _a _s^+(v- v)n_s,a (u_s,a-T_s( D_ TV(s,u)-δ)- v ) = 2∑a∈s+(v−v¯)ns,aus,a− 2∑a∈s+(v−v¯)ns,a(Ts(D^TV(s,u)−δ)+v¯). \;=\;2\! _a _s^+(v- v)n_s,a\,u_s,a\;-\;2\! _a _s^+(v- v)n_s,a (T_s( D_ TV(s,u)-δ)+ v ). The chain of inequalities us,a>va≥va−v¯u_s,a>v_a≥ v_a- v gives s+(v−v¯)⊆s+(v)⊆s+(u)X_s^+(v- v) _s^+(v) _s^+(u). The probability constraint yields ∑a∈s+(u)ns,aus,a=∑a∈s+(−u)ns,a(−us,a) _a _s^+(u)n_s,a\,u_s,a= _a _s^+(-u)n_s,a\,(-u_s,a) (from ∑ans,aus,a=0 _an_s,a\,u_s,a=0) and consequently 2∑a∈s+(u)ns,aus,a=TsD^TV(s,u)2 _a _s^+(u)n_s,a\,u_s,a=T_s\, D_ TV(s,u). Combining these: TsD^TV(s,v−v¯) T_s\, D_ TV(s,v- v) ≤ 2∑a∈s+(u)ns,aus,a− 2∑a∈s+(v−v¯)ns,a(Ts(D^TV(s,u)−δ)+v¯) \;≤\;2\! _a _s^+(u)n_s,a\,u_s,a\;-\;2\! _a _s^+(v- v)n_s,a (T_s( D_ TV(s,u)-δ)+ v ) =TsD^TV(s,u)−N(Ts(D^TV(s,u)−δ)+v¯) \;=\;T_s\, D_ TV(s,u)\;-\;N (T_s( D_ TV(s,u)-δ)+ v ) =TsNδ−Ts(N−1)D^TV(s,u)−Nv¯ \;=\;T_sNδ-T_s(N-1)\, D_ TV(s,u)-N v =Tsδ−Ts(N−1)(D^TV(s,u)−δ)−Nv¯, \;=\;T_sδ-T_s(N-1) ( D_ TV(s,u)-δ )-N v, where N:=2∑a∈s+(v−v¯)ns,a∈ℕN:=2 _a _s^+(v- v)n_s,a is necessarily even (being twice a sum of positive integers). Noting that proj((u))=v−v¯proj(G(u))=v- v, two cases arise. If N=0N=0, then va≤v¯v_a≤ v for all a, so D^TV(s,proj((u)))=0<δ D_ TV(s,proj(G(u)))=0<δ (since ∑ans,a(va−v¯)=0 _an_s,a(v_a- v)=0 forces va=v¯v_a= v for all a). If N≥2N≥ 2 (since N is even and positive), then Ts(N−1)≥Ts≥1>0T_s(N-1)≥ T_s≥ 1>0. Combined with D^TV(s,u)−δ>0 D_ TV(s,u)-δ>0 and Nv¯≥0N v≥ 0, this gives Ts(N−1)(D^TV(s,u)−δ)+Nv¯>0T_s(N-1)( D_ TV(s,u)-δ)+N v>0, and therefore TsD^TV(s,proj((u)))<TsδT_s\, D_ TV(s,proj(G(u)))<T_sδ, i.e., D^TV(s,proj((u)))<δ D_ TV(s,proj(G(u)))<δ. In either case, the strict inequality holds. ∎ D.7 Main Theorem: Sample-Based TV-TRPO and APC-Obj Equivalence Theorem D.10 (Sample-based TV-TRPO APC-Obj equivalence). Suppose θ satisfies standard function approximation assumptions, then APC-Obj and sample-based TV-TRPO produce the same policy update: πθnew(APC-Obj)=πθnew(TV-TRPO). _ _ new^( APC-Obj)\;=\; _ _ new^( TV-TRPO). Proof. By Lemma D.6, optimizing the APC-Obj objective over θ is equivalent to optimizing the surrogated APC-Obj objective j(APC-Obj)(Δ,δ(APC-Obj))j^( APC-Obj)( ,δ^( APC-Obj)) over ratio deviations Δ . We therefore work with j(APC-Obj)j^( APC-Obj) directly. Rewriting j(APC-Obj)j^( APC-Obj) in terms of Definition D.8: j(APC-Obj)(Δ,δ(APC-Obj)) j^( APC-Obj)( ,δ^( APC-Obj)) =1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,a[clip(Δs,a,Tsδ(APC-Obj)−∑a′(ns,a′−δa,a′)|Δs,a′|) \;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a [clip ( _s,a,\;T_sδ^( APC-Obj)- _a (n_s,a - _a,a )\,| _s,a | ) ⋅A^s,aθold+(1−(Δ)s)A^s,aθold] · A _ old_s,a+(1-C( )_s)\, A _ old_s,a ] =1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,a[(clip(Δs,a,Tsδ(APC-Obj)−∑a′(ns,a′−δa,a′)|Δs,a′|) \;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a [ (clip ( _s,a,\;T_sδ^( APC-Obj)- _a (n_s,a - _a,a )\,| _s,a | ) −(Δ)s)A^s,aθold+A^s,aθold] -C( )_s ) A _ old_s,a+ A _ old_s,a ] =1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,a[proj((Δ))s,aA^s,aθold \;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a [proj(G( ))_s,a\, A _ old_s,a +A^s,aθold]. + A _ old_s,a ]. (75) Here the second equality collects the (1−(Δ)s)(1-C( )_s) factor, and the third identifies the clipping-minus-correction expression with proj((Δ))s,aproj(G( ))_s,a via Definition D.8. Observe that the per-action clip term admits the equivalent form clip(Δs,a,Tsδ(APC-Obj)−∑a′∈s(ns,a′−δa,a′)|Δs,a′|)=clip(Δs,a,Ts(δ(APC-Obj)−D^TV(s,Δ))+|Δs,a|),clip\! ( _s,a,\;T_sδ^( APC-Obj)-\!\!\! _a _s\!\!(n_s,a - _a,a )\,| _s,a | )\;=\;clip\! ( _s,a,\;T_s(δ^( APC-Obj)- D_ TV(s, ))+| _s,a| ), which connects the surrogated APC-Obj clipping to the operator G in Definition D.8. The APC-Obj maximizer satisfies the trust-region constraint. Let Δ∗ ^* denote the maximizer of j(APC-Obj)(⋅,δ(APC-Obj))j^( APC-Obj)(·,δ^( APC-Obj)). We claim D^TV(s,Δ∗)≤δ(APC-Obj) D_ TV(s, ^*)≤δ^( APC-Obj) for all s∈[]s [X]. Suppose for contradiction that D^TV(s,Δ∗)>δ(APC-Obj) D_ TV(s, ^*)>δ^( APC-Obj) for some s. By Lemma D.9, D^TV(s,proj((Δ∗)))<δ(APC-Obj) D_ TV\! (s,\,proj(G( ^*)) )<δ^( APC-Obj). The maximizer linearity (70) then yields: ∑a∈sns,aproj((Δ∗))s,aA^s,aθold≤D^TV(s,proj((Δ∗)))M(s). _a _sn_s,a\,proj(G( ^*))_s,a\, A _ old_s,a\;≤\; D_ TV\! (s,\,proj(G( ^*)) )\,M(s). Construct Δ∗ ^** by setting Δs,a∗:=δ(APC-Obj)Δ^s,a∗ ^**_s,a:=δ^( APC-Obj)\, ^*_s,a for actions at state s and Δs′,a′∗:=Δs′,a′∗ ^**_s ,a := ^*_s ,a for all other states. Since D^TV(s,Δ∗)=δ(APC-Obj) D_ TV(s, ^**)=δ^( APC-Obj), the clipping at state s is inactive for Δ∗ ^**, so proj((Δ∗))s,a=δ(APC-Obj)Δ^s,a∗proj(G( ^**))_s,a=δ^( APC-Obj)\, ^*_s,a, achieving per-state value δ(APC-Obj)M(s)δ^( APC-Obj)\,M(s). Therefore: j(APC-Obj)(Δ∗,δ(APC-Obj)) j^( APC-Obj)( ^*,δ^( APC-Obj)) =1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,a[proj((Δ∗))s,aA^s,aθold+A^s,aθold] \;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a [proj(G( ^*))_s,a\, A _ old_s,a+ A _ old_s,a ] =1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,a[proj((Δ∗))s,aA^s,aθold+A^s,aθold] \;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a [proj(G( ^**))_s,a\, A _ old_s,a+ A _ old_s,a ] −Γ(s)(1−γ)ΓM(s)Ts(δ(APC-Obj)−D^TV(s,proj((Δ∗)))) \;-\; (s)(1-γ) \, M(s)T_s (δ^( APC-Obj)- D_ TV(s,\,proj(G( ^*))) ) =j(APC-Obj)(Δ∗,δ(APC-Obj))−Γ(s)(1−γ)ΓM(s)Ts(δ(APC-Obj)−D^TV(s,proj((Δ∗)))) \;=\;j^( APC-Obj)( ^**,δ^( APC-Obj))\;-\; (s)(1-γ) \, M(s)T_s (δ^( APC-Obj)- D_ TV(s,\,proj(G( ^*))) ) <j(APC-Obj)(Δ∗,δ(APC-Obj)), \;<\;j^( APC-Obj)( ^**,δ^( APC-Obj)), contradicting the optimality of Δ∗ ^*. Hence D^TV(s,Δ∗)≤δ(APC-Obj) D_ TV(s, ^*)≤δ^( APC-Obj) for all s. Reduction to constrained linear maximization. Since D^TV(s,Δ∗)≤δ(APC-Obj) D_ TV(s, ^*)≤δ^( APC-Obj) for all s, the clipping terms in (75) reduce to the identity ((Δ)s,a=Δs,aG( )_s,a= _s,a) and (Δ∗)s=0C( ^*)_s=0. The surrogated APC-Obj objective thus simplifies to: j(APC-Obj)(Δ,δ(APC-Obj))=1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aΔs,aA^s,aθold+1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aA^s,aθold.j^( APC-Obj)( ,δ^( APC-Obj))\;=\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, _s,a\, A _ old_s,a\\ +\; 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, A _ old_s,a. Dropping the constant, the maximization becomes: Δ∙∗=argmaxΔ:D^TV(s,Δ)≤δ(APC-Obj)∀s,∑a∈sns,aΔs,a=0∀s1(1−γ)Γ∑s∈[]Γ(s)Ts∑a∈sns,aΔs,aA^s,aθold. ^*_ \;=\; _ subarrayc :\; D_ TV(s, )≤δ^( APC-Obj)\;∀ s,\\ _a _sn_s,a\, _s,a=0\;∀ s subarray 1(1-γ) _s [X] (s)T_s _a _sn_s,a\, _s,a\, A _ old_s,a. By the maximizer linearity (70), the unique maximizer is Δ∙∗=δ(APC-Obj)Δ^∙∗ ^*_ =δ^( APC-Obj)\, ^*_ . Identification with the sample-based TV-TRPO maximizer. By definition, δ(APC-Obj)=1−γ8γ‖A^∙θold‖∞M=t∗δ^( APC-Obj)= 1-γ8γ\,\| A _ old_ \|_∞\,M=t^*, coinciding with the TV-TRPO optimal step size from Theorem D.3. Therefore the APC-Obj maximizer is Δ∙∗=δ(APC-Obj)Δ^∙∗=t∗Δ^∙∗ ^*_ =δ^( APC-Obj)\, ^*_ =t^*\, ^*_ , which is precisely the TV-TRPO maximizer. Lifting to policies. Let θ∗θ^* denote the policy update that maximizes the surrogated APC-Obj objective j(APC-Obj)(Δ(θ),δ(APC-Obj))j^( APC-Obj)( (θ),δ^( APC-Obj)). If Δ(θ∗) (θ^*) did not coincide with Δ∗ ^*, then under the standard function approximation assumptions there would exist θ∗θ^** with Δ(θ∗)=Δ∗ (θ^**)= ^*, giving j(APC−Obj)(Δ(θ∗),δ(APC-Obj))>j(APC−Obj)(Δ(θ∗),δ(APC-Obj))j^( APC-Obj)( (θ^**),δ^( APC-Obj))>j^( APC-Obj)( (θ^*),δ^( APC-Obj)), contradicting the optimality of θ∗θ^*. Hence Δ(θ∗)=Δ∗=δ(APC-Obj)Δ^∙∗ (θ^*)= ^*=δ^( APC-Obj)\, ^*_ . By Lemma D.6, Δ(θ∗) (θ^*) also maximizes the original APC-Obj objective. By Theorem D.3, Δ(θ∗)=t∗Δ^∙∗ (θ^*)=t^*\, ^*_ is also the TV-TRPO maximizer. Therefore: πθnew(APC-Obj)=(δ(APC-Obj)Δ^∙∗+1)πθold=πθnew(TV-TRPO).∎ _ _ new^( APC-Obj)\;=\;(δ^( APC-Obj)\, ^*_ +1)\, _ _ old\;=\; _ _ new^( TV-TRPO). D.8 Trust-Region Bound for APC-Obj and TV-TRPO Define the TRPO trust-region radius δ(TRPO):=1−γ8γ.δ^( TRPO):= 1-γ8γ. (76) Lemma D.11 (APC-Obj trust-region radius is bounded). δ(APC-Obj)≤δ(TRPO)δ^( APC-Obj)≤δ^( TRPO). Proof. By Lemma D.2, M≤‖A^∙θold‖∞M≤\| A _ old_ \|_∞. Therefore, δ(APC-Obj)=1−γ8γ‖A^∙θold‖∞M≤1−γ8γ‖A^∙θold‖∞‖A^∙θold‖∞=1−γ8γ=δ(TRPO).∎δ^( APC-Obj)= 1-γ8γ\,\| A _ old_ \|_∞\,M≤ 1-γ8γ\,\| A _ old_ \|_∞\,\| A _ old_ \|_∞= 1-γ8γ=δ^( TRPO). Lemma D.12 (TV-TRPO and APC-Obj updates lie within the trust region). The TV-TRPO and APC-Obj policy updates lie in the trust region ℬδ(TRPO)TV-TR(θold)B TV-TR_δ^( TRPO)( _ old). Proof. By Theorem D.3, DTVmax(πθold∥πθnew(TV-TRPO))=t∗=1−γ8γ‖A^∙θold‖∞M≤1−γ8γ=δ(TRPO),D _ TV( _ _ old\,\|\, _θ^( TV-TRPO)_ new)=t^*= 1-γ8γ\,\| A _ old_ \|_∞\,M≤ 1-γ8γ=δ^( TRPO), so πθnew(TV-TRPO)∈ℬδ(TRPO)TV-TR(θold) _θ^( TV-TRPO)_ new TV-TR_δ^( TRPO)( _ old). By Theorem D.10, πθnew(APC-Obj)=πθnew(TV-TRPO)∈ℬδ(TRPO)TV-TR(θold) _θ^( APC-Obj)_ new= _θ^( TV-TRPO)_ new TV-TR_δ^( TRPO)( _ old). ∎ Appendix E Fiber Bundle Theory E.1 Fiber Bundle The fiber bundle is the algebraic object underlying the local-global modelling. We first recall the relevant concepts before defining the specific bundle used in this work. Definition E.1 (Fiber bundle). A smooth fiber bundle is a tuple (E,B,πE,F)(E,B, _E,F) where E,B,FE,B,F are manifolds and πE:E→B _E:E→ B is a smooth surjection satisfying the local triviality condition. Specifically, for every b∈Bb∈ B, there exists an open neighborhood U⊂BU⊂ B containing b and a diffeomorphism ΦU _U (called a local trivialization): ΦU:πE−1(U)→≅U×F _U: _E^-1(U) U× F such that proj1(ΦU(x))=πE(x)for all x∈πE−1(U),proj_1( _U(x))= _E(x) all x∈ _E^-1(U), where proj1:U×F→Uproj_1:U× F→ U is the projection onto the first factor, defined by (u,f)↦u(u,f) u. The preimage Eb:=πE−1[b]≅FE_b:= _E^-1[\b\] F is called the fiber over b, and B is called the base. Remark E.2 (Triviality, base space, and extensibility). In the setting of this paper, the base space B is finite (trajectory indices times sign channels), so every open cover admits a refinement into singletons and all transition functions are trivial. The fiber bundle is therefore globally trivializable (E≅B×FE B× F). Despite not yet having utilized much of the interesting topological structure that the fiber bundle enables, the fiber bundle formalism already has immediate merits in policy optimization despite this triviality. Moreover, the algebraic compositionality of fibrations, rather than their topology, is what enables the Fibration Gating Hierarchy (Section 6.1) and its concrete instantiation FiberPO-Domain (Section 6.2), demonstrating that the formalism provides genuine structural utility beyond the topologically trivial two-level case. The immediate merits are as follows: (i): It always formalizes general dependency, local-global structure, and for that it provides grounds to define well-behaved algebraic operations (in our paper, it is manifested with canonical decomposition structure of densities that contains policy ratio’s information into base and fiber components so that the FBG gating operator (Definition 4.3) can exploit upon). It enables treatment for continuous state-action spaces without modification, which is also one of the central merit of fiber bundle gating(FBG) and fibration gating hierarchy(FGH) formulations. (i): It enables the synthetic treatment of heterogeneous architectures at LLM settings. More specifically, the choice of base space B determines what constitutes “global information” in the fiber bundle model, and the choice B is the primary key for extending FBG to richer, more heterogeneous settings. With B=Tjθold×−1,+1B=Tj _ old×\-1,+1\ as in FiberPO-Trajectory, the base gate already provides refined control over trajectory-level drift while preserving update signals from well-behaved individual tokens, manifested in training stability and policy entropy behaviors. The same framework accommodates finer-grained structure without modifying the gating machinery. In Section 6, we generalize FBG to the Fibration Gating Hierarchy (FGH) by composing fibrations into a chain, and derive FiberPO-Domain, a four-level instantiation that enriches the base with domain and prompt group indices to produce per-domain and per-group aggregates, so the base gate regulates each domain’s drift independently. Even more, the natural fibration of permitted actions over space of states forms a natural and possibly highly nontrivial, geometrically rich fiber bundle, which also fits naturally into the FBG framework, and may be further exploited for stability control and achieving better performance. In each case, the fiber Eb:=πE−1[b]⊆E_b:= _E^-1[\b\] E collects exactly those tokens belonging to class b∈Bb∈ B, the Markov kernel K reflects gated base densities back to the bundle, and the atomic gate gBaseg_ Base operates per class, gFiberg_ Fiber operates per token. The local-global information decomposition is formalized by the concept of densities and Markov kernels (see Appendix E.2 and E.3) and remains unchanged for all these cases. E.2 Density on a Manifold Definition E.3 (Density on a vector space). A density on an n-dimensional vector space V is a function μ:V×⋯×V⏟n→ℝμ: V×·s× V_n satisfying the following condition: if T:V→VT:V→ V is a linear map, then μ(Tv1,⋯,Tvn)=|detT|μ(v1,⋯,vn).μ(Tv_1,·s,Tv_n)=|det\,T|\,μ(v_1,·s,v_n). Let (V)D(V) denote the set of all densities on V. Definition E.4 (Density on a manifold). Let ℳ M be a smooth manifold. The set ℳ=⨆p∈ℳ(Tpℳ)D M= _p∈ MD(T_p M) is called the density bundle of ℳ M. A section of ℳD M is called a density on ℳ M. We denote the space of all densities on the manifold ℳM as 1ℳ D^1M or M. We denote the space of all global sections of the density bundle of ℳ M as Γ(ℳ,1ℳ) (M,D^1M). Thus, Γ(ℳ,1ℳ)=1ℳ≡. (M,D^1M)= D^1M . The fiber over a point x∈ℳx is denoted x1ℳD^1_xM. It can be understood as the space of densities localized at this particular point. Remark E.5. Any density μ defines, at least locally, a smooth signed or complex measure ν on ℳ M, so the integral ∫Kμ=ν(K) _Kμ=ν(K) is well defined for any compact K⊂ℳK⊂ M, as is ∫Kfμ=∫Kfdν _Kfμ= _Kf\,dν for any f∈Cc(ℳ)f∈ C_c( M). E.3 Fiber Bundle Gating: Detailed Exposition We provide a detailed exposition of the Fiber Bundle Gating (FBG) framework introduced in Definition 4.3. The FBG instance is specified by the tuple (E,B,πE,K,gBase,gFiber,−;ℱ,ℛ)(E,B, _E,K,g_ Base,g_ Fiber,-;\,F,R), with πE:E→B _E:E→ B being a fiber bundle. Recall ,E,B are the spaces of densities on the bundle space and base space E,BE,B, respectively. The associated FBG gating function G(πE,K,gBase,gFiber,−):→G_( _E,K,g_ Base,g_ Fiber,-):E is defined as: G(σ):=K((⨁b∈BgBase,b)(πE∗σ))+((⨁i∈EgFiber,i,πE∗(σ))(σ−K(πE∗σ))).G(σ):=K\! ( ( _b∈ Bg_ Base,b )( _E*σ) )+ ( ( _i∈ Eg_ Fiber,i, _E*(σ) ) (σ-K( _E*σ) ) ). We now discuss each constituent object in detail. Object 1: Fiber bundle. πE:E→B _E:E→ B is a fiber bundle. The local and global information of interest are associated with the points inside the bundle space and base space, respectively. This association is formalized by the densities on the total space and base space. The choice of (E,B,πE)(E,B, _E) is a design decision in FBG: it determines the granularity at which the model operates, while the bundle projection πE _E determines the structural decomposition into classes. Different base spaces yield qualitatively different gating behaviors from the same FBG machinery. With B=Tjθold×−1,+1B=Tj _ old×\-1,+1\, the base gate controls trajectory-level drift. The remaining FBG objects (K, gBaseg_ Base, gFiberg_ Fiber, ℱF, ℛR) then specialize to the chosen granularity. Object 2: Fibration decomposition map and recovery map. Recall the definition of ¯ X, the augmented state-action space, and ℝ>0¯R_>0 X, the (augmented) policy ratio space (see Appendix A.4). To incorporate the policy ratios r∙r_ (the real-world information available during training) into the fiber bundle (which is so far a purely mathematical construct), we need to define a way to associate local information with points in E and global information with points in B. Because “local” and “global information” are intuitive but imprecise notions, we formalize them as densities on a manifold: the fibration decomposition map ℱF converts policy ratios into densities on the total space E, these densities intuitively represent the weight, importance or relevance the corresponding local information has in the global quantities; algebraically, they are unique, canonical forms for the many possible expression of essentially the same global aggregation quantities. The pushforward πE∗ _E* aggregates these to densities on the base B to produce interested global quantities, and the gating operators act on these densities. This density-based formulation makes the interaction between local and global information precise and compositional. In summary, we introduce: 1. The density spaces E and B (see Appendix E.2). 2. The fibration decomposition map ℱ:ℝ>0¯→F:R_>0 X , which converts policy ratios r∙r_ to densities on the total space. 3. The policy ratio recovery map ℛ:→ℝ>0¯R:E _>0 X, which converts densities back to policy ratios. ℛR is a left-inverse of ℱF. Definition E.6 (Local information in FBG). Given any point i∈Ei∈ E in the bundle space, the local information qiq_i associated to i is then represented as qi=ℱ(r∙)iq_i=F(r_ )_i, the density ℱ(r∙)F(r_ ) localized at the point i. Definition E.7 (Global information in FBG). Given any point b∈Bb∈ B, the global information qbq_b associated to b is then represented as qb=(πE∗ℱ(r∙))bq_b=( _E*F(r_ ))_b. Here πE∗ℱ(r∙) _E*F(r_ ) is the pushforward of the density ℱ(r∙)∈F(r_ ) (which consists of local information) by the bundle projection map πE _E. Finally, the recovery map ℛ:→ℝ>0¯R:E _>0 X retrieves the gated information from the density space back to policy ratios, enabling their use in the surrogate objective. Object 3: Reflecting Markov kernel. K:→K:B is a Markov kernel that reflects densities on the base space to densities on the bundle space, satisfying πE∗∘K=id _E* K=id_B. The reflecting condition has a direct geometric interpretation. For a density σ∈σ (local information), the pushforward πE∗(σ)∈ _E*(σ) yields the global information. The subtraction σ−K(πE∗σ)σ-K( _E*σ) decouples the influence of the global information πE∗σ _E*σ from σ through the kernel K. After this decoupling, the local residual should no longer ”contain” global information, i.e., no longer contribute to any global information, equivalent to requiring πE∗(σ−K(πE∗σ))=0 _E*(σ-K( _E*σ))=0. This condition is equivalent to the reflecting property πE∗∘K=id _E* K=id_B. Intuition for the FBG gating function. The two terms in G(σ)G(σ) admit the following interpretation: 1. The term K((⨁b∈BgBase,b)(πE∗σ))K\! ( ( _b∈ Bg_ Base,b )( _E*σ) ): given the density (local information) σ∈σ , the atomic gating function gates its global portion πE∗σ _E*σ, and the Markov kernel K brings the gated global information back to the total space. 2. The term (⨁i∈EgFiber,i,πE∗(σ))(σ−K(πE∗σ)) ( _i∈ Eg_ Fiber,i, _E*(σ) ) (σ-K( _E*σ) ): the residual σ−K(πE∗σ)σ-K( _E*σ) captures within-context variation after removing the global contribution. The fiber gating functions gFiberg_ Fiber gate this residual independently at each point. Object 4: Atomic gating functions. Recall x1ℳD^1_xM is the space of densities on manifold ℳM localized at point x∈ℳx (Definition E.4). The atomic gating functions are: gBase,b:b1B→b1B,gFiber,i,pB:i1E→i1E,g_ Base,b:D^1_bB ^1_bB, g_ Fiber,i,p_B:D^1_iE ^1_iE, defined pointwise, where b∈Bb∈ B, i∈Ei∈ E, and pB∈p_B . Note that the fiber gating function gFiber,i,pBg_ Fiber,i,p_B may depend on an extra parameter666In the current FiberPO formulation, this dependence is not used. However, in general it is natural to introduce the global quantity/information influence on the gating of local quantities. We will be exploring this dependency in the future., the base density pB∈p_B . This reflects the fact that local gating may be conditioned on global context. FBG gating form. Recalling that an FBG instance induces a RGF surrogate objective (Definition 4.4): J^(G;ℱ,ℛ)(θ|θold):=∑τ∈Tjπθold[∑t=0Tτ−11Tτℛ∘G∘ℱ(r∙)st,atA^st,at]. J_(G;F,R)(θ| _ old):= _τ _ _ old [ _t=0^T_τ-1 1T_τ\,R G (r_ )_s_t,a_t\; A_s_t,a_t ]. Using the maps ℱ,G,ℛF,G,R, the final policy ratio transformation is r∙↦ℛ∘G∘ℱ(r∙)∈ℝ>0¯r_ G (r_ ) _>0 X. For all r∙∈ℝ>0¯r_ _>0 X, this transformation leverages the fiber bundle gating framework to treat local and global information in a structured way. Extensibility. All four objects above are defined relative to the chosen base space B. Extending B to include additional classifying factors (domain, expert index, temporal grouping) naturally extends the FBG instance: the pushforward πE∗ _E* produces finer aggregates, the Markov kernel K reflects finer base densities back to the bundle, and the atomic gates gBaseg_ Base, gFiberg_ Fiber operate per class. Theorem 4.5 guarantees first-order agreement with the true RL objective for any such extension, provided the atomic gates satisfy the identity conditions at the on-policy point. The FiberPO-Trajectory objective derived in Section 5 is the simplest nontrivial instantiation of this framework. The Fibration Gating Hierarchy (Section 6.1) generalizes this construction to arbitrary hierarchical depth, and FiberPO-Domain (Section 6.2) demonstrates the concrete four-level case incorporating domain and prompt group structure. Global-to-local information exchange of FBG. A crucial structural advantage of FBG, beyond first-order agreement near on-policy, is that it provides a principled mechanism for exchanging global gating effects with individual token ratios. We explain why this is non-trivial and why naive alternatives might fail. In the RGF framework (Definition 3.1), the ratio gating map :ℝ>0ℰ→ℝ>0ℰG:R_>0^E _>0^E takes the full ratio tuple r∙r_ as input and produces a gated ratio for each entry. Global quantities—such as the trajectory-level total variation distance D¯TV(tj) D_ TV (tj), or the sequence aggregate ratio sτ±s_τ^±—are computed from many individual ratios. While these global quantities are well defined as functions of r∙r_ , they have no natural counterpart at the level of a single token ratio rs,ar_s,a. That is, there is no canonical way to “assign” or “attribute” a global quantity to any individual entry in ratio space. This creates a fundamental difficulty for any approach that attempts to incorporate global gating directly in ratio space. Suppose one tries to define a gating map G that simultaneously enforces a global budget (e.g., a trajectory-level TV bound) and local per-token clipping, without the structured decomposition that FBG provides. Several problems arise: 1. Loss of first-order agreement. The global quantity is a nonlinear function of many ratios. Naively incorporating it into per-token clipping changes the Jacobian ∂(r∙)i/∂rj (r_ )_i/∂ r_j at the on-policy point r∙=r_ =1. Unless the coupling is carefully designed, the surrogate will no longer agree with the true RL objective to first order near on-policy. 2. Uncontrolled gradient directions. When global and local gating are entangled in ratio space, the gradient ∇θJ _θ J at points r∙≠r_ 1 may point in directions that are undesirable: for instance, the gradient may encourage individual token ratios to move in a direction that increases the trajectory-level divergence D¯TV(tj) D_ TV (tj), rather than respecting the trust region. This is because the global constraint, when projected onto individual token ratios without a structured decomposition, can create competing objectives between the global budget and local per-token updates. 3. Loss of fine-grained control. Without the orthogonal decomposition that the reflecting condition πE∗∘K=id _E* K=id_B provides, there is no way to gate the pure local variation—the part of each token’s density that remains after the global contribution has been cleanly removed. When global and local information are entangled, the local gate inevitably re-gates global information that has already been gated, or conversely, the global gate inadvertently constrains genuinely local variation. This double-counting makes it difficult for the two gates to operate at their intended granularity, potentially compromising the precision of the trajectory-level budget and the locality of per-token control. On top of having the first-order agreement, FBG avoids the problems 2 and 3 through the density-based decoupling. The pushforward πE∗ _E* extracts global information from local densities in a canonical way. The Markov kernel K reflects the gated global signal back to the total space, and the reflecting condition ensures that the residual σ−K(πE∗σ)σ-K( _E*σ) carries no base-level information. As a result: • The base gate gBaseg_ Base controls the global budget in isolation, operating on the trajectory-level base density πE∗σ _E*σ without interference from local variation. • The fiber gate gFiberg_ Fiber operates on the purest form of local variation: the residual σ−K(πE∗σ)σ-K( _E*σ), from which all global influence has been subtracted. This means gFiberg_ Fiber gates each token’s deviation from the trajectory aggregate, not its absolute value—the most refined level of local control possible. • Because global and local components are orthogonal (πE∗(σ−K(πE∗σ))=0 _E*(σ-K( _E*σ))=0), the two gates compose without cross-contamination: Their effects are independent and additive in the density space, guaranteeing that the combined gating inherits the desired properties of each component. In summary, the structured decomposition σ↦K(πE∗σ)+(σ−K(πE∗σ))σ K( _E*σ)+(σ-K( _E*σ)) is not merely an organizational convenience; it enables non-interfering gating at both the global and local scales, while faithfully exchanging global gating effects with individual token ratios. E.4 Proof of First-Order Agreement (Theorem 4.5) We prove that the FBG surrogate objective agrees with the true RL objective to first order at θ=θoldθ= _ old. Proof. Value agreement. At θ=θoldθ= _ old, all importance ratios equal 1, so ℱ(r∙)=ℱ()F(r_ )=F(1). The identity conditions on the atomic gating functions state: gFiber,i,pB(ℱ()i−K(πE∗ℱ())i)=ℱ()i−K(πE∗ℱ())i,gBase,b(πE∗ℱ())=πE∗ℱ().g_ Fiber,i,p_B(F(1)_i-K( _E*F(1))_i)=F(1)_i-K( _E*F(1))_i, g_ Base,b( _E*F(1))= _E*F(1). We verify that G(ℱ())=ℱ()G(F(1))=F(1). The base component gives: K((⨁b∈BgBase,b)(πE∗ℱ()))=K(πE∗ℱ()).K\! ( ( _b∈ Bg_ Base,b )( _E*F(1)) )=K( _E*F(1)). The fiber component gives: (⨁i∈EgFiber,i,πE∗ℱ())(ℱ()−K(πE∗ℱ()))=ℱ()−K(πE∗ℱ()), ( _i∈ Eg_ Fiber,i, _E*F(1) ) (F(1)-K( _E*F(1)) )=F(1)-K( _E*F(1)), where the last equality uses the identity condition on gFiberg_ Fiber applied to each fiber component. Summing gives G(ℱ())=K(πE∗ℱ())+ℱ()−K(πE∗ℱ())=ℱ()G(F(1))=K( _E*F(1))+F(1)-K( _E*F(1))=F(1). Since ℛR is a left-inverse of ℱF, we obtain ℛ∘G∘ℱ()=R G (1)=1, and the surrogate reduces to the linear surrogate value J(θold)J( _ old). Gradient agreement. Taking the gradient with respect to θ at θ=θoldθ= _ old, we need to show that ∇θ(ℛ∘G∘ℱ)|θold=∇θ(ℛ∘ℱ)|θold _θ(R G )|_ _ old= _θ(R )|_ _ old. By the chain rule, it suffices to show that the Jacobian of G at the on-policy density ℱ()F(1) is the identity. The derivative conditions gFiber,i,pB′(ℱ()i−K(πE∗ℱ())i)=1g _ Fiber,i,p_B(F(1)_i-K( _E*F(1))_i)=1 and gBase,b′(πE∗ℱ())=1g _ Base,b( _E*F(1))=1 ensure that each atomic gating function has unit derivative at the on-policy point. Since G is composed of the base and fiber gating applied to the orthogonal decomposition σ=K(πE∗σ)+(σ−K(πE∗σ))σ=K( _E*σ)+(σ-K( _E*σ)), the Jacobian of G at ℱ()F(1) is the identity on both the base and fiber components. Therefore: ∇θ(ℛ∘G∘ℱ)|θold=∇θ(ℛ∘ℱ)|θold, _θ(R G ) |_ _ old= _θ(R ) |_ _ old, which recovers the true policy gradient ∇θJ(θ)|θ=θold _θJ(θ) |_θ= _ old. ∎ E.5 Derivation of FiberPO Recall the APC-Obj objective (in RGF form): J^APC-Obj(θ|θold)=1T∑(s,a,ℐ)∈ℰ[clip(rs,a−1,±(Tsδ(APC-Obj)−∑(s,a′,ℐ′)∈ℰ(s,a′,ℐ′)≠(s,a,ℐ),|rs,a′−1|))+1]A^s,a. J^APC-Obj(θ| _ old)= 1T _(s,a,I) [clip (r_s,a-1,\;± (T_sδ^(APC-Obj)-\!\!\!\! _ subarrayc(s,a ,I ) \\ (s,a ,I )≠(s,a,I), subarray\!\!\!\!|r_s,a -1| ) )+1 ] A_s,a. , where ℰ:=(s,a,ℐ)=¯:=(s,a,τ,t)E:=\(s,a,I)\= X:=\(s,a,τ,t)\ is the space of all sampled state–action pairs augmented with trajectory membership τ and time step t; each element is a distinct sampled token, so |¯|=T| X|=T and |¯s|=Ts| X_s|=T_s (Definition 3.2). The derivation begins from the APC-Obj objective (Section 3.2), which enforces a per-state TV budget Tsδ(APC-Obj)T_sδ^(APC-Obj) through cross-token interaction terms (∑(s,a′,ℐ′)∈ℰ(s,a′,ℐ′)≠(s,a,ℐ),|rs,a′−1|) ( _ subarrayc(s,a ,I ) \\ (s,a ,I )≠(s,a,I), subarray\!\!\!\!|r_s,a -1| ). We transform it into a form in which two scales of information are made explicit: the aggregate drift of a trajectory, measured by the positive and negative aggregate ratios sτ+,sτ−s_τ^+,s_τ^- (which reflect the trajectory-level TV distance, as established in Section 3), and the per-token ratio rs,ar_s,a (which captures how much each individual token has moved from the reference policy). Our goal is to use the APC-Obj objective as a reliable starting point, make reasonable relaxations, and arrive at the FBG form (Definition 4.3). The resulting objective can then inherit APC-Obj’s stability properties and the structural guarantees of the FBG form (first-order agreement, global-local decoupling, extensibility). Figure 5 illustrates this derivation roadmap. Figure 5: Derivation roadmap from APC-Obj to FiberPO. The APC-Obj objective uses a single clipping with cross-token interaction to enforce a per-state TV budget; the two scales of stability information (trajectory-level aggregate ratios sτ±s_τ^± and per-token ratios rs,ar_s,a) are implicitly entangled within this clipping. Through four successive relaxations, the key step (Derivation IV) decomposes the single clipping to make the two scales explicit, yielding separate global and local gates that fit naturally into the FBG form. The resulting FiberPO inherits trust-region stability from APC-Obj and structural guarantees (first-order agreement, global-local decoupling, information exchange, extensibility) from the FBG framework. E.5.1 Derivation I: δ-Relaxation Replace the vanishing δ(APC-Obj)δ^(APC-Obj) with a positive tunable hyperparameter δ. Since δ(APC-Obj)=δ(TRPO)=0δ^( APC-Obj)=δ^( TRPO)=0 at γ=1γ=1 (Theorem 2.1), this relaxation trades the exact TRPO trust-region guarantee for a controllable approximation: the resulting objective no longer enforces the classical bound, but δ directly quantifies the departure, and the TV-based stability structure is retained as a design guide (Remark 3.4). E.5.2 Derivation I: Logarithmic Approximation Apply the logarithmic approximation logri≈ri−1 r_i≈ r_i-1 and 1+clip(⋯)≈exp(clip(⋯))1+clip(·s)≈ (clip(·s)). This provides two benefits: 1. Positivity by construction. Policy ratios must satisfy r>0r>0. The map logr∈ℝ↦exp(⋅)>0 r (·)>0 guarantees this automatically: all subsequent gating operations—clipping, piecewise-linear gates gaggg agg, additions—can be designed as unconstrained operations on ℝR and exponentiated back, yielding valid positive ratios without explicit positivity constraints. This opens opportunity to many gating function design, not just a map from positive real line to positive real line. 2. Better algebraic properties for subsequent simplifications, in particular the additive decomposition of the clipping bound in Derivation IV. 3. Natural asymmetry. The common asymmetry of clipping parameters on ratios (where ϵ−<ϵ+ _-< _+) is naturally captured: replacing clip(r; 1−ϵ−, 1+ϵ+)clip(r;\,1- _-,\,1+ _+) by exp(clip(logr,±ϵ)) (clip( r,\,±ε)), the symmetric “log clipping” naturally exhibits this asymmetry. Under this approximation, the cross-token TV budget terms |rs,a′−1||r_s,a -1| in the APC-Obj objective (3) become |logrs,a′|| r_s,a |, so the per-state estimated TV divergence D^TV(s) D_ TV(s) is approximated by the mean absolute log-ratio, which we call the absolute log divergence. This identification is used in Derivation IV to interpret the clipping bound as a TV-budget constraint expressed in log space. The objective becomes: J^(θ|θold)=1T∑(s,a,ℐ)∈ℰexp(clip(logrs,a,±(Tsδ−∑(s,a′,ℐ′)∈ℰ(s,a′,ℐ′)≠(s,a,ℐ)|logrs,a′|)))A^s,a. J(θ| _ old)= 1T _(s,a,I) \! (clip ( r_s,a,\;± (T_sδ-\!\!\!\! _ subarrayc(s,a ,I ) \\ (s,a ,I )≠(s,a,I) subarray\!\!\!\!| r_s,a | ) ) ) A_s,a. E.5.3 Derivation I: Sequence-Level Aggregation Two changes restrict the cross-token coupling to within each trajectory. First, the summation inside the clip bound changes from all sampled tokens ∑(s,a′,ℐ′)∈ℰ,(a′,ℐ′)≠(a,ℐ) _(s,a ,I ) ,\,(a ,I )≠(a,I) to only those sharing the same trajectory ∑(s′,a′,ℐ′)∈ℰ,(a′,ℐ′)≠(a,ℐ),τ(ℐ′)=τ(ℐ) _(s ,a ,I ) ,\,(a ,I )≠(a,I),\,τ(I )=τ(I). Second, the per-state TV budget TsδT_sδ is replaced by a per-trajectory budget Tτ(ℐ)δT_τ(I)δ, distributing the total budget proportionally to trajectory length. The objective becomes: J^(θ|θold)=1|Tjθold|∑(s,a,ℐ)∈ℰ1Tτ(ℐ)×exp(clip(logrs,a,±(Tτ(ℐ)δ−∑(s′,a′,ℐ′)∈ℰ(s′,a′,ℐ′)≠(s,a,ℐ),τ(ℐ′)=τ(ℐ)|logrs′,a′|)))A^s,a, J(θ| _ old)= 1|Tj _ old| _(s,a,I) 1T_τ(I)\\ × \! (clip ( r_s,a,\;± (T_τ(I)δ-\!\!\!\! _ subarrayc(s ,a ,I ) \\ (s ,a ,I )≠(s,a,I),\;τ(I )=τ(I) subarray\!\!\!\!| r_s ,a | ) ) ) A_s,a, where τ(ℐ)τ(I) is the trajectory given by the extra information ℐI (see Definition 3.1) associated with the pair (s,a,ℐ)∈ℰ(s,a,I) , and Tτ(ℐ)T_τ(I) is its length. As with GRPO, the 1/Tτ(ℐ)1/T_τ(I) weighting also rewards shorter responses. E.5.4 Derivation IV: Clipping Decomposition into Fiber Bundle Gating Form This is the most important and distinctive derivation step compared to the derivation of PPO, GRPO, and GSPO. The goal is to decompose the current clipping, which enforces only a trajectory-level TV budget (∑(s′,a′,ℐ′)∈ℰ,τ(ℐ′)=τ(ℐ)|logrs′,a′|≤Tτ(ℐ)δ _(s ,a ,I ) ,\;τ(I )=τ(I)| r_s ,a |≤ T_τ(I)δ, i.e. the sum of token log-ratios within the same trajectory is bounded), into explicit local and global clipping. We focus on the clipping term inside the objective: clip(logrs,a,±(Tτ(ℐ)δ−∑(s′,a′,ℐ′)≠(s,a,ℐ)τ(ℐ′)=τ(ℐ)|logrs′,a′|)).clip ( r_s,a,\;± (T_τ(I)δ-\!\!\!\! _ subarrayc(s ,a ,I )≠(s,a,I)\\ τ(I )=τ(I) subarray\!\!\!\!| r_s ,a | ) ). Introducing aggregate ratios. Define the positive and negative (trajectory) aggregate ratios: log(sτ+):=1Tτ∑t=0Tτ−1max(log(rst(τ),at(τ)),0),log(sτ−):=1Tτ∑t=0Tτ−1max(log(rst(τ),at(τ)−1),0), (s^+_τ):= 1T_τ _t=0^T_τ-1 ( (r_s_t(τ),a_t(τ)),0), (s^-_τ):= 1T_τ _t=0^T_τ-1 ( (r^-1_s_t(τ),a_t(τ)),0), and denote ls,a:=sign(logrs,a)l_s,a:=sign( r_s,a). The clipping term can be re-expressed as: clip(logrs,a,±(Tτδ−Tτlogsτ+−Tτlogsτ−+|logrs,a|)).clip ( r_s,a,\;±(T_τδ-T_τ s^+_τ-T_τ s^-_τ+| r_s,a|) ). Negative skew of unsplit D¯^TV(τ) D_ TV^(τ) gating. Using a single, unsplit estimate of D¯^TV(τ) D_ TV^(τ) (Appendix A.3) as the global gating quantity introduces a systematic negative skew. Concretely, the absolute log divergence logsτ++logsτ−=(1/Tτ)∑t|logrst,at| s^+_τ+ s^-_τ=(1/T_τ) _t| r_s_t,a_t| approximates the trajectory-level TV distance in log-ratio coordinates (Derivation I, Section E.5.2). By definition, logsτ+,logsτ−≥0 s^+_τ, s^-_τ≥ 0, so their sum is always nonnegative. In the FBG framework (Definition 4.3), local-global decoupling subtracts the pushed-forward aggregate K(πE∗σ)K( _E*σ) from every fiber. When this aggregate is the unsplit (nonneg.) TV estimate, the subtraction removes a nonnegative quantity from every per-token log-ratio, imparting a systematic negative bias to all fiber residuals. The remedy is to split by sign into the positive channel sτ+s^+_τ and negative channel sτ−s^-_τ: a token with logrs,a>0 r_s,a>0 is then gated only by logsτ+ s^+_τ, and one with logrs,a<0 r_s,a<0 only by logsτ− s^-_τ, so the same-sign component is subtracted rather than the combined nonnegative total. Per-channel budgets and the C−<C+C^-<C^+ condition. After splitting, each channel is gated independently, eliminating the sign-mixing skew. However, an intrinsic asymmetry between the two channels remains. Observe that logsτ−logsτ+=−1Tτ∑t=0Tτ−1logrst(τ),at(τ), s^-_τ- s^+_τ\;=\;- 1T_τ _t=0^T_τ-1 r_s_t(τ),a_t(τ), which is a trajectory-averaged Monte Carlo estimate of the reverse KL divergence D¯KL(tj)(πθold∥πθ) D (tj)_ KL( _ _ old\| _θ). Since this quantity is nonnegative, we have logsτ−≥logsτ+ s^-_τ≥ s^+_τ in expectation, so the negative channel generically carries more aggregate drift. Imposing a symmetric budget δ/2δ/2 on each channel therefore insufficiently constrains the faster-drifting negative channel: because logs− s^- grows faster, it reaches the shared threshold δ/2δ/2 sooner and dominates the base weight wτbasew base_τ (Eq. 7), skewing the effective trust region toward the negative direction. To compensate, we decompose δ into per-channel budgets C+C^+ and C−C^- satisfying C++C−=δC^++C^-=δ, C+,C−>0C^+,C^->0, with the recommendation C−<C+C^-<C^+: the tighter negative budget ensures that the faster-drifting channel reaches rollback earlier, preventing the overall gated ratio (r∙)i=wτbase⋅r~ifiberG(r_ )_i=w base_τ· r_i fiber (Eq. 7) from being biased toward policy-ratio-decreasing updates. The symmetric case C+=C−=δ/2C^+=C^-=δ/2 is recovered as a special case. We decompose the above into a stricter clipping by clipping positive and negative log ratios separately: For logrs,a>0(ls,a=1): r_s,a>0\;(l_s,a=1): clip(logrs,a,±(TτC+−Tτlogsτ++logrs,a)), ( r_s,a,\;±(T_τC^+-T_τ s^+_τ+ r_s,a) ), For logrs,a<0(ls,a=−1): r_s,a<0\;(l_s,a=-1): clip(logrs,a,±(TτC−Tτlogsτ−logrs,a)). ( r_s,a,\;±(T_τC^--T_τ s^-_τ- r_s,a) ). Or equivalently: clip(logrs,a,±(TτC(ls,a)−Tτlogsτ(ls,a)+ls,alogrs,a)).clip ( r_s,a,\;±(T_τC^(l_s,a)-T_τ s^(l_s,a)_τ+l_s,a r_s,a) ). Decomposing aggregate and individual clipping. The channel-split clipping above still acts jointly on the aggregate and per-token ratios. To introduce local clipping behavior that constrains each token individually and to further structure the objective into the fiber and base components required by the FBG framework, we decompose the clipping of the aggregate ratio explicitly out of the clipping of individual policy ratios, yielding a stricter clipping (absolute value strictly less than the above), where ls,a:=sign(logrs,a)l_s,a:=sign( r_s,a): ls,aclip(ls,alogrs,a−logsτ(ls,a),±ϵ)+clip(ls,alogsτ(ls,a),±(TτC(ls,a)−Tτlogsτ(ls,a)−(ϵ−ls,alogrs,a))).l_s,aclip(l_s,a r_s,a- s_τ^(l_s,a),\,±ε)\;+\;clip (l_s,a s^(l_s,a)_τ,\;±(T_τC^(l_s,a)-T_τ s^(l_s,a)_τ-(ε-l_s,a r_s,a)) ). The decomposition naturally introduces an extra constant ϵε, representing the clipping bound for individual policy ratios. Now we are ready to define an intermediate object, object gaggg agg, which will be used to define gBaseg_ Base in FBG, the above is equivalent to: ls,aclip(ls,alogrs,a−logsτ(ls,a),±ϵ)+ls,agagg(logsτ(ls,a),TτC(ls,a)−(ϵ−ls,alogrs,a)Tτ+1,Tτ),l_s,a\,clip(l_s,a r_s,a- s_τ^(l_s,a),\,±ε)\;+\;l_s,a\,g agg\! ( s^(l_s,a)_τ,\; T_τC^(l_s,a)-(ε-l_s,a r_s,a)T_τ+1,\;T_τ ), (77) where gagg(x,C,k):=xif |x|≤Csign(x)(k+1)C−kxif C<|x|<(1+k−1)C0otherwiseg agg(x,C,k):= \ array[]lx&if |x|≤ C\\[5.0pt] sign(x)(k+1)C-kx&if C<|x|<(1+k^-1)C\\[5.0pt] 0&otherwise array . (78) Removing the nonnegative factor ls,alogrs,al_s,a r_s,a from gaggg agg, and noting that TτT_τ is typically large in the LLM context so that ϵTτ+1 εT_τ+1 quickly vanishes and TτC(ls,a)Tτ+1≈C(ls,a) T_τC^(l_s,a)T_τ+1≈ C^(l_s,a), we obtain the simplified clipping term: ls,aclip(ls,alogrs,a−logsτ(ls,a),±ϵ)+ls,agagg(logsτ(ls,a),C(ls,a),Tτ).l_s,a\,clip(l_s,a r_s,a- s_τ^(l_s,a),\,±ε)\;+\;l_s,a\,g agg\! ( s^(l_s,a)_τ,\;C^(l_s,a),\;T_τ ). (79) Remark E.8 (Three-regime structure of gaggg agg). The function gaggg agg that emerges from this decomposition has three regimes: pass-through when |logsτ(l)|≤C(l)| s_τ^(l)|≤ C^(l), linear rollback when C(l)<|logsτ(l)|<(1+Tτ−1)C(l)C^(l)<| s_τ^(l)|<(1+T_τ^-1)C^(l), and zeroing beyond that threshold. See Figure 2 for a visualization. Fiber Bundle Gating objects. The two terms in (79) correspond to fiber and base gating components respectively; we now identify the FBG objects that realize this decomposition. By defining E=¯×−1,+1E= X×\-1,+1\, (s,a,τ,t;l)∈E(s,a,τ,t;\,l)∈ E, ℱ(r∙)(s,a,τ,t;l):=1Tτmax(llogrs,a,0)=μ¯(τ)max(llogrs,a,0)∈(s,a,τ,t;l)1E≃ℝF(r_ )_(s,a,τ,t;\,l):= 1T_τ (l r_s,a,0)= _ X^(τ) (l r_s,a,0) ^1_(s,a,τ,t;\,l)E as local information, and B=Tjθold×−1,+1B=Tj _ old×\-1,+1\ with the projection πE:E→B _E:E→ B, πE(s,a,τ,t;l):=(τ,l) _E(s,a,τ,t;\,l):=(τ,l), one finds that logsτ(l)=πE∗ℱ(r∙)|(τ,l)∈B. s^(l)_τ= _E*F(r_ ) |_(τ,l)∈ B. The reflecting Markov kernel is defined as K(⋅|(τ,l)):=μ¯(τ)K(·|(τ,l)):= _ X^(τ) (see Appendix A.2), so that K((s,a,τ,t;l)|(τ,l)):=μ¯(τ)((s,a,τ,t))=1TτK((s,a,τ,t;\,l)|(τ,l)):= _ X^(τ)(\(s,a,τ,t)\)= 1T_τ. One can verify that πE∗K(pB)=pB=id(pB) _E*K(p_B)=p_B=id_B(p_B). The fibration recovery map is defined as ℛ(σ∙)(s,a,τ,t):=exp(Tτ(σ(s,a,τ,t;+1)−σ(s,a,τ,t;−1))).R( _ )_(s,a,τ,t):= (T_τ( _(s,a,τ,t;\,+1)- _(s,a,τ,t;\,-1)) ). The atomic gating functions are: gFiber,i,pB(σi):=clip(σi,±ϵTτ(i)),i∈E,gBase,(τ,l)(σ(τ,l)):=gagg(σ(τ,l),C(l),Tτ),(τ,l)∈B.g_ Fiber,i,p_B( _i):=clip\! ( _i,\,± εT_τ(i) ), i∈ E, g_ Base,(τ,l)( _(τ,l)):=g agg\! ( _(τ,l),\,C^(l),\,T_τ ), (τ,l)∈ B. Verification. The expression (79) fits exactly into the fiber bundle gating formalism: clip(ls,alogrs,a−logsτ(ls,a),±ϵ)+gagg(logsτ(ls,a),C(ls,a),Tτ) (l_s,a r_s,a- s_τ^(l_s,a),\,±ε)+g agg\! ( s^(l_s,a)_τ,\;C^(l_s,a),\;T_τ ) =gFiber,i,πE∗σ(σ−K(πE∗σ))i+K(gBase,(τ,ls,a)(πE∗σ))i =g_ Fiber,i, _E*σ (σ-K( _E*σ) )_i+K (g_ Base,(τ,l_s,a)( _E*σ) )_i ≡G(σ)i, ≡ G(σ)_i, where σ≡ℱ(r∙)∈σ (r_ ) and i≡(s,a,τ,t;ls,a)∈Ei≡(s,a,τ,t;\,l_s,a)∈ E. The clipping-decomposed objective is therefore in standard Fiber Bundle Gating form: J^(G;ℱ,ℛ)(θ|θold):=∑τ∈Tjπθold[∑t=0Tτ−11Tτℛ∘G∘ℱ(r∙)st,atA^st,at]. J_(G;F,R)(θ| _ old):= _τ _ _ old [ _t=0^T_τ-1 1T_τ\,R G (r_ )_s_t,a_t\; A_s_t,a_t ]. E.5.5 The FiberPO Objective By fully expanding the FBG form, we obtain the FiberPO objective. Definition E.9 (Fiber-Aware Clipping Policy Optimization (FiberPO) objective). In standard RGF form: J^FiberPO(θ|θold)=∑(s,a,τ,t)∈¯[1|Tjθold|1Tτ⋅exp∘gagg(logsτ+,C+,Tτ)exp∘gagg(logsτ−,C−,Tτ)×logclip(((sτ(ls,a))−ls,ars,a),ϵ)logclip(((sτ(−ls,a))−ls,a),ϵ)⋅A^s,a], J^FiberPO(θ| _ old)= _(s,a,τ,t)∈ X [ 1|Tj _ old| 1T_τ· \;g agg( s^+_τ,C^+,T_τ) \;g agg( s^-_τ,C^-,T_τ)\\ × logclip ( ((s_τ^(l_s,a))^-l_s,ar_s,a ),ε )logclip ( ((s_τ^(-l_s,a))^-l_s,a ),ε )· A_s,a ], (80) where: • ls,a:=sign(logrs,a)l_s,a:=sign( r_s,a). • log(sτ+):=1Tτ∑t=0Tτ−1max(log(rst(τ),at(τ)),0) (s^+_τ):= 1T_τ _t=0^T_τ-1 ( (r_s_t(τ),a_t(τ)),0), • log(sτ−):=1Tτ∑t=0Tτ−1max(log(rst(τ),at(τ)−1),0) (s^-_τ):= 1T_τ _t=0^T_τ-1 ( (r^-1_s_t(τ),a_t(τ)),0), • logclip(x,a):=exp(clip(log(x),±a))logclip(x,a):= (clip( (x),\,± a)), • gagg(x,C,k)g agg(x,C,k) is as defined in (78), • ℰ=¯E= X is the augmented state-action pair space (see Appendix A.2), • C+C^+ and C−C^- are the per-channel trust-region budgets with C++C−=δC^++C^-=δ; ϵε and δ are hyperparameters. E.6 FiberPO Objective in LLM Notation We restate the FiberPO objective (Definition 5.1) using standard LLM notation (g,j,i)(g,j,i), where g indexes the query group, j indexes the j-th trajectory within group g, and i indexes the i-th token within trajectory j. The translation from RL notation is τ,t↦g,j,i:=s0(τ),τ,tτ,t g,j,i:=s_0(τ),τ,t (see Appendix C.3). The FiberPO objective in LLM notation is: J^FiberPO(θ|θold)=g∼[1|Tjθold(g)|∑j∈Tjθold(g)1Tj(g)∑i=1Tj(g)exp∘gagg(logsj(g)+,C+,Tj(g))exp∘gagg(logsj(g)−,C−,Tj(g))×logclip(((sj(g,lj,i(g)))−lj,i(g)rj,i(g)),ϵ)logclip(((sj(g,−lj,i(g)))−lj,i(g)),ϵ)⋅A^j,i(g)], J^FiberPO(θ| _ old)=E_g D [ 1|Tj _ old(g)| _j _ old(g) 1T_j^(g) _i=1^T_j^(g) \;g agg( s^(g)+_j,C^+,T_j^(g)) \;g agg( s^(g)-_j,C^-,T_j^(g))\\ × logclip ( ((s_j^(g,\,l_j,i^(g)))^-l_j,i^(g)r_j,i^(g) ),ε )logclip ( ((s_j^(g,\,-l_j,i^(g)))^-l_j,i^(g) ),ε )· A_j,i^(g) ], (81) where lj,i(g):=sign(logrj,i(g))l_j,i^(g):=sign( r_j,i^(g)), and the constituent quantities are: • Positive and negative trajectory aggregate ratios: logsj(g)+:=1Tj(g)∑i=0Tj(g)−1max(logrj,i(g), 0),logsj(g)−:=1Tj(g)∑i=0Tj(g)−1max(−logrj,i(g), 0). s^(g)+_j:= 1T^(g)_j _i=0^T_j^(g)-1 ( r^(g)_j,i,\,0), s^(g)-_j:= 1T^(g)_j _i=0^T_j^(g)-1 (- r^(g)_j,i,\,0). • Log-clipping function: logclip(x,ϵ):=exp(clip(logx,±ϵ))logclip(x,ε):= (clip( x,\,±ε)). • Aggregate gating function: gagg(x,C,k)g agg(x,C,k) as defined in (10). • C+C^+ and C−C^- are the per-channel trust-region budgets with C++C−=δC^++C^-=δ. • Tj(g)T_j^(g) is the sequence length of the j-th trajectory in group g; ϵε and δ are hyperparameters. E.7 FiberPO-Domain: FGH Objects The FiberPO-Domain objective (30) can equivalently be written in the FGH gating form (29): J^FiberPO-Domain(θ|θold)=∑i∈¯1|Tjθold|1Tτℛ∘G∘ℱ(r∙)iA^i, J^FiberPO-Domain(θ| _ old)= _i∈ X 1|Tj _ old|\, 1T_τ\;R G (r_ )_i\; A_i, (82) where (r∙)i=wiBase⋅r~iFiber=ℛ∘G∘ℱ(r∙)iG(r_ )_i=w_i Base· r_i Fiber=R G (r_ )_i corresponds to the FGH case n=3n=3. The constituent FGH objects are: 1. Fibration hierarchy (n=3n=3). The four strata are: B0 B_0 :=Domain×−1,+1, :=Domain×\-1,+1\, B1 B_1 :=PromptGroup×−1,+1, :=PromptGroup×\-1,+1\, B2 B_2 :=Tjθold×−1,+1, :=Tj _ old×\-1,+1\, B3≡E B_3≡ E :=¯×−1,+1, := X×\-1,+1\, with fibrations π3(s,a,τ,t;l):=(τ,l) _3(s,a,τ,t;\,l):=(τ,l), π2(τ,l):=(gτ,l) _2(τ,l):=(g_τ,l), π1(g,l):=(Dg,l) _1(g,l):=(D_g,l). 2. Reflecting Markov kernels. K0(g,l∣D,l′) K_0(g,l D,l ) :=|Tjθold(g)||Tjθold(D)|g∈Dl=l′, := |Tj _ old(g)||Tj _ old(D)|\,I_g∈ D\;I_l=l , K1(τ,l∣g,l′) K_1(τ,l g,l ) :=1|Tjθold(g)|τ∈gl=l′, := 1|Tj _ old(g)|\,I_τ∈ g\;I_l=l , K2(i,l∣τ,l′) K_2(i,l τ,l ) :=1Tτi∈τl=l′. := 1T_τ\,I_i∈τ\;I_l=l . 3. Atomic gating functions. At the domain level (k=0k=0): g0,(D,l)(σ0):=|Tjθold(D)|gagg(1|Tjθold(D)|(σ0)(D,l),C(l),TD).g_0,(D,l)( _0):=|Tj _ old(D)|\;g agg\! ( 1|Tj _ old(D)|\,( _0)_(D,l),\;C^(l),\;T_D ). At the prompt group level (k=1k=1): g1,p<1,(g,l)(σ1):=|Tjθold(g)|gagg(1|Tjθold(g)|(σ1)(g,l),C(l),Tg).g_1,p_<1,(g,l)( _1):=|Tj _ old(g)|\;g agg\! ( 1|Tj _ old(g)|\,( _1)_(g,l),\;C^(l),\;T_g ). At the trajectory level (k=2k=2): g2,p<2,(τ,l)(σ2):=gagg((σ2)(τ,l),C(l),Tτ).g_2,p_<2,(τ,l)( _2):=g agg\! (( _2)_(τ,l),\;C^(l),\;T_τ ). At the token level (k=3k=3): g3,p<3,(i,l)(σ):=1Tτiclip(Tτiσ(i,l),ϵ).g_3,p_<3,(i,l)(σ):= 1T_ _i\,clip\! (T_ _i\, _(i,l),\;ε ). 4. Fibration decomposition and recovery. ℱ(r∙)i,l:=li=lTτililogriF(r_ )_i,l:= I_l_i=lT_ _i\,l_i r_i, ℛ(σ)i:=exp(Tτi(σi,+1−σi,−1))R(σ)_i:= \! (T_ _i\,( _i,+1- _i,-1) ). References [1] S. Kakade and J. Langford (2002) Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, p. 267–274. Cited by: §A.1, §2. [2] M. Kim, J. Kim, M. Choi, and J. Park (2022) Adaptive discount factor for deep reinforcement learning in continuing tasks with uncertainty. Sensors 22 (19), p. 7266. Cited by: §B.3, §2. [3] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, p. 27730–27744. Cited by: §B.3, §1. [4] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015) Trust region policy optimization. In International Conference on Machine Learning, p. 1889–1897. Cited by: §B.1, §B.1, §1, §2. [5] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §C.4, §1, §1, §2, §3.1, Remark 3.4. [6] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §C.4, §1, §1, §2, §3.1, Remark 3.4. [7] Y. Wang, H. He, and X. Tan (2020) Truly proximal policy optimization. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, p. 113–122. Cited by: §1, 2nd item. [8] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §1. [9] C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025) Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §C.4, §1, §2, §3.1, Remark 3.4, Remark 3.6.