Paper deep dive
Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training
Anas Barakat, Souradip Chakraborty, Khushbu Pahwa, Amrit Singh Bedi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 1:13:25 PM
Summary
This paper investigates why optimizing for Pass@k (probability that at least one of k samples is correct) can degrade Pass@1 (single-sample accuracy) in large language models. The authors identify 'prompt interference' as the root cause: Pass@k optimization implicitly reweights prompts toward those with low success probabilities. If these low-success prompts are 'negatively interfering' (i.e., their gradients conflict with the average Pass@1 gradient), upweighting them causes the overall Pass@k gradient to conflict with the Pass@1 gradient, leading to a decrease in single-shot accuracy despite improvements in multi-sample metrics.
Entities (7)
Relation Signals (5)
Pass@k Optimization → implicitlyreweights → Low-Success Prompts
confidence 95% · pass@k optimization implicitly reweights prompts toward low-success prompts
pass@k → isusedfor → Verifiable LLM Tasks
confidence 95% · Pass@k is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning.
Prompt Interference → leadsto → Gradient Conflict
confidence 92% · pass@k policy gradients can conflict with pass@1 gradients because pass@k optimization implicitly reweights prompts toward low-success prompts; when these prompts are what we term negatively interfering... can rotate the pass@k update direction away from the pass@1 direction.
Pass@k Optimization → causes → Pass@1 Degradation
confidence 90% · prior work reports a recurring trade-off: pass@k improves while pass@1 degrades under such methods.
Negatively Interfering Prompts → haveconflictinggradientswith → Pass@1 Gradient
confidence 90% · when these prompts are what we term negatively interfering, their upweighting can rotate the pass@k update direction away from the pass@1 direction.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Pass@k is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of $k$ independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@$k$. However, prior work reports a recurring trade-off: pass@k improves while pass@1 degrades under such methods. This trade-off is practically important because pass@1 often remains a hard operational constraint due to latency and cost budgets, imperfect verifier coverage, and the need for a reliable single-shot fallback. We study the origin of this trade-off and provide a theoretical characterization of when pass@k policy optimization can reduce pass@1 through gradient conflict induced by prompt interference. We show that pass@$k$ policy gradients can conflict with pass@1 gradients because pass@$k$ optimization implicitly reweights prompts toward low-success prompts; when these prompts are what we term negatively interfering, their upweighting can rotate the pass@k update direction away from the pass@1 direction. We illustrate our theoretical findings with large language model experiments on verifiable mathematical reasoning tasks.
Tags
Links
- Source: https://arxiv.org/abs/2602.21189v2
- Canonical: https://arxiv.org/abs/2602.21189v2
Trouble viewing inline? Open PDF directly →
Full Text
148,378 characters extracted from source content.
Expand or collapse full text
*Work does not relate to position at Amazon. Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training Anas Barakat1, Souradip Chakraborty2, Khushbu Pahwa*, Amrit Singh Bedi3 1Singapore University of Technology and Design 2University of Maryland, College Park 3University of Central Florida Abstract Pass@k is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of k independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@k. However, prior work reports a recurring trade-off: pass@k improves while pass@1 degrades under such methods. This trade-off is practically important because pass@1 often remains a hard operational constraint due to latency and cost budgets, imperfect verifier coverage, and the need for a reliable single-shot fallback. We study the origin of this trade-off and provide a theoretical characterization of when pass@k policy optimization can reduce pass@1 through gradient conflict induced by prompt interference. We show that pass@k policy gradients can conflict with pass@1 gradients because pass@k optimization implicitly reweights prompts toward low-success prompts; when these prompts are what we term negatively interfering, their upweighting can rotate the pass@k update direction away from the pass@1 direction. We illustrate our theoretical findings with large language model experiments on verifiable mathematical reasoning tasks. 1 Introduction Pass@k optimization. In many verifiable tasks, such as code generation and short-answer math, a system can afford multiple response attempts for the same prompt and check the correctness of each response attempt with an automatic verifier (e.g., unit tests, exact-match) that can check the correctness of generated answers. This naturally leads to pass@k inference: sample k independent candidate solutions and declare success if any candidate is correct. The corresponding metric, pass@k, measures the probability that at least one of k i.i.d. samples solves the prompt, and has become a standard performance measure when multiple attempts are available [chen-et-al21eval-llms-code, li-et-al22alphacode, gehring-et-al25rlef]. This “k attempts with verification” setting suggests that the training objective should align with deployment: if inference uses k attempts, training should directly optimize the same pass@k objective rather than optimizing only the single-sample expected reward. Recent work develops policy-gradient estimators and practical training procedures that optimize pass@k, and demonstrates large gains in pass@k performance on reasoning and code benchmarks [tang-et-al25icml, walder-karkhanis25neurips]. Trade-off: optimizing pass@k can reduce pass@1. A key complication is that pass@k optimization does not necessarily improve single-shot pass@1 performance. Empirically, some recent works report objective-dependent trade-offs where pass@k improves while pass@1 drops [tang-et-al25icml, walder-karkhanis25neurips]. For example, walder-karkhanis25neurips explicitly highlight a trade-off between pass@1 and pass@k during code-generation fine-tuning (e.g. Tables 1 to 4 in their paper), and their results show that optimizing a pass@k objective can lower pass@1 even as higher-k metrics increase substantially (Table 1). These regressions make pass@k optimization risky in settings where single-shot accuracy is a hard requirement. Why this degradation matters. Pass@1 remains operationally important: many deployments have limited latency or cost budgets, may not have reliable verifiers for every prompt, and often need a strong single-shot fallback even when retries are possible. In these regimes, a training procedure that improves pass@k at the cost of pass@1 can reduce overall system reliability and complicate model selection and deployment. Figure 1: (a) Empirical trade-off. Under pass@k policy optimization, pass@k increases while pass@11 may decrease. We explain this empirically observed trade-off in (b) and (c), which schematically illustrate the pass@11 and pass@k (k>1k>1) gradients for three prompts and their expectations in policy-parameter space. (b) Pass@1 gradients with negatively interfering prompt. This panel shows a setting in which prompt 33 is negatively interfering with prompts 11 and 22, i.e., the per-prompt pass@1 gradient for prompt 33 has negative inner product with the per-prompt pass@11 gradients for prompts 11 and 22 (see Section 3). Here, ∇ @11 denotes the population pass@11 gradient, given by the average (expectation over prompts; here under a uniform distribution) of the per-prompt pass@11 gradients. (c) Pass@k vs pass@1 gradient conflict. Per-prompt pass@k gradients are scaled versions of the corresponding per-prompt pass@1 gradients (Eq. (2)). This reweighting amplifies the magnitude of the pass@k gradient for the negatively interfering prompt 33, causing the resulting population pass@k gradient to conflict with the population pass@11 gradient (their inner product becomes negative, corresponding to an obtuse angle, shown here as approximately 120∘120 ). Consequently, a policy update in the pass@k gradient direction can increase pass@k while decreasing pass@1. Open question. Despite growing adoption of pass@k objectives, it is still not well understood why pass@k optimization can hurt pass@1, and when we should expect this trade-off to appear. Without a principled explanation, it is difficult to design reliable inference-aware fine-tuning methods that deliver multi-attempt gains while preserving strong single-shot performance. This leads to our research question: When and why can pass@k policy optimization degrade pass@1 performance? Our key insight. Compared to pass@1, optimizing the pass@k objective induces an implicit prompt reweighting toward prompts with lower success probability (i.e., prompts the current policy rarely solves). When these prompts contribute gradients that conflict with the population pass@1 gradient, upweighting them increases their influence on the pass@k policy gradient update. Consequently, the pass@k gradient can conflict with the pass@1 gradient direction. In this regime, pass@k improves under pass@k updates while pass@1 decreases (see Figure 1 for a schematic explanation). We identify the prompts that can lead to gradient conflict as negatively interfering using the concept of prompt interference we introduce in this work. Our main contributions are as follows: ∙ Prompt interference. We introduce the concept of prompt interference. We say that two given prompts are positively (resp. negatively) interfering if a policy parameter update which increases the probability of providing a correct response for that prompt tends to increase (resp. decrease) the probability of success of the other prompt. To capture the similarity between prompts in terms of pass@1 gradient representation, we introduce a similarity kernel to define the concept of prompt interference mathematically. The induced distribution over negatively interfering prompts is crucial to understand the pass@k and pass@1 trade-off. ∙ Gradient conflict characterization. We show that pass@k and pass@1 gradients can be conflicting in the sense that they can form an obtuse angle. This implies that a policy update following pass@k’s policy gradient tends to increase pass@k while decreasing pass@1. We provide a characterization of this gradient conflict by establishing an interpretable expression for the inner product between pass@k and pass@1. This characterization crucially uses the fact that per-prompt pass@k gradients are weighted pass@1 per-prompt policy gradients. The gradient conflict expression shows that gradient conflict occurs when negatively interfering prompt regions are upweighted by the implicit pass@k reweighting and hence dominate the contribution of positively interfering prompts. ∙ Sufficient conditions and influence of k. Using our gradient conflict characterization, we provide sufficient conditions under which gradient conflict occurs. We further study the influence of the parameter k and show that increasing k encourages gradient conflict under some conditions in the relative probability of success in negatively versus positively interfering prompts. ∙ Pass@1 degradation under pass@k updates. We prove that pass@1 decreases while pass@k increases (simultaneously) under one-step pass@k policy updates satisfying an explicit stepsize condition. To show this result, we establish and use smoothness of the pass@k objective under a standard policy parameter regularity assumption. ∙ Empirical validation. We illustrate our main findings on large language models for solving math reasoning questions. 2 Preliminaries In this section, we formally define the pass@k objective and recall its policy gradients as used in pass@k policy optimization. We consider a setting where a language model is represented as a stochastic policy πθ _θ parameterized by weights θ∈ℝdθ ^d over a set of responses Y given a set of prompts denoted by X (e.g., math questions). The conditional probability distribution of the model’s output responses when prompted with x is πθ(⋅|x). _θ(·|x). We denote by D the distribution over prompts. Pass@k objective. Given a prompt x sampled from a distribution D, a single ‘attempt’ is a response y∼πθ(⋅|x).y _θ(·|x). We suppose we have access to a binary reward verifier r(x,y)∈0,1r(x,y)∈\0,1\ which encodes correctness of response y for prompt x. The per-prompt success probability for any x∈x and any policy parameter θ∈ℝdθ ^d will be denoted: pθ(x):=y∼πθ(⋅|x)[r(x,y)]=ℙy∼πθ(⋅∣x)(r(x,y)=1),p_θ(x):=E_y _θ(·|x)[r(x,y)]=P_y _θ(· x)(r(x,y)=1)\,, (1) where the last identity follows from the fact that rewards are binary (r(x,y)∈0,1r(x,y)∈\0,1\). The pass@k objective is defined as the probability that at least one response is correct among k responses, i.e., Jk(θ):=ℙ(max1≤i≤kr(x,yi)=1)=x∼[1−(1−pθ(x))k],J_k(θ):=P ( _1≤ i≤ kr(x,y_i)=1 )=E_x [1-(1-p_θ(x))^k]\,, (pass@k) where x∼,yi∼πθ(⋅|x)x ,y_i _θ(·|x) for i=1,⋯ki=1,·s k are k independent response samples. Hence pass@k can be written using the nonlinear transformation fk(p):=1−(1−p)kf_k(p):=1-(1-p)^k as Jk(θ)=x∼[fk(pθ(x))].J_k(θ)=E_x [f_k(p_θ(x))]\,. It immediately follows that pass@k is an increasing function in k: ∀m≥k≥1,∀θ∈ℝd,Jm(θ)≥Jk(θ).∀ m≥ k≥ 1,∀θ ^d,J_m(θ)≥ J_k(θ)\,. Moreover, pass@k can be bounded by pass@1 using Jensen’s inequality and concavity of fkf_k w.r.t. its argument: for all k≥1,θ∈ℝdk≥ 1,θ ^d, J1(θ)≤Jk(θ)≤min(1,fk(J1(θ))).J_1(θ)≤ J_k(θ)≤ (1,f_k(J_1(θ)) )\,. Note that under our notation J1(θ)=x∼[pθ(x)].J_1(θ)=E_x [p_θ(x)]. Pass@k policy gradients. Pass@k optimization can be performed using pass@k policy gradients. It follows from the chain rule that the gradient of pass@k w.r.t. the policy parameters is given by: ∇Jk(θ)=x∼[wk(pθ(x))∇pθ(x)],wk(p):=k(1−p)k−1,∇ J_k(θ)=E_x [w_k(p_θ(x))∇ p_θ(x)]\,,w_k(p):=k(1-p)^k-1, (2) as noted in a few recent works [mahdavi-et-al25pg-pass@k, yu25]. Pass@k gradients reweight prompts by wk,θ(x):=wk(pθ(x))=k(1−pθ(x))k−1.w_k,θ(x):=w_k(p_θ(x))=k(1-p_θ(x))^k-1\,. (3) This weighting emphasizes low-probability of success prompts and downweights prompts with high probability of success. To compute pass@k gradients in (2), note that the gradient ∇pθ(x)∇ p_θ(x) of the prompt-wise probability of success for any prompt x∈x and any policy parameter θ∈ℝdθ ^d immediately follows from (1) and is given by: ∇pθ(x)=y∼πθ(⋅|x)[r(x,y)sθ(x,y)],∇ p_θ(x)=E_y _θ(·|x)[r(x,y)s_θ(x,y)]\,, (4) where sθ(x,y):=∇logπθ(y|x)s_θ(x,y):=∇ _θ(y|x) denotes the score function. This gradient expression has been recently used in tang-et-al25icml, walder-karkhanis25neurips for instance. Per-prompt pass@k and pass@1 are positively collinear as ∇Jk(x;θ):=wk,θ(x)∇pθ(x)=wk,θ(x)∇J1(x;θ)∇ J_k(x;θ):=w_k,θ(x)∇ p_θ(x)=w_k,θ(x)∇ J_1(x;θ) and wk,θ(x)w_k,θ(x). However we will crucially see that pass@k and pass@1 as expectation over prompts will not necessarily be positively collinear. This discrepancy which was overlooked in the literature highlights the importance of the expectation over prompts and the structure of per-prompt pass@1 gradients. This motivates our next main section in which we introduce the concept of prompt interference capturing the potential conflict between the diversity of prompts in the underlying prompt distribution and the shared policy parameters across prompts. 3 Prompt Interference In this section, we introduce the phenomenon of prompt interference which will play an important role in explaining later in the paper why pass@1 can degrade under pass@k policy updates. This section is focused on understanding the structure of pass@1 policy gradients over the set of prompts. The role of the pass@k objective and its influence on optimization will be discussed in section 4. 3.1 Prompt similarity measure and interference The pass@1 objective is defined as the expectation of the per-prompt probability of success with respect to the distribution of prompts (see pass@k with k=1k=1). As a function of policy parameters, this pass@1 objective induces a shared-parameter coupling across prompts: the same policy parameters are shared across the entire distribution of prompts which might be diverse. As a consequence, a (shared) policy parameter update can improve the probability of success (pθ(x)p_θ(x)) on some prompts while it can degrade it on other prompts. When this negative transfer occurs, we say that there is negative prompt interference. We provide a formal definition of this phenomenon in the rest of this section. To define prompt interference, we investigate the structure of the pass@1 policy gradient, looking at the per-prompt pass@1 policy gradients and comparing them across prompts. We define the following natural prompt gradient similarity measure κθ:×→ℝ _θ:X×X defined for any policy parameter θ∈ℝdθ ^d and any prompts x,x′∈x,x by: κθ(x,x′):=⟨∇pθ(x),∇pθ(x′)⟩. _θ(x,x ):= ∇ p_θ(x),∇ p_θ(x ) \,. (5) This kernel captures similarities between prompts in terms of their pass@1 gradients which encode their ability to improve the probability of success. Using the policy gradient expressions of the gradients of pass@1 in (4), we obtain: κθ(x,x′)=y∼πθ(⋅|x)y′∼πθ(⋅|x′) [δ(x,y,x’,y’) ⟨s_θ(x,y), s_θ(x’,y’) ⟩], where δ(x,y,x′,y′):=r(x,y)r(x′,y′) which is equal to 1 if and only if both y,y′ are correct completions for x,x′ respectively (as rewards are binary). The kernel informs on whether improving pass@1 on one prompt tends to also improve pass@1 on another prompt under the shared policy parameters. To see this, consider a policy update θ′=θ+η∇pθ(x) with a small stepsize η>0 in the direction ∇pθ(x) improving the probability of success on x. A first-order Taylor expansion of θ↦pθ(x′) gives: pθ+(x′) ≃+pθ(x′)η⟨∇pθ(x′),∇pθ(x)⟩=+pθ(x′)ηκθ(x′,x). This leads us to the following prompt interference definition. Definition 3.1 (Prompt interference). We will say that two prompts x and x′ are interfering positively if κθ(x′,x)>0 where the similarity kernel κθ is defined in (5) (or (48)): a step that increases the per-prompt pass@1 pθ(x) at x also tends to increase pθ(x′). Otherwise, if κθ(x′,x)<0, the two prompts will be said to interfer negatively and a step that increases pθ(x) tends to decrease pθ(x′). We remark that our definition is similar to the definition of interference in schaul-et-al19ray-interference in the context of multi-task learning. See also yu-et-al20gradient-surgery-mtl who also discuss gradient interference between the gradients of different task loss function gradients in the same multi-task learning context. Recently, in the context of LLM post-training, qu-et-al26pope invoked ray interference as introduced in schaul-et-al19ray-interference to explain why learning can be inhibited on heterogeneous prompt mixtures and why exploration on hard problems can be hurt. We refer the reader to Section 6 for further discussion regarding the concept of interference and prior work. In the next section, we illustrate the prompt interference concept on a toy example. 3.2 Toy example Setting. We consider a minimal contextual-bandit abstraction of verifier-based multi-sample LLM evaluation as a running example. A prompt is a context x∈ drawn from a mixture distribution with a latent difficulty label ℓ(x)∈easy,hard with a fixed proportion of hard labels: ℙ(ℓ(x)=hard)=phard. Each prompt is represented by a 2D feature vector ψ(x)=[1,s(x)]⊤∈ℝ2 with a bias 1 and a scalar feature s(x)∈ℝ. Conditioned on difficulty, the scalar feature follows overlapping Gaussians whose separation is controlled by a parameter δ>0: if ℓ(x)=easy then s(x)∼(−δ/2, 1), and if ℓ(x)=hard then s(x)∼(+δ/2, 1). Smaller δ yields greater overlap between easy and hard feature distributions. A completion is represented as a discrete action y∈0,1. Each prompt has a unique correct completion determined by its difficulty: y⋆(x)=0 if ℓ(x)=easy and y⋆(x)=1 if ℓ(x)=hard. The verifier reward is deterministic binary: r(x,y)=y=y⋆(x). We use a two-action softmax policy parameterized by a single vector θ∈ℝ2 such that πθ(1|x)=σ(θ⊤ψ(x)) and πθ(0|x)=1−σ(θ⊤ψ(x)), where σ(t)=11+exp(−t) is the sigmoid function. The policy shares parameters across heterogeneous subpopulations (easy vs hard). For later use, we also note that we initialize the policy at a reference policy θref that is globally biased toward action y=0, inducing high initial success on easy prompts and low initial success on hard prompts. Illustration of negative prompt interference. It follows that the per-prompt success probability defined in (1) is given by pθ(x)=πθ(y⋆(x)|x) for any prompt x∈. A simple computation yields: =∇pθ(x)-zθ(x)ψ(x),=if ℓ(x)easy,+zθ(x)ψ(x),=if ℓ(x)hard, (7) where zθ(x):=σ(θ⊤ψ(x))(1−σ(θ⊤ψ(x)))≥0. As a consequence, two prompts that have a similar representation will have opposite per-prompt pass@1 gradients and will hence be negatively interfering. This will be the case in our example for prompts in the overlap between the two Gaussians: two such prompts can have different labels (hard and easy) while having a similar prompt representation (with a similar scalar feature s(x)). See Figure 2 below for an illustration. Figure 2: Cosine kernel heatmap: cos(∇pθ(x),∇pθ(x′)) for subsamples of prompts: 120 easy and 80 hard among a total of 6000 samples. Blue regions correspond to negative prompt interference. In the next section, we show in our toy example how negative prompt interference can lead to pass@1 decrease under one-step pass@k policy gradient ascent. 3.3 Tradeoff: pass@k ascent can decrease pass@1 Using the toy example introduced in the previous section, we show in this section that pass@1 can decrease under one-step pass@ k policy gradient ascent. Fix δ=0.2 and consider two realized prompts in the overlap region between the 2 Gaussians: a prompt xe with ℓ(xe)=easy and ψ(xe)=[1,−0.1]⊤, and xh with ℓ(xh)=hard and ψ(xh)=[1,+0.1]⊤. These two prompts have nearly identical representations as cos(ψ(xe),ψ(xh))≃0.98 is close to 1. Because of (7), they are therefore negative interfering as κθ(xe,xh)=⟨∇pθ(xe),∇pθ(xh)⟩=−zθ(xe)zθ(xh)⟨ψ(xe),ψ(xh)⟩<0 since ⟨ψ(xe),ψ(xh)⟩>0. Numerically we have κθ(xe,xh)≃−0.01 and cos(∇pθ(xe),∇pθ(xh))≃−0.98 which means that the per-prompt pass@1 gradients are almost antiparallel. On this two-point empirical distribution, the pass@1 and pass@k gradients are given by: ∇J1(θ) =12(+∇pθ(xe)∇pθ(xh)), ∇J10(θ) =12(+we∇pθ(xe)wh∇pθ(xh)), where the pass@10 weight we:=w10,θ(xe)≃2.33×10−7 whereas wh:=w10,θ(xh)≃3.88. Pass@10 upweights the hard prompt (pθ(xh)≃0.10 low) and aggressively downweights the easy prompt xe (large pθ(xh)≃0.86). Indeed, recall that wk,θ(x)=k(1−pθ(x))k−1 which is near k when pθ(x)≈0 but and near 0 when pθ(x)≈1. As a consequence, the pass@10 gradient is therefore dominated by ∇pθ(xh) which conflicts with ∇pθ(xe). Importantly, we have cos(∇J1(θ),∇J10(θ))=−0.77<0 meaning that the pass@1 and pass@10 are conflicting and strongly anti-aligned. With one gradient-ascent step on pass@10 θ+=θ+η∇J10(θ) with η=5, we have: J1(θ) ≃0.48→J1(θ+)≃0.46(↓), J10(θ) ≃0.83→J10(θ+)=0.95(↑). This shows that pass@1 decreases while pass@k increases under pass@k optimization. In our toy example, we have observed that negative prompt interference can induce a conflict between pass@1 and pass@k gradients when it is amplified by the induced pass@k prompt reweighting. In the next section, we go beyond our toy example to characterize this conflict between pass@k and pass@1 gradients in the general setting. 4 Pass@k and Pass@1 Gradients Conflict In this section, we exploit the observation that pass@ k policy optimization induces an implicit prompt reweighting, biased toward upweighting low-success prompts. To connect with the previous section, if this reweighting coincides with upweighting negatively interfering prompts, then pass@k policy updates may result in conflicting pass@k and pass@1 gradients, leading to pass@1 degradation under pass@k policy optimization updates. We make this chain of arguments precise in this section. 4.1 Gradient conflict: Reweighting meets interference To explain why optimizing pass@ k can hurt pass@1, we study a key quantity: the angle between the policy gradients of pass@k and pass@1 respectively. We show that this angle can be obtuse and we refer to this setting as a case of conflicting gradients. This observation might seem a little surprising from the theory viewpoint at first glance. Indeed, the per-prompt gradient ∇Jk(x;θ) of pass@k is always a positive multiple of the per-prompt policy gradient ∇J1(x;θ) of pass@1 since ∇Jk(x;θ)=wk,θ(x)∇J1(x;θ) and wk,θ(x)≥0 for any prompt x∈. Nevertheless, the population gradients ∇Jk(θ)=[∇Jk(x;θ)] and ∇J1(θ)=[∇J1(x;θ)] need not form an acute angle as their inner product can be negative as we will see. The following proposition provides an explicit expression of this inner product to characterize the gradients conflict. To state this result, we introduce the prompt-wise gradient agreement score for pass@1 defined for any policy parameter θ∈ℝd and any prompt x∈ as follows: aθ(x):=⟨∇J1(x;θ),∇J1(θ)⟩=E∼x′D[κθ(x,x′)]. (8) This agreement score quantifies the prompt-level gradient interference with the average pass@1 objective. If aθ(x)>0, that prompt’s pass@1 gradient points in a direction that also improves average pass@1. If aθ(x)<0, it is anti-aligned and improving that prompt’s pass@1 tends to decrease population pass@1 under the shared policy parameters θ. Using this agreement score, our characterization is as follows. Proposition 4.1 (Gradients conflict characterization). For any k≥1 and any θ∈ℝd, ⟨∇Jk(θ),∇J1(θ)⟩ =x∼[wk,θ(x)aθ(x)] (9) =[wk,θ(x)]⋅‖∇J1(θ)‖2+cov(wk,θ(x),aθ(x)). (10) In particular, ⟨∇Jk(θ),∇J1(θ)⟩<0 is equivalent to each one of the following conditions: (i) x∼[wk,θ(x)aθ(x)]<0, (i) cov(wk,θ(x),aθ(x))<−[wk,θ(x)]⋅‖∇J1(θ)‖2. Discussion of conditions. We discuss each condition and provide two interpretations explaining gradients conflict and we refer the reader to figures 3 and 4 for illustrations on the toy example of section 3.2: (i) Prompt distribution shift: pass@k reweights prompts. Recalling that the weights wk,θ(x) are nonnegative, define the reweighted prompt distribution: ∝~Dk,θ(dx)wk,θ(x)D(dx), inducing the expectation ~k,θ[f(x)]=[wk,θ(x)f(x)][wk,θ(x)] for any measurable function f of the prompt random variable x. Note that the distribution ~k,θ places higher mass on prompts with smaller success probability pθ(x) (hard prompts) since wk,θ(x) is decreasing in pθ(x). Therefore, we can rewrite (9) as follows: =⟨∇Jk(θ),∇J1(θ)⟩⋅ED[wk,θ(x)]E~Dk,θ[aθ(x)], (11) and since [wk,θ(x)]>0 (as wk,θ(x)≥0 and not equal to zero a.e.), we obtain the following equivalence between ⟨∇Jk(θ),∇J1(θ)⟩<0 and ~k,θ[aθ(x)]<0. This characterization shows that conflict of gradients of pass@1 and pass@k occurs if and only if the average of the agreement score aθ(x) under the pass@k induced prompt distribution ~k,θ is negative. In contrast note that [aθ(x)]=∥∇J1(θ)∥2≥0. Pass@k changes the prompt weighting from to ~k,θ and the average agreement under this shifted distribution can become negative. In summary, we obtain the following causal chain: pass@k induces a prompt reweighting toward harder prompts (i.e. lower pθ(x) since wk,θ(x) decreases in pθ(x)). If hard prompts have negative agreement score aθ(x) (i.e. per-prompt pass@1 gradients are conflicting with the pass@1 gradient) then the reweighted average agreement becomes negative and the gradient of pass@k and pass@1 form an obtuse angle as a consequence, i.e. they are conflicting. Figure 3: Contour plots of pass@1 and pass@k objectives in the policy parameter space. Gradients of pass@k and pass@1 with respect to policy parameters are conflicting in the gray area. Figure 4: Agreement score aθ(x) (see (8)) on the toy example. The figure shows that the prompt distribution has both prompts with negative and positive agreement scores. Prompt with negative agreement scores are responsible for gradient conflict and hence pass@1 decrease. After pass@k policy optimization, negative agreement scores get even more negative. (i) Covariance between weights and agreement scores. The covariance cov(wk,θ(x),aθ(x)) captures the coupling between ‘how much pass@k emphasizes a prompt’ (via weight wk,θ(x)) and ‘whether that prompt helps pass@1’ (via alignment score aθ(x)). The covariance term quantifies whether pass@k’s hardness reweighting concentrates gradient mass on prompts whose success-probability gradients are anti-aligned with the pass@1 gradient. Conflict happens when this anti-aligned hard-prompt mass overwhelms the positive term. See Appendix A for more details. Remark 4.2. For a more general discussion of the conflict between pass@k and pass@m gradients for any ≥k,m1 using our kernel viewpoint, see Appendix B. 4.2 Sufficient conditions and threshold in k We now provide sufficient conditions for pass@ k and pass@1 gradients conflict, as a corollary of Proposition 4.1. Before stating this result, we make a standard smoothness assumption on the policy parameterization. Assumption 4.3 (Lipschitz and smooth policy). There exist positive constants G,F s.t. for all prompts ∈xX, we have the expected score bounds: ≤E∼yπθ(⋅|x)[∥∇logπθ(|yx)∥2]G2,≤E∼yπθ(⋅|x)[‖∇2logπθ(|yx)‖]F. (12) This assumption is satisfied for instance by Gaussian policies with bounded state feature vectors and softmax tabular policies, see e.g. Lemma 4.4, 4.8 and corollary 4.3 in yuan-et-al22vanilla-pg. This is a standard assumption in several works on the convergence analysis of policy gradient methods, see, e.g., yuan-et-al22vanilla-pg, papini-et-al22smoothing. To quantify the magnitude of negative interference and its prevalence under the distribution , we define the m-strongly negatively interfering prompt set with margin m>0: :=X-(θ,m)∈xX:≤aθ(x)-m, (13) and denote by qθ:=(−(θ,m))∈[0,1] its probability mass under the distribution of prompts. Then, the expected pass@k weight contributions inside and outside the m-strongly interfering set −(θ,m) are given by: W-(k,θ) :=E[wk,θ(x)1∈xX-(θ,m)], (14) W+(k,θ) :=E[wk,θ(x)1∉xX-(θ,m)]. (15) The following corollary of Proposition 4.1 shows that gradients of pass@1 and pass@k conflict when the contribution of pass@k weights in negatively interfering prompt regions dominates its counterpart for positively interfering prompts. Corollary 4.4 (Dominating negatively interfering prompts). Let Assumption 4.3 hold. Suppose that qθ>0 and let k≥2. Then for any θ∈ℝd, ⟨∇Jk(θ),∇J1(θ)⟩≤−δ(θ) where δ(θ):=mW−(k,θ)−G2W+(k,θ). If in addition δ(θ)<0, then ⟨∇Jk(θ),∇J1(θ)⟩≤−δ(θ)<0. Discussion of assumption. Note that the assumption qθ>0 is necessary for gradient conflict. Indeed, if instead we have for instance for almost every x∈,aθ(x)≥0, then it follows from (9) that ⟨∇Jk(θ),∇J1(θ)⟩≥0 as the weights wk,θ(x) are also always nonnegative. Negative agreement score is a necessary condition for gradient conflict. The gradient conflict condition of Corollary 4.4 depends on the relative weights contribution ratio W−(k,θ)/W+(k,θ) in negative vs positive prompt agreement score regions. This ratio is a function of both the chosen parameter k and the probability of success pθ(x) (1) under policy πθ. We now exploit the closed form of the pass@k weights wk,θ(x):=k(1−pθ(x))k−1 to evaluate the ratio under assumptions on the prompt-wise probabilities of success. The next result shows that beyond a threshold k⋆, the pass@k and pass@1 gradients form an obtuse angle. We make a separation assumption on the probability of success over prompts: the probability of success for m-strongly interfering prompts is supposed to be uniformly smaller than the probability of success of the remaining prompts. Intuitively, this boils down to assume that negatively interfering prompts are relatively harder to solve than the rest of the prompts. Proposition 4.5 (Influence of k). In the setting of corollary 4.4, assume there exist ε,δ,q>0 s.t. supx∈−(m,θ)pθ(x)≤ε<δ≤infx∉−(m,θ)pθ(x), (16) and qθ≥q for any θ, then ⟨∇Jk(θ),∇J1(θ)⟩≤−δ where δ=k[(1−ε)k−1mq−(1−δ)k−1G2(1−q)]. If in addition k>k⋆:=1+log((1−q)G2qm)log(1−ε1−δ), then ⟨∇Jk(θ),∇J1(θ)⟩<0. This result shows there is a phase transition in k: if k is large enough then gradients of pass@k and pass@1 will be conflicting and hence pass@1 will degrade while pass@k will improve under a small step size policy update following pass@k’s gradient. The threshold k⋆ becomes smaller when the probability mass of the negatively interfering region is larger. This means that under larger negative prompt interference, the conflict between pass@1 and pass@k becomes more severe as it occurs for even smaller k. 4.3 Pass@1 degradation under pass@k updates In this section, we show and quantify pass@1 decrease and pass@ k increase under pass@k policy optimization, exploiting the conflict between pass@k and pass@1 gradients shown in the previous sections. To quantify improvement and degradation under policy updates, we prove smoothness of the pass@k objective for any k≥1 under Assumption 4.3 (see Lemma C.1 in appendix C.4). Using this smoothness property, we show that gradient conflict results in provable pass@1 decrease under pass@k optimization. Proposition 4.6 (Pass@1 decrease under pass@k updates). In the setting of corollary 4.4, set θ+=θ+η∇Jk(θ) where k≥2 and η>0. If η≤min(δ(θ)C2,1Lk) where C2:=k2G2L1/2 then J1(θ+) <J1(θ)−ηδ(θ)+C2η2<J1(θ), (17) Jk(θ+) ≥Jk(θ)+η2‖∇Jk(θ)‖2≥Jk(θ). (18) The stepsize condition guarantees that the stepsize is small enough to ensure both decrease of pass@1 and increase of pass@k using the different smoothness constants of pass@k and pass@1 compared to the gradient conflict strength δ(θ). See Figure 5 for an illustration over multiple iterations. Figure 5: Pass@5 vs pass@1 in example of section 3.2 with pass@5 policy optimization, ‘pop’ refers to ‘population’ i.e. pass@k as defined in (pass@k), ‘easy’ and ‘hard’ mean expectation is only taken over prompts labeled as easy respectively hard. Pass@5 increases while pass@1 decreases. 5 Experiments We empirically test whether the pass@ k objective can induce gradient conflict with pass@1 on math reasoning, as predicted by Proposition 4.1. Dataset and models. We use the MATH dataset [hendrycksmath2021], which contains competition-style high school math problems spanning seven subjects (Algebra, Counting & Probability, Geometry, Intermediate Algebra, Number Theory, Precalculus, and Prealgebra). We randomly sample 2,000 problems for evaluation. We run experiments with two reasoning models: DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B. Pass@k gradient computation. For computational efficiency, we compute pass@1 gradients with respect to policy parameters in the language model’s final hidden layer (dimension d=4096 for Llama-8B, d=3584 for Qwen-7B). We compute pass@k gradients using Monte Carlo estimates based on (2) and pass@k estimates developed in prior work [walder-karkhanis25neurips, chen-et-al21eval-llms-code]. Setup. We create filtered data sets δ1,δ2 of prompts and responses with varying difficulty thresholds (δ1,δ2) consisting of (i) hard prompts (pθ(xi)<δ2) and (i) easy prompts (pθ(xi)>δ1). We test 7 combinations with δ1∈0.80,0.85,0.90 and δ2∈0.05,0.10,0.15. For each combination, we compute estimates of the agreement scores aθ(x) as defined in (8), pass@k weights wk,θ(x) (see (3)) and estimated average weighted agreement scores over prompts, which correspond to pass@k and pass@1 gradient inner product as shown in Proposition 4.1. See Appendix D.2 for all threshold configurations. (a) DeepSeek-R1-Distill-Llama-8B: =δ10.85, =δ20.10 (550 prompts: 86 hard, 464 easy, ratio 5.4:1) (b) DeepSeek-R1-Distill-Qwen-7B: =δ10.85, =δ20.05 (656 prompts: 29 hard, 627 easy, ratio 21.6:1) Figure 6: Pass@k optimization hurts pass@1 performance: Gradient conflict across two models. Both models demonstrate how pass@k reweighting causes gradient conflict with pass@1, despite different threshold settings. Agreement scores are computed relative to the pass@1 gradient of filtered prompts. Left column (A): Agreement scores show separation between hard negatives (red) and easy positives (green), with unweighted means shown as blue dashed lines. Middle column (B): Pass@k weights grouped by pass@1 probability, showing extreme weight disparity (∼1028:1) between hard and easy prompts. Right column (C): Weighted contributions (estimates of pass@k vs pass@1 inner products) demonstrate how pass@k reweighting hurts pass@1. Purple arrows highlight the critical downward shift from the unweighted mean (blue dotted, same as plot A) to the weighted mean (red dashed). (a) Deepseek-Llama: The downward shift from +×2.8010-3 to -×1.1210-3 (=Δ-×3.9210-3) flips the gradient from positive to negative, resulting in inner product −0.613. (b) DeepSeek-Qwen: The dramatic downward shift from +×2.9710-2 to -×2.7410-1 (=Δ-×3.0410-1) demonstrates even stronger gradient conflict. Despite only 29 hard prompts vs 627 easy (21.6:1 ratio), the extreme weight disparity causes hard negatives to completely dominate, resulting in inner product −181. Discussion. Our experiments validate the theoretical predictions in several ways. First, the agreement scores (column A in Figure 6) show clear separation between hard prompts (red points, negative agreement scores clustered below zero) and easy prompts (green points, positive agreement scores above zero), confirming that prompt interference exists in practice. The unweighted mean agreement (blue dashed line) is positive, indicating that under uniform weighting, the population gradient would improve pass@1. Second, the pass@k weights (column B in Figure 6) reveal extreme disparity in how pass@k values different prompts. The histogram shows that hard prompts (red bars, low pass@1 values ≲0.1) receive weights of n=38 at scale 12, n=21 at scale 4, and n=27 at scale 1, while easy prompts (green bars, high pass@1 values ≳0.85) receive negligible weights near 10−28. This creates weight ratios exceeding 1028:1, demonstrating the extreme reweighting mechanism our theory identifies: pass@k dramatically amplifies the influence of low-success prompts while effectively ignoring high-success prompts. Third, column C demonstrates the consequence of this reweighting through weighted agreement scores. The purple arrows in panel C highlight the critical downward shift from the unweighted mean (blue dotted line, matching column A) to the weighted mean (red dashed line). For Llama-8B (Figure 6(a), this shift is Δ=−3.92×10−3, moving from +2.80×10−3 to −1.12×10−3 and flipping the gradient alignment from positive to negative, resulting in an inner product of −0.613. For Qwen-7B (Figure 6(b)), despite having only 29 hard prompts versus 627 easy prompts (a 21.6:1 ratio heavily favoring easy prompts), the extreme weight disparity causes an even more dramatic shift of Δ=−3.04×10−1, moving from +2.97×10−2 to −2.74×10−1, yielding a strongly negative inner product of −181. This validates Proposition 4.1: when the negative covariance between pass@k weights wk,θ(x) and agreement scores aθ(x) dominates the positive term [wk,θ(x)]⋅∥∇J1(θ)∥2, gradient conflict occurs. The visualization confirms our causal chain: pass@k upweights hard prompts (column B) → these hard prompts exhibit negative interference aθ(x)<0 (column A) → the reweighted gradient conflicts with pass@1’s gradient (column C). The robustness across all seven threshold configurations (see Appendix D.2, Figure 8) demonstrates this is not an artifact of threshold selection but a fundamental property of pass@k optimization in the presence of negatively interfering prompts. Additional ablations: Which prompts are amplified by pass@k? To have a complementary view of the amplification mechanism, we directly plot weights against agreement scores, with color encoding pass@1 probability. See Figure 7 which shows results for both models across three representative threshold configurations. The critical observation is where these weights concentrate in agreement-score space (x-axis). High-weight prompts cluster in the negative-agreement region (aθ(x)<0), while low-weight prompts cluster in the positive-agreement region. This means pass@k’s design, which correctly prioritizes hard prompts for its own objective, systematically amplifies negatively interfering prompts that hurt pass@1. (a) Llama: =δ10.80, =δ20.10 (b) Llama: =δ10.85, =δ20.15 (c) Llama: =δ10.90, =δ20.10 (d) Qwen: =δ10.80, =δ20.10 (e) Qwen: =δ10.85, =δ20.15 (f) Qwen: =δ10.90, =δ20.10 Figure 7: Pass@k amplification pattern across models and thresholds. Each point represents one prompt, with the corresponding agreement score (x-axis), pass@k weight (y-axis), and pass@1 probability shown by color (red = low, green = high). Consistent pattern: Prompts with negative alignment (left), low pass@1 (red), and high weights (4-12) cluster upper-left, while prompts with positive alignment (right), high pass@1 (green), and zero weights (∼10-28) appear lower-right. Top row (a-c): DeepSeek-R1-Distill-Llama-8B across three δ2 values (0.05, 0.10, 0.15). Bottom row (d-f): DeepSeek-R1-Distill-Qwen-7B with same thresholds. 6 Related Work Inference-time fine-tuning. Inference-time objective optimization has recently witnessed a surge of interest for optimizing language model performance for specific inference-time procedures such as win rate [balashankar-et-al25infalign], Best-of-N [sessa-et-al25bond, amini-et-al25vBoN, chow-et-al25infaware-BoN, beirami-et-al25theoretical-BoN, aminian-et-al25bon], majority voting [uesato-et-al22, lightman-et-al24, du-et-al25] or more recently their combination [di-et-al25best-of-majority]. davis-recht25 showed that several existing algorithms for post-training large language models (such as rejection sampling fine-tuning (see e.g. xiong-et-al25minimalist) and GRPO [shao-et-al24deepseekmath]) can be seen as instances of stochastic gradient ascent for maximizing a monotonically increasing function of the probability of giving a correct answer given a prompt, in the binary reward setting. Pass@k optimization. In this work, we focus on the special case of the pass@ k metric which has been introduced as an evaluation metric in chen-et-al21eval-llms-code for code generation tasks, similarly to a metric proposed earlier in kulal-et-al19spoc. Given access to verifiable rewards, chen-et-al21eval-llms-code proposed an unbiased estimator of the pass@k metric. Inspired by inference-aware optimization gains, tang-et-al25icml recently proposed to directly perform pass@k policy optimization with variance-reduced pass@k policy gradients using the control-variates method. Concurrently, walder-karkhanis25neurips introduced new lower variance unbiased estimators for pass@k and its gradient using more general U-statistics and demonstrated its performance for solving hard problems. mahdavi-et-al25pg-pass@k derived expressions for the pass@k objective and pass@k policy gradients as presented in (2) and proposed a new pass@k gradient reweighting method with a tunable parameter γ different from k. yu25 also derives per-prompt pass@k policy gradients and shows that per-prompt pass@k and pass@1 gradients are positively collinear vectors, questioning the soundness of considering the pass@k objective and advocating for different exploration mechanisms while keeping pass@k as an inference-time evaluation metric. We show in our work that the expected pass@k and pass@1 gradients over the prompt distribution may not necessarily be positively collinear. chen-et-al25pass@ktraining-exploration investigated how pass@k model training balances the exploration and exploitation capabilities of large language models. jiang-et-al25risk-sensitive-rl further considered a different risk-sensitive objective than pass@k based on a soft maximum (log-sum-exp) objective depending on a temperature parameter, providing a knob to trade-off between pass@k and pass@1 performance. peng-et-al25simko proposed Simple Pass@K Optimization (SimKO) which redistributes gradient updates across the top-K token-level probability candidates to promote exploration and improve the balance between exploitation and exploration. thrampoulidis-et-al25 connects policy gradient algorithms for pass@k optimization [mahdavi-et-al25pg-pass@k, tang-et-al25icml, walder-karkhanis25neurips] to advantage shaped GRPO methods [chen-et-al25pass@ktraining-exploration, shao-et-al24deepseekmath, deng-et-al25, mroueh25rlvr-grpo]. To the best of our knowledge, prior work on pass@k optimization does not theoretically characterize when and why optimizing pass@k can degrade pass@1 under pass@k policy optimization; we provide such a characterization. Interference. The concept of ray interference has been previously introduced and investigated in multi-task reinforcement learning [schaul-et-al19ray-interference]. As an analogy with multi-task learning, tasks can be viewed as prompts (or problems to solve in the special case of mathematical reasoning) in our LLM post-training setting. Instead of composite optimization objectives in the form of a sum over tasks, pass@k objectives are expected probabilities of success over a prompt distribution, which may be discrete or continuous (see (pass@k)). Our definition of prompt interference (Definition 3.1) parallels schaul-et-al19ray-interference, using gradients of pass@1 in our LLM post-training setting rather than gradients of task loss functions in multi-task learning. Related notions of gradient conflict/interference have also been studied in multi-task learning (e.g., yu-et-al20gradient-surgery-mtl). To extend the multi-task learning analogy discussed in qu-et-al26pope, tasks can also be thought of as clusters of prompts labeled by degree of difficulty. Inspired by schaul-et-al19ray-interference, qu-et-al26pope recently investigated learning to reason on hard problems when no correct rollouts are sampled and argue that pass@k optimization does not address this challenge. They emphasize the role of ray interference in hindering learning on heterogeneous prompt mixtures; in particular, Section 3.2 discusses negative interference between easy and hard problems, and how pass@k policy optimization does not mitigate its consequences. Conceptually, qu-et-al26pope study interference as an exploration/solvability bottleneck on heterogeneous mixtures, while we analyze the pass@k objective itself and show how prompt-level gradient conflict can induce an explicit pass@k–pass@1 trade-off. Our findings are consistent with the empirical insights and intuitions in [qu-et-al26pope, Section 3.2]. We offer a complementary theoretical view by formalizing prompt interference for LLM post-training and providing conditions under which pass@1 degradation can occur under pass@k policy optimization via prompt-level gradient conflict (see Proposition 4.1, Corollary 4.4, and Proposition 4.6). Our results also connect to several open questions highlighted by qu-et-al26pope. While our paper focuses on the theoretical understanding of the pass@k vs pass@1 trade-off and limitations of pass@k policy optimization, qu-et-al26pope propose a framework (Privileged On-Policy Exploration, POPE) to mitigate ray interference and improve the solvability of hard problems by guiding on-policy exploration. Our theoretical insights may suggest alternative mitigation mechanisms and clarify when such interventions are necessary and effective. 7 Conclusion and Future Work In this work, we investigated why pass@ k policy optimization can degrade pass@1, showing that pass@k and pass@1 gradients can be conflicting when pass@k weights amplify negative prompt interference. Our findings open several avenues for future work. An immediate direction is to design and evaluate methods to mitigate this conflict issue. More broadly, it would be interesting to investigate more general inference-time objectives beyond pass@k (such as other concave transforms of the probability of success) in view of solving harder reasoning tasks and enhancing exploration with more flexible and targeted prompt reweighting without sacrificing pass@1 performance. Our similarity metric can offer a good starting point for gradient surgery. Acknowledgements We thank Aviral Kumar for pointing us to relevant related work, which helped us to update the related work discussion at the end of Section 6 and to add remarks on prior work on interference immediately after Definition 3.1. References Table of Contents 1 Introduction 2 Preliminaries 3 Prompt Interference 3.1 Prompt similarity measure and interference 3.2 Toy example 3.3 Tradeoff: pass@k ascent can decrease pass@1 4 Pass@k and Pass@1 Gradients Conflict 4.1 Gradient conflict: Reweighting meets interference 4.2 Sufficient conditions and threshold in k 4.3 Pass@1 degradation under pass@k updates 5 Experiments 6 Related Work 7 Conclusion and Future Work References A Discussion of the covariance condition in Proposition 4.1 B A kernel view of gradient conflict beyond =k1 C Proofs C.1 Proof of Proposition 4.1: Gradients conflict characterization C.2 Proof of Corollary 4.4: Dominating negatively interfering prompts C.3 Proof of Proposition 4.5: Influence of k C.4 Lemma C.1: Smoothness of pass@k C.5 Proof of Proposition 4.6: Pass@1 degradation under pass@k updates D Additional Experiments D.1 More details regarding experimental setting D.2 Threshold robustness analysis Appendix A Discussion of the covariance condition in Proposition 4.1 Introducing the correlation ρw,a(θ):=cov(wk,θ(x),aθ(x))σw(θ)⋅σa(θ) where σw(θ),σa(θ) are respectively the standard deviations of wk,θ(x) and aθ(x), we can then rewrite condition (10) equivalently as: <ρw,a(θ)-⋅E[wk,θ(x)]‖∇J1(θ)‖2⋅σw(θ)σa(θ). (19) This condition emphasizes that conflict requires strong negative correlation between hardness weights and agreement scores as well as sufficient variance. If the weights barely vary (σw(θ)≃0), pass@k cannot strongly shift focus across prompts and conflict is harder. If agreement scores barely vary (σa(θ)≃0) there is no meaningful conflict structure to exploit that can lead to misaligned gradients. Appendix B A kernel view of gradient conflict beyond =k1 In this section, we further investigate the gradient conflict phenomenon between pass@ k and pass@m for any k,m≥1. Using the kernel κθ defined in (5), we now relate pass@k and pass@m gradients for any k,m≥1 using (2) to obtain: =⟨∇Jk(θ),∇Jm(θ)⟩E∼x,x′D[wk,θ(x)wm,θ(x′)κθ(x,x′)]. (20) This identity separates two factors: (1) where each objective focuses via the weights wk,θ(x),wm,θ(x′) with prompt reweighting toward ‘hard’ prompts as k grows; and (2) whether those regions interfere via the sign structure of κθ(x,x′) which is independent of the parameter k. Conflict between pass@k and pass@m gradients happens when the objectives emphasize prompt regions whose cross-similarity (encoded by the kernel) is negative. The following result which is an immediate consequence of (20) shows that conflict requires interference. Proposition B.1 (No interference regime: No negative transfer ⟹ no conflict). For any policy parameter ∈θRd, if ≥κθ(x,x′)0 for ×D a.e. (x,x′) then, ≥⟨∇Jk(θ),∇Jm(θ)⟩0 for all ≥k,m1. If pairwise prompt similarities are nonnegative on the support of the prompt distribution then all pass@k objectives are mutually aligned. Therefore, conflict can only occur when pairwise similarities are not always nonnegative and depends on how the weights wk,θ,wm,θ shift mass. In other words, pass@k objectives cannot conflict unless there is negative transfer between some prompts. Appendix C Proofs C.1 Proof of Proposition 4.1: Gradients conflict characterization Proof. First recall from (2) that for any k ≥1andanyθ∈R^d, ∇Jk(θ)=E∼xD[wk(pθ(x))∇pθ(x)]=E∼xD[wk(pθ(x))∇J1(x;θ)]. (21) Thenitimmediatelyfollowsthat:⟨∇Jk(θ),∇J1(θ)⟩=⟨E∼xD[wk(pθ(x))∇J1(x;θ)],∇J1(θ)⟩=E∼xD[⟨wk(pθ(x))∇J1(x;θ),∇J1(θ)⟩]=E∼xD[wk(pθ(x))aθ(x)],wherewerecallthata_θ(x) := ⟨∇J_1(x;θ), ∇J_1(θ) ⟩.Bydefinitionofthecovariancebetweentworeal-valuedrandomvariables:cov(wk,θ(x),aθ(x))=-E∼xD[⋅wk,θ(x)aθ(x)]⋅E∼xD[wk,θ(x)]E∼xD[aθ(x)]=-⟨∇Jk(θ),∇J1(θ)⟩⋅E∼xD[wk,θ(x)]‖∇J1(θ)‖2,wherethesecondequalityfollowsfromusing(C.1)andnoticingthatE_x ∼D[ a_θ(x)] = ∥∇J_1(θ)∥^2bydefinitionofa_θ(x) .:Rearrangingtheaboveidentitythengives =⟨∇Jk(θ),∇J1(θ)⟩+⋅E∼xD[wk,θ(x)]‖∇J1(θ)‖2cov(wk,θ(x),aθ(x)). (22) Asaconsequenceof(C.1)and(22),wehavethat⟨∇J_k(θ), ∇J_1(θ) ⟩< 0:ifandonlyifE_x ∼D[w_k(p_θ(x)) a_θ(x)] < 0:whichisalsoequivalenttothecondition <cov(wk,θ(x),aθ(x))-⋅E[wk,θ(x)]‖∇J1(θ)‖2. (23) ∎ C.2 Proof of Corollary 4.4: Dominating negatively interfering prompts Proof. First, Proposition 4.1 yields: =⟨∇Jk(θ),∇J1(θ)⟩E[ωk,θ(x)aθ(x)]. (24) We decompose this inner product into 2 regions: the m-strongly negatively interfering region X-(θ,m) and its complement, ⟨∇Jk(θ),∇J1(θ)⟩=E[ωk,θ(x)aθ(x)]=+E[ωk,θ(x)aθ(x) 1∈xX-(θ,m)]E[ωk,θ(x)aθ(x) 1∉xX-(θ,m)]. (25) We now upperbound each one of the above terms. For the first term, by definition of the set of prompts X-(θ,m), we have ≤aθ(x)-m for any ∈xX-(θ,m). As for the second term, we show that for any ∉xX-(θ,m), ≤|aθ(x)|G2 using Assumption 4.3 as follows. Bounding aθ(x). Using the definition of the agreement score aθ(x) (see (8)) together with Cauchy-Schwarz inequality, we have aθ(x)=⟨∇J1(x;θ),∇J1(θ)⟩≤⋅‖∇J1(x;θ)‖E[∇J1(x;θ)]‖≤⋅‖∇J1(x;θ)‖E[‖∇J1(x;θ)‖]. (26) Then recalling the policy gradient expression of ∇J1(x;θ) in (4) and using Jensen’s inequality, we have: ‖∇J1(x;θ)‖2=‖E∼yπθ(⋅|x)[r(x,y)sθ(x,y)]‖2≤E∼yπθ(⋅|x)[‖sθ(x,y)‖2]≤G2, (27) where the last inequality follows from using Assumption 4.3. Using this inequality in (26) yields the desired inequality: ∈∀xX,≤|aθ(x)|G2. (28) Back to (25) and using the inequalities established above, we conclude the proof: ≤⟨∇Jk(θ),∇J1(θ)⟩-δ(θ), (29) where :=δ(θ)-mW-(k,θ)G2W+(k,θ) and the aggregated weights W-(k,θ),W+(k,θ) are defined in (14). If in addition <δ(θ)0, then ⟨∇Jk(θ),∇J1(θ)⟩≤-δ(θ)<0. ∎ C.3 Proof of Proposition 4.5: Influence of k Proof. Recall from Proposition 4.1 that: =⟨∇Jk(θ),∇J1(θ)⟩E[ωk,θ(x)aθ(x)]. (30) Using again the same decomposition as in (25), we have: ⟨∇Jk(θ),∇J1(θ)⟩=E[ωk,θ(x)aθ(x)]=+E[ωk,θ(x)aθ(x) 1∈xX-(θ,m)]E[ωk,θ(x)aθ(x) 1∉xX-(θ,m)]. (31) Depending on the prompt region, we have two distinct ways to bound the pass@k weights: (i) For ∈xX(m,θ), we have by assumption, ≤pθ(x)ε, hence ≥ωk,θ(x)k(-1ε)-k1. (i) For ∉xX(m,θ) we have by assumption, ≥pθ(x)δ, hence ≤ωk,θ(x)k(-1δ)-k1. Observing in addition to both items above that ≤aθ(x)-m for any ∈xX-(θ,m) and ≤aθ(x)G2 for any ∈xX (as shown in (28) under Assumption 4.3) and hence for any ∉xX-(θ,m) in particular, we obtain from (31) the following inequality: ≤⟨∇Jk(θ),∇J1(θ)⟩+-k(-1ε)-k1mqk(-1δ)-k1G2(-1q), (32) where we also use the assumption >qθq. Thus it suffices to have <k(-1δ)-k1G2(-1q)k(-1ε)-k1mq, to ensure that <⟨∇Jk(θ),∇J1(θ)⟩0. The above condition is equivalent to: <G2(-1q)mq(-1ε-1δ)-k1. Taking log in the above inequality, recalling that >-1ε-1δ and reordering the inequality yields: k>k⋆:=+ 1log((-1q)G2qm)log(-1ε-1δ), (33) which concludes the proof. ∎ C.4 Lemma C.1: Smoothness of pass@k Lemma C.1 (Smoothness of pass@k). Under Assumption 4.3, for all ≥k1, the pass@k objective Jk is Lk-smooth with :=Lk+k2G2kF, i.e. for all ∈θ,θ′Rd, ≤|-Jk(θ′)Jk(θ)⟨∇Jk(θ),-θ′θ⟩|Lk2‖-θ′θ‖22. Proof. To show smoothness of the pass@k objective Jk, we bound its Hessian ∇2Jk(θ) uniformly in the policy parameter θ using Assumption 4.3. Since =∇Jk(θ)E∼xD[wk(pθ(x))∇pθ(x)] as shown in (2), the chain rule gives the Hessian of Jk for every ∈θRd, =∇2Jk(θ)E∼xD[+ωk′(pθ(x))∇pθ(x)∇pθ(x)⊤ωk(pθ(x))∇2pθ(x)]. (34) Taking the operator norm and using the triangular inequality yields: ≤‖∇2Jk(θ)‖E∼xD[+⋅|ωk′(pθ(x))|‖∇pθ(x)‖2⋅|ωk(pθ(x))|‖∇2pθ(x)‖]. (35) In the rest of the proof, we bound each term separately. Step 1: Bounding ‖∇pθ(x)‖2. Recall from (4) that the pass@1 policy gradient is given by: =∇pθ(x)E∼yπθ(⋅|x)[r(x,y)∇logπθ(|yx)]. (36) By Jensen’s inequality and since ∈r(x,y)0,1, we have ‖∇pθ(x)‖2 ≤E∼yπθ(⋅|x)[∥r(x,y)2∇logπθ(|yx)∥2]≤E∼yπθ(⋅|x)[∥∇logπθ(|yx)∥2]≤G2, (37) where the last bound follows from Assumption 4.3. Step 2: Bounding ‖∇2pθ(x)‖. Differentiating the pass@1 policy gradient, we obtain: =∇2pθ(x)E∼yπθ(⋅|x)[+r(x,y)∇logπθ(|yx)∇logπθ(|yx)⊤r(x,y)∇2logπθ(|yx)]. Taking the operator norm and using the triangle inequality, we obtain: ‖∇2pθ(x)‖ ≤+E∼yπθ(⋅|x)[|r(x,y)|⋅∥∇logπθ(|yx)∥2]E∼yπθ(⋅|x)[⋅|r(x,y)|‖∇2logπθ(|yx)‖] ≤+E∼yπθ(⋅|x)[∥∇logπθ(|yx)∥2]E∼yπθ(⋅|x)[‖∇2logπθ(|yx)‖]≤+G2F, (38) where the last inequality again follows from using Assumption 4.3. Step 3: Bounding weights ωk′ and ωk. By definition of the weights wk(p)=fk′(p)=k(-1p)-k1 for any ∈p[0,1]. Therefore it follows that =ωk′(p)-k(-k1)(-1p)-k2 for ≥k2. Hence, for any ∈p[0,1] and any ≥k2, ≤|ωk(p)|k,≤|ωk′(p)|k(-k1). (39) Conclusion. Plugging (37)–(38) and the bounds on ωk,ωk′ (39) into (35) yields ‖∇2Jk(θ)‖≤+k(-k1)G2k(+G2F)≤+k2G2kF. Finally, boundedness of the Hessian norm implies that Jk is Lk-smooth with =Lk+k2G2kF, i.e., for any ∈θ,θ′Rd, ≤|-Jk(θ′)Jk(θ)⟨∇Jk(θ),-θ′θ⟩|Lk2‖-θ′θ‖2. ∎ C.5 Proof of Proposition 4.6: Pass@1 degradation under pass@k updates Proof. We prove the decrease of pass@1 and increase of pass@k under pass@k policy gradient ascent separately. Recall the policy update rule with step size >η0: =θ++θη∇Jk(θ). (40) (i) Pass@1 descent. By L1-smoothness of J1 (Lemma C.1) and using the policy update rule (40), we have: J1(θ+) ≤+J1(θ)⟨∇J1(θ),-θ+θ⟩L12‖-θ+θ‖2 =+J1(θ)η⟨∇J1(θ),∇Jk(θ)⟩L1η22‖∇Jk(θ)‖2. (41) We now bound each one of the last two terms in the above inequality. Bounding ‖∇Jk(θ)‖2. Recall the pass@k policy gradient expression =∇Jk(θ)Ex[ωθ(x)∇J1(x;θ)], we have by Jensen and Cauchy-Schwarz inequalities: ‖∇Jk(θ)‖2=‖Ex[ωθ(x)∇J1(x;θ)]‖2≤Ex[|ωθ(x)|2‖∇J1(x;θ)‖2]≤k2G2, (42) where the last inequality follows from using the pass@1 policy gradient bound in (27) together with immediate weight bound ≤|ωθ(x)|k and the fact that the reward is binary (this can be generalized to a continuous uniformly bounded reward function). Bounding ⟨∇J1(θ),∇Jk(θ)⟩. By Corollary 4.4, we have ≤⟨∇J1(θ),∇Jk(θ)⟩-δ(θ). (43) Combining the bounds (42) and (43), we immediately obtain from (41): ≤J1(θ+)+-J1(θ)ηδ(θ)C2η2, (44) where :=C2L1k2G22 and =L1+G2F as shown in Lemma C.1, G,F being smoothness constants defined in Assumption 4.3. Hence, if >δ(θ)0 and 0<η≤η+:=δ(θ)C2, then <+-ηδ(θ)C2η20 and it follows that J1(θ+)≤+-J1(θ)ηδ(θ)C2η2<J1(θ), (45) which is the first desired inequality. (i) Pass@k ascent. As for pass@k under pass@k policy gradient ascent, the proof is a standard application of smoothness in optimization. We provide a brief proof in our setting for completeness. Using Lk-smoothness of the pass@k objective shown in Lemma C.1 together with the policy update rule (40), we have: Jk(θ+) ≥-+Jk(θ)⟨∇Jk(θ),-θ+θ⟩Lk2‖-θ+θ‖2 =-+Jk(θ)η‖∇Jk(θ)‖2Lkη22‖∇Jk(θ)‖2 =+Jk(θ)η(-1Lkη2)‖∇Jk(θ)‖2. (46) We pick then ≤η1Lk to obtain the second desired inequality: Jk(θ+)≥+Jk(θ)η2‖∇Jk(θ)‖2≥Jk(θ). (47) Overall, a sufficient step-size condition to guarantee simultaneous increase of pass@k and decrease of pass@1 (i.e., in particular <J1(θ+)J1(θ) and >Jk(θ+)Jk(θ) when ≠∇Jk(θ)0) is: 0<η≤min(δ(θ)C2,1Lk), (48) and this concludes the proof. ∎ Appendix D Additional Experiments D.1 More details regarding experimental setting Sampling configuration. For each problem, we generate =k32 independent responses using temperature sampling with temperature =T0.7 and nucleus sampling with =p0.95. Responses are evaluated using exact match against ground truth answers and binary rewards indicate correctness. D.2 Threshold robustness analysis Figure 8 shows gradient misalignment across all 6 additional threshold configurations for DeepSeek-R1-Distill-Llama-8B (beyond the main text example of =δ10.85, =δ20.10). All combinations exhibit negative inner products (range: −0.49 to −0.65), demonstrating that the misalignment phenomenon is robust to threshold choices. (a) =δ10.80, =δ20.10 (636 prompts: 86 hard, 550 easy, IP=−0.652) (b) =δ10.80, =δ20.15 (647 prompts: 97 hard, 550 easy, IP=−0.621) (c) =δ10.85, =δ20.05 (502 prompts: 38 hard, 464 easy, IP=−0.551) (d) =δ10.90, =δ20.10 (505 prompts: 86 hard, 419 easy, IP=−0.527) Figure 8: Gradient conflict across all threshold configurations (DeepSeek-R1-Distill-Llama-8B). Each panel shows agreement scores, weights, and weighted contributions for a different (δ1,δ2) configuration. All exhibit negative inner products, confirming robust misalignment across thresholds. The consistent downward shift from unweighted to weighted means shows pass@k optimization systematically opposes pass@1 improvement. ***Note that the denominator is nonzero as the weights are not equal to zero a.e. _θ(x,x )=E_ $ array[c]@c@ y _θ(·|x) y _θ(·|x ) array$ [δ(x,y,x',y') s_θ(x,y), s_θ(x',y') ], equation where $δ(x,y,x ,y ):=r(x,y)\,r(x ,y )$ which is equal to 10000\ $1$ if and only if both $y,y $ are correct completions for $x,x $ respectively (as rewards are binary). The kernel informs on whether improving pass@1 on one prompt tends to also improve pass@1 on another prompt under the shared policy parameters. To see this, consider a policy update $θ =θ+η∇ p_θ(x)$ with a small stepsize 10000\ $η>0$ in the direction $∇ p_θ(x)$ improving the probability of success on $x$. A first-order Taylor expansion of $θ p_θ(x )$ gives: \@@amsalign p_θ^+(x )& p_θ(x )+η ∇ p_θ(x ),∇ p_θ(x) =p_θ(x )+η _θ(x ,x)\,. This leads us to the following prompt interference definition. tcbcolframergb1,0.85,0.7 tcbcolbackrgb1,0.975,0.95 to433.62pt to225.89pt 0.0pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke [named]pgffillcolorrgb1,0.85,0.7 @color@rgb@fill10.850.7 @invoke @fill@opacity1.0 @invoke @moveto0.0pt4.26788pt @lineto0.0pt221.61902pt @curveto0.0pt223.97614pt1.91077pt225.8869pt4.26788pt225.8869pt @lineto429.35211pt225.8869pt @curveto431.70923pt225.8869pt433.62pt223.97614pt433.62pt221.61902pt @lineto433.62pt4.26788pt @curveto433.62pt1.91077pt431.70923pt0.0pt429.35211pt0.0pt @lineto4.26788pt0.0pt @curveto1.91077pt0.0pt0.0pt1.91077pt0.0pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke [named]pgffillcolorrgb1,0.975,0.95 @color@rgb@fill10.9750.95 @invoke @fill@opacity1.0 @invoke @moveto1.42262pt4.26788pt @lineto1.42262pt221.61902pt @curveto1.42262pt223.19043pt2.69647pt224.46428pt4.26788pt224.46428pt @lineto429.35211pt224.46428pt @curveto430.92352pt224.46428pt432.19737pt223.19043pt432.19737pt221.61902pt @lineto432.19737pt4.26788pt @curveto432.19737pt2.69647pt430.92352pt1.42262pt429.35211pt1.42262pt @lineto4.26788pt1.42262pt @curveto2.69647pt1.42262pt1.42262pt2.69647pt1.42262pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke @fill@opacity1.0 @invoke @beginscope @invoke @transformcm1.00.00.01.06.26788pt214.75792pt @invoke [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 [b]421.08423pt [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 definition[Prompt interference]We will say that two prompts 10000\ $x$ and 10000\ $x $ are interfering positively if $ _θ(x ,x)>0$ where the similarity kernel 10000\ $ _θ$ is defined in eq:kernel-def-1 (or eq:kernel-def-2): a step that increases the per-prompt pass@1 $p_θ(x)$ at $x$ also tends to increase $p_θ(x )$. Otherwise, if $ _θ(x ,x)<0$, the two prompts will be said to interfer negatively and a step that increases $p_θ(x)$ tends to decrease $p_θ(x )$. definition @invoke @endscope @invoke @endscope @discardpath @invoke @endscope We remark that our definition is similar to the definition of interference in [citet]\@@bibrefAuthors Phrase1YearPhrase2schaul-et-al19ray-interference\@@citephrase[\@@citephrase, Definition 10000\ 1] in the context of multi-task learning. See also [cite]\@@bibrefAuthors Phrase1YearPhrase2yu-et-al20gradient-surgery-mtl\@@citephrase[\@@citephrase] who also discuss gradient interference between the gradients of different task loss function gradients in the same multi-task learning context. Recently, in the context of LLM post-training, [cite]\@@bibrefAuthors Phrase1YearPhrase2qu-et-al26pope\@@citephrase[\@@citephrase] invoked ray interference as introduced in [citet]\@@bibrefAuthors Phrase1YearPhrase2schaul-et-al19ray-interference\@@citephrase[\@@citephrase] to explain why learning can be inhibited on heterogeneous prompt mixtures and why exploration on hard problems can be hurt. We refer the reader to Section 10000\ sec:related-work for further discussion regarding the concept of interference and prior work. In the next section, we illustrate the prompt interference concept on a toy example. \@@numbered@sectionsubsectiontocToy example Setting. We consider a minimal contextual-bandit abstraction of verifier-based multi-sample LLM evaluation as a running example. A prompt is a context $x $ drawn from a mixture distribution with a latent difficulty label $ (x)∈\ easy, hard\$ with a fixed proportion of hard labels: $P( (x)= hard)=p_hard.$ Each prompt is represented by a 2D feature vector $ψ(x)=[1,s(x)] ^2$ with a bias $1$ and a scalar feature $s(x) .$ Conditioned on difficulty, the scalar feature follows overlapping Gaussians whose separation is controlled by a parameter $δ>0$: if $ (x)= easy$ then $s(x) \! (-δ/2,\,1 ),$ and if $ (x)= hard$ then $s(x) \! (+δ/2,\,1 ).$ Smaller $δ$ yields greater overlap between easy and hard feature distributions. A completion is represented as a discrete action $y∈\0,1\.$ Each prompt has a unique correct completion determined by its difficulty: $y (x)=0$ if $ (x)=easy$ and $y (x)=1$ if $ (x)=hard$. The verifier reward is deterministic binary: $r(x,y)=1\y=y (x)\\,.$ We use a two-action softmax policy parameterized by a single vector $θ ^2$ such that $ _θ(1|x)=σ(θ ψ(x))$ and $ _θ(0|x)=1-σ(θ ψ(x))$, where $σ(t)= 11+ (-t)$ is the sigmoid function. The policy shares parameters across heterogeneous subpopulations (easy vs hard). For later use, we also note that we initialize the policy at a reference policy $ _ref$ that is globally biased toward action $y=0$, inducing high initial success on easy prompts and low initial success on hard prompts. Illustration of negative prompt interference. It follows that the per-prompt success probability defined in eq:ptheta-def is given by $p_θ(x)= _θ(y (x)|x)$ for any prompt $x .$ A simple computation yields: equation∇ p_θ(x)= cases-z_θ(x)\,ψ(x),&if (x)=easy,\\ +\,z_θ(x)\,ψ(x),&if (x)=hard\,, cases equation where $z_θ(x):=σ(θ ψ(x))(1-σ(θ ψ(x)))≥ 0\,.$ As a consequence, two prompts that have a similar representation will have opposite per-prompt pass@1 gradients and will hence be negatively interfering. This will be the case in our example for prompts in the overlap between the two Gaussians: two such prompts can have different labels (hard and easy) while having a similar prompt representation (with a similar scalar feature $s(x)$). See Figure 10000\ fig:cosine-kernel-heatmap below for an illustration. figure[h] [width=260.17464pt]Figs/cosine_kernel_heatmap.pdf \@@toccaption @tag[ ]2Cosine kernel heatmap: $ (∇ p_θ(x),∇ p_θ(x ))$ for subsamples of prompts: 120 easy and 80 hard among a total of 6000 samples. Blue regions correspond to negative prompt interference.\@@caption @tag[: ] Figure 2 Cosine kernel heatmap: $ (∇ p_θ(x),∇ p_θ(x ))$ for subsamples of prompts: 120 easy and 80 hard among a total of 6000 samples. Blue regions correspond to negative prompt interference. \@add@centering figure In the next section, we show in our toy example how negative prompt interference can lead to pass@1 decrease under one-step pass@$k$ policy gradient ascent. \@@numbered@sectionsubsectiontocTradeoff: pass@$k$ ascent can decrease pass@1 Using the toy example introduced in the previous section, we show in this section that pass@1 can decrease under one-step pass@$k$ policy gradient ascent. Fix $δ=0.2$ and consider two realized prompts in the overlap region between the 2 Gaussians: a prompt $x_e$ with $ (x_e)=easy$ and $ψ(x_e)=[1,-0.1] $, and $x_h$ with $ (x_h)=hard$ and $ψ(x_h)=[1,+0.1] $. These two prompts have nearly identical representations as $ (ψ(x_e),ψ(x_h)) 0.98$ is close to 1. Because of eq:grad-pass@1-toy, they are therefore negative interfering as $ _θ(x_e,x_h)= ∇ p_θ(x_e),∇ p_θ(x_h) =-z_θ(x_e)z_θ(x_h) ψ(x_e),ψ(x_h) <0$ since $ ψ(x_e),ψ(x_h) >0.$ Numerically we have $ _θ(x_e,x_h) -0.01$ and $ (∇ p_θ(x_e),∇ p_θ(x_h)) -0.98$ which means that the per-prompt pass@1 gradients are almost antiparallel. On this two-point empirical distribution, the pass@1 and pass@$k$ gradients are given by: \@@amsalign∇ J_1(θ)&= 12(∇ p_θ(x_e)+∇ p_θ(x_h))\,,\\ ∇ J_10(θ)&= 12(w_e∇ p_θ(x_e)+w_h∇ p_θ(x_h))\,, where the pass@$10$ weight $w_e:=w_10,θ(x_e) 2.33× 10^-7$ whereas $w_h:=w_10,θ(x_h) 3.88.$ Pass@$10$ upweights the hard prompt ($p_θ(x_h) 0.10$ low) and aggressively downweights the easy prompt $x_e$ (large $p_θ(x_h) 0.86$). Indeed, recall that $w_k,θ(x)=k(1-p_θ(x))^k-1$ which is near $k$ when $p_θ(x)≈ 0$ but and near $0$ when $p_θ(x)≈ 1\,.$ As a consequence, the pass@$10$ gradient is therefore dominated by $∇ p_θ(x_h)$ which conflicts with $∇ p_θ(x_e)$. Importantly, we have $ (∇ J_1(θ),∇ J_10(θ))=-0.77<0$ meaning that the pass@1 and pass@$10$ are conflicting and strongly anti-aligned. With one gradient-ascent step on pass@$10$ $θ^+=θ+η∇ J_10(θ)$ with $η=5$, we have: \@@amsalign J_1(θ)& 0.48\;→\;J_1(θ^+) 0.46 ( ), 16.38895pt\\ J_10(θ)& 0.83\;→\;J_10(θ^+)=0.95 ( ). This shows that pass@1 decreases while pass@$k$ increases under pass@$k$ optimization. In our toy example, we have observed that negative prompt interference can induce a conflict between pass@1 and pass@$k$ gradients when it is amplified by the induced pass@$k$ prompt reweighting. In the next section, we go beyond our toy example to characterize this conflict between pass@$k$ and pass@1 gradients in the general setting. \@@numbered@sectionsectiontocPass@$k$ and Pass@1 Gradients Conflict In this section, we exploit the observation that pass@$k$ policy optimization induces an implicit prompt reweighting, biased toward upweighting low-success prompts. To connect with the previous section, if this reweighting coincides with upweighting negatively interfering prompts, then pass@$k$ policy updates may result in conflicting pass@$k$ and pass@1 gradients, leading to pass@1 degradation under pass@$k$ policy optimization updates. We make this chain of arguments precise in this section. \@@numbered@sectionsubsectiontocGradient conflict: Reweighting meets interference To explain why optimizing pass@$k$ can hurt pass@1, we study a key quantity: the angle between the policy gradients of pass@$k$ and pass@1 respectively. We show that this angle can be obtuse and we refer to this setting as a case of conflicting gradients. This observation might seem a little surprising from the theory viewpoint at first glance. Indeed, the per-prompt gradient $∇ J_k(x;θ)$ of pass@$k$ is always a positive multiple of the per-prompt policy gradient $∇ J_1(x;θ)$ of pass@1 since $∇ J_k(x;θ)=w_k,θ(x)∇ J_1(x;θ)$ and $w_k,θ(x)≥ 0$ for any prompt $x \,.$ Nevertheless, the population gradients $∇ J_k(θ)=E[∇ J_k(x;θ)]$ and $∇ J_1(θ)=E[∇ J_1(x;θ)]$ need not form an acute angle as their inner product can be negative as we will see. The following proposition provides an explicit expression of this inner product to characterize the gradients conflict. To state this result, we introduce the prompt-wise gradient agreement score for pass@1 defined for any policy parameter 10000\ $θ ^d$ and any prompt 10000\ $x $ as follows: equationa_θ(x):= ∇ J_1(x;θ),∇ J_1(θ) =E_x [ _θ(x,x )]\,. equation This agreement score quantifies the prompt-level gradient interference with the average pass@1 objective. If $a_θ(x)>0$, that prompt's pass@1 gradient points in a direction that also improves average pass@1. If $a_θ(x)<0$, it is anti-aligned and improving that prompt's pass@1 tends to decrease population pass@1 under the shared policy parameters 10000\ $θ\,.$ Using this agreement score, our characterization is as follows. tcbcolframergb1,1,1 to433.62pt to1040.16pt 0.0pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke [named]pgffillcolorrgb1,1,1 @color@gray@fill1 @invoke @fill@opacity1.0 @invoke @moveto0.0pt4.26788pt @lineto0.0pt1035.89595pt @curveto0.0pt1038.25307pt1.91077pt1040.16383pt4.26788pt1040.16383pt @lineto429.35211pt1040.16383pt @curveto431.70923pt1040.16383pt433.62pt1038.25307pt433.62pt1035.89595pt @lineto433.62pt4.26788pt @curveto433.62pt1.91077pt431.70923pt0.0pt429.35211pt0.0pt @lineto4.26788pt0.0pt @curveto1.91077pt0.0pt0.0pt1.91077pt0.0pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke [named]pgffillcolorrgb0.95,0.95,0.95 @color@gray@fill0.95 @invoke @fill@opacity1.0 @invoke @moveto1.42262pt4.26788pt @lineto1.42262pt1035.89595pt @curveto1.42262pt1037.46736pt2.69647pt1038.74121pt4.26788pt1038.74121pt @lineto429.35211pt1038.74121pt @curveto430.92352pt1038.74121pt432.19737pt1037.46736pt432.19737pt1035.89595pt @lineto432.19737pt4.26788pt @curveto432.19737pt2.69647pt430.92352pt1.42262pt429.35211pt1.42262pt @lineto4.26788pt1.42262pt @curveto2.69647pt1.42262pt1.42262pt2.69647pt1.42262pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke @fill@opacity1.0 @invoke @beginscope @invoke @transformcm1.00.00.01.06.26788pt1030.88206pt @invoke [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 [b]421.08423pt [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 proposition[Gradients conflict characterization]For any $k≥ 1$ and any $θ ^d$, \@@amsalign ∇ J_k(θ),∇ J_1(θ) &=E_x [w_k,θ(x)a_θ(x)]\\ &=E[w_k,θ(x)]·\|∇ J_1(θ)\|^2+cov(w_k,θ(x),a_θ(x))\,. In particular, $ ∇ J_k(θ),∇ J_1(θ) <0$ is equivalent to each one of the following conditions: enumerate[label=( *),leftmargin=0pt,itemindent=0.0pt,align=left] @item@$E_x [w_k,θ(x)a_θ(x)]<0\,,$ @item@ $cov(w_k,θ(x),a_θ(x))<-E[w_k,θ(x)]·\|∇ J_1(θ)\|^2\,.$ enumerate proposition @invoke @endscope @invoke @endscope @discardpath @invoke @endscope Discussion of conditions. We discuss each condition and provide two interpretations explaining gradients conflict and we refer the reader to figures 10000\ fig:front-page and 10000\ fig:agreement-score for illustrations on the toy example of section 10000\ sec:toy-example: distrib-shift Prompt distribution shift: pass@$k$ reweights prompts. Recalling that the weights $w_k,θ(x)$ are nonnegative, define the reweighted prompt distribution: equation* D_k,θ(dx)\, w_k,θ(x)D(dx)\,, equation* inducing the expectation $E_ D_k,θ[f(x)]= E_D[w_k,θ(x)f(x)]E_D[w_k,θ(x)]$ for any measurable function $f$ of the prompt random variable 10000\ $x$. Note that the distribution $ D_k,θ$ places higher mass on prompts with smaller success probability $p_θ(x)$ (hard prompts) since $w_k,θ(x)$ is decreasing in $p_θ(x).$ Therefore, we can rewrite eq:conflict-1 as follows: equation ∇ J_k(θ),∇ J_1(θ) =E_D[w_k,θ(x)]·E_ D_k,θ[a_θ(x)]\,, equation and since $E_D[w_k,θ(x)]>0$ (as $w_k,θ(x)≥ 0$ and not equal to zero a.e.), we obtain the following equivalence between $ ∇ J_k(θ),∇ J_1(θ) <0$ and $E_ D_k,θ[a_θ(x)]<0\,.$ This characterization shows that conflict of gradients of pass@1 and pass@$k$ occurs if and only if the average of the agreement score $a_θ(x)$ under the pass@$k$ induced prompt distribution $ D_k,θ$ is negative. In contrast note that $E_D[a_θ(x)]=\|∇ J_1(θ)\|^2≥ 0\,.$ Pass@$k$ changes the prompt weighting from $D$ to $ D_k,θ$ and the average agreement under this shifted distribution can become negative. In summary, we obtain the following causal chain: pass@$k$ induces a prompt reweighting toward harder prompts (i.e. lower $p_θ(x)$ since $w_k,θ(x)$ decreases in $p_θ(x)$). If hard prompts have negative agreement score $a_θ(x)$ (i.e. per-prompt pass@1 gradients are conflicting with the pass@1 gradient) then the reweighted average agreement becomes negative and the gradient of pass@$k$ and pass@1 form an obtuse angle as a consequence, i.e. they are conflicting. figure[!t] [width=303.53267pt]Figs/frontpage-figure.pdf \@@toccaption @tag[ ]3 Contour plots of pass@1 and pass@$k$ objectives in the policy parameter space. Gradients of pass@$k$ and pass@1 with respect to policy parameters are conflicting in the gray area. \@@caption @tag[: ] Figure 3 Contour plots of pass@1 and pass@$k$ objectives in the policy parameter space. Gradients of pass@$k$ and pass@1 with respect to policy parameters are conflicting in the gray area. \@add@centering figure figure[!h] [width=433.62pt]Figs/agreement_hist_init_vs_after.pdf \@@toccaption @tag[ ]4Agreement score 10000\ $a_θ(x)$ (see eq:alignment-score) on the toy example. The figure shows that the prompt distribution has both prompts with negative and positive agreement scores. Prompt with negative agreement scores are responsible for gradient conflict and hence pass@1 decrease. After pass@$k$ policy optimization, negative agreement scores get even more negative.\@@caption @tag[: ] Figure 4 Agreement score 10000\ $a_θ(x)$ (see eq:alignment-score) on the toy example. The figure shows that the prompt distribution has both prompts with negative and positive agreement scores. Prompt with negative agreement scores are responsible for gradient conflict and hence pass@1 decrease. After pass@$k$ policy optimization, negative agreement scores get even more negative. \@add@centering figure covariance Covariance between weights and agreement scores. The covariance 10000\ $cov(w_k,θ(x),a_θ(x))$ captures the coupling between `how much pass@$k$ emphasizes a prompt' (via weight $w_k,θ(x)$) and `whether that prompt helps pass@1' (via alignment score 10000\ $a_θ(x)$). The covariance term quantifies whether pass@$k$’s hardness reweighting concentrates gradient mass on prompts whose success-probability gradients are anti-aligned with the pass@1 gradient. Conflict happens when this anti-aligned hard-prompt mass overwhelms the positive term. See Appendix 10000\ appx:covar-condition for more details. remarkFor a more general discussion of the conflict between pass@$k$ and pass@$m$ gradients for any $k,m≥ 1$ using our kernel viewpoint, see Appendix 10000\ sec:kernel-grad-conflict-k-m. remark \@@numbered@sectionsubsectiontocSufficient conditions and threshold in $k$ We now provide sufficient conditions for pass@$k$ and pass@1 gradients conflict, as a corollary of Proposition 10000\ prop:grad-conflict. Before stating this result, we make a standard smoothness assumption on the policy parameterization. assumption[Lipschitz and smooth policy]There exist positive constants $G,F$ s.t. for all prompts $x ,$ we have the expected score bounds: equationE_y _θ(·|x)[\|∇ _θ(y|x)\|^2]≤ G^2\,, _y _θ(·|x)[\|∇^2 _θ(y|x)\|]≤ F\,. equation assumption This assumption is satisfied for instance by Gaussian policies with bounded state feature vectors and softmax tabular policies, see e.g. Lemma 4.4, 4.8 and corollary 4.3 in [citet]\@@bibrefAuthors Phrase1YearPhrase2yuan-et-al22vanilla-pg\@@citephrase[\@@citephrase]. This is a standard assumption in several works on the convergence analysis of policy gradient methods, see, e.g., [citet]\@@bibrefAuthors Phrase1YearPhrase2yuan-et-al22vanilla-pg,papini-et-al22smoothing\@@citephrase[\@@citephrase]. To quantify the magnitude of negative interference and its prevalence under the distribution 10000\ $D$, we define the $m$-strongly negatively interfering prompt set with margin 10000\ $m>0$: equationX_-(θ,m):=\x :a_θ(x)≤-m\\,, equation and denote by $q_θ:=D(X_-(θ,m))∈[0,1]$ its probability mass under the distribution 10000\ $D$ of prompts. Then, the expected pass@$k$ weight contributions inside and outside the $m$-strongly interfering set 10000\ $X_-(θ,m)$ are given by: \@@amsalign W_-(k,θ)&:=E[w_k,θ(x) 1_\x _-(θ,m)\]\,,\\ W_+(k,θ)&:=E[w_k,θ(x) 1_\x _-(θ,m)\]\,. The following corollary of Proposition 10000\ prop:grad-conflict shows that gradients of pass@1 and pass@$k$ conflict when the contribution of pass@$k$ weights in negatively interfering prompt regions dominates its counterpart for positively interfering prompts. tcbcolframergb1,1,1 to433.62pt to304.72pt 0.0pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke [named]pgffillcolorrgb1,1,1 @color@gray@fill1 @invoke @fill@opacity1.0 @invoke @moveto0.0pt4.26788pt @lineto0.0pt300.45235pt @curveto0.0pt302.80946pt1.91077pt304.72023pt4.26788pt304.72023pt @lineto429.35211pt304.72023pt @curveto431.70923pt304.72023pt433.62pt302.80946pt433.62pt300.45235pt @lineto433.62pt4.26788pt @curveto433.62pt1.91077pt431.70923pt0.0pt429.35211pt0.0pt @lineto4.26788pt0.0pt @curveto1.91077pt0.0pt0.0pt1.91077pt0.0pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke [named]pgffillcolorrgb0.95,0.95,0.95 @color@gray@fill0.95 @invoke @fill@opacity1.0 @invoke @moveto1.42262pt4.26788pt @lineto1.42262pt300.45235pt @curveto1.42262pt302.02376pt2.69647pt303.2976pt4.26788pt303.2976pt @lineto429.35211pt303.2976pt @curveto430.92352pt303.2976pt432.19737pt302.02376pt432.19737pt300.45235pt @lineto432.19737pt4.26788pt @curveto432.19737pt2.69647pt430.92352pt1.42262pt429.35211pt1.42262pt @lineto4.26788pt1.42262pt @curveto2.69647pt1.42262pt1.42262pt2.69647pt1.42262pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke @fill@opacity1.0 @invoke @beginscope @invoke @transformcm1.00.00.01.06.26788pt295.43846pt @invoke [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 [b]421.08423pt [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 corollary[Dominating negatively interfering prompts]Let Assumption 10000\ as:E-LS hold. Suppose that $q_θ>0$ and let $k≥ 2.$ Then for any $θ ^d,$ $ ∇ J_k(θ),∇ J_1(θ) ≤-δ(θ)$ where $δ(θ):=mW_-(k,θ)-G^2W_+(k,θ)$. If in addition $δ(θ)<0$, then $ ∇ J_k(θ),∇ J_1(θ) ≤-δ(θ)<0\,.$ corollary @invoke @endscope @invoke @endscope @discardpath @invoke @endscope Discussion of assumption. Note that the assumption $q_θ>0$ is necessary for gradient conflict. Indeed, if instead we have for instance for almost every $x ,a_θ(x)≥ 0$, then it follows from eq:conflict-1 that $ ∇ J_k(θ),∇ J_1(θ) ≥ 0$ as the weights $w_k,θ(x)$ are also always nonnegative. Negative agreement score is a necessary condition for gradient conflict. The gradient conflict condition of Corollary 10000\ cor:dominating-neg-interfer-prompts depends on the relative weights contribution ratio $W_-(k,θ)/W_+(k,θ)$ in negative vs positive prompt agreement score regions. This ratio is a function of both the chosen parameter $k$ and the probability of success 10000\ $p_θ(x)$ 10000\ eq:ptheta-def under policy 10000\ $ _θ\,.$ We now exploit the closed form of the pass@$k$ weights 10000\ $w_k,θ(x):=k(1-p_θ(x))^k-1$ to evaluate the ratio under assumptions on the prompt-wise probabilities of success. The next result shows that beyond a threshold $k $, the pass@$k$ and pass@1 gradients form an obtuse angle. We make a separation assumption on the probability of success over prompts: the probability of success for $m$-strongly interfering prompts is supposed to be uniformly smaller than the probability of success of the remaining prompts. Intuitively, this boils down to assume that negatively interfering prompts are relatively harder to solve than the rest of the prompts. tcbcolframergb1,1,1 to433.62pt to700.96pt 0.0pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke [named]pgffillcolorrgb1,1,1 @color@gray@fill1 @invoke @fill@opacity1.0 @invoke @moveto0.0pt4.26788pt @lineto0.0pt696.6877pt @curveto0.0pt699.04482pt1.91077pt700.95558pt4.26788pt700.95558pt @lineto429.35211pt700.95558pt @curveto431.70923pt700.95558pt433.62pt699.04482pt433.62pt696.6877pt @lineto433.62pt4.26788pt @curveto433.62pt1.91077pt431.70923pt0.0pt429.35211pt0.0pt @lineto4.26788pt0.0pt @curveto1.91077pt0.0pt0.0pt1.91077pt0.0pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke [named]pgffillcolorrgb0.95,0.95,0.95 @color@gray@fill0.95 @invoke @fill@opacity1.0 @invoke @moveto1.42262pt4.26788pt @lineto1.42262pt696.6877pt @curveto1.42262pt698.25911pt2.69647pt699.53296pt4.26788pt699.53296pt @lineto429.35211pt699.53296pt @curveto430.92352pt699.53296pt432.19737pt698.25911pt432.19737pt696.6877pt @lineto432.19737pt4.26788pt @curveto432.19737pt2.69647pt430.92352pt1.42262pt429.35211pt1.42262pt @lineto4.26788pt1.42262pt @curveto2.69647pt1.42262pt1.42262pt2.69647pt1.42262pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke @fill@opacity1.0 @invoke @beginscope @invoke @transformcm1.00.00.01.06.26788pt691.67381pt @invoke [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 [b]421.08423pt [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 proposition[Influence of $k$]In the setting of corollary 10000\ cor:dominating-neg-interfer-prompts, assume there exist $ ,δ,q>0$ s.t. equation _x _-(m,θ)p_θ(x)≤ <δ≤ _x _-(m,θ)p_θ(x), equation and $q_θ≥ q$ for any $θ$, then $ ∇ J_k(θ),∇ J_1(θ) ≤-δ$ where $δ=k[(1- )^k-1mq-(1-δ)^k-1G^2(1-q)].$ If in addition $k>k :=1+ ( (1-q)G^2qm) ( 1- 1-δ)\,,$ then $ ∇ J_k(θ),∇ J_1(θ) <0\,.$ proposition @invoke @endscope @invoke @endscope @discardpath @invoke @endscope This result shows there is a phase transition in $k$: if $k$ is large enough then gradients of pass@$k$ and pass@1 will be conflicting and hence pass@1 will degrade while pass@$k$ will improve under a small step size policy update following pass@$k$'s gradient. The threshold $k $ becomes smaller when the probability mass of the negatively interfering region is larger. This means that under larger negative prompt interference, the conflict between pass@1 and pass@$k$ becomes more severe as it occurs for even smaller $k$. \@@numbered@sectionsubsectiontocPass@1 degradation under pass@$k$ updates In this section, we show and quantify pass@1 decrease and pass@$k$ increase under pass@$k$ policy optimization, exploiting the conflict between pass@$k$ and pass@1 gradients shown in the previous sections. To quantify improvement and degradation under policy updates, we prove smoothness of the pass@$k$ objective for any $k≥ 1$ under Assumption 10000\ as:E-LS (see Lemma 10000\ lem:smoothness-pass@k in appendix 10000\ appx:lemma-smoothness). Using this smoothness property, we show that gradient conflict results in provable pass@1 decrease under pass@$k$ optimization. tcbcolframergb1,1,1 to433.62pt to226.61pt 0.0pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke [named]pgffillcolorrgb1,1,1 @color@gray@fill1 @invoke @fill@opacity1.0 @invoke @moveto0.0pt4.26788pt @lineto0.0pt222.34041pt @curveto0.0pt224.69753pt1.91077pt226.60829pt4.26788pt226.60829pt @lineto429.35211pt226.60829pt @curveto431.70923pt226.60829pt433.62pt224.69753pt433.62pt222.34041pt @lineto433.62pt4.26788pt @curveto433.62pt1.91077pt431.70923pt0.0pt429.35211pt0.0pt @lineto4.26788pt0.0pt @curveto1.91077pt0.0pt0.0pt1.91077pt0.0pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke [named]pgffillcolorrgb0.95,0.95,0.95 @color@gray@fill0.95 @invoke @fill@opacity1.0 @invoke @moveto1.42262pt4.26788pt @lineto1.42262pt222.34041pt @curveto1.42262pt223.91182pt2.69647pt225.18567pt4.26788pt225.18567pt @lineto429.35211pt225.18567pt @curveto430.92352pt225.18567pt432.19737pt223.91182pt432.19737pt222.34041pt @lineto432.19737pt4.26788pt @curveto432.19737pt2.69647pt430.92352pt1.42262pt429.35211pt1.42262pt @lineto4.26788pt1.42262pt @curveto2.69647pt1.42262pt1.42262pt2.69647pt1.42262pt4.26788pt @closepath @fill @invoke @invoke @endscope @beginscope @invoke @fill@opacity1.0 @invoke @beginscope @invoke @transformcm1.00.00.01.06.26788pt217.32652pt @invoke [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 [b]421.08423pt [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 proposition[Pass@$1$ decrease under pass@$k$ updates]In the setting of corollary 10000\ cor:dominating-neg-interfer-prompts, set $θ^+=θ+η∇ J_k(θ)$ where $k≥ 2$ and $η>0$. If $η≤ ( δ(θ)C_2, 1L_k)$ where $C_2:=k^2G^2L_1/2$ then \@@amsalign J_1(θ^+)&<J_1(θ)-ηδ(θ)+C_2η^2<J_1(θ)\,,\\ J_k(θ^+)&≥ J_k(θ)+ η2\|∇ J_k(θ)\|^2≥ J_k(θ)\,. proposition @invoke @endscope @invoke @endscope @discardpath @invoke @endscope The stepsize condition guarantees that the stepsize is small enough to ensure both decrease of pass@1 and increase of pass@$k$ using the different smoothness constants of pass@$k$ and pass@1 compared to the gradient conflict strength 10000\ $δ(θ).$ See Figure 10000\ fig:pass@k-pass@1 for an illustration over multiple iterations. figure[H] [width=346.89731pt]Figs/fig-toy-pass_k-vs-pass_1.pdf \@@toccaption @tag[ ]5Pass@5 vs pass@1 in example of section 10000\ sec:toy-example with pass@5 policy optimization, `pop' refers to `population' i.e. pass@$k$ as defined in eq:pass@k, `easy' and `hard' mean expectation is only taken over prompts labeled as easy respectively hard. Pass@5 increases while pass@1 decreases.\@@caption @tag[: ] Figure 5 Pass@5 vs pass@1 in example of section 10000\ sec:toy-example with pass@5 policy optimization, `pop' refers to `population' i.e. pass@$k$ as defined in eq:pass@k, `easy' and `hard' mean expectation is only taken over prompts labeled as easy respectively hard. Pass@5 increases while pass@1 decreases. \@add@centering figure \@@numbered@sectionsectiontocExperiments We empirically test whether the pass@$k$ objective can induce gradient conflict with pass@1 on math reasoning, as predicted by Proposition 10000\ prop:grad-conflict. Dataset and models. We use the MATH dataset 10000\ [citep][\@@bibrefAuthorsPhrase1Yearhendrycksmath2021\@@citephrase, ], which contains competition-style high school math problems spanning seven subjects (Algebra, Counting \& Probability, Geometry, Intermediate Algebra, Number Theory, Precalculus, and Prealgebra). We randomly sample 2,000 problems for evaluation. We run experiments with two reasoning models: DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B. Pass@$k$ gradient computation. For computational efficiency, we compute pass@1 gradients with respect to policy parameters in the language model's final hidden layer (dimension $d=4096$ for Llama-8B, $d=3584$ for Qwen-7B). We compute pass@$k$ gradients using Monte Carlo estimates based on eq:pass@k-grad and pass@$k$ estimates developed in prior work [citep][\@@bibrefAuthorsPhrase1Yearwalder-karkhanis25neurips,chen-et-al21eval-llms-code\@@citephrase, ]. Setup. We create filtered data sets $D_ _1, _2$ of prompts and responses with varying difficulty thresholds $( _1, _2)$ consisting of (i) hard prompts ($p_θ(x_i)< _2$) and (i) easy prompts ($p_θ(x_i)> _1$). We test 7 combinations with $ _1∈\0.80,0.85,0.90\$ and $ _2∈\0.05,0.10,0.15\$. For each combination, we compute estimates of the agreement scores 10000\ $a_θ(x)$ as defined in eq:alignment-score, pass@$k$ weights 10000\ $w_k,θ(x)$ (see eq:pass@k-weights-formula) and estimated average weighted agreement scores over prompts, which correspond to pass@$k$ and pass@1 gradient inner product as shown in Proposition 10000\ prop:grad-conflict. See Appendix 10000\ app:threshold_robustness for all threshold configurations. figure[htbp] subfigure345.0pt [width=345.0pt]Figs/deepseek_llama_threshold_variations_2000_k32/1x3_delta1_0.85_delta2_0.10.pdf \@@toccaption @tag[ ](a)DeepSeek-R1-Distill-Llama-8B: $ _1=0.85$, $ _2=0.10$ (550 prompts: 86 hard, 464 easy, ratio 5.4:1)\@@caption @tag[ ] (a) DeepSeek-R1-Distill-Llama-8B: $ _1=0.85$, $ _2=0.10$ (550 prompts: 86 hard, 464 easy, ratio 5.4:1) \@add@centering subfigure 7.97224pt subfigure345.0pt [width=345.0pt]Figs/deepseek_qwen_threshold_variations_2000_k32/1x3_delta1_0.85_delta2_0.05.pdf \@@toccaption @tag[ ](b)DeepSeek-R1-Distill-Qwen-7B: $ _1=0.85$, $ _2=0.05$ (656 prompts: 29 hard, 627 easy, ratio 21.6:1)\@@caption @tag[ ] (b) DeepSeek-R1-Distill-Qwen-7B: $ _1=0.85$, $ _2=0.05$ (656 prompts: 29 hard, 627 easy, ratio 21.6:1) \@add@centering subfigure \@@toccaption @tag[ ]6 Pass@$k$ optimization hurts pass@1 performance: Gradient conflict across two models. Both models demonstrate how pass@$k$ reweighting causes gradient conflict with pass@1, despite different threshold settings. Agreement scores are computed relative to the pass@1 gradient of filtered prompts. Left column (A): Agreement scores show separation between hard negatives (red) and easy positives (green), with unweighted means shown as blue dashed lines. Middle column (B): Pass@$k$ weights grouped by pass@1 probability, showing extreme weight disparity ($ $10$^28$:1) between hard and easy prompts. Right column (C): Weighted contributions (estimates of pass@$k$ vs pass@1 inner products) demonstrate how pass@$k$ reweighting hurts pass@1. Purple arrows highlight the critical downward shift from the unweighted mean (blue dotted, same as plot A) to the weighted mean (red dashed). (a) Deepseek-Llama: The downward shift from $+2.80× 10^-3$ to $-1.12× 10^-3$ ($ =-3.92× 10^-3$) flips the gradient from positive to negative, resulting in inner product $-0.613$. (b) DeepSeek-Qwen: The dramatic downward shift from $+2.97× 10^-2$ to $-2.74× 10^-1$ ($ =-3.04× 10^-1$) demonstrates even stronger gradient conflict. Despite only 29 hard prompts vs 627 easy (21.6:1 ratio), the extreme weight disparity causes hard negatives to completely dominate, resulting in inner product $-181$. \@@caption @tag[: ] Figure 6 Pass@$k$ optimization hurts pass@1 performance: Gradient conflict across two models. Both models demonstrate how pass@$k$ reweighting causes gradient conflict with pass@1, despite different threshold settings. Agreement scores are computed relative to the pass@1 gradient of filtered prompts. Left column (A): Agreement scores show separation between hard negatives (red) and easy positives (green), with unweighted means shown as blue dashed lines. Middle column (B): Pass@$k$ weights grouped by pass@1 probability, showing extreme weight disparity ($ $10$^28$:1) between hard and easy prompts. Right column (C): Weighted contributions (estimates of pass@$k$ vs pass@1 inner products) demonstrate how pass@$k$ reweighting hurts pass@1. Purple arrows highlight the critical downward shift from the unweighted mean (blue dotted, same as plot A) to the weighted mean (red dashed). (a) Deepseek-Llama: The downward shift from $+2.80× 10^-3$ to $-1.12× 10^-3$ ($ =-3.92× 10^-3$) flips the gradient from positive to negative, resulting in inner product $-0.613$. (b) DeepSeek-Qwen: The dramatic downward shift from $+2.97× 10^-2$ to $-2.74× 10^-1$ ($ =-3.04× 10^-1$) demonstrates even stronger gradient conflict. Despite only 29 hard prompts vs 627 easy (21.6:1 ratio), the extreme weight disparity causes hard negatives to completely dominate, resulting in inner product $-181$. \@add@centering figure Discussion. Our experiments validate the theoretical predictions in several ways. First, the agreement scores (column A in Figure 10000\ fig:gradient_misalignment_two_models) show clear separation between hard prompts (red points, negative agreement scores clustered below zero) and easy prompts (green points, positive agreement scores above zero), confirming that prompt interference exists in practice. The unweighted mean agreement (blue dashed line) is positive, indicating that under uniform weighting, the population gradient would improve pass@1. Second, the pass@$k$ weights (column B in Figure 10000\ fig:gradient_misalignment_two_models) reveal extreme disparity in how pass@$k$ values different prompts. The histogram shows that hard prompts (red bars, low pass@1 values $ 0.1$) receive weights of $n=38$ at scale 12, $n=21$ at scale 4, and $n=27$ at scale 1, while easy prompts (green bars, high pass@1 values $ 0.85$) receive negligible weights near $10^-28$. This creates weight ratios exceeding $10^28:1$, demonstrating the extreme reweighting mechanism our theory identifies: pass@$k$ dramatically amplifies the influence of low-success prompts while effectively ignoring high-success prompts. Third, column C demonstrates the consequence of this reweighting through weighted agreement scores. The purple arrows in panel C highlight the critical downward shift from the unweighted mean (blue dotted line, matching column A) to the weighted mean (red dashed line). For Llama-8B (Figure 10000\ fig:llama_085_010, this shift is $ =-3.92× 10^-3$, moving from $+2.80× 10^-3$ to $-1.12× 10^-3$ and flipping the gradient alignment from positive to negative, resulting in an inner product of $-0.613$. For Qwen-7B (Figure 10000\ fig:deepseek_085_005), despite having only 29 hard prompts versus 627 easy prompts (a 21.6:1 ratio heavily favoring easy prompts), the extreme weight disparity causes an even more dramatic shift of $ =-3.04× 10^-1$, moving from $+2.97× 10^-2$ to $-2.74× 10^-1$, yielding a strongly negative inner product of $-181$. This validates Proposition 10000\ prop:grad-conflict: when the negative covariance between pass@$k$ weights $w_k,θ(x)$ and agreement scores $a_θ(x)$ dominates the positive term $E[w_k,θ(x)]·\|∇ J_1(θ)\|^2$, gradient conflict occurs. The visualization confirms our causal chain: pass@$k$ upweights hard prompts (column B) $→$ these hard prompts exhibit negative interference $a_θ(x)<0$ (column A) $→$ the reweighted gradient conflicts with pass@1's gradient (column C). The robustness across all seven threshold configurations (see Appendix 10000\ app:threshold_robustness, Figure 10000\ fig:all_threshold_configs_llama) demonstrates this is not an artifact of threshold selection but a fundamental property of pass@$k$ optimization in the presence of negatively interfering prompts. Additional ablations: Which prompts are amplified by pass@$k$? To have a complementary view of the amplification mechanism, we directly plot weights against agreement scores, with color encoding pass@1 probability. See Figure 10000\ fig:weight_vs_alignment_scatter which shows results for both models across three representative threshold configurations. The critical observation is where these weights concentrate in agreement-score space (x-axis). High-weight prompts cluster in the negative-agreement region ($a_θ(x)<0$), while low-weight prompts cluster in the positive-agreement region. This means pass@$k$'s design, which correctly prioritizes hard prompts for its own objective, systematically amplifies negatively interfering prompts that hurt pass@1. figure*[!h] subfigure110.40253pt [width=345.0pt,height=137.5pt,keepaspectratio]Figs/weight_alignment_plots_llama/weight_vs_alignment_delta1_0.80_delta2_0.10.pdf \@@toccaption @tag[ ](a)Llama: $ _1=0.80$, $ _2=0.10$\@@caption @tag[ ] (a) Llama: $ _1=0.80$, $ _2=0.10$ \@add@centering subfigure subfigure110.40253pt [width=345.0pt,height=137.5pt,keepaspectratio]Figs/weight_alignment_plots_llama/weight_vs_alignment_delta1_0.85_delta2_0.15.pdf \@@toccaption @tag[ ](b)Llama: $ _1=0.85$, $ _2=0.15$\@@caption @tag[ ] (b) Llama: $ _1=0.85$, $ _2=0.15$ \@add@centering subfigure subfigure110.40253pt [width=345.0pt,height=137.5pt,keepaspectratio]Figs/weight_alignment_plots_llama/weight_vs_alignment_delta1_0.90_delta2_0.10.pdf \@@toccaption @tag[ ](c)Llama: $ _1=0.90$, $ _2=0.10$\@@caption @tag[ ] (c) Llama: $ _1=0.90$, $ _2=0.10$ \@add@centering subfigure 11.95836pt subfigure110.40253pt [width=345.0pt,height=137.5pt,keepaspectratio]Figs/weight_alignment_plots_qwen/weight_vs_alignment_delta1_0.80_delta2_0.10.pdf \@@toccaption @tag[ ](d)Qwen: $ _1=0.80$, $ _2=0.10$\@@caption @tag[ ] (d) Qwen: $ _1=0.80$, $ _2=0.10$ \@add@centering subfigure subfigure110.40253pt [width=345.0pt,height=137.5pt,keepaspectratio]Figs/weight_alignment_plots_qwen/weight_vs_alignment_delta1_0.85_delta2_0.15.pdf \@@toccaption @tag[ ](e)Qwen: $ _1=0.85$, $ _2=0.15$\@@caption @tag[ ] (e) Qwen: $ _1=0.85$, $ _2=0.15$ \@add@centering subfigure subfigure110.40253pt [width=345.0pt,height=137.5pt,keepaspectratio]Figs/weight_alignment_plots_qwen/weight_vs_alignment_delta1_0.90_delta2_0.10.pdf \@@toccaption @tag[ ](f)Qwen: $ _1=0.90$, $ _2=0.10$\@@caption @tag[ ] (f) Qwen: $ _1=0.90$, $ _2=0.10$ \@add@centering subfigure \@@toccaption @tag[ ]7 Pass@$k$ amplification pattern across models and thresholds. Each point represents one prompt, with the corresponding agreement score (x-axis), pass@$k$ weight (y-axis), and pass@1 probability shown by color (red = low, green = high). Consistent pattern: Prompts with negative alignment (left), low pass@1 (red), and high weights (4-12) cluster upper-left, while prompts with positive alignment (right), high pass@1 (green), and zero weights ($ $10$^-28$) appear lower-right. Top row (a-c): DeepSeek-R1-Distill-Llama-8B across three $ _2$ values (0.05, 0.10, 0.15). Bottom row (d-f): DeepSeek-R1-Distill-Qwen-7B with same thresholds. \@@caption @tag[: ] Figure 7 Pass@$k$ amplification pattern across models and thresholds. Each point represents one prompt, with the corresponding agreement score (x-axis), pass@$k$ weight (y-axis), and pass@1 probability shown by color (red = low, green = high). Consistent pattern: Prompts with negative alignment (left), low pass@1 (red), and high weights (4-12) cluster upper-left, while prompts with positive alignment (right), high pass@1 (green), and zero weights ($ $10$^-28$) appear lower-right. Top row (a-c): DeepSeek-R1-Distill-Llama-8B across three $ _2$ values (0.05, 0.10, 0.15). Bottom row (d-f): DeepSeek-R1-Distill-Qwen-7B with same thresholds. \@add@centering figure* \@@numbered@sectionsectiontocRelated Work Inference-time fine-tuning. Inference-time objective optimization has recently witnessed a surge of interest for optimizing language model performance for specific inference-time procedures such as win rate [citep][\@@bibrefAuthorsPhrase1Yearbalashankar-et-al25infalign\@@citephrase, ], Best-of-N [citep][\@@bibrefAuthorsPhrase1Yearsessa-et-al25bond,amini-et-al25vBoN,chow-et-al25infaware-BoN,beirami-et-al25theoretical-BoN,aminian-et-al25bon\@@citephrase, ], majority voting [citep][\@@bibrefAuthorsPhrase1Yearuesato-et-al22,lightman-et-al24,du-et-al25\@@citephrase, ] or more recently their combination [citep][\@@bibrefAuthorsPhrase1Yeardi-et-al25best-of-majority\@@citephrase, ]. [citet]\@@bibrefAuthors Phrase1YearPhrase2davis-recht25\@@citephrase[\@@citephrase] showed that several existing algorithms for post-training large language models (such as rejection sampling fine-tuning (see e.g. [citet]\@@bibrefAuthors Phrase1YearPhrase2xiong-et-al25minimalist\@@citephrase[\@@citephrase]) and GRPO [citep][\@@bibrefAuthorsPhrase1Yearshao-et-al24deepseekmath\@@citephrase, ]) can be seen as instances of stochastic gradient ascent for maximizing a monotonically increasing function of the probability of giving a correct answer given a prompt, in the binary reward setting. Pass@$k$ optimization. In this work, we focus on the special case of the pass@$k$ metric which has been introduced as an evaluation metric in [citet]\@@bibrefAuthors Phrase1YearPhrase2chen-et-al21eval-llms-code\@@citephrase[\@@citephrase] for code generation tasks, similarly to a metric proposed earlier in [citet]\@@bibrefAuthors Phrase1YearPhrase2kulal-et-al19spoc\@@citephrase[\@@citephrase]. Given access to verifiable rewards, [citet]\@@bibrefAuthors Phrase1YearPhrase2chen-et-al21eval-llms-code\@@citephrase[\@@citephrase] proposed an unbiased estimator of the pass@$k$ metric. Inspired by inference-aware optimization gains, [citet]\@@bibrefAuthors Phrase1YearPhrase2tang-et-al25icml\@@citephrase[\@@citephrase] recently proposed to directly perform pass@$k$ policy optimization with variance-reduced pass@$k$ policy gradients using the control-variates method. Concurrently, [citet]\@@bibrefAuthors Phrase1YearPhrase2walder-karkhanis25neurips\@@citephrase[\@@citephrase] introduced new lower variance unbiased estimators for pass@$k$ and its gradient using more general U-statistics and demonstrated its performance for solving hard problems. [citet]\@@bibrefAuthors Phrase1YearPhrase2mahdavi-et-al25pg-pass@k\@@citephrase[\@@citephrase] derived expressions for the pass@$k$ objective and pass@$k$ policy gradients as presented in eq:pass@k-grad and proposed a new pass@$k$ gradient reweighting method with a tunable parameter $γ$ different from $k$. [citet]\@@bibrefAuthors Phrase1YearPhrase2yu25\@@citephrase[\@@citephrase] also derives per-prompt pass@$k$ policy gradients and shows that per-prompt pass@$k$ and pass@1 gradients are positively collinear vectors, questioning the soundness of considering the pass@$k$ objective and advocating for different exploration mechanisms while keeping pass@$k$ as an inference-time evaluation metric. We show in our work that the expected pass@$k$ and pass@1 gradients over the prompt distribution may not necessarily be positively collinear. [citet]\@@bibrefAuthors Phrase1YearPhrase2chen-et-al25pass@ktraining-exploration\@@citephrase[\@@citephrase] investigated how pass@$k$ model training balances the exploration and exploitation capabilities of large language models. [citet]\@@bibrefAuthors Phrase1YearPhrase2jiang-et-al25risk-sensitive-rl\@@citephrase[\@@citephrase] further considered a different risk-sensitive objective than pass@$k$ based on a soft maximum (log-sum-exp) objective depending on a temperature parameter, providing a knob to trade-off between pass@$k$ and pass@1 performance. [citet]\@@bibrefAuthors Phrase1YearPhrase2peng-et-al25simko\@@citephrase[\@@citephrase] proposed Simple Pass@$K$ Optimization (SimKO) which redistributes gradient updates across the top-$K$ token-level probability candidates to promote exploration and improve the balance between exploitation and exploration. [citet]\@@bibrefAuthors Phrase1YearPhrase2thrampoulidis-et-al25\@@citephrase[\@@citephrase] connects policy gradient algorithms for pass@$k$ optimization [citep][\@@bibrefAuthorsPhrase1Yearmahdavi-et-al25pg-pass@k,tang-et-al25icml,walder-karkhanis25neurips\@@citephrase, ] to advantage shaped GRPO methods [citep][\@@bibrefAuthorsPhrase1Yearchen-et-al25pass@ktraining-exploration,shao-et-al24deepseekmath,deng-et-al25,mroueh25rlvr-grpo\@@citephrase, ]. To the best of our knowledge, prior work on pass@$k$ optimization does not theoretically characterize when and why optimizing pass@$k$ can degrade pass@1 under pass@$k$ policy optimization; we provide such a characterization. Interference. The concept of ray interference has been previously introduced and investigated in multi-task reinforcement learning [citep][\@@bibrefAuthorsPhrase1Yearschaul-et-al19ray-interference\@@citephrase, ]. As an analogy with multi-task learning, tasks can be viewed as prompts (or problems to solve in the special case of mathematical reasoning) in our LLM post-training setting. Instead of composite optimization objectives in the form of a sum over tasks, pass@$k$ objectives are expected probabilities of success over a prompt distribution, which may be discrete or continuous (see eq:pass@k). Our definition of prompt interference (Definition 10000\ def:prompt-interference) parallels [citet]\@@bibrefAuthors Phrase1YearPhrase2schaul-et-al19ray-interference\@@citephrase[\@@citephrase, Definition 10000\ 1], using gradients of pass@1 in our LLM post-training setting rather than gradients of task loss functions in multi-task learning. Related notions of gradient conflict/interference have also been studied in multi-task learning (e.g., [citep]\@@bibrefAuthorsPhrase1Yearyu-et-al20gradient-surgery-mtl\@@citephrase, ). To extend the multi-task learning analogy discussed in [cite]\@@bibrefAuthors Phrase1YearPhrase2qu-et-al26pope\@@citephrase[\@@citephrase], tasks can also be thought of as clusters of prompts labeled by degree of difficulty. Inspired by [citet]\@@bibrefAuthors Phrase1YearPhrase2schaul-et-al19ray-interference\@@citephrase[\@@citephrase], [citet]\@@bibrefAuthors Phrase1YearPhrase2qu-et-al26pope\@@citephrase[\@@citephrase] recently investigated learning to reason on hard problems when no correct rollouts are sampled and argue that pass@$k$ optimization does not address this challenge. They emphasize the role of ray interference in hindering learning on heterogeneous prompt mixtures; in particular, Section 10000\ 3.2 discusses negative interference between easy and hard problems, and how pass@$k$ policy optimization does not mitigate its consequences. Conceptually, [citet]\@@bibrefAuthors Phrase1YearPhrase2qu-et-al26pope\@@citephrase[\@@citephrase] study interference as an exploration/solvability bottleneck on heterogeneous mixtures, while we analyze the pass@$k$ objective itself and show how prompt-level gradient conflict can induce an explicit pass@$k$--pass@1 trade-off. Our findings are consistent with the empirical insights and intuitions in [cite][\@@bibrefAuthorsPhrase1Yearqu-et-al26pope\@@citephrase, , Section 10000\ 3.2]. We offer a complementary theoretical view by formalizing prompt interference for LLM post-training and providing conditions under which pass@1 degradation can occur under pass@$k$ policy optimization via prompt-level gradient conflict (see Proposition 10000\ prop:grad-conflict, Corollary 10000\ cor:dominating-neg-interfer-prompts, and Proposition 10000\ prop:pass@1-degradation). Our results also connect to several open questions highlighted by [citet]\@@bibrefAuthors Phrase1YearPhrase2qu-et-al26pope\@@citephrase[\@@citephrase]. While our paper focuses on the theoretical understanding of the pass@$k$ vs pass@1 trade-off and limitations of pass@$k$ policy optimization, [citet]\@@bibrefAuthors Phrase1YearPhrase2qu-et-al26pope\@@citephrase[\@@citephrase] propose a framework (Privileged On-Policy Exploration, POPE) to mitigate ray interference and improve the solvability of hard problems by guiding on-policy exploration. Our theoretical insights may suggest alternative mitigation mechanisms and clarify when such interventions are necessary and effective. \@@numbered@sectionsectiontocConclusion and Future Work In this work, we investigated why pass@$k$ policy optimization can degrade pass@1, showing that pass@$k$ and pass@1 gradients can be conflicting when pass@$k$ weights amplify negative prompt interference. Our findings open several avenues for future work. An immediate direction is to design and evaluate methods to mitigate this conflict issue. More broadly, it would be interesting to investigate more general inference-time objectives beyond pass@$k$ (such as other concave transforms of the probability of success) in view of solving harder reasoning tasks and enhancing exploration with more flexible and targeted prompt reweighting without sacrificing pass@1 performance. Our similarity metric can offer a good starting point for gradient surgery. \@@unnumbered@sectionsectionAcknowledgements We thank Aviral Kumar for pointing us to relevant related work, which helped us to update the related work discussion at the end of Section 10000\ sec:related-work and to add remarks on prior work on interference immediately after Definition 10000\ def:prompt-interference. @newpage @bibliographyreferences @newpage \@@numbered@sectionappendixtocDiscussion of the covariance condition in Proposition~ prop:grad-conflict Introducing the correlation $ _w,a(θ):= cov(w_k,θ(x),a_θ(x)) _w(θ)· _a(θ)$ where $ _w(θ), _a(θ)$ are respectively the standard deviations of $w_k,θ(x)$ and $a_θ(x)$, we can then rewrite condition eq:conflict-2 equivalently as: equation _w,a(θ)<- E[w_k,θ(x)]·\|∇ J_1(θ)\|^2 _w(θ)· _a(θ)\,. equation This condition emphasizes that conflict requires strong negative correlation between hardness weights and agreement scores as well as sufficient variance. If the weights barely vary ($ _w(θ) 0$), pass@$k$ cannot strongly shift focus across prompts and conflict is harder. If agreement scores barely vary ($ _a(θ) 0$) there is no meaningful conflict structure to exploit that can lead to misaligned gradients. \@@numbered@sectionappendixtocA kernel view of gradient conflict beyond $k=1$ In this section, we further investigate the gradient conflict phenomenon between pass@$k$ and pass@$m$ for any $k,m≥ 1$. Using the kernel 10000\ $ _θ$ defined in eq:kernel-def-1, we now relate pass@$k$ and pass@$m$ gradients for any $k,m≥ 1$ using eq:pass@k-grad to obtain: equation ∇ J_k(θ),∇ J_m(θ) =E_x,x [w_k,θ(x)w_m,θ(x ) _θ(x,x )]\,. equation This identity separates two factors: (1) where each objective focuses via the weights $w_k,θ(x),w_m,θ(x )$ with prompt reweighting toward `hard' prompts as $k$ grows; and (2) whether those regions interfere via the sign structure of $ _θ(x,x )$ which is independent of the parameter $k$. Conflict between pass@$k$ and pass@$m$ gradients happens when the objectives emphasize prompt regions whose cross-similarity (encoded by the kernel) is negative. The following result which is an immediate consequence of eq:inner-grad-k-grad-m shows that conflict requires interference. proposition[No interference regime: No negative transfer $ $ no conflict]For any policy parameter $θ ^d$, if $ _θ(x,x )≥ 0$ for $D×D$ a.e. $(x,x )$ then, $ ∇ J_k(θ),∇ J_m(θ) ≥ 0$ for all $k,m≥ 1\,.$ proposition If pairwise prompt similarities are nonnegative on the support of the prompt distribution 10000\ $D$ then all pass@$k$ objectives are mutually aligned. Therefore, conflict can only occur when pairwise similarities are not always nonnegative and depends on how the weights $w_k,θ,w_m,θ$ shift mass. In other words, pass@$k$ objectives cannot conflict unless there is negative transfer between some prompts. \@@numbered@sectionappendixtocProofs \@@numbered@sectionsubsectiontocProof of Proposition~ prop:grad-conflict: Gradients conflict characterization \@proof First recall from eq:pass@k-grad that for any $k ≥ 1$andany$θ ^d$, equation∇ J_k(θ)=E_x [w_k(p_θ(x))∇ p_θ(x)]=E_x [w_k(p_θ(x))∇ J_1(x;θ)]\,. equationThenitimmediatelyfollowsthat: aligned ∇ J_k(θ),∇ J_1(θ) &= _x [w_k(p_θ(x))∇ J_1(x;θ)],∇ J_1(θ) \\ &=E_x [ w_k(p_θ(x))∇ J_1(x;θ),∇ J_1(θ) ]\\ &=E_x [w_k(p_θ(x))a_θ(x)]\,, alignedwherewerecallthat$a_θ(x) := ∇ J_1(x;θ), ∇ J_1(θ) $\,.Bydefinitionofthecovariancebetweentworeal-valuedrandomvariables: aligned cov(w_k,θ(x),a_θ(x))&=E_x [w_k,θ(x)· a_θ(x)]-E_x [w_k,θ(x)]·E_x [a_θ(x)]\\ &= ∇ J_k(θ),∇ J_1(θ) -E_x [w_k,θ(x)]·\|∇ J_1(θ)\|^2\,, alignedwherethesecondequalityfollowsfromusing eq:inner-prod-proofandnoticingthat$E_x [ a_θ(x)] = \|∇ J_1(θ)\|^2$bydefinitionof$a_θ(x)\,.$Rearrangingtheaboveidentitythengives: equation ∇ J_k(θ),∇ J_1(θ) =E_x [w_k,θ(x)]·\|∇ J_1(θ)\|^2+cov(w_k,θ(x),a_θ(x))\,. equationAsaconsequenceof eq:inner-prod-proofand eq:covar-inner-prod,wehavethat$ ∇ J_k(θ), ∇ J_1(θ) < 0$ifandonlyif:$E_x [w_k(p_θ(x)) a_θ(x)] < 0$whichisalsoequivalenttothecondition: equationcov(w_k,θ(x),a_θ(x))<-E[w_k,θ(x)]·\|∇ J_1(θ)\|^2\,. equation @proof \@@numbered@sectionsubsectiontocProof of Corollary~ cor:dominating-neg-interfer-prompts: Dominating negatively interfering prompts \@proof First, Proposition 10000\ prop:grad-conflict yields: equation ∇ J_k(θ),∇ J_1(θ) =E\! [ _k,θ(x)\,a_θ(x) ]\,. equation We decompose this inner product into 2 regions: the $m$-strongly negatively interfering region 10000\ $X_-(θ,m)$ and its complement, equation ∇ J_k(θ),\ ∇ J_1(θ) =E\! [ _k,θ(x)\,a_θ(x) ]=E\! [ _k,θ(x)\,a_θ(x)\, 1_\x _-(θ,m)\ ]+E\! [ _k,θ(x)\,a_θ(x)\, 1_\x _-(θ,m)\ ]\,. equation We now upperbound each one of the above terms. For the first term, by definition of the set of prompts $X_-(θ,m)$, we have $a_θ(x)≤-m$ for any $x _-(θ,m)\,.$ As for the second term, we show that for any $x _-(θ,m),$ $|a_θ(x)|≤ G^2$ using Assumption 10000\ as:E-LS as follows. Bounding $a_θ(x).$ Using the definition of the agreement score $a_θ(x)$ (see eq:alignment-score) together with Cauchy-Schwarz inequality, we have equationa_θ(x)= ∇ J_1(x;θ),∇ J_1(θ) ≤\|∇ J_1(x;θ)\|·\|E[∇ J_1(x;θ)]\|≤\|∇ J_1(x;θ)\|·E[\|∇ J_1(x;θ)\|]\,. equation Then recalling the policy gradient expression of $∇ J_1(x;θ)$ in eq:pass@1pg and using Jensen's inequality, we have: equation\|∇ J_1(x;θ)\|^2=\|E_y _θ(·|x)[r(x,y)s_θ(x,y)]\|^2 _y _θ(·|x)[\|s_θ(x,y)\|^2]≤ G^2\,, equation where the last inequality follows from using Assumption 10000\ as:E-LS. Using this inequality in eq:proof-bound-atheta yields the desired inequality: equation∀ x , |a_θ(x)|≤ G^2\,. equation Back to eq:inner-prod-decomp and using the inequalities established above, we conclude the proof: equation ∇ J_k(θ),\ ∇ J_1(θ) ≤-δ(θ)\,, equation where $δ(θ):=mW_-(k,θ)-G^2W_+(k,θ)$ and the aggregated weights 10000\ $W_-(k,θ),W_+(k,θ)$ are defined in eq:aggregated-weights. If in addition $δ(θ)<0$, then $ ∇ J_k(θ),∇ J_1(θ) ≤-δ(θ)<0\,.$ @proof \@@numbered@sectionsubsectiontocProof of Proposition~ prop:phase-transition-in-k: Influence of $k$ \@proof Recall from Proposition 10000\ prop:grad-conflict that: equation ∇ J_k(θ),∇ J_1(θ) =E\! [ _k,θ(x)\,a_θ(x) ]\,. equation Using again the same decomposition as in eq:inner-prod-decomp, we have: equation ∇ J_k(θ),\ ∇ J_1(θ) =E\! [ _k,θ(x)\,a_θ(x) ]=E\! [ _k,θ(x)\,a_θ(x)\, 1_\x _-(θ,m)\ ]+E\! [ _k,θ(x)\,a_θ(x)\, 1_\x _-(θ,m)\ ]\,. equation Depending on the prompt region, we have two distinct ways to bound the pass@$k$ weights: enumerate[label=( *),leftmargin=0pt,itemindent=0.0pt,align=left] @item@For $x (m,θ)$, we have by assumption, $p_θ(x)≤ $, hence $ _k,θ(x)\ ≥\ k(1- )^k-1.$ @item@For $x (m,θ)$ we have by assumption, $p_θ(x)≥δ$, hence $ _k,θ(x)\ ≤\ k(1-δ)^k-1.$ enumerate Observing in addition to both items above that $a_θ(x)≤-m$ for any $x _-(θ,m)$ and $a_θ(x)≤ G^2$ for any $x $ (as shown in eq:bound-agreement-score under Assumption 10000\ as:E-LS) and hence for any $x _-(θ,m)$ in particular, we obtain from eq:inner-prod-decomp2 the following inequality: equation ∇ J_k(θ),∇ J_1(θ) \ ≤\ -\,k(1- )^k-1\,m\,q\ +\ k(1-δ)^k-1\,G^2\,(1-q)\,, equation where we also use the assumption 10000\ $q_θ>q\,.$ Thus it suffices to have $$k(1-δ)^k-1G^2(1-q)\ <\ k(1- )^k-1mq,$$ to ensure that $ ∇ J_k(θ),∇ J_1(θ) <0\,.$ The above condition is equivalent to: $$ G^2(1-q)mq\ <\ ( 1- 1-δ )^k-1.$$ Taking $ $ in the above inequality, recalling that $1- >1-δ$ and reordering the inequality yields: equationk\ >k :=\ 1\ +\ \! ( (1-q)\,G^2q\,m ) \! ( 1- 1-δ )\,, equation which concludes the proof. @proof \@@numbered@sectionsubsectiontocLemma~ lem:smoothness-pass@k: Smoothness of pass@$k$ lemma[Smoothness of pass@$k$]Under Assumption 10000\ as:E-LS, for all $k≥ 1$, the pass@$k$ objective $J_k$ is $L_k$-smooth with $L_k:=k^2G^2+kF,$ i.e. for all $θ,θ ^d,$ equation*|J_k(θ )-J_k(θ)- ∇ J_k(θ),θ -θ |≤ L_k2\|θ -θ\|_2^2\,. equation* lemma \@proof To show smoothness of the pass@$k$ objective $J_k$, we bound its Hessian $∇^2J_k(θ)$ uniformly in the policy parameter 10000\ $θ$ using Assumption 10000\ as:E-LS. Since $∇ J_k(θ)=E_x [w_k(p_θ(x))∇ p_θ(x)]$ as shown in eq:pass@k-grad, the chain rule gives the Hessian of $J_k$ for every $θ ^d$, equation∇^2J_k(θ)=E_x [ _k (p_θ(x))\,∇ p_θ(x)\,∇ p_θ(x) + _k(p_θ(x))\,∇^2p_θ(x) ]. equation Taking the operator norm and using the triangular inequality yields: equation \|∇^2J_k(θ) \| _x [ | _k (p_θ(x)) |·\,\|∇ p_θ(x)\|^2+ | _k(p_θ(x)) |·\,\|∇^2p_θ(x)\| ]. equation In the rest of the proof, we bound each term separately. \@@unnumbered@sectionparagraphtocStep 1: Bounding $\|∇ p_θ(x)\|^2$. Recall from eq:pass@1pg that the pass@1 policy gradient is given by: equation∇ p_θ(x)=E_y _θ(·|x)\! [r(x,y)\,∇ _θ(y|x) ]. equation By Jensen's inequality and since $r(x,y)∈\0,1\$, we have \@@amsalign\|∇ p_θ(x)\|^2& _y _θ(·|x) [r(x,y)^2\,\|∇ _θ(y|x)\|^2 ] _y _θ(·|x) [\|∇ _θ(y|x)\|^2 ]≤ G^2\,, where the last bound follows from Assumption 10000\ as:E-LS. \@@unnumbered@sectionparagraphtocStep 2: Bounding $\|∇^2 p_θ(x)\|$. Differentiating the pass@1 policy gradient, we obtain: $$∇^2p_θ(x)=E_y _θ(·|x)\! [r(x,y)\,∇ _θ(y|x)\,∇ _θ(y|x) +r(x,y)\,∇^2 _θ(y|x) ].$$ Taking the operator norm and using the triangle inequality, we obtain: \@@amsalign\|∇^2p_θ(x)\|& _y _θ(·|x) [|r(x,y)|·\,\|∇ _θ(y|x)\|^2 ]+E_y _θ(·|x) [|r(x,y)|·\,\|∇^2 _θ(y|x)\| ]\\ & _y _θ(·|x) [\|∇ _θ(y|x)\|^2 ]+E_y _θ(·|x) [\|∇^2 _θ(y|x)\| ]≤ G^2+F\,, where the last inequality again follows from using Assumption 10000\ as:E-LS. \@@unnumbered@sectionparagraphtocStep 3: Bounding weights $ _k'$ and $ _k$. By definition of the weights $w_k(p)=f_k (p)=k(1-p)^k-1$ for any $p∈[0,1]$. Therefore it follows that $ _k (p)=-k(k-1)(1-p)^k-2$ for $k≥ 2$. Hence, for any $p∈[0,1]$ and any $k≥ 2$, equation| _k(p)|≤ k\,, | _k (p)|≤ k(k-1)\,. equation \@@unnumbered@sectionparagraphtocConclusion. Plugging eq:grad-e-bound-- eq:hess-e-bound and the bounds on $ _k, _k $ eq:weights-bounds into eq:hess-norm-split yields $$\|∇^2J_k(θ)\|≤ k(k-1)\,G^2+k\,(G^2+F)≤ k^2G^2+kF.$$ Finally, boundedness of the Hessian norm implies that $J_k$ is $L_k$-smooth with $L_k=k^2G^2+kF$, i.e., for any $θ,θ ^d\,,$ $$ |J_k(θ )-J_k(θ)- ∇ J_k(θ),θ -θ |≤ L_k2\|θ -θ\|^2.$$ @proof \@@numbered@sectionsubsectiontocProof of Proposition~ prop:pass@1-degradation: Pass@$1$ degradation under pass@$k$ updates \@proof We prove the decrease of pass@1 and increase of pass@$k$ under pass@$k$ policy gradient ascent separately. Recall the policy update rule with step size $η>0$: equationθ^+=θ+η\,∇ J_k(θ). equation (i) Pass@1 descent. By $L_1$-smoothness of $J_1$ (Lemma 10000\ lem:smoothness-pass@k) and using the policy update rule 10000\ eq:pass@k-gd, we have: \@@amsalign J_1(θ^+)&≤ J_1(θ)+ ∇ J_1(θ),θ^+-θ + L_12\|θ^+-θ\|^2\\ &=J_1(θ)+η\, ∇ J_1(θ),∇ J_k(θ) + L_1η^22\|∇ J_k(θ)\|^2. We now bound each one of the last two terms in the above inequality. Bounding $\|∇ J_k(θ)\|^2$. Recall the pass@$k$ policy gradient expression $∇ J_k(θ)=E_x\! [ _θ(x)\,∇ J_1(x;θ) ]$, we have by Jensen and Cauchy-Schwarz inequalities: equation\|∇ J_k(θ)\|^2= \|E_x\! [ _θ(x)\,∇ J_1(x;θ) ] \|^2\;≤\;E_x\! [| _θ(x)|^2\,\|∇ J_1(x;θ)\|^2 ]≤ k^2G^2\,, equation where the last inequality follows from using the pass@1 policy gradient bound in eq:pass@1-grad-bound together with immediate weight bound $| _θ(x)|≤ k$ and the fact that the reward is binary (this can be generalized to a continuous uniformly bounded reward function). Bounding $ ∇ J_1(θ),∇ J_k(θ) $. By Corollary 10000\ cor:dominating-neg-interfer-prompts, we have equation ∇ J_1(θ),∇ J_k(θ) ≤-δ(θ)\,. equation Combining the bounds 10000\ eq:grad-bound-1 and eq:innerprod-bound, we immediately obtain from eq:smooth-J1: equationJ_1(θ^+)≤ J_1(θ)-ηδ(θ)+C_2η^2\,, equation where $C_2:= L_1k^2G^22$ and $L_1=G^2+F$ as shown in Lemma 10000\ lem:smoothness-pass@k, $G,F$ being smoothness constants defined in Assumption 10000\ as:E-LS. Hence, if $δ(θ)>0$ and $0<η≤ _+:= δ(θ)C_2,$ then $-ηδ(θ)+C_2η^2<0$ and it follows that equationJ_1(θ^+)≤ J_1(θ)-ηδ(θ)+C_2η^2<J_1(θ)\,, equation which is the first desired inequality. (i) Pass@$k$ ascent. As for pass@$k$ under pass@$k$ policy gradient ascent, the proof is a standard application of smoothness in optimization. We provide a brief proof in our setting for completeness. Using $L_k$-smoothness of the pass@$k$ objective shown in Lemma 10000\ lem:smoothness-pass@k together with the policy update rule 10000\ eq:pass@k-gd, we have: \@@amsalign J_k(θ^+)&≥ J_k(θ)+ ∇ J_k(θ),θ^+-θ - L_k2\|θ^+-θ\|^2\\ &=J_k(θ)+η\|∇ J_k(θ)\|^2- L_kη^22\|∇ J_k(θ)\|^2\\ &=J_k(θ)+η (1- L_kη2 )\|∇ J_k(θ)\|^2. We pick then $η≤ 1L_k$ to obtain the second desired inequality: equationJ_k(θ^+)≥ J_k(θ)+ η2\|∇ J_k(θ)\|^2≥ J_k(θ)\,. equation Overall, a sufficient step-size condition to guarantee simultaneous increase of pass@$k$ and decrease of pass@1 (i.e., in particular $J_1(θ^+)<J_1(θ)$ and $J_k(θ^+)>J_k(θ)$ when $∇ J_k(θ)≠ 0$) is: equation0<η≤ ( δ(θ)C_2, 1L_k )\,, equation and this concludes the proof. @proof @newpage \@@numbered@sectionappendixtocAdditional Experiments \@@numbered@sectionsubsectiontocMore details regarding experimental setting Sampling configuration. For each problem, we generate $k=32$ independent responses using temperature sampling with temperature $T=0.7$ and nucleus sampling with $p=0.95$. Responses are evaluated using exact match against ground truth answers and binary rewards indicate correctness. \@@numbered@sectionsubsectiontocThreshold robustness analysis Figure 10000\ fig:all_threshold_configs_llama shows gradient misalignment across all 6 additional threshold configurations for DeepSeek-R1-Distill-Llama-8B (beyond the main text example of $ _1=0.85$, $ _2=0.10$). All combinations exhibit negative inner products (range: $-0.49$ to $-0.65$), demonstrating that the misalignment phenomenon is robust to threshold choices. figure[p] subfigure310.4979pt [width=345.0pt]Figs/deepseek_llama_threshold_variations_2000_k32/1x3_delta1_0.80_delta2_0.10.pdf \@@toccaption @tag[ ](a)$ _1=0.80$, $ _2=0.10$ (636 prompts: 86 hard, 550 easy, IP=$-0.652$)\@@caption @tag[ ] (a) $ _1=0.80$, $ _2=0.10$ (636 prompts: 86 hard, 550 easy, IP=$-0.652$)\@add@centering subfigure 3.98611pt subfigure310.4979pt [width=345.0pt]Figs/deepseek_llama_threshold_variations_2000_k32/1x3_delta1_0.80_delta2_0.15.pdf \@@toccaption @tag[ ](b)$ _1=0.80$, $ _2=0.15$ (647 prompts: 97 hard, 550 easy, IP=$-0.621$)\@@caption @tag[ ] (b) $ _1=0.80$, $ _2=0.15$ (647 prompts: 97 hard, 550 easy, IP=$-0.621$)\@add@centering subfigure 3.98611pt subfigure293.2521pt [width=345.0pt]Figs/deepseek_llama_threshold_variations_2000_k32/1x3_delta1_0.85_delta2_0.05.pdf \@@toccaption @tag[ ](c)$ _1=0.85$, $ _2=0.05$ (502 prompts: 38 hard, 464 easy, IP=$-0.551$)\@@caption @tag[ ] (c) $ _1=0.85$, $ _2=0.05$ (502 prompts: 38 hard, 464 easy, IP=$-0.551$)\@add@centering subfigure 3.98611pt subfigure293.2521pt [width=345.0pt]Figs/deepseek_llama_threshold_variations_2000_k32/1x3_delta1_0.90_delta2_0.10.pdf \@@toccaption @tag[ ](d)$ _1=0.90$, $ _2=0.10$ (505 prompts: 86 hard, 419 easy, IP=$-0.527$)\@@caption @tag[ ] (d) $ _1=0.90$, $ _2=0.10$ (505 prompts: 86 hard, 419 easy, IP=$-0.527$)\@add@centering subfigure \@@toccaption @tag[ ]8 Gradient conflict across all threshold configurations (DeepSeek-R1-Distill-Llama-8B). Each panel shows agreement scores, weights, and weighted contributions for a different $( _1, _2)$ configuration. All exhibit negative inner products, confirming robust misalignment across thresholds. The consistent downward shift from unweighted to weighted means shows pass@$k$ optimization systematically opposes pass@1 improvement. \@@caption @tag[: ] Figure 8 Gradient conflict across all threshold configurations (DeepSeek-R1-Distill-Llama-8B). Each panel shows agreement scores, weights, and weighted contributions for a different $( _1, _2)$ configuration. All exhibit negative inner products, confirming robust misalignment across thresholds. The consistent downward shift from unweighted to weighted means shows pass@$k$ optimization systematically opposes pass@1 improvement. \@add@centering figure \@add@PDF@RDFa@triples document (48)