Paper deep dive

Peer-Predictive Self-Training for Language Model Reasoning

Shi Feng, Hanlin Zhang, Fan Nie, Sham Kakade, Yiling Chen

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 64

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 4/18/2026, 1:26:47 AM

Summary

Peer-Predictive Self-Training (PST) is a label-free fine-tuning framework for language models that enables collaborative self-improvement without external supervision. By generating responses sequentially, models create an aggregated output that serves as an internal training target. The framework uses Pointwise Mutual Information (PMI) to quantify the informativeness of individual responses relative to the aggregate, using this signal to adaptively scale training updates. PST demonstrates consistent improvements in mathematical reasoning benchmarks (SimulEq, Math500, MultiArith) across various model families (Gemma-2, LLaMA-3.2, Qwen-2.5) by leveraging cross-model interactions and the inherent asymmetry between generation and verification.

Entities (8)

Gemma-2-2B · language-model · 100%Llama-3.2-1B · language-model · 100%MATH500 · benchmark · 100%MultiArith · benchmark · 100%Peer-Predictive Self-Training · framework · 100%Qwen-2.5-1.5B · language-model · 100%SimulEq · benchmark · 100%Pointwise Mutual Information · metric · 95%

Relation Signals (3)

Peer-Predictive Self-Training → usesmetric → Pointwise Mutual Information

confidence 100% · We measure how informative each intermediate response is about the aggregate using pointwise mutual information (PMI)

Peer-Predictive Self-Training → improvesperformanceon → SimulEq

confidence 95% · PST improves exact-match accuracy by 2.2 to 4.3 percentage points across... SimulEq

Gemma-2-2B → trainedvia → Peer-Predictive Self-Training

confidence 95% · PST improves exact-match accuracy... across Gemma-2-2B

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. We propose Peer-Predictive Self-Training (PST), a label-free fine-tuning framework in which multiple language models improve collaboratively by leveraging a cross-model aggregated response as an internal training signal. Given a prompt question, the models generate responses sequentially; the final aggregated answer, often more reliable than individual responses in practice, serves as an internal target for learning. We measure how informative each intermediate response is about the aggregate using pointwise mutual information (PMI), and use this signal to scale self-training updates. Responses already aligned with the aggregate are updated less, while less informative or misaligned responses are updated more. On mathematical reasoning benchmarks (SimulEq, Math500, and MultiArith), PST improves exact-match accuracy by 2.2 to 4.3 percentage points across Gemma-2-2B, LLaMA-3.2-1B, and Qwen-2.5-1.5B, and reduces the average generator-verifier gap (GV-Gap) by 26 to 40 percent, while requiring no external supervision or teacher-student hierarchy and relying solely on cross-model interactions. These results suggest that cross-model generations and peer-predictive feedback can serve as an effective approach for self-supervised training.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

63,712 characters extracted from source content.

Expand or collapse full text

Peer-Predictive Self-Training for Language Model Reasoning Shi Feng Harvard University shifeng-thu@outlook.com &Hanlin Zhang11footnotemark: 1 Harvard University hanlinzhang@g.harvard.edu &Fan Nie Stanford University niefan@stanford.edu &Sham Kakade Harvard University sham@seas.harvard.edu &Yiling Chen Harvard University yiling@seas.harvard.edu Equal contribution. Abstract Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. We propose Peer-Predictive Self-Training (PST), a label-free fine-tuning framework in which multiple language models improve collaboratively by leveraging a cross-model aggregated response as an internal training signal. Given a prompt question, the models generate responses sequentially; the final aggregated answer–often more reliable than individual responses in practice–serves as an internal target for learning. We measure how informative each intermediate response is about the aggregate using pointwise mutual information (PMI), and use this signal to scale self-training updates. Responses already aligned with the aggregate are updated less, while less informative or misaligned responses are updated more. On mathematical reasoning benchmarks (SimulEq, Math500, and MultiArith), PST improves exact-match accuracy by 2.2–4.3 percentage points across Gemma-2-2B, LLaMA-3.2-1B, and Qwen-2.5-1.5B, and reduces the average generator–verifier gap (GV-Gap) by 26–40%, while requiring no external supervision or teacher–student hierarchy and relying solely on cross-model interactions. These results suggest that cross-model generations and peer-predictive feedback can serve as an effective approach for self-supervised training. 1 Introduction Continued self-improvement of large language models (LLMs) without external supervision is a central open problem. Many successful post-training pipelines rely on labeled data, explicit reward models, or carefully engineered feedback signals (Wang et al., 2025; Lu et al., 2025). While effective in controlled settings, these methods fundamentally limit scalability and adaptability, especially when ground truth is expensive, incomplete, or unavailable, and when improvement must proceed continually without assuming access to stronger external supervision. A growing body of recent work therefore explores self-improvement driven by internally generated signals, including self-training and multi-model interaction (Song et al., 2024; Sun et al., 2025). These approaches have proven particularly effective for reasoning-oriented tasks, where correct solutions often require multi-step logical consistency rather than surface-level pattern matching (Wang et al., 2024; 2025). In these settings, models may struggle to reliably generate correct answers, yet still retain the ability to recognize or validate correct reasoning trajectories when presented with candidate solutions, reflecting a fundamental asymmetry between generation and verification (Song et al., 2024; Sun et al., 2025). This verification–generation asymmetry makes reasoning problems especially amenable to self-training based on internal feedback signals. Recent work further shows that homogeneous model populations can exhibit emergent coordination and collective behavior through open-ended interaction, even in the absence of external labels (Jiang et al., 2025). However, how to elicit reliable and stable supervision signals in a fully unsupervised manner remains unresolved, particularly in the continual self-improvement regime where error accumulation and confirmation bias can severely limit long-term gains (Song et al., 2024). A key insight from both learning theory and economics is that reliability can emerge from aggregation, a phenomenon often referred to as the wisdom of crowds. Across domains, aggregating multiple independent or weakly correlated predictions can effectively pool dispersed information and yield more accurate outcomes than relying on a single predictor (Song et al., 2024; Lu et al., 2025; Wolfers and Zitzewitz, 2004; Surowiecki, 2004). In the context of language models, different models—or different reasoning trajectories produced by the same model—may encode complementary partial information about the underlying solution, such that aggregation implicitly filters noise while amplifying consistent and informative signals. As a result, the aggregated output is frequently more reliable than most individual responses. Recent theoretical work formalizes this intuition through the coverage principle, which argues that post-training or selection-based improvement is possible as long as the base model assigns non-negligible probability mass to correct solutions, even if they are rare (Chen et al., 2025). Under this view, the central challenge is not the absence of correct solutions, but how to identify and amplify them without access to ground truth. This challenge is closely related to prior work on information elicitation under unverifiable outcomes, which studies how informative signals can be extracted from cross-agent statistical structure without relying on ground truth Miller et al. (2005); Shnayder et al. (2016); Kong and Schoenebeck (2019); Kong (2020); Feng et al. (2022). Rather than external labels, such approaches leverage mutual predictiveness across agents, and recent work shows that mutual information–based signals can reliably evaluate LLM judgments in no-gold-standard settings (Xu et al., 2024). Taken together, these observations suggest a natural synthesis of aggregation and peer-based evaluation: a pool of heterogeneous language models can not only aggregate dispersed beliefs into a stronger collective answer, but also provide informative cross-model signals about answer quality even when no external supervision is available. Building on this perspective, we propose Peer-Predictive Self-Training (PST), a cooperative self-improvement framework for language model reasoning. Given a prompt, multiple language models generate responses sequentially, and the final aggregated response serves as a crowd-based internal reference signal. Rather than treating this aggregate as a hard pseudo-label, PST quantifies how informative each intermediate response is about the final aggregate using pointwise mutual information (PMI). This peer-predictive signal is then used to adaptively modulate the strength of self-training updates: responses that already align with the collective decision receive smaller updates, while less informative or misaligned responses are updated more strongly. The key intuition behind PST is that aggregation induces a form of lightweight verification signal. By pooling diverse reasoning trajectories, the aggregated response often reflects a stronger evaluation signal than any single generator alone, even in the absence of an explicit verifier. Crucially, PST is not an aggregation method, but a mechanism for converting peer-predictive information into supervision signals. PST leverages this verification–generation asymmetry to guide self-improvement, encouraging models to internalize reasoning patterns aligned with the strongest implicit peer feedback. Importantly, PST requires no ground-truth supervision, no reward models, and no fixed teacher–student hierarchy, relying solely on cross-model interaction and peer predictiveness. We evaluate PST on mathematical reasoning benchmarks including MATH-500-Numeric, MultiArith, and SimulEq, using three heterogeneous instruction-tuned model families: Gemma-2-2B, LLaMA-3.2-1B, and Qwen-2.5-1.5B. We intentionally focus on mathematical reasoning, as it provides a clean and objectively verifiable setting for studying self-improvement, allowing us to isolate the effect of the learning mechanism without confounding factors such as subjective evaluation. Across all settings, PST yields consistent improvements of 2.2–4.3 percentage points in exact-match accuracy over the original models and reduces the average generator–verifier gap by 26–40%. While the absolute gains are modest, they are consistent across models of different scales and mathematical reasoning datasets, and are achieved without any external supervision or verifier training. These results demonstrate that peer-predictive feedback provides a simple yet effective foundation for continued, fully unsupervised self-improvement of language models on reasoning tasks. Additional experiments on larger-scale models, including Gemma 3 (4B), LLaMA 3.1 (8B), and Qwen 2.5 (3B), are provided in Appendix C.2, where we observe similar improvements, further validating the effectiveness of PST at larger scales. 2 Related Work Recent work has examined the limits of self-improvement in language models through the lens of generator–verifier interactions. Lu et al. (2025) show that generator–verifier similarity fundamentally constrains gains from self-verification, and that distributional diversity among models is critical for effective verification. This insight directly motivates the use of heterogeneous peers in our framework. Beyond verifier-based selection, a growing body of work studies self-improvement without external supervision. Classical self-training and bootstrapping methods iteratively fine-tune models on their own predictions, but are prone to confirmation bias and error reinforcement when incorrect generations are amplified (Song et al., 2024; Kadlčík and Štefánik, 2024). Recent approaches mitigate this issue by relying on coverage assumptions, arguing that post-training is effective as long as the base model assigns non-negligible probability mass to correct solutions, even if they are rare (Chen et al., 2025). However, these methods typically operate in a single-model regime and do not explicitly exploit cross-model interactions. Recent work has also explored learning from multiple models or peer interactions. For example, Wu et al. (2026) and Luo et al. (2025) study how models can improve by leveraging signals from other models or agents, often through interaction or selection at inference time. These approaches highlight the potential of cross-model learning. In contrast, PST focuses on using peer agreement to derive a quantitative signal that can guide parameter updates during training. Figure 1: Overview of the PST pipeline. Another related line of work explores collective behavior and multi-model interaction. Jiang et al. (2025) study homogeneous populations of language models and show that open-ended interaction can induce emergent coordination, while also revealing limitations due to model similarity. Debate-style or discussion-based methods, as well as aggregation-based methods such as majority voting and self-consistency, rely on explicit comparison rules or external judges to improve performance at inference time, but do not provide a training signal that can be used to update model parameters. In contrast, Peer-Predictive Self-Training (PST) derives supervision implicitly from aggregation and peer predictiveness, without explicit debate, ranking, or teacher–student roles. Our method is closely related to peer prediction and information elicitation mechanisms, which aim to recover truthful signals when ground truth is unavailable Miller et al. (2005); Kong (2020). Recent work demonstrates that mutual information–based signals can reliably evaluate LLM judgments without gold labels (Xu et al., 2024). PST operationalizes this idea in a training loop: pointwise mutual information quantifies how much a model’s response contributes to a stronger collective decision, and this signal directly modulates the strength of self-training updates. From a generator–verifier perspective, PST builds on analyses of generator–verifier gaps in LLM self-improvement (Song et al., 2024; Sun et al., 2025). Prior work emphasizes that improvement requires access to verifiers strictly stronger than the generator. Unlike approaches that assume an explicit verifier, PST induces a stronger evaluation signal through aggregation across peers. Empirically, this leads to reduced generator–verifier gaps after training, and theoretically, to monotone convergence driven by cross-model capability asymmetries. Finally, our theoretical motivation aligns with recent results on the computational asymmetry between generation and verification. Verification can often be implemented by shallow or low-precision transformers, whereas long-horizon planning and generation are computationally harder (Merrill and Sabharwal, 2023; Wang et al., 2024; 2025). PST leverages this asymmetry implicitly: even when individual models struggle to generate correct solutions, their combined verification signals can still provide a reliable self-supervised training target. 3 Peer-Predictive Self-Training (PST) We first introduce our Peer-Predictive Self-Training (PST) framework for multi-model self-improvement. PST is guided by two ideas: (i) wisdom of crowds – a pool of heterogeneous language models can aggregate dispersed “beliefs” and produce an aggregated answer that is usually stronger than individual responses, akin to how prediction markets aggregate information; (i) peer prediction – when no ground truth is available, cross-agent consistency still provides valuable information for answer quality. Concretely, PST scores each model’s response by how informative it is about the final aggregate via pointwise mutual information (PMI) and uses this peer-predictive signal to weigh self-training updates. Figure 1 provides an overview of the PST pipline. Cross-model generation. Let Mkk=1N\M_k\_k=1^N be a set of N≥2N≥ 2 autoregressive language models with parameters θkk=1N\ _k\_k=1^N. At the start of each epoch, we sample a random permutation π of 1,…,N\1,…,N\. The model acting at position i in the resultant order is Mπ(i)M_π(i) with parameters θπ(i) _π(i). Given a question prompt x, the models produce responses sequentially. The i-th model Mπ(i)M_π(i) produces a response yiy_i conditioned on x and all previous responses y<i=(y1,…,yi−1)y_<i=(y_1,…,y_i-1): yi∼pθπ(i)(⋅∣x,y<i).y_i p_ _π(i)(· x,y_<i). We treat the final response yNy_N as an ensemble-aggregated answer, since it is produced after observing all previous candidate solutions. Peer-predictive score via PMI. Taking yNy_N as an internal target for self-training, we want to quantify how much each intermediate response yiy_i helps “predict” the final aggregate yNy_N. For each position i∈1,…,Ni∈\1,…,N\, we compute a pointwise mutual information (PMI) Xu et al. (2025) score under the same model Mπ(i)M_π(i) that produced yiy_i: ri=log⁡pθπ(i)(yN∣x,yi)−log⁡pθπ(i)(yN∣x).r_i= p_ _π(i)(y_N x,y_i)- p_ _π(i)(y_N x). Intuitively, rir_i is large when model Mπ(i)M_π(i)’s own response yiy_i strongly increases its likelihood of producing the final aggregate yNy_N, relative to its baseline from the prompt alone. Algorithm 1 Peer-Predictive Self-Training Input: dataset D; models Mk\M_k\ with params θk\ _k\; temperature τ; epochs E for e=1e=1 to E do Randomly generate a permutation π of 1,2,…N\1,2,… N\ for each prompt x∈x do y<1←∅y_<1← for i=1i=1 to N do sample yi∼pθπ(i)(⋅∣x,y<i)y_i p_ _π(i)(· x,y_<i) y<i+1←(y<i,yi)y_<i+1←(y_<i,y_i) end for Let yNy_N be the final aggregated response for i=1i=1 to N do ri←log⁡pθπ(i)(yN∣x,yi)−log⁡pθπ(i)(yN∣x)r_i← p_ _π(i)(y_N x,y_i)- p_ _π(i)(y_N x), αi←σ(−ri/τ) _i←σ(-r_i/τ) ℒCE(i)←−∑tlog⁡pθπ(i)(yi,t+1∣x,y<i,yi,≤t)L^(i)_CE←- _t p_ _π(i)(y_i,t+1 x,y_<i,y_i,≤ t), ℒPST(i)←αi⋅ℒCE(i)L^(i)_PST← _i·L^(i)_CE Update parameters θπ(i) _π(i) by backpropagating ℒPST(i)L^(i)_PST with AdamW end for end for end for Output: fine-tuned parameters θk\ _k\ We convert rir_i into a non-negative update weight using a temperature-controlled sigmoid: αi=σ(−riτ),σ(u)=11+e−u, _i=σ\! (- r_iτ ), σ(u)= 11+e^-u, where temperature τ>0τ>0 controls sensitivity. Higher PMI therefore yields smaller αi _i, resulting in a smaller update. The idea is that a model whose response already aligns with the aggregated solution should change less, which encourages a natural division of labor instead of redundant reasoning. Weighted self-training. Each model Mπ(i)M_π(i) is trained on its own generated trajectory yi=(yi,1,…,yi,Ti)y_i=(y_i,1,…,y_i,T_i) with next-token cross-entropy loss: ℒCE(i)=−∑t=1Ti−1log⁡pθπ(i)(yi,t+1∣x,y<i,yi,≤t).L_CE^(i)=- _t=1^T_i-1 p_ _π(i)\! (y_i,t+1 x,y_<i,y_i,≤ t ). We scale this loss by the peer-predictive weight: ℒPST(i)=αiℒCE(i).L_PST^(i)= _i\,L_CE^(i). We then backpropagate through ℒPST(i)L_PST^(i) and update parameters θπ(i) _π(i) (e.g., with AdamW). The permutation π is resampled each epoch so that different models take on different positions—including the final “aggregator” position—over training. Algorithm 1 summarizes the PST framework. The final response yNy_N can be viewed as an aggregated answer, as the last model conditions on all previous responses and thus integrates information across models without requiring an external aggregation module. Randomizing the permutation ensures that each model is equally likely to appear at any position in expectation, mitigating position bias. 4 Experiments (a) SimulEq (b) MATH-500-Numeric (c) MultiArith Figure 2: Accuracy before and after fine-tuning on SimulEq, MATH-500-Numeric, and MultiArith. Each panel reports pre- and post-fine-tuning by PST performance for Gemma-2-2B, LLaMA-3.2-1B, and Qwen-2.5-1.5B. Experimental Setup. We evaluate PST using three instruction-tuned LMs with different architectures and sizes: Gemma-2-2B-Instruct Google (2024), LLaMA-3.2-1B-Instruct Meta (2024), and Qwen2.5-1.5B-Instruct Alibaba Cloud and Qwen Team (2024). We fine-tune each base model using LoRA adapters Hu et al. (2022), while keeping all original model parameters frozen. Training updates are applied exclusively to the LoRA parameters. Model optimization is carried out using AdamW Loshchilov and Hutter (2019) with a learning rate of 1×10−61× 10^-6 and weight decay 0.010.01. We adopt a cosine-annealing learning-rate schedule, with the maximum annealing horizon set to the total number of training epochs. Forward computation is performed under bfloat16 automatic mixed precision. For each training instance, models generate responses sequentially. PMI scores rir_i are computed for all non-final models, and the training objective for model Mπ(i)M_π(i) is defined as a scaled next-token cross-entropy loss αiℒCE _iL_CE, where αi=σ(−ri/τ) _i=σ(-r_i/τ) and τ=3.0τ=3.0. Gradients are clipped to a global norm of 1.01.0, and optimization steps are skipped whenever non-finite gradients are detected. At the beginning of each epoch, the ordering of models is reshuffled so that different models assume the final (aggregator) role across epochs. Each model maintains its own optimizer and scheduler, and training is conducted for a total of 55 epochs. Datasets. We consider 3 standard reasoning benchmarks: • SimulEq Kushman et al. (2014), a collection of problems focused on solving systems of simultaneous equations of variable complexity. • MATH-500-Numeric Hendrycks et al. (2021), a challenging benchmark of mathematical problem solving derived from the broader MATH dataset that contains 500 problems with integer answers. • MultiArith Roy and Roth (2016), a dataset of multi-step arithmetic word problems requiring combination of basic operations such as addition, subtraction, multiplication, and division. These benchmarks provide a clean and verifiable setting for isolating the effect of the learning mechanism. All datasets are formatted uniformly as Question: <problem> and presented to all models sequentially. We record each model’s raw response (given only the question as a prompt) before and after fine-tuning using PST, and compute the exact-match accuracy for each individual model, with error bars. Evaluation and Figures. After five epochs of PST fine-tuning, we evaluate model performance on SimulEq, MATH-500-Numeric, and MultiArith. Figure 2 shows paired bars representing pre- and post-fine-tuning accuracy for each model. In all benchmarks, fine-tuning using PST consistently improves exact-match accuracy across all models. Averaged over models, PST yields a mean accuracy improvement of approximately 3.0 percentage points on SimulEq, 2.2 percentage points on MATH-500-Numeric, and 4.3 percentage points on MultiArith. These gains are obtained using standard fine-tuning settings without external supervision, indicating that PMI-based modulation alone provides a stable and effective training signal. These results indicate that PST effectively enhances reasoning reliability across heterogeneous models by allocating larger updates to less informative responses and smaller updates to highly predictive ones. 5 Why Does PST Work? 5.1 Intuition PST can be viewed as self-training with an endogenously improved reference signal and an information-weighted update rule. For each prompt, multiple models generate candidate solutions sequentially, and the final response (conditioned on all prior candidates) acts as an aggregated reference. PST then updates each model on its own trajectory, but scales the update magnitude using a PMI-based signal. Intuitively, if a model’s response already makes the aggregate highly likely (high PMI), then it is already aligned with the crowd and should change little; if it is weakly predictive or misaligned (low PMI), the model receives a larger update, pushing it toward behaviors that better match the aggregated decision. A helpful lens is to view each model as playing both a generator and a verifier role. During sequential generation, model Mπ(i)M_π(i) produces yi∼pθπ(i)(⋅∣x,y<i),y_i p_ _π(i)(· x,y_<i), where conditioning on y<iy_<i implicitly reflects an evaluation of earlier candidates before producing yiy_i. The final output yNy_N pools information from all peers and can be interpreted as the result of applying the strongest available implicit verification signal within the group to the set of proposed candidates. Why can yNy_N be a stronger training target than any single model’s output? As discussed in Appendix B, verification and generation can differ sharply in computational difficulty: when abstracting a reasoning problem as a path-finding problem on a directed graph, verifying a proposed path can lie in a low-depth circuit class (TC0TC^0), while constructing a correct path can be substantially harder (PP- or NLNL-hard). This verification–generation asymmetry implies that even when individual models struggle to generate correct solutions from scratch, they may still collectively provide a comparatively strong evaluation signal when multiple candidate solutions are available. PST uses this aggregated output yNy_N as a self-generated reference and assigns each response yiy_i a reward ri=PMI(yN,yi),r_i=PMI(y_N,y_i), which measures how informative yiy_i is about the final decision. Correct or on-track responses tend to increase the likelihood of yNy_N and thus receive larger rewards, leading to stronger reinforcement. Incorrect responses provide little information about yNy_N and are correspondingly suppressed. From a generator–verifier perspective, this training process encourages each model to generate candidates that align with the strongest implicit verifier in the peer group. Equivalently, PST drives models to internalize the behavior of the best available verification signal. In Section C.1, we quantify this effect using an empirical generator–verifier (GV) gap metric and observe that PST consistently reduces this gap, indicating improved generator–verifier alignment through peer-predictive self-training. 5.2 Theoretical Analysis We analyze PST at the level of aggregate model capabilities, following the generator–verifier viewpoint of prior work Sun et al. (2025). Our objective is not to characterize the underlying optimization procedure, but to understand how relative capability gaps drive improvement. We consider n models trained jointly. For each model i∈1,…,ni∈\1,…,n\, we associate a scalar capability variable si(t)∈[0,1],s_i(t)∈[0,1], representing its expected performance on the target task at time t≥0t≥ 0. These variables are abstract and summarize population-level performance rather than specific parameters or losses. PST produces an effective self-reward signal by aggregating model outputs. Rather than modeling individual verifiers explicitly, we represent this signal by a single scalar v(t)∈[0,1],v(t)∈[0,1], which captures the strength of the aggregated evaluation signal used for training. Motivated by the wisdom-of-the-crowd effect and the generator–verifier asymmetry discussed in Appendix B, we allow the aggregated evaluation signal to be initially stronger than individual generators, i.e., v(0)>maxi⁡si(0).v(0)> _is_i(0). Model improvement is assumed to be gap-driven: a model can only improve when trained against an evaluation signal that is strictly stronger than itself. In addition, learning saturates as capabilities approach their upper bound. Under these assumptions, we model PST by the following dynamical system: dsidt=α(1−si)[v−si]+,dvdt=β(1−v)maxi⁡si, ds_idt=α(1-s_i)\,[v-s_i]_+, dvdt=β(1-v)\, _is_i, (1) where α,β>0α,β>0 are scaling constants and [x]+:=max⁡x,0[x]_+:= \x,0\. The first equation captures improvement driven by positive generator–verifier gaps, while the second reflects that the aggregate evaluation signal strengthens as generators improve and saturates near its maximum. We now characterize the long-run behavior of the system. Theorem 5.1 (Convergence of PST dynamics). Assume si(0),v(0)∈[0,1]s_i(0),v(0)∈[0,1] for all i. Let (s1(t),…,sn(t),v(t))(s_1(t),…,s_n(t),v(t)) follow (1). Then: 1. For ∀i∀ i, si(t)s_i(t) is monotone non-decreasing and si∗:=limt→∞si(t)s_i := _t→∞s_i(t) exists with si∗≥si(0)s_i ≥ s_i(0). 2. v(t)v(t) is monotone non-decreasing and the limit v∗:=limt→∞v(t)v := _t→∞v(t) exists with v∗≥v(0)v ≥ v(0). 3. For any model i with si(0)<v∗s_i(0)<v , we have si∗=v∗s_i =v . In particular, at convergence no model remains strictly below the limiting evaluation signal unless it started at or above it. The dynamics induced by PST are governed by monotonicity and boundedness. Each solver’s capability increases whenever it falls below the current aggregated evaluation signal and is upper bounded, ensuring convergence. The aggregated signal itself evolves monotonically as a function of the strongest solver. Intuitively, any solver that were to remain strictly below the limiting aggregation level would continue to receive a positive update signal, making such a gap unsustainable at convergence. As a result, PST induces monotone capability improvement driven by gaps relative to a strong aggregated evaluation signal, and converges to a saturated state in which self-reward no longer provides additional learning signal. A formal proof is provided in Appendix A. Empirically, we observe behavior consistent with Theorem 5.1 through ablation studies. Across all datasets, the row-wise maximum GV values—which capture the strongest verifier available to each generator—consistently decrease after training. This indicates that generators progressively align with stronger peer verifiers, reducing the gap between their own capability and the strongest available evaluation signal. In contrast, standard self-training is limited by each model’s own generator–verifier value and cannot exploit stronger cross-model signals. Detailed definitions, metrics, and full empirical results are provided in Appendix C.1, which further show that the observed improvements go beyond what standard self-training can achieve. These results provide empirical support for the monotone improvement dynamics and convergence behavior predicted by our analysis. 6 Ablation Study To isolate the effect of PST from cross model knowledge transfer and to ensure that the observed gains are not simply the consequence of a stronger model guiding a weaker one, we conduct an ablation study using two models from the same family with different scales: LLaMA-3.2-1B-Instruct and LLaMA-3.2-3B-Instruct. Both models are jointly fine tuned on MATH-500-Numeric following the exact PST procedure described in Section 3, including PMI based modulation, model order shuffling across epochs, and same optimization setup. (a) Ablation study on MATH-500-Numeric using LLaMA 3.2 models. (b) Comparison on MATH-500 between SFT, GRPO, and PST. (c) Comparison on MultiArith between SFT, GRPO, and PST. (d) Comparison on SimulEq between SFT, GRPO, and PST. Figure 3: Results across model scales and benchmarks. Figure 3(a) reports the pre training and post training accuracies for both models. We observe consistent improvements across the two scales. The 1B model shows a substantial increase in accuracy, and the stronger 3B model also improves despite already having much higher initial performance. This observation confirms that PST does not rely on a teacher like relationship between weaker and stronger models. Instead, both models benefit from the PMI-guided multi-model training loop, which appears to amplify helpful reasoning trajectories already present in each model and to reduce the impact of uninformative steps. The fact that both models improve together suggests that PST functions as a cooperative self-refinement process rather than a form of implicit distillation. To ensure that the observed gains of PST are not simply due to additional fine-tuning steps or exposure to external supervision, we compare against strong single-model fine-tuning baselines constructed under a matched optimization budget. Specifically, we sample 500 training instances from OpenMathInstruct-2 NVIDIA Research (2024) and fine-tune each model independently using either supervised fine-tuning (SFT) Ouyang et al. (2022) or group relative policy optimization (GRPO) Shao et al. (2024). All baselines are initialized from the same instruction-tuned checkpoints as PST and are trained using the same optimizer, learning rate schedule, number of updates, batch size, and decoding configuration. Crucially, baseline training does not involve any multi-model interaction, PMI-based modulation, or cross-model feedback. After baseline fine-tuning, all models are evaluated without any further adaptation on the same three downstream numerical reasoning benchmarks: MATH-500-Numeric, MultiArith, and SimulEq. Importantly, no information from these evaluation datasets is used during baseline training, either directly or indirectly, consistent with the PST setup. This protocol follows a standard and widely adopted single-model fine-tuning paradigm, in which models are optimized independently on a fixed training set and then tested on held-out tasks. As such, it provides a strong and controlled baseline for assessing whether improvements arise from improved single-model optimization alone or from cross-model interaction. Figures 3(b), 3(c) and 3(d) compare the performance of these baselines with models trained using PST. Across all three benchmarks, both supervised fine-tuning (SFT) and group relative policy optimization (GRPO) lead to only modest or inconsistent gains over the original instruction-tuned models, and their effects vary across model families and datasets. In contrast, PST consistently improves accuracy across all models and benchmarks, even when compared against the stronger GRPO baseline. This consistent gap suggests that the gains achieved by PST cannot be explained by improved single-model training or better reward shaping alone. Instead, they arise from the cooperative, PMI-guided multi-model training dynamics in PST, which leverage cross-model predictiveness to amplify informative reasoning trajectories while suppressing uninformative ones. 7 Conclusion and Future Directions We propose Peer-Predictive Self-Training (PST), a fully unsupervised framework for continued self-improvement of language models based on cross-model interaction. By aggregating sequential generations from heterogeneous models and using PMI-based modulation to scale self-training updates, PST enables models to leverage peer predictiveness as an internal supervisory signal. Across multiple mathematical reasoning benchmarks, PST consistently improves exact-match accuracy and reduces the generator–verifier gap, without relying on external supervision, reward models, or fixed teacher–student hierarchies. More broadly, our results show that peer-predictive feedback enables stable self-improvement in reasoning-oriented tasks by providing a reliable source of internal supervision through aggregation and mutual predictiveness, even without ground truth. These results also highlight several limitations, along with promising directions for further exploration. First, it would be valuable to investigate PST at larger scales, both in terms of model size and training data. While we provide additional experiments on larger models in Appendix C.2, these are still limited in scale, as many of the most recent and capable models are not publicly available. Larger models may exhibit even larger initial generator–verifier gaps, and training PST on sufficiently large datasets would enable evaluation on held-out questions not seen during PST training, allowing a clearer assessment of generalization. Second, PST naturally extends to deeper and longer multi-round interaction. Allowing more rounds of debate or aggregation and training for more epochs may further improve the quality of the aggregated reference signal, while also raising important questions about training stability, early stopping, and how to detect degeneration or collapse during prolonged self-training. Third, while our experiments focus on reasoning-oriented benchmarks in a clean and verifiable setting, it remains to be explored whether peer-predictive self-training can be effective for other task domains, such as knowledge-intensive question answering, instruction following, or open-ended generation. Finally, beyond PMI-based modulation, it would be interesting to explore alternative peer-predictive signals or modulation schemes, and to better understand which properties of the modulation mechanism are essential for stable and effective self-improvement. References Alibaba Cloud and Qwen Team (2024) Qwen2.5 1.5b instruct (qwen2.5-1.5b-instruct) model card. Note: Hugging FaceAccessed: 2026-01-28 External Links: Link Cited by: §4. F. Chen, A. Huang, N. Golowich, S. Malladi, A. Block, J. T. Ash, A. Krishnamurthy, and D. J. Foster (2025) The coverage principle: how pre-training enables post-training. arXiv preprint arXiv:2510.15020. Cited by: §1, §2. S. Feng, F. Yu, and Y. Chen (2022) Peer prediction for learning agents. Advances in Neural Information Processing Systems 35, p. 17276–17286. Cited by: §1. Google (2024) Gemma 2 2b instruct (gemma-2-2b-it) model card. Note: Hugging FaceAccessed: 2026-01-28 External Links: Link Cited by: §4. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: 2nd item. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4. Hugging Face (2024) TRL: transformer reinforcement learning (grpotrainer / grpoconfig). Note: GitHub repositoryAccessed: 2026-01-28 External Links: Link Cited by: §C.3. L. Jiang, Y. Chai, M. Li, M. Liu, R. Fok, N. Dziri, Y. Tsvetkov, M. Sap, A. Albalak, and Y. Choi (2025) Artificial hivemind: the open-ended homogeneity of language models (and beyond). arXiv preprint arXiv:2510.22954. Cited by: §1, §2. M. Kadlčík and M. Štefánik (2024) Self-training language models for arithmetic reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 12378–12386. Cited by: §2. Y. Kong and G. Schoenebeck (2019) An information theoretic framework for designing information elicitation mechanisms that reward truth-telling. ACM Transactions on Economics and Computation (TEAC) 7 (1), p. 1–33. Cited by: §1. Y. Kong (2020) Dominantly truthful multi-task peer prediction with a constant number of tasks. In Proceedings of the fourteenth annual acm-siam symposium on discrete algorithms, p. 2398–2411. Cited by: §1, §2. N. Kushman, R. Barzilay, et al. (2014) Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: 1st item. I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4. J. Lu, R. Teehan, J. Jin, and M. Ren (2025) When does verification pay off? a closer look at llms as solution verifiers. arXiv preprint arXiv:2512.02304. Cited by: §C.1, §1, §1, §2. T. Luo, W. Du, J. Bi, S. Chung, Z. Tang, H. Yang, M. Zhang, and B. Wang (2025) Learning from peers in reasoning models. arXiv preprint arXiv:2505.07787. Cited by: §2. W. Merrill and A. Sabharwal (2023) The parallelism tradeoff: limitations of log-precision transformers. Transactions of the Association for Computational Linguistics 11, p. 531–545. Cited by: §B.2, Lemma B.1, §2. Meta (2024) Llama 3.2 1b instruct (llama-3.2-1b-instruct) model card. Note: Hugging FaceAccessed: 2026-01-28 External Links: Link Cited by: §4. N. Miller, P. Resnick, and R. Zeckhauser (2005) Eliciting informative feedback: the peer-prediction method. Management Science 51 (9), p. 1359–1373. Cited by: §1, §2. NVIDIA Research (2024) OpenMathInstruct-2: a large-scale math instruction dataset. Note: Hugging FaceAccessed: 2026-01-28 External Links: Link Cited by: §6. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS). External Links: Link Cited by: §6. B. Rossman (2014) Formulas vs. circuits for small distance connectivity. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, p. 203–212. Cited by: 3rd item, §B.2. S. Roy and D. Roth (2016) Solving general arithmetic word problems. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: 3rd item. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, et al. (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: Link Cited by: §6. V. Shnayder, A. Agarwal, R. Frongillo, and D. C. Parkes (2016) Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation, p. 179–196. Cited by: §1. Y. Song, H. Zhang, C. Eisenach, S. Kakade, D. Foster, and U. Ghai (2024) Mind the gap: examining the self-improvement capabilities of large language models. arXiv preprint arXiv:2412.02674. Cited by: §C.1, §C.1, §1, §1, §1, §2, §2. Y. Sun, Y. Liang, Z. Zhang, and J. Teng (2025) Theoretical modeling of llm self-improvement training dynamics through solver-verifier gap. arXiv preprint arXiv:2507.00075. Cited by: §1, §2, §5.2. J. Surowiecki (2004) The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. Doubleday. Cited by: §1. S. Wang, Y. Shen, S. Feng, H. Sun, S. Teng, and W. Chen (2024) Alpine: unveiling the planning capability of autoregressive learning in language models. Advances in neural information processing systems 37, p. 119662–119688. Cited by: Appendix B, §1, §2. S. Wang, Y. Shen, H. Sun, S. Feng, S. Teng, L. Dong, Y. Hao, and W. Chen (2025) Benefits and pitfalls of reinforcement learning for language model planning: a theoretical perspective. arXiv preprint arXiv:2509.22613. Cited by: Appendix B, §1, §1, §2. J. Wolfers and E. Zitzewitz (2004) Prediction markets. Journal of Economic Perspective 18 (2), p. 107–126. Cited by: §1. M. Wu, Z. Zhang, Q. Dong, Z. Xi, J. Zhao, S. Jin, X. Fan, Y. Zhou, H. Lv, M. Zhang, et al. (2026) Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 33944–33952. Cited by: §2. S. Xu, Y. Lu, G. Schoenebeck, and Y. Kong (2024) Benchmarking llms’ judgments with no gold standard. arXiv preprint arXiv:2411.07127. Cited by: §1, §2. S. Xu, Y. Lu, G. Schoenebeck, and Y. Kong (2025) Benchmarking llms’ judgments with no gold standard. In Proceedings of the 13th International Conference on Learning Representations (ICLR 2025), Note: Open access via arXiv External Links: Link Cited by: §3. Appendices Appendix A Proof of Theorem 5.1 Proof. For any i, the right-hand side of (1) satisfies dsidt≥0 ds_idt≥ 0 because (1−si)≥0(1-s_i)≥ 0 on [0,1][0,1] and [v−si]+≥0[v-s_i]_+≥ 0. Hence si(t)s_i(t) is monotone non-decreasing. Moreover, if si(t)=1s_i(t)=1 then dsidt=0 ds_idt=0, so si(t)s_i(t) cannot exceed 11. If si(t)=0s_i(t)=0 then dsidt≥0 ds_idt≥ 0, so it cannot become negative. Thus si(t)∈[0,1]s_i(t)∈[0,1] for all t≥0t≥ 0. Similarly, dvdt=β(1−v)maxi⁡si≥0 dvdt=β(1-v) _is_i≥ 0 whenever v∈[0,1]v∈[0,1], so v(t)v(t) is monotone non-decreasing. Also, if v(t)=1v(t)=1 then dvdt=0 dvdt=0, so v(t)v(t) cannot exceed 11; if v(t)=0v(t)=0 then dvdt≥0 dvdt≥ 0, so it cannot become negative. Thus v(t)∈[0,1]v(t)∈[0,1] for all t≥0t≥ 0. Since each si(t)s_i(t) is monotone non-decreasing and bounded in [0,1][0,1], the limit si∗=limt→∞si(t)s_i = _t→∞s_i(t) exists for every i. Likewise, since v(t)v(t) is monotone non-decreasing and bounded in [0,1][0,1], the limit v∗=limt→∞v(t)v = _t→∞v(t) exists. Fix any model i with si(0)<v∗s_i(0)<v . Suppose for contradiction that si∗<v∗s_i <v . Let δ:=13(v∗−si∗)>0δ:= 13(v -s_i )>0. By convergence, there exists T such that for all t≥Tt≥ T, v(t)≥v∗−δandsi(t)≤si∗+δ.v(t)≥ v -δ s_i(t)≤ s_i +δ. Therefore, for all t≥Tt≥ T, v(t)−si(t)≥(v∗−δ)−(si∗+δ)=(v∗−si∗)−2δ=δ,v(t)-s_i(t)≥(v -δ)-(s_i +δ)=(v -s_i )-2δ=δ, so [v(t)−si(t)]+≥δ[v(t)-s_i(t)]_+≥δ. Also, since si(t)≤si∗+δ<1s_i(t)≤ s_i +δ<1 for t large enough (because si∗≤1s_i ≤ 1 and δ>0δ>0), we have 1−si(t)≥1−(si∗+δ)=:c>01-s_i(t)≥ 1-(s_i +δ)=:c>0 for all sufficiently large t. Plugging into (1) yields, for all sufficiently large t, dsidt≥αcδ> 0, ds_idt\;≥\;α\,c\,δ\;>\;0, which implies that si(t)s_i(t) must increase by at least αcδ(t−T)α cδ\,(t-T) for t≥Tt≥ T, contradicting the existence of the finite limit si∗s_i . Hence the assumption si∗<v∗s_i <v is impossible, and we conclude si∗=v∗s_i =v . This proves the third claim and completes the proof. ∎ Appendix B Why Is Verification Easier Than Generation: A Transformer Architecture Perspective In this section, we provide theoretical intuition for a fundamental asymmetry underlying our method: in reasoning-oriented tasks, verifying a candidate solution is often substantially easier than generating a correct solution from scratch. This verification–generation gap enables reliable learning signals even in the absence of external supervision and plays a central role in our analysis. Following Wang et al. (2024; 2025), we view reasoning as a long-range logical deduction problem. Such tasks require deriving a final conclusion through a sequence of logically dependent intermediate steps. Maintaining global coherence across many steps is inherently challenging for autoregressive language models and makes direct generation of correct solutions difficult. A long-range logical deduction naturally induces a dependency structure. Each intermediate conclusion corresponds to a reasoning state, and each valid inference corresponds to a transition between states. Although this structure may not be explicitly represented, it can be abstracted as a directed dependency graph whose nodes represent intermediate reasoning states and whose edges represent valid logical implications. Solving the deduction task corresponds to identifying a valid chain of inferences connecting the initial premises to the final conclusion. Crucially, there is a qualitative difference between constructing such a chain and verifying one. Generation requires discovering a globally consistent sequence of deductions, whereas verification only requires checking whether a proposed sequence is logically valid. This asymmetry suggests that even when a model struggles to generate a correct answer, it may still be capable of reliably recognizing correctness when presented with a candidate solution. Formally, we represent a long-range logical deduction task by a directed graph G=(V,E)G=(V,E) with a designated source node s and target node t. Nodes correspond to intermediate reasoning states, and directed edges represent valid inference steps. Under this abstraction, generation corresponds to constructing a valid sequence of deductions that connects s to t, while verification corresponds to checking whether a given candidate sequence indeed constitutes such a valid connection. As we show below, these two tasks differ sharply in computational complexity. B.1 Complexity Analysis of the Verification Task We first show that the verification task can be carried out by a constant-depth transformer with polynomial embedding size in the input length. Intuitively, verification only requires local consistency checks across a proposed deduction chain and can be performed in a highly parallel manner. Lemma B.1. Every problem in the uniform-TC0 family can be simulated by a constant-depth, log-precision transformer with polynomial embedding size in the input length. Conversely, every such transformer can be simulated by a threshold circuit of polynomial size and constant depth Merrill and Sabharwal (2023). 111It is conjectured that an O(logk⁡n)O( ^kn)-depth, log-precision transformer with polynomial embedding size has computational power equivalent to the class TCk. Theorem B.2 (Verification lies in TC0). For both in-context and pre-learned graph settings, the verification task, namely checking whether a candidate deduction chain is valid or confirming that no valid chain exists, belongs to the complexity class TC0. Consequently, it can be simulated by a constant-depth, log-precision transformer with polynomial embedding size. Proof. We show that the verification task lies in TC0 for both the in-context and pre-learned graph settings. TC0 consists of problems solvable by uniform, constant-depth, polynomial-size threshold circuits with AND, OR, and MAJORITY gates. Given a graph G=(V,E)G=(V,E) and a candidate path (v0,v1,…,vk)(v_0,v_1,…,v_k), verification reduces to checking in parallel whether each pair (vi,vi+1)(v_i,v_i+1) is an edge in E. Each edge-membership test is a simple table lookup or adjacency-matrix access, which can be performed in constant depth using threshold circuits. The final correctness check is the AND of all edge-validity results, which also remains in TC0. In the pre-learned setting, the graph is assumed to be stored in the model’s internal representation from training; verifying a candidate path amounts to matching queried edges against this stored structure. In the in-context setting, the graph is explicitly provided as part of the input. In both cases, verification again reduces to parallel edge-existence queries, each implementable with constant-depth circuitry. If the output claims that no path exists, correctness reduces to confirming that none of the proposed edge transitions in the candidate path appear in E, which is likewise a constant-depth check. All required operations therefore fit within uniform threshold circuits of constant depth and polynomial size. Hence the verification task belongs to TC0, establishing the theorem. ∎ This result follows because each local inference check (vj,vj+1)∈E(v_j,v_j+1)∈ E can be performed in constant parallel time, and the results can be aggregated using AND, OR, or MAJORITY gates. In pre-learned graphs, this corresponds to table lookup. In in-context graphs, it corresponds to explicit consistency checking within the prompt. The entire verification procedure therefore fits within TC0. Thus, determining whether a proposed response is correct (C) or incorrect (Z) is computationally lightweight and well within the capabilities of low-depth transformers. B.2 Complexity Analysis of the Generation Task We next analyze the generation task, which requires constructing a valid deduction chain rather than merely verifying one. We show that this process is substantially harder and exceeds the computational power of constant-depth transformers. Theorem B.3 (Generation exceeds TC0). For both in-context and pre-learned graph settings, constructing a valid deduction chain is computationally harder than verification: • In and-or graphs, the PATH problem is P-hard. Hence, generation in the in-context setting exceeds TC0 unless TC0=PTC^0=P. • In or-graphs, the PATH problem is NL-complete. Hence, generation exceeds TC1 unless NL=TC1NL=TC^1. • Even in the pre-learned setting, evaluating deduction chains on and-or graphs can simulate monotone circuit evaluation, which is at least NL-hard Rossman (2014). Proof. We analyze the computational complexity of constructing a valid s–t path in the three scenarios stated in the theorem. 1. In-context and-or graphs. We first show that path construction on and-or graphs in the in-context setting is P-hard. The reduction is from Horn clause satisfiability, a P-complete problem. Given a Horn formula φ , we construct an and-or graph GφG_ in polynomial time as follows. Each positive literal is represented as a seed node, and each negative literal is represented as a target node. A clause containing one positive literal and one or more negative literals introduces directed edges from the positive literal node to the corresponding negative literal nodes. Clauses with multiple negative literals introduce a new target node that aggregates their dependencies. Finally, we add a single global target node connected to all clause-level target nodes. A truth assignment satisfying φ corresponds exactly to a path from some seed node to the global target node, and conversely the existence of such a path yields a satisfying assignment. The size of GφG_ is polynomial in |φ|| |, so the reduction is valid. Since Horn satisfiability is P-complete, path construction on and-or graphs is P-hard. Under the hypothesis TC0⊊PTC^0 , this implies that constructing a valid path on and-or graphs cannot be carried out by constant-depth threshold circuits and therefore cannot be simulated by constant-depth, log-precision transformers with polynomial embedding size. 2. In-context or-graphs. For or-graphs, the in-context problem is the classical directed reachability problem: given the explicitly provided graph G=(V,E)G=(V,E) and distinguished nodes s,ts,t, determine whether there exists a directed path from s to t. This is the PATH problem, which is NL-complete. Since NC1⊆L⊆NL⊆NC2andNCi⊆ACi⊆TCi⊆NCi+1,NC^1 ^2 ^i ^i ^i ^i+1, solving an NL-hard problem within TC1TC^1 would imply NL=TC1NL=TC^1, an equality that is widely conjectured to be false. In the transformer setting, a constant-depth, log-precision model computing with polynomial embedding corresponds to a TC1TC^1 circuit family Merrill and Sabharwal (2023). Therefore, assuming NL≠TC1NL ^1, path construction on or-graphs in the in-context setting cannot be simulated by such transformers. 3. Pre-learned and-or graphs. In the pre-learned setting, the transformer is assumed to have memorized the entire graph structure during training. We show that even under this assumption, constructing a valid path may require computational power at least as strong as NL. The key observation is that reachability in an and-or graph can simulate the evaluation of an arbitrary monotone circuit of polynomial size. Each gate of the monotone circuit is encoded as a node in the and-or graph: an AND gate corresponds to an “and-node” that requires all predecessors to be reachable, and an OR gate corresponds to an “or-node” requiring at least one predecessor to be reachable. The output gate of the monotone circuit is designated as the target node whose reachability determines acceptance. This construction is standard and preserves polynomial size; see Rossman (2014). Since monotone circuits of polynomial size can express all NL computations, the existence of a constant-depth (TC0) implementation for reachability on these graphs would imply TC0=NLTC^0=NL, contrary to prevailing complexity assumptions. Therefore, unless TC0=NLTC^0=NL, there exist pre-learned and-or graphs for which path construction exceeds the computational power of constant-depth, log-precision transformers. Conclusion. Across all three settings, constructing a valid path (planning) is at least as hard as solving P-hard or NL-hard problems. Under standard complexity separation assumptions (TC0⊊PTC^0 and NL≠TC1NL ^1), these tasks exceed the computational power of constant-depth, log-precision transformers with polynomial embedding size, establishing the theorem. ∎ These results follow from standard complexity-theoretic reductions. Horn clause satisfiability, which is P-complete, reduces to PATH on and-or graphs, establishing P-hardness in the in-context setting. The classical PATH problem on or-graphs is NL-complete. In the pre-learned case, deduction-chain evaluation corresponds to evaluating monotone circuits composed of AND and OR gates, which is known to be NL-hard. If constant-depth transformers could solve these tasks, it would imply TC0=NLTC^0=NL or even TC0=PTC^0=P, which is widely believed to be unlikely. B.3 Discussion and Implications The analysis above highlights a fundamental asymmetry between verification and generation in long-range logical deduction tasks. Verification lies in TC0 and is therefore feasible for constant-depth, log-precision transformers, whereas generation is P- or NL-hard and substantially more demanding. This asymmetry directly motivates the assumption used in Section 4.2 that the aggregated evaluation signal can be initially stronger than any individual generator. Since verification is computationally easier than generation, models may possess relatively strong verification capabilities even when their generation capabilities are weak. Aggregating multiple models allows these verification capabilities to be combined, yielding an effective evaluation signal that satisfies v(0)>maxi⁡si(0).v(0)> _is_i(0). As training progresses, improvements in generation reduce this gap, leading to saturation when models internalize the evaluation signal. Appendix C Additional Experiments and Details C.1 Empirical Improvements on GV-Gap (a) SimulEq (b) Math500 (c) MultiArith Figure 4: Initial GV matrices on SimulEq, Math500, and MultiArith, computed using initial models as both generators and verifiers. Row-wise maxima at initialization are highlighted by gray boxes. The green numbers on the right report the corresponding row-wise maximum GV values after fine-tuning, computed between fine-tuned generators and fine-tuned verifiers (green indicates a decrease, while black indicates no change). The analysis above predicts that the gains of PST arise from cross-model interactions, in which stronger verifiers provide a cleaner training signal. Empirically, for a generator MgM_g and a verifier MvM_v, given a question–answer pair (x,a)(x,a) and k candidate responses y(1),…,y(k)∼pMg(⋅∣x)\y^(1),…,y^(k)\ p_M_g(· x), where MgM_g is run in generative mode to sample candidate responses, we define the GV value as GV-Value(Mg,Mv∣x)≜Ans(y⋆)=a, -Value(M_g,M_v x) 1\Ans(y )=a\, y⋆=arg⁡maxt⁡sMv(x,y(t)). y = _ts_M_v(x,y^(t)). Here sMv(x,y)∈ℝs_M_v(x,y) denotes a scalar evaluation score assigned by the verifier model to a completed response y given input x. The verifier MvM_v is run in evaluation mode, performing a deterministic forward pass on the fixed input–response pair (x,y)(x,y) without sampling, dropout, or parameter updates. The score is computed as a sequence-level value by summing token-level log-probabilities over the response tokens and is used solely to rank candidate responses, rather than to generate outputs, following the generator–verifier framework of Song et al. (2024). Figure 4 shows the initial GV matrices on SimulEq, Math500, and MultiArith. Rows correspond to generators and columns to verifiers. For each generator, the row-wise maximum GV value at initialization is highlighted by a gray box. In most cases, this maximum occurs off the diagonal and is strictly positive, indicating that a generator aligns more strongly with another model acting as a verifier than with itself. This reveals systematic cross-model asymmetries that PST can exploit during training, suggesting that peer interaction provides stronger supervision than self-training within a single model. Moreover, this also aligned with the findings in (Song et al., 2024; Lu et al., 2025) that cross-verification from diverse models are effective. The number shown to the right of each row reports the corresponding row-wise maximum GV value after fine-tuning, where finetuned models are used as both generators and verifiers. Across all datasets, these values consistently decrease after training. Specifically, the sum of row-wise maximum GV values is reduced by approximately 39.8% on SimulEq, 27.1% on MATH-500-Numeric, and 26.0% on MultiArith, indicating that generators internalize strategies aligned with the strongest verifiers. In contrast, standard self-training relies on each model’s own predictions and evaluations, whose generator–verifier value is inherently bounded by its own capability and thus cannot match the maximum GV value achieved by stronger peer models. By leveraging cross-model interactions, PST allows each generator to align with signals induced by stronger verifiers, leading to systematically reduced generator–verifier gaps. This behavior is consistent with our theoretical interpretation, in which probability mass is progressively reallocated toward correct strategies, and the capability gap between each generator and the strongest verifier narrows over training. C.2 Experiments on Larger-Scale LLMs To further evaluate the scalability of PST, we conduct additional experiments on larger-scale language models. Except for the use of larger models, all experimental settings remain identical to those described in Section 4 and Section C.3, including datasets, training procedure, optimization, and evaluation protocols. Specifically, we follow the same sequential generation process, PMI-based modulation, and LoRA-based fine-tuning setup as in the main experiments. (a) SimulEq (b) MATH-500-Numeric (c) MultiArith Figure 5: Accuracy before and after fine-tuning on SimulEq, MATH-500-Numeric, and MultiArith. Each panel reports pre- and post-fine-tuning by PST performance for Gemma-3-4B, LLaMA-3.1-8B, and Qwen-2.5-3B. We observe consistent performance improvements of PST over the original models in Figure 5, consistent with the trends reported in Section 4. C.3 Additional Reproducibility Details This section reports only implementation-level details not specified in the main text; all training and evaluation code is provided in the supplementary materials. For evaluation, all models use stochastic decoding with temperature 0.70.7 and sampling enabled, and results are averaged over 1010 independent runs with different random seeds to estimate variance. Error bars denote ±1± 1 standard deviation across runs. For baseline supervised fine-tuning (SFT), each model is trained independently on 500500 examples from OpenMathInstruct-2 for 55 epochs using causal language modeling, with per-device batch size 11, gradient accumulation of 88 steps, and learning rate 10−610^-6. The learning rate for all baselines is lightly tuned and found to have no significant impact; we therefore use the same learning rate as PST to ensure a consistent comparison. For verifiable reinforcement learning, we adopt group relative policy optimization (GRPO) as implemented in the TRL library Hugging Face (2024), which optimizes the policy directly using scalar rewards assigned to generated completions. The reward is binary and fully verifiable: a completion receives reward 11 if the gold numeric answer appears as a standalone number in the output text and 0 otherwise. GRPO training uses 55 epochs, per-device batch size 11, gradient accumulation 88, learning rate 10−610^-6, and samples 44 completions per prompt during training, with prompt and completion lengths capped at 512512 and 256256 tokens respectively. For PST training, models are shuffled at the start of each epoch using a fixed seed so that the identity of the final aggregating model rotates across epochs; the final model produces the aggregate response but is updated only via standard next-token prediction on its own output, while non-final models are updated by next-token cross-entropy on their own generated responses scaled by a sigmoid function of the PMI-based reward with temperature τ=3.0τ=3.0. Gradients are clipped to global norm 1.01.0, and optimization steps are skipped whenever non-finite gradients are detected.

Instant research discovery