Paper deep dive

Cloud-ScPO: Hidden-State Geometry for Semi-Supervised Preference Optimization in LLM Reasoning

Yuzhou Liu, Xiyang Hu

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 70

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Preference optimization improves mathematical reasoning in large language models (LLMs), but reliable chosen-rejected pairs usually require verified answers, human annotations, or external reward models. We investigate whether preference supervision can instead be derived from the model's internal representation geometry in a semi-supervised setting. Our analysis shows that reasoning trajectories generated across different mathematical problems form structured global point clouds in which correct and incorrect trajectories exhibit different geometric organization. Based on this observation, we propose Cloud--ScPO, a topology-guided preference-mining framework that uses a small labeled set to construct multiple correct and incorrect reference Clouds. Each trajectory is represented by a mean-pooled hidden state and scored against connectivity-induced components using a component-level soft $k$-nearest-neighbor measure averaged across reference banks. We combine this cross-problem Cloud signal with prompt-level self-consistency: self-consistency determines the answer-level preference direction, while Cloud scoring selects concrete trajectories and filters pairs by their score margin. Experiments on GSM8K and MATH-Numeric across four model settings show that Cloud--ScPO consistently improves over ScPO, with gains of up to 4.49\% on GSM8K and 4.19\% on MATH-Numeric. Pair-level analyses further show that Cloud--ScPO maintains comparable correctness reliability while more effectively separating informative chosen trajectories from incomplete, repetitive, or otherwise low-quality rejected responses.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

68,496 characters extracted from source content.

Expand or collapse full text

Cloud–ScPO: Hidden-State Geometry for Semi-Supervised Preference Optimization in LLM Reasoning Written by AAAI Press Staff1 AAAI Style Contributions by Peter Patel Schneider, Sunil Issar, J. Scott Penberthy, George Ferguson, Hans Guesgen, Francisco Cruz , Marc Pujol-Gonzalez With help from the AAAI Publications Committee. Yuzhou Liu1, Xiyang Hu2 Corresponding author. Preference optimization improves mathematical reasoning in large language models (LLMs), but reliable chosen–rejected pairs usually require verified answers, human annotations, or external reward models. We investigate whether preference supervision can instead be derived from the model’s internal representation geometry in a semi-supervised setting. Our analysis shows that reasoning trajectories generated across different mathematical problems form structured global point clouds in which correct and incorrect trajectories exhibit different geometric organization. Based on this observation, we propose Cloud–ScPO, a topology-guided preference-mining framework that uses a small labeled set to construct multiple correct and incorrect reference Clouds. Each trajectory is represented by a mean-pooled hidden state and scored against connectivity-induced components using a component-level soft k-nearest-neighbor measure averaged across reference banks. We combine this cross-problem Cloud signal with prompt-level self-consistency: self-consistency determines the answer-level preference direction, while Cloud scoring selects concrete trajectories and filters pairs by their score margin. Experiments on GSM8K and MATH-Numeric across four model settings show that Cloud–ScPO consistently improves over ScPO, with gains of up to 4.49% on GSM8K and 4.19% on MATH-Numeric. Pair-level analyses further show that Cloud–ScPO maintains comparable correctness reliability while more effectively separating informative chosen trajectories from incomplete, repetitive, or otherwise low-quality rejected responses. Introduction LLMs have shown strong potential in mathematical and symbolic reasoning, especially when combined with post-training and preference optimization methods. Representative approaches include reinforcement learning from human feedback (RLHF) (Ouyang et al. 2022), direct preference optimization (DPO) (Rafailov et al. 2023), and group relative policy optimization (GRPO) (Shao et al. 2024). Their effectiveness depends on how preferred and rejected responses are constructed (Christiano et al. 2017; Stiennon et al. 2020; Ouyang et al. 2022). Human comparisons are expensive, external verifiers may be unavailable, and final-answer labels provide little information about the quality of intermediate reasoning. These limitations are especially restrictive when each prompt produces several long trajectories. In semi-supervised reasoning, only a small subset of prompts has verified answers, while most prompts and generated trajectories are unlabeled. Existing methods reduce annotation costs by learning from model-generated reasoning. STaR iteratively trains on self-generated rationales that reach correct answers (Zelikman et al. 2022), and self-consistency aggregates multiple sampled trajectories at inference time (Wang et al. 2023). ScPO converts self-consistency into training supervision by preferring trajectories from the majority-answer cluster over those from minority clusters (Prasad et al. 2025). Nevertheless, determining which rollout should be preferred remains difficult when several trajectories share the same answer or when prompt-level consensus is unreliable. Semi-supervised reward modeling provides another solution by assigning pseudo-preferencesto unlabeled responses through iterative reward-model training (He et al. 2024). However, such methods still depend on a learned reward model, and their pseudo-label quality may be sensitive to the limited labeled preference data available for training. In this work, we investigate a different source of preference signal: the internal representation geometry of the language model itself. Our key observation is that reasoning trajectories generated across many distinct mathematical problems collectively form a structured global point cloud in the model’s representation space, rather than an uninformative collection of independent samples. More importantly, this global geometry is correlated with answer correctness. Correct trajectories from different problems tend to occupy denser and more coherent regions and become connected at smaller filtration scales, whereas incorrect trajectories are generally more dispersed and exhibit greater geometric and topological variation. Motivated by this observation, we use mean-pooled response-token hidden states to obtain a trajectory-level representation that summarizes the complete reasoning process. Motivated by this observation, we propose Cloud–ScPO, a topology-guided preference-mining framework for semi-supervised reasoning optimization. We construct multiple reference Clouds from labeled trajectory representations and score each unlabeled rollout using a component-level soft k-nearest-neighbor measure averaged across reference banks. Cloud scoring is combined with self-consistency to select concrete trajectories within answer clusters and retain high-confidence preference pairs without requiring gold labels or reward-model scores for every unlabeled rollout. Experiments on GSM8K and MATH-Numeric show that Cloud–ScPO consistently improves over ScPO across multiple model backbones. Pair-level analysis further indicates that Cloud scoring preserves correctness reliability while better separating informative chosen trajectories from low-quality rejected responses. Our contributions are a global geometry-based trajectory-quality signal, a robust multi-bank Cloud-scoring method, and its integration with self-consistency for semi-supervised preference construction. Related work Latent-space supervision for reasoning optimization. Prior work has shown that language-model internal representations encode signals related to latent knowledge, truthfulness, and hallucination risk (Burns et al. 2024; Azaria and Mitchell 2023; Chen et al. 2024). Building on this direction, Silence the Judge introduces Latent-GRPO, which derives intrinsic rewards from the hidden-state geometry of trajectories generated for the same prompt (Zhang et al. 2026). It estimates a prompt-specific latent center and assigns higher rewards to trajectories closer to this center. Although this avoids external verifiers, the resulting signal is limited to the sampled rollout group and may be sensitive to incorrect within-prompt consensus. In contrast, our method constructs multi-bank reference Clouds from correct trajectories collected across different labeled problems. This global structure transfers supervision across prompts and is combined with self-consistency to construct high-confidence preference pairs for subsequent optimization. Preference optimization from sampled reasoning trajectories. Prior work has explored improving reasoning through self-generated rationales, iterative self-training, and verifier- or preference-based learning (Zelikman et al. 2022; Hosseini et al. 2024; Wang et al. 2024b). Building on this direction, IRPO and ScPO convert multiple sampled chain-of-thought trajectories into preference supervision. IRPO iteratively constructs chosen–rejected pairs by preferring trajectories that lead to correct answers and optimizes them using a modified DPO objective augmented with a negative log-likelihood term (Rafailov et al. 2023; Pang et al. 2024). ScPO instead builds on self-consistency (Wang et al. 2023) to reduce dependence on gold answers, treating trajectories associated with the majority answer on unlabeled prompts as preferred over inconsistent alternatives (Prasad et al. 2025). Despite this difference, both methods derive their primary selection signals from outcomes observed within each prompt and do not explicitly exploit geometric regularities shared across problems. Our method retains the reliable answer-grouping mechanism of ScPO but complements it with multi-bank Cloud scoring constructed from correct trajectories across many labeled problems. This global representation signal distinguishes trajectories within answer groups and enables Cloud-gap-based confidence filtering, thereby providing preference supervision beyond prompt-level correctness or consistency alone. Topology and geometry of LLM representation spaces. Prior work has shown that language-model representations exhibit meaningful geometric and topological structure. Persistent homology has been used to analyze hidden-state point clouds and relate their structure to model behavior and robustness (Chauhan and Kaul 2022), while activation-space geometry has been found to encode behaviorally relevant properties such as factual truth (Marks and Tegmark 2024). More recently, The Shape of Adversarial Influence shows that adversarial conditions induce systematic changes in the global topology of LLM activation spaces (Fay et al. 2026). In contrast to these primarily diagnostic studies, we use cross-problem representation geometry to construct multi-bank reference Clouds and convert this structure into trajectory scores and high-confidence preference pairs for semi-supervised reasoning optimization. Method As illustrated in Figure 1, both Cloud–ScPO and Pure Cloud follow a semi-supervised pipeline that begins with a base model and a small labeled dataset. Figure 1: Overview of the proposed framework. (1) Response generation: the base model generates multiple trajectories for the labeled problems. Gold answers are used to identify correct and incorrect trajectories, and the verified correct trajectories are used for supervised fine-tuning. The resulting model then generates trajectories for the unlabeled problems. (2) Cloud generation: the labeled trajectories are mean-pooled and divided into correct and incorrect reference pools, from which multiple reference banks are sampled. (3) Scoring and pair construction: each unlabeled trajectory is scored against every reference bank, and the bank-level scores are averaged to construct preference pairs for Pure Cloud and Cloud–ScPO. Problem Setup We consider a semi-supervised reasoning setting with a small labeled set L=(xi,ai)i=1nLD_L=\(x_i,a_i)\_i=1^n_L and a larger unlabeled set U=xjj=1nUD_U=\x_j\_j=1^n_U, where nL≪nUn_L n_U. We first use the base model to generate K reasoning trajectories for each of the 600 labeled problems. The extracted answer of each trajectory is compared with the corresponding gold answer, allowing the labeled trajectories to be partitioned into correct and incorrect pools. The verified correct trajectories are used for supervised fine-tuning, producing a tuned policy πSFT _SFT, while both labeled trajectory pools are retained for constructing the multi-bank reference Clouds. Finally, πSFT _SFT generates K trajectories for each problem in UD_U. These unlabeled trajectories are scored against the reference Clouds and used to construct preference pairs without accessing any ground-truth answers from UD_U. To characterize local connectivity within each reference Cloud, we use zero-dimensional persistent homology (H0H_0). Under a Vietoris–Rips filtration, each representation initially forms an individual connected component, and components merge as the distance threshold increases; H0H_0 records this evolution of connected structures (Zomorodian and Carlsson 2005; Gabrielsson et al. 2020). Additional persistent-homology visualizations, including exploratory H1H_1 analyses and the motivation for our connectivity-based component construction, are provided in Appendix A. Trajectory Representations and Reference Clouds Let ht∈ℝdh_t ^d be the final-layer hidden state of the t-th valid response token. We represent a trajectory by the normalized mean hidden state z¯(x,y)=z(x,y)‖z(x,y)‖2,z(x,y)=1T∑t=1Tht. z(x,y)= z(x,y)\|z(x,y)\|_2, z(x,y)= 1T _t=1^Th_t. (1) Eq. (1) summarizes the full response rather than using only its final token. Using the labeled trajectories, we form a correct pool +P^+ and an incorrect pool −P^- according to whether their extracted answers match the verified answers. We then construct R balanced reference banks (r+,r−)(C_r^+,C_r^-) by repeatedly sampling from these two pools. This global construction differs from prompt-specific clouds formed by multiple responses to one problem(Zhang et al. 2026). Topology-Guided Cloud Scoring Each reference cloud may contain several local structures. To identify them, we process pairwise edges in ascending Euclidean distance, as in the zero-dimensional Vietoris–Rips filtration (Gabrielsson et al. 2020). Starting with one component per point, we stop after ⌈ρ(N−1)⌉ ρ(N-1) successful merges and discard components smaller than a predefined threshold. The remaining early-H0H_0 components of class c∈+,−c∈\+,-\ in bank r are denoted by rc=Gr,jcjG_r^c=\G_r,j^c\_j. For a candidate z, let dr,jc(z)=minu∈Gr,jc⁡‖z−u‖2d_r,j^c(z)= _u∈ G_r,j^c\|z-u\|_2 be its distance to component Gr,jcG_r,j^c. Let q,rc(z)N_q,r^c(z) contain its q nearest components, and define the component-size weight as ωr,jc=|Gr,jc|, _r,j^c=|G_r,j^c|, where |Gr,jc||G_r,j^c| is the number of reference trajectories in the component. We use the distance kernel κ(d)=exp⁡(−dpτ),κ(d)= (- d^pτ ), where p>0p>0 controls the distance-decay rate. We set p=2p=2 in all experiments. The component compatibility is Src(z)=log⁡∑j∈q,rc(z)ωr,jcκ(dr,jc(z))∑j∈q,rc(z)ωr,jc.S_r^c(z)= Σ _j _q,r^c(z) _r,j^cκ(d_r,j^c(z))Σ _j _q,r^c(z) _r,j^c. (2) Thus, Eq. (2) aggregates over topology-induced components rather than individual reference points. The final Cloud score contrasts compatibility with the correct and incorrect clouds and averages the result across reference banks: scloud(x,y)=1R∑r=1Rsr(z¯(x,y)),sr(z)=Sr+(z)−λnegSr−(z). array[]rcls_cloud(x,y)&=& 1R _r=1^Rs_r( z(x,y)),\\[3.0pt] s_r(z)&=&S_r^+(z)- _negS_r^-(z). array (3) In Eq. (3), λneg _neg controls the incorrect-cloud penalty. Cloud scores are compared only among trajectories generated for the same problem. Preference Pair Construction Pure Cloud. Let xvalid⊆xY_x^valid _x denote the trajectories that pass the response-validity checks. For each unlabeled problem, we choose the highest- and lowest-scoring valid trajectories: yx+=arg⁡maxy∈xvalid⁡scloud(x,y),yx−=arg⁡miny∈xvalid⁡scloud(x,y). array[]rcly_x^+&=& _y _x^valids_cloud(x,y),\\ y_x^-&=& _y _x^valids_cloud(x,y). array (4) Pairs are ranked by the Cloud margin scloud(x,yx+)−scloud(x,yx−)s_cloud(x,y_x^+)-s_cloud(x,y_x^-), and only the highest-confidence fraction (α, which is tunable) is retained. Hence, Eq. (4) constructs preferences without answer-level self-consistency. Cloud–ScPO Hybrid. We first extract and canonicalize the final answer of each valid trajectory and group trajectories producing the same answer into answer clusters. Following ScPO, we retain only problems with a unique majority-answer cluster. The majority answer is selected as the preferred answer ax+a_x^+, while a least-frequent eligible non-majority answer is selected as the rejected answer ax−a_x^-. When multiple minority clusters have the same vote count, we select the cluster whose lowest-scoring trajectory has the smallest Cloud score. After determining the preferred and rejected answer clusters, Cloud scores are used to select the concrete trajectories within them: yx+=arg⁡maxy∈x+⁡sCloud(y),yx−=arg⁡miny∈x−⁡sCloud(y),y_x^+= _y _x^+s_Cloud(y), y_x^-= _y _x^-s_Cloud(y), (5) where x+C_x^+ and x−C_x^- denote the preferred and rejected answer clusters, respectively. We define the Cloud-based confidence margin of the resulting pair as cxHybrid=sCloud(yx+)−sCloud(yx−).c_x^Hybrid=s_Cloud(y_x^+)-s_Cloud(y_x^-). (6) Eligible pairs are ranked by this margin, and the top-α fraction is retained, where α is a tunable retention hyperparameter. Thus, self-consistency determines the answer-level preference direction, whereas Cloud scoring selects the concrete reasoning trajectories and filters pairs according to their representation-based separation. Since ScPO and Cloud–ScPO may select different trajectories before the final response-validity checks, their numbers of successfully constructed pairs may differ slightly. For optimization, we retain the normalized vote-margin weight used by ScPO: w(x)=Vx(ax+)−Vx(ax−)K,w(x)= V_x(a_x^+)-V_x(a_x^-)K, (7) where Vx(a)V_x(a) is the number of sampled trajectories producing answer a, and K is the total number of rollouts for problem x. Because tied-majority problems are excluded, every retained pair has a positive vote margin. Complete pseudocode and implementation details for Pure Cloud and Cloud–ScPO preference-pair construction are provided in Appendix B. Preference Optimization Pure Cloud optimization. Pure Cloud uses standard DPO (Rafailov et al. 2023). Define rθ(x,y)=log⁡πθ(y∣x)−log⁡πref(y∣x)r_θ(x,y)= _θ(y x)- _ref(y x) and Δrθ=rθ(x,y+)−rθ(x,y−). r_θ=r_θ(x,y^+)-r_θ(x,y^-). Its objective is ℒPureCloud=−(x,y+,y−)∼pref[log⁡σ(βΔrθ)].L_PureCloud=-E_(x,y^+,y^-) _pref [ σ\! (β r_θ ) ]. (8) Equation (8) increases the relative likelihood of the Cloud-preferred trajectory with respect to the reference policy. Cloud–ScPO optimization. Following ScPO, Cloud–ScPO additionally uses a length-normalized negative log-likelihood objective for the chosen response: ℓNLL(x,y+)=−1|y+|∑t=1|y+|log⁡πθ(yt+∣x,y<t+). _NLL(x,y^+)=- 1|y^+| _t=1^|y^+| _θ (y_t^+ x,y_<t^+ ). (9) Let ℓDPO(x,y+,y−)=−log⁡σ(βΔrθ). _DPO(x,y^+,y^-)=- σ\! (β r_θ ). (10) The final Cloud–ScPO objective is ℒCloud-ScPO=(x,y+,y−)∼pref[w(x)(ℓDPO+λNLLℓNLL)].L_Cloud -ScPO=E_(x,y^+,y^-) _pref [w(x) ( _DPO+ _NLL _NLL ) ]. (11) In Eq. (11), the normalized vote margin defined in Eq. (7) weights both optimization terms, while λNLL _NLL controls the contribution of the chosen-response likelihood objective. The DPO term learns the pairwise preference, whereas the NLL term directly reinforces the Cloud-selected trajectory from the majority-answer cluster. Experiments Experiments setup Datasets and Metrics. We evaluate our methods on GSM8K and MATH-Numeric, and conduct additional Pure Cloud ablations on MATH-Numeric. • GSM8K. GSM8K (Cobbe et al. 2021) contains approximately 7.5K training problems and 1.3K test problems covering grade-school mathematical reasoning. Following the original ScPO setup, we reserve 10% of the training set as a development set for hyperparameter tuning and checkpoint selection. The resulting train, development, and test splits contain approximately 6.7K, 0.8K, and 1.3K problems, respectively. We report exact-match accuracy of the extracted final numeric answer on test set. • MATH-Numeric. MATH (Hendrycks et al. 2021) consists of challenging high-school mathematics competition problems. We retain only examples whose final answers can be evaluated through numeric extraction and normalization. Following the same protocol as for GSM8K, we reserve 10% of the filtered training set as a development set for hyperparameter tuning and checkpoint selection, while the official test split is used only for final evaluation. We report exact-match accuracy of the normalized final answer. Base Models. We conduct experiments on GSM8K (Cobbe et al. 2021) using Llama-3-8B Base (Grattafiori and others 2024) and Mistral-7B-v0.3 (Jiang et al. 2023). For MATH-Numeric, a numeric-answer subset of MATH (Hendrycks et al. 2021), we use Llama-3-8B Base and Qwen3-8B (Yang et al. 2025). We additionally evaluate Qwen3-4B-Instruct-2507 (Yang et al. 2025), a stronger reasoning-oriented instruction-tuned model, to investigate whether Cloud-based signals become more informative as the model’s reasoning capability improves. Baselines. We compare our method with these baselines under a semi-supervised setting, using 600 randomly sampled training problems as labeled data and treating the remainder as unlabeled. • Seed Model (Zero-shot CoT). We evaluate the seed model M0M_0 using zero-shot chain-of-thought prompting (Kojima et al. 2023) and greedy decoding, without any task-specific training. • Supervised Fine-Tuning (SFT-600). We use the base model to generate trajectories for 600 labeled problems and verify their extracted answers against the corresponding gold answers. The verified correct trajectories are then used for supervised fine-tuning. This baseline measures the benefit obtained from gold-verified model-generated supervision derived from the 600 visible problems. • Self-Consistency Preference Optimization (ScPO). Following ScPO (Prasad et al. 2025), we group sampled responses by their extracted final answers and select responses associated with the most and least frequent answers as the chosen and rejected responses, respectively. Each pair is weighted by the normalized difference between their answer frequencies. • Reward-Model-Based Preference Optimization (IRPORMIRPO_RM). We implement a semi-supervised adaptation of the reward-model baseline considered in ScPO (Prasad et al. 2025). For the 600 visible examples, gold-answer correctness determines the preference direction: correct responses form the preferred candidate set, while incorrect responses form the rejected candidate set. ArmoRM-Llama3-8B (Wang et al. 2024a) is then used to select the highest-scoring correct response as the chosen response and the lowest-scoring incorrect response as the rejected response. For each remaining unlabeled example, the same reward model scores all sampled responses, and the highest- and lowest-scoring responses are selected as the chosen and rejected responses, respectively. We construct at most one preference pair per eligible example and define its confidence using the reward-score margin cxIRPORM=RRM(x,yx+)−RRM(x,yx−).c_x^IRPO_RM=R_RM(x,y_x^+)-R_RM(x,y_x^-). All retained preference pairs are assigned a uniform training weight of 11. Hyperparameters. For ScPO and Cloud–ScPO, we sample K=8K=8 responses per problem and retain the top α=0.30α=0.30 candidate pairs, and set the merge ratio ρ=0.2ρ=0.2; Pure Cloud uses K=16K=16 and α=0.10α=0.10. Rollouts are generated with temperature 1.01.0, top-p=0.95p=0.95, and dataset-specific maximum lengths. Cloud scoring uses R=20R=20 banks with 200 labeled problems each, q=5q=5, τ=2.0τ=2.0, and λneg=1.0 _neg=1.0. Unless otherwise specified, DPO training uses β=0.10β=0.10, learning rate 5×10−65× 10^-6, effective batch size 16, up to 20 epochs, and early-stopping patience 5; Cloud–ScPO sets λNLL=1.0 _NLL=1.0. All test results use greedy decoding and exact-match accuracy. Computing Infrastructure. NVIDIA A40*4, each with 48 GB of memory. Main results and analysis Results on GSM8K Cloud–ScPO consistently improves reasoning accuracy. Table 1 reports the GSM8K results for Llama-3-8B and Mistral-7B. Across both models, SFT on the 600 visible examples improves over the corresponding base model, while preference optimization provides substantially larger gains. ScPO increases accuracy from 41.62% to 49.74% for Llama-3-8B and from 12.28% to 28.43% for Mistral-7B. Cloud–ScPO achieves the best performance on both models, reaching 52.24% and 32.92%, respectively. This corresponds to improvements of 2.50% over ScPO for Llama-3-8B and 4.49% for Mistral-7B. More preference pairs do not necessarily yield better performance. IRPO-RM uses substantially more preference pairs than either ScPO or Cloud–ScPO, but does not achieve comparable improvements. It reaches 46.40% accuracy with 3,794 pairs on Llama-3-8B and 15.23% with 2,511 pairs on Mistral-7B, remaining below both ScPO-based methods. These results indicate that preference-pair quality and selection confidence are more important than the raw number of training pairs. In particular, combining self-consistency with Cloud-based trajectory scoring produces more informative preference signals than reward-model-only selection. Table 1: Single-rollout test accuracy and number of preference pairs on GSM8K. The best accuracy for each backbone is shown in bold. Llama-3-8B Mistral-7B Method Acc. (%) # Pairs Acc. (%) # Pairs Base Model 39.50 – 10.31 – SFT-600 41.62 – 12.28 – ScPO 49.74 1,311 28.43 1,113 IRPO-RM 46.40 3,794 15.23 2,511 Cloud–ScPO 52.24 1,329 32.92 1,163 Experiment results on Math numeric Cloud-based selection is most effective for stronger reasoning models. As shown in Table 2, all preference-optimization methods improve over SFT-600 on MATH-Numeric. For Llama-3-8B, IRPO-RM achieves the highest accuracy of 23.51%, narrowly outperforming Cloud–ScPO at 23.01%. In contrast, Cloud–ScPO performs best on Qwen3-8B, reaching 62.33% and exceeding ScPO and IRPO-RM by 4.19% and 5.34%, respectively. Notably, Cloud–ScPO obtains this result with only 963 pairs, compared with 4,178 pairs used by IRPO-RM, highlighting the importance of pair quality rather than preference-data volume. The stronger improvement on Qwen3-8B further suggests that Cloud-based trajectory signals become more informative as the underlying model’s reasoning capability increases. Table 2: Single-rollout test accuracy and number of preference pairs on Math-numeric. Llama-3-8B Qwen3-8B Method Acc. (%) # Pairs Acc. (%) # Pairs Base Model 8.03 – 53.61 – SFT-600 21.01 – 55.52 – ScPO 22.10 409 58.14 963 IRPO-RM 23.51 2569 56.99 4178 Cloud–ScPO 23.01 410 62.33 963 Preference-Pair Quality Across Datasets Cloud–ScPO improves coverage while preserving correctness reliability. Table 3 compares ScPO and Cloud–ScPO across four dataset–model settings. Cloud–ScPO constructs at least as many valid pairs as ScPO in every setting. On GSM8K, it increases the ideal-pair rate from 89.02% to 90.37% for Llama-3-8B and from 56.51% to 59.67% for Mistral-7B, while reducing the rate of both-incorrect pairs. On MATH-Numeric, the correctness composition is nearly unchanged for Llama-3-8B and improves slightly for Qwen3-8B, whose ideal-pair rate increases from 98.65% to 98.75%. Importantly, Cloud–ScPO does not increase risky preference reversals. On GSM8K, the reversal rate decreases from 1.98% to 1.88% for Llama-3-8B and from 3.05% to 2.24% for Mistral-7B, while it remains unchanged in the two MATH-Numeric settings. Cloud scoring yields clearer chosen–rejected separation. Cloud–ScPO selects shorter chosen responses in three of the four settings and longer rejected responses in all four, producing a larger length gap between the preferred and rejected trajectories. The final two diagnostics are computed using a fixed deterministic text-analysis protocol(details are provided in Appendix D). Incompleteness is identified from empty or unfinished answer markers, unclosed expressions, and visibly cut-off endings, while repetition is identified from repeated sentences, paragraphs, lines, or recurring 10-token sequences. Across all settings, Cloud–ScPO assigns more incomplete, truncated, and repetitive trajectories to the rejected side. The effect is most pronounced for Llama-3-8B on GSM8K, where the incomplete-or-truncated rate increases from 27.84% to 41.46% and the repetition rate from 27.99% to 38.22%. Because these metrics characterize undesirable properties of rejected responses, their higher values under Cloud–ScPO suggest that Cloud scoring more frequently assigns structurally weaker trajectories to the rejected side of the preference pair. Pair quality remains constrained by rollout quality. For Llama-3-8B on MATH-Numeric, both methods obtain an ideal-pair rate of only about 7.6%, while more than 92% of pairs contain two incorrect responses. This suggests that preference construction becomes fundamentally limited when the candidate rollout pool is dominated by incorrect trajectories. Overall, Cloud–ScPO maintains or improves pair coverage and correctness composition, avoids additional preference reversals, and provides cleaner separation between informative chosen responses and low-quality rejected trajectories. Representative examples of cleaner chosen trajectories and repetitive, truncated, or off-topic rejected trajectories are provided in Appendix C. Table 3: Preference-pair comparison across datasets and backbone models. Bold values indicate better pair coverage or correctness composition within the same setting. Response lengths are measured in characters. GSM8K MATH-Numeric Llama-3-8B Mistral-7B Llama-3-8B Qwen3-8B Metric ScPO Cloud– ScPO ScPO Cloud– ScPO ScPO Cloud– ScPO ScPO Cloud– ScPO Pair coverage Successfully parsed pairs ↑ 1,311 1,329 1,113 1,163 409 410 963 963 Correctness composition Ideal pairs ↑ 1,167 1,201 629 694 31 31 950 951 Rate 89.02% 90.37% 56.51% 59.67% 7.58% 7.56% 98.65% 98.75% Both responses incorrect ↓ 118 103 450 443 378 379 7 6 Rate 9.00% 7.75% 40.43% 38.09% 92.42% 92.44% 0.73% 0.62% Risky reversed pairs ↓ 26 25 34 26 0 0 6 6 Rate 1.98% 1.88% 3.05% 2.24% 0.00% 0.00% 0.62% 0.62% Response diagnostics Average chosen length 452 328 545 429 797 782 372 422 Average rejected length 520 617 563 581 1,184 1,309 420 444 Incomplete or truncated rejected responses↑ 27.84% 41.46% 13.12% 19.43% 13.45% 17.80% 1.14% 1.25% Rejected responses with obvious repetition↑ 27.99% 38.22% 27.40% 31.90% 46.70% 50.73% 3.12% 3.43% Cloud-Gap Threshold Analysis Larger Cloud-score gaps concentrate pairs with stronger answer-level contrast. We directly examine the relationship between Cloud-score separation and pair distinctiveness using the GSM8K rollouts generated by Llama-3-8B. Specifically, we construct candidate pairs using Cloud scores and rank them according to the score difference between the chosen and rejected trajectories. We then measure the proportion of pairs whose two trajectories produce different final answers under different retention thresholds. Table 4: Cloud-gap analysis on GSM8K with Llama-3-8B. Candidate pairs are ranked by Cloud-score gap, and the pairs with the largest gaps are retained. Retained candidates # Pairs # Answer-disagreeing Rate (%) Top 5% 344 326 94.77 Top 10% 687 550 80.06 Top 20% 1,373 693 50.47 Top 30% 2,059 752 36.52 All candidates 6,862 953 13.89 As shown in Table 4, answer-disagreeing pairs are strongly concentrated among candidates with larger Cloud-score gaps. This result provides direct evidence, under the Llama-3-8B setting on GSM8K, that the Cloud-score gap can serve as a practical confidence signal for identifying more distinctive preference pairs. Ablation test The mean-token configuration achieves the strongest observed Pure Cloud result. Table 5 compares Pure Cloud configurations using mean-token and last-token trajectory representations. The mean-token configuration achieves the best accuracy of 55.32%, outperforming SFT-600 by 2.12% and the base model by 12.62 points. The last-token configuration also improves over both baselines, reaching 53.56%, but remains 1.76 points below the mean-token result. These results suggest that aggregating hidden states across the full response provides a more informative trajectory-level representation than relying only on the terminal token. However, because the two configurations use different retention ratios and numbers of preference pairs, the comparison reflects the combined effect of representation and pair-selection settings rather than a strictly controlled representation-only ablation. Reference-bank configuration affects global discrimination and ranking at the extremes. Table 6 compares reference-bank configurations under the same prompt pool and sampling strategy. With M=100M=100, using R=10R=10 banks provides the strongest overall discrimination, achieving the best AUC, balanced accuracy, and raw accuracy. Increasing the number of banks to R=20R=20 slightly reduces these aggregate metrics, but improves both extreme-ranking measures, raising Top-10 Correct from 77.63% to 78.00% and Bottom-10 Wrong from 72.96% to 74.98%. Holding R=20R=20 fixed, reducing the bank size from 100 to 50 support problems further improves Bottom-10 Wrong to 75.38%, but decreases the overall classification metrics. These results suggest that larger banks provide a more stable global estimate of trajectory quality, whereas additional or smaller banks can improve the identification of low-quality trajectories at the rejection end. We therefore view M=100M=100 and R=10R=10 as the strongest configuration for overall discrimination, while M=100M=100 and R=20R=20 offers a more balanced chosen–rejected ranking. Table 5: Pure Cloud ablation on MATH-Numeric using Qwen3-4B-Instruct. The best test accuracy is shown in bold. Method Retention Ratio # Pairs DPO β Learning Rate Accuracy (%) Base Model – – – – 42.70 SFT-600 – – – – 53.20 Mean-token 30% 1,280 0.10 5×10−65× 10^-6 55.32 Last-token 10% 426 0.05 2×10−62× 10^-6 53.56 Table 6: Reference-bank ablation on GSM8K with a fixed prompt pool of 400 and balanced sampling. M denotes the support problems per bank, and R denotes the number of reference banks. All values are percentages. M R AUC Bal. Acc. Raw Acc. Top-10 Correct Bottom-10 Wrong 100 10 67.02 63.07 65.42 77.63 72.96 100 20 66.73 62.43 65.14 78.00 74.98 50 20 66.50 62.22 64.56 77.56 75.38 Conclusion We proposed Cloud–ScPO, a semi-supervised preference-mining framework that combines global trajectory-representation geometry with prompt-level self-consistency. Using multi-bank reference Clouds built from a small labeled set, the method scores unlabeled reasoning trajectories and constructs high-confidence preference pairs. Experiments on GSM8K and MATH-Numeric across multiple model backbones show consistent improvements over SFT and ScPO. Pair-level analysis further indicates that Cloud scoring preserves correctness reliability while more effectively separating informative chosen responses from low-quality rejected trajectories. References A. Azaria and T. Mitchell (2023) The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 967–976. External Links: Link, Document Cited by: Related work. C. Burns, H. Ye, D. Klein, and J. Steinhardt (2024) Discovering latent knowledge in language models without supervision. External Links: 2212.03827, Link Cited by: Related work. J. Chauhan and M. Kaul (2022) BERTops: studying bert representations under a topological lens. External Links: 2205.00953, Link Cited by: Related work. C. Chen, K. Liu, Z. Chen, Y. Gu, Y. Wu, M. Tao, Z. Fu, and J. Ye (2024) INSIDE: llms’ internal states retain the power of hallucination detection. External Links: 2402.03744, Link Cited by: Related work. P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei (2017) Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, Cited by: Introduction. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: 1st item, Datasets and Metrics.. A. Fay, I. García-Redondo, Q. Wang, H. Dubossarsky, and A. Monod (2026) The shape of adversarial influence: characterizing llm latent spaces with persistent homology. In International Conference on Learning Representations, Cited by: Related work. R. B. Gabrielsson, B. J. Nelson, A. Dwaraknath, and P. Skraba (2020) A topology layer for machine learning. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 108, p. 1553–1563. Cited by: Appendix A, Problem Setup, Topology-Guided Cloud Scoring. A. Grattafiori et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: Datasets and Metrics.. Y. He, H. Wang, Z. Jiang, A. Papangelis, and H. Zhao (2024) Semi-supervised reward modeling via iterative self-training. In Findings of the Association for Computational Linguistics: EMNLP 2024, Cited by: Introduction. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. In Advances in Neural Information Processing Systems, Cited by: 2nd item, Datasets and Metrics.. A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal (2024) V-star: training verifiers for self-taught reasoners. External Links: 2402.06457, Link Cited by: Related work. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed (2023) Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: Datasets and Metrics.. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa (2023) Large language models are zero-shot reasoners. External Links: 2205.11916, Link Cited by: 1st item. S. Marks and M. Tegmark (2024) The geometry of truth: emergent linear structure in large language model representations of true/false datasets. External Links: 2310.06824, Link Cited by: Related work. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Cited by: Introduction. R. Y. Pang, W. Yuan, K. Cho, H. He, S. Sukhbaatar, and J. Weston (2024) Iterative reasoning preference optimization. External Links: 2404.19733, Link Cited by: Related work. A. Prasad, W. Yuan, R. Y. Pang, J. Xu, M. Fazel-Zarandi, M. Bansal, S. Sukhbaatar, J. Weston, and J. Yu (2025) Self-consistency preference optimization. External Links: 2411.04109, Link Cited by: Introduction, Related work, 3rd item, 4th item. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Cited by: Introduction, Related work, Pure Cloud optimization.. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Introduction. N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano (2020) Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, Cited by: Introduction. H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang (2024a) Interpretable preferences via multi-objective reward modeling and mixture-of-experts. External Links: 2406.12845, Link Cited by: 4th item. T. Wang, S. Li, and W. Lu (2024b) Self-training with direct preference optimization improves chain-of-thought reasoning. External Links: 2407.18248, Link Cited by: Related work. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou (2023) Self-consistency improves chain of thought reasoning in language models. External Links: 2203.11171, Link Cited by: Introduction, Related work. A. Yang, A. Li, B. Yang, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Datasets and Metrics.. E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman (2022) STaR: bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, Cited by: Introduction, Related work. N. Zhang, W. Ma, Z. Ma, J. Xu, J. Gao, J. Hao, R. He, and J. Xu (2026) Silence the judge: reinforcement learning with self-verifier via latent geometric clustering. arXiv preprint arXiv:2601.08427. Cited by: Related work, Trajectory Representations and Reference Clouds. A. Zomorodian and G. Carlsson (2005) Computing persistent homology. Discrete & Computational Geometry 33 (2), p. 249–274. External Links: Document Cited by: Appendix A, Problem Setup. Appendix A Appendix A: Topological Motivation and Representation Analysis Persistent-homology background. Let =zii=1N⊂ℝdZ=\z_i\_i=1^N ^d denote a point cloud of trajectory representations. We construct a Vietoris–Rips filtration by gradually increasing a distance threshold ϵε. At each filtration scale, nearby points are connected, and higher-dimensional simplices are added whenever all of their pairwise edges are present. Persistent homology records the birth and death of topological structures as ϵε increases (Zomorodian and Carlsson 2005; Gabrielsson et al. 2020). The zero-dimensional homology group H0H_0 describes connected components. At ϵ=0ε=0, every trajectory forms an independent component. As the filtration value increases, nearby components merge. The death time of an H0H_0 interval therefore records the distance scale at which one component joins another. Earlier component deaths indicate that the corresponding representations become connected at smaller filtration scales. The one-dimensional homology group H1H_1 describes independent cycles or loops. An H1H_1 feature is born when a closed cycle appears but has not yet been filled by higher-dimensional simplices, and it dies when the enclosed region becomes filled. Each feature is represented by a persistence interval [b,d),pers⁡(b,d)=d−b,[b,d), (b,d)=d-b, (12) where b and d are the birth and death scales, respectively. Longer intervals correspond to loop structures that persist over a wider range of filtration values, whereas very short intervals may reflect local variation or sampling noise. Although the main Cloud–ScPO method is motivated by the more stable H0H_0 connectivity pattern, we additionally report H1H_1 barcodes to provide a broader view of the latent-space topology. The H1H_1 results are exploratory and are not used in trajectory scoring or preference construction. Representation-processing comparison. We analyze representative Level 3 and Level 4 subsets of MATH. For each subset, we construct correct and incorrect point clouds using the same number of sampled trajectories, with N=200N=200 points in each cloud. We compare two representation-processing pipelines. The first uses unnormalized last-token hidden states. The second applies mean pooling over valid response-token hidden states, followed by ℓ2 _2 normalization. Because both the pooling strategy and normalization are changed, this experiment should be interpreted as a comparison between two complete representation-processing pipelines rather than as a controlled ablation that isolates either factor independently. Moreover, absolute filtration values should not be directly compared across the two pipelines because their distance scales differ substantially. (a) MATH Level 3 using unnormalized last-token representations. Correct and incorrect H0H_0 and H1H_1 barcodes are overlaid using matched point-cloud sizes. (b) MATH Level 4 using unnormalized last-token representations. Visible H1H_1 intervals occur over a broad filtration range. (c) MATH Level 3 after mean pooling over response-token hidden states and ℓ2 _2 normalization. Correct and incorrect H0H_0 and H1H_1 barcodes are displayed separately. (d) MATH Level 4 after mean pooling and ℓ2 _2 normalization. The processed representations retain a visible H0H_0 connectivity difference, while the H1H_1 separation becomes weaker. Figure 2: Persistent-homology analysis of correct and incorrect reasoning trajectories on representative MATH Level 3 and Level 4 subsets. Panels (a)–(b) use unnormalized last-token hidden representations and overlay the correct and incorrect barcodes. Panels (c)–(d) use mean-pooled response-token representations followed by ℓ2 _2 normalization and separately display the H0H_0 and H1H_1 barcodes of the two classes. Representation processing produces a more stable filtration scale and a more interpretable H0H_0 connectivity pattern, while the H1H_1 signal becomes weaker. Unnormalized last-token representations. Panels (a) and (b) of Figure 2 show the Level 3 and Level 4 results obtained from unnormalized last-token hidden states. The filtration values span a comparatively large range, and the correct and incorrect H0H_0 barcodes overlap substantially. The raw representations also exhibit visible H1H_1 intervals, indicating that the point clouds contain nontrivial one-dimensional structures. However, these structures occur over a broad and potentially unstable distance scale. Euclidean distances between unnormalized hidden states are affected by variation in vector norms, so part of the observed persistence may reflect representation magnitude rather than only structural similarity. In addition, a single terminal-token state can be influenced by answer formatting, punctuation, or sequence-ending behavior and may not summarize the complete reasoning trajectory. Mean-pooled and normalized representations. Panels (c) and (d) show the corresponding analyses after mean pooling over response-token hidden states and applying ℓ2 _2 normalization. The filtration values occupy a substantially more stable numerical range, making the relative connectivity of the correct and incorrect point clouds easier to interpret. Across both representative difficulty levels, the correct point clouds begin to merge into coherent components at smaller filtration values. The incorrect point clouds remain fragmented over a wider range before forming larger connected structures. This pattern suggests that correct reasoning trajectories occupy locally denser and more coherent regions of the processed representation space, whereas incorrect trajectories are distributed more heterogeneously. These observations are qualitative and based on representative Level 3 and Level 4 subsets. They motivate the proposed method but should not be interpreted as establishing an identical topological pattern for every difficulty level or model configuration. Behavior of the H1H_1 signal. The unnormalized last-token representations exhibit visible differences in their H1H_1 barcodes. However, these features are observed over a large and norm-sensitive filtration range. After mean pooling and ℓ2 _2 normalization, the H1H_1 signal becomes considerably weaker and less consistently separated between correct and incorrect trajectories. This sensitivity suggests that the observed loop structures are less robust to representation processing than the corresponding H0H_0 connectivity pattern. At present, we have not identified a reliable mechanism for converting H1H_1 features into an effective trajectory-scoring or preference-selection objective. We therefore treat the H1H_1 results as exploratory evidence of additional latent-space organization rather than as a component of Cloud–ScPO. Developing a robust use of higher-dimensional persistent-homology features is left for future work. From H0H_0 connectivity to Cloud scoring. In contrast to H1H_1, the relative connectivity behavior captured by H0H_0 remains interpretable after mean pooling and normalization. The earlier merging behavior of correct trajectories provides a direct operational signal: locally coherent regions can be identified by processing pairwise distances in ascending order and examining the components formed during the early portion of the filtration. This observation motivates the connectivity-induced components used in Cloud–ScPO. Within each correct or incorrect reference Cloud, we begin with one component per trajectory representation and process pairwise edges in ascending Euclidean distance. We stop after a predefined fraction of successful component merges and remove components smaller than a minimum-size threshold. The remaining early local components summarize coherent regions of the labeled trajectory distribution. A candidate trajectory is then evaluated through its soft nearest-component compatibility with the correct and incorrect reference Clouds. Compared with scoring against isolated reference points, this component-level procedure incorporates local structural information from the labeled trajectory distribution. The persistent H0H_0 analysis therefore motivates the component construction, while the resulting connectivity components provide the practical geometric objects used by the Cloud-scoring function. Appendix B Appendix B: Preference-Pair Construction Details This section provides the complete pair-construction procedures for Pure Cloud and Cloud–ScPO. For each unlabeled problem x, let x=yx,kk=1KY_x=\y_x,k\_k=1^K denote the K sampled reasoning trajectories, and let sCloud(x,y)s_Cloud(x,y) denote the averaged multi-bank Cloud score of trajectory y. Both methods first select concrete trajectories and apply the final response- and pair-validity checks. Candidate pairs that pass these checks are then ranked by their Cloud-score margins, after which the top-α fraction is retained. We use ⌈α||⌉ α|P| when converting the retention ratio into an integer number of pairs. Pure Cloud pair construction. Pure Cloud does not use answer extraction, answer clustering, or self-consistency. It directly selects the highest- and lowest-scoring eligible trajectories for each problem. Algorithm 1 Pure Cloud Preference-Pair Construction 1:Unlabeled problems UD_U; rollouts xx∈U\Y_x\_x _U; Cloud scores sClouds_Cloud; retention ratio α 2:Preference dataset PureCloudD_PureCloud 3:←∅P← 4:for all x∈Ux _U do 5: x←y∈x:ValidResponse(y)V_x←\y _x: ValidResponse(y)\ 6: if |x|<2|V_x|<2 then 7: continue 8: end if 9: yx+←arg⁡maxy∈x⁡sCloud(x,y)y_x^+← _y _xs_Cloud(x,y) 10: yx−←arg⁡miny∈x⁡sCloud(x,y)y_x^-← _y _xs_Cloud(x,y) 11: if not ValidPair(x,yx+,yx−) ValidPair(x,y_x^+,y_x^-) then 12: continue 13: end if 14: cx←sCloud(x,yx+)−sCloud(x,yx−)c_x← s_Cloud(x,y_x^+)-s_Cloud(x,y_x^-) 15: ←∪(x,yx+,yx−,cx)P ∪\(x,y_x^+,y_x^-,c_x)\ 16:end for 17:Sort P in descending order of cxc_x 18:m←⌈α||⌉m← α|P| 19:PureCloud←D_PureCloud← first m pairs in P 20:return PureCloudD_PureCloud Here, ValidResponse applies the response-level eligibility conditions used in the experiments, while ValidPair verifies that the selected responses are non-identical and satisfy the final serialization and parsing requirements. Pure Cloud assigns a uniform training weight to every retained pair and optimizes the resulting dataset using standard DPO. Cloud–ScPO pair construction. Cloud–ScPO follows ScPO in using answer-level self-consistency to determine the preference direction. It requires a unique majority-answer cluster and discards tied-majority problems. Cloud scores are then used to resolve ambiguities among equally frequent minority clusters and to select concrete trajectories within the preferred and rejected clusters. Algorithm 2 Cloud–ScPO Preference-Pair Construction 1:Unlabeled problems UD_U; rollouts xx∈U\Y_x\_x _U; Cloud scores sClouds_Cloud; total rollout count K; retention ratio α 2:Weighted preference dataset Cloud-ScPOD_Cloud -ScPO 3:←∅P← 4:for all x∈Ux _U do 5: x←y∈x:ValidResponse(y)∧CanonicalAnswer(y)≠∅V_x←\y _x: ValidResponse(y) CanonicalAnswer(y)≠ \ 6: Group trajectories in xV_x by canonical answer: x(a)=y∈x:CanonicalAnswer(y)=aC_x(a)=\y _x: CanonicalAnswer(y)=a\ 7: x←a:x(a)≠∅A_x←\a:C_x(a)≠ \ 8: if |x|<2|A_x|<2 then 9: continue 10: end if 11: Vx(a)←|x(a)|V_x(a)←|C_x(a)| for every a∈xa _x 12: max←arg⁡maxa∈x⁡Vx(a)A_ ← _a _xV_x(a) 13: if |max|≠1|A_ |≠ 1 then 14: continue ⊳ Discard tied-majority problem 15: end if 16: Let ax+a_x^+ be the unique answer in maxA_ 17: vmin←mina∈x∖ax+⁡Vx(a)v_ ← _a _x \a_x^+\V_x(a) 18: min←a≠ax+:Vx(a)=vminA_ ←\a≠ a_x^+:V_x(a)=v_ \ 19: Select the rejected answer cluster by ax−←arg⁡mina∈min⁡[miny∈x(a)⁡sCloud(x,y)]a_x^-← _a _ [ _y _x(a)s_Cloud(x,y) ] 20: yx+←arg⁡maxy∈x(ax+)⁡sCloud(x,y)y_x^+← _y _x(a_x^+)s_Cloud(x,y) 21: yx−←arg⁡miny∈x(ax−)⁡sCloud(x,y)y_x^-← _y _x(a_x^-)s_Cloud(x,y) 22: if not ValidPair(x,yx+,yx−) ValidPair(x,y_x^+,y_x^-) then 23: continue 24: end if 25: cxHybrid←sCloud(x,yx+)−sCloud(x,yx−)c_x^Hybrid← s_Cloud(x,y_x^+)-s_Cloud(x,y_x^-) 26: w(x)←Vx(ax+)−Vx(ax−)Kw(x)← V_x(a_x^+)-V_x(a_x^-)K 27: ←∪(x,yx+,yx−,cxHybrid,w(x))P ∪\(x,y_x^+,y_x^-,c_x^Hybrid,w(x))\ 28:end for 29:Sort P in descending order of cxHybridc_x^Hybrid 30:m←⌈α||⌉m← α|P| 31:Cloud-ScPO←D_Cloud -ScPO← first m pairs in P 32:return Cloud-ScPOD_Cloud -ScPO Several implementation details are worth emphasizing. First, Cloud–ScPO requires a unique majority-answer cluster; tied-majority problems are discarded rather than recovered through Cloud scoring. Second, when several minority clusters have the same minimum vote count, Cloud scoring selects the cluster containing the lowest-scoring trajectory. Third, the highest-scoring trajectory in the majority cluster is selected as yx+y_x^+, whereas the lowest-scoring trajectory in the selected minority cluster is used as yx−y_x^-. The final response- and pair-validity checks are applied before confidence ranking and top-α retention. The normalized vote weight uses the nominal total number of generated rollouts K as its denominator, rather than the number of valid or successfully parsed trajectories. Consequently, even when fewer than K rollouts remain valid, the pair weight is still w(x)=Vx(ax+)−Vx(ax−)K.w(x)= V_x(a_x^+)-V_x(a_x^-)K. Cloud–ScPO and ScPO may yield slightly different numbers of successfully constructed pairs because they select different concrete trajectories before the final validity checks. A trajectory selected by one method may pass these checks while the trajectory selected by the other method may not. The difference in pair counts is therefore not attributed to recovering tied-majority cases. Appendix C Appendix C: Qualitative Preference-Pair Examples We present representative examples from the Llama-3-8B MATH preference-pair files to illustrate how Cloud–ScPO changes the concrete chosen and rejected trajectories selected by ScPO. All examples appear in both pair datasets and preserve the same majority and minority answer counts. The difference therefore arises from trajectory-level Cloud selection rather than from a change in the answer-level preference direction. Response excerpts are shortened for readability, and omitted continuations are denoted by “[…]”. Lengths are measured in characters. Table 7: Summary of the qualitative examples. Response lengths are reported as chosen/rejected. The Cloud gap is the difference between the Cloud scores of the selected chosen and rejected trajectories. Selection ranks are reported within the retained 410-pair set; full-pool percentiles are approximated relative to all eligible candidates before top-30%30\% retention. Prompt ID Primary observation Votes ScPO length Cloud–ScPO length Cloud gap Selection rank math_train_610 Cleaner chosen trajectory 5/15/1 3660/28753660/2875 707/2875707/2875 0.03330.0333 10/41010/410 (approx. top 0.73%0.73\%) math_train_5707 Repetitive trajectory rejected 4/14/1 412/1084412/1084 412/2308412/2308 0.03730.0373 110/410110/410 (approx. top 8.05%8.05\%) math_train_115 Corrupted continuation rejected 5/15/1 418/473418/473 663/1903663/1903 0.03310.0331 50/41050/410 (approx. top 3.66%3.66\%) Example 1: selecting a cleaner trajectory within the majority-answer cluster. Consider the following problem: Problem. BoatWorks built 3 canoes in January and twice as many in each subsequent month. How many canoes were built in total by the end of March? Both methods identify 2121 as the majority answer, with a vote count of 55, and select a minority answer with count 11. The ScPO chosen trajectory first reaches the correct result but then continues with several unrelated questions: ScPO chosen excerpt. “In January they built 33 canoes, in February 66, and in March 1212. Thus, 3+6+12=213+6+12= 21. Q: What is the 13th digit in the decimal expansion of 17 17? A: Let’s think step by step. […]” Cloud–ScPO instead selects a self-contained trajectory from the same majority-answer cluster: Cloud–ScPO chosen. “In January they built 33 canoes. In February they built 3×2=63× 2=6, and in March they built 6×2=126× 2=12. Therefore, 3+6+12=21.3+6+12=21. The final answer is 21 21.” The Cloud-selected chosen response is reduced from 36603660 to 707707 characters while preserving the correct answer and the same answer-cluster preference. Its Cloud score is 0.03730.0373, compared with 0.00400.0040 for the rejected trajectory. This example illustrates how Cloud scoring can resolve trajectory-level ambiguity within a majority cluster whose members are indistinguishable under answer frequency alone. Example 2: assigning a repetitive continuation to the rejected side. For the problem Problem. What is the value of 3×(7−5)−53×(7-5)-5? both methods select the same concise chosen trajectory, which correctly computes 3×(7−5)−5=3×2−5=1.3×(7-5)-5=3× 2-5=1. Their rejected trajectories differ. The Cloud–ScPO rejected response begins by solving the original problem but then repeatedly introduces unrelated worked examples: Cloud–ScPO rejected excerpt. “What is 7−57-5? A: 7−5=27-5=2. A: 3×2=63× 2=6. A: 6−5=16-5=1. […] Example 1: What is 4×(8÷4)4×(8 4)? […] Example 2: What is 6×(12÷3)6×(12 3)? […]” The selected response extends to 23082308 characters and contains repeated templates and off-task continuations. Its Cloud score is −0.0947-0.0947, compared with −0.0574-0.0574 for the chosen response. This example is consistent with the aggregate diagnostic result that Cloud–ScPO places a larger proportion of repetitive or structurally weak trajectories on the rejected side. Example 3: rejecting a structurally corrupted response. Consider the problem Problem. If 2x2+1=19 2x^2+1= 19, find the average of all possible values of x. Both methods identify the correct majority answer 0, supported by five trajectories. The ScPO rejected trajectory remains on topic but makes an arithmetic error and reports −14- 14: ScPO rejected excerpt. “We obtain x=±3x=± 3. If x=3x=3, the average is 3+(−3)2=−12 3+(-3)2=- 12. If x=−3x=-3, the average is 0. Therefore, the average is −14 - 14.” Cloud–ScPO selects a more severely degraded trajectory as the rejected response. It begins with an incorrect derivation and subsequently continues with unrelated problems: Cloud–ScPO rejected excerpt. “Squaring both sides gives 2x2+1=192x^2+1=19, so x=±322x=± 3 22. But since x is an integer, x=3x= 3. Q: If 2x=220162^x=2^2016, find x. A: x=2016x=2016. Q: Find the greatest positive integer […]” The continuation produces a malformed final-answer extraction and grows from 473473 characters under ScPO to 19031903 characters under Cloud–ScPO. The selected chosen and rejected Cloud scores are −0.0799-0.0799 and −0.1130-0.1130, respectively. This case further illustrates that Cloud scoring can place off-topic and structurally corrupted trajectories on the rejected side even when answer frequency cannot distinguish trajectories within the same answer cluster. Interpretation. These examples are qualitative illustrations rather than an independent evaluation. They show two recurring behaviors reflected in the aggregate statistics: Cloud–ScPO often selects a more concise and self-contained trajectory from the preferred answer cluster, and it more consistently assigns repetitive, malformed, or off-task continuations to the rejected side. The method does not change the majority-based answer preference in these cases; instead, it provides a finer ordering over trajectories that have already been grouped by self-consistency. Appendix D Appendix D: Prompt Templates and Response Diagnostics Response-generation prompts. We use dataset-specific zero-shot chain-of-thought prompts for rollout generation. Within each dataset, the same prompt template and answer format are used across all compared methods. The base model uses these prompts to generate trajectories for the labeled problems, while the SFT model uses the same templates to generate trajectories for the unlabeled problems. Response Generation: GSM8K Prompt: Answer the following question step-by-step. When you are ready, place the final answer on a new line in the following format: #### <number> Q: question A: Let’s think step by step. Response Generation: MATH-Numeric Prompt: Answer the following question step-by-step. When you are ready, place the final answer on a new line in the following format: The final answer is <your answer>.The final answer is <your answer>. Q: question A: Let’s think step by step. Directly computed pair diagnostics. Pair coverage, correctness composition, preference reversals, and response lengths in Table 3 of the main paper are computed directly from the constructed preference-pair files. Let ai∗a_i denote the gold answer and let a^(yi+) a(y_i^+) and a^(yi−) a(y_i^-) denote the canonicalized answers extracted from the chosen and rejected responses. Each successfully parsed pair is classified as Ideal: a^(yi+)=ai∗,a^(yi−)≠ai∗, a(y_i^+)=a_i , a(y_i^-)≠ a_i , Both incorrect: a^(yi+)≠ai∗,a^(yi−)≠ai∗, a(y_i^+)≠ a_i , a(y_i^-)≠ a_i , Risky reversed: a^(yi+)≠ai∗,a^(yi−)=ai∗. a(y_i^+)≠ a_i , a(y_i^-)=a_i . A risky reversed pair places an incorrect trajectory on the chosen side and a correct trajectory on the rejected side. The corresponding rate is ReversalRate=#i:a^(yi+)≠ai∗∧a^(yi−)=ai∗Nparsed,ReversalRate= \# \i: a(y_i^+)≠ a_i a(y_i^-)=a_i \N_parsed, (13) where NparsedN_parsed is the number of successfully parsed preference pairs. Gold answers are used only for this post-hoc pair-quality analysis and are not accessed when constructing preferences for unlabeled problems. Chosen and rejected response lengths are measured directly in characters. Deterministic rejected-response diagnostics. The final two rows of Table 3—incomplete or truncated rejected responses and rejected responses with obvious repetition—are computed using a fixed deterministic text-analysis protocol. The same rules are applied to the rejected responses produced by ScPO and Cloud–ScPO in every dataset–backbone setting. A rejected response is labeled incomplete or truncated when at least one of the following conditions is detected: • the response is empty or contains no substantive generated content; • the expected final-answer marker is present but its answer is missing or unfinished; • the response contains an unclosed parenthesis, bracket, brace, mathematical environment, or boxed-answer expression; or • the response ends with a visibly incomplete sentence, equation, calculation, or reasoning step. A rejected response is labeled as containing obvious repetition when at least one of the following patterns is detected: • an identical normalized sentence occurs multiple times; • an identical non-empty line or paragraph is repeated; • a question–answer block or reasoning segment is reproduced without meaningful progression; or • a normalized sequence of ten consecutive tokens recurs within the same response. Normalization for repetition detection removes inconsequential whitespace differences before comparing textual units. Ordinary reuse of mathematical variables, short function words, or necessary intermediate expressions is not treated as repetition. The two labels are assigned independently, so a rejected response may satisfy both diagnostic conditions. Aggregation and interpretation. For a method with N evaluated rejected responses, the reported rate for diagnostic d is Rated=#i:d(yi−)=1N,Rate_d= \#\i:d(y_i^-)=1\N, (14) where yi−y_i^- denotes the rejected response in the i-th preference pair. Because these diagnostics characterize undesirable properties of the rejected side, higher values do not imply that the underlying model generates more defective trajectories. Instead, they indicate that the preference-construction method more frequently assigns incomplete, truncated, or repetitive trajectories to rejection rather than selecting them as preferred responses. Across all four dataset–backbone settings, Cloud–ScPO produces higher rates for both diagnostics. This result supports the conclusion that Cloud scoring provides clearer response-level separation by placing a larger proportion of structurally degraded trajectories on the rejected side.

Instant research discovery