Paper deep dive
Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning
Sida Li, Yuanxin Liu, Shicheng Li, Linli Yao, Xu Sun, Haoli Bai, Xiaohui Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/12/2026, 6:58:58 AM
Summary
The paper introduces TQS (Trajectory Quality Score), a structural scoring framework for evaluating coding-agent interaction trajectories without relying on LLM judges. TQS assesses trajectory quality through seven linear-time metrics focused on structured convergence: information focus (OEC, PSN), intent-action alignment (IAC), policy dynamics (TTE, SCP, PED), and a guardrail for dirty patterns (DPI). The authors applied TQS to 155,144 successful trajectories from the CoderForge-Preview corpus, filtering them into top, bottom, and random subsets. Fine-tuning Qwen3-Coder-30B-A3B-Instruct on top-ranked TQS trajectories resulted in superior performance on SWE-Bench-Verified compared to models trained on random or low-quality subsets, demonstrating that process-level trajectory quality is a critical factor in supervised fine-tuning.
Entities (12)
Relation Signals (10)
Qwen3-Coder-30B-A3B-Instruct → evaluatedon → SWE-bench Verified
confidence 95% · Models trained on top-ranked trajectories consistently outperform those trained on random or bottom-ranked subsets on SWE-Bench-Verified.
TQS → usesmetric → Intent-Action Consistency
confidence 95% · tqs uses seven components... (B1) Intent-Action Consistency (iac).
TQS → usesmetric → Tool Transition Entropy
confidence 95% · tqs uses seven components... (C1) Tool Transition Entropy (tte).
TQS → usesmetric → Solution Commitment Point
confidence 95% · tqs uses seven components... (C2) Solution Commitment Point (scp).
TQS → usesmetric → Post-Error Diversity
confidence 95% · tqs uses seven components... (C3) Post-Error Diversity (ped).
TQS → usesmetric → Dirty Pattern Indicator
confidence 95% · tqs uses seven components... (D1) Dirty Pattern Indicator (dpi).
TQS → usesmetric → Progressive Scope Narrowing
confidence 95% · tqs uses seven components... (A2) Progressive Scope Narrowing (psn).
TQS → usesmetric → Observation Entropy Collapse
confidence 95% · tqs uses seven components... (A1) Observation Entropy Collapse (oec).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Open coding agents are commonly trained through supervised fine-tuning on interaction trajectories whose final patches pass the benchmark test suite. However, this outcome-level filtering criterion ignores process quality: trajectories that eventually succeed may still contain inefficient interaction patterns such as repetitive file inspection, unstable tool usage, or delayed modification behavior, all of which become part of the imitation target. We propose TQS (Trajectory Quality Score), a structural scoring framework for coding-agent trajectories that operates directly on trajectory structure, requires no LLM judge, and runs in linear time. TQS characterizes trajectory quality through structural signals organized around structured convergence: information focus, intent--action alignment, and policy dynamics. Using 155,144 successful trajectories from a public open-license corpus, we rank trajectories by TQS and fine-tune coding agents on filtered subsets. Models trained on top-ranked trajectories consistently outperform those trained on random or bottom-ranked subsets on SWE-Bench-Verified. Our results suggest that process-level trajectory quality is an important factor in coding-agent supervision and that structural trajectory filtering provides an effective and scalable alternative to LLM-based data selection.
Tags
Links
- Source: https://arxiv.org/abs/2610.13152v1
- Canonical: https://arxiv.org/abs/2610.13152v1
Trouble viewing inline? Open PDF directly →
Full Text
69,181 characters extracted from source content.
Expand or collapse full text
Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning Sida Li Affiliation: National Key Laboratory for Multimedia Information Processing, Peking University Email: baihaoli@huawei.com Yuanxin Liu Affiliation: National Key Laboratory for Multimedia Information Processing, Peking University Email: lixiaohui33@huawei.com Shicheng Li Affiliation: National Key Laboratory for Multimedia Information Processing, Peking University Linli Yao Affiliation: National Key Laboratory for Multimedia Information Processing, Peking University Xu Sun Affiliation: National Key Laboratory for Multimedia Information Processing, Peking University Haoli Bai Affiliation: Huawei Technologies Co., Ltd. lisida,liuyuanxin,linliyao@stu.pku.edu.cn lisc99,xusun@pku.edu.cn Xiao-Hui Li Affiliation: Huawei Technologies Co., Ltd. lisida,liuyuanxin,linliyao@stu.pku.edu.cn lisc99,xusun@pku.edu.cn Abstract Open coding agents are commonly trained through supervised fine-tuning on interaction trajectories whose final patches pass the benchmark test suite. However, this outcome-level filtering criterion ignores process quality: trajectories that eventually succeed may still contain inefficient interaction patterns such as repetitive file inspection, unstable tool usage, or delayed modification behavior, all of which become part of the imitation target. We propose tqs (Trajectory Quality Score), a structural scoring framework for coding-agent trajectories that operates directly on trajectory structure, requires no LLM judge, and runs in linear time. tqs characterizes trajectory quality through structural signals organized around structured convergence: information focus, intent–action alignment, and policy dynamics. Using 155,144 successful trajectories from a public open-license corpus, we rank trajectories by tqs and fine-tune coding agents on filtered subsets. Models trained on top-ranked trajectories consistently outperform those trained on random or bottom-ranked subsets on SWE-Bench Verified. Our results suggest that process-level trajectory quality is an important factor in coding-agent supervision and that structural trajectory filtering provides an effective and scalable alternative to LLM-based data selection. Project page: https://github.com/DuHeng0128/TQS.git. 1 Introduction Coding agents that interact with sandboxed environments to inspect repositories, execute tests, and edit files have recently achieved strong performance on SWE-Bench Verified-style software engineering tasks (Jimenez et al., 2024; OpenAI, 2024). Beyond advances in foundation models and inference-time scaffolding, an increasingly important ingredient in these systems is trajectory supervision: training on multi-turn agent demonstrations consisting of reasoning traces, tool calls, tool observations, and final patches (Pan et al., 2025; Ma et al., 2024; Ariyak et al., 2026). For medium-scale open-weight models, a common and effective practice is to collect such trajectories from more capable teacher agents and retain those whose final patches pass the benchmark test suite. However, outcome-level correctness does not necessarily imply process-level quality. Among successful trajectories, behavioral patterns vary substantially. Some trajectories localize the bug quickly, maintain coherent search scope, and transition efficiently from exploration to modification. Others repeatedly inspect the same files, interrupt stalled shell commands, or postpone meaningful edits until late in the interaction budget, yet still eventually reach a passing patch through extended trial-and-error. These differences matter because trajectory-level supervised fine-tuning exposes the student to the full interaction process, including reasoning traces, tool use, and the final patch (Kamoi et al., 2024). Inefficient or unstable interaction patterns therefore become part of the training signal. Figure 1: Overview of tqs. Contrasting high-quality (focused and convergent) with low-quality (repetitive and unstable) successful trajectories, TQS evaluates trajectory quality along three structural dimensions: Information Focus (OEC, PSN), Intent–Action Alignment (IAC), and Policy Dynamics (TTE, SCP, PED), with a Dirty Pattern Indicator (DPI) guardrail. These measures are aggregated into a weighted score in [0,1][0,1]. The instruction-tuning literature has shown that curated subsets can outperform larger, noisier training corpora (Zhou et al., 2023; Chen et al., 2024; Liu et al., 2024; Li et al., 2024b; Xia et al., 2024b). Coding-agent trajectories pose a different selection problem. They are long interaction histories whose ordering records exploration, editing, testing, and recovery. Perplexity-style scores are confounded by length and tokenization, while LLM-based judges are costly at the scale of hundreds of thousands of trajectories and billions of tokens. Furthermore, these approaches are not designed to characterize how effectively an agent organizes its search process over time. In this work, we propose tqs (Trajectory Quality Score), a structural scoring framework for OpenHands-style coding-agent trajectories. The framework is built around structured convergence: as an agent approaches a solution, a strong trajectory becomes increasingly organized around the relevant fix. This organization appears at three levels. The agent should focus its observations and file operations on task-relevant evidence, keep its stated intent aligned with its tool actions, and move from broad exploration toward productive edit, test, and recovery behavior. We operationalize these levels through seven trajectory-level measures, each computable in linear time from event logs alone. Outcome labels are used only to validate candidate structural signals during metric development; once the components are fixed, tqs ranks successful trajectories without inspecting the final patch, held-out test result, or downstream SFT performance. Empirically, we score 155,144 successful trajectories from CoderForge-Preview (Ariyak et al., 2026) and construct matched top-ranked, bottom-ranked, and random subsets from the eligible fixed-budget SFT pool under task- and repository-level diversity constraints. We LoRA-fine-tune the same Qwen3-Coder-30B-A3B-Instruct model (Ariyak et al., 2026; Yang et al., 2025a; Hu et al., 2022) on each subset using identical hyperparameters. Models trained on top-ranked trajectories consistently outperform those trained on random or bottom-ranked subsets on SWE-Bench Verified (OpenAI, 2024) under the OpenHands framework (Wang et al., 2025). These results suggest that trajectory-level process structure is an important and previously underexplored dimension of coding-agent supervision. Our contributions are threefold: (i) we formulate coding-agent trajectory quality through structured convergence and operationalize it with seven efficient trajectory-level measures; (i) we introduce a scalable framework for trajectory scoring and filtering without relying on LLM judges or external reward models; and (i) we present an end-to-end open pipeline for trajectory scoring, filtering, fine-tuning, and evaluation, demonstrating that quality-aware trajectory selection improves downstream coding-agent performance. 2 Related Work Coding agents and trajectory supervision. Recent coding agents interact with repositories through tool-augmented environments that support file inspection, shell execution, testing, and patch generation. Frameworks such as SWE-agent (Yang et al., 2024) and OpenHands (Wang et al., 2025) standardized the agent-computer interaction pipeline and enabled large-scale collection of executable agent trajectories. Building on these infrastructures, datasets such as SWE-Gym (Pan et al., 2025), Lingma SWE-GPT (Ma et al., 2024), and CoderForge-Preview (Ariyak et al., 2026) progressively scaled teacher-generated trajectory supervision for open-weight coding agents. Recent work has also explored trajectory synthesis and reward-guided curation for software and GUI agents (Sun et al., 2025). While reinforcement-learning approaches for software engineering agents (Wei et al., 2025) further improve exploration and decision-making, they typically still rely on strong supervised initialization, making trajectory quality important for both SFT and RL-based pipelines. Data selection for supervised fine-tuning. Data-selection methods for supervised fine-tuning usually use an external judge or a training-derived importance signal. AlpaGasus (Chen et al., 2024) and DEITA (Liu et al., 2024) ask stronger LLMs to score samples. LESS (Xia et al., 2024b), Cherry-LLM (Li et al., 2024b), and Superfiltering (Li et al., 2024a) estimate utility through gradient influence or model-internal signals; related methods have also been applied to code instruction tuning (Wang et al., 2024). These selectors were developed for short, single-turn samples. Applying them to an agent trajectory would require redefining their sample-level signals over a long interaction history. tqs instead uses the temporal structure already present in the event log. Process-level evaluation and trajectory quality. Recent work has increasingly emphasized evaluating intermediate reasoning and interaction processes rather than final outcomes alone. Prior studies on chain-of-thought faithfulness analyze whether intermediate reasoning traces are consistent with the eventual answer (Lanham et al., 2023; Bentham et al., 2024). AgentDiagnose (Ou et al., 2025) applies an LLM rubric to browser-agent trajectories and shows that process-aware trajectory selection can improve training efficiency. Beyond this, several methods use entropy or behavioral diversity to analyze and optimize agent policies (Chen et al., 2026; Li et al., 2026; Dong et al., 2025; Yu et al., 2025a). Recent work has also explored process supervision and trajectory curation for software-engineering agents using privileged patch-level information (Ma et al., 2026). 3 Trajectory Quality Score 3.1 Preliminaries and Notation A trajectory τ=(s0,s1,…,sn−1)τ=(s_0,s_1,…,s_n-1) is a sequence of n steps. Each step is represented as st=(at,mt,ot,Ft,εt),s_t=(a_t,m_t,o_t,F_t, _t), where at∈a_t is an action type (one of think, read_file, write_file, execute_bash, search, submit, other); mtm_t is the free-text thought that precedes the action; oto_t is the tool observation returned after the action; Ft⊆ℱF_t is the set of file paths touched by the action; and εt∈0,1 _t∈\0,1\ indicates whether oto_t matches a predefined error pattern. This tuple is recoverable from any OpenHands-style event log without model inference. 3.2 Design Intuition and Score Construction A coding-agent trajectory is a search process over observations, files, actions, and intermediate hypotheses. We view a high-quality trajectory as one that exhibits structured convergence: over the course of the interaction, the search becomes organized around the emerging fix. tqs relies on event-log evidence that the agent is narrowing its search toward a task-relevant region. At minimum, it provides a low-cost way to filter out clearly undesirable trajectory patterns. Structured convergence has three observable levels. (A) Information focus asks whether the agent narrows the relevant evidence through its observations and file operations. (B) Intent–action alignment asks whether the stated thought before a tool call points in the same direction as the action that follows. (C) Policy dynamics asks whether the interaction policy develops a productive rhythm of exploration, execution, and error recovery. Some failures are categorical rather than gradual; the trajectories associated with them are inefficient. A run may trigger an early-stop pattern, never make a substantive edit, or repeat the same failing command. dpi therefore sits outside the three-dimensional organization as a guardrail for these cases. The axes motivate candidate measures, and the selection protocol decides which ones enter tqs. Each component is computable from the step tuple in O(n)O(n) CPU time without auxiliary LLM judging, so the scorer can process event logs spanning roughly 6B tokens. The checks in §3.4 retain measures that discriminate above chance, have limited correlation with step count, and preserve behavioral diversity during subset selection. 3.3 Components and Aggregation tqs uses seven components, each normalized to [0,1][0,1] with larger values indicating cleaner structure. A component is left undefined when its required event is absent; Eq. 1 handles these cases during aggregation. (A1) Observation Entropy Collapse (oec). For each observation oto_t, we compute character-level Shannon entropy (Shannon, 1948). Let H~t H_t be its three-step moving average, H¯open H_open the mean entropy of the first min(5,n) (5,n) observations, and rt=H~t/H¯openr_t= H_t/ H_open. Then H(ot)=−∑cpt(c)log2pt(c),soec=clamp(1−mintrt,0,1). gatheredH(o_t)=- _cp_t(c) _2p_t(c),\\ s_ oec=clamp\! (1- _tr_t,0,1 ). gathered Lower relative entropy indicates that observations have concentrated around a smaller body of evidence. Since an unproductive loop can produce the same pattern, oec is used with tte and dpi. (A2) Progressive Scope Narrowing (psn). We count the distinct files touched in a five-step window and compute Kendall’s τK _K (Kendall, 1938) between this scope and the step index over the second half of the trajectory: scopet=|⋃k=max(0,t−4)tFk|,spsn=clamp(1−τK2,0,1). gatheredscope_t= | _k= (0,t-4)^tF_k |,\\ s_ psn=clamp\! ( 1- _K2,0,1 ). gathered A shrinking active file set gives negative τK _K and therefore a higher score, matching the localization pattern used by coding agents (Xia et al., 2024a; Yu et al., 2025b). (B1) Intent–Action Consistency (iac). A fixed lexicon-based parser maps the thought preceding each non-think action to a coarse intended action a^t a_t: reading, editing, execution/testing, or search. For the extractable steps ℐ=t:a^t is definedI=\t: a_t is defined\, siac=1|ℐ|∑t∈ℐ[a^t=at].s_ iac= 1|I| _t 1[ a_t=a_t]. iac is undefined when ℐ=∅I= . It captures coarse thought–action consistency characteristic of ReAct-style interaction (Yao et al., 2023), without assessing semantic correctness. (C1) Tool Transition Entropy (tte). Let p(i→j)p(i\!→\!j) be the empirical frequency of adjacent transition (i,j)(i,j) in the seven-action alphabet. We normalize its entropy by the maximum over 49 transition types: stte=−∑i,jp(i→j)log2p(i→j)log249.s_ tte= - _i,jp(i\!→\!j) _2p(i\!→\!j) _249. tte penalizes long single-tool loops. Focus and guardrail measures provide the context needed to distinguish useful variation from erratic switching. (C2) Solution Commitment Point (scp). A substantive edit is a write_file action whose observation contains no error pattern. Let t∗t^* be the first such edit, u=t∗/nu=t^*/n, d(u)=max(0,0.2−u,u−0.5)d(u)= (0,0.2-u,u-0.5), and σ=0.2σ=0.2. We set sscp=0,no substantive edit;exp(−d(u)2/(2σ2)),otherwise.s_ scp= \ array[]@l@r@0,&no substantive edit;\\ \! (-d(u)^2/(2σ^2) ),&otherwise. array . The score is one for u∈[0.2,0.5]u∈[0.2,0.5] and decays for earlier or later commitment. Because u is relative time, scp does not prefer a particular absolute trajectory length. (C3) Post-Error Diversity (ped). Let ℰτ=t:εt=1,t<n−1E_τ=\t: _t=1,t<n-1\ be the error steps followed by an action. We combine a change in action type with the Jaccard distance between successive file sets: Jacc(Ft+1,Ft)=1−|Ft+1∩Ft||Ft+1∪Ft|,dt=12([at+1≠at]+Jacc(Ft+1,Ft)),sped=1|ℰτ|∑t∈ℰτdt, gatheredJacc(F_t+1,F_t)=1- |F_t+1∩ F_t||F_t+1∪ F_t|,\\ d_t= 12\! (1[a_t+1≠ a_t]+Jacc(F_t+1,F_t) ),\\ s_ ped= 1|E_τ| _t _τd_t, gathered ped is undefined when ℰτE_τ is empty. Repeating the same action on the same files gives zero; changing the tool, file scope, or both raises the score (Shinn et al., 2023; Kamoi et al., 2024). (D1) Dirty Pattern Indicator (dpi). dpi captures categorical failures that may survive the continuous measures: early stopping, no substantive edit, repeated shell-command loops, and repeated errors. With repeated-loop fraction lp and error-repetition rate er, p=0.4 1[early]+0.4 1[noedit]+min(0.3,3lp)+min(0.3,0.6er),sdpi=1−min(1,p). gatheredp=0.4\,1[early]+0.4\,1[no\ edit]\\ + (0.3,3\,lp)+ (0.3,0.6\,er),\\ s_ dpi=1- (1,p). gathered A trajectory with no flagged pattern receives full guardrail credit; each detected pattern reduces the contribution of dpi. Aggregation. We use fixed, hand-set weights throughout: woec=0.30w_ oec=0.30, wiac=0.25w_ iac=0.25, wpsn=wtte=wped=wdpi=0.10w_ psn=w_ tte=w_ ped=w_ dpi=0.10, and wscp=0.05w_ scp=0.05. The resulting totals are 0.40 for information focus, 0.25 for intent–action alignment, 0.25 for policy dynamics, and 0.10 for the guardrail. Perturbing woecw_ oec, wiacw_ iac, or wdpiw_ dpi one at a time retains 93.1–99.6% of the Top-1,024 selection (Appendix I). The final tqs is a renormalized weighted mean over the components defined for a trajectory: tqs(τ)=∑c∈∗(τ)wc⋅sc(τ)∑c∈∗(τ)wc, tqs(τ)\;=\; _c ^*(τ)w_c· s_c(τ) _c ^*(τ)w_c, (1) where ∗(τ)C^*(τ) is the set of components defined on τ. Trajectories with n<3n<3 or component coverage below 4/74/7 are excluded. Outcome labels are used only for component admission during metric development (as described in §3.4); after the components and weights are fixed, tqs ranks successful trajectories using event-log structure alone. 3.4 Discriminant Validity and External Sanity Checks We use task outcome as a weak admission signal for candidate components. Ineffective process patterns often make failure more likely, so a useful component should tend to rank a passing trajectory above a failing trajectory from the same task. This comparison filters signals with the wrong direction or strong confounds. Once the components are fixed, tqs ranks only task-successful trajectories for training, using event-log structure alone and no further patch-level supervision. Concretely, we pair each passing trajectory with a failing trajectory from the same task. For each candidate measure, we compute two quantities. The first is concordance: the rate at which the measure assigns a higher score to the passing trajectory than to its paired failing trajectory, with 0.5 corresponding to chance. The second is the absolute Spearman correlation |ρ||ρ| with trajectory length, which serves as a simple check against length-driven proxies. A measure is admitted only if its concordance exceeds 0.5 and |ρ|<0.25|ρ|<0.25. Table 1 summarizes the results. The seven retained components pass both criteria, and the composite tqs reaches a concordance of 0.732, above every individual component. The same criteria exclude several candidates: BUS, TGCR, and DCS-file fail the concordance criterion, while DCS-Jaccard is dominated by length correlation (|ρ|=0.729|ρ|=0.729). hcr passes the numerical thresholds, but including it caused the Top-1K set to concentrate 99.9% of its mass on a single binary behavior pattern. This mode collapse is invisible to pairwise concordance but matters for SFT, where trajectories should preserve behavioral diversity. We therefore add behavioral-diversity preservation as a third admission criterion and exclude hcr under it. Additional details on the rejected measures are provided in Appendix E. Measure Conc. |ρ||ρ| Outcome OEC 0.640 0.103 ✓kept IAC 0.612 0.087 ✓kept PSN 0.585 0.142 ✓kept SCP 0.598 0.058 ✓kept TTE 0.563 0.218 ✓kept PED 0.541 0.034 ✓kept DPI 0.591 0.071 ✓guardrail TQS Composite 0.732 0.092 — HCR 0.527 0.014 × diversity BUS/TGCR/DCS-f. ≤0.479≤ 0.479 <0.25<0.25 × conc. DCS-Jacc. — 0.729 × length Table 1: Discriminant validity on paired success/failure trajectories from CoderForge-Preview. tqs’s composite concordance exceeds every individual component. Dropped measures reflect the role of admission protocol: hcr is rejected for behavioral mode collapse, BUS/TGCR/DCS-file for low concordance, and DCS-Jaccard for length correlation. Figure 2: External validity of tqs on the released scoring records. We bin 413,278 records into ten equal-frequency deciles by tqs. These records include the source shards and the release’s reward-1 convenience shards. The resolved rate increases monotonically from 54.1% in D1 to 86.9% in D10. Figure 3: Two passing trajectories on the same pygments LassoLexer task. The high-tqs run reaches the fix in 36 steps. The low-tqs run takes 73 steps and includes many interrupts and forced process terminations. Their final outcomes are equal, and their tqs values differ by 0.36. External outcome check. We also compare the finalized score with the test harness’s resolved label, which is absent from the scoring function. Figure 2 shows a monotonic rise in resolved rate across the ten tqs deciles, from 54.1% in D1 to 86.9% in D10. The 32.9-point spread has no inversion at a decile boundary. Within a fixed step-count band, pair-AUC is 1.000 for the composite and above 0.96 for oec, iac, and ped (Appendix G). 4 Experimental Setup Qualitative motivation. Figure 3 shows two passing OpenHands runs on the same pygments LassoLexer task. Both resolve the task. The high-tqs trajectory reaches the fix in 36 steps through a compact read–diagnose–plan–review pattern; the low-tqs trajectory takes 73 steps and contains many interrupt and kill actions. The largest component gaps occur in observation focus (oec: 0.738 vs. 0.312), commitment timing (scp: 0.990 vs. 0.528), and post-error adaptation (ped: 1.000 vs. 0.000), revealing substantial behavioral differences that the final outcome alone does not capture. Appendix A reports all seven component scores. Data and subset construction. Subset Top Bottom Random # trajectories 1,024 1,024 1,024 Mean tqs 0.714 0.454 0.589 Mean steps 34.2 53.0 41.0 Mean tokens (K) 28.1 26.3 26.9 Tasks / repos 847/209 847/204 825/217 Max per task 30 30 59 Source mix (%) 61/20/18 61/20/18 61/20/18 Dirty flag (%) 1.0 64.3 13.6 Table 2: Training subsets. Top and Bottom have zero tqs overlap (Top.min =0.706>=0.706> Bottom.max =0.470=0.470). Source mix is Smith/Reb./Gym. We score CoderForge-Preview (Ariyak et al., 2026), an open-license corpus of 258,134 unique OpenHands-format trajectories produced by a Qwen3-Coder-480B teacher on 51,201 tasks from R2E-Gym (Jain et al., 2025), SWE-Smith (Yang et al., 2025b), and SWE-Rebench (Badertdinov et al., 2025). The 155,144 passing trajectories form the candidate pool and were deduplicated by trajectory_id before subset construction. For fixed-budget SFT, we keep trajectories with at least 3 steps, tqs coverage ≥4/7≥ 4/7, and tokenized length at most 32,768, leaving 38,382 eligible trajectories. From this pool we construct matched Top-K, Bottom-K, and Random-K subsets of 1,024 trajectories, controlling task coverage, repository coverage, and source proportions. Table 2 summarizes the resulting subsets. Sampler details are in Appendix C.1. Training and evaluation. All SFT runs use Qwen3-Coder-30B-A3B-Instruct (Yang et al., 2025a) with LoRA (Hu et al., 2022). The training recipe is fixed across conditions: adapters are placed on the attention projections, loss is applied to assistant and tool_call segments only, and every hyperparameter except the selected trajectories is held constant (Appendix C). We evaluate pass@1 on SWE-Bench Verified (OpenAI, 2024) using OpenHands v0.52.1 (Wang et al., 2025), matching the scaffold and version used to generate the training trajectories. We also recompute tqs components on evaluation rollouts to diagnose process-level behavioral changes. pass@1 by difficulty (%) ID Method |||D| Easy Medium Hard Overall Δ No-SFT control B0 Qwen3-Coder-30B-A3B (base, no SFT) — 36.6 19.2 2.2 24.4 — Subset baselines (matched 1,024 trajectories, identical SFT recipe) B1 Random-K 1,024 40.7 16.9 2.2 24.8 ++0.4 B2 Bottom-K (lowest tqs) 1,024 33.5 16.9 0.0 21.8 −-2.6 B3 Shortest-K (length-only) 1,024 37.6 18.4 0.0 24.2 −-0.2 B4 DPI-only-K (guardrail only) 1,024 37.6 19.9 2.2 25.2 ++0.8 B5 LLM-Judge-K (GPT-4o rubric) 1,024 43.3 21.1 6.7 28.4 ++4.0 B6 Top-LM-K (length-matched to Random) 1,024 40.7 20.7 4.4 27.0 ++2.6 Ours Top-K (highest tqs) 1,024 42.3 20.3 6.7 27.6 ++3.2 Table 3: Main results on SWE-Bench Verified. All SFT runs use the same Qwen3-Coder-30B-A3B-Instruct base, LoRA hyperparameters, and OpenHands evaluation scaffold; only the selected training trajectories differ. Difficulty buckets follow OpenAI (2024). Baselines. We compare against the untuned base model, Random-K, Bottom-K, Shortest-K by token count, dpi-only-K, GPT-4o LLM-Judge-K under an AgentDiagnose-style (Ou et al., 2025) rubric, and Top-LM-K, which length-matches high-tqs trajectories to the Random-K length distribution. All baselines are drawn from the same eligible pool and trained with the same SFT recipe. 5 Results 5.1 Main Results Table 3 reports the main results. Under the same backbone, SFT recipe, and evaluation scaffold, the selected trajectories have a clear effect on downstream performance. Random-K improves only slightly over the base model, from 24.4 to 24.8 pass@1. In contrast, tqs-Top reaches 27.6, improving over Random-K by 2.8 points and over the base model by 3.2 points. Bottom-K moves in the opposite direction, dropping to 21.8. This ordering shows that trajectories with the same reward-1 outcome can differ substantially in their value as supervision. The simpler baselines do not explain the tqs gain. Shortest-K reaches 24.2, below Random-K, so selecting short trajectories alone is not sufficient as a selection criterion. DPI-only-K gives a small improvement over Random-K but remains 2.4 points below tqs-Top. The length-matched control, Top-LM-K, reaches 27.0 after matching the length distribution of Random-K. It retains most of the Top–Random gain, which indicates that length is correlated with trajectory quality but does not fully account for the effect of tqs. Among the non-tqs selectors, LLM-Judge-K is the strongest baseline at 28.4 pass@1. This is 0.8 points above tqs-Top, showing that rubric-based GPT-4o judging remains competitive. Still, tqs-Top recovers most of its improvement over Random-K and matches LLM-Judge-K on Hard tasks at 6.7. Figure 4 summarizes this trade-off between accuracy and selection cost. Their selection costs differ more sharply. Under the prompt, token-count, and pricing assumptions used for the baseline, GPT-4o judging is estimated to cost ∼$2,750 \!\$2,750 for the 38,382 eligible trajectories and ∼$18.6K \!\$18.6K for all 258,134 unique trajectories (Appendix F). In comparison, the current tqs implementation processes a 3,462-trajectory benchmark (median 53 steps) at 43 ms per trajectory with one worker on an AMD EPYC 7413. Extrapolating this measured rate gives about 3.1 CPU-hours for the unique corpus. With four workers, the measured throughput is 87.5 trajectories per second, corresponding to about 49 minutes for a full-corpus pass, or roughly $0.1–$0.2 in on-demand cloud CPU time at representative 4-vCPU rates (Appendix C). These measurements make tqs suitable for corpus-scale ranking, while semantic judging can be reserved for settings where its additional cost is acceptable. The benefit of tqs selection is larger on harder tasks. The Top–Random gap increases from 1.6 points on Easy to 3.4 on Medium and 4.5 on Hard. This pattern suggests that generic successful supervision may suffice for easier problems, while more difficult tasks are more sensitive to the structure and quality of the selected trajectories. Figure 4: Main results on SWE-Bench Verified. Pass@1 on the full test set (top) and Hard subset (bottom). tqs-Top recovers most of the LLM-as-Judge gain at much lower selection cost. 5.2 Direct Validation among Successful Runs Validation statistic Human Sonnet-5 High vs. Low agreement 85.2 (208/244) 81.1 (198/244) Mid vs. Low agreement 74.2 (190/256) 77.0 (197/256) High vs. Mid agreement 76.8 (172/224) 67.4 (151/224) Overall agreement 78.7 (570/724) 75.4 (546/724) Pooled concordance 73.3 (631/861) 76.2 (656/861) Full-order match 45.7 (137/300) 41.3 (124/300) Spearman ρ 0.417 0.402 Table 4: Direct validation of the tqs ordering among successful runs. Agreement (%) uses order-consistent pairwise judgments; pooled concordance uses pointwise rubric scores with ties excluded. We next test whether the tqs ordering agrees with content-based judgments while holding both task and outcome fixed. For each of 300 tasks with at least three resolved runs, we construct high–low, mid–low, and high–mid pairs, yielding 900 comparisons. Five software-engineering students split the human evaluation, with each pair assigned to one student, while Claude Sonnet-5 evaluates the same pairs separately. Both evaluations are blinded to tqs and its components. Each pair is presented in both orders; 176 pairs receive inconsistent verdicts and are treated as ties, leaving 724 order-consistent comparisons. Table 4 shows consistent agreement across the three validation views. On the 724 order-consistent pairs, human and Sonnet-5 judgments agree with the tqs ordering on 78.7% and 75.4% of comparisons, respectively, with the strongest agreement on high–low pairs (85.2% and 81.1%). Pointwise rubric scores yield similar pooled concordance, at 73.3% for humans and 76.2% for Sonnet-5. More stringently, the complete high>>mid>>low ordering matches tqs on 45.7% and 41.3% of tasks, well above the 16.7% chance rate for three-item rankings. Together, these results show that tqs captures a substantial portion of the quality differences recognized by content-based evaluators, while remaining deliberately narrower than a semantic judge. Appendix B reports the full statistics and examples of semantic distinctions not captured by tqs. 5.3 Ablation of tqs Dimensions Selection signal pass@1 Full tqs-Top 27.6 w/o Information focus 24.6 w/o Intent–action alignment 23.0 w/o Policy dynamics 25.8 Random-K 24.8 Base (no SFT) 24.4 Table 5: Dimension ablation of tqs. Each variant removes one structural dimension from data selection and retrains with the same SFT recipe. Random-K and the untuned base model are shown for reference. We next test whether tqs’s downstream benefit comes from one structural dimension or their combination. From the full scorer, we remove information focus, intent–action alignment, or policy dynamics from data selection, reconstruct the Top-K subset, and retrain with the same SFT recipe. Table 5 shows that all three dimensions contribute to the downstream gain. Removing intent–action alignment produces the largest drop, from 27.6 to 23.0 pass@1, falling below both Random-K and the untuned base model. Removing information focus reduces performance to 24.6, while removing policy dynamics gives 25.8. None of the ablated variants matches the full tqs result, indicating that the gain is not carried by a single dimension. The particularly large drop without intent–action alignment suggests that selecting trajectories whose stated reasoning is consistently reflected in subsequent actions is especially important for effective SFT supervision. 5.4 Behavioral Diagnosis on Rollouts Training Data Fine-tuned models Process metric Top-K Top Rand Bottom Mean Steps 34.2 50.66 57.64 63.46 Mean oec 0.603 0.571 0.521 0.484 Mean psn 0.753 0.570 0.548 0.568 Mean scp 0.994 0.777 0.798 0.772 Mean ped 0.826 0.192 0.267 0.164 Mean dpi 0.996 0.931 0.923 0.906 Table 6: Process metrics of the Top-K training data and the fine-tuned models’ own SWE-Bench Verified rollouts. Finally, we ask whether data selection is reflected in the models’ own test-time behavior. We recompute the tqs components on their SWE-Bench Verified rollouts and report the means in Table 6. The Top model uses 50.66 steps on average, compared with 57.64 for Random and 63.46 for Bottom. It also has higher oec and dpi than both models; its psn is higher than Random and close to Bottom. The pattern is not uniform across all components in these rollouts, since Random has slightly higher scp and ped than Top. This suggests that selecting high-tqs supervision leaves a measurable behavioral imprint on the fine-tuned policy rather than affecting only final task accuracy. At the same time, the mismatch on scp and ped indicates that the model does not simply reproduce the component profile of its training data. The clearest rollout differences are shorter interaction, more focused observations, and fewer dirty patterns. 6 Conclusion "Passing" is too coarse a quality signal for coding-agent SFT data. We introduced tqs, a lightweight event-log scorer that evaluates successful trajectories through information focus, intent–action alignment, policy dynamics, and a guardrail for categorical failure patterns. Across our experiments, high-tqs trajectories provide more effective supervision than simpler outcome-, length-, or pattern-based selection, while low-tqs trajectories can even hurt downstream performance. At the same time, tqs approaches the effectiveness of rubric-based GPT-4o judging at substantially lower selection cost, making repeated rescoring and corpus-scale data selection practical. These results suggest that successful trajectories are not equally valuable as supervision, and that trajectory structure offers a useful, scalable signal beyond task outcome alone. Limitations Our experiments use CoderForge-Preview trajectories generated by a single teacher model under the OpenHands scaffold. Although the components rely on general event-log fields, their parsers and thresholds may require recalibration for different action spaces, multilingual traces, or other step granularities. Transfer to GUI or multimodal agents would also require new definitions of scope and intent. Outside the current setting, the hand-set component weights may likewise require recalibration. tqs also scores the structure of an observed process rather than verifying whether a passing patch addresses the underlying root cause. Our SFT experiments focus on a fixed-budget setting with 1,024 selected trajectories per condition. We also restrict training trajectories to at most 32K tokens so that they fit within the 40,960-token context after template overhead. This excludes a long-trajectory tail that may contain useful recovery behavior. We train each condition once because of compute cost, so the reported pass@1 differences do not include multi-seed variance estimates. Our downstream evaluation is limited in model scale. Preliminary experiments with a 7B backbone did not show stable selector differences, so we focus the reported SFT comparison on the 30B model. Larger backbones remain untested. Acknowledgments We thank the anonymous reviewers for their constructive feedback. This work was supported by the Fundamental Research Funds for the Central Universities, Peking University. Xu Sun is the corresponding author of this paper. References Ariyak et al. (2026) A. Ariyak, J. Zhang, J. Wang, S. Zhu, F. Bianchi, S. Srivastava, A. Panda, S. Bharti, C. Xu, J. Heo, X. S. Wu, J. Zou, P. Liang, L. Song, C. Zhang, B. Athiwaratkun, Z. Zhou, and Q. Wu CoderForge-Preview: SOTA open dataset for training efficient coding agents. Note: Dataset card and technical release External Links: Document, Link Cited by: §1, §1, §2, §4. Badertdinov et al. (2025) I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Litvintseva, and B. Yangel SWE-rebench: an automated pipeline for task collection and decontaminated evaluation of software engineering agents. External Links: 2505.20411 Cited by: §4. Bentham et al. (2024) O. Bentham, N. Stringham, and A. Marasović Chain-of-thought unfaithfulness as disguised accuracy. External Links: 2402.14897 Cited by: §2. Chen et al. (2026) G. Chen, C. Sun, C. Fu, Q. Wang, Z. Huang, C. Wei, G. Chen, F. Fang, A. Argha, B. Zhao, X. Xu, Q. Han, H. Alinejad-Rokny, Q. Qu, B. Li, S. Ni, M. Yang, H. Wei, and Y. Li Beyond quantity: trajectory diversity scaling for code agents. External Links: 2602.03219 Cited by: §2. Chen et al. (2024) L. Chen, S. Li, J. Yan, H. Wang, K. Gunaratna, V. Yadav, Z. Tang, V. Srinivasan, T. Zhou, H. Huang, and H. Jin AlpaGasus: training a better alpaca with fewer data. In Proceedings of the 12th International Conference on Learning Representations (ICLR), Cited by: §1, §2. Dong et al. (2025) G. Dong, H. Mao, K. Ma, L. Bao, Y. Chen, Z. Wang, Z. Chen, J. Du, H. Wang, F. Zhang, G. Zhou, Y. Zhu, J. Wen, and Z. Dou Agentic reinforced policy optimization (ARPO). External Links: 2507.19849 Cited by: §2. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations (ICLR), Cited by: §1, §4. Jain et al. (2025) N. Jain, J. Singh, M. Shetty, L. Zheng, K. Sen, and I. Stoica R2E-Gym: procedural environments and hybrid verifiers for scaling open-weights SWE agents. External Links: 2504.07164 Cited by: §4. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan SWE-bench: can language models resolve real-world GitHub issues?. In Proceedings of the 12th International Conference on Learning Representations (ICLR), Cited by: §1. Kamoi et al. (2024) R. Kamoi, Y. Zhang, N. Zhang, J. Han, and R. Zhang When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics 12, p. 1417–1440. Cited by: §1, §3.3. Kendall (1938) M. G. Kendall A new measure of rank correlation. Biometrika 30 (1/2), p. 81–93. Cited by: §3.3. Lanham et al. (2023) T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion, K. Lukošiūtė, K. Nguyen, N. Cheng, N. Joseph, N. Schiefer, O. Rausch, R. Larson, S. McCandlish, S. Kundu, S. Kadavath, S. Yang, T. Henighan, T. Maxwell, T. Telleen-Lawton, T. Hume, Z. Hatfield-Dodds, J. Kaplan, J. Brauner, S. R. Bowman, and E. Perez Measuring faithfulness in chain-of-thought reasoning. External Links: 2307.13702 Cited by: §2. Li et al. (2024a) M. Li, Y. Zhang, S. He, Z. Li, H. Zhao, J. Wang, N. Cheng, and T. Zhou Superfiltering: weak-to-strong data filtering for fast instruction-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2. Li et al. (2024b) M. Li, Y. Zhang, Z. Li, J. Chen, L. Chen, N. Cheng, J. Wang, T. Zhou, and J. Xiao From quantity to quality: boosting LLM performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: §1, §2. Li et al. (2026) Z. Li, H. Wang, Y. Zhao, G. Chen, Y. Li, K. Chen, Y. Cao, G. Ye, H. Chai, M. Wang, and Z. Yin Rethinking the role of entropy in optimizing tool-use behaviors for large language model agents. External Links: 2602.02050, Link Cited by: §2. Liu et al. (2024) W. Liu, W. Zeng, K. He, Y. Jiang, and J. He What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning. In Proceedings of the 12th International Conference on Learning Representations (ICLR), Cited by: §1, §2. Ma et al. (2026) M. Ma, T. Chen, Y. Lin, S. Lu, Q. Zhu, Y. Gong, Z. Huang, P. Cheng, Y. Lu, and J. S. Dong From patches to trajectories: privileged process supervision for software-engineering agents. External Links: 2605.21996, Link Cited by: §2. Ma et al. (2024) Y. Ma, R. Cao, Y. Cao, Y. Zhang, J. Chen, Y. Liu, Y. Liu, B. Li, F. Huang, and Y. Li Lingma SWE-GPT: an open development-process-centric language model for automated software improvement. External Links: 2411.00622 Cited by: §1, §2. OpenAI (2024) OpenAI Introducing SWE-bench Verified. Note: OpenAI Blog External Links: Link Cited by: §1, §1, §4, Table 3. Ou et al. (2025) T. Ou, W. Guo, A. Gandhi, G. Neubig, and X. Yue AgentDiagnose: an open toolkit for diagnosing LLM agent trajectories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP Demos), p. 207–215. External Links: Document, Link Cited by: §2, §4. Pan et al. (2025) J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang Training software engineering agents and verifiers with SWE-Gym. In Proceedings of the 42nd International Conference on Machine Learning (ICML), Cited by: §1, §2. Shannon (1948) C. E. Shannon A mathematical theory of communication. Bell System Technical Journal 27 (3), p. 379–423. Cited by: §3.3. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36 (NeurIPS), Cited by: §3.3. Sun et al. (2025) Q. Sun, K. Cheng, Z. Ding, C. Jin, Y. Wang, F. Xu, Z. Wu, C. Jia, L. Chen, Z. Liu, B. Kao, G. Li, J. He, Y. Qiao, and Z. Wu OS-Genesis: automating GUI agent trajectory construction via reverse task synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5555–5579. External Links: Document, Link Cited by: §2. Wang et al. (2025) X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig OpenHands: an open platform for AI software developers as generalist agents. In Proceedings of the 13th International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1, §2, §4. Wang et al. (2024) Y. Wang, K. He, D. Fu, Z. GongQue, H. Xu, Y. Chen, Z. Wang, Y. Fu, G. Dong, M. Diao, J. Wang, M. Zhang, X. Cai, and W. Xu How do your code LLMs perform? empowering code instruction tuning with really good data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 14027–14043. External Links: Link, Document Cited by: §2. Wei et al. (2025) Y. Wei, O. Duchenne, J. Copet, Q. Carbonneaux, L. Zhang, D. Fried, G. Synnaeve, R. Singh, and S. I. Wang SWE-RL: advancing LLM reasoning via reinforcement learning on open software evolution. External Links: 2502.18449 Cited by: §2. Xia et al. (2024a) C. S. Xia, Y. Deng, S. Dunn, and L. Zhang Agentless: demystifying LLM-based software engineering agents. External Links: 2407.01489 Cited by: §3.3. Xia et al. (2024b) M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen LESS: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning (ICML), Cited by: §1, §2. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, et al. Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1, §4. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37 (NeurIPS), Cited by: §2. Yang et al. (2025b) J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang SWE-smith: scaling data for software engineering agents. External Links: 2504.21798 Cited by: §4. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR), Cited by: §3.3. Yu et al. (2025a) J. Yu, Z. Cheng, X. Wu, and X. Xing Building coding agents via entropy-enhanced multi-turn preference optimization. External Links: 2509.12434 Cited by: §2. Yu et al. (2025b) Z. Yu, H. Zhang, Y. Zhao, H. Huang, M. Yao, K. Ding, and J. Zhao OrcaLoca: an LLM agent framework for software issue localization. In Proceedings of the 42nd International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 267, p. 73416–73436. External Links: Link Cited by: §3.3. Zhou et al. (2023) C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy LIMA: less is more for alignment. In Advances in Neural Information Processing Systems 36 (NeurIPS), External Links: Link Cited by: §1. Appendix A Worked-Example Component Scores Table 7 gives the component breakdown for the two pygments trajectories in Figure 3. The table uses the fixed weights defined in §3.3. Appendix B Direct Validation and Semantic Gaps Within-task protocol. We sample 300 tasks with at least three resolved runs: 181 from SWE-Smith, 85 from SWE-Rebench, and 34 from R2E-Gym. High, middle, and low tqs tiers yield 900 pairs. Each pair is presented in both orders, giving 1,800 judgments in the human study and another 1,800 in the Claude Sonnet-5 study. Five software-engineering students split the human labeling, with one student assigned to each pair. Neither study sees tqs or its components. The rubric covers problem comprehension, solution quality and minimality, exploration efficiency, error recovery, and verification rigor. Of the 900 pairs, 176 reverse under presentation order and are treated as ties. The main table reports the remaining 724 pairs, or 80.4% coverage. Component Weight Run 2 Run 4 oec 0.30 0.7384 0.3120 iac 0.25 0.6667 0.5610 psn 0.10 0.8491 0.5708 tte 0.10 0.5718 0.4046 scp 0.05 0.9904 0.5280 ped 0.10 1.0000 0.0000 dpi 0.10 1.0000 0.6000 TQS 0.7798 0.4178 Table 7: Component scores for the two resolved runs in Figure 3. Run 2 is the high-tqs trajectory and Run 4 is the low-tqs trajectory. Human Claude Sonnet-5 Contrast Agreement 95% CI p Agreement 95% CI p High vs. Low 85.2% (208/244) 80.3–89.1 ×10−301.34\!×\!10^-30 81.1% (198/244) 75.8–85.6 ×10−231.18\!×\!10^-23 Mid vs. Low 74.2% (190/256) 68.5–79.2 ×10−154.27\!×\!10^-15 77.0% (197/256) 71.4–81.7 ×10−181.51\!×\!10^-18 High vs. Mid 76.8% (172/224) 70.8–81.8 ×10−163.43\!×\!10^-16 67.4% (151/224) 61.0–73.2 ×10−72.04\!×\!10^-7 Overall 78.7% (570/724) 75.6–81.6 ×10−575.94\!×\!10^-57 75.4% (546/724) 72.1–78.4 ×10−442.70\!×\!10^-44 Table 8: Agreement with the tqs ordering among order-consistent pairs. Confidence intervals are 95% Wilson intervals. The p-values use two-sided exact binomial tests against 0.5. The rubric scores also give a pointwise comparison that does not require the two presentations to produce the same discrete verdict. Table 9 reports pairwise concordance after excluding equal-score ties. Across the three pair types, the human rubric agrees with tqs on 631/861 pairs and Claude Sonnet-5 agrees on 656/861. The full high>>mid>>low ordering matches tqs for 137/300 tasks under the human rubric and 124/300 tasks under Claude Sonnet-5, compared with 1/61/6 of tasks under a random ordering. Human rubric Claude Sonnet-5 rubric Contrast Conc. 95% CI p Conc. 95% CI p Ties High vs. Low 78.3% (227/290) 73.2–82.6 ×10−236.52\!×\!10^-23 82.1% (238/290) 77.2–86.1 ×10−291.35\!×\!10^-29 10 Mid vs. Low 69.5% (203/292) 64.0–74.5 ×10−112.11\!×\!10^-11 75.7% (221/292) 70.5–80.3 ×10−194.39\!×\!10^-19 8 High vs. Mid 72.0% (201/279) 66.5–77.0 ×10−131.11\!×\!10^-13 70.6% (197/279) 65.0–75.6 ×10−124.39\!×\!10^-12 21 Pooled 73.3% (631/861) 70.2–76.1 ×10−446.59\!×\!10^-44 76.2% (656/861) 73.2–78.9 ×10−551.05\!×\!10^-55 39 Table 9: Concordance between pointwise rubric scores and the tqs tier ordering after excluding equal-score ties. Confidence intervals are 95% Wilson intervals; p-values use two-sided exact binomial tests against 0.5. What semantic judging adds. We inspect disagreements in which Claude Sonnet-5 prefers the lower-tqs trajectory. On python__mypy-10430, the preferred run traces a timing-dependent crash to a flag missing from VAR_FLAGS across serialization and fixes it in one line. The higher-tqs run duplicates lookup logic without resolving the timing failure. On the sqlfluff func_pm_op_change task, the preferred run corrects an indexing error, while the higher-tqs run handles a secondary tuple-subtraction failure and adds bounds checks around the original bug. Verification produces another type of disagreement. On wfondrie__mokapot-65, both runs apply the same one-line fix. The preferred run builds a reproduction, executes the existing test, and adds edge-case tests. The higher-tqs run stops after static inspection without confirming a passing test. These examples show information available to semantic judging that tqs does not encode. Score geometry and selection stability. tqs is a scalar functional, tqs:→[0,1] tqs:T→[0,1], rather than a structural distance between trajectories. Its role is to induce the total preorder used for subset selection: τi⪰τj _i _j whenever tqs(τi)≥tqs(τj) tqs( _i)≥ tqs( _j). Distinct trajectories may therefore be equivalent for selection if they receive the same score. Accordingly, d(τi,τj)=|tqs(τi)−tqs(τj)|d( _i, _j)= | tqs( _i)- tqs( _j) | is a pseudometric on T, and becomes a metric after identifying trajectories with equal scores. This only describes separation in score space; nearby TQS values do not imply structural similarity between trajectories. For subset selection, the relevant quantity is the score margin. Let s(τ)=tqs(τ)s(τ)= tqs(τ) and let s~ s be a perturbed scoring function such that supτ∈|s~(τ)−s(τ)|≤ϵ. _τ | s(τ)-s(τ) |≤ε. Then, for any pair τi,τj _i, _j, s~(τi)−s~(τj)≥s(τi)−s(τj)−2ϵ. s( _i)- s( _j)≥ s( _i)-s( _j)-2ε. Hence, any pair whose original score gap exceeds 2ϵ2ε preserves its ordering. For scores sorted as s(1)≥⋯≥s(N)s_(1)≥·s≥ s_(N), define the Top-K boundary margin as ΔK=s(K)−s(K+1) _K=s_(K)-s_(K+1). If ΔK>2ϵ _K>2ε, the Top-K subset is unchanged. More generally, only trajectories close to the selection boundary can enter or leave the selected set under a bounded perturbation. This also gives a simple robustness condition for the TQS weights. Writing tqs(τ)=∑m=1Mwmϕm(τ),ϕm(τ)∈[0,1], tqs_w(τ)= _m=1^Mw_m _m(τ), _m(τ)∈[0,1], a perturbation δ satisfies |tqs+δ(τ)−tqs(τ)|≤‖δ‖1. | tqs_w+ (τ)- tqs_w(τ) |≤\| \|_1. Therefore, ΔK>2‖δ‖1 _K>2\| \|_1 is a sufficient worst-case condition for preserving the Top-K subset. This bound is conservative, while the empirical Top-K overlap under weight perturbations directly measures the local stability of the selected subset around the chosen TQS configuration. Appendix C Implementation Details Parser. Trajectories are stored as OpenHands-format parquet files with the columns trajectory_id, instance_id, repo, messages, tools, reward, and finish_reason. Our parser converts each row into the step tuple in §3. Tool calls are recovered from the OpenAI-style messages/tool_calls schema, and observations are attached to the corresponding tool step whenever the event log contains a returned tool message. File paths are extracted from tool arguments and normalized before scope-based measures are computed. Scoring cost. The scorer is a deterministic CPU pipeline, and each component is linear in trajectory length. We benchmark the current implementation on an AMD EPYC 7413 using 3,462 mixed-source trajectories with a median length of 53 steps. A single worker processes this batch in 149.2 seconds, or 23.2 trajectories per second (43 ms per trajectory). At this rate, scoring all 258,134 unique trajectories requires about 3.1 CPU-hours. Four workers reach 87.5 trajectories per second, which corresponds to about 49 minutes for a full-corpus pass. Profiling attributes 58% of the single-worker runtime to regular-expression error detection and 13% to the character-level entropy calculation used by oec. Loss masking. For SFT, ms-swift computes loss on assistant responses, including tool calls, while masking user messages and tool responses. Under the Qwen3-Coder agent template, tool calls are rendered in XML. For example: <tool_call> <function=execute_bash> <parameter=command> pytest tests/ </parameter> </function> </tool_call> This setup trains the model to imitate both natural-language reasoning and tool invocation, while excluding user messages and tool observations from the supervised loss. As a result, the loss focuses on the agent’s reasoning and actions, while tool outputs serve only as conditioning context. Error flags. The binary error flag εt _t is set from tool observations using a small set of string and regular-expression patterns, including common execution failures, missing-file messages, traceback markers, failed test invocations, and tool-level error statuses. The flag is intentionally conservative: it is used to detect repeated failure patterns and post-error behavior, not to infer semantic correctness of the final patch. C.1 Subset Sampler We construct fixed-budget training subsets from the eligible pool described in §3.3. Each subset contains 1,024 trajectories. Top-K and Bottom-K are selected by descending and ascending tqs, respectively, while Random-K is sampled with a fixed seed. To keep the comparison focused on trajectory quality, the sampler controls three sources of variation: task identity, repository coverage, and source provenance. For Top-K and Bottom-K, we first sort eligible trajectories by tqs in the corresponding direction. The sampler then fills the subset greedily. In the first pass, it selects at most one trajectory per task. In the second pass, it prioritizes trajectories from repositories not yet represented in the subset. In the final pass, it fills any remaining slots in score order, subject to a hard per-task cap of 30 trajectories. This preserves tqs ranking as the primary selection criterion while avoiding subsets dominated by a small number of tasks or repositories. Because CoderForge-Preview combines trajectories from multiple data sources, we also stratify each subset by source. Target source proportions are computed from the eligible pool and applied to Top-K, Bottom-K, and Random-K. Random-K uses the same eligibility filters and source proportions, but samples uniformly within each source stratum rather than sorting by tqs. All subset construction is performed after deduplication by trajectory_id. Ties at selection boundaries are broken deterministically by trajectory_id. Appendix D Score and Component Distributions Figure 5: tqs distribution on the 38,382-trajectory eligible pool. The distribution peaks around 0.58–0.60. The Top-1024 threshold is 0.7061, and the Bottom-1024 threshold is 0.4696, leaving a gap of 0.2365 between the two selected subsets. Component Mean Std. Undefined oec 0.429 0.118 0.0% psn 0.646 0.153 0.0% iac 0.570 0.098 0.0% tte 0.550 0.038 0.0% scp 0.978 0.060 0.0% ped 0.580 0.232 0.7% dpi 0.941 0.127 0.0% Table 10: Marginal statistics of the seven retained tqs components on the eligible pool. scp and dpi are concentrated near 1, while oec, psn, iac, and ped provide broader variation. Figure 5 shows the distribution of the final tqs over the 38,382 eligible trajectories. The scores span a broad range rather than collapsing into a narrow band, providing sufficient resolution for fixed-budget ranking. The Top-1,024 cutoff is 0.7061, whereas the Bottom-1,024 cutoff is 0.4696, leaving a score gap of 0.2365 between the two selected regions. Random-K, by construction, remains centered around the eligible-pool distribution rather than either tail. This separation is important operationally: tqs does not merely reorder trajectories with nearly identical scores, but identifies distinct high- and low-scoring regions from which fixed-size subsets can be constructed. The separation alone does not establish downstream quality, but provides the score resolution required for the selection experiments in §5. (a) oec (b) psn (c) iac (d) tte (e) scp (f) ped (g) dpi Figure 6: Marginal distributions of the seven retained tqs components on the eligible pool. All components are normalized to [0,1][0,1] before aggregation. Figure 7: Spearman correlation matrix among retained tqs components on the eligible pool. The largest off-diagonal correlation is 0.320 between tte and ped, indicating no severe redundancy among the retained components. Table 10 and Figure 6 characterize the seven retained components individually. Their marginal distributions differ substantially, consistent with their different roles in structured convergence. ped has the largest dispersion (σ=0.232σ=0.232), followed by psn (σ=0.153σ=0.153) and oec (σ=0.118σ=0.118), providing substantial variation across the eligible pool. iac also varies meaningfully around its mean of 0.570, whereas tte is more tightly concentrated (σ=0.038σ=0.038). In contrast, scp and dpi have high means of 0.978 and 0.941, respectively, because most eligible trajectories already avoid their most severe failure patterns. Their value therefore lies primarily in identifying the smaller tail of runs with weak commitment or dirty interaction patterns rather than providing continuous discrimination across the entire pool. The components also differ in coverage. Six are defined for every eligible trajectory, while ped is undefined for only 0.7% of runs because its error-recovery signal requires an applicable error transition. This small amount of missingness is handled by the renormalized aggregation in Eq. 1, rather than assigning an artificial value to trajectories for which the component cannot be evaluated. Overall, the heterogeneous marginals show that the final score combines both broadly varying signals such as oec, psn, and ped and more selective guardrail-like signals such as scp and dpi. Figure 7 further examines whether these signals are redundant. Pairwise Spearman correlations remain modest across the retained components; the largest off-diagonal correlation is only 0.320, between tte and ped. Thus, no component is well approximated by another through a simple monotonic relationship. This is consistent with the intended design of tqs: information focus, intent–action alignment, policy dynamics, and categorical failure patterns contribute overlapping but distinct structural evidence. The correlation analysis by itself does not show that every component is necessary, but it rules out severe marginal redundancy; the downstream dimension ablations in §5.3 provide the complementary test of whether these signals matter for data selection. Appendix E Failed Candidate Measures We also considered several candidate measures that were motivated by the same structured-convergence view but failed the admission protocol in §3.4. Their definitions are included here to clarify the boundary of the final score. HCR (Hypothesis–Commit Ratio). HCR characterizes how an agent allocates steps between exploration and execution. We define exploration as ℛ=think,read_file,searchR=\ think, read\_file, search\ and execution as ℰ=write_file,execute_bashE=\ write\_file, execute\_bash\, and compute their ratio within the early, middle, and late thirds of the trajectory: Rg=#t∈Gg:at∈ℛ#t∈Gg:at∈ℰ.R_g= \#\t∈ G_g:a_t \\#\t∈ G_g:a_t \. HCR assigns score 1 when Rearly≥Rmid≥RlateR_early≥ R_mid≥ R_late, favoring an explore-then-commit pattern, and 0 otherwise. Although it passes the numerical thresholds (concordance 0.527, |ρ|=0.014|ρ|=0.014), including HCR caused 99.9% of the Top-1K subset to share the same binary pattern, so we exclude it under the behavioral-diversity criterion. BUS (Backtracking Utility Score). BUS distinguishes productive redirection from unproductive wandering. A backtracking event is detected when the agent’s current target-file set has no overlap with that of the preceding k steps. The event is considered useful if, within the following k steps, the search scope narrows or the agent makes a substantive edit: sBUS=# useful backtracking events# backtracking events.s_BUS= \# useful backtracking events\# backtracking events. Higher BUS therefore indicates that changes in direction more often lead to focused progress. Its concordance is only 0.202, so we exclude it. TGCR (Tool-Graph Cycle Ratio). TGCR captures structural redundancy in file navigation. We construct a directed graph whose nodes are files and whose edges represent successive file-access transitions, and use DFS coloring to identify back edges: sTGCR=# back edges# total edges.s_TGCR= \# back edges\# total edges. A larger value indicates more cyclic revisitation, whereas a trajectory closer to a DAG reflects a cleaner access path. TGCR obtains concordance 0.479, close to chance, and is therefore excluded. DCS-Jaccard (Decision Consistency Score). DCS measures local consistency between reasoning, action, and subsequent feedback. For each step, it combines intent–action alignment with the extent to which the resulting observation informs the next intent: sDCS s_DCS =12[sim(intentt,actiont) = 12 [sim(intent_t,action_t) +sim(obst,intentt+1)]. +sim(obs_t,intent_t+1) ]. DCS-Jaccard instantiates sim(⋅,⋅)sim(·,·) with Jaccard similarity and averages the resulting scores over valid steps. Although intended to capture coherent local transitions, it has |ρ|=0.729|ρ|=0.729 with step count and is therefore rejected as strongly length-confounded. DCS-file (file-alignment variant). DCS-file replaces lexical Jaccard overlap with file-level alignment when evaluating the same intent–action and observation–intent transitions. It therefore asks whether the files implied by the agent’s reasoning agree with those actually acted on and whether observed files are reflected in the next plan. The resulting score obtains concordance 0.238, substantially below chance in the intended direction, and is excluded. Appendix F LLM-Judge Baseline Rubric The B5 baseline uses GPT-4o as a trajectory quality judge under an AgentDiagnose-style rubric. The judge sees the issue description, the trajectory’s reasoning and tool-call sequence, and the four rubric dimensions below. It does not see the reference patch or held-out test outcome. Each dimension is scored on a 1–5 Likert scale, and the trajectory score is the mean of the four dimensions. The top 1,024 trajectories by mean score, after the same eligibility filters and subset constraints as Top-K, form the B5 training subset. Planning. Does the agent decompose the issue into a coherent plan before acting, and does the plan track the issue’s structure rather than the first failure mode that comes to mind? Grounding. Are claims and edits grounded in observations that the trajectory has actually surfaced, or does the agent assert facts about the codebase without prior evidence? Reflection. When an action fails, does the agent diagnose the failure before retrying, or does it loop on minor variations of the same action? Efficiency. Does each step advance the trajectory toward a fix, or does the trajectory contain redundant exploration, repeated reads of already-viewed files, or stuck-process interrupts? Cost estimate. On the 38,382-trajectory eligible pool, with approximately 28K input tokens and 200 output tokens per call, GPT-4o pricing of $2.50/M input tokens and $10/M output tokens gives a per-pool cost of roughly ∼$2,750 \!\$2,750. Applying the same judge to all 258,134 unique trajectories would cost roughly ∼$18.6K \!\$18.6K. Appendix G Length-Controlled Pair-AUC Table 11 gives a stricter length-confound check than the |ρ|<0.25|ρ|<0.25 criterion in Table 1. We restrict trajectories to the modal step-count band, total_steps∈[25,40]total\_steps∈[25,40], and ask whether retained components still separate Top from Bottom. The composite tqs remains perfectly separating in this band. Among the largest components, oec, iac, and ped remain above 0.96 pair-AUC, while psn drops to 0.839. This drop is diagnostic: scope narrowing is partly entangled with trajectory length, but the composite score is not driven by this single component. Score / Component Pair-AUC tqs composite 1.000 oec 0.996 iac 0.963 ped 0.961 psn 0.839 Table 11: Length-controlled pair-AUC in the modal step-count band total_steps∈[25,40]total\_steps∈[25,40]. Source Passing Eligible Retention SWE-Smith 89,501 23,569 26.3% SWE-Rebench 44,739 7,821 17.5% R2E-Gym 20,904 6,992 33.4% All 155,144 38,382 24.7% Table 12: Retention after applying the fixed-budget SFT eligibility filters to passing trajectories. All passing trajectories satisfy the step-count and coverage filters; the main reduction comes from the tokenized-length cap and subsequent deduplication by trajectory_id. Appendix H Tokenized-Length Cutoff The 32,768-token cap is a packing-budget choice for SFT. It fits inside the 40,960-token context window while leaving room for chat-template and assistant-tag tokens. Of the 155,144 reward=1=1 trajectories, 38,382 survive the eligibility filters and deduplication, giving a retention rate of 24.7%. The discarded tail is dominated by long trajectories, especially from SWE-Smith. The cap should be interpreted as part of the fixed-budget SFT setting. Longer trajectories may contain useful recovery behavior, but they require longer-context or sequence-parallel training to imitate faithfully. We leave this setting to future work. Appendix I Sensitivity to Component Weights The main experiments use the fixed, hand-set weights defined in §3.3. They allocate 0.40 to information focus, 0.25 to intent–action alignment, 0.25 to policy dynamics, and 0.10 to the dpi guardrail. No downstream pass@1 result was used to tune them. We test local selection stability by perturbing the two largest individual weights, woecw_ oec and wiacw_ iac, and the guardrail weight wdpiw_ dpi. For each setting, we rebuild the Top-K subset and report its overlap with the main selection. This experiment measures how much the selected set changes near the chosen weights. Weight setting Top-1024 overlap Main weights 100.0% woec=0.25w_ oec=0.25 93.1% woec=0.35w_ oec=0.35 93.6% wiac=0.20w_ iac=0.20 93.1% wiac=0.30w_ iac=0.30 94.1% wdpi=0.05w_ dpi=0.05 98.3% wdpi=0.15w_ dpi=0.15 99.6% Table 13: Weight sensitivity of tqs selection on the eligible pool. We perturb one weight at a time while keeping the other weights fixed, then measure overlap between the resulting Top-1024 set and the main Top-1024 set. Figure 8: Weight sensitivity of Top-1024 selection under single-weight perturbations. Figure 8 visualizes the same result. OEC and IAC perturbations retain 93–94% overlap with the main Top-1024 set, while DPI perturbations retain over 98% overlap. These results indicate that the selected subset is locally stable to moderate changes in the hand-set weights. This analysis does not establish that the chosen weights are globally optimal, but it shows that small perturbations around them leave most of the selected training set unchanged.