Paper deep dive
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Kefeng Duan, Dewu Zheng, Yanlin Wang, Xiwen Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jiachi Chen, Mingwei Liu, Zibin Zheng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/14/2026, 6:17:18 AM
Summary
The paper introduces PTA-IRT, a framework for efficient Software Engineering (SWE) agent benchmarking that utilizes Privileged Trajectory-Aware Item Response Theory. It addresses the high cost of evaluating agents on full benchmarks by selecting a representative calibration subset. Unlike prior methods that rely solely on pass/fail outcomes, PTA-IRT incorporates historical execution trajectories (process-level evidence) as privileged information to improve item selection and ability estimation. Experiments on four SWE benchmarks demonstrate that PTA-IRT outperforms existing IRT baselines in score and ranking recovery under low calibration budgets.
Entities (13)
Relation Signals (24)
PTA-IRT → evaluatedon → SWE-bench Verified
confidence 95% · we conduct experiments on four SWE benchmarks... SWE-bench Verified
PTA-IRT → evaluatedon → SWE-Bench Pro
confidence 95% · we conduct experiments on four SWE benchmarks... SWE-bench Pro
PTA-IRT → evaluatedon → SWE-bench Full
confidence 95% · we conduct experiments on four SWE benchmarks... SWE-bench Full
PTA-IRT → evaluatedon → SWE-bench Lite
confidence 95% · we conduct experiments on four SWE benchmarks... SWE-bench Lite
PTA-IRT → uses → Learning Using Privileged Information
confidence 95% · transfers measurement knowledge to an ability estimator through Learning Using Privileged Information (LUPI)
PTA-IRT → uses → Item Response Theory
confidence 95% · PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework
Zhejiang University → affiliationof → Jiachi Chen
confidence 90% · Jiachi Chen... 3College of Computer Science and Technology, Zhejiang University
Huawei Cloud Computing Technologies Co., Ltd. → affiliationof → Ensheng Shi
confidence 90% · Ensheng Shi... 2Huawei Cloud Computing Technologies Co., Ltd.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.01603v1
- Canonical: https://arxiv.org/abs/2609.01603v1
Trouble viewing inline? Open PDF directly →
Full Text
51,782 characters extracted from source content.
Expand or collapse full text
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation Kefeng Duan Dewu Zheng Yanlin Wang Xiwen Wang Ensheng Shi Xilin Liu Yuchi Ma Jiachi Chen Mingwei Liu Zibin Zheng Abstract Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT. 1School of Software Engineering, Sun Yat-sen University, Zhuhai, China 2Huawei Cloud Computing Technologies Co., Ltd., China 3College of Computer Science and Technology, Zhejiang University, Hangzhou, China duankf, zhengdw5, wangxw86@mail2.sysu.edu.cn, wangylin36, liumw26, zhzibin@mail.sysu.edu.cn shiensheng, liuxilin3, mayuchi1@huawei.com, chenjiachi@zju.edu.cn 1 Introduction As Large Language Models (LLMs) are increasingly deployed as software engineering agents, more challenging benchmarks emerge to evaluate their ability to solve real-world software engineering tasks. However, these agentic benchmarks make evaluation substantially more expensive than conventional single-turn settings. Repository-level benchmarks require agents to reason over long horizons, explore codebases, invoke tools, edit source files, and execute tests. These multi-step interactions result in substantial inference costs and long evaluation cycles. For example, SWE-bench contains more than 2,000 programming tasks; a single full-benchmark SWE-agent run has an estimated upper-bound cost exceeding $8,000 under a $4-per-task limit, while the average cost on successfully resolved instances is $1.59 for SWE-agent with GPT-4 Turbo [51, 16]. This motivates the central question of our work: can we evaluate a new agent on only a small budgeted subset of tasks while reliably recovering its performance and relative ranking on the full benchmark? To address this question, prior work on efficient benchmarking introduces Item Response Theory (IRT) to select a small but representative subset of benchmark instances and use limited observations to estimate full-benchmark performance [36, 28, 17, 59]. IRT-based methods fit historical response matrices to estimate item measurement characteristics and to infer a model’s latent ability from a limited set of observed responses. IRT therefore provides a promising framework for budgeted SWE agent evaluation. However, prior IRT methods for efficient benchmarking typically represent each task solely through its final pass/fail response, while a SWE agent’s outcome is produced through a multi-step trajectory of reasoning, context exploration, tool use, code editing, and verification. Reducing a trajectory to a binary outcome discards crucial process-level measurement signals about how an agent reaches success or failure, including what context it explores, which actions or edits it attempts, and where its problem-solving process breaks down. To address this limitation, we propose PTA-IRT (Privileged Trajectory-Aware Item Response Theory), which incorporates historical agent trajectories as privileged information into IRT-based data selection and evaluation. PTA-IRT first converts trajectories into structured semantic summaries and uses them to learn trajectory-aware four-parameter logistic (4PL) item measurement characteristics. It then selects a difficulty-stratified calibration subset using trajectory-aware Fisher information and transfers measurement knowledge to an ability estimator through Learning Using Privileged Information (LUPI). Given a new agent, PTA-IRT executes only the selected calibration tasks and uses limited calibration observations to estimate its full-benchmark performance and ranking. Dataset curation Budgeted evaluation (subset→ ) Dimension Lite Verified Classical IRT Deep/PSN-IRT AutoJudger PTA-IRT Target domain SWE SWE General General Multimodal SWE Observation signal — — Pass/Fail Pass/Fail Pass/Fail Pass/Fail++Agent Traj. Modeling method Heuristic rules Human judgment Parametric IRT Neural IRT IRT ++ LLM selection Traj.-aware IRT LLM involvement — — — — Selects tasks Summarizes traj. Human involvement — Heavy — — — — Table 1: Comparison of efficient benchmarking for SWE evaluation. Dataset curation: SWE-bench Lite and Verified are fixed smaller subsets filtered from SWE-bench (Lite: rule-based; Verified: human annotation). Budgeted evaluation recovers full-benchmark scores from a calibration subset. “—” denotes not applicable / not involved. To validate the effectiveness of PTA-IRT, we conduct experiments on four SWE benchmarks and compare it with prior IRT baselines. Our results show that: • Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery. • Privileged trajectory information is the source of these gains: outcome-only modeling or corrupted trajectory summaries weaken recovery. • Structured trajectory summaries carry process signals beyond issue text and submitted patches, making budgeted evaluation more informative. Our contributions are as follows: • We formulate efficient SWE benchmarking as a budgeted subset-to-full evaluation problem under a realistic protocol in which a new agent is executed only on the calibration subset, while historical trajectories are used offline for training. • We introduce PTA-IRT, which incorporates historical agent trajectories as privileged information into IRT-based item selection and full-benchmark performance estimation. • Extensive experiments on four SWE benchmarks show that PTA-IRT improves budgeted score and ranking recovery over prior IRT baselines. 2 Background Figure 1: Limitations of prior work and an overview of PTA-IRT. 2.1 Related Work Large language models and LLM-based agents have reshaped a wide range of software engineering tasks [48, 22, 41]. On code generation, they now cover function-level synthesis as well as repository-level completion, secure generation, and token-efficient decoding [1, 11, 55, 56, 20, 24, 54, 46, 47]. Code search and retrieval-augmented generation further connect natural-language intent to relevant code context [4, 42, 5, 12, 45]. Code summarization and comment generation recover intent from source context [43, 6], while retrieval-augmented commit message generation links edits to natural-language descriptions [32]. Issue resolution has become a central testbed for agentic software engineering, with systems that explore repositories, apply patches, and verify fixes against tests [14, 51, 34, 7, 8, 13, 52, 58]. Related directions include debugging, repository-level reasoning, code translation, and code review [35, 44, 40, 57]. These advances have driven more demanding repository-level and agentic benchmarks, which require long-horizon reasoning, tool use, editing, and test execution [14, 50, 2, 8, 7, 46, 47, 55, 40, 56, 57]. Such evaluations are substantially more expensive than conventional single-turn settings [1, 11]. Recent research therefore examines the efficiency and reliability of LLM benchmarks, with a growing interest in reducing evaluation cost through a smaller set of test instances [29, 39, 53]. IRT provides a data-driven, item-level framework for this purpose [26]. By modeling agents’ responses across instances, IRT estimates item properties, such as difficulty and discriminability, together with the latent abilities of evaluated models. In natural language processing, IRT was initially used to diagnose fundamental properties of datasets and test items, including item difficulty and discriminability [18]. More recently, its principles have been applied to efficient evaluation. Deep-IRT and PSN-IRT extend conventional IRT with neural architectures and more flexible response functions to model complex model–item interactions [36, 59]. In a different direction, AutoJudger proposes an agent-driven framework for efficient benchmarking of multimodal large language models [3]. However, IRT remains underexplored for SWE agent evaluation specifically, where agent–instance response data are sparse and trajectories are long. Instead, current SWE benchmarks still rely on rule-based filtering or human verification to reduce evaluation cost, as in the curated SWE-bench Lite and Verified subsets [14]. Table 1 contrasts these approaches along target domain, observation signal, modeling method, and human/LLM involvement. 2.2 Preliminary: Item Response Theory IRT models the response behavior between respondents and items through respondents’ latent abilities and items’ measurement properties [9, 30]. In SWE agent evaluation, agents correspond to respondents, while benchmark instances correspond to items. IRT uses an Item Characteristic Curve (ICC) to model the conditional probability of a correct response. Under the one-parameter logistic model, the probability that agent i successfully solves instance j is defined as P(Xij=1∣θi)=11+exp[−(θi−bj)],P(X_ij=1 _i)= 11+ [-( _i-b_j)], (1) where θi _i denotes the latent ability of agent i and bjb_j denotes the difficulty parameter of instance j. As IRT research advances, subsequent work incorporates additional item parameters, explores more expressive ICC functions, and adopts deep-learning approaches to improve its modeling capacity in complex evaluation settings [36, 59]. 2.3 Preliminary: Learning Using Privileged Information LUPI leverages additional information that is available during training but unavailable at test time [38, 31]. Formally, each training example is represented as (,∗,y)(x,x^*,y), where x denotes the standard input, ∗x^* denotes privileged information, and y is the supervision signal. The goal of LUPI is to use ∗x^* during training to improve a predictor that relies only on x at inference time. A neural realization of LUPI uses a teacher–student paradigm, in which the teacher accesses privileged information and transfers its knowledge to a student through distillation. The student then relies only on the standard input, without requiring privileged information at inference time [37, 25, 19]. In our setting, agent trajectories serve as privileged information, as they provide process-level evidence beyond binary responses. They are available for historical agent–item interactions, but are not required when estimating a new agent’s performance on the full benchmark from its responses on a calibration subset. 3 PTA-IRT 3.1 Overview Evaluating a SWE agent on a full benchmark is costly. We study how to construct a small calibration subset from a full benchmark under a limited evaluation budget, and how to accurately estimate an agent’s full-benchmark performance using only its outcomes on this subset. As illustrated in Figure 1, PTA-IRT consists of three consecutive stages: • Stage 1: Construct Agent Trajectory Representations. We compress heterogeneous and lengthy execution logs into comparable structured process summaries, preserving process-level evidence that binary pass/fail labels cannot capture. • Stage 2: Select Informative Calibration Items. We fit a trajectory-aware 4PL IRT model, score tasks by Fisher information weighted by trajectory reliability, and select a difficulty-stratified calibration subset that is both informative and representative. • Stage 3: Estimate Agent Ability via LUPI. On the same trajectory-aware 4PL backbone, a teacher–student objective transfers privileged process knowledge to a deployable student. At test time, we fit only a scalar ability on the calibration subset and extrapolate to the full benchmark. 3.2 Construct Agent Trajectory Representations Raw agent trajectories contain lengthy commands, tool outputs, and intermediate messages that vary across agents and frameworks. We therefore build a trajectory parser to extract action steps from heterogeneous logs, and design a prompt protocol that compresses each historical trajectory into a structured process summary. The summary has four fields: Task Goal, Context Explored, Edits Executed, and Path Overview. The summarization model receives the task description, extracted action steps, the benchmark reference patch, and the agent’s submitted patch. The reference patch serves to align task-relevant files, APIs, and change scope across heterogeneous trajectories, and is not treated as an outcome label. The protocol further instructs the summarizer to describe observable actions and edits, without inferring correctness or unobserved reasoning. This compression preserves process-level evidence absent from binary pass/fail labels. We encode process summaries as Z=ijZ=\z_ij\ and use them together with the response matrix Y=yijY=\y_ij\ during offline training. Under our protocol, a new agent is evaluated on the calibration subset to estimate full-benchmark performance, so process summaries are available for historical interactions during training but unavailable for unevaluated tasks at test time. 3.3 Select Informative Calibration Items This stage constructs a calibration subset S that reduces uncertainty about agent ability while remaining representative of full-benchmark difficulty. Purely maximizing item information tends to select overly hard, mutually similar tasks and harms score recovery. We therefore select items by a trajectory-aware information score under difficulty stratification. Trajectory-aware scorer. We fit a 4PL IRT model on historical agents A only. Each agent i has latent ability θi _i, and each task j has global parameters (aj,bj,cj,dj)(a_j,b_j,c_j,d_j) for discriminability, difficulty, guessing, and feasibility ceiling. Under these parameters, the response probability is pij=cj+(dj−cj)σ(aj(θi−bj)),p_ij=c_j+(d_j-c_j)\,σ\! (a_j( _i-b_j) ), (2) with 0≤cj<dj≤10≤ c_j<d_j≤ 1. Architecturally, an agent encoder maps a one-hot agent identifier to θi _i, an item encoder maps a one-hot item identifier to (aj,bj,cj,dj)(a_j,b_j,c_j,d_j), and a summary encoder maps each privileged summary ijz_ij to bounded residuals (Δaij,Δbij)=αtanh(g(ij))( a_ij, b_ij)=α (g(z_ij)). Each encoder is a two-layer multilayer perceptron (MLP). On cells with a usable summary (mij=1m_ij=1), we form effective parameters aij a_ij =aj+mijΔaij,bij=bj+mijΔbij, =a_j+m_ij a_ij, b_ij=b_j+m_ij b_ij, (3) pij p_ij =cj+(dj−cj)σ(aij(θi−bij)). =c_j+(d_j-c_j)\,σ\! (a_ij( _i-b_ij) ). Here (aj,bj,cj,dj)(a_j,b_j,c_j,d_j) remain shared item parameters for the agent population. A process summary provides agent–item interaction evidence, which we inject as residuals on discriminability and difficulty, i.e., the parameters that govern the ability-linked transition of the item characteristic curve. In contrast, the asymptotes cjc_j and djd_j encode structural bounds of the task and evaluation setting, namely chance success and feasibility limits. These bounds are properties of the item and harness rather than of a single solving path, so we do not make them trajectory-dependent. Information-aware stratified selection. We score each task by how much its outcomes inform ability estimation, then select under difficulty-stratified budget allocation. The contribution of interaction (i,j)(i,j) is the 4PL Fisher information [27] Iij(θi)=aij2(pij−cj)2(dj−pij)2(dj−cj)2pij(1−pij).I_ij( _i)= a_ij^2 (p_ij-c_j )^2 (d_j-p_ij )^2(d_j-c_j)^2p_ij(1-p_ij). (4) Aggregating over A with offline quality weights ωij _ij yields Fisher(j)=∑i∈ωijIij(θi)∑i∈ωij,ESS(j)=∑i∈mijωij,Info(j)=Fisher(j)⋅log(1+ESS(j)). array[]lclFisher(j)&=& _i _ijI_ij( _i) _i _ij,\\[6.0pt] ESS(j)&=& _i m_ij\, _ij,\\[6.0pt] Info(j)&=&Fisher(j)· (1+ESS(j) ). array (5) Here mij∈0,1m_ij∈\0,1\ marks a usable process summary, and ωij∈[0,1] _ij∈[0,1] down-weights incomplete or malformed trajectory summaries. ESS(j)ESS(j) is the effective sample size of privileged evidence on item j. Multiplying Fisher(j)Fisher(j) by log(1+ESS(j)) (1+ESS(j)) therefore reduces the score of items whose privileged evidence is of low quality. Let rj=1||∑i∈yijr_j= 1|A| _i y_ij be the pass rate of task j on historical agents only. We partition tasks into difficulty bins by rjr_j, allocate the budget k across bins in proportion to bin size, and within each bin select the tasks with the largest Info(j)Info(j). The resulting subset S therefore favors high-information items while covering easy-to-hard regimes, which is essential for recovering full-benchmark scores from a small calibration budget. 3.4 Estimate Agent Ability via LUPI A new agent is evaluated only on the calibration subset S, and process summaries arise only after the agent is actually executed. Full-benchmark summaries are therefore unavailable at scoring time, which matches the LUPI setting. We design a teacher–student estimator on the same trajectory-aware 4PL backbone: privileged summaries supervise the teacher offline, while the student uses only identifiers and shared item parameters, estimates ability from S, and extrapolates to all tasks. Teacher–student training. The teacher and student share the agent, item, and summary encoders. The student predicts with global item parameters (aj,bj,cj,dj)(a_j,b_j,c_j,d_j). The teacher predicts with trajectory-adjusted parameters (aij,bij)(a_ij,b_ij), so the training objective depends on process-summary information. We minimize ℒ =BCE(pijS,yij) =BCE(p^S_ij,y_ij) (6) +λtmijωijBCE(pijT,yij) +\, _t\,m_ij _ijBCE(p^T_ij,y_ij) +λKLmijωijKL(pTij∥pSij) +\, _KL\,m_ij _ijKL\! (p^T_ij\,\|\,p^S_ij ) +λΔmij(‖Δaij‖2+‖Δbij‖2), +\, _ \,m_ij (\| a_ij\|^2+\| b_ij\|^2 ), where pijSp^S_ij and pijTp^T_ij are the student and teacher pass probabilities. The teacher and KL terms distill summary-conditioned measurements into the student. The residual penalty keeps (Δa,Δb)( a, b) small. Test-time estimation. We freeze all network weights and fit a scalar ability θ∗ _* on S by L-BFGS [21] from the observed outcomes, optionally with teacher distillation on S. With θ∗ _* and the shared item parameters (aj,bj,cj,dj)(a_j,b_j,c_j,d_j), the student predicts pass probabilities on every benchmark task and aggregates them into overall performance and ranking. 4 Experimental Setup 4.1 Datasets As shown in Table 2, we evaluate on four SWE-bench versions (Lite, Verified, Full, and Pro). Lite and Verified are curated subsets of Full [14]: Lite uses rule-based filtering toward more self-contained functional fixes, while Verified uses human annotation to remove underspecified or problematic instances. SWE-bench Pro [2] further extends SWE-bench to long-horizon, multi-file enterprise-style tasks across broader repositories and languages. Because Lite and Verified are the community’s default reporting targets, they attract substantially more public agent submissions with parseable trajectories than Full or Pro. We therefore obtain denser model coverage on Lite and Verified, while Full and Pro emphasize larger task collections under sparser agent pools. We use each official release and retain all models with complete, successfully parsed trajectories. Benchmark # Tasks # Evaluated Models SWE-bench Lite 300 35 SWE-bench Verified 500 70 SWE-bench Full 2,294 14 SWE-bench Pro 730 14 Table 2: Overview of the benchmarks and evaluated models. Lite Verified Full Pro Avg Method MAE↓ τ↑ ρ↑ MAE↓ τ↑ ρ↑ MAE↓ τ↑ ρ↑ MAE↓ τ↑ ρ↑ MAE↓ τ↑ ρ↑ MLE .164±.022 .718 .894 .235±.045 .648 .831 .120±.056 .319 .424 .165±.039 .429 .675 .171±.041 .528 .706 MCMC .217±.048 .647 .805 .130±.041 .769 .923 .073±.050 .780 .921 .084±.041 .846 .952 .126±.045 .761 .900 VI .239±.027 .731 .898 .142±.012 .855 .967 .281±.140 .934 .982 .226±.028 .868 .952 .222±.052 .847 .950 VIBO .206±.031 .691 .866 .291±.036 .583 .737 .157±.069 .890 .960 .211±.057 .692 .864 .216±.048 .714 .857 Deep-IRT .305±.145 .682 .863 .191±.049 .724 .909 .362±.120 .934 .982 .289±.144 .692 .868 .287±.114 .758 .905 PSN-IRT .114±.035 .650 .817 .216±.045 .786 .941 .198±.133 .758 .921 .073±.013 .692 .886 .150±.057 .722 .891 AutoJudger .128±.014 .620 .826 .200±.011 .673 .871 .151±.051 .626 .736 .120±.047 .626 .807 .150±.031 .637 .810 PTA-IRT .045±.004 .836 .950 .043±.015 .872 .976 .048±.025 .956 .991 .029±.017 .890 .974 .041±.015 .888 .973 Table 3: Comparison at a 10%10\% calibration budget across four SWE benchmarks. Lower MAE and higher Kendall’s τ / Spearman’s ρ are better. Best values in each column are bolded (all ties are bolded). Values are rounded to three decimals with leading zeros omitted. Figure 2: Budget sensitivity on SWE-bench Lite under 55–25%25\% calibration. Each panel shows one metric (MAE, Kendall’s τ, and Spearman’s ρ). 4.2 Baselines We compare PTA-IRT against a representative set of prior IRT baselines spanning classical IRT, neural IRT, and agent-driven selection. Classical baselines include Maximum Likelihood Estimation (MLE), Markov Chain Monte Carlo (MCMC) [10], Variational Inference (VI) [15], and VIBO [49]. Advanced methods include Deep-IRT [36] and PSN-IRT [59]. We additionally include AutoJudger [3], which combines IRT-based difficulty estimation with an LLM agent that adaptively selects calibration items. 4.3 Metrics We adopt three metrics: Mean Absolute Error (MAE), Kendall’s τ, and Spearman’s ρ. • MAE: the average deviation between predicted and ground-truth full-benchmark performance, MAE=1N∑i=1N|y^i−yi|MAE= 1N _i=1^N | y_i-y_i |, where y^i y_i and yiy_i are the predicted and ground-truth scores of model i, and N is the number of evaluated models. • Kendall’s τ: pairwise ordinal agreement between the predicted and ground-truth rankings, τ=C−D(N2)τ= C-D N2, where C and D are the numbers of concordant and discordant model pairs. • Spearman’s ρ: the global monotonic correlation between the predicted and ground-truth rankings, ρ=corr(rank(^),rank())ρ=corr (rank( y),rank(y) ), where y and y are the vectors of predicted and ground-truth performance. 4.4 Implementation Details To ensure reliability, we use four-fold cross-validation: in each fold, 75% of the models are used for training and the remaining 25% for testing. Process summaries are generated with DeepSeek-V4-Flash11 1 https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash and embedded with all-MiniLM-L6-v2.22 2 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 5 Main Results 5.1 RQ1: Effectiveness of PTA-IRT Lite Verified Full Pro Avg Method MAE↓ τ↑ ρ↑ MAE↓ τ↑ ρ↑ MAE↓ τ↑ ρ↑ MAE↓ τ↑ ρ↑ MAE↓ τ↑ ρ↑ PTA-IRT .045±.004 .836 .950 .043±.015 .872 .976 .048±.025 .956 .991 .029±.017 .890 .974 .041±.015 .888 .973 w/o Traj. scorer .055±.021 .778 .930 .065±.008 .860 .969 .048±.029 .912 .982 .047±.010 .890 .969 .054±.017 .860 .963 w/o LUPI .085±.004 .573 .755 .042±.013 .848 .964 .053±.032 .802 .912 .092±.040 .868 .965 .068±.022 .773 .899 + Top-K .196±.031 .812 .942 .193±.012 .752 .917 .228±.163 .934 .982 .086±.025 .802 .921 .176±.058 .825 .941 + Clustering .182±.037 .819 .947 .189±.015 .757 .921 .225±.162 .912 .974 .090±.027 .758 .908 .171±.060 .812 .937 Table 4: Ablation of PTA-IRT at a 10%10\% calibration budget. Lower MAE and higher Kendall’s τ / Spearman’s ρ are better. Best values in each column are bolded (all ties are bolded). Values are rounded to three decimals with leading zeros omitted. Figure 3: Contribution of trajectory summaries under summary dropout (10%10\% calibration). Solid lines: With ESS; dashed lines: Without ESS. Colors denote benchmarks. RQ1 examines how PTA-IRT performs under limited calibration budgets relative to prior IRT baselines. Table 3 reports the main comparison at a 10%10\% budget on four SWE benchmarks, and Figure 2 evaluates budget sensitivity on SWE-bench Lite under 55–25%25\% calibration. Finding 1. With only 10%10\% calibration, PTA-IRT consistently outperforms prior IRT baselines on both score recovery and ranking. As shown in Table 3, PTA-IRT is best on every MAE/τ/ρ column across the four SWE benchmarks, averaging MAE ±0.0150.041\!±\!0.015, τ=0.888τ\!=\!0.888, and ρ=0.973ρ\!=\!0.973. This advantage holds under two complementary regimes. On Lite and Verified, where agent pools are dense, PTA-IRT attains the lowest MAE and the highest τ/ρ. On Full and Pro, where task sets are large but agents are few, PTA-IRT likewise achieves the best score and ranking recovery. Finding 2. PTA-IRT is robust across calibration budgets, and a small budget already yields practically useful ranking agreement. As shown in Figure 2, 5%5\% calibration already gives τ=0.768τ\!=\!0.768, within the practical range reported in prior analyses [23, 33] (τ≥0.7τ\!≥\!0.7–0.80.8). Raising the budget from 5%5\% to 20%20\% further improves ranking on Lite (τ: →0.8860.768\!→\!0.886) and tends to lower MAE. PTA-IRT remains the best method across the full 55–25%25\% range, underscoring the robustness of our approach under varying calibration budgets. 5.2 RQ2: Ablation Analysis of PTA-IRT RQ2 examines the contribution of each component of PTA-IRT to recovering full-benchmark scores and rankings. Table 4 reports ablations at a 10%10\% calibration budget. w/o Traj. scorer removes the summary encoder from the 4PL IRT used for informative selection. w/o LUPI drops teacher–student LUPI and estimates ability with the student model alone, without privileged summary supervision. + Top-K and + Clustering replace stratified information selection with global Top-K by information and pass-rate clustering, respectively. Finding 3. Each component of PTA-IRT contributes to recovering full-benchmark scores and rankings. As shown in Table 4, the full configuration achieves the best average MAE/τ/ρ. Removing the trajectory-aware scorer (w/o Traj. scorer) or LUPI (w/o LUPI) raises Avg MAE and lowers ranking agreement; the only local exception is a marginal MAE improvement on Verified without LUPI, which does not overturn the average. Replacing stratified selection with Top-K or clustering likewise worsens the average on both score and ranking. 5.3 RQ3: Contribution of Trajectory Summaries RQ3 examines the contribution of historical trajectory summaries to PTA-IRT. We apply summary dropout: a controlled fraction of train-summary embeddings is set to zero before retraining the trajectory-aware scorer and rebuilding the stratified calibration set S. We compare With ESS, which down-weights corrupted cells, against Without ESS, which still feeds corrupted embeddings at full weight. Figure 3 reports MAE, Kendall’s τ, and Spearman’s ρ on four SWE benchmarks under a 10%10\% calibration budget. Finding 4. Trajectory summaries improve recovery quality; their benefit comes from summary content rather than a vacuous privileged channel. As shown in Figure 3, increasing summary dropout generally weakens score recovery and ranking relative to the clean setting, though the curves are not strictly monotonic across benchmarks. At full dropout without ESS, Lite approaches the RQ2 ablation that removes the trajectory-aware scorer (MAE 0.0560.056, τ=0.778τ\!=\!0.778), indicating that the gains of PTA-IRT mainly come from summary content rather than merely having an unused summary pathway. Finding 5. ESS effectively protects PTA-IRT when trajectory summaries are unreliable. Under the same dropout levels, With ESS stays closer to the clean setting than Without ESS on most benchmark–metric pairs. The protective gap is clearest at low-to-mid dropout (e.g., on Lite, Without ESS already falls to τ≈0.735τ\!≈\!0.735 at 50%50\% dropout), where ESS can still down-weight corrupted cells. At extreme dropout, both degrade as usable process signal disappears. 5.4 RQ4: Characteristics of Trajectory Summaries RQ4 examines what distinctive structure trajectory summaries capture relative to issue text and submitted patches, and how that structure is organized across summary fields and same-item outcome pairs. We answer this with two complementary views: cross-channel similarity (Table 5) and same-item outcome geometry (Table 6). Cross-channel similarity. For each (agent, item) cell we compute cosine similarity between the process summary (and its four fields) and (i) the question embedding jq_j and (i) the submitted-answer embedding. As a control for question alignment, we also report Δ vs. randQ: the gap between similarity to the true question and the mean similarity to a randomly sampled question. Table 5 summarizes the results. Channel ↔ ↔ . Δ randQ Summary 0.714 0.638 +0.473+0.473 Task Goal 0.776 0.532 +0.587+0.587 Context Explored 0.502 0.533 +0.314+0.314 Edits Executed 0.530 0.670 +0.333+0.333 Path Overview 0.468 0.441 +0.272+0.272 Table 5: Cross-channel cosine similarity on SWE-bench Verified. Δ vs. randQ applies to the ↔ Question column. Finding 6. Trajectory summaries exhibit a clear division of labor: Task Goal tracks the issue text, Edits Executed tracks the submitted patch, while Context Explored and Path Overview remain process-specific. Whole-summary similarity to the question (0.7140.714) exceeds a random-question baseline by a large margin (Δ=+0.473 \!=\!+0.473). Field-level scores reveal complementary roles rather than redundancy. Task Goal is closest to the question (0.7760.776), which largely explains the high Summary↔ score. Edits Executed is closest to the submitted answer (0.6700.670), whereas Context Explored and Path Overview stay lower on both axes (≤0.533≤\!0.533). A trajectory summary therefore combines question-facing goal understanding with edit-level content and independent exploration/path structure. Same-item outcome geometry. We next compare representations of successful and failed solving paths on the same task. For every item with multiple agents, we form agent pairs and report mean embedding distance (1−cosine1-cosine) for both-pass (P), both-fail (F), and pass–fail (PF) pairs, together with Δout=d(PF)−12(d(P)+d(F)) _out=d(PF)- 12 (d(P)+d(F) ). A larger Δout _out indicates stronger outcome-related structure. Table 6 reports Answer, Summary, and the four summary fields. Channel P F PF Δout _out Answer 0.175 0.290 0.265 0.033 Summary 0.143 0.163 0.158 0.005 Task Goal 0.127 0.147 0.141 0.003 Context Explored 0.337 0.387 0.372 0.010 Edits Executed 0.366 0.422 0.414 0.020 Path Overview 0.363 0.387 0.389 0.013 Table 6: Same-item pair distances on SWE-bench Verified. P/F/PF: both-pass / both-fail / pass–fail. Finding 7. Submitted answers concentrate outcome geometry, while trajectory summaries preserve process diversity. For answers, same-outcome successes are much closer than failures (P 0.1750.175 vs. F 0.2900.290), and Δout=0.033 _out\!=\!0.033. For full summaries, P and F distances remain close (0.1430.143 vs. 0.1630.163) and Δout _out is only 0.0050.005; among fields, Context Explored, Edits Executed, and Path Overview keep large absolute distances (≥0.337≥\!0.337), and Edits Executed shows the largest Δout _out (0.0200.020), consistent with its stronger answer alignment in Table 5. Thus pass/fail is expressed most directly in the final patch, whereas summaries retain heterogeneous solving paths that are not collapsed by binary labels and patches alone. 6 Conclusion Evaluating SWE agents on full benchmarks is expensive because each task can require long-horizon exploration, editing, and testing. We present PTA-IRT, which treats historical execution trajectories as privileged information for efficient IRT-based evaluation. PTA-IRT builds structured trajectory summaries, selects a difficulty-stratified calibration subset with trajectory-aware measurement features, and estimates a new agent’s full-benchmark score and ranking from limited calibration observations via LUPI. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks under our evaluation protocol, showing that privileged trajectory information makes budgeted SWE agent evaluation more informative. References [1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating large language models trained on code. External Links: 2107.03374, Link Cited by: §2.1, §2.1. [2] X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V. Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler (2025) SWE-bench pro: can ai agents solve long-horizon software engineering tasks?. External Links: 2509.16941, Link Cited by: §2.1, §4.1. [3] X. Ding, C. Pan, Z. Li, J. Zhang, S. Wang, and Z. Wei (2026) AutoJudger: an agent-driven framework for efficient benchmarking of MLLMs. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 15009–15034. External Links: Link, Document, ISBN 979-8-89176-390-6 Cited by: §2.1, §4.2. [4] J. Gong, Y. Wu, L. Liang, Y. Wang, J. Chen, M. Liu, and Z. Zheng (2025) Cosqa+: enhancing code search evaluation with a multi-choice benchmark and test-driven agents. IEEE Transactions on Software Engineering 52 (1), p. 206–220. Cited by: §2.1. [5] W. Gu, J. Chen, Y. Wang, T. Jiang, X. Li, M. Liu, X. Liu, Y. Ma, and Z. Zheng (2025) What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond. arXiv preprint arXiv:2503.20589. Cited by: §2.1. [6] H. Guo, X. Chen, Y. Huang, Y. Wang, X. Ding, Z. Zheng, X. Zhou, and H. Dai (2023) Snippet comment generation based on code context expansion. ACM Transactions on Software Engineering and Methodology 33 (1), p. 1–30. Cited by: §2.1. [7] L. Guo, W. Tao, R. Jiang, Y. Wang, J. Chen, X. Liu, Y. Ma, M. Mao, H. Zhang, and Z. Zheng (2025) Omnigirl: a multilingual and multimodal benchmark for github issue resolution. Proceedings of the ACM on Software Engineering 2 (ISSTA), p. 24–46. Cited by: §2.1, §2.1. [8] L. Guo, Y. Wang, C. Li, W. Tao, P. Yang, J. Chen, H. Song, D. Tang, and Z. Zheng (2026) SWE data construction, automatically!. Proceedings of the ACM on Software Engineering 3 (FSE), p. 525–546. Cited by: §2.1, §2.1. [9] R. K. Hambleton and H. Swaminathan (2013) Item response theory: principles and applications. Springer Science & Business Media. Cited by: §2.2. [10] W. K. Hastings (1970) Monte carlo sampling methods using markov chains and their applications. Biometrika 57 (1), p. 97–109. External Links: ISSN 0006-3444, Document, Link, https://academic.oup.com/biomet/article-pdf/57/1/97/23940249/57-1-97.pdf Cited by: §4.2. [11] N. Jain, Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica (2025) LiveCodeBench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 58791–58831. External Links: Link Cited by: §2.1, §2.1. [12] T. Jiang, Y. Wang, Y. Wang, D. Guo, E. Shi, Y. Ma, J. Chen, and Z. Zheng (2025) AlignCoder: aligning retrieval with target intent for repository-level code completion. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 971–982. Cited by: §2.1. [13] T. Jiang, Y. Wang, X. He, D. Guo, J. Chen, M. Wen, E. Shi, X. Liu, Y. Ma, and G. Li (2026) PhoenixRepair: rethinking repair strategy exploration in software agents. External Links: 2607.18859, Link Cited by: §2.1. [14] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024) SWE-bench: can language models resolve real-world github issues?. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 54107–54157. External Links: Link Cited by: §2.1, §2.1, §2.1, §4.1. [15] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul (1999) An introduction to variational methods for graphical models. Machine Learning 37 (2), p. 183–233. External Links: ISSN 1573-0565, Document, Link Cited by: §4.2. [16] S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan (2025) AI Agents That Matter. Transactions on Machine Learning Research. External Links: Link Cited by: §1. [17] A. Kipnis, K. Voudouris, L. Schulze Buschoff, and E. Schulz (2025) Metabench - a sparse benchmark of reasoning and knowledge in large language models. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 31734–31770. External Links: Link Cited by: §1. [18] J. P. Lalor, H. Wu, and H. Yu (2016) Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras (Eds.), Austin, Texas, p. 648–657. External Links: Link, Document Cited by: §2.1. [19] W. Lee, J. Lee, D. Kim, and B. Ham (2020) Learning with privileged information for efficient image super-resolution. In Computer Vision – ECCV 2020, A. Vedaldi, H. Bischof, T. Brox, and J. Frahm (Eds.), Cham, p. 465–482. External Links: ISBN 978-3-030-58586-0 Cited by: §2.3. [20] Y. Li, E. Shi, D. Zheng, K. Duan, J. Chen, and Y. Wang (2024) Repomincoder: improving repository-level code generation based on information loss screening. In Proceedings of the 15th Asia-Pacific Symposium on Internetware, p. 229–238. Cited by: §2.1. [21] D. C. Liu and J. Nocedal (1989) On the limited memory bfgs method for large scale optimization. Mathematical Programming 45 (1), p. 503–528. External Links: ISSN 1436-4646, Document, Link Cited by: §3.4. [22] J. Liu, K. Wang, Y. Chen, X. Peng, Z. Chen, L. Zhang, and Y. Lou (2026) Large language model-based agents for software engineering: a survey. ACM Trans. Softw. Eng. Methodol.. Note: Just Accepted External Links: ISSN 1049-331X, Link, Document Cited by: §2.1. [23] N. F. Liu, T. Lee, R. Jia, and P. Liang (2023) Do question answering modeling improvements hold across benchmarks?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 13186–13218. External Links: Link, Document Cited by: §5.1. [24] S. Liu, Y. Huang, M. Liu, J. Chen, E. Shi, Y. Ma, H. Zhang, Y. Zhang, and Y. Wang (2026) ShortCoder: knowledge-augmented syntax optimization for token-efficient code generation. External Links: 2601.09703, Link Cited by: §2.1. [25] D. Lopez-Paz, L. Bottou, B. Schölkopf, and V. Vapnik (2016) Unifying distillation and privileged information. External Links: 1511.03643, Link Cited by: §2.3. [26] F. M. Lord and M. R. Novick (2008) Statistical theories of mental test scores. IAP. Cited by: §2.1. [27] F. M. Lord (1980) Applications of item response theory to practical testing problems.. ERIC. Cited by: §3.3. [28] F. Maia Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin (2024) TinyBenchmarks: evaluating LLMs with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 34303–34326. External Links: Link Cited by: §1. [29] Y. Perlitz, E. Bandel, A. Gera, O. Arviv, L. Ein-Dor, E. Shnarch, N. Slonim, M. Shmueli-Scheuer, and L. Choshen (2024) Efficient benchmarking (of language models). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 2519–2536. External Links: Link, Document Cited by: §2.1. [30] B. B. Reeve and P. Payers (2005) Applying item response theory modelling for evaluating. Assessing quality of life in clinical trials: Methods and practice 55. Cited by: §2.2. [31] V. Sharmanska, N. Quadrianto, and C. H. Lampert (2013) Learning to rank using privileged information. In Proceedings of the 2013 IEEE International Conference on Computer Vision, ICCV ’13, USA, p. 825–832. External Links: ISBN 9781479928408, Link, Document Cited by: §2.3. [32] E. Shi, Y. Wang, W. Tao, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun (2022) Race: retrieval-augmented commit message generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 5520–5530. Cited by: §2.1. [33] K. Sun, A. Williams, and D. Hupkes (2023) The validity of evaluation results: assessing concurrence across compositionality benchmarks. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), J. Jiang, D. Reitter, and S. Deng (Eds.), Singapore, p. 274–293. External Links: Link, Document Cited by: §5.1. [34] W. Tao, Y. Zhou, Y. Wang, W. Zhang, H. Zhang, and Y. Cheng (2024) Magis: llm-based multi-agent framework for github issue resolution. Advances in Neural Information Processing Systems 37, p. 51963–51993. Cited by: §2.1. [35] R. Tian, Y. Ye, Y. Qin, X. Cong, Y. Lin, Y. Pan, Y. Wu, H. Haotian, L. Weichuan, Z. Liu, and M. Sun (2024) DebugBench: evaluating debugging capability of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 4173–4198. External Links: Link, Document Cited by: §2.1. [36] E. Tsutsumi, R. Kinoshita, and M. Ueno (2021) Deep item response theory as a novel test theory based on deep learning. Electronics 10 (9). External Links: Link, Document Cited by: §1, §2.1, §2.2, §4.2. [37] V. Vapnik and R. Izmailov (2015) Learning using privileged information: similarity control and knowledge transfer. Journal of Machine Learning Research 16 (61), p. 2023–2049. External Links: Link Cited by: §2.3. [38] V. Vapnik and A. Vashist (2009) A new learning paradigm: learning using privileged information. Neural Networks 22 (5), p. 544–557. Note: Advances in Neural Networks Research: IJCNN2009 External Links: ISSN 0893-6080, Document, Link Cited by: §2.3. [39] R. Vivek, K. Ethayarajh, D. Yang, and D. Kiela (2024) Anchor points: benchmarking models with much fewer examples. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Y. Graham and M. Purver (Eds.), St. Julian’s, Malta, p. 1576–1601. External Links: Link, Document Cited by: §2.1. [40] Y. Wang, Y. Wang, S. Wang, D. Guo, J. Chen, J. Grundy, X. Liu, Y. Ma, M. Mao, H. Zhang, et al. (2025) Repotransbench: a real-world multilingual benchmark for repository-level code translation. IEEE Transactions on Software Engineering. Cited by: §2.1, §2.1. [41] Y. Wang, K. Duan, D. Zheng, E. Shi, F. Zhang, Y. Wang, J. Chen, X. Liu, Y. Ma, H. Zhang, Q. Wang, and Z. Zheng (2026) Towards an understanding of context utilization in code intelligence. ACM Comput. Surv. 58 (11). External Links: ISSN 0360-0300, Link, Document Cited by: §2.1. [42] Y. Wang, L. Guo, E. Shi, W. Chen, J. Chen, W. Zhong, M. Wang, H. Li, H. Zhang, Z. Lyu, et al. (2023) You augment me: exploring chatgpt-based data augmentation for semantic code search. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), p. 14–25. Cited by: §2.1. [43] Y. Wang, E. Shi, L. Du, X. Yang, Y. Hu, Y. Wang, D. Guo, S. Han, H. Zhang, and D. Zhang (2025) Context-aware code summarization with multi-relational graph neural network. Automated Software Engineering 32 (1), p. 19. Cited by: §2.1. [44] Y. Wang, S. Wang, Y. Wang, B. Zhang, D. Guo, J. Chen, and Z. Zheng (2026) RepoReasoner: evaluating repository-level code reasoning ability of long-context language models. Proceedings of the ACM on Software Engineering 3 (FSE), p. 2790–2812. Cited by: §2.1. [45] Y. Wang, J. Wu, T. Jiang, M. Liu, J. Chen, C. Wang, E. Shi, X. Liu, Y. Ma, and Z. Zheng (2025) DrainCode: stealthy energy consumption attacks on retrieval-augmented code generation via context poisoning. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 778–790. Cited by: §2.1. [46] Y. Wang, B. Zhang, Y. Wang, D. Guo, T. Y. Zhuo, J. Chen, M. Liu, X. Zhang, and Z. Zheng (2026) ArkRepoBench: a repository-level code completion benchmark for harmonyos development. In Findings of the Association for Computational Linguistics: ACL 2026, p. 19409–19429. Cited by: §2.1, §2.1. [47] Y. Wang, Z. Zhang, C. Wang, X. Xu, M. Liu, Y. Wang, J. Chen, and Z. Zheng (2026) RealSec-bench: a benchmark for evaluating secure code generation in real-world repositories. In Findings of the Association for Computational Linguistics: ACL 2026, p. 35866–35883. Cited by: §2.1, §2.1. [48] Y. Wang, W. Zhong, Y. Huang, E. Shi, M. Yang, J. Chen, H. Li, Y. Ma, Q. Wang, and Z. Zheng (2025) Agents in software engineering: survey, landscape, and vision. Automated Software Engineering 32 (2), p. 70. Cited by: §2.1. [49] M. Wu, R. L. Davis, B. W. Domingue, C. Piech, and N. Goodman (2020) Variational item response theory: fast, accurate, and expressive.. International Educational Data Mining Society. Cited by: §4.2. [50] J. Yang, C. E. Jimenez, A. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. Narasimhan, D. Yang, S. Wang, and O. Press (2025) SWE-bench multimodal: do ai systems generalize to visual software domains?. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 2794–2829. External Links: Link Cited by: §2.1. [51] J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 50528–50652. External Links: Document, Link Cited by: §1, §2.1. [52] H. Ye, A. Z. Yang, C. Hu, Y. Wang, T. Zhang, and C. Le Goues (2025) Adverintent-agent: adversarial reasoning for repair based on inferred program intent. Proceedings of the ACM on Software Engineering 2 (ISSTA), p. 1398–1420. Cited by: §2.1. [53] P. Yuan, Y. Zhang, S. Feng, Y. Li, X. Wang, J. Shi, C. Tan, B. Pan, Y. Hu, and K. Li (2025) Beyond one-size-fits-all: tailored benchmarks for efficient evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 15591–15615. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2.1. [54] Z. Zhang, C. Wang, Y. Wang, E. Shi, Y. Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng (2025) Llm hallucinations in practical code generation: phenomena, mechanism, and mitigation. Proceedings of the ACM on Software Engineering 2 (ISSTA), p. 481–503. Cited by: §2.1. [55] D. Zheng, Y. Wang, E. Shi, X. Liu, Y. Ma, H. Zhang, and Z. Zheng (2024) Top general performance= top domain performance? domaincodebench: a multi-domain code generation benchmark. arXiv preprint arXiv:2412.18573. Cited by: §2.1, §2.1. [56] D. Zheng, Y. Wang, E. Shi, R. Zhang, Y. Ma, H. Zhang, and Z. Zheng (2025) Humanevo: an evolution-aware benchmark for more realistic evaluation of repository-level code generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), p. 1372–1384. Cited by: §2.1, §2.1. [57] D. Zheng, Y. Wang, X. Wang, K. Duan, H. Zhang, X. Liu, Y. Ma, and Z. Zheng (2026) From static to dynamic: benchmarking real-world code review with mcr-bench. External Links: 2608.27442, Link Cited by: §2.1, §2.1. [58] D. Zheng, R. Ye, Y. Wang, Y. Ye, H. Zhang, E. Shi, X. Liu, Y. Ma, J. Yu, and Z. Zheng (2026) SWE-prime: fewer trajectories, better performance. External Links: 2608.27449, Link Cited by: §2.1. [59] H. Zhou, H. Huang, Z. Zhao, L. Han, H. Wang, K. Chen, M. Yang, W. Bao, J. Dong, and B. Xu (2026) Lost in benchmarks? rethinking large language model benchmarking with item response theory. Proceedings of the AAAI Conference on Artificial Intelligence 40 (41), p. 35085–35093. External Links: Link, Document Cited by: §1, §2.1, §2.2, §4.2.