Paper deep dive

Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First

Ishaan Bhola, Adithyan Krishnan, Mukunda NS

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 80

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandbox-verified, with false claims stripped before delivery. The searcher's hidden states, together with the task text, then feed a resume-based router that dispatches the task to one of four frontier fixers. Adding a new fixer requires no retraining. On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark's official capped budget tier, SuperScout matches the best single model's solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline. A no-router ablation, always the cheapest fixer with the handoff, ties the routed system on this benchmark, so the handoff rather than the routing decision carries the result. A paired calibration study points to the mechanism: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, though at $N=99$ the per-fixer effects are directional only; the searcher's hidden states improve cost routing on the calibration labels while the handoff's own text does not. The searcher's compute adds less than half a cent of GPU time per task.

PDF

Open source PDF →

PDF not stored locally. Use the link above to view on the source site.

Full Text

77,036 characters extracted from source content.

Expand or collapse full text

Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First†thanks: Model and data: https://huggingface.co/SuperAGI/SuperScout-7B (weights), https://huggingface.co/datasets/SuperAGI/superscout-sft-search (training corpus), https://huggingface.co/datasets/SuperAGI/superscout-router-features (router features). Code and data: https://github.com/TransformerOptimus/superscout. Ishaan Bhola Adithyan Krishnan Mukunda NS Abstract Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandbox-verified, with false claims stripped before delivery. The searcher’s hidden states, together with the task text, then feed a résumé-based router that dispatches the task to one of four frontier fixers. Adding a new fixer requires no retraining. On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark’s official capped budget tier, SuperScout matches the best single model’s solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline. A no-router ablation, always the cheapest fixer with the handoff, ties the routed system on this benchmark, so the handoff rather than the routing decision carries the result. A paired calibration study points to the mechanism: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, though at N=99N=99 the per-fixer effects are directional only; the searcher’s hidden states improve cost routing on the calibration labels while the handoff’s own text does not. The searcher’s compute adds less than half a cent of GPU time per task. 1 Introduction Frontier-model agents now resolve real repository issues, from locating bugs to generating patches [1], but each solve carries a real per-task price. Strong open-weights models cost a fraction yet close fewer tasks. Any team deploying an issue-solving system therefore faces a standing choice between the expensive best model and cheaper alternatives. Existing LLM routers [2, 3] make that choice from the task text alone, before anything has engaged with the actual repository. Replaying learned routers over the public per-task results of three SWE benchmarks reveals that solve sets are largely nested: models that solve more tasks nearly always subsume the solve sets of weaker ones, and no router reliably improves on always selecting the strongest model (§3). Routing for accuracy therefore offers little headroom. Routing for cost, by contrast, only requires predicting when a cheaper model will suffice. $0.20$0.30$0.60$1.205252%5454%5656%5858%6060%SuperScoutOpus 4.6 soloKimi K2.5 soloGPT-5.2 soloblind-mixing lineCost per solve (log scale)Solve rate Figure 1: Cost per solve versus solve rate on SWE-bench Pro (Python-266). The dashed curve is the blind-mixing line (random cost-blind mix of Kimi K2.5 and Claude Opus 4.6). SuperScout matches Claude Opus 4.6 at about a fifth of its cost per solve, well above the line. The system point is all-in; solo points are fixer API only. The x-axis is logarithmic. SuperScout routes after scouting, a pattern we call scrouting. A 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff: a list of implicated files, diagnostic notes, and a candidate reproduction test. These claims pass through a sandbox verification step, and claims that do not check out are stripped before a downstream model ever sees them. A résumé router then selects among four frontier fixers using the task text together with the searcher’s own hidden states; adding a new fixer requires no retraining. Figure 1 compares SuperScout to solo frontier models on SWE-bench Pro’s full Python slice [4] (266 tasks), matching the benchmark’s official capped budget tier exactly. SuperScout matches the best single frontier model’s solve rate, resolving 159 of 266 tasks versus 158, at $0.230 total cost per solve compared to $1.274: about a fifth. The configuration sits above the blind-mixing line (the accuracy/cost segment any random split of traffic between a cheap and a strong model would achieve). The searcher’s contribution to system cost is negligible: SuperScout-7B’s entire GPU bill for the evaluation was $1.13. A paired calibration study on 100 fresh tasks points to why this cost reduction holds: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, and the searcher’s hidden states improve cost routing where the handoff’s own text does not (§8). We make five contributions: • An engage-then-route architecture with a trained searcher whose verified handoff is consumed by the chosen fixer. • A zero-cost replay audit of published per-task outcomes on three SWE benchmarks showing that solve sets are largely nested and that no learned router reliably beats always calling the strongest model, which motivates routing for cost rather than accuracy. • Routing features drawn from the searcher’s hidden states, with a résumé-based N-way pool where adding a new fixer requires no retraining. • A matched-protocol evaluation on SWE-bench Pro’s Python census showing frontier-matching accuracy at about a fifth of the total cost per solve. • A paired calibration study measuring the handoff’s redistribution pattern and the router’s feature design space, plus a verification gate that strips the searcher’s false reproduction claims. 2 Related Work SWE-agent [5] introduced the agent-computer-interface paradigm for autonomous issue resolution in real repositories. Agentless [6] showed that a fixed pipeline, with no agent autonomy at all, can achieve competitive resolve rates. SWE-smith [7] addresses data scarcity by synthesizing large-scale training corpora for such agents, while SWE-Gym [8] and R2E-Gym [9] supply executable training environments built from real repositories. The community evaluates these systems on SWE-bench [1] and its harder successor SWE-bench Pro [4]. A parallel line of work trains small models specifically for code localization. SWE-Fixer [10] trains a 7B retriever whose output feeds one fixed larger editor. LocAgent [11] and SweRank [12] similarly train compact models to identify fault locations within a repository. These systems produce exactly the kind of evidence a router could consume, yet none of them routes: the localizer’s output terminates in a single, predetermined consumer. LLM routing has been studied along two axes. Cost-quality routers such as RouteLLM [2], FrugalGPT [3], and Hybrid LLM [13] learn to dispatch queries to cheaper or stronger models based on predicted difficulty. Profile-based selectors take a complementary approach, building per-model signatures from benchmark outcomes or learned embeddings and matching incoming queries against them [14, 15, 16]. Both families share a structural property: they decide from the task prompt, and optionally from model profiles, before any system has engaged with the concrete problem instance. Several concurrent efforts address routing in code-generation settings. SWE-Router [17] probes each issue with a weak model and then escalates between exactly two models; on escalation the strong model restarts from scratch, so the probe’s work is not consumed. Its theoretical analysis of trajectory-conditioned routing nonetheless supports the design direction we pursue. CodeRescue [18] routes among recovery actions (reflect, replan, or escalate) inside a single agent’s trajectory rather than among fixer models. TRACE-Router [19] formulates model selection as a contextual bandit over an N-way pool, though it is evaluated outside software engineering. Self-play SWE-RL [20] trains solvers via self-play and evaluates on SWE-bench Pro but does not route. As Table I summarizes, SuperScout is distinct in combining a trained searcher, a handoff the fixer actually consumes (after verification), an N-way fixer pool where adding a new fixer requires no retraining, and routing features drawn from the searcher’s hidden states. TABLE I: Capability comparison of SuperScout with related routing and localization systems. Columns: Searcher, a model trained to scout the repository before any fix is attempted; Handoff, the searcher’s work product is consumed by the downstream fixer rather than discarded; N-way, selection over a pool of more than two fixers; Onboard, adding a new fixer requires no retraining of any learned component; States, routing features include the searcher’s internal hidden states. Concurrent code-routing work escalates between a fixed pair and discards the cheap model’s evidence on escalation; profile-based routers select from the task prompt alone, before any exploration; trained small localizers produce evidence a router could use but perform no routing. ✓† : SWE-Fixer’s retriever feeds one fixed larger editor, while LocAgent and SweRank stop at localization. Rows: SWE-Router [17], CodeRescue [18], TRACE-Router [19], profile routers [15, 16, 14], small localizers [10, 11, 12]. System Searcher Handoff N-way Onboard States SWE-Router × × × × × CodeRescue × × × × × TRACE-Router × × ✓ × × Profile routers × × ✓ ✓ × Small localizers ✓ ✓† × × × SuperScout ✓ ✓ ✓ ✓ ✓ 3 Problem Setup and Metrics 3.1 Task and Cost Model We study repo-level issue resolution: given the text of an issue and a snapshot of the repository, the system must produce a source-code patch whose correctness is judged by held-out tests [1, 4]. Frontier language models already solve these tasks at high rates, but their per-task cost is substantial. Strong open-weights models are far cheaper yet less reliable. In a deployment that selects from a pool of candidate fixer models, the natural question is whether one can retain the strongest model’s effectiveness while spending less. To answer that question we adopt a single yardstick, defined before any experiment: total cost per solve at a matched solve rate. “Total” means every system component’s spend, including searcher GPU time, verification sandboxes, and fixer API calls, not just the final model invocation. A system claims a cost improvement only if its solve rate matches that of the best individual model. This metric exposes a trivial lower bound. Any point on the line segment between a cheap model’s (cost, accuracy) and a strong model’s (cost, accuracy) is achievable by randomly splitting traffic between the two, a strategy we call blind mixing. A routing system is interesting only if it operates above that segment. We describe the operational details of the blind-mixing line in §6. 3.2 Why Accuracy-Only Routing Is Insufficient Before building SuperScout we replayed a set of learned routers over the publicly available per-task results of three SWE-coding benchmarks: SWE-bench Verified [21], SWE-bench Multilingual [22], and SWE-bench Pro [4]. The replay incurs zero model cost, and the routers span four families: embedding retrieval (k-N), trained heads (logistic and MLP), clustering, and a language-rule baseline. Two findings shaped the design that followed. First, solve sets are strongly nested: the tasks a weaker model solves are largely a subset of those the strongest model solves. Measured as set containment, the overlap is 0.941, 0.912, and 0.773 across the three benchmarks. Nor is this an artifact of comparing cheap models with frontier ones: replaying pools built from frontier models alone, one per lab, leaves containment at 0.90 to 0.93. These models are generally strong rather than complementary specialists, so an accuracy-oriented router has almost no room to combine their strengths. Figure 2 sketches the consequence: between any two models the accuracy prize is only a thin sliver, while the large shared region is where a cheaper model would have sufficed all along. (a) any two frontier peersall taskssolved by both solved by only one: rare (b) a cheaper model vs. the strongest Strongest model’s solve set Cheaper model’s solve set a cheaper model suffices here: route for cost solved only by the cheaper model: rare solved by neither Figure 2: Solve sets are nearly nested (schematic). (a) Even two frontier peers solve almost the same tasks, so accuracy routing has little to win. (b) A cheaper model’s set sits mostly inside the strongest model’s; that shared region is where routing for cost pays. Second, no learned router we tested exceeded the solve rate of always picking the strongest model; every observed gap fell within noise. Taken together, these results suggest a plain conclusion: routing for accuracy has little headroom on current issue-resolution benchmarks. Routing for cost, by contrast, does not require complementary skills at all. It requires only the ability to predict when a cheaper model will suffice. Full audit tables appear in Appendix A. 3.3 Goal Our objective inverts the premise of prior routing work. Rather than seeking accuracy gains by combining models, we aim to match the strongest available model’s solve rate at a materially lower total cost per solve. The thesis is that routing should happen after engaging with the problem, not from the task text alone. In SuperScout a small searcher model first explores the repository; both its explicit work product (a structured handoff) and its internal representation of the task (hidden states) then inform the routing decision. The evaluation partly supports this thesis: the engagement’s payoff arrives through the handoff more than through the routing decision (§7). §4 details the architecture. SearchHandoffGateRouteFixTaskissue text SuperScout-7B explores the repo sampled decoding bounded turns Structured handoff implicated filesrepo notesrepro test Verify-then-strip replays the repro claim✓ replays: kept× fails: stripped Résumé router task text ++ SuperScout-7B hidden states vs. one résumé per fixer Fixer pool GPT-5.2Claude Opus 4.6Gemini 3 FlashKimi K2.5+ new fixer: résumé only▶ the chosen fixer gets the task ++ surviving handoff no handoff: the chosen fixer proceeds from the issue text alone Figure 3: The SuperScout pipeline. SuperScout-7B explores the repository and emits a structured handoff, which a sandbox gate verifies before a résumé router selects one of four frontier fixers. The dashed path is the fallback: when no handoff is produced, the chosen fixer proceeds from the issue text alone. Adding a new fixer requires only a résumé, not retraining. 4 The SuperScout System Figure 3 traces a single task through the pipeline. The task enters a search phase, where SuperScout-7B explores the repository and writes a structured handoff. A sandbox gate then verifies the handoff’s reproduction claims before a résumé router selects one fixer from a pool of frontier models. That fixer receives the task and the surviving handoff, produces a patch, and submits it to official scoring. The entire trajectory is a single pass: no parallel sampling, no cross-model escalation mid-task. When the searcher produces no handoff at all, the chosen fixer proceeds from the issue text alone; this fallback defines the floor as fixer-solo performance. 4.1 The Searcher and Its Handoff SuperScout-7B is a 7B model built on Qwen2.5-Coder [23], trained exclusively for the search phase (§5). Given a task, it explores the repository, localizes the implicated files, attempts to write a failing reproduction test, and then produces a handoff document before stopping. The handoff is a structured artifact kept short by design, typically about a page of text (4 KB). It contains implicated files with line regions, ranked by confidence; a reproduction attempt specifying a file, command, and observed output; dead ends the searcher already tried; and free-form repository notes. A verified example appears in Appendix M. When the search exhausts its turn budget without committing to a handoff, a single extra generation step demands one. Such handoffs are tagged forced and are never pooled with spontaneous ones anywhere in this paper. 4.2 Verify-then-Strip The searcher’s reproduction claims are not trusted. Before any fixer sees a handoff, a sandbox replays the claimed reproduction command against the unpatched repository. A claim that does not genuinely fail is stripped: both the test file and the claim itself are removed from the handoff. A verified claim, by contrast, is materialized so the fixer can use it directly. Calibration revealed that most reproduction claims are in fact false (§8), and §7 quantifies the guard’s effect at benchmark scale. Injection is blanket: every routed fixer receives the surviving handoff identically. 4.3 The Résumé Router TABLE I: The résumé router, end to end. Each fixer’s résumé stores two embedding centroids and a base rate, computed from public per-task outcomes. A logistic head per feature space scores P(solve)P(solve); the router walks the pool in cheap-first order, stopping at the first fixer clearing θ. Adding a new fixer requires only 25–50 public outcomes and no retraining. Component Specification Embedder Qwen3-Embedding-0.6B frozen 4096-token cap, L2L_2-normalized Résumé per fixer: mean embedding of solved, mean of failed, base rate from 25–50 public outcomes Features cos⁡(x,s) (x,s), cos⁡(x,f) (x,f), cos⁡(x,s)−cos⁡(x,f) (x,s)- (x,f), p, cos⁡(x,s)⋅p (x,s)·p, (cos⁡(x,s)−cos⁡(x,f))⋅p( (x,s)- (x,f))·p uncentered, per feature space Hidden state SuperScout-7B layer −4-4, pre-decode final-position state (3584-d) Scorer logistic regression per space, uniform probability average, θ=0.30θ=0.30, cheapest-adequate New fixer 25–50 public outcomes; no retraining of SuperScout-7B, the embedder, or the scorer Each fixer in the pool is summarized by a résumé built from 25–50 public per-task outcomes. A résumé stores three quantities: the mean embedding of tasks the fixer solved, the mean embedding of tasks it failed, and a base solve rate. Table I lists the full specification. At routing time the task is embedded in two complementary feature spaces. Its text is encoded by a frozen off-the-shelf embedder [24]. Its content as SuperScout-7B experienced it is captured through the searcher’s 3,584-dimensional pre-decode hidden state, drawn from the fourth-from-last layer at the final token position. In each feature space the router computes uncentered cosine similarities between the task and every résumé’s solved and failed centroids, their difference, and the base rate; a single logistic regression per feature space, shared across fixers, then scores P(solve)P(solve) for each fixer from its résumé-relative features. The routing head blends the task-text and hidden-state feature sets by uniformly averaging their predicted probabilities. Routing walks the pool in cheap-first order and assigns the task to the first fixer whose predicted probability clears a caution threshold θ; if none clears it, the task falls to a designated anchor model (§6). Logistic regression was chosen over a multi-layer perceptron during calibration, where the simpler scorer won repeatedly (§8). 4.4 Extensibility Adding a new fixer requires no retraining. A new model’s résumé consists of two averaged embeddings and a base rate, computed from its public outcomes. The embedder is frozen and SuperScout-7B is untouched; §7 presents a measured discussion. 5 Training the Searcher 5.1 Data and Supervised Training SuperScout-7B is trained on search-phase demonstrations sliced from openly licensed agent trajectories produced by other systems. Each demonstration captures one complete search episode: the agent explores a repository, localizes the fault, and writes a failing reproduction, with a synthesized handoff-emission turn appended as the final supervised target. Within that synthesized turn, the file list is extracted deterministically from the trace and the reproduction record is copied verbatim; only the free-form notes are written by a model, the open-weights gpt-oss-120b [25]. Traces are drawn from three public sources (Open-SWE-Traces [26], SWE-rebench-openhands [27, 28], and SWE-Hero [29]; the latter two are trajectories of the OpenHands scaffold [30]) and then success-filtered against the gold patch, keeping only trajectories that actually found the right files. Deduplication retains the two highest quality-ranked traces per issue, so the corpus counts distinct bugs rather than retellings of the same fix. The resulting dataset contains 19,911 examples. Six rows carrying malformed tool-call wrappers are dropped at load time, giving the 19,905 examples actually trained on. The realized language mix is Python 37.3%, Go 36.7%, TypeScript 19.7%, and JavaScript 6.3%. Table I summarizes the corpus composition. TABLE I: Composition of SuperScout-7B’s supervised search corpus. Each example is a full search trajectory with a synthesized handoff-emission turn as the final supervised step; loss is on assistant turns only. Traces are success-filtered and deduplicated to two per issue. The 23 evaluation repositories and 450 held-out vault issues are excluded; both exclusions were verified by a programmatic gate. Component Value Corpus Examples trained on 19,905 Built / frozen set 19,911 Held-back shelf (untrained) 9,478 Tokens per epoch 382.4M Packed 32k blocks (fill) 12,723 (91.7%) Language mix Go 7,304 (36.7%) Python 7,417 (37.3%) TypeScript 3,932 (19.7%) JavaScript 1,258 0(6.3%) Trace sources Open-SWE-Traces Go/TS/JS + Py SWE-rebench-openhands Python SWE-Hero-openhands Python Filters Traces kept per issue top 2 Verified reproduction 97.7% Excluded repositories 23 Excluded held-out issues 450 Contamination control was decided on day one: a 23-repository blocklist covering all 11 SWE-bench Pro repositories and 12 SWE-bench Verified [21] repositories is excluded from every training and calibration set. A programmatic gate verified zero blocklist hits in the frozen file. We fine-tune Qwen2.5-Coder-7B [23] with LoRA [31] on a single GPU. Before training, the base model’s localization rate is near zero; the search behavior is entirely learned from the demonstrations above. Hyperparameters are reported in Appendix L. 5.2 Decoding Matters More Than Expected The most consequential training-era finding was not about the training itself but about inference. On a 450-task held-out exam, greedy decoding finds the right files at a rate of 0.1104. A single sampled draw at temperature 0.9 reaches 0.3058, a 2.65× gain after matching for infrastructure timeouts. greedytemp 0.900.10.10.20.20.30.30.40.42.65×2.65×timeout-matched0.1100.1100.3060.306Mean gold-file recallFind rate (vault-450)commitratehandoffquality0112233parity3.03×3.03×0.96×0.96×Ratio, temp 0.9 // greedyDecomposition (dial-100) Figure 4: Sampled decoding is what makes the searcher useful. Left: switching from greedy to temperature 0.90.9 raises the find rate from 0.1100.110 to 0.3060.306 (2.65×2.65×, n=426n=426 of 450450). Right: on a separate 100-task dial set, commitment rises 3.03×3.03× while per-handoff quality holds at 0.96×0.96×; on the exam itself emission rises 3.4×3.4× against an 18% per-handoff recall cost (Appendix K). The two panels use different task sets; the right panel is a mechanism check, not a replication. The exam’s own decomposition shows what sampling buys and what it spends: the searcher’s emission rate rises from 0.213 to 0.718 (3.4×), while recall per emitted handoff falls from 0.517 to 0.426, an 18% quality cost (Appendix K). Sampling trades a slice of per-handoff quality for a much larger gain in willingness to commit, a strongly net-positive exchange. A separate 100-task dial set shows the same mechanism with commitment up 3.03× and per-handoff quality flat at 0.96× (Figure 4); the quality cost visible on the exam does not appear there, so the dial set is a mechanism check, not a replication. Greedy decoding, it turns out, makes the searcher reluctant to declare a result; sampling at moderate temperature recovers ability the model already has. The shipped configuration is therefore a single sampled draw at temperature 0.9 (§6). 5.3 Language Transfer SuperScout-7B is trained on four languages (Go, Python, TypeScript, JavaScript), yet SuperScout is deployed against repositories in languages the searcher has never seen. On the nine-language SWE-bench Multilingual evaluation [22], which contains no Python and so covers three of the four training languages (Figure 5), spontaneous-handoff localization quality is higher on the six never-trained languages (file-level F1=0.630F_1=0.630) than on the three trained ones (F1=0.455F_1=0.455). The inversion survives a difficulty-matched control restricted to single-gold-file tasks, where recall on never-trained languages reaches 0.791 versus 0.524 for the trained pool. JavaScript and TypeScript account for much of the trained pool’s lower average; the TypeScript and C++ cells each rest on 12 assigned tasks, of which 7 and 8 respectively produced the spontaneous handoffs analyzed, and are indicative only (full per-language table in Appendix J). 00.20.20.40.40.60.60.80.8GoJavaScriptTypeScript*JavaRubyRustPHPCC++*0.5460.5460.3890.3890.3430.3430.6120.6120.7010.7010.6670.6670.6300.6300.4350.4350.7080.708Spontaneous handoff file F1F_1trained (Go/JS/TS)never trained (6 langs)trained pool 0.455never-trained pool 0.630 Figure 5: Localization transfers to languages SuperScout-7B was never trained on. Per-language file-level F1F_1 of spontaneous handoffs (searcher alone, single draw, SWE-bench Multilingual). Solid bars: three trained languages; hatched: six never-trained. Dashed lines mark pooled aggregates (0.4550.455 trained, 0.6300.630 never-trained). The TypeScript and C++ cells (12 assigned tasks each; 7 and 8 spontaneous handoffs analyzed) are indicative only. The contrast carries a confound: JavaScript and TypeScript are at once the trained pool’s weakest cells and its smallest training slices, and both may be structurally harder to localize in (dynamic imports, build artifacts) independently of training exposure, so the trained-versus-never-trained comparison should not be read as causal. Our hypothesis for the pattern, and it is a hypothesis, is that SuperScout-7B learned a search method, not a language-specific vocabulary. The exploration strategies it relies on (reading directory trees, tracing imports, scanning test files) are structural operations that generalize across languages; the specific tokens involved matter less than the strategy of following them. Reinforcement learning. We built and validated a GRPO [32] training rig and ran 50 clean steps. Learning was flat. A trace-level autopsy showed that the reward signal was real but aimed at a behavior already near its ceiling after supervised training: the model could localize when it chose to commit, and sampling had already recovered that commitment. We shelved RL in favor of the decoding fix described above; rig details are in Appendix L. 6 Experimental Setup 6.1 Benchmark and Protocol We evaluate on SWE-bench Pro [4], using the full Python slice: all 266 tasks, a census rather than a sample. These span three repositories: 96 from ansible, 91 from openlibrary, and 79 from qutebrowser, out of 731 tasks in the public set. We chose Pro because its tasks are harder and less saturated than earlier SWE-bench variants, and because all 11 of its repositories were excluded from SuperScout-7B’s training data from day one via the contamination blocklist of §5. Our protocol matches the benchmark’s official capped budget tier exactly: a 50-LLM-call turn cap and a $2.00 per-attempt cost limit. The cap is silent; the agent is never told a budget exists. Source-code inspection of the official scaffold found no budget sentence in any prompt; when the cap is exhausted, the current working diff is auto-submitted. Task input follows the scaffold’s exact three-field concatenation of problem statement, requirements, and interface. For 105/266 tasks a byte-lossless JSON decode of double-serialized fields was applied; this deviation is logged. Fixer prompts are byte-identical to the SWE-bench Multilingual leaderboard templates [22], and a single task-text definition is used throughout, for the searcher, every fixer, and the router embedding alike. 6.2 Frozen System Configuration The searcher, gate, résumés, embedder, and router weights were frozen before first contact with the benchmark. SuperScout-7B serves at temperature 0.9, single draw, with pinned sampling parameters, per-episode cache isolation, a serving-health canary gate, and a 40-turn search budget. The handoff policy is blanket injection: every routed fixer receives the surviving handoff identically. Verify-then-strip runs first on every reproduction claim (mechanism in §4). The router’s logistic head was fit on all calibration data (recipe in Table I) and operates at caution threshold θ=0.30θ=0.30. This threshold was initially calibrated against solo-fixer outcomes; because the handoff systematically lifts inexpensive fixers, it was recalibrated on the 99-task lab set using with-handoff outcomes via the pre-existing matched-point rule. The router’s benchmark result is evaluated by routing each task and scoring it against that fixer’s measured episode under the official capped protocol; every one of the 266 routed picks has a real measured episode behind it. 6.3 Arms, Baselines, and Cost Accounting Four measured arms run fully paired on the same 266 tasks: three solo fixers (Claude Opus 4.6 [33], GPT-5.2 [34], and Kimi K2.5 [35]) and the SuperScout fixer runs with injected handoffs. Both frontier models were run in full as solo baselines because our calibration study and a small paid probe disagreed about which was stronger; the resolution is itself a result (§7). At the calibrated operating point (θ=0.30θ=0.30) the router’s fallback never fires, so anchor choice is moot. Two baselines anchor interpretation. The strongest-solo baseline is the single fixer with the highest solve rate. The blind-mixing line traces the accuracy-versus-cost segment produced by randomly assigning each task to Kimi K2.5 or Claude Opus 4.6 in varying proportions; a system point above this line does something smarter than chance mixing. Cost accounting is deliberately conservative against SuperScout. System costs are all-in: routed-fixer API spend plus SuperScout-7B’s entire GPU bill plus verify-then-strip sandbox infrastructure, all amortized per task. Solo arms count only their pure fixer API spend. We report total cost per solve at a matched solve rate; measured spend is disclosed in Appendix N. Scoring uses the benchmark’s official harness; a gold-patch control passes 265/266 tasks. The single failure stems from a dataset artifact (a truncated test-spec identifier) that is unwinnable for every arm equally and is kept in all denominators. 7 Results 7.1 Headline TABLE IV: Main results on SWE-bench Pro (Python-266). SuperScout matches the pool’s best single model (Claude Opus 4.6) at about a fifth of its cost per solve; 159 versus 158 is a one-task gap at n=266n=266, a match, not a win. The last row is the no-router ablation, which ties the routed system on this benchmark (§7.1). System Solves/266 Rate $/task $/solve vs. mix SuperScout (router, θ=0.30θ=0.30) 159 59.77% $0.137 $0.230 +3.60+3.60 Claude Opus 4.6 solo 158 59.40% $0.757 $1.274 — Kimi K2.5 solo 149 56.02% $0.106 $0.190 — GPT-5.2 solo 139 52.26% $0.570 $1.091 — No-router ablation (Kimi K2.5 ++ handoff) 159 59.77% $0.136 $0.227 +3.60+3.60 Solves are out of n=266n=266, top to bottom: 159, 158, 149, 139, 159. The SuperScout row and the ablation row share one all-in cost convention: fixer API spend plus the SuperScout-7B searcher’s GPU bill plus verify-then-strip sandbox infrastructure, amortized over 266 tasks. Solo rows are fixer API spend only. “vs. mix” is percentage points above the Kimi K2.5–Claude Opus 4.6 blind-mixing line at the row’s own $/task. Table IV summarizes the primary evaluation. SuperScout solves 159 of 266 tasks (59.77%) at a total cost per solve of $0.230. The pool’s best single model, Claude Opus 4.6 run solo, solves 158 of 266 (59.40%) at $1.274 per solve. SuperScout matches this ceiling at about a fifth of the cost per solve. The one-task gap between 159 and 158 at n=266n=266 is not a meaningful difference; it is a match. The no-router ablation sends every task to Kimi K2.5 with the handoff and no routing decision at all, also solving 159 of 266 at $0.227 per solve. The calibrated router concentrates 263 of 266 tasks on the cheapest fixer and adds no solves over this ablation; its three diversions to Gemini 3 Flash [36] cost $0.003 per solve, insurance rather than accuracy. On this benchmark, then, the handoff carries the result and routing collapses to cost allocation. Whether blanket assignment to one cheap fixer reaches parity is a property of this pool and benchmark, not a general rule: in deployment that answer is not known in advance, and the router is the component that predicts it, from free public data, before any spend. The ablation is ex-ante specifiable too, but only hindsight shows it reaches parity; the router turns that gamble into a calibrated decision. Both system rows sit above the blind-mixing line defined in §6 by ++3.60 percentage points at their respective cost points (visible in Figure 1). SuperScout-7B’s entire GPU bill for all 266 search episodes was $1.13, under half a cent per task. Why were both frontier models run in full as solo baselines? Our calibration study and a small paid probe disagreed about which was stronger. On the fresh calibration tasks (§8), GPT-5.2 was the strongest solo fixer at 60.6% solve rate. A small paid probe of 15 tasks per fixer leaned the other way: Claude Opus 4.6 solved two more tasks, but p=0.625p=0.625, well below the pre-declared decision bar, while inverting the calibration prior. Both were therefore run in full. The complete paired comparison resolved the question: Claude Opus 4.6 beats GPT-5.2 by ++7.14 percentage points on the same 266 tasks, with 44 tasks solved only by Claude Opus 4.6 versus 25 solved only by GPT-5.2 (exact McNemar test [37], p=0.029p=0.029). One further observation from the solo baselines deserves note. Kimi K2.5 solo resolves 149 of 266 tasks (56.02%) at $0.106 per task, dominating GPT-5.2 solo (139/266, 52.26%, $0.570 per task) on both axes, gaining ++3.76 percentage points of accuracy at roughly one fifth of the per-task cost. 7.2 Component Analysis Localization quality. TABLE V: SuperScout-7B’s file localization on SWE-bench Pro (Python-266). Per-task mean recall and precision against the gold patch’s file set. Spontaneous and forced handoffs are reported separately. All numbers are from a single sampled draw at temperature 0.90.9 (no majority vote). Handoff type Recall Precision All-gold Overall (n=266n=266) 0.566 0.821 24.8% Spontaneous (n=206n=206) 0.586 0.833 28.2% Forced (n=60n=60) 0.499 0.780 13.3% Table V reports the quality of SuperScout-7B’s file localization. Against a mean of 3.44 gold files per task, the searcher names a median of 2.0 files, achieving per-task mean recall of 0.566 and precision of 0.821; in 24.8% of tasks every gold file appears in the handoff. Spontaneous handoffs (n=206n=206) localize substantially better than forced ones (n=60n=60) across the board: recall 0.586 versus 0.499, precision 0.833 versus 0.780, and all-gold rate 28.2% versus 13.3%. This gap is unsurprising: tasks that exhaust the turn budget are typically harder. It underscores why the two handoff types are never pooled when reporting component behavior (per-repository detail in Appendix D). Verify-then-strip guard. Of 266 handoffs, 249 included a claimed verified reproduction of the bug. Replaying every claim inside the task sandbox revealed that only 50 of them (20%) were genuine, while 174 (70%) were demonstrably false. The guard stripped every false claim before any fixer saw it. Forced handoffs overclaimed more aggressively: just 9% of their reproduction claims were genuine, compared with 22% for spontaneous handoffs. These benchmark rates are worse than those observed during calibration (32% genuine, 56% false; §8), consistent with harder tasks producing more overclaiming. The guard neutralizes this overclaiming before any fixer sees it: 174 fixer prompts were stripped of confident misinformation that would otherwise have been treated as ground truth (full outcome classes in Appendix C); we did not run a pass-through arm, so the guard’s contribution to the headline is an inference from this claim census rather than a measurement (§9). Protocol findings. The auto-submit-at-cap mechanism rescued 24 of Claude Opus 4.6’s 158 solves and 33 of GPT-5.2’s 139. Without it, both frontier models’ solve rates would read roughly 9–13 percentage points lower. The 50-call cap binds often: it was reached in 36.5% of GPT-5.2 attempts, 17.3% of Claude Opus 4.6 attempts, and 48% of Kimi K2.5 attempts. The $2 cost cap bound only 5 times (all Claude Opus 4.6, maximum $2.09). The gold-control census confirmed 265 of 266 tasks (§6) (full cap statistics in Appendix E). Router behaviour. 0.10.10.20.20.30.30.40.40.50.50.60.60.70.70.80.80.90.91.01.00252550507575100100θ=0.30θ=0.30P(solve)P(solve) for the cheapest fixer (Kimi K2.5)taskstask-text featuresrouting head (blend)hidden-state features Figure 6: Two feature spaces, two very different confidence distributions at the routing gate (θ=0.30θ=0.30). Histograms of P(solve)P(solve) for Kimi K2.5 (cheapest fixer) across all 266266 tasks. The task-text head spans nearly the full unit interval (sd 0.200.20), while the hidden-state head concentrates tightly (sd 0.070.07, support entirely above θ). Their blend, the routing head, gates 263263 of 266266 tasks to the cheapest fixer. The router sends 263 of 266 tasks to Kimi K2.5: the first fixer in cheap-to-expensive order whose blended solve probability clears θ=0.30θ=0.30 receives the task. Figure 6 reveals how the two underlying feature spaces distribute that probability. The task-text head spreads wide (sd 0.200.20, support 0.080.08–0.970.97), while the hidden-state head concentrates tightly (sd 0.070.07, support 0.400.40–0.740.74) and sits entirely above the threshold. Because that support lies entirely above θ, part of the hidden-state head’s effect at the gate is mechanical: blending shrinks the text head’s predictions toward the state head’s near-constant mean, which by itself admits more tasks to the cheapest fixer. We did not run a matched shrinkage control, so the feature’s informational contribution cannot be fully separated from this threshold-shifting effect; the held-out separation gap (AUC 0.6000.600 versus 0.5610.561 for task text at N=99N=99, Appendix F) is suggestive rather than decisive. The hidden state’s contribution is not boosting accuracy per se but shifting when the cheap model is trusted, consistent with its cost-side rather than accuracy-side value observed during calibration (§8). 7.3 Onboarding a New Fixer Step 1Step 2Step 3 Collect outcomes ×✓ ×✓ 25–50 public per-task outcomes Average into a résumé solvedfailed two centroids ++ a base rate Routable at once existing fixerexisting fixerexisting fixernew fixer the router reads it searcher, embedder, scorer: all frozen, nothing retrains Figure 7: Onboarding a fixer costs a résumé, not a training run. A short public outcome record averages into two centroids and a base rate, and the router reads that résumé at inference time. No component is retrained. Adding a new fixer requires no retraining. A fixer’s résumé consists of two averaged embeddings (one over its solved tasks, one over its failed tasks) plus a base solve rate, all buildable from 25–50 public per-task outcomes (Table I). The embedder is frozen infrastructure and SuperScout-7B never retrains; the router simply reads each new résumé at inference time (Figure 7). This design keeps the pool extensible: a fixer released tomorrow can be routed to as soon as a short public-benchmark run produces its outcome record. Appendix H bounds what such public outcomes can carry: on our fresh calibration tasks every fixer’s public rate drops and the public ordering does not survive. A résumé built from public outcomes is therefore a cold-start device, adequate for admitting a new fixer to the pool but not for fine-grained ranking, and the thresholds used in this paper were calibrated on freshly measured labels for exactly this reason. 8 Calibration on Fresh Tasks 8.1 The Label Run SuperScout’s router assigns each incoming task to a fixer on the basis of per-task résumés and cost thresholds (§4). Both mechanisms require outcome labels: which fixer solved which task, and at what price. Public leaderboard data supplied these labels for several languages but contained zero usable Python outcomes for any fixer in the pool. Benchmark contamination policies lock those results away, and Appendix H documents why leaderboard-derived labels mislead when applied outside their original distribution. We therefore collected fresh labels: 100 Python bugs drawn from 2026 repositories, each attempted by all four fixers twice, once cold and once with SuperScout-7B’s handoff injected (99 of the 100 yielded a valid paired comparison). This paired design let every task serve as its own control, isolating the handoff’s effect from task difficulty. The same 100 tasks exposed a problem with SuperScout-7B’s reproduction claims. Replaying those claims against ground truth revealed that 32% were genuine and 56% demonstrably false. The verify-then-strip gate described in §4 was built as a direct consequence, before any benchmark contact. 8.2 The Handoff Redistributes Figure 8 shows the paired comparison; exact rates and intervals appear in Appendix G. Confidence intervals here and in the appendix are 95% percentile intervals from a bootstrap resampling tasks [38]. The handoff lifted the three weaker fixers: Claude Opus 4.6 rose from 48.5% to 53.5% (+5.1+5.1 p, p=0.125p=0.125), Kimi K2.5 from 52.5% to 56.6% (+4.0+4.0 p, p=0.481p=0.481), and Gemini 3 Flash from 55.6% to 57.6% (+2.0+2.0 p, p=0.774p=0.774). The strongest fixer, GPT-5.2, moved in the opposite direction, dropping from 60.6% to 56.6% (−4.0-4.0 p, p=0.424p=0.424). Pooled across all four, the effect is +1.8+1.8 percentage points with a 95% confidence interval of [−1.0,+4.5][-1.0,+4.5] and p=0.401p=0.401, not statistically significant. At N=99N=99 every per-fixer test is underpowered; the confidence intervals, not the point estimates, carry the information. What the pattern does establish is that fixer rankings survive handoff conditioning, with only one adjacent flip falling within the confidence interval. That stability is what the router’s résumé assumption requires. The pattern is consistent with redistribution rather than addition: solving ability shifts toward the cheaper models, which is exactly the shape cost-aware routing needs, though no single contrast reaches significance. The redistribution pattern recurs on the benchmark’s router-selected subset, though that comparison is uncontrolled and serves only as corroboration; exact numbers appear in Appendix I. 4545%5050%5555%6060%6565%Claude Opus 4.6Kimi K2.5Gemini 3 FlashGPT-5.2pooled+5.1+5.1+4.0+4.0+2.0+2.0−4.0-4.0+1.8+1.8Solve rate (paired, N=99N=99)solowith handoff Figure 8: The handoff pattern is redistributive, not additive. Paired comparison (N=99N=99 tasks, four fixers). Open circles: solo rates; filled markers: with-handoff. Horizontal bars span the 95% CI of each paired delta, anchored at the solo rate. The three weaker fixers gain; the strongest loses. All deltas are directional only (p>0.05p>0.05 at this sample size). 8.3 The Router Design Space We compared four input-feature variants under an offline simulation of the capped cost protocol, pinning each to the same matched solve rate so that the bars in Figure 9 differ only in spend. Variant A, using task text alone, saves 30.5% on total cost per solve at a matched solve rate. Adding SuperScout-7B’s hidden states produces variant C, the routing blend, raising the saving to 34.3%. The two variants that incorporate handoff-text features tell a sharply different story: variant B collapses to 8.0% and variant D to 9.4%. The lesson is concrete. The searcher’s internal state helps the router, while the handoff’s text helps the fixers; feeding the handoff memo to the router actively hurts. Supporting signals reinforce this split. As an outcome predictor, the hidden state reaches an AUC of 0.600 (fold-stable in four of five folds), while handoff text manages only 0.510. A simple logistic regression beat a multi-layer perceptron in every comparison, so the simpler scorer shipped. The full grid appears in Appendix F. The C-versus-A cost gap, a different quantity from the AUC separation, holds in only three of five folds, and the gate-level caveat of §7 applies: at deployment the hidden state’s informational contribution cannot be fully separated from its threshold-shifting effect. On the calibration labels, the cost-aware router matched the strongest fixer’s solve rate at 29% lower total cost per solve ($0.49 versus $0.69). Atask textB++handoffC++stateD++both0%1010%2020%3030%4040%deployedrouterhandoff-texthandoff-text30.530.5%8.08.0%9.49.4%34.334.3%Held-out cost saving Figure 9: Which router features buy cost savings. Held-out cost saving against an always-best-model anchor, all variants pinned to the same solve rate (.606.606). Adding SuperScout-7B’s hidden states (variant C, the routing blend) lifts savings from 30.5%30.5\% to 34.3%34.3\%. Variants with handoff-text features (B, D) collapse to 88–9%9\%. The C-vs-A gap is fold-fragile at N=99N=99. Calibration settled four design choices: the routing blend (variant C), the matched-point threshold (θ=0.30θ=0.30), logistic regression as the scoring function, and blanket handoff injection. It left one question open. GPT-5.2 was the strongest fixer on these fresh tasks, a ranking the benchmark later inverted (§7). 9 Limitations Our headline result rests on a single benchmark’s Python slice, comprising 266 tasks. While the searcher’s localization ability transfers across languages (§5), that evidence covers the searcher alone, not the full routed system. Cross-benchmark generality remains untested, and the ablation finding, that blanket assignment to the cheapest fixer ties the routed system, is a property of this task pool; it need not transfer to other pools or benchmarks. The 266 tasks themselves span three repositories, with per-repository recall ranging from 0.472 to 0.671 (Appendix D); the effective diversity is closer to three codebases than to 266 independent draws. At n=266n=266 the headline is a one-task edge, which is precisely why we claim a match and nothing stronger. The robustness evidence is structural: both the routed system and the no-router ablation sit +3.60+3.60 percentage points above the blind-mixing line, so on this pool the margin over random traffic splitting is carried by the handoff rather than by the routing decision. Like the ablation finding itself, this attribution is specific to this pool and benchmark and can change with either. The calibration label run uses n=100n=100 tasks. No handoff contrast in this paper reaches statistical significance, on the calibration labels or on the benchmark subset; the mechanism rests on the consistency of the pattern across two independent settings, not on any single test. The benchmark-side handoff comparisons themselves come from router-selected subsets rather than random assignment, so they carry observational caveats. The controlled reference point is the paired calibration study on held-out tasks (§8). Contamination control is one-sided. The searcher’s training corpus excludes a 23-repository blocklist covering every SWE-bench Pro and SWE-bench Verified repository, enforced by a programmatic gate on the frozen training file. No such control is possible on the fixer side: the three evaluation repositories are public, we have no visibility into the four fixers’ training data, and differential fixer-side contamination would distort the relative solve rates that the résumés and the calibration are fit to. We also did not run a pass-through arm in which fixers receive the handoff with its unverified reproduction claims intact, so the guard’s contribution to the headline solve rate is asserted from the claim census (§7) rather than measured; a pass-through ablation is future work. All costs are provider list prices recorded at measurement time. Prices drift, and one fixer was served as a quantized build through a pinned provider, so cost conclusions are snapshots rather than stable constants. The cost-accounting asymmetry described in §6 (all-in for the system versus API-only for solo baselines) is conservative in our disfavor, but it is still an accounting choice that readers should weigh. SuperScout-7B achieves an overall localization recall of 0.566, a moderate figure drawn from a single sampled inference pass whose variance is uncharacterized. The system-level result shows that this recall suffices for effective routing; it does not establish SuperScout-7B as a state-of-the-art localizer. All absolute solve rates reported here are specific to the capped, silent-budget, auto-submit evaluation tier defined in §6. Under other tiers these numbers would change. Our protocol comparisons (§7) show that such tier choices are worth percentage points of solve rate, underscoring the relativity of any single absolute figure. Finally, the hidden-state features that drive routing are tied to the exact SuperScout-7B checkpoint. Retraining the searcher would require re-extracting hidden states and refitting the router heads. The zero-retrain onboarding property applies to adding new fixers, not to changing the searcher itself. 10 Conclusion SuperScout shows that engaging with the repository before dispatch pays, though not where we expected: the verified handoff, not the routing decision, carries the result. A small trained searcher, SuperScout-7B, explores the repository and writes a structured handoff whose reproduction claims are verified in a sandbox (false claims are stripped), and whose hidden states feed a résumé router that selects the downstream fixer. On the benchmark’s full Python slice, this pipeline solves 159 of 266 problems, matching the best single frontier model (158 of 266) at about a fifth of the total cost per solve, with the searcher’s own compute contributing a rounding error to the budget. Two mechanism-level findings underpin this result. The handoff appears to redistribute rather than add ability, lifting the three cheaper fixers while slightly hurting the strongest, a directional pattern at N=99N=99. The searcher’s hidden states, meanwhile, change which tasks the router trusts to the cheap model, yet feeding the handoff’s text directly to the router hurts it. Because adding a new fixer requires no retraining, the system is built for a model market that changes quarter by quarter. Natural next steps include wider benchmark coverage across languages and problem domains. Equally valuable would be measuring how quickly a newly released fixer can be onboarded into the live pipeline without retraining the searcher or the router. References Jimenez et al. [2024] C. E. Jimenez et al., “SWE-bench: Can language models resolve real-world GitHub issues?” in International Conference on Learning Representations (ICLR), 2024, arXiv:2310.06770. Ong et al. [2024] I. Ong et al., “RouteLLM: Learning to route LLMs with preference data,” 2024, arXiv:2406.18665. Chen et al. [2023] L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” 2023, arXiv:2305.05176. Deng et al. [2025] X. Deng et al., “SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks?” 2025, arXiv:2509.16941. Yang et al. [2024] J. Yang et al., “SWE-agent: Agent-computer interfaces enable automated software engineering,” in Advances in Neural Information Processing Systems (NeurIPS), 2024, arXiv:2405.15793. Xia et al. [2024] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying LLM-based software engineering agents,” 2024, arXiv:2407.01489. Yang et al. [2025a] J. Yang et al., “SWE-smith: Scaling data for software engineering agents,” 2025, arXiv:2504.21798. Pan et al. [2025] J. Pan et al., “Training software engineering agents and verifiers with SWE-Gym,” in International Conference on Machine Learning (ICML), 2025. Jain et al. [2025] N. Jain, J. Singh, M. Shetty, L. Zheng, K. Sen, and I. Stoica, “R2E-Gym: Procedural environments and hybrid verifiers for scaling open-weights SWE agents,” in Conference on Language Modeling (COLM), 2025, arXiv:2504.07164. Xie et al. [2025] C. Xie et al., “SWE-Fixer: Training open-source LLMs for effective and efficient GitHub issue resolution,” 2025, arXiv:2501.05040. Chen et al. [2025] Z. Chen et al., “LocAgent: Graph-guided LLM agents for code localization,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, 2025, p. 8697–8727, arXiv:2503.09089. Reddy et al. [2025] R. G. Reddy et al., “SweRank: Software issue localization with code ranking,” 2025, arXiv:2505.07849. Ding et al. [2024] D. Ding et al., “Hybrid LLM: Cost-efficient and quality-aware query routing,” in International Conference on Learning Representations (ICLR), 2024, arXiv:2404.14618. Shnitzer et al. [2023] T. Shnitzer et al., “Large language model routing with benchmark datasets,” 2023, arXiv:2309.15789. Zhuang et al. [2025] R. Zhuang, T. Wu, Z. Wen, A. Li, J. Jiao, and K. Ramchandran, “EmbedLLM: Learning compact representations of large language models,” in International Conference on Learning Representations (ICLR), 2025, arXiv:2410.02223. Wang et al. [2026] C. Wang et al., “ICL-router: In-context learned model representations for LLM routing,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 39, 2026, p. 33 413–33 421, arXiv:2510.09719. Son et al. [2026] S. Son, S. Yoon, J. Tang, S. Wang, L. Wolf, and I. Bogunovic, “SWE-Router: Routing in multi-turn agentic software engineering tasks,” 2026, presented at the 5th Deep Learning for Code Workshop, ICML 2026; arXiv:2607.00053. He et al. [2026] Q. He et al., “CodeRescue: Budget-calibrated recovery routing for coding agents,” 2026, arXiv:2607.19338. Raj et al. [2026] R. Raj, S. Kundu, S. Banerjee, D. Joshi, I. Vohra, and T. Krishna, “TRACE-Router: Task-consistent and adaptive online routing for agentic AI,” 2026, arXiv:2607.22465. Wei et al. [2026] Y. Wei et al., “Toward training superintelligent software agents through self-play SWE-RL,” in International Conference on Machine Learning (ICML), 2026, arXiv:2512.18552. OpenAI [2024] OpenAI, “Introducing SWE-bench Verified,” https://openai.com/index/introducing-swe-bench-verified/, 2024, accessed 2026-08-03. Yang et al. [2025b] J. Yang et al., “SWE-bench multilingual,” https://w.swebench.com/multilingual.html, 2025, dataset released alongside SWE-smith. Hui et al. [2024] B. Hui et al., “Qwen2.5-Coder technical report,” 2024, arXiv:2409.12186. Zhang et al. [2025] Y. Zhang et al., “Qwen3 Embedding: Advancing text embedding and reranking through foundation models,” 2025, arXiv:2506.05176. OpenAI [2025a] OpenAI, “gpt-oss-120b & gpt-oss-20b model card,” 2025, arXiv:2508.10925. Ahmad et al. [2026] W. U. Ahmad, N. Ludwig, S. Majumdar, and B. Ginsburg, “Open-SWE-Traces: Advancing dual-mode multilingual distillation for software engineering agents,” 2026, arXiv:2606.16038; dataset: https://huggingface.co/datasets/nvidia/Open-SWE-Traces. Trofimova et al. [2025] M. Trofimova, A. Shevtsov, I. Badertdinov, K. Pyaev, S. Karasik, and A. Golubev, “OpenHands Trajectories with Qwen3-Coder-480B-A35B-Instruct,” Nebius blog / HuggingFace dataset, 2025, dataset: https://huggingface.co/datasets/nebius/SWE-rebench-openhands-trajectories. Badertdinov et al. [2025] I. Badertdinov et al., “SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. Ludwig et al. [2026] N. Ludwig, W. U. Ahmad, S. Majumdar, and B. Ginsburg, “From SWE-ZERO to SWE-HERO: Execution-free to execution-based fine-tuning for software engineering agents,” 2026, arXiv:2604.01496; dataset: https://huggingface.co/datasets/nvidia/SWE-Hero-openhands-trajectories. Wang et al. [2025] X. Wang et al., “OpenHands: An open platform for AI software developers as generalist agents,” in International Conference on Learning Representations (ICLR), 2025, arXiv:2407.16741. Hu et al. [2022] E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations (ICLR), 2022, arXiv:2106.09685. Shao et al. [2024] Z. Shao et al., “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,” 2024, arXiv:2402.03300. Anthropic [2026] Anthropic, “Introducing Claude Opus 4.6,” https://w.anthropic.com/news/claude-opus-4-6, 2026, accessed 2026-08-03. OpenAI [2025b] OpenAI, “Introducing GPT-5.2,” https://openai.com/index/introducing-gpt-5-2/, 2025, accessed 2026-08-03. Kimi Team [2026] Kimi Team, “Kimi K2.5: Visual agentic intelligence,” 2026, arXiv:2602.02276. Google DeepMind [2025] Google DeepMind, “Gemini 3 Flash model card,” https://deepmind.google/models/model-cards/gemini-3-flash/, 2025, accessed 2026-08-03. McNemar [1947] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, p. 153–157, 1947. Efron [1979] B. Efron, “Bootstrap methods: Another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, p. 1–26, 1979. Appendix A The Prior Routing Audit The cost-routing premise of §3 rests on a zero-cost replay of published per-task outcome matrices. Table VI records the structural statistics behind that replay: set containment, the unique-solver shell, and the gap between the best learned router and always picking the strongest model, across three benchmarks and matched four-model pools. TABLE VI: Why we do not route for accuracy. Zero-cost audit of published per-task outcome matrices on matched four-model pools. Containment measures solve-set nesting; the unique-solver shell is the share of routable tasks only one model solves. On all three benchmarks, no learned router beat always calling the strongest model. This is background motivation, not a system result. Verified Multiling. Pro Best single model 0.768 0.726 0.449 Mean containment 0.941 0.912 0.773 Unique-solver shell 21.5% 20.8% 30.8% Best router −- best model −0.006-0.006 +0.010+0.010 −0.004-0.004 p 0.66 0.24 0.90 Appendix B Router Threshold Sweep Table VII sweeps the routing threshold θ that the calibration’s matched-point rule selects. At θ=0.25θ=0.25 every task clears the cheapest fixer’s gate and the router degenerates into the no-router ablation: 159 solves at $0.227 per solve, identical to always calling Kimi K2.5. Raising the threshold to the operating point θ=0.30θ=0.30 diverts three tasks to Gemini 3 Flash at a marginal cost of $0.003 per solve, without changing the solve count. Above the operating point, the router’s growing caution pushes progressively more tasks onto expensive fixers: cost per task rises monotonically while the solve count ceases to be exactly measurable, because the diverted tasks lack a with-handoff outcome on their new fixer. We report those rows as bracketing intervals rather than point estimates (the table note details the convention). On this benchmark, routing collapses to a cost-allocation decision: no threshold examined buys additional solves beyond the 159 that the cheapest fixer already delivers. TABLE VII: The gate threshold sweep. Raising θ pushes tasks off the cheapest fixer onto more expensive ones: cost per task rises monotonically while the solve count ceases to be exactly measurable, since diverted tasks lack a with-handoff outcome on their new fixer. The operating point θ=0.30θ=0.30 routes all but three tasks to Kimi K2.5 and is the last threshold at which every number is exact. Routing split (tasks) θ Kimi K2.5 Gemini 3 Flash GPT-5.2 Claude Opus 4.6 Solves $/task $/solve 0.250.25 266 0 0 0 159 $0.136 $0.227 0.30†0.30 263 3 0 0 159 $0.137 $0.230 0.350.35 236 19 10 1 [149, 169] $0.154 $0.272∗ 0.400.40 203 34 28 1 [132, 179] $0.178 $0.337∗ 0.500.50 124 34 72 36 [94, 196] $0.334 $0.672∗ †Operating point: the threshold used for every SuperScout number in the paper. ∗$/solve from a solo-filled point estimate of the solve count, since tasks routed to a fixer with no measured with-handoff outcome take that fixer’s solo outcome; rows whose solve count is not measured exactly report the bracketing interval instead, and no midpoint. Rows above the operating point send tasks that pass no gate to the most capable fixer (Claude Opus 4.6); at the operating point every task passes at least one gate, so the fall-through never fires. Solves are out of n=266n=266. All rows use the board’s cost convention: measured fixer API spend plus a pinned $5.13 of searcher GPU and sandbox infrastructure, amortized over 266 tasks. Appendix C Verify-then-Strip Outcome Classes §7 reports the headline guard statistics; the full failure-class census behind those numbers appears in Table VIII, broken down by outcome category and by handoff type. TABLE VIII: What the verify-then-strip sandbox found in 266 handoffs. Top: outcome class of each reproduction claim, replayed before any fixer saw it (5050 genuine, 174174 false and stripped). Bottom: split by handoff type; spontaneous handoffs are more than twice as likely to be genuine as forced ones (never pooled). Forced-handoff percentages use the 4343 of 6060 that claim a reproduction as their denominator. The label-run row shows the same measurement on an earlier, easier task set. Outcome of the replayed reproduction n Passed at base (claim stripped) 174 Genuinely failed (claim kept) 050 Claim not true 016 Import error 012 Non-zero exit, other 009 Missing file 003 No reproduction block 001 No command 001 Total 266 Handoff set Claiming Genuine Stripped Spontaneous (n=206n=206) 206 22% 68% Forced (n=60n=60) 043 09% 79% All Pro handoffs (n=266n=266) 249 20% 70% Label-run baseline 091 32% 56% Appendix D Localization by Repository The per-task mean localization quality reported in Table V pools three repositories that differ substantially in how many files their gold patches touch. Table IX disaggregates by repository and provides the solved-versus-unsolved cross-tab behind the main-text numbers. TABLE IX: Localization by repository, and against outcome. Top: SuperScout-7B’s file-localization quality per repository; spread tracks the number of gold files per task. Bottom: quality split by whether the system solved the task. Better localization correlates weakly with solving (+0.024+0.024 recall), so localization is not a gate on the outcome. Repository Recall Prec. All-gold Gold files ansible (n=96n=96) 0.472 0.846 10.4% 3.80 openlibrary (n=91n=91) 0.574 0.766 28.6% 3.58 qutebrowser (n=79n=79) 0.671 0.853 38.0% 2.84 All (n=266n=266) 0.566 0.821 24.8% 3.44 Outcome Recall Prec. All-gold Solved (n=159n=159) 0.575 0.830 26.4% Unsolved (n=107n=107) 0.552 0.808 22.4% Appendix E Capped-Tier Statistics The protocol findings in §7 quote selected cap-binding rates and auto-submit rescue counts. Table X gives the complete picture for all three fixer arms. TABLE X: How often the evaluation caps bound, and what auto-submission was worth. All arms ran the official capped tier (5050 turns, $2.00 per attempt). The turn cap is the binding constraint; the cost cap bound only five times. Auto-submission rescued 2424 of Claude Opus 4.6’s solves and 3333 of GPT-5.2’s; without it, both anchors’ rates would fall roughly 99–1313 points. The gold-patch control passes on 265265 of 266266 tasks. GPT-5.2 Claude Opus 4.6 Kimi K2.5 Hit 50-turn cap 36.5% 17.3% 48% Hit $2 cost cap 0 5 0 Max attempt cost $1.59 $2.09 — Auto-submitted at cap 94 48 126 Solves rescued 33 24 — Solves 139 158 149 Rate without rescue 39.8% 50.4% — Appendix F Router Design Space, Full Grid The calibration analysis in §8 summarizes the four feature variants under the logistic-regression scorer. Table XI extends that comparison to include the MLP head and provides the outcome-separation (AUC) measurements the cost numbers rest on. TABLE XI: The complete router design space. Top: held-out cost saving for every feature-set × scorer combination, all pinned to solve rate .606.606. Adding hidden states helps under both scorers; handoff-text features collapse savings to 88–9%9\%. The MLP halves every variant’s saving. At N=99N=99 the best variant is only 3/53/5 fold-stable. Bottom: the hidden state separates solved from failed tasks at AUC .600.600, while handoff-text embedding sits at chance. Router features Saving Folds Brier LR MLP A text only 30.5% 15.4% 3/5 0.270 B ++ handoff text 08.0% 05.3% 1/5 0.254 C ++ hidden state (dep.) 34.3% 18.7% 3/5 0.249 D ++ both 09.4% 04.2% 4/5 0.251 Routing representation Pooled AUC Stable Hidden state (pre-decode, layer −4-4) 0.600 4/5 Task text 0.561 — Handoff text (prereg. baseline) 0.510 — Appendix G Redistribution, Exact Numbers Figure 8 in the main text visualizes the paired handoff ablation; Table XII gives the exact per-fixer rates, deltas, confidence intervals, discordant-pair counts, and p-values behind that figure. TABLE XII: The handoff pattern is redistributive, not additive. Exact per-fixer rates, deltas, and p-values behind Figure 8 (9999 paired tasks, 396396 attempts). The three weaker fixers gain and the strongest loses; pooled effect is +1.8+1.8 p with CI including zero. No fixer reaches p<0.05p<0.05; every row is directional only. b counts rescued tasks, c counts broken ones. Fixer Solo Handoff Δ (95% CI) b/cb/c p Claude Opus 4.6 48.5 53.5 +5.1+5.1 [0.00.0, 10.110.1] 6/1 0.125 Kimi K2.5 52.5 56.6 +4.0+4.0 [−4.0-4.0, 12.112.1] 11/7 0.481 Gemini 3 Flash 55.6 57.6 +2.0+2.0 [−5.1-5.1, 9.19.1] 7/5 0.774 GPT-5.2 60.6 56.6 −4.0-4.0 [−11.1-11.1, 3.03.0] 5/9 0.424 pooled 54.3 56.1 +1.8+1.8 [−1.0-1.0, 4.54.5] 29/22 0.401 Appendix H Benchmark-Versus-Fresh Calibration Receipts The router’s résumés are built from public per-task outcomes, so the router inherits whatever biases those outcomes carry. Buying fresh labels was not a design preference but a measured necessity. Table XIII records the calibration receipts that motivated that decision: every fixer’s public rate compared with its freshly measured rate under a single controlled harness. TABLE XIII: Why we bought fresh labels: public rates do not transfer to unseen tasks. Each fixer’s public multilingual rate versus its freshly measured rate on 100100 post-cutoff Python tasks under one harness. Every fixer drops; the ordering does not survive. Rates use the full 100100-task solo denominator; the paired analysis in Appendix G uses the 9999 tasks with both arms. These are calibration receipts, not a benchmark claim. Fixer Public Fresh (95% CI) Gap Rank Gemini 3 Flash 0.727 0.550 [.450, .650] −17.7-17.7 #1 → #2 Claude Opus 4.6 0.720 0.480 [.380, .570] −24.0-24.0 #1 → #4 Kimi K2.5 0.673 0.520 [.420, .610] −15.3-15.3 #2 → #3 GPT-5.2 0.667 0.610 [.510, .700] −5.7-5.7 #3 → #1 Appendix I Handoff Effect on Router-Selected Benchmark Subset The redistribution pattern observed under controlled conditions (§8.2) also appears in the benchmark evaluation, where the tasks the router sends to Kimi K2.5 were available both with and without a handoff. Table XIV reports that comparison. The subset was selected by the router rather than randomly assigned; it remains measured, uncontrolled corroboration rather than a controlled experiment. TABLE XIV: The handoff effect on the deployment path. The tasks routed to Kimi K2.5 were also run without a handoff, giving a same-task comparison: the workhorse gains 3.83.8 p on 263263 tasks. b/cb/c counts tasks solved only with the handoff versus only solo; the exact McNemar test on those discordant pairs gives p=0.245p=0.245. The router selected which tasks enter the subset, not the within-task contrast, so the pairing is intact, but the subset is not randomly assigned and this table remains corroboration, not a controlled measurement. The three tasks routed elsewhere have no solo arm for their fixer and are omitted. Fixer Routed n Handoff Solo Δ b/cb/c p Kimi K2.5 263 60.1% 56.3% +3.8+3.8 35/25 0.245 Appendix J Nine-Language Localization Detail Figure 5 in the main text shows the per-language F1F_1 bars; Table XV gives the full precision, recall, F1F_1, and all-gold counts behind that figure, split by trained and never-trained language pools. TABLE XV: Localization transfers to languages SuperScout-7B never saw in training. Per-language precision, recall, and F1F_1 of spontaneous handoffs against gold patch files. The six never-trained languages outperform the three trained ones (F1F_1 0.6300.630 vs. 0.4550.455). JavaScript and TypeScript are genuine soft spots; the TypeScript and C++ cells rest on 1212 assigned tasks each, with 77 and 88 spontaneous handoffs analyzed († ). Multi-file recall is weak across all languages (0.260.26–0.340.34). Forced handoffs excluded; searcher alone, one sampled draw. Language n P R F1F_1 All-gold Go 27 0.605 0.542 0.546 13/27 JavaScript 26 0.500 0.365 0.389 07/26 TypeScript† 07 0.333 0.357 0.343 02/7 trained pool 60 0.528 0.444 0.455 22/60 Java 27 0.722 0.574 0.612 11/27 Ruby 36 0.833 0.661 0.701 18/36 Rust 20 0.800 0.626 0.667 10/20 PHP 33 0.697 0.632 0.630 17/33 C 18 0.546 0.435 0.435 06/18 C++† 08 0.688 0.813 0.708 06/8 never-trained 142 0.731 0.613 0.630 68/142 Appendix K Vault Decoding Comparison The decoding finding reported in §5.2 is demonstrated on a 100-task dial subset; Table XVI gives the full A/B comparison on the 450-task held-out vault that established the finding. TABLE XVI: Sampled decoding is what makes the searcher commit. The same checkpoint on 450450 held-out issues, greedy versus temperature 0.90.9. Find rate nearly triples, from higher commitment (72%72\% vs. 21%21\%) against an 18% drop in per-handoff recall. The greedy arm lost 2424 episodes to infrastructure timeouts; the 2.65×2.65× figure adjusts for this. The vault is excluded from all training data. Metric (n=450n=450) Sampled Greedy Find rate 0.306 0.110 95% CI [.273, .337] — timeout-matched ratio 2.65×2.65× (n=426n=426) Emission rate 0.718 0.213 Handoffs emitted 323 096 Recall per handoff 0.426 0.517 Precision 0.643 0.726 Files per handoff 1.93 2.05 All-gold handoffs 071 027 Malformed 000 000 Appendix L Training Hyperparameters §5 describes the supervised recipe at the level relevant to the contribution; Table XVII records the full configuration as realized in the actual training run. TABLE XVII: Supervised fine-tuning configuration, as run. Rank-6464 LoRA on every linear projection, trained in bf16 for two epochs on packed 3232k-token blocks at 91.7%91.7\% fill; loss is on assistant turns only. All values are the realized run, not the planned recipe. The complete run cost $86 of GPU time and peaked at 24.224.2 GB. Setting Value Base model Qwen2.5-Coder-7B-Instruct Adaptation LoRA, r=64r=64, α=128α=128 Target modules all linear projections (no embedding, no output head) Dropout 0 Precision bf16 Learning rate 1×10−41×10^-4, cosine to 10%10\% Warmup 3% of steps Epochs 2 (3,182 steps) Context 32k, packed blocks Micro-batch 1×321×32k Gradient accum. 8 Gradient ckpt. on Seed 3407 Tokens per epoch 382.4M (91.7% fill) Wall clock 31.8 h Peak memory 24.2 GB Final loss 0.114 Cost $86 Reinforcement-learning rig. We built and validated a GRPO training rig before concluding that RL was unnecessary (§5). The rig used a two-GPU topology: one GPU trained while the other generated rollouts, with kill-and-resume checkpointing so that preempted runs lost no more than a single episode. Exact backpropagation through long episodes was verified against ground truth before any paid run began. The reward function was validated separately, by replaying candidate reward assignments against thousands of previously saved episodes and confirming agreement before committing GPU spend. 50 clean steps then ran flat, as reported in §5, and the rig was shelved in favor of the decoding fix. Appendix M A Verified Handoff, Before and After the Gate Figure 10 reproduces one real spontaneous handoff from the SWE-bench Pro evaluation, shown in the exact post-strip form the fixer received. This handoff’s reproduction claim was replayed in the task’s own sandbox and genuinely failed, so the claim was forwarded intact rather than deleted. <files> qutebrowser/utils/utils.py qutebrowser/utils/qtutils.py qutebrowser/browser/downloads.py qutebrowser/mainwindow/tabbedbrowser.py </files> <repro> command: cd /app && python test_issue.py reproduced: true observed: OBSERVATION: SUCCESS: interpolate_color exists in utils Interpolation at 50%: 128:0:128 Traceback (most recent call last): File "/app/test_issue.py", line 26, in <module> interp = qtutils.interpolate_color AttributeError: module ’qutebrowser.utils.qtutils’ has no attribute ’interpolate_color’ </repro> <notes> Search surfaced definitions of interpolate_color in qutebrowser/utils/utils.py (line 260) and references to utils.interpolate_color in qutebrowser/browser/downloads.py (line 563) and qutebrowser/mainwindow/tabbedbrowser.py (lines 866, 883). No occurrences of interpolate_color were found in qutebrowser/utils/qtutils.py. Grep for _get_color_percentage returned nothing. Attempted import of qutebrowser.utils.qtutils showed only top-level symbols, confirming the function is absent there. Running a custom script test_issue.py via cd /app && python test_issue.py raised AttributeError: module ’qutebrowser.utils.qtutils’ has no attribute ’interpolate_color’. No build or test tooling issues observed; commands executed directly with python. No false leads beyond missing function in qtutils. </notes> STOP Figure 10: One real handoff, exactly as the fixer received it. A spontaneous handoff from the SWE-bench Pro evaluation, reproduced verbatim after the verify-then-strip stage. This reproduction claim was genuine and forwarded intact; 174174 of 249249 claims were false and stripped. Line wrapping in the notes is ours; no content is added or redacted. Appendix N Compute and Cost Disclosure Table XVIII itemizes the measured spend across the three eras of this project: evaluation, label collection, and training. The one-off cost of building the system, comprising the label run (≈$370≈ 370), the fine-tuning run (≈$86≈ 86), and corpus note generation (≈$9≈ 9), totals roughly $465. Against the $0.62 per-task saving over the best solo fixer ($0.757 API-only versus $0.137 all-in), that sunk cost amortizes in roughly 750 tasks. TABLE XVIII: Measured spend. Every figure is a measured value from the run ledgers. The evaluation era covers all five arms plus infrastructure; the label-run era covers fresh per-task outcomes; the training era is the single SFT run. SuperScout-7B’s entire evaluation contribution was $1.13 of GPU time (∼0.4 0.4 cents per task). Era totals are not summed. Item $ Evaluation era Probe (40 attempts) 19.24 GPT-5.2 solo arm 151.64 Claude Opus 4.6 solo arm 201.25 Kimi K2.5 solo arm 28.23 System arm 86.49 Smoke 1.32 Unrecorded partials ≈ 11 GPU (all pods) ≈ 5.9 Sandbox compute 40–55 era total ≈ 545–560 Label-run era Fixer API 340.03 GPU pod ≈ 4 Sandbox compute ≈ 25 era total ≈ 370 Training era Fine-tuning run ≈ 86 Corpus note generation ≈ 9

Instant research discovery