Paper deep dive
From BERT to Frontier Agents: Eight Years of Language-Model Progress, the Collapse of the Capability-Cost Curve, and the Rise of Task-Targeted Models
Pranav Kumar Kaliaperumal
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 8/17/2026, 4:03:21 AM
Summary
This paper reviews the evolution of language models from 2018 to 2026, highlighting the shift from encoder-based models like BERT to agentic systems. It documents a ~5.8x annual growth in solving real-world coding tasks (SWE-bench Verified) and a significant collapse in the capability-cost curve, where budget models like GPT-5.6 Luna match or exceed previous flagship performance at a fraction of the cost. The study finds that the 2026 frontier is fragmented by task, with different models leading in specific domains (e.g., Claude Opus 5 in frontend coding, GPT-5.6 Sol in terminal work). Additionally, it demonstrates that inference-time techniques such as self-consistency, confidence estimation, and model routing can significantly improve performance on fixed models.
Entities (15)
Relation Signals (11)
BERT → releasedby → Google
confidence 99% · In October 2018, Google released BERT
GPT-5.6 Luna → releasedby → OpenAI
confidence 99% · GPT-5.6 Luna, OpenAI’s budget tier
claude-opus-5 → releasedby → Anthropic
confidence 99% · Anthropic released Claude Opus 5
claude-opus-5 → leadsin → frontend_coding
confidence 97% · Claude Opus 5 leads in frontend coding
Claude-fable-5 → leadsin → repository_level_coding
confidence 97% · Claude Fable 5 excels at repository level coding
GPT 5.6 Sol → leadsin → terminal_tasks
confidence 97% · GPT 5 point 6 Sol dominates terminal tasks
GPT-5.6 Luna → haslowercostthan → GPT-5.5
confidence 96% · at one-fifth the token price
Kimi K3 → isopenweight → true
confidence 96% · a 2.8-trillion-parameter open-weight mixture-of-experts model
GPT-5.6 Luna → matchesperformanceof → GPT-5.5
confidence 95% · GPT-5.6 Luna... matches or beats the eleven-week-old GPT-5.5 flagship
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Between October 2018 and July 2026 AI models progressed from simple systems like BERT to massive agents that solve complex math and write software. The ability to resolve real coding issues improved by nearly six times per year since late 2024. During this time costs dropped sharply with OpenAIs budget model GPT 5 point 6 Luna matching flagship capabilities for just one to six dollars per million tokens beating older versions at a fraction of the price. Top performance is now split across specialized models as Claude Opus 5 leads in frontend coding Claude Fable 5 excels at repository level coding and GPT 5 point 6 Sol dominates terminal tasks. In a grade school math test using the Qwen 2 point 5 model basic methods solved 58 of 100 problems while advanced sampling solved up to 79. A confidence ranking tool correctly identified 47 right answers in its top 50 choices proving highly useful for sorting tasks with all research materials made fully public.
Tags
Links
- Source: https://arxiv.org/abs/2608.13675v1
- Canonical: https://arxiv.org/abs/2608.13675v1
Trouble viewing inline? Open PDF directly →
Full Text
42,669 characters extracted from source content.
Expand or collapse full text
From BERT to Frontier Agents: Eight Years of Language-Model Progress, the Collapse of the Capability–Cost Curve, and the Rise of Task-Targeted Models An empirical review with reproducible measurements, 2018–2026 Pranav Kumar Kaliaperumal prka5235@colorado.edu August 2026 Abstract Between October 2018 and July 2026, language models moved from BERT, a 340-million- parameter encoder that could barely generate text, to trillion-parameter agentic systems that solve International Mathematical Olympiad problems, repair software repositories, and operate desktop computers. We trace that progression and measure its practical consequences with public benchmarks, prices, and reproducible experiments. A log- linear fit to SWE-bench Verified estimates∼5.8× annual growth in the odds of resolving a real GitHub issue since October 2024. Over the same period, the capability–cost curve collapsed: GPT-5.6 Luna, OpenAI’s budget tier at$1/$6 per million input/output tokens, matches or beats the eleven-week-old GPT-5.5 flagship on most agentic and professional- work benchmarks in the vendor’s tables at one-fifth the token price. It also exceeds the August 2025 GPT-5 SWE-bench Verified score by ∼18 points in our reconstruction of an independent harness. Task-level results reveal a fragmented frontier: Claude Opus 5 leads human-preference frontend coding, Claude Fable 5 leads repository-level coding, GPT-5.6 Sol leads agentic terminal work, and Opus 5 quadrupled the ARC-AGI-3 record. A two-model router recovers the full per-benchmark gain of a six-model oracle on our comparison suite. We also run a traced grade-school-math experiment. After selecting the prompt, temperature, model size, and selector on a locked 16-item development split, we freeze a Qwen2.5-1.5B configuration and evaluate it once on 100 disjoint items. Greedy decoding solves 58/100; four-sample plurality and a prespecified verifier each solve 62/100 (paired exact McNemar p = 0.481), while the any-sample oracle reaches 79/100. A post-evaluation confidence model ranks the vote-at-four outputs with five- fold out-of-fold ROC AUC 0.833 and a Brier score of 0.147, versus 0.236 for a prevalence baseline. Its highest-confidence 50 predictions contain 47 correct answers, suggesting a useful triage signal rather than a new full-coverage accuracy gain. The repository includes all data, code, complete traces, tests, and compilation scripts. Keywords: large language models; evaluation; scaling; inference economics; task specializa- tion; self-consistency; confidence estimation; model routing Contents 1 Introduction3 1 arXiv:2608.13675v1 [cs.LG] 13 Aug 2026 2 Eight years in four paradigms3 2.1 The encoder era: BERT and the fine-tuning paradigm (2018–2019) . . . . . .3 2.2 Scale and in-context learning: GPT-3 (2020–2021) . . . . . . . . . . . . . . .4 2.3 Alignment and instruction following (2022–2023) . . . . . . . . . . . . . . . .4 2.4 Reasoning and agents (2024–2026) . . . . . . . . . . . . . . . . . . . . . . . .4 3 Data and methods4 3.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 3.2 Analysis methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 4 Results6 4.1 Capability growth: three benchmark eras . . . . . . . . . . . . . . . . . . . .6 4.2 The collapse of the capability–cost curve . . . . . . . . . . . . . . . . . . . . .7 4.3 The frontier fragmented by task . . . . . . . . . . . . . . . . . . . . . . . . . .8 5 Improving fixed models at inference time9 5.1 Sample, then aggregate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 5.2 Estimate confidence for selective use . . . . . . . . . . . . . . . . . . . . . . .11 5.3 Match reasoning effort to the workload . . . . . . . . . . . . . . . . . . . . . .11 5.4 Route work across specialized models . . . . . . . . . . . . . . . . . . . . . . .12 5.5 Reporting priorities for model builders . . . . . . . . . . . . . . . . . . . . . .12 6 Discussion: how models get good at specific tasks12 7 Conclusion13 2 1 Introduction In October 2018, Google released BERT, a bidirectional Transformer encoder that set the state of the art on eleven natural-language understanding tasks and pushed the GLUE bench- mark to 80.5% 1 . BERT could not write a coherent paragraph; it was a reading model, fine- tuned per task. In July 2026, Anthropic released Claude Opus 5 and reported 42/42 under its post-contest, no-tools grading protocol for the 2026 International Mathematical Olympiad problems; ARC Prize independently measured a roughly fourfold jump on ARC-AGI-3 2,3 . Between those two releases sit eight years in which the field’s dominant paradigm changed at least four times: from task-specific fine-tuning, to few-shot prompting of large pretrained decoders, to instruction tuning and human-feedback alignment, to reasoning models and tool-using agents. We ask three related questions. How did the field move from BERT to the 2026 frontier of GPT-5.6, Claude Fable 5 and Opus 5, Kimi K3, and their contemporaries? How quickly did capability and cost efficiency improve when measured from public benchmark and pricing series rather than impressions? And how much more performance can a practitioner obtain from fixed model weights by spending inference compute carefully, calibrating reasoning effort, estimating confidence, and routing work across models? Contributions. 1. A consolidated, sourced timeline of language-model capability from 2018 to 2026 across three benchmark eras (Section 2, Table 1). 2. Trend fits showing∼5.8× annual growth in the odds of solving SWE-bench Verified tasks, alongside the saturation of knowledge benchmarks such as MMLU (Section 4.1). 3. A cost analysis showing a∼60× decline in input-token prices from GPT-3 (2020) to GPT- 5.6 Luna (2026), and a demonstration that today’s budget tier matches the flagship of one quarter ago on most agentic/professional benchmarks (Section 4.2). 4. Evidence that the 2026 frontier is task-fragmented, and that simple model routing recov- ers the per-task optimum (Section 4.3). 5. A locked development/evaluation study of self-consistency for a small open model, fol- lowed by an exploratory confidence model that ranks predictions for triage. Complete traces, paired uncertainty, and reasoning-effort analyses accompany the results (Sec- tion 5). 2 Eight years in four paradigms 2.1 The encoder era: BERT and the fine-tuning paradigm (2018–2019) BERT 1 established the pretrain-then-fine-tune recipe: masked-language-model pretraining on unlabeled text, followed by a small task-specific head. RoBERTa 4 showed the recipe was undertrained rather than wrong, reaching 88.5 on GLUE with more data and longer training; T5 5 unified all tasks as text-to-text and reached 90.3 on SuperGLUE. GPT-2 6 demonstrated that a left-to-right decoder trained on enough web text acquired tasks without fine-tuning, but its zero-shot quality was not yet competitive. The era’s signature: capability lived in task-specific heads, and benchmarks (GLUE, SuperGLUE, SQuAD) saturated within roughly two years of introduction. 3 2.2 Scale and in-context learning: GPT-3 (2020–2021) GPT-3 7 scaled the decoder recipe to 175B parameters and showed that few-shot prompting — a handful of examples in the context window — could substitute for gradient updates on many tasks. Its 43.9% five-shot MMLU 8 looks modest today, but the conceptual shift was permanent: capability could be elicited from a frozen general model. The limits were equally clear: GPT-3 few-shot scored 71.8 on SuperGLUE, well below fine-tuned systems, and its unaligned samples were unreliable. 2.3 Alignment and instruction following (2022–2023) InstructGPT 9 applied reinforcement learning from human feedback (RLHF) to make models follow instructions; ChatGPT (November 2022) packaged this for general users. GPT-4 (March 2023) jumped to 86.4% on MMLU 10 , near the estimated human-expert level, and established the modern pattern of a capability report accompanying release. Meta’s Llama series opened weights and created an open ecosystem; Claude and Gemini established a multi-vendor frontier. 2.4 Reasoning and agents (2024–2026) Test-time compute changed the scaling target. OpenAI’s o1 (December 2024) showed that longer, trained reasoning traces and additional inference tokens could lift math and coding performance; its 91.8% MMLU score effectively saturated that benchmark 11 . Agency then changed the task itself. SWE-bench Verified 12,13 made “resolve a real GitHub issue” the standard hard problem. Scores on independent harnesses climbed from 49% for Claude 3.5 Sonnet in October 2024 to 97% for Claude Opus 5 in July 2026 14 . By mid-2026 the flag- ship releases — Claude Fable 5 (July 1), GPT-5.6 Sol/Terra/Luna (July 9), Kimi K3 (July 16), Claude Opus 5 (July 24) — were evaluated primarily on agentic, long-horizon, and economically-weighted tasks rather than static question answering 15–18 . Kimi K3 addition- ally marked the near-closure of the open–closed gap: a 2.8-trillion-parameter open-weight mixture-of-experts model competitive with the best proprietary systems 17 . Table 1 summarizes the arc, and Figure 2 shows the agentic benchmark series at its current endpoint. 3 Data and methods 3.1 Data The repository contains three analysis tables, a locked evaluation split, and the complete traces needed to reproduce every reported result. D1: Timeline. Twenty-seven (model, benchmark, score, date, source) rows spanning 2018–2026, drawn from the original papers and from leaderboard mirrors (Table 1; full file data/models timeline.csv). Where multiple evaluation variants exist we record the variant explicitly (e.g., vendor-reported vs. independent SWE-bench Verified). D2: 2026 frontier matrix. Per-model benchmark scores and API prices for the July 2026 frontier — GPT-5.6 Sol/Terra/Luna, Claude Fable 5, Claude Opus 5, Kimi K3, plus the April 2026 flagship GPT-5.5 and Claude Opus 4.8 — from OpenAI’s GPT-5.6 launch tables 16 , Anthropic’s Opus 5 launch and system card 2,18 , Artificial Analysis 19,20 , the Vals AI independent SWE-bench Verified harness 14 , and arena.ai’s human-preference Frontend Code Arena 21 . 4 Table 1: Selected milestones, 2018–2026. Scores are as originally reported by the cited source; benchmark variants are noted. DateModelOrgMilestone metricScore 2018-10BERT-LargeGoogleGLUE80.5 2019-07RoBERTaMetaGLUE88.5 2019-10T5-11BGoogleSuperGLUE90.3 2020-05GPT-3OpenAIMMLU (5-shot)43.9 2023-03GPT-4OpenAIMMLU (5-shot)86.4 2024-05GPT-4oOpenAIMMLU88.7 2024-12o1OpenAIMMLU91.8 2024-10Claude 3.5 SonnetAnthropicSWE-bench Ver.49.0 2025-05Claude Opus 4AnthropicSWE-bench Ver.72.5 2025-08GPT-5OpenAISWE-bench Ver.74.9 2025-11Claude Opus 4.5AnthropicSWE-bench Ver.76.8 2026-04GPT-5.5OpenAISWE-bench Ver. (vendor)85.1 2026-07Claude Fable 5AnthropicSWE-bench Ver. (indep.)95.0 2026-07GPT-5.6 SolOpenAISWE-bench Ver. (indep.)96.2 2026-07Kimi K3MoonshotSWE-bench Ver. (indep.)93.4 2026-07Claude Opus 5AnthropicSWE-bench Ver. (indep.)97.0 Public evidence vendor + independent Versioned inputs CSV / parquet Deterministic analyses trends / cost / routing Paper outputs figures + exact claims Official GSM8K + local Qwen weights Seeded decoding + resumable checkpoint Traces + paired tests + OOF confidence Reproducibility architecture Figure 1: Reproducibility architecture. Public evidence feeds deterministic trend, cost, and routing analyses. The live evaluation records complete traces and paired statistics; the exploratory confidence stage consumes those versioned traces. Every path produces auditable paper outputs. D3: Pricing. Input/output prices per million tokens for representative API models from 2020 to 2026, using vendor documentation where available 16,22 . D4:Live-evaluation items. The committed official GSM8K test split contains 1,319 question/answer pairs 23 .We retain the original 16 pilot identifiers as a locked development split and sample 100 evaluation identifiers once from the remaining rows (random state=20260810) before v1 tuning. The split manifest records both identifier lists and the source-parquet SHA-256 digest. Provenance caveat. Mid-2026 numbers mix vendor-reported and independent evalua- tions. We label provenance wherever it matters and prefer third-party measurements (Vals AI, Artificial Analysis, ARC Prize) when both exist. Vendor launch numbers are claims, not independent results; we treat them as such and note where vendor and third-party harnesses differ. 5 3.2 Analysis methods Trend fitting. For the SWE-bench Verified series we fit a log-linear model to the odds of solving a task, logit(p), against release date, and report the implied annual multiplicative growth of those odds with the fit R 2 (Figure 2). For MMLU we compare the raw series with the estimated human-expert level of 89.8% 8 . Pareto analysis. Using Artificial Analysis’s all-effort dataset for the GPT-5.6 family (15 model×effort settings with intelligence-index scores and per-task costs 19,20 ), we compute the cost–capability Pareto frontier and identify dominated settings. Routing oracle. On the fourteen benchmarks that OpenAI’s GPT-5.6 launch table reports for all six of Sol, Terra, Luna, GPT-5.5, Fable 5, and Opus 4.8, we normalize each benchmark to its best model (100), then compare (i) the best single model’s mean, (i) an oracle that picks the best model per benchmark, and (i) restricted routers (a cheapest-within-one-point router and a two-model router). Live experiment. Section 5.1 describes a controlled development/evaluation protocol. On development data we screen three prompts (official chat-template concise, four-shot, and solve-and-check) at temperatures 0.4 and 0.7 using Qwen2.5-0.5B-Instruct, confirm the lead- ing four-shot/0.4 setting on all 16 items, and compare it with Qwen2.5-1.5B-Instruct. The primary selector is four-sample plurality; a greedy candidate verifier is prespecified as sec- ondary. Before evaluation, a manifest freezes Qwen2.5-1.5B-Instruct, the official chat tem- plate with four worked examples, k = 4, temperature 0.4, top-p = 0.95, seed 1234, and the plurality selector. We then run that configuration once on all 100 disjoint evaluation items. Every record stores the greedy completion, four sampled completions, canonical numeric an- swers, prefix votes, verifier decision, and correctness. We report exact counts, Wilson 95% intervals, the any-sample oracle ceiling, and two-sided exact McNemar tests against paired greedy outcomes. Atomic checkpoints, JSONL traces, split/config hashes, and integrity tests are provided. Exploratory confidence model. After the frozen evaluation labels were available, we specified a simple model to estimate whether vote-at-four was correct. Its seven reference- free features are plurality share, normalized answer entropy, the number of distinct sampled answers, agreement of plurality with greedy and verifier outputs, and the mean and standard deviation of sample-completion length. We standardize these features inside each training fold and fit an L2-regularized logistic regression. Five-fold stratified cross-validation with shuffling and seed 20260811 yields one out-of-fold probability for every item. We report ROC AUC, Brier score against a constant 0.62 prevalence baseline, and accuracy among the predictions retained at fixed 50% and 75% coverage. The features, folds, metrics, and coverage levels were set before scoring this added analysis, but the analysis itself was designed after the v1 outcomes were known. It is exploratory and does not constitute a second held-out evaluation. 4 Results 4.1 Capability growth: three benchmark eras Knowledge benchmarks saturated. The MMLU series rose from 43.9% (GPT-3, 2020) to 86.4% (GPT-4, 2023) to 91.8% (o1, late 2024), past the 89.8% estimated human-expert level. By 2024 the benchmark had lost discriminative power: four strong models clustered within 4 points 24 , and benchmark mirrors now mark MMLU as “saturated” and exclude it 6 Figure 2: SWE-bench Verified, October 2024 – July 2026 (mix of vendor-reported and independent harness results; see Table 1). The log-linear fit on solve odds implies ∼5.8× growth per year. from scoring 11 . The field responded with harder knowledge tests (MMLU-Pro 24 , Humanity’s Last Exam) and, increasingly, with agentic tasks. Agentic coding became the frontier metric — and moved fast. Figure 2 shows SWE-bench Verified from October 2024 to July 2026. A log-linear fit on the solve odds gives odds t+1yr ≈ 5.8× odds t (R 2 = 0.78, n = 14): the odds of resolving a real GitHub issue multiplied by almost six each year. Independent evaluation puts the current frontier at 96–97% 14 , implying the benchmark itself will saturate within roughly a year — the same lifecycle GLUE and MMLU went through, on a harder task. Harder variants (SWE-bench Pro, Frontier-Bench, on which even the best model scores only 43–64% 16,18 ) are already taking over. Benchmark turnover accelerated. GLUE remained discriminative for about two years and MMLU for roughly four; SWE-bench Verified appears headed toward a two-year lifespan. Capability keeps advancing, but each measuring stick loses headroom soon after laboratories optimize against it. 4.2 The collapse of the capability–cost curve API input prices fell from GPT-3’s $60 per million tokens (2020) to GPT-5.6 Luna’s $1 (2026): a ∼60× decline, compounding at roughly 2× per year, while capability itself was compounding at a similar rate. The two effects multiply: a dollar buys orders of magnitude more capability than a dollar in 2023. Budget tier vs. last quarter’s flagship. The sharpest illustration is internal to Ope- nAI’s own July 2026 launch table 16 . GPT-5.6 Luna ($1/$6 per M input/output tokens at the official standard API rate 16 ) against GPT-5.5 ($5/$30), the flagship released eleven weeks earlier: across the 23 benchmarks reported for both, Luna wins 9 and loses 14 — but the wins concentrate exactly where the economic value is. On agentic and professional work (Agents’ Last Exam, GDPval-A, SWE-bench Pro, DeepSWE, HealthBench Profes- sional, AutomationBench, GraphWalks) Luna wins 7 of 10; it trails mainly on academic knowledge (GPQA), competition math (FrontierMath), multimodal understanding, and — 7 sharply — long-context recall (41.3% vs. 81.5% on MRCR). Against the flagship of one year ago the comparison is not close: our reconstruction of overall SWE-bench Verified from Vals AI’s per-difficulty rows gives Luna 92.9% vs. GPT-5’s 74.9% vendor-reported score in August 2025 14,25 . In practical terms, capability that was flagship-class 2–12 months ago now costs ∼20% of the flagship token price. The cheapest tier still gives up long-context recall and the hardest academic reasoning. The mid tier is dead. Our Pareto analysis of the fifteen GPT-5.6 settings reproduces and confirms Artificial Analysis’s finding 19,20 : every Terra configuration is dominated — there is always a Luna or Sol setting that is smarter at no extra cost or equally smart and cheaper. Six of fifteen settings lie off the frontier. The economic structure of a model family is now bimodal: a volume tier and a premium tier, with the middle squeezed out. 4.3 The frontier fragmented by task No single model leads the 2026 frontier; leadership is task-dependent (Figure 3): • Frontend and design-oriented coding: Claude Opus 5. On the live arena.ai Web- Dev leaderboard checked August 10, Opus 5 led blind developer preference at 1,712 Elo, with open-weight Kimi K3 second at 1,682; both values were preliminary and will move as votes accumulate 21 . • Repository-level coding: Claude Fable 5. Fable 5 leads SWE-bench Pro at 80.0%, 15.4 points ahead of GPT-5.6 Sol (64.6%) 16 . • Agentic terminal and long-horizon professional work: GPT-5.6 Sol. Sol leads the Artificial Analysis Coding Agent Index (80), Terminal-Bench 2.1 (88.8%; 91.9% in multi-agent Ultra mode), and Agents’ Last Exam (52.7%, +12.2 over the best non- OpenAI model) 16,19 . • Novel reasoning and computer use: Claude Opus 5. Opus 5 scored 30.2% on ARC- AGI-3 — nearly 4× the previous record of 7.8% (GPT-5.6 Sol) — plus a vendor-reported 42/42 on IMO 2026 under Anthropic’s grading protocol and the lead on OSWorld 2.0 computer use 2,3,18 . • Best-value agentic work: Claude Opus 5 again, reaching within 0.5% of Fable 5’s CursorBench peak at half the task cost 18 . Routing recovers the optimum. If no model wins everywhere, a router should beat every individual model. On the fourteen fully-covered benchmarks, the best single model (GPT-5.6 Sol) averages 97.6% of the per-benchmark best; an oracle router averages 100% (+2.4 points). A router with only two models — GPT-5.6 Sol and Claude Fable 5 — already captures the entire oracle gain on this suite because their strengths are complementary (agentic vs. repository-level and mathematical work). A cost-constrained router that picks the cheapest model within one point of the best matches the oracle score at 95% of its cost — the savings are modest precisely because Sol’s wins are usually by wide margins (Section 5 discusses implications). Why models diverge. Specialization is manufactured, not accidental. Moonshot opti- mized K3 for agentic coding with a 2.5× scaling-efficiency gain from architectural changes (Kimi Delta Attention, Attention Residuals) and a training pipeline openly aimed at the agentic benchmark suite 17 ; its 1M-token single-agent setup was deliberately tuned for long- horizon tasks like BrowseComp 17 . OpenAI trained GPT-5.6 for terminal and multi-agent 8 GPT-5.6 SolClaude Opus 5Claude Fable 5Kimi K3 Agentic work (Agents' Last Exam) Repo coding (SWE-bench Pro) SWE tasks (SWE-bench Verified) Frontend (Arena Elo, 2026-08-10) Novel reasoning (ARC-AGI-3) Knowledge work (GDPval-A Elo) 100 n/a 77 n/a 81 n/a 100 n/a 991009896 951009598 2610066 n/a 99 n/a 10096 Capability as % of category best (2026-07/08) 80.0 82.5 85.0 87.5 90.0 92.5 95.0 97.5 100.0 Figure 3: Capability as a percentage of the category best, July–August 2026. No column is uniformly dark: each frontier model has at least one category where another model clearly leads. Cells marked n/a lack a comparable public number — itself a symptom of vendors reporting different benchmarks. workflows (Ultra mode natively fans out subagents) 16 . And the cautionary case: Opus 5’s record ARC-AGI-3 gain partially failed to transfer to Witness, an independent held-out puz- zle benchmark (43.4, statistically tied with K3 and Fable 5), consistent with training on genre-adjacent data 26,27 . Targeting a task family works, although gains measured on its benchmark can overstate what transfers to new settings. Section 6 examines this tension. 5 Improving fixed models at inference time We evaluate three ways to raise effective performance without retraining: sample more than once, match reasoning effort to the workload, and route work to specialized models. 5.1 Sample, then aggregate Why multiple attempts can help. On problems with verifiable answers, independent samples can expose solution paths that greedy decoding misses. Self-consistency aggregates those attempts without changing model weights 28 . The selector still matters: a poor rule can discard the only correct candidate. Development and model selection. We use the original 16-item Qwen2.5-0.5B-Instruct run as development evidence, not a final estimate, and lock a disjoint 100-item evaluation set before tuning. The first eight development items screen three official-chat prompts (concise, four-shot, and solve-and-check) at temperatures 0.4 and 0.7. We confirm the leading four- shot/0.4 prompt on all 16 items, compare 0.5B and 1.5B models, and measure plurality prefixes alongside a candidate verifier. A frozen manifest records the selected configuration before evaluation begins. Development results. The legacy raw-prompt 0.5B pilot produced 1/16 greedy and 2/16 vote-at-eight. The official instruction template, four worked examples, and temperature 0.4 raised the same model to 5/16 greedy and 6/16 with vote-at-four or the verifier. With that setup held fixed, the 1.5B model reached 8/16 under greedy, vote-at-four, and verifier decod- ing on CPU. Prompt formatting supplied the largest development gain. Model size improved 9 Table 2: Development path and frozen evaluation counts. The legacy vote uses k = 8; all v1 vote rows use k = 4. Development rows guided configuration selection and must not be read as holdout estimates. ConfigurationnGreedyVoteVerifierOracle Legacy raw prompt, 0.5B (dev.)1612– Chat + few-shot, 0.5B (dev.)1656610 Chat + few-shot, 1.5B (dev.)1688810 Frozen 1.5B configuration (eval.)10058626279 Figure 4: Frozen evaluation and exploratory confidence triage. The left panel reports exact-match accuracy with Wilson 95% intervals; vote-at-four is the primary result and the any-sample oracle is a diagnostic ceiling. The right panel ranks vote-at-four predictions by out-of-fold confidence and shows accuracy at fixed coverage. Dashed lines mark the full-set baselines. greedy and the selected vote, whereas the verifier added no development accuracy. Because these results guided selection, we do not treat them as unbiased performance estimates. Frozen evaluation. The manifest fixes Qwen2.5-1.5B-Instruct, four-shot official chat for- matting, k = 4, temperature 0.4, top-p = 0.95, seed 1234, and plurality as the primary selector; it names the verifier as secondary. We run the 100 locked items once on CUDA and retain every paired trace. On the untouched 100-item evaluation split, greedy decoding solved 58/100 (58.0%; Wilson 95% CI [48.2, 67.2]), while the prespecified four-sample plu- rality solved 62/100 (62.0%; Wilson 95% CI [52.2, 70.9]). The verifier also solved 62/100 (62.0%). The primary paired exact McNemar test gave p = 0.481. This four-point gain equals a 6.9% relative improvement in the saved run, but the con- fidence intervals overlap and the paired test does not reject equality. We therefore describe an observed improvement, not an established population effect. Selection remains the bottleneck. All 400 sampled answers were parseable, and at least one sample was correct on 79/100 items. Plurality and the verifier each recovered 62/100, leaving a 17-point gap to the oracle ceiling. The verifier tied plurality on evaluation after trailing it in the CUDA development check (6/16 versus 10/16). Complexity alone did not help; a better selector must find correct minority candidates without disturbing correct consensus. 10 Table 3: Frozen v1 evaluation results. Statistical inference is paired against greedy; the primary exact McNemar test gives p = 0.481. MethodCorrectAccuracyWilson 95% CIRole Greedy58/10058.0%[48.2, 67.2]baseline Sample @ 161/10061.0%[51.2, 70.0]diagnostic Vote @ 261/10061.0%[51.2, 70.0]diagnostic Vote @ 462/10062.0%[52.2, 70.9]prespecified primary Verifier62/10062.0%[52.2, 70.9]prespecified secondary Oracle: any sample79/10079.0%[70.0, 85.8]ceiling diagnostic Reproducibility. The same seed produced different candidate draws on CPU and CUDA, although greedy correctness remained 8/16 in the development check. The result JSON therefore records the execution device, and the paper uses the CUDA artifact from the 100- item evaluation. This experiment mirrors frontier multi-sample and multi-agent inference at small scale; for example, Sol Ultra reports 91.9% on Terminal-Bench versus 88.8% for the single-agent setting 16 . 5.2 Estimate confidence for selective use After the v1 labels were available, we asked whether the traces could identify reliable vote- at-four outputs without looking at reference answers. We fixed seven inexpensive signals before scoring the new analysis: plurality share, answer entropy and diversity, agreement with greedy and verifier outputs, and two completion-length statistics. An L2-regularized logistic regression produces one out-of-fold confidence score for each item under five-fold stratified cross-validation. The model separates correct from incorrect vote-at-four outputs with ROC AUC 0.833. Its Brier score is 0.147, compared with 0.236 for a constant prediction equal to the 62% prevalence. Confidence ranking is more useful operationally: the top 50 predictions contain 47 correct answers (94.0%), and the top 75 contain 53 (70.7%). Full coverage necessarily returns to the frozen 62/100 result. Table 4: Exploratory selective accuracy from five-fold out-of-fold confidence scores. Coverage levels were fixed before scoring. Top refers to confidence rank, not a second evaluation split. Confidence-ranked setCoverageCorrectAccuracy Top half50/10047/5094.0% Top three quarters75/10053/7570.7% All items100/10062/10062.0% A deployed system could accept the high-confidence half and send the remainder to additional sampling, a larger model, or human review. This procedure improves the accuracy of the retained set, not full-coverage exact match. Because the same 100 labels both train and assess the cross-validated model, independent data must confirm the result. 5.3 Match reasoning effort to the workload Maximum effort can cost more while producing a worse answer. Anthropic’s score–cost curves put Opus 5’s Frontier-Bench peak at xhigh effort (44.4%); max effort falls to 43.3% despite consuming more compute. A community re-analysis of Anthropic’s Frontier-code chart places that peak near medium effort (∼53%) 18 . Our Pareto analysis shows a different part of the 11 same tradeoff: each roughly four-point intelligence gain from Luna-max to Terra-max to Sol-max multiplies per-task cost by 2.6× and then 1.9× 20 . Longer chains can wander on tasks with direct solution paths. Deployments should therefore measure effort per workload and use its empirical optimum rather than treating maximum effort as a universal quality setting. 5.4 Route work across specialized models The fragmented frontier in Section 4.3 makes routing more effective than a single default model. Across the fourteen fully covered benchmarks, the best individual model averages 97.6% of the per-benchmark maximum. An oracle reaches 100, and a two-model combination of Sol and Fable 5 captures that entire gain. Production systems increasingly expose this choice through model-abstraction layers 29 ; practitioners also pair a strong planner with a cheaper executor, such as “Sol medium for architecting, Luna high for coding” 30 . Two complementary frontier models can cover quality-sensitive work, while a budget tier handles well-defined subtasks at lower cost. 5.5 Reporting priorities for model builders Model builders can make these tradeoffs easier to evaluate by publishing score–cost curves for every benchmark; single endpoint numbers invite cherry-picking 18 . Budget models also need better long-context recall: Luna scores 41.3% on MRCR against Sol’s 91.5% 16 , the clearest break from its near-flagship pattern. Held-out transfer suites should accompany targeted benchmarks, as the ARC-AGI-3/Witness gap in Section 4.3 demonstrates. K3’s BrowseComp result further points toward better mid-context retrieval, not longer context windows alone, as the more valuable investment target 17 . 6 Discussion: how models get good at specific tasks The 2026 pattern — one frontier, several winners — follows directly from the way laboratories now build frontier models. Domain-weighted post-training. Reinforcement learning on verifiable, domain-specific rewards (unit tests for code, proof checkers for math, task-completion signals for agents) lets a lab buy capability in a chosen vertical. Moonshot’s agentic suite results and OpenAI’s terminal/agentic lead both track heavy RL investment in those task families 16,17 . Architecture as targeting. Kimi K3’s hybrid linear attention (Kimi Delta Attention) and expert-routing changes were chosen for long-context agentic efficiency, and the payoff appeared precisely on long-horizon benchmarks (BrowseComp 91.2, state of the art at re- lease) 17 . Inference-time structure. GPT-5.6’s Ultra mode makes multi-agent fan-out a first-class feature and buys Terminal-Bench points (88.8% → 91.9%) at higher cost 16 . Our self- consistency experiment (Section 5.1) applies the same lever at small scale. The exploratory confidence model adds a routing signal: accept high-confidence answers, then reserve addi- tional compute or review for the rest. Capability depends partly on how a system orchestrates inference. 12 Benchmark-shaped optimization and its limits. When a benchmark becomes a target, labs optimize toward it, and two failure modes follow: the benchmark saturates (GLUE, MMLU, soon SWE-bench Verified), or gains fail to transfer off-distribution. The Witness evaluation of Opus 5 is the cleanest 2026 example: a ∼4× record on ARC-AGI-3, but only a statistical tie with K3 and Fable 5 on held-out puzzles of the same genre, including one environment where Opus 5 stated the hidden rules before its first move — suggesting familiarity, not just reasoning 26,27 . METR’s maintainer-review study adds a second caution from a different angle: a growing share of SWE-bench-passing patches would be rejected by human maintainers (quality, breakage, or missing the issue’s point), and the grader– maintainer gap is widening over model generations 31 . Targeting works, and that is exactly why single-number claims deserve suspicion. Implication for evaluation. Score-versus-cost curves, held-out transfer suites, and maintainer-grade review are becoming the minimum standard of evidence. Vendors already report curves 18 ; ARC Prize now administers runs independently with published traces 3,27 ; and METR’s maintainer study is a template for auditing what benchmark passes actually mean. Implication for buyers. The economic shape of 2026 — a budget tier at 20% of flagship token price that matches last quarter’s flagship on most professional work, plus a fragmented frontier — means the unit of procurement is no longer “a model” but a routing policy with per-task effort settings. The 2023 question “which model is best?” has been replaced by “which combination, at which effort, for which task?” Limitations. Mid-2026 scores mix vendor and independent evaluations. Provenance labels cannot fully remove harness differences: GPT-5.5, for example, scores 85.1% in the vendor report and 82.6% in our reconstruction of Vals AI’s independent harness 14,25 . The SWE- bench trend also pools both sources across only 22 months, so 5.8× indicates scale rather than a stable law. Our routing oracle knows benchmark identity; a production router must infer task type from unlabeled traffic. The v1 intervention used 16 development items and one 100-item GSM8K evaluation. It covers a single model family, task, and sampling seed. Its four-point primary gain is not statistically significant (p = 0.481), and the Wilson intervals remain broad. We designed the confidence model after those labels were visible and assessed it by cross-validation on the same 100 items. Its AUC and 94% top-half accuracy are exploratory until a new dataset confirms them. Seeded candidates also differ between CPU and CUDA, so exact replay requires the recorded software and device context. Public benchmark figures are current as of August 10, 2026, in a field that changes weekly. 7 Conclusion Eight years separate BERT’s 80.5 GLUE score from Opus 5’s reported 42/42 on the 2026 IMO problems. Progress did not follow one smooth exponential. Each shift — fine-tuning, few-shot scale, alignment, reasoning, and agency — moved capability onto a new axis and exhausted the previous era’s benchmarks. Agentic-coding solve odds are still rising at roughly six-fold per year, while the price of a fixed capability level falls fast enough that a $1/M budget tier now reproduces most of the professionally relevant performance of an eleven- week-old flagship. Different models lead frontend coding, repository repair, terminal work, and novel reasoning, so deployment increasingly depends on routing policy rather than one model choice. 13 Fixed weights still leave room for system-level gains. In our frozen GSM8K evaluation, four-sample plurality moved exact match from 58% to 62%, though the four-point difference remains uncertain (p = 0.481). At least one sample was correct on 79% of items, which makes selection the clearest next target. An exploratory logistic model found a useful signal in answer agreement and trace statistics: out-of-fold AUC reached 0.833, and 47 of its 50 highest-confidence predictions were correct. That result supports selective triage, not a claim that full-coverage accuracy exceeds 62%, and it still requires confirmation on untouched data. Reproducibility. The accompanying archive contains all datasets (code/data/), analysis scripts (exp1–exp5), locked manifests, complete inference traces, unit tests (tests/), figure generation, and the paper build script; see README.md. Reproduction needs no proprietary access beyond the live benchmark numbers, whose sources and retrieval dates appear in code/data sources.md. References [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- training of deep bidirectional transformers for language understanding. NAACL-HLT, 2019. [2] Anthropic. Claude opus 5 system card. https://w.anthropic.com/claude-opus-5 -system-card, 2026. July 24, 2026. [3] ARC Prize. Claude opus 5 — ARC-AGI results. https://arcprize.org/results/a nthropic-claude-opus-5, 2026. Independently administered results, July 24, 2026. [4] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692, 2019. [5] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21, 2020. [6] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Technical Report, 2019. [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. [8] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask understanding. ICLR, 2021. [9] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human feedback. NeurIPS, 2022. [10] OpenAI. GPT-4 technical report. arXiv:2303.08774, 2023. [11] BenchLM. MMLU leaderboard & scores. https://benchlm.ai/benchmarks/mmlu, 2026. Retrieved August 7, 2026; benchmark marked saturated/display-only. [12] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? ICLR, 2024. 14 [13] OpenAI. Introducing SWE-bench verified. https://openai.com/index/introducing -swe-bench-verified/, 2024. [14] Vals AI. SWE-bench verified — independent evaluation results. https://vals.ai/be nchmarks/swebench, 2026. Retrieved August 6, 2026. [15] Anthropic. Claude fable 5 and claude mythos 5. https://w.anthropic.com/ne ws/claude-fable-5-mythos-5, 2026. Official launch post, June 9, 2026; availability updated July 1, 2026. [16] OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition. https://open ai.com/index/gpt-5-6/, 2026. Launch tables, July 9, 2026; retrieved August 2026. [17] Moonshot AI. Kimi k3: Open frontier intelligence. https://github.com/MoonshotA I/Kimi-K3, 2026. Official repository and technical report, July 16, 2026. [18] Anthropic. Introducing claude opus 5. https://w.anthropic.com/news/claude-o pus-5, 2026. Official launch post, July 24, 2026. [19] Artificial Analysis. GPT-5.6 benchmarks across intelligence, speed and cost. https: //artificialanalysis.ai/articles/gpt-5-6-has-landed, 2026. July 9, 2026. [20] Igor Warzocha. Comparison for all 5.6 models (artificial analysis anchor measurements). https://gist.github.com/IgorWarzocha/60bfd11731f15cf8802f0b6e80d47ac7, 2026. July 11, 2026. [21] Arena AI. Code arena: Webdev overall leaderboard. https://arena.ai/leaderboard /code/webdev/overall, 2026. Live blind-preference leaderboard, retrieved August 10, 2026. [22] DeployBase. Cost per token over time: How LLM API pricing has dropped. https: //deploybase.ai/articles/cost-per-token-over-time-how-llm-api-pricing-h as-dropped, 2026. March 4, 2026. [23] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rei Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.14168, 2021. [24] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. NeurIPS Datasets and Benchmarks, 2024. [25] LocalAIMaster. GPT-5.5 review: ChatGPT default, tested. https://localaimaster. com/models/gpt-5-5, 2026. May 9, 2026. [26] The Decoder. Anthropic’s opus 5 blows past fable 5 and gpt-5.6 sol on the benchmark designed to measure real intelligence. https://the-decoder.com/anthropics-opus-5 -blows-past-fable-5-and-gpt-5-6-sol-on-the-benchmark-designed-to-measu re-real-intelligence/, 2026. July 26, 2026; includes Witness benchmark results. [27] TechTimes. Claude opus 5 took ARC-AGI-3 record with an equation no AI had written before. https://w.techtimes.com/articles/321661/20260727/, 2026. July 27, 2026. 15 [28] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ICLR, 2023. [29] Yash Thakker. GPT-5.6: Rolling out july 9 — sol, terra, luna. https://explainx.ai/ blog/gpt-5-6-release-date-features-benchmarks-2026, 2026. June–July 2026. [30] Hacker News. Advancing the price-performance frontier with GPT-5.6 — discussion. https://news.ycombinator.com/item?id=49112867, 2026. July 30, 2026; community benchmark comparisons. [31] METR. Many SWE-bench-passing PRs would not be merged into main. https://me tr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merge d-into-main/, 2026. March 10, 2026. 16