Paper deep dive
All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code
Dipayan Banik, Kowshik Chowdhury, Shazibul Islam Shamim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 2:12:50 AM
Summary
This paper investigates the quality of test code generated by AI coding agents in open-source pull requests (PRs). Through an empirical study of 86,156 test-file patches from 33,596 agent-authored PRs, the authors identify a significant gap between test presence and actual verification strength. They propose a syntactic taxonomy of eight oracle signal categories, ranging from 'Weak' (no or no meaningful assertion) to 'Strong' (behavioral/value comparisons). The study finds that 80.2% of agent-authored test patches contain weak or no explicit oracle signals. While strong oracles are associated with more complex and heavily reviewed PRs, a multivariate logistic regression reveals that strong oracle signals significantly increase the likelihood of a PR being merged (OR = 1.28), even after controlling for agent, size, and repository popularity.
Entities (9)
Relation Signals (4)
AI Coding Agent → authors → Pull Request
confidence 100% · AI coding agents that generate test code alongside production code in open source pull requests (PRs).
Strong Oracle Signal → improves → Merge Likelihood
confidence 100% · strong oracles significantly improve merge likelihood (OR = 1.28, p < 0.001).
OpenAI Codex → isa → AI Coding Agent
confidence 100% · produced by five coding agents: OpenAI Codex...
Claude Code → produces → Strong Oracle Signals
confidence 90% · Claude Code and Devin producing stronger oracle profiles than Copilot, Cursor, and OpenAI Codex.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Software practitioners increasingly use AI coding agents that generate test code alongside production code in open source pull requests (PRs). Recent studies report more than 932,000 agent-authored PRs across more than 116,000 repositories, yet whether their test files contain meaningful verification logic remains underexplored. Test files lacking explicit assertions execute code without verifying behavior, so quality gates based on test-file presence overestimate verification strength. The goal of this paper is to help practitioners assess the verification strength of agent-authored patches by characterizing oracle signals and their link to merge outcomes and review effort. We conduct an empirical study of 86,156 test-file patches from 33,596 agent-authored PRs across 2,807 GitHub repositories produced by five coding agents: OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code. A qualitative analysis of 384 stratified patches informs a syntactic taxonomy of eight oracle signal categories. Applied at scale, 80.2% of test patches contain weak or no explicit oracle signals. While raw merge rates are lower for strong-oracle PRs, a regression analysis adjusting for agent, PR size, repository popularity, task type, and language shows strong oracles significantly improve merge likelihood (OR = 1.28, p < 0.001). Our findings suggest that test file counts substantially overestimate verification strength and that practitioners can adopt oracle-aware quality checks to more accurately evaluate agent-authored contributions.
Tags
Links
- Source: https://arxiv.org/abs/2606.18168v1
- Canonical: https://arxiv.org/abs/2606.18168v1
Trouble viewing inline? Open PDF directly →
Full Text
23,307 characters extracted from source content.
Expand or collapse full text
All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code Dipayan Banik Danovo Energy Solutions Raleigh, North Carolina, USA dbanik@danovoenergy.com Kowshik Chowdhury Kennesaw State University Kennesaw, Georgia, USA kchowdh1@students.kennesaw.edu Shazibul Islam Shamim Kennesaw State University Kennesaw, Georgia, USA mshamim@kennesaw.edu Abstract—Software practitioners increasingly use AI coding agents that generate test code alongside production code in open-source pull requests (PRs). Recent studies report more than 932,000 agent-authored PRs across more than 116,000 reposito- ries, yet whether their test files contain meaningful verification logic remains underexplored. Test files lacking explicit assertions execute code without verifying behavior, so quality gates based on test-file presence overestimate verification strength. The goal of this paper is to help practitioners assess the verification strength of agent-authored patches by characterizing oracle signals and their link to merge outcomes and review effort. We conduct an empirical study of 86,156 test-file patches from 33,596 agent- authored PRs across 2,807 GitHub repositories produced by five coding agents: OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code. A qualitative analysis of 384 stratified patches informs a syntactic taxonomy of eight oracle signal categories. Applied at scale, 80.2% of test patches contain weak or no explicit oracle signals. While raw merge rates are lower for strong-oracle PRs, a regression analysis adjusting for agent, PR size, repository popularity, task type, and language shows strong oracles significantly improve merge likelihood (OR = 1.28, p < 0.001). Our findings suggest that test file counts substantially overestimate verification strength and that practitioners can adopt oracle-aware quality checks to more accurately evaluate agent-authored contributions. Index Terms—test oracle, test assertions, AI coding agents, empirical software engineering, mining software repositories I. INTRODUCTION The landscape of software development is shifting from AI- assisted code completion to autonomous code authorship. Interactive assistants such as early GitHub Copilot offered line-level suggestions that developers could accept or ig- nore, preserving human control over commits [1]. A new generation of autonomous agents, including OpenAI Codex, Devin, Cursor, and Claude Code operates at the repository level: given an issue or prompt, they plan implementations, write production code, generate tests, open pull requests, and in some cases iterate on reviewer feedback without human intervention [2], [3]. Industry projections estimate that 75% of enterprise software engineers will use AI code assistants by 2028 [4]. This transition represents not merely a productivity gain but a fundamental change in who authors the verification logic that guards against regression. As agents take on this authorship, the quality of the test code they produce becomes a critical concern. In a typical pull request, a developer writes a test that calls a function and then asserts that the returned value matches an expected result. This assertion is the test oracle, which determines whether the test passes or fails [5]. Without it, a test executes code but never checks whether the output is correct. A pull request that includes test files appears well-verified to a reviewer scanning a file list: CI pipelines report coverage, dashboards turn green, and the contribution looks complete. However, practitioners are increasingly reporting a gap between this appearance and the underlying verification strength. Developers describe AI- generated tests that “confirm your code does what it already does”— a phenomenon called “test theater” [6]. Common patterns include assertion-free functions that execute code paths but never verify the output [7], and test suites that mock the module under test itself, passing by construction rather than by correctness [8]. Research on LLM-generated test oracles reinforces this concern. Controlled experiments show that LLM-generated assertions frequently encode actual program behavior rather than expected behavior, turning bugs into passing tests [9]. Fine-tuned assertion generators detect only a fraction of known faults even at 59.5% exact-match precision [10], and mutation- guided feedback improves oracle quality but requires curated prompts unavailable in autonomous agent workflows [11]. These findings expose a gap unaddressed by existing metrics: code coverage correlates only weakly with fault-detection effectiveness [12], with mature test suites showing gaps of up to 51% between executed and checked code [13]. A systematic categorization of what agents actually produce in test patches ranging from no oracle signal to strong behavioral assertions would give both CI tooling and human reviewers a vocabulary to distinguish verified contributions from struc- tural scaffolding. Understanding how these categories relate to merge outcomes and review effort would further clarify whether oracle strength carries practical consequences in real- world development workflows, or whether reviewers treat all test-bearing PRs alike regardless of assertion quality. Existing research has examined oracle quality in controlled settings and test file presence in real-world repositories, but not the oracle content of agent-authored test patches at scale. We address this gap through a large-scale empirical study of 86,156 cumulative test-file patches from 33,596 agent- authored pull requests across 2,807 GitHub repositories from the AIDev dataset [14]. We investigate two research questions: arXiv:2606.18168v1 [cs.SE] 16 Jun 2026 • RQ1: What oracle signal patterns characterize agent- authored test-file patches? • RQ2: How does oracle signal strength relate to PR merge outcomes and human review effort? This paper makes three primary contributions: (1) We pro- pose a syntactic taxonomy of eight oracle signal categories for classifying test-file patches. (2) We characterize oracle signal distributions across five AI coding agents. (3) We analyze the relationship between oracle signal strength and PR merge outcomes through a multivariate logistic regression. I. METHODOLOGY We use the AIDev-pop subset of the AIDev dataset [14], which contains 33,596 agent-authored pull requests across 2,807 repositories with at least 100 GitHub stars. We extract file- level patches from the pr_commit_details table, which stores the diff for each file modified in each commit within a pull request. Our analysis scripts, classified dataset, and figures are available at [15]. A. RQ1: Oracle Signal Classification Test File Identification: We identify test files using two cri- teria applied jointly. First, the file path must match test-related directory or filename patterns, including test/, tests/, __tests__/, testing/, or filenames containing _test., .test., or .spec. Second, the file extension must belong to a set of 30 source code extensions spanning Python, Type- Script, JavaScript, Go, Java, C#, Rust, Ruby, and others. We exclude non-code artifacts (fixtures, snapshots, documentation) that appear in test directories but contain no executable test logic. This filtering yields 103,976 test code patches from the total of 711,923 file patches. Cumulative Patch Construc- tion: A single test file may appear in multiple commits within the same pull request. To avoid double-counting and capture the full oracle signal introduced by the PR, we concatenate all patches for each unique (PR, filename) pair in commit order. This produces 86,156 cumulative patches for our taxonomy analysis. Oracle Signal Taxonomy: For each cumulative patch, we extract added lines and search for assertion patterns from widely used testing frameworks (e.g., JUnit, pytest, Jest, Go testing, RSpec). Through iterative open coding, we find that oracle signals fall along a spectrum – from no assertion to checks that code merely executes, to checks of specific output values or behavior – which we group into eight categories in Table I. Weak categories (W1–W5) confirm execution without verifying output correctness, mirroring the documented gap between executed and output checked code [12], [13]; strong categories (S1–S3) compare output to expected values, verify error handling or types, or combine multiple assertion types. To assess reliability, two authors independently labeled a stratified random sample of 384 patches (95% confidence, 5% margin); we resolved disagreements by discussion and reported agreement with Cohen’s Kappa [16]. B. RQ2: Oracle Signals and PR Outcomes PR-Level Quality Groups: We aggregate patch-level classi- fications to the PR level by assigning each PR the highest TABLE I ORACLE SIGNAL TAXONOMY CATEGORIES. CodeTierDefinition W1WeakNo assertion pattern present W2WeakExistence/non-null checks only W3WeakBoolean asserts only (no value compared) W4WeakMock/call-verification only W5WeakSnapshot match only S1StrongValue equality or comparison S2StrongError, containment, or type checks S3StrongTwo or more distinct strong types oracle-signal category among its test-file patches. We de- fine three quality groups: all weak (best patch is W1–W5), hasstrong (best patch is S1 or S2), and hasmultistrong (best patch is S3). Outcome Variables: We collect merge outcome, review effort (formal reviews + inline + discussion comments), PR size (files changed, lines added), repository popularity (star count), and time to close for each PR. Controlled Comparisons: We compare the three quality groups across all outcome variables. To control for confounding factors, we repeat the comparison within each agent, within each task type, and within PR size buckets (small 1–5 files, medium 6– 20, large >20). We apply size-bucket controls to both merge rate and review effort to distinguish whether oracle signal strength has an independent association with PR outcomes. We additionally fit a multivariate logistic regression including agent, log-transformed PR size, log-transformed repository stars, task type, and primary language as covariates. I. ANALYSIS AND FINDINGS A. RQ1: Oracle Signals in Test-File Patches Figure 1 presents the oracle signal distribution across all 86,156 cumulative test-file patches and for newly created files separately. As shown in the summary header, 80.2% of patches contain weak or no explicit oracle signals (W1–W5), with value assertions (S1) and multi-signal strong oracles (S3) accounting for 11.3% and 5.7%, respectively. Two authors independently labeled 384 stratified patches (95% confidence, 5% margin), reaching Cohen’s κ = 0.77. On the same sample, the classifier matches the human labels for the oracle-signal category in 86.7% of patches. Two patterns emerge from the per-agent breakdown. First, agents differ in how often they introduce oracle signals (χ 2 = 2497.3, p < 0.001), with Claude Code and Devin producing stronger oracle profiles than Copilot, Cursor, and OpenAI Codex. Second, newly created files show higher strong-oracle rates than modified files (χ 2 = 810.2, p < 0.001), ranging from 18% for OpenAI Codex to 67% for Claude Code. The newly created-file comparison carries the strongest claim because the full file content is the patch and no prior oracle content exists. Key Finding. 80.2% of agent-authored test-file patches contain weak or no explicit oracle signals. On newly added test files, strong-oracle rates range from 18% to 67% across agents. W1W2W3W4W5S1S2S3 Claude Code Copilot Cursor Devin OpenAI Codex Overall 550.10.33-18419 730.30.20.50.21726 590.212217316 710.20.320.412310 850.00.30.30.0834 790.10.30.70.21136 41% strong (n=1,923) 26% strong (n=18,209) 36% strong (n=2,072) 26% strong (n=13,147) 15% strong (n=50,805) 20% strong (n=86,156) (a) All Test-File Patches W1W2W3W4W5S1S2S3 Claude Code Copilot Cursor Devin OpenAI Codex Overall 24-0.29-30631 550.40.40.30.029311 53-11417221 410.20.74122625 810.00.50.3-1125 710.10.50.80.31629 67% strong (n=461) 44% strong (n=3,475) 40% strong (n=603) 52% strong (n=2,108) 18% strong (n=15,648) 27% strong (n=22,295) (b) Newly Added Files Overall: 80.2% weak (W1W5) | S1: 11.3% | S3: 5.7% | N = 86,156 | Newly added: 18%67% strong across agents Fig. 1. Oracle signal distributions and breakdowns for RQ1 analyses. B. RQ2: Oracle Strength, Merge Outcomes, and Review Effort S3 PRs merge at 59.7% compared to 72.6% for weak-oracle PRs, but S3 PRs also contain 4.2× more code additions (Mann-Whitney test, p < 0.001, r = 0.38), attract 2.4× more review effort (Mann-Whitney test, p < 0.001, r = 0.36), and reside in repositories with 3.8× more stars. This review effort gap persists within PR size buckets, with ratios ranging from 1.7× for small PRs to 2.5× for large PRs. Figure 2 shows that the aggregate pattern weakens within individual agents. Within Claude Code, S3 PRs merge at 62% compared to 52% for weak-oracle PRs; Copilot shows the same direction at 49% versus 44%. For test-type PRs, S3 PRs merge at 81% compared to 77%. A multivariate logistic regression adjusting for agent, PR size, repository popularity, task type, and primary language confirms that S3 oracles associate with significantly higher merge rates after adjustment (OR = 1.28, p < 0.001). Key Finding. Stronger oracle signals are associated with more complex, heavily reviewed pull requests. After adjusting for agent, PR size, repository popularity, task type, and language, S3 oracles significantly increase the likelihood of merge. IV. DISCUSSION Our findings highlight a meaningful gap between test-file presence and oracle quality in agent-authored pull requests. This pattern may reflect an optimization mismatch: coding agents are trained on code completion objectives that reward structural plausibility, that is, creating test files, importing frameworks, and defining function signatures rather than be- havioral reasoning about expected program output. Generat- ing a correct oracle requires an understanding of intended semantics, a capability that current agent architectures do not explicitly optimize for. The variation across agents (18% to 67% strong-oracle rate on new test files) further suggests that training data, prompt design, and tool integration influence testing discipline. While our data cannot isolate the prompts behind each PR, oracle strength also varies with task type – a proxy for what the agent was asked to do: feature work yields strong oracles in only 18.2% of patches, versus 25.6% for bug fixes and 24.9% for test-focused tasks. This suggests practitioners obtain more meaningful oracles when verification is scoped as an explicit objective rather than bundled into feature PRs, and that adopting oracle-aware CI checks that flag newly added test files lacking assertion patterns can surface this gap early and separate verified contributions from structural scaffolding. V. THREATS TO VALIDITY Our regex-based taxonomy classifies syntactic oracle signals in patch diffs, not complete source files; a modified-file patch classified as W1 may belong to a file with existing assertions, which we address by reporting newly created files separately. A patch classified as S1 confirms that an equality check exists, not that it checks the right property. The association between oracle strength and merge outcomes may be confounded by unobserved factors such as developer experience, merge-bot interactions, or CI requirements; we mitigate this through multivariate regression adjusting for agent, PR size, stars, task type, and language. Our dataset covers five agents active between December 2024 and July 2025, with OpenAI Codex contributing 65% of PRs, and filters for repositories with 100+ stars; results may not generalize to smaller repositories or ecosystems with different testing conventions. Additionally, our taxonomy does not capture implicit oracles such as crash-on-failure or timeout-based detection, which some testing frameworks use as default verification mecha- nisms. Our sensitivity analysis shows that reclassifying W4 (mock/interaction) and W5 (snapshot) as strong shifts the overall weak rate by less than one percentage point, indicating that boundary decisions between categories do not materially affect the findings. VI. RELATED WORK Our work intersects three areas: the test oracle problem, LLM-based oracle generation, and empirical studies of agent- authored contributions. The oracle problem has a long re- search history [5]. Coverage correlates only weakly with fault- detection effectiveness [12], and mature test suites exhibit gaps of up to 51 percentage points between executed and checked code [13]. Jain et al. [17] formalized this as the “oracle gap”, showing that high coverage alone does not capture test ade- quacy. Our study examines whether this distinction between executing code and verifying its behavior holds for agent- authored test code at scale. In the LLM setting, generated oracles frequently encode actual rather than expected behavior [9], fine-tuned generators detect only a fraction of known faults [10], and mutation-guided feedback improves quality but requires curated prompts [11]. These studies evaluate oracle quality in controlled settings; our work shifts the analysis to real-world agent-authored patches. For real-world agent Claude Code CopilotCursorDevinOpenAI Codex 0 20 40 60 80 100 Merge Rate (%) 52 44 66 53 82 53 42 60 45 79 62 49 62 41 81 (a) Merge Rate by Agent Median Additions Mean Review Effort Median Stars 0 200 400 600 800 1000 1200 Value 123 3 306 184 4 307 512 7 1177 (b) PR Complexity by Quality Group featfixtestrefactor 0 20 40 60 80 100 Merge Rate (%) 76 63 77 71 68 53 80 64 58 50 81 61 (c) Merge Rate by Task Type All weak (W1-W5)Has strong (S1-S2)Has multi-strong (S3) Fig. 2. Merge outcomes and confounding-factor analyses for RQ2. behavior, Haque et al. measured test file presence on the AIDev dataset [14], Milanese et al. compared test quality between agent and human PRs using test-smell detection, and Chowdhury et al. analyzed how code-review agents and feedback signal quality relate to PR outcomes [18], [19], [20]. VII. CONCLUSION This study provides large-scale empirical evidence that test- file presence is an insufficient proxy for verification strength in agent-authored pull requests. By classifying 86,156 cumu- lative test-file patches across 2,807 repositories, we show that 80.2% contain weak or no explicit oracle signals, with the strong-oracle rate varying from 18% to 67% across agents on newly created files. These findings suggest that coding agents generate test structure far more reliably than they generate oracle logic. Strong oracle signals coincide with larger, more heavily reviewed PRs; once this complexity is controlled, they predict higher merge likelihood, making oracle strength a positive signal masked by confounding rather than a merge disadvantage. REFERENCES [1] H. Li, H. Zhang, and A. E. Hassan, “The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering,” arXiv preprint arXiv:2507.15003, 2025. [2] C. Reddington, “From idea to PR: A guide to GitHub Copilot’s agentic workflows,” https://github.blog/ai- and- ml/github- copilot/from- idea- t o- pr- a- guide- to- github- copilots- agentic- workflows/, 2025, accessed: 2026-03-30. [3] M. Watanabe, H. Li, Y. Kashiwa, B. Reid, H. Iida, and A. E. Hassan, “On the use of agentic coding: An empirical study of pull requests on GitHub,” arXiv preprint arXiv:2509.14745, 2025. [4] Gartner, “Gartner says 75 percent of enterprise software engineers will use AI code assistants by 2028,” https://w.gartner.com/en/newsroom/ press-releases/2024-04-11-gartner-says-75-percent-of-enterprise-softw are-engineers-will-use-ai-code-assistants-by-2028, 2024, press release, April 11, 2024. [5] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering, vol. 41, no. 5, p. 507–525, 2015. [6] B. Houston, “The rise of test theater,” https://ben3d.ca/blog/the- rise- o f-test-theater, 2025, accessed: 2026-03-29. [7] H. Flores, “I let an AI agent write 275 tests. here’s what it was actually optimizing for,” https://dev.to/htekdev/i- let- an- ai- agent- write- 275- tes ts- heres- what- it- was- actually- optimizing- for- 32n7, 2026, accessed: 2026-03-29. [8] M. Bar-Zeev, “Why testing after with AI is even worse,” https://dev.to/m barzeev/why-testing-after-with-ai-is-even-worse-4jc1, 2026, accessed: 2026-03-29. [9] M. Konstantinou, R. Degiovanni, and M. Papadakis, “Do LLMs generate test oracles that capture the actual or the expected program behaviour?” arXiv preprint arXiv:2410.21136, 2024. [10] S. Primbs, B. Fein, and G. Fraser, “AsserT5: Test assertion generation us- ing a fine-tuned code language model,” in Proceedings of the IEEE/ACM International Conference on Automation of Software Test (AST 2025), 2025, arXiv:2502.02708. [11] A. Moradi Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,” Information and Software Technology, vol. 171, 2024. [12] L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” in Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), 2014, p. 435–445. [13] S. B. Hossain, M. B. Dwyer, S. Elbaum, and A. Nguyen-Tuong, “Measuring and mitigating gaps in structural testing,” in Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023), 2023. [14] H. Li, H. Zhang, and A. E. Hassan, “AIDev: Studying AI coding agents on GitHub,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026), 2026, arXiv:2602.09185. [15] D. Banik, K. Chowdhury, and S. I. Shamim, “Replication package for “all smoke, no alarm: Oracle signals in agent-authored test code”,” https: //doi.org/10.6084/m9.figshare.32032107, 2026. [16] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, p. 159–174, 1977. [17] K. Jain, G. T. Kalburgi, C. Le Goues, and A. Groce, “Mind the gap: The difference between coverage and mutation score can guide testing efforts,” in Proceedings of the 34th IEEE International Symposium on Software Reliability Engineering (ISSRE 2023), 2023, p. 102–113. [18] S. Haque, S. Ingale, and C. Csallner, “An empirical study of tests in agentic pull requests,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026), 2026, arXiv:2601.03556. [19] R. Milanese, F. Salzano, A. Spina, A. Vitale, R. Pareschi, F. Fasano, and M. Fazzini, “Human-agent versus human pull requests: A testing- focused characterization and comparison,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026), 2026, arXiv:2601.21194. [20] K. Chowdhury, D. Banik, K. M. Ferdous, and S. I. Shamim, “From industry claims to empirical reality: An empirical study of code re- view agents in pull requests,” in Proceedings of the 23rd Interna- tional Conference on Mining Software Repositories (MSR 2026), 2026, arXiv:2604.03196.