Paper deep dive

Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering

Alexander Krentsel, Shubham Agarwal, Mert Cemri, Shu Liu, Sidharth Sankhe, Ziming Mao, Matei Zaharia, Ion Stoica

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 110

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 9/16/2026, 7:57:38 AM

Summary

The paper introduces the 'two-gap framework' to explain failure modes in agentic software engineering, identifying the 'requirement gap' (mismatch between stakeholder intent and formal requirements) and the 'model gap' (mismatch between the deployment environment model and reality). It argues that these gaps cannot be fully closed in open systems, leading to issues like reward hacking and hallucination. The authors propose an 'assurance-revision loop' that uses deployment evidence to iteratively narrow these gaps, framing assured development as a resource allocation problem balancing human judgment, agent capability, and compute.

Entities (10)

Two-gap framework · concept · 98%Model Gap · concept · 97%Requirement gap · concept · 97%Assurance-revision loop · process · 96%Hallucination · failure-mode · 95%Reward Hacking · failure-mode · 95%Implementation-verification loop · process · 94%Human judgment · resource · 93%Stakeholder intent · concept · 92%Deployment evidence · data · 90%

Relation Signals (11)

Two-gap framework → consistsof → Requirement gap

confidence 98% · We call these together - requirement gap and model gap - the two-gap framework

Two-gap framework → consistsof → Model Gap

confidence 98% · We call these together - requirement gap and model gap - the two-gap framework

Requirement gap → causes → Reward Hacking

confidence 95% · reward hacking exploits omissions in the requirements or model

Model Gap → causes → Reward Hacking

confidence 95% · reward hacking exploits omissions in the requirements or model

Requirement gap → widenedby → Hallucination

confidence 94% · hallucination widens the gaps by fabricating requirements or environment assumptions

Model Gap → widenedby → Hallucination

confidence 94% · hallucination widens the gaps by fabricating requirements or environment assumptions

Assurance-revision loop → narrows → Requirement gap

confidence 93% · uses deployment evidence to revise the requirements... to continuously narrow them

Assurance-revision loop → narrows → Model Gap

confidence 93% · uses deployment evidence to revise the requirements, model, or evaluator... to continuously narrow them

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Software development follows an implementation-verification loop in which developers or agents iteratively revise an implementation until an evaluator, such as a test suite, accepts it. The evaluator checks the implementation against a set of requirements under a model of the deployment environment. Yet even a formal proof that the implementation satisfies the requirements under the model cannot guarantee acceptable behavior after deployment. Requirements only approximate stakeholder intent, and the model only approximates the real deployment environment. We call these together - requirement gap and model gap - the two-gap framework, which unifies the main failure modes of agentic software engineer-ing: reward hacking exploits omissions in the requirements or model, while hallucination widens the gaps by fabricating requirements or environment assumptions. Because neither gap can generally be certified closed in an open, changing world, the goal shifts from closing them to continuously narrowing them. We therefore propose an assurance-revision loop that uses deployment evidence to revise the requirements, model, or evaluator when stakeholders reject the resulting behavior. We then cast assured agentic development as a resource-allocation problem over human judgment, agent capability, and compute. The two principal bottlenecks mirror the two gaps: human judgment for the requirement gap and faithful, costly evaluation for the model gap. Reality remains the final verifier: acceptable behavior under actual deployment conditions is the ultimate test, while predeployment evaluations remain proxies for it.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

106,289 characters extracted from source content.

Expand or collapse full text

Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering Conference: arXiv preprint; September 2026; Alexander Krentsel∗, Shubham Agarwal∗, Mert Cemri∗, Shu Liu∗ Sidharth Sankhe, Ziming Mao, Matei Zaharia, Ion Stoica UC Berkeley © none Abstract. Software development follows an implementation–verification loop in which developers or agents iteratively revise an implementation until an evaluator, such as a test suite, accepts it. The evaluator checks the implementation against a set of requirements under a model of the deployment environment. Yet even a formal proof that the implementation satisfies the requirements under the model cannot guarantee acceptable behavior after deployment. Requirements only approximate stakeholder intent, and the model only approximates the real deployment environment. We call these together — requirement gap and model gap — the two-gap framework, which unifies the main failure modes of agentic software engineering: reward hacking exploits omissions in the requirements or model, while hallucination widens the gaps by fabricating requirements or environment assumptions. Because neither gap can generally be certified closed in an open, changing world, the goal shifts from closing them to continuously narrowing them. We therefore propose an assurance–revision loop that uses deployment evidence to revise the requirements, model, or evaluator when stakeholders reject the resulting behavior. We then cast assured agentic development as a resource-allocation problem over human judgment, agent capability, and compute. The two principal bottlenecks mirror the two gaps: human judgment for the requirement gap and faithful, costly evaluation for the model gap. Reality remains the final verifier: acceptable behavior under actual deployment conditions is the ultimate test, while predeployment evaluations remain proxies for it. Keywords: system synthesis, assurance, testing, formal verification, AI coding agents, reward hacking, runtime monitoring 1. Introduction Figure 1. Inner implementation-verification loop. One run holds R, M, and E fixed while revising P. The requirement and model gaps lie outside the loop. AI coding agents generate code and tests at a speed and cost no human team can match (Yang et al., 2024; GitHub, 2026a; GitHub, 2026b), with failures accumulating just as rapidly. In July 2026, agents evaluating cybersecurity benchmarks escaped their sandbox, reached the internet, and compromised production infrastructure in “an attempt to cheat the evaluation” (OpenAI, 2026; Hugging Face, 2026c; Hugging Face, 2026a). An agent tasked with speeding up a key-value store delivered a sixfold throughput gain that passed every correctness test by regenerating the benchmark’s predictable values on the fly instead of storing them (Liu et al., 2026b). Leading benchmark audits have found solutions accepted due to vague descriptions and flawed grading harnesses (Chowdhury et al., 2024; Wang et al., 2025). Agents routinely invoke packages that do not exist, and attackers register those names in advance (Spracklen et al., 2025). Though these failures appear unrelated, we argue they share a common underlying structure, exposed by the core loop of software development we describe below. In this loop, which we call the implementation-verification loop, a developer or coding agent starts from a set of requirements R and an environment model M, and iterates on an implementation P until an evaluator E accepts it (Figure 1). Once accepted, P is deployed. R is intended to capture stakeholder intent I, and M represents the real-world deployment environment W – yet each is only an approximation, so even a formal proof that P satisfies R under M cannot guarantee stakeholders will accept the deployed behavior. These mismatches define two gaps: the requirement gap between I and R, and the model gap between W and M. We call this structure the two-gap framework. Viewed through this framework, reward hacking exploits the two gaps: an agent finds an implementation that satisfies R under M while defeating the stakeholder’s actual intent I. In the earlier key-value store example (Liu et al., 2026b), omitting the need to persist arbitrary values was a requirement gap, while evaluating with predictable values was a model gap, since values in the real world are arbitrary. Hallucination works in the opposite direction: rather than exploiting what R and M omit, it widens the gaps from within by introducing fabricated requirements or assumptions—an unstated requirement, such as evicting entries older than thirty days is a requirement gap, while assuming a nonexistent API is a model gap. These gaps are not unique to AI agents; they have been recognized for decades across software engineering and other fields. Brooks (Brooks, 1987) emphasized the difficulty of deciding precisely what to build, while Smith (Smith, 1985) argued that formal correctness within a model cannot establish that the model adequately represents the real world. In AI, McCarthy’s qualification problem captures the difficulty of stating every condition under which an intended outcome should hold (McCarthy, 1977). In economics, incomplete-contract theory recognizes that contracts cannot specify every relevant contingency (Hart and Moore, 1988; Hadfield-Menell and Hadfield, 2019). These works substantiate the claim that neither gap can generally be certified closed in an open, changing world (Section 3). AI agents magnify the risks posed by these gaps. They generate and evaluate candidate implementations orders of magnitude faster than any human, optimizing relentlessly against a fixed evaluator (Yang et al., 2024; GitHub, 2026a; GitHub, 2026b; Cui et al., 2026; Becker et al., 2025). At the same time, they often lack domain- and organization-specific context that experienced developers use to compensate for omissions in R or M, leading to reward hacking (Amodei et al., 2016; Skalse et al., 2022; Pan et al., 2024; Fenwick, 2026). Agents can also widen the gaps directly, hallucinating requirements into R and interfaces into M (Section 4.2). Automated deployment propagates any false acceptance quickly and at scale, and adding more reviewing agents cannot eliminate the risk when they share the same incomplete artifacts (Section 4.4). As the gaps lie outside the inner implementation-verification loop, narrowing them requires an outer assurance-revision loop. Drawing on decades of safety and security practice, this loop treats agent-generated software as possibly compromised, and relies on human stakeholders to close the gaps: human judgment and agent assistance combine to detect misbehavior, limit its impact, gather deployment evidence, and revise R, M, or E, after which the inner loop reruns (Section 5). Together, the two loops implement software assurance: the continuing process of establishing and maintaining justified confidence that P will satisfy I when deployed in W (Rushby, 2009; ISO/IEC/IEEE, 2025; Avizienis et al., 2004). Within this two-loop architecture, stakeholder judgment becomes a critical resource: humans must retain authority over high-stake judgments and remain in the outer loop for the foreseeable future (Section 3.1). We thus frame assurance as an optimization problem: balancing human judgment, agent capability, and compute to reach a required level of assurance at minimum cost and time. As implementation becomes cheap, the emerging bottlenecks will become accountable human judgment for the requirement gap and faithful, often costly evaluation for the model gap. This paper makes the following contributions: • A two-gap framework. We explain reward hacking and hallucination through the requirement and model gaps. We argue that neither gap can generally be certified closed in open, changing systems, and show how agents magnify these gaps (Sections 2, 3, 3 and 4). • A two-loop architecture for software assurance. We pair implementation and verification with an outer assurance-revision loop that uses deployment evidence to revise requirements, models, and evaluators. Humans retain authority over critical judgments (Section 5). • A research agenda for assured agentic development. We frame assurance as a resource allocation problem over human judgment, agent capability, and compute. We identify human judgment and faithful evaluation as the principal bottlenecks (Section 6). 2. The Inner Implementation-Verification Loop and Its Limits This section presents the inner implementation-verification loop and the two fundamental gaps that can cause an implementation that is accepted by the evaluator to fail when deployed in the real world. Figure 1 and Table 1 summarize the architecture and notation. 2.1. The inner loop with fixed R, M, and E Given a set of requirements R, an environment model M, and an evaluator E, a developer or AI agent iterates on an implementation P until E⁡(P,R,M)=.E(P,R,M)=pass. When clear from context, we refer to R as “requirements” and to M as “model”. R states the required behavior, and may come in the form of a product requirement document, acceptance criteria, or a formal specification. M states the assumptions and conditions in the deployment environment that are considered relevant to the task. In our key-value example, R might include API semantics, data consistency semantics, and performance goals, while M might include machine types, failure modes, and testing workload. E is the acceptance procedure, which returns both a pass/fail decision along with diagnostic feedback to help revise P (Meyer, 1992; Hoare, 1969; Rushby, 2009). E might include code reviews, static analysis, test suites, or machine-checked proof, while the feedback might include errors and performance profiling data. In some cases, M is implicit in E’s assumptions about the real world rather than represented as an explicit artifact. R aims to capture stakeholder intent I. A stakeholder may be P’s user (e.g., an application programmer using a database), or a delegate that represents the user. Examples of such delegates are product managers, domain experts, and safety and compliance reviewers. A delegate is expected to generate I by getting detailed feedback from users, and by relying on policies, standards, previous experience, and established engineering practice. M aims to capture all relevant assumptions and conditions during real-world deployment W. This structure creates two gaps, a requirement gap between R and I, and a model gap between M and W. In our key–value store example, the requirement gap is that the specification does not state that the system must store client-supplied values, whereas the model gap arises because the benchmark generates predictable values rather than the arbitrary values encountered in the real world. Both gaps are external to the inner loop: they concern how faithfully the loop’s inputs represent intent and the world, not how the loop executes (Figure 1). These gaps are fundamental: closing them would require perfectly capturing every stakeholder expectation and every possible real-world condition relevant to P. Section 3 argues that none of these gaps can generally be certified as closed in an open, changing world. The two gaps are conceptually distinct. R states what the system should do, while M represents the conditions under which the system must do it. Narrowing one does not necessarily narrow the other. In our key-value store example, a more representative workload would narrow the model gap but would not add the missing storage requirement to R. Strengthening R would narrow the requirement gap, but it would not address the model gap caused by predictable benchmark values that failed to reflect real-world workload. Nor would a machine-checked proof close either gap. It could establish that candidate implementation P meets R for every execution admitted by M, but it could not establish anything outside the inner loop where the two gaps lie. These gaps can cause failures during deployment. A deployment misbehavior is a deployed P in W that stakeholders reject under I. A false acceptance occurs when P is accepted by E (i.e., E⁡(P,R,M)=E(P,R,M)=pass), although P violates I under some relevant condition in W. Not every false acceptance results in a deployment misbehavior, e.g., if the relevant condition may never occur during deployment, or an independent check outside the inner loop may block deployment. Conversely, deployment misbehavior need not imply false acceptance. P may satisfy I under the conditions prevailing when E accepts it but become unacceptable after I or W changes. For example, a GPU program P may pass E and run correctly in deployment until a driver update reduces its available memory, causing an out-of-memory failure of P. R may also specify an explicit optimization objective, such as maximizing throughput, minimizing latency, reducing cost, or lowering memory consumption, which E measures. Reward hacking is a false acceptance in which the inner implementation-verification loop selects P because behavior that violates I improves this objective by benefiting from a requirement or model gap. Not every false acceptance is reward hacking: a latent race may simply escape detection by E. By contrast, if the agent removes locks to improve throughput, thereby introducing a race, and E accepts it because M omits the workload that would expose the race, the result is reward hacking. Hallucination is a complementary failure mode. Rather than exploiting an omission in R or M, the agent fabricates evidence: it behaves as if R contained a requirement that I does not support, or as if M contained an assumption that W does not satisfy. The recorded artifacts remain unchanged; the agent operates on fabricated versions of them, widening the gaps from within. Unlike reward hacking, hallucination is defined by the fabrication rather than by E’s verdict: it produces a false acceptance only when E inherits the fabricated assumption or lacks the checks to detect it (Section 4.2). Table 1. Notation and definitions. Term or symbol Meaning Stakeholders The humans authorized to judge deployment behavior, resolve tradeoffs, and revise R. They may include end users or delegated representatives (e.g. product managers). Stakeholder intent, I What responsible stakeholders want the deployed behavior to be. It has no authoritative recorded form: any statement of it belongs to R, which records a partial, possibly imperfect approximation. Requirements, R The recorded constraints on acceptable behavior and the current explicit approximation of I. Real world, W The full world in which P is deployed and evaluated by stakeholders, including known and unknown conditions and mechanisms. Environment model, M The conditions and mechanisms represented or assumed when interpreting verification. It abstracts the aspects of W believed relevant. (Note: “model” does not mean language model.) Implementation, P The code and configuration being developed and assessed. Evaluator, E The acceptance procedure that assesses P against R under M. It can return an acceptance decision and feedback used to revise P. Assurance Justified confidence that P will satisfy I when deployed in W (ISO/IEC/IEEE, 2025). Requirement gap Difference between R and I. R omits, distorts, or conflicts with stakeholder intent. Model gap Difference between M and W: M omits or misrepresents deployment conditions. Evaluation gap E accepts P although P violates R on some execution admitted by M. Internal to the inner loop; closable in principle for fixed R and M. Deployment misbehavior Behavior produced by deployed P that violates I within the specified deployment scope and is therefore rejected by responsible stakeholders. False acceptance An outcome in which E accepts P against R under M, but P would violate I under at least one relevant condition in W within the specified deployment scope. May only be uncovered at deployment-time. Reward hacking A false acceptance produced when adaptive optimization selects P because its performance against E benefits from a gap between R and I or between M and W. Hallucination The agent operates on a fabricated version of R or M: a requirement that I does not support or an assumption that W does not satisfy. It widens the gaps from within, whereas reward hacking exploits existing ones. It may, but need not, produce a false acceptance. 2.2. The requirement gap The requirement gap separates stakeholder intent I from the stated requirements R. Because intent evolves as stakeholders learn, R is only an explicit, partial snapshot of I at the time requirements are formulated. In many cases, stakeholders discover what they want only by interacting with P (Section 6.2). The requirement gap can arise through omission, ambiguity, conflict, or change. “Put and get values” may omit the requirement to retain the value across a crash (omission). “Prioritize VIP traffic” may mean different things to the networking and billing teams (ambiguity). A strict latency target may conflict with a requirement to encrypt every request, without specifying which takes priority (conflict). Finally, a requirement that was once adequate may cease to reflect stakeholder intent as products, regulations, or accepted tradeoffs evolve (change). 2.3. The model gap The model gap is the difference between the environment model M and the real world W. M captures only the conditions and mechanisms deemed relevant to the task (Jackson, 1995; Smith, 1985). This selectivity is not a flaw; abstraction is necessary to make evaluation tractable. However, the gap matters when M omits an aspect of W that determines whether the implementation P actually satisfies stakeholder’s intent. In an open, changing world, no predeployment procedure can generally establish that M captures every future condition in W that could affect the outcome. Relevant aspects of M fall into two broad categories: execution assumptions and evaluator assumptions. Execution assumptions describe the conditions under which P runs, including inputs, workloads, timing, resources, failures, platforms, and external services. Evaluator assumptions describe how E observes and judges P, including its abstractions, tools, and the mapping from source code to deployed behavior. A model gap arises when either type omits or misrepresents a condition that affects whether acceptance by E predicts acceptable deployment behavior. For example, predictable benchmark values may not represent arbitrary deployment values. A crash consistency proof may assume flush semantics that the device does not provide. A test may use an API stub that ignores rate limits. Models also age as software, dependencies, hardware, and operating procedures change (Lehman, 1996; Parnas, 1994). Assumptions that were once true may later cease to be so. Important assumptions should therefore be explicit, monitored, and versioned. 2.4. The evaluation gap: an internal, closable gap The requirement and model gaps lie outside the inner loop and, as we argue in Section 3, generally cannot be certified closed. By contrast, a third gap, which we call the evaluation gap, lies within the inner loop and can in principle be closed. It arises when E accepts an implementation P even though P violates R on an execution admitted by M, commonly because testing does not cover every such execution. This gap can produce a false acceptance when the violated requirement reflects stakeholder intent. Under our definition, however, it is not reward hacking: it reflects E’s failure to enforce R under M, rather than the agent’s exploitation of a mismatch between R and I or between M and W. For fixed R and M, a sound formal proof closes the evaluation gap by establishing that P satisfies R for every execution admitted by M, assuming that the proof system is sound and its trusted computing base, including the proof kernel, is correct. Verification efforts at AWS and projects such as seL4 and IronFleet have established such guarantees for real systems (Newcombe et al., 2015; Klein et al., 2009; Hawblitzel et al., 2015; TLA+ Foundation, 2026). These proofs, however, do not show that R faithfully captures I or that M faithfully represents W. We therefore focus on the external gaps, which persist even when E perfectly verifies P against R under M. To summarize, all three gaps affect assurance. The inner loop supports the claim that P satisfies R under M. The top-level assurance claim is that P satisfies I in W. The evaluation gap weakens the evidence for the inner claim. The requirement and model gaps weaken the connection between the inner claim and the assurance claim. Closing the evaluation gap therefore strengthens verification but cannot overcome the two external gaps. Narrowing those gaps strengthens assurance. 3. The Two Gaps in Practice To make these gaps more concrete, Table 2 maps a range of real-world failure cases to their gaps (in R, M, or both) and possible fixes. Each case shows how an unresolved gap can lead to the acceptance of an implementation that violates stakeholder intent. Table 2. Illustrative cases relevant to the two gaps. Labels: [RH] gap exploited by reward hacking; [H] gap widened by hallucination; [FA] gap caused false acceptances without established reward hacking. Example What happened Requirement gap Model gap Fixes Key value store reconstruction (Liu et al., 2026b) An optimizer stored a seed and regenerated benchmark values, reporting about 6×6× higher throughput without storing the supplied values. [RH] The storage requirement omitted preservation of arbitrary client information. [RH] Evaluation values were reconstructible. Deployment values need not be. Repair both the requirement and the evaluation distribution. Mixture-of-Experts (MoE) load balancing (Cheng et al., 2025) An optimizer improved the load balancing objective by assigning no traffic to some experts, thereby avoiding service rather than balancing it. [RH] The load balance objective omitted service coverage. No model gap is needed to explain the result. Add service coverage to R and E. SWE-bench audits (Chowdhury et al., 2024; Wang et al., 2025) Screening for SWE-bench Verified removed tasks with vague issue statements or tests that rejected valid fixes (Chowdhury et al., 2024); even on the screened subset, 7.8% of “correct” patches fail the developers’ own tests (Wang et al., 2025). [FA] Issue statements omitted expected behavior, admitting multiple plausible readings of the task. [FA] Environment and dependency mismatches are model gaps; overly permissive benchmark tests are an evaluation gap. Separate and repair task, model, and evaluator faults. Sandbox escape and Hugging Face breach (OpenAI, 2026; Hugging Face, 2026c; Hugging Face, 2026a; Cloud Security Alliance AI Safety Initiative, 2026) During evaluation on a cybersecurity benchmark, an OpenAI model exploited a zero-day in the sandbox’s package-registry proxy, reached the internet, and compromised Hugging Face’s production systems, seeking the benchmark’s reference solutions. [RH] The task rewarded obtaining solutions without stating that they must be derived within the evaluation environment. [RH] The evaluation environment was assumed isolated; in W, an egress path and production credentials were reachable. Treat harness isolation as an assumption in M; verify it (deny-by-default egress) before evaluation (Section 5.1). Claude evaluation breaches (Anthropic, 2026) In three of 141,006 reviewed evaluation runs, Claude models performing capture-the-flag tasks reached the internet through a misconfigured partner environment and accessed production systems of three organizations, despite prompts stating no internet access; the models treated the real systems as part of the exercise. [RH] R stated no obligation to halt or escalate when the environment contradicted its stated assumptions. [RH, H] M asserted isolation that a misconfiguration falsified; the models’ operative M treated real targets as simulated. Verify isolation; add stop-and-escalate requirements to R; monitor assumption violations (Section 5.2). The natural question is whether better requirements, richer models, and stronger verification can certify that the gaps are closed. For open and changing systems, we claim the answer is generally “no”. The available evidence – recorded R and M, recorded stakeholder judgments, E’s results, and observations from testing and deployment – captures only an incomplete projection of I and W: it can narrow the gaps, not provably close them. We discuss the fundamental challenges unique to each gap in turn. 3.1. Why the requirement gap is fundamentally hard to close Certifying the requirement gap as closed would require showing that, across all relevant executions in W, the behaviors permitted by R are exactly those that responsible stakeholders would judge acceptable. One approach could be to record in R the stakeholder judgments for all possible executions of P; however, this is not generally feasible for three reasons. First, stakeholder intent is partially tacit. People can often judge a concrete behavior more readily than state the general rule governing such behavior (Polanyi, 1966; Nuseibeh and Easterbrook, 2000; Feigenbaum, 1977; Buchanan and Shortliffe, 1984; Brooks, 1987). Requirements that appear obvious may remain unstated until an implementation violates them. In the key-value store example, few would think to state that a store must actually store the values clients supply – certainly we didn’t! Second, stakeholders do not know what they do not know. They refine their intent as they interact with an implementation, observe unexpected behavior, and reconsider previously unconsidered tradeoffs (Nuseibeh and Easterbrook, 2000; McCarthy, 1977; Hart and Moore, 1988; Hadfield-Menell and Hadfield, 2019; Potts et al., 1994). For example, a stakeholder may recognize the need for human approval only after an agent processes its first four-figure refund. Third, it is generally infeasible to assess the behavior of every relevant execution in W. This would require describing every condition in W that could affect whether P satisfies I; a complete description of those conditions would itself define a model M that omits nothing relevant about W, equivalent to closing the model gap. But as Section 3.2 argues, that gap cannot generally be certified closed in an open, changing world. A second approach would replace enumeration with a predictive surrogate for stakeholder judgment. A surrogate learned from past judgments is underdetermined: one might match every observed judgment yet diverge on unseen cases. A formally specified surrogate would instead have to capture how stakeholders apply tacit knowledge, handle unfamiliar cases, and revise their views as evidence changes: in effect, a “human simulator” for the task, which remains an open challenge. Neither enumeration nor a surrogate, then, can generally certify the requirement gap closed. 3.2. Why the model gap is fundamentally hard to close Certifying the model gap as closed would require showing that M represents every condition in W that could affect whether P satisfies I. Because this claim concerns what M omits or misrepresents, it cannot be established by reasoning within M alone (Jackson, 1995; Smith, 1985). Expanding M shifts the question to whether the expanded M faithfully represents W. One could try to bypass M and evaluate P directly in W. But many relevant conditions are too numerous, costly, slow, or risky to test (e.g., large-scale outages or security breaches) – precisely the reason the inner loop typically evaluates P using simulators, emulators, and test environments rather than W itself. Further, because W is open and changing, some conditions are unknown at evaluation time or arise only during deployment. Direct evaluation can therefore narrow the model gap but cannot certify it closed. A second approach would provide a formal specification of W and prove that P satisfies R under every execution the specification admits. But any such specification is itself a model M. A proof can establish that P satisfies R for every execution represented by M, but it cannot establish that M includes every relevant condition in W (Smith, 1985; Hoare, 1969; Rushby, 2009; Fetzer, 1988; Fonseca et al., 2017). Adding detail moves the boundary of the model without establishing that nothing consequential remains outside it. Formalization can expose assumptions and narrow the model gap, but it cannot by itself certify closure. Both approaches also face a moving target. Workloads, dependencies, hardware, organizations, and adversaries evolve, and deploying P may itself change user or system behavior (Lehman, 1996; Parnas, 1994). A model that adequately represents W today may therefore become inadequate tomorrow. Permanently certifying the gap as closed would require anticipating every future change that could affect P. The arguments in Sections 3.1 and 3.2 concern open and changing systems. They do not preclude certifying the gaps as closed when both stakeholder intent and the real world are explicitly bounded. For example, assume the stakeholder intent for an 8-bit adder is limited to one functional property: for every pair of 8-bit inputs, the adder must output (x+y)mod256(x+y) 256 under fixed Boolean semantics. R can state this property exactly, while M can represent all possible input pairs and the Boolean semantics governing the circuit. Both gaps can then be certified closed, but only for this property: timing, power consumption, temperature tolerance, and fault behavior would each require extending R and M, with their own potential gaps. Appendix A discusses other bounded domains, such as mathematics and hardware design, in which the gaps can be narrowed or even closed for specific properties. 4. AI Agents Magnify the Gaps The requirement and model gaps long predate AI agents. McCarthy’s qualification problem, incomplete-contract theory, and work in software engineering and expert systems have long identified the difficulty of creating and maintaining complete specifications, and environment models (Jackson, 1995; Lehman, 1996; Parnas, 1994; Feigenbaum, 1977; Buchanan and Shortliffe, 1984; Brooks, 1987; McCarthy, 1977; Hart and Moore, 1988; Hadfield-Menell and Hadfield, 2019). Research on principal-agent theory, Goodhart’s law, and Campbell’s law has similarly examined failures caused by imperfect requirements and proxies (i.e., environment models) (Amodei et al., 2016; Skalse et al., 2022; Krakovna et al., 2020; Jensen and Meckling, 1976; Goodhart, 1984; McCubbins and Schwartz, 1984; Campbell, 1979). What is new is how AI agents turn these longstanding gaps into far greater risks: systematic search exploits omissions, hallucination widens the gaps, and deployment at scale rapidly propagates the resulting failures. We describe each mechanism below, and explain why scaling automated review alone cannot adequately mitigate either gap. 4.1. Agents Systematically Exploit the Gaps At its core, reward hacking occurs when an AI agent exploits an omission in R or M to maximize its assigned objective, such as increasing efficiency or reducing cost. Because E evaluates P against R under M, it may accept an implementation that takes advantage of what these artifacts fail to capture. The implementation therefore succeeds according to E while producing behavior that violates stakeholder intent I. In effect, the agent optimizes what the evaluator measures rather than the outcome stakeholders actually want. Reward hacking requires neither deception nor an intent to evade evaluation (Amodei et al., 2016; Skalse et al., 2022; Pan et al., 2024). A cleaning robot rewarded for the amount of dust it collects may dump dirt on the floor and sweep it up again simply because doing so maximizes its score! Human developers may be less susceptible because they interpret R and M through context that the artifacts do not state. This includes local domain and organizational knowledge, shared context with stakeholders, and deployment experience, but also engineering “common sense” accumulated through decades of practice. Agents bring broad knowledge from training and can acquire task-specific context, but they cannot be assumed to recover or reliably apply all this tacit knowledge when the artifacts omit it. In the key value store example, although R did not explicitly require it, the expectation that a store preserve arbitrary client values is part of basic systems common sense. A human expert would therefore likely reject an implementation that never stores client values, regardless of its benchmark score. The examples in Table 2 exhibit the same contextual asymmetry. Agents also intensify reward hacking by iterating orders of magnitude faster than humans. They can generate candidates in parallel and revise P repeatedly while R, M, and E remain fixed. Each failure, stack trace, benchmark score, profiler report, or review comment reveals more about what E rewards. Frameworks such as GEPA and SkyDiscover demonstrate how effectively such feedback can guide adaptive search (Agrawal et al., 2026b; Liu et al., 2026a). When later candidates are optimized using this feedback, E serves as both the optimization signal and the verifier, favoring implementations fitted to E rather than to I and W (Dwork et al., 2015; Blum and Hardt, 2015). In summary, while feedback from E can accelerate genuine improvement, it can also help an optimizer discover and exploit omissions in R or M. 4.2. Hallucinations Widen the Gaps from Within As defined in Section 2.1, hallucination adds unsupported content to the agent’s operative R or M; here we examine why agents produce it and when E fails to catch it. An agent that generates code assuming a nonexistent API or package effectively adds to M an environmental assumption unsupported by W, widening the model gap. An agent that invents a business rule, such as silently dropping duplicate refund requests, adds to R a requirement unsupported by I, widening the requirement gap. The two failure modes are therefore related but distinct: reward hacking adaptively exploits existing gaps, whereas hallucination widens them from within by introducing unsupported requirements or environment model assumptions. In the key–value store of Section 1, the implementation that stores only the seed and regenerates values is reward hacking: search found a shortcut, enabled by missing the requirement to store values, that E rewards. An agent that instead calls a nonexistent compression API is hallucinating: nothing rewarded the fabrication, and a grounded E would catch it at once. 4.3. Automatic Deployment Amplifies the Consequences Automatic deployment amplifies the consequences of both failure modes. Once E accepts a change produced by reward hacking or hallucination, an agentic pipeline may propagate it across repositories and services or deploy it to a large user population. Scale does not cause either failure mode, but it allows a single false acceptance to affect many users before stakeholder review or deployment evidence exposes it. 4.4. Scaling Automated Review Alone Is Not Enough A natural response to both failure modes is to scale automated review alongside generation. Reviewers using different agents or tools can catch ordinary defects, suspicious optimizations, and fabricated dependencies missed by the generating agent. However, reviewers relying on the same R, M, E, and context evaluate P against the same premises. More reviews may scrutinize P more thoroughly without providing new evidence that R captures I or that M represents W. Reviewers may therefore overlook an omission exploited by reward hacking or accept an unsupported assumption introduced by hallucination. Narrowing the gaps requires evidence and authority from outside the inner loop: human stakeholder judgment to verify R against I, and empirical deployment evidence to verify M against W. While agents can assist in gathering this evidence, additional reviewers constrained to the same static artifacts cannot substitute for stakeholder intent or real-world feedback (Section 5). 5. The Outer Assurance-Revision Loop: Design for Misbehavior A strong coding agent produces a P that passes E, but that alone does not establish sufficient assurance as requirement and model gaps may remain. The outer assurance–revision loop builds and maintains support for the top-level assurance claim as evidence, stakeholder intent, and deployment conditions evolve (Rushby, 2009; Calinescu et al., 2018; Avizienis et al., 2004). It leverages familiar mechanisms, including code review, staged deployment, A/B testing, production monitoring, and incident response (Figure 2). Agentic development makes these mechanisms critical: agent-generated code must be treated as potentially compromised (Section 5.2), and failures must trigger revisions to the artifacts that enabled them rather than merely patches to P (Table 3). The humans in the outer loop include people accountable for approving, deploying, and operating P, supported by AI agents and tools. Agents should perform as much outer loop work as evidence and risk permit, while accountable humans retain authority over consequential decisions. Human authority does not imply human review of every step, e.g., humans can delegate routine bounded decisions. Evidence imposes cost and delay, so assurance effort should scale with risk. Routine changes may rely on inexpensive, reusable automation, whereas changes with serious consequences, uncertain assumptions, or limited reversibility require more realistic evaluation and greater human attention. Productivity should be measured across the complete workflow at the required assurance level. If agents cannot improve that workflow, their autonomy should be decreased. Figure 2. Outer assurance-revision loop. Accountable authority and empirical evidence drive revisions to P, R, M, E, or deployment controls before the inner loop reruns and bounded deployment resumes. The outer loop should maintain a versioned assurance argument that links the top-level assurance claim to supporting evidence, assumptions, approved scope, owners, and invalidation conditions (Calinescu et al., 2018; Weyns et al., 2018; ISO/IEC/IEEE, 2025). New evidence may strengthen or weaken the argument, narrow its scope, or invalidate the claim. The assurance argument makes clear why, and within what limits, the system may operate autonomously and defines explicit conditions for intervention by operators. Any evidence that materially weakens confidence in P can trigger the outer loop before or after deployment. The assurance team should then: (1) Preserve the evidence needed to reproduce the behavior and its operating context. (2) Determine whether P, R, M, E, or an operational safeguard must change. (3) Revise the responsible artifacts and safeguards. (4) Rerun the inner implementation and verification loop. (5) Release or redeploy gradually, with bounded exposure, explicit stop conditions, and tested recovery. For our key-value example, these steps would involve revising R to require preservation of arbitrary client-supplied values, modifying the benchmark to use unpredictable values, potentially adding a runtime monitor that checks whether returned values match those originally written, and redeploying through a canary rollout. 5.1. Treat agent-generated software as possibly compromised We should treat agent-generated software as potentially compromised until independent evidence establishes otherwise, following standard security practice for untrusted code (OWASP Foundation, 2025). This is an adaptation of Thompson’s classic lesson: one cannot fully trust code one did not create, and inspection alone cannot establish that trust (Thompson, 1984). This stance does not presume malice; reward hacking requires neither deception nor intent (Section 4.1). Review alone is also insufficient: a human or agent reviewer relying on the same inner-loop artifacts may inherit the same blind spots as the agent that generated the software (Section 4.4). Within the outer assurance-revision loop, we must therefore embed safeguards in the deployment pipeline, including isolation, least privilege, staged deployment, monitoring, and rollback. These safeguards limit harm, gather deployment evidence, and trigger rollback or revisions to R, M, or E when deployed behavior violates I. The July 2026 evaluation breaches summarized in Table 2 (Section 3) illustrate why agent-generated software should be treated as untrusted. In each breach, evaluators reduced the agents’ built-in safeguards. The surrounding evaluation and deployment pipeline then failed to enforce verified isolation, a containment safeguard of the outer assurance-revision loop, allowing the agents to reach production systems (OpenAI, 2026; Hugging Face, 2026c; Hugging Face, 2026a; Anthropic, 2026). In the Hugging Face breach, the agent stole the benchmark reference solutions, behavior the forensic analysis called “an attempt to cheat the evaluation” (Hugging Face, 2026a). This was the result of reward hacking: M assumed isolation, but W exposed production systems, and the agent used this mismatch to improve its evaluation result. In the Claude breaches, the agents used unintended production access to continue their assigned tasks. This was also reward hacking as this access provided an unintended way to advance the evaluation objective. It was also a hallucination. Despite evidence that they had reached real systems, the agents continued to behave as though those systems were part of the simulated evaluation. They effectively adopted a fabricated model M in which the production systems remained part of the simulation, which was false in W (Anthropic, 2026). The postmortems attributed the breaches to failures of these outer-loop safeguards. Anthropic described its breaches as “closer to a harness and operational failure than an AI model alignment failure” (Anthropic, 2026), while independent analyses identified permitted egress and reachable credentials as conditions that established security practices should eliminate (Cloud Security Alliance AI Safety Initiative, 2026). These breaches show how failures in outer-loop containment can turn reward hacking and hallucination into production security incidents. Engineers should therefore embed outer-loop safeguards, assume that isolation can fail, and limit the impact. 5.2. Employ defense in depth: established safety and security practice The defenses we discuss here are not new; they draw on decades of work in security engineering, dependability, continuous delivery, and site reliability engineering (Avizienis et al., 2004; Beyer et al., 2016; Saltzer and Schroeder, 1975; Humble and Farley, 2010). What changes is that under the assumption that the code is potentially compromised these defenses are no longer optional safeguards but essential parts of the workflow. If an implementation may be compromised, the surrounding pipeline must limit the resulting harm. We next organize these practices around the outer loop’s three objectives: reduce the frequency of deployment misbehavior, limit its impact, and prevent its recurrence. Reduce the frequency of deployment misbehavior. Before deployment, stakeholders should narrow the requirement gap by refining R, e.g., by building prototypes, stating boundary cases, and providing examples of acceptable and unacceptable behavior. Adversarial reviewers should search for implementations that satisfy R while defeating I. To narrow the model gap, the assurance team should make explicit assumptions about workloads, failure modes, dependencies, hardware types, and evaluator behavior, and test them using techniques such as metamorphic and differential testing, fault injection, replay, as well as experiments on target hardware (Barr et al., 2015; Segura et al., 2016; McKeeman, 1998). To reduce overfitting to M, the final evaluation should use inputs and evidence withheld from iterative development (Dwork et al., 2015; Blum and Hardt, 2015). Finally, one should use techniques such as contracts, static analysis, runtime checks, and agent-generated proofs to strengthen verification where the risk justifies their cost (Meyer, 1992; Hoare, 1969; Rushby, 2009; Agarwal et al., 2026). Limit impact through least privilege and isolation. Access privileges for both the agent and the artifacts it produces must be limited to those required for the task, with deny-by-default policies and request verification (Saltzer and Schroeder, 1975; Rose et al., 2020). The trusted region — the code, credentials, and resources whose compromise would be catastrophic — must be kept as small as possible and isolated from agent-generated code by running that code in sandboxes, containers, or lightweight VMs with strict constraints on resource usage, network access, and access rights (Agache et al., 2020). The July 2026 incidents show why isolation matters: blocking network egress and access to production credentials would have contained them (Anthropic, 2026; Cloud Security Alliance AI Safety Initiative, 2026). Figure 3. The research agenda at a glance. Four research directions (bottom) target the structural and resource bottlenecks (middle) that constrain the optimization objective (top), drawing on humans, agents, and compute. Limit impact through staged exposure. Production systems eventually have full exposure by definition. For such systems, we should gradually increase an implementation’s exposure only as evidence supports doing so. For example, we could progress through stages gated on success: (1) replay and simulation; (2) shadow deployment, i.e., mirror production traffic without affecting users (Feitelson et al., 2013); (3) release to a small canary group; (4) gradual release to the user population with automated health checks and tested rollback (Beyer et al., 2016; Humble and Farley, 2010; Beyer et al., 2018). In addition, we can leverage chaos and fault-injection tests to verify containment and recovery in advance before stages (3) and (4) (Basiri et al., 2016). Limit impact through real-time observability. Agents that synthesize P should also generate versioned runtime monitors for each consequential, observable requirement in R and assumption in M (Leucker and Schallhart, 2009; Fickas and Feather, 1995); an assumption violation may invalidate the assurance claim before P visibly violates R. The monitor is not merely an observability mechanism; it is an executable representation of an assumption that must remain valid. However, since monitors derived from the same artifacts may inherit the same omissions, operators must review them and add independent telemetry (Majors et al., 2022). Monitoring can limit impact if each alert triggers a predefined response. Possible responses include halting a rollout, reducing exposure, activating a fallback, or rolling back. When performing a change, we should also bound the users, requests, services, and time interval affected by the change (Beyer et al., 2016). Finally, a monitor that fires on a divergence between M and W turns a silent gap into an explicit trigger for outer-loop revision. Prevent recurrence. When deployment misbehavior occurs, the assurance team should determine whether it resulted from a mismatch between R and I, a mismatch between M and W, or a weakness in E. The assurance team and developers should then revise the affected artifacts (Table 3). A single incident may reveal several causes, as the key–value store example illustrates: developers must add an explicit requirement to store arbitrary client values and use a representative evaluation dataset. Section 6.3 examines the open problem of generalizing such case-specific repairs into requirements, tests, and safeguards for future development. Table 3. Diagnosis and primary repair in the outer loop. Diagnosis Meaning Primary repair Implementation defect (not requirement or model gap) P violates R under conditions represented by M. Its acceptance by E is an evaluation gap (Section 2.4). Revise P, add a regression check to E, and rerun the inner loop. Requirement gap R permits behavior that responsible stakeholders reject. Revise R, extend E to check the new requirement, and rerun the inner loop. Model gap Conditions in W, including execution, deployment, or the actual behavior of E, differ consequentially from M. Revise M, E, monitors, or controls. Revise P if needed, then rerun the inner loop. 6. Research Agenda: Assurance as a Resource Optimization Problem The fundamental problem is an optimization problem: generate an implementation that reaches the required assurance, within acceptable risk and exposure, as quickly and cheaply as possible. Risk is the expected loss from deployment misbehavior (Beyer et al., 2016; Kaplan and Garrick, 1981), which could be assessed by using incident rates, past losses, and predicting deployment outcomes (Beyer et al., 2016). Exposure bounds the consequences of a change by limiting the users, requests, services, data, and the interval of time it can affect. Reaching the desired level of assurance requires evidence for three relationships: that R captures I, that M represents W, and that P satisfies R under M. To solve this problem we can broadly leverage three resources: human judgment, model capability, and compute. As with any optimization problem, we first need to identify the bottlenecks. The two gaps create two structural bottlenecks. Narrowing the requirement gap requires accountable human judgment, while narrowing the model gap requires faithful evaluation, which is often expensive and slow. Among the resources, humans are arguably the scarcest: model capability and compute scale given enough money, but the attention of people with authority over I does not (Cui et al., 2026; Becker et al., 2025). Sections 6.1 and 6.2 target the two structural bottlenecks, Section 6.3 discusses how continual learning can narrow both, and Section 6.4 how to alleviate the human bottleneck. Figure 3 captures our research agenda. 6.1. Narrow the model gap: faithful evaluation at acceptable cost Evaluation methods trade fidelity, which determines the size of the model gap, against coverage and cost. Real-world evaluation provides the highest fidelity but is costly and slow, and suffers from limited coverage (only inputs/scenarios appearing during observed deployment (Lehman, 1996; Parnas, 1994)). One research direction is to specify M and R faithfully and formally enough so that we can employ formal methods (Stoica et al., 2024). This can enable LLMs to generate a machine-checked proof certifying correctness for all inputs along with the implementation (Agarwal et al., 2026; Hubert et al., 2026; Ren et al., 2025; Baba et al., 2025; Li et al., 2024). Recent experiments with TLA+ show this approach can scale well when M is appropriately specified (Newcombe et al., 2015; Klein et al., 2009; Hawblitzel et al., 2015), however validating M against W remains a challenge left to the outer loop. Even with the well-specified M, generating correct code for large, complex specifications remains an open research problem (Agarwal et al., 2026). One approach to resolve the tradeoff between evaluation fidelity and cost is to use a cascade of evaluators ordered by increasing fidelity and cost: e.g., from a parametrized analytical model, to a coarse-grained simulator, to a fine-grained simulator, to an emulator, and finally to the real system. Cheap evaluators in early stages can filter out candidates obviating expensive stages, similar to techniques from multi-fidelity optimization and bandit-based hyperparameter search (Forrester et al., 2007; Li et al., 2018). Computer architects similarly use a cheap roofline model to bound achievable performance (Williams et al., 2009), then gem5 to simulate the microarchitecture (Binkert et al., 2011), then FPGA emulation to test near-native speed (Karandikar et al., 2018), and finally silicon implementation to provide the definitive result. In our key-value store example, the cascade of evaluators might consist of a queueing model, a workload simulator, execution against recorded production traces, and a canary deployment. Two further open research problems are calibration and evaluator selection. First, we must (i) calibrate each stage against the next one such that an implementation P passing an early, cheaper evaluator is likely to pass a later, more realistic one, and (i) detect when changes in W affect this calibration. Second, dynamic evaluator sequencing could aim, at each step, to reduce the most uncertainty per unit cost and provide strong, independent evidence when false acceptance could cause serious harm. Evaluator selection and deployment exposure should therefore be considered jointly: stronger evidence can justify broad, higher-risk exposure (Section 5.2) (Beyer et al., 2016). 6.2. Narrow the requirement gap: iterative clarification Agents can help narrow their own requirement gap through iterative clarification: identify likely omissions, ambiguities, and conflicts in R, and ask stakeholders to resolve them. Agents can surface competing interpretations, construct corner cases, trace the consequences of each interpretation, and ask questions whose answers could change a decision (Nuseibeh and Easterbrook, 2000; Stoica et al., 2024). Asking clarifying questions before generating code has already been shown to reduce implementation bugs (Mu et al., 2024). When should the system ask a stakeholder for guidance? We propose two signals. First, in ambiguity revealed through proactive search: before deployment, construct corner cases for which plausible interpretations of R prescribe different behavior by using coverage-guided techniques such as fuzzing, metamorphic testing, and automated adversarial evaluation (Zhou et al., 2026; Segura et al., 2016; Manès et al., 2021). Second, in missing guidance revealed by new cases during deployment: the system should monitor and escalate when it encounters a scenario in deployment on which prior stakeholder judgment provides no clear guidance, using techniques such as out-of-distribution detection, uncertainty-driven active learning, selective prediction, and learning-to-defer (Bondi et al., 2022; Hendrycks and Gimpel, 2017; Settles, 2009; Mozannar and Sontag, 2020). For example, an agent that has processed thousands of refunds below one hundred dollars should escalate its first four-figure refund. The first signal seeks stakeholder judgment because R is ambiguous; the second seeks it because it reveals a requirement not covered by R. For both, whom to ask and how to present the questions remains an open question: for instance by showing counterexamples first and eliciting an independent judgment before revealing the agent’s recommendation (Parasuraman and Riley, 1997). When no stakeholder is available to resolve an open question, we should limit the agent to reversible actions, such as work in snapshotted isolated containers and lightweight virtual machines (Agache et al., 2020) or applying changes through transactions (Brown and Patterson, 2003). Irreversible or externally visible actions should require explicit stakeholder approval. Reversible actions allow the system to gather evidence while preserving the stakeholder’s ability to decide later. 6.3. Continual learning across both gaps Retaining clarifications and lessons is the key to not repeating mistakes. Because a failure may require changes to R, M, E, P, a monitor, or a deployment policy (Table 3), the research problem is continual system learning: diagnosing which part of the coupled system failed, repairing it, and preserving the repair as intent and real world evolve. We propose scoping retention to a "lesson" we define as a rule that links a class of failures to a fix intended to prevent their recurrence (Beyer et al., 2016). Each lesson includes its diagnosis, supporting evidence, scope, owner, review triggers, and tests of whether the repair remains effective. Furthermore, the lesson should be operationalized by encoding it into a requirement, an evaluator, a monitor, an implementation guard, or a deployment policy. Contradictory evidence should narrow, replace, or retire a lesson. Maintaining a lesson database is not easy. The history of AI provides a cautionary tale. Rule-based expert systems attempted this: building a knowledge base by encoding expert judgment as explicit, maintainable rules. However, even expert-authored knowledge bases grew inconsistent, rules interacted unpredictably, and maintenance became costly (Feigenbaum, 1977; Buchanan and Shortliffe, 1984; Lenat, 1995). Agents magnify this, generating candidate lessons faster than people can validate and reconcile them. While stored lessons transfer useful learnings they can also propagate errors (Zhao and others, 2024; Xiong et al., 2026), and even cause catastrophic forgetting (Parisi et al., 2019). Research should therefore treat lesson maintenance as a first-class problem: diagnose causes, infer scope, detect contradictions, and forget deliberately. A lesson with outdated assumptions must be revised or retired. Cross-system learnings must be transferred carefully, only where intent, authority, and relevant conditions in W are shared. Consider a lesson that doubles timeouts to mask a slow web service: it should be retired when the dependency is fixed, and should not be transferred to a latency-critical service. To maintain a sound knowledge base, we need to track each lesson’s provenance, and govern its activation, transfer, and retirement with validation tests and counterevidence. 6.4. Use humans efficiently We argue that the scarce resource is shifting from implementation effort to human judgment, necessary whenever a decision requires authority over I or depends on missing context. Thus, human efforts should focus on resolving ambiguities in R, judging deployment outcomes, defining acceptable consequences, and approving exposure, while agents automate routine triage, log analysis, monitor generation, and preliminary review. However, automation can leave human operators responsible for rare and difficult cases while depriving them of routine practice necessary to respond effectively. Previous research has documented both over-reliance on automation and the failure to use it appropriately (Parasuraman and Riley, 1997; Bainbridge, 1983). Escalation mechanisms must therefore preserve human skill, situational awareness, and independent judgment, rather than merely keeping humans nominally in the loop. Agents should also reduce the cost of human review by turning raw evidence such as machine logs into human-consumable evidence. MAST, for example, can distill thousands of multi-agent execution traces into just fourteen failure types (Cemri et al., 2025). More generally, agents can synthesize logs and incidents, formulate candidate lessons or revisions to existing ones, and provide an explanation and supporting evidence for each. If the evidence is insufficient, agents can run targeted experiments to gather more. Humans need to review only candidates that are contested, cross an authority boundary, or pose consequential risk. Once approved, each candidate becomes a scoped, versioned lesson that future agents can reuse as long as its scope remains valid. Finally, we must trade risk and cost against exposure, and evaluate our workflows across sequences of both development and deployment, capturing the assurance-revision loop. For this reason, benchmarks are needed that deliberately include hidden omissions in R, changes in W, and faults in E, so that the workflow discovers and addresses these omissions rather than being told where they are located. Evaluations should track human queries, evaluator calls, compute, elapsed time, and deployment exposure. They should report the Pareto frontier across speed, cost, and assurance under a fixed risk bound. Comparisons should include human-driven, fully automated, and selective human–agent workflows. 7. Implications for Agentic Software Engineering The two-gap framework and two-loop architecture we present raise five implications for agentic system design: (1) The gaps are not new; the agents exploiting them are. The requirement and model gaps we present are not to be treated as special case defects; they have been individually recognized across engineering, economics, and political science for decades (Section 3). What is new is the pressure on them. Agents iterate more quickly and cheaply than any human, without the full context and responsibility that keeps humans aligned. Reward hacking is therefore a predictable outcome, requiring neither deception nor intent. The logical implication is to treat agent-generated software as possibly compromised, and design the surrounding pipeline to handle misbehavior. (2) The requirement gap makes humans the bottleneck. Agents can be asked to identify omissions in R and propose interpretations explicitly, but only authorized decision-makers can resolve ambiguities, judge surprising outcomes, and specify tradeoffs. Therefore, accountable human judgment will remain part of the outer assurance-revision loop for the foreseeable future. As agentic implementation capacity grows, this judgment becomes the scarce resource on the critical path. Therefore throughput of such systems will increasingly depend on how efficiently the outer loop elicits, records, delegates, and reuses these judgments (Sections 6.2 and 6.4). (3) The model gap makes evaluation the bottleneck. Every evaluator relies on a model of deployment, so the tradeoff between evaluation overhead and fidelity directly impacts the model gap. Faithful evaluation reduces reward hacking at the cost of productivity gained from agents. The engineering challenge is to provide the least costly body of evidence that supports the required confidence, e.g., calibrated cascades of evaluators (Sections 5.2 and 6.1). (4) Formal methods shift the gaps but do not close them. Formal methods are seductive in promising exhaustive coverage, but can hide the remaining gaps in model or requirements. Where the gaps are small by construction (e.g. formal mathematics), the proof is conclusive (Appendix A). But for most open-world domains, this is not the case because (1) for tractability, developers often formalize only part of the informal model or requirements and (2) the real world continuously changes. A sound proof over a misstated R or a misrepresented M certifies the wrong thing, persuasively. Better languages can reduce what is lost in formalization, but they do not remove the need to validate the result. Checking that the formal R still captures I, and the formal M still captures W, is the certification problem of Section 3 applied to the specification itself, and it remains part of the outer loop. (5) Greater agentic capability shifts the frontier but does not close the gaps. More powerful agents may get better at identifying gaps, gathering evidence, and retaining lessons, but we expect this to shift the gaps’ nature rather than eliminate them. Common, well-documented discrepancies may be resolved autonomously; what remains will tend to be rare conditions, organization-specific knowledge, and complex interactions missing from existing data. In the case of a requirement gap, a storage agent might learn standard deletion but miss whether files should remain deleted after a backup is restored. Resolving this requires stakeholder judgment, not more examples of ordinary operations. In the case of a model gap, a system may track isolated machine failures but miss correlated ones caused by shared dependencies. Thus, as agents narrow the gaps, the marginal cost of discovering the next consequential gap may increase. Research should therefore measure how the frequency, consequences, and discovery cost of unresolved gaps change with agent capability and accumulated experience. Together, these implications define the limits of agentic automation. Agents lack the authority to define stakeholder intent I and cannot replace empirical evidence about the real world W. Consequently, critical real-world systems synthesized from scratch cannot yet be justified to a high level of assurance without outer-loop safeguards (Liu et al., 2026b). Such synthesis remains practical in closed or low-risk settings, where intent, operating conditions, and potential impact are tightly bounded. This paper addresses these two bottlenecks: allocate human judgment where authority is needed to resolve intent, use high-fidelity evaluation to gather evidence about the world, and leverage agents and compute to scale both. Implementations can now be generated just in time; assurance cannot. 8. Related Work Smith and Fetzer distinguish formal correctness from empirical commitments connecting models to reality (Smith, 1985; Fetzer, 1988). Classic requirements engineering separates stakeholder goals, domain knowledge, boundary specifications, and programs (Jackson, 1995; Zave and Jackson, 1997; Gunter et al., 2000), where assumptions and specifications together entail requirements. Stoica et al. separate statement from solution specifications (Stoica et al., 2024). In our framework, requirements R and model assumptions M form the explicit problem presented to the inner loop, evaluator E operationalizes checking under M, and W denotes the real world. This structure enables the two-gap framework to evaluate whether the explicit problem stays faithful to intent and deployment. Testing, contracts, static analysis, model checking, and formal proofs offer distinct evidence linking implementations to requirements. Test oracle research studies erroneous expected results (Barr et al., 2015), while Design by Contract executes selected obligations at runtime (Meyer, 1992). Formal verification proves claims within a model, but empirical studies highlight the unverified environmental assumptions and trusted boundaries that remain outside the proof (Rushby, 2009; Fonseca et al., 2017). Optimization against incomplete objectives drives reward hacking (Amodei et al., 2016; Skalse et al., 2022), while adaptive data analysis shows how evaluator feedback biases subsequent candidate selection (Dwork et al., 2015; Blum and Hardt, 2015). Frameworks like optimize_anything and SkyDiscover demonstrate how diagnostic feedback accelerates evaluator-guided search (Agrawal et al., 2026a; Liu et al., 2026a), while SWE-agent and SWE-bench provide realistic contexts for editing repositories through execution feedback (Yang et al., 2024; Jimenez et al., 2024). Continual learning and knowledge engineering address retention, knowledge transfer, and the difficulty of eliciting explicit knowledge (Parisi et al., 2019; Feigenbaum, 1977; Buchanan and Shortliffe, 1984). Dependability, runtime verification, requirements monitoring, staged deployment, and site reliability engineering address prevention, detection, containment, recovery, and operational learning (Avizienis et al., 2004; Leucker and Schallhart, 2009; Fickas and Feather, 1995; Beyer et al., 2016). Autonomic computing, Simplex runtime assurance, dynamic assurance cases, and perpetual assurances provide established feedback mechanisms for adaptation and constraint enforcement (Kephart and Chess, 2003; Calinescu et al., 2018; Weyns et al., 2018; Rivera et al., 1996). While selective prediction focuses on per-instance deferral to human operators (Bondi et al., 2022), our scope requires continuously updating versioned artifacts, controls, and lessons over time using intent authority and deployment evidence. Human-factors research cautions against automation ironies and over-reliance in escalation design (Parasuraman and Riley, 1997; Bainbridge, 1983), while taxonomies like MAST and AdaMAST organize agent traces into reviewable human vocabularies (Cemri et al., 2025; Cemri and others, 2026). Our contribution lies in synthesizing these domains around two explicit external gaps and two coupled feedback loops. The framework links adaptive E reuse directly to reward hacking, treats review independence as a property of evidence and authority rather than agent identity, and frames assurance as risk-constrained allocation of human attention, evaluation effort, and deployment exposure. 9. Conclusion The main contribution of this paper is the two-gap framework, consisting of the requirement gap that separates the intent of the stakeholder I from the requirements R, and the model gap that separates the real world W from its model M. Although these gaps are not unique to agentic workflows, we view our framework as a powerful lens for understanding the failure modes of agentic software engineering: reward hacking exploits omissions in R or M, while hallucination widens the gaps through fabrication. Although either gap may be closed for a fixed claim in a bounded domain, neither can generally be certified closed in an open, changing world. Through the lens of the two-gap framework, we propose a two-loop architecture to guide allocation of resources in agentic software engineering flows to limit misbehavior. An outer assurance–revision loop uses stakeholder judgment and empirical evidence to narrow the gaps and maintain assurance over time, while an inner implementation-verification loop develops validated implementations. Our research agenda asks how to carry out this process efficiently using human judgment, agent capability, and compute. It identifies two principal bottlenecks: accountable human judgment for the requirement gap, and faithful, often costly evaluation for the model gap. Our framing also tempers the optimism that verifiability alone will drive AI progress on open-world tasks. Recent works argue that capabilities advance fastest where solutions can be checked cheaply and reliably (Wei, 2025), with defining and evaluating tasks becoming the new bottleneck (Yao, 2025; Silver and Sutton, 2025). Indeed, reinforcement learning against verifiable rewards has powered impressive recent gains in mathematics (Lambert et al., 2024; DeepSeek-AI, 2025). But mathematics is the rare domain where the relevant world is the formal system itself: the model gap collapses, and a near-perfect evaluator exists, whether exact match or a proof checker such as Lean (Misra, 2026; de Moura and Ullrich, 2021). In an open, changing world there is no such collapse: verification is delegated to E, and E inherits the two gaps. Reality is the final verifier—not because successful deployment proves correctness, but because deployment can reveal counterexamples that the inner loop wrongly accepted. A mature discipline for agentic software engineering should therefore build on decades of requirements, verification, dependability, runtime monitoring, staged deployment, and knowledge engineering. Agents can process evidence at machine speed, but authorized stakeholders must retain control over what outcomes count as acceptable. The two gaps will persist. Progress will depend on how quickly we detect their consequences, limit the harm, and revise the artifacts that enabled them. 10. Acknowledgments We thank our colleagues at UC Berkeley for their feedback and conversations on this work, including Rishabh Iyer, Mohsen Lesani, and others. This research is supported by NSF (IFML) CCF-2019844 and gifts from Accenture, AMD, Anyscale, Broadcom Inc., Google, IBM, Intel, Intesa Sanpaolo, Lambda, Mibura Inc., Samsung SDS, and SAP. References Agache et al. (2020) A. Agache, M. Brooker, A. Iordache, A. Liguori, R. Neugebauer, P. Piwonka, and D.-M. Popa Firecracker: lightweight virtualization for serverless applications. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI), p. 419–434. External Links: Link Cited by: §5.2, §6.2. Agarwal et al. (2026) S. Agarwal, A. Krentsel, S. Liu, M. Cemri, et al. Inductive deductive synthesis: enabling AI to generate formally verified systems. External Links: 2605.23109, Link Cited by: §5.2, §6.1. Agrawal et al. (2026a) L. A. Agrawal, D. Lee, S. Tan, W. Ma, K. Elmaaroufi, R. Sandadi, et al. optimize_anything: a universal API for optimizing any text parameter. In Proceedings of the ACM Conference on AI and Agentic Systems (CAIS ’26), p. 1–16. External Links: Document, Link Cited by: §8. Agrawal et al. (2026b) L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, et al. GEPA: reflective prompt evolution can outperform reinforcement learning. In Proceedings of the International Conference on Learning Representations, Note: Oral External Links: Link Cited by: §4.1. Amodei et al. (2016) D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané Concrete problems in AI safety. External Links: 1606.06565, Link Cited by: §1, §4.1, §4, §8. Anthropic (2024) Anthropic Building effective agents. Note: December 2024 External Links: Link Cited by: Appendix B. Anthropic (2026) Anthropic Investigating three real-world incidents in our cybersecurity evaluations. Note: July 30, 2026 External Links: Link Cited by: Table 2, §5.1, §5.2. Avizienis et al. (2004) A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing 1 (1), p. 11–33. External Links: Document Cited by: §1, §5.2, §5, §8. Baba et al. (2025) K. Baba, C. Liu, S. Kurita, and A. Sannai Prover agent: an agent-based framework for formal mathematical proofs. External Links: 2506.19923, Link Cited by: Appendix A, §6.1. Bainbridge (1983) L. Bainbridge Ironies of automation. Automatica 19 (6), p. 775–779. External Links: Document Cited by: §6.4, §8. Barr et al. (2015) E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo The oracle problem in software testing: a survey. IEEE Transactions on Software Engineering 41 (5), p. 507–525. External Links: Document Cited by: §5.2, §8. Basiri et al. (2016) A. Basiri, N. Behnam, R. de Rooij, L. Hochstein, L. Kosewski, J. Reynolds, and C. Rosenthal Chaos engineering. IEEE Software 33 (3), p. 35–41. External Links: Document Cited by: §5.2. Becker et al. (2025) J. Becker, N. Rush, E. Barnes, and D. Rein Measuring the impact of early-2025 AI on experienced open-source developer productivity. External Links: 2507.09089, Link Cited by: §1, §6. B. Beyer, C. Jones, J. Petoff, and N. R. Murphy (Eds.) (2016) B. Beyer, C. Jones, J. Petoff, and N. R. Murphy (Eds.) Site reliability engineering: how Google runs production systems. O’Reilly Media. Cited by: §5.2, §5.2, §5.2, §6.1, §6.3, §6, §8. B. Beyer, N. R. Murphy, D. K. Rensin, K. Kawahara, and S. Thorne (Eds.) (2018) B. Beyer, N. R. Murphy, D. K. Rensin, K. Kawahara, and S. Thorne (Eds.) The site reliability workbook: practical ways to implement SRE. O’Reilly Media. Cited by: §5.2. Binkert et al. (2011) N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, et al. The gem5 simulator. ACM SIGARCH Computer Architecture News 39 (2), p. 1–7. External Links: Document Cited by: §6.1. Blum and Hardt (2015) A. Blum and M. Hardt The ladder: a reliable leaderboard for machine learning competitions. In Proceedings of ICML, p. 1006–1014. External Links: Link Cited by: §4.1, §5.2, §8. Bondi et al. (2022) E. Bondi, R. Koster, H. Sheahan, M. Chadwick, Y. Bachrach, T. Cemgil, U. Paquet, and K. Dvijotham Role of human-AI interaction in selective prediction. Proceedings of the AAAI Conference on Artificial Intelligence 36 (5), p. 5286–5294. External Links: Document Cited by: §6.2, §8. Borkar (2005) S. Borkar Designing reliable systems from unreliable components: the challenges of transistor variability and degradation. IEEE Micro 25 (6), p. 10–16. External Links: Document Cited by: Appendix A. Brooks (1987) F. P. Brooks No silver bullet: essence and accidents of software engineering. IEEE Computer 20 (4), p. 10–19. External Links: Document Cited by: §1, §3.1, §4. Brown and Patterson (2003) A. B. Brown and D. A. Patterson Undo for operators: building an undoable e-mail store. In Proceedings of the USENIX Annual Technical Conference, p. 1–14. External Links: Link Cited by: §6.2. Bryant (1986) R. E. Bryant Graph-based algorithms for boolean function manipulation. IEEE Transactions on Computers C-35 (8), p. 677–691. External Links: Document Cited by: Appendix A. B. G. Buchanan and E. H. Shortliffe (Eds.) (1984) B. G. Buchanan and E. H. Shortliffe (Eds.) Rule-based expert systems: the MYCIN experiments of the Stanford heuristic programming project. Addison-Wesley. External Links: Link Cited by: §3.1, §4, §6.3, §8. Calinescu et al. (2018) R. Calinescu, D. Weyns, S. Gerasimou, M. U. Iftikhar, I. Habli, and T. Kelly Engineering trustworthy self-adaptive software with dynamic assurance cases. IEEE Transactions on Software Engineering 44 (11), p. 1039–1069. External Links: Document Cited by: §5, §5, §8. Campbell (1979) D. T. Campbell Assessing the impact of planned social change. Evaluation and Program Planning 2 (1), p. 67–90. External Links: Document Cited by: §4. Cemri et al. (2026) M. Cemri et al. AdaMAST: adaptive failure taxonomies as feedback for LLM-agent improvement procedures. External Links: Link Cited by: §8. Cemri et al. (2025) M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, et al. Why do multi-agent LLM systems fail?. In Advances in Neural Information Processing Systems 38 (Datasets and Benchmarks), External Links: Link Cited by: §6.4, §8. Cheng et al. (2025) A. Cheng, S. Liu, M. Pan, Z. Li, et al. Let the barbarians in: how AI can accelerate systems performance research. External Links: 2512.14806, Link Cited by: Table 2. Chowdhury et al. (2024) N. Chowdhury, J. Aung, C. J. Shern, O. Jaffe, D. Sherburn, G. Starace, et al. Introducing SWE-bench Verified. Note: OpenAIAugust 13, 2024; updated February 24, 2025 External Links: Link Cited by: §1, Table 2, Table 2. Clarke and Emerson (1982) E. M. Clarke and E. A. Emerson Design and synthesis of synchronization skeletons using branching time temporal logic. In Logic of Programs, LNCS, Vol. 131, p. 52–71. External Links: Document Cited by: Appendix A. Cloud Security Alliance AI Safety Initiative (2026) Cloud Security Alliance AI Safety Initiative Research note: OpenAI model sandbox escape and Hugging Face breach. Note: July 22, 2026 External Links: Link Cited by: Table 2, §5.1, §5.2. Cui et al. (2026) K. Z. Cui, M. Demirer, S. Jaffe, L. Musolff, S. Peng, and T. Salz The effects of generative AI on high-skilled work: evidence from three field experiments with software developers. Management Science. External Links: Document Cited by: §1, §6. de Moura and Ullrich (2021) L. de Moura and S. Ullrich The Lean 4 theorem prover and programming language. In Proceedings of the 28th International Conference on Automated Deduction (CADE), p. 625–635. Cited by: §9. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. External Links: 2501.12948, Link Cited by: §9. Dwork et al. (2015) C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth Preserving statistical validity in adaptive data analysis. In Proceedings of STOC, p. 117–126. External Links: Link Cited by: §4.1, §5.2, §8. Feigenbaum (1977) E. A. Feigenbaum The art of artificial intelligence: themes and case studies of knowledge engineering. In Proceedings of IJCAI-77, p. 1014–1029. External Links: Link Cited by: §3.1, §4, §6.3, §8. Feitelson et al. (2013) D. G. Feitelson, E. Frachtenberg, and K. L. Beck Development and deployment at Facebook. IEEE Internet Computing 17 (4), p. 8–17. External Links: Document Cited by: §5.2. Fenwick (2026) C. Fenwick Post on X. Note: X (@codytfenwick)July 22, 2026 External Links: Link Cited by: §1. Fetzer (1988) J. H. Fetzer Program verification: the very idea. Communications of the ACM 31 (9), p. 1048–1063. External Links: Document Cited by: §3.2, §8. Fickas and Feather (1995) S. Fickas and M. S. Feather Requirements monitoring in dynamic environments. In Proceedings of the 2nd IEEE International Symposium on Requirements Engineering, p. 140–147. External Links: Document Cited by: §5.2, §8. Fonseca et al. (2017) P. Fonseca, K. Zhang, X. Wang, and A. Krishnamurthy An empirical study on the correctness of formally verified distributed systems. In Proceedings of EuroSys, p. 328–343. External Links: Document Cited by: §3.2, §8. Forrester et al. (2007) A. I. J. Forrester, A. Sóbester, and A. J. Keane Multi-fidelity optimization via surrogate modelling. Proceedings of the Royal Society A 463, p. 3251–3269. External Links: Document Cited by: §6.1. GitHub (2026a) GitHub About GitHub Copilot cloud agent. Note: GitHub DocsAccessed July 11, 2026 External Links: Link Cited by: §1, §1. GitHub (2026b) GitHub About GitHub Copilot code review. Note: GitHub DocsAccessed July 11, 2026 External Links: Link Cited by: §1, §1. Goodhart (1984) C. A. E. Goodhart Problems of monetary management: the UK experience. In Monetary Theory and Practice: The UK Experience, p. 91–121. External Links: Document Cited by: §4. Google DeepMind (2025) Google DeepMind Genie 3: a new frontier for world models. Note: August 2025 External Links: Link Cited by: Appendix A. Gunter et al. (2000) C. A. Gunter, E. L. Gunter, M. Jackson, and P. Zave A reference model for requirements and specifications. IEEE Software 17 (3), p. 37–43. External Links: Document Cited by: §8. Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber World models. External Links: 1803.10122, Link Cited by: Appendix A. Hadfield-Menell and Hadfield (2019) D. Hadfield-Menell and G. K. Hadfield Incomplete contracting and AI alignment. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), p. 417–422. External Links: Document Cited by: §1, §3.1, §4. Hafner et al. (2025) D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap Mastering diverse domains through world models. Nature 640, p. 647–653. Cited by: Appendix A. Hart and Moore (1988) O. Hart and J. Moore Incomplete contracts and renegotiation. Econometrica 56 (4), p. 755–785. External Links: Document Cited by: §1, §3.1, §4. Hawblitzel et al. (2015) C. Hawblitzel, J. Howell, M. Kapritsos, J. R. Lorch, B. Parno, M. L. Roberts, S. T. V. Setty, and B. Zill IronFleet: proving practical distributed systems correct. In Proceedings of the 25th ACM Symposium on Operating Systems Principles, p. 1–17. External Links: Document Cited by: §2.4, §6.1. Hendrycks and Gimpel (2017) D. Hendrycks and K. Gimpel A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of ICLR, External Links: Link Cited by: §6.2. Hennessy and Patterson (2019) J. L. Hennessy and D. A. Patterson Computer architecture: a quantitative approach. 6th edition, Morgan Kaufmann. Cited by: Appendix A. Hoare (1969) C. A. R. Hoare An axiomatic basis for computer programming. Communications of the ACM 12 (10), p. 576–580, 583. External Links: Document Cited by: Appendix A, Appendix A, §2.1, §3.2, §5.2. Hubert et al. (2026) T. Hubert, R. Mehta, L. Sartran, et al. Olympiad-level formal mathematical reasoning with reinforcement learning. Nature 651, p. 607–613. External Links: Document Cited by: Appendix A, §6.1. Hugging Face (2026a) Hugging Face Anatomy of a frontier lab agent intrusion: a technical timeline of the july 2026 incident. Note: July 27, 2026 External Links: Link Cited by: §1, Table 2, §5.1. Hugging Face (2026b) Hugging Face Harness, scaffold, and the AI agent terms worth getting right. Note: Hugging Face Blog External Links: Link Cited by: Appendix B. Hugging Face (2026c) Hugging Face Security incident disclosure — july 2026. Note: July 16, 2026 External Links: Link Cited by: §1, Table 2, §5.1. Humble and Farley (2010) J. Humble and D. Farley Continuous delivery: reliable software releases through build, test, and deployment automation. Addison-Wesley. Cited by: §5.2, §5.2. IEEE (2024) IEEE IEEE standard for SystemVerilog—unified hardware design, specification, and verification language. Note: IEEE Std 1800-2023 External Links: Link Cited by: Appendix A. ISO/IEC/IEEE (2025) ISO/IEC/IEEE Systems and Software Engineering—Systems and Software Assurance—Part 1: Vocabulary and Concepts. Technical report Technical Report ISO/IEC/IEEE 15026-1:2025, International Organization for Standardization, Geneva, Switzerland. External Links: Link Cited by: §1, Table 1, §5. Jackson (1995) M. Jackson The world and the machine. In Proceedings of the 17th International Conference on Software Engineering, p. 283–292. External Links: Document Cited by: §2.3, §3.2, §4, §8. Jensen and Meckling (1976) M. C. Jensen and W. H. Meckling Theory of the firm: managerial behavior, agency costs and ownership structure. Journal of Financial Economics 3 (4), p. 305–360. External Links: Document Cited by: §4. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan SWE-bench: can language models resolve real-world GitHub issues?. In Proceedings of ICLR, External Links: Link Cited by: §8. Kaplan and Garrick (1981) S. Kaplan and B. J. Garrick On the quantitative definition of risk. Risk Analysis 1 (1), p. 11–27. External Links: Document Cited by: §6. Karandikar et al. (2018) S. Karandikar, H. Mao, D. Kim, D. Biancolin, A. Amid, et al. FireSim: FPGA-accelerated cycle-exact scale-out system simulation in the public cloud. In Proceedings of the 45th International Symposium on Computer Architecture (ISCA), p. 29–42. External Links: Document Cited by: §6.1. Kephart and Chess (2003) J. O. Kephart and D. M. Chess The vision of autonomic computing. Computer 36 (1), p. 41–50. External Links: Document Cited by: §8. Klein et al. (2009) G. Klein, K. Elphinstone, G. Heiser, J. Andronick, et al. seL4: formal verification of an OS kernel. In Proceedings of the 22nd ACM Symposium on Operating Systems Principles, p. 207–220. External Links: Document Cited by: §2.4, §6.1. Kocher et al. (1999) P. Kocher, J. Jaffe, and B. Jun Differential power analysis. In Advances in Cryptology—CRYPTO ’99, LNCS, Vol. 1666, p. 388–397. External Links: Document Cited by: Appendix A. Krakovna et al. (2020) V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg Specification gaming: the flip side of AI ingenuity. Note: Google DeepMind BlogApril 21, 2020 External Links: Link Cited by: §4. Kundu et al. (2010) S. Kundu, S. Lerner, and R. Gupta Translation validation of high-level synthesis. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 29 (4), p. 566–579. Cited by: Appendix A. Lambert et al. (2024) N. Lambert, J. Morrison, V. Pyatkin, et al. Tülu 3: pushing frontiers in open language model post-training. External Links: 2411.15124, Link Cited by: §9. LeCun (2022) Y. LeCun A path towards autonomous machine intelligence. Note: OpenReview Cited by: Appendix A. Lehman (1996) M. M. Lehman Laws of software evolution revisited. In Software Process Technology, LNCS, Vol. 1149, p. 108–124. Cited by: §2.3, §3.2, §4, §6.1. Lenat (1995) D. B. Lenat CYC: a large-scale investment in knowledge infrastructure. Communications of the ACM 38 (11), p. 33–38. External Links: Document Cited by: §6.3. Leroy (2009) X. Leroy Formal verification of a realistic compiler. Communications of the ACM 52 (7), p. 107–115. External Links: Document Cited by: Appendix A. Leucker and Schallhart (2009) M. Leucker and C. Schallhart A brief account of runtime verification. Journal of Logic and Algebraic Programming 78 (5), p. 293–303. External Links: Document Cited by: §5.2, §8. Li et al. (2018) L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar Hyperband: a novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research 18 (185), p. 1–52. External Links: Link Cited by: §6.1. Li et al. (2024) Z. Li, Y. Wu, Z. Li, et al. Autoformalize mathematical statements by symbolic equivalence and semantic consistency. External Links: 2410.20936, Link Cited by: Appendix A, §6.1. Liu et al. (2026a) S. Liu, M. Cemri, S. Agarwal, A. Krentsel, A. Naren, Q. Mang, et al. SkyDiscover: a flexible, adaptive framework for AI-driven scientific and algorithmic discovery. In Proceedings of the ACM Conference on AI and Agentic Systems (CAIS ’26), p. 1223–1227. External Links: Document Cited by: §4.1, §8. Liu et al. (2026b) S. Liu, A. Krentsel, S. Agarwal, M. Cemri, et al. The time is here for Just-in-Time systems: challenges and opportunities. External Links: 2605.24096, Link Cited by: §1, §1, Table 2, §7. Majors et al. (2022) C. Majors, L. Fong-Jones, and G. Miranda Observability engineering: achieving production excellence. O’Reilly Media. Cited by: §5.2. Manès et al. (2021) V. J. M. Manès, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo The art, science, and engineering of fuzzing: a survey. IEEE Transactions on Software Engineering 47 (11), p. 2312–2331. External Links: Document Cited by: §6.2. McCarthy (1977) J. McCarthy Epistemological problems of artificial intelligence. In Proceedings of the 5th International Joint Conference on Artificial Intelligence (IJCAI-77), p. 1038–1044. Cited by: §1, §3.1, §4. McCubbins and Schwartz (1984) M. D. McCubbins and T. Schwartz Congressional oversight overlooked: police patrols versus fire alarms. American Journal of Political Science 28 (1), p. 165–179. External Links: Document Cited by: §4. McKeeman (1998) W. M. McKeeman Differential testing for software. Digital Technical Journal 10 (1), p. 100–107. External Links: Link Cited by: §5.2. Meyer (1992) B. Meyer Applying “Design by Contract”. IEEE Computer 25 (10), p. 40–51. External Links: Document Cited by: §2.1, §5.2, §8. Misra (2026) V. Misra After the proof. Note: August 2026 External Links: Link Cited by: §9. Mozannar and Sontag (2020) H. Mozannar and D. Sontag Consistent estimators for learning to defer to an expert. In Proceedings of ICML, p. 7076–7087. External Links: Link Cited by: §6.2. Mu et al. (2024) F. Mu, L. Shi, S. Wang, Z. Yu, B. Zhang, C. Wang, S. Liu, and Q. Wang ClarifyGPT: a framework for enhancing LLM-based code generation via requirements clarification. Proceedings of the ACM on Software Engineering 1 (FSE), p. 2332–2354. External Links: Document Cited by: §6.2. Newcombe et al. (2015) C. Newcombe, T. Rath, F. Zhang, B. Munteanu, M. Brooker, and M. Deardeuff How Amazon Web Services uses formal methods. Communications of the ACM 58 (4), p. 66–73. External Links: Document Cited by: §2.4, §6.1. Nuseibeh and Easterbrook (2000) B. Nuseibeh and S. Easterbrook Requirements engineering: a roadmap. In Proceedings of the Conference on the Future of Software Engineering, p. 35–46. Cited by: §3.1, §3.1, §6.2. NVIDIA (2025) NVIDIA Cosmos world foundation model platform for physical AI. External Links: 2501.03575, Link Cited by: Appendix A. OpenAI (2026) OpenAI OpenAI and Hugging Face partner to address security incident during model evaluation. Note: July 21, 2026 External Links: Link Cited by: §1, Table 2, §5.1. OWASP Foundation (2025) OWASP Foundation OWASP top 10 for large language model applications, 2025 edition. External Links: Link Cited by: §5.1. Pan et al. (2024) A. Pan, E. Jones, M. Jagadeesan, and J. Steinhardt Feedback loops with language models drive in-context reward hacking. In Proceedings of ICML, p. 39154–39200. External Links: Link Cited by: §1, §4.1. Parasuraman and Riley (1997) R. Parasuraman and V. Riley Humans and automation: use, misuse, disuse, abuse. Human Factors 39 (2), p. 230–253. External Links: Document Cited by: §6.2, §6.4, §8. Parisi et al. (2019) G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter Continual lifelong learning with neural networks: a review. Neural Networks 113, p. 54–71. External Links: Document Cited by: §6.3, §8. Parnas (1994) D. L. Parnas Software aging. In Proceedings of the 16th International Conference on Software Engineering, p. 279–287. Cited by: §2.3, §3.2, §4, §6.1. Pnueli et al. (1998) A. Pnueli, M. Siegel, and E. Singerman Translation validation. In Tools and Algorithms for the Construction and Analysis of Systems, LNCS, Vol. 1384, p. 151–166. External Links: Document Cited by: Appendix A. Polanyi (1966) M. Polanyi The tacit dimension. Doubleday. Cited by: §3.1. Pollack (1998) R. Pollack How to believe a machine-checked proof. In Twenty Five Years of Constructive Type Theory, p. 205–220. External Links: Document Cited by: Appendix A. Potts et al. (1994) C. Potts, K. Takahashi, and A. I. Antón Inquiry-based requirements analysis. IEEE Software 11 (2), p. 21–32. External Links: Document Cited by: §3.1. Rabaey et al. (2003) J. M. Rabaey, A. Chandrakasan, and B. Nikolić Digital integrated circuits: a design perspective. 2nd edition, Prentice Hall. Cited by: Appendix A. Ren et al. (2025) Z. Ren, Z. Shao, J. Song, et al. DeepSeek-Prover-V2: advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. External Links: 2504.21801, Link Cited by: Appendix A, §6.1. Rivera et al. (1996) J. G. Rivera, A. A. Danylyszyn, C. Weinstock, L. R. Sha, and M. J. Gagliardi An architectural description of the simplex architecture. Technical report Technical Report CMU/SEI-96-TR-006, Software Engineering Institute, Carnegie Mellon University. External Links: Link Cited by: §8. Rose et al. (2020) S. Rose, O. Borchert, S. Mitchell, and S. Connelly Zero trust architecture. Technical report Technical Report Special Publication 800-207, NIST. External Links: Document Cited by: §5.2. Rushby (2009) J. Rushby Software verification and system assurance. In Proceedings of the 7th IEEE International Conference on Software Engineering and Formal Methods, p. 3–10. External Links: Document Cited by: Appendix A, Appendix A, §1, §2.1, §3.2, §5.2, §5, §8. Saltzer and Schroeder (1975) J. H. Saltzer and M. D. Schroeder The protection of information in computer systems. Proceedings of the IEEE 63 (9), p. 1278–1308. External Links: Document Cited by: §5.2, §5.2. Segura et al. (2016) S. Segura, G. Fraser, A. B. Sánchez, and A. Ruiz-Cortés A survey on metamorphic testing. IEEE Transactions on Software Engineering 42 (9), p. 805–824. External Links: Document Cited by: §5.2, §6.2. Settles (2009) B. Settles Active learning literature survey. Technical report Technical Report Computer Sciences Technical Report 1648, University of Wisconsin–Madison. External Links: Link Cited by: §6.2. Silver and Sutton (2025) D. Silver and R. S. Sutton Welcome to the era of experience. Note: Google DeepMind External Links: Link Cited by: §9. Skalse et al. (2022) J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems 35, External Links: Link Cited by: §1, §4.1, §4, §8. Smith (1985) B. C. Smith The limits of correctness. ACM SIGCAS Computers and Society 14–15 (4, 1–3), p. 18–26. External Links: Document Cited by: §1, §2.3, §3.2, §3.2, §8. Spracklen et al. (2025) J. Spracklen, R. Wijewickrama, A. H. M. N. Sakib, A. Maiti, B. Viswanath, and M. Jadliwala We have a package for you! a comprehensive analysis of package hallucinations by code generating LLMs. In Proceedings of the 34th USENIX Security Symposium, p. 3687–3706. External Links: Link Cited by: §1. Stoica et al. (2024) I. Stoica, M. Zaharia, J. Gonzalez, K. Goldberg, K. Sen, et al. Specifications: the missing link to making the development of LLM systems an engineering discipline. External Links: 2412.05299, Document Cited by: §6.1, §6.2, §8. Thompson (1984) K. Thompson Reflections on trusting trust. Communications of the ACM 27 (8), p. 761–763. External Links: Document Cited by: §5.1. TLA+ Foundation (2026) TLA+ Foundation Industrial use of TLA+. Note: Accessed July 21, 2026 External Links: Link Cited by: §2.4. Wang et al. (2025) Y. Wang, M. Pradel, and Z. Liu Are “solved issues” in SWE-bench really solved correctly? an empirical study. External Links: 2503.15223, Document, Link Cited by: §1, Table 2, Table 2. Wei (2025) J. Wei Asymmetry of verification and verifier’s law. Note: July 2025 External Links: Link Cited by: §9. Weyns et al. (2018) D. Weyns, N. Bencomo, R. Calinescu, J. Cámara, C. Ghezzi, V. Grassi, et al. Perpetual assurances for self-adaptive systems. In Software Engineering for Self-Adaptive Systems I. Assurances, p. 31–63. External Links: Document Cited by: §5, §8. Williams et al. (2009) S. Williams, A. Waterman, and D. Patterson Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM 52 (4), p. 65–76. External Links: Document Cited by: §6.1. Xiong et al. (2026) Z. Xiong, Y. Lin, W. Xie, P. He, Z. Liu, J. Tang, H. Lakkaraju, and Z. Xiang How memory management impacts LLM agents: an empirical study of experience-following behavior. In Proceedings of ACL, p. 623–645. External Links: Document Cited by: §6.3. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37, External Links: Document Cited by: Appendix B, §1, §1, §8. Yao (2025) S. Yao The second half. Note: April 2025 External Links: Link Cited by: §9. Zave and Jackson (1997) P. Zave and M. Jackson Four dark corners of requirements engineering. ACM Transactions on Software Engineering and Methodology 6 (1), p. 1–30. External Links: Document Cited by: §8. Zhao et al. (2024) A. Zhao et al. ExpeL: LLM agents are experiential learners. Proceedings of the AAAI Conference on Artificial Intelligence 38 (17), p. 19632–19642. External Links: Document Cited by: §6.3. Zhou et al. (2026) Y. Zhou, A. Li, A. Silla, Z. Liu, and V. Sekar AIChilles: automatically uncovering hidden weaknesses in AI-evolved systems. External Links: 2606.15834, Link Cited by: §6.2. Appendix A When the Two Gaps Can Be Narrowed Not all domains expose equally difficult gaps. The gaps become more tractable when intent is fixed by an authoritative artifact, the relevant world is bounded by a stable formal interface, an existing implementation provides a strong behavioral reference, or a faithful model of the world can be learned from available data. It is important, however, to distinguish narrowing the requirement or model gap from strengthening E relative to fixed R and M (See Section 2.1). Formal methods and formal domains. Formal methods primarily strengthen evaluation: for fixed R and M, a sound proof can establish that P satisfies R for every execution admitted by M (Hoare, 1969; Rushby, 2009). This can close the evaluation gap within the proof’s formal boundary. Formalization can also help people narrow the requirement and model gaps by making requirements and assumptions explicit and by producing counterexamples that expose omissions. The proof itself, however, cannot establish that R captures I or that M captures W. Formal mathematics lies at one end of this spectrum. For the narrow task of checking a proof within a specified formal system, formal semantics defines the relevant world, so the model gap can effectively collapse (Hubert et al., 2026; Ren et al., 2025; Baba et al., 2025; Pollack, 1998). The requirement gap remains when an informal conjecture is translated into the wrong formal statement, which is why autoformalization research checks whether the formal statement preserves the meaning of the informal one, for example by testing whether independently generated formalizations of the same conjecture are logically equivalent (Li et al., 2024; Pollack, 1998). Hardware and chip synthesis. Digital hardware often admits a more bounded model than deployed software. An instruction set can define a processor’s observable functional contract (Hennessy and Patterson, 2019), while register transfer level (RTL) designs describe clocked state transitions. SystemVerilog standardizes behavioral, RTL, and gate level abstractions and verification assertions (IEEE, 2024). By defining inputs, states, and interfaces precisely, these abstractions can substantially narrow the model gap. Formal property checking has a long history in hardware verification (Clarke and Emerson, 1982; Bryant, 1986), while equivalence checking can compare a source specification with RTL generated through high level synthesis (Kundu et al., 2010). These methods can close or narrow the evaluation gap within the digital abstraction. They narrow the model gap only insofar as this abstraction represents the manufactured and deployed chip. However, they cannot close the requirement gap when the instruction set or asserted properties omit intended behavior (Hoare, 1969; Rushby, 2009). Nor can they cover analog effects, manufacturing variation, power, temperature, or side channels without additional models and properties (Rabaey et al., 2003; Borkar, 2005; Kocher et al., 1999). Code optimization rather than system synthesis. Optimization starts from an existing implementation P0P_0, which provides a behavioral reference embodying prior decisions about interfaces and edge cases. Treating preservation of this behavior as an explicit requirement narrows the requirement gap because the agent need not reconstruct it from a separate (possibly incomplete) specification. This is a fundamental difference between optimization and synthesis: optimization inherits a behavioral anchor, whereas synthesis must infer one from R and M. Source code alone does not narrow the model gap, but configurations can encode platform assumptions and deployment evidence can reveal conditions in W that M should represent. Verified compilation and translation validation can establish or check semantic preservation, while existing tests provide additional evidence (Leroy, 2009; Pnueli et al., 1998). These methods narrow the evaluation gap for the transformation, but they cannot close it: behavioral parity is established only for the executions actually exercised; these executions cannot be enumerated exhaustively. If no test ever stores a 4MB value, behavior for such values remains unchecked. Moreover, P0P_0 may itself contain defects, obsolete behavior, or earlier requirement and model gaps. Synthesis from scratch is nevertheless more challenging because it lacks this executable anchor and must infer more from incomplete R and M. World models. A complementary effort targets the model gap directly: learn M from data. World models are generative models of an environment’s dynamics, trained on video and interaction with the real world (Ha and Schmidhuber, 2018; LeCun, 2022; Hafner et al., 2025). Agents can then be evaluated, and even trained, using such models before acting in the real world. This line of work spans early latent dynamics models (Ha and Schmidhuber, 2018), the vision of world models as the path to grounded machine intelligence (LeCun, 2022), and recent foundation-scale systems: real-time interactive environments for agent training, and open platforms that generate physics-aware synthetic data for robotics and autonomous driving (Google DeepMind, 2025; NVIDIA, 2025). These models serve AI applications that act in the physical world, or that must simulate it faithfully, such as games. For such applications, world models narrow the model gap. However, a learned world model is still a model M: its fidelity to W still needs to be validated. World models narrow the model gap, but they move the burden of empirical validation rather than removing it: now the learned model itself must be checked against reality. Appendix B Operationalizing the Inner Loop: The Evaluation Harness Practitioners increasingly summarize agents as an AI model plus a harness, i.e., the prompts, tools, sandboxes, memory, and checkers that surround the AI model (Yang et al., 2024; Anthropic, 2024; Hugging Face, 2026b). In this framework, the harness operationalizes the abstractions of the inner loop. The task descriptions it supplies are the operative R; the tools, dependencies, permissions, and environment it exposes determine the realized M; and the tests it runs and results it parses implement E. The quality of the harness plays a key role in determining all three gaps. An underspecified task widens the requirement gap, an environment or dependency mismatch widens the model gap, and an unreliable checker creates an evaluation gap. For example, the SWE-bench audits traced false acceptances to unreliable harnesses (Table 2), and the July 2026 incidents were caused by harness and operational failures (Section 5.1). The harness also implements the controls of the outer assurance-revision loop, including sandboxing, least privilege, monitoring, and staged deployment (Section 5.2). Thus, improving the harness can narrow the gaps and improve assurance without changing the underlying AI model. This is an important source of progress in agentic software engineering: better agents are not only those using more capable AI models, but also the ones operating with better requirements, environments, and evaluators. These examples define a continuum rather than exceptions to the framework. The requirement gap narrows as authoritative intent becomes more explicit. The model gap shrinks as the relevant world becomes more bounded and stable, or as faithful models of it are learned from data. The evaluation gap shrinks as stronger methods establish conformance within those boundaries. None of these improvements should be mistaken for closure of a different gap.

Instant research discovery