Paper deep dive
Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites
Xin Xu, Siru Tao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Invariant suites and runtime monitors increasingly gate agent deployment decisions, and the evidence offered for any particular suite is almost always a single observation: it passes an implementation believed correct and fails one believed broken. We measure what that observation is worth. Applying mutation analysis to an invariant suite for a multi-session agent state-projection layer, we first find that this standard validation certifies a suite in which a first-order mutant removing event-identity deduplication survives every check. We then freeze the repaired twelve-check suite, record its hash, and run it once against ten mutants specified by an adversarial reader who designed none of its fixtures: it kills five. Instrumenting the five survivors against the reference shows they fail in two distinct ways, not one. Three are never activated, because no fixture supplies an input on which the mutated code behaves differently at all. The other two corrupt internal state that no oracle in the suite can observe. The two modes need different repairs, and neither is visible from a pass/fail report. Treating the missing inputs as a coverage question, we enumerate seven discriminating dimensions of the input space, register in advance which are uncovered and which survivors they should explain, and add one fixture per uncovered dimension while reusing the existing oracles verbatim. All five survivors then die, each to the check written for its predicted dimension. We report this as a repair result on the same challenge set rather than a second held-out estimate, and give the artifact, including the frozen hash, the registered predictions, all mutants and the run logs, so the distinction is checkable.
Tags
Links
- Source: https://arxiv.org/abs/2610.02928v1
- Canonical: https://arxiv.org/abs/2610.02928v1
Trouble viewing inline? Open PDF directly â
Full Text
38,860 characters extracted from source content.
Expand or collapse full text
Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites Xin Xu Carnegie Mellon University Eragon xuxin@cmu.edu Siru Tao Carnegie Mellon University sirutao@andrew.cmu.edu Abstract Invariant suites and runtime monitors increasingly gate agent deployment deci- sions, and the evidence offered for any particular suite is almost always a single observation: it passes an implementation believed correct and fails one believed broken. We measure what that observation is worth. Applying mutation analysis to an invariant suite for a multi-session agent state-projection layer, we first find that this standard validation certifies a suite in which a first-order mutant removing event-identity deduplication survives every check. We then freeze the repaired twelve-check suite, record its hash, and run it once against ten mutants specified by an adversarial reader who designed none of its fixtures: it kills five. Instrumenting the five survivors against the reference shows they fail in two distinct ways, not one. Three are never activated, because no fixture supplies an input on which the mutated code behaves differently at all. The other two corrupt internal state that no oracle in the suite can observe. The two modes need different repairs, and neither is visible from a pass/fail report. Treating the missing inputs as a coverage question, we enumerate seven discriminating dimensions of the input space, register in ad- vance which are uncovered and which survivors they should explain, and add one fixture per uncovered dimension while reusing the existing oracles verbatim. All five survivors then die, each to the check written for its predicted dimension. We report this as a repair result on the same challenge set rather than a second held-out estimate, and give the artifact, including the frozen hash, the registered predictions, all mutants and the run logs, so the distinction is checkable. 1 Introduction Agentic systems that operate over multiple turns and multiple concurrent sessions are typically wrapped in a layer of automated checking: invariant assertions, runtime monitors, safety validators, or judge models that decide whether an observed trajectory is acceptable. These checks are then used, directly or indirectly, to gate deployment, trigger human escalation, or certify that a system behaves as intended. The evidentiary weight placed on this layer is substantial, and yet the layer itself is rarely subjected to the same scrutiny as the system it monitors. A verification suite that never triggers a failure is observationally identical, from outside the suite, to a system that never has one. Reuel et al. [2025] name this gap as an open problem in technical AI governance: it is hard to be confident that an evaluationâs results reflect the property being measured rather than the evaluation methodology itself. They pose it for models. We ask it about the infrastructure agents run on. The technique for answering it at the software layer is mutation analysis [DeMillo et al., 1978, Jia and Harman, 2011], and the conditions under which a fault escapes are standard [Ammann and Offutt, 2016]: it must be reached, must infect state, and that infection must propagate to an observed output. We take the technique as given. The question we bring to it is not the one mutation analysis usually answers. Mutation score is normally a developer-facing measure of test adequacy, consumed to decide Preprint. arXiv:2610.02928v1 [cs.SE] 2 Oct 2026 where to write more tests. Here the suiteâs output is offered as evidence, to license a deployment or an intervention, so the operative question is what a verifierâs silence entitles anyone to conclude. That question has an answer we can measure, and the answer turns out to be governed by something other than the properties the suite asserts. We study the client-side projection layer of a multi-session agent chat interface, which rebuilds visible conversation state from asynchronous events (WebSocket pushes, HTTP acknowledgments, periodic snapshots) arriving unreliably, per session, per user. Six failure classes follow from asynchronous multi-channel replication: duplicate delivery, non-idempotent replay, cross-session leakage, out-of- order rendering, non-convergent concurrent updates, and stale-state regression. We write a reference implementation handling all six, an eleven-check invariant suite, and first-order mutants each altering exactly one mechanism. The paper is one argument in three steps. The validation everyone uses is uninformative. Passing the reference and failing an all-defects variant certifies our suite while a first-order mutant survives all eleven checks, because the reference enforces the property twice and no fixture separates the two enforcements. Measured against externally specified mutants, the certified suite resolves half. We froze the repaired twelve-check suite, recorded its hash, and ran it once against ten mutants specified by an adversarial reader of this manuscript who took no part in designing any fixture: five survive. The survivors fail in two distinct ways, and the distinction dictates the repair. Instrumenting each survivor against the reference at every step of every fixture shows that three are never activated. The mutated code runs but produces identical internal state, because no fixture supplies an input on which the change matters. Two others do corrupt internal state that no oracle in the suite can observe. Both are coverage problems, but of different kinds: the first needs new input values, the second needs input values that carry an existing corruption to an observable output. We enumerate seven dimensions along which the referenceâs control flow can discriminate two executions, register which are uncovered before writing anything, and add one fixture per uncovered dimension while reusing the existing oracle predicates unchanged. All five survivors then die, each to the check written for its predicted dimension. Contributions. 1. A measurement where there was previously an assurance. We report a post-freeze mutation kill rate for a verification suite guarding agent infrastructure: 5 of 10 externally specified first-order mutants, with the frozen-suite hash, the mutants, and the survivors published. The usual evidence for such a suite is an unbroken record of not firing, which is also what a suite of no resolution produces. (The 10/10 reported after repair is a result on the same challenge set, not a second held-out estimate, and we keep the two separate throughout.) 2. A demonstration that the standard validation cannot detect this. The correct/all-defects pair certifies a suite that a first-order mutant survives outright. The mechanism is redundancy in the system under test: two guards independently suppress a redelivered event, so no fixture exercising redelivery can attribute the outcome to either. This is the masking problem that condition-coverage criteria such as MC/DC exist to address, arising here between two implementation mechanisms rather than two clauses of one predicate. 3.A two-mode account of why a suite misses, measured rather than asserted. We instru- ment every mutant against the reference at each step of each fixture and record where the two first diverge. Three survivors never diverge internally at all. Two diverge internally but never visibly. A pass/fail report cannot distinguish these, and they call for different repairs. We are not aware of this breakdown being reported for a verification suite used as deployment evidence. 4.A registered prediction that the repair confirms. Treating the gap as an input-space coverage problem, we enumerate seven discriminating dimensions, record in advance which are uncovered and which survivors they should account for, then add one fixture per uncovered dimension with the oracles reused verbatim. All five predictions hold. 5.A negative result on invariant-family taxonomies. Mutants co-designed with the suiteâs five families localise cleanly to their own family. Externally specified mutants do not: a deduplication fault is caught by a convergence check, and two unrelated snapshot faults by the same check. Families organise checks well and describe coverage badly. 2 2 Related Work Contract- and invariant-based verification of agent infrastructure. The closest work to ours is Paduraru et al. [2026], who propose a trace-based assurance framework for agentic orchestra- tion combining contracts, testing, and governance. Their framework validates itself through fault injection and budgeted counterexample search over an orchestration graph, targeting failures such as non-termination, role drift, and claim propagation across agents. Our target failure surface is different, namely client-side session-state consistency rather than orchestration semantics, and more importantly our validity argument is different in kind: rather than searching for counterexamples against a specification, we construct a fixed, interface-compatible mutant known in advance to violate the specification, and ask whether the checking apparatus notices. The two approaches are com- plementary: fault injection is well suited to open-ended search for unknown failure modes, while a differential mutation argument is well suited to auditing whether known failure modes are reliably caught. Empirical failure taxonomies for deployed agents. Wu [2026] give a longitudinal taxonomy of silent failures in a production LLM agent runtime, drawn from postmortems. That work characterises what goes wrong. Ours asks whether an automated checker would have caught it. Active testing of agent frameworks. Long et al. [2026] test popular agent frameworks using fuzzing combined with an active, ReAct-style oracle that inspects runtime state during execution. The oracle mechanism there is active inspection against a libraryâs exposed API. Ours is passive differential replay against a fixed set of scenarios, and the target is a deployed session-state layer rather than a framework itself. Evaluation of monitors rather than systems. Guerin et al. [2022] evaluate the discriminative power of machine-learning safety monitors using detection metrics against labeled data. We have no labeled corpus of real failures and construct the fault set by design instead, which is what makes a kill rate available at all. Relation to mutation testing. Constructing faulty program variants to measure whether a test suite detects them is mutation analysis, with a mature literature on first- versus higher-order mutants, surviving mutants as evidence of test inadequacy, mutation score as an adequacy criterion, and equivalent-mutant detection [DeMillo et al., 1978, Jia and Harman, 2011]. Our five single-fault variants are first-order mutants and the all-defects variant is a higher-order mutant. A suite that kills the higher-order mutant while a constituent first-order mutant survives is exactly the situation mutation analysis exists to expose. Our contribution is a post-freeze mutation score for a verification suite guarding agent infrastructure, together with a per-mutant account of where in the reachabilityâ infectionâpropagation chain each surviving fault stops. We are not aware of either being reported for a checking layer used as deployment evidence. Just et al. [2014] show that mutant detection correlates with real-fault detection, which is what makes a mutation score meaningful as evidence about a suite. Relation to metamorphic and differential testing.Redelivery, replay, reordering, and convergence under permuted arrival are metamorphic relations between executions rather than assertions about a single output, and the survivors we report are failures of source-test selection under those relations [Segura et al., 2016]. Differential testing classically compares independently written implementations as pseudo-oracles, and has a long history for stateful and distributed systems [Zhang et al., 2022, Gu et al., 2023]. We do not do that. Our oracle is explicit invariant assertions, and the distinction matters for what our result can claim. 3 The Verification Target: A Multi-Session Projection Layer We study a common architectural pattern in multi-session agent chat interfaces: each session maintains a client-side projection, a visible message list and a run-status field, built by applying a stream of asynchronous events to a prior state. Events arrive through more than one channel: an optimistic local echo when the user sends a message, an HTTP acknowledgment once the backend accepts it, a WebSocket push once the agent responds, and a periodic snapshot for reconnection and late- joining clients. We formalize the interface any implementation of this layer must satisfy as a 3 ProjectionAdapter: given a session identifier it produces an initial state, given a state and an event envelope it produces a new state, and it can render the visible messages from any state. Three additional hooks are optional,applyOptimistic,applyHttpAckandapplySnapshot, and a check that needs one skips (rather than silently passing) when an adapter does not implement it. This pattern admits six failure classes, which motivate the checks in Section 4: 1.Non-idempotent delivery. The same event, delivered twice (e.g., after a network retry), changes the visible projection rather than leaving it unchanged. 2.Non-idempotent replay. Re-consuming the full event stream from the start (e.g., on a reconnect that lacks a resumption cursor) changes the projection relative to consuming it once. 3.Cross-session leakage. An event addressed to one session is applied to another, typically the result of a broadcast channel that is not scoped by session identifier before dispatch. 4. Order-insensitive rendering. Events that arrive out of logical order are rendered in arrival order, with no buffering, gap detection, or explicit resynchronization signal. 5.Non-convergent concurrent update paths. The same logical message reaches the projection through more than one channel, an optimistic echo, an HTTP acknowledgment and a WebSocket push, and the three paths are reconciled differently depending on arrival order, rather than converging to one canonical message regardless of order. 6. Stale-state regression. A late-arriving snapshot or status update overwrites a projection that has already advanced past it, silently regressing terminal or otherwise-settled state. We take these six classes as given, as recurring consequences of this architectural pattern, and use them to construct the seven adapters described next. Nothing about the six classes or the adapters below is a claim about any specific deployed system. They are a fixed target for validating a checking methodology, not an audit of one. 4 Method: A Reference Implementation and Its Mutants Reference adapter.The correct implementation deduplicates by event identifier, upserts by message identifier rather than appending, scopes every event to its originating session before mutating state, buffers out-of-order events behind a monotonic cursor and flags a gap rather than rendering past it, reconciles the optimistic/acknowledgment/WebSocket paths for the same logical message into one entry regardless of arrival order, merges snapshots into live state rather than replacing it, and treats run-status updates as monotonic (a terminal status cannot be overwritten by a stale in-flight one). All-defects mutant. The all-defects mutant, NAIVE, is constructed, not discovered: it appends on every event with no deduplication or upsert logic, applies no session-scope guard before mutating state, renders strictly in arrival order with no buffering or gap detection, lets the HTTP-acknowledgment and WebSocket paths both append independently without reconciliation, replaces the entire projection on every snapshot rather than merging it, and takes the last write on run-status regardless of whether it regresses a terminal state. Each of these six choices is a direct, by-construction instantiation of one failure class from Section 3. Its faults are built in rather than inferred from observed behaviour, so whether a check catches them is a question about the check and not about ground truth. First-order mutants. A correct/all-defects pair separates two extremes, which a suite of very low resolution also does: firing on everything passes the broken side, firing on nothing passes the correct side. We therefore construct five first-order mutants, each identical to REFERENCE except for one injected fault, one per invariant family, a choice whose effect on family localisation Section 5 measures. â˘DEDUP-DEFECT: the event-identity deduplication step is removed. Redelivered events are reap- plied. â˘REGRESSION-DEFECT: the version guard on run-status updates is removed. Last write wins, so a stale replay can regress a terminal status. ⢠ISOLATION-DEFECT: the session-scope guard is removed. An event addressed to another session mutates this one. 4 1. Standard validation 2. First-order mutants 3. Held-out measurement 4. Dimension- guided repair reference vs. all-defects one fault each, co-designed specified externally, suite frozen first dimensions registered before fixtures written Suite certified 11/11 on reference 8 failures on all-defects 1 mutant survives all eleven checks +1 fixtureâ killed 5 of 10 killed 5 survivors two failure modes 10 of 10 killed 5 fixtures added oracles unchanged The evidence most suites have. Compatible with any resolution. Two guards in the reference both suppress the fault, so no fixture attributes it. A number instead of a clean record. Survivors locate the gap. Each survivor dies to the check written for its registered dimension. Three survivors were never activated. Two were activated but oracle-masked. Both are input gaps, and the oracles were reused unchanged. Figure 1: Four stages of evidence about one suite. Stage 1 is the validation such a suite normally receives, and it certifies this one. Stage 2 shows what the certificate omits: a mutant removing a single guard survives every check, because the reference enforces the property twice and no fixture separates the two enforcements. Stage 3 is the measurement, against mutants specified by a reader who designed none of the fixtures, on a suite frozen and hashed beforehand. Stage 4 is a repair evaluated on those same survivors, not a second measurement: its value is that the dimensions and their per-survivor predictions were registered before any fixture was written. â˘ORDERING-DEFECT: on a gap, the cursor advances to the highest sequence observed rather than the highest contiguous sequence applied. This is an ordinary mistake, tracking âlatest seenâ instead of âlast contiguousâ, and it compounds: once the cursor jumps the gap, the missing event is later rejected as stale, so the gap never heals. â˘CONVERGENCE-DEFECT: snapshots replace the projection instead of merging into it, so a late snapshot erases live state. Every other line of each mutant is the reference implementation verbatim, so a firing check is respond- ing to one syntactic change. It does not follow that the change has one behavioural consequence: a cursor or merge fault can violate several properties at once, and Section 5 shows that it does. Invariant suite. We start from eleven checks (C-01âC-11), grouped into five invariant families, exercised against eleven fixed, deterministic scenarios (duplicate delivery, full-stream replay, distinct- but-similar messages, a foreign-session event, a sequence gap, out-of-order delivery, three orderings of optimistic/ack/WebSocket convergence, two orderings of snapshot-plus-live-events convergence, a stale status update, and a same-sequence conflict). The five families are: IDEMPOTENCE (duplicate and full-stream replay must not change the projection), DISTINCTNESS / NO REGRESSION (messages that should stay distinct must not be merged, and terminal state must not regress), ISOLATION (a session must not be mutated by another sessionâs event), ORDERING (gaps must not silently advance the cursor, out-of-order delivery must render correctly or flag itself, and duplicate-sequence conflicts must not both silently win), and CONVERGENCE (all update paths for one logical message, and snapshot-plus-live-events under either arrival order, must reach the same state). Section 5 adds a twelfth check, then five more, for reasons the mutants supply. Requirements on the suite.Figure 1 summarizes the four stages. Two requirements are uncontro- versial: the suite must not fire on REFERENCE, and it must fire on NAIVE, since a suite that never fires certifies nothing. Both are satisfied, and Section 5 shows that satisfying both is compatible 5 with carrying a hole. The requirement that does discriminate is a post-freeze one: given mutants specified by someone who did not design the fixtures, against a suite frozen before those mutants exist, what fraction does it kill? Family localisation, where a single-fault mutant fails only inside its own invariant family, is not among the requirements: Section 5 shows it is a property of how mutants are chosen rather than of the suite. 5 Results 5.1 Outcome of the Standard Validation Against the reference/all-defects pair the eleven-check suite behaves as a validated instrument should: REFERENCE passes all eleven, NAIVE fails eight and is skipped on one. This is the evidence such a pair can produce, and it is the evidence most suites have. A first-order mutant that removes event-identity deduplication and changes nothing else fails zero of the eleven, including both checks whose stated subject is idempotence under redelivery. Two properties of the fixture set independently prevent the mutant from becoming distinguishable from the reference. The duplicate and replay fixtures deliver both copies with a sequence number, and the reference rejects any event whose sequence does not exceed the cursor, so the ordering guard suppresses the redelivery before the absent deduplication matters. Independently, those fixtures redelivermessage.created, applied by upserting onmessageId. Writing the same key twice yields the same state, so even unsequenced the duplicate leaves no trace. The idempotence family was not testing deduplication. It was testing ordering. The repair is a check-level change even though the cause is a fixture: an input that is simultaneously unsequenced and append-shaped defeats both suppressors. We add one, a streamingmessage.delta with a repeated eventId and no sequence, as C-12. 5.2 Post-Freeze Mutation Kill Rate Repairing against the mutant that exposed a hole and then reporting that the mutant is caught measures nothing. We froze the twelve-check suite, recorded its hash, and only then obtained ten further first-order mutants specified by an adversarial reader of this manuscript who had the system description and the check titles but designed none of the fixtures and did not see their contents. All ten specifications were implemented, and none was discarded as invalid or equivalent. The suite was run once and not modified. It killed five of ten (Table 1, column A). REFERENCE still passed 12/12, so the survivors are not an artifact of a suite that had stopped firing. Because the mutants were specified adversarially rather than sampled, 5/10 is a challenge-set score and not an estimate of the detection rate under any natural fault distribution. Detection across invariant families. The five mutants written alongside the families each fail only inside their own family. The externally specified set does not. H1 keys deduplication onmessageId rather thaneventId, an idempotence fault under any reasonable taxonomy, and is caught only by C-08, a convergence check, because the optimistic/acknowledgment/WebSocket path is the sole fixture where the two keys diverge. H4 and H9 are structurally unrelated snapshot faults and are both caught by C-09 alone. Family localisation is a property of how mutants are chosen, not of the suite. 5.3 Two Modes of Missing a Fault Why a suite misses is not visible from its pass/fail output, so we measured it. For every (mutant, fixture) pair we replayed the fixture step by step against both the reference and the mutant, serialising the full internal state and the observable projection after each step, and recorded where the two first diverge. A fault that never diverges internally was never activated by that input. One that diverges internally but never observably is masked at the oracle boundary. Only the latter is in reach of a better assertion on the same inputs. Under the twelve pre-repair fixtures, seven of the ten mutants are activated and five of those propagate to an observable output. The five that propagate (H1, H4, H5, H7 and H9) are exactly the five the suite kills, and the five survivors split into the two modes below. 6 â˘Never activated (H6, H8, H10). The mutated code executes but computes the same result. H6 compares session identifiers case-insensitively and every fixture uses identifiers differing by more than case, so the altered comparison returns what the original returned. H8 permits an equal-version regression and the stale-status fixture supplies a strictly lower version, so the widened branch is never taken. H10 lets an acknowledgment overwrite a richer status and no fixture delivers an acknowledgment after the event it acknowledges. No assertion over these inputs could have caught any of the three, however strong. â˘Activated but oracle-masked (H2, H3). Internal state does diverge. H2 removes deduplication for non-message events, so the mutantâs seen-event set differs from the referenceâs after the first run.*event, and no fixture then redelivers one, so the divergence stays internal. H3 bounds the deduplication cache to a single entry, and the difference is real from the second distinct event onward, but no fixture redelivers an event after an intervening one. These two are within reach of a wider oracle as well as of new inputs. The distinction matters because it separates two repairs that a bare kill rate conflates. Three of five gaps could only ever be closed by new input values. Two could be closed either by new inputs or by exposing more internal state to the checks. We take the input route for both, because it keeps the oracles fixed and therefore keeps the experiment interpretable, but a practitioner facing H2 or H3 has a genuine choice. 5.4 Discriminating Fixture Coverage as a Registered Prediction If the survivors fail because no input distinguishes mutant from reference, then what a suite resolves is fixed by the shapes of its inputs, and input shape is enumerable in a way that âassert more propertiesâ is not. This is testable. Before writing any new fixture we enumerated the dimensions along which the referenceâs control flow can discriminate two executions, recorded which the existing twelve fixtures cover, and recorded which survivors the uncovered ones should account for. D1 event shape: upsert-shaped vs. append-shapedcovered D2 sequencing: sequenced vs. unsequencedcovered (by C-12) D3 event family: message.* vs. run.*uncoveredâ H2 D4 redelivery distance: immediate vs. after an intervening eventuncoveredâ H3 D5 session identity: exact vs. differing only in caseuncoveredâ H6 D6 version relation: strictly lower vs. equaluncoveredâ H8 D7 channel order: WebSocket before ack vs. ack afteruncoveredâ H10 We generated five fixtures, one per uncovered cell, without consulting any mutantâs source, and added checks C-13 to C-17 that reuse existing oracle predicates. All five survivors died (Table 1, column B), and every registered prediction held: H2 to C-13, H3 to C-14, H6 to C-15, H8 to C-16, H10 to C-17. One correspondence is not exclusive. C-13, written for therun.*dimension, also kills H3, which is worth reporting because it shows the dimensions are not orthogonal: a redeliveredrun.*event also sits at distance one from its original. REFERENCE passes all seventeen checks, so the added fixtures produce no failures on correct code. Repair result versus held-out measurement. The dimensions were derived by inspecting the same five survivors the repair is evaluated on. What the experiment establishes is that a written-down diagnosis correctly predicted which inputs would expose which faults, five times out of five, before those inputs existed. What it does not establish is that the seventeen-check suite would score 10/10 against unseen mutants, or that seven dimensions are complete. Oracle reuse. Each new check pairs a new input with an assertion already in the suite (Table 2). This is what licenses locating the gap in the inputs: had we also strengthened the assertions, the two causes would be confounded. 6 Discussion Activation gaps and masking gaps. The usual response to an inadequate suite is to assert more properties. For three of our five survivors that response could not have worked: the mutated code 7 Table 1: Externally specified mutants against the frozen suite.âmeans the mutant was killed (at least one check failed).âmeans it survived. Column A: the twelve-check suite, frozen and hashed before any of these mutants was written. Column B: after adding one fixture per uncovered dimension, oracles unchanged. The dimension column records the prediction registered before the fixtures existed. REFERENCE fails no check in either column (12/12 and 17/17). Injected faultABKilled by (registered dim.) H1deduplication keyed on messageIdâC-08 (convergence) H2deduplication only for message.*ââC-13(D3) H3deduplication cache bounded to one entryââC-14, also C-13(D4) H4snapshot resets the cursor unconditionallyâC-09 H5over-aggressive merge on role + textâC-03 H6session identifier compared case-insensitivelyââC-15(D5) H7off-by-one contiguity testâC-05, C-06 H8equal-version update regresses a terminal stateââC-16(D6) H9snapshot drops live-only messages from orderâC-09 H10acknowledgment overwrites a richer statusââC-17(D7) Killed5/1010/10 Table 2: The five added checks reuse existing oracle predicates unchanged, and only the input differs. digestOfcompares the rendered projection before and after a step,runStatusreads a runâs status, visibleMessages reads the rendered list. New checkNew input dimensionOracle predicate, reused from C-13 run.* event family (D3) digestOf + runStatus, as C-01 and C-10 C-14redelivery after an intervening event (D4) digestOf before/after, as C-01 C-15session identifier differing only in case (D5) digestOf before/after, as C-04 C-16equal version rather than strictly lower (D6) runStatus, as C-10 C-17 acknowledgment after the WebSocket event (D7)digestOf + visibleMessages, as C-07 produced identical internal state on every fixture, so no assertion over those inputs had anything to distinguish. For the other two an assertion over internal state would have worked, and so did a new input. Distinguishing the modes requires instrumenting mutant against reference, which a pass/fail report does not do, and it changes what the practitioner should do next. This is also why we take the input route uniformly: it holds the oracles fixed, so the repair is attributable. Evidential value of a monitorâs silence. A suite that has never fired unexpectedly is compatible with a suite that cannot fire. Both produce the same record, and that record is what is usually offered when a monitorâs output licenses a deployment. A post-freeze challenge score breaks the tie: it is falsifiable, degrades visibly when the suite is weakened, and can be recomputed after any change. Ours went from certified-by-the-standard-method to 5/10 against an adversarial challenge set, a statement about what a determined reader could get past this suite. Redundancy in the system under test. The original miss was not a coding error. Every check does what its title says. Deduplication by event identity and rejection by sequence staleness both suppress a redelivered event, so no fixture exercising redelivery can attribute the outcome to either. Condition-coverage criteria such as MC/DC exist precisely to force inputs on which one condition independently determines the outcome [Hayhurst et al., 2001]. The same masking arises here between two implementation mechanisms rather than two clauses of one predicate, and it is not addressed by any coverage criterion the suite tracks. Relation to input-space partitioning.Our seven dimensions are a small input-parameter model in the tradition of category-partition and combinatorial testing [Ammann and Offutt, 2016, Kuhn et al., 2013]. What is new here is the direction of use: rather than deriving partitions from a specification up front, we use externally specified surviving mutants to reveal which partitions a verification suite lacks, and then test whether the resulting model predicts the repair. The model extends as challenge sets grow: redelivery distance beyond one intervening event (a cache bounded to two entries needs 8 distance two), session identity under whitespace, Unicode normalisation and encoding variants, version relations with absence and wraparound, and channel orders over longer traces. Invariant families as an organising unit.Grouping checks into families is useful for writing and reading them, and is the natural unit in which coverage gets described. Our data show it is the wrong unit for the second purpose: a deduplication fault surfaced in a convergence check, and two unrelated snapshot faults in the same check. Coverage arguments of the form âevery family has checksâ are therefore not arguments about detection. Deployment semantics of a skip. One check cannot be attempted against NAIVE, which exposes neither hook it requires. Three-valued verdicts are standard in runtime verification. What a gate additionally needs is the decision attached to the third value: for a property whose violation is unrecoverable, inability to observe it plausibly has to fail closed. Scope. All implementations are synthetic and deterministic. The controlled design establishes internal validity of the mutation outcomes, since we know exactly which fault is present, but limits external validity to hand-written first-order faults, which are an imperfect proxy for those developers introduce [Just et al., 2014]. The reader who specified the held-out mutants saw the system description and the check titles but not the fixtures, which makes the set adversarial with respect to the stated properties rather than blind. Two extensions follow directly: a second challenge round against the frozen seventeen-check suite by a reader who has not seen these survivors, and the same checks bound to an implementation the authors did not write. 7 Conclusion The evidence normally offered for a verification suite guarding agent infrastructure is that it passes code believed correct and fails code believed broken. We measured what that evidence is worth for one such suite and found it worth little: the certified suite was survived outright by a first-order mutant, and killed five of ten faults specified by a reader who had not designed its fixtures. The failures were not scattered, but neither were they uniform. Instrumenting each survivor against the reference showed two modes: three faults that no fixture ever activated, and two that corrupted internal state no oracle could observe. Only the second kind is reachable by a stronger assertion on the same inputs, so a bare kill rate conflates two problems with different remedies. Both are gaps in the input set, and treating them that way is testable: we enumerated the dimensions along which the implementation can discriminate two executions, registered which were uncovered and which survivors they should explain, and added one fixture per uncovered dimension with the oracles unchanged. All five predictions held, and because they were registered before any fixture existed, the repair is checkable as a prediction on the survivors that motivated it rather than read as a second measurement. For a layer whose output is meant to license a deployment decision, the failure direction that matters is not a monitor that fires too readily but one that stays quiet in a way a passing report presents as success. The procedure that surfaces this needs little beyond what such a suite already has: each mutant is the reference implementation with a single edit, and the measurement is one run of the existing suite. References P. Ammann and J. Offutt. Introduction to Software Testing. Cambridge University Press, 2nd edition, 2016. R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34â41, 1978. J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu. Acto: Automatic end-to-end testing for operation correctness of cloud system management. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), 2023. 9 J. Guerin, R. S. Ferreira, K. Delmas, and J. Guiochet. Unifying evaluation of machine learning safety monitors. In IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE), pages 414â422, 2022. K. J. Hayhurst, D. S. Veerhusen, J. J. Chilenski, and L. K. Rierson. A practical tutorial on modified condition/decision coverage. Technical Report NASA/TM-2001-210876, NASA Langley Research Center, 2001. Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649â678, 2011. R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 2014. D. R. Kuhn, R. N. Kacker, and Y. Lei. Introduction to Combinatorial Testing. Chapman and Hall/CRC, 2013. M. Long, Y. Zhao, and H. Wang. LogicHunter: Testing LLM agent frameworks with an agentic oracle. arXiv preprint arXiv:2607.06195, 2026. C. Paduraru, P.-L. Bouruc, and A. Stefanescu. A trace-based assurance framework for agentic AI orchestration: Contracts, testing, and governance. arXiv preprint arXiv:2603.18096, 2026. A. Reuel, B. Bucknall, S. Casper, T. Fist, L. Soder, O. Aarne, L. Hammond, et al. Open problems in technical AI governance. Transactions on Machine Learning Research, 2025. S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-CortĂŠs. A survey on metamorphic testing. IEEE Transactions on Software Engineering, 42(9):805â824, 2016. W. Wu. When errors become narratives: A longitudinal taxonomy of silent failures in a production LLM agent runtime. arXiv preprint arXiv:2606.14589, 2026. Z. Zhang, B. Yuan, K. Yang, D. Zou, and H. Jin. StateDiver: Testing deep packet inspection systems with state-discrepancy guidance. In Proceedings of the 38th Annual Computer Security Applications Conference (ACSAC), 2022. 10