Paper deep dive
FaultLens: Learning Compact Behavioral Test Suites for Generated Operational Programs
Zeming Liu, Hang Lyu, Jingtao Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/28/2026, 4:07:04 AM
Summary
The paper introduces FaultLens, a method for learning compact behavioral test suites for generated operational programs. It combines a fault-driven greedy ranking with a mutation-independent diversity ranking to select probes from an exhaustive domain. Evaluated across twenty generated policies and four environments, the hybrid approach achieves 99.0% coverage of dynamically killable faults using only 1.2-2.0% of the exhaustive probe domain. The method also demonstrates improved generalization to unseen fault families and reduces severe tail regressions in downstream deployment scenarios.
Entities (10)
Relation Signals (7)
FaultLens â achievescoverageon â Generated Operational Programs
confidence 95% ¡ A 32-probe hybrid learned on generations 1-3 covers 576/582 (99.0%) dynamically killable faults in generations 4-5
FaultLens â uses â Diversity Ranking
confidence 95% ¡ 2. a diversity ranking ignores mutant outcomes and covers structural features of the probe domain.
FaultLens â uses â Active Greedy Ranking
confidence 95% ¡ FaultLens therefore combines two rankings: 1. an active ranking greedily covers training mutants using their executed kill relation
FaultLens â creates â Sparse Executed Outcome Cache
confidence 90% ¡ It executes a rich probe domain once, stores the fault-probe kill relation as a sparse outcome cache
Diversity Ranking â improvesgeneralizationfor â Unseen Fault Families
confidence 90% ¡ When an entire fault family is withheld from training, diversity raises scenario-family macro coverage from 84.6% to 94.9%.
Downstream Admission Rule â reduces â Severe Tail Regressions
confidence 90% ¡ a conservative admission rule reduces severe tail regressions from 15/20 program-environment groups to 0/20.
Sparse Executed Outcome Cache â stores â Fault-Probe Kill Relation
confidence 90% ¡ stores the fault-probe kill relation as a sparse outcome cache
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generated operational programs are often validated with either a few hand-written examples or exhaustive regression suites. The former can miss sparse boundary and interaction faults, while the latter can be unnecessarily expensive. We introduce FaultLens, a method for learning compact behavioral test suites while preserving an auditable connection to executed evidence. It executes a rich probe domain once, stores the fault-probe kill relation as a sparse outcome cache, and learns probe orderings only from earlier program generations. A fault-driven greedy component exploits known kill structure, while a mutation-independent diversity component covers probe families, cases, templates, and temporal bins. Their alternating hybrid remains useful when a new program contains a fault mechanism absent from ordering construction. We evaluate twenty generated operational policies across four environments, ten execution seeds, 1,200 measured run summaries, 2,160 controlled program transformations, and 4,120,200 executed program-probe pairs. Of 1,960 intended faulty transformations, 1,779 alter a contract or output somewhere in the finite audit domain; 200 additional controls preserve behavior. A 32-probe hybrid learned on generations 1-3 covers 576/582 (99.0%) dynamically killable faults in generations 4-5 using 1.2-2.0% of the exhaustive domain. With an entire fault family withheld from training, diversity raises scenario-family macro coverage from 84.6% to 94.9%. In a downstream deployment study, a conservative admission rule reduces severe tail regressions from 15/20 program-environment groups to 0/20. FaultLens provides a prioritized evidence mechanism, not a proof of correctness, and makes its budget, evidence source, generalization split, and misses explicit.
Tags
Links
- Source: https://arxiv.org/abs/2608.26746v1
- Canonical: https://arxiv.org/abs/2608.26746v1
Trouble viewing inline? Open PDF directly â
Full Text
35,444 characters extracted from source content.
Expand or collapse full text
FaultLens: Learning Compact Behavioral Test Suites for Generated Operational Programs Zeming LiuHang LyuJingtao Zhang Independent Researchers August 2026 Abstract Generated operational programs are commonly validated with either a few hand-written examples or an exhaustive regression suite. The former misses sparse boundary and interaction faults; the latter can be unnecessarily expensive when validation crosses time, observations, thresholds, and action budgets. We introduce FaultLens, a method for learning compact behavioral test suites while preserving an auditable connection to executed evidence. The method executes a rich probe domain once, stores the resulting faultâprobe kill relation as a sparse outcome cache, and learns probe orderings only from earlier program generations. A fault-driven greedy component exploits known kill structure. A mutation-independent diversity component covers probe families, cases, templates, and temporal bins. Their alternating hybrid is designed to remain useful when the next program contains a fault mechanism not seen during ordering construction. We study twenty generated operational policies in a resource-selection case study with four environments, ten execution seeds, 1,200 measured run summaries, 2,160 controlled program transformations, and 4,120,200 executed programâprobe pairs. Of 1,960 intended faulty transformations, 1,779 alter a contract or output somewhere in the finite audit domain; 200 additional controls preserve behavior. A 32-probe hybrid learned on program generations 1â3 covers 576/582 (99.0%) dynamically killable faults in generations 4â5, using 1.2â2.0% of the exhaustive domain. When an entire fault family is withheld from ordering construction, diversity raises scenarioâfamily macro coverage from 84.6% to 94.9%. As a downstream deployment study, a conservative admission rule reduces severe tail regressions from 15/20 programâenvironment groups to 0/20. The results show that compact test selection can generalize across generated programs and partially across fault mechanisms, while also making its misses explicit. 1 Introduction Code-generating models increasingly produce small operational programs that map telemetry and state to concrete actions. Examples include selecting a cache policy, choosing a replica set, activating a feature configuration, or allocating a limited set of database indexes. Such programs are attractive because they can be inspected, versioned, tested, and rolled back. They are also difficult to validate: a program can parse and return legal actions while encoding a shifted phase boundary, an order dependency, stale state, a damaged threshold, or an unsafe resource set. Other generated artifacts expose similarly structured requirements for physical consistency, temporal prompt transitions, and contentâstyle separation [1â3]. Executing generated artifacts is now a standard part of agent evaluation [4], but the validation budget is rarely treated as a first-class object. A few examples provide weak behavioral evidence. Exhaustive scenario-derived suites provide better evidence but may cross every round, observation type, threshold neighborhood, multiplicity, and permutation. That cost is especially visible when a probe invokes an external planner or remote system rather than a microsecond-scale local function. 1 arXiv:2608.26746v1 [cs.SE] 27 Aug 2026 This paper asks a narrower question than program synthesis or policy learning: given an interpretable exhaustive probe domain and historical generated programs, can we learn a much smaller suite that retains fault sensitivity on future programs? The distinction matters. We do not generate the operational program, optimize its task objective, or claim semantic correctness. We learn which already-defined behavioral probes should run first under a declared budget. The main challenge is generalization. Mutation-guided prioritization can find high-yield tests, but it can overfit to correlated mutants. A selector trained on several threshold faults may repeatedly choose nearly identical threshold probes. The next failure may instead be temporal or order-sensitive. FaultLens therefore combines two rankings: 1. an active ranking greedily covers training mutants using their executed kill relation; and 2. a diversity ranking ignores mutant outcomes and covers structural features of the probe domain. The hybrid alternates unique probes from both lists. This fixed allocation spends half the nominal budget exploiting observed evidence and half exploring scenario structure. We evaluate it with two separation protocols. The first learns from program generations 1â3 and evaluates generations 4â5. The second removes an entire fault family from training and evaluates only that family on future programs. Our contributions are: ⢠a sparse counterfactual outcome cache that evaluates arbitrary probe subsets without synthe- sizing outcomes or rerunning programs; ⢠a hybrid activeâdiversity ordering for compact behavioral validation; ⢠cross-program and whole-fault-family protocols that expose selector overfitting; and ⢠an executed case study comprising twenty generated programs, 2,160 transformations, 4.12 mil- lion programâprobe executions, downstream operational measurements, and fully reproducible figures. The strongest result is not exhaustive detection. The complete domain defines which transformations are behaviorally effective. The substantive result is that a ranking learned without the held-out programs retains 99.0% coverage at 32 probes, and structural diversity materially improves generalization when the held-out fault mechanism is unknown. 2 Problem Setting 2.1 Generated Operational Policies At decision roundt, an operational program receives observationsO t and returns an action set A t under resource budget B: Ď(t,O t )â A t , A t âA, |A t |⤠B,(1) whereAis the declared action catalog. The program is expected to be deterministic for identical inputs, invariant to duplicate records when observations represent a set, and invariant to input permutation when order is not part of the task semantics. These properties are domain contracts, not universal properties of all software. The validation input is a generated candidateĎ c and a finite probe domainP s for environment s. Every probe records a round, an observation multiset, a family, and a human-readable case. The output validator normalizes action order and checks type, uniqueness, budget, and catalog membership. 2.2 FaultâProbe Relation LetM s be controlled transformations of historical programs. A probepkills mutantmwhen the normalized output differs from the unmodified program or becomes invalid: K(m,p) = 1[out(m,p)̸= out(Ď,p)].(2) 2 historical programs full be- havioral probe domain sparse executed outcome cache active + diversity probe ranking budgeted test of a new program Figure 1: FaultLens learns a compact ordering from executed historical evidence. The new program executes the selected prefix rather than replaying cached outputs. Static contract changes are tracked separately. A transformed program is domain-effective if it changes a static contract or an output somewhere in the complete finite domain. Domain- equivalent mutants are reported and excluded from dynamic-coverage denominators, following standard mutation-study practice [5]. For a selected probe set QâP s , dynamic coverage is C(Q) = 1 |M Ⲡs | X mâM Ⲡs 1[QâŠp : K(m,p) = 1̸=â ],(3) whereM Ⲡs is the evaluation universe of dynamically killable mutants. The selector must choose an ordered prefix under budgetbusing only training programs and allowed training fault families. 2.3 Evaluation Questions We ask four questions: RQ1. Which probe families expose each controlled fault family? RQ2. How much cross-program fault coverage remains as the probe budget shrinks? RQ3. Does structural diversity help when an entire fault family is absent from training? RQ4. Can compact behavioral validation support a safer downstream deployment decision? 3 Method Figure 1 summarizes the workflow. Historical programs are used offline to construct executed outcome evidence and a probe ranking. A new program still executes the selected probes; the cache does not predict its outputs. Task measurements, when available, are a separate downstream admission layer. 3.1 Contracts Before Prioritization Test prioritization is useful only after inexpensive fail-closed checks. Our case-study implementa- tion parses Python into an AST, verifies one declared entry point, rejects imports and dynamic call targets, and disallows global or nonlocal mutation. Accepted code executes in a fresh process under CPU, address-space, output-size, and wall-time limits. Return values are checked again for type, duplicates, resource budget, and catalog membership. These checks reduce accidental failures; they are not an adversarial sandbox. The prioritizer operates only on baseline programs that pass the static and metamorphic contracts. Static-only faults do not consume a dynamic probe budget. 3.2 Scenario-Derived Probe Domain The exhaustive domain crosses time with observation structure. At every round it includes: ⢠two identical empty inputs for repeatability; â˘eight values around each declared threshold: 0, 0.5, 0.95, 0.999, 1.0, 1.001, 1.05, and 2.0 times the threshold; 3 ⢠duplicate and triplicate observations for every template; ⢠both orders of every distinct template pair; ⢠mixed highâlow template pairs; and ⢠an unknown observation template. The resulting domains contain 1,620 probes for two-template environments and 2,700 probes for the three-template environment. Repeat, duplicate, and permutation pairs also define metamorphic relations [6]. The grammar is intentionally interpretable: a selected probe maps back to a round, threshold case, template combination, and declared relation. 3.3 Sparse Executed Outcome Cache The full domain is executed once for every baseline and transformed program. Instead of storing a dense|M|Ă|P|matrix, the cache stores one record per mutant with static outcomes and only the IDs of killing probes. Equation 3 can then be evaluated by set intersection. This representation enables thousands of counterfactual subset evaluations without further program or system execution. It is exact relative to the frozen full-suite trace; it does not establish equivalence under arbitrary probe reorderings for a hidden-state program. Static state rejection and repeatability relations reduce this concern, while high-assurance deployments can execute each selected probe in a fresh process. 3.4 Active Greedy Ranking For training mutant set U, the active selector repeatedly chooses p â = arg max pâP |mâ U : K(m,p) = 1|,(4) addsp â toQ, and removes its newly killed mutants fromU. Ties are broken by stable probe ID. This is a set-cover heuristic [7] and accounts for overlap ignored by sorting probes on individual kill frequency. Pure greedy selection can nevertheless over-specialize. Once it finds a high-yield region, its early ranks can be dominated by structurally similar probes. This is desirable exploitation when future faults match training, but fragile when the fault mechanism shifts. 3.5 Mutation-Independent Diversity Ranking The diversity ranking never reads K. It maps each probe to categorical features: ⢠probe family and case; ⢠five-round temporal bin; ⢠familyâtime and caseâtime interactions; and ⢠nonnumeric template tokens encoded by the stable probe ID. A second greedy pass covers the largest number of unseen structural features. The hybrid alternates unique probes from the active and diversity rankings. The 1:1 ratio is fixed before evaluation. At very small budgets, the hybrid is expected to trail active selection because some choices explore rather than kill known mutants. At larger budgets, that exploration may improve transfer to unseen programs and unseen fault families. 4 Study Design 4.1 Operational Resource-Selection Case Study We evaluate generated functions that select a limited action set over 60 rounds. The concrete action is activating database indexes, but the validation method sees only rounds, observation 4 Table 1: Case-study environments and full behavioral domains. Environment Templates Actions Budget Rounds Probes Burst25260 1,620 Composite37160 2,700 Cycle24160 1,620 Rare25160 1,620 Table 2: Controlled transformations. âEffectiveâ means a static contract or output changes somewhere in the complete finite domain. CategoryGenerated Effective Families Contract integrity440440 import, state, catalog, budget Temporal behavior560488 dropout, shift, partial schedule Observations720621 drop, duplicate, order, threshold, template Action choice240230 action substitution Fault candidates1,9601,779 13 families Equivalent controls200200 5 control families records, action catalogs, and budgets. The four environments represent incident bursts, a three- phase composite workload, periodic switching, and rare high-cost events. Each environment uses a 120k-row SQLite instance. Five independent generation runs produce one Python policy per environment, yielding twenty programâenvironment groups. Ten execution seeds and six operational baselines yield 1,200 frozen measurement rows. Program generation is not evaluated in this paper; the generated artifacts are treated as fixed subjects identified by hashes. 4.2 Transformation Corpus Each program receives 98 intended faulty transformations across thirteen families and ten behavior-preserving controls across five families. Fault parameters vary temporal modulus and offset, active windows, round shifts, templates, thresholds, and replacement actions. The complete execution contains 4,120,200 programâprobe pairs. Of 1,960 fault candidates, 181 are equivalent on the finite domain, leaving 1,779 effective instances. Sixty are detected only statically, 57 through both channels, and 1,662 only dynamically. None of the 200 controls changes a static contract or output. The complete-domain numbers establish the replay universe. They should not be read as an independent 100% detection claim: a dynamically effective mutant is defined by changing at least one full-domain output. 4.3 Generalization Protocols For cross-program evaluation, generations 1â3 construct a separate ranking for each environment and generations 4â5 are used only for evaluation. Only contract-passing baseline programs enter the dynamic experiment. The held-out universe contains 582 dynamically killable instances. For whole-family holdout, one fault family is removed from ordering construction. The ranking is evaluated on that family in future programs. We report macro coverage over 42 nonempty environmentâfault-family cells, so large mutation families do not dominate. Baselines are active greedy, the hybrid, individual kill frequency, and 200 deterministic random rankings per environment. Random curves report means and 5thâ95th percentile envelopes. 5 Fault candidates Detection channels Equivalent controls 1,779 applicable 181 equivalent on audit domain 1,662 60 static-only ¡ 57 both 0 / 200 flagged static only both dynamic only A Full-suite evidence 1248163264128256512 Probe budget (full suite: 1,620â2,700) 50 60 70 80 90 100 Held-out killable faults covered (%) 99.0% at 32 probes (1.2â2.0% of full suite) B Cross-policy probe efficiency (trials 1â3 â 4â5) Active greedy Hybrid: active + diversity Frequency Random Figure 2: Complete executed evidence and cross-program probe efficiency. The ranking is learned on generations 1â3 and evaluated on generations 4â5. Random shading is the weighted 5thâ95th percentile range over 200 rankings. 4.4 Downstream Operational Admission Behavioral validity does not imply operational utility. To study downstream use, each program is paired with a validated restricted reference policy. Seeds 1â5 are used for admission and seeds 6â10 are held out. For execution seed i, normalized loss is L i (Ď) = 1 2 P i (Ď) P i (Ď 0 ) + 1 2 V i (Ď) max(1,V i (Ď 0 )) ,(5) wherePis p95 latency andVis the number of operational threshold violations. A free program is admissible only when it passes behavioral contracts, improves calibration loss, and one-sided 95% bootstrap upper bounds on latency and violation harm satisfy a declared operating profile. The Safe profile permits at most 5% p95 harm; Balanced permits 30%. Both permit 0.02 normalized violation harm. These profiles are fixed before held-out evaluation. 5 Results 5.1 RQ1: Evidence Is Distributed Across Probe Families Figure 2A summarizes the complete executed evidence. The dynamic channel is essential: 1,662 effective faults do not change a static contract. Figure 3 provides the more informative view. Some faults, including budget overflow and complete round dropout, are exposed by nearly every probe family. Other rows are selective. Duplicate-sensitive programs require multiplicity probes. Threshold corruption requires threshold coverage. Order-sensitive programs require multi-observation inputs. Round shifts combine temporal and catalog evidence. No individual probe family dominates all fault rows. This motivates both a rich complete domain and explicit structural exploration in the compact ranking. 5.2 RQ2: Compact Suites Transfer Across Programs Table 3 reports scenario-weighted coverage. Active greedy is strongest at budgets 4 and 8 because every early selection exploits observed kills. At 16 probes, active and hybrid both cover 571/582 (98.1%) held-out faults, versus 89.2% for random. At 32 probes, hybrid reaches 576/582 (99.0%) 6 repeat threshold multiplicity permutation mixed catalog budget overflow duplicate sensitive index swap observation drop order sensitive partial schedule round dropout round shift template swap threshold corruption unknown index 100100100100100100 00100000 100100100100100100 010010050500 001001001000 100100100100100100 100100100100100100 10025255050100 085851001000 010048484848 100100100100100100 Failure atlas: which probe families expose which faults? 0%50%100% Coverage 56 38 80 32 42 56 84 32 52 54 56 n Figure 3: Held-out failure atlas. Each cell gives the fraction of dynamically killable future-program mutants in a fault row exposed by all probes in the column family. Table 3: Cross-program held-out dynamic coverage (%). Budget Active Hybrid Frequency Random 486.477.755.259.9 896.491.966.576.9 1698.198.177.789.2 3298.199.081.895.6 6498.199.086.698.1 128 100.099.193.599.1 while using only 1.2% of the 2,700-probe domain or 2.0% of a 1,620-probe domain. Composite, Cycle, and Rare reach 100%; the six remaining misses occur in Burst. The result also shows that hybrid is not uniformly superior. Exploration costs coverage below 16 probes. At 128 probes, active reaches 100% while the alternating hybrid remains at 99.1%. A deployment with an extremely small budget should prefer exploitation when the historical fault model is trusted; a higher-assurance deployment can run a longer active prefix or the full domain. The full batch audit has median wall time 59.1 ms per baseline program, while measured policy calls account for roughly 1.5 ms; process startup dominates in this micro-policy case study. We therefore claim a reduction in executed probe count and validation payload, not a measured 50Ăwall-time speedup. Savings should be larger when probes invoke planners, containers, or remote systems. 5.3 RQ3: Diversity Helps on Unseen Fault Families Whole-family holdout creates a stronger distribution shift. At budget 32, pure active greedy achieves 84.6% macro coverage over the 42 nonempty environmentâfamily cells. Hybrid reaches 94.9%. Figure 4 aggregates by fault family. Diversity is equal or better on all eleven dynamic families. The largest gains occur for threshold corruption, round shifts, and partial schedulesâ precisely the faults that require coverage across structural cases or temporal bins rather than 7 25%50%75%100% Coverage of an entirely held-out fault family at 32 probes threshold corruption round shift partial schedule template swap observation drop duplicate sensitive budget overflow index swap order sensitive round dropout unknown index n=54 n=32 n=56 n=52 n=32 n=38 n=56 n=80 n=42 n=84 n=56 â Active greedyâ Hybrid scenarioâfamily macro: 84.6% â 94.9% Diversity improves unseen-fault generalization Figure 4: Whole-fault-family holdout at budget 32. Ordering construction excludes the rowâs fault family on generations 1â3; evaluation uses only that family in generations 4â5. Table 4: Held-out operational results. A severe regression is a programâenvironment group with mean p95 more than 5% above the reference. MethodLoss p95 ms Violations Regressions Free programs0.08380.85229.1515/20 Restricted reference 0.02690.30310.330/20 Calibration mean0.02470.3575.165/20 FaultLens-Safe0.02690.30310.330/20 FaultLens-Balanced 0.02510.3456.194/20 Evaluation oracle0.02460.3565.155/20 another high-frequency training probe. The hybrid is not complete under this shift. Coverage is 68.8% for round shifts and 81.2% for observation drop. Diversity is therefore a robustness mechanism, not a substitute for adding incident-derived faults and probes to the domain. 5.4 RQ4: Downstream Admission Reduces Tail Regressions Table 4 evaluates held-out seeds. Deploying every free program yields normalized loss 0.0838 and severe p95 regressions in 15 of 20 programâenvironment groups. Always using the restricted reference yields 0.0269 and zero regressions. Selecting by calibration mean lowers loss to 0.0247 but accepts five regression groups. The Safe profile selects two free programs, achieves loss 0.0269, and has zero observed severe regressions. Balanced selects six, reduces mean violations from 10.33 to 6.19, and lowers loss to 0.0251 while accepting four regression groups. The evaluation-only oracle also selects five regressing groups because scalar loss can exchange large tail harm for violation reduction. Figure 5 separates latency and violations. The declared p95-harm margin creates a visible transition: margins through 0.20 have zero observed severe regressions; at 0.30, four appear. A single scalar score would hide this operating choice. 8 0.30.40.50.60.70.8 Held-out mean p95 latency (ms) â better 5 10 15 20 25 30 Held-out operational violations â better Free program Structured â Safe FaultLens-Balanced Mean select â Oracle marker size â severe p95-regression groups A Riskâutility frontier 0.00.10.20.30.40.5 Declared p95-harm margin 0.0250 0.0255 0.0260 0.0265 0.0270 Held-out normalized loss 4R 5R5R zero observed severe regressions B Declared operating point 6 7 8 9 10 Mean violations Figure 5: Downstream risk and utility. Left: deployment strategies in the latencyâviolation plane. Right: the declared p95-harm margin sweep; marker labels give severe regression counts. 6 Analysis and Ablations 6.1 Why Frequency Is Not Enough Frequency prioritization chooses probes with many individual training kills but does not account for redundant coverage. At budget 32 it reaches 81.8%, compared with 98.1% for active set cover. The 16.3-point difference measures the value of updating the uncovered set after every selection. Test prioritization work has long emphasized early fault revelation [8,9]; generated programs add the need to define a scenario grammar and an explicit program-generation split. 6.2 Why Diversity Helps On the ordinary cross-program split, active and hybrid differ by less than one point at budget 32. The whole-family holdout reveals a 10.3-point macro gap. The diversity half of the ranking is therefore not primarily protecting against new source syntax; it protects against a change in the relationship between fault mechanism and informative probe structure. This conclusion is limited to the chosen structural features. Five-round bins, probe cases, and template tokens are natural for the case study but not universal. Other operational programs may require topology, permission, API, or concurrency features. 6.3 Layer Contributions Static contracts alone detect 117 effective transformations but miss 1,662 output-only faults. Behavioral validation alone admits seventeen of twenty baseline programs, yet the conservative downstream rule admits only two. Conversely, statistical task measurements cannot reject imports, unknown actions, hidden state, or order dependence. The layers answer different questions: ⢠contracts establish admissible code and output structure; ⢠compact probes establish behavioral evidence under a cost budget; and â˘task measurements establish whether a valid program should be deployed at a declared risk operating point. 9 6.4 Outcome Cache The sparse cache is 27 MiB for 2,160 transformations and 4,120,200 executed programâprobe pairs. Recomputing every budget curve, 200 random rankings per environment, whole-family holdouts, and the failure atlas requires no program or database execution. Each aggregate point still maps to concrete killing probe IDs, avoiding a learned surrogate whose predictions would themselves require validation. 7 Limitations Program domain. The study covers twenty short deterministic Python policies and four environments. Programs that call tools, manipulate schemas, coordinate distributed components, or use concurrency need stronger isolation and different structural features. Mutation representativeness. Controlled transformations measure sensitivity, not real- world fault prevalence. The thirteen families may favor the grammar that motivated them. Cross-program splitting prevents direct source memorization, and whole-family holdout reduces operator leakage, but neither replaces a corpus of real incidents. Finite-domain equivalence. The 181 excluded fault candidates are equivalent only on the complete audit domain, not necessarily for every possible input. Likewise, the 200 behavior- preserving controls cover comments, whitespace, dead branches, identity transforms, and input copies rather than all semantics-preserving transformations. State and ordering. Cached subset coverage is exact for frozen full-domain executions. It may not match a different execution order for a hidden-state program. The method rejects explicit state and tests repeatability; this is not a formal purity proof. Operational statistics. The downstream study uses five calibration and five held-out seeds per group. Bootstrap bounds cannot correct deployment distribution shift, and the 5% severe- regression threshold is an operational definition rather than a theorem. Stronger distributional guarantees require additional assumptions, as in conformal risk control [10]. Security. AST restrictions and resource limits reduce accidental damage but do not safely execute adversarial code. Untrusted deployment requires OS isolation, credential separation, syscall controls, and security review. 8 Related Work Property and behavioral testing. QuickCheck popularized generated property tests [11]; metamorphic testing derives relations when individual outputs lack complete oracles [6]; CheckList organizes behavioral capabilities for model evaluation [12]. FaultLens uses a finite, scenario- derived grammar and learns an ordering over its probes. Mutation testing. Mutation analysis evaluates whether a suite exposes controlled faults [5]. Mutation-driven generation can also construct tests and oracles [13]. Our full-domain executions establish a sparse kill relation; program and operator holdouts evaluate whether a compact subset transfers beyond the transformations used to prioritize it. Regression-test prioritization. Prior work orders regression tests to reveal faults earlier [8,9]. The active component is a deterministic set-cover heuristic. The diversity component addresses correlated mutation evidence by reserving budget for structural exploration. Operational learning and risk. Learned database components often use fallback mech- anisms because single harmful decisions can dominate average benefit [14,15]. Tail latency is especially important in operational systems [16]. Bootstrap intervals quantify sampling uncer- tainty [17]; they do not provide distribution-free deployment guarantees. FaultLens keeps compact behavioral evidence separate from the downstream performance rule. 10 9 Reproducibility The evaluation artifact records stable probe IDs, transformation parameters, static outcomes, sparse killing-probe lists, program hashes, deterministic random seeds, bootstrap labels, and every plotted CSV row. The complete source run executes 4,120,200 programâprobe pairs. Cached replay reconstructs the probe-budget curves and whole-family holdouts without executing programs or the case-study system. The manuscript figures are vector PDFs and the present arXiv source has no paths outside its upload directory. Reproduction should distinguish two targets. A full run reconstructs executed outcomes from frozen program sources. A replay run treats those outcomes as immutable evidence and reconstructs selection experiments. This separation makes it possible to audit prioritization logic without trusting a synthetic performance model. 10 Conclusion Generated operational programs need more evidence than a parser and a few examples, but exhaustive validation is not always the right default. FaultLens learns compact behavioral test suites from executed historical evidence while reserving part of the budget for mutation- independent structural diversity. In the case study, 32 probes retain 99.0% fault coverage across future programs, and diversity improves whole-fault-family macro coverage from 84.6% to 94.9%. The remaining misses are as important as the gains: compact validation is a prioritized evidence mechanism, not a proof of correctness. Its value is that the budget, evidence source, generalization split, and deployment operating point are all explicit and auditable. A Per-Environment Budget Curves Figure 6 expands the weighted curve into four environments. The hybrid reaches 100% by 16 probes in Composite, Cycle, and Rare. Burst requires a longer prefix and accounts for all six cross-program misses at budget 32. Reporting the weighted aggregate alone would conceal this heterogeneity. B Policy-Level Admission Landscape Figure 7 shows all twenty downstream decisions. P denotes the free generated program and S the restricted reference. The free programs in Composite and Cycle often exhibit extremely large calibration upper bounds and held-out tail ratios, whereas several Burst programs improve tail latency. The aggregate result therefore does not arise from a uniform reject-all policy. References [1]Guangting Zheng, Haojing Chen, Hao Li, Jingtao Zhang, Zhen Yang, Xiaosong Jia, Xue Yang, Shaofeng Zhang, and Yanyong Zhang. Enhancing video physical consistency via role-aware joint training and modality-decoupled denoising. arXiv preprint arXiv:2607.04653, 2026. URLhttps: //arxiv.org/abs/2607.04653. [2]Shanwen Tan, Hao Li, Jingtao Zhang, Xiaosong Jia, Xue Yang, Shaofeng Zhang, and Yanyong Zhang. SWIFT: Prompt-adaptive memory for efficient interactive long video generation. arXiv preprint arXiv:2605.09442, 2026. URL https://arxiv.org/abs/2605.09442. [3] Jingtao Zhang, Haorui Gao, Youqing Liang, and Zeming Liu. Scale-separated conditioning for style-encoder-free diffusion stylization. arXiv preprint arXiv:2608.19719, 2026. URLhttps://arxiv. org/abs/2608.19719. 11 0 20 40 60 80 100 Fault coverage (%) Burst ActiveHybridRandom Composite 2 0 2 2 2 4 2 6 2 8 Probe budget 0 20 40 60 80 100 Fault coverage (%) Cycle 2 0 2 2 2 4 2 6 2 8 Probe budget Rare Budget curves on held-out policy generations Figure 6: Cross-program probe-budget curves for each evaluation environment. [4] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. [5]Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. Mutation testing advances: An analysis and survey. IEEE Transactions on Software Engineering, 45(3): 345â378, 2019. [6]Tsong Yueh Chen, Shing Chi Cheung, and Shiu Ming Yiu. Metamorphic testing: A new approach for generating next test cases. Technical Report HKUST-CS98-01, Hong Kong University of Science and Technology, 1998. [7]VĂĄclav ChvĂĄtal. A greedy heuristic for the set-covering problem. Mathematics of Operations Research, 4(3):233â235, 1979. doi: 10.1287/moor.4.3.233. [8] Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. Test case prioritization: An empirical study. In Proceedings of the IEEE International Conference on Software Maintenance, pages 179â188, 1999. doi: 10.1109/ICSM.1999.792604. [9]Shin Yoo and Mark Harman. Regression testing minimization, selection and prioritization: A survey. Software Testing, Verification and Reliability, 22(2):67â120, 2012. doi: 10.1002/stvr.430. [10]Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In International Conference on Learning Representations, 2024. [11]Koen Claessen and John Hughes. Quickcheck: A lightweight tool for random testing of haskell programs. In Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming, pages 268â279, 2000. doi: 10.1145/351240.351266. [12] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902â4912, 2020. doi: 10.18653/v1/2020.acl-main. 442. 12 Contract Safe choice Balanced choice Cal. p95 UCB Held-out free/ref P = free program; S = structured fallback T1 ¡ burst T1 ¡ composite T1 ¡ cycle T1 ¡ rare T2 ¡ burst T2 ¡ composite T2 ¡ cycle T2 ¡ rare T3 ¡ burst T3 ¡ composite T3 ¡ cycle T3 ¡ rare T4 ¡ burst T4 ¡ composite T4 ¡ cycle T4 ¡ rare T5 ¡ burst T5 ¡ composite T5 ¡ cycle T5 ¡ rare passSS-0.000.96Ă failSS+40.8738.07Ă passSS+3.364.39Ă passSP+0.291.27Ă passSS+0.030.96Ă failSS+39.4540.50Ă passSS+3.324.40Ă passSP+0.291.29Ă passPP-0.000.97Ă passSS+39.9441.48Ă passSS+3.414.37Ă passSS+0.341.30Ă passSS+0.051.00Ă failSS+39.8040.56Ă passSS+3.754.41Ă passSP+0.271.27Ă passPP+0.050.98Ă passSS+38.6639.17Ă passSS+3.424.42Ă passSP+0.271.27Ă Policy-level admission landscape Figure 7: Program-level downstream admission evidence. [13]Gordon Fraser and Andreas Zeller. Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering, 38(2):278â292, 2012. doi: 10.1109/TSE.2011.93. [14]Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. Bao: Making learned query optimization practical. In Proceedings of the 2021 International Conference on Management of Data, pages 1275â1288, 2021. doi: 10.1145/3448016.3452838. [15] Dana Van Aken, Andrew Pavlo, Geoffrey J. Gordon, and Bohan Zhang. Automatic database management system tuning through large-scale machine learning. In Proceedings of the 2017 ACM International Conference on Management of Data, pages 1009â1024, 2017. doi: 10.1145/3035918. 3064029. [16]Jeffrey Dean and Luiz AndrĂŠ Barroso. The tail at scale. Communications of the ACM, 56(2):74â80, 2013. doi: 10.1145/2408776.2408794. [17]Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall/CRC, 1994. 13