Paper deep dive
The Patchwork Problem in LLM-Generated Code
Viraaji Mothukuri, Reza M. Parizi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/18/2026, 3:17:24 PM
Summary
This paper introduces the 'patchwork problem,' where LLM-generated code is locally valid but globally incoherent, leading to structural failures in deployed systems. The authors propose a formal taxonomy of eight structural failure categories (e.g., Symbol Resolution Failures, Dependency Hallucination) and a hybrid verification framework that uses graph representations of repository artifacts to detect these invariants. Empirical results show that standard tools like type checkers and SAST miss the majority of these failures, which are prevalent in real-world AI-generated code.
Entities (24)
Relation Signals (17)
Reza M. Parizi → authored → The Patchwork Problem in LLM-Generated Code
confidence 95% · The Patchwork Problem in LLM-Generated Code Viraaji Mothukuri, Reza M. Parizi
Viraaji Mothukuri → authored → The Patchwork Problem in LLM-Generated Code
confidence 95% · The Patchwork Problem in LLM-Generated Code Viraaji Mothukuri, Reza M. Parizi
Patchwork Problem → causes → Structural Failures
confidence 95% · The root cause is frequently structural rather than logical. A generated endpoint references configuration keys never declared in the project...
LLM-Generated Code → evadesdetectionby → Type Checking
confidence 92% · Empirical evaluation across two frontier models under four prompting strategies reveals that the vast majority of structural failures evade type checking, testing, and SAST entirely
LLM-Generated Code → evadesdetectionby → Static Analysis Tools
confidence 92% · Empirical evaluation across two frontier models under four prompting strategies reveals that the vast majority of structural failures evade type checking, testing, and SAST entirely
Reza M. Parizi → affiliatedwith → Kennesaw State University
confidence 90% · Decentralized Science Lab, College of Computing and Software Engineering Kennesaw State University, GA, USA
Viraaji Mothukuri → affiliatedwith → Kennesaw State University
confidence 90% · Decentralized Science Lab, College of Computing and Software Engineering Kennesaw State University, GA, USA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-generated code often compiles, passes tests, and appears correct, yet breaks once deployed. The root cause is frequently structural rather than logical. A generated endpoint references configuration keys never declared in the project, an import targets a package that does not exist in any registry, or a new route omits the authentication guard applied to every sibling endpoint. Each patch is locally valid but globally incoherent, and standard CI toolchains rarely surface these failures. As LLM-powered coding tools see widespread adoption, this blind spot poses a growing risk to software quality. We call this the \textbf{patchwork problem}. This paper formalizes structural coherence as consistency invariants over graph representations of repository artifacts, including import, call, dependency, configuration, schema, resource, control-flow, and routing graphs, and introduces an eight-category failure taxonomy distinguishing defects specific to LLM generation from those merely amplified by it. We present a hybrid verification framework that delegates to mature static analysis tools where they already excel and deploys purpose-built detectors for cross-cutting invariants underserved by existing toolchains, targeting provable constraint violations rather than heuristic pattern matching. Empirical evaluation across two frontier models under four prompting strategies reveals that the vast majority of structural failures evade type checking, testing, and SAST entirely, and that failure patterns diverge qualitatively between models in ways that challenge model-agnostic mitigation strategies. External validation on real-world AI-generated repositories confirms that these failures are not artifacts of controlled experimentation but are prevalent wherever LLMs write code with minimal human oversight.
Tags
Links
- Source: https://arxiv.org/abs/2607.08981v1
- Canonical: https://arxiv.org/abs/2607.08981v1
Trouble viewing inline? Open PDF directly →
Full Text
49,070 characters extracted from source content.
Expand or collapse full text
The Patchwork Problem in LLM-Generated Code Viraaji Mothukuri, Reza M. Parizi Decentralized Science Lab, College of Computing and Software Engineering Kennesaw State University, GA, USA vmothuku@students.kennesaw.edu rparizi1@kennesaw.edu Abstract LLM-generated code often compiles, passes tests, and appears correct, yet breaks once deployed. The root cause is frequently structural rather than logical. A generated endpoint references configuration keys never declared in the project, an import targets a package that does not exist in any registry, or a new route omits the authentication guard applied to every sibling endpoint. Each patch is locally valid but globally incoherent, and standard CI toolchains rarely surface these failures. As LLM-powered coding tools see widespread adoption, this blind spot poses a growing risk to software quality. We call this the patchwork problem. This paper formalizes structural coherence as consistency invariants over graph representations of repository artifacts, including import, call, dependency, configuration, schema, resource, control-flow, and routing graphs, and introduces an eight-category failure taxonomy distinguishing defects specific to LLM generation from those merely amplified by it. We present a hybrid verification framework that delegates to mature static analysis tools where they already excel and deploys purpose-built detectors for cross-cutting invariants underserved by existing toolchains, targeting provable constraint violations rather than heuristic pattern matching. Empirical evaluation across two frontier models under four prompting strategies reveals that the vast majority of structural failures evade type checking, testing, and SAST entirely, and that failure patterns diverge qualitatively between models in ways that challenge model-agnostic mitigation strategies. External validation on real-world AI-generated repositories confirms that these failures are not artifacts of controlled experimentation but are prevalent wherever LLMs write code with minimal human oversight. Keywords: LLM code generation, structural coherence, static analysis, graph invariants, code quality, neural code synthesis. 1 Introduction Code generation from Large Language Models has achieved remarkable results on isolated programming tasks [7, 4, 11, 1], driving rapid adoption, with millions of engineers using LLM-powered assistants daily. Yet a gap persists between benchmark performance and production utility [8]. Code that appears correct in isolation frequently fails when integrated into real software systems [27], and the dominant failure mode is structural rather than functional. A generated patch may compile, pass type checking, and satisfy local tests while violating invariants that span the repository. Consider a FastAPI endpoint referencing a Pydantic model with hallucinated field names, or a Django view assuming environment variables that are never declared in the project configuration. Such patches exhibit local correctness but global incoherence: they pass the checks developers rely on and fail only when exercised in the context of the full system. Current evaluation methodologies do not surface these failures systematically. Type checking and linting often miss semantic inconsistencies that cross file boundaries. Test suites cannot cover every integration point. SAST tools typically focus on taint flows rather than structural coherence. The result is a blind spot in which generated code enters codebases carrying latent defects that remain invisible to standard toolchains. We term this the patchwork problem. LLM-generated patches may be individually well-formed yet fail to cohere into a consistent whole, particularly at repository scale, where consistency constraints span imports, dependencies, configurations, schemas, and security contracts [12]. Our approach formalizes structural coherence as consistency invariants over graph representations of repository artifacts [25]. A key design insight is that reliable detection requires matching each invariant class to an appropriate verification strategy. Categories where mature static analysis tools already capture the relevant language semantics can be delegated to those tools, while categories that require cross-graph reasoning, absent from existing toolchains, call for purpose-built detectors that target constraint violations under explicit assumptions and produce actionable evidence. This work makes three contributions: (1) We introduce a taxonomy of eight structural failure categories, each defined by graph-based consistency invariants, distinguishing failures characteristic of LLM-generated patches from issues merely amplified by them. (2) We present a multi-graph verification framework that integrates mature static analysis tools (mypy, tsc, pylint, ESLint) with purpose-built cross-graph detectors over eight repository graphs, producing localized evidence traces for each violation. (3) We provide an empirical study across 336 generations from two frontier models under four prompting conditions, along with external validation on 43 real-world AI-generated repositories. 2 Related Work Table 1 situates our contribution relative to prior work across four research threads. Hallucination characterization work [27, 20] establishes the empirical prevalence of structural defects in generated code but characterizes them descriptively rather than as verifiable constraint violations. Repository-level benchmarks, including RepoBench [12], SWE-bench [8], EvoCodeBench [9], BaxBench [22], SecRepoBench [19], SecureVibeBench [3], and SWE-agent [26] demonstrate that snippet-level performance does not transfer reliably to repository-scale tasks, yet their evaluation criteria remain outcome-oriented (test passage, exploit success) rather than diagnosing which structural invariants are violated. Graph-based representations such as Code Property Graphs [25], CODE-MVP [23], and GALLa [28] leverage graphs as representational substrates but do not operationalize them as a constraint verification layer. Secure generation approaches, including CodeGuard+ [6] and SafeGenBench [10], target vulnerability prevention without formalizing structural coherence. Our work addresses this gap by defining structural incoherence as violated consistency constraints across graph representations and producing localized evidence traces that attribute failures to specific constraint violations. Table 1: Summary of related work Work Focus Gap [27] Taxonomy of hallucinations in repo-level generation Descriptive; no formal invariants or automated detection [20] Quantitative measurement of fabricated package references Dependency-only; no cross-artifact verification [12] Cross-file code completion with retrieval subtasks Evaluates completion, not structural coherence [8] GitHub issue resolution via test passage Test-based; misses failures that evade tests [9] Repo-aligned generation with dependency annotations Pass@k metric; no structural invariant checking [22] Backend generation with exploit-based security assessment Exploit-focused; no config or schema verification [19] Security-focused patch evaluation Vulnerability labels, not structural constraints [3] Multi-file vulnerability scenarios Security-scoped; no general structural coverage [26] Autonomous multi-step repo editing Evaluates resolution rate, not edit coherence [25] Unified code property graph for vulnerability discovery Single-file; no cross-file or config constraints [23] Multi-view contrastive pre-training Graphs as training signal, not verification layer [28] Graph-aligned fine-tuning for structural semantics Alignment target, not constraint checking [6] Constrained decoding for secure generation Decoding-time; no post-hoc structural verification [10] Dual-judge vulnerability detection Vulnerability-scoped; no structural coherence 3 Structural Failure Taxonomy The patchwork problem manifests through structural failures, defined as violations of consistency invariants at the repository scale, verifiable via static graph analysis without execution. These failures differ from functional bugs in that individual patches appear correct yet collectively violate project contracts. Our taxonomy comprises eight categories, each defined by formal invariants, required graph artifacts, and failure characteristics specific to LLM outputs. Table 2 summarizes the classification. Symbol Resolution Failures (SRF): A symbol resolution failure occurs when a referenced name cannot be resolved within the repository’s module graph. Formally, for symbol reference r in file f with module graph ℳM, the invariant requires ∀r∈refs(f):∃d s.t. resolve(r,f,ℳ)→d∀ r (f):∃ d s.t. resolve(r,f,M)→ d. Detection requires the import graph and symbol table, optionally augmented with type information for generic resolution. Evidence traces record file, line, symbol, expected module, and resolution outcome. This failure class is amplified in LLM outputs because models operating with incomplete context frequently invent plausible but non-existent module names or reference deprecated APIs. Phantom Internal API (PIA): Phantom API failures occur when generated code invokes internal functions with incorrect signatures or semantics inconsistent with declared interfaces. The invariant requires signature compatibility such that for call site c invoking symbol s with signature registry Σ , compatible(sig(c),Σ(s))=truecompatible(sig(c), (s))=true. Detection leverages the call graph and signature registry extracted from type annotations and protocol declarations. This category is strongly amplified as LLMs hallucinate method signatures based on naming conventions rather than actual declarations, particularly for internal APIs underrepresented in training data. Dependency Hallucination (DHI): Algorithm 2 validates that all external imports reference packages declared in the project’s dependency manifests. External imports not found in the dependency graph trigger registry queries to PyPI or npm, distinguishing fully hallucinated packages (nonexistent in registries) from undeclared-but-existing packages. For npm packages, import names match registry names directly. For PyPI, the detector assumes direct correspondence between import names and package names, which holds for the majority of packages but not for cases where import and distribution names diverge (e.g., yaml vs. PyYAML, cv2 vs. opencv-python). The resulting phantom module set feeds downstream into SRF and PIA detection. Build/Configuration Incoherence (BCI): Build configuration failures arise when generated code assumes configurations inconsistent with the repository’s declared state. The invariant requires that for each configuration assumption a implied by generated code, a satisfying declaration exists in the configuration space C. Detection operates over the build graph and configuration graph encompassing entrypoints, environment variables, and framework settings. Four invariant classes apply across languages, namely entrypoint existence, environment variable declaration, module system consistency, and framework configuration alignment. This category is amplified under weak retrieval context where models default to standard configurations divergent from project settings. Resource Coherence Failures (RCF): Resource coherence failures occur when code fails to provide declared resources, encompassing both filesystem resources and computational contracts. Filesystem resource failures arise when code references files, assets, templates, or migrations that do not exist, with the existence invariant requiring exists(resolve_path(r,root))=trueexists(resolve\_path(r,root))=true for each resource reference r and additional ordering constraints for sequential resources such as database migrations. Return contract failures arise when functions with declared return types contain execution paths that do not produce a value matching the declared type, violating the contract that the function’s signature promises to callers. Schema completeness failures arise when model definitions omit fields required by consuming code. Detection operates over the resource graph mapping code references to filesystem paths, the CFG for return-path reachability analysis, and the schema graph for field completeness validation. LLMs exhibit amplified failure rates across all three sub-categories by generating plausible paths based on conventions rather than actual repository structure, omitting return statements on error-handling branches, and producing incomplete schema definitions. Control Flow Coherence (CFC): Control flow failures manifest as CFG anomalies including unreachable blocks, contradictory conditions, exception flow misuse, and dead error handling. Invariants require full reachability from entry (∀v:reachable(v0,v)∀ v:reachable(v_0,v)) and exception handler type consistency. Detection operates on intraprocedural CFGs with exception edge annotations. While unreachable code is a general bug class, LLMs exhibit amplified characteristic patterns such as overbroad exception handling, redundant null checks, and copy-paste control flow inconsistencies. Cross-File Contract Violations (CCV): Contract violations occur when producer and consumer modules exhibit interface mismatches across file boundaries, including wrong field names, incompatible serialization, incorrect error codes, and misaligned assumptions. The invariant requires schema compatibility whereby schema(P.output)⊇schema(C.input)schema(P.output) (C.input) with type consistency. Detection requires the call graph with edges spanning files and the schema graph extracted from OpenAPI specs, Pydantic models, or Zod schemas. LLMs amplify this failure by hallucinating response fields based on naming conventions. Figure 1: Overview of the Proposed Framework Security Structural Regressions (SSR): Security structural regressions occur when application wiring violates security contracts absent classic taint flow vulnerabilities. The invariant requires guard coverage such that ∀r∈R:guarded_by(r,M)∀ r∈ R:guarded\_by(r,M) for security-critical routes R and required guards M. Detection requires routing and middleware attachment graphs specific to each framework. We scope detection to FastAPI (dependency injection guards), Django (permission decorators), Express (middleware chains), and Next.js (middleware matchers). LLMs amplify this failure by wiring middleware incorrectly, even when producing syntactically correct code. Table 2: Structural failure taxonomy ID Category Primary Graph(s) LLM Profile Type check evasion Test Evasion SRF Symbol Resolution Import + Symbol Table Amplified Partial Partial PIA Phantom Internal API Call + Signature Strongly Amplified Yes Yes DHI Dependency Hallucination Dependency + Registry Specific to LLMs Yes Yes BCI Build/Config Incoherence Build + Config Amplified Yes Yes RCF Resource Coherence Resource + CFG + Schema Amplified Yes Yes CFC Control Flow Coherence CFG General/Amplified Partial Partial CCV Cross-File Contracts Call + Schema Amplified Yes Yes SSR Security Structural Routing + Middleware Amplified Yes Yes 4 Verification Framework Architecture Overview: The verification framework employs a hybrid architecture guided by a precision-first design philosophy in which each taxonomy category is matched to the verification strategy that maximizes detection precision for its invariant class. Categories where mature static analysis tools already handle the relevant language semantics (symbol resolution, signature compatibility) delegate to those tools, inheriting their years of edge-case handling. Control flow coherence employs a hybrid approach combining custom graph reachability and pattern matching with SAST tool delegation, reflecting the observation that no single layer achieves adequate coverage alone. Categories requiring cross-graph reasoning absent from existing toolchains (configuration incoherence, dependency hallucination, security regressions, resource coherence, cross-file contracts) employ purpose-built detectors that target provable constraint violations rather than heuristic pattern matching. This division reflects the empirical observation that reimplementing established analyses from scratch produces low-precision detectors due to the long tail of language-specific edge cases (exception flows, generators, context managers, async patterns), while the novel cross-cutting invariants central to the patchwork problem have no existing tool coverage. The framework operates on each repository state after generation, constructing graph representations from the combined original and generated code and routing each to the appropriate verification backend. The framework’s output for each finding is a localized evidence trace recording the violated invariant, the implicated files and line numbers, and the constraint that would need to hold for the code to be structurally sound. All source code, graph construction scripts, detection pipelines, evaluation configurations, and external validation datasets required to reproduce our results are publicly available [13]. Figure 1 illustrates the end-to-end verification pipeline, showing how graph construction connects prompting conditions to failure detection. Graph Construction: For each repository state after generation, the framework constructs eight graph representations spanning structural, behavioral, and configurational dimensions. Table 3 summarizes the construction method for each graph type. Table 3: Graph construction methods by language Graph Python TypeScript Import ast module; relative import resolution ts.createProgram with tsconfig.json resolution Call pycg flow-insensitive points-to analysis Compiler API type-directed resolution Dependency pyproject.toml, requirements.txt, poetry.lock + PyPI validation package.json, package-lock.json, yarn.lock + npm validation Schema Pydantic BaseModel, SQLAlchemy Column Zod z.object(), Prisma schema.prisma Config .env, .env.example, Docker Compose, framework settings Same sources Resource open(), pathlib.Path, template loaders, migration deps Equivalent TS patterns CFG ast branch analysis; pylint delegation ts-morph branch analysis; ESLint delegation Routing FastAPI route decorators, Django URL conf Express router chains, Next.js middleware matchers Figure 2: Graph-to-category mapping Figure 2 visualizes the many-to-many mapping between graph representations and failure categories, illustrating why the framework requires multiple coordinated analyses rather than a single monolithic pass. Chords connect each graph representation (left) to the failure categories it enables detecting (right). Some categories require multiple graphs, and some graphs serve multiple categories, motivating the hybrid architecture. Detection Algorithms: The following paragraphs formalize detection for each taxonomy category. Purpose-built detectors (Algorithms 1–3, 4, 5, and 7) target provable constraint violations, while control flow coherence (Algorithm 6) combines custom analysis with SAST tool delegation. Detection order reflects data dependencies, with DHI running first to produce the phantom module set consumed by SRF and PIA. Configuration Incoherence Detection (BCI): Algorithm 1 detects provable runtime failures from unguarded environment variable accesses. It extracts strict access patterns that throw on missing values (os.environ["KEY"] in Python, unguarded process.env.KEY in TypeScript), eliminates accesses protected by guards (try/except, membership tests, fallback operators), and validates remaining accesses against the repository’s configuration space. Arithmetic expressions and safe access patterns with explicit defaults are excluded at extraction time. Every reported finding represents a configuration access that will produce a runtime crash if the variable is absent. Algorithm 1 Configuration Incoherence Detection (BCI) 0: Repository R with generated code G, config files C 0: Set of validated configuration incoherence findings F 1: Extract unsafe accesses 2: A←∅A← 3: for each file f in G do 4: if f is Python then 5: A←A∪os.environ["K"]A← A∪\ os.environ["K"] patterns in ff\ 6: else if f is TypeScript then 7: A←A∪A← A∪\unguarded process.env.K in ff\ 8: end if 9: end for 10: Guard elimination 11: for each access a∈Aa∈ A do 12: if a inside try/except KeyError or preceded by membership test or has ||/?? fallback then 13: A←A∖aA← A \a\ 14: end if 15: end for 16: Config-space validation 17: ←C← keys from .env, .env.example, docker-compose.yml, settings 18: F←∅F← 19: for each access a∈Aa∈ A with key k do 20: if k∉k then 21: F←F∪(a,k,file,line)F← F∪\(a,k,file,line)\ 22: end if 23: end for 24: return F Dependency Hallucination Detection (DHI): Algorithm 2 validates that all external imports reference packages declared in the project’s dependency manifests. External imports not found in the dependency graph trigger registry queries to PyPI or npm, distinguishing fully hallucinated packages (nonexistent in registries) from undeclared-but-existing packages. The resulting phantom module set feeds downstream into SRF and PIA detection. Algorithm 2 Dependency Hallucination Detection (DHI) 0: Repository R with manifests M, generated code G 0: Set of dependency hallucination findings F 1: D←D← parse packages from M (pyproject.toml, requirements.txt, package.json) 2: Extract and validate external imports 3: F←∅F← 4: for each import i in G referencing package p do 5: if p∉p∉ stdlib and p∉p∉ local modules and p∉Dp∉ D then 6: Query registry ℛR (PyPI/npm) for p 7: if p∉ℛp then 8: F←F∪(i,p,“hallucinated”)F← F∪\(i,p,``hallucinated′)\ 9: else 10: F←F∪(i,p,“undeclared”)F← F∪\(i,p,``undeclared′)\ 11: end if 12: end if 13: end for 14: return F Symbol Resolution and Phantom API Detection (SRF, PIA): Algorithm 3 leverages the phantom module set from Algorithm 2. A LocalModulePattern filter excludes intra-generation cross-references and intentional placeholders to prevent false positives from multi-file generation tasks. Algorithm 3 Symbol Resolution (SRF) and Phantom API (PIA) Detection 0: Import graph ℐI, call graph K, phantom set P from Algorithm 2 0: Sets of SRF findings FSF_S and PIA findings FPF_P 1: FS←∅F_S← , FP←∅F_P← 2: for each import edge (f,m,s)(f,m,s) in ℐI do 3: if (m∈Pm∈ P or ¬resolve(s,m) (s,m)) and m∉m∉ LocalModulePattern then 4: FS←FS∪(f,m,s)F_S← F_S∪\(f,m,s)\ 5: end if 6: end for 7: for each call edge (f,m.s,args)(f,m.s,args) in K do 8: if m∈Pm∈ P and m∉m∉ LocalModulePattern then 9: FP←FP∪(f,m,s,args)F_P← F_P∪\(f,m,s,args)\ 10: end if 11: end for 12: return FS,FPF_S,F_P Resource Coherence Detection (RCF): Algorithm 4 targets three sub-categories. Return contract violations construct intraprocedural CFGs for functions with declared return types and identify execution paths that terminate without producing a value, excluding branches guarded by exception handlers that re-raise or call sys.exit. Filesystem resource violations check that referenced paths (templates, migrations, assets) resolve to existing files. Schema completeness violations flag consuming code that accesses fields absent from the producing model’s definition. Every finding represents a provable violation: a reachable path missing a declared return, a path literal that does not resolve, or a field access targeting an undefined name. Algorithm 4 Resource Coherence Detection (RCF) 0: CFGs ℱF, resource graph ℛR, schema graph S, generated code G 0: Set of resource coherence findings F 1: F←∅F← 2: for each function f in G with declared return type T do 3: Build intraprocedural CFG; for each path π to exit, if π has no return of type T and is not exception-terminated, add (f,π,T)(f,π,T) to F 4: end for 5: for each resource ref r with path p in G do 6: if ¬exists(resolve(p,root)) (resolve(p,root)) then 7: F←F∪(r,p)F← F∪\(r,p)\ 8: end if 9: end for 10: for each consumer access c.fieldc.field produced by model M do 11: if field∉fields(M)field (M) then 12: F←F∪(c,M,field)F← F∪\(c,M,field)\ 13: end if 14: end for 15: return F Cross-File Contract Violation Detection (CCV): Algorithm 5 detects interface mismatches across module boundaries via call graph and schema graph analysis. It targets four patterns: disconnected middleware (registered but never imported in route modules), unused decorators referencing nonexistent permission classes, duplicate middleware registrations causing double execution, and field naming mismatches between producer response schemas and consumer access patterns (e.g., user_name vs. username). Each finding identifies a statically verifiable disconnect between two code locations that must agree for correct execution. Algorithm 5 Cross-File Contract Violation Detection (CCV) 0: Call graph K, schema graph S, middleware config ℳM, generated code G 0: Set of contract violation findings F 1: F←∅F← 2: for each middleware m registered in ℳM do 3: if m∉m∉ import edges of any route module then 4: F←F∪(m,disconnected)F← F∪\(m,disconnected)\ 5: end if 6: if m appears >>1 time in registration then 7: F←F∪(m,duplicate)F← F∪\(m,duplicate)\ 8: end if 9: end for 10: for each cross-file call edge (producer,consumer)(producer,consumer) in K do 11: SP←S_P← output fields of producerproducer; SC←S_C← accessed fields in consumerconsumer 12: if ∃f∈SC:f∉SP∃\,f∈ S_C:f∉ S_P then 13: F←F∪(producer,consumer,SC∖SP)F← F∪\(producer,consumer,S_C S_P)\ 14: end if 15: end for 16: return F Control Flow Coherence Detection (CFC): Algorithm 6 employs a hybrid three-layer approach with findings deduplicated by line number. Layer 1 performs BFS reachability from function entry nodes, flagging only entirely dead functions. Layer 2 applies pattern matching for dead code after terminators, tautological conditions, duplicate handlers, and infinite loops. Layer 3 delegates to pylint/ESLint with post-processing filters suppressing known false positives from context managers, generators, and heavy exception scaffolding. Algorithm 6 Control Flow Coherence Detection (CFC) 0: Generated code files G, constructed CFGs ℱF 0: Set of control flow findings F 1: F←∅F← 2: for each file f in G do 3: Graph-based reachability (Layer 1) 4: for each function CFG g∈ℱ(f)g (f) do 5: R←R← BFS from entry node of g 6: if all body nodes ∉R∉ R then 7: F←F∪(f,g,“dead function”)F← F∪\(f,g,``dead function′)\ 8: end if 9: end for 10: Pattern-based detection (Layer 2) 11: F←F∪F← F∪ detect dead-code-after-terminator in f 12: F←F∪F← F∪ detect tautological conditions in f 13: F←F∪F← F∪ detect duplicate except/switch-case in f 14: F←F∪F← F∪ detect infinite loops without exit in f 15: SAST delegation (Layer 3) 16: r←r← invoke pylint/ESLint on f (skip if syntax error) 17: Remove findings in context manager, generator, or heavy try/finally contexts 18: F←F∪rF← F∪ r 19: Deduplicate F by line number (priority: graph >> pattern >> SAST) 20: end for 21: return F Security Structural Regression Detection (SSR): Algorithm 7 identifies endpoints lacking authentication guards present on sibling routes. Routes are clustered by resource segment, public endpoints are filtered, and majority-rule analysis flags routes where a dominant guard (≥ 90% coverage) is absent on destructive HTTP methods (POST, PUT, DELETE, PATCH). Algorithm 7 Security Structural Regression Detection (SSR) 0: Routing graph G with routes (path,method,guards)\(path,method,guards)\ 0: Set of security regression findings F 1: Filter and cluster 2: for each route r in G do 3: Assign r to cluster CresourceC_resource by path segment 4: end for 5: Remove clusters matching public whitelist (health, auth, docs, webhook, metrics) 6: Majority-rule guard analysis 7: F←∅F← 8: for each remaining cluster C with |C|≥4|C|≥ 4 do 9: g∗←g^*← most common guard in C 10: ratio←|r∈C:g∗∈guards(r)|/|C|ratio←|\r∈ C:g^* (r)\|/|C| 11: if ratio≥0.9ratio≥ 0.9 then 12: for each route r∈Cr∈ C where g∗∉guards(r)g^* (r) do 13: if method(r) ∈ POST, PUT, DELETE, PATCH then 14: F←F∪(r,g∗,C)F← F∪\(r,g^*,C)\ 15: end if 16: end for 17: end if 18: end for 19: return F Illustrative Example: We trace a real-world AI-generated repository through the verification pipeline to illustrate how structural failures manifest and evade standard toolchains. Illustrative Example: hypertropher-app [24] A Next.js/React web application built with AI coding tools and published on GitHub. 72 files analyzed tsc-strict ✓ SAST ✓ 11 structural failures × BCI — Configuration Incoherence 7 findings Four environment variables (NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY, SUPABASE_SECRET_API_KEY, NEXT_PUBLIC_GOOGLE_MAPS_API_KEY) accessed without defaults across three Supabase client files and the application layout. None declared in any .env or config file. Each resolves to undefined at runtime. Invisible to tsc because process.env access is structurally valid regardless of key existence. DHI — Dependency Hallucination 1 finding Import references @vercel/analytics/next, a package absent from package.json. The path alias filter correctly excludes 98 @/components/ui/* local aliases, isolating the single genuinely unresolvable external dependency. RCF — Resource Coherence 2 findings Functions loadingCities and previewUrl declare return types but contain conditional branches that never return a value. CFG reachability analysis identifies the gaps. The type checker misses these because exception flow masks the incomplete returns. CFC — Control Flow Coherence 1 finding Dead code after a return statement at line 461. This category was absent from all 336 controlled generations yet appears in real-world AI-generated code, consistent with the hypothesis that less supervised generation surfaces failure modes that controlled experiments do not elicit. A companion repository, VoiceTradeWithSchwab [2] (voice-controlled stock trading, 100% AI-generated Python), exhibits 92 findings across five categories including phantom imports, an infinite loop, and unguarded trading configuration variables. 5 Evaluation and Results 5.1 Experimental Setup We evaluate structural failure detection across 336 code generations from two frontier models, GPT-4o (2024-08-06, 128K context) and Claude 3.5 Sonnet (2024-10-22, 200K context), both at temperature zero. The evaluation corpus consists of 10 curated open-source production repositories spanning Python (Django [5], FastAPI [17]) and TypeScript (Express [15], Next.js [21]), selected for active maintenance, with a minimum 50 files and 10K LOC, type annotation coverage exceeding 50%, test coverage above 60%, and explicit schema definitions.111Please refer to [13] for the complete list of curated repositories, external validation datasets, and all evaluation metadata. From these repositories, we extract 60 tasks derived from merged pull requests and closed issues at three complexity levels, namely L1 single-file (30 tasks), L2 multi-file (20 tasks), and L3 cross-cutting (10 tasks). Four prompting strategies control context richness ranging from P1 (minimal, task description only) through P2 (local, 2–5 same-directory files) and P3 (retrieved, 10 similarity-ranked files) to P4 (oracle, 5–15 ground-truth files). The evaluation has a partially unbalanced design in which 24 early tasks were evaluated with GPT-4o under P1 and P2 only, while the remaining 36 tasks received both models across all four strategies, yielding 192 GPT-4o generations (60 each for P1/P2, 36 each for P3/P4) and 144 Claude generations (36 per strategy). P1 and P2 therefore contain 96 generations each and P3 and P4 contain 72 each; all analyses use appropriate denominators to account for this asymmetry. We compare our framework against four baselines, representing static CI checks: type checking (mypy [14] and tsc, test execution, SAST via bandit [16] and semgrep [18], and regex heuristics. Dependency installation is excluded as the evaluation operates on generated code before environment builds; it would catch at most the 3 DHI findings but none of the remaining 64. Detection metrics include per-category precision against ground truth and evasion rates quantifying findings that pass each baseline undetected. Ground truth labels were established through two approaches. For categories with small finding counts (BCI, DHI, PIA, SRF), every finding was manually reviewed and verified as a provable constraint violation. For categories with larger counts (RCF, CCV), precision was established through iterative pipeline refinement that systematically eliminated false positive patterns, with boundary cases resolved by consulting the formal invariants for each category. 5.2 Results Detection Performance: Table 4 reports detection results across 336 generations (192 GPT-4o, 144 Claude 3.5 Sonnet) under four prompting conditions. Our framework identifies 67 structural failures across eight active categories, with 65 (97.0%) invisible to all baseline methods. Compilation and type checking detect only 2 findings independently (both RCF return-type violations), while test execution, SAST, and regex heuristics detect none. By category, RCF accounts for the most findings (29), followed by CCV (18), BCI (12), DHI (3), PIA (3), and SRF (2). Manual validation confirms 100% precision for BCI, DHI, PIA, and SRF (20 of 20 verified as provable constraint violations), while RCF and CCV precision was established through iterative pipeline refinement. No baseline method detects any CCV, BCI, DHI, PIA, or SRF finding; type checkers catch only 2 of 29 RCF findings. Evasion rates reinforce this gap: 97.0% of findings evade compilation (mypy --strict/tsc --strict), and 100% evade test suites and SAST tools. Two categories, CFC and SSR, produced zero findings in the controlled evaluation despite having active detectors. To determine whether these detectors function correctly or whether controlled generation simply does not elicit these failure modes, we applied the full pipeline to 43 real-world AI-generated repositories spanning vibe-coded projects (Cursor AI, Google Gemini, GitHub Copilot), GPT-Engineer/Lovable applications, and self-declared fully AI-generated projects.222Please refer to [13] for the complete list of all repositories with per-repo metadata and finding counts. Across 1,581 analyzed files the pipeline detected 1,152 findings in 35 of 43 repositories (81.4% repo-level incidence), with 474 DHI findings, 270 RCF findings, 177 PIA findings, 148 BCI findings, 62 SRF findings, 16 CFC findings (6 duplicate switch cases, 2 dead-code-after-return, 2 infinite loops), and 5 CCV findings. The CFC findings confirm that the hybrid three-layer detector functions correctly; controlled frontier generation with explicit task specifications simply does not produce the unstructured code patterns that trigger control flow failures. SSR remained at zero across all evaluations. This is consistent with two properties of the evaluation corpus. The detector requires route clusters with at least 4 endpoints exhibiting a dominant per-route guard pattern, and most vibe-coded projects apply authentication at the framework level (e.g., global middleware, app-level decorators) rather than per-route, leaving no inconsistency to detect. The two highest-finding repositories, hypertropher-app and VoiceTradeWithSchwab (detailed in the Illustrative Example above), exemplify how structural failures cluster and compound in real-world AI-generated code. Table 4: Detection results across 336 generations Method Findings TP Precision Unique Type Check/Lint 2 2 100% 0 Test Execution 0 0 N/A 0 SAST 0 0 N/A 0 Regex Heuristics 0 0 N/A 0 Patch Work Framework (Ours) 67 67 see text 65 Table 5: Per-category detection results Category N Precision Baseline Detection Strategy RCF 29 Refined 2/29 CFG return-path + schema CCV 18 Refined 0/18 Cross-graph disconnect BCI 12 100% 0/12 Unsafe-access + config-space DHI 3 100% 0/3 Registry validation PIA 3 100% 0/3 Cross-graph phantom check SRF 2 100% 0/2 Import + symbol resolution CFC 0 N/A N/A Hybrid 3-layer SSR 0 N/A N/A Resource-clustered auth Model Comparison: Failure distributions diverge qualitatively between GPT-4o and Claude 3.5 Sonnet as visualized in Figure 3. Overall failure rates are comparable (GPT-4o: 39 findings in 25/192 generations, 13.0%; Claude: 28 findings in 17/144 generations, 11.8%), but the models exhibit distinct failure profiles rather than simply differing in magnitude. GPT-4o produces all 18 CCV findings and all import-related failures (DHI, PIA, SRF) exclusively, while Claude generates 22 of 29 RCF findings. Both contribute equally to BCI (6 each). Chi-squared testing confirms distributional independence (χ2=25.1χ^2=25.1, p=2.73×10−7p=2.73× 10^-7). With 67 total findings, these patterns are descriptive observations warranting replication rather than definitive model characterizations. Figure 3: Model divergence in structural failure categories. Bars extend left for GPT-4o and right for Claude 3.5 Sonnet. Prompt Sensitivity: Table 6 reports failure counts by prompting strategy. P3 (retrieved context) exhibits the highest count (24) and P1 (minimal) the lowest (8), indicating that richer context reshapes rather than uniformly reduces failure distributions. Notably, BCI appears in P1 through P3 but not P4 (oracle), consistent with ground-truth files helping models identify correct configuration variables. L3 (cross-cutting) tasks exhibit 44.6% finding incidence compared to 16.1% for L1 and 13.4% for L2, confirming that tasks spanning configuration, middleware, and schema boundaries are substantially more failure-prone. Figure 4 shows how failure category composition shifts across complexity levels, with BCI and CCV concentrated in L3 tasks that require cross-layer reasoning. Table 6: Findings by prompt strategy and category Category P1 P2 P3 P4 RCF 0 8 13 8 CCV 4 5 5 4 BCI 4 2 6 0 DHI 0 1 0 2 PIA 0 1 0 2 SRF 0 0 0 2 Total 8 17 24 18 Figure 4: Failure category composition by task complexity level. Runtime: The pipeline analyzes each file in a median of 47 ms end-to-end (graph construction through all seven detectors), with a 120-second timeout per external analysis script. Registry validation queries (PyPI/npm) are cached across runs. The largest repository in Track D (435 files, 70K LOC) completes in 233 seconds. Runtime is dominated by graph construction (99.8% of per-file time); all detectors combined complete in under 1ms per file. These times indicate the framework is practical as a CI integration for repositories of moderate size. 5.3 Practical Implications These findings have direct consequences for teams adopting LLM-generated code. First, standard CI pipelines (type checking, testing, SAST) are insufficient as quality gates for generated code; 97% of detected failures pass all four baselines, meaning structurally broken code can merge undetected. Teams relying solely on existing toolchains face a growing blind spot as LLM-generated code volume increases. Second, the qualitative divergence between models suggests that switching or combining models does not uniformly reduce risk; different models produce different failure profiles, and mitigation strategies should be model-aware. Third, the concentration of failures in L3 cross-cutting tasks (44.6% incidence versus 13–16% for simpler tasks) indicates that structural verification is most critical for tasks spanning configuration, routing, and schema boundaries, precisely the tasks where LLMs are increasingly deployed. The framework’s localized evidence traces are designed to support both automated CI integration and developer review workflows, providing actionable diagnostics rather than opaque pass/fail signals. 6 Conclusion This work formalized the patchwork problem in LLM-generated code through a graph-based failure taxonomy and a hybrid verification framework combining mature static analysis tools with purpose-built detectors. Across controlled generations and real-world repositories, the overwhelming majority of detected structural failures evade type checking, testing, and SAST entirely, and failure patterns diverge qualitatively between models. Our future work will focus on extending the framework to additional models and programming languages and on subjecting the categories currently validated through iterative refinement to independent precision audits. Further directions include repair mechanisms that leverage detected constraint violations to prompt for missing declarations, and integration into agentic coding workflows and continuous integration pipelines where incremental patch review replaces complete-file generation. References [1] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton (2021) Program synthesis with large language models. External Links: 2108.07732, Link Cited by: §1. [2] BSalita (2025) VoiceTradeWithSchwab. Note: https://github.com/BSalita/VoiceTradeWithSchwab Cited by: §4. [3] J. Chen, H. Huang, Y. Lyu, J. An, J. Shi, C. Yang, T. Zhang, H. Tian, Y. Li, Z. Li, X. Zhou, X. Hu, and D. Lo (2026-07) SecureVibeBench: benchmarking secure vibe coding of AI agents via reconstructing vulnerability-introducing scenarios. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 24144–24168. External Links: Link, Document, ISBN 979-8-89176-390-6 Cited by: Table 1, §2. [4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating large language models trained on code. External Links: 2107.03374, Link Cited by: §1. [5] Django Software Foundation (2024) Django: the web framework for perfectionists with deadlines. Note: https://w.djangoproject.comVersion 5.0 Cited by: §5.1. [6] Y. Fu, E. Baker, Y. Ding, and Y. Chen (2024) Constrained decoding for secure code generation. External Links: 2405.00218, Link Cited by: Table 1, §2. [7] J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim (2026-01) A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35 (2), p. 1–72. External Links: ISSN 1557-7392, Link, Document Cited by: §1. [8] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan (2024) SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1, Table 1, §2. [9] J. Li, G. Li, X. Zhang, Y. Dong, and Z. Jin (2024) EvoCodeBench: an evolving code generation benchmark aligned with real-world code repositories. External Links: 2404.00599, Link Cited by: Table 1, §2. [10] X. Li, J. Ding, C. Peng, B. Zhao, X. Gao, H. Gao, and X. Gu (2025) Safegenbench: a benchmark framework for security vulnerability detection in llm-generated code. arXiv preprint arXiv:2506.05692. Cited by: Table 1, §2. [11] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Sutherland Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals (2022-12) Competition-level code generation with alphacode. Science 378 (6624), p. 1092–1097. External Links: ISSN 1095-9203, Link, Document Cited by: §1. [12] T. Liu, C. Xu, and J. McAuley (2024) Repobench: benchmarking repository-level code auto-completion systems. In International Conference on Learning Representations, Vol. 2024, p. 47832–47850. Cited by: §1, Table 1, §2. [13] V. Mothukuri (2026) Source code of the paper: the patchwork problem in llm-generated code. Note: https://github.com/decentralizedsciencelab/PatchWork.gitSource code, evaluation pipelines, detection configurations, and datasets Cited by: §4, footnote 1, footnote 2. [14] mypy (2024) Mypy: optional static typing for Python. Note: https://mypy-lang.orgVersion 1.8 Cited by: §5.1. [15] OpenJS Foundation (2024) Express: fast, unopinionated, minimalist web framework for Node.js. Note: https://expressjs.comVersion 4.x Cited by: §5.1. [16] PyCQA (2024) Bandit: a tool designed to find common security issues in Python code. Note: https://bandit.readthedocs.ioVersion 1.7 Cited by: §5.1. [17] S. Ramírez (2024) FastAPI: modern, fast (high-performance) web framework for building APIs with Python. Note: https://fastapi.tiangolo.comVersion 0.110 Cited by: §5.1. [18] Semgrep, Inc. (2024) Semgrep: lightweight static analysis for many languages. Note: https://semgrep.devOpen-source edition Cited by: §5.1. [19] C. Shen, C. Dilgren, P. Chiniya, L. Griffith, Y. Ding, and Y. Chen (2026) SecRepoBench: benchmarking code agents for secure code completion in real-world repositories. External Links: 2504.21205, Link Cited by: Table 1, §2. [20] J. Spracklen, R. Wijewickrama, A. N. Sakib, A. Maiti, and B. Viswanath (2025) We have a package for you! a comprehensive analysis of package hallucinations by code generating \llms\. In 34th USENIX Security Symposium (USENIX Security 25), p. 3687–3706. Cited by: Table 1, §2. [21] Vercel (2024) Next.js: the React framework for the web. Note: https://nextjs.orgVersion 14 Cited by: §5.1. [22] M. Vero, N. Mündler, V. Chibotaru, V. Raychev, M. Baader, N. Jovanović, J. He, and M. Vechev (2025) BaxBench: can LLMs generate correct and secure backends?. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: Table 1, §2. [23] X. Wang, Y. Wang, Y. Wan, J. Wang, P. Zhou, L. Li, H. Wu, and J. Liu (2022) CODE-mvp: learning to represent source code from multiple views with contrastive pre-training. In Findings of the Association for Computational Linguistics: NAACL 2022, External Links: Link Cited by: Table 1, §2. [24] wowashuwow (2025) Hypertropher-app. Note: https://github.com/wowashuwow/hypertropher-app Cited by: §4. [25] F. Yamaguchi, N. Golde, D. Arp, and K. Rieck (2014) Modeling and discovering vulnerabilities with code property graphs. In 2014 IEEE symposium on security and privacy, p. 590–604. Cited by: §1, Table 1, §2. [26] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. External Links: 2405.15793, Link Cited by: Table 1, §2. [27] Z. Zhang, C. Wang, Y. Wang, E. Shi, Y. Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng (2025) Llm hallucinations in practical code generation: phenomena, mechanism, and mitigation. Proceedings of the ACM on Software Engineering 2 (ISSTA), p. 481–503. Cited by: §1, Table 1, §2. [28] Z. Zhang, H. Yu, S. Li, P. Di, J. Li, and R. Wang (2025) GALLa: graph aligned large language models for improved source code understanding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), External Links: Link Cited by: Table 1, §2.