Paper deep dive
Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation
Geert Trooskens, Aaron Karlsberg, Anmol Sharma, Lamara De Brouwer, Max Van Puyvelde, Matthew Young, John Thickstun, Gil Alterovitz, Walter A. De Brouwer
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 4/10/2026, 3:02:38 AM
Summary
Compiled AI is a paradigm for enterprise workflow automation that replaces repeated LLM invocations with a one-time compilation phase. By generating and validating static code artifacts from business logic, the system achieves deterministic execution, significant cost reduction (up to 57x fewer tokens), and improved auditability, making it particularly suitable for high-stakes healthcare environments.
Entities (5)
Relation Signals (3)
Compiled AI → evaluatedon → BFCL
confidence 100% · We evaluate on two task types: function-calling (BFCL, n=400)
Compiled AI → utilizes → Multi-stage validation pipeline
confidence 100% · every artifact passes a four-stage validation pipeline before deployment
Compiled AI → reduces → Token Consumption
confidence 95% · reducing token consumption by 57x at 1,000 transactions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which workflows execute deterministically without further model invocation. This paradigm has antecedents in prior work on declarative pipeline optimization (DSPy) and hybrid neural-symbolic planning (LLM+P); our contribution is a systems-oriented study of its application to high-stakes enterprise workflows, with particular emphasis on healthcare settings where reliability and auditability are critical. By constraining generation to narrow business-logic functions embedded in validated templates, compiled AI trades runtime flexibility for predictability, auditability, cost efficiency, and reduced security exposure. We introduce (i) a system architecture for constrained LLM-based code generation, (ii) a four-stage generation-and-validation pipeline that converts probabilistic model output into production-ready code artifacts, and (iii) an evaluation framework measuring operational metrics including token amortization, determinism, reliability, security, and cost. We evaluate on two task types: function-calling (BFCL, n=400) and document intelligence (DocILE, n=5,680 invoices). On function-calling, compiled AI achieves 96% task completion with zero execution tokens, breaking even with runtime inference at approximately 17 transactions and reducing token consumption by 57x at 1,000 transactions. On document intelligence, our Code Factory variant matches Direct LLM on key field extraction (KILE: 80.0%) while achieving the highest line item recognition accuracy (LIR: 80.4%). Security evaluation across 135 test cases demonstrates 96.7% accuracy on prompt injection detection and 87.5% on static code safety analysis with zero false positives.
Tags
Links
- Source: https://arxiv.org/abs/2604.05150v1
- Canonical: https://arxiv.org/abs/2604.05150v1
Trouble viewing inline? Open PDF directly →
Full Text
39,039 characters extracted from source content.
Expand or collapse full text
Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation Geert Trooskens1 Aaron Karlsberg1 Anmol Sharma1 Lamara De Brouwer1 Max Van Puyvelde2 Matthew Young1 John Thickstun3 Gil Alterovitz4 Walter A. De Brouwer2 1XY.AI Labs, Palo Alto, CA 2Stanford University School of Medicine, Stanford, CA 3Cornell University, Department of Computer Science, Ithaca, NY 4Brigham and Women’s Hospital / Harvard Medical School, Boston, MA Correspondence to: Geert Trooskens <geert@xy.ai> (April 2026) Abstract We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which workflows execute deterministically without further model invocation. This paradigm has antecedents in prior work on declarative pipeline optimization (DSPy) and hybrid neural-symbolic planning (LLM+P); our contribution is a systems-oriented study of its application to high-stakes enterprise workflows, with particular emphasis on healthcare settings where reliability and auditability are critical. By constraining generation to narrow business-logic functions embedded in validated templates, compiled AI trades runtime flexibility for predictability, auditability, cost efficiency, and reduced security exposure. We introduce (i) a system architecture for constrained LLM-based code generation, (i) a four-stage generation-and-validation pipeline that converts probabilistic model output into production-ready code artifacts, and (i) an evaluation framework measuring operational metrics including token amortization, determinism, reliability, security, and cost. We evaluate on two task types: function-calling (BFCL, n=400) and document intelligence (DocILE, n=5,680 invoices). On function-calling, compiled AI achieves 96% task completion with zero execution tokens, breaking even with runtime inference at approximately 17 transactions and reducing token consumption by 57× at 1,000 transactions. On document intelligence, our Code Factory variant matches Direct LLM on key field extraction (KILE: 80.0%) while achieving the highest line item recognition accuracy (LIR: 80.4%). Security evaluation across 135 test cases demonstrates 96.7% accuracy on prompt injection detection and 87.5% on static code safety analysis with zero false positives. 1 Introduction Large language models (LLMs) are increasingly deployed to automate enterprise workflows, evolving from question-answering systems toward autonomous agent architectures (Wu et al., 2024; CrewAI, 2024; LangChain, 2022). While these approaches demonstrate flexibility, they rely on repeated model invocation during execution, leading to high token consumption, variable latency, and non-deterministic behavior. The idea of using LLMs as compilers rather than interpreters is not new. DSPy (Khattab et al., 2023) compiles declarative LLM calls into optimized pipelines, achieving 25–65% improvement over prompt engineering. LLM+P (Liu et al., 2023) translates natural language to PDDL, then uses classical planners to produce optimal solutions—the LLM handles intent understanding, the deterministic solver handles execution. Text-to-SQL systems instantiate the same pattern: an LLM generates a query once, and the database executes it deterministically at scale. Our contribution is to study this paradigm as a first-class production systems concern, evaluating it with operational metrics suited to enterprise deployment, and demonstrating its particular value in healthcare settings where reliability and auditability are regulatory requirements. Empirical studies reveal persistent reliability challenges in runtime agent systems. Cemri et al. (2025) found that 79% of multi-agent failures stem from specification and coordination issues rather than infrastructure. Salesforce’s CRMArena-Pro benchmark shows agent success rates degrading from 58% in single-turn to 35% in multi-turn interactions (Salesforce, 2025). Non-determinism persists even at temperature=0: Atil et al. (2024) found accuracy varies up to 15% across runs, and Ouyang et al. (2023) demonstrated 18–75% output variance due to architectural factors including Mixture-of-Experts routing. These limitations are especially problematic in healthcare, where prior authorization workflows, clinical document processing, and billing automation require determinism, auditability, and compliance with HIPAA and CMS decision timeframes. This paper investigates compiled AI as a design point optimized for well-specified, high-volume, compliance-sensitive workflows. The key observation is that many enterprise workflows require intelligence to design but not to execute repeatedly. Once business logic is specified and validated, repeated execution can be handled by conventional software infrastructure—with full observability enabling targeted refinement of compiled artifacts over time. Rather than interpreting natural language at runtime, an LLM is constrained to generate narrow business-logic functions within pre-validated templates. These artifacts pass a multi-stage validation pipeline (security analysis, syntactic verification, execution testing, output accuracy) before deployment as static code with predictable latency, zero stochasticity, and near-zero marginal inference cost. Definition (Compiled AI). A workflow system satisfying three properties: 1. One-time LLM invocation — the model runs once at generation time, not at transaction time. 2. Zero-token deterministic execution — deployed workflows run as static code with no further model calls. 3. Mandatory multi-stage validation — every artifact passes security, syntax, execution, and accuracy checks before deployment. System Runtime LLM? Deterministic? Validation required? Agents (AutoGen, LangChain) Yes No No DSPy Partial Partial No Compiled AI No Yes Yes Total TokensTransactions (n)200K400K600K800K1.0MAutoGen (805/tx)LangChain (740/tx)Direct LLM (552/tx)Compiled (9.6K)9.6Kgenn∗=17n^*\!=\!17Break-even57–84× savingsat n=10001005001000 Figure 1: Token consumption comparison across baselines (BFCL, n=400). Compiled AI incurs a one-time 9,600-token generation cost, then executes at zero marginal cost. Break-even vs. Direct LLM at n∗≈17n^*≈ 17; at 1,000 transactions, compiled AI reduces consumption by 57× vs. Direct LLM and 84× vs. AutoGen. Contributions. This paper makes three contributions: (1) a system architecture for constrained LLM-based code generation producing bounded business-logic functions in durable workflow templates (Section 3); (2) a four-stage generation-and-validation pipeline converting probabilistic model output into production-ready code artifacts (Section 3); and (3) an evaluation framework measuring operational viability—token amortization, determinism, reliability, validation effectiveness, and cost—evaluated across two distinct task types (Sections 4–6). 2 Related Work 2.1 The Compilation Paradigm The most directly relevant prior work validates compilation as a productive paradigm for LLM systems. DSPy (Khattab et al., 2023) compiles declarative LLM calls into optimized pipelines, achieving 25–65% improvement over prompt engineering and enabling small 770M-parameter models to compete with GPT-3.5 on several tasks. LLM+P (Liu et al., 2023) translates natural language to PDDL and delegates execution to classical planners; LLMs alone fail to produce even feasible plans, but the hybrid approach succeeds. WorkflowLLM (Fan et al., 2024) demonstrates one-shot workflow learning with zero-shot transfer to unseen APIs. FlowMind (Zeng et al., 2024) generates workflows that prevent runtime hallucinations via structured intermediate representations. AlphaCodium (Ridnik et al., 2024) shows that structured multi-stage flows with verification improve GPT-4 accuracy from 19% to 44%. MetaGPT’s Standard Operating Procedures (Hong et al., 2024) are essentially compiled workflows: deterministic procedures reducing cascading hallucinations by constraining agent behavior. Our work differs from these efforts in two ways. First, we treat production viability as a first-class concern—not just accuracy improvement, but operational properties (token economics, determinism, auditability, cost at scale) that determine enterprise deployability. Second, we study compilation specifically in high-stakes domains where reliability is a regulatory requirement, demonstrating both where the approach excels (structured function-calling) and where it must adapt (noisy document extraction via bounded agentic invocation). 2.2 The Reliability Crisis in LLM Agents A substantial body of empirical work motivates moving the LLM out of the execution loop. Cemri et al. (2025) analyzed 1,600+ annotated traces across seven major frameworks, identifying specification failures (41.8%) and inter-agent coordination failures (36.9%) as the dominant failure modes—79% of failures are not infrastructure problems. Pan et al. (2025) surveyed 306 practitioners across 26 domains and found 68% limit agents to ≤ 10 autonomous steps before human intervention, with reliability cited as the primary deployment barrier. Critically, non-determinism persists even at temperature=0 (Ouyang et al., 2023; Atil et al., 2024), undermining reproducibility in interpreted AI approaches. Our core architectural choice—removing the LLM from the execution loop and confining it to a one-time compilation phase—directly addresses these failure modes. 2.3 Constrained Generation and Formal Verification Structured generation techniques offer complementary paths to reliability. SynCode (Ugare et al., 2024) achieves 96% reduction in syntax errors through grammar-constrained decoding; type-constrained decoding (Mündler et al., 2025) reduces compilation errors by >>50% while improving functional correctness; XGrammar (Dong et al., 2024) delivers up to 100× speedup over prior constrained decoding approaches. Formal verification shows promise for critical systems: Astrogator (Councilman et al., 2025) verifies correct code in 83% of cases. However, Xu et al. (2024) demonstrate that structural hallucinations cannot be eliminated by larger training sets alone, positioning constrained decoding as complementary rather than complete. Our approach combines constrained generation with multi-stage validation. For healthcare specifically, Neupane et al. (2025) present a HIPAA-compliant agentic AI framework integrating attribute-based access control with PHI sanitization—an approach our template-based architecture can encode as compliance-by-construction. 2.4 Benchmark Limitations and the Case for Operational Metrics The gap between benchmark performance and production viability is well-documented and motivates our evaluation framework. HumanEval is saturated (>>95% for frontier models) and contaminated (Yang et al., 2023; Xu et al., 2024); top model performance on contamination-resistant SWE-Bench Pro collapses to approximately 23%. The synthetic-to-real gap is severe: models achieving 84–89% on synthetic benchmarks attain only 25–34% on real-world class-level tasks (Rao et al., 2025). Perhaps most striking, the METR developer productivity study found developers perceived 20% productivity gains while objective measurements showed 19% slower task completion (METR, 2025). These findings motivate our framework’s emphasis on token amortization, execution determinism, and validation convergence over generation accuracy alone. 3 System Architecture The compiled AI architecture is motivated by a fundamental asymmetry in enterprise workflow automation: generating correct business logic benefits from LLM reasoning, but executing that logic thousands of times per day does not. Runtime agent systems conflate these two phases, invoking an LLM at every transaction. Our architecture separates them, confining LLM invocation to a one-time compilation step and deploying validated, static code for all subsequent execution. This separation yields several benefits. Deterministic execution eliminates output variance and enables full auditability under this architecture—every decision traces to a specific line of code, a property essential for CMS compliance in prior authorization and for FDA audit readiness in AI-enabled medical workflows (FDA, 2025). Static code can be subjected to established software engineering tooling: static analysis, type checking, security scanning, and regression testing. And because the LLM exits the execution loop, the attack surface for prompt injection shrinks—from one LLM call per transaction to one call per workflow type compiled. Design Principles. Four principles shape the architecture. Constrained Generation limits LLM output to narrow, well-defined functions (20–50 lines); templates provide infrastructure, bounding the output space and reducing hallucination risk. Compilation over Interpretation means generated code is validated, tested, and deployed as static artifacts—the LLM exits the execution loop entirely. Validation as Requirement means every artifact passes a four-stage pipeline before deployment, feasible precisely because we generate code rather than interpret configurations. Compliance by Construction encodes regulatory constraints (HIPAA, PCI-DSS, SOC 2) directly in templates and prompt blocks, ensuring generated code inherits compliance properties by default. Component Overview. The system takes a YAML workflow specification as input and produces a validated Temporal activity as output. An Orchestrator receives the specification and selects appropriate templates, modules, and compliance constraints. A Template Library provides tested code patterns for common workflow types: synchronous handlers, streaming processors, batch processors with checkpointing, and input validators with fallback logic. A Module Library provides reusable functional capabilities including database access with connection pooling, HTTP clients with retry logic, and notification delivery. Prompt Blocks encode domain constraints such as HIPAA data handling rules. The orchestrator assembles these components into a prompt, invokes the LLM once to generate business logic, and passes the assembled artifact through a four-stage validation pipeline before deployment. On validation failure, the system regenerates with error context rather than deploying a broken artifact. Figure 2 illustrates this flow; Algorithm 1 formalizes it. YAMLSpecOrch.TemplatesModulesPromptsLLMGenerateAssembleSecuritySyntaxExecuteAccuracyValidation PipelineTemporalActivityRegenerate on failureIntentDeterministicOne-time Figure 2: The code foundry architecture. Business intent (YAML) enters; validated Temporal activities emerge. Dashed red arrows show the regeneration loop on validation failure. The LLM runs once at generation time, not at transaction time. Algorithm 1 Code Foundry Generation Process 1:Workflow specification S, Template library T, Module library M 2:Validated Temporal activity A 3:t←SelectTemplate(S,T)t← SelectTemplate(S,T) 4:m←SelectModules(S,M)m← SelectModules(S,M) 5:p←AssemblePrompt(S,t,m)p← AssemblePrompt(S,t,m) 6:code←LLMGenerate(p)code← LLMGenerate(p) ⊳ One-time generation 7:A←Assemble(t,m,code)A← Assemble(t,m,code) 8:for stage∈Security,Syntax,Execution,Accuracystage∈\Security,Syntax,Execution,Accuracy\ do 9: result←Validate(A,stage)result← Validate(A,stage) 10: if result=FAILresult=FAIL then 11: code←Regenerate(p,result.errors)code← Regenerate(p,result.errors) 12: A←Assemble(t,m,code)A← Assemble(t,m,code) 13: end if 14:end for 15:return A ⊳ Deterministic artifact Validation Pipeline. The four stages serve distinct purposes. Security applies static analysis (Bandit, Semgrep, custom rules) checking for SQL injection, command injection, path traversal, and secrets exposure. Syntax performs AST parsing, type checking (mypy), and linting (ruff). Execution runs sandboxed execution against test fixtures verifying successful completion, error handling, and output structure. Accuracy compares outputs against golden datasets using task-specific thresholds. Bounded Agentic Invocation and Security Architecture. Some workflow steps require runtime judgment that cannot be fully precompiled. Our architecture supports bounded agentic invocation: generated code may call LLMs for specific, narrow subtasks (e.g., extracting structured fields from an ambiguous clinical note) while maintaining deterministic overall flow, operating under defined schemas, fallback logic, drift monitoring, and human escalation thresholds. Beyond correctness validation, a three-gate security pipeline protects against OWASP LLM Top 10 vulnerabilities: an Input Gate (DeBERTa-v3 prompt injection detection + Presidio PII scanning), a Code Gate (static analysis for CWE-94, CWE-89, CWE-502, CWE-78, CWE-328), and an Output Gate (cryptographic canary token injection for prompt leakage detection). A Dual LLM pattern separates privileged and quarantined model instances. Compiled AI reduces runtime LLM invocations from one-per-transaction to zero, effectively eliminating prompt-based attack surface in the execution path. 4 Evaluation Framework Existing benchmarks primarily measure task-level capability—code correctness or tool-use success rates. While necessary, these metrics are insufficient for evaluating production workflow automation in domains like healthcare, where economic efficiency, determinism, and reliability are more critical than expressiveness. The gap between benchmark accuracy and production viability is empirically well-documented: systems achieving 95% benchmark accuracy may succeed in only 35% of multi-turn production interactions (Salesforce, 2025), and perceived productivity gains can diverge sharply from objective measurements (METR, 2025). We therefore evaluate compiled AI as a systems artifact rather than a reasoning agent, using metrics across seven categories: token efficiency, latency, consistency and determinism, reliability, code quality, validation pipeline effectiveness, and security. The token efficiency analysis centers on the break-even transaction count: n∗=GenTokenscompiledRuntimeTokensper-tx, runtime−RuntimeTokensper-tx, compiledn^*= GenTokens_compiledRuntimeTokens_per-tx, runtime-RuntimeTokens_per-tx, compiled (1) For determinism, we measure output entropy H=−∑pilogpiH=-Σ p_i p_i over N=1,000 identical inputs; compiled AI targets H=0H=0, contrasting with the 18–75% output variance documented in runtime inference even at temperature=0 (Ouyang et al., 2023). Reliability targets far exceed the 35% agent baseline from Salesforce (2025). Security metrics evaluate the three-gate architecture against OWASP LLM Top 10 threats. Validation pipeline metrics—FirstPassRate per stage and OverallFirstPass—are novel; to our knowledge no prior work reports these for LLM-generated production code. Full metric tables for all seven categories are provided as supplementary material. 5 Experiments We evaluate on two distinct task types to demonstrate that compiled AI generalizes beyond a single benchmark and that the architecture is not overfit to a particular input modality. Task 1: Function Calling (BFCL). The Berkeley Function-Calling Leaderboard (Patil et al., 2024) measures function-calling capability on 400 instances where the model must identify the correct function and extract parameters from natural language queries. This task validates the core compilation-vs-runtime trade-off in a structured setting with clean inputs. Task 2: Document Intelligence (DocILE). To stress-test the architecture on noisy, semantic inputs, we evaluate on DocILE—5,680 invoice documents with degraded OCR quality, heterogeneous formats across hundreds of vendors, and semantic ambiguity. DocILE uses two evaluation metrics: KILE (Key Information Line-level Extraction), measuring exact-match accuracy on key fields such as invoice number, date, and total amount; and LIR (Line Item Recognition), measuring accuracy on structured line items within documents. This task also motivates the Code Factory variant: when pure deterministic approaches fail due to semantic complexity, compiled activities wrapping focused LLM calls provide a principled middle ground. Baselines. We compare four approaches across both tasks: Direct LLM (Claude Opus 4.5, one call per transaction), LangChain and AutoGen (same LLM with orchestration overhead), and Compiled AI / Code Factory (LLM invoked once at compilation; execution is deterministic code). For DocILE, we additionally evaluate a Deterministic variant (pure regex, no LLM) to bound the non-learning approach. All experiments use Claude Opus 4.5 (claude-opus-4-5-20251101) at temperature 0, PydanticAI for structured orchestration, and a custom DSL executor for workflow definitions. 6 Results 6.1 Task 1: Function Calling (BFCL) Token efficiency and cost. Compiled AI incurs a one-time 9,600-token generation cost and zero execution tokens thereafter (Table 1). Break-even against direct LLM occurs at n∗≈17n^*≈ 17 transactions. At 1,000 transactions, compiled AI uses 57× fewer tokens than direct LLM and 84× fewer than AutoGen. At 1M transactions/month, total cost of ownership is $555 vs. $22,000 for direct LLM—a 40× cost ratio (Table 2). Table 1: Token consumption comparison (BFCL, n=400). Method GenTokens Runtime/tx Total@1K BreakEven CompRatio Direct LLM 0 552 552,000 — — LangChain 0 740 740,000 — 0.75× AutoGen 0 805 805,000 — 0.69× Compiled AI 9,600 0 9,600 ∼ 17 tx ∼ 57× Table 2: Total cost of ownership at 1M transactions/month (Claude Opus 4.5: $15/1M input, $75/1M output tokens). Method Inference Infra Total TCO CostRatio Direct LLM $21,500 $500 $22,000 40× LangChain $28,900 $500 $29,400 53× AutoGen $31,400 $500 $31,900 57× Compiled AI $55 $500 $555 1× Latency and consistency. Compiled AI executes at 4.5ms P50 (vs. 2,004ms for direct LLM, a 450× improvement) with near-zero jitter (10.5ms vs. 1,123ms). It empirically achieves 100% reproducibility and zero output entropy; runtime inference shows 95% reproducibility due to model variance at temperature=0, consistent with Atil et al. (2024). Reliability and validation. Task completion is 96% (384/400). All 16 failures are compilation-time failures—once successfully compiled, workflows execute with 100% reliability. Validation pipeline first-pass rates are 100% through syntax and execution stages; 96% at the accuracy gate reflects cases where generated code executed correctly but did not match expected outputs. These failures are detectable before deployment, unlike runtime inference errors. This compile-time detection is meaningful: without the accuracy validation gate, the 4% of semantically-incorrect workflows would deploy as artifacts with 100% execution success but wrong outputs—silent failures invisible to operators. Multi-stage validation converts these into detectable compile-time failures, achieving 100% deployed-workflow reliability at the cost of a 4% compilation failure rate (simulated ablation). Security. The prompt injection validator achieves 95.8% recall with 100% precision (zero false positives) across 30 adversarial inputs. The code safety gate achieves 75% recall on 20 vulnerable code fixtures with 100% precision (zero false positives on 20 benign workflows). Output gate canary detection is 12.5% recall in simulation-based testing, reflecting methodology rather than production performance. Overall precision across 135 security test cases is 96.1%. Output gate canary recall (12.5%) reflects the simulation methodology: canary tokens trigger only when the prompt leaks into the LLM’s generation context, a condition that requires specific attacker access patterns not fully represented in our synthetic test set; production red-teaming against real attack corpora would provide a more complete picture. Code quality. Compiled artifacts achieve 99% type coverage and 96% test pass rate. Cyclomatic complexity averages 23.8 vs. 8 for human code, a known characteristic of LLM-generated switch-case structures (GitClear, 2025) and an area for template refinement. 6.2 Task 2: Document Intelligence (DocILE) Table 3 presents results on DocILE. Pure deterministic extraction (regex) runs 4,915× faster than direct LLM but achieves only 20.3% KILE—an 80 percentage point gap driven by failures on semantic fields: customer names (10.1% KILE) and currency amounts (9.2% KILE), where contextual understanding is required. Code Factory bridges this gap by compiling task-specific LLM calls as code artifacts rather than constructing prompts per-request. It achieves competitive KILE accuracy (80.0%, matching direct LLM) and the highest LIR accuracy (80.4%), while running at 2.3× lower latency than direct LLM. Combined with Pydantic output validation and automatic retry on schema violations, Code Factory demonstrates that compiled code wrapping focused LLM calls achieves parity with hand-tuned prompt engineering while enabling systematic software engineering benefits: versioning, testing, and full auditability. Table 3: DocILE results. KILE: Key Information Line-level Extraction (exact-match on key fields, e.g. invoice number, date, total). LIR: Line Item Recognition (accuracy on structured line items). Code Factory makes compiled LLM calls; Deterministic uses pure regex. Paradigm Approach KILE LIR Latency LLM Calls Compiled AI Deterministic (Regex) 20.3% 59.7% 0.6 ms None Code Factory 80.0% 80.4% 2,695 ms† Compiled Runtime AI Direct LLM 80.0% 74.5% 6,339 ms Per-request LangChain 80.0% 75.6% 6,207 ms Per-request AutoGen 77.8% 78.9% 13,742 ms Per-request †Median latency; mean 10,263 ms due to 10% retry outliers (max 183s). First compile: 57s. Together, the two task types span the spectrum from fully structured to highly noisy semantic inputs. BFCL represents the structured end: inputs are well-typed function signatures with unambiguous correct outputs, making pure deterministic compilation feasible. DocILE represents the semantic end: invoice documents vary in layout, language, and field placement, requiring contextual understanding that a regex-based approach cannot provide. That compiled AI generalizes across both extremes—achieving competitive accuracy on function-calling and matching Direct LLM on document intelligence—suggests the architecture is not overfit to a single input modality. The Code Factory variant provides a principled path for the semantic end of this spectrum: bounded LLM invocation as compiled code rather than ad-hoc runtime prompting. 7 Discussion Compiled AI is not a general replacement for runtime inference. Our results suggest a practical decision rule: use compiled extraction for structured, predictable inputs where determinism matters; use runtime LLM for noisy, open-ended content where semantic understanding is important; and for mixed-content production systems, a hybrid approach—compiled extraction with confidence-based LLM fallback—can reduce latency and cost while maintaining near-LLM accuracy. Healthcare administrative workflows exemplify the regime where compiled AI excels. Prior authorization, billing reconciliation, and clinical document processing are high-volume, well-specified by regulation, compliance-sensitive, latency-sensitive, and demand full auditability. Qiu et al. (2024) identify administrative workflow automation as a candidate application for LLM-based systems, and regulatory frameworks increasingly emphasize auditability (FDA, 2025). Our prior authorization example (Appendix A) shows how compiled AI encodes insurance coverage rules as deterministic code, confining LLM invocation to the narrow step of extracting structured clinical factors from unstructured notes. Constraining generation improves reliability by bounding the error space: the model cannot hallucinate incorrect API calls or database schemas since these come from pre-tested templates. Deterministic execution also yields full observability—when accuracy degrades, we can pinpoint which code segment underperformed and trigger targeted regeneration. This positions compiled AI not as a one-shot process but as an evolutionary system: deterministic execution with adaptive improvement, where production metrics continuously inform artifact refinement. There is also an economic timing argument. LLM inference costs have fallen roughly 10× per year, but enterprise deployment volumes are growing faster still—driving total inference spend upward even as per-token costs decline (OpenAI, 2026). Architectures that amortize inference across transactions, rather than invoking a model per request, become economically compelling at scale. Compiled AI is one such architecture: generation cost is fixed at compile time, and execution cost is zero regardless of transaction volume. As LLM costs come to dominate enterprise operating budgets, compile-once-run-many approaches are not merely convenient—they are likely to become the default deployment pattern for well-specified, high-volume workflows. 8 Limitations Several limitations warrant acknowledgment. Specification quality: compiled AI assumes users can accurately specify workflows in YAML; the “specification problem” remains fundamental and iterative refinement is often necessary. Bounded applicability: not all workflows reduce to deterministic code, and tasks requiring genuine creativity or adaptation to novel situations may require runtime inference; our evaluation covers two task types, and broader generalization requires further study. Generation failures: 4% of compilations (16/400 on BFCL) failed to produce accurate outputs, though all were syntactically valid and detectable before deployment. Model dependence: generated code quality depends on the underlying LLM, and model updates may require re-validation. Benchmark scope: our evaluation framework has not yet been adopted by other systems, limiting cross-system comparability. 9 Conclusion We studied compiled AI as a production systems design point for LLM-based workflow automation, with emphasis on the healthcare domain where reliability and auditability are regulatory requirements. Building on the compilation paradigm established in DSPy, LLM+P, and related work, we characterized compiled AI’s operational properties across two task types, showed that it generalizes through the Code Factory variant to semantic tasks requiring bounded LLM invocation, and demonstrated consistent advantages in token economics (57× reduction at 1,000 transactions), latency (450× improvement on function-calling), determinism (100% reproducibility vs. 95% for runtime inference), and cost at scale (40–57× TCO reduction at 1M transactions/month). The approach trades flexibility for determinism, cost efficiency, auditability, and reduced attack surface—properties that matter most in well-specified, high-volume, compliance-sensitive workflow regimes. We release our evaluation framework and benchmark suite to enable systematic comparison across the community. Future Work. Key open directions include natural language specification (moving beyond YAML), automatic workflow decomposition from high-level intent, continuous optimization through runtime accuracy metrics enabling targeted regeneration cycles, formal verification of generated code properties, and cross-system benchmark adoption. References Anthropic [2025] Anthropic. Introducing Claude Opus 4.5. Technical report, November 2025. https://w.anthropic.com/news/claude-opus-4-5 Atil et al. [2024] Nurullah Atil, Pedro Henrique Luz de Araujo, and Benjamin Roth. Non-Determinism of “Deterministic” LLM Settings. arXiv:2408.04667, August 2024. Breunig [2025] Drew Breunig. Does the Bitter Lesson Have Limits? https://dbreunig.com, August 2025. Cemri et al. [2025] Mert Cemri et al. Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657, March 2025. Chakrabarti [2025] Debojyoti Chakrabarti. The Bitter Lesson is Misunderstood. https://obviouslywrong.org, August 2025. Chen et al. [2023] Wenhu Chen et al. Program of Thoughts Prompting. TMLR, November 2023. Councilman et al. [2025] Craig Councilman et al. Astrogator: Towards Formal Verification of LLM-Generated Code. arXiv:2507.13290, July 2025. CrewAI [2024] João Moura. CrewAI. https://github.com/crewAIInc/crewAI, 2024. Dalrymple et al. [2024] David Dalrymple et al. Towards Guaranteed Safe AI. arXiv:2405.06624, May 2024. Dong et al. [2024] Yixin Dong et al. XGrammar. arXiv:2411.15100, November 2024. Fan et al. [2024] Shuheng Fan et al. WorkflowLLM. arXiv:2411.05451, November 2024. FDA [2025] U.S. Food and Drug Administration. AI/ML-Enabled Medical Devices. https://w.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices, 2025. Gao et al. [2023] Luyu Gao et al. PAL: Program-aided Language Models. In ICML, 2023. GitClear [2025] GitClear. AI Copilot Code Quality: 2025 Data. Technical report, February 2025. https://w.gitclear.com/ai_assistant_code_quality_2025_research Hong et al. [2024] Sirui Hong et al. MetaGPT. In ICLR, 2024. Jimenez et al. [2024] Carlos E. Jimenez et al. SWE-bench. In ICLR (Oral), 2024. Karpathy [2017] Andrej Karpathy. Software 2.0. Medium, November 2017. Karpathy [2026] Andrej Karpathy. Spec-driven development. X (Twitter), January 2026. https://x.com/karpathy/status/1883601522500329783 Khattab et al. [2023] Omar Khattab et al. DSPy. arXiv:2310.03714, October 2023. LangChain [2022] Harrison Chase. LangChain. https://github.com/langchain-ai/langchain, October 2022. Li et al. [2024] Chengshu Li et al. Chain of Code. In ICML (Oral), 2024. Liu et al. [2023] Bo Liu et al. LLM+P. arXiv:2304.11477, April 2023. METR [2025] METR. Measuring the Impact of Early-2025 AI Models on Developer Productivity. Technical report, 2025. Mündler et al. [2025] Niels Mündler et al. Type-Constrained Code Generation. In PLDI, 2025. OpenAI [2026] OpenAI (2026). API Pricing. Retrieved 2026. https://openai.com/api/pricing Neupane et al. [2025] Adarsh Neupane et al. Towards a HIPAA Compliant Agentic AI System. arXiv:2504.17669, April 2025. Ouyang et al. [2023] Shuyin Ouyang et al. Non-determinism of ChatGPT in Code Generation. arXiv:2308.02828, August 2023. Pan et al. [2025] Alex Pan et al. Measuring Agents in Production. arXiv:2512.04123, December 2025. Patil et al. [2024] Shishir G. Patil et al. Gorilla / BFCL. In NeurIPS, 2024. Qiu et al. [2024] Shijie Qiu et al. LLM-based agentic systems in medicine. Nature Machine Intelligence, December 2024. Rao et al. [2025] Shubham Rao et al. Beyond Synthetic Benchmarks. arXiv:2510.26130, October 2025. Ridnik et al. [2024] Tal Ridnik et al. AlphaCodium. arXiv:2401.08500, January 2024. Rozière et al. [2023] Baptiste Rozière et al. Code Llama. arXiv:2308.12950, August 2023. Salesforce [2025] Kung-Hsiang Huang et al. CRMArena-Pro. Salesforce AI Research, May 2025. Shi et al. [2025] Yizhou Shi et al. The SWE-Bench Illusion. arXiv:2506.12286, June 2025. Sutton [2019] Richard S. Sutton. The Bitter Lesson. http://w.incompleteideas.net/IncIdeas/BitterLesson.html, March 2019. Temporal [2024] Temporal Technologies. Temporal Documentation. https://docs.temporal.io, 2024. Ugare et al. [2024] Shubham Ugare et al. SynCode. arXiv:2403.01632, March 2024. Veracode [2025] Veracode. 2025 GenAI Code Security Report. Technical report, September 2025. Wu et al. [2024] Qingyun Wu et al. AutoGen. In COLM, 2024. Xu et al. [2024] Yucheng Xu et al. Benchmark Data Contamination Survey. arXiv:2406.04244, June 2024. Xu et al. [2024] Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is Inevitable. arXiv:2401.11817, 2024. Yang et al. [2023] Yichen Yang et al. Rethinking Benchmark and Contamination. arXiv:2311.04850, November 2023. Zeng et al. [2024] Jin Zeng et al. FlowMind. In ACM ICAIF, 2024. Zhou et al. [2024] Andy Zhou et al. Language Agent Tree Search. In ICML, 2024. McCabe [1976] Thomas J. McCabe. A Complexity Measure. IEEE TSE, SE-2(4):308–320, December 1976. Appendix A Example: Prior Authorization Workflow This example demonstrates bounded agentic invocation for healthcare prior authorization, illustrating compiled AI in the healthcare regime discussed in Section 7: the LLM is confined to structured data extraction, while coverage decisions execute as deterministic, fully auditable code. Listing 1: Workflow specification for GLP-1 prior authorization ⬇ # workflow_spec: prior_auth_glp1_v1.yaml metadata: name: "GLP-1␣Agonist␣Medical␣Necessity␣Review" compliance: ["HIPAA", "CMS_Decision_Timeframes"] inputs: - name: "patient_chart_summary" type: "text" logic_requirements: - step: "extract_clinical_factors" type: "bounded_invocation" schema: has_t2d_diagnosis: boolean current_a1c: float bmi: float has_step_therapy_failure: boolean validation: - "bmi␣>␣10␣AND␣bmi␣<␣100" - step: "evaluate_coverage" type: "deterministic_rule" logic: | IF has_t2d_diagnosis THEN APPROVE ELSE IF (bmi >= 30) AND has_step_therapy_failure THEN APPROVE ELSE DENY Listing 2: Compiled artifact isolating probabilistic extraction from deterministic decisions ⬇ @workflow.defn class PriorAuthWorkflow: @workflow.run async def run(self, input_data: WorkflowInput) -> AuthResult: # PHASE 1: BOUNDED AGENTIC INVOCATION clinical_data = await workflow.execute_activity( "extract_clinical_data_activity", args=[input_data.chart_text], retry_policy=RetryPolicy(maximum_attempts=3) ) # PHASE 2: HALLUCINATION VALIDATION if clinical_data.a1c and not (3.0 < clinical_data.a1c < 20.0): return await self.trigger_human_review("Suspicious␣A1C") # PHASE 3: DETERMINISTIC DECISION decision, reason = "DENIED", "Does␣not␣meet␣criteria" if clinical_data.has_t2d_diagnosis: decision, reason = "APPROVED", "Type␣2␣Diabetes␣Diagnosis" elif (clinical_data.bmi >= 30.0 and clinical_data.has_step_therapy_failure): decision, reason = "APPROVED", "BMI>=30␣+␣Step␣Therapy" # PHASE 4: AUDIT TRAIL await workflow.execute_activity("log_decision_event", "decision": decision, "logic_version": "v1.0") return AuthResult(status=decision, reason_code=reason) This example demonstrates three properties: (1) Safety Sandwich—the probabilistic model is constrained between input validation and deterministic logic [Dalrymple et al., 2024]; (2) Auditability—decisions trace to specific code lines, enabling full auditability under this architecture; (3) Token Economics—coverage rules are compiled once rather than sent per-request. Appendix B Benchmark Suite We release our benchmark suite at https://github.com/XY-Corp/CompiledAI, including 540 YAML workflow specifications, golden outputs for accuracy validation, measurement scripts for all seven metric categories, and baseline implementations.