Paper deep dive
Agent Gym: A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback
Pouya Ghiasnezhad Omran, Michael Zimmermann, Duncan Cambridge, Ashmita Kapoor, Tanya Dixit
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/22/2026, 2:33:27 AM
Summary
The paper introduces Agent Gym, a modular, domain-agnostic framework designed to address the 'static agent dilemma' in production LLM agents. It enables continuous post-deployment evaluation and evolution through a three-zone architecture: a Constitution layer for declarative domain knowledge, a Runtime Inference Pipeline for acting and investigation, and a Learning Loop for human-in-the-loop rule correction. Key technical contributions include a hybrid deterministic-LLM correction engine (ALF), a ground-truth-free three-layer investigation architecture, and a programmatic safety loop for rule validation.
Entities (8)
Relation Signals (6)
Agent Gym → contains → ALF
confidence 95% · ALF is the correction engine that sits downstream of the acting agent... part of the framework
Agent Gym → contains → Investigation Agent
confidence 95% · Zone 2 executes the runtime inference pipeline: Acting Agent→ Investigation Agent→ALF engine.
Agent Gym → supports → Human-in-the-Loop
confidence 95% · continuous evaluation and evolution of LLM agents through human-in-the-loop feedback
Agent Gym → demonstratedon → Invoice Processing
confidence 90% · An open-source reference implementation for invoice processing demonstrates that the framework is fully operational
Investigation Agent → uses → Constitution
confidence 90% · The Investigation Agent validates agent decisions against the rules book... The Constitution... serves as the system’s constitution.
ALF → utilizes → LLM
confidence 90% · hybrid deterministic-LLM correction engine... correction leverages LLMs when contextual understanding is required.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Model (LLM) agents deployed in production environments face a fundamental tension: the agent's behavior is frozen at deployment time, while the business rules and edge cases it must handle continue to evolve. Existing approaches address agent construction and one-time evaluation but provide no structured mechanism for continuous post-deployment behavioral correction without modifying the agent's source code. Most of the approaches offered in the market, require intense collection of logs and traces, and re-examining the agent design by the engineering team, a process which is heavy, long and negates the economical value of agentic transformation. We introduce Agent Gym, a modular, domain-agnostic framework that wraps any existing LLM-based agent in a continuous evaluation-and-evolution loop. The framework provides six composable capabilities --- Act, Evaluate, Investigate, Correct, Learn, and Observe --- organized across three architectural zones: a constitution layer that codifies domain knowledge in configuration artifacts, a runtime inference pipeline that chains acting, investigation, and adaptive correction, and a learning loop that enables subject matter experts to discover and validate new correction rules through natural language interaction. The key technical contributions include a hybrid deterministic-LLM correction engine with 21 condition operators and three-tier actions, a three-layer investigation architecture for ground-truth-free compliance validation, and a programmatic safety loop that guarantees rule correctness before human approval. We further introduce the Spec-to-Note Gap, an autoencoder-inspired view of agentic system transparency. An open-source reference implementation for invoice processing demonstrates that the framework is fully operational and ready for adoption.
Tags
Links
- Source: https://arxiv.org/abs/2608.15591v1
- Canonical: https://arxiv.org/abs/2608.15591v1
Trouble viewing inline? Open PDF directly →
Full Text
53,307 characters extracted from source content.
Expand or collapse full text
2026 Agent Gym: A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback Pouya Ghiasnezhad Omran 1 , Michael Zimmermann 1 , Duncan Cambridge 1 , Ashmita Kapoor 1 and Tanya Dixit 1 1 Google Cloud Large Language Model (LLM) agents deployed in production environments face a fundamental tension: the agent’s behavior is frozen at deployment time, while the business rules and edge cases it must handle continue to evolve. Existing approaches address agent construction and one-time evaluation but provide no structured mechanism for continuous post-deployment behavioral correction without modifying the agent’s source code. Most of the approaches offered in the market, require intense collection of logs and traces, and re-examining the agent design by the engineering team, a process which is heavy, long and negates the economical value of agentic transformation. We introduce Agent Gym, a modular, domain- agnostic framework that wraps any existing LLM-based agent in a continuous evaluation-and-evolution loop. The framework provides six composable capabilities — Act, Evaluate, Investigate, Correct, Learn, and Observe — organized across three architectural zones: a constitution layer that codifies domain knowledge in configuration artifacts, a runtime inference pipeline that chains acting, investigation, and adaptive correction, and a learning loop that enables subject matter experts to discover and validate new correction rules through natural language interaction. The key technical contributions include a hybrid deterministic-LLM correction engine with 21 condition operators and three-tier actions, a three-layer investigation architecture for ground-truth-free compliance validation, and a programmatic safety loop that guarantees rule correctness before human approval. We further introduce the Spec-to-Note Gap, an autoencoder-inspired view of agentic system transparency. An open-source reference implementation for invoice processing demonstrates that the framework is fully operational and ready for adoption. Keywords: LLM agents, human-in-the-loop, agent evaluation, adaptive learning, rule engine, domain- agnostic framework 1. Introduction LLM-based agents are increasingly deployed to automate complex, rule-governed business pro- cesses such as document classification, data ex- traction, compliance validation, and decision- making (Brown et al., 2020; Gemini Team, Google, 2023). Frameworks such as ReAct (Yao et al., 2022), LangChain (Chase, 2022), and Au- toGen (Wu et al., 2023) have made it straight- forward to build capable agents that achieve im- pressive initial accuracy. Yet this initial success masks a deeper structural problem that we term the static agent dilemma: the agent’s behavior is frozen at deployment time, while the business environment it operates in is not. Three categories of failure emerge in produc- tion. First, edge-case failures: the agent encoun- ters document configurations or regulatory ex- ceptions absent from its training. Second, sys- tematic misinterpretations: the agent consistently misapplies a business rule across an entire cat- egory of cases. Third, drift and rule evolution: business rules change over time, and the agent has no mechanism to absorb these changes with- out code modification. We are offering extensi- ble framework, which in the future, can uncover more learning opportunities, like discovering new skills, tools, MCP servers, etc. The conventional path from “subject matter expert (SME) identifies a problem” to “agent be- havior is corrected” passes through a software Corresponding author(s): pgomran@google.com arXiv:2608.15591v1 [cs.AI] 16 Aug 2026 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback engineering bottleneck: the SME describes the is- sue informally, an engineer interprets the descrip- tion, modifies agent code or prompts, re-deploys, and the SME verifies. This workflow is slow, lossy, error-prone (i.e. human interactions) and scales poorly (Amershi et al., 2019; Sculley et al., 2015). A further challenge lies in evaluation. Tradi- tional frameworks compare agent output against labeled ground truth (Liang et al., 2022; Liu et al., 2024). However, in production, ground truth may not exist for novel cases, and even when refer- ence data is available, it may not capture the nuanced business rules that should govern the decision. What is needed is a dual evaluation model: one that validates against reference data when available, and one that validates decisions against the business rules themselves. The real enteprise world is adverserial to agents, and the agent need to learn from example of one (and not from a well curated dataset). We introduce Agent Gym, a modular frame- work that addresses these challenges. Our contri- butions are: 1.A domain-agnostic framework for contin- uous agent evaluation and evolution that treats the acting agent as a black box, re- quiring no modification to the agent’s source code. Domain-specific understanding of the agent’s pipeline is derived entirely from declar- ative configuration artifacts (the constitution), not from custom-coded critic, rule-engine, or learning components; adopting the framework for a new domain requires changing only these artifacts, while all framework code remains unchanged. 2.A hybrid deterministic-LLM correction en- gine (ALF) with 21 condition operators for deterministic error detection and a three-tier action model for targeted correction. 3.A three-layer investigation architecture for compliance validation without ground truth, employing cached rule discovery and ultra- conservative triple-check verification. 4. A programmatic safety loop for human-in- the-loop rule discovery that guarantees new rules are validated for schema correctness, cross-case impact, and collateral elimination before SME approval. While human-in-the- loop interaction establishes baseline learning, the safety loop actively interrogates proposed rules and solicits clarifying constraints to en- sure robust generalization and prevent gover- nance drift. 5.A constitution-based governance model where domain knowledge is captured in declarative configuration artifacts that serve as both human-readable documentation and machine-parseable specifications. 6.The Spec-to-Note Gap, an autoencoder- inspired view of agentic system transparency that generalizes round-trip correctness (Alla- manis et al., 2024) to whole-system granular- ity. 7.An open-source reference implementation 1 for invoice processing that demonstrates the complete framework is operational and ready for adoption. The remainder of this paper is organized as fol- lows. Section 2 positions Agent Gym against re- lated work. Section 3 presents the framework ar- chitecture. Section 4 discusses constitution-based agent creation. Section 5 introduces the Spec- to-Note Gap. Section 6 describes the reference implementation. Section 7 discusses limitations and future directions. Section 8 concludes. 2. Related Work LLM Agent Frameworks. Recent frameworks have focused on agent construction: ReAct (Yao et al., 2022) combines reasoning and acting, Au- toGPT (Significant Gravitas, 2023) pursues full autonomy, LangChain (Chase, 2022) provides composable chains, MetaGPT (Hong et al., 2024) assigns roles to agents, and AutoGen (Wu et al., 2023) enables multi-agent conversation. These frameworks address how to build capable agents but provide no structured mechanism for post- deployment behavioral evolution. Agent Gym is complementary: it wraps any agent built with these frameworks in a continuous correction loop. 1 https://github.com/google/adk-samples/ tree/main/python/agents/invoice-processing 2 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback Agent Evaluation. Benchmarks such as HELM (Liang et al., 2022), AgentBench (Liu et al., 2024), and SWE-bench (Jimenez et al., 2024) evaluate agent capabilities on static test sets. The LLM-as-judge paradigm (Zheng et al., 2023) enables scalable evaluation without human labels. These approaches measure agent quality at a point in time but do not address continuous production monitoring or behavioral correction. Agent Gym integrates both deterministic and LLM-based evaluation as components of a broader lifecycle. Self-Improving Agents. Self-Refine (Madaan et al., 2023)enables iterative refinement through self-feedback, and Reflexion (Shinn et al., 2023) uses verbal reinforcement learning for agent self- correction. Both approaches modify the agent itself. In contrast, Agent Gym leaves the agent untouched, applying corrections in a separate downstream layer. This preserves auditability and enables independent version control of the agent and its correction rules. Constitutional AI and Alignment. Constitu- tional AI (Bai et al., 2022) uses a set of prin- ciples to guide model behavior during training. Agent Gym extends the constitutional metaphor to the operational layer: a reconstructed rules book serves as the system’s constitution, govern- ing runtime validation and correction rather than model training. Human-in-the-Loop Learning. RLHF (Chris- tiano et al., 2017; Ouyang et al., 2022) and ac- tive learning (Settles, 2009) incorporate human feedback into model training. Agent Gym op- erates at a different level: Human feedback pro- duces structured correction rules that are applied deterministically at inference time, without re- training the underlying model. This makes cor- rections immediate, editable, and reversible. In- context and rule-based learning is essential in production settings because knowledge external to model weights remains transparent, auditable, and easily editable. Furthermore, enterprise pol- icy updates and data unlearning can be enacted instantly without incurring the latency and cost of continuous fine-tuning or model retraining. Business Rule Management. Traditional busi- ness rule management systems (BRMS) such as Drools provide deterministic rule engines for en- terprise applications (Graham, 2006; Ross, 2003). Agent Gym’s ALF engine extends this paradigm with LLM-driven actions, enabling corrections that require understanding of context and seman- tics beyond what deterministic rules alone can express. Model Documentation. Model Cards (Mitchell et al., 2019) and Datasheets (Gebru et al., 2021) establish standards for documenting ML systems. Microsoft’s Transparency Notes (Microsoft Cor- poration, 2020) extend this to deployed services. The Spec-to-Note Gap concept introduced in Sec- tion 5 proposes automated generation of such documentation, connecting to round-trip correct- ness (Allamanis et al., 2024) at the system level. Table 1 summarizes the positioning of Agent Gym relative to key approaches across five dimensions critical for production agent management. 3. The Agent Gym Framework 3.1. Design Principles Agent Gym is built on four foundational princi- ples that guide every architectural decision: Principle 1: The agent is a black box. The framework makes no assumptions about the agent’s internal architecture. It observes only input-output behavior: which documents went in, which artifacts came out, and what decisions were rendered. Any agent — whether a mono- lithic pipeline, a multi-agent system, or a single LLM call — can be wrapped without modification. Principle 2: Corrections are layered, not in- vasive. Rather than modifying the agent when an error is discovered, corrections are applied in a separate downstream layer. The agent’s orig- inal output is preserved; the corrected output 3 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback Case / Invoice Acting Agent Classify Extract ValidateTransform Output Audit Investigation Agent stopok ALF Rule Base Collect Plan Execute Final Output The Constitution Source of Truth & Transparency Note Rule Learning Agent (RLA) Human Expert (SME) Update Rules Feedback Sample Review Periodic System Review Promote Frequent Patterns Zone 1: The Constitution Architecture Zone 2: The Runtime Inference Pipeline Zone 3: The Learning & Evolution Loop Data Flow Reference Line Figure 1|Three-zone architecture of Agent Gym. Zone 1 provides the constitutional foundation (rules book and master data). Zone 2 executes the runtime inference pipeline: Acting Agent→ Investigation Agent→ALF engine. Zone 3 enables human-guided evolution through SME feedback and the Rule Learning Agent. Frequent correction patterns are periodically promoted into the Acting Agent’s permanent logic. Table 1|Comparison of Agent Gym with related approaches across key dimensions for production agent management.✓ indicates full support;∼ indicates partial support; – indicates no support. Approach Post-Deploy Correction Domain Agnostic Human Governance No Agent Modification GT-Free Evaluation ReAct / LangChain / AutoGen–✓– HELM / AgentBench / SWE-bench–∼–✓– Self-Refine / Reflexion∼✓– RLHF–∼✓– Traditional BRMS✓∼✓– Agent Gym (ours)✓ sits alongside it. This preserves auditability and enables independent rollback. Principle 3: Domain knowledge lives in con- figuration, not code. All domain-specific infor- mation is captured in a declarative configura- tion layer. When the framework is adopted for a new domain, only the configuration changes; the framework’s code remains untouched. Principle 4: Humans govern the loop. Every correction rule proposed by the system must be reviewed, approved, and validated by a subject matter expert before it is added to the production rule base (ALF). The framework provides candi- date discovery; the human provides authoritative judgment. Principle 5: Governance must be tiered. To prevent malicious rule injection or unauthorized policy drift, the framework enforces strict sepa- ration of duties between runtime rule discovery and constitutional modification. While domain SMEs discover and validate case-level ALF correc- tion rules, permanent modifications to the core constitution and acting agent logic require multi- stakeholder administrative approval. 3.2. Architecture Overview Agent Gym organizes its capabilities into three architectural zones, illustrated in Figure 1: Zone 1 — Constitution Architecture. A do- main configuration layer consisting of a master 4 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback data specification (YAML) and a reconstructed rules book (Markdown) that together serve as the system’s constitution. These artifacts can be authored manually by domain experts or, in the general case, generated by a bootstrap agent — a four-stage LLM pipeline that analyzes the acting agent’s source code and sample outputs. The current reference implementation uses man- ually authored artifacts; the bootstrap pipeline is a planned automation. Access to the master data specification (YAML) and reconstructed rules book should be carefully managed to ensure no unauthorized modification as these files wield significant power in critical agents. Zone 2 — Runtime Inference Pipeline. Three sequential stages process each case: the acting agent produces initial output, the investigation agent validates compliance against the constitu- tion, and the ALF engine applies targeted correc- tions based on learned rules. Zone 3 — Learning and Evolution Loop. A conversational learning agent enables SMEs to re- view cases, identify error patterns, and discover new correction rules through a programmatic safety loop that guarantees rule correctness be- fore approval. The framework provides six composable capa- bilities that span these zones: Act (the existing agent processes documents), Evaluate (output is compared against ground truth), Investigate (decisions are validated against the rules book), Correct (known error patterns are fixed deter- ministically), Learn (SMEs discover new rules conversationally), and Observe (operational dash- boards, planned for a future release, will provide visibility). 3.3. Domain Configuration Layer The domain configuration layer makes Agent Gym domain-agnostic through two generated artifacts: Master Data is a structured YAML document that fully describes a domain across eleven sec- tions: document types, extraction schemas, tax- onomies, validation pipeline definitions, output schema, evaluation comparison groups, investi- gation file maps, configuration defaults, domain- specific detection logic, artifact naming conven- tions, and rejection templates. Every framework component reads from this single source of truth. Rules Book is a human-readable Markdown document that captures the complete business rules governing the agent’s processing logic. It de- scribes every validation step, threshold, rejection template, and decision outcome. The investiga- tion agent consumes this document as context for LLM-based validation. We think this is a major innovation area, where the agent DNA is exposed (and editable) not only for engineers, but for do- main experts (which are far from code editing and complex configuration files) In the general case, both artifacts can be gen- erated by a Bootstrap Agent, a four-stage LLM pipeline that: (1) analyzes the acting agent’s source code to extract pipeline structure, schemas, and validation logic; (2) scans sample outputs to discover artifact schemas and status values; (3) synthesizes a comprehensive rules book; and (4) produces the structured master data YAML. Alternatively, domain experts can author the arti- facts directly. The current reference implementa- tion uses manually authored artifacts; the boot- strap pipeline is a planned automation that will reduce onboarding effort for new domains. Adopting Agent Gym for a new domain re- quires providing the acting agent’s code and sam- ple outputs, along with manually authored or bootstrap-generated configuration artifacts. 3.4. Runtime Inference Pipeline 3.4.1. Acting Agent The acting agent is the system under observation. Agent Gym imposes only one structural require- ment: the agent must produce JSON artifacts as output. Beyond this, the agent’s architecture is unconstrained. The acting agent processes in- put documents and produces a folder of output artifacts per case, including intermediate results and a final structured output file containing the agent’s decision. 5 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback 3.4.2. Investigation Agent The Investigation Agent validates agent decisions against the rules book without requiring any ground truth data, making it applicable to every case, including novel ones. The agent employs a three-layer validation ar- chitecture, summarized in Table 2: Layer 1 — Deterministic Checks. At each case, the agent performs data source validation, verify- ing that the acting agent drew field values from the correct sources (extraction vs. preprocessing). Bypass detection identifies cases where the agent skipped required validation steps. These checks involve no LLM calls. Layer 2 — LLM Rule Discovery. At initializa- tion, the rules book is sent to an LLM, which dis- covers and categorizes validation rules into struc- tured groups. The discovered rules are cached with a SHA-256 hash of the rules book content, so subsequent runs skip rediscovery unless the rules book has changed. Layer 3 — Conservative Cross-Validation. An ultra-conservative layer evaluates each rule group against the case. When a potential violation is detected, a triple-check mechanism re-runs the evaluation twice more. Only violations confirmed by all three independent runs are reported. This dramatically reduces false positives. Overhead control. Four mechanisms bound the investigation agent’s LLM costs in practice. Content-hash caching ensures that Layer 2 com- putes a SHA-256 hash of the rules book content; on subsequent runs the cached rule groups are reused unless the rules book has changed, amor- tizing Layer 2’s LLM cost to zero after the first in- vocation. Section-filtered context reduces prompt size by extracting only the rules book section relevant to the current validation phase rather than sending the full document to the LLM for each call. Batch grouping consolidates determin- istic rule groups (data-source priorities, tolerance thresholds, entity whitelists, and keyword lists) into a single batched LLM call; only subjective groups such as work-type classification receive in- dividual calls. Finally, early exit from triple-check Table 2|Three-layer investigation architecture. Each layer adds progressively deeper validation with controlled LLM usage. Layer MethodLLMScope 1DeterministicNoPer-field 2Rule discoveryYes ∗ Global 3Cross-validation YesPer-group ∗ Cached by content hash; amortized to zero after first run. issues the second and third verification calls only when the first call detects a potential violation. For compliant cases, which constitute the major- ity under a well-functioning agent, Layer 3 incurs a single LLM call per rule group. Together, these mechanisms ensure that steady-state investiga- tion cost is dominated by Layer 1 deterministic checks, with LLM usage concentrated on the ini- tial run and on genuine violations. The investigation agent produces per-case compliance scores on a 0–100% scale, cat- egorized as Fully_Compliant (≥80%), Partial_Violation (60–80%), orMa- jor_Violation (<60%). A major violation halts the pipeline, preventing potentially non- compliant output from reaching the correction stage. 3.4.3. ALF — Adaptive Learning Framework ALF is the correction engine that sits downstream of the acting agent. Its design separates detection from correction: detection is fully deterministic, while correction leverages LLMs when contextual understanding is required. Detection. Each ALF rule specifies conditions evaluated against the agent’s output using a li- brary of 21 operators, including equality, contain- ment, regex matching, numeric comparison, list membership, null checks, prefix matching, and dynamic field references. Conditions are joined with AND logic. There is no LLM involvement in detection — a rule either matches or it does not, and the result is perfectly reproducible. Dynamic field references (e.g.,_DYNAMIC_ preprocessing.vendor_name_) are resolved at 6 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback Table 3|ALF action tiers. Tier selection depends on the nature and extent of the error. Tier ActionLLM Scope 1 Field editNo Single field 2 Surgical patch Yes Target fields 3 Pipeline cont. Yes Full output evaluation time, enabling parameterized condi- tions that adapt to case-specific context. Correction. When a rule matches, its action specifies how to correct the output. ALF supports a three-tier action model, summarized in Table 3: Tier 1 applies deterministic field edits for cases where the correct value is a known constant. Tier 2 invokes an LLM to determine correct values for specific fields, patching them into the existing output. Tier 3 is used when the agent terminated early (e.g., rejecting at an initial phase when it should have continued); the LLM continues the pipeline from the resumption point, producing a complete revised output. Aggregation. When multiple rules match, ALF uses a Collect-Plan-Execute pipeline, illustrated in Figure 2. All matching rules are collected, merged into a revision plan that combines actions by tier, and executed in fixed order: pipeline continua- tion first, surgical patches second, deterministic edits last. Scope-based mutual exclusion guaran- tees at most one rule fires per scope, preventing conflicts. This ensures at most two LLM calls per case regardless of how many rules match. Each correction is fully audited: which rules were evaluated, which matched, the revision plan, and complete LLM metadata (model, to- ken counts, latency). 3.5. Learning and Evolution Loop The Learning Agent enables SMEs to review agent output, describe desired corrections in natural language, and collaboratively discover new ALF rules. It orchestrates four sub-modules: Case Loader presents the acting agent’s output Rule Base Collect (eval conditions) Plan (group by tier) Tier 3 Continue Tier 2 Patch Tier 1 Edit Corrected Output Audit Trail Figure 2|ALF Collect-Plan-Execute pipeline. Conditions are evaluated deterministically (Col- lect), actions are grouped by tier (Plan), and exe- cuted in order: Tier 3→Tier 2→Tier 1. A full audit trail is maintained throughout. for a specific case, showing the decision, phase- by-phase validation results, extracted data, and correction history. Rule Discoverer accepts the SME’s natural lan- guage feedback and interprets it in the context of the case data, existing rules, and the rules book. Using an LLM grounded in these artifacts, it generates candidate conditions following a con- servative domain principle: conditions should be narrow enough to match only the intended cases but general enough to cover similar future cases. Impact Assessor evaluates the proposed rule deterministically against a sample of existing cases (with the target case always included), re- porting target matches (the case under review must match), collateral matches (other cases that also match), and safe non-matches. Rule Writer validates the schema, checks for conflicts with existing rules, backs up the rule base, and persists the approved rule with full metadata: who approved it, when, which cases it was designed for, and which rules book section it relates to. Each learning session is logged for auditability. The critical innovation is the programmatic safety loop, illustrated in Figure 3. This loop is enforced in code — not prompt instructions — so 7 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback it cannot be bypassed by the LLM. Each candidate rule passes through an iterative validation cycle: 1.Schema validation: The rule’s JSON structure is verified against the ALF schema. 2.Target match verification: The rule’s condi- tions are evaluated against the target case. If the rule fails to match, the LLM automatically broadens conditions while remaining conser- vative. 3.Collateral assessment: Conditions are eval- uated deterministically against a sample of cases. If unintended matches are found, the LLM automatically adds narrowing conditions (vendor name, amount range, service cate- gory) to eliminate them. This cycle repeats up to three times. If col- lateral persists after three attempts, the rule is presented to the SME with a warning. The SME may also request iterative revisions to a proposed rule; each revision re-enters the safety loop. At no point is a rule applied without explicit human approval. Care must be taken to control access to the approval process for rule persistence and in some cases a two step approval process may be required with an administrator or senior manager providing approval, after SME approval, before a rule is persisted. Rule lifecycle management. As the rule base grows, systematic lifecycle management becomes necessary to prevent it from becoming unwieldy. The framework provides several structural sup- ports for this. Each rule carries anenabledflag that allows deactivation without deletion, preserv- ing audit history while removing the rule from runtime evaluation. The Rule Writer creates a timestamped backup of the entire rule base be- fore any modification, enabling rollback of any change. Before a new rule is persisted, conflict detection checks for duplicate identifiers, prior- ity collisions within the same scope, and scope overlap with existing rules. Structured metadata fields (severity, root cause, rules-book section) enable rules to be queried, filtered, and priori- tized for review. Rules can also be permanently deleted, with a backup created before removal. Discover Rule Validate Schema Assess Impact Target Matched? Broaden Collateral? Auto- Tighten SME Approval Persist Rule Yes No No Yes ≤3× Figure 3|Programmatic safety loop for rule dis- covery. Candidate rules are iteratively validated, checked for target coverage, and auto-tightened to eliminate collateral matches before SME ap- proval. Beyond these building blocks, the architecture anticipates a periodic system review cycle, illus- trated in Figure 1. When a correction rule fires so consistently that it reveals a systematic defi- ciency in the acting agent, the underlying pattern should be promoted into the agent’s permanent logic, and the corresponding ALF rule retired. This promotion path keeps the rule base lean by ensuring that well-established corrections gradu- ate out of the runtime correction layer and into the agent itself. The current implementation does not yet automate this promotion, nor does it in- clude review queues, performance tracking per rule, or sunset policies; concrete mechanisms for these capabilities are discussed in Section 7. 3.6. Evaluation Engine The Evaluation Engine provides quantitative ac- curacy measurement when ground truth is avail- able. It operates as a schema-driven comparison framework with two layers: 8 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback Deterministic comparison performs field-by- field matching across configured comparison groups, applying financial tolerances (default: $0.02) for numeric fields and exact matching for text fields after normalization. LLM-as-judge (Zheng et al., 2023) (op- tional) provides a holistic alignment verdict per case: Aligned, Partially_Aligned, or Not_Aligned. This captures semantic align- ment beyond what field-level comparison can ex- press. Both layers read their configuration from the master data, ensuring that adding a new compar- ison group requires only a YAML change, not a code change. 4. Constitution-Based Agent Creation A distinctive property of Agent Gym’s architec- ture is that the constitutional artifacts — the rules book and master data — can serve not only as governance instruments for an existing agent but also as specifications for creating new agents. We outline this bidirectional relationship between constitution and agent. Constitution as specification. The rules book captures every validation step, threshold, rejec- tion condition, and decision outcome in human- readable form. The master data YAML codifies document types, extraction schemas, taxonomies, and pipeline configurations. Together, these ar- tifacts constitute a complete, machine-parseable specification of the desired agent behavior. A de- veloper building a new acting agent can use these artifacts as requirements documents, and an LLM can use them as grounding context for code gen- eration. Bootstrap and refinement cycle. A bootstrap agent can establish an initial constitution from an existing agent’s code. Alternatively — and as demonstrated in the reference implementation — the constitution can be authored first by a do- main expert describing the desired processing logic, then used to guide agent construction. This inverts the conventional flow: rather than code →constitution, the flow becomes constitution→ code→ refined constitution. Co-evolution. As the learning loop produces correction rules that address systematic agent er- rors, patterns emerge that indicate where the acting agent’s logic should be updated. The con- stitution captures these patterns in structured form. Over time, the constitution evolves to re- flect not just the original design intent but also the accumulated operational experience encoded in correction rules. This positions the constitu- tion as a living document that bridges the gap between initial specification and deployed behav- ior, providing a foundation for principled agent re-engineering when code-level changes become warranted. Onboardingnewagents. Adopting Agent Gym for a new acting agent requires two inputs: the acting agent itself and a constitution that describes its intended behavior. The frame- work’s computational components, including the ALF engine, the investigation agent, and the learning agent, are domain-independent and require no modification; they operate entirely from the constitutional artifacts. A bootstrap agent, itself an LLM-based pipeline, can automate constitution generation by analyzing the new agent’s source code and sample outputs. Because agent frameworks such as ADK impose consistent structural conventions, the bootstrap agent can extract pipeline stages, extraction schemas, validation logic, and decision rules from the code, then synthesize a reconstructed rules book and master data YAML without manual authoring. This bootstrap pipeline is described in Section 3.3 and is planned as a future automation; the current reference implementation uses manually authored artifacts. Once the constitution is in place, the ALF rule base starts empty. Subject matter experts then populate it through the learning loop as they observe the new agent’s behavior, discover error patterns, and validate correction rules through the programmatic safety loop described in Section 3.5. This separation ensures that the framework scales to new domains and new agents without engineering 9 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback effort beyond constitution generation. This bidirectional relationship distinguishes Agent Gym from approaches that treat agent development and agent monitoring as separate concerns. The framework’s configuration artifacts are simultaneously governance instruments and development specifications. 5.The Spec-to-Note Gap: An Autoen- coder View of Agentic System Trans- parency Modern agentic systems are rarely a single model. They are compositions of planners, tools, retriev- ers, guardrails, and sometimes multiple agents managing control flow (Dorri et al., 2018; Guo et al., 2024). Documenting what such a system actually does by hand is tedious and goes stale immediately. We propose a structural pattern that addresses this challenge. 5.1. The Autoencoder Analogy Consider the lifecycle of an agentic system as a transformation chain, illustrated in Figure 4: 1. A specification (natural language) describes the intended behavior. 2.Implementation (code, prompts, tools, config- uration) encodes this spec into an executable system. 3.An LLM-based auditor inspects the implemen- tation artifacts — code, prompt templates, tool definitions, evaluation results, sample exe- cution traces — and generates a transparency note in natural language. This chain forms an autoencoder over natu- ral language: the spec is the input, the imple- mented system is the latent representation, and the transparency note is the reconstruction. The analogy is not merely metaphorical — it yields a concrete diagnostic: contrasting the spec against the transparency note functions as a reconstruction loss, surfacing missing capabilities, silent scope creep, and behaviors the evaluation suite never measured. 5.2. Connections to Prior Work This pattern generalizes the round-trip correct- ness idea of Allamanis et al. (2024) from function- level consistency to whole-system granularity. It also connects to verbal feedback loops such as Self-Refine (Madaan et al., 2023) and Reflex- ion (Shinn et al., 2023), with two key differ- ences: the artifact being critiqued is a system rather than a single output, and the critique is itself a deliverable — a transparency note in the tradition of Model Cards (Mitchell et al., 2019) and Datasheets (Gebru et al., 2021). 5.3. SME Interface The transparency note serves as a shared inter- face with subject matter experts. SMEs cannot read evaluation harnesses, and even when they can, evaluations only check what the develop- ers thought to measure. But they can read a structured natural-language note about how the system behaves in their domain, and they will quickly flag wrong assumptions, missed popula- tions, and regulatory edge cases. Comments on the note become tickets on the system. Within Agent Gym, the Spec-to-Note pattern maps naturally to the Observe capability. An au- ditor agent can be run on a CI cadence over the system’s artifacts, regenerating the transparency note as the constitution and rule base evolve. 5.4. Open Questions Several questions warrant further investigation: how to measure the auditor’s own faithfulness to the system it inspects; when to regenerate the note (on every commit, on every rule change, on a fixed cadence); whether the gap between spec and note can be turned into an actual optimiza- tion signal rather than a review aid; and whether an adversarially-prompted auditor could serve as a release-gate red-teamer. 6. Reference Implementation We have developed an open-source reference im- plementation that demonstrates the complete Agent Gym framework in the domain of in- 10 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback Specification (Natural Language) Implementation (Code / System) LLM-Based Auditor Transparency Note (Natural Language) encode inspectgenerate reconstruction loss Figure 4|The Spec-to-Note autoencoder. Natural language specification is encoded into an imple- mented system, then decoded back to natural language by an LLM auditor. The gap between spec and transparency note surfaces missing or unintended behaviors. voice processing. The implementation is built on Google’s Agent Development Kit (ADK) and uses Gemini (Gemini Team, Google, 2023) models for all LLM operations. 6.1. System Overview The reference implementation is a unified dual- mode agent — a singleLlmAgentinstance with 18 registered function tools that supports both inference (document processing) and learning (SME-guided rule discovery) modes. The entire system is packaged as a self-contained Python module: all data, configurations, test cases, rules, and evaluation artifacts reside within the package, enabling deployment without external infrastruc- ture dependencies. 6.2. Acting Pipeline The acting agent implements a nine-stage sequen- tial pipeline: 1.Classifier: Identifies document types (invoice, work authorization form, email). 2.Extractor: Pulls structured data from PDFs using LLM-based extraction with Pydantic schema validation (Wang et al., 2023). 3.Phase 1–4 Validators: Progressive compli- ance validation — intake checks, content vali- dation, external validation (e.g., tax ID check- sum), and calculation validation. 4. Transformer: Normalizes line items against standard taxonomies. 5.Output Generator: Produces the final struc- tured decision. 6. Audit Logger: Creates a compliance trail. Each stage produces a numbered JSON arti- fact (e.g.,01_classification.jsonthrough 09_audit_log.json), providing full traceabil- ity. An early-exit mechanism allows the pipeline to skip remaining validation phases when a rejec- tion is determined, proceeding directly to output generation. 6.3. Investigation and Correction The investigation agent implements the three- layer architecture described in Section 3.4.2. Layer 2 caches discovered rules using SHA-256 hashing of the rules book content, eliminating redundant LLM calls across runs. Layer 3’s triple- check mechanism uses confidence thresholds of 90% for violations and 70% for ambiguous cases, treating ambiguity as compliant (conservative bias). Deterministic rule groups are batched into a single LLM call, and only the rules-book section relevant to each validation phase is included in the prompt rather than the full document, further reducing per-case token consumption. The ALF engine implements all 21 condi- tion operators described in Section 3.4.3, includ- ing dynamic field references that resolve values like_DYNAMIC_preprocessing.vendor_name_ at evaluation time. The Collect-Plan-Execute pipeline handles multi-rule scenarios with scope- based mutual exclusion. 6.4. Learning Mode The learning mode implements the full safety loop (Section 3.5). The rule discoverer generates candidate rules grounded in the rules book and existing rule base. The impact assessor evalu- ates conditions deterministically against a sam- ple of cases (always including the target). Auto- tightening adds conditions based on vendor name, amount range, rejection template text, and ser- 11 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback vice category to eliminate collateral matches. 6.5. Evidence of Framework Viability The reference implementation includes a com- prehensive test suite covering condition opera- tors, action executors, schema validation, con- flict detection, and impact assessment is publicly available 2 for invoice processing use case. A two- layer evaluation framework (deterministic field comparison with configurable tolerances, plus op- tional LLM-as-judge) enables systematic quality measurement. The implementation demonstrates several properties that validate the framework design: •Domain adaptability: Replacing the master data YAML and acting pipeline adapts the en- tire framework to a new document type. All downstream components — evaluation, inves- tigation, ALF, and learning — automatically adjust. • Operational readiness: The system runs via ADK’s web interface or command line, with documented paths to production deployment on cloud infrastructure. • Self-containment: The complete system, in- cluding test cases with PDF documents, ground truth, and pre-configured rules, ships as a sin- gle installable package. 7. Discussion and Future Work Strengths. Agent Gym’s principal strength is its separation of concerns: the acting agent, the correction layer, and the learning loop are inde- pendently versioned, tested, and evolved. The domain-agnostic design, achieved through the configuration layer, means the framework code is reusable across business processes. The human- in-the-loop governance model ensures that cor- rections are traceable and reversible. Limitations. The current framework has sev- eral limitations that future work should address. 2 https://github.com/google/adk-samples/ tree/main/python/agents/invoice-processing First, the bootstrap process is currently a man- ual workflow; an LLM-assisted bootstrap agent, as described in Section 3.3, would reduce the ef- fort required to author constitutional artifacts for new domains. Second, the investigation agent’s LLM-based validation incurs costs that scale with the number of cases and rule groups, though the caching, batching, section filtering, and early-exit mechanisms described in Section 3.4.2 substan- tially reduce this in practice; quantifying the cost reduction across large-scale deployments remains future work. Third, our reference implementa- tion validates the framework in a single domain; multi-domain validation is needed to fully estab- lish domain agnosticism. Future directions. Several extensions follow naturally from the current architecture: Automated rule suggestion. As the investigation agent accumulates compliance data, recurring vi- olation patterns can be identified and proactively surfaced to the SME, shifting the learning agent from reactive to proactive operation. Multi-run trend analysis. An administrative dashboard could support loading multiple agent runs, enabling comparative analysis across con- figurations and time periods. Rule lifecycle management. As the rule base ac- cumulates corrections over time, several capabili- ties become necessary. Periodic review, referenced in Figure 1 as “Periodic System Review”, would flag rules that have not matched any case within a configurable time window for SME review and po- tential deprecation. Performance tracking would record per-rule match counts and correction out- comes, and whether the corrected output was sub- sequently validated as correct by the evaluation engine, surfacing rules with high match rates but poor outcomes for revision. Consolidation would identify clusters of rules with overlapping condi- tions that could be merged into a single, more general rule, reducing rule-base size without sac- rificing coverage. Most importantly, promotion into the acting agent would address cases where a correction rule fires so consistently that it reveals a systematic agent deficiency; the corresponding logic should be promoted into the acting agent’s 12 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback permanent behavior and the ALF rule retired. Currently, the learning agent modifies only the ALF rule base; closing this loop is an important direction for future work. Cross-domain transfer. Organizations operat- ing multiple agents across different business pro- cesses could share a single Agent Gym instal- lation, with domain-specific behavior governed entirely by configuration. Effective rule structures and condition patterns discovered in one domain may transfer to others. Automated bootstrap. The bootstrap agent de- scribed in Sections 3.3 and 4 would enable fully automated onboarding of new acting agents by analyzing their source code and sample outputs to generate the constitutional artifacts. Investi- gating how to make this pipeline robust across different agent frameworks and how to handle agents whose internal structure does not follow conventional framework patterns are important open questions. Spec-to-Note as release gate. The autoencoder pattern described in Section 5 could be integrated into CI/CD pipelines, with the reconstruction loss serving as an automated quality gate for agent deployments. 8. Conclusion We have presented Agent Gym, a modular framework for continuous evaluation and evo- lution of LLM agents through human-in-the-loop feedback. The framework addresses the static agent dilemma — the fundamental tension be- tween frozen agent behavior and evolving busi- ness environments — by wrapping any exist- ing agent in a structured observation-correction- learning loop without modifying the agent’s source code. The framework’s technical contributions in- clude a hybrid deterministic-LLM correction en- gine that separates reliable detection from flexible correction, a three-layer investigation architec- ture that validates compliance without ground truth, and a programmatic safety loop that guar- antees rule correctness before human approval. The constitution-based governance model cap- tures domain knowledge in declarative artifacts that serve simultaneously as human documen- tation and machine specifications. The Spec-to- Note Gap concept provides a principled approach to automated system transparency. An open-source reference implementation demonstrates that the framework is not merely theoretical but fully operational. Agent Gym does not replace the need for well-engineered agents; it provides the environment in which those agents can be continuously measured, un- derstood, and improved — not by engineers rewriting code, but by domain experts contribut- ing the knowledge that only they possess. Acknowledgments We would like to thank Mitesh Agarwal for his valuable feedback and for reviewing this paper. Declaration on Generative AI During the preparation of this work, the author(s) used LLM-based tools in order to: assist with drafting, perform grammar and spell checks, as- sist with LaTeX formatting and bibliography man- agement. After using these tools, the author(s) reviewed and edited the content as needed and take full responsibility for the publication’s con- tent. References M. Allamanis, S. Panthaplackel, and P. Yin. Unsu- pervised evaluation of code LLMs with round- trip correctness. In International Conference on Machine Learning (ICML), 2024. S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zim- mermann. Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engi- neering: Software Engineering in Practice (ICSE- SEIP), pages 291–300. IEEE, 2019. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, 13 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback C. McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural In- formation Processing Systems, 33:1877–1901, 2020. H. Chase. LangChain.https://github.com/ langchain-ai/langchain, 2022. P. F. Christiano, J. Leike, T. Brown, M. Marber, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017. A. Dorri, S. S. Kanhere, and R. Jurdak. Multi- agent systems: A survey. IEEE Access, 6:28573– 28593, 2018. T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. Gemini Team, Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. I. Graham. Business rules management and ser- vice oriented architecture: A pattern language. 2006. T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang. Large lan- guage model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024. S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhou, et al. Metagpt: Meta programming for a multi- agent collaborative framework. In International Conference on Learning Representations, volume 2024, pages 23247–23275, 2024. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github is- sues? In International Conference on Learning Representations, volume 2024, pages 54107– 54157, 2024. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al. Holistic evalu- ation of language models. arXiv preprint arXiv:2211.09110, 2022. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. Agent- bench: Evaluating llms as agents. In Interna- tional Conference on Learning Representations, volume 2024, pages 52989–53046, 2024. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prab- humoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in neu- ral information processing systems, 36:46534– 46594, 2023. Microsoft Corporation. Microsoft AI transparency note.https://learn.microsoft.com/ en-us/legal/cognitive-services/ openai/transparency-note, 2020. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wain- wright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language mod- els to follow instructions with human feedback. Advances in Neural Information Processing Sys- tems, 35:27730–27744, 2022. R. G. Ross. Principles of the Business Rule Approach. Addison-Wesley Professional, 2003. D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison. Hidden techni- cal debt in machine learning systems. Advances in neural information processing systems, 28, 2015. 14 Agent Gym : A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback B. Settles. Active learning literature survey. Com- puter Sciences Technical Report 1648, University of Wisconsin–Madison, 2009. N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neu- ral information processing systems, 36:8634– 8652, 2023. Significant Gravitas. AutoGPT: An autonomous GPT-4 experiment.https://github.com/ Significant-Gravitas/AutoGPT, 2023. B. Wang, Z. Wang, X. Wang, Y. Cao, R. A Saurous, and Y. Kim. Grammar prompting for domain- specific language generation with large lan- guage models. Advances in Neural Information Processing Systems, 36:65030–65055, 2023. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. Au- toGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2023. 15