Paper deep dive
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses
Zhixuan Tan, Pengjie Gu, Zhao Li, Yihan Hu, Xu He, Dong Li, Jianye Hao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning, complex state management, and continuous cross-domain abstraction. However, current evaluations of skill evolution in autonomous agents suffer from a critical identifiability problem: they structurally confound genuine capability abstraction with rote solution leakage (i.e., copying highly similar code from historical training data). To resolve this, we introduce EngramBench, a rigorous, capability-grounded benchmark governed by the strict axiom of capability overlap without solution overlap. Comprising 30 diverse learning tasks and 13 unseen transfer tasks, EngramBench challenges agents to navigate interactive, multi-hour development cycles driven by LLM-simulated users. Our extensive evaluation across 48 multi-hour execution trajectories -- corroborated by human-expert validation -- reveals a profound insight into procedural memory. We demonstrate that static skill banks do not magically bypass the "last mile" of exact code implementation, which remains bottlenecked by the base model's inherent reasoning limits. However, they serve as an indispensable execution compass. By navigating agents away from catastrophic, token-heavy trial-and-error, genuine capability abstraction slashes redundant context bloat and reduces overall coding time by over 55%. Ultimately, EngramBench shifts the evaluation paradigm from trivial pattern matching to the verifiable measurement of deep, cross-domain capability transfer.
Tags
Links
- Source: https://arxiv.org/abs/2609.39284v1
- Canonical: https://arxiv.org/abs/2609.39284v1
Trouble viewing inline? Open PDF directly â
Full Text
59,458 characters extracted from source content.
Expand or collapse full text
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses Zhixuan Tan 1,* , Pengjie Gu 2,* , Zhao Li 2 , Yihan Hu 2 , Xu He 2 , Dong Li 2 , Jianye Hao 2 1 The Chinese University of Hong Kong, Shenzhen 2 MemoraX AI V 1.0¡ Sep 2026 * Equal contribution. Corresponding author: jianye.hao@tju.edu.cn Code: https://github.com/Virgil-Tan/EngramBench Abstract. While large language models have achieved remarkable success in isolated code genera- tion, authentic software engineering requires sustained reasoning, complex state management, and continuous cross-domain abstraction. However, current evaluations of skill evolution in au- tonomous agents suffer from a critical identifiability problem: they structurally confound genuine capability abstraction with rote solution leakage (i.e., copying highly similar code from histori- cal training data). To resolve this, we introduce EngramBench, a rigorous, capability-grounded benchmark governed by the strict axiom of capability overlap without solution overlap. Comprising 30 diverse learning tasks and 13 unseen transfer tasks, EngramBench challenges agents to nav- igate interactive, multi-hour development cycles driven by LLM-simulated users. Our extensive evaluation across 48 multi-hour execution trajectoriesâcorroborated by human-expert valida- tionâreveals a profound insight into procedural memory. We demonstrate that static skill banks do not magically bypass the âlast mileâ of exact code implementation, which remains bottlenecked by the base modelâs inherent reasoning limits. However, they serve as an indispensable execu- tion compass. By navigating agents away from catastrophic, token-heavy trial-and-error, genuine capability abstraction slashes redundant context bloat and reduces overall coding time by over 55%. Ultimately, EngramBench shifts the evaluation paradigm from trivial pattern matching to the verifiable measurement of deep, cross-domain capability transfer. Keywords long-horizon software engineering, agent benchmarks, skill evolution, procedural memory, cross-project transfer 1 Introduction Large Language Models (LLMs) have demonstrated strong capabilities in code generation and problem-solving (Chen et al. 2021; Yang et al. 2024). Building on this foundation, the field is increasingly focusing on skill evolution (Z. Li and Hu 2026; Wang et al. 2023). By equipping agents with memory and reflection mechanisms, this approach allows them to accumulate engineering experience and learn from execution feedback over time (Shinn et al. 2023). Rather than treating every task as an isolated challenge, agents can learn from past successes and mistakes, evolving MemoraX AI Research¡ EngramBench1 / 19 arXiv:2609.39284v1 [cs.SE] 30 Sep 2026 EngramBenchTechnical Report from static models into lifelong learners. Real-world software engineering requires agents to distill reusable practices from past projects and apply them to new, unfamiliar contexts. Therefore, establishing a benchmark to measure this cross-project learning is critical for determining whether AI agents can operate as continuous software engineers rather than one-off script generators. However, evaluating skill-learning frameworks poses a fundamental design challenge: structurally decoupling genuine skill abstraction from rote code memorization. This challenge is driven by a task-design dilemma. On the one hand, constructing learning and testing tasks within the same environment (e.g., a single repository) can ensure knowledge relevance, but reusable procedures must be separated from task-specific answer paths to prevent âsolution leakageâ (X. Li et al. 2026). Without this boundary, agents can achieve artificially high scores by simply copying historical abstract syntax trees (ASTs) or specific APIs, masquerading memorization as learning. Conversely, preventing solution reuse is not enough: learning and testing tasks must retain transferable skill dependencies for historical experience to carry instructional value (Zheng et al. 2025). Without both safeguards, evaluations risk either rewarding solution duplication or providing irrelevant histories, producing diagnostic ambiguity. Ultimately, when an agent fails on a new project, researchers cannot isolate the root cause: did the memory framework fail to abstract the skill, or did the benchmark simply provide the wrong history? This ambiguity exposes a fundamental flaw in current evaluations: the confusion problem. Downstream scores confound two entirely separate variables: Downstream Score= í Distillation Capability | z What we aim to measure Ă Presence of Evidence in History | z What the benchmark must guarantee (1) By treating the right side as an unverified black box, existing benchmarks make the left side impossible to isolate. A high score might merely reflect the base modelâs zero-shot intuition, while a low score might simply mean the required knowledge was absent from the historical tasks. To escape this trap and properly isolate the distillation capability, we must look to human developers. Human learning is a many-to-many abstraction process, not a 1-to-1 code mapping. An engineer who learns "transaction rollbacks" in e-commerce and "state synchronization" in IoT does not copy specific APIs when building a new medical system. The concrete solutions are entirely different, but the abstracted engineering capabilities are reused. Real learning happens when core challenges overlap, but specific implementations do not. Thus, a scientific benchmark must strictly enforce a golden design rule: Capability Overlap Without Solution Overlap(2) By strictly ensuring tasks share high-level challenges, we make the presence of evidence required in our first equation structurally auditable. Crucially, this axiom clearly delineates the boundaries of experience transfer: it strictly forbids the retention of project-specific implementations, hardcoded business constants, or exact API paths (solution leakage), while explicitly expecting the abstraction of general transaction templates, fault-recovery pseudocode, and cross-cutting engineering workflows (capability overlap). By running them on completely isolated systems with zero shared codebase, we physically prevent cheating. This axiom finally breaks the design trap and enables the measurement of true skill abstraction. Figure 1 illustrates this many-to-many transfer of reusable capabilities across otherwise distinct projects. To materialize this design axiom, we introduce EngramBench, a strictly decoupled, capability- grounded benchmark built to evaluate true skill evolution. First, to guarantee the presence of learning evidence, EngramBench features 30 representative learning tasks across 6 diverse business domains. Rather than merely providing the final correct code, the benchmark supplies feedback-rich MemoraX AI Research¡ EngramBench2 / 19 EngramBenchTechnical Report Learning Tasks 30 tasks, 6 domains Shared Skill Bank Reusable engineering skills Transfer Tasks 13 tasks Task L1 (order scheduling) Task L2 (file processing) Task L3 (payment handling) Task L30 (device deployment) Transactionality and Idempotency Resource and Ledger Consistency State Transition and Version Control Durable Execution and Recovery Deterministic Replay and Consistent Projection Cross-layer Validation and Delivery Closure Task T1 (DockChain) Task T2 (CarbonLedger) Task T3 (EscrowGuard) Task T13 (FlagFoundry) Figure 1: Many-to-many capability transfer in EngramBench. Experience from different learning tasks is abstracted into reusable skills that support multiple transfer tasks. Arrows denote illustrative capability relationships, not direct reuse of project-specific code. trajectoriesâincluding the agentâs initial mistakes, test-case failures, and iterative fixes. This ensures the framework has sufficient data to abstract underlying engineering principles. Second, to enforce solution isolation and evaluate many-to-many composition, we design 13 long-horizon transfer projects. These test projects share zero business logic or APIs with the learning tasks. To succeed, the agent cannot simply retrieve old code; it must dynamically combine abstract skills learned from different domains. Crucially, to test sustained execution beyond static code generation, our transfer projects are not single-turn tests. Instead, they are multi-turn, interactive software projects driven by an LLM-simulated user, following an interaction paradigm established in tool-agent-user evaluation (Yao et al. 2024). Throughout the multi-hour development cycle, this simulated user actively asks follow-up questions, checks progress against the fixed requirements, and rejects incomplete drafts. Reliable multi-turn execution presents a distinct challenge beyond single-turn capability (Laban et al. 2025). EngramBench tests whether distilled, long-term skills help agents meet that challenge. Finally, to maintain strict scientific discipline, EngramBench enforces a two- phase decoupled protocol. Once the learning phase is complete, the agentâs generated skill bank is strictly frozen, enforcing a strict information boundary between historical distillation and transfer execution. For controlled evaluation, we standardize the base model and learning trajectories across all methods, allowing us to purely compare the algorithmic efficiency of different skill-distillation frameworks. To demonstrate the value of this rigorous design, our evaluations establish a clear baseline for future research. We show that while current base models naturally struggle with these long-horizon tasks without prior knowledge, a proper skill-evolution framework can produce a real, positive learning delta, successfully guiding the agent through complex, multi-turn user interactions. In summary, the main contributions of this work are as follows: â˘A Clear Rule for Skill Evolution: We clearly define the identifiability problem in current benchmarks and introduce a strict design rule: capability overlap without solution overlap. ⢠Building EngramBench: We introduce a strictly decoupled, capability-grounded benchmark. It includes 30 diverse learning tasks with rich feedback, and 13 unseen, complex testing tasks driven by multi-turn simulated users. MemoraX AI Research¡ EngramBench3 / 19 EngramBenchTechnical Report â˘Strict Evaluation Protocols: We design a two-phase frozen testing process. With these fair and strict rules, we provide the community with a reliable standard to measure true skill evolution without hidden bias. 2 Related Work 2.1 Code Generation vs. Software Engineering The evaluation of Large Language Models (LLMs) in programming has historically centered on short- horizon code generation. Benchmarks such as HumanEval (Chen et al. 2021) and MBPP (Austin et al. 2021) evaluate the ability of models to synthesize standalone, algorithmic functions from natural language prompts. While these benchmarks effectively measure a modelâs static knowledge of syntax and basic algorithms, real-world software engineering extends far beyond isolated code snippets (Jimenez et al. 2023). Authentic software engineering is a long-horizon, dynamic process characterized by massive contextual dependencies, environmental feedback, and iterative debugging across multiple modules. To succeed in these environments, autonomous agents cannot rely solely on static pre-training; they must possess the ability to continuously interact, learn from execution failures, and accumulate reusable experience. This fundamental shiftâfrom single-turn code generation to continuous software engineeringânecessitates new paradigms for evaluating how agents abstract and reuse engineering skills over time. Recent empirical work by METR evaluates task-completion reliability as a function of task length, measured by the time human experts require, highlighting the need to assess sustained execution beyond short tasks (Kwa et al. 2025). EngramBench directly addresses this need through long, interactive development lifecycles. 2.2 Current Agent Benchmarks and Their Limitations To evaluate agents in realistic development environments, the community has introduced repository- level benchmarks, most notably SWE-bench (Jimenez et al. 2023) and its variants (Jimenez et al. 2023). These benchmarks represent a significant milestone by challenging agents to resolve real-world GitHub issues within complex, real-world codebases. Recently, some evaluations have attempted to adapt these static benchmarks into lifelong learning or skill-evolution scenarios by providing agents with historical issue-resolution trajectories as training data. However, when repurposed for skill evolution, these benchmarks suffer from the structural dilemma outlined in Section 1. Because their task sequences are inherently intra-domainâtypically requiring the agent to solve consecutive issues within the exact same repository (e.g., Django or SymPy)âthey inadvertently trigger solution leakage. Agents can achieve artificially high transfer scores simply by retrieving and copying highly similar abstract syntax trees (ASTs), specific local APIs, or database schemas from the historical tasks. Consequently, these benchmarks test an agentâs capacity for exact code retrieval and rote context-matching, rather than its ability to distill cross- cutting engineering capabilities. EngramBench diverges fundamentally from these paradigms by explicitly enforcing cross-domain task sequences, ensuring capability overlap without solution overlap. 2.3 Procedural Memory and Skill Evolution in LLMs To bridge the gap between static LLMs and lifelong learning agents, recent literature has explored ar- chitectures equipped with procedural memory or skill libraries. Frameworks such as Voyager (Wang et al. 2023), ExpeL (Zhao et al. 2024), and various agentic continual learning systems (Shu et al. 2026) attempt to distill execution trajectories into reusable text guidelines or executable code repositories. These harnesses aim to abstract past successes and failures into a persistent memory bank, which is then retrieved to guide decision-making in novel environments. MemoraX AI Research¡ EngramBench4 / 19 EngramBenchTechnical Report Despite the algorithmic innovations introduced by these harnesses, their reported efficacies remain confounded by the benchmarks on which they are evaluated. Because existing evaluations fail to decouple a methodâs true abstraction capability from the sheer availability of highly similar in-domain data, the true utility of current procedural memory mechanisms remains opaque. We cannot ascertain whether an agent has genuinely learned an abstract principle (e.g., concurrent state synchronization) or merely memorized a repository-specific workaround. EngramBench provides the missing, capability-grounded infrastructure necessary to causally verify and benchmark these procedural memory frameworks. 3 Constructing EngramBench Unlike traditional benchmarks that repurpose isolated GitHub issues for static code completion, EngramBench is architected as a requirement-driven, long-horizon software engineering sandbox. It challenges agents to synthesize complete business systems from lightweight API contracts, forcing them to navigate cross-layer dependencies, environmental constraints, and interactive feedback. 3.1 Task Curation & Domains Instead of scraping disparate open-source commits, EngramBench features meticulously constructed, requirement-driven sandbox projects that mirror real-world business constraints. The accumulation phase consists of 30 learning tasks strategically distributed across 6 core business domains: Resource Booking & Logistics, Data & Collaboration, Transaction & Ledger, IoT & Configuration, Security & Moderation, and Infrastructure & Communication. This taxonomy guarantees that abstract engineer- ing capabilities (e.g., transactional idempotency, failure recovery) appear repeatedly across disjoint business contexts. Figure 2 summarizes the learning-task distribution and the cross-cutting capabilities shared across domains. Each task follows a strict developmental lifecycle: a definitive README contract, a frozen execution plan, and a scenario-driven progression. Agents are provided with a deterministic scaf- folding that defines request/response contracts, initialization hooks, and public interface checkers. To provide a rigorous capability-grounded foundation, the learning phase yields 30 complete, feedback-rich development trajectories. The current dataset archives 583 interactive execution turns, capturing simulated user dialogue, agent decisions, tool invocation events, compiler outputs, and hidden evaluation results. Crucially, all tasks and distilled skills have been rigorously verified through human-expert baseline executions to ensure their feasibility and intrinsic engineering utility. Complementing the learning tasks, the benchmark includes a transfer evaluation set of 13 tasks. These tasks share no business logic with the accumulation set but are explicitly designed to test the cross-domain extraction of the underlying engineering capabilities mapped in the learning curriculum. To ground this capability overlap in actual learning experiences, we make the evidence directly inspectable. We systematically map representative public requirements from the transfer tasks to reusable engineering mechanisms, tracing them back to concrete implementation, repair, and verification episodes in the learning histories. These links are established through recorded execution trajectoriesâsuch as local failure-repair loops or fault-injection validationsârather than superficial similarities in task names. Table 1 presents representative mappings that validate this capability coverage. Appendix A provides the complete task-level mapping across all 13 transfer tasks. MemoraX AI Research¡ EngramBench5 / 19 EngramBenchTechnical Report Figure 2: The 30 learning tasks span six business domains. Cross-cutting engineering capabilities recur across domains, providing a basis for skill abstraction and transfer. Table 1: Learning Evidence Mapping for Representative Transfer Requirements Transfer Task Reusable Mechanism Learning Episode (Source)Evidence Type Eval CreatorRights- Exchange Append-only correction and read-projection consistency BillForge (T7): Ledger reads omitted adjustment entries; a query repair restored their visibility. Local repair loop B-10 IncidentRelayStable event identity and persistent retry handling QueueForge (T13): SIGKILL before ACK persistence. ConfigRelay (T12): HTTP 503 followed by same-event retry and ordering checks. Impl + Fault injection C-07 CarbonLedger Publish consistency during post-generation crash ArtifactVault (T26): SIGKILL injected after file rename but before DB commit. Impl + Verification C-03 CapacityLease Expired lease recovery and backlog drain throughput QueueForge (T16): Insufficient recovery count; worker batching and test-counting fixes followed by a passing re-test. Local repair loop E-06 Note: Evidence Type distinguishes implementation-and-verification sequences from iterative local failure-repair loops recorded in execution trajectories. T denotes the recorded turn number; ArtifactVault T26 is from the original development history, while the other cited turns are from V2 migration histories. These are mechanism-level links, not proof that entire transfer tasks were solved in the learning phase. The full task-level table appears in Appendix A. MemoraX AI Research¡ EngramBench6 / 19 EngramBenchTechnical Report 3.2 Codebase Complexity and Cross-Layer Implementation EngramBench operates in a modern, full-stack engineering environment primarily utilizing Type- Script, Node.js 22, React, and PostgreSQL 16. Rather than dropping agents into massive legacy codebases to perform localized one-line bug fixes, the benchmark evaluates the ability to build complex, cross-layer implementations from a unified structural foundation. For the 13 transfer tasks, the initial execution environment provides a clean V2 API scaffold. On average, each starter repository contains 23.46 files (including documentation and contract definitions) and 8 source code files, totaling 622 non-empty lines of code (LOC) and 268 public operations across the entire transfer set. The true complexity of EngramBench lies in its deep engineering constraints rather than raw initial LOC. Agents are required to implement full-stack business logic that spans database migrations, background dispatchers, OpenAPI integrations, and browser-observable UI states. For instance, in a task like CapacityLease, the agent must not only implement a booking endpoint but also guarantee time-range capacity conservation, atomic admission control, concurrent race-condition handling, and crash recovery. These challenges demand that the agent orchestrates persistent state synchronization and fault tolerance, decisively elevating the evaluation beyond superficial syntax generation. 3.3 Environment Capabilities and Multi-Layered Feedback To support these long-horizon tasks, EngramBench provides an isolated, stateful OCI-compliant (Docker) execution sandbox. The environment natively provisions necessary toolchains, including shell access, file I/O operations, source code search utilities, TypeScript/npm build environments, PostgreSQL databases, and Chromium for end-to-end (E2E) browser testing. A critical differentiator of EngramBench is its dynamic, multi-layered feedback loop, which organically guides the agent through multi-hour development cycles: ⢠Execution Feedback: Agents autonomously invoke tools and immediately receive real-world standard output/error, including compiler traces, database connection states, and unit test failures. ⢠Interactive Progression: A Simulated User Agent actively monitors the execution against the frozen plan. This multi-turn interaction paradigm has precedent in benchmarks such así-bench, which pairs a language agent with a simulated user and domain-specific tools (Yao et al. 2024). Rather than injecting new out-of-scope requirements, the simulated user queries progress, handles blockages, and forces the coding agent to cross-reference the original README to determine missing implementations. â˘Delivery and Evaluation: The development cycle is governed by a strict two-stage delivery pipeline. Agents first iterate against a public interface checker. Once the public contracts are satisfied, the code is strictly frozen and dispatched to an isolated hidden evaluation environment. Here, comprehensive testingâspanning integration, concurrency, performance, and crash- recovery scenariosâgenerates the definitive test feedback, determining whether the underlying engineering capabilities were successfully synthesized. The Long-Horizon Scale. Unlike traditional benchmarks that execute in a single prompt- completion cycle, EngramBench accurately simulates the marathon nature of real-world software engineering. A single transfer task typically demands 25 to 30 interactive development turns, characterized by continuous exchanges between the Simulated User and the Coding Agent. Throughout this prolonged process, the agent must continuously maintain context, parse massive MemoraX AI Research¡ EngramBench7 / 19 EngramBenchTechnical Report execution logs, and navigate cascading errors, culminating in hours of continuous execution and tens of millions of processed tokens per project. 4 Rigorous Evaluation Protocol To ensure that performance gains genuinely reflect capability abstraction rather than mere context matching or test-set overfitting, EngramBench establishes a highly stringent, multi-dimensional evaluation protocol governed by physical isolation boundaries. 4.1 The Confirmed-Case Pass Fraction (CCPF) Traditional code generation benchmarks predominantly rely on binary Pass@1 metrics. However, for long-horizon software engineering, binary success is overly coarse: it fails to distinguish between an agent that implemented 95% of the core business logic but missed an edge case, and an agent that failed to even initialize the database. To provide a high-resolution signal of requirement coverage, we introduce the Confirmed-Case Pass Fraction (CCPF). For a given taskíĄwith an applicable hidden test case setC íĄ , the CCPF is defined as: CCPF íĄ = 1 |C íĄ | âď¸ íâC íĄ 1[í passes]= í íĄ |C íĄ | (3) whereí íĄ represents the number of confirmed passed cases, andC íĄ is the set of all applicable, non-excluded evaluation cases. All applicable cases are equally weighted, and the metric evaluates the finalized delivery without arbitrarily excluding challenging edge cases. Crucially, self-authored unit tests generated by the agent are explicitly ignored. The denominator|C íĄ |strictly includes all applicable evaluation cases. Consequently, unresolved states, compilation crashes, or unreached test cases remain in the denominator but contribute nothing to the passed numerator, implicitly penalizing incomplete executions. For macro-level performance across multiple runs, we utilize the micro-average CCPF to accurately reflect the holistic completion rate of complex engineering contracts. 4.2 Multi-Dimensional Verification and Hidden Stress Testing EngramBench evaluates systems far beyond superficial API responses. As demonstrated by the EvalPlus framework (Liu et al. 2023), inadequate test suites can fail to expose incorrect code and distort model rankings. To reduce this âfake-greenâ risk, our hidden evaluators systematically stress-test the agentâs implementation across five critical engineering dimensions: (A) Contracts & Lifecycle, (B) Data Idempotency & Concurrency, (C) Worker Recovery & Durability, (D) Cross-Layer UI Consistency, and (E) Operational Compatibility. The hidden evaluators employ aggressive, adversarial testing mechanisms to verify true engi- neering capabilities. For example, in the CapacityLease task, the evaluator does not simply check the successful return of a booking API. Instead, it enforces: â˘Concurrency & Idempotency: The evaluator fires 64 concurrent requests with identical idempotency keys to multiple API instances. It strictly asserts that the database yields exactly one lease and one corresponding business event, penalizing naive implementations that bypass race conditions. ⢠Controlled Crash & Recovery: The evaluator intentionally pauses a background worker and terminates it viaSIGKILL. It then spawns a replacement worker to verify if the uncompleted task is correctly resumed and finished without duplicating business side-effects. MemoraX AI Research¡ EngramBench8 / 19 EngramBenchTechnical Report ⢠Anti-Hallucination (Fake-Green) Checks: To prevent agents from writing vacuous tests that always pass, the evaluator dynamically breaks dependencies (e.g., swapping the database for an unreachable address or masking the Chromium path). If the agentâs integration or E2E tests still pass under these broken conditions, the submission is failed for hallucinating execution logic. 4.3 Anti-Cheating and Sandboxing Protocols To eradicate solution leakage and ensure causal evaluation, EngramBench enforces a strict two-phase decoupled protocol supported by OCI-compliant (Docker) container isolation. Phase 1: Skill Generation & Freezing. During the learning phase, the harness distills historical trajectories into a reusable Skill Bank (e.g.,SKILL.md). Crucially, this output is compiled into a manifest with SHA-256 hashes. This guarantees deterministic, immutable inputs for the downstream evaluation. Phase 2: Isolated Transfer Execution. For the 13 transfer tasks, the agent is instantiated in a completely independent environment. It is provided with the current taskâs README, a clean workspace scaffold, and the frozen Skill Bank. The agent is physically restricted from mounting or accessing the raw workspaces, interaction histories, or codebases from the learning phase. Furthermore, skill evolution is strictly disabled during this phase, preventing the agent from continuously optimizing its memory bank using test-set feedback. This freeze applies within each run: the S-Full-20 bank was selected by qualitative coverage of engineering procedures rather than transfer scores, whereas the Coreless-3 ablation was designed after inspecting the preceding S-Full/S-Ablated comparison and is therefore exploratory. Hidden Evaluation Boundaries. To prevent look-ahead bias, the hidden evaluators are entirely unmounted during the development phase. Once the agent passes the public interface checker, the harness automatically freezes the commit and logs a secure code digest. The evaluation phase then mounts this frozen snapshot into a separate, dedicated testing container. This multi-stage isolation keeps hidden-test code outside the development workspace, reducing opportunities for direct test-specific optimization and supporting evaluation of transfer to new projects. 5 Experimental Setup To empirically validate the necessity of procedural memory, we conduct a highly controlled ablation study. We focus on four representative transfer tasks (CapacityLease, EscrowGuard, MeterSettle, and FlagFoundry) and evaluate four skill configurations. With three independent trials per configuration- task pair, this yields 48 strictly audited execution trajectories. Across all runs, the primary Coding Agent is powered by GPT-5.5 Medium. To standardize the in- teractive progression and eliminate human variability, we deploy DeepSeek V4 Pro (thinking/high mode) as the Simulated User Agent. The developmental boundaries are strictly enforced: the task README defines the immutable requirements, and the Frozen Plan dictates the initial architectural strategy. Fixing the plan reduces variation in initial planning and focuses the comparison on ex- ecution capability; controlled experiments show that even when explicit knowledge and plans are provided, per-step execution accuracy can degrade as the horizon grows (Sinha et al. 2025). The user Scenario strictly drives the interaction forward without injecting out-of-scope business requirements. MemoraX AI Research¡ EngramBench9 / 19 EngramBenchTechnical Report 5.1 Baselines: Native Skills vs. Ablations To disentangle the value of the provided memory from the base modelâs inherent reasoning, we deploy the following configurations: ⢠No Skills (Zero-Shot Baseline): Evaluates the modelâs native capability to navigate long-horizon challenges relying solely on its pre-trained weights. ⢠S-Full-20 (Complete Memory): A static repository of 20 consolidated skills generated via a rigorous two-stage distillation and compression pipeline. First, the full historical trajectories of the 30 learning tasks are locally distilled into a massive raw skill bank (368 skill packages). Second, a global compression phase (using DeepSeek V4 Pro at temperature 0) semantically merges redundant mechanisms and eliminates generic advice, retaining exactly 20 highly actionable and cross-domain transferable engineering principles. â˘S-Ablated-20 (History Filtering): To test the frameworkâs sensitivity to evidence quality, this bank is derived from a deliberately weakened historical dataset. Targeted removal of critical implementation, debugging, and verification records reduces the raw historical text volume by 69.66%. Crucially, these deprived trajectories undergo the exact same two-stage pipeline and compression prompt as S-Full-20, distilling down to 20 skills. This isolates the impact of severe evidence deprivation on the final synthesized knowledge. â˘Coreless-3 (Capability Ablation): A surgically edited version of the S-Full-20 bank that removes the targeted core software engineering procedures (e.g., concurrency control), leaving only 3 basic interface skills. While this exploratory ablation reduces both the volume and scope of the memory, it serves to test whether the base model can sustain performance when these procedures are absent from the supplied memory. 5.2 Evaluation Metrics A robust skill-evolution framework must balance functional correctness with computational effi- ciency. As emphasized by prior work on agent evaluation (Kapoor et al. 2024), success rates alone are insufficient; evaluation should jointly measure accuracy and cost. Accordingly, we report token consumption and development time alongside functional correctness. 1. Functional Correctness. We report task success using the Micro-Average CCPF, governed by the strict rules defined in Section 4. Performance deltas are strictly reported in absolute percentage points. 2. Computational Cost (Tokens & Time). We track the exact resource burden introduced by the developmental loop: â˘Total Tokens: Merges input and output (reasoning) tokens. We strictly deduplicate cached contexts by session to reflect true computational overhead. â˘Pure Development Time: The Coding Agentâs exact wall-clock execution time, explicitly subtracting all off-cycle phases (e.g., Simulated User processing, hidden evaluations, and verified pauses). 3. Behavioral Dynamics. To differentiate between an agent that "gives up early" and one that "works efficiently", we track internal actions per turn: command execution counts, patch invocations, reasoning token generation, and context compaction triggers. Reductions in resource overhead are reported as relative percentages against the No Skills baseline. MemoraX AI Research¡ EngramBench10 / 19 EngramBenchTechnical Report 6 Experimental Results & Analysis We evaluate four configurationsâNo Skills, S-Full-20, S-Ablated-20, and Coreless-3âacross 48 strictly audited execution trajectories. Rather than merely reporting binary success rates, our analysis deconstructs the exact behavioral impact of procedural memory, revealing a profound shift in how agents navigate long-horizon engineering tasks. 6.1 Aggregate Performance: The âLast Mileâ Bottleneck We first evaluate the impact of skills on functional correctness using the micro-average Confirmed- Case Pass Fraction (CCPF) across the 12 runs for No Skills and S-Full-20. Table 2: Task-Level CCPF: No Skills vs. S-Full-20 (12 Runs Each) TaskNo Skills S-Full-20 Change (p) MeterSettle76.42%80.49%+4.07 FlagFoundry46.34%47.15%+0.81 EscrowGuard18.52%33.33%+14.81 CapacityLease47.83%46.38%-1.45 Micro-Average46.63%51.25%+4.62 As shown in Table 2, injecting the complete skill bank provides a verifiable overall gain (+4.62 percentage points). However, the improvement is uneven. While complex transactional tasks like EscrowGuard see massive gains (+14.81 p), CapacityLease experiences a slight regression. This non-monotonic benefit exposes a critical philosophical insight into autonomous learning: a reusable engineering principle is not an executable implementation. Procedural memory provides a high-level architectural compass (e.g., âhow to structure idempotent transactionsâ), but it in- tentionally omits exact syntax. The âlast mileâ of engineeringâwriting the precise code, bridging specific APIs, and passing edge-case assertionsâstill fundamentally relies on the base modelâs inherent reasoning limits. If the base model struggles with final execution constraints, even the best navigational guide cannot single-handedly guarantee a perfect pass rate. 6.2 The Efficiency Paradigm: Memory as an Execution Compass While correctness improvements are bound by the base modelâs âlast mileâ capabilities, the impact of the skill bank on execution efficiency is transformative. Table 3: Average Efficiency Metrics per Run: No Skills vs. S-Full-20 MetricNo SkillsS-Full-20 Reduction Input + Output Tokens50,025,849 40,192,20319.66% Non-Cached Input Tokens3,211,135612,84980.91% Completed Dev Turns27.0026.501.85% Coding Time (Mins)189.0084.4655.3% Table 3 reveals a massive 55.3% reduction in total coding time (saving over 104 minutes per run). Crucially, this time saving is not achieved by simply cutting conversations short (completion turns only decreased by 1.85%). Furthermore, network logs reveal that reduced external wait times account for onlyâź6 minutes of this saving. The true source of this acceleration lies in how the skill bank fundamentally alters the modelâs internal cognitive loop: by navigating the agent away from MemoraX AI Research¡ EngramBench11 / 19 EngramBenchTechnical Report catastrophic architectural errors early, it slashes non-cached input tokens by a staggering 80.91%, eliminating redundant context bloat. 6.3 Ablation Insights: Capability Trumps History Volume To disentangle the source of this efficiency, we analyze our two targeted ablations. S-Ablated-20 tests evidence deprivation (removing 69.66% of historical text prior to distillation), while Coreless-3 tests capability deprivation (surgically retaining only 3 basic interface skills). Table 4: Aggregate Correctness and Coding Time Across All Configurations ConfigurationCCPF Avg. Coding Time (Mins) No Skills46.63%189.00 S-Full-2051.25%84.46 S-Ablated-2052.60%91.74 Coreless-348.36%âĽ190.87 History volume is not skill value. Despite a 69.66% reduction in raw historical text, S-Ablated-20 surprisingly achieves the highest CCPF (52.60%), albeit with a slight execution cost penalty (+8.6% coding time compared to S-Full-20). This confirms that as long as the distillation pipeline abstracts the correct engineering principles, effective skill transfer can be sustained by a substantially reduced evidence base. The volume of historical experience is secondary to the quality of the abstracted knowledge. Execution degradation under Coreless-3. Conversely, while Coreless-3 represents a compound reduction in both skill count and content, its observed execution degradation supports the interpre- tation that abstracted engineering capabilities contribute to the performance gains of S-Full-20. When the targeted core software engineering skills are removed from the supplied memory, the configuration fails to preserve the correctnessâtime profile. CCPF drops to 48.36%, and the average coding time skyrockets back to over 190 minutesâworse than the No Skills baseline. 6.4 Behavioral Audit: Decisive Execution vs. Trial-and-Error To understand the mechanics behind the Coreless-3 collapse and the S-Full-20 acceleration, we audit internal execution actions across all 48 runs (Table 5). Table 5: Behavioral Action Audits (Aggregate Totals Across 12 Runs) Action / EventNo Skills S-Full-20 S-Ablated-20 Coreless-3 Tool Invocations6,4844,8505,9798,195 Return Text (Chars)13.66M6.06M10.11M14.76M Patch Invocations865607751988 Added Patch Lines40,39338,91441,50046,059 Reasoning Tokens441,803302,016360,958699,961 Context Compactions21111626 The behavioral data reveals two distinct execution paradigms: MemoraX AI Research¡ EngramBench12 / 19 EngramBenchTechnical Report 1.Concentrated Edits over Fragmented Patches (S-Full-20): Guided by the complete skill bank, the agent invoked the patch tool 29.8% less often than the baseline, yet total lines added only decreased by 3.66%. This means edits grew from an average of 47 to 64 lines per patch. The skill bank helps the agent consolidate its architectural plans, acting with decisive execution rather than falling into a fragmented âsmall editâ checkâ small editâ loop. 2.Token-Heavy Thrashing (Coreless-3): Without core capabilities, the agentâs execution profile becomes severely bloated. Compared to S-Full-20, tool invocations spike by 68.97%, and generated reasoning tokens explode by 131.76% (reaching nearly 700k tokens). Without the execution compass, the agent relies on exhaustive, blind exploration, flooding the context window with 14.76M characters of error logs and forcing 26 context compactions. Ultimately, EngramBench makes these dynamics visible. Fewer skills do not necessarily mean less work; rather, the absence of abstracted core capabilities forces the base model into catastrophic, token-heavy trial-and-error. 7 Limitations and Future Work While EngramBench establishes a rigorous framework for evaluating skill evolution, we acknowledge two boundary conditions that direct future research: Computational Scale. Authentic long-horizon execution is computationally exorbitant. Because a single run can consume up to 66 million tokens and require manual trajectory auditing, our deep ablation study is concentrated on 48 strictly controlled runs across 4 representative tasks. Scaling this high-resolution audit to the entire 13-task suite remains an ongoing effort. Evaluation Scope and Infrastructure. To causally isolate the algorithmâs distillation capability, we evaluated a strictly frozen skill bank, omitting the complexities of dynamic memory retrieval (e.g., Active RAG). Furthermore, our empirical findings are currently grounded in GPT-5.5 Medium. Future work will test diverse open-weights architectures and enforce stricter, air-gapped container boundaries to further immunize the hidden evaluators. 8 Conclusion In this work, we exposed a critical vulnerability in how the community evaluates skill evolution in autonomous agents: the structural confounding of genuine capability abstraction with rote solution leakage. To resolve this identifiability problem, we introduced EngramBench, the first capability-grounded benchmark governed by the strict axiom of capability overlap without solution overlap. Crucially, EngramBench shifts the evaluation focus beyond mere functional correctness to equally prioritize execution efficiencyâstrictly measuring token consumption, development time, and the elimination of meaningless code churn. Our rigorous empirical analysis across 48 multi-hour execution trajectoriesâfurther corroborated by human-expert validation to confirm the intrinsic utility of the tasks and distilled skillsârevealed a profound insight into procedural memory. We demonstrated that static skill banks do not magically bypass the âlast mileâ of exact code imple- mentation, which remains bottlenecked by the base modelâs inherent reasoning limits. However, they serve as an indispensable execution compass. By navigating agents away from catastrophic, token-heavy trial-and-error and redundant code modifications, genuine capability abstraction dras- tically streamlines the internal cognitive loopâslashing context bloat and reducing overall coding time by over 55%. MemoraX AI Research¡ EngramBench13 / 19 EngramBenchTechnical Report Ultimately, EngramBench shifts the agent evaluation paradigm from trivial pattern matching to the verification of deep, cross-domain capability transfer. We provide this infrastructure to the community as a definitive standard, ensuring that future advancements in lifelong learning agents are measured by their true ability to abstract, generalize, and engineer efficiently. References Austin, J., A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton (2021). Program Synthesis with Large Language Models. doi: 10.48550/arXiv.2108.07732. arXiv: 2108.07732. url: https://arxiv.org/abs/2108.07732. Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. doi: 10.48550/arXiv.2107.03374. arXiv: 2107.03374. url: https://arxiv.org/abs/2107.03374. Jimenez, C. E., J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? doi: 10.48550/arXiv.2310.06770. arXiv: 2310.06770. url: https://arxiv.org/abs/2310.06770. Kapoor, S., B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan (2024). AI Agents That Matter. arXiv: 2407.01502. url: https://arxiv.org/abs/2407.01502. Kwa, T. et al. (2025). Measuring AI Ability to Complete Long Tasks. arXiv: 2503.14499v1. url: https://arxiv.org/abs/2503.14499v1. Laban, P., H. Hayashi, Y. Zhou, and J. Neville (2025). LLMs Get Lost In Multi-Turn Conversation. arXiv: 2505.06120v1. url: https://arxiv.org/abs/2505.06120v1. Li, X. et al. (2026). SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. doi: 10.48550/arXiv.2602.12670. arXiv: 2602.12670. url: https://arxiv.org/abs/2602.12670. Li, Z. and Y. Hu (2026). SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History. arXiv: 2606.08671. url: https://arxiv.org/abs/2606.08671. Liu, J., C. S. Xia, Y. Wang, and L. Zhang (2023). âIs Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationâ. In: Advances in Neural Information Processing Systems. Vol. 36. url: https://arxiv.org/abs/2305.01210. Shinn, N., F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. doi: 10.48550/arXiv.2303.11366. arXiv: 2303.11366. url: https://arxiv.org/abs/2303.11366. Shu, Y., B. JimĂŠnez GutiĂŠrrez, S. P. Jonnalagedda, Y. Yao, H. Sun, and Y. Su (2026). AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents. doi:10.48550/arXiv.2606.02461. arXiv:2606.02461. url: https://arxiv.org/abs/2606.02461. Sinha, A., A. Arun, S. Goel, S. Staab, and J. Geiping (2025). The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs. arXiv: 2509.09677v1. url: https://arxiv.org/abs/2509.09677v1. Wang, G., Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. doi:10.48550/arXiv.2305.16291. arXiv:2305.16291. url: https://arxiv.org/abs/2305.16291. Yang, J., C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. doi: 10.48550/arXiv.2405.15793. arXiv: 2405.15793. url: https://arxiv.org/abs/2405.15793. Yao, S., N. Shinn, P. Razavi, and K. Narasimhan (2024). í-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv: 2406.12045. url: https://arxiv.org/abs/2406.12045. Zhao, A., D. Huang, Q. Xu, M. Lin, Y.-J. Liu, and G. Huang (2024). âExpeL: LLM Agents Are Experiential Learnersâ. In: Proceedings of the AAAI Conference on Artificial Intelligence 38.17, p. 19632â19642. doi: 10.1609/aaai.v38i17.29936. url: https://ojs.aaai.org/index.php/AAAI/article/view/29936. Zheng, J., X. Cai, Q. Li, D. Zhang, Z.-Z. Li, Y. Zhang, L. Song, and Q. Ma (2025). LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners. arXiv: 2505.11942v1. url: https://arxiv.org/abs/2505.11942v1. MemoraX AI Research¡ EngramBench14 / 19 EngramBenchTechnical Report A Learning Evidence Coverage Across the Transfer Set Table A.1 extends the representative mappings in Section 3 to all 13 transfer tasks, followed by a shared cross-layer acceptance mapping. Each row links a public requirement to a reusable engineering mechanism, a recorded learning episode, and the corresponding evaluation dimen- sion. Coverage is complete at the task level, not an exhaustive audit of every README clause; a mechanism-level link does not establish that a skill bank extracted the mechanism or that an agent used it successfully. Evidence notation. Repair loop denotes an observed failure or incorrect result, an applied change, and subsequent executed verification. Implementation + verification denotes implemented behavior followed by executed checks without a recorded pre-repair failure of that same behavior. HTTP observation distinguishes observed responses from assertion-based verification. T denotes the turn number in the archived learning trajectory: all cited turns are from V2 migration histories except ArtifactVault T26, which is from its original development history. Case IDs refer to the evaluator version audited on 30 September 2026, not to historical pass results. Table A.1: Requirement-level capability mapping across all 13 transfer tasks and a shared cross-layer acceptance mapping. Transfer task and public requirement Reusable mechanism Learning sources Recorded failure, repair, and verification evidence Evaluation dimension / cases MeterSettle Select rates and billing periods by event time; append corrections without destroying historical records or their read relationships. Event-time version binding; append-only correction; consistent read projections. GeoPulse T15; BillForge T7. Implementation + verification: GeoPulse bound historical queries and events to the applicable revision and ran integration regression checks. Repair loop: BillForgeâs adjustment returned HTTP 500; a migration fix enabled creation, but ledger reads still omitted the adjustment. A query repair restored PAYMENT and ADJUSTMENT entries in subsequent HTTP output. These episodes do not establish MeterSettleâs specific pricing arithmetic. B: rating, correction, concurrency. B-01; B-07. DockChain Allocate Berth, Tug, and Yard resources as a complete bundle; two linked Movements must commit atomically without exceeding capacity. Multi-resource commit boundaries; capacity contention; transactional idempotency. QuotaMesh T6; BillForge T12. Implementation + verification: QuotaMesh repaired allocation/revision boundaries and ran integration tests including hierarchical capacity contention. BillForge repaired nested database connections, shared the transaction between business writes and idempotency, and executed concurrent same-key checks with a single-record assertion. The cited history does not contain a complete multi-resource deadlock-to-lock-order repair loop. B: bundle allocation, concurrency. B-02; B-09. Continued on next page MemoraX AI Research¡ EngramBench15 / 19 EngramBenchTechnical Report Table A.1 (continued) Transfer task and public requirement Reusable mechanism Learning sources Recorded failure, repair, and verification evidence Evaluation dimension / cases IncidentRelay After an unknown Domain Event ACK, retry the same event identity/body while preserving per-aggregate order. Durable outbox; unknown-result retry; stable event identity; ordered delivery. QueueForge T13; ConfigRelay T12. Implementation + fault injection: QueueForge killed the dispatcher after webhook 2x but before ACK persistence, restarted it, and checked redelivery of the same event ID. ConfigRelay used an HTTP receiver returning 503 on the first request, then asserted stable event identity and that later events did not overtake the unacknowledged event. These were controlled fault-injection checks, not observed production incidents. C: delivery, recovery. C-07: unknown webhook ACK. FlagFoundry Advance progressive rollout over frozen cohorts; resolve pass/rollback at the deadline without letting late Outcomes rewrite the terminal state. Frozen target sets; staged progression; deadline- triggered durable compensation. ConfigRelay T6âT7. Implementation + verification: ConfigRelay implemented frozen cohorts and automatic rollback. Executed HTTP/worker scenarios observed a successful first cohort, a failed second cohort, completed ROLLBACK Work, and delivered rollback commands. This supports the progression and compensation mechanisms, not verification of FlagFoundryâs bucket algorithm or all Outcome/deadline races. B: rollout correctness. B-09: Out- come/deadline race. CarbonLedger Expose a Certificate only with the RETIRED commit; recover after file generation without duplicate certificates or deductions. File/DB publication consistency; recoverable intermediate state; lease fencing. ArtifactVault original T26; MediaDock T8. Repair loop: ArtifactVaultâs 81 ms retry interval failed its recovery check; worker and test-isolation changes were followed by a passing re-test. Implementation + verification: the same turn added recovery from a renamed blob before DB commit and checked a unique commit after SIGKILL. MediaDock asserted publication CAS, revision-bound grants, and downloaded bytes. No pre-fix execution failure was found for the rename/DB gap itself. C: worker recovery, persistence. C-03: post-generation SIGKILL. ParcelFlow Split an Order into warehouse Fulfillments; allow at most one Shipment per group and keep the parent terminal state consistent with every group. Frozen membership; fan-out; child/parent state consistency; atomic persistence. ConfigRelay T6âT7; BillForge T12. Implementation + verification: ConfigRelayâs frozen cohorts, per-member progression, and asynchronous rollback produced actual HTTP and Work results. BillForge supplied repaired transaction boundaries and executed idempotency regression checks. These are composable mechanisms, not evidence that the learning tasks already verified cross-warehouse greedy allocation or Shipment/cancel business rules. B: grouped allocation, concurrency. B-03; B-10. Continued on next page MemoraX AI Research¡ EngramBench16 / 19 EngramBenchTechnical Report Table A.1 (continued) Transfer task and public requirement Reusable mechanism Learning sources Recorded failure, repair, and verification evidence Evaluation dimension / cases ColdChainControl Late readings may correct excursion history without regressing the current projection; Recall freezes its target set at publication. Event-time rules; late-event classification; history/current separation; frozen membership. GeoPulse T14âT15; ConfigRelay T6âT7. Implementation + verification: GeoPulse asserted late-event classification and continuous transition sequences in T14, then repaired historical revision selection and ran regression checks in T15. ConfigRelayâs staged cohorts supplied executed frozen-membership behavior. These excerpts do not directly establish a complete excursion rebuild or non-regression of the current route. B: projection, concurrency. B-06; B-10. CreatorRights- Exchange Correct CLOSED royalties only by appending to a later OPEN period; preserve old postings/periods and trace each adjustment. Append-only ledger correction; compensation rather than rewriting; original-entry links and read projections. BillForge T7; LedgerBridge T6. Repair loop: BillForge moved from adjustment HTTP 500 to successful creation with missing ledger visibility, then repaired the query and observed both entry types. Implementation + HTTP observation: LedgerBridge added REVERSAL; reversal and same-key replay matched, and both statements retained TRANSFER/REVERSAL entries. The single-leg scenario is not complete cross-period financial verification. B: data correctness, idempotency, concurrency. B-10: adjustment uniqueness/bal- ance. AccessSentinel After revocation commits, checks must fail closed; stale credentials/caches cannot authorize, and concurrent operations must use current authority. Credential generations; rotation/reuse; monotonic revocation; conditional checks on authoritative state. IdentityMesh T10/T12; ClinicGrid T43. Implementation + HTTP observation: IdentityMesh recorded refresh replay/reuse, revoked-subject rejection, and stale key-rotation rejection. Repair loop: after a Work state/terminal HTTP 500, ClinicGrid changed to a DB-conditional claim; the re-test no longer produced that error. The latter supports atomic authority checks, not complete multi-API cached revocation behavior in AccessSentinel. B: authority, idempotency, concurrency. B-09: cross-process revocation. Commerce- Command Retain UNKNOWN after a provider accepts payment but loses the response; recover with stable request identity without another charge or journal entry. Unknown versus failed outcomes; stable side-effect identity; durable retry; transactional records. QueueForge T13; BillForge T12. Implementation + fault injection: QueueForgeâs kill-before-persisted-ACK check exercised a side effect whose response had already arrived, followed by redelivery after restart. BillForge supplied repaired joint business/idempotency commit behavior and concurrent same-key regression checks. This is a mechanism-level composition; webhook redelivery does not itself verify exactly-once payment charging. C: worker recovery, persistence. C-05: payment reconciliation. Continued on next page MemoraX AI Research¡ EngramBench17 / 19 EngramBenchTechnical Report Table A.1 (continued) Transfer task and public requirement Reusable mechanism Learning sources Recorded failure, repair, and verification evidence Evaluation dimension / cases EscrowGuard Conserve available/re- leased/refunded funds; create all Beneficiary Payouts atomically, without releasing and refunding the same amount twice. Integer fund conservation; immutable compensation entries; same-transaction idempotency and settlement. LedgerBridge T6; BillForge T12. Implementation + HTTP observation: LedgerBridge appended reversal entries and observed both partiesâ statements and replay. Repair + verification: BillForge repaired shared transaction handling and ran concurrent idempotency checks. These episodes supply financial correction and commit-boundary experience; the single-leg history does not establish complete multi-beneficiary contention behavior. B: fund correctness, concurrency. B-01; B-09. PermitForge Only a valid current-Stage Claim may write a Decision; stage evidence and revisions must remain correctly scoped. Stage-local identity; Claim/revision fencing; decision-to-audit consistency. Moderation- Flow T3âT4; ClinicGrid T43. Implementation + assertion-based verification: ModerationFlow implemented revision/lease-aware claims and exercised real HTTP claim-to-decision flows with Case, audit, and snapshot assertions. ClinicGrid supplied its DB-conditional claim repair. These positive flows do not verify every stale Claim, role quorum, or repeated reviewer across Stages. A-14 reflects the current public Stage-policy revision, not an unchanged V1 requirement. A: public contract, A-14. B: concurrency, B-10. CapacityLease Preserve capacity and atomic Gang state; recover expired leases, fence stale owners, and actually drain the backlog. Capacity/commit invariants; DB leases and fencing; recovery throughput; post-condition checks. QuotaMesh T6; QueueForge T16. Implementation + verification: QuotaMesh repaired capacity/revision boundaries and ran hierarchical contention tests. Repair loop: QueueForge recovered only 572/2,000 expired leases; batch-result commits and SKIP LOCKED changes raised recovery to 2,000/2,000. Its test query also needed a join-duplicate counting fix before the final public performance command passed. This is not proof of complete Gang fairness. B: B-09, B-10. C: C-05, expired-lease fencing. E: E-06, recovery at scale. Continued on next page MemoraX AI Research¡ EngramBench18 / 19 EngramBenchTechnical Report Table A.1 (continued) Transfer task and public requirement Reusable mechanism Learning sources Recorded failure, repair, and verification evidence Evaluation dimension / cases Shared cross-layer acceptance CapacityLease and CreatorRights- Exchange: production UI actions must traverse real APIs and appear in persistent state/public projections. Browser-to-API- to-storage closure; independent acceptance; non-empty projection checks. MergeBoard T5; EvidenceChain T10; MediaDock T8. Repair loops: MergeBoardâs Chromium test could not find controls; production asset-loading repairs were followed by a passing browser test. EvidenceChainâs split succeeded but snapshot returned HTTP 500; adding the parentâs aliquots repaired the subsequent HTTP chain. Implementation + verification: MediaDock compared actual downloaded bytes with uploaded bytes. These are concrete exercised paths, not proof of whole-project acceptance. D: OpenAPI, UI, cross-layer checks. CapacityLease: D-01, D-02, D-08. CreatorRights- Exchange: D-02, D-08. MemoraX AI Research¡ EngramBench19 / 19