Paper deep dive

From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python

Jinhua Wang, Biswa Sengupta

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 65

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%

Last extracted: 4/14/2026, 2:35:45 AM

Summary

The paper presents a methodology for LLM-assisted continuous code translation, migrating a production Rust-based AI coding agent (CODEX CLI) to Python. By using public agent benchmarks (SWE-bench, Terminal-Bench) as an objective function, the authors achieved functional parity and subsequently evolved the Python port into a capability superset with 30 feature-flagged extensions, demonstrating that Python's expressiveness provides significant advantages for agentic systems with negligible performance costs.

Entities (6)

CODEX CLI · software-system · 100%JP Morgan Chase & Co. · organization · 100%Python · programming-language · 100%Rust · programming-language · 100%SWE-BENCH · benchmark · 100%Terminal-Bench · benchmark · 100%

Relation Signals (3)

SWE-BENCH → evaluates → CODEX CLI

confidence 100% · The Python port resolves 59/80 SWE-bench Verified tasks

CODEX CLI → migratedfrom → Rust

confidence 100% · We present a methodology for LLM-assisted continuous code translation in which a large language model translates a production Rust codebase... into Python

CODEX CLI → migratedto → Python

confidence 100% · We translate it to Python using LLM-assisted translation

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Cross-language migration of large software systems is a persistent engineering challenge, particularly when the source codebase evolves rapidly. We present a methodology for LLM-assisted continuous code translation in which a large language model translates a production Rust codebase (648K LOC, 65 crates) into Python (41K LOC, 28 modules), with public agent benchmarks as the objective function driving iterative refinement. Our subject system is Codex CLI, a production AI coding agent. We demonstrate that: (1) the Python port resolves 59/80 SWE-bench Verified tasks (73.8%) versus Rust's 56/80 (70.0%), and achieves 42.5% on Terminal-Bench versus Rust's 47.5%, confirming near-parity on real-world agentic tasks; (2) benchmark-driven debugging, revealing API protocol mismatches, environment pollution, a silent WebSocket failure mode, and an API 400 crash, is more effective than static testing alone; (3) the architecture supports continuous upstream synchronisation via an LLM-assisted diff-translate-test loop; and (4) the Python port has evolved into a capability superset with 30 feature-flagged extensions (multi-agent orchestration, semantic memory, guardian safety, cost tracking) absent from Rust, while preserving strict parity mode for comparison. Our evaluation shows that for LLM-based agents where API latency dominates, Python's expressiveness yields a 15.9x code reduction with negligible performance cost, while the benchmark-as-objective-function methodology provides a principled framework for growing a cross-language port from parity into an extended platform.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

64,824 characters extracted from source content.

Expand or collapse full text

From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python Jinhua Wang LLM Suite Team JP Morgan Chase & Co. jinhua.wang@jpmorgan.com Biswa Sengupta LLM Suite Team JP Morgan Chase & Co. biswa.sengupta@jpmorgan.com Abstract—Cross-language migration of large software systems is a persistent engineering challenge, particularly when the source codebase evolves rapidly. We present a methodology for LLM- assisted continuous code translation in which a large language model translates a production Rust codebase (648K LOC, 65 crates) into Python (41K LOC, 28 modules), and public agent benchmarks serve as the objective function that drives iterative refinement. Our subject system is CODEX CLI, a production AI coding agent. We demonstrate that: (1) the LLM- translated Python port resolves 59/80 SWE-bench Verified tasks (73.8%) versus the Rust original’s 56/80 (70.0%), and achieves 42.5% accuracy on Terminal-Bench versus Rust’s 47.5% (post- fix complete rerun), demonstrating near-parity on real-world agentic tasks with Python slightly ahead on SWE-bench and Rust slightly ahead on Terminal-Bench; (2) benchmark-driven debugging—where failing tasks reveal API protocol mismatches, environment pollution, tool-availability gaps, a silent WebSocket empty-response failure mode present in both implementations, and a model-generated invalid content-item type that crashed the agent—is more effective than static testing alone at closing the parity gap; (3) the translation architecture supports continuous upstream synchronisation, enabling the Python port to absorb new Rust commits through an LLM-assisted diff-translate-test loop; and (4) the Python port has evolved into a capability superset of the Rust original through a codex.enhancements module that adds 30 feature-flagged extensions—multi-agent orchestration, semantic memory, persistent plans, cost track- ing, IDE bridge, guardian safety assessment, voice mode, and more—absent from the Rust implementation, while a layered flag-resolution system preserves strict parity mode for head- to-head comparison. Our evaluation across code complexity, test coverage, runtime performance, and head-to-head agent benchmarking shows that for LLM-based agents where API latency dominates, Python’s expressiveness yields a 15.9× code reduction with negligible performance cost, while the benchmark- Disclaimer: This paper was prepared for informational purposes by the LLM Suite group of JP Morgan Chase and its affiliates (‘JPMC’) and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation, warranty or undertaking whatsoever and disclaims all liability for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful. © 2026 JP Morgan Chase & Co. All rights reserved. as-objective-function methodology provides a principled frame- work for growing a cross-language port from parity into a first- class extended platform. I. INTRODUCTION Cross-language migration of large software systems is a recurring challenge in software engineering. Teams migrate codebases for many reasons—performance, ecosystem access, contributor accessibility—but the process is labour-intensive, error-prone, and typically a one-time effort. Once migration is complete, the source and target diverge permanently; upstream improvements in the original language must be manually re- ported or abandoned. This problem is especially acute for fast- moving projects where the upstream ships daily. Recent advances in large language models (LLMs) suggest a different approach: continuous, LLM-assisted code translation where an LLM performs the bulk of cross-language transla- tion, and automated benchmarks serve as the objective func- tion that validates correctness and drives iterative refinement. Rather than a one-shot migration, this methodology establishes a living bridge between the source and target codebases. In this paper, we apply this methodology to CODEX CLI [1], a production AI coding agent originally implemented in Rust (648K LOC across 65 crates). We translate it to Python using LLM-assisted translation, producing a 41K LOC implementa- tion across 28 modules—a 15.9× code reduction. Crucially, we validate the translation not only through 2,621 unit tests but through head-to-head agent benchmarking: running both the original Rust CLI and our Python port on Terminal-Bench [2], an 80-task benchmark of complex terminal operations. Our contributions are: • Benchmark-as-objective-functionmethodology.We demonstrate that public agent benchmarks (Terminal- Bench, SWE-bench) serve as effective objective functions for cross-language translation. Our Python port achieves 42.5% on Terminal-Bench versus the original’s 47.5%, and 73.8% on SWE-bench Verified versus Rust’s 70.0%— near-parity on both benchmarks—with benchmark-driven debugging revealing protocol mismatches, environment 1© 2026 JP Morgan Chase & Co. arXiv:2604.11518v1 [cs.SE] 13 Apr 2026 pollution, tool-availability gaps, and transport-layer failure modes that unit tests missed. • From parity to superset. We show that the same LLM- assisted translation methodology that achieves parity can continue to evolve the port beyond the original. The Python port ships a codex.enhancements module with 30 feature-flagged extensions—multi-agent orchestration, se- mantic memory, guardian safety assessment, cost tracking, and more—absent from the Rust implementation, while preserving a strict parity mode for fair comparison. • Continuous upstream synchronisation architecture. We describe an LLM-assisted diff-translate-test loop that en- ables the Python port to continuously absorb upstream Rust commits. The architecture uses git submodule tracking, au- tomated diff extraction, LLM-driven translation of changed modules, and benchmark regression testing to maintain parity as the upstream evolves. • Comprehensive empirical evaluation. We evaluate the migration across nine dimensions: code size, cyclomatic complexity, test parity, runtime performance, API sur- face, dependency structure, migration effort, end-to-end agent benchmarks, and head-to-head benchmarking on both Terminal-Bench and SWE-bench Verified. • LLM-assisted translation patterns. We document the systematic patterns by which an LLM translates Rust idioms to Python equivalents, including error handling (Result<T,E> → exceptions), concurrency (Tokio → asyncio), and serialization (serde → Pydantic), achieving idiomatic target code rather than mechanical transpilation. Our findings have broad implications beyond this specific migration. The benchmark-as-objective-function approach is language-agnostic and applicable to any translation where functional equivalence can be measured through automated evaluation. For the emerging class of API-latency-bound LLM applications, our results demonstrate that Python’s expressive- ness advantages substantially outweigh Rust’s performance benefits, yielding a more maintainable and extensible system. The remainder of this paper is organized as follows: Sec- tion I provides background on CODEX CLI and AI coding agents. Section I details the system architecture. Section IV describes the LLM-assisted translation methodology and con- tinuous sync architecture. Section V presents our empirical evaluation including head-to-head benchmark results. Sec- tion VI surveys related work, and Section VII concludes. I. BACKGROUND AND SYSTEM OVERVIEW A. AI Coding Agents AI coding agents are LLM-powered systems that interact with a developer’s local environment to accomplish software engineering tasks. Unlike code completion tools that generate snippets, coding agents operate in a loop: they analyze the current state of a codebase, formulate a plan, invoke tools (shell commands, file edits, web searches), observe results, and iterate until the task is complete or a human intervenes. CODEX CLI is a production AI coding agent that runs as a terminal application. It connects to OpenAI’s Responses API, supports multiple LLM backends (GPT-4, o3, o4-mini, and local models via Ollama/LM Studio), and provides both interactive chat and non-interactive batch execution modes. The system enforces configurable security policies through platform-native sandboxing and supports extensibility through the Model Context Protocol (MCP) [3]. B. System Scope The system comprises 28 Python modules (Table IV), organized into six architectural layers: 1) Agent Layer: Core agent runner, tool orchestration, guardian (automated approval), multi-agent coordination, and memory management. 2) Security Layer: Sandbox enforcement (Seatbelt, Bub- blewrap, seccomp), execution policy evaluation, and pro- cess hardening. 3) Protocol Layer: Wire protocol types, approval workflows, model abstractions, and JSON-RPC event system. 4) Integration Layer: MCP client/server, authentication (OAuth/PKCE), backend API client, and cloud task man- agement. 5) Presentation Layer: Terminal UI (Textual framework), ap- plication server (WebSocket/HTTP), and CLI entry points. 6) Infrastructure Layer: State persistence (SQLite), configu- ration management, telemetry, analytics, feature flags, and utilities. C. Original Rust Implementation The system was originally implemented in Rust using the Tokio async runtime, comprising 65 crates with approximately 648K lines of code. The Rust implementation leveraged the language’s type system for compile-time safety guarantees, particularly in the sandbox and protocol subsystems. However, the build system complexity (Bazel + Cargo), compilation times, and the cognitive overhead of Rust’s ownership model motivated an exploration of alternative implementation lan- guages. D. Migration Context The migration to Python was motivated by three factors: (1) iteration velocity—the ability to rapidly prototype and deploy agent capabilities is critical in a fast-evolving LLM landscape; (2) ecosystem integration—Python’s dominance in the ML/AI ecosystem simplifies integration with model providers, data processing tools, and community extensions; and (3) contributor accessibility—Python lowers the barrier for external contributors and plugin developers. I. SYSTEM ARCHITECTURE Figure 1 illustrates the high-level architecture of CODEX CLI. We describe each major subsystem below. A. Agent Runner The agent runner implements the core execution loop: (1) send the current conversation context to the LLM API, (2) receive a response that may contain tool calls, (3) execute each tool call through the orchestrator, (4) append results to 2© 2026 JP Morgan Chase & Co. core Agent, TurnCtx, ConvHistory exec ToolDef, ShellExec, Sandbox tui ratatui, EventLoop cli clap, Subcommands mcp Client, Server, stdio/SSE login OAuth, JWT, Keyring Rust Original (codex-rs) 6 crates agent.runner QueryEngine, StreamToolExec config_loader, sandbox, perms agent.tools 34 handlers: shell, patch, list_dir, mcp, permissions cli.tui Textual, widgets, events cli click, exec/chat/review mcp Client, Server, OAuth agent.auth+sandbox Seatbelt, Bubblewrap seccomp, ExecPolicy Python Port (codex. * ) 1:1 parity with Rust port agents/ • MULTI_AGENT, ORCHESTRATION • MULTI_AGENT_V2 guardian • GUARDIAN (on) ◦ VOICE_MODE (off) memory/ ◦ SEMANTIC_MEMORY • DENIAL_REPLAY tools/ • SKILLS, NOTEBOOK_EDIT ◦ LSP_INTEGRATION state/ • PERSISTENT_PLANS • COST_TRACKING bridge/resilience/ • SYSTEM_REMINDERS ◦ WORKTREE_TOOLS Python Enhancements (codex.enhancements. * ) 35 flags:•=on◦=opt-in (Python only) extends Fig. 1: Three-tier architecture comparison. Left: the six Rust crates of codex-rs. Center: the Python port (codex. * ), with one module per Rust crate; arrows denote 1:1 correspondence. Right: codex.enhancements — a Python-only superset of 30 flag-gated capabilities absent from Rust, layered incrementally above the port. the conversation, and (5) repeat until the model produces a final response or the maximum turn count is reached (default: 50 turns). In the Python port, the originally monolithic runner was decomposed into focused, single-responsibility modules: a stateful QueryEngine that owns the agent loop lifecy- cle; an auth module for credential resolution and JWT refresh; a config_loader for layered TOML configu- ration; a sandbox module for platform-specific command isolation; a permissions module providing a unified three- layer approval pipeline; a StreamingToolExecutor for semaphore-based concurrent tool dispatch; and a tool_result_budget module for oversized output man- agement. A thin runner.py orchestrator delegates to these modules while maintaining backward-compatible re-exports. Key design decisions include: • Event-drivenarchitecture: The runner emits typed events(TurnStarted, ToolCall, ToolResult, TurnCompleted) consumed by the presentation layer, enabling loose coupling between agent logic and UI rendering. • Turn-scoped approval caching: When a user approves a tool invocation (e.g., a shell command), the approval is cached for the current turn, preventing redundant prompts for semantically equivalent operations within the same log- ical step. • Multi-phase context management: Context compaction operates in three phases: microcompaction strips stale tool outputs inline; snip compaction removes low-value mes- sages below a token threshold; and full compaction performs LLM-based summarization with boundary markers, post- compact file restoration (up to 5 files, 50K token budget), and ghost snapshot preservation. • Toolresultbudgeting:Oversizedtooloutputs (>100K chars) are saved to disk and replaced with a compact pointer message containing head/tail previews, preventing context window exhaustion during long-running sessions. • Layeredpermissionmiddleware:Asingle can_use_tool() entry point evaluates three layers sequentially—config-based pattern matching, automated guardian LLM risk assessment, and interactive user prompts—with results cached per-turn. B. Tool Orchestration The tool orchestrator manages a registry of ToolHandler implementations, each responsible for a specific tool type: 3© 2026 JP Morgan Chase & Co. • shell: Executes shell commands within the security sand- box. • apply_patch: Applies unified diffs to files with conflict detection. • list_dir: Directory listing with configurable depth. • mcp_handler: Delegates to MCP server tools. • request_permissions: Handles runtime permission escalation. Each tool invocation passes through a three-stage pipeline: (1) policy check against the execution policy engine, (2) ap- proval check via the guardian or user prompt, and (3) sand- boxed execution with result capture. C. Security Sandbox The sandbox subsystem enforces filesystem and network isolation for tool executions. Three platform-specific imple- mentations are provided: • macOS (Seatbelt): Generates dynamic Seatbelt profiles that whitelist specific filesystem paths and network operations based on the configured policy. • Linux(Bubblewrap/seccomp): Uses Bubblewrap for mount namespace isolation and Landlock for filesystem ac- cess control, with seccomp filters for system call restriction. • Windows: Restricted process tokens with job object con- straints. Sandboxpoliciesareorganizedintothreemodes ofincreasingpermissiveness: read-only(default), workspace-write (write access to the project directory), and full-access (unrestricted, requires explicit opt-in). D. Execution Policy Engine The execution policy engine evaluates tool invocations against a declarative rule set before sandbox enforcement. Rules are specified in a Python-based DSL that supports prefix matching on command strings, network access control by host/port, and host executable whitelisting. This separation of policy from mechanism enables data-driven policy composi- tion: users, organizations, and the system can contribute policy layers that are merged at runtime. E. Multi-Agent Orchestration The multi-agent subsystem enables hierarchical task dele- gation. A parent agent can spawn child agents with inherited conversation context (via history forking), assign them specific tasks, and coordinate their results. Safety bounds prevent unbounded recursion: maximum depth of 5, maximum 10 children per parent, and maximum 100 total agents per session. Each child agent operates in its own sandbox scope and approval context. F. Model Context Protocol (MCP) The MCP integration provides bidirectional protocol sup- port: as a client, CODEX CLI connects to external MCP servers to access additional tools and resources; as a server, it exposes its own capabilities to other MCP-compliant systems. The client supports two transports—stdio (for local servers) TABLE I: Python-Specific Enhancement Module Categories CategoryFlagsDefault Multi-agent MULTI_AGENT, MULTI_AGENT_V2, MULTI_AGENT_ORCHESTRATION On Safety GUARDIANOn Tool extensions FILE_TOOLS, FILE_EDIT, WEB_FETCH, NOTEBOOK_EDIT, WORKTREE_TOOLS Mixed Memory MEMORY_SYSTEM, TYPED_MEMORY, SEMANTIC_MEMORY, AUTO_MEMORY Mixed Context mgmt MULTI_STRATEGY_COMPACTION, FORKED_COMPACTION Mixed Productivity SKILLS, CRON_TOOL, VOICE_MODE Mixed Session PERSISTENT_PLANS, SYSTEM_REMINDERS, DENIAL_REPLAY On IDE integration IDE_BRIDGE, LSP_INTEGRATION Mixed Observability COST_TRACKING, APP_STATE, STARTUP_PREFETCH On and Streamable HTTP with Server-Sent Events (for remote servers)—with OAuth-based authentication and automatic to- ken refresh. G. State Management Session state is persisted to SQLite databases with WAL (Write-Ahead Logging) mode for concurrent access. The state subsystem manages conversation history, thread metadata, agent job records, and extracted memories. Schema versioning (currently v5 for state, v1 for logs) enables forward-compatible migrations. H. Python-Specific Enhancements Beyond wire-level parity with the Rust original, the Python port has evolved into an extended platform through a codex.enhancements module that ships 30 feature- flagged capabilities absent from the Rust implementation. These extensions are entirely additive: loaded lazily on first use, they never modify core data structures and are excluded from parity tests. A four-tier flag-resolution system (build- time compiled flags → environment variables → runtime --enable/--disable overrides→ defaults) lets operators run in strict parity mode (all enhancement flags off) for apples- to-apples benchmark comparison, or in extended mode to access the full capability surface. The sub-packages under codex/enhancements/ im- plement each category: agents/ for hierarchical multi- agent spawning and coordination beyond the base multi- agent runner; memory/ for typed and semantic memory extraction; compaction/ for multi-strategy and forked com- paction policies; resilience/ for source-aware retry logic; 4© 2026 JP Morgan Chase & Co. startup/ for prefetch optimisations; state/ for app- level state management and runner bridging; and tools/ for tool registration and command registry extensions. Intent-preservation enhancements (PERSISTENT_PLANS, SYSTEM_REMINDERS, DENIAL_REPLAY) add session- level coherence mechanisms that maintain goal context across long-running agents—a category of functionality with no analogue in the Rust codebase. This architecture reflects a key advantage of Python for agentic systems: the ecosystem’s rapid-prototyping culture enables new capabilities to be implemented and shipped as opt-in extensions in the time it would take to prototype them in Rust, while the flag system ensures the additions never compromise the benchmarked parity baseline. IV. LLM-ASSISTED TRANSLATION METHODOLOGY We describe our methodology for LLM-assisted cross- language translation, the idiom mapping patterns that emerged, and the continuous upstream synchronisation architecture that keeps the translation current. A. Translation Process Unlike traditional transpilation tools that perform syntax- level conversion, our approach uses an LLM as the translation engine. The process operates at the module level: for each Rust crate, we provide the LLM with the source code, its test suite, and the target Python module’s existing context (imports, dependent modules). The LLM produces idiomatic Python that preserves behavioral semantics while adapting to Python conventions. The translation proceeded in dependency order: foundation modules (protocol types, configuration, utilities) first, then infrastructure (state management, authentication), then core logic (agent runner, tool handlers), and finally presentation (CLI, TUI). This ordering ensures that each translated module can import its already-translated dependencies. a) The role of benchmarks as objective functions.: Unit tests provide necessary but insufficient validation of translation correctness. Many subtle bugs—API protocol mismatches, tool registration errors, output format differences—only manifest when the full agent pipeline executes end-to-end against real tasks. We discovered that public agent benchmarks (Terminal- Bench, SWE-bench) serve as powerful objective functions for translation quality: 1) Detect integration failures. Our initial Terminal-Bench run scored 0% because the adapter used a simplistic LLM- to-tmux bridge instead of the full agent runner. Benchmark failure immediately revealed the gap. 2) Expose API protocol bugs. The Python port initially sent ‘type’: ‘local_shell’ to the Responses API, which returned HTTP 400. This was invisible to unit tests but caused 100% fallback to the Chat Completions API. Benchmark comparison (31% vs 49%) exposed the issue. 3) Reveal environment assumptions. Installing the Python port via pip polluted the container’s Python environment, TABLE I: Terminal-Bench accuracy across translation itera- tions IterationFix AppliedAccuracy v0 (baseline)Original Rust CLI47.5% v1Naive tmux adapter0.0% v2Full agent runner (Chat Completions)31.3% v3Responses API function tool fix35.0% v4Conversation history fix35.0% v5Venv isolation + ripgrep install45.0% TABLE I: Rust-to-Python Idiom Mapping Rust PatternPython Equivalent Result<T, E>Exceptions (raise/try) Option<T> Optional[T] enum (algebraic) @dataclass + Union[...] enum (simple) enum.Enum struct @dataclass(frozen=True) impl Trait Protocol / ABC async/await (Tokio) async/await (asyncio) Arc<Mutex<T>>Plain objects (GIL) serde (de/serialize)Pydantic BaseModel reqwest (HTTP) httpx ratatui (TUI)Textual clap (CLI)Click sqlx (SQL) sqlite3 (stdlib) Cargo workspaceSingle pyproject.toml breaking tasks that depended on pre-installed packages (e.g., pandas, pyarrow). The original Rust CLI, installed via npm, had no such interference. 4) Quantify parity. Each benchmark run produces a scalar accuracy metric that directly measures functional equiva- lence, enabling iterative refinement toward the target. Table I shows how benchmark-driven debugging progres- sively closed the parity gap. B. Idiom Mapping Table I summarizes the systematic translation patterns em- ployed during migration. The LLM was instructed to produce idiomatic Python—not mechanical transliterations—meaning Rust patterns are mapped to their natural Python equivalents. C. Key Design Decisions a) Pydantic for Protocol Types.: The protocol layer— comprising 4,016 LOC of type definitions, approval workflows,andconfigurationschemas—usesPydantic BaseModel for automatic JSON serialization, validation, and schema generation. This replaces Rust’s serde derive macros while adding runtime type checking that catches protocol violations early. Hot-path internal types use @dataclass to avoid Pydantic’s validation overhead. b) Textual for Terminal UI.: The Rust implementation used ratatui with a custom event loop. The Python port uses Textual, a modern Python TUI framework with CSS- like styling, widget composition, and built-in async support. 5© 2026 JP Morgan Chase & Co. Upstream Rust Repo Diff Extraction LLM Translation Benchmark Validation Python Port pass fail: refine Fig. 2: Continuous upstream synchronisation pipeline. Bench- mark regression triggers re-translation of the failing module. Despite the framework difference, the UI achieves visual and behavioral parity. c) asyncio for Concurrency.: Rust’s Tokio runtime was mapped to Python’s asyncio, with anyio as an abstraction layer. The GIL eliminates the need for Arc<Mutex<T>> patterns. While this sacrifices CPU parallelism, the workload is overwhelmingly I/O-bound (LLM API calls, file operations), making the trade-off favorable. D. Continuous Upstream Synchronisation A key contribution of this work is an architecture for contin- uous translation—not a one-time migration. The upstream Rust codebase ships daily updates; our Python port must absorb them to remain useful. We achieve this through a four-stage pipeline: 1) Track. The upstream Rust repository is tracked as a git submodule. Periodic git pull fetches new commits. 2) Diff.Aconversionscript (scripts/convert-diff.py) extracts the changed Rust modules and maps them to their Python equivalents usingamodule-levelcorrespondencetable(e.g., codex-rs/exec → codex.exec). 3) Translate. An LLM translates the diff—not the entire crate, but only the changed portions—guided by the existing Python module as context. This incremental approach is more efficient and less error-prone than re-translating entire modules. 4) Validate. The translated changes are tested at three lev- els: (a) unit tests (pytest), (b) type checking (mypy --strict), and (c) benchmark regression (tb run on Terminal-Bench). If the benchmark score regresses, the translation is refined until parity is restored. This architecture treats the benchmark score as a loss function: when a translated change causes regression, the LLM re-examines its translation with the failing test/benchmark as additional context. In practice, most upstream changes translate cleanly on the first attempt; only API-level changes or new tool types require iterative refinement. E. Validation Strategy Migration correctness was validated through a four-tier testing strategy: 1) Unit tests: 2,621 test functions mirroring the Rust test suite. 2) Integration tests: End-to-end verification that all 28 mod- ules interact correctly. 3) Parity tests: Explicit tests verifying rendering output and protocol serialization match the Rust implementation. 4) Benchmark regression: Head-to-head Terminal-Bench evaluation ensuring the Python port’s task-solving accuracy remains within 5% of the Rust baseline. V. EVALUATION We evaluate the Rust-to-Python migration across four dimensions: code metrics (Section V-A), test parity (Sec- tion V-B), runtime performance (Section V-C), and real task benchmarks (Section V-D). All experiments were conducted on a MacBook Pro (Apple M4 Pro Max, 128GB RAM, ma- cOS 15.4) using Python 3.13. Head-to-head agent evaluations (Terminal-Bench, SWE-bench Verified) use GPT-5.4 via the OpenAI Responses API. A. Code Metrics a) Lines of Code.: Table IV presents the module-level LOC comparison. The Python implementation comprises 52,685 lines of code across 328 files, compared to 648,789 lines across 1,555 files in Rust—a 12.3× reduction. The agent module was recently refactored from a monolithic runner into focused submodules (auth, sandbox, config loader, query en- gine, permissions, streaming tool executor, tool result budget), increasing its file count but improving maintainability. The reduction ratio varies by module: infrastructure modules like state and config show moderate reduction due to inherent complexity, while the TUI subsystem achieves the largest reduction (largely due to Textual’s higher-level abstractions vs. ratatui’s low-level rendering model). Figure 3 visualizes the per-module LOC comparison on a logarithmic scale, highlighting the consistent reduction across all subsystems. b) Cyclomatic Complexity.: We measured cyclomatic complexity [4] for all 4,692 Python functions using the radon static analysis tool. Figure 4 shows the distribution across the top 12 modules by function count. The mean complexity is 2.70 (rank A), with 89% of functions achieving the mini- mal complexity rank (A). Only 23 functions (0.5%) exceed complexity rank C, concentrated in the agent runner (which handles complex state transitions) and the sandbox manager (which implements platform-specific branching logic). The recent decomposition of the runner module improved its per- function complexity while adding new modules with focused, low-complexity functions. c) Code Density.: Figure 5 plots the code density (LOC per file) against module size (number of files). Python modules cluster at higher density (mean 145 LOC/file) compared to Rust (mean 417 LOC/file), reflecting Python’s more concise expression of equivalent functionality. B. Test Parity We use the term test parity to mean functional and behavioral equivalence, not a one-to-one mapping of test 6© 2026 JP Morgan Chase & Co. TABLE IV: Lines of Code Comparison by Module ModulePy FilesPy LOCRs LOCRatio agent8212,829181,90314.2x analytics23199893.1x app server63,45254,24415.7x auth101,8965,1182.7x backendclient36829131.3x cli72,0984,8222.3x cloud56677,32111.0x code mode66012,1663.6x config91,4963,1832.1x core152,561181,90371.0x exec71,70111,4936.8x execpolicy79024,6035.1x features47751,1221.4x hooks181,6544,8422.9x instructions3831662.0x integrations33151,0133.2x mcp92,4587,6583.1x plugin32212931.3x protocol212,71825,4159.4x rollout76055,7179.4x sandbox93,09313,0364.2x sdk88911,1851.3x skills66684,7837.2x state121,33710,1187.6x telemetry64984,99210.0x tui445,53294,67517.1x utils162,63315,1165.7x Total32852,685648,78912.3x cases. The two implementations have fundamentally different testing needs: Rust requires extensive tests for memory safety, ownership, lifetimes, and borrow-checker edge cases that simply do not exist in Python, while Python’s higher-level abstractions let each test cover more behavioral surface area. Table V presents the test function counts by module. The Python implementation contains 2,902 test functions com- pared to 8,490 in Rust. The 3× difference reflects three factors: (1) Python’s higher-level abstractions eliminate entire categories of tests (memory safety, ownership, lifetime edge cases) that Rust must cover; (2) Rust’s convention of co- locating unit tests with source code inflates the count with trivial accessor and trait-implementation tests; and (3) the Python suite is written to verify behavioral contracts—“does this agent turn produce the correct tool call?”—rather than internal implementation invariants. The test-to-KLOC ratio provides a normalized comparison: the Python suite averages 62 tests per KLOC, indicating thorough behavioral coverage relative to codebase size. C. Runtime Performance Table VI presents the Python runtime performance metrics. We measure startup time (importing the CLI entry point), peak memory consumption, and import time. a) Startup Time.: The Python CLI startup (importing the main entry point) averages 53.9ms (σ=2.1ms, n=20), which is approximately 3–5× slower than a compiled Rust binary. 10 2 10 3 10 4 10 5 Lines of Code instructions plugin integrations analytics telemetry code_mode rollout cloud skills backend_client features sdk execpolicy state config hooks exec auth cli mcp core utils protocol sandbox app_server tui agent Lines of Code by Module Python Rust Fig. 3: Lines of code by module (log scale). Python consis- tently requires fewer lines across all 28 modules. However, this overhead is amortized over a typical agent session that runs for minutes to hours, during which LLM API calls dominate latency at 1–10 seconds per round trip. The startup overhead represents less than 1% of total session time. b) Memory Usage.: Peak resident memory for the Python process is 30.3 MB (σ=0.1MB, n=5). While higher than a compiled Rust binary (typically 10–15MB), this foot- print is modest for a desktop application running on systems with 8–64GB RAM. The overhead is attributable to the Python interpreter and loaded standard library modules. D. Real Task Benchmarks Beyond micro-benchmarks, we evaluate the Python im- plementation on representative real-world tasks that exercise multiple subsystems simultaneously. Table VII presents timing results for eight operational tasks. Figure 7 visualizes the latency distribution across all harness benchmarks on a logarithmic scale, with a reference line indicating typical LLM API latency. The harness benchmarks exercise the actual subsystem code paths used during agent operation, providing realistic performance data beyond synthetic micro-benchmarks. Key observations: 7© 2026 JP Morgan Chase & Co. 0204060 Cyclomatic Complexity sandbox agent app_server auth hooks mcp tui utils config core exec protocol Complexity Distribution (Top 12 Modules) Fig. 4: Cyclomatic complexity distribution by module. The majority of functions (90%) achieve rank A (complexity 1–5). 0100200300400 Files per Module 0 100 200 300 400 500 600 700 LOC per File Code Density: Python vs Rust Python Rust Fig. 5: Code density comparison. Python achieves higher information density per file. • Tool orchestration overhead is negligible (∼30μs): the approval pipeline, handler dispatch, and result packaging add virtually no latency to tool execution. • Shell execution dominates local latency: spawning a subprocess and capturing output takes 3–7ms, which is the primary source of local computation cost. Even so, this is 3 orders of magnitude faster than a typical LLM API call. • Patch parsing and policy matching are sub-microsecond: the data-structure operations at the core of code modification TABLE V: Test Parity: Python vs Rust ModulePy TestsRs TestsPy KLOCTests/KLOC agent661251915.343.1 analytics12100.336.7 app server1314483.735.4 auth77672.136.7 backendclient1280.717.2 cli142682.362.7 cloud28410.739.4 code mode35100.749.0 config74381.842.3 core20225193.164.1 exec751421.940.0 execpolicy121661.0121.6 features35220.842.5 hooks46581.726.6 instructions540.153.2 integrations17190.447.1 mcp165512.858.0 plugin1910.279.2 protocol1022832.836.2 rollout33370.745.7 sandbox1751283.648.7 sdk111141.0108.8 skills30780.838.9 state46841.727.2 telemetry28400.648.4 tui26314017.236.7 utils2123343.071.2 e2e450nannan Total29028490 TABLE VI: Runtime Performance Metrics (Python) MetricMeanStdN Startup Time (ms)53.92.120 Import Time (μs)57.00.01 Peak Memory (MB)30.30.15 Importable Modules328.00.01 and security enforcement are extremely fast in Python. • The full pipeline (orchestrator→ approval→ shell→ result capture) completes in 3.5ms, confirming that the Python implementation adds no perceptible overhead to the agent’s tool-use loop. These results demonstrate that in a typical agent session where each LLM round-trip takes 1–10 seconds, local Python computation accounts for less than 0.1% of total latency. E. Code Quality Analysis We analyze the API surface, type safety, and dependency structure of the Python codebase to assess software quality beyond LOC metrics. a) API Surface.: Table VIII compares the API surface of both implementations. The Python codebase defines 1,385 classes and 2,363 functions/methods, while Rust exposes 2,675 structs/enums and 20,525 functions/methods. Python achieves a higher API density: fewer definitions serving equivalent functionality, reflecting the expressiveness of higher-level ab- 8© 2026 JP Morgan Chase & Co. 05001000150020002500 Test Count instructions analytics backend_client integrations plugin cloud telemetry skills rollout code_mode features hooks state config exec auth protocol sdk execpolicy app_server cli mcp sandbox core utils tui agent Test Functions by Module Python Rust Fig. 6: Test function counts by module: Python vs Rust. stractions. Notably, 395 Python methods are explicitly async, directly mapping Rust’s async trait implementations. b) Type Coverage.: Running mypy --strict on the codebase reveals 248 type errors across 69 of 282 files, yielding a 75.5% strict type-clean rate. The majority of errors stem from third-party library type stubs (Textual, httpx) rather than application logic, indicating strong internal type discipline. Zero TODO, FIXME, or HACK comments were found in the codebase, suggesting a clean, production-ready state. c) Dependency Structure.: Figure 9 shows the cross- module import dependency heatmap. The architecture exhibits clear layering: foundation modules (core, config, utils) are widely depended upon, while peripheral modules (cli, tui) consume many dependencies but are not imported by others. The agent module serves as the primary integration point with both high fan-in and fan-out, consistent with its role as the system’s orchestration core. F. Migration Effort We analyze the git history to quantify the migration effort (Table IX). The repository contains 4,947 total commits, with 116 commits during the intensive migration period (March 25– 27, 2026). During this period, 109,427 lines were added TABLE VII: Harness Benchmark Results: Real Subsystem Operations TaskMean (ms)P50 (ms) Tool Orchestration Orchestrator (skip approval)0.0300.028 Orchestrator (with approval)0.0290.028 Tool Registry (10 tools)0.0330.030 Shell Execution Shell Handler (echo)3.3193.289 Shell Handler (ls | head)6.6726.619 Full Pipeline (orch→shell)3.5123.389 Code Operations Patch Parsing (add file)0.0020.002 Patch Parsing (update hunks)0.0030.003 ExecPolicy Matching (5 rules)0.0010.001 State & Memory Token Estimation (2K words)<0.001 <0.001 Should Compact Decision0.5000.501 SQLite State (session + 20 msgs)0.0740.068 Config TOML Merge0.0010.001 Feature Flags Lookup0.0010.001 10 2 10 0 10 2 Latency (ms) ExecPolicy Rule Matching (5 patterns) Config TOML Merge Feature Flags Lookup (all) Patch Parsing (add file) Patch Parsing (update hunks) Tool Orchestrator (skip approval) Tool Registry (10 tools, dispatch) Tool Orchestrator (with approval) SQLite State (session + 20 msgs) Should Compact Decision Full Pipeline (orchshellresult) Shell Handler (echo) Shell Handler (ls | head) LLM API (~1-10s) Harness Benchmark Latencies Fig. 7: Harness benchmark latencies (log scale). The red dashed line marks typical LLM API round-trip time (∼1–10s). All local operations complete 2–6 orders of magnitude faster. and 906,656 lines were deleted, yielding a net reduction of 797,229 lines—reflecting the consolidation from the verbose Rust codebase to the more concise Python implementation. The test suite evolved progressively during migration: com- mit messages document milestones at 1,881, 2,343, and finally 2,621 passing tests, demonstrating a test-driven approach where each module’s conversion was validated incrementally. The 45 test-related commits during the migration period 9© 2026 JP Morgan Chase & Co. TABLE VIII: API Surface Comparison MetricPythonRustRatio Source Files3291,4484.4x Classes / Structs+Enums1,5902,6751.7x Top-level Functions7819,32311.9x Methods1,82811,2026.1x Async Methods4590— Properties1400— Doc Comments3,87116,7394.3x TODO/FIXME0126— Classes/ Structs FunctionsMethodsAsync Methods 10 3 10 4 Count API Surface Comparison Python Rust Fig. 8: API surface comparison (log scale). Python provides equivalent functionality with fewer, higher-level abstractions. underscore the emphasis on correctness validation throughout the process. G. End-to-End Agent Evaluation To validate that the Python implementation can execute realistic agent workflows, we evaluate 8 representative tasks that exercise the full tool execution pipeline—from task spec- ification through tool dispatch, shell execution, and result capture. Each task is run 5 times with results averaged. All 8 tasks achieve a 100% success rate (40/40 runs), confirming that the Python tool execution pipeline is function- ally correct across diverse task types including file creation, shell command execution, multi-step pipelines, and patch application. Task latencies range from 0.07ms (patch-only operations) to 24.7ms (complex multi-tool pipelines involving patch ap- plication and shell execution). The dominant cost factor is subprocess spawning for shell commands (∼3–7ms per com- mand), consistent with the harness benchmark findings. Multi- tool tasks scale linearly with the number of shell invocations, confirming the absence of systemic overhead in the orchestra- tion layer. H. LLM Pipeline Benchmarks To evaluate the LLM-adjacent operations that are unique to AI agent systems, we benchmark 26 operations across the agent app_server backend_client cli cloud core exec execpolicy features hooks mcp protocol rollout sandbox state tui utils agent app_server backend_client cli cloud core exec execpolicy features hooks mcp protocol rollout sandbox state tui utils 00000000010101001 00000000000000000 00000000000000000 21000001101001010 00100000000000000 00000000000000000 00000100000000000 00000000000000000 00000000000000000 00000000000000000 00000000000000000 00000000000000000 00000000000000100 00000000000000000 00000000000000000 20010000000000001 00000000000000000 Cross-Module Import Dependencies 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Fig. 9: Cross-module import dependency heatmap. Darker cells indicate stronger coupling. TABLE IX: Migration Effort Summary MetricValue Total Commits4957 Migration Period Commits146 Lines Added (migration)137,331 Lines Deleted (migration)912,510 Net Line Change-775,179 Test-related Commits55 Rust Crate Count6 Python Module Count27 Avg Commit Size (lines)1314 full LLM pipeline (Figure 11). These benchmarks use mocked API responses—pre-recorded JSON payloads that simulate the OpenAI Responses API without making real network calls. This isolates the Python orchestration overhead (prompt construction, response parsing, context management) from variable network latency, enabling reproducible microsecond- precision measurements of the local computation that sur- rounds each LLM call. a) Token Estimation and Context Management.: To- ken estimation scales linearly with text length but re- mains sub-microsecond even for 1M-character inputs. The should_compact decision takes 0.49ms for 100K-token conversations, enabling real-time compaction triggering with- out perceptible delay. b) Conversation Compaction.: The full compaction pipeline (history formatting, prompt construction, mock LLM call, response parsing, history reconstruction) completes in 0.67ms with mocked API, confirming that the Python orches- tration overhead is negligible—the real bottleneck is the LLM API latency. 10© 2026 JP Morgan Chase & Co. TABLE X: End-to-End Agent Task Evaluation TaskToolsSuccessMean (ms)P50 (ms) Create Python File1100 Shell Echo1100 Multi-step File Creation2100 Directory Listing1100 Git Status1100 Complex Pipeline3100 File Update with Patch2100 Token Estimation Accuracy1100 05001000 Mean Latency (ms) Create Python File Shell Echo Multi-step File Creation Directory Listing Git Status Complex Pipeline Token Estimation Accuracy File Update with Patch 1 tools 1 tools 2 tools 1 tools 1 tools 3 tools 1 tools 2 tools LLM API (~1s) Agent Eval: Task Latencies Fig. 10: Agent evaluation task latencies. All tasks complete in under 25ms. Green bars indicate 100% success rate. c) Guardian Risk Assessment.: Fast-path pattern match- ing for 10 safe commands (e.g., ls, cat, git status) takes 0.035ms and 5 dangerous commands (e.g., rm -rf /, curl | bash, chmod 777, d if=/dev/zero, sudo rm) takes 0.005ms, enabling sub-millisecond approval deci- sions for common operations without LLM invocation. The full guardian review with mocked LLM completes in 0.36ms. d) SSE Streaming and Protocol.: Parsing 50 Server-Sent Events (simulating a streaming LLM response) takes 0.030ms. Constructing a 30-turn conversation with tool calls and serial- izing to JSON takes 0.093ms. These results confirm that the Python protocol layer introduces no meaningful overhead to the streaming pipeline. I. Head-to-Head Agent Benchmarking The most demanding test of translation correctness is func- tional parity on real tasks: does the Python port solve the same problems as the Rust original? We evaluate both im- plementations on Terminal-Bench [2], an 80-task benchmark of complex terminal operations including kernel compilation, cryptographic hash cracking, ML model training, maze solv- ing, and repository manipulation. 10 2 10 0 10 2 Latency (ms) Context Mgr Tokens (Long prompt) should_auto_compact (Under threshold) Context Budget Calc (7 models) Context Mgr Tokens (Unicode text) Tool Call Parsing (5 calls) Format History (20 turns) Guardian Fast-Path (5 dangerous cmds) should_compact (Short (1K tokens)) Event Construction (22 events + serialize) SSE Parsing (50 deltas + metadata) Guardian Fast-Path (10 safe cmds) Conversation Build (30 turns + tools) should_compact (Medium (20K tokens)) should_compact (Many messages (50)) Guardian Review (mock LLM) Memory Phase1 Extract (mock LLM) should_compact (Long (100K tokens)) Compact Task (mock LLM, 20 turns) LLM API (~1-10s) LLM Pipeline Benchmark Latencies Fig. 11: LLM pipeline operation latencies (log scale). All local operations complete orders of magnitude faster than LLM API calls. TABLE XI: Terminal-Bench head-to-head results (80 tasks, GPT-5.4) AgentResolvedAccuracyUnique Passes Original Codex CLI (Rust)38 / 8047.5%11 codex-python (ours)34 / 8042.5%9 Both solved25— a) Experimental setup.: Both agents are installed na- tively inside Terminal-Bench Docker containers and run exec with identical flags (--sandbox danger-full-access --model gpt-5.4). The original Rust CLI (v0.117.0) in- stalls via npm; the Python port installs from a pre-built wheel into an isolated virtualenv. Both use the same OpenAI API key and GPT-5.4 model. b) Results.: Table XI presents the head-to-head results. The Python port resolves 34 of 80 tasks (42.5%), compared to the original Rust CLI’s 38/80 (47.5%). This figure reflects the complete post-fix rerun after applying the API 400 error- recovery fix (Section V-J). Of the 80 tasks, 25 are solved by both implementations, 9 are solved only by the Python port, and 13 are solved only by the Rust CLI. c) Analysisofuniquepasses.:The9 taskssolvedonlybythePythonportinclude fibonacci-server, jupyter-notebook-server, 11© 2026 JP Morgan Chase & Co. TABLE XII: SWE-bench Verified head-to-head results (80 tasks, GPT-5.4, 4 workers) AgentTasksPatchesResolvedRate codex-python (ours) astropy22221254.5% django58584781.0% Total80805973.8% Original Codex CLI (Rust) Total80805670.0% and pytorch-model-cli (2 variants)—tasks involving Python-ecosystemtoolingwherethePythonagent’s environment familiarity provides an advantage. The 13 tasks solved only by the Rust CLI include chess-best-move (requiring extended multi-turn reasoning), path-tracing, and sqlite-db-truncate. Notably, both sets of unique passes involve the same LLM and tools—the differences arise from non-deterministic LLM behavior and subtle environmental factors, not from architectural limitations of either implementation. d) Iterative refinement.: The current 42.5% accuracy was reached through six iterations of benchmark-driven debugging (see Table I in Section IV), with the final fix being API 400 error recovery (Section V-J). Each iteration identified a specific category of failure—API protocol mismatch, missing conversation history, Python environment pollution, absent CLI tools, invalid content-item type—and a targeted fix. This demonstrates the effectiveness of benchmarks as an objective function for translation quality. The adapter runs each task by cloning the target repository, checking out the base commit, and executing cdx exec with the issue description as the prompt. Patches are captured via git diff. With 4 parallel workers and a 1800-second per- task timeout, the 80-task Verified run completes in approxi- mately 2 hours. Initial runs revealed two bugs that suppressed resolve rates. First, ws_transport.py silently returned empty responses when the WebSocket API exhausted its quota, which the agent misinterpreted as successful no-op completions; fix- ing the fallback to detect empty responses and retry via HTTP SSE brought the patch production rate to 100% for both benchmarks. Second, the memory-extraction model was mis-specified as a non-existent model identifier; updating to gpt-5.4-nano fixed a 404 Not Found error that oc- curred after every completed rollout. Both bugs are structural vulnerabilities shared by the Rust and Python implementa- tions; the Rust baseline was re-run with a clean API key to confirm its 70.0% figure unaffected by the WebSocket issue. On SWE-bench Verified (80 astropy and django tasks), the Python agent resolves 59/80 tasks (73.8%) versus the Rust original’s 56/80 (70.0%), with particularly strong performance on django (81.0%). The Python port’s 3.8 percentage-point advantage is within the margin of LLM non-determinism. The adapter and all prediction artifacts are included in our reproducibility package. J. Bugs Discovered Through Benchmarking The benchmarking process revealed four bugs that would have been invisible to unit tests: • WebSockettransportrobustnessimprovement (ws_transport.py):WhentheAPIquotawas exhausted,WebSocketconnectionsreturnedempty responses silently. The agent treated these as successful no-ops and marked tasks complete without doing any work. Beyond fixing the silent failure, the HTTP SSE fallback path delivers robustness improvements not present in the Rust transport: per-request 429 detection with Retry-After headerparsing,exponentialbackoff,andautomatic fallback-API-key rotation on quota exhaustion. This structural vulnerability is present in both implementations; only the Python run was initially affected because the Rust baseline used a fresh API key. The fix constitutes a net robustness improvement of the Python transport over its Rust counterpart. • Memory extraction model error (phase1.py): The model identifier for post-rollout memory extraction did not resolve to a valid endpoint, causing a 404 Not Found error after every completed agent turn. Fixed by updating the model name to gpt-5.4-nano. • _cost_tracker NameError (runner.py): A cost- tracking variable was referenced before assignment in the Docker-run code path, crashing a subset of tasks. Fixed by initialising the variable at the top of the enclosing scope. • Comprehensive API 400 error recovery (runner.py): Under certain prompts, the model generated a response containing an unsupported content-item type. The Re- sponses API returned HTTP 400 (invalid_value on input[N]), the agent had no recovery path, and the trial died immediately before writing any output. Rather than patching only this failure mode, we implemented a systematic 400 recovery layer covering four distinct er- ror scenarios: (1) unsupported previous_response_id parameter—stripped and retried; (2) invalid_value on a specific input item—offending item removed by index and retried; (3) local_shell tool type not supported by the endpoint—gracefully degraded; and (4) context-window overflow—oldest input items trimmed and retried. This recovery system is Python-specific; the Rust implementa- tion has no equivalent error-recovery layer for these 400 scenarios. K. Discussion Our evaluation reveals four key findings. First, the Rust- to-Python migration achieves near-parity across both bench- marks: the Python port leads on SWE-bench Verified (73.8% vs. 70.0%) while trailing on Terminal-Bench (42.5% vs. 47.5%). The Terminal-Bench gap is partly attributable to the now-fixed API crash, LLM non-determinism, and a safety- refusal failure mode not present in the Rust baseline. Second, the Python agent resolves 59/80 SWE-bench Verified tasks 12© 2026 JP Morgan Chase & Co. (73.8%) with GPT-5.4, demonstrating that the translation sup- ports realistic software engineering workloads at scale. Third, benchmarks are more effective than unit tests at detecting translation bugs: • 2,621 unit tests passed from the first complete transla- tion, yet Terminal-Bench accuracy was initially 0% (wrong adapter architecture). • API protocol bug (sending local_shell instead of function tools) was invisible to tests but caused 100% Chat Completions fallback. • Environment pollution from pip-installing into the system Python was only detectable through end-to-end task execu- tion in Docker containers. • WebSocket empty-response and API 400 bugs (Sec- tion V-J) were each invisible to all unit tests and surfaced only through live benchmark runs. These findings support our thesis that for complex system translations, public benchmarks should be treated as first-class objective functions, not merely downstream validation. Fourth, the Python port is now a capability superset of the Rust original, not merely a parity replica. The codex.enhancements module (Section I-H) ships 30 feature-flagged extensions—multi-agent orchestration, seman- tic memory, persistent plans, cost tracking, IDE bridge, guardian safety assessment, voice mode, and more—none of which exist in the Rust codebase. Two bug fixes uncovered during benchmarking also delivered net improvements over Rust: the WebSocket fallback now brings 429 detection, Retry-After backoff, and API-key rotation that the Rust transport lacks; and the API 400 recovery layer handles four distinct error scenarios with no Rust equivalent. The feature- flag architecture is critical here: by setting all enhancement flags to off, one obtains a strict parity build suitable for head-to-head comparison; enabling flags progressively unlocks the extended platform. This demonstrates that LLM-assisted translation need not stop at functional equivalence—the same methodology that achieves parity can continue to evolve the port into a first-class, independently-capable system. The broader engineering picture is equally clear: for AI agents where the computational bottleneck is the LLM API (1– 10 seconds per round-trip), Python’s local overhead (<25ms per tool execution) accounts for less than 0.1% of total session latency, while delivering 15.9× code reduction and 90% rank- A cyclomatic complexity. VI. RELATED WORK a) AI Coding Agents.: The landscape of AI coding agents has expanded rapidly. SWE-agent [5] introduced an agent-computer interface for automated software engi- neering, achieving strong performance on SWE-bench [6]. OpenDevin [7] provides an open platform for generalist AI developer agents. ChatDev [8] explores multi-agent col- laboration for software development through communica- tive agents. These systems share architectural patterns with CODEX CLI—tool-use loops, sandboxed execution, and itera- tive refinement—but differ in their deployment model (cloud- hosted vs. local), security architecture, and protocol support. b) Tool-Augmented LLMs.: The ReAct framework [9] established the reasoning-action paradigm that underlies mod- ern coding agents. Toolformer [10] demonstrated that lan- guage models can learn to use external tools autonomously. The Model Context Protocol (MCP) [3] standardizes tool integration for LLM applications, enabling interoperable tool ecosystems. Our work contributes an architectural perspective on how these capabilities are composed into a production system. c) Language Migration Studies.: Prior work on cross- language code translation has focused on LLM-based tran- spilation [11] and neural machine translation of code [12]. These studies primarily evaluate translation correctness at the function level. Our contribution differs in scope—we analyze a complete system-level migration comprising 28 subsystems and 648K lines of Rust—and in methodology, providing a multi-dimensional quantitative comparison rather than a correctness-focused evaluation. d) Software Complexity Metrics.: McCabe’s cyclomatic complexity [4] and Halstead’s software science metrics [13] provide foundational frameworks for quantifying code com- plexity. We apply these metrics in a novel context: evaluating the complexity characteristics of an AI coding agent architec- ture and comparing them across implementation languages. e) Code Generation and Analysis.: Codex [1] and sub- sequent models [14] demonstrated that LLMs trained on code can generate functionally correct programs. Our work complements this line of research by examining the systems that deploy these models in production, analyzing the ar- chitectural decisions that determine reliability, security, and maintainability. VII. CONCLUSION We presented a methodology for LLM-assisted continuous code translation, demonstrated on CODEX CLI—a produc- tion AI coding agent migrated from Rust (648K LOC) to Python (41K LOC). Our central finding is that public agent benchmarks serve as effective objective functions for cross- language translation: the Python port achieves 42.5% accuracy on Terminal-Bench, within 5% of the original Rust imple- mentation’s 47.5%, with benchmark-driven debugging proving more effective than unit tests alone at exposing integration- level bugs. Crucially, the port has grown beyond parity into a capability superset: the codex.enhancements module adds 30 feature-flagged extensions absent from Rust, while a layered flag-resolution system preserves a strict parity build for reproducible head-to-head comparison. Four principles emerge from this work: 1) Benchmarks over tests. While 2,621 unit tests passed from the initial translation, Terminal-Bench accuracy was 0%. Six iterations of benchmark-driven debugging—fixing API protocol mismatches, conversation history bugs, en- vironment pollution, missing tools, and an API 400 error 13© 2026 JP Morgan Chase & Co. crash—closed the gap to 5%. Public benchmarks detect the integration failures that unit tests cannot. 2) Continuous translation, not one-shot migration. Our up- stream synchronisation architecture (track → diff → trans- late → validate) treats translation as an ongoing process. The LLM translates only changed code; the benchmark score serves as a regression gate. This makes it practical to maintain a living Python port of a fast-moving Rust codebase. 3) Language choice follows the bottleneck. For AI agents where LLM API latency (1–10s) dominates, Python’s sub- millisecond local overhead is negligible. The 15.9× code reduction, 90% rank-A complexity, and broader contributor base are decisive advantages that the benchmark results validate in practice. 4) Translation enables divergence. Once benchmark parity is achieved, the translated port can evolve independently. Python’s rapid-prototyping culture allowed 30 additive extensions to be developed and shipped behind feature flags in the time it would take to prototype them in Rust. Bug fixes uncovered during benchmarking delivered net improvements over the original: the WebSocket transport now has 429 detection and API-key rotation that Rust lacks; the API 400 recovery layer handles four error scenarios with no Rust equivalent. Parity is a starting point, not a ceiling. a) Future Work.: Several directions merit investigation: (1) applying the benchmark-as-objective-function methodol- ogy to other large-scale translations (e.g., Java → Kotlin, C++ → Rust); (2) automated upstream sync where the diff- translate-test loop runs as a CI pipeline with no human intervention; (3) multi-benchmark objective functions that combine Terminal-Bench, SWE-bench, and domain-specific benchmarks for richer translation validation; and (4) studying how the benchmark-driven approach compares to formal ver- ification methods for establishing cross-language equivalence. REFERENCES [1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” in arXiv preprint arXiv:2107.03374, 2021. [2] Laude Institute, “Terminal-bench: Benchmarking LLM agents on real- world terminal tasks,” arXiv preprint arXiv:2601.11868, 2025, https: //w.tbench.ai/. [3] Anthropic, “Model context protocol: A standard for tool-augmented LLM systems,” 2025, https://modelcontextprotocol.io. [4] T. J. McCabe, “A complexity measure,” IEEE Transactions on Software Engineering, vol. SE-2, no. 4, p. 308–320, 1976. [5] J. Yang, C. E. Jimenez, A. Wettig, K. Liber, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” arXiv preprint arXiv:2405.15793, 2024. [6] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” in International Conference on Learning Representa- tions (ICLR), 2024. [7] X. Wang et al., “OpenDevin: An open platform for AI software developers as generalist agents,” arXiv preprint arXiv:2407.16741, 2024. [8] C. Qian et al., “ChatDev: Communicative agents for software develop- ment,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. [9] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023. [10] T. Schick, J. Dwivedi-Yu, R. Dess ` ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” Advances in Neural Infor- mation Processing Systems, vol. 36, 2024. [11] R. Pan, A. R. Ibrahimzada, R. Krishna, D. J. Murali, J. Pavez et al., “Lost in translation: A study of bugs introduced by large language models while translating code,” Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), 2024. [12] M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” ACM Transactions on Software Engineering and Methodology, vol. 28, no. 4, p. 1–29, 2019. [13] M. H. Halstead, “Elements of software science.” Elsevier, 1977. [14] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic eval- uation of large language models of code,” in International Symposium on Machine Programming (MAPS), 2022. APPENDIX TableXIIIlistsall30featureflagsinthe codex.enhancements module, grouped by sub-package. Each flag can be toggled at runtime via --enable FLAG / --disable FLAG, via the CODEX_ENABLE_FLAG environment variable, or through the layered configuration system. Setting all flags to off produces a strict-parity build identical in behavior to the Rust original. 14© 2026 JP Morgan Chase & Co. TABLE XIII: Complete list of Python enhancement flags Sub-packageFlagDefaultDescription agents/ MULTI_AGENTOnHierarchical child-agent spawning with inherited context MULTI_AGENT_V2OnImproved coordination protocol with result aggregation MULTI_AGENT_ORCHESTRATIONOnParallel agent dispatch with safety bounds (max depth 5, max 100 agents) guardian GUARDIANOnAutomated LLM-based risk assessment before tool execution bridge/ IDE_BRIDGEOffWebSocket bridge for IDE integration (VS Code, JetBrains) LSP_INTEGRATIONOffLanguage Server Protocol support for code intelligence compaction/ MULTI_STRATEGY_COMPACTIONOnThree-phase context compaction (micro, snip, full) FORKED_COMPACTIONOffBranch-and-merge compaction for multi-agent sessions memory/ MEMORY_SYSTEMOnPost-turn memory extraction and retrieval TYPED_MEMORYOnStructured memory with typed schemas (user, project, feedback) SEMANTIC_MEMORYOffEmbedding-based similarity search over memory store AUTO_MEMORYOnAutomatic memory extraction without explicit user request state/ PERSISTENT_PLANSOnGoal and plan persistence across turns SYSTEM_REMINDERSOnPeriodic system-message injection for long sessions DENIAL_REPLAYOnRe-attempt denied actions with adjusted parameters tools/ FILE_TOOLSOnEnhanced file read/write/edit beyond base shell FILE_EDITOnStructured file editing with conflict detection WEB_FETCHOffHTTP fetching and web content extraction NOTEBOOK_EDITOffJupyter notebook cell manipulation WORKTREE_TOOLSOffGit worktree management for parallel development productivity/ SKILLSOnLoadable skill definitions for domain-specific workflows CRON_TOOLOffScheduled task creation and management VOICE_MODEOffVoice input/output for conversational interaction resilience/ COST_TRACKINGOnPer-session and per-turn API cost accounting APP_STATEOnApplication-level state management and checkpointing STARTUP_PREFETCHOnParallel prefetch of config, auth tokens, and model metadata 15© 2026 JP Morgan Chase & Co.

Instant research discovery