Paper deep dive
Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation
Jean-Baptiste Espinasse, Djamel Eddine Khelladi, Mathieu Acher
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/24/2026, 4:09:03 AM
Summary
The paper introduces Spec2COBOLRot, an agentic AI pipeline designed to generate realistic COBOL corpora for benchmarking modernization tools. The system combines specification-driven generation with iterative degradation, injecting patterns from real production code to match structural complexity metrics. Evaluation on three programs shows the pipeline produces syntactically valid code with realistic structural complexity, though preserving business behavior during degradation remains a challenge.
Entities (10)
Relation Signals (9)
Spec2COBOLRot → targets → COBOL
confidence 100% · We propose a systematic agentic AI pipeline for generating realistic COBOL programs...
Spec2COBOLRot → developedby → Djamel Eddine Khelladi
confidence 95% · Authors listed: Jean-Baptiste Espinasse, Djamel Eddine Khelladi, and Mathieu Acher.
Spec2COBOLRot → developedby → Mathieu Acher
confidence 95% · Authors listed: Jean-Baptiste Espinasse, Djamel Eddine Khelladi, and Mathieu Acher.
Spec2COBOLRot → developedby → Jean-Baptiste Espinasse
confidence 95% · Authors listed: Jean-Baptiste Espinasse, Djamel Eddine Khelladi, and Mathieu Acher.
Spec2COBOLRot → measures → SLOC
confidence 95% · For the scale dimension, we used Source Lines of Code (SLOC)...
Spec2COBOLRot → measures → Cyclomatic Complexity
confidence 95% · For the code complexity dimension, we used cyclomatic complexity (CC)...
Sopra Steria → providedcorpusfor → Spec2COBOLRot
confidence 90% · We evaluate whether degradation reaches target complexity levels... provided by our industrial partner Sopra Steria
Spec2COBOLRot → publishedin → AISM 2026
confidence 90% · In Proceedings of the 2nd International Workshop on AI for Software Modernization (AISM ’26)...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:COBOL remains widely deployed, yet representative corpora reflecting real production code are rarely available, limiting rigorous benchmarking of modernization approaches. We propose a systematic agentic AI pipeline for generating realistic COBOL programs, combining specification-driven generation with iterative degradation guided by patterns and complexity targets extracted from real production code. Here, realism is understood as structural fidelity to production code as captured by our metrics. We evaluate whether degradation reaches target complexity levels while preserving business behavior, and examine the limits of the approach, across three programs from distinct business domains. Results show the pipeline reliably produces syntactically valid programs and moves them toward realistic structural complexity. However, preserving business behavior is not always achieved by construction, and targeting structural metrics independently of business logic risks producing programs whose complexity does not reflect a plausible maintenance history. We discuss these limitations and outline a more realistic alternative as a direction for future work, generating legacy programs from scratch along a simulated development history.
Tags
Links
- Source: https://arxiv.org/abs/2609.26835v1
- Canonical: https://arxiv.org/abs/2609.26835v1
Trouble viewing inline? Open PDF directly →
Full Text
46,176 characters extracted from source content.
Expand or collapse full text
Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation Jean-Baptiste Espinasse Sopra Steria Nantes, France Inria Rennes, France jb.espinasse@soprasteria.com Djamel Eddine Khelladi CNRS, Univ. of Rennes, IRISA, Inria Rennes, France djamel-eddine.khelladi@irisa.fr Mathieu Acher INSA, Univ. of Rennes, IRISA, Inria Rennes, France mathieu.acher@irisa.fr Abstract COBOL remains widely deployed, yet representative corpora re- flecting real production code are rarely available, limiting rigorous benchmarking of modernization approaches. We propose a system- atic agentic AI pipeline for generating realistic COBOL programs, combining specification-driven generation with iterative degrada- tion guided by patterns and complexity targets extracted from real production code. Here, realism is understood as structural fidelity to production code as captured by our metrics. We evaluate whether degradation reaches target complexity levels while preserving busi- ness behavior, and examine the limits of the approach, across three programs from distinct business domains. Results show the pipeline reliably produces syntactically valid programs and moves them to- ward realistic structural complexity. However, preserving business behavior is not always achieved by construction, and targeting structural metrics independently of business logic risks producing programs whose complexity does not reflect a plausible mainte- nance history. We discuss these limitations and outline a more real- istic alternative as a direction for future work, generating legacy programs from scratch along a simulated development history. CCS Concepts • Software and its engineering→Software maintenance tools. Keywords Automated Software Engineering, COBOL Generation, Large Lan- guages Models ACM Reference Format: Jean-Baptiste Espinasse, Djamel Eddine Khelladi, and Mathieu Acher. 2026. Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation. In Proceedings of the 2nd International Workshop on AI for Software Modernization (AISM ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3843775.3844543 1 Introduction Legacy systems are running critical business activity across banking, insurance, public administration, etc, where reliability is important. COBOL remains one of the most widely deployed programming languages in production systems, with billions of lines still actively AISM ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 2nd International Workshop on AI for Software Modernization (AISM ’26), October 12–16, 2026, Munich, Germany, https://doi.org/10.1145/3843775.3844543. executed today [5]. Originally designed for business applications, COBOL was adopted for its readability relative to today languages, making it accessible to a broad range of practitioners. However, the characteristics that once made COBOL well-suited for its context have become sources of difficulty. Compared to mod- ern languages, COBOL systems are increasingly hard to maintain, the generation of developers who originally built these systems is retiring, leaving entry-level developers to face significant chal- lenges in code comprehension and defect location [8,9,15]. Beyond the workforce transition, the software engineering paradigms and conventions applied during the development of these systems differ substantially from modern practices. Code that was considered well-structured at the time has accumulated decades of patches, feature additions, and emergency fixes, often without correspond- ing updates to documentation. This progressive drift between code and its specifications has produced large and opaque codebases. Structured programming standards such as Jackson Structured Pro- gramming [1] were introduced to address some of these issues, but their adoption was uneven, and the accumulation of technical debt has continued. Furthermore, these legacy technologies are largely absent from contemporary academic curricula, further widening the skills gap. Recent work has explored generative AI and agentic AI for COBOL modernization tasks such as retro-documentation, business rules extraction and code transformation [4,9,11,16], showing promise over rule-based approaches that operate at the syntactic level and therefore fail to capture business intent[4]. How- ever, they come with significant limitations. LLM outputs are non- deterministic, opaque, difficult to control, and the resulting trans- lations cannot be trusted without extensive validation [2,10,14]. Given that these legacy systems have sustained mission-critical operations for decades, any modernization effort introduces sub- stantial risk. This concern is widely shared among practitioners and organizations responsible for such systems [13]. As GenAI-based modernization approaches show promise but remain difficult to val- idate and control, the need for representative evaluation resources becomes paramount. Without corpus that reflect the structural and semantic characteristics of real production COBOL, it is not possi- ble to rigorously benchmark these approaches, characterize their failure modes, or establish the conditions under which they can be trusted. Representative COBOL corpus reflecting production main- frame programs are rarely available in open access. Existing public collections are highly heterogeneous, ranging from algorithmic ex- ercises to domain-specific scripts, and do not exhibit the structural patterns characteristic of enterprise legacy systems, such as copy- books, sequential and indexed file handling, or business-oriented arXiv:2609.26835v1 [cs.SE] 21 Sep 2026 AISM ’26, October 12–16, 2026, Munich, GermanyJean-Baptiste Espinasse, Djamel Eddine Khelladi, and Mathieu Acher control flows. The proprietary nature of enterprise COBOL code- bases severely limits the availability of realistic public corpus [15], and existing open-source collections rely on filtering heuristics such as GitHub stars that do not reflect production code quality [3, 9]. To address these challenges and the lack of strong benchmarks, we propose a systematic agentic AI approach to the generation of re- alistic COBOL programs, combining agentic loop. The pipeline first generates a syntactically valid and functionally correct program from a natural language specification, then iteratively degrades it toward a calibrated target complexity, using an asset library of patterns extracted from real production code and complexity levels derived by correlation-based regression on a reference corpus. Here, realism is understood as structural fidelity to production code, as captured by our metrics. We evaluate whether iterative degradation reaches the target complexity intervals derived from real production programs pro- vided by our industrial partner Sopra Steria, while business behavior is preserved in the process, and assess the practical limits of the approach. We also validate through expert review. Across three business domains, our evaluation shows that the pipeline reliably produces syntactically valid programs and moves them toward realistic structural complexity, while also surfacing practical limita- tions, in particular around functional preservation and scalability, which we discuss in detail. Our replication package is available as an artifact at https:// github.com/jbespi/spec2cobolrot-samples.git. 2 Related Work Approaches to building COBOL datasets are centered on program translation, program augmentation, and full synthetic generation. Mining public repositories is a fourth, complementary source, but suffers from representativeness issues due to heterogeneous quality and GitHub-star-based filtering heuristics [3, 15]. Program translation approaches construct COBOL corpora by translating programs from a higher-resource source language. COBOL- Coder [9] translates Java programs into COBOL using an LLM, then validates the result through compiler-based checks and back- translation similarity scoring. Cassano et al. [6] propose MultiPL-T, which synthesizes and filters tests on high-resource language code before translating it into low-resource languages, using test cover- age as a validation signal. These methods depend on the availability of a source-language corpus and inherit its structural style. The resulting COBOL has no inherent reason to exhibit the conventions of legacy production code, since it is derived from the syntax and idioms of the source language rather than from COBOL itself. Program augmentation approaches start from an existing, often small, code corpus and transform it to increase diversity or realism. Saito et al. [21] propose grammar- and coverage-based augmen- tation, identifying underrepresented grammar paths in a training corpus and prompting an LLM to generate code that fills these gaps. Their approach targets syntactic diversity but not legacy-specific structural realism. LintSeq [19] refactors real code into sequences of linter-verified edits to increase solution diversity while preserving functionality. Closest in spirit to our degradation phase, Leinonen et al. [17] prompt an LLM to imitate typical student mistakes and show that the resulting error distribution is statistically close to that of real student submissions. We pursue an analogous idea, injecting plausible legacy patterns rather than plausible bugs, and targeting COBOL production realism rather than student code. Full synthetic generation approaches generate programs from scratch, typically via instruction-based prompting, without trans- forming an existing corpus. Code Alpaca [7] generates 20K syn- thetic instruction-solution pairs via Self-Instruct. WizardCoder [18] extends this with Evol-Instruct to iteratively increase instruction complexity. Our approach combines elements of the latter two cat- egories. Like full synthetic generation, Phase 1 produces COBOL directly from a natural language specification, without a source corpus to translate. Like program augmentation, Phase 2 transforms the generated program using patterns extracted from real produc- tion code. Unlike prior augmentation work, this transformation targets a calibrated, corpus-derived complexity profile rather than syntactic coverage or edit diversity alone. 3 Corpus Analysis 3.1 Corpus Description Our industrial partner provided us with a corpus of COBOL pro- grams, which we used to evaluate our work. This corpus of pro- grams consist of 14 batch COBOL payroll programs from a real legacy application. It covers several functions of a complete process- ing pipeline, from the extraction and sorting of business files to re- quest generation, dictionary-based enrichment, result merging and payslip production. From a technical perspective, the programs are characterized by a file-oriented design, extensive use of copybooks and large global data structures. The source code is incrementally developed and maintained over years. It’s a representative sample in terms of size, complexity of legacy COBOL programs. 3.2 Structural Complexity Metrics Table 1: Structural complexity metrics. MetricSymbolRationale Program-level metrics Source Lines of CodeSLOCProgramsizeexcludingcom- ments/blanks Number of paragraphsNPARStructural decomposition granularity Data/Procedure Div. LOC ratioDIVRBalance between data and logic Cyclomatic ComplexityCCControl-flow complexity Average paragraph lengthAPLCode organization, modularity Legacy pattern metrics Number of GOTO statementsNGOLack of structured control flow Variable Usage Rate (%)VUROver-declaration Dead Paragraph Rate (%)DPRStructural dead code To assess the structural complexity of the COBOL programs, we selected a set of metrics capturing scale, code complexity, main- tenance drift, and modularization. These combine classic metrics commonly used to characterize program-level complexity with met- rics specifically designed to capture traits typical of legacy code, such as maintenance drift and control-flow degradation. Table 1 summarizes them. For the scale dimension, we used Source Lines of Code (SLOC), number of paragraphs (NPAR), and the Data/Procedure Division Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus GenerationAISM ’26, October 12–16, 2026, Munich, Germany LOC ratio (DIVR). SLOC is a widely used indicator of program size, and of the effort already invested and still required to maintain it; it is important to keep this size realistic and not too small. NPAR captures the structural decomposition of the code, i.e., the number of minimal logical units in the program. DIVR assesses the balance between data and logic: in legacy COBOL, the data division is often large due to extensive use of global data structures and copybooks. For the code complexity dimension, we used cyclomatic complex- ity (C) and the number of GOTO statements (NGO). C measures the number of independent logic paths through the code and is a good indicator of testing and maintenance difficulty; it is typically high in legacy COBOL due to its procedural style with many condi- tionals and loops. NGO captures the lack of structured control flow, often associated with code that is hard to understand and maintain; combined with C, it gives a more complete picture of control-flow complexity, since C alone is not sensitive to control-flow structure. For the maintenance drift dimension, we used the Variable Usage Rate (VUR) and the Dead Paragraph Rate (DPR). VUR measures the proportion of declared variables never referenced in the pro- cedure division, capturing variable over-declaration, common in legacy COBOL. DPR measures the proportion of paragraphs never executed. Together, these metrics assess how far the program has drifted from its original design, and how much noise this drift adds for modernization tools relying on generative AI. For the modularization dimension, we used the Average Para- graph Length (APL), the mean number of statements per paragraph. Very short paragraphs fragment related logic across units, while very long ones concentrate too much responsibility in a single one, increasing the difficulty for generative AI tools during moderniza- tion. 3.3 Descriptive Statistics Structural metrics are computed using two tools provided by our in- dustrial partner: a static extractor that parses each COBOL program to extract structural and syntactic information, and an analyzer that computes the metrics of Table 1 from this extracted information. Table 2 reports summary statistics (minimum, maximum, mean, median, and standard deviation) for each structural metrics across the 14-program corpus. Given the limited size of this corpus and the specific scope, these statistics should be interpreted as descrip- tive characteristics rather than generalizable estimates of COBOL legacy systems. Figure 1 complements Table 2 by visualizing the shape of each metric’s distribution, overlaying individual program values. Most metrics exhibit dispersion and right-skewed distributions, with a majority of programs concentrated at moderate values and a small number of outlier programs reaching high complexity. This is particularly visible for SLOC, C and NGO, where a single program (the same across all three metrics) exhibits values several times higher than the corpus median, indicating some degraded legacy implementation coexisting with comparatively cleaner programs in the same industrial codebase. In contrast, DPR shows nearly all programs with a value of 0% (median = 0%), with the exception of the same outlier program reach- ing 1.36%. This suggests that dead paragraphs are not a widespread Table 2: Structural metric distributions on the 14-programs Symbol Min Max Mean MedianStd Unit SLOC360283878957.144660.010203.45lines NPAR121015286.71157.0329.83count DIVR0.251.530.830.670.57ratio C2974341859.43765.52369.94count APL13.3331.7222.2024.007.12lines NGO41678286.7195.5465.52count VUR41.89386.31666.16367.92518.947% DPR0.001.360.100.000.36% issue in this corpus, but can be significant in modernization use cases. That’s why we keep this metric here in our corpus analysis. 3.4 Inter-Metric Correlation To assess if our metrics vary independently or jointly, we com- pute pairwise Spearman correlations across the corpus. Size and control-flow-related metrics (SLOC, NPAR, C, NGO, VUR) are strongly correlated with each other (휌>0.90), while DR is strongly anti-correlated with this group (휌<−0.90), and APL shows only moderate correlation (휌 ≈0.6–0.7). DPR remains largely indepen- dent of all other metrics (|휌|<0.25). These indicate that complexity manifests as a coherent, multi-dimensional pattern rather than as independently varying properties, motivating the correlation-based approach for defining target complexity levels (see Section 5). 4 Approach Figure 2 illustrates the overall pipeline of our approach. The pipeline consists of two main phases: Specification driven COBOL Genera- tion and Iterative degradation. The first phase focuses on generating COBOL programs based on provided specifications, while the sec- ond phase iteratively degrades these programs to enhance their realism by incorporating patterns from an asset library. This modu- lar approach allows for efficient handling of multiple specifications and facilitates the integration of various components in the pipeline. We separate generation from degradation rather than producing a degraded program directly. A program already containing legacy patterns is difficult for a domain expert to read and validate against the specification. By first producing a clean program, we obtain a version that can be readily reviewed against the specification, and that serves as the reference against which the degraded pro- gram’s input/output behavior is later compared (Section 6.2). This separation is what allows us to certify that a realistic degraded program still conforms to its original specification, a guarantee that is particularly valuable for modernization use cases such as busi- ness rule extraction or retro-documentation, where the generated benchmark’s ground truth must be known with confidence. 4.1 Inputs: Specifications and Asset Library Specifications. Each task in the pipeline is driven by a natural language specification that describes the business logic and require- ments of a target COBOL program. Specifications are generated from real-world use cases with an agent and subsequently validated by a human expert from our industrial partner to ensure alignment AISM ’26, October 12–16, 2026, Munich, GermanyJean-Baptiste Espinasse, Djamel Eddine Khelladi, and Mathieu Acher 0 10000 20000 lines SLOC 0 250 500 750 1000 count NPAR 0.5 1.0 1.5 ratio DR 0 2000 4000 6000 count C 15 20 25 30 lines APL 0 500 1000 1500 count NGO 40 50 60 70 80 % VUR 0.0 0.5 1.0 % DPR Figure 1: Metrics across the 14-program corpus. Boxes show interquartile ranges; individual points represent each program. Assets Idioms Smells Syntactic ExtractionAnalysisRules validation spec.mdinstruction.md PROG.cbl instruction.md PROG.cbl loop-check iteration loop-context Done Iterative Degradation Specification-driven Generation PROG.cbl CompilationExecution loop-check iteration loop-context Pass Fail Pass Fail Figure 2: Overview of the Spec2COBOLRot pipeline. with real enterprise use cases. Specifications are provided in a struc- tured format that captures the essential elements of the program’s functionality, including functional objective, business context, re- quired inputs, produced outputs, and business rules. The specifications of our running examples, described in Ta- ble 3, are available in our replication package. Each specification comprises four levels: the functional objective, an input data defini- tion, an output data definition, and a business rule expressed as an explicit computation. Asset Library. To guide the degradation phase, we rely on an asset library containing recurring patterns extracted from real pro- duction COBOL programs. Each asset belongs to one of three fami- lies: idioms are recurring code fragments characteristic of legacy COBOL batch programs, reflecting common operational conven- tions rather than any defect; smells are poor coding practices that erode maintainability, such as unstructured control flow or dead state, without altering functional behavior; mismatches are COBOL-specific constructs that force a non-trivial adaptation deci- sion during later modernization. Each asset is validated to ensure it accurately reflects a pattern observed in real-world code, so the library serves as a grounded reference for the degradation agent rather than an arbitrary source of complexity. Beyond its family, each asset is described by a short rationale, a canonical code form, and guidance on when and how it should be injected. The full set of assets can be found in our replication package (the full library composition is reported in Section 5). 4.2 Phase 1: Specification driven COBOL Generation Generation. Given a specification and an instruction prompt (instruction.md), the generation agent is responsible for produc- ing an initial COBOL program that adheres to the specified require- ments. Data files are generated alongside the COBOL program to ensure that the program can be compiled and executed successfully. Synthetic data is generated from the specification context. This syn- thetic data is reused in Phase 2 to verify that degradation preserves the program’s input/output behavior (Section 4.3). Syntactic validation. The generated COBOL program under- goes syntactic validation through an iterative loop. At each iteration, the program is compiled using GnuCOBOL and executed against the generated data files. If compilation and execution both succeed, the program is passed to Phase 2. Otherwise, the compiler and runtime errors are fed back to the generation agent as fresh context (loop-context, Figure 2), which produces a revised program for the Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus GenerationAISM ’26, October 12–16, 2026, Munich, Germany next iteration. This loop is bounded by a budget of푘iterations. This step is crucial to confirm that the generated code is not only syntactically correct but also executable, providing a foundation for degradation in the next phase. Output. It is a syntactically correct and executable COBOL pro- gram that meets the requirements specified in the input specifi- cation. However, at this stage, the program may lack the realism and complexity observed in production COBOL code, which will be addressed in Phase 2 through iterative degradation. If the pro- gram still fails to compile or execute after푘iterations, the pipeline terminates for that specification and Phase 2 is not launched. 4.3 Phase 2: Iterative Degradation Asset injection. An degradation agent is employed to iteratively enhance the realism of the generated COBOL program by incor- porating patterns from the asset library. With an other instruction prompt (instruction.md), the agent analyzes the initial program and identifies opportunities to inject relevant assets that align with the program’s structure and functionality. This process involves se- lecting appropriate patterns from the library and integrating them into the codebase, thereby increasing its complexity and resem- blance to real-world COBOL programs. Assets are carefully chosen to ensure that they do not violate the original specification and maintain the program’s correctness while enhancing its realism. Deriving Complexity Levels. To evaluate the pipeline’s abil- ity to generate realistic programs across the complexity spectrum, rather than at isolated points, we define target complexity levels as intervals over SLOC, spanning a reference corpus’s observed range. For each level, we derive a target interval for metrics strongly corre- lated with SLOC (푅 2 >0.7). With linear regression, we predict the metric at both interval bounds, take the envelope of the two pre- dictions, and widen it by±1 residual standard deviation to account for the variability not explained by SLOC alone. Lower bounds are clipped at the corpus-wide minimum observed value for each met- ric, avoiding physically implausible targets that could otherwise result from residual extrapolation at high SLOC values. Metrics whose correlation with SLOC is too weak to support a reliable pre- diction are instead assigned the corpus-wide observed range. This method requires only the structural metrics already computed on the reference corpus (Section 3.3), and can be applied to any cor- pus of production programs to derive realistic, complexity-scaled degradation targets. Stopping criterion. The target intervals derived above are used to drive and stop the degradation loop. They are computed once, ahead of the loop, by using the Deriving Complexity Levels method. In our experiments, these are the min–max bounds derived per complexity level (Section 5). At each iteration, the structural met- rics defined in Section 3.2 (Table 1) are recomputed on the current program using the extraction and analysis tools. If any metric falls outside its target interval, degradation continues with further asset injection; otherwise, the process stops. If the target intervals are not reached within a budget of푘iterations, the process stops af- ter the last iteration and returns the program in its current state. The pipeline does not depend on target intervals being derived from a real reference corpus. In the absence of such a corpus, these bounds could instead be set manually or from other sources, at the risk of driving degradation toward target profiles that are less representative of actual production code. 5 Experimental Setup Table 3: Specifications used in the evaluation. NameDescription TRNRECBTNightly banking transaction processing against account positions, with fee and interest appli- cation and end-of-day interbank reconciliation. PAYCOMBTMonthly payroll computation with social con- tributions, tax withholding, and ordered deduc- tions, producing pay slips, bank transfers, and a statutory declaration. CLMPREMTAnnual insurance premium recomputation with experience-based adjustment, and claim settlement across motor, property, and health portfolios. Specifications Used. We use three specifications from three busi- ness domains (banking, payroll, insurance), each generated from a real-world use case and validated by a COBOL expert (Table 3). Asset Library Construction. The library was created and ex- tracted from the production programs provided by our industrial partner (Section 3.1) and contains 26 assets: 8 idioms, 15 smells, and 3 mismatches (Table 4). Table 4: Asset library composition. Family#Examples Idioms8EOF-driven iteration; indexed lookup; reject-record output Smells15 unstructured GO TO flow; working- storage overdeclaration; mixed I/O and business logic Mismatches3memory-overlay views; packed-decimal representation, redefine Tools and configuration. The generation and degradation agents are both using GPT-5.4 . Each phase is bounded by an iteration bud- get of푘=5 loop iterations (Figure 2): Phase 1 recompiles and re-executes the program up to 5 times to reach a syntactically valid and executable state; Phase 2 performs up to 5 asset-injection iter- ations to reach the target complexity profile. Syntactic validation relies on GnuCOBOL; structural metrics are computed using the extraction and analysis tools provided by our industrial partner. The pipeline is orchestrated with Harbor [12]. Target Complexity Levels. The four complexity levels are de- fined by manually chosen SLOC intervals, spanning the observed range of the industrial partner corpus (Section 3.3, Table 2) from small (∼2.5k SLOC) to large (∼25k SLOC) programs. These SLOC in- tervals are the only manually set parameter of the method; all other target intervals are derived from them via regression. Applying the Deriving Complexity Levels method of Section 4.3 to the industrial partner corpus, NPAR, DIVR, C, NGO, and VUR are strongly cor- related with SLOC (푅 2 >0.7) and are assigned regression-based AISM ’26, October 12–16, 2026, Munich, GermanyJean-Baptiste Espinasse, Djamel Eddine Khelladi, and Mathieu Acher intervals; APL’s correlation is too weak (푅 2 =0.36) and is instead assigned the corpus-wide observed range. DPR is excluded due to its near-zero variance across the corpus. Table 5 reports the resulting target intervals for four complexity levels. Note that intervals for adjacent levels may partially overlap, reflecting residual variability around the SLOC-metric trend rather than an inconsistency in the level definitions. 6 Evaluation To evaluate the effectiveness of our agentic pipeline, we designed a set of research questions that focus on the syntactic validity, struc- tural completeness, and realism of the generated COBOL programs. RQ1Does the agentic pipeline generate syntactically valid and structurally complete COBOL programs from natural language specifications ? RQ2 Does the iterative degradation process increase the tar- get structural complexity while preserving their functional behavior ? RQ3What are the practical scale limits beyond which the degradation process fails to produce compilable and func- tionnaly equivalent COBOL programs ? RQ4Do domain experts perceive the generated programs as realistic production COBOL ? Table 5: Correlation of target intervals for our metrics across complexity levels, derived by linear regression on SLOC. Metric L1 (2.5–3.5k) L2 (4.5–6.5k) L3 (8–12k) L4 (20–25k) NPAR33–15998–255210–431594–847 DIVR0.80–1.440.66–1.340.39–1.180.25–0.61 C29–1009439–16961240–29553988–5932 NGO4–2934–41324–633504–1153 APL13.3–31.713.3–31.713.3–31.713.3–31.7 VUR (%)68.6–84.363.4–80.853.9–74.841.9–54.1 6.1 RQ1 - Syntactic Validity For each generated program, we verify that it compiles with Gnu- COBOL together and executes with data files without runtime errors, and produces all input and output files required by the spec- ification. Across all generation runs for the three specifications (TRNRECBT, PAYCOMBT, CLMPREMT), every program satisfies all three checks. Beyond these automated checks, manual inspec- tion of the generated code against the specification suggests that the key business rules, in particular the computations explicitly required by the specification, are mostly reflected in the procedure division. Data formats specified for input and output fields (types, lengths, value ranges) are also respected in most cases. This inspec- tion is qualitative and not exhaustive; a systematic, rule-by-rule and field-by-field validation is left for future work. 6.2 RQ2 - Realism Degradation For each of the three specifications (TRNRECBT, PAYCOMBT, CLM- PREMT), we run the pipeline 3 times and report results per pro- gram. Structural complexity increases across all three programs after degradation. Figure 3 reports, for each program, the structural metrics before (pre-degradation, red) and after (post-degradation, blue) the degradation phase, together with the calibrated target band for L1 and L2 levels. Each axis is log 10 -scaled and normal- ized so that a raw value of zero maps to the center and the upper bound of the target interval maps to the outer edge, making rela- tive progress toward the target comparable across metrics of very different raw scale (SLOC, C, and NGO span several orders of magnitude in the reference corpus 3.3). Across all three programs, the pre-degradation profile lies inside the target band on most axes, reflecting the intentionally simple programs produced by Phase 1. After degradation, the profile expands outward and enters the target band on nearly every axis, indicating that asset injection systemati- cally moves the program toward the complexity levels observed in the reference corpus. Functional equivalence, however, is not always preserved. We check functional equivalence by executing both the pre- and post- degradation program on the same synthetic test dataset (Section 4.2) and diffing their output files. Note that non-terminating programs are caught by an execution timeout and rejected earlier in the loop. Results on L1 are mixed: all 3 runs of CLMPREMT preserve output files unchanged, while 2 of 3 runs of PAYCOMBT and 2 of 3 runs of TRNRECBT introduce differences across all output files, for an overall preservation rate of 5/9 runs. This choice reflects a delib- erate design trade-off, we favored giving the degradation agent freedom to explore structural and pattern-level transformations without an additional hard constraint at every iteration, to avoid over-restricting its ability to reach the target complexity profile. In hindsight, our results suggest this freedom comes at a measurable cost to functional preservation, and that a tighter, per-iteration veri- fication loop, similar to the one already used in Phase 1 (Section 4.2), would likely be a better trade-off in practice (Section 8). 6.3 RQ3 - Practical Scale Limits Across all generated programs for L3 and L4, we did not observe practical scale limits in terms of compilability or functional equiva- lence: every degraded program compiled successfully, and the data manipulated between the pre-degradation and post-degradation phases remained often consistent. The practical limit we identified is instead qualitative in nature and relates to realism rather than correctness. When the degradation process is constrained to reach a target size or complexity threshold, the underlying LLM tends to compensate by duplicating paragraphs that are structurally and semantically very similar to one another, rather than introducing genuinely diverse legacy-like constructs. While this strategy allows the imposed thresholds to be met, it comes at the cost of realism, as such repetitive patterns are not representative of how legacy COBOL programs typically grow in size and complexity over time. This suggests that scale constraints should be interpreted not as a limit on the technical feasibility of the degradation process, but rather as a factor that can silently erode the realism of the generated defects. 6.4 RQ4 - Expert Review To validate these findings, the generated specifications were re- viewed by 3 domain experts with 20 to 30 years of experience in legacy systems, and the resulting programs by 2 other experts with comparable experience. Overall, experts were impressed by the Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus GenerationAISM ’26, October 12–16, 2026, Munich, Germany SLOC NPAR C APL DIVR NGO VUR 1 2 3 4 5 (a) PAYCOMBT (L1) SLOC NPAR C APL DIVR NGO VUR 1 2 3 4 5 (b) CLMPREMT (L1) SLOC NPAR C APL DIVR NGO VUR 1 2 3 4 5 (c) TRNRECBT (L1) SLOC NPAR C APL DIVR NGO VUR 1 2 3 4 5 (d) PAYCOMBT (L2) SLOC NPAR C APL DIVR NGO VUR 1 2 3 4 5 (e) CLMPREMT (L2) SLOC NPAR C APL DIVR NGO VUR 1 2 3 4 5 (f) TRNRECBT (L2) Target band (min→ max) Pre-degradationPost-degradation Figure 3: Comparing the realism of generated COBOL before/after the degradation phase, for the L1 (top row) and L2 (bottom row) target complexity levels. agent’s ability to generate COBOL code for a complex business do- main from the specifications, though the resulting programs were judged very clean, arguably too clean for real industrial COBOL. Figure 4 shows one excerpt from PAYCOMBT that experts recog- nized as a typical legacy pattern encountered during modernization audits. Other structural aspects, however, revealed the limits of the injected patterns: the deliberate absence ofCOPYstatements, ubiquitous in real legacy codebases;IFblocks systematically closed with an explicitEND-IFrather than the more common, error-prone implicit termination by period (.); an overuse ofSTRINGstatements withDELIMITED BY SIZE, several also relying recurrently on|as a delimiter; andREDEFINESclauses, which our degradation process rarely managed to introduce due to the lack of sub-variable format definitions in the generated data structures. Overall, experts con- sidered the use of legacy assets a promising first step, but still too synthetic to fully capture the diversity of real legacy defects. 01 WS-AMTS. 05 WS-ADJ-SAL PIC 9(7)V99. 05 WS-HR-RATE PIC 9(7)V99. 05 WS-OT1-AMT PIC 9(7)V99. ... COMPUTE WS-HR-RATE = WS-ADJ-SAL / CAL-HRS (WS-CAL-IDX) COMPUTE WS-OT1-AMT = AGG-OT1 (WS-I) * WS-HR-RATE * 1.25 Figure 4: Excerpt from the degraded PAYCOMBT program 7 Threats to Validity Construct validity. The structural metrics used in this study are based on syntactic extraction that do not directly measure the com- prehension difficulties of the business domain. This threat is miti- gated by the fact that the specification documents were validated by experts as realistic, although expert review remains subjective and may vary across reviewers with different experience. Moreover, we deliberately targeted a level of structural complexity comparable to production programs, which increases the representativeness of our generated programs relative to real-world code and contributes to the overall realism of the study. Internal validity. The generation and enrichment steps rely on non-deterministic LLM calls. Running the pipeline on the same specification may produce different programs, this limits exact reproducibility. So the overall pipeline is not expected to be exactly reproducible. But it’s mitigated by the fact that in our use case, we want variability in the generated programs to cover a wide range of possible implementations. External validity. Our study relies on a limited number of generated programs, though spanning distinct domains (banking, payroll, insurance), which constrains the generalization of our re- sults. The reference metrics are derived from 14 programs provided by a single industrial partner in one business domain (HR payroll), which may further limit generalization to other enterprises and domains. Our pipeline is nonetheless designed to be generic, and applicable to arbitrary business contexts, domains, and COBOL dialects, and could be extended to cover a wider range of PIC clause variations and more complexDATA DIVISIONstructures. Executing programs relying on such constructs, or on CICS and embedded AISM ’26, October 12–16, 2026, Munich, GermanyJean-Baptiste Espinasse, Djamel Eddine Khelladi, and Mathieu Acher SQL, is not a limitation of the methodology, but requires an execu- tion environment capable of supporting them, which our current toolchain does not provide. 8 Discussion The degradation process successfully increases the structural com- plexity of the generated programs towards the reference corpus. Functional correctness, verified by input/output comparison on a synthetic dataset, holds only in part and remains limited by test data coverage, an impact future work could also extend to runtime performance. Structural complexity is primarily driven by business logic and shaped by decades of patches and modifications, so targeting ab- solute metric values independently of business complexity may appear artificial. This does not undermine our objective: our goal is to produce programs exhibiting the structural characteristics of legacy code (poor practices, accumulated complexity, maintenance drift) while remaining functionally correct. A program "too com- plex" for its business logic is an even harder case for modernization tools, and therefore a more valuable benchmark artifact. A natural baseline is direct LLM degradation from a generic "make it look like legacy code" prompt, without the asset library. We made the opposite choice to keep injected patterns traceable to real produc- tion code rather than the model’s priors, and leave a quantitative comparison of the two to future work. A promising direction is to keep the decoupling between gener- ation and degradation, but replace the degradation process itself with a more realistic alternative: rather than injecting patterns di- rectly, a maintenance plan (a to-do list of patches and modifications spanning several decades) would drive a legacy-developer agent that incrementally adds and modifies code as would have happened over time. This more realistic process could plausibly yield more realistic programs, though at a higher generation cost, and we leave its evaluation to future work. We did not include comment-related metrics, such as comment density or comment-code alignment. The latter, capturing whether comments remain semantically consistent with the code, would be useful for assessing documentation reliability in legacy systems, where documentation is often outdated [20]. However, it requires semantic analysis beyond static structural parsing, which we leave for future work. 9 Conclusion We proposed a systematic agentic AI pipeline for generating realis- tic COBOL programs, combining specification-driven generation with iterative degradation guided by patterns and complexity tar- gets extracted from real production code. Our evaluation across three programs from distinct business domains showed that degra- dation reliably produces syntactically valid programs and moves them to the target complexity levels, with business-behavior preser- vation achieved in part, yielding programs that exhibit the struc- tural properties of real-world legacy systems. On edge cases, experts appreciated the resulting programs but noted that some legacy pat- terns were still missing despite matching metrics, a limitation of pattern coverage rather than of the methodology itself. As future work, we plan to scale the pipeline further and extend our reference corpus to other business domains. Acknowledgements. This work is supported by the Inria Défi LLM4Code (https://project.inria.fr/llm4code/). References [1]2025. Jackson Structured Programming. http://w.jacksonworkbench.co.uk/ jsp.htm. [2]Sandeep Hans al. 2025. Automated Testing of COBOL to Java Transformation. arXiv:2504.10548 [cs.SE] https://arxiv.org/abs/2504.10548 [3] Mir Sameed Ali, Nikhil Manjunath, and Sridhar Chimalakonda. 2023. X-COBOL: A Dataset of COBOL Repositories. doi:10.48550/arXiv.2306.04892 [4]Chiranjeevi B S and Sridhar Chimalakonda. 2025. LLM Vs Rule-Based - The COBRAIN Tool and An Empirical Study on Extracting Business Rules from COBOL. In EASE ’25. 294–304. doi:10.1145/3756681.3756982 [5] Rina Diane Caballar and Cole Stryker. 2025. What Is COBOL Modernization? https://w.ibm.com/think/topics/cobol-modernization. IBM Think. Published 27 November 2025, updated 06 April 2026. Accessed: 2026-06-23. [6]Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Carolyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. 2024. Knowledge Transfer from High-Resource to Low- Resource Programming Languages for Code LLMs. Proc. ACM Program. Lang. 8, OOPSLA2, Article 295 (Oct. 2024), 32 pages. doi:10.1145/3689735 [7]Sahil Chaudhary. 2023. Code Alpaca: An Instruction-following LLaMA model for code generation. https://github.com/sahil280114/codealpaca. [8]Agnieszka Ciborowska, Aleksandar Chakarov, and Rahul Pandita. 2021. Con- temporary COBOL: Developers’ Perspectives on Defects and Defect Location. In 2021 ICSME. IEEE, 227–238. doi:10.1109/icsme52107.2021.00027 [9] Anh T. V. Dau, Shin Hwei Tan, Jinqiu Yang, Nghi D. Q. Bui, and Anh Tuan Nguyen. 2026. COBOL-Coder: Domain-Adapted Large Language Models for COBOL Code Generation and Translation. doi:10.48550/arXiv.2604.03986 [10] Shmulik Froimovich, Raviv Gal, Wesam Ibraheem, and Avi Ziv. 2025. Quality Evaluation of COBOL to Java Code Transformation. arXiv:2507.23356 [cs.SE] https://arxiv.org/abs/2507.23356 [11]Shubham Gandhi, Manasi Patwardhan, Jyotsana Khatri, Lovekesh Vig, and Raveendra Kumar Medicherla. 2024. Translation of Low-Resource COBOL to Logically Correct and Readable Java leveraging High-Resource Java Refinement. In LLM4Code ’24 (Lisbon, Portugal). 46–53. doi:10.1145/3643795.3648388 [12]Harbor Framework Team. 2026. Harbor: A framework for evaluating and optimiz- ing agents and models in container environments. doi:10.5281/zenodo.20953922 Version v0.16.1. [13]Makena Kelly. 2025. DOGE Plans to Rebuild SSA Code Base in Months, Risking Benefits and System Collapse. https://w.wired.com/story/doge-rebuild-social- security-administration-cobol-benefits/. [14] Atul Kumar and al. 2025. Automated Validation of COBOL to Java Transformation. https://arxiv.org/abs/2506.10999 [15] Dylan Lee, Austin Henley, Bill Hinshaw, and Rahul Pandita. 2022. OpenCBS: An Open-Source COBOL Defects Benchmark Suite. arXiv:2206.06260 [cs] doi:10. 48550/arXiv.2206.06260 [16] Fangjian Lei, Jiawen Liu, Shayan Noei, Ying Zou, Derek Truong, and William Alexander. 2025. Enhancing COBOL Code Explanations: A Multi-Agents Ap- proach Using Large Language Models. arXiv:2507.02182 [cs.SE] https://arxiv. org/abs/2507.02182 [17] Juho Leinonen, Paul Denny, Olli Kiljunen, Stephen MacNeil, Sami Sarsa, and Arto Hellas. 2024. LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education. arXiv:2411.10455 [cs.CY] https://arxiv.org/abs/2411.10455 [18]Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2025.Wiz- ardCoder: Empowering Code Large Language Models with Evol-Instruct. arXiv:2306.08568 [cs.CL] https://arxiv.org/abs/2306.08568 [19]Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. 2025. Training Language Models on Synthetic Edit Sequences Improves Code Synthesis. arXiv:2410.02749 [cs.LG] https://arxiv.org/abs/2410.02749 [20]Rock Sabetto and al. 2025. Impact of Comments on LLM Comprehension of Legacy Code. arXiv:2506.11007 [cs.SE] https://arxiv.org/abs/2506.11007 [21]Shin Saito, Takaaki Tateishi, and Yasuharu Katsuno. 2025. Grammar- and Coverage-based Augmentation of Programs for Training LLMs. In ASE Workshops. 1–4. doi:10.1109/ASEW67777.2025.00010 Received 2026-08-11; accepted 2026-08-23