Paper deep dive
Towards Standardized Evaluation in Automated Domain Modeling: Introducing a Benchmark
Vasiliy Seibert
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/18/2026, 5:57:02 AM
Summary
This paper introduces a standardized benchmark for evaluating automated domain modeling approaches. The benchmark combines two datasets: the 45-record Golden UML Modelset and an 8-record reference archive, totaling 53 records. It utilizes the Triandini metric (metrik-4) to compare generated PlantUML class diagrams against ground truth. The study evaluates three candidate strategies: a heuristic rule-based method and two LLM-driven zero-shot strategies using GLM-5.1 and Kimi K2 models.
Entities (8)
Relation Signals (10)
Benchmark → usesmetric → metrik-4
confidence 99% · The benchmark chooses metrik-4, the Triandini (30) metric... as the fixed metric.
Benchmark → usesdataset → Chen et al. Reference Archive
confidence 98% · The benchmark combines... with the 8-record reference archive of Chen et al.
Benchmark → usesdataset → Golden UML Modelset
confidence 98% · The benchmark combines the 45-record Golden UML Modelset... with the 8-record reference archive
metrik-4 → derivedfrom → Triandini
confidence 95% · metrik-4, the Triandini (30) metric
Chen et al. Strategy → usesllm → Kimi-K2
confidence 95% · Chen et al. (15; 14) Kimi K2 (22) ... 0.699
Chen et al. Strategy → usesllm → GLM 5.1
confidence 95% · Chen et al. (15; 14) GLM-5.1 (35) ... achieves the highest scores
Calamo et al. Strategy → usesllm → GLM 5.1
confidence 95% · Calamo et al. (9; 10) GLM-5.1 (35) ... 0.694
Calamo et al. Strategy → usesllm → Kimi-K2
confidence 95% · Calamo et al. (9; 10) Kimi K2 (22) ... 0.613
Benchmark → followsguideline → FAIR4RS
confidence 90% · In accordance with the FAIR4RS recommendations... the benchmark is provided as a research artifact
Rule-based Strategy → usestool → SpaCy
confidence 90% · The rule_based candidate is a Python-over-spaCy re-implementation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Domain modeling plays an essential role in domain-driven design, capturing essential entities and their relationships within a specific domain. Despite advancements in automated domain modeling, the absence of standardized benchmarks has hindered the comparative assessment of existing approaches. This paper introduces a benchmark designed to address this gap. The benchmark combines the 45-record Golden UML Modelset (Verbruggen et al., 2025) on Zenodo, as distributed by the Text2UML project of Calamo, Mecella, and Snoeck (Calamo et al., 2025), with the 8-record reference archive of Chen et al. (Chen et al., 2023a,b), enabling the evaluation of automated domain modeling approaches across different levels of complexity and scale. Given a natural language description, the task is to generate a corresponding domain model. For each description, a reference domain model is provided as ground truth. A metric is used to compare the generated domain model with the corresponding ground-truth model. To demonstrate the utility of the benchmark, we evaluate multiple automated domain modeling approaches, including heuristic rule-based methods and LLM-driven strategies. In accordance with the FAIR4RS recommendations (Chue Hong et al., 2022), the benchmark is provided as a research artifact to encourage reuse and support future research on automated domain modeling.
Tags
Links
- Source: https://arxiv.org/abs/2608.15255v1
- Canonical: https://arxiv.org/abs/2608.15255v1
Trouble viewing inline? Open PDF directly →
Full Text
35,442 characters extracted from source content.
Expand or collapse full text
Towards Standardized Evaluation in Automated Domain Modeling: Introducing a BenchmarkCCS: Software and its engineering → Software creation and management → Software verification and validation Vasiliy Seibert Affiliation: Institute for Software and Systems Engineering, TU Clausthal , Clausthal-Zellerfeld , Lower Saxony , Germany email: vasiliy.seibert@tu-clausthal.de OrcID: 0000-0002-7121-6816 © none; Received 2026-07-03 Abstract. Domain modeling plays an essential role in domain-driven design, capturing essential entities and their relationships within a specific domain. Despite advancements in automated domain modeling, the absence of standardized benchmarks has hindered the comparative assessment of existing approaches. This paper introduces a benchmark designed to address this gap. The benchmark combines the 45-record Golden UML Modelset (Verbruggen et al., 2025) on Zenodo, as distributed by the Text2UML project of Calamo, Mecella, and Snoeck (Calamo et al., 2025), with the 8-record reference archive of Chen et al. (Chen et al., 2023a,b), enabling the evaluation of automated domain modeling approaches across different levels of complexity and scale. Given a natural language description, the task is to generate a corresponding domain model. For each description, a reference domain model is provided as ground truth. A metric is used to compare the generated domain model with the corresponding ground-truth model. To demonstrate the utility of the benchmark, we evaluate multiple automated domain modeling approaches, including heuristic rule-based methods and LLM-driven strategies. In accordance with the FAIR4RS recommendations (Chue Hong et al., 2022), the benchmark is provided as a research artifact to encourage reuse and support future research on automated domain modeling. Keywords: domain modeling, class diagrams, large language models, benchmark, evaluation, FAIR principles 1. Introduction Domain modeling is central to model-driven software engineering (18), and a domain model captures the relevant entities of a domain and their relationships using the syntax of UML class diagrams. The benefit of using UML class-diagram syntax is that the resulting model is precise and well known by software developers, who will use it as a foundation for building the system, and that the model facilitates clear communication in requirements discussions and reduces misunderstandings between stakeholders. Assisting or even automating domain modelling has great potential regarding increasing quality and reducing cost. Traditionally, heuristic rule-based methods (2; 1) have been formulated to analyze Requirement Descriptions given in Natural Language Text using tools like the Stanford CoreNLP API and subsequently apply rules to extract entities and meaningful relationships between them. However, these approaches are getting surpassed by modern approaches that rely entirely on Large Language Models (LLMs) (12). Measuring the performance of automated domain modelling approaches has been neglected in the literature (12). A benchmark analogous to Stanford’s HELM framework (29), consisting of a clearly defined task, a public dataset, and a deterministic metric, would help the research community by providing a place for healthy competition. The literature has repeatedly called for such a benchmark: Cámara et al. (12) state that the assessment of generative AI in modeling tasks is under-investigated, and Cámara, Burgueño, and Troya (11) argue for robust methodologies to standardize it. On the dataset side, a community-curated domain model corpus (31; 9) has emerged as a common reference set. On the metric side, the Seibert study (24) implements five metrics behind a single interface and evaluates them against human expert ratings (15). The contribution of this paper is to address the previously described gap by proposing a benchmark. The task is to take a natural-language description of a domain (18) and to generate a domain model in PlantUML class-diagram notation. The benchmark uses a dataset of 53 domain models with their corresponding natural-language descriptions, drawn from two sources (31; 9; 15; 14). The metric is taken from the Seibert study (24), which implements five metrics from four source papers (27; 13; 34; 30) behind a single interface, each returning three sub-scores in the unit interval, one per element type (class, attribute, association), where each sub-score is the F1 of the element-level match between candidate and reference. The benchmark uses metrik-4, the Triandini (30) metric, on the basis that the Seibert study (24) shows it to be the strongest of the five on per-element agreement with the Chen et al. (15) human expert ratings, winning 2 of 3 element-by-statistic cells. To encourage reuse, the benchmark is released as a research artifact in accordance with the FAIR4RS recommendations (16; 33) https://doi.org/10.5281/zenodo.21166386. 2. Related Work Prior approaches to automated domain modeling each evaluate on their own dataset, and the field’s adoption of a common reference set is recent. Heuristic rule-based methods (17; 23; 5; 32; 26; 4; 3; 6; 21; 1) each report a private, case-specific, or industrial evaluation, with no public artefact deposited at a stable identifier. Chen et al. (15) compare different LLMs and prompting strategies, bring their own dataset, and grade candidate models through human expert comparisons despite automated metrics being available, though they do deposit their data on Zenodo (14). Parts of the Chen data are incorporated in a community-curated domain model corpus (31), published at a stable identifier with peer-reviewed provenance. The Calamo, Mecella, and Snoeck corpus (9; 10) is a cleaned encoding of the Verbruggen 45 published alongside a generation pipeline, and is, to our knowledge, the first reference corpus in this space to be published with the pipeline that consumes it. Prior approaches to grading a candidate domain model against a reference are diverse in approach, and no metric has become a community standard. In the education application area, Modi, Taher, and Mahmud (20) develop a Java-based tool that produces an automatic grade from student and instructor solution diagrams, Boubekeur, Mussbacher, and McIntosh (8) combine heuristics with machine learning to predict approximate letter grades, and Bouali et al. (7) compare LLM-generated scores against teaching assistants on 92 student submissions. Singh, Boubekeur, and Mussbacher (27) propose a rule-based Mistake Detection System covering 83 of 97 identified mistake types, and Fauzan, Siahaan, Rochimah, and Triandini (19; 30) decompose assessment into a structural and a semantic component, reporting agreement with expert judgements. For model reuse and repository mining, Čech (13) defines a class-model distance via graph edit distance, Yuan, Yan, and Ma (34) decompose structural similarity into intra- and inter-structure components over a UML Class Graph, and Song et al. (28) adopt a graph neural network on use case graphs with TF-IDF cosine similarity. None of these approaches is cross-compared with a metric from another family, and each evaluates on its own dataset. The Seibert study (24) implements five metrics from four source papers (27; 13; 34; 30) behind a single interface and compares them against human expert ratings (15) on a common corpus. This benchmark reuses one of the five implementations, namely metrik-4, the Triandini (30) metric, as the fixed metric. 3. Approach 3.1. Benchmark Approach The benchmark decomposes into three axes: a data axis, a task axis, and a metric axis. The data axis holds 53 records, each a pair of a natural-language text and a reference PlantUML class diagram drawn from two community-curated corpora. The task axis runs each candidate strategy against the natural-language text to produce a generated PlantUML class diagram. The metric axis is a deterministic comparator that accepts the reference and generated diagrams and returns a 3-tuple of per-element similarity scores, one each for classes, attributes, and associations. 3.2. Data Figure 1. Overview of the benchmark: a (natural-language text, reference PlantUML class diagram) pair is fed to a candidate strategy, which produces a candidate PlantUML diagram that a metric compares against the reference. The benchmark runs on 53 records drawn from two PlantUML class-diagram corpora: a 45-record community-curated corpus (31; 9) and an 8-record reference archive (15; 14). The eight reference diagrams were derived from the ground truth of the Chen et al. study; while that study produced many more models, those are generated by LLMs, whereas these eight were crafted by human experts in the field of education. Each record is a pair of a natural-language text and a reference PlantUML class diagram that the text describes. Together, the 53 records contribute 540 classes, 667 attributes, 184 enums, and 710 relationships. The two corpora differ in scale and in per-record composition: the 45-record corpus averages 9.4 classes and 13.0 attributes per record, while the 8-record archive averages 14.4 classes and 10.3 attributes, reflecting the denser modelling style of its source domain. Table 1 breaks down the 53 records per dataset and decomposes the 710 relationships into seven relationship types included in the dataset, ranging from 355 plain associations to 2 association-class links. Across both corpora, 70% of relationship endpoints carry a multiplicity, indicating that the majority of relationships specify cardinality information that a candidate model must recover. DS-1 (45) DS-2 (8) Total (53) NLT words (mean) 316 423 — Classes (mean / total) 9.4 / 425 14.4 / 115 540 Attributes (mean / total) 13.0 / 585 10.3 / 82 667 Enums (mean / total) 2.4 / 109 9.4 / 75 184 Relationships (mean / total) 11.9 / 537 21.6 / 173 710 Association (--) 268 87 355 Aggregation (o--) 6 0 6 Inheritance (-->|) 102 34 136 Composition (*--) 116 35 151 Directed assoc. (-->) 32 1 33 Dependency (..) 13 14 27 Association-class 0 2 2 Card (% with multiplicity) 68 76 70 Table 1. Per-dataset overview. LabTracker natural-language text (15; 14): The LabTracker software helps (i) doctors manage the requisition of tests and examinations for patients and (i) patients book appointments for tests and examinations at a lab. For the remainder of this description, tests and examinations are used interchangeably. For a requisition, a doctor must provide their numeric practitioner number and signature for verification as well as their full name, their address, and their phone number. The signature is a digital signature, i.e., an image of the actual signature of the doctor. Furthermore, the doctor indicates the date from which the requisition is valid. The requisition must also show the patient’s information including their alpha-numeric health number, first name and last name, date of birth, address, and phone number. A doctor cannot prescribe a test for themselves but can prescribe tests to someone else who is a doctor. Several tests can be combined on one requisition but only if they belong to the same group of tests. For example, only blood tests can be combined on one requisition or only ultrasound examinations can be combined. It is not possible to have a blood test and an ultrasound examination on the same requisition. For each test, its duration is defined by the lab network, so that it is possible to schedule appointments accordingly. The duration of a test is the same at each lab. For some kinds of tests, it does not matter how many tests are performed. They take as long as a single test. For example, several blood tests can be performed on a blood sample, i.e., it takes as long to draw the blood sample for a single blood test as it does for several blood tests. A doctor may also indicate that the tests on a requisition are to be repeated for a specified number of times and interval. The interval is either weekly, monthly, every half year, or yearly. All tests on a requisition are following the same repetition pattern. The doctor and the patient can view the results of each test (either negative or positive) as well as the accompanying report. A patient is required to make an appointment for some tests while others are walk-in only. For example, x-ray examinations require an appointment, but blood tests are walk-in only (i.e., it is not possible to make an appointment for a blood test). On the other hand, some tests only require a sample to be dropped off (e.g., a urine or stool sample). To make an appointment for a requisition, a patient selects the desired lab based on the lab’s address and business hours. For requisitions with repeated tests, a patient is only allowed to make one appointment at a time. The confirmation for an appointment also shows a confirmation number, the date as well as start/end times, and the name of the lab as well as its registration number. It is possible to change or cancel an appointment at any time but doing so within 24 hours of the appointment incurs a change/cancellation fee. Each lab determines its own fee and business hours. All labs are open every day of the year and offer all tests. The business hours of a lab do not change from one week to the next. Each day a lab is open from the day’s start time to its end time, i.e., there are no breaks. LabTracker reference PlantUML (truncated) (15; 14): @startuml enum Interval WEEKLY MONTHLY HALF_YEARLY YEARLY class Doctor +signature: string class Patient +dateOfBirth: string class Requisition +effectiveDate: string +repetitionInterval: Interval (...) Doctor "1" -- "*" Requisition Requisition -- "*" Test Patient "1" -- "0..*" Requisition Appointment "0..*" -- "1" Lab Test "1" -- "0..1" Appointment @enduml 3.3. Metric This benchmark makes use of the metric implementations provided by the Seibert study (24), which packages five metrics from four source papers (27; 13; 34; 30) behind a single interface that accepts two PlantUML strings and returns a 3-tuple of per-element similarity scores, one each for classes, attributes, and associations. The study identifies two risks in reusing published metrics: a wrong implementation of the metric, and a wrong translation of the metric’s native output format into the per-element F1 scale of the human expert ratings. To address the first risk, each metric is restated as a design-by-contract specification with pre- and post-conditions, decomposed into sub-functions, and handed unchanged to two independent LLMs through the same harness; if both implementations produce identical output on all 39 ground-truth comparisons from Chen et al. (15), the specification is assumed to be well-defined and the implementations are assumed to be correct. To address the second risk, the specification is extended to translate each metric’s native output into the per-element F1 scale, and the translation is tested for partial-order preservation against the metric’s native output, so that the ordering of the metric results is consistent with the ordering of the original output across the 39 comparisons. The five implementations are then compared against human expert ratings using a consistency table of four statistics — per-element MAD, residual standard deviation, absolute bias, and Pearson r (linear correlation between metric score and human F1) — computed across the 39 comparisons for each of the three element types. No single metric dominated across all cells: the per-element winners split across M-1, M-3, M-4, and M-5, with M-3 achieving the highest correlation on attributes (r = 0.65) and M-4 achieving the best correlation on classes (r = 0.42) and relationships (r = 0.42). A qualitative analysis complemented the statistical comparison, examining where each metric agreed with or diverged from the human judgements and identifying systematic biases such as M-4’s constant overestimation on relationships. The benchmark chooses metrik-4, the Triandini (30) metric, because it wins 2 of the 3 element-by-r cells and also achieves the best attribute MAD (0.14), making it the most balanced metric on the consistency table. The implementation is reused from the upstream domain-model-metrics package (v1.0.0) without modification, so the per-element scores are reproducible from the version pin. Seibert study metric interface: Metric M compute(ReferencePlantUML, GeneratedPlantUML) -> (class_score : [0, 1], attribute_score : [0, 1], association_score : [0, 1]) 3.4. Candidates The benchmark reuses three open-source candidate strategies: one deterministic rule-based pipeline and two LLM-driven zero-shot strategies. The rule_based candidate is a Python-over-spaCy re-implementation of the NLP-based pipeline of Abdelnabi, Maatuk, Abdelaziz, and Elakeili (1), which parses the natural-language text with spaCy’s dependency parser and applies heuristic rules for extracting classes, attributes, and relationships, emitting a PlantUML class diagram with no LLM call. The two zero-shot candidates were derived from the literature: one from Chen et al. (15; 14) and one from Calamo, Mecella, and Snoeck (9; 10). The Chen et al. prompt asks the LLM to extract the class diagram as an intermediate domain-specific language (Enumerations, Class, Relationships); the Calamo et al. prompt asks the LLM to emit the PlantUML class diagram directly, guided by a 5-step directive covering classes, attributes, relations and inheritance, relation-to-class assignment, and cardinalities. Each LLM-driven candidate is run with two LLMs (GLM-5.1 and Kimi K2) and repeated three times to account for the non-determinism of cloud-served LLMs. To ensure that the LLM output adheres to the PlantUML syntax accepted by the parser, the Chen et al. candidate makes a second LLM call prompted with a grammar encoding the parser’s strict grammar, which translates the intermediate DSL into canonical PlantUML. Chen et al. (15; 14) — Stage 1, system prompt: Generate the lists of model classes and associations from a given description. Chen et al. (15; 14) — Stage 1, task prompt: Create a class diagram for the following description by giving the enumerations, classes, and relationships using format: Enumerations: enumerationName(literals) (there might be no or multiple enumerations) Class: className(attributeType attributeName (there might be multiple attributes)) (there might be multiple classes) Relationships mul1 class1 associate mul2 class2 (class1 and2 are classes above. mul1 and mul2 are one of the following options [0..*, 1, 0..1, 1..*]) (there might be multiple associations) class1 inherit class2 (class1 and class2 are classes above) (there might be multiple inheritance) mul1 class1 contain mul2 class2 (class1 and2 are classes above. mul1 and mul2 are one of the following options [0..*, 1, 0..1, 1..*]) (there might be multiple composition) Chen et al. (15; 14) — Stage 2, translate prompt: You are a PlantUML translator. You will be given a draft class diagram extracted from a natural-language specification. The draft may be in a free-form text format (the upstream zenodo S1 format) OR in a malformed PlantUML form. Your job is to produce a single, clean PlantUML class diagram in the canonical grammar described below. (...) Calamo, Mecella, and Snoeck (9; 10) — prompt: You will be asked by the user to create a plant UMl model from specification text. Do so in the most clear way possible, avoid class properties and assign molteplicity. Do include attributes for classes. For example the class Book would be: class Book String Title, String Author, Date PublicationDate Use only bi-directional arc for relations and no description. For example a relation between the class Book and the class Page, if the Book can have from one to many pages and the pages could have exactly one book, would be: Book "1..1" -- "1..*" Page Adapt the cardinality to each case. Where no specific cardinality is specified, use the default "0..*". If necessary, feel free to use inheritance. The plantuml has to be the class diagram. In generating the diagram perform this steps in order 1. Extract class from text 2. Extract attributes for each class 3. Extract relations form text and look for the inheritance 4. Assign the relation to the corresponding class 5. Add cardinality to the relations Put everything in this order: first all classes and then all relations. In our example would be: @startuml class Book String Title, String Author, Date PublicationDate class Page String Content Book "1..1" -- "1..*" Page @enduml Output plantuml without futher text or explaination. 4. Results The benchmark results are summarised in Tables 2 and 3, which report the cross-run stability of the three candidate strategies on the two community-curated corpora introduced in Section 3.1: DS-1, the 45-record community-curated corpus of Verbruggen et al. (31) as re-encoded by Calamo, Mecella, and Snoeck (9), and DS-2, the 8-record reference archive of Chen et al. (15; 14), whose reference diagrams were crafted by human experts in the field of education. Together the two corpora contribute 53 records, comprising 540 classes, 667 attributes, 184 enumerations, and 710 relationships. Each cell in the tables corresponds to one (candidate, LLM) combination, repeated over three independent runs to capture the non-determinism of cloud-served LLMs. The three candidates are the rule-based pipeline of Abdelnabi, Maatuk, Abdelaziz, and Elakeili (1), the two-stage zero-shot strategy of Chen et al. (15; 14), and the single-stage zero-shot strategy of Calamo, Mecella, and Snoeck (9; 10). The two LLMs are GLM-5.1 (35) and Kimi K2 (22), invoked through Ollama with think mode disabled, at an extract temperature of 0.7 and a translate temperature of 0.0 for the Chen et al. candidate’s second-stage grammar prompt. Scoring is performed with metrik-4 from the Seibert study (24), pinned to v1.0.0 of the domain-model-metrics package (25), which returns one F1 score per element type (class, attribute, relationship), each in the unit interval. For each cell, the tables report the 3-run median and cross-run standard deviation of the per-run mean score, along with the median percentage of records that failed validation across the three runs. The rule-based candidate is deterministic and involves no LLM call, so it appears with no LLM and zero cross-run variance on every element. Candidate LLM Fail % Class Attribute Relationships Rule-based (1) — 0.0 0.417 ± 0.000 0.447 ± 0.000 0.348 ± 0.000 Chen et al. (15; 14) GLM-5.1 (35) 0.0 0.746 ± 0.006 0.780 ± 0.006 0.676 ± 0.007 Chen et al. (15; 14) Kimi K2 (22) 8.9 0.699 ± 0.013 0.717 ± 0.014 0.641 ± 0.014 Calamo et al. (9; 10) GLM-5.1 (35) 2.2 0.694 ± 0.024 0.691 ± 0.025 0.701 ± 0.024 Calamo et al. (9; 10) Kimi K2 (22) 4.4 0.613 ± 0.015 0.603 ± 0.016 0.637 ± 0.015 Table 2. Benchmark results on DS-1 (45 records). Scores are 3-run median ± cross-run std; Fail % is the median percentage of records that failed validation. Best results in bold. Candidate LLM Fail % Class Attribute Relationships Rule-based (1) — 0.0 0.312 ± 0.000 0.353 ± 0.000 0.215 ± 0.000 Chen et al. (15; 14) GLM-5.1 (35) 12.5 0.625 ± 0.013 0.650 ± 0.014 0.576 ± 0.030 Chen et al. (15; 14) Kimi K2 (22) 25.0 0.596 ± 0.057 0.603 ± 0.053 0.581 ± 0.066 Calamo et al. (9; 10) GLM-5.1 (35) 0.0 0.655 ± 0.062 0.640 ± 0.064 0.689 ± 0.059 Calamo et al. (9; 10) Kimi K2 (22) 0.0 0.506 ± 0.080 0.487 ± 0.070 0.548 ± 0.104 Table 3. Benchmark results on DS-2 (8 records). Scores are 3-run median ± cross-run std; Fail % is the median percentage of records that failed validation. Best results in bold. On DS-1, the 45-record community-curated corpus (31; 9), both LLM-driven candidates outperform the rule-based baseline (1) on every element. The Chen et al. candidate (15; 14) paired with GLM-5.1 (35) achieves the highest scores on class (0.746) and attribute (0.780), while the Calamo et al. candidate (9; 10) paired with GLM-5.1 achieves the highest score on relationship (0.701). The rule-based baseline scores 0.417 on class, 0.447 on attribute, and 0.348 on relationship, so the LLM candidates lead by 0.15–0.35 absolute on every element. Cross-run stability is high: the standard deviation across the three runs is at most 0.025 for every (candidate, LLM) cell, and as low as 0.006 for the Chen et al. candidate with GLM-5.1. The Chen et al. candidate with GLM-5.1 also produces zero validation failures across all three runs, while the Calamo et al. candidate with GLM-5.1 has a 2.2% failure rate. Kimi K2 (22) is competitive but scores lower than GLM-5.1 on both candidates, with failure rates of 8.9% (Chen et al.) and 4.4% (Calamo et al.). The rule-based baseline is deterministic with zero cross-run variance and zero failures, as expected. On DS-2, the 8-record reference archive of Chen et al. (15; 14), the LLM candidates again outperform the rule-based baseline (1) on every element, but the cross-run standard deviations are 4–10× higher than on DS-1. The Calamo et al. candidate (9; 10) with GLM-5.1 (35) achieves the best class (0.655) and relationship (0.689) scores, while the Chen et al. candidate (15; 14) with GLM-5.1 achieves the best attribute score (0.650). The rule-based baseline scores 0.312 on class, 0.353 on attribute, and 0.215 on relationship, so the LLM leads are comparable to DS-1 in absolute terms (0.15–0.47). However, the cross-run standard deviation reaches 0.104 for the Calamo et al. candidate with Kimi K2 (22) on relationship, compared to 0.015 on DS-1 — a 6.9× inflation. The smaller corpus amplifies single-record noise: with only 8 records, one record flipping its per-record score shifts the cell median by roughly 0.06. Validation failure rates are also higher on DS-2: the Chen et al. candidate with GLM-5.1 has a 12.5% failure rate (versus 0.0% on DS-1), and with Kimi K2 it reaches 25.0% (versus 8.9%). The Calamo et al. candidate has zero validation failures on DS-2 across both LLMs, consistent with its single-stage design that emits PlantUML directly. References Abdelnabi et al. (2020) E. A. Abdelnabi, A. M. Maatuk, T. M. Abdelaziz, and S. M. Elakeili Generating UML class diagram using NLP techniques and heuristic rules. In 2020 20th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA), Monastir, Tunisia, p. 277–282. External Links: Document Cited by: §1, §2, §3.4, Table 2, Table 3, §4, §4, §4. Ahmed et al. (2022) S. Ahmed, A. Ahmed, and N. U. Eisty Automatic transformation of natural to unified modeling language: A systematic review. In 2022 IEEE/ACIS 20th International Conference on Software Engineering Research, Management and Applications (SERA), Las Vegas, NV, USA, p. 112–119. External Links: Document Cited by: §1. Arora et al. (2016) C. Arora, M. Sabetzadeh, L. Briand, and F. Zimmer Extracting domain models from natural-language requirements: approach and industrial evaluation. In Proceedings of the ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems (MODELS ’16), New York, NY, USA, p. 250–260. External Links: Document Cited by: §2. Bajwa and Choudhary (2012) I. S. Bajwa and M. A. Choudhary From natural language software specifications to UML class models. In Proceedings of the 13th International Conference on Enterprise Information Systems (ICEIS 2011), Revised Selected Papers, p. 224–237. External Links: Document Cited by: §2. Bajwa et al. (2009) I. S. Bajwa, A. Samad, and S. Mumtaz Object oriented software modeling using NLP based knowledge extraction. European Journal of Scientific Research 35 (1), p. 22–33. Cited by: §2. Ben Abdessalem Karaa et al. (2016) W. Ben Abdessalem Karaa, Z. Ben Azzouz, A. Singh, N. Dey, A. S. Ashour, and H. Ben Ghazala Automatic builder of class diagram (ABCD): an application of UML generation from functional requirements. Software: Practice and Experience 46 (11), p. 1443–1458. External Links: Document Cited by: §2. Bouali et al. (2025) A. Bouali et al. LLM-based grading of software models: an exploratory study. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), p. 423–435. Cited by: §2. Boubekeur et al. (2020) Y. Boubekeur, G. Mussbacher, and S. McIntosh Automatic grading of software modeling assignments. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS ’20), p. 267–278. External Links: Document Cited by: §2. Calamo et al. (2025) M. Calamo, M. Mecella, and M. Snoeck Text2UML: A tool for the automated generation of UML class diagrams from natural language text. In Proceedings of the ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems (MODELS ’25), Cited by: §1, §1, §2, §3.2, §3.4, §3.4, Table 2, Table 2, Table 3, Table 3, §4, §4, §4. Calamo et al. (2026) M. Calamo, M. Mecella, and M. Snoeck Text2UML results with golden UML dataset. Note: https://doi.org/10.5281/zenodo.19599470 Cited by: §2, §3.4, §3.4, Table 2, Table 2, Table 3, Table 3, §4, §4, §4. Cámara et al. (2024) J. Cámara, L. Burgueño, and J. Troya Towards standardized benchmarks of LLMs in software modeling tasks: A conceptual framework. Software and Systems Modeling. External Links: Document Cited by: §1. Cámara et al. (2023) J. Cámara, J. Troya, L. Burgueño, and A. Vallecillo On the assessment of generative AI in modeling tasks: An experience report with ChatGPT and UML. Software and Systems Modeling 22 (3), p. 781–793. External Links: Document Cited by: §1, §1. Cech et al. (2019) I. Cech et al. Matching object-oriented models: a graph-edit-distance approach. In 2019 IEEE 27th International Requirements Engineering Conference (RE), p. 217–227. Cited by: §1, §2, §3.3. Chen et al. (2023a) K. Chen, Y. Yang, B. Chen, J. A. Hernández López, G. Mussbacher, and D. Varró Automated domain modeling with large language models: a comparative study — research artefact. Note: https://doi.org/10.5281/zenodo.8105098 Cited by: §1, §2, §3.2, §3.2, §3.2, §3.4, §3.4, §3.4, §3.4, Table 2, Table 2, Table 3, Table 3, §4, §4, §4. Chen et al. (2023b) K. Chen, Y. Yang, B. Chen, J. A. Hernández López, G. Mussbacher, and D. Varró Automated domain modeling with large language models: a comparative study. In 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS), Västerås, Sweden, p. 162–172. External Links: Document Cited by: §1, §1, §2, §2, §3.2, §3.2, §3.2, §3.3, §3.4, §3.4, §3.4, §3.4, Table 2, Table 2, Table 3, Table 3, §4, §4, §4. Chue Hong et al. (2022) N. P. Chue Hong et al. FAIR4RS principles. In Proceedings of the 21st ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS ’18 Workshop), External Links: Document Cited by: §1. Clavel et al. (2007) M. Clavel, M. S. E. González, and V. T. da Silva The MOVA tool: a rewriting-based UML modeling, measuring, and validation tool. In Proceedings of the XII Jornadas de Ingeniería del Software y Bases de Datos (JISBD 2007), p. 393–394. Cited by: §2. Evans (2004) E. Evans Domain-driven design: tackling complexity in the heart of software. Addison-Wesley Professional. Cited by: §1, §1. Fauzan et al. (2021) R. Fauzan et al. Automated assessment of object-oriented models using structural and lexical similarity. In 2021 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS), p. 112–122. Cited by: §2. Mahmud (2021) H. Mahmud MOSS: a tool to grade object-oriented programming assignments. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (SIGCSE ’21), New York, NY, USA, p. 534–540. External Links: Document Cited by: §2. Mohanan and Samuel (2018) M. Mohanan and P. Samuel Natural language processing approach for UML class model generation from software requirement specifications via SBVR. International Journal on Artificial Intelligence Tools 27 (6), p. 1850027. External Links: Document Cited by: §2. Moonshot AI (2025) Moonshot AI Kimi K2.6 (K2.7-code). Note: https://kimi.moonshot.cn/ Cited by: Table 2, Table 2, Table 3, Table 3, §4, §4, §4. Popescu et al. (2008) D. Popescu, S. Rugaber, N. Medvidovic, and D. M. Berry Reducing ambiguities in requirements specifications via automatically created object-oriented models. In Proceedings of the 14th Monterey Workshop on Software Engineering for Self-Adaptive Systems, p. 103–124. External Links: Document Cited by: §2. Seibert (2025a) V. Seibert Comparing domain-model similarity: A reference implementation of five literature metrics behind one interface, evaluated against human expert ratings. Note: https://doi.org/10.5281/zenodo.20942597 Cited by: §1, §1, §2, §3.3, §4. Seibert (2025b) V. Seibert domain-model-metrics: Python package of five domain-model similarity metrics. Note: https://github.com/VasiliySeibert/domain-model-metrics Cited by: §4. Sharma et al. (2009) V. S. Sharma, S. Sarkar, K. Verma, A. Panayappan, and A. Kass Extracting high-level functional design from software requirements. In Proceedings of the 16th Asia-Pacific Software Engineering Conference (APSEC ’09), p. 35–42. External Links: Document Cited by: §2. Singh et al. (2022) C. K. Singh et al. Detecting mistakes in object-oriented models: a synonym-aware approach. In 2022 IEEE 30th International Requirements Engineering Conference (RE), p. 1–11. Cited by: §1, §2, §3.3. Song et al. (2024) W. Song et al. A deep learning approach to UML use case model similarity. In Proceedings of the 27th International Conference on Model Driven Engineering Languages and Systems (MODELS ’24), Cited by: §2. Stanford Center for Research on Foundation Models (CRFM) (2024) Stanford Center for Research on Foundation Models (CRFM) Holistic Evaluation of Language Models (HELM). Note: https://crfm.stanford.edu/helm/ Cited by: §1. Triandini et al. (2021) E. Triandini et al. Automated class diagram assessment using semantic and structural similarities. International Journal of Intelligent Engineering and Systems. External Links: Document Cited by: §1, §2, §3.3. Verbruggen et al. (2025) C. Verbruggen, L. Netz, P. Glaser, M. Scholz, C. Huemer, M. Calamo, B. Rumpe, M. Snoeck, and D. Bork Toward a community-curated golden dataset of UML models. In Proceedings of the ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems (MODELS ’25 Educators Symposium), New York, NY, USA. Note: Dataset: https://doi.org/10.5281/zenodo.16985873 Cited by: §1, §1, §2, §3.2, §4, §4. Vinay et al. (2009) S. Vinay, S. Aithal, and P. Desai An approach towards automation of requirements analysis. In Proceedings of the International MultiConference of Engineers and Computer Scientists (IMECS 2009), Vol. 1. Cited by: §2. Wilkinson et al. (2016) M. D. Wilkinson et al. The FAIR guiding principles for scientific data management and stewardship. In Scientific Data, External Links: Document Cited by: §1. Yuan et al. (2020) Y. Yuan, Y. Yan, and Y. Ma Structural matching of UML class diagrams: a graph-pattern approach. In 2020 IEEE 28th International Requirements Engineering Conference (RE), p. 1–11. Cited by: §1, §2, §3.3. Zhipu AI (2025) Zhipu AI GLM-5.1/GLM-5.2 Model Card. Note: https://github.com/THUDM/GLM Cited by: Table 2, Table 2, Table 3, Table 3, §4, §4, §4.