Paper deep dive
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis
Viet-Man Le, Thi Ngoc Trang Tran, Sebastian Lubos, Alexander Felfernig, Damian Garber
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/27/2026, 4:19:22 AM
Summary
This study evaluates the capability of 12 Large Language Models (LLMs) to perform 16 Automated Analysis Operations (AOs) on semi-formal textual blueprints for early-stage Software Product Line (SPL) scoping. The research compares LLM performance against the solver-based oracle FLAMA across both general-purpose and reasoning-optimized models. Results indicate that reasoning-optimized models (e.g., Gemini 2.5 Pro, Grok 4 Fast Reasoning) achieve 88-89% accuracy, demonstrating that LLMs can serve as lightweight assistants for validating feature hierarchies and constraints before formal model construction.
Entities (8)
Relation Signals (4)
FLAMA → comparedto → Large Language Models
confidence 100% · we compare their outputs against the solver-based oracle FLAMA
Gemini 2.5 Pro → isa → Reasoning-optimized Model
confidence 100% · reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro)
Large Language Models → perform → Automated Analysis Operations
confidence 100% · We study whether Large Language Models (LLMs) can perform feature model analysis operations (AOs) directly on semi-formal textual blueprints
Universal Variability Language → usedfor → Feature Modeling
confidence 100% · The Universal Variability Language (UVL) has emerged as a unified textual DSL for feature modeling
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We study whether Large Language Models (LLMs) can perform feature model analysis operations (AOs) directly on semi-formal textual blueprints, i.e., concise constrained-language descriptions of feature hierarchies and constraints, enabling early validation in Software Product Line scoping. Using 12 state-of-the-art LLMs and 16 standard AOs, we compare their outputs against the solver-based oracle FLAMA. Results show that reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro) achieve 88-89% average accuracy across all evaluated blueprints and operations, approaching solver correctness. We identify systematic errors in structural parsing and constraint reasoning, and highlight accuracy-cost trade-offs that inform model selection. These findings position LLMs as lightweight assistants for early variability validation.
Tags
Links
- Source: https://arxiv.org/abs/2604.20523v1
- Canonical: https://arxiv.org/abs/2604.20523v1
Trouble viewing inline? Open PDF directly →
Full Text
49,714 characters extracted from source content.
Expand or collapse full text
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis Viet-Man Le v.m.le@tugraz.at Graz University of Technology Graz, Austria Thi Ngoc Trang Tran trang.tran@tugraz.at Graz University of Technology Graz, Austria Sebastian Lubos sebastian.lubos@tugraz.at Graz University of Technology Graz, Austria Alexander Felfernig alexander.felf ernig@tugraz.at Graz University of Technology Graz, Austria Damian Garber damian.garber@tugraz.at Graz University of Technology Graz, Austria Abstract We study whether Large Language Models (LLMs) can perform fea- ture model analysis operations (AOs) directly on semi-formal textual blueprints, i.e., concise constrained-language descriptions of feature hierarchies and constraints, enabling early validation in Software Product Line scoping. Using 12 state-of-the-art LLMs and 16 stan- dard AOs, we compare their outputs against the solver-based oracle FLAMA. Results show that reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro) achieve 88–89% average accuracy across all evaluated blueprints and operations, approaching solver correctness. We identify systematic errors in structural parsing and constraint reasoning, and highlight accuracy–cost trade-offs that in- form model selection. These findings position LLMs as lightweight assistants for early variability validation. CCS Concepts • Software and its engineering→Abstraction, modeling and modularity; Model-driven software engineering; Automated static analysis;• Computing methodologies→Natural language pro- cessing. Keywords Feature Model Analysis Operations, Software Product Line Engi- neering, Software Product Line Scoping, Large Language Models ACM Reference Format: Viet-Man Le, Thi Ngoc Trang Tran, Sebastian Lubos, Alexander Felfernig, and Damian Garber. 2026. Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis. In The 41st ACM/SIGAPP Sympo- sium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, Greece. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3748522.3779903 1 Introduction Software Product Line Engineering (SPLE) systematically manages variability and enables large-scale reuse in software-intensive sys- tems [2, 9]. A critical early stage is Software Product Line (SPL) scoping, where engineers and domain experts define the boundaries This work is licensed under a Creative Commons Attribution 4.0 International License. SAC ’26, Thessaloniki, Greece © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2294-3/2026/03 https://doi.org/10.1145/3748522.3779903 of the product line, identify candidate features, and make assump- tions about variability [25]. Scoping outcomes have a profound influence on all subsequent phases, since they determine which requirements, architectures, and reusable assets will be engineered. Empirical studies confirm that incorrect or incomplete scoping decisions can cascade into costly rework and misalignment with business goals. Industrial practitioners continue to view scoping as one of the most persistent challenges in SPLE adoption [3]. To ensure correct variability decisions, researchers have devel- oped a broad catalogue of automated analysis operations (AOs) for feature models (FMs) [5, 13]. These operations extract information from FMs and verify properties that are difficult or infeasible to as- sess manually, especially as models scale to hundreds or thousands of features [5]. Typical examples include detecting dead or false optional features, validating satisfiability, computing valid config- urations, and estimating configuration-space size. Such analyses are vital for maintaining FM correctness and scalability [13]. They are typically applied only after an FM has been constructed during domain requirements engineering, delaying feedback until late in the process when corrections are costly and disruptive [14, 20]. This paper investigates the potential of applying automated anal- ysis already during the scoping phase, where assumptions about variability are first articulated. We introduce an early validation workflow in which these assumptions are expressed in semi-formal textual blueprints [26] and analyzed directly with Large Language Models (LLMs). The workflow provides immediate feedback on feasibility, consistency, and potential defects before formal feature models are constructed, thereby bridging informal scoping prac- tices with automated validation and enabling domain experts to iteratively refine scope decisions. This early validation step avoids the need for a solver or formal knowledge base, reducing modeling effort and enabling earlier detection and discussion of design issues. Recent advances in LLMs suggest that this approach is feasible. Studies show that, when carefully prompted, LLMs can perform deductive and logical inference beyond surface-level text under- standing [17, 24, 30, 31, 37], and can map natural language into formal specifications such as answer set programs [18], constraint satisfaction problems (CSPs) [16, 26], or Universal Variability Lan- guage (UVL) feature models [11]. These findings indicate that LLMs may function as lightweight inference engines over semi-formal representations like blueprints. However, their out-of-the-box abil- ity to perform feature model AOs on such inputs has not been arXiv:2604.20523v1 [cs.SE] 22 Apr 2026 SAC ’26, March 23–27, 2026, Thessaloniki, GreeceV.M. Le et al. systematically studied. This paper addresses this gap through a large-scale empirical evaluation of 12 state-of-the-art LLMs on 16 AOs, comparing their accuracy, runtime, and failure modes against a solver-based oracle. The contributions of this paper are threefold. Firstly, we formal- ize and motivate a workflow for early product-line validation, which combines semi-formal blueprints with lightweight LLM-based AOs to provide feedback already during SPL scoping. Secondly, we con- duct a large-scale systematic evaluation of off-the-shelf LLMs on feature model analysis tasks using blueprint inputs. Our study covers 12 LLMs (both general-purpose and reasoning-optimized) and 16 AOs across solver-free and solver-based categories. Finally, we systematically evaluate LLM accuracy, cost, and failure modes against the solver-based oracle FLAMA [12], providing insights into their suitability as lightweight assistants for SPL scoping. 2 Background and Related Work 2.1 Software Product Line Engineering The primary objective of SPLE is to reduce development cost and time-to-market while improving quality by deriving families of related products from shared assets rather than engineering each product independently [2, 9]. Its lifecycle is commonly organized into two complementary processes: domain engineering, which es- tablishes reusable core assets, and application engineering, which derives concrete products from them [32]. Domain engineering is typically structured into four phases: (i) scoping, where product- line boundaries, candidate features, and variability assumptions are defined; (i) domain requirements engineering, which captures common and variable requirements; (i) domain design and imple- mentation, which produce reusable architectures and components; and (iv) domain testing, which validates the shared assets before reuse [2]. Application engineering then configures and assembles products from these assets, while evolution and maintenance activ- ities ensure long-term adaptation of the product line [7]. 2.2 Feature Models and Analysis Operations A central artifact of domain requirements engineering is the feature model (FM), which represents a product line’s commonalities and variabilities through a hierarchical feature tree and cross-tree con- straints such as requires and excludes [2, 19]. To validate FMs, the community has established automated analysis operations (AOs) [5, 13]. Solver-free AOs compute structural metrics (e.g., feature counts, tree depth), while solver-based AOs employ SAT/CSP/BDD engines to verify semantic properties such as satisfiability, dead features, and valid configurations [4, 13]. These operations transform FMs from descriptive artifacts into analyzable models, enabling system- atic detection of defects difficult to assess manually at scale [5]. The Universal Variability Language (UVL) has emerged as a unified textual DSL for feature modeling [6]. In this study, we use FLAMA [12] to execute solver-based AOs on UVL inputs as our ground-truth oracle. We focus on the Boolean level of UVL, capturing feature hierarchies and cross-tree constraints. 2.3 SPL Scoping Scoping defines the boundaries of a product line, identifies can- didate features, and establishes variability assumptions [25, 32]. These decisions shape all subsequent engineering activities and are widely recognized as among the most critical yet challenging in SPLE [3]. Current validation practices rely heavily on informal stakeholder workshops, product roadmaps, and expert judgment, leaving con- sistency and feasibility unchecked until formal feature models ex- ist [14, 20]. This delay causes late-stage rework when variability conflicts or infeasible assumptions surface during requirements engineering or implementation. Integrating automated analysis operations (AOs) directly into scoping can close this gap by provid- ing early feedback on variability properties before formalization, enabling domain experts to iteratively refine scope decisions with confidence [5]. 2.4 Large Language Models for Reasoning Large Language Models (LLMs) such as GPT, Claude, and Gemini are transformer-based architectures with up to hundreds of billions of parameters, pre-trained on large text corpora [8]. Beyond tradi- tional NLP tasks, they exhibit emerging reasoning capabilities when properly prompted, including deductive inference, logical implica- tion, and structured problem solving [17, 30, 37]. Benchmarks such as LogicBench [31] and ZebraLogic [24] confirm this potential but also reveal limitations, e.g., accuracy drops under multi-step infer- ence or complex constraints. Studies further note that many models behave as “greedy reasoners,” favoring short reasoning paths and failing under negation or deep logic. Prompting techniques, such as few-shot, chain-of-thought [8, 21, 35], and rationale decomposition [17], can elicit more structured rea- soning without fine-tuning. These approaches have enabled LLMs to generate formal artifacts such as answer-set programs, CSPs, and UVL feature models directly from natural language [11, 16, 18, 26]. For product line engineering, this suggests that well-prompted LLMs may serve as lightweight inference engines for analyzing semi-formal blueprints, providing early validation without relying exclusively on solver technology. 3 Early Validation Workflow for SPL Scoping We propose an early validation workflow that integrates automated analysis directly into the scoping phase. This is achieved by intro- ducing a lightweight, semi-formal blueprint [26] and leveraging LLMs to analyze it. The workflow proceeds in three steps: (1)Step 1 - Blueprint creation: Domain experts consolidate scope boundaries, candidate features, and variability assumptions into a semi-formal blueprint. A blueprint is a set of textual semi- formal constraints that collectively specify the feature hierarchy and cross-tree relations. Examples include “Feature A requires Feature B”, “Feature C can be Feature D or Feature E”, “Feature B excludes Feature D”, or “Feature E can be Feature G, Feature H, or both”. This representation is easy to author yet structured enough for automated checks, bridging free-text scoping notes and formal UVL models. Figure 1 (top-left panel) illustrates a concrete blueprint for a smartwatch product line. (2)Step 2 - LLM-based analysis. The blueprint is fed to an LLM analysis engine that executes AOs. Each AO is guided by a tailored prompt that encodes its reasoning task, ensuring the LLM applies an appropriate inference strategy. The set of AOs Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint AnalysisSAC ’26, March 23–27, 2026, Thessaloniki, Greece is extensible, allowing any operation expressible in natural language to be incorporated into the workflow. (3)Step 3 - Feedback and refinement: The engine returns action- able findings such as inconsistencies, dead features, or product counts. Based on this feedback, the domain expert revises the blueprint and, where necessary, updates scoping outputs. All these three steps are repeated until stabilization. The iteration yields a validated blueprint and updated scoping dossier. These can (i) be compiled to a formal feature model for solver-based verification and (i) seed draft UVL generation to re- duce modeling effort [11]. In short, blueprints capture early variabil- ity decisions in a form LLMs can analyze, providing timely feedback before full formalization. The workflow provides the context for our empirical evaluation. Our study does not assess the workflow itself but investigates a crit- ical question it raises: Are off-the-shelf LLMs capable of performing feature model analysis operations directly on semi-formal blueprints? The next section presents our methodology. 4 Methodology 4.1 Research Questions We evaluate whether off-the-shelf LLMs can execute AOs on semi- formal blueprints and how they compare to a solver oracle (FLAMA [12]). The evaluation is structured around three research questions: RQ1Accuracy. How accurately can different LLMs perform solver- free and solver-based AOs on blueprint inputs? RQ2Cost. How do end-to-end costs (runtime and token usage) relate to achieved accuracy across models? RQ3 Failure modes. What types of errors do LLMs exhibit when performing AOs? How do these vary across model families? 4.2 LLMs Under Study We evaluate 12 publicly available LLMs, spanning both general- purpose (e.g., GPT-4.1, Claude Sonnet 4, DeepSeek Chat) and reasoning-optimized variants (e.g., Grok 4 Fast Reasoning, GPT- 5 mini, Gemini 2.5 Pro). All models are accessed via their official APIs without any fine-tuning, enabling a direct comparison be- tween mainstream and reasoning-focused architectures. To ensure comparability, every model is queried under identi- cal conditions:temperature=0, unrestricted context and output lengths within provider limits, and a uniform prompting and eval- uation pipeline. This setup yields deterministic completions that expose each model’s inherent reasoning behavior. Table 1 lists the evaluated models along with their type, context-window size, and maximum output length. 4.3 Blueprints and Dataset To evaluate LLM-based analysis in a controlled and reproducible set- ting, we derive blueprints from existing UVL feature models rather than from raw scoping inputs. While scoping artifacts represent the natural source for early validation, they are rarely standardized and lack solver-based ground truth. In contrast, UVL models are publicly available, semantically precise, and compatible with solver analysis, enabling reproducible comparison against an oracle. We collected models from two established repositories, UVL- Hub [33] and the Feature-Model-Benchmark v1.0 [34], and selected ten representative cases covering both toy (e.g.,SW,SMW) and large, real-world product lines (e.g.,BDB,CNNl/f). Each model contains at least one cross-tree constraint and varies widely in feature count, tree depth, and constraint density. For each UVL model, we constructed a corresponding blueprint by restating its hierarchy and constraints in constrained natural language, as described in Section 3. For generalization-related AOs, each blueprint was paired with its variant obtained by swapping se- lected relationships (mandatory↔optional, or↔alternative). This setting yields controlled pairs that allow testing if one blueprint’s variability space includes the other. All blueprints were manually verified for semantic equivalence to their UVL sources. Table 2 summarizes structural metrics and blueprint sizes, ranging from fewer than 100 to over 70,000 tokens, thus spanning both compact and complex models. 4.4 Analysis Operations We consider 16 commonly used AOs from feature model research and practice [4, 13], covering both structural metrics (e.g., feature counts, tree depth) and solver-based reasoning tasks (e.g., satisfiabil- ity, core features, configuration counting). Table 3 lists all evaluated AOs. Each AO is implemented through a dedicated prompt template that defines its reasoning steps and enforces a standardized output format for comparison with the solver-based oracle.AO12(#valid configurations) is limited to the first eight FMs due to scalability constraints on the largest cases (CNNl,CNNf). ForAO16(generaliza- tion), each test involves a pair of blueprints: the original and its modified variant (see Section 4.3). 4.5 Evaluation Protocol Prompt Design. For each AO, we use a three-part pipeline illus- trated in Figure 1 using dead feature detection as an example. The figure shows how the system prompt, user prompt, and output con- tract work together to guide the LLM through the analysis of a smartwatch blueprint. We now describe each component: •System prompt. The system prompt defines the model’s role as a domain-specific assistant for feature model analysis. It introduces the blueprint format, explains the elements of feature models (features, hierarchy, and cross-tree con- straints), and specifies the semantics and expected outcome of the target AO using precise definitions and result formats. •User prompt. Combines task instructions, illustrative exam- ples, and the target input. It begins with a Learn from Exam- ples section containing 2–4 exemplars that pair blueprints with their corresponding AO results. A subsequent step-by- step procedure describes the reasoning strategy (e.g., feature enumeration, constraint propagation, contradiction check- ing), followed by the blueprint to be analyzed. This design grounds the task in concrete examples and guides the model through a consistent, reproducible reasoning process. •Output contract. Each AO uses an XML-based output schema to ensure machine-checkable results. The con- tract enforces standardized tags (e.g.,<dead_features>, <core_features>,<feature_model_analysis>)with SAC ’26, March 23–27, 2026, Thessaloniki, GreeceV.M. Le et al. Table 1: Evaluated LLMs ordered by variant type, context-window capacity, and max output length. ModelModel IDTypeContext Window Max Output (tokens)(tokens) Grok 4 Fast Non Reasoning [36] grok-4-non-reasoningGeneral-purpose2MN/A GPT-4.1 [27] gpt-4.1General-purpose1M32K Llama 4 Scout [28] llama-4-scoutGeneral-purpose328K16K Claude Sonnet 4 [1] claude-sonnet-4General-purpose200K64K DeepSeek V3.1 Chat [10] deepseek-chatGeneral-purpose128K8K Grok 4 Fast Reasoning [36] grok-4-reasoningReasoning-optimized2MN/A Gemini 2.5 Flash [15] gemini-2.5-flashReasoning-optimized1M65K Gemini 2.5 Pro [15] gemini-2.5-proReasoning-optimized1M65K Llama 4 Maverick [29] llama-4-maverickReasoning-optimized1M16K GPT-5 mini [27] gpt-5-miniReasoning-optimized400K128K Claude Sonnet 4 Thinking [1] claude-sonnet-4-thinkReasoning-optimized200K64K DeepSeek V3.1 Reasoner [10] deepseek-reasonerReasoning-optimized128K64K Notes: “K” = thousand tokens, “M” = million tokens, “N/A” = not specified. Table 2: Feature models used in the experiments with structural metrics and blueprint sizes. Feature Model SW SMW IDE SMG COM SEA CVE BDB CNNl CNNf #Features6131433481451691173,2966,867 #Relationships481119257315541,5613,516 #Cross-tree constraints12242113153282769 Tree Depth2223310451011 #Blueprint tokens681431783607401,4562,3673,78033,82371,812 Abbreviations: SW=Sandwich, SMW=Smartwatch, IDE=IDE product line, SMG=Strategy Mobile Game, COM=Computer, SEA=Subsea Control System, CVE=Cybersecurity Vulnerability, BDB=Berkeley DB, CNNl=light CNN architectures, CNNf=full CNN architectures. Metrics extracted with FLAMA 2.0.1 and Glucose3; blueprint token counts measured in tokens. Table 3: AOs considered in the experiments. IDSolver-free AOsIDSolver-based AOs AO1#FeaturesAO10Satisfiable/Void AO2#Leaf FeaturesAO11Configuration Satisfiable AO3Tree DepthAO12#Valid Configurations AO4#Mandatory FeaturesAO13Core Features AO5#Optional Features AO14Dead Features AO6#Or GroupsAO15False Optional Features AO7#Alternative GroupsAO16Generalization AO8#Requires AO9#Excludes canonical content: integers for counts,true/falsefor booleans, and newline-separated or tag-enclosed lists for feature sets. Any deviation (e.g., unparseable text, missing tags, malformed lists) is automatically considered incorrect. For set-valued AOs, exact set equality with the solver-based oracle is required. Models are also instructed to include concise natural language justifications, such as explaining why a feature is classified as dead. This uniform pipeline ensures consistent results across models and AOs by prioritizing fair capability comparison over tailored optimization. Full templates for all 16 AOs are provided in the replication package 1 . Inference. Each triple (including model, blueprint, and AO) is ex- ecuted once with deterministic decoding (temperature=0, greedy) via the provider’s public API only, i.e., no tool use, no solver calls, no post-processing. We record request/response payloads, the raw XML, the rationale, and wall-clock runtime. Besides, timeouts and truncations are kept (and evaluated) as produced. Metrics and error taxonomy. We report the results along the fol- lowing two metrics: •Accuracy. Exact-match agreement with the FLAMA oracle, broken down by AO, blueprint, and model family (general- purpose vs. reasoning-optimized). •Cost. End-to-end runtime (in seconds) and token usage (prompt+completion). Errors are categorized into four mutually exclusive failure modes aligned with our analysis: (i) Unparseable (violates XML contract), (i) Format-correct but wrong (includes semantic slips such as mis- reading alternative vs. mandatory), (i) Partial/Truncated (incom- plete outputs due to context/output limits), and (iv) Hallucinated elements (items not in the blueprint). 1 Replication package: https://github.com/AIG-ist-tugraz/llm-blueprint-analysis Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint AnalysisSAC ’26, March 23–27, 2026, Thessaloniki, Greece <feature_model_analysis> <dead_features>[[analog]]</dead_features> <explanation> [analog] is a dead feature because [screen] is mandatory, and [screen] must be either [analog] or [hires]. However, [screen] also requires [camera], and [camera] in turn requires [hires]. This creates a conflict if [analog] is selected, as it cannot satisfy the transitive requirement for [hires]. Therefore, [hires] is always included, and [analog] can never be selected. </explanation> </feature_model_analysis> Analysis Results LLMs - A [smartwatch] must have at least one type of [connector] and a [screen]. - The [connector] can be [gps], [cellular], or both. - The [screen] can be [analog] or [hires] (high resolution). - The [smartwatch] may include a [camera] and a [compass]. - [compass] requires a [gps]. - [camera] requires a [hires]. - [cellular] and [analog] exclude each other. - If the [screen] is selected, a [camera] must also be included. Feature Model Blueprint You are a domain-specific AI assistant specializing in analyzing feature models described in “Blueprint Format”. Your purpose is to: - Interpret the structural and semantic elements of a feature model... - Detect dead features using logical reasoning and configuration validation. - Provide explanations grounded in feature relationships and constraint interactions. Use the following foundational knowledge to support your analysis: Feature Model A feature model is a structured representation... Formal Representation A feature configuration task is defined as a triple (F, D, C) where:... Additional Concepts - Dead features: Features that *cannot* be included in *any* valid configuration. Blueprint Format The Blueprint format is a bullet-point, human-readable way for describing feature models... Each bullet clarifies the relationships between features,... System Prompt Now, analyze the following feature model. Follow these steps: Step 1: Learn from Examples: Use the annotated examples below to understand how dead features are inferred... Step 2: Reasoning: For each feature: - Attempt to construct a valid configuration where: - That feature is selected. - Satisfies all relationships and constraints - Avoids triggering any exclusion or logical conflict - If no valid configuration exists due to unsatisfiable relationship/constraint interactions, classify the feature as dead. - Explain your reasoning using structural and logical justifications. Wrap your detailed reasoning inside:... *Output Format* At the end, wrap your final reasoning in: ... Step 3: Analysis this feature model: Now, analyze the following model to identify any dead features. <feature_model_blueprint_to_analyze> feature_model_blueprint </feature_model_blueprint_to_analyze> User Prompt Figure 1: Prompting pipeline for dead feature detection, showing how the system and user prompts guide reasoning and how the LLM outputs XML results for solver comparison. 4.6 Baseline Oracle and Implementation We use FLAMA 2.0.1 [12] as the solver-based oracle, executing AOs on UVL inputs through the Glucose3 SAT solver and the D library to obtain exact and reproducible ground-truth results for all comparisons. The LLM-based analysis engine and the evalua- tion framework are implemented in Python 3.10 within a unified evaluation harness. To ensure modularity and reproducibility, we employ LangChain [22] for standardized API integration, and Lang- Graph [23] for orchestrating prompting pipelines. All prompts, blueprints, model settings, evaluation scripts, raw outputs, and per-run runtimes are publicly released in the replication package 1 . 5 Results Full tables and results are shown in Appendix 2 . 5.1 RQ1: Accuracy of LLM-based AOs Figure 2 and Figure 3 summarize accuracy values across all 16 AOs. Overall, reasoning-optimized LLMs clearly outperform general- purpose ones, achieving an average accuracy of 81.1% compared to 61.0%. This gap reflects stronger multi-step reasoning and con- straint handling. Accuracy patterns, however, vary markedly across operation types and blueprint complexity. Accuracy Across AOs. Solver-based AOs yield higher accura- cies than solver-free ones for both model families: 68.8% vs. 56.0% for general-purpose and 83.5% vs. 79.2% for reasoning-optimized 2 Supplementary appendix: https://doi.org/10.5281/zenodo.17913681 AO1AO2AO3AO4AO5AO6AO7AO8AO9 AO10AO11AO12AO13AO14AO15AO16 claude- sonnet-4 gpt-4.1 grok-4-non- reasoning deepseek-chat llama-4-scout 10050804070100708090100905090807090 504050305010070708010010050807060100 404030205090806080908050507060100 303060104010070608080802560605090 30203020304050407090901340402040 Figure 2: Accuracy (%) of general-purpose LLMs across AOs. AO1AO2AO3AO4AO5AO6AO7AO8AO9 AO10AO11AO12AO13AO14AO15AO16 grok-4- reasoning gpt-5-mini gemini-2.5-pro claude- sonnet-4-think deepseek- reasoner gemini-2.5- flash llama-4- maverick 1008010080901008090100100100100609070100 100807080801008010010010010075909080100 10070908080100809010010090100909080100 100708060801007080901001007590908080 908010030501008090100100907590807090 606080508080807080100906370806080 603030306080706080701001360604090 Figure 3: Accuracy (%) of reasoning-opt. LLMs across AOs. LLMs. Rather than requiring exhaustive enumeration, these solver- based tasks are operationalized as verification. For instance, inAO10, the model is asked to search for contradictions in the blueprint and concludes “satisfiable” if none are found.AO11analogously checks whether a given configuration violates any constraint be- fore deciding its satisfiability.AO16reduces to judging whether SAC ’26, March 23–27, 2026, Thessaloniki, GreeceV.M. Le et al. SW SMWIDESMGCOMSEACVEBDB CNNlCNNf claude- sonnet-4 gpt-4.1 grok-4-non- reasoning deepseek-chat llama-4-scout 1001009488946288563863 1001008181566250693150 100948181565050443125 94816962445069441956 75885050312544381313 Figure 4: Accuracy (%) of general-purpose LLMs on 16 AOs across 10 blueprints. one model subsumes another (i.e., no counterexample is identified). This verification-style framing lowers the need for complete search and tends to be more robust to local parsing noise. By contrast, solver-free AOs (e.g.,AO4–AO9) require the model to parse and count structural relationships such as mandatory, optional, or alternative features; small semantic misunderstandings, e.g., interpreting “A must have B or C” as two mandatory children instead of an alterna- tive group, lead to systematic counting errors and lower accuracies. Difficulty levels vary markedly across AOs and model families. For general-purpose LLMs, the hardest AOs areAO2(leaf count), AO4(mandatory count), andAO12(#valid configurations), all av- eraging below 50% accuracy, reflecting persistent semantic errors and limited reasoning capacity. In contrast, reasoning-optimized LLMs achieve high performance (≥85%) on most tasks but still struggle withAO4(58.6%) due to semantic misunderstandings, and moderately on AO12 and AO15, which demand constraint propaga- tion or enumeration. Across both families, the easiest AOs areAO6, AO9,AO10,AO11, andAO16(all above 85%), which involve simpler verification steps rather than complex structural reasoning. Accuracy Across Blueprints. Figures 4 & 5 show accuracies aggre- gated by blueprint. Across both model families, accuracy decreases steadily with increasing blueprint complexity. All models achieve near-perfect performance on small and shallow blueprints (SW,SMW), moderate accuracy on medium-sized ones (IDE,SMG,COM), and sub- stantial drops on complex models such asSEA,BDB,CNNl, andCNNf. These challenging cases combine large feature counts (up to 7,000), deep hierarchies (depth≥10), and numerous cross-tree constraints (>300), which amplify semantic misunderstandings and reasoning limitations. Importantly, no single model achieves 100% accuracy across all blueprints, highlighting the difficulty of handling large, constraint-dense feature models. Model Ranking. Among reasoning-optimized LLMs, three mod- els form the top-performing group: Grok 4 Fast Reasoning (89.7%), GPT-5 mini (88.9%), and Gemini 2.5 Pro (88.2%) (see Figure 5). They achieve perfect scores on several simple AOs and maintain stable performance on complex ones, suggesting potential for ensemble- based aggregation. The second-tier reasoning models, Claude Son- net 4 Thinking (84.5%) and DeepSeek Reasoner (82.8%), show mod- erate drops onAO12(both 75%) andAO15(80% and 70%). They also exhibit structural counting weaknesses: Claude Sonnet 4 Thinking performs lower onAO4(60%) andAO2/AO7(70%), while DeepSeek Reasoner falls sharply onAO4(30%) andAO5(50%). Gemini 2.5 Flash SW SMWIDESMGCOMSEACVEBDB CNNlCNNf grok-4- reasoning gpt-5-mini gemini-2.5-pro claude- sonnet-4-think deepseek- reasoner gemini-2.5- flash llama-4- maverick 100100100100949488886073 10010010010010010094755367 10010010010010081100815367 100100100941006994814067 100948188948881756067 10010010088946262752727 100948181505644381327 Figure 5: Accuracy (%) of reasoning-optimized LLMs on 16 AOs across 10 blueprints. Table 4: Average runtime (in second), token usage, and accu- racy across all AOs and blueprints for representative models. Model IDRuntime (s) Tokens Accuracy (%) grok-4-reasoning3,046.7505,87489.7 gemini-2.5-pro2,390.8602,25188.2 gpt-5-mini6,118.3843,25188.9 deepseek-reasoner9,508.4561,29082.8 claude-sonnet-41,168.5439,52378.3 llama-4-scout680.2374,60842.7 FLAMA (Solver) † 18.45–100.0 † Results on formal inputs; solver cannot process semi-formal blueprints. and Llama 4 Maverick perform worse (60–75%), mainly due to con- text overflow and output truncation. Within the general-purpose family (see Figure 4), Claude Sonnet 4 leads with 78.3%, followed by GPT-4.1 and Grok 4 Fast Non Reasoning (60–68%), while Llama 4 Scout is the weakest (42.7%). Takeaway. Reasoning-optimized models consistently outper- form general-purpose ones on blueprint AOs, but accuracy declines with blueprint size/depth and remains uneven for structural counts (e.g., mandatory/alternative counting). 5.2 RQ2: Efficiency of LLM-Based AOs Since AOs demand high precision, computational cost is mean- ingful only in relation to achieved accuracy. A model that spends more time but delivers correct results is preferable over a faster yet unreliable one. Table 4 reports average runtime, token usage, and accuracy across all AOs and blueprints. Reasoning-optimized LLMs are notably more resource-intensive, requiring 2.4k–9.5k sec- onds and 500K–840K tokens, compared to 0.7k–1.9k seconds and 370K–440K tokens for general-purpose models. The symbolic solver baseline (FLAMA) remains by far the fastest (18.45 seconds) but cannot process semi-formal blueprints directly. Accuracy–Cost Balance. High computational cost does not neces- sarily guarantee better accuracy. For instance, DeepSeek Reasoner is the slowest model (9,508 seconds) yet achieves only 82.8% accuracy, Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint AnalysisSAC ’26, March 23–27, 2026, Thessaloniki, Greece while Grok 4 Fast Reasoning reaches 89.7% with one-third of the run- time. In contrast, Gemini 2.5 Pro delivers similar accuracy (88.2%) with the lowest runtime among reasoning models (2,391 seconds), representing the most efficient high-accuracy trade-off. GPT-5 mini attains top accuracy (88.9%) but at substantially higher runtime and token cost, suggesting diminishing returns. Meanwhile, general- purpose models such as Claude Sonnet 4 offer a moderate balance (78.3%, 1,169 seconds), and lightweight models like Llama 4 Scout are fast yet unreliable (42.7%). 5.3 RQ3: Error Analysis We identify three dominant failure modes. First, semantic slips in solver-free AOs (AO4–AO9), most notably misreading “A must have B or C” as two mandatory children instead of an alternative group. Second, incomplete propagation or enumeration in solver-based AOs, especiallyAO12(#valid configurations) andAO15(#false optional), which require constraint reasoning beyond surface parsing. Third, context and output limits on very large blueprints (CNNl,CNNf), causing truncation or early stopping. These problems intensify in deep or constraint-dense models (SEA,BDB,CNNl/f) and are most pronounced inAO12,AO15, and structural AOs (AO4–AO9). Detailed counts per model and AO are in Appendix 2 . Model behaviors reflect these patterns. Grok 4 Fast Reasoning, GPT-5 Mini, and Gemini 2.5 Pro make few but systematic propaga- tion errors. Claude Sonnet 4 often overcounts mandatory features due to confusion between or/alternative and mandatory relation- ships. DeepSeek Reasoner shows partial outputs while Gemini 2.5 Flash and Llama 4 Maverick suffer truncation near context lim- its. Lightweight models (Llama 4 Scout, DeepSeek Chat) tend to produce format-correct but semantically wrong answers. These recurring patterns underscore the need for disambiguation rules, output chunking, and model ensembles. 5.4 Overall Synthesis Across RQ1–RQ3, reasoning-optimized LLMs approach solver-level accuracy on many analyses but remain sensitive to structural ambi- guity, scale, and context limits. Most errors stem from incomplete constraint reasoning rather than random noise. In summary, LLMs can already serve as reliable early validators of variability models when paired with simple safeguards and ensemble strategies. 6 Discussion Where LLMs fit in early scoping. Our results support a pragmatic placement of LLMs at the earliest stage, when blueprints are semi- formal and solver-ready model is not yet available. In this context, reasoning-oriented models provide verification-style feedback with near-solver accuracy (e.g., forAO10,AO11,AO16), quick structural sanity checks (e.g., forAO1,AO3,AO6,AO9), and concise explanations that help stakeholders refine scope decisions before formalization. Accuracy and explainability over speed. Because early validation is iterative and not latency-critical, we prioritize correctness and interpretability over raw speed. The observed runtime/token over- heads are acceptable in practice given the benefit of solver-like guidance without requiring a formal model. Reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro, GPT-5 mini) therefore offer a practical operating point: near-solver accuracy with stable formatting and useful rationales that fit within minutes- scale review cycles typical of scoping workshops. Leveraging model complementarity. Top performers exhibit com- plementary strengths across blueprints and AOs. Grok 4 Fast Rea- soning demonstrates exceptional robustness on large, constraint- dense models such asBDB,CNNl, andCNNf, where other models suffer truncation or incomplete propagation. In contrast, GPT-5 mini achieves the highest consistency on mid-size blueprints (COM, SEA) and complex solver-based tasks (AO12,AO15), benefiting from deeper constraint reasoning. Gemini 2.5 Pro performs best on com- pact to medium blueprints (CVE,SMG) and excels in maintaining strict XML conformance and stable formatting even under long outputs. These differences suggest that the models compensate for each other’s weaknesses: Grok handles scale and constraint den- sity, GPT-5 mini excels in fine-grained constraint reasoning, and Gemini maintains strict output conformance. Simple ensemble ag- gregation (majority vote or confidence-weighted fusion) mitigates idiosyncratic errors and raises reliability with minimal engineering, matching the “low-friction” goal of early validation. Actionable guidance. To harden early analyses, we recommend four concrete measures aligned with the observed failure modes: (i) add short disambiguation rules that test group semantics before any counting (distinguish or/alternative from mandatory) to reduce semantic slips inAO4–AO9, (2) plan outputs for long blueprints by chunking lists and capping free text to avoid truncation and partial results on large cases (e.g.,CNNl/CNNf), (i) use model ensembles on medium and large blueprints (majority vote or confidence-weighted fusion across top models) to smooth per-model weaknesses in prop- agation and enumeration (notablyAO12andAO15), (iv) maintain human oversight through stakeholder review and verification pro- tocols to prevent automation bias, as approximately 1 in 10 analyses may contain errors. Threats to Validity. Our evaluation uses blueprints derived from UVL models to ensure reproducibility and oracle comparabil- ity. While this enables consistent benchmarking, it may under- represent ambiguity and noise found in truly informal scoping artifacts. Moreover, the 16 evaluated AOs cover representative but not exhaustive FM tasks. We also evaluated a broad set of public LLM APIs, but not all commercially or academically available mod- els. Results may vary for future versions or domain-specific fine- tuning. Finally, our strict exact-match scoring penalizes partially correct outputs, which may underestimate the practical reasoning capabilities of models in near-missing cases. 7 Conclusion and Future Work We evaluated whether off-the-shelf LLMs can perform feature model AOs directly on semi-formal blueprints, where early feedback is most valuable. Reasoning-optimized models reach near-solver accuracy and offer actionable explanations, making them practical assistants for SPL scoping. Their minute-scale runtime is acceptable for accurate, automation-ready feedback before formalization. Re- maining issues, such as semantic slips and incomplete propagation, can be mitigated through ensemble use and prompt safeguards. Overall, LLMs provide a feasible path to automated, explainable SAC ’26, March 23–27, 2026, Thessaloniki, GreeceV.M. Le et al. variability checks prior to solver-based verification, though human oversight remains necessary given their 88–89% accuracy. Future work will expand both the analytical scope and the work- flow. We plan to include additional AO (e.g., redundancy, atomic sets, diagnosis/conflict sets) to better approximate the full AO cata- logue, and to conduct workflow evaluations in practice with domain engineers to assess usability and feedback quality. We also aim to support blueprint-to-UVL translation with round-trip consistency checks and to develop learned routers that select the most reliable model. Finally, we envision extending the workflow toward scope completeness checks, where LLMs ensure traceability between goals, scenarios, and features—detecting orphaned or missing links to reduce underscoping and misalignment with business objectives. Acknowledgments The work presented in this paper has been developed within the research project GenRE funded by the Austrian Research Promotion Agency under the project number 915086. References [1]2025. All models overview - Anthropic — docs.anthropic.com. Retrieved Sep- tember 29, 2025 from https://docs.claude.com/en/docs/about-claude/models/o verview#model-comparison-table. [2]S. Apel, D. Batory, C. Kästner, and G. Saake. 2013. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer. [3] M. Becker, R. Rabiser, and G. Botterweck. 2024. Not quite there yet: remaining challenges in systems and software product line engineering as perceived by industry practitioners. In Proceedings of the 28th ACM International Systems and Software Product Line Conference (SPLC ’24). ACM, Dommeldange, Luxembourg, 179–190. doi:10.1145/3646548.3672587. [4]D. Benavides, A. Felfernig, J. Galindo, and F. Reinfrank. 2013. Automated Analysis in Feature Modelling and Product Configuration. In ICSR’13 (LNCS) number 7925. Springer, Pisa, Italy, 160–175. [5]D. Benavides, S. Segura, and A. Ruiz-Cortes. 2010. Automated analysis of feature models 20 years later: A literature review. Inf. Sys., 35, 615–636, 6. [6]D. Benavides, C. Sundermann, K. Feichtinger, J.A. Galindo, R. Rabiser, and T. Thüm. 2025. UVL: feature modelling with the universal variability language. Journal of Systems and Software, 225, 112326. doi:https://doi.org/10.1016/j.jss.2 024.112326. [7] T. Berger, J.-P. Steghöfer, T. Ziadi, J. Robin, and J. Martinez. 2020. The state of adoption and the challenges of systematic variability management in industry. Empirical Software Engineering, 25, 3, (May 2020), 1755–1797. doi:10.1007/s106 64-019-09787-6. [8]T. Brown et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, (Eds.) Vol. 33. Curran Associates, Inc., 1877–1901. [9]P.C. Clements and L. Northrop. 2002. Software product lines. Addison-wesley. [10] 2025. Models & Pricing | DeepSeek API Docs — api-docs.deepseek.com. Re- trieved September 29, 2025 from https://api-docs.deepseek.com/quick_start/pr icing/. [11]J. A. Galindo, A. J. Dominguez, J. White, and D. Benavides. 2023. Large language models to generate meaningful feature model instances. In Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A (SPLC ’23). ACM, Tokyo, Japan, 15–26. doi:10.1145/3579027.3608973. [12]J. A. Galindo, J.-M. Horcas, A. Felferning, D. Fernandez-Amoros, and D. Be- navides. 2023. Flama: a collaborative effort to build a new framework for the automated analysis of feature models. In Proceedings of the 27th ACM Inter- national Systems and Software Product Line Conference - Volume B (SPLC ’23). ACM, Tokyo, Japan, 16–19. doi:10.1145/3579028.3609008. [13] J.A. Galindo, D. Benavides, P. Trinidad, A. Gutiérrez-Fernández, and A. Ruiz- Cortés. 2019. Automated Analysis of Feature Models: Quo Vadis? In 23rd Inter- national Systems and Software Product Line Conference - Volume A (SPLC ’19). ACM, Paris, France, 302. doi:10.1145/3336294.3342373. [14]S. Ghosh, D. Elenius, W. Li, P. Lincoln, N. Shankar, and W. Steiner. 2016. Arse- nal: automatic requirements specification extraction from natural language. In NASA Formal Methods. S. Rayadurgam and O. Tkachuk, (Eds.) Springer International Publishing, Cham, 41–46. [15] 2025. Gemini models | Gemini API | Google AI for Developers — ai.google.dev. Retrieved September 29, 2025 from https://ai.google.dev/gemini-api/docs/mod els. [16]L. Hotz, C. Bähnisch, S. Lubos, A. Felfernig, A., and J. Twiefel. 2024. Exploiting large language models for the automated generation of constraint satisfaction problems. 26th International Workshop on Configuration, ConfWS 2024. CEUR Workshop Proceedings, 3812, 91–100. [17]J. Huang and K. C.-C. Chang. 2023. Towards reasoning in large language models: a survey. In Findings of the Association for Computational Linguistics: ACL 2023. A. Rogers, J. Boyd-Graber, and N. Okazaki, (Eds.) Association for Computational Linguistics, Toronto, Canada, (July 2023), 1049–1065. doi:10.18 653/v1/2023.f indings-acl.67. [18]A. Ishay, Z. Yang, and J. Lee. 2023. Leveraging Large Language Models to Gen- erate Answer Set Programs. In Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning. (Aug. 2023), 374–383. doi:10.24963/kr.2023/37. [19]K. Kang, S. Cohen, J. Hess, W. Novak, and S. Peterson. 1990. Feature-oriented Domain Analysis (FODA) – Feasibility Study. Tech.Rep. – SEI-90-TR-21. [20]C. Khor and R. R. Lutz. 2024. Enhancing the requirements engineering of configurable systems by the ongoing use of variability models. Requirements Engineering, 29, 3, (Sept. 2024), 303–328. doi:10.1007/s00766-024-00421-6. [21]T. Kojima, S. (S.) Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, (Eds.) Vol. 35. Curran Associates, Inc., 22199–22213. [22]LangChain. 2025. GitHub - langchain-ai/langchain: Build context-aware rea- soning applications — github.com. Retrieved April 20, 2025 from https://github .com/langchain-ai/langchain. [23] LangChain. 2025. GitHub - langchain-ai/langgraph: Build resilient language agents as graphs. — github.com. Retrieved April 20, 2025 from https://github.c om/langchain-ai/langgraph. [24]B. Y. Lin, R. Le Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi. 2025. Zebralogic: on the scaling limits of LLMs for logical reasoning. In Forty-second International Conference on Machine Learning. [25] L. Marchezan, E. Rodrigues, W. K. G. Assunção, M. Bernardino, F. P. Basso, and J. Carbonell. 2022. Software product line scoping: a systematic literature review. In Proceedings of the 26th ACM International Systems and Software Product Line Conference - Volume A (SPLC ’22). ACM, Graz, Austria, 256. doi:10.1145/354693 2.3547012. [26] K. Michailidis, D. Tsouros, and T. Guns. 2024. Constraint Modelling with LLMs Using In-Context Learning. In 30th International Conference on Principles and Practice of Constraint Programming (CP 2024) (Leibniz International Proceedings in Informatics (LIPIcs)). Vol. 307. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 20:1–20:27. doi:10.4230/LIPIcs.CP.2024.20. [27]2025. OpenAI models. Retrieved September 29, 2025 from https://platf orm.ope nai.com/docs/models. [28] 2025. Llama 4 Scout - API, Providers, Stats — openrouter.ai. Retrieved September 29, 2025 from https://openrouter.ai/meta-llama/llama-4-scout. [29] 2025. Llama 4 Maverick - API, Providers, Stats — openrouter.ai. Retrieved September 29, 2025 from https://openrouter.ai/meta-llama/llama-4-maverick. [30]L. Pan, V. Ganesh, J. Abernethy, C. Esposo, and W. Lee. 2025. Can transformers reason logically? a study in SAT solving. In Forty-second International Confer- ence on Machine Learning. [31] M. Parmar, N. Patel, N. Varshney, M. Nakamura, M. Luo, S. Mashetty, A. Mitra, and C. Baral. 2024. Logicbench: towards systematic evaluation of logical rea- soning ability of large language models. In ACL (1), 13679–13707. https://doi.o rg/10.18653/v1/2024.acl-long.739. [32]K. Pohl, G. Böckle, and F. J. van der Linden. 2010. Software Product Line Engi- neering: Foundations, Principles and Techniques. (1st ed.). Springer Publishing Company, Incorporated. [33]D. Romero-Organvidez, J. A. Galindo, C. Sundermann, J.-M. Horcas, and D. Benavides. 2024. Uvlhub: a feature model data repository using uvl and open science principles. Journal of Systems and Software, 216, 112150. doi:https://doi .org/10.1016/j.jss.2024.112150. [34]C. Sundermann, V. F. Brancaccio, E. Kuiter, S. Krieter, T. Heß, and T. Thüm. 2024. Collecting feature models from the literature: a comprehensive dataset for benchmarking. In Proceedings of the 28th ACM International Systems and Software Product Line Conference (SPLC ’24). ACM, Dommeldange, Luxembourg, 54–65. doi:10.1145/3646548.3672590. [35]J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter b., F. Xia, E. Chi, Q. V. Le, and D. Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, (Eds.) Vol. 35. Curran Associates, Inc., 24824–24837. [36]XAI. 2025. Models and Pricing | xAI Docs — docs.x.ai. Retrieved September 29, 2025 from https://docs.x.ai/docs/models. [37]J. Yan, C. Wang, J. Huang, and W. Zhang. 2024. Do large language models understand logic or just mimick context? CoRR, abs/2402.12091. https://doi.org /10.48550/arXiv.2402.12091.