Paper deep dive
Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations
Anna Arnaudo, Riccardo Coppola, Maurizio Morisio, Flavio Giobergia, Andrea Bioddo, Angelo Bongiorno, Luca Dadone
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/27/2026, 2:32:28 AM
Summary
This paper evaluates a multi-agent LLM-based architecture designed to automate the Goal-Oriented Requirements Engineering (GORE) process. The proposed system uses a fixed-structure chain involving two LLMs: GPT-4 as a 'generator' and Llama 3.3 70B as an 'evaluator' (critic) in an iterative feedback loop. The pipeline consists of documentation preprocessing, actor identification, high-level goal extraction, and low-level goal extraction. The study finds that a Zero-shot prompting strategy combined with a feedback loop outperforms stand-alone Few-shot prompting, though the system's accuracy (61% for low-level goals) suggests it is currently best suited as an acceleration tool for manual extraction rather than a full replacement. Future work aims to integrate RAG and Chain-of-Thought prompting.
Entities (8)
Relation Signals (4)
Llama 3.3 70B → actsasevaluatorin → multi-agent architecture
confidence 100% · Llama 3.3 70B operates as the 'evaluator'
GPT-4 → actsasgeneratorin → multi-agent architecture
confidence 100% · GPT-4 functions as the 'generator'
multi-agent architecture → automates → Goal-Oriented Requirements Engineering
confidence 90% · discuss a possible approach for automating the Goal-Oriented Requirements Engineering (GORE) process
Zero-shot prompting → outperforms → Few-shot prompting
confidence 90% · The feedback-loop mechanism with Zero-shot outperformed stand-alone Few-shot
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Due to the textual and repetitive nature of many Requirements Engineering (RE) artefacts, Large Language Models (LLMs) have proven useful to automate their generation and processing. In this paper, we discuss a possible approach for automating the Goal-Oriented Requirements Engineering (GORE) process by extracting functional goals from software documentation through three phases: actor identification, high and low-level goal extraction. To implement these functionalities, we propose a chain of LLMs fed with engineered prompts. We experimented with different variants of in-context learning and measured the similarities between input data and in-context examples to better investigate their impact. Another key element is the generation-critic mechanism, implemented as a feedback loop involving two LLMs. Although the pipeline achieved 61% accuracy in low-level goal identification, the final stage, these results indicate the approach is best suited as a tool to accelerate manual extraction rather than as a full replacement. The feedback-loop mechanism with Zero-shot outperformed stand-alone Few-shot, with an ablation study suggesting that performance slightly degrades without the feedback cycle. However, we reported that the combination of the feedback mechanism with Few-shot does not deliver any advantage, possibly suggesting that the primary performance ceiling is the prompting strategy applied to the 'critic' LLM. Together with the refinement of both the quantity and quality of the Shot examples, future research will integrate Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting to improve accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2604.22207v1
- Canonical: https://arxiv.org/abs/2604.22207v1
Trouble viewing inline? Open PDF directly →
Full Text
60,856 characters extracted from source content.
Expand or collapse full text
Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations Anna Arnaudo anna.arnaudo@polito.it Department of Control and Computer Engineering, Politecnico di Torino Torino, IT Riccardo Coppola Department of Control and Computer Engineering, Politecnico di Torino Torino, IT Maurizio Morisio Department of Control and Computer Engineering, Politecnico di Torino Torino, IT Flavio Giobergia Department of Control and Computer Engineering, Politecnico di Torino Torino, IT Andrea Bioddo Politecnico di Torino Torino, IT Angelo Bongiorno Politecnico di Torino Torino, IT Luca Dadone Politecnico di Torino Torino, IT Abstract Due to the textual and repetitive nature of many Requirements Engineering (RE) artefacts, Large Language Models (LLMs) have proven useful to automate their generation and processing. In this paper, we discuss a possible approach for automating the Goal- Oriented Requirements Engineering (GORE) process by extract- ing functional goals from software documentation through three phases: actor identification, high and low-level goal extraction. To implement these functionalities, we propose a chain of LLMs fed with engineered prompts. We experimented with different variants of in-context learning and measured the similarities between in- put data and in-context examples to better investigate their im- pact. Another key element is the generation-critic mechanism, implemented as a feedback loop involving two LLMs. Although the pipeline achieved 61% accuracy in low-level goal identifica- tion — the final stage — these results indicate the approach is best suited as a tool to accelerate manual extraction rather than as a full replacement. The feedback-loop mechanism with Zero-shot outperformed stand-alone Few-shot, with an ablation study sug- gesting that performance slightly degrades without the feedback cycle. However, we reported that the combination of the feedback mechanism with Few-shot does not deliver any advantage, possibly suggesting that the primary performance ceiling is the prompting strategy applied to the ’critic’ LLM. Together with the refinement of both the quantity and quality of the Shot examples, future re- search will integrate Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting to improve accuracy. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. EASE 2026, Glasgow, Scotland, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n CCS Concepts • Software and its engineering→Requirements analysis; • General and reference→Empirical studies;• Computing methodologies→ Natural language generation. Keywords Large Language Models, Software Engineering, Requirements En- gineering, Goal Oriented Requirements Engineering, Prompt Engi- neering ACM Reference Format: Anna Arnaudo, Riccardo Coppola, Maurizio Morisio, Flavio Giobergia, An- drea Bioddo, Angelo Bongiorno, and Luca Dadone. 2026. Evaluating LLM- Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/n.n 1 Introduction Rising interest in LLMs for software engineering [24] highlights their effectiveness in analysing and generating structured arti- facts [2,15]. In the field of Requirements Engineering (RE) some studies have explored the extraction of goal models from natural language requirement specifications [7], the extraction of domain models from textual requirements [5], the enhancement of Use Case definition with LLM-based agents [8], goal-model genera- tion from user stories [20], class/behavioral model synthesis and benchmarking [6]. While most current applications of LLMs in software engineering rely on limited interactions with a single model instance, emerging research highlights a shift toward more collaborative paradigms. Integrating Large Language Models into multi-agent systems (MAS) represents a significant advancement, enabling agents to assume specialised roles, coordinate their actions, and collectively address complex software engineering challenges [12,25]. Drawing inspi- ration from the literature, we envision a system architecture organ- ised around multiple LLMs emulating the different steps typically involved in complex processes. arXiv:2604.22207v1 [cs.SE] 24 Apr 2026 EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United KingdomArnaudo et al. Moreover, increasing attention is directed towards prompting strategies and their effectiveness in supporting software engineer- ing[]. For this reason, we experimented with different variants of in-context learning to assess how they affect the proposed archi- tecture. The ultimate objective of our architecture is the refinement of high-level goals into increasingly concrete, operationalised forms, thus obtaining the list of the system’s functional requirements. Although mapping from low-level goals to API endpoints is not the main focus of this study, we include it in our conceptual architecture because it may provide useful insights about future extensions of this work. The contributions of this work can be listed as follows: •We proposed a novel architecture that decomposes the GORE process into multiple steps involving a feedback-loop mech- anism; • We evaluated the proposed architecture by varying the prompt- ing strategy of the generator model; •We demonstrated that the combination of Zero-shot prompt- ing with a feedback-loop outperforms Few-shot prompting applied to GPT alone in the target tasks; •We measured the similarity between the Few-shot and the ground truth examples, analysing the possible implications on the performances of our system. The replication package of the study is available on Zenodo 1 . 1.1 Research Questions (RQs) We formalised the evaluation of the proposed architecture through the following RQs: • RQ1: What is the effectiveness of the multi-agent architec- ture in extracting the list of actors, and how it is influenced by Shot-prompting? •RQ2: What is the effectiveness of the multi-agent architec- ture in modelling high-level goals, and how it is influenced by Shot-prompting? •RQ3: What is the effectiveness of the multi-agent architec- ture in decomposing high-level goals into low-level goals, and how it is influenced by Shot-prompting? • RQ4: Does the critic mechanism actually improve the ex- traction of actors, high and low-level goals? 2 Background 2.1 The GORE Framework Introduced by Van Lamsweerde in 2001 [21], Goal-Oriented Re- quirements Engineering (GORE) is a technique with the primary objective of identifying all goals of a system, defined as Objectives that the system under consideration should achieve, which can be then mapped to functional or non-functional requirements. Goals can be formulated at different levels of abstraction, ranging from high- level strategic concerns to low-level technical ones. The technique also includes the identification of the actors — i.e., stakeholders of the system, including its final users — goal prioritisation, and conflict detection. While low-level goals are typically derived from parent goals by asking ‘how’, the reverse path — goal abstraction 1 https://zenodo.org/records/18919525 — is achieved by answering ‘why’ [21]. For sake of simplicity, we consider only functional goals in our work, as the total number of non-functional goals may raise to an unmanageable amount, especially in case of manual datasets curation. 2.2 Large Language Models in Requirements Engineering Zadenoori et al. [26] provide a comprehensive survey of LLM inte- gration into RE, noting an exponential trajectory in publications beginning in 2023. Their analysis reveals that the majority of current studies utilise GPT-family models, with a heavy emphasis on Zero- shot (used in 44% of the surveyed studies) and Few-shot prompting (29%). A key finding of the survey is the current inclination to- wards using models without further task-specific optimisation; the authors argue that this trend highlights a requirement for more rigorous investigation into advanced model architectures and or- chestration — as performed by this work. Moreover, significant hurdles remain concerning the deterministic reliability of these systems and the integration of human supervisory oversight. Sami et al. [19] developed a multi-agent system where a Prod- uct Owner agent generates user stories that are subsequently val- idated by a Quality Assurance agent against the INVEST [3] and ISO/IEC/IEEE 29148-2011 [1] standards. Their findings, which ad- dress a challenge similar to that addressed herein, emphasise the advantages of inter-agent communication and orchestration over a single, monolithic LLM instance. However, that study only covers the ’user story’ format, while in the present work we target system goals. Das et al. [7] propose NLP-driven techniques for extracting struc- tured goal models from unstructured textual input, reducing manual effort and enhancing requirement elicitation accuracy. Similarly, re- cent work on API Alignment [11] integrates GORE with LLM-based techniques by leveraging multiple iterative prompts and the GPT model to extract goals and map them to existing APIs. The paper demonstrates the potential of LLMs for automating goal extraction and API selection, but it also suffers from limitations in validation, including inconsistent goal decomposition, unexplained omissions, and a lack of structured quality control. Our approach, building on existing work, aims to expand the investigation of a multi-LLM pipeline for goal extraction and to offer a further exploratory evaluation of its feasibility and limitations. 3 Approach 3.1 Architecture We based our architecture on a fixed-structure LLM chain, rep- resented in Figure 1. Since GORE consists of clearly defined and sequential steps, agents’ inherent autonomy and flexibility were not well-suited. On the other hand, a structured chain ensures greater control over the execution flow. At the core of our architecture, two LLMs — GPT-4 and Llama 3.3 70B — collaborate through an iterative feedback loop. GPT-4 functions as the ’generator’, while Llama 3.3 70B operates as the ’evaluator’. Although the research prioritises the optimisation of prompting strategies and interaction patterns, this configuration has been chosen since it leverages GPT-4 for high-quality text gen- eration and Llama for a lower-cost response evaluation. If Llama’s Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their LimitationsEASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom evaluation score is below 8.5/10 — that will be referred as Quality Threshold 2 — the critique is inserted into the prompt fed to GPT-4 in the next iteration. Until the Quality Threshold is not met, the system can perform a maximum of 3 iterations before continuing to the next phase, as we empirically found that this number is a good trade-off between accuracy and computational overhead. Moreover, we experimentally observed that if the ’generator’ and the ’critic’ agents fail to reach an agreement within this number of iterations, then it is unlikely to reach convergence. Future works may investigate in more depth the effect on performance when these parameters are varied. 3.2 Multi-step Pipeline As represented in Figure 1, the pipeline we propose is composed of multiple phases: (1)Documentation Preprocessing (optional): It transforms a raw README file into a natural language project description, serving as a preprocessing step to improve the accuracy of downstream tasks. This is automated through the use of GPT-4, prompted as reported in Table 3; (2)Actor Identification: Actors are active entities that carry actions to achieve one or multiple goals [16]; (3) High-level Goals Extraction: High-level goals are the broad project objectives. For each identified actor, this step aims to extract the main goals that he may want to achieve when interacting with the software; (4)Low-level Goals Extraction: Low-level goals are specific, actionable objectives. Complex high-level goals are decom- posed into a hierarchy of low-level goals; (5)API Mapping: Goals can be finally mapped to the API end- points that implement the related functionalities. This step needs as input the set of low-level goals and the API docu- mentation of the software under analysis. The first step is optional because natural language project de- scriptions were already available for the London Ambulance Service and Urbain Maintenance case studies (described in Section 4.2). Actors, high-level and low-level goals extraction is implemented through the iterative feedback loop described in Section 3.1, em- ploying the prompts reported in Table 3. 3.3 Prompting Strategy The chain integrates diverse in-context learning strategies, includ- ing Zero-shot, One-shot, and Few-shot. Three Shot examples have been manually curated for each task, by adapting documentation and requirements of existing open-source projects. Specifically, they are related to (i) an application that extracts statistics associ- ated to GitHub accounts; (i) a food delivery application; and (i) a home maintenance service locator. Some examples of prompts are reported in Table 3, while Table 4 contains some instances of Few-shot examples integrated into prompts. The cosine similarities between the ground truth examples — described in Section 4.2 — and the Shot examples are reported in Table 1. 2 This threshold has been empirically derived, by qualitatively analysing the generated goals. Table 1: Average cosine similarities between the Shot exam- ples used in the prompts for the generator agent — i.e., GPT — and the different software projects constituting our ground truth dataset. DatasetTaskAverage Simi- larity Genome NexusActors0.5137 high-level goals0.5151 low-level goals0.5323 Gestao HospitalActors0.5219 high-level goals0.5113 low-level goals0.5082 London AmbulanceActors0.5319 high-level goals0.5327 low-level goals0.5327 Urban MaintenanceActors0.5007 high-level goals0.5067 low-level goals0.5144 Average per TaskActors0.5171 high-level goals0.5164 low-level goals0.5219 Table 2: Average cosine similarities between the Shot exam- ples fed to the critic agent — i.e., Llama — in each task and the different software projects forming the ground truth. DatasetTaskAverage Simi- larity Genome NexusActors0.5514 high-level goals0.5407 low-level goals0.5257 Gestao HospitalActors0.5289 high-level goals0.4991 low-level goals0.5066 London AmbulanceActors0.5031 high-level goals0.4687 low-level goals0.5037 Urban MaintenanceActors0.5382 high-level goals0.5236 low-level goals0.5631 Average per TaskActors0.5304 high-level goals0.5080 low-level goals0.5248 While our study evaluates diverse prompting strategies for the ’generator’ agent, the ’evaluator’ was restricted to a Few-shot config- uration to mitigate scoring insensitivity. Empirical observations re- vealed that — absent explicit benchmarks for ’major’ versus ’minor’ errors — the Llama model lacked the necessary evaluative anchors to assign nuanced ratings. Without these calibrated exemplars, the model exhibited a systemic bias toward invariant scoring across disparate iterations. The cosine similarities between the ground truth examples and the Shot examples feed to the ’evaluator’ are reported in Table 2. At the present stage of development, the framework is restricted to functional requirements by embedding explicit constraints within EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United KingdomArnaudo et al. Figure 1: Schema of the proposed architecture, consisting of a LLM chain the model prompts. However, the architecture maintains the flexi- bility to encompass also non-functional goals. This extension would necessitate removing the existing prompt instructions and updating both the ground-truth data and the Few-shot exemplars to reflect these broader goals. To enable the reproducibility of the results, the temperature has been set to zero to improve determinism in both GPT and Llama’s generations. Moreover, prompts are enhanced by a priming technique 3 , as can be seen from the prompt samples reported in Table 3. Finally, it is fundamental to ensure that model outputs are struc- tured and consistent. For this purpose, we use OpenAI’s Pydan- tic library 4 , which allows us to enforce a structured schema for GPT-generated responses, while Llama outputs are formatted using standard JSON serialisation. 4 Evaluation Method 4.1 Evaluation Pipeline The evaluation procedure consists of the following steps: (1) Preprocessing: Stopwords remotion, stemming and lem- matisation are applied to reduce variation in text represen- tation. This can enhance the accuracy of the subsequent similarity computation. Due to their simpler formulation, preprocessing is not applied to actor names, but only on goals descriptions; (2)Encoding: Both generated and reference data are trans- formed into vector embeddings using a BERT-based model; (3) Similarity Computation: Cosine similarity is calculated between generated and reference goal embeddings, forming a similarity matrix; (4)Maximum Weight Bipartite Matching: For each actor or goal, it is necessary to find the element of the ground 3 In the context of prompt engineering, priming is the practice of strategically providing contextual input within a prompt to shape the model’s responses toward a desired style, reasoning process, or domain of information. 4 https://ai.pydantic.dev/models/openai/ truth that it aims to resemble. To do so, we employ the al- gorithm proposed by Munkres et al. [18] to solve the Maxi- mum Weight Bipartite Matching problem, which prescribes to model the generated and the reference items as nodes of a bipartite graph. In this undirected graph, the arcs are weighted by the cosine similarity between the two strings associated to the nodes. By finding the set of arcs generating the maximum sum of weights, each string is linked to its best match. 5 (5)Computation of Precision, Recall and F1-score: For the computation of these performance metrics, we employed the formulas from Zhang et al. [27] reported below: 6 푅푒푐푎푙= 1 |푋| Í (푖,푗)∈퐽 x ⊤ 푖 y 푗 푃푟푒푐푖푠푖표푛= 1 |푌| Í (푖,푗)∈퐽 x ⊤ 푖 y 푗 F1-score= 2∗푅푒푐푎푙∗푃푟푒푐푖푠푖표푛 푅푒푐푎푙+푃푟푒푐푖푠푖표푛 Where X is the set of embeddings from the generated strings; Y is the set of embeddings from the ground truth; J is the set of arcs identified in the previous step. The scalar product betweenx 푖 andy 푖 computes the cosine similarity between the relative strings. The approach of computing cosine similarity between BERT embeddings effectively distinguishes true semantic equivalence from surface-level resemblance, ensuring robust adherence to the ground truth. At the same time, solving the Maximum Weight Bipartite Matching problem allows us to effectively measure the similarity between the generated and the reference sets of actors or goals. 5 In case the lists of the generated and the reference strings have different lengths, some items remain unpaired. If this happens, those items contribute to the counting of false positives or false negatives respectively. 6 They applied these formulas on two sets of words, while we applied them to two sets of sentences. Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their LimitationsEASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom Table 3: Prompts provided to the LLM model to accomplish the different tasks of the proposed pipeline. Preprocess the README file System prompt: You a r e a t e c h n i c a l w r i t i n g a s s i s t a n t s p e c i a l i z e d i n s u m m a r i z i n g s o f t w a r e d o c u m e n t a t i o n . Y o u r g o a l i s t o e x t r a c t a c l e a r , w e l l− w r i t t e n , and a c c u r a t e d e s c r i p t i o n o f a p r o j e c t f r o m i t s README f i l e . The d e s c r i p t i o n s h o u l d b e n a t u r a l and i n f o r m a t i v e , w i t h o u t u n n e c e s s a r y d e t a i l s o r i m p l e m e n t a t i o n s p e c i f i c s . A v o i d m a r k e t i n g l a n g u a g e , v a g u e c l a i m s , o r f i l l e r c o n t e n t . T a l k a s you w e r e a s t a k e h o l d e r d e s c r i b i n g t h e s y s t e m he w a n t s t o b e i m p l e m e n t e d ( e . g . , d u r i n g r e q u i r e m e n t s e l i c i t a t i o n ) . Prompt: H e r e i s t h e README f i l e o f a s o f t w a r e p r o j e c t :[ . . . ] E x p l a i n i t s p u r p o s e , t h e p r o b l e m i t a d d r e s s e s ( i f m e n t i o n e d ) , and i t s main f u n c t i o n a l i t i e s . Generate a new list of actors System prompt: You a r e a h e l p f u l a s s i s t a n t e x p e r t i n s o f t w a r e e n g i n e e r i n g t a s k s , s p e c i a l i s e d i n e x t r a c t i n g end− u s e r s r o l e s f r o m a h i g h l e v e l d e s c r i p t i o n o f a s o f t w a r e p r o j e c t . Y o u r t a s k i s t o e x t r a c t t h e a c t o r s ( r o l e s o f e n d u s e r s o f t h e s y s t e m ) f r o m t h e g i v e n d e s c r i p t i o n . I f a c t o r s a r e n o t e x p l i c i t l y m e n t i o n e d , i n f e r them b a s e d on t y p i c a l u s e r s o f s i m i l a r s o f t w a r e s y s t e m s . E a c h e x t r a c t e d a c t o r name s h o u l d b e a c c o m p a n i e d by a v e r y s h o r t d e s c r i p t i o n . Prompt: Now e x t r a c t t h e a c t o r s ( r o l e s o f e n d u s e r s ) f r o m t h e f o l l o w i n g s o f t w a r e d e s c r i p t i o n . ∗ D e s c r i p t i o n :∗ d e s c r i p t i o n Generate a new list of high-level goals System prompt: You a r e a h e l p f u l a s s i s t a n t e x p e r t i n s o f t w a r e e n g i n e e r i n g t a s k s . You' r e t a s k e d t o e x t r a c t h i g h− l e v e l g o a l s f r o m a s o f t w a r e d e s c r i p t i o n f o r e a c h p r o v i d e d a c t o r t h a t i s e x p e c t e d t o i n t e r a c t w i t h t h e s o f t w a r e . F o l l o w i n g t h e G o a l− O r i e n t e d R e q u i r e m e n t s E n g i n e e r i n g ( GORE ) f r a m e w o r k s , h i g h− l e v e l g o a l s a r e s t r a t e g i c o b j e c t i v e s t h a t d e f i n e t h e' why' b e h i n d a s y s t e m . They a r e u s u a l l y a b s t r a c t , b u s i n e s s− o r i e n t e d , and i n d e p e n d e n t o f t e c h n i c a l i m p l e m e n t a t i o n . They r e p r e s e n t t h e n e e d s o f s t a k e h o l d e r s o r t h e o r g a n i z a t i o n . F o c u s : V i s i o n and j u s t i f i c a t i o n . G e n e r a t e ONLY t h e f u n c t i o n a l g o a l s . Prompt: B a s e d on y o u r u n d e r s t a n d i n g o f t h e t y p i c a l n e e d s and i n t e r e s t s o f t h e f o l l o w i n g a c t o r s i n t h e f o l l o w i n g s o f t w a r e p r o j e c t , h e l p g e n e r a t e a l i s t o f h i g l l e v e l g o a l s . ∗ D e s c r i p t i o n :∗ p r o j e c t _ d e s c r i p t i o n ∗ A c t o r s :∗ a c t o r s Generate a new list of low-level goals System prompt: You a r e a h e l p f u l a s s i s t a n t e x p e r t i n s o f t w a r e e n g i n e e r i n g t a s k s .E l i c i t low− l e v e l g o a l s f o r a s p e c i f i c s t a k e h o l d e r i n a s o f t w a r e p r o j e c t . A v o i d g e n e r i c g o a l s . I n s t e a d , b r e a k them down i n t o a t o m i c a c t i o n s l i n k e d t o s y s t e m c a p a b i l i t i e s . Don' t b e t o o g e n e r i c , f o r e x a m p l e , a v o i d g o a l s l i k e' make t h e s o f t w a r e f a s t' ,' d e v e l o p a web i n t e r f a c e' e t c . F o l l o w i n g t h e G o a l− O r i e n t e d R e q u i r e m e n t s E n g i n e e r i n g ( GORE ) f r a m e w o r k , low− l e v e l g o a l s a r e t e c h n i c a l o b j e c t i v e s t h a t d e s c r i b e' how' t h e h i g h− l e v e l g o a l s w i l l b e a c h i e v e d . They a r e more c o n c r e t e and a r e e v e n t u a l l y r e f i n e d i n t o s p e c i f i c r e q u i r e m e n t s o r s o f t w a r e s p e c i f i c a t i o n s . F o c u s : I m p l e m e n t a t i o n and c o n s t r a i n t s . G e n e r a t e ONLY t h e f u n c t i o n a l g o a l s . Prompt: B a s e d on y o u r u n d e r s t a n d i n g o f t h e t y p i c a l t a s k s t h a t c o m p o s e t h e f o l l o w i n g s e q u e n c e o f h i g h− l e v e l g o a l s , p r o v i d e i f p o s s i b l e a d e c o m p o s i t i o n o f g o a l s i n t o s u b− g o a l s . E a c h low− l e v e l g o a l s h o u l d t h e o r e t i c a l l y c o r r e s p o n d t o a s i n g l e a c t i o n o f t h e a c t o r w i t h t h e s o f t w a r e . ∗ High− l e v e l g o a l s :∗ h i g h L e v e l G o a l s Critique the response previously generated System prompt: You' r e an h e l p f u l a s s i s t a n t , e x p e r t i n t h e f i e l d o f s o f t w a r e e n g i n e e r i n g . Prompt: You' r e an h e l p f u l a s s i s t a n t , e x p e r t i n t h e f i e l d o f s o f t w a r e e n g i n e e r i n g and s p e c i a l i s e d i n t h e G o a l− O r i e n t e d R e q u i r e m e n t s E n g i n e e r i n g ( GORE ) f r a m e w o r k . F o l l o w i n g t h e G o a l− O r i e n t e d R e q u i r e m e n t s E n g i n e e r i n g ( GORE ) f r a m e w o r k : −an a c t o r i s a c t i v e e n t i t y t h a t h a s t h e c a p a b i l i t y t o p e r f o r m a c t i o n s t o a c h i e v e g o a l s . U n l i k e g o a l s , w h i c h a r e ' what' o r' why ,' a c t o r s a r e t h e' who .' −h i g h− l e v e l g o a l s a r e s t r a t e g i c o b j e c t i v e s t h a t d e f i n e t h e' why' b e h i n d a s y s t e m . They a r e u s u a l l y a b s t r a c t , b u s i n e s s− o r i e n t e d , and i n d e p e n d e n t o f t e c h n i c a l i m p l e m e n t a t i o n . They r e p r e s e n t t h e n e e d s o f s t a k e h o l d e r s o r t h e o r g a n i z a t i o n . F o c u s : V i s i o n and j u s t i f i c a t i o n . −low− l e v e l g o a l s a r e t e c h n i c a l o b j e c t i v e s t h a t d e s c r i b e' how' t h e h i g h− l e v e l g o a l s w i l l b e a c h i e v e d . They a r e more c o n c r e t e and a r e e v e n t u a l l y r e f i n e d i n t o s p e c i f i c r e q u i r e m e n t s o r s o f t w a r e s p e c i f i c a t i o n s . F o c u s : I m p l e m e n t a t i o n and c o n s t r a i n t s . You c a n p r o p o s e new g o a l s t a k i n g i n t o a c c o u n t t h e a l r e a d y p r e s e n t o n e s . C o n s i d e r t h a t h i g h− l e v e l g o a l s o f t e n a n s w e r t h e WHY q u e s t i o n , w h i l e low− l e v e l g o a l s o f t e n a d d r e s s t h e HOW. You m us t e n s u r e t h a t ONLY f u n c t i o n a l g o a l s a r e p r e s e n t . ∗ D e s c r i p t i o n :∗ d e s c r i p t i o n ∗ A c t o r s :∗ a c t o r s EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United KingdomArnaudo et al. Table 4: Exemplars of Few-shot examples used either for prompting the ’generator’ or the ’critic’ LLM. Other instances can be found in the online appendix. Generation of high-level goals − D e s c r i p t i o n : C a t W a t c h i s a web a p p l i c a t i o n t h a t t r a c k s and s t o r e s G i t H u b s t a t i s t i c s f o r a c c o u n t s . I t p r o v i d e s p r o j e c t p o p u l a r i t y and c o n t r i b u t o r d a t a t h r o u g h a REST API , o f f e r i n g a g g r e g a t e d s t a t s . − A c t o r s : [ name : " G i t H u b a c c o u n t " , d e s c r : " I n d i v i d u a l s who own G i t H u b a c c o u n t s " ] ∗ O u t p u t :∗ [ " The s t a k e h o l d e r a i m s t o e f f o r t l e s s l y m o n i t o r t h e p o p u l a r i t y m e t r i c s o f t h e i r o p e n s o u r c e p r o j e c t s a c r o s s v a r i o u s G i t H u b a c c o u n t s u s i n g C a t W a t c h . " , " The s t a k e h o l d e r s e e k s a f e a t u r e t h a t h i g h l i g h t s t h e m o s t a c t i v e c o n t r i b u t o r s and c o l l a b o r a t o r s i n t h e i r G i t H u b r e p o s i t o r i e s t h r o u g h C a t W a t c h . " , " The s t a k e h o l d e r d e s i r e s a n o t i f i c a t i o n s y s t e m w i t h i n C a t W a t c h t h a t a l e r t s them p r o m p t l y a b o u t s i g n i f i c a n t a c t i v i t i e s , s u c h a s new c o n t r i b u t i o n s o r r i s i n g p r o j e c t t r e n d s " , " The s t a k e h o l d e r i n s i s t s on C a t W a t c h i m p l e m e n t i n g r o b u s t d a t a s e c u r i t y m e a s u r e s and c o m p l i a n c e w i t h p r i v a c y s t a n d a r d s t o s a f e g u a r d t h e i r G i t H u b a c c o u n t i n f o r m a t i o n . " , " The s t a k e h o l d e r r e q u i r e s C a t W a t c h t o s e a m l e s s l y i n t e g r a t e w i t h t h e i r e x i s t i n g w o r k f l o w t o o l s and d e v e l o p m e n t e n v i r o n m e n t s , e n h a n c i n g p r o d u c t i v i t y and u s e r e x p e r i e n c e . " , " The s t a k e h o l d e r a i m s t o a c c e s s d e t a i l e d a n a l y t i c s and r e p o r t s g e n e r a t e d by CatWatch , o f f e r i n g i n s i g h t s i n t o p r o j e c t p e r f o r m a n c e , community e n g a g e m e n t , and o t h e r r e l e v a n t m e t r i c s . " ] Critic of high-level goals − D e s c r i p t i o n : C a t W a t c h i s a web a p p l i c a t i o n t h a t t r a c k s and s t o r e s G i t H u b s t a t i s t i c s f o r a c c o u n t s . I t p r o v i d e s p r o j e c t p o p u l a r i t y and c o n t r i b u t o r d a t a t h r o u g h a REST API , o f f e r i n g a g g r e g a t e d s t a t s . − A c t o r s : [ name : " G i t H u b a c c o u n t " , d e s c r : " I n d i v i d u a l s who own G i t H u b a c c o u n t s " ] − High− l e v e l G o a l s : [ " The s t a k e h o l d e r a i m s t o e f f o r t l e s s l y m o n i t o r t h e p o p u l a r i t y m e t r i c s o f t h e i r o p e n s o u r c e p r o j e c t s a c r o s s v a r i o u s G i t H u b a c c o u n t s u s i n g C a t W a t c h . " , " The s t a k e h o l d e r s e e k s a f e a t u r e t h a t h i g h l i g h t s t h e m o s t a c t i v e c o n t r i b u t o r s and c o l l a b o r a t o r s i n t h e i r G i t H u b r e p o s i t o r i e s t h r o u g h C a t W a t c h . " , " The s t a k e h o l d e r w a n t s t o u s e t h e s y s t e m t o t r a c k t h e h e a l t h , f e e d i n g s c h e d u l e s , and GPS l o c a t i o n s o f a c t u a l c a t s i n a r e s c u e s h e l t e r . " , " The s t a k e h o l d e r i n s i s t s on C a t W a t c h i m p l e m e n t i n g r o b u s t d a t a s e c u r i t y m e a s u r e s and c o m p l i a n c e w i t h p r i v a c y s t a n d a r d s t o s a f e g u a r d t h e i r G i t H u b a c c o u n t i n f o r m a t i o n . " , " The s t a k e h o l d e r r e q u i r e s C a t W a t c h t o s e a m l e s s l y i n t e g r a t e w i t h t h e i r e x i s t i n g w o r k f l o w t o o l s and d e v e l o p m e n t e n v i r o n m e n t s , e n h a n c i n g p r o d u c t i v i t y and u s e r e x p e r i e n c e . " , " The s t a k e h o l d e r a i m s t o a c c e s s d e t a i l e d a n a l y t i c s and r e p o r t s g e n e r a t e d by CatWatch , o f f e r i n g i n s i g h t s i n t o p r o j e c t p e r f o r m a n c e , community e n g a g e m e n t , and o t h e r r e l e v a n t m e t r i c s . " ] ∗ S c o r e :∗ 3 / 1 0 ∗ Comment :∗ Out o f c o n t e x t . D e s p i t e t h e name " CatWatch , " t h e g o a l r e g a r d i n g t r a c k i n g p h y s i c a l c a t s i s c o m p l e t e l y u n r e l a t e d . 4.2 Experiment Setup To evaluate the proposed architecture, we selected four samples of software projects, as reported in Table 5. Future work may experi- ment with more datasets. We selected two enterprise applications from the WFD (for- merly EMB) dataset [4]. This dataset includes software projects with corresponding READMEs and Swagger-formatted API docu- mentation. Our focus is on GestaoHospital 7 , a public health manage- ment system, and GenomeNexus 8 , which automates the annotation of cancer-related genetic variants. These projects constitute good examples of commissioned applications involving software engi- neering activities. 7 https://github.com/ValchanOficial/GestaoHospital 8 https://github.com/WebFuzzing/Dataset/tree/master/jdk_8_maven/cs/rest- gui/genome-nexus Table 5: Number of annotations present in the ground truth datasets adopted in this study. HL = High-level, L = Low- level datasetActors HL Goals L Goals GestaoHospital5420 GenomeNexus5934 Urban Maintenance6918 London Ambulance System4210 Total202482 Table 6: Example of output produced by the proposed system when processing the Urban Maintenance use case under the Few-shot setting. Red text represents false positives, while blue highlights false negatives. Examples of false positives are the high-level goals related to account registration and management. While low-level goals have not been reported for brevity, we observed that the relative low-level goals have not been consistently generated. Actors "Citizens", "Municipal Operators", "External Maintenance Personnel", "Administrators", "Non-registered Users", "Technical Office Staff", "Or- ganizational Office Staff" High-level goals "Citizens should be able to submit reports by selecting locations on a map and providing necessary details, ensuring their voices are heard in urban management.", "Municipal operators need tools to review, approve, and manage reports effectively, ensuring timely responses to urban issues.", "External maintenance personnel should be able to receive assignments, update report statuses, and communicate with municipal operators to resolve issues efficiently.", "Administrators require access to both public and private statistics to monitor system performance and citizen engage- ment effectively.", "Non-registered users should be able to access public reports and statistics to foster trust and engagement with the munici- pality’s initiatives.", "The system should allow citizens to submit reports anonymously, ensuring their identity is not disclosed in public listings.", "Citizens should be able to create a new account through registration.", "Registered users should be able to manage their account information", "Registered users should be able to start following a report to receive timely notifications", "Administrators should be able to configure the auto-assignment for report categories., "External maintenance users should be able to communicate with the municipal staff, and eventually manage a report’s lifecycle through the platform." Furthermore, we examined the London Ambulance Service case study, a seminal exemplar frequently cited within the GORE liter- ature [14,22]. The associated annotations were synthesised from the extant body of research. Finally, we included a software project sourced from an uni- versity course, describing an urban maintenance ticketing system. The relative annotations were manually curated by subject-matter experts within the teaching faculty. The ground truth was manually curated by three of the authors of this paper, producing the annotations described in Table 5 and available in the replication package. Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their LimitationsEASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom Table 7: Results for the tasks described in Section 3.2, per- formed in Zero-Shot (ZS), One-Shot (OS), or Few-shot (FS) setting. ActorsHigh-Level Goals Low-Level Goals ZSOSFSZSOSFSZSOSFS Prec.0.750.68 0.78 0.630.570.63 0.780.720.77 Recall0.78 0.800.67 0.610.600.59 0.510.490.45 F10.760.740.72 0.620.590.61 0.610.590.57 5 Results and Discussion All the measurements obtained during the evaluation are arranged in Table 7, while a qualitative evaluation is reported in Table 6. 5.1 Extraction of Actors As summarised in Table 7, the Zero-shot setting achieved the best F1-score — of about 0.76 — in actor extraction. At the same time, One-shot and Few-shot obtained comparable values for this metric. One-shot is associated with the highest recall (0.80), while Few- shot seem to favour precision — achieving the peak value of 0.78. This scenario highlights that the absence of Shot examples leads to the best trade-off between precision and recall, but the system’s behaviour can be shifted by applying different Shot-prompting strategies. To assess the impact of the refinement loop described in Section 3.1 — the core innovation of our architecture — Table 9 reports results from a pipeline using only the GPT model without any feed- back mechanism. The comparison reveals that Actor Extraction does not benefit from the feedback loop. Although the observed differences are marginal — never exceeding three percentage points — the F1-scores achieved during the ablation study consistently outperform those of the full architecture across all prompting con- figurations, reaching a peak of 0.78 in the Zero-shot setting. To further evaluate the impact of the Few-shot prompts, Table 8 details the performance metrics across the ground truth case studies (already described in Section 4.2). Notably, in the Genome Nexus case the system achieved perfect precision, albeit with a limited recall of approximately 0.40. This precision peak aligns with Tables 1 and 2, which identify this dataset as having the highest cosine similarity to the Few-shot exemplars utilised in both GPT and Llama prompts. Regarding the Urban Maintenance dataset — which yielded the second-highest precision and a substantial recall of 0.70 — the av- erage cosine similarity does not rank second for GPT prompts; however, it does hold the second-highest position for Llama-based prompts (Table 2). A consistent pattern emerges for the Gestao Hospital and London Ambulance Service datasets, which rank third and fourth in precision, respectively. Their precision scores corre- late proportionally with the average cosine similarities observed in the Llama prompts. This suggests that the order of precision is preserved specifically in relation to the Llama Few-shot similarities, potentially indicating a direct correlation between the diversity of exemplars provided to the critic agent and the system’s overall precision. However, analogous patterns cannot be found when considering neither system’s recall nor F1-score. Similarly, the cosine similar- ities between the case studies forming the ground truth and the Few-Shot examples used in GPT prompting seem not to be involved. Table 8: Performance metrics for the Actor identification task using Few-shot prompting, disaggregated by the individual software projects within the ground truth and ordered by precision. DatasetRecall Precision F1-score Genome Nexus0.401.000.57 Urban Maintenance0.700.840.77 London Ambulance Service0.760.510.61 Gestao Hospital0.850.710.77 Answer to RQ1: Our evaluation reported an F1-Score of 0.76 in the optimal prompting configuration — specifically Zero-shot. This indicates that our architecture takes limited benefit from in-context learning. Furthermore, a distinct trade-off is evident between the maximum precision of 0.78 achieved via Few-shot and the maxi- mum recall of 0.80 associated with One-shot prompting. The Llama ablation study yielded a slightly superior F1-score of 0.78 — associ- ated with Zero-shot again. This suggests that the generator agent — i.e., GPT — is the primary responsible of the suboptimal exploita- tion of the Shot examples. Finally, we found a possible correlation between the precision in Actor extraction and the cosine similarity between the software project’s description and the Few-Shot ex- amples provided to the critic agent — i.e., the Llama model. This correlation suggests that the Few-shot strategy applied to Llama should be enriched to improve the critic mechanism’s effectiveness. 5.2 Extraction of High-Level Goals As reported in Table 7, high-level goal extraction demonstrates a distinct behavioural pattern across prompting settings compared to actor extraction. The Zero-shot strategy yielded the optimal overall F1-score, precision, and recall — recorded as 0.62, 0.63, and 0.61, respectively. These findings confirm the detrimental impact of Shot-prompting if integrated with the feedback mechanism. However, in the results of the Llama ablation study — reported in Table 9 — the opposite trend can be observed, with the supremacy of Few-shot prompting. This may suggest that — for what concerns the task of identifying high-level goals — the GPT model takes advantage from Shot examples, but this effect is neutralised by the critique mechanism. Table 9: Results for the Llama’s feedback ablation, performed in Zero-Shot (ZS), One-Shot (OS), or Few-shot (FS) setting. ActorsHigh-Level Goals Low-Level Goals ZSOSFSZSOSFSZSOSFS Prec.0.800.80 0.860.650.66 0.680.790.76 0.79 Recall0.770.740.670.460.46 0.530.380.36 0.40 F10.780.770.750.540.54 0.600.510.50 0.53 EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United KingdomArnaudo et al. In high-level goals extraction, the contribution of Llama proves to be slightly beneficial: during the ablation study, the maximum F1- score reached was 0.60, which is two points below the one relative to the complete architecture. These results may indicate that, while Few-shot prompting im- proves the ’generator’ model alone, high-level goals extraction takes greater advantage by the introduction of the feedback mechanism. Finally, in Table 10 we report the performance achieved by the complete architecture divided by case study, when Few-shot is ap- plied. As was observed for the actors extraction task, a proportional relation — albeit not linear — can be found when comparing the precision metric achieved in each case study and the cosine similar- ities reported in Table 2 — which have been computed between the case studies documentation and the few Shot examples fed to the Llama model. This may suggest the primary bottleneck resided in the quality of the Few-Shot examples provided to the ’critic’ agent, and not to the ’generator’ model. Consistently with the results presented in Section 5.1, we did not observe any correlation between the similarity to the GPT’s Few-shot examples of each case study and the respective perfor- mance achieved by the system. Table 10: Performance metrics for the high-level goals iden- tification task using Few-shot prompting, disaggregated by the individual software projects within the ground truth and ordered by precision. DatasetRecall Precision F1-score Genome Nexus0.510.760.61 London Ambulance Service0.750.210.33 Urban Maintenance0.530.690.60 Gestao Hospital0.810.460.59 Answer to RQ2: The maximum F1-score reached by our archi- tecture for the extraction of high-level goals is 0.62 (Zero-shot). The ablation of the critique mechanism reveals an opposite trend, with Few-shot achieving the highest F1-score — which was 0.60. This may suggest that the performance improvements achieved by the addition of the feedback loop overcome the ones related to the application of Few-shot to the GPT model alone. The performance with the feedback mechanism during Few-shot learning shows a specific trend: precision correlates proportionally — though not lin- early — with the cosine similarity between the case studies and the examples provided to the Llama model. This pattern consistently mirrors the results previously observed during the actors extraction task, possibly indicating that the primary bottleneck resided in the quality of the in-context examples provided to the ’critic’ model. 5.3 Extraction of Low-Level Goals As shown in Table 7, low-level Goal extraction yields a F1-score slightly lower than high-level goals extraction (0.61 versus 0.62). This can indicate that the errors may have propagated up to this point in the pipeline, effectively establishing a performance ceiling. Indeed, as detailed below, it is possible to confirm some patterns already found when analysing low-level goals extraction in Section 5.2. In the Zero-Shot setting, the system achieves the best recall (0.78), precision (0.51), and F1-score (0.61). At the same time, Few-shot proves to be the best strategy in the absence of the feedback mech- anism: in the results of the Llama’s ablation study — reported in Table 9 — Few-shot is associated with a great precision (0.78), a low recall (0.40), and the highest F1-score (0.53). These values confirm the phenomena already observed in the extraction of high-level goals, suggesting that the introduction of the feedback mechanism yields greater benefits than the application of Few-shot prompting to the GPT model alone. The iterative interaction between GPT and Llama failed to exploit Shot-prompting, which suggests that the primary bottleneck resides within the prompting of the Llama model. Following the established methodology, we compared the cosine similarities between the case studies and the Llama’s Few-shot ex- emplars — detailed in Table 2 — against the performance metrics achieved by the architecture in the Few-shot setting for each corre- sponding case study (Table 11). Differently from the previous two sections, we were unable to find any correlation pattern. This may suggest that the errors made during the extraction of high-level goals may have introduced perturbations in this final step. As can be seen by the prompts in Table 3, the models are tasked to extract low-level goals by starting only from the high-level ones. Future work may investigate this effect more deeply by isolating the gen- eration of low-level goals. Table 11: Performance metrics for the Low-level goals iden- tification task using Few-shot prompting, disaggregated by the individual software projects within the ground truth and ordered by precision. DatasetRecall Precision F1-score Genome Nexus0.250.850.39 Gestao Hospital0.560.760.64 Urban Maintenance0.610.690.65 London Ambulance Service0.680.680.68 Answer to RQ3: The maximum F1-Score obtained for low-Level Goals extraction is 0.61 (Zero-shot). This value is slightly lower than the one achieved for the previous task in the pipeline, suggesting that the propagation of errors may have introduced performance ceiling. By removing the feedback loop from the architecture, the system achieves a maximum F1-score of 0.53 in the Few-shot setting. While this indicates that the introduction of the critic agent delivers a tangible advantage, it confirms that this benefit comes at the cost of neutralising the effects of in-context, as this phenomenon was already observed in the high-level goals extraction. 5.4 API Mapping Given the limited F1-scores observed in the preceding phases, this component is considered exploratory rather than part of the Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their LimitationsEASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom Table 12: Examples of generated API mappings High Level Goal NameLow Level Goal NameAPI Name Manage Healthcare OperationsRegister a new hospital with essential details, including name, address, and contact information.insertUsingPOST Manage Healthcare OperationsRetrieve a list of all registered hospitals with their registration status and details.findAllUsingGET formal evaluation. In Table 12, we therefore provide only a qualita- tive illustration of how such a mapping might look when applied to the extracted goals. These mappings should be interpreted as plausible candidates generated by the model rather than validated correspondences. Nevertheless, following a manual review of the project’s API documentation, the authors regard them as substan- tively sound. This provides preliminary insight into how such a component could support analysts during design or requirements traceability tasks. 6 Limitations 6.1 Bias Some bias may have been introduced through the examples em- ployed in the in-context learning. Specifically, if the examples used were more closely aligned with the target problem, the results could have been biased towards higher performance without actu- ally being the consequence of better design choices. This could be mitigated by expanding the evaluation to further benchmarks, or by introducing RAG-augmented Few-shot prompting. 9 Moreover, a potential source of bias lies in the input software documents (Section 3.2). As highlighted in the prompt engineering literature [10,17], the clarity and quality of information contained within input prompts are crucial determinants, and the adopted preprocessing strategy may not be enough for bare README files. Future work may assess the impact of feeding the system with more extensive documentation. 6.2 Threats to Construct Validity In the present study, README files have been used as a proxy for preliminary documents from which the requirements might be elicited. To provide more realistic application scenarios, the ap- proach should be evaluated with natural language requirements (e.g., transcripts of interviews with stakeholders) rather than README files. Furthermore, the multi-step pipeline enforces a ’waterfall’ method- ology that precludes the modification of high-level goals during the elicitation of low-level objectives. This contrasts with the estab- lished literature [21], which advocates for the late-stage discovery of high-level goals through obstacle analysis and by addressing ’why’ queries relative to low-level goals. Future work may enhance architectural flexibility by integrating these additional mechanisms. 6.3 Threats to Internal Validity We acknowledge that our study did not answer the proposed re- search questions exhaustively. Indeed, more experiments could be performed by varying the values of the Quality Threshold and 9 Retrieval Augmented Generation (RAG) with Few-shot prompting refers to the use of a RAG system to retrieve the most pertinent Few-Shot examples at inference time, based on their similarity with the system’s current input. the maximum number of iterations of the generation-critique loop (both defined in Section 3.1). Furthermore, our method stipulates that the final response pro- duced by GPT-4 is retained once the maximum number of iterations has been reached. Although retaining the final iteration may be suboptimal compared to selecting the highest-scoring response, Llama’s feedback effectively mitigates quality degradation across successive outputs. 6.4 Threats to External Validity As previously stated, we concentrated solely on functional goals. While this limitation was imposed to keep the number of goals per use case manageable for manual annotation, we acknowledge that it may limit the generalisability of our results to real-world scenarios where non-functional goals are critical. Finally, it is worth noting that the reported results are highly dependent on the chosen architecture, the specific models employed, and the datasets used to validate the approach. Although they offer valuable insights into the capabilities of LLMs, these findings may not be generalisable to other combinations of models or alternative configurations (e.g., changing the number of max iterations) within a processing chain. 7 Conclusion and Future Work We presented a semi-structured approach for automating parts of the RE process, assessing the impact of in-context learning, and exploiting models with diverging base knowledge for refining the outputs through an iterative feedback mechanism. However, the results were not entirely satisfactory, with an F1-score of 61% in low-level goals extraction — the last step of the pipeline. We ac- knowledge that the values observed are insufficient for fully auto- mated use and would still require substantial manual supervision in practice. We therefore view the proposed approach not as a re- placement for human annotation, but as a starting point that can assist and accelerate manual extraction. Improving recall is a key direction for future work, and systematic comparison with human recall on the same data would provide a more meaningful upper bound and evaluation target. Crucially, the ablation study — conducted by removing the Llama- based feedback mechanism — demonstrated that the proposed ar- chitecture yields consistent advantages over a conventional, lin- ear pipeline utilising GPT in isolation. Moreover, we were able to demonstrate that our feedback-loop mechanism with Zero-shot prompting outperforms Few-shot prompting applied to the GPT model alone. Measurements of the cosine similarity between the case study descriptions and the shot examples provided to the GPT model revealed no correlation with performance per case study. This sug- gests a balanced variety among the in-context examples. EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United KingdomArnaudo et al. Conversely, analysis of the Few-shot examples used for Llama prompting indicated a potential correlation between their similarity to the case studies and the precision values achieved. This suggests that the quantity and variety of in-context examples provided to the Llama model should be expanded to improve robustness. While our approach may serve as an encouraging starting point for developing more accurate systems, we recognise that there re- mains substantial room for improvement. Future work may involve conducting repeated runs to assess stability, evaluating additional datasets (Section 4.2), and testing alternative embedding models — such as Sentence BERT — in place of BERT (Section 4.1). In its current form, our method - although iterative - did not involve a human in the loop, which may account for the suboptimal outcomes. Indeed, various studies in the RE literature [9,13,23] indicate that reliable results are difficult to achieve without involv- ing humans. Furthermore, Llama’s evaluation process could be enhanced with an increased number of Few-Shot examples, which are fundamental to provide to the model the references to assign the scores. Moreover, the model could be provided with literature- grounded instructions - drawing from RE and GORE studies - to guide its outputs towards a more informed evaluation, rather than depending exclusively on its internal knowledge and probabilistic reasoning. This enhancement could be realised through the inte- gration of a Retrieval-Augmented Generation (RAG) system and the application of Chain of Thought (CoT) prompting. Additionally, our architecture could be modified to better align to the GORE procedures described in the literature. Specifically, a further loop mechanism should be introduced, encompassing both the high-level and low-level goals extraction phases to enable late discovery of high-level goals. Finally, our study was conducted without imposing constraints on computational resources or processing time. Although we relied on remote API calls to access both the GPT and Llama models, future research could examine in greater depth the computational and economic costs. References [1]Systems and software engineering – Life cycle processes –Requirements engi- neering, 2011. ISBN: 9780738165912. [2] Akbar, M. A., Khan, A. A., and Liang, P. Ethical aspects of chatgpt in software engineering research. arXiv preprint arXiv:2306.07557 (2023). [3] Alliance, A. What does INVEST Stand For? | Agile Alliance, Dec. 2015. [4] Arcuri, A., Zhang, M., Golmohammadi, A., Belhadi, A., Duman, O., Seran, S., Galeotti, J. P., and Ghianni, H. WebFuzzing/EMB: v3.4.0, Jan. 2025. [5]Arulmohan, S., Meurs, M.-J., and Mosser, S. Extracting domain models from textual requirements in the era of large language models. In 2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) (2023), IEEE, p. 580–587. [6]Bozyigit, F., Bardakci, T., Khalilipour, A., Challenger, M., Ramackers, G., Babur, Ö., and Chaudron, M. R. Generating domain models from natural language text using nlp: a benchmark dataset and experimental comparison of tools. Software and Systems Modeling 23, 6 (2024), 1493–1511. [7]Das, S., Deb, N., Cortesi, A., and Chaki, N. Extracting goal models from natural language requirement specifications. Journal of Systems and Software (2024), 111981. [8] De Vito, G., Palomba, F., Gravino, C., Di Martino, S., and Ferrucci, F. Echo: An approach to enhance use case quality exploiting large language models. In 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (2023), IEEE, p. 53–60. [9]Ebrahim, M., Guirguis, S., and Basta, C. Enhancing software requirements engineering with language models and prompting techniques: Insights from the current research and future directions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Work- shop) (2025), J. Zhao, M. Wang, and Z. Liu, Eds., Association for Computational Linguistics, p. 486–496. [10]Errica, F., Sanvito, D., Siracusano, G., and Bifulco, R. What Did I Do Wrong? Quantifying LLMs’ Sensitivity and Consistency to Prompt Engineering. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (Albuquerque, New Mexico, 2025), Association for Computational Linguistics, p. 1543–1558. [11]Feldt, R., and Coppola, R. Semantic api alignment: Linking high-level user goals to apis. In 2025 IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) (2025), IEEE, p. 17–20. [12]Feldt, R., Kang, S., Yoon, J., and Yoo, S. Towards autonomous testing agents via conversational large language models, 2023. [13] Ferrari, A., and Spoletini, P. Formal requirements engineering and large language models: A two-way roadmap. [14]Finkelstein, A., and Dowell, J. A comedy of errors: the london ambulance service case study. In Proceedings of the 8th International Workshop on Software Specification and Design, IEEE Comput. Soc. Press, p. 2–4. [15] Kang, S., Yoon, J., and Yoo, S. Large language models are few-shot testers: Ex- ploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) (2023), IEEE, p. 2312–2323. [16]Kavakli, E. Goal-oriented requirements engineering: A unifying framework. Requirements Engineering 6 (2002), 237–251. [17] Lin, Z. How to write effective prompts for large language models, Sept. 2023. [18]Munkres, J. Algorithms for the assignment and transportation problems. 32–38. [19]Sami, M. A., Waseem, M., Zhang, Z., Rasheed, Z., Systä, K., and Abrahamsson, P. AI based Multiagent Approach for Requirements Elicitation and Analysis. [20]Siddeshwar, V., Alwidian, S., and Makrehchi, M. A comparative study of large language models for goal model extraction. In Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems (2024), p. 253–263. [21]Van Lamsweerde, A. Goal-oriented requirements engineering: A guided tour. In Proceedings fifth ieee international symposium on requirements engineering (2001), IEEE, p. 249–262. [22]Van Lamsweerde, A., and Letier, E. Handling obstacles in goal-oriented re- quirements engineering. 978–1005. [23]Vogelsang, A. Prompting the future: Integrating generative LLMs and require- ments engineering. [24]Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). [25] Yoon, J., Feldt, R., and Yoo, S. Autonomous large language model agents enabling intent-driven mobile gui testing, 2023. [26]Zadenoori, M. A., Dąbrowski, J., Alhoshan, W., Zhao, L., and Ferrari, A. Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review, Sept. 2025. arXiv:2509.11446 [cs]. [27] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. BERTScore: Evaluating text generation with BERT.