Paper deep dive

Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews

Barbara Kitchenham, Sebastián Pizard, Lech Madeyski, Ronnie de Souza Santos, Martin Shepperd, David Budgen

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 234

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 8/1/2026, 10:54:25 AM

Summary

This paper presents preliminary guidelines, named GUEST (GenAI Use and Evaluation in SLR Tasks), for using and evaluating Generative AI (GenAI) and Large Language Models (LLMs) in Systematic Literature Reviews (SLRs). The authors identify methodological issues such as hallucination, non-determinism, data leakage, and bias, arguing that GenAI requires human oversight and cannot currently perform unsupervised systematic studies. The recommendations aim to ensure rigor, reliability, and transparency in SLRs conducted with GenAI support.

Entities (9)

GenAI · technology · 99%Kitchenham · author · 99%LLMs · technology · 99%GUEST · framework · 98%SLR · methodology · 97%hallucination · issue · 95%Data Leakage · issue · 94%RAISE · guideline · 92%PRISMA · standard · 90%

Relation Signals (7)

GenAI → requires → Human Oversight

confidence 96% · We argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies.

GUEST → providesguidancefor → SLR

confidence 95% · We summarize our results as a set of process recommendations, which we name GUEST (GenAI Use and Evaluation in SLR Tasks).

GenAI → suffersfrom → hallucination

confidence 95% · They can occasionally produce invalid or nonsensical output. This is called hallucinating...

GenAI → suffersfrom → Data Leakage

confidence 93% · This phenomenon is referred to as data leakage or data contamination. Data contamination represents a serious threat...

GUEST → addressesissues → hallucination

confidence 90% · We discuss the problems researchers face when evaluating GenAI for SLRs... identify and explain process issues... hallucinating...

SLR → iscomplementedby → GenAI

confidence 88% · GenAI offers the prospect of cost-effective assistance for some repetitive tasks and for additional validation of some complex tasks.

RAISE → similarto → GUEST

confidence 85% · The Responsible use of AI in Evidence SynthEsis (RAISE 2026) guidelines are a community-based set of recommendations... position our work.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, while capable of summarizing text, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require. Objectives: To support researchers intending to conduct SLRs using GenAI or those conducting empirical studies evaluating how well GenAI supports SLR tasks. Methods: First, we conducted a rapid review to identify studies that propose guidelines for evaluating and using GenAI and LLMs to support SLRs. Second, we drew on thought experiments, relevant guidance from the literature, and our own experience conducting SLRs and evaluating tools to develop recommendations for how to use and assess GenAI in the context of SLRs. Results: We discuss the problems researchers face when evaluating GenAI for SLRs. We identify and explain process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools. Finally, we summarize our results as a set of process recommendations, which we name GUEST (GenAI Use and Evaluation in SLR Tasks). Conclusion: We argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies. However, it offers the prospect of cost-effective assistance for some repetitive tasks and for additional validation of some complex tasks. Our GUEST recommendations should help software engineering researchers both to conduct and report trustworthy SLRs using GenAI and to provide rigorous independent evaluation studies.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

232,767 characters extracted from source content.

Expand or collapse full text

Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews Barbara Kitchenham 0000-0002-6134-8460 Keele UniversityUnited Kingdom , Sebastián Pizard 0000-0002-5535-8646 Universidad de la RepúblicaUruguay , Lech Madeyski 0000-0003-3907-3357 Wrocław University of Science and TechnologyPoland lech.madeyski@pwr.edu.pl , Ronnie de Souza Santos 0000-0003-3235-6530 University of CalgaryCanada , Martin Shepperd 0000-0003-1874-6145 Brunel University of LondonUnited Kingdom and David Budgen 0000-0001-7143-0241 Durham UniversityUnited Kingdom (2026) Abstract. Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, while capable of summarizing text, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require. Objectives: To support researchers intending to conduct SLRs using GenAI or those conducting empirical studies evaluating how well GenAI supports SLR tasks. Methods: First, we conducted a rapid review to identify studies that propose guidelines for evaluating and using GenAI and LLMs to support SLRs. Second, we drew on thought experiments, relevant guidance from the literature, and our own experience conducting SLRs and evaluating tools to develop recommendations for how to use and assess GenAI in the context of SLRs. Results: We discuss the problems researchers face when evaluating GenAI for SLRs. We identify and explain process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools. Finally, we summarize our results as a set of process recommendations, which we name GUEST (GenAI Use and Evaluation in SLR Tasks). Conclusion: We argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies. However, it offers the prospect of cost-effective assistance for some repetitive tasks and for additional validation of some complex tasks. Our GUEST recommendations should help software engineering researchers both to conduct and report trustworthy SLRs using GenAI and to provide rigorous independent evaluation studies. Evaluation studies, Large Language Models (LLMs), Generative AI (GenAI) tools, Systematic Review, GenAI tool use, GenAI tool evaluation, Software Engineering (SE), Guidelines, Recommendations †copyright: none†journalyear: 2026†journal: TOSEM†journalvolume: 1†journalnumber: 1†article: 1†publicationmonth: 1†ccs: General and reference Surveys and overviews†ccs: General and reference Empirical studies†ccs: Computing methodologies Natural language processing†ccs: Software and its engineering Software development methods 1. Introduction As advocates of the use of systematic literature reviews (SLRs),111In this paper, we use the term systematic literature review to refer to all types of systematically conducted literature review, including qualitative systematic reviews, quantitative systematic reviews, mapping studies, rapid reviews, tertiary studies, and updates to such studies. we are used to both conducting SLRs ourselves and reviewing SLRs conducted by other researchers. Recently, however, we have been asked to review empirical studies investigating the use of Generative AI (GenAI) tools such as Large Language Models (LLMs) to assist with performing systematic review tasks. Unfortunately, there is no well-established methodology for evaluating the performance of GenAI tools in the context of systematic reviews, and we have noticed some significant problems with the methods being adopted. We believe that methodology problems must be addressed quickly before they become adopted as de facto standards by other researchers or even by GenAI tools. We note that bad practices can be extremely difficult to eradicate once they become established. An example of this in software engineering (SE) is the use of the Mean Magnitude Relative Error (MMRE) in cost estimation research. After the publication of Conte et al.’s book on software metrics and models in 1986 (Conte et al., 1986), MMRE was widely adopted as a means of measuring the performance of cost and fault estimation models. However, MMRE is a biased metric. MMRE bias was first subjected to detailed analysis in the early 2000s (Kitchenham et al., 2001; Foss et al., 2003), but the problem of its continued use was later noted both in 2005 (Myrtveit et al., 2005) and in 2012 (Myrtveit and Stensrud, 2012). Another example is incorrect analysis and meta-analysis of crossover experiments, where many researchers failed to realize that, when each participant in an experiment uses all experimental methods, correct statistical analysis is based on the change in the participants’ outcomes, not a simple analysis of all observations (Kitchenham et al., 2020). The goal of this paper is to provide process recommendations to assist researchers who want either to evaluate or to use GenAI tools for conducting SLRs. We discuss a number of methodological issues concerning the use of GenAI tools and identify process recommendations for planning, conducting, and reporting that we hope will be of value to researchers using and evaluating GenAI tools to support SLRs. Our recommendations are primarily intended to support software engineering (SE) research, which is the focus of our own research, but may be of interest to the wider research community. For brevity, we refer to this preliminary set of recommendations as GUEST (GenAI Use and Evaluation in SLR Tasks): like a guest, a GenAI tool should make its contributions under the researchers’ oversight, never replace them. These recommendations are preliminary because we expect them to evolve as both the technology and the empirical evidence mature. In Section 2, we look at the issues that affect the study and use of GenAI tools and LLMs from two perspectives: methodological issues arising from the nature of GenAI tools and LLMs, and their implications for conducting trustworthy SLRs; and methodological issues arising from the nature of SLRs, and the constraints these impose on GenAI tool usage. In addition, we report prior work related to the automation of SLRs, particularly in light of recent advances in GenAI and LLMs. In Section 3, we discuss how we developed our process recommendations. In the first stage, we undertook a search for related work, particularly guidelines for using GenAI tools. In the second stage, using the outcomes from the previous stage and our own experience conducting SLRs and evaluating tools, we drew on thought experiments to develop recommendations for how to use and assess GenAI in the context of SLRs. In Section 4, we report the characteristics of the thought experiments and their results; in particular, Section 4.4 presents the recommendations for both using and evaluating GenAI tools for SLRs. We set out the threats to the validity of our work in Section 5, discuss the implications of our recommendations in Section 6, raise the main open research questions in Section 7, and present some closing remarks in Section 8. 2. Background and Related Work This section discusses factors that affect the way researchers interact with and evaluate GenAI tools, including LLMs, in the context of conducting SLRs. In addition, we report related work on automating the SLR process using GenAI tools. 2.1. The Nature of GenAI tools Generative AI refers to algorithmic approaches based on deep generative models trained on large datasets to learn the underlying data distribution, enabling them to generate new, realistic content (for example, text, images, or code) from simple prompts (Banh and Strobel, 2023). They are able to produce plausible natural-language answers to textual questions on any topic. However, GenAI tools have basic limitations that make their use in the context of SLRs challenging: (1) They create something of a moving target for evaluators and users, and this happens at two levels. The first level is the major model updates that occur (e.g., GPT-4 to GPT-4o to GPT-5.6), though fortunately, these releases are differentiated by formal release announcements, version numbers, and some documentation. Researchers can at least explicitly reference these version details. More problematic is the second, or infrastructure level. This includes: (i) safety filters and content policies, (i) system prompts and guardrails, (i) input preprocessing and output filtering, which can be highly relevant for say PDF processing capabilities, (iv) rate limiting and abuse detection, and finally (v) general UI/API wrapper changes. Unfortunately, Level 2 changes can happen daily or even hourly without any public notice, leading to reproducibility problems. (2) They can occasionally produce invalid or nonsensical output. This is called hallucinating and is the focus of much current research. Huang et al. (Huang et al., 2025) provide a deeper discussion and taxonomy. The causes have been discussed from different perspectives: Kalai et al. (Kalai et al., 2025) argue that hallucinations arise because standard training and evaluation reward confident guessing over acknowledging uncertainty, while He and Thinking Machine Labs (He et al., 2025) attribute the related non-determinism of LLM endpoints to load- and batch-size variation. Mitigation is an active research area (for example, retrieval-augmented generation and fact-checking (Greengard, 2025)), but none of these approaches yet eliminates the problem, so hallucination remains a limitation that users must guard against. (3) They can produce inconsistent outputs from the same input. They respond to questions based on the frequency of textual patterns with an added element of randomization to simulate variations in human responses. In contrast to hallucinations, inconsistencies are not invalid. They represent the variations in responses we find in spoken and written communication. However, it is the responsibility of the GenAI tool user to distinguish between responses that are harmless linguistic variations and those that are invalid hallucinations. (4) They are designed to identify frequent conversational threads and move such threads into their long-term memory. This means that frequent use of the same test material may lead to the test questions and answers being stored in long-term memory, so the GenAI tool can obtain the answers by accessing the long-term memory without actually processing the specific questions. This phenomenon is referred to as data leakage or data contamination. Data contamination represents a serious threat to the use of publicly available datasets to test GenAI tools (Woelfle et al., 2024). In addition, in the context of aggregating evidence from primary studies, the risk of data contamination means that without extremely careful prompting, LLMs may give priority to the most commonly reported outcomes among a set of papers, while ignoring minority opinions and the possibility of publication bias. (5) Some, but not all, GenAI tools allow researchers access to parameters that influence their behaviour. For example, to reduce the risk of inconsistent outputs, researchers can adjust the temperature parameter in LLMs that directly affects the variability and randomness of generated responses. A lower LLM temperature value (close to 0) produces more deterministic and focused outputs for tasks requiring greater levels of reproducibility. In addition, there are different parameters (i.e., top_p, max tokens, frequency penalty, or presence penalty) that can also be adjusted (Brown et al., 2020). However, parameters can interact in unpredictable ways, making the effects of changing multiple settings hard to predict. (6) Currently, GenAI tools are accessed via chatbot prompts. A recent paper discussing the prompting interface has identified some problems with prompting in the context of SE research (Morris, 2024). In particular, Morris identified the problem of researchers not reporting how many prompts they used to obtain the results they reported, and failing to assess and report the robustness of their prompts across different GenAI models and different generations of the same model. (7) Most LLMs are optimized to produce fluent, plausible responses to prompts (Banh and Strobel, 2023) rather than to explain how they arrived at an answer or to justify why it is valid — a limitation that matters for SLRs, where conclusions must be traceable and justified. Furthermore, the major commercial chatbots tend to keep internal reasoning private. The default approach is to hide chain-of-thought or other internal reasoning steps (due to safety and misuse prevention, as well as to keep intellectual property away from competition) and offer only curated, higher-level explanations that aim to foster user trust without compromising safety or competitive advantage. Although there is currently more understanding of the importance of providing justifications and explanations of outputs, it remains the responsibility of the user to put in place a prompt process that will provide any necessary explanatory information. (8) GenAI tools can, and do, provide support for unethical behaviour. In particular, they can readily be used to support plagiarism and will not be restricted by concerns about breach of copyright when used by humans who, themselves, are not concerned about ethical behaviour. (9) They are likely to exhibit bias. They are trained on enormous volumes of Internet-based data and, therefore, “inherit stereotypes, misrepresentations, derogatory and exclusionary language” (Gallegos et al., 2025). In the context of SLR tasks, this bias could lead a tool to downplay or overlook relevant studies — for example, studies not written in English, or whose authors are women or have non-European-sounding names. A recent preprint reports a preference for male-authored references in LLM-driven reference selection (He, 2025). However, we are not aware of peer-reviewed, SE-specific evidence that quantifies this risk for SLR screening, so we flag it as a concern that warrants empirical investigation rather than an established finding. (10) Globally, they consume power and have wider environmental costs (Crawford, 2024; Dauvergne, 2022). However, even in the context of individual empirical studies, power consumption or costs can be an issue. For example, Woelfle et al. (Woelfle et al., 2024) report that they duplicated all LLM prompts to assess reliability, except for GPT-4, where they duplicated prompts only on 25% of the SLRs being evaluated “due to high cost”. Thus, like humans, GenAI tools can make mistakes (although the causes of error are different), and they can behave without regard for scientific ethics or scientific method. In addition, they are designed to be non-deterministic and to conceal their working methods. This means that while some systems are able to produce seductively plausible literature reviews, such reviews are, at best, equivalent to expert opinion reviews, not systematic literature reviews. 2.2. The Nature of SLRs We undertake SLRs to reduce the risk of validity problems associated with expert reviews, such as: • Personal bias. • Incomplete personal knowledge of the relevant literature. • Inclusion of poor-quality studies. • Transcription errors during data collection (as a result of fatigue, misreading, or misunderstanding). • Analysis and synthesis errors (as a result of personal bias, fatigue, or personal knowledge gaps). Because of such problems, expert literature reviews can be unreliable, so there is no way to determine whether specific results and recommendations can be trusted. Such validity risks were undermining scientific progress in many disciplines, particularly medicine and health care (Mulrow, 1987). Hence, to address these problems, medical researchers developed guidelines for conducting SLRs (see (Cumpston et al., 2019) for the latest guidelines), which were subsequently accepted by other empirical disciplines, including Software Engineering (SE). An important feature of these guidelines is that they do not only define a set of SLR activities, but also identify associated SLR process control activities intended to minimize errors. We accept the overheads of process control (e.g., broad searches often adopting a variety of different strategies and multiple researchers doing the same tasks) because we want to reduce the risks of invalid results that are inherent in expert reviews. These process controls cannot guarantee SLR reliability. However, they should increase the likelihood of identifying and correcting human errors. We want our SLRs to produce evidence that is Valid, Traceable, Qualified, and Verifiable. Validity requires that authors find all the relevant literature and draw trustworthy conclusions. It also demands that authors use appropriate methods for all of their SLR process and SLR process control tasks. Traceability requires authors to make clear the links among their research questions, search and selection process, data collection process, and data aggregation process and how each contributes to the final conclusions and recommendations. In the context of evidence from SLRs, Qualified means that authors assess the strength of the evidence supporting their recommendations and conclusions, which requires assessing both the methodological validity of the individual primary studies and the combined strength of the evidence from the set of available primary studies. Finally, to be Verifiable (also referred to as Reproducible or Auditable), authors need to take care to fully document both their study protocols and the conduct of their SLRs, using current reporting standards (Kitchenham et al., 2023). Rapid Reviews (R) and Mapping studies (MS) are sometimes considered to be forms of SLR. But it is important to note that they have different constraints: • RRs trade-off some aspects of the SLR process to obtain results in a short time scale and/or with limited resources. • Mapping studies (other than tertiary studies) seldom require an assessment of the weaknesses of individual primary study methodology or the strength of evidence but instead typically aim to classify primary studies related to a topic area and sometimes develop a taxonomy of primary study types. Nonetheless, such studies can still claim to be systematic if researchers apply process control to the subset of SLR processes that they do use and to deliver reports that conform with SLR reporting standards. Specific SLR tasks. An SLR is a structured research method that answers defined questions by systematically locating, selecting, and analysing all relevant scientific literature. This involves a number of very different but related tasks, each of which requires a valid starting point and needs to deliver a valid input to the subsequent activity. Typically, this is done by inspecting the output of each task, not only to assess whether it is valid, but also to ensure it was produced appropriately (that is, that the process used to obtain the output does not invalidate the chain of evidence from the research question to the final conclusion). To support this, well-established good practices exist for each task (for example, study selection generally involves two or more reviewers independently performing the same activity), along with validation procedures (for example, in independent selection, calculating inter-reviewer agreement metrics). The different SLR tasks are discussed in more detail in Appendix C. 2.3. Related Work SLRs have been adopted by many different academic disciplines and, particularly in healthcare, are an important source of information for practitioners. In SE, we find wide adoption of SLRs, although primarily by academics. For example, Kamei et al. identified 446 SLRs published only in the top SE journals and conferences before 2019 (Kamei et al., 2021), and many more have been published subsequently. However, planning and conducting SLRs are complex activities that require substantial resources and effort. This has long motivated work on their automation. The International Collaboration for the Automation of Systematic Reviews (ICASR)222https://icasr.github.io/ is an international collaboration among several groups working on systematic review automation since 2015. In their first paper (Beller et al., 2018), they argued that automation for SLRs should support and improve tasks across the full evidence-synthesis workflow, be developed through continuous, user-informed refinement (starting with key bottlenecks), and maintain rigorous review standards (for example, PRISMA and MECIR). They also emphasized building flexible, interoperable, modular tools; fostering multidisciplinary collaboration; openly sharing methods, tools, and resources; and evaluating automation rigorously and reproducibly, including both technical performance and usability in real review workflows. They meet annually to share their progress on SLR automation. While early work on SLR automation showed promise, it was only in the last few years, with the emergence of GenAI, that broad automation began to seem realistically achievable. In an SLR on the topic, Van Dinter et al. (van Dinter et al., 2021) found only 41 papers on SLR automation up to June 2020, and these studies were confined to just two domains: software engineering and medicine. Since around that time, there has been a rapid increase in both the number and the diversity of studies investigating how LLMs and other GenAI techniques can automate one or more stages of the SLR process (see, e.g., (Dennstädt et al., 2024; Guo, 2024; Guo et al., 2024)). Several commercial tools tailored to SLRs have also emerged (see, e.g., Elicit333Elicit is an AI research assistant that searches, summarizes, and extracts data from academic papers to automate parts of literature reviews and SLRs. https://elicit.com/ ). However, in the absence of clear guidelines and recommendations, and given the limited technical understanding of GenAI among many users, some initiatives pursue SLR automation in an overly naive way, which can compromise the quality of both the intermediate results and the final outputs of SLRs conducted with these technologies. Against this backdrop, a group of mostly health researchers is developing guidelines to support the use of GenAI in SLRs (Thomas et al., 2026a). The Responsible use of AI in Evidence SynthEsis (RAISE 2026) guidelines are a community-based set of recommendations and companion guidance intended to ensure the responsible, transparent, and ethical use of AI in evidence synthesis, grounded in research integrity. They make clear that their guidelines apply to several different researcher roles. In particular, they point out the distinction between researchers using GenAI tools and LLMs to support the conduct of their SLRs and researchers studying the performance of GenAI and LLM tools when applied to SLR-related tasks. They also make clear that researchers who want to use GenAI tools to support their SLRs need to be able to justify the use of the GenAI tools and report how the tools were trained and their performance evaluated. Finally, given the risks of using GenAI in research within our community, a group of researchers has begun developing guidelines for experimenting with LLMs in software engineering (Wagner et al., 2025). Although these preliminary guidelines apply more broadly to empirical studies that use LLMs, they are also relevant to research that involves planning and conducting SLRs. Two further guideline efforts help to position our work. Baltes et al. (Baltes et al., 2025) provide community guidelines for empirical SE studies involving LLMs, organized around a taxonomy of seven study types. In that taxonomy, systematic reviews are one study type rather than the focus, so the guidance is not tailored to specific SLR tasks. In the evidence-synthesis community, Farotimi et al. (Farotimi et al., 2025), in an editorial in Research Synthesis Methods, give guidance for manuscripts that test GenAI for systematic review and meta-analysis, framed around reporting, validation, and reproducibility requirements rooted in the medical and broader evidence-synthesis literature. Table 1. Positioning of our guidelines relative to related GenAI evidence-synthesis guidance. Work Domain SLR-specific? Roles / granularity Grounding This paper Software engineering Yes Two researcher roles (using vs evaluating); per-SLR-task, SE-tailored methods Rapid review, thought experiments, companion SE studies RAISE 2026 (Thomas et al., 2026a, b, c) Evidence synthesis; health/Cochrane-rooted; SE not addressed Yes Eight ecosystem roles; principles plus task-level metrics (RAISE-2) and tool-use recommendations (RAISE-3) Large multi-organization expert consensus Farotimi et al. (Farotimi et al., 2025) SRMA; health/cross-discipline Yes Manuscript authors; reporting/validation by SRMA stage Editorial; adopts RAISE and DEST Baltes et al. (Baltes et al., 2025) Empirical SE (general) No (seven study types) Researchers and reviewers; study-type level, not SLR-stage Community guidelines plus checklist Wagner et al. (Wagner et al., 2025) Empirical SE (general) No SE researchers; foundational, high-level Position paper (precursor to Baltes et al.) Table 1 summarizes how our guidelines relate to these efforts. RAISE is the closest related work: it is task-aware (RAISE-2 discusses evaluation metrics and RAISE-3 gives tool-use recommendations by synthesis stage) and addresses both the using and evaluating roles within a broader, eight-role evidence-synthesis ecosystem. RAISE originates in the health and broader evidence-synthesis tradition, where evidence often takes relatively standardized forms such as randomized controlled trials. Software engineering, by contrast, exhibits greater heterogeneity of SLR and study types, and RAISE does not specifically address it. The same applies to the Farotimi et al. editorial, which addresses authors in the evidence-synthesis community rather than software engineering. Baltes et al. and Wagner et al. are SE-focused but target empirical LLM studies in general rather than SLRs. Our contribution is distinctively SE-specific: (i) a focus on the heterogeneity of SLR and study types in software engineering, especially the prevalence of qualitative SLRs; (i) an in-depth treatment of the two researcher roles (using versus evaluating GenAI for SLRs); (i) SE-tailored, task-level methods; and (iv) empirical grounding through our companion studies on literature screening (Madeyski et al., 2026) and qualitative synthesis (Pizard et al., 2026). Relationship to our companion studies. Two of these works are our own peer-reviewed studies, and we use them here as the empirical backbone for our screening and synthesis recommendations rather than re-deriving that evidence. Our study of literature screening (Madeyski et al., 2026) (published in Information and Software Technology) provides the performance-metric analysis, the reanalyses of published SE screening evaluations, and the validation guidance that underpin our screening recommendations. Our study of qualitative synthesis (Pizard et al., 2026) (published at WSESE’26) provides the trial-based failure modes and soundness criteria that underpin our synthesis recommendations. This paper makes a distinct, higher-level contribution: a role-based framework that covers all the main SLR tasks, integrates GenAI tool validation into the SE SLR planning, conduct, and reporting process, and positions this guidance against related efforts (Table 1). To keep the present paper focused on that framework, we summarize the companion findings where they motivate a recommendation and refer readers to the companion studies for the task-level detail, which we do not reproduce here. 2.4. Rationale for our guidelines As noted earlier, our goal is to provide recommendations to guide researchers when they evaluate or use GenAI to support SLRs. With this in mind, our guidelines contribute to and differ from prior work in the following ways: • They build on prior recommendations from the field where SLRs originated, but also draw on other related work and our own experience in software engineering. In particular, our work complements guidance developed in the health and broader evidence-synthesis tradition, such as RAISE (Thomas et al., 2026a) and the Farotimi et al. editorial (Farotimi et al., 2025), by focusing specifically on software engineering, where SLRs and primary studies are more heterogeneous and qualitative SLRs are common. Our contribution is SE-specific: SE-tailored, task-level methods and a detailed account of how to integrate GenAI tool validation into the SE SLR workflow. • They focus only on the two predominant roles in our discipline: (1) researchers who evaluate GenAI to support SLRs and (2) reviewers who conduct an SLR using GenAI. This both simplifies our proposal by focusing on the roles we believe will most benefit from the guidelines and allows us to go deeper, providing more deliberate analyses of the challenges these roles face and more detailed guidance to support them. • They also draw on our computing expertise regarding the nature, limitations, and risks of GenAI and LLMs. We see this as essential, especially after observing that many studies evaluating or using these technologies in SLRs underestimate risks or conflate key concepts related to GenAI. 3. Developing Our Recommendations: Method and Related Guidelines Guidelines for Evaluating & Using GenAI for SLRs Informal Searches & Experience as Reviewers Stage 1. Conductof Literature Reviewon GenAI for SLRs(Rapid Review)Stage 2. Elaborationof Guidelines(Thought Experiment) Conduct of Study on Using LLMs for Qualitative Synthesis (Pizard et al., 2026) Conduct of Study on Screening Papers Using GenAI (Madeyski et al., 2026) Outcomes:∙ Guidelines for Using GenAI for∙ Academic Research (Table 2).∙ Study found on GenAI-Assisted∙ Quality Appraisal of SLRs (Woelfle et al., 2024).Grey Literature Found:∙ Guidelines for Empirical SE & LLMs (Wagner et al., 2025).∙ Guidelines for Using LLMs in SLRs (Thomas et al., 2026a).motivatedmotivatedmotivatedinputs toinputs to Figure 1. Context and development process for the guidelines on evaluating and using GenAI for SLRs. Our goal is to propose guidelines for evaluating and using GenAI to support SLR planning and conduct. As mentioned earlier, these guidelines were motivated by our recent experience as reviewers of papers reporting related studies. This motivation also led some of us to conduct two complementary studies that informed the guidelines: one on using GenAI for paper screening (Madeyski et al., 2026)444When we began to develop our recommendations, we used a preliminary draft of the paper. and another on using LLMs for qualitative synthesis (Pizard et al., 2026). Figure 1 situates the guidelines within this broader body of work, showing the stages we carried out and the inputs used in their development. 3.1. Background When we started to develop process recommendations, we were aware of two extremely important grey literature reports that have directly influenced our process recommendations: (1) Wagner et al., “Towards Evaluation Guidelines for Empirical Studies involving LLMs” (Wagner et al., 2025)555When we began our study, only the preliminary guidelines were available.. (2) Thomas et al., “Responsible use of AI in Evidence SynthEsis (RAISE 2026) 1: Recommendations for practice” (Thomas et al., 2026a)666When we began our study we used an earlier version of this document (Thomas et al., 2024). However, because the list of recommendations we used when developing our recommendations were unchanged in the different versions of the document, we reference the most recent version of the article throughout this article.. 3.2. Stage 1: Literature Search Given the possibility of other useful papers, we conducted a Rapid Review (R) of the literature (Pizard et al., 2025), searching for any English-language papers that reported guidelines for evaluating the performance of GenAI tools and LLMs in the context of SLRs. The R process, including the research questions, eligibility criteria, search strategy, the search strings, search results, data extraction, data synthesis and limitations, is discussed in the Appendix A. Here we summarize the review protocol and its main results and conclusions. In brief, the protocol was as follows. • Research question. Which papers report guidelines or recommendations for evaluating the performance of GenAI tools in the context of conducting SLRs? • Sources. We searched a single digital library, SCOPUS, because it indexes a wide range of important journals and conferences across many domains, including the computer science and medical domains (where SLR guidance has often been developed first); it supports advanced search capabilities including Boolean search strings; and drawing candidate papers from one source reduces the risk of including multiple versions of the same study report in the candidate set, which could distort the synthesis. We did not restrict the search by discipline or by GenAI tool, and we checked sufficiency by verifying that the searches recovered a pre-defined set of known relevant papers (Tables 7 and 8). • Eligibility. We included peer-reviewed papers (journal or conference, including editorials and letters) that presented guidelines or recommendations for evaluating or using GenAI tools in an SLR context and whose full text was available in English. We excluded papers available only on self-publication sites (such as arXiv), and papers that used SLR results to make practice recommendations (for example, clinical guidelines). • Search and validation. We ran two SCOPUS searches: Search 1 (13/03/2025) targeted GenAI use in SLRs in general, and Search 2 (29/03/2025) broadened the terms to cover specific SLR tasks such as screening, data extraction, risk-of-bias assessment, and qualitative synthesis. The search strings (Tables 9 and 10) were drafted by one author and refined by another, and their effectiveness was validated against the known papers. Search 1 recovered all 13 papers in our known set at that stage, and Search 2 recovered all 18 known papers before it was restricted to guideline terms, which indicates that a single-source SCOPUS search was sufficient for our goal of covering the important themes rather than retrieving every relevant paper. We analysed guidelines obtained from six papers that reported general guidelines for using GenAI tools to support academic research and identified six major guidelines reported in Table 2 (from now on, these are referred to as G1 to G6). G6 is related to peer review, so only the first five guidelines are of direct relevance to our process recommendations. These guidelines provided a legal and ethical framework within which to develop our process recommendations. The integrated guidelines are not completely independent. Human accountability is an overriding requirement, while guidelines G2 to G6 represent specific means of ensuring human accountability. Table 2. General Guidelines for Using GenAI tools for Academic Research ID Guideline References G1 Humans are accountable for AI tool use in terms of outcome validity and adherence to legal and scientific ethics. (da Veiga, 2025; Uribe et al., 2025; Kim et al., 2023; Perkins and Roe, 2024a) G2 All AI-assisted outputs should be reviewed by humans. (da Veiga, 2025; Inam et al., 2024; Kim et al., 2023; Lubowitz, 2024; Perkins and Roe, 2024a) G3 Do not assign authorship to AI tools. (da Veiga, 2025; Uribe et al., 2025; Inam et al., 2024; Kim et al., 2023; Lubowitz, 2024; Perkins and Roe, 2024a) G4 All AI tool use, with the exception of editing and refining text, should be fully reported. (da Veiga, 2025; Uribe et al., 2025; Inam et al., 2024; Lubowitz, 2024; Kim et al., 2023) G5 AI tools should not create the full text of reports, nor create, alter or manipulate images, data, citations, results, conclusions or recommendations. (da Veiga, 2025; Inam et al., 2024; Kim et al., 2023; Lubowitz, 2024) G6 AI tools should not be used for peer review. (Inam et al., 2024; Lubowitz, 2024) Subsequent to our R, Communications of the ACM published a paper describing how computer science conferences are reacting to the use of GenAI tools in scientific writing by Nahar et al., (Nahar et al., 2025). Apart from G6, their recommendations mirror the guidelines in Table 2. Nahar et al. note that risks related to exposing sensitive information during peer review can be mitigated by configuring LLM settings to prevent models from learning or retaining user-provided data. This suggests that G6 is over restrictive in the context of GenAI tools, although editors would need to ensure that appropriate safeguards are in place777However, configuring model settings may not be a sufficient safeguard for Agentic AI tools: whether built on GenAI models or not, they act autonomously and may go beyond their instructions.. After completing our R, we concluded that: (1) There were no readily available English language research papers indexed by SCOPUS (up to 29/03/2025) that addressed our goal of producing recommendations for using GenAI tools to support SLRs. (2) The literature screening recommendations proposed by Madeyski et al. (Madeyski et al., 2026) were at a lower level of abstraction than Thomas et al.’s recommendations (Thomas et al., 2026a), and recommendations from the two articles could not be easily integrated. (3) Although we would continue to develop general recommendations for evaluating and using GenAI tools for conducting SLRs, more detailed recommendations (e.g., similar to those produced by Madeyski et al. (Madeyski et al., 2026) and later by Pizard et al. (Pizard et al., 2026)) related to different SLR tasks (including risk of bias assessment, data extraction, qualitative synthesis, strength of evidence assessment) would also be of value. Detailed task-specific investigations of all SLR processes are beyond the scope of our current study, but we produced briefing notes describing the individual tasks included in each of these SLR processes to assist the development of preliminary task-related recommendations. (4) The paper by Woelfle et al. (Woelfle et al., 2024) provided very valuable insight into the problems associated with using GenAI tools to support qualitative assessment. Our R deliberately targeted papers that report guidelines or recommendations, as stated in its research question. It was not designed to aggregate the empirical literature on how GenAI tools actually perform on SLR tasks. We address that empirical evidence separately. Section 4.3 synthesizes the failure modes reported in empirical studies — including our companion studies on literature screening (Madeyski et al., 2026) and qualitative synthesis (Pizard et al., 2026) — and maps them to the SLR tasks they affect. We use this synthesis as an evidence backbone for our recommendations. Many of these empirical studies were also among the known papers our searches recovered (Tables 7 and 8). Although the searches were run in March 2025, our recommendations do not rest on the guideline literature frozen at that date: we continued to track the field, verifying that successive versions of the RAISE recommendations left unchanged the list we had used (Thomas et al., 2026a), positioning our work against guidance published after the searches — Baltes et al. (Baltes et al., 2025) and the Farotimi et al. editorial (Farotimi et al., 2025) (Table 1) — and cross-checking our integrated guidelines against the conference policies analysed by Nahar et al. (Nahar et al., 2025). These conclusions and Table 2 are the final outcomes of our rapid review. The approach used to develop recommendations addressing the use of GenAI tools to support SLRs is presented in Section 3.3. 3.3. Stage 2: Developing Recommendations for Evaluating GenAI tools The general guidelines in Table 2 do not address the specific process issues related to using GenAI in the context of SLRs. However, they make it clear that our recommendations must recognize that academic research involving GenAI tools, including conducting SLRs, requires human oversight. In addition, the results of our R made it clear that we could not rely on published research and needed a different approach to develop useful recommendations for evaluating GenAI tools in the context of SLR. We considered it important that our recommendations support two different researcher roles in the context of SLR-related research: (1) Tool Evaluators whose primary interest is assessing the performance of SLR tools. Such researchers will need to consider amending their standard research practices when studying GenAI tool capability. In this study, we restrict ourselves to researchers undertaking independent assessments of GenAI tools. Evaluators working as part of an AI tools development team should refer to Thomas et al. (Thomas et al., 2026b). (2) Reviewers whose primary interest is conducting high-quality SLRs. In order to use such tools, researchers will need to know how to conduct any necessary pre-deployment validation of GenAI tools, to assess the reliability of tool outputs and to maintain human oversight. We do not include any discussion of issues related to the selection of specific GenAI tools. For tool selection advice refer to (Thomas et al., 2026c). In addition, useful recommendations should support both SLR article reviewers and subsequent readers. As well as complying with the general recommendations in Table 2, we wanted our recommendations to be consistent with the Responsible use of AI in Evidence SynthEsis (RAISE 2026) guidelines (Thomas et al., 2026a) and the guidelines for empirical SE studies (Wagner et al., 2025). In addition, we wanted to take advantage of the results reported in three other papers: (1) Madeyski et al. (Madeyski et al., 2026) discuss the selection and interpretation of performance metrics used to evaluate GenAI tools when screening titles, abstracts and keywords (TAK) of candidate primary studies for eligibility. This article informed our initial recommendations, and a later revision, which extended its recommendations to full-text screening and (in principle, pending empirical validation) to data extraction, was used to refine them. (2) Woelfle et al. (Woelfle et al., 2024) discuss in detail a study investigating the use of GenAI tools to evaluate the quality and rigour of SLR reports using three different evaluation instruments. (3) Pizard et al. (Pizard et al., 2026) who discuss the issues involved when using GenAI tools to support qualitative synthesis. We also found another relevant SE study by Baltes et al. (Baltes et al., 2025), which we used to validate our recommendations. There are also two other RAISE-related studies that cover issues not explicitly addressed in our recommendations: (1) Thomas et al. (Thomas et al., 2026b) which discusses issues relating to building GenAI tools. (2) Thomas et al. (Thomas et al., 2026c), which discusses the selection of tools, summarizes the current capabilities of a number of existing tools. Although these papers are of value for researcher teams producing and evaluating GenAI tools for SLRs, they are outside the scope of our recommendations. 3.4. Stage 2: Method used to Develop Recommendations The method we used to develop role-related recommendations was to use a Thought Experiment of the process issues we would need to address when planning, conducting and reporting evaluations of GenAI tool performance. A thought experiment is an organized exercise of the mind in which a hypothetical situation is imaginatively devised and explored to examine, assess, refine, or illuminate ideas, theories, or propositions (Stuart et al., 2018). This method has been used previously in software engineering (see, e.g.,(Melegati et al., 2024; Ko, 2019; Monperrus, 2014)), and several authors describe it as a useful complement to purely empirical research (see, e.g., (Génova et al., 2012)). The advantages of a thought experiment based approach are: (1) Considering the processes used for planning, conducting and reporting, should provide a common level of abstraction for recommendations. (2) The two roles and the three process stages provide a framework for assessing the completeness of the recommendations. (3) A thought experiment would allow us to utilize our own experience of SE experimentation and conducting SE SLRs to identify process recommendations that are proactive rather than retrospective. (4) A process-based presentation identifying the difference between the standard process for tools evaluation and SLR conduct would provide a practical context for researchers wanting to use the recommendations. It would also make clear the scope and limitations of the guidelines. If researchers find they require a process step not covered by the guidelines, they could use the issues raised in Section 2 or their own thought experiment to assess whether there are any GenAI tool or SLR requirements they need to be aware of in the context of the novel process step. (5) SE researchers with experience of conducting SLRs should be familiar with the use of thought experiments, because it is the method used to develop SLR protocols. We focused our thought experiment on identifying how standard evaluation and SLR processes would need to be changed to incorporate GenAI tools. We decided to conduct a thought experiment for each researcher role because, although the roles overlap, there are significant differences. Our R of existing guidelines identified that a general problem with developing recommendations from existing guidelines was that both the existing guidelines we found and other related articles treated the evaluation at different levels of abstraction. For example, our existing recommendations related to screening the literature ((Madeyski et al., 2026)) and qualitative synthesis ((Pizard et al., 2026)) provided some detailed phase-related recommendations, while other guidelines (i.e., (Wagner et al., 2025) and (Thomas et al., 2026a)) were generally at a higher level of abstraction. In order to support SLR researchers, we felt that we needed to consider the specific issues related to all the main SLR tasks. To do this, we developed briefing notes detailing the tasks required to conduct risk of bias assessment888Often referred to as quality assessment in the SE literature. and strength of evidence assessment (also referred to as certainty of evidence). These are presented in Appendix C. We also include briefing notes on literature screening (both abstract-based and full-text), based on Madeyski et al. (Madeyski et al., 2026); on data extraction, which draws on that study’s principled extension of its metric recommendations to extraction tasks (pending empirical validation); and on qualitative synthesis, based on Pizard et al. (Pizard et al., 2026). This allowed us the opportunity to check that all appropriate issues were included in our recommendations. We exclude discussion of the initial literature search process because it usually depends on the capabilities of external search engines and data libraries, which are outside the control of individual researchers or research groups. 4. Summary of the Thought Experiments This section describes the initial assumptions about the roles that influenced our thought experiments, provides a general overview of the thought experiment results, and identifies related failure modes reported in empirical studies. Then we report the recommendations for both evaluators and reviewers developed from our thought experiments and SLR task briefing notes. Finally, we validate our recommendations by comparing them with related studies ((Baltes et al., 2025; Thomas et al., 2026a)). 4.1. Context and Scope of the Thought Experiments This section discusses the starting point for our thought experiments by highlighting some of the similarities and differences between the two researcher roles. For the evaluator role, we adopted the viewpoint of researchers with experience in empirical research methods and GenAI tools, intending to undertake independent evaluations of GenAI tools’ performance. We assumed that any rigorous evaluation of a GenAI tool requires an experimental framework consistent with best practice that: • Allows the performance of a GenAI tool to be compared with current practice. • Does not unfairly favour either the GenAI tool or current practice. • Allows limitations of the tool and risks associated with tool use to be identified, as well as tool benefits and opportunities. • Is reported at a level of detail that supports repeatability and meta-analysis. Evaluators can have many different goals, including but by no means limited to: • Comparative evaluation and benchmarking of different GenAI tools. • Comparative evaluation of GenAI tools and other more deterministic SLR support tools. • Evaluation of different prompting strategies. • Evaluation of methods to predict the performance of GenAI tools in the context of a specific SLR. In addition, evaluators need to be ready to assess new GenAI tools, intended to address SLR tasks that are beyond the capability of existing tools. In particular, tools intended to address qualitative analysis and synthesis, and tools intended to support (or assist) the evaluation of the reliability of primary studies and the strength of evidence supporting SLR recommendations. For the reviewer role, we adopted the viewpoint of members of a review team looking to adopt GenAI tools. We expect members of a review team to have experience with the basic SLR process standards and reporting guidelines, and a general understanding of the nature of GenAI tools. We assumed that the review team would want to ensure that: • Any use of GenAI tools would conform with all relevant legal and ethical requirements for academic use of GenAI tools in terms of ensuring human oversight and full disclosure of use, as outlined in Table 2. • The use of GenAI tools did not compromise SLR validity. • Any use of GenAI tools to replace a human researcher during the conduct of an SLR was justified by a well-conducted validation of tool performance, which provides empirical support for tool adoption. Note, undertaking a validation intended to decide whether to use an AI tool to replace a human researcher is an example of an important overlap between the evaluator and reviewer roles. For both roles, our thought experiments were aimed at identifying any changes researchers would need to make to their study planning, conduct and reporting process when using or evaluating GenAI tools. However, we have assumed that any research study is being undertaken in the context of academic research where the general ethos supports free and open reporting of research studies. We have not considered the issues involved when software engineers are working in any industry context and may have a requirement to protect their company’s commercial interests. For industry-based researchers, the basic planning and conduct recommendation may be of value, but some of the reporting requirements might be unnecessary. In addition to considering the process changes needed to support GenAI tool evaluation and use of GenAI tools, we also wanted to explore issues related to the individual SLR processes. SLR processes have different purposes and characteristics, which any GenAI tools aimed at automating specific SLR processes need to properly address. For this reason, we constructed briefing notes that describe the different processes and their sub-tasks and how GenAI tools could support them. Our briefing notes are reported in Appendix C and provide an important addition to our recommendations for evaluators and reviewers. We did not produce any briefing notes related to preliminary SLR planning, the search for candidate primary studies, quantitative analysis, or production of SLR reports, because: • Preliminary planning includes developing research questions, search strings and eligibility criteria. We discuss these topics in Appendix B.2.1. Wang et al. (Wang et al., 2023) reported an evaluation of automating the construction of Boolean queries. One of the prompting methods Wang et al. tested was a designed guided prompt based on the method human researchers use to create a prompt. This approach is similar to the approach we used to develop our technical briefing notes. However, in our opinion, automating these initial planning tasks may not require a formal validation, as long as such GenAI tool use is fully reported, and reviewers take responsibility for how any GenAI tool output is used. • Searching for candidate studies relies on the facilities provided by digital libraries, which are outside the control of SLR researchers. • For quantitative analysis, the use of meta-analysis is based on existing statistical methods and GenAI tools are not needed. • When using GenAI tools to assist the production of SLR reports, as long as assistance is restricted to linguistic and grammatical improvements and all such help is overseen and assessed by human researchers, it is sufficient to report such usage without any additional tests or validation. 4.2. Overview of the Thought Experiment Results In the context of evaluation studies aimed at benchmarking GenAI tool performance, it is critical to properly address the risk of data contamination associated with retrospective analyses of previously published SLRs. Recommendations we found in the literature emphasized that prospective studies are the most reliable means of addressing this risk. Fortunately, reviewers using GenAI tools during SLR are in an excellent position to undertake prospective studies of GenAI performance. Thus, SE researchers who usually have experience in both SE tool evaluation and using SLRs as an integral part of their standard research process are in a particularly good position to contribute to evaluating GenAI tool support for SLRs. Furthermore, the need for prospective studies confirms the value of SE-oriented recommendations for GenAI tool performance studies for conducting GenAI-supported SLRs that consider both roles. Our briefing notes in Appendix C highlight major differences between the types of SLR tasks that would impact both reviewers and evaluators: • Screening candidate primary studies for inclusion in an SLR involves multiple repetitions of the same task, often involving processing thousands of individual research papers. It is the main scenario in which a review team might want to perform a full validation based on a random selection of candidate studies with the aim of deciding whether a human researcher could be replaced. The literature screening process is repetitive, time-consuming and error-prone, and represents a clear opportunity for GenAI tools to increase SLR efficiency even if validation samples are relatively large, as demonstrated by (Cao et al., 2024). The use of GenAI tools in the context of an ongoing SLR can deliver prospective assessments of GenAI tool performance while reducing the risk of data contamination. However, the risk is not eliminated if evaluators use multiple GenAI tools and multiple prompt strategies on the same data sample. • Data extraction, including classification of primary studies for mapping studies, is also repetitive, but is also more varied, with major differences between the numerical values required for quantitative primary studies and textual data required for qualitative primary studies. Also, data extraction is based only on the available primary studies. The number of primary studies might be a relatively large number for mapping studies, but it will probably be relatively small for quantitative and qualitative SLRs. Thus, apart from mapping studies, the opportunity for reducing human researcher effort is likely to be relatively small, while the effort required to construct GenAI tool prompts and related data collation processes could be considerable. • Qualitative synthesis is problematic because there are many different methods of qualitative analysis and synthesis, and they are less well defined than quantitative methods. Using interpretive approaches, it may be difficult for different human researchers to agree on what constitutes a valid synthesis, making it difficult to establish a baseline for comparison with a GenAI-based synthesis. In general, outcomes are extremely varied, including, for example, textual descriptions of identified themes, models of the relationships between influencing factors, or lists of risks and benefits. Only simple lists lend themselves easily to the use of quantitative agreement metrics, without which judging the relative performance of GenAI tools and human researchers objectively is difficult. • Risk of bias (RoB), which is often referred to as quality assessment in the SE literature, involves assessing the methodology used in each primary study to identify whether there are any serious weaknesses that cast doubts on the study results. Primary studies with serious methodological weaknesses may be excluded from the SLR synthesis. RoB assessment is similar to the process researchers use when they are asked to review manuscripts submitted to conferences or journals. Although unnecessary for mapping studies, RoB is required when reviewers intend to synthesize primary study findings. Appendix C.3 identifies that in the case of SE, the problems associated with automating RoB relate to identifying appropriate evaluation instruments for SE primary studies and to the expertise required to understand and use those criteria. In terms of researchers conducting SLRs, the individual sub-tasks discussed in C.3 offer the opportunity for GenAI tools to act as an additional research assistant, for example, by highlighting primary study text that addresses a specific criterion, or making suggestions about the value of a specific criterion. • Strength of evidence assessment involves assessing whether the evidence supporting a specific SLR finding is trustworthy. Noting that SLRs are retrospective studies, the strength of evidence assessment is similar to identifying and investigating possible confounding factors, which is required when assessing the limitations of a quantitative primary study or the search for alternative viewpoints in the context of a qualitative study. Like the RoB assessment, this process is not required for mapping studies. However, it is essential for assessing the reliability of findings derived by synthesizing results from multiple primary studies. Overall, the briefing notes suggest that GenAI tools are well-suited to automating mapping study processes. With well-defined eligibility criteria and well-defined research questions, it would seem possible to fully automate literature screening and primary study classification for mapping studies. Furthermore, mapping studies do not usually require risk of bias (RoB) assessment or strength of evidence assessment, which are more complex processes. Even so, for a systematic mapping, providing traceability between study goals and study findings and supporting reproducibility are essential, so GenAI tools would need to provide a final report that conforms with mapping study reporting requirements (Kitchenham et al., 2023). For formal SLRs that require RoB assessment, detailed data extraction, data syntheses and strength of evidence assessment, the prospect of full automation is poor. In the context of data extraction for quantitative and qualitative synthesis, the variety of different study types typically found in SE SLRs makes full automation unlikely. RoB and the strength of evidence rely on knowledge not contained in any specific primary study text. In the case of RoB, this process requires both knowledge of the methodological weaknesses of different empirical methods and the experience to make inferences from what is not reported, as well as what is reported in individual primary studies. In the case of strength of evidence assessment, reviewers need to make an assessment of the reliability of individual findings obtained from a set of primary studies. This requires a rigorous assessment of the factors that could bias findings, even from a group of good-quality primary studies. For example, a group of laboratory experiments may all be of good methodological rigour, but may be based on student participants or relatively simple tasks, so experimental findings might not generalize to practitioners and industry-scale tasks. In terms of researchers conducting SLRs, the individual sub-tasks for complex tasks such as RoB assessment, qualitative synthesis and strength of evidence assessment offer the opportunity for GenAI tools to act as an additional research assistant, for example by highlighting primary study text snippets that address a specific issue and/or issues related to a specific evaluation criterion, or suggesting an answer to an evaluation question, or proposing a qualitative synthesis finding. In such a scenario, the purpose of the GenAI tool usage would not be to save research effort, but: (1) To provide independent validation of the task performance of the human researchers and, thus increase the reliability of the SLR findings. (2) To provide a prospective assessment of GenAI tool performance on more difficult SLR processes. 4.3. Failure Modes Reported in Empirical Studies A recurring concern with guidelines of this kind is that they can rest on expert judgement rather than reported evidence. To address this, we synthesized the failure modes that empirical studies have reported when GenAI tools are applied to SLR tasks, and we mapped each task to the recommendation that responds to it (Table 3). The clearest evidence concerns two tasks. For literature screening, our companion study (Madeyski et al., 2026) reanalyses several software engineering screening evaluations and shows how conventional performance metrics mislead under class imbalance and how readily relevant studies are lost. In one such 9,695-article evaluation, the metric used to choose the best-performing LLM largely determined how much evidence was lost: the LLM ranked best by accuracy missed 63.3% of the relevant studies and the best by plain MCC missed 43.9%, whereas the best by the cost-sensitive Weighted Matthews Correlation Coefficient (WMCC) proposed in that study (Madeyski et al., 2026) missed only 5.8%. We summarize these findings here and refer the reader to that study for the underlying metric analysis and data. For qualitative synthesis, Pizard et al. (Pizard et al., 2026) report, from autoethnographic trials, failure modes such as plausible-but-incorrect output, loss of interpretation, sensitivity to prompts, and a verification burden as large as the manual task. For data extraction, risk-of-bias assessment, and strength-of-evidence assessment, direct empirical evidence specific to these tasks is currently limited, so we treat the related guidance as open research questions (Section 7). This synthesis forms the evidence backbone for the recommendations that follow. Table 3. Failure modes reported in empirical studies, mapped to SLR tasks and to the recommendations that address them. SLR task Reported failure mode (source) Recommendation(s) that address it Literature screening Conventional metrics (accuracy, specificity, F1F_1) mislead under the severe class imbalance of screening: an LLM can score over 96% on accuracy yet detect none of the relevant studies. False negatives cause lost evidence (the proportion of relevant studies missed, 1−Recall1-Recall); incomplete reporting of confusion-matrix counts prevents re-analysis and meta-analysis; and workload-saving claims that count the cost of false positives but ignore the much higher cost of false negatives (falsely rejecting relevant studies loses evidence, potentially irretrievably) can hide a serious threat to the validity of the review (Madeyski et al., 2026). Above all, use WMCC so the far higher cost of false negatives (lost evidence) than false positives is reflected in the score (Scr2); plain MCC (Scr1) corrects for chance and class imbalance but not for this cost asymmetry. Report full confusion-matrix counts (Scr3), so recall and lost evidence (1−Recall1-Recall) can be computed and monitored; address the risk of data contamination (Plan4). Data extraction Direct empirical evidence on GenAI performance for SE data extraction is currently limited. Data-extraction recommendations (Ext1–Ext7); treated as an open research question pending further evidence (Section 7). Qualitative synthesis LLMs produce plausible but incorrect syntheses and deliver plausibility rather than interpretation, so output can look useful while distorting the primary studies; small prompt changes, or even the same prompt, yield different output, undermining traceability and reproducibility; models carry training biases; and verifying that a synthesis is sound can take as much effort as doing it manually (Pizard et al., 2026). Qualitative-synthesis recommendations Syn1–Syn3 (specify the method; specify the performance criteria; specify how human and tool syntheses are compared); for reviewers, restrict tool use to validating human work and review all output (RevA1–RevA2); planning recommendation Plan4 (built-in biases). Risk-of-bias assessment RoB assessment depends on methodological knowledge not contained in any single primary study; direct empirical evidence on GenAI performance is currently limited. RoB/strength-of-evidence recommendations (A1–A3); framed as open research questions (Section 7). Strength-of-evidence assessment SoE assessment requires judging the reliability of findings across multiple studies; direct empirical evidence on GenAI performance is currently limited. RoB/strength-of-evidence recommendations (A1–A3); framed as open research questions (Section 7). 4.4. Summary of Recommendations This section presents the detailed recommendations for evaluators and reviewers arising from integrating the results of our role-based, process-oriented thought experiments (see Appendix B) with the issues identified by our briefing notes (see Appendix C). Our thought experiments make it clear that evaluators interested in studying the GenAI tools and reviewers wanting to use GenAI tools to support the conduct of a specific SLR share a number of process-related issues, particularly issues related to planning and reporting the use of GenAI tools. To avoid repetition, we organized our recommendations into three groups: those common to both roles and those specific to each role. We report the more general recommendations first, in terms of planning, conducting, and reporting recommendations, followed by the recommendations related to individual SLR processes. Our recommendations are of four kinds, summarized in Table 4, and we separate them so that readers can tell which ones are specific to SLRs. The first kind covers SLR practice changes due to GenAI tool use. Such changes arise in the tasks that belong to systematic reviews in particular: literature screening, data extraction, qualitative synthesis, and risk-of-bias and strength-of-evidence assessment. These recommendations are the distinctive contribution of this paper, and most of them form the SLR-process recommendations below. The second kind restates good practice that is already well established for secondary studies in general, with or without GenAI. Examples are documenting exclusion decisions and guarding against lost evidence. We treat these as known concerns for secondary studies and align them with established reporting guidance: SEGRESS (Kitchenham et al., 2023) for software engineering, and PRISMA 2020 (Page et al., 2021) for evidence synthesis in a broader context. Some of these concerns also take an SLR-specific form. Guarding against lost evidence is a general aim of any secondary study, but measuring and limiting it when a GenAI tool screens the literature is specific to our setting: it needs cost-sensitive metrics that weight a missed relevant study above a wrongly retrieved one, which our companion study (Madeyski et al., 2026) develops in detail (see Scr1–Scr4 and ScrRev1–ScrRev3 below). The third kind is generic to almost any empirical evaluation of a GenAI tool, not just to SLRs. Examples are justifying the choice of tool, reporting the model version, its parameters and the prompts used, and using an open tool as a baseline. We align these with the emerging SE guidance on empirical studies that use LLMs: Baltes et al. (Baltes et al., 2025), Wagner et al. (Wagner et al., 2025), and RAISE (Thomas et al., 2026a). The fourth kind relates to good practice for any evaluation study, whether or not it involves a GenAI tool. This includes ensuring that objectives, goals and research questions are formally stated, using appropriate evaluation methods, and reporting evaluation results with reproducibility in mind, for example by providing replication packages and open data (Huotala et al., 2025a). There are many academic papers discussing such issues. In Table 4 we cite only a few, including studies in which we have been directly involved. Most of our planning, conduct, and reporting recommendations are of the second, third or fourth kind. We include them for completeness, so that the guidance can be used on its own, not as a claim to novelty. The SLR-process recommendations, where our distinctive contribution lies, are mostly of the first kind. In Table 4 the four kinds are labelled (i) to (iv). Table 5 and Table 6 annotate each recommendation with its kind. Table 4. Classification of our recommendations: which are specific to SLRs, and which existing guidance we align the general ones with. Category What it covers (examples) Related existing guidance SLR practice changes due to GenAI tool use (i) Tasks particular to SLRs: literature-screening metrics, data-extraction performance assessment, qualitative synthesis, and risk-of-bias and strength-of-evidence assessment. The distinctive contribution of this paper. — (this paper; see also our companion studies (Madeyski et al., 2026; Pizard et al., 2026)) General secondary-study practice (i) Good practice for any secondary study, with or without GenAI: documenting exclusion decisions and guarding against lost evidence (general aim; SLR-specific control under screening metrics). SEGRESS (Kitchenham et al., 2023); PRISMA 2020 (Page et al., 2021) General GenAI evaluation (i) Practice generic to almost any empirical evaluation of a GenAI tool: justifying tool choice, reporting model version, parameters and prompts, and using an open tool as a baseline. Baltes et al. (Baltes et al., 2025); Wagner et al. (Wagner et al., 2025); RAISE (Thomas et al., 2026a) General evaluation practice (iv) Good practice for almost any evaluation study whether or not GenAI-based: defining goals and hypotheses, using an appropriate study design and data analysis methods, and supporting reproducibility, including replication packages and open data. Kitchenham et al. (Kitchenham et al., 2002, 2019); Ralph (Ralph, 2021); Huotala et al. (Huotala et al., 2025a) (replication packages) 4.4.1. Planning, Conduct, and Reporting Recommendations These recommendations are based on an assessment of the different roles. Table 5 lists them, grouped into planning, conduct, and reporting, together with the role each recommendation applies to and its kind. Table 5. GUEST planning, conduct and reporting recommendations. Id Recommendation Role Kind Planning Plan1 Justify the choice of GenAI tools and critically appraise them. Both (i) Plan2 Justify the choice of study design. Both (iv) Plan3 Define the methods to be used to evaluate GenAI tool performance. Both (iv) Plan4 Ensure that the evaluation processes are designed to detect tool errors and to minimize risks to validity from built-in tool biases and from data contamination caused by repeated analyses of the same dataset within the study. Both (i) Plan5 Identify the data needed to assess tool performance and plan the data collection and storage process and any quality control procedures. Both (iv) Plan6 Base comparisons on results from at least two human researchers. Both (i) PlanE1 Clearly define evaluation goals, research questions, and hypotheses. Evaluators (iv) PlanE2 Use an open GenAI tool as a baseline. Evaluators (i) PlanE3 Identify any assumption made at the start of the evaluation (e.g. define what input data is required to initiate the task). Evaluators (iv) PlanE4 Justify the use of existing publicly available datasets, given that the datasets, and published analyses of them, may already be part of a GenAI tool’s training data. Evaluators (i) PlanE5 Plan an appropriate data analysis, including meta-analysis if the evaluation aims to summarize results from multiple SLRs. Evaluators (iv) PlanR1 Consider using GenAI tools as an additional research assistant to assist in the specification of research questions and eligibility criteria. Reviewers (i) PlanR2 Specify all the SLR processes and tasks to be supported by GenAI tools. Reviewers (i) PlanR3 Adopt a case-study-based approach to using GenAI tools to provide prospective evaluations of tool performance. Reviewers (i) Conduct Con1 Ensure that any deviations from the plan are identified and reported. Both (iv) Con2 Ensure data collection processes are used correctly. Both (iv) Con3 For GenAI case studies within SLRs, ensure that all human researcher effort required to support GenAI tool use is reported. Both (i) Reporting Rep1 Report the prompt strategy, the prompt development process, the prompts used to deliver the results reported in the study, and the date when the final prompt was executed. Both (i) Rep2 Report all GenAI tool settings and parameters. Both (i) Rep3 Accurately assess and report all costs of GenAI tool use. Both (i) Rep4 Report the results and conclusions of the evaluation results with process reproducibility and future meta-analysis in mind. In particular, support open access to data, analysis algorithms and results. Both (iv) Rep5 Report any study limitations and constraints, including any outstanding concerns regarding the reliability of GenAI tools’ results. Both (i) RepE1 When making recommendations aimed at conducting SLRs, explain how the GenAI tool(s) can be incorporated into current SLR processes. Evaluators (i) RepE2 When making recommendations aimed at conducting SLRs, explain how the GenAI tool use complies with requirements for human oversight. Evaluators (i) RepE3 Report any additional use of GenAI tools to plan, conduct or report the tool evaluation process. Evaluators (i) RepE4 Report any financial interest or any other special issues, and how bias was avoided. Evaluators (iv) RepR1 Follow appropriate SLR reporting guidelines (i.e., SEGRESS (Kitchenham et al., 2023) or PRISMA 2020 (Page et al., 2021)). Reviewers (i) RepR2 Report any additional use of GenAI tools to report the SLR. Reviewers (i) 4.4.2. Recommendations Related to SLR Processes Based on analyses of specific SLR processes. Table 6 lists the recommendations for the individual SLR processes: literature screening (title, abstract, keyword (TAK) and full text), data extraction, qualitative synthesis, and risk of bias (RoB) and strength of evidence (SoE) assessment. For literature screening, our companion study develops these principles into ten operational recommendations (R1–R10), differentiated by benchmarking and deployment study types (Madeyski et al., 2026). We are not yet aware of SE-specific empirical evidence on GenAI tool performance for RoB and SoE assessment, so those recommendations are preliminary (see Section 7 for the main open research questions). Table 6. GUEST SLR-process recommendations. Id Recommendation Role Kind Literature screening Scr1 Use Matthews correlation coefficient (MCC) to ensure results are better than chance.a Both (i) Scr2 Use WMCC to properly account for cost differences associated with False Positives (FPs) and False Negatives (FNs). Both (i) Scr3 Report full confusion matrix counts to support meta-analysis and re-analysis of results.a Both (i) Reviewer-based validation studies undertaken during the conduct of an SLR: ScrRev1 Planning: Acceptable performance levels for GenAI tools (or the process to set them) should be identified and justified prior to the validation exercise. Reviewers (i) ScrRev2 Planning: Have appropriate contingency plans to address the validation results. Reviewers (i) ScrRev3 Conduct and Reporting: Provide confidence intervals for the selected performance metrics based on resampling without replacement. Reviewers (i) Data extraction Ext1 Plan for data extraction to include descriptive textual information such as study type, study goals, study setting, publication date, first author, and publication type. Both (i) Ext2 For mapping studies requiring classifying primary studies against pre-defined categories, use MCC (with an extension if multiple outcomes or multiple categories are required). Both (i) Ext3 For mapping studies requiring classifying primary studies against pre-defined categories, avoid WMCC unless there is an explicit justification for assuming differential costs for data extraction errors. Both (i) Ext4 For quantitative data extraction, correctness metrics should use the values explicitly reported in the primary study text as the gold standard whether the extraction was performed by human researchers, or by a tool. Both (iv) Ext5 For more complex data extraction, such as that supporting qualitative analysis, define which sub-tasks in the data extraction process are being automated.b Both (i) Ext6 For more complex data extraction, such as that supporting qualitative analysis, define the starting point of the automated process in terms of knowledge about primary studies.b Both (i) Ext7 For more complex data extraction, such as that supporting qualitative analysis, specify how comparisons between human researchers and GenAI tools will be conducted and assessed. Both (i) Qualitative synthesis Syn1 Specify and report the qualitative synthesis method being adopted and explain why it is suitable given the SLR research questions and the SLR primary studies. Both (i) Syn2 Specify and report the criteria used to assess GenAI tool performance, given the qualitative synthesis method being adopted. Both (i) Syn3 Specify and report how the synthesis of GenAI tools will be compared with the synthesis provided by human researchers, in particular, how relative performance will be measured and assessed. Both (i) Risk of bias and strength of evidence A1 Define which sub-tasks are being automated.b Both (i) A2 Define the information required as input to each sub-task and how it is obtained.b Both (i) A3 Specify how the synthesis of GenAI tools will be compared with the synthesis provided by human researchers and the relative performance measured. Both (i) Qualitative synthesis, risk of bias, and strength of evidence — reviewers RevA1 Restrict GenAI tool use to providing additional process validation of tasks performed by human researchers. Reviewers (i) RevA2 Ensure human researchers review all GenAI tools’ outcomes and take full responsibility for how the outcomes are used. Reviewers (i) a Although MCC and full confusion-matrix reporting are long established in general classifier evaluation, we classify Scr1 and Scr3 as SLR-specific. Applying them to SLR screening evaluations is not established practice: conventional metrics such as accuracy and F1F_1 mislead under the severe class imbalance of screening (Section 4.3), and our companion study develops the metric guidance for this setting in detail (Madeyski et al., 2026). b We classify Ext5, Ext6, A1, and A2 as SLR-specific rather than general practice. They presuppose automation, so they do not restate established secondary-study practice, and the decomposition they require is of the specific SLR task: which parts of it are delegated to a GenAI tool, and what knowledge of the primary studies the automated process starts from. The general SLR process changes for reviewers that are implied by these recommendations are summarized in Figure 2 (Appendix B.2). 4.5. Validation After constructing our recommendations, we compared our set of recommendations with those constructed by Baltes et al. (Baltes et al., 2025), which we did not use for developing our recommendations, and those reported by Thomas et al. (Thomas et al., 2026a), which we used while developing our recommendations. The Baltes et al. study, which refines and extends the authors’ earlier workshop position paper (Wagner et al. (Wagner et al., 2025)) and is therefore a closely related line of work rather than a fully independent one, provided a recent SE baseline against which to cross-check our recommendations, while the study by Thomas et al. provided a check that the different methods we trialled to report our results had not led to any important issues being lost. We compared our recommendations with the eight guidelines of Baltes et al. (Baltes et al., 2025). Six are covered by our recommendations: declaring GenAI use and role; reporting model versions, configurations, and customizations; using suitable baselines, benchmarks, and metrics; using an open LLM as a baseline; using human validation of outputs; and reporting limitations. Of the remaining two, “Report System and Prompt Design” is covered for prompts but not for tool architecture beyond the model, which concerns building new tools and so falls outside our scope (which excludes the GenAI tool developer role); and “Report Session Traces” is a complementary reporting practice that our data-management recommendation partly supports. We note in our threats to validity (Section 5) that we do not address GenAI tool development. We compared our recommendations with the recommendations provided by Thomas et al. (Thomas et al., 2026a) for evidence synthesists (which we refer to as reviewers) and evidence synthesist methodologists (which we refer to as evaluators). For both roles, Thomas et al. provide guidelines relating to the wider SLR community, which they refer to as the “evidence synthesis ecosystem”. The ecosystem comprises the two researcher roles we consider, plus the other six roles they consider: AI tool building teams; organizations that produce evidence synthesis; funders and commissioners of evidence synthesis and AI; publishers of evidence synthesis; users of evidence synthesis; trainers of evidence synthesis methods. In our recommendations and thought experiments, we have considered only the relationship between reviewers and evaluators in terms of the overlap between the roles and the opportunity to conduct prospective studies while conducting SLRs. Given the limited scope of our guidelines, we have not formally compared our recommendations with any recommendations related to developing the ecosystem. Excluding recommendations related to the ecosystem, Thomas et al. specified ten numbered recommendations for reviewers, one of which included four subitems, of which one was split again into two further subitems. The following three recommendations were not included in our recommendations: • Reviewers must take a critical view of available tools. This is important for evaluators as well as reviewers, so we have reworded our original Plan1 recommendation to identify this issue. • Reviewers need to be aware that research papers may be written by AI. The implications of this recommendation are unclear, so we have made no change to our recommendations. • Report any financial or other special issue. For reviewers, this is covered by our reference to using current reporting standards (RepR1). Excluding recommendations related to the ecosystem, Thomas et al. specified eight main recommendations for evaluators. One was split into three subitems, while another related to reporting was split into six subitems. The following recommendations were not included in our recommendations: • Adopt best practice. This seemed unnecessary, so we have not changed our recommendation but we have added a comment to our section discussing the thought experiment context. • Use the most appropriate study design. This seemed to be an example of current best practice, but we have added a new recommendation to make this issue explicit. We believe it is relevant to adopting GenAI as well as evaluating AI, so recommendation Plan2 asks evaluators and reviewers to justify their choice of study design, because it is difficult for a reader to know whether a study design was appropriate or not unless researchers explain why it was used. • Register review-related studies. This practice is not currently adopted for SE studies, so we have not made any change to our recommendation. • Report and publish results. We have identified reporting as an integral part of our thought experiments and have identified various recommendations related to reporting. We have not made any additional changes to our recommendations but have identified in our assumptions that the recommendations are aimed at researchers. • Report data collection process. We have added the need to plan data collection and storage, and any necessary quality control procedures, to an existing recommendation to identify the data needed to assess tool performance. • Report any exclusion criteria. We are unsure about the scope of the recommendation, so we have not made any changes. • Report data collection and collation. This was addressed by a change to the recommendation to define required data. • Report data quality assurance. This was addressed by a change to the recommendation to define the required data. • Support open access. We recommended that evaluators and reviewers report results with reproducibility and future meta-analysis in mind. However, we have revised the recommendation to explicitly add the need for open access. • Report any financial interest and how bias was avoided. We have added a new recommendation that applies to evaluators (RepE4). In the RAISE-2 document (Thomas et al., 2026b), Thomas et al. discuss performance metrics. However, they provide general definitions of various metrics and their use, rather than the more SLR process-focused and detailed recommendations we propose. For this reason, we have not undertaken any item-based comparison of our performance metric recommendations and the RAISE-2 discussions. An issue raised by our validation was the importance of ensuring adequate explanation of recommendations. We were unsure how to interpret Thomas et al.’s recommendations related to research papers being written by AI, and exclusion criteria. In contrast, our attempt to relate recommendations clearly to the evaluation and SLR processes may help readers correctly interpret our recommendations. In addition, the recommendation to report evaluation studies may be obvious in the context of research-based evaluators and reviewers, but is more difficult in the context of industry-based tool builders, who may be required to protect the commercial interests of their companies. This emphasizes that it is important to make explicit the assumptions underlying a set of recommendations. 5. Threats to Validity Given our research characteristics and the use of several methods, we did not adopt a particular framework to analyse the threats to validity; instead, as suggested in (Lago et al., 2024), we considered the dimensions most significant for our research objectives. In particular, we studied the construct, internal, and external validity, as well as the reliability of our process. 5.1. Construct validity Construct validity concerns whether our recommendations, and the risks they are intended to address, correspond to the real threats that GenAI tools pose to the validity of an SLR. The main threat is that recommendations derived from thought experiments could target the wrong issues, or miss important ones. We reduced this threat in three ways. First, we grounded the recommendations in empirical evidence: the failure modes they address are drawn from empirical studies of GenAI use in SLR tasks, including our companion studies on screening (Madeyski et al., 2026) and qualitative synthesis (Pizard et al., 2026), and are mapped to the tasks they affect (Section 4.3). Second, we informed the recommendations with a rapid review of existing guidelines for using GenAI in research (Section 3). Third, we cross-checked our set against the independent guidelines of Baltes et al. (Baltes et al., 2025), which we deliberately did not use while developing our recommendations, and found that they agreed on most points (Section 4.5). 5.2. Internal validity Internal validity concerns whether our derivation process could have produced biased or incorrect recommendations. The main threat is that a thought experiment draws on the researchers’ own experience, so unstated or unrecognized assumptions can shape its results. We saw direct evidence of this: comparing our evaluator recommendations with those of Thomas et al. (Thomas et al., 2026a), we found several items that they stated explicitly but that we had treated as restatements of standard good practice. We therefore added those generally accepted good-practice items for completeness, even though we had assumed the role of experienced evaluators. Even so, we cannot guarantee that we have articulated every assumption. Two features of our process reduce this threat. The recommendations were developed and reviewed by several authors with extensive experience of conducting and reviewing SLRs, and they were corroborated by an independent guideline set (Section 4.5). A further threat affects the rapid review that informed our work: a single researcher selected the candidate papers, which could have introduced selection bias (Appendix A). 5.3. External validity External validity concerns how far our recommendations generalize beyond the setting in which we produced them. First, our thought experiments and briefing notes rest on our experience as software engineering researchers, so some recommendations may be less relevant to other disciplines, and researchers in those fields will need to judge which ones transfer. Because our recommendations state general good practice, researchers must also interpret them in the context of their own study, with its particular research questions and hypotheses. Second, we address only the evaluator and reviewer roles, not the GenAI tool developer role. That role is important in software engineering and would benefit from its own recommendations, and is taken up by related RAISE guidance on building tools (Thomas et al., 2026b). Third, GenAI tools are changing quickly and their performance varies across disciplines and study types, so recommendations that hold today, or in one setting, may need revisiting in another. We return to both points in Section 6. 5.4. Reliability Reliability concerns whether other researchers, repeating our process, would arrive at similar recommendations. A thought experiment and a rapid review are both judgement-based, so an exact replication is unlikely. We reduce this threat by documenting the process so that others can follow and scrutinize it: we report the two researcher roles and the three process stages that framed the thought experiments (Section 3.3), the briefing notes for each SLR task (Appendix C), and the research question, eligibility criteria, search strings and known-paper validation for the rapid review (Appendix A). This documentation will not make the process deterministic, but it makes our reasoning transparent and open to challenge. 6. Discussion Our recommendations support the viewpoint that the use of GenAI tools in the context of academic research must be based on human oversight. Although GenAI tools are already capable of providing accurate summaries of the research literature, such summaries cannot be equated to systematic literature reviews because currently they do not deliver associated documentation, allowing their summaries to be replicated by other human researchers using a trustworthy systematic literature review process. The requirement for human supervision will not disappear even if we have GenAI tools that are theoretically capable of delivering a complete SLR, together with associated documentation. Scientific progress has been undermined by expert opinions in the past, and results based on one (or more) GenAI tool cannot be regarded as inherently more reliable than results based on expert opinion. To use such output, we still require human researchers to take responsibility for validating all the SLR processes and their outcomes. Furthermore, if we allow GenAI tools to take over some SLR tasks, human researchers may lose the ability to properly assess the reliability of GenAI tool outcomes. In the context of screening literature for inclusion in an SLR, Spillias et al. (Spillias et al., 2024) point out that, for PhD students the goal of undertaking an SLR is not just to contribute to the literature, but also to learn more about their chosen topic. We would add that it is also important to learn about the SLR process. It is unlikely that researchers who do not know how to undertake an SLR manually will understand how best to prompt a GenAI tool to perform an SLR task, or how to evaluate the outcome of its use.999See Binz et al. 2025 (Binz et al., 2025) for a broader discussion of the implications of GenAI tools for scientific research. Currently, one of the most problematic issues for evaluations seems to be the danger of contaminated test data. This issue is not unique to the evaluation of GenAI tool performance. However, in the case of GenAI tools, it is not caused by poorly understood or badly compiled data; it is due to GenAI tool processes that are invisible to the researcher conducting the study. This means, even if we suspect there are data problems (for example, the evaluation results are too good to believe), it is extremely difficult to detect and confirm data contamination. This means that standard data mining approaches based on re-analysing publicly available datasets, which have served empirical software engineering research well in the past (when data sets are used with care), are unreliable for assessing GenAI tool performance. Dennstadt et al. (Dennstädt et al., 2024) point out that “To objectively assess how well an LLM-based solution can evaluate scientific publications for new research questions, large cultivated [sic] independent prospective data sets on many different topics would be needed”. It is important that researchers compiling such data sets provide the results of their own GenAI tool analyses, so other researchers can use their results without re-running the GenAI tool analyses. Huotala et al. (Huotala et al., 2025b) provide this kind of benchmark for SE researchers. Although their benchmark is based on past completed SLRs, the re-analysis provided by (Madeyski et al., 2026), was obtained from the reported results without the contamination risks inherent in re-running the original data analyses. In addition, we agree with Thomas et al. (Thomas et al., 2026a) and Devane et al. (Devane et al., 2022) that one important option for gathering evidence about GenAI tool performance is to incorporate evaluation of GenAI tools into our basic SLR process. For example, using GenAI tools to perform SLR tasks in parallel with human reviewers would mean that each SLR would provide a prospective case study of GenAI tool support for SLR tasks. Furthermore, the technology is changing rapidly, so it would be foolish to assume that any process recommendations will not need to be adapted as the technology improves.101010See, for example, Jovanović and Campbell’s discussion of large reasoning models and large concept models (Jovanović and Campbell, 2025). Thus, if we make the evaluation of tools to support SLRs part of our SLR process, we will be in a better position to keep our evaluation methods in line with technology changes. Using GenAI tools in an SLR is also an iterative and incremental activity rather than a single linear pass. Planning, prompt development, tool testing and validation, and conduct form feedback loops, which we describe in the summary of process changes (Appendix B.2). Our discussion in Section 2.1 and comments in (Woelfle et al., 2024) suggest that it is challenging to ensure GenAI tools produce trustworthy results, because the underlying design of such tools does not readily support reproducibility and transparency. Bearing this in mind, we should not forget that if our goal is to improve the way we conduct SLRs, tools based on more deterministic algorithms should also be investigated. For example, Šuster et al. (Šuster et al., 2024) report an experiment used to compare FlanT5XL with a supervised tool called RobotReviewer that showed RobotReviewer outperforming FlanT5XL. In addition, if we want simply to improve the validity of SLR reports, using GenAI tools as an additional research assistant provides prospective assessment of tool performance and may reveal errors or omissions made by human researchers. Our recommendations do not assume that only a single GenAI tool is used. Where it improves robustness, a review team can run several tools, or the same tool with several prompts, and combine the outputs — an ensemble — with human judgement. Tools such as AISysRev (Huotala et al., 2026) support multi-model title–abstract screening. Combining tools can reduce idiosyncratic errors, but it does not remove the risk of data contamination, which can affect several tools trained on overlapping data, and it increases cost, so the trade-off should be reported. Our recommendations were developed for GenAI tools applied to individual SLR tasks, but agentic AI platforms — end-to-end systems in which LLM agents plan and execute multiple review stages automatically (Thomas et al., 2026b) — fall within their scope, because each embedded stage raises the same process issues we discuss. However, two of those issues become more severe. First, our validation process operates on a task-by-task basis, and end-to-end platforms do not always expose the intermediate outputs of each stage. Thomas et al. (Thomas et al., 2026b) identify two evaluation strategies for such platforms: evaluating each stage in isolation when the platform permits, or evaluating only the final report. The second strategy inherits the problems discussed in Section 2.1, since a report whose provenance cannot be inspected is at best equivalent to expert opinion, and the evaluation is invalid if the reviews used to judge the report were part of the tool’s training data (Thomas et al., 2026b). Second, the behaviour of an agentic run is steered not only by the initial prompts but also by the context that accumulated during the run, such as the plans the agents generate and the outputs of the tools they call. Reporting the initial prompts is therefore not enough to make an agentic study reproducible. We agree with Thomas et al. (Thomas et al., 2026b) that reports generated by end-to-end platforms should not be treated as a rigorous evaluation of such platforms, and we regard the rigorous evaluation of such platforms as an open research question. We hope that our recommendations will help SE researchers to improve the evaluation of GenAI tools, because improved evaluation studies will support more reliable meta-analyses. Single evaluation exercises cannot provide all the evidence we need to properly assess the risks and benefits of GenAI tool use. However, we urge meta-analysts to expect discipline-related study heterogeneity. For example, Spillias et al. (Spillias et al., 2024) point out that studies from the medical domain may deliver better performance when screening papers for inclusion in SLRs because they have a better-defined and more stable vocabulary than other disciplines. Another issue is that medical studies often rely on RCTs, which can be assessed using tools such as the Cochrane RoB-2 method. When aggregating medical primary studies based on RCTs, meta-analysts are better able to identify poor quality studies than when aggregating primary studies based on a wider range of experimental methods. 7. Open Research Questions For literature screening and qualitative synthesis we can draw on empirical evidence (Section 4.3). For three further tasks — data extraction, risk-of-bias (RoB) assessment, and strength-of-evidence (SoE) assessment — there is little or no empirical evidence on GenAI tool performance for the heterogeneous study types found in SE, and what evidence exists (largely from medical randomized controlled trials) is contradictory. Our recommendations for these tasks are therefore preliminary and process-oriented, and we set out the main questions we believe must be answered before firmer, evidence-based guidance is possible. For RoB assessment: • Which evaluation instruments are appropriate for the heterogeneous study types found in SE SLRs? The SE community has not agreed on such instruments (Appendix C.3), and without agreement GenAI tool performance cannot be assessed consistently. • Can GenAI tools apply RoB instruments reliably? Current evidence is contradictory: applying RoB questionnaires to randomized controlled trials, Šuster et al. (Šuster et al., 2024) found LLMs “seldom surpassing trivial baselines”, whereas Lai et al. (Lai et al., 2024) reported “substantial accuracy and consistency”. • Can GenAI tools help to construct or refine RoB instruments for SE methods, rather than only apply existing ones? For SoE assessment: • Can GenAI tools assess the strength of evidence for a finding, given that this requires reasoning about confounders and bias across a set of studies using knowledge not contained in any single primary study? • How should a GenAI tool’s SoE judgements be validated against human assessment, when that human assessment is itself subjective? For data extraction, the question is narrower: how reliably can GenAI tools extract the heterogeneous quantitative and qualitative data items found in SE primary studies, and how should that reliability be measured? Finally, several questions cut across all of these tasks. Can a GenAI tool serve as a judge or oracle for an SLR task, given that the human researchers used as its comparison can themselves misjudge, and what should serve as the reference “correct” answer when no infallible human baseline exists? And why are two human researchers required for these subjective tasks — is this an irreducible safeguard, or could a validated GenAI tool justifiably replace one of them, and what level of agreement with human researchers would justify such a replacement? We raise these as open questions rather than resolving them, because answering them will require the kind of empirical evidence the field is only beginning to gather. 8. Conclusions GenAI tools are a hot topic. They are widely adopted by industry and widely discussed in academia. They represent a new technology that few researchers understand but that has great potential to change practices in industry and academia. Furthermore, the nature of GenAI tools means we can no longer rely on our usual processes for evaluating the accuracy of software tools. Thus, we need to rethink our evaluation methods when we investigate the performance of GenAI tools. In this paper, we have presented GUEST, a preliminary set of recommendations that extend existing research processes to address some of the problems faced by researchers using, testing, and evaluating GenAI tools in the context of SLR conduct. We argue that GenAI tools are unable to produce SLRs, but that the ability of such tools to produce summaries of research papers provides a strong incentive for further research. These recommendations will help researchers to develop more rigorous evaluation methods that properly address both the limitations and the potential of GenAI tools. They will help journal editors and reviewers avoid accepting papers simply because the topic is important, rather than because the authors have made a compelling case that their evaluation methods, and hence their results, are trustworthy. Appendix A Details of our Rapid Review At the start of our review we were aware of two relevant reports: (1) A preliminary version of Thomas et al. (Thomas et al., 2026a), Responsible use of AI in Evidence SynthEsis (RAISE 2026) 1: Recommendations for practice. (2) Wagner et al. (Wagner et al., 2025), Towards Evaluation Guidelines for Empirical Studies involving LLMs. A.1. Goal and Research Questions At the beginning of our research, we sought to identify papers that would inform our development of guidelines for evaluating and using GenAI tools in the context of SLRs. Specifically, we searched for papers that already provided guidelines to help researchers assess the performance of GenAI tools in SLRs, as well as any related work that could help us formulate useful guidelines. This led to the research question: RQ Which papers report guidelines or recommendations for evaluating the performance of GenAI tools in the context of conducting SLRs? A.2. Research Method Strategy The following issues affected our choice of methodology: • We did not want to limit our search to Software Engineering and Computer Science sources, as guidelines for conducting SLRs have often been first developed within the medical research community. • As a result of requiring cross-domain searches, we expected to find large numbers of irrelevant papers. We decided that our search goal should be to obtain coverage of themes (i.e. issues that were considered important by any existing guideline papers) rather than attempt to identify all existing papers. We expected any attempt at achieving full coverage would fail because of the large numbers of papers being published. Furthermore, we had no intention of relying solely on existing research, we wanted to adopt a risk assessment approach using our own analysis of the issues facing use of GenAI tools, the processes used to reduce threats to validity for SLRs, and our experiences of performing tool evaluations. We wanted to make use of our own experiences in order to identify potential problems before they become embedded in de facto standards in the same way that MMRE did. • We wanted a reproducible baseline for identifying relevant articles, so we intended to include only articles available in journals or conference proceedings, not on self-publication sites. However, we did not intend to exclude editorials or letters. • Finally, we wanted to deliver the results of such a review in as expeditious a manner as possible to practitioners and researchers given the incredible growth in research into, and use of, GenAI based tools. A key feature of a Rapid Review (R) —as the name implies— is its timeliness. We also intended to use the evidence ourselves as input for developing our guidelines. Therefore, we followed the recommendation that RRs be used primarily in collaborations with a clearly defined requester or knowledge user, especially in low-resource settings (Pizard et al., 2025). These issues led us to adopt a Rapid Review ((Cartaxo et al., 2018) and (Pizard et al., 2025)) rather than a full SLR. We decided to use SCOPUS as our only digital library because: (1) Using output from a single source reduces the issue of including multiple versions of the same study report in the set of candidate primary studies. (2) SCOPUS indexes a wide range of important journals and conferences in many different domains, including the medical and computer science domains. (3) Its search processes conforms with the rules of Boolean algebra meaning that we could use more complex search strings without inflating the number of irrelevant studies found. (4) Scopus indexes articles published in journals and conferences, including editorials and letters. A.3. Eligibility Criteria We were seeking articles that: • I1: Presented guidelines or recommendations for the evaluation of GenAI tools. • I2: Presented guidelines or recommendations for using GenAI tools to support academic research. • I3: Had the full text of the article available in English. • I4: Had a context that involved the use of systematic literature reviews or tasks associated with SLRs. We did not want our searches to be restricted to a specific GenAI tool or LLM. Neither did we want to restrict our search discipline. These criteria were the basis of our keyword-based search process. We also excluded studies based on the following criteria: • E1: Articles that would not address our research questions were ones that used the results of one or more SLRs to generate recommendations for practice, such as health care guidelines. We planned to exclude any such articles during the literature screening process. • E2: Articles that were self-published and available only on a site such as arXiv. A.4. Search Process Our search process was developed to identify two types of paper: (1) Studies that evaluated the use of GenAI tools to support various types of SLR (hereafter, Search 1). (2) Studies that evaluated the use of GenAI tools to support specific SLR tasks rather than SLR types (hereafter, Search 2). Our search strings were intended to address inclusion items I1, I2, and I4 and exclusion item E2 reported in A.3. The search strings were initially designed by Pizard and reviewed and refined by Madeyski. Each search string was built from groups of synonyms for the main concepts in our research question, combined with Boolean operators. The string for Search 1 combined a group of SLR-type terms (drawn from standard SLR methodology vocabulary, for example, “systematic review”, “mapping study” and “scoping review”) with a group of technology terms (general terms such as “large language model”, “LLM” and “generative AI”, plus the names of specific GenAI front-end systems). String validation was based on assessing the effectiveness of the string (without any restriction related to guidelines papers) at finding the 13 papers without asterisks in Table 7. In addition, Table 8 reports other known papers that were not expected to be found by our searches. For Search 2 we evolved the string in two ways: we broadened the SLR-type group to name individual SLR tasks (for example “screening”, “data extraction”, “risk of bias”, “qualitative synthesis” and “strength of evidence”), so that we would also find papers addressing a single task, and we added a group of guideline terms (“guideline”, “guidance” and “recommendation”) together with further GenAI model names. String validation was based on assessing the effectiveness of the string (without any restriction related to guideline papers) at finding all the 18 papers in Table 7. The source of the known papers were the authors of this paper, some citations from identified papers, and a set of 20 papers found by an SLR that evaluated the use of GenAI to support qualitative analysis methods (de Morais Leça et al., 2025). The papers related to qualitative analysis are indicated by an asterisk and were only used to validate String 2. Search 1 was based on a stepwise search strategy and conducted on 13/03/2025. First, we performed a general search for studies on LLMs and SLRs (see first row of Table 9). Second, since authors sometimes used proprietary technology names instead of general terminology, we used the names of specific frontend systems (see second and third rows of Table 9) that we were familiar with and those mentioned in (Pearson, 2024) and (Gibney, 2025). Third, the results of the individual searches were merged (obtaining 2462 papers) and validated against the known papers, finding the 13 known papers that we expected to find. As we expected, none of the papers in Table 8 were found by the search, although some were formally published subsequently. Finally, from this list of papers, we selected those studies that included “guidance” or “guideline” in their titles, resulting in 13 candidate primary studies. Madeyski and Pizard continued refining the search process because they were concerned that we would miss papers that referred to papers addressing issues related to specific SLR tasks rather than SLRs in general. For Search 2 Madeyski changed which document fields were matched. Search 1 had used the combined TITLE-ABS-KEY field, whereas in Search 2, the TITLE and AUTHKEY fields are searched separately. In Scopus, AUTHKEY matches only the keywords an author assigns to a document, whereas the combined keyword field (KEY) used within TITLE-ABS-KEY also matches Scopus-assigned index terms, trade names and chemical names.111111See the Scopus search field definitions: https://schema.elsevier.com/dtds/document/bkapi/search/SCOPUSSearchTips.htm. Restricting the keyword match to AUTHKEY therefore returned less polluted results. Table 10 shows the SCOPUS string used for Search 2. It was conducted on 25/03/2025 and returned 873 papers. These included the 18 known papers: 13 were the same relevant papers found by Search 1, and 5 were related to qualitative assessment. The search string was then restricted to select papers related to guidelines (see Table 10). This search string was applied on 29/03/2025 and delivered 20 candidate primary studies. Table 7. Relevant Known papers (The asterisk identifies papers included in this set after expanding the search strings to consider the stages of the SLR. They were not used to assess Search 1.) Id Title Reference Source KN1 Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study Guo et al. (Guo et al., 2024) Huotala (Huotala et al., 2024) KN2 Can ChatGPT write a good boolean query for systematic review literature search? Wang et al. (Wang et al., 2023) Petersen (Petersen and Gerken, 2025) KN3 Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages Khraisha et al. (Khraisha et al., 2024) Huotala (Huotala et al., 2024) KN4* CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models Gao et al. (Gao et al., 2024) Leca et al. (de Morais Leça et al., 2025) KN5* Exploring the Use of Artificial Intelligence for Qualitative Data Analysis: The Case of ChatGPT Morgan (Morgan, 2023) Leca et al. (de Morais Leça et al., 2025) KN6 Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions Alshami et al. (Alshami et al., 2023) Felizardo (Felizardo et al., 2024) KN7 Leveraging Large Language Models for Literature Review Tasks - A Case Study Using ChatGPT Zimmermann et al. (Zimmermann et al., 2024) Shepperd KN8 Leveraging the potential of generative AI to accelerate systematic literature reviews: an example in the area of educational technology Castillo-Segura et al. (Castillo-Segura et al., 2023) Shepperd KN9 On the road to interactive LLM-based systematic mapping studies Petersen and Gerken (Petersen and Gerken, 2025) Madeyski KN10* Performing an Inductive Thematic Analysis of Semi-Structured Interviews With a Large Language Model: An Exploration and Provocation on the Limits of the Approach De Paoli (De Paoli, 2024) Leca et al. (de Morais Leça et al., 2025) KN11 PRISMA Systematic Literature Review, including with Meta-Analysis vs. Chatbot/GPT (AI) regarding Current Scientific Data on the Main Effects of the Calf Blood Deproteinized Hemoderivative Medicine (Actovegin) in Ischemic Stroke Anghelescu et al. (Anghelescu et al., [n. d.]) Felizardo (Felizardo et al., 2024) KN12* Prompts, Pearls, Imperfections: Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis Wachinger et al. (Wachinger et al., 2024) Leca et al. (de Morais Leça et al., 2025) KN13 Screening articles for systematic reviews with ChatGPT Syriani et al. (Syriani et al., 2024) Shepperd KN14 Should we collaborate with AI to conduct literature reviews? Changing epistemic values in a flattening world Ngwenyama and Frantz (Ngwenyama and Rowe, 2024) Shepperd KN15 The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews Huotala et al. (Huotala et al., 2024) Leca et al. (de Morais Leça et al., 2025) KN16* The use of Generative AI in qualitative analysis: Inductive thematic analysis with ChatGPT Perkins and Roe (Perkins and Roe, 2024b) Leca et al. (de Morais Leça et al., 2025) KN17 Transforming literature screening: The emerging role of large language models in systematic reviews Delgado-Chaves et al. (Delgado-Chaves et al., 2025) Kitchenham KN18 Zero-shot generative large language models for systematic review screening automation Wang et al. (Wang et al., 2024) Shepperd Table 8. Other known papers not expected to be found by SCOPUS Title Reference Source Reason Assessing the ability of chatgpt to screen articles for systematic reviews Syriani et al. (Syriani et al., 2024) Shepperd Published version available after search Automated Title and Abstract Screening for Scoping Reviews Using the GPT-4 Large Language Model Wilkins (Wilkins, 2023) Huotala (Huotala et al., 2024) Not peer reviewed Automating research synthesis with domain-specific large language model fine-tuning Susnjak et al. (Susnjak et al., 2025) Shepperd Not in SCOPUS Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Robinson et al. (Robinson et al., 2023) Huotala (Huotala et al., 2024) Not peer reviewed ChatGPT application in Systematic Literature Reviews in Software Engineering: an evaluation of its accuracy to support the selection activity Felizardo et al. (Felizardo et al., 2024) - Indexed by SCOPUS after search. Exploring the Use of AI in Qualitative Analysis: A Comparative Study of Guaranteed Income Data Hamilton (Hamilton et al., 2023) Leca et al. (de Morais Leça et al., 2025) Not mention any variant of generative AI or LLMs in the title or author keywords. Prompting is all you need: LLMs for systematic review screening Cao et al. (Cao et al., 2024) Kitchenham Published version available after search Towards Evaluation Guidelines for Empirical Studies involving LLMs Wagner et al. (Wagner et al., 2025) Shepperd Published version available after search Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology Natukunda and Muchene (Natukunda and Muchene, 2023) Felizardo (Felizardo et al., 2024) Not mention any variant of generative AI or LLMs in the title or author keywords. Utilizing Large Language Models to Update Systematic Literature Reviews: A Case Study on Time Pressure and Well-Being in Software Engineering Guo (Guo, 2024) Shepperd Published version available after search Table 9. Search strings for Search 1 General Search TITLE-ABS-KEY ( ( ”systematic review*” OR ”systematic literature review*” OR ”mapping stud*” OR ”mapping review*” OR ”systematic map*” OR ”literature review*” OR ”scoping review*” OR ”scoping stud*” OR ”rapid review*” ) AND ( ”language large model*” OR llm* OR ”generative AI” ) ) AND PUBYEAR >2022 AND PUBYEAR <2026 Specific searches TITLE-ABS-KEY ( ( ”systematic review*” OR ”systematic literature review*” OR ”mapping stud*” OR ”mapping review*” OR ”systematic map*” OR ”literature review*” OR ”scoping review*” OR ”scoping stud*” OR ”rapid review*” ) AND ( #SPECIFIC FRONTEND SYSTEMS#) ) AND PUBYEAR >2022 AND PUBYEAR <2026 Specific Frontend Systems related to LLMs and generative IA (separated by semicolon). chatgpt OR o3-mini; Scopus AI; Perplexity; Research Kick; Scite; PaperQA2; Undermind; Elicit; DeepSeek; Llama; Claude; Olmo Table 10. Search string for Search 2 ( TITLE ( ( ”systematic review*” OR ”systematic literature review*” OR ”mapping stud*” OR ”mapping review*” OR ”systematic map*” OR ”literature review*” OR ”scoping review*” OR ”scoping stud*” OR ”rapid review*” OR ”screening” OR ”eligibility assessment” OR ”data extraction” OR ”data collection” OR ”quality assessment” OR ”risk of bias” OR ”peer evaluation” OR ”peer review” OR ”qualitative synthesis” OR ”meta-synthesis” OR ”meta synthesis” OR ”thematic analysis” OR ”qualitative analysis” OR ”qualitative data analysis” OR ”strength of evidence” OR ”certainty of evidence” ) AND (”guideline*” OR ”guidance” OR ”recommendation*” ) AND ( ”generative artificial intelligence” OR ”generative AI” OR ”large language model*” OR llm* OR ”GPT*” OR o1 OR ”o1-mini” OR ”o3-mini” OR claude OR grok OR llama OR gemini OR bard OR titan OR nova OR phi OR orca OR mai OR qwen OR deepseek OR chatgpt OR ”Perplexity” OR ”Scopus AI” OR claude OR ”Research Kick” OR scite OR paperqa2 OR undermind OR elicit ) ) OR AUTHKEY ( ( ”systematic review*” OR ”systematic literature review*” OR ”mapping stud*” OR ”mapping review*” OR ”systematic map*” OR ”literature review*” OR ”scoping review*” OR ”scoping stud*” OR ”rapid review*” OR ”screening” OR ”eligibility assessment” OR ”data extraction” OR ”data collection” OR ”quality assessment” OR ”risk of bias” OR ”peer evaluation” OR ”peer review” OR ”qualitative synthesis” OR ”meta-synthesis” OR ”meta synthesis” OR ”thematic analysis” OR ”qualitative analysis” OR ”qualitative data analysis” OR ”strength of evidence” OR ”certainty of evidence” ) AND ( ”guideline*” OR ”guidance” OR ”recommendation*” ) AND ( ”generative artificial intelligence” OR ”generative AI” OR ”large language model*” OR llm* OR ”GPT*” OR o1 OR ”o1-mini” OR ”o3-mini” OR claude OR grok OR llama OR gemini OR bard OR titan OR nova OR phi OR orca OR mai OR qwen OR deepseek OR chatgpt OR ”Perplexity” OR ”Scopus AI” OR claude OR ”Research Kick” OR scite OR paperqa2 OR undermind OR elicit ) ) ) AND PUBYEAR >2022 AND PUBYEAR <2026 A.5. Final Primary Study selection Search 1 delivered 13 articles that used the terms guidelines or guidance in their titles. Search 2 identified 20 articles that used the terms guidelines or guidance or recommendations. The article abstracts, title and keywords were then assessed by Kitchenham using the inclusion and exclusion criteria reported in A.3. However, particular attention was paid to inclusion criterion I3 and exclusion criterion E1 because neither of these criteria were addressed by the search strings. Search 1 and Search 2 were processed separately. Based on their abstracts, Kitchenham identified three relevant papers from Search 1 and five from Search 2, which are shown in Table 11. The most common reason for exclusion was due to criterion E1 i.e., the paper presented recommendations for practice based on using GenAI tools to summarize SLR results. Subsequently, two of the five papers found by Search 2 (Chen, 2024; Luo et al., 2024) were excluded because the full text was not in English (i.e., they failed inclusion criteria I3). Search 2 found another paper of interest, i.e., Woelfle et al., (Woelfle et al., 2024). Although not a paper aimed at producing recommendations, it reported a rigorous large-scale study of the evaluation of using GenAI tools to support qualitative assessment of systematic literature reviews. In addition, one of its co-authors was John Ioannidis who is recognized as one of the world’s leading systematic review and meta-analysis researchers. This paper seemed potentially very useful for identifying good practices for evaluating GenAI performance on SLR tasks requiring assessment of primary study methodology. Table 11. Guidelines Papers included in the Rapid Review based on Abstract and Title Authors Title Reference Found By Final Decision Inam et al A review of top cardiology and cardiovascular medicine journal guidelines regarding the use of generative artificial intelligence tools in scientific writing (Inam et al., 2024) Search 1 Include Kim et al. ChatGPT and large language model (LLM) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicine (Kim et al., 2023) Search1 Include Lubowitz Guidelines for the Use of Generative Artificial Intelligence Tools for Biomedical Journal Authors and Reviewers (Lubowitz, 2024) Search 1 Include Veiga Ethical guidelines for the use of generative artificial intelligence and artificial intelligence-assisted tools in scholarly publishing: a thematic analysis (da Veiga, 2025) Search 2 Include Uribe et al. Integrating Generative AI in Dental Education: A Scoping Review of Current Practices and Recommendations (Uribe et al., 2025) Search 2 Include Perkins & Roe Academic publisher guidelines on AI usage: A ChatGPT supported thematic analysis (Perkins and Roe, 2024a) Search 2 Include Chen Utilizing ChatGPT in Systematic Reviews and Meta-Analyses (Chen, 2024) Search 2 Exclude (failed I3) Luo The application of large language models in the field of evidence-based medicine (Luo et al., 2024) Search 2 Exclude (failed I3) The two searches were processed separately. Investigating the two sets of results we found that: • None of the papers would have been found by a search restricted to the computing domain. • None of the six papers referenced any of the other five papers. • Superficial analysis suggested that none of the papers found by Search 1 was found by Search 2 and vice versa. However, Search 2 found all three papers found by Search 1, but they were not identified when the search results were restricted to guideline papers. Search 1 found one of the papers identified by Search 2 (i.e., (Uribe et al., 2025)). However, it was not included in the papers selected from Search 1 because it used the term ”recommendations” but not ”guidelines” or ”guidance” in its title. A.6. Data Extraction and Synthesis Our raw data comprised the recommendations obtained from each paper. Santos extracted the guidelines specified by six papers, i.e., each of the three papers found by Search 2, the two papers that we were aware of when we started our study (i.e.,the preliminary version of (Thomas et al., 2026a) and (Wagner et al., 2025)) and an initial version of a paper that Madeyski, Kitchenham and Shepperd authored discussing evaluation of LLMs in the context of literature screening process. Kitchenham extracted the guidelines specified by the three papers identified by Search 1. Data synthesis was intended to integrate the recommendations from different papers using thematic analysis. In practice, this was an iterative process in which we tried various methods of organizing a thematic framework. The first synthesis was undertaken by Santos based on the raw data he extracted from the six articles. He extracted 41 recommendations from the articles and then identified 7 higher-level themes. He mapped the 41 recommendations to these themes and finally identified what the high-level themes meant in the context of SLRs. This approach had a number of issues: • The level of abstraction was appropriate for guidelines related to the use of GenAI tools to support academic research. • Thomas et al.’s recommendations relating to the use of GenAI tools to support SLR conduct arose from different usage scenarios, which had overlapping recommendations, and it was unclear which specific recommendations we should include. • Wagner et al.’s recommendation related to general tool evaluation studies, and although useful, the recommendations were not focused on support for SLR conduct. • Madeyski et al.’s recommendations were aimed at assessing the performance of GenAI tools in assessing the eligibility of candidate primary studies. They covered details related to that task, and could not be easily incorporated with recommendations from studies reporting issues related to other SLR tasks. Nonetheless, it seemed apparent that recommendations about specific SLR tasks were of potential value. The second synthesis was undertaken by Kitchenham. Initially, it was intended to analyse the papers found by Search 1 to validate the high-level recommendations extracted from the synthesis of papers by Search 2. The second synthesis did not find any issues relevant to SLRs not already found by Santos. However, it was unable to integrate recommendations from sources which were at different levels of abstraction, and gave us no confidence that the current recommendations had covered all the major GenAI tool evaluation issues. In order to address the problems found during the second synthesis, we decided to produce three different types of guidelines using Thomas et al.’s approach of defining guidelines for different research roles: (1) To support all researchers using AI tools for academic research, Kitchenham integrated the recommendations from the six relevant papers to produce Table 2. For reasons of completeness, we included an additional theme (i.e., G6) which was related to peer review. This issue was not relevant to guidelines intended for use in the context of SLR conduct, but is relevant to academic research. (2) To support researchers intending to assess how well GenAI tools can support SLR tasks, we decided to undertake a thought experiment working through the processes that need to be taken to plan, conduct and report evaluations of tools to support SLRs. (3) To support researchers intending to use GenAI tools to support the conduct of a specific SLR, we decided to undertake a thought experiment working through the processes used to plan, conduct and report an SLR, and assessing how the use of GenAI tools would affect those processes. A.7. Limitations A major limitation of our search was that it was limited to a single source (i.e., SCOPUS) and hence there is a high probability that it will have missed some relevant papers. However, SCOPUS covers an extremely large range of journals and conferences and, in particular, indexes all the major computing and software journals and conferences. Our searches also recovered all of a pre-specified set of known relevant papers (Table 7), which gives direct evidence that the single-source search was sufficient for our goal of theme coverage. Another limitation arises because the selection of relevant papers from the candidate primary studies was done by a single researcher, which might also have led to incorrectly rejecting some relevant papers found by the searches. The possibility of missing papers during search and selection means that we may have failed to identify important issues concerning the use of LLMs to support SLRs. This was the goal of our R, and we found only two papers that directly addressed that goal (i.e., (Chen, 2024) and (Luo et al., 2024)). These papers were omitted from our study because they were not available in English. However, the title and abstracts of those two papers suggested that our search should have found other similar papers if any such papers were indexed by SCOPUS, and it was probable that we would have found them if they existed. In terms of identifying guidelines for using GenAI tools to support academic research, our R seems more successful. The various themes reported in Table 2 were repeated by many of the individual papers, and the individual papers did not cite each other. In addition, with one caveat, our set of guidelines matches recommendations reported by Nahar et al.’s (Nahar et al., 2025) who analysed the recommendations currently being used by the organizers of computer conferences. Thus, it is likely that we have achieved completeness with respect to the most important issues. A further limitation is the search horizon: the searches were completed on 29/03/2025 and were not re-run before submission. The R’s goal was theme coverage to inform the development of our recommendations, not an exhaustive or continuously updated survey. That development is complete, and the recommendations are grounded in the failure-mode synthesis (Section 4.3) and our companion studies (Madeyski et al., 2026; Pizard et al., 2026), so re-running the searches would not, by itself, change them. We therefore addressed post-search developments narratively instead: guidance published after the searches — Baltes et al. (Baltes et al., 2025), Farotimi et al. (Farotimi et al., 2025), and Nahar et al. (Nahar et al., 2025) — is incorporated and positioned in Section 2 and Section 3. We also verified that successive versions of the RAISE recommendations left unchanged the list we had used (Thomas et al., 2026a). Appendix B Details of the Thought Experiment Results This section reports the detailed outcomes of the thought experiments related to evaluators of GenAI tools of SLRs and reviewers using GenAI tools to support their review. B.1. Evaluating GenAI tools This section addresses the processes needed to plan, conduct and report GenAI tool evaluations. B.1.1. Planning GenAI tool Evaluations : Evaluators need to specify their goals and hypotheses clearly. In particular, they should clarify whether the GenAI tools being evaluated are intended to replace human researchers (i.e., to fully automate a specific process) or to support human researchers (i.e., act as an assistant, replacing one of the human researchers). If investigating full automation, evaluators should discuss how their proposals can support current ethical requirements for human oversight. Another critical planning issue is identifying the data needed to address their research questions and hypotheses. There are two basic evaluation approaches: (1) Retrospective analysis. This involves identifying and extracting appropriate data from previously completed SLRs. Keeping in mind the issue of data leakage if data sets are reused (Woelfle et al., 2024), evaluators planning to use data extracted from previously published SLRs must ensure that the data they require has not already been widely used in previous evaluation studies. (2) Prospective analysis. This involves identifying and extracting appropriate data during the conduct of an SLR. Evaluators can use GenAI tools to perform tasks in parallel with human researchers conducting an SLR. In addition, evaluators can also undertake some of the SLR tasks themselves to better assess the performance levels achieved by human researchers. From a scientific viewpoint, a prospective analysis is preferable because it avoids the possibility that evaluators design their research questions to suit the available data. In addition, prospective analysis provides some protection against the risk of data leakage. We note that SLRs, themselves, are retrospective studies and that the complexity of the SLR process arises from the need to reduce the risk of bias inherent in such studies. The risk of data leakage is a particular problem for evaluating GenAI tool performance. This can occur if retrospective datasets are used, particularly well-known curated datasets that have been used to evaluate other non-generative SLR tools. However, it can also occur if evaluators test and retest multiple LLMs using multiple prompt strategies on the same datasets or dataset partitions without putting in place procedures to prevent models from learning or retaining the data from previous evaluation sessions. Many issues are similar for researchers evaluating GenAI tool performance in the context of SLRs, and also for researchers training and assessing GenAI tools for use on a specific SLR. However, from the perspective of evaluators, the planning process should: • Identify an open GenAI tool as a baseline and explain the choice of any other GenAI tools being evaluated (Thomas et al., 2026a; Wagner et al., 2025; Woelfle et al., 2024). An open tool can be pinned to a specific, archived version that other researchers can rerun, which supports reproducibility. Proprietary, hosted tools also change, but it is often hard to tell exactly what has changed, an earlier version usually cannot be re-accessed, and the tool may become unavailable or prohibitively expensive. • Produce a data-management plan for: – Collecting and storing all performance evaluation data and applying any necessary quality control processes. – Storing prompts, outputs, and model versions (Thomas et al., 2026a; Wagner et al., 2025; Woelfle et al., 2024). • Put in place procedures needed to ensure data privacy if GenAI tool use requires uploading PDFs of published papers, sensitive data, or patent-level information without appropriate permissions and safeguards. • Specify the basic prompt strategy, which must include: – Ensuring that the tool is prompted to deliver the information needed to investigate the nature of disagreements between tools and humans — for example, its answer to each eligibility criterion or evaluation question, and the supporting text extracts or page references on which its decision rests. – Ensuring that final prompts are duplicated to assess GenAI tool reliability (Woelfle et al., 2024). – Identifying the process to be used to measure the power and environmental costs of GenAI use. For hosted tools the provider controls the infrastructure, so precise energy use is hard to measure. In practice, we recommend reporting feasible proxies such as the number of API calls or tokens and the monetary cost. • If required given the study goals and hypotheses, compare GenAI tool results with a baseline derived from two or more human researchers and report agreement metrics (Thomas et al., 2026a; Wagner et al., 2025; Woelfle et al., 2024). • Include an evaluation of environmental and power costs in any assessment of GenAI tool efficiency. In addition, evaluators need to consider the impact of obtaining data from multiple SLRs, in particular: • The data management plan must address the collection and collation of data from multiple SLRs. • The data analysis plan must consider how to aggregate the results from several different SLRs. Also, for prospective evaluation studies, evaluators must specify the procedures that will be followed when collaborating with the SLR review teams. Evaluators must take into account both the nature of specific SLR tasks and the limitations of GenAI tools when planning their experimental process and performance measures (Thomas et al., 2026a; Woelfle et al., 2024; Madeyski et al., 2026). Individual SLR processes and their requirement are discussed in Appendix C. B.1.2. Conducting GenAI tool Evaluations Evaluators should follow the processes and procedures defined during planning and keep a record of any deviations from the planned experimental procedures. Evaluation of GenAI tools does not change this. However, if needing to obtain fair comparisons between human researchers and GenAI tools, human researchers would need to be rigorous with respect to both time keeping for any SLR tasks they conduct, and to ensuring that they also record any additional information that evaluators require to assess the validity of the GenAI tool. B.1.3. Reporting GenAI tool Evaluations As is usual for empirical studies, evaluators must report their experimental goals and design and the conduct of their study, including any deviations from their planned process. This section identifies any additional requirements that apply to the evaluation of GenAI tools. Evaluators should report: (1) The rationale for evaluating the specific GenAI tools (Woelfle et al., 2024). (2) A description of the SLR(s) that provided the evaluation data. (3) If data were obtained from multiple SLRs, the contribution of the different SLRs to generalizability should be discussed. (4) The role of any human participants contributing to the assessment of human performance should be described (Woelfle et al., 2024). (5) The prompt engineering process, including prompt templates and the methods used to validate prompt outcomes. This information may be included in the final report or in accessible supplementary materials (Thomas et al., 2026a; Wagner et al., 2025; Woelfle et al., 2024). (6) Model versioning and configuration details, including all parameter values (Thomas et al., 2026a; Wagner et al., 2025; Woelfle et al., 2024). (7) Any metrics used to assess tool performance (Wagner et al., 2025; Woelfle et al., 2024). (8) GenAI tool intra-tool reliability, which can be assessed by running final prompts several times (Woelfle et al., 2024). (9) The results of the evaluation in terms of the effectiveness (validity) and efficiency (cost-benefit) (Madeyski et al., 2026). (10) Any concerns regarding possible GenAI tool bias (Thomas et al., 2026a). (11) Any financial or other personal interests in the evaluation results and, if necessary, how any risk of evaluator bias due to reported interests was mitigated (Thomas et al., 2026a). In addition, evaluators should discuss any limitations or constraints associated with their study, for example: • Are the results topic or discipline dependent? For example, GenAI performance in the medical domain may depend on the well-defined, unambiguous terminology used in many medical research papers. • Given the current requirements for human oversight in academic research, are the results compatible with this requirement? If they are, are there any process issues users of research need to be aware of? These recommendations assume that the evaluators have not used GenAI tools to support their evaluation process. However, if the evaluators have used GenAI tools not just as the objects of the evaluation but also as tools to assist the evaluation (e.g., for data collection and collation, or report writing), they should ensure that their report adheres to the general reporting requirements for GenAI tool use described in the recommendations for reviewers and Table 2. For example, Spillias et al. (Spillias et al., 2024), who undertook a case study to evaluate GenAi tool performance on literature screening, included an explicit declaration of their use of generative AI and AI-assisted technologies in the writing process. B.2. Using GenAI tools In this section, we identify the extensions to the SLR process and reporting guidelines needed to address the issues raised by using GenAI tools. We consider the three main phases of the SLR process (Planning, Conduct, and Reporting) together with a new phase we refer to as Tool Testing and Validation. This new phase takes place in parallel with the planning and conducting phases. B.2.1. Tool Testing and Validation There are several different scenarios to consider: • Reviewers may be concerned mainly with the conduct of a specific SLR. In such circumstances, they are likely only to be interested in using tools that reduce effort or that can be used to improve SLR validity. However, if review teams have only limited expertise in GenAI tools, they may need to include a GenAI researcher in their research team. • Reviewers may be collaborating with evaluators to conduct a case study of GenAI tool performance. In this case, the nature of the collaboration needs to be defined. At one extreme, the collaboration can be fully independent, where the reviewers conduct their SLR without the use of any GenAI tools, but make all their process results available to a separate evaluation team that conducts the same processes using GenAI tools. A problem with this approach is that by using a GenAI tool an independent evaluation team might identify a flaw in the work of the reviewers. It would violate independence if the evaluators alerted the reviewers to the flaw, but it would endanger the validity of the SLR if they did not and would damage the prospects for further collaboration. Alternatively, the evaluators may be part of the SLR team. This would mean that it would not be possible to investigate automation of the full SLR process, because the requirements of the SLR would mean that human researchers maintained oversight of each process and GenAI tool results could not be included in the SLR without human validation. Regardless of how the testing and validation are organized, it is critical that all stakeholders understand and accept the goals of the activity, are fully aware of how it will be managed and know what their personal roles and responsibilities will be. The first testing and validation task is to identify all SLR processes to be GenAI-assisted and to identify appropriate candidate tools. Candidate tools can be identified based on previous tool evaluation studies, which need to be critically evaluated in order to assess tool capability, risks and limitations (Thomas et al., 2026a). All reviewers required to use GenAI tools should be given any necessary training. During SLR planning, researchers might consider using GenAI tools to support: (1) Research question development. (2) Boolean query development. (3) Specification of eligibility criteria. In the case of query development, Wang et al. (Wang et al., 2023) conducted an experiment investigating whether ChatGPT could write an effective Boolean query. They reported that improvements in precision sometimes came with decreased recall, although this could be mitigated by better MeSH terms (MeSH is a standardized, controlled vocabulary thesaurus created by the U.S. National Library of Medicine). A useful element of their study is that they provide a good description of how they developed their prompts to support unguided and guided query refinement. In particular, the guided process was based on a current automated query formulation method. However, in our opinion, their study had several serious threats to validity: • They evaluated results with respect to precision, F1, and recall, without considering the different costs of false negatives and false positives. • Although they reported F1 values, they did not discuss the fact that all their tests report low F1 values (the highest F1 values were close to 0.6, and most were less than 0.5) indicating a high proportion of false positives and false negatives. • They used a well-known public data source and refined their prompts by multiple iterations of the same task without considering the possibility of data leakage. • Dependence on MeSH means their results do not generalize to other topic areas. Assessing the use of GenAI tools for any of these three planning tasks would not necessarily require a separate evaluation process. It could be restricted to using the tool and checking the outcomes against the opinions of the SLR research team. The procedures adopted for prompting the tool and checking the outcomes should be reported. In particular, the prompt inputs, tool output, and the final version of the output agreed by the research team should be reported in the protocol and the final SLR report or supplementary materials. Using GenAI tools to support subsequent SLR processes is more complex because subsequent processes are extremely varied. For processes that involve performing the same task on a set of different studies, such as screening candidate primary studies based on the inclusion and exclusion criteria (i.e., eligibility criteria) based either on abstracts, or full text, and extracting information about the type of a primary study in the context of mapping study data extraction, reviewers could undertake a GenAI tool evaluation to assess whether using a tool (or tools) could allow tasks usually requiring two human researchers to be performed by one human researcher and one (or more) GenAI tools. For these processes, the basic test and validation process would be the same as an independent sample-based prospective evaluation which would involve: (1) Selecting a random subset of individual items (e.g., 5% to 20% of the required inputs). (2) Assigning the items to at least two human researchers. We need to have the validation sample performance results from two human researchers because this represents current best practice for SLRs. Disagreements between the human researchers should be discussed and resolved to produce an agreed decision for each item. (3) Assigning the same items to one or more GenAI tools. (4) Specifying appropriate agreement metrics to measure GenAI performance. (5) Identifying the decision criteria that will be used to assess GenAI tool performance, if decisions about GenAI tool adoption are required. (6) Measuring performance data for processing the items to assess effectiveness and effort, elapsed time and all related costs to assess efficiency (for GenAI tools, this includes human time and effort for prompt engineering and all other costs related to GenAI tool use). (7) Assessing the reliability of GenAI tools, for example, running the agreed prompts at least twice (Woelfle et al., 2024).121212When a large number of items are being assessed (e.g., the candidate primary studies in a screening validation sample), each item contributes a pair of repeated prompt executions, so two repetitions of each prompt are sufficient to assess reliability. If only a few items are involved, researchers might need to consider more repetitions. (8) Inconsistent results should be identified as “Requiring further assessment” — they should not be excluded from performance evaluation. (9) Analysing agreement metrics to calculate: • Agreement between two human researchers. • Agreement between each human researcher and the GenAI tools. • Agreement between the GenAI tools and the agreed human researcher decision. • Sample-based confidence intervals for GenAI performance metrics and GenAI tool reliability. (10) Deciding whether, in the context of the SLR goals, the performance of the tool(s), in terms of both cost saving and threats to validity, is good enough to allow the remainder of the tasks to be performed by one human researcher and the tool(s). (11) If the reviewers intend to adopt the use of the tools for their current SLR, the test process and results must be included in the SLR report or supplementary materials. Additionally, it is important to ensure that the evaluation appropriately identifies the full costs of using a GenAI tool, or tools, to replace a human researcher. In particular, • What changes, if any, would need to be made to the standard human researcher-based SLR process? • How would disagreements between the tool(s) and the human researcher be resolved? • How would full oversight of GenAI tool output be ensured? Appendix C discusses more details related to eligibility screening and data extraction together with process-oriented discussions of the more complex SLRS tasks including qualitative synthesis, risk of bias assessment and strength of evidence assessment. We agree with Thomas et al.(Thomas et al., 2026c), that current GenAI tools cannot be relied on to fully automate all complex SLR tasks. Nonetheless, in Appendix C we discuss the possibility that evaluators could collaborate with reviewers to assess GenAI performance on all or part of these processes. B.2.2. SLR Planning SLR planning procedures need to be extended to cover the use of GenAI tools. Specifically, the protocol should include the following information for any GenAI-assisted SLR task, including: (1) The basic prompt strategy, which should include ensuring that the tool is prompted to produce additional information to support assessment of the validity of its outputs (Thomas et al., 2026a; Wagner et al., 2025; Woelfle et al., 2024). (2) A data-management plan for (Thomas et al., 2026a): (a) Storing prompts, outputs, and model versions. (b) Labelling, collecting and storing all performance human and tool performance measures. (c) Any quality assurance processes applied to collected data. (3) The procedures needed to ensure data privacy if GenAI tool use requires uploading PDFs of published papers, sensitive data, or patent-level information without appropriate permissions and safeguards. (4) Specifying how any GenAI tool testing and validation will be integrated with SLR conduct. (5) The processes used to monitor tool output for errors and bias. Process methods must include: • Running the finally agreed prompt twice and checking for consistency (Woelfle et al., 2024). • Comparing the GenAI tool output with that of at least two human researchers (Thomas et al., 2026a; Wagner et al., 2025; Woelfle et al., 2024). • For any SLR process that involves running multiple equivalent tasks, reporting agreement matrices for tasks (excluding tasks that were part of the GenAI tool test) (Madeyski et al., 2026). • Checking outputs for possible GenAI bias, such as ignoring or downgrading studies with female authors or authors with non-European names, or that express minority opinions or uncommon results (Thomas et al., 2026a; Pizard et al., 2026). (6) Compiling an environmental and computing cost statement for use of the GenAI tool. (7) Defining the process to be adopted if GenAI errors or biases are detected, which would usually refer the task back to another human researcher (Madeyski et al., 2026; Woelfle et al., 2024; Pizard et al., 2026). (8) Contingency plans to deal with inadequate GenAI tool performance, which could be: • Referring the task(s) back to human reviewers not previously involved in the specific task(s). • Revising the prompt strategy. B.2.3. SLR Conduct SLR guidelines have always specified that the conduct of an SLR should follow the processes and procedures defined in the protocol and report any deviations from the protocol. The use of GenAI tools does not change this. However, human participants in GenAI tool training and validation may need to be more rigorous with respect to: (1) Time keeping. Usually, members of the SLR review team are not required to accurately report the time they spend on particular SLR tasks. However, for validation purposes, human researchers should keep accurate records, including an allowance for break periods, while performing tasks which are part of GenAI tool validation, without which they are likely to become tired and error-prone, but excluding time lost due to external interruptions. (2) Providing any extra information needed to support comparisons between AI tool results and results obtained by human researchers. For example, if the GenAI tool is required to provide information to support or explain its response, human researchers will need to do the same, even if such details would not normally be recorded. B.2.4. SLR Reporting Current SLR standards (PRISMA-2020 (Page et al., 2021) and SEGRESS (Kitchenham et al., 2023)) already require full reporting of the use of GenAI tools, including training and validation details (Thomas et al., 2026a). However, certain issues need to be emphasized: (1) Explain and justify the use of GenAI tools (Thomas et al., 2026a; Woelfle et al., 2024). (2) Ensure humans review of all GenAI-assisted results to reduce the risk of fabricated conclusions, data, graphics, quotations, or references (Pizard et al., 2026). (3) Report the prompt engineering process, including prompt templates, and the methods used to validate prompts, either in the final report or in accessible supplementary materials (Thomas et al., 2026a; Wagner et al., 2025; Woelfle et al., 2024; Pizard et al., 2026). (4) Report model versioning and configuration details, including all parameter values in the methods section (Thomas et al., 2025; Wagner et al., 2025; Woelfle et al., 2024; Pizard et al., 2026). (5) Report any concerns regarding possible GenAI tool bias (Thomas et al., 2026a; Pizard et al., 2026). (6) Report the full environmental impact and computing-cost statement, including both training and use of the GenAI tool(s). (7) Adhere to legal and ethical issues relating to the use of GenAI tools reported in Table 2, and in addition: • Do not upload copyrighted or sensitive data to commercial APIs. • Ensure that GenAI-assisted results do not breach plagiarism regulations. • Report any financial interests or other special interests associated with the adoption of GenAI tools. B.2.5. Summary of Process Changes Figure 2 summarizes the general process changes required for using GenAI tools to conduct an SLR. It does not consider the detailed process change discussed in Appendix C. We only report the extensions to the general process, not the savings from GenAI tool use, since the human researchers will still need to perform a part of all the SLR tasks manually, even with the support of GenAI tools. The process is iterative and incremental rather than strictly linear. Planning fixes the acceptable performance levels and the initial prompts. Tool testing and validation then check the tool against those levels. If the tool falls short, the team refines the prompts or revises the plan — for example, by invoking the contingency plans agreed during planning — and validates again before the tool is used during conduct. Issues noticed during conduct, such as unexpected outputs or borderline cases, may likewise send the team back to refine prompts or re-validate. Figure 2 shows the main extensions. In practice, the steps it lists are revisited as needed. Without GenAI tool supportWith GenAI tool supportA. Tool EvaluationB. PlanningC. ConductD. Reporting Test & Evaluate GenAI Tools HumanassessmentsRandom sampleof papersDocumented process& reports of results Plans for searching, incl/excl rules, conflict resolution strategies, data extraction. ++ Prompt strategy, data management, data privacy, integration of tools, monitoring of tools. Record outcomes of each step, deviations from the plan and reasons. ++ Record time, responses for all eligibility criteria, added details about synthesis. Reporting of plan and process as per PRISMA or SEGRESS. ++ Reporting of all details of use of AI tools, screening outcomes and any legal and ethical issues. re-validate onnew issuesvalidate &refine prompts Figure 2. Summary of SLR Process Extensions Appendix C Details of SLR Process and GenAI Considerations This section discusses the detailed tasks needed to conduct the following SLR processes and considerations for incorporating GenAI: (1) eligibility of candidate primary studies based on title, abstract and keyword (TAK) information, (2) eligibility based on full text assessment, (3) data extraction, (4) risk of bias (RoB) assessment, (5) synthesis of qualitative primary studies, (6) Strength of evidence (SoE) assessment. The goal of this section is to provide a framework to help develop preliminary task-specific SLR recommendations. C.1. Literature Screening In this section, we summarize the most important issues from our revision of (Madeyski et al., 2026). We consider both abstract screening and full-text screening. Human-based TAK and full-text eligibility assessment require the following process steps: (1) Defining the eligibility criteria. These should be based on the SLR research questions (which would also have driven the candidate’s primary study search process). Note, eligibility criteria may need to be revised when proceeding from TAK-based screening to full-text-based screening. (2) Trialling the eligibility criteria on a random selection of candidate primary studies, to ensure that the criteria are properly understood and used correctly by human researchers. (3) Assessing each of the candidate primary studies identified by the search process against the eligibility criteria and retaining any study that cannot be rejected from the list of candidate primary studies. For a study claiming to be a formal SLR, this process should be conducted by two human researchers working independently, and disagreements would need to be identified and resolved either by discussion or by independent arbitration conducted by a third researcher. Mapping studies, scoping studies and rapid reviews often weaken this requirement. Madeyski et al. (Madeyski et al., 2026) point out that for studies related to screening candidate primary studies, evaluators should plan for highly imbalanced data. Evaluators must also recognize that the cost of False Negatives is lost evidence, whereas the cost of False Positives is additional human effort, so False Negatives are more of a problem than False Positives in the context of SLR validity. Therefore, we recommend: (1) Use performance metrics that are reliable when classification groups are imbalanced. The Matthews Correlation Coefficient (MCC) is a good option because, like all correlation coefficients, values close to zero suggest our predictions are not much different from chance. (2) False negatives are more important than false positives, so either use WMCC to support comparisons of performance, or conduct a cost-benefit analysis based on the cost ratio of FP to FN. (3) Full confusion matrix counts are essential to support meta-analysis. These recommendations are consolidated as Scr1–Scr3, and the validation-study items below as ScrRev1–ScrRev3, in Section 4.4. For researchers conducting an SLR who need to decide whether or not to deploy a GenAI tool, deployment decisions should be based on the results of a validation study where: • Researchers will need to decide which GenAI tools to consider, which prompt strategies to use and which deployment process(es) to evaluate. • Acceptable rates of false negatives and their relative costs relative to false positives should be set in advance, based on the type of SLR and the requirements of stakeholders. • To support decision making, validation results should include estimates of performance metric variability (e.g., confidence intervals based on resampling). • If the validation results support GenAI tool use, mechanisms to maintain human oversight need to be specified. Screening full text is similar to TAK screening, however: (1) We would not expect the extreme imbalance observed in TAK screening. However, our recommendations for appropriate metrics are unchanged. (2) It may be necessary to amend/refine the eligibility criteria and/or prompt design. Cao et al. (Cao et al., 2024) changed their prompt design for full text screening. However, Khraisha (Khraisha et al., 2024) used the same prompt process for full text screening. (3) There may not be enough primary studies to allow a realistic validation study. In such circumstances, reviewers may decide to use GenAI tools in the role of an additional research assistant while still involving two human researchers in each eligibility assessment. This would provide both additional quality assurance on the screening task and the opportunity to assess GenAI tool capability. C.2. Data Extraction Data Extraction involves the following subtasks: (1) creating a data extraction form or set of forms, (2) validating data extraction forms, (3) using the data extraction forms, (4) converting extracted data into usable formats. C.2.1. Creating Data Extraction Forms involves defining what data items to extract. General issues related to data form definition include: • Defining the scope of the data collection exercise, which depends on research questions and planned synthesis method. • Addressing the need for both structured fields and free-text fields. The main difficulty in the context of SE, is that data items tend to be very heterogeneous: (1) For quantitative SLRs, data includes effect sizes, sample sizes, p-values, and confidence intervals. In addition, outside the framework of families of experiments (Basili et al., 2002), unlike medical researchers, SE researchers do not have well-defined vocabularies to describe their metrics and experimental conditions. For these reasons, it may be necessary to specify the unit and/or metric definition used in each primary study, as well as the value of each required data item. (2) For qualitative SLRs, data includes: study context, methodology descriptions, participant characteristics, and findings. (3) For mapping studies, data usually involves multi-class categorization of primary studies along characteristics such as research method, SE domain, contribution type, and venue type. For researchers conducting an SLR, GenAI could assist in form creation (e.g., suggesting relevant data items for the given research questions), but the form(s) would need to be validated by human researchers. C.2.2. Validating Data Extraction Forms This involves piloting the agreed data collection form on a subset of primary studies. The human-based validation process involves: (1) Two or more researchers independently extracting data from a few randomly selected studies. (2) Discussing any disagreements in order to refine data form definitions and instructions. (3) Iterative refinement until consistent use of the form is achieved. Researchers considering the use of GenAI tools to support a specific SLR could incorporate a GenAI tool into the data form validation process. GenAI tools could replace one human extractor or provide additional support for human data extraction, but this would require evaluation of: (1) The accuracy achieved for extracting numerical data, which requires an exact match. However, if the data item is used in a specific primary study, the gold standard “true” value exists in that primary study and applies whether the extraction is performed by human researchers or by a GenAI tool. The existence of a gold standard value offers an exception to the general approach of relying on the results obtained by two human researchers to assess GenAI tool performance131313We note that, since primary study analyses and reporting could be flawed, the gold standard value could still prove to be incorrect when judged against a re-analysis of the raw data. While human researchers might be able to recognize implausible or impossible values during data extraction, a GenAI tool targeting data extraction might not be able to make such assessments.. (2) For mapping studies, classifications of primary study type require an exact match. Nonetheless, in the short term, this is likely to be the most viable context for incorporating GenAI tool data extraction into the SLR process because classifying a primary study against a set of defined categories is similar to checking candidate primary studies against eligibility criteria. For example, Khraisha et al. (Khraisha et al., 2024) treated each extraction field as a binary detection problem and used the same performance metrics they used for data screening. (3) The accuracy of qualitative/textual data. For this, semantic similarity may be sufficient, but it is not necessarily easy to assess. It is likely to require additional effort to evaluate the equivalence between the textual results obtained by human researchers performing the data extraction task and the outputs of GenAI tools. (4) Handling of missing data when a primary study simply does not report a data item the extraction form requires, e.g., no standard deviation reported, no explicit description of participant selection method. (5) Managing hallucination risks, including the fabrication of values not present in the source paper, fabrication of textual content, misattribution of textual information (to the wrong study, the wrong RQ, the wrong experimental condition), and plausible confabulation (i.e., output that is internally consistent and sounds reasonable but does not correspond to anything in the source. Data extraction performance metrics need to address all the above issues. Madeyski et al., (Madeyski et al., 2026) point out that reliable metrics such as the Matthews correlation coefficient (MCC) can be extended to address the classification issue (i.e. multiple categories for a nominal scale variable) and multiple outcomes (i.e., missing correct, incorrect). However, there is currently little research literature addressing this SLR stage. C.2.3. Using the Data Extraction Form This means applying the data collection form and process across all primary studies (except those used for the validation exercise - unless researchers agree that further cross-checking is needed). The human data extraction process involves at least two researchers who extract independently, and disagreements are resolved. When researchers are conducting an SLR, GenAI tools can support this process or might replace a human researcher. However, to maintain human oversight, tools should be prompted to provide source quotes/page references for each extracted value to support traceability and verification. A key difference between performance metrics used for screening candidate primary studies and performance metrics used for data extraction is that data extraction errors are harder to detect downstream (e.g., a wrong, missing or spurious effect size can silently corrupt meta-analysis). This makes the issue of validating GenAI tool performance critical. In addition, there is no inherent difference between types of error, which implies researchers should rely on MCC as the main performance metric without needing to weight different sorts of errors. C.2.4. Collating Extracted Data This involves converting extracted data into usable formats, and preparation for synthesis. This includes issues such as: • standardizing units, • converting between effect size measures • handling incomplete reporting, • highlighting anomalous values for additional validation. GenAI tools could assist with unit conversions, anomaly detection and deriving missing statistics, but such computations would need to be verified because GenAI tools are known to make arithmetic errors. C.2.5. GenAI Performance Evaluation For researchers wanting to assess the performance of GenAI tools outside the context of a specific SLR, it is critical to define the specific aims of the investigation (i.e., which subtasks are being addressed) and the starting and end points of the study. For example, is the data form available or is the GenAI tool(s) expected to construct a data form? Are the GenAI tool(s) expected to provide data that is reliable enough to be used without further validation? If the tool(s) is required to replace human researchers as opposed to assisting them, how will human oversight be maintained? C.3. Risk of Bias (RoB) Assessment The goal of risk of bias assessment is to assess whether there are any weaknesses in the research methodology adopted by each of the included primary studies. Poor research methodology casts doubts on the trustworthiness of primary study results and findings which, in turn, would undermine any SLR results relying on the primary study. Although risk of bias is considered essential for quantitative and qualitative SLRs intended to provide recommendations for SE practice or academic research, it is not necessary for mapping or scoping studies. It is optional for rapid reviews. Characteristics of the risk of bias (RoB) assessment task are that: (1) RoB assessment relies on having an appropriate set of criteria for assessing the methodological weaknesses of a primary study. Such criteria are usually organized as a set of questions with explanations about how the questions should be answered and are referred to as an evaluation (or assessment) instrument (or tool). (2) RoB evaluation questions include: • was the study prospective or retrospective? • Was the method appropriate to address the research question(s)? • Were best practices for the specific method used? • Were known weaknesses of the methods properly addressed? • Were data correctly analysed/synthesized? • Were reported findings clearly linked to the analysis/synthesis results? (3) To provide reliable assessments, RoB evaluation questions are usually extended to provide more details about how to answer them. For example, for formal (laboratory) experiments, good practices include randomized assignment to treatment and methods to avoid participant and experimenter bias, weaknesses include small sample sizes, reliance on student participants and small-scale or over-simplified tasks and task materials. (4) Evaluation assessments require a means of aggregating the answers to each question into an overall assessment of the RoB associated with a primary study. Such an assessment is usually based on a subjective qualitative ordinal scale of the type: Critical RoB, High RoB, Moderate RoB, Low RoB (5) Quantitative and qualitative primary studies usually require different evaluation instruments because they ask different types of questions using different research methods. (6) Even in the case of quantitative primary studies, an evaluation instrument usually involves some subjective judgements. For this reason, RoB evaluation is usually performed by two or more human evaluators and disagreements need to be discussed and resolved. In the next sections, we discuss the four main tasks involved in RoB evaluation: (1) Creating Evaluation Instruments (2) Validating Evaluation Instruments for a specific SLR (3) Using an Evaluation Instrument (4) Converting the evaluation results into a RoB assessment C.3.1. Creating Evaluation Instruments Currently, research has concentrated on evaluating whether GenAI tools can be trained to use the RoB or the updated RoB2 instrument (for a comparison of the two instruments see (Nejadghaderi et al., 2024)). This is only applicable to quantitative experiments that conducted randomized controlled trials (referred to as RCTs), and is, therefore, of little value in the context of SE studies. However, there are major disagreements about the performance of GenAI tools even for these well-defined instruments that assess a well-defined experimental method. For example, (Šuster et al., 2024) assessed LLM performance by applying the RoB2 questionnaire to 213 papers describing randomized controlled trials. They used four LLMs (FlanT5XL, ChatGPT (gpt-3.5-turbo), Meditron-70B, and Med42) and reported that “The results fall short of expectations, with LLMs seldom surpassing trivial baselines”. In contrast, (Lai et al., 2024), assessed the RoB questionnaire on 30 papers, and using ChatGPT and Claude and concluded that the LLMs “demonstrated substantial accuracy and consistency in evaluating RCTs, suggesting their potential as supportive tools in systematic review processes”. These studies used different risk of bias questionnaires, different primary studies, different GenAI tools, and different prompting strategies. So, they illustrate why it is essential that we develop appropriate procedures for evaluating GenAI tools. Without agreement on reliable research methods, it will not be possible to evaluate the RoB of current research, which will handicap attempts to meta-analyse results and to assess the reasons for differences. Thus, a critical issue for SE researchers intending to evaluate GenAI performance on the RoB tasks is to identify an appropriate evaluation instrument. Furthermore, the SE community needs to come to some agreement about such instruments or we will be unable to aggregate results from different experiments. Studies relating to using GenAI tool to construct evaluation instruments would need to consider: (1) The scope of any generated evaluation instrument i.e., what type of empirical studies should be considered. The scope of a specific evaluation instrument would depend on the type of empirical studies expected in a specific SLR. (2) How to construct RoB evaluation questions. Evaluation questions need to be based on the known process requirements for using a specific method, any known weaknesses of the specific method, and the extent to which the study’s design has correctly conducted the required process and addressed any weaknesses. To generate evaluation instruments, it is necessary to find source material from which to generate and/or refine appropriate evaluation questions. (3) How to assess the validity of any constructed evaluation instrument. This process is difficult because: (1) The different study types used in SE include data mining studies, case studies, quasi-experiments, formal randomized experiments, opinion surveys and a variety of different qualitative methods. Furthermore, a specific SR may include numerous different study types, all of which would need to be assessed with respect to an evaluation instrument calibrated to the likely methodological weaknesses associated with each of the adopted research methods. It should also be noted that a single primary study may use several different methods. (2) SE researchers do not always correctly label their study types, nor do they always analyse their data correctly (see, for example, (Vegas et al., 2016; Wohlin, 2021; Kitchenham et al., 2020, 2019)). Thus, evaluators cannot rely on selecting an appropriate evaluation instrument based solely on the method claimed by study authors. Furthermore, reference to a specific analysis method does not guarantee that the method was applied correctly. It is also clear that publication in a reputable journal does not, by itself, guarantee low RoB. (3) Woelfle et al. (Woelfle et al., 2024) report that GenAI tool accuracy using an evaluation instrument depends on the level of subjectivity of the instrument. They assessed GenAI tool performance on three different evaluation instruments and confirmed that the GenAI tools performed best on the instrument which involved the least amount of subjective assessment, and worst on the instrument that involved extensive subjective assessments. Given that SE researchers seldom use randomized clinical trials (which are more rigorous than standard laboratory experiments), and SE methodologists have reported problems with the use of empirical methods, we do not believe that it is currently feasible to use GenAI tools to support the generation of evaluation instruments for use in a specific SLR. Nonetheless, research aimed at investigating the construction of RoB instruments would be extremely useful to improve current empirical SE practice. C.3.2. Validating an Evaluation Instrument In order to use an agreed evaluation instrument in the context of a specific SLR, it is essential both to validate the instrument and train researchers to use the instrument: (1) Initial validation requires the researchers responsible for constructing the instrument (or tailoring an existing instrument to the types of primary studies included in a specific SLR) to apply the instrument independently to a few randomly selected primary studies and compare their results. Disagreements need to be discussed with a view to determining whether the instrument needs further refinement. (2) Subsequently, if other researchers need to use the evaluation instrument, they should also apply the instrument to the selected primary studies. Then, the research group need to discuss any remaining disagreements in terms of possible changes to the instrument, and/or gaining a better common understanding of how to use the instrument. For researchers conducting an SLR, this task would provide an opportunity to investigate whether GenAI tools could be used to support the RoB evaluation of the remaining primary studies C.3.3. Using the Evaluation Instrument The evaluation instrument(s) must then be used to assess each primary study. There are different approaches to conducting the human researcher-based process. Issues include deciding: (1) How many researchers evaluate each primary study, assuming that at least two researchers should assess each primary study. (2) Whether each researcher initially applies the evaluation instrument independently and they discuss disagreements at a subsequent meeting, or they both read the primary study independently but work together to complete the evaluation instrument. (3) Whether the process is done in tranches, i.e., an initial set of primary studies is evaluated, followed by a discussion of any problems and issues arising from the first set of reviews. This is useful if a fairly large and diverse set of primary studies is being evaluated. (4) How the evaluation results are converted into a RoB assessment (see the following section). During the conduct of an SLR, it would be possible to incorporate a prospective study aimed at assessing the performance of GenAI tools alongside the research process conducted by humans. A prospective study would investigate (1) Whether the performance of one or more of the GenAI tools is reliable enough to replace one or both human researchers. In the case of a single GenAI tool, the tool should be applied several times to the same primary study using the same prompt. In the case of multiple GenAI tools, the results from the different tools can be compared. In either case, it is necessary to identify the rate of disagreements. (2) Whether the performance of one or more of the GenAI tools working with a single human researcher is reliable enough to replace one human researcher. In both cases, a high rate of disagreement between human researchers and GenAI tools would suggest that the value of the GenAI tools was limited. It is also worth noting that GenAI tools might be useful for highlighting text related to specific evaluation instrument questions, even if human researchers remain responsible for answering the question. C.3.4. Converting the evaluation results into a RoB assessment The final issue with the use of evaluation instruments is converting the set of evaluation criteria into a statement of risk of bias. It is important to note that RoB is not additive. If one criterion has a high risk of bias, the study as a whole will be classified as having a high risk of bias, even if other criteria have a low risk of bias. Formalizing the aggregation process for each evaluation instrument is necessary for human researchers and would also support investigations of the performance of GenAI tools. C.4. Qualitative Synthesis During the Qualitative synthesis (QS) findings are extracted from individual studies, are integrated and interpreted to build a collective understanding, identify common patterns, and generate new interpretations or hypotheses. Qualitative synthesis methods are difficult to understand, use and report because: (1) There is a wide range of methods that can be used to conduct QS (many adapted from qualitative analysis), which have different approaches and processes. (2) One important aspect of qualitative synthesis methods is the form of data transformation, which exists along a continuum. At one end are aggregative approaches, which summarize and integrate findings across studies with minimal reinterpretation. At the other end are interpretive (or configurative) approaches, which develop new conceptual understandings by reanalysing and reinterpreting evidence. (3) Another important aspect is the researchers’ epistemological stance. This can be described as a spectrum ranging from an idealist position, in which reality is constructed through multiple, alternative human interpretations, to a realist position, in which reality exists independently of thought and can be directly observed and known. In general, idealist stances align with constructivist perspectives, whereas realist stances align with positivist perspectives. (4) Researcher subjectivity plays a significant role in the reliability of QS and their outcomes. It must be considered in a way that aligns with the epistemological stance. In realist approaches, such as content analysis, subjectivity is viewed as a potential source of bias that needs to be measured and controlled to ensure coding reliability. Conversely, idealist approaches recognize subjectivity as an inherent and valuable aspect of research. These approaches emphasize the importance of critically reflecting on the researcher’s positionality and influence on the study. (5) Several other concepts are important in QS. These include analytical orientation (inductive or deductive elaboration), the use of coding as an intermediate instrument, conceptual innovation (the use of concepts not present in primary studies141414It may be difficult to distinguish conceptual innovation by human researchers from plausible confabulation by GenAI tools.), data heterogeneity, and the intended use of the synthesis outcomes. C.4.1. The Qualitative Synthesis Process The QS process involves three main steps: (1) Plan the synthesis: Define the characteristics of the synthesis to be conducted and decide on the method(s) to be used, the role of the researchers, and how the process and its results will be validated. This involves considering the research questions, the intended use of the synthesis results, the types of data found in the primary studies, and the resources of the review team. In software engineering, it is quite common to use different methods for the RQs of an SR. For example, some questions that aim to characterize primary studies can be answered with categorization, while other open questions may be addressed with more interpretative approaches that allow for an inductive orientation (e.g., thematic analysis). (2) Conduct the synthesis: Carry out the synthesis of the primary studies. Generally, this involves following the plan developed earlier and executing the steps of the selected methods. However, it can also include incorporating activities based on the results obtained. For instance, if categorizing studies, new dimensions to explore may be found151515This is also common in mapping studies.. It is also important to maintain a record of the process, decisions made, and intermediate results to allow for auditing of the process later. (3) Validate the synthesis: In this stage, evaluate whether the process was conducted and the results are adequate. We recommend using criteria predefined in the planning stage. Pizard et al. (Pizard et al., 2026) propose the following list of characteristics for a sound QS, as the basis of evaluation criteria: (1) The findings should be grounded in data from the primary studies and account for the observed patterns ((Sandelowski and Barroso, 2007; Lewin et al., 2018; Flemming and Noyes, 2021)). They should provide a full and fair representation of the perspectives and understandings of the primary study authors ((Sandelowski and Barroso, 2007; Krefting, 1991)). Any contradictory evidence from the primary studies should be reported and, if possible, explained. (2) The findings should have practical relevance to the research question, and provide information to support their transferability to other contexts ((Sandelowski and Barroso, 2007; Lewin et al., 2018; Krefting, 1991)). (3) The findings should be appropriately weighted and considered for use, which may require identifying and reporting the methodological limitations of the primary studies that underpin them ((Lewin et al., 2018; Krefting, 1991)). (4) The choice of synthesis method should be appropriate and well justified ((Sandelowski and Barroso, 2007; Lewin et al., 2018; Krefting, 1991)), and should be consistent with the research question, the chosen epistemological approach, and the available primary study data. (5) The conduct of the method should be consistent with good practice, its reporting should be transparent, and there should be sufficient information to enable auditing ((Sandelowski and Barroso, 2007; Lewin et al., 2018; Krefting, 1991)). (6) The researchers should address their subjectivity appropriately and in alignment with the epistemological approach they have chosen (Krefting, 1991). In realist approaches, addressing subjectivity typically involves independent coding by multiple researchers and the calculation of agreement metrics. In contrast, more idealist (non-positivist) approaches require authors to demonstrate reflexivity about the interpretive process. This includes critically examining potential influences, such as their background, positions, and interests (especially on the research topic), as well as biases introduced by the software tools used (Weitzmann, 2003). C.4.2. Considerations for Using GenAI tools to Support Qualitative Synthesis Given the heterogeneity of methods and possible adaptations to use in the QS process and the few studies on the subject, it is difficult to consider an appropriate and safe way to use GenAI in QS. It is critical to ensure that any QS is sound using criteria such as those discussed above. The mechanisms to achieve this should be considered and defined during the planning stage. We suggest using predefined evaluation criteria, always involving humans in the process, and proactively identifying and addressing the risks associated with incorporating GenAI into the QS process. The current state of GenAI and the results reported by Pizard et al. indicate that when conducting an SLR: (1) GenAI tools could be used for realistic aggregative synthesis (e.g., classifying studies into predefined categories with little interpretation), taking on the role of an independent reviewer. For validation, similar evaluations to those proposed for study selection involving reviewer agreement could be conducted. (2) For more idealist or interpretative syntheses, many risks arise. It is important to identify and report biases introduced by the researchers or the tools used. GenAI generally presents many inherent biases that cannot be as easily studied or identified. In addition, in many idealist approaches, cross-checking between independent reviewers is not suitable as a form of validation, so incorporating GenAI as an additional researcher does not add value and could even introduce additional problems. In these cases, we agree with the views of other qualitative analysis researchers who suggest that GenAI tools can be used as a secondary analysis/synthesis method to validate the work done and to obtain new ideas and perspectives complementary to the work previously carried out by researchers (Binz et al., 2025). In either case, outcomes or findings from any GenAI tool would need to be scrutinized by human researchers before being considered for inclusion in the final SLR report. For researchers evaluating GenAI tool performance, prospective studies could act as trials for GenAI tools (even if there is no intention of incorporating the GenAI tool outcomes into the ongoing SLR). In addition, and subject to considerations of data leakage, retrospective studies could be based on existing SLRs by applying GenAI tools to the defined set of primary studies. The critical issues would be defined: (1) Appropriate criteria (and/or performance metrics) to assess GenAI tool performance. (2) A method to assess whether agreement between GenAI tools and human researchers was better than chance. (3) A procedure to assess the importance of disagreements between the GenAI tool and human researchers, particularly in the case of interpretative synthesis. The briefing notes in this section are based on research by Pizard et al. (Pizard et al., 2026). A limitation of that study is that it did not include an independent data extraction stage. Instead, the complete papers were provided to the GenAI tool as input to conduct the steps of the synthesis methods. More research is needed to evaluate data extraction and qualitative synthesis separately. C.5. Strength of Evidence Assessment Assessing the strength of evidence (also called the certainty of evidence) of individual review findings aims to assess whether the individual findings of an SLR are trustworthy. It is an important part of any quantitative or qualitative systematic literature review (Dybå and Dingsøyr, 2008). It can also be of value in the context of Rapid Reviews. However, it is not usually relevant for Mapping or Scoping reviews. The human-based assessment process involves five steps: (1) Identifying an evaluation instrument either by creating a new instrument or by selecting (and amending if necessary) an existing instrument. (2) Collating the information required from each primary study for each finding. Some information should be available from the RoB evaluation, but other information may be available in tables of data extracted during the synthesis process or only by extracting further data from each primary study. (3) Trialling the instrument and training the human evaluators on at least one finding. (4) Applying the instrument to each finding and resolving any disagreements between evaluators. (5) Agreeing and reporting the overall strength of evidence for each finding. Note, Lewin et al (2018) recommend reporting the completed strength of evidence tables for each finding. Like risk of bias assessment, the strength of evidence is usually based on a subjective assessment of a number of criteria relating to the trustworthiness of a specific finding. This is usually conducted by two or more researchers using an evaluation instrument, including criteria such as: (1) Methodological limitations: The extent to which there are concerns about the design or conduct of the primary studies that contributed evidence to an individual review finding, based on the risk of bias assessment of the relevant primary studies ((Dybå and Dingsøyr, 2008; Lewin et al., 2018; Pizard et al., 2026)). (2) Synthesis/Analysis Method: The extent to which the choice of synthesis/analysis method used to construct a finding is appropriate and justified (Pizard et al., 2026). For qualitative analysis, consider: • How the researchers addressed their own subjectivity. • How the researchers aligned their synthesis method with their own epistemological approach. (3) Synthesis/Analysis Conduct: The extent to which the conduct of the synthesis/analysis method is adequate, its reporting is transparent, and sufficient information is provided to enable auditing (i.e., reporting is based on specific SLR reporting guidelines) (Pizard et al., 2026). (4) Coherence: The extent to which the review finding is grounded in data from the relevant primary studies and accounts for the observed patterns (Lewin et al., 2018). (5) Adequacy of data: An overall determination of the degree of richness and quantity of data supporting a review finding ((Dybå and Dingsøyr, 2008; Lewin et al., 2018)). (6) Practical Relevance: The extent to which the body of evidence from the primary studies supporting a review finding is applicable to the context (perspective or population, phenomenon of interest, setting) specified in the review question ((Dybå and Dingsøyr, 2008; Lewin et al., 2018; Pizard et al., 2026)). (7) Risk of publication bias: based on whether SLR findings could be biased due to characteristics of the set of relevant primary studies ((Dybå and Dingsøyr, 2008; Pizard et al., 2026)). Examples of sources of bias include: • Only having small-scale positive studies, particularly concept validation studies, rather than rigorous evaluations. • Only having studies that agreed with one another, with no reports of any minority opinions. • Only having English language studies with male authors of European/North American ancestry. • Having studies that were mainly produced by a single research group. Individual assessments for each criterion need to be scored on a qualitative scale, such as No Serious Problem, Minor Problem, or Serious Problem. Individual assessments are then combined (after disagreements are resolved) into an overall assessment on a qualitative Strength of Evidence scale, such as: Strong (no serious problems), Medium (not more than one serious problem), Low (two or three serious problems), Very Low (four or more serious problems). Researchers conducting SLRs, who want to use GenAI tools to support their research, will need to be sure that any gains (e.g., reduced human effort or more consistent assessments) of using such a tool outweigh the potential risks. An important issue is that there are seldom a large number of findings from a single SLR, so it remains unclear whether the effort involved, both in training a GenAI tool to perform subjective assessments and in assessing whether the tool is performing adequately, outweighs any potential gains from using such a tool. However, there are two areas where GenAI tools are particularly likely to be valuable, even if the main strength of evidence assessments is conducted by two human researchers: (1) Using one or more GenAI tools to help organize the information from primary studies related to each individual finding, and extracting any additional contextual information needed to support the strength of evidence assessment. (2) Using one or more GenAI tools as additional independent researchers. Independent assessments reduce the risk of bias arising when the human researchers producing the strength of evidence assessment were also responsible for the conduct of the SLR. Researchers evaluating GenAI capabilities should start by investigating the GenAI tool’s capability to support human researchers. Such investigations can be organized as case studies in collaboration with researchers conducting SLRs. However, for researchers looking to fully automate systematic reviews, the strength of evidence is a critical task without which a systematic review is formally incomplete. Case studies undertaken in collaboration with researchers conducting SLRs would seem to be a sound approach to collecting the data needed to evaluate all aspects of fully automating strength of evidence assessment. C.6. General Issues Raised by the Briefing Notes Although addressing different SLR processes, we can see some common issues arising from the qualitative data synthesis, risk of bias analysis, and strength of evidence assessments briefing notes: (1) These tasks are more complex than the literature screening task because they rely on the understanding and experience of researchers outside the context of a specific SLR. For example, SLR researchers need knowledge of best practices for a variety of different quantitative and qualitative research methods. (2) They involve more different sub-tasks than literature screening, and those sub-tasks vary in complexity. Some of the tasks would be supported by flexible text searching tools with copy and paste facilities. Other tasks require an assessment of the extent to which a study provides reliable findings, which are hard to define at a level of detail to ensure consistent, justifiable assessments. (3) It may be difficult for human researchers to agree on what constitutes a correct assessment, making any assessment of the validity of a GenAI tool assessment problematic. (4) A particular problem with strength of evidence assessment is that it is based on issues that cannot be assessed by reviewing a specific primary study. It can only be assessed by considering the weaknesses associated with a set of primary studies. For these three processes, researchers conducting SLRs would have the option of using one or more GenAI tools as an extra set of eyes. GenAI outcomes could be incorporated into an SLR report, if they were scrutinized by two human researchers who performed the same tasks, and the results from the GenAI tool(s) were directed to identify all supporting evidence. The purpose of using GenAI tools would not be to reduce the effort to conduct the SLR, but to: (1) Increase the depth and breadth of SLR process validation and, hence, improve the trustworthiness of the SLR findings. (2) Deliver prospective case studies of GenAI performance for these processes and their individual sub-tasks. Data extraction depends very much on the goal of the SLR and the methods adopted by individual primary studies. For mapping and scoping studies, the closer the data extraction processes is to a Yes/No classification process, the more it resembles the literature screening tasks and can use the same basic prompt approaches and performance metrics used for literature screening. If the classification process is extended to cover multiple categories and multiple output values, the performance metrics recommended for use in literature screening can be adapted (Madeyski et al., 2026). However, it is more difficult to specify a process to support data extraction for qualitative data synthesis or quantitative data analysis. Again, SLR researchers have the option of using GenAI tools, not to save time and effort, but for process validation and prospective case studies of the task. CRediT statement Barbara Kitchenham: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Visualization, Project administration (initial submission). Sebastian Pizard: Conceptualization, Investigation, Writing – review & editing, Visualization. Lech Madeyski: Conceptualization, Investigation, Writing – review & editing, Visualization, Project administration (revision process). Ronnie de Souza Santos Conceptualization, Investigation, Writing – review & editing. Martin Shepperd: Conceptualization, Writing – review & editing. David Budgen: Validation, Writing – review & editing. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgements.We thank Shari Lawrence Pfleeger for reviewing our paper and recommending changes to improve the readability and clarity of our paper. References (1) Alshami et al. (2023) Ahmad Alshami, Moustafa Elsayed, Eslam Ali, Abdelrahman E. E. Eltoukhy, and Tarek Zayed. 2023. Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions. Systems 11, 7 (2023). doi:10.3390/systems11070351 Anghelescu et al. ([n. d.]) Aurelian Anghelescu, Florentina Carmen Firan, Gelu Onose, Constantin Munteanu, Andreea-Iulia Trandafir, Ilinca Ciobanu, Stefan Gheorghita, and Vlad Ciobanu. [n. d.]. PRISMA Systematic Literature Review, including with Meta-Analysis vs. Chatbot/GPT (AI) regarding Current Scientific Data on the Main Effects of the Calf Blood Deproteinized Hemoderivative Medicine (Actovegin) in Ischemic Stroke. ([n. d.]). Baltes et al. (2025) Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, et al. 2025. Guidelines for empirical studies in software engineering involving large language models. arXiv preprint arXiv:2508.15503 (2025). Banh and Strobel (2023) Leonardo Banh and Gero Strobel. 2023. Generative artificial intelligence. Electronic Markets 33, 1 (06 Dec 2023), 63. doi:10.1007/s12525-023-00680-1 Basili et al. (2002) Victor R Basili, Forrest Shull, and Filippo Lanubile. 2002. Building knowledge through families of experiments. IEEE transactions on software engineering 25, 4 (2002), 456–473. Beller et al. (2018) Elaine Beller, Justin Clark, Guy Tsafnat, Clive Adams, Heinz Diehl, Hans Lund, Mourad Ouzzani, Kristina Thayer, James Thomas, Tari Turner, Jun Xia, Karen Robinson, Paul Glasziou, Olga Ahtirschi, Robin Christensen, Julian Elliott, Sergio Graziosi, Joel Kuiper, Rasmus Moustgaard, Annette O’Connor, Jacob Riis, Karla Soares-Weiser, Kris Thayer, Camilo Vergara, Ida Wedel-Heinen, and On behalf of the founding members of the ICASR group. 2018. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Systematic Reviews 7, 1 (19 May 2018), 77. doi:10.1186/s13643-018-0740-7 Binz et al. (2025) Marcel Binz, Stephan Alaniz, Adina Roskies, Balazs Aczel, Carl T Bergstrom, Colin Allen, Daniel Schad, Dirk Wulff, Jevin D West, Qiong Zhang, et al. 2025. How should the advancement of large language models affect the practice of science? Proceedings of the National Academy of Sciences 122, 5 (2025), e2401227121. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901. Cao et al. (2024) Christian Cao, Jason Sang, Rohit Arora, Robbie Kloosterman, Matt Cecere, Jaswanth Gorla, Richard Saleh, David Chen, Ian Drennan, Bijan Teja, Michael Fehlings, Paul Ronksley, Alexander A Leung, Dany E Weisz, Harriet Ware, Mairead Whelan, David B Emerson, Rahul Arora, and Niklas Bobrovitz. 2024. Prompting is all you need: LLMs for systematic review screening. medRxiv (2024). arXiv:https://w.medrxiv.org/content/early/2024/06/03/2024.06.01.24308323.full.pdf doi:10.1101/2024.06.01.24308323 Cartaxo et al. (2018) Bruno Cartaxo, Gustavo Pinto, and Sergio Soares. 2018. The role of rapid reviews in supporting decision-making in software engineering practice. In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018. 24–34. Castillo-Segura et al. (2023) Pablo Castillo-Segura, Carlos Alario-Hoyos, Carlos Delgado Kloos, and Carmen Fernandez Panadero. 2023. Leveraging the Potential of Generative AI to Accelerate Systematic Literature Reviews: An Example in the Area of Educational Technology. In 2023 World Engineering Education Forum - Global Engineering Deans Council (WEEF-GEDC). 1–8. doi:10.1109/WEEF-GEDC59520.2023.10344098 Chen (2024) Hsiu-Min Chen. 2024. Utilizing ChatGPT in Systematic Reviews and Meta-Analyses. Hu Li Za Zhi 71, 5 (2024), 21–28. Conte et al. (1986) Samuel Daniel Conte, Hubert E Dunsmore, and YE Shen. 1986. Software engineering metrics and models. Benjamin-Cummings Publishing Co., Inc. Crawford (2024) Kate Crawford. 2024. Generative AI is guzzling water and energy. Cumpston et al. (2019) Miranda Cumpston, Tianjing Li, Matthew J Page, Jacqueline Chandler, Vivian A Welch, Julian PT Higgins, and James Thomas. 2019. Updated guidance for trusted systematic reviews: a new edition of the Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane database of systematic reviews 2019, 10 (2019), ED000142. da Veiga (2025) Adéle da Veiga. 2025. Ethical guidelines for the use of generative artificial intelligence and artificial intelligence-assisted tools in scholarly publishing: a thematic analysis. Science Editing 12, 1 (2025), 28–34. doi:10.6087/kcse.352 Dauvergne (2022) Peter Dauvergne. 2022. Is artificial intelligence greening global supply chains? Exposing the political economy of environmental costs. Review of International Political Economy 29, 3 (2022), 696–718. de Morais Leça et al. (2025) Matheus de Morais Leça, Lucas Valença, Reydne Santos, and Ronnie de Souza Santos. 2025. Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering. In Proceedings of the 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (Ottawa, Ontario, Canada) (WSESE ’25). IEEE Press, 36–43. doi:10.1109/WSESE66602.2025.00013 De Paoli (2024) Stefano De Paoli. 2024. Performing an inductive thematic analysis of semi-structured interviews with a large language model: An exploration and provocation on the limits of the approach. Social Science Computer Review 42, 4 (2024), 997–1019. Delgado-Chaves et al. (2025) Fernando M. Delgado-Chaves, Matthew J. Jennings, Antonio Atalaia, Justus Wolff, Rita Horvath, Zeinab M. Mamdouh, Jan Baumbach, and Linda Baumbach. 2025. Transforming literature screening: The emerging role of large language models in systematic reviews. Proceedings of the National Academy of Sciences 122, 2 (2025), e2411962122. doi:10.1073/pnas.2411962122 Dennstädt et al. (2024) Fabio Dennstädt, Johannes Zink, Paul Martin Putora, Janna Hastings, and Nikola Cihoric. 2024. Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Systematic Reviews 13, 1 (15 Jun 2024), 158. doi:10.1186/s13643-024-02575-4 Devane et al. (2022) Declan Devane, Nikita N Burke, Shaun Treweek, Mike Clarke, James Thomas, Andrew Booth, Andrea C Tricco, and KM Saif-Ur-Rahman. 2022. Study within a review (SWAR). Journal of Evidence-Based Medicine 15, 4 (2022), 328. Dybå and Dingsøyr (2008) Tore Dybå and Torgeir Dingsøyr. 2008. Strength of evidence in systematic reviews in software engineering. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement. 178–187. Farotimi et al. (2025) Oluwaseun Farotimi, Adam Dunn, Caspar J. Van Lissa, Joshua Richard Polanin, Dimitris Mavridis, and Terri D. Pigott. 2025. Guidance for manuscript submissions testing the use of generative AI for systematic review and meta-analysis. Research Synthesis Methods (2025). Editorial; published online 11 December 2025. Felizardo et al. (2024) Katia Romero Felizardo, Márcia Sampaio Lima, Anderson Deizepe, Tayana Uchôa Conte, and Igor Steinmacher. 2024. ChatGPT application in Systematic Literature Reviews in Software Engineering: an evaluation of its accuracy to support the selection activity. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Barcelona, Spain) (ESEM ’24). Association for Computing Machinery, New York, NY, USA, 25–36. doi:10.1145/3674805.3686666 Flemming and Noyes (2021) Kate Flemming and Jane Noyes. 2021. Qualitative evidence synthesis: where are we at? International journal of qualitative methods 20 (2021), 1609406921993276. Foss et al. (2003) Tron Foss, Erik Stensrud, Barbara Kitchenham, and Ingunn Myrtveit. 2003. A simulation study of the model evaluation criterion MMRE. IEEE Transactions on Software Engineering 29, 11 (2003), 985–995. Gallegos et al. (2025) I.O. Gallegos, R.A. Rossi, J. Barrow, M.M. Tanjim, S. Kim, F. Dernoncourt, T. Yu, R. Zhang, and N.K. Ahmed. 2025. Bias and fairness in large language models: A survey. Computational Linguistics 50, 3 (2025), 1097–1179. Gao et al. (2024) Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Perrault. 2024. CollabCoder: a lower-barrier, rigorous workflow for inductive collaborative qualitative analysis with large language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–29. Gibney (2025) Elizabeth Gibney. 2025. What are the best AI tools for research? Nature’s guide. Nature (Feb 2025). Greengard (2025) Samuel Greengard. 2025. Shining a Light on AI Hallucinations. Commun. ACM 68, 5 (April 2025), 9–11. doi:10.1145/3715691 Guo et al. (2024) Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Michael Paget, and Christopher Naugler. 2024. Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study. J Med Internet Res 26 (jan 2024), e48996. Guo (2024) Jingyi Guo. 2024. Utilizing Large Language Models to Update Systematic Literature Reviews: A Case Study on Time Pressure and Well-Being in Software Engineering. Master’s thesis. University of Helsinki, Faculty of Science, Helsinki, Finland. https://helda-test-22.hulib.helsinki.fi/server/api/core/bitstreams/d36e3a4b-c3ab-4430-b26d-9dcd956bda06/content Génova et al. (2012) Gonzalo Génova, Juan Llorens, and Jorge Morato. 2012. Software Engineering Research: The Need to Strengthen and Broaden the Classical Scientific Method. In Research Methodologies, Innovations and Philosophies in Software Systems Engineering and Information Systems. IGI Global, Hershey, PA, USA, 106–125. doi:10.4018/978-1-4666-0179-6.ch006 Hamilton et al. (2023) Leah Hamilton, Desha Elliott, Aaron Quick, Simone Smith, and Victoria Choplin. 2023. Exploring the use of AI in qualitative analysis: A comparative study of guaranteed income data. International journal of qualitative methods 22 (2023), 16094069231201504. He et al. (2025) Horace He et al. 2025. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism (2025). doi:10.64434/tml.20250910 https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/. He (2025) Jiangen He. 2025. Who Gets Cited? Gender- and Majority-Bias in LLM-Driven Reference Selection. arXiv:2508.02740 [cs.DL] https://arxiv.org/abs/2508.02740 Huang et al. (2025) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Trans. Inf. Syst. 43, 2, Article 42 (Jan. 2025), 55 pages. doi:10.1145/3703155 Huotala et al. (2025a) Aleksi Huotala, Miikka Kuutila, and Mika Mäntylä. 2025a. Research artifacts in secondary studies: A systematic mapping in software engineering. Information and Software Technology 187 (Nov. 2025), 107830. doi:10.1016/j.infsof.2025.107830 Huotala et al. (2025b) Aleksi Huotala, Miikka Kuutila, and Mika Mäntylä. 2025b. SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews. In 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). ACM, 01–12. Huotala et al. (2024) Aleksi Huotala, Miikka Kuutila, Paul Ralph, and Mika Mäntylä. 2024. The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (Salerno, Italy) (EASE ’24). Association for Computing Machinery, New York, NY, USA, 262–271. https://doi.org/10.1145/3661167.3661172 Huotala et al. (2026) Aleksi Huotala, Miikka Kuutila, Olli-Pekka Turtio, Simo Sipilä, and Mika Mäntylä. 2026. AISysRev – LLM-based Tool for Title-abstract Screening. arXiv:2510.06708 [cs.SE] https://arxiv.org/abs/2510.06708 Inam et al. (2024) Maha Inam, Sana Sheikh, Abdul Mannan Khan Minhas, Elizabeth M Vaughan, Chayakrit Krittanawong, Zainab Samad, Carl J Lavie, Adeel Khoja, Melaine D’Cruze, Leandro Slipczuk, et al. 2024. A review of top cardiology and cardiovascular medicine journal guidelines regarding the use of generative artificial intelligence tools in scientific writing. Current Problems in Cardiology 49, 3 (2024), 102387. doi:10.1016/j.cpcardiol.2024.102387 Jovanović and Campbell (2025) Mlađan Jovanović and Mark Campbell. 2025. Reasoning AI: Progress, Challenges, and the Path Forward for Large Reasoning and Concept Models. Computer 58, 11 (2025), 113–119. Kalai et al. (2025) Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. 2025. Why Language Models Hallucinate. arXiv:2509.04664 [cs.CL] https://arxiv.org/abs/2509.04664 Kamei et al. (2021) Fernando Kamei, Igor Wiese, Crescencio Lima, Ivanilton Polato, Vilmar Nepomuceno, Waldemar Ferreira, Márcio Ribeiro, Carolline Pena, Bruno Cartaxo, Gustavo Pinto, and Sérgio Soares. 2021. Grey Literature in Software Engineering: A critical review. Information and Software Technology 138 (2021). Khraisha et al. (2024) Qusai Khraisha, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. 2024. Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods 15, 4 (mar 2024), 616–626. Kim et al. (2023) Jin K Kim, Michael Chua, Mandy Rickard, and Armando Lorenzo. 2023. ChatGPT and large language model (LLM) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicine. Journal of Pediatric Urology 19, 5 (2023), 598–604. doi:10.1016/j.jpurol.2023.05.018 Kitchenham et al. (2019) Barbara Kitchenham, Lech Madeyski, and Pearl Brereton. 2019. Problems with statistical practice in human-centric software engineering experiments. In Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering. 134–143. doi:10.1145/3319008.3319009 Kitchenham et al. (2020) Barbara Kitchenham, Lech Madeyski, and Pearl Brereton. 2020. Meta-analysis for families of experiments in software engineering: a systematic review and reproducibility and validity assessment. Empirical Software Engineering 25 (2020), 353–401. doi:10.1007/s10664-019-09747-0 Kitchenham et al. (2023) Barbara Kitchenham, Lech Madeyski, and David Budgen. 2023. SEGRESS: Software engineering guidelines for reporting secondary studies. IEEE Transactions on Software Engineering 49, 3 (2023), 1273–1298. doi:10.1109/TSE.2022.3174092 Kitchenham et al. (2002) Barbara A Kitchenham, Shari Lawrence Pfleeger, Lesley M Pickard, Peter W Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on software engineering 28, 8 (2002), 721–734. Kitchenham et al. (2001) Barbara A. Kitchenham, Lesley M. Pickard, Stephen G. MacDonell, and Martin Shepperd. 2001. What accuracy statistics really measure. IEE Proceedings - Software Engineering 148, 3 (2001), 81–85. Ko (2019) Amy J. Ko. 2019. Why We Should Not Measure Productivity. Apress, Berkeley, CA, 21–26. doi:10.1007/978-1-4842-4221-6_3 Krefting (1991) Laura Krefting. 1991. Rigor in qualitative research: The assessment of trustworthiness. The American journal of occupational therapy 45, 3 (1991), 214–222. Lago et al. (2024) Patricia Lago, Per Runeson, Qunying Song, and Roberto Verdecchia. 2024. Threats to Validity in Software Engineering – hypocritical paper section or essential analysis?. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’24). Association for Computing Machinery, New York, NY, USA, 314–324. Lai et al. (2024) Honghao Lai, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang, Jiayi Liu, Jianing Liu, Ziying Ye, et al. 2024. Assessing the risk of bias in randomized clinical trials with large language models. JAMA Network Open 7, 5 (2024), e2412687–e2412687. Lewin et al. (2018) Simon Lewin, Andrew Booth, Claire Glenton, Heather Munthe-Kaas, Arash Rashidian, Megan Wainwright, Meghan A Bohren, Özge Tunçalp, Christopher J Colvin, Ruth Garside, et al. 2018. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implementation Science 13, Suppl 1 (2018), 2. Lubowitz (2024) James H Lubowitz. 2024. Guidelines for the use of generative artificial intelligence tools for biomedical journal authors and reviewers. Arthroscopy: The Journal of Arthroscopic & Related Surgery 40, 3 (2024), 651–652. doi:10.1016/j.arthro.2023.10.037 Luo et al. (2024) Xufei Luo, Han Lyu, Qianling Shi, Zijun Wang, Hui Liu, Di Zhu, Ye Wang, and Yaolong Chen. 2024. The application of large language models in the field of evidence-based medicine. Chinese Journal of Evidence-Based Medicine (2024). Madeyski et al. (2026) Lech Madeyski, Barbara Kitchenham, and Martin Shepperd. 2026. LLM4SCREENLIT: Recommendations on assessing the performance of large language models for screening literature in systematic reviews. Information and Software Technology 198 (2026), 108204. doi:10.1016/j.infsof.2026.108204 Melegati et al. (2024) Jorge Melegati, Nicolas Nascimento, Rafael Chanin, Afonso Sales, and Igor Wiese. 2024. Exploring potential implications of intelligent tools for human aspects of software engineering. In Proceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engineering (Lisbon, Portugal) (CHASE ’24). Association for Computing Machinery, New York, NY, USA, 121–132. doi:10.1145/3641822.3641877 Monperrus (2014) Martin Monperrus. 2014. A critical review of ”automatic patch generation learned from human-written patches”: essay on the problem statement and the evaluation of automatic software repair. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 234–242. doi:10.1145/2568225.2568324 Morgan (2023) David L Morgan. 2023. Exploring the use of artificial intelligence for qualitative data analysis: The case of ChatGPT. International journal of qualitative methods 22 (2023), 16094069231211248. Morris (2024) Meredith Ringel Morris. 2024. Prompting considered harmful. Commun. ACM 67, 12 (2024), 28–30. Mulrow (1987) Cynthia D Mulrow. 1987. The medical review article: state of the science. Annals of internal medicine 106, 3 (1987), 485–488. Myrtveit and Stensrud (2012) Ingunn Myrtveit and Erik Stensrud. 2012. Validity and reliability of evaluation procedures in comparative studies of effort prediction models. Empirical Software Engineering 17 (2012), 23–33. Myrtveit et al. (2005) Ingunn Myrtveit, Erik Stensrud, and Martin Shepperd. 2005. Reliability and validity in comparative studies of software prediction models. IEEE Transactions on Software Engineering 31, 5 (2005), 380–391. Nahar et al. (2025) Mahjabin Nahar, Sian Lee, Rebekah Guillen, and Dongwon Lee. 2025. Generative Artificial Intelligence Policies under the Microscope. Commun. ACM 68, 7 (2025), 29–33. Natukunda and Muchene (2023) Agnes Natukunda and Leacky K. Muchene. 2023. Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology. Systematic Reviews 12, 1 (03 Jan 2023), 1. doi:10.1186/s13643-022-02163-4 Nejadghaderi et al. (2024) Seyed Aria Nejadghaderi, Maryam Balibegloo, and Nima Rezaei. 2024. The Cochrane risk of bias assessment tool 2 (RoB 2) versus the original RoB: A perspective on the pros and cons. Health Science Reports 7, 6 (2024), e2165. Ngwenyama and Rowe (2024) Ojelanki Ngwenyama and Frantz Rowe. 2024. Should We Collaborate with AI to Conduct Literature Reviews? Changing Epistemic Values in a Flattening World. Journal of the Association for Information Systems 25 (01 2024), 122–136. doi:10.17705/1jais.00869 Page et al. (2021) Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, et al. 2021. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372 (2021). Pearson (2024) Helen Pearson. 2024. Can AI review the scientific literature - and figure out what it all means? Nature 635, 8038 (nov 2024), 276–278. Perkins and Roe (2024a) Mike Perkins and Jasper Roe. 2024a. Academic publisher guidelines on AI usage: A ChatGPT supported thematic analysis. 1398 pages. doi:10.12688/f1000research.142411.2 Perkins and Roe (2024b) Mike Perkins and Jasper Roe. 2024b. The use of Generative AI in qualitative analysis: Inductive thematic analysis with ChatGPT. Journal of Applied Learning and Teaching 7, 1 (2024), 390–395. Petersen and Gerken (2025) Kai Petersen and Jan M. Gerken. 2025. On the road to interactive LLM-based systematic mapping studies. Information and Software Technology 178 (2025), 107611. doi:10.1016/j.infsof.2024.107611 Pizard et al. (2025) Sebastián Pizard, Joaquín Lezama, Rodrigo García, Diego Vallespir, and Barbara Kitchenham. 2025. Using rapid reviews to support software engineering practice: a systematic review and a replication study. Empirical Software Engineering 30, 1 (2025), 10. Pizard et al. (2026) Sebastián Pizard, Ramiro Moreira, Federico Galiano, Ignacio Sastre, and Lorena Etcheverry. 2026. On the Use of Large Language Models for Qualitative Synthesis. In Proceedings of the 2026 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE ’26). Association for Computing Machinery, New York, NY, USA, 9–16. doi:10.1145/3786149.3788299 Ralph (2021) Paul Ralph. 2021. ACM SIGSOFT empirical standards released. ACM SIGSOFT Software Engineering Notes 46, 1 (2021), 19–19. Robinson et al. (2023) Ambrose Robinson, William Thorne, Ben P. Wu, Abdullah Pandor, Munira Essat, Mark Stevenson, and Xingyi Song. 2023. Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation. arXiv:2308.06610 [cs.CL] https://arxiv.org/abs/2308.06610 Sandelowski and Barroso (2007) M Sandelowski and J Barroso. 2007. Optimizing the validity of qualitative research synthesis studies. Sandelowski M, Barroso J. Handbook for synthesizing qualitative research. NewYork: Springer Publishing (2007), 227–230. Spillias et al. (2024) Scott Spillias, Paris Tuohy, Matthew Andreotta, Ruby Annand-Jones, Fabio Boschetti, Christopher Cvitanovic, Joseph Duggan, Elisabeth A Fulton, Denis B Karcher, Cecile Paris, et al. 2024. Human-AI collaboration to identify literature for evidence synthesis. Cell Reports Sustainability 1, 7 (2024), 1–11. Stuart et al. (2018) Michael T. Stuart, Yiftach Fehige, and James Robert Brown. 2018. Thought Experiments: State of the Art. In The Routledge Companion to Thought Experiments, Michael T. Stuart, Yiftach Fehige, and James Robert Brown (Eds.). Routledge, 1–28. Susnjak et al. (2025) Teo Susnjak, Peter Hwang, Napoleon Reyes, Andre L. C. Barczak, Timothy McIntosh, and Surangika Ranathunga. 2025. Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning. ACM Transactions on Knowledge Discovery from Data 19, 3 (March 2025), 1–39. doi:10.1145/3715964 Šuster et al. (2024) Simon Šuster, Timothy Baldwin, and Karin Verspoor. 2024. Zero-and few-shot prompting of generative large language models provides weak assessment of risk of bias in clinical trials. Research Synthesis Methods 15, 6 (2024), 988–1000. Syriani et al. (2024) Eugene Syriani, Istvan David, and Gauransh Kumar. 2024. Screening articles for systematic reviews with ChatGPT. Journal of Computer Languages 80 (2024), 101287. doi:10.1016/j.cola.2024.101287 Thomas et al. (2025) James Thomas, Ella Flemyng, Anna Noel-Storr, Will Moy, Iain Marshall, Raouf Hajji, Zoe Jordan, Edoardo Aromataris, Samer Mheissen, Justin Clark, Paweł Jemioło, Ashrita Saran, Michelle Angrish, Biljana Macura, Rene Spijker, Neal Haddaway, Wojciech Kusa, Yuan Chi, Isabel Fletcher, Jan Minx, Emma McFarlane, Claudia Kapp, Kaitlyn Hair, Marion Spring, Isabel Kempner, Lisa Askie, Birte Snilstveit, Joerg J Meerpohl, Gerald Gartlehner, Laurenz Mahlanza-Langer, Declan Devane, Rigmor C Berg, Justine Karpusheff, and Jonas Goretzko. 2025. Responsible AI in Evidence SynthEsis (RAISE) 1: guidance and recommendations Draft for Consultation and Revision. Open Science Framework (OSF) (2025). Thomas et al. (2024) James Thomas, Ella Flemyng, Anna Noel-Storr, Will Moy, Iain Marshall, Raouf Hajji, Zoe Jordan, Edoardo Aromataris, Samer Mheissen, Justin Clark, Paweł Jemioło, Ashrita Saran, Vickie R. Walker, Michelle Angrish, Biljana Macura, Rene Spijker, Neal Haddaway, Wojciech Kusa, Yuan Chi, and Isabel Fletcher. 2024. Responsible AI in Evidence SynthEsis (RAISE): guidance and recommendations Draft for Consultation and Revision. Open Science Framework (OSF) (2024). Thomas et al. (2026a) James Thomas, Kaitlyn Hair, Anna Noel-Storr, Michelle Angrish, Edoardo Aromataris, Lisa Askie, Rigmor C. Berg, Yuan Chi, A. Justin Clark, Declan Devane, Emma McFarlane, Isabel Fletcher, Gerald Gartlehner, Jonas Goretzko, Neal Haddaway, Raouf Hajji, Paweł Jemioło, Zoe Jordan, Justine Karpusheff, Isabel Kempner, Wojciech Kusa, Claudia Lenkewitz, Biljana Macura, Laurenz Mahlanza-Langer, Iain J. Marshall, Joerg J. Meerpohl, Samer Mheissen, Jan Minx, Will Moy, Andrew A. Rooney, Ashrita Saran, Birte Snilstveit, Rene Spijker, Marion Spring, Vickie R. Walker, and Ella Flemyng. 2026a. Responsible Use of AI in Evidence Synthesis (RAISE 2026) 1: Recommendations for Practice. Open Science Framework. doi:10.17605/OSF.IO/FWAUD Version 3, updated 13 March 2026. Thomas et al. (2026b) James Thomas, Kaitlyn Hair, Anna Noel-Storr, Michelle Angrish, Edoardo Aromataris, Lisa Askie, Rigmor C. Berg, Yuan Chi, A. Justin Clark, Declan Devane, Emma McFarlane, Isabel Fletcher, Gerald Gartlehner, Jonas Goretzko, Neal Haddaway, Raouf Hajji, Paweł Jemioło, Zoe Jordan, Justine Karpusheff, Isabel Kempner, Wojciech Kusa, Claudia Lenkewitz, Biljana Macura, Laurenz Mahlanza-Langer, Iain J. Marshall, Joerg J. Meerpohl, Samer Mheissen, Jan Minx, Will Moy, Andrew A. Rooney, Ashrita Saran, Birte Snilstveit, Rene Spijker, Marion Spring, Vickie R. Walker, and Ella Flemyng. 2026b. Responsible Use of AI in Evidence Synthesis (RAISE 2026) 2: Building and Evaluating AI Evidence Synthesis Tools. Open Science Framework. doi:10.17605/OSF.IO/FWAUD Version 3, updated 13 March 2026. Thomas et al. (2026c) James Thomas, Kaitlyn Hair, Anna Noel-Storr, Michelle Angrish, Edoardo Aromataris, Lisa Askie, Rigmor C. Berg, Yuan Chi, A. Justin Clark, Declan Devane, Emma McFarlane, Isabel Fletcher, Gerald Gartlehner, Jonas Goretzko, Neal Haddaway, Raouf Hajji, Paweł Jemioło, Zoe Jordan, Justine Karpusheff, Isabel Kempner, Wojciech Kusa, Claudia Lenkewitz, Biljana Macura, Laurenz Mahlanza-Langer, Iain J. Marshall, Joerg J. Meerpohl, Samer Mheissen, Jan Minx, Will Moy, Andrew A. Rooney, Ashrita Saran, Birte Snilstveit, Rene Spijker, Marion Spring, Vickie R. Walker, and Ella Flemyng. 2026c. Responsible Use of AI in Evidence Synthesis (RAISE 2026) 3: Selecting and Using AI Evidence Synthesis Tools. Open Science Framework. doi:10.17605/OSF.IO/FWAUD Version 3.1, updated 13 March 2026. Uribe et al. (2025) Sergio E. Uribe, Ilze Maldupa, and Falk Schwendicke. 2025. Integrating Generative AI in Dental Education: A Scoping Review of Current Practices and Recommendations. European Journal of Dental Education 29, 2 (2025), 341–355. doi:10.1111/eje.13074 van Dinter et al. (2021) Raymon van Dinter, Bedir Tekinerdogan, and Cagatay Catal. 2021. Automation of systematic literature reviews: A systematic literature review. Information and Software Technology 136 (2021), 106589. doi:10.1016/j.infsof.2021.106589 Vegas et al. (2016) Sira Vegas, Cecilia Apa, and Natalia Juristo. 2016. Crossover Designs in Software Engineering Experiments: Benefits and Perils. IEEE Transactions on Software Engineering 42, 2 (2016), 120–135. doi:10.1109/TSE.2015.2467378 Wachinger et al. (2024) Jonas Wachinger, Kate Bärnighausen, Louis N Schäfer, Kerry Scott, and Shannon A McMahon. 2024. Prompts, pearls, imperfections: comparing ChatGPT and a human researcher in qualitative data analysis. Qualitative Health Research (2024), 10497323241244669. Wagner et al. (2025) Stefan Wagner, Marvin Muñoz Barón, Davide Falessi, and Sebastian Baltes. 2025. Towards Evaluation Guidelines for Empirical Studies involving LLMs. arXiv:2411.07668 [cs.SE] https://arxiv.org/abs/2411.07668 Wang et al. (2023) Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023. Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Taiwan) (SIGIR ’23). Association for Computing Machinery, New York, NY, USA, 1426–1436. doi:10.1145/3539618.3591703 Wang et al. (2024) Shuai Wang, Harrisen Scells, Shengyao Zhuang, Martin Potthast, Bevan Koopman, and Guido Zuccon. 2024. Zero-Shot Generative Large Language Models for Systematic Review Screening Automation. In Advances in Information Retrieval, Nazli Goharian, Nicola Tonellotto, Yulan He, Aldo Lipani, Graham McDonald, Craig Macdonald, and Iadh Ounis (Eds.). Springer Nature Switzerland, Cham, 403–420. Weitzmann (2003) E.A. Weitzmann. 2003. Software and Qualitative Research. In Collecting and Interpreting Qualitative Materials, N. K. Denzin and Y. S. Lincoln (Eds.). Sage, 310–339. Wilkins (2023) David Wilkins. 2023. Automated title and abstract screening for scoping reviews using the GPT-4 Large Language Model. arXiv:2311.07918 [cs.CL] https://arxiv.org/abs/2311.07918 Woelfle et al. (2024) Tim Woelfle, Julian Hirt, Perrine Janiaud, Ludwig Kappos, John PA Ioannidis, and Lars G Hemkens. 2024. Benchmarking Human–AI collaboration for common evidence appraisal tools. Journal of Clinical Epidemiology 175 (2024), 111533. Wohlin (2021) Claes Wohlin. 2021. Case Study Research in Software Engineering—It is a Case, and it is a Study, but is it a Case Study? Information and Software Technology 133 (2021), 106514. Zimmermann et al. (2024) Robert Zimmermann, Marina Staab, Mehran Nasseri, and Patrick Brandtner. 2024. Leveraging Large Language Models for Literature Review Tasks - A Case Study Using ChatGPT. In Advanced Research in Technologies, Information, Innovation and Sustainability, Teresa Guarda, Filipe Portela, and Jose Maria Diaz-Nafria (Eds.). Springer Nature Switzerland, Cham, 313–323.

Instant research discovery