Paper deep dive
Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?
Aleksi Huotala, Miikka Kuutila, Mika MÀntylÀ
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/9/2026, 7:44:12 AM
Summary
This study evaluates the performance of eight new Large Language Models (LLMs) in automating title-abstract screening for Software Engineering Systematic Reviews (SRs), using the SESR-Eval and SESR-Eval-Mini datasets. Results indicate that newer LLMs offer only marginal improvements over older models (avg. MCC rose from 0.347 to 0.365), with performance variations between secondary studies being greater than between models. LLMs show high inter-model agreement (Gwet's AC1 = 0.830) but are not yet reliable enough to replace human screeners due to low recall and sensitivity to criteria clarity. Refining inclusion/exclusion criteria provided modest gains.
Entities (12)
Relation Signals (8)
Aleksi Huotala â affiliatedwith â University of Helsinki
confidence 95% · Aleksi Huotala ... Affiliation: University of Helsinki, Helsinki, Finland
Mika MĂ€ntylĂ€ â affiliatedwith â University of Helsinki
confidence 95% · Mika MÀntylÀ ... Affiliation: University of Helsinki, Helsinki, Finland
Miikka Kuutila â affiliatedwith â LUT University
confidence 95% · Miikka Kuutila ... Affiliation: LUT University, Lahti, Finland
LLMs â usedfor â Systematic Reviews
confidence 95% · Prior work has explored large language models (LLMs) for automating this step [screening in systematic reviews]
SESR-Eval â usedin â Software Engineering
confidence 95% · We used an existing software engineering SR screening benchmark (SESR-Eval) as our data.
SESR-Eval-Mini â derivedfrom â SESR-Eval
confidence 90% · We also power-sampled a new, smaller dataset (SESR-Eval-Mini) that allows evaluation at lower costs.
New LLMs â outperforms â Old LLMs
confidence 85% · The eight new LLMs performed marginally better than the seven old ones: avg. MCC across secondary studies rose from 0.347 to 0.365.
Refining criteria â improves â Recall
confidence 80% · Refining the inclusion and exclusion criteria slightly improved recall and made decisions easier for some LLMs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Screening in systematic reviews (SRs) is manual and time-consuming. Prior work has explored large language models (LLMs) for automating this step, but LLMs are evolving rapidly, so earlier performance claims may no longer accurately reflect their screening performance. We used an existing software engineering SR screening benchmark (SESR-Eval) as our data. We also power-sampled a new, smaller dataset (SESR-Eval-Mini) that allows evaluation at lower costs. Using this data, we evaluated eight new LLMs for screening performance. Additionally, we tested different prompts, analyzed LLM agreement in screening decisions and criteria, and examined the effect of refining the inclusion and exclusion criteria on screening performance. The eight new LLMs performed marginally better than the seven old ones: avg. MCC across secondary studies rose from 0.347 to 0.365. Differences between secondary studies are still bigger than between LLMs. Computing the overall screening decision from criterion-level decisions degraded screening performance only slightly. LLMs generally agree with each other in their corresponding screening decisions (mean Gwet's AC1 = 0.830), though certain inclusion and exclusion criteria showed larger disagreement than others. Refining the inclusion and exclusion criteria slightly improved recall and made decisions easier for some LLMs, but overall impacts of criteria refinement were modest. LLMs are not yet ready to replace humans in paper screening and the advantages new, more costly models bring, appear to be very limited. Agent-based approaches, prompt engineering, and further criteria refinement are three potential future research avenues.
Tags
Links
- Source: https://arxiv.org/abs/2610.10633v1
- Canonical: https://arxiv.org/abs/2610.10633v1
Trouble viewing inline? Open PDF directly â
Full Text
150,333 characters extracted from source content.
Expand or collapse full text
Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews? Aleksi Huotala Email: aleksi.huotala@helsinki.fi Affiliation: University of Helsinki, Helsinki, Finland Miikka Kuutila Email: miikka.kuutila@lut.fi Affiliation: LUT University, Lahti, Finland Mika MĂ€ntylĂ€ Email: mika.mantyla@helsinki.fi Affiliation: University of Helsinki, Helsinki, Finland Abstract Screening in systematic reviews (SRs) is manual and time-consuming. Prior work has explored large language models (LLMs) for automating this step, but LLMs are evolving rapidly, so earlier performance claims may no longer accurately reflect their screening performance. We used an existing software engineering SR screening benchmark (SESR-Eval) as our data. We also power-sampled a new, smaller dataset (SESR-Eval-Mini) that allows evaluation at lower costs. Using this data, we evaluated eight new LLMs for screening performance. Additionally, we tested different prompts, analyzed LLM agreement in screening decisions and criteria, and examined the effect of refining the inclusion and exclusion criteria on screening performance. The eight new LLMs performed marginally better than the seven old ones: avg. MCC across secondary studies rose from 0.347 to 0.365. Differences between secondary studies are still bigger than between LLMs. Computing the overall screening decision from criterion-level decisions degraded screening performance only slightly. LLMs generally agree with each other in their corresponding screening decisions (mean Gwetâs AC1 = 0.830), though certain inclusion and exclusion criteria showed larger disagreement than others. Refining the inclusion and exclusion criteria slightly improved recall and made decisions easier for some LLMs, but overall impacts of criteria refinement were modest. LLMs are not yet ready to replace humans in paper screening and the advantages new, more costly models bring, appear to be very limited. Agent-based approaches, prompt engineering, and further criteria refinement are three potential future research avenues. keywordssystematic reviews, large language models, title-abstract screening, systematic review automation 1 Introduction Evidence-Based Software Engineering (EBSE) highlights systematic reviews (SRs) as a key method for gathering and synthesizing evidence in software engineering (SE) (Kitchenham et al., 2004). However, conducting SRs is labor-intensive and time-consuming (Felizardo et al., 2025). According to Marshall and Wallace (2019), a single SR can take an average of 67 weeks to complete, with paper screening representing one of the most demanding phases. Prior work has shown that humans spend approximately 89 seconds screening each individual paper (Huotala et al., 2024), which makes paper screening a substantial bottleneck in the SR process as thousands of papers may require screening. Automating or supporting paper screening has therefore attracted considerable attention. Earlier approaches have used traditional Machine Learning (ML) and Natural Language Processing (NLP) techniques (Hamel et al., 2020), while recently, Large Language Models (LLMs) have been used because of their ability to reason over textual input (Huotala et al., 2024; Huotala et al., 2025a; Felizardo et al., 2024; Li et al., 2024; Delgado-Chaves et al., 2025; Shameem et al., 2026; Thode et al., 2025; Tran et al., 2023; Matsui et al., 2024; Guo et al., 2024; Serretti, 2026; Trad et al., 2025; Adam et al., 2026). Existing studies show promise for paper screening using LLMs, as LLMs achieve performance comparable to that of non-expert human screeners (Huotala et al., 2024; Huotala et al., 2025a; Felizardo et al., 2024). Yet, against expert human screeners, LLMs still make a considerable number of mistakes stemming from sources such as different interpretations of key terms (MĂ€ntylĂ€ et al., 2026). Guidelines for LLM usage have been proposed to mitigate their errors (Madeyski et al., 2026). Human-in-the-loop systems have also been developed to support researchers in applying LLMs and other automation methods to paper screening (Huotala et al., 2026a; Deizepe et al., 2026; Afonso et al., 2026; Ouzzani et al., 2016). This paper extends our prior work (Huotala et al., 2025a), which introduced the SESR-Eval dataset for monitoring artificial intelligence (AI) performance in paper screening. Nine LLMs were benchmarked, and most of them performed similarly. However, the differences in screening accuracy between the secondary studies were greater than differences between LLMs, but LLMs were fairly cheap to use per secondary study. While the benchmark dataset can be used to benchmark LLMs, LLMs were still not recommended for automating paper screening process. Next we explain the difference between this paper and Huotala et al. (2025a). First, the evaluated LLMs covered only a single generation of models, prompt, and an output schema, which leaves it unclear whether newer LLMs, different user prompts or output schemas would further improve screening performance and make paper screening feasible with LLMs. Claims of earlier generation models may no longer accurately reflect the capabilities of newer models in paper screening, and benchmarking additional LLMs for paper screening benefit researchers and practitioners in choosing a cost-efficient LLM with an acceptable performance for the task (RQ1 â RQ2). Benchmarking LLMs on a full dataset (N = 34,528) may not be financially sensible, as LLMs are generally expensive to run (Felizardo et al., 2025), and token costs of LLMs have increased due to test-time inference and additional model capabilities, such as thinking models (RQ1.2). Computing the overall decision separately from criterion-level decisions can possibly reduce hallucinations, falsely judged screening decisions, and allows for additional flexibility (RQ2). Second, the agreement in overall screening decisions and across inclusion and exclusion criteria was not analyzed in the paper, even though it could reveal whether LLM screening decisions are uniform across different models, or sensitive to the selected model (RQ3.1 â RQ3.2). Third, the modification of inclusion and exclusion criteria was not studied in the paper, even when certain criteria cannot possibly be evaluated purely by title and abstract. Improving poorly formatted, unclear, or ambiguous criteria may increase the likelihood of a correct overall or criterion screening decision, which directly motivates RQ4. We therefore address the following research questions: RQ1: How do LLMs perform in paper screening? RQ1.1: How do LLMs perform in paper screening on the full SESR-Eval dataset? RQ1.2: How do LLMs perform in paper screening on the sampled SESR-Eval-Mini dataset? RQ2: How does computing the overall screening decision from criterion-level decisions affect LLM screening performance on the sampled data? RQ3: To what extent do LLMs agree with each other? RQ3.1: To what extent do LLMs agree with each other in their overall screening decisions? RQ3.2: To what extent do LLMs agree with each other in individual screening criteria? RQ4: To what extent does refining the inclusion and exclusion criteria affect LLM screening performance? For RQ1, we benchmark eight new state-of-the-art LLMs against seven older ones. On the full dataset (RQ1.1), we benchmark two of the new LLMs, and all eight new LLMs on a smaller 5% sample, SESR-Eval-Mini (RQ1.2). We then investigate how prompt design and output structuring affect screening performance in RQ2. For RQ3, we analyze agreement in overall screening decisions across LLMs (RQ3.1) and also across individual inclusion and exclusion criteria (RQ3.2). Finally, for RQ4, we examine whether refining the inclusion and exclusion criteria improves screening performance. The remainder of the paper is structured as follows. Section 2 reviews related work. Section 3 provides the necessary background. Section 4 describes the methodology, and Section 5 presents the results. Section 6 discusses the findings, and Section 7 concludes the paper. 2 Related Work Table 1: Related work of paper screening with LLMs. Study Year Discipline LLMs Huotala et al. (2024) 2024 Software Engineering 2 Huotala et al. (2025a) 2025 Software Engineering 9 Felizardo et al. (2024) 2024 Software Engineering 1 Li et al. (2024) 2024 Software Engineering 9 Delgado-Chaves et al. (2025) 2025 Software Engineering 18 Shameem et al. (2026) 2026 Software Engineering 1 Thode et al. (2025) 2025 Software Engineering 5 DennstĂ€dt et al. (2024) 2024 Biomedical 4 Tran et al. (2023) 2023 Medical 1 Matsui et al. (2024) 2024 Medical 2 Guo et al. (2024) 2024 Medical 2 Serretti (2026) 2026 Medical 1 Trad et al. (2025) 2025 Medical 1 Adam et al. (2026) 2026 Medical 3 JĂłzwiak et al. (2026) 2026 Health Technology Assessment 9 Qiu et al. (2026) 2026 Health Technology Assessment 1 Sujau et al. (2025) 2025 Veterinary Medicine 2 Behrouzian et al. (2026) 2026 Artificial Intelligence 2 Table 1 presents related work on using LLMs to automate paper screening in SRs. The remainder of the section presents the current state of LLM-assisted paper screening in multiple disciplines / domains. LLM-assisted paper screening has been studied in various disciplines, including SE (Huotala et al., 2024; Huotala et al., 2025a; Felizardo et al., 2024; Li et al., 2024; Delgado-Chaves et al., 2025; Shameem et al., 2026; Thode et al., 2025), medicine (Tran et al., 2023; Matsui et al., 2024; Guo et al., 2024; Serretti, 2026; Trad et al., 2025; Adam et al., 2026), biomedicine DennstĂ€dt et al. (2024), health technology assessment (JĂłzwiak et al., 2026; Qiu et al., 2026), veterinary medicine (Sujau et al., 2025), and artificial intelligence (Behrouzian et al., 2026). Next, we first go over software engineering studies, followed by studies from other disciplines, and conclude with a summary on the related work. 2.1 Software Engineering Huotala et al. (2024) found that GPT-3.5 and GPT-4 LLMs perform as well as Masterâs students in paper screening. By simplifying abstracts they were faster to read, but simplification did not increase human performance. GPT-3.5 and GPT-4 LLMs had a precision and recall of (0.650, 0.176) and (0.500, 0.417), respectively. Including task examples in the screening task (one-shot (OS) and few-shot (FS)) was observed to improve screening performance, but the authors must have a list of high-quality seed papers to start with. The usage of prompt optimization was recommended. Huotala et al. (2025a) created the SESR-Eval benchmark dataset for benchmarking LLMs in paper screening, and nine LLMs were benchmarked in paper screening using the dataset. Most LLMs performed similarly in paper screening, and the performance differences between secondary studies were greater than for LLMs. GPT-4.1 mini performed the best, but small models, such as Ministral 8B and GPT-4.1 Nano, had poor screening performance. The cost of using LLMs to screen papers was relatively low. The authors concluded that LLMs are not yet recommended for automating paper screening, since accuracy varied widely across secondary studies, and as no LLM managed a high recall with reasonable precision. Felizardo et al. (2024) evaluated the accuracy of GPT-4 (ChatGPT) in paper screening for two SRs. They replicated the first selection phase of two SRs with the help of ChatGPT, where the LLMsâ screening results were compared to those of humans. The results were that the overall accuracy in paper screening was 80.7%, and that in both SRs the LLMs made false judgments regarding included and excluded papers. For the first SR, there were 16 false negatives (FN) and 17 false positives (FP). For the second SR, there were 35 FNs and 27 FPs. The recommendation was not to rely completely on LLMs for paper screening, but LLMs can be a valuable tool with SRs as they can mitigate bias and inaccuracies in the included evidence. Li et al. (2024) selected benchmark datasets from the SYNERGY dataset. Using that data, they benchmarked nine LLMs in terms of overall accuracy, sensitivity, and specificity. The results of the LLMs were then compared to gold-standard human decisions. The performance of LLMs varied significantly, but GPT-4 demonstrated a good balance of sensitivity and specificity and the accuracy was consistently in the high 90%s. This indicates a high potential for LLMs in paper screening tasks, but they are not yet ready to fully replace humans in the task. Delgado-Chaves et al. (2025) benchmarked 18 LLMs in paper screening. They picked three existing SRs, from which about 6,500 primary studies were evaluated by 18 different LLMs. From the primary studies, 92%, 88%, and 40% of the papers were correctly classified by the LLMs, respectively. There was a significant variation between secondary studies, and the structure of the inclusion and exclusion criteria significantly affected the performance of LLMs in the task. The best-performing LLMs in terms of Matthews Correlation Coefficient (MCC) were GPT-4o (MCC=0.349) and GPT-4o mini (MCC=0.342). The authors emphasized that extra caution should be taken when articulating inclusion and exclusion criteria. A workload reduction between 33% and 93% was observed during paper screening. Shameem et al. (2026) followed a mixed-methods research to benchmark LLMs on multiple phases of SRs - including formulating research questions, generating search strings, quality assessment, developing taxonomies, and paper screening. First, they selected four SRs as "benchmark studies" (BS), and GPT-4o was used to screen the relevant papers. For all benchmark studies (BS1, BS2, BS3, BS4), GPT-4o correctly classified 64.5%, 68%, 58%, and 72% of papers correctly, respectively. Recall was 55.17%, 50%, 100%, and 72%, for the respective benchmark studies. Precision was 76.1%, 31.2%, 55.3%, and 72%, respectively. However, there were certain domain-specific challenges (e.g., domain-specific literature) that humans would have identified, but the LLM failed to do so. Thode et al. (2025) benchmarked five LLMs in paper screening, and used the SRâs labeled studies as the ground truth. They designed three different prompts for the task. In addition, they created a dual-LLM system to test how a two-reviewer screening workflow would work with LLMs. As a result, LLMs exhibited a high recall (up to 98%) but low precision (27%). The dual-LLM system managed an average recall of 99% with a precision of 27%. The best-performing LLM was Meta Llama 2 with an average recall of 98%, while GPT-4 Turbo had the highest average precision (72%). 2.2 Other Disciplines DennstĂ€dt et al. (2024) used LLMs to screen papers based on the predefined inclusion and exclusion criteria and the paperâs title and abstract. The performance of LLMs in paper screening varied depending on the LLM being used and on the dataset. FlanT5 was the best-performing model with sensitivity (recall / true positive rate) and specificity (true negative rate) of 94.48%/31.78%. OpenHermes-NeuralChat, Mixtral and Platypus 2 were trailing quite a bit behind, with sensitivity/specificity of 97.58%/19.12%, 93%/75.19% and 97.58%/38.34% for the three models, respectively. Even small changes to the user prompt or the Likert scale from 1-5 to 1-10 significantly impacted the screening performance of LLMs. Tran et al. (2023) evaluated the GPT-3.5 Turbo LLM for paper screening by first developing prompts that aim to mimic the process of screening by human researchers. The LLM was tested on 22,666 papers. The LLM in the paper screening task had a sensitivity of 97.1%, but with poor specificity (37.7%). A workload reduction of 65% was observed. While the tested LLM had a good sensitivity, human intervention was recommended because of the poor specificity. Matsui et al. (2024) evaluated paper screening with GPT-3.5 and GPT-4 LLMs in two SRs containing 1,381 and 3,146 papers, respectively. Both of the LLMs had a fairly high paper screening throughput - about 110 records per minute. The total screening task was completed in two hours. In the first SR, GPT-3.5 had a sensitivity/specificity of 0.900/0.709 versus GPT-4âs 0.806/0.996. For the second SR, the performance was fairly similar, with sensitivity/specificity of GPT-3.5âs 0.958/0.116 versus GPT-4âs 0.875/0.855. A deeper analysis found that GPT-3.5 excluded papers because of insufficient domain knowledge, while GPT-4 misinterpreted inclusion criteria. Guo et al. (2024) benchmarked the GPT-4 LLM in paper screening. They screened over 24,000 title-abstract pairs across six clinical reviews. GPT-4 had an accuracy of 91.0%, F1 score of 0.60, sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The agreement between models was high (kappa=0.96) and the LLMs could give justifications for their incorrect classifications. The authors concluded that LLMs have the potential to improve SR efficiency and lead to more accurate and reliable conclusions in medical research. Serretti (2026) developed a fully local LLM pipeline using the open-weight gpt-oss-20b LLM to benchmark title/abstract screening for three SRs. The performance (F1 score) of the LLM in paper screening was 98.9%, 76.0%, and 83.6% for the three SRs, respectively. The findings demonstrate that a fully locally run LLM can support researchers in paper screening. Trad et al. (2025) evaluated the performance of a commercial paper screening tool called Rayyan, and an LLM-based paper screening system in automating the screening process. Out of 14,439 papers, two thousand papers were used to train the paper classifier of Rayyan. The paper screening performance was analyzed using different thresholds for paper inclusion. While Rayyan performed adequately (exclusion rate of 50.7%-72.1%), the LLM-based screening system was far superior, achieving an exclusion rate of 99.5%, and a workload reduction of 95.5%, as the screening took 25.5 hours. The paper demonstrates that LLM-based screening systems are far superior compared to traditional ML-based systems, such as Rayyan. Adam et al. (2026) conducted a large experiment with 515 different prompts against open-source LLMs (Meta Llama and Mistral) to study how the prompt structure affects screening performance in four SRs from the medical domain. The resulting F1, recall and precision were 27.6%, 83.6% and 18.5%, respectively. The paper screening performance was observed to increase significantly when 1) larger-parameter LLMs were used; 2) the prompt included the title of the SR being conducted; and 3) the prompt included the selection criteria. Screening accuracy decreased when abstracts were used in screening rather than paper titles. The best-performing prompt achieved 72.2% recall and 66.1% specificity. This translated into a workload reduction of around 34.5â37.5%. The paper highlighted that there is a precision-recall tradeoff to be made and the prompt structure significantly affected screening performance. JĂłzwiak et al. (2026) benchmarked Gemini 3 Flash and GPT-5.2 LLMs in paper screening and experimented with how prompt optimization could improve paper screening performance. A dataset of 100 papers was constructed from the papers that had the highest inter-model disagreement ratio, and was labeled by experts for the experiment. Prompt optimization improved recall for all LLMs by +3.7% to +37.1%. The best-performing LLM was Gemini 3 Flash with an accuracy of 91%, and F1 score of 81.6%, while costing 25 times less than GPT-5.2 per abstract. Qiu et al. (2026) benchmarked LLMs for paper screening by using a stepwise training framework with three studies - one for each phase. The first study was used to optimize the technical parameters of the experiment with 1000 abstracts and 50 full texts. The second and third studies validated this approach. The LLMs achieved 100% sensitivity and a specificity between 85.6% and 93%. Full-text screening had an accuracy between 82.1% and 96.2%. Sujau et al. (2025) evaluated the effectiveness of LLMs to rank primary studies for climate-sensitive zoonotic disease research. The dataset contained 2905 title-abstract pairs that were related to four zoonotic diseases, and a set of paper selection criteria was constructed. Five researchers screened abstracts using the defined criteria. After that, the papers were screened using GPT-4o LLM with zero-shot (ZS) and Chain-of-Thought (CoT) prompting techniques. A workload reduction of at least 70% was achieved, while the LLM maintained a recall of 95%. The authors emphasized that further research is needed on standardized benchmark datasets, but the findings demonstrate that LLMs can substantially reduce the screening effort. Behrouzian et al. (2026) used GPT-5.1 and GPT-3.5 Turbo LLMs to benchmark paper screening against a dataset from the Cochrane review corpus, which contained 1,000 abstracts. The recall of GPT-5.1 and GPT-3.5 Turbo ranged from 0.72 to 0.32 and 0.92 to 0.18, respectively. Excluding papers immediately, rather than labeling papers as unsure, made both models perform better with consistency. This demonstrated that LLMs do not handle uncertainty well in paper screening. 2.3 Summary The common consensus of the related work is that LLMs show significant promise in paper screening, by reliably reducing the workload of practitioners and researchers in paper screening tasks, while LLMs have been observed to perform as well as non-expert humans in the task. Despite these promises, LLMs are not yet recommended for fully replacing humans in paper screening, as the screening performance is greatly affected by the structure and clarity of the inclusion and exclusion criteria, the structure of the prompt, certain domain-specific issues, and the trade-off of selecting a cost-effective LLM instead of a state-of-the-art LLM to reduce costs. Existing literature predominantly treats paper screening as a binary classification task, as they examine only the overall screening performance, and provide only a minimal criterion-level analysis of how LLMs handle specific inclusion/exclusion criteria or whether the performance varies across criteria types. Furthermore, only a few studies investigate inter-LLM agreement and the impact of dataset sampling, or how refining criteria affects paper screening performance. 3 Background In this section, we introduce the core concepts for this study. First, we describe the paper screening process in SRs. Then, we describe paper screening as a classification task, and how the criterion-level decisions are aggregated into an overall screening decision. Finally, we discuss the agreement between multiple screeners. 3.1 Inclusion and exclusion criteria In SRs, inclusion (IC) and/or exclusion (EC) criteria are applied to research papersâ titles and abstracts to determine whether the paper should be included or excluded (Kitchenham and Charters, 2007). Excluded papers are discarded, and the remaining included papers are further examined in the full-text screening phase (Kitchenham and Charters, 2007). Each of these criteria map to a specific research question (RQ) of the study, so the selected primary studies have the possibility to contain an answer to the RQ. The SR guidelines state that the criteria should be crafted during the SR protocol development, and may be refined during the literature search process (Kitchenham and Charters, 2007). Importantly, criteria should be formulated so that they can be reliably assessed and classify studies with high accuracy. Some inclusion and exclusion criteria may be applied to the paperâs metadata, such as language, date of publication, and publisher information. While human researchers can easily determine whether a paper was published in a certain year or period, LLMs can have issues making such decisions without having access to the paperâs metadata or, in some cases, the publisherâs copyright statement in the abstract (Felizardo et al., 2025). In addition, it is noted that criteria that assess the language of the paper, or are biased towards the knowledge of the authors, can influence the decisions based on the criteria (Kitchenham and Charters, 2007; Kitchenham et al., 2004). The implications of these observations raise important questions regarding how LLMs handle screening based on a set of rules. First, since inclusion and exclusion criteria have differences in how they can be assessed objectively, decisions made by LLMs can be more consistent with certain criteria than with others. This motivates RQ3, where we analyze how LLM screening decisions actually agree across paper selection criteria (RQ3.2). Second, as Kitchenham and Charters (2007) states that the criteria may be refined during the literature search process, it remains uncharted whether such refinement also improves the performance of LLMs in paper screening. This motivates our RQ4, which investigates the effect of refining the paper selection criteria on LLM screening performance. 3.2 Title-abstract screening as a classification task Title-abstract screening (paper screening) can be considered a binary classification task (Du et al., 2024; van de Schoot et al., 2021), where each paper is either included or excluded based on its title and abstract (Kitchenham and Charters, 2007). However, SRs typically include multiple inclusion and exclusion criteria, and the overall inclusion/exclusion decision depends on how individual criterion-level decisions are combined or aggregated. A common pattern is that a paper is excluded if it matches even one exclusion criterion, or if it does not match every inclusion criterion. However, variations of these rule combinations do exist. This complexity of handling multiple criteria raises important questions about inter-screener agreement, and how criterion-level decisions should be aggregated. This also motivates our RQ3.1 and RQ3.2. When LLMs are applied in paper screening, a common pattern is that the LLM produces an overall decision in binary format. This binary classification task can be expanded to support probabilistic (a floating point between 0.0 and 1.0), or e.g., Likert-scale decisions (Felizardo et al., 2024; Huotala et al., 2025a). The aggregation behind the overall decision of the LLM can be done in two ways. In the first way, the LLM produces judgments for each criterion and also the overall decision. This adds a considerable amount of flexibility, but there is an issue. Because most LLMs are decoder-only transformers (Vaswani et al., 2023), they predict the next token(s) based on the earlier tokens, the overall decision may contradict the LLMâs own criterion-level judgments, if e.g., the overall decision is generated before the criterion-level decisions. In the second way, the LLM produces only the judgments for the criteria, and the overall decision is aggregated manually from the judgments of the criteria. The benefit here is that the decision boundary can be adjusted flexibly, if one wants to e.g., maximize recall (Huotala et al., 2025a). However, if a study has a significant number of criteria, a single incorrect criterion judgment is enough to falsely exclude a possibly relevant paper. This trade-off here motivates RQ2. 3.3 Agreement between multiple raters 55101015150.40.40.60.60.80.811Number of decision makersAccuracy of a majority votep=0.70p=0.70p=0.60p=0.60Random guess Figure 1: Accuracy of majority voting as a function of the number of independent decision makers. The standard SR methodology recommends that at least two independent reviewers screen papers, to reduce bias and ensure rigor (Kitchenham and Charters, 2007). The benefit of multiple reviewers can be formalized by the Condorcet Jury Theorem (List and Goodin, 2001; Shteingart et al., 2020), which demonstrates that when individual decision-makers make the correct binary decision with a higher than 50/50 chance, the accuracy of a majority vote increases as the number of independent decision-makers increases (Figure 1) (List and Goodin, 2001; Shteingart et al., 2020). Empirical evidence supports this principle. A study by Abramowitz et al. (2025) found that multiple independent reviewers provide more reliable quality assessment than a single reviewer in peer review contexts. Three conditions of the Condorcet Jury Theorem are important when LLMs screen papers. First, the benefit of multiple raters requires a majority. Second, with two LLMs as decision makers, a majority vote cannot resolve a possible disagreement. Using two LLMs to screen is therefore only useful to flag disagreements, or to include a paper if either of the models includes it. This increases recall, but lowers precision. Third, the LLMs making the decisions must reach their decisions independently. It is known that LLMs share much of their training data, so choosing e.g. models from only a single provider might reduce the collaborative usage of these LLMs, and most likely the errors of the LLMs are correlated. A high agreement between LLMs is therefore ambiguous. This might indicate that the LLMs themselves are reliable, or they make the same mistakes, which makes it pointless to use multiple LLMs for screening. A comprehensive evaluation of LLM reliability requires examining both inter-model agreement and the consistency of decisions across different LLMs and criteria. Such analysis is essential for assessing whether LLMs can reliably replace or augment human screeners in the paper screening phase of SRs. For these reasons, LLM agreement is analyzed between LLMs (RQ3.1) and between inclusion and exclusion criteria (RQ3.2). 4 Methodology Figure 2: The study methodology, including numbered actions and the conducted experiments. Next, we cover the methodology used in the study. We followed the SIGSOFT empirical standards to conduct this research (Ralph et al., 2020). The high-level overview of the methodology can be seen in Figure 2. First, in Subsection 4.1, we describe how LLMs were selected for the experiments â . Then, we describe how the data sampling for the SESR-Eval-Mini dataset was conducted in Subsection 4.2 âĄ. After this, we give a detailed overview of how we benchmarked the LLMs (Subsection 4.3), how the computation of the overall screening decision was separated (Subsection 4.4), the setup behind the LLM agreement between each other and criteria (Subsection 4.5), and the criteria modification process (Subsection 4.6). In Subsection 4.7, we cover the statistical methods used. 4.1 LLM selection (all RQs) Table 2: Selected models with their reasoning effort setting, costs per 1M tokens (in U.S. dollars), and Pydantic AI output type for OpenRouter. SESR-Eval models are picked from Huotala et al. (2025a). 1 Pricing as of September 2026 2 Pricing as of April 2025 Publisher Model Reasoning effort Input ($) / 1M tokens Output ($) / 1M tokens Output type LiveBench OpenAI GPT-5.5 Low $5.00 1 $30.00 1 Native Anthropic Claude Opus 4.7 Low $5.00 1 $25.00 1 Tool Google Gemini 3.1 Pro Low $2.00 1 $12.00 1 Tool Z.AI GLM 5.1 Low $1.05 1 $3.50 1 Tool Moonshot AI Kimi K2.6 Low $0.74 1 $3.49 1 Tool SpaceXAI Grok 4.20 Beta Low $1.25 1 $2.50 1 Tool Alibaba Qwen 3.6 Plus Medium $0.32 1 $3.20 1 Prompted DeepSeek V4 Pro High $0.435 1 $0.87 1 Tool SESR-Eval OpenAI o3-mini High $1.10 2 $4.40 2 Native OpenAI GPT-4o N/A $2.50 2 $10.00 2 Native OpenAI GPT-4.1 N/A $2.00 2 $8.00 2 Native OpenAI GPT-4.1 mini N/A $0.40 2 $1.60 2 Native Anthropic Claude 3.7 Sonnet N/A $3.00 2 $10.00 2 Native DeepSeek DeepSeek R1 N/A $0.54 2 $2.18 2 Native Meta Llama 4 Maverick N/A $0.19 2 $0.85 2 Native For the LLM selection â , we first picked existing LLMs that were benchmarked with the SESR-Eval dataset (Huotala et al., 2025a), excluding Ministral 8B and GPT-4.1 Nano, as they exhibited poor screening performance compared to the rest of the models in the work. Selecting the existing LLMs gives us a good baseline that we can use to directly compare the newer models against. In addition to the existing LLMs, we selected some new LLMs from the LiveBench leaderboard by White et al. (2025) (version 2026-01-08 11 1 https://web.archive.org/web/20260504142848/https://livebench.ai/#/?highunseenbias=true), to see how the current state-of-the-art performs in paper screening. LiveBench actively maintains a list of LLMs alongside the performance results in various domains and contexts, including reasoning, coding, mathematics, and data analysis. From the leaderboard, we first picked eight distinct publishers: OpenAI 22 2 https://openai.com, Anthropic 33 3 https://w.anthropic.com, Google 44 4 https://ai.google, Z.AI 55 5 https://z.ai, Moonshot AI 66 6 https://w.moonshot.ai, SpaceXAI 77 7 https://x.ai, Alibaba 88 8 https://qwen.ai, DeepSeek 99 9 https://deepseek.com â not in any particular order. Next, for each publisher, we selected their LLM with the highest global average performance score to minimize the risk of picking LLMs that fare better in some domain or context. As demonstrated by Huotala et al. (2025a), selecting the most expensive model does not guarantee an increase in screening performance. While most new LLMs were set to a âlowâ reasoning effort, the reasoning effort of DeepSeek V4 Pro and Qwen 3.6 Plus were set to âhighâ and âmediumâ, since they were the first LLMs we tested, we mirrored the reasoning effort setting of the existing o3-mini LLM. As we previously demonstrated that "thinking" models give negligible improvements in screening performance when reasoning effort is increased, and that it increases token output significantly (Huotala et al., 2025a), we then decided to configure the rest of the models to a âlowâ reasoning effort. In total, we selected 15 LLMs for the experiments (Table 2) â seven of which were previously benchmarked in the SESR-Eval paper, and eight new LLMs, which represent the current state-of-the-art. 4.2 Data sampling â SESR-Eval-Mini (all RQs) Table 3: Number of papers per secondary study, and the number of papers in the 5% power-allocated sample, α=0.5α=0.5. Included:Excluded (I:E) is the percentage split of included to excluded papers. Secondary studies sourced from Huotala et al. (2025a). Full Sampled Study N I:E N I:E Casalaro et al. (2022) 2,260 3:97 109 3:97 Sharbaf et al. (2023) 322 53:47 41 46:54 Tambon et al. (2022) 1,024 18:82 73 25:75 RodrĂguez-PĂ©rez et al. (2021) 2,748 8:92 120 6:94 Bjarnason et al. (2023) 3,703 1:99 139 1:99 Kuutila et al. (2020) 4,186 5:95 148 5:95 Geraldi et al. (2020) 167 34:66 30 27:73 Zhang and Li (2020) 104 100:0 23 100:0 Tebes et al. (2020) 495 72:28 51 78:22 Ebrahimi et al. (2021) 111 50:50 24 46:54 Lewowski and Madeyski (2022) 1,651 10:90 93 10:90 Van Dinter et al. (2022) 559 25:75 54 19:81 Stradowski and Madeyski (2023) 1,194 62:38 79 59:41 Alonso et al. (2023) 222 86:14 34 79:21 Somers et al. (2023) 405 33:67 46 26:74 Teixeira et al. (2017) 318 59:41 41 63:37 Lenarduzzi et al. (2021) 301 14:86 40 15:85 Pereira et al. (2021) 69 100:0 19 100:0 BariĆĄiÄ et al. (2022) 326 47:53 41 51:49 Deckers and Lago (2022) 321 6:94 41 7:93 Rani et al. (2023) 2,079 3:97 104 1:99 Zakeri-Nasrabadi et al. (2023) 9,695 5:95 225 7:93 Shevtsov et al. (2018) 1,487 11:89 88 17:83 Lin et al. (2022) 781 16:84 63 14:86 Total 34,528 12:88 1,726 21:79 After the LLM selection â , we continued with data sampling âĄ. We decided to conduct data sampling, as our main goal was to provide a cheaper alternative of SESR-Eval dataset, which is motivated by Table 2 as it shows that newer flagship models have significantly increased input and output token costs. In addition, we did not want to trade better performance for lower costs by choosing smaller LLMs, which are evidently cheaper (Felizardo et al., 2025). As the data source, we used SESR-Eval by Huotala et al. (2025a), which is a gold-standard paper screening dataset. The dataset contains 34,528 pre-labeled primary studies from 24 software engineering secondary studies, which were rigorously sourced from high-reputation SE journals (Huotala et al., 2025b). The labels of the primary studies are the overall screening decisions. For the paper sampling strategy, we decided to use power-allocated sampling with α=0.5α=0.5 (Bankier, 1988). Power-allocated sampling balances the proportional and equal allocation to control for variance across strata (Bankier, 1988). In practice, pure random sampling (a power-allocated sample with an α=1.0α=1.0) at 5% would give us a sample where the smallest secondary study (with 69 primary studies) would get an average of 3.45 primary studies, which would not allow any sort of secondary study-level inference on the accuracy. On the other hand, with an α=0.0α=0.0, all secondary studies would get equal weight, which at 5% would be 72 primary studies for each secondary study (34,52834,528 Ă 0.050.05 / 24). This would distort the overall dataset too much in favor of small studies. The value of α=0.5α=0.5 is a compromise as it balances between equally allocating across all strata, and allocating proportional to the size of each stratum Bankier (1988). Table 3 lists the secondary studies, their full primary study count and the sampled primary study count. The full dataset used in the experiments contains 34,528 rows, whereas the 5% sample set contains 1,726 papers. One can notice that the sampled data has a different I:E ratio overall (21:79) than the full dataset (12:88). This is because smaller secondary studies had more inclusions and their proportional weight increases in power-allocated sampling. For RQ4, the subset of studies was further reduced to 448 papers, as we selected only those papers that were part of the seven secondary studies most suitable for criteria improvement â€. The 5% sampled dataset is called the SESR-Eval-Mini dataset, which is available in our prior workâs (Huotala et al., 2025a) research artifact as an update (Huotala et al., 2026c). 4.3 Benchmarking the LLMs (all RQs) Table 4: Tested models for each research question, with sample counts. F = full (N=34,528), S = 5% power-allocated sample (N=1,726), R = papers of the 7 secondary studies from 5% power-allocated sample (N=448). Bottom row indicates the user prompt and output schema version used. Model RQ1.1 RQ1.2 RQ2 RQ3.1 RQ3.2 RQ4 LiveBench GPT-5.5 - S S - - R Claude Opus 4.7 - S S - - R Gemini 3.1 Pro - S S - - R GLM 5.1 - S S - - R Kimi K2.6 - S S - - R Grok 4.20 Beta - S S - - R Qwen 3.6 Plus F S S F, S F, S R DeepSeek V4 Pro F S S F, S F, S R SESR-Eval o3-mini F S - F, S F, S - GPT-4o F S - F, S F, S - GPT-4.1 F S - F, S F, S - GPT-4.1 mini F S - F, S F, S - Claude 3.7 Sonnet F S - F, S F, S - DeepSeek R1 F S - F, S F, S - Llama 4 Maverick F S - F, S F, S - User prompt & Output schema version V1 V1 V2 V1 V1 V2 Next, we describe how we benchmarked the LLMs for each research question. Tested models for each RQ are listed in Table 4. For screening on the full dataset (RQ1.1), we re-used the screening results of the seven existing LLMs that were gathered in prior work (Huotala et al., 2025a), and benchmarked two new LLMs (see Table 4). For RQ1.2, we derived the sampled screening results for the seven existing LLMs and two new LLMs from RQ1.1, but ran the benchmarks for the remaining six new LLMs that were not benchmarked in RQ1.1. For RQ2, all eight new LLMs were benchmarked on the 5% sample. In the case of RQ3.1 and RQ3.2, the same old and new LLMs as for RQ1.1 were used, as RQ3.1 and RQ3.2 are based on the data gathered from RQ1.1. As benchmarking on the full dataset made it possible to derive the results for the sampled data, we decided to also derive the results for the sampled data in RQ3.1 and RQ3.2, respectively. As all new LLMs were benchmarked for RQ2, we decided to also benchmark them on criteria refinement for RQ4. This makes sense, as both RQ2 and RQ4 use the same user prompt and output schema. One might wonder why certain LLMs were not tested for certain RQs. After we first had benchmarked two new LLMs, Qwen 3.6 Plus and DeepSeek V4 Pro, for RQ1.1, we decided to not run rest of them (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, GLM 5.1, Kimi K2.6, or Grok 4.20 Beta) on the full dataset, as running these LLMs on the full dataset would have inflated costs significantly, and current LLMs are quite expensive to run (Felizardo et al., 2025).1010 10 Our current cost for this study was $1,100. Had we tested those models with the full SESR-Eval dataset, we estimated a cost of about $6,500. After reviewing prior observations that the performance differences between LLMs are fairly narrow (Huotala et al., 2025a), testing only two of the new LLMs for RQ1.1 was justified. As the sampled data represents 5% of the full dataset with greatly reduced costs, we were able to benchmark all new LLMs for RQ1.2, RQ2, and RQ4. We used OpenRouter (Aubakirova et al., 2026) to access the LLMs, which provides a unified application programming interface (API) to call the LLMs routed through the service. An additional benefit of using OpenRouter was that it supports Pydantic AI out of the box, which is a model-agnostic Python framework. It provides automatic output validation, error handling, and retry support that makes it possible to call LLMs reliably. Our recent work used OpenRouter and Pydantic AI for paper screening and it worked very well for benchmarking multi-vendor LLMs (Huotala et al., 2025a). The scripts are available in the paperâs research artifact (Huotala et al., 2026b). The benchmarking was done with a parallelized Python script, which used the Pydantic AI library for calling LLMs. The LLMs were configured with a top_p value of 0.1, and the following system prompt: You are an expert research assistant. Recent works by Huotala et al. (2024); Huotala et al. (2025a) and Felizardo et al. (2024) have used the same parameters and system prompt to increase the reproducibility and determinism of LLM outputs without affecting output quality. We did not set a temperature parameter this time, as OpenAIâs current recommendation is to adjust either top_p or temperature, not both (Chiang and Lee, 2023). The results were output as Comma-Separated Values (CSV) files, which were further processed using pandas (3.0.1) and scikit-learn (1.8.0) Python libraries. For both RQ1.1 and RQ1.2, we computed accuracy, precision, recall, F1, MCC, and lost evidence. 4.4 Separating the computation of the overall screening decision (RQ2) Figure 3: User prompt and output schema versions, the changes made for each version, and the RQs in which they are used. V1 user prompt and output schema are from Huotala et al. (2025a). Figure 3 describes the changes made to the user prompt and the LLM output schema, respectively. The first version (V1) of the user prompt was the same as that used in our prior work (Huotala et al., 2025a). This same user prompt was then used for RQ1.1, RQ1.2, RQ3.1, and RQ3.2, as these RQs were directly compared with the results of our prior work, so no changes were required nor made to the user prompt âą. For RQ2, the changes made to the user prompt in V2 were tightly coupled with the motivation behind the respective RQ. Removing the overall screening decision field necessitated refining the user prompt, as the screening decision of the LLMs relied on the criterion decisions. Because of this, the instructions given to the LLMs were refined to better guide the LLMsâ interpretation of the criteria. Also for RQ4, the V2 version user prompt was used, because it focused solely on refining inclusion and exclusion criteria; the refinements made for V2 version of user prompt aimed to provide an improved guidance for the LLMs and to reduce the ambiguity of applying criteria âŁ. No changes were made for the output schema for RQ1.1, RQ1.2, RQ3.1, or RQ3.2, as the results were directly compared with prior work (Huotala et al., 2025a). On the other hand, significant changes were implemented to the V2 output schema (RQ2 and RQ4). First, the overall decision field was removed from the output schema, as the overall decision was computed from the inclusion and exclusion criterion decisions, making the overall decision field redundant. Second, we added extra validation for the criterion name format as an attempt to improve reliability of the structured responses. The increased number of LLMs tested increases the likelihood of LLMs hallucinating the output, and letting the Pydantic AI library apply stricter output validations is an attempt to mitigate this. Third, the Python classes for Criterion and Decision were split for inclusion and exclusion criteria, as we wanted to make sure that the LLMs rigorously understand the difference between inclusion and exclusion criterion evaluations. Lastly, we clarified how the binary and probabilistic decisions are related to each other, e.g. the probabilistic decision means the probability of an inclusion criteria being met, and for exclusion criteria, this means the probability of the exclusion criterion being met. This was to mitigate a potential misunderstanding of the LLMs where the LLM could misunderstand the probability of exclusion as "how certain the LLM is about its answer". If the LLM output JavaScript Object Notation (JSON) that did not meet the output schema, Pydantic AI automatically retried the request and instructed the LLM to correct the mistake. Because the overall decision field was removed for RQ2 and RQ4, the overall decision was determined by the results of the inclusion and exclusion criterion. Each secondary study in the SESR-Eval dataset contained inclusion and exclusion criteria, which we converted into YAML Ainât Markup Language (YAML) files and compared against the original inclusion and exclusion criteria to ensure correctness. The resulting YAML files were consumed by the experiment scripts. Each secondary study had rules, by which the inclusion criteria (IC) and exclusion criteria (EC) are evaluated. This information was also provided to the LLM as a boolean formula, e.g. (IC1 AND IC2 AND IC3) AND NOT (EC1 OR EC2) Both versions of the user prompts and output structures are included in the research artifact (Huotala et al., 2026b). For RQ2, we also computed accuracy, precision, recall, F1, MCC, and lost evidence. 4.5 LLM agreement - Overall decision and criteria (RQ3.1 & RQ3.2) To study the agreement between LLMs in overall decision RQ3.1 and criteria RQ3.2, we first retrieved the raw responses of the LLMs originally collected for RQ1.1. These raw responses contained the original screening responses, which we parsed as objects. From the objects, we retrieved the binary_decision field, which indicates the LLMs overall screening decision. For studying the agreement across the inclusion and exclusion criteria in RQ3.2, we extracted the results corresponding to the inclusion and/or exclusion criteria, that were also found from the raw responses originally collected for RQ1.1. 4.6 Modifying criteria (RQ4) Table 5: Criteria modification process. # Description 1 Expand abbreviations and acronyms 4.6.1 2 Split compound criteria into multiple rules where possible 4.6.2 3 Remove criteria not verifiable from title/abstract (e.g., publication year, venue) 4.6.3 4 Convert negations to affirmative form 4.6.4 5 Preserve original wording â minimize rewording 4.6.5 6 If a rule contains vague terms, search the SR for their definition or description 4.6.6 Our motivation behind modifying criteria in RQ4 was that improving poorly formatted, unclear, or ambiguous criteria may decrease the likelihood of an incorrect overall or criterion screening decision. As this part was somewhat experimental and as modifying criteria takes a lot of time, we focused on the secondary studies that LLMs had challenges making the correct decision for includes and excludes. We began by analyzing the criterion-level agreements from Online resource 3 (Huotala et al., 2026b) and looking at Table 8 to see, which secondary studies had most disagreement â€. We then selected seven secondary studies from Table 8 (RodrĂguez-PĂ©rez et al., 2021; Bjarnason et al., 2023; Zhang and Li, 2020; Tebes et al., 2020; Alonso et al., 2023; Lenarduzzi et al., 2021; Deckers and Lago, 2022). For this experiment, we selected all new LLMs from Table 4 â€, as we wanted to make the results fully comparable with RQ2. The criteria modification process â„ was performed by the third author. The second author verified the third authorâs criteria modification suggestions. After the criteria modification process was finished, we then re-benchmarked the primary studies using eight LLMs (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, GLM 5.1, Kimi K2.6, Grok 4.20 Beta, Qwen 3.6 Plus, and DeepSeek V4 Pro) on the 5% sampled data, filtered to the selected seven secondary studies (N=448), using the V2 user prompt and output schema. For RQ4, we computed accuracy, precision, recall, F1, MCC, Gwetâs First-order Agreement Coefficient (Gwetâs AC1), and lost evidence. Next, we describe each item of the criteria modification process in detail, in Subsections 4.6.1 to 4.6.6. The following modifications were made to the criteria (Table 5): 4.6.1 Expand abbreviations and acronyms Abbreviations and acronyms were expanded, as acronyms can be misleading, e.g. if the LLM fails to understand the context of the abbreviation or acronym. For instance, continuous integration (CI) and confidence interval (CI) have matching acronyms, but mean totally different things. This might lead the LLM to misclassify a paper that mentions continuous integration in the abstract, but the LLM interprets the acronym as confidence interval. 4.6.2 Split compound criteria into multiple rules where possible Long criteria that contain multiple different rules were split into separate, simpler rules so each can be evaluated clearly and independently by the LLM. 4.6.3 Remove criteria not verifiable from title/abstract (e.g., publication year, venue) If the LLM cannot determine a criterionâs outcome by the paperâs title/abstract, it can lead to false negatives or false positives. Paper abstracts rarely contain information regarding e.g. publication year or venue. 4.6.4 Convert negations to affirmative form We converted negations to affirmative form, so they describe the desired criteria directly. Affirmative rules are easier to interpret and we have witnessed LLMs tripping over negation, especially when present in exclusion rules (MĂ€ntylĂ€ et al., 2026). 4.6.5 Preserve original wording â minimize rewording We kept the criteria as close as possible to their original wording to preserve the intent of the underlying SR. The rewording was limited to changes needed for consistency, clarity, and LLM interpretability, as larger changes to the criterion could alter the meaning of it. 4.6.6 If a rule contains vague terms, search the SR for their definition or description We searched the SR for definitions and explanations, or contextual descriptions of vague terms used in the criteria. When available, these definitions were incorporated into the criteria to make them more precise. 4.7 Statistical methods used Next, we describe each statistical method we used and motivate their use. For all the metrics (excluding regression model), we used the confusion matrices as a single source of truth, and include them in our research artifact (Huotala et al., 2026b) as recommended by Madeyski et al. (2026). True positives (TP) indicate correctly included studies, while true negatives (TN) indicate correctly excluded studies. On the contrary, false negatives (FN) are studies incorrectly excluded and false positives (FP) are studies incorrectly included. 4.7.1 Accuracy, Precision, Recall, F1 In our experiments we reported accuracy, precision, recall, and F1 scores to make the screening results comparable with prior work, and the corresponding metrics are industry-standard metrics for reporting empirical software engineering research. Accuracy was used to measure the general correctness of the LLMs, and precision and recall were used to measure the LLMâs correctness in correctly included studies, while F1 presents precision and recall as a harmonic mean (Manning et al., 2008). Accuracy, precision, recall, and F1 are computed as follows: Aâcâcâuârâaâcây=TâP+TâNnAccuracy= TP+TNn (1) Where n is the total number of screened studies (included or excluded). Pârâeâcâiâsâiâoân=TâPTâP+FâPPrecision= TPTP+FP (2) Râeâcâaâlâl=TâPTâP+FâNRecall= TPTP+FN (3) Fâ1=2ĂTâP2ĂTâP+FâP+FâNF1= 2Ă TP2Ă TP+FP+FN (4) 4.7.2 MCC We reported the Matthews Correlation Coefficient (MCC), as it was recommended by Madeyski et al. (2026) for ranking, and because the SESR-Eval dataset (Huotala et al., 2025a) contains secondary studies that sample far more excluded than included papers. The SESR-Eval dataset by Huotala et al. (2025a) used in our experiments contains 34,528 papers, from which 4,197 were labeled as included, and 30,331 as excluded. This gives an I:E ratio of 12:88, which makes the excluded papers a highly prevalent category. The same situation holds for the 5% sampled data, which also had a similar level of class imbalance (21:79). MCC uses the whole confusion matrix and behaves adequately with class imbalance, which the F1 score does not. MCC is computed as follows: MâCâC=TâPĂTâNâFâPĂFâN(TâP+FâP)â(TâP+FâN)â(TâN+FâP)â(TâN+FâN)MCC= TPĂ TN-FPĂ FN (TP+FP)(TP+FN)(TN+FP)(TN+FN) (5) 4.7.3 Lost evidence Lost evidence (FN) is the number and percentage of relevant studies that the LLM excludes incorrectly (false negatives). In the context of SRs, missing relevant papers is more costly than including irrelevant ones, as the irrelevant papers are usually caught in the full-text screening phase (Kitchenham and Charters, 2007). Reporting lost evidence is useful because two LLMs can have a similar F1 score, but the lost evidence of one LLM can be significantly higher than the other. The confusion matrices of each experiment are included in our research artifact (Huotala et al., 2026b). 4.7.4 LLM Agreement - Gwetâs AC1 Gwetâs AC1 (Gwet, 2008) was used instead of Cohenâs kappa as our agreement metric. Cohenâs kappa is well known to suffer from the "prevalence problem"1111 11 Also known as the prevalence paradox and prevalence effect making it unsuited for datasets with high category imbalance. In our data, we have secondary studies where nearly all decisions are includes or excludes. Should there be any mistakes in LLMsâ screening decisions in those studies, Cohenâs kappa would collapse close to zero as it overestimates chance agreement from the base rate. As our data has a prevalence problem, Gwetâs AC1 becomes the most suitable metric for it. Gwetâs AC1 does not artificially deflate the agreement estimate simply because one category was highly prevalent (Wongpakaran et al., 2013). Gwetâs AC1 is computed as follows: AâCâ1=paâpe1âpeAC1= p_a-p_e1-p_e (6) where pap_a is the percentage of observed agreement and pep_e is the probability of chance agreement. 4.7.5 Bootstrap confidence intervals and regression model for statistical significance Our main metrics, F1 and MCC, come from machine learning and are not typically accompanied by statistical tests. Because these are composite metrics, traditional statistical tests like the t-test are not applicable. Therefore, to test the statistical significance of our results, we reported the 95% percentile bootstrap confidence intervals (Efron and Tibshirani, 1994) for precision, recall, F1, MCC, lost evidence, and Gwetâs AC1. We used B=2000B=2000 samples. Because the secondary studies in the SESR-Eval dataset differ significantly in the size and I:E ratio, we resampled papers with replacement within each secondary study, so that every resample preserved the composition of the dataset. The 2.5th and 97.5th percentiles of each metricâs bootstrap distribution formed the 95% confidence interval. For differences between conditions evaluated on the same primary studies, we computed the difference on each resample. This yielded paired confidence intervals, whose difference we considered meaningful when its confidence interval excluded 0. To compare the screening performance between all LLMs and secondary studies, we chose models benchmarked for RQ1.2 (see Table 4). Then, we fit a logistic regression model that predicted the correctness of LLMsâ screening decisions based on the LLM and the primary study. The logistic regression model was fitted and the odds ratios were calculated using the R programming language and the methods glm and coef. We chose the model and secondary study with highest overall probability of correct prediction for the reference classes. 4.8 AI disclosure Claude Code was used to support the development of scripts that we used to run the LLMs and to generate and format tables. All outputs generated by the tool were independently reviewed and validated by the authors, who retain full responsibility for the final scripts, tables, and other associated outputs. 5 Results 5.1 RQ1.1: Paper screening on full dataset Table 6: Screening performance metrics of LLMs across primary studies (RQ1.1). The models above the double line are from Huotala et al. (2025a). Model Accuracy Precision Recall F1 MCC Lost Evidence o3-mini 0.86 [0.86, 0.86] 0.43 [0.42, 0.45] 0.52 [0.51, 0.53] 0.47 [0.46, 0.48] 0.39 [0.38, 0.41] 2018 (48%) [1944, 2095] GPT-4o 0.84 [0.83, 0.84] 0.40 [0.39, 0.41] 0.66 [0.64, 0.67] 0.50 [0.48, 0.51] 0.42 [0.41, 0.44] 1443 (34%) [1378, 1506] GPT-4.1 0.83 [0.83, 0.83] 0.38 [0.37, 0.39] 0.63 [0.61, 0.64] 0.47 [0.46, 0.48] 0.39 [0.38, 0.41] 1565 (37%) [1499, 1633] GPT-4.1 mini 0.90 [0.89, 0.90] 0.60 [0.59, 0.62] 0.43 [0.42, 0.45] 0.50 [0.49, 0.52] 0.46 [0.44, 0.47] 2375 (57%) [2294, 2454] Claude 3.7 Sonnet 0.89 [0.89, 0.89] 0.56 [0.54, 0.58] 0.46 [0.45, 0.48] 0.51 [0.50, 0.52] 0.45 [0.44, 0.46] 2249 (54%) [2171, 2328] DeepSeek R1 0.89 [0.89, 0.89] 0.56 [0.55, 0.58] 0.42 [0.41, 0.43] 0.48 [0.47, 0.49] 0.43 [0.41, 0.44] 2433 (58%) [2348, 2517] Llama 4 Maverick 0.87 [0.86, 0.87] 0.47 [0.45, 0.48] 0.61 [0.59, 0.62] 0.53 [0.51, 0.54] 0.46 [0.44, 0.47] 1655 (39%) [1587, 1721] Average 0.867 0.486 0.532 0.494 0.429 1962.6 (47%) DeepSeek V4 Pro 0.87 [0.87, 0.88] 0.48 [0.47, 0.50] 0.47 [0.46, 0.49] 0.48 [0.46, 0.49] 0.41 [0.39, 0.42] 2214 (53%) [2135, 2290] Qwen 3.6 Plus 0.86 [0.86, 0.87] 0.45 [0.44, 0.46] 0.52 [0.50, 0.53] 0.48 [0.47, 0.49] 0.40 [0.39, 0.42] 2034 (48%) [1959, 2110] Average 0.869 0.466 0.494 0.479 0.405 2124.0 (51%) Table 7: Screening performance metrics of LLMs across secondary studies (RQ1.1). The models above the double line are from Huotala et al. (2025a). Model Accuracy Precision Recall F1 MCC Lost Evidence P95 time per paper (s) Cost per study (USD) o3-mini 0.73 [0.72, 0.74] 0.60 [0.58, 0.62] 0.49 [0.48, 0.51] 0.46 [0.44, 0.48]1 0.54 [0.52, 0.56]2 0.30 [0.29, 0.32] 84.1 (51%) [81.0, 87.3] 86.1 36.7â GPT-4o 0.73 [0.72, 0.74] 0.53 [0.51, 0.54] 0.66 [0.64, 0.67] 0.51 [0.49, 0.52]1 0.59 [0.57, 0.60]2 0.31 [0.29, 0.32] 60.1 (34%) [57.4, 62.8] 14.1 13.4â GPT-4.1 0.73 [0.72, 0.74] 0.53 [0.51, 0.54] 0.60 [0.58, 0.62] 0.48 [0.47, 0.50]1 0.56 [0.55, 0.57]2 0.29 [0.27, 0.31] 65.2 (40%) [62.5, 68.0] 24.7 12.1â GPT-4.1 mini 0.73 [0.72, 0.74] 0.62 [0.59, 0.64] 0.38 [0.36, 0.40] 0.41 [0.39, 0.42]1 0.47 [0.45, 0.49]2 0.28 [0.26, 0.30] 99.0 (62%) [95.6, 102.3] 10.5 2.2â Claude 3.7 Sonnet 0.73 [0.72, 0.74] 0.58 [0.56, 0.60] 0.45 [0.44, 0.47] 0.44 [0.43, 0.46]1 0.51 [0.49, 0.52]2 0.28 [0.26, 0.30] 93.7 (55%) [90.5, 97.0] 33.7 24.0â DeepSeek R1 0.72 [0.71, 0.73] 0.58 [0.55, 0.60] 0.41 [0.39, 0.43] 0.41 [0.39, 0.43]1 0.48 [0.46, 0.50]2 0.27 [0.25, 0.29] 101.4 (59%) [97.8, 104.9] 42.8 3.8â Llama 4 Maverick 0.74 [0.73, 0.75] 0.54 [0.52, 0.55] 0.61 [0.59, 0.62] 0.50 [0.48, 0.51]1 0.57 [0.56, 0.58]2 0.31 [0.29, 0.33] 69.0 (39%) [66.1, 71.7] 23.0 1.4â Average 0.731 0.566 0.515 0.4591 0.5312 0.294 81.8 (49%) DeepSeek V4 Pro 0.73 [0.73, 0.74] 0.62 [0.60, 0.64] 0.47 [0.45, 0.48] 0.45 [0.43, 0.46]1 0.53 [0.52, 0.55]2 0.31 [0.29, 0.33] 92.3 (53%) [89.0, 95.4] 103.0 12.1 Qwen 3.6 Plus 0.73 [0.72, 0.74] 0.59 [0.57, 0.62] 0.51 [0.49, 0.52] 0.46 [0.45, 0.48]1 0.55 [0.53, 0.56]2 0.31 [0.29, 0.33] 84.8 (49%) [81.6, 87.9] 94.7 11.2 Average 0.733 0.608 0.486 0.4561 0.5392 0.309 88.5 (51%) 1 Mean of the per-secondary-study F1 scores. 2 F1 computed from the averaged precision and recall. â Old models: costs previously recorded â only an estimate. Table 8: Accuracy, precision, recall, F1 score, MCC, and lost evidence for the secondary studies across all LLMs (RQ1.1, full dataset). Lost evidence is the share of relevant papers missed (false negatives), pooled across LLMs. MCC is N/A where it is undefined (all papers, or all predictions, belong to one class); the average MCC is taken over the remaining studies. Study Accuracy Precision Recall F1 MCC Lost evidence Casalaro et al. (2022) 0.93 0.27 0.74 0.39 0.42 26% Sharbaf et al. (2023) 0.69 0.73 0.68 0.70 0.39 32% Tambon et al. (2022) 0.87 0.74 0.46 0.57 0.52 54% RodrĂguez-PĂ©rez et al. (2021) 0.91 0.41 0.18 0.25 0.23 82% Bjarnason et al. (2023) 0.86 0.05 0.64 0.10 0.16 36% Kuutila et al. (2020) 0.94 0.39 0.40 0.39 0.36 60% Geraldi et al. (2020) 0.65 0.47 0.46 0.47 0.20 54% Zhang and Li (2020) 0.26 1.00 0.26 0.41 N/A 74% Tebes et al. (2020) 0.36 0.80 0.16 0.27 0.07 84% Ebrahimi et al. (2021) 0.62 0.59 0.78 0.67 0.25 22% Lewowski and Madeyski (2022) 0.91 0.56 0.64 0.60 0.55 36% Van Dinter et al. (2022) 0.70 0.45 0.72 0.55 0.37 28% Stradowski and Madeyski (2023) 0.83 0.89 0.82 0.85 0.64 18% Alonso et al. (2023) 0.23 0.77 0.15 0.25 -0.11 85% Somers et al. (2023) 0.71 0.59 0.45 0.51 0.32 55% Teixeira et al. (2017) 0.52 0.83 0.24 0.37 0.22 76% Lenarduzzi et al. (2021) 0.73 0.17 0.24 0.20 0.04 76% Pereira et al. (2021) 0.79 1.00 0.79 0.88 N/A 21% BariĆĄiÄ et al. (2022) 0.67 0.78 0.41 0.53 0.35 59% Deckers and Lago (2022) 0.78 0.19 0.81 0.31 0.32 19% Rani et al. (2023) 0.96 0.32 0.27 0.29 0.27 73% Zakeri-Nasrabadi et al. (2023) 0.89 0.29 0.68 0.41 0.40 32% Shevtsov et al. (2018) 0.85 0.36 0.55 0.44 0.36 45% Lin et al. (2022) 0.89 0.63 0.70 0.67 0.60 30% Average 0.73 0.55 0.51 0.46 0.32 49% We started our research by rerunning the full dataset of our prior SESR-Eval paper (Huotala et al., 2025a) with the models we selected, starting from the cheapest: DeepSeek V4 Pro and Qwen 3.6 Plus, see Table 2. We did not run the other models on the full dataset, as we estimated it would have cost $6,500; instead, we opted to run them only on the sampled data, see next section. Table 7 and Table 7 show the accuracy, precision, recall, F1, MCC, and lost evidence scores of the tested models for primary studies and secondary studies, respectively. We report results for both primary and secondary studies for the same reasons as for our prior work (Huotala et al., 2025a), as some secondary studies contain a large number of papers and therefore dominate the primary study scores. Screening performance across primary studies. Across primary studies (Table 7) we see that in terms of MCC, the two new LLMs (DeepSeek V4 Pro and Qwen 3.6 Plus) do not exceed the best-performing old LLMs (GPT-4.1 mini and Llama 4 Maverick, MCC=0.46) that were first benchmarked 18 months prior. The same is observed for F1 score, where DeepSeek V4 Pro and Qwen 3.6 Plus have exactly the same F1=0.48, while the old LLM with the best F1 score was Llama 4 Maverick (F1=0.53). In terms of lost evidence for primary studies, both DeepSeek V4 Pro (53%) and Qwen 3.6 Plus (48%) lose less evidence than GPT-4.1 mini (57%), Claude 3.7 Sonnet (54%) and DeepSeek R1 (58%), but do not reach the lost evidence levels of GPT-4o (34%) or GPT-4.1 (37%). Accuracy, precision, and recall for the two new LLMs are at about the same level as for the seven old LLMs. On average, the old LLMs perform marginally better than the new LLMs (old MCC = 0.429 vs. new MCC = 0.405). Screening performance across secondary studies. In Table 7, we can see that there is a four-way tie as the best-performing LLM in paper screening over the full dataset in terms of MCC â the two new LLMs (DeepSeek V4 Pro and Qwen 3.6 Plus) have the same MCC as GPT-4o and Llama 4 Maverick (MCC=0.31). If we look at screening performance between secondary studies (Table 8), the F1 score ranges between 0.10 and 0.88, MCC between -0.11 and 0.64 averaged over all LLMs. Lost evidence varies significantly per-study â the average is 49% while it ranges from 18% to 85%. For studies Zhang and Li (2020) and Pereira et al. (2021), MCC was not defined as they did not contain excluded studies at all (Table 3). Next, if we look at the more detailed two-dimensional table (Online resource 1) (Huotala et al., 2026b), the performance of DeepSeek V4 Pro and Qwen 3.6 Plus are fairly similar compared to the rest of the LLMs, where F1 for DeepSeek V4 Pro ranges from 0.08 to 0.86, and 0.07 to 0.86 for Qwen 3.6 Plus, respectively. Meanwhile, MCC for DeepSeek V4 Pro ranges from -0.27 to 0.70. For Qwen 3.6 Plus, the MCC ranges between -0.26 and 0.65. In terms of lost evidence for secondary studies (Table 7), both DeepSeek V4 Pro and Qwen 3.6 Plus lose 53% and 49% evidence on average, respectively. Compared to older LLMs, this is at almost the same level as o3-mini (51%), and Claude 3.7 Sonnet (55%). The new LLMs are not even close to GPT-4o, which loses only 34% of evidence on average. When looking at the average across secondary studies, the new LLMs perform marginally better than the old LLMs (old MCC = 0.294 vs. new MCC = 0.309). Looking at the cost per secondary study and the time taken to screen one paper (95th percentile), we see that both of the new models (DeepSeek V4 Pro = 103.0s, Qwen 3.6 Plus = 94.7s) are the two slowest models around. The previous slowest model was o3-mini with a screening speed of 86.1s per paper. The fastest LLM to screen one paper is GPT-4.1 mini (10.5s), which is almost ten times faster. In addition, the two models are about as expensive as GPT-4o and GPT-4.1, where DeepSeek V4 Pro costs 12.1 dollars per secondary study, and Qwen 3.6 Plus cost 11.2 dollars per secondary study on average. Summary: RQ1.1 results show that the two new models (DeepSeek V4 Pro and Qwen 3.6 Plus) tested on full datasets are not clearly more accurate than the older ones. Average MCC is marginally higher across secondary studies (0.309 vs 0.294) but lower across primary studies (0.405 vs 0.429). Differences between secondary studies are still significant. 5.2 RQ1.2: Paper screening on sampled dataset Table 9: Screening performance metrics of LLMs across primary studies (RQ1.2). The models above the double line are from Huotala et al. (2025a). Model Accuracy Precision Recall F1 MCC Lost Evidence o3-mini 0.83 [0.81, 0.84] 0.60 [0.55, 0.66] 0.50 [0.45, 0.55] 0.55 [0.50, 0.59] 0.44 [0.39, 0.49] 179 (50%) [159, 200] GPT-4o 0.80 [0.78, 0.81] 0.51 [0.46, 0.55] 0.64 [0.60, 0.69] 0.57 [0.53, 0.60] 0.44 [0.39, 0.49] 128 (36%) [111, 145] GPT-4.1 0.80 [0.78, 0.82] 0.51 [0.47, 0.56] 0.61 [0.56, 0.65] 0.55 [0.51, 0.59] 0.43 [0.38, 0.48] 141 (39%) [123, 159] GPT-4.1 mini 0.84 [0.83, 0.85] 0.69 [0.63, 0.75] 0.41 [0.37, 0.46] 0.52 [0.47, 0.56] 0.45 [0.40, 0.50] 210 (59%) [189, 231] Claude 3.7 Sonnet 0.82 [0.80, 0.84] 0.59 [0.54, 0.65] 0.42 [0.38, 0.47] 0.49 [0.45, 0.54] 0.40 [0.35, 0.45] 207 (58%) [186, 228] DeepSeek R1 0.83 [0.81, 0.84] 0.61 [0.55, 0.67] 0.44 [0.40, 0.49] 0.51 [0.47, 0.56] 0.42 [0.36, 0.47] 200 (56%) [179, 222] Llama 4 Maverick 0.82 [0.80, 0.83] 0.56 [0.51, 0.61] 0.58 [0.53, 0.62] 0.57 [0.53, 0.61] 0.45 [0.40, 0.50] 152 (42%) [134, 171] Average 0.818 0.583 0.514 0.537 0.433 173.9 (49%) Claude Opus 4.7 0.85 [0.83, 0.86] 0.75 [0.69, 0.81] 0.38 [0.34, 0.43] 0.51 [0.46, 0.55] 0.46 [0.41, 0.51] 221 (62%) [201, 242] DeepSeek V4 Pro 0.83 [0.81, 0.84] 0.61 [0.56, 0.67] 0.44 [0.40, 0.49] 0.51 [0.47, 0.55] 0.42 [0.37, 0.47] 200 (56%) [180, 220] Gemini 3.1 Pro 0.83 [0.82, 0.85] 0.67 [0.61, 0.73] 0.39 [0.35, 0.44] 0.50 [0.45, 0.54] 0.43 [0.38, 0.48] 217 (61%) [196, 239] GLM 5.1 0.83 [0.82, 0.85] 0.71 [0.65, 0.77] 0.33 [0.29, 0.38] 0.45 [0.41, 0.50] 0.41 [0.36, 0.46] 239 (67%) [216, 261] GPT-5.5 0.82 [0.80, 0.84] 0.60 [0.54, 0.66] 0.41 [0.36, 0.45] 0.48 [0.44, 0.53] 0.39 [0.34, 0.44] 213 (59%) [192, 234] Grok 4.20 Beta 0.82 [0.80, 0.83] 0.58 [0.53, 0.63] 0.40 [0.36, 0.45] 0.48 [0.43, 0.52] 0.38 [0.33, 0.43] 214 (60%) [193, 235] Kimi K2.6 0.83 [0.81, 0.84] 0.62 [0.56, 0.67] 0.42 [0.37, 0.46] 0.50 [0.45, 0.54] 0.41 [0.36, 0.46] 209 (58%) [188, 230] Qwen 3.6 Plus 0.82 [0.80, 0.83] 0.57 [0.51, 0.62] 0.48 [0.44, 0.53] 0.52 [0.48, 0.56] 0.41 [0.36, 0.46] 185 (52%) [165, 206] Average 0.827 0.638 0.407 0.493 0.412 212.3 (59%) Lost Evidence CI is for the false-negative count. Table 10: Screening performance metrics of LLMs across secondary studies (RQ1.2). Model Accuracy Precision Recall F1 MCC Lost Evidence P95 time per paper (s) Cost per sample (USD) o3-mini 0.75 [0.73, 0.78] 0.68 [0.59, 0.73] 0.57 [0.46, 0.61] 0.53 [0.44, 0.57]1 0.62 [0.52, 0.65]2 0.40 [0.31, 0.44] 7.5 (43%) [6.6, 8.3] 86.1 1.82â GPT-4o 0.72 [0.70, 0.75] 0.53 [0.48, 0.58] 0.74 [0.63, 0.77] 0.53 [0.47, 0.57]1 0.62 [0.55, 0.65]2 0.35 [0.29, 0.40] 5.3 (26%) [4.6, 6.0] 14.1 0.67â GPT-4.1 0.74 [0.71, 0.76] 0.58 [0.50, 0.62] 0.69 [0.57, 0.73] 0.53 [0.45, 0.57]1 0.63 [0.54, 0.66]2 0.36 [0.28, 0.40] 5.9 (31%) [5.1, 6.6] 24.7 0.60â GPT-4.1 mini 0.75 [0.72, 0.77] 0.70 [0.56, 0.74] 0.44 [0.35, 0.49] 0.45 [0.37, 0.49]1 0.54 [0.44, 0.58]2 0.35 [0.27, 0.40] 8.8 (56%) [7.9, 9.6] 10.5 0.11â Claude 3.7 Sonnet 0.73 [0.70, 0.75] 0.53 [0.45, 0.61] 0.45 [0.36, 0.49] 0.42 [0.35, 0.46]1 0.49 [0.41, 0.53]2 0.27 [0.20, 0.32] 8.6 (55%) [7.8, 9.5] 33.7 1.20â DeepSeek R1 0.73 [0.71, 0.76] 0.59 [0.49, 0.64] 0.53 [0.41, 0.56] 0.47 [0.38, 0.51]1 0.56 [0.46, 0.59]2 0.34 [0.25, 0.38] 8.3 (47%) [7.5, 9.3] 42.8 0.19â Llama 4 Maverick 0.74 [0.72, 0.77] 0.56 [0.51, 0.60] 0.67 [0.55, 0.70] 0.51 [0.46, 0.55]1 0.61 [0.54, 0.64]2 0.35 [0.28, 0.40] 6.3 (33%) [5.6, 7.1] 23.0 0.07â Average 0.738 0.596 0.583 0.4931 0.5802 0.347 7.2 (42%) Claude Opus 4.7 0.74 [0.72, 0.77] 0.79 [0.61, 0.81] 0.47 [0.35, 0.52] 0.49 [0.39, 0.53]1 0.59 [0.46, 0.62]2 0.42 [0.32, 0.46] 9.2 (53%) [8.4, 10.1] 15.6 2.75 DeepSeek V4 Pro 0.74 [0.72, 0.76] 0.66 [0.55, 0.70] 0.54 [0.42, 0.58] 0.50 [0.42, 0.54]1 0.59 [0.49, 0.62]2 0.38 [0.29, 0.42] 8.3 (46%) [7.5, 9.2] 103.0 0.60 Gemini 3.1 Pro 0.74 [0.72, 0.76] 0.65 [0.52, 0.68] 0.44 [0.33, 0.48] 0.45 [0.37, 0.49]1 0.52 [0.41, 0.55]2 0.35 [0.26, 0.39] 9.0 (56%) [8.2, 10.0] 14.3 0.93 GLM 5.1 0.73 [0.71, 0.76] 0.68 [0.51, 0.71] 0.40 [0.29, 0.44] 0.41 [0.32, 0.45]1 0.50 [0.38, 0.54]2 0.34 [0.24, 0.38] 10.0 (60%) [9.0, 10.9] 51.2 0.50 GPT-5.5 0.73 [0.71, 0.75] 0.65 [0.53, 0.70] 0.47 [0.36, 0.52] 0.46 [0.37, 0.49]1 0.54 [0.43, 0.58]2 0.35 [0.26, 0.40] 8.9 (53%) [8.0, 9.8] 17.6 1.83 Grok 4.20 Beta 0.73 [0.71, 0.75] 0.69 [0.58, 0.74] 0.47 [0.35, 0.51] 0.47 [0.38, 0.51]1 0.56 [0.45, 0.59]2 0.36 [0.26, 0.40] 8.9 (53%) [8.0, 9.8] 53.0 0.61 Kimi K2.6 0.74 [0.72, 0.76] 0.63 [0.52, 0.67] 0.50 [0.38, 0.54] 0.47 [0.38, 0.50]1 0.56 [0.45, 0.59]2 0.35 [0.26, 0.39] 8.7 (50%) [7.8, 9.6] 225.6 0.66 Qwen 3.6 Plus 0.73 [0.70, 0.75] 0.63 [0.53, 0.68] 0.57 [0.45, 0.60] 0.50 [0.42, 0.54]1 0.60 [0.50, 0.63]2 0.37 [0.28, 0.41] 7.7 (43%) [6.9, 8.6] 94.7 0.55 Average 0.736 0.672 0.481 0.4691 0.5582 0.365 8.8 (52%) 1 Mean of the per-secondary-study F1 scores. 2 F1 computed from the averaged precision and recall. â Old models: costs previously recorded â only an estimate; Scaled to the sample by the modelâs token-based cost per paper. Table 11: Accuracy, precision, recall, F1 score, MCC, and lost evidence for the secondary studies across all LLMs (RQ1.2, 5% sample). Lost evidence is the share of relevant papers missed (false negatives). MCC is N/A where it is undefined (all sampled papers, or all predictions, belong to one class). Study Accuracy Precision Recall F1 MCC Lost evidence Casalaro et al. (2022) 0.94 [0.90, 0.97] 0.25 [0.00, 0.51] 0.67 [0.00, 0.93] 0.37 [0.00, 0.59] 0.38 [0.17, 0.59] 33% [0, 47] Sharbaf et al. (2023) 0.76 [0.66, 0.86] 0.80 [0.62, 0.92] 0.65 [0.48, 0.81] 0.72 [0.56, 0.83] 0.52 [0.32, 0.70] 35% [19, 52] Tambon et al. (2022) 0.82 [0.75, 0.89] 0.81 [0.56, 0.95] 0.34 [0.19, 0.51] 0.48 [0.29, 0.64] 0.45 [0.26, 0.59] 66% [49, 81] RodrĂguez-PĂ©rez et al. (2021) 0.94 [0.91, 0.97] 0.53 [0.17, 0.86] 0.23 [0.08, 0.37] 0.32 [0.11, 0.46] 0.32 [0.11, 0.48] 77% [63, 92] Bjarnason et al. (2023) 0.83 [0.79, 0.88] 0.04 [0.00, 0.12] 0.87 [0.00, 0.87] 0.07 [0.00, 0.21] 0.16 [0.14, 0.30] 13% [0, 13] Kuutila et al. (2020) 0.95 [0.92, 0.97] 0.51 [0.23, 0.75] 0.58 [0.29, 0.87] 0.55 [0.27, 0.73] 0.52 [0.25, 0.72] 42% [13, 71] Geraldi et al. (2020) 0.73 [0.60, 0.85] 0.50 [0.17, 0.81] 0.44 [0.16, 0.73] 0.47 [0.17, 0.68] 0.29 [-0.03, 0.58] 56% [27, 84] Zhang and Li (2020) 0.18 [0.08, 0.30] 1.00 [1.00, 1.00] 0.18 [0.08, 0.30] 0.30 [0.14, 0.46] N/A 82% [70, 92] Tebes et al. (2020) 0.31 [0.20, 0.42] 0.83 [0.57, 1.00] 0.15 [0.08, 0.24] 0.26 [0.14, 0.37] 0.05 [-0.18, 0.21] 85% [77, 92] Ebrahimi et al. (2021) 0.62 [0.46, 0.76] 0.56 [0.32, 0.76] 0.80 [0.63, 0.96] 0.66 [0.44, 0.80] 0.27 [0.00, 0.52] 20% [4, 37] Lewowski and Madeyski (2022) 0.93 [0.89, 0.96] 0.65 [0.37, 0.82] 0.56 [0.32, 0.81] 0.60 [0.36, 0.77] 0.56 [0.33, 0.74] 44% [19, 68] Van Dinter et al. (2022) 0.84 [0.77, 0.90] 0.55 [0.31, 0.75] 0.74 [0.55, 0.92] 0.63 [0.42, 0.77] 0.54 [0.34, 0.71] 26% [8, 45] Stradowski and Madeyski (2023) 0.85 [0.78, 0.91] 0.89 [0.81, 0.95] 0.85 [0.76, 0.93] 0.87 [0.80, 0.92] 0.69 [0.55, 0.81] 15% [7, 24] Alonso et al. (2023) 0.23 [0.12, 0.35] 0.58 [0.25, 0.94] 0.09 [0.03, 0.17] 0.15 [0.05, 0.28] -0.20 [-0.44, 0.06] 91% [83, 97] Somers et al. (2023) 0.75 [0.66, 0.83] 0.52 [0.28, 0.73] 0.42 [0.27, 0.58] 0.46 [0.29, 0.61] 0.30 [0.12, 0.48] 58% [42, 73] Teixeira et al. (2017) 0.51 [0.38, 0.65] 0.92 [0.79, 0.99] 0.25 [0.13, 0.38] 0.39 [0.22, 0.55] 0.27 [0.13, 0.41] 75% [62, 87] Lenarduzzi et al. (2021) 0.79 [0.69, 0.87] 0.30 [0.03, 0.57] 0.33 [0.04, 0.71] 0.32 [0.04, 0.57] 0.19 [-0.09, 0.48] 67% [28, 96] Pereira et al. (2021) 0.71 [0.56, 0.86] 1.00 [1.00, 1.00] 0.71 [0.56, 0.86] 0.83 [0.72, 0.92] N/A 29% [14, 44] BariĆĄiÄ et al. (2022) 0.62 [0.49, 0.74] 0.93 [0.83, 0.99] 0.27 [0.17, 0.39] 0.42 [0.28, 0.55] 0.35 [0.24, 0.46] 73% [61, 83] Deckers and Lago (2022) 0.73 [0.63, 0.83] 0.18 [0.00, 0.38] 0.76 [0.00, 0.87] 0.29 [0.00, 0.51] 0.27 [0.11, 0.44] 24% [0, 40] Rani et al. (2023) 0.99 [0.98, 1.00] 0.54 [0.00, 0.83] 1.00 [0.00, 1.00] 0.70 [0.00, 0.91] 0.73 [0.62, 0.92] 0% [0, 0] Zakeri-Nasrabadi et al. (2023) 0.91 [0.88, 0.93] 0.38 [0.21, 0.54] 0.60 [0.41, 0.77] 0.46 [0.28, 0.60] 0.43 [0.26, 0.57] 40% [23, 59] Shevtsov et al. (2018) 0.85 [0.79, 0.91] 0.58 [0.34, 0.76] 0.49 [0.30, 0.68] 0.53 [0.34, 0.68] 0.45 [0.24, 0.61] 51% [32, 70] Lin et al. (2022) 0.91 [0.86, 0.95] 0.67 [0.38, 0.84] 0.70 [0.41, 0.98] 0.69 [0.42, 0.84] 0.63 [0.36, 0.81] 30% [2, 59] Average 0.74 [0.72, 0.75] 0.60 [0.54, 0.64] 0.53 [0.43, 0.56] 0.48 [0.42, 0.51] 0.37 [0.32, 0.42] 47% [43, 50] Table 12: Logistic regression model predicting the correctness of a single screening decision with the LLM and the secondary study. Models below the double horizontal line are from Huotala et al. (2025a). Est. std.err z p Odds ratio (OR) (Intercept) 4.9956 0.2881 17.342 < 2e-16 *** 147.76 Reference - Claude Opus 4.7 GPT-5.5 -0.2348 0.1035 -2.269 0.02326 * 0.79 Grok 4.20 Beta -0.2747 0.1030 -2.666 0.00766 ** 0.76 Kimi K2.6 -0.1941 0.1040 -1.867 0.06186 . 0.82 Qwen 3.6 Plus -0.2747 0.1030 -2.666 0.00766 ** 0.76 Gemini 3.1 Pro -0.1102 0.1050 -1.050 0.29393 0.90 DeepSeek V4 Pro -0.1890 0.1040 -1.817 0.06926 . 0.83 GLM 5.1 -0.1209 0.1049 -1.153 0.24899 0.89 Claude 3.7 Sonnet -0.2348 0.1035 -2.269 0.02326 * 0.79 DeepSeek R1 -0.1890 0.1040 -1.817 0.06926 . 0.83 GPT-4.1 -0.4177 0.1014 -4.118 3.83e-05 *** 0.66 GPT-4.1 mini -0.0559 0.1057 -0.529 0.59705 0.95 GPT-4o -0.4361 0.1013 -4.307 1.66e-05 *** 0.65 Llama 4 Maverick -0.2499 0.1033 -2.419 0.01558 * 0.78 o3-mini -0.1683 0.1043 -1.614 0.10657 0.85 Reference - Rani et al. (2023) Casalaro et al. (2022) -2.0906 0.2964 -7.053 1.75e-12 *** 0.124 Sharbaf et al. (2023) -3.6242 0.2942 -12.320 < 2e-16 *** 0.027 Tambon et al. (2022) -3.2768 0.2894 -11.324 < 2e-16 *** 0.038 RodrĂguez-PĂ©rez et al. (2021) -1.9675 0.2966 -6.634 3.28e-11 *** 0.140 Bjarnason et al. (2023) -3.1666 0.2847 -11.123 < 2e-16 *** 0.042 Kuutila et al. (2020) -1.8817 0.2944 -6.392 1.64e-10 *** 0.152 Geraldi et al. (2020) -3.7823 0.2982 -12.685 < 2e-16 *** 0.023 Zhang and Li (2020) -6.3278 0.3123 -20.262 < 2e-16 *** 0.002 Tebes et al. (2020) -5.5829 0.2893 -19.298 < 2e-16 *** 0.004 Ebrahimi et al. (2021) -4.3087 0.2989 -14.413 < 2e-16 *** 0.013 Lewowski and Madeyski (2022) -2.2296 0.2971 -7.505 6.14e-14 *** 0.108 Van Dinter et al. (2022) -3.1357 0.2944 -10.650 < 2e-16 *** 0.043 Stradowski and Madeyski (2023) -3.0547 0.2901 -10.529 < 2e-16 *** 0.047 Alonso et al. (2023) -6.0230 0.2980 -20.208 < 2e-16 *** 0.002 Somers et al. (2023) -3.7030 0.2920 -12.683 < 2e-16 *** 0.025 Teixeira et al. (2017) -4.7432 0.2900 -16.355 < 2e-16 *** 0.009 Lenarduzzi et al. (2021) -3.4868 0.2958 -11.790 < 2e-16 *** 0.031 Pereira et al. (2021) -3.8763 0.3078 -12.594 < 2e-16 *** 0.021 BariĆĄiÄ et al. (2022) -4.3035 0.2907 -14.806 < 2e-16 *** 0.014 Deckers and Lago (2022) -3.7793 0.2930 -12.896 < 2e-16 *** 0.023 Zakeri-Nasrabadi et al. (2023) -2.4996 0.2848 -8.777 < 2e-16 *** 0.082 Shevtsov et al. (2018) -3.0345 0.2891 -10.496 < 2e-16 *** 0.048 Lin et al. (2022) -2.4916 0.3004 -8.294 < 2e-16 *** 0.083 Here, we use the 5% sampled dataset (SESR-Eval-Mini) for all the models we selected for RQ1.2 (Table 4). The LLMs benchmarked on the SESR-Eval-Mini show similar screening performance figures as on the full dataset1212 12 We can see that 95% confidence intervals are wider in sampled than in full dataset.. This result was somewhat expected, as the 5% sample was drawn using power-allocated sampling, which attempted to sample by balancing the proportional and equal allocation. However, if we look closer at the proportional and equal allocation of the sample (Table 3), we see that the power-allocated sampling had an I:E ratio of 21:79, while the original dataset had an I:E ratio of 12:88, as power-allocated sampling increased the proportional weight of smaller studies that had higher include ratio. Tables 9 and 10 show the accuracy, precision, recall, MCC, F1, and lost evidence scores of the models for primary studies and secondary studies, respectively. Table 12 shows the logistic regression model, which predicts the correctness of a single screening decision with the LLM and the secondary study. The regression model was computed for RQ1.2 only, as we wanted to compare all new LLMs with the old LLMs from our prior work (Huotala et al., 2025a). From the regression model we can see that Claude Opus 4.7 has the highest odds for a correct screening decision, which surpasses the old GPT-4.1 mini by a slight margin. The statistically significant intercept (log-odds = 4.9956, p < 2e-16) corresponds to high baseline odds of a correct screening decision prediction, when using the reference model (Claude Opus 4.7) with the reference study (Rani et al., 2023) (odds ratio â 147.76). Gemini 3.1 Pro, GLM 5.1, Kimi K2.6, DeepSeek R1, o3-mini and GPT-4.1 have roughly similar performance in correctness as the reference category, while the estimate is negative it is not statistically significant. The largest negative effects go for GPT-4o (ÎČ = -0.4361, p < 1.66e-05, Oddâs ratio (OR) â 0.65) and GPT-4.1 (ÎČ = -0.4177, p < 3.83e-05, OR â 0.66), which suggests that these two old models have substantially poorer screening performance in terms of correctness compared to the newer LLMs (and also some older LLMs). For secondary studies, LLMs predicted the correct screening decision most on the study by Rani et al. (2023), with significantly worse performance on all other secondary studies. Screening performance across secondary studies. The top-performing model with respect to the MCC score over secondary studies (n=24n=24, see Table 10) is Claude Opus 4.7 (new model) with MCC=0.420.42, followed by o3-mini (old model) with MCC=0.400.40, and in third place is DeepSeek V4 Pro (new model) with MCC=0.380.38. Then there is a bunch of models as all remaining models score between MCC=0.340.34 and MCC=0.370.37, with the exception of CLaude 3.7 Sonnet (MCC = 0.270.27). With respect to OpenAI, we can state that previously their models â GPT-4.1, GPT-4o, and o3-mini fared well when comparing across model providers. However, we now notice that GPT-5.5, which was their latest model, actually scores the same as GPT-4.1 mini and GPT-4o at MCC=0.350.35, indicating that those models have not progressed at all. As a coincidence, DeepSeek R1 (old) and DeepSeek V4 Pro (new) had the identical accuracy, precision, recall, F1, lost evidence scores and MCC (on primary study level), but a different MCC score on secondary study level. On average, we can see that the new LLMs perform marginally better (avg. new MCC = 0.365 vs. avg. old MCC = 0.347). Then for the F1 score across secondary studies, we can see that the old GPT family still dominates, sharing the top-3 place with a mean F1 score of 0.5310.53^1 shared between GPT-4.1, o3-mini, and GPT-4o. The best new models are DeepSeek V4 Pro and Qwen 3.6 Plus, which are tied for fifth place with an F1 score of 0.50, followed by Claude Opus 4.7 with F1=0.4910.49^1. Looking at the extended statistics from Online resource 3 (Huotala et al., 2026b), screening performance between secondary studies varies significantly, where MCC ranges from â0.27-0.27 to 1.001.00, and F1 from 0.000.00 to 1.001.00 for the new models. Looking at the average cost per secondary study, the cost is under a dollar for 11 LLMs. Claude Opus 4.7 is the most expensive model with a cost per secondary study of 2.75 dollars. The second most expensive new model is GPT-5.5, which costs 1.83 dollars per secondary study. The cost per secondary study for DeepSeek V4 Pro, GLM 5.1, Grok 4.20 Beta, Kimi K2.6, and Qwen 3.6 Plus are on the 0.60 dollar area. By far the cheapest model is Llama 4 Maverick, which costs 7 cents (0.07 dollars) per secondary study. Looking at the paper screening speed - Kimi K2.6 is the slowest (225.6s) with DeepSeek V4 Pro in second place (103.0s). Third is Qwen 3.6 Plus (94.7s). Claude Opus 4.7 (15.6s), Gemini 3.1 Pro (14.3s), and GPT-5.5 (17.6s) are fairly fast in paper screening, and are almost as fast as the old GPT-4o (14.1s), but no LLM was faster than the old GPT-4.1 mini (10.5s). Screening performance across primary studies. If we look at the primary-study averaged scores (Table 9), Claude Opus 4.7 still leads with MCC=0.460.46, followed by the old Llama 4 Maverick (old) (MCC=0.450.45) and GPT-4.1 mini (old) with MCC=0.450.45 in the shared second place. On the shared third place are o3-mini (old) and GPT-4o (old) with MCC of 0.440.44 for both. GPT-5.5 and Grok 4.20 Beta are the worst-performing in terms of primary-study averaged MCC, with 0.390.39 and 0.380.38, respectively. On average, we see the opposite result than with the secondary studies: the old LLMs perform marginally better (avg. old MCC = 0.433 vs. avg. new MCC = 0.412). However, the amount of lost evidence (Table 9 rightmost column), differs significantly from the old LLMs. Claude Opus 4.7 loses 221 studies, which is substantially more than GPT-4o (128), GPT-4.1 (141), and Llama 4 Maverick (152). A similar pattern is observed for DeepSeek V4 Pro (200), GLM 5.1 (239), GPT-5.5 (213) and Grok 4.20 Beta (214). Looking at the per-study metrics averaged over all LLMs (Table 11), MCC ranges between â0.20-0.20 and 0.720.72 and F1 score between 0.150.15 and 0.870.87, so differences between secondary studies are still quite significant. Summary: RQ1.2 results show that overall, the secondary studies have a larger effect than LLMs on screening performance. Model differences are modest: across secondary studies, the new models perform marginally better than old ones (avg. new MCC = 0.365 vs. avg. old MCC = 0.347), but when compared across primary studies, old models are still better (avg. new MCC = 0.412, avg. old MCC = 0.433). The new model Claude Opus 4.7 leads in the MCC metric by a narrow margin, and in logistic regression predicting the correctness of a screening decision. However, the old GPT-family still dominates when we look at the F1 score. New models tend to also lose more evidence (59% vs. 49%) than old models. Benchmarking on the SESR-Eval-Mini is cost-effective. 5.3 RQ2: Computing the overall decision from criterion-level decisions Table 13: Per-model accuracy, precision and recall, 5% sample (RQ2). Model Accuracy Precision Recall V1 V2 Î V1 V2 Î V1 V2 Î GPT-5.5 0.82 [0.80, 0.84] 0.83 [0.81, 0.84] 0.00 [-0.01, +0.02] 0.60 [0.54, 0.66] 0.64 [0.58, 0.71] +0.04 [0.00, +0.09] 0.41 [0.36, 0.45] 0.35 [0.31, 0.40] -0.05 [-0.09, -0.02] Claude Opus 4.7 0.85 [0.83, 0.86] 0.84 [0.83, 0.85] -0.01 [-0.01, 0.00] 0.75 [0.69, 0.81] 0.75 [0.68, 0.81] -0.01 [-0.05, +0.03] 0.38 [0.34, 0.43] 0.34 [0.30, 0.39] -0.04 [-0.07, -0.01] Gemini 3.1 Pro 0.83 [0.82, 0.85] 0.82 [0.81, 0.84] -0.01 [-0.02, 0.00] 0.67 [0.61, 0.73] 0.62 [0.56, 0.68] -0.05 [-0.09, -0.02] 0.39 [0.35, 0.44] 0.38 [0.34, 0.43] -0.01 [-0.04, +0.02] GLM 5.1 0.83 [0.82, 0.85] 0.82 [0.81, 0.84] -0.01 [-0.02, 0.00] 0.71 [0.65, 0.77] 0.66 [0.59, 0.73] -0.05 [-0.09, -0.01] 0.33 [0.29, 0.38] 0.31 [0.27, 0.35] -0.03 [-0.05, 0.00] Kimi K2.6 0.83 [0.81, 0.84] 0.82 [0.81, 0.84] 0.00 [-0.02, +0.01] 0.62 [0.56, 0.67] 0.60 [0.54, 0.66] -0.02 [-0.06, +0.03] 0.42 [0.37, 0.46] 0.43 [0.38, 0.47] +0.01 [-0.03, +0.05] Grok 4.20 Beta 0.82 [0.80, 0.83] 0.80 [0.79, 0.82] -0.01 [-0.02, 0.00] 0.58 [0.53, 0.63] 0.54 [0.48, 0.59] -0.04 [-0.08, -0.01] 0.40 [0.36, 0.45] 0.37 [0.32, 0.41] -0.04 [-0.07, 0.00] Qwen 3.6 Plus 0.82 [0.80, 0.83] 0.82 [0.80, 0.83] 0.00 [-0.01, +0.01] 0.57 [0.51, 0.62] 0.58 [0.53, 0.64] +0.02 [-0.01, +0.05] 0.48 [0.44, 0.53] 0.45 [0.40, 0.49] -0.03 [-0.06, -0.01] DeepSeek V4 Pro 0.83 [0.81, 0.84] 0.82 [0.80, 0.84] -0.01 [-0.02, 0.00] 0.61 [0.56, 0.67] 0.59 [0.54, 0.65] -0.02 [-0.05, +0.02] 0.44 [0.40, 0.49] 0.41 [0.37, 0.46] -0.03 [-0.06, 0.00] Avg. 0.83 [0.81, 0.84] 0.82 [0.81, 0.84] -0.01 [-0.01, 0.00] 0.64 [0.59, 0.69] 0.62 [0.57, 0.67] -0.02 [-0.03, 0.00] 0.41 [0.37, 0.45] 0.38 [0.34, 0.42] -0.03 [-0.04, -0.01] Table 14: Per-model F1, MCC and lost evidence, 5% sample (RQ2). Model F1 MCC Lost Evidence (FN) V1 V2 Î V1 V2 Î V1 V2 Î GPT-5.5 0.48 [0.44, 0.53] 0.45 [0.41, 0.50] -0.03 [-0.07, +0.01] 0.39 [0.34, 0.44] 0.38 [0.33, 0.44] -0.01 [-0.05, +0.03] 213 [192, 234] 232 [209, 256] +19 [+6, +33] Claude Opus 4.7 0.51 [0.46, 0.55] 0.47 [0.42, 0.52] -0.04 [-0.07, -0.01] 0.46 [0.41, 0.51] 0.43 [0.38, 0.48] -0.03 [-0.06, 0.00] 221 [201, 242] 235 [213, 257] +14 [+4, +25] Gemini 3.1 Pro 0.50 [0.45, 0.54] 0.47 [0.43, 0.52] -0.03 [-0.05, 0.00] 0.43 [0.38, 0.48] 0.39 [0.34, 0.44] -0.04 [-0.07, -0.01] 217 [196, 239] 222 [201, 244] +5 [-6, +15] GLM 5.1 0.45 [0.41, 0.50] 0.42 [0.37, 0.47] -0.03 [-0.06, -0.01] 0.41 [0.36, 0.46] 0.36 [0.31, 0.42] -0.04 [-0.07, -0.01] 239 [216, 261] 248 [225, 271] +9 [+1, +18] Kimi K2.6 0.50 [0.45, 0.54] 0.50 [0.45, 0.54] 0.00 [-0.04, +0.04] 0.41 [0.36, 0.46] 0.40 [0.35, 0.45] 0.00 [-0.05, +0.04] 209 [188, 230] 205 [183, 226] -4 [-19, +11] Grok 4.20 Beta 0.48 [0.43, 0.52] 0.44 [0.39, 0.48] -0.04 [-0.08, -0.01] 0.38 [0.33, 0.43] 0.33 [0.28, 0.38] -0.05 [-0.09, -0.01] 214 [193, 235] 227 [205, 250] +13 [0, +27] Qwen 3.6 Plus 0.52 [0.48, 0.56] 0.51 [0.47, 0.55] -0.01 [-0.04, +0.01] 0.41 [0.36, 0.46] 0.40 [0.36, 0.45] -0.01 [-0.04, +0.03] 185 [165, 206] 197 [176, 218] +12 [+2, +23] DeepSeek V4 Pro 0.51 [0.47, 0.55] 0.49 [0.44, 0.53] -0.02 [-0.05, 0.00] 0.42 [0.37, 0.47] 0.39 [0.34, 0.44] -0.03 [-0.06, +0.01] 200 [180, 220] 210 [190, 232] +10 [-1, +21] Avg. 0.49 [0.45, 0.53] 0.47 [0.43, 0.51] -0.03 [-0.04, -0.01] 0.41 [0.37, 0.45] 0.39 [0.35, 0.43] -0.02 [-0.04, -0.01] 212 [193, 232] 222 [203, 242] +10 [+4, +16] Table 14 shows per-model accuracy, precision and recall, and Table 14 the F1, MCC, and lost evidence on whether computing the overall decision from individual criteria is better than letting the LLM make the overall decision. Technically speaking computing the decision from the individual criteria is a more correct approach. However, deviating from that can allow weighting of the criteria to occur in the LLM that then makes the overall decision. We can see that computing the overall screening decision from criterion-level decisions decreases the average accuracy, precision, recall, F1, and MCC. For GPT-5.5, the accuracy increased by one percentage point (from 0.82 to 0.83). Lost evidence increases by +10 on average, and the only LLM that had less lost evidence was Kimi K2.6 (-4). Bootstrapping demonstrates that the reduction in accuracy, precision, and MCC for Gemini 3.1 Pro, GLM 5.1, and Grok 4.20 Beta are statistically significant as zero is not part of the 95% confidence interval. Recall reduction for GPT-5.5, Claude Opus 4.7, GLM 5.1, and Qwen 3.6 Plus are also statistically significant. The F1 score reduction is statistically significant for Claude Opus 4.7, GLM 5.1, and Grok 4.20 Beta. In terms of lost evidence, the increase in lost evidence is significant for most of the LLMs. Summary: RQ2 shows that computing overall decisions from individual criteria makes screening stricter but not more precise: recall fell (0.41 to 0.38) and there were more false negatives (+10 on average). Given that false negatives (lost evidence) are worse than false positives, this change cannot be recommended. 5.4 RQ3.1: LLM agreement - Overall decision Table 15: Gwetâs AC1 between LLMs on binary include/exclude decisions on the full dataset and on the 5% sample, with bootstrap confidence intervals (RQ3.1). 95% percentile bootstrap confidence intervals in brackets (B=2000B=2000; papers resampled with replacement within each secondary study). AC1 LLM pair Full dataset 5% sample Claude 3.7 â DeepSeek R1 0.911 [0.908, 0.915] 0.861 [0.840, 0.881] Claude 3.7 â DeepSeek V4 Pro 0.891 [0.887, 0.895] 0.847 [0.826, 0.868] Claude 3.7 â GPT-4.1 0.813 [0.807, 0.818] 0.768 [0.742, 0.793] Claude 3.7 â GPT-4.1 mini 0.909 [0.906, 0.913] 0.853 [0.833, 0.872] Claude 3.7 â GPT-4o 0.826 [0.821, 0.831] 0.771 [0.744, 0.797] Claude 3.7 â Llama 4 Mav. 0.874 [0.869, 0.878] 0.822 [0.797, 0.844] Claude 3.7 â o3-mini 0.862 [0.858, 0.866] 0.817 [0.794, 0.839] Claude 3.7 â Qwen 3.6 Plus 0.878 [0.873, 0.881] 0.833 [0.811, 0.854] DeepSeek R1 â DeepSeek V4 Pro 0.890 [0.886, 0.894] 0.860 [0.839, 0.879] DeepSeek R1 â GPT-4.1 0.814 [0.809, 0.820] 0.806 [0.781, 0.829] DeepSeek R1 â GPT-4.1 mini 0.925 [0.922, 0.928] 0.887 [0.868, 0.904] DeepSeek R1 â GPT-4o 0.829 [0.824, 0.834] 0.792 [0.766, 0.818] DeepSeek R1 â Llama 4 Mav. 0.871 [0.867, 0.876] 0.838 [0.815, 0.860] DeepSeek R1 â o3-mini 0.861 [0.856, 0.865] 0.832 [0.808, 0.853] DeepSeek R1 â Qwen 3.6 Plus 0.873 [0.869, 0.877] 0.849 [0.826, 0.870] DeepSeek V4 Pro â GPT-4.1 0.826 [0.821, 0.831] 0.799 [0.773, 0.823] DeepSeek V4 Pro â GPT-4.1 mini 0.896 [0.892, 0.899] 0.873 [0.854, 0.891] DeepSeek V4 Pro â GPT-4o 0.833 [0.828, 0.838] 0.775 [0.747, 0.801] DeepSeek V4 Pro â Llama 4 Mav. 0.864 [0.860, 0.869] 0.824 [0.800, 0.846] DeepSeek V4 Pro â o3-mini 0.918 [0.914, 0.921] 0.897 [0.879, 0.914] DeepSeek V4 Pro â Qwen 3.6 Plus 0.938 [0.935, 0.941] 0.921 [0.905, 0.936] GPT-4.1 â GPT-4.1 mini 0.823 [0.817, 0.828] 0.794 [0.769, 0.819] GPT-4.1 â GPT-4o 0.872 [0.867, 0.877] 0.853 [0.828, 0.875] GPT-4.1 â Llama 4 Mav. 0.840 [0.835, 0.845] 0.825 [0.800, 0.848] GPT-4.1 â o3-mini 0.838 [0.833, 0.843] 0.812 [0.786, 0.836] GPT-4.1 â Qwen 3.6 Plus 0.826 [0.820, 0.831] 0.814 [0.789, 0.838] GPT-4.1 mini â GPT-4o 0.826 [0.821, 0.831] 0.776 [0.750, 0.802] GPT-4.1 mini â Llama 4 Mav. 0.877 [0.873, 0.881] 0.832 [0.810, 0.853] GPT-4.1 mini â o3-mini 0.870 [0.866, 0.874] 0.852 [0.831, 0.872] GPT-4.1 mini â Qwen 3.6 Plus 0.879 [0.875, 0.882] 0.855 [0.835, 0.876] GPT-4o â Llama 4 Mav. 0.859 [0.855, 0.864] 0.832 [0.807, 0.855] GPT-4o â o3-mini 0.837 [0.832, 0.842] 0.794 [0.768, 0.819] GPT-4o â Qwen 3.6 Plus 0.829 [0.824, 0.834] 0.786 [0.759, 0.813] Llama 4 Mav. â o3-mini 0.850 [0.845, 0.854] 0.815 [0.790, 0.839] Llama 4 Mav. â Qwen 3.6 Plus 0.857 [0.852, 0.862] 0.813 [0.788, 0.837] o3-mini â Qwen 3.6 Plus 0.919 [0.916, 0.923] 0.900 [0.882, 0.917] Mean over pairs 0.864 [0.861, 0.867] 0.830 [0.815, 0.844] Table 15 shows Gwetâs AC1 score on the binary decision on the full dataset â the agreement between LLMs is generally high. The mean Gwetâs AC1 score over pairs is 0.864. When looking at the range, the lowest Gwetâs AC1 between any two LLMs is 0.813 (Claude 3.7 Sonnet vs. GPT-4.1), and the highest one is 0.938 between DeepSeek V4 Pro and Qwen 3.6 Plus. Looking at the 5% sampled data (Table 15), the agreement between LLMs is slightly lower, where the mean AC1 score decreases to 0.830 and the range of agreement is 0.768 - 0.921. Out of the 36 pairs in the table, nine of them have an agreement less than 0.80, and contain either GPT-4.1 or GPT-4o as the other model. Summary: RQ3.1 shows that LLMs make mostly the same decisions in paper screening, and the most disagreement comes from pairs where GPT-4.1 and GPT-4o are part of. 5.5 RQ3.2: LLM agreement - Individual criteria Table 16: Mean inter-model agreement (Gwetâs AC1) on binary screening decision. Type N Mean AC1 IC 93 0.778 [0.773, 0.783] EC 110 0.868 [0.865, 0.870] IC â- EC -0.090 [-0.095, -0.086] In order to get a better insight of the agreements and disagreements, we decided to look into individual criteria in RQ3.2. Table 16 shows the mean pairwise inter-model agreement grouped by criterion type. Exclusion criteria show slightly higher mean agreement than inclusion criteria (0.868 vs 0.778), which is statistically significant. As shown in Online resource 3 (Huotala et al., 2026b), agreement levels for inclusion criteria vary considerably across the secondary studies. For example, in study Stradowski and Madeyski (2023), agreement ranges from 0.7090.709 to 0.9650.965 across the inclusion criteria, whereas in study Alonso et al. (2023), it ranges from 0.0650.065 to 0.8340.834. A similar pattern is also observed for studies Casalaro et al. (2022); Zakeri-Nasrabadi et al. (2023); Deckers and Lago (2022); Tambon et al. (2022); Pereira et al. (2021); RodrĂguez-PĂ©rez et al. (2021); BariĆĄiÄ et al. (2022); Lenarduzzi et al. (2021); Lin et al. (2022), where substantial variation in agreement levels exists between individual inclusion criteria. The same pattern is observed partially for the exclusion criteria: agreement levels vary considerably across the secondary studies, and about half of the studies exhibit substantial variance between individual exclusion criteria. Despite the variability, some individual inclusion and exclusion criteria show consistently high agreement across the secondary studies. For example, exclusion criteria related to the paperâs language being other than English (Stradowski and Madeyski, 2023; Alonso et al., 2023; Somers et al., 2023; Zakeri-Nasrabadi et al., 2023; Lewowski and Madeyski, 2022; Van Dinter et al., 2022; Casalaro et al., 2022; Deckers and Lago, 2022; Tambon et al., 2022; Rani et al., 2023; Pereira et al., 2021; RodrĂguez-PĂ©rez et al., 2021; BariĆĄiÄ et al., 2022; Ebrahimi et al., 2021; Tebes et al., 2020) generally exhibited very high levels of agreement in AC1 from 0.911 to 1.000. Similarly, criteria based on simple and objective characteristics of a study tended to yield higher agreement than criteria requiring greater subjective interpretation. To illustrate criteria where LLMs have high agreement: âą Stradowski and Madeyski (2023) "The paper is a primary study" (AC1 0.9650.965) âą Lewowski and Madeyski (2022) "Title or abstract of the paper indicates that at least one code smell/anti-pattern plays an important part of the study." (AC1 0.8760.876) âą Rani et al. (2023) "The topic of the paper is about code comment quality" (AC1 0.9670.967) When looking at the criteria where LLMs disagree, we find two groups. Criteria that look simple and straightforward but whose agreement is still low, such as: âą Tambon et al. (2022) "Papers that thrive to solely improve a prediction metric such as accuracy, precision, or recall" (AC1 0.4430.443) âą Ebrahimi et al. (2021) "Studies which explicitly addresses privacy aspects of apps" (AC1 0.6180.618) âą Lenarduzzi et al. (2021) "Publications that only mention prioritization of TD in an introductory statement and do not fully or partly focus on it" (AC1 0.2590.259) âą Alonso et al. (2023) "Do not mainly comprise software development-related MVPs (e.g., mainly hardware or IoT related MVPs)" (AC1 0.0650.065) We must recognize that agreement is also influenced by the data, i.e. papers that are screened. Many criteria in the two bullet lists above have a good deal of subjectivity, e.g. "important part of the study" (Lewowski and Madeyski, 2022) (0.8760.876), while "do not fully or partly focus on it" (0.2590.259) both have vagueness in them yet they score totally differently. Then finally we have criteria that try to say too much and therefore do not perform very well: âą RodrĂguez-PĂ©rez et al. (2021) "We removed papers that analyze perceived diversity from a software management perspective (i.e., planning, scheduling, resource allocation, execution, delivery) and a team management perspective (i.e., communication) because these papers were not related to perceived diversity in the software development." (AC1 0.5300.530) âą Bjarnason et al. (2023) "Articles that merely describe the use of prototypes without considering methodological perspectives such as principles and procedures for the prototyping, articles that are not peer reviewed or written in English, and duplicates of studies already included in the systematic map" (AC1 0.6630.663) 5.6 RQ4: Refining the inclusion and exclusion criteria Table 17: Gwetâs AC1 across the GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, GLM 5.1, Kimi K2.6, Grok 4.20 Beta, Qwen 3.6 Plus, and DeepSeek V4 Pro LLMs for the refined criteria (RQ4). ID AC1 Study Old New Old New Î RodrĂguez-PĂ©rez et al. (2021) IC1 IC1 0.902 0.898 -0.003 [-0.022, +0.016] IC2 IC2 0.985 0.984 -0.001 [-0.010, +0.009] IC3 IC3 1.000 1.000 0.000 [0.000, 0.000] IC4 â 0.775 â â IC5 â 0.716 â â IC6 â 0.318 â â IC7 â 0.759 â â EC1 EC1 0.996 0.998 +0.002 [-0.004, +0.011] EC2 â 1.000 â â EC3 EC2 0.941 0.972 +0.031 [+0.011, +0.056] EC4 EC3 0.975 0.991 +0.016 [+0.001, +0.032] EC5 EC4 0.876 0.890 +0.014 [-0.008, +0.038] EC6 EC5 0.987 0.994 +0.007 [-0.004, +0.018] EC7 EC6 0.992 0.987 -0.004 [-0.012, 0.000] Bjarnason et al. (2023) IC1 IC1 0.731 0.734 +0.003 [-0.031, +0.041] EC1 IC2 0.475 1.000 +0.525 [+0.443, +0.608] EC1 0.416 -0.059 [-0.118, +0.005] Zhang and Li (2020) IC1 IC1 0.759 0.920 +0.162 [+0.079, +0.256] IC2 IC2 0.848 0.875 +0.027 [-0.060, +0.135] IC3 IC3 0.955 0.944 -0.011 [-0.039, 0.000] EC1 â 0.988 â â EC2 IC4 1.000 1.000 0.000 [0.000, 0.000] EC3 â 1.000 â â EC4 IC5 0.851 0.944 +0.093 [-0.001, +0.200] Tebes et al. (2020) IC1 â 0.748 â â IC2 IC1 0.948 0.963 +0.016 [-0.002, +0.038] IC3 IC2 0.868 0.704 -0.163 [-0.251, -0.079] EC1 EC1 0.945 0.948 +0.003 [-0.028, +0.038] EC2 IC3 0.919 0.955 +0.036 [-0.008, +0.088] EC3 IC4 1.000 1.000 0.000 [0.000, 0.000] EC4 IC5 0.889 0.829 -0.059 [-0.115, -0.011] EC5 â 1.000 â â EC6 â 1.000 â â EC7 â 0.995 â â Alonso et al. (2023) IC1 IC1 0.870 0.848 -0.022 [-0.083, +0.023] IC2 EC2 0.741 0.130 -0.610 [-0.769, -0.424] IC3 EC1 0.138 0.750 +0.612 [+0.384, +0.778] EC1 IC2 1.000 1.000 0.000 [0.000, 0.000] EC2 â 0.949 â â EC3 â 0.962 â â Lenarduzzi et al. (2021) IC1 IC1 0.883 0.800 -0.083 [-0.146, -0.028] IC2 IC2 0.841 0.817 -0.024 [-0.075, +0.023] IC3 IC3 0.873 0.833 -0.040 [-0.097, +0.007] IC4 IC4 0.921 0.881 -0.041 [-0.086, +0.002] EC1 IC5 0.993 1.000 +0.007 [0.000, +0.021] EC2 â 0.989 â â EC3 â 1.000 â â EC4 EC1 0.921 0.892 -0.030 [-0.104, +0.039] EC5 â 0.994 â â EC6 EC2 0.148 0.215 +0.067 [-0.011, +0.136] EC7 â 1.000 â â Deckers and Lago (2022) IC1 â 0.934 â â IC2 IC1 1.000 1.000 0.000 [0.000, 0.000] IC3 â 0.688 â â EC1 IC2 0.674 0.798 +0.124 [+0.043, +0.208] EC2 IC3 0.956 0.769 -0.187 [-0.323, -0.064] EC3 EC3 0.994 1.000 +0.006 [0.000, +0.019] EC4 EC4 0.986 0.987 +0.001 [0.000, +0.002] EC5 EC5 0.596 0.889 +0.293 [+0.167, +0.424] EC6 EC6 0.885 0.939 +0.054 [+0.008, +0.117] Mean 0.847 0.866 +0.019 [+0.008, +0.030] Table 18: Lost evidence (false negatives) per model: RQ2 vs. RQ4 (refined criteria), restricted to the 7 secondary studies with refined criteria, 5% sample. Percentages are relative to the total relevant papers in the sample. Lost Evidence (FN) Model RQ2 RQ4 Î (RQ4â-RQ2) GPT-5.5 95 (89%) [84, 106] 87 (81%) [76, 98] -8 [-15, -2] Claude Opus 4.7 99 (93%) [89, 109] 95 (89%) [84, 105] -4 [-8, -1] Gemini 3.1 Pro 92 (86%) [81, 103] 89 (83%) [78, 100] -3 [-8, +1] GLM 5.1 99 (93%) [89, 110] 98 (92%) [87, 109] -1 [-6, +4] Kimi K2.6 94 (88%) [83, 105] 92 (86%) [81, 102] -2 [-8, +3] Grok 4.20 Beta 94 (88%) [84, 105] 92 (86%) [81, 102] -2 [-7, +3] Qwen 3.6 Plus 89 (83%) [78, 100] 86 (80%) [74, 97] -3 [-8, +2] DeepSeek V4 Pro 95 (89%) [84, 105] 74 (69%) [63, 86] -21 [-30, -13] Avg. 95 (88%) [85, 105] 89 (83%) [79, 99] -6 [-9, -2] Table 19: RQ2 vs. RQ4 screening metrics. Model F11 MCC1 Recall Lost Evidence (FN) RQ2 RQ4 Î RQ2 RQ4 Î RQ2 RQ4 Î RQ2 RQ4 Î GPT-5.5 0.18 [0.07, 0.27] 0.23 [0.13, 0.31] +0.05 [-0.03, +0.13] 0.12 [0.00, 0.22] 0.12 [0.01, 0.20] -0.01 [-0.10, +0.09] 0.11 [0.06, 0.17] 0.19 [0.11, 0.26] +0.07 [+0.02, +0.14] 95 [84, 106] 87 [76, 98] -8 [-15, -2] Claude Opus 4.7 0.15 [0.05, 0.23] 0.16 [0.07, 0.24] +0.01 [-0.03, +0.06] 0.16 [0.03, 0.23] 0.15 [0.03, 0.22] -0.01 [-0.03, +0.03] 0.07 [0.03, 0.13] 0.11 [0.06, 0.17] +0.04 [+0.01, +0.08] 99 [89, 109] 95 [84, 105] -4 [-8, -1] Gemini 3.1 Pro 0.19 [0.09, 0.27] 0.21 [0.11, 0.29] +0.02 [-0.03, +0.08] 0.12 [0.01, 0.21] 0.15 [0.04, 0.23] +0.03 [-0.02, +0.08] 0.14 [0.08, 0.21] 0.17 [0.10, 0.24] +0.03 [-0.01, +0.07] 92 [81, 103] 89 [78, 100] -3 [-8, +1] GLM 5.1 0.13 [0.04, 0.21] 0.14 [0.05, 0.21] +0.01 [-0.03, +0.06] 0.12 [-0.01, 0.19] 0.11 [-0.02, 0.19] -0.01 [-0.07, +0.06] 0.07 [0.03, 0.13] 0.08 [0.04, 0.14] +0.01 [-0.04, +0.06] 99 [89, 110] 98 [87, 109] -1 [-6, +4] Kimi K2.6 0.15 [0.07, 0.22] 0.17 [0.08, 0.25] +0.02 [-0.03, +0.08] 0.09 [-0.01, 0.17] 0.10 [-0.02, 0.19] +0.01 [-0.08, +0.10] 0.12 [0.06, 0.18] 0.14 [0.08, 0.21] +0.02 [-0.03, +0.07] 94 [83, 105] 92 [81, 102] -2 [-8, +3] Grok 4.20 Beta 0.17 [0.07, 0.25] 0.17 [0.08, 0.25] +0.01 [-0.06, +0.07] 0.11 [-0.01, 0.21] 0.06 [-0.06, 0.14] -0.05 [-0.15, +0.05] 0.12 [0.07, 0.18] 0.14 [0.08, 0.21] +0.02 [-0.03, +0.07] 94 [84, 105] 92 [81, 102] -2 [-7, +3] Qwen 3.6 Plus 0.18 [0.09, 0.26] 0.19 [0.11, 0.28] +0.02 [-0.03, +0.06] 0.13 [0.02, 0.20] 0.09 [-0.02, 0.18] -0.03 [-0.09, +0.02] 0.17 [0.10, 0.24] 0.20 [0.13, 0.27] +0.03 [-0.02, +0.08] 89 [78, 100] 86 [74, 97] -3 [-8, +2] DeepSeek V4 Pro 0.17 [0.07, 0.24] 0.27 [0.18, 0.34] +0.10 [+0.04, +0.16] 0.14 [0.03, 0.21] 0.08 [-0.02, 0.17] -0.05 [-0.12, +0.02] 0.11 [0.06, 0.17] 0.31 [0.22, 0.40] +0.20 [+0.12, +0.28] 95 [84, 105] 74 [63, 86] -21 [-30, -13] Avg. 0.16 [0.08, 0.23] 0.19 [0.11, 0.26] +0.03 [0.00, +0.07] 0.12 [0.02, 0.19] 0.11 [0.01, 0.18] -0.01 [-0.05, +0.02] 0.12 [0.07, 0.17] 0.17 [0.11, 0.23] +0.05 [+0.02, +0.08] 95 [85, 105] 89 [79, 99] -6 [-9, -2] 1 Mean of the per-secondary-study scores (undefined MCC counts as 0). Recall and Lost Evidence are pooled over papers. Avg. is the mean over LLMs. Table 20: Per-study RQ2 vs. RQ4 screening metrics (V2 prompt, 5% sample, updated-criteria subset, averaged across LLMs). Study Accuracy Precision Recall F1 MCC RQ2 RQ4 RQ2 RQ4 RQ2 RQ4 RQ2 RQ4 RQ2 RQ4 Lenarduzzi et al. (2021) 0.87 0.86 0.73 0.81 0.23 0.23 0.33 0.33 0.34 0.36 Deckers and Lago (2022) 0.74 0.68 0.20 0.19 0.75 0.96 0.31 0.31 0.29 0.34 Tebes et al. (2020) 0.30 0.34 0.80 0.79 0.13 0.19 0.23 0.30 0.02 0.04 Zhang and Li (2020) 0.09 0.11 0.88 1.00 0.09 0.11 0.16 0.20 0.00 0.00 Alonso et al. (2023) 0.21 0.24 0.13 0.71 0.01 0.09 0.02 0.15 0.02 -0.11 Bjarnason et al. (2023) 0.78 0.74 0.04 0.03 1.00 1.00 0.08 0.06 0.17 0.15 RodrĂguez-PĂ©rez et al. (2021) 0.94 0.94 0.06 0.00 0.02 0.00 0.03 0.00 0.02 -0.01 Avg. 0.56 0.56 0.40 0.50 0.32 0.37 0.16 0.19 0.12 0.11 In analyzing RQ3.2, we noticed that some criteria could have benefited from refinement, and hypothesized that such refinement might lead to improvements in screening accuracy. Therefore, in RQ4, we selected studies for criteria refinement based on two factors: low screening accuracy (RQ1) and low agreement in screening criteria (RQ3.1 and RQ3.2). This rests on the assumption that an increase in agreement between LLMs would lead to an increase in accuracy. Next we go over the results in two separate sections. See Tables 17, 18, 19, and 20 for the results. The detailed criteria refinement results are available as Online resource 4 (Huotala et al., 2026b). 5.6.1 Agreement â Gwetâs AC1 On average, Gwetâs AC1 score increased slightly from 0.8470.847 to 0.8660.866 after the criteria refinement. However, Gwetâs AC1 comparison (Table 17) shows that when criteria were reworded, the AC1 score for the reworded criteria increased largely, or had statistical significance in some studies. For example, see the following changes in Table 17: âą RodrĂguez-PĂ©rez et al. (2021): EâCâ3âEâCâ2EC3â EC2 and EâCâ4âEâCâ3EC4â EC3 âą Bjarnason et al. (2023): EâCâ1âIâCâ2EC1â IC2 âą Zhang and Li (2020) IâCâ1IC1 âą Alonso et al. (2023): IâCâ3âEâCâ1IC3â EC1 âą Deckers and Lago (2022): EâCâ5EC5 Conversely, the AC1 score decreased notably in Alonso et al. (2023) for IâCâ2âEâCâ2IC2â EC2, Deckers and Lago (2022) (EâCâ2âIâCâ3EC2â IC3), and Tebes et al. (2020) (IâCâ3âIâCâ2IC3â IC2). The rest of the refined criteria exhibited similar levels of agreement, staying within a tenth of the original AC1. Based on bootstrapping, the change in AC1 score is statistically significant for some criteria â refining those criteria did either improve or degrade AC1. For more detailed analysis of study-wise changes, we analyze three studies in more detail (RodrĂguez-PĂ©rez et al., 2021; Bjarnason et al., 2023; Alonso et al., 2023). In study RodrĂguez-PĂ©rez et al. (2021), criteria IC4, IC5, IC6, IC7, and EC2 were deleted, as they were related to the paperâs metadata, not the paperâs title and abstract. Most of the reworded criteria exhibited similar levels of performance, e.g. EâCâ3âEâCâ2EC3â EC2 and EâCâ4âEâCâ3EC4â EC3 had an improvement in Gwetâs AC1 of +0.031+0.031 and +0.016+0.016, respectively. The exclusion criterion EC3 was originally written as: "We removed discussion papers, i.e., those papers studying the perceived diversity in a psychological, social, or philosophical context because they were not related to SE activities." (RodrĂguez-PĂ©rez et al., 2021) Then EC3 was reworded as: "Discussion papers, i.e., those papers studying the perceived diversity in a psychological, social, or philosophical context." This rewording was statistically significant, given the rewording process attempted to preserve the original wording while improving the clarity of the criteria. In study Bjarnason et al. (2023), criteria EC1 was split into IC2 and EC1. The new IC2 was simply written as: "Written in English" (AC1 = 1.0001.000), which was assumed to have a high level of agreement among LLMs. The other split criterion (EC1) states that "Articles that merely describe the use of prototypes without considering methodological perspectives such as principles and procedures for the prototyping." has no more ambiguity. As the original Gwetâs AC1 score for criterion EC1 was 0.4750.475, this is clear evidence that criteria that contain multiple conditions benefit significantly by splitting them into parts. While the paper was written in English, it would have skewed the original LLMâs reliability. In study Alonso et al. (2023), EC2 and EC3 were deleted because they were related to the paperâs publication year and the published journal, which yet again are impossible to be determined from the title and abstract without having access to the paperâs metadata. The rewording exhibited similar behavior compared to study RodrĂguez-PĂ©rez et al. (2021) and Bjarnason et al. (2023) in rewording, and moving the criteria from inclusion to exclusion criteria for IâCâ3âEâCâ1IC3â EC1. IC3 that was previously written as "Do not mainly comprise software development-related MVPs (e.g., mainly hardware or IoT related MVPs)" (Alonso et al., 2023) was reworded and flipped as EC1: "Comprise other than software development-related Minimum Viable Products (e.g., hardware or IoT related Minimum Viable Products)" Here, expanding acronyms and flipping the inclusion criterion into an exclusion criterion increased Gwetâs AC1 from 0.1380.138 to 0.7500.750. 5.6.2 Screening accuracy Lost evidence (Table 18) indicates that refining the criteria led to a small increase in retaining evidence, i.e. lost evidence reduction (-6), when compared to RQ2. Per-model differences follow the same pattern more or less, where GLM 5.1 exhibited the smallest reduction in lost evidence (-1) compared to DeepSeek V4 Pro (-21), GPT-5.5 (-8) and Claude Opus 4.7 (-4). F1 and MCC (Table 19) show that the average F1 score of the tested LLMs improved slightly from 0.16 to 0.19, whereas MCC decreased from 0.12 to 0.11. Looking at the model-level results, DeepSeek V4 Pro had a statistically significant increase in F1 score from 0.17 to 0.27. The remaining seven LLMs exhibited only a slight improvement in F1 score. MCC increased only for Gemini 3.1 Pro (+0.03) and Kimi K2.6 (+0.01). Recall was observed to improve with every LLM, and DeepSeek V4 Pro having a statistically significant increase of +0.20 along with GPT-5.5 (+0.07) and Claude Opus 4.7 (+0.04). The same was observed for lost evidence, where the three LLMs above had also a reduction in lost evidence that was statistically significant. The per-study results (Table 20) were surprising, although the average F1 score improved slightly from 0.16 to 0.19. Average accuracy did not improve, while recall increased from 0.32 to 0.37. Precision also increased from 0.40 to 0.50, which indicates that refining the criteria helped identify more relevant papers and also reduced false positives. MCC degraded by 0.01 on average. At the study level, study Deckers and Lago (2022) showed the strongest improvement in recall, while study Lenarduzzi et al. (2021) showed an increase only in precision and MCC. Summary: RQ4 showed modest improvement in LLM agreement, in retaining evidence and recall when original screening criteria were refined. 6 Discussion 6.1 LLM performance (RQ1) Despite the 18-month gap between the first execution of SESR-Eval and the re-run in this paper, the newer models on the full dataset (RQ1.1) are at best marginally more accurate than the seven earlier ones. On both MCC and F1 metrics, DeepSeek V4 Pro (F1=0.48, MCC=0.41) and Qwen 3.6 Plus (F1=0.48, MCC=0.40) did not reach the F1 and MCC of older LLMs, such as GPT-4.1 mini (F1=0.50, MCC=0.46). Given that these two new models are approximately five times more expensive than GPT-4.1 mini (the best-performing old LLM in terms of MCC alongside Llama 4 Maverick), conducting paper screening on these expensive models is not recommended. In addition, the new models were among the slowest ones tested. However, the results on the SESR-Eval-Mini dataset (RQ1.2) diverged: Claude Opus 4.7 (MCC=0.46) claimed the crown as the best model for paper screening by a narrow margin over GPT-4.1 mini and Llama 4 Maverick (MCC=0.45). As we did not test Claude Opus 4.7 on the full dataset, we cannot rule out that it is genuinely better. However, its per-token costs make screening thousands of papers expensive, so this gain must be weighed. Taken together, the results suggest that we may be approaching a ceiling, where latency and costs grow faster than screening performance. Claude Opus 4.7 demonstrates that performance improvements are still possible. Part of this latency might be accounted for by the larger models, where additional inference time is required, as was observed for the old LLMs (DeepSeek R1 and o3-mini) in the prior work (Huotala et al., 2025a). The inflated token costs across the field (Table 2) might also be accounted for by the larger models, increased demand of LLMs or reasoning tokens, which all necessitate additional compute power. All the new LLMs were tested with reasoning turned on, so this itself inflates costs, even if token prices stay at the same level. Since the new models do not clearly perform better in paper screening, we might have to look somewhere else for improvement. One possibility is that the abstracts may not provide enough information for LLMs to accurately make the screening decision, which has been previously discussed in Felizardo et al. (2025). Human reviewers perform paper screening using the same information available, so we would assume that LLMs should be able to complete the same task with acceptable accuracy. However, human reviewers bring domain expertise and knowledge of the SRâs goals, which LLMs might have difficulties in. Human reviewers also tend to disagree with one another at this stage, which may cap LLM screening performance regardless of model capability. Providing LLMs with additional context, that may be implicit to human reviewers, about the screening task may therefore be one potential direction for improving screening performance. This would help in keeping costs under control, as larger and newer models tend to have higher per-token costs. Agent-based approaches are a second direction worth studying, as the design paradigm for generative AI has gradually been shifting towards agentic AI. In particular, task decomposition could split the screening process into smaller tasks, with each AI agent evaluating a specific inclusion or exclusion criterion. This could improve screening performance by letting each LLM focus more on a single criteria, and aggregating the results in the end. However, multiple AI agents multiply LLM calls per paper, which could further inflate costs. In addition, LLMs could be provided more context about the SR being conducted, so their context aligns better with the interests of the inclusion and exclusion criteria. 6.2 Effects of prompt design and output structure on screening performance (RQ2) Computing the overall screening decision from the criterion-level decision decreased precision, recall, F1, MCC, and accuracy metrics. In addition, the V2 prompt and output structure made LLMs lose more relevant evidence. Adam et al. (2026) tested 515 different prompts, and had an F1 score of 0.276, recall of 0.836 and precision of 0.185. While we tested two versions of prompts and output schemas (V1 and V2), our prompts achieved a better F1 score of 0.47, but precision and recall (0.62 and 0.38) did not have the same pattern. As previously mentioned, targeting a good precision while maximizing recall is the goal for LLM-based paper screening (Huotala et al., 2025a), which our V2 prompt and structure did not yet achieve. This would indicate that the user prompt would need even bigger changes compared to the baseline, to reach the goal of good precision with maximized recall. DennstĂ€dt et al. (2024) mentioned that even small changes to the user prompt significantly impacted the screening performance. Our results show that the changes we implemented to the V2 user prompt impacted screening performance ever so slightly. This would indicate that LLMs behave very differently on certain changes made to the prompt, and more radical changes would have been needed, or modifying the system prompt instead of the user prompt, as LLMs tend to respect and follow the system prompt more strictly than the user prompt. Matsui et al. (2024) and Shameem et al. (2026) observed that LLMs classified papers incorrectly because of insufficient domain knowledge, misinterpreted inclusion criteria, or domain-specific literature. While our changes to the V2 included clarifications to the interpretation of inclusion and exclusion criteria, the amount of lost evidence increased slightly and the overall performance degrades. This result aligns with Matsui et al. and Shameem et al., which would indicate that the LLMs had trouble understanding the criteria, even after our changes were meant to clarify their application and improve understanding. JĂłzwiak et al. (2026) observed that prompt optimization (which is an area of prompt design) improved recall by +3.7% to +37.1% for the LLMs â in our results we observed an average reduction of -0.03 and between -0.06 and +0.01 for the tested LLMs. Our result is the opposite, and we believe that more radical changes are needed to observe a larger difference. 6.3 LLM agreement with each other and across inclusion and exclusion criteria (RQ3.1 & RQ3.2) In our experiments, LLMs generally had an acceptable level of agreement on the overall binary include/exclude decisions (AC1=0.864), and on per-criterion decisions (include AC1=0.778, exclude AC1=0.868). In one prior work, Guo et al. (2024) agreement between models was also acceptable, with a quite high kappa = 0.96. Paradoxically, getting benefits from the Condorcet Jury Theorem, see Section 3, would require us to have high correctness (MCC) in individual LLM decisions but low agreement. Such a setup would provide the biggest benefits, as the mistakes by one LLM would not be repeated by another LLM. Unfortunately, it seems that we have low correctness (MCC) and high agreement between LLMs, making the possible gains of combining decisions limited. In another prior work, Thode et al. (2025) used multiple LLMs and included a paper if either LLM would include it. Obviously, this increased recall while decreasing precision, as each paper gets multiple tries to be included. 6.4 Inclusion and exclusion criteria refinement (RQ4) Refining the inclusion and exclusion criteria produced encouraging and informative results. We hypothesized that ambiguous criteria in the secondary studies were degrading screening performance. The mean F1 score increased only by a margin, from 0.16 to 0.19, showing that rewording the criteria does not alone explain the low precision. However, the Gwetâs AC1 scoring shows that several criteria became considerably easier for the LLMs to interpret. The criteria that combined multiple conditions made the LLMsâ judgment unable to distinguish between the two conditions. For instance, as checking two conditions that a paper was written in English and that it described prototypes without a methodological perspective in a single criterion, the LLM lost the language check outcome completely. Therefore, we strongly recommend that inclusion and exclusion criteria that are used with LLMs to screen papers should be singular, with acronyms explained and written in a form that is easy to understand by the LLMs. Using paper metadata in inclusion and exclusion criteria for paper screening is also not advisable, as the LLMs cannot make that information up without being fed additional context or allowing the LLM to retrieve that metadata from a third party source. 6.5 Limitations and Threats to Validity Next, the limitations and threats to the validity of our work are outlined. Generalization of results. We benchmarked only SE-specific secondary studies and their primary studies in this work. This was because our prior work (Huotala et al., 2025a), which established the gold-standard dataset used, is itself limited to the SE domain. We suspect our findings might not generalize to other domains, such as medicine or health technology assessment, as the conventions of inclusion and exclusion criteria, and the structure of abstracts can differ. Reliability of screening results. There is always the risk that LLMs can hallucinate and therefore have an effect on the screening results. We have attempted to mitigate this threat by benchmarking multiple LLMs, to minimize the effect of possible hallucinations the LLMs might have in their token generation chains. Only zero-shot prompting technique was tested. We tested only the zero-shot prompting technique, as the additional prompting strategies would have introduced an additional variable that could disrupt the differences between LLMs. However, our prior work evaluated one-shot (OS), few-shot (FS), and few-shot with chain-of-thought (FS-CoT) prompting techniques, which showed that using prompting strategies improves LLM performance (Huotala et al., 2024). Our evaluation in this work therefore reflects the "baseline" result, which is assumed to improve with OS, FS and FS-CoT prompting techniques. One LLM did not support structured outputs via native or tool calling. In this case, we resorted to prompted output for Qwen 3.6 Plus, where the LLM was instructed in-prompt to output JSON directly rather than by tool-calling or outputting natively. This inflated costs, as the schema had to be included in every request, which also might have increased the possibility of malformed outputs. However, we configured Pydantic AI to retry the requests if the output did not match the expected schema, to minimize this threat. Fortunately, most LLMs supported either tool calling or native output. Undefined MCC scores. We noticed that the scripts we used to compute MCC scores for Tables 7 and 10 (and the per-study Tables 8 and 11) warned us about undefined MCC scores, because two of the studies, Zhang and Li (2020) and Pereira et al. (2021), contained no excluded papers, in both the full dataset and the sampled dataset. This makes MCCâs denominator zero regardless of the modelâs prediction. We believe this might have negatively impacted the validity of the MCC if averaged over secondary studies, as the zero values could have dragged down the MCC score by approximately 2/24 (8.3%). We decided not to filter out the two studies, so that the average is still taken from the n=24 secondary studies. As the possible reduction of the MCC score here is relative for every LLM, this does not change how the LLMs compare against each other. Most new models were run at low reasoning effort. Six of the eight models in the experiments were run with reasoning effort set to low, with Qwen 3.6 Plus at medium and DeepSeek V4 Pro at high. The choice was motivated by prior evidence that the reasoning setting brings negligible gains in screening performance (Huotala et al., 2025a). However, the evidence predates the models evaluated in this paper, and reasoning effort may have different effects for newer models. Our results for the low-effort models should be read as a lower bound rather than their peak performance. Notably, the two models (Qwen 3.6 Plus and DeepSeek V4 Pro) run at different reasoning settings did not have meaningful improvement in screening performance. Possibility of training data contamination. The SESR-Eval dataset was released in October 2025, so models released after that date may have been trained on the dataset. However, the data from the secondary studies might be in the training data, as they were released years before the dataset itself. As we observed only marginal performance improvements for the newer models, we believe this threat is unlikely to have affected our benchmark results. It cannot be fully ruled out, however, as most LLM vendors do not disclose what data the models have been trained on. Incorrect ground truths in SESR-Eval. It is likely that some ground truth labels in SESR-Eval are incorrect, as the original secondary study authors may have also made mistakes. Furthermore, it is also possible that not all decisions in the SESR-Eval dataset were made based solely on the title and abstract, even though SESR-Eval purposefully sourced studies where such decision points were visible. However, we think it is highly unlikely that such issues alone would explain the rather limited LLM screening performance. Further support that LLM mistakes in screening software engineering papers are real â and not an artifact of poor ground truth â is provided by MĂ€ntylĂ€ et al. (2026), where the original secondary study authors screened new papers manually, compared their results with LLM decisions, and analyzed the reasons for LLM screening mistakes. In that study, as in this paper, LLM performance in the screening task was modest. Refinement of inclusion and exclusion criteria. The refined inclusion and exclusion criteria are our interpretation of the original authorsâ intent. There is a risk that the rewritten or split criteria might not match the original authorâs intent. We mitigated this threat by limiting the changes to: 1) expanding abbreviations and acronyms, which does not affect the original meaning of the criterion; 2) splitting compound criteria into separate rules, if there is a distinct separation between them; 3) removing criteria that cannot be verified from the title or abstract, since we do not want the LLMs to make up the information and guess; 4) converting negated criteria to affirmative form, which LLMs generally interpret more reliably; 5) preserving the original wording as much as possible; and 6) clarifying vague terms in the criteria using definitions straight from the paper. In addition, all deleted, reworded, and added rules were reviewed by two of the authors. Nevertheless, as the refinement process was a manual and laborious process, human error cannot be ruled out. 7 Conclusions and future work In this paper, we extended the SESR-Eval (Huotala et al., 2025a) study considerably by benchmarking multiple new LLMs for paper screening, creating the new SESR-Eval-Mini dataset, investigating if prompt design and output structuring affected screening performance, analyzing agreement in screening decisions across LLMs and across inclusion and exclusion criteria, and finally, examining whether refining the inclusion and exclusion criteria had any effect on screening performance. Despite the 18-month gap between Huotala et al. (2025a) and this work, we found at most marginal improvements in LLM screening performance. This suggests that the publicly reported advancements of LLMs do not translate into improved screening performance for software engineering papers. Thus, the evidence indicates we must state that LLM progress has largely stalled in this respect. Differences in screening performance are larger between secondary studies than between LLMs, and also with the new LLMs, meaning that the performance varies more based on the study than the used model. This means that while benchmarking new LLMs in screening tasks, it is important to test a broad selection of secondary studies. Following the finding that differences between secondary studies are large, we improved the screening rules for certain secondary studies that performed poorly in screening, but found only modest improvements in screening performance. As improving screening rules makes little difference, this means that secondary study differences are likely to stem from other sources. One such source to be investigated in future studies is the primary studies that each secondary study screens. It is quite possible that some datasets (sets of primary studies) are more difficult to screen correctly than others. Taken together, LLMs are still not reliable enough for replacing humans in the paper screening process of SRs. However, we feel that LLMs can still be a useful companion in paper screening. The current bottleneck of paper screening of LLMs is not the raw compute and model capabilities, but how much context paper abstracts and the prompts contain, and in what kind of prompt the screening task is provided to the LLMs. As future work, we are specifically looking into designing a distributed screening architecture, where inclusion and exclusion criteria are split between multiple small language model (SLM) agents. We believe this approach has high potential in increasing screening accuracy, as each SLM agent would focus on evaluating a single criterion rather than juggling between varying amounts of criteria simultaneously while trying to arrive at a decision from a single piece of text. As another future work, we are looking into ways that would maximize recall with the usage of probability thresholds, which would open the possibility of using LLMs to filter out studies before human screening. Filtering would be especially valuable in review contexts where the terminology used is not settled, as database searches often yield thousands of results for human screening resulting in high amounts of manual work and excluded papers. Lastly, we are looking at investigating the effect of other prompting techniques in a more robust manner, and the combination of few-shot (FS) and chain-of-thought (CoT) prompting together with multi-agent systems. Data availability The scripts, data, and other materials used to conduct this work are available in the research artifact (Huotala et al., 2026b). Acknowledgments The first and third authors have been supported by the Strategic Research Council of Research Council of Finland (Grant ID 358471). Declarations The authors declare that they have no conflict of interest. References Aubakirova et al. (2026) Aubakirova, M., Atallah, A., Clark, C., Summerville, J., Midha, A.: State of AI: An Empirical 100 Trillion Token Study with OpenRouter. arXiv (2026). https://doi.org/10.48550/ARXIV.2601.10088 Adam et al. (2026) Adam, T.J., Abosabie, S.A.S., Dittmer, M., Wolf, E., Abosabie, S.A., Behnke, C., Baier, F., Weickmann, A., Köser, L., Correll, C.U., Rutsch, N.: Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study. Research Synthesis Methods, 1â18 (2026) https://doi.org/10.1017/rsm.2026.10093 Alonso et al. (2023) Alonso, S., Kalinowski, M., Ferreira, B., Barbosa, S.D.J., Lopes, H.: A systematic mapping study and practitioner insights on the use of software engineering practices to develop MVPs. Information and Software Technology 156, 107144 (2023) https://doi.org/10.1016/j.infsof.2022.107144 Abramowitz et al. (2025) Abramowitz, B., Lev, O., Mattei, N.: Who Reviews The Reviewers? A Multi-Level Jury Problem. In: Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems. AAMAS â25, p. 14â22. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2025). https://doi.org/10.5555/3709347.3743512 Afonso et al. (2026) Afonso, M., Saavedra, N., Lourenço, B., Mendes, A., Ferreira, J.F.: ProfOlaf: Semi-automated Tool for Systematic Literature Reviews. In: Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering. FSE Companion â26, p. 262â266. ACM, New York, NY, USA (2026). https://doi.org/10.1145/3803437.3806432 Bankier (1988) Bankier, M.D.: Power Allocations: Determining Sample Sizes for Subnational Areas. The American Statistician 42(3), 174â177 (1988) https://doi.org/10.2307/2684995 Bjarnason et al. (2023) Bjarnason, E., Lang, F., Mjöberg, A.: An empirically based model of software prototyping: A mapping study and a multi-case study. Empirical Software Engineering 28(5), 115 (2023) https://doi.org/10.1007/s10664-023-10331-w Behrouzian et al. (2026) Behrouzian, A.M., Meleti, M., Colangelo, M.T., Calciolari, E., Galli, C.: Linguistic Polarity and Decision Architecture in LLM-Based Abstract Screening for Systematic Reviews. Information 17(5) (2026) https://doi.org/10.3390/info17050449 BariĆĄiÄ et al. (2022) BariĆĄiÄ, A., Ruchkin, I., SaviÄ, D., Mohamed, M.A., Al-Ali, R., Li, L.W., Mkaouar, H., Eslampanah, R., Challenger, M., Blouin, D., Nikiforova, O., Cicchetti, A.: Multi-paradigm modeling for cyberâphysical systems: A systematic mapping review. Journal of Systems and Software 183, 111081 (2022) https://doi.org/10.1016/j.jss.2021.111081 Casalaro et al. (2022) Casalaro, G.L., Cattivera, G., Ciccozzi, F., Malavolta, I., Wortmann, A., Pelliccione, P.: Model-driven engineering for mobile robotic systems: A systematic mapping study. Software and Systems Modeling 21(1), 19â49 (2022) https://doi.org/10.1007/s10270-021-00908-8 Chiang and Lee (2023) Chiang, C.-H., Lee, H.-y.: Can Large Language Models Be an Alternative to Human Evaluations? Association for Computational Linguistics (2023). https://doi.org/10.18653/v1/2023.acl-long.870 Deizepe et al. (2026) Deizepe, A., Felizardo, K.R., Pereira, R.: AI LitReview: A web-based tool for LLM-assisted systematic literature reviews. In: Proceedings of the XL SimpĂłsio Brasileiro de Engenharia de Software (SBES 2026) (2026) Delgado-Chaves et al. (2025) Delgado-Chaves, F.M., Jennings, M.J., Atalaia, A., Wolff, J., Horvath, R., Mamdouh, Z.M., Baumbach, J., Baumbach, L.: Transforming literature screening: The emerging role of large language models in systematic reviews. Proceedings of the National Academy of Sciences 122(2), 2411962122 (2025) https://doi.org/10.1073/pnas.2411962122 Deckers and Lago (2022) Deckers, R., Lago, P.: Systematic literature review of domain-oriented specification techniques. Journal of Systems and Software 192, 111415 (2022) https://doi.org/10.1016/j.jss.2022.111415 Du et al. (2024) Du, J., Soysal, E., Wang, D., He, L., Lin, B., Wang, J., Manion, F.J., Li, Y., Wu, E., Yao, L.: Machine learning models for abstract screening task - A systematic literature review application for health economics and outcome research. BMC Medical Research Methodology 24, 108 (2024) https://doi.org/10.1186/s12874-024-02224-3 DennstĂ€dt et al. (2024) DennstĂ€dt, F., Zink, J., Putora, P.M., Hastings, J., Cihoric, N.: Title and abstract screening for literature reviews using large language models: An exploratory study in the biomedical domain. Systematic Reviews 13(1), 158 (2024) https://doi.org/10.1186/s13643-024-02575-4 Efron and Tibshirani (1994) Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap, 1st edn., p. 1â456. Chapman and Hall/CRC, New York, NY, USA (1994). https://doi.org/10.1201/9780429246593 Ebrahimi et al. (2021) Ebrahimi, F., Tushev, M., Mahmoud, A.: Mobile app privacy in software engineering research: A systematic mapping study. Information and Software Technology 133, 106466 (2021) https://doi.org/10.1016/j.infsof.2020.106466 Felizardo et al. (2025) Felizardo, K.R., Deizepe, A., Coutinho, D., Gomes, G., Meireles, M., Gerosa, M., Steinmacher, I.: On the Difficulties of Conducting and Replicating Systematic Literature Reviews Studies Using LLMs in Software Engineering. In: 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE), p. 20â23. IEEE, Ottawa, ON, Canada (2025). https://doi.org/10.1109/WSESE66602.2025.00010 Felizardo et al. (2024) Felizardo, K.R., Lima, M.S., Deizepe, A., Conte, T.U., Steinmacher, I.: ChatGPT application in Systematic Literature Reviews in Software Engineering: An evaluation of its accuracy to support the selection activity. In: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ESEM â24, p. 25â36. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3674805.3686666 Guo et al. (2024) Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., Naugler, C.: Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study. Journal of Medical Internet Research 26(1), 48996 (2024) https://doi.org/10.2196/48996 Geraldi et al. (2020) Geraldi, R.T., Reinehr, S., Malucelli, A.: Software product line applied to the internet of things: A systematic literature review. Information and Software Technology 124, 106293 (2020) https://doi.org/10.1016/j.infsof.2020.106293 Gwet (2008) Gwet, K.L.: Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology 61(1), 29â48 (2008) https://doi.org/10.1348/000711006X126600 Huotala et al. (2025a) Huotala, A., Kuutila, M., MĂ€ntylĂ€, M.: SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews. In: Proceedings of the 19th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ESEM â25, p. 1â12. IEEE, Piscataway, NJ, USA (2025). https://doi.org/10.48550/arXiv.2507.19027 Huotala et al. (2025b) Huotala, A., Kuutila, M., MĂ€ntylĂ€, M.: Research artifacts in secondary studies: A systematic mapping in software engineering. Information and Software Technology 187, 107830 (2025) https://doi.org/10.1016/j.infsof.2025.107830 Huotala et al. (2026b) Huotala, A., Kuutila, M., MĂ€ntylĂ€, M.: Dataset for: Has LLM Screening Performance Stalled in Systematic Reviews? Zenodo (2026). https://doi.org/10.5281/zenodo.23117229 Huotala et al. (2026c) Huotala, A., Kuutila, M., MĂ€ntylĂ€, M.: Dataset for: SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews. Zenodo (2026). https://doi.org/10.5281/zenodo.23205534 Huotala et al. (2024) Huotala, A., Kuutila, M., Ralph, P., MĂ€ntylĂ€, M.: The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. EASE â24, p. 262â271. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3661167.3661172 Huotala et al. (2026a) Huotala, A., Kuutila, M., Turtio, O.-P., SipilĂ€, S., MĂ€ntylĂ€, M.: AISysRev - LLM-based Tool for Title-Abstract Screening. In: Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering. FSE Companion â26, p. 142â146. ACM, New York, NY, USA (2026). https://doi.org/10.1145/3803437.3806408 Hamel et al. (2020) Hamel, C., Kelly, S.E., Thavorn, K., Rice, D.B., Wells, G.A., Hutton, B.: An evaluation of DistillerSRâs machine learning-based prioritization tool for title/abstract screening â impact on reviewer-relevant outcomes. BMC Medical Research Methodology 20(1), 256 (2020) https://doi.org/10.1186/s12874-020-01129-1 JĂłzwiak et al. (2026) JĂłzwiak, A., Imre, A., HagymĂĄsy, J., Tittmann, J., Nagy, A., KovĂĄcs, S., Kardas, P., van Boven, J.F., Mommers, I., Ăgh, T.: REFLECTIVE-TIAB: cost-effective prompt optimization for large language model-based title and abstract screening in literature reviews. Expert Review of Pharmacoeconomics & Outcomes Research 26(6), 747â755 (2026) https://doi.org/10.1080/14737167.2026.2691995 Kitchenham and Charters (2007) Kitchenham, B., Charters, S.: Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical report, Keele University (January 2007) Kitchenham et al. (2004) Kitchenham, B.A., Dyb, T., JĂžrgensen, M.: Evidence-Based Software Engineering. In: Proceedings of the 26th International Conference on Software Engineering. ICSE â04, p. 273â281. IEEE Computer Society, USA (2004). https://doi.org/10.5555/998675.999432 Kuutila et al. (2020) Kuutila, M., MĂ€ntylĂ€, M., Farooq, U., Claes, M.: Time pressure in software engineering: A systematic review. Information and Software Technology 121, 106257 (2020) https://doi.org/10.1016/j.infsof.2020.106257 Lenarduzzi et al. (2021) Lenarduzzi, V., Besker, T., Taibi, D., Martini, A., Arcelli Fontana, F.: A systematic literature review on Technical Debt prioritization: Strategies, processes, factors, and tools. Journal of Systems and Software 171, 110827 (2021) https://doi.org/10.1016/j.jss.2020.110827 Lin et al. (2022) Lin, B., Cassee, N., Serebrenik, A., Bavota, G., Novielli, N., Lanza, M.: Opinion Mining for Software Development: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 31(3), 38â13841 (2022) https://doi.org/10.1145/3490388 List and Goodin (2001) List, C., Goodin, R.E.: Epistemic Democracy: Generalizing the Condorcet Jury Theorem. Journal of Political Philosophy 9(3), 277â306 (2001) https://doi.org/10.1111/1467-9760.00128 Lewowski and Madeyski (2022) Lewowski, T., Madeyski, L.: How far are we from reproducible research on code smell detection? A systematic literature review. Information and Software Technology 144, 106783 (2022) https://doi.org/10.1016/j.infsof.2021.106783 Li et al. (2024) Li, M., Sun, J., Tan, X.: Evaluating the effectiveness of large language models in abstract screening: A comparative analysis. Systematic Reviews 13(1), 219 (2024) https://doi.org/10.1186/s13643-024-02609-x Madeyski et al. (2026) Madeyski, L., Kitchenham, B., Shepperd, M.: LLM4SCREENLIT: Recommendations on assessing the performance of large language models for screening literature in systematic reviews. Information and Software Technology 198, 108204 (2026) https://doi.org/10.1016/j.infsof.2026.108204 MĂ€ntylĂ€ et al. (2026) MĂ€ntylĂ€, M., Matsubara, P., Felizardo, K.R., Kuutila, M., Gerosa, M., de Sousa Sampaio, S., Conte, T., Steinmacher, I.: Understanding LLMs in Title-Abstract Screening: From disagreements to recommendations. In: Euromicro Conference on Software Engineering and Advanced Applications, p. 226â242. Springer, Cham, Switzerland (2026). https://doi.org/10.1007/978-3-032-36587-3_16 Manning et al. (2008) Manning, C.D., Raghavan, P., SchĂŒtze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008) Matsui et al. (2024) Matsui, K., Utsumi, T., Aoki, Y., Maruki, T., Takeshima, M., Takaesu, Y.: Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews. Journal of Medical Internet Research 26(1), 52758 (2024) https://doi.org/10.2196/52758 Marshall and Wallace (2019) Marshall, I.J., Wallace, B.C.: Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic Reviews 8(1), 163 (2019) https://doi.org/10.1186/s13643-019-1074-9 Ouzzani et al. (2016) Ouzzani, M., Hammady, H., Fedorowicz, Z., Elmagarmid, A.: Rayyanâa web and mobile app for systematic reviews. Systematic reviews 5(1), 210 (2016) https://doi.org/10.1186/s13643-016-0384-4 Pereira et al. (2021) Pereira, J.A., Acher, M., Martin, H., JĂ©zĂ©quel, J.-M., Botterweck, G., Ventresque, A.: Learning software configuration spaces: A systematic literature review. Journal of Systems and Software 182, 111044 (2021) https://doi.org/10.1016/j.jss.2021.111044 Qiu et al. (2026) Qiu, T., Jiang, H., Chen, M.S., Lin, T.: A Prompt-Based Tutorial for Large Language ModelâAssisted Screening in Systematic Reviews and Meta-Analyses. International Journal of Technology Assessment in Health Care, 1â31 (2026) https://doi.org/10.1017/S0266462326104073 Ralph et al. (2020) Ralph, P., Ali, N., Baltes, S., Bianculli, D., Diaz, J., Dittrich, Y., Ernst, N., Felderer, M., Feldt, R., Filieri, A., de França, B.B.N., Furia, C.A., Gay, G., Gold, N., Graziotin, D., He, P., Hoda, R., Juristo, N., Kitchenham, B., Lenarduzzi, V., MartĂnez, J., Melegati, J., Mendez, D., Menzies, T., Molleri, J., Pfahl, D., Robbes, R., Russo, D., SaarimĂ€ki, N., Sarro, F., Taibi, D., Siegmund, J., Spinellis, D., Staron, M., Stol, K., Storey, M.-A., Taibi, D., Tamburri, D., Torchiano, M., Treude, C., Turhan, B., Wang, X., Vegas, S.: Empirical Standards for Software Engineering Research (2020). https://doi.org/10.48550/arXiv.2010.03525 Rani et al. (2023) Rani, P., Blasi, A., Stulova, N., Panichella, S., Gorla, A., Nierstrasz, O.: A decade of code comment quality assessment: A systematic literature review. Journal of Systems and Software 195, 111515 (2023) https://doi.org/10.1016/j.jss.2022.111515 RodrĂguez-PĂ©rez et al. (2021) RodrĂguez-PĂ©rez, G., Nadri, R., Nagappan, M.: Perceived diversity in software engineering: A systematic literature review. Empirical Software Engineering 26(5), 102 (2021) https://doi.org/10.1007/s10664-021-09992-2 Shameem et al. (2026) Shameem, M., Alshayeb, M., Niazi, M., Mahmood, S., Ahmad, A., Al Dallal, J.: An exploratory study on effectiveness of GPT-4o in conducting sub-tasks of systematic literature reviews. Automated Software Engineering 33(3), 81 (2026) https://doi.org/10.1007/s10515-026-00629-6 Shevtsov et al. (2018) Shevtsov, S., Berekmeri, M., Weyns, D., Maggio, M.: Control-Theoretical Software Adaptation: A Systematic Literature Review. IEEE Transactions on Software Engineering 44(8), 784â810 (2018) https://doi.org/10.1109/TSE.2017.2704579 Somers et al. (2023) Somers, R.J., Douthwaite, J.A., Wagg, D.J., Walkinshaw, N., Hierons, R.M.: Digital-twin-based testing for cyberâphysical systems: A systematic literature review. Information and Software Technology 156, 107145 (2023) https://doi.org/10.1016/j.infsof.2022.107145 Serretti (2026) Serretti, A.: Benchmarking a Local Schema-Constrained Large Language Model Pipeline for Abstract Screening and Evidence Mapping. Cureus 18(6) (2026) https://doi.org/10.7759/cureus.111193 Stradowski and Madeyski (2023) Stradowski, S., Madeyski, L.: Machine learning in software defect prediction: A business-driven systematic mapping study. Information and Software Technology 155, 107128 (2023) https://doi.org/10.1016/j.infsof.2022.107128 Shteingart et al. (2020) Shteingart, H., Marom, E., Itkin, I., Shabat, G., Kolomenkin, M., Salhov, M., Katzir, L.: Majority Voting and the Condorcetâs Jury Theorem. arXiv (2020). https://doi.org/10.48550/arXiv.2002.03153 Sujau et al. (2025) Sujau, M., Wada, M., VallĂ©e, E., Hillis, N., SuĆĄnjak, T.: Accelerating Disease Model Parameter Extraction: An LLM-Based Ranking Approach to Select Initial Studies for Literature Review Automation. Machine Learning and Knowledge Extraction 7(2), 28 (2025) https://doi.org/10.3390/make7020028 Sharbaf et al. (2023) Sharbaf, M., Zamani, B., SunyĂ©, G.: Conflict management techniques for model merging: A systematic mapping review. Software and Systems Modeling 22(3), 1031â1079 (2023) https://doi.org/10.1007/s10270-022-01050-9 Teixeira et al. (2017) Teixeira, S., Agrizzi, B.A., Filho, J.G.P., Rossetto, S., Baldam, R.D.L.: Modeling and automatic code generation for wireless sensor network applications using model-driven or business process approaches: A systematic mapping study. Journal of Systems and Software 132, 50â71 (2017) https://doi.org/10.1016/j.jss.2017.06.024 Tran et al. (2023) Tran, V.-T., Gartlehner, G., Yaacoub, S., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Aboulayeh, F., Afach, S., Meerpohl, J., Ravaud, P.: Sensitivity, Specificity and Avoidable Workload of Using a Large Language Models for Title and Abstract Screening in Systematic Reviews and Meta-Analyses. medRxiv (2023). https://doi.org/10.1101/2023.12.15.23300018 Thode et al. (2025) Thode, L., Iftikhar, U., Mendez, D.: Exploring the use of LLMs for the selection phase in systematic literature studies. Information and Software Technology 184 (2025) https://doi.org/10.1016/j.infsof.2025.107757 Tambon et al. (2022) Tambon, F., Laberge, G., An, L., Nikanjam, A., Mindom, P.S.N., Pequignot, Y., Khomh, F., Antoniol, G., Merlo, E., Laviolette, F.: How to certify machine learning based safety-critical systems? A systematic literature review. Automated Software Engineering 29(2), 38 (2022) https://doi.org/10.1007/s10515-022-00337-x Tebes et al. (2020) Tebes, G., Peppino, D., Becker, P., Matturro, G., Solari, M., Olsina, L.: Analyzing and documenting the systematic review results of software testing ontologies. Information and Software Technology 123, 106298 (2020) https://doi.org/10.1016/j.infsof.2020.106298 Trad et al. (2025) Trad, F., Yammine, R., Charafeddine, J., Chakhtoura, M., Rahme, M., El-Hajj Fuleihan, G., Chehab, A.: Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation. BMC Medical Research Methodology 25(1), 130 (2025) https://doi.org/10.1186/s12874-025-02583-5 van de Schoot et al. (2021) van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., Oberski, D.L.: An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence 3(2), 125â133 (2021) https://doi.org/10.1038/s42256-020-00287-7 Van Dinter et al. (2022) Van Dinter, R., Tekinerdogan, B., Catal, C.: Predictive maintenance using digital twins: A systematic literature review. Information and Software Technology 151, 107008 (2022) https://doi.org/10.1016/j.infsof.2022.107008 Vaswani et al. (2023) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (arXiv:1706.03762) (2023) https://doi.org/10.48550/arXiv.1706.03762 White et al. (2025) White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., Shubh-Agrawal, Sandha, S.S., Naidu, S.V., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W., Goldblum, M.: LiveBench: A challenging, contamination-free LLM benchmark. In: The Thirteenth International Conference on Learning Representations (2025). https://doi.org/10.48550/arXiv.2406.19314 Wongpakaran et al. (2013) Wongpakaran, N., Wongpakaran, T., Gwet, K.: A comparison of Cohenâs Kappa and Gwetâs AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology 13 (2013) https://doi.org/10.1186/1471-2288-13-61 Zhang and Li (2020) Zhang, J., Li, J.: Testing and verification of neural-network-based safety-critical control software: A systematic literature review. Information and Software Technology 123, 106296 (2020) https://doi.org/10.1016/j.infsof.2020.106296 Zakeri-Nasrabadi et al. (2023) Zakeri-Nasrabadi, M., Parsa, S., Ramezani, M., Roy, C., Ekhtiarzadeh, M.: A systematic literature review on source code similarity measurement and clone detection: Techniques, applications, and challenges. Journal of Systems and Software 204, 111796 (2023) https://doi.org/10.1016/j.jss.2023.111796