Paper deep dive
How broad is that claim? Mapping Generalisation in NLP Research
Chenxin Diao, Nataliya Stepanova, Emily Allaway
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generalisations are common in scientific communication, even though they are semantically ambiguous. An automated method is needed to identify and categorise claims according to their level of generalisation, in order help detect an over-reliance on generalisations and possible misrepresentations of scientific findings. We introduce a comprehensive taxonomy of generalisations in the scientific domain, NLPGenX, which labels claims according to their level of generality and framing within the text. We operationalise this taxonomy with an LLM-powered framework, NLPGenA, that automatically classifies sentences from scientific articles into 5 different generalisation classes. We validate our framework with human annotators and use the framework to construct a large-scale dataset of NLP papers annotated according to generality, with auxiliary labels for hedging and vague descriptors (NLPGens). We use NLPGens to analyse the use of generalisations in NLP papers across multiple venues and subdomains, and to examine associations with citation counts, hedging, and vague descriptors.
Tags
Links
- Source: https://arxiv.org/abs/2609.14770v1
- Canonical: https://arxiv.org/abs/2609.14770v1
Trouble viewing inline? Open PDF directly â
Full Text
117,448 characters extracted from source content.
Expand or collapse full text
How broad is that claim? Mapping Generalisation in NLP Research Chenxin Diao Nataliya Stepanova Affiliation: School of Informatics, University of Edinburgh Email: chenxin.diao,nstepano,emily.allaway@ed.ac.uk Emily Allaway Abstract Generalisations are common in scientific communication, even though they are semantically ambiguous. An automated method is needed to identify and categorise claims according to their level of generalisation, in order help detect an over-reliance on generalisations and possible misrepresentations of scientific findings. We introduce a comprehensive taxonomy of generalisations in the scientific domain, NLPGenX, which labels claims according to their level of generality and framing within the text. We operationalise this taxonomy with an LLM-powered framework, NLPGenA, that automatically classifies sentences from scientific articles into 5 different generalisation classes. We validate our framework with human annotators and use the framework to construct a large-scale dataset of NLP papers annotated according to generality, with auxiliary labels for hedging and vague descriptors (NLPGens). We use NLPGens to analyse the use of generalisations in NLP papers across multiple venues and subdomains, and to examine associations with citation counts, hedging, and vague descriptors. The dataset and code are available at https://github.com/cx-diao/nlpgen. 1 Introduction Generalisations allow researchers to extrapolate from concrete findings to broader implications (Bunge, 1960; Peters et al., 2022). As a result, they are abundant in scientific communication, including in research articles (DeJesus et al., 2019). However, by obscuring variability, these generalisations may also exaggerate or overgeneralise findings beyond what is actually supported by evidence, leading to potential misinterpretations. Generalisations can take many forms in language (see Fig 1), with varying semantic interpretations (Haigh et al., 2020; Bowker, 2023). Notably, generics express generalisations without qualifying how broadly they apply. In both humans (Leslie et al., 2011) and large language models (LLMs) (Peters et al., 2025; Allaway et al., 2024), this has been shown to lead to overgeneralisation of a conclusion. While generalisations, particularly generics, have been extensively studied (Krifka et al., 1995a; Leslie, 2008; Carlson, 2010; Neufeld et al., 2025, e.g.,), most studies focus on relatively simple sentences. The resulting theoretical definitions are then either too abstract or syntactically limited (e.g., to only bare plural subjects) to translate directly to the scientific domain. Therefore, we propose a new taxonomy that captures differing forms of generalisations in science. Figure 1: Our taxonomy NLPGenX for classifying outcome-reporting sentences in the NLP domain. Our taxonomy (NLPGenX) unifies definitions from across multiple recent studies on generalisations in science (DeJesus et al., 2019; Peters et al., 2024; Peters and Lemeire, 2024). Focusing on sentences about results from the NLP domain, we define five categories that capture different degrees of generalisation. In scientific articles, very general claims are often embedded within study-specific framing (see Fig. 1c). We draw on DeJesus et al. (2019) to capture this in our taxonomy. Automatically identifying and classifying generalisations is challenging due to the complexity of both the definitions and of scientific language. While prior studies have manually identified generalisations in multiple domains (e.g., medical research, experimental philosophy; Peters et al., 2024; Peters and Lemeire, 2024), these relied on extensive manual annotation, limiting their scale. Additionally, systems to automatically identify generalisations have limited coverage, focusing only on a subset of generalisations that are identifiable through heuristic short-cuts â e.g., kind-referring plural noun phrases in the subject position (Doddington et al., 2004; Mitchell et al., 2003; Walker et al., 2006; Poesio et al., 2018; Cilleruelo et al., 2025). To facilitate large-scale automatic analysis, we propose a novel framework that uses LLMs to operationalise our taxonomy in the NLP domain. We considerably extend on previous automation efforts by providing an LLM-driven framework (NLPGenA) to categorise sentences following our taxonomy. Our framework uses nine consolidated annotation tasks: seven filter outcome sentences and derive taxonomy labels, while two auxiliary tasks identify hedging and vague descriptors. We collect human annotations to validate our framework, demonstrating substantial agreement between the LLM annotations and the human experts. We apply our framework to the 4.5K documents in the NLP subset of the NLPeer dataset (Dycke et al., 2023), a corpus of research articles in the NLP domain from 2017â2024. From this we create a new corpus, NLPGens, of 180K sentences filtered and then labelled according to our taxonomy, hedging, and vague descriptors. We then use NLPGens to study how generalisations are used in NLP literature. We conduct analyses on how the distribution of different generalisation types varies across venues and NLP sub-domains. We also conduct exploratory analyses into paper citation counts. We find that both hedging and vague descriptors are associated with generalisations in NLP papers. Our contributions are as follows: 1. A taxonomy of generalisations in NLP papers, NLPGenX. 2. A robust automated annotation framework, NLPGenA, consisting of classifiers to label the sentences according to our taxonomy. 3. A large-scale dataset, NLPGens, of sentences from NLP papers annotated for levels of genericity, hedging, and vague descriptors, obtained via our framework. 4. Analyses on generalisations in NLP papers, including their associations with other linguistic features (hedging and vague descriptors) and paper impact (measured through citation counts). 2 Prior Work Generalisations are common across languages (Behrens, 2005; Carlson and Pelletier, 1995), and are often classified as habitual or causal (Tessler and Goodman, 2019). Generic statements are a special subtype of generalisations that omit explicit quantification. They have garnered significant interest as they can be judged true based on weak evidence but have strong implications (Cimpian et al., 2010). Usage of generalisations and generics has implications for science, as the language used for communicating findings impacts the conclusions people draw (Peters et al., 2025). Labelling generalisations in language. Early annotation schemas for generalisations were restricted to labelling the genericity of noun phrases (Doddington et al., 2004; Mitchell et al., 2003; Walker et al., 2006). Later works also labelled the genericity of a clause, using event descriptions and co-reference taxonomies (Cybulska and Vossen, 2014; OâGorman et al., 2016; Friedrich et al., 2015; Friedrich et al., 2016) and classified events as episodic or habitual (Krifka et al., 1995b; Mathew, 2009; Smith, 2003). More recently, Govindarajan et al. (2019) presented a semantic framework for labelling predicates and their arguments. In contrast to these works, we categorise and annotate generalisations at the sentence level. Initial automated approaches explored the use of syntactic and grammatical features for annotating generalisations (Friedrich and Pinkal, 2015a; Friedrich and Pinkal, 2015b). More recent works have fine-tuned language models (Devlin et al., 2019) for automatic annotation (Bhakthavatsalam et al., 2020; Cilleruelo et al., 2025). However, these approaches require labelled data for training, which is difficult to obtain. So, we use zero-shot prompting with an instruction-tuned LLM for our annotation framework. Analysing scientific articles. The claims in scientific articles have been analysed for a number of aspects including discourse role (Fisas et al., 2015), citation purpose (Fisas et al., 2016), and argumentation relations (Lauscher et al., 2018). Prior works have also studied the relationship between scientific claims and news headlines about them (WĂŒhrl et al., 2024). These aspects relate primarily to the pragmatics of a claim and its role or relationships. In contrast, we focus on the semantics of claims by labelling their generality. Manual annotations have been conducted to label generalisations in scientific articles from psychology (DeJesus et al., 2019), medicine (Peters et al., 2024), and experimental philosophy (Peters and Lemeire, 2024). While these labelling schema had overlapping categories, such as hedging and framing, their definitions often included domain-specific elements or were designed to cover only a limited set of claims (e.g., only titles and abstracts). We define a novel taxonomy for generalisations that draws on these works while also providing concrete, actionable definitions. 3 NLPGenX: A Taxonomy of Scientific Outcome Generalisations Figure 2: Our framework for annotating generalisations in NLP papers (§4) using our proposed taxonomy (§3). Generalisations are claims whose interpretation supports inference beyond the particular observations explicitly reported in the sentence. Here we focus specifically on generalisations about the findings (outcomes) of a study. We categorise these generalisations along two dimensions (§4.5): framing and genericity. Genericity concerns how broadly a claim is interpreted as applying. Framing concerns how the generalisation is presented in relation to the study. These dimensions draw on prior work in psychology, medicine, and experimental philosophy (DeJesus et al., 2019; Peters et al., 2024; Peters and Lemeire, 2024), which we unify into our taxonomy. Their recurrence across fields suggests that NLPGenX may extend beyond NLP, but this requires domain-specific validation (see §Limitations). 3.1 Dimensions of Generalisation Framing. Inspired by (DeJesus et al., 2019), we take into account the framing of generalisations in our taxonomy. Framing is central to generalisation in scientific communication. Authors often embed a generalisation inside a reporting frame, such as âour results suggestâ or âthis study demonstratesâ. This frame then links the generalisation to the particular study (Fig, 1c and d). Framing may also take the form of references to specific tables or figures in the article. Genericity. Generalisations can be either generic or non-generic. Generic generalisations lack syntactic features (e.g., quantification) that signal how broadly they apply (e.g., Fig. 1b). As a result, generics are often interpreted overgenerally as holding broadly (or even universally) (Leslie et al., 2011). In contrast, non-generic generalisation include language that qualifies their scope, such as quantification (e.g., âNLPGenA is mostly reliableâ) or references to specific instances. 3.2 Taxonomic Categories We define five categories derived from the different types of generality and framing. The first level of our taxonomy differentiates between non-generalised and generalised/* claims, distinguished by how broadly they are implied to apply. Non-generalisations mention only specific results in the claim (Fig. 1e); whereas generalisations are implied to apply more broadly (e.g., Fig. 1b). The second level (*/framed/*) marks whether the scope of a generalisation is constrained through syntactic framing. Finally, the third level distinguishes */generic versus */non-generic generalisations. generalised/bare/generic. Bare generic generalisations are claims that express generalisations without any overt study frame nor explicit cues as to how broadly they apply (Fig. 1d). generalised/framed/generic. Framed generic generalisations consist of a generic claim embedded within explicit study framing. These claims present a broadly expressed finding within the context of the study (Fig. 1c). generalised/framed/non-generic. Framed non-generic generalisations present a finding as applying beyond specific instances but within the context of the study (e.g., âResultsâ in Fig. 1b). These claims also include explicit markers of how broadly the claim applies (e.g., Fig. 1b applies to scientific claims). generalised/bare/non-generic. Bare non-generic generalisations constrain the scope of the claim through means other than framing. That is, a constrained generalisation is presented without explicitly being presented as a finding of the current study (Fig. 1a). non-generalised. Non-generalised claims describe specific results or instances without drawing general conclusions about them (Fig. 1e). 4 NLPGenA: A Framework for Annotating Generalisations in NLP We propose a framework to annotate generalisations in NLP papers according to our proposed taxonomy (see §3). The framework uses nine consolidated annotation tasks: seven filter outcome sentences and derive taxonomy labels, while two auxiliary tasks identify hedging and vague descriptors respectively. Since our taxonomy is specifically designed for claims about outcomes, the first step of our framework labels sentences for this (§4.1). Then, we identify generalisations (§4.2) and, for all generalisations, determine whether they are framed in reference to the study (§4.3) and whether they contain generic language (§4.4). Independently of the taxonomy, we annotate hedging and vague descriptors (§4.5). Figure 2 shows our framework. See Appendix A for more details. Appendix E evaluates a deterministic rule-based counterpart to explore which annotation decisions require LLM-based semantic judgement. We find that although some of the annotation decisions can be made with deterministic rules, there are significant gaps in performance for annotation tasks that require semantic judgement. For consistency, we use LLM-based classification for all experiments, and we discuss the potential for rule-based classification in Section Limitations. 4.1 Is it about outcomes? The first step of our framework determines whether a sentence is about the outcomes of the study. Outcomes include not only experimental results and findings but also other scientific artefacts such as datasets and leaderboards. Following the definitions from Fisas et al. (2015), we label a sentence as being about outcomes if it is primarily about the authorâs own work and: âą Introduces or presents some scientific artefact (e.g., âWe presentâŠâ). âą Describes, interprets, or analyses experimental results (e.g., âThe results showâŠâ). âą Describes how the research will contribute to knowledge in the field. 4.2 Is it generalised? We label a claim as gen/* if it meets at least one of three conditions, combined in one classifier (is_generalised): âą Underspecified quantification (e.g., âmostâ, âoftenâ). We exclude quantifiers with specific thresholds (e..g, â50% ofâ, âallâ) because these refer to specific instances and therefore do not make generalisations. âą Expressions of causality (e.g., âresults inâ). âą Only a minor change in meaning, in context, after the insertion âusuallyâ or âgenerallyâ. The first two conditions capture quantified, habitual, or causal generalisations in line with the distinctions made in prior work (Tessler and Goodman, 2019). The last condition captures potentially generic generalisations following a test proposed by Krifka et al. (1995b). 4.3 Is it framed? If a claim is generalised, we use two classifiers to determine whether it is framed. A */framed/* claim should refer to the study in some way (§3.1). This can be through a reference to the study population (ref_study_population), the broader domain most closely related to the the specific instances that were studied (e.g., âannotatorsâ, âtested datasetsâ). Or it can be through more general references to the study (ref_study), such as mentioning the authors (e.g., âwe findâ), the study results (e.g., âThe results showâ), or tables and figures (e.g., âsee Table Nâ). Claims without a reference to the study are unframed. 4.4 Is it generic? Following Krifka et al. (1995b); DeJesus et al. (2019), we define a generic claim (using three classifiers) as meeting all of the following conditions âą Is in the present tense (is_present_tense): as noted in DeJesus et al. (2019), past tense restricts the scope of a claim to the specific instances from the study. âą Does not have quantification (quantified): this is a hallmark of generic sentences (Krifka et al., 1995b) because quantifiers signal how broadly a claim applies. âą Does not refer to the study population (§4.3): framing may reference the study itself but not the sample or population because this constrains the scope of the claim. Unframed claims that meet these conditions are gen/bare/generic. For framed claims, the framing is removed (removable_framing) before determining genericity. This is done by determining whether the study framing can be separated from a complete embedded sentence11 1 We use three separate prompts for this but aggregate them for simplicity. See Appendix D for full details. For example, âour results show that attention is usefulâ can be (and in fact is) a gen/framed/generic; in contrast, âour results show the effectiveness of our methodâ can not be generic because removing the framing leaves only a nominal phrase. If the embedded sentence meets the conditions of genericity, then it is a gen/framed/generic. 4.5 Hedging and Vague Descriptors We use two classifiers to annotate auxiliary linguistic attributes independently of the taxonomy: hedging (contains_hedging) and vague descriptors (is_vague). Neither attribute contributes to the final taxonomy label. Hedging is the use of linguistic expressions (e.g., âlikelyâ) to show that the author is not fully committed to a claim (Lakoff, 1973). It is common in academic writing, where authors use it to temper certainty and allow for alternatives and criticism (Hyland, 1994). We allow both generic and non-generic claims to be labelled as hedged, following DeJesus et al. (2019). We define a vague descriptor as an under-specified adjective or adverb whose degree is left unclear, such as âsignificantâ, âgoodâ, or âefficientâ without an explicit threshold. venue # papers # sents # section sents # outcome sents acl â17 133 26K 8.9K 4.1K coling â20 89 16K 6.3K 2.9K arr â22 473 97K 42,45 19K emnlp â23 2.0K 413K 133K 68K emnlp â24 1.7K 440K 205K 87K Total 4.5K 991K 396K 180K Table 1: Statistics of NLPGens. âSentsâ is sentences. Section sents are sentences from only relevant paper sections (abstract, introduction, results, discussion, conclusion; §5.1). Outcome sents are sentences labelled as being about the study outcomes (§4.1). 5 NLPGens: A Large-Scale Dataset of Generalisations in NLP Research 5.1 Implementation Source data. We source NLP papers from the NLPeer corpus (Dycke et al., 2023). We use all NLP subsets of both v1 and v2 of the corpus (see Table 1). We use keywords to identify and extract all sentences from five standard paper sections: Abstract, Introduction, Results, Discussion, and Conclusion. See Appendix A for details. LLM setup. We construct a separate prompt for each of the tasks following the questions described in §4 (see Fig. 2) and query gpt-oss-120b (Agarwal et al., 2025) with temperature =0=0 three times 22 2 Since we used vLLM (Kwon et al., 2023) to host the LLM, even with temperature =0=0, the output is not deterministic.. We take the majority vote for each classifier and aggregate following a decision tree to acquire the final LLM label. For determining whether a sentence is about outcome, generalised, and whether it refers to a study population, we provide the previous two sentences as context in the prompt. See Appendix L for full prompts and details. 5.2 Dataset Statistics Our dataset contains 180,426180,426 fully labelled about_outcome sentences distributed across the five taxonomy classes. On average, 51%51\% of sentences in a paper were labelled as about outcomes (see Appendix Table 10 for specifics). Each sentence also carries the auxiliary contains_hedging and is_vague annotations. As shown in Table 3, our dataset is dominated by gen/framed/non-generic (54.6%54.6\%) and non-gen (31.6%31.6\%), with the three remaining classes together accounting for 13.8%13.8\% of the labelled sentences. We present detailed analysis in §6. Final label Example non-gen CrossAligner exceeds the F-Score of the Previous SOTA by 2.7 points (82.5 versus 79.8). gen/framed/non-generic We believe that our entropy-based perspective will help provide a strong starting point for this pursuit. gen/framed/generic We believe the improvement is from the accurate recognition of these sentences. gen/bare/non-generic Most researchers do not take advantage of recent metrics that correlate better with human judgments. gen/bare/generic HiTab also presents cross-domain and complicated calculation challenges. Table 2: Examples from our NLPGens corpus (§5). See Appendix Table 9 for more examples. Full Corpus End-to-End Annotated Subset Final label n % label dist âĄ()R(L) n_G correct acc non-gen 57050 31.6 1.00 29 28 0.966 gen/framed/non-generic 98449 54.6 0.95 60 58 0.967 gen/bare/generic 15852 8.8 0.94 3 3 1.000 gen/bare/non-generic 6956 3.9 0.92 5 4 0.800 gen/framed/generic 2119 1.2 0.75 3 3 1.000 Total 180426 100.0 - 100 96 0.960 Table 3: Label statistics and annotator validation for our corpus. n is the number (% label dist is the percentage) of about_outcome sentences in the corpus assigned each label by our framework. nGn_G is the number of sentences for each final label where all 9 classifier labels were validated by humans, correct is the number where LLM-label was validated as correct and acc is the corresponding LLM accuracy. RâĄ(L)R(L) is data-weighted per-label reliability (Eq. 1). 5.3 Human Validation Setup We evaluate our annotation framework with two human protocols over the same sampled items. In human verification, annotators see the frameworkâs output and reasoning and judge whether the output is correct. In independent annotation, annotators receive the same inputs and task instructions but assign the label themselves without seeing the frameworkâs answer. Verification asks the narrower question of whether our framework output is defensible under the instructions, allowing annotators to accept edge cases for which more than one answer is plausible; we revisit such cases in Appendix F. For each of the 11 underlying annotation modules, we sample 200 rows from the corpus. Where applicable, we balance the framework label (100 positive and 100 negative) so that the negative class is not swamped by the prevalent positive class. One hundred outcome-positive rows are shared across the seven taxonomic modules and the two auxiliary attributes, giving an evaluation subset with consistent coverage; the conditionally invoked get_target_sent module is sampled separately. The full sampling procedure is detailed in Appendix B. Each item under each protocol is examined by three annotators with at least undergraduate-level background in computer science. Annotators received approximately thirty minutes of training per classifier and were paid $18.28 per hour for both training and annotation. See Appendix B for details. Verification Independent Classifier Acc α Acc α Is it about outcomes? (§4.1) about_outcome 0.990 0.853 0.870 0.619 Is it generalised? (§4.2) is_generalised 0.990 0.760 0.765 0.065 Is it framed? (§4.3) ref_study 0.975 0.853 0.905 0.713 ref_study_population 0.980 0.800 0.765 0.428 Is it generic? (§4.4) is_present_tense 0.985 0.933 0.980 0.933 quantified 0.985 0.900 0.880 0.670 removable_framing 33 3 For removable_framing, α measures only the Boolean gate derived from is_reporting_clause and sentence_embedded; unlike Acc, it does not include target-sentence extraction quality. See Appendix D. 0.970 0.967 0.870 0.640 Auxiliary attributes (§4.5) contains_hedging 0.995 0.833 0.875 0.558 is_vague 0.975 0.771 0.710 0.090 Table 4: Acc is majority-to-framework agreement: under verification, it is the proportion of items for which the majority judged the framework output correct; under independent annotation, it is the proportion for which the majority-assigned label matches the framework. We also report nominal Krippendorffâs α. Verification α uses reconstructed verifier-implied labels; independent α uses directly assigned labels. See Appendix C for details. Agreement and accuracy. Table 4 summarises the results. We use Acc for agreement between the human majority vote and our framework. Under the verification setting, this is the proportion of items for which most annotators judged the framework output correct. Under independent annotation, it is the proportion for which the majority of directly assigned labels matches the framework output. For verification, we reconstruct the label implied by each annotator: an accepted framework label is retained and a rejected Boolean label is negated. We compute Krippendorffâs α (Krippendorff, 1989) over these individual reconstructed labels. For independent annotation, we compute α directly over the individual labels assigned without seeing the framework output. Under verification, accuracy is at least 0.9700.970 for every classifier and α ranges from 0.7600.760 to 0.9670.967. Independent annotation produces lower accuracy (0.7100.710â0.9800.980), while its α varies substantially across modules (0.0650.065â0.9330.933). In particular, agreement is very low for is_generalised (α=0.065α=0.065) and is_vague (α=0.090α=0.090), moderate for ref_study_population (α=0.428α=0.428), and high for is_present_tense (α=0.933α=0.933). Thus, the independent annotations do not provide an equally stable reference for every semantic classifier. We therefore use verification as our primary reliability result and treat independent annotation as a diagnostic. The verification scores measure whether the framework outputs are defensible under our taxonomy, showing that the LLM annotations are reliable. Note that the LLM annotations are obtained using a single off-the-shelf open-weight model and do not represent an upper bound on framework performance. See Appendix B and Appendix C for more details. Per-label reliability. The verification accuracy captures the reliability of a single classifier, but the final taxonomic label for a sentence is determined through a decision tree that combines multiple classifiers (see Fig. 2). We therefore translate the per-classifier validation numbers into a per-label reliability score: an estimate of the probability that an LLM annotation carrying a given final label is correct, composed from per-classifier accuracy scores. Let L denote the final label from our taxonomy. We define the data-weighted reliability for label L by summing over paths pâLp _L, the set of paths through the decision tree that terminate at label L RâĄ(L)=âpâLnp,LnLâPâ(Lâis correct|p)R(L)\;=\; _p _L n_p,Ln_LP(L\ is correct|p) (1) where nLn_L is the number of claims in the data with final label L and nL,pn_L,p is the number of those claims that were produced with path p. We can compute the conditional probability as the path reliability PâĄ(Lâis correct|p)=âcâpPâĄ(vcâis correct)P(L\ is correct|p)\;=\; _câ pP(v_c\ is correct) (2) where c is a classifier and vcv_c is the value predicted by that classifier. The individual per-classifier probabilities are the per-classifier accuracies computed from the verification judgements. Table 3 shows the per-label reliability scores (see Appendix B for a detailed breakdown). Every final label is therefore accompanied by a quantitative reliability estimate, which downstream analyses can use to weight or filter the annotated data. 6 Analysis Using our NLPGens corpus, we conduct five analyses into how generalisations are used in NLP research papers. Note that the analysis is limited to the sentences annotated as about_outcome (the first filtering step for genericity annotation). See Appendix G for the distribution of about_outcome sentences per section across venues. 6.1 Trends Across Sections Figure 3: Class distribution of about_outcome sentences across all venues, separated by section and venue. We probe at whether the distribution of generalisations changes per paper section. Across different venues, the most abundant class was gen/framed/non-generic (Fig. 3). Interestingly, this class seems to grow slightly over the years, but further fine-grained analysis is needed to determine why. The overall ratio of gen/* to non-gen sentences is around 60:40, with the most generalisations found in the discussion. We observe the most */generic claims in the introduction and the fewest in the abstract and results sections. The majority of these are not framed generalised/bare/non-generic. 6.2 Trends Across Tracks Figure 4: Class distribution of about_outcome sentences in COLING â20 papers, separated by track. We next probe at whether different sub-topics of NLP research result in different rates of generalisations. Out of the 5 venues represented in the dataset, only the COLING-2020 papers (89) contained track information. See Appendix H for experiments with assigning soft track labels to other venues. We visualise the per-track distributions of gen labels for COLING-2020 papers in Figure 4. The Applications and Summar-Simpl (Summarization and Simplification) tracks contain the most non-gen sentences. The Applications class also has one of of the lowest rates of bare generics overall. This makes intuitive sense - Application papers are usually constrained to a specific subfield (e.g., BioNLP, Social Media), limiting the usage of broad statements. We also see a large difference in the rates of non-gen across the tracks, with almost half of the sentences in the Summar-Simpl track being non-gen while only 15% of the CogMod (Cognitive Modelling) track. We leave a thorough investigation for the reasons behind track-level generalisation differences for future work. 6.3 Generalisations and Citation Count We next probe at the correlation between a paperâs impact (approximated through citation count) and its usage of generalising language. We retrieve citation counts using SemanticScholar (see Appendix I for details). For each paper, we compute the distribution for each category over all about_outcome sentences. In addition to the five categories in our taxonomy, we consider three special groupings: total generalisations (all gen/* categories), total generics (both */generic categories), total framed generalisations (both */framed/* categories). We then calculate the Spearman rho (Ï) correlation between these eight types (five original labels and three additional) and the total citation count of the paper. We perform this analysis per venue. Category Ï p-value non-gen -0.15 0.173 gen/framed/non-generic 0.29 0.005 gen/bare/generic -0.29 0.007 gen/bare/non-generic -0.06 0.604 gen/framed/generic -0.08 0.484 */generic -0.28 0.008 gen/* 0.15 0.176 gen/framed/* 0.28 0.008 Table 5: Spearman Ï correlation between the generalisation classes and citation counts for ACL â17. Statistically significant effects (p-value < 0.05) in bold. Predicting citation counts is a hard task, especially when using a single linguistic feature (generalisability) without any further information about semantic content. Nevertheless, we observe a statistically signification correlation for ACL â17 (Table 5), with Spearman Ï of almost 0.3 for gen/framed/non-generic sentences. This relationship is similar to that observed by Peters et al. (2024). Intuitively, stronger papers effectively generalise over their findings, framing and scoping their claims when necessary to prevent over-claiming. We find similar (but not significant) trends across other venues (see Appendix I), indicating this topic requires further study. 6.4 Hedging tracks generalisation Category % hedged % of hedged % vague % of vague non-gen 14.6 15.1 64.8 27.2 gen/framed/non-generic 36.7 65.5 80.5 58.3 gen/bare/generic 40.2 11.5 80.2 9.3 gen/bare/non-generic 49.2 6.2 76.7 3.9 gen/framed/generic 41.5 1.6 80.7 1.3 */generic 40.3 13.1 80.2 10.6 gen/* 37.9 84.9 80.2 72.8 gen/framed/* 36.8 67.1 80.5 59.5 Table 6: Hedging and vague-descriptor prevalence and composition for all 180K outcome sentences from five venues. The five taxonomy labels partition the corpus; aggregate categories overlap. We next probe whether the use of hedging is associated with generalised claims about outcomes. Table 6 reports hedging prevalence and composition for the five taxonomy labels and the three aggregate categories. Across all 180K fully labelled about_outcome sentences in our corpus, hedging is roughly 2.6 times more frequent in generalised than in non-generalised sentences (37.9% vs. 14.6%). The inverse direction is even sharper: of all hedged claims about outcomes, around 84.9% are also annotated as generalised, indicating that hedging is a strong (though not perfect) surface cue for generalisation. The per-label breakdown refines this picture. Among generalised labels, the highest hedging rate is on gen/bare/non-generic at 49.2%, while gen/framed/non-generic has the lowest at 36.7%. That is, when the generalising frame is made explicit, authors hedge less; bare generalisations that lack a generic noun phrase attract the most hedging, plausibly because the epistemic load is otherwise unmarked. 6.5 Vague descriptors track generalisation Finally, we examine is_vague, the auxiliary attribute for under-specified adjectives and adverbs introduced in §4.5. It is labelled independently from the classifiers that determine the generalisation label. Across all five venues, 72.8%72.8\% of outcome sentences containing a vague descriptor are labelled as generalised; the standard deviation across the five venue-specific proportions is 1.4 percentage points. Table 6 shows the association in both directions. Vague descriptors occur in 80.2%80.2\% of generalised sentences, compared with 64.8%64.8\% of non-generalised sentences. The odds that a generalised sentence contains a vague descriptor are therefore 2.20 times those for a non-generalised sentence. The association is similar across the five venues. We treat this co-occurrence as exploratory. Independent majority agreement with the framework is only 0.7100.710 for is_vague, with low chance-corrected agreement (α=0.090α=0.090; Table 4). Moreover, the vague-descriptor and generalisation labels are produced by separate prompts but the same LLM pipeline, so shared annotation artefacts may contribute to the measured association. 7 Conclusion We present a novel taxonomy NLPGenX for classifying the generality of sentences communicating scientific findings in NLP research. We also provide a fully automated annotation framework NLPGenA that operationalises the classification of a sentence into one of our five generalisation categories using an ensemble of linguistics-inspired classifiers. To our knowledge, NLPGenA is the first fully automated solution leveraging instruction-tuned LLMs for annotating generalisation in language, as prior works have either been manual or focused on smaller scale experiments. We use our NLPGenA to also construct NLPGens, a dataset of 180K NLP claims annotated for generalisation according to our taxonomy NLPGenX, as well as for hedging and vague descriptors. Again, to our knowledge, this is the first large-scale dataset annotating generalisation in scientific claims. Our analyses on NLPGens reveal preliminary patterns that warrant further investigation: the relative percentage of generalisations has grown over time, NLP sub-domains have slightly different distributions, the relative rate of generalisations may be correlated with a paperâs citation count, and both hedging and vague descriptors are associated with generalised claims. Limitations Our taxonomy is developed and validated specifically for sentences describing outcomes in NLP papers. While this covers a large portion of claims in a paper, there can also be claims about the work of other authors. These background claims (Lauscher et al., 2018) are not captured by our taxonomy. Additionally, the semantics of generics are highly nuanced (Krifka et al., 1995b, e.g.,). Combining this with the complexity of many scientific sentences means that some edge cases may not be accounted for by our taxonomy. This includes nested claims, which may have components from different categories, and pronoun ambiguity. The use of computer graphics papers during initial development (Fisas et al., 2015; Fisas et al., 2016), together with the cross-domain foundations discussed in §3, suggests potential applicability to other empirical scientific domains. This does not establish cross-domain validity; differences in terminology and discourse conventions, particularly in less constrained genres (e.g., broad social-science discourse), may require adaptation and separate validation. NLPGenA uses LLM prompting for annotation. While we validate the quality of this framework, the nature of LLMs limits its explainability, reproducibility, and efficiency. Our rule-based baseline (Appendix E) approaches NLPGenA on several surface-oriented decisions, but substantial performance gaps remain for tasks requiring semantic judgement and for end-to-end classification. We therefore use LLM-based classification consistently across all experiments, while future work could explore hybrid frameworks that replace suitable modules with deterministic rules to improve reproducibility and reduce computational cost. Acknowledgements We would like to thank the anonymous reviewers for their valuable suggestions. The authors also thank Huawei Technologies Co. for their generous support towards this research. References Agarwal et al. (2025) S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, et al. Gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Cited by: Appendix A, §5.1. Allaway et al. (2024) E. Allaway, C. Bhagavatula, J. D. Hwang, K. McKeown, and S. Leslie Exceptions, instantiations, and overgeneralization: insights into how language models process generics. Computational Linguistics 50 (4), p. 1211â1275. Cited by: §1. Behrens (2005) L. Behrens Genericity from a cross-linguistic perspective.. Linguistics 43 (2). Cited by: §2. Bhakthavatsalam et al. (2020) S. Bhakthavatsalam, C. Anastasiades, and P. Clark Genericskb: a knowledge base of generic statements. arXiv preprint arXiv:2005.00660. Cited by: §2. Bowker (2023) M. Bowker A problem for generic generalisations in scientific communication. Journal of Applied Philosophy 40 (1), p. 123â132. Cited by: §1. Bunge (1960) M. Bunge The place of induction in science. Philosophy of Science 27 (3), p. 262â270. Cited by: §1. G. N. Carlson and F. J. Pelletier (Eds.) (1995) G. N. Carlson and F. J. Pelletier (Eds.) The generic book. University of Chicago Press. Cited by: §2. G. N. Carlson (Ed.) (2010) G. N. Carlson (Ed.) The generic book. Digit. print. edition, Univ. of Chicago Press, Chicago. Cited by: §1. Cilleruelo et al. (2025) G. Cilleruelo, E. Allaway, B. Haddow, and A. Birch MGEN: millions of naturally occurring generics in context. Society for Computation in Linguistics 8 (1). Cited by: §1, §2. Cimpian et al. (2010) A. Cimpian, A. C. Brandone, and S. A. Gelman Generic statements require little evidence for acceptance but have powerful implications. Cognitive science 34 (8), p. 1452â1482. Cited by: §2. Cybulska and Vossen (2014) A. Cybulska and P. Vossen Guidelines for ecb+ annotation of events and their coreference. Cited by: §2. DeJesus et al. (2019) J. M. DeJesus, M. A. Callanan, G. Solis, and S. A. Gelman Generic language in scientific communication. Proceedings of the National Academy of Sciences 116 (37), p. 18370â18377. External Links: https://w.pnas.org/doi/pdf/10.1073/pnas.1817706116, Document Cited by: §1, §1, §2, §3.1, §3, 1st item, §4.4, §4.5. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 4171â4186. External Links: Document Cited by: §2. Doddington et al. (2004) G. R. Doddington, A. Mitchell, M. A. Przybocki, L. A. Ramshaw, S. M. Strassel, R. M. Weischedel, et al. The automatic content extraction (ace) program-tasks, data, and evaluation.. In Lrec, Vol. 2, p. 837â840. Cited by: §1, §2. Dycke et al. (2023) N. Dycke, I. Kuznetsov, and I. Gurevych NLPeer: a unified resource for the computational study of peer review. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, p. 5049â5073. External Links: Link Cited by: Appendix J, Appendix J, Appendix K, §1, §5.1. Fisas et al. (2016) B. Fisas, F. Ronzano, and H. Saggion A multi-layered annotated corpus of scientific papers. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LRECâ16), N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis (Eds.), PortoroĆŸ, Slovenia, p. 3081â3088. External Links: Link Cited by: §2, Limitations. Fisas et al. (2015) B. Fisas, H. Saggion, and F. Ronzano On the discoursive structure of computer graphics research papers. In Proceedings of the 9th Linguistic Annotation Workshop, A. Meyers, I. Rehbein, and H. Zinsmeister (Eds.), Denver, Colorado, USA, p. 42â51. External Links: Link, Document Cited by: §2, §4.1, Limitations. Friedrich et al. (2016) A. Friedrich, A. Palmer, and M. Pinkal Situation entity types: automatic classification of clause-level aspect. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1757â1768. Cited by: §2. Friedrich et al. (2015) A. Friedrich, A. Palmer, M. P. SĂžrensen, and M. Pinkal Annotating genericity: a survey, a scheme, and a corpus. In Proceedings of the 9th linguistic annotation workshop, p. 21â30. Cited by: §2. Friedrich and Pinkal (2015a) A. Friedrich and M. Pinkal Automatic recognition of habituals: a three-way classification of clausal aspect. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, p. 2471â2481. Cited by: §2. Friedrich and Pinkal (2015b) A. Friedrich and M. Pinkal Discourse-sensitive automatic identification of generic expressions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 1272â1281. Cited by: §2. Govindarajan et al. (2019) V. Govindarajan, B. Van Durme, and A. S. White Decomposing generalization: models of generic, habitual, and episodic statements. Transactions of the Association for Computational Linguistics 7, p. 501â517. Cited by: §2. Haigh et al. (2020) M. Haigh, H. A. Birch, and T. V. Pollet Does âscientists believeâŠâimply âall scientists believeâŠâ? individual differences in the interpretation of generic news headlines. Collabra: Psychology 6 (1), p. 17174. Cited by: §1. Hyland (1994) K. Hyland Hedging in academic writing and eaf textbooks. English for Specific Purposes 13 (3), p. 239â256. External Links: Document, Link Cited by: §4.5. Krifka et al. (1995a) M. Krifka, F. J. Pelletier, G. Carlson, A. ter Meulen, G. Chierchia, and G. Link Genericity: an introduction. In The Generic Book, G. N. Carlson and F. J. Pelletier (Eds.), p. 1â124. Cited by: §1. Krifka et al. (1995b) M. Krifka, F. J. Pelletier, G. Carlson, A. Ter Meulen, G. Chierchia, and G. Link Genericity: an introduction. Cited by: §2, 2nd item, §4.2, §4.4, Limitations. Krippendorff (1989) K. Krippendorff Content analysis. International encyclopedia of communication (226), p. 403â407. Cited by: §5.3. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: Appendix A, footnote 2. Lakoff (1973) G. Lakoff Hedges: a study in meaning criteria and the logic of fuzzy concepts. Journal of philosophical logic 2 (4), p. 458â508. Cited by: §4.5. Lauscher et al. (2018) A. Lauscher, G. GlavaĆĄ, S. P. Ponzetto, and K. Eckert Investigating the role of argumentation in the rhetorical analysis of scientific publications with neural multi-task learning models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, p. 3326â3338. External Links: Link, Document Cited by: §2, Limitations. Leslie et al. (2011) S. Leslie, S. Khemlani, and S. Glucksberg Do all ducks lay eggs? the generic overgeneralization effect. Journal of Memory and Language 65 (1), p. 15â31. Cited by: §1, §3.1. Leslie (2008) S. Leslie Generics: cognition and acquisition. The Philosophical Review 117 (1), p. 1â47. External Links: Document Cited by: §1. Mathew (2009) T. A. Mathew Supervised categorization for habitual versus episodic sentences. Georgetown University. Cited by: §2. Mitchell et al. (2003) A. Mitchell, S. Strassel, M. Przybocki, J. Davis, G. Doddington, R. Grishman, A. Meyers, A. Brunstein, L. Ferro, and B. Sundheim ACE-2 version 1.0. Linguistic Data Consortium, Philadelphia. Cited by: §1, §2. Neufeld et al. (2025) E. Neufeld, A. Bosse, G. Del Pinal, and R. Sterken Giving generic language another thought. Wiley Interdisciplinary Reviews: Cognitive Science 16 (1), p. e70000. Cited by: §1. OâGorman et al. (2016) T. OâGorman, K. Wright-Bettner, and M. Palmer Richer event description: integrating event coreference with temporal, causal and bridging annotation. In Proceedings of the 2nd workshop on computing news storylines (CNS 2016), p. 47â56. Cited by: §2. Peters et al. (2025) U. Peters, A. Bertazzoli, J. M. DeJesus, G. J. van der Velden, and B. Chin-Yee Generics in science communication: misaligned interpretations across laypeople, scientists, and large language models. Public Understanding of Science, p. 09636625261425891. Cited by: §1, §2. Peters et al. (2022) U. Peters, A. Krauss, and O. Braganza Generalization bias in science. Cognitive science 46 (9), p. e13188. Cited by: §1. Peters and Lemeire (2024) U. Peters and O. Lemeire Hasty generalizations are pervasive in experimental philosophy: a systematic analysis. Philosophy of Science 91 (3), p. 661â681. Cited by: §1, §1, §2, §3. Peters et al. (2024) U. Peters, H. R. Sherling, and B. Chin-Yee Hasty generalizations and generics in medical research: a systematic review. Plos one 19 (7), p. e0306749. Cited by: §1, §1, §2, §3, §6.3. Poesio et al. (2018) M. Poesio, Y. Grishina, V. Kolhatkar, N. S. Moosavi, I. Roesiger, A. Roussel, F. Simonjetz, A. Uma, O. Uryupina, J. Yu, et al. Anaphora resolution with the arrau corpus. In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference, p. 11â22. Cited by: §1. Reiter and Frank (2010) N. Reiter and A. Frank Identifying generic noun phrases. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, J. HajiÄ, S. Carberry, S. Clark, and J. Nivre (Eds.), Uppsala, Sweden, p. 40â49. External Links: Link Cited by: Appendix E. Singh et al. (2022) A. Singh, M. DâArcy, A. Cohan, D. Downey, and S. Feldman SciRepEval: a multi-format benchmark for scientific document representations. In Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: Appendix H. Smith (2003) C. S. Smith Modes of discourse: the local structure of texts. Vol. 103, Cambridge University Press. Cited by: §2. Tessler and Goodman (2019) M. H. Tessler and N. D. Goodman The language of generalization.. Psychological review 126 (3), p. 395. Cited by: §2, §4.2. Walker et al. (2006) C. Walker, S. Strassel, J. Medero, and K. Maeda Ace 2005 multilingual training corpus. (No Title). Cited by: §1, §2. WĂŒhrl et al. (2024) A. WĂŒhrl, D. Wright, R. Klinger, and I. Augenstein Understanding fine-grained distortions in reports of scientific findings. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand. Cited by: §2. Appendix A LLM Annotation Details Sentence extraction and coverage check. Each NLPeer paper is supplied as a structured ITG JSON file that enumerates headings, paragraphs, list items, and formulas in reading order. We segment every paragraph into sentences with spaCy44 4 https://spacy.io/ and retain only those drawn from sections whose heading matches one of the keywords abstract, introduction, result, discussion, conclusion, or ablation. To guard against silent drops from section detection or sentence segmentation, we run a per-paper coverage check on the Introduction: every non-empty token of the raw Introduction text must reappear, after punctuation-and-whitespace normalisation, within the union of the extracted sentences. Papers that fail this check are quarantined and excluded from downstream annotation; the remaining papers proceed with a CSV listing the selected sentences, keyed by (paper, section, within-paper sentence index), which serves as the input to the annotation pipeline described above. Full decision tree and data flow. Sentence input claim two_sent_context about_outcome return skip downstream sentence_embedded is_reporting_clause get_target_sent iff both True get_target_sent_label sentence with frame removed is_generalised ref_study ref_study_population is_present_tense quantified final_label rule-based merge (Alg. 1) contains_hedging is_vague FalseTruetwo_sent_contextTrueremovable_framing(on claim)taxonomic classifiers(on get_target_sent_label) â _labelauxiliary attributes on raw claimoutside decision tree Figure 5: Data flow across the annotation pipeline (one sentence at a time). The dashed right branch records auxiliary attributes that do not feed final_label. Algorithm 1 Get final taxonomy label 1: Boolean labels from the seven taxonomic classifiers (§B), given the about_outcome label is true 2: Final label â 3: if is_generalised is false then 4: ââ â non-gen 5: else 6: if ref_study_population is true then 7: ââ â gen/framed/non-generic 8: else if ref_study is true then 9: if is_present_tense is false then 10: ââ â gen/framed/non-generic 11: else if is_reporting_clause is false then 12: ââ â gen/framed/non-generic 13: else if sentence_embedded is false then 14: ââ â gen/framed/non-generic 15: else if quantified is true then 16: ââ â gen/framed/non-generic 17: else 18: ââ â gen/framed/generic 19: end if 20: else 21: if is_present_tense is false then 22: ââ â gen/bare/non-generic 23: else if quantified is true then 24: ââ â gen/bare/non-generic 25: else 26: ââ â gen/bare/generic 27: end if 28: end if 29: end if 30: return â A more fine-grained decision tree is shown in Algorithm 1. Figure 5 summarises how classifier outputs are chained before the final label is computed. about_outcome gates the rest: when False, no other LLM call is made for the sentence. get_target_sent only calls the LLM when both is_reporting_clause and sentence_embedded are True; otherwise get_target_sent_label is set to the original claim. The five taxonomic classifiers all read get_target_sent_label; the blue dashed path marks the additional two_sent_context input used by is_generalised and ref_study_population. These five classifiers feed the rule-based final_label of Algorithm 1. contains_hedging and is_vague run on the raw claim and sit outside the decision tree. Hardware and performance. We ran experiments on a mixture of NVIDIA A100 (80 GB), H100 (80 GB), and H200 (80 GB) GPUs, serving gpt-oss-120b (Agarwal et al., 2025) via vLLM (Kwon et al., 2023) with asynchronous requests. As a rough throughput estimate, annotating a 100-sentence paper with num_passes=1num\_passes=1 on a 4ĂH200 node takes approximately 10 minutes, though this figure varies with the fraction of sentences that pass the about_outcome gate and with the inherent variability of asynchronous, parallelised inference. Annotation statistics. As introduced in Section 5, each about_outcome sentence is annotated three times by the LLM, and the categorical classifier outputs are merged by majority vote across the three passes, with pass 1 as the tie-breaker. Across the ten Boolean modules reported in the paper, this produces 2,019,5502,019,550 majority-voted classifier cells (gated by about_outcome for downstream classifiers). Only 1313 remain missing and 9494 are tied after the merge: 1111 sentences across 77 papers are incomplete, and a further 9494 sentences across 9494 papers carry a tie-broken label. Missingness is already low before merging (830830 empty cells summed across the three passes, âŒ0.01% 0.01\%), so the vote serves chiefly to suppress single-pass failures rather than to compensate for systematic gaps. We conclude that the annotated corpus is effectively complete at scale. Appendix B Human Verification Details Human verification. We recruited seven annotators with at least undergraduate-level training in computer science and prior experience providing teaching support. All annotators were familiar with the NLP field and with scientific writing style. In this verification protocol, annotators did not relabel sentences from scratch. Instead, they audited the LLM pipeline: for each classifier, they judged whether the modelâs decision was correct, given the same inputs available to the model (§ 5.3). Each sampled row was judged independently by three annotators. Assignment sheets for each annotation step were identical across annotators, ensuring that every row received three votes. Before annotation, annotators received approximately 3030 minutes of training for each annotation step. During training, they reviewed the corresponding prompt template (Appendix L) and were instructed to judge each LLM label strictly according to that template. The workbooks themselves contained only the instantiated inputs and outputs for each sampled row. For each classifier, we distributed one Excel workbook per annotator, with one file per classifier. Each workbook contained the following columns: âą Row identifiers: corpus split, paper path, and within-paper sentence index. âą Sentence context matching the prompt placeholders for that classifier: typically the original sentence claim or the extracted framed sentence get_target_sent_label, sometimes supplemented with a two-sentence local context two_sent_context. âą The LLM label and its free-text reasoning. The annotators were asked to judge each LLM label strictly according to the corresponding prompt template, and fill in the following fields: âą A mandatory True/False field recording whether the LLM decision is correct. âą An optional free-text comments field. Annotators were paid equivalent to US$18.28 per hour for both training and annotation. They are informed that the annotation task is part of a research project and that their contributions will be used to evaluate the performance of a annotation framework, and the data will be made available to the public. The data collection protocol was reviewed and approved by our institutionâs research ethics review board prior to recruitment. Human verification set sampling. Population df_all_selected 5 splits â · completed rows Anchor S 200 rows â · about_outcome 100 T / 100 F about_outcome export 200 rows =S=S cohort_true 100 about_outcome-True rows in S (shared core: 9 exports) Pool: about_outcome True 7 taxonomic + 2 auxiliary exports 200 rows â · 100 T / 100 F cohort_true + top-up Pool: gate-cond. eligible (is_reporting_clause ⧠sentence_embedded) get_target_sent export 200 rows partial cohort_true + top-up Figure 6: Human-validation sampling pipeline (seed 42). Anchor S stratifies on about_outcome; cohort_true is the 100-row core reused in the seven taxonomic exports plus contains_hedging and is_vague, but is not the direct core for get_target_sent. For get_target_sent, a dashed arrow shows cohort_true feeding the module-specific export via conditional filtering; a solid arrow then shows the pool as the primary sampling source. For taxonomic exports, a solid arrow shows direct cohort-core inclusion; a dashed arrow shows external top-up from the about_outcome-True eligible pool. We draw verification rows from the merged annotated corpus: all sentences with completed LLM annotation across five NLPeer subsets, majority-voted per classifier and keyed by corpus split, paper index, and within-paper sentence index. For each of the 11 annotation modules we export 200 rows (random seed 42). Out of the 11 annotation modules, about_outcome is the gating module, get_target_sent is a helper that provides input for downstream modules, and contains_hedging and is_vague are auxiliary attributes outside the decision tree. We call the remaining seven modules, which directly determine the final label, taxonomic modules. We first form an anchor set S by stratifying on merged about_outcome labels (100 rows with LLM label True and 100 with False) without balancing by corpus split or paper. Let cohort_true denote the 100 rows in S with about_outcome True; this cohort is the shared core included in the seven taxonomic verification sheets and both auxiliary-attribute sheets. For get_target_sent, since whether it is invoked is conditional on is_reporting_clause and sentence_embedded, we retain eligible rows from cohort_true and top up from other about_outcome-True rows that satisfy both conditions. The audited error rate egts,Te_gts,T therefore estimates extraction quality on rows where the gating predicates fired; the role of this rate in the per-label reliability score is detailed below in Appendix B. For each of the seven taxonomic modules that implement the generalisation decision tree and the two auxiliary attributes, we build a 200-row export balanced 100/100 on that moduleâs LLM label: sample up to 100 True and 100 False rows from cohort_true (when available), then top up without replacement from the pool of about_outcome-True sentences until the quota is met. The about_outcome verification set is anchor S itself. Per-label reliability: get_target_sent and the asymmetric veto rule. The path product SpS_p in (2) treats get_target_sent as one more classifier in the sequence actually evaluated by the pipeline. Because get_target_sent runs only when both is_reporting_clause and sentence_embedded are predicted true, this asymmetric gating warrants a separate accounting of how it can fail. Let T denote the truth-side trigger event âboth is_reporting_clause and sentence_embedded are truly trueâ, and let LpL_p be the LLM gate decision implied by the path p. We decompose the path-level failure into three disjoint events: âą VtreeV_tree: a tree classifier on p is wrong; this is captured by the marginal factors (1âec,vc)(1-e_c,v_c) in (2). âą VextV_ext: get_target_sent was invoked but the extraction is wrong; captured by (1âegts,T)(1-e_gts,T) on Lp=(T,T)L_p=(T,T) paths. âą VskipV_skip: T holds but Lpâ (T,T)L_pâ (T,T); the LLM skipped get_target_sent when it should have run, so downstream classifiers received the original sentence in place of the extracted target. Over-calling get_target_sent (i.e., Lp=(T,T)L_p=(T,T) but ÂŹT T) is not counted as a failure mode: the prompt is designed to return the original sentence in that case, so its only contribution to the path is the extraction pass rate 1âegts,T1-e_gts,T. Marginalising over the truth gates yields a case-wise definition of the gate-block factor MpM_p of SpS_p, with r and m shorthand for the is_reporting_clause and sentence_embedded subscripts: Lp L_p =(T,T):Mp=(1âer,T)â(1âem,T)â(1âegts,T), =(T,T):M_p=(1-e_r,T)(1-e_m,T)(1-e_gts,T), Lp L_p =(T,F):Mp=(1âer,T)â(1âem,F), =(T,F):M_p=(1-e_r,T)(1-e_m,F), Lp L_p =(F,â ):Mp=(1âer,F). =(F,\,·\,):M_p=(1-e_r,F). On non-(T,T)(T,T) paths the F-stratum gate factor (1âec,F)(1-e_c,F) already equals PrâĄ(ÂŹVskip) ( V_skip): âLLM gate F rightâ implies âtruth gate Fâ implies âno missed callâ. Adding a separate VskipV_skip term would therefore double-count. Hence the path product in (2), evaluated on the actually executed sequence, is numerically the marginalised score; the asymmetric rule does not add a new factor, only a new interpretation. The end-to-end shared-set check in Section 5.3 reports the two corresponding row-level vetoes separately: a skip-veto fires when the rectified path triggers get_target_sent but the LLM did not call it, and an extraction-veto fires when the LLM called get_target_sent and the human majority judged the extraction wrong. Over-calls remain unvetoed and are absorbed into the corpus-level egts,Te_gts,T. Agreement and reconstructed labels. The verification judgements themselves (LLM correct vs. LLM incorrect) are heavily skewed towards the positive class, making chance-corrected agreement on those judgements difficult to interpret. For each Boolean module, we instead reconstruct the task label implied by each verifier. If the verifier accepts the framework label, we retain it; if the verifier rejects it, we negate it. We then compute nominal Krippendorffâs α over these individual reconstructed labels before majority voting. This moves the chance correction onto the task-label scale while preserving the observed pairwise disagreements. The reconstruction is unavailable for get_target_sent because rejecting an extracted string does not identify the verifierâs preferred alternative. Appendix C Independent Annotation Details Verification Independent Classifier Acc Acc+ Acc- α Acc Acc+ Acc- α Is it about outcomes? (§4.1) about_outcome 0.990 0.990 0.990 0.853 0.870 0.810 0.930 0.619 Is it generalised? (§4.2) is_generalised 0.990 0.980 1.000 0.760 0.765 0.590 0.940 0.065 Is it framed? (§4.3) ref_study 0.975 0.960 0.990 0.853 0.905 0.850 0.960 0.713 ref_study_population 0.980 0.980 0.980 0.800 0.765 0.770 0.760 0.428 Is it generic? (§4.4) get_target_sent 55 5 Label-level α is unavailable for get_target_sent. Raw pairwise agreement on whether the extraction matches the framework is 0.9370.937 under verification and 0.7970.797 under independent annotation. 1.000 1.000 â â 0.865 0.865 â â is_present_tense 0.985 0.990 0.980 0.933 0.980 0.990 0.970 0.933 quantified 0.985 0.970 1.000 0.900 0.880 0.770 0.990 0.670 is_reporting_clause 0.985 0.970 1.000 0.887 0.950 0.930 0.970 0.637 sentence_embedded 0.925 0.850 1.000 0.810 0.890 0.780 1.000 0.415 Auxiliary attributes (§4.5) contains_hedging 0.995 0.990 1.000 0.833 0.875 0.760 0.990 0.558 is_vague 0.975 0.950 1.000 0.771 0.710 0.450 0.970 0.090 Table 7: Per-module results before merging is_reporting_clause, sentence_embedded, and get_target_sent into removable_framing. Acc is majority-to-framework agreement: under verification, the majority judged the framework output correct; under independent annotation, the majority-assigned label matches the framework output. Acc+ and Acc- report this proportion conditional on positive and negative framework outputs. Verification α uses reconstructed verifier-implied labels, while independent α uses directly assigned labels. Protocol and sampling. Independent annotation complements the human verification described in Appendix B. We reuse the same 200 sampled items for each of the eleven underlying annotation modules, but withhold the framework output and its reasoning. Annotators receive only the task instructions and the text fields supplied to the corresponding classifier. For Boolean modules, they assign a True/False label directly; for get_target_sent, they write the target proposition. Each item receives three annotations. Metrics. Acc denotes majority-to-framework agreement in both tables. Under verification, an item is correct when the majority judged the framework output correct. Under independent annotation, a Boolean item is correct when the majority of the three directly assigned labels matches the framework output. Acc+ and Acc- report this quantity separately for positive and negative framework outputs. Krippendorffâs α is computed over the three individual labels for each item, before taking the majority. The labels entering α differ by protocol. Let yiy_i be the frameworkâs Boolean label for item i, and let ciâac_ia denote whether verifier a judged it correct. The verifier-implied label is y~iâa=yiif âciâa=True,ÂŹyiif âciâa=False. y_ia= casesy_i&if c_ia=True,\\ y_i&if c_ia=False. cases (3) Verification α is computed over y~iâa y_ia. Independent α is instead computed directly over each annotatorâs raw True/False label. Both coefficients therefore operate on the same task-label scale and sampled items. However, the reconstructed verification labels remain conditioned on seeing the framework output and reasoning, so they do not constitute independent annotations. Per-module results. Table 4 reports the nine modules after consolidating the three removable-framing operations, while Table 7 reports all eleven operations. Independent majority agreement with the framework ranges from 0.7100.710 to 0.9800.980 in the consolidated table. Verification α ranges from 0.7600.760 to 0.9670.967, whereas independent α ranges from 0.0650.065 to 0.9330.933. The largest protocol difference occurs for is_generalised (0.7600.760 vs. 0.0650.065). Independent α is also low for is_vague (0.0900.090), and remains lower for ref_study_population (0.4280.428) and sentence_embedded (0.4150.415), while is_present_tense reaches 0.9330.933 under both protocols. Thus, independently applying the instructions is highly reproducible for some surface-oriented modules but does not yield a stable reference for every semantic distinction. The removable_framing coefficients measure its Boolean gate and are derived in Appendix D. Target-sentence extraction. We normalise each independently written target with Unicode compatibility normalisation and case folding, then remove punctuation and whitespace before exact comparison. All 200 items receive three target strings. The three normalised strings are identical for 138 items, two distinct strings occur for 52 items, and three distinct strings occur for 10 items. The human majority matches the framework extraction for 173 items (Acc=0.865Acc=0.865). We do not report label-level α for this free-text operation because the Boolean label-level α used above is not applicable. The footnote to Table 7 instead reports raw pairwise agreement on whether each extraction matches the framework. End-to-end comparison and interpretation. On the 100 items shared across the decision-tree modules, substituting the independent human-majority decisions into the final tree gives a final label matching the original framework output for 48 items. The corresponding verification result is 96 items, and the two human-implied final labels agree on 61 items. These differences compound the module-level disagreement and show that verification and independent annotation answer different questions: verification assesses whether a proposed output is defensible, whereas independent annotation measures whether annotators reproduce that output without seeing it. The low independent agreement for is_generalised and the moderate agreement for ref_study_population also help contextualise the rule-based baseline in Appendix E. If trained annotators have difficulty reproducing these semantic distinctions without a proposed answer, deterministic surface rules should not be expected to recover them reliably. Appendix D Composite Label: removable_framing Table 4 consolidates is_reporting_clause, sentence_embedded, and get_target_sent into a single removable_framing row. This section defines the merge and explains how its extraction-aware accuracy and Boolean-gate Krippendorffâs α are computed under both human protocols. Population. We inner-join the is_reporting_clause and sentence_embedded human-annotation sheets on the shared row key (split, input_path, row_idx), then intersect with the 100 rows common to all taxonomic evaluation exports. This yields 100 rows. The get_target_sent annotation sheet is left-joined onto the same key set; only 28 of the 100 rows have both is_reporting_clause and sentence_embedded labelled True by the framework and therefore have a get_target_sent human vote. Merged framework label. The composite framework label used to stratify Acc+ / Acc- is yrf=yircâ§yces,y_rf=y_irc y_ces, (4) where yircy_irc and ycesy_ces are the framework booleans for is_reporting_clause and sentence_embedded respectively. This mirrors the pipeline gate that determines whether get_target_sent is invoked: 28 rows have yrf=Truey_rf=True and 72 have yrf=Falsey_rf=False. Composite majority accuracy. For each of the 100 rows, we compute a row-level correctness outcome as follows. Let m^c m_c denote whether the human majority judged framework module câs output correct on that row. The composite majority verdict is correctrf _rf =m^ircâ§m^cesâ§g, = m_irc m_ces g, (5) g g =m^gtsif âyrf=True,Trueotherwise, = cases m_gts&if y_rf=True,\\ True&otherwise, cases where m^gts m_gts is the human majority judgement for get_target_sent on that row. This matches the trust-score path product in (2): on non-invoked rows the extraction factor is absent, so the audited pass rate already conditions on the correct event. Boolean-gate Krippendorffâs α. For verification, let r~iâa r_ia and e~iâa e_ia be the labels implied by verifier a for item i on is_reporting_clause and sentence_embedded, respectively. As described in Appendix B, each label retains the framework value when the verifier accepts it and negates the value when the verifier rejects it. For independent annotation, let riâar_ia and eiâae_ia denote the corresponding raw labels. We derive one removable-framing gate label per item and annotator: m~iâaver m_ia^ver =r~iâaâ§e~iâa, = r_ia e_ia, (6) miâaind m_ia^ind =riâaâ§eiâa. =r_ia e_ia. We compute nominal Krippendorffâs α separately over m~iâaver\ m_ia^ver\ and miâaind\m_ia^ind\ on the same 100 items, with three labels per item. The resulting coefficients are 0.9670.967 for verification and 0.6400.640 for independent annotation. Composite results. Under verification setting, 97 of the 100 composite decisions agree with the framework (Acc=0.970Acc=0.970). Under independent annotation, 87 decisions agree: 24 of 28 positive framework outputs (Acc+=0.857Acc^+=0.857) and 63 of 72 negative outputs (Accâ=0.875Acc^-=0.875). Unlike the labels in Equation 6, these accuracy values also require get_target_sent to match when the framework invokes extraction. The α values for removable_framing in Table 4 measure only agreement on the Boolean removable-framing gate; get_target_sent extraction quality is reported separately in Appendix C. Table 7 reports both protocols for the eleven individual modules before is_reporting_clause, sentence_embedded, and get_target_sent are merged. Appendix E Rule-based Baseline To explore how much of NLPGenA can be reproduced without an LLM, we implement deterministic counterparts for nine annotation modules that are used in the final decision tree (Algorithm 1) for determining the generalisation label. Each counterpart translates the corresponding prompt instructions into regular-expression and lexicon rules, supplemented by spaCy part-of-speech, morphological, and dependency features. The baseline receives the same text fields as NLPGenA, executes the modules in the same dependency order, and applies the same final decision tree. The rules were derived from the prompts and frozen before evaluation; they do not use LLM predictions, rationales, or human annotations. We report the three operations consolidated as removable_framing in the main paper separately so that target-sentence extraction can be evaluated directly. about_outcome. The rule requires a finite verb. It accepts patterns such as âwe introduce/propose/presentâ, âour results showâ, and inflected forms of âachieveâ, âoutperformâ, and âimproveâ. It rejects navigational patterns such as âTable/Figure N showsâ, future-work cues, method verbs such as âwe use/train/evaluateâ without an outcome cue, and prior-work cues without a current-study reference. sentence_embedded. The rule accepts a finite clausal complement (ccomp), a that-marked finite clause attached to a content noun such as âfindingâ or âevidenceâ, or a post-colon span containing both a subject and a finite verb. is_reporting_clause. The rule requires a matrix verb with a reporting lemma such as âargueâ, âfindâ, âindicateâ, âreportâ, âshowâ, or âsuggestâ. Matches in relative, adverbial, or open complements (relcl, acl, advcl, or xcomp) are excluded; âfindâ must be a root or coordinated root. get_target_sent. This operation runs only when both sentence_embedded and is_reporting_clause are True. It takes the dependency subtree of the first finite complement, removes an initial âthatâ, âwhetherâ, or âifâ, trims punctuation, and returns the resulting proposition. Otherwise, it copies the original sentence. is_generalised. The rule first accepts causal markers such as âbecauseâ, âdue toâ, âcauseâ, âlead toâ, and âresult inâ. It then maps vague frequencies such as âoftenâ, âusuallyâ, and âmost of the timeâ to True, but exact or absolute frequencies such as â3 out of 10â, â62% of casesâ, âalwaysâ, and âneverâ to False. When there is no frequency marker, it approximates NLPGenAâs insertion test: a present-tense predication is accepted unless blocked by a study-specific expression (e.g., âthis studyâ, âTable 2â, or an exact percentage), a singular proper-name subject, or a past-tense event. ref_study. The rule accepts explicit current-study expressions: first-person forms such as âweâ and âourâ; âthis studyâ and âthe present studyâ; âthese resultsâ and âthese findingsâ; âthe proposed methodâ; references such as âTable/Figure/Section Nâ; and numbered studies or experiments. ref_study_population. The rule first matches explicit populations such as âparticipantsâ, âannotatorsâ, âsample/cohort ofâ, âdatasetâ, âbenchmarkâ, and âcorpusâ. It also matches experimentâpopulation relations such as âevaluated ⊠models/datasetsâ, âacross ⊠tasks/languagesâ, or âon task/benchmarkâ when the sentence contains a measurement such as accuracy, BLEU, or an improvement. Anaphoric expressions such as âthese modelsâ require an experimental cue in the context. is_present_tense. The rule inspects the dependency root and its coordinated main predicates. A VBP/VBZ predicate or present-tense auxiliary is accepted. The modals âcanâ and âmayâ count as present, whereas âshallâ, âwillâ, âcouldâ, and âwouldâ do not. quantified. The rule matches phrases such as âa fewâ, âthe majority ofâ, âsome/most/all ofâ, and âmore/fewer/less than N%â, as well as tokens such as âeachâ, âeveryâ, âmanyâ, and âseveralâ when they modify a noun. Adjectival uses such as âmost effectiveâ are excluded because the token is not attached to a noun. Evaluation. We evaluate each module on the same 200 rows used for human verification. For each Boolean module, the majority verification judgement supplies the reference label: the NLPGenA label is retained when most annotators judged it correct and inverted otherwise. For get_target_sent, all audited extractions were majority-approved, so we report normalised exact match against these reference strings. The final-category comparison executes the complete baseline on the shared 100-row verification set. Table 8 reports accuracy alongside the corresponding NLPGenA result; rows whose absolute difference is below 0.10.1 are bold. Module NLPGenA Rule baseline about_outcome 0.990 0.630 sentence_embedded 0.925 0.810 is_reporting_clause 0.985 0.860 get_target_sent 1.000 0.460 is_generalised 0.990 0.630 ref_study 0.975 0.895 ref_study_population 0.980 0.570 is_present_tense 0.985 0.965 quantified 0.985 0.845 Generalisation category 0.960 0.120 Table 8: Verification-derived accuracy of NLPGenA and its prompt-derived rule-based counterparts. Component rows use 200 verified examples; get_target_sent uses normalised exact match, and the final row uses the shared 100-example verification set. The rule baseline approaches NLPGenA on ref_study and is_present_tense. These components could therefore be candidates for deterministic replacement, which would reduce LLM calls and improve reproducibility. However, the final-category accuracy of 0.1200.120 shows that wholesale replacement is not viable: semantic distinctions such as generality and references to the study population remain difficult to capture with surface rules, and component errors propagate through the pipeline. This result aligns with the observation that certain semantic distinctions, particularly genericity, are not transparently marked in English (Reiter and Frank, 2010). Independent annotation likewise yields very low chance-corrected agreement for is_generalised (α=0.065α=0.065) and only moderate agreement for ref_study_population (α=0.428α=0.428; Appendix C). This difficulty in reproducing the semantic distinctions partly contextualises the rule baselineâs performance. A hybrid framework combining deterministic rules for suitable components with LLM-based semantic judgements is therefore a promising direction. We also considered a supervised baseline trained on the 100 non-shared verification rows for each classification step and evaluated on the shared set. This would not yield a meaningful comparison: beyond the small training set, labels were balanced only across the complete 200-row audit rather than within the non-shared subset. The resulting training data are severely imbalanced for several steps (e.g., 99:199:1 for outcome detection and 89:1189:11 for present-tense detection), while the test split is often skewed in the opposite direction. A learned modelâs performance would therefore largely reflect the sampling procedure. The current independent round is also too small to provide an adequately scaled training set. We therefore leave supervised baselines to future work using a substantially larger, independently labelled dataset. Appendix F Dataset Examples and Error Analysis for Annotation Decision path LLM Label Example (†= correct, â„ = incorrect) Error Analysis is_generalised False non-gen †Our method outperforms Set by a maximum of 5.89% micro F1 in the ZH dataset, highlighting its effectiveness in mitigating order bias. â„ This paper detects and classifies fallacies. The â„ example should be annotated as gen/* as it should pass the insertion test. is_generalised True, ref_study_population True gen/framed/non-generic †73% of respondents agree that model-internal interventions are likely necessary to achieve CB. â„ There are two advantages brought by such a data augmentation approach. The âdata augmentation approachâ in the â„ example is the research outcome rather than the study population. is_generalised True, ref_study_population False, ref_study True, is_present_tense True, is_reporting_clause True, sentence_embedded True, quantified True gen/framed/non-generic †Our experiments showed that our method offers several advantages over the prevailing paradigm of direct image-to-report modeling. â„ A thorough analysis of the prototype-based clustering method demonstrates that the learned prototype vectors are able to implicitly capture various relations between events. There is no quantifier in the â„ example. is_generalised True, ref_study_population False, ref_study True, is_present_tense True, is_reporting_clause True, sentence_embedded False gen/framed/non-generic †The results indicate the importance of not freezing too many layers. â„ Our extensive experimental results have demonstrated the effectiveness of TPT. The main verb in the â„ example is in the past tense. is_generalised True, ref_study_population False, ref_study True, is_present_tense True, is_reporting_clause False gen/framed/non-generic †Our game requires more sophisticated analyses to capture to what extent the emergent languages are compositional, and what the decomposed features are. â„ In conclusion, offering primary education has few advantages. The â„ example does not refer to the study. is_generalised True, ref_study_population False, ref_study True, is_present_tense False gen/framed/non-generic †We believe that our entropy-based perspective will help provide a strong starting point for this pursuit. â„ By integrating these methods, we achieved a 7.35% improvement in decompilation performance, reaching a new state-of-the-art level of 55.03%. The â„ should be annotated as non-gen as it should not pass the insertion test. Note that is_present_tense asserts the main verb of the embedded sentence in the â„ example. is_generalised True, ref_study_population False, ref_study True, is_present_tense True, is_reporting_clause True, sentence_embedded True, quantified False gen/framed/generic †Thus, we argue that annotated arguments are actually used for AE, not for ED in existing joint methods. â„ Note, however, that we should be aware of the possibility of unreliable results from the self-ensemble. The â„ example, although satisfying the designed criteria, its framing is not removable. It shows that our heuristics are not perfect. is_generalised True, ref_study_population False, ref_study False, is_present_tense True, quantified False gen/bare/generic †The contextualized semantic space does not require additional knowledge, and therefore can scale to other languages, especially low-resource languages where a large knowledge base is unavailable. â„ Contrary to our initial assumption we found that the lower the correlation, the higher the predictive performance with FRESH. The â„ exmaple clearly does refer to the study. is_generalised True, ref_study_population False, ref_study False, is_present_tense True, quantified True gen/bare/non-generic †Many novel methods are proposed based on IA findings and highly influenced by them, but highly influential non-IA work cites IA findings without being driven by them. â„ Such a workflow can provide a large improvement in translation quality at each iteration âlargeâ in the â„ example is not a frequency expression. is_generalised True, ref_study_population False, ref_study False, is_present_tense False gen/bare/non-generic †In addition, L2 regularization worked better than L0 . â„ Effectiveness of Proof-Enhancement Method. Incomplete sentences such as the â„ example should have been discarded during about_outcome stage. Table 9: Example and error analysis for each possible decision path. The †examples illustrate typical sentences that are expected to fall into the corresponding decision path, while the â„ examples illustrate potential LLM errors and limitations of the current heuristics. Table 9 gives representative examples for every terminal path in the annotation decision tree. For each path, the †example shows a sentence whose classifier decisions support the assigned taxonomy label, while the â„ example illustrates either a plausible LLM error or a limitation of our heuristic operationalisation. Several error patterns recur across paths. First, some mistakes arise at the initial generalisation boundary: result claims can be incorrectly treated as non-gen, while highly specific numerical results may be over-classified as gen/*. Second, the boundary between study framing and study population references is delicate. For example, phrases such as âdata augmentation approachâ may look like a study population but actually name the research outcome itself. Third, genericity decisions are sensitive to the exact predicate being evaluated after a reporting frame is removed. Errors occur when the model misses tense, treats vague descriptors (e.g., âlargeâ) as quantifiers, or evaluates the surface reporting clause rather than the embedded proposition. The table also highlights a residual limitation of the decision-tree approximation. Some sentences satisfy the observable criteria used by the prompts but remain difficult to classify under the theoretical taxonomy, especially when the study frame is pragmatically non-removable even though it has the right syntactic shape. We therefore use these examples as qualitative diagnostics for the pipeline, complementing the human-validation results in Appendix B. Appendix G about_outcome Across Sections In Table 10 we report how many sentences across each section ended up with an about_outcome annotation (note that this is not an average per paper, rather, we normalise by the total set of all sentences from all papers per venue) in Table 10. Most of the sections are relatively consistent across the tested venues, with the biggest range in the rates of generalisations across the conclusion. acl 2017 coling 2020 arr 2022 emnlp 2023 emnlp 2024 abstract 57.3 48.3 54.9 55.1 53.5 intro 32.2 29.5 35.1 37.8 37.4 results 55.5 60.7 60.1 57.2 62.8 discuss. 56.1 64.1 54.0 54.2 57.6 concl. 55.8 55.0 36.3 71.5 34.1 total 51.4 51.5 48.1 55.2 49.1 Table 10: Percentage of about_outcome across different paper sections (across total set of all sentences in each venue from a specific section). Appendix H Track Assignment The only venue in our dataset which came with track data for each paper was COLING-2020, with the tracks summarised in Table 11. Short Name Full Track Applications Applications (BioNLP/Legal/SocialâŠ) Dialogue Dialogue and Interactive Systems Discourse Discourse and Pragmatics TopicModel Topic Modelling, IR & CR LangGen Language Generation LangModelling Language Modelling KG Info Extraction, âŠ, KGs LR-Eval Language Resources and Eval CogMod Ling Theories, Cognitive Models ML Machine Learning for CL/NLP MT Machine Translation WordSeg Morphology and Word Segmentation Semantics Semantics (Words/Sents/Ontologies) SA-Opinion Sentiment Analysis, Opinion Mining Speech ASR, TTS & Spoken Language Summar-Simpl Summarization and Simplification SyntaxParsing Tagging/Chunking/Syntax Parsing Text-QA Textual Inference and QA GroundLA Grounded Language Acquisition Table 11: COLING 2020 tracks, both full name and our shortened acronyms for plots. For more track information see the call for submissions: https://groups.google.com/g/ml-news/c/pP0wKlJdW6s?pli=1 For all COLING-2020 papers, we got an embedding of the title + abstract using the allenai/specter2_base pre-trained model (Singh et al., 2022). We then obtained centroid vectors for all tracks by considering all COLING-2020 embeddings in each one. We then attached a soft track label to all track-less papers from the other four venues by embedding their title + abstract using the same allenai/specter2_base pre-trained model and simply labelling the paper as belonging to the closest track (lin-alg norm for distance from track centroids). Figure 7: Generalisation rates across ACL â17 papers with soft track labels. Figure 8: Generalisation rates across ARR â22 papers with soft track labels. Figure 9: Generalisation rates across EMNLP â23 papers with soft track labels. Figure 10: Generalisation rates across EMNLP â24 papers with soft track labels. We then plot the generalisation class distirbution across the different venues, separating results across papers assigned to the same track. ACL â17 in Figure 7, ARR â22 in Figure 8, EMNLP â23 in Figure 9, EMNLP â24 in Figure 10. We find that rates of generalisations do differ across the different NLP subdomains, although the trends are not consistent across the venues. Additionally, the relative orderings of the tracks in terms of most to least generalised are not consistent across the different years. There are several explanations to this. First, the NLP field changed quite substantially over the past couple of years, so some of the original COLING-2020 tracks are not as applicable anymore. Additionally, our soft-labelling clustering algorithm is quite naive, and although we manually inspected the soft labels for each venue to confirm that something sensible was going on, the approach is far from perfect. We believe it is still useful to consider different rates of generalisations across different scientific subfields, but perhaps more gold labels for tracks/domains are necessary for a robust analysis. Appendix I Citation Analysis To obtain citation counts, we first parse each paperâs ID from the ACL Anthology URL (if provided) and then pass this to the SemanticScholarAPI66 6 https://api.semanticscholar.org/graph/v1/paper/batch. Note that we were not able to obtain citation counts for papers that did not contain the ACL URL in their metadata, which was the case for EMNLP â23 papers. In Table 12 we report the results of the same analysis as was done for ACL â17 papers (obtaining the Spearman rho (Ï) correlation coefficient between generalisation classes and paper citation counts) for COLING â20, ARR â22, and EMNLP â24 papers. COLING â20 ARR â22 EMNLP â24 Ï p-value Ï p-value Ï p-value non-gen -0.07 0.541 -0.06 0.168 -0.02 0.365 gen/framed/non-generic 0.14 0.218 0.08 0.070 -0.01 0.623 gen/bare/generic -0.04 0.712 0.03 0.538 0.02 0.321 gen/bare/non-generic -0.06 0.587 -0.07 0.107 0.01 0.584 gen/framed/generic 0.04 0.741 -0.05 0.287 0.04 0.111 */generic -0.03 0.769 0.00 0.933 0.02 0.367 gen/* 0.07 0.541 0.06 0.168 0.02 0.361 gen/framed/* 0.14 0.199 0.08 0.088 -0.01 0.648 Table 12: Spearman Ï correlation between the generalisation classes and citation counts for COLING â20, ARR-22, EMNLP â24 papers. Unlike for ACL â17, none of the correlations are statistically significant. However, we still see the gen/framed/* class as one of the more positively correlated ones across COLING â20 and ARR â22, which is consistent with what we observed with ACL â17, even though the effects are not strong enough statistically. As noted in the main text, correlating citation counts with a single linguistic feature, without any consideration for the semantics of the paperâs claims, is an incredibly hard task. Therefore, the fact that we are seeing even a weak signal is already an indication that this relationship should be robustly investigated further, perhaps alongside some combination of semantic features. Also, the different venues have really different citation distributions (see Figure 11). ACL â17 specifically has more spread, whereas for the other venues either the citation counts have not stabilised (for EMNLP â24, see the sharp spike around 0 citations), or the citation counts are all all bundled in the same log space. It is possible that the distributions of the other conferences simply are not spread out enough and are too clustered/noisy to find a strong effect. Figure 11: Kernel Density Estimation (KDE) plot of the citation distributions for papers across the venues for which we could obtain citation counts using the Semantic Scholar API (ACL â17, COLING â20, ARR â22, EMNLP â24). Appendix J Data Access, Licensing, and Intended Use We build NLPGens on the NLPeer corpus (Dycke et al., 2023), accessed by submitting the standard data request through the official TUdatalib deposits for v177 7 https://tudatalib.ulb.tu-darmstadt.de/items/eeb96857-15ea-4075-9857-2f98a9728145 and v288 8 https://tudatalib.ulb.tu-darmstadt.de/items/d4a4061b-e4e3-4b1e-a90d-d48a3d69e3c0; no separate data use agreement was attached. Both deposits are released under C BY-NC 4.0 except where otherwise noted, with subset-level Creative Commons terms (e.g., C-BY for ACL-17, C-BY-NC-SA 4.0 for COLING-20 and ARR-22) and per-item licence strings documented in the resource paper (Dycke et al., 2023) and shipped as metadata with each paper. Our use is non-commercial academic research with attribution, and we do not redistribute the raw corpus. We modify the source material only by parsing manuscripts into intertextual graphs, segmenting them into sentences, filtering to five standard sections, and adding sentence-level annotations. NLPeer is intended for the computational study of peer review and NLP-based reviewing assistance, and the authors stress that author or reviewer profiling violates the intended use; they explicitly position NLPeer as a basis for new annotated datasets on its clearly licensed, structured manuscript text (Dycke et al., 2023). Our work falls within this scope: we process only parsed manuscript text, annotating linguistic properties of scientific claims. We do not use peer-review text, attempt profiling, or automate review generation or scoring. We will release the resulting NLPGens annotations under C BY-NC-SA 4.0 to inherit the most restrictive source-subset licence, with NLPeer attributed and the per-row source licence retained alongside each annotation. Appendix K Personal Data and Offensive Content We use manuscript text only and do not access peer-review reports, reviewer identities, or any non-public author metadata. Identifying information about reviewers in NLPeer was already curated upstream: review texts were manually checked by the NLPeer team for personal information, and identifying metadata was stripped from review reports in all subsets except the openly identified F1000-22 (Dycke et al., 2023); F1000-22 is excluded from our work. Author names that appear in the manuscripts (e.g., in âweâ-statements, acknowledgements, or in-text citations) reflect public scholarly publication records rather than private identifiers we collected; we never use them for profiling, demographic inference, or any analysis at the level of individual people. Beyond venue publication norms, we did not perform a separate systematic screen for offensive content; given that the input is accepted NLP/CL research papers, we judged this risk to be low. Appendix L Prompt Templates about_outcome (§4.1) # Task: Determine if the target claim is an outcome, given the previous two sentences as context. When you are not sure, answer with Yes. # Outcome: * Introduction, e.g. * "we introduce", "we propose", "we present" * Results: observations of the experimental results, e.g. * "Our model achieved significant improvements over the baselines." * Discussion: interpretation or analysis of the results, e.g. * "The results show that our approach significantly outperforms baseline methods." * Contribution: how the research will contribute to knowledge in the field, e.g. * "This work contributes to the field by introducing a novel architecture." * Conclusion * Limitations # Not Outcome: * Background: statements about knowledge in the domain or previous related work, e.g. * "Deep learning models achieve high accuracy in most tasks." * Experimental setup details, e.g. * "We used a BM25 retriever with a top k of 44 and a width penalty ÎČ=0.6ÎČ=0.6." * Methodology/Approach, including actions done by the authors, e.g. * "We evaluate the performance of our method on various benchmarks." * "GEAR takes both prediction and evaluation metrics as input." * "We believe that GEAR is able to take both prediction and evaluation metrics as input." * Navigational sentences: referring to text, figures, tables, etc., e.g. * "Table 1 shows the results of the experiments." * Incomplete claims or parsing errors, e.g. * "in both the pre-training and the fine-tuning stages." * "2 3 5 5 1 ] ] ] " * Future work # More instructions: * Outcome should be mainly about the authorâs own work. It should not be fully about othersâ work. * It can be an outcome if othersâ work is only mentioned to elaborate on the findings of the authors, e.g. * "Unlike previous works, GEAR nails single-stage summarisation, aligning with the finding of Gu et al. (2024)." * Modal verbs or hedging (e.g., "can", "may", "tend to") do not change the classification. Classify by the content. * Being an opinion (e.g., "we believe") does not change the classification. * If both outcome and non-outcome are present, choose outcome, e.g. * "We propose the method to address the limitations by introducing a novel architecture." # Target Claim: Context: two_sent_context Claim: claim Is the target claim an outcome, given the previous two sentences as context? When you are not sure, answer with Yes. Give your reasoning first and then answer with Yes or No. The output should be in the format below. Reasoning: <your reasoning> Answer: Yes/No is_generalised (§4.2) # Task: Determine if the target claim is generalised. Answer with generalised or not generalised. # Decision process: 1. Check the context The context provided is the previous two sentences before the target claim. 2. Ignore hedging Do not let hedging affect the decision. Examples of hedging: can, could, would, may, might, appears to, seem, likely, necessarily. 3. Check for explicit causality Check whether the sentence explicitly expresses causality, for example by using words or phrases such as: because, reason, due to, so, lead to, result in, cause. If YES -> generalised If NO -> continue Example: "The improvement is due to better attention alignment." -> generalised 4. Check for a frequency expression in the whole sentence 4.1. If there is no frequency expression -> continue to 5 4.2. If there is already a frequency expression, check whether it expresses or implies one of the following: * a specific frequency, e.g. "3 out of 10", "in 62% of cases" * 100% frequency, e.g. "in all cases", "always", "all the time", "consistently" * 0% frequency, e.g. "never", "no", "none" If YES -> not generalised If NO -> generalised Examples: "The model succeeds half of the time." -> not generalised "The model always outperforms the baseline." -> not generalised "The model often/usually/sometimes outperforms the baseline." -> generalised "The model outperforms the baseline most of the time/more than half of the time." -> generalised 5. If there is no frequency expression, apply the insertion test Try inserting either "usually" or "generally" into the **main clause**, taking the context into account. Then ask: Does the modified sentence still sound natural in the context? If the modified sentence still sounds natural in the context -> generalised If the modified sentence does not sound natural in the context -> not generalised Notes on interpretation * "generally" ONLY allows a collective reading * the property applies to the subject as a class/kind or whole, not to a single instance * "usually" can allow: * a distributive reading: indicates the proportion of subject instances that the property applies to, e.g. "The scores of A are usually lower than B" * a temporal reading: indicates the proportion of time that the property applies, e.g. "A usually works better than B" * Use whichever reading fits the context best. 6. Mixed cases If a sentence contains both a generalised part and a non-generalised part, annotate it as generalised. Example: "On this dataset, the model consistently achieved 85% accuracy; this shows that graph structure helps." -> contains both 100% frequency ("consistently") and a generalised claim ("graph structure helps") -> generalised # Decision summary Use this decision order: * Read the previous three sentences for context. 1. Read the previous three sentences for context. 2. Ignore hedging. 3. If it explicitly expresses causality -> **generalised**. 4. Otherwise, check for a frequency expression in the whole sentence: * no frequency expression -> continue to 5 * specific / 100% / 0% frequency -> **not generalised** * other frequency expressions -> **generalised** 5. If there is no frequency expression, try inserting "usually/generally" into the main clause: * the modified sentence still sounds natural in the context -> **generalised** * the modified sentence does not sound natural in the context -> **not generalised** 6. If both types are present, choose **generalised**. # Target Claim: context: two_sent_context claim: get_target_sent_label Following the instructions above, is the target claim generalised? Give the reasoning process first and then answer with "generalised" or "not generalised". The output should be in the format below. Reasoning Process: * explicit causality exists: Yes/No * given no explicit causality, frequency expression exists: Yes/No/NA * given frequency expression exists, it is a specific / 100% / 0% frequency: Yes/No/NA * given frequency expression exists, it is other frequency expression: Yes/No/NA * given no frequency expression, adding "usually/generally" into the main clause sounds natural: Yes/No/NA Answer: generalised/not generalised ref_study (§4.3) # Task: Determine if the target claim includes a reference to the study itself. Find the reference first, if there is any, and then answer with Yes or No, in the same format as the examples. # Instruction: The task is to determine whether the target claim is explicitly portrayed as a fact that the current study conducted by the authors tells us. Such references could include, but are not limited to: * the authors, e.g. "we", "our" * the research findings or outcomes, e.g. "the finding", "the proposed method" * experimental results, e.g. "the experiment results", "the table shows" * direct reference to the study, e.g. "study 2", "(the third experiment)" * elements of the paper, e.g. "Table 1", "Figure 5", "Section 3", "top 5 rows of Table 1" # Target Claim: Claim: get_target_sent_label Determine if the target claim includes a reference to the study itself. Find the reference first (no more than 3 words), if there is any, and then answer with Yes or No, in the format as below. Reference: <reference to the study from the target claim> Answer: Yes/No ref_study_population (§4.3) # Task: Determine whether the target claim, based on the context, identifies the population that the experiments are carried out on. If it does, state the population. # Instructions: Consider the question: "What are the experiments carried out on?" A valid answer may include, but is not limited to: * datasets, e.g. "the MuSE dataset", "SemEval-2016 Task 10", "the top 5 rows" * models, e.g. "the language models we evaluate", "all auto-regressive models we tested", "we testedGPT-4-128k" * participants, e.g. "participants", "annotators", "markers", "students" # Target Claim: Context: two_sent_context Claim: "get_target_sent_label" Determine whether the target claim, based on the context, could answer the question "What are the experiments carried out on?" State any population you find, or "none", then answer with "Yes" or "No". Use the format below: Population: <population identified in the claim, or "none"> Answer: Yes/No is_present_tense (§4.4) # Task: Determine if the main clause of the target claim is in the present tense. State the main verb in the main clause, then answer with "Yes" or "No". # Instructions: * Predication combined with present tense and other tenses should be counted as present tense. For example, "it passed the test and shows great improvement" should be classified as present tense. * Treat "can" and "may" as present tense. * Treat "shall" and "will" as future tense. * Treat "could", "would" as past tense # Examples: Claim: "it passed the test and shows great improvement" Main verb: "passed", "shows" Answer: Yes Claim: Adultâs performance on a spatial reasoning task was not related to their scores on a verbal reasoning task Main verb: "was" Answer: No Claim: The three groups did not differ in their ability to discriminate non-modified word Main verb: "did not differ" Answer: No # Target Claim: Claim: get_target_sent_label Determine whether the target claim is in the present tense. State the main verb in the main clause, then answer with "Yes" or "No". The output should be in the format below. Main verb: <main verb in the main clause> Answer: Yes/No quantified (§4.4) # Task: Determine whether the target claim includes non-numeric quantifiers. # Definition of Quantifiers: Quantifiers are words that limit how broadly a claim applies. This includes words that specify the extent, frequency, or proportion of the claim. # Examples of Quantifiers: * Universal / distributive: all, both, each, every * Existential / indefinite: some, any, several, a few, few, many, much, little, a little, a lot of, lots of, plenty of * Negative: no, none, neither, hardly any * Proportional: most, half (of), the majority of, the minority of * Comparative / superlative before **nouns**: more (than), most, fewer (than), fewest, less (than), least * Partitives (of-phrases): some of, most of, all of, none of, many of, much of, a lot of, plenty of # Not Quantifiers: * Anything that is not before a noun is not a quantifier. * Specific numerals are not quantifiers, e.g. "0.9", "70%". # More Examples: Claim: Most reasons do not adequately justify violating a sacred value. Reasoning: In this case, "most" is before a noun, hence it is a quantifier. Answer: Yes Claim: The algorithm is the most effective one. Reasoning: "most" here is before an adjective, not a noun, hence it is not a quantifier. Answer: No Claim: Children generally score high on measures of academic achievement. Reasoning: "generally" is not before a noun, it is not a quantifier. Answer: No Claim: While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off. Reasoning: Specific numerals are not quantifiers. Answer: No Claim: Children score highest on measures of academic achievement. Reasoning: "highest" here is not before a noun, hence it is not a quantifier. Answer: No Claim: The algorithm achieves more than 95% accuracy. Reasoning: Although specific numerals "95%" is not a quantifier, "more than" here is before a noun, hence it is a quantifier. Answer: Yes # Target Claim: get_target_sent_label Determine whether the target claim includes non-numeric quantifiers. Give your reasoning first, then answer with Yes or No. Use the format below. Reasoning: <your reasoning> Answer: Yes/No is_reporting_clause (§4.4) # Task: Determine whether the target claim contains a reporting verb. # Instructions: * A reporting clause is a clause that indicates you are referring to what someone said or thought. For example, in "She said that she was hungry", "She said" is a reporting clause. * In the context of academic writing, a reporting clause is often used to report the findings of a study. For example, "We found âŠ", "the results demonstrate âŠ", and "it suggests âŠ" are reporting clauses. * A reporting verb is the main verb in the reporting clause, e.g. "found", "demonstrate", "suggest". # Examples: Claim: Our results speak to the importance of psychological distancing in the expression of conscious control over thought and action from a young age Reasoning: "speak" is a reporting verb Answer: Yes Claim: The finding that the developing system favours risky visuomotor choices forms a first step towards understanding how children deal with everyday activities Reasoning: "forms" is not a reporting verb Answer: No Claim: Childrenâs ability to improve executive function by mentally transcending their egocentric perspective underscores the critical role of representational capacities in self-control Reasoning: "underscores" is a reporting verb Answer: Yes Claim: Across four studies it shows that people do better on shape-recognition tasks Reasoning: "shows" is a reporting verb Answer: Yes Claim: the MedBERT model achieves state-of-the-art results for multiple tasks. Reasoning: "achieves" is not a reporting verb Answer: No # Target Claim: Claim: claim Determine whether the target claim contains a reporting verb. Give your reasoning first, and then answer with Yes or No. The output should be in the format below. Reasoning: <your reasoning> Answer: Yes/No sentence_embedded (§4.4) # Task: Determine whether the target claim contains a complement clause. # Instructions: A complement clause can be distinguished from a relative clause because a relative complementiser can be replaced by a relative wh-pronoun (e.g. who or which), whereas a complement clause cannot be replaced in this way. # Examples: Claim: The present study confirms that there exist significant associations between musical practice and intelligence. Reasoning: âThe present study confirms which there exist significant associations between musical practice and intelligenceâ does NOT make sense. Answer: Yes Claim: These findings suggest a promising new avenue for early EF intervention. Reasoning: There is no subordinate clause in the sentence; the part after âsuggestâ is a noun phrase. Answer: No Claim: The present study confirms the existence of significant associations between musical practice and intelligence. Reasoning: There is no subordinate clause in the sentence; the part after âconfirmsâ is a noun phrase. Answer: No Claim: These findings suggest that this is a promising new avenue for early EF intervention. Reasoning: âThese findings suggest which this is a promising new avenue for early EF interventionâ does NOT make sense. Answer: Yes Claim: Our results speak to the fact that psychological distancing is important in the expression of conscious control over thought and action from a young age. Reasoning: âOur results speak to the fact which psychological distancing is important in the expression of conscious control over thought and action from a young ageâ does NOT make sense. Answer: Yes Claim: Our results speak to the importance of psychological distancing in the expression of conscious control over thought and action from a young age. Reasoning: There is no subordinate clause in the sentence; the part after âspeak toâ is a noun phrase. Answer: No Claim: Our results show L1 loss is not a crucial condition for acquiring an L2. Reasoning: âOur results show which L1 loss is not a crucial condition for acquiring an L2â does NOT make sense. Answer: Yes # Target Claim: claim Does the target claim contain a complement clause? Give your reasoning first, then answer with Yes or No. Use the format below. Reasoning: <your reasoning> Answer: Yes/No get_target_sent (§4.4) # Task You will be given a reporting clause (e.g., "this table suggestsâŠ", "we foundâŠ", "it showsâŠ"). Your task is to analyse it and extract the embedded sentence (complement clause) being reported. If the sentence is not a reporting clause, or if the reported content is not a complete sentence, return the original sentence. # Instructions * Possible reporting verbs include, but are not limited to: suggest, indicate, demonstrate, show, argue, highlight, report, note, observe, conclude, state, claim, illustrate, examine, propose, and find. * The reported content should be a complete sentence. # Examples Sentence: This table suggests that each handcrafted feature contributes to some extent, whereas features with large weights vary across prompts. Reasoning: "suggests" is a reporting verb, and the reported content is a complete sentence, so we extract it; the "whereas" clause should also be included. Answer: Each handcrafted feature contributes to some extent, whereas features with large weights vary across prompts. Sentence: Table 4 shows that by incorporating handcrafted essay-level features, the proposed method drastically improves the accuracy of all base DNN-AES models. Reasoning: "shows" is a reporting verb, and the reported content is a complete sentence, so we extract it; the "by incorporating" clause should also be included. Answer: By incorporating handcrafted essay-level features, the proposed method drastically improves the accuracy of all base DNN-AES models. Sentence: Table 4 shows the experimental results. Reasoning: "shows" is a reporting verb, but the reported content is not a complete sentence, so we return the original sentence. Answer: Table 4 shows the experimental results. Sentence: This task presents specific challenges: the output is subject to strong structural constraints and is significantly longer than the input. Reasoning: "presents" functions as a reporting verb here, and the reported content is a complete sentence, so we extract it. Answer: The output is subject to strong structural constraints and is significantly longer than the input. Sentence: Another reason is that reducing the attention key size hurts model quality. Reasoning: "is" is not a reporting verb, so we return the original sentence. Answer: Another reason is that reducing the attention key size hurts model quality. Sentence: The results, shown in the "p-value" column in Table 4, indicated that the proposed method improved performance at the 5% significance level for the LSTM- and BERT-based models, and at the 10% significance level for the conventional hybrid model. Reasoning: "indicated" is a reporting verb, and the reported content is a complete sentence, so we extract it. Answer: The proposed method improved performance at the 5% significance level for the LSTM- and BERT-based models, and at the 10% significance level for the conventional hybrid model. # Target Sentence Sentence: claim Analyse the sentence and extract the embedded sentence being reported. If the sentence is not a reporting clause, or if the reported content is not a complete sentence, return the original sentence. Give your reasoning first, then provide the extracted sentence. Use the format below. Reasoning: <your reasoning> Answer: <extracted sentence> contains_hedging (§4.5) # Task Determine whether the target sentence includes explicit hedging. # Instructions * A sentence includes explicit hedging if it contains words or phrases that directly express uncertainty, possibility, approximation, limited generality, or reduced commitment, e.g. "may", "might", "tend to", "likely", "necessarily", "probably", "could", "would", "should", "can". * Do not infer hedging unless it is explicitly signalled by the wording. # Target sentence Sentence: claim Does the target sentence include explicit hedging? State the hedging expression in the sentence first and then answer with Yes or No. The output should be in the format below. Hedging expression: <exact word(s) or phrase(s), or "none"> Answer: Yes/No is_vague (§4.5) # Task Determine whether the target sentence includes an under-specified adjective or adverb. # Instructions * If there is an adjective or adverb in the sentence, consider whether it is a valid question to ask "how <adjective/adverb>?" If so, then the adjective/adverb is under-specified, answer Yes. * If there is no adjective or adverb in the sentence, then answer No. * When there are both specified and under-specified adjectives/adverbs in the sentence, count it as under-specified. * Example of valid question: "how significant?", "how good?", "how efficient?" * Example of non-valid question: "how true?", "how better?", "how best?", "how middle?", "how average?" * More examples: Sentence: "Furthermore, we observe significant discrepancies when applying the models to different domains." Question: "how significant?" Valid: Yes Answer: Yes Sentence: "The same is true here." Question: "how true?" Valid: No Answer: No Sentence: "In order to guarantee the annotation consistency, people turn to predefined annotation guidelines according to specific tasks or applications." Question: "how predefined?", "how specific?" Valid: No, Yes Answer: Yes Sentence: "The boxplots confirm the strength of CRF over the other models." Question: None Valid: NA Answer: No # Target sentence Sentence: claim Does the target sentence include an under-specified adjective or adverb? State the question you would ask, determine whether it is valid, and then answer with Yes or No. The output should be in the format below. Question: <question(s) you would ask, or "None"> Valid: <Yes/No/NA for each question> Answer: Yes/No