Paper deep dive

Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers

Mingmeng Geng, Yuhang Dong, Thierry Poibeau

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 68

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/27/2026, 1:38:27 AM

Summary

This paper analyzes the impact of Large Language Models (LLMs) on academic writing by tracking shifts in word usage patterns within arXiv abstracts and titles. The authors demonstrate that LLMs introduce distinct, evolving linguistic idiosyncrasies—such as the increased frequency of 'via' and 'beyond'—and that these patterns can be used to estimate LLM influence. The study employs a linear regression approach to model baseline word frequency trends and compares human-written text with LLM-generated content across various models and prompts.

Entities (7)

arXiv · datasetplatform · 100%Claude · llm · 100%DeepSeek · llm · 100%Gemini · llm · 100%GPT-3.5 · llm · 100%BERTScore · metric · 95%ROUGE · metric · 95%

Relation Signals (3)

arXiv → contains → Academic Papers

confidence 100% · This paper analyzes abstracts of arXiv papers

LLM → influences → Word Usage

confidence 90% · LLMs have their own preferences for word usage.

GPT-5 nano → favors → via

confidence 85% · newer LLMs such as DeepSeek and GPT-5 tend to favor the words 'via' and 'beyond'

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Through an analysis of arXiv papers, we report several shifts in word usage that are likely driven by large language models (LLMs) but have not previously received sufficient attention, such as the increased frequency of "beyond" and "via" in titles and the decreased frequency of "the" and "of" in abstracts. Due to the similarities among different LLMs, experiments show that current classifiers struggle to accurately determine which specific model generated a given text in multi-class classification tasks. Meanwhile, variations across LLMs also result in evolving patterns of word usage in academic papers. By adopting a direct and highly interpretable linear approach and accounting for differences between models and prompts, we quantitatively assess these effects and show that real-world LLM usage is heterogeneous and dynamic.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

67,966 characters extracted from source content.

Expand or collapse full text

Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Mingmeng Geng * 1 2 Yuhang Dong * 3 Thierry Poibeau 1 2 Abstract Through an analysis of arXiv papers, we report several shifts in word usage that are likely driven by large language models (LLMs) but have not previously received sufficient attention, such as the increased frequency of “beyond” and “via” in titles and the decreased frequency of “the” and “of” in abstracts. Due to the similarities among different LLMs, experiments show that current classifiers struggle to accurately determine which specific model generated a given text in multi- class classification tasks. Meanwhile, variations across LLMs also result in evolving patterns of word usage in academic papers. By adopting a di- rect and highly interpretable linear approach and accounting for differences between models and prompts, we quantitatively assess these effects and show that real-world LLM usage is heteroge- neous and dynamic. 1 1. Introduction The increasing impact of large language models (LLMs) in academic publications has been observed (Liang et al., 2024; Geng & Trotta, 2024; Kobak et al., 2024). As LLMs continue to develop, has their impact evolved more recently? For example, some researchers might have begun to reduce their use of certain LLM-style (more precisely, ChatGPT- style) expressions, such as “delve”, around late 2023 and early 2024 (Geng & Trotta, 2025). Different LLMs possess their own idiosyncrasies, which can be used to classify the text they generate (Wu et al., 2023; Antoun et al., 2024; Sun et al., 2025). Considering the multiple updates to Chat- GPT and the emergence of other models, this paper aims 1 ́ Ecole Normale Sup ́ erieure (ENS) – Universit ́ e Paris Sci- ences et Lettres (PSL) 2 Laboratoire Lattice 3 Friedrich-Alexander- Universit ̈ at Erlangen-N ̈ urnberg (FAU). Correspondence to: Ming- meng Geng<mingmeng.geng@ens.psl.eu>. Preprint. March 27, 2026. 1 Visualization of word usage patterns in arXiv abstracts: https://llm-impact.github.io/word-usage-arx iv-abstract/ 0100200300 Word Frequency Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek via 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 200 300 400 500 600 700 Frequency (per 10,000 Titles) via Real Data Linear Trend Fit Period (a) In arXiv paper titles. 020406080 Word Frequency Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude together 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 200 250 300 350 400 Frequency (per 10,000 abstracts) together Real Data Linear Trend Fit Period 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 80000 85000 90000 95000 100000 105000 Frequency (per 10,000 abstracts) the Real Data Linear Trend Fit Period 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 45000 50000 55000 60000 65000 of (b) In arXiv paper abstracts. Figure 1. Top-left & Middle-left: Word frequency comparison for titles or rewritten abstracts produced by different LLMs from 2,000 real arXiv abstracts; error bars denote variance across models and prompts. Remaining panels: Temporal trends of word frequencies in real arXiv data, with the yellow dashed line fitted on data from 2015 to 2021 and extended to early 2026. to analyze and estimate the impact of LLMs on academic publications in relation to these developments. As depicted in Figure 1a, certain LLMs have a preference for the word “via” when generating titles from real arXiv abstracts, whose frequency is also becoming more com- mon in real arXiv paper titles from 2025. Meanwhile, Fig- ure 1b shows that the frequency of “together” first declines markedly and is then followed by a rapid increase, likely reflecting the contrasting preferences between newer and older LLMs regarding this term. Moreover, “the” and “of”, 1 arXiv:2603.25638v1 [cs.CL] 26 Mar 2026 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers two of the most commonly used words in many English cor- pora, have experienced a clear decline in frequency within arXiv abstracts. The use of words and expressions in academic writing has always been evolving (Bizzoni et al., 2020; Wang et al., 2023; Cheng et al., 2024). Researchers are also continuously monitoring the use of specific indicative words to estimate the usage of LLMs in academic writing (Gray, 2025; He & Bu, 2025). Comparisons between texts generated by LLMs and those written by humans can now be carried out using different techniques (Wu et al., 2025). In this work, we first collect arXiv papers based on a publicly available dataset and use different LLMs to simulate parts of the abstracts submitted before the emergence of ChatGPT, either by rewriting the abstracts or generating titles. We then compare the similarities and differences between the texts produced by LLMs and those written by humans from different perspectives, as well as the variations between texts generated by different LLMs. Although models have been advancing, their outputs still differ in some way from human- written texts. Finally, we choose to analyze and estimate the impact of LLMs in academic publications through word usage, since different LLMs favor different words and word frequency is a relatively interpretable measure. Our findings suggest that the influence of LLMs on aca- demic publications is growing, and the variety and the de- velopment of LLMs has led to further variations. The tools and metrics for analyzing and monitoring their impact need to evolve as well. Although more sophisticated methods may provide numerically more accurate estimates, direct ap- proaches can also offer new insights into the understanding and visualizing of the societal impact of LLMs. 2. Data 2.1. arXiv Data This paper analyzes abstracts of arXiv papers, based on a dataset that is updated weekly on Kaggle (arXiv.org sub- mitters, 2024) 2 . These data include more than 2.9 million arXiv papers across different fields. Considering the quan- tity and rapid update rate of arXiv papers, they may well be the best avenue for timely monitoring of LLM influence on academic writing. As our study requires analyzing how effects vary over time, we need to incorporate timestamps into the analysis. In this dataset, two such timestamps are available: the paper IDid (indicating the initial submission) and the last update time updatedate (indicating the most recent revision). 2 https://w.kaggle.com/datasets/Cornel l-University/arxiv 2.2. Data Preparation Dataset versions from version 1 (released in August 2020) to version 277 (released in March 2026) were collected, with only newly added papers retained in each version. This means that the time implied by the arXiv ID is essentially the submission time, and even with revisions, the delay will not exceed a week. In this way, we can reduce the temporal delay introduced by paper updates. We categorize papers based on arXiv categories, consid- ering only the first category for papers with multiple cat- egories. We also applyre.findall(r’ + ’, x.lower())in Python to strip punctuation and convert all text to lowercase. Based on their update dates in version 265, we randomly selected 2,000 papers from January to October 2022 for simulation using LLMs, i.e., prior to the release of ChatGPT. 2.3. Other Data Only frequent words are considered to estimate the impact of LLMs. In addition to word frequencies in the arXiv dataset, we also used data from the Google Books Ngram dataset to establish a threshold for selecting common words. We are also interested in how stopwords are used in LLMs, with ref- erence to the list in Natural Language Toolkit (NLTK) (Bird, 2006). 3. Methods 3.1. Trend of Word Frequency Over Time We denote the observed frequency of wordwin the dataset at timetbyf d w (t) , which is defined by the following equa- tion: f d w (t) = f w (t) + ε w (t)(1) wheref w (t)refers to the pattern of how word frequency changes over time, whileε w (t)represents random varia- tions. Even without considering randomness, the frequency of words can be influenced by multiple factors, such as changes in research topics or writing conventions. In the following, we assume that the baseline word frequency changes linearly over time and then model the effect of LLM usage as a factor that modifies this established linear trend, as shown in the formula below: f w (t) = a w + b w t + c w (t)(2) wherea w andb w are word-specific coefficients, whilec w (t) represents the effect introduced by LLMs. Based on observational data from the pre-LLM periodt∈ T 1 , we estimate the parametersa w andb w andˆa w and ˆ b w . 2 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers These estimates are then used to predict the word frequency f pred w (t) over the later time interval t∈T 2 as follow: f pred w (t) = ˆa w + ˆ b w t.(3) As shown in Figures 1, the linear regression results based on data from 2021 and earlier can also reasonably predict the word frequencies for the first ten months of 2022, i.e., before the emergence of ChatGPT. Therefore, we can use the linear regression described above to predict how word frequencies would change over time as if LLMs did not exist. 3.2. Impact Estimation For different modelsm, they have varying preferences for words, which also depend on the prompt. Letf 0 w represent the word frequency in human-written text. For modelm under promptp, the word frequency isf m,p w , and its mean across prompts is denoted by f m w . For the observed values in Equation 1 and the predicted values in Equation 3, we can easily obtain the following ratio: r w (t) = f d w (t) f pred w (t) .(4) If we further assume thatη 0 (t)represents the proportion in human-written text at timet, andη m,p (t)represents the proportion in text generated by modelxand promptp. Then, under the assumption that the observed text is a mixture of these components, we can obtain the following approximate relationship: r w (t)≈ η 0 (t) + X m∈M,p∈P η m,p (t) f m w f 0 w ,(5) whereMrepresents the set of all LLMs under consideration, andP represents the set of all prompts being considered. If we denote all the unknowns related toηas the vector η(t), and defineM ∗ andP ∗ to include both human and non-prompted cases to accommodateη 0 . Then for the set of all considered wordsW,η(t)can be determined using the following relation: min η(t) X w∈W   r w (t)− X m∈M ∗ ,p∈P ∗ η m,p (t) f m,p w f 0 w   2 s.t. η 0 (t) + X m∈M ∗ ,p∈P η m,p (t) = 1, η m,p (t)∈ [ℓ m,p ,u m,p ], ∀m∈M ∗ , p∈P ∗ . (6) It is important to note that the value ofη 0 may be very close to 1, for example, before the emergence of ChatGPT. At the same time, other values ofηmay be close to 0. Therefore, although the loss function can be adjusted, it needs to be done with caution. 3.3. Text Similarity Comparison and Distinction We explore two different tasks: generating the title and modifying the abstract. To examine the similarities and differences between human-authored texts and those gen- erated by LLMs, as well as across outputs from different LLMs, we initially assess textual similarity using multiple metrics. We then attempt to train classifiers to categorize texts generated by different LLMs. For example, we consider three specific cases of ROUGE (Lin, 2004) to compare the lexical similarity of texts: ROUGE-1 (word-level comparison), ROUGE- 2 (bigram-level comparison), and ROUGE-L (comparison based on the longest common subsequence). We also use BERTScore (Zhang et al., 2019), based on a pre-trained semantic neural network, to assess their semantic similarity between two pieces of text. Referring to the design and parameters of a previous study (Sun et al., 2025), we also train classifiers to dis- tinguish texts generated by different LLMs, based on three classical models: BERT (Devlin et al., 2019), GPT- 2 (Radford et al., 2019), T5 (Raffel et al., 2020), and LLM2Vec (BehnamGhader et al., 2024). The selection of parameters, such as the learning rate is consistent with that of the previous paper (Sun et al., 2025). Although adjusting the model and its parameters might yield better classification results, the purpose of this section is to validate our dataset using the current best open-source classifier, so no modifications were made. 3.4. Word Frequency Comparison Word frequency can also be analyzed from various perspec- tives. For example, we can define frequency change ratio as follows: r w,m = f m w − f 0 w f 0 w .(7) The variance of the frequency of a wordwin the same texts given a promptpacross different modelsmis expressed as: Var w,p = 1 |M ∗ |− 1 X m∈M ∗ (f m,p w − ̄ f p m ) 2 (8) where the average value ̄ f p m can be represented as ̄ f p m = 1 |M ∗ | X m∈M ∗ f m,p w .(9) Subsequently, the coefficient of variation is defined as CV w,p = p Var w,p ̄ f p m .(10) 3 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Table 1. Release timeline of the LLMs used in this paper. Dates indicate the initial public availability, verified by official announce- ments. Model NameRelease Date GPT-3.5 (OpenAI, 2022)Dec 30, 2022 GPT-4o mini (OpenAI, 2024)Jul 18, 2024 GPT-5 nano (OpenAI, 2025)Aug 07, 2025 DeepSeek V3 (DeepSeek-AI, 2024)Dec 26, 2024 DeepSeek R1 (DeepSeek-AI, 2025a)Jan 20, 2025 DeepSeek V3.2 (DeepSeek-AI, 2025b)Dec 01, 2025 Gemini 2.5 Flash (Google, 2025a)Jun 17, 2025 Gemini 2.5 Pro (Google, 2025a)Jun 17, 2025 Gemini 3 Flash (Google, 2025b)Dec 17, 2025 Claude 3 Haiku (Anthropic, 2024)Mar 13, 2024 Claude Haiku 4.5 (Anthropic, 2025)Oct 15, 2025 Therefore, we can identify which words behave differently across different LLMs. 4. Simulation 4.1. Models Simulations were conducted using nine different models from the GPT, DeepSeek, Gemini, and Claude for simu- lation. Their release dates range from November 2022 to December 2025, as illustrated in Table 1. More details on the simulation using the API can be found in Appendix Section A.2. We are aware that some LLMs are not covered in this study, and some versions of GPT and Claude are not included in the experiments. But the models we choose are expected to yield fairly stable estimation results. 4.2. Prompts Given the diversity of real-world application scenarios, we adopt two prompts of different lengths, one long and one short, for simulation. Short Prompt for Abstract Rewriting : It is possible that authors restrict their use of LLM tools to light assistance, such as minor rewrites. Therefore, we consider the short prompt as follows. Revise the following sentences. Please only output the revised text in JSON format. Only one version. No explanations. Only plain text. EXAMPLE JSON OUTPUT:"text": "Your revised version of the provided sentences goes here." Other Prompts: We can also provide more details in a longer prompt to simulate a researcher maximizing LLM capabilities for deep editing, like requesting the persona of a 0510 Word Frequency Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek beyond 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 20 40 60 80 100 120 Frequency (per 10,000 Titles) beyond Real Data Linear Trend Fit Period (a) Left: the frequency of the word “beyond” in the titles of 2,000 simulated papers. Right: its frequency in actual arXiv titles (across all categories). 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 200 400 600 800 1000 Frequency (per 10,000 Titles) via (cs) Real Data Linear Trend Fit Period 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 150 200 250 300 350 400 via (non-cs) (b) Temporal change in the frequency of the word “via” in actual arXiv paper titles for CS and non-CS categories. Figure 2. Examples of word frequencies in paper titles. professional academic editor. The long prompt for abstract rewriting can be found in Appendix A.1, along with the prompt used for title generation. 5. Results 5.1. Word Choice in the Titles LLMs are asked to generate titles based on real abstracts. The word distributions of these titles also differ from those written by humans, which is not surprising. For example, as illustrated in Figures 1a and 2a, newer LLMs such as DeepSeek and GPT-5 tend to favor the words “via” and “beyond” in paper titles, and their actual frequency only begins to significantly exceed the predicted values starting from 2025. We also analyze the differences between CS (computer sci- ence) and non-CS papers. As shown in Figure 2b, the fre- quency of “via” in titles has increased noticeably in both categories. Given the relatively small number of words in titles, we focus primarily on the analysis of abstracts. 5.2. Commonly Used Words LLMs have their own preferences for word usage. Some commonly used words are often regarded as stopwords that carry little informational content, and are therefore removed in natural language processing (Sarica & Luo, 2021), but there are differences between LLMs and humans in the usage of many common words. For instance, for the 20 most common words in the abstracts, their word variation 4 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers the of and a to in we is for that with on this are by as an from which be GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2 DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 Word Frequency Change Ratio (Post- vs. Pre-LLM Processing) 0.6 0.4 0.2 0.0 0.2 0.4 0.6 Figure 3. Frequency change ratios of the 20 most frequent words in 2,000 abstracts using multiple LLMs and the shorter prompt. 05000100001500020000 Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude the 02500500075001000012500 of Word Frequency Figure 4. Word frequencies in 2,000 abstracts or in the corre- sponding LLM-processed content. Error bars denote the standard deviation of word frequencies across outputs produced by different LLMs and/or prompts. frequency defined in Equation 7 is illustrated in Figure 3. Similar to previous results, although it is difficult to estimate LLM usage based on individual words, some correlations can still be observed. The earlier Figure 1b shows a decline in the frequencies of “the” and “of”, and Figure 4 demon- strates that most LLMs may indeed avoid using these two words. To better present word preferences across LLMs, we plot the frequency of the 20 words most favored and the frequency of the 20 words least favored by LLMs, based on the coefficient of variation defined in Equation 10, in Figures 5a and 5b. Thus, in theory, the changes in the frequencies of these common words can also be used to estimate the influence of LLMs on academic publications. 5.3. Changes and Evolution of LLMs LLMs are also continuously evolving, and different LLMs have distinct word preferences. The word “together” in Fig- ure 1b serves as a clear example. Words previously consid- ered characteristic of ChatGPT, like “delve” and “intricate”, are being abandoned by newer LLMs, as seen in Figure 9 in the Appendix. There are also more examples of the change in word usage preferences, including even some more common words. For outcomes crucial utilized additionally methodology findings regarding characterized conducted advances utilize precise focuses indicate inherent furthermore examine illustrate thereby characteristics Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2 DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 Word Frequency 100 200 300 400 500 600 700 800 (a) The 20 words with the highest change coefficients that LLMs prefer to use. usually besides etc give mainly great done able comes behaviour greatly gives hence takes uses enough together whole interesting idea Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2 DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 Word Frequency 0 20 40 60 80 100 (b) The 20 words with the highest change coefficients that humans prefer to use. Figure 5. The frequencies of some words that appeared at least 20 times in the 2,000 abstracts used for the simulation. instance, as illustrated in Figure 10 in the appendix, the frequency of “and” has recently exceeded the predicted up- ward trend, whereas the frequency of “this” first decreased and then increased. These shifts may reflect differences in word preference between some recent models such as GPT-5 nano and earlier models, as shown in Figure 3. In addition, Figure 5a indicates that, except for GPT-5 Nano, other models show a strong preference for the word “furthermore”. The frequency of this term rises sharply but returns to a level close to its previous growth trajectory recently, as illustrated in Figure 11 of the appendix. This pattern could also indicate that GPT-5 Nano or models like it are more widely used in academic writing. The changes in word frequency in abstracts discussed above are observed across the entire arXiv corpus. Similar patterns also emerge when CS and non-CS papers are examined sep- arately. For example, Figure 12 in the appendix shows the frequency changes of “the” and “of” in these two groups. In- teractive results for more words and categories are available on the website 3 . 3 https://llm-impact.github.io/word-usage -arxiv-abstract/ 5 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.5 0.6 0.7 0.8 0.9 1.0 ROUGE-1 GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash ROUGE-1 over Release Dates (Abstract) Original Abstract vs LLM-Generated Abstract (long prompt) Original Abstract vs LLM-Generated Abstract (short prompt) LLM-Generated Abstract (long prompt) vs LLM-Generated Abstract (short prompt) GPT DeepSeek Gemini Claude 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.86 0.88 0.90 0.92 0.94 0.96 0.98 1.00 BERTScore GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash BERTScore over Release Dates (Abstract) Original Abstract vs LLM-Generated Abstract (long prompt) Original Abstract vs LLM-Generated Abstract (short prompt) LLM-Generated Abstract (long prompt) vs LLM-Generated Abstract (short prompt) GPT DeepSeek Gemini Claude (a) Similarity between the processed texts of the two prompts and their similarity to the real abstracts. 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.12 0.14 0.16 0.18 0.20 0.22 ROUGE-1 GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash ROUGE-1 over Release Dates (Title) GPT DeepSeek Gemini Claude GPT DeepSeek Gemini Claude 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.76 0.77 0.78 0.79 0.80 0.81 BERTScore GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash BERTScore over Release Dates (Title) GPT DeepSeek Gemini Claude GPT DeepSeek Gemini Claude (b) Similarity between the titles generated by LLMs and the real titles. Figure 6. Comparison Results of Text Similarity. The x-axis represents the model release dates. The changes and evolution of LLMs also require us to esti- mate and monitor their impact with a dynamic perspective. The differences among various LLMs offer valuable insights for estimating the usage and influence of different models. 5.4. Text Similarity Analysis Apart from the differences in word usage frequency, we aim to examine it from a broader perspective of text similarity (e.g., ROUGE-1 and BERTScore). The comparison between the original abstracts and titles and the content processed or generated by LLMs, as well as the comparison of the same model’s outputs across different prompts, are shown Table 2. Classification results. Prompt: long, Classifier: BERT, Model Type: old. The black squares indicate the source of the detected text, and the accuracy shown is the highest value obtained across the parameter set. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Acc (%)78.6 90.1 90.1 95.3 76.9 93.4 91.8 Table 3. Classification results. Prompt: long, Classifier: BERT, Model Type: new. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Acc (%)63.0 83.0 90.4 84.9 83.8 70.9 88.2 in Figure 6. The more comprehensive comparison results can be found in Figure 13 of the Appendix. Figures 6a and 13a indicate that the abstracts processed by the two prompts are close to each other, whereas they show a larger difference from the original ones. Claude Haiku 3 stands as an exception, as the abstracts resulting from the shorter prompt are more similar to the original abstracts. In general, the results from the two prompts of the new LLMs are gradually aligning more closely with the original abstracts under ROUGE metrics. However, semantically, this trend is not as evident. Similar conclusions can be drawn from the title generation in Figures 6b and 13b: while the new LLMs are closer to the true titles according to ROUGE, this is not reflected in the semantic results from BERTScore. The analysis in this part further highlights the differences between various LLMs, as well as the changes occurring within models of the same series. It is important to note that higher similarity does not necessarily mean stronger model performance, which may simply indicate that the model has made fewer modifications to the output text. 5.5. Classification We consider two scenarios in classification tasks: distin- guishing between texts generated by different LLMs, and a multi-class classification that includes texts written by humans. The method and code we use here are based on the rare open-source work that involves classifying texts from different LLMs (Sun et al., 2025). In the first scenario, we consider binary and four-class clas- 6 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Real abstract GPT 3.5 GPT 4o mini GPT 5 nano DeepSeek Gemini Claude Predicted Class Real abstract GPT 3.5 GPT 4o mini GPT 5 nano DeepSeek Gemini Claude True Class 77.47%1.10%2.20%9.34%1.10%2.20%6.59% 2.75%71.98%14.84%1.10%1.65%5.49%2.20% 0.00%18.13%67.58%0.00%3.30%9.34%1.65% 6.59%0.00%1.65%66.48%6.04%3.85%15.38% 3.30%7.69%14.29%8.79%19.78%31.87%14.29% 1.10%6.59%9.89%2.20%12.09%59.34%8.79% 13.19%2.75%5.49%8.24%19.78%18.13%32.42% GPT-2 Confusion Matrix 0 10 20 30 40 50 60 70 Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude Predicted Class Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude True Class 84.07%0.00%0.00%3.85%1.65%0.00%10.44% 0.55%87.36%6.59%0.55%2.20%0.55%2.20% 0.00%8.79%80.77%0.00%7.69%1.65%1.10% 1.65%0.00%0.00%76.37%6.59%4.40%10.99% 1.10%3.30%3.85%2.75%49.45%18.13%21.43% 0.00%0.55%5.49%0.55%13.74%72.53%7.14% 9.34%1.10%0.55%6.59%18.68%6.04%57.69% LLM2Vec Confusion Matrix 0 10 20 30 40 50 60 70 80 Figure 7. Confusion matrix of classification results from the de- tectors based on GPT-2 and LLM2Vec. In the training and test data, GPT-3.5, GPT-4o mini, and GPT-5-nano are mixtures of two prompts, while DeepSeek, Gemini, and Claude are mixtures of multiple versions and two prompts. After mixing, each class contains 2,000 abstracts. sification. An earlier version (GPT-3.5, DeepSeek V3, Gem- ini 2.5 Flash, Claude 3 Haiku) and the most recent ver- sion (GPT-5 nano, DeepSeek V3.2, Gemini 3 Flash, Claude Haiku 4.5) from each of the four model families (GPT, DeepSeek, Gemini, and Claude) are chosen for comparison. Tables 2 and 3 present the classification results of the BERT- based classifier for texts generated with long prompt, while Table 4 to Table 13 in the Appendix provide more compari- son results. The values in the table represent the results with the highest accuracy in the test set with different parameters. In general, the results indicate that the abstracts polished by different LLMs show some differences, with binary classifi- cation accuracy reaching around 80%-90% and four-class classification accuracy at about 60%. In the old models, the long prompt generally achieves higher classification accu- racy than the short prompt. In other words, the differences between different LLMs are narrowing, which may indicate a homogenizing effect. For the second scenario, we directly perform a seven-class or thirteen-class classification, including texts written by humans. The results of the seven categories shown in Fig- ure 7 and Figure 14 of the appendix refer to three GPT models, along with DeepSeek, Gemini, and Claude (with mixed outputs from different models in each series), as well as human-written texts. In Figures 16 to 19 of the appendix, the thirteen categories correspond to the classification of each model separately. The table 15 in the Appendix sum- marizes some of the results. It can be observed from these results that the accuracy de- clines with the increase in categories. While the accuracy of classifying human-written texts and GPT-processed texts is quite high, it’s also fairly common for human-written text to be mistakenly identified as LLM-generated. In real- world scenarios, the number of prompts is far greater than two, and there are more models involved than those in our experiment. 5.6. Impact Estimation Given that real abstracts have about 20% or even more chance of being misclassified as LLM-generated with the classifier described above, we use the previously described word frequency-based estimation, which is easier to inter- pret. Based on Equation 6, the estimated impact of LLMs varies depending on the different words and parameters. For example, we can set η m,p (t)∈ [0, 1], ∀m∈M ∗ , p∈P ∗ (11) To demonstrate the effectiveness of the simple method, we used SLSQP (Sequential Least Squares Programming) to solve this equation, with the initial value ofη m,p (t)uni- formly distributed. Figure 8 presents the estimates derived from the models in Table 1 and the prompts in Section 4.2. The results in Figure 8a are based on 2,741 words that are relatively common in both the Google Books Ngram dataset and arXiv abstracts, while the results in Figure 8b are derived from the stopwords in NLTK that appear at least once in the 2,000 arXiv abstracts. These results are generally consistent with our expectations, such as the LLM impact estimates being close to zero before October 2022. We can also observe in Figure 8a that the proportion of text similar to the GPT-3.5 style first increases and then decreases. The outputs of various LLMs show some similarities (Jiang et al., 2025), which could reduce the accuracy of predictions. The estimates based on stopwords in Figure 8b show larger fluctuations, likely due to the low differentiation of these 7 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers 2022-012023-012024-012025-012026-01 0.0 0.2 0.4 0.6 0.8 1.0 (t) Human-writtenGPT-3.5GPT-4o miniGPT-5 nanoDeepSeekGeminiClaude (a) The 2,741 words used for estimation are drawn from the 10,000 most frequent words in the Google Books Ngram dataset that occur at least 10 times in the 2,000 preprocessed abstracts used for simulation. 2022-012023-012024-012025-012026-01 0.0 0.2 0.4 0.6 0.8 1.0 (t) Human-writtenGPT-3.5GPT-4o miniGPT-5 nanoDeepSeekGeminiClaude (b) The 126 words used for estimation are drawn from NLTK’s English stopwords in the Google Books Ngram dataset that occur in the 2,000 cleaned abstracts used for simulation. Figure 8. Estimation of LLM impact in arXiv abstracts. words across different models, even though they differ from human-written text. 6. Related Work LLM-Generated Text Detection. Researchers have pro- posed many detectors for identifying text generated by LLMs, with a variety of approaches and techniques (Wu et al., 2025). However, in real-world scenarios, such as academic writing, human modifications to LLM outputs can indeed affect the effectiveness of detection (Russell et al., 2025). Considering the variety of models and use cases, it may be an impossible task to accurately estimate the numerical usage of LLMs (Geng & Poibeau, 2025). LLM Fingerprint. Some LLMs each have their own lin- guistic fingerprints, such as in vocabulary and lexical diver- sity, the frequency of certain lexical and morphosyntactic features (Reviriego et al., 2024; McGovern et al., 2025). These features can be used to trace their origins (Nikolic et al., 2025) as well as to detect LLM-generated text (An- toun et al., 2024; Sun et al., 2025) and check API calls (Gao et al., 2024). Similar studies have also appeared in other languages, although some LLMs may be quite similar and difficult to distinguish (Zaitsu et al., 2025). LLMs for Scientific Writing. Many people are dis- cussing the use of LLMs in scientific research, and this has become an unstoppable trend (Eger et al., 2025). The disclosure of AI usage may impact how people perceive the article (Hazra et al., 2025). Despite the widespread use of AI in academic writing, there are few papers that disclose its usage (He & Bu, 2025). The scope of LLMs’ use in sci- entific research goes beyond just writing, such as research fields (Hao et al., 2026), citations (Algaba et al., 2025), and more. Impact EstimationThe influence of LLMs on academic publications is diverse (Kusumegi et al., 2025). In addi- tion to the previously mentioned works, some papers at- tempt to quantify these impacts, such as through the use of words (Liang et al., 2025) and different linguistic shifts (Bao et al., 2025; Lin et al., 2025). 7. Discussion The focus of this paper is on the impact of LLMs rather than on whether the text was generated by them. Classifiers are likely to perform well in binary classification, but their accuracy significantly decreases in multi-class tasks. In real-world scenarios, however, the latter situation is more common. Although the method of analyzing word frequency may sound simple, this intuitive approach proves to be quite ef- fective in analyzing the impact of LLMs. Focusing on more common words may provide better estimates, and our sim- ple method can fill the gap left by complex classifiers. Our estimates can also be improved, for example, by considering more models and prompts. In the era of LLMs, the way people use words is continually evolving (Geng & Trotta, 2025; Mak & Walasek, 2025). The effect of LLMs on text goes beyond academic publications. As similar impacts are likely to increase in the future, the methods and findings presented in this paper could serve as important references for subsequent research. The language style of LLMs, including their preference for certain words, requires further exploration (Juzek & Ward, 2025). The opportunities these tools provide differ for authors from various regions (e.g., Eastern vs. Western) (Khan et al., 2025), though our paper does not explore this aspect. The modification of content and information is more troubling than any change in wording (Mohamed et al., 2025; Ab- dulhai et al., 2026). The generation of content may require more caution, such as hallucinated references (Sakai et al., 2026). 8 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers 8. Conclusion We systematically analyze similarities and differences be- tween arXiv abstracts and titles processed by different LLMs. Compared to human-written texts, LLM-generated outputs exhibit distinct stylistic characteristics, with addi- tional variation observed across different models. While complex classifiers can be employed for detection, the application of such black-box methods in real-world scenarios raises concerns. Nevertheless, it remains possible to estimate the influence of LLMs by examining the differ- ences between their outputs and human-written text. Our study also shows that LLMs, especially the GPT series, have a significant impact on the writing in arXiv abstracts, but other models likely have considerable usage as well. Considering both similarity comparisons and detector out- comes, model outputs may indeed become progressively more human-like, raising more challenges for AI-generated text detection. Furthermore, humans could be subtly shaped by machines. Consequently, approaches for tracking and evaluating their impact will need to be continuously refined as LLMs advance. Acknowledgments This work benefited from funding from the French State, managed by the Agence Nationale de la Recherche, under the France 2030 program (grant reference ANR-23-IACL- 0008). This research also received support from the ENS- PSL BeYs Chair in Data Science and Cybersecurity. References Abdulhai, M., White, I., Wan, Y., Qureshi, I., Leibo, J., Kleiman-Weiner, M., and Jaques, N. How llms distort our written language. arXiv preprint arXiv:2603.18161, 2026. Algaba, A., Holst, V., Tori, F., Mobini, M., Verbeken, B., Wenmackers, S., and Ginis, V. How deep do large lan- guage models internalize scientific literature and citation practices? arXiv preprint arXiv:2504.02767, 2025. Anthropic. Claude 3 Haiku.https://w.anthropi c.com/news/claude-3-haiku , 2024. Accessed: 2026-01-25. Anthropic. Claude Haiku 4.5.https://w.anth ropic.com/news/claude-haiku-4-5 , 2025. Accessed: 2026-01-25. Antoun, W., Sagot, B., and Seddah, D. From text to source: Results in detecting large language model-generated con- tent. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Re- sources and Evaluation (LREC-COLING 2024), p. 7531– 7543, 2024. arXiv.org submitters. arxiv dataset, 2024. URLhttps: //w.kaggle.com/dsv/7548853. Bao, T., Zhao, Y., Mao, J., and Zhang, C. Examining linguis- tic shifts in academic writing before and after the launch of chatgpt: a study on preprint papers. Scientometrics, p. 1–31, 2025. BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. Llm2vec: Large lan- guage models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961, 2024. Bird, S. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 interactive presentation sessions, p. 69–72, 2006. Bizzoni, Y., Degaetano-Ortlieb, S., Fankhauser, P., and Te- ich, E. Linguistic variation and change in 250 years of english scientific writing: A data-driven approach. Fron- tiers in Artificial Intelligence, 3:73, 2020. Cheng, M., Gligori ́ c, K., Piccardi, T., and Jurafsky, D. An- throscore: A computational linguistic measure of anthro- pomorphism. In Proceedings of the 18th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume 1: Long Papers), p. 807–825, 2024. DeepSeek-AI. DeepSeek V3 release news. https://ap i-docs.deepseek.com/zh-cn/news/news1 226, 2024. Accessed: 2026-01-20. DeepSeek-AI. DeepSeek R1 release news.https://ap i-docs.deepseek.com/zh-cn/news/news2 50120, 2025a. Accessed: 2026-01-20. DeepSeek-AI. DeepSeek V3.2 release news.https: //api-docs.deepseek.com/zh-cn/news/n ews251201, 2025b. Accessed: 2026-01-20. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186, 2019. Eger, S., Cao, Y., D’Souza, J., Geiger, A., Greisinger, C., Gross, S., Hou, Y., Krenn, B., Lauscher, A., Li, Y., et al. Transforming science with large language models: A survey on ai-assisted scientific discovery, experimenta- tion, content generation, and evaluation. arXiv preprint arXiv:2502.05151, 2025. 9 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Gao, I., Liang, P., and Guestrin, C. Model equality test- ing: Which model is this api serving? arXiv preprint arXiv:2410.20247, 2024. Geng, M. and Poibeau, T. On the detectability of llm- generated text: What exactly is llm-generated text? arXiv preprint arXiv:2510.20810, 2025. Geng, M. and Trotta, R. Is chatgpt transforming academics’ writing style? arXiv preprint arXiv:2404.08627, 2024. Geng, M. and Trotta, R. Human-llm coevolution: Evidence from academic writing. arXiv preprint arXiv:2502.09606, 2025. Google. Gemini 2.5 model family expands.https:// blog.google/products-and-platforms/p roducts/gemini/gemini-2-5-model-famil y-expands/, 2025a. Accessed: 2026-01-20. Google. Gemini 3 Flash.https://blog.google/pr oducts-and-platforms/products/gemini /gemini-3-flash/, 2025b. Accessed: 2026-01-20. Gray, A. Estimating the prevalence of llm-assisted text in scholarly writing. arXiv preprint arXiv:2512.01560, 2025. Hao, Q., Xu, F., Li, Y., and Evans, J. Artificial intelligence tools expand scientists’ impact but contract science’s fo- cus. Nature, p. 1–7, 2026. Hazra, S., Lee, D., Majumder, B. P., and Kumar, S. Accepted with minor revisions: Value of ai-assisted scientific writ- ing. arXiv preprint arXiv:2511.12529, 2025. He, Y. and Bu, Y. Academic journals’ ai policies fail to curb the surge in ai-assisted academic writing. arXiv preprint arXiv:2512.06705, 2025. Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., and Choi, Y. Artificial hivemind: The open-ended homogeneity of language models (and beyond). arXiv preprint arXiv:2510.22954, 2025. Juzek, T. S. and Ward, Z. B. Why does chatgpt “delve” so much? exploring the sources of lexical overrepresentation in large language models. In Proceedings of the 31st international conference on computational linguistics, p. 6397–6411, 2025. Khan, F. K., Ibrahim, H., Aldahoul, N., Rahwan, T., and Zaki, Y. Who gets seen in the age of ai? adoption patterns of large language models in scholarly writing and citation outcomes. arXiv preprint arXiv:2509.08306, 2025. Kobak, D., Gonz ́ alez-M ́ arquez, R., Horv ́ at, E.- ́ A., and Lause, J.Delving into chatgpt usage in academic writing through excess vocabulary.arXiv preprint arXiv:2406.07016, 2024. Kusumegi, K., Yang, X., Ginsparg, P., de Vaan, M., Stuart, T., and Yin, Y. Scientific production in the era of large language models. Science, 390(6779):1240–1243, 2025. Liang, W., Izzo, Z., Zhang, Y., Lepp, H., Cao, H., Zhao, X., Chen, L., Ye, H., Liu, S., Huang, Z., et al. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. arXiv preprint arXiv:2403.07183, 2024. Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., Cao, H., Liu, S., He, S., Huang, Z., et al. Quantifying large language model usage in scientific papers. Nature Human Behaviour, p. 1–11, 2025. Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, p. 74–81, 2004. Lin, D., Zhao, N., Tian, D., and Li, J. Chatgpt as linguistic equalizer? quantifying llm-driven lexical shifts in aca- demic writing. arXiv preprint arXiv:2504.12317, 2025. Mak, M. H. and Walasek, L. Style, sentiment, and quality of undergraduate writing in the ai era: A cross-sectional and longitudinal analysis of 4,820 authentic empirical reports. Computers and Education: Artificial Intelligence, p. 100507, 2025. McGovern, H. E., Stureborg, R., Suhara, Y., and Alikaniotis, D. Your large language models are leaving fingerprints. In Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect), p. 85–95, 2025. Mohamed, A., Geng, M., Vazirgiannis, M., and Shang, G. Llm as a broken telephone: Iterative generation distorts information. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7493–7509, 2025. Nikolic, I., Baluta, T., and Saxena, P.Model prove- nance testing for large language models. arXiv preprint arXiv:2502.00706, 2025. OpenAI. Introducing ChatGPT.https://openai.c om/index/chatgpt/, 2022. Accessed: 2026-01-20. OpenAI. GPT-4o mini: Advancing cost-efficient intelli- gence.https://openai.com/index/gpt-4 o-mini-advancing-cost-efficient-intel ligence/, 2024. Accessed: 2026-01-20. 10 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers OpenAI. Introducing GPT-5.https://openai.com /index/introducing-gpt-5/, 2025. Accessed: 2026-01-20. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1–67, 2020. Reviriego, P., Conde, J., Merino-G ́ omez, E., Mart ́ ınez, G., and Hern ́ andez, J. A. Playing with words: Comparing the vocabulary and lexical diversity of chatgpt and humans. Machine Learning with Applications, 18:100602, 2024. Russell, J., Karpinska, M., and Iyyer, M. People who fre- quently use chatgpt for writing tasks are accurate and robust detectors of ai-generated text. arXiv preprint arXiv:2501.15654, 2025. Sakai, Y., Kamigaito, H., and Watanabe, T. Hallucitation matters: Revealing the impact of hallucinated references with 300 hallucinated papers in acl conferences. arXiv preprint arXiv:2601.18724, 2026. Sarica, S. and Luo, J. Stopwords in technical language processing. Plos one, 16(8):e0254937, 2021. Sun, M., Yin, Y., Xu, Z., Kolter, J. Z., and Liu, Z. Id- iosyncrasies in large language models. arXiv preprint arXiv:2502.12150, 2025. Wang, G., Wang, H., Sun, X., Wang, N., and Wang, L. Linguistic complexity in scientific writing: A large-scale diachronic study from 1821 to 1920. Scientometrics, 128 (1):441–460, 2023. Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., and Wong, D. F. A survey on llm-generated text detection: Necessity, methods, and future directions. Computational Linguistics, 51(1):275–338, 2025. Wu, K., Pang, L., Shen, H., Cheng, X., and Chua, T.-S. Llmdet: A third party large language models generated text detection tool. In Findings of the association for computational linguistics: emnlp 2023, p. 2113–2133, 2023. Zaitsu, W., Jin, M., Ishihara, S., Tsuge, S., and Inaba, M. Stylometry can reveal artificial intelligence authorship, but humans struggle: A comparison of human and seven large language models in japanese. PLoS One, 20(10): e0335369, 2025. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. 11 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers A. Simulation Details A.1. Other prompts Long prompt for abstract rewriting: Role: Act as a professional academic editor and reviewer for top-tier scientific journals (e.g., Nature, IEEE Transactions, ACM). You have expertise in technical writing, grammar, and scientific logic. Task: Refine and polish the specific text provided below. Your goal is to improve clarity, coherence, and academic tone while strictly maintaining the original technical meaning. Constraints & Guidelines: 1. Grammar & Syntax: Correct all grammatical, spelling, and punctuation errors. 2. Academic Tone: Use formal, objective, and precise language. Avoid colloquialisms, contractions, or overly flowery language. 3. Clarity & Flow: Improve sentence structure to enhance readability. Break down overly complex sentences if necessary, but ensure the logical flow remains smooth. 4. Vocabulary: Replace weak or repetitive words with more precise academic vocabulary suited for a technical context. 5. Preservation: - Do NOT change the core scientific meaning or data. - Do NOT alter specific variable names, LaTeX formulas, citations (e.g., [1]), or technical terminology unless they are clearly incorrect. 6. Conciseness: Remove redundancy and fluff. Output Format: Please only output the polished text in JSON format. Only one version. No explanations. Only plain text. EXAMPLE JSON OUTPUT: "text": "Your polished version of the provided text goes here." Prompts for title generation: Read the following abstract of a research paper. Generate a representative title that accurately reflects the main contribution of the work. Constraint: You must output ONLY a valid JSON object. Do not include any introductory or concluding text. Format:"title": "Your generated title here" A.2. API Usage All polished texts utilized in this study were generated via cloud-based LLM services rather than local execution. To ensure data consistency and mitigate potential disruptions caused by network instability, all LLM responses were ac- quired using Python-based batch API processing instead of interactive chat interfaces. The APIs were configured to output data in JSON format to strictly exclude extraneous explanatory text. All other settings were retained at their default values. The specific model APIs employed are listed below: 1.OpenAI Models:gpt-3.5-turbo, gpt-4o-mini, gpt-5-nano. Access period: January 13, 2025 – January 14, 2025. 2.DeepSeek Models:deepseek-chat, deepseek-reasoner. Note: DeepSeek provides only the latest model versions on its official website without specific version reference. According to the official documentation, the actual version that was executed during the access period was DeepSeek-V3.2. Access period: January 13, 2025 – January 15, 2025. 3.AlibabaCloud Models:deepseek-v3, deepseek-r1. Note: These models utilized open-source code provided by DeepSeek and were hosted on servers operated by the Alibaba Cloud internet vendor. Access period: January 13, 2025 – January 16, 2025. 4.Google Models:gemini-2.5-flash, gemini-2.5-pro, gemini-3-flash-preview. Access period: January 13, 2025 – January 15, 2025. 5.Anthropic Models: claude-3-haiku-20240307, claude-haiku-4-5-20251001. Access period: January 21, 2026. B. Other Results 12 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers 0204060 Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude delve 0255075100125 intricate Word Frequency 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 0 10 20 30 40 50 Frequency (per 10,000 abstracts) delve Real Data Linear Trend Fit Period 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 0 25 50 75 100 125 150 intricate Figure 9. Supplementary comparison results. 02000400060008000 Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude and 0100020003000 this Word Frequency 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 35000 40000 45000 50000 Frequency (per 10,000 abstracts) and Real Data Linear Trend Fit Period 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 10000 11000 12000 13000 14000 this Figure 10. Supplementary comparison results. 0200400600 Real abstracts GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude furthermore 0100200300 thereby Word Frequency 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 400 500 600 700 800 900 Frequency (per 10,000 abstracts) furthermore Real Data Linear Trend Fit Period 2015-012016-012017-012018-012019-012020-012021-012022-012023-012024-012025-012026-01 100 200 300 400 500 thereby Figure 11. Supplementary comparison results. (a) Word frequency in CS abstracts. (b) Word frequency in non-CS abstracts. Figure 12. Comparison Results of Text Similarity. The x-axis represents the model release dates. 13 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Table 4. Classification results. Prompt: long, Classifier: GPT-2, Model Type: old. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy78.2%87.1%89.0%94.8%71.7%90.1%89.8% Table 5. Classification results. Prompt: long, Classifier: GPT-2, Model Type: new. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy65.2%84.6%90.1%81.9%77.7%62.9%82.7% Table 6. Classification results. Prompt: long, Classifier: T5, Model Type: old. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy72.8%90.7%91.2%94.8%73.4%91.2%91.8% Table 7. Classification results. Prompt: long, Classifier: T5, Model Type: new. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy57.3%88.5%93.1%84.1%83.8%58.5%82.4% Table 8. Classification results. Prompt: short, Classifier: BERT, Model Type: old. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy73.1%82.1%89.3%91.5%71.2%89.3%93.4% Table 9. Classification results. Prompt: short, Classifier: BERT, Model Type: new. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy65.5%80.8%89.8%83.8%87.6%76.6%86.8% 14 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Table 10. Classification results. Prompt: short, Classifier: GPT-2, Model Type: old. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy69.0%80.2%88.2%89.6%73.1%85.7%90.9% Table 11. Classification results. Prompt: short, Classifier: GPT-2, Model Type: new. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy58.2%76.6%89.6%83.0%80.8%70.6%83.0% Table 12. Classification results. Prompt: short, Classifier: T5, Model Type: old. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy70.9%83.5%91.2%92.9%69.8%86.3%92.9% Table 13. Classification results. Prompt: short, Classifier: T5, Model Type: new. GPT■ · · · DeepSeek■ · ·■ · Gemini■ ·■ ·■ ·■ Claude■ · ·■ ·■ Accuracy63.5%82.7%90.4%85.4%83.2%64.6%84.3% Table 14. Summary of Binary Classification Accuracy (%) across different Model Types, Prompts, and Classifiers. Abbreviations: G: GPT, D: DeepSeek, Ge: Gemini, C: Claude. Columns represent the ensemble combination. Model TypePromptClassifier AllG + DG + GeG + CD + GeD + CGe + C (G+D+Ge+C) New Long BERT63.083.090.484.983.870.988.2 GPT-265.284.690.181.977.762.982.7 T557.388.593.184.183.858.582.4 Short BERT65.580.889.883.887.676.686.8 GPT-258.276.689.683.080.870.683.0 T563.582.790.485.483.264.684.3 Old Long BERT78.690.190.195.376.993.491.8 GPT-278.287.189.094.871.790.189.8 T572.890.791.294.873.491.291.8 Short BERT73.182.189.391.571.289.393.4 GPT-269.080.288.289.673.185.790.9 T570.983.591.292.969.886.392.9 15 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ROUGE-1 GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash ROUGE-1 over Release Dates Original Abstract vs LLM-Generated Abstract (long prompt) Original Abstract vs LLM-Generated Abstract (short prompt) LLM-Generated Abstract (long prompt) vs LLM-Generated Abstract (short prompt) GPT DeepSeek Gemini Claude 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ROUGE-2 GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash ROUGE-2 over Release Dates Original Abstract vs LLM-Generated Abstract (long prompt) Original Abstract vs LLM-Generated Abstract (short prompt) LLM-Generated Abstract (long prompt) vs LLM-Generated Abstract (short prompt) GPT DeepSeek Gemini Claude 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ROUGE-L GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash ROUGE-L over Release Dates Original Abstract vs LLM-Generated Abstract (long prompt) Original Abstract vs LLM-Generated Abstract (short prompt) LLM-Generated Abstract (long prompt) vs LLM-Generated Abstract (short prompt) GPT DeepSeek Gemini Claude 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.800 0.825 0.850 0.875 0.900 0.925 0.950 0.975 1.000 BERTScore GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash BERTScore over Release Dates Original Abstract vs LLM-Generated Abstract (long prompt) Original Abstract vs LLM-Generated Abstract (short prompt) LLM-Generated Abstract (long prompt) vs LLM-Generated Abstract (short prompt) GPT DeepSeek Gemini Claude (a) Similarity between the processed texts of the two prompts and their similarity to the real abstracts. 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.050 0.075 0.100 0.125 0.150 0.175 0.200 0.225 0.250 ROUGE-1 GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash ROUGE-1 over Release Dates GPT DeepSeek Gemini Claude GPT DeepSeek Gemini Claude 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 ROUGE-2 GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash ROUGE-2 over Release Dates GPT DeepSeek Gemini Claude GPT DeepSeek Gemini Claude 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.050 0.075 0.100 0.125 0.150 0.175 0.200 0.225 0.250 ROUGE-L GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash ROUGE-L over Release Dates GPT DeepSeek Gemini Claude GPT DeepSeek Gemini Claude 2023-012023-052023-092024-012024-052024-092025-012025-052025-092026-01 Release Date 0.700 0.725 0.750 0.775 0.800 0.825 0.850 0.875 0.900 BERTScore GPT-3.5 Claude 3 Haiku GPT-4o mini DeepSeek V3 DeepSeek R1 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5 nano Claude Haiku 4.5 DeepSeek V3.2-chat DeepSeek V3.2-reasoner Gemini 3 Flash BERTScore over Release Dates GPT DeepSeek Gemini Claude GPT DeepSeek Gemini Claude (b) Similarity between the titles generated by LLMs and the real titles. Figure 13. Comparison Results of Text Similarity. The x-axis represents the model release dates. 16 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Table 15. Comparison of 7-class classification performance across BERT, GPT-2, T5, and LLM2Vec models. Metrics are reported in percentage (%). P denotes Precision, R denotes Recall. Class BERTGPT-2T5LLM2Vec PRF1PRF1PRF1PRF1 Real abstract82.2468.6874.8574.2177.4775.8185.0671.9877.9886.9384.0785.47 GPT 3.574.0776.9275.4766.5071.9869.1379.6268.6873.7586.4187.3686.89 GPT 4o mini64.7172.5368.3958.2967.5862.6060.6878.0268.2783.0580.7781.89 GPT 5 nano68.4563.1965.7169.1466.4867.7967.5074.1870.6884.2476.3780.12 DeepSeek36.1830.2232.9331.0319.7824.1638.7126.3731.3749.4549.4549.45 Gemini44.5671.9855.0445.5759.3451.5548.5573.6358.5270.2172.5371.35 Claude41.7426.3732.3239.8632.4235.7648.0634.0739.8751.9857.6954.69 Overall Accuracy58.5656.4460.9972.61 17 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Real abstract GPT 3.5 GPT 4o mini GPT 5 nano DeepSeek Gemini Claude Predicted Class Real abstract GPT 3.5 GPT 4o mini GPT 5 nano DeepSeek Gemini Claude True Class 68.68%0.00%0.55%12.09%1.10%1.65%15.93% 1.65%76.92%9.89%0.55%3.85%5.49%1.65% 0.00%13.74%72.53%0.00%3.85%9.34%0.55% 4.40%0.00%0.55%63.19%14.29%5.49%12.09% 0.55%4.40%13.19%5.49%30.22%40.66%5.49% 0.00%4.95%9.89%1.10%10.99%71.98%1.10% 8.24%3.85%5.49%9.89%19.23%26.92%26.37% BERT Confusion Matrix 0 10 20 30 40 50 60 70 Real abstract GPT 3.5 GPT 4o mini GPT 5 nano DeepSeek Gemini Claude Predicted Class Real abstract GPT 3.5 GPT 4o mini GPT 5 nano DeepSeek Gemini Claude True Class 71.98%0.55%0.00%12.64%2.75%2.20%9.89% 1.10%68.68%18.13%1.10%4.40%4.95%1.65% 0.00%9.34%78.02%0.55%3.30%7.69%1.10% 2.75%1.10%2.20%74.18%6.04%6.04%7.69% 1.10%4.40%14.29%6.59%26.37%34.07%13.19% 0.00%1.10%11.54%1.65%8.79%73.63%3.30% 7.69%1.10%4.40%13.19%16.48%23.08%34.07% T5 Confusion Matrix 0 10 20 30 40 50 60 70 Figure 14. Confusion matrix of classification results from the GPT-2-based detector and T5-based detector. 18 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude Predicted Class Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek Gemini Claude True Class 84.07%0.00%0.00%3.85%1.65%0.00%10.44% 0.55%87.36%6.59%0.55%2.20%0.55%2.20% 0.00%8.79%80.77%0.00%7.69%1.65%1.10% 1.65%0.00%0.00%76.37%6.59%4.40%10.99% 1.10%3.30%3.85%2.75%49.45%18.13%21.43% 0.00%0.55%5.49%0.55%13.74%72.53%7.14% 9.34%1.10%0.55%6.59%18.68%6.04%57.69% LLM2Vec Confusion Matrix 0 10 20 30 40 50 60 70 80 Figure 15. Confusion matrix of classification results from the LLM2Vec-based detector. 19 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 Predicted Class Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2 DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 True Class 71.43%0.55%0.55%4.95%0.00%0.00%1.10%1.10%0.55%1.10%0.00%16.48%2.20% 0.55%71.43%10.44%0.00%4.95%0.55%0.55%1.65%0.55%1.65%2.75%4.95%0.00% 0.00%18.13%64.84%0.00%5.49%0.55%1.10%0.55%1.65%3.30%2.75%1.65%0.00% 5.49%0.00%1.65%54.95%0.55%0.55%4.40%2.20%0.00%3.85%4.40%7.14%14.84% 0.00%6.59%14.84%0.55%24.73%4.95%7.14%5.49%9.34%3.85%14.29%3.85%4.40% 2.20%1.10%0.55%1.65%17.58%17.03%4.40%8.24%11.54%9.34%14.29%3.30%8.79% 1.65%3.30%8.24%2.75%15.93%2.75%22.53%8.24%3.85%6.59%7.14%6.59%10.44% 2.20%3.30%9.89%3.85%12.09%4.40%12.64%12.09%6.04%6.59%11.54%7.69%7.69% 0.00%1.65%6.59%0.00%8.24%2.20%3.30%4.95%39.56%12.09%18.13%1.65%1.65% 0.55%0.55%0.55%1.65%1.65%2.20%2.75%0.00%13.74%53.85%18.13%1.65%2.75% 0.00%1.10%1.65%1.65%6.04%2.20%0.55%2.20%7.14%13.19%61.54%0.55%2.20% 17.03%1.10%2.75%5.49%0.00%1.10%6.59%4.95%1.65%4.40%2.20%47.25%5.49% 1.10%0.55%1.10%9.34%8.79%7.69%7.14%6.04%2.20%6.59%9.89%2.20%37.36% bert (lr=1e-04, accuracy=44.51%) Confusion Matrix 0 10 20 30 40 50 60 70 Figure 16. Confusion matrix of BERT (lr=1e-04) 20 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 Predicted Class Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2 DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 True Class 70.33%0.00%1.10%8.79%0.55%0.55%0.55%3.30%0.55%2.20%0.00%11.54%0.55% 2.20%75.82%6.04%1.10%4.40%0.55%1.10%0.55%2.20%1.10%1.65%3.30%0.00% 0.00%16.48%59.89%0.55%6.04%0.00%1.10%1.10%6.04%1.10%4.40%3.30%0.00% 8.24%0.55%0.55%56.59%0.55%1.65%2.75%4.95%0.55%3.85%3.30%5.49%10.99% 1.10%10.44%7.69%3.30%33.52%3.85%4.95%4.40%9.34%4.95%7.69%3.85%4.95% 4.40%2.20%2.20%4.40%10.99%13.74%3.85%7.14%15.38%9.34%12.09%2.20%12.09% 4.40%3.30%9.89%6.59%11.54%4.95%11.54%10.44%6.04%11.54%7.14%5.49%7.14% 7.14%4.95%4.95%9.89%10.44%3.30%7.14%13.19%5.49%10.99%9.34%6.04%7.14% 1.10%5.49%6.59%3.30%4.40%5.49%1.10%4.95%44.51%12.09%9.89%0.00%1.10% 1.65%1.10%2.20%2.20%4.95%3.30%1.65%4.95%13.74%47.80%8.24%3.30%4.95% 0.55%2.20%3.85%5.49%5.49%2.75%1.65%2.20%9.34%14.29%49.45%1.10%1.65% 21.43%0.00%2.20%9.34%1.10%2.20%3.85%0.55%2.75%3.30%2.20%43.41%7.69% 2.20%0.00%0.00%14.29%5.49%10.99%3.85%6.04%5.49%9.34%6.04%2.20%34.07% gpt2 (lr=3e-04, accuracy=42.60%) Confusion Matrix 0 10 20 30 40 50 60 70 Figure 17. Confusion matrix of GPT-2 (lr=3e-04) 21 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 Predicted Class Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2 DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 True Class 72.53%0.00%0.00%6.04%0.00%1.10%0.00%1.65%0.00%1.10%0.00%15.38%2.20% 0.55%65.38%19.78%0.55%4.95%0.00%1.10%1.10%2.20%0.00%0.55%2.75%1.10% 0.00%12.09%74.73%0.00%2.20%0.55%0.55%1.65%2.75%0.00%2.75%1.65%1.10% 3.30%0.00%2.20%63.19%0.00%0.55%3.30%4.40%0.55%3.30%5.49%4.40%9.34% 0.55%5.49%12.64%1.65%29.12%5.49%4.40%4.95%10.99%2.20%15.38%2.20%4.95% 0.55%1.10%3.85%3.30%14.29%12.09%3.85%7.14%8.79%9.34%13.19%6.04%16.48% 2.75%1.10%12.64%8.24%9.89%4.40%14.29%10.99%5.49%6.59%6.59%8.24%8.79% 2.75%1.10%9.34%5.49%7.69%2.75%12.64%14.29%9.34%10.99%9.89%6.59%7.14% 0.00%3.30%9.89%0.55%3.30%3.30%3.30%2.75%37.91%16.48%15.38%0.55%3.30% 0.00%0.55%3.30%1.10%1.65%1.10%2.20%1.10%9.89%53.85%17.03%1.65%6.59% 0.00%0.00%3.85%2.20%4.40%2.20%0.00%2.75%8.79%12.64%59.34%0.55%3.30% 21.43%0.55%4.40%8.79%0.55%1.65%5.49%4.95%1.65%1.65%2.20%40.66%6.04% 2.20%0.00%1.10%12.09%8.24%4.95%4.95%4.40%2.20%10.44%7.69%2.75%39.01% t5 (lr=3e-04, accuracy=44.34%) Confusion Matrix 0 10 20 30 40 50 60 70 Figure 18. Confusion matrix of T5 (lr=3e-04) 22 Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 Predicted Class Real abstract GPT-3.5 GPT-4o mini GPT-5 nano DeepSeek V3 DeepSeek R1 DeepSeek V3.2 DeepSeek V3.2 Thinking Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude 3 Haiku Claude Haiku 4.5 True Class 81.87%0.00%0.00%4.95%0.00%2.20%0.00%0.55%0.00%0.00%0.00%10.44%0.00% 0.55%81.87%7.69%0.55%2.20%0.55%0.55%1.10%1.65%0.00%0.55%2.75%0.00% 0.00%8.79%74.73%0.00%3.85%1.10%2.20%4.95%1.65%0.00%0.55%2.20%0.00% 2.20%0.00%0.00%75.82%0.55%1.65%2.20%3.85%0.00%2.20%3.30%4.40%3.85% 0.00%4.40%4.40%0.55%47.80%9.89%3.85%8.24%5.49%3.85%6.04%1.10%4.40% 0.00%2.75%1.65%2.20%10.44%36.81%4.95%9.89%9.34%4.40%4.95%3.30%9.34% 1.65%1.65%3.85%6.59%13.74%6.04%19.23%19.23%3.85%8.24%2.75%4.95%8.24% 1.10%2.20%2.20%4.95%10.44%5.49%9.89%35.71%5.49%4.95%3.30%6.59%7.69% 0.00%0.55%2.20%0.00%6.04%6.59%2.20%6.59%59.89%5.49%4.95%2.20%3.30% 0.00%0.55%0.55%1.10%0.55%0.55%1.10%3.30%6.04%71.98%10.99%1.10%2.20% 0.00%0.55%0.55%2.20%3.85%2.75%0.55%1.10%2.75%13.19%70.88%1.10%0.55% 17.58%1.10%0.55%4.95%1.10%2.20%2.20%4.95%3.30%0.55%2.20%56.04%3.30% 0.55%0.55%0.00%5.49%3.30%9.34%7.14%4.95%2.20%4.40%2.20%3.85%56.04% llm2vec (lr=5e-05, accuracy=59.13%) Confusion Matrix 0 10 20 30 40 50 60 70 80 Figure 19. Confusion matrix of LLM2Vec (lr=5e-05) 23

Instant research discovery