Paper deep dive
A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2
Marcin PietroĹ, Filip Gampel, Jakub GomuĹka, Andrzej Tomski, RafaĹ Olszowski
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 12:10:38 PM
Summary
This study evaluates the performance of state-of-the-art Large Language Models (LLMs), specifically GPT-5.2, Llama 4, and DeepSeek R1, on argument classification tasks using the UKP and Args.me datasets. The research investigates the impact of advanced prompting strategies, including Chain-of-Thought, prompt rephrasing, voting, and certainty estimation. Results indicate that GPT-5.2 achieves the highest accuracy (91.9% on Args.me, 78.0% on UKP), and ensemble techniques improve performance by 2-8%. Qualitative analysis highlights systematic failure modes such as instability with prompt formulation and difficulty in detecting implicit criticism.
Entities (8)
Relation Signals (6)
GPT-5.2 â achievesaccuracyon â Args.me
confidence 95% ¡ The best-performing model in the study (GPT-5.2) achieves a classification accuracy of 78.0% (UKP) and 91.9% (Args.me).
GPT-5.2 â achievesaccuracyon â UKP
confidence 95% ¡ The best-performing model in the study (GPT-5.2) achieves a classification accuracy of 78.0% (UKP) and 91.9% (Args.me).
LLMs â outperforms â Traditional Machine Learning
confidence 90% ¡ Recent advances in large language models (LLMs) have significantly improved the performance of argument classification compared to traditional machine learning approaches.
GPT-5.2 â outperforms â Llama-4
confidence 90% ¡ The results reveal that the GPT model family outperforms the Llama and DeepSeek models.
Chain-of-Thought â improves â Argument Classification
confidence 85% ¡ The use of prompt rephrasing, multi-prompt voting, and certainty estimation further improves classification performance and robustness.
Voting â improves â Argument Classification
confidence 85% ¡ The use of prompt rephrasing, multi-prompt voting, and certainty estimation further improves classification performance and robustness.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Argument mining (AM) is an interdisciplinary research field focused on the automatic identification and classification of argumentative components, such as claims and premises, and the relationships between them. Recent advances in large language models (LLMs) have significantly improved the performance of argument classification compared to traditional machine learning approaches. This study presents a comprehensive evaluation of several state-of-the-art LLMs, including GPT-5.2, Llama 4, and DeepSeek, on large publicly available argument classification corpora such as this http URL and UKP. The evaluation incorporates advanced prompting strategies, including Chain-of- Thought prompting, prompt rephrasing, voting, and certainty-based classification. Both quantitative performance metrics and qualitative error analysis are conducted to assess model behavior. The best-performing model in the study (GPT-5.2) achieves a classification accuracy of 78.0% (UKP) and 91.9% (this http URL). The use of prompt rephrasing, multi-prompt voting, and certainty estimation further improves classification performance and robustness. These techniques increase the accuracy and F1 metric of the models by typically a few percentage points (from 2% to 8%). However, qualitative analysis reveals systematic failure modes shared across models, including instabilities with respect to prompt formulation, difficulties in detecting implicit criticism, interpreting complex argument structures, and aligning arguments with specific claims. This work contributes the first comprehensive evaluation that combines quantitative benchmarking and qualitative error analysis on multiple argument mining datasets using advanced LLM prompting strategies.
Tags
Links
- Source: https://arxiv.org/abs/2603.19253v1
- Canonical: https://arxiv.org/abs/2603.19253v1
Trouble viewing inline? Open PDF directly â
Full Text
106,319 characters extracted from source content.
Expand or collapse full text
A COMPREHENSIVE STUDY OF LLM-BASED ARGUMENT CLASSIFICATION: FROM LLAMA THROUGH DEEPSEEK TO GPT-5.2 Marcin Pietro Ě n Faculty of Computer Science, Electronics, and Telecommunications AGH University of Krakow pietron@agh.edu.pl Filip Gampel Faculty of Humanities AGH University of Krakow fgampel@agh.edu.pl Jakub GomuĹka Faculty of Humanities AGH University of Krakow jgomulka@agh.edu.pl Andrzej Tomski Institute of Mathematics University of Silesia andrzej.tomski@us.edu.pl RafaĹ Olszowski Faculty of Humanities AGH University of Krakow rolszowski@agh.edu.pl ABSTRACT Argument mining (AM) is an interdisciplinary research field focused on the automatic identification and classification of argumentative components, such as claims and premises, and the relationships between them. Recent advances in large language models (LLMs) have significantly improved the performance of argument classification compared to traditional machine learning approaches. However, there remains a lack of systematic comparative evaluation of modern LLMs across multiple benchmark datasets, as well as a limited understanding of their error patterns and failure modes. This study presents a comprehensive evaluation of several state-of-the-art LLMs, including GPT-5.2, Llama 4, and DeepSeek R1, on large publicly available argument classification corpora such as Args.me and UKP. The evaluation incorporates advanced prompting strategies, including Chain-of- Thought prompting, prompt rephrasing, voting, and certainty-based classification. Both quantitative performance metrics and qualitative error analysis are conducted to assess model behavior. The best-performing model in the study (GPT-5.2) achieves a classification accuracy of 78.0% (UKP) and 91.9% (Args.me). The use of prompt rephrasing, multi-prompt voting, and certainty estimation further improves classification performance and robustness. These techniques increase the accuracy and F1 metric of the modeuls by typically a few percentage points (from 2% to 8%). However, qualitative analysis reveals systematic failure modes shared across models, including instabilities with respect to prompt formulation, difficulties in detecting implicit criticism, interpreting complex argument structures, and aligning arguments with specific claims. The findings provide new insights into the strengths and limitations of LLMs in automated argument mining and highlight the importance of prompt engineering and ensemble techniques. This work contributes the first comprehensive evaluation that combines quantitative benchmarking and qualitative error analysis on multiple argument mining datasets using advanced LLM prompting strategies. Keywords Argument mining¡ Transformers¡ Large Language Models¡ Cognitive intelligence¡ NLP 1 Introduction Argument mining (AM) is an interdisciplinary research area spanning philosophy, logic, linguistics, rhetoric, law, psychology, and computer science. Within artificial intelligence, AM focuses on the automatic identification and analysis of argumentative structures in natural language texts. The goal is to extract components such as claims and premises, as well as the relationships between them, enabling structured representations of argumentative discourse arXiv:2603.19253v1 [cs.CL] 25 Feb 2026 A comprehensive study of LLM-based argument classification (Park & Cardie, 2018). In recent years, AM has become an important subfield of Data Mining and Natural Language Processing (NLP), with applications in decision support systems, opinion analysis, legal informatics, and computational social science. Early computational approaches to AM relied primarily on feature-based machine learning methods and manually engineered representations (Cabrio and Villata, 2018, Schaefer and Stede, 2022, Lawrence and Reed, 2020). With the rise of neural architectures, including recurrent and convolutional neural networks, performance improved substantially, particularly for component detection and relation classification tasks. The introduction of Transformer-based architec- tures, such as BERT, marked a significant breakthrough in NLP, enabling contextualized representations that improved performance in many AM benchmarks (Pietron et al., 2024; Stab et al., 2018a). More recently, large language models (LLMs) such as Llama and GPT have demonstrated strong generalization capabilities across diverse reasoning and language understanding tasks. Despite these advances, the application of LLMs to structured argument classification remains insufficiently explored. Most standardized LLM benchmarks focus on multiple-choice reasoning, question answering, logical puzzles, or code generation. In contrast, argument classification requires the identification of implicit premises, pragmatic context, and nuanced semantic relations between propositions. Unlike many benchmark tasks with clearly defined rule-based solutions, argumentative reasoning often involves ambiguity, contextual interpretation, and subtle distinctions between support, attack, and neutrality relations. This makes argument classification a particularly challenging and informative testbed for evaluating the reasoning capabilities of LLMs. Recent research has proposed prompting strategies that encourage intermediate reasoning steps, such as Chain-of- Thought prompting and tree-structured exploration mechanisms (Wei et al., 2022). These approaches aim to improve performance in tasks that require multi-step inference rather than direct answer prediction. However, it remains unclear whether such reasoning-oriented prompting strategies translate into measurable improvements in structured argument mining tasks. In particular, the extent to which LLMs can replicate or approximate human annotation behavior in argument relation classification is still an open question. In AM, the analytical pipeline is typically divided into three main subtasks of increasing complexity: (i) identification of argumentative components, distinguishing argumentative from non-argumentative segments; (i) classification of component types, such as premises and claims; and (i) identification of relational properties, determining whether one proposition supports, attacks, or is unrelated to another. The third subtaskâargument relation classificationârequires simultaneous semantic and logical analysis of multiple textual units and therefore constitutes a suitable framework to evaluate the reasoning abilities of modern LLMs. This paper investigates the capability of contemporary LLMs to perform argument relation classification using two widely recognized datasets derived from established AM projects: UKP argument corpora and the Args.me corpus. Both datasets contain manually annotated argumentative relations, providing a reliable ground truth for systematic evaluation. We compare model predictions with human annotations and perform both quantitative and qualitative analyses to assess strengths and limitations. Our study demonstrates how different models, varying in size and reasoning capabilities, perform in argument classification tasks. The comparison is conducted using the Accuracy and F1-score metrics. The results reveal that the GPT model family outperforms the Llama and DeepSeek models. Additionally, we show that improving the prompt algorithm with rephrased prompts and incorporating a voting strategy can significantly improve performance, increasing Accuracy and F1 (from 2% to 8%). The study is guided by the following research questions: RQ1. How does a particular syntax of prompts affect the quality of argument classification? RQ2. What are the performance differences between different language models and how does the size of the LLMs increase the capabilities of argument classification? RQ3. How do prompting and reasoning algorithms improve argument classification by LLMs? RQ4. How can a certainty-based multiprompt improve the Accuracy of argument classification? RQ5. What types of error do LLMs make in automatic argument classification? RQ6. What are the shortcomings of existing annotated datasets commonly used to train or evaluate networks and models in argument classification, and how should they be improved? The main contributions of this work are: ⢠A systematic evaluation of state-of-the-art LLMs on two established argument mining datasets. ⢠A comparative analysis of different prompting strategies. ⢠A qualitative error analysis highlighting limitations of LLMs in structured argument reasoning. To the best of our knowledge, this study presents one of the first evaluations across a wide spectrum of modern large language models (LLMs) on argument classification benchmarks. In addition, it introduces the development of an 2 A comprehensive study of LLM-based argument classification efficient prompt algorithm for analyzing arguments in natural language. Additionally, it presents a qualitative in-depth analysis of LLM errors in the argument classification process. 2 Related works Argument classification (AC) is a specialized subtask in the broader field of AM. It concentrates on categorizing the identified elements of an argument into predetermined classes. These classes often include differentiating claims and premises, or identifying whether a component supports or opposes the argument. Additionally, argument classification can entail assessing the nature of the argument, such as determining its type or evaluating its quality (e.g., whether it is a strong or weak argument). This task is integral to constructing a structured representation of arguments. By clarifying the role and relationship of each component, it enhances our understanding of the dynamics within an argumentative discourse (Lippi & Torroni, 2016). Argument classification (AC), as a specialized subtask within AM, was described by Daxenberger et al. (2020), Lippi and Torroni (2016), Dusmanu et al. (2017) and many others. According to these authors, AC involves categorizing the identified components of an argument into predefined categories, such as differentiating between claims and premises or determining the stance of an argumentative component, whether supporting or opposing. Various data science techniques that utilize natural language processing have proven effective in AC and, more broadly, in AM. Initially, argument structures were often represented using trees or tree-like models, facilitating computation due to the availability of tree-based parsing techniques. However, real-world arguments frequently deviate from these idealized structures. More recently, researchers have shifted towards exploring non-tree-based argument structures in argument mining. Before the advent of BERT and other Transformer-based models, Support Vector Machines (SVMs) and neural networks were pivotal in AM, leveraging their pattern recognition and classification capabilities to identify and analyze argumentative structures in the text. Architectures such as Recurrent Neural Networks (RNN), e.g. Long Short-Term Memory (LSTM), and Convolutional Neural Networks (CNN) have been used to incorporate contextual information into machine decision-making processes. Niculae et al. (2017) introduced the first non-tree model for argument mining, using a factor graph model combined with structured Support Vector Machines and Bidirectional Long Short-Term Memory. SVMs have been widely used in argument mining due to their effectiveness in binary classification tasks, which are well-suited to identifying whether a particular sentence or phrase is an argumentative component (e.g., claim vs. non-claim). Subsequently, Galassi et al. (2018) utilized LSTMs and residual network links to predict the connections among argument components. LSTM based approach for argument classification is also presented in Stab et al., 2018a. These LSTM-based solutions have limits and can achieve up to 45% accuracy in the UKP dataset, (Li, 2020; Stab et al., 2018a). While SVMs and neural networks significantly contributed to the development of AM, they had limitations, such as the need for extensive feature engineering and difficulties in capturing long-distance dependencies in text. The introduction of BERT (Bidirectional Encoder Representations from Transformers) and subsequent Transformer-based models revolutionized AM, (Li, 2020; Pietron et al., 2024). The Transformer architecture, featuring its self-attention mechanism, was initially introduced by Vaswani et al. (2017) as a solution to the increasing computational and memory demands of recurrent neural networks (RNNs), which were considered state of the art at the time. Research on how transformers work, especially the BERT model (Devlin et al., 2019), has attracted significant interest. Fine-tuning large pre-trained Transformer-based models resulted in remarkable performance gains across a wide range of tasks. Several recent studies have employed transformer-based models for argument mining. Reimers et al. (2019) leveraged contextual word embeddings such as BERT and ELMo to significantly enhance argument/non-argument classification and proposed methods for argument clustering. Chakrabarty et al. (2019) introduced a BERT-based model for argument component classification and relation detection within persuasive online conversations. Going further, Chen (2021) is using pre-trained BERT based models for predicting arguments where the structure forms a directed acyclic graph. Moreover, Ruiz-Dolz et al. (2020) present an analysis of the behavior of transformer-based models (i.e., BERT, XLNET, RoBERTa, DistilBERT and ALBERT) when predicting argument relations, and evaluate the models in five different domain specific corpora, with the objective of finding the less domain dependent model. The work presented in Li, 2020 shows that DistilBERT achieves 54.2% and the BERT precision is around 57.7% in the UKP benchmark. In Pietron et al., 2024, the authors present a hybrid of BERT and ChatGPT-4. The proposed architecture significantly outperforms other ML-based solutions. It achieves 89.5% accuracy in the Args.me and 68.5% on the UKP benchmark. In this work, the LLM is used as an additional classifier on selected subsets of arguments for which the BERT model shows low response confidence. The main shortcomings of this work are the lack of evaluation of the GPT-4 model on entire datasets and the lack of in-depth error analysis. Recent Transformer-based models include BLOOM (Le Scao et al., 2023), Llama (Touvron et al., 2023), GPT-5 and DeepSeek-R1 (Guo et al., 2025). There is a lack of works which study the performance of these models in argument mining, especially those with reasoning capabilities. Chain-of-Thought is an approach that simulates human-like reasoning processes by delineating complex tasks into a sequence of logical steps 3 A comprehensive study of LLM-based argument classification towards a final solution. This methodology reflects a fundamental aspect of human intelligence, offering a structured mechanism for problem-solving. This technique can be applied in prompt or incorporated into the LLM, forming a "reasoning model". Given the challenges posed by fully automated AM methods, an interesting research direction has emerged in exploring hybrid approaches that combine the efforts of human annotators with AI. The automated AM methods often struggle to determine whether two arguments express the same viewpoint (Chakrabarty et al., 2019; Daxenberger et al., 2017), and reliance on a limited set of labeled data can lead to the exclusion of minority opinions, thereby creating a bias towards more popular or frequently repeated arguments. For example, a hybrid method called HyEnA (Van der Meer et al., 2024) employs a sampling algorithm that guides human annotators individually through an opinion corpus. Then, an intelligent merging strategy helps annotators combine their results into clusters of arguments, integrating both manual and automatic labeling. 3 Data Dataset choice In our research, we decided to conduct a comparative study using two corpora containing different argument datasets, each of which was developed by a different research team. These are: the UKP corpus (Stab et al., 2018b) and the Args.me corpus (Ajjour et al., 2019). These corpora have gained recognition in recent argument mining research, but they have not yet been studied comparatively or with recent large language models such as Llama, DeepSeek or GPT-5. Previous studies, including Bar-Haim et al., 2017, Boltuzic and Snajder, 2014, Stab et al., 2018b were limited to research on a single corpus, and mostly covered pre-transformer-based NLP technologies. Both corpora used in our study share fundamental structural characteristics of argumentation, namely: ⢠Presence of a thesis â a central conclusion or topic around which the argumentation is constructed; ⢠Provision of a list of premises â each labeled as supporting or opposing the thesis, or with no relation (UKP) The form and quality of argument descriptions vary between datasets; for instance, some include off-topic entries, non-arguments, or rephrased conclusions in place of genuine premises. The UKP corpus (Stab et al., 2018b) comprises datasets of arguments derived from online comments on eight controversial topics: abortion, cloning, the death penalty, gun control, minimum wage, nuclear energy, school uniforms, and marijuana legalization. This corpus includes over 25,000 instances. The sentences were independently annotated by seven individuals recruited through the Amazon Mechanical Turk (AMT) crowdsourcing platform. For each classification, an agreement level was required, with Cohenâs kappa (Îş) set at 0.723, surpassing the commonly accepted threshold of 0.7 for reliable results (Carletta, 1996). The classification labels used were: (1) supporting argument (Argument_for), (2) opposing argument (Argument_against), and (3) non-argument (NoArgument). The Args.me corpus (version 1.0, cleaned) provided by Ajjour et al. (2019) consists of arguments collected from four debate portals in mid-2019: Debatewise, IDebate.org, Debatepedia, and Debate.org. The arguments were extracted using heuristics specifically designed for each debate portal. The datasets used in our simulations are Idebate.org, Debatepedia, and Debatewise, which together contain 47,992 arguments. Debatepedia has the highest number of arguments, while IDebate.org shows the least disproportion between PRO and CON arguments, with the greatest class imbalance observed in Debatepedia. The annotations in this dataset include conclusions and premises, which are further categorized into PRO premises (arguments supporting the thesis) and CON premises (arguments opposing the thesis). Samples used in this study We performed an initial screening of all datasets to crop very long records (>2000 characters in theargument/sentencefield). In the context of this study, these will be referred to as the "full" sets. For each dataset, we then created a "trimmed" version of 2000 records for heavy calculations, involving large or proprietary models, etc. The trimmed versions were sampled randomly from the full sets, ensuring that the original class imbalance is preserved. The record counts of both sets can be found in tables 1 and 2. Appendix E shows detailed sample sizes for each calculation performed in the study. 4 Experimental setup 4.1 System architecture and model choice The general architecture of the proposed approach is illustrated in Fig. 1. The input to the system comes from the different datasets. From the aggregated data, arguments are forwarded to the Prompt Generator module; optionally, a thesis statement may be included. 4 A comprehensive study of LLM-based argument classification Table 1: UKP class counts for full and trimmed datasets DatasetFull ForFull AgainstFull NoArgFull TotalTrim ForTrim AgainstTrim NoArgTrim Total abortion6808222427392934641812362000 cloning706839149430394655529832000 death45711112083365125060911412000 gun7876651889334147139811312000 marijuana5876261262247547450610202000 nuclear6068522118357633947611852000 school5457291734300836248511532000 wage5765511346247346644610882000 Table 2: Args.me class counts for full and trimmed datasets DatasetFull ForFull AgainstFull TotalTrim ForTrim AgainstTrim Total debatepedia1578754062119314905102000 debatewise810956421375111798212000 idebate644566031304898810122000 The Prompt Generator produces a set of prompts instructing the models to classify the relation of the argument to the thesis, and to estimate the certainty of their answer. If the thesis is not provided (as in the case of the UKP corpus), the Prompt Generator injects a thesis based on the argument domain. In the context of this study, for each argument-thesis pair we generate N = 4 prompts. Subsequently, a set of instances of a LLM responds to the generated prompts. The output of each instance is parsed into a label prediction along with a confidence score. For our ensemble method, the outputs are then passed to the Voting module, which applies several ensemble voting strategies to generate the final system prediction. As we will show, the voting mechanisms help mitigate errors produced by individual prompts. In our study, we test the performance of various recent popular large language models, coming from the Llama, Deepseek and GPT family. Details on all models chosen for the study are provided in Appendix A. 4.2 Prompting strategies Our study focuses on assessing the performance of general-purpose LLMâs without any prior specific training on a similar task or dataset related to argument mining. All experiments are related to the task of argument classification. Since prompting techniques severely impact the performance of LLM models in many areas, an attempt was made to examine some of them. mRAR promptingRephrase and Respond (RaR) is a prompting technique in which LLMâs are asked to rephrase and expand questions posed by the user and provide responses to these (Deng et al., 2023). For the purpose of our study, we develop a standard set of four prompts, all aimed at the same task of classifying the stance of an argumantative text with respect to a topic or thesis. We will call this manual rephrased and respond strategy (mRAR). The prompts differ in: â˘Response format: Two prompts ask the model to format the response as a single letter (F/A/N), while two ask for verbal labels (For/Against/No argument). â˘Thesis presentation: Two prompts are rather verbose, while the other two feature a shorter and simpler formulation of the task. These relations are visualized in Fig. 2. The prompts were minimally adapted to match the dataset format (three answer options for UKP vs. two for Args.me). The full texts of all prompts can be found in Appendix B. Chain-of-Thought prompting The Chain of Thought (CoT) technique aims at mirroring human reasoning and problem-solving through a coherent series of logical deductions (Wei et al., 2022). As a prompting strategy, the LLM is usually asked to break down the task into smaller steps, sequentially analyzing and summarizing all available information and context before answering (Zhou et al., 2023). At the time of writing, "Reasoning models" (Besta et al., 2025) are the de-facto industry standard for proprietary and high-end language models. These models are specifically trained to perform multiple steps of logical reasoning in order to solve complex tasks. However, some 5 A comprehensive study of LLM-based argument classification Figure 1: System architecture with voting strategy popular open-weight models like the Llama series do not have inbuilt reasoning capacities. For these models, we study the effect of explicitly encouraging the model to arrive at its answer through step-by-step reasoning (see Appendix B). Certainty self-assessment Finally, for the four mRaR prompts, a second prompt was appended to the conversation, after the model provided its classification answer. The prompt asked the model to assess how certain its answer was on a percentage scale from 0 to 100 (see Appendix B). These self-assessments allow as to combine the answers from the four prompts through a number of voting algorithms to obtain a final aggregate answer. The algorithms are described in detail in the following section. 4.3 Voting algorithms The classification of the textpusing the promptiby the modelT Î , together with a certainty rating, may be formally denoted as T Î (p i )7âLĂ [0, 1](1) HereL = F,A,Nis the set of possible predictions, whereFindicates an argument predicted as âForâ,Athe prediction âAgainstâ, andNsignifies âNo Argumentâ. LetT Î (p i ) = (q i ,c i )denote the prediction of a labelq i âL with certainty c i â [0, 1]. We will consider three versions of the aggregated voting alogrithm: â˘Simple vote The most frequent answer among the promptings is chosen. In case of a tie, the result is chosen randomly from the winning options. This version of the voting algorithm ignores the certainty self-ratings. Let m be the number of prompts. For each class q âF,A,N, define the vote count: 6 A comprehensive study of LLM-based argument classification Figure 2: Overview of prompt complexity used in the study. V (q) = m X i=1 1[q i = q],(2) where 1[¡] denotes the indicator function. Now define the set of winners: W = q âL : V (q) = max kâL V (k) .(3) The final classification is given by (U is the uniform distribution): q pred âźU (W)(4) â˘Tiebreak vote Like in simple vote, the result is the most frequent answer. However in case of a tie, the certainty self-ratings are used to determine the winner. ForW as above, define the aggregated certainty score: S(q) = m X i=1 c i 1[q i = q], q âW(5) Then the final prediction is: q pred = arg max qâW S(q)(6) ⢠Weighted vote For each category, a certainty-weighted sum of votes is calculated. The winner is the option with the highest score. For all classes q, consider certainty score: S(q) = m X i=1 c i 1[q i = q].(7) The predicted label is: q pred = arg max qâL S(q).(8) The Tiebreak vote is presented in Algorithm 1. The main loop iterates over the set of prompts and queries the model for the argument classification and the certainty of its answer (lines from 3 to 15). After each answer, it updates the aggregated results. Then it checks which classes were recognized most frequently (line 16). If there is only one such class, it constitutes the final answer (line 19). Otherwise, the confidence level (certainty) of the answers is taken into account and the final predicted class is determined on the basis of this (line 21). 7 A comprehensive study of LLM-based argument classification Algorithm 1 Majority Voting with Certainty-Based Tie-Breaking Require: L â LLM model returning (q,c) Require: ⌠â list of prompts 1: δ f ,δ a ,δ n â 0 2: c f ,c a ,c n â 0 3: for Ď â ⌠do 4:(q,c)â L(Ď) 5:if q == âForâ then 6: δ f â δ f + 1 7: c f â c f + c 8:else if q == âAgainstâ then 9: δ a â δ a + 1 10: c a â c a + c 11:else 12: δ n â δ n + 1 13: c n â c n + c 14:end if 15: end for 16: ââ max(δ f ,δ a ,δ n ) 17: W âk âf,a,n : δ k = â 18: if|W| = 1 then 19: outâ k such thatW =k 20: else 21: outâ arg max kâW c k 22: end if 23: return out 5 Numerical results In this section results from running a wide range of argument classification experiments are reported. To better assess the reliability of the final model performance calculations, we will first analyze how model performance depends on factors like prompt formulation, dataset and sampling randomness. 5.1 Variability Prompt variability (RQ1) As described in Sec. 4.2, the two main distinctions within our basic prompt set are answer format (P1,P3 vs. P2,P4) and verbosity (P1,P4 vs P2,P3). The effect of prompt choice with respect to average performance is shown in Fig. 3. First we calculate the accuracy for each model-dataset pair, averaged over all four prompts. Then we calculate the deviations of the accuracy for the particular prompts. The distribution of these deviations for all model-prompt-dataset combinations, grouped by prompt and corpus, is shown in the figure. We first note that, on average, classification with natural language answer format (P1,P3) works better than the abbreviated symbolic format (P2,P4). This is especially clear in the case of the UKP corpus. Furthermore, more elaborate formulation of the task (P2,P3) seem preferable to the shorter formulation (P1,P4). This certainly holds for the Args.me database. In this corpus, we note a marked instability for P4, which combined symbolic format with a brief task formulation. For the UKP corpus, the effect of prompt verbosity is not as clear: The verbose prompt P2 shows notably poor performance. This may be related to the different data structures between the corpora: Within Args.me, each record provides a pair of thesis and argument. In contrast, the UKP corpus consist only of arguments, grouped into datasets by discussion topic. The thesis is taken to be implied, since these topics are usually highly controversial issues, dividing debate participants clearly into two opposing camps. For the purpose of prompt 2, we attempted to reconstruct the theses which would best characterize these opposing standpoints (see Appendix B). In cases where the scope of the original debate was broader than the reconstructed thesis, this might have impaired the argument classification. Thus for UKP, higher accuracy is achieved when stating the thesis very roughly (e.g. being simply "for" or "against" abortion, minimum wage etc.) Variance between datasetsWe now calculate how model performance varies with different datasets. For each model, we take its average accuracy, and calculate the standard deviation based on performance in model-dataset pairs. We again perform calculations seperately on UKP and Args.me data. The Llama models have a notably higher variance for 8 A comprehensive study of LLM-based argument classification Figure 3: Impact on accuracy due to prompt choice (calculations exclude Llama 3.1 1B) the UKP datasets, while for Args.me, GPT and Deepseek performance varies slightly more. All in all however, for a given dataset the typical deviation from average is around 2 to 4 percentage points. We also note that some datasets lead to systematically poorer classification accuracy then others (cf. Fig. 11). This is particularly true for the debatewise dataset within Args.me. One obvious reason is variance in data and annotation quality. However, the differences could also stem from LLM bias, as discussed in Sec. 6.1. Sampling variance due to temperature In our study, we consequently set the temperature to 0.6, which is the recommended temperature for the Llama models. This introduces randomness into the LLM output. To estimate how strongly the classification varies due to this decoding strategy, we perform a variability study: We use the "trimmed" datasets to repeatedly perform inference on identical prompts. Due to cost restraints, the study is performed on models which we were able to run locally (i.e. all except GPT 5.2). For each record, prompt and model inference is performed three times. We then calculate the Fleiss kappa, which is a measure of agreement for raters in categorical ratings (Fleiss, 1971). It is a generalization of the popular Cohen kappa, which measures agreement between two raters. We note that intra-model agreement is especially strong for large, well-performing models. This is an intuitive result assuming that stronger models more often "know" the answer and less frequently resort to "guessing", as is the case with smaller models. The typicalÎşfor models larger than Llama 8b is above 0.8, which indicates very good agreement for these models. 5.2 Model performance We now turn to assess how the models in our study compare to each other in argument classification. For the accuracies in this section, we always report the standard error. Unless stated otherwise, for each model we first average accuracy across prompts within each dataset. We then compute the mean and standard error across datasets to obtain the values reported in the tables and figures. Zero-shot accuracy (RQ2)We begin by comparing the accuracy of the models in the absence of prompting strategies. Figures 6 and 7 illustrate the performance averaged across the four basic prompts and all datasets, relative to the random 9 A comprehensive study of LLM-based argument classification Figure 4: Standard deviation from model average depending on dataset Figure 5: Average Fleissâ kappa by model for UKP and Args.me. guessing baselines (33.3% for UKP and 50% for Args.me). As anticipated, performance declines significantly for the smallest models. While Llama 8B and 3B may still be utilized under severe computational constraints, the 1B modelâs performance is comparable to random guessing, rendering it ineffective for the task in a zero-shot capacity. Consequently, we exclude it from most subsequent analyses. (Note: For the Args.me corpus, Llama 1Bâs performance falls below the 50%). Only minuscule performance differences are observed between Llama 3.1 70b, 3.3 70b and Llama 4 Scout. As expected, the flagship OpenAI model GPT-5.2 â the only proprietary model in the study â shows the best performance. Second ranking is the large GPT oss-120b. ds70b, which is the Llama 3.3 70b model fine tuned with Deepseek R1, shows notable improvement over its base model. Perhaps most noteworthy is the performance of GPT oss-20b. With a modest parameter count, it significantly outperforms the much larger Llama 70b models. The authors believe this speaks to the potential of achieving argument mining performance on par with state-of-the-art general purpose models using aptly tuned, much smaller open-weight models like GPT oss-20b. Investigating this possibility represents a promising area for future research stemming from this work. 10 A comprehensive study of LLM-based argument classification Figure 6: UKP aggregate Accuracy by model Figure 7: Args.me aggregate Accuracy by model Chain-of-thought prompting (RQ3) As mentioned, explicit chain-of-thought prompting was performed only for the Llama models, since these are the only models in the studies which do not have an inbuilt "reasoning mode". The models were encouraged to perform step-by-step reasoning using a special prompt format, see Appendix B. The resulting Accuracy is compared to average zero-shot performance in Fig. 8. The efficacy of the CoT prompting method in this context is rather mixed. Improvements of a few percentage points were noted for the larger models as well as Llama 3b in case of the UKP corpus. On the other hand, performance worsened for all models in case of the Args.me corpus. Voting algorithms (RQ4)We now proceed to a discussion of the results obtained from the various voting algorithms introduced in Section 4.3. A comparison against the zero-shot baseline for each model is illustrated in Figure 9. In nearly all instances, the voting algorithms yield a statistically significant improvement over zero-shot prompting. The average percentage-point gains across different algorithms are detailed in Table 3. Overall, the Tiebreak method achieves the most substantial improvement, followed closely by Simple voting. While Weighted voting still outperforms average zero-shot inference, it underperforms relative to the other two methods within the UKP corpus. 11 A comprehensive study of LLM-based argument classification Figure 8: P1âP4 average vs P5 (CoT) accuracy for UKP and Args.me. Table 3: Average improvement in percentage points over baseline accuracy by voting algorithm UKPArgs.me Simple+4.4%+2.6% Tiebreak+4.5%+2.8% Weighted+2.4%+2.9% These results suggest that repeatedly querying a model with slight prompt variations allows to extract information than a single zero-shot trial. Consequently, reasonable aggregation methods are expected to enhance performanceâa principle that aligns with recent advancements in Mixture-of-Experts (MoE) architectures (Cai et al., 2025). Our findings also indicate that a modelâs self-assessment of certainty is of limited reliability. This is particularly evident in the poor performance of the weighted vote for Llama 3B on the UKP corpus; the model frequently assigned a certainty score of "0," implying a random guess despite actually outperforming the baseline. This is why more robust results are achieved using two-stage methods like "Simple" or "Tiebreak". However, even noisy indicators like self-assessed certainty appear to contain useful signal, as using these values for tie-breaking still yields an improvement over the random selection used in the Simple algorithm. Finally, the performance boost from voting is more pronounced for open-weight models than for the proprietary GPT-5.2. Notably, in the UKP corpus, this voting ensemble enables GPT oss-120B to surpass the state-of-the-art GPT-5.2. 12 A comprehensive study of LLM-based argument classification Figure 9: Voting algorithm comparison (simple, tiebreak, weighted) for UKP and Args.me. Figure 10: Three-prompt vs four-prompt voting by model. Table 4: Ablation study of the impact of presented prompts (UKP tiebreak accuracy). modelP2,3,4P1,3,4P1,2,4P1,2,3avgP1,2,3,4 Llama 1b30.747.247.632.539.542.2 Llama 3b59.860.960.462.760.961.7 Llama 8b70.370.568.972.970.671.4 Llama 3.1 70b74.173.874.475.174.375.0 Llama 3.3 70b75.174.474.774.774.775.6 Llama 4-Scout-70B78.177.176.378.877.678.7 DS 70b77.979.278.478.778.678.6 gpt-oss-20B79.379.379.379.779.479.7 gpt-oss-120b81.581.581.281.981.582.1 GPT-5.279.280.279.379.879.680.4 13 A comprehensive study of LLM-based argument classification Ablation study (RQ4) Given the performance variability observed across different prompts (cf. Sec. 5.1), we examine whether excluding lower-performing "voters" from the ensemble can further enhance accuracy. We investigate this through an ablation study, performing voting within the four possible three-voter subsetsâderived by removing one prompt at a timeâusing the UKP corpus. The Tiebreak algorithm identified in the previous section is employed to obtain these results. As shown in Figure 10, removing a voter generally leads to a degradation in average performance across models. Nevertheless, it remains to be seen whether removing a specific underperforming prompt might yield improvements. Table 4 presents the accuracy results for these configurations, with the top-performing three-prompt combination in each row highlighted in bold. The final two columns provide the mean accuracy across all three-voter groups and the original four-voter tiebreak result for reference. For Llama 3.1 (1B, 3B, 8B, 70B), Llama 4 Scout, and the DeepSeek model, at least one three-prompt combination outperforms the four-prompt configuration. Conversely, for Llama 3.3 70B, GPT oss-20B, GPT oss-120B, and GPT-5.2, the four-prompt ensemble remains the superior choice. This suggests that maximizing the voter pool is particularly advantageous for larger models. While the P1,P2,P3 combination most frequently achieves the highest accuracy among the subsets, every combination serves as the optimal choice for at least one model. Consequently, there is no universally superior sub-selection of prompts. In summary, the most robust performance gains are achieved with a larger number of voters. Table 5: Average model Accuracy in argument classification on the UKP and Args.me datasets UKP (mRaR)UKP (voting)UKP (CoT)Args.me (mRaR)Args.me (voting)Args.me (CoT) Llama-3.1-1B37.3Âą 0.542.2Âą 0.526.6Âą 0.541.6Âą 4.141.5Âą 4.141.0Âą 4.1 Llama-3.1-3B53.5Âą 1.161.7Âą 1.155.4Âą 1.158.8Âą 1.561.4Âą 1.555.8Âą 1.5 Llama-3.1-8B65.4Âą 1.471.4Âą 1.465.4Âą 1.469.6Âą 1.973.2Âą 1.965.2Âą 1.9 Llama-3.1-70B70.9Âą 1.875.0Âą 1.875.3Âą 1.880.0Âą 2.185.7Âą 2.175.8Âą 2.1 Llama-3.3-70B72.5Âą 1.875.6Âą 1.876.8Âą 1.880.8Âą 2.186.3Âą 2.176.6Âą 2.1 Llama-4-Scout-70B73.1Âą 1.478.7Âą 1.476.9Âą 1.479.6Âą 1.885.2Âą 1.874.2Âą 1.8 ds70b75.9Âą 1.278.6Âą 1.2â84.4Âą 2.687.0Âą 2.6â gpt-oss-20b76.1Âą 1.279.7Âą 1.2â86.2Âą 2.488.2Âą 2.4â gpt-oss-120b77.2Âą 0.682.1Âą 0.6â88.5Âą 2.889.6Âą 2.8â GPT-5.278.0Âą 1.180.4Âą 1.1â91.9Âą 2.492.4Âą 2.4â Table 6: Average model F1 in argument classification on the UKP and Args.me datasets UKP (mRaR)UKP (voting)UKP (CoT)Args.me (mRaR)Args.me (voting)Args.me (CoT) Llama-3.1-1B25.2Âą 1.435.7Âą 1.422.7Âą 1.427.8Âą 2.135.7Âą 2.130.9Âą 2.1 Llama-3.1-3B37.5Âą 2.058.1Âą 2.038.1Âą 2.037.9Âą 0.559.1Âą 0.538.1Âą 0.5 Llama-3.1-8B46.2Âą 1.467.2Âą 1.446.6Âą 1.445.4Âą 1.071.7Âą 1.044.0Âą 1.0 Llama-3.1-70B52.5Âą 1.574.1Âą 1.574.2Âą 1.552.9Âą 1.384.6Âą 1.350.4Âą 1.3 Llama-3.3-70B53.7Âą 1.474.7Âą 1.475.5Âą 1.453.5Âą 1.385.3Âą 1.376.4Âą 1.3 Llama-4-Scout-70B52.7Âą 3.876.0Âą 3.873.0Âą 3.852.6Âą 1.483.9Âą 1.449.4Âą 1.4 ds70b55.9Âą 1.877.0Âą 1.8â55.8Âą 1.586.1Âą 1.5â gpt-oss-20b54.8Âą 2.876.6Âą 2.8â57.0Âą 10.487.2Âą 10.4â gpt-oss-120b74.6Âą 1.179.8Âą 1.1â58.3Âą 11.688.8Âą 11.6â GPT-5.277.3Âą 1.179.3Âą 1.1â61.0Âą 11.991.9Âą 11.9â 14 A comprehensive study of LLM-based argument classification Figure 11: Heatmap visualizing the performance of the seven top-performing LLM models across debate topics in the UKP and Args.me datasets. Table 7: Accuracy comparison across different argument mining studies UKP (F1)Args.me (Acc) GPT-5.277.391.9 gpt-oss-120b74.688.5 gpt-oss-20b73.186.2 ds70b74.684.4 Llama 3.3 70b71.680.8 Llama 3.1 8b61.669.6 BERT (Stab et al., 2018a)57.785.3 LSTM (Stab et al., 2018a)42.85- Pietron et al., 202468.589.6 Akiki and Potthast, 2020-75.5 15 A comprehensive study of LLM-based argument classification Performance summary and comparison with other studies (RQ2-4) At the end of this section we present an overview over the performance achieved by the different models using mRaR (i.e. zero-shot averaged over four prompts), Tiebreak-voting and Chain of Thought. We report Accuracy (Tab. 5) and F1 scores (Tab. 6). Detailed results such as Accuracy, Recall, Precision, F1 and sample size for all dataset, model and prompt combinations are available in Appendix E. A visualization of model performance broken down by dataset and prompt is given in the heatmap Fig. 11. We also present a comparison to previous studies using the UKP and Args.me benchmarks 7. As could be expected, LLMs of the latest generation largely surpass older approaches. As a notable exception, the ensemble approach of Pietron et al., 2024, which featured a combined architecture of BERT and ChatGPT-4, achieved an accuracy on par with leading models of this study for the Args.me benchmark. 5.3 Error types in UKP dataset Figure 12: Error types in UKP by prompt. Percentages are given with respect to all predicted arguments, including correct ones. Despite achieving overall good performance in argument classification, the models still make errors. Interesting patterns emerge when we analyze the types of errors that occur. Figures 12 - 14 show the proportion of the most common error types out of all errors within prompts, datasets and models, following this classification: ⢠AF and AN refer to statements labeled by human annotators as against but incorrectly classified by the model as for or neutral, respectively. ⢠FA and FN refer to statements labeled as for but incorrectly classified as against or neutral. â˘NA and NF represent statements annotated as neutral but misclassified by the model as against or for, respectively. The most frequent type of error made by the models is classifying neutral utterancesâlabeled as such by annotatorsâ making them arguments (Fig. 12). This suggests that the promptâs directive to find arguments may lead the models to overinterpret content in search of argumentative structure. All prompts tend to produce NA and NF errors - that is, misclassifying neutral statements as arguments. Prompt P2, in particular, exhibits a significantly stronger tendency than the others to mislabel neutral utterances as supporting arguments (NF errors). In contrast, prompt P3 shows the most balanced distribution of error types, indicating no strong bias toward any specific classification error. Figure 13 presents an analysis of error types by debate topic in the UKP dataset, which reveals several noteworthy patterns. When it comes to the misclassification of posts labeled as neutral by human annotators, the topics most prone to being incorrectly classified as counter-arguments (NA errors) are death penalty and school uniforms. In contrast, the topics most susceptible to NF errors â where neutral statements are misclassified as supporting arguments â are 16 A comprehensive study of LLM-based argument classification Figure 13: Error types in UKP by dataset. Percentages are given with respect to all predicted arguments, including correct ones. Figure 14: Error types in UKP by model. Percentages are given with respect to all predicted arguments, including correct ones. marijuana and cloning. The topic of gun access stands out due to a notably high frequency of both FA and NA errors â where pro-access and neutral statements are misclassified as opposing arguments. This makes it the topic most prone to false negatives and suggests potential bias in the modelsâ training data. Among the leading models, GPT-5.2 and DS70B exhibit the strongest tilt towards NF and NA errors, meaning that they incorrectly classify neutral statements as arguments. In contrast, both gpt-oss models are significantly more balanced in terms of error types (see Fig. 14). 6 Qualitative analysis of misclassification patterns In addition to quantitative evaluation, we conducted a qualitative error analysis to examine the nature of misclassifications produced by the evaluated LLMs, with particular focus on DS70B, gpt-oss-120b, and GPT-5.2, as these models achieved the highest F1 scores on the UKP dataset. The analysis aimed to identify systematic failure modes and to better 17 A comprehensive study of LLM-based argument classification understand the linguistic and argumentative conditions under which errors occur. The results revealed a range of recurring error patterns associated with discourse complexity, domain-specific characteristics, and architectural differences between the models. The findings were categorized according to the identified linguistic and structural phenomena. The results of this analysis are presented below. 6.1 Impact of domain-specific context on error patterns (RQ5) Error distributions varied significantly across topics. The most extreme case was the death penalty topic, where 43.2% of all errors were NA errors. For example: The defects in death-penalty laws, conceded by the Supreme Court in the early 1970s, have not been appreciably altered by the shift from unrestrained discretion to âguided discretionâ. This statement describes procedural developments but was misclassified as opposing the claim. Similarly, in the marijuana topic, NF errors accounted for 33.00% of all errors. For example: For instance, cocaine has a medical purpose and can be prescribed by doctors as Erythroxylum coca, yet its true production and distribution are controlled by drug cartels and organized crime. Although this statement provides contextual information, the oss-120b model incorrectly interpreted it as supporting the claim. These findings indicate that models tend to systematically misinterpret descriptive or contextual statements as argumentative stance. The table below presents the dominant error type for each topic and its percentage share among all errors within that topic. Table 8: Dominant error type by topic and its percentage share TopicDominant error typePercentage of errors within topic Death penaltyNA43.2% Marijuana legalizationNF33.0% AbortionNA32.3% Gun access limitationNA32.3% CloningNA31.2% Minimum wageNF30.3% School uniformsNA29.4% Nuclear energyNA28.8% The Table 8 presents the dominant error type for each topic and its percentage share among all errors made by DS70B, gpt-oss-120b, and GPT-5.2 within that topic. The analysis reveals a systematic tendency of the models to implicitly associate arguments within specific topical domains with particular argumentative positions. For instance, the models tend to misclassify statements as opposing, in debates on the death penalty and abortion, and as supporting in discussions on marijuana legalization and minimum wage policies. This pattern suggests that the models may implicitly associate specific topics with dominant argumentative frames, potentially reflecting biases present in their training data, and consequently assign stance based on topic-related expectations rather than the actual semantic and pragmatic content of the statement. As certain topics appeared disproportionately associated with specific types of classification errors, we conducted an additional analysis using only the oss-120b model, which was hypothesized to exhibit topic-related bias. We examined whether, for particular topics, errors more frequently involved the misclassification of Against arguments (AF, AN errors) than For arguments (FA, FN errors), or vice versa. Once again, the most notable patterns emerged in datasets previously identified as particularly susceptible to model bias. For example, in the Death penalty dataset, oss-120b reversed the polarity of For arguments substantially more often (37.7% of all errors) than Against arguments (21.2% of all errors). Moreover, when misclassifications involved statements labeled as No argument, a large majority of them (66.7%) were interpreted by oss-120b as opposing the death penalty. These findings suggest the presence of systematic bias in the modelâs interpretation of argumentative content within this topic, leading to a disproportionate attribution of oppositional stance even when such intent was not explicitly expressed. 6.2 Structural and discourse-level complexity in argumentation (RQ5) Most misclassifications across all evaluated models resulted from failures to accurately map complex syntactic structures to the correct argumentative polarity. â˘Failure to interpret contrastive discourse structures. The most frequent failure mode across all evaluated models involved misinterpretation of contrastive discourse structures, accounting for app. 6,876 error cases 18 A comprehensive study of LLM-based argument classification from the analysed sample. Contrastive constructions, marked by discourse operators such as but, however, although, despite, yet, nevertheless, or even though, introduce a concessive clause followed by the main argumentative conclusion. In such structures, the clause following the contrastive marker typically expresses the dominant argumentative polarity, while the preceding clause provides background, qualification, or rhetorical concession. The evaluated models in some cases failed to correctly identify this hierarchical relationship. Instead, polarity was often assigned based on the first clause, while the conclusion expressed after the contrastive marker was ignored or underweighted. This resulted in systematic polarity inversion or neutralization errors, particularly in the NA (neutral misclassified as against; 2,310 cases) and NF (neutral misclassified as in favor; 2,032 cases) categories. Example from the gun access debate (oss120b, AN): We are strong supporters of the second amendment, but weâve got to do something to stop guns from getting into the wrong hands. The model incorrectly prioritized the concessive clause instead of the main conclusion supporting regulation. ⢠Misinterpretation of argument structure and multi-faceted reasoning. Another major source of errors involved incorrect interpretation of argument structure, particularly in statements containing inference indica- tors such as because, therefore, since, thus, consequently, and as a result. These markers signal inferential relationships between premises and conclusions and are essential for identifying argumentative polarity. Correct classification requires recognizing the direction of inference and distinguishing between premises, intermediate reasoning steps, and the final evaluative conclusion. However, the evaluated models frequently failed to map these inferential structures onto the correct polarity. In app. 1,291 cases, arguments containing inference indicators were misclassified. Models often relied on surface-level lexical cues, such as references to risks, constraints, or negative consequences, without correctly interpreting their inferential role. Example form the abortion debate: (ds70b, FA): Should we ban birth control altogether then since weâd be killing a âhumanâ? The marker since introduces a premise within a rhetorical question, forming a reductio-type argument. The model misclassified the argument due to failure to interpret the inferential structure â˘Loss of referential alignment. Another important failure mode involved loss of referential alignment between the argument and its target claim, accounting for app. 433 cases of complete polarity reversal, primarily AF (against misclassified as in favor) and FA (in favor misclassified as against) errors. These errors were strongly associated with arguments beginning with deictic expressions such as it, this, that, these, or those, which function as anaphoric markers linking the statement to a previously introduced claim or context. The evaluated models frequently failed to resolve these referential dependencies, interpreting statements in isolation rather than linking pronouns to their intended targets. This resulted in polarity inversion or neutralization and reveals a limitation in discourse-level reasoning and context integration. Example form the gun access limitation debate (ds70b, FA): Thatâs because pulling a gun out and shooting back in the chaos of a mass shooting just makes things worse. In this case, that refers to a previously stated claim concerning defensive gun use. The argument provides a justification opposing that claim. The model failed to correctly identify the referential target and assigned incorrect polarity. 6.3 Lexical features and pragmatic inference (RQ5) ⢠Literal adherence to facts. A total of 2,030 errors resulted from literal interpretation of empirical or statistical statements, which models classified as neutral (FN, AN), ignoring their argumentative intent. This failure mode disproportionately affected oss-120b (app. 1,222 cases), indicating a strong tendency to treat factual evidence as informational rather than argumentative. Example (oss120b, AN): A 2003 study by researchers at Arizona State University found that " students from schools without uniforms reported higher self-perception scores than students from schools with uniform policies. Although this statement implicitly criticizes uniform policies, the model interpreted it as neutral factual information. â˘Incorporating external discourse context. In app. 1,146 cases, models incorporated context that was not present in the analyzed content but is often found in public debate. Example (DS70B, FN): DNA cloning has been used in genetic engineering to create plants that offer better nutritional value. Here, the model failed to recognize that the statement refers to a specific application of cloning and implicitly supports its permissibility in general. Instead, it incorrectly concluded that the statement does not directly address the broader ethical or social implications of cloning humans or animals and therefore classified it as not an argument. â˘Implicit criticism, rhetorical questions, and counterfactual reasoning. Models frequently failed to detect implicit evaluative intent, including rhetorical questions and counterfactual constructions. This included app. 813 cases of undetected implicit criticism and app. 1,392 cases involving counterfactual or ironic reasoning. Example (oss-120b, AN): If the students do nât like them, why should we force them to wear these garments anyway?. This rhetorical question implicitly opposes uniform policies but was misclassified due to failure to recognize implicit argumentative intent. 19 A comprehensive study of LLM-based argument classification â˘Attributing meaning beyond textual evidence. In app. 685 cases, models attributed argumentative intent that was not supported by the original text, leading to polarity reversals. Example (oss-120b, AF): Some abortions happen because of societal pressures. This statement implicitly criticizes abortion by emphasizing external coercion, yet oss-120b classified it as an argument in favor of abortion. This suggests that the model relied on generalized discourse associations rather than the actual evaluative direction expressed in the statement. â˘Negation and emotional language. Negation and emotionally charged language also disrupted classification. In app. 154 cases, sentences containing multiple negations caused polarity misclassification, particularly in oss120b and ds70b. Additionally, 617 cases involved emotionally expressive informal language that models failed to interpret correctly. Example (ds70b, AN): "The fact that some states or countries which do not use the death penalty have lower murder rates than jurisdictions which do is not evidence of the failure of deterrence" Here, multiple negations and emotional emphasis interfered with polarity detection, therefore the model is overlooking the final negation present in the argument, concluding that this statement is evidence of the failure of deterrence. These findings demonstrate that pragmatic inference represents a major limitation in current LLM-based argument mining. Models correctly interpret literal semantic content but sometimes fail to infer argumentative intent, particularly when arguments rely on empirical evidence, rhetorical structure, implicit criticism, emotional emphasis, or contextual reference. 6.4 Model-Specific Differences in Error Patterns (RQ5) Although all evaluated models exhibited similar general limitations related to discourse complexity and pragmatic inference, some differences emerged in their susceptibility to specific linguistic and argumentative phenomena. The observed phenomena are presented in Table 9. Overall, these findings indicate a clear trade-off between pragmatic sensitivity and structural reasoning. The gpt-oss120b model struggles primarily with pragmatic inference and implicit intent, GPT-5.2 demonstrates strong pragmatic understanding but remains vulnerable to structural complexity, and ds70b occupies an intermediate position, with particular sensitivity to emotional language. Table 9: Model-specific characteristics in argument misclassification ModelMajor misclassification categoriesModel-specific characteristics oss120b ⢠Contrastive discourse structures ⢠Literal interpretation of facts ⢠Complex argumentation ⢠Implicit criticism More vulnerable to literal interpretation and failure to rec- ognize argumentative intent. This model more frequently misclassifies statements containing numbers, statistics, and research findings as purely neutral or informational content. It performs worse in detecting irony, rhetorical questions, and implicit criticism. gpt-5.2 ⢠Contrastive discourse structures ⢠Complex argumentation ⢠Misinterpretation of argument structure High resistance to literal interpretation, but occasional difficulty in processing argument structure. This model performs best in inferring argumentative intent from statis- tical evidence and rarely misclassifies factual arguments. However, its main limitation involves multi-faceted and contrastive sentences, where it may fail to identify the main conclusion. It may occasionally overinterpret logical structure markers. ds70b ⢠Contrastive discourse structures ⢠Literal interpretation of facts ⢠Complex argumentation ⢠Incorporation of external context ⢠Emotional language misinterpreted as ar- gument More susceptible to emotional and informal language. ds70b demonstrates better recognition of argumentative intent in statistical statements than gpt-oss120b, but per- forms worse than GPT-5.2. It may misinterpret emotional expressions as indicative of argumentative stance and ex- hibits difficulties in correctly handling references to exter- nal discourse context. 20 A comprehensive study of LLM-based argument classification Figure 15: Gold re-annotation sample breakdown (n=442). 6.5 Errors made by LLMs and borderline cases (RQ6) The accuracy of the models was assessed on the basis of discrepancies between their evaluations and those produced by the annotators involved in creating the datasets used in this research. In other words, the values presented in the tables represent the degree of agreement between the modelsâ judgements and those of the annotators. It should be noted, however, that the test datasets were constructed from real instances of argumentative exchange and, as a consequence, necessarily include borderline cases that do not lend themselves to straightforward classification. This raises the question of the extent to which the calculated inaccuracy rates of the models result from disagreements concerning the annotation of precisely such cases. A comprehensive answer to this question would require a meticulous analysis of the datasets employed, which lies well beyond the scope of the present study. We therefore conducted only a limited pilot investigation of this issue. Its results cannot be regarded as statistically significant and therefore warrant only a single, modest conclusion: namely, that the quantitative results reported in this paper may not fully reflect the modelsâ actual performance with respect to the assigned task. Our preliminary investigation focused exclusively on records from the UKP abortion dataset for which, using prompt 3, the best-performing model (GPT-5.2) produced 442 responses that diverged from those of the annotators (Fig. 15). This particular datasetâprompt combination was selected because GPT-5.2âs results for it were closest to the modelâs average performance on the UKP collection (78.0%). An examination of all these cases revealed that a substantial proportion - 205 instances (46.4%) - consisted of borderline expressions that could plausibly be classified differently from the way they were originally annotated. Our annotator not only identified the problematic cases but also independently re-annotated all 442 instances. It transpired that 90 out of the 205 borderline cases (43.9%) were annotated differently from the original UKP labels. Of these 90 cases, 83 (92.2%) were classified in accordance with the modelâs output. By contrast, among the 237 cases not identified as borderline, only 6 (2.5%) were annotated differently from the UKP dataset; in 4 of these cases, our classification again coincided with that of the model. It should be emphasised that the vast majority of divergently classified recordsâ355 cases (80.3%)âwere nevertheless judged to constitute genuine model inaccuracies. However, if our limited observations were to be confirmed by statistically robust studies, they would suggest that the actual accuracy of the models may be somewhat higher than indicated by our current results. Any further investigation would also need to consider records for which the modelsâ outputs were deemed âcorrect,â since annotation errors in such cases would in fact increase the number of genuine model inaccuracies. One may conjecture that within cases of agreement there are significantly fewer borderline expressions and hence fewer annotation errors (the low proportion of disagreement among non-borderline cases appears to support 21 A comprehensive study of LLM-based argument classification this assumption). At present, however, this conjecture lacks sufficient empirical support and cannot serve as a reliable basis for definitive conclusions. 7 Conclusions and future work The findings of this study suggest that argument classification represents a particularly sensitive benchmark for evaluating the reasoning capabilities of large language models. Our results also highlight the importance of complementing standard quantitative evaluation with systematic qualitative error analysis. While aggregate accuracy and F1 scores suggest substantial progress in argument classification, qualitative inspection reveals persistent structural limitations shared across models, including failures in referential resolution, contrastive reasoning, and pragmatic inference. These findings suggest that future benchmarks should incorporate evaluation protocols specifically designed to test discourse-level reasoning, rather than relying solely on aggregate classification performance. In particular, the inclusion of adversarial examples, contrastive constructions, counterfactual statements and pragmatically implicit arguments may provide a more accurate assessment of true reasoning capability. An important practical implication of this study is that a substantially smaller open-weight model, such as gpt-oss-120b, can perform only marginally worse than the flagship proprietary model GPT-5.2 in argument mining tasks. When enhanced with structured prompting and certainty-based voting strategies, the performance gap narrows further, in some configurations approaching parity. This result demonstrates that near state-of-the-art argument classification does not necessarily require the largest or proprietary architectures. From a computational perspective, this finding is particularly relevant: achieving comparable performance with a smaller open model translates into significantly lower infrastructure demands, reduced server load, and improved cost-efficiency. Consequently, high-quality argument mining systems can be developed in a more accessible, scalable, and reproducible manner, without exclusive dependence on closed commercial models. These observations are further supported by our pilot re-annotation study conducted on 442 disagreement cases in the UKP abortion dataset. Nearly half of these instances (46.4%) were identified as borderline expressions, and in 43.9% of those cases the re-annotation differed from the original labelsâmost often aligning with the modelâs prediction. This indicates that a non-negligible proportion of reported âmodel errorsâ may in fact reflect annotation ambiguity rather than genuine reasoning failure. Consequently, improving dataset quality should not only involve stricter annotation procedures but also clearer formalization of borderline categories, including explicit guidelines on the treatment of implicit arguments, rhetorical questions, descriptive-statistical statements, and context-dependent evaluative claims. Transparent publication of annotation policies and decision criteria would enhance reproducibility, allow fairer model comparison, and provide a more reliable benchmark for evaluating discourse-level reasoning in LLMs. The limitations identified in this study have important implications for the deployment of LLM-based argument mining systems in real-world contexts, such as public policy analysis, legal reasoning, and social media monitoring. In particular, the tendency of models to infer argumentative intent based on topic associations rather than textual evidence raises concerns about potential bias amplification and misrepresentation of viewpoints. This suggests that fully automated argument classification systems should be used cautiously in high-stakes contexts and should ideally be complemented by human oversight or hybrid humanâAI annotation workflows. The observed failure modes indicate that further progress in argument mining will likely require architectural and training advances specifically targeting discourse-level reasoning. In particular, improvements may be achieved through training regimes that explicitly incorporate discourse structure, argumentation theory, and pragmatic inference, rather than relying solely on next-token prediction objectives. Overall, these results confirm that while modern LLMs represent a major advancement in automated argument mining, their limitations in pragmatic reasoning and discourse-level interpretation remain a fundamental challenge. Addressing these shortcomings will require improvements not only in model architecture but also in training data quality, annotation frameworks, and reasoning-oriented training methods. Based on our results, future work might focus on developing more sophisticated prompt engineering-based algorithms that improve argument classification. The basic binary and three-label argument classification might be extended to more complex argument mining tasks. Another plausible next step would be to adapt the RAG technique and fine tuning of models such as gpt-oss-20b for accurate and highly efficient argument mining. References Ajjour, Y., Wachsmuth, H., Kiesel, J., Potthast, M., Hagen, M., & Stein, B. (2019). Data acquisition for argument search: The args.me corpus. In C. BenzmĂźller & H. Stuckenschmidt (Eds.), Ki 2019: Advances in artificial intelligence (Vol. 11793). Springer, Cham. https://doi.org/10.1007/978-3-030-30179-8_4 22 A comprehensive study of LLM-based argument classification Akiki, C., & Potthast, M. (2020). Exploring argument retrieval with transformers. CLEF. https://ceur-ws.org/Vol-2696 Bar-Haim, R., Bhattacharya, I., Dinuzzo, F., Saha, A., & Slonim, N. (2017, April). Stance classification of context- dependent claims. In M. Lapata, P. Blunsom, & A. Koller (Eds.), Proceedings of the 15th conference of the European chapter of the association for computational linguistics: Volume 1, long papers (p. 251â261). Association for Computational Linguistics. https://aclanthology.org/E17-1024/ Besta, M., Barth, J., Schreiber, E., Kubicek, A., Catarino, A., Gerstenberger, R., Nyczyk, P., Iff, P., Li, Y., Houliston, S., Sternal, T., Copik, M., Kwa Ě sniewski, G., MĂźller, J., Flis, Ĺ., Eberhard, H., Chen, Z., Niewiadomski, H., & Hoefler, T. (2025). Reasoning language models: A blueprint. https://doi.org/10.48550/ARXIV.2501.11223 Boltuzic, F., & Snajder, J. (2014). Back up your stance: Recognizing arguments in online discussions. Proceedings of the First Workshop on Argumentation Mining, 49â58. Cabrio, E., & Villata, S. (2018). Five years of argument mining: A data-driven analysis. IJCAI, 5427â5433. Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025). A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 1â20. https://doi.org/10.1109/tkde.2025. 3554028 Carletta, J. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2), 249â254. Chakrabarty, T., Hidey, C., Muresan, S., McKeown, K., & Hwang, A. (2019). Ampersand: Argument mining for persuasive online discussions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2933â2943. Chen, T. (2021). Bert argues: How attention informs argument mining (Publication No. 1589) [Honors Theses]. University of Richmond. https://scholarship.richmond.edu/honors-theses/1589 Daxenberger, J., Eger, S., Habernal, I., Stab, C., & Gurevych, I. (2017). What is the essence of a claim? cross- domain claim identification. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2055â2066. Daxenberger, J., Schiller, B., Stahlhut, C., et al. (2020). Argumentext: Argument classification and clustering in a generalized search scenario [Issue Date: July 2020]. Datenbank Spektrum, 20, 115â121. https://doi.org/10. 1007/s13222-020-00347-7 Deng, Y., Zhang, W., Chen, Z., & Gu, Q. (2023). Rephrase and respond: Let large language models ask better questions for themselves. arXiv. https://doi.org/10.48550/arXiv.2311.04205 Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, June). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (p. 4171â4186). Association for Computational Linguistics. https://doi.org/ 10.18653/v1/N19-1423 Dusmanu, M., Cabrio, E., & Villata, S. (2017, September). Argument mining on Twitter: Arguments, facts and sources. In M. Palmer, R. Hwa, & S. Riedel (Eds.), Proceedings of the 2017 conference on empirical methods in natural language processing (p. 2317â2322). Association for Computational Linguistics. https: //doi.org/10.18653/v1/D17-1245 Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378â382. https://doi.org/10.1037/h0031619 Galassi, A., Lippi, M., & Torroni, P. (2018). Argumentative link prediction using residual networks and multi-objective learning. Proceedings of the 5th Workshop on Argument Mining, 1â10. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., & Zhu, Q. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. https://arxiv.org/abs/2501.12948. Lawrence, J., & Reed, C. (2020). Argument Mining: A Survey. Computational Linguistics, 45(4), 765â818. https: //doi.org/10.1162/coli_a_00364 Le Scao, T., Fan, A., Akiki, C., & Pavlick, E. (2023). Bloom: A 176b-parameter open-access multilingual language model. https://arxiv.org/abs/2211.05100. Li, K. (2020). An empirical study of transformer-based neural language model adaptation. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7934â7938. https: //doi.org/10.1109/ICASSP40776.2020.9053399 Lippi, M., & Torroni, P. (2016). Argumentation mining: State of the art and emerging trends. ACM Trans. Internet Technol., 16(2), 10:1â10:25. https://doi.org/10.1145/2850417 Niculae, V., Park, J., & Cardie, C. (2017). Argument mining with structured svms and rnns. arXiv preprint arXiv:1704.06869. Park, J., & Cardie, C. (2018, May). A corpus of eRulemaking user comments for measuring evaluability of arguments. In N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, & T. Tokunaga (Eds.), Proceedings of the eleventh international 23 A comprehensive study of LLM-based argument classification conference on language resources and evaluation (LREC 2018). European Language Resources Association (ELRA). https://aclanthology.org/L18-1257 Pietron, M., Olszowski, R., & GomuĹka, J. (2024). Efficient argument classification with compact language models and chatgpt-4 refinements. In N. e. a. Nguyen (Ed.), Computational collective intelligence. iccci 2024 (Vol. 14810). Springer, Cham. https://doi.org/10.1007/978-3-031-70816-9_20 Reimers, N., Schiller, B., Beck, T., Daxenberger, J., Stab, C., & Gurevych, I. (2019). Classification and clustering of arguments with contextualized word embeddings. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 567â578. Ruiz-Dolz, R., Heras, S., Alemany, J., & GarcĂa-Fornes, A. (2020). Transformer-based models for automatic identifica- tion of argument relations: A cross-domain evaluation. arXiv. https://doi.org/10.48550/arXiv.2011.13187 Schaefer, R., & Stede, M. (2022). Gercct: An annotated corpus for mining arguments in german tweets on climate change. Proceedings of the Thirteenth Language Resources and Evaluation Conference, 6121â6130. Stab, C., Miller, T., Schiller, B., Rai, P., & Gurevych, I. (2018a). Cross-topic argument mining from heterogeneous sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3664â3674. Stab, C., Miller, T., Schiller, B., Rai, P., & Gurevych, I. (2018b). Cross-topic argument mining from heterogeneous sources. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3664â 3674. https://doi.org/10.18653/v1/d18-1402 Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, E., A. Grave, & Lample, G. (2023). Llama: Open and efficient foundation language models. https://arxiv.org/abs/2302.13971. Van der Meer, M., Liscio, E., Jonker, C. M., Plaat, A., Vossen, P., & Murukannaiah, P. K. (2024). A hybrid intelligence method for argument mining. Journal of Artificial Intelligence Research, 80, 1187â1222. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ĺ., & Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.c/paper_files/paper/2017/file/3f5e243547dee91fbd053c1c4a845a-Paper.pdf Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022). Chain- of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Advances in neural information processing systems (p. 24824â 24837, Vol. 35). Curran Associates, Inc. https : / / proceedings . neurips . c / paper _ files / paper / 2022 / file / 9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Zhou, Y., Geng, X., Shen, T., Tao, C., Long, G., Lou, J.-g., & Shen, J. (2023). Thread of thought: Unraveling chaotic contexts. arXiv. https://doi.org/10.48550/arXiv.2311.08734 A Models used in the study ModelDeveloperRelease dateHuggingface codeLicenseCode Llama 3.2 1BMetaSep 2024meta-llama/Llama-3.2-1B-Instructopen1b Llama 3.2 3BMetaSep 2024meta-llama/Llama-3.2-3B-Instructopen3b Llama 3.1 8BMetaJul 2024meta-llama/Llama-3.1-8B-Instructopen8b Llama 3.1 70BMetaJul 2024meta-llama/Llama-3.1-70B-Instructopen31_70b Llama 3.3 70BMetaDec 2024meta-llama/Llama-3.3-70B-Instructopen33_70b Llama 4 ScoutMetaApr 2025meta-llama/Llama-4-Scout-17B-16E-Instructopenscout DeepSeek R1 Dist. 70BDeepSeekJan 2025deepseek-ai/DeepSeek-R1-Distill-Llama-70Bopends70b gpt-oss-20BOpenAIAug 2025openai/gpt-oss-20bopenoss20b gpt-oss-120bOpenAIAug 2025openai/gpt-oss-120bopenoss120b GPT-5.2OpenAIDec 2025âproprietarygpt-5.2 The models in this study are summarized in the table below. The last column represents the code which was used in this paper to refer to the models within figures and tables. B Prompts We used several distinct prompts to query the models, varying in complexity and expected output format. The prompts differed slightly between the UKP and Args.me datasets due to the different nature of the corpora (UKP includes a "No 24 A comprehensive study of LLM-based argument classification Argument" class, whereas Args.me is binary). In addition, for each prompt 1-4, a version with certainty estimation was also used. In each case, we used the standard system prompt (if provided) and chat template for the given model. UKP All prompts for the UKP dataset operate on a sentence and a thesis. No.Text 1Is the sentence: "sentence" an argument for or againstthesis, or is it no argument? Return one of the expressions: âForâ, âAgainstâ or âNo argumentâ, without any additional commentary. 2 The thesis is: "thesis" Indicate if the argument "sentence" is for this thesis (F), against this thesis (A), or neutral (N). Please respond with only one letter: F, A, or N, without any additional commentary. 3In the context of the ongoing public debate, evaluate whether the text "sentence" represents an argument supporting or opposing "thesis", or whether it does not qualify as an argument at all. Respond with one of the expressions: âForâ, âAgainstâ or âNo Argumentâ. 4Is the sentence: "sentence" an argument for (F) or against (A)thesis, or is it no argument (N)? Return a single letter: F, A, or N, without any additional commentary. Below are the thesis variants for the eight UKP datasets. For Prompt 2 we used the elaborate "long" formulation, for Prompts 1, 3 and 4 the "short" formulations. DatasetShort thesisLong thesis AbortionabortionAbortion should be fully accessible. CloningcloningCloning should be allowed. Death penaltydeath penaltyThe death penalty should be allowed. Marijuanalegalisation of marijuanaMarijuana should be legal. Gun lawsstricter gun lawsGun access should be limited. Minimum wageminimum wageThe minimum wage is justified and should be increased. Nuclear energynuclear energyNuclear energy should be developed. School uniformsschool uniformsSchool uniforms should be the standard in education. Args.me For the Args.me dataset, the "No Argument" option was removed. No.Text 1Is the sentence: "sentence" an argument for or against "thesis"? Return one of the expressions: "For" or "Against", without any additional commentary. 2 The thesis is: "thesis" Indicate if the argument "sentence" is for this thesis (F) or against this thesis (A). Please respond with only one letter: F or A, without any additional commentary. 3In the context of the ongoing public debate, evaluate whether the text "sentence" represents an argument supporting or opposing "thesis". Respond with one of the expressions: "For" or "Against". 4Is the sentence: "sentence" an argument for (F) or against (A)thesis? Return a single letter: F or A, without any additional commentary. Certainty For the certainty self-rating, the model was first queried with one of the above prompts (1-4). Then, the following follow-up prompt was appended to the conversation history: "Return the certainty of your answer as a percentage. Output only a single number between 0 and 100, with no additional text." 25 A comprehensive study of LLM-based argument classification Chain of Thought To evaluate reasoning capabilities of the Llama models, we used a Chain of Thought (CoT) prompt. This prompt explicitly instructs the model to think step-by-step. CorpusCoT Prompt UKPIs the sentence: "sentence" an argument for or against thesis or is it no argument? Solve the argument classification problem. Think through the problem step by step to solve it. Then output one final line exactly in the format: FINAL: <For|Against|No Argument> Do not include anything after the FINAL line. Args.meIs the sentence: "sentence" an argument for or against thesis? Solve the argument classification problem. Think through the problem step by step to solve it. Then output one final line exactly in the format: FINAL: <For|Against> Do not include anything after the FINAL line. C Experimental Setup and Hyperparameters All local model inferences (Llama models, DeepSeek 70B, gpt-oss 20b, and gpt-oss 120b) were conducted on the Athena supercomputer at the Academic Computer Centre Cyfronet AGH. The computations were performed on nodes equipped with 8Ă NVIDIA A100 GPUs (40GB VRAM each). Local models were deployed using the vLLM library. We utilized the default vLLM sampling parameters, with the exception of temperature, which was set to0.6for all models. The maximum number of new tokens generated was set to 4096. Other hyperparameters such as reasoning effort for the GPT models were also left to the default values. Inference for GPT-5.2 was performed via cloud inference using the OpenRouter API. The total expenditure for cloud inference amounted to USD 300 and was funded by the Department of Humanities, AGH University of Science and Technology. D Answer Parsing To extract the final classification from the model outputs, we applied regular expressions (Regex) designed to capture the expected label formats while ignoring extraneous text (e.g., "The answer is..."). The regex patterns generally looked for the keywords at the beginning or end of the string, or emphasized within the text (e.g., inside quotes or asterisks). Below are the regex patterns used for parsing the cleaned model outputs (whitespace and punctuation removed, converted to lowercase): CorpusPrompt No.Pattern UKP1, 3 râ (for|against|no argument) | (for|against|no argument) $â UKP2, 4 râ (f|a|n) | (f|a|n) $â Args.me1, 3 râ (for\|against) | (for\|against) $â Args.me2, 4 râ (f|a) | (f|a) $â If a direct match was not found using the patterns above, a fallback search was performed to find emphasized keywords (e.g., inside quotes or asterisks): r"(?:â|\"|\*\*)( *)(for|against|no argument)( *)(?:â|\"|\*\*)" The certainty value was obtained by searching for the patternrâ +â, i.e. extracting the first number found in the output. 26 A comprehensive study of LLM-based argument classification E Detailed Results by Prompt We present the detailed results for each run. nSample size AccAccuracy PrecPrecision RecRecall F1F1 score (macro average) Inv. ans.Invalid answer: answers which were empty (mostly API issues) or could not be parsed according to the rules in App. D Inv. cert.Invalid certainty (same as above) CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. Argsme1bdebatepedia12119325.755.150.120.725224 Argsme1bdebatepedia22119341.553.153.341.5217259 Argsme1bdebatepedia32119324.552.251.531.14785137 Argsme1bdebatepedia42119350.254.055.148.7513266 Argsme1bdebatepediaCoT200043.055.757.252.3410- Argsme1bdebatewise11375141.156.150.229.840120 Argsme1bdebatewise21375147.350.250.247.0110142 Argsme1bdebatewise31375135.650.650.339.02576139 Argsme1bdebatewise41375150.149.849.849.585379 Argsme1bdebatewiseCoT200039.252.152.251.6489- Argsme1bidebate11304850.554.750.235.081102 Argsme1bidebate21304851.151.050.949.409669 Argsme1bidebate31304846.055.553.248.01845132 Argsme1bidebate41304851.251.451.451.4466435 Argsme1bidebateCoT200040.756.056.055.9545- Argsme3bdebatepedia12119351.063.164.450.90246 Argsme3bdebatepedia22119365.963.167.162.4238 Argsme3bdebatepedia32119365.359.661.559.8227216 Argsme3bdebatepedia42119362.451.751.751.7054 Argsme3bdebatepediaCoT200055.863.267.458.6149- Argsme3bdebatewise11375152.258.056.551.20300 Argsme3bdebatewise21375159.157.156.756.71384 Argsme3bdebatewise31375156.255.655.255.1383234 Argsme3bdebatewise41375156.552.651.950.52106 Argsme3bdebatewiseCoT200055.259.059.358.4114- Argsme3bidebate11304861.862.661.761.10227 Argsme3bidebate21304861.263.061.560.1746 Argsme3bidebate31304856.158.557.255.220589 Argsme3bidebate41304852.454.052.848.6072 Argsme3bidebateCoT200056.259.159.159.198- Argsme8bdebatepedia12119374.271.778.271.6017 Argsme8bdebatepedia22119373.166.969.867.7138 Argsme8bdebatepedia32119369.570.677.167.93117 Argsme8bdebatepedia42119371.563.765.064.224 Argsme8bdebatepediaCoT200068.270.777.168.972- Argsme8bdebatewise11375166.666.567.166.3152 Argsme8bdebatewise21375169.668.667.868.01915 Argsme8bdebatewise31375166.469.369.467.3183124 Argsme8bdebatewise41375159.556.955.755.223 Argsme8bdebatewiseCoT200062.965.065.564.658- Argsme8bidebate11304873.173.273.173.1032 Argsme8bidebate21304872.274.072.471.801 Argsme8bidebate31304874.975.675.074.93239 27 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. Argsme8bidebate41304859.162.559.456.501 Argsme8bidebateCoT200064.567.066.966.972- Argsme31_70bdebatepedia12119388.383.989.185.830 Argsme31_70bdebatepedia22119390.286.688.687.531 Argsme31_70bdebatepedia32119390.586.989.087.9102 Argsme31_70bdebatepedia42119358.763.166.757.602 Argsme31_70bdebatepediaCoT200077.275.082.675.10- Argsme31_70bdebatewise11375178.978.679.578.7120 Argsme31_70bdebatewise21375185.284.785.484.91215 Argsme31_70bdebatewise31375179.179.579.379.417591 Argsme31_70bdebatewise41375158.560.860.758.511 Argsme31_70bdebatewiseCoT200073.773.974.773.62- Argsme31_70bidebate11304886.886.886.886.810 Argsme31_70bidebate21304891.191.191.191.100 Argsme31_70bidebate31304885.686.386.085.94817 Argsme31_70bidebate41304863.563.863.563.300 Argsme31_70bidebateCoT200076.476.976.376.20- Argsme33_70bdebatepedia12119388.283.889.485.800 Argsme33_70bdebatepedia22119389.885.988.787.100 Argsme33_70bdebatepedia32119390.987.489.788.410 Argsme33_70bdebatepedia42119359.264.968.858.400 Argsme33_70bdebatepediaCoT200078.275.683.276.00- Argsme33_70bdebatewise11375179.679.880.879.500 Argsme33_70bdebatewise21375186.185.686.585.810 Argsme33_70bdebatewise31375180.780.180.380.2210 Argsme33_70bdebatewise41375160.064.263.359.800 Argsme33_70bdebatewiseCoT200073.974.675.373.80- Argsme33_70bidebate11304887.587.787.587.510 Argsme33_70bidebate21304891.691.791.691.600 Argsme33_70bidebate31304886.987.287.086.940 Argsme33_70bidebate41304867.268.367.166.600 Argsme33_70bidebateCoT200077.678.577.577.40- Argsmescoutdebatepedia12119383.978.883.180.430 Argsmescoutdebatepedia22119382.877.580.878.810 Argsmescoutdebatepedia32119392.289.190.989.900 Argsmescoutdebatepedia42119361.160.463.758.300 ArgsmescoutdebatepediaCoT200073.772.880.071.71- Argsmescoutdebatewise11375177.377.478.377.190 Argsmescoutdebatewise21375182.682.283.082.370 Argsmescoutdebatewise31375182.682.182.282.180 Argsmescoutdebatewise41375163.165.765.563.110 ArgsmescoutdebatewiseCoT200072.073.173.772.15- Argsmescoutidebate11304886.086.086.086.000 Argsmescoutidebate21304888.788.888.788.701 Argsmescoutidebate31304888.588.788.688.530 Argsmescoutidebate41304866.766.966.766.600 ArgsmescoutidebateCoT200076.977.576.876.70- Argsmeds70bdebatepedia1200090.186.090.587.801 Argsmeds70bdebatepedia2200092.289.091.190.004 Argsmeds70bdebatepedia3200092.689.591.990.601 Argsmeds70bdebatepedia4200080.676.984.178.116 Argsmeds70bdebatewise1200079.478.778.978.801 Argsmeds70bdebatewise2200084.784.483.884.1010 Argsmeds70bdebatewise3200081.581.180.480.711 Argsmeds70bdebatewise4200073.572.973.673.0016 Argsmeds70bidebate1200085.585.585.585.500 Argsmeds70bidebate2200089.890.089.989.9116 28 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. Argsmeds70bidebate3200085.986.185.985.900 Argsmeds70bidebate4200076.776.776.776.7112 Argsmeoss20bdebatepedia1200089.585.290.387.10164 Argsmeoss20bdebatepedia2200090.787.688.588.02247 Argsmeoss20bdebatepedia3200091.087.291.288.805 Argsmeoss20bdebatepedia4200088.283.888.085.5179 Argsmeoss20bdebatewise1200080.980.280.680.40112 Argsmeoss20bdebatewise2200084.484.383.283.61197 Argsmeoss20bdebatewise3200082.481.981.781.803 Argsmeoss20bdebatewise4200078.878.278.078.1164 Argsmeoss20bidebate1200086.786.786.786.70100 Argsmeoss20bidebate2200089.489.789.589.40206 Argsmeoss20bidebate3200087.988.188.087.902 Argsmeoss20bidebate4200084.484.684.484.4056 Argsmeoss120bdebatepedia1200092.589.392.090.500 Argsmeoss120bdebatepedia2200092.890.890.390.503 Argsmeoss120bdebatepedia32119393.290.392.591.300 Argsmeoss120bdebatepedia42119391.187.391.088.8055 Argsmeoss120bdebatewise1200082.782.282.082.102 Argsmeoss120bdebatewise2200086.786.885.586.000 Argsmeoss120bdebatewise31375183.182.582.582.500 Argsmeoss120bdebatewise41375181.580.980.980.9115 Argsmeoss120bidebate1200089.389.489.489.300 Argsmeoss120bidebate2200091.491.791.591.400 Argsmeoss120bidebate3200090.090.190.089.900 Argsmeoss120bidebate41304888.588.788.588.5011 Argsmegpt-5.2debatepedia1200096.294.395.795.002 Argsmegpt-5.2debatepedia2200094.893.392.993.100 Argsmegpt-5.2debatepedia3200096.494.795.895.300 Argsmegpt-5.2debatepedia4199895.192.795.093.840 Argsmegpt-5.2debatewise1199986.786.386.186.224 Argsmegpt-5.2debatewise2199689.389.688.488.9100 Argsmegpt-5.2debatewise3199887.987.587.487.542 Argsmegpt-5.2debatewise4199985.985.585.485.522 Argsmegpt-5.2idebate1199892.692.792.792.640 Argsmegpt-5.2idebate2200093.593.793.693.500 Argsmegpt-5.2idebate3200093.293.393.393.202 Argsmegpt-5.2idebate4200091.691.791.691.602 UKP1babortion1392941.132.634.330.1785 UKP1babortion2392925.537.534.225.600 UKP1babortion3392919.444.533.912.780 UKP1babortion4392952.431.633.731.500 UKP1babortionCoT200025.036.634.931.3510- UKP1bcloning1303947.144.836.031.100 UKP1bcloning2303931.839.337.530.300 UKP1bcloning3303923.741.933.513.610 UKP1bcloning4303944.931.533.428.600 UKP1bcloningCoT200029.540.141.037.3416- UKP1bdeath1365153.633.334.030.600 UKP1bdeath2365130.636.437.229.300 UKP1bdeath3365113.431.233.08.860 UKP1bdeath4365152.233.334.332.100 UKP1bdeathCoT200025.736.937.432.6491- UKP1bgun1334148.941.037.032.300 UKP1bgun2334124.040.133.923.700 UKP1bgun3334125.631.034.115.700 UKP1bgun4334148.635.035.831.600 29 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. UKP1bgunCoT200024.237.637.132.7532- UKP1bmarijuana1247550.139.635.730.700 UKP1bmarijuana2247532.642.439.132.100 UKP1bmarijuana3247524.541.333.514.410 UKP1bmarijuana4247545.929.533.027.800 UKP1bmarijuanaCoT200027.538.938.736.4509- UKP1bnuclear1357656.330.835.832.200 UKP1bnuclear2357628.337.136.627.700 UKP1bnuclear3357618.327.133.311.710 UKP1bnuclear4357654.232.133.530.000 UKP1bnuclearCoT200026.140.042.835.4532- UKP1bschool1300852.537.435.732.600 UKP1bschool2300828.640.436.827.300 UKP1bschool3300818.928.633.611.300 UKP1bschool4300848.829.232.128.900 UKP1bschoolCoT200027.238.641.235.5477- UKP1bwage1247351.430.936.031.300 UKP1bwage2247329.936.836.930.000 UKP1bwage3247324.926.433.615.310 UKP1bwage4247348.131.435.531.600 UKP1bwageCoT200027.840.741.537.2521- UKP3babortion1392957.652.552.948.200 UKP3babortion2392935.549.151.036.000 UKP3babortion3392961.647.444.043.520 UKP3babortion4392957.447.748.547.400 UKP3babortionCoT200054.245.245.845.551- UKP3bcloning1303955.860.056.152.200 UKP3bcloning2303944.754.855.742.800 UKP3bcloning3303951.647.142.341.020 UKP3bcloning4303953.351.051.350.800 UKP3bcloningCoT200056.856.355.756.042- UKP3bdeath1365154.751.448.343.600 UKP3bdeath2365131.146.546.128.800 UKP3bdeath3365156.145.542.439.620 UKP3bdeath4365152.147.247.745.211 UKP3bdeathCoT200054.048.349.948.860- UKP3bgun1334153.054.453.047.000 UKP3bgun2334136.951.950.536.700 UKP3bgun3334161.151.649.348.220 UKP3bgun4334155.351.651.750.300 UKP3bgunCoT200052.548.149.048.379- UKP3bmarijuana1247561.359.259.657.300 UKP3bmarijuana2247546.256.558.444.800 UKP3bmarijuana3247555.554.743.341.120 UKP3bmarijuana4247555.951.350.950.900 UKP3bmarijuanaCoT200054.652.451.451.631- UKP3bnuclear1357664.362.561.357.700 UKP3bnuclear2357637.952.454.938.100 UKP3bnuclear3357662.554.345.044.300 UKP3bnuclear4357659.951.752.051.200 UKP3bnuclearCoT200055.650.251.550.741- UKP3bschool1300867.365.063.761.700 UKP3bschool2300840.656.257.741.000 UKP3bschool3300861.360.141.739.210 UKP3bschool4300865.257.457.857.600 UKP3bschoolCoT200058.153.451.051.545- UKP3bwage1247359.861.261.556.800 30 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. UKP3bwage2247343.354.657.442.700 UKP3bwage3247361.155.848.547.720 UKP3bwage4247358.254.555.053.700 UKP3bwageCoT200057.054.755.554.960- UKP8babortion1392964.653.652.852.9053 UKP8babortion2392960.058.163.757.582 UKP8babortion3392968.157.350.251.630 UKP8babortion4392960.350.051.550.000 UKP8babortionCoT200063.255.253.954.237- UKP8bcloning1303968.868.268.267.100 UKP8bcloning2303962.767.069.663.550 UKP8bcloning3303966.966.660.962.340 UKP8bcloning4303964.663.165.663.500 UKP8bcloningCoT200063.064.961.862.954- UKP8bdeath1365167.060.760.359.300 UKP8bdeath2365151.358.665.150.820 UKP8bdeath3365168.660.160.059.220 UKP8bdeath4365158.756.759.454.400 UKP8bdeathCoT200067.564.762.863.439- UKP8bgun1334163.460.561.159.100 UKP8bgun2334155.056.961.554.640 UKP8bgun3334166.758.754.254.910 UKP8bgun4334152.450.248.747.000 UKP8bgunCoT200061.357.557.757.651- UKP8bmarijuana1247568.967.666.065.800 UKP8bmarijuana2247564.267.170.664.930 UKP8bmarijuana3247567.968.661.062.910 UKP8bmarijuana4247563.961.262.761.100 UKP8bmarijuanaCoT200063.263.563.563.559- UKP8bnuclear1357675.772.167.569.000 UKP8bnuclear2357663.763.071.063.030 UKP8bnuclear3357670.767.258.760.850 UKP8bnuclear4357671.364.765.865.100 UKP8bnuclearCoT200067.464.763.063.827- UKP8bschool1300875.371.972.871.400 UKP8bschool2300862.663.470.462.330 UKP8bschool3300874.173.063.865.310 UKP8bschool4300870.665.869.166.700 UKP8bschoolCoT200072.570.369.870.035- UKP8bwage1247372.069.070.969.100 UKP8bwage2247363.265.770.863.941 UKP8bwage3247372.369.765.166.760 UKP8bwage4247363.060.963.960.800 UKP8bwageCoT200065.464.063.963.846- UKP31_70babortion1392969.866.971.466.800 UKP31_70babortion2392965.864.974.564.811 UKP31_70babortion3392967.063.770.564.6550 UKP31_70babortion4392963.958.061.257.700 UKP31_70babortionCoT200073.868.773.870.60- UKP31_70bcloning1303974.175.276.374.200 UKP31_70bcloning2303971.573.577.971.910 UKP31_70bcloning3303971.773.175.973.5770 UKP31_70bcloning4303972.873.974.572.500 UKP31_70bcloningCoT200076.575.777.376.20- UKP31_70bdeath1365167.670.668.966.200 UKP31_70bdeath2365168.767.877.567.700 UKP31_70bdeath3365167.466.571.366.3530 31 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. UKP31_70bdeath4365166.367.267.763.900 UKP31_70bdeathCoT200072.571.774.671.60- UKP31_70bgun1334169.067.773.167.700 UKP31_70bgun2334164.766.773.064.610 UKP31_70bgun3334165.965.669.965.1530 UKP31_70bgun4334154.953.152.350.410 UKP31_70bgunCoT200070.968.273.669.40- UKP31_70bmarijuana1247574.673.577.274.500 UKP31_70bmarijuana2247571.372.778.071.810 UKP31_70bmarijuana3247573.274.578.975.2630 UKP31_70bmarijuana4247571.871.074.771.400 UKP31_70bmarijuanaCoT200073.872.876.774.00- UKP31_70bnuclear1357675.673.777.674.100 UKP31_70bnuclear2357672.570.780.071.800 UKP31_70bnuclear3357674.973.279.174.8741 UKP31_70bnuclear4357675.873.177.673.900 UKP31_70bnuclearCoT200076.272.776.974.30- UKP31_70bschool1300881.779.082.780.200 UKP31_70bschool2300876.874.782.876.310 UKP31_70bschool3300880.077.481.579.0530 UKP31_70bschool4300880.476.581.078.100 UKP31_70bschoolCoT200083.280.283.681.70- UKP31_70bwage1247374.172.576.472.800 UKP31_70bwage2247372.672.479.072.710 UKP31_70bwage3247372.572.277.273.2540 UKP31_70bwage4247370.568.772.668.900 UKP31_70bwageCoT200075.573.076.474.30- UKP33_70babortion1392971.070.471.667.800 UKP33_70babortion2392967.065.875.365.800 UKP33_70babortion3392970.066.072.967.060 UKP33_70babortion4392967.760.762.560.000 UKP33_70babortionCoT200073.969.072.069.90- UKP33_70bcloning1303974.776.477.174.900 UKP33_70bcloning2303973.074.679.173.300 UKP33_70bcloning3303975.075.479.275.6130 UKP33_70bcloning4303974.175.975.273.700 UKP33_70bcloningCoT200078.177.578.577.80- UKP33_70bdeath1365167.273.770.067.300 UKP33_70bdeath2365169.669.378.968.800 UKP33_70bdeath3365170.672.275.670.760 UKP33_70bdeath4365168.773.468.466.800 UKP33_70bdeathCoT200075.074.776.574.10- UKP33_70bgun1334168.469.073.567.500 UKP33_70bgun2334166.468.074.566.200 UKP33_70bgun3334167.667.572.466.8240 UKP33_70bgun4334160.056.656.554.500 UKP33_70bgunCoT200071.868.573.269.70- UKP33_70bmarijuana1247573.673.277.373.600 UKP33_70bmarijuana2247573.374.279.473.700 UKP33_70bmarijuana3247573.373.978.874.0170 UKP33_70bmarijuana4247573.072.574.972.400 UKP33_70bmarijuanaCoT200075.274.177.875.20- UKP33_70bnuclear1357676.275.279.075.200 UKP33_70bnuclear2357673.271.480.672.600 UKP33_70bnuclear3357676.274.680.975.9320 UKP33_70bnuclear4357678.076.176.975.400 UKP33_70bnuclearCoT200077.674.378.175.80- 32 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. UKP33_70bschool1300883.581.484.982.300 UKP33_70bschool2300878.376.284.577.800 UKP33_70bschool3300883.179.585.281.620 UKP33_70bschool4300882.079.481.779.800 UKP33_70bschoolCoT200084.581.684.783.00- UKP33_70bwage1247373.473.276.672.400 UKP33_70bwage2247373.372.979.673.400 UKP33_70bwage3247374.974.379.974.9160 UKP33_70bwage4247373.472.075.471.700 UKP33_70bwageCoT200078.576.179.477.40- UKPscoutabortion1392972.065.457.257.900 UKPscoutabortion2392960.562.370.459.910 UKPscoutabortion3392976.771.274.872.700 UKPscoutabortion4392967.453.851.352.100 UKPscoutabortionCoT200074.171.462.565.20- UKPscoutcloning1303974.978.869.771.600 UKPscoutcloning2303968.972.676.069.500 UKPscoutcloning3303978.077.479.377.800 UKPscoutcloning4303973.875.569.871.400 UKPscoutcloningCoT200075.378.870.973.00- UKPscoutdeath1365176.176.968.270.200 UKPscoutdeath2365160.465.270.560.300 UKPscoutdeath3365177.776.376.475.600 UKPscoutdeath4365173.367.367.167.100 UKPscoutdeathCoT200080.481.673.876.60- UKPscoutgun1334170.865.362.963.500 UKPscoutgun2334160.563.668.860.400 UKPscoutgun3334174.971.475.572.700 UKPscoutgun4334164.056.456.656.200 UKPscoutgunCoT200072.269.065.466.40- UKPscoutmarijuana1247573.976.268.369.700 UKPscoutmarijuana2247569.771.275.870.100 UKPscoutmarijuana3247577.876.479.277.500 UKPscoutmarijuana4247570.369.264.765.600 UKPscoutmarijuanaCoT200074.676.670.272.50- UKPscoutnuclear1357679.079.670.473.300 UKPscoutnuclear2357669.768.576.969.200 UKPscoutnuclear3357679.375.779.377.200 UKPscoutnuclear4357676.573.369.471.000 UKPscoutnuclearCoT200077.977.370.973.40- UKPscoutschool1300882.983.277.079.200 UKPscoutschool2300871.271.779.971.500 UKPscoutschool3300885.282.885.383.800 UKPscoutschool4300881.278.177.377.700 UKPscoutschoolCoT200083.884.079.481.40- UKPscoutwage1247374.278.064.467.800 UKPscoutwage2247372.071.677.371.700 UKPscoutwage3247380.178.278.878.510 UKPscoutwage4247373.270.265.467.100 UKPscoutwageCoT200076.876.971.173.20- UKPds70babortion1200077.575.069.771.800 UKPds70babortion2200062.162.272.261.701 UKPds70babortion3200076.071.075.472.800 UKPds70babortion4200076.071.271.171.203 UKPds70bcloning1200075.376.372.573.900 UKPds70bcloning2200072.572.977.773.001 UKPds70bcloning3200076.275.576.275.800 33 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. UKPds70bcloning4200076.376.075.575.600 UKPds70bdeath1200077.575.774.874.800 UKPds70bdeath2200071.568.977.170.104 UKPds70bdeath3200077.374.676.675.000 UKPds70bdeath4200075.474.575.474.107 UKPds70bgun1200075.171.874.672.900 UKPds70bgun2200065.766.573.165.402 UKPds70bgun3200074.171.076.172.500 UKPds70bgun4200071.868.672.369.402 UKPds70bmarijuana1200078.578.077.877.900 UKPds70bmarijuana2200073.273.478.473.700 UKPds70bmarijuana3200078.277.079.678.000 UKPds70bmarijuana4200076.174.976.175.502 UKPds70bnuclear1200078.376.273.874.900 UKPds70bnuclear2200070.869.178.670.3018 UKPds70bnuclear3200079.375.978.277.000 UKPds70bnuclear4200078.074.875.475.106 UKPds70bschool1200083.682.081.381.600 UKPds70bschool2200079.576.984.479.010 UKPds70bschool3200084.381.784.082.800 UKPds70bschool4200083.581.582.481.913 UKPds70bwage1200078.676.776.976.800 UKPds70bwage2200072.471.377.372.301 UKPds70bwage3200078.876.479.477.600 UKPds70bwage4200076.373.975.774.603 UKPoss20babortion1392975.871.866.568.60185 UKPoss20babortion2392967.964.272.265.601108 UKPoss20babortion3392976.875.066.269.4015 UKPoss20babortion4392974.868.767.868.20540 UKPoss20bcloning1303976.578.273.475.00116 UKPoss20bcloning2303973.373.177.573.50760 UKPoss20bcloning3303975.579.970.373.108 UKPoss20bcloning4303976.476.674.375.20257 UKPoss20bdeath1365178.679.269.673.00186 UKPoss20bdeath2365171.868.477.369.901011 UKPoss20bdeath3365177.579.967.471.4017 UKPoss20bdeath4365178.175.271.373.00510 UKPoss20bgun1334174.771.567.268.80157 UKPoss20bgun2334166.965.470.965.70989 UKPoss20bgun3334175.075.266.168.9015 UKPoss20bgun4334173.969.569.069.20415 UKPoss20bmarijuana1247574.276.669.371.80125 UKPoss20bmarijuana2247572.171.676.172.31499 UKPoss20bmarijuana3247573.279.066.469.808 UKPoss20bmarijuana4247574.573.571.372.20417 UKPoss20bnuclear1357679.877.875.176.30139 UKPoss20bnuclear2357671.069.178.170.40797 UKPoss20bnuclear3357679.779.373.275.7010 UKPoss20bnuclear4357675.269.571.270.20513 UKPoss20bschool1300884.785.080.482.4089 UKPoss20bschool2300880.377.384.279.40671 UKPoss20bschool3300883.385.876.780.207 UKPoss20bschool4300883.280.979.780.30295 UKPoss20bwage1247380.479.176.677.50142 UKPoss20bwage2247373.172.078.072.80625 UKPoss20bwage3247379.579.674.776.309 UKPoss20bwage4247380.177.777.477.30282 34 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. UKPoss120babortion1392978.475.470.572.506 UKPoss120babortion2392971.167.976.669.200 UKPoss120babortion3392979.475.673.774.500 UKPoss120babortion4392978.073.074.173.505 UKPoss120bcloning1303978.980.076.477.701 UKPoss120bcloning2303976.276.080.476.401 UKPoss120bcloning3303979.479.777.578.300 UKPoss120bcloning4303977.777.177.777.102 UKPoss120bdeath1365179.880.572.175.302 UKPoss120bdeath2365174.671.280.473.001 UKPoss120bdeath3365179.679.972.375.300 UKPoss120bdeath4365180.078.175.176.404 UKPoss120bgun1334177.275.471.272.903 UKPoss120bgun2334171.169.675.970.100 UKPoss120bgun3334178.476.573.774.700 UKPoss120bgun4334177.674.374.674.302 UKPoss120bmarijuana1247577.478.873.975.800 UKPoss120bmarijuana2247575.175.179.675.400 UKPoss120bmarijuana3247579.680.676.578.100 UKPoss120bmarijuana4247579.778.978.978.902 UKPoss120bnuclear1357681.179.876.177.703 UKPoss120bnuclear2357674.371.880.973.501 UKPoss120bnuclear3357659.246.233.525.300 UKPoss120bnuclear4357680.977.579.478.407 UKPoss120bschool1300885.985.682.583.902 UKPoss120bschool2300881.778.686.280.800 UKPoss120bschool3300858.156.234.727.800 UKPoss120bschool4300886.083.685.384.303 UKPoss120bwage1247380.880.376.677.802 UKPoss120bwage2247375.774.280.275.500 UKPoss120bwage3247382.281.378.879.600 UKPoss120bwage4247381.579.879.579.401 UKPgpt-5.2abortion1200079.074.481.776.902 UKPgpt-5.2abortion2200070.668.578.969.700 UKPgpt-5.2abortion3199878.073.581.576.042 UKPgpt-5.2abortion4200077.673.281.675.706 UKPgpt-5.2cloning1200078.077.080.477.902 UKPgpt-5.2cloning2200073.574.278.573.700 UKPgpt-5.2cloning3200078.377.380.978.102 UKPgpt-5.2cloning4200076.275.679.476.200 UKPgpt-5.2death1200078.675.381.077.000 UKPgpt-5.2death2200071.470.279.970.400 UKPgpt-5.2death3199978.875.382.077.322 UKPgpt-5.2death4199976.573.681.175.320 UKPgpt-5.2gun1199875.973.579.674.842 UKPgpt-5.2gun2200070.270.377.569.900 UKPgpt-5.2gun3199976.974.280.075.626 UKPgpt-5.2gun4199975.473.480.174.522 UKPgpt-5.2marijuana1200079.778.482.779.600 UKPgpt-5.2marijuana2200074.475.180.574.700 UKPgpt-5.2marijuana3199879.478.282.679.440 UKPgpt-5.2marijuana4199978.978.183.079.022 UKPgpt-5.2nuclear1199979.575.782.377.920 UKPgpt-5.2nuclear2200073.171.480.972.400 UKPgpt-5.2nuclear3199979.175.182.177.322 UKPgpt-5.2nuclear4200078.074.482.076.600 UKPgpt-5.2school1200086.383.388.785.300 35 A comprehensive study of LLM-based argument classification CorpusModelDatasetPromptnAccPrecRecF1Inv. ans.Inv. cert. UKPgpt-5.2school2200080.777.985.879.900 UKPgpt-5.2school3199986.283.187.985.022 UKPgpt-5.2school4199984.481.187.683.320 UKPgpt-5.2wage1199882.880.484.381.842 UKPgpt-5.2wage2199974.273.479.874.120 UKPgpt-5.2wage3199982.580.184.281.720 UKPgpt-5.2wage4199681.178.783.580.382 36