Paper deep dive

How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding

Jeongyeon Kim, John Mitchell

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 137

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 9/11/2026, 4:23:09 AM

Summary

This paper investigates the effectiveness of multi-agent Large Language Models (LLMs) in performing deductive qualitative coding. The authors developed a baseline pipeline where AI agents independently code, debate, and reconcile disagreements across four diverse qualitative datasets. Key findings indicate that coding accuracy is influenced by codebook length, data similarity, and the intensity of agent disagreement, with intense debates leading to higher accuracy. The study highlights that while LLMs emulate human discussion behaviors, they lack adaptive responsiveness to context. Based on an exploratory literature review of user needs, the paper proposes design goals and recommendations for building transparent, reliable, and user-centered automated qualitative coding systems.

Entities (8)

Jeongyeon Kim · person · 99%John Mitchell · person · 99%Stanford University · organization · 98%Qualitative Coding · task · 97%Multi-agent LLM coding · method · 95%Agent Disagreement · factor · 92%Codebook Length · factor · 90%Design Goals · concept · 88%

Relation Signals (6)

Jeongyeon Kim → affiliatedwith → Stanford University

confidence 99% · Affiliation: Stanford University, Stanford, CA, USA

John Mitchell → affiliatedwith → Stanford University

confidence 99% · Affiliation: Stanford University, Stanford, CA, USA

Multi-agent LLM coding → usedfor → Qualitative Coding

confidence 95% · quantifying the effectiveness of multi-agent LLM coding across varied qualitative datasets

Agent Disagreement → influences → Coding Accuracy

confidence 92% · intense and unresolved debates between agents led to higher accuracy

Codebook Length → influences → Coding Accuracy

confidence 90% · coding accuracy depends on factors such as codebook length

Exploratory Literature Review → informs → Design Goals

confidence 88% · The findings of the exploratory literature review emphasize the significance of transparency and reliability... we proposed a set of design goals

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The utility of AI in multi-coder qualitative coding has been widely discussed, yet little empirical evidence exists to delineate the contexts in which it performs reliably. We address this gap by quantifying the effectiveness of multi-agent LLM coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes. We developed a literature-informed baseline pipeline that enables AI agents to independently code, debate, and reconcile disagreements. Results revealed that coding accuracy depends on factors such as codebook length, qualitative data similarity, and agent disagreement. Notably, intense and unresolved debates between agents led to higher accuracy. Our analysis showed that while LLMs emulate many human discussion behaviors, they lack adaptive responsiveness to context. From these findings, we offer design recommendations for building automated coding systems. Our open-source AI discussion dataset and methodological framework lay the groundwork for advancing the design of AI-mediated automated thematic analysis.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

135,185 characters extracted from source content.

Expand or collapse full text

How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding CCS: Human-centered computing Collaborative and social computing systems and toolsCCS: Human-centered computing HCI design and evaluation methodsCCS: Computing methodologies Discourse, dialogue and pragmatics Jeongyeon Kim email: jeongyeon.kim@stanford.edu Affiliation: Stanford University, Stanford, CA, USA and John Mitchell email: John.Mitchell@stanford.edu Affiliation: Stanford University, Stanford, CA, USA © none Abstract. The utility of AI in multi-coder qualitative coding has been widely discussed, yet little empirical evidence exists to delineate the contexts in which it performs reliably. We address this gap by quantifying the effectiveness of multi-agent LLM coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes. We developed a literature-informed baseline pipeline that enables AI agents to independently code, debate, and reconcile disagreements. Results revealed that coding accuracy depends on factors such as codebook length, qualitative data similarity, and agent disagreement. Notably, intense and unresolved debates between agents led to higher accuracy. Our analysis showed that while LLMs emulate many human discussion behaviors, they lack adaptive responsiveness to context. From these findings, we offer design recommendations for building automated coding systems. Our open-source AI discussion dataset and methodological framework lay the groundwork for advancing the design of AI-mediated automated thematic analysis. 1. Introduction Deductive qualitative coding is the process of applying predefined frameworks to categorize and interpret qualitative data. It is a time-consuming and labor-intensive task even for experts, demanding meticulous examination and contextual understanding of nuanced data (Jiang et al., 2021; Kantor, 2024). The central role of qualitative coding in social science has led to a growing body of recent work on automating qualitative coding through artificial intelligence (AI) and natural language processing (NLP) techniques. Motivated by our own interest in qualitative research with large amounts of data, we investigate AI-based coding methods modeled on human practices, such as multi-coder comparison and consensus. Our study quantifies the effectiveness of multi-agent Large Language Model (LLM) coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes. In leveraging AI for multi-coder qualitative coding, we investigate how effectively AI can engage with the complexities of data, identifying specific conditions — when and in which contexts — AI demonstrates consistent reliability. Our analysis is intended to help researchers using our methods avoid both over-reliance that can amplify automation bias by sidelining necessary human interpretation and under-trust that could preclude their potential utility. Although AI tools may perform adequately in structured cases, they often falter in interpreting the nuanced subtleties of qualitative data like interview transcripts. This form of data may resist rigid categorization, requiring interpretive judgment that extends beyond mechanical rule-based classification. Even among experienced human coders, variability in interpretation is common, which is why measures like intra-rater reliability and consensus-building are indispensable. Qualitative coding is thus not merely a technical task but a deeply interpretive practice grounded in human insight. Despite these inherent complexities of qualitative coding, recent advancements in LLMs have accelerated efforts to automate this process. They explored prompt engineering (Chew et al., 2023; Zhang et al., 2024) and incorporated human feedback loops into the automation process (Zambrano et al., 2023; Barany et al., 2024). However, these efforts remain confined to narrowly defined contexts, addressing limited or domain-specific datasets. Moreover, the prevailing focus on coding accuracy marginalizes user-centered considerations and broader system design concerns, such as transparency and interpretability. As a result, system designers are left without comprehensive, practically grounded frameworks to guide the development of automated systems that align with the needs and expectations of end users. Our research bridges these gaps by (1) investigating user needs with automated coding systems through conducting an exploratory literature review, (2) evaluating LLM capabilities across diverse datasets, and (3) proposing actionable design guidelines for future systems. The findings of the exploratory literature review emphasize the significance of transparency and reliability in automated coding systems. The reviewed studies demonstrate that users seek to understand the underlying coding process, including why codes are assigned and how disagreements among coders are resolved. The literature also stresses the necessity of communicating system reliability, allowing users to set realistic expectations about system outputs. Another recurrent theme across the literature is a lack of explicit optimization strategies or actionable guidance to enhance system performance, which often leads to user frustration. Drawing upon the findings from the literature review, we proposed a set of design goals aimed at improving the transparency and reliability of automated qualitative coding systems. Following a synthesis of user needs derived from our review, we investigated the capacity of LLMs with the aim of translating identified user needs into actionable design recommendations. Based on the extracted design goals, we developed a baseline pipeline for qualitative coding, which was applied across four distinct qualitative datasets. This pipeline adopts a multi-agent architecture where AI agents simulate the collaborative coding process of human coders. The experiment results demonstrated how factors such as dataset characteristics, labeling uncertainty, and coder consensus levels influence the effectiveness of LLM-based systems. For example, coding accuracy improves when the codebook is more concise and when AI coders engage in more intense disagreements during the coding process. Additionally, by analyzing and comparing the behaviors of LLMs and human coders, we identified strategies for developers and designers to adjust LLMs for automated coding tasks. The analysis revealed that while LLMs replicate basic human interaction patterns, such as acknowledging other discussants’ perspectives and proposing alternatives, they fall short in nuances, such as questioning to clarify uncertainties or drawing on personal experiences. These findings culminated in a set of design recommendations for creating LLM-based qualitative coding systems. The recommendations introduce strategies for designers, including guiding users of systems through data preprocessing, leveraging uncertainty and conflict as diagnostic signals for accuracy, and tuning agent discussion styles based on task ambiguity. These recommendations serve as a conceptual toolkit that helps users better understand and interact with model outputs. To advance the field, we made our baseline pipeline and the LLM agents’ discussion dataset publicly available. The contributions of our work can be summarized as below: • Design goals for automated qualitative coding systems based on exploratory literature review • Design of baseline pipeline for LLM-based qualitative coding • Discussion dataset of LLM coders across four qualitative coding tasks • Data analysis of LLM capabilities and factors influencing coding accuracy • Design recommendations for LLM-based qualitative coding systems 2. Related Work Even for experts, qualitative coding involves considerable cognitive and temporal investment, owing to the need for detailed scrutiny and deep contextual understanding of nuanced data (Jiang et al., 2021; Kantor, 2024). Specifically, our work focused on deductive coding, which applies predefined theoretical frameworks to categorize and interpret qualitative data. This section addresses two dimensions in the evolution of qualitative coding systems: the automation of qualitative coding through NLP techniques and the role of human users in engaging with these automated systems. 2.1. Automation of Qualitative Coding Efforts to automate qualitative coding in NLP have evolved from simple rule-based approaches to interactive machine-learning systems. Marathe and Toyama (Marathe and Toyama, 2018) proposed a semi-automated tool using basic NLP methods like keyword matching, while Chen et al. (Chen et al., 2018) introduced SVM models with a visualization on coder disagreement. Several studies broadened the scope of AI-assisted automated coding to encompass workflows that involve collaboration between multiple human coders (Drouhard et al., 2017; Gao et al., 2023b). Building on these efforts, the advancement of LLMs has accelerated the development of automated qualitative coding techniques. For example, Dunivin et al. (Dunivin, 2024) and Ashwin et al. (Ashwin et al., 2023) demonstrated that chain-of-thought reasoning enhances coding accuracy alongside a few-shot learning method (Xiao et al., 2023). Several papers (Chew et al., 2023; Zhang et al., 2024) introduced stepwise prompt-guided coding and Liu et al. (Liu et al., ) highlighted the role of contextual data, such as dialogue flow and timestamps. Other studies have shown that prompt engineering can greatly enhance the accuracy of coding with LLMs. Examples include using precise codebook definitions in social science domains (Halterman and Keith, 2024), adding coding rules informed by debates over borderline cases (Meng et al., 2024), and including the reasoning behind code assignments (Siiman et al., 2023). Following these advances in prompt engineering, researchers have explored ways to make the LLM-based coding process more rigorous and systematic to improve the precision and adaptability of coding. Subramaniam et al. (Subramaniam et al., 2024) proposed DebateGPT, a multi-agent system refining outputs through summarization and confidence scoring. Bryda and Sadowski (Bryda and Sadowski, 2024) iteratively adjusted prompt design to guide AI in deductive coding with clarity and consistency. Meanwhile, recognizing the need for methodological rigor and reproducibility, Tai et al. (Tai et al., 2024) introduced a metric to measure coding consistency over repeated runs, and several other studies (Bijker et al., 2024; Kirsten et al., 2024) also ran multiple repeated LLM experiments to ensure reliability and robustness in their findings. Existing research is confined to experiments on individual datasets, leaving a gap in providing a comprehensive guide for designing scalable and generalizable automated qualitative coding systems. Furthermore, as far as we are aware, there is no existing research that takes an analytical approach to how LLMs perform under varying conditions (e.g., codebook length, complexity of data to be coded). This lack of analysis of the factors influencing LLM performance makes it difficult for system developers and designers to optimize LLM performance, leaving it as an unguided trial-and-error process. Our study seeks to bridge this gap by performing a statistical analysis of LLM capabilities in qualitative coding across diverse datasets. By analyzing the relationship between dataset features and LLM performance, our work offers a foundation for understanding and optimizing LLM capabilities in qualitative coding. 2.2. Humans’ Roles in Automated Qualitative Coding Despite advancements in technologies for automating qualitative coding, practical adoption may remain limited if the roles and needs of human users are overlooked. In this context, prior research highlights efforts to address users’ needs and challenges in automating the process of coding. Through interviews, Zhang et al. (Zhang et al., 2023b) uncovered several user concerns regarding the use of LLMs in qualitative coding, such as a lack of transparency and limited understanding of LLMs’ capabilities. Similarly, Chen et al. (Chen et al., 2016) highlighted users’ concerns regarding the black-box nature and lack of transparency in automation models. Jiang et al. (Jiang et al., 2021), based on interviews with 17 qualitative researchers, suggested that AI-based tools should embrace ambiguity, enhance human agency, and encourage serendipitous discoveries instead of removing uncertainty. In light of these findings, further research has emphasized the active role of users in addressing limitations and refining automation processes. For instance, a few studies stressed the indispensability of user input in automated qualitative coding (Christou, 2023b), such as reflecting and adapting the automated results (Khan et al., 2024) and interpreting nuances of data (Bryda and Sadowski, 2024; Carius and Teixeira, 2024; Morgan, 2023). Preserving user agency (Spinoso-Di Piano, 2023; Kantor, 2024) and ensuring transparency (Feuston and Brubaker, 2021; Rietz and Maedche, 2021; Hong et al., 2022) have also been identified as key features of automated systems. These perspectives collectively highlight the importance of designing systems that not only streamline the coding process but also foster human-in-the-loop collaboration and maintain user control. In this thread, Zambrano et al. (Zambrano et al., 2023) have explored the use of AI agents as collaborative coding companions, and another study (Barany et al., 2024) showed that human-AI collaboration achieves higher accuracy compared to using AI or human coders alone. Meanwhile, interactive tools like PaTAT (Gebreegziabher et al., 2023) and CollabCoder (Gao et al., 2024) exemplified mixed-initiative approaches, allowing users to iteratively guide AI while retaining autonomy. These approaches align with the primary goal of qualitative research, which is to foster a deeper understanding of qualitative data rather than simply speeding up the coding process. Basit et al. (Basit, 2003) further underscored that the goal of qualitative coding in thematic analysis is not merely to label data but to support researchers in discovering, supporting, and confirming their theories through the data. Meanwhile, challenges of automated systems remain as Bano et al. (Bano et al., 2023b) noted the limitations of LLMs in understanding context and nuances, cautioning against AI agents fostering a single, echo-chamber perspective. Ziems et al. (Ziems et al., 2024) evaluated LLMs’ capabilities on diverse benchmarks, showing potential to enhance computational social science but highlighting the need for human oversight in the process. Building on previous studies, our exploratory literature review investigates user requirements and expectations in automating qualitative coding. We then designed LLM experiments based on these findings. By comparing the findings of the exploratory literature review with the outcomes of the experiments, we formulate concrete design recommendations for automated qualitative coding systems. By addressing these gaps, our study lays the groundwork for an integration of LLMs into qualitative research practices through the investigation of user needs, statistical analysis, and design recommendations. 3. Exploratory Literature Review Design Goals (DG) Supporting Literature DG1. Provide reasons and evidence for how codes are assigned. (Rietz and Maedche, 2021; Gebreegziabher et al., 2023; Lopez-Fierro and Nguyen, 2024; Overney, 2023; Jiang et al., 2021; Feuston and Brubaker, 2021; Siiman et al., 2023; Kirsten et al., 2024; Katz et al., 2024; Zhang et al., 2023a; Overney et al., 2024; Palea et al., 2024; Yang and Ma, 2025; Hamilton et al., 2023; Davison et al., 2024; Christou, 2023b; Lennon et al., 2021; Zhang et al., 2023b; Bennis and Mouwafaq, 2025; Amani et al., 2025; Rietz, 2021; Arlinghaus et al., 2024; Hasija and Esper, 2022; Tschisgale et al., 2023; Chen et al., 2024; Chen et al., 2018; Cameron, 2025; Johs et al., 2022; Costa, 2023; Li et al., 2024b; Piano and others, 2024; Sabbaghan, 2024; Than et al., 2024; Christou, 2023a; Şen et al., 2023; Perkins and Roe, 2024a; Wachinger et al., 2024; Gao et al., 2025; Brent and Slusarz, 2003; Tai et al., 2024; Zięba-Kulawik, 2025; De Paoli, 2024; Hong et al., 2022; Turobov et al., 2024; Yan et al., 2024; Nyaaba et al., 2025; Qiao et al., ; Mathis et al., 2024; Parfenova et al., 2024; Ashwin et al., 2023; Player et al., 2024; T"̈ornberg, 2023; Bano et al., 2023a; Bano et al., 2023b; Chew et al., 2023; Xu et al., 2025; De Paoli, 2023) DG2. Illuminate uncertainties and interpretative variability in code assignments. (Rietz and Maedche, 2021; Gebreegziabher et al., 2023; Morgan, 2023; Bryda and Sadowski, 2024; Lopez-Fierro and Nguyen, 2024; Overney, 2023; Jiang et al., 2021; Siiman et al., 2023; Kirsten et al., 2024; Marshall and Naff, 2024; Overney et al., 2024; Mortelmans, 2024; Palea et al., 2024; Gao et al., 2023b; Gao et al., 2023a; Hamilton et al., 2023; Pattyn, 2024; Davison et al., 2024; Sinha et al., 2024; Christou, 2023b; Amani et al., 2025; Hitch, 2024; Jalali and Akhavan, 2024; Nelson et al., 2020; Paulus and Marone, 2024; Mehrad et al., 2024; Christou, 2024; Rietz, 2021; Spinoso-Di Piano, 2023; Tschisgale et al., 2023; Longo, 2019; Perkins and Roe, 2024b; Chopra and Haaland, 2023; Chen et al., 2024; Zhao et al., 2024; Chen et al., 2018; Costa, 2023; Klemmer et al., 2024; Gao et al., 2024; Piano and others, 2024; Sabbaghan, 2024; Goyanes et al., 2024; Than et al., 2024; Wheeler, 2025; Christou, 2023b; Kantor, 2024; Şen et al., 2023; Perkins and Roe, 2024a; Li et al., 2021; Dunivin, 2024; Dengel et al., 2023; Zade et al., 2018; Tai et al., 2024; Christou, 2025; Zięba-Kulawik, 2025; Anis and French, 2023; De Paoli, 2024; Dai et al., 2023; Yan et al., 2024; Nyaaba et al., 2025; Nguyen-Trung and Nguyen, 2025; Lee et al., 2024; Friedman et al., 2024; Hayes, 2025; Eschrich and Sterman, 2024; Qiao et al., ; De Paoli and Mathis, 2024; Atas et al., ; Mathis et al., 2024; Parfenova et al., 2024; Carius and Teixeira, 2024; Rasheed et al., 2024; Ritschl et al., ; Smirnov, 2025; de Morais Leça et al., 2024; Nejjar et al., 2025; Khan et al., 2024; Omizo, 2024; Bano et al., 2024; Bano et al., 2023a; Roberts et al., 2024; Chew et al., 2023; Rao et al., 2024; De Paoli, 2023; Nelson et al., 2021; Nguyen-Trung, 2024) DG3. Enable users to calibrate expectations for coding accuracy. (Gebreegziabher et al., 2023; Marshall and Naff, 2024; Zhang et al., 2023a; Overney et al., 2024; Palea et al., 2024; Gao et al., 2023b; Gao et al., 2023a; Christou, 2023b; Zhang et al., 2023b; Xiao et al., 2023; Hasija and Esper, 2022; Johs et al., 2022; Klemmer et al., 2024; Zade et al., 2018; Hong et al., 2022; Bano et al., 2024) DG4. Facilitate users’ understanding of how data characteristics affect coding accuracy. (Rietz and Maedche, 2021; Bryda and Sadowski, 2024; Overney, 2023; Feuston and Brubaker, 2021; Siiman et al., 2023; Kirsten et al., 2024; Marshall and Naff, 2024; Prescott et al., 2024; Overney et al., 2024; Mortelmans, 2024; Gao et al., 2023a; Yang and Ma, 2025; Hamilton et al., 2023; Amani et al., 2025; Bijker et al., 2024; Liu et al., 2025; Chubb, 2023; Tschisgale et al., 2023; Gao et al., 2024; Giri, 2024; Mesec, 2023; Wachinger et al., 2024; Dunivin, 2024; Brondani et al., 2024; Anis and French, 2023; Guetterman et al., 2018; Hayes, 2025; Atas et al., ; de Morais Leça et al., 2024; Ashwin et al., 2023; Player et al., 2024; Bano et al., 2023b; Chew et al., 2023) Table 1. An overview of the design goals for automated qualitative coding systems and literature that informed their formulation. The exploratory literature review explores user needs and expectations for automated qualitative coding systems, aiming to derive design goals that inform the development of a baseline LLM-based coding pipeline. We adopted the PRISMA statement (Moher et al., 2009) for review. 3.1. Review Methodology 3.1.1. Identification Process For the identification of relevant literature, we conducted a database search using the following keywords: (“AI” OR “artificial intelligence” OR “LLMs” OR “large language models” OR “automated”) AND (“qualitative coding” OR “qualitative analysis” OR “thematic analysis” OR “thematic coding”). This search yielded an initial set of 988 articles, which were sorted and selected based on their relevance to the specified keywords. 3.1.2. Screening Process One of the authors reviewed the titles and abstracts of all retrieved papers. Papers were excluded if their focus was unrelated to automated qualitative coding or if they were not written in English. For example, papers using the term "coding" in the context of computer programming rather than thematic analysis were removed. This screening procedure resulted in a final set of 180 relevant papers. 3.1.3. Assessing Eligibility To determine the eligibility of relevant literature, one of our authors read full articles of 180 papers. Through an iterative process of reading and analysis, the author developed the following exclusion criteria: • Papers were excluded if the primary focus was not on automated qualitative coding. For instance, studies that employed qualitative methods such as interviews or surveys as part of broader research themes (e.g., AI in Education: A Qualitative Analysis) were omitted. • Papers that discussed solely human-driven coding practices without reference to automated systems were excluded (e.g., Analyzing Qualitative Data via Collaborative Coding: A Practical Guide). • Papers were excluded if they lacked explicit discussions of user expectations or design considerations in automated qualitative coding systems. This includes works that merely presented use cases of technologies or technical architectures without addressing user-side design requirements. The exclusion criteria were refined throughout the iterative review process. As a result of this eligibility evaluation, a final corpus of 118 papers was selected for analysis. The complete list of reviewed papers is provided in the Supplementary Materials. 3.2. Analysis Given that our aim was to inform design goals for our baseline pipeline of LLM-based qualitative coding, we conducted an exploratory review rather than a comprehensive, systematic literature review. One of our authors generated preliminary codes through a full reading of the corpus, followed by a second round of analysis to finalize the codebook. We then invited a qualitative research expert with a PhD in sociology to independently assess the codebook against the full corpus. 3.3. Review Results We present the design goals identified through the review of prior literature. Table 1 provides an overview of these goals and their supporting studies. 3.3.1. Transparency Coding automation should help users make informed decisions by presenting outputs that are transparent in their reasoning and limitations, including an explanation of how results are derived and any associated uncertainties. DG1. Provide reasons and evidence for how codes are assigned. Prior work emphasizes that automated coding systems should offer explanations for their coding outputs (Rietz and Maedche, 2021; Zhang et al., 2023a) to support user understanding of AI reasoning (Gebreegziabher et al., 2023; Christou, 2023b). To reflect this, we prompted the LLM to include justifications when generating codes. DG2. Illuminate uncertainties and interpretative variability in code assignments. Literature highlights the importance of exposing uncertainties in AI-generated codes, often represented through confidence scores (Li et al., 2021; Gebreegziabher et al., 2023; Klemmer et al., 2024). Furthermore, prior work underscores interpretive variability, across both human and AI coders (Siiman et al., 2023; Dai et al., 2023). Based on this, we used two LLM coders to perform coding independently and resolve disagreements through discussion, mimicking human coding processes. We instructed the models to estimate confidence level, allowing LLM coders to assign an ‘Undecidable’ label when uncertain. 3.3.2. Reliability Automated coding systems should enable users to better understand the system’s capabilities and develop strategies to optimize performance. DG3. Enable users to calibrate expectations for coding accuracy. Users often struggle when they cannot align expectations with system performance (Christou, 2023b; Palea et al., 2024). In line with established approaches (Bano et al., 2023b; Zade et al., 2018), we used coder disagreements during discussions as a proxy for evaluating automated coding accuracy. DG4. Facilitate users’ understanding of how data characteristics affect coding accuracy. Literature suggests that factors such as codebook complexity (Bryda and Sadowski, 2024), the abstractness of language (Prescott et al., 2024; Overney, 2023), and data similarity (Amani et al., 2025; Bijker et al., 2024) affect coding accuracy. Our study investigated how variations in data and codebook (e.g., length, similarity, and specialization) impact coding reliability. 4. Large Language Model Experiments and Data Analysis In this section, we report the pipeline and results of LLM experiments designed based on the design goals. Our goal is to examine the capabilities of LLMs in qualitative coding by evaluating their performance across different datasets. 4.1. Datasets We conducted experiments using LLMs across four distinct domains: education, law, sociology, and medicine, utilizing datasets from prior studies. The datasets include a dialogue corpus in learning (Wachsmuth and Alshomary, 2022), cases from the European Court of Human Rights (Chalkidis et al., 2021), open-ended interview responses (Sanfilippo et al., 2020), and medical paper abstracts (Schopf et al., 2022). To standardize dataset sizes, we randomly selected five labels from each corpus and sampled 500 instances per label. Meanwhile, we have released the prompt design, coding pipeline, and LLM-generated discussions as open-source materials11 1 https://anonymous.4open.science/r/qual-coding-llm-experiment-1DA3/README.md. 4.2. Design of Coding Pipeline Based on the exploratory literature review, we designed an automated coding pipeline powered by AI agents. The process mirrors human coding in that the AI coders first independently code the entire dataset and two AI coders engage in discussions about conflicting cases (DG2). In the discussion process, they aim to resolve conflicts and derive new labeling rules based on the discussions. An important note is that, before any of these steps, an initial phase of extracting excerpts to be coded was conducted. This data preprocessing step aimed to standardize the units of analysis. In this step, the LLM was supplied with the codebook and the data to be coded, and it was instructed to extract data excerpts likely to correspond to the codes in the codebook. As a result, only the extracted excerpts, rather than the entire qualitative dataset, were used for coding. This approach aligns with methodologies discussed in recent studies, which emphasize the importance of preprocessing to improve the reliability of qualitative coding by ensuring only pertinent excerpts are subjected to analysis (Gao et al., 2023a; Than et al., 2024; Schwitter, 2025). The prompts used in the coding process were structured as follows. AI coders were instructed to justify their coding decisions (DG1) and had three possible response options: the code is present in the data, absent from the data, or undecidable (DG2). We grounded the prompt design in prior work, adhering to principles of constructive discussions (Scannapieco, 1997; Deutsch et al., 2011; Summers et al., 1950) and efficient AI-agent dialogue frameworks (Li et al., 2024a; Wang et al., 2024; Chan et al., 2023; Smit et al., ). To reduce ambiguity in the coding task for AI models, we followed established conventions in the literature by including only one code per coding cycle and repeating the process for each code individually. As the discussion progressed through multiple rounds and turns, LLM coders were encouraged to formulate their answers by considering both their previous reasoning and the other coder’s perspectives. If disagreements arose initially, but consensus was achieved through rounds of discussions, the coders collaboratively developed a new labeling rule to resolve similar disputes in the future. Each back-and-forth exchange constituted one round of discussion. If no agreement was reached after three rounds, the case was marked as “disagreed,” and the process moved on to the next case. To avoid ordering bias, we adhered to methodologies established in reasoning and QA frameworks for AI-agent discussions (Du et al., 2023). We utilized OpenAI’s ChatGPT with the ‘gpt-4o-mini’ model for this process. The temperature settings were 0 for coding and 0.7 for discussions. These configurations align with previous studies on qualitative coding, but future researchers may adjust them to meet their objectives. After completing the discussion phase, both AI coders re-coded the entire dataset using shared knowledge derived from discussions and newly established labeling rules. At each coding round, we calculated inter-rater reliability (IRR) using Cohen’s Kappa. 4.3. Evaluation Method Following the pipeline design, we addressed data imbalance using an undersampling technique (Liu et al., 2008) and computed F1 scores to assess coding accuracy. We then examined the relationship between coding accuracy and factors related to datasets and AI-agent discussions (DG4). These factors included dataset characteristics, such as length of codebooks, semantic similarity of content within codebooks, length of data excerpt to be coded, semantic similarity of content within excerpts, and disparity of specialization degree between codebooks and excerpts. Additionally, labeling dynamics, such as labeling uncertainty, degree of consensus and controversy between coders, were also considered (DG3), drawing upon the literature detailed in Table 2, 3, 4, and 5. We quantify corpus’ semantic proximity using cosine similarity, one of the most common similarity metrics, enabling a scale-invariant comparison of textual representations (Reimers and Gurevych, 2019). Meanwhile, to measure the degree of specialization of corpus, we adopted a frequency-based approach that follows practice in corpus linguistics and psycholinguistics, lower reference-corpus frequency indicates higher lexical difficulty (Breland, 1996; Chen and Meurers, 2016). Specifically, we used the Corpus of Contemporary American English (COCA) to compute a Word Difficulty Score (WDS) for each word, defined by the inverse frequency of the word (Tang et al., 2023; Corpus, ). For statistical analysis, we adopted Mixed-Effects Models, an approach designed to handle both fixed and random effects, making it particularly suited for grouped or hierarchical data (Schmettow, 2015; Agresti et al., 2000; Luger et al., 2014). Given the grouping structure of our data — comprising four datasets with five labels in each dataset — we modeled these groupings as controlled random effects. Model appropriateness was verified through assessments of VIF, ICC, and residual normality. In addition, we conducted a qualitative evaluation of AI coder discussion patterns. Using the Thomas-Kilmann Conflict Mode Instrument (Thomas, 2008) and frameworks for discursive moves (Walton, 2008; Zakharov et al., 2021), we compared the behaviors of human and AI discussions. One of our authors iteratively coded the patterns in AI-generated discussions and resolved disagreements. 4.4. Statistical Analysis Results Across all datasets and labels, the IRRs between two AI coders exceeded 0.85. The F1 scores ranged from 0.31 to 0.89 depending on dataset and label, with an average of 0.68 and a standard deviation of 0.16. From this data, we specifically analyzed cases where either one of the two AI coders assigned the ‘Undecidable’ label or where the two coders provided conflicting results, estimating the accuracy of these cases. The average accuracy was 0.60 with a standard deviation of 0.25, highlighting the tradeoff between coverage and accuracy. Lower confidence levels and consensus degrees resulted in reduced accuracy. Full evaluation details are available in the Supplementary Material. The Mixed Effects Model analysis revealed how coding accuracy is influenced by data characteristics and coding dynamics, as summarized in Table 2, 3, 4, and 5. Detailed statistics, including minimum, maximum, average, and standard deviations for the analysis, are available in the Supplementary Materials. 4.4.1. Relationship between Coding Accuracy and Dataset Characteristics Table 2. Dependency on Dataset Characteristics. This table summarizes how different dataset characteristics affect the accuracy of AI agents in initial coding. The summary of the analysis, conducted using a Mixed Effects Model, includes statistical significance, reasons and interpretations of the findings, and connections to relevant literature. Green cell indicates positive, red negative, and gray no relationship with coding accuracy. *: p<0.05, **: p<0.01, ***: p<0.001, ****: p<0.0001. Metric Dataset Characteristic P-value (T-statistic, Effect Size) Relationship Reason and Interpretation Relevant Literature Accuracy of Initial Coding Length of Codebook < 0.0001 **** (-11.702, -0.287) The shorter the codebook, the higher the initial accuracy of automated coding. LLMs’ ability to retrieve and retain relevant information from the input declines as the context length increases. - The performance of LLMs degrades when identifying relevant information within lengthy contexts (Liu et al., 2024; Li et al., 2024c). - As context length increases, the model’s ability to focus on the most relevant information can decrease, leading to a “dilution” of attention across the input (Xu et al., 2023). - To create a good codebook in qualitative data analysis, ensure it is clear and concise (Barany et al., 2024; Ritchie et al., 2022; DeCuir-Gunby et al., 2011). Cosine Similarity of Codebook < 0.0001 **** (-11.738, -0.276) The less similar the semantics of words in each codebook, the higher the initial accuracy of automated coding. Describing code’s meaning in varied language provides multiple perspectives and contexts for the code compared to using overly specific or narrow language. Not Applicable Length of Excerpt < 0.0001 **** (-9.249, -0.438) The shorter the excerpt to code, the higher the initial accuracy of automated coding. Concise data contains less noise from extraneous details, making it easier for AI to focus on the main idea. - The performance of LLMs degrades when identifying relevant information within lengthy contexts (Liu et al., 2024; Li et al., 2024c). - The long and unsegmented text segments made coding decisions more complex (Hruschka et al., 2004). Cosine Similarity of Excerpt < 0.0001 **** (7.372, 0.189) The more similar the semantics of words in excerpt to code, the higher the initial accuracy of automated coding. Limited or consistent topics of data to code make it easier for the AI to identify patterns and relationships within the data. - LLMs perform better in cases where fewer, more concentrated topics allow the model to capture relevant information easily (Doi et al., 2024). Gap of Degree of Specialization Between Codebook and Excerpt < 0.0001 **** (-7.629, -0.331) The smaller the gap of degree of specialization between codebook and excerpt to code, the higher the initial accuracy of automated coding. Text alignment in NLP involves mapping corresponding segments between two texts by measuring their similarity and connecting the most similar parts. Qualitative coding requires similar alignment between the codebook and data. A gap in technical terminology use between the two lowers the similarity score, which can reduce coding accuracy. - Text alignment relies on the assessment of similarity to establish accurate correspondences across different text segments (Kay and Roscheisen, 1993; Zha et al., 2023). - Significant terminological differences between sources — such as technical versus lay language — impair the model’s ability to form coherent personas. Lacking shared features like vocabulary and style, the model struggles to maintain cross-domain accuracy and truthfulness (Joshi et al., 2023). Table 2 illustrates the relationship between dataset characteristics and coding accuracy. Lengthier codebooks, higher cosine similarity within the codebook, lengthier excerpts, and greater gaps in specialization between codebooks and excerpts all negatively affect accuracy. In contrast, higher cosine similarity within the excerpt positively impacts coding accuracy. We observed that concise codebooks improve AI coders’ labeling accuracy, as LLMs struggle to retrieve and retain information with extended input contexts (Table 2, Row 1). This finding aligns with previous studies emphasizing the efficiency of concise inputs for LLMs (Liu et al., 2024; Li et al., 2024c; Xu et al., 2023; Barany et al., 2024; Ritchie et al., 2022; DeCuir-Gunby et al., 2011). One should be cautious in interpreting this finding, as straightforward codes often require less detailed descriptions, whereas complex codes necessitate nuanced, lengthy explanations. This means that the observed high accuracy with concise inputs may partly reflect the complexity of the code content itself, rather than the conciseness alone. To account for this, we incorporated specialized terminology and Word Sense Disambiguation (WSD) to minimize the complexity’s influence. This further analysis mitigated the potential confounding effects of complexity, which our analysis found to be statistically insignificant. The same pattern was also observed in the data to be coded (Table 2, Row 3), where AI coders performed more accurately with concise data. The analysis revealed opposing effects of cosine similarity within the codebook (Table 2, Row 2) versus the data to be coded (Table 2, Row 4). Codebooks employing diverse languages improved accuracy, as variety aids in contextual understanding of the code. Conversely, consistency in topics within the data to be coded improved accuracy, as fewer topics made it easier for the model to capture relevant information easily (Doi et al., 2024). To summarize, coding accuracy improves when codebooks employ diverse language while the target data remains consistent. Since the coding process requires determining whether individual data points fall within codebook categories, this difference in linguistic breadth — varied vocabulary in the codebook and the homogeneity of languages in data — leads to more accurate coding results. It suggests that designing a corpus requires balancing diversity with specificity, depending on the role of each corpus. This balance highlights the complementary nature of broad comprehension (for codebooks) and targeted application (for data). Additionally, drawing parallels from text alignment tasks in NLP domain, which involves mapping corresponding text segments across sources, we explored the impact of similarity alignment between codebooks and data. The result showed that significant disparities in technical terminology diminished similarity alignment, impairing coding accuracy (Table 2, Row 5). This result corroborates existing research showing that differences in terminology reduce LLM efficacy (Kay and Roscheisen, 1993; Zha et al., 2023; Joshi et al., 2023). 4.4.2. Relationship between Coding Accuracy Increase and Labeling Uncertainty Table 3. Dependency on Labeling Uncertainty. This table summarizes how labeling uncertainty among AI agents affects the accuracy improvements achieved through AI agent discussions. The summary of the analysis, conducted using a Mixed Effects Model, includes statistical significance, reasons and interpretations of the findings, and connections to relevant literature. Green cell indicates positive, red negative, and gray no relationship with coding accuracy. *: p<0.05, **: p<0.01, ***: p<0.001, ****: p<0.0001. Metric Labeling Uncertainty P-value (T-statistic, Effect Size) Relationship Reason and Interpretation Relevant Literature Accuracy Increase via Discussions Number of Undecidable Labels in Initial Coding < 0.01 ** (3.274, 0.123) The higher the number of ‘Undecidable’ labels before discussion, the greater the accuracy improvement through AI agent discussions. Dialogue process naturally clarifies uncertainties and builds a precise, shared understanding, thereby improving coding accuracy. Language model performance is enhanced by emphasizing the uncertain data during the training process (Yoo and Kweon, 2019; Settles, 2009). Number of Undecidable Labels in Discussion-Prompted Coding < 0.0001 **** (-14.476, -0.369) The lower the number of ‘Undecidable’ labels after discussion, the greater the accuracy improvement through AI agent discussions. Low frequency of ‘Undecidable’ labels serves as an indicator of the effect of AI agent discussions. Low confidence and agreement among coders are indicators of poor performance (Ganji et al., 2018). Table 3 presents the relationship between coding accuracy improvement through discussions and labeling uncertainty, measured by the frequency of ‘Undecidable’ labels assigned by AI coders. The results reveal a positive relationship between coding accuracy increase via discussions and the ratio of ‘Undecidable’ labels during initial coding before discussions. In contrast, after discussion-prompted coding (i.e., coding performed by LLM coders using a prompt that included the two coders’ discussion process and the new labeling rule derived from it), this trend reverses, showing a negative relationship. The positive relationship suggests that greater ambiguity in the initial coding task enhances the effectiveness of AI agent discussions, as the dialogue process naturally clarifies uncertainties and builds a shared, precise understanding (Table 3, Row 1). These findings corroborate the studies that revealed the positive relationship between clarification of labeling uncertainty and language model performance (Yoo and Kweon, 2019; Settles, 2009). On the other hand, the frequency of ‘Undecidable’ labels persisted following the discussions serves as an indicator of the effect of AI agent discussions — more occurrences indicate a small improvement in accuracy through discussions (Table 3, Row 2). This result parallels findings from human coding studies, where low coder confidence levels often signal poor coding performance (Ganji et al., 2018). 4.4.3. Relationship between Coding Accuracy Increase and Consensus Level of AI Coders Table 4. Dependency on Consensus Among Coders. This table summarizes how consensus among AI agents during coding affects the accuracy improvements achieved through AI agent discussions. The summary of the analysis, conducted using a Mixed Effects Model, includes statistical significance, reasons and interpretations of the findings, and connections to relevant literature. Green cell indicates positive, red negative, and gray no relationship with coding accuracy. *: p<0.05, **: p<0.01, ***: p<0.001, ****: p<0.0001. Metric Consensus among Coders P-value (T-statistic, Effect Size) Relationship Reason and Interpretation Relevant Literature Accuracy Increase via Discussions Number of Conflicting Labels in Initial Coding 0.059 (-1.888, -0.037) No relationship between the number of conflicting labels before discussion and the accuracy improvement through AI agent discussions. There is no evidence that the number of conflicting labels before discussion serves as an indicator of the accuracy improvement through AI agent discussions. Not Applicable Number of Conflicting Labels in Discussion-Prompted Coding < 0.0001 **** (-6.861, -0.147) The lower the number of conflicting labels after discussion, the greater the accuracy improvement through AI agent discussions. Low frequency of conflicting labels serves as an indicator of the effect of AI agent discussions. Low confidence and agreement among coders are indicators of poor performance (Ganji et al., 2018). As shown in Table 4, coding accuracy improvement varies depending on the agreement levels between AI coders. Our analysis revealed that the number of conflicting labels before discussion did not influence the accuracy gains achieved through AI agent discussions (Table 4, Row 1). In contrast, a low frequency of conflicting labels after discussion is indicative of the positive impact of AI agent discussions (Table 4, Row 2). This is consistent with existing literature that identifies low agreement among human coders as a marker of poor qualitative coding accuracy (Ganji et al., 2018). 4.4.4. Relationship between Coding Accuracy and Controversy Level of AI Coders Table 5. Dependency on Controversy in Discussions. This table summarizes how the level of controversy in AI agent discussions affects the accuracy of AI agents when prompted with the discussion process. The summary of the analysis, conducted using a Mixed Effects Model, includes statistical significance, reasons and interpretations of the findings, and connections to relevant literature. Green cell indicates positive, red negative, and gray no relationship with coding accuracy. *: p<0.05, **: p<0.01, ***: p<0.001, ****: p<0.0001. Note that the term “competing/compromising/collaborating mode” relates to the content’s characteristics, not the discussion styles of the AI agents, as we did not adjust the AI agents’ discussion personas. Metric Controversy in Discussions P-value (T-statistic, Effect Size) Relationship Reason and Interpretation Relevant Literature Accuracy of Discussion-Prompted Coding Number of Turns in Discussions < 0.0001 **** (7.997, 0.168) The more controversial the discussions, the higher the accuracy of discussion-prompted coding. Discussions about controversial cases reveal nuanced details that clarify the scope of the code’s application. To create a good codebook in qualitative data analysis, include clear inclusion and exclusion criteria for each code (MacQueen et al., 1998; Ritchie et al., 2022). Ratio of Resolved Conflicts < 0.0001 **** (-6.300, -0.139) The lower the agreement ratio in the discussions, the higher the accuracy of discussion-prompted coding. Debates with disagreements reveal nuanced details that clarify the scope of the code’s application. Ratio of Correctly Resolved Conflicts < 0.0001 **** (-9.720, -0.169) The lower the number of correctly resolved conflicting cases via discussions, the greater the accuracy of discussion-prompted coding. Small number of correctly resolved conflicting cases indicates that the AI agent discussions involved a high proportion of challenging and borderline cases. These discussions about borderline cases reveal nuanced details that clarify the scope of the code’s application. Not Applicable Ratio of Competing Modes in Discussions < 0.01 ** (2.789, 0.054) No relationship between the ratio of competing modes of AI agents in discussions and the accuracy of discussion-prompted coding. There is no evidence that the ratio of competing modes of AI agents in discussions serves as an indicator of the accuracy of discussion-prompted coding. Not Applicable Ratio of Compromising Modes in Discussions < 0.0001 **** (-4.251, -0.066) The less compromising the discussions, the higher the accuracy of discussion-prompted coding. Discussions about controversial cases, which provokes less compromising discussions among AI agents, reveal nuanced details that clarify the scope of the code’s application. - Addressing conflicts in a collaborative, not competitive manner turns disagreements into chances for improved decision-making (Tjosvold et al., 2019). - Collaborating mode is desired in coding discussion, seeking for consensual decision (Thomas, 2008). Ratio of Collaborating Modes in Discussions < 0.0001 **** (-8.420, -0.255) The less collaborating the discussions, the higher the accuracy of discussion-prompted coding. Discussions about controversial cases, which provokes less collaborating discussions among AI agents, reveal nuanced details that clarify the scope of the code’s application. Table 5 illustrates the relationship between accuracy of discussion-prompted coding and the degree of controversy between AI coders. Accuracy was positively related to the number of discussion turns, with contentious disagreements extending debates and leading to an increase in turns. In contrast, accuracy showed a negative relationship with the ratio of resolved conflicts, correctly resolved conflicts, as well as discussions conducted in compromising and collaborating modes. There was no significant relationship observed with the ratio of competing modes. The high coding accuracy observed in controversial discussions can be attributed to the nuanced understanding generated during debates. Lengthier debates allowed for clarification of the inclusion and exclusion criteria for each code, aligning with principles for designing effective codebooks by exposing clear inclusion and exclusion criteria (MacQueen et al., 1998; Ritchie et al., 2022) (Table 5, Row 1). Similarly, these effects were also prominent in discussions without consensus (Table 5, Row 2). Interestingly, a smaller number of correctly resolved conflicts correlated with higher accuracy, indicating that debates about challenging or borderline cases uncovered subtle details that revealed the scope of code application (Table 5, Row 3). This underscores that the value of engaging in nuanced debates may outweigh the importance of achieving correct resolutions, as the process itself drives a deeper understanding of ambiguous cases. Contrary to findings from human discussions, where collaborative modes are typically desired, which provide emotional support and psychological safety (Tjosvold et al., 2019; Thomas, 2008), less compromise and collaboration among AI coders produced better results (Table 5, Rows 5 and 6). This discrepancy can be explained by the nature of LLM agents, which operate free of emotional and social dynamics. Competitive discussions, which might induce stress in humans, function as optimization tools for LLMs as they can drive more rigorous refinement of reasoning. Additionally, LLMs typically share the same foundational architecture and training data, thereby sometimes generating similar opinions among agents. Accordingly, collaborative modes can lead to overly convergent answers and reinforce biases (Anderson et al., 2024; Chuang et al., 2023; Bano et al., 2023b). 4.5. Qualitative Observation of AI-Agent Discussions Table 6. Comparison of Discursive Moves Between LLM Coders and Humans. This table summarizes the discursive moves observed in LLM coders’ discussions to those reported for humans in existing research. The table also includes detailed descriptions and examples for each move. Discursive Move of LLMs Description Discursive Move of Humans in Literature Examples from AI Agent Discussions Stance Maintenance Reaffirming a viewpoint to emphasize commitment to a particular perspective. Clarification (Grice, 1990; Bakhtin, 2010; Chan et al., 2023) “I maintain that it should be labeled as 1 for the presence of a check question.” “However, I still believe it primarily serves a rhetorical purpose rather than acting as a direct check question.” “I still assert that the claim fundamentally advocates for the inclusion of minority rights within a multicultural framework.” Acknowledgment Before Disagreement Acknowledging the opposing coder’s viewpoint before clearly expressing disagreement. Tolerant Divergence (Felton et al., 2009; Lu et al., 2011; Angouri and Tseliga, 2010; Cheng et al., 2017), Partial Concurrence (Felton et al., 2009; Lu et al., 2011; Angouri and Tseliga, 2010; Cheng et al., 2017) “I understand Coder 2’s concerns about the nuances of the phrase … However, I still assert that this utterance functions as a check question.” “I understand your perspective, but I would argue that the mere presence of poor conditions alone does not meet the threshold of a violation of the right to life as defined in Article 2.” “While I acknowledge Coder 2’s point regarding ‘severe cardiorespiratory collapse,’ it is essential to clarify that this term alone does not constitute a cardiovascular disease as defined by the provided description, which focuses on chronic or specific conditions related to the circulatory system.” Softening Language Using softer and cooperative language to maintain a positive and constructive atmosphere. Affirming Communication (Felton et al., 2009; Lu et al., 2011; Angouri and Tseliga, 2010; Cheng et al., 2017) “I appreciate Coder 2’s reevaluation and the acknowledgment of ambiguity.” “Coder 1’s assessment correctly identifies it as a form of agreement that confirms the sentiment previously discussed.” Reassessing Evidence Revisiting the description of code, often mentioning specific lines to reinforce their position or make it more convincing. Referenced Argumentation (Felton et al., 2009; Lu et al., 2011; Angouri and Tseliga, 2010; Cheng et al., 2017) “According to the requirements of D01, the utterance must specifically ask a check question.” “The right to life [which is one of the codes in the codebook] encompasses not only protection from direct life-threatening actions but also the provision of a safe and healthy environment.” “The essence of Article 5 [which is one of the codes in the codebook] is focused on actual deprivation of liberty through arrest or detention.” Direct Challenge Confronting the opposing coder’s perspective head-on, with clear language indicating a strong contrasting view. Explicit Rejection (Felton et al., 2009; Bar-Haim et al., 2017; Locher, 2004), Constructive Disagreement (Felton et al., 2009; Bar-Haim et al., 2017; Locher, 2004) “Coder 1’s interpretation of the question as a check question misrepresents the essence of what constitutes a check question.” “Coder 1’s perspective overlooks this critical relationship, which justifies the labeling of Wilson’s disease under digestive system diseases.” Alternative Proposal Suggesting an alternative viewpoint. Alternative Proposal (Bakhtin, 2010; Parker, 2006; Chan et al., 2023), Reframing Perspective (Bakhtin, 2010; Parker, 2006) “Therefore, it might be better categorized as an open-ended question [a third alternative option distinct from the views of Coder1 and Coder2] instead of a clear check question.” “Therefore, I propose labeling it as undecidable case [a third alternative option distinct from the views of Coder1 and Coder2] due to this ambiguity.” Tentative Stance Expressing an opinion with caution, acknowledging ambiguity or lack of certainty. Tentative Expression (Felton et al., 2009; Lu et al., 2011; Angouri and Tseliga, 2010; Cheng et al., 2017) “Thus, it is ambivalent; it does not definitively confirm or refute the existence of guiding principles.” “However, the ambiguity surrounding the intention behind "right?" makes it difficult to firmly categorize this utterance as either a check question or not.” “The lack of clear legal standards or precedents regarding the threshold for inhumane treatment in this context leaves room for interpretation.” Seeking Consensus Showing willingness to align or compromise with others. Mutual Agreement (Teasley et al., 2008; Diziol et al., 2010; Schwarz et al., 2018; Chan et al., 2023; Zhang et al., 2023c) “I agree with Coder 2’s revised assessment that the case presents complexities that make it difficult to categorize definitively under Article 5.” “I concur with Coder 1’s conclusion that this utterance should be labeled as 0.” Collaborative Reflection Explicitly referencing how the interaction with the other coder’s viewpoint has led to a new understanding or shift in thought. Dual Perspective Recognition (Felton et al., 2009; Lu et al., 2011; Angouri and Tseliga, 2010; Cheng et al., 2017) “After considering Coder 1’s perspective that the utterance does not ask a check question and instead presents a rhetorical comparison, …” “Upon reevaluating the concerns raised in Coder 1’s position, I acknowledge that hepatoblastoma, being a malignancy originating in the liver, is indeed related to the digestive system as the liver plays a critical role in digestion and metabolism.” Redundant Restatement Restating an earlier argument without introducing new information or distinct perspectives. Repetition (Zakharov et al., 2021) Coder 1: For the code “Person” to be applicable, there should be a sense of emotional resonance or significance linked to that person in relation to the music, which is absent here. Coder 2: … Coder 1: For the “Person” code to apply, there needs to be a layer of emotional resonance or significance that connects the person to the music, which is not evident in this instance. Explicit Contrast Directly contrasting their own reasoning with the other coder’s point, highlighting the differences before affirming a stance. Arguments by Comparison (Walton, 2008) “While Coder 1 argues it lacks the explicit structure of a check question [the other coder’s opinion], it does in fact seek clarification about the action [their own opinion].” “The utterance ‘Could you clarify what you mean by ‘having the waves together’?’ does not ask a check question [the other coder’s opinion]; instead, it requests clarification [their own opinion].” Reflective Reconsideration Indicating a review of prior discussion before concluding. Partial Concurrence (Felton et al., 2009; Lu et al., 2011; Angouri and Tseliga, 2010; Cheng et al., 2017) “After reconsidering the earlier discussion …” “After reviewing both opinions, …” “After reflecting on the ongoing discussion, …” We observed the discussion behaviors of LLM agents without additional fine-tuning or instruction about discursive moves and compared them with human discursive moves, providing a baseline that other researchers can expand and tailor for diverse applications. Table 6 contains the comparison between human discursive moves from the literature and the ones that are observed from AI coders’ discussions. The discursive moves that are absent from the AI coders but still used by humans are organized in the table in Appendix A. LLMs replicate many of the discussion patterns of human coders, though key distinctions remain evident. For example, LLMs emulate basic patterns of interaction, such as maintaining stances, acknowledging others’ perspectives, reassessing evidence, explicitly contrasting ideas, expressing a tentative stance, and proposing alternatives. These behaviors highlight their capacity for logical and systematic reasoning. Additionally, LLMs are capable of supporting essential aspects of constructive discussions, such as fostering collaborative reflection and seeking consensus. However, certain key aspects of human discursive behavior are absent in LLMs. Human discussants actively influence the flow and direction of discussions through dynamic interactions, such as raising questions to address uncertainties (e.g., clarifying ambiguities or challenging evidence), expanding on others’ ideas, sharing personal experiences, and moderating discussions based on evolving conversational contexts. These actions reflect humans’ ability to adjust their communication based on subtle shifts in the discussion and the reactions of other participants. In contrast, LLMs tend to operate within a more fixed framework, often missing the fluidity and adaptability required to respond to implicit cues or evolving conversational dynamics. Moreover, LLMs do not replicate the emotional expressiveness inherent in human discourse. Human discussants often incorporate sarcasm and complaints, which can disrupt the focus of discussions or lead to unnecessary tension. This absence of emotionally charged or overly critical tones in LLMs can sometimes foster a more neutral and focused environment for logical reasoning. Instead, agents use softening language to foster a cordial atmosphere during the discussion. However, LLMs do share certain non-constructive discursive behaviors with humans, such as redundant restatement, which reiterates their earlier arguments without introducing new perspectives. Meanwhile, humans use rhetorical strategies, such as analogies, to enhance persuasiveness and concretize abstract concepts. By comparison, while LLMs are proficient in generating well-structured and logical narratives, they frequently lack the creative subtlety needed to craft compelling narratives. It is worth noting that LLMs engage in significantly less questioning compared to humans. While humans frequently ask clarifying or probing questions to refine understanding or actively explore the perspectives of other participants, LLMs tend to prioritize delivering coherent and direct answers over engaging in exploratory dialogue and mutual learning. The differences between humans and LLMs are not limitations but complementary features that can be harnessed for mutual reinforcement in collaborative settings. Researchers can design human-AI collaborations by integrating the structured and efficient reasoning of LLMs with the creativity, emotional intelligence, and contextual awareness of humans. The discussion dataset we created enables the evaluation of LLMs’ baseline discussion capabilities without fine-tuning, offering a foundation for guiding and benchmarking automated qualitative coding methodologies. 5. Design Recommendations for Automated Qualitative Coding Systems Based on results from LLM experiments and data analysis, we propose several design recommendations for developers and designers of automated qualitative coding systems. The quantitative findings of experiments, including mean, standard deviation, minimum, and maximum values, are available in the Supplementary Materials. However, the primary aim of design recommendations is not to specify rigid thresholds but to offer a conceptual and methodological foundation upon which future research may further extend by focusing more directly on quantitative metrics. 5.1. Design Recommendation for Designers 1: Provide users with guidance on preprocessing their datasets to optimize system performance. One key recommendation is that designers should provide users with targeted guidance on how to preprocess their datasets to optimize system performance. This includes embedding features in the system that offer diagnostic feedback on dataset properties and provide actionable suggestions. For instance, systems can notify users when certain data characteristics are likely to result in reduced performance, such as when particular data for coding or code definitions may yield lower accuracy. In such cases, users can be advised to allocate additional scrutiny to the associated segments, or the system can generate preprocessing recommendations accordingly. The example suggestions may address codebook development such as encouraging concise codebook creation. Below are representative examples from the data analysis. <Example of Short Codebook> To be labeled as “Article03”, the court case should violate the Article03. Article03: Prohibition of torture No one shall be subjected to torture or to inhuman or degrading treatment or punishment. <Example of Long Codebook> To be labeled as “Article06”, the court case should violate the Article06. Article06: Right to a fair trial 1. In the determination of his civil rights and obligations or of any criminal charge against him, everyone is entitled to a fair and public hearing within a reasonable time by an independent and impartial tribunal established by law. Judgment shall be pronounced publicly but the press and public may be excluded from all or part of the trial in the interests of morals, public order or national security in a democratic society, where the interests of juveniles or the protection of the private life of the parties so require, or to the extent strictly necessary in the opinion of the court in special circumstances where publicity would prejudice the interests of justice. 2. Everyone charged with a criminal offence shall be presumed innocent until proved guilty according to law. 3. Everyone charged with a criminal offence has the following minimum rights: … Another recommendation for the codebook can include avoiding repetitive or circular code definitions, which result in a high similarity between words within a codebook, and instead employing multiple, distinct descriptions that have a low similarity between words within a codebook. <Example of Low Similarity Codebook> To be labeled as “Person”, the respondent should associate the track with a person such as a family member, a friend, the loss of a person, a group of people, or not being with a person or being alone. <Example of High Similarity Codebook> To be labeled as “Negative Evaluations”, the respondent should make a negative evaluation about the track or an aspect of the track. Users may also be prompted to split data for coding into chunks of one to two sentences. <Example of Short Excerpt> So do you know what virtual reality is? <Example of Long Excerpt> This is a great question because it really gets to the difference between some of our different tools in our machine learning tool belt in addressing problems like this. So if we were to use a supervised learning classic classification approach, a person would need to think about those features and creatively come up with them in approach we call the kitchen sink approach, which is just try everything you can possibly think of and see what works. The qualitative data for coding should then be organized around coherent or limited topics using NLP techniques such as TextTiling. <Example of Low Similarity Excerpt> Slightly like slipknot. It reminds me of the game “need for speed underground 2” a little. The repetitive riffs make me think of grinding the same missions or tasks at work or in certain games. There is a certain anger, or energy brought by the vocals that makes me knuckle down, but the guitar is mostly happy and upbeat. I went through a phase of listening to the GazettE, but I still think it’s good. Energetic and upbeat. I don’t get too distracted by the lyrics as I don’t speak Japanese. It helps me focus and power through. Nope, it’s just a song I quite like. <Example of High Similarity Excerpt> A sense of calmness and serenity. Because it puts my life into perspective when I feel anxious or worried about something. Because it calms me down and makes me think of only positive things, all negative things just fall away. Yes, it’s a Christian track, so when I play it, it makes me think of God and his promises, so it gives me a perspective and reminder that I am loved. Furthermore, designers should encourage users to match the degree of specialization in their codebooks with that in their datasets. 5.2. Design Recommendation for Designers 2: Allow users to leverage (1) the “Undecidable” label and (2) the conflicting label as a performance indicator to predict potential accuracy improvements resulting from AI agent discussions. Beyond preprocessing, systems should enable users to understand and use performance signals. In particular, the “Undecidable” and conflicting labels should be made interpretable as indicators of potential accuracy gains via AI agent discussions. For example, users can be guided to interpret the number of uncertainty prior to AI agent discussions as a potentially positive sign, and the number of uncertain labels after discussion as a negative indicator of limited benefit from the process. Likewise, persistent conflict between agents following discussions should be treated as a potential signal of limited effects of AI agent discussions. Incorporating such diagnostic tools allows users to develop informed mental models of the system’s behavior and adjust their expectations. 5.3. Design Recommendation for Designers 3: Allow users to leverage the level of controversy within AI agent discussions as a performance indicator for evaluating the expected accuracy of discussion-prompted coding. Another major recommendation addresses the value of the degree of agent disagreement. While disagreements might be intuitively understood as limitations, our findings suggest that they are often more informative than their resolution. Designers should reconceptualize agent disagreement during coding discussions not as a failure but as a signal of interpretive richness. Our analysis indicates that prolonged or unresolved disagreements between agents often yield clarification of coding criteria and foster a more nuanced understanding of edge cases. In some cases, the resolution of disagreements may correspond with low information gain, suggesting that users should capitalize on the presence and structure of disagreement rather than its resolution alone. Specifically, in gauging the expected accuracy of discussion-prompted coding, designers can guide users to interpret high volumes of conversational exchange between agents as a positive sign while viewing high proportions of resolved or even correctly resolved conflicts as potential negative indicators. Below are representative examples from the data analysis. <Example of Controversial Discussion> … Coder 1: I appreciate Coder 2’s interpretation that “what about negative harmony” could serve as a discussion prompt. However, the nature of the phrase “what about” often leans towards rhetorical questioning rather than a genuine inquiry seeking specific information. … Coder 2: After considering Coder 1’s arguments, I maintain that the phrase “what about negative harmony” does indeed present a question. While Coder 1 suggests it functions more as a prompt, I argue that it invites exploration of a specific topic, which aligns with the essence of D02. … Coder 1: I appreciate Coder 2’s perspective that “what about negative harmony” could be seen as an invitation for discussion; however, I still contend that it lacks the qualities of a genuine inquiry. … Coder 2: I understand Coder 1’s view regarding the rhetorical nature of “what about negative harmony,” yet I continue to assert that it functions as an inquiry. … <Example of Non-Controversial Discussion> … Coder 1: While I understand Coder 2’s perspective that the excerpt discusses potential and challenges, I believe it still conveys specific information about opportunities in synthetic worlds and the limitations of current simulation technologies. … Coder 2: Upon reevaluating my position and considering Coder 1’s reasoning, I recognize that the utterance does indeed convey substantial information about the opportunities and limitations involved in creating synthetic fantasy worlds. Phrases like “there’s tons of opportunities for the synthetic case” and “we still don’t do people well simulated” provide clear insights into the current state of technology in this area. Therefore, I agree with Coder 1 that this aligns with the requirements for D09, as it effectively informs the listener about specific aspects of the topic. 5.4. Design Recommendation for Designers 4: Adjust AI agents’ discussion style based on the intended objectives of the coding task. A final recommendation focuses on discussion dynamics and the adjustment of agent behaviors according to the objectives of the coding task. Designers may consider providing users with the option to activate a discussion mode that deliberately introduces controversy in tasks where ambiguity is prevalent in the dataset. When a label is undecidable, confidence is low, or prior performance has been unreliable, the system can shift into a debate-enhancing mode that fosters disagreement, prompting agents to adopt explicitly opposing perspectives. Our findings suggest that introducing structured opposition within agent discussion improves the accuracy of discussion-prompted coding. <Example of Discussion with Competing Mode> Upon further reflection, I maintain my stance that the excerpt contains positive evaluations. … While Coder 1 argues these are not explicit evaluations, they do reflect a positive judgment about the track and its context, fulfilling the criteria for Positive Evaluations as outlined in the description. <Example of Discussion with Compromising Mode> While I understand Coder 2’s perspective regarding closed head injury being a significant condition, I believe we are dealing with an undecidable case here. … Therefore, based on the provided data, it is difficult to definitively categorize it as either present or absent under this specific label [, suggesting the Undecidable label as an alternative]. <Example of Discussion with Collaborating Mode> Upon reevaluating Coder 1’s reasoning, I recognize that the term “nostalgic” serves as a descriptive adjective that evokes specific emotional qualities associated with the track. … Therefore, I change my initial assessment to agree with Coder 1. 6. Discussion This section examines the key findings of our work, acknowledges the study’s limitations, and outlines directions for future research. 6.1. Summary of Key Findings and Implications Our study reveals that LLM “coders” can engage in a structured dialogue to perform qualitative coding, with performance influenced by specific dataset characteristics and discussion dynamics. First, we found that coding accuracy is shaped by dataset’s length and semantic similarity, and the gap of degree of specialization between codebook and data for coding. These findings highlighted that preprocessing codebooks and qualitative data for coding can boost AI coding performance. Second, our results show that the frequency of “Undecidable” label and conflict between AI agents can be utilized as a performance indicator. These signals assist users in calibrating their expectations of automated systems. We discovered that instances of disagreement and debate between the AI coders can be a positive signal for high accuracy of coding. Rather than viewing disagreement as a failure, our study indicates that when the two AI coders disagreed on how to code a segment, the conversation often increased interpretive depth. The AI’s convergence is a double-edged sword: on one hand, it led to consistent consensus in most cases, but on the other hand, we suspect it might sometimes mask uncertainty. For instance, in a few discussions we examined, one agent proposed a code interpretation that the other accepted quickly. Such instances resulted in a consensus label that both agents agreed on, yet the underlying ambiguity of the data might not have been fully resolved, leading to an incorrect labeling outcome. This behavior underscores a design consideration on how to interpret automated outcomes: simply because two AI agents agree after a short discussion does not guarantee the decision was correct or well-founded. Furthermore, prolonged exchanges were associated with a more nuanced understanding of edge cases, whereas quick resolutions sometimes signaled that less information was gained through discussions. Together, these findings ground our discussion in concrete evidence from the experiment, moving beyond abstract statements to actionable recommendations for design. Third, our qualitative analysis of LLM discourse behavior versus human coder discussions uncovered both parallels and gaps. On one hand, the AI agents replicated many constructive discussion patterns observed in human teams. They maintained or defended their initial stances, acknowledged each other’s perspectives, contrasted differing interpretations explicitly, reassessed evidence, and proposed alternatives. These behaviors suggest that LLMs are capable of logical, systematic reasoning. On the other hand, we identified several human discussion strategies that were absent or weaker in the AI conversations. For example, human coders often interject questions to clarify uncertainties, challenge assumptions, or seek additional evidence; in our study, the LLM agents did not engage in clarifying or probing questions. Humans also dynamically adjust the discussion flow, moderate the direction of debate, share personal anecdotes, and resort to figurative language and rhetorical strategies to enhance engagement and clarity. In contrast, the AI discussions followed a more fixed pattern, not fully adapting to nuanced conversational cues or shifting strategy mid-discussion. Emotional or stylistic nuances present in human dialogue, such as sarcasm or frustration, were essentially absent from the AI’s exchanges. This lack of emotional reactivity in LLMs kept the tone neutral and focused, potentially creating a more consistently cordial environment for deliberation. Meanwhile, the AI agents did exhibit some less productive tendencies akin to humans, such as repetitive restatement of points without adding new information. In summary, LLM-based coders can mirror basic human reasoning moves but miss the adaptive and inquisitive depth of human interaction, suggesting that while LLMs excel at structured reasoning, they may fall short in demonstrating the more affective and socially attuned aspects of human discourse without additional fine-tuning. 6.2. Limitations While our study provides encouraging evidence for LLM-based qualitative coding, several limitations temper our conclusions. First, the AI model we used was not fine-tuned for qualitative coding tasks or any domain-specific data. All agent behaviors were elicited through prompting alone. This choice allowed us to examine out-of-the-box capabilities, but it also means performance might be improved with customization. A fine-tuned model or one explicitly instructed in effective discussion strategies could produce different (potentially better) outcomes. Second, the evaluation of coding accuracy and consensus has inherent challenges. We treated the original human-coded dataset labels or expert consensus as the “ground truth” for accuracy calculations. However, in qualitative research, coder disagreements can sometimes reflect multiple valid interpretations rather than clear-cut errors. Our use of accuracy as a metric, while necessary for quantitative analysis, oversimplifies this nuance. An AI-assigned code different from the ground truth was counted as incorrect, even if it might be a justifiable interpretation in context. Third, the rapid pace of LLM development introduces another layer of complexity. New releases of LLMs may significantly alter the dynamics of agent discussions, potentially changing both the quality and style of coding collaboration. This evolving baseline makes it challenging to draw stable conclusions from experiments tied to a particular model version, and highlights the need for ongoing evaluation as the technology advances. Meanwhile, although our experiments were tied to GPT-4o-mini, the underlying dynamics — such as the tendency for accuracy gains to emerge from prolonged debate rather than rapid consensus, with controversy about borderline cases clarifying inclusion and exclusion criteria for codes and — represent model-agnostic mechanisms. These dynamics are likely to persist across LLM families, since they stem from the interactional structure of multi-agent debate rather than from model-specific capabilities. At the same time, given the rapid evolution of LLMs, we frame our findings less as static benchmarks and more as design-oriented insights that persist across model generations. Finally, our study setting was a simulated one: AI agents discussed with each other, and we compared their behavior to documented human coding discussions from literature. We did not conduct a user study of human analysts actually working with these AI agents. As a result, we have not directly measured human researchers’ trust, satisfaction, or workflow integration when partnering with AI coders. These human factors are crucial for real-world adoption and were beyond our current scope. The absence of direct human-in-the-loop experimentation is a limitation that we acknowledge and aim to address in future work. 6.3. Theoretical Implications: Rethinking Reliability and Agreement Beyond practical design, our work invites HCI and computational social science researchers to reconsider some theoretical assumptions in qualitative research when one of the coders is an AI. In traditional qualitative coding, intercoder reliability (e.g., Cohen’s Kappa) between human coders serves as a check on consistency and code clarity. But what does reliability mean when a coder is an AI? We found that measuring agreement between two AI coders was useful for our analysis, yet it differs from human scenarios. For one, AI coders (especially if they are instances of the same LLM model) may have correlated errors or shared biases, violating the assumption of independent judgments. In our study, both AI coders were based on the same underlying model architecture and were exposed to the same training data distribution, which is unlike two independent human coders with distinct backgrounds. This means a high agreement between AI coders might not guarantee the validity of the coding — they could be consistently wrong in the same way, implying that reliability statistics need reinterpretation Inter-rater reliability with an AI partner could be overly optimistic (if the AI’s decisions are rubber-stamped by a human or vice versa) or overly pessimistic (if the AI systematically challenges a human coder on edge cases that humans might ordinarily skip as noise). One theoretical extension suggested by our work is to develop new metrics that account for the discussion process. Instead of just percent agreement on final codes, one could measure the proportion of cases that required discussion to reach agreement, or the stability of the AI’s decisions before and after debate. These measures might capture the nuance of AI coding interactions better than a single agreement coefficient. Additionally, we propose rethinking the role of codebooks and coding guidelines for AIs. When an AI is involved, perhaps the codebook should be written not just for human interpretability but also optimized for machine interpretability. Another implication is how we define consensus. In human teams, consensus is often the end of discussion; in human–AI teams, we might deliberately choose not to fully automate consensus. It could be acceptable, or even desirable, to have some lasting disagreements, if those indicate areas for further human analysis. Theorists and practitioners should be open to the idea that a discord between AI coder or between a human and an AI coder could spark deeper insights. Meanwhile, the distinction between what is ‘reasonable practice’ and what is ‘inherent to qualitative analysis’ illuminates a question: is qualitative coding reducible to a standardized analytic procedure, or is it constitutively interpretive? We take the latter view. The presence of ambiguity and disagreement is not a byproduct of methodological insufficiency of imperfect method but a constitutive condition of qualitative knowledge production. Our contribution is to formalize its role in computational pipelines. By operationalizing disagreement and undecidability, we reposition what qualitative research has treated as unavoidable and generative into a measurable feature that informs the design of automated systems. In summary, our research calls for an expanded notion of reliability and consensus in coding when integrating AI – one that values the process of reaching agreement (or not reaching it) as much as the final agreement itself. 6.4. Design Implications: LLM-Based Coding Systems Our findings suggest several important design implications for systems that incorporate LLMs into qualitative coding workflows. One key design recommendation is to offer tunable AI behavior modes, specifically a toggle between a “strict” mode and a “debate” mode. In the debate-enhancing mode, the system could detect low-confidence scenarios — such as undecidable cases or potential coding conflicts — and then trigger divergent perspectives among AI agents. By encouraging structured disagreement or devil’s advocacy, the system can prompt deeper analytical engagement and help surface nuanced considerations that may be lost in overly harmonious exchanges. Our results indicate that such structured debate is especially beneficial for ambiguous or complex coding situations, improving the overall accuracy and depth of analysis. On the other hand, for routine or well-defined coding tasks, a concise exchange may be preferable. In these scenarios, a “strict” mode could streamline interactions. This dynamic control over the discussion mode offers alignment with the user’s task goals. For example, a complicated qualitative analysis might benefit from competitive and discursive outputs clarifying challenging cases, whereas quick coding for a simple survey might prefer efficiency. While our study focused on AI–AI coder pairs, a future direction is the integration of human coders directly into the discussion loop. A human-in-the-loop design could better align the coding process to users’ goals and intent by allowing human users to guide and challenge AI-generated codes in real time. Qualitative coding has long been theorized as more than a procedural act of labeling; it is an interpretive practice in which researchers mediate between data, theory, and their own positionality. Reflexivity — acknowledging how one’s standpoint and assumptions shape interpretation — is central to ensuring validity in this tradition. While our study demonstrates that LLMs reproduce discursive moves reminiscent of human coders (e.g., stance maintenance, acknowledgment, tentative stances), these moves operate as surface-level simulations of interpretive dialogue. They do not embody the meta-cognitive work of interrogating one’s assumptions, nor the positional disclosure that grounds qualitative validity. As a result, AI “agreement” may project an illusion of rigor while bypassing the epistemic labor that human researchers contribute through reflexive engagement. For automated qualitative coding, this implies that validity must be reconceptualized as a co-constructed property of human – AI interaction. AI coders can highlight ambiguity, surface discursive conflicts, and generate systematic consistency, but reflexivity remains the province of human researchers. Rather than viewing LLM consensus as a substitute for interpretive rigor, we argue that its true value lies in provoking reflexive engagement: Why did the models disagree? What kinds of ambiguity do their debates reveal? How does a researcher’s theoretical stance shape the interpretation of such ambiguity? Designing systems that scaffold these questions — by embedding reflexive checkpoints, prompting positional notes, and visualizing where AI outputs diverge from human interpretive expectations—can transform automation from a tool of decontextualization into a partner in reflexive inquiry. In this way, the uniquely human aspects of coding are not displaced but re-centered, with AI disagreements serving as catalysts for deeper interpretive practice. We also encourage longitudinal studies and domain-specific deployments of LLM-based coding to see how our findings hold in real-world practice. For instance, deploying an AI coding assistant in a social science research lab for several months could uncover how human coders adapt their strategies when an AI is part of the team — do they come to trust the AI for certain codes more than others? Does the presence of an AI change how teams construct their codebook? Different domains (e.g., healthcare interviews vs. Twitter data vs. HCI user study feedback) may present unique challenges for LLM coders. Future work should examine a variety of contexts, potentially leading to specialized design guidelines. Several directions emerge from our study that could extend and deepen this line of inquiry: • Richer analysis dimensions: Future studies could vary the number of AI agents, assign them distinct personas or expertise areas, and experiment with multiple iterations of discussion to observe how arguments evolve or converge over time. • Tracking dynamics of iterative discussions: Our study involved a single iteration of discussion. Meanwhile, investigating how the structure and tone of AI discussions change across multiple iterations could offer insights into discussion quality and decision-making patterns. • Customized discussion behaviors: Based on observed behaviors, systems could adapt the agents’ conversational styles or debate strategies to match the user’s preferences or the coding context. • Human-in-the-loop pipeline design: Designing workflows where human coders participate proactively — adding feedback to AI debates, inserting their own counterpoints, or steering the discussion toward specific frames — can make the process better align with the user’s intent and goals. Future tools could allow humans to reflect their opinions within AI debates, not just through final selections but as part of the ongoing discussion, enabling richer co-creation and mutual learning between humans and AI. 7. Conclusion Our work contributes a foundational step toward understanding and improving automated qualitative coding using LLMs. By simulating coder discussions, examining the impact of data characteristics, and analyzing AI behaviors qualitatively, we provide a nuanced account of the strengths and limitations of LLMs in replicating human coding dynamics. Our findings emphasize that disagreement and ambiguity are not merely challenges but opportunities for deeper insight, and they should be strategically harnessed in system design. From these findings, we propose a set of design recommendations for designers: maintain concise and diverse codebooks, structure excerpts around coherent topics, use coding uncertainty like “Undecidable” labels as diagnostic tools for coding accuracy, and tune agent discussion style to suit task goals. References Agresti et al. (2000) A. Agresti, J. G. Booth*, J. P. Hobert*, and B. Caffo* Random-effects modeling of categorical response data. Sociological Methodology 30 (1), p. 27–80. Cited by: §4.3. Amani et al. (2025) S. Amani, L. White, T. Balart, K. Watson, and K. Shryock Applying generative ai for thematic analysis: a model for researchers exploring qualitative methods. Available at SSRN 5128905. Cited by: §3.3.2, Table 1, Table 1, Table 1. Anderson et al. (2024) B. R. Anderson, J. H. Shah, and M. Kreminski Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th Conference on Creativity & Cognition, p. 413–425. Cited by: §4.4.4. Angouri and Tseliga (2010) J. Angouri and T. Tseliga “You have no idea what you are talking about!” from e-disagreement to e-impoliteness in two online fora. Cited by: Table 6, Table 6, Table 6, Table 6, Table 6, Table 6. Anis and French (2023) S. Anis and J. A. French Efficient, explicatory, and equitable: why qualitative researchers should embrace ai, but cautiously. Business & Society 62 (6), p. 1139–1144. Cited by: Table 1, Table 1. Arlinghaus et al. (2024) C. S. Arlinghaus, C. Wulff, G. W. Maier, C. Arlinghaus, C. Wulff, and G. Maier Inductive coding with chatgpt-an evaluation of different gpt models clustering qualitative data into categories. Open Science Framework. https://doi. org/10.31219/osf. io/gpnye. Cited by: Table 1. Ashwin et al. (2023) J. Ashwin, A. Chhabra, and V. Rao Using large language models for qualitative analysis can introduce serious bias. arXiv preprint arXiv:2309.17147. Cited by: §2.1, Table 1, Table 1. [8] A. H. Atas, B. Celik, F. Balzan, and B. Shahrokhian Automating thematic analysis with multi-agent llm systems. Cited by: Table 1, Table 1. Bakhtin (2010) M. M. Bakhtin The dialogic imagination: four essays. University of texas Press. Cited by: Table 7, Table 7, Table 6, Table 6. Bano et al. (2024) M. Bano, R. Hoda, D. Zowghi, and C. Treude Large language models for qualitative research in software engineering: exploring opportunities and challenges. Automated Software Engineering 31 (1), p. 8. Cited by: Table 1, Table 1. Bano et al. (2023a) M. Bano, D. Zowghi, and J. Whittle AI and human reasoning: qualitative research in the age of large language models. The AI Ethics Journal 3 (1). Cited by: Table 1, Table 1. Bano et al. (2023b) M. Bano, D. Zowghi, and J. Whittle Exploring qualitative research using llms. arXiv preprint arXiv:2306.13298. Cited by: §2.2, §3.3.2, Table 1, Table 1, §4.4.4. Bar-Haim et al. (2017) R. Bar-Haim, I. Bhattacharya, F. Dinuzzo, A. Saha, and N. Slonim Stance classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, p. 251–261. Cited by: Table 6. Barany et al. (2024) A. Barany, N. Nasiar, C. Porter, A. F. Zambrano, A. L. Andres, D. Bright, M. Shah, X. Liu, S. Gao, J. Zhang, et al. ChatGPT for education research: exploring the potential of large language models for qualitative codebook development. In International conference on artificial intelligence in education, p. 134–149. Cited by: §1, §2.2, §4.4.1, Table 2. Basit (2003) T. Basit Manual or electronic? the role of coding in qualitative data analysis. Educational research 45 (2), p. 143–154. Cited by: §2.2. Bennis and Mouwafaq (2025) I. Bennis and S. Mouwafaq Advancing ai-driven thematic analysis in qualitative research: a comparative study of nine generative models on cutaneous leishmaniasis data. BMC Medical Informatics and Decision Making 25 (1), p. 1–14. Cited by: Table 1. Bijker et al. (2024) R. Bijker, S. S. Merkouris, N. A. Dowling, and S. N. Rodda ChatGPT for automated qualitative research: content analysis. Journal of medical Internet research 26, p. e59050. Cited by: §2.1, §3.3.2, Table 1. Breland (1996) H. M. Breland Word frequency and word difficulty: a comparison of counts in four corpora. Psychological Science 7 (2), p. 96–99. Cited by: §4.3. Brent and Slusarz (2003) E. Brent and P. Slusarz “Feeling the beat” intelligent coding advice from metaknowledge in qualitative research. Social Science Computer Review 21 (3), p. 281–303. Cited by: Table 1. Brondani et al. (2024) M. Brondani, C. Alves, C. Ribeiro, M. M. Braga, R. C. M. Garcia, T. Ardenghi, and K. Pattanaporn Artificial intelligence, chatgpt, and dental education: implications for reflective assignments and qualitative research. Journal of Dental Education 88 (12), p. 1671–1680. Cited by: Table 1. Bryda and Sadowski (2024) G. Bryda and D. Sadowski From words to themes: ai-powered qualitative data coding and analysis. In World Conference on Qualitative Research, p. 309–345. Cited by: §2.1, §2.2, §3.3.2, Table 1, Table 1. Cameron (2025) H. Cameron Breaking up with nvivo: ai delivers faster, deeper, and better qualitative insights. Cited by: Table 1. Carius and Teixeira (2024) A. C. Carius and A. J. Teixeira Artificial intelligence and content analysis: the large language models (llms) and the automatized categorization. AI & SOCIETY, p. 1–12. Cited by: §2.2, Table 1. Chalkidis et al. (2021) I. Chalkidis, M. Fergadiotis, D. Tsarapatsanis, N. Aletras, I. Androutsopoulos, and P. Malakasiotis Paragraph-level rationale extraction through regularization: a case study on european court of human rights cases. arXiv preprint arXiv:2103.13084. Cited by: §4.1. Chan et al. (2023) C. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu Chateval: towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201. Cited by: §4.2, Table 6, Table 6, Table 6. Chen et al. (2024) J. Chen, A. Lotsos, L. Zhao, C. Wang, J. Hullman, B. Sherin, U. Wilensky, and M. Horn A computational method for measuring"" open codes"" in qualitative analysis. arXiv preprint arXiv:2411.12142. Cited by: Table 1, Table 1. Chen et al. (2018) N. Chen, M. Drouhard, R. Kocielnik, J. Suh, and C. R. Aragon Using machine learning to support qualitative coding in social science: shifting the focus to ambiguity. ACM Transactions on Interactive Intelligent Systems (TiiS) 8 (2), p. 1–20. Cited by: §2.1, Table 1, Table 1. Chen et al. (2016) N. Chen, R. Kocielnik, M. Drouhard, V. Peña-Araya, J. Suh, K. Cen, X. Zheng, C. R. Aragon, and V. Peña-Araya Challenges of applying machine learning to qualitative coding. In ACM SIGCHI Workshop on Human-Centered Machine Learning, Cited by: §2.2. Chen and Meurers (2016) X. Chen and D. Meurers Characterizing text difficulty with word frequencies. In Proceedings of the 11th workshop on innovative use of nlp for building educational applications, p. 84–94. Cited by: §4.3. Cheng et al. (2017) J. Cheng, M. Bernstein, C. Danescu-Niculescu-Mizil, and J. Leskovec Anyone can become a troll: causes of trolling behavior in online discussions. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, p. 1217–1230. Cited by: Table 6, Table 6, Table 6, Table 6, Table 6, Table 6. Chew et al. (2023) R. Chew, J. Bollenbacher, M. Wenger, J. Speer, and A. Kim LLM-assisted content analysis: using large language models to support deductive coding. arXiv preprint arXiv:2306.14924. Cited by: §1, §2.1, Table 1, Table 1, Table 1. Chopra and Haaland (2023) F. Chopra and I. Haaland Conducting qualitative interviews with ai. Cited by: Table 1. Christou (2025) P. Christou Looking beyond numbers in qualitative research: from data saturation to data analysis. The Qualitative Report 30 (1), p. 3088–3100. Cited by: Table 1. Christou (2023a) P. A. Christou A critical perspective over whether and how to acknowledge the use of artificial intelligence (ai) in qualitative studies. The Qualitative Report 28 (7), p. 1981–1991. Cited by: Table 1. Christou (2023b) P. A. Christou How to use artificial intelligence (ai) as a resource, methodological and analysis tool in qualitative research?. Qualitative Report 28 (7). Cited by: §2.2, §3.3.1, §3.3.2, Table 1, Table 1, Table 1. Christou (2024) P. A. Christou Thematic analysis through artificial intelligence (ai). Qualitative Report 29 (2). Cited by: Table 1. Chuang et al. (2023) Y. Chuang, A. Goyal, N. Harlalka, S. Suresh, R. Hawkins, S. Yang, D. Shah, J. Hu, and T. T. Rogers Simulating opinion dynamics with networks of llm-based agents. arXiv preprint arXiv:2311.09618. Cited by: §4.4.4. Chubb (2023) L. A. Chubb Me and the machines: possibilities and pitfalls of using artificial intelligence for qualitative data analysis. International journal of qualitative methods 22, p. 16094069231193593. Cited by: Table 1. [39] B. Corpus Corpus of contemporary american english. Cited by: §4.3. Costa (2023) A. P. Costa Qualitative research methods: do digital tools open promising trends?. Revista Lusófona de Educação 59 (59). Cited by: Table 1, Table 1. Dai et al. (2023) S. Dai, A. Xiong, and L. Ku LLM-in-the-loop: leveraging large language model for thematic analysis. arXiv preprint arXiv:2310.15100. Cited by: §3.3.1, Table 1. Davison et al. (2024) R. M. Davison, H. Chughtai, P. Nielsen, M. Marabelli, F. Iannacci, M. van Offenbeek, M. Tarafdar, M. Trenz, A. A. Techatassanasoontorn, A. Diaz Andrade, et al. The ethics of using generative ai for qualitative data analysis.. Information Systems Journal 34 (5). Cited by: Table 1, Table 1. de Morais Leça et al. (2024) M. de Morais Leça, L. Valença, R. de Souza Santos, and R. Santos Applications and implications of large language models in qualitative analysis: a new frontier for empirical software engineering. arXiv e-prints, p. arXiv–2412. Cited by: Table 1, Table 1. De Paoli and Mathis (2024) S. De Paoli and W. S. Mathis Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with llms. Quality & Quantity, p. 1–27. Cited by: Table 1. De Paoli (2023) S. De Paoli Can large language models emulate an inductive thematic analysis of semi-structured interviews? an exploration and provocation on the limits of the approach and the model. arXiv preprint arXiv:2305.13014. Cited by: Table 1, Table 1. De Paoli (2024) S. De Paoli Performing an inductive thematic analysis of semi-structured interviews with a large language model: an exploration and provocation on the limits of the approach. Social Science Computer Review 42 (4), p. 997–1019. Cited by: Table 1, Table 1. DeCuir-Gunby et al. (2011) J. T. DeCuir-Gunby, P. L. Marshall, and A. W. McCulloch Developing and using a codebook for the analysis of interview data: an example from a professional development research project. Field methods 23 (2), p. 136–155. Cited by: §4.4.1, Table 2. Dengel et al. (2023) A. Dengel, R. Gehrlein, D. Fernes, S. G"̈orlich, J. Maurer, H. H. Pham, G. Großmann, and N. D. g. Eisermann Qualitative research methods for large language models: conducting semi-structured interviews with chatgpt and bard on computer science education. In Informatics, Vol. 10, p. 78. Cited by: Table 1. Deutsch et al. (2011) M. Deutsch, P. T. Coleman, and E. C. Marcus The handbook of conflict resolution: theory and practice. John Wiley & Sons. Cited by: §4.2. Diziol et al. (2010) D. Diziol, E. Walker, N. Rummel, and K. R. Koedinger Using intelligent tutor technology to implement adaptive support for student collaboration. Educational Psychology Review 22, p. 89–102. Cited by: Table 6. Doi et al. (2024) T. Doi, M. Isonuma, and H. Yanaka Comprehensive evaluation of large language models for topic modeling. External Links: 2406.00697 Cited by: §4.4.1, Table 2. Drouhard et al. (2017) M. Drouhard, N. Chen, J. Suh, R. Kocielnik, V. Pena-Araya, K. Cen, X. Zheng, and C. R. Aragon Aeonium: visual analytics to support collaborative qualitative coding. In 2017 IEEE Pacific Visualization Symposium (PacificVis), p. 220–229. Cited by: §2.1. Du et al. (2023) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325. Cited by: §4.2. Dunivin (2024) Z. O. Dunivin Scalable qualitative coding with llms: chain-of-thought reasoning matches human performance in some hermeneutic tasks. arXiv preprint arXiv:2401.15170. Cited by: §2.1, Table 1, Table 1. Eschrich and Sterman (2024) J. Eschrich and S. Sterman A framework for discussing llms as tools for qualitative analysis. arXiv preprint arXiv:2407.11198. Cited by: Table 1. Felton et al. (2009) M. Felton, M. Garcia-Mila, and S. Gilabert Deliberation versus dispute: the impact of argumentative discourse goals on learning and reasoning in the science classroom. Informal Logic 29 (4), p. 417–446. Cited by: Table 6, Table 6, Table 6, Table 6, Table 6, Table 6, Table 6. Feuston and Brubaker (2021) J. L. Feuston and J. R. Brubaker Putting tools in their place: the role of time and perspective in human-ai collaboration for qualitative analysis. Proceedings of the ACM on Human-Computer Interaction 5 (CSCW2), p. 1–25. Cited by: §2.2, Table 1, Table 1. Friedman et al. (2024) C. Friedman, A. Owen, and L. VanPuymbrouck Should chatgpt help with my research? a caution against artificial intelligence in qualitative analysis. Qualitative Research, p. 14687941241297375. Cited by: Table 1. Ganji et al. (2018) A. Ganji, M. Orand, and D. W. McDonald Ease on down the code: complex collaborative qualitative coding simplified with’code wizard’. Proceedings of the ACM on human-computer interaction 2 (CSCW), p. 1–24. Cited by: §4.4.2, §4.4.3, Table 3, Table 4. Gao et al. (2023a) J. Gao, J. Cao, S. Yeo, K. T. W. Choo, Z. Zhang, T. J. Li, S. Zhao, and S. T. Perrault Impact of human-ai interaction on user trust and reliance in ai-assisted qualitative coding. arXiv preprint arXiv:2309.13858. Cited by: Table 1, Table 1, Table 1, §4.2. Gao et al. (2023b) J. Gao, K. T. W. Choo, J. Cao, R. K. Lee, and S. Perrault CoAIcoder: examining the effectiveness of ai-assisted human-to-human collaboration in qualitative analysis. ACM Transactions on Computer-Human Interaction 31 (1), p. 1–38. Cited by: §2.1, Table 1, Table 1. Gao et al. (2024) J. Gao, Y. Guo, G. Lim, T. Zhang, Z. Zhang, T. J. Li, and S. T. Perrault CollabCoder: a lower-barrier, rigorous workflow for inductive collaborative qualitative analysis with large language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, p. 1–29. Cited by: §2.2, Table 1, Table 1. Gao et al. (2025) J. Gao, Z. Shu, and S. Y. Yeo Using large language model to support flexible and structural inductive qualitative analysis. arXiv preprint arXiv:2501.00775. Cited by: Table 1. Gebreegziabher et al. (2023) S. A. Gebreegziabher, Z. Zhang, X. Tang, Y. Meng, E. L. Glassman, and T. J. Li Patat: human-ai collaborative qualitative coding with explainable interactive rule synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, p. 1–19. Cited by: §2.2, §3.3.1, §3.3.1, Table 1, Table 1, Table 1. Giri (2024) R. C. Giri Rapid ai: an ai application for thematic analysis and enhanced qualitative research. Cited by: Table 1. Goyanes et al. (2024) M. Goyanes, C. Lopezosa, and B. Jordá Thematic analysis of interview data with chatgpt: designing and testing a reliable research protocol for qualitative research. Center for Open Science. https://osf. io/8mr2f/download. Cited by: Table 1. Grice (1990) H. P. Grice 1975 logic and conversation. The Philosophy of Language. Cited by: Table 7, Table 7, Table 6. Guetterman et al. (2018) T. C. Guetterman, T. Chang, M. DeJonckheere, T. Basu, E. Scruggs, and V. V. Vydiswaran Augmenting qualitative text analysis with natural language processing: methodological study. Journal of medical Internet research 20 (6), p. e231. Cited by: Table 1. Halterman and Keith (2024) A. Halterman and K. A. Keith Codebook llms: adapting political science codebooks for llm use and adapting llms to follow codebooks. arXiv preprint arXiv:2407.10747. Cited by: §2.1. Hamilton et al. (2023) L. Hamilton, D. Elliott, A. Quick, S. Smith, and V. Choplin Exploring the use of ai in qualitative analysis: a comparative study of guaranteed income data. International journal of qualitative methods 22, p. 16094069231201504. Cited by: Table 1, Table 1, Table 1. Hasija and Esper (2022) A. Hasija and T. L. Esper In artificial intelligence (ai) we trust: a qualitative investigation of ai technology acceptance. Journal of Business Logistics 43 (3), p. 388–412. Cited by: Table 1, Table 1. Hayes (2025) A. S. Hayes “Conversing” with qualitative data: enhancing qualitative research through large language models (llms). International Journal of Qualitative Methods 24, p. 16094069251322346. Cited by: Table 1, Table 1. Hitch (2024) D. Hitch Artificial intelligence augmented qualitative analysis: the way of the future?. Qualitative Health Research 34 (7), p. 595–606. Cited by: Table 1. Hong et al. (2022) M. Hong, L. A. Marsh, J. L. Feuston, J. Ruppert, J. R. Brubaker, and D. A. Szafir Scholastic: graphical human-ai collaboration for inductive and interpretive text analysis. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, p. 1–12. Cited by: §2.2, Table 1, Table 1. Hruschka et al. (2004) D. J. Hruschka, D. Schwartz, D. C. St. John, E. Picone-Decaro, R. A. Jenkins, and J. W. Carey Reliability in coding open-ended data: lessons learned from hiv behavioral research. Field methods 16 (3), p. 307–331. Cited by: Table 2. Jalali and Akhavan (2024) M. S. Jalali and A. Akhavan Integrating ai language models in qualitative research: replicating interview data analysis with chatgpt. System Dynamics Review 40 (3), p. e1772. Cited by: Table 1. Jiang et al. (2021) J. A. Jiang, K. Wade, C. Fiesler, and J. R. Brubaker Supporting serendipity: opportunities and challenges for human-ai collaboration in qualitative analysis. Proceedings of the ACM on Human-Computer Interaction 5 (CSCW1), p. 1–23. Cited by: §1, §2.2, §2, Table 1, Table 1. Johs et al. (2022) A. J. Johs, D. E. Agosto, and R. O. Weber Explainable artificial intelligence and social science: further insights for qualitative investigation. Applied AI letters 3 (1), p. e64. Cited by: Table 1, Table 1. Joshi et al. (2023) N. Joshi, J. Rando, A. Saparov, N. Kim, and H. He Personas as a way to model truthfulness in language models. arXiv preprint arXiv:2310.18168. Cited by: §4.4.1, Table 2. Kantor (2024) J. Kantor Best practices for implementing chatgpt, large language models, and artificial intelligence in qualitative and survey-based research. JAAD international 14, p. 22–23. Cited by: §1, §2.2, §2, Table 1. Katz et al. (2024) A. Katz, G. C. Fleming, and J. Main Thematic analysis with open-source generative ai and machine learning: a new method for inductive qualitative codebook development. arXiv preprint arXiv:2410.03721. Cited by: Table 1. Kay and Roscheisen (1993) M. Kay and M. Roscheisen Text-translation alignment. Computational linguistics 19 (1), p. 121–142. Cited by: §4.4.1, Table 2. Khan et al. (2024) A. H. Khan, H. Kegalle, R. D’Silva, N. Watt, D. Whelan-Shamy, L. Ghahremanlou, and L. Magee Automating thematic analysis: how llms analyse controversial topics. arXiv preprint arXiv:2405.06919. Cited by: §2.2, Table 1. Kirsten et al. (2024) E. Kirsten, A. Buckmann, A. Mhaidli, and S. Becker Decoding complexity: exploring human-ai concordance in qualitative coding. arXiv preprint arXiv:2403.06607. Cited by: §2.1, Table 1, Table 1, Table 1. Klemmer et al. (2024) J. H. Klemmer, S. A. Horstmann, N. Patnaik, C. Ludden, C. Burton Jr, C. Powers, F. Massacci, A. Rahman, D. Votipka, H. R. Lipford, et al. Using ai assistants in software development: a qualitative study on security practices and concerns. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 2726–2740. Cited by: §3.3.1, Table 1, Table 1. Lee et al. (2024) V. V. Lee, S. C. van der Lubbe, L. H. Goh, and J. M. Valderas Harnessing chatgpt for thematic analysis: are we ready?. Journal of Medical Internet Research 26, p. e54974. Cited by: Table 1. Lennon et al. (2021) R. P. Lennon, R. Fraleigh, L. J. Van Scoy, A. Keshaviah, X. C. Hu, B. L. Snyder, E. L. Miller, W. A. Calo, A. E. Zgierska, and C. Griffin Developing and testing an automated qualitative assistant (aqua) to support qualitative analysis. Family medicine and community health 9 (Suppl 1), p. e001287. Cited by: Table 1. Li et al. (2024a) J. Li, Q. Zhang, Y. Yu, Q. Fu, and D. Ye More agents is all you need. arXiv preprint arXiv:2402.05120. Cited by: §4.2. Li et al. (2024b) K. D. Li, A. M. Fernandez, R. Schwartz, N. Rios, M. N. Carlisle, G. M. Amend, H. V. Patel, and B. N. Breyer Comparing gpt-4 and human researchers in qualitative analysis of healthcare data: qualitative description study. J Med Internet Res. Cited by: Table 1. Li et al. (2024c) T. Li, G. Zhang, Q. D. Do, X. Yue, and W. Chen Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060. Cited by: §4.4.1, Table 2, Table 2. Li et al. (2021) Z. Li, D. Dohan, and C. M. Abramson Qualitative coding in the computational era: a hybrid approach to improve reliability and reduce effort for coding ethnographic interviews. Socius 7, p. 23780231211062345. Cited by: §3.3.1, Table 1. Liu et al. (2024) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, p. 157–173. Cited by: §4.4.1, Table 2, Table 2. Liu et al. (2025) X. Liu, A. F. Zambrano, R. S. Baker, A. Barany, J. Ocumpaugh, J. Zhang, M. Pankiewicz, N. Nasiar, and Z. Wei Qualitative coding with gpt-4: where it works better. Journal of Learning Analytics, p. 1–17. Cited by: Table 1. [94] X. Liu, J. Zhang, and A. Barany Assessing the potential and limits of large language models in qualitative coding. Cited by: §2.1. Liu et al. (2008) X. Liu, J. Wu, and Z. Zhou Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 39 (2), p. 539–550. Cited by: §4.3. Locher (2004) M. Locher Power and politeness in action: disagreements in oral communication. Mouton de Gruyter. Cited by: Table 6. Longo (2019) L. Longo Empowering qualitative research methods in education with artificial intelligence. In World conference on qualitative research, p. 1–21. Cited by: Table 1. Lopez-Fierro and Nguyen (2024) S. Lopez-Fierro and H. Nguyen Making human-ai contributions transparent in qualitative coding. In Proceedings of the 17th International Conference on Computer-Supported Collaborative Learning-CSCL 2024, p. 3-10, Cited by: Table 1, Table 1. Lu et al. (2011) J. Lu, M. M. Chiu, and N. W. Law Collaborative argumentation and justifications: a statistical discourse analysis of online discussions. Computers in Human Behavior 27 (2), p. 946–955. Cited by: Table 6, Table 6, Table 6, Table 6, Table 6, Table 6. Luger et al. (2014) T. M. Luger, J. Suls, and M. W. Vander Weg How robust is the association between smoking and depression in adults? a meta-analysis using linear mixed-effects models. Addictive behaviors 39 (10), p. 1418–1429. Cited by: §4.3. MacQueen et al. (1998) K. M. MacQueen, E. McLellan, K. Kay, and B. Milstein Codebook development for team-based qualitative analysis. Cam Journal 10 (2), p. 31–36. Cited by: §4.4.4, Table 5. Marathe and Toyama (2018) M. Marathe and K. Toyama Semi-automated coding for qualitative research: a user-centered inquiry and initial prototypes. In Proceedings of the 2018 CHI conference on human factors in computing systems, p. 1–12. Cited by: §2.1. Marshall and Naff (2024) D. T. Marshall and D. B. Naff The ethics of using artificial intelligence in qualitative research. Journal of Empirical Research on Human Research Ethics 19 (3), p. 92–102. Cited by: Table 1, Table 1, Table 1. Mathis et al. (2024) W. S. Mathis, S. Zhao, N. Pratt, J. Weleff, and S. De Paoli Inductive thematic analysis of healthcare qualitative interviews using open-source large language models: how does it compare to traditional methods?. Computer Methods and Programs in Biomedicine 255, p. 108356. Cited by: Table 1, Table 1. Mehrad et al. (2024) A. Mehrad, M. H. T. Zangeneh, M. Bouzedif, N. G. Rweramila, and H. Boutament Qualitative methods in statistics and methodology: artificial intelligence (ai). Journal of Advanced Studies in Social Sciences 2 (2). Cited by: Table 1. Meng et al. (2024) H. Meng, Y. Yang, Y. Li, J. Lee, and Y. Lee Exploring the potential of human-llm synergy in advancing qualitative analysis: a case study on mental-illness stigma. arXiv preprint arXiv:2405.05758. Cited by: §2.1. Mesec (2023) B. Mesec The language model of artificial inteligence chatgpt-a tool of qualitative analysis of texts. Authorea Preprints. Cited by: Table 1. Moher et al. (2009) D. Moher, A. Liberati, J. Tetzlaff, and D. G. Altman Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. Bmj 339. Cited by: §3. Morgan (2023) D. L. Morgan Exploring the use of artificial intelligence for qualitative data analysis: the case of chatgpt. International journal of qualitative methods 22, p. 16094069231211248. Cited by: §2.2, Table 1. Mortelmans (2024) D. Mortelmans NVivo and ai:(semi)-automatic coding. In Doing Qualitative Data Analysis with NVivo, p. 229–250. Cited by: Table 1, Table 1. Nejjar et al. (2025) M. Nejjar, L. Zacharias, F. Stiehle, and I. Weber Llms for science: usage for code generation and data analysis. Journal of Software: Evolution and Process 37 (1), p. e2723. Cited by: Table 1. Nelson et al. (2020) C. A. Nelson, L. M. Pérez-Chada, A. Creadore, S. J. Li, K. Lo, P. Manjaly, A. B. Pournamdari, E. Tkachenko, J. S. Barbieri, J. M. Ko, et al. Patient perspectives on the use of artificial intelligence for skin cancer screening: a qualitative study. JAMA dermatology 156 (5), p. 501–512. Cited by: Table 1. Nelson et al. (2021) L. K. Nelson, D. Burk, M. Knudsen, and L. McCall The future of coding: a comparison of hand-coding and three types of computer-assisted text analysis methods. Sociological Methods & Research 50 (1), p. 202–237. Cited by: Table 1. Nguyen-Trung and Nguyen (2025) K. Nguyen-Trung and N. L. Nguyen Narrative-integrated thematic analysis (nita): ai-supported theme generation without coding. Cited by: Table 1. Nguyen-Trung (2024) K. Nguyen-Trung ChatGPT in thematic analysis: can ai become a research assistant in qualitative research. OSF Preprint, researchgate. net. Cited by: Table 1. Nyaaba et al. (2025) M. Nyaaba, M. SungEun, M. A. Apam, K. O. Acheampong, and E. Dwamena Optimizing generative ai’s accuracy and transparency in inductive thematic analysis: a human-ai comparison. arXiv preprint arXiv:2503.16485. Cited by: Table 1, Table 1. Omizo (2024) R. M. Omizo Automating research in business and technical communication: large language models as qualitative coders. Journal of Business and Technical Communication 38 (3), p. 242–265. Cited by: Table 1. Overney et al. (2024) C. Overney, B. Saldías, D. Dimitrakopoulou, and D. Roy Sensemate: an accessible and beginner-friendly human-ai platform for qualitative data analysis. In Proceedings of the 29th International Conference on Intelligent User Interfaces, p. 922–939. Cited by: Table 1, Table 1, Table 1, Table 1. Overney (2023) C. Overney SenseMate: an ai-based platform to support qualitative coding. Master’s Thesis, Massachusetts Institute of Technology. Cited by: §3.3.2, Table 1, Table 1, Table 1. Palea et al. (2024) D. Palea, G. Vadhul, and D. T. Lee Annota: peer-based ai hints towards learning qualitative coding at scale. In Proceedings of the 29th International Conference on Intelligent User Interfaces, p. 455–470. Cited by: §3.3.2, Table 1, Table 1, Table 1. Parfenova et al. (2024) A. Parfenova, A. Denzler, and J. Pfeffer Automating qualitative data analysis with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), p. 83–91. Cited by: Table 1, Table 1. Parker (2006) W. C. Parker Public discourses in schools: purposes, problems, possibilities. Educational Researcher 35 (8), p. 11–18. Cited by: Table 6. Pattyn (2024) F. Pattyn The value of generative ai for qualitative research: a pilot study. Journal of Data Science and Intelligent Systems. Cited by: Table 1. Paulus and Marone (2024) T. M. Paulus and V. Marone “In minutes instead of weeks”: discursive constructions of generative ai and qualitative data analysis. Qualitative Inquiry, p. 10778004241250065. Cited by: Table 1. Perkins and Roe (2024a) M. Perkins and J. Roe Generative ai tools in academic research: applications and implications for qualitative and quantitative research methodologies. arXiv preprint arXiv:2408.06872. Cited by: Table 1, Table 1. Perkins and Roe (2024b) M. Perkins and J. Roe The use of generative ai in qualitative analysis: inductive thematic analysis with chatgpt. Journal of Applied Learning and Teaching 7 (1). Cited by: Table 1. Piano et al. (2024) S. Piano et al. Qualitative code suggestion: a human-centric approach to qualitative coding. Cited by: Table 1, Table 1. Player et al. (2024) L. Player, R. Hughes, K. Mitev, L. Whitmarsh, C. Demski, N. Nash, T. Papakonstantinou, and M. Wilson The use of large language models for qualitative research: decota. Cited by: Table 1, Table 1. Poppel (2020) L. v. Poppel The study of metaphor in argumentation theory. In Argumentation Through Languages and Cultures, p. 177–208. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7. Prescott et al. (2024) M. R. Prescott, S. Yeager, L. Ham, C. D. Rivera Saldana, V. Serrano, J. Narez, D. Paltin, J. Delgado, D. J. Moore, and J. Montoya Comparing the efficacy and efficiency of human and generative ai: qualitative thematic analyses. JMIR AI 3, p. e54482. Cited by: §3.3.2, Table 1. [131] T. Qiao, C. Walker, C. W. Cunningham, and Y. S. Koh Thematic-lm: a llm-based multi-agent system for large-scale thematic analysis. In THE WEB CONFERENCE 2025, Cited by: Table 1, Table 1. Rao et al. (2024) V. N. Rao, E. Agarwal, S. Dalal, D. Calacci, and A. Monroy-Hernández QuaLLM: an llm-based framework to extract quantitative insights from online forums. arXiv preprint arXiv:2405.05345. Cited by: Table 1. Rasheed et al. (2024) Z. Rasheed, M. Waseem, A. Ahmad, K. Kemell, W. Xiaofeng, A. N. Duc, and P. Abrahamsson Can large language models serve as data analysts? a multi-agent assisted approach for qualitative data analysis. arXiv preprint arXiv:2402.01386. Cited by: Table 1. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-bert: sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. Cited by: §4.3. Rietz and Maedche (2021) T. Rietz and A. Maedche Cody: an ai-based system to semi-automate coding for qualitative research. In Proceedings of the 2021 CHI conference on human factors in computing systems, p. 1–14. Cited by: §2.2, §3.3.1, Table 1, Table 1, Table 1. Rietz (2021) T. Rietz Designing ai-based systems for qualitative data collection and analysis. Ph.D. Thesis, Dissertation, Karlsruhe, Karlsruher Institut f"̈ur Technologie (KIT), 2021. Cited by: Table 1, Table 1. Ritchie et al. (2022) M. J. Ritchie, K. L. Drummond, B. N. Smith, J. L. Sullivan, and S. J. Landes Development of a qualitative data analysis codebook informed by the i-parihs framework. Implementation science communications 3 (1), p. 98. Cited by: §4.4.1, §4.4.4, Table 2, Table 5. [138] V. Ritschl, L. Grespan, and D. U. Ort Assessing the effectiveness of large language models in qualitative content analysis of interview data. Cited by: Table 1. Roberts et al. (2024) J. Roberts, M. Baker, and J. Andrew Artificial intelligence and qualitative research: the promise and perils of large language model (llm)‘assistance’. Critical Perspectives on Accounting 99, p. 102722. Cited by: Table 1. Sabbaghan (2024) S. Sabbaghan Exploring the synergy of human and ai-driven approaches in thematic analysis for qualitative educational research. Journal of Applied Learning and Teaching 7 (2). Cited by: Table 1, Table 1. Sanfilippo et al. (2020) K. R. M. Sanfilippo, N. Spiro, M. Molina-Solana, and A. Lamont Do the shuffle: exploring reasons for music listening through shuffled play. PloS one 15 (2), p. e0228457. Cited by: §4.1. Scannapieco (1997) F. A. Scannapieco Formal debate: an active learning strategy. Journal of Dental Education 61, p. 955–961. Cited by: §4.2. Schmettow (2015) M. Schmettow Tutorial: modern regression techniques for hci researchers. In Human-Computer Interaction–INTERACT 2015: 15th IFIP TC 13 International Conference, Bamberg, Germany, September 14-18, 2015, Proceedings, Part IV 15, p. 651–654. Cited by: §4.3. Schopf et al. (2022) T. Schopf, D. Braun, and F. Matthes Evaluating unsupervised text classification: zero-shot and similarity-based approaches. In Proceedings of the 2022 6th International Conference on Natural Language Processing and Information Retrieval, p. 6–15. Cited by: §4.1. Schwarz et al. (2018) B. B. Schwarz, N. Prusak, O. Swidan, A. Livny, K. Gal, and A. Segal Orchestrating the emergence of conceptual learning: a case study in a geometry class. International Journal of Computer-Supported Collaborative Learning 13, p. 189–211. Cited by: Table 6. Schwitter (2025) N. Schwitter Using large language models for preprocessing and information extraction from unstructured text: a proof-of-concept application in the social sciences. Methodological Innovations, p. 20597991251313876. Cited by: §4.2. Şen et al. (2023) M. Şen, Ş. Şen, and T. Şahin A new era for data analysis in qualitative research: chatgpt!. Shanlax International Journal of Education 11 (1). Cited by: Table 1, Table 1. Settles (2009) B. Settles Active learning literature survey. Cited by: §4.4.2, Table 3. Siiman et al. (2023) L. A. Siiman, M. Rannastu-Avalos, J. P"̈oys"̈a-Tarhonen, P. H"̈akkinen, and M. Pedaste Opportunities and challenges for ai-assisted qualitative data analysis: an example from collaborative problem-solving discourse data. In International Conference on Innovative Technologies and Learning, p. 87–96. Cited by: §2.1, §3.3.1, Table 1, Table 1, Table 1. Sinha et al. (2024) R. Sinha, I. Solola, H. Nguyen, H. Swanson, and L. Lawrence The role of generative ai in qualitative research: gpt-4’s contributions to a grounded theory analysis. In Proceedings of the 2024 Symposium on Learning, Design and Technology, p. 17–25. Cited by: Table 1. Smirnov (2025) E. Smirnov Enhancing qualitative research in psychology with large language models: a methodological exploration and examples of simulations. Qualitative Research in Psychology 22 (2), p. 482–512. Cited by: Table 1. [152] A. P. Smit, N. Grinsztajn, P. Duckworth, T. D. Barrett, and A. Pretorius Should we be going mad? a look at multi-agent debate strategies for llms. In Forty-first International Conference on Machine Learning, Cited by: §4.2. Spinoso-Di Piano (2023) C. Spinoso-Di Piano Qualitative code suggestion: a human-centric approach to qualitative coding. McGill University (Canada). Cited by: §2.2, Table 1. Subramaniam et al. (2024) V. Subramaniam, A. Torralba, and S. Li DebateGPT: fine-tuning large language models with multi-agent debate supervision. Cited by: §2.1. Summers et al. (1950) H. B. Summers, F. L. Whan, and T. A. Rousse How to debate: a textbook for beginners. (No Title). Cited by: §4.2. T"̈ornberg (2023) P. T"̈ornberg How to use llms for text analysis. arXiv preprint arXiv:2307.13106. Cited by: Table 1. Tai et al. (2024) R. H. Tai, L. R. Bentley, X. Xia, J. M. Sitt, S. C. Fankhauser, A. M. Chicas-Mosier, and B. G. Monteith An examination of the use of large language models to aid analysis of textual data. International Journal of Qualitative Methods 23, p. 16094069241231168. Cited by: §2.1, Table 1, Table 1. Tang et al. (2023) J. Tang, T. Wang, and S. Zhu Assessing word difficulty: a mapping method. In 2023 International Conference on Blockchain Technology and Applications (ICBTA), p. 1–4. Cited by: §4.3. Teasley et al. (2008) S. Teasley, F. Fischer, P. Dillenbourg, M. Kapur, M. Chi, A. Weinberger, and K. Stegmann Cognitive convergence in collaborative learning. Cited by: Table 6. Than et al. (2024) N. Than, L. Fan, T. Law, L. K. Nelson, and L. McCall Updating “the future of coding”: qualitative coding with generative large language models. Cited by: Table 1, Table 1, §4.2. Thomas (2008) K. W. Thomas Thomas-kilmann conflict mode. TKI Profile and Interpretive Report 1 (11). Cited by: §4.3, §4.4.4, Table 5. Tjosvold et al. (2019) D. Tjosvold, A. S. Wong, and N. Y. F. Chen Managing conflict for effective leadership and organizations. In Oxford research encyclopedia of business and management, Cited by: §4.4.4, Table 5. Tschisgale et al. (2023) P. Tschisgale, P. Wulff, and M. Kubsch Integrating artificial intelligence-based methods into qualitative research in physics education research: a case for computational grounded theory. Physical Review Physics Education Research 19 (2), p. 020123. Cited by: Table 1, Table 1, Table 1. Turobov et al. (2024) A. Turobov, D. Coyle, and V. Harding Using chatgpt for thematic analysis. arXiv preprint arXiv:2405.08828. Cited by: Table 1. Wachinger et al. (2024) J. Wachinger, K. B"̈arnighausen, L. N. Sch"̈afer, K. Scott, and S. A. McMahon Prompts, pearls, imperfections: comparing chatgpt and a human researcher in qualitative data analysis. Qualitative Health Research, p. 10497323241244669. Cited by: Table 1, Table 1. Wachsmuth and Alshomary (2022) H. Wachsmuth and M. Alshomary "" Mama always had a way of explaining things so i could understand”: a dialogue corpus for learning to construct explanations. arXiv preprint arXiv:2209.02508. Cited by: §4.1. Walton (2008) D. Walton Argumentation schemes. Cambridge University Press. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, §4.3, Table 6. Wang et al. (2024) Q. Wang, Z. Wang, Y. Su, H. Tong, and Y. Song Rethinking the bounds of llm reasoning: are multi-agent discussions the key?. arXiv preprint arXiv:2402.18272. Cited by: §4.2. Wheeler (2025) K. Wheeler How to use generative ai to assist the analysis of qualitative data. Sage Research Methods How to Guides. Cited by: Table 1. Wise and Chiu (2011) A. F. Wise and M. M. Chiu Analyzing temporal patterns of knowledge construction in a role-based online discussion. International Journal of Computer-Supported Collaborative Learning 6, p. 445–470. Cited by: Table 7. Xiao et al. (2023) Z. Xiao, X. Yuan, Q. V. Liao, R. Abdelghani, and P. Oudeyer Supporting qualitative analysis with large language models: combining codebook with gpt-3 for deductive coding. In Companion proceedings of the 28th international conference on intelligent user interfaces, p. 75–78. Cited by: §2.1, Table 1. Xu et al. (2025) H. Xu, S. Yi, T. Lim, J. Xu, A. Well, C. Mery, A. Zhang, Y. Zhang, H. Ji, K. Pingali, et al. TAMA: a human-ai collaborative thematic analysis framework using multi-agent llms for clinical interviews. arXiv preprint arXiv:2503.20666. Cited by: Table 1. Xu et al. (2023) P. Xu, W. Ping, X. Wu, L. McAfee, C. Zhu, Z. Liu, S. Subramanian, E. Bakhturina, M. Shoeybi, and B. Catanzaro Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025. Cited by: §4.4.1, Table 2. Yan et al. (2024) L. Yan, V. Echeverria, G. M. Fernandez-Nieto, Y. Jin, Z. Swiecki, L. Zhao, D. Gašević, and R. Martinez-Maldonado Human-ai collaboration in thematic analysis using chatgpt: a user study and design recommendations. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, p. 1–7. Cited by: Table 1, Table 1. Yang and Ma (2025) Y. Yang and L. Ma Artificial intelligence in qualitative analysis: a practical guide and reflections based on results from using gpt to analyze interview data in a substance use program. Quality & Quantity, p. 1–24. Cited by: Table 1, Table 1. Yoo and Kweon (2019) D. Yoo and I. S. Kweon Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 93–102. Cited by: §4.4.2, Table 3. Zade et al. (2018) H. Zade, M. Drouhard, B. Chinh, L. Gan, and C. Aragon Conceptualizing disagreement in qualitative coding. In Proceedings of the 2018 CHI conference on human factors in computing systems, p. 1–11. Cited by: §3.3.2, Table 1, Table 1. Zakharov et al. (2021) S. Zakharov, O. Hadar, T. Hakak, D. Grossman, Y. B. Kolikant, and O. Tsur Discourse parsing for contentious, non-convergent online discussions. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 15, p. 853–864. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, §4.3, Table 6. Zambrano et al. (2023) A. F. Zambrano, X. Liu, A. Barany, R. S. Baker, J. Kim, and N. Nasiar From ncoder to chatgpt: from automated coding to refining human coding. In International conference on quantitative ethnography, p. 470–485. Cited by: §1, §2.2. Zha et al. (2023) Y. Zha, Y. Yang, R. Li, and Z. Hu Text alignment is an efficient unified model for massive nlp tasks. Advances in Neural Information Processing Systems 36, p. 77942–77968. Cited by: §4.4.1, Table 2. Zhang et al. (2023a) H. Zhang, C. Wu, J. Xie, C. Kim, and J. M. Carroll QualiGPT: gpt as an easy-to-use tool for qualitative coding. arXiv preprint arXiv:2310.07061. Cited by: §3.3.1, Table 1, Table 1. Zhang et al. (2023b) H. Zhang, C. Wu, J. Xie, Y. Lyu, J. Cai, and J. M. Carroll Redefining qualitative analysis in the ai era: utilizing chatgpt for efficient thematic analysis. arXiv preprint arXiv:2309.10771. Cited by: §2.2, Table 1, Table 1. Zhang et al. (2024) H. Zhang, C. Wu, J. Xie, F. Rubino, S. Graver, C. Kim, J. M. Carroll, and J. Cai When qualitative research meets large language model: exploring the potential of qualigpt as a tool for qualitative coding. arXiv preprint arXiv:2407.14925. Cited by: §1, §2.1. Zhang et al. (2023c) J. Zhang, X. Xu, N. Zhang, R. Liu, B. Hooi, and S. Deng Exploring collaboration mechanisms for llm agents: a social psychology view. arXiv preprint arXiv:2310.02124. Cited by: Table 6. Zhao et al. (2024) F. Zhao, F. Yu, and Y. Shang A new method supporting qualitative data analysis through prompt generation for inductive coding. In 2024 IEEE International Conference on Information Reuse and Integration for Data Science (IRI), p. 164–169. Cited by: Table 1. Zięba-Kulawik (2025) K. Zięba-Kulawik Generative ai: new framework of using large language models for analysing descriptive qualitative data. Cited by: Table 1, Table 1. Ziems et al. (2024) C. Ziems, W. Held, O. Shaikh, J. Chen, Z. Zhang, and D. Yang Can large language models transform computational social science?. Computational Linguistics 50 (1), p. 237–291. Cited by: §2.2. Appendix A Comparison of Discursive Moves Between LLM Coders and Humans Table 7. Comparison of Discursive Moves Between LLM Coders and Humans. This table shows the discursive moves used by humans in the literature, which did not appear in the discussions of LLM coders. Category of Discursive Moves Discursive Move of Humans in Literature Moderating Moderation (Grice, 1990; Wise and Chiu, 2011; Bakhtin, 2010) Illustrating Example and Illustration (Walton, 2008; Poppel, 2020) Metaphor and Analogy (Walton, 2008; Poppel, 2020) Expansive Idea Expansion (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Personal Story Personal Narrative (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Questioning Clarification Request (Grice, 1990; Bakhtin, 2010) Questioning Evidence (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Answer (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Critical Question (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Aggressive Confrontational Expression (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Dismissive Humor (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Sarcasm (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Reframing Critique (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Disruptive Relevance Critique (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Nitpicking (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Complaint (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Disruptive Behavior (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Topic Derailment (Zakharov et al., 2021; Walton, 2008; Poppel, 2020) Non-reasoned Disagreement (Zakharov et al., 2021; Walton, 2008; Poppel, 2020)

Instant research discovery