Paper deep dive
Antipatterns in AI-assisted Qualitative Data Analysis: A Catalog of Temptations and Pitfalls for Software Engineering Researchers
Rashina Hoda, Carolyn Seaman, Victoria Gomes, Rodrigo Spinola
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/31/2026, 3:42:02 AM
Summary
This paper presents a catalog of antipatterns in AI-assisted Qualitative Data Analysis (QDA) for Software Engineering (SE) researchers. It identifies risks associated with uncritical use of AI, such as LLMs, which can undermine analytical rigor, validity, reflexivity, and transparency. The antipatterns are categorized into Dangerous Drivers (problematic motivations), Operational Missteps (suboptimal usage), and Analytical Failures (flawed methods). The work aims to provide guidance and vocabulary for researchers and reviewers to navigate the methodological challenges of integrating AI into qualitative research.
Entities (16)
Relation Signals (11)
Carolyn Seaman → affiliatedwith → University of Maryland, Baltimore County
confidence 99% · Carolyn Seaman is with University of Maryland Baltimore County
Rashina Hoda → affiliatedwith → Monash University
confidence 99% · Rashina Hoda is with Monash University
Antipatterns → categorizedas → Analytical Failures
confidence 96% · The antipatterns are grouped into three categories... Dangerous Drivers, Operational Missteps, and Analytical Failures.
Antipatterns → categorizedas → Dangerous Drivers
confidence 96% · The antipatterns are grouped into three categories... Dangerous Drivers, Operational Missteps, and Analytical Failures.
Antipatterns → categorizedas → Operational Missteps
confidence 96% · The antipatterns are grouped into three categories... Dangerous Drivers, Operational Missteps, and Analytical Failures.
AI-assisted QDA → posesrisk → Loss of Reflexivity
confidence 95% · Loss of reflexivity: Since reflexivity requires self-awareness... it is an inherently human process that cannot be meaningfully reproduced or automated by LLMs.
AI-assisted QDA → posesrisk → Loss of Traceability
confidence 94% · Loss of traceability and transparency: The inherent non-deterministic nature... can complicate the ability of future researchers to replicate the process
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI-assisted qualitative data analysis (QDA) offers unprecedented opportunities to streamline software engineering (SE) research, yet uncritical use risks compromising analytical rigor and flooding the field with accelerated production of low-quality research. While tactical best practices will naturally evolve over time, SE researchers currently lack strategic guidance to identify and mitigate methodological risks when attempting AI-assisted QDA. Based on our decades of qualitative SE research expertise and experience combined with an understanding of the emerging landscape of AI-assisted QDA, this paper presents a catalog of antipatterns in AI-assisted QDA - a set of assumptions and practices that initially appear advantageous but ultimately undermine analytical rigor and validity. The antipatterns are grouped into three categories reflecting escalating impact: Dangerous Drivers, Operational Missteps, and Analytical Failures. As more SE researchers attempt AI-assisted QDA, these antipatterns will help them identify and avoid common temptations and pitfalls, while reviewers can be equipped with the vocabulary and criteria to call out problematic and failed practice. Ultimately, this catalog of antipatterns can serve as a stepping stone in our responsible methodological evolution toward principled and meaningful human-AI collaboration in qualitative research.
Tags
Links
- Source: https://arxiv.org/abs/2608.27927v1
- Canonical: https://arxiv.org/abs/2608.27927v1
Trouble viewing inline? Open PDF directly →
Full Text
77,311 characters extracted from source content.
Expand or collapse full text
UNDER REVIEW: HODA ET AL., 20261 Antipatterns in AI-assisted Qualitative Data Analysis: A Catalog of Temptations and Pitfalls for Software Engineering Researchers Rashina Hoda, Carolyn Seaman, Vict ́ oria Gomes, Rodrigo Sp ́ ınola ✦ Abstract—AI-assisted qualitative data analysis (QDA) offers unprece- dented opportunities to streamline software engineering (SE) research, yet uncritical use risks compromising analytical rigor and flooding the field with accelerated production of low-quality research. While tactical best practices will naturally evolve over time, SE researchers currently lack strategic guidance to identify and mitigate methodological risks when attempting AI-assisted QDA. Based on our decades of qualitative SE research expertise and experience combined with an understanding of the emerging landscape of AI-assisted QDA, this paper presents a catalog of antipatterns in AI-assisted QDA – a set of assumptions and practices that initially appear advantageous but ultimately undermine analytical rigor and validity. The antipatterns are grouped into three categories reflecting escalating impact: Dangerous Drivers, Operational Missteps, and Analytical Failures. As more SE researchers attempt AI-assisted QDA, these antipatterns will help them identify and avoid common temptations and pitfalls, while reviewers can be equipped with the vocabulary and criteria to call out problematic and failed practice. Ul- timately, this catalog of antipatterns can serve as a stepping stone in our responsible methodological evolution toward principled and meaningful human-AI collaboration in qualitative research. Index Terms—Qualitative Data Analysis, Qualitative Research, Artificial Intelligence, AI, Antipatterns, Risks, Pitfalls, Failures, Software Engi- neering, Research. 1 INTRODUCTION Q UALITATIVE data analysis (QDA) plays an important role in software engineering (SE) research, enabling rich, contextualized insights into complex and contempo- rary socio-technical phenomena such as developer use of large language models (LLMs), team dynamics in agile soft- ware development, and human-artificial intelligence (AI) collaboration in SE. Core principles that guide QDA in SE research include grounding in data, iterative and systematic analysis (e.g., coding and constant comparison) [1], [2], reflexivity of the researcher in interpreting socio-technical phenomena [3], [4], and transparency of the analytical process [5]. Together, these principles ensure that findings are credible, rigorously Manuscript received; revised [date]. Rashina Hoda is with Monash University, Melbourne, Australia (e-mail: rashina.hoda@monash.edu). Carolyn Seaman is with University of Maryland Baltimore County, Baltimore, USA (e-mail: cseaman@umbc.edu). Vict ́oria Gomes and Rodrigo Sp ́ınola are with Virginia Commonwealth University, Richmond, USA (e-mail: oliveiragov@vcu.edu, spinolaro@vcu.edu) derived, and traceable, aligning with established criteria for trustworthiness in qualitative research [6]. Rather than merely summarizing or categorizing data, QDA aims to generate explanations, identify patterns, and construct con- cepts or theory that are deeply grounded in the data [1] and situated within context through interpretive practices [7]. The recent emergence of modern AI (e.g., generative AI and agentic AI) is rapidly transforming how researchers engage with data through QDA techniques such as the- matic analysis [7], [8], socio-technical grounded theory (STGT) for data analysis [3], [4], discourse analysis [9], and narrative analysis [10]. Modern AI tools based on LLMs promise to accelerate the pace of the analytic process, reduce manual effort, and enable the processing of larger datasets than pre- viously possible [11], [12]. In SE, where qualitative methods are often seen to be constrained by time and resource limi- tations, these capabilities present compelling opportunities and unprecedented temptations to expand the scope and scale of QDA [3], [13]. However, the integration of AI into QDA also raises significant methodological concerns. When used uncriti- cally, AI can obscure grounding in data, reduce iterative engagement, diminish reflexivity, and weaken transparency, thereby undermining the very principles that ensure the rigor and validity of qualitative research[5], [14]–[16]. Though it offers compelling potential, there is a growing risk that the uncritical use of AI-assisted QDA may enable the production of low-quality qualitative research, faster and at scale, raising concerns about a potential incoming methodological crisis. This tension is reflected in debates across the wider qualitative research community. At the one end of the spectrum, a group of 419 social science scholars, including pioneering methodologists, have strongly rejected the use of AI in reflexive qualitative research, arguing that inter- pretive analysis cannot be meaningfully delegated to au- tomated systems without undermining its epistemological foundation such as reflexivity [17]. On the other end of the spectrum, AI adoption is accelerating in research, as evidenced by both the rapid incorporation of AI-based fea- tures into established QDA platforms, e.g., MAXQDA [18], NVivo [19], ATLAS.ti [20], and the concurrent emergence of methodological guidelines intended to support AI-assisted arXiv:2608.27927v1 [cs.SE] 28 Aug 2026 UNDER REVIEW: HODA ET AL., 20262 qualitative workflows [21]. This creates a paradoxical sit- uation in which AI-assisted QDA is being simultaneously resisted on principled grounds while also being rapidly promoted in practice. As socio-technical researchers, it is unsurprising that the SE research community is increasingly intrigued by the prospect of leveraging AI to assist with QDA. Based on emerging work (summarized in the next section) and discussions at SE conferences and workshops, a spectrum seems to be emerging, from researchers and reviewers who are inclined to reject AI-assisted QDA to those who are readily embracing it, with those who are cautiously curious about its use standing in between. The first two groups represent opposing polarized re- sponses, neither of which is likely to be productive. Em- bracing AI-assisted QDA without critical reflection risks eroding methodological rigor, while rejecting it altogether risks missing opportunities to address chronic limitations such as speed, scalability, and procedural consistency. Those cautiously curious about AI-assisted QDA seem to be un- sure how to proceed due to concerns about invalidating its epistemological foundations and a lack of credible guidance. Addressing this need for guidance presents an unusual challenge for the SE research community, which has a long history of borrowing, contextualizing, and adapting existing credible guidance on qualitative methodologies from the social sciences to suit our unique socio-technical contexts, e.g., Case Studies [22], Ethnography [23], STGT [3], and Mixed- Methods Research [24]. In the case of AI-assisted QDA, there is no existing playbook to follow or adapt from. If any- thing, prominent social science research experts are largely rejecting AI [17]. For a change, as experienced qualitative SE research experts, we may be better positioned to explore the opportunity more thoroughly because of our dual expertise and offer some early guidance to our field. As a community, we stand on the shore of uncharted waters. Calibrated maps, in the form of tactical guidelines and best practices, will likely emerge in the coming months and years from the attempts, successes, and failures of those undertaking this journey. In the meantime, we cannot rely solely on trial and error. As we begin navigating this emerging space, we need guidance on what can go wrong (e.g., epistemological misalignment, loss of reflexivity, bias encoding), why such issues arise (based on a deep under- standing and hands-on experience of QDA as well as tech- nical expertise in understanding capabilities and limitations of AI), and how to avoid such issues when attempting AI-assisted QDA. Such guidance can help move the field from cautious interest toward careful applications. As SE researchers with reasonable understanding of this emerging technology and substantial prior experience in designing, conducting, teaching, and evaluating qualitative research in SE and adapting qualitative research methods for SE [3]– [5], we find ourselves uniquely positioned, and somewhat obliged, to contribute to addressing this early guidance gap. In this paper, we aim to help SE researchers and review- ers, regardless of their current stance towards AI, by ac- knowledging the emerging opportunities and highlighting the temptations, risks, and pitfalls of AI-assisted QDA in SE. To this end, we present a catalog of antipatterns in AI- assisted qualitative data analysis, providing a timely and critical lens for examining where and how methodological issues can emerge. We define antipatterns in this context as emerging assumptions and practices that initially appear desirable and seemingly reasonable, but ultimately undermine the rigor and validity of qualitative data analysis. We organize eleven antipatterns into three intercon- nected categories that reflect escalating methodological is- sues and impact: (i) Dangerous Drivers, which capture problematic or unexamined motivations for adopting AI- assisted QDA; (i) Operational Missteps, which represent misconceptions and incorrect or suboptimal uses of AI during the analytical process; and (i) Analytical Failures, which correspond to fundamentally flawed uses of AI- assisted QDA that compromise the methodological integrity and validity of the analysis. With this catalog, we aim to provide both researchers and reviewers with a structured vocabulary to communi- cate and reason about AI-assisted QDA. For researchers, these antipatterns serve as both warnings and guidance to avoid common temptations and pitfalls when attempting AI-assisted QDA. For reviewers, they offer a lens to iden- tify and articulate problematic practices when assessing submitted work. While neither exhaustive nor final, we hope the catalog will contribute to emerging yet critical dialogue and understanding of what constitutes purposeful, meaningful, and valuable QDA in the age of AI [11], [21]. The rest of this paper is organized as follows. Section 2 presents a brief overview of the emerging landscape of AI- assisted QDA as represented in current literature. Section 3 presents the catalog of antipatterns in AI-assisted QDA, including a discussion of their cascading and compounding effects and the scope, application, and limitations of the catalog. Section 4 presents recommendations for researchers and reviewers. Finally, Section 5 outlines future directions toward responsible methodological evolution in qualitative research in the age of AI. 2 EMERGING LANDSCAPE OF AI-ASSISTED QDA The current discussion on AI-assisted QDA across research communities builds on a long history of computational support for QDA, including computer-assisted qualitative data analysis (CAQDAS) tools [45], topic modeling [46], clustering [47], classification [48], sentiment analysis [49], and other machine learning or natural language processing strategies [50]. These perspectives are increasingly conse- quential because of the kind of analytical work now being delegated to AI. Generative AI and LLMs are increasingly being used in tasks that move closer to the interpretive core of QDA, including meaning making, interpretation, code sug- gestion, summarization, categorization, theme generation, and analytic synthesis [12], [13], [51]. While traditional software tools, such as older versions of MAXQDA, NVivo, and AT- LAS.ti, supported human-led analysis through structuring, cross-referencing, and some visualizations, modern AI systems seem to overtake the interpretive process, generating codes, categories, themes, and even analytical narratives. AI is no longer seen only as a low-level aid for organizing qualitative data; it now can shape what is noticed, how patterns are described, which interpretations appear plausible, and how qualitative meaning is constructed. UNDER REVIEW: HODA ET AL., 20263 TABLE 1 Risks associated with AI-assisted qualitative data analysis in software engineering and related areas, classified by the type of evidence reported. EV: Empirically validated. EB: Experience-based. UA: Unvalidated assumptions. OC: Opinion-based claims. Risk Type of Evidence EVEBUAOC Environmental and planetary sustainability costs: Training and deploying LLMs require substantial compu- tational resources, which translate into high energy consumption and non-trivial carbon emissions. [25]–[28][17] Bias encoding: The model may introduce or amplify systematic biases in its outputs.[11], [29][30]–[32] [12], [17], [33]–[35] Loss of depth: Deep qualitative insight often emerges through in-depth and prolonged engagement with data, careful attention to ambiguity, and sustained effort to reconcile contradictions and tensions within participants’ accounts. Automated shortcuts that accelerate coding or summarization risk compressing this process, thereby reducing opportunities for rich, layered interpretation. [11] [12], [32], [36] [34][13], [37] Loss of reflexivity: Since reflexivity requires self-awareness, ethical responsibility, and situated judgment, it is an inherently human process that cannot be meaningfully reproduced or automated by LLMs. [11][12], [31][17], [38] Loss of traceability and transparency: The inherent non-deterministic nature and the frequent version updates of proprietary foundational and frontier LLMs can complicate the ability of future researchers to replicate the process, as the underlying interpretive engine functions as a black box that evolves over time. [12][39][13] Loss of creativity: LLM-assisted analyses may gravitate toward familiar narratives and commonly observed themes, potentially flattening the creative and exploratory dimensions of qualitative inquiry. [11][40] Loss of privacy: Qualitative datasets in software engineering frequently contain sensitive information. When such data are processed using LLMs, particularly via external, cloud-based APIs, there is a risk that sensitive information may be exposed, stored, logged, or reused in ways that are opaque to researchers. [41][12] Loss of human expertise and learning: When key analytic activities such as coding, memoing, and preliminary interpretation are delegated extensively to LLMs, researchers, particularly novices, may miss critical opportu- nities to engage deeply with the data and to cultivate these foundational competencies. [31] [13], [35], [38], [40] Loss of nuance and diversity: LLMs generate new codes based on patterns learned from existing datasets, and they tend to favor conventional or statistically dominant interpretations. This creates a feedback loop where researchers inadvertently prioritize existing global concepts over emerging local ones, leading to a loss of nuance and diversity. [42][43] Loss of emergence: Emergence refers to the gradual, often unexpected development of concepts, relationships, or theoretical insights that were not specified in advance but arise through iterative engagement with data. This process depends on time, openness, and interaction, conditions that are difficult to preserve when analytic workflows are overly automated or optimized for speed. [36][44] Table 1 summarizes the risks captured by the emerging body of knowledge in this area, supported by varying levels of evidence. In the social sciences, this shift has led to strong methodological objections, with 419 experienced qualitative researchers from 32 countries, including prominent schol- ars and methodologists, completely rejecting the use of generative AI for reflexive qualitative research [17]. Their critique is grounded in the argument that QDA depends on human meaning-making, subjectivity, situated interpretation, and reflexive accountability. From this perspective, the central problem is not only that AI may produce inaccurate, biased, or superficial outputs, but that it cannot participate in the epistemic work that gives QDA its methodological charac- ter. Delegating interpretive tasks to AI may alter the very conditions under which qualitative knowledge is produced, not to mention the overarching issues with accuracy and hallucinations. Others adopt a less prohibitive, but still cautious, po- sition. Rather than rejecting AI assistance altogether, stud- ies such as Nguyen-Trung [42], Meyer [36] and Teixeira et al. [43] examine how AI might support selected ana- lytic activities while preserving human analytical authority. Across this work, AI is commonly framed as a possible aid for familiarization, preliminary coding, codebook development, summarization, or theme refinement. However, this framing is repeatedly accompanied by concerns that AI may narrow the analytic space, privilege conventional interpretations, reduce engagement with ambiguity, or create a false sense of analytic completeness. The central issue, therefore, is whether AI use preserves the depth, emergence, reflexiv- ity, and situated judgment that QDA demands. These risks are also being examined within SE con- texts [13], [52]–[54], driven by practical and methodological tensions. SE researchers often work with large volumes of socio-technical data in textual formats, e.g., interview transcripts, commit messages, developer communication traces, and more recently, AI-generated texts and human-AI conversations [4]. These data sources are central to under- standing human, organizational, and collaborative aspects of software development, but they also require substantial interpretive effort. Because AI is seen to handle multiple data sources and modalities (e.g., text, audio, video) at scale and speed, modern AI-assisted QDA is emerging as a promising but methodologically delicate concern within SE research. In one of the early works in this area, Bano et al. [13] dis- cuss both the opportunities created by LLMs in qualitative research and the methodological challenges they introduce. They emphasize that QDA depends on contextual under- standing, interpretive judgment, and researcher reflexivity, which cannot be directly delegated to automated systems. Lec ̧a et al. [12] confirm that LLMs have often been explored for QDA with the expectation that they can reduce manual effort and support the analysis of larger qualitative datasets. However, they also identify limitations related to variability, transparency, privacy, and reproducibility. These concerns highlight the tradeoffs inherent in AI-assisted QDA. Broader SE work on the use of LLMs in empirical studies also provides some relevant guidance and emphasizes the need to report model versions, prompts, parameters, and usage settings when LLMs are used in empirical research UNDER REVIEW: HODA ET AL., 20264 [30], [39]. Although not focused on qualitative analysis, this guidance applies in part to QDA outputs that are also sensi- tive to prompt wording, model updates, sampling decisions, and non-deterministic behavior. These risks can be understood as recurring patterns rather than as isolated limitations, i.e., as antipatterns – a di- agnostic taxonomy previously used to guide SE researchers on “what not to do” and “what to avoid” [3], [24]. 3 A CATALOG OF ANTIPATTERNS We present a catalog of antipatterns in AI-assisted QDA - emerging assumptions and practices that initially appear desirable and seemingly reasonable, but ultimately undermine the rigor and validity of qualitative data anal- ysis. Importantly, these antipatterns should not be read as an argument that AI can never be used to assist QDA. Rather, they identify the conditions under which AI assistance becomes methodologically risky and point to ways to mitigate those risks. We believe that there is a place in QDA for effective and prudent AI assistance. Best practices and guidelines in this area will necessarily emerge and evolve as researchers experiment with approaches and critique the results. In the meantime, guidance is needed on why not and how not to apply modern AI tools and techniques in QDA to avoid damage from uncritical or rushed applications. Based on our decades of qualitative SE research expertise and experience combined with an understanding of the emerging landscape of AI-assisted QDA (summarized above), we present a catalog of antipatterns in AI-assisted QDA. To provide structure and clarity, we organize eleven antipatterns into three categories that reflect the different types of issues that can arise in AI-assisted QDA, from the initial motivation for adopting AI (Dangerous Drivers), to how it is operationalized in practice (Operational Missteps), to how it can seriously threaten the overall integrity of the analytical process (Analytical Failures). Table 2 presents the details of each antipattern, including an illustrative example, an explanation of why it is tempting, associated risks, and some possible mitigation strategies. In the following subsections, we summarize the antipatterns in each category, including how reviewers can evaluate these antipatterns with commensu- rate review recommendations. 3.1 Dangerous Drivers The first category of antipatterns is Dangerous Drivers, which are rooted in the motivation for adopting AI-assisted QDA. These reflect problematic or unexamined reasons for using AI that can bias methodological decisions from the outset. Although some underlying reasons might be legitimate in some cases, the danger is that the reasons and motivations are usually implicit, thus unexamined, and assumed to be beneficial by default. One might think of the antipatterns in the Dangerous Drivers category as grounds for reviewers to demand better explanations and methodological justifications for using AI-assisted QDA. The because we can antipattern is represented by a hammer icon to symbolize Maslow’s law of the instrument, which says ‘if you have a hammer, everything looks like a nail’. As technologists, we are embedded in a socio-technical context that encourages technological experimentation, so our proverbial ‘hammer’ is whatever the latest technology buzzword is. Trying something new technologically is rarely questioned and often admired for its own sake. Using AI in QDA because we have access to AI is an easy antipattern to adopt without realizing it. Equally subtle, because it can arise from the fear of missing out and a desire to ”keep up”, is the productivity spiral 7 antipattern. This arises from the pressures that the ever-increasing use of AI can bring, leading to a vicious cycle between increased scale and speed of AI-assisted QDA on the one hand, and increased community expectations of greater scale and speed from qualitative studies on the other. This phenomenon is not limited to QDA. Concerns about the productivity spiral emerged from Anthropic’s analysis of data from 81,000 interviews about what people from all domains and walks of life want and fear from AI [55]. Ironically, Anthropic’s study itself is an extreme example of this antipattern (also evident from the brag of its scale in the title), as it demonstrates an investigative scale that may make sense for its purposes, but cannot and should not be expected from other qualitative studies. Although the Anthropic study is an obvious outlier, the increases in scale and productivity from the use of AI are often more incremental. The human as gold standard ð antipattern arises when studies compare AI-generated coding results to human- generated results based on the unquestioned assumption that all human coding is equal and of high quality. In practice, there is a marked difference in the quality of coding produced by individuals with varying levels of QDA expertise and experience. Without assessing the quality of the human coding, it is dangerous to assume that it is an acceptable benchmark. Coding done by a novice human analyst or based on a coding scheme that is mediocre at best hardly qualifies as a benchmark. Even coding judged high- quality cannot be considered the only high-quality coding of a particular corpus of data, so its use as a benchmark is equally suspect. Therefore, the quality of the AI-generated coding that is deemed ’just as good’ becomes questionable. Although tempting, benchmarking against any human an- alyst cannot guarantee good quality AI coding if there is no or low actual human engagement in the QDA process – described later as part of more serious Analytical Failures. 3.2 Operational Missteps The second category of antipatterns is Operational Mis- steps, which capture incorrect or suboptimal uses of AI during the analysis process, stemming from misconceptions about the nature of qualitative analysis, the role of the researcher, and the capabilities and limitations of AI. Most instances of antipatterns in this category are fixable because they could be embedded in a valid analytical process, but these misuses weaken overall quality. One might think of these Operational Missteps as grounds for a ”major revision” recommendation on a paper under review. The QDA as categorization V antipattern employs AI to do what AI actually seems to do really well (categorizing UNDER REVIEW: HODA ET AL., 20265 TABLE 2 A catalog of antipatterns in AI-assisted qualitative data analysis. AntipatternExampleWhy TemptingAssociated RisksPossible Mitigation Strategies DANGEROUS DRIVERS Because We Can: Using AI-assisted QDA because AI is able to process qualitative data and is readily available to human researchers, without due consideration of fit, relevance, risks, and limitations. A paper reports the use of an AI tool for open coding, describing the tool used, the prompts, the interaction, and the results, without giving a justification for why AI usage was considered and how the AI tool was selected. SE researchers are technologists at heart, and we are hard wired to explore the limits of new technology. The coolness of what AI can do, along with apparent gains in speed and scale seem sufficient. Loss of privacy, human expertise, learning, reflexivity and environmental and planetary sustainability costs, along with threats to research validity and missed opportunities for deeper insight, make it imperative to weigh AI’s apparent benefits against its real but less visible costs. Researchers should perform and share a thoughtful cost-benefit analysis to ensure AI use is meaningfully justified, rather than assuming speed and scale are sufficient rationales. 7 The Productivity Spiral: A vicious cycle between increased scale and speed of AI-assisted QDA on the one hand, and increased community expectations of greater scale and speed from qualitative studies on the other. The Anthropic study “What 81,000 people want from AI” illustrates how large-scale AI-assisted studies can become a visible marker of research ambition and prestige. Researchers may be subject to pressure to use AI to produce more to keep up with peers and expectations from institutions, funding agencies, and publication venues. Prioritizing productivity over sustained analytical engagement can lead to loss of depth, nuance, diversity, emergence, reflexivity, human expertise, learning, traceability, and transparency, while normalizing unrealistic expectations. Researchers should use qualitative values (depth, richness, reflexivity) to select and justify their QDA approach, rather than be driven by quantitative markers such as scale and speed. ð Human as Gold Standard: Assuming human data analysis is an appropriate comparison or benchmark for AI-assisted QDA without considering the expertise and experience of the human analyst(s). A well-designed empirical comparison finds an AI and a human coder assign similar codes to 90% of the data and concludes that AI coding is acceptable without considering the human analyst’s expertise, experience, and QDA quality. Manual coding is time-consuming and can be tedious. Getting what seem to be the same results using AI can be seen to justify AI use, saving substantial time. Benchmarking against human analyst(s) may be insufficient and misses an opportunity to do better, leading to bias encoding, loss of depth, nuance, diversity, and reflexivity. Other avenues for evaluating AI-generated QDA results should be explored, based on known quality dimensions for qualitative research. OPERATIONAL MISSTEPS V QDA as Categorization: Using AI to group data into ”buckets” based on internal quantification (e.g., most popular themes), training data, and existing knowledge (instead of staying true to the research data). Limiting QDA to categorization. Coarse-grained, quantitatively-led categorization missing qualitative richness, e.g., ”Types of test debt identified: missing test cases (mentioned by 60% participants) and broken test cases (10%)”. Categorization is a low hanging fruit which LLMs seem to be good at (e.g., content analysis). In low risk, non-research contexts (e.g., market analysis), it may be all that is needed. Reducing QDA to surface-level categorization can drive loss of depth, nuance and diversity; forcing data into fixed buckets can reinforce bias encoding and cause loss of emergence and loss of creativity. Deductive QDA does not end at categorization; categories should be analyzed for richness and relationships. In inductive QDA, AI can assist with individual and incremental steps (e.g., open coding and constant comparison), rather than categorizing across the dataset. " Epistemological Slide: A shift away from the constructivist foundations of qualitative research toward positivism. AI assumed by some to reflect the ‘objective truth’ and thus being employed as a ‘superior’ analyst to subjective humans. Presentation of findings marked by quantification of qualitative data. Positioning AI as ‘objective’ and free from human intervention can be used to justify its use in QDA. Reducing the role of the human in constructing knowledge and over-reliance on AI can lead to loss of nuance, diversity, depth, emergence, reflexivity and creativity and risks bias encoding. Reviewers should watch for and question overly positivist research approaches (quant heavy, claiming to be objective, looking to verify ‘the truth’) when a constructivist approach is more meaningful. e The Black Box Trap: Treating AI as an authoritative, opaque analyst, where generated codes, groupings, and interpretations are accepted at face value without fully understanding either QDA principles or the AI-based mechanisms being used. A researcher prompts a general purpose LLM to “identify themes” from interview transcripts. The researcher cannot explain the generated themes, how they relate to the data, or whether alternative interpretations exist. For researchers unfamiliar with QDA, AI can seem like a shortcut around training in methodological principles and steps. Likewise, researchers unfamiliar with AI tools may use them bluntly, without understanding the risks or how to use them well. Treating opaque AI outputs as authoritative and removing researchers from the analytic process can lead to bias encoding and loss of traceability, transparency, reflexivity, human expertise and learning while weakening the justification of coding decisions and the grounding of findings in the data. Ensure that researchers have at least a foundational understanding of both QDA and AI tools before integrating them. Require explicit documentation of prompts, intermediate steps, and researcher decisions [39], not merely explanations offered by AI. Ç Too much Trust: Overreaching analysis caused by hallucinations, generalizations, and over interpretations. A strong inverse relationship between developer experience and use of AI tools for code generation is found in data analyzed using AI. This is based on stereotypes represented in the underlying general-purpose model and is not present in the research data. Overreaching analysis can sound grand and impressive, and might be couched in a way that implies more confidence than is warranted. Without careful review, AI hallucinations, generalizations, and overinterpretations can misdirect decisions and research agendas, reinforcing bias encoding, loss of traceability and transparency, loss of nuance and diversity, and loss of reflexivity. Findings generated using AI should be vetted by humans familiar with the underlying data and with the risks of AI-based analysis. UNDER REVIEW: HODA ET AL., 20266 TABLE 2 A catalog of antipatterns in AI-assisted qualitative data analysis (Continued). AntipatternExampleWhy TemptingAssociated RisksPossible Mitigation Strategies AI Output as Insight: AI-produced QDA findings can sound insightful while remaining shallow, leading researchers and readers to mistake superficial output for meaningful analysis A researcher uses AI to generate themes from developer interview data about cloud vs on-premises servers. The polished output sounds insightful, but the findings are generic and weakly grounded in data. AI tools produce fluent, well-structured outputs that resemble legitimate qualitative findings, giving a strong illusion of depth, rigor, and expertise. Mistaking persuasive AI output for genuine insight creates loss of depth and loss of nuance and diversity; accepting superficial findings without scrutiny also drives loss of reflexivity and loss of traceability and transparency. AI-produced results should always be critiqued and guided by human researchers, pushing for depth and nuanced understanding of the underlying data [13]. ANALYTICAL FAILURES 6 Low Human Engagement: Researchers lacking enough familiarity with the raw data to be able to spot mistakes, biases, and hallucinations when applying AI to QDA. A researcher uses AI to analyze interview data collected by others without reading or becoming familiar with much of it, resulting in hallucinated findings they fail to recognize due to limited engagement with the material. Reading data is time consuming, and often tedious. Fully reading all the data may seem to erase much of the productivity gains of using AI. Because AI can produce plausible but incorrect results, detecting errors requires close familiarity with the data. Limited engagement with raw data can therefore reinforce bias encoding and drive loss of depth, loss of reflexivity, and loss of human expertise and learning. Read the data. ` Fishing for Results: Exploiting AI’s ”eagerness to please” (AKA AI sycophancy) by steering it toward confirming assumptions, prior hypotheses, or external theories rather than staying true to the data. A researcher believes a proposition is true, but it is not supported by the AI-generated findings. When prompted to consider it, the LLM complies by generating a tenuous chain of ”evidence” from the data to support the suggested proposition. QDA techniques are rigorous, in part, to counter the human tendency to look for the evidence we want in the data (confirmation bias). It is tempting to cede responsibility for countering that tendency to AI when it appears to have found what we are looking for. We know that human bias is real, and so we have built methodological infrastructure to bound it (e.g., member checking, code reviews). Using AI to legitimize preferred conclusions can bypass these safeguards, reinforcing bias encoding and driving loss of emergence, loss of reflexivity, and loss of nuance and diversity. Prompt histories should be reported when AI is used for QDA [39]. Even more important, prompting techniques should be devised and taught that do not allow researchers to transfer their own biases to AI-based analysis. AI can be trained to be aware of the researcher’s assumptions list [4]. Æ Automated QDA: Fully automated, end-to-end AI driven QDA with little to no human engagement with the overall process and analysis. A research team feeds a set of research questions and a large corpus of interview data into an AI-based QDA tool and asks it to generate a publishable paper on the analysis and results. This is taking the efficiency and scale of AI use to the max and exhibits the downsides related to all previous antipatterns. Fully automating QDA leaves AI decisions and biases unchecked, reinforcing bias encoding and causing loss of traceability and transparency; removing human engagement also drives loss of reflexivity, loss of human expertise and learning, loss of depth, and loss of emergence. As a community, we need to develop good workflows that effectively involve AI as a partner in QDA, with frequent and meaningful human-AI collaboration, both for learning and for evaluation. data into buckets), but this does not constitute QDA [11]. This is problematic based, first, on how AI performs cat- egorization and, second, on the fact that categorization is only one part of QDA. Most generative AI models approach the categorization of qualitative data through an underlying quantitative approach, surfacing most commonly appear- ing words or phrases (similar to content analysis) or by relying on their vast underlying training datasets and pre- established conceptualizations (through internet searches). This avoids remaining true to the research data, risking misinterpretations and overfitting meanings[11]. Force- fitting data into context windows (e.g., uploading entire in- terview transcripts into a single query) does not help. Even when a retrieval augmented generation (RAG) approach is applied or a custom LLM is built, risks such as the loss of nuance, creativity, emergence, and reflexivity cannot be denied (Table 1). Furthermore, reducing QDA to a categorization task is analogous to reducing software engineering to writing code. Writing code is but one important activity in SE, supported by many others such as requirements specification, negotia- tion, design, architecture, testing, deployment, maintenance, and operations. Software can be produced simply by writing code, but the results are suboptimal in most cases. Similarly, categorization is one step within QDA that is supported by many others such as understanding the underlying data, ap- plying reasoning, analytical questioning of the data, creating and refining codes, constant comparison across the dataset, memoing, raising the data through levels of abstraction, reflexive practice, and critical evaluation [4]. Does this mean AI can never be used for categoriza- tion? No. Simple categorization in the form of coarse-grained grouping of simple data – such as text responses made up of a few words – done by AI under human supervision may be suitable in cases where both the likelihood and impact of incorrect classification are low. QDA performed within scholarly studies on non-trivial data is often more complex and nuanced and demands a careful human-led approach [4]. Similarly, the epistemological slide " characterizes a shift (whether it be in a particular study or over time) towards increasing levels of positivism at the expense of the foundationally important constructivist origins of qual- itative research. This antipattern capitalizes on AI’s pen- chant for quantification, typically a hallmark of positivism. Within a single study, this antipattern might manifest as UNDER REVIEW: HODA ET AL., 20267 unnecessary quantification or quantitative language. The assumption that AI is an ‘objective’ entity and hence ‘su- perior’ to subjective humans, when AI bias encoding is well documented (section 2), further pushes QDA along the epistemological slide. Such assumptions and practices do not serve greater understanding of the questions being studied, and often bring into question why a qualitative approach was chosen in the first place. The role of the human in constructing knowledge and leveraging subjective understanding grounded in lived and professional experi- ence is a feature of constructivist research, not a bug. The black box trape antipattern is about using AI tools without really understanding how they work or what their limitations are. For example, one-shot queries that lead to ‘averagification’ of results ignore the need to incorporate context into AI-assisted analysis [11]. On the flip side, this antipattern also captures the case where AI is used by researchers to compensate for a lack of expertise, training, experience, or understanding of QDA itself, leading to a variety of methodological weaknesses, such as the QDA as categorizationV antipattern, described above. Another type of Operational Misstep is placing too much trust Ç in the tool’s output, ignoring its tendency towards reductionism and overstatement [12]. The logical extreme of the too much trustÇ antipattern is the AI output as insight antipattern: LLMs, in particular, are extremely good at masking a lack of deep insight with well-crafted and intelligent-sounding narratives that are hard to ignore. 3.3 Analytical Failures Our last category of antipatterns is Analytical Failures, i.e., fundamentally flawed uses of AI-assisted QDA. These involve breakdowns in the methodological integrity of the analysis process itself, rendering the results invalid by any standard of rigorous qualitative research. These antipatterns should typically attract a reject recommendation on a paper under review. The fishing for results `antipattern, for example, results from exploiting AI’s tendency to please (also known as AI sycophancy), to agree with suggestions from a human analyst, and even generate ”evidence” to support such sug- gestions, even when those suggestions are so biased as to be spurious. In this way, AI tools can be used to amplify human biases (e.g., confirmation bias) rather than avoid them [11]. This can be exacerbated by the low human engagement with the data 6 antipattern, which avoids the analyst’s obligation to immerse themselves in the data, to read it, and to be familiar enough with the data to be able to spot errors, biases, and hallucinations generated by the AI assistant. Finally, the pinnacle of Analytical Failures is the auto- mated QDA Æ antipattern, which results from all other Analytical Failures taken to the extreme. Automated QDA Ædescribes an approach to QDA in which the entire process, from data preparation to writing up of the results, is handed over to an AI tool with no human involvement, no human review of intermediate results, and no opportunity for course correction. 3.4 Compounding and Cascading Antipatterns Antipatterns hardly exist in isolation. Rather, they often co- manifest, imposing a cascading and combined effect. For example, because we can represents a technology push or AI first attitude and approach to QDA that risks the manifestation of other antipatterns. AI-enabled QDA can be seen as a silver bullet to overcome chronic limitations of manual QDA such as speed and scale, making antipatterns such as the productivity spiral7 seem justified and logical. Similarly, human as gold standardð can accompany either or both of these Dangerous Drivers. For example, once an AI native approach is deployed, it is easy to fall into the trap of trusting a benchmarking exercise against a human analyst(s), without consideration for their level of QDA expertise, to calibrate and accept AI outputs. Dangerous drivers such as because we can can also cause a cascading effect onto other antipatterns, such as too much trustÇ , an Operational Misstep, or low human engagement with the data6 , an Analytical Failure. QDA as categorizationV is likely to trigger the episte- mological slide " . In QDA as categorizationV , concepts most commonly occurring in the data are typically grouped to create categories. Such an approach shifts the QDA goal from developing rich and layered understanding and con- ceptualization toward listing coarse-grained categories that abstract away nuance and may draw on pre-existing ideas from its training. Thus, quantification of qualitative data happens quietly in the background through representing averages, percentages, coverage, means, medians, statistical significance, or even inter-coder reliability statistics. Simi- larly, even if the human as gold standard ð is partially ad- dressed by comparing to an expert human analyst, without continuous and significant human engagement, it can lead to low human engagement with the data 6 or automated QDAÆ , more serious Analytical Failures. Human as gold standard ðcan also lead to an epistemological slide " as the human analyst’s view is treated as the benchmark or baseline truth (positivism), without leaving room for multiple interpretations (constructivism). Individually and together, these antipatterns move the epistemological dial of inquiry from constructivism toward positivism. As mentioned earlier, AI output as insight is a logical extreme of the too much trustÇ antipattern, while the black box trap e can help explain why these two antipatterns may manifest. By treating AI as a black box and not apply- ing critical thinking, human analysts can become prone to too much trust Ç and consequently accepting any and all AI output as insight . These antipatterns are also likely to form a vicious cycle with low human engagement with the data 6 , as the human analyst over-trusts and over-relies on AI, but is not familiar enough with the data to recognize when AI has gone off the rails. The examples above suggest that methodological weak- nesses introduced early in an AI-assisted QDA workflow may propagate throughout the remainder of the analysis. Decisions that initially appear minor, such as adopting AI without clear justification or accepting its outputs un- critically, can progressively distance researchers from the data, ultimately compromising grounding, reflexivity, trans- parency, and the overall trustworthiness of qualitative anal- ysis. UNDER REVIEW: HODA ET AL., 20268 3.5 Scope, Application, and Limitations While our goal is to provide a structured vocabulary and analytical foundation for reasoning about problematic attempts at AI-assisted QDA, we do not claim that the proposed taxonomy or the relationships described are definitive or exhaustive. Similarly, some of these antipatterns may arise in other AI-assisted research processes,forexampleinquantitativeresearch. Nevertheless, this catalog is situated within QDA as the antipatterns are drawn from an understanding of the possible violations of qualitative research principles. Applicability beyond software engineering research. The authors are all SE researchers and thus are qualified to propose guidance for our own discipline. However, many of the presented antipatterns appear to be more widely relevant. For example, the productivity spiral 7 is observed in the results of the Anthropic interview study as a phenomenon cutting across disciplines. Researchers in related disciplines such as human computer interaction, human robot interaction, information systems, AI, etc. may also find these antipatterns relevant. Evolving nature of AI-assisted QDA antipatterns. The antipatterns identified in this paper were informed by the capabilities, interaction paradigms, and limitations of contemporary AI systems. However, AI technologies are evolving rapidly, and advances in reasoning, context management,transparency,multimodalcapabilities, and autonomous workflows are likely to reshape how researchers interact with these systems in the future. Consequently, some antipatterns described in this work may become less relevant, manifest differently, or even disappear as AI systems mature, while entirely new antipatterns may emerge, e.g., with the proliferation of agentic, neurosymbolic, or other forms of AI in the future. Rather than representing a fixed taxonomy, we, therefore, view the antipattern catalog as a conceptual framework that should evolve over time alongside advances in AI technologies and empirical evidence from their use. Compounding relationships between antipatterns. Finally, the interactions among antipatterns described in section 3.4 are conceptual and intended to illustrate plausible ways in which problematic AI-assisted QDA practices may reinforce one another. In practice, some antipatterns may co-occur more frequently than others, different interaction patterns may emerge, and additional cascading effects may exist beyond those discussed in this paper. These aspects also highlight opportunities for future research. We therefore view this paper not as a definitive or exhaustive taxonomy of AI-assisted QDA antipatterns, but as an initial conceptual foundation intended to stimulate empirical investigation, methodological refinement, theoret- ical debate, and the development of rigorous human–AI collaboration practices for QDA. We expect formal experi- mentation with emerging AI tools conducted and reported by the SE community in the future to help improve the conceptual depth and breadth of this catalog. 4 RECOMMENDATIONS The antipatterns presented in this paper highlight not only ways in which AI-assisted QDA can be problematic or fail, but also how such problems and failures can be recognized and mitigated. In this section, we translate these insights into actionable recommendations for both researchers and reviewers. 4.1 Recommendations for Researchers The antipatterns provide a lens for researchers to anticipate, recognize, and avoid problematic uses of AI-assisted QDA. By reflecting on the motivations, assumptions, and practices that give rise to these antipatterns, researchers can better safeguard the integrity of their analysis and maintain alignment with the epistemic foundations of qualitative research. · QDA should follow a human-in-the-lead approach: A human-in-the-loop approach to QDA assumes that the process is AI-driven, with a human analyst primarily as a reviewer or quality assurance auditor of ‘truth’ (rooted in positivism). This ignores the role of the human analyst in shaping the analysis (constructivism), manifesting an epistemological slide ". Even assuming highly sophisticated AI tools in the future (i.e., zero hallucination, sycophancy, or inaccuracy), a human simply reviewing AI’s steps and outputs will have little influence over the human knowledge building required to construct the findings. They can also suffer from possible cognitive fatigue, cognitive offloading, overreliance, erosion of critical thinking, and loss of motivation [56], [57]. This would offset much of the perceived efficiency gains. Even if the human analyst overcomes these practical challenges and is satisfied with both the AI’s process and outcomes, this approach still does not leverage what it means for a human to understand the data, derive meaning, enable emergence and reflexivity, and capture nuance and creativity, which are core QDA values. A human-led AI-assisted QDA approach counters these risks, preserves the values of qualitative research, and celebrates the motivation and joy of research work. · Build expertise in both QDA and AI before applying AI-assisted QDA: AI-assisted QDA should not be treated as a shortcut to learning either QDA or AI. To lead QDA, researchers need sufficient methodological and technical literacy to question AI outputs, rather than treating them as neutral evidence or finished findings. This does not mean that researchers must master every technical detail of AI systems. Rather, they need enough understanding to recognize that fluent sounding outputs may still be incomplete, biased, or weakly grounded in the data. They also need enough grounding in QDA to notice when AI use reduces interpretation to categorization, obscures alternative readings, or weakens the connection between evidence and analytic claims. Without such critical expertise, researchers are more likely to engage in QDA as categorization V , accept AI output as insight , place too much trust Ç in the system, or fall into the black box trap e . Critical AI use therefore depends on researchers’ capacity to ask informed questions about how AI has UNDER REVIEW: HODA ET AL., 20269 shaped the analysis, where its outputs require verification, and when its use may be methodologically inappropriate. · Do not give in to peer pressure or the fear of missing out: Methodological issues in AI-assisted QDA often originate before analysis begins, at the level of the motivations for using AI, often driven by peer pressure and the fear of missing out on new research trends and tools (e.g., from seeing other researchers and publications using AI, even if without clear justification or careful application). Antipatterns such as because we can and productivity spiral7 reflect unexamined or convenience-driven reasons for adopting AI that can jeopardize the entire research process from the start. Research design should be built thoughtfully and carefully, with each element thoroughly justified and thought through. The design rationales need to be honest and convincing to those outside the research team. This is true, of course, even when AI is not part of the process, but becomes crucial in the presence of AI because these antipatterns can be subtle and easy to miss. They can lead to premature or inappropriate integration of AI into the research workflow, increasing the likelihood of downstream methodological issues. · Contain issues from cascading and compounding: Antipatterns rarely occur in isolation. Issues originating at earlier stages of research, such as problematic motivations, can propagate and manifest as Operational Missteps or full Analytical Failures at later stages of the research process. Researchers should not only address issues at the point where they become visible, but also reflect on upstream decisions and assumptions that may have contributed. This perspective encourages a more holistic view of methodological rigor, where preventing early-stage temptations and missteps can reduce the likelihood of severe downstream failures. · Recognize the impact beyond a single study (community concerns): Not every antipattern may manifest in every study. However, antipatterns manifesting across different studies over time can lead to the degradation and deval- uation of qualitative research in the community. A related concern is the loss of qualitative research skills from the community. Qualitative data analysis skills, if increasingly outsourced to AI, will lead to them eroding from the com- munity’s methodological repertoire, one study at a time, one researcher at a time. This in turn erodes the ability of qualitative researchers to remain effective arbiters and evaluators of AI-generated results, even if much of the analytical labor is outsourced to AI. 4.2 Recommendations for Reviewers The increasing use of AI in QDA introduces new challenges for the evaluation of empirical studies, particularly inassessingrigor,transparency,andvalidity.The antipatterns identified in this paper provide reviewers with a structured lens and vocabulary to identify and articulate methodological concerns specific to AI-assisted QDA. Rather than focusing on whether AI was used, the key task for reviewers is to examine the risks that may have been introduced in the analytical process as a result of its application. By leveraging the antipattern catalog, reviewers can more systematically identify and name problematic assumptions, missteps, and failures, and provide more precise and actionable feedback on the credibility of AI- assisted QDA. · Identify red flags of analytical failure: Certain patterns should be treated as strong indicators of invalid or methodologicallyunsoundanalysis.Theseinclude fully automated analysis pipelines, absence of a clear chain of evidence, lack of human engagement with raw data, and findings that are not clearly grounded in the data. Such cases correspond to the Analytical Failures identified in this paper and are not acceptable as scholarly qualitative research. Their presence substantially weakens the credibility of the study and should warrant rejection. · Assess the degree of human engagement in QDA: Reviewers should pay close attention to the role of the human researcher in the analytical process. Minimal engagement with the data or over-reliance on AI outputs may signal antipatterns such as low human engagement with the data 6and automated QDA Æ . Evidence of sustained human involvement, familiarity with the data, and critical interpretation of AI outputs are essential indicators that the analysis has not been inappropriately delegated to the AI system. · Focus on traceability and accountability: A central concern in evaluating AI-assisted QDA is whether the analytical process is transparent and traceable. Reviewers should assess whether the study provides sufficient information to understand how AI was used, how outputs were generated, and how they were interpreted in relation to the data. Lack of traceability and transparency should be treated as a significant limitation, as it prevents meaningful evaluation of the validity and credibility of the findings. In particular, missing details about prompts, interactions, and intermediate analytical steps may indicate deeper methodological issues. · Use the antipatterns as a diagnostic lens: For reviewers, the antipattern catalog provides a structured vocabulary for identifying and articulating methodological concerns in AI- assisted QDA. Rather than relying on general impressions, reviewers can use specific antipatterns to diagnose where and how an application may be problematic. For example, inadequate documentation of methods may indicate the black box trap e , lack of methodological details about how AI-generated codes, findings, or themes were evaluated by human researchers may reflect too much trust Ç , and absence of explicit links between data and findings may suggest low human engagement with the data6 . · Evaluate holistically: Since antipatterns can have cascading and compounding effects, reviewers should look out for how the manifestation of one antipattern may hint at another. This will enable reviewers to evaluate the study more holistically and question both apparent and hidden antipatterns. UNDER REVIEW: HODA ET AL., 202610 · Help authors improve: As the community begins to experiment with the use of AI for QDA, some grace is in order. Reviewers should avoid knee-jerk responses either for or against the use of AI. Instead, give authors guidance and, where possible, ask for a revision. For example, many instances of Dangerous Drivers could be addressed by asking authors to more carefully share their research design rationale. Similarly, at least some Operational Missteps can be remedied in a revision that better explains, for example, how human and AI collaborated. The community would benefit from good published examples, and reviewers can facilitate that process with a focus on improvement rather than exclusion. On the other hand, bad published examples can do a lot of long-term damage to the field, so reviewers also need to take their role as gatekeepers seriously and thoughtfully. Ultimately, the goal of this work is not to discourage the use of AI in qualitative research, but to make visible the ways in which its misuse can undermine methodological rigor. By framing these risks through a structured catalog of antipatterns, we aim to support more critical, transparent, and accountable use of AI in qualitative data analysis. 5 FUTURE DIRECTIONS The antipatterns identified in this paper highlight some critical issues in the way AI is currently, and potentially, integrated into qualitative data analysis. Achieving rigorous and meaningful human–AI collaboration in QDA requires a shift from reacting to problematic practices toward proactive development of frameworks, methods, and evaluation criteria. In this section, we outline key steps towards this goal, emerging from the limitations and risks exposed by the antipatterns catalog. Improve and articulate intentions. A recurring theme across the identified antipatterns is the presence of weak or unexamined motivations for adopting AI, often driven by convenience, novelty, or perceived efficiency gains (e.g., because we can , productivity spiral 7 ). Future research should focus on articulating purposeful uses of AI in QDA. This includes developing conceptual frameworks to articulate when and why AI should be used, how its role aligns with research objectives, how its use supports the core purpose of doing QDA, and how to distinguish between superficial and substantive contributions resulting from the use of AI in qualitative research. Such work is essential to move the field beyond opportunistic adoption toward intentional and methodologically grounded practice. Design principles for human-AI collaboration. While current uses of AI in QDA often emphasize efficiency and scale, the antipatterns reveal that such gains can come at the cost of analytical depth, contextual richness, and methodological rigor. Future work should articulate what constitutes meaningful human-AI collaboration in qualitative data analysis, focusing on interaction models, division of analytical responsibilities, and workflows that preserve the interpretive and reflexive nature of QDA. In particular, there is a need to better understand how AI can support, rather than replace, core analytical processes, avoiding antipatterns such as the black box trap e or automated QDA Æ . Moving in this direction is critical to ensure that AI augments, rather than undermines, the epistemic foundations of qualitative research. Develop systematic evaluation guidelines. The integration of AI into QDA introduces new challenges for evaluating the quality and validity of qualitative studies. There is limited guidance on how to assess AI-assisted analysis, leading to inconsistent and likely extreme or superficial evaluation practices. Future work should focus on developing evaluation and reporting guidelines that explicitly account for the role of AI in the analytical process. These criteria should enable reviewers and researchers to systematically identify antipatterns, assess the degree of human engagement, evaluate traceability and grounding in data, and determine whether AI use contributes to or detracts from the overall quality of the study. Establishing such evaluation mechanisms is essential to supporting consistent, transparent, and rigorous evaluation of human- AI collaboration in qualitative research. Together, these directions point toward a broader re- search agenda aimed at enabling the responsible evolution of QDA in the age of AI. Identifying antipatterns is a critical first step towards the difficult (and exciting) work of articulating principles, guidelines, and exemplars of what constitutes purposeful, meaningful, and valuable use. By embarking on this journey, the research community will be able to better harness the potential of AI while preserving the methodological integrity of qualitative research. ACKNOWLEDGMENTS We express our gratitude to Margaret-Anne Storey, Christoph Treude, and Fabio Palomba for their thoughtful feedback on drafts. REFERENCES [1]K. Charmaz, Constructing Grounded Theory, 2nd ed. Sage, 2014. [2]M. B. Miles, A. M. Huberman, and J. Salda ̃ na, Qualitative Data Analysis: A Methods Sourcebook, 3rd ed. Sage, 2014. [3]R. Hoda, “Socio-technical grounded theory for software engineer- ing,” IEEE Transactions on Software Engineering, vol. 48, no. 10, p. 3808–3832, 2022. [4]—, Qualitative Research with Socio-Technical Grounded Theory. Springer, 2024. [5]C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” IEEE Transactions on Software Engineering, vol. 25, no. 4, p. 557–572, 1999. [6]Y. S. Lincoln and E. G. Guba, Naturalistic Inquiry. Sage, 1985. [7]V. Braun and V. Clarke, “Using thematic analysis in psychology,” Qualitative Research in Psychology, vol. 3, no. 2, p. 77–101, 2006. [8]—, “Reflecting on reflexive thematic analysis,” Qualitative Re- search in Sport, Exercise and Health, vol. 11, no. 4, p. 589–597, 2019. [9]J. Jiao, “An introduction to discourse analysis: Theory and method,” Language in Society, vol. 41, no. 3, p. 402–403, 2012. [10] C. K. Riessman, Narrative Analysis, ser. Qualitative Research Meth- ods. Newbury Park, CA: Sage Publications, 1993, vol. 30. [11] C. F. Barros, B. B. Azevedo, V. V. G. Neto, M. Kassab, M. Kali- nowski, H. A. D. do Nascimento, and M. C. Bandeira, “Large language model for qualitative research: A systematic mapping study,” in Proceedings of the 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineer- ing (WSESE), 2025, p. 48–55. UNDER REVIEW: HODA ET AL., 202611 [12] M. De Morais Leca, L. Valenc ̧a, R. Santos, and R. de Souza Santos, “Applications and implications of large language models in qual- itative analysis: A new frontier for empirical software engineer- ing,” in Proceedings of the 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE), 2025. [13] M. Bano, R. Hoda, D. Zowghi, and C. Treude, “Large language models for qualitative research in software engineering: exploring opportunities and challenges,” Automated Software Engg., vol. 31, no. 1, Dec. 2023. [Online]. Available: https: //doi-org.proxy.library.vcu.edu/10.1007/s10515-023-00407-8 [14] L. S. Nowell, J. M. Norris, D. E. White, and N. J. Moules, “Thematic analysis: Striving to meet the trustworthiness crite- ria,” International journal of qualitative methods, vol. 16, no. 1, p. 1609406917733847, 2017. [15] E. G. Guba and Y. S. Lincoln, Fourth Generation Evaluation.Sage Publications, 1989. [16] J. W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 4th ed. Sage, 2014. [17] T. Jowsey, V. Braun, V. Clarke, D. Lupton, and M. Fine, “We reject the use of generative artificial intelligence for reflexive qualitative research,” Qualitative Inquiry, vol. 0, no. 0, p. 10778004251401851, 0. [Online]. Available: https://doi.org/10.1177/10778004251401851 [18] “AI assist: AI-powered qualitative data analysis with MAXQDA,” https://w.maxqda.com/products/ai-assist, Apr 2026. [19] “How to do thematic analysis in qualitative research with NVivo + Lumivero AI assistant,” https://lumivero.com/resources/blo g/how-to-do-thematic-analysis-in-qualitative-research-with-nvi vo-lumivero-ai-assistant/, Apr 2026. [20] “Introducing: AI coding beta powered by OpenAI,” https://atla sti.com/ai-coding-powered-by-openai, Apr 2026. [21] T. Ornelas, A. A. Ara ́ ujo, J. Ara ́ ujo, M. Ara ́ ujo, B. Trinkenreich, and M. Kalinowski, “LLM-assisted thematic analysis: Opportunities, limitations, and recommendations,” 2025. [Online]. Available: https://arxiv.org/abs/2511.14528 [22] P. Runeson and M. H ̈ ost, “Guidelines for conducting and reporting case study research in software engineering,” Empirical Software Engineering, vol. 14, no. 2, p. 131–164, 2009. [23] H. Sharp, Y. Dittrich, and C. R. De Souza, “The role of ethno- graphic studies in empirical software engineering,” IEEE Transac- tions on Software Engineering, vol. 42, no. 8, p. 786–804, 2016. [24] M.-A. Storey, R. Hoda, A. Maciel Paz Milani, and M. T. Baldas- sarre, “Guiding principles for mixed methods research in software engineering,” Empirical Software Engineering, vol. 30, no. 5, p. 138, 2025. [25] G. LLC, “Our 2024 environmental report,” https://blog.google/ outreach-initiatives/sustainability/2024-environmental-report/, 2024, accessed Dec 2025. [26] —, “2025 environmental report,” https://sustainability.google/ reports/google- 2025- environmental- report/, 2025, accessed Dec 2025. [27] J. Shi, Z. Yang, and D. Lo, “Efficient and green large language models for software engineering: Literature review, vision, and the road ahead,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, 2025. [28] E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in nlp,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, p. 3645–3650. [29] M. Alshaikh, T. Alshaher, R. Vieira, B. Santana, C. Xavier, J. Aman- cio, G. Carneiro, J. Leite, S. Freire, and M. Mendonca, “Prompt en- gineering strategies for LLM-based qualitative coding of psycho- logical safety in software engineering communities: A controlled empirical study,” in Proceedings of the 30th International Conference on Evaluation and Assessment in Software Engineering, 2026. [30] S. Baltes, F. Angermeir, C. Arora, M. M. Bar ́ on, C. Chen, L. B ̈ ohme, F. Calefato, N. Ernst, D. Falessi, B. Fitzgerald, D. Fucci, M. Kalinowski, S. Lambiase, D. Russo, M. Lungu, L. Prechelt, P. Ralph, R. van Tonder, C. Treude, and S. Wagner, “Guidelines for empirical studies in software engineering involving large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2508.15503 [31] T. Ornelas, A. Ara ́ ujo, J. Ara ́ ujo, M. Ara ́ ujo, B. Trinkenreich, and M. Kalinowski, “LLM-assisted thematic analysis: Opportunities, limitations, and recommendations,” in 2026 IEEE/ACM Interna- tional Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE), 2026. [32] S. Pizard, R. Moreira, F. Galiano, I. Sastre, and L. Etcheverry, “On the use of large language models for qualitative synthesis,” in 2026 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE), 2026. [33] R. Navigli, S. Conia, and B. Ross, “Biases in large language models: Origins, inventory, and discussion,” J. Data and Information Quality, vol. 15, no. 2, Jun. 2023. [Online]. Available: https://doi.org/10.1145/3597307 [34] R. M. Davison, H. Chughtai, P. Nielsen, M. Marabelli, F. Iannacci, M. van Offenbeek, M. Tarafdar, M. Trenz, A. A. Techatassanasoontorn, A. D ́ ıaz Andrade, and N. Panteli, “The ethics of using generative AI for qualitative data analysis,” Information Systems Journal, vol. 34, no. 5, p. 1433–1439, 2024. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10 .1111/isj.12504 [35] D. Kang, Z. Han, J. Tian, M. Zhang, and J. M. Rzeszotarski, “Themeviz: Understanding the effect of human-AI collaboration in theme development with an LLM-enhanced interactive visual system,” Proc. ACM Hum.-Comput. Interact., vol. 9, no. 7, Oct. 2025. [Online]. Available: https://doi-org.proxy.library.vcu.edu /10.1145/3757675 [36] T. Meyer, C. Baker, and J. Keefe, “Enhancing thematic analysis with local LLMs: A scientific evaluation of prompt engineering techniques,” in Usability and User Experience, ser. AHFE Open Access, T. Z. Ahram and C. Falc ̃ ao, Eds., vol. 194.USA: AHFE International, 2025. [37] R. Sinha, I. Solola, H. Nguyen, H. Swanson, and L. Lawrence, “The role of generative AI in qualitative research: Gpt-4’s contributions to a grounded theory analysis,” in Proceedings of the 2024 Symposium on Learning, Design and Technology, ser. LDT ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 17–25. [Online]. Available: https://doi.org/10.1145/3663433.36 63456 [38] E. M. Bender, T. Gebru, A. McMillan-Major, and M. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021, p. 610–623. [39] S. Wagner, M. M. n. Bart ́ on, D. Falessi, and S. Baltes, “Towards evaluation guidelines for empirical studies involving LLMs,” in Proceedings of the 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, ser. WSESE ’25.IEEE Press, 2025, p. 24–27. [Online]. Available: https://doi.org/10.1109/WSESE66602.2025.00011 [40] B.Trinkenreich,F.Calefato,G.Hanssen,K.Blincoe, M. Kalinowski, M. Pezz ` e, P. Tell, and M.-A. Storey, “Get on the train or be left on the station: Using LLMs for software engineering research,” in Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, ser. FSE Companion ’25.New York, NY, USA: Association for Computing Machinery, 2025, p. 1503–1507. [Online]. Available: https://doi.org/10.1145/3696630.3731666 [41] Z. Rasheed, M. Waseem, A. Ahmad, K.-K. Kemell, W. Xiaofeng, A. N. Duc, and P. Abrahamsson, “Can large language models serve as data analysts? a multi-agent assisted approach for qualitative data analysis,” 2025. [Online]. Available: https: //arxiv.org/abs/2402.01386 [42] K. Nguyen-Trung, “Chatgpt in thematic analysis: Can AI become a research assistant in qualitative research?” Quality & Quantity, vol. 59, p. 4945–4978, 2025. [43] M. C. Teixeira, M. Tschopp, and A. Jobin, “Generative artificial intelligence in qualitative research methods: Between hype and risks?” 2025. [Online]. Available: https://arxiv.org/abs/2511.084 61 [44] C. Treude and M.-A. Storey, “ Generative AI and Empirical Software Engineering: A Paradigm Shift ,” in 2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware).Los Alamitos, CA, USA: IEEE Computer Society, Nov. 2025, p. 233–239. [Online]. Available: https://doi.ieeecomputersociety.or g/10.1109/AIware69974.2025.00033 [45] N. G. Fielding and R. M. Lee, Computer Analysis and Qualitative Research, 1st ed. USA: Sage Publications, Inc., 1998. [46] S. I. Nikolenko, S. Koltcov, and O. Koltsova, “Topic modelling for qualitative studies,” J. Inf. Sci., vol. 43, no. 1, p. 88–102, Feb. 2017. [Online]. Available: https://doi.org/10.1177/0165551515617393 [47] G. E. Mitchell and H. P. Schmitz, “Using model-based clustering to improve qualitative inquiry: Computer-aided qualitative data analysis, latent class analysis, and interpretive transparency,” UNDER REVIEW: HODA ET AL., 202612 VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, vol. 34, no. 1, p. 162–169, 2023. [Online]. Available: https://doi.org/10.1007/s11266-021-00409-8 [48] K. Crowston, X. Liu, and E. E. Allen, “Machine learning and rule-based automated coding of qualitative data,” Proceedings of the American Society for Information Science and Technology, vol. 47, no. 1, p. 1–2, 2010. [Online]. Available: https://asistdl.onlinelibr ary.wiley.com/doi/abs/10.1002/meet.14504701328 [49] I. Becker, S. Parkin, and M. A. Sasse, “Combining qualitative coding and sentiment analysis: Deconstructing perceptions of usable security in organisations,” in The LASER Workshop: Learning from Authoritative Security Experiment Results (LASER 2016).San Jose, CA: USENIX Association, May 2016, p. 43–53. [Online]. Available: https://w.usenix.org/conference/laser2016/prog ram/presentation/becker [50] M. Marathe and K. Toyama, “Semi-automated coding for qualitativeresearch:Auser-centeredinquiryandinitial prototypes,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI ’18.New York, NY, USA: Association for Computing Machinery, 2018, p. 1–12. [Online]. Available: https://doi.org/10.1145/3173574.3173922 [51] S. T. U. Shah, M. Hussein, A. Barcomb, and M. Moshirpour, “QuaRUM: Qualitative data analysis-based retrieval-augmented UML domain model from requirements documents,” Automated Software Engineering, vol. 33, no. 2, p. 44, 2026. [Online]. Available: https://doi.org/10.1007/s10515-025-00587-5 [52] A. Adeseye, J. Isoaho, and T. Mohammad, “LLM-assisted qualitative data analysis: Security and privacy concerns in gamified workforce studies,” Procedia Computer Science, vol. 257, p. 60–67, 2025, the 16th International Conference on Ambient Systems, Networks and Technologies Networks (ANT)/ the 8th International Conference on Emerging Data and Industry 4.0 (EDI40). [Online]. Available: https://w.sciencedirect.com/scie nce/article/pii/S1877050925007471 [53] D. Garcia Quevedo and J. Kuri, Using LLMs in Qualitative Analysis. Cham: Springer Nature Switzerland, 2026, p. 75–79. [Online]. Available: https://doi.org/10.1007/978-3-032-08872-7 6 [54] T. Ngo, A. M. Nguyen, D. Nguyen Van, and A. Nguyen- Quoc, “Qualitative coding analysis through open-source large language models: A user study and design recommendations,” in Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’26. New York, NY, USA: Association for Computing Machinery, 2026. [Online]. Available: https://doi.org/10.1145/3772363.3798320 [55] S. Huang, S. Carter, J. Eaton, S. Pollack, D. C. I, N. Makagiansar, M. Gonzalez, S. Carr, J. Hong, K. Handa, M. McCain, T. Millar, M. Julapalli, G. Yun, A. Alt, C. Larsson, J. Leibrock, M. Gallivan, T. Sumers, E. Durmus, M. Kearney, J. H. Shen, J. Clark, M. Stern, and D. Ganguli. (2026) What 81,000 people want from ai. [Online]. Available: https://anthropic.com/features/81k-interviews [56] M. Gerlich, “AI tools in society: Impacts on cognitive offloading and the future of critical thinking,” Societies, vol. 15, no. 1, p. 6, 2025. [57] F. Dell’Acqua, E. McFowland I, E. R. Mollick, H. Lifshitz-Assaf, K. Kellogg, S. Rajendran, L. Krayer, F. Candelon, and K. R. Lakhani, “Navigating the jagged technological frontier: Field ex- perimental evidence of the effects of AI on knowledge worker productivity and quality,” Harvard business school technology & operations mgt. Unit working paper, no. 24-013, 2023.