Paper deep dive
On the Prospects of Dynamic LLM Conversations in Software Development
Annemarie Wittig, Alina Mailach, Janet Siegmund, Norbert Siegmund
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a \emph{context}-aware group received intent-based conversation augmentation, (2) a \emph{proactive} group received follow-up suggestions and tailored advice, and (3) a \emph{control} group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the \emph{proactive} treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.
Tags
Links
- Source: https://arxiv.org/abs/2608.30756v1
- Canonical: https://arxiv.org/abs/2608.30756v1
Trouble viewing inline? Open PDF directly ā
Full Text
76,967 characters extracted from source content.
Expand or collapse full text
On the Prospects of Dynamic LLM Conversations in Software Development Annemarie Wittig Ā·Alina Mailach Ā· Janet Siegmund Ā·Norbert Siegmund AbstractLarge language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively sup- port their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developersābe it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) acontext-aware group re- ceived intent-based conversation augmentation, (2) aproactivegroup received follow-up suggestions and tailored advice, and (3) acontrolgroup without in- tervention. Our augmentations are minimal: (i) to reduce confounding factors and (i) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with theproactivetreatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer- A. Wittig Leipzig, Leipzig University, Leipzig, 04109, Germany E-mail: annemarie.wittig@cs.uni-leipzig.de A. Mailach Leipzig, Leipzig University, Leipzig, 04109, Germany J. Siegmund Chemnitz University of Technology, Chemnitz, 09107, Germany N. Siegmund ScaDS.AI Dresden/Leipzig, Leipzig University, Leipzig, 04109, Germany arXiv:2608.30756v1 [cs.SE] 31 Aug 2026 2Annemarie Wittig et al. LLM interactions show observable effects, such that more severe augmenta- tions may have the potential to substantially improve developer satisfaction. KeywordsLLMĀ·DevelopersĀ·Conversation GuidanceĀ·Coding Assistance 1 Introduction The growing popularity of large language models (LLMs) has made them es- sential to assist software engineers in their daily work. These models have proven useful for many software engineering tasks, including code genera- tion[4,5,23,25], requirements elicitation[2], debugging[5,23], and documen- tation[15,25,28]. Research Track 0 10 20 30 40 Number of Papers ICSE ICSE ICSE ASE 2023 FSE FSE LLM-focused Paper Conversationally Focused Paper Updated Conversations Fig. 1Number of research papers on LLMs published in FSEā24, ICSEā24, and ASEā23 A number of studies have been conducted that focus on how developers rely on LLM- based tools, investigating usage and interac- tions[ 24], and the impact on productivity: Find- ings regarding the usefulness are mixed, with some studies finding no[6], little[15], or consid- erable improvement[31]. Despite such numer- ous research activities, only few papers focused on how to guide users through the interaction with an LLM to increase their usefulness.Fig- ure 1shows that, among 43 LLM-centered pa- pers presented at the research tracks of recent ICSE, ASE, and FSE conferences, only seven analyze the conversational structure and devel- oper interactions with LLMs. Just two of these papers attempt to improve these interactions. This underscores a critical gap: While LLMs are widely applied, the central question of how users interact with them and how to better support this interaction remains underexplored. Closing this gap is essential to unlock the full potential of LLM-developer interactions. The lack of research prioritizing the guid- ance of developers in their conversations with LLMs is surprising, as it has been shown in different research fields that sup- porting chat conversations of individuals can yield more positive results for both sides of the conversation, for example, by causing a better understand- ing of conversational contexts[ 3,10,12,20]. Moreover, studies in software en- gineering found that developers often do not provide suļ¬icient context to an LLM, reducing its ability to effectively solve a development task[ 1,15,18,19]. Despite the advances in prompting techniques, retrieval augmented genera- tion, and agentic development, there is limited insight into how conversations may benefit from additional guidance. Thus, we ask whether and how guided On the Prospects of Dynamic LLM Conversations in Software Development3 developerāLLM conversations benefit software development, and how they af- fect developer behavior and perceptions as well as the resulting code. In this study, we investigate the effects of two approaches intended to support developer-LLM interactions: The first approach, referred to ascontext treatment, enhances the system message provided to the LLM with context based on a user-selected intention chosen before each conversation. The second treatment, referred to asproactive treatment, provides additional guidance by recommending improved and follow-up prompts, as well as advice tailored to the userās conversation. We show such a workflow inFigure 2, highlighting the different mechanisms of each treatment. Our goal is to understand how these direct interventions into interactions with an LLM can affect software developers and the software development process. To this end, we evaluate both treatments against a control group in a large study with computer science students in their 3rd semester working on afull-stack web development projectforfour consecutive months. This study design, focusing on an entire project rather than a small programming task, and lasting for four months instead of a few hours, is a distinguishing feature of this work. It enables us to draw long-term effects in a more practical setting. Despite relying on students, our setting has high ecological validity due to the size, complexity, variety, and length of the task. We provided developers with a user interface to interact with the adjusted LLM and logged all interactions. Additionally, we conducted a monthly survey tracking the userās experience throughout the experiment. Overall, our treatments are designed to follow a conservative approach with a focus on internal validity, so we explore whether even minimal interventions can already affect the development process. Although we do not expect large effect sizes with such a setup, we thereby minimizing the risk of introducing too many confounding factors (e.g., the way interventions are provided or whether they disturb the LLM interaction) and can better pinpoint effects to certain treatments. Based on this controlled design, we provide the foundation for research on extensive and targeted interventions to unlock further productivity gains with LLMs. The main goal of the study is to identify whether and to what degree even small interventions in the communication of an LLM affect development. In summary, we make the following contributions: āA description of two novel approaches to guide developers during software development with LLMs. āAn analysis of longitudinal usage data from three different LLM assistants (control,context,proactive) over a four-month, realistic software develop- ment project. āA comprehensive replication package containing an application with both guidance approaches and all material and data from the empirical study. 1 1 https://anonymous.4open.science/r/On-the-Prospects-of-Dynamic-LLM-Conversat ions-in-Software-Development-D22C 4Annemarie Wittig et al. 2 Related Work Pitfalls of LLM Usage.Several studies have observed that LLMs misunder- stand what users want and need to successfully achieve their goal. For in- stance, LLMs misunderstand requirements[15,24], task context[24], and prob- lems, goals, and rationales resulting in faulty conversation[6,24]. Particularly in more complex software engineering tasks, LLMs appear to be less beneficial compared to simple coding tasks[30]. These studies provide valuable insights in the context of developer-LLM interactions, but none of these take a close look at how we can guide developers to overcome these challengesāthe key motivation for our work. Thus, we focus on automatically intercepting in de- velopersā interactions to provide situation-specific support in complex software engineering projects. Improving LLM Interactions for Development.Several studies focus on fine- tuning LLMs by incorporating specific information sources to enhance LLM capabilities. Kong et al.[14] use pre-collected data to fine-tune an LLM, such that it automatically refines the latest prompt of a conversation using role prompting [13], a technique that assigns a model a role via prompts to guide its responses in line with that roleās expertise and perspective. While promising, updating only the latest prompt in a conversation ignores the conversational flow that chatbot interactions usually have. Ouyang et al.[ 22] fine-tuned LLMs with human feedback to better align them with user intent, which results in improved truthfulness and less toxic replies. Similarly, Wang[29] applied re- inforcement learning from human feedback to adapt interactions. While these studies explore similar directions to ours, they directly update models, which reduces flexibility and ties them to specific model instances. Moreover, a model update is static by definition and cannot account for new conversational in- tents and contexts. By contrast, our approach is dynamically intercepting and updating conversations while they are ongoing, allowing for larger flexibility with less time and compute. Another approach to improve user-LLM interaction is to intervene directly in conversations to enhance the interaction by including specific context in- formation. Nam et al.[ 19] enhance open-ended prompts with user-highlighted code, resulting in more task completions. They conclude that adding even more contextual information holds potential to improve interactions. Arteaga Garcia et al.[ 1] validates this conclusion by demonstrating that, with more con- text, such as environment variables, previously executed code, and previous chat messages, the LLM yields better results. Similarly, Li et al.[16] add IDE native contexts, hereby focusing on code completion instead of conversations. While these studies follow partially related ideas, they do not concentrate on the prompt flow, but include different context information. A distinguishing feature of our study is that it is not being limited to coding tasks, specific IDEs, and laboratory environments, but aims at analyzing effects on interven- ing chat interactions in a long-term, practical development scenario. That is, it is orthogonal to the aforementioned studies. On the Prospects of Dynamic LLM Conversations in Software Development5 Guidance of Chat Interactions in Different Fields.The need for supporting chat interactions is not limited to SE contexts, but has been recognized in multiple different fields. Researchers enhanced chatbots for more empathetic and eļ¬icient responses for embodied conversational agents in general conver- sations to better react and understand participants intents[ 20]. Another study on LLMs for procrastination management found that LLMs require nuanced contexts to properly react to userās needs which are usually not provided, thus recommends the inclusion of automatically generated dynamic contexts[3]. Overall, even if the approaches differ, research fields beyond SE recognized that conversations can benefit from additional support and guidance to im- prove the overall experience and effectiveness. 3 Methodology In this section, we present our experimental design by introducing and moti- vating the research questions, then operationalizing the independent and de- pendent variables, and finally detailing the qualitative analysis, participants, and materials. 3.1 Research Questions The overarching goal of this research is to understand the effect of guidance mechanisms on developers and the code they produce. Specifically, we are interested in how different levels of guidance affect developersā interactions with an LLM, their perception of the LLM, and the code they produce. To this end, we define three research questions: RQ 1. How does the level of guidance influence developer behavior when in- teracting with an LLM? RQ 2. How do developers perceive LLMs with guided conversations? RQ 3. How do different guidance and support mechanisms influence code pro- duced by developers? We address these questions through three dependent variables: user behav- ior, perception, and generated code usage, each capturing interaction, experi- ence, and code artifacts, respectively. Independent Variable.The level of guidance is the independent variable and has three alternatives:contextual guidance,proactive guidance, andno guid- ance(our control condition).Figure 2illustrates how these levels play out in a conversation with the same user prompt. We operationalizecontextual guidance(lower left part inFigure 2), such that we enhance user-conversations with the context of the project task given by us (i.e., developing a web application) C2 and the developerās intention for a conversation C3, which they provide at the start of every conversation by 6Annemarie Wittig et al. Fig. 2The system message (gray bubble), developer input (green bubble), bot response ( blue bubble), and how they are updated in each treatment, using the example input prompt āThe appās background can change to a darker color scheme.ā, representing a minimal feature description selecting from a list of pre-defined intentions or providing their ownC1. This way, we ensure suļ¬icient context for the LLM, so that it can generate more tailored responses, for example, to either provide code, a description, or a user story. We chose this treatment because prior research has shown that providing suļ¬icient context to an LLM is essential for good developerāLLM interactions: Challenges often arise when developers do not share the necessary context[ 7, 11,18], leading to frustration[11], out-of-context interactions[7], and incorrect or unsatisfying replies[ 18]. Furthermore, since we found in a previous study that participants have diļ¬iculty providing suļ¬icient context[ 18], we decided to automatically provide the context to the LLM without participant interaction. This allows us to focus on the effect of additional context, independent of par- ticipantsā ability to provide it. Related approaches have also been successfully explored by other researchers[ 1,16,19]. On the Prospects of Dynamic LLM Conversations in Software Development7 Next, we operationalizeproactive guidance(lower right part inFigure 2) by instructing the LLM to act as a helpful assistant and predict follow-up ques- tions that the user may state based on the generated answer P1 . Additionally, we give the user feedback on the quality of their prompts and hints for more effective interactions with the LLM P3 (based on a hidden LLM). Inspired by automatic prompt refinement techniques[17], we use the reply to suggest a better suited prompt to the user in an informative block P5 , alongside the generated hints and adviceP6. This operationalization is also inspired by our previous study, in which participants also faced diļ¬iculties in formulating prompts[18]. Furthermore, our experience shows that participants find it diļ¬i- cult to navigate programming projects and determine which direction to take next. Thus, we included this treatment to provides support in these areas. For the control condition,no guidance, we use a LLM without any inter- vention (upper part in Figure2). We refer to this model as the control LLM to ensure clear separation from the treatment conditions. Both treatments are minimally invasive compared to the control LLM, for two reasons: First, we focus on internal validity to understand possible effects in detail. Thus, if the treatments differ only in few aspects, we can more reliably attribute cause and effect relationship between treatments and outcome. Second, we reduce ethical issues to a minimum. If we design a treatment that is highly invasive and worsens the performance of students, the respective students would be disadvantaged disproportionally. We provide a short summary of each of the levels of guidance in Table 1. Table 1Overview of our levels of guidance and their operationalization Level of GuidanceOperationalizationRationale No GuidanceBase LLM, no intervention Control Group Contextual Guidance Enhanced user conversations with task and intention con- text Mitigation of challenges caused by LLMs lacking context Proactive Guidance Additional follow-up ques- tions, feedback on prompt quality and further hints, al- ternative prompts Mitigation of issues with prompt formulation and find- ing directions in program- ming projects Dependent Variables.To answer our RQs, we analyze 3 dependent variables, one per RQ:user behavior,perception, andgenerated code usage. We provide a summary of each in Table 2. The outcomeuser behaviordescribes the interaction of developers with the LLM and represents the most direct observable impact of the treatments. We operationalized it three-ways. First, we checked theconversation length(num- ber of prompts per conversation, their evolvement over time, and number of prompt tokens, i.e. tokens from user-created messages, per conversation) to measure how extensively participants interacted with the LLM and whether 8Annemarie Wittig et al. this changed over time or under treatment. Second, we evaluated thespecified intentionsfor starting a conversation, provided by the participants, to exam- ine whether treatments steered developers toward switching their intention (e.g. explanations versus code generation). Third, we analyzed theintention alignmentof participantsā provided intention with conversation content. This allows us to explore underlying reasons of potential treatment impacts. To as- sess whether participantsā specified intentions align with the prompt contents within conversations, we selected 45 conversations per treatment to inspect manually. The resulting sampling size of 135 conversations was chosen to en- sure suļ¬icient coverage across treatments while keeping manual inspection feasible. We used stratified random sampling, such that we order the conver- sations by time, assign them to buckets of 100 conversations, and randomly selected five conversations per treatment from each bucket. Then, two of the authors assigned each of these prompts the intention that best fit the con- tent of the prompt. We compared whether our assignments align with the user-specified intention and logged each mismatch. Perceptionrefers toperceived helpfulnessof the LLM, its perceived impact on developersāfrustration, and theirproductivity. We assessed all three aspects with monthly questionnaires. This provides insights into how participants per- ceive guided interactions, and determines whether these guidance mechanisms could serve as a foundation for future research on improving developer-LLM interactions. Last, we observe thegenerated code usagefor the projects. Specifically, we analyze theamount of generated codepushed into a repository, themodifica- tion rateof committed code, and thedurationfor how long this code typically remains in the project. We collected this data to analyze how much the LLM supports development efforts, how long-lasting this support is, and how our treatment changes these effects. To this end, we randomly selected 50 con- versations from each treatment for manual review. We restricted the selected conversations to intentions that tend to naturally produce code:code genera- tion,language question,debugging, andtest generation. A conversation must also contain more than two prompts, and at least one response with generated code that not just repeats code of the prompt and does not solely contain shell commands or settings. For each conversation, we reviewed the commit history of the corresponding repository from the day the code was generated to the final release tag. When the generated code was indeed committed, we analyzed the commit history and counted line modifications to evaluate how the code evolved. 3.2 Study Conduct We conducted the study in the context of a mandatory 3rd semester software engineering course offered at two universities. This course included practical work to develop a full-stack web development project that lasted four months. The project was either preceded by a lecture on software engineering (one On the Prospects of Dynamic LLM Conversations in Software Development9 Table 2Dependent Variables, the RQ for which they are collected, and their operational- ization. VariableOperationalization Rationale RQ 1 : User Behavior Conversation length Changes in interaction intensity under treatment and time Specified intentionsTreatment influence on the type of in- teraction developers initiate Intention alignmentExploration of reasons for treatment impacts RQ 2 : Developer Perception Helpfulness, frustra- tion, productivity Participantsā perception of guided in- teractions RQ 3 : Generated Code Usage Amount pushedAdoption rate of AI-generated code in development Modification rateDegree of adaptation required for AI- generated code DurationPersistence of AI-generated code in the codebase university) or accompanied by one (second university), covering the software engineering lifecycle, development practices, and prompt engineering basics. With the start of the project, participants received access to the chatbot that was connected to GPT-4[21]. Using the most recent model at the time ensured participantsā interest in the LLM while reducing the risk of abandon- ment due to perceived tool limitations. They first gave their informed con- sent, confirming that they want to participate in the study, and agreed to the pseudonymized use of their data. They could revoke these agreements and stop participating in the study at any time without any repercussions on their course fulfillment. Once consent was given, participants could use the chatbot throughout the four-month project while completing their regular course ac- tivities. Throughout this time, we collected participant information through questionnaires, including short monthly questionnaires to collect longitudinal data on their opinions and satisfaction. Participants could discontinue the study at any time without any negative impact. In that case, they would have received access the chatbot using an account without tracking and without questionnaire prompts. No participant withdrew consent or requested removal of their data. There was no ethics proposal, which is not required at either university; we followed standard research guidelines on these kinds of studies. There was no restriction in using the chatbot. At the beginning of each conversation, we asked our participants to specify the intention they have and whether they use the LLM for the project or other use cases. The study pro- cess for participants is visualized in Figure 3. We analyze only project-related conversations. We restrict the chatbot history to read-only. This ensures the validity of our experiment by maintaining the relevance of intentions to the current conversation. 10Annemarie Wittig et al. Conversa)ons Conversations Informed Consent Programming Experience Questionnaire Chatbot Experience Questionnaire Request Reply Conversations Request Reply Request Reply and Prompt Suggestions Enhance System Message with Context Advice and enhanced Prompt Control Context Proac)ve Beginning of the Project Once a month Monthly Mood Questionnaire Conversation Intention Fig. 3Process of the study after participants received access depending on the treatment 3.3 Material Intentions.When initializing a conversation with the LLM, participants were asked to specify their intention for this conversation. The intention choice cannot be changed within a conversation, as it is intended that participants start new conversations when their intentions change. This ensures that con- versations are clearly assigned to intentions, prioritizing internal validity. This process is the same for all treatments. For starting a conversation, selecting one of the following intentions was necessary:to generate code(referred to ascode generation,code gen),to de- bug code(debugging,debug),to test code(test generation,test gen),to have code explained(code comprehension,code comp),to identify requirements(re- quirements elicitation,req elic),to ask questions about lecture content(lecture question,lect ques), and lastly,to ask questions about the programming lan- guage/framework(language question,lang ques). The specification of a custom intention was possible if none of these were suitable. The intention pool has been derived from our previous study by Mailach et al.[ 18]. Questionnaires.During the course of the study, we applied several question- naires to answer our research questions. At the beginning, we asked partici- pants about their programming knowledge, following the guidelines by Sieg- mund et al.[26]. Details are in the replication package. Additionally, we assessed information about experience with prompt engi- neering (PE), including whether they have attended a dedicated PE lecture of the course. Questions included how often participants have used LLMs, whether previous experience was positive or negative, and whether they would like to use them in the future. In a free text field, participants could also share any thoughts that they might feel relevant for the study. On the Prospects of Dynamic LLM Conversations in Software Development11 The monthly surveys assessed participantsā perception of the LLM to an- swer our second research question. Completing these questionnaires was manda- tory to continue interacting with the LLM. These surveys asked, on a 5-point- Likert scale, whether they find the chatbot helpful, frustrating, and beneficial for productivity. The monthly questionnaire also included an optional open- text field for any thoughts participants liked to share with us. We chose a one-month timeframe to collect regular insights and track changes throughout the experiment, while accommodating the typical bursts and pauses of activity in university projects. This duration also minimized participant inconvenience, reducing the risk of participants stopping using the chatbot due to excessive interruptions. 3.4 Participants To work on the programming project, students were randomly divided into teams of 4 to 8 students. The teams were then randomly assigned to the levels of the independent variable, ensuring similar experiences, even when multiple participants interacted with one LLM, for example, in the context of pair programming. In total, 93 students started the experiment. For answering RQ 1 and RQ 3 , we cleaned the data to remove outliers in three steps: 1.Removing all conversations / prompts with a non-project intention (re- moved 7 participants from the dataset). 2.Removing participants who sent fewer than 10 prompts in total (19 par- ticipants) to exclude users who did not suļ¬iciently use the LLM. 3.Removing superusers (3 participants). These users are considered statis- tical outliers according to thetwo-standard-deviation criterion, as they considerably exceed the mean number of 70 prompts per person (59 after removal); they sent between 235 and 415 prompts. This left us with the data of 64 participants from 25 project teams. Per treatment, we had 9 project teams in the control group, 8 in the context group, and 7 in the proactive group. We give an overview of our individual participant demographics in Table 3. The programming experience level (Median Prog. ExperienceinTable 3) is comparable for all groups (confirmed by a Kruskal-Wallis-Test W= 2.84,p= 0.24), so different levels of programming experience cannot explain differences between groups in the dependent variables (further confounding variables, such as preceding programming courses, are also similar among the groups, see projectās replication package). The same counts for the opinions on and usage of LLMs; although there are slight variations between the groups, there is no systematic difference, so that we can assume that the attitude toward LLMs is comparable between groups. For the second research question, which focuses on perception rather than behavior, we did not remove the outliers from the questionnaire responses, be- cause super-usersā opinions are still relevant here and do not skew any statis- tics. Instead, we removed responses only from participants with less than five 12Annemarie Wittig et al. Table 3Background of participants. Likert-scale responses are shown as bars, with the following values on theleftandrightrespectively: [D-A]Strongly disagreetostrongly agree [N-D]NevertoDaily [I-E]Very inexperiencedtoVery experienced [W-B]Much worsetoMuch better ControlContextProactive (N=25)(N=18)(N=21) Background and Experience Median Prog. Experience (yrs) 3.0 (n=23) 2.0 (n=15) 3.0 (n=18) Attended āPEā Lectureyes (n=25) yes (n=18) yes (n=21) Perceived Prog. Experience compared to Peers [W-B] Logical Prog. Experience [I-E] Opinions on Chatbots Use in daily life [N-D] Chatbots are helpful... ...in software dev. [D-A] ...in programming [D-A] ...in daily life [D-A] ...in studies [D-A] Chatbots are fun [D-A] Confident using chatbots [D-A] More future chatbot usage [D-A] prompts in the month previous to the questionnaire. For example, to be in- cluded in the data from the 2024-01-14 (second measurement date), users must have sent a minimum of 5 requests between 2023-12-06 (first measure- ment date) and 2024-01-14. Additionally, if participants did not use the LLM in-between two measurement dates at all, they could not answer the question- naire and thus are not included either. This way, we ensure that participants have used the LLMs suļ¬iciently to be able to give an informed response, while also preventing the loss of responses from participants who worked on the projects infrequently or became too frustrated to continue participating. 4 Results and Discussion We present the results of each research question in this section, separated into data presentation and the discussion of the interpretation and implications of the results. On the Prospects of Dynamic LLM Conversations in Software Development13 4.1 RQ 1 : How does guiding a conversation influence developer behavior when interacting with an LLM? 4.1.1 User Behavior Conversation Length.After removing outliers (conversations where the amount of prompts deviated more than two standard deviations of the mean), partici- pants wrote on average 3.1 prompts per conversation in thecontrol, 3.6 in the context, and 3.3 in theproactivegroup. For the number of tokens per prompt per conversation, also with outliers removed, there are, on average, 4638 for thecontrol, 7191 for thecontext, and 5995 for theproactivegroup. We present an overview inFigure 4. (a)(b) Fig. 4Average conversation length by treatment group per conversation by number of prompts per conversation (a) and number of prompt tokens per conversation (b) A Kruskal-Wallis test shows no significant difference in the number of prompts (W= 3.39,p= 0.182, Cliffās delta ranging from -0.07 to 0.06), while it does for the prompt tokens (W= 17.18,p<0.05, Cliffās delta ranging from -0.21 to 0.04). A Post-hoc Dunnās Test with Bonferroni correction confirmed the differences to be significant between thecontrolandcontext(p<0.05, CliffāsĪ“=ā0.21), and thecontrolandproactivegroup (p<0.05, Cliffās Ī“=ā0.17). Notably, these differences are not merely significant based on the large number of tokens, but also the effect sizes are larger than for the number of prompts. Figure 5shows that the number of prompts per conversation drops with the project duration. This holds for all groups and is most severe forcon- texttreatment. Interestingly,proactivetreatment seems already right at the beginning requiring fewer prompts. Specified Intentions.Figure 6gives an overview of the distribution of con- versation intentions. Across all groups, the most frequent intentions arecode 14Annemarie Wittig et al. Fig. 5Length of conversations over time, averaged per day and divided by treatment generation(370),debugging(235), andlanguage questions(148). Notably, par- ticipants in thecontrolgroup started far more conversations with the intention language question(Lang ques) than any other group (66.9% of such conver- sations, compared to 3% (context), and 29% (proactive)). It is their most frequent intention (99 of 323 conversations, 30% of all their conversations). In thecontextandproactivegroup, the most frequent intention wascode gen- eration(Code gen) with 117 of 226 (54%) and 156 of 352 (44%) conversa- tions, respectively. These differences in frequency are statistically significant (Ļ 2 = 111.40,df= 14,p<0.001). Furthermore, theproactivegroup used code generation intentions 60% and 33% more often than thecontrolandcontext groups, respectively. Interaction of conversation length and specified intentions.We also evaluated whether conversation length in terms of number of prompts is specific to the conversationsā intention. Most of the conversations are kept short, but in- tentionslanguage questions,debugging, andcode generationrelate to longer conversations. Figure7shows how many conversations (numbers in circles) with a specific conversation length (x-axis) were found for each intention (y- axis). The median length of conversations varies between 2 (custom intentions, marked asother) and 4 (test generation). A two-way ANOVA indicates a potentially significant impact of the inten- tion on the conversation length (F= 2.02, df= 7,p= 0.049), but shows no significant impact of neither the treatment (F= 2.3, df= 2,p>0.05) nor the interaction of treatment and intention (F= 0.8, df= 14,p>0.05). In post-hoc tests, the significant effect of intention vanishes. On the Prospects of Dynamic LLM Conversations in Software Development15 020406080100 Distribution of conversations per treatment Code comp Code gen Debug Lang ques Lect ques Req elic Test gen Other Intention 24 97 76 99 3 4 8 12 21 117 60 5 2 12 4 5 22 156 99 44 1 3 7 20 67 370 235 148 6 19 19 37 Total ControlContextProactive Fig. 6Conversation intentions per treatment. The number of conversations with an inten- tion is shown in the bars; the total number of conversations is shown on the right Fig. 7Distribution of conversation lengths per intention Intention alignment with conversation content.The manual analysis of 135 conversations showed whether and how participantsā prompt contents diverge from their initially specified intentions. Table 4provides details on misalignments between intention and actual conversation. Over all groups,38.52% of the analyzed conversations con- tain only prompts that entirely match the user-specified intention. Across all groups, about 25% of conversations have been started with a prompt that does not match the user-specified intention (the upper part of the table, column(% 1st )), which is similar for all treatments. If any deviation occurs, it 16Annemarie Wittig et al. Table 4Misalignments between user-specified intentions and prompt contents, split by treatment group or intention. Columns describe conversations with at least one misaligned prompt (% min1 ), more than half misaligned prompts (% half ), and the mean and median first positions in a conversation where a misalignment occurs (avg pos , med pos ). Prompt-level metrics include the percentage of first prompts of conversations that were misaligned (% 1st ) and the percentage of prompts where an intention that was assigned by us did not align with the userās given intention (% F N ). Prompt-level% 1st % F N Conversation-level N% min1 % half avg pos med pos By treat- ment Control45 26.67 62.22 35.56 2.462.0 Context45 28.89 68.89 37.78 2.192.0 Proactive 45 22.22 53.33 26.67 2.002.0 By given intention Code gen 52 36.54 75.00 46.15 2.032.0 11.43 Code comp 7 14.29 85.71 42.86 2.002.0 5.71 Debug37 10.81 48.65 10.81 2.723.0 22.86 Lang ques 28 14.29 46.43 25.00 2.922.0 45.71 Test gen3 66.67 66.67 66.67 1.001.0 2.86 usually occurs with the second prompt (columns med pos and avg pos ) for all treatments. Furthermore, 35.56% of all conversations in thecontrolgroup, 37.78% in thecontextgroup, and 26.67% in theproactivegroup had more than half of their prompts misaligned with the user-specified intention. Thus, a considerable number of conversations do not align with the user-specified intention and shift quickly, mostly after the first answer. Interestingly, the proactivegroup has the highest agreement in intention and actual conversa- tion. Breaking down the conversations per group, 37.78% of conversations in the controlgroup, 31.11% in thecontextgroup, and 46.67% conversations in the proactivegroup solely contain aligned prompts. AĻ 2 -test of independence on the number of aligned and misaligned prompts shows no significant difference between the alignment of prompts and the treatments (Ļ 2 = 5.31,df= 2,p= 0.069). While these misalignments may have influenced previously analyzed vari- ables for this RQ, the majority of conversations and prompts remain aligned; thus, we consider the data suitable for analysis of previous RQs. The inability to express oneās intentions and the tendency to shift intentions quickly during a conversation is interesting in its own right. This insight has not been empir- ically observed in related studies, yet it will affect future chat mechanisms. Viewing the data by user-specified intention (lower part of Table 4), par- ticipants indicating that they intend to debug deviate the least. Additionally, if participants deviate fromdebugging, they do so later in the conversations compared to any other intention (on median after the 3rd prompt; column āMedian pos ā). A potential reason for this observation is that participants might need more prompts to find a bug using the LLM than they need to fulfill other tasks. The highest number of deviations occurs when participants specify to generate code and tests (85.71 and 66.67%, column % min1 ). In cases such mis- On the Prospects of Dynamic LLM Conversations in Software Development17 alignments occur, the most often used intention islanguage questions(45%; column % F N ), which might mean that, once code is returned by the LLM, it needs to be understood, so the intention shifts tolanguage questionsorcode comprehension. RQ 1 : Results Conversational behavior of participants shows no major differences among treatments regarding the number of prompts, but regarding the number of prompt tokens per conversation. Conversations also get shorter over time whereasproactivealready starts with short conver- sations. There is a significant imbalance of our treatment groups between the number of conversations for specific intentions, despite the overall num- ber of conversations being similar. This difference is most prominent for participants asking about language-specific details. In thecontrol group, participants are starting conversations with this intention al- most 20 times more often than participants in thecontextgroup. More- over, programming related intentions, such as code generation and de- bugging intentions, are used substantially more often for theproactive group than for the two others. About a quarter of conversations start with a mismatch of the stated intention and the actual content of the prompt. This number rises after the first answer and is most pronounced forcode generationandcom- prehension. Theproactivegroup has the highest alignment of actual and user-stated intentions. 4.1.2 Discussion: Effects of guidance levels on developer-LLM-interactions The results regarding specified intentions align with other studies, indicating that developers use AI assistants for code generation and debugging[24,18,31, 5]. While not entirely surprisingly, it is interesting that the participants of our study only rarely asked the LLM for test generation, while Sergeyuk et al.[24] found that this was an often used intention. This discrepancy might stem from the different contexts of our programming-beginner setting, compared to the professional setting of their study, as automated tests were not mandatory for the programming project (yet the final product must still run and fulfill the requirements). Looking at the specified intentions by treatment, thecontrolgroup specifies most often language questions. This intention describes participants wanting to start a conversation to learn more about a programming language. At first sight, this might indicate that the control setting motivates participants toward using the LLM to dive more into understanding the programming language, while theproactiveandcontextnudges participants toward using the LLM to provide a working software product, less toward learning something about the code. However, in conjunction with the result that the actual content of the 18Annemarie Wittig et al. conversation in many cases does not align with the specified intention draws a different picture. Especially conversations with the specified intention ofcode generationoften contain prompts about language-specific details (19.5% of prompts within code generation conversations). Thus, even though the LLM interface explicitly guides users to think about what they want from the conversation, they have trouble doing so. We interpret this to mean that early-career developers, like those in our sample, might struggle with deciding up-front what exactly they want from the LLM. If they are not sure about their intention, it is likely that their prompt reflects that in vagueness, such that the LLM is used ineffectively. This insight shows the need to clarify intentions of users upfront, which may also yield a more effective prompt design. We are not aware of any user study that identified the extend of this mismatch despite its importance for the soundness of empirical evalutions and design of new chat mechanisms. Regarding eļ¬iciency, we see a longitudinal trend towards shorter discus- sions, indicating that developers get more eļ¬icient in their LLM communica- tion over time as they require fewer prompts in a single discussion. A main observation is that with theproactivetreatment, conversations were already short at the first time period, which stands out and may point to an increased eļ¬iciency. In our specific setting, intentions within a conversation could not be changed without starting a new conversation; a trade-off of our study design. This can be unproductive, as a new conversation does not have the relevant context. We found clear evidence that a static intention context is insuļ¬icient, as the nat- ural flow of conversations shifts the focus. Theproactivetreatment accounts for this dynamic nature and shows the lowest misalignments in intentions and actual conversation content. A possible explanation is also that this treatment ensures that the user stays on track with its initial intent and does not slide into other intention activities. So either requiring users to specify their inten- tions with every prompt or leveraging systems, that predict intentions without additional user input, could improve effectiveness of LLMs. On the Prospects of Dynamic LLM Conversations in Software Development19 The answer to RQ 1 is nuanced. At first glance, there is no signifi- cant difference in developersā general behavior when interacting with the LLM, regardless of the treatment. Even for thecontextgroup, we observed no substantial behavior changes, despite identifying frequent misalignments of intentions throughout conversations. A closer look reveals that theproactivegroup seems to better align their prompts to their initial conversation intention. However this difference is not sig- nificant. Additionally, participants in this group used the LLM most often for development specific intentions, such as debugging and code generation, and used fewer prompts to finish a conversation, point- ing to higher eļ¬iciency. Nevertheless, this could show a tendency of the ability of dynamic approaches to support participants in keeping their conversations aligned with their initial goal when using the LLM specifically for software development. 4.2 RQ 2 : How do developers perceive LLMs with guided conversations? 4.2.1 Developer Perception To evaluate RQ 2 , we show the monthly survey results of participants inFig- ure 8. As is common in longitudinal studies (especially in voluntary university experiments), the number of participants reduces with each month. We started with 47 replies for the first survey and ended up with only 13 replies for the last. Starting with the second survey, the group sizes were small, making sig- nificance tests underpowered and the results likely inconclusive. Therefore, we compared the results of surveys 2ā4 without testing for significance. This ap- proach helps identify tendencies between treatments that can inform future research directions. The first survey was conducted during the period with the highest de- velopment effort for the projects, so more developers answered it. Therefore, we performed pairwise Mann-Whitney U-tests for this survey to compare in- dividual groups. Thus, we compare the values for tendencies, but note that differences are minor. More detailed information on this can be found on our Web site. Helpfulness.The total average value on perceived helpfulness (TotalinFig- ure 8) is 3.9 for thecontrol, 3.5 for thecontext, and 4.3 in theproactivegroup (5 meaning very helpful), so all groups found the LLM to be at least somewhat helpful. A pair-wise Mann-Whitney U-test revealed significant differences be- tween theproactiveand both, thecontroland thecontextgroup (U= 452.0, p= 0.008and U= 269.0,p= 0.003respectively). Overall, there is a downward trend of perceived helpfulness for all groups from the first to the third survey, and an increase for the last survey. Theproactivegroup reported the highest helpfulness rating ( Figure 8A) in all surveys, while thecontrolgroup reported 20Annemarie Wittig et al. 2023 12-06 (N=47) 2024 01-14 (N=25) 2024 02-14 (N=16) 2024 03-15 (N=13) Total N=(101) 1 2 3 4 5 Average Answer (1-5) n=19 n=12 n=7 n=2 n=40 n=12 n=6 n=4 n=5 n=27 n=16 n=7 n=5 n=6 n=34 (A) The chatbot was helpful in the last few weeks. 2023 12-06 (N=47) 2024 01-14 (N=25) 2024 02-14 (N=16) 2024 03-15 (N=13) Total N=(101) 1 2 3 4 5 n=19 n=12 n=7 n=2 n=40 n=12 n=6 n=4 n=5 n=27 n=16 n=7 n=5 n=6 n=34 (B) Using the chatbot frustrated me in the last few weeks. 2023 12-06 (N=47) 2024 01-14 (N=25) 2024 02-14 (N=16) 2024 03-15 (N=13) Total N=(101) 1 2 3 4 5 n=19 n=12 n=7 n=2 n=40 n=12 n=6 n=4 n=5 n=27 n=16 n=7 n=5 n=6 n=34 (C) Using the chatbot made me more productive in the last few weeks. ControlContextProactive Fig. 8Averaged answers for perceived helpfulness, frustration, and productivity. The an- swers range from 1 to 5, with 1 being āstrongly disagreeā and 5 āstrongly agreeā. Higher values indicate a positive rating in A and C, but negative in B. the lowest in the first survey (U =88.5,p= 0.025). Cliffās delta effect sizes for the first survey range fromā0.41(when comparing thecontrolandproac- tivegroup) to0.06(contexttocontrol), confirming that theproactivegroup viewed the LLM more helpful at this point. Thecontextgroup has the lowest helpfulness rating in the second and third surveys, dropping notably from the first. Frustration.The average participant frustration scores are 2.9 forcontrol, 2.7 forcontext, and 2.4 forproactive(1 meaning not frustrated), indicating low frustration. There does not seem to be an overall picture of how frustration evolves. Participants in thecontrolgroup showed the highest frustration ( Fig- ure 8B) in the first two surveys, then align with theproactivegroup on a medium-low level. After a considerable increase in frustration from the second to third survey, thecontextgroup is the most frustrated in the third survey. Theproactivegroup consistently reported low frustration levels. None of the differences in the first survey is statistically significant, and the Cliffās delta ranges fromā0.14to0.11. ProductivityOn average, the total perceived productivity of participants is 3.8 for thecontrolandproactive, and 3.4 for thecontextgroup (5 meaning very productive). Participantsā perceived productivity (Figure 8C) increased in each survey for thecontrolgroup, showing the highest ratings of all groups in the second and third surveys. Again, with only two people from thecontrol group giving responses, we cannot reliably compare their results for the last survey. Thecontextandproactivegroupsā perceived productivity drops from the first to the second survey. However, the drop is more pronounced for the context group, resulting in the lowest perceived productivity in the second and third surveys. Again, none of the differences in the first survey are sta- tistically significant, with Cliffās delta effect sizes ranging from mediumā0.30 (comparing thecontextandproactivegroup) to null0.0(contextandcontrol). On the Prospects of Dynamic LLM Conversations in Software Development21 RQ 2 : Results Theproactivegroup perceived the LLM as the most helpful and as more beneficial to their productivity while rarely feeling frustrated, while maintaining the highest user engagement throughout the experiment. Thecontextgroup viewed the LLM the most critical, while both, the contextandcontrolgroups felt frustrated by the LLM at some point during the semester. 4.2.2 Discussion: Developer perceptions of guided conversations with LLMs The perception of participants differs marginally between groups, indicating that all variants of LLMs are perceived as helpful, supporting productivity, and only rarely lead to frustration. Especially theproactivegroup shows the most positive perception, and one participant highlighted this in the open text fieldā[...] I can only say that programming itself can be frustrating sometimes and I think without the chatbot[...] it would be even more exhausting and frustratingā P1 . The consistently lower helpfulness and productivity ratings of thecontext group might be linked with the misalignment of the user-specified with the actual intentions of a conversation, which is most prominent for thecontext group (cf. Section 4.1.1). The intention is not used for thecontrolgroup, and has limited effect for the dynamic conversation in theproactivegroup. So, they do not receive misaligned responses, resulting in a more positive perception. In general, these results align with previous studies on perceived produc- tivity[ 15], and contradict others that found significant productivity improve- ments[31] or that observed increased frustration and no productivity gains[6]. This further highlights the mixed effects of LLMs. Thus, these findings add another piece to the puzzle to better understand how different prompting techniques and guidance mechanisms affect software developers. For theproactivegroup, we further manually analyzed how users adopted the features of follow-up questions and improved prompts. About two-thirds of participants used follow-up questions or improved prompts, indicating that the functionality holds some appeal for the majority of users. Looking more closely, especially for the user-specified intentions ofcode generationandlanguage questions, participants used either follow-up questions or improved prompts to continue the conversation. Over time, this decreased, which could indicate that participants might have learned how to improve their conversations, thus not needing any more assistance. Furthermore, since there does not seem to be negative perception of participants regarding guidance, this might be a promising endeavor to integrate relevant tips and techniques in custom-based chatbots, at least for early-career developers. 22Annemarie Wittig et al. To answer RQ 2 , we find that LLMs are perceived differently depend- ing on the treatment. Especially developers using theproactiveLLM perceived the LLM the most positive, suggesting that active guidance features positively impact developers perceptions of LLMs. This is note- worthy, considering that we used a conservative, low-impact approach. By contrast, developers using thecontextLLM rated it the least helpful, which is likely due to the misalignment of given prompts with conver- sational contents. These findings suggest that, while guiding LLMs can enhance user experience, not all treatments are equally well-received. Given the better results in theproactivegroup, enhancing developer- LLM-interactions with proactive guidance elements is a promising di- rection for future work. 4.3 RQ 3 : How do different guidance and support mechanisms influence code produced by developers? 4.3.1 Results To answer RQ 3 , we evaluate the amount of generated code that is pushed into the repository, the modification rate, and the duration of how long generated code remains in the project. Amount of generated code pushed to repositories.Over all treatments, partic- ipants made a median of 59 commits. Per treatment, the median commits by person were 72 (control), 42 (context) and 44 (proactive) commits. ControlContextProactive Treatment 0 10 20 30 Conversation count 23 34 26 27 16 24 Resulting code committed No resulting code committed Fig. 9Pushed vs. not pushed gen- erated code per treatment; solid and dashed lines mark their respec- tive averages. To verify whether participants used code generated by the LLM, we sampled 150 conver- sations for manual review (cf. Section 3.1).Fig- ure 9shows that, for 46% (23) conversations in thecontrolgroup, 68% (34) in thecontext group, and 52% (26) in theproactivegroup, generated code was committed to the reposi- tory. Thus, most generated code was commit- ted in thecontextgroup. This difference is sig- nificant between thecontrolgroup andcontext group (Ļ 2 = 4.07, df= 1,p= 0.04, Cliffās delta=ā0.22), but it disappears afterBonfer- roni correction. Modification rate.Overall, participants change 74.01% of the generated code over time.Table 5gives an overview of the code changes by treatment (% total ), showing that thecontextandproactivegroups have a similar modification rate, and thecontrolgroup shows the lowest number of modifications. The difference of the total modification rates is significant (Ļ 2 = 20.95,p<0.01). On the Prospects of Dynamic LLM Conversations in Software Development23 We normalized the changes by computing the average percentage of mod- ified lines per committed code segment (% segment ) to better account for code- segment specific modifications (e.g., ignoring formatting related changes). A code segment comprises coherent code that was generated by the LLM, and pushed to the same file. The lowest modification rate still occurs in thecon- trolgroup (60.25%), but the rate of modification of thecontextgroup is now more similar to thecontrolgroup, and the change for theproactivegroup is negligible. After the normalization, the difference is not significant anymore, as shown by a Kruskal-Wallis test (W= 1.46,p= 0.48). Table 5Changes made to the LLM-generated code after it was committed to the project, including the total percentage of changes to generated code (% total ), the average percentage of changes per committed code segment (% segment ), and the average number of days until changes occurred (Time days ). Treatment % total % segment Time days Control65.26 60.2518.72 Context76.47 65.548.70 Proactive79.95 77.5920.16 Duration until code is changed.Over all treatments, code remains unchanged for, on average, 15.37 days. Members of theproactivetreatment leave their code unchanged for the longest time period of 20 days, on average. Thecon- troltreatment follows with 18 days, while thecontexttreatment had the short- est time until changes occurred, averaging 8 days (right column, Time days in Table 5). This difference is not significant (W= 5.0,p= 0.08). RQ 3 : Results Participants used more LLM-generated code in their projects if they used one of the guiding treatments. The discrepancy is large with the contextgroup using generated code from 2 out of 3 conversations, com- pared to less than half of the conversations in thecontrolgroup. Around 70% of LLM-generated code per file was eventually changed in all groups, with teams in theproactivegroup changing code the most and thecontrolgroup changing the least. The alteration occurred the fastest in thecontextgroup. 4.3.2 Discussion: Effects of guidance levels on produced code Thecontextgroup had more conversations resulting in generated code be- ing pushed. This is in line with the results of SpasiÄ and JankoviÄ[27], who observed that incorporating roles, specific instructions, and seed words into prompts improved model outputs to a more human-like level. Since thecontext 24Annemarie Wittig et al. treatment works similarly, this might explain that thecontextgroups commit- ted more generated code to the repository. At first, it seems counterintuitive that thecontextgroup has the highest number of generated code commits in the repositories, despite having lower helpfulness and productivity ratings. Simultaneously, the LLM code is modified the fastest, that is, after 10 days. A likely explanation is that, since the participants modify code more quickly, they are likely not satisfied with the generated code, but since they pushed it to the repository and now must modify it, they feel less productive and less supported by the LLM. Interestingly, this insight contrasts another study re- garding perceived productivity: Ziegler et al.[32] found that higher perceived productivity correlates with higher acceptance rate of generated code. This might also be caused by the different AI assistance (chat-based vs. direct IDE integration with code generation suggestions) or by the university context. The concrete underlying causes for this mismatch in perceived productivity and code contribution is an interesting research direction for future studies. The overall high modification rates are in line with Harding and Kloster[9], who estimate the code churn rate (the percentage of code lines that are mod- ified within 2 weeks) to double in 2024 compared to a 2021 baseline from before the wide usage of AI-tools. One developer in thecontrolgroup men- tions:āSome of my fellow students used the chatbot to write the frontend [...]. Now, [it] is practically unmaintainable because it is just a pile of spaghetti codeā P2 . Within the same group, another developer had a similar experience, when their team members did not update LLM-generated code to fit their cur- rent projectās boundaries, resulting inā[...] others [having] to spend the time saved by [the LLM] to get the code into an acceptable state.ā P4 The observa- tion was summarized by another developer using the control LLM, pointing out how the āchatbot was very helpful for general basic understanding, but it reached its limits when it came to more concrete, project-specific things [...]ā P3 . This perception is supported by current work, which often focuses on including the codebase as context in LLM conversations[1,16,19], highlighting the gap between the LLMās knowledge of the codebase and the knowledge needed to create code that can be integrated nicely. We argue that, partially, differences of our setting with other studies come from our project-specific setting that does not focus on a single, well-encapsulated tasks, but rather a more realistic open-ended project scenario, which is essential to understand the full range of of an LLMās impact on coding activities. While we acknowledge that interpretations based on limited data and marginal significance should be made cautiously, the observed trends offer valuable insights for future research. Furthermore, the alignment of our find- ings regarding modification with participant perceptions and prior work sup- ports the viability of these findings. To answer RQ 3 , we find that treated LLMs cause more LLM-generated code to be pushed into repositories, with the highest amount in reposi- On the Prospects of Dynamic LLM Conversations in Software Development25 tories of developers using thecontextLLM. Despite this disparity, sim- ilar amounts of code were changed over time in all groups. This high- lights how even the more conservative interventions in conversations can increase the LLMās capabilities to generate code that is committed without causing negative trade-offs in that regard, further highlighting the potential of guidance mechanisms. 5 Threats to Validity Internal Validity.The drop-out rate of participants could be a threat to the studyās internal validity. This might suggest that participants who continued using the LLMs were more motivated or had a more positive attitude toward them, which particularly threatens conclusions for RQ 2 . Hence, we collected participants general attitude toward LLM usage at the beginning of the study and found no systematic pattern between participants completing the study and those who drop out. Moreover, since the main development activities were in the first time period, the decreasing numbers may indicate that fewer participants required the help of the LLM. Statistical Conclusion Validity.Most participants sent fewer than 200 prompts, except three, one of whom sent over 400. Including them can majorly impact statistical conclusions. We mitigated this by removingsuperusersduring pre- processing to ensure that they are not skewing aggregated statistics. External Validity.The study population consists of novice developers. As the participants were students rather than professional developers, this might limit the generalizability to the full developer population. However, the study was embedded in a longitudinal, realistic software project resembling conditions developers encounter in their first professional projects. As such, the study supports viewing the population as novice developers, as is done commonly in Software Engineering research[ 8]. Consequently, the results are interpreted with respect to novice developers rather than the educational setting itself. As novice developers, most of our participants developed their first proper software project during our study, while more experienced developers might have done a lot more projects. However, a significant portion of developers in practice are junior developers and developers in new positions. Thus, while our results may not be generalizable to the entire developer population, they are still relevant to an important subpopulation. The goal of this study is to investigate the benefits and harms of interventions in LLM conversations in order to pave the way for new interaction mechanisms. Ecological Validity.The university context threatens the applicability of our results for practical settings and may explain deviations of results from other 26Annemarie Wittig et al. studies. However, conducting a study as controlled as ours, which includes ran- domly assigning developers to teams and letting all teams develop the same project for four consecutive months, is infeasible in a realistic setting. This dedicated focus inherently comes with cost for ecological validity. To increase ecological validity, we designed a realistic project setting, with the projectās longevity aligning more closely with real-world development processes, espe- cially compared to studies that focus on small or isolated tasks. 6 Conclusion LLMs have become a well-used tool for developers and a popular research topic. In this study, we asked whether we can support and guide developers when interacting with LLMs and, more specifically, how such guidance impacts software development. To this end, we implemented two guidance approaches with minimal in- terference in the conversation. Thecontexttreatment supplied the LLM with additional context based on the userās intention, while theproactivetreatment offered active guidance through advice, improved prompts, and follow-up ques- tions. We tested these approaches in a large, longitudinal 4-month study with CS students building a full-stack Web application in their third semester. We found that conservative guidance mechanisms in LLMs did not majorly influence participants in their general usage behavior with the LLMs. Across the measured outcomes (interaction patterns, self-reported frustration/help- fulness/productivity, and downstream code adoption/modification), we did not observe evidence of negative effects attributable to the interventions. How- ever, we find significant variation in conversation intentions between treatment groups, though the reason for this remains unclear. In terms of helpfulness, frustration, and productivity, we find that developers receiving theproactive guidance perceived the LLM as the most positive. Thecontexttreatment was perceived critically, but still led to higher usage of LLM-generated code. What does it mean?We conclude that even such minimal interventions as ours already yield positive effects on developersā satisfaction and how they adopt and modify code. The interventions do not disrupt the workflow compared to the freedom of the control LLM. During the course of the project, participants grew more eļ¬icient, as conversations grew shorter over time. Based on this, it is promising to study how higher levels of guidance, such as regular updates of intentions or requesting more context during a conversation, may affect results and user satisfaction. Our initial setup with only sparse intervention of user-LLM interaction can serve as a starting point to better guide (early-career) software developers to eļ¬iciently use LLMs in their daily workflow, and, importantly, dispel ethical concerns regarding harmful effects of studies that investigate interventions in user-LLM interactions. On the Prospects of Dynamic LLM Conversations in Software Development27 Statements and Declarations Funding This work was supported by the Saxon State Ministry for Science, Culture and Tourism (SMWK) through the āCenter for Scalable Data Analytics and Artificial Intelligence Dresden/Leipzigā (ScaDS.AI), by the European Social Fund (ESF) together with the German state of Saxony under grant number A.100760692 (Project: Teaching-AI), and by the German Research Foundation DFG (SI 2171/3-2). Ethical approval This study involved voluntary participation of computer science students and did not require formal ethical approval according to the institutional regu- lations of the participating universities. All participants provided informed consent prior to participation. Responses were pseudonymized, and no per- sonally identifiable information was published. Participation was voluntary, and participants could withdraw at any time without consequence. Data was handled in accordance with applicable data protection regulations. Informed consent Participants were informed about the purpose of the study, the voluntary nature of their participation, what data would be collected, how it would be used and protected, and that they could withdraw at any time. Data was pseudonymized during analysis and anonymized prior to publication. Informed consent was obtained before any data was collected. Informed consent was obtained prior to data collection. Author Contributions All authors contributed to the study conception and design. Material prepa- ration and data collection were performed by Annemarie Wittig and Alina Mailach; Analysis was performed by Annemarie Wittig. The first draft of the manuscript was written by Annemarie Wittig and subsequently revised by Alina Mailach; All authors commented, updated and rephrased previous ver- sions of the manuscript. All authors read and approved the final manuscript. Data Availability Statement The data and material from this study is available in our replication package: https://anonymous.4open.science/r/On-the-Prospects-of-Dynamic-L LM-Conversations-in-Software-Development-D22C/README.md 28Annemarie Wittig et al. Conflict of Interest The authors have no competing interests to declare that are relevant to the content of this article. Clinical Trial Number Not applicable References 1.Arteaga Garcia, E.J., Nicolaci Pimentel, J.a.F., Feng, Z., Gerosa, M., Steinmacher, I., Sarma, A.: How to support ml end-user programmers through a conversational agent. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ā24. Association for Computing Machinery, New York, NY, USA (2024). DOI 10.1145/3597503.3608130. URLhttps://doi.org/10.1145/3597503.3608130 2.Ataei, M., Cheong, H., Grandi, D., Wang, Y., Morris, N., Tessier, A.: Elicitron: An llm agent-based simulation framework for design requirements elicitation (2024). URL https://arxiv.org/abs/2404.16045 3.Bhattacharjee, A., Zeng, Y., Xu, S.Y., Kulzhabayeva, D., Ma, M., Kornfield, R., Ahmed, S.I., Mariakakis, A., Czerwinski, M.P., Kuzminykh, A., Liut, M., Williams, J.J.: Under- standing the role of large language models in personalizing and scaffolding strategies to combat academic procrastination (2023). URLhttps://arxiv.org/abs/2312.13581 4.Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., Gazit, I.: Taking flight with copilot: Early insights and opportunities of ai-powered pair-programming tools. Queue20(6), 35ā57 (2023). DOI 10.1145/3582083. URL https://doi.org/10.1145/3582083 5.Chatterjee, S., Liu, C.L., Rowland, G., Hogarth, T.: The impact of ai tool on engineering at anz bank an empirical study on github copilot within coporate environment. ArXiv abs/2402.05636(2024). URLhttps://api.semanticscholar.org/CorpusID:267547 947 6.Choudhuri, R., Liu, D., Steinmacher, I., Gerosa, M., Sarma, A.: How far are we? the triumphs and trials of generative ai in learning software engineering (2023). URL https: //arxiv.org/abs/2312.11719 7.Davila, N., Wiese, I.S., Steinmacher, I., da Silva, L.L., Kawamoto, A.L.S., Favaro, G.J.P., Nunes, I.: An industry case study on adoption of ai-based programming as- sistants. 2024 IEEE/ACM 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) p. 92ā102 (2024). URL https: //api.semanticscholar.org/CorpusID:270174221 8.Falessi, D., Juristo, N., Wohlin, C., Turhan, B., MĆ¼nch, J., Jedlitschka, A., Oivo, M.: Empirical software engineering experts on the use of students and profession- als in experiments. Empirical Software Engineering23(1), 452ā489 (2018). DOI 10.1007/s10664-017-9523-3. URL https://doi.org/10.1007/s10664-017-9523-3 9.Harding, W., Kloster, M.: Coding on copilot: 2023 data shows downward pressure on code quality. Tech. rep., GitClear (2024). URLhttps://gitclear-public.s3.us-wes t-2.amazonaws.com/Coding-on-Copilot-2024-Developer-Research.pdf 10.Hassib, M., Buschek, D., Wozniak, P.W., Alt, F.: Heartchat: Heart rate augmented mobile chat to support empathy and awareness. In: Proceedings of the 2017 CHI Con- ference on Human Factors in Computing Systems, CHI ā17, p. 2239ā2251. Association for Computing Machinery, New York, NY, USA (2017). DOI 10.1145/3025453.3025758 11.Khojah, R., Mohamad, M., Leitner, P., de Oliveira Neto, F.G.: Beyond code generation: An observational study of chatgpt usage in software engineering practice (2024). URL https://arxiv.org/abs/2404.14901 On the Prospects of Dynamic LLM Conversations in Software Development29 12.Kim, S., Eun, J., Oh, C., Suh, B., Lee, J.: Bot in the bunch: Facilitating group chat discussion by improving eļ¬iciency and participation with a chatbot. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ā20, p. 1ā13. Association for Computing Machinery, New York, NY, USA (2020). DOI 10.1145/3313 831.3376785 13.Kong, A., Zhao, S., Chen, H., Li, Q., Qin, Y., Sun, R., Zhou, X., Wang, E., Dong, X.: Better zero-shot reasoning with role-play prompting. In: Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4099ā4113 (2024) 14.Kong, A., Zhao, S., Chen, H., Li, Q., Qin, Y., Sun, R., Zhou, X., Zhou, J., Sun, H.: Self-prompt tuning: Enable autonomous role-playing in llms (2024). URLhttps://ar xiv.org/abs/2407.08995 15.Kuhail, M.A., Mathew, S.S., Khalil, A., Berengueres, J., Shah, S.J.H.: āwill i be re- placed?ā assessing chatgptās effect on software development and programmer percep- tions of ai tools. Science of Computer Programming235, 103111 (2024). DOI https://doi.org/10.1016/j.scico.2024.103111. URLhttps://w.sciencedirect. com/science/article/pii/S0167642324000340 16.Li, Y., Peng, Y., Huo, Y., Lyu, M.R.: Enhancing llm-based coding tools through native integration of ide-derived static context (2024). URLhttps://arxiv.org/abs/2402.0 3630 17.Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B.P., Hermann, K., Welleck, S., Yazdanbakhsh, A., Clark, P.: Self-refine: Iterative refinement with self-feedback. In: NeurIPS (2023) 18.Mailach, A., Gorgosch, D., Siegmund, N., Siegmund, J.: āok pal, we have to code that nowā: Interaction patterns of programming beginners with a conversational chatbot. Empirical Software Engineering (EMSE)30(1), 34 (2025). URL https://sws.inform atik.uni-leipzig.de/wp-content/uploads/2024/10/OkPal-InteractionPatternsOf ProgrammingBeginners.pdf 19.Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., Myers, B.: Using an llm to help with code understanding. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ā24. Association for Computing Machinery, New York, NY, USA (2024). DOI 10.1145/3597503.3639187 20.Ćzge Nilay YalƧın: Empathy framework for embodied conversational agents. Cognitive Systems Research59, 123ā132 (2020). DOI https://doi.org/10.1016/j.cogsys.2019.09.0 16. URL https://w.sciencedirect.com/science/article/pii/S1389041719304826 21.OpenAI: Gpt-4 (2024). URLhttps://openai.com/research/gpt-4. Accessed: 2024- 10-17 22.Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P.F., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback. In: NeurIPS (2022) 23.Russo, D.: Navigating the complexity of generative ai adoption in software engineering (2024). URL https://arxiv.org/abs/2307.06081 24.Sergeyuk, A., Golubev, Y., Bryksin, T., Ahmed, I.: Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward (2024). URLhttps://arxiv. org/abs/2406.07765 25.Shin, J., Tang, C., Mohati, T., Nayebi, M., Wang, S., Hemmati, H.: Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks. ArXivabs/2310.10508(2023). URL https://api.semanticscho lar.org/CorpusID:264145854 26.Siegmund, J., KƤstner, C., Liebig, J., Apel, S., Hanenberg, S.: Measuring and modeling programming experience. Empirical Software Engineering19, 1299ā1334 (2014) 27.SpasiÄ, A.J., JankoviÄ, D.S.: Using chatgpt standard prompt engineering techniques in lesson preparation: Role, instructions and seed-word prompts. In: 2023 58th Interna- tional Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST), p. 47ā50 (2023). DOI 10.1109/ICEST58410.2023.10187269 30Annemarie Wittig et al. 28.Vaillant, T.S., de Almeida, F.D., Neto, P.A.M.S., Gao, C., Bosch, J., de Almeida, E.S.: Developersā perceptions on the impact of chatgpt in software development: A survey (2024). URLhttps://arxiv.org/abs/2405.12195 29.Wang, B.: A proactive system for supporting users in interactions with large language models. In: Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, CHIIR ā24, p. 441ā444. Association for Computing Machinery, New York, NY, USA (2024). DOI 10.1145/3627508.3638325 30.Wang, W., Ning, H., Zhang, G., Liu, L., Wang, Y.: Rocks coding, not developmentāa human-centric, experimental evaluation of llm-supported se tasks (2024). URLhttps: //arxiv.org/abs/2402.05650 31.Ziegler, A., Kalliamvakou, E., Li, X.A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., Aftandilian, E.: Measuring github copilotās impact on productivity. Commun. ACM 67(3), 54ā63 (2024). DOI 10.1145/3633453. URLhttps://doi.org/10.1145/3633453 32.Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., Aftandilian, E.: Productivity assessment of neural code completion (2022). URL https://arxiv.org/abs/2205.06537