Paper deep dive
Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming
Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, Amanpreet Kapoor
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/26/2026, 2:40:23 AM
Summary
This study investigates the impact of pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (Full Context vs. No Context) on student learning experiences with AI Teaching Assistants (AI TAs) in an introductory programming course. Through a randomized controlled trial with 132 students, the research found that the Socratic AI TA with full context was rated least favorably for task completion support, showing higher interaction stress and a higher rate of external LLM use, although some differences were not statistically significant. The findings suggest that guardrailed AI TAs are not automatically beneficial and that effectiveness depends on balancing pedagogical guidance and contextual awareness to ensure student support and engagement.
Entities (10)
Relation Signals (7)
University of Florida ā affiliatedwith ā Madeleine Eastwood
confidence 95% Ā· Madeleine Eastwood Affiliation: University of Florida
University of Auckland ā affiliatedwith ā Paul Denny
confidence 95% Ā· Paul Denny Affiliation: University of Auckland
AI Teaching Assistant ā backedby ā GPT-4o
confidence 95% Ā· AI TAs were backed by the OpenAI GPT-4o LLM
Socratic Guidance ā combinedwith ā Full Context
confidence 95% Ā· Students rated the Socratic AI TA with full context least favorably
Socratic Guidance ā produces ā Lower Perceived Support
confidence 90% Ā· Students rated the Socratic AI TA with full context least favorably, reporting significantly lower perceived support for task completion.
Socratic Guidance ā associatedwith ā Higher External LLM Use
confidence 85% Ā· the highest rate of external LLM use
Socratic Guidance ā associatedwith ā Higher Interaction Stress
confidence 85% Ā· this condition also showed the highest observed interaction stress
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI teaching assistants (AI TAs) backed by large language models (LLMs) and pedagogical guardrails are increasingly being integrated into programming courses, providing students with scalable access to hints, conceptual explanations, and code-level feedback. However, guardrails may also create friction. If students feel that the support provided is overly restrictive or poorly contextualized to their current progress, they may bypass approved tools for general-purpose LLMs. To investigate how AI TA design affects students' learning experiences, we conducted a randomized controlled trial with 132 students in an introductory programming course. Students completed three tasks related to code-writing and debugging and were randomly assigned to one of four AI TAs varied across two dimensions: pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (no context vs. full context of the problem and student solution). We examined students' perceptions, interaction behaviors, and evidence of post-task comprehension. Students rated the Socratic AI TA with full context least favorably, reporting significantly lower perceived support for task completion. Descriptively, this condition also showed the highest observed interaction stress, the highest rate of external LLM use, and the lowest proportion of post-task explanations demonstrating full comprehension, though these differences were not statistically significant. These findings suggest that guardrailed AI TAs are not automatically better for learning. Instead, their effectiveness depends on how pedagogical guidance and contextual awareness are balanced in ways that students experience as useful, supportive, and worth continuing to use.
Tags
Links
- Source: https://arxiv.org/abs/2609.29995v1
- Canonical: https://arxiv.org/abs/2609.29995v1
Trouble viewing inline? Open PDF directly ā
Full Text
46,062 characters extracted from source content.
Expand or collapse full text
Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming CCS: Social and professional topics Computing education Madeleine Eastwood Affiliation: University of Florida, Gainesville, FL, United States email: meastwood@ufl.edu , Harshith Narne Affiliation: University of Florida, Gainesville, FL, United States email: h.narne@ufl.edu , Joseph Hilby Affiliation: University of Florida, Gainesville, FL, United States email: joseph.hilby@gmail.com , Paul Denny Affiliation: University of Auckland, Auckland, New Zealand email: paul@cs.auckland.ac.nz , Ashish Aggarwal Affiliation: University of Florida, Gainesville, FL, United States email: ashishjuit@ufl.edu and Amanpreet Kapoor Affiliation: University of Florida, Gainesville, FL, United States email: kapooramanpreet@ufl.edu Ā© none Abstract. AI teaching assistants (AI TAs) backed by large language models (LLMs) and pedagogical guardrails are increasingly being integrated into programming courses, providing students with scalable access to hints, conceptual explanations, and code-level feedback. However, guardrails may also create friction. If students feel that the support provided is overly restrictive or poorly contextualized to their current progress, they may bypass approved tools for general-purpose LLMs. To investigate how AI TA design affects studentsā learning experiences, we conducted a randomized controlled trial with 132 students in an introductory programming course. Students completed three tasks related to code-writing and debugging and were randomly assigned to one of four AI TAs varied across two dimensions: pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (no context vs. full context of the problem and student solution). We examined studentsā perceptions, interaction behaviors, and evidence of post-task comprehension. Students rated the Socratic AI TA with full context least favorably, reporting significantly lower perceived support for task completion. Descriptively, this condition also showed the highest observed interaction stress, the highest rate of external LLM use, and the lowest proportion of post-task explanations demonstrating full comprehension, though these differences were not statistically significant. These findings suggest that guardrailed AI TAs are not automatically better for learning. Instead, their effectiveness depends on how pedagogical guidance and contextual awareness are balanced in ways that students experience as useful, supportive, and worth continuing to use. Keywords: AI tutor, Feedback, Digital TAs, Automated Tutors, Programming 1. Introduction LLM-backed AI teaching assistants (AI TAs) are increasingly being deployed in programming courses to provide scalable, always-available help while using pedagogical guardrails to avoid simply giving students solutions (Liffiton et al., 2024; Kazemitabaar et al., 2024; Sheese et al., 2024; Kapoor et al., 2026). Unlike general-purpose LLM chat tools, these systems are designed to support learning by providing tutor-like guidance without directly completing the work for students. Yet these guardrails create a design tension, because if a system is too restrictive or if students perceive the help as poorly contextualized to their request, they may disengage, avoid seeking help, or bypass approved tools in favor of unrestricted general-purpose LLMs (Ko et al., 2025; Kapoor et al., 2026). This creates a problem for educators because when students migrate to external systems, opportunities to capture learning-related data are lost, and students may become more susceptible to over-reliance on AI-generated solutions (Bo et al., 2026; Denny et al., 2024; Shein, 2024). At the same time, student favorability is not necessarily equivalent to learning effectiveness: a guardrailed AI TA that provides more friction between the student and the answer may be perceived as less helpful or favorable to the student, while still supporting the productive struggle associated with deeper learning. The goal, then, is not to maximize favorability alone, but to identify configurations that are both pedagogically sound and experienced by students as useful and worth continuing to use. A central challenge for our community is to understand how design choices shape studentsā experiences with AI TAs. Prior work has introduced and evaluated guardrailed AI systems for programming support (Liffiton et al., 2024; Kazemitabaar et al., 2024; Kapoor et al., 2026), and instructors have expressed a strong preference for customizing AI chatbots to fit their course-specific needs (Hou et al., 2026). However, most prior work has evaluated fixed AI TA systems as complete interventions, without explicitly testing how different configurations of the same system affect student perceptions, behaviors, and learning-related outcomes. In this paper, we address this gap through a randomized controlled experiment comparing four configurations of the same AI TA. We vary pedagogical guidance style by comparing a Socratic AI TA that guides students through questions with a Direct AI TA that provides brief explanations while still withholding complete solutions. We vary context awareness by comparing an AI TA with full access to the problem statement and the studentās current solution with an AI TA that has access only to the chat history. We investigate studentsā learning experiences with these AI TAs by answering the following research question: RQ: To what extent does an AI TAās pedagogical guidance style and context awareness affect introductory programming studentsā learning experiences, as measured through their (a) perceptions, (b) behaviors, and (c) task comprehension? We conducted a 2Ć2 factorial randomized controlled experiment with 132 students in an introductory Python course. Students completed one code-writing and two code-debugging tasks, and each student was randomly assigned to interact with one of four AI TAs configured with a specific combination of pedagogical style and context. We analyzed studentsā perceptions from a post-survey, interaction behaviors from tool logs, and task comprehension from a post-task explanation of a debugging solution. Notably, the AI TA that combined Socratic guidance with full context was rated least favorably for supporting task completion, showing that pedagogically motivated designs are not automatically experienced by students as more helpful. Our findings provide evidence-based guidance for educators and developers seeking to deploy such tools effectively in real classroom settings. 2. Related Work Recent work in computing education exploring the use of large language models (LLMs) can be broadly grouped into two strands. The first examines the pedagogical capabilities and limitations of LLMs, including their instructional soundness, learning implications, and ability to generate programming tasks or feedback (Raihan et al., 2025; Sarsa et al., 2022; Hellas et al., August 7-11, 2023). The second investigates student reception and adoption of scaffolded, pedagogically grounded LLM systems, including AI TA tools that are deliberately constrained to promote productive learning behaviors (Liu et al., 2024; Liffiton et al., 2024; Kazemitabaar et al., 2024; Zamfirescu-Pereira et al., February 26-March 1, 2025; AkƧapınar and Sidan, 2024; Li and Ma, 2025; Denny et al., July 8-10, 2024). Evaluations of systems such as CodeHelp (Liffiton et al., 2024) and CodeAid (Kazemitabaar et al., 2024) are well known examples of this latter strand of work. There is much interest in this space even beyond computing education, where similar systems have been developed, deployed, and evaluated across disciplines such as chemistry (Wang et al., 2025), medicine (Fan et al., 2025), architecture (Wang and Wu, 2025), engineering (Shi et al., 2026; Ter Sarkisian et al., 2026), and physics (Alipour et al., 2025). As such systems become more common, there is a need for research that moves towards understanding how students interact with AI TAs, and especially how particular design choices shape those interactions. One important design dimension is pedagogical guidance style. Socratic questioning is pedagogically appealing because it guides students toward discovering solutions themselves rather than providing direct answers (Vygotsky and Kozulin, 2012). Prior work in computing education has shown that Socratic techniques can support learning (Tamang et al., 2021), and recent studies have explored whether LLMs can generate Socratic guidance for programming tasks (Al-Hossami et al., 2024; Liding et al., 2026) and whether Socratic AI TAs can improve learning and retention (Tran et al., 2026). For instance, Tran et al. found that slower-paced, Socratic-style AI TA interactions produced greater learning gains, particularly among students with less prior knowledge (Tran et al., 2026). They also found that faster-paced interactions provided initial learning benefits for more advanced students, but these gains were accompanied by reduced retention. Our work complements Tran et al. by extending the study of AI TA guidance beyond interaction pacing and learning/retention outcomes to examine how pedagogical guidance style and context availability shape studentsā perceptions, interaction behaviors, and immediate post-task comprehension. Additionally, previous work has found that studentsā reception of Socratic feedback is mixed (Ahmed et al., 2025). For example, Ahmed et al. found that students perceived Socratic hints as less helpful, less accurate, and more time-consuming than other feedback approaches, even when experts viewed the feedback more favorably (Ahmed et al., 2025). This suggests a tension between pedagogical intent and student experience that warrants further investigation in conversational AI TA settings. A second important design dimension is context awareness. Instructors have expressed a strong preference for customizing AI chatbots to fit their course-specific needs, including how and where the chatbot is deployed (Hou et al., 2026). In programming courses, context may include the problem statement, the studentās current code, and the history of the interaction. Providing this context may allow an AI TA to offer more relevant support, while withholding it may encourage students to articulate their problem more explicitly. Prior work has generally reported AI TA systems with fixed configurations (e.g. (Diaz et al., 2025; Liu et al., 2024; Feng et al., 2026)) rather than isolating how specific configurations affect student perceptions and learning. Our study addresses this important gap by experimentally comparing AI TAs that vary across both pedagogical guidance style and context awareness. 3. Methods11footnotemark: 1 3.1. Study Design Our study employed a between-subjects, 2Ć2 factorial parallel-arm randomized design (MacKenzie, 2012) to examine the impact of AI TAsā pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (Full Context of the problem statement and studentās current solution vs. No Context) on studentsā learning experiences. These experiences were operationalized through student perceptions, interaction behaviors, and task comprehension. The decision to manipulate these dimensions was motivated by prior work identifying task context and pedagogical strategy as important configurations preferred by instructors in AI TAs (Hou et al., 2026). Students were asked to solve three programming problems (one code-writing and two code-debugging tasks) on a web-based coding platform with assistance from one of four randomly assigned AI TAs. Following completion of the assignment, all participants completed a survey that gauged their perceptions of the AI TA, their help-seeking behavior, and their comprehension of one of the debugging tasks. The studyās methodology and data collection procedures were reviewed and approved by the universityās Institutional Review Board (IRB). Figure 1. Student-facing Interface of the Web-based Coding PlatformScreenshot of the coding tool showing the problem statement, submission history, AI assistant, code editor, and test results. 3.2. Coding Tool and AI TA design The study was implemented using a coding platform, which allows instructors to author and host programming questions (similar to (Kapoor et al., 2026). A screenshot of the web-based platform is shown in Figure 1. The application has a split-screen interface with the problem statement, student submission history, āAI assistantā on the left, and an embedded text editor on the right. Students could run their own tests or submit their code to run against the instructor-defined test suite. AI TAs were backed by the OpenAI GPT-4o LLM and returned responses to student queries based on custom prompts with pedagogical guardrails similar to CodeHelp (Liffiton et al., 2024). The four AI TAs differed only in their system prompts and contextual information; the interface presented to students was otherwise identical. In the Full Context conditions, the chatbotās system prompt was supplied with the problem statement, the studentās current code, and the chat history, whereas in the No Context conditions, it was supplied only with the studentās chat history. In the Direct conditions, the chatbot was instructed to provide brief, direct responses without producing complete code solutions, whereas in the Socratic conditions, it was instructed to guide students exclusively through targeted questions without providing direct answers. These two manipulations resulted in four AI TA configurations: (1) Direct + Full Context, (2) Direct + No Context, (3) Socratic + Full Context, and (4) Socratic + No Context. 3.3. Participants Students were recruited from an introductory Python programming course (CS1) at a large public research university in the United States during the Fall 2025 semester. Enrolled students (N=598N=598) were not familiar with the coding platform used in the study, but were invited to participate in our study through in-person and learning management system announcements. They were offered 0.5% extra credit toward their course grade for completing three programming tasks and a post-survey, and could receive the incentive while opting to exclude their data from research analysis. Data were collected during the final two weeks of the semester, by which point students were familiar with Python lists, loops, and conditionals. Table 1. Participant Distribution across 2Ć22Ć 2 Experimental Conditions (N=132N=132) Context Awareness Feedback Style Full Context No Context Row Total Direct Feedback 37 32 69 Socratic Feedback 30 33 63 Column Total 67 65 132 Of the 598 students enrolled in the course, 254 completed the intent-to-participate pre-survey. After excluding non-consenting students, incomplete responses, students who did not use the chatbot, and cases with missing or inconsistent chatbot logs, the final analytic dataset consisted of 132 consenting students. Although participants were randomly and evenly assigned to the four conditions at the pre-survey stage, some imbalance occurred because not all students who initially expressed interest completed the study. The final distribution of participants across conditions is shown in Table 1. Prior programming experience was broadly distributed: 24% reported being Experienced or Very experienced, 50% reported Average experience, and 25% reported being Inexperienced or Very inexperienced, using a self-estimation measure validated in prior work (Feigenspan et al., 2012). Table 2. Research Question Operationalization: Measures, Survey Items/Log Metrics, and Analysis Methods Experience Dimension Measures Survey Item/Log Metric Analysis Method (a) Perceptions Task completionā The [platform] chatbot helped me complete the tasks successfully. Kruskal-Wallis H test Task comprehensionā The [platform] chatbot helped me better understand the problem statements. Kruskal-Wallis H test Correct informationā The [platform] chatbot provided me with correct information. Kruskal-Wallis H test Stressful interactionā My experience using the [platform] chatbot was stressful. Kruskal-Wallis H test (b) Behaviors Used another LLM# Did you use another LLM or chatbot? Chi-square test Reason for LLM useā” Why did you use another LLM or chatbot? (if answered āYesā to previous question) Inductive coding Average no. of chats ā¢ Log data Kruskal-Wallis H test Average chat length ā¢ Log data Kruskal-Wallis H test (c) Task comprehension Recall solutionā” In the tasks you completed, you were asked to debug a function bar() that takes in a 10Ć10 matrix (2D List) for input, then gives the sum of numbers surrounding a specific cell. Below is the code snippet provided, with the error still present. def bar(values, row, col): xxxxsum = 0 xxxxfor i in range(row-2, row+3): xxxxxxxxfor j in range(col-2, col+3): xxxxxxxxxxxxif i != row or j != col: xxxxxxxxxxxxxxxxsum += values[i][j] xxxxreturn sum Please re-identify the error and briefly explain your solution to it. Inductive coding Note. Scale used: ā 5-point Likert scale (1 = Strongly Disagree, 5 = Strongly Agree). #Binary (Yes/No). ā”Open-ended response. 3.4. Study Tasks We selected three tasks that had been clearly described in prior studies on AI TAs in computing education (Denny et al., July 8-10, 2024; Kapoor et al., 2026). The first task was a code-writing task in which students completed a function that determined whether a value was prime. The second task was a debugging task in which students fixed a function intended to determine whether a list contained repeated values. The third task was a debugging task in which students fixed a function that summed values surrounding a specific cell in a 10Ć10 matrix; the provided implementation had a boundary-checking bug. Function names were anonymized across tasks to reduce the extent to which the chatbot could infer the intended solution from the function name, which was important for evaluating the role of context awareness. Each student was randomly invited to one of four course shells in our tool, each with a different AI TA but the three identical tasks. Students were given one week to complete the study. Each task included ten instructor-provided test cases, and students were instructed to use the chatbot to help solve the problems. Most students (128 out of 132) completed all three tasks, passing all test cases. Across all conditions, the average score was 2.98 out of 3 (māeādāiāaān=3,māiān=1,māaāx=3,Ļ=0.19median=3,min=1,max=3,Ļ=0.19). Students initiated a total of 2,467 messages to the AI TA across the three problems (ā¼ 6 messages per student per problem). 3.5. Data Collection and Analysis Data were collected from a post-survey and tool logs that recorded task completion and AI TA interaction behavior. Survey questions were informed by prior work in the community (Liffiton et al., 2024; Denny et al., July 8-10, 2024). Perception measures used 5-point Likert-scale items assessing whether the chatbot helped students complete the tasks, understand the problem statements, receive correct information, and whether the interaction was stressful. Behavioral measures included whether students used another LLM or chatbot, their reasons for doing so, the number of chat messages sent to the AI TA, and average message length. Task comprehension was assessed through an open-ended post-task response in which students re-identified and explained their solution to the boundary-checking bug from the matrix debugging task. Table 2 summarizes how each research question dimension was operationalized, including the corresponding survey items or log metrics and analysis methods. Quantitative Analysis. Quantitative survey and log data were analyzed using descriptive statistics, followed by inferential tests. Differences across the four experimental conditions were examined using the non-parametric Kruskal-Wallis H test, chosen because the Likert-scale ordinal responses and numerical log data did not follow a normal distribution (Shapiro-Wilk test, p<.001p<.001). Studentsā use of another LLM was analyzed using a chi-square test. The significance level was set at Ī±=.05Ī±=.05 for all tests. Qualitative Analysis. Responses to the open-ended questions on task comprehension and studentsā reasons for using an alternative LLM were analyzed using inductive content analysis (Saldana, 2011). Student responses were inductively coded by two of the authors, with each author coding one of the open-ended questions. The initial codes were iteratively grouped and abstracted into higher-order categories. Two researchers reviewed all codes and categories to verify their consistency, with discrepancies resolved through discussion. 4. Results 4.1. Perception Measures Figure 2. Mean Student Ratings for AI TAs by Interaction of Context Awareness and Pedagogical Guidance Style (N=132) Note. Ratings measured on a 5-point Likert scale (1 = Strongly Disagree, 5 = Strongly Agree). A KruskalāWallis H test revealed a statistically significant effect of condition on perceived support of the AI TA for task completion (Ļ2ā(3)=12.14Ļ^2(3)=12.14, p=.007p=.007, N=132N=132). Students in the Socratic + Full Context condition rated the AI TA lowest on this measure (mean rank = 48.63, Ī¼=3.53Ī¼=3.53), substantially below the other three conditions: Direct + Full Context (mean rank = 76.85, Ī¼=4.27Ī¼=4.27), Direct + No Context (mean rank = 70.75, Ī¼=4.16Ī¼=4.16), and Socratic + No Context (mean rank = 67.02, Ī¼=4.12Ī¼=4.12), as shown in Figure 2. Differences across the four conditions were not statistically significant for perceived support for task comprehension (Ļ2ā(3)=4.79Ļ^2(3)=4.79, p=.188p=.188), provision of correct information (Ļ2ā(3)=1.71Ļ^2(3)=1.71, p=.635p=.635), or stressful interaction (Ļ2ā(3)=4.50Ļ^2(3)=4.50, p=.213p=.213). Descriptively, however, students in the Direct conditions rated the AI TA higher on task comprehension than those in the Socratic conditions, and students in the Socratic + Full Context condition rated their interaction as more stressful than students in the other three conditions. Perception findings: Students assigned to a Socratic AI TA with full context rated the AI TA least favorably across the perception measures. They reported significantly lower perceived support for task completion, and descriptively the lowest support for task comprehension and the highest stressful interaction rating. 4.2. Behavior 4.2.1. Using another LLM Overall, 15% of students (20 of 132) reported using another LLM or chatbot during the experiment. Although usage rates differed descriptively across conditions, the differences were not statistically significant (Ļ2ā(3)=2.47Ļ^2(3)=2.47, p=.48p=.48). Students in the Socratic + Full Context condition reported the highest rate of external LLM use (23%, 7 of 30 students), compared with 9ā15% in the other three conditions. We coded studentsā open-ended explanations for why they used another LLM. Their reasons were abstracted into three not mutually exclusive categories: inadequate guidance (n=17n=17), going in circles (n=4n=4), and forgetting context (n=2n=2). The most common concern was that the chatbot did not provide enough support, gave explanations that were too general or inflexible, or failed to work dynamically with the student. For example, one student in the Socratic + Full Context condition wrote that the chatbot ādoesnāt really expand or work with the student very wellā and was āsuper rigidā. Another student in the Socratic + No Context condition described the chatbot as using ācircle reasoningā, where it repeatedly answered questions with more questions rather than helping with the code. 4.2.2. Interaction volume A KruskalāWallis H test indicated a statistically significant difference in the number of queries students sent to the AI TA across the four conditions (Ļ2ā(3)=13.28Ļ^2(3)=13.28, p=.004p=.004). Students in the Socratic + No Context condition sent substantially more queries (Ī¼=11.1Ī¼=11.1 per problem) than students in the Socratic + Full Context (Ī¼=5.1Ī¼=5.1), Direct + Full Context (Ī¼=4.1Ī¼=4.1), and Direct + No Context (Ī¼=4.7Ī¼=4.7) conditions. This suggests that students who received Socratic guidance without problem context required more back-and-forth with the chatbot to make progress. A KruskalāWallis H test also revealed a significant difference in average message length across conditions (Ļ2ā(3)=29.96Ļ^2(3)=29.96, p<.001p<.001). Students in the Direct + No Context condition wrote the longest messages on average (Ī¼=126Ī¼=126 characters per message), followed by Direct + Full Context (Ī¼=81Ī¼=81), Socratic + No Context (Ī¼=69Ī¼=69), and Socratic + Full Context (Ī¼=54Ī¼=54). This pattern suggests that when the chatbot lacked context, students often had to describe their situation in more detail before receiving useful support. Behavior findings: Behavioral engagement differed significantly across conditions in query volume and message length. Students in the Socratic + No Context condition sent the most queries per problem, while students in the Socratic + Full Context condition sent the shortest messages on average. Descriptively, students in the Socratic + Full Context condition also reported the highest, though non-significant, rate of external LLM use. Figure 3. Decision Flowchart for Task ComprehensionDecision flowchart for task comprehension categories 4.3. Task Comprehension Analysis We analyzed studentsā responses to a post-task survey question, administered immediately after task completion, asking them to re-identify the error and explain their solution for Task 3 (āSurrounding Sumā). This assessment reflects studentsā immediate comprehension of the problem alongside their own debugging solution, rather than a longer-term retention of the underlying concept, and allowed us to examine how the AI TAās guidance style and context awareness may have shaped that comprehension. Responses were coded into four comprehension categories (see Figure 3). Full comprehension required students to identify the indices out-of-bounds bug and explain a correct fix. Partial/impure comprehension captured responses that identified the correct bug but had incomplete, unclear, or faulty reasoning. Minimal comprehension captured responses that did not identify the out-of-bounds bug. Ungaugable comprehension captured responses where it was unclear whether they had identified the bug. Figure 4. Comprehension Level by Group AssignmentComprehension Level by Group Assignment Across all non-blank responses (n=128n=128), 57% students showed full comprehension, 30% showed partial/impure comprehension, 11% showed minimal comprehension, and 2% were ungaugable. When broken down by condition, the Direct + Full Context group had the highest proportion of full-comprehension responses (67%), followed by Direct + No Context (61%), Socratic + No Context (50%), and Socratic + Full Context (48%). The Socratic + Full Context group also had the highest proportion of minimal-comprehension responses (14%, see Figure 4). Overall, comprehension patterns appeared to differ more by pedagogical guidance style than by context awareness: Direct conditions produced a higher average proportion of full-comprehension responses than Socratic conditions. Task comprehension findings: Descriptively, students in the Direct conditions showed a higher proportion of responses that demonstrated full-comprehension than students in the Socratic conditions. The Direct + Full Context condition had the highest proportion of full-comprehension responses, while the Socratic + Full Context condition had the lowest. 5. Discussion Role of Pedagogical Guidance Style: Our results suggest that the AI TAās guidance style shaped how students engaged with the tool. Students in the Socratic conditions sent the most queries, suggesting that a chatbot that responds with guiding questions rather than direct answers can require more back-and-forth before students make progress. In this sense, Socratic guidance appeared to increase interaction volume as observed in other domains outside of computing (Ter Sarkisian et al., 2026), especially when the chatbot lacked task context. However, the perception and comprehension results tell a more cautionary story. Students rated the Socratic + Full Context condition least favorably for supporting task completion, and this condition also had the lowest proportion of full-comprehension post-task explanations. This raises an important tension for educators considering AI TA deployment. A pedagogically motivated approach that encourages students to reason through problems independently may still be experienced as frustrating or unhelpful (similar to results from Ahmed et al. (Ahmed et al., 2025)), especially when students are focused on completing a programming task. Educators should therefore consider whether the goal of a given assignment is task completion, conceptual understanding, or productive struggle, and configure the AI TAās guidance style accordingly. Role of Contextual Awareness: Context awareness also played a notable role in shaping student behavior. Students in the No Context conditions, where the chatbot had access only to the chat history and not to the problem statement or studentās current code, tended to write longer individual queries than those in the Full Context conditions. This suggests that when the chatbot lacked information about the problem, students compensated by providing more detail in their messages, essentially doing the work of explaining their situation before the chatbot could help them. At the same time, Full Context did not guarantee a better student experience. The Socratic + Full Context condition produced the shortest messages on average, suggesting that students did not need to explain much before receiving a response. Yet this same condition was rated least favorably for task completion and had the lowest proportion of full-comprehension responses on the post-task. This points to a practical implication for educators, which is that providing an AI TA with full task context may reduce the effort required to formulate queries, but contextual awareness alone does not ensure that the support will feel useful or lead to stronger understanding. Instructors deploying AI TAs in programming courses should therefore consider integrating the tool with the assignment environment while also carefully designing how the chatbot uses that context pedagogically. Interaction Between Pedagogical Guidance Style and Context Awareness: The interaction plots in Figure 2 show why guidance style and context awareness should not be interpreted independently. When Direct guidance was available, students rated the chatbot similarly for supporting task completion regardless of how much context was provided. In contrast, when the chatbot was Socratic, providing full context reduced studentsā perceived support for task completion. This suggests that context awareness was not uniformly beneficial, and that its effect depended on the pedagogical style with which it was paired. The behavioral results reinforce this interpretation. The Socratic + No Context condition produced the highest query volume, which makes intuitive sense because when a chatbot avoids direct answers and also lacks knowledge of the task, students must work harder to get useful help. The result may be a high-effort interaction that feels more exhausting than educational, particularly for students who are already struggling. In contrast, the Direct + Full Context condition produced the lowest query volume, suggesting that students could get help quickly when the chatbot provided direct guidance and already had access to the problem and their current code. These findings suggest that guardrails can become roadblocks when pedagogical guidance and contextual awareness are not well balanced. A Socratic chatbot with limited context may demand too much effort from students, while a highly contextualized chatbot that remains rigidly Socratic may still fail to provide the kind of support students find useful, as reflected in the weaker descriptive post-task comprehension pattern for the Socratic + Full Context condition. Rather than treating āmore scaffoldingā or āmore contextā as inherently better, educators and system designers should consider AI TAs as configurable learning supports whose effectiveness depends on the fit between the task, the studentās needs, and the form of help provided. Future work should explore adaptive AI TAs that can shift between Socratic and more direct guidance based on studentsā progress, confusion, or help-seeking behavior. Limitations: Our study was conducted in an introductory programming course at a large research university. Although this controlled setting supports high internal validity, it may limit the generalizability of the findings to other contexts. Another limitation is that we measured task comprehension for only one task, a design choice we made to reduce survey fatigue. This design captures immediate comprehension rather than knowledge retention. Future studies could investigate the impact of AI TA configurations on long-term knowledge retention. Finally, qualitative coding of open-ended responses involves some degree of subjectivity; we mitigate this by developing a codebook, transparent reporting of our process, and by providing a detailed flowchart of our approach. 6. Conclusion AI TAs are increasingly being deployed in programming courses to provide scalable help while using pedagogical guardrails to avoid simply giving students complete solutions. However, our findings show that not all guardrailed designs are experienced by students as helpful ā their effect depends on pedagogical guidance style and context awareness. In this study, we moved beyond evaluating a single AI TA system as a complete intervention (similar to (Kazemitabaar et al., 2024; Li and Ma, 2025; Liffiton et al., 2024; Liu et al., 2024)) by experimentally comparing four configurations of the same AI TA, varying both guidance style and context awareness. Across studentsā perceptions, interaction behaviors, and post-task comprehension, the Socratic + Full Context condition showed the least favorable descriptive pattern. Students rated it significantly lower in perceived support for task completion, and it also showed less favorable descriptive patterns for perceived comprehension support, higher interaction stress, greater external LLM use, and lower post-task comprehension. These results highlight a key design challenge for AI TAs in programming courses, which is that pedagogically motivated configurations can become roadblocks if students experience them as too rigid or insufficiently useful. Effective AI TA design will therefore require balancing pedagogical guidance with contextual awareness in ways that support learning while keeping students engaged with course-approved tools. References Ahmed et al. (2025) U. Z. Ahmed, S. Sahai, B. Leong, and A. Karkare Feasibility study of augmenting teaching assistants with ai for cs1 programming feedback. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, SIGCSETS 2025, New York, NY, USA, p. 11ā17. External Links: ISBN 9798400705311, Link, Document Cited by: Ā§2, Ā§5. AkƧapınar and Sidan (2024) G. AkƧapınar and E. Sidan AI chatbots in programming education: guiding success or encouraging plagiarism. Discover Artificial Intelligence 4 (1). Note: Cited by: 16; All Open Access, Gold Open Access External Links: Document, Link Cited by: Ā§2. Al-Hossami et al. (2024) E. Al-Hossami, R. Bunescu, J. Smith, and R. Teehan Can language models employ the socratic method? experiments with code debugging. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2024, New York, NY, USA, p. 53ā59. External Links: ISBN 9798400704239, Link, Document Cited by: Ā§2. Alipour et al. (2025) S. Alipour, L. Coward, J. Burris, B. Karki, and T. Liao Evaluating an ai teaching assistant in vr physics education: a case study of user experience and effectiveness. In 2025 IEEE International Symposium on Emerging Metaverse (ISEMV), Vol. , p. 62ā68. External Links: Document Cited by: Ā§2. Bo et al. (2026) J. Y. Bo, M. Kazemitabaar, M. Deng, M. Inzlicht, and A. Anderson Invisible saboteurs: sycophantic llms mislead novices in problem-solving tasks. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI ā26, New York, NY, USA. External Links: ISBN 9798400722783, Link, Document Cited by: Ā§1. Denny et al. (July 8-10, 2024) P. Denny, S. MacNeil, J. Savelka, L. Porter, and A. Luxton-Reilly Desirable characteristics for ai teaching assistants in programming education. In Proceedings of the 2024 ACM Conference on Innovation and Technology in Computer Science Education V.1, ITiCSE ā24, New York, NY, USA. External Links: Document, Link Cited by: Ā§2, Ā§3.4, Ā§3.5. Denny et al. (2024) P. Denny, J. Prather, B. A. Becker, J. Finnie-Ansley, A. Hellas, J. Leinonen, A. Luxton-Reilly, B. N. Reeves, E. A. Santos, and S. Sarsa Computing education in the era of generative ai. Commun. ACM 67 (2), p. 56ā67. External Links: ISSN 0001-0782, Link, Document Cited by: Ā§1. Diaz et al. (2025) M. Diaz, D. Karp, P. Tuli, and A. Kapoor Edugator: an ai-enabled tool for creating and delivering interactive computing content. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 2, SIGCSE TS 2025, New York, NY, USA, p. 1732. External Links: ISBN 9798400705328, Link, Document Cited by: Ā§2. Fan et al. (2025) H. Fan, W. Du, J. Yan, and H. Jiang Application of deepseek-based ai teaching assistant in teaching anesthesiology theories. BMC Medical Education. Cited by: Ā§2. Feigenspan et al. (2012) J. Feigenspan, C. KƤstner, J. Liebig, S. Apel, and S. Hanenberg Measuring programming experience. In 2012 20th IEEE International Conference on Program Comprehension (ICPC), Vol. , p. 73ā82. External Links: Document Cited by: Ā§3.3. Feng et al. (2026) T. H. Feng, S. Hooper, P. Denny, B. C. WĆ¼nsche, and A. Luxton-Reilly Bot or prof: comparing the characteristics of ai-generated and instructor-written responses to student queries in an introductory programming course forum. In Proceedings of the 28th Australasian Computing Education Conference, ACE ā26, New York, NY, USA, p. 63ā72. External Links: ISBN 9798400723520, Link, Document Cited by: Ā§2. Hellas et al. (August 7-11, 2023) A. Hellas, J. Leinonen, S. Sarsa, C. Koutcheme, L. KujanpƤ, and J. Sorva Exploring the responses of large language models to beginner programmersā help requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1, ICER ā23 V.1, New York, NY, USA. External Links: Document, Link Cited by: Ā§2. Hou et al. (2026) I. Hou, Z. Xiong, P. J. Guo, and A. Y. Wang āBespoke botsā: diverse instructor needs for customizing generative ai classroom chatbots. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI ā26, New York, NY, USA. External Links: ISBN 9798400722783, Link, Document Cited by: Ā§1, Ā§2, Ā§3.1. Kapoor et al. (2026) A. Kapoor, P. Denny, L. Porter, S. MacNeil, and M. Diaz Exploring student behaviors and motivations when using ai teaching assistants with optional guardrails. In Proceedings of the 28th Australasian Computing Education Conference, ACE ā26, New York, NY, USA, p. 22ā31. External Links: ISBN 9798400723520, Link, Document Cited by: Ā§1, Ā§1, Ā§3.2, Ā§3.4. Kazemitabaar et al. (2024) M. Kazemitabaar, R. Ye, X. Wang, A. Z. Henley, P. Denny, M. Craig, and T. Grossman CodeAid: evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ā24, New York, NY, USA. External Links: ISBN 9798400703300, Link, Document Cited by: Ā§1, Ā§1, Ā§2, Ā§6. Ko et al. (2025) S. Ko, M. Zahn, K. Stephens-Martinez, Y. Velasco, L. Battestilli, and S. Heckman Relationships between computing studentsā characteristics, help-seeking approaches, and help-seeking behavior in introductory courses and beyond. In Proceedings of the 2025 ACM Conference on International Computing Education Research V.1, ICER ā25, New York, NY, USA, p. 313ā326. External Links: ISBN 9798400713408, Link, Document Cited by: Ā§1. Li and Ma (2025) H. Li and B. Ma CodeRunner agent: integrating ai feedback and self-regulated learning to support programming education. In International Conference on Computers in Education, Cited by: Ā§2, Ā§6. Liding et al. (2026) Z. C. Liding, M. Osmolovskiy, H. Lanka, R. Howard, N. Roy, and R. Borela To tell or to ask? comparing the effects of targeted vs. socratic ai hints. In Proceedings of the 57th ACM Technical Symposium on Computer Science Education V.2, SIGCSE TS 2026, New York, NY, USA, p. 1423ā1424. External Links: ISBN 9798400722554, Link, Document Cited by: Ā§2. Liffiton et al. (2024) M. Liffiton, B. E. Sheese, J. Savelka, and P. Denny CodeHelp: using large language models with guardrails for scalable support in programming classes. In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research, Koli Calling ā23, New York, NY, USA. External Links: ISBN 9798400716539, Link, Document Cited by: Ā§1, Ā§1, Ā§2, Ā§3.2, Ā§3.5, Ā§6. Liu et al. (2024) R. Liu, C. Zenke, C. Liu, A. Holmes, P. Thornton, and D. J. Malan Teaching cs50 with ai: leveraging generative artificial intelligence in computer science education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2024, New York, NY, USA, p. 750ā756. External Links: ISBN 9798400704239, Link, Document Cited by: Ā§2, Ā§2, Ā§6. MacKenzie (2012) I. S. MacKenzie Human-computer interaction: an empirical research perspective. Morgan Kaufmann. Cited by: Ā§3.1. Raihan et al. (2025) N. Raihan, M. L. Siddiq, J. C.S. Santos, and M. Zampieri Large language models in computer science education: a systematic literature review. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, SIGCSETS 2025, New York, NY, USA, p. 938ā944. External Links: ISBN 9798400705311, Link, Document Cited by: Ā§2. Saldana (2011) J. Saldana Fundamentals of qualitative research. Oxford university press. Cited by: Ā§3.5. Sarsa et al. (2022) S. Sarsa, P. Denny, A. Hellas, and J. Leinonen Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1, ICER ā22, New York, NY, USA, p. 27ā43. External Links: ISBN 9781450391948, Link, Document Cited by: Ā§2. Sheese et al. (2024) B. Sheese, M. Liffiton, J. Savelka, and P. Denny Patterns of student help-seeking when using a large language model-powered programming assistant. In Proceedings of the 26th Australasian Computing Education Conference, ACE ā24, New York, NY, USA, p. 49ā57. External Links: ISBN 9798400716195, Link, Document Cited by: Ā§1. Shein (2024) E. Shein The impact of ai on computer science education. Commun. ACM 67 (9), p. 13ā15. External Links: ISSN 0001-0782, Link, Document Cited by: Ā§1. Shi et al. (2026) K. Shi, P. Xu, J. Zhang, F. Tang, X. Zhuan, X. Dong, and Y. Chen Research on the impact of ai teaching assistants on electrical engineering coursesāa case study of the course āexploring the world of electricityā. In The Proceedings of the International Council on Electrical Engineering Conference 2025(ICEE 2025), X. Dong (Ed.), Singapore, p. 273ā282. External Links: ISBN 978-981-95-6915-1 Cited by: Ā§2. Tamang et al. (2021) L. J. Tamang, Z. Alshaikh, N. A. Khayi, P. Oli, and V. Rus A comparative study of free self-explanations and socratic tutoring explanations for source code comprehension. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, SIGCSE ā21, New York, NY, USA, p. 219ā225. External Links: ISBN 9781450380621, Link, Document Cited by: Ā§2. Ter Sarkisian et al. (2026) A. G. Ter Sarkisian, C. Miller, N. Poonjolai, C. Aramburu, and J. Atweh Albert or alberta? investigating socratic dialogue and modality in ai-assisted engineering problem solving. In 2026 Systems and Information Engineering Design Symposium (SIEDS), Vol. , p. 1ā6. External Links: Document Cited by: Ā§2, Ā§5. Tran et al. (2026) K. Tran, G. Gao, A. Lombard, T. Yu, H. Jiang, and T. Y. Yeh Pacing for mastery: optimizing llm interactions for learning. In Proceedings of the 57th ACM Technical Symposium on Computer Science Education V.1, SIGCSE TS 2026, New York, NY, USA, p. 1068ā1074. External Links: ISBN 9798400722561, Link, Document Cited by: Ā§2. Vygotsky and Kozulin (2012) L. S. Vygotsky and A. Kozulin Thought and language, revised and expanded edition. The MIT Press Paperback Series, MIT Press. External Links: ISBN 9780262517713, LCCN 2011048675 Cited by: Ā§2. Wang and Wu (2025) A. Wang and X. Wu Building a triadic model of technology, motivation, and engagement: a mixed-methods study of ai teaching assistants in design theory education. Frontiers in Psychology Volume 16 - 2025. External Links: Link, Document, ISSN 1664-1078 Cited by: Ā§2. Wang et al. (2025) X. Wang, L. Zhang, Y. Mao, D. J. McGillivray, and Z. Wang ChEdu: a guided ai teaching assistant for chemistry education and exam support. Journal of Chemical Education 102 (12), p. 5347ā5354. External Links: Document, Link Cited by: Ā§2. Zamfirescu-Pereira et al. (February 26-March 1, 2025) J. D. Zamfirescu-Pereira, L. Qi, B. Hartmann, J. DeNero, and N. Norouzi 61A bot report: AI assistants in CS1 save students homework time and reduce demands on staff. (now what?). In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V.1, SIGCSE TS ā25, New York, NY, USA. External Links: Document, Link Cited by: Ā§2.