Paper deep dive
Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems
Eduard Frankford, Erik Cikalleshi, Ruth Breu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/10/2026, 4:01:17 AM
Summary
This paper presents a systematic review of conversational assessment approaches in programming education, identifying three architectural families: rule-based, LLM-based, and hybrid systems. It proposes a 'Hybrid Socratic Framework' for integrating conversational verification into Automated Programming Assessment Systems (APASs) to address the challenge of students submitting functionally correct code without conceptual understanding.
Entities (4)
Relation Signals (2)
Hybrid Socratic Framework → integratesinto → Automated Programming Assessment Systems
confidence 95% · The paper synthesizes these findings into a Hybrid Socratic Framework for integrating conversational verification into Automated Programming Assessment Systems (APASs).
Large Language Models → challenges → Automated Programming Assessment Systems
confidence 90% · Large Language Models (LLMs) challenge conventional automated programming assessment
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) challenge conventional automated programming assessment because students can now produce functionally correct code without demonstrating corresponding understanding. This paper makes two contributions. First, it reports a saturation-based scoping review of conversational assessment approaches in programming education. The review identifies three dominant architectural families: rule-based or template-driven systems, LLM-based systems, and hybrid systems. Across the literature, conversational agents appear promising for scalable feedback and deeper probing of code understanding, but important limitations remain around hallucinations, over-reliance, privacy, integrity, and deployment constraints. Second, the paper synthesizes these findings into a Hybrid Socratic Framework for integrating conversational verification into Automated Programming Assessment Systems (APASs). The framework combines deterministic code analysis with a dual-agent conversational layer, knowledge tracking, scaffolded questioning, and guardrails that tie prompts to runtime facts. The paper also discusses practical safeguards against LLM-generated explanations, including proctored deployment modes, randomized trace questions, stepwise reasoning tied to concrete execution states, and local-model deployment options for privacy-sensitive settings. Rather than replacing conventional testing, the framework is intended as a complementary layer for verifying whether students understand the code they submit.
Tags
Links
- Source: https://arxiv.org/abs/2604.07304v1
- Canonical: https://arxiv.org/abs/2604.07304v1
Trouble viewing inline? Open PDF directly →
Full Text
49,208 characters extracted from source content.
Expand or collapse full text
Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems Eduard Frankford 1 a , Erik Cikalleshi 1b and Ruth Breu 1c 1 University of Innsbruck, Department of Computer Science, Innsbruck, Austria eduard.frankford@uibk.ac.at, erik.cikalleshi@student.uibk.ac.at, ruth.breu@uibk.ac.at Keywords: Systematic Review, Conversational Agents, Automated Programming Assessment Systems, Large Language Models, Program Comprehension, Socratic Questioning Abstract: Large Language Models (LLMs) challenge conventional automated programming assessment because stu- dents can now produce functionally correct code without demonstrating corresponding understanding. This paper makes two contributions. First, it reports a saturation-based scoping review of conversational assessment approaches in programming education. The review identifies three dominant architectural families: rule-based or template-driven systems, LLM-based systems, and hybrid systems. Across the literature, conversational agents appear promising for scalable feedback and deeper probing of code understanding, but important limi- tations remain around hallucinations, over-reliance, privacy, integrity, and deployment constraints. Second, the paper synthesizes these findings into a Hybrid Socratic Framework for integrating conversational verification into Automated Programming Assessment Systems (APASs). The framework combines deterministic code analysis with a dual-agent conversational layer, knowledge tracking, scaffolded questioning, and guardrails that tie prompts to runtime facts. The paper also discusses practical safeguards against LLM-generated ex- planations, including proctored deployment modes, randomized trace questions, stepwise reasoning tied to concrete execution states, and local-model deployment options for privacy-sensitive settings. Rather than replacing conventional testing, the framework is intended as a complementary layer for verifying whether stu- dents understand the code they submit. 1 Introduction Large Language Models have fundamentally changed programming education and assessment. The grow- ing complexity of modern coding skills has made AI a necessary tool for supporting the learning process (Manorat et al., 2025). However, this shift also in- troduces substantial assessment risks. While tools such as ChatGPT can provide personalized feedback, they can also generate complete solutions that stu- dents may submit without understanding (Lin et al., 2025). This creates a difficult situation for assessment. At present, most Automated Programming Assess- ment Systems (APASs) still rely primarily on unit tests and basic static checks to validate functional cor- rectness. These mechanisms remain useful, but they are weak proxies for conceptual understanding when students can generate plausible code with LLMs (Vin- a https://orcid.org/0009-0005-5959-4936 b https://orcid.org/0009-0007-4349-4249 c https://orcid.org/0000-0001-7093-4341 tila, 2024). Consequently, assessment must move be- yond the final artifact and examine whether students can explain the logic, execution, and design decisions behind their submissions (Lehtinen et al., 2023). In this paper, the term conversational agent is used as an umbrella term for chatbots and dialog-based tu- toring or assessment systems that maintain multi-turn natural-language interaction with a learner and adapt subsequent turns based on previous answers (Debets et al., 2025). Conversational agents offer a promis- ing way to bridge the assessment gap. Instead of re- lying only on static quizzes or black-box functional tests, these systems can engage students in dialogue and use scaffolding techniques to probe how well they understand the code they submitted (Cheng et al., 2025). Isolated examples of this idea already exist, such as systems that ask questions about a learner’s code (Lehtinen et al., 2023) or tools designed to verify authorship and understanding (Vintila, 2024). How- ever, there is still no widely adopted design synthe- sis for integrating these conversational elements into APASs in a transparent and scalable manner. arXiv:2604.07304v1 [cs.SE] 8 Apr 2026 Accordingly, this paper is structured in two parts. First, it presents a systematic review of chatbot-based and conversational assessment approaches in pro- gramming education. Second, it derives a framework proposal from the resulting findings. The overall goal is to identify effective practices, categorize technical approaches, surface limitations, and synthesize de- sign guidance for a next generation of APASs that can assess both code functionality and student under- standing. 2 Related Work The use of AI in computer science classrooms is expanding rapidly. A recent review covered more than 100 papers to categorize how such tools are be- ing used, from course design to grading (Manorat et al., 2025). While that review highlights the util- ity of AI for feedback generation and grading sup- port, it addresses the educational landscape broadly. The present study narrows the focus to conversational agents used to verify knowledge during programming assessment. When looking at chatbots specifically, Debets et al. found that most existing tools are designed for teaching rather than testing (Debets et al., 2025). In their review of 71 papers, they noted that many chat- bots rely on platforms such as Dialogflow but often lack a strong theoretical foundation. The present work focuses on the intersection of chatbots, program com- prehension, and formal assessment. Traditional assessment systems usually rely on unit tests and static code analysis. However, pass- ing a test case does not necessarily mean that a stu- dent understands the underlying concept. Research by Lehtinen et al. shows that students often achieve unproductive success, where they pass the assignment through trial and error or copying without truly under- standing the code (Lehtinen et al., 2023; Canic ̧o and Santos, 2025). Approaches such as asking students to explain their own code have been proposed to address this gap (Lehtinen et al., 2023), but a systematic ac- count of how such conversational verification can be integrated into an automated grading pipeline is still missing. 3 Methodology 3.1 Review Questions and Design Objective This paper intentionally separates review questions from the framework proposal. The systematic review answers two research questions, and the framework section then synthesizes the findings into a design proposal. • RQ1: How can conversational assessment ap- proaches in programming education be catego- rized based on their technical implementation, pedagogical strategies, and assessment methods? • RQ2: What are the benefits, limitations, and pedagogical implications of chatbot-based assess- ment approaches with respect to scalability, fair- ness, and learning outcomes? Based on the answers to RQ1 and RQ2, the sec- ond part of the paper derives a design objective: to propose a framework for integrating a conversa- tional agent into an APAS in a way that is transparent, grounded in program analysis, and scalable enough for larger cohorts. 3.2 Search Strategy and PRISMA-Guided Review Protocol The review followed a PRISMA-style workflow cov- ering identification, screening, eligibility, and inclu- sion. Figure 2 summarizes the selection process. To improve reproducibility, the full search strings, qual- ity assessment rubric, and row-level screening deci- sions are provided in the supplementary materials on Zenodo. 1 3.2.1 Information Sources Google Scholar was used as the primary meta-search engine to retrieve literature and citation links across the ACM Digital Library, IEEE Xplore, ScienceDi- rect, arXiv, and SpringerLink. To reduce the risk of missing papers because of ranking artifacts, targeted validation searches were also performed directly on these sources where appropriate. Searches were con- ducted between 1 October 2025 and 30 November 2025. 3.2.2 Search Strings Five search strings were developed using the PICOC framework (Population, Intervention, Comparison, 1 https://zenodo.org/records/18335209 Outcome, Context) to capture different aspects of conversational assessment in programming education (Kitchenham, 2012). The keyword groups were de- fined as follows: • Population: students in computer science or pro- gramming courses; APASs • Intervention:chatbots, conversational agents, conversational assessment, intelligent tutoring systems • Comparison: traditional methods, manual review, non-conversational assessment • Outcome:code understanding, learning out- comes, explanation quality, authorship verifica- tion • Context: university-level education in the LLM era To keep the main text concise, Figure 1 summa- rizes the keyword groups and query template. The five full search strings are provided in the supplemen- tary materials on Zenodo. 3.2.3 Search Execution and Screening Process Given the rapid evolution of LLMs in education, a saturation-based scoping review methodology was adopted. This approach was selected because the do- main remains small, fast-moving, and heterogeneous. 1. Initial Search: Each search string was executed across the selected sources between October and November 2025. Results were sorted by relevance where possible. Title and abstract screening were combined into a single stage because titles alone were often insufficient to determine relevance. 2. Iterative Screening: Search results were re- viewed page by page. Papers were retained for full-text review when they appeared to address conversational assessment, chatbot-based ques- tioning, code understanding, or intelligent tutor- ing in programming education. 3. Saturation-Based Stopping: For each search string and source combination, screening contin- ued until 3 consecutive result pages yielded no new relevant papers. The saturation point was usually reached after pages 3 to 5 of Google Scholar results. 4. Duplicate Removal and Decision Logging: Du- plicates were removed manually during screening. A screening log captured the source, search string identifier, retrieval window, screening stage, in- clusion decision, and exclusion reason for each full-text decision. 3.2.4 Snowballing To ensure comprehensive coverage, one iteration (k = 1) of snowballing was conducted (Wohlin, 2014), ap- plying both backward and forward snowballing to the papers that passed the initial inclusion and exclu- sion criteria. Forward snowballing examined papers that cited the included studies using Google Scholar’s Cited by feature, while backward snowballing re- viewed the reference lists of included papers. During this process, the same inclusion and exclu- sion criteria were applied. Papers already identified during the systematic search were excluded to avoid duplication. Very short papers such as workshop ab- stracts and posters were also excluded. Many cited papers were initially assessed by title and publication year. A substantial share of the retrieved work fo- cused on tutoring, debugging assistance, or one-way code explanations rather than using conversational in- teraction to verify student understanding during as- sessment. 3.2.5 Selection Criteria To ensure relevance and quality, studies were selected according to the following criteria. Inclusion was limited to peer-reviewed papers or reputable gray lit- erature published between 2018, marking the post- transformer era (Vaswani et al., 2017), and 2025. Se- lected studies were required to be written in English and to address programming or computer science ed- ucation, including transferable methods from closely related domains such as mathematics or logic. Cru- cially, the primary focus had to be on assessing code understanding via conversational agents, chatbots, or intelligent tutoring systems, or explicitly discussing the impact of LLMs on programming assessment. Records were excluded if they were promotional material, advertisements, or opinion pieces without technical or empirical substance.Studies focused on non-programming domains without a clear link to computer science education, or studies that men- tioned chatbots or assessment only tangentially, were discarded. Research published prior to 2018 and arti- cles whose full text was inaccessible through institu- tional access or author contact were also excluded. 4 Review Findings 4.1 Categorization of Approaches This section answers RQ1. Based on the system- atic analysis of the primary studies, conversational as- sessment approaches in programming education can Figure 1: Keyword groups and query template used to derive the five search string variants. Paper identified through database searching (n = 123) Papers screened by title and abstract (n = 123) Papers included from systematic search (n=24) Primary studies (n = 15) Supportive papers (n = 9) Papers eligible for snowballing (QA Score >= 3.5) (n = 9) Snowballing (k = 1) Backward + Forward (n = 462) Total papers included in review (n = 31) PRISMA Flow Diagram Paper title screening (n = 56) Additional papers identified (n = 7) Excluded (n = 99) IC5: Not about assesment/questions generation (n=64) IC4: Not context relevant (n=20) IC1: Publication Year (n = 7) Other criteria (n=3) Figure 2: PRISMA-style flow diagram of identification, screening, eligibility, and inclusion. be grouped into three dominant categories: (1) rule- based or template-driven systems, (2) LLM-based systems, and (3) hybrid systems. Using the domi- nant architecture of the concrete assessment systems discussed in the corpus, hybrid approaches appeared most frequently (5/12), followed by rule-based or template-driven systems (4/12) and LLM-based sys- tems (3/12). 4.1.1 Rule-Based and Template-Driven Systems Rule-based approaches represent the most structured method for conversational assessment. These sys- tems rely on deterministic algorithms and predefined templates to generate questions. Typically, they use static and dynamic code analysis to understand the student’s code (Alshaikh et al., 2021; Santos et al., 2022; Thomas et al., 2019). Static analysis often em- ploys Abstract Syntax Trees (ASTs) to identify code constructs such as variable declarations, loops, and conditional statements. Dynamic analysis simulates the execution of the code to track variable values and runtime behavior. Question generation is deterministic, meaning that the same input consistently produces the same ques- tions. This predictability supports consistency and auditability, but it also limits flexibility.Techni- cally, these systems parse code to extract structural elements such as variable declarations, loop con- structs, conditional statements, and function defini- tions. Questions are then generated by matching these elements against template libraries (Santos et al., 2022; Stankov et al., 2023; Thomas et al., 2019). For instance, when a for loop is detected, the sys- tem may generate questions about initialization, con- tinuation conditions, termination, and incrementation. Some rule-based systems extend this approach with dynamic execution and use simulated runs to gener- ate questions about runtime behavior, variable val- ues, and execution traces (Santos et al., 2022; Thomas et al., 2019). The primary advantage of rule-based systems lies in their consistency, modest computational overhead, scalability, and immediate feedback (Thomas et al., 2019; Santos et al., 2022). However, they face signifi- cant limitations regarding flexibility and variety. They cannot easily handle code patterns outside their prede- fined templates, which can lead to repetitive question- ing (Alshaikh et al., 2021). Building larger and more expressive template libraries also demands substantial time and maintenance effort. 4.1.2 LLM-Based Systems LLM-based systems use models such as GPT, Llama, or Mistral to generate natural and contextually rel- evant questions without relying on predefined tem- plates (Kargupta et al., 2024; Wang and Zhan, 2024). Unlike rule-based approaches, these systems can sus- tain multi-turn dialogues that adapt to student re- sponses and maintain conversational context. How- ever, they face a central challenge: general-purpose LLMs are optimized to be helpful assistants and may therefore provide direct solutions instead of guid- ing students through Socratic questioning (Kargupta et al., 2024). To address this, more advanced systems imple- ment structured workflows that constrain LLM be- havior toward pedagogically sound interaction. These systems draw on established teaching methods such as the Socratic method and scaffolding theory to shape how questions are generated and sequenced (Alshaikh et al., 2021; Al-Hossami et al., 2023). One notable idea is the trace-and-verify workflow. Tree- Instruct (Kargupta et al., 2024), for example, tracks specific concepts or bugs as binary state variables and uses this state to build dynamic question trees, where sibling questions probe the same misconception from different angles. The main strengths of LLM-based systems are their ability to generate natural-sounding questions, handle diverse code patterns, and adapt pedagogical strategies. Their weaknesses include inconsistent out- puts, higher computational cost, and dependence on the capabilities and reliability of the underlying model (Al-Hossami et al., 2023; Kargupta et al., 2024; Wang and Zhan, 2024). 4.1.3 Hybrid Systems Hybrid systems strategically combine the structure of rule-based approaches with the flexibility of machine- learning components.This architecture acknowl- edges that coherent conversation flow and reliable fact extraction benefit from explicit structure, while tasks such as response evaluation, paraphrasing, and dis- tractor generation benefit from the pattern-recognition capacity of ML (Al-Hossami et al., 2023; Vimalaksha et al., 2021; Wang et al., 2025). ChatDAC (Chuang and Wang, 2025) exemplifies this approach by integrating dynamic assessment into a chatbot. It uses GPT-4 to evaluate student explana- tions against a reference reason and compute a simi- larity score. This score determines the level of scaf- folding provided. A low score triggers broad hints, while a higher score allows more focused guidance. Similarly, Sakshm AI (Gupta et al., 2025) uses a chat- bot named Disha within explicit Socratic guardrails and provides context-aware hints and structured feed- back without revealing direct code solutions. AVERT (Vintila, 2024) further illustrates the value of com- bining deterministic program evidence with conversa- tional verification when authorship and understanding must both be examined. The primary advantage of hybrid systems is that they can balance coverage with control. They reduce the fragility of pure rule-based systems and the unpre- dictability of pure LLM systems, but they also intro- duce higher architectural complexity and additional integration effort (Al-Hossami et al., 2023; Vimalak- sha et al., 2021). RQ1 Summary:Categorization of Ap- proaches • Rule-based or template-driven:These systems rely on deterministic AST analysis and predefined templates to ensure consis- tency and immediate feedback (Santos et al., 2022; Thomas et al., 2019; Stankov et al., 2023). Their main limitation is rigidity and long-term maintenance cost. • LLM-based: These approaches use gener- ative models to facilitate natural, multi-turn Socratic dialogue that adapts to student in- put (Kargupta et al., 2024; Wang and Zhan, 2024; Lin et al., 2025). Their key risks are hallucinations, direct answer-giving, and in- consistent grading behavior. • Hybrid: These frameworks combine de- terministic code analysis with generative dialogue management (Chuang and Wang, 2025; Gupta et al., 2025; Vintila, 2024). They appear most suitable when scalabil- ity, transparency, and pedagogical control are all required. 4.2 Benefits, Limitations, and Pedagogical Implications This section answers RQ2.The integration of chatbot-based assessment systems into programming education offers clear advantages for scalability and personalized learning, but it also introduces technical, behavioral, and pedagogical challenges. 4.2.1 Benefits: Scalability, Engagement, and Self-Improvement The primary benefit of automated conversational as- sessment is the ability to provide immediate and per- sonalized feedback at a scale that human instruc- tors cannot easily match. Empirical studies consis- tently report measurable improvements in student per- formance.For instance, the Socratic Author sys- tem demonstrated a 43% learning gain in program- ming knowledge compared to a control group (Al- shaikh et al., 2021). Similarly, the implementation of ChatDAC resulted in a significant increase in post- test scores, and stronger engagement with tiered hints correlated positively with final exam performance (Chuang and Wang, 2025). Beyond academic performance, these systems also affect student psychology and engagement. Re- search on PythonPal suggests that personalized chat- bot feedback can reduce transactional distance in online learning and foster a stronger sense of en- gagement (Palahan, 2025). Furthermore, the AIval- uate system suggests that such tools can reduce teacher burden in performance-based assessments while maintaining assessment quality in larger co- horts (Yusuf et al., 2025). 4.2.2 Limitations: Reliability, Over-Reliance, and Integrity Despite their potential, conversational assessment tools face major limitations. A central technical chal- lenge is the tendency of LLMs to hallucinate, omit relevant execution details, or produce inconsistent evaluations. Research using the Let’s Ask AI frame- work showed that even strong models such as GPT-4 can still make errors similar to novice programmers, for example by misreading execution paths (Lehtinen et al., 2024). Behavioral challenges are equally important. Stu- dents may game the system or become over-reliant on AI support. An exploratory study by Rahe et al. (Rahe and Maalej, 2025) observed repeated debugging loops in which students excessively prompted the chatbot for fixes rather than trying to understand the underly- ing logic. This over-reliance threatens academic in- tegrity, particularly when students can produce cor- rect code or polished explanations without mastery (Elhambakhsh, 2025). Students have also reported disappointment with generic or repetitive responses from AI tutors when those systems fail to adapt to the actual context of the submission (Frankford et al., 2024). 4.2.3 Pedagogical Implications: The Gap Between Writing and Understanding The most important pedagogical implication in this literature is the visibility of unproductive success, where students produce functionally correct code without understanding how it works. Findings from the Jask system show a strong contrast between per- formance on static code-structure questions and dy- namic execution questions: while students achieved success rates above 80% on static-structure questions, performance dropped below 50% on dynamic execu- tion questions (Santos et al., 2022). This gap is con- cerning because code tracing skill is a strong predictor of overall programming competence (Lehtinen et al., 2023; Stankov et al., 2023). Interaction analyses further suggest that the most effective systems encourage reflection rather than merely providing answers (Khor and Chan, 2025). Taken together, these findings support a shift away from purely functional grading and toward conver- sational validation that verifies conceptual depth and reasoning quality (Vintila, 2024). RQ2 Summary: Benefits, Limitations, and Implications • Benefits: Conversational agents can pro- vide personalized feedback at scale, improve learning gains, and reduce teacher burden in large cohorts (Alshaikh et al., 2021; Yusuf et al., 2025). • Limitations: Technical risks such as hallu- cinations (Lehtinen et al., 2024) are com- pounded by behavioral risks such as gam- ing, over-reliance, and dependence on AI- generated fixes (Rahe and Maalej, 2025). • Pedagogical implications: The literature re- peatedly exposes a gap between producing correct code and understanding it (Santos et al., 2022). This supports shifting assess- ment from purely functional testing toward conversational verification of reasoning. 4.3 Synthesis of Design Requirements The review findings suggest five requirements for a practical conversational layer in APASs:(1) grounding questions in deterministic code facts, (2) supporting multi-turn probing instead of one- shot explanation, (3) discouraging answer copying and LLM-generated explanations through runtime- specific prompts, (4) preserving privacy and trans- parency in data handling, and (5) separating forma- tive support from high-stakes summative use. These requirements motivate the framework proposed in the next section. 5 Framework Proposal Derived from the Review Synthesizing the findings from RQ1 and RQ2, this section proposes a Hybrid Socratic Framework for in- tegrating conversational verification into APASs. The proposal is intended as a design synthesis rather than a validated end product. Its purpose is to balance the conversational fluency of LLMs with the reliability of deterministic code analysis. The framework uses a hybrid architecture in which the conversational layer is constrained by static and dynamic analysis of the submitted code. The So- cratic method serves as the pedagogical foundation because it requires students to articulate intermediate reasoning rather than merely supply an answer. The overall architecture is shown in Figure 3. 5.1 Essential Components The proposed framework consists of five modules op- erating in a closed loop: • Code Analysis Engine: This component uses static analysis and dynamic execution to extract deterministic facts about the student’s submission. Example: If a loop never terminates, the engine can identify the relevant line and the missing state transition, providing ground truth that constrains later prompting. • State Space and Knowledge Tracker: This module maintains a persistent model of the stu- dent’s understanding by tracking knowledge com- ponents and misconceptions across turns. Example: If a student can explain variable scope but fails to explain pointer arithmetic, the tracker prioritizes follow-up questions about pointer be- havior. • Dual-Agent Conversational Core: Splitting the generative task into two specialized agents sepa- rates questioning from evaluation: 1. Instructor Agent: Formulates Socratic ques- tions from the extracted code facts. Example: Look at variable i in the loop body. What value does it have immediately before the final iteration executes? 2. Verifier Agent: Evaluates student responses against a reference reason grounded in the code-analysis output. Example: It compares the student’s explana- tion against the trace-based reference explana- tion and estimates whether mastery has been demonstrated. • Assessment Engine: A hybrid grading layer combines evidence from code functionality and dialogue quality. Example: Code that passes tests but is paired with a weak or inconsistent explanation triggers addi- tional questioning and may lower the final score. • Socratic Guardrails: These constraints prevent the agents from reverting to a solution-giving as- sistant role. Example: If a student asks for the fix directly, the system must redirect the interaction toward a trace, boundary condition, or design rationale in- stead of supplying code. 5.2 Integrity Safeguards Against LLM-Generated Explanations A central challenge is that students may also use LLMs to generate explanations. For this reason, the framework should not rely on generic Why does your code work? questions. Instead, it should bind expla- nation requests to concrete and potentially random- ized execution evidence. 1. Proctored deployment for high-stakes use: In high-stakes settings, the conversational layer should be deployed in supervised labs, controlled viva-style sessions, or remote proctoring contexts. Unproctored use is more suitable for formative practice. 2. Randomized trace questions: The Code Anal- ysis Engine should generate trace questions from runtime states that are specific to the submitted program and, where possible, to randomized in- puts. This makes generic LLM-generated answers less useful because the student must refer to con- crete execution states. 3. Stepwise reasoning tied to execution states: Rather than accepting a single polished explana- tion, the system should require students to reason through successive states, for example by predict- ing the next value of a variable, identifying the last valid array access, or explaining why a loop terminates. 4. Adaptive follow-up questions: When the Veri- fier Agent detects vague or generic language, the Instructor Agent should issue a follow-up that nar- rows the reasoning space, such as changing the in- Static Analysis (AST) Dynamic Execution Unit TestCoverage Code Analysis Engine Dual-Agent LLM Question & Interaction Instructor Agent (Generator) Weak Explanation Verifier Agent (Grader) Update KCs Assessment Engine Score = Functional Test + Dialogue Score Grade & Feedback APAS Grade + LLM Feedback Student Interaction Student & Instructor Dashboard Expected Concept Current Knowledge State State & Progress State Space & Knowledge Tracker JSON Facts System Prompts & Socratic Guardrails No direct solutions Stay on objectiveTiered hints Prompt TemplatesContext Valid Explanation Student Code Submission Figure 3: Hybrid Socratic Framework for chatbot-based assessment in APASs. put, asking for the next trace state, or focusing on a specific branch. Together, these safeguards reduce the usefulness of copy-pasted explanations and shift the burden of proof toward real-time reasoning about the student’s actual code. 5.3 Agent Prompting Strategy To ensure that the LLMs behave within the Socratic guardrails, each agent requires a dedicated system prompt. These prompts are adapted from design pat- terns reported in Sakshm AI, TreeInstruct, and Chat- DAC (Gupta et al., 2025; Kargupta et al., 2024; Chuang and Wang, 2025). Rather than relying on a single generic instruction, the framework separates question generation from response evaluation and constrains both tasks with explicit roles, inputs, and output expectations. 5.3.1 Instructor Agent Prompt This agent receives the code-analysis output, the cur- rent dialogue state, and the target concept to be probed. In practical terms, the prompt defines the agent as a Socratic tutor whose task is to ask short, fo- cused questions tied to concrete program facts, such as a specific variable update, branch condition, loop boundary, or runtime state. It is explicitly instructed not to reveal the fix, not to provide corrected code, and not to confirm correctness too early. Instead, it should move from simpler comprehension checks to- ward deeper reasoning prompts and use the dialogue history to adapt the next question to the student’s last answer. The overall purpose of the prompt is therefore to turn deterministic evidence from the code-analysis layer into guided questioning that triggers explanation rather than solution copying. 5.3.2 Verifier Agent Prompt This agent evaluates student responses against a ref- erence reason grounded in the facts produced by the Code Analysis Engine. Its role is not to define the truth space from scratch, but to compare the stu- dent’s explanation against deterministic program evi- dence and then decide whether additional scaffolding is needed. The prompt therefore frames the agent as a constrained evaluator that checks conceptual align- ment with the expected execution logic, tolerates vari- ation in wording, and returns a structured judgment about whether the answer is sufficient, partially cor- rect, or incorrect. It also determines which miscon- ception or missing step should be targeted next. In this way, the Verifier Agent supports consistent as- sessment while keeping the grading process anchored in the actual behavior of the submitted program rather than in an unconstrained model interpretation. 5.4 Design Principles The framework is governed by three core design prin- ciples. 5.4.1 Two-Tier Assessment Strategy To reduce lucky guessing and superficial interaction, the framework adopts a two-tier approach similar to ChatDAC. • Tier 1 (Selection): The chatbot presents a scenario or code-behavior query generated by the Instruc- tor Agent. • Tier 2 (Explanation): The student must provide a natural-language explanation for the selected an- swer. Scores are calculated by the Verifier Agent using a weighted formula: Score = 20+(Similarity× 0.8) The base score of 20 acknowledges a correct selection in Tier 1, but high scores are only achievable with a valid explanation in Tier 2. 5.4.2 Socratic Guardrails To prevent the LLM from reverting to an assis- tant role and giving direct solutions, strict prompt- ing guardrails are enforced. The system redirects off- topic questions back to the current learning objec- tive and refuses to generate code until the student has demonstrated conceptual understanding through dia- logue. 5.4.3 Scaffolded Decomposition Complex programs are decomposed into smaller logic units. Instead of assessing an entire program in one step, the system uses code analysis to ask targeted Figure 4: Proof of concept, Tier 1: scenario-based multiple- choice question generated by the Instructor Agent. questions about specific branches, states, or logic blocks, so that the student must demonstrate under- standing component by component. 5.5 Proof of Concept Implementation and Current Limitations To validate the technical feasibility of the proposed framework, a functional prototype was developed in a Python-based stack. Source code and a demo video are available on Zenodo. 2 The prototype uses Streamlit for the interac- tive student interface and session state manage- ment.While the initial version used Gemini 2.0 Flash, the updated prototype is model-agnostic. It supports multiple backends, including hosted infer- ence APIs and local transformer models through llama-cpp-python, which improves flexibility and enables privacy-sensitive deployment.A fallback mode allows operation without active API keys by us- ing pre-generated datasets. The implementation enforces the dual-agent ar- chitecture illustrated in Figures 4 and 5: 1. Instructor Agent: Receives raw C code input and generates scenario-based multiple-choice ques- tions targeting specific conceptual elements such as pointer management. 2. Verifier Agent: Acts as a semantic grader by comparing the student’s natural-language expla- nation against a hidden reference reason gener- ated from the same grounded context. The current prototype nevertheless has important limitations. It demonstrates the end-to-end conversa- tional flow, but it does not yet constitute a production- ready APAS component. In particular, deterministic unit-testing and regression-testing hooks are not yet integrated into a complete assessment pipeline, lan- guage support is limited, no classroom deployment 2 https://zenodo.org/records/18335209 Figure 5: Proof of concept, Tier 2: open-ended explanation graded by the Verifier Agent. study has been conducted, and no inter-rater reliabil- ity study has compared automated judgments with hu- man examiners. In addition, latency, operational cost, model drift, and adversarial prompt resistance have not yet been evaluated systematically. For these rea- sons, the prototype should currently be interpreted as a feasibility demonstration rather than validated evi- dence of assessment reliability. 6 Discussion The review and the framework proposal together in- dicate that APASs are moving from a primary fo- cus on functional outputs toward stronger verification of learners’ reasoning. Dialogue-based verification increases the effort required to submit AI-generated code without understanding, particularly when ques- tions are tied to execution traces and code-specific states. A second implication is the growing importance of cognitive depth in automated questioning. Rather than posing generic prompts, effective conversational assessors align questions with explicit comprehen- sion targets such as code tracing, control-flow rea- soning, or explanation of design tradeoffs. This sug- gests that future APASs should map chatbot turns to well-defined comprehension targets rather than treat- ing conversation as an unstructured add-on. A third implication is the persistent tradeoff be- tween validity and scalability. LLM-based systems scale and support natural multi-turn interaction but introduce reliability risks through hallucinations and inconsistent outputs, while rule-based systems offer stronger auditability but weaker coverage and adapt- ability. The literature therefore points toward hybrid architectures that ground conversational assessment in deterministic program analysis. 6.1 Technology Acceptance and Classroom Integration Technology acceptance is likely to determine whether such systems are used successfully in practice. Stu- dents need to understand whether the chatbot is acting as a tutor, an examiner, or both. Instructors need clear controls over prompt policies, scoring thresholds, ap- peals, and when the conversational layer is formative versus summative. Transparent communication about what is being assessed and how explanations are eval- uated is therefore essential for trust and adoption. 6.2 Ethics, Fairness, and Accountability Ethical concerns arise when conversational systems influence grades. Students must be protected against opaque grading logic, biased language judgments, and inconsistent treatment across language back- grounds or communication styles. A grounded hy- brid architecture helps by tying evaluation to program facts, but this does not remove the need for human oversight, appeal procedures, and ongoing validation against instructor judgments. 6.3 Digital Sustainability Digital sustainability also deserves attention. Vendor- dependent cloud deployments may be difficult to maintain over time because of changing APIs, pric- ing, and model availability. Local-model options im- prove institutional control and privacy, but they intro- duce hardware, maintenance, and energy costs. Sus- tainable adoption therefore depends not only on peda- gogical effectiveness but also on maintainable infras- tructure, reproducible prompt and model versioning, and realistic operating costs. 7 Threats to Validity While this study followed a systematic methodology, several risks to validity remain. 7.1 Technological Volatility A significant challenge in reviewing LLM-based tools is the pace of technological change. Several studies in the corpus rely on models or frameworks that were al- ready being replaced during the review period. Some reported limitations may therefore reflect the matu- rity of specific models rather than fundamental limits of conversational assessment as a paradigm. 7.2 Framework Design Validity The proposed Hybrid Socratic Framework is derived from the review, but it has not yet been validated through a full classroom study. As a result, design choices such as similarity thresholds, scaffolding se- quences, or the weighting of explanation quality re- main provisional. The framework should therefore be interpreted as a reasoned synthesis, not as a validated assessment standard. 8 Conclusion and Future Work Large Language Models make purely functional pro- gramming assessment increasingly insufficient be- cause students can now generate correct code with- out necessarily understanding it. This paper therefore addressed conversational assessment in two stages. First, it reviewed the emerging literature on con- versational agents in programming assessment and grouped the identified approaches into rule-based or template-driven, LLM-based, and hybrid systems. The review indicates that conversational agents can provide scalable and personalized probing of code understanding, but it also highlights substantial risks related to hallucinations, over-reliance, privacy, fair- ness, and academic integrity. Second, based on these findings, the paper pro- posed a Hybrid Socratic Framework that combines deterministic code analysis with a constrained conver- sational layer. The framework is intended to support code-understanding verification by grounding ques- tions in program facts, requiring stepwise reasoning, and separating question generation from explanation evaluation. Overall, the literature and the proposed frame- work suggest that the future of programming assess- ment will likely involve conversational verification as a complement to conventional testing rather than as a replacement for it. The most promising path is not to ban AI outright, but to use carefully constrained con- versational agents to examine whether students can reason about the code they submit. 8.1 Future Work Several directions remain open.First, future work should validate the framework empirically through classroom deployments and comparisons with instructor-led oral checks.Second, stronger knowledge-tracing mechanisms could help the sys- tem maintain a more stable model of student un- derstanding across multiple sessions. Third, multi- modal extensions may eventually allow students to explain code through traces, diagrams, or memory- state sketches rather than text alone.Finally, fu- ture studies should investigate when conversational assessment is appropriate for formative use, when it can support summative decisions, and how deploy- ment choices affect privacy, fairness, and student be- havior. REFERENCES Al-Hossami, E., Bunescu, R., Teehan, R., Powell, L., Maha- jan, K., and Dorodchi, M. (2023). Socratic question- ing of novice debuggers: A benchmark dataset and preliminary evaluations. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Ed- ucational Applications (BEA 2023), pages 709–726, Toronto, Canada. Association for Computational Lin- guistics. Alshaikh, Z., Tamang, L., and Rus, V. (2021). Experiments with auto-generated socratic dialogue for source code understanding. In Proceedings of the 13th Interna- tional Conference on Computer Supported Education (CSEDU), pages 35–44. SCITEPRESS – Science and Technology Publications. Canic ̧o, A. B. and Santos, A. L. (2025). Integrating ques- tions about learners’ code in an automated assess- ment system. In 6th International Computer Program- ming Education Conference (ICPEC 2025), volume 133 of Open Access Series in Informatics (OASIcs), pages 5:1–5:14, Dagstuhl, Germany. Schloss Dagstuhl – Leibniz-Zentrum f”ur Informatik. Cheng, G., Wong, W., Luo, L., and Yu, M. (2025). Inte- grating a scaffolding-based, LLM-driven chatbot into programming education: A university case study. In Proceedings of the 2025 International Symposium on Educational Technology (ISET 2025), pages 196–200. IEEE. Chuang, Y.-T. and Wang, H.-T. (2025). A ChatGPT-based dynamic assessment chatbot. Journal of Computer Languages, 85:101366. Debets, T., Banihashem, S. K., Joosten-Ten Brinke, D., Vos, T. E. J., Maillette de Buy Wenniger, G., and Camp, G. (2025). Chatbots in education: A systematic review of objectives, underlying technology and theory, eval- uation criteria, and impacts. Computers & Education, 234:105323. Elhambakhsh, S. E. (2025). Evaluating ChatGPT-3’s ef- ficacy in solving coding tasks: implications for aca- demic integrity in english language assessments. Lan- guage Testing in Asia, 15(1):37. Frankford, E., Sauerwein, C., Bassner, P., Krusche, S., and Breu, R. (2024). AI-tutoring in software engineering education. In Proceedings of the 46th International Conference on Software Engineering: Software En- gineering Education and Training (ICSE-SEET ’24), pages 309–319, New York, NY, USA. Association for Computing Machinery. Gupta, R., Goyal, H., Kumar, D., Mehra, A., Sharma, S., Mittal, K., and Challa, J. S. (2025).Sakshm AI: Advancing AI-assisted coding education for en- gineering students in india through socratic tutor- ing and comprehensive feedback.arXiv preprint arXiv:2503.12479. Kargupta, P., Agarwal, I., Tur, D. H., and Han, J. (2024). Instruct, not assist: LLM-based multi-turn planning and hierarchical questioning for socratic code debug- ging. In Findings of the Association for Computa- tional Linguistics: EMNLP 2024, pages 9475–9495, Miami, Florida, USA. Association for Computational Linguistics. Khor, E. T. and Chan, L. (2025). Exploring the effect of scaffolding strategies in GenAI chatbot on student en- gagement and programming skill development. In GCCCE 2025 English Conference Proceedings, pages 32–39. Global Chinese Society for Computers in Ed- ucation. Kitchenham, B. A. (2012). Systematic review in software engineering: Where we are and where we should be going. In Proceedings of the 2nd International Work- shop on Evidential Assessment of Software Technolo- gies, EAST ’12, pages 1–2, New York, NY, USA. As- sociation for Computing Machinery. Lehtinen, T., Haaranen, L., and Leinonen, J. (2023). Auto- mated questionnaires about students’ JavaScript pro- grams: Towards gauging novice programming pro- cesses.In Proceedings of the 25th Australasian Computing Education Conference, pages 49–58, Mel- bourne, VIC, Australia. Association for Computing Machinery. Lehtinen, T., Koutcheme, C., and Hellas, A. (2024). Let’s ask AI about their programs: Exploring ChatGPT’s answers to program comprehension questions. In Pro- ceedings of the 46th International Conference on Soft- ware Engineering: Software Engineering Education and Training (ICSE-SEET ’24), pages 221–232. As- sociation for Computing Machinery. Lin, Y., Ferdous Khan, M. F., and Sakamura, K. (2025). Athena: A GenAI-powered programming tutor based on open-source LLM.In 2025 1st International Conference on Consumer Technology (ICCT-Pacific), pages 1–4. IEEE. Manorat, P., Tuarob, S., and Pongpaichet, S. (2025). Artifi- cial intelligence in computer programming education: A systematic literature review. Computers and Edu- cation: Artificial Intelligence, 8:100403. Palahan, S. (2025). PythonPal: Enhancing online program- ming education through chatbot-driven personalized feedback. IEEE Transactions on Learning Technolo- gies, 18:335–350. Rahe, C. and Maalej, W. (2025). How do programming students use generative AI? Proceedings of the ACM on Software Engineering, 2(FSE):978–1000. Santos, A., Soares, T., Garrido, N., and Lehtinen, T. (2022). Jask: Generation of questions about learners’ code in Java. In Proceedings of the 27th ACM Conference on Innovation and Technology in Computer Science Ed- ucation Vol. 1, pages 117–123, Dublin, Ireland. Asso- ciation for Computing Machinery. Stankov, E., Jovanov, M., and Madevska Bogdanova, A. (2023). Smart generation of code tracing questions for assessment in introductory programming. Computer Applications in Engineering Education, 31(1):5–25. Thomas, A., Stopera, T., Frank-Bolton, P., and Simha, R. (2019). Stochastic tree-based generation of program- tracing practice questions. In Proceedings of the 50th ACM Technical Symposium on Computer Science Ed- ucation, pages 91–97, Minneapolis, MN, USA. Asso- ciation for Computing Machinery. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neu- ral Information Processing Systems 30 (NIPS 2017), pages 5998–6008. Vimalaksha, A., Prekash, A., Kumar, V., and Srinivasa, G. (2021). DiGen: Distractor generator for multi- ple choice questions in code comprehension. In 2021 IEEE International Conference on Engineering, Tech- nology & Education (TALE), pages 1073–1078. IEEE. Vintila, F. (2024). AVERT (Authorship Verification and Evaluation Through Responsive Testing): an LLM- based procedure that interactively verifies code au- thorship and evaluates student understanding.In 2024 21st International Conference on Information Technology Based Higher Education and Training (ITHET), pages 1–7. IEEE. Wang, J., Dai, Y., Zhang, Y., Ma, Z., Li, W., and Chai, J. (2025). Training turn-by-turn verifiers for dialogue tu- toring agents: The curious case of LLMs as your cod- ing tutors. In Findings of the Association for Compu- tational Linguistics: ACL 2025, pages 12416–12436, Vienna, Austria. Association for Computational Lin- guistics. Wang, L. and Zhan, S. (2024). How can generative AI benefit educators in designing assessments in com- puter science? Education Research and Perspectives, 51:82–101. Wohlin, C. (2014). Guidelines for snowballing in system- atic literature studies and a replication in software en- gineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, pages 1–10, New York, NY, USA. Asso- ciation for Computing Machinery. Yusuf, H., Money, A., and Daylamani-Zad, D. (2025). Towards reducing teacher burden in performance- based assessments using aivaluate: an emotionally in- telligent LLM-augmented pedagogical AI conversa- tional agent. Education and Information Technolo- gies, 30:24649–24693.