Paper deep dive
Is Solving Better Than Evaluating GenAI Solutions?
Ethan Dickey, Marios Mertzanidis, Alexandros Psomas
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/1/2026, 1:49:02 AM
Summary
This study investigates the impact of evaluating Generative AI (GenAI)-generated solutions versus traditional problem solving on student learning in a junior-level algorithms course. Using a randomized A/B crossover design with 220 students, the research found that while evaluating GenAI solutions led to higher homework scores, it did not result in statistically significant improvements in midterm, final exam, or overall course grades. Survey data indicated that most students did not change their study habits, though those who adapted strategies found the evaluation tasks helpful. The findings suggest that GenAI evaluation redistributes effort toward verification and diagnosis but requires deliberate scaffolding to produce meaningful conceptual transfer.
Entities (8)
Relation Signals (5)
Purdue University â conductedstudyin â Introduction to the Analysis of Algorithms
confidence 95% · This study was conducted during the Spring 2025 semester at Purdue University... in a junior-level 'Introduction to the Analysis of Algorithms' course.
GenAI Evaluation â showsnosignificantdifferencein â Final Exam Scores
confidence 95% · We found no statistically significant differences between groups in midterm scores, final exam scores...
GenAI Evaluation â yieldshigherscorein â Homework Scores
confidence 92% · Students received significantly higher homework scores when evaluating GenAI-generated solutions
GenAI Evaluation â doesnotautomaticallyproduce â Stronger Conceptual Transfer
confidence 90% · does not automatically produce stronger conceptual transfer.
GenAI Evaluation â redistributeseffortto â Verification, Diagnosis, and Judgment
confidence 85% · GenAI evaluation redistributes student effort from open-ended solution construction toward verification, diagnosis, and judgment
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As Generative AI (GenAI) tools become increasingly capable of generating solutions to computing assignments, the computing education community is exploring pedagogical approaches that emphasize solution evaluation, verification, and critique alongside traditional solution generation. However, evidence regarding the impact of such evaluation-centered tasks on student learning remains limited, particularly in upper-division, theory-heavy courses. We conducted a randomized A/B crossover study (N=220) in a junior-level algorithms course to compare evaluating GenAI-generated solutions with traditional problem solving. Across six assignments, student working groups either solved challenging algorithmic problems directly or evaluated often-flawed GenAI-generated solutions, with roles reversed midway through the semester. We found no statistically significant differences between groups in midterm scores, final exam scores, overall course grades, or exam problems structurally aligned with the homework interventions. Students received significantly higher homework scores when evaluating GenAI-generated solutions, but this localized advantage did not translate into downstream summative gains. Survey data further indicated that most students reported no change in study habits in response to the intervention; however, those who reported adapting their study strategies rated the GenAI-evaluation assignments as significantly more helpful. These findings suggest that GenAI evaluation redistributes student effort from open-ended solution construction toward verification, diagnosis, and judgment, but does not automatically produce stronger conceptual transfer. We conclude that GenAI-evaluation activities can be incorporated into algorithms coursework without broad performance losses, but meaningful learning gains may require deliberate scaffolding that pushes students beyond simple error diagnosis.
Tags
Links
- Source: https://arxiv.org/abs/2607.27586v1
- Canonical: https://arxiv.org/abs/2607.27586v1
Trouble viewing inline? Open PDF directly â
Full Text
83,923 characters extracted from source content.
Expand or collapse full text
Is Solving Better Than Evaluating GenAI Solutions? Ethan Dickey Purdue UniversityDepartment of Computer ScienceWest LafayetteIndianaUSA47907 dickeye@purdue.edu 0009-0007-3706-5253 , Marios Mertzanidis Purdue UniversityDepartment of Computer ScienceWest LafayetteIndianaUSA47907 mmertzan@purdue.edu and Alexandros Psomas Purdue UniversityDepartment of Computer ScienceWest LafayetteIndianaUSA47907 apsomas@purdue.edu Abstract. As Generative AI (GenAI) tools become increasingly capable of generating solutions to computing assignments, the computing education community is exploring pedagogical approaches that emphasize solution evaluation, verification, and critique alongside traditional solution generation. However, empirical evidence regarding the impact of such evaluation-centered tasks on student learning remains limited, particularly in upper-division, theory-heavy courses. We conducted a randomized A/B crossover study (N=220N=220) in a junior-level algorithms course to investigate how evaluating GenAI-generated solutions compares with traditional problem solving. Across six assignments, student working groups either solved challenging algorithmic problems directly or evaluated often-flawed GenAI-generated solutions to corresponding problems, with group roles reversed midway through the semester. We found no statistically significant differences between groups on midterm scores, final exam scores, overall course grades, or exam problems structurally aligned with the homework interventions. Students received significantly higher homework scores during periods in which they evaluated GenAI-generated solutions, but this localized advantage on the modified homework items did not translate into downstream summative gains. Survey data further indicated that most students did not report changing their study habits in response to the intervention; however, students who did report adapting their study strategies rated the GenAI-evaluation assignments as significantly more helpful. These findings suggest that GenAI evaluation redistributes student effort from open-ended solution construction toward verification, diagnosis, and judgment, but does not automatically produce stronger conceptual transfer. We conclude that GenAI-evaluation activities can be incorporated into algorithms coursework without broad performance losses, but that meaningful learning gains may require deliberate scaffolding that pushes students beyond simple error diagnosis. Generative AI, large language models, computing education, algorithms, experimental design â ccs: Computing methodologies Artificial intelligenceâ ccs: Social and professional topics Computing education 1. Introduction In recent years, advances in Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs) have fundamentally reshaped computing education. Modern LLMs are capable of generating syntactically correct code and solving many introductory programming and algorithms problems (Denny et al., 2024b; Finnie-Ansley et al., 2022, 2023; Reeves et al., 2023; Savelka et al., 2023a). As these tools become increasingly accessible, students are no longer merely learning about artificial intelligence; they are actively incorporating AI systems such as ChatGPT and GitHub Copilot into their day-to-day programming workflows (Bird et al., 2022). This rapid adoption has sparked widespread discussion regarding how computing curricula, pedagogical practices, and assessment methods should adapt in response to these technologies (Prather et al., 2023a; Denny et al., 2023a; Qiao et al., 2026). The emergence of GenAI presents both substantial opportunities and significant risks for computing education. Prior work has shown that LLMs can provide detailed code explanations, help debug errors, deliver personalized assistance, and offer interactive support that may improve student engagement and productivity (Leinonen et al., 2023a; MacNeil et al., 2023; Leinonen et al., 2023b; Bird et al., 2022; Kazemitabaar et al., 2023; Prather et al., 2023b). However, researchers have raised concerns that students may become overly reliant on AI-generated solutions, develop only a superficial understanding of programming concepts, or acquire an inflated perception of their own competence (Becker et al., 2023; Finnie-Ansley et al., 2022; Leinonen et al., 2023a; Li et al., 2022; Prather et al., 2018, 2023b; Denny et al., 2024a; Vaithilingam et al., 2022). Recent studies further suggest that struggling students may be particularly vulnerable to these effects, often relying on GenAI tools in ways that bypass deeper learning processes (Margulieux et al., 2024; Prather et al., 2024). These concerns are especially relevant in algorithmic problem solving, where conceptual understanding, abstraction, and design reasoning play a central role. Despite the growing literature on GenAI in computing education, much of the existing work focuses on evaluating the capabilities of LLMs themselves rather than measuring their impact on student learning. A recent large-scale review by Prather et al. found that the majority of published studies investigate code generation performance, AI-assisted programming, or student perceptions, while comparatively little work provides controlled empirical evidence regarding learning outcomes (Prather et al., 2023a). Similarly, a recent systematic review by Liu et al. (Liu et al., 2025) in algorithms education notes that rigorous controlled studies remain relatively rare in computing education research overall; the authors argue that many open questions remain regarding how algorithms should be taught effectively and equitably, particularly in contexts that require deeper conceptual understanding and problem-solving skills. At the same time, the role of students in programming tasks may itself be changing. As GenAI systems become more capable of producing complete solutions, the educational emphasis may increasingly shift from code production toward code comprehension, evaluation, and critique (Denny et al., 2024c; Nam et al., 2024; Prather et al., 2023a). Recent work argues that understanding, validating, and reasoning about AI-generated code are becoming essential skills for modern programmers (Qiao et al., 2026). This shift is particularly important given that modern LLM-based tools, which are freely and openly accessible to students, are already widely used for assignment assistance, and can often generate solutions to introductory programming and algorithms problems with minimal prompting (Bird et al., 2022; Finnie-Ansley et al., 2022, 2023; Reeves et al., 2023; Savelka et al., 2023a). Compounding this challenge, prior work has noted that AI-generated solutions are difficult to reliably detect using existing plagiarism or AI-detection tools, whose effectiveness remains limited and prone to false positives (Orenstrakh et al., 2024). Consequently, if the goal is to support genuine learning rather than merely restrict tool use, many researchers argue that simply prohibiting AI usage may be neither practical nor pedagogically effective, and that computing education instead needs instructional approaches that explicitly incorporate and critically engage with AI-generated content (Bellettini et al., 2023; Denny et al., 2023b; Poldrack et al., 2023; Savelka et al., 2023b; Prather et al., 2023a). This raises an important pedagogical question: rather than asking students only to produce solutions themselves, can structured interaction with AI-generated solutions improve conceptual understanding and learning outcomes? We investigate the impact of a course-integrated GenAI activity in a junior-level algorithms course. Across a sequence of homework assignments, students participated in one of two instructional conditions: they either (a) solved a challenging algorithmic problem in the traditional manner, or (b) generated, evaluated, and graded an LLM-produced solution to the same type of problem. To study the effects of these activities, we employed a randomized A/B crossover design at the homework working-group level, allowing all groups to experience both instructional conditions at different stages of the semester through a mid-semester treatment switch. We evaluate the impact of this intervention using several complementary outcome measures, including overall performance on summative assessments, performance on exam questions specifically designed to test conceptual insights emphasized in the GenAI-based assignments, and studentsâ self-reported perceptions and study strategy adaptations throughout the course. Our study contributes empirical evidence to the growing discussion surrounding GenAI in computing education. Rather than evaluating the raw capabilities of LLMs, we examine how a specific AI-mediated pedagogical intervention influences student learning outcomes in an algorithms context. More broadly, our findings advance ongoing efforts to rethink assessment and instructional design in the era of Generative AI. 2. Related Work 2.1. Generative AI and the Changing Validity of Traditional Computing Assessments Community syntheses argue that the rapid improvement of LLMs challenges the validity of most out-of-class assessments, since high quality solutions and explanations can be generated with minimal effort (Prather et al., 2023a, 2025). Empirical studies have documented this over a large number of languages and assessment formats. For example, LLMs are capable of solving a large portion of introductory programming tasks (and, in some cases, entire programming course assessment suites) (Savelka et al., 2023a; Finnie-Ansley et al., 2022). Evaluations in undergraduate programming courses in different languages report that LLMs can generate plausible solutions quickly, motivating a redesign of assessments, learning tasks, and policy (Ouh et al., 2023). In domains that require structured reasoning, LLMs also show increasing competence. For example, Poulsen et al. (Poulsen et al., 2024) showed that newer models perform substantially better on proof-related tasks, although important limitations persist. Collectively, these studies suggest that assessment redesign is a pedagogical necessity, rather than merely an academic-integrity response. 2.2. Pedagogical Responses: Verification, Critique, and Responsible GenAI Use A growing number of classroom interventions have responded by foregrounding skills that are more durable under ubiquitous code generation: understanding, testing, debugging, decomposition, and critique (Vadaparty et al., 2024; Donekal Chandrashekar et al., 2026). This emphasis on structured integration also appears in course-level frameworks that introduce GenAI through explicit norms, guided activities, and reflection on appropriate use, rather than treating LLMs as an unsupervised add-on to existing programming coursework (Dickey et al., 2024). Similarly, tool-based interventions attempt to cultivate skepticism by confronting students with mixed- or low-quality AI suggestions and requiring them to identify errors (thus âoperationalizingâ critical use as an assessable learning objective) (MacNeil et al., 2025). Complementary observational studies, e.g., Smith et al. (Smith et al., 2025) and Prather et al. (Prather et al., 2024), highlight why scaffolding matters: when students encounter LLM errors, they vary widely in their ability to detect missteps, calibrate trust, and decide on their next actions. Together, these findings motivate tasks in which evaluation is part of the design: the learning objective is not merely to produce an answer, but to develop justified confidence through verification and explanation (Prather et al., 2025; Tankelevitch et al., 2024). 2.3. Metacognition and Self-Regulated Learning in GenAI Evaluation Tasks Metacognition and self-regulated learning provide a principled lens for why evaluating a GenAI solution might support learning: it can require learners to monitor understanding, detect gaps, and regulate strategy (e.g., rederiving, testing, and seeking counterexamples) (Loksa et al., 2022). Formative feedback theory argues that learners benefit when feedback processes help them generate internal feedback and act on it, rather than passively receiving correctness signals (Nicol and MacfarlaneâDick, 2006). LLM-mediated student workflows increase these demands: human-AI interaction research characterizes GenAI use as requiring planning, evaluation, trust calibration, and strategy control (Tankelevitch et al., 2024). Empirical computer science education studies, e.g., Prather et al. (Prather et al., 2024) and Margulieux et al. (Margulieux et al., 2024), suggest these regulatory skills mediate whether GenAI helps or harms: some learners accelerate their learning by using GenAI strategically while others struggle, accept incorrect suggestions, or experience an âillusion of confidence.â Altogether, this evidence motivates interventions that explicitly scaffold monitoring and evaluation (e.g., justifying why an answer is correct, asking for tests or invariants, or requiring reflection on when the AI was reliable) rather than assuming these behaviors will emerge spontaneously (Donekal Chandrashekar et al., 2026; MacNeil et al., 2025). 2.4. Learning from Solutions: Worked Examples, Erroneous Examples, and Self-Explanation Evaluation-centered GenAI tasks can be grounded in the worked-example literature. A review by Muldner et al. (Muldner et al., 2022) focused on programming activities concludes that worked examples can support learning, but outcomes strongly depend on how students engage with the example (tracing/predicting/explaining/debugging), and on alignment to learnersâ prior knowledge. This connection is reinforced by self-explanation research, which shows that learners who generate principled explanations while studying a worked solution develop more transferable understanding (Chi et al., 1989). Experimental work in learning science, e.g., Durkin and Rittle-Johnson (Durkin and Rittle-Johnson, 2012) and Booth et al. (Booth et al., 2013), has found that studying incorrect examples (usually in contrast with correct ones) can improve conceptual understanding, especially when paired with explanation prompts that direct attention to critical features. These results provide a more theoretical basis for pairing an LLM-solution evaluation with structured prompts (e.g., âstate the claim, identify first invalid step, justify why it fails, and propose a corrected invariant/argumentâ) rather than treating âspot the bugâ as a purely unguided activity (Chi et al., 1989; Booth et al., 2013). 2.5. Learning by Reviewing: Peer Assessment and Pedagogical Code Review Evaluating othersâ work is a well-established pedagogical structure in computing education, most prominently through peer assessment and pedagogical code review. A systematic review of peer code review in higher education reported common benefits (exposure to alternative solutions, increased attention to quality criteria) and recurring difficulties such as novice reviewersâ limited expertise, reliability concerns, and the need for rubrics and scaffolding (Indriasari et al., 2020). Empirical studies of Pedagogical Code Review (PCR) environments by Hundhausen et al. (Hundhausen et al., 2010, 2013) found that the design of the review activity itself and the review interface shaped whether or not students engaged in substantive critique. Related work by Reily et al. (Reily et al., 2009) also suggests that aggregating multiple peer reviews can improve accuracy, and that participating in the review process can yield learning benefits for reviewers themselves, though outcomes depend on implementation details. For GenAI evaluation tasks, this body of work suggests that grading an AI solution should not be framed as a trivial addition; it is a form of review activity that requires (1) explicit criteria, (2) accountability for studentsâ justification, and (3) carefully calibrated assignment difficulty so novices can succeed at meaningful diagnosis. 2.6. LLM-Generated Instructional Artifacts and Reproducibility Challenges LLMs are increasingly studied not only as solution generators, but also as feedback providers and evaluators. Systematic evidence on automated grading and feedback tools highlights both the potential value of scalable feedback and persistent concerns about reproducibility and evaluation datasets (Messer et al., 2024). Recent work by Smith and Zilles (Smith IV and Zilles, 2024) has explored LLM-mediated grading of free response explanations (e.g., evaluating âexplain in plain Englishâ answers) and reported moderate agreement patterns with human graders and systematic disagreement patterns that are important for instructional validity. Other classroom tools use LLMs to generate learning content (such as worked examples) and evaluated those experiences via expert review and student studies, e.g., Jury et al. (Jury et al., 2024). Related platform work by Sinha et al. (Sinha et al., 2024) has also explored instructor-facing uses of GenAI, such as forum support systems in which model-generated responses are reviewed and revised before reaching students. This highlights a complementary design space in which AI is mediated through instructional workflows rather than used directly by students without oversight. However, all of these approaches struggle with LLM variability: outputs can differ with prompts, context, and model version. These differences can threaten fairness and replicability if students receive (meaningfully) different artifacts to evaluate (Prather et al., 2025; MacNeil et al., 2025). Practical interventions therefore increasingly recommend explicit controls over factors such as prompts, âchat freshness,â artifact reporting, and instructor-generated canonical outputs, as well as explicit instruction on verification approaches (Donekal Chandrashekar et al., 2026; MacNeil et al., 2025). 2.7. Gaps in Existing Evidence for Post-Introductory Computing Courses Recent synthesis studies, e.g., Bouvier et al. (Bouvier et al., 2026) and Prather et al. (Prather et al., 2025), explicitly call out the relative scarcity of evidence in post-introductory courses where abstraction and proof-like reasoning play a bigger role. In these settings, the question is usually not âCan the students write code?â but rather, âCan the students justify correctness, reason about tradeoffs, and transfer abstract paradigms?â Evidence that LLMs can increasingly handle structured reasoning tasks (with limitations) strengthens the case that upper-division courses must develop evaluation-centered learning objectives rather than relying on traditional creation-centric tasks alone (Poulsen et al., 2024; Prather et al., 2023a). Our study responds to this gap by examining an evaluation-centered intervention in a junior-level algorithms course using a randomized crossover design and outcomes tied to summative exam performance and structurally aligned reasoning tasks. 3. Methods This study employed a randomized A/B crossover experimental design to examine the relationship between grading AI-generated solutions and studentsâ ability to internalize difficult, abstract concepts in a junior-level âIntroduction to the Analysis of Algorithmsâ course. Specifically, for each homework assignment, approximately half of the students were asked to solve a problem independently, while the remaining students were tasked with evaluating a GenAI-generated solution to the same problem. Student outcomes under these two instructional approaches were then compared using performance measures from the midterm and final examinations, as well as studentsâ self-reported perceptions of the two types of exercises. The course covers classical algorithm design paradigms, including divide-and-conquer, dynamic programming, greedy algorithms, maximum flow, and reductions, and is traditionally most challenging during the post-midterm portion, when students are required to engage in increasingly complex abstract reasoning. The study was conducted during the Spring 2025 semester at Purdue University. 3.1. Research Questions This study investigates how engaging with AI-generated solutions as a grading task, compared to traditional problem solving, relates to studentsâ learning outcomes and perceptions in a junior-level algorithms course. Specifically, we address the following research questions: Research Question 1. How does evaluating GenAI-generated solutions, relative to directly solving problems, relate to studentsâ performance on summative assessments (midterm and final examinations)? Research Question 2. To what extent does participation in GenAI-based grading activities relate to studentsâ ability to internalize and transfer abstract algorithmic reasoning, as reflected in performance on exam problems structurally aligned with prior homework exercises? Research Question 3. How do students perceive the helpfulness of grading GenAI-generated solutions for learning algorithmic concepts, compared to completing problems independently? Research Question 4. Does participation in GenAI-based grading activities relate to self-reported changes in studentsâ study behaviors and preparation strategies for examinations? 3.2. Participants and Group Formation A total of 227 undergraduate students were enrolled in the course. After excluding seven students who audited the course, received incompletes, or did not complete the final examination, the final analytic sample consisted of 220 students. At the beginning of the semester, students were informed that they would be divided into two groups, each receiving a slightly different set of homework assignments. To support existing collaboration practices and reduce cross-group contamination, students were asked to self-report their regular working groups (up to six peers), with the assurance that members of the same working group would be assigned to the same set of homework questions. This approach was intended to ensure that students who routinely collaborated remained together, thereby minimizing interactions across conditions (e.g., in cases where one group was tasked with solving a problem while the other evaluated AI-generated solutions for the same problem). While collaboration within reported working groups was permitted under the course collaboration policy, students submitted homework individually and were expected to write up their own solutions. Once working groups were identified, they were randomly assigned to one of the two experimental conditions. The resulting groups were labeled Monte Carlo and Las Vegas, referencing two classes of randomized algorithms. These labels were intentionally chosen to avoid any implied ordinal or performance-based hierarchy between conditions, thereby reducing the risk of expectancy effects or motivational bias. 3.3. Ethics and Consent The study protocol was reviewed and approved by the Purdue University Institutional Review Board (IRB-2025-39, approved January 21, 2025). Survey participation was voluntary, and survey responses were collected anonymously. Course performance data were analyzed after de-identification. 3.4. Experimental Design We employed an A/B crossover design to compare learning outcomes associated with solving problems versus evaluating GenAI-generated solutions. The course included six homework assignments, one midterm examination, and one final examination. Homework assignments were released biweekly, each with a two-week submission window, and each focused on a distinct course topic: (1) mathematical induction, (2) divide-and-conquer algorithms, (3) dynamic programming, (4) graph algorithms, (5) max-flow/min-cut, and (6) reductions. Each homework assignment consisted of four exercises of increasing difficulty. The first three exercises were traditional problem-solving tasks in which students were required to produce complete solutions; these exercises were identical across both experimental conditions and were each worth 30 points. The fourth exercise was deliberately designed to be substantially more challenging, such that contemporary GenAI systems (e.g., ChatGPT) could not reliably produce correct solutions: before release, we tested each prompt in multiple independent ChatGPT-4o sessions and selected problems for which generated solutions were consistently incomplete or incorrect. This exercise was worth 10 points for students in both conditions. The number of points was chosen so that no group gained a significant advantage over the other. During the first half of the semester (Homework 1â3), students assigned to the Monte Carlo condition were required to solve the fourth exercise directly. This typically involved designing an algorithm, formally proving its correctness, and analyzing its runtime. In contrast, students in the Las Vegas condition were provided with a standardized prompt to submit to ChatGPT and were then tasked with evaluating the generated solution, including identifying and explaining any errors or omissions. Below, we present an example of a homework problem as assigned to the Monte Carlo group, along with the corresponding instructions provided to the Las Vegas group. For clarity in the analyses that follow, the Las Vegas condition is reported as Group A and the Monte Carlo condition as Group B. Monte Carlo Assignment. You are trying to build an amusement park ride out of spare parts that you found at the University Surplus Store. Each part has a certain shape that we represent as a sequence of pluses and minuses, as in: â+âŁ+âŁââŁââŁââŁââŁââŁ+++-----+,â representing a part of length 8, that climbs up 2 feet, drops down 5 feet, then climbs 1 foot (ending 2 feet below where it started). Formally, you are given as input an array of n strings. Each string is of length at most â , and it consists only of +,â\+,-\ symbols. You want to find the length of the longest ride you can make using (a subset of) these parts, where a ride must start on the ground, must end on the ground, and must never go below ground. For example, if you are given [â+++,â+++,++,+,â][--+++,--+++,++,+,---], then the longest ride you can make is of length 10 and starts with the 3rd part, then the 2nd part, then the 5th part, â+âŁ+âŁââŁââŁ+âŁ+âŁ+âŁââŁââŁâ++--+++---.â Devise a polynomial time algorithm, which given as input the array of parts, returns the length of the longest ride. Prove the correctness and runtime of your algorithm using clear and concise arguments. Las Vegas Assignment. Create a new chat in ChatGPT-4o and ask it to provide a (succinct) solution to the following problem. Provide a link to your chat (using the âShare Chatâ functionality) and a (legible) screenshot of the response. You can include this in LaTeX using the figure environment. If you are working in a study group, everyone in the group can submit the same screenshot. âYou are trying to build an amusement parkâŠâ Grade ChatGPTâs solution from Part (a). Specifically, if both the algorithm and its proof of correctness work, state that this is the case. If there is a bug in the proof, point to the bug, and argue whether this is a minor bug (e.g., a calculation error) but the proof is more or less correct, or whether it is a major bug (i.e., a crucial step is wrong or missing). In the GenAI-evaluation condition, âgradingâ did not mean only assigning a numerical score. Students were expected to evaluate the correctness of the generated algorithm and proof, identify any substantive errors or omissions, and explain why those errors affected the validity of the solution. In particular, students were asked to distinguish minor issues, such as arithmetic or presentation errors, from major issues, such as an incorrect algorithm, an invalid recurrence, a missing invariant, or a flawed proof step. Students were not required to provide a complete corrected solution, although they could do so as part of their explanation. To reduce variation across GenAI-generated artifacts, students in the evaluation condition were instructed to begin a new ChatGPT-4o chat and use the provided prompt without modifying the problem statement. Students submitted both a share link and a screenshot of the generated response, allowing the instructional staff to verify that a GenAI-generated solution had been produced and evaluated. We did not require all students to evaluate an identical instructor-generated output; instead, the intervention intentionally reflected a realistic setting in which students interact directly with GenAI tools. Despite variation across individual responses, generated solutions for each assignment tended to fall into a small number of recurring error patterns, summarized in Table 1. Students were instructed to evaluate the initial generated solution rather than iteratively prompting the model until it produced a correct answer. The exercise described above requires first preprocessing the input by sorting the list of parts according to the minimum height each part can reach independently. The solution then proceeds by constructing a two-dimensional dynamic programming table, where the first dimension corresponds to the part under consideration and the second dimension represents the height of the roller coaster. Across all homework assignments, we observed that the GenAI-generated solutions consistently fell into two to three distinct categories, each characterized by recurring and systematic errors. In the present exercise, the GenAI solution failed to sort the input and, in many instances, incorrectly formulated the dynamic programming approach by using a one-dimensional table instead of the required two-dimensional structure. After the first three homework assignments were completed, students took the midterm examination. The midterm was administered in person, proctored by the course instructor and teaching assistants, with closed books and the allowance of a single handwritten notes sheet per student. The exam assessed material covered in the first three assignments. Notably, one midterm problem required reasoning closely aligned with that of a prior homework exercise, specifically, an exercise for which one of the student groups had evaluated a GenAI-generated solution. The corresponding midterm problem is presented below. Midterm Exercise. You have decided to take a job trading in Wall Street. Using your superpower, algorithm design, you figure out n trading strategies, where strategy i, when executed, gives you a profit of viâ„0v_iâ„ 0 million dollars. You want to execute your strategies over n days. Strategies cannot be re-used, so once youâve used a strategy i, you cannot use it again. You would really like to use all your strategies, but you are scared that youâl attract too much attention to yourself, and then others will start stealing your strategies, taking your competitive advantage away. Therefore, you decide that on the k-th day, you should make sure to have accumulated a profit of at most 5âk5k million dollars so far. That is, the strategy on the first day you should make at most 55 million dollars, on the first two days you should make at most 1010 million dollars, and so on. To make sure this constraint is satisfied, some days you show up at the office, and execute a strategy that does nothing. Unfortunately, your boss notices when you do this, so, in order to keep your job, you want to maximize the number of strategies (ârealâ strategies, from the set s1,âŠ,sn\s_1,âŠ,s_n\) you execute, while keeping a low profile, and, of course, making a lot of money. Devise an efficient algorithm that, given as input n strategies s1,âŠ,sns_1,âŠ,s_n with profits v1,âŠ,vnv_1,âŠ,v_n (in millions of dollars), constructs a play-book for n days which describes whether on day i you execute some strategy sjs_j (at most one strategy) or do nothing, such that (1) you maximize the number of strategies from the set s1,âŠ,sns_1,âŠ,s_n, (2) for all k=1,âŠ,nk=1,âŠ,n the profit you accumulate in the first k days is at most 5âk5k million dollars, (3) the total profit you make is exactly x million dollars, for some x<5ânx<5n. Prove the correctness of your algorithm, and analyze its runtime. For the exercise described above, the solution requires first sorting the input based on the profit associated with each strategy, followed by the construction of a three-dimensional dynamic programming table. The first dimension corresponds to the day under consideration, the second to the strategy being evaluated, and the third to the cumulative profit achieved. This line of reasoning closely mirrors the structure and logic of the roller coaster exercise introduced earlier in the homework assignment. During the second half of the semester, the roles of the two groups were reversed. Students in the Las Vegas condition were tasked with directly solving the exercise, whereas students in the Monte Carlo condition evaluated a GenAI-generated solution. At the end of the semester, students completed a final examination covering all course topics, with a stronger emphasis on material from Homework Assignments 4â6. As in the midterm, one exam problem was intentionally designed to closely align with the structure and reasoning of a prior homework exercise, specifically, an exercise for which the Monte Carlo group had evaluated a GenAI-generated solution. 3.5. Data Collection and Instruments We analyzed multiple sources of quantitative and self-reported data: (1) item-level and aggregate scores from all homework assignments and the two course examinations; (2) final course percentage grades; (3) responses from a pre-midterm survey assessing topic-specific comfort and confidence (N=208N=208); and (4) responses from a post-final survey examining studentsâ perceived helpfulness of GenAI-based grading activities and self-reported changes in study habits (N=200N=200). All homework assignments and examinations were graded by the course instructor and the teaching assistant team, which consisted of seven graduate teaching assistants (GTAs) and seventeen undergraduate teaching assistants (UTAs). All UTAs had previously completed the course and earned a final grade of A or A+. UTAs together with GTAs were responsible for grading the first three (lower-difficulty) exercises on each homework assignment, whereas the fourth, most challenging exercise on each assignment was graded exclusively by the GTAs. Prior to grading, all graders responsible for a given exercise met to jointly develop a grading rubric and align on evaluation criteria, with the goal of minimizing inter-grader variability. All homework and exam grading was conducted using the Gradescope platform. For the examinations, graders additionally worked in a shared physical space and graded concurrently to further reduce grading inconsistencies. The grading criteria for the fourth exercise differed by condition but emphasized the same underlying algorithmic insight. For students solving the problem directly, credit was awarded for a correct algorithm, proof of correctness, and runtime analysis. For students evaluating a GenAI-generated solution, credit was awarded for accurately diagnosing the generated solution, identifying major correctness issues, and justifying the evaluation with reference to the algorithm, proof, or runtime analysis. Thus, the two conditions were not graded using identical rubrics, but both were designed to assess engagement with the same core algorithmic idea. Survey data were collected using a university-supported platform that required students to authenticate with their institutional credentials. The surveys did not collect identifying information, ensuring anonymity and encouraging honest responses. Approximately one hour prior to the midterm examination, at a point when nearly all exam preparation had been completed, we administered a Qualtrics survey to assess studentsâ self-reported comfort and confidence across the core algorithmic topics covered on the midterm. Response rates were high (Group A: n=101n=101; Group B: n=107n=107; total N=208N=208, 95% response rate). All items used a five-point Likert scale (1 = Very uncomfortable / Not confident; 5 = Very comfortable / Very confident). Due to technical constraints, an equivalent survey could not be administered prior to the final examination. Immediately following the final examination, we administered a second Qualtrics survey to both experimental groups (Group A: n=97n=97; Group B: n=103n=103; total N=200N=200, 91% response rate). The survey included two closed-ended items. The first assessed studentsâ perceived helpfulness of grading a GenAI-generated solution relative to completing the problem themselves, measured on a five-point Likert scale (1 = Very unhelpful; 5 = Very helpful). The second item examined whether the modified homework assignments influenced studentsâ exam preparation strategies, using a three-point nominal scale (1 = No; 2 = Somewhat; 3 = Yes). 3.6. Statistical Analysis For each outcome measure, we followed a three-step analytic workflow: (1) assumption screening using the ShapiroâWilk test for normality and Leveneâs test for homogeneity of variance; (2) test selection based on the results of assumption checks and the measurement scale of the data; and (3) effect size reporting to assess practical significance. For normally distributed, interval-scale data with homogeneous variances, we employed independent-samples t-tests and reported Cohenâs d. When these assumptions were violated, or when outcomes were ordinal, we used appropriate nonparametric alternatives, including the MannâWhitney U test (two independent groups), the Wilcoxon signed-rank test (paired samples), and the KruskalâWallis H or Friedman tests (three or more related samples). Corresponding effect sizes were reported using rank-biserial r or Kendallâs W, as appropriate. Two-sample KolmogorovâSmirnov tests were additionally used as shape-based robustness checks. Associations between ordinal variables were analyzed using Spearmanâs Ï, while categorical frequency data were analyzed using Ï2Ï^2 tests with CramĂ©râs V reported as the effect size. For the problem-level analysis of final examination performance, where score distributions exhibited substantial skew, we report the median and interquartile range (IQR) and rely on nonparametric tests. To control family-wise error rates in post hoc comparisons, we applied Holmâs sequential Bonferroni correction. All statistical tests were two-tailed with a significance threshold of α=.05α=.05. 4. Results !80 Topic Key Solution Idea LLMâs Mistake GenAI Group Score Summary (group, ÎŒ, median, Ï) Non-GenAI Group Score Summary (group, ÎŒ, median, Ï) Total Score Summary (ÎŒ, median, Ï) Induction Add-Subtract value and use triangle inequality to produce bound Wrong proof of correctness A, 5.41, 6.00, 3.89 B, 6.23, 6.80, 2.77 5.82, 6.00, 3.38 Divide & Conquer Even & odds cannot create invalid pairs, adding & dividing does not create new invalid pairs Wrong Algorithm A, 6.38, 8.00, 3.46 B, 5.95, 6.00, 3.28 6.16, 7.11, 3.37 Dynamic Programming Sort entries by an auxiliary value, create 2D DP table Wrong Algorithm A, 6.11, 7.00, 3.59 B, 5.36, 5.60, 3.33 5.73, 6.00, 3.47 !40 Average - - A, 5.97, 6.33, 2.74 B, 5.85, 6.00, 2.45 5.91, 6.00, 2.59 Graph Problems Create multiple copies of given graph and connect copies through appropriate edges Wrong Algorithm B, 6.41, 7.50, 3.51 A, 2.99, 1.50, 3.26 4.71, 4.00, 3.79 Network Flow Problem is equivalent to finding a partition of nodes to disjoint cycles in a bipartite graph Wrong proof of correctness B, 7.21, 8.00, 3.38 A, 3.90, 3.59, 3.07 5.57, 6.00, 3.62 Reductions We can enforce equality on first inequality by choosing tight number for the second inequality Wrong Algorithm B, 6.54, 8.00, 3.85 A, 5.55, 6.00, 3.31 6.05, 7.00, 3.62 !40 Average - - B, 6.72, 7.33, 2.67 A, 4.15, 3.68, 2.56 5.44, 5.63, 2.91 Midterm Question Sort entries by an auxiliary value, create 3D DP table - A, 1.62, 0.00, 3.29 B, 1.95, 0.00, 3.35 1.79, 0.00, 3.32 !40 Midterm Overall - - A, 60.69, 62.00, 14.45 B, 61.78, 62.00, 13.41 61.24, 62.00, 13.91 Final Question Create multiple copies of given graph and connect copies through appropriate edges - B, 5.43, 2.00, 5.22 A, 5.79, 6.00, 5.15 5.61, 6.00, 5.17 !40 Final Overall - - B, 59.06, 56.50, 16.72 A, 58.18, 59.00, 17.07 58.62, 58.00, 16.86 Table 1. HW grades are out of 10, Midterm Question Grades are out of 15, Midterm Overall Grades are out of 101, Final Question Grades are out of 15, Final Overall Grades are out of 113. âGenAI Groupâ denotes the group that evaluated a GenAI-generated solution for the corresponding homework topic or prior homework block; âNon-GenAI Groupâ denotes the group that solved the corresponding problem directly. The selected midterm and final questions required insight from one of the GenAI-evaluation questions during the prior 3 homeworks. A summary of homework LLM questions and midterm and final questions and grades can be found in Table 1. 4.1. Baseline Equivalence and Internal Validity To assess whether the two groups were comparable prior to the GenAI-grading intervention, we examined a baseline achievement measure derived from the sum of the three non-AI problems on Homeworks 1â3, rescaled to a 0â10 scale (Group A: n=109n=109; Group B: n=111n=111). Descriptive statistics indicated similar performance across groups (Group A: M=7.91M=7.91, SD=1.36SD=1.36, median = 8.04; Group B: M=8.04M=8.04, SD=1.35SD=1.35, median = 8.30), with overlapping 95% confidence intervals for the mean (Group A: [7.65,8.17][7.65,8.17]; Group B: [7.79,8.30][7.79,8.30]). Assumption screening revealed departures from normality for both groups (ShapiroâWilk: WA=0.94W_A=0.94, WB=0.91W_B=0.91, p<.001p<.001), while homogeneity of variance was satisfied (Leveneâs test: F=0.15F=0.15, p=.69p=.69). Given the large and balanced sample sizes, we report both parametric and nonparametric comparisons. An independent-samples t-test indicated no statistically significant difference between groups, tâ(218)=â0.72t(218)=-0.72, p=.47p=.47. Consistent with this result, a MannâWhitney U test likewise showed no significant difference (U=5658U=5658, p=.407p=.407), with a trivial effect size (r=0.06r=0.06). A two-sample KolmogorovâSmirnov test further confirmed that the distributions did not differ in shape (D=0.097D=0.097, p=.62p=.62). Taken together, these analyses converge to indicate that the two cohorts were statistically indistinguishable at baseline. This supports the internal validity of subsequent comparisons, suggesting that later differences in outcomes are unlikely to be attributable to pre-existing achievement differences between groups. 4.2. Primary Outcome: Overall Exam and Course Performance With baseline equivalence established, we next examined whether the order in which students experienced the two instructional conditions was associated with differences in overall course performance, as measured by midterm examination scores, final examination scores, overall course percentages, and relative change in exam performance over the semester. As detailed below, performance on all measures remained statistically indistinguishable across groups. For each high-stakes assessment, we first evaluated assumptions relevant to test selection. ShapiroâWilk tests failed to reject normality for midterm scores, final scores, normalized change scores (Final â Midterm), and overall course percentages in both groups (most p>.35p>.35, one p>.07p>.07). Leveneâs tests further indicated homogeneity of variances for all outcomes (all p>.19p>.19). Given that these assumptions were satisfied, independent-samples t-tests were used as the primary inferential procedure. For all comparisons, we report 95% confidence intervals (CIs) for the mean differences and Cohenâs d as a measure of effect size. Across all summative measures, group means were closely aligned. On the midterm examination (maximum score = 101), Group A achieved a mean score of M=60.7M=60.7 (SD=14.5SD=14.5), while Group B achieved a mean score of M=61.8M=61.8 (SD=13.4SD=13.4). On the final examination (maximum score = 113), Group A obtained a mean of M=58.2M=58.2 (SD=17.1SD=17.1), compared to M=59.1M=59.1 (SD=16.7SD=16.7) for Group B. Final course percentages were similarly comparable, with Group A averaging M=63.9M=63.9 (SD=12.1SD=12.1) and Group B averaging M=65.4M=65.4 (SD=11.4SD=11.4). To capture relative changes in exam performance over time, we additionally computed a normalized change score (Final â Midterm, scaled to 10 points). Group A exhibited a mean change of M=â0.86M=-0.86 (SD=1.3SD=1.3), while Group B exhibited a mean change of M=â0.89M=-0.89 (SD=1.1SD=1.1). The negative values reflect the differing exam maxima and the greater difficulty of post-midterm material rather than an absolute decline in learning. Inferential analyses revealed no statistically significant differences between groups on any summative performance measure. Midterm examination scores did not differ significantly between groups, tâ(218)=â0.58t(218)=-0.58, p=.56p=.56, with a 95% CI of [â4.8,2.6][-4.8,2.6] points and a negligible effect size (Cohenâs d=â0.08d=-0.08). Final examination scores likewise showed no significant difference, tâ(218)=â0.39t(218)=-0.39, p=.70p=.70, 95% CI [â5.4,3.6][-5.4,3.6] points, d=â0.05d=-0.05. The normalized change in exam performance (Final â Midterm) was also statistically indistinguishable across groups, tâ(218)=0.19t(218)=0.19, p=.85p=.85, 95% CI [â0.3,0.3][-0.3,0.3] points, d=0.03d=0.03. Similarly, overall course percentages did not differ significantly, tâ(218)=â0.95t(218)=-0.95, p=.34p=.34, 95% CI [â4.6,1.6][-4.6,1.6] points, with a negligible effect size (d=0.13d=0.13). Finally, distributions of letter grades were comparable across groups, Ï2â(9)=9.69Ï^2(9)=9.69, p=.38p=.38, with a small effect size (CramĂ©râs V=.07V=.07), further supporting the absence of meaningful differences in overall course outcomes. 4.3. Performance in GenAI-Aligned Exam Question In each examination (midterm and final), we included a problem whose solution required the same core techniques and sequence of logical steps as a prior homework exercise. On the midterm aligned question, Group A had previously evaluated a GenAI-generated solution to the corresponding homework problem, while Group B had solved the problem directly. On the final aligned question, this mapping was reversed. This crossover structure therefore provides two within-group observations, enabling us to examine whether prior participation in GenAI-based grading is associated with differences in the internalization and transfer of algorithmic reasoning. 4.3.1. Analytic approach and descriptive statistics. ShapiroâWilk tests indicated that all four samples deviated from normality (W=.57W=.57â.87.87, p<10â7p<10^-7), while Leveneâs tests supported the assumption of homogeneous variances (midterm p=.46p=.46, final p=.77p=.77). Accordingly, group comparisons were conducted using the MannâWhitney U test as the primary inferential procedure, with two-sample KolmogorovâSmirnov (KS) tests used as robustness checks. Effect sizes are reported using rank-biserial r, with α=.05α=.05. Table 1 summarizes the score distributions for the midterm and final exam items (each out of 15 points). The midterm item exhibited a pronounced floor effect (median = 0 in both groups), whereas the final item showed higher central tendency and greater dispersion, consistent with improved performance over the course of the semester. The midterm itemâs pronounced floor effect (median = 0 in both groups) limits its sensitivity to detect group differences; the transfer comparison is therefore carried primarily by the final-exam item, and conclusions about midterm transfer should be regarded as inconclusive rather than null. 4.3.2. Inferential results. For the midterm, the MannâWhitney test indicated no statistically significant difference between groups (U=5,644.5U=5,644.5, p=.30p=.30), with a trivial effect size (r=â0.058r=-0.058). The two-sample KolmogorovâSmirnov test yielded a consistent result (D=.067D=.067, p=.95p=.95). Similarly, for the final, no statistically significant difference was observed on the final exam item (U=6,191.5U=6,191.5, p=.76p=.76), with a negligible effect size (r=.020r=.020). The KS test again corroborated this result (D=.073D=.073, p=.90p=.90). Across both assessment points, we do not observe evidence of differences in performance associated with prior participation in GenAI-based grading activities. 4.4. Final Exam Performance by Problem Category To examine whether the timing of GenAI-assisted grading (first- versus second-half assignments) was associated with end-of-course performance, we partitioned the final examination into three categories: first-half, second-half, and hybrid problems. For each student, we computed a percentage score within each category by summing earned points and normalizing by the maximum possible points for that category. For first-half problems, Group A achieved M=71.43%M=71.43\% (SD=17.31SD=17.31; x~=71.88% x=71.88\%; IQR=25.00IQR=25.00), while Group B achieved M=73.23%M=73.23\% (SD=18.28SD=18.28; x~=75.00% x=75.00\%; IQR=31.25IQR=31.25). For second-half problems, performance was lower in both groups: Group A obtained M=45.93%M=45.93\% (SD=17.41SD=17.41; x~=45.39% x=45.39\%; IQR=25.00IQR=25.00), and Group B obtained M=46.36%M=46.36\% (SD=16.73SD=16.73; x~=44.08% x=44.08\%; IQR=25.66IQR=25.66). For hybrid problems (reported descriptively), Group A achieved M=51.98%M=51.98\% (SD=21.92SD=21.92; x~=47.37% x=47.37\%; IQR=21.05IQR=21.05), and Group B achieved M=53.22%M=53.22\% (SD=21.85SD=21.85; x~=47.37% x=47.37\%; IQR=26.32IQR=26.32). 4.4.1. Inferential analyses. ShapiroâWilk tests indicated non-normal distributions for both categories (p<.05p<.05 for all), while Leveneâs tests supported the assumption of homogeneous variances (first-half: p=.384p=.384; second-half: p=.600p=.600). Accordingly, group comparisons were conducted using MannâWhitney U tests, with rank-biserial r reported as the effect size. For first-half problems, the MannâWhitney test indicated no statistically significant difference between groups (U=5,624.50U=5,624.50, p=.368p=.368), with a trivial effect size (r=â0.061r=-0.061). The 95% confidence interval for the median difference was [â9.38, 3.13][-9.38,\,3.13]. For second-half problems, similarly, no statistically significant difference was observed (U=5,932.00U=5,932.00, p=.804p=.804), with a negligible effect size (r=â0.017r=-0.017). The 95% confidence interval for the median difference was [â5.92, 6.58][-5.92,\,6.58]. 4.5. Change in Homework Performance We next examined whether GenAI-based grading was related to changes in studentsâ homework performance over the course of the semester. For each student, we computed two within-student difference scores: (1) the change in performance on the GenAI-graded problem across the two halves of the course (with positive values indicating higher performance during periods of GenAI-based grading), and (2) the corresponding change in performance on the three problems solved independently. 4.5.1. Descriptive statistics. Metric Group A (M±SDM ) Group B (M±SDM ) GenAI Questions (%) 18±2518± 25 9±239± 23 Non-GenAI Questions (%) 3±143± 14 â3±13-3± 13 Descriptively, Group A exhibited a larger positive change on the GenAI-graded problems and a modest advantage on the student-solved problems. 4.5.2. Assumption checks. ShapiroâWilk tests indicated deviations from normality in three of the four distributions (p<.05p<.05), with the exception of Group Aâs GenAI-graded difference scores. Leveneâs tests supported the assumption of homogeneous variances across groups (p>.11p>.11). Given these mixed normality results, we report nonparametric analyses as the primary inferential procedure, supplemented by parametric tests for the GenAI-graded problems as a robustness check. 4.5.3. Inferential analyses. For the GenAI-graded problems, the MannâWhitney test indicated a statistically significant difference between groups (U=7,046U=7,046, p=.035p=.035), with a small effect size (r=.14r=.14). A confirmatory independent-samples t-test yielded a consistent result, tâ(218)=2.90t(218)=2.90, p=.004p=.004, with a small effect size (Cohenâs d=.39d=.39). Two cautions apply in interpreting this difference. First, the two conditions were not graded with identical rubrics (see Section 3.5), so the comparison partly reflects differing scoring structures for solving versus evaluating, not solely a difference in learning. Second, because randomization occurred at the working-group level, this student-level result is the finding most sensitive to within-group dependence (see Section 5.1). For the student-solved (non-GenAI) problems, the MannâWhitney test likewise indicated a statistically significant difference between groups (U=7,119U=7,119, p=.024p=.024), with a small effect size (r=.15r=.15). Notably, these tasks were identical across conditions, so this difference cannot be attributed to the intervention; it more likely reflects time-varying factors such as topic difficulty, exam timing, and accumulated course experience across the two halves of the course. This pattern reinforces our broader interpretation that the homework-level differences should not be read as direct evidence of a learning effect from the GenAI activity. 4.6. Emerging Affective Outcomes (Midterm Self-Efficacy) After students had completed three of the six homework assignments under their assigned condition, we administered a Qualtrics survey approximately one hour prior to the midterm examination, at a point when most exam preparation had been completed. The survey achieved a high response rate (Group A: n=101n=101; Group B: n=107n=107; overall N=208N=208, 95%) and measured studentsâ self-reported comfort and confidence across the five core algorithmic topics assessed on the midterm: asymptotic analysis, divide & conquer, dynamic programming, greedy methods, and graph algorithms. Each item used a five-point Likert scale (1 = Very uncomfortable / Not confident; 5 = Very comfortable / Very confident). Due to technical limitations, an equivalent survey could not be administered prior to the final examination. 4.6.1. Data preparation and analysis. Likert-scale responses were recoded to ordinal values (1â5). Internal consistency was assessed using Cronbachâs α, yielding acceptable reliability for both the Comfort (α=0.79α=0.79) and Confidence (α=0.84α=0.84) scales, supporting the use of composite scores. All analyses were conducted at α=.05α=.05 using nonparametric procedures. Between-group differences in Comfort and Confidence composites were evaluated using MannâWhitney U tests. Within-subject variation across topics was assessed using Friedman tests, followed by Wilcoxon signed-rank post hoc comparisons with Holm correction. Associations between Comfort and Confidence composites were examined using Spearmanâs rank correlation (Ï). 4.6.2. Results. No statistically significant differences were observed between groups on either the Comfort composite (U=5,788U=5,788, p=.374p=.374) or the Confidence composite (U=5,152U=5,152, p=.561p=.561) after three assignments under their respective conditions. Friedman tests indicated significant within-subject variation across topics for both Comfort (Ï2â(4)=156.78Ï^2(4)=156.78, p<.001p<.001) and Confidence (Ï2â(4)=166.50Ï^2(4)=166.50, p<.001p<.001). Holm-adjusted post hoc comparisons showed that Asymptotic Analysis (Comfort median = 4; Confidence median = 3â4) and Divide & Conquer (Comfort median = 4; Confidence median = 3) were rated significantly higher than Dynamic Programming, Greedy Methods, and Graph Algorithms (all medians = 3) on both scales (pholm<.001p_holm<.001), with no statistically significant differences among the latter three topics (all pholm>.05p_holm>.05). Finally, a strong positive association was observed between Comfort and Confidence composites (Spearmanâs Ï=0.714Ï=0.714, p<.001p<.001), indicating substantial concordance between the two measures. 4.7. End-of-Course Perceptions (Post-Final Survey) Immediately after the final exam, we administered a Qualtrics survey to both experimental groups (Group A: n=97n=97; Group B: n=103n=103; overall N=200N=200, 91% response rate). Two closed-ended items were included: âą Helpfulness of Grading. âHow much did grading GenAIâs solution (versus completing the problem yourself) help you learn the material?â 5-point Likert (1 = Very unhelpful ⊠5 = Very helpful). âą Change in Studying. âDid the modified homework assignments change how you studied for exams?â 3-point nominal (1 = No; 2 = Kind of; 3 = Yes). All responses were converted to ordinal codes; group completion rates did not differ, so we proceeded with our preregistered nonparametric tests at α=.05α=.05. 4.7.1. Descriptive statistics. Both groups reported a neutral median rating on the helpfulness item (x~=3.0 x=3.0), indicating that perceptions generally ranged between âSomewhat unhelpfulâ and âSomewhat helpful.â With respect to changes in study habits, the majority of students reported no change (154/200, 77.0%), followed by âKind ofâ (32/200, 16.0%) and âYesâ (14/200, 7.0%). 4.7.2. Between-group comparisons. No statistically significant difference was observed between groups in perceived helpfulness of grading (MannâWhitney U=5,132U=5,132, p=.728p=.728). Similarly, a chi-square test of independence indicated no association between group assignment and reported changes in study habits (Ï2â(2,N=200)=0.97Ï^2(2,N=200)=0.97, p=.616p=.616), with a small effect size (CramĂ©râs V=0.07V=0.07). 4.7.3. Association between helpfulness and study-habits change. A KruskalâWallis test indicated a statistically significant difference in helpfulness ratings across the three study-change categories (Hâ(2)=17.74H(2)=17.74, p<.001p<.001). Follow-up pairwise MannâWhitney tests with Bonferroni-adjusted p-values showed that students who reported changing their study habits rated the grading activity as more helpful than those who reported no change (No vs. Kind of: U=1,742U=1,742, padj=.019p_adj=.019; No vs. Yes: U=505U=505, padj=.002p_adj=.002). The difference between the âKind ofâ and âYesâ categories was not statistically significant (U=143U=143, padj=.122p_adj=.122). 5. Discussion Our study investigated how evaluating AI-generated solutions compares with traditional problem-solving in a junior-level algorithms course. Across several outcome measures, we found no evidence that evaluating GenAI-generated solutions led to higher overall exam performance or course grades than directly solving problems. The two groups were comparable at baseline and remained statistically indistinguishable on midterm scores, final exam scores, course percentages, and changes in exam performance over time. We also found no significant differences on exam questions designed to be structurally aligned with prior homework exercises. Thus, in this implementation, prior exposure to a GenAI-generated solution and its evaluation did not measurably improve transfer to a related exam problem relative to solving the corresponding homework problem directly. Our findings complicate two common intuitions about GenAI-based coursework. One concern is that asking students to evaluate AI-generated solutions instead of solving problems themselves will substantially weaken learning, because students lose practice constructing complete solutions. A competing hope is that evaluating flawed AI output may strengthen learning by prompting students to inspect, critique, and explain algorithmic reasoning. In our study, neither extreme prediction was supported. Students who engaged in GenAI-solution evaluation did not outperform students who solved the problems directly, but we also found no evidence of broad performance losses on summative assessments or structurally aligned exam questions. These results suggest that GenAI-evaluation tasks should not be viewed simply as weaker substitutes for problem solving, nor as automatically superior forms of metacognitive practice. Rather, they appear to redistribute the work students do: students engage less in open-ended solution generation and more in verification, diagnosis, and judgment. This redistribution is also visible in the homework data: students tended to receive higher scores on the GenAI-evaluation problems than on the corresponding direct-solution problems. However, in this implementation, this localized homework advantage did not translate into measurable differences in downstream exam performance. Our findings have practical implications for the design of GenAI-based assignments. The absence of broad performance losses suggests that structured GenAI-evaluation activities can be incorporated into algorithms coursework without necessarily displacing all meaningful practice. However, the absence of measurable gains also suggests that simply asking students to grade an AI-generated solution may not be sufficient to improve transfer. This conclusion is supported by our survey results: the vast majority of students reported that the GenAI-based assignments did not change their study habits, but students who reported adapting their study behaviors rated the evaluation activity as significantly more helpful than those who did not. Future versions of such assignments may need to push students beyond simple diagnosis, for example by asking them to identify the first invalid step in an argument or algorithm, construct a counterexample where a proposed algorithm fails, propose a corrected algorithm or invariant, and explain why the corrected solution works. In this sense, the pedagogical value of GenAI evaluation may depend less on exposure to AI output itself and more on how carefully the evaluation task is scaffolded. 5.1. Threats to Validity Several limitations should be considered when interpreting these results. First, although students were assigned to conditions using a randomized crossover design, randomization occurred at the homework-working-group level rather than at the individual-student level. This choice reflected the collaborative structure of the course and was intended to reduce contamination across conditions, but it also means that student outcomes within working groups may not be fully independent. Because clustering inflates Type I error rather than masking true effects, this concern bears asymmetrically on our findings: our null summative results would, if anything, be reinforced by accounting for within-group dependence, whereas our one positive finding (the localized homework difference) should be interpreted with corresponding caution as a result that may not survive a clustered analysis. Second, the intervention compares two authentic instructional activities rather than isolating a single mechanism. Students in the GenAI-evaluation condition generated an LLM response, read and interpreted that response, evaluated its correctness, and explained any errors. Thus, the treatment should be understood as a structured GenAI-evaluation activity, not as a pure measure of the effect of exposure to AI-generated solutions or of grading alone. Third, the GenAI-generated artifacts themselves may have varied across students. Although students were given standardized prompts and asked to use the same model, LLM outputs can differ across sessions and contexts. This variation is pedagogically realistic, but it may also introduce noise into the intervention. Instructors who use similar activities may need to decide whether to let students generate their own AI outputs or instead provide a common instructor-generated artifact for all students to evaluate. Fourth, the crossover design took place within a live course in which topics changed over time. The first half of the course covered different material from the second half, and later topics were generally more difficult. As a result, differences between first-half and second-half outcomes may reflect topic difficulty, student maturation, accumulated course experience, or exam timing in addition to the instructional condition itself. Fifth, our measures capture only some forms of learning. Exam scores and structurally aligned exam questions provide meaningful evidence about summative performance and transfer of algorithmic reasoning, but they may not capture all competencies targeted by GenAI-evaluation tasks, such as trust calibration, debugging AI-generated explanations, or recognizing subtle flaws in plausible-looking solutions. Conversely, self-reported helpfulness and study behavior are subjective and may be affected by recall, social desirability, or studentsâ differing interpretations of the survey items. Finally, the absence of statistically significant differences should not be interpreted as proof that the two activities are equivalent. Our results indicate that we did not observe evidence of meaningful performance differences in this course implementation. Establishing equivalence or non-inferiority would require an analysis designed specifically for that purpose. 6. Conclusion and Future Work As GenAI tools become increasingly integrated into studentsâ programming workflows, computing education faces a practical design question: how should coursework adapt when students can readily generate plausible solutions to traditional assignments? This study examined one response: replacing some direct solution construction with structured evaluation of GenAI-generated solutions in a junior-level algorithms course. Using a randomized crossover design, we found no evidence that this intervention improved summative performance or transfer to structurally aligned exam problems relative to traditional problem solving, but also no evidence of broad performance losses. Together, these findings suggest that structured GenAI-evaluation activities may be feasible components of algorithms instruction, but that their value depends on careful task design rather than on the mere presence of AI-generated content. Future work should investigate which forms of scaffolding make GenAI-evaluation activities most effective. In particular, it would be useful to compare evaluation-only tasks with evaluation-and-repair tasks, to contrast student-generated AI outputs with standardized instructor-provided outputs, and to study whether these activities benefit some students more than others. More broadly, future research should examine how evaluation-centered GenAI activities affect skills that are increasingly important in AI-rich programming environments, including verification, critique, explanation, repair, and calibrated reliance on generated solutions, as well as the development of validated assessment instruments to measure these specific competencies. References B. A. Becker, P. Denny, J. Finnie-Ansley, A. Luxton-Reilly, J. Prather, and E. A. Santos (2023) Programming is hard-or at least it used to be: educational opportunities and challenges of ai code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, p. 500â506. Cited by: §1. C. Bellettini, M. Lodi, V. Lonati, M. Monga, and A. Morpurgo (2023) DaVinci goes to bebras: a study on the problem solving ability of gpt-3. In CSEDU 2023-15th International Conference on Computer Supported Education, Vol. 2, p. 59â69. Cited by: §1. C. Bird, D. Ford, T. Zimmermann, N. Forsgren, E. Kalliamvakou, T. Lowdermilk, and I. Gazit (2022) Taking flight with copilot: early insights and opportunities of ai-powered pair-programming tools. Queue 20 (6), p. 35â57. Cited by: §1, §1, §1. J. L. Booth, K. E. Lange, K. R. Koedinger, and K. J. Newton (2013) Using example problems to improve student learning in algebra: differentiating between correct and incorrect examples. Learning and Instruction 25, p. 24â34. External Links: ISSN 0959-4752, Document, Link Cited by: §2.4. D. J. Bouvier, B. Pereira Cipriano, R. Glassey, O. Petrovska, E. Anderson, A. Birillo, R. Dougherty, R. Pettit, N. Pombo, E. Rahimi, C. Ramakrishnan, A. Steinmaurer, S. Taneja, M. Usman, and A. Vadaparty (2026) The rest of the robots: generative ai in post-introductory computing education. In Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education, ITiCSE-WGR 2025, New York, NY, USA, p. 61â107. External Links: ISBN 9798400721670, Link, Document Cited by: §2.7. M. T.H. Chi, M. Bassok, M. W. Lewis, P. Reimann, and R. Glaser (1989) Self-explanations: how students study and use examples in learning to solve problems. Cognitive Science 13 (2), p. 145â182. External Links: ISSN 0364-0213, Document, Link Cited by: §2.4. P. Denny, B. A. Becker, J. Leinonen, and J. Prather (2023a) Chat overflow: artificially intelligent models for computing education - renaissance or apocaiypse?. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2023, New York, NY, USA, p. 3â4. External Links: ISBN 9798400701382, Link, Document Cited by: §1. P. Denny, V. Kumar, and N. Giacaman (2023b) Conversing with copilot: exploring prompt engineering for solving cs1 problems using natural language. In Proceedings of the 54th ACM technical symposium on computer science education V. 1, p. 1136â1142. Cited by: §1. P. Denny, J. Leinonen, J. Prather, A. Luxton-Reilly, T. Amarouche, B. A. Becker, and B. N. Reeves (2024a) Prompt problems: a new programming exercise for the generative ai era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, p. 296â302. Cited by: §1. P. Denny, J. Prather, B. A. Becker, J. Finnie-Ansley, A. Hellas, J. Leinonen, A. Luxton-Reilly, B. N. Reeves, E. A. Santos, and S. Sarsa (2024b) Computing education in the era of generative ai. Communications of the ACM 67 (2), p. 56â67. Cited by: §1. P. Denny, D. H. Smith IV, M. Fowler, J. Prather, B. A. Becker, and J. Leinonen (2024c) Explaining code with a purpose: an integrated approach for developing code comprehension and prompting skills. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, p. 283â289. Cited by: §1. E. Dickey, A. Bejarano, and C. Garg (2024) AI-lab: a framework for introducing generative artificial intelligence tools in computer programming courses. SN Computer Science 5 (6), p. 720. External Links: Document, Link, ISSN 2661-8907 Cited by: §2.2. N. Donekal Chandrashekar, S. B. Nizamani, M. Ellis, and N. Ramakrishnan (2026) Demystify, use, reflect: preparing students to be informed llm-users. In Proceedings of the 57th ACM Technical Symposium on Computer Science Education V.2, SIGCSE TS 2026, New York, NY, USA, p. 1299â1300. External Links: ISBN 9798400722554, Link, Document Cited by: §2.2, §2.3, §2.6. K. Durkin and B. Rittle-Johnson (2012) The effectiveness of using incorrect examples to support learning about decimal magnitude. Learning and Instruction 22 (3), p. 206â214. External Links: ISSN 0959-4752, Document, Link Cited by: §2.4. J. Finnie-Ansley, P. Denny, B. A. Becker, A. Luxton-Reilly, and J. Prather (2022) The robots are coming: exploring the implications of openai codex on introductory programming. In Proceedings of the 24th Australasian Computing Education Conference, ACE â22, New York, NY, USA, p. 10â19. External Links: ISBN 9781450396431, Link, Document Cited by: §1, §1, §1, §2.1. J. Finnie-Ansley, P. Denny, A. Luxton-Reilly, E. A. Santos, J. Prather, and B. A. Becker (2023) My ai wants to know if this will be on the exam: testing openaiâs codex on cs2 programming exercises. In Proceedings of the 25th Australasian Computing Education Conference, p. 97â104. Cited by: §1, §1. C. Hundhausen, A. Agrawal, and K. Ryan (2010) The design of an online environment to support pedagogical code reviews. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education, SIGCSE â10, New York, NY, USA, p. 182â186. External Links: ISBN 9781450300063, Link, Document Cited by: §2.5. C. D. Hundhausen, A. Agrawal, and P. Agarwal (2013) Talking about code: integrating pedagogical code reviews into early computing courses. ACM Trans. Comput. Educ. 13 (3). External Links: Link, Document Cited by: §2.5. T. D. Indriasari, A. Luxton-Reilly, and P. Denny (2020) A review of peer code review in higher education. ACM Trans. Comput. Educ. 20 (3). External Links: Link, Document Cited by: §2.5. B. Jury, A. Lorusso, J. Leinonen, P. Denny, and A. Luxton-Reilly (2024) Evaluating llm-generated worked examples in an introductory programming course. In Proceedings of the 26th Australasian Computing Education Conference, ACE â24, New York, NY, USA, p. 77â86. External Links: ISBN 9798400716195, Link, Document Cited by: §2.6. M. Kazemitabaar, J. Chow, C. K. T. Ma, B. J. Ericson, D. Weintrop, and T. Grossman (2023) Studying the effect of ai code generators on supporting novice learners in introductory programming. In Proceedings of the 2023 CHI conference on human factors in computing systems, p. 1â23. Cited by: §1. J. Leinonen, P. Denny, S. MacNeil, S. Sarsa, S. Bernstein, J. Kim, A. Tran, and A. Hellas (2023a) Comparing code explanations created by students and large language models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, p. 124â130. Cited by: §1. J. Leinonen, A. Hellas, S. Sarsa, B. Reeves, P. Denny, J. Prather, and B. A. Becker (2023b) Using large language models to enhance programming error messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, p. 563â569. Cited by: §1. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. (2022) Competition-level code generation with alphacode. Science 378 (6624), p. 1092â1097. Cited by: §1. J. Liu, S. Poulsen, E. Goodwin, H. Chen, G. Williams, Y. Gertner, and D. Franklin (2025) Teaching algorithm design: a literature review. ACM Trans. Comput. Educ. 25 (2). External Links: Link, Document Cited by: §1. D. Loksa, L. Margulieux, B. A. Becker, M. Craig, P. Denny, R. Pettit, and J. Prather (2022) Metacognition and self-regulation in programming education: theories and exemplars of use. ACM Trans. Comput. Educ. 22 (4). External Links: Link, Document Cited by: §2.3. S. MacNeil, J. Prather, R. A. Nabid, S. Gutierrez, S. Carvalho, S. Shrestha, P. Denny, B. N. Reeves, J. Leinonen, and R. L. Rossetti (2025) Fostering responsible ai use through negative expertise: a contextualized autocompletion quiz. In Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2025, New York, NY, USA, p. 326â332. External Links: ISBN 9798400715679, Link, Document Cited by: §2.2, §2.3, §2.6. S. MacNeil, A. Tran, A. Hellas, J. Kim, S. Sarsa, P. Denny, S. Bernstein, and J. Leinonen (2023) Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, p. 931â937. Cited by: §1. L. E. Margulieux, J. Prather, B. N. Reeves, B. A. Becker, G. Cetin Uzun, D. Loksa, J. Leinonen, and P. Denny (2024) Self-regulation, self-efficacy, and fear of failure interactions with how novices use llms to solve programming problems. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2024, New York, NY, USA, p. 276â282. External Links: ISBN 9798400706004, Link, Document Cited by: §1, §2.3. M. Messer, N. C. C. Brown, M. Kölling, and M. Shi (2024) Automated grading and feedback tools for programming education: a systematic review. ACM Trans. Comput. Educ. 24 (1). External Links: Link, Document Cited by: §2.6. K. Muldner, J. Jennings, and V. Chiarelli (2022) A review of worked examples in programming activities. ACM Trans. Comput. Educ. 23 (1). External Links: Link, Document Cited by: §2.4. D. Nam, A. Macvean, V. Hellendoorn, B. Vasilescu, and B. Myers (2024) Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, p. 1â13. Cited by: §1. D. J. Nicol and D. MacfarlaneâDick (2006) Formative assessment and selfâregulated learning: a model and seven principles of good feedback practice. Studies in Higher Education 31 (2), p. 199â218. External Links: Document, Link, https://doi.org/10.1080/03075070600572090 Cited by: §2.3. M. S. Orenstrakh, O. Karnalim, C. A. Suarez, and M. Liut (2024) Detecting llm-generated text in computing education: comparative study for chatgpt cases. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), p. 121â126. Cited by: §1. E. L. Ouh, B. K. S. Gan, K. Jin Shim, and S. Wlodkowski (2023) ChatGPT, can you generate solutions for my coding exercises? an evaluation on its effectiveness in an undergraduate java programming course.. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2023, New York, NY, USA, p. 54â60. External Links: ISBN 9798400701382, Link, Document Cited by: §2.1. R. A. Poldrack, T. Lu, and G. BeguĆĄ (2023) AI-assisted coding: experiments with gpt-4. arXiv preprint arXiv:2304.13187. Cited by: §1. S. Poulsen, S. Sarsa, J. Prather, J. Leinonen, B. A. Becker, A. Hellas, P. Denny, and B. N. Reeves (2024) Solving proof block problems using large language models. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2024, New York, NY, USA, p. 1063â1069. External Links: ISBN 9798400704239, Link, Document Cited by: §2.1, §2.7. J. Prather, P. Denny, J. Leinonen, B. A. Becker, I. Albluwi, M. Craig, H. Keuning, N. Kiesler, T. Kohn, A. Luxton-Reilly, S. MacNeil, A. Petersen, R. Pettit, B. N. Reeves, and J. Savelka (2023a) The robots are here: navigating the generative ai revolution in computing education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education, ITiCSE-WGR â23, New York, NY, USA, p. 108â159. External Links: ISBN 9798400704055, Link, Document Cited by: §1, §1, §1, §2.1, §2.7. J. Prather, J. Leinonen, N. Kiesler, J. Gorson Benario, S. Lau, S. MacNeil, N. Norouzi, S. Opel, V. Pettit, L. Porter, B. N. Reeves, J. Savelka, I. Smith, S. Strickroth, and D. Zingaro (2025) Beyond the hype: a comprehensive review of current trends in generative ai research, teaching practices, and tools. In 2024 Working Group Reports on Innovation and Technology in Computer Science Education, ITiCSE 2024, New York, NY, USA, p. 300â338. External Links: ISBN 9798400712081, Link, Document Cited by: §2.1, §2.2, §2.6, §2.7. J. Prather, R. Pettit, K. McMurry, A. Peters, J. Homer, and M. Cohen (2018) Metacognitive difficulties faced by novice programmers in automated assessment tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research, p. 41â50. Cited by: §1. J. Prather, B. N. Reeves, P. Denny, B. A. Becker, J. Leinonen, A. Luxton-Reilly, G. Powell, J. Finnie-Ansley, and E. A. Santos (2023b) âItâs weird that it knows what i wantâ: usability and interactions with copilot for novice programmers. ACM transactions on computer-human interaction 31 (1), p. 1â31. Cited by: §1. J. Prather, B. N. Reeves, J. Leinonen, S. MacNeil, A. S. Randrianasolo, B. A. Becker, B. Kimmel, J. Wright, and B. Briggs (2024) The widening gap: the benefits and harms of generative ai for novice programmers. In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1, ICER â24, New York, NY, USA, p. 469â486. External Links: ISBN 9798400704758, Link, Document Cited by: §1, §2.2, §2.3. Y. Qiao, M. I. H. Shihab, and C. Hundhausen (2026) A systematic literature review of the use of genai assistants for code comprehension: implications for computing education research and practice. ACM Transactions on Computing Education 26 (2), p. 1â33. Cited by: §1, §1. B. Reeves, S. Sarsa, J. Prather, P. Denny, B. A. Becker, A. Hellas, B. Kimmel, G. Powell, and J. Leinonen (2023) Evaluating the performance of code generation models for solving parsons problems with small prompt variations. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, p. 299â305. Cited by: §1, §1. K. Reily, P. L. Finnerty, and L. Terveen (2009) Two peers are better than one: aggregating peer reviews for computing assignments is surprisingly accurate. In Proceedings of the 2009 ACM International Conference on Supporting Group Work, GROUP â09, New York, NY, USA, p. 115â124. External Links: ISBN 9781605585000, Link, Document Cited by: §2.5. J. Savelka, A. Agarwal, M. An, C. Bogart, and M. Sakr (2023a) Thrilled by your progress! large language models (gpt-4) no longer struggle to pass assessments in higher education programming courses. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1, ICER â23, New York, NY, USA, p. 78â92. External Links: ISBN 9781450399760, Link, Document Cited by: §1, §1, §2.1. J. Savelka, A. Agarwal, C. Bogart, and M. Sakr (2023b) Large language models (gpt) struggle to answer multiple-choice questions about code. arXiv preprint arXiv:2303.08033. Cited by: §1. A. Sinha, S. Goyal, Z. Sy, R. Kuperus, E. Dickey, and A. Bejarano (2024) BoilerTAI: a platform for enhancing instruction using generative ai in educational forums. In 2024 IEEE Frontiers in Education Conference (FIE), Vol. , p. 1â8. External Links: Document Cited by: §2.6. D. H. Smith IV and C. Zilles (2024) Code generation based grading: evaluating an auto-grading mechanism forâ explain-in-plain-englishâ questions. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, p. 171â177. Cited by: §2.6. S. B. Smith, H. Wei, A. OâNeill, A. Durai, J. DeNero, J.D. Zamfirescu-Pereira, and N. Norouzi (2025) Spotting ai missteps: students take on llm errors in cs1. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 2, SIGCSETS 2025, New York, NY, USA, p. 1627â1628. External Links: ISBN 9798400705328, Link, Document Cited by: §2.2. L. Tankelevitch, V. Kewenig, A. Simkute, A. E. Scott, A. Sarkar, A. Sellen, and S. Rintel (2024) The metacognitive demands and opportunities of generative ai. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI â24, New York, NY, USA. External Links: ISBN 9798400703300, Link, Document Cited by: §2.2, §2.3. A. Vadaparty, D. Zingaro, D. H. Smith IV, M. Padala, C. Alvarado, J. Gorson Benario, and L. Porter (2024) CS1-llm: integrating llms into cs1 instruction. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2024, New York, NY, USA, p. 297â303. External Links: ISBN 9798400706004, Link, Document Cited by: §2.2. P. Vaithilingam, T. Zhang, and E. L. Glassman (2022) Expectation vs. experience: evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts, p. 1â7. Cited by: §1.