Paper deep dive
How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles
Shang Wu, Catarina G Belem, Shuyuan Fu, Mark Steyvers, Padhraic Smyth
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While AI assistance can improve human task performance in the short term, it may also undermine the development of skills in the longer term. We examine this tension in a controlled logic-puzzle experiment involving on-demand AI assistance, where participants complete tasks before, during, and after AI is available. By experimentally varying AI request costs, we find that lower-cost assistance induces more frequent AI use. We also find that participants who request AI assistance during the AI-access phase perform worse at the task after assistance is removed, and their subsequent unassisted performance is overestimated when predicted from earlier AI-assisted performance. We use a Bayesian latent ability model to separate initial ability, post-AI ability, and participant-specific skill change, while estimating how independent reasoning during the AI-access phase relates to skill development. The results show that greater independent problem-solving effort is associated with larger gains in latent ability, consistent with the interpretation that skill development is weaker when AI assistance substitutes for independent reasoning.
Tags
Links
- Source: https://arxiv.org/abs/2608.23543v1
- Canonical: https://arxiv.org/abs/2608.23543v1
Trouble viewing inline? Open PDF directly ā
Full Text
68,599 characters extracted from source content.
Expand or collapse full text
How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles DOI: X.XXXXXXXConference: Make sure to enter the correct conference title from your rights confirmation email; June 03ā05, 2018; Woodstock, NYISBN: 978-1-4503-X-X/2018/06Conference: 2026 ACM Conference on Human-AI Complementarity and Alignment; September 27ā30, 2026; Alexandria, VA, USA2026 ACM Conference on Human-AI Complementarity and Alignment (HCOMP 2026), September 27ā30, 2026, Alexandria, VA, USADOI: 10.1145/3834580.3838741ISBN: 979-8-4007-2894-5/2026/09CCS: Human-centered computing Empirical studies in HCI Shang Wu email: shangw13@uci.edu Affiliation: University of California, Irvine, USA , Catarina G Belem email: cbelem@uci.edu Affiliation: University of California, Irvine, USA , Shuyuan Fu email: shuyuanf@uci.edu Affiliation: University of California, Irvine, USA , Mark Steyvers email: msteyver@uci.edu Affiliation: University of California, Irvine, USA and Padhraic Smyth email: smyth@uci.edu Affiliation: University of California, Irvine, USA Ā© c Abstract. While AI assistance can improve human task performance in the short term, it may also undermine the development of skills in the longer term. We examine this tension in a controlled logic-puzzle experiment involving on-demand AI assistance, where participants complete tasks before, during, and after AI is available. By experimentally varying AI request costs, we find that lower-cost assistance induces more frequent AI use. We also find that participants who request AI assistance during the AI-access phase perform worse at the task after assistance is removed, and their subsequent unassisted performance is overestimated when predicted from earlier AI-assisted performance. We use a Bayesian latent ability model to separate initial ability, post-AI ability, and participant-specific skill change, while estimating how independent reasoning during the AI-access phase relates to skill development. The results show that greater independent problem-solving effort is associated with larger gains in latent ability, consistent with the interpretation that skill development is weaker when AI assistance substitutes for independent reasoning. Keywords: human-AI interaction, individual skill development, reliance, Bayesian modeling ā c-license: by 1. Introduction As AI systems become increasingly used as aids in everyday cognitive activities, an important question is not only whether they improve human cognitive performance but also whether they promote or hinder skill development. While AI assistance can improve immediate task performance in the short term (Gajos and Mamykina, 2022; Karny et al., 2024), it may also alter the degree to which people engage with a cognitive task. In particular, one significant concern is that readily available AI assistance could reduce the need for sustained independent reasoning and encourage humans to offload cognitive effort to the AI system (BuƧinca et al., 2021; Risko and Gilbert, 2016; Lee et al., 2025). This creates a central tension in AI-assisted cognition: the same assistance that helps users solve problems in the short run may weaken the development of the underlying skills needed to solve similar problems independently in the future. This tension is especially important in settings where learning depends on active engagement. Skill development requires repeated opportunities to reason, test strategies, revise mistakes, and internalize problem structure (Anderson et al., 1995). If AI assistance provides a substitute for these processes, humans may appear to be successful at a task while the assistance is available, but have weaker performance once the assistance is removed (Wu et al., 2026; Liu et al., 2026). At the same time, AI use is not necessarily harmful in itself. Humans may request assistance while still engaging deeply with the task, and the consequences of AI use may depend on whether it complements or replaces independent problem-solving effort (Shen and Tamkin, 2026; Lehmann et al., 2024; Poulidis et al., 2025). Understanding the effect of AI on learning, therefore, requires distinguishing between (a) the frequency of AI use, and (b) how users use it. We study this issue through a controlled logic-puzzle experiment that tracks skill development across three phases: Phase 1, an initial AI-free assessment; Phase 2, an intermediate phase in which AI assistance may be available; and Phase 3, a final AI-free assessment (see Figure 1). During Phase 2, participants can request AI assistance on demand. We experimentally vary the costs of AI requests across conditions to induce different levels of AI engagement across participants. By comparing participantsā performance in Phases 1 and 3, the design allows us to examine how the cost of AI shapes reliance behavior and how participantsā engagement with AI during Phase 2 relates to subsequent performance after assistance is removed. We investigate relationships between various factors related to individual behavior and performance by fitting a Bayesian latent ability model that treats pre- and post-AI accuracy and response time as indicators of underlying ability, and that models changes in skill levels as a function of observed behavior, performance measures, and initial ability. Our findings suggest that weaker short-term skill development is associated with AI assistance that displaces independent reasoning. Lower-cost access leads participants to use AI more frequently, and participants who request AI perform worse once assistance is removed. The Bayesian model further suggests that short-term skill development is associated less with how often participants request AI help and more with whether they continue to engage in independent reasoning. Participants who devote more effort to independent problem solving show greater gains in latent ability, whereas the frequency of requests for AI assistance is not associated with gains in skill development after accounting for initial ability and independent effort. This paper contributes to research on human-AI interaction by adding nuance to how AI assistance affects learning. Our findings suggest that, in a controlled logical puzzle task, the central issue is not AI use itself, but whether assistance displaces the independent reasoning through which skills are built. This distinction is particularly important in educational and training settings, where AI support should ideally improve current task performance without weakening future problem-solving ability. Figure 1. Overview of the three-phase study design. Participants completed Phase 1 and Phase 3 without simulated AI assistance, providing pre- and post-AI measures of unassisted performance. During Phase 2, participants were assigned either to a no-AI control condition or to an AI-access condition with experimentally varied AI request costs.A horizontal timeline shows the three-phase study progressing from a pre-assessment to an experimental phase and then a post-assessment. During the middle phase, participants are randomly assigned to one of three conditions: no AI assistance, high-cost AI assistance, or low-cost AI assistance. 2. Related Work Although a large body of work examines human-AI complementarity in joint decision-making contexts (Steyvers et al., 2022; Zhang et al., 2022; De Toni et al., 2024), recent studies have begun to investigate how AI assistance influences subsequent performance once it is no longer available. A common approach uses pre- and post-assessment designs to compare behavior before AI exposure, during AI-assisted work, and after assistance is removed. In mathematical reasoning and reading-comprehension tasks, Liu et al. (2026) find that AI assistance improves short-run performance but reduces persistence and impairs later independent performance. In programming, Shen and Tamkin (2026) show that AI assistance can weaken conceptual understanding, code reading, and debugging skills, while also documenting cognitively engaged interaction patterns that preserve learning. Related evidence from writing tasks suggests that LLM assistance may reduce cognitive engagement and weaken later unaided performance (Kosmyna et al., 2025). These findings are consistent with broader work on AI as a ātool for thoughtā, which emphasizes that AI systems can either scaffold reasoning or automate cognitive steps that users would otherwise perform themselves (Tankelevitch et al., 2025). Together, these studies suggest that AI-assisted success does not necessarily translate into durable skill acquisition, and that AIās effects on human learning are context-dependent. Our work adds to this growing literature by focusing on whether learners continue to spend time on independent problem-solving during periods when AI assistance is available, and how this independent effort relates to subsequent skill development. One mechanism linking AI assistance to skill development is cognitive offloading: the use of external tools to reduce internal cognitive effort (Risko and Gilbert, 2016). Offloading can improve immediate performance, but it may also reduce learning when externally supported operations replace the reasoning, memory, or problem-solving processes through which skill is acquired (Grinschgl et al., 2021; Sparrow et al., 2011). This connects directly to work on AI reliance, which studies when users request, accept, reject, or defer to AI assistance (Madhavan and Wiegmann, 2007; Hoff and Bashir, 2015). Prior work shows that reliance depends on features of the task environment, such as AI accuracy, timing, and time pressure (Cao et al., 2023; Nourani et al., 2020; Swaroop et al., 2024). Reliance, however, is not necessarily harmful: on-demand assistance can act as a cognitive forcing mechanism that reduces over-reliance (BuƧinca et al., 2021), and learning-oriented assistance can be beneficial when it scaffolds understanding rather than allowing users to bypass the task (Gajos and Mamykina, 2022; Baker et al., 2004). Thus, a key distinction is not simply whether users rely on AI, but whether AI use preserves or displaces the cognitive work needed for learning. Prior work also suggests that the consequences of AI assistance depend on how the system changes the userās role in the task. In chess, Poulidis et al. (2025) show that the effects of AI assistance depend on whether it prescribes actions or instead directs usersā attention, reinforcing the distinction between AI that substitutes for vs. complements human reasoning. In our earlier work (Wu et al., 2026), we explored the role of AI informativeness and usage in logical reasoning. In this paper we build on this framework to focus more directly on the behavioral mechanisms linking AI assistance to skill development. Specifically, we distinguish between the frequency and timing of AI requests and the extent to which participants preserve independent problem-solving effort when assistance is available. In this context, we develop a Bayesian latent ability model that allows us to disentangle the different factors affecting skill development over time. This allows us to examine whether weaker subsequent learning is better explained by AI use itself or by the displacement of independent reasoning during the AI-access phase. 3. Experiment To understand how AI assistance affects human skill development, we conducted a controlled user study in the specific context of solving logic puzzles under time constraints, where participants could autonomously request AI assistance while solving problems. By experimentally varying the cost of AI requests, the design allows us to examine how different help-seeking environments shape AI engagement, task performance, and learning. 3.1. Task Description We build on a logic-puzzle paradigm used in prior work on AI-assisted skill development (Wu et al., 2026), but modify the AI assistance format to allow for cost-driven help-seeking. In each problem, participants were presented with six objects and a set of five logical constraints that jointly define a unique ordering (Figure 2(a)). Participants solved the problem by arranging the six objects according to these constraints. An answer was deemed correct (from the userās perspective) only if all six objects were placed in the correct positions. Participants could submit up to two attempts per problem. After the first attempt, participants received feedback indicating the number of correctly positioned objects, but not which specific objects were correctly positioned. Given this feedback, participants could either revise and resubmit their response or accept the initial attempt as final and proceed to the next problem. The correct solution was revealed after each problem to facilitate learning across problems. While more open-ended and more realistic tasks could in principle be used in a study like this, this more controlled setting reduces variability in terms of prior human knowledge and supports a clearer focus on problem-solving behavior under optional AI assistance. The task was designed to be intuitive for participants without specialized training, while still requiring logical reasoning under time constraints, reflecting decision settings in which both speed and accuracy are important. In addition, some objects contained recurring visual markers that were associated with specific positions, providing hidden structural cues that participants could learn to exploit across problems. The study employed a three-phase design that separates baseline performance, AI-assisted task completion, and post-assistance performance, allowing us to examine how AI request costs relate to AI usage patterns and subsequent skill development (Figure 1). Phase 1 served as a pre-AI assessment and lasted 8 minutes or until participants completed at least four problems. Phase 2 lasted 20 minutes and introduced the experimental conditions, including optional AI assistance under different costs as described below in Section 3.2. Phase 3 mirrored Phase 1 as a post-AI assessment without AI assistance. Problem order was randomized within each phase to mitigate confounding effects of problem difficulty on learning. Participants earned one point for each correctly solved problem. During Phase 2, points were deducted for each AI request according to the assigned condition. Bonus payments were calculated at $0.15 per point in Phases 1 and 2 (up to $2 total) and $0.18 per point in Phase 3 (up to $1.5). (a) Without AI assistance (b) With AI assistance Figure 2. User-interface examples from the logic-based puzzle taskExamples of two user interfaces for the logic-based puzzle task, consisting of 6 objects. (a) The user interface without AI assistance; (b) The user interface with AI assistance, identical to (a) except for an added dialog on the left that displays the AI suggestion as text and displays the location of a randomly chosen object on the screen. 3.2. Conditions We used a between-subjects design in which participants were randomly assigned to one of three conditions that varied the availability and cost of AI assistance during Phase 2. AI assistance was available only in Phase 2, and participants in AI-available conditions could request it at any time while solving the puzzles. Each request revealed the location of one randomly selected object and incurred a condition-specific point deduction, creating a trade-off between obtaining help and maximizing rewards. Participants were informed of the applicable AI request cost structure in the instructions before starting the study. Figure 2(b) shows the user interface for an AI-enabled condition. This interface is representative of all AI-enabled conditions, which share the same interaction design and differ only in the cost of AI requests. Participants could click āGet AI helpā to reveal the location of a randomly selected object in the puzzle. Each request updated the right-hand panel by incrementing the number of assists and the deducted points. Participants were randomly assigned to one of the following conditions: ā¢ No-AI: No AI assistance was available, and the AI panel was not shown (see Figure 2(a)). ā¢ Low-cost AI: Each AI request incurred a 0.1-point deduction. ā¢ High-cost AI: Each AI request incurred a 0.18-point deduction. To ensure that variation in AI quality did not confound the effect of AI cost on participantsā help-seeking behavior, we fixed the simulated AI agent to be perfectly correct (100%). Under this design, each AI request consistently returned the correct location of a randomly selected object, allowing us to attribute behavioral differences across AI-access conditions to request cost rather than output quality. This design choice aligns with prior work in humanāAI interaction that uses simulated agents to isolate specific behavioral mechanisms (e.g., (Srivastava et al., 2022; BuƧinca et al., 2021; Gajos and Mamykina, 2022; Vasconcelos et al., 2023)). Participants were not informed of the AIās accuracy and were instructed only that AI assistance would be available during Phase 2 at a cost. 3.3. Procedure After providing informed consent, participants were randomly assigned to one of the three conditions and presented with task instructions. To ensure task understanding, participants completed two comprehension checks, and only those who passed both proceeded to the main study. The main study consisted of three sequential phases, separated by short breaks, during which participants completed a set of logic puzzle problems under their assigned condition. To monitor data quality during task performance, an attention check was embedded in Phase 2. After completing all phases, participants completed a short post-study survey about their experience, perceived effort, strategy use, prior task familiarity, and, when applicable, their perceptions and timing of AI use (see Supplementary Material B.1 for details). 3.4. Participants We recruited 150 English-speaking adults (18+) via Prolific, restricted to adults residing in the U.S. with at least an undergraduate degree. We excluded 26 participants: 1 failed the attention check, 14 showed inattentiveness (off-screen for ā„5ā„ 5 seconds on over three problems), 8 had inconsistent self-reported and logged AI use, 2 were extreme time outliers, and 1 showed low-effort responses with near-zero accuracy.11 1 Off-screen time is total problem time minus active on-screen time. Inconsistent AI-use reporting refers to disagreement between post-study reports and usage logs. The low-effort participant expressed dislike for the study and answered randomly throughout. The final sample consisted of 124 participants (42 No-AI, 43 Low-cost AI, 39 High-cost AI). Participants received $9 for participation (median completion time: 56 minutes) and could earn up to an additional $3.50 in performance-based bonuses. The study was approved by the Institutional Review Board (IRB) at the University of California, Irvine (protocol #7206). The final sample consisted of 124 participants with a mean age of 40 years (SD = 12). The sample included 59 male participants, 64 female participants, and 1 participant who preferred not to disclose gender. In terms of education, 66 participants held an undergraduate degree and 58 held a masterās degree or higher. 3.5. Measures We study both observed task performance and latent skill development. For observed performance, we use accuracy, defined as the number of objects (0ā6) placed in the correct position for a problem, and response time, measured in seconds. Both are computed per participant for each of the 3 phases, averaging over the different problems that a participant attempted during a phase. To summarize the speed-accuracy tradeoff, we also compute reward rate, as used in cognitive modeling (Standage et al., 2015). For each participant in each phase, reward rate is defined as the average accuracy divided by the average response time, in units of correct objects per minute, where higher values indicate better performance. To characterize reliance on AI during Phase 2, we measure AI usage as the total number of assistance requests made by a participant. We also measure independent problem-solving effort using solo share. At the problem level, this measure is defined as the fraction of response time spent working independently before requesting AI assistance. If no AI assistance is requested, the time until final submission is counted as the time working independently. Participant-level solo share is then defined as the average problem-level solo share across Phase 2 problems. In Section 4.1 we use reward rate, accuracy, and response time to empirically summarize performance, to characterize AI reliance behavior, and to examine whether Phase 2 performance predicts subsequent unassisted performance differently across levels of AI use. Our Bayesian analyses in Section 4.2 use Phase 1 and Phase 3 performance as observed signals of latent ability, and use Phase 2 solo share as the main behavioral measure to predict the final (Phase 3) skill levels of participants, both with and without AI assistance. 4. User Study Results We begin below with a high-level summary of results from our user study, followed by a more detailed Bayesian analysis to quantify the effect of AI assistance on skill development at both individual and population-wide levels. 4.1. Empirical Findings Skill Development: To examine whether participants in general showed task-specific skill improvement over time, we show in Figure 3 the average reward rate across phases for participants who did not request any AI assistance. The unassisted task performance of this set of participants improved substantially from the pre-AI assessment (Phase 1) to the post-AI assessment (Phase 3). The mean reward rate increased from 2.03 correct objects per minute in Phase 1 to 3.86 in Phase 3, corresponding to a 90.2% increase in the number of correct objects per minute (N=75N=75, t=7.40t=7.40, p<0.01p<0.01), illustrating substantial short-term improvement in participantsā skills on this task. Effects of AI-cost condition: We next examine the effects of the randomized AI-cost manipulation. Phase 1 reward rates were well balanced across conditions (ANOVA: F=0.20F=0.20, p=0.82p=0.82). As shown in Figure 4, participants in the Low-cost AI condition made significantly more AI requests during Phase 2 than those in the High-cost AI condition (6.67 vs. 3.33 requests; one-sided Welch test: t=1.48t=1.48, p<0.10p<0.10). From Phase 1 to Phase 3, reward rate increased by 1.95 in the No-AI condition, 1.66 in the High-cost AI condition, and 1.32 in the Low-cost AI condition; the increase was smaller in the Low-cost AI than in the No-AI condition in a one-sided t-test (t=1.30t=1.30, p<0.10p<0.10). In Phase 3, accuracy did not differ across conditions (ANOVA: F=0.097F=0.097, p=0.91p=0.91), while response times were longer in the Low-cost AI condition (M=111.81M=111.81s, SāE=6.13SE=6.13) than in the High-cost AI (M=86.78M=86.78s, SāE=4.18SE=4.18; p<0.01p<0.01) or No-AI condition (M=92.21M=92.21s, SāE=4.88SE=4.88; p=0.01p=0.01). Post-AI Performance: Descriptively, participants who requested AI assistance in Phase 2 had a lower average reward rate (3.42 correct items/minute) in Phase 3 (where no AI assistance was available) compared to the average reward rate in Phase 3 (3.86 correct items/minute) of participants who did not request AI in Phase 2. To explore this further, we use a regression model to predict Phase 3 reward rate as a function of Phase 2 reward rate and find that predictions are systematically biased by AI use. Specifically, as shown in Figure 5, for participants who did not request AI, Phase 3 performance was underestimated by 0.15 reward-rate units on average. In contrast, for AI users, Phase 3 performance was overestimated by 0.22 reward-rate units on average. In other words, the apparent performance of participants while using AI (in Phase 2) systematically overestimates their subsequent performance when conducting the task on their own without AI assistance (in Phase 3). These patterns suggest that Phase 2 AI-use behavior is associated with differences in short-term skill improvement from the pre-AI phase (Phase 1) to the post-AI phase (Phase 3). We investigate these relationships further below via a Bayesian modeling approach, separating initial ability, post-AI ability, and latent skill change while incorporating Phase 2 AI usage behavior. Figure 3. Reward rate across phases for participants with zero AI use in Phase 2. Labels 2-1 and 2-2 denote the first and second halves of Phase 2. Error bars indicate standard errors.A line plot shows reward rate increasing steadily across the four displayed time points, from 2.03 in Phase 1 to 3.86 in Phase 3, with intermediate increases during the two halves of Phase 2. A two-bar chart shows that participants in the low-cost AI condition made roughly twice as many AI requests as participants in the high-cost AI condition, 6.67 vs. 3.33 requests on average. Figure 4. Average AI usage in Phase 2 by AI cost condition. Error bars indicate standard errors.A two-bar chart shows that participants in the low-cost AI condition made roughly twice as many AI requests as participants in the high-cost AI condition, 6.67 vs. 3.33 requests on average. An error-bar plot compares prediction residuals for non-AI users and AI users. The average residual is 0.15 for non-AI users and -0.22 for AI users, indicating opposite directions of prediction error across the two groups. Figure 5. Residuals by AI-use status. Residuals are defined as observed minus predicted Phase 3 reward rate, yāy^y- y; error bars indicate standard errors.An error-bar plot compares prediction residuals for non-AI users and AI users. The average residual is 0.15 for non-AI users and -0.22 for AI users, indicating opposite directions of prediction error across the two groups. 4.2. Bayesian Analysis We use a Bayesian model to infer latent individual ability and skill development, conditioned on observed individual-level data (accuracy, response time, AI usage, solo share) across the three experiment phases. This approach follows a long tradition in education and psychometrics in modeling ability as a latent trait inferred from observed performance (Lord and Novick, 2008; Embretson and Reise, 2025). It is useful in our setting because observed performance is a noisy proxy for ability: accuracy and response time provide complementary signals, and the same observed Phase 3 performance can correspond to different amounts of learning depending on participantsā Phase 1 baseline performance and Phase 2 AI reliance behavior. A Bayesian latent ability framework allows us to separate initial ability from post-AI ability and participant-level skill change, while propagating uncertainty in both individual- and population-level quantities. In particular, we use the model to infer each participantās individual latent initial ability, latent post-AI ability, and latent skill change, as well as to estimate how short-term skill change is associated with independent problem-solving effort in Phase 2. We outline below the structure of the model in correspondence with the three phases of the experiment. We standardize all observed performance measures prior to fitting the model to improve the interpretability of coefficients across variables measured on different scales (Gelman et al., 2013; Gelman and Hill, 2007), and we use weakly informative priors as outlined in the Supplementary Material A.1. 4.2.1. A Bayesian latent ability model of short-term skill development, with and without AI assistance Beginning with Phase 1 (pre-AI, initial assessment), for individual i, let Īøiā1 _i1 denote latent initial ability, with a prior Īøiā1ā¼ā”(0,1) _i1 (0,1). We model an individualās Phase 1 outcomes (accuracy Aiā1A_i1 and log response time Riā1R_i1) as conditionally independent noisy functions of their latent ability Īøiā1 _i1: (1) Aiā1|Īøiā1,Ī¼A,ĻAā¼ā”(Ī¼A+Īøiā1,ĻA) A_i1 _i1, _A, _A ( _A+ _i1, _A) Riā1|Īøiā1,Ī¼R,ĻRā¼ā”(Ī¼RāĪøiā1,ĻR) R_i1 _i1, _R, _R ( _R- _i1, _R) where the effect of Īøiā1 _i1 is reversed for the two variables (in terms of the signs in the mean terms) to reflect the expectation that higher values of ability will correspond (on average) to higher accuracy and to lower response times. Here, Ī¼A _A and Ī¼R _R are measurement intercepts for accuracy and log response time, respectively, and ĻA _A and ĻR _R capture residual measurement variation. Because Aiā1A_i1 and Riā1R_i1 are phase-level measures aggregated over the problems completed by each participant, the model does not separately estimate problem-level effects. Problems were designed to be comparable in difficulty and their order was randomized within each phase, so any remaining variation in problem difficulty, order, or the number of problems completed is treated as part of the residual measurement variation. These population-level parameters are inferred jointly during Bayesian inference along with the individual-level latent abilities and skill changes introduced below. For Phase 2, where AI is available to some participants, we use solo share SoloiSolo_i as the observed behavioral measure in our main model, where SoloiSolo_i is the participant-level fraction of Phase 2 problem-solving time that participant i spends reasoning independently prior to (or without) AI assistance (see Section 3.5). We model participant-specific latent skill change, Ī“i _i, as a function of initial ability and the Phase 2 engagement measure SoloiSolo_i: (2) Ī“i|Īøiā1,Soloiā¼ā”(Ī±0+Ī±ĪøāĪøiā1+Ī±soloāSoloi,ĻĪ“). _i _i1,Solo_i ( _0+ _Īø _i1+ _soloSolo_i,\ _Ī“ ). Since Ī“i _i is not constrained to be positive, the model can accommodate either skill improvement or skill decline after AI assistance. The intercept Ī±0 _0 represents the expected latent skill change for a participant with average initial ability and average solo share. The coefficient Ī±Īø _Īø allows latent skill change to vary with initial ability, accounting for the possibility that lower- and higher-ability participants have different room to improve or learn at different rates (e.g., allowing for saturation effects for individuals with very high initial ability). The Ī±solo _solo coefficient captures the association of independent reasoning with latent skill change, conditioned on initial ability. Note that because solo share reflects observed participant behavior rather than a randomized treatment, this coefficient should be interpreted as associational rather than causal. The parameter ĻĪ“ _Ī“ reflects residual heterogeneous variability in latent skill change that is not explained by initial ability or solo share. An alternative option would be to model latent skill change using AI usage as a Phase 2 behavioral predictor, either in addition to or instead of solo share. We use solo share in the main model because, as shown later, it led to better held-out predictive performance than AI request frequency. For Phase 3, where AI is not available (as in Phase 1), let Aiā3A_i3 denote Phase 3 accuracy and Riā3R_i3 denote Phase 3 log response time. Phase 3 outcomes are modeled analogously to Phase 1, i.e., as noisy functions of an individualās latent Phase 3 ability Īøiā3 _i3: (3) Aiā3|Īøiā3,Ī¼A,ĻAā¼ā”(Ī¼A+Īøiā3,ĻA) A_i3 _i3, _A, _A ( _A+ _i3, _A) Riā3|Īøiā3,Ī¼R,ĻRā¼ā”(Ī¼RāĪøiā3,ĻR) R_i3 _i3, _R, _R ( _R- _i3, _R) We model an individualās latent ability Īøiā3 _i3 as the deterministic sum of their initial ability Īøiā1 _i1 offset by the individualās change in ability Ī“i _i, i.e., Īøiā3=Īøiā1+Ī“i _i3= _i1+ _i, where posterior uncertainty about Īøiā3 _i3 is driven by uncertainty about Īøiā1 _i1 and Ī“i _i. Figure 6. Bayesian model of latent skill development. Phase 1 and Phase 3 accuracy and response time are modeled as noisy measurements of latent abilities Īøiā1 _i1 and Īøiā3 _i3. Participant-specific skill change Ī“i _i depends on initial ability and solo share. Shaded nodes are observed variables, unshaded nodes are latent variables or unknown parameters, and the plate denotes participants i=1,ā¦,Ni=1,ā¦,N.A plate diagram shows the Bayesian model linking initial latent ability to Phase~1 accuracy and response time, and linking Phase~3 latent ability to the corresponding Phase~3 measures. Initial ability and Phase~2 solo share jointly predict participant-specific skill change, which in turn connects initial ability to post-AI ability. We summarize the overall structure of the Bayesian model as a directed graphical model in Figure 6, where shaded nodes denote observed variables and unshaded nodes denote latent variables and unknown parameters in the model. The nodes inside the plate reflect replication over the N participants, where nodes inside the plate are at the individual level (with subscript i) and are independent (across the N individuals) conditioned on the population-level parameters outside the plate. The accuracy A and response time R in Phases 1 and 3 depend on the corresponding latent abilities Īøiā1 _i1 and Īøiā3 _i3 for these Phases. The dependence of Īøiā3 _i3 on Ī“i _i and Īøiā1 _i1 represents the latent transition from initial ability to post-AI ability. Finally, latent ability change, Ī“i _i, is modeled as a function of initial ability Īøiā1 _i1 and solo share, allowing the model to capture associations between individual skill development and the amount of independent reasoning participants engage in during Phase 2. The graphical model represents the probabilistic dependencies assumed in the model and is not intended as a causal model of how Phase 2 behavior affects skill change. We note that the model does not include variables for observed participant Phase 2 accuracy Aiā2A_i2 or response time Riā2R_i2. We omit these measures to keep the specification focused on whether Phase 2 independent reasoning is associated with subsequent latent skill change beyond initial ability, rather than modeling Phase 2 task performance directly. Given the model described above, Bayesian inference proceeds by conditioning on =Aiā1,Aiā3,Riā1,Riā3,Soloii=1ND=\A_i1, A_i3, R_i1, R_i3, _i\_i=1^N, and estimating the joint posterior distribution over all unknown quantities. Because the likelihood and priors do not yield a conjugate closed-form posterior, we approximate the posterior using Markov Chain Monte Carlo (MCMC) (see implementation details in Supplementary Material A.2). Standard MCMC diagnostics indicated satisfactory convergence, and we additionally assess model fit using posterior predictive checks and the uncertainty of held-out predictive comparisons; details are reported in Supplementary Material A.3. We note that the randomized AI-cost condition (low-cost, high-cost) affects participantsā opportunities and incentives to request assistance, and this condition could be modeled as an upstream parent of solo share in the model. Because the primary specification conditions directly on this observed behavioral measure, we omit the condition node from the graphical model in Figure 6 for simplicity. In Supplementary Material A.4 and Supplementary Material A.5, we report the results of alternative model specifications that add AI-cost condition controls and that replace solo share with AI usage, and find that these alternative model specifications do not add any additional explanatory power to the model. Participant-specific posterior estimates of latent skill change are ordered from smallest to largest. Most posterior modes lie to the right of the zero-change line, while many credible intervals are wide, showing substantial uncertainty and heterogeneity across participants. Figure 7. Posterior modes and 95% credible intervals of participant-specific latent ability changes, Ī“i _i, ordered by posterior mode. The vertical dashed line indicates zero change.Participant-specific posterior estimates of latent skill change are ordered from smallest to largest. Most posterior modes lie to the right of the zero-change line, while many credible intervals are wide, showing substantial uncertainty and heterogeneity across participants. 4.2.2. Independent reasoning is more predictive of short-term skill development than AI request frequency We summarize the results obtained from the posterior samples after running MCMC inference for the model described in Figure 6. Figure 7 visualizes the posterior modes and 95% credible intervals of participant-specific latent ability changes, Ī“i _i. Almost all participant-level posterior modes are positive, indicating positive estimated skill development for most participants. This is consistent with the earlier empirical observations of skill development at the population level in Figure 3. The credible intervals are wide, indicating substantial uncertainty and heterogeneity in individual-level skill development, but nonetheless broadly indicate positive skill gains. (a) Ī±0 _0: Average expected skill change (b) Ī±Īø _Īø: Impact of initial ability on skill change (c) Ī±solo _solo: Solo share Figure 8. Posterior distributions of latent ability regression coefficients. Histograms show posterior draws; orange solid lines denote posterior means, orange dotted lines denote 95% credible intervals, and black dashed lines mark zero.Three posterior histograms summarize the population-level coefficients. Average expected skill change is clearly positive, the coefficient on initial ability is clearly negative, and the coefficient on solo share is positive, with its lower credible bound remaining slightly above zero. The posterior distributions of the three population-level coefficients in the latent ability model are shown in Figure 8. The posterior distribution of the average expected latent skill change, Ī±0 _0, is positive: for a participant with average initial latent ability and average solo share, the expected latent ability change is Ī±0=0.668 _0=0.668, with 95% CI [0.485,0.851][0.485,0.851]. Because latent ability is measured jointly through accuracy and response time, a positive gain corresponds to improved accuracy and/or faster responses in the post-AI assessment. Figure 8(b) shows that the coefficient Ī±Īø _Īø linking latent skill change to initial ability is strongly negative, with an expected posterior value of ā0.531-0.531 and 95% CI [ā0.756,ā0.307][-0.756,-0.307], with posterior probability Pā”(Ī±Īø<0)=1.000P( _Īø<0)=1.000. This indicates that participants with lower initial latent ability tend to show larger gains, consistent with greater room for improvement among initially lower-performing participants. Importantly, Figure 8(c) shows that solo share is positively associated with latent skill change. The posterior mean of Ī±solo _solo is 0.1250.125, with 95% CI [0.002,0.246][0.002,0.246] and posterior probability Pā”(Ī±solo>0)=0.977P( _solo>0)=0.977. Thus, participants who spent more time independently solving problems (without AI assistance) during Phase 2 exhibited larger gains in latent ability from the pre-AI to the post-AI assessment. To interpret the magnitude of this association, we compute a model-implied contrast between two otherwise average participants whose Phase 2 solo share differs by one minute out of the 20-minute AI-access phase. Relative to the observed population mean value of solo share, this one-minute difference corresponds to a 1.5% increase in predicted Phase 3 accuracy and a 2.5% reduction in predicted response time. Together, these results suggest that, in this context, the key behavioral distinction is not how often participants use AI, but how much independent reasoning they preserve while using it. Participants who preserve more independent reasoning show greater short-term skill gains, consistent with weaker skill development when AI assistance is accompanied by greater cognitive offloading. As an alternative specification, we also fit a model that replaces solo share with AI usage, measured as the total number of assistance requests during Phase 2 (see details in Supplementary Material A.5). We find that AI usage is not associated with latent skill change after accounting for initial ability, with a posterior for Ī±usage _usage centered at a posterior mean of 0.00040.0004, 95% CI [ā0.122,0.124][-0.122,0.124], and Pā”(Ī±usage>0)=0.501P( _usage>0)=0.501. Thus, in this setting, AI request frequency alone does not appear to explain latent skill change. The results are consistent with the hypothesis that time spent on independent reasoning is more predictive of skill development than simple frequency of AI use. We further compare the main solo-share model discussed above against three alternative specifications: a solo-share model with direct AI-cost condition controls, a usage-only model that replaces solo share with AI usage (the count of the total number of AI requests), and a constant-change baseline in which all participants share the same expected latent skill change, i.e., Ī“i=Ī±0 _i= _0, with no participant-level effects for latent skill change. We evaluate the four specifications using 5-fold cross-validation at the participant level. In each fold, the model is fit on four-fifths of participants and used to predict Phase 3 accuracy and log response time for the remaining participants based on their Phase 1 performance and the Phase 2 predictors included in the corresponding model. We evaluate predictive performance using the total held-out log-likelihood, where higher values indicate better overall predictive fit, and the mean-squared error (MSE) for standardized Phase 3 accuracy and log response time, where lower values indicate better point prediction. Table 1 summarizes the results. The solo-share model achieves the best predictive performance across all three metrics, with the highest held-out log-likelihood and the lowest MSE for both accuracy and log response time. Additional details of the evaluation procedure are provided in Supplementary Material A.6. Table 1. Five-fold held-out prediction of Phase 3 unassisted performance across Bayesian latent-ability models. The main model uses Phase 2 solo share to predict latent skill change; the condition model adds AI-cost indicators; the AI-usage model replaces solo share with total AI requests; and the constant-change baseline assumes identical expected skill change across participants. Test LogL is the total held-out log predictive likelihood (higher is better). MSE is the mean squared prediction error for standardized Phase 3 accuracy and log response time (lower is better). Model Test LogLā Accuracy MSEā Log RT MSEā Solo share (main) -329.94 0.809 0.850 Solo share + AI-cost conditions -331.45 0.830 0.854 AI usage -332.96 0.831 0.882 Constant change (baseline) -339.02 0.893 0.915 5. Discussion and Conclusion This study examines the role of AI assistance in short-term, task-specific skill development. In a controlled three-phase logic-puzzle experiment, we find that lower-cost AI requests result in increased AI use, and that participants who request AI assistance perform worse after AI assistance is removed. Our Bayesian latent ability model further suggests that skill development is more closely associated with the amount of independent reasoning participants preserve during the AI-access phase than with AI request frequency itself. These findings contribute to the growing literature on AI-enabled deskilling, cognitive offloading, and learning. Prior research shows that external support can improve immediate performance while reducing the cognitive effort needed for later memory, reasoning, or skill formation (Risko and Gilbert, 2016; Grinschgl et al., 2021; Sparrow et al., 2011). More recent studies similarly raise concerns that AI assistance can reduce persistence, weaken conceptual understanding, or encourage cognitive disengagement (Shen and Tamkin, 2026; Kosmyna et al., 2025; Dizon et al., 2026). Our results suggest that, in this setting, the central issue is not necessarily AI use per se, but how much independent thinking participants preserve while AI assistance is available. The practical implication is that AI systems should be designed not only to improve immediate performance, but also to preserve usersā active engagement. Interfaces that encourage an initial attempt, delay assistance, provide partial hints, or prompt reflection may help users benefit from AI without fully offloading the independent reasoning process. This study also points to several directions for future work. Our experiment uses a simulated AI assistant and focuses on short-term skill development in a controlled logic-puzzle task, which helps isolate the relationship between AI use, independent reasoning, and subsequent unassisted performance. Future studies could examine whether similar patterns persist over longer time horizons, in larger and more diverse samples, and in more realistic domains such as education, professional training, and other high-stakes settings. It would also be useful to study how variation in AI accuracy, explanation quality, and interaction design affects both immediate performance and longer-term skill development. In addition, because the Bayesian model uses participant-level phase aggregates rather than problem-level responses, it does not separately identify residual differences in problem difficulty or problem exposure, providing another potential dimension for future studies. In conclusion, our work suggests that weaker short-term skill development is associated with greater AI use, particularly when that use involves less independent reasoning. In the context of skill development, designing AI systems that support rather than replace human thinking may therefore be important for improving not only immediate task performance but also subsequent unassisted performance. Acknowledgements. We thank the reviewers for their valuable feedback that helped improve this paper. This work was supported by the National Science Foundation under awards NSF 2505006 and NSF IIS-2046873, by the Hasso Plattner Institute (HPI) Research Center in Machine Learning and Data Science at UCI, and by funding support from SAP. References Anderson et al. (1995) J. R. Anderson, A. T. Corbett, K. R. Koedinger, and R. Pelletier Cognitive tutors: lessons learned. The Journal of the Learning Sciences. 4 (2), p. 167ā207. Cited by: Ā§1. Baker et al. (2004) R. S. Baker, A. T. Corbett, K. R. Koedinger, and A. Z. Wagner Off-task behavior in the cognitive tutor classroom: when students āgame the system". In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p. 383ā390. Cited by: Ā§2. BuƧinca et al. (2021) Z. BuƧinca, M. B. Malaya, and K. Z. Gajos To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5 (CSCW1), p. 1ā21. Cited by: Ā§1, Ā§2, Ā§3.2. Cao et al. (2023) S. Cao, C. Gomez, and C. Huang How time pressure in different phases of decision-making influences human-AI collaboration. Proceedings of the ACM on Human-Computer Interaction 7 (CSCW2), p. 1ā26. Cited by: Ā§2. De Toni et al. (2024) G. De Toni, N. Okati, S. Thejaswi, E. Straitouri, and M. Rodriguez Towards human-ai complementarity with prediction sets. Advances in Neural Information Processing Systems 37, p. 31380ā31409. Cited by: Ā§2. Dizon et al. (2026) J. I. W. T. Dizon, N. B. Mendoza, D. Gasevic, and F. A. Ganotice Assessing AI-driven metacognitive offloading: initial development and validation of the metacognitive laziness scale. ECNU Review of Education 9 (2). External Links: Document Cited by: Ā§5. Embretson and Reise (2025) S. E. Embretson and S. P. Reise Item response theory: foundations for psychologists and social scientists. Taylor & Francis. Cited by: Ā§4.2. Gajos and Mamykina (2022) K. Z. Gajos and L. Mamykina Do people engage cognitively with AI? impact of AI assistance on incidental learning. In Proceedings of the 27th International Conference on Intelligent User Interfaces, p. 794ā806. Cited by: Ā§1, Ā§2, Ā§3.2. Gelman et al. (2013) A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin Bayesian data analysis. 3 edition, CRC Press. Cited by: Ā§4.2. Gelman and Hill (2007) A. Gelman and J. Hill Data analysis using regression and multilevel/hierarchical models. Cambridge University Press. Cited by: Ā§4.2. Gelman (2006) A. Gelman Prior distributions for variance parameters in hierarchical models (comment on article by browne and draper). Bayesian Analysis 1 (3), p. 515 ā534. External Links: Document, Link Cited by: Ā§A.1. Grinschgl et al. (2021) S. Grinschgl, F. Papenmeier, and H. S. Meyerhoff Consequences of cognitive offloading: boosting performance but diminishing memory. Quarterly Journal of Experimental Psychology 74 (9), p. 1477ā1496. Cited by: Ā§2, Ā§5. Hoff and Bashir (2015) K. A. Hoff and M. Bashir Trust in automation: integrating empirical evidence on factors that influence trust. Human Factors 57 (3), p. 407ā434. Cited by: Ā§2. Hoffman et al. (2014) M. D. Hoffman A. Gelman et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo.. J. Mach. Learn. Res. 15 (1), p. 1593ā1623. Cited by: Ā§A.2. Karny et al. (2024) S. Karny, L. W. Mayer, J. Ayoub, M. Song, H. Su, D. Tian, E. Moradi-Pari, and M. Steyvers Learning with AI assistance: a path to better task performance or dependence?. In Proceedings of the ACM Collective Intelligence Conference, p. 10ā17. Cited by: Ā§1. Kosmyna et al. (2025) N. Kosmyna, E. Hauptmann, Y. T. Yuan, J. Situ, X. Liao, A. V. Beresnitzky, I. Braunstein, and P. Maes Your brain on ChatGPT: accumulation of cognitive debt when using an AI assistant for essay writing task. arXiv preprint arXiv:2506.08872 4. Cited by: Ā§2, Ā§5. Lee et al. (2025) H. Lee, A. Sarkar, L. Tankelevitch, I. Drosos, S. Rintel, R. Banks, and N. Wilson The impact of generative AI on critical thinking: self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, p. 1ā22. Cited by: Ā§1. Lehmann et al. (2024) M. Lehmann, P. B. Cornelius, and F. J. Sting AI meets the classroom: when does ChatGPT harm learning?. arXiv preprint arXiv:2409.09047. Cited by: Ā§1. Liu et al. (2026) G. Liu, B. Christian, T. Dumbalska, M. A. Bakker, and R. Dubey AI assistance reduces persistence and hurts independent performance. arXiv preprint arXiv:2604.04721. Cited by: Ā§1, Ā§2. Lord and Novick (2008) F. M. Lord and M. R. Novick Statistical theories of mental test scores. IAP. Cited by: Ā§4.2. Madhavan and Wiegmann (2007) P. Madhavan and D. A. Wiegmann Effects of information source, pedigree, and reliability on operator interaction with decision support systems. Human Factors 49 (5), p. 773ā785. Cited by: Ā§2. Nourani et al. (2020) M. Nourani, J. King, and E. Ragan The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8, p. 112ā121. Cited by: Ā§2. Poulidis et al. (2025) S. Poulidis, H. Ge, H. Bastani, and O. Bastani Action vs. attention signals for human-AI collaboration: evidence from chess. The Wharton School Research Paper. Cited by: Ā§1, Ā§2. Risko and Gilbert (2016) E. F. Risko and S. J. Gilbert Cognitive offloading. Trends in Cognitive Sciences 20 (9), p. 676ā688. Cited by: Ā§1, Ā§2, Ā§5. Shen and Tamkin (2026) J. H. Shen and A. Tamkin How AI impacts skill formation. arXiv preprint arXiv:2601.20245. Cited by: Ā§1, Ā§2, Ā§5. Sparrow et al. (2011) B. Sparrow, J. Liu, and D. M. Wegner Google effects on memory: cognitive consequences of having information at our fingertips. Science 333 (6043), p. 776ā778. Cited by: Ā§2, Ā§5. Srivastava et al. (2022) D. K. Srivastava, J. M. Lilly, and K. M. Feigh Improving human situation awareness in AI-advised decision making. In 2022 IEEE 3rd International Conference on Human-Machine Systems (ICHMS), p. 1ā6. Cited by: Ā§3.2. Stan Development Team (2024) Stan Development Team Stan userās guide. External Links: Link Cited by: Ā§A.2. Stan Development Team (2025) Stan Development Team Prior choice recommendations. Note: Stan Wiki Cited by: Ā§A.1. Standage et al. (2015) D. Standage, D. H. Wang, R. P. Heitz, and P. Simen Toward a unified view of the speed-accuracy trade-off. Frontiers in Neuroscience 9, p. 139. Cited by: Ā§3.5. Steyvers et al. (2022) M. Steyvers, H. Tejeda, G. Kerrigan, and P. Smyth Bayesian modeling of human-AI complementarity. Proceedings of the National Academy of Sciences 119 (11), p. e2111547119. Cited by: Ā§2. Swaroop et al. (2024) S. Swaroop, Z. BuƧinca, K. Z. Gajos, and F. Doshi-Velez Accuracy-time tradeoffs in AI-assisted decision making under time pressure. In Proceedings of the 29th International Conference on Intelligent User Interfaces, p. 138ā154. Cited by: Ā§2. Tankelevitch et al. (2025) L. Tankelevitch, E. L. Glassman, J. He, A. Kittur, M. Lee, S. Palani, A. Sarkar, G. Ramos, Y. Rogers, and H. Subramonyam Understanding, protecting, and augmenting human cognition with generative ai: a synthesis of the CHI 2025 tools for thought workshop. arXiv preprint arXiv:2508.21036. Cited by: Ā§2. Vasconcelos et al. (2023) H. Vasconcelos, M. Jƶrke, M. Grunde-McLaughlin, T. Gerstenberg, M. S. Bernstein, and R. Krishna Explanations can reduce overreliance on AI systems during decision-making. Proceedings of the ACM on Human-Computer Interaction 7 (CSCW1), p. 1ā38. Cited by: Ā§3.2. Wu et al. (2026) S. Wu, H. Yao, C. BelĆ©m, S. Fu, M. Steyvers, and P. Smyth The impact of AI usage and informativeness on skill development in logical reasoning. In Proceedings of the 5th International Conference on Hybrid Human-Artificial Intelligence (HHAI 2026), p. 145ā159. External Links: Document Cited by: Ā§1, Ā§2, Ā§3.1. Zhang et al. (2022) Q. Zhang, M. L. Lee, and S. Carter You complete me: human-AI teams and complementary expertise. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, p. 1ā28. Cited by: Ā§2. Appendix A Bayesian Modeling and Inference Details A.1. Prior specification We use weakly informative priors for all parameters in the Bayesian latent ability model. Because the observed performance measures and predictors are standardized before model fitting, these priors place coefficients on a common scale. For latent initial ability, we use a standard normal prior, Īøiā1ā¼ā”(0,1) _i1 (0,1). For the latent skill-change component, we assign normal priors to the population-level regression coefficients, Ī±0,Ī±Īø,Ī±soloā¼ā”(0,1) _0, _Īø, _solo (0,1). We place an exponential prior on the residual skill-change scale, ĻĪ“ā¼Exponentialā”(2) _Ī“ (2). For the measurement model, we use normal priors for the accuracy and response-time intercepts, Ī¼A,Ī¼Rā¼ā”(0,1) _A, _R (0,1), and exponential priors for the residual standard deviations, ĻA,ĻRā¼Exponentialā”(1) _A, _R (1). These exponential priors restrict scale parameters to positive values and provide weak regularization toward smaller residual variation, consistent with common recommendations for priors on scale parameters in hierarchical Bayesian models (Gelman, 2006; Stan Development Team, 2025). A.2. MCMC inference We fit the Bayesian latent ability model in Stan through the Python interface CmdStanPy, using the No-U-Turn Sampler (NUTS), an adaptive Hamiltonian Monte Carlo algorithm (Hoffman et al., 2014; Stan Development Team, 2024). For the main specification, we run four chains, each with 1,000 warmup iterations and 2,000 post-warmup sampling iterations, yielding 8,000 posterior draws in total. We set the target acceptance probability to 0.99 and the maximum tree depth to 12. The model is estimated using a non-centered parameterization for participant-specific latent skill changes, which improves sampling efficiency in hierarchical models by separating the standard-normal skill-change residuals from the residual skill-change scale. Posterior summaries are computed empirically from the retained MCMC draws. Specifically, posterior means are computed as the average of posterior draws, 95% credible intervals are computed using the 2.5th and 97.5th posterior percentiles, and posterior probabilities such as Pā”(Ī±solo>0)P( _solo>0) are computed as the fraction of posterior draws satisfying the corresponding inequality. Participant-level summaries of latent initial ability, post-AI ability, and latent skill change are computed in the same way from the posterior draws of Īøiā1 _i1, Īøiā3 _i3, and Ī“i _i. A.3. Model Diagnostics and Predictive Checks MCMC convergence. We assess MCMC convergence using standard sampling diagnostics. For the main solo-share model, all reported R R values are below 1.01, and the rank-normalized effective sample sizes are satisfactory. We observe no divergent transitions, and the tree-depth and E-BFMI diagnostics indicate no sampling problems. Together, these diagnostics indicate satisfactory convergence and mixing of the posterior samples. Posterior predictive checks. We assess posterior predictive fit by comparing observed Phase 1 and Phase 3 accuracy and log response-time distributions with replicated outcomes from the posterior predictive distribution. The observed means of all four measures and the standard deviations of three of the four fall within their corresponding 95% posterior predictive intervals; the model somewhat overpredicts the dispersion of Phase 1 log response time. Overall, the model reproduces the central tendencies and most of the observed dispersion well. Uncertainty in cross-validation comparisons. In addition to the aggregate held-out results reported in Table 1, we examine uncertainty in predictive performance across held-out participants. The mean held-out log predictive density per participant (standard error) is ā2.661-2.661 (0.0900.090) for the main solo-share model, ā2.673-2.673 (0.0900.090) for the solo-share model with AI-cost condition indicators, ā2.685-2.685 (0.0900.090) for the AI-usage model, and ā2.734-2.734 (0.0880.088) for the constant-change baseline. Thus, although the solo-share model has the highest held-out predictive performance, the differences among the specifications are modest. The cross-validation results therefore provide additional predictive evidence in favor of the solo-share specification. A.4. Alternative specification with solo share and AI-cost condition controls As an alternative specification to the main solo-share model, we allow the assigned AI-cost condition to enter the Bayesian latent ability model directly in addition to solo share. This specification tests whether the randomized condition explains additional variation in latent skill change after accounting for initial ability and solo share. We use the no-AI condition as the reference group and define two indicator variables: LowCosti=1LowCost_i=1 for participants assigned to the low-cost AI condition, and HighCosti=1HighCost_i=1 for participants assigned to the high-cost AI condition. The latent skill-change component is then Ī“iā¼ā”(Ī±0+Ī±ĪøāĪøiā1+Ī±lowāLowCosti+Ī±highāHighCosti+Ī±soloāSoloi,ĻĪ“). _i ( _0+ _Īø _i1+ _lowLowCost_i+ _highHighCost_i+ _soloSolo_i,\ _Ī“ ). The remaining measurement model is unchanged from the main specification. Thus, Phase 1 and Phase 3 accuracy and log response time are modeled as noisy measurements of latent ability, and post-AI ability is defined as Īøiā3=Īøiā1+Ī“i _i3= _i1+ _i. The posterior estimates show that adding direct condition effects does not provide clear additional explanatory information. The coefficient for the low-cost AI condition is negative but highly uncertain, Ī±low=ā0.084 _low=-0.084, 95% CI [ā0.399,0.225][-0.399,0.225]. The coefficient for the high-cost AI condition is positive but also highly uncertain, Ī±high=0.062 _high=0.062, 95% CI [ā0.238,0.359][-0.238,0.359]. Both credible intervals contain zero. In contrast, the initial-ability coefficient remains strongly negative, Ī±Īø=ā0.531 _Īø=-0.531, 95% CI [ā0.762,ā0.304][-0.762,-0.304], consistent with the main model. The solo-share coefficient remains positive but is somewhat attenuated relative to the main specification, Ī±solo=0.114 _solo=0.114, 95% CI [ā0.017,0.245][-0.017,0.245]. Overall, this alternative specification does not indicate that the assigned AI-cost condition has an additional direct association with latent skill change once initial ability and solo share are included. This supports the main specification, which focuses on preserved independent reasoning during Phase 2 rather than modeling assigned condition as a direct predictor of skill development. A.5. Alternative specification with AI usage As an alternative specification, we replace solo share with AI usage in the Bayesian latent ability model. This specification tests whether the total number of AI assistance requests explains latent skill change, rather than the amount of independent reasoning preserved during Phase 2. Let UsageiUsage_i denote standardized AI usage, measured as the total number of assistance requests made by participant i during Phase 2. The latent skill-change component is then Ī“iā¼ā”(Ī±0+Ī±ĪøāĪøiā1+Ī±usageāUsagei,ĻĪ“). _i ( _0+ _Īø _i1+ _usageUsage_i,\ _Ī“ ). The remaining measurement model is unchanged from the main specification. Phase 1 and Phase 3 accuracy and log response time are modeled as noisy measurements of latent ability, and post-AI ability is defined as Īøiā3=Īøiā1+Ī“i _i3= _i1+ _i. The posterior estimates show that AI usage does not explain latent skill change. The coefficient on AI usage is essentially zero, Ī±usage=0.0004 _usage=0.0004, 95% CI [ā0.122,0.124][-0.122,0.124], with Pā”(Ī±usage>0)=0.501P( _usage>0)=0.501. Thus, higher request frequency is not associated with either higher or lower latent skill change after accounting for initial ability. The initial-ability coefficient remains negative, Ī±Īø=ā0.495 _Īø=-0.495, 95% CI [ā0.725,ā0.268][-0.725,-0.268], consistent with the main specification and indicating that participants with lower initial latent ability tended to show larger skill gains. Overall, this alternative specification suggests that latent skill development is not explained by AI request frequency itself. Together with the main solo-share model, this supports the interpretation that the relevant behavioral signal is the extent to which participants preserved independent reasoning during Phase 2, rather than how often they requested AI assistance. A.6. Held-out model evaluation We compare four Bayesian latent ability model specifications. The first is the main solo-share model described in Section 4.2.1, in which latent skill change depends on initial latent ability and Phase 2 solo share. The second adds direct AI-cost condition controls to the solo-share model, as described in Supplementary Material A.4. The third is the AI-usage model in Supplementary Material A.5, which replaces solo share with the total number of AI requests. The fourth is a constant-change baseline, in which all participants share the same expected latent skill change, Ī“i=Ī±0 _i= _0, so post-AI ability differs only through baseline latent ability and measurement noise. We evaluate these specifications using 5-fold held-out prediction of Phase 3 outcomes. In each fold, the model is fit using only the four-fifths of participants assigned to the training fold. For these training participants, the model observes the Phase 1 performance measures, Phase 3 outcomes, and any behavioral predictors used by the corresponding specification. The remaining one-fifth of participants are excluded from model fitting entirely. After fitting, we use only the held-out participantsā prediction inputs to compute posterior predictive distributions for their Phase 3 outcomes. These inputs are Phase 1 performance and solo share for the main solo-share model; Phase 1 performance, solo share, and condition indicators for the condition-control model; Phase 1 performance and AI usage for the usage-only model; and Phase 1 performance only for the constant-change baseline. The primary evaluation metric is held-out log likelihood, which measures out-of-sample predictive accuracy by evaluating the posterior predictive probability assigned to held-out observations. For each held-out participant, we compute the log predictive probability of the observed Phase 3 accuracy and log response time under the posterior predictive distribution, averaging over posterior draws from the model fit on the training fold. We then sum this quantity across held-out participants and folds, with higher values indicating better predictive fit. We also report held-out mean squared error (MSE), computed by comparing posterior mean predictions with the observed held-out Phase 3 accuracy and log response time; lower MSE values indicate better point prediction. As reported in Table 1, the main solo-share model achieves the highest held-out log likelihood and the lowest held-out MSE for both Phase 3 accuracy and log response time. Thus, the held-out results favor the solo-share model on overall predictive fit. Appendix B Experiment Details B.1. Post-Study Survey After completing the three-phase study, participants were asked to fill out a short post-study survey. This survey was originally used in the pilot study to gather feedback and adjust the game settings, and we kept it in the main experiments for consistency. In the analyses reported in this paper, we only use responses from participants in AI conditions regarding whether they reported using AI assistance, comparing these with actual usage logs to identify inconsistent behavior as a filtering condition. No other survey responses were used in our analyses. The survey content is shown below. Questions with 5-point Likert scale response options: Strongly Disagree, Disagree, Neutral, Agree, and Strongly Agree: ā¢ I found the game to be boring. ā¢ I felt confident in my performance throughout the game. ā¢ I put in considerable effort to achieve my level of performance. ā¢ I developed a strategy to improve my performance during the game. ā¢ I find seeing the correct solution for each problem helps me learn. ā¢ Iāve had prior experience with a similar task. Two optional free-response questions: ā¢ Did you notice any hidden rules during the game? If so, please describe them. Additionally, please share the strategy you used while solving the puzzles. ā¢ Please share your overall experience with the game and any suggestions for improvement. For participants in AI assistance conditions, we further asked: ā¢ Rate the AI assistantās accuracy (0ā100, or select Not Applicable). ā¢ The AI assistant was helpful during the experiment (5-point Likert scale, with an additional choice Not Applicable). ā¢ When did you typically use the AI assistant during the experiment? For example: Right after seeing the problem, after attempting a submission, to check your answer, etc. (optional free response question)