Paper deep dive

Available but Unclaimed: An Empirical Study of Human-AI Synergy

Robin Welsch, Michelle Rausch, Pascal Knierim, Thomas Kosch, Jochen Kuhn, Albrecht Schmidt, Daniela Fernandes

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 134

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:People increasingly reason with large language models (LLMs), yet complementary capabilities do not guarantee outperforming both components. In a between-subjects study, participants (N=535) solved a 40-item battery of matrix reasoning, mental rotation, syllogisms, and letter-string analogies, unaided or with GPT-5.6-Luna, Claude Opus 4.8, Gemini 3.6 Flash, or Kimi K3. Each assisted trial required consultation with the model. Each model answered every item alone 100 times under matched elicitation. The assisted-unaided accuracy difference increased with item-level LLM competence. Deference varied across tasks and increased with competence within tasks. Post-advice confidence distinguished correct from incorrect answers less strongly than unaided confidence. In a reference comparison, about half the increase in LLM accuracy carried through to assisted accuracy. How much of that accuracy gain reached participants differed across the models. These findings motivate evaluating LLMs in interaction with humans and designing support for selective deference that preserves independent reasoning.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

131,691 characters extracted from source content.

Expand or collapse full text

Available but Unclaimed: An Empirical Study of Human–AI Synergy Conference: ; ; CCS: Human-centered computing Human computer interaction (HCI) Robin Welsch Affiliation: Aalto University, Espoo, Finland email: robin.welsch@aalto.fi , Michelle Rausch Affiliation: Aalto University, Espoo, Finland , Pascal Knierim Affiliation: University of Innsbruck, Innsbruck, Austria email: pascal.knierim@uibk.ac.at , Thomas Kosch Affiliation: Humboldt-Universität zu Berlin, Berlin, Germany email: thomas.kosch@hu-berlin.de , Jochen Kuhn Affiliation: Faculty of Physics, Ludwig-Maximilians-Universität München, Munich, Germany email: jochen.kuhn@lmu.de , Albrecht Schmidt Affiliation: Ludwig-Maximilians-Universität München, Munich, Germany email: albrecht.schmidt@ifi.lmu.de and Daniela Fernandes Affiliation: Aalto University, Espoo, Finland email: daniela.dasilvafernandes@aalto.fi Date: September 11, 2026 Abstract. People increasingly reason with large language models (LLMs), yet complementary capabilities do not guarantee outperforming both components. In a between-subjects study, participants (N=535) solved a 40-item battery of matrix reasoning, mental rotation, syllogisms, and letter-string analogies, unaided or with GPT-5.6-Luna, Claude Opus 4.8, Gemini 3.6 Flash, or Kimi K3. Each assisted trial required consultation with the model. Each model answered every item alone 100 times under matched elicitation. The assisted–unaided accuracy difference increased with item-level LLM competence. Deference varied across tasks and increased with competence within tasks. Post-advice confidence distinguished correct from incorrect answers less strongly than unaided confidence. In a reference comparison, about half the increase in LLM accuracy carried through to assisted accuracy. How much of that accuracy gain reached participants differed across the models. These findings motivate evaluating LLMs in interaction with humans and designing support for selective deference that preserves independent reasoning. Keywords: human-AI interaction, complementarity, reliance, large language models, reasoning, decision support Figure 1. Assisted and assistant-alone accuracy differ across tasks. Each assistant’s own accuracy (left, 100-run chat-parity benchmark) and its teams’ accuracy (right), overall and per task. The dotted rule marks unaided accuracy and the dashed rule chance (undefined for letter strings). Small points show per-item competences. Bars show 95% Jeffreys intervals on pooled runs and trials, treating observations as independent.Five panels, overall and four tasks, with a small schematic of the task at the top of each and the task name at its foot. In each panel four lines run from an assistant-alone point on the left to a team point on the right, each marked with its provider's logo, and a cloud of small points behind the left column shows that assistant's per-item competence spread. A dotted line labelled unaided marks the unaided group. A dashed line labelled chance marks each task's chance level. The lines are not parallel. The assistants that gain most between the two columns are not the ones highest on the left. Thin vertical bars behind each mark show 95\% intervals. They are narrow throughout and mostly hidden by the marks. 1. Introduction Large language models (LLMs) are unevenly reliable within a single domain. A model may solve one reasoning problem on nearly every attempt and a problem of the same form at chance (Lewis and Mitchell, 2024; McCoy et al., 2024), while its replies need not reveal the difference (Zhou et al., 2024). Whether a person can distinguish correct from incorrect advice and rely on the model accordingly matters for effective human–AI interaction (HAI). People and LLMs exhibit different strengths and weaknesses when working on a problem. One can theoretically compensate for the other’s errors (Hemmer et al., 2025; Steyvers et al., 2022). Complementarity denotes this potential, created where human and AI errors differ. Synergy denotes the realized case in which the team actually outperforms both components (Vaccaro et al., 2024). During an interaction, answers can change as the conversation proceeds. Differing errors therefore create an opportunity, but do not guarantee that the person can recognize and correct them. We call the realized share of independent-error reference headroom synergy capture. However, achieving synergy is not built into an LLM’s configuration. It depends on the interaction, particularly on what the user understands and how the assistant communicates its reliability (Zhou et al., 2024). That is the familiar concern of reliance, together with its failure mode of overreliance (Lee and See, 2004). Effective human–AI synergy depends not only on how capable the LLM is, but also on how people use its advice. LLM assistants are typically reluctant to express uncertainty, and users rely on their output even when it is marked as uncertain (Zhou et al., 2024). Users accordingly underestimate performance variation (Kelly et al., 2023) and form biased estimates of model quality (Fernandes et al., 2026). Where AI reliance in HAI is easily quantifiable (e.g., in simple decision-making tasks with low to no interactivity), synergy is rare. Meta-analytically, human–AI teams fail on average to exceed the better of their parts in non-conversational interactions (Vaccaro et al., 2024), where per-item correctness is known, reliance can be scored as appropriate (Schemmer et al., 2023), and complementarity is formally defined (Hemmer et al., 2025). Interactive studies likewise show that solo LLM accuracy incompletely predicts assisted performance (Chang et al., 2025; Riedl and Weidmann, 2026). Four questions organize this paper. Because LLM competence is measured per item, we ask (RQ1) where consulting pays and where it costs rather than whether it helps on average. Because we classified each trial’s advice, its correctness, and whether it was adopted, we ask (RQ2) whether deference follows that boundary. We ask (RQ3) whether confidence supports selective deference and what gains reference policies predict. Because four assistants encountered the same items, we ask (RQ4) whether a more capable LLM yields a stronger team. To address these, we report a study in which 535 participants worked the same 40-item battery, spanning matrix reasoning, mental rotation, syllogistic reasoning, and letter-string analogies, either unaided (N=187N=187) or with one of four LLM assistants, namely gpt-5.6-luna (N=179N=179), claude-opus-4-8 (N=61N=61), gemini-3.6-flash (N=52N=52), or kimi-k3 (N=56N=56). Each assistant also answered every item alone over 100 independent runs, using matched stimuli and API settings. This estimates LLM competence on the study’s own items, with a binomial standard error of approximately .05 or less. We find that benefits increase with the LLM’s competence on an item, with positive effects resolved in the two upper competence bands. Across the examined contrast, assisted accuracy increased about half as steeply as LLM competence. Deference (i.e., submitting the assistant’s answer on a given trial) varied across tasks and increased with competence within tasks. Stated confidence distinguished correct from incorrect answers overall, with a modest positive association between discrimination and synergy capture. Retrospective item-level selection estimates a gain of about six points, or two or three questions in forty. Reliability, then, must be measured at the item level, not reduced to a single score for the assistant. This paper makes three contributions to human–AI interaction. • An account of answer agreement and switching in open-ended chat, based on conversations classified trial by trial across four assistants. Under mandatory consultation, deference varied substantially by task and also tracked item competence. The own-answer association was uncertain on disjoint trials. These findings motivate examining interaction design, rather than treating inappropriate deference simply as a user deficit. • Evidence that complementary errors are not enough. Formal accounts show how metacognitive sensitivity can support beneficial combination under specified conditions (Li and Steyvers, 2026). Here, confidence distinguished correct from incorrect answers, yet participants still frequently followed incorrect advice. The policy ladder estimates what more selective use of the two components could achieve, distinguishing task- and item-level selection from an oracle that assumes independent errors. • Assisted accuracy increases less steeply than LLM competence at the specified contrast. Assistants within a few points of one another alone differed approximately twofold in benchmark-based pass-through, the association between item-level LLM competence and assisted accuracy (Table 5). The spread was smaller on the realized-advice axis. Together with deference and capture, these slopes extend prior comparisons of solo and assisted performance (Chang et al., 2025). 2. Related Work 2.1. Complementarity in Human–AI Teams Complementarity is necessary for human–AI synergy because differing errors create the potential to outperform either component (Bansal et al., 2021; Hemmer et al., 2025; Zöller et al., 2025; Steyvers et al., 2022). Empirically, synergy is rare (Vaccaro et al., 2024). A stronger model is a harder baseline to outperform, and overreliance can erode the human contribution until the pair trails the person alone (Klingbeil et al., 2024). How much of that complementarity a pair collects depends in part on metacognitive sensitivity (i.e., how well confidence separates one’s own right answers from one’s wrong ones). A Bayes-optimal combination can improve on both components, even with a less accurate assistant, provided accuracy differences and the dependence between human and LLM confidence satisfy the model’s conditions (Li and Steyvers, 2026). This guarantee concerns independently formed judgments combined by a rule. Calibration alone offers no general guarantee either. In binary classification, any deterministic rule combining calibrated probabilities that does not essentially always follow one agent can perform worse than both for some joint distribution of predictions and outcomes (Peng et al., 2025). The benefit of combining predictions also depends on their joint error structure. How complementarity and metacognition support synergy during an open interaction therefore remains an empirical question. Work on model ensembles illustrates why error dependence matters. Models trained on overlapping corpora fail the same cases (Kim et al., 2024), and systems that share foundation components homogenize decisions (Bommasani et al., 2026; Kleinberg and Raghavan, 2021). A recent audit of several hundred models reports that two models agree on roughly six of every ten items both answer incorrectly, with error correlation increasing rather than decreasing as models grow larger and more accurate, across architectures and providers (Kim et al., 2025). Routing and ensembling, by contrast, benefit where models genuinely differ in which items they answer well (Jiang et al., 2023; Kim et al., 2026). These findings concern model–model combinations, but reinforce the need to measure rather than assume complementary errors in human–AI pairs. Evaluating synergy requires an LLM-alone reference on the study’s own items. Major evaluation traditions benchmark assistants in isolation on standard item pools (Legg and Hutter, 2007; Chollet, 2019; Srivastava et al., 2023; Bringsjord, 2011; Pellert et al., 2023). Scores can sit at ceiling (Klein and Kovacs, 2024) or depend on elicitation conditions. ChatBench addresses this mismatch by measuring human-alone, LLM-alone, and assisted accuracy on shared questions, including a free-text benchmark with matched model settings (Chang et al., 2025). We extend this approach to generated reasoning items, relating repeated competence estimates to deference, confidence, and realized synergy. 2.2. Reliance and Its Failure Modes Complementarity can be lost when accepting advice replaces independent reasoning. People accept LLM output even when it is worse than their unaided judgement (Klingbeil et al., 2024; Shekar et al., 2024). Without corrective feedback, reliance hardens into coarse heuristics about whether to follow the system at all (Lu and Yin, 2021), and impressions of its competence persist despite repeated disconfirmation (Colombatto and Fleming, 2023). Interface designs have attempted to restore displaced deliberation with mixed results. Making acceptance a deliberate action reduces overreliance, yet users rate these designs lowest (Buçinca et al., 2021). Explanations increase acceptance regardless of advice correctness (Bansal et al., 2021; Wang and Yin, 2021). Disclosing weaknesses reduces reliance on incorrect advice only when errors are difficult to detect (Rieger et al., 2026; Qian and Wexler, 2024). The target has accordingly been reframed as appropriate reliance, defined as following correct advice while refusing incorrect advice (Schemmer et al., 2023)—with the most effective designs being those that users resist. Judge–adviser measures score advice-taking as the shift between pre- and post-advice judgements (Bonaccio and Dalal, 2006; Yaniv, 2004), with algorithmic as well as human advisers (Logg et al., 2019). Evaluating appropriate reliance and automation bias (Parasuraman and Riley, 1997; Skitka et al., 1999; Wickens et al., 2015) additionally requires relating advice-taking to advice correctness on each trial and tracing how answers change during interaction. Advice-taking measures also miss metacognitive sensitivity, or how well confidence distinguishes one’s own correct from incorrect answers. Confidence can rise or fall wholesale without improving metacognitive sensitivity, which is why signal-detection accounts were developed to disentangle performance, mean confidence, and metacognitive sensitivity (Rahnev, 2025), and why increased confidence conveys no information about whether judgment improved. Advice affects both dimensions unequally. An adviser’s expressed confidence shifts a judge’s own confidence and trust regardless of advice quality (Pescetelli and Yeung, 2021). With an LLM as adviser, confidence can drift toward what the system displays and remain displaced after withdrawal (Li et al., 2025), while offloading can decouple confidence from accuracy (Fernandes et al., 2026). Connecting such shifts to performance requires identifying the advice participants actually received and whether they adopted it. We measure deference under required consultation by identifying what the LLM advised, whether the advice was right, and whether the participant submitted it. Deference can thus be checked item by item against where following the assistant actually paid. Selective deference also requires effort. Bounded rationality sets the standard as the best rule available under limited information and time (Simon, 1955). Rational analysis asks which policy is adapted to the environment’s structure (Anderson, 1990), resource-rational analysis prices that policy against the computation it costs (Lieder and Griffiths, 2020), and computational rationality sharpens it to which policy maximises what a person cares about, net of the effort it demands (Oulasvirta et al., 2022), so an apparent error may be a correctly priced decision not to spend effort. Offloading is least costly when verification costs exceed its benefits (Risko and Gilbert, 2016), interface designs that facilitate delegation further reduce its cost (Grinschgl et al., 2020), and errors introduced by the resource can make that strategy costly (Weis and Wiese, 2022). These accounts all assume the agent can price the option in front of it, and pricing requires a signal that tracks what that option is worth. Offloading research shows that people defer based on their own confidence (Boldt and Gilbert, 2019), and their decisions approach the optimal policy when confidence is informative (Gilbert et al., 2020). These accounts motivate comparisons with explicit alternative policies. In this study, item-level measurement lets us evaluate reference policies—task- and item-level selection and an independent-error oracle—and compare their conditional gains. 2.3. Team Performance and LLM Capability These reliance decisions also matter when comparing assistants. One might expect that as LLM competence increases and people defer to the advice, team accuracy would rise accordingly. Which advice people take also depends on the person, interface, and assistant. The complementarity literature characterizes how a team performs relative to its components (Bansal et al., 2021; Hemmer et al., 2025), and a large-scale meta-analysis of human–AI performance finds that team performance tracks LLM competence in aggregate (Vaccaro et al., 2024). Note, however, that this trend conceals cases where a stronger AI yields a weaker team, and the mechanism remains unidentified. Work on tuning decision-support systems finds that the match between the assistant’s error profile and the user’s baseline matters as much as its overall accuracy (Inkpen et al., 2023). Some studies hold the model fixed and vary its stated or observed accuracy (Yin et al., 2019), its error profile at constant accuracy (Inkpen et al., 2023), or its explanations (Bansal et al., 2021). Within a single system, higher accuracy does yield larger team gains (Yu et al., 2024). One study pairs clinicians with five models on the same vignettes (Zöller et al., 2025), evaluating aggregated ranked outputs rather than interactive assistance. ChatBench instead compares two LLMs in human interaction and finds that their solo accuracy gap narrows with assistance (Chang et al., 2025). Reanalyzing ChatBench, Riedl and Weidmann (Riedl and Weidmann, 2026) separate individual and collaborative ability using Bayesian item-response models and associate conversational perspective-taking with collaborative performance. Without humans, a model’s rank alone and its rank as a helper in a team of LLMs are only modestly related (Wongchamcharoen et al., 2026), and combining models stops paying once the single-model baseline is strong (Kim et al., 2026). The candidate mechanism is misallocated reliance. Gains from upgrading the LLM depend partly on which advice people follow. Users treat LLM accuracy as steadier across items than it is (Kelly et al., 2023), so a stronger LLM may raise its own bar without improving the team’s selectivity. We extend these evaluations by estimating pass-through, the association between item-level LLM competence and assisted accuracy, alongside realized synergy and synergy capture. We examine slopes across forty items per assistant rather than relying on the rank order of four assistants. Whether a better LLM produces a better team is thus an empirical question, not a given. 3. Method In the following, we motivate and document our methodological choices. We built and deployed a purpose-made benchmark with a 4040-item cognitive battery spanning four task domains, a web application that administers it under proctoring, and an automated evaluation of state-of-the-art LLMs on the identical items under prompting conditions matched to what participants see. 3.1. Participants A two-sample power calculation for a person-level accuracy difference of d=.30d=.30 (α=.05α=.05, two-sided, power =.80=.80, pwr::pwr.t.test) gives n=175.4n=175.4, rounded to 176176 per group. The unaided and gpt-5.6-luna groups exceed that target (N=187N=187 and 179179). The additional groups (N=61N=61 claude-opus-4-8, 5656 kimi-k3, and 5252 gemini-3.6-flash) extend the comparison to other assistants. An item-level proportion based on roughly 50 participants has a maximum binomial standard error of about .07.07, compared with .04.04 for 179 participants. Hierarchical models account for repeated responses and partially pool group estimates. We recruited through Prolific and redirected participants to our own web application. Each group was a separate Prolific posting, so assignment was by posting rather than randomized within one. Appendix D reports per-group demographics and AI experience. The groups are broadly comparable on age, education, and AI-use frequency, with the gemini-3.6-flash group skewing female (67% vs. 45–52%). Eligible were adults aged 18 or older whose first language was English, who resided in the United Kingdom, and who had not taken part in an earlier batch of the study. The study had to be completed on a desktop computer to avoid excessive scrolling. Participants gave informed consent through a stepped briefing covering the data collected, storage keyed only to the Prolific identifier with no name or contact details recorded, the browser-based proctoring by AutoProctor, the retention period, and the right to withdraw at any time. Anyone unwilling to be proctored was instructed to return the study on Prolific instead. Assessed against Aalto University’s criteria for mandatory ethics review under the Finnish national research-integrity guidelines, the study met none of them (intervention in physical integrity, departure from informed consent, minors without guardian consent, exceptionally strong stimuli, mental harm beyond that of ordinary life, or a security threat) and therefore did not require committee review. Participants were paid £9 9 per hour plus performance bonuses (Bianco et al., 2021). Each group had a separate leaderboard. Its top ten received an additional half of the hourly rate for their session time, and first place received a further £100 100, split among ties. Participants saw the bonus rules before starting, but no scores or standings were shown beforehand. Completed sessions were paid whether or not their data were retained, except where the session was returned. Returns were requested for incomplete sessions, for participants failing two or more attention checks, and for those the proctoring review indicated had consulted an external AI assistant or been offline for more than 10 minutes. Where a participant contested a return, we paid full or partial compensation rather than pursue it. Proctoring data were used solely for the manual integrity review. We excluded sessions with fewer than 4040 scored responses or a missing questionnaire section, two or more failed attention checks, or any flag in the AutoProctor manual review, which caught behavior such as consulting an external AI assistant or being offline for more than 10 minutes. Fifty-seven sessions were excluded, thirty because of a technical error and twenty-seven following proctoring review. One of the latter participants had also failed all three attention checks. The analyzed sample comprises 535 participants, including 270 men, 260 women, and 5 who reported non-binary or multiple categories or did not disclose, aged 18–76 (M=35.2M=35.2, SD=10.8SD=10.8), with 71% holding at least a bachelor’s degree. The sample was AI-experienced — 81% reported using AI at least weekly and 44% daily, most often naming OpenAI as their preferred provider (56%) — so the reliance we observe is not a novice’s first contact with an assistant. The full demographic breakdown by condition is included with the study materials (see the availability note opening Section 4). 3.2. Experimental Design The user study used a between-subjects design with one manipulated factor, assistant availability (AI versus no-AI). In the AI condition, a conversational AI assistant appeared alongside every task. In no-AI the battery was identical but the assistant panel was suppressed entirely. Separately from the user study, each assistant was measured alone on the identical items as a reference benchmark (Section 3.5). The four assistant groups are described in Section 3.1. Items were generated from seeded parameters to reduce the risk of using published instruments present in model training data. In AI, participants had to consult the assistant at least once per item before submission, but answer controls remained available so they could form an answer first. The study therefore measures mandated consultation, not naturalistic opt-in use. Each group’s assistant and low reasoning setting were fixed server-side throughout the session. Google used a 1,024-token thinking budget. Benchmark runs used the same settings (Section 3.5). 3.2.1. The item battery We selected four reasoning paradigms used in human research and AI evaluation, with tunable difficulty to sample different human and assistant strengths. Seeded offline generators produced a frozen battery with documented item parameters and answer keys (Appendix A). Table 1 gives its composition and Figure 2 shows one item per task. Table 1. Composition of the administered 40-item battery. Difficulty labels are the generator’s designed levels, not empirical difficulties. Task type n Difficulty mix Response format Matrix reasoning 10 2 medium, 4 hard, 4 expert Multiple choice, 8 options Mental rotation 10 2 medium, 4 hard, 4 expert Multiple choice, 2 options Syllogistic reasoning 10 1 easy, 4 medium, 5 harda Multiple choice, 3 options Letter-string analogies 10 3 hard, 4 expert, 3 combination Free text entry aPresented as 6 scenarios, two carrying three questions each and four carrying one. Matrix reasoning follows Raven-like rule designs (Carpenter et al., 1990; Matzen et al., 2010), with related items studied psychometrically (Harris et al., 2020) and used in AI benchmarks (Barrett et al., 2018; Zhang et al., 2019). Participants complete a 3×33× 3 grid of geometric figures by choosing its missing bottom-right cell from eight images (Appendix A.1). Mental rotation follows Shepard and Metzler (1971). Participants judge whether two three-dimensional cube figures are the same object rotated or mirror images. Related rotation tasks assess spatial ability (Branoff, 2009) and AI performance (Ramakrishnan et al., 2024) (Appendix A.2). Syllogisms, used in human (Khemlani and Johnson-Laird, 2012) and LLM reasoning research (Eisape et al., 2024; Ozeki et al., 2024), present two-premise narrative arguments judged Valid, Invalid, or Cannot be determined. The ten questions span six scenarios, with difficulty based on mental-model counts (Johnson-Laird and Bara, 1984) (Appendix A.3). Letter-string analogies build on the Copycat paradigm (Hofstadter et al., 1995) and its use in human–LLM comparisons (Lewis and Mitchell, 2024; Webb et al., 2022). Participants complete a target sequence from two worked examples, using a fictional alphabet whose ordering is stated in the stimulus (Appendix A.4). The answers were scored server-side against the stored key, by whitespace- and case-normalized match for letter strings. On the two letter-string items whose worked examples leave a second rule defensible, the answer following that narrower rule was also accepted as correct (Appendix A.4). (a) Matrix reasoning. Complete the 3×33× 3 pattern using answer options 1–8 below. (b) Mental rotation. Is figure B the same object as figure A and just rotated, or its mirror image? (c) Letter-string analogy (free-text answer). Alphabet (in order): > * + < ! @ $ ) & = : - ( % ~ Study the following patterns: [@ ) = -] → [@ ) = - %] [$ & : (] → [$ & : ( ~] Using the same alphabet, complete this: [> + ! $] → ? (d) Syllogistic reasoning — the conclusion is judged Valid / Invalid / Cannot be determined. Excerpt (record-keeping filler omitted). “At Dunmore Bakery, the manager keeps a daily log of all baked goods and their status. […] Whenever an entry in the bakery’s records is classified as items logged by the opening team, it is also classified as items assigned to batch code 3. Among the bakery’s records, there are entries classified as items assigned to batch code 3 that do not carry the items recorded under the seasonal range code classification. […] Conclusion: All items logged by the opening team are items recorded under the seasonal range code.” Figure 2. One example item per task type; (d) is excerpted for legibility. (a) matrix item mat_025 (correct answer option 1), (b) mental-rotation item rot_014 (80∘80 depth rotation, correct answer same object), (c) letter-string item ls_014 (correct answer > + ! $ &), and (d) syllogism item syl_013 (correct answer cannot be determined). The premises in (d) are embedded in record-keeping filler so the quantifiers must be inferred from prose.Four example items, one per task type. (a) A matrix-reasoning item shows a three-by-three grid of cells whose triangles vary in number, size, and orientation, with the final cell replaced by a question mark. Eight answer options appear below in two rows, numbered 1–4 and 5–8. (b) A mental-rotation item shows two three-dimensional shapes made of cubes side by side, labelled A and B, with B rotated in depth. The response options are same object or mirrored object. (c) A letter-string item shows two example transformations of short symbol strings followed by a new string whose transformed form must be typed as free text. (d) A syllogism item shows a short workplace narrative containing quantified statements, followed by a conclusion to judge as follows, does not follow, or cannot be determined. 3.2.2. Measures Task performance was scored as the total number of correct responses out of 4040 and, per task type, out of 1010. Because the four response formats carry different chance levels — roughly 0%0\% for free-text letter strings, 12.5%12.5\% for one-of-eight matrices, 33%33\% for three-way syllogisms, and 50%50\% for the two-way rotation judgment — each task type is interpreted against its own chance level. Before and after the battery, participants estimated their own score out of 4040. Assisted participants also estimated their unaided score and the assistant’s solo score. After each task, all participants estimated their own, assistant-alone, and counterfactual scores out of 1010, with the counterfactual referring to the assistance condition they had not experienced. Every scored response carried a confidence rating from 00 (Unsure) to 100100 (Certain), with one rating per conclusion in multi-question syllogisms. The post-battery global estimates inform the supplementary belief–performance comparison (Appendix E). Additional global percentile and difficulty judgments are documented in the study materials. Backward navigation was disabled to preserve initial estimates. Every AI-assisted item also includes the full chat transcript with message counts and timings, from which the advice-quality and reliance measures of Section 3.6 are derived. Every letter-string item includes a mandatory free-text field asking participants to describe the pattern they identified and how they applied it (a wrong answer may reflect item ambiguity rather than a reasoning failure). Furthermore, the closing questionnaire asked for an optional strategy description for each task type. After the battery, we administered four questionnaire sections. The closing questionnaires asked about AI consultation strategy and behavior on disagreement, perceived helpfulness, trust, and frustration (AI only), followed by task feedback, demographics, and AI-use frequency (all participants). Three attention checks were embedded. Full item wording is included with the study materials (see the availability note opening Section 4). The questionnaire structure and attention checks are summarized in Appendix B. 3.3. Task and Procedure Participants completed the 4040 items of the battery (Section 3.2.1) in a randomized order. Tasks remained contiguous, and syllogism items were shuffled at the scenario level, keeping each scenario’s questions together. The AI interface placed a streaming chat panel beside the task (Figure 3). The panel was hidden on questionnaire pages. Figure 3. The study application used a split interface, with the reasoning task and the response and confidence controls on the left and the assistant on the right. In no-AI the same pages were shown without the chat panel.Screenshot of the study web application in the assisted condition. The left side shows the current reasoning item with its answer controls and a progress indicator. The right side shows a chat panel with the AI assistant, a message history, and a text box for sending messages. A header carries the task name and a timer. After briefing, consent and the pre-task assessment, participants completed four tasks. Each began with instructions and a practice trial, followed by ten scored items and a task-level assessment. Post-task questionnaires and the leaderboard closed the session. Median duration was 8686 minutes (interquartile range 6464–108108, AI 9898, no-AI 7171). Screen-by-screen documentation accompanies the study materials (Section 4). 3.4. Apparatus The study ran in a purpose-built web application. Implementation details accompany the research software (Section 4). AutoProctor monitored screen content, tab switches and navigation away from the study. Sessions flagged in manual review were excluded as described in Section 3.1. The chat panel provided buttons to copy the stimulus and answer options, excluding confidence and strategy reports, and to upload stimulus images. Both actions were logged. 3.5. Model Benchmarking Each assistant was benchmarked on the study’s items using matched stimuli and API settings. Unlike participants, assistants received no instruction page or practice trial, and their histories reset between tasks rather than accumulating across the session. Each assistant completed 100 runs of the 40 items. LLM competence is the proportion of retained replies scored correct per item. With 95–100 replies per cell, the maximum binomial standard error is approximately .05.05 (roughly ±.10±.10 at 95% confidence). Item order was paired across assistants within each run (Miller, 2024). Shared history makes responses within a run dependent. Runs with API errors were excluded rather than scored. The models answer in free-form prose, requiring it to be mapped onto the item’s answer set before scoring. A second-model extractor, gemini-3.5-flash-lite run with thinking disabled, performs that mapping for every reply. Deterministic normalization and an accepted-alternate table are applied to its output, and for letter strings, a deterministic reading of the reply’s own bracketed answer overrules the extractor wherever that answer is unambiguous. Replies flagged as needing review were excluded. Refusals without that flag were scored as produced. This left 15,968 observations. Assistant rankings were unchanged under alternative flag treatments (Appendix C). 3.6. Coding AI Advice Quality and Reliance We processed each item’s complete transcript with a second-model extraction pipeline. Each transcript, stripped of the assistant’s reasoning summaries but keeping the turn labels, was passed to gemini-3.5-flash-lite with thinking disabled, instructed to extract, rather than solve for, the first answer the assistant put forward, its last, and the answer the conversation as a whole landed on, each in the item’s answer format. For multi-question syllogism scenarios, the conclusion under analysis is named in the prompt. From these we derive whether the final advice matched the key, whether the participant’s answer matched that advice, whether first and final differed, and whether the assistant never committed to an answer at all, which counts as incorrect advice but stays distinguishable from a wrong one. Crossing advice quality with whether it was followed and, where it was declined, whether the participant was right, yields the categories used throughout, namely appropriate reliance, costly override, overreliance, successful and unsuccessful override — kept apart deliberately, since collapsing the last two would leave “overrode bad advice” ambiguous between the participant’s own competence and mere disagreement — and no advice. To check the extractor’s followed-versus-declined reading, one author hand-coded 200 transcripts, 25 per group and advice-correctness cell. Nine trials were marked unclear and excluded from the agreement calculation. Agreement on the remaining 191 was 89.5% (κ=.64κ=.64). Task-specific agreement was .97 and .98 on the image tasks, .86 on syllogisms, and .71 on letter strings. The audit and sensitivity analysis concern adoption labels. Advice-correctness extraction requires separate validation. Main intervals treat all extracted labels as fixed. 3.7. Statistical Approach We report posterior medians, 95% highest-density intervals (HDIs), and directional probabilities P⁡(b>0)P(b>0). HDIs accommodate skewed posteriors. We call an interval excluding zero resolved. One spanning zero leaves the direction uncertain. We estimate descriptive associations on this battery at item, trial, and participant levels. Group contrasts do not isolate effects of access or assistant identity. Item-level models use either 40 item summaries or 160 item–assistant cells. Trial-level models use Bernoulli likelihoods with crossed participant and item effects. Participant-level models use one summary per person. Gaussian models describe confidence, accuracy summaries, and capture. Capture is a ratio modeled on an unbounded scale. Most fixed coefficients have Normal(0, 1.5) priors, with Exponential(1) priors on standard deviations. Intercept and correlation priors vary by model, including brms defaults. The supplementary notebook lists the exact priors for each fit. The benchmark’s 100 elicitations per item and assistant are distinct from posterior sampling. The reported Bayesian fits use four chains of 20,000 iterations, half warmup. Intervals for aggregated item summaries condition on the estimated component accuracies. Following the Bayesian analysis checklist (Depaoli and van de Schoot, 2017) (WAMBS), we audited convergence and prior sensitivity. Across the reported models, the largest R R was 1.003, the smallest bulk effective sample size 1,663, and residual divergences reached 0.49% of post-warmup draws. Half-width, double-width and flat-prior refits moved probability-scale quantities by at most .015.015, without changing direction or resolution. Models were fit in R with brms (2.23) and cmdstanr. The supplementary notebook gives full diagnostics, sensitivity results and specifications. 4. Results Throughout the Results, accuracy and proportion correct name the same quantity, the share of items answered correctly, for participants and teams alike. LLM competence is the assistant’s accuracy on an item, measured over one hundred benchmark runs (Section 3.5), and keeps its own name because it serves as the predictor throughout. The study application, item generators, and chat-parity benchmark harness are maintained in HAI_Benchmark. A minimal release of de-identified model inputs, the analysis notebook, the corresponding R script, and reported posterior summaries will be added to the same repository. These inputs support model reproduction rather than reconstruction of raw conversations. 4.1. Team Accuracy by Item Competence (RQ1) Five stacked rows of individual accuracy, each a density curve above a strip of jittered points, one point per participant. The unaided curve is broad and centered near .53. The four assisted curves sit further right and are narrower, centered near .69, .71, .73 and .79, and their point strips are visibly sparser, the unaided group and gpt-5.6-luna rows carrying three times as many participants as the others. A dashed vertical line in each assisted row marks that assistant's own accuracy, falling inside the bulk of its own team's distribution in every case except kimi-k3, where it sits above it. Figure 4. Every assisted group outperforms the unaided one, and none exceeds its own assistant by much. Distribution of individual accuracy over the 40 items, one row per group. Note. Each row is a kernel density over participants above jittered points, one per participant. Group sizes are given beside each row. The dashed rule and diamond in each assisted row mark that assistant’s own accuracy on the same 40 items without a person in the loop (Section 4.4).Five stacked rows of individual accuracy, each a density curve above a strip of jittered points, one point per participant. The unaided curve is broad and centered near .53. The four assisted curves sit further right and are narrower, centered near .69, .71, .73 and .79, and their point strips are visibly sparser, the unaided group and gpt-5.6-luna rows carrying three times as many participants as the others. A dashed vertical line in each assisted row marks that assistant's own accuracy, falling inside the bulk of its own team's distribution in every case except kimi-k3, where it sits above it. We compare unaided participants, assistants answering alone, and participants working with assistants (teams). Accuracy differences use proportion correct, with .1.1 equivalent to four items. Battery-level comparisons average each component first. Realized synergy S instead compares the team with the better component on each item, a stronger reference. Capture expresses gains relative to an independent-error oracle. Pass-through describes the association between item competence and assisted accuracy. We now trace through several analyses how LLM competence unfolded in performance through interaction. Every assisted group outperformed the unaided group (Figure 4). Averaged over items, assisted accuracy exceeds unaided accuracy by .200.200 [.084.084, .316.316], approximately eight of forty items. The comparison with the assistant alone is uncertain, at −.006-.006 [−.157-.157, .140.140] (see Table 5). We define realized synergy S as the team’s accuracy on an item minus whichever of its two parts scored higher there, so it is positive only where the pair outperformed both of them. Teams beat the unaided participant on 132 of the 160 item-by-assistant cells, and the LLM alone on 61 — but both at once on only 42 (Figure 5). Averaged over cells, S is negative, −.062-.062 [−.118-.118, −.005-.005], or about two and a half items of the forty.11 1 Excluding the 37 cells where the team answered more than .90.90 of items correctly gives −.074-.074 [−.128-.128, −.018-.018] on the remaining 123, so the negative average is not a ceiling artifact. S contrasts independent groups. On pre-reply trials, final accuracy minus the higher of the two pooled component accuracies is −.003-.003 [−.019-.019, .013.013]. This paired estimate pools component accuracies over the selected pre-reply trials, and its direction remains uncertain. Negative average S can therefore coexist with assisted accuracy above the assistant’s battery-level average. Per-task results are in Table 2. Synergy capture expresses observed gains relative to a complementary reference. The reference is h+c−hch+c-hc, where h and c are unaided-group and assistant accuracy on the item. Capture divides the gain over h by the reference headroom c⁡(1−h)c(1-h). Unlike accuracy, capture is a ratio. A value of .500.500 means half the reference gain over unaided performance was obtained. Overall capture is .584.584 [.356.356, .809.809], varying across tasks (Table 2). The reference assumes independent human and assistant errors. Human–assistant co-failure is measured on trials with an answer committed before the first reply, and therefore inherits that subset’s self-selection. On these trials, excess co-failure averages .036.036 [.010.010, .064.064] over the item–assistant cells. Positive dependence lowers this oracle, so capture quantifies reference headroom rather than observed paired potential. Point-and-interval plot of three accuracy series across 40 items ordered by LLM competence. The LLM series rises from zero to one across the ordering, the unaided group series is roughly flat, and the assisted series follows the unaided group series on the left of the plot and the assistant series on the right. A letter above each item classifies the team on that item, and a small stacked strip at the top left summarizes the shares, with synergy on 35 percent of items, augmentation on 60 percent, and interference on 5 percent. Figure 5. Teams track the assistant where it is strong and the unaided participant where the assistant is weak. Per-item accuracy for the unaided group, gpt-5.6-luna alone, and the group working with it, with the 40 items ordered by LLM competence (the proportion of one hundred benchmark runs answered correctly). Vertical lines are 95% credible intervals on the item mean. Letters classify each item by point estimates, not resolved differences, with S where it exceeded both of its parts (synergy, 14 items), A where it exceeded one (augmentation, 24), and I where it exceeded neither (interference, 2). The strip at the top left gives the three shares.Point-and-interval plot of three accuracy series across 40 items ordered by LLM competence. The LLM series rises from zero to one across the ordering, the unaided group series is roughly flat, and the assisted series follows the unaided group series on the left of the plot and the assistant series on the right. A letter above each item classifies the team on that item, and a small stacked strip at the top left summarizes the shares, with synergy on 35 percent of items, augmentation on 60 percent, and interference on 5 percent. Table 2. Realized synergy never resolves positive in any task, capture is uneven across them, and error overlap resolves only on the image-delivered tasks. Per-task estimates for the three item-level quantities, with tasks ordered by LLM competence. Note. S compares assisted accuracy with the better component on each item. Capture is relative to independent-error headroom above unaided accuracy. It is modelled on an unbounded scale, so an interval may extend above 1, and mental rotation leaves almost no headroom, so its ratio is unidentified. Excess co-failure is the amount by which the pair failed together above the product of its two failure rates. Bracketed values are 95% credible intervals. An interval excluding zero is what we call resolved, and resolved estimates are set in bold throughout the paper’s tables. Matrices and mental rotation were delivered as images, letter strings and syllogisms as text. Every task contributes 10 items. Realized synergy S Synergy capture Excess co-failure Task Est. 95% HDI Est. 95% HDI Est. 95% HDI Mental rotation −.034-.034 [−.100-.100, .027] .071 [−.483-.483, .547] .062 [.035, .090] Matrices .015 [−.049-.049, .078] .643 [.383, .888] .074 [.046, .101] Letter strings −.153-.153 [−.218-.218, −.090-.090] .484 [.293, .689] .013 [−.014-.014, .041] Syllogisms −.074-.074 [−.139-.139, −.009-.009] .820 [.641, .997] −.003-.003 [−.031-.031, .023] Table 3. Benefits rise with competence, with uncertain effects in the lower competence bands. The effect function over the 160 item-by-assistant cells, the effect within each competence band, and the slope for each assistant separately. Note. The assisted–unaided difference is per-item team accuracy minus unaided accuracy on the same item. Cell-level quantities are on the accuracy scale. The trial-level slope is in log-odds. Resolved estimates are in bold. The zero crossing −b0/b1-b_0/b_1 and the two lowest competence bands are unresolved, so these estimates do not identify a sharp decision threshold. Quantity Estimate [95% HDI] The effect function Effect at mean competence .200 [.112, .277] Slope on competence .401 [.193, .612] Effect where the assistant always fails −.103-.103 [−.201-.201, −.014-.014] Zero crossing .238 [−.153-.153, .527] Effect within competence band LLM right on ≤ .2 of runs −.020-.020 [−.135-.135, .103] .2–.5 .056 [−.052-.052, .158] .5–.8 .181 [.076, .283] ≥ .8 .283 [.183, .385] Slope, per assistant gpt-5.6-luna (low) .487 [.378, .597] gemini-3.6-flash .447 [.341, .549] claude-opus-4-8 .382 [.280, .486] kimi-k3 .264 [.142, .382] Because the four assistants exhibit different item-level competence profiles (Figure 1), we can estimate how the benefit of access varies with LLM competence by regressing the item-level assisted–unaided difference on the assistant’s competence on that item. The effect function (Figure 6) relates assisted-minus-unaided accuracy to benchmark competence across the 160 item–assistant cells. Its positive slope indicates larger benefits on items the assistant handles better. Positive effects resolve in the two upper bands. The lower bands remain compatible with both benefit and harm (Table 3). The fitted zero crossing is .238.238 [−.153-.153, .527.527], with 91.7% of posterior mass inside [0,1][0,1]. The .50.50 boundary defines our reporting bands.22 2 The task-order adjustment is reported in Appendix E. Within-task trial slopes are positive in all four tasks, from 2.1772.177 [1.5251.525, 2.8302.830] in letter strings to 3.0113.011 [1.8461.846, 4.1784.178] in syllogisms. These slopes describe average trends across items within each task. A scatter plot of the per-item assisted--unaided difference against the assistant's competence on that item, with one upward fitted line and its credible band. Points from all four assistants are interleaved rather than separated, with no assistant occupying a distinct region. Figure 6. Assisted–unaided differences increase with LLM competence. The 160 item–assistant cells, coloured by assistant, with a pooled fit and 95% interval. Negative values indicate lower assisted accuracy. The pooled line summarizes the trend; assistant-specific slopes are reported separately.A scatter plot of the per-item assisted–unaided difference against the assistant's competence on that item, with one upward fitted line and its credible band. Points from all four assistants are interleaved rather than separated, with no assistant occupying a distinct region. To describe differences across observed score levels, we split participants into thirds by battery accuracy within each group, then compared assisted and unaided strata twice, once on the items where their assistant was reliable (≥.80≥.80 of its runs correct), and once on those where it was unreliable (≤.40≤.40). On reliable items, assisted–unaided differences were largest in the lowest score stratum. Gains fall steadily as scores rise, from +.451+.451 [.409.409, .492.492] in the lowest third to +.163+.163 [.121.121, .207.207] in the highest. Where it was unreliable, the losses do not mirror those gains. It is the middle third that pays, −.137-.137 [−.206-.206, −.071-.071], while the lowest does not resolve and the highest comes out ahead (Figure 7). Refitting one group at a time, the middle-third loss resolves for gpt-5.6-luna, gemini-3.6-flash and kimi-k3, between −.119-.119 and −.160-.160. No third resolves for claude-opus-4-8. But “weak items” means different items for each assistant, drawn from tasks of varying difficulty. Here, the two most dissimilar sets share none. Figure 7. Assisted–unaided differences vary across observed score strata and assistant reliability. Assisted minus unaided accuracy by within-group score tertile, shown separately for items where the assistant is reliable (≥.8≥.8) and unreliable (≤.4≤.4). Note. Tertiles use observed scores within each group, not pretreatment ability. Contrasts come from separate trial-level models for reliable and unreliable items. Bars are 95% credible intervals, and solid fill marks a contrast whose interval excludes zero. The interval spans benefit and harm only in the lowest score tertile on unreliable items, marked by a hollow bar.Grouped bar chart of assisted minus unaided accuracy across three score tertiles. Reliable-item bars are positive throughout and shrink from low to high scores, from about .45 to .16. Unreliable-item bars are near zero and unresolved in the lowest stratum, clearly negative in the middle stratum at about $-.13$, and slightly positive in the highest stratum at about .07. 4.2. Deference and Answer Switching (RQ2) The extraction pass classified all 13,920 assisted trials from 348 participants by extracted advice, its scored correctness, and submitted-answer agreement. On 374 trials the assistant never settled on an answer, leaving 13,546 on which there was something to take up. The proportions below are observed trial shares with exact Beta intervals that ignore the clustering of trials within participants and items. Refit as a Bernoulli mixed model with the same crossed random effects as the task models, estimates with participant and item effects set to zero are .946.946 [.930.930, .960.960] for correct advice, .771.771 [.715.715, .821.821] for incorrect, and a discrimination of .175.175 [.137.137, .217.217] — a different conditional estimand from the observed trial shares. The extractor identified different first and final answers on .082 of advice-bearing trials, with revisions mostly improving scored accuracy. Of these, 749 moved a wrong first answer to a right one against 236 moving the other way, so three revisions in four improved the answer. Extracted advice accuracy rose from .713 [.706, .721] at the assistant’s first answer to .751 [.744, .759] at its last. The revision analysis pools all follow-up types. Participants submitted answers matching the assistant’s advice on .908 [.902, .913] of the trials where it was correct and on .743 [.730, .756] of the trials where it was wrong, a discrimination of just .165 [.150, .179] (Table 4). Incorrect answers were not rare. Participants met a median of nine of them over the battery. A quarter nonetheless never declined a single one. Those who did decline were usually right to, participants who held their own answer against incorrect advice were correct .678 [.658, .701] of the time. We use deference operationally for submitting an answer matching the assistant’s advice, including independent agreement. Recorded answer switches identify replacement of an earlier judgment. Deference varied by task. The four task rates run from .751 on letter strings to .947 on syllogisms, and they do not follow the tasks’ advice quality, which itself differs by nearly four tenths (Table 4). Letter-string analogies had the second-best advice quality of the four tasks but drew the least deference. The task ordering is sensitive to adoption-coding error.33 3 An audit-calibrated sensitivity analysis retained frequent wrong-advice adoption (.787.787, 95% sensitivity interval [.731,.828][.731,.828]), but left the letter-string–syllogism contrast uncertain. It assumes coding-error rates transfer within task, advice-correctness and extracted-adoption strata, pooling groups. Advice correctness itself is held fixed. Within participants, deference is positively associated with current-task advice accuracy (b=.093b=.093 [.035.035, .152.152]). The association with previous-task accuracy remains uncertain (b=.063b=.063 [−.065-.065, .194.194]). Holding task fixed, deference also rises with item competence (b=1.331b=1.331 [0.9830.983, 1.6951.695] in log-odds). Its probability-scale change depends on the task’s baseline deference, so it cannot be summarized by a universal four-point change. Both task-level differences and within-task sensitivity contribute to the pattern. Participants therefore showed some sensitivity to item-level reliability. As a descriptive illustration for the 40-item comparison, deference was .858 [.824, .892] where the assistant outperformed the unaided group and .893 [.833, .950] where it did not. The uncertainty does not support a clear difference, and the unaided-group comparison is not a measure of each participant’s own expected accuracy. Trials with a pre-reply answer show where switching occurred. On the 4,873 trials carrying an answer entered before the reply, participants switched from correct to incorrect answers more often on low-competence items — .198 [.084, .566] of held correct answers in the lowest competence band against .024 [.008, .113] in the highest — and corrected a wrong one least often there, .244 [.121, .394] rising to .676 [.552, .794] (Figure 8). This pattern is also compatible with differing exposure to incorrect advice. Two lines across four competence bands. The probability of giving up a correct answer falls from about .20 to about .02, while the probability of correcting a wrong answer rises from about .24 to about .68. Figure 8. Correct-to-incorrect switching is more frequent on low-competence items. Probability of switching away from a correct answer, and of switching to a correct answer, after the assistant replied, by LLM competence on the item. Note. Error bars are 95% credible intervals from models with crossed random effects for participant and item. Probabilities condition on initial correctness, not on disagreement with advice. The analysis covers the 35% of assisted trials with an answer recorded before the assistant’s first reply. The test each series carries is its change across the competence range, −.174-.174 [−.456-.456, −.069-.069] for abandoning and .425 [.296, .550] for correcting. The two series do not cross at any level of competence.Two lines across four competence bands. The probability of giving up a correct answer falls from about .20 to about .02, while the probability of correcting a wrong answer rises from about .24 to about .68. 4.3. Confidence, Calibration, and the Policy Ladder (RQ3) Confidence concerned the submitted answer and was rated after consultation, including in the pre-reply-answer sensitivity check. An earlier-sample sensitivity analysis includes 1,686 rated trials with an answer entered before the reply. The confidence gap is .052.052 [.030.030, .074.074], versus .105.105 [.095.095, .114.114] unaided. Table 4. Extracted deference rates differ across tasks. Advice accuracy, deference and overreliance by task, on the 13,546 classified trials that carried committed advice, pooled over all four assistants and ordered by advice accuracy. Note. Advice correct is the share of trials on which the assistant’s extracted answer was right, and deference the share on which the participant submitted it. Overreliance is deference where the advice was wrong, and discrimination is deference to right minus deference to wrong advice. Task rows are model-based estimates. The pooled row gives observed trial shares. Interval widths are summarized below. Task Trials Advice correct Deference Overreliance Discrim. Syllogisms 3,402 .969 .947 .851 .100 Letter strings 3,384 .846 .751 .482 .319 Mental rotation 3,335 .652 .902 .806 .154 Matrices 3,425 .577 .867 .775 .161 All tasks 13,546 .751 .859 .743 .165 Widest intervals are on advice accuracy in matrices and mental rotation (±.12±.12), which pools four assistants that differ sharply there, then ±.05±.05 on overreliance and discrimination in syllogisms, where only 144 trials carried wrong advice, and ±.04±.04 on discrimination in letter strings. Every other interval is narrower than ±.03±.03. Overall, stated confidence was M=77.0M=77.0, SD=25.1SD=25.1 against an accuracy of .690. Mean confidence varied little across competence bands while accuracy rose (Figure 9). The pooled item-level confidence slope is −.010-.010 [−.020-.020, .000.000], and the within-task estimate is .000.000 [−.011-.011, .012.012] (Appendix E). Confidence discriminated correct from incorrect answers overall (b=.329b=.329 [.264.264, .396.396]). In the weakest band its slope was .111.111 [−.146-.146, .375.375], compared with point estimates of .283.283–.380.380 in the other bands. The weak-band interval spans negative and positive confidence–correctness slopes. On the trials that both followed the intended sequence and carried an extracted verdict, confidence was fit on four cells crossing what was submitted (own answer vs. LLM’s) with whether the advice was right. Among the 11,741 adoption trials where participants handed in the assistant’s answer, accuracy was 1.000 when the advice was right, and .008 when it was wrong (the residual reflects trials the extractor read as submitted-as-advised that were not, at a rate consistent with the audit in Section 3.6), yet stated confidence was 78.4 and 78.0. These descriptive means are similar. Submitting the assistant’s answer rather than one’s own was associated with 11.611.6 [10.310.3, 12.812.8] points higher confidence, and 48% of participants were on average more confident when the advice was wrong. In the selected own-answer robustness analysis, capture changes by −.028-.028 [−.140-.140, .084.084] per SD of discrimination. This association remained imprecisely estimated. Two lines across four competence bands. Realized accuracy rises steeply from about .49 to about .86 while stated confidence stays almost flat near .75, so the vertical gap between them closes and then reverses. Figure 9. Stated confidence barely moves while accuracy swings by a third. Mean stated confidence and realized accuracy by LLM competence on the item, on trials following the intended sequence. Note. Post-consultation certainty ratings are divided by 100. Their gaps from accuracy are descriptive scale comparisons, not elicited probability calibration. The figures along the foot give the confidence–accuracy gap in each band, posterior median and 95% interval. Capped bars on the markers are 95% credible intervals.Two lines across four competence bands. Realized accuracy rises steeply from about .49 to about .86 while stated confidence stays almost flat near .75, so the vertical gap between them closes and then reverses. We next examine whether declining wrong advice is associated with metacognitive discrimination or with accuracy on trials where participants retained their own answer. These are observational predictors derived from participants’ behaviour during the study. Metacognitive discrimination was associated with less adoption of wrong advice, with a coefficient of −.184-.184 [−.363-.363, −.004-.004] in log-odds per SD, with an uncertain discrimination interaction. Selected own-answer accuracy was associated with these outcomes in separate models, with coefficients of −1.018-1.018 [−1.197-1.197, −.840-.840] for taking wrong advice and 1.7211.721 [1.5591.559, 1.8891.889] for discrimination (Table 7). These are observational associations. Own-answer accuracy partly overlaps the behaviour being predicted. Its association remained uncertain when source and outcome trials were separated (Table 7). Confidence discrimination, measured as the area under the receiver operating characteristic curve (AUC), exceeded .5 in 36 of 40 item-level point estimates, but the slopes relating AUC to synergy and the assisted–unaided difference remained uncertain. AUC counts correct–incorrect confidence pairs, with ties contributing one half. Participant AUC requires both outcomes and at least eight rated trials for deference models, twelve for capture. Participant-level AUC was modestly associated with capture, at .046.046 [.001.001, .093.093] per SD, with P=.98P=.98 for a positive association. The interval only narrowly excludes zero. Figure 10. Answer agreement and reference-policy gains describe different aspects of the interaction. (A) Submitted-answer agreement against the assistant’s item-level advantage, with task means. (B) Three reference policies, each as a gain over observed performance. Note. Task-level selection gains .035.035 [.004.004, .066.066], item-level selection .061.061 [.030.030, .092.092], and an independent-error oracle .141.141 [.110.110, .172.172].Two panels. The left panel plots submitted-answer agreement against the assistant's advantage across 160 item–assistant cells, with larger points marking task means. The right panel shows three reference rules as gains over the observed teams, each with an interval above zero. The policy ladder compares observed performance with retrospective reference rules (Figure 10B). Task- and item-aware references select between estimated unaided-group and benchmark accuracy. Both selection and evaluation use the same data, so their gains may be optimistic.44 4 Two-fold held-out selection with 2,000 participant/run resamples, reselecting each policy, gave task gain .035.035 [.017.017, .052.052] and item gain .058.058 [.038.038, .074.074]. These 95% percentile intervals condition on the same 40 items. The trial oracle additionally assumes independent errors. These are conditional reference gains, not demonstrated improvements from deployable policies. The full specification is in Appendix E. The confidence gap between correct and incorrect submitted answers was lower in every assisted group than in the unaided group (Table 6). These are between-group comparisons of post-advice confidence. Mean confidence was negatively associated with competence across items in each group, but the association was carried between tasks and was inconsistent within them (Appendix E). 4.4. Team Accuracy Across the Four Assistants (RQ4) Table 5. Solo and assisted point-estimate rankings differ. Pass-through also varies by group. Per-assistant solo accuracy, team accuracy, realized synergy, and pass-through, ordered by solo accuracy. Note. All values are posterior medians with 95% credible intervals. Resolved estimates are in bold. Pass-through is the slope of team correctness on the assistant’s item competence, in log-odds, partially pooled across assistants. Slopes are group-specific. Realized synergy S compares the team with the better component separately on each item. The pooled row is over the 348 assisted participants. The unaided row is shown for reference only. The lower section gives the six pairwise differences in pass-through. Assistant n LLM alone Team Realized synergy S Pass-through gemini-3.6-flash 52 .761 [.674, .845] .781 [.720, .841] −.055-.055 [−.091-.091, −.016-.016] 2.81 [2.37, 3.25] claude-opus-4-8 61 .735 [.653, .816] .733 [.673, .791] −.055-.055 [−.092-.092, −.017-.017] 2.21 [1.80, 2.61] kimi-k3 56 .730 [.648, .812] .691 [.631, .750] −.087-.087 [−.129-.129, −.046-.046] 1.39 [.89, 1.88] gpt-5.6-luna (low) 179 .713 [.627, .796] .711 [.651, .770] −.052-.052 [−.088-.088, −.013-.013] 2.76 [2.39, 3.14] unaided 187 — .528 [.474, .583] — — pooled 348 .735 [.633, .834] .728 [.622, .828] −.062-.062 [−.118-.118, −.005-.005] 2.46 [2.18, 2.75] Pairwise differences in pass-through (log-odds) gemini-3.6-flash −- kimi-k3 1.43 [0.82, 2.03] gpt-5.6-luna −- kimi-k3 1.37 [.87, 1.88] claude-opus-4-8 −- kimi-k3 .82 [.29, 1.35] gemini-3.6-flash −- claude-opus-4-8 .60 [.05, 1.13] gpt-5.6-luna −- claude-opus-4-8 .55 [.13, .98] gemini-3.6-flash −- gpt-5.6-luna .05 [−.45-.45, .57] Table 6. Post-advice confidence separates right from wrong answers less in every assisted group than unaided, and the drop does not scale with what the assistant knows. The confidence gap between correct and incorrect answers, per group, and each assisted group’s drop against unaided. Note. The gap is the posterior difference in mean stated confidence (rescaled to 00–11) between correct and incorrect answers, from one model over every rated trial. Drops are differences from the unaided gap on the same scale. Bracketed values are 95% credible intervals. Every interval excludes zero, so all estimates are set in bold. Group Confidence gap Drop vs. unaided unaided .102.102 [.092, .112] — gpt-5.6-luna (low) .062.062 [.051, .073] −.040-.040 [−.054-.054, −.026-.026] claude-opus-4-8 .057.057 [.037, .075] −.046-.046 [−.067-.067, −.025-.025] gemini-3.6-flash .059.059 [.037, .080] −.043-.043 [−.066-.066, −.019-.019] kimi-k3 .053.053 [.035, .072] −.049-.049 [−.070-.070, −.028-.028] Table 7. Selected own-answer accuracy and metacognitive AUC are associated with resistance to wrong advice. Posterior effect of each candidate predictor, per standard deviation of that predictor. Note. Overreliance and discrimination are as defined in Table 4. Both are trial-level outcomes, so Panel A is in log-odds. The two nns give participants in the overreliance and discrimination fits, respectively. AUC coefficients adjust for shrunken own-work ability. Own-answer coefficients are unadjusted. The disjoint estimate uses declined correct-advice trials (median three per person) to predict separate wrong-advice trials. Capture is the realised share of headroom, a participant-level quantity, so Panel B is on the capture scale itself. Bracketed values are 95% credible intervals, P is the posterior probability that the effect carries the sign of its median, and resolved estimates are in bold. Panel A. Who declines wrong advice (log-odds per SDSD) Predictor Overreliance Discrimination Metacognitive discrimination (AUCAUC), n=320/321n=320/321 −.184-.184 [−.363-.363, −.004-.004] −.003-.003 [−.127-.127, .117] P=.98P=.98 P=.52P=.52 Own-answer accuracy, n=333/334n=333/334 −1.018-1.018 [−1.197-1.197, −.840-.840] 1.7211.721 [1.559, 1.889] P>.999P>.999 P>.999P>.999 Own-answer accuracy, disjoint trials, n=283n=283 −.063-.063 [−.283-.283, .154.154] — Panel B. Who captures the headroom (capture per SDSD, n=274n=274) Predictor Synergy capture Metacognitive discrimination (AUCAUC) .046.046 [.001, .093], P=.98P=.98 Overconfidence, raw −.223-.223 [−.261-.261, −.185-.185], P>.999P>.999 Overconfidence, own accuracy held fixed −.001-.001 [−.013-.013, .010], P=.59P=.59 Four assisted groups are compared against one unaided group. Table 5 reports solo and assisted accuracy, item-wise realized synergy, and pass-through, the association between item competence and assisted accuracy. Figure 1 shows task profiles. Gemini and Opus rank first and second both alone and with participants, while Luna overtakes Kimi with assistance. Luna also has a relatively steep pass-through slope despite its lower solo score. Kimi has the lowest slope on both competence measures. These point-estimate rankings remain descriptive. On the realized-advice axis, the slope spread narrows from roughly twofold to 1.4×1.4×, and only the gpt-5.6-luna–kimi-k3 contrast resolves (Appendix E). The matched four-assistant benchmark provides a common reference for studying solo and assisted performance. At the trial level, the pooled competence slope is 2.4652.465 [2.1822.182, 2.7502.750] in log-odds. For a reference change in benchmark competence from .25.25 to .75.75, the model predicts an accuracy increase of .263.263 [.222.222, .303.303] with participant and item effects set to zero, equivalent to .526.526 [.443.443, .605.605] of the competence change. This roughly half-sized pass-through is specific to that contrast and evaluation convention. The probability-scale fraction varies with baseline competence. The more competent assistant belongs to the better team on .776.776 [.701.701, .857.857] of item-level pairs that differ. The task profiles also differ (Figure 1). Mental-rotation benchmark scores span thirteen points while assisted scores span three, whereas near-ceiling syllogism benchmarks accompany a wider assisted spread. These descriptive comparisons motivate evaluating performance with people alongside performance alone. 5. Discussion We asked when a person and an assistant working together outperform both components. Assisted accuracy fell below the item-wise better-component reference, while the battery-level assisted–LLM difference remained uncertain. An AI assistant is not uniformly reliable. On two items of the same kind, it can be near-certain on one and no better than guessing on the next. We measured that variation by running each assistant repeatedly on the same items. Participants’ deference varied substantially by task and also increased with item competence. Consistent with Vaccaro et al. (2024), we found no clear advantage over the better component. Riedl and Weidmann’s modeled AI benefit instead compares assisted with unaided performance (Riedl and Weidmann, 2026). 5.1. Where Consulting Pays (RQ1) Complementarity requires that the person and the assistant fail on different questions. Comparing the answer a participant had entered before the reply arrived against the advice that then came yields two separately recorded answers on the same item. Excess co-failure was positive on mental rotation and matrices, while error dependence remained uncertain on letter strings and syllogisms. Thus, complementarity guarantees very little in terms of overall team performance. For letter strings, the assistant was most often reliably correct, although its errors did not always coincide with participants’ errors. Teams, however, scored below the assistant alone. Whether errors differ sets how much a pair could gain. A participant recognizing where the assistant fails may help determine how much they realize. In line with Dell’Acqua et al. (2026), we find assisted–unaided differences vary with item competence. Their jagged frontier runs between whole tasks. Ours runs between items of the same task, which is one level further. Understanding the frontier at that level may help explain a split in the field-experiment literature, where assistance lifts the weakest workers most in some studies and widens the gap in others (Noy and Zhang, 2023; Brynjolfsson et al., 2025; Otis et al., 2025). Both can be true, depending partly on the mix of items. An accurate assistant offers more potential gain when unaided accuracy is low. Where reliability varies across items, wrong advice can displace correct answers. The mix of items may therefore contribute to these different outcomes. 5.2. Deference Varies by Task and Tracks Item Competence (RQ2) Li and Steyvers’ (Li and Steyvers, 2026) model of human–AI synergy, grounded in metacognitive sensitivity, derives conditions under which confidence-informed combination improves on either component. One interpretation of our results is a granularity mismatch. A direct test of the combination rule would require assistant-confidence measurements and implementation of the rule. One possible explanation concerns how each source is weighted by its reliability. LLM competence differed item by item, while deference showed task-level differences. Deference also increased with item competence. The size of that increase depended on the task’s baseline deference. It appears that users represent an assistant as having a single generalizable accuracy rather than multiple (Kelly et al., 2023). Such a belief could favor task-level heuristics, consistent with bounded rationality (Simon, 1955; Oulasvirta et al., 2022). Item-level feedback could help correct that belief. Participants received advice without calibrated item-level reliability estimates. In contemporary LLM interactions, a reply arrives in the same form whether it is right or wrong, and is often overly confident (Zhou et al., 2024), making confidence of expression an unreliable cue to correctness. This account invites an objection regarding what is rational for the human in interaction with AI. On the ecological-rationality view, a rule is judged not in the abstract but by its fit to the environment it runs in (Payne et al., 1993), and a simple rule matched to its environment routinely beats an elaborate one (Gigerenzer and Brighton, 2009). On the resource-rational view, the cost of the finer rule counts against it (Lieder and Griffiths, 2020). Following a task-level rule may be reasonable when reliability is hard to judge for each problem. Participants may therefore have adapted to limited information rather than failed to reason. The policy ladder estimates gains from more selective use of advice, assuming the relevant correctness information is available. This estimated potential is what we call “available but unclaimed.” The design challenge is, thus, to help users judge reliability without demanding excessive effort. Whether such support delivers the estimated gains remains to be tested. As assistants become more reliable, routine deference may become more reasonable, while users have less incentive to check the remaining errors. 5.3. Confidence Alone May Not Supply the Missing Signal (RQ3) Our findings motivate testing two approaches, supporting users’ assessment of their own answers and providing information about the assistant’s reliability. Participants rated confidence in their submitted answers. Post-advice metacognitive discrimination had a modest positive association with capture. The second approach is to display the assistant’s reliability on the screen, and here Rieger et al. (2026) give reason for caution. In their study, explanations were most helpful for items of moderate difficulty, providing the least benefit when errors were obvious and when items were too difficult to evaluate the explanation. Our results likewise suggest examining how item difficulty affects the usefulness of reliability information. Our participants’ confidence distinguished their correct from incorrect answers, yet they still frequently adopted incorrect advice. Peng et al. (2025) identify a related theoretical limit. Individual calibration need not hold after conditioning on another agent’s prediction. Their theorem concerns calibrated binary predictions. It clarifies why the ladder’s additional information matters. Task/item selection uses comparative performance, while the oracle assumes independent errors. Whether displayed confidence can support comparable gains is a design question that can be empirically tested. 5.4. How Solo Competence Relates to Assisted Performance (RQ4) Luna’s rise above Kimi with assistance, despite its lower solo score, and Kimi’s consistently lower pass-through suggest differences in usefulness across the observed groups. The pattern does not fully establish a more ergonomic model. This extends ChatBench’s finding that solo-performance differences can shrink in interaction (Chang et al., 2025). Following Steyvers et al. (2022) and Hemmer et al. (2025), we distinguish gains over unaided performance from outperforming both components. On most items, the more competent assistant is also the one whose team does better, on .776.776 [.701.701, .857.857] of the pairs that differ, so the ranking between models largely holds. Benefiting from a better assistant, however, is not the same as outperforming that assistant through selective combination. An assistant’s advantage can be lost either on the way to the user, if a model performs worse in a chat window than on a benchmark, or after it arrives, in what the person does with the reply. We reduced the elicitation mismatch by matching stimuli and API settings. Each assistant received the stimulus message composed by the participants’ chat panel, replayed through the same API call with no system prompt or format instruction. However, benchmark histories reset between tasks and omit participant follow-up turns. The gap may, thus, reflect both elicitation differences and participants’ use of advice. 5.5. Implications For AI development. These data compare solo and assisted performance on the same items. Joint evaluation has both conceptual (Haupt and Brynjolfsson, 2025) and empirical precedents (Chang et al., 2025; Riedl and Weidmann, 2026), while benchmark construct validity remains a concern (Bean et al., 2025). Pass-through adds a measure of how assisted accuracy varies with item competence under a specified protocol. Estimating correctness on new queries requires a proxy for the answer key used here. Repeated sampling offers self-consistency as one candidate whose relation to correctness needs evaluation. At the worked competence contrast of .25 to .75, predicted assisted accuracy increased by .263. This measures responsiveness, not a fraction of available value lost. Even selecting the better component can yield a half-sized response when unaided accuracy exceeds LLM competence at the lower endpoint. Policy comparisons separately estimate unrealized gains. Interactive evaluation should assess assisted accuracy alongside solo accuracy and users’ discrimination between correct and incorrect advice. For design. An answer-first design provides a competing judgment against which to evaluate advice, rather than only a confidence rating. The own-answer association was uncertain when predictor and outcome trials were separated, leaving answer-first design as a hypothesis to test. Making acceptance a deliberate action reduces overreliance, and users rate those designs lowest (Buçinca et al., 2021). Eliciting an answer first differs by changing when the interface acts rather than adding friction to acceptance—the cost occurs before the reply, during an additional step that may itself require effort. A controlled trial could request a revisable answer or an explicit inability to answer before revealing advice, measuring accuracy, harmful switches, effort, and perceived control. Reliability displays and routing should compare the assistant with the user’s available alternative, while preserving the user’s final choice. Whether reliance is warranted is not a fixed property of the assistant. It varies with the item at hand. For augmentation. These results belong to an old question. Engelbart (1962) claimed that intellect is augmented by a system in which a person, a tool, and a method are adapted to one another, not by the tool alone. Our four assistants differed in the tool and left the method untouched, but solo performance did not translate one-for-one into assisted performance. The findings suggest evaluating methods that help people decide when to follow advice, comparing human judgment against always following the assistant on matched trials, not only against unaided performance. 5.6. Limitations & Future Work Three limitations bound these results. First, results concern four assistants, one interface, and 40 reasoning items under mandatory consultation. Groups were recruited through separate postings, with differing periods and leaderboard sizes. Measured demographics were similar, but unmeasured ability, motivation, and period effects may contribute to group differences. The separate unaided reference estimates group performance rather than each participant’s alternative. Optional consultation and longer-term or expert use require further study. Second, the estimated gains depend partly on uncertainty in LLM competence, whose binomial standard error is at most about .05.05. We assessed this uncertainty by resampling participants and benchmark runs while keeping the 40 items and four groups fixed. Held-out selection checked whether choosing and evaluating policies on the same observations inflated their gains. These checks support our claim of potential gains from more selective advice use on this battery. Achieving those gains in practice remains an open question. The oracle assumes independent errors, and users would need reliability information that is both available and worth the effort of using. Third, syllogisms provide few incorrect replies, limiting precision for responses to wrong advice. Pre-reply answers are self-selected, and own-answer accuracy uses few, selected trials per participant, including trials contributing to the deference outcome. Randomizing an answer-first requirement would help distinguish the effect of forming an answer from participant selection and shared-data effects. Collecting pre-advice confidence would allow a direct comparison with the post-advice judgments studied here. 6. Conclusion Assisted participants generally outperformed the unaided group but did not clearly exceed their assistants. Gains increased with item competence, while deference reflected both task differences and within-task reliability. Post-advice confidence discriminated correctness, but did not establish a precise account of who captured more of the reference headroom. Retrospective policies estimated additional gains under assumptions about component accuracy and error dependence. These findings motivate testing interactions that support selective deference and independent reasoning, with effectiveness measured against both unaided and assistant-alone performance. Use of Generative AI We used large language models (Anthropic’s Claude Opus 5 and Claude Fable 5.1, OpenAI’s Astra with medium reasoning effort, z-ai/glm-5.2, moonshotai/kimi-k2.6, and qwen/qwen3-235b-a22b-2507) to assist with copy-editing the manuscript and with writing plotting and analysis code. All study design, data collection, statistical analysis, and interpretation were carried out by the authors, who verified every reported number against the analysis output and take full responsibility for the content. Acknowledgements. This work was supported by the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme, AmplifAI (grant agreement No. 101217557). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. Daniela Fernandes is funded by the Finnish Doctoral Program Network in Artificial Intelligence, AI-DOC (decision number VN/3137/2024-OKM-6). References Anderson (1990) J. R. Anderson The adaptive character of thought. Lawrence Erlbaum Associates, Hillsdale, NJ. External Links: ISBN 9780805804195 Cited by: §2.2. Bansal et al. (2021) G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. Weld Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Chi ’21, New York, NY, USA, p. 1–16. External Links: ISBN 978-1-4503-8096-6, Link, Document Cited by: §2.1, §2.2, §2.3, §2.3. Barrett et al. (2018) D. Barrett, F. Hill, A. Santoro, A. Morcos, and T. Lillicrap Measuring abstract reasoning in neural networks. In Proceedings of the 35th International Conference on Machine Learning, J. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, p. 511–520. External Links: Link Cited by: §3.2.1. Bean et al. (2025) A. M. Bean, R. O. Kearns, A. Romanou, F. S. Hafner, H. Mayne, J. Batzner, N. Foroutan, C. Schmitz, K. Korgul, H. Batra, O. Deb, E. Beharry, C. Emde, T. Foster, A. Gausen, M. Grandury, S. Han, V. Hofmann, L. Ibrahim, H. Kim, H. R. Kirk, F. Lin, G. K. Liu, L. Luettgau, J. Magomere, J. Rystrøm, A. Sotnikova, Y. Yang, Y. Zhao, A. Bibi, A. Bosselut, R. Clark, A. Cohan, J. Foerster, Y. Gal, S. A. Hale, I. D. Raji, C. Summerfield, P. H.S. Torr, C. Ududec, L. Rocher, and A. Mahdi Measuring what matters: construct validity in large language model benchmarks. In Advances in Neural Information Processing Systems 38, Datasets and Benchmarks Track, Note: arXiv:2511.04703 External Links: Document Cited by: §5.5. Bianco et al. (2021) R. Bianco, G. Mills, M. de Kerangal, S. Rosen, and M. Chait Reward enhances online participants’ engagement with a demanding auditory task. Trends in Hearing 25. External Links: ISSN 2331-2165, Link, Document Cited by: §3.1. Boldt and Gilbert (2019) A. Boldt and S. J. Gilbert Confidence guides spontaneous cognitive offloading. Cognitive Research: Principles and Implications 4 (1), p. 45. External Links: Document Cited by: §2.2. Bommasani et al. (2026) R. Bommasani, S. H. Bana, K. A. Creel, D. Jurafsky, and P. Liang Algorithmic monocultures in hiring. In The 2026 ACM Conference on Fairness, Accountability, and Transparency, p. 6351–6382. External Links: Document Cited by: §2.1. Bonaccio and Dalal (2006) S. Bonaccio and R. S. Dalal Advice taking and decision-making: an integrative literature review, and implications for the organizational sciences. Organizational Behavior and Human Decision Processes 101 (2), p. 127–151. External Links: Document Cited by: §2.2. Branoff (2009) T. J. Branoff Spatial Visualization Measurement: A Modification of the Purdue Spatial Visualization Test — Visualization of Rotations. The Engineering Design Graphics Journal 64 (2). External Links: ISSN 1949-9167, 0046-2012, Link, Document Cited by: §3.2.1. Bringsjord (2011) S. Bringsjord Psychometric artificial intelligence. J. Exp. Theor. Artif. Intell. 23 (3), p. 271–277. External Links: Document, Link Cited by: §2.1. Brynjolfsson et al. (2025) E. Brynjolfsson, D. Li, and L. Raymond Generative AI at work. The Quarterly Journal of Economics 140 (2), p. 889–942. External Links: Document Cited by: §5.1. Buçinca et al. (2021) Z. Buçinca, M. B. Malaya, and K. Z. Gajos To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making. Proc. ACM Hum.-Comput. Interact. 5 (Cscw1). External Links: Link, Document Cited by: §2.2, §5.5. Carpenter et al. (1990) P. A. Carpenter, M. A. Just, and P. Shell What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices Test.. Psychological Review 97 (3), p. 404–431 (en). External Links: ISSN 1939-1471, 0033-295X, Link, Document Cited by: §A.1, §3.2.1. Chang et al. (2025) S. Chang, A. Anderson, and J. M. Hofman ChatBench: from static benchmarks to human-AI evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 26009–26038. External Links: Document, Link Cited by: 3rd item, §1, §2.1, §2.3, §5.4, §5.5. Chollet (2019) F. Chollet On the measure of intelligence. arXiv preprint arXiv:1911.01547. Cited by: §2.1. Colombatto and Fleming (2023) C. Colombatto and S. Fleming Illusions of confidence in artificial systems. p. . External Links: Document Cited by: §2.2. Dell’Acqua et al. (2026) F. Dell’Acqua, E. McFowland, E. Mollick, H. Lifshitz-Assaf, K. C. Kellogg, S. Rajendran, L. Krayer, F. Candelon, and K. R. Lakhani Navigating the jagged technological frontier: field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality. Organization Science 37 (2), p. 403–423. External Links: Document Cited by: §5.1. Depaoli and van de Schoot (2017) S. Depaoli and R. van de Schoot Improving transparency and replication in Bayesian statistics: the WAMBS-checklist. Psychological Methods 22 (2), p. 240–261. External Links: Document Cited by: §3.7. Eisape et al. (2024) T. Eisape, M. Tessler, I. Dasgupta, F. Sha, S. van Steenkiste, and T. Linzen A systematic comparison of syllogistic reasoning in humans and language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 8425–8444. External Links: Link, Document Cited by: §3.2.1. Engelbart (1962) D. C. Engelbart Augmenting human intellect: a conceptual framework. Menlo Park, CA, p. 21. External Links: Document Cited by: §5.5. Fernandes et al. (2026) D. Fernandes, S. Villa, S. Nicholls, O. Haavisto, D. Buschek, A. Schmidt, T. Kosch, C. Shen, and R. Welsch AI makes you smarter but none the wiser: the disconnect between performance and metacognition. Computers in Human Behavior 175, p. 108779. External Links: Document Cited by: §1, §2.2. Gigerenzer and Brighton (2009) G. Gigerenzer and H. Brighton Homo heuristicus: why biased minds make better inferences. Topics in Cognitive Science 1 (1), p. 107–143. External Links: Document Cited by: §5.2. Gilbert et al. (2020) S. J. Gilbert, A. Bird, J. M. Carpenter, S. M. Fleming, C. Sachdeva, and P. Tsai Optimal use of reminders: metacognition, effort, and cognitive offloading.. Journal of Experimental Psychology: General 149 (3), p. 501. Cited by: §2.2. Grinschgl et al. (2020) S. Grinschgl, H. S. Meyerhoff, and F. Papenmeier Interface and interaction design: How mobile touch devices foster cognitive offloading. Comput. Hum. Behav. 108, p. 106317. External Links: Document, Link Cited by: §2.2. Harris et al. (2020) A. Harris, J. McMillan, B. Listyg, L. Matzen, and N. Carter Measuring Intelligence with the Sandia Matrices: Psychometric Review and Recommendations for Free Raven-Like Item Sets. Personnel Assessment and Decisions 6 (3). External Links: ISSN 23778822, Link, Document Cited by: §3.2.1. Haupt and Brynjolfsson (2025) A. Haupt and E. Brynjolfsson Position: AI should not be an imitation game: centaur evaluations. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research. Cited by: §5.5. Hemmer et al. (2025) P. Hemmer, M. Schemmer, N. Kühl, M. Vössing, and G. Satzger Complementarity in human–AI collaboration: concept, sources, and evidence. European Journal of Information Systems 34 (6), p. 979–1002. External Links: Document Cited by: §1, §1, §2.1, §2.3, §5.4. Hofstadter et al. (1995) D. R. Hofstadter M. Mitchell et al. The copycat project: a model of mental fluidity and analogy-making. Advances in connectionist and neural computation theory 2, p. 205–267. Cited by: §3.2.1. Inkpen et al. (2023) K. Inkpen, S. Chappidi, K. Mallari, B. Nushi, D. Ramesh, P. Michelucci, V. Mandava, L. H. Vepřek, and G. Quinn Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making. Vol. 30. External Links: ISSN 1073-0516, Link, Document Cited by: §2.3, §2.3. Jiang et al. (2023) D. Jiang, X. Ren, and B. Y. Lin LLM-Blender: ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, p. 14165–14178. External Links: Document Cited by: §2.1. Johnson-Laird and Bara (1984) P.N. Johnson-Laird and B. G. Bara Syllogistic inference. Cognition 16 (1), p. 1–61. External Links: ISSN 0010-0277, Link, Document Cited by: §A.3, §3.2.1. Kelly et al. (2023) M. Kelly, A. Kumar, P. Smyth, and M. Steyvers Capturing humans’ mental models of AI: an item response theory approach. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’23, New York, NY, USA. External Links: Document, Link Cited by: §1, §2.3, §5.2. Khemlani and Johnson-Laird (2012) S. Khemlani and P. N. Johnson-Laird Theories of the syllogism: A meta-analysis.. Psychological Bulletin 138 (3), p. 427–457 (en). External Links: ISSN 1939-1455, 0033-2909, Link, Document Cited by: §3.2.1. Kim et al. (2025) E. M. Kim, A. Garg, K. Peng, and N. Garg Correlated errors in large language models. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 30038–30066. External Links: Link Cited by: §2.1. Kim et al. (2024) S. S. Kim, Q. V. Liao, M. Vorvoreanu, S. Ballard, and J. W. Vaughan " I’m not sure, but…": examining the impact of large language models’ uncertainty expression on user reliance and trust. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, p. 822–835. External Links: Document Cited by: §2.1. Kim et al. (2026) Y. Kim, K. Gu, C. Park, C. Park, S. Schmidgall, A. A. Heydari, Y. Yan, Z. Zhang, Y. Zhuang, Y. Liu, et al. Capable language models can outgrow the benefits of collaboration. Nature Machine Intelligence 8 (7), p. 1157–1172. External Links: Document Cited by: §2.1, §2.3. Klein and Kovacs (2024) B. Klein and K. Kovacs The performance of ChatGPT and Bing on a computerized adaptive test of verbal intelligence. PloS one 19 (7), p. e0307097. External Links: Document Cited by: §2.1. Kleinberg and Raghavan (2021) J. Kleinberg and M. Raghavan Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences 118 (22), p. e2018340118. Cited by: §2.1. Klingbeil et al. (2024) A. Klingbeil, C. Grützner, and P. Schreck Trust and reliance on AI - An experimental study on the extent and costs of overreliance on AI. Comput. Hum. Behav. 160, p. 108352. External Links: Document, Link Cited by: §2.1, §2.2. Lee and See (2004) J. D. Lee and K. A. See Trust in automation: designing for appropriate reliance. Human Factors 46 (1), p. 50–80. Note: PMID: 15151155 External Links: Document Cited by: §1. Legg and Hutter (2007) S. Legg and M. Hutter Universal intelligence: a definition of machine intelligence. Minds and Machines 17 (4), p. 391–444. External Links: Document Cited by: §2.1. Lewis and Mitchell (2024) M. Lewis and M. Mitchell Evaluating the robustness of analogical reasoning in large language models. arXiv. External Links: Document, Link Cited by: §A.4, §1, §3.2.1. Li et al. (2025) J. Li, Y. Yang, Q. V. Liao, J. Zhang, and Y. Lee As confidence aligns: understanding the effect of ai confidence on human self-confidence in human-ai decision making. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, p. 1–16. External Links: Document Cited by: §2.2. Li and Steyvers (2026) Z. Li and M. Steyvers Modeling the joint impact of human and AI metacognitive sensitivity on human–AI collaboration. Journal of Mathematical Psychology 129, p. 102988. External Links: Document Cited by: 2nd item, §2.1, §5.2. Lieder and Griffiths (2020) F. Lieder and T. L. Griffiths Resource-rational analysis: understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences 43, p. e1. External Links: Document Cited by: §2.2, §5.2. Logg et al. (2019) J. M. Logg, J. A. Minson, and D. A. Moore Algorithm appreciation: people prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes 151, p. 90–103. External Links: Document Cited by: §2.2. Lu and Yin (2021) Z. Lu and M. Yin Human reliance on machine learning models when performance feedback is limited: heuristics and risks. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, p. 1–16. External Links: Document Cited by: §2.2. Matzen et al. (2010) L. E. Matzen, Z. O. Benz, K. R. Dixon, J. Posey, J. K. Kroger, and A. E. Speed Recreating raven’s: software for systematically generating large numbers of raven-like matrix problems with normed properties. Behavior Research Methods 42 (2), p. 525–541. External Links: ISSN 1554-3528, Link, Document Cited by: §A.1, §3.2.1. McCoy et al. (2024) R. T. McCoy, S. Yao, D. Friedman, M. D. Hardy, and T. L. Griffiths Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences 121 (41), p. e2322420121. External Links: Document Cited by: §1. Miller (2024) E. Miller Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv. Note: Version Number: 1Other 14 pages External Links: Link, Document Cited by: §3.5. Noy and Zhang (2023) S. Noy and W. Zhang Experimental evidence on the productivity effects of generative artificial intelligence. Science 381 (6654), p. 187–192. External Links: Document, Link, https://w.science.org/doi/pdf/10.1126/science.adh2586 Cited by: §5.1. Otis et al. (2025) N. G. Otis, R. P. Clarke, S. Delecourt, D. Holtz, and R. Koning The uneven impact of generative AI on entrepreneurial performance. Note: Working paperVersion 3 External Links: Document Cited by: §5.1. Oulasvirta et al. (2022) A. Oulasvirta, J. P. P. Jokinen, and A. Howes Computational Rationality as a Theory of Interaction. In CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, S. D. J. Barbosa, C. Lampe, C. Appert, D. A. Shamma, S. M. Drucker, J. R. Williamson, and K. Yatani (Eds.), p. 359:1–359:14. External Links: Document, Link Cited by: §2.2, §5.2. Ozeki et al. (2024) K. Ozeki, R. Ando, T. Morishita, H. Abe, K. Mineshima, and M. Okada Exploring reasoning biases in large language models through syllogism: insights from the NeuBAROCO dataset. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 16063–16077. External Links: Link, Document Cited by: §3.2.1. Parasuraman and Riley (1997) R. Parasuraman and V. Riley Humans and automation: use, misuse, disuse, abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society 39 (2), p. 230–253. External Links: Document Cited by: §2.2. Payne et al. (1993) J. W. Payne, J. R. Bettman, and E. J. Johnson The adaptive decision maker. Cambridge University Press, Cambridge. External Links: ISBN 9780521415057, Document Cited by: §5.2. Pellert et al. (2023) M. Pellert, C. M. Lechner, C. Wagner, B. Rammstedt, and M. Strohmaier Ai psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, p. 17456916231214460. External Links: Document Cited by: §2.1. Peng et al. (2025) K. Peng, N. Garg, and J. Kleinberg A no free lunch theorem for human–AI collaboration. Proceedings of the AAAI Conference on Artificial Intelligence 39 (13), p. 14369–14376. External Links: Document Cited by: §2.1, §5.3. Pescetelli and Yeung (2021) N. Pescetelli and N. Yeung The role of decision confidence in advice-taking and trust formation.. Journal of Experimental Psychology: General 150 (3), p. 507. Cited by: §2.2. Qian and Wexler (2024) C. Qian and J. Wexler Take it, leave it, or fix it: measuring productivity and trust in human-ai collaboration. In Proceedings of the 29th International Conference on Intelligent User Interfaces, Iui ’24, New York, NY, USA, p. 370–384. External Links: ISBN 9798400705083, Link, Document Cited by: §2.2. Rahnev (2025) D. Rahnev A comprehensive assessment of current methods for measuring metacognition. Nature Communications 16 (1), p. 701. External Links: Document Cited by: §2.2. Ramakrishnan et al. (2024) S. K. Ramakrishnan, E. Wijmans, P. Kraehenbuehl, and V. Koltun Does Spatial Cognition Emerge in Frontier Models?. arXiv. Note: Version Number: 2Other Published in ICLR 2025 External Links: Link, Document Cited by: §3.2.1. Riedl and Weidmann (2026) C. Riedl and B. Weidmann Quantifying human–AI synergy. PsyArXiv preprint. External Links: Document, Link Cited by: §1, §2.3, §5.5, §5. Rieger et al. (2026) T. Rieger, H. Schindler, K. Koch, and L. Onnasch AI error difficulty modulates the effectiveness of explainability in decision support systems. ACM Transactions on Computer-Human Interaction 33 (4), p. 1–34. External Links: Document Cited by: §2.2, §5.3. Risko and Gilbert (2016) E. F. Risko and S. J. Gilbert Cognitive offloading. Trends in cognitive sciences 20 (9), p. 676–688. External Links: Document Cited by: §2.2. Schemmer et al. (2023) M. Schemmer, N. Kühl, C. Benz, A. Bartos, and G. Satzger Appropriate reliance on AI advice: conceptualization and the effect of explanations. In Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23, New York, NY, USA. External Links: Document, Link Cited by: §1, §2.2. Shekar et al. (2024) S. Shekar, P. Pataranutaporn, C. Sarabu, G. A. Cecchi, and P. Maes People over trust ai-generated medical responses and view them to be as valid as doctors, despite low accuracy. External Links: Document Cited by: §2.2. Shepard and Metzler (1971) R. N. Shepard and J. Metzler Mental rotation of three-dimensional objects. Science 171 (3972), p. 701–703. External Links: ISSN 1095-9203, Link, Document Cited by: §3.2.1. Simon (1955) H. A. Simon A behavioral model of rational choice. The Quarterly Journal of Economics 69 (1), p. 99–118. External Links: Document Cited by: §2.2, §5.2. Skitka et al. (1999) L. J. Skitka, K. L. Mosier, and M. Burdick Does automation bias decision-making?. International Journal of Human-Computer Studies 51 (5), p. 991–1006. External Links: Document Cited by: §2.2. Srivastava et al. (2023) A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res. 2023. External Links: Link Cited by: §2.1. Steyvers et al. (2022) M. Steyvers, H. Tejeda, G. Kerrigan, and P. Smyth Bayesian modeling of human–AI complementarity. Proceedings of the National Academy of Sciences 119 (11), p. e2111547119. Note: Publisher: Proceedings of the National Academy of Sciences External Links: Link, Document Cited by: §1, §2.1, §5.4. Vaccaro et al. (2024) M. Vaccaro, A. Almaatouq, and T. Malone When combinations of humans and ai are useful: a systematic review and meta-analysis. Nature Human Behaviour, p. 1–11. External Links: Document Cited by: §1, §1, §2.1, §2.3, §5. Wang and Yin (2021) X. Wang and M. Yin Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-assisted Decision-making. In IUI ’21: 26th International Conference on Intelligent User Interfaces, College Station, TX, USA, April 13-17, 2021, T. Hammond, K. Verbert, D. Parra, B. P. Knijnenburg, J. O’Donovan, and P. Teale (Eds.), p. 318–328. External Links: Document, Link Cited by: §2.2. Webb et al. (2022) T. Webb, K. J. Holyoak, and H. Lu Emergent analogical reasoning in large language models. arXiv. External Links: Document, Link Cited by: §3.2.1. Weis and Wiese (2022) P. P. Weis and E. Wiese Know Your Cognitive Environment! Mental Models as Crucial Determinant of Offloading Preferences. Hum. Factors 64 (3), p. 499–513. External Links: Document, Link Cited by: §2.2. Wickens et al. (2015) C. D. Wickens, B. A. Clegg, A. Z. Vieane, and A. L. Sebok Complacency and automation bias in the use of imperfect automation. Human factors 57 (5), p. 728–739. External Links: Document Cited by: §2.2. Wongchamcharoen et al. (2026) P. K. Wongchamcharoen, K. Gulati, M. M. Fong, and A. Nagaraj CentaurBench: benchmarking llm capabilities on augmenting vs. automating real-world work tasks. arXiv preprint arXiv:2608.18554. External Links: Document Cited by: §2.3. Yaniv (2004) I. Yaniv Receiving other people’s advice: influence and benefit. Organizational Behavior and Human Decision Processes 93 (1), p. 1–13. External Links: Document Cited by: §2.2. Yin et al. (2019) M. Yin, J. Wortman Vaughan, and H. Wallach Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 chi conference on human factors in computing systems, p. 1–12. External Links: Document Cited by: §2.3. Yu et al. (2024) F. Yu, A. Moehring, O. Banerjee, T. Salz, N. Agarwal, and P. Rajpurkar Heterogeneity and predictors of the effects of ai assistance on radiologists. Nature Medicine 30 (3), p. 837–849. External Links: Document Cited by: §2.3. Zhang et al. (2019) C. Zhang, F. Gao, B. Jia, Y. Zhu, and S. Zhu RAVEN: a dataset for relational and analogical visual reasoning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 5312–5322. External Links: Document Cited by: §3.2.1. Zhou et al. (2024) K. Zhou, J. D. Hwang, X. Ren, and M. Sap Relying on the unreliable: the impact of language models’ reluctance to express uncertainty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3623–3643. External Links: Document Cited by: §1, §1, §1, §5.2. Zöller et al. (2025) N. Zöller, J. Berger, I. Lin, N. Fu, J. Komarneni, G. Barabucci, K. Laskowski, V. Shia, B. Harack, E. A. Chu, et al. Human–ai collectives most accurately diagnose clinical vignettes. Proceedings of the National Academy of Sciences 122 (24), p. e2426153122. External Links: Document Cited by: §2.1, §2.3. Appendices The appendices document item generation (A), the questionnaire and attention checks (B), the benchmarking harness (C), sample composition (D), and supplementary results (E). Appendix A Item generation A.1. Matrix reasoning Cells vary in shape, size, fill, count and orientation. Each varying attribute is governed by one of four relation types after Carpenter et al. (1990) and Matzen et al. (2010). • Constant. The same value in all three cells of a row, differing between rows • Progressive. The value cycles across the columns, in the same order in every row • Unique. The value follows a diagonal cycle, so that the cell in row r and column c takes the value at index (r+c)mod3(r+c) 3 • Distribute-three. Each of the three values appears exactly once per row, in an order drawn at random for each row Difficulty is set by how many of the attributes vary at once and by how many of them use distribute-three, as shown in Table 8. Table 8. Difficulty levels for matrix items. Level Attributes varying Relations drawn from Additional constraint easy 2 classic only none medium 3 classic only none hard 4 all four at least one distribute-three expert 4a all four at least two distribute-three aAt the expert level all five attributes are sampled, but because orientation is always among them, the shape rule is removed by the orientation–shape exclusion below and no unused attribute remains to replace it, so every expert item varies the four non-shape attributes (count, fill, size, orientation), with shape fixed. Two rendering constraints prevent visually ambiguous options. Orientation rules are restricted to shapes on which orientation is visible (triangles, with the three orientation values separated by at least 25%25\% of the 120∘120 rotational period). A sampled rule set pairing orientation with a shape rule swaps the shape rule onto an unused attribute, or drops it at the expert level (Table 8). Distractors are deduplicated on appearance rather than description. A visual key collapses orientation variants that a shape’s rotational symmetry renders indistinguishable, and any distractor sharing the correct answer’s visual key, or duplicating another distractor’s, is removed. Element size is constant across counts, keeping count and size orthogonal cues, and grid and option images are rendered from shared geometry constants, so a shape is pixel-identical between question and options. A.2. Mental rotation Figure A is always shown at its canonical orientation. How Figure B is produced depends on the condition. In the depth condition the object itself is rotated in three dimensions. In the picture-plane condition the object is not rotated at all and the camera is rolled by the same angular disparity, replicating the picture-plane condition of the original paradigm. Both conditions are crossed with the four difficulty levels in Table 9. Table 9. Difficulty levels for mental rotation. The rotation axis applies to the depth condition only, and it switches with the angle, so no axis effect is separable from an effect of angular disparity. With the camera azimuth fixed at 45∘45 the switch is not itself a difficulty manipulation. Level Angular disparity Rotation axis (depth condition) easy 40∘40 world Y medium 80∘80 world Y hard 120∘120 world X expert 160∘160 world X Every figure is ten cubes in four segments, giving the three right-angled elbows of the original form. Each new cube must be adjacent to exactly one prior cube, which makes closed rings impossible, and base figures are screened for chirality against all 24 proper lattice rotations, so a “different” pair is never an achiral figure some rotation could match. Three implementation choices protect the construct. Rotation is applied to polygon face vertices as exact floats rather than integer cube centers, so rotated figures carry no lattice-rounding artifacts. The camera’s viewing direction never changes (elevation 25∘25 , azimuth 45∘45 ), stationary in the depth condition as in the original method and adding only a roll about the fixed viewing axis in the picture-plane condition. Face shading is reassigned after rotation to the nearest of the six canonical shades by world-space normal, so shading follows orientation rather than identity and cannot signal handedness. A.3. Syllogistic reasoning Each item is a two-premise categorical argument with a stated conclusion, judged as Valid (necessarily true given the premises), Invalid (necessarily false), or Cannot be determined (neither entailed nor refuted, or the premises are inconsistent). The generator dresses each syllogism as a record-keeping narrative in one of six mundane domains (a bakery, a school, a sports club, a library, an animal shelter, a garden center), rewriting the premises so that the canonical quantifier words (all, no, some) are replaced by prose paraphrases from which the quantifier must be recovered, and weaving in neutral distractor sentences. Difficulty is computed rather than hand-estimated. Tiers follow the Johnson–Laird mental-model count (Johnson-Laird and Bara, 1984) — the number of distinct subject–predicate relationships the premises permit — computed by an independent solver, replacing hand-typed per-form accuracy estimates that could not be re-derived. The resulting mapping is given in Table 10. Table 10. Difficulty levels for syllogism items, and the mean of the hand-typed per-form human-accuracy estimates the computed tiers replaced (shown for validation only). Tier Mental models Mean estimated human accuracy easy 1 0.690.69 medium 2 0.510.51 hard 3 0.350.35 Mean estimated unaided participant accuracy falls monotonically across the three counts, validating the tiers against the estimates they replaced. A fourth tier would need a subjective cut-point, reintroducing the free parameter the computed measure removed. Multi-question scenarios with disjoint category terms Reading a scenario carries a fixed comprehension cost paid once per page, but reusing one premise pair with varied conclusions would make the answers mutually derivable (“All A are C” exactly negates “Some A are not C”). A three-question scenario therefore uses nine category phrases in three term-disjoint triples woven into one narrative. No category term is shared between triples, so the premises of one question are logically silent about another’s, and disjointness is asserted at generation time. We accepted the price — a longer page and a “sorting” demand — as a realistic reasoning load. Balancing response categories during item selection All 1818 hard (three-model) single-question items have the answer “Cannot be determined” — a conclusion is valid only if it holds in every model, so the more models a form admits, the less likely any conclusion is entailed — and difficulty-first selection empirically produced a battery that was 9/109/10 “Cannot be determined”, solvable by a constant response. Difficulty is therefore only a tiebreaker among equally label-useful candidates. “Invalid” is scarce because only two of the 4848 forms yield a provably false conclusion, and a hard assertion after selection guarantees at least one valid and one invalid item. A.4. Letter-string analogies Every item presents the alphabet ordering, two worked examples of one transformation, and one target sequence to complete. The two alphabets are a permutation of the 2626-letter Latin alphabet in which 2020 of the 2626 positions are deranged (six letters keep their alphabetic position) and a 1515-symbol non-alphabetic set (> * + < ! @ $ ) & = : - ( % ~). Items are built from four single transformations — add-letter, fix-alphabet, sort, and a simultaneous successor-and-predecessor operation on opposite ends — and three ordered pairs of transformations, namely remove-redundant + add-letter, remove-redundant + sort, and sort + add-letter. Difficulty is built from the generalization axes of Lewis and Mitchell (2024), including a step size greater than one, a longer sequence, and grouping, meaning each symbol is displayed twice so the sequence must be parsed before the rule can apply. Each item records how many axes are active at once, counted relative to a baseline of step one, length five and no grouping — the medium level in Table 11, which was not administered. The alphabet is not one of the counted axes. It changes between levels alongside them — hard moves to the symbol set as the step rises to two, and expert returns to the 2626-letter alphabet as length and grouping are added — so no effect of the alphabet can be separated from the axes it moves with. Table 11 gives the parameters of each level. Responses are typed as free text and scored server-side by whitespace- and case-normalized exact match against the key. Runs of whitespace collapse to single spaces and case is ignored, so spacing and capitalization cannot make a correct sequence wrong, and the two documented alternate answers (see below) are also accepted. Table 11. Parameters of the letter-string levels. The medium level is shown as the baseline against which generalization axes are counted, but was not administered. Level Alphabet Step Base lengtha Grouping Axes active medium (baseline) 2626-letter 1 5 no 0 hard 1515-symbol 2 5 no 1 expert 2626-letterb 2 9 yes 3 combination mixedc 2 5 yes 3c aAn element is one position in the sequence, that is, one symbol of the alphabet. Base length is the length of the sequence the generator draws. The strings actually shown are derived from it, so how many elements they contain depends on the transformation. An add-letter item shows one element fewer before the arrow than after, a remove-redundant item one more. Shown strings run from 44 to 66 elements at the medium, hard and combination levels and from 77 to 1010 at expert. Where grouping is used every element is printed twice, so [k k r r e e g g] is four elements and eight printed symbols. bA nine-element step-two sequence spans 1717 alphabet positions, so expert items use the 2626-letter alphabet rather than the 1515-symbol set. At the shorter levels either alphabet fits, so the alphabet varies alongside the axes rather than independently of them, and it is not one of the counted axes. cThe alphabet is drawn per item. Two of the three combination items use the 1515-symbol set and one the 2626-letter set. Their third counted generalization axis is the two-rule composition itself rather than one of the axes above. The ambiguity verifier A single worked example under-determines the rule, e.g. when considering a sort item where the example is equally consistent with “sort the whole sequence into alphabet order” and with “swap the elements at positions i and j”. These agree on the example and disagree on the target. Showing a second example only helps if it differs on the disambiguating feature, and two randomly drawn examples can share the same accident and leave the ambiguity intact. The generator therefore constructs a hypothesis set ℋH of rules a solver might plausibly induce, including the intended structural families parameterised over step sizes 11–44, plus enumerated positional swaps, positional removals, per-position “fix the odd one out” readings, alphabet-order sorting, duplicate removal, arithmetic-progression repair, and a literal per-position index-shift reading of each shown example. An item is accepted only if every rule in ℋH that is consistent with all shown examples maps the target to the same intended answer, and is resampled otherwise. After generation, a self-check re-parses every emitted sequence-based item from its stored display strings and re-runs the verification independently of the in-memory state used during generation, asserting that the guarantee holds. Verification is relative to the enumerated hypothesis families in ℋH. Every positional rule in ℋH is indexed by position, leaving content-addressed rules such as “swap whichever adjacent pair is out of order” outside the hypothesis set. The two exceptions below illustrate this limit. The combination level Three items apply two transformations at once (remove-redundant ++ sort, remove-redundant ++ add-letter, sort ++ add-letter). For these the hypothesis set extends to all ordered two-operation compositions of a curated primitive set, so the verifier can represent the intended pair and rule out confusable ones. The extension is gated behind a flag so single-operation levels keep their original hypothesis set and their generated output byte-identical. Accepted alternate answers Two combination-level items were genuinely under-determined in a way the verifier did not catch. Each worked example needed only one adjacent-pair swap to reach the intended order, so “sort into alphabet order” and “swap the single out-of-order pair” fit both examples equally well, and only the target distinguishes them. For these two items the answer implied by the narrower rule is also accepted as correct. Appendix B Questionnaire sections and attention checks Four post-task questionnaire sections were administered. About you (all participants) covered age, gender, education, profession, frequency of AI-tool use (55-point, Never–Every day), and preferred AI provider and model. AI consultation strategy (AI only) asked for the percentage of items with interaction beyond the mandated minimum, a four-option description of the typical approach, and behavior when the assistant disagreed with the participant’s initial answer — a self-report companion to the behavioral reliance measure. AI interaction (AI only) rated perceived helpfulness, trust, and frustration on fully labeled 55-point scales. Task feedback (all) rated the task set’s suitability and collected optional strategy descriptions and free-text comments. Full item wording is included with the study materials (see the availability note opening Section 4). Three attention checks required the instructions to have been read rather than skimmed. The first asked how the £100 100 bonus is earned, with the strongest distractor being the other bonus in the same incentive card. A second condition-specific check probed the engagement disclaimer (AI) or the unguessable size of the top-10 bonus (no-AI). The third was a standard instructed-response item after the battery (“select the leftmost option”), carried by a condition-appropriate scale. Exclusions applied to participants failing two or more (Section 3.1). In the analyzed sample no one failed more than one, by construction. Of the 535 participants, 459 passed all three, and the 76 single failures fell mostly on the first check (60), whose strongest distractor is the other bonus in the same incentive card. Appendix C Model benchmarking harness Section 3.5 gives the scoring and exclusion rules. Eighteen claude-opus-4-8 letter-string replies were stopped by a provider-side safety classifier. 32 replies carried needs-review flags, overlapping on 13 replies, so 37 of 16,000 had at least one flag. Custom-alphabet prompts appeared to trigger a cipher/jailbreak classifier, truncating or suppressing replies. This occurred systematically with claude-opus-5 in early testing but emerged for claude-opus-4-8 only in the 100-run campaign, not its ten-run screening. All 3232 needs-review rows and all 1818 refusals received incorrect scores before exclusions. The needs-review rows were then excluded, including the 1313 that also carried refusal flags. The influence is bounded. The ordering of the four assistants is identical whether the needs-review rows are kept, dropped, or all flipped to correct, and no assistant’s competence moves by more than .004.004 across those three treatments. Excluding the refusals instead of scoring them would move the affected group’s pooled competence by .002.002. The identified flags fall 1515 / 1212 / 55 / 00 across claude-opus-4-8, gpt-5.6-luna, kimi-k3 and gemini-3.6-flash. The sensitivity checks above assess these flagged replies. Message parity. A model that is briefed better than a human is not a fair reference. The chat-parity harness that produced the reported numbers therefore sends the assistant exactly the messages a participant’s browser sent and no system prompt, no output-format constraint and no prefill. Benchmark history accumulates within a task and resets between tasks. The runs replay each item’s stimulus message rather than participants’ follow-up turns, whose effect Section 4.2 bounds (advice accuracy .713 to .751 after revision). The benchmark campaign ran 13–19 August 2026, after all participant sessions (22 July–12 August). Two residual asymmetries favor the humans. Participants receive an instruction page and a practice trial per task that the assistant does not, and the assistant’s conversation history is reset at each task, whereas participants’ chat history accumulated across the whole session. We used gemini-3.5-flash-lite with thinking disabled for both the benchmark answer-mapping described above and the transcript extraction in Section 3.6. Note that this makes the two pipelines share a failure mode. Run bookkeeping and robustness. Each run is written to its own timestamped directory with an explicit run identifier. Error-flagged runs are dropped from the average. In the chat-parity campaign two runs aborted mid-way (one from a provider spending cap and one from an empty reply that poisoned the conversation history) and were re-run cleanly, with only the discarded partial attempts quarantined. Appendix D Sample composition Table 12reports the analyzed sample’s composition per group. Table 12. Composition of the analyzed sample by group. Age is M (SDSD). Remaining columns are percentages. Other combines non-binary, multiple categories, and undisclosed gender. AI use is the share reporting at least weekly, and daily, use. Provider is the share naming OpenAI as preferred provider. Group n Age Men Women Other Bachelor+ Weekly+ Daily OpenAI pref. unaided 187 36.8 (10.9) 53 47 1 73 77 37 55 gpt-5.6-luna 179 34.3 (10.8) 54 45 1 68 84 47 56 claude-opus-4-8 61 37.7 (10.8) 52 48 0 77 85 54 56 gemini-3.6-flash 52 33.0 (10.0) 33 67 0 71 79 42 67 kimi-k3 56 32.3 (10.4) 45 52 4 71 79 43 50 Appendix E Supplementary results E.1. Robustness checks Input and sampling uncertainty. Resampling participants within groups and paired benchmark runs 4,000 times gave fixed-battery S=−.062S=-.062 [−.080-.080, −.046-.046] and pooled capture .587.587 [.527.527, .645.645]. These percentile intervals propagate component uncertainty on the same 40 items; they differ from the hierarchical-model intervals. Stricter-sampler refits of S, capture, the item-level competence gradient, and the ladder retained data, formulas, and priors. Each had 1–5 divergences in 40,000 draws. Point estimates were stable, but capture’s 95% HDI widened from [.356,.809][.356,.809] to [.317,.839][.317,.839]. Full comparisons accompany the supplement. Anisotropy. The two sides of that boundary are not mirror images. Each unit by which the assistant leads the unaided participant is worth .578.578 [.473.473, .686.686] of team accuracy, while each unit by which it trails costs −.293-.293 [−.445-.445, −.151-.151], with a benefit-minus-harm magnitude contrast of .285.285 [.069.069, .501.501]. Harm is real and resolves, but accrues at about half the rate the benefit does. The team is more responsive to what the assistant knows than to what it does not. Pass-through on the realized-advice axis. Benchmark competence is a noisy proxy for delivered advice, which can also change through participant follow-ups. Refitting the per-assistant slopes on the realized-advice axis — the share of that group’s trials with correct advice on the item, same trial-level specification on both axes — gives 4.3084.308 [3.8703.870, 4.7654.765] for gpt-5.6-luna, 3.8293.829 [3.3253.325, 4.3244.324] for claude-opus-4-8, 3.7473.747 [3.2163.216, 4.3004.300] for gemini-3.6-flash and 3.1563.156 [2.4472.447, 3.8573.857] for kimi-k3. The spread narrows to roughly 1.4×1.4×, and only the gpt-5.6-luna–kimi-k3 contrast resolves (1.1501.150 [0.4710.471, 1.8351.835]), where on benchmark competence every kimi-k3 contrast does. Changing axes changes both the predictor’s operationalization and its measurement error. Benchmark competence is a noisy stand-in for the advice participants actually met. Fitting the same trial-level model on each of two competence axes in turn — the benchmark, then what the assistant actually said in this study’s own transcripts, which correlate at .898.898 — steepens the slope from 2.5342.534 [2.2452.245, 2.8292.829] to 4.2084.208 [3.8483.848, 4.5844.584]. Both are separate fits from the pooled pass-through model above, so the three slopes are not competing estimates of one quantity. Task-order adjustment. Adjusting for task order moves the trial-level competence slope from 2.1332.133 [1.0461.046, 3.0423.042] to 2.1242.124 [1.0101.010, 3.0043.004] log-odds per unit. This adjusts for position in the session, not task identity. Separately, competence slopes were positive within all four tasks (Section 4.1). Censoring robustness. Competence is right-censored, with 38 of the 160 cells at exactly 1.00, but the outcome ceiling does not bind. No item leaves the unaided group above .90, and the room to improve is flat across competence, so the rising effect is not an artefact of shrinking headroom. E.2. The full policy ladder This section reports the levels behind the contrasts in Figure 10B. The model has 1,120 rows, comprising 160 item–assistant cells crossed with seven reference levels (unaided, observed assisted, first answer, benchmark, task-aware, item-aware, and independent-error oracle). A Gaussian model estimates level-specific means with random intercepts for item and assistant. The task/item references choose between the separate unaided group’s accuracy and benchmark competence. First-answer accuracy uses the earliest extracted assistant answer and covers about 97% of assisted trials, whereas observed performance includes all assisted trials. These levels have different eligibility and are reported descriptively. The fitted unaided level is .525 [.432, .613], observed assisted accuracy .726 [.634, .814], and the conditional first-answer reference .727 [.634, .814]. The task-aware reference is .761 [.670, .851], the item-aware reference .787 [.696, .876], and the independent-error oracle .867 [.776, .956]. The item-aware and oracle gains over observed performance are .061.061 [.030.030, .092.092] and .141.141 [.110.110, .172.172]. These model-based reference contrasts use the component-accuracy estimates and assumptions specified above. Of the 348 assisted participants, the 88 who estimated that the assistant was no better than themselves reached .745 [.733, .758], versus .713 [.705, .721] among the other 260, a difference of .032 [.017, .046]. These post-task beliefs may reflect experienced performance. Mean estimated assistant performance was 29.7 against 28.8 correct answers out of 40. This comparison concerns mean beliefs and mean performance. E.3. Metacognition across the four assistants This section reports per-assistant item-level detail for Section 4.3. The per-group confidence gaps and their drops against unaided are in Table 6. Across the 40 items, correlations between mean confidence and LLM competence were r=−.363r=-.363 for gpt-5.6-luna, −.383-.383 for claude-opus-4-8, −.502-.502 for gemini-3.6-flash, and −.613-.613 for kimi-k3. Competence correlated .105.105 with unaided accuracy, and adjusting for that measure strengthened the negative associations. The pattern arose between tasks and was inconsistent within them, where two tasks offered little competence variation. Mean post-advice confidence therefore did not consistently track item-level LLM competence. This differs from discriminating the correctness of one’s submitted answer and leaves pre-advice confidence untested.

Instant research discovery