Paper deep dive

From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Cost

Saki Imai, Mert İnan, Malihe Alikhani

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 106

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 9/21/2026, 3:57:23 AM

Summary

This paper introduces a productivity-oriented framework for evaluating human-AI collaboration that measures outcome quality relative to interaction cost, rather than treating collaboration as a black box. Using two datasets (CoGym and a new visualization dataset) across four tasks, the authors demonstrate that sessions with identical quality ratings can vary by up to 70 times in interaction cost. The study finds that quality-cost relationships are task-dependent, subjective user ratings are unreliable proxies for productivity, and productive sessions are characterized by agents probing earlier and users spending less effort on repair.

Entities (15)

Interaction Cost · metric · 95%Outcome Quality · metric · 95%CoGym · dataset · 90%Costly Success · concept · 90%LLM-as-Judge · method · 90%Productive Success · concept · 90%Related Work Synthesis · task · 90%Tabular Analysis · task · 90%Travel Planning · task · 90%Visualization · task · 90%Grounding Acts · concept · 85%Positive Friction · concept · 85%Gemini 2.0 Flash · model · 80%GPT-4o · model · 80%GPT-5.1 · model · 80%

Relation Signals (11)

Productive Success → definedby → High Quality and Low Interaction Cost

confidence 95% · We define productive collaboration as high quality output achieved with relatively low interaction cost.

Costly Success → definedby → High Quality and High Interaction Cost

confidence 95% · Costly SuccessHigh quality, high cost

CoGym → includestasks → Related Work Synthesis

confidence 95% · CoGym... spanning three diverse tasks: ... and related work synthesis

CoGym → includestasks → Travel Planning

confidence 95% · CoGym... spanning three diverse tasks: travel planning... tabular analysis... and related work synthesis

CoGym → includestasks → Tabular Analysis

confidence 95% · CoGym... spanning three diverse tasks: ... tabular analysis...

Productive Sessions → characterizedby → Agent Probing Earlier

confidence 90% · productive sessions are characterized by agents probing earlier

Productive Sessions → characterizedby → Less User Repair Effort

confidence 90% · users spending less effort repairing the interaction

Related Work Synthesis → correlateswith → Positive Quality-Cost Relationship

confidence 90% · Related work shows the strongest positive association... suggesting that additional interaction often supports more complete synthesis.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:AI productivity is often measured by task completion time, economic value, or improvements in outcome quality. However, these measures usually treat collaboration as a black box where they capture what output was produced, but not the interaction cost required to produce it. Motivated by economics literature, we introduce a productivity-oriented framework for evaluating human-AI collaboration as outcome quality relative to interaction cost. Across two datasets spanning four tasks, we show that: (1) sessions with identical quality ratings can differ by up to 70 times in interaction cost; (2) quality-cost relationships vary by task, with some tasks rewarding extended interaction and others favoring fast convergence; (3) subjective user ratings are not reliable substitutes for productivity; and (4) productive sessions are characterized by agents probing earlier and users spending less effort repairing the interaction. By distinguishing productive success from costly success, our framework makes interactional cost visible and shows how dialogue analysis can inform the evaluation and design of AI systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

103,478 characters extracted from source content.

Expand or collapse full text

From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Cost Saki Imai Mert İnan Malihe Alikhani Affiliation: Northeastern University, Boston MA Affiliation: imai.s, inan.m, m.alikhani@northeastern.edu Abstract AI productivity is often measured by task completion time, economic value, or improvements in outcome quality. However, these measures usually treat collaboration as a black box where they capture what output was produced, but not the interaction cost required to produce it. Motivated by economics literature, we introduce a productivity-oriented framework for evaluating human-AI collaboration as outcome quality relative to interaction cost. Across two datasets spanning four tasks, we show that: (1) sessions with identical quality ratings can differ by up to 70× in interaction cost; (2) quality-cost relationships vary by task, with some tasks rewarding extended interaction and others favoring fast convergence; (3) subjective user ratings are not reliable substitutes for productivity; and (4) productive sessions are characterized by agents probing earlier and users spending less effort repairing the interaction. By distinguishing productive success from costly success, our framework makes interactional cost visible and shows how dialogue analysis can inform the evaluation and design of AI systems. 1 Introduction Two human-AI collaborations can achieve equally successful outcomes while requiring vastly different amounts of interaction from the user. In our data, for example, two travel planning sessions with the same quality rating differed by approximately 70 times in interaction cost. Task success treats these sessions as equivalent, but from the user’s perspective they are not. Figure 1 illustrates this distinction: one is a productive success, while the other is a costly success. As large language models become embedded in work practices, researchers have renewed efforts to measure AI productivity: what work AI systems can perform, how much economic value they may create, and whether they allow people to complete tasks faster or at higher quality. Recent work estimates the economic exposure of occupations to LLMs Handa et al. (2025); Tamkin and McCrory (2025), measures the time horizon over which agents can complete tasks autonomously Kwa et al. (2026); METR (2026); Patwardhan et al. (2025), and evaluates productivity gains in domains such as writing, coding, consulting, and customer support. These studies show that AI systems can affect productivity, but they often treat the interaction itself as a black box. They measure the final output, elapsed time, or economic value, without explaining how the human and AI system reached that outcome. Interaction CostOutput QualityLowHighLowHighProductive SuccessHigh quality, low cost Costly SuccessHigh quality, high cost Cheap FailureLow quality, low cost Unproductive FailureLow quality, high cost Figure 1: Productivity distinguishes productive success from costly success. Task success captures output quality, but not the interaction cost required to achieve it. We define productive collaboration as high quality output achieved with relatively low interaction cost. We argue that productive human-AI collaboration should be evaluated as a relationship between outcome quality and interaction cost. This view builds on the principle of least collaborative effort Clark and Wilkes-Gibbs (1986), which holds that collaborators coordinate not to minimize effort for one party alone, but to minimize the joint effort required to establish sufficient common ground Clark and Brennan (1991). In human-AI settings, this distinction is especially important, where an agent can appear helpful while shifting substantial grounding work onto the user. To make this distinction measurable, we introduce a productivity oriented evaluation framework for human-AI collaboration that defines collaborative productivity as outcome quality relative to interaction cost. We validate the framework across two datasets, including a newly collected human-AI collaboration dataset with full interaction trajectories, submitted artifacts, and post task perception measures, for a visualization task.11 1 Code and data to reproduce our work can be found at https://github.com/sakimai/human-AI-productivity Our analysis yields three main findings. First, sessions with identical quality ratings can differ by one to two orders of magnitude in interaction cost, variation that subjective user ratings do not consistently capture. Second, the relationship between cost and quality is task dependent: in related work writing, additional interaction tends to accompany better outcomes, while in visualization, higher quality sessions tend to converge with less interaction. These patterns suggest that productivity must be interpreted within task rather than treated as a universal preference for shorter conversations. Third, productive collaboration is not simply shorter interaction. Productive sessions are characterized by agents probing and clarifying earlier, while less productive sessions require users to spend more effort clarifying, redirecting, and repairing. This suggests that interaction can be productive when agent initiated grounding reduces later clarification, and correction, but costly when the burden of resolving ambiguity falls primarily to the user. Our contributions are as follows: 1. We introduce a productivity oriented evaluation framework for human-AI collaboration as outcome quality relative to interaction cost (§ 3.1), and show that it captures variation not reflected in user subjective ratings (§ 4.3). 2. We show that collaborations with similar outcome can require dramatically different amounts of user effort (§ 4.1) and that tasks differ in whether additional interaction helps or hurts quality (§ 4.2). 3. We identify dialogue patterns associated with productive collaboration, showing that productive agents reduce user side grounding burden by front loading clarification (§ 4.4). 2 Related Work Evaluating productivity of human-AI collaboration. A growing literature measures whether LLMs make people more productive, by measuring outcomes rather than interactions. Economic analyses estimate aggregate value and labor market effects (Handa et al., 2025; Tamkin and McCrory, 2025; Eloundou et al., 2024; Webb, 2020; Gans and Goldfarb, 2026), and benchmarks measure autonomous task completion (Kwa et al., 2026; METR, 2026; Patwardhan et al., 2025; Vidgen et al., 2025). Empirical experiments with users report substantial speedups for professional writing (Noy and Zhang, 2023), customer support (Brynjolfsson et al., 2025), code generation (Peng et al., 2023; Imai, 2022; Qian and Wexler, 2024), ad creation (Ju and Aral, 2026), and consulting Dell’Acqua et al. (2026) with the largest gains among less experienced workers. Other studies report null or negative effects, where experienced developers were 19% slower with AI while believing they were 20% faster (Becker et al., 2025), and consultants who used AI on tasks outside its capabilities performed worse than those without it (Dell’Acqua et al., 2026). These studies measure whether AI helps on average, not how language use during the interaction shapes the outcome. We close this gap by measuring collaborative productivity at the session level and analyzing the dialogue that produced each outcome. Earlier work combined task success and dialogue cost into a single score for spoken dialogue systems (Walker et al., 1997). We adapt this framing to human-AI collaboration, but shift the question from comparing systems to explaining variation between sessions on the same task. Evaluating human-AI collaboration. Recent work has argued that evaluation should move beyond static, AI alone benchmarks Lee et al. (2023); Fragiadakis et al. (2025); Xuan et al. (2026). Interaction centered datasets Lee et al. (2022); Chiang et al. (2024); Zheng et al. (2024); Zhao et al. (2024) and benchmarks Lin et al. (2025); Chang et al. (2025); Li et al. (2025); Zheng et al. (2023) are aimed to evaluate model behavior in use. A related line of work evaluates human-agent collaboration, where the AI system can communicate with the user while also taking actions in a shared task environment (Yao et al., 2025; Barres et al., 2025; Shao et al., 2026; Shen et al., 2025). Recent work on LLM training further incorporates efficiency into multiturn objectives, penalizing excessive user reading and writing costs alongside task success and interaction quality (Wu et al., 2025). Most existing evaluations still center on end-to-end task success rates Shridhar et al. (2021); Zhou et al. (2024); Xie et al. (2024); Jimenez et al. (2024), and the cost of reaching that outcome is often ignored. By introducing a framework that jointly measures outcome quality and interaction cost, it allows us to distinguish productive and unproductive collaboration. Grounding acts and positive friction. Collaboration requires participants to build a shared conceptual workspace Roschelle and Teasley (1995), and both speakers share the effort of building it. The principle of least collaborative effort (Clark and Wilkes-Gibbs, 1986) holds that participants minimize joint rather than individual effort, and Clark and Brennan (1991) decompose this into formulation, reception, and repair costs distributed across both speakers. This framing motivates our framework: a productivity measure for human-AI collaboration must account for the burden agent output places on users. Recent work shows that LLMs do not naturally reproduce human grounding behavior. Shaikh et al. (2024) find that LLM generations contain fewer grounding acts than human responses, often presuming common ground rather than actively constructing it. Shaikh et al. (2025) further show that grounding failures in human-LLM interaction can produce downstream rifts. In contrast, positive friction argues that some slowdowns improve reliability İnan et al. (2025). Our framework reconciles these views, that positive friction is productive when it prevents larger downstream repair costs, and unproductive when it shifts repair burden to the user. 3 Methods We evaluate human-AI collaboration at the session level. Each session produces a final output and an interaction trajectory. Our framework asks whether the session achieved high outcome quality relative to the interaction cost required to produce it. We first define the productivity framework, then describe how we instantiate quality, cost, and dialogue features in our two datasets. 3.1 Collaborative Productivity Framework We define collaborative productivity as outcome quality relative to interaction cost. This follows the productivity intuition of output relative to input (Solow, 1957; Schreyer and Pilat, 2001) in economics, but adapts it to human-AI collaboration,22 2 We use human-AI collaboration as an umbrella term covering both chat-only and tool-using LLM agents. where the relevant input is the interactional effort required to reach an outcome. This formulation is motivated by theories of grounding in communication. Collaborative work requires participants to establish sufficient common ground for the task at hand. Clark and Brennan (1991) distinguish several costs involved in this process, including formulation costs for producing utterances, reception costs for interpreting them, and repair costs for resolving misunderstandings. In human-AI collaboration, these costs are distributed across the user and the agent, where users write prompts, read and interpret model outputs, and repair when the interaction goes off track. We therefore operationalize interaction cost as a property of the full trajectory rather than only the user’s messages. Our framework also builds on PARADISE, which evaluates spoken dialogue agents by combining task success with dialogue costs (Walker et al., 1997). PARADISE showed that dialogue evaluation should separate what was accomplished from the costs incurred in accomplishing it, and that these quantities can be combined into a performance function. We adapt this idea from spoken dialogue systems to human-AI collaboration. Let Y denote the space of artifacts a collaborative session can produce and T the space of interaction trajectories. We define a quality function Q:→ℝQ:Y scoring the artifact and a cost function C:→ℝ≥0C:T _≥ 0 scoring the interaction. The collaborative productivity of session i on a task is Pz(i)=z⁡(Qi)−z⁡(Ci)P_z(i)=z(Q_i)-z(C_i) where z⁡(⋅)z(·) denotes within task z-score normalization and Qi,CiQ_i,C_i are the quality and cost of session i. We standardize Q and C because tasks differ in both quality scales and expected interaction length, so raw scores are not directly comparable across tasks. Following multi-attribute utility approaches (Keeney and Raiffa, 1993), we combine the standardized attributes additively, treating quality as a benefit and interaction cost as a penalty. Thus, PzP_z can be interpreted as an index that shows whether a session achieved higher quality for that task while requiring lower interaction cost. In Appendix F, we report sensitivity analyses varying the relative weight on quality and cost and show that the main qualitative findings are stable across weights. Quality as a task dependent design choice. The appropriate quality function (Q) depends on the task and evaluation goal. In code generation, the standard target is functional correctness, usually reported as pass@k Chen et al. (2021) or test case pass rate Hendrycks et al. (2021). For high stakes domains, such as medical Ayers et al. (2023); Tu et al. (2025) or mental health Scholich et al. (2025); Franke Föyen et al. (2025) support, expert ratings may be appropriate because correctness and harm require domain judgment. For service oriented tasks, user ratings may be appropriate when the target outcome is whether the user’s goal was satisfied Budzianowski et al. (2018). For open-ended but rubric evaluable tasks, LLM-as-a-judge rubric based scoring provides a consistent way to assess artifact quality across sessions. Interaction cost as grounding burden. Cost measures should reflect the interactional burden imposed by the collaboration. User tokens approximate formulation effort, agent tokens approximate reception burden, and turns or elapsed time capture coarser interaction overhead. Repair costs are not added as a separate term in our primary measure because they accumulate through the trajectory itself where repeated clarification and reformulation increase the number of tokens. In this sense, token and turn based costs operationalize the idea from grounding theory that formulation, reception, and repair effort accumulate over the interaction. 3.2 Data We instantiate the framework on two datasets spanning four tasks: three structured human-agent tasks from Collaborative Gym and one newly collected human-LLM visualization task. CoGym We use human-agent collaboration sessions from the CoGym framework Shao et al. (2026) spanning three diverse tasks: travel planning (n = 112), tabular analysis (n = 69), and related work synthesis (n = 47). In these tasks, users collaborate with an LLM agent that can act in a shared environment, such as searching databases, editing a shared document, or executing code. Sessions were collected from real users interacting with LLM agents (GPT-4o, Gemini 2.0 Flash) through a web interface Hurst et al. (2024); Team et al. (2025). We restrict primary analyses to sessions producing a final artifact (n=184), and provide additional details of each task in Appendix A. Visualization We additionally collect a new human-LLM collaboration dataset of 42 sessions in which participants used GPT-5.1 Singh et al. (2026) to analyze the WildChat-1M dataset Zhao et al. (2024). Participants were asked to formulate a hypothesis about the dataset, test it using Python, and submit a visualization with supporting code and a short interpretation. The task is open-ended but the final output can still be evaluated with a rubric for relevance, correctness, and evidential support. We log the full chat trajectory, submitted code, final visualization, and post task survey responses. After submission, participants rated their confidence in the output, the usefulness of the AI, and perceived interaction productivity. This dataset complements CoGym by covering a human-AI collaboration task without environment actions. We provide additional details on the study setting, custom platform, and procedure in Appendix B. Defining Q and C. Across both datasets, we use LLM-as-judge rubric scored task performance as the primary quality measure Q (Chiang et al., 2024), using the task specific rubrics reported in Appendix C. This choice reflects our goal of measuring productivity as the quality of the final artifact relative to the interaction cost required to produce it. On the other hand, user Likert ratings capture how participants perceived the outcome or interaction. We therefore reserve them for comparison analyses rather than using them to define Q. This separation allows us to ask whether perceived productivity aligns with an independently scored quality-cost tradeoff, and avoids circularity in later analyses. Moreover, this avoids using the same user survey both to define Q and to test whether users perceived the interaction as productive (Podsakoff et al., 2003). In our data, subjective Likert ratings are strongly correlated within each dataset, so we keep them as comparison measures rather than the primary quality measure (Appendix D). Our primary cost measure is weighted token cost: C=user tokens+0.5⋅agent tokensC=user tokens+0.5·agent tokens User tokens approximate formulation effort, while agent tokens approximate reception burden (Clark and Brennan, 1991). We assign agent tokens a lower weight because reading is typically faster than composing text Brysbaert (2019); Karat et al. (1999), while still treating long model outputs as a cost imposed on the user. Because no token based measure can fully capture effort such as cognitive load and attention, we test whether our conclusions depend on this particular operationalization. Appendix F repeats the analyses with user tokens only, total tokens, turn count, and alternative agent token weights; the main conclusions remain qualitatively stable. Human validation. To validate the LLM-as-judge quality scores, two authors independently scored a subset of sessions using the same task specific rubrics. For CoGym, we sample 20 sessions from each task and across the range of LLM-rubric scores; for the visualization dataset, we score all 42 sessions. Human scores correlate strongly with LLM-as-judge scores (CoGym ρ=.68ρ=.68; visualization ρ=.74ρ=.74), with high human–human agreement (ρ=.81ρ=.81). This suggests that the primary quality signal used in PzP_z tracks human rubric judgments. 3.3 Grounding and Friction Features To characterize productive collaboration, we annotate each utterance using two taxonomies: grounding acts (Shaikh et al., 2025) and positive friction movements (İnan et al., 2025). Grounding acts characterize whether and how participants establish mutual understanding; friction movements characterize deliberate slowdowns that surface assumptions or invite reflection. The two taxonomies capture different aspects of collaborative discourse. Grounding acts. We adapt the taxonomy of Shaikh et al. (2025), which distinguishes acts that advance grounding (next turn, acknowledge, follow-up), acts that signal ambiguity (clarification, overresponse), and acts that address grounding failures (repair, reformulation, restart). User and agent acts are labeled separately, since the same surface act carries different functional weight depending on the speaker (e.g., an agent acknowledgment vs. a user acknowledgment). An utterance may receive multiple labels. Positive friction movements. We adapt the five category positive friction taxonomy from İnan et al. (2025): assumption reveal (surfacing beliefs about the environment, the interlocutor, or the task), reflective pause (verbal or behavioral signals of internal deliberation), reinforcement (restating a prior utterance for emphasis), overspecification (providing more information than requested), and probing (questioning to redirect or clarify). Each utterance receives a single label from these five categories or Not Friction if none applies. We retain the single label scheme of the original taxonomy. Annotation. We use GPT-5.1 at temperature 0 to label each utterance with grounding acts and positive friction movements, using the annotation guidelines from Shaikh et al. (2025) and İnan et al. (2025). We validate these labels on a subset of 20 sessions, balanced across dataset and productivity quartile. Two human annotators independently label the same utterances. Agreement was substantial for positive friction labels (human–human κ=.72κ=.72; model–human κ=.64κ=.64) and acceptable for grounding acts (human–human micro-F1 =.76=.76; model–human micro-F1 =.68=.68). 4 Findings Figure 2: Identical quality ratings can hide large differences in interaction cost. Each panel shows the distribution of interaction cost within quality buckets for one task. Cost is shown on a log scale. Wide horizontal spread within a quality bucket indicates that task success alone cannot distinguish productive success from costly success. Figure 3: Task specific relationships between outcome quality and interaction cost. Each point is a session, with quality and cost standardized within task; dashed lines indicate task means. The pattern shows that interaction cost is not uniformly beneficial or harmful, where additional interaction accompanies higher quality in some tasks. We organize the results around four questions. First, does task success hide interaction costs? (§ 4.1) Second, does the quality-cost relationship vary by task? (§ 4.2) Third, do subjective ratings capture the same quality-cost tradeoff? (§ 4.3) Finally, what dialogue patterns characterize productive collaboration? (§ 4.4) These analyses show that productivity is not simply shorter interaction. It depends on whether interaction cost helps produce quality, and on who bears the grounding work required to reach that quality. 4.1 Cost Variation Within Quality Levels Identical quality ratings can require very different amounts of interaction. Figure 2 plots interaction cost within each quality bucket separately for all three CoGym tasks and the visualization task. Across tasks, sessions with the same rating often differ by one to two orders of magnitude in cost. The clearest example appears in travel planning. Among completed sessions in the top quality bucket, interaction costs range from 854 to 60,324 tokens, a 70.6 times difference. Similar within quality variation appears in tabular analysis, related work, and visualization. This motivates measuring productivity as quality relative to cost since task success alone cannot distinguish productive success, where high quality is reached with low interaction cost, from costly success, where similar quality requires substantially more interaction. The same pattern holds when quality is measured with human Likert outcome rating rather than LLM rubric scores (Appendix E). 4.2 Task Structures Shape Quality & Cost Interaction cost is not uniformly helpful or harmful; its relationship to quality depends on the task. We compute the within task Spearman correlation between standardized quality z⁡(Q)z(Q) and standardized cost z⁡(C)z(C), and Figure 3 visualizes the relationship at the session level. Related work shows the strongest positive association (ρQC=+0.55 _QC=+0.55), suggesting that additional interaction often supports more complete synthesis. Travel planning is also positive but weaker (ρQC=+0.27 _QC=+0.27), consistent with a task where iteration helps but gains diminish as plans converge. In contrast, tabular analysis shows little evidence of a positive quality-cost relationship (ρQC=−0.13 _QC=-0.13) where additional exchanges do not reliably improve quality. This interpretation is consistent with works showing that data analysis is iterative and often repetitive Wongsuphasawat et al. (2019); Kandel et al. (2012). Visualization shows the most negative association (ρQC=−0.48 _QC=-0.48), suggesting that high quality sessions depend less on iteration and tend to converge quickly on a hypothesis and visualization. These differences matter for evaluation. A universal penalty on interaction length would mischaracterize tasks like related work, where more interaction can reflect useful elaboration. Conversely, rewarding longer interaction would mischaracterize tasks where additional exchanges reflect debugging, uncertainty, or delayed convergence. Productivity therefore needs to be interpreted within a task rather than a universal preference for brevity. 4.3 Subjective Ratings Do Not Reliably Capture Productivity If subjective ratings captured productivity, then users should penalize costly sessions when quality is held fixed. We test this by computing, for each subjective measure S, the partial Spearman correlation ρ⁡(C,S∣Q)ρ(C,S Q) between interaction cost C and the subjective rating, controlling for quality Q. A productivity sensitive subjective measure should show a negative partial correlation, where among sessions of similar quality, higher cost should correspond to lower ratings. Across six subjective measures in two datasets, this pattern appears reliably in only one case (Table 1). CoGym satisfaction shows the predicted cost penalty, both unconditionally (ρ=−0.22ρ=-0.22, p<0.01p<0.01) and after controlling for quality (ρ=−0.24ρ=-0.24, p<0.01p<0.01). In contrast, CoGym outcome quality and communication rating are cost blind: holding quality fixed, longer or more costly interactions are not rated lower. But, the visualization study shows Speed is weakly cost-blind, effectiveness is cost-rewarding, and confidence is positively associated with cost even after controlling for quality (ρ=+0.41ρ=+0.41, p<0.01p<0.01). These divergences suggest that subjective prompts elicit different constructs. Quality focused prompts (outcome quality, communication rating) anchor users to the final artifact, where interaction cost may be less salient. Experience focused prompts can split into two interpretations, where satisfaction appears to treat cost as negative experience, while confidence may treat cost as investment or commitment. Subjective ratings are therefore useful, but they are not interchangeable proxies for productivity. Dataset Measure ρ⁡(Q,S)ρ(Q,S) ρ⁡(C,S∣Q)ρ(C,S Q) n CoGym outcome quality +0.14+0.14 −0.04-0.04 155 CoGym comm. rating −0.09-0.09 −0.14-0.14 41 CoGym satisfaction +0.03+0.03 −0.24∗⁣∗-0.24^** 184 Visualization speed −0.02-0.02 −0.07-0.07 42 Visualization effectiveness −0.05-0.05 +0.13+0.13 42 Visualization confidence −0.00-0.00 +0.41∗⁣∗+0.41^** 42 Table 1: Partial Spearman correlations of interaction cost C with subjective measures S, controlling for LLM rubric quality Q. CoGym measures are either cost-blind or cost-penalizing; Visualization metrics are either cost-blind or cost-rewarding. *p<0.05p<0.05, **p<0.01p<0.01. 4.4 Productive Collaboration Shifts Interactional Labor to the Agent Figure 4: Agent and user probing in top vs. bottom PzP_z quartile (n=59 each). (a) Rate per turn: productive sessions show a labor crossover, with the agent probing more and the user probing less. (b) Counts per session: agent probe counts are identical, but user probe counts more than double in unproductive sessions. Error bars: ± 1 SEM. We next ask what productive collaboration looks like in the dialogue itself. We compare the top and bottom PzP_z quartiles (n=59 each) on grounding acts and positive friction movements. For each feature, we report Cohen’s d for the difference in means between quartiles, where positive values indicate features that are more frequent in productive sessions. Significance is assessed with Mann Whitney U tests; full feature results are in Appendix G. Friction is not inherently unproductive; its speaker matters. The rate of any friction per turn is nearly identical between productive and unproductive sessions (0.77 vs. 0.79, d=−0.31d=-0.31, n.s.). However, unproductive sessions contain more total friction movements because they are longer (8.0 vs. 14.1, d=−0.65d=-0.65, p<0.001p<0.001). The difference between these sessions is therefore not how often friction occurs, but who produces it. The clearest example is probing (Figure 4). Productive sessions devote 44% of agent turns to probing, compared to 27% in unproductive sessions (d=+0.73d=+0.73, p<0.001p<0.001). Across the full sample, agent probing rate is positively correlated with PzP_z (ρ=+0.19ρ=+0.19, p<0.01p<0.01, n=233). This effect is concentrated in tasks where iterative clarification is useful, such as related work (ρ=+0.35ρ=+0.35) and travel planning (ρ=+0.24ρ=+0.24), consistent with the task level patterns in §4.2 (Table 4). Agent probe counts are identical across quartiles (n¯=1.81 n=1.81 in both; Figure 4b), which shows that productive sessions reach the same amount of clarification in roughly half the number of turns. The pattern is stronger early in the session where 48% of agent turns in the first 30% of productive sessions contain probing, vs. 31% in unproductive ones. User probing shows the opposite pattern (d=−0.52d=-0.52, p<0.01p<0.01), suggesting that when the agent does not probe, users must do the clarification work themselves. Assumption reveal follows the same trend, concentrated on the user side in unproductive sessions (n=0.31n=0.31 vs. 1.371.37, d=−0.57d=-0.57). Grounding acts show the same asymmetry. User repair (0.22 vs. 0.83, d=−0.58d=-0.58), clarification (0.27 vs. 0.63, d=−0.49d=-0.49), follow-up (3.05 vs. 6.02, d=−0.55d=-0.55) are all elevated in unproductive sessions. By contrast, user acknowledgment and agreement rates run roughly 3× higher in productive sessions (d=+0.58d=+0.58 for both, p<0.05p<0.05). Productive collaboration therefore looks like agent probes → user confirms; unproductive collaboration leaves the grounding work to the user to repair, clarify, and probe. This matches the principle of least collaborative effort (Clark and Wilkes-Gibbs, 1986; Clark and Brennan, 1991). Agent side friction can be productive when it surfaces ambiguity early and prevents downstream repair. The same friction becomes costly when the user must perform it later as repair work. 5 Discussion Our findings suggest that productive human-AI collaboration is about how interactional work is distributed across the collaboration. Productive collaboration depends on who bears the grounding work. Our dialogue analysis shows that productive collaboration is not simply less interactive or less frictional. Productive and unproductive sessions contain similar rates of friction; what differs is who performs the grounding work (§4.4). In productive sessions, agents probe earlier and users confirm; in unproductive sessions, users repair, clarify, and probe the agent. Calls for seamless interaction often imply that friction should be minimized, while positive friction argues that strategic slowdowns can improve reliability (İnan et al., 2025). Our results suggest that friction is productive when the agent pays a small clarification cost up front, and costly when unresolved ambiguity later becomes user-side repair. This implies that agents should be designed to front-load clarification, surface assumptions, and ask targeted probes before acting, rather than waiting for users to initiate repair. Productivity is task relative. The quality-cost relationship varies substantially across tasks, from positive in related work synthesis (ρQC=+0.55 _QC=+0.55) to negative in visualization (ρQC=−0.48 _QC=-0.48; §4.2). This means there is no universal effort to outcome curve for human-AI collaboration. In some tasks, additional interaction reflects useful elaboration, exploration, or synthesis; in others, it may reflect delayed convergence, debugging, or confusion. Evaluation protocols that uniformly penalize interaction length may therefore underrate systems on tasks where extended interaction is productive, while protocols that reward engagement may overrate systems on tasks where high quality outcomes should converge quickly. This implies that productivity should be evaluated and interpreted within a task before comparing across tasks or systems. Subjective ratings measure different constructs. Subjective ratings are valuable, but they are not interchangeable measures of productivity. In our data, most subjective measures do not penalize interaction cost once quality is held fixed, and some move in the opposite direction (§4.3). This suggests that quality focused prompts direct users to the artifact, satisfaction may integrate cost as negative experience, and confidence may reflect commitment. As a result, two studies could reach different conclusions about the same system depending on whether they ask users about satisfaction, confidence, usefulness, or speed. This has implications in studies of human-AI collaboration which should specify what each subjective prompt is intended to measure, and should avoid treating perceived quality, usefulness, confidence, and productivity as interchangeable outcomes. Productivity is conditional on the evaluation objective. Our framework does not assume that interaction is inherently costly or that shorter interactions are always preferable. Interaction cost is meaningful only relative to the outcome the collaboration is intended to achieve, and the quality function (Q) must therefore reflect the values relevant to that task. For example, in an educational setting, immediately giving a student the correct answer may minimize interaction while undermining the objective of learning. If learning is the intended outcome, Q should instead capture outcomes such as retention, transfer, or independent problem solving. We therefore view collaborative productivity as efficiency conditional on task relevant goals and not as a prescription to minimize interaction. 6 Conclusion Task success tells us whether a human-AI collaboration reached a desired outcome, but not what it cost the user to get there. We introduced a productivity oriented framework for evaluating collaboration as outcome quality relative to interaction cost, and applied it across four tasks from two datasets. Our results show that identical quality ratings can hide large differences in interaction cost, that the quality-cost relationship varies by task, and that subjective ratings do not consistently capture this tradeoff. Dialogue analysis further shows that productive collaboration shifts grounding work toward the agent. Productive collaboration is not just successful collaboration made shorter. It is successful collaboration in which interactional effort is spent where it contributes meaningfully to the outcome rather than being unnecessarily transferred to the user. These findings motivate the design of agents that surface ambiguity and probe when appropriate to reduce unnecessary user repair. Limitations Interaction cost is an approximation of user effort. We use weighted token cost as our primary measure, treating user tokens as a proxy for formulation effort and agent tokens as a proxy for reception burden. This operationalization captures part of the interactional burden, but it does not directly measure cognitive load, attention, frustration, multitasking, or the effort required to evaluate whether an answer is correct. Dialogue patterns are correlational. Our dialogue analysis identifies features associated with productive sessions, such as agent probing and reduced user repair. However, these results are observational. Future experiments should test whether prompting or training agents to front load clarification improves productivity without introducing unnecessary friction. Generalizability is limited by tasks. We evaluate four tasks across two datasets. These settings provide useful variation, but they do not cover the full range of human-AI collaboration. Productivity may behave differently in long horizon work, creative collaboration, or domains where trust, safety, and accountability matter more than speed. Use of AI Assistants The authors used AI assistants for language polishing after the initial draft. The authors verified and edited all generated text and remain responsible for the content of the paper. Acknowledgments This research was supported in part by a grant from the Institute for Information, the Internet, and Democracy (IIID) at Northeastern University. We thank Asteria Kaeberlein for their helpful feedback. References Ayers et al. (2023) J. W. Ayers, A. Poliak, M. Dredze, E. C. Leas, Z. Zhu, J. B. Kelley, D. J. Faix, A. M. Goodman, C. A. Longhurst, M. Hogarth, and D. M. Smith Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Internal Medicine 183 (6), p. 589–596. External Links: ISSN 2168-6106, Document, Link, https://jamanetwork.com/journals/jamainternalmedicine/articlepdf/2804309/jamainternal_ayers_2023_oi_230030_1685974538.66672.pdf Cited by: §3.1. Barres et al. (2025) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan τ2τ^2-Bench: evaluating conversational agents in a dual-control environment. External Links: 2506.07982, Link Cited by: §2. Becker et al. (2025) J. Becker, N. Rush, E. Barnes, and D. Rein Measuring the impact of early-2025 ai on experienced open-source developer productivity. External Links: 2507.09089, Link Cited by: §2. Brynjolfsson et al. (2025) E. Brynjolfsson, D. Li, and L. Raymond Generative ai at work. The Quarterly Journal of Economics 140 (2), p. 889–942. Cited by: §2. Brysbaert (2019) M. Brysbaert How many words do we read per minute? a review and meta-analysis of reading rate. Journal of Memory and Language 109, p. 104047. External Links: ISSN 0749-596X, Document, Link Cited by: §3.2. Budzianowski et al. (2018) P. Budzianowski, T. Wen, B. Tseng, I. Casanueva, S. Ultes, O. Ramadan, and M. Gašić MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, p. 5016–5026. External Links: Link, Document Cited by: §3.1. Chang et al. (2025) S. Chang, A. Anderson, and J. M. Hofman ChatBench: from static benchmarks to human-AI evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 26009–26038. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. External Links: 2107.03374, Link Cited by: §3.1. Chiang et al. (2024) W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica Chatbot arena: an open platform for evaluating llms by human preference. External Links: 2403.04132, Link Cited by: §2, §3.2. Clark and Brennan (1991) H. H. Clark and S. E. Brennan Grounding in communication.. Cited by: §1, §2, §3.1, §3.2, §4.4. Clark and Wilkes-Gibbs (1986) H. H. Clark and D. Wilkes-Gibbs Referring as a collaborative process. Cognition 22 (1), p. 1–39. Cited by: §1, §2, §4.4. Dell’Acqua et al. (2026) F. Dell’Acqua, E. McFowland, E. Mollick, H. Lifshitz, K. C. Kellogg, S. Rajendran, L. Krayer, F. Candelon, and K. R. Lakhani Navigating the jagged technological frontier: field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality. Organization Science 37 (2), p. 403–423. External Links: Document, Link, https://doi.org/10.1287/orsc.2025.21838 Cited by: §2. Eloundou et al. (2024) T. Eloundou, S. Manning, P. Mishkin, and D. Rock GPTs are gpts: labor market impact potential of llms. Science 384 (6702), p. 1306–1308. External Links: Document, Link, https://w.science.org/doi/pdf/10.1126/science.adj0998 Cited by: §2. Fragiadakis et al. (2025) G. Fragiadakis, C. Diou, G. Kousiouris, and M. Nikolaidou Evaluating human-ai collaboration: a review and methodological framework. External Links: 2407.19098, Link Cited by: §2. Franke Föyen et al. (2025) L. Franke Föyen, E. Zapel, M. Lekander, E. Hedman-Lagerlöf, and E. Lindsäter Artificial intelligence vs. human expert: licensed mental health clinicians’ blinded evaluation of ai-generated and expert psychological advice on quality, empathy, and perceived authorship. Internet Interventions 41, p. 100841. External Links: ISSN 2214-7829, Document, Link Cited by: §3.1. Gans and Goldfarb (2026) J. S. Gans and A. Goldfarb O-ring automation. Working Paper Technical Report 34639, Working Paper Series, National Bureau of Economic Research. External Links: Document, Link Cited by: §2. Handa et al. (2025) K. Handa, A. Tamkin, M. McCain, S. Huang, E. Durmus, S. Heck, J. Mueller, J. Hong, S. Ritchie, T. Belonax, K. K. Troy, D. Amodei, J. Kaplan, J. Clark, and D. Ganguli Which economic tasks are performed with ai? evidence from millions of claude conversations. External Links: 2503.04761, Link Cited by: §1, §2. Hendrycks et al. (2021) D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt Measuring coding challenge competence with apps. External Links: 2105.09938, Link Cited by: §3.1. Hurst et al. (2024) A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Mądry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. T. Passos, A. Kirillov, A. Christakis, A. Conneau, A. Kamali, A. Jabri, A. Moyer, A. Tam, A. Crookes, A. Tootoochian, A. Tootoonchian, A. Kumar, A. Vallone, A. Karpathy, A. Braunstein, A. Cann, A. Codispoti, A. Galu, A. Kondrich, A. Tulloch, A. Mishchenko, A. Baek, A. Jiang, A. Pelisse, A. Woodford, A. Gosalia, A. Dhar, A. Pantuliano, A. Nayak, A. Oliver, B. Zoph, B. Ghorbani, B. Leimberger, B. Rossen, B. Sokolowsky, B. Wang, B. Zweig, B. Hoover, B. Samic, B. McGrew, B. Spero, B. Giertler, B. Cheng, B. Lightcap, B. Walkin, B. Quinn, B. Guarraci, B. Hsu, B. Kellogg, B. Eastman, C. Lugaresi, C. Wainwright, C. Bassin, C. Hudson, C. Chu, C. Nelson, C. Li, C. J. Shern, C. Conger, C. Barette, C. Voss, C. Ding, C. Lu, C. Zhang, C. Beaumont, C. Hallacy, C. Koch, C. Gibson, C. Kim, C. Choi, C. McLeavey, C. Hesse, C. Fischer, C. Winter, C. Czarnecki, C. Jarvis, C. Wei, C. Koumouzelis, D. Sherburn, D. Kappler, D. Levin, D. Levy, D. Carr, D. Farhi, D. Mely, D. Robinson, D. Sasaki, D. Jin, D. Valladares, D. Tsipras, D. Li, D. P. Nguyen, D. Findlay, E. Oiwoh, E. Wong, E. Asdar, E. Proehl, E. Yang, E. Antonow, E. Kramer, E. Peterson, E. Sigler, E. Wallace, E. Brevdo, E. Mays, F. Khorasani, F. P. Such, F. Raso, F. Zhang, F. von Lohmann, F. Sulit, G. Goh, G. Oden, G. Salmon, G. Starace, G. Brockman, H. Salman, H. Bao, H. Hu, H. Wong, H. Wang, H. Schmidt, H. Whitney, H. Jun, H. Kirchner, H. P. de Oliveira Pinto, H. Ren, H. Chang, H. W. Chung, I. Kivlichan, I. O’Connell, I. O’Connell, I. Osband, I. Silber, I. Sohl, I. Okuyucu, I. Lan, I. Kostrikov, I. Sutskever, I. Kanitscheider, I. Gulrajani, J. Coxon, J. Menick, J. Pachocki, J. Aung, J. Betker, J. Crooks, J. Lennon, J. Kiros, J. Leike, J. Park, J. Kwon, J. Phang, J. Teplitz, J. Wei, J. Wolfe, J. Chen, J. Harris, J. Varavva, J. G. Lee, J. Shieh, J. Lin, J. Yu, J. Weng, J. Tang, J. Yu, J. Jang, J. Q. Candela, J. Beutler, J. Landers, J. Parish, J. Heidecke, J. Schulman, J. Lachman, J. McKay, J. Uesato, J. Ward, J. W. Kim, J. Huizinga, J. Sitkin, J. Kraaijeveld, J. Gross, J. Kaplan, J. Snyder, J. Achiam, J. Jiao, J. Lee, J. Zhuang, J. Harriman, K. Fricke, K. Hayashi, K. Singhal, K. Shi, K. Karthik, K. Wood, K. Rimbach, K. Hsu, K. Nguyen, K. Gu-Lemberg, K. Button, K. Liu, K. Howe, K. Muthukumar, K. Luther, L. Ahmad, L. Kai, L. Itow, L. Workman, L. Pathak, L. Chen, L. Jing, L. Guy, L. Fedus, L. Zhou, L. Mamitsuka, L. Weng, L. McCallum, L. Held, L. Ouyang, L. Feuvrier, L. Zhang, L. Kondraciuk, L. Kaiser, L. Hewitt, L. Metz, L. Doshi, M. Aflak, M. Simens, M. Boyd, M. Thompson, M. Dukhan, M. Chen, M. Gray, M. Hudnall, M. Zhang, M. Aljubeh, M. Litwin, M. Zeng, M. Johnson, M. Shetty, M. Gupta, M. Shah, M. Yatbaz, M. J. Yang, M. Zhong, M. Glaese, M. Chen, M. Janner, M. Lampe, M. Petrov, M. Wu, M. Wang, M. Fradin, M. Pokrass, M. Castro, M. O. T. de Castro, M. Pavlov, M. Brundage, M. Wang, M. Khan, M. Murati, M. Bavarian, M. Lin, M. Yesildal, N. Soto, N. Gimelshein, N. Cone, N. Staudacher, N. Summers, N. LaFontaine, N. Chowdhury, N. Ryder, N. Stathas, N. Turley, N. Tezak, N. Felix, N. Kudige, N. Keskar, N. Deutsch, N. Bundick, N. Puckett, O. Nachum, O. Okelola, O. Boiko, O. Murk, O. Jaffe, O. Watkins, O. Godement, O. Campbell-Moore, P. Chao, P. McMillan, P. Belov, P. Su, P. Bak, P. Bakkum, P. Deng, P. Dolan, P. Hoeschele, P. Welinder, P. Tillet, P. Pronin, P. Tillet, P. Dhariwal, Q. Yuan, R. Dias, R. Lim, R. Arora, R. Troll, R. Lin, R. G. Lopes, R. Puri, R. Miyara, R. Leike, R. Gaubert, R. Zamani, R. Wang, R. Donnelly, R. Honsby, R. Smith, R. Sahai, R. Ramchandani, R. Huet, R. Carmichael, R. Zellers, R. Chen, R. Chen, R. Nigmatullin, R. Cheu, S. Jain, S. Altman, S. Schoenholz, S. Toizer, S. Miserendino, S. Agarwal, S. Culver, S. Ethersmith, S. Gray, S. Grove, S. Metzger, S. Hermani, S. Jain, S. Zhao, S. Wu, S. Jomoto, S. Wu, Shuaiqi, Xia, S. Phene, S. Papay, S. Narayanan, S. Coffey, S. Lee, S. Hall, S. Balaji, T. Broda, T. Stramer, T. Xu, T. Gogineni, T. Christianson, T. Sanders, T. Patwardhan, T. Cunninghman, T. Degry, T. Dimson, T. Raoux, T. Shadwell, T. Zheng, T. Underwood, T. Markov, T. Sherbakov, T. Rubin, T. Stasi, T. Kaftan, T. Heywood, T. Peterson, T. Walters, T. Eloundou, V. Qi, V. Moeller, V. Monaco, V. Kuo, V. Fomenko, W. Chang, W. Zheng, W. Zhou, W. Manassra, W. Sheu, W. Zaremba, Y. Patil, Y. Qian, Y. Kim, Y. Cheng, Y. Zhang, Y. He, Y. Zhang, Y. Jin, Y. Dai, and Y. Malkov GPT-4o system card. External Links: 2410.21276, Link Cited by: §3.2. Imai (2022) S. Imai Is github copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ICSE ’22, New York, NY, USA, p. 319–321. External Links: ISBN 9781450392235, Link, Document Cited by: §2. İnan et al. (2025) M. İnan, A. Sicilia, S. Dey, V. Dongre, T. Srinivasan, J. Thomason, G. Tür, D. Hakkani-Tür, and M. Alikhani Better slow than sorry: introducing positive friction for reliable dialogue systems. External Links: 2501.17348, Link Cited by: Table 3, §2, §3.3, §3.3, §3.3, §5. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. Ju and Aral (2026) H. Ju and S. Aral Collaborating with ai agents: field experiments on teamwork, productivity, and performance. External Links: 2503.18238, Link Cited by: §2. Kandel et al. (2012) S. Kandel, A. Paepcke, J. M. Hellerstein, and J. Heer Enterprise data analysis and visualization: an interview study. IEEE Transactions on Visualization and Computer Graphics 18 (12), p. 2917–2926. External Links: Document Cited by: §4.2. Karat et al. (1999) C. Karat, C. Halverson, D. Horn, and J. Karat Patterns of entry and correction in large vocabulary continuous speech recognition systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’99, New York, NY, USA, p. 568–575. External Links: ISBN 0201485591, Link, Document Cited by: §3.2. Keeney and Raiffa (1993) R. L. Keeney and H. Raiffa Decisions with multiple objectives: preferences and value trade-offs. Cambridge University Press. Cited by: §3.1. Kwa et al. (2026) T. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, N. Rush, S. V. Arx, R. Bloom, T. Broadley, H. Du, B. Goodrich, N. Jurkovic, L. H. Miles, S. Nix, T. Lin, N. Parikh, D. Rein, L. J. K. Sato, H. Wijk, D. M. Ziegler, E. Barnes, and L. Chan Measuring ai ability to complete long software tasks. External Links: 2503.14499, Link Cited by: §1, §2. Lee et al. (2022) M. Lee, P. Liang, and Q. Yang CoAuthor: designing a human-ai collaborative writing dataset for exploring language model capabilities. In CHI Conference on Human Factors in Computing Systems, CHI ’22, p. 1–19. External Links: Link, Document Cited by: §2. Lee et al. (2023) M. Lee, M. Srivastava, A. Hardy, J. Thickstun, E. Durmus, A. Paranjape, I. Gerard-Ursin, X. L. Li, F. Ladhak, F. Rong, R. E. Wang, M. Kwon, J. S. Park, H. Cao, T. Lee, R. Bommasani, M. S. Bernstein, and P. Liang Evaluating human-language model interaction. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §2. Li et al. (2025) T. Li, W. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica From crowdsourced data to high-quality benchmarks: arena-hard and benchbuilder pipeline. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 34209–34231. External Links: Link Cited by: §2. Lin et al. (2025) B. Y. Lin, Y. Deng, K. Chandu, A. Ravichander, V. Pyatkin, N. Dziri, R. Le Bras, and Y. Choi WildBench: benchmarking llms with challenging tasks from real users in the wild. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 47852–47870. External Links: Link Cited by: §2. METR (2026) METR Task-completion time horizons of frontier ai models. Note: https://metr.org/time-horizons/ Cited by: §1, §2. Noy and Zhang (2023) S. Noy and W. Zhang Experimental evidence on the productivity effects of generative artificial intelligence. Science 381 (6654), p. 187–192. External Links: Document, Link, https://w.science.org/doi/pdf/10.1126/science.adh2586 Cited by: §2. Patwardhan et al. (2025) T. Patwardhan, R. Dias, E. Proehl, G. Kim, M. Wang, O. Watkins, S. P. Fishman, M. Aljubeh, P. Thacker, L. Fauconnet, N. S. Kim, P. Chao, S. Miserendino, G. Chabot, D. Li, M. Sharman, A. Barr, A. Glaese, and J. Tworek GDPval: evaluating ai model performance on real-world economically valuable tasks. External Links: 2510.04374, Link Cited by: §1, §2. Peng et al. (2023) S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer The impact of ai on developer productivity: evidence from github copilot. External Links: 2302.06590, Link Cited by: §2. Podsakoff et al. (2003) P. Podsakoff, S. MacKenzie, J. Lee, and N. Podsakoff Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology 88, p. 879–903. External Links: Document Cited by: §3.2. Qian and Wexler (2024) C. Qian and J. Wexler Take it, leave it, or fix it: measuring productivity and trust in human-ai collaboration. In Proceedings of the 29th International Conference on Intelligent User Interfaces, IUI ’24, p. 370–384. External Links: Link, Document Cited by: §2. Roschelle and Teasley (1995) J. Roschelle and S. D. Teasley The construction of shared knowledge in collaborative problem solving. In Computer supported collaborative learning, p. 69–97. Cited by: §2. Scholich et al. (2025) T. Scholich, M. Barr, S. Wiltsey Stirman, and S. Raj A comparison of responses from human therapists and large language model–based chatbots to assess therapeutic communication: mixed methods study. JMIR Ment Health 12, p. e69709. External Links: ISSN 2368-7959, Document, Link, Link Cited by: §3.1. Schreyer and Pilat (2001) P. Schreyer and D. Pilat Measuring productivity. OECD Economic studies 33 (2), p. 127–170. Cited by: §3.1. Shaikh et al. (2024) O. Shaikh, K. Gligorić, A. Khetan, M. Gerstgrasser, D. Yang, and D. Jurafsky Grounding gaps in language model generations. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 6279–6296. External Links: Link, Document Cited by: §2. Shaikh et al. (2025) O. Shaikh, H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz Navigating rifts in human-LLM grounding: study and benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 20832–20847. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: Table 3, §2, §3.3, §3.3, §3.3. Shao et al. (2026) Y. Shao, V. Samuel, Y. Jiang, J. Yang, and D. Yang Collaborative gym: a framework for enabling and evaluating human-agent collaboration. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix C, §2, §3.2. Shen et al. (2025) S. Z. Shen, V. Chen, K. Gu, A. Ross, Z. Ma, J. Ross, A. Gu, C. Si, W. Chi, A. Peng, J. J. Shen, A. Talwalkar, T. Wu, and D. Sontag Completion ≠ collaboration: scaling collaborative effort with agents. External Links: 2510.25744, Link Cited by: §2. Shridhar et al. (2021) M. Shridhar, X. Yuan, M. Cote, Y. Bisk, A. Trischler, and M. Hausknecht ALFWorld: aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, External Links: Link Cited by: §2. Singh et al. (2026) A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, A. Nathan, A. Luo, A. Helyar, A. Madry, A. Efremov, A. Spyra, A. Baker-Whitcomb, A. Beutel, A. Karpenko, A. Makelov, A. Neitz, A. Wei, A. Barr, A. Kirchmeyer, A. Ivanov, A. Christakis, A. Gillespie, A. Tam, A. Bennett, A. Wan, A. Huang, A. M. Sandjideh, A. Yang, A. Kumar, A. Saraiva, A. Vallone, A. Gheorghe, A. G. Garcia, A. Braunstein, A. Liu, A. Schmidt, A. Mereskin, A. Mishchenko, A. Applebaum, A. Rogerson, A. Rajan, A. Wei, A. Kotha, A. Srivastava, A. Agrawal, A. Vijayvergiya, A. Tyra, A. Nair, A. Nayak, B. Eggers, B. Ji, B. Hoover, B. Chen, B. Chen, B. Barak, B. Minaiev, B. Hao, B. Baker, B. Lightcap, B. McKinzie, B. Wang, B. Quinn, B. Fioca, B. Hsu, B. Yang, B. Yu, B. Zhang, B. Brenner, C. R. Zetino, C. Raymond, C. Lugaresi, C. Paz, C. Hudson, C. Whitney, C. Li, C. Chen, C. Cole, C. Voss, C. Ding, C. Shen, C. Huang, C. Colby, C. Hallacy, C. Koch, C. Lu, C. Kaplan, C. Kim, C. Minott-Henriques, C. Frey, C. Yu, C. Czarnecki, C. Reid, C. Wei, C. Decareaux, C. Scheau, C. Zhang, C. Forbes, D. Tang, D. Goldberg, D. Roberts, D. Palmie, D. Kappler, D. Levine, D. Wright, D. Leo, D. Lin, D. Robinson, D. Grabb, D. Chen, D. Lim, D. Salama, D. Bhattacharjee, D. Tsipras, D. Li, D. Yu, D. Strouse, D. Williams, D. Hunn, E. Bayes, E. Arbus, E. Akyurek, E. Y. Le, E. Widmann, E. Yani, E. Proehl, E. Sert, E. Cheung, E. Schwartz, E. Han, E. Jiang, E. Mitchell, E. Sigler, E. Wallace, E. Ritter, E. Kavanaugh, E. Mays, E. Nikishin, F. Li, F. P. Such, F. de Avila Belbute Peres, F. Raso, F. Bekerman, F. Tsimpourlas, F. Chantzis, F. Song, F. Zhang, G. Raila, G. McGrath, G. Briggs, G. Yang, G. Parascandolo, G. Chabot, G. Kim, G. Zhao, G. Valiant, G. Leclerc, H. Salman, H. Wang, H. Sheng, H. Jiang, H. Wang, H. Jin, H. Sikchi, H. Schmidt, H. Aspegren, H. Chen, H. Qiu, H. Lightman, I. Covert, I. Kivlichan, I. Silber, I. Sohl, I. Hammoud, I. Clavera, I. Lan, I. Akkaya, I. Kostrikov, I. Kofman, I. Etinger, I. Singal, J. Hehir, J. Huh, J. Pan, J. Wilczynski, J. Pachocki, J. Lee, J. Quinn, J. Kiros, J. Kalra, J. Samaroo, J. Wang, J. Wolfe, J. Chen, J. Wang, J. Harb, J. Han, J. Wang, J. Zhao, J. Chen, J. Yang, J. Tworek, J. Chand, J. Landon, J. Liang, J. Lin, J. Liu, J. Wang, J. Tang, J. Yin, J. Jang, J. Morris, J. Flynn, J. Ferstad, J. Heidecke, J. Fishbein, J. Hallman, J. Grant, J. Chien, J. Gordon, J. Park, J. Liss, J. Kraaijeveld, J. Guay, J. Mo, J. Lawson, J. McGrath, J. Vendrow, J. Jiao, J. Lee, J. Steele, J. Wang, J. Mao, K. Chen, K. Hayashi, K. Xiao, K. Salahi, K. Wu, K. Sekhri, K. Sharma, K. Singhal, K. Li, K. Nguyen, K. Gu-Lemberg, K. King, K. Liu, K. Stone, K. Yu, K. Ying, K. Georgiev, K. Lim, K. Tirumala, K. Miller, L. Ahmad, L. Lv, L. Clare, L. Fauconnet, L. Itow, L. Yang, L. Romaniuk, L. Anise, L. Byron, L. Pathak, L. Maksin, L. Lo, L. Ho, L. Jing, L. Wu, L. Xiong, L. Mamitsuka, L. Yang, L. McCallum, L. Held, L. Bourgeois, L. Engstrom, L. Kuhn, L. Feuvrier, L. Zhang, L. Switzer, L. Kondraciuk, L. Kaiser, M. Joglekar, M. Singh, M. Shah, M. Stratta, M. Williams, M. Chen, M. Sun, M. Cayton, M. Li, M. Zhang, M. Aljubeh, M. Nichols, M. Haines, M. Schwarzer, M. Gupta, M. Shah, M. Y. Guan, M. Huang, M. Dong, M. Wang, M. Glaese, M. Carroll, M. Lampe, M. Malek, M. Sharman, M. Zhang, M. Wang, M. Pokrass, M. Florian, M. Pavlov, M. Wang, M. Chen, M. Wang, M. Feng, M. Bavarian, M. Lin, M. Abdool, M. Rohaninejad, N. Soto, N. Staudacher, N. LaFontaine, N. Marwell, N. Liu, N. Preston, N. Turley, N. Ansman, N. Blades, N. Pancha, N. Mikhaylin, N. Felix, N. Handa, N. Rai, N. Keskar, N. Brown, O. Nachum, O. Boiko, O. Murk, O. Watkins, O. Gleeson, P. Mishkin, P. Lesiewicz, P. Baltescu, P. Belov, P. Zhokhov, P. Pronin, P. Guo, P. Thacker, Q. Liu, Q. Yuan, Q. Liu, R. Dias, R. Puckett, R. Arora, R. T. Mullapudi, R. Gaon, R. Miyara, R. Song, R. Aggarwal, R. Marsan, R. Yemiru, R. Xiong, R. Kshirsagar, R. Nuttall, R. Tsiupa, R. Eldan, R. Wang, R. James, R. Ziv, R. Shu, R. Nigmatullin, S. Jain, S. Talaie, S. Altman, S. Arnesen, S. Toizer, S. Toyer, S. Miserendino, S. Agarwal, S. Yoo, S. Heon, S. Ethersmith, S. Grove, S. Taylor, S. Bubeck, S. Banesiu, S. Amdo, S. Zhao, S. Wu, S. Santurkar, S. Zhao, S. R. Chaudhuri, S. Krishnaswamy, Shuaiqi, Xia, S. Cheng, S. Anadkat, S. P. Fishman, S. Tobin, S. Fu, S. Jain, S. Mei, S. Egoian, S. Kim, S. Golden, S. Mah, S. Lin, S. Imm, S. Sharpe, S. Yadlowsky, S. Choudhry, S. Eum, S. Sanjeev, T. Khan, T. Stramer, T. Wang, T. Xin, T. Gogineni, T. Christianson, T. Sanders, T. Patwardhan, T. Degry, T. Shadwell, T. Fu, T. Gao, T. Garipov, T. Sriskandarajah, T. Sherbakov, T. Korbak, T. Kaftan, T. Hiratsuka, T. Wang, T. Song, T. Zhao, T. Peterson, V. Kharitonov, V. Chernova, V. Kosaraju, V. Kuo, V. Pong, V. Verma, V. Petrov, W. Jiang, W. Zhang, W. Zhou, W. Xie, W. Zhan, W. McCabe, W. DePue, W. Ellsworth, W. Bain, W. Thompson, X. Chen, X. Qi, X. Xiang, X. Shi, Y. Dubois, Y. Yu, Y. Khakbaz, Y. Wu, Y. Qian, Y. T. Lee, Y. Chen, Y. Zhang, Y. Xiong, Y. Tian, Y. Cha, Y. Bai, Y. Yang, Y. Yuan, Y. Li, Y. Zhang, Y. Yang, Y. Jin, Y. Jiang, Y. Wang, Y. Wang, Y. Liu, Z. Stubenvoll, Z. Dou, Z. Wu, and Z. Wang OpenAI gpt-5 system card. External Links: 2601.03267, Link Cited by: §3.2. Solow (1957) R. M. Solow Technical change and the aggregate production function. The Review of Economics and Statistics 39 (3), p. 312–320. External Links: ISSN 00346535, 15309142, Link Cited by: §3.1. Tamkin and McCrory (2025) A. Tamkin and P. McCroryEstimating ai productivity gains from claude conversations(Website) External Links: Link Cited by: §1, §2. Team et al. (2025) G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen, E. Pitler, T. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan, B. Lee, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Moreira, K. Ayoub, M. Goel, J. Krawczyk, C. Du, E. Chi, H. Cheng, E. Ni, P. Shah, P. Kane, B. Chan, M. Faruqui, A. Severyn, H. Lin, Y. Li, Y. Cheng, A. Ittycheriah, M. Mahdieh, M. Chen, P. Sun, D. Tran, S. Bagri, B. Lakshminarayanan, J. Liu, A. Orban, F. Güra, H. Zhou, X. Song, A. Boffy, H. Ganapathy, S. Zheng, H. Choe, Á. Weisz, T. Zhu, Y. Lu, S. Gopal, J. Kahn, M. Kula, J. Pitman, R. Shah, E. Taropa, M. A. Merey, M. Baeuml, Z. Chen, L. E. Shafey, Y. Zhang, O. Sercinoglu, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, I. Danihelka, B. Roelofs, A. White, A. Andreassen, T. von Glehn, L. Yagati, M. Kazemi, L. Gonzalez, M. Khalman, J. Sygnowski, A. Frechette, C. Smith, L. Culp, L. Proleev, Y. Luan, X. Chen, J. Lottes, N. Schucher, F. Lebron, A. Rrustemi, N. Clay, P. Crone, T. Kocisky, J. Zhao, B. Perz, D. Yu, H. Howard, A. Bloniarz, J. W. Rae, H. Lu, L. Sifre, M. Maggioni, F. Alcober, D. Garrette, M. Barnes, S. Thakoor, J. Austin, G. Barth-Maron, W. Wong, R. Joshi, R. Chaabouni, D. Fatiha, A. Ahuja, G. S. Tomar, E. Senter, M. Chadwick, I. Kornakov, N. Attaluri, I. Iturrate, R. Liu, Y. Li, S. Cogan, J. Chen, C. Jia, C. Gu, Q. Zhang, J. Grimstad, A. J. Hartman, X. Garcia, T. S. Pillai, J. Devlin, M. Laskin, D. de Las Casas, D. Valter, C. Tao, L. Blanco, A. P. Badia, D. Reitter, M. Chen, J. Brennan, C. Rivera, S. Brin, S. Iqbal, G. Surita, J. Labanowski, A. Rao, S. Winkler, E. Parisotto, Y. Gu, K. Olszewska, R. Addanki, A. Miech, A. Louis, D. Teplyashin, G. Brown, E. Catt, J. Balaguer, J. Xiang, P. Wang, Z. Ashwood, A. Briukhov, A. Webson, S. Ganapathy, S. Sanghavi, A. Kannan, M. Chang, A. Stjerngren, J. Djolonga, Y. Sun, A. Bapna, M. Aitchison, P. Pejman, H. Michalewski, T. Yu, C. Wang, J. Love, J. Ahn, D. Bloxwich, K. Han, P. Humphreys, T. Sellam, J. Bradbury, V. Godbole, S. Samangooei, B. Damoc, A. Kaskasoli, S. M. R. Arnold, V. Vasudevan, S. Agrawal, J. Riesa, D. Lepikhin, R. Tanburn, S. Srinivasan, H. Lim, S. Hodkinson, P. Shyam, J. Ferret, S. Hand, A. Garg, T. L. Paine, J. Li, Y. Li, M. Giang, A. Neitz, Z. Abbas, S. York, M. Reid, E. Cole, A. Chowdhery, D. Das, D. Rogozińska, V. Nikolaev, P. Sprechmann, Z. Nado, L. Zilka, F. Prost, L. He, M. Monteiro, G. Mishra, C. Welty, J. Newlan, D. Jia, M. Allamanis, C. H. Hu, R. de Liedekerke, J. Gilmer, C. Saroufim, S. Rijhwani, S. Hou, D. Shrivastava, A. Baddepudi, A. Goldin, A. Ozturel, A. Cassirer, Y. Xu, D. Sohn, D. Sachan, R. K. Amplayo, C. Swanson, D. Petrova, S. Narayan, A. Guez, S. Brahma, J. Landon, M. Patel, R. Zhao, K. Villela, L. Wang, W. Jia, M. Rahtz, M. Giménez, L. Yeung, J. Keeling, P. Georgiev, D. Mincu, B. Wu, S. Haykal, R. Saputro, K. Vodrahalli, J. Qin, Z. Cankara, A. Sharma, N. Fernando, W. Hawkins, B. Neyshabur, S. Kim, A. Hutter, P. Agrawal, A. Castro-Ros, G. van den Driessche, T. Wang, F. Yang, S. Chang, P. Komarek, R. McIlroy, M. Lučić, G. Zhang, W. Farhan, M. Sharman, P. Natsev, P. Michel, Y. Bansal, S. Qiao, K. Cao, S. Shakeri, C. Butterfield, J. Chung, P. K. Rubenstein, S. Agrawal, A. Mensch, K. Soparkar, K. Lenc, T. Chung, A. Pope, L. Maggiore, J. Kay, P. Jhakra, S. Wang, J. Maynez, M. Phuong, T. Tobin, A. Tacchetti, M. Trebacz, K. Robinson, Y. Katariya, S. Riedel, P. Bailey, K. Xiao, N. Ghelani, L. Aroyo, A. Slone, N. Houlsby, X. Xiong, Z. Yang, E. Gribovskaya, J. Adler, M. Wirth, L. Lee, M. Li, T. Kagohara, J. Pavagadhi, S. Bridgers, A. Bortsova, S. Ghemawat, Z. Ahmed, T. Liu, R. Powell, V. Bolina, M. Iinuma, P. Zablotskaia, J. Besley, D. Chung, T. Dozat, R. Comanescu, X. Si, J. Greer, G. Su, M. Polacek, R. L. Kaufman, S. Tokumine, H. Hu, E. Buchatskaya, Y. Miao, M. Elhawaty, A. Siddhant, N. Tomasev, J. Xing, C. Greer, H. Miller, S. Ashraf, A. Roy, Z. Zhang, A. Ma, A. Filos, M. Besta, R. Blevins, T. Klimenko, C. Yeh, S. Changpinyo, J. Mu, O. Chang, M. Pajarskas, C. Muir, V. Cohen, C. L. Lan, K. Haridasan, A. Marathe, S. Hansen, S. Douglas, R. Samuel, M. Wang, S. Austin, C. Lan, J. Jiang, J. Chiu, J. A. Lorenzo, L. L. Sjösund, S. Cevey, Z. Gleicher, T. Avrahami, A. Boral, H. Srinivasan, V. Selo, R. May, K. Aisopos, L. Hussenot, L. B. Soares, K. Baumli, M. B. Chang, A. Recasens, B. Caine, A. Pritzel, F. Pavetic, F. Pardo, A. Gergely, J. Frye, V. Ramasesh, D. Horgan, K. Badola, N. Kassner, S. Roy, E. Dyer, V. C. Campos, A. Tomala, Y. Tang, D. E. Badawy, E. White, B. Mustafa, O. Lang, A. Jindal, S. Vikram, Z. Gong, S. Caelles, R. Hemsley, G. Thornton, F. Feng, W. Stokowiec, C. Zheng, P. Thacker, Ç. Ünlü, Z. Zhang, M. Saleh, J. Svensson, M. Bileschi, P. Patil, A. Anand, R. Ring, K. Tsihlas, A. Vezer, M. Selvi, T. Shevlane, M. Rodriguez, T. Kwiatkowski, S. Daruki, K. Rong, A. Dafoe, N. FitzGerald, K. Gu-Lemberg, M. Khan, L. A. Hendricks, M. Pellat, V. Feinberg, J. Cobon-Kerr, T. Sainath, M. Rauh, S. H. Hashemi, R. Ives, Y. Hasson, E. Noland, Y. Cao, N. Byrd, L. Hou, Q. Wang, T. Sottiaux, M. Paganini, J. Lespiau, A. Moufarek, S. Hassan, K. Shivakumar, J. van Amersfoort, A. Mandhane, P. Joshi, A. Goyal, M. Tung, A. Brock, H. Sheahan, V. Misra, C. Li, N. Rakićević, M. Dehghani, F. Liu, S. Mittal, J. Oh, S. Noury, E. Sezener, F. Huot, M. Lamm, N. D. Cao, C. Chen, S. Mudgal, R. Stella, K. Brooks, G. Vasudevan, C. Liu, M. Chain, N. Melinkeri, A. Cohen, V. Wang, K. Seymore, S. Zubkov, R. Goel, S. Yue, S. Krishnakumaran, B. Albert, N. Hurley, M. Sano, A. Mohananey, J. Joughin, E. Filonov, T. Kępa, Y. Eldawy, J. Lim, R. Rishi, S. Badiezadegan, T. Bos, J. Chang, S. Jain, S. G. S. Padmanabhan, S. Puttagunta, K. Krishna, L. Baker, N. Kalb, V. Bedapudi, A. Kurzrok, S. Lei, A. Yu, O. Litvin, X. Zhou, Z. Wu, S. Sobell, A. Siciliano, A. Papir, R. Neale, J. Bragagnolo, T. Toor, T. Chen, V. Anklin, F. Wang, R. Feng, M. Gholami, K. Ling, L. Liu, J. Walter, H. Moghaddam, A. Kishore, J. Adamek, T. Mercado, J. Mallinson, S. Wandekar, S. Cagle, E. Ofek, G. Garrido, C. Lombriser, M. Mukha, B. Sun, H. R. Mohammad, J. Matak, Y. Qian, V. Peswani, P. Janus, Q. Yuan, L. Schelin, O. David, A. Garg, Y. He, O. Duzhyi, A. Älgmyr, T. Lottaz, Q. Li, V. Yadav, L. Xu, A. Chinien, R. Shivanna, A. Chuklin, J. Li, C. Spadine, T. Wolfe, K. Mohamed, S. Das, Z. Dai, K. He, D. von Dincklage, S. Upadhyay, A. Maurya, L. Chi, S. Krause, K. Salama, P. G. Rabinovitch, P. K. R. M, A. Selvan, M. Dektiarev, G. Ghiasi, E. Guven, H. Gupta, B. Liu, D. Sharma, I. H. Shtacher, S. Paul, O. Akerlund, F. Aubet, T. Huang, C. Zhu, E. Zhu, E. Teixeira, M. Fritze, F. Bertolini, L. Marinescu, M. Bölle, D. Paulus, K. Gupta, T. Latkar, M. Chang, J. Sanders, R. Wilson, X. Wu, Y. Tan, L. N. Thiet, T. Doshi, S. Lall, S. Mishra, W. Chen, T. Luong, S. Benjamin, J. Lee, E. Andrejczuk, D. Rabiej, V. Ranjan, K. Styrc, P. Yin, J. Simon, M. R. Harriott, M. Bansal, A. Robsky, G. Bacon, D. Greene, D. Mirylenka, C. Zhou, O. Sarvana, A. Goyal, S. Andermatt, P. Siegler, B. Horn, A. Israel, F. Pongetti, C. ". Chen, M. Selvatici, P. Silva, K. Wang, J. Tolins, K. Guu, R. Yogev, X. Cai, A. Agostini, M. Shah, H. Nguyen, N. Ó. Donnaile, S. Pereira, L. Friso, A. Stambler, A. Kurzrok, C. Kuang, Y. Romanikhin, M. Geller, Z. Yan, K. Jang, C. Lee, W. Fica, E. Malmi, Q. Tan, D. Banica, D. Balle, R. Pham, Y. Huang, D. Avram, H. Shi, J. Singh, C. Hidey, N. Ahuja, P. Saxena, D. Dooley, S. P. Potharaju, E. O’Neill, A. Gokulchandran, R. Foley, K. Zhao, M. Dusenberry, Y. Liu, P. Mehta, R. Kotikalapudi, C. Safranek-Shrader, A. Goodman, J. Kessinger, E. Globen, P. Kolhar, C. Gorgolewski, A. Ibrahim, Y. Song, A. Eichenbaum, T. Brovelli, S. Potluri, P. Lahoti, C. Baetu, A. Ghorbani, C. Chen, A. Crawford, S. Pal, M. Sridhar, P. Gurita, A. Mujika, I. Petrovski, P. Cedoz, C. Li, S. Chen, N. D. Santo, S. Goyal, J. Punjabi, K. Kappaganthu, C. Kwak, P. LV, S. Velury, H. Choudhury, J. Hall, P. Shah, R. Figueira, M. Thomas, M. Lu, T. Zhou, C. Kumar, T. Jurdi, S. Chikkerur, Y. Ma, A. Yu, S. Kwak, V. Ähdel, S. Rajayogam, T. Choma, F. Liu, A. Barua, C. Ji, J. H. Park, V. Hellendoorn, A. Bailey, T. Bilal, H. Zhou, M. Khatir, C. Sutton, W. Rzadkowski, F. Macintosh, R. Vij, K. Shagin, P. Medina, C. Liang, J. Zhou, P. Shah, Y. Bi, A. Dankovics, S. Banga, S. Lehmann, M. Bredesen, Z. Lin, J. E. Hoffmann, J. Lai, R. Chung, K. Yang, N. Balani, A. Bražinskas, A. Sozanschi, M. Hayes, H. F. Alcalde, P. Makarov, W. Chen, A. Stella, L. Snijders, M. Mandl, A. Kärrman, P. Nowak, X. Wu, A. Dyck, K. Vaidyanathan, R. R, J. Mallet, M. Rudominer, E. Johnston, S. Mittal, A. Udathu, J. Christensen, V. Verma, Z. Irving, A. Santucci, G. Elsayed, E. Davoodi, M. Georgiev, I. Tenney, N. Hua, G. Cideron, E. Leurent, M. Alnahlawi, I. Georgescu, N. Wei, I. Zheng, D. Scandinaro, H. Jiang, J. Snoek, M. Sundararajan, X. Wang, Z. Ontiveros, I. Karo, J. Cole, V. Rajashekhar, L. Tumeh, E. Ben-David, R. Jain, J. Uesato, R. Datta, O. Bunyan, S. Wu, J. Zhang, P. Stanczyk, Y. Zhang, D. Steiner, S. Naskar, M. Azzam, M. Johnson, A. Paszke, C. Chiu, J. S. Elias, A. Mohiuddin, F. Muhammad, J. Miao, A. Lee, N. Vieillard, J. Park, J. Zhang, J. Stanway, D. Garmon, A. Karmarkar, Z. Dong, J. Lee, A. Kumar, L. Zhou, J. Evens, W. Isaac, G. Irving, E. Loper, M. Fink, I. Arkatkar, N. Chen, I. Shafran, I. Petrychenko, Z. Chen, J. Jia, A. Levskaya, Z. Zhu, P. Grabowski, Y. Mao, A. Magni, K. Yao, J. Snaider, N. Casagrande, E. Palmer, P. Suganthan, A. Castaño, I. Giannoumis, W. Kim, M. Rybiński, A. Sreevatsa, J. Prendki, D. Soergel, A. Goedeckemeyer, W. Gierke, M. Jafari, M. Gaba, J. Wiesner, D. G. Wright, Y. Wei, H. Vashisht, Y. Kulizhskaya, J. Hoover, M. Le, L. Li, C. Iwuanyanwu, L. Liu, K. Ramirez, A. Khorlin, A. Cui, T. LIN, M. Wu, R. Aguilar, K. Pallo, A. Chakladar, G. Perng, E. A. Abellan, M. Zhang, I. Dasgupta, N. Kushman, I. Penchev, A. Repina, X. Wu, T. van der Weide, P. Ponnapalli, C. Kaplan, J. Simsa, S. Li, O. Dousse, F. Yang, J. Piper, N. Ie, R. Pasumarthi, N. Lintz, A. Vijayakumar, D. Andor, P. Valenzuela, M. Lui, C. Paduraru, D. Peng, K. Lee, S. Zhang, S. Greene, D. D. Nguyen, P. Kurylowicz, C. Hardin, L. Dixon, L. Janzer, K. Choo, Z. Feng, B. Zhang, A. Singhal, D. Du, D. McKinnon, N. Antropova, T. Bolukbasi, O. Keller, D. Reid, D. Finchelstein, M. A. Raad, R. Crocker, P. Hawkins, R. Dadashi, C. Gaffney, K. Franko, A. Bulanova, R. Leblond, S. Chung, H. Askham, L. C. Cobo, K. Xu, F. Fischer, J. Xu, C. Sorokin, C. Alberti, C. Lin, C. Evans, A. Dimitriev, H. Forbes, D. Banarse, Z. Tung, M. Omernick, C. Bishop, R. Sterneck, R. Jain, J. Xia, E. Amid, F. Piccinno, X. Wang, P. Banzal, D. J. Mankowitz, A. Polozov, V. Krakovna, S. Brown, M. Bateni, D. Duan, V. Firoiu, M. Thotakuri, T. Natan, M. Geist, S. tan Girgin, H. Li, J. Ye, O. Roval, R. Tojo, M. Kwong, J. Lee-Thorp, C. Yew, D. Sinopalnikov, S. Ramos, J. Mellor, A. Sharma, K. Wu, D. Miller, N. Sonnerat, D. Vnukov, R. Greig, J. Beattie, E. Caveness, L. Bai, J. Eisenschlos, A. Korchemniy, T. Tsai, M. Jasarevic, W. Kong, P. Dao, Z. Zheng, F. Liu, F. Yang, R. Zhu, T. H. Teh, J. Sanmiya, E. Gladchenko, N. Trdin, D. Toyama, E. Rosen, S. Tavakkol, L. Xue, C. Elkind, O. Woodman, J. Carpenter, G. Papamakarios, R. Kemp, S. Kafle, T. Grunina, R. Sinha, A. Talbert, D. Wu, D. Owusu-Afriyie, C. Du, C. Thornton, J. Pont-Tuset, P. Narayana, J. Li, S. Fatehi, J. Wieting, O. Ajmeri, B. Uria, Y. Ko, L. Knight, A. Héliou, N. Niu, S. Gu, C. Pang, Y. Li, N. Levine, A. Stolovich, R. Santamaria-Fernandez, S. Goenka, W. Yustalim, R. Strudel, A. Elqursh, C. Deck, H. Lee, Z. Li, K. Levin, R. Hoffmann, D. Holtmann-Rice, O. Bachem, S. Arora, C. Koh, S. H. Yeganeh, S. Põder, M. Tariq, Y. Sun, L. Ionita, M. Seyedhosseini, P. Tafti, Z. Liu, A. Gulati, J. Liu, X. Ye, B. Chrzaszcz, L. Wang, N. Sethi, T. Li, B. Brown, S. Singh, W. Fan, A. Parisi, J. Stanton, V. Koverkathu, C. A. Choquette-Choo, Y. Li, T. Lu, A. Ittycheriah, P. Shroff, M. Varadarajan, S. Bahargam, R. Willoughby, D. Gaddy, G. Desjardins, M. Cornero, B. Robenek, B. Mittal, B. Albrecht, A. Shenoy, F. Moiseev, H. Jacobsson, A. Ghaffarkhah, M. Rivière, A. Walton, C. Crepy, A. Parrish, Z. Zhou, C. Farabet, C. Radebaugh, P. Srinivasan, C. van der Salm, A. Fidjeland, S. Scellato, E. Latorre-Chimoto, H. Klimczak-Plucińska, D. Bridson, D. de Cesare, T. Hudson, P. Mendolicchio, L. Walker, A. Morris, M. Mauger, A. Guseynov, A. Reid, S. Odoom, L. Loher, V. Cotruta, M. Yenugula, D. Grewe, A. Petrushkina, T. Duerig, A. Sanchez, S. Yadlowsky, A. Shen, A. Globerson, L. Webb, S. Dua, D. Li, S. Bhupatiraju, D. Hurt, H. Qureshi, A. Agarwal, T. Shani, M. Eyal, A. Khare, S. R. Belle, L. Wang, C. Tekur, M. S. Kale, J. Wei, R. Sang, B. Saeta, T. Liechty, Y. Sun, Y. Zhao, S. Lee, P. Nayak, D. Fritz, M. R. Vuyyuru, J. Aslanides, N. Vyas, M. Wicke, X. Ma, E. Eltyshev, N. Martin, H. Cate, J. Manyika, K. Amiri, Y. Kim, X. Xiong, K. Kang, F. Luisier, N. Tripuraneni, D. Madras, M. Guo, A. Waters, O. Wang, J. Ainslie, J. Baldridge, H. Zhang, G. Pruthi, J. Bauer, F. Yang, R. Mansour, J. Gelman, Y. Xu, G. Polovets, J. Liu, H. Cai, W. Chen, X. Sheng, E. Xue, S. Ozair, C. Angermueller, X. Li, A. Sinha, W. Wang, J. Wiesinger, E. Koukoumidis, Y. Tian, A. Iyer, M. Gurumurthy, M. Goldenson, P. Shah, M. Blake, H. Yu, A. Urbanowicz, J. Palomaki, C. Fernando, K. Durden, H. Mehta, N. Momchev, E. Rahimtoroghi, M. Georgaki, A. Raul, S. Ruder, M. Redshaw, J. Lee, D. Zhou, K. Jalan, D. Li, B. Hechtman, P. Schuh, M. Nasr, K. Milan, V. Mikulik, J. Franco, T. Green, N. Nguyen, J. Kelley, A. Mahendru, A. Hu, J. Howland, B. Vargas, J. Hui, K. Bansal, V. Rao, R. Ghiya, E. Wang, K. Ye, J. M. Sarr, M. M. Preston, M. Elish, S. Li, A. Kaku, J. Gupta, I. Pasupat, D. Juan, M. Someswar, T. M., X. Chen, A. Amini, A. Fabrikant, E. Chu, X. Dong, A. Muthal, S. Buthpitiya, S. Jauhari, N. Hua, U. Khandelwal, A. Hitron, J. Ren, L. Rinaldi, S. Drath, A. Dabush, N. Jiang, H. Godhia, U. Sachs, A. Chen, Y. Fan, H. Taitelbaum, H. Noga, Z. Dai, J. Wang, C. Liang, J. Hamer, C. Ferng, C. Elkind, A. Atias, P. Lee, V. Listík, M. Carlen, J. van de Kerkhof, M. Pikus, K. Zaher, P. Müller, S. Zykova, R. Stefanec, V. Gatsko, C. Hirnschall, A. Sethi, X. F. Xu, C. Ahuja, B. Tsai, A. Stefanoiu, B. Feng, K. Dhandhania, M. Katyal, A. Gupta, A. Parulekar, D. Pitta, J. Zhao, V. Bhatia, Y. Bhavnani, O. Alhadlaq, X. Li, P. Danenberg, D. Tu, A. Pine, V. Filippova, A. Ghosh, B. Limonchik, B. Urala, C. K. Lanka, D. Clive, Y. Sun, E. Li, H. Wu, K. Hongtongsak, I. Li, K. Thakkar, K. Omarov, K. Majmundar, M. Alverson, M. Kucharski, M. Patel, M. Jain, M. Zabelin, P. Pelagatti, R. Kohli, S. Kumar, J. Kim, S. Sankar, V. Shah, L. Ramachandruni, X. Zeng, B. Bariach, L. Weidinger, T. Vu, A. Andreev, A. He, K. Hui, S. Kashem, A. Subramanya, S. Hsiao, D. Hassabis, K. Kavukcuoglu, A. Sadovsky, Q. Le, T. Strohman, Y. Wu, S. Petrov, J. Dean, and O. Vinyals Gemini: a family of highly capable multimodal models. External Links: 2312.11805, Link Cited by: §3.2. Tu et al. (2025) T. Tu, M. Schaekermann, A. Palepu, K. Saab, J. Freyberg, R. Tanno, A. Wang, B. Li, M. Amin, Y. Cheng, E. Vedadi, N. Tomasev, S. Azizi, K. Singhal, L. Hou, A. Webson, K. Kulkarni, S. S. Mahdavi, C. Semturs, J. Gottweis, J. Barral, K. Chou, G. S. Corrado, Y. Matias, A. Karthikesalingam, and V. Natarajan Towards conversational diagnostic artificial intelligence. Nature 642 (8067), p. 442–450. External Links: Link, Document Cited by: §3.1. Vidgen et al. (2025) B. Vidgen, A. Fennelly, E. Pinnix, J. Benchek, D. Khan, Z. Richards, A. Bridges, C. Huang, K. Sahu, A. Kottamasu, B. Ma, B. Hunsberger, I. Robinson, A. Datta, C. Mahapatra, D. Barton, C. R. Sunstein, E. Topol, B. Foody, and O. Nitski The ai productivity index (apex). External Links: 2509.25721, Link Cited by: §2. Walker et al. (1997) M. A. Walker, D. J. Litman, C. A. Kamm, and A. Abella PARADISE: a framework for evaluating spoken dialogue agents. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics, Madrid, Spain, p. 271–280. External Links: Link, Document Cited by: §2, §3.1. Webb (2020) M. Webb The impact of artificial intelligence on the labor market. SSRN Electronic Journal. External Links: Link, Document Cited by: §2. Wongsuphasawat et al. (2019) K. Wongsuphasawat, Y. Liu, and J. Heer Goals, process, and challenges of exploratory data analysis: an interview study. External Links: 1911.00568, Link Cited by: §4.2. Wu et al. (2025) S. Wu, M. Galley, B. Peng, H. Cheng, G. Li, Y. Dou, W. Cai, J. Zou, J. Leskovec, and J. Gao CollabLLM: from passive responders to active collaborators. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §2. Xie et al. (2024) T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu OSWORLD: benchmarking multimodal agents for open-ended tasks in real computer environments. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §2. Xuan et al. (2026) K. Xuan, P. Song, P. Lu, P. Han, W. Li, Z. Zhang, Z. He, W. Hua, M. Li, J. You, A. Weller, Y. Wang, and J. Pei Interactive evaluation requires a design science. External Links: 2605.17829, Link Cited by: §2. Yao et al. (2025) S. Yao, N. Shinn, P. Razavi, and K. R. Narasimhan $τ$-bench: a benchmark for tool- agent- user interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Zhao et al. (2024) W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng WildChat: 1m chatgpt interaction logs in the wild. External Links: 2405.01470, Link Cited by: Appendix B, §2, §3.2. Zheng et al. (2024) L. Zheng, W. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang LMSYS-chat-1m: a large-scale real-world LLM conversation dataset. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. Gonzalez, and I. Stoica Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 46595–46623. External Links: Link Cited by: §2. Zhou et al. (2024) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig WebArena: a realistic web environment for building autonomous agents. External Links: 2307.13854, Link Cited by: §2. Appendix A CoGym Dataset Task Details We use three task environments from Collaborative Gym (CoGym): travel planning, related work writing, and tabular analysis. CoGym is designed for human-agent collaboration in shared task environments, where both the human and the agent can communicate and interact with task specific tools. Travel planning. In the travel planning task, the human and agent collaborate to produce a detailed itinerary that satisfies a user’s travel goals and constraints. The task is challenging because the initial request may underspecify important preferences, such as preferred activities, destinations, budget constraints, accommodation needs, or transportation choices. The agent can use travel-related search tools, including city, attraction, restaurant, flight, and accommodation search, and can update a shared itinerary editor. The final artifact is the completed travel plan. Related work writing. In the related work task, the human and agent collaborate to write a related work section for a given research topic. The agent can search for relevant papers, add or remove papers from a shared library, transfer selected papers into a draft, and update a shared text editor. The final artifact is the related work section. Tabular analysis. In the tabular analysis task, the human and agent collaborate to derive analytical insights from provided tabular data. The environment includes shared access to the data, a Jupyter notebook for code execution, and a text editor for documenting findings. The agent can execute notebook cells and update the written analysis. The final artifact is an analytical report or finding supported by computations over the table. These three tasks provide variation in the role of interaction. Travel planning emphasizes latent preferences and constraint satisfaction; related work writing emphasizes synthesis, domain expertise, and iterative refinement; and tabular analysis emphasizes tool use, code execution, and evidence generation. This variation is important for our analysis because productivity should not be interpreted as a universal preference for shorter interactions. Instead, the relationship between quality and cost depends on the task structure. Appendix B Visualization Dataset Details Task. The visualization dataset was collected from an IRB approved study. Participants completed a structured exploratory analysis task using WildChat-1M (Zhao et al., 2024), a large scale dataset of real world ChatGPT interactions. The task required participants to formulate a testable hypothesis about the dataset, write or adapt Python code to analyze the data, produce a visualization, and submit a short written interpretation of their findings. This design was intended to capture a realistic human–LLM workflow in which users must move from problem formulation to analysis, visualization, and explanation, rather than simply rate model outputs or complete isolated microtasks. Participants. Participants were university students. This population is appropriate for the task because the study focuses on how users collaborate with an LLM to complete an open-ended analytical task. The study therefore captures collaboration in a setting where participants had a genuine task objective and were expected to produce a usable final artifact. In total, we collected 42 completed sessions. Interface and model. Participants interacted with the LLM through a custom web platform designed for the study. The interface presented the task instructions, provided access to the chat based assistant, and collected the final submission. The assistant was powered by GPT-5.1. The platform logged the full interaction trajectory, including user messages and model responses, so that each final artifact could be analyzed together with the process that produced it. Collected data. For each session, we collect the full chat transcript, timestamps if available, submitted code, final visualization, written interpretation, rubric score, and post task survey responses. Privacy and release. Because the dataset contains natural language interactions with an LLM, chat logs may include incidental personal information or sensitive content entered by participants. Before any release, we will remove direct identifiers, anonymize participant IDs, and filter or redact potentially identifying content. Appendix C Task Quality Rubrics We evaluate the final artifact from each collaboration session using task specific grading rubrics. Each rubric assigns an integer score from 1 to 5, where higher scores indicate higher task quality. The judge is given the original task request and the final artifact produced by the human-AI team. For tabular analysis, the judge is also given excerpts from the Jupyter execution history so that the final report can be evaluated against the computations used to produce it. The judge is instructed to provide a short justification and then conclude with a final numeric score. For the related work writing task, we reuse the rubric provided by Collaborative Gym in Figure 13 of Shao et al. (2026). We use custom rubrics for travel planning, tabular analysis, and the visualization task, shown below. C.1 Travel Planning Rubric The travel planning rubric evaluates whether the final itinerary satisfies the user’s request, covers the requested dates, includes concrete travel entities, and is internally realistic. Scoring Rubric for Travel Planning The evaluator is given: (a) the user’s travel-planning request, and (b) the final travel plan that the human-AI team produced, defined as the contents of the shared travel-plan editor at the end of the session. Evaluation Criteria Score = 1. The plan is missing entirely, gibberish, or unrelated to the user’s request, including the origin, destination, or dates. Score = 2. The plan exists but only sketches one or two days. Required slots, such as transportation, accommodation, attraction, breakfast, lunch, and dinner, are missing for most days. Specifics such as flight numbers, restaurant names, or hotel names are absent or appear fabricated. Score = 3. The plan covers all requested days but is partially incomplete. Some meals or attractions are left blank; flight numbers, accommodations, or restaurants are present but generic; or there are commonsense issues, such as destination mismatches, impossible timing, or missing intercity transportation. Score = 4. The plan covers every day and fills most meal, attraction, transportation, and accommodation slots with concrete entities, such as named flights, hotels, restaurants, or attractions. The itinerary is internally consistent, with transportation between cities scheduled before the day’s activities. Only minor gaps remain. Score = 5. The plan is complete and realistic for every day. It includes named flights with flight numbers, named hotels, named restaurants for each meal, and named attractions. It respects the user’s preferences, including budget, interests, and number of travelers; places transportation between cities correctly; and contains no obvious commonsense violations. The evaluator provides 2–4 sentences of justification and concludes with: Therefore, the final score is N. C.2 Tabular Analysis Rubric The tabular analysis rubric evaluates whether the final report answers the user’s analytical request, reports concrete numerical findings, and supports its conclusions with appropriate computations. Scoring Rubric for Tabular Data Analysis The evaluator is given: (a) the user’s analytical request, (b) the final analytical report, defined as the contents of the result editor at the end of the session, and (c) excerpts of the Jupyter execution history that produced the report. The evaluator is instructed to evaluate the final report. Evaluation Criteria Score = 1. There is no report, the report is gibberish, or it is unrelated to the question. Score = 2. The report addresses the question only superficially. It contains no specific numerical findings, no statistical justification, unsupported or generic conclusions, or conclusions that contradict the executed code. Score = 3. The report answers the question with at least one concrete numerical finding, but lacks rigor. For example, it omits error bars, significance tests, or sample sizes; examines only one slice of the data; or provides conclusions that are only partially supported. Score = 4. The report addresses the question with multiple specific numerical findings and uses appropriate aggregations or simple statistics. The conclusions follow from the analysis, with only minor gaps, such as missing caveats about data quality or the absence of a formal hypothesis test where one would have helped. Score = 5. The report directly answers the user’s question with concrete numbers, uses appropriate statistical testing where relevant, considers alternative explanations or confounders, and presents conclusions with clear caveats. The findings are consistent with the executed code. The evaluator provides 2–4 sentences of justification and concludes with: Therefore, the final score is N. C.3 Visualization Task Rubric The visualization rubric evaluates the quality of the submitted WildChat-1M analysis, including the hypothesis, code, statistical analysis, visualization, and final interpretation. Scoring Rubric for WildChat-1M Analysis The evaluator is given: (a) the prompt and constraints, (b) the final code submitted by the participant, and (c) the final output captured at submission. The evaluator is instructed to evaluate the quality of the submitted analysis. Evaluation Criteria Score = 1. No code is submitted, the submission is gibberish, or it is unrelated to the task. This score is also assigned when the submission violates the data-handling constraints in an obvious way, such as pasting raw conversations. Score = 2. The code attempts to load WildChat but does not actually compute the hypothesized analysis. There is no plot, only a placeholder plot, no stated hypothesis, or the output is empty or trivially limited to commands such as df.head(). Score = 3. The code states a clear hypothesis, runs at least one aggregation or group comparison consistent with the hypothesis, and produces some output. However, statistical testing is missing or weak; the visualization is basic or difficult to interpret; or the conclusions are missing or hand-waved. Score = 4. The hypothesis is clear and testable. The code uses appropriate aggregations and at least one statistical test or correlation. The visualization matches the hypothesis and includes proper labels and titles. The output is consistent with the analysis, and the data-handling constraints are respected. Score = 5. The hypothesis is well motivated and operationalized. The analysis combines aggregations, an appropriate statistical test with a reported statistic and p-value or effect size, and a polished, well-labeled visualization that supports the conclusion. The code is clean and self-contained, constraints are fully respected, and the conclusion explicitly follows from the results with appropriate caveats. The evaluator provides 2–4 sentences of justification and concludes with: Therefore, the final score is N. Appendix D Likert Correlations In addition to rubric scored task quality, both datasets include subjective Likert ratings collected after the task. These ratings capture participants’ perceptions of the outcome and interaction, but they are not interchangeable with our primary quality measure. Table 2 reports pairwise Spearman correlations among the available Likert measures within each dataset. For CoGym, participants rated the quality of the outcome, the agent, communication with the agent, and overall satisfaction. These measures are moderately to strongly correlated with one another, suggesting that participants’ subjective impressions of the interaction are not independent constructs. For example, outcome quality is strongly correlated with both agent rating and communication rating. For the visualization dataset, perceived usefulness, speed, and confidence are also positively correlated, with the strongest relationship between usefulness and speed. These correlations motivate our decision not to use subjective ratings as the primary quality signal Q. If subjective ratings were used to define outcome quality, later analyses comparing productivity against users’ perceived experience would become partially circular. Instead, we use independently rubric scored task performance as the primary quality measure and reserve subjective ratings for comparison analyses. This allows us to test whether users’ perceptions track the quality-cost tradeoff captured by our productivity metric. Dataset Rating pair ρ p n CoGym outcome – agent .60 <.001<.001 191 CoGym outcome – communication .62 <.001<.001 50 CoGym agent – communication .59 <.001<.001 50 Website usefulness – speed .67 <.001<.001 42 Website usefulness – confidence .56 <.001<.001 42 Website speed – confidence .26 .099 42 Table 2: Inter-item Spearman correlations among subjective Likert ratings. Subjective ratings are substantially correlated within each study, motivating our decision to use independently rubric-scored task performance as the primary quality signal Q and reserve subjective ratings for comparison analyses. Figure 5: Interaction cost within subjective outcome-rating levels. For CoGym, the subjective measure is the participant’s outcome quality rating; for the visualization task, it is confidence in the submitted result, since a direct outcome quality rating was not collected. Each point represents one collaboration session, and cost is shown on a log scale. Wide variation within the same rating level shows that subjective outcome ratings can hide large differences in the interaction cost required to produce the final artifact. Appendix E Cost Variation Within Human Likert Quality Levels Section 4.1 shows that sessions with identical rubric scored quality can require very different amounts of interaction. Here, we repeat the analysis using the closest available subjective outcome measure in each dataset. For CoGym, this is the participant’s Likert rating of outcome quality. For the visualization task, we use confidence in the submitted result as the closest subjective outcome oriented measure. Figure 5 plots interaction cost within each human outcome rating level. The same qualitative pattern appears: sessions with the same perceived outcome quality often differ substantially in interaction cost. Even when users assign the same rating to the final output, the amount of interaction required to reach that output can vary widely. This reinforces the motivation for our framework: outcome ratings alone, whether produced by a rubric judge or by the user, do not reveal whether the collaboration was productively achieved or costly to produce. Productivity therefore requires measuring quality and cost jointly rather than treating perceived quality as a complete evaluation signal. Appendix F Productivity Rankings Are Robust to Cost and Weighting Choices We validate our productivity metric: Pz=z⁡(Q)−z⁡(C)P_z=z(Q)-z(C) along two dimensions. First, we vary the relative weight assigned to agent tokens. Second, we compare productivity rankings under several alternative cost definitions: total tokens, user tokens only, agent tokens only, total turns, and user turns. F.1 Varying the Agent-Token Weight We define a family of weighted token costs: Cλ=user tokens+λ⋅agent tokens,C_λ=user tokens+λ·agent tokens, where λ∈0.50,0.75,1.00,1.50,2.00λ∈\0.50,0.75,1.00,1.50,2.00\. For each value of λ, we recompute productivity as Pz(λ)=z⁡(Q)−z⁡(Cλ),P_z^(λ)=z(Q)-z(C_λ), with Q and CλC_λ standardized within task. We then compute pairwise Spearman correlations between the resulting productivity rankings. Figure 6 shows that productivity rankings are stable across agent-token weights. For CoGym, pairwise Spearman correlations range from .77 to 1.00; for the visualization dataset, they range from .93 to 1.00. As expected, the lowest correlations occur when comparing the most different weighting choices, such as λ=.50λ=.50 versus λ=2.00λ=2.00. Even in these cases, rankings remain strongly correlated. This suggests that our conclusions are not driven by the specific choice of assigning agent tokens half the weight of user tokens. Figure 6: Productivity rankings are stable across alternative agent token weights. Each cell reports the Spearman correlation between productivity rankings computed using Cλ=user tokens+λ⋅agent tokensC_λ=user tokens+λ·agent tokens, for λ∈0.50,0.75,1.00,1.50,2.00λ∈\0.50,0.75,1.00,1.50,2.00\. The primary setting used in the main text is λ=.50λ=.50. Rankings remain strongly correlated across weights for both CoGym and the visualization dataset. F.2 Alternative Cost Measures We also test whether the productivity rankings are robust to broader changes in the cost measure. In addition to CdefaultC_default, we compute productivity using five alternative cost definitions: • Total tokens: user tokens + agent tokens. • User tokens: only tokens written by the user. • Agent tokens: only tokens generated by the agent. • Total turns: number of user and agent turns. • User turns: number of user turns. Figure 7 reports pairwise Spearman correlations between productivity rankings under these cost definitions. The rankings are highly stable. In CoGym, all pairwise correlations are at least .70, and the primary measure is especially close to total tokens (ρ=.99ρ=.99) and agent tokens (ρ=.96ρ=.96). In the visualization dataset, all pairwise correlations are at least .91, indicating even stronger agreement across cost definitions. The somewhat lower correlations for user only measures in CoGym are expected because CoGym agents can take actions in shared task environments and sometimes produce long tool mediated outputs. In these settings, considering only user messages can miss part of the reception burden imposed by the agent. Nevertheless, the overall pattern remains stable that sessions identified as productive under the primary cost definition tend to remain productive under alternative token- and turn-based definitions. Figure 7: Productivity rankings hold across alternative cost definitions. Each cell reports the Spearman correlation between productivity rankings computed with different cost measures. CdefaultC_default is the primary weighted token measure used in the main text. Rankings are strongly correlated across token-based and turn-based alternatives, suggesting that the main findings are not an artifact of a single cost operationalization. Appendix G Dialogue Feature Analysis: Full Results Section 4.4 reports the main dialogue patterns associated with productive collaboration. Here, we provide the full feature comparison between the top and bottom productivity quartiles. We compare sessions in the top and bottom quartiles of PzP_z across all tasks (n=59n=59 per quartile). For each feature, we compute the mean rate per turn in each quartile, Cohen’s d for the difference in means, and a Mann–Whitney U test. Positive effect sizes indicate features that are more frequent in productive sessions, while negative effect sizes indicate features that are more frequent in unproductive sessions. The full results in Table 3 support the interpretation in the main text. Productive sessions are characterized by more agent probing and more user acknowledgments or agreements. In contrast, unproductive sessions show more user probing, user assumption reveal, user repair, agent overresponse, and agent overspecification. This pattern suggests that productive collaboration is not defined by the absence of friction. Rather, productivity depends on how grounding work is distributed: productive sessions show agent side clarification followed by user confirmation, while unproductive sessions leave more repair and clarification work to the user. Feature (rate per turn) Top quartile Bottom quartile Cohen’s d p Friction taxonomy (İnan et al., 2025) agent probing 0.506 0.245 +0.73+0.73 0.0010.001*** probing (combined) 0.423 0.344 +0.36+0.36 0.0340.034* user reinforcement 0.047 0.032 +0.14+0.14 0.7830.783 reinforcement (combined) 0.060 0.050 +0.11+0.11 0.6500.650 user reflective pause 0.008 0.004 +0.09+0.09 0.5800.580 agent reinforcement 0.064 0.061 +0.02+0.02 0.4250.425 user overspecification 0.109 0.112 −0.01-0.01 0.1620.162 reflective pause (combined) 0.009 0.017 −0.18-0.18 0.1550.155 any friction 0.741 0.800 −0.31-0.31 0.0870.087 overspecification (combined) 0.204 0.270 −0.32-0.32 0.0200.020* agent assumption reveal 0.054 0.112 −0.33-0.33 0.1150.115 agent overspecification 0.321 0.459 −0.38-0.38 0.0220.022* agent reflective pause 0.007 0.033 −0.38-0.38 0.1870.187 user probing 0.273 0.425 −0.52-0.52 0.0050.005** user assumption reveal 0.038 0.120 −0.61-0.61 0.0000.000*** assumption reveal (combined) 0.045 0.119 −0.66-0.66 0.0010.001** Grounding act taxonomy (Shaikh et al., 2025) user acknowledgement 0.165 0.049 +0.58+0.58 0.0400.040* user agreement 0.156 0.045 +0.58+0.58 0.0090.009** user repeat 0.048 0.021 +0.29+0.29 0.9220.922 user disagreement 0.022 0.017 +0.07+0.07 0.3490.349 user topic switch 0.029 0.026 +0.04+0.04 0.3090.309 agent agreement 0.000 0.000 +0.00+0.00 1.0001.000 user clarification 0.054 0.067 −0.11-0.11 0.2200.220 agent clarification 0.025 0.042 −0.12-0.12 0.8230.823 user grounding friction 0.141 0.167 −0.15-0.15 0.2570.257 agent display 0.024 0.044 −0.22-0.22 0.2030.203 agent acknowledgement 0.032 0.077 −0.35-0.35 0.0380.038* user repair 0.039 0.080 −0.36-0.36 0.0100.010* user followup 0.401 0.508 −0.36-0.36 0.0720.072 agent response 0.605 0.739 −0.41-0.41 0.0400.040* agent overresponse 0.469 0.620 −0.46-0.46 0.0230.023* Table 3: Full rate based feature comparison between top and bottom PzP_z quartiles (n=59 each). Cohen’s d is computed on the difference in means; p-values from Mann–Whitney U tests. Features marked “(combined)” aggregate user and agent turns. Significance: *p<0.05p<0.05, **p<0.01p<0.01, ***p<0.001p<0.001. G.1 Per Task Correlations for Agent Probing The main text highlights agent probing as the clearest dialogue feature associated with productive collaboration. Table 4 reports the Spearman correlation between agent probing rate and PzP_z, both pooled across tasks and separately by task. The pooled correlation is positive and significant (ρ=.19,p=.003ρ=.19,p=.003), but the relationship is task dependent. Agent probing is most strongly associated with productivity in related work writing (ρ=.35ρ=.35) and travel planning (ρ=.24ρ=.24), the two tasks where clarification and iterative refinement are often useful. In contrast, the association is near zero in tabular analysis and visualization. This supports the broader claim that productive friction is task relative: agent probing appears most useful when the task benefits from eliciting preferences, constraints, or framing decisions, but less useful when success depends on quickly converging on an executable analysis or visualization. Task ρ p n All tasks +0.19+0.19 0.0030.003** 233 Related work +0.35+0.35 0.0160.016* 47 Travel planning +0.24+0.24 0.0290.029* 83 Tabular analysis +0.03+0.03 0.8300.830 61 Visualization −0.05-0.05 0.7310.731 42 Table 4: Spearman correlations between agent probing rate (rate agent probing) and PzP_z, pooled across all sessions and per-task. The effect is concentrated in tasks where iterative clarification helps (related work, travel planning) and absent in tasks where success depends more on quick convergence (tabular analysis, visualization). *p<0.05p<0.05, **p<0.01p<0.01.

Instant research discovery