Paper deep dive
Thinkink: 2D Spatial Ink-native Interaction with LLMs
Mohammad Hasan Payandeh, Daniel Vogel, Jian Zhao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/25/2026, 1:11:48 AM
Summary
The paper introduces Thinkink, a digital inking tool that integrates Large Language Models (LLMs) into a 2D spatial canvas for ideation. It allows users to input handwritten text and sketches, which the system interprets via a semantic tree to generate LLM responses as ink-like text and sketches directly on the canvas. The design was informed by a three-stage user study process involving formative research, a technical probe, and a final usage study with 28 participants total.
Entities (8)
Relation Signals (6)
Thinkink â uses â Large Language Models
confidence 95% ¡ Thinkink integrates large language models (LLMs) into this practice.
Thinkink â supports â 2D Spatial Canvas
confidence 92% ¡ ...write and draw in a shared 2D canvas.
Formative Study â informed â Technical Probe
confidence 90% ¡ The results informed a technical probe for a diagnostic study...
Thinkink â uses â Semantic Tree
confidence 90% ¡ A semantic tree streamlines ink interpretation...
Thinkink â uses â State Machine
confidence 88% ¡ ...provides explicit control using a state machine.
Technical Probe â uses â Gemini-3-flash-preview
confidence 85% ¡ It leverages LLMs (Gemini 3 Flash Preview [11]) to continuously analyze user drawings...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:People often use handwritten notes and sketches to externalize ideas for ideation. To integrate large language models (LLMs) into this practice, we propose Thinkink. Prompts can be handwritten text or drawn sketches with LLM-generated responses visualized as ink-like text and sketches spatially integrated into a shared canvas. A semantic tree streamlines ink interpretation, and a lightweight UI provides explicit control using a state machine. The tool was designed using a three-stage process. A formative study (N=12) examined current practices with conventional and digital inking methods. The results informed a technical probe for a diagnostic study (N=6) identifying usability and human-LLM interaction challenges. This motivated the design of Thinkink, with a final study (N=10) examining how people incorporate it into their ideation practices. We contribute design implications and a tool for ink-native LLM interaction where the user and LLM write and draw in a shared 2D canvas.
Tags
Links
- Source: https://arxiv.org/abs/2607.21468v1
- Canonical: https://arxiv.org/abs/2607.21468v1
Trouble viewing inline? Open PDF directly â
Full Text
96,574 characters extracted from source content.
Expand or collapse full text
arXiv:2607.21468v1 [cs.HC] 23 Jul 2026 Thinkink: 2D Spatial Ink-native Interaction with LLMs Mohammad Hasan Payandeh Cheriton School of Computer Science, University of Waterloo Waterloo, Canada mpayandeh@uwaterloo.ca Daniel Vogel Cheriton School of Computer Science, University of Waterloo Waterloo, Canada dvogel@uwaterloo.ca Jian Zhao Cheriton School of Computer Science, University of Waterloo Waterloo, Canada jianzhao@uwaterloo.ca Figure 1: Example Thinkink usage scenario for logo ideation: (a) user writes high-level ideation goal; (b) system in âAI insightsâ mode responds with writing around the goal, including a reflective question, a proofreading point, a text generation request, and image generation request; (c) user considers reflective question and adds logo design ideas including âThinkinkâ, sketches of a pen and star, and âbrain drawingâ concept; (d) user switches to âAsk AIâ mode, sketches an arrow, and writes âDrawâ, the system sets the context shown in red and responds with an initial logo concept; (e) user regenerates an alternative logo with the same context; (f) user adds the star to the context and regenerates a refined a logo concept. Abstract People often use handwritten notes and sketches to externalize ideas for ideation. To integrate large language models (LLMs) into this practice, we propose Thinkink. Prompts can be handwritten text or drawn sketches with LLM-generated responses visualized as ink-like text and sketches spatially integrated into a shared canvas. A semantic tree streamlines ink interpretation, and a lightweight UI provides explicit control using a state machine. The tool was designed using a three-stage process. A formative study (N=12) examined current practices with conventional and digital inking methods. The results informed a technical probe for a diagnostic study (N=6) identifying usability and human-LLM interaction chal- lenges. This motivated the design of Thinkink, with a final study (N=10) examining how people incorporate it into their ideation practices. We contribute design implications and a tool for ink- native LLM interaction where the user and LLM write and draw in a shared 2D canvas. Keywords Digital Inking, 2D Spatial Canvas, Human-AI Interaction, Large Language Models (LLMs) 1 Introduction People often use handwritten notes and drawings to externalize their thoughts spatially on paper and digital devices [18,21,25]. This can support ideation by helping people reflect, validate, and explore new thoughts [1,12]. However, people can reach a dead- lock and need support to explore new information, directions, and connections to move forward [24,37,41]. LLMs can fill this gap by expanding the depth and breadth of existing thoughts [13,36]. However, the typical chatbot-style of interaction does not match the non-linear two-dimensional nature of writing and drawing. Prior work has explored LLM support for ideation [4,7,34], sketch-based code editing [44], ink-based note-taking [9,35,43], and sketching in general [17,26,38]. These systems demonstrate the value of LLM assistance for either inking or ideation, but they do not explore the intersection of the two. Our research builds on these past work but is distinguished by its focus on using LLMs for ideation with all interaction entirely within an ink-native canvas. To explore this space, we conducted a user-centered, iterative process including three stages. Each stage investigated different questions, but they were connected through four cross-cutting aspects, consisting of Ideation workflow, UI and Interaction Design, LLM Integration, and LLM Outputs. Throughout this process, we developed a technical probe and then refined it into Thinkink, a digital ink-native LLM-assisted ideation tool that enables people to interact with an LLM through ink input and receive ink-like responses directly on the canvas. First, we conducted a formative focus group with 12 participants to understand RQ1: What user practices, needs, and expectations should inform the design of a digital ink-native LLM-assisted tool for ideation? Participants completed individual and collaborative 1 Payandeh et al. ideation activities using their preferred media and reflected on their workflows. Based on participant feedback and observations, we derived four design guidelines. In Stage 2, we translated these guidelines into a technical probe: a pen-first canvas with a semantic tree as backbone that inter- prets ink as drawings, concepts, requests, and generations, enabling context-aware, on-canvas LLM responses. Using the probe, we con- ducted a diagnostic study with 6 participants to understand RQ2: What human-LLM interaction challenges emerge when users engage with the technical probe? By observing participants using the probe across two ideation tasks and conducting follow-up interviews, we identified four key interaction design challenges. In Stage 3, we addressed these challenges in refined prototype tool called Thinkink. It uses a state-machine interaction design that separates note-taking from LLM assistance, by making explicit prompting, insights, request iteration, and generation inspection distinct modes. We conducted a usage study with 10 participants to examine RQ3: How do users utilize Thinkink in practice? The results revealed four usage patterns that validate the Thinkink design and also suggest approaches and priorities for ink-native LLM interaction in general. In summary, we make the following contributions: â˘Empirical knowledge that includes design guidelines, improve- ment areas, usage patterns, and discussions for building ink- native tools that unify inking workflows with LLM support, grounded in findings from a user-centered, iterative process. â˘Thinkink, a digital inking tool for ideation that enables ink-based prompting and ink-like LLM responses directly on a 2D canvas. 2 Background and Related Work This section discusses related work on ideation, digital inking, and LLM-supported inking. We do not review the broader creativity- support literature spanning early uncertain phases to later well- defined stages, as our work focuses specifically on ideation. 2.1 Ideation and Supporting Technologies Ideation is the process of generating new evidence or information that satisfies a goal, it is often described as divergent thinking or brainstorming [22]. When people engage in ideation, they rely on abductive reasoning, which means starting from evidence, usually incomplete, to find a plausible explanation [37,41]. This is differ- ent than other types of reasoning, such as deductive reasoning which applies rules to derive necessary conclusions and inductive reasoning which generalizes from examples. In ideation, people specifically rely on abductive reasoning to externalize concepts, notice patterns, and identify new information, including new in- terpretations or next possibilities [24]. Thus, ideation is not only idea generation, but also an abductive and explanatory process of making sense of partial evidence [12]. Externalization enhances abductive reasoning [18,21,25]. One approach to externalization is self-explanation, or âRubberduckingâ, where people explain their internal thoughts out loud to articulate assumptions and deepen understanding [5,31]. Our focus is exter- nalization using visuals by writing and sketching thoughts, an ap- proach that works well for visual thinkers [1,37]. Both approaches help inspect and reorganize own thinking, but by definition they largely work with knowledge a person already has; they do not contribute new perspectives, analogies, or candidate directions. Recent work uses LLMs to address this issue, framed as part of a broader agenda to create âtools for thoughtâ that augment rather than replace cognition [36]. In ideation, LLMs can support divergence in three ways: they can broaden semantic breadth by surfacing concepts from a wider conceptual space, suggest alterna- tive framings or directions, and sustain divergence by helping users exploring before converging too early on a single solution. Prior work has been explored LLMs as brainstorming through prompt frameworks for eliciting creative thoughts [4], collabora- tive brainwriting , a group ideation method in which participants iteratively write and build on one anotherâs ideas, as well as idea evaluation [34], and role-based accounts of humanâAI ideation that describe different divisions of labor, such as AI as the primary idea generator versus AI as support for human-led ideation [7]. [13]. Case studies have also shown how LLMs can act as brainstorming partners in complex domains such as mathematics, scientific re- search, and other technical problem-solving tasks [13]. Collectively, this work suggests that LLMs can expand the space of possibilities considered during ideation. However, interaction with LLMs re- mains largely conversational and text-centric, with ideas exchanged through prompts, chat, or documents. This mode of interaction can be suboptimal, especially for visual thinkers who externalize ideas through sketching and handwriting, as it requires constant back- and-forth between the inking canvas and the LLM interface. Our work fills this gap by intertwining peopleâs sketches and handwrit- ing with LLM-generated responses on a unified canvas. 2.2 Digital Inking Tools Digital tools demonstrate how pen input (i.e. the âinkâ) can enhance note-taking, sketching, and thinking through direct, low-friction interaction. They have shown that ink is not only a recording medium, but also an interaction primitive. For example, ActiveInk [33] supports sensemaking by letting users annotate data and then activate those strokes for analytic actions, linking externalization and interaction. Work on Sketchnot- ing [45] further characterizes how people combine text, diagrams, layout, and styling in visual notes, highlighting the importance of structure, recomposition, and lightweight expressiveness in digital tools. These systems establish key design principles for pen-based environments: preserve the immediacy of ink, support mixed visual representations, and keep interaction close to the workspace. InkSeine [15] uses existing ink to initiate in-situ search, treats queries as first-class objects, and lets users fluidly interleave note- taking with retrieval [3]. It shows how ink can act as a simple input command without forcing users to leave the canvas. How- ever, InkSeine routes those interpretations into a separate search interface rather than using the canvas itself as a shared space for AI-mediated ideation and response. We are inspired from this work by using ink as input, but we also integrate the resulting informa- tion in an ink-like form directly within a 2D spatial canvas, not in a separate interface. This demands careful interaction design, as a single workspace must support both traditional inking actions, such as writing, erasing, and moving drawings, and AI-specific actions, such as writing prompts, erasing prompts, and generating 2 Thinkink responses, while remaining easy to understand and fluid to use during open-ended exploration [14, 42]. More broadly, whereas prior digital inking tools treat ink pri- marily as user-authored input for note-taking, search, or analytic commands, we treat the canvas as a shared ideation space in which AI also contributes directly in ink. 2.3 AI-powered Digital Inking A growing body of prior work utilize AI/LLMs combined with digital ink, but for tasks other than ideation. Some systems im- prove writing itself: SkipWriter [43] uses LLMs for abbreviated handwritten text input, or Korovai et al. [23]propose handwrit- ing enhancement techniques that beautify handwritten text while preserving personal style. Others focus on recognizing and ma- nipulating ink. Inkeraction segments and classifies handwritten objects, identifies relationships, and synthesizes strokes to support editing and writing workflows [35], while InkFM provides a foun- dation for full-page handwritten note understanding across text, math, and drawings [9]. These works share an important technical foundation, namely the ability of AI to interpret ink correctly. How- ever, when they model relationships, the semantics are typically relatively simple and static; they are less suited to interpreting the evolving, context-dependent meaning of mixed sketches and notes during open-ended ideation (e.g. a circle with a line under it can be a balloon, but the same circle, when the user draws a hand for the line, is the head of a stickman). A few systems come closer to our work. Code Shaping shows that free-form sketches can communicate editing intent to AI, but in the specialized context of code manipulation [44]. Visual Sketch- pad gives multimodal language models an internal visual chain-of- thought [16] aimed at improving model reasoning, not supporting end-user ideation on a shared canvas. SketchGPT lets users interact with LLMs using sketches and speech to generate context-aware responses [17], producing confirmable operation lists on the under- lying UI. Common across all three works, the interaction is mainly one-way, from user ink to system interpretation, and responses are delivered through interface actions rather than as ink-native contributions within the same 2D workspace. ImaginationVellum presents a generative-AI canvas where the inks in a 2D workspace acts as a prompt for visual generation [26], but its domain is image ideation rather than ink-based conceptual exploration. SketchAgent supports conversational sequential sketch generation and refine- ment [38], but it primarily completes or modifies a drawing rather than helping users explore divergent idea spaces. What is missing is an ideation-focused LLM that shares the page: it incrementally grounds in the evolving, spatial semantics of mixed ink and responds in ink (placing lightweight annotations/sketches directly in the same 2D workspace), rather than only triggering UI actions or generating standalone images. Our work brings together ideation with ink-native support. Un- like prior ideation tools, it is not primarily chat-based; unlike prior ink systems, it is not limited to recognition, beautification, or com- mand invocation. Instead, it treats ink as both a cognitive artifact and an interaction medium for generative exploration, enabling users and LLM to co-develop ideas directly on the same canvas. 3 Formative Study To address RQ1: What user practices, needs, and expectations should inform the design of a digital ink-native LLM-assisted tool for ideation? we conducted a formative study with a focus group [2]. Our goal was to identify design guidelines for designing a technical probe as an initial prototype. 3.1 Method We conducted a 2-hour in-person focus group with 12 participants from an HCI laboratory (M=7, F =3, ND=2; age 23â40). The cohort included two professors, and ten Masterâs and PhD students. Four participants listed data visualization as their primary research area, one worked in XR/VR, one in interactive art and technology, and the remainder indicated general HCI. Participants took part voluntarily and did not receive any compensation. The session combined an externalization activity with a group reflection. After a short introduction and consent process, partici- pants read a brief sheet describing question categories and examples that encourage ideation (See Appendix A.1). Each participant then selected or authored a question to answer individually and, option- ally, another for pair work (See Appendix A.2). They were asked to write the question, list at least five thought or evidence they have in their mind, externalize these evidence visually, think, ex- plore, and draft an answer. Participants could use any medium they preferred, including pen and paper, digital inking tools, or other non-inking digital tools such as Excalidraw [8], draw.io [19], or Figma [10], and they were also allowed to consult LLMs such as ChatGPT [29]. We supplied paper, pens, sticky notes, and colored pencils. The individual activity lasted approximately 20 minutes, followed by an optional 20-minute collaborative activity and a 20- minute semi-structured group discussion (See Appendix A.3 for the discussion points). Discussion points elicited contributions from multiple participants; quotes are reported without identifiers due to the shared recording setup. We collected photographs or screen- shots of the final artifacts, resulting in 15 sketches: 12 on paper, 2 created using a digital inking tool, and 1 in Figma; 12 individual and 3 collaborative (see Appendix A.2 for table of participant medium choices and A.4 for the artifacts). 3.2 Design Guidelines We transcribed the discussion verbatim and conducted an inductive thematic analysis [20] of the transcript together with an inspection of the artifacts. Four different high-level aspects served as analytic lenses throughout our process: Ideation workflow concerns how users externalize, organize, and develop ideas; UI and Interaction Design concerns the pen-first interactions and interface structures that support this process; LLM Integration concerns how and when LLM capabilities are invoked within the digital inking tool; and LLM Outputs concerns what the LLM generates and how those responses are presented on the canvas. Ideation workflow: Support Fluid Ideation Through Evolving Externalization Workflows. Participants described externaliza- tion as generative rather than merely representational: âI definitely think externalizing context is helpful ... it helps me to open my mindâ, and âI personally feel like once I start visualizing this causation, I start 3 Payandeh et al. brainstorming more causations ... expanding thisâ. However, their workflows were rarely linear. Some struggled to begin (âItâs hard for me to find a starting point ... how would I visualize my thoughts?â ), some moved from text to visuals (âYou first write down the context, and then visualizeâ ), and others felt a fixed sequence was constrain- ing (âI follow the strict process ... and I feel like that kind of restricts meâ ). Several also found it difficult to decide on an appropriate representation (âI couldnât really think about how to visualize itâ ). Together, these findings suggest that an ink-native LLM-assisted ideation tool should support evolving, non-linear externalization: users should be able to begin with partial text, rough marks, or simple structures, receive lightweight scaffolds when needed, and continuously revise and reorganize the canvas as ideas develop. UI and Interaction Design: Enable a Minimal, Paper-Like Ink- ing Experience. Medium choice strongly favoured paper: 12 of the 15 artifacts were created on pen and paper. Participants associated paper with low distraction and immediacy, noting that âPen and paper is less distracting ... I lose my train of thought more easily with softwareâ and âI could get many notifications on an iPadâ. Digital tools were mainly valued for their malleability, as one participant noted: âiPad is quite limited ... but moving things around is super easyâ. Artifacts also relied mostly on simple diagrammatic elements such as arrows, circles, rectangles, clouds, and handwritten labels, with colours appearing in only 2 of the 15 artifacts (see Appen- dix A.4). A digital ink-native LLM-assisted tool should therefore minimize interface chrome and preserve a paper-like experience: a full-screen canvas, pen-first interaction, simple strokes and ba- sic shapes, and lightweight touch-based manipulation that retains digital flexibility without interrupting thought. LLM Integration: Integrate LLMs as an Ideation Partner Within the Canvas. Many participants said they view chatbots such as ChatGPT as tools for ideation and filling knowledge gaps, rather than as replacements for their own reasoning. Some used chatbots in the study. As one participant explained, âthe AI (LLMs) helped me to really get to think deeper and if I get stuck then I have helpâ. At the same time, participants noted that this support may be more helpful in domain-specific contexts: âif youâre doing some- thing...highly specialized...you might get as much helpâ. This en- courages integrating LLMs as a contextual ideation partner that works from the evolving sketch itself, helping users expand, con- nect, and elaborate ideas in place. Rather than functioning as a separate text-only assistant, the LLMs should be grounded in the current canvas state and contribute support that remains tightly coupled to the userâs ongoing externalization. LLM Outputs: Favour Exploratory, Question-Asking LLM Outputs Over Direct Answers. Participants were cautious about LLM Outputs that could bias their thinking or narrow the space of possibilities. Instead of direct answers, some of them preferred prompts that stimulate exploration, as reflected in the suggestion: âWhat if it[LLMs] can provoke us, ask us some questions...that will be more helpful in terms of opening upâ. For digital ink-native ideation tools, LLM Outputs should therefore prioritize exploratory support, such as reflective questions, alternative perspectives, tensions, and missing considerations, over declarative answers. This style of out- put is better aligned with sketch-based ideation because it expands the search space, preserves user agency, and helps users continue reasoning through the canvas rather than deferring to the model. 4 Technical Probe To explore how LLM support can be integrated directly into an inking workflow, we developed a technical probe using a Vite- based React and TypeScript canvas application [27,32,40]. The probe maintains a semantic tree behind the scenes (unseen by users) as a backend structure. It leverages LLMs (Gemini 3 Flash Preview [11]) to continuously analyze user drawings and translate the evolving workspace into a structured representation, enabling the system to better understand user input and respond to requests. This representation enables the system to interpret what the user has drawn and subsequently provide context-aware LLM responses directly on the canvas (Figure 2). Interaction traces are persisted using a lightweight Express logging server [30]. Following the fluid externalization ideation workflow guideline form the formative study, the technical probe supports external- ization by letting users begin with partial text, rough sketches, or simple marks and then continue adding, moving, and refining content on an open canvas without following a fixed sequence. After short idle periods, the system re-analyzes the workspace. Fol- lowing the minimal paper-like inking UI guideline, it preserves a minimal, paper-like interaction style through a full-screen drawing surface, direct pen input, and lightweight touch gestures for nav- igation and editing rather than menus or chat panels. Following the on-canvas LLM ideation partner guideline, it integrates LLMs as an ideation partner within the canvas by grounding requests in specific drawings and returning responses as on-canvas textual annotations positioned near the relevant part of the sketch. Finally, following the exploratory, question-first LLM outputs guideline, the probe emphasizes exploratory support by recognizing opportuni- ties for follow-up questions, information retrieval, or suggestions for what to draw next, enabling the system to extend the userâs thinking in place rather than only delivering detached answers. 4.1 User Interface The probe interface is a large, infinite white canvas with a subtle background grid and no visible UI buttons, so user drawings and LLM-generated content coexist directly on the workspace. User strokes remain primary in black, while LLM outputs are positioned near the triggering sketch so they read as spatial additions rather than separate chat messages; LLM generations they as tentative, semi-transparent annotations in two forms: teal for generation without request and indigo for requests. Tapping a request produces its response in teal. Tapping any generated content (with or without a request) finalizes it by turning it black and merging it into the evolving artifact, preserving the immediacy of paper while keeping the modelâs contributions legible and integrated. Interaction is optimized for pen-and-touch on a tablet: the pen is used for drawing and erasing, two-finger gestures pan and zoom, and a one-finger double tap toggles eraser mode before erasing with the pen. 4.2 Semantic Tree Behind the UI, the probe maintains a semantic tree that helps the system better interpret the user drawings, as shown in Figure 2. The 4 Thinkink Figure 2: semantic tree of the user canvas; black=drawings, or- ange=concepts, indigo=requests, teal=generations with/with- out request; red stars mark the context selected to produce generation (9) from request (9). tree serves as an intermediate structure between raw ink and LLM prompting: rather than asking the model to reason over isolated strokes or a flat image alone, the system organizes the workspace into meaningful units and relations. Its Base Tree is produced using the prompt in Appendix B.1. Black nodes denote drawing nodes and orange nodes denote concept nodes. The treeâs Base Tree is produced using the prompt in Appendix B.1. Black nodes denote drawing nodes and orange nodes denote concept nodes. Drawing nodes represent individual pieces of ink, such as hand- written words, arrows, circles, or other sketch elements, while concept nodes represent higher-level groupings of related drawings or subclusters. Because users may work in several disconnected re- gions of the canvas, the representation allows multiple roots rather than forcing the entire workspace into a single hierarchy. On top of the Base Tree, the system adds an Insight Nodes Layer using the prompt in Appendix B.2. This layer introduces indigo request nodes, which mark places where the system identifies an opportunity to provide textual support, and teal generation-without- request nodes, which represent unsolicited model outputs (e.g., Socratic questions or suggestions for what to draw next). Finally, the system adds a Response Layer, which includes gener- ations produced in response to request nodes (i.e., generation-with- request nodes attached to their corresponding requests). To produce generation-with-request nodes, the system utizlize user request and the context related to the request as a prompt for node generation (see Appendix B.3). The request is gathered by analysis of drawings. The context is gathered from the requestâs local and hierarchical neighborhood, such as its parent, siblings, relevant ancestors, and related sibling branches. For example, Figure 2 shows dark red stars indicating the context selected for request node (9)âthe drawing and concept nodes from the tree used to ground its generation. This layered structure clarifies the distinction between the userâs drawings, the systemâs interpretation as requests for support, and the modelâs subsequent contributions to the canvas. The tree is built and updated through an analysis pipeline that runs whenever the user becomes idle. The system first sends the model a composed view of the canvas together with individual drawings and asks it to generate the Base Tree (Appendix B.1). It then prompts the model to augment that structure with insight nodes (Appendix B.2) and, when a request is present, to produce the corresponding Response Layer node using the prompt in Appen- dix B.3. The resulting text is placed near the relevant drawing, and if the user accepts it, it becomes part of the evolving workspace and is incorporated into subsequent analyses. In this way, the semantic tree allows the probe to treat ink not only as display content, but also as a structured input language for LLM insights. 5 Diagnostic Study To address RQ2: What human-LLM interaction challenges emerge when users engage with the technical probe? we conducted a diag- nostic study with the technical probe. Our goal was to identify key interaction challenges before building Thinkink, exploring four aspects of the design. Our findings revealed challenges in three aspects, while Ideation workflow showed no issues, indicating that participants were satisfied with the technical probeâs support in fluid ideation through evolving externalization workflows. 5.1 Method We recruited 6 participants from a university community (M=4, F =2; age 26â37), each from a different domain: quantum information and computation, HCI, computational arts/game design, fluid mechan- ics, reinforcement learning and AI ethics, and vision science. The cohort included one postdoctoral researcher, three PhD students, and two Masterâs students. Participants received $30 compensation. Overall, participants reported frequent use of pen and paper for thinking and ideation (M=3.67, SD=1.03 on a 5-point scale), infre- quent use of digital inking tools (M=2.50, SD=0.84), and high use of LLMs for thinking and ideation (M=4.67, SD=0.52). The study was conducted in person on a 13-inch iPad Pro using our technical probe. Each session lasted approximately 90 minutes. After consent, demographics, and pre-task questions, participants completed a short tutorial. They then completed two ideation tasks: one based on a question from the same set used in the formative study (see Appendix A.1) and one related to their own area of expertise, chosen by the participants themselves. For each task, participants spent 10 minutes to externalize what they knew, ex- plore and ideate with the LLM, and then reach the best conclusion they could. We collected think-aloud data, screen recordings, audio, interaction logs, and final artifacts (see Appendix C.3), followed by a semi-structured interview and an 11-item 5-point Likert question- naire. The full study materials are provided in Appendix C. 5 Payandeh et al. 5.2 Challenges We analyzed the qualitative data using the same method as in the formative study, employing the four high-level design aspects as analytical lenses. We further enriched our analysis with interaction logs, artifacts, questionnaire ratings, and semantic tree inspections. Participants highlighted several strengths of the probe that echoed our formative design guidelines. Following the âIdeation workflowâ guideline form the formative study, they valued the spa- tial canvas for maintaining a big-picture view of their evolving externalization: âwith the canvas, I zoom out and see what hap- penedâ [P6]. Following the âUI and Interaction Designâ guideline, they appreciated the minimal, no-UI design for staying out of the way of thinking: âvery minimal ... Iâm a fan of minimal stuffâ [P5]. Following the âLLM Integrationâ guideline, participants found that even unconfirmed Insights could still spark further exploration: âThe good thing about that is that when it offers, like, a suggestion, I donât need to take it as is....â [P2]. Following the âLLM Outputsâ guideline, participants also saw value in Socratic questioning that expands the ideation space, suggesting that â[i like that] every time I ask a question, it responds with, like, another question....â [P2]. The study also revealed challenges, reflected in participantsâ interviews and survey responses (Figure 4; Appendix E). UI and Interaction Design: Unclear Boundaries Between Note-Taking and LLM Modes, Causing Interaction Confusion. Although most participants appreciated the minimal interface, the lack of explicit cues also created confusion. Participants often could not tell when they were simply writing notes and when the system had started interpreting their writing as input for LLMs response: âWhenever I was writing something, I was thinking if the model is... considering... everythingâ [P1]. They were also sometimes unsure how to perform different actions, especially when interactions over- lapped: âwhen I tried to move the screen... it moved some part of that text that I wrote before....â [P6]. These difficulties led participants to ask for more explicit interface support, including visible controls âI would honestly prefer having a toolbar....â [P2], fewer and more clearly separated modes âhaving two different proactive and active sessions, that would make it easier....â [P4], and lightweight onboard- ing embedded in the interface: âfor any new application... they have kind of introduction... that would be more helpful...â [P6]. UI and Interaction Design: Limited Error Prevention, Recov- ery From Errors, and Control. Participants encountered a system that neither prevented common errors nor supported smooth recov- ery once errors occurred. Editing was brittle: âthereâs no razor, so itâs a little bit painful that you cannot immediately erase it....â [P1], âI feel it should have a control Z. A button...â [P4], âYeah, I need to have an option for undoing... the process....â [P6], and âbecause of 2sec merg- ing, the writing seperated to two text (which shouldnât)...â [P1]. Also, LLM Outputs were difficult to preserve or revisit, as indicated by âMaybe if that would have stayed a bit longer, so that I can choose to stay there....â [P4] and âwhen I press it, and then I continue writing, it disappears....â [P3]. Users also wanted more control over automation, noting premature intervention (âI was trying to actually write very fast... the AI (LLM) might start thinking about that part, and give me...â [P1] âI donât want the AI (LLM) to examine everything as soon as Iâm done....â [P4]), and opaque and uneditable context selection for output generation (âIt was not clear to me which part of the text my question, the AI (LLMs), is considering....â [P1] and âwhat if it caused... this mistake caused it to have the wrong prompt in future?...â [P4]). LLM Integration: Inability to Interact Directly with LLMs. Participants wanted an explicit way to prompt the LLMs in addition to receiving Insights, because autonomous feedback did not always match their immediate intent. Several emphasized the need to de- cide when the system should engage (âit is nice that it starts thinking when I am saying, hey, start thinking now....â [P1] and âI would prefer to write everything down... then... now I want you to examine this process...â [P4]), while others were frustrated when direct queries led to more questioning rather than answers (âI feel that itâs, like, keep asking other questions... itâs not giving any answer....â [P4], and âI wouldnât prefer questions, honestly, in this stage....â [P3]). LLM Outputs: Misalignment Between LLM Output Type- s/Formats and User Needs. LLM Outputs sometimes failed to match the representational needs of participantsâ work. Users ex- pected the system to respond visually when appropriate, e.g., âit would be really cool if it starts, like, sketching with me...â [P2], âif it could draw some diagrams for me, some shapes for me. it would be top-notch...â [P5], and âDo you want to draw a cube? I can draw a nice cu...â [P4]. This mismatch was especially evident in technical tasks, where plain-text responses were insufficient (âthe presenta- tion format of the AI (LLM) responses, if it is just text, is terribleâ [P1]) and where unreadable formatting and rigid typography further reduced usefulness, as reflected in âItâs not easy to read the latex format....â [P4] and âCan I adjust the size of the text, or no?â [P2]. 5.3 Improvement Areas From these challenges, we derive the following improvement areas to inform the refinement of the technical probe. UI and Interaction Design: Make System Modes Explicit and Constrain Interactions Within Each Mode. The interface should make writing, editing, selection, and LLM-based modes clearly distinct and easy to switch between, so users can under- stand what the system is attending to and what actions are currently possible. Limiting each mode to a small, well-defined set of inter- actions, supported by lightweight cues such as visible controls, mode indicators, and brief onboardingm, would reduce accidental actions, clarify system behavior, and improve learnability without sacrificing interface simplicity. UI and Interaction Design: Prioritize Recoverability and Con- trollable Automation. Hybrid inkâLLM systems should be de- signed around reversibility, persistence, and inspectability. Robust undo/redo, editable merge outcomes, persistent response history, explicit context scoping, and manual LLM triggering would help users prevent errors, recover from them quickly, and override au- tomation when it acts at the wrong time or with the wrong context. LLM Integration: Enable Both Explicit Prompting and In- sights. The system should distinguish between on-demand prompt- ing and Insights as two explicit but interoperable modes. Allowing users to manually invoke the LLM, while separately controlling Insights, would preserve user agency and reduce interruptions without sacrificing serendipitous assistance. 6 Thinkink LLM Outputs: Align Content and Types with Users Demands. LLM responses should adapt in both content and presentation. Beyond text, outputs can include proofreading and critique-oriented feedback, ink-like sketches, rendered mathematics, Markdown, and context-aware typography, ensuring they remain clear and useful across technical and presentation-oriented workflows. 6 Thinkink 6.1 Overview Thinkink is the refined version of our technical probe, redesigned in response to the diagnostic study (Figure 1; see Appendix D.5 for all screenshots). The main changes are: adding stronger mechanisms for control and recovery such as confirmation, history, and man- ual context adjustment, separating explicit LLM prompting from Insights, expanding the types and formats of LLM Outputs, and making system modes explicit through a state machine interaction design [6,39] (overview in Figure 3; detailed version in Appen- dix D.5). In contrast to the probe, Thinkink does not rely on loosely inferred LLMs interaction alone; instead, it structures interaction into clear substates that make available actions, transitions, and LLM behaviour more legible. Thinkink retains the semantic tree as its backbone, but builds it on demand rather than regenerating the entire structure after every canvas change, as in the technical probe. Considering the state machine design shown in Figure 3, entering âPromptâ or âInsightsâ from âDrawingsâ, âPromptâ, or âInsightsâ regenerates only the base tree. Entering âInsightsâ from âDrawingsâ or âPromptâ adds the Insight Nodes Layer on top of the base tree. Entering âIterateâ adds the Response Layer, but only includes a generation with request node for the active request: if reached from âInsightsâ, the request is the selected request node from the Insight Nodes Layer; if reached from âPromptâ, it is the explicit request drawing, attached as a request node to the relevant drawing. Constructing the semantic tree incrementally reduces the number of LLM calls and the input tokens per call, thereby decreasing user wait times. 6.2 User Interface Thinkink incorporates mechanisms for user control and recov- ery during interaction with LLMs. Users can compose explicit on- canvas requests, browse Insights, and manually adjust selected drawings by the model as context used for response generation. LLM Outputs are no longer treated as only temporary overlays: they can be inspected, browsed as alternatives, accepted into the workspace, and revisited through history. A history bar supports both revision and exploration by enabling users to move backward and forward through edits, revisit earlier LLM Outputs, and trig- ger regeneration of new alternative responses or Insights. During regeneration, previous outputs are included in the prompt so the model can produce different alternatives rather than repeating prior ones, which better supports ideation. Thinkink retains the probeâs pen-first infinite canvas and mini- mal visual style, but adds a set of lightweight UI elements to make interactions with LLMs more explicit and usable. At the top left, a persistent set of buttons supports navigation across substates, letting users move between note-taking, explicit LLM prompting, Insights, and back or confirm/cancel flows when needed. At the Figure 3: State Machine Interaction Design: Rectangles rep- resent substates grouped into two superstates: âNote-takingâ (âDrawings,â âEditâ) and âLLM assistantâ (âPrompt,â âIn- sights,â âIterate,â âInspectâ). Double-headed arrows show bidi- rectional transitions; single-headed arrows, unidirectional. Circular loops denote self-transitions (within-substate inter- actions). Edge colors encode transition type: black for tran- sitions within âNote-taking,â indigo for transitions within âLLM assistant,â and cyan for cross-superstate transitions. bottom right, a second set of controls provides the actions avail- able within the current substate. This separation was designed to help users mentally distinguish transitions between substates from actions within a substate, while also supporting bimanual use: right- handed users can use the left hand for the top-left controls and the right hand with the pen for the bottom-right controls, while left-handed use reverses this arrangement. Thinkink expands the types and formats of LLM Outputs. In addition to generation without request for Insights such as Socratic questions, it adds proofreading-oriented Insights; and in addition to generation with request for text-generation requests, it also supports image-generation requests, such as addressing usersâ di- rect image-generation requests or suggesting what to draw next. Thinkink also improves how these outputs are presented by sup- porting better-formatted text, including mathematical notation, so they are more useful in technical ideation tasks. In addition, both selected drawings and LLM generations can be revised through normal edits or LLM edits by annotation over text or image. 6.3 Interaction Design using State Machine Thinkink organizes interactions as a substate machine. This design choice is motivated by the first two UI and interaction challenges identified in our diagnostic study (i.e.unclear boundaries between note-taking and LLM assistant, and limited support for error pre- vention, recovery, and user control) by making modes explicit, restricting each mode to a clear, non-overlapping set of interac- tions, and defining clear transitions for easier recovery and control. The state machine includes two superstates: Note-taking and LLM assistant, and six substates: Drawings and Edit under Note-taking, and Prompt, Insights, Iterate, and Inspect under LLM assistant. Fol- lowing Figure 3, we refer to between-substate transitions using labels such as Prompt->Iterate, and to self-transitions as actions performed while in a given substate. In Note-taking superstate, Drawings is the default substate for ordinary canvas work. While in Drawings, users can move existing drawings through âdragging a drawing.â Starting a new drawing with the drawing pen or selecting an existing drawing through 7 Payandeh et al. âtap-and-hold on a drawingâ triggers Drawings->Edit, making that drawing the current focus of interaction. While in Edit, users can edit the selected drawing in two ways. First, they can perform normal editing using the drawing pen and the lasso eraser. Second, they can perform LLM editing using the LLM annotation pen and LLM annotation eraser to annotate the drawing and invoke LLM-supported revision of that note. In both editing modes, users can use the history bar to undo or redo revi- sions. Users can enter Edit through Drawings->Edit or Prompt->Edit. When they leave Edit, they transit to the previous substate. In LLM assistant superstate, Prompt supports explicit prompting. While in A, the drawings on the canvas are treated as available context. Users can start an explicit request with the LLM prompting pen, which triggers Prompt->Iterate. Users can also move existing drawings through âdragging a drawingâ or open a drawing in Edit through âtap-and-hold on a drawing,â which triggers Prompt->Edit. This makes explicit prompting a distinct interaction path rather than something inferred implicitly from general writing. Insights enables working with the Insight Nodes Layer. While in Insights, users inspect suggestions generated from the existing canvas without first authoring a request. Selecting a request node through âtap-and-hold on a requestâ triggers Insights->Iterate, while selecting a generation-without-request insight node through âtap on a generationâ triggers Insights->Inspect. Users can also use the history bar to move across Insights sets and regenerate a new set; during regeneration, previous Insights are included in the prompt so the model is encouraged to produce different suggestions. Leav- ing this substate with the top-left navigation buttons returns users through Insights->Prompt, preserving the distinction between In- sights browsing and explicit prompting. Iterate is the main substate for refining an explicit LLM request and producing different generations from it. While in AR, users can continue writing and editing their requests using the LLM prompt- ing pen, reposition requests via dragging, manually select which drawings serve as context using the lasso-based tool, and switch between generation types, such as text or image, which are auto- matically identified by the model. As they refine the request or its context, new generations can be shown as responses. Users can also browse alternative generated responses while staying in AR using the history bar, which supports both revisiting prior alternatives and regenerating new ones. When regenerating, previous gener- ations are included in the prompt so the model can produce new responses rather than repeat earlier ones. Selecting one of those generations through âtap on a generationâ triggers Iterate->Inspect. Leaving this substate with the top-left navigation buttons returns users through Iterate->Prompt or Iterate->Insights, depending on whether the request was reached from Prompt or Insights. Inspect is the inspection and decision substate for one LLM Output. While in AG, users can inspect the selected generation, edit it with the LLM annotation pen and LLM annotation eraser, move it on the canvas through âdragging the generation,â and use the history bar to undo or redo revisions. Back-navigation through the top-left buttons returns users through Inspect->Iterate when the generation was reached from AR, or through Inspect->Insights when it was reached directly from Insights. Confirming a generation is done by tapping the top-left confirm button, which accepts it into the workspace, turns it into a drawing, and returns to the previous Figure 4: Average of participants ratings different prototypes active substate through Inspect->Prompt or Inspect->Insights. When Inspect->Insights occurs, the system preserves the previously shown Insights. This allows users to work through one insight at a time, finalize it, and then return to continue with the remaining insights. 7 Usage Study To address RQ3: How do users utilize Thinkink in practice? we con- ducted a usage study to understand how people use Thinkink for ideation. Our goal was to understand common usage patterns of such as tool in real-world scenarios. Additionally, we highlight how changes in Thinkink, compared to the technical probe, address challenges identified in the diagnostic study, while also revealing new challenges encountered by participants. 7.1 Method The method followed the diagnostic study, with two modifications: the interview questions were revised to focus on usage patterns rather than system diagnosis (see Appendix D.1), and additional sur- vey questions were added to assess whether the challenges identi- fied in the diagnostic study with the technical probe were addressed by Thinkink (see Appendix E). Regarding participants, we invited five participants back from the diagnostic study (P1âP5; M=3, F =2; age 26â37) and recruited five new participants (P7âP11; M=2, F =3; age 22â42), with a variety in major: quantum information and com- putation, HCI, computational arts/game design, fluid mechanics, reinforcement learning and AI ethics, vision science, statistics, en- trepreneurship, health and aging, and biotechnology. The cohort included 3 undergraduate students, 2 Masterâs students, 4 PhD stu- dents, and 1 postdoctoral researcher. Overall, participants reported moderate use of pen and paper for thinking and ideation (M=3.50, SD=0.97 on a 5-point scale), low use of digital inking tools (M=2.60, SD=0.70), and moderate to frequent use of LLMs for thinking and ideation (M=3.90, SD=0.74). The full study materials are provided in Appendix D, including examples of final artifacts (Appendix D.3) and pictures captured during task performance (Appendix D.5). 7.2 Improvements and New Challenges The changes introduced in ThinkInk helped participants overcome several of the challenges identified in the diagnostic study, as re- flected in interviews and survey responses (Figure 4; Appendix E). Participants also highlighted further improvements. Regarding UI and Interaction Design, ThinkInk improved the previously unclear boundary between note-taking and LLM inter- action by making modes more legible through in-situ visual cues. For example, P7 reported that distinguishing modes was ânot really 8 Thinkink hardâ [P7] because âthis ink color appears on the cornerâ [P7], suggest- ing that the interface better signaled when participants were writing notes versus engaging the model. ThinkInk also improved usersâ ability to recover from errors and maintain control during interac- tion. After the model misunderstood part of a technical question, P1 was able to revise the prompt directly on the canvas rather than restart the task, explaining, âI erased it, and I made it simpler.â [P1] This indicates that the editable canvas supported lightweight error recovery and iterative refinement. A new issue, however, was that some participants felt a steep learning curve around modes and controls, with some users describing the experience as âa little bit confusedâ and warning that, without onboarding, they âmight be confused in the first placeâ: âI was a little bit confusedâ [P1]; âif you didnât give me instructions, I might be confused at the firstâ [P9]. Regarding LLM Integration, ThinkInk addressed the earlier inability to interact directly with LLMs by allowing participants to invoke the model when they were ready, instead of relying only on automatic suggestions. P5 highlighted this increased sense of agency: âSo sometimes I knew what I wanted, so I right away asked it.â [P5] This suggests that ThinkInk restored more direct, user- initiated interaction with the LLM while keeping the exchange embedded in the shared note-taking workspace. Compared with conventional chat interfaces, this made LLM use feel more tightly coupled with participantsâ ongoing work on the canvas. A new area for improvement was that participants wanted their prompts (requests) to persist on the canvas so they could feel a sense of ownership and review their finalizing, rather than having requests disappear after generation: âthe request disappearsâ [P2]; âI want to keep track of my questionsâ [P1]. Finally, regarding LLM Outputs, ThinkInk partially addressed the earlier misalignment between LLM output formats and usersâ needs by situating model responses within a canvas that also sup- ported sketching and diagrammatic work, rather than constraining interaction to text-only chat. P1 explicitly contrasted ThinkInk with ChatGPT, noting that âThe picture is much easier to draw, and it is impossible to do that in ChatGPT.â [P1] This suggests that ThinkInk better aligned LLM-supported work with the visual and diagram- matic nature of participantsâ tasks. At the same time, output re- mained uneven: answers were sometimes too short, generic, or hallucinated âthey were so short, sometimes they were very unrelated to what I askedâ [P9] âit starts hallucinatingâ [P4]; and the model sometimes failed to follow instructions or produce accurate outputs in terms of the content âit generated me... a flow chart [instead of what the participant wanted]â [P3]. Participants also sometimes felt that the modelâs default context selection did not match their intent, leading to inaccurate responses. In such cases, they had to manually adjust the contexts themselves: âyou have to go and deselect all the onesâ [P4]; âit would not give me the response I wanted... so I had to undo (unselect) those contextsâ [P9]. 7.3 Usage Patterns We identified the following usage patters across the four design aspects. As the data collected mirrored the diagnostic study, we used the same analysis process. Ideation Workflow: Note-taking paired with LLM assistant Supports Non-linear Iterative Ideation. Participants often de- scribed ThinkInk as âone more partnerâ [P3] in ideation. They used it for ideation through a spatial, non-linear workflow. They contrasted the 2D canvas with linear chat because it âgives you more free- domâ [P1], and P3 described serendipitous exploration [28] through keeping multiple branches visible: âA is good, B is good, and C is good... I keep all of them... B and C, I will reach out to them later.â [P3]. We found that Thinkink can be used flexibly for varied by task (see Appendix D.6 for usage patterns). For example, in a design creativity task, P2 generated alternative sketches and iterated on them, âIâm just sketching something... to see what kind of things AI (LLM) can do.â [P2], while also noting that âas artists, we like to keep the history of the different iterations.â [P2]; P7 used the canvas to ideate plots as a branching map, âstarting from the center... it leads to the next state of thinking.â [P7]. Table D.2 lists participantsâ chosen tasks, and Appendix E shows they found Thinkink highly support ideation, when performing their tasks. Note-taking also functioned as prompt grounding: P1 added written or drawn context because âit would be hard for this model to understand, thatâs why I was pro- viding contextâ [P1], and P2, when Insights were insufficient, turned back to Note-taking to supply more context and âhighlight the part thatâs relevant, and then work from there.â [P2]. UI and Interaction Design: Cognitively Separating Note- Taking from the LLM assistant and Minimal UI Improves Usability. Most participants described the interface as simple and minimal: âvery simple... one person can understand and useâ [P8], while giving them the flexibility in using the features however they workflow requires (see Figure D.6, including navigation and us- age patterns). They used the state machine interaction design as a understandable and memorable structure for available actions, although not all of them enacted the intended bimanual workflow of using the top-left controls with the left hand and in-substate interactions with the right. Some carried over familiar one-handed sketching habits (âwhen I sketch or create something in real life, Iâm always just using my right hand... My left hand doesnât really do anythingâ [P2]) while most still referred to the leftâright division as shaping their understanding of the interface because it âhelps... build my mental model of what features does whatâ [P2] and made it âvery helpful to have different icons... be on the other sideâ [P9]. Relat- edly, some participants who used the right hand for all interactions described the hand movement from right-side to left-side buttons as creating a brief pause for thinking: âthat time kind of works re- ally well for when my right hand... moves thereâ [P2]. Participants also used lightweight UI elements in ways that reduced interaction overhead: output-type, including text and image, selection reduced prompt-writing burden (âI donât need to always specify my request... generate me an imageâ [P3]), and iconography helped them distin- guish LLM functions from ordinary note-taking (âeasy to identify... the AI (LLM) versus normal note tweakingâ [P11]). LLM Integration: While Insights Used for Discovery, Ex- plicit Prompts used for Follow-Up. Participants used ThinkInkâs Prompt mode in conjunction with note-making and steering. No- tably, P8 used Prompt multimodally, effectively turning written text and even shapes into prompts, while emphasizing that successful interpretation depended on grounding the model in the current 9 Payandeh et al. context, âSo thatâs the reason I was making sure that the context is rightâ [P8]. They also used Insights to surface overlooked ideas and context-aware ideation: âit was giving me questions that I didnât know about.â [P10]; âknows, like, what all the stuff that, I had written. Based on that, it was providing me things that I could, like, askâ [P10]. P8 also used Insights as an initial probe, but when those suggestions surfaced a different avenue, they returned to Prompt to re-steer, again foregrounding context management: âSo thatâs the reason I was making sure that the context is rightâ [P8]. They sometimes confirmed one insight and switched to the Prompt mode to explore it further: âI would pick one of them that I like and continue with the AI querying mode to ask more questions about that insightâ [P9]. LLM Outputs: Context Management is Necessary, But it En- ables Shaping, Comparing, and Refining LLM responses. Par- ticipants managed LLM Outputs on the spatial canvas differently from chat, noting that responses remained traceable rather than âall linearâ âitâs all linear.â [P3]. Across Prompt and Insights, they refined outputs through regeneration and history, using iteration to recover from wrong answers or move the model toward a new gen- eration: âwhen you regenerate, the new one is correct.â [P1]; âtry again ... check if thereâs new things.â [P7]. Context selection and manipula- tion were used to test different assumptions, specially in a quantum computing task done by P1 or a design creativity task by P2. Also, it was used to adjust relevance and make the systemâs focus more explicit by revealing the context that is used to generate response âitâs pointing out the focus (context)â [P7], allowing users to remove distracting material âcan you just deselect these contexts?â [P1]. âLLM editing was used mainly during convergenceâ, when users knew what to change âthe LLM edit feature work better.â [P2] and annotation afforded control âgiving a good control of the context.â [P7]. 8 Discussion Ink-native interaction with LLMs could enable building new tools across domains such as mathematics, physics, and graphic design, as well as in scenarios including ideation, collaboration, and AR/VR. This paper represents a stepping stone toward realizing ink-native interactions with LLMs, an area that remains largely underexplored. Although our studies focused on a specific scenario and domains, and involved a limited number of participants, they provide useful insights for future research. Across our formative, diagnostic, and usage studies, we demonstrate how to build a foundation for trans- forming unstructured drawings on a 2D spatial canvas into struc- tured representations that is more understandable for LLMs. Our findings further suggest that designing ink-native LLM-powered tools for ideation is not a matter of identifying a single optimal interface, but rather of balancing competing design goals across different users and tasks, focusing on different design aspects. Ideation Workflow: Linear vs. Non-linear Progression. The formative study showed that externalization is non-linear and often begins with incomplete marks, words, or structures. The diagnostic and usage studies then showed why this matters: participants used the 2D canvas to branch, keep alternatives visible, and ground prompts in drawings. However, not all users benefited from the same degree of openness. Some valued free-form exploration, while others wanted clearer structure. This suggests a key design choice for future systems: whether to privilege open spatial exploration or provide more linear scaffolds, summaries, or templates. The right balance will likely differ for different tasks, such as design creativity, technical reasoning, and everyday planning, as well as for users who naturally work in organized versus free-form ways. UI and Interaction Design: Constrained vs. Unconstrained Interactions. The studies consistently supported the value of a minimal interface design. Participants appreciated low visual clut- ter and pen-first interaction, which aligned with the immediacy of paper. The diagnostic study also showed that minimalism can become ambiguity when modes, system attention, and available actions are unclear. Thinkinkâs state-machine design improved leg- ibility and control, but the usage study still revealed a learning curve. This requires a design decision: preserving a low-friction interface versus adding explicit constraints, cues, and onboarding. More constrained interactions may especially benefit users with less experience using digital inking tools, while expert users may prefer fewer guardrails and more interaction choices. Adaptive dis- closure may therefore be promising: simple defaults for novices, with more flexible interactions revealed as users gain confidence. LLM Integration: User vs. LLM Agency. Participants across studies wanted LLMs to act as an ideation partner embedded in the canvas, not as a separate chatbot. They valued Insights that expanded thinking, but also wanted ways to explicitly ask the model, control when it acted, and inspect or adjust context. Together, these findings suggest that Insights and Prompt should be treated as separate but interoperable design dimensions rather than a single continuum. Different tasks may require varying levels of LLMs initiative: tasks where users are uncertain about their goals, such as everyday decision-making, can benefit from more Insights, whereas tasks where users have clearer objectives, such as specialized work like math problem-solving, may require more explicit prompting. LLM Outputs: Generic vs. Task-specific Content and Repre- sentations. Exploratory questions were useful for broad ideation, but generic text-only outputs were insufficient for technical and creative work. Participants wanted diagrams, better formatting, mathematical notation, and editable visuals. Thus, future ink-native LLM-powered tools should decide whether to optimize for general- purpose support or for domain-tuned outputs matched to target tasks such as mathematics, design creativity, or scientific reasoning. 9 Conclusion Handwritten and sketch-based ideation helps people externalize and explore ideas, yet current LLM interfaces remain largely chat- based and poorly suited to spatial ink workflows. We presented Thinkink, a LLM-assisted ideation tool that supports ink-based prompting and on-canvas LLM responses, including a semantic- tree representation of the workspace and an explicit state-machine interaction design. Through a user-centered, three-stage process, a formative focus group (í=12), a diagnostic study of a technical probe (í=6), and a usage study of Thinkink (í=10), we identified and refined four core design aspects: âIdeation Workflow,â âUI and Interaction Design,â âLLM Integration,â and âLLM Outputs,â. Our results suggest that future ink-native LLM-assisted tools should treat these aspects as balances to tune for target users and tasks. It 10 Thinkink should explore adaptive interfaces, richer multimodal outputs, and domain-specific versions for settings such as math, design creativity, and everyday decision making. References [1]Christopher Andrews, Alex Endert, and Chris North. 2010. Space to think: large high-resolution displays for sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI â10). Association for Computing Machinery, New York, NY, USA, 55â64. doi:10.1145/1753326.1753336 [2]Rosanna L. Breen. 2006. A Practical Guide to Focus-Group Research. Jour- nal of Geography in Higher Education 30, 3 (2006), 463â475.doi:10.1080/ 03098260600927575 [3] Donald O Case and Lisa M Given. 2016. Looking for information: A survey of research on information seeking, needs, and behavior. (2016). [4]Hung-Fu Chang and Tong Li. 2025. A framework for collaborating a Large Language Model tool in brainstorming for triggering creative thoughts. Thinking Skills and Creativity 56 (2025), 101755. doi:10.1016/j.tsc.2025.101755 [5]Michelene T.H. Chi, Nicholas De Leeuw, Mei-Hung Chiu, and Christian La- vancher. 1994. Eliciting self-explanations improves understanding. Cognitive Science 18, 3 (1994), 439â477. doi:10.1016/0364-0213(94)90016-7 [6]Paulo Pinheiro da Silva, Norman W Paton, Rick Kazman, Len Bass, and Jan Bosch. 2003. Improving UML Support for User Interface Design: A Metric Assessment of UMLi. In Proceedings of ICSE 2003 Workshop on Bridging the Gaps Between Software Engineering and Human-Computer Interaction, May 3-4, 2003, Portland, Oregon, USA. IFIP, 76â83. [7]Julian De Freitas, Gideon Nave, and Stefano Puntoni. 2025. Ideation with Generative AIâin Consumer Research and Beyond. Journal of Consumer Research 52, 1 (05 2025), 18â31. arXiv:https://academic.oup.com/jcr/article- pdf/52/1/18/63186652/ucaf012.pdf doi:10.1093/jcr/ucaf012 [8] Excalidraw Contributors. 2024. Excalidraw: Virtual whiteboard for sketching hand-drawn like diagrams. https://excalidraw.com/ Accessed: 2024-03-21. [9] Anastasiia Fadeeva, Vincent Coriou, Diego Antognini, Claudiu Musat, and Andrii Maksai. 2025. InkFM: A Foundational Model for Full-Page Online Handwritten Note Understanding. arXiv:2503.23081 [cs.CV] https://arxiv.org/abs/2503.23081 [10]Figma, Inc. 2024. Figma: The Collaborative Interface Design Tool. https://w. figma.com/ Accessed: 2024-03-21. [11] Google. 2026. Gemini API Documentation. https://ai.google.dev/gemini-api/docs. Official documentation for Gemini models including Gemini 3 Flash Preview. Accessed: 2026-03-22. [12]ARTHUR C. GRAESSER, WILLIAM BAGGETT, and KENT WILLIAMS. 1996. Question-driven Explanatory Reasoning. Applied Cognitive Psychology 10, 7 (1996), 17â31. doi:10.1002/(SICI)1099-0720(199611)10:7<17::AID-ACP435>3.0. CO;2-7 [13]Sophia Gu. 2023. LLMs as Potential Brainstorming Partners for Math and Science Problems. arXiv:2310.10677 [cs.CL] https://arxiv.org/abs/2310.10677 [14] Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen + touch = new tools. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST â10). Association for Computing Machinery, New York, NY, USA, 27â36. doi:10.1145/1866029.1866036 [15]Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman, and Desney Tan. 2007. InkSeine: In Situ search for active note taking. In Proceedings of the SIGCHI Conference on Human Factors in Comput- ing Systems (San Jose, California, USA) (CHI â07). Association for Computing Machinery, New York, NY, USA, 251â260. doi:10.1145/1240624.1240666 [16]Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. 2024. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. Advances in Neural Information Processing Systems 37 (2024), 139348â139379. doi:10.52202/079017-4423 [17]Zeyuan Huang, Cangjun Gao, Yaxian Shan, Haoxiang Hu, Qingkun Li, Xiaoming Deng, Cuixia Ma, Yu-Kun Lai, Yong-Jin Liu, Feng Tian, Guozhong Dai, and Hongan Wang. 2025. SketchGPT: A Sketch-based Multimodal Interface for Application-Agnostic LLM Interaction. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST â25). Association for Computing Machinery, New York, NY, USA, Article 157, 18 pages. doi:10.1145/ 3746059.3747598 [18] Edwin Hutchins. 1995. Cognition in the Wild. MIT Press. [19]JGraph Ltd. 2024. draw.io: Free online diagram software. https://app.diagrams. net/ Accessed: 2024-03-21. [20] Helene Joffe. 2011. Thematic Analysis. John Wiley & Sons, Ltd, Chapter 15, 209â 223. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781119973249.ch15 doi:10.1002/9781119973249.ch15 [21] David Kirsh. 1995. The Intelligent Use of Space. Artificial Intelligence 73, 1â2 (1995), 31â68. doi:10.1016/0004-3702(94)00017-U [22]Arthur Koestler. 1964. The Act of Creation. Macmillan, Oxford, England. 751 pages. APA PsycInfo record: 1965-07821-000. [23]Karina Korovai, Dmytro Zhelezniakov, Oleg Yakovchuk, Olga Radyvonenko, Nataliya Sakhnenko, and Ivan Deriuga. 2024. Handwriting Enhancement: Recognition-Based and Recognition-Independent Approaches for On-Device Online Handwritten Text Alignment. IEEE Access 12 (2024), 99334â99348. doi:10.1109/ACCESS.2024.3412433 [24]Stephen C.-Y. Lu and Ang Liu. 2012. Abductive reasoning for design synthesis. CIRP Annals 61, 1 (2012), 143â146. doi:10.1016/j.cirp.2012.03.062 [25]Lorenzo Magnani. 2007. Logic and Abduction: Cognitive Externalizations in Demonstrative Environments. 275â284 pages. https://philsci-archive.pitt.edu/ 10414/ ISSN: 0495-4548 (print). [26] Nicolai Marquardt, Asta Roseway, Hugo Romat, Payod Panda, Michel Pahud, Gonzalo Ramos, Steven M. Drucker, Andrew D. Wilson, Ken Hinckley, and Nathalie Riche. 2025. ImaginationVellum: Generative-AI Ideation Canvas with Spatial Prompts, Generative Strokes, and Ideation History. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST â25). Association for Computing Machinery, New York, NY, USA, Article 159, 19 pages. doi:10.1145/3746059.3747631 [27]Microsoft. 2026. TypeScript. https://w.typescriptlang.org/. Official documen- tation. Accessed: 2026-03-22. [28] Saeed Mirvahedi and Sussie Morrish. 2017. The role of serendipity in op- portunity exploration. Journal of Research in Marketing and Entrepreneur- ship 19, 2 (10 2017), 182â200. arXiv:https://w.emerald.com/jrme/article- pdf/19/2/182/1831465/jrme-10-2017-0045.pdf doi:10.1108/JRME-10-2017-0045 [29] OpenAI. 2024. ChatGPT. https://openai.com/chatgpt/ [30] OpenJS Foundation and Express Contributors. 2026. Express - Fast, unopinion- ated, minimalist web framework for Node.js. https://expressjs.com/. Official documentation. Accessed: 2026-03-22. [31] Maria Teresa Parreira, Sarah Gillet, and Iolanda Leite. 2023. Robot Duck De- bugging: Can Attentive Listening Improve Problem Solving?. In Proceedings of the 25th International Conference on Multimodal Interaction (Paris, France) (ICMI â23). Association for Computing Machinery, New York, NY, USA, 527â536. doi:10.1145/3577190.3614160 [32] React Contributors. 2026. React. https://react.dev/. Official documentation. Accessed: 2026-03-22. [33] Hugo Romat, Nathalie Henry Riche, Ken Hinckley, Bongshin Lee, Caroline Ap- pert, Emmanuel Pietriga, and Christopher Collins. 2019. ActiveInk: (Th)Inking with Data. In Proceedings of the 2019 CHI Conference on Human Factors in Com- puting Systems (Glasgow, Scotland Uk) (CHI â19). Association for Computing Machinery, New York, NY, USA, 1â13. doi:10.1145/3290605.3300272 [34]Orit Shaer, Angelora Cooper, Osnat Mokryn, Andrew L Kun, and Hagit Ben Shoshan. 2024. AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI â24). Association for Computing Ma- chinery, New York, NY, USA, Article 1050, 17 pages. doi:10.1145/3613904.3642414 [35]Lei Shi, Rachel Campbell, Peggy Chi, Maria Cirimele, Mike Cleron, Kirsten Climer, Chelsey Q Fleming, Ashwin Ganti, Philippe Gervais, Pedro Gonnet, Tayeb A Karim, Andrii Maksai, Chris Melancon, Rob Mickle, Claudiu Musat, Palash Nandy, Xiaoyu Iris Qu, David Robishaw, Angad Singh, and Mathangi Venkatesan. 2024. Inkeraction: An Interaction Modality Powered by Ink Recognition and Synthesis. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI â24). Association for Computing Machinery, New York, NY, USA, Article 181, 26 pages. doi:10.1145/3613904.3642498 [36]Lev Tankelevitch, Elena L. Glassman, Jessica He, Majeed Kazemitabaar, Aniket Kittur, Mina Lee, Srishti Palani, Advait Sarkar, Gonzalo Ramos, Yvonne Rogers, and Hari Subramonyam. 2025. Tools for Thought: Research and Design for Understanding, Protecting, and Augmenting Human Cognition with Generative AI. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA â25). Association for Computing Machinery, New York, NY, USA, Article 804, 8 pages. doi:10.1145/3706599.3706745 [37]Paul Thagard and Cameron Shelley. 1997. Abductive Reasoning: Logic, Visual Thinking, and Coherence. Springer Netherlands, Dordrecht, 413â427. doi:10. 1007/978-94-017-0487-8_22 [38]Yael Vinker, Tamar Rott Shaham, Kristine Zheng, Alex Zhao, Judith E Fan, and Antonio Torralba. 2025. SketchAgent: Language-Driven Sequential Sketch Gen- eration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 23355â23368. [39] Visual Paradigm. 2024. Visual Paradigm: Software Design and Modeling Tool. https://w.visual-paradigm.com/ Accessed: 2024-03-21. [40]Vite Contributors. 2026. Vite Guide. https://vite.dev/guide/. Official documenta- tion. Accessed: 2026-03-22. [41] Douglas Walton. 2014. Abductive reasoning. University of Alabama Press. [42] Haijun Xia, Ken Hinckley, Michel Pahud, Xiao Tu, and Bill Buxton. 2017. Writ- Large: Ink Unleashed by Unified Scope, Action, & Zoom. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI â17). Association for Computing Machinery, New York, NY, USA, 3227â3240. doi:10.1145/3025453.3025664 11 Payandeh et al. [43]Zheer Xu, Shanqing Cai, Mukund Varma T, Subhashini Venugopalan, and Shumin Zhai. 2024. SkipWriter: LLM-Powered Abbreviated Writing on Tablets. In Proceed- ings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST â24). Association for Computing Machinery, New York, NY, USA, Article 22, 13 pages. doi:10.1145/3654777.3676423 [44]Ryan Yen, Jian Zhao, and Daniel Vogel. 2025. Code Shaping: Iterative Code Editing with Free-form AI-Interpreted Sketching. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI â25). Association for Computing Machinery, New York, NY, USA, Article 872, 17 pages. doi:10.1145/ 3706598.3713822 [45]Rebecca Zheng, Marina FernĂĄndez Camporro, Hugo Romat, Nathalie Henry Riche, Benjamin Bach, Fanny Chevalier, Ken Hinckley, and Nicolai Mar- quardt. 2021. Sketchnote Components, Design Space Dimensions, and Strategies for Effective Visual Note Taking. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI â21). Asso- ciation for Computing Machinery, New York, NY, USA, Article 466, 15 pages. doi:10.1145/3411764.3445508 12 Thinkink A Formative Study A.1 Question Categories and Examples CategoryDescription and Example Brainstorming QuestionsEncourage creative thinking, idea generation, and innovation by exploring a wide range of possibilities. Example: âIâm designing a system for a research paper. What would be a suitable name for it?â Personal QuestionsAsk about an individualâs experiences, opinions, or feelings. Example: âWhat motivates me to start my day?â Open-Ended QuestionsEncourage detailed responses and cannot be answered with a simple âyesâ or âno.â Example: âHow do technological advancements shape the way we communicate?â Thought-Provoking QuestionsStimulate deeper reflection and contemplation, often challenging assumptions. Example: âWhat does success mean?â Cause-and-Effect QuestionsExamine relationships between actions or events and their outcomes. Example: âHow does employee engagement influence productivity?â Comparative QuestionsAsk for comparisons between two or more ideas or experiences. Example: âWhich is more fulfilling: working independently or in a team, and why?â Exploratory QuestionsInvestigate new or unknown areas to gain understanding. Example: âWhat opportunities could emerge from this new technology?â Planning QuestionsFocus on creating a roadmap or strategy for future actions. Example: âHow would you structure a six-month plan to achieve this goal?â Problem-Solving QuestionsAim to identify and resolve issues or challenges. Example: âHow can we fix issues in the current system?â Decision-Making QuestionsInvolve choosing between options or prioritizing actions. Example: âIf you had to focus on one project this quarter, which would it be and why?â Critical Thinking QuestionsRequire analyzing and evaluating information to form judgments. Example: âWhat factors contribute to procrastination?â Career QuestionsFocus on professional development and career paths. Example: âDid I choose the right career path?â Table A.1: Question categories used in the formative study. A.2 Participants Choices for the tasks Participant IDTask TypeMediumSelected Question 1047IndividualPen & PaperWhy do plans get delayed, and what factors cause procrastination? 1153IndividualPen & PaperDid I choose the right career path? 1253IndividualPen & Paper- 1300IndividualPen & PaperHow would I structure a plan to get a faculty job immediately after I graduate? 1473IndividualPen & PaperWhat motivates me to purchase online courses? 1540IndividualDigital Inking- 1571IndividualFigmaHow to plan my project to meet the next deadline? 1766IndividualPen & PaperHow do I add a notification feature to my current system? 1820IndividualDigital InkingHow could personal habits/behaviors gradually change a system or inter- face to better serve users over time? 1842IndividualPen & PaperIf you had to focus on only one project this quarter, which would it be and why? 1902IndividualPen & PaperWhat makes a good presentation? âIndividualPen & Paperâ âCollaborativePen & PaperHow to not be overwhelmed? âCollaborativePen & Paperâ âCollaborativePen & Paperâ Table A.2: Participantsâ selected tasks 13 Payandeh et al. A.3 Discussion Points â˘How did the process of externalizing evidence influence your understanding and the quality of your answer to the question? Please explain why it had an impact, or why it did not. â˘Did visualizing the evidence lead you to discover any new connections or insights that you hadnât previously considered? If so, please elaborate. â˘In what other types of questions or scenarios do you think externalizing evidence would be beneficial? Conversely, in what situations might it not be helpful? ⢠How easy or difficult was it for you to externalize the evidence, and what factors influenced that experience? ⢠Why do you think the tool you used is or isnât suitable for this task? ⢠In what aspects did you collaborate, and what challenges did you encounter (if any)? ⢠Why did you choose or not choose to use ChatGPT for this task? if you chose, for what? What are the scenarios that ChatGPT might help? A.4 Artifacts Figure A.1: Examples of artifacts made by participants during the formative study B Technical Probe B.1 Prompt for Base Semantic Tree Generation System instruction for generating new code (promptType: gen_code) Analyze this hand-drawn thinking canvas and build its base semantic tree. Build the semantic tree for this hand-drawn canvas. The tree must contain ONLY "drawing" and "concept" nodes. No other node types are allowed. For each drawing, analyze and include in the drawing node: - drawingType (text/shape) - text: extracted text content (if drawingType is text) - desc: description considering canvas context without mentioning other drawings (if drawingType is shape; maximum 5 words); as it is hand-drawn, donât invent overly specific details. If the shape contains text, include the text as part of the description (e.g. "Rectangle with text âHelloâ"). NODE TYPES (only these two): - "drawing" = individual sketch/text on the canvas (one per drawing) - "concept" = meaningful group of 2+ related items (drawings or concepts) RELATIONSHIP & GROUPING RULES â READ CAREFULLY: 14 Thinkink - Only group drawings under a shared "concept" node if they have a CLEAR, SPECIFIC, and MEANINGFUL semantic relationship. - DO NOT group drawings together just because they are on the same canvas, are spatially near each other, or are both hand-drawn items. - DO NOT create vague concept labels like "Ideas", "Notes", "Thoughts", "Canvas Items", "Topics", "Concepts", or "Main Ideas". - If drawings cover DIFFERENT, UNRELATED topics, they MUST remain as separate root nodes. - Prefer FLAT structures with multiple independent roots over a single deeply nested tree. - A concept node is only justified when removing it would lose meaningful structural information about how its children relate to each other. STRUCTURAL RULES: - Total drawing nodes must EXACTLY equal total individual drawings provided - Concept = parent with 2+ children (drawing or concept) - Multiple roots are expected and preferred when drawings are unrelated - No duplication of similar nodes Node fields: drawing: id, nodeType, drawingId (string index), drawingType, text/desc, children concept: id, nodeType, conceptLabel, children Return ONLY valid JSON, no additional text or explanations. Return JSON format: "semanticTree": [...] CRITICAL - DRAWING INDEX MAPPING: - There are EXACTLY N drawings (indices 0 to N-1) - The "semanticTree" MUST contain a drawing node with drawingId for ALL N drawings using their indices (0-N-1) - DO NOT skip any drawing index - Use drawingId as an INDEX (0, 1, 2, ...) not as a string ID B.2 Prompt for the Insight Nodes Layer Generation System instruction for generating new code (promptType: gen_code) Given this existing semantic tree of a hand-drawn thinking canvas: <treeJson> <DRAWING DIMENSIONS section if available> Generate AI insights nodes and attach them as children of the appropriate drawing nodes or concept nodes. Node types to generate: "text_generation_request": a concise request (5â15 words) for LLM-generated text information related to the parent nodeâs content "generation_without_request": insight output with one of these subtypes: - insightSubtype="socratic": a thought-provoking Socratic question (5â15 words) that deepens thinking about the parent node - insightSubtype="suggest_next_drawing": a concise suggestion (1â4 words) for a specific next drawing element that the user might add near the parent node RULES: - These nodes can be children of drawing nodes OR concept nodes - Multiple insight nodes of different types can be attached to the same parent - No duplication of similar nodes - A node should be attached to the parent where the content is most relevant - Generate insight nodes only when they meaningfully support ideation - generation_without_request nodes may use ONLY these two subtypes: "socratic" and "suggest_next_drawing" - For suggest_next_drawing, output the name of a specific drawing element, NOT an instruction sentence. Examples: "arrow", "circle", "label", "branch", "example box" - For responsePosition, choose a location based on available space and meaningful placement relative to the parent drawing or concept Node fields: text_generation_request: id, nodeType, requestPrompt, responsePosition, fontSize, suggestedWidth, suggestedHeight, children generation_without_request: id, nodeType, insightSubtype ("socratic"|"suggest_next_drawing"), requestPrompt, responsePosition, fontSize, suggestedWidth, suggestedHeight, children fontSize / suggestedWidth / suggestedHeight rules: - fontSize: integer (12â220), estimated from parent drawing scale - suggestedWidth: max(parent drawing widthĂ 2, fontSizeĂ 12), minimum 350px - suggestedHeight: roughly parent drawing height, minimum 50px Return the full modified semantic tree: "semanticTree": [...] B.3 Prompt for Text Response Generation System instruction for generating new code (promptType: gen_code) The user has drawn something and is asking: "<requestPrompt>" CONTEXT: <context item 1> 15 Payandeh et al. <context item 2> ... Respond in relation to the context. Provide a brief, direct answer (max 25 words). C Diagnostic Study C.1 Interview Questions General 1. What are your general thoughts and insights? 2. How would you compare this experience to trying to solve this problem using a standard chatbot (like ChatGPT)? 3. How would you compare this to solving the problem on a physical whiteboard or paper without AI? Four aspects: Ideation Workflows, UI and Interaction Design, AI Integration, and AI Outputs: 1. What challenges did you face when doing the tasks? 2. You performed two tasks. Did you find yourself using the tool differently between the two? How so? 3. How can the tool help you perform your task? What feature is missing? 4. Were there moments where you wanted to express an idea to the AI but felt the sketching/writing tools limited you? If so, what were these moments? 5.Did you encounter limitations when trying to convey ideas to the AI using the sketches themselves? If so, what were you trying to communicate? 6. How did you feel about using a 2D space to communicate with AI? 7. What was your experience using a 2D canvas to perform the tasks? Did it feel like using pen and paper or not? Why? 8. What was your experience with the merging feature? 9. When the AI responded, did the content match what you were looking for? If not, what did you expect to see? 10. When the AI responded, did the presentation/format match what you were looking for? If not, what did you expect to see? 11. How did you feel about the visual presentation of the AIâs answers? Did they clutter your space, or did they fit into your flow? 12. Can you point to a specific moment where the AIâs response triggered a âlightbulb momentâ or a shift in your reasoning? Closing 1. Do you have any further thoughts or suggestions for enhancing the tool? C.2 Participants Choices for the tasks PID Task 1: ChosenTask 2: Self-defined P1How can a professor increase efficiency and research output without constantly pushing group members? Does an âarea lawâ exist for tripartite GHZ entanglement? P2 Which is more fulfilling: working independently or in a team, and why?The ATC horror game. P3How can I maintain a healthy lifestyle?In computer science education, how should AI tools be designed to support studentsâ reasoning and learning? P4How can I plan my day and reduce my social media usage?What is the best assumption for a specific fluid problem? P5How do philosophers draw conclusions from a given idea?How can I integrate social learning into an ethical reinforcement learning process? P6Which does an LLM suggest: traveling by official transportation services or using rideshare? How can I simulate the polarization profile after passing through optical elements? Table C.3: Participantsâ selected tasks organized by category 16 Thinkink C.3 Artifacts Figure C.2: Examples of artifacts made by participants during the diagnostic study 17 Payandeh et al. D Usage Study D.1 Interview Questions General. 1. What are your general thoughts and insights? 2. How would you compare this experience to trying to solve this problem using a standard chatbot (like ChatGPT)? 3. How would you compare this to solving the problem on a physical whiteboard or paper without AI? Four aspects: Ideation Workflows, UI and Interaction Design, AI Integration, and AI Outputs: 1. How did the tool support or hinder your ability to connect ideas and build a conclusion today? What worked well, and what felt unnatural? 2.Can you describe a specific moment where you reached a mental block or knowledge gap, and how the tool helped (or failed to help) you move forward? 3.Did the spatial layout of the canvas change the way you reasoned through your problem compared to a standard vertical document? How? 4. How did the AI change what you wrote/drew or how you organized the canvas? 5. How did having the AI responses directly on the 2D canvas (rather than in a linear chat) affect how you synthesized information? 6. Did the tool feel like a unified workspace, or did you feel like you were switching between âthinking modeâ and âAI modeâ? 7. How do you feel about the interactions? Was it easy to get the most of the tool using the interactions it required? 8. At what moments did you decide to use (or not use) the AI? What triggered those decisions? 9. In what scenarios did you prefer Ask AI over AI insights, or vice versa? 10. How transparent did the AI feel to you in terms of which parts of the canvas it was reading or using to generate its responses? 11. Did you feel you had enough manual control to change information the AI was looking at before it gave you an answer? 12. How did this level of control affect your trust in the accuracy and relevance of the AIâs responses? 13. How did you manage the sheer amount of information the AI gave you? Did you use the Generation History? 14. How was your experience modifying or editing the AIâs generated text/images? Challenges: 1. Were there any moments where the tool did something you didnât expect? If so, how easily were you able to undo it or recover your flow? 2. Did the AI insights ever feel intrusive or interrupt your train of thought? 3.Did you ever feel anxious about âlosingâ an AI response like you might in a continuous chat thread? How did the tool alleviate or worsen this? Returning Participants vs. New Participants ⢠For Returning Participants: You tested an earlier version of the tool. What stood out to you as the biggest improvement today? Did any of the changes introduce new problems for your workflow? â˘For New Participants: If you were to adopt this tool for your domain tomorrow, what is the very first feature you would use and what would you change? Closing. 1. Do you have any further thoughts and insights? Additionally, do you have any suggestions for enhancing the tool? D.2 Participants Choices for the tasks PIDTask 1: ChosenTask 2: Self-defined P1What motivates me to learn about historyThe capacity of wiretap channel P2Decide about weapon purchaseBrainstorming my next painting P3What does success mean?How to teach people to use a debugger P4How would you structure a six-month plan to achieve my goal?Dependency flow inside the pipe to geometric parameters P5Gym planningUnderstanding semi MDP P7How would you structure a six-month plan to achieve this goal?Ways to do detrending P8If you had to focus on only one project this quarter, which would it be and why? Learn about how entrepreneurs contribute to sustainability P9What motivates me to continue my career?Brainstorming my PhD thesis topic (health) P10How can we fix the issues weâre facing with food shortages?How to use AI for sustainable development? P11Where do you see yourself in five years professionally?Drawing a plasmid to enter into E. coli host, ensuring I have all features needed Table D.4: Participantsâ selected tasks. Returning participants were P1âP5; new participants were P7âP11. Returning participant P6 did not join this study. 18 Thinkink D.3 Artifacts Figure D.3: Examples of artifacts made by participants during the usage study D.4 Pictures of Participants Using Thinkink Figure D.4: Pictures of Participants Using Thinkink 19 Payandeh et al. D.5 Pictures of Participants Using Thinkink Figure D.5: Detailed version of State Machine Interaction Design, incluging Thinkink Screenshots. Rectangles represent substates grouped into two superstates: âNote-takingâ (âDrawings,â âEditâ) and âLLM assistantâ (âPrompt,â âInsights,â âIterate,â âInspectâ). Double-headed arrows indicate bidirectional transitions, while single-headed arrows denote unidirectional transitions. Circular loops represent self-transitions (within-substate interactions). Edge colors encode transition types: black for transitions within âNote-taking,â indigo for transitions within âLLM assistant,â and cyan for cross-superstate transitions. This detailed version augments this state machine with accompanying screenshots, providing concrete examples of each substate and illustrating how transitions are manifested in the interface. 20 Thinkink D.6 State Machine Interaction Design Usage Patterns Figure D.6: Per-participant state diagrams demonstrate how each participant navigated among the six substates of ThinkInk, with edge thickness indicating transition frequency and self-loops indicating repeated interaction within a substate. Arrow colors encode transition type: black edges indicate transitions within "Note-taking" substates, indigo edges indicate transitions within "AI assistant" substates, and cyan edges indicate transitions between the two superstates. The figure highlights substantial flexibility in use patterns: some participants, including P2, P4, P7, and P8, highly used the "Note-taking" superstate, including "Drawings" and "Edit" substates, whereas others, including P3, P9, and P10, concentrated much more of their interaction in the "AI assistant" superstate than the "Note-taking" superstate. "AI Insights" was used by all participants except P3, P8, and P10, further underscoring that the interface afforded multiple viable ways of working rather than steering users into a single prescribed workflow. At the same time, "Iterate" emerged as the most consistently prominent substate across participants, suggesting that refinement of generations was a central part of interaction with the system. 21 Payandeh et al. E Diagnostic and Usage Studies survey questionsâ responses Diagnostic study (technical probe) (í=6) Usage study (Thinkink) (í=10) Code QuestionMedian (IQR)Mean (SD)Median (IQR)Mean (SD) Ideation Support and User Workflows3.754.38 Q1.1The tool helped me generate new explanations, hypotheses, or information I had not considered. 4.00 (2.50â4.00)3.50 (1.22)4.00 (4.00â4.75)4.30 (0.48) Q1.2I felt satisfied with the conclusion I reached at the end of the session.4.00 (3.25â4.75)3.83 (1.17)4.00 (4.00â4.00)4.20 (0.42) Q1.3The integrated LLM helped me reach a satisfactory conclusion.4.00 (4.00â4.00)3.83 (0.98)4.00 (4.00â4.00)4.10 (0.57) Q1.4The sketching interface helped me think clearly and easily.4.50 (2.50â5.00)3.83 (1.47)5.00 (5.00â5.00)4.80 (0.42) Q1.5Using this interface, I was able to make better progress toward my goal.â5.00 (4.00â5.00)4.50 (0.71) User Interface and Interaction Design3.543.90 Q2.1It was easy to convey my intent to the AI using sketches and handwriting.4.00 (3.25â4.00)3.67 (1.03)4.00 (4.00â4.75)3.90 (1.10) Q2.2The AI can successfully interpret my intention in the spatial arrangement of my drawings. 4.00 (3.25â4.00)3.83 (0.75)3.50 (3.00â4.00)3.70 (0.82) Q2.3I did not feel like I had to fight the tool to make the AI understand me.4.00 (2.50â4.75)3.67 (1.37)4.00 (4.00â4.00)3.90 (0.74) Q2.4The inking experience mimics the feel of pen on paper. 3.00 (2.00â4.00)3.00 (1.55)4.00 (2.25â4.75)3.50 (1.43) Q2.5 It was clear and easy to navigate between different screens of the app, including note-taking and AI modes. â4.00 (4.00â5.00)4.40 (0.52) Q2.6The interface provided enough visible cues (labels, icons, or feedback) for me to confidently find and use features. â5.00 (4.00â5.00)4.30 (1.06) Q2.7I had enough control to adjust, undo, or override the systemâs merging/grouping of my notes. â4.00 (3.25â4.00)3.60 (1.26) AI Integrationâ4.40 Q3.1AI responses appeared at appropriate times and did not interrupt or distract from my thinking process. â4.50 (4.00â5.00)4.40 (0.70) AI Outputs3.834.12 Q4.1The AIâs responses were useful to the context of my problem.4.00 (4.00â4.00)4.17 (0.41)4.00 (4.00â4.75)4.30 (0.48) Q4.2The presentation/format of the AIâs response as text on the canvas was useful to the context of my problem. 4.50 (2.50â5.00)3.83 (1.47)5.00 (4.25â5.00)4.50 (0.85) Q4.3The AIâs responses appeared in the location that I expected them to.4.00 (2.25â5.00)3.50 (1.76)5.00 (4.00â5.00)4.20 (1.23) Q4.4I could understand and control what information the AI used from my canvas when generating a response. â3.50 (3.00â5.00)3.70 (1.25) Q4.5 It was easy to revisit, recover, and compare previous AI generations when I needed them. â4.00 (4.00â4.00)3.90 (0.88) Table E.5: Combined descriptive statistics for post-task questionnaires. Categorical headers display aggregate mean scores. Items were rated on a 5-point Likert scale. Bold indicates the higher Mean (SD) between studies for that row; where only one study was available, its Mean (SD) is bolded. 22