Paper deep dive
Investigating In-Context Privacy Learning by Integrating User-Facing Privacy Tools into Conversational Agents
Mohammad Hadi Nezhad, Francisco Enrique Vicente Castro, Ivon Arroyo
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/23/2026, 12:04:30 PM
Summary
This study investigates in-context, experiential privacy learning by integrating a just-in-time privacy notice panel into a simulated ChatGPT interface. The research evaluates how such tools influence user perceptions of sensitive information and privacy-protective behaviors among Computer Science students, while also identifying UI/UX design features that facilitate or hinder engagement with privacy protections.
Entities (5)
Relation Signals (3)
Privacy Notice Panel â integratedwith â ChatGPT
confidence 95% ¡ we integrated a simulated ChatGPT interface with our privacy panel
Computer Science Students â interactedwith â Privacy Notice Panel
confidence 95% ¡ students interact with our privacy panel, which provides learning opportunities
Privacy Notice Panel â facilitates â Experiential Learning
confidence 90% ¡ investigate in-context, experiential learning by examining how interactions with privacy tools during chatbot use enhance users' privacy learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Supporting users in protecting sensitive information when using conversational agents (CAs) is crucial, as users may undervalue privacy protection due to outdated, partial, or inaccurate knowledge about privacy in CAs. Although privacy knowledge can be developed through standalone resources, it may not readily translate into practice and may remain detached from real-time contexts of use. In this study, we investigate in-context, experiential learning by examining how interactions with privacy tools during chatbot use enhance users' privacy learning. We also explore interface design features that facilitate engagement with these tools and learning about privacy by simulating ChatGPT's interface which we integrated with a just-in-time privacy notice panel. The panel intercepts messages containing sensitive information, warns users about potential sensitivity, offers protective actions, and provides FAQs about privacy in CAs. Participants used versions of the chatbot with and without the privacy panel across two task sessions designed to approximate realistic chatbot use. We qualitatively analyzed participants' pre- and post-test survey responses and think-aloud transcripts and describe findings related to (a) participants' perceptions of privacy before and after the task sessions and (b) interface design features that supported or hindered user-led protection of sensitive information. Finally, we discuss future directions for designing user-facing privacy tools in CAs that promote privacy learning and user engagement in protecting privacy in CAs.
Tags
Links
- Source: https://arxiv.org/abs/2603.19416v1
- Canonical: https://arxiv.org/abs/2603.19416v1
Trouble viewing inline? Open PDF directly â
Full Text
75,019 characters extracted from source content.
Expand or collapse full text
Investigating In-Context Privacy Learning by Integrating User-Facing Privacy Tools into Conversational Agents Mohammad Hadi Nezhad University of Massachusetts Amherst mhadinezhad@cs.umass.edu Francisco Enrique Vicente Castro New York University Ivon Arroyo University of Massachusetts Amherst Abstract Supporting users in protecting sensitive information when using conversational agents (CAs) is crucial, as users may undervalue privacy protection due to outdated, partial, or in- accurate knowledge about privacy in CAs. Although privacy knowledge can be developed through standalone resources, it may not readily translate into practice and may remain detached from real-time contexts of use. In this study, we in- vestigate in-context, experiential learning by examining how interactions with privacy tools during chatbot use enhance usersâ privacy learning. We also explore interface design fea- tures that facilitate engagement with these tools and learning about privacy by simulating ChatGPTâs interface which we integrated with a just-in-time privacy notice panel. The panel intercepts messages containing sensitive information, warns users about potential sensitivity, offers protective actions, and provides FAQs about privacy in CAs. Participants used ver- sions of the chatbot with and without the privacy panel across two task sessions designed to approximate realistic chatbot use. We qualitatively analyzed participantsâ pre- and post- test survey responses and think-aloud transcripts and describe findings related to (a) participantsâ perceptions of privacy before and after the task sessions and (b) interface design features that supported or hindered user-led protection of sen- sitive information. Finally, we discuss future directions for designing user-facing privacy tools in CAs that promote pri- vacy learning and user engagement in protecting privacy in CAs. 1 Introduction Protecting privacy during conversational agent (hereafter CAs or chatbots) use requires users to continuously assess what informationâabout themselves or othersââis appropriate, or fitting, to reveal in a particular contextâ [35]. However, users often disclose sensitive information that may not be necessary for completing tasks [28, 30, 43]. Even when such information is relevant, disclosure decisions are not always made by a careful weighing of immediate benefits against potential future privacy risks. Examples include treating CAs as therapists and sharing emotional or behavioral details, pro- viding medical or fitness information to receive advice, or uploading lengthy documents (e.g., emails or contracts) for summarization or search [30, 43]. In pursuing these goals, users may overlook the sensitivity of the information they are disclosing, be unaware of available actions to protect their information, or find such protective actions time-consuming or inconvenient. As a result, they may undervalue privacy protection and its importance during CA interactions. This makes it essential to support users in more context-aware and privacy-conscious decision-making. Given these concerns, it is essential to support different user groups in developing a stronger understanding of privacy in the context of CAs, as supporting them can promote greater and more meaningful engagement with privacy-conscious be- haviors [9, 44]. While such understanding can be fostered through standalone learning resources (e.g., educational tools or academic programs) [13, 15], it may not easily translate into practice, as it is often detached from the real-time con- texts of use in which privacy decisions must be made (e.g., what information to protect and how, depending on the tasks and constraints including time and effort) and where timely awareness and support are most critical. We thus explore how exposing users to privacy tools during realistic chatbot use scenarios can support experiential, in-context learning about privacy, while also exploring user interface and experience (UI/UX) design features that encourage user engagement with privacy protections and facilitate learning. We first examine key aspects of privacy perceptions among undergraduate and masterâs students in a Computer Science (CS) program in the US (participant details in Section 3.3). We then analyze how these perceptions evolve through in- teractions with a just-in-time privacy notice panel (Section 3.1) that embeds learning opportunities directly within CA interfaces. To this end, we integrated a simulated ChatGPT interface with our privacy panel that intercepts interactions im- mediately after users attempt to submit a message containing 1 arXiv:2603.19416v1 [cs.HC] 19 Mar 2026 sensitive information. Our panel (1) detects and informs users about potentially sensitive information in their messages (e.g., names, physical addresses); (2) offers features for applying anonymization strategies (retracting, faking, generalizing); (3) surfaces two built-in ChatGPT privacy controls (disabling memory, opting out of data sharing); and (4) includes FAQs that describe how to think about sensitive information and factors to consider when making disclosure decisions. We analyzed our panelâs effectiveness through a five-phase study (Section 3.2) with ten participants: (1) a pre-test survey examining baseline privacy perceptions; (2â3) two study ses- sions to complete tasks using the simulated chatbot with and without our privacy panel; (4) an immediate post-test survey collecting reflections on the task sessions; and (5) a delayed post-test survey to examine changes in privacy perceptions. Our task and study design closely approximate realistic chat- bot use for our participant group by incorporating a range of real-world use scenarios involving sensitive information (e.g., writing emails, searching documents), withholding the studyâs privacy goals until after participation to reduce bias, and al- lowing participants to freely direct their own interactions (e.g., prompting, disclosure decisions). Thus, we examine the fol- lowing research question (RQ): RQ1. Before and after completing tasks using the chatbot, what did students think aboutâ 1.1. what they consider as sensitive information? 1.2.the importance of protecting sensitive information when using chatbots? 1.3.what they can do to protect sensitive information when using ChatGPT? Given the critical role of UI/UX design in usersâ engage- ment with and adoption of interactive tools, there are gaps and opportunities to better understand how interface design deci- sions support or hinder the protection of sensitive information when using CAs. Thus, we adopt an exploratory approach to examine such design features in our privacy panel by analyz- ing studentsâ think-aloud transcripts and interface interactions from task sessions and responses to open-ended survey ques- tions. Therefore, we examine the following RQ: RQ2.What interface design features of the privacy panel sup- port or hinder the protection of sensitive information? Through a detailed qualitative analysis of data (Section 3.4), we present findings on how our participants conceptualized sensitive information (Section 4.1), viewed the importance of protecting sensitive information during CA use (Section 4.2), and what protective actions they thought they can take (Section 4.3). We also report findings on interface design deci- sions that supported or hindered usersâ protection of sensitive information when interacting with our privacy panel (Section 4.5). We then discuss opportunities for future work on en- hancing in-context support and experiential privacy learning (Section 5.1), designing UI/UX to promote engagement and learning in user-facing privacy tools (Section 5.2), support- ing users in navigating privacy trade-offs (Section 5.3), and identifying learning opportunities for students (Section 5.4). 2 Background and Related Work 2.1User Knowledge and Agency Over Privacy When Using Conversational Agents (CA) Users need to understand and consider various aspects of privacy in their decision-making to meaningfully engage in privacy-protective behaviors in CAs use. Decisions about what to disclose or withhold may be shaped by several factors, including individual preferences (e.g., sensitivity of financial information may differ across individuals [5]), the features of the technology (e.g., privacy policies, security safeguards), and the perceived trustworthiness of the technology providers in protecting user privacy [25, 31, 37, 43]. However, usersâ awareness and understanding of these factors may not be up- dated by, for example, current data practices. For instance, research shows that users often rely on partial, simplified, or inaccurate mental models of how LLM-based CAs process their data [43]. They may not recognize what types of infor- mation can be used to identify individuals [31, 37], or may underestimate the potentials of these tools in exacerbating the expected consequences of data leakageâfor themselves and othersâthrough capabilities such as real-time, detailed profiling that can heighten the risks of data access and mis- use [20, 26]. Moreover, even when users intend to protect sen- sitive information, they may be unaware of available privacy- protective actions, whether built into the tool (e.g., ChatGPTâs option to opt-out of content sharing for model training) or actions they can take themselves to address the toolâs privacy limitations (e.g., anonymizing sensitive data before submis- sion) [31, 43]. These gaps in knowledge and awareness can lead users to overlook or undervalue privacy protection, es- pecially when weighing trade-offs such as deciding whether to disclose more sensitive information to improve chatbot performance or withhold it to safeguard privacy. Research has suggested that privacy perceptions (e.g., re- garding information sensitivity) can vary across people based on factors such as age, gender, education, and geographic location [5, 12, 27, 28, 40]. This highlights the need to exam- ine privacy perceptions across different user groups, avoid overgeneralizing findings, and tailor support to the specific needs of each group. For example, a study with UK-based CA users aged 18 and over found that participants lacked reliable strategies for protecting privacy, had difficulty under- standing privacy features and their outcomes, and showed low baseline awareness of privacy risks [28]. In another study, researchers investigated how users of womenâs period and fertility tracking apps define and understand personally identi- fiable information (PII), revealing mismatches between usersâ 2 perceptions and regulations regarding PII protections [37]. In this study, we examined the privacy perceptions of Com- puter Science (CS) undergraduate and masterâs students at a US university (details in Section 3.3). Specifically, we ex- plored how they conceptualize sensitive information, how important it is to protect such information during CA use, and what protective actions they think they can take. We also investigate how their views changed after interacting with our privacy panel while completing realistic chatbot tasks. 2.2 User-Facing Privacy Tools for Enhancing Knowledge During CA Use Interactive systems can support knowledge development by embedding learning opportunities directly within usersâ work- flows. This can enable learning to occur in context and through experienceâe.g., through real tasks, constraints, and goals (see also Experiential Learning [16, 23]). Accordingly, integrating user-facing privacy tools into CAs can support users in engaging with privacy-protective practices, such as recognizing sensitive information and protective approaches [31, 44]. When these tools are integrated into the interaction context, they can help highlight the necessity of protections (e.g., observing the extent of sensitive info being disclosed, reflecting whether protections impact chatbot performance). This, in turn, can incentivize users to take protective actions, engage more deeply with thinking and learning about privacy in CAs, and apply this acquired knowledge in their future interactions. Scholars have developed user-facing privacy tools for CAs to promote privacy awareness and encourage protective ac- tions. For example, Clear [9] is a just-in-time interface that au- tomatically identifies sensitive information in user messages and informs users of relevant privacy policies and potential disclosure risks. Rescriber [44] assists users in detecting and sanitizing sensitive content in their prompts. Casper [10] automatically anonymizes PII and notifies users about sen- sitive topics in their messages. However, these studies did not systematically analyze changes in usersâ lasting privacy knowledge, focusing instead on short-term awareness. Only the Rescriber study reported insights from short-term reflec- tions, suggesting that participants had perceptions of learning. In this study, students interact with our privacy panel, which provides learning opportunities through identifying and in- forming users about certain types of potentially sensitive infor- mation in their messages, offering anonymization strategies (retracting, faking, and generalizing), surfacing two built-in chatbot privacy controls (opting out of content sharing for model training, disabling memory), and presenting a set of FAQs about privacy in CAs. We assess changes in partici- pantsâ lasting privacy knowledge (i.e., ability to recall and explain) by administering pre- and post-test surveys at least one week before and after the task sessions. 2.3UI/UX Design for Engagement with Pri- vacy Tools The design of the UI/UX plays a critical role in the adoption and effective engagement with user-facing tools. In interactive systems, usersâ choices are often shaped by factors including how options are presented, when they are surfaced, what al- ternative choices are offered, and which default options are provided (particularly given that default options tend to be selected more frequently) [19, 29, 39]. For example, prior work shows that certain entry points to privacy settings can improve their discoverability, findability, and perceived usabil- ity [17]. Similarly, privacy notices that are presented at the mo- ment of decision-making and tailored to usersâ context have been found to more effectively support privacy-protective choices [9, 21, 36]. Scholars have raised related concerns in the context of privacy tools in CAs. For example, ChatGPT uses user content for model training by default and although an opt-out option exists, studies show that many users are unaware of it or do not understand how it works [31, 43]. The convenience of user actions in the tool also plays a key role in disclosure and protection behaviors. For instance, chatbot users are more likely to share information when disclosure is easily afforded by the interface (e.g., features for upload- ing lengthy documents) [43]. On the privacy-tool side, users value designs that simplify and streamline sanitization ef- forts (e.g., bulk sanitization), while also reporting challenges in understanding certain sanitization strategies (e.g., abstrac- tion) [31,44]. This highlights a need to further examine UI/UX designs that support engagement with privacy tools in CAs. In this work, we adopt an exploratory approach to identify fea- tures in the design of our privacy notice panel that supported or hindered the protection of sensitive information. 3 Methods In this section, we describe our ChatGPT interface simula- tion and privacy notice panel design (Section 3.1), the study procedure (Section 3.2), participants (Section 3.3), and data collection and analysis approach (Section 3.4). 3.1 Chatbot and Privacy Notice Panel Design We built a simulation of ChatGPTâs interface (as of Fall 2024) (Figure 1), including a message input, conversation panel, and a profile icon that provides access to a settings menu mirroring ChatGPTâs design and allowing users to opt in/out of content sharing for model training and to toggle the chatbotâs memory (Appendix A.1). When memory is enabled, the simulation considers the last seven messages; when disabled, it considers only the most recent message. Although this design does not replicate ChatGPTâs exact memory mechanism, it still creates a trade-off between contextual retention and potential privacy 3 protection. The simulation sends messages to the GPT-4o API and displays the resulting responses. Adding to this interface, we developed a privacy notice panel that intercepts message submissions immediately after users attempt to submit a message (i.e., clicking the submit button) containing any of the following types of potentially sensitive information embedded in the materials used in our user tasks (Section 3.2): names of people, email addresses, phone numbers, physical addresses, social security numbers, and dates of birth. The panel appears on the right side of the screen without blocking the ongoing interaction and its use is optional. The panel displays (as annotated in Figure 1): A.Warning Messageâinforms users that their message con- tains potentially sensitive information. B. Anonymization Panelâlists types of sensitive informa- tion detected, each expandable to show additional details. For each type, individual instances are displayed, and users can: (B.1) locate and highlight an instance in their message; (B.2) anonymize it using one of three options: retract (replace with a type label, e.g., 123-456-7890 â[Phone number]), generalize (retain coarse details such as US state and ZIP code for addresses, or year for date of birth), or fake (replace with a dummy value, e.g., CameronâArron); and (B.3) restore the original value. Alternatively, users can use Anonymize All and Restore All to apply an action to all instances of a given information type. A note at the bottom of the panel re- minds users that the list may not capture all sensitive information in their message. C. Shortcuts to Built-in Privacy Controlsâtwo buttons providing direct access to ChatGPTâs built-in controls for (a) opting in/out of content sharing for model training and (b) enabling or disabling memory (Appendix A.1). D.FAQsâincludes five questions expandable for reading the answer. Each answer is presented in two layers: a brief sentence or heading, with an optional expansion providing a one-paragraph explanation. The questions and answers are meant to provide users additional in- formation on how they can conceptualize sensitive in- formation, view their anonymity when interacting with CAs, balance privacy protection with chatbot utility, un- derstand differences between âgenerativeâ CAs and ear- lier database-driven systems, and consider the shared responsibility for privacy protection between users, the technology, and its providers. E. Proceed with Sendingâsends the current message. 3.2 Study Procedure Our study consists of five phases. To reduce bias, we adopted incomplete disclosure strategy by withholding studyâs privacy goals from participants until after participation. Our study design and protocol was approved by our institutionâs IRB. Pre-Test Survey (Phase 1): Participants completed a sur- vey with open-ended questions designed to explore initial perceptions of privacy. Specifically, we asked participants (a) what they think sensitive information means, (b) how impor- tant it is to protect sensitive information when interacting with chatbots like ChatGPT, (c) what they can do to protect sensi- tive information when interacting with ChatGPT, and to (d) describe any ChatGPT features that would help them protect sensitive information during interaction. Participants were instructed to respond based on their own thoughts, without searching online or consulting others. Task Sessions (Phases 2 and 3): At least one week after Phase 1, participants attended two remote Zoom 1 sessions, separated by a 1-4 day gap depending on availability. In the first session, they completed one version of the task assign- ments (A or B) (see next paragraph) using the chatbot without the privacy notice panel; in the second session, they completed the alternate task assignment using the chatbot with the pri- vacy notice panel. In both sessions, participants shared their screen while thinking aloud to verbalize their thoughts. The first author facilitated all sessions, beginning with an introduc- tion to the study, tasks, and chatbot, followed by providing think-aloud instructions and asking participants to complete a brief practice task. To minimize distractions, the researcher muted himself and disabled his video during tasks. We also allowed participants to freely direct their own chatbot interac- tions (e.g., prompting, anonymization, evaluating responses). User Tasks. We created two versions of task assignments (A and B) and alternated them across the two task sessions (Phases 2 and 3) for each participant to counterbalance order and content effects. Half of the participants completed Assign- ment A in the first session and Assignment B in the second session, while the order was reversed for the remaining partic- ipants (Table 1). Both assignments included comparable tasks involving text summarization, text classification, searching lengthy content (e.g., contracts), and drafting emails. Each assignment consisted of three tasks, each with two steps. For each step, participants were provided with task descriptions, required data embedded with sensitive information, and a text entry field for submitting responses. We designed the tasks to (1) approximate real-world chatbot use scenarios and be relevant to CS undergraduate and masterâs students; (2) place participants in common situations involving sensitive information disclosure (e.g., unintentional disclosure when sharing lengthy content with chatbots); and (3) avoid explicit mentions of privacy in task descriptions to minimize bias, while encouraging participants to treat the data as real or as their own. A detailed description of the tasks and embedded sensitive information is provided in (Appendix A.2). Immediate Post-Test Survey (Phase 4): Within one day 1 https://w.zoom.com/ 4 Figure 1: ChatGPT Interface Simulation With Privacy Notice Panel. The panel appears after sending a message containing sensitive info (highlighted in the input). It includes (A) a warning message, (B) an anonymization panel, (C) shortcuts to built-in privacy controls, (D) FAQs, and (E) a proceed with sending button. The anonymization panel further includes, for each detected instance, a (B.1.) locate icon, (B.2.) drop-down menu of anonymization options, and (B.3.) restore button. after Phase 3, participants completed a survey capturing their reflections on the task sessions. We included open-ended ques- tions asking participants to describe chatbot features (includ- ing the privacy panel) that supported them in protecting sen- sitive information during interaction; to share their thoughts about the design of each component of the privacy notice panel (i.e., the anonymization panel, FAQs, and shortcuts to built-in privacy controls); to describe how easy or difficult they found it to use the panel; to suggest ways to improve the panel; and to provide any additional comments. Delayed Post-Test Survey (Phase 5): At least one week af- ter Phase 3, participants completed a survey that repeated the pre-test (Phase 1) questions to examine changes in their pri- vacy perceptions. The survey also included questions asking whether and how participation in the study influenced partic- ipantsâ thinking about privacy when using chatbots, as well as inviting any additional comments about their participation. We administered all surveys through Qualtrics 2 . 3.3 Participants Our work focuses on undergraduate and masterâs students in a CS program in the US. We selected this participant group because students are frequent users of chatbots (particularly ChatGPT) [38, 42], young users are more prone to disclosing sensitive information [24], and CS students are likely future developers of chatbots, making their privacy perceptions espe- cially relevant to examine. Focusing on a specific participant group also allowed us to tailor the design of tasks, surveys, and interfaces to participantsâ contexts, thereby strengthening the ecological validity of the studyâe.g., by including data 2 https://w.qualtrics.com/ 5 analysis tasks (e.g., classification), using documents related to participantsâ semester schedules designed for CS students, and adapting the language of FAQ content and survey ques- tions to be relevant to participantsâ context. Additionally, this population enabled us to control for variation in age and geographic context while allowing diversity in gender and frequency of chatbot use [12, 27, 28, 40]. We recruited participants through a poster distributed at a large public university in the northeastern US. Interested indi- viduals completed a screening survey collecting background information (e.g., degree level, gender, and ChatGPT use). We recruited 11 students, one of whom completed a pilot session that informed refinements to the study protocol. The final sample included ten participants, ranging from third-year undergraduate to second-year masterâs students, evenly split by (self-reported) gender (five male and five female), with varying frequency of ChatGPT use (e.g., daily or several times per week). Participants completed all study phases between March and June 2025. We paused recruitment after ten partic- ipants, as our iterative analysis suggested thematic saturation, with no substantively new themes observable in additional responses or sessions. Each participant received a $70 USD gift card. Participant details are summarized in Table 1. 3.4 Data Collection and Analysis Approach For each participant, we collected responses to all pre- and post-test surveys along with audio and video recordings of the two task sessions, including think-aloud transcripts. Each task session, excluding the introduction and practice task, lasted between 15 and 80 minutes. We analyzed qualitative data using thematic coding [6], with procedures tailored to each research question (RQ). To examine RQ1.1, RQ1.2, and RQ1.3, we grouped participant responses from the pre-test (Phase 1) and delayed post-test (Phase 5) surveys according to the relevance of each survey question to the corresponding RQ. For data relevant to RQ1.1 and RQ1.2, we conducted inductive thematic coding to derive codes directly from responses. For RQ1.3, we employed a hybrid thematic coding (inductive and deductive), initializing the codebook with codes corresponding to privacy-protective actions supported by the chatbot and privacy notice panel (e.g., retracting, faking, disabling memory) and iteratively refining the codebook based on the data. For each data group, we followed an iterative process in which recurring and related codes were identified and grouped to develop a codebook with representative themes and detailed descriptions. Two authors then independently applied each codebook to its associated dataset and met in weekly collaborative interpretation sessions to resolve ambiguities, reconcile disagreements, and finalize theme definitions through shared interpretation of the data. The resulting themes are presented in Table 2 and described in Section 4. To examine RQ2, we inductively coded the think- aloud transcripts from the task sessions (Phases 2 and 3) and responses to the immediate post-test survey (Phase 4). While themes were derived inductively from the data, we oriented the analysis toward understanding participantsâ perspectives on interface and experience design features of our privacy panel that supported or hindered interactions (Section 4.5). 4 Results We present themes from participantsâ responses concerning their conceptualizations of sensitive information (RQ1.1; Sec- tion 4.1), views on the importance of protecting it during chatbot use (RQ1.2; Section 4.2), and available protective actions (RQ1.3; Section 4.3). These are summarized in Ta- ble 2. For each theme, we provide a brief description followed by illustrative quotes from pre- and post-test responses. Par- ticipant IDs shown in bold indicate those who articulated a theme after the study but not before. We then present stu- dentsâ reflections on how participation in the study influenced their privacy thinking (Section 4.4) and how the design of our privacy panel supported or hindered user-led protection of sensitive information (RQ2; Section 4.5). 4.1 How students conceptualize sensitive infor- mation (RQ1.1) 4.1.1 Potentials for Harm if Leaked Under this conceptualization, students viewed information as sensitive if access by others could enable harm to information owners or to whom the information pertains. Before the study, responses from seven students (P1, P2, P3, P5, P6, P7, P8) reflected this viewâe.g., P3 noted that the leakage of sensi- tive information âleads to severe consequences to the person including identity theft.â Similarly, P6 explained: â[...][ac- cess to such information] could potentially result in security threats or danger of some kind.â In post-test responses, five students (P1, P2, P3, P7, P10) showed this conceptualization, including P10, who had not articulated it before the study, saying: âIt is a piece of data that we canât share with anybody and its loss can lead to serious crimes such as identity theft, misuse of data and so on.â 4.1.2 Restricted Access Under this theme, types of info are considered sensitive when students want them to remain confidential or shared only with specific people, rather than accessed publicly through viewing, knowing, or possession. Before the study, seven students (P1, P4, P5, P6, P7, P9, P10) articulated this conceptualizationâ e.g., P10 stated: âIt is similar to having FERPA rights and only certain people should have access to that info.â and P5 noted: â[this information] should only be shared with a selected group of people.â After the study, we observed this theme in the responses of seven students (P3, P4, P5, P6, P7, 6 Table 1: Participant Details (self-reported) and their Task Assignment Versions. Session 1 refers to the task session using the chatbot without the panel, and Session 2 refers to the session using the chatbot with the panel. IDDegree - YearGenderChatGPT UseTask Assignment Version DurationFrequencySession 1Session 2 P1Master - 1st yearFemale~2.5 yearDailyAB P2Undergrad - 4th yearFemale~2.5 yearsDailyBA P3Master - 1st yearFemale~2.5 yearsSeveral times a weekAB P4Master - 2nd yearMale~2.5 yearsDailyBA P5Undergrad - 3rd yearMale~1.5 yearsDailyAB P6Undergrad - 4th yearMale~2 yearsSeveral times a weekBA P7Undergrad - 4th yearFemale~2 yearsSeveral times a weekAB P8Undergrad - 3rd yearMale~1 yearOnce a weekBA P9Master - 1st yearMale~3 yearsDailyAB P10Undergrad - 3rd yearFemale~1.5 yearsDailyBA P9, P10), including P3 who had not articulated it prior to the study. 4.1.3 Linkable to Identity This theme characterizes information as sensitive when it can be used to identify individuals, either directly or indirectly (similar to Personally Identifiable Information, PIIs). Five students (P2, P3, P4, P8, P10) expressed this view in their pre- test responsesâe.g., P2 emphasized the traceability of such information: âI think sensitive information is any information that can be traced back to find out who it belongs to [...].â Similarly, P3 noted: âSensitive information is information that [...] can lead to the identity of an individual.â Post-test responses reflected this theme for three students (P2, P3, P8), all of whom had articulated this view prior to the study. 4.1.4 Context-based Sensitivity This theme emphasizes that what is considered sensitive can vary across individuals and context. Two students showed this view prior to the study (P5, P9)âe.g., P5 described the subjectivity of sensitive information: âSensitive info depends on who it is sensitive for [...]â, and P9 emphasized its context dependence: âIt depends largely on the context for me.â After the study, responses from three students reflected this perspec- tive (P6, P9, P10), including P6 and P10, who had not noted it before the studyâe.g., P6 described protections based on ownerâs preferences â[...][sensitive info] can be protected depending on what the source/owner of the info wants.â 4.1.5 Security Measures This theme captures responses that characterize sensitive in- formation as those often protected or secured by specific technical measures. P5 articulated this prior to the study, and P6 talked about it after the studyâe.g., P5 stated: âThis in- formation is usually handled with multiple levels of security to prevent leaks.â 4.2 Studentsâ views on the importance of pro- tecting sensitive info in chatbot use (RQ1.2) 4.2.1 Unclear Data Lifecycle This theme captures views on the importance of protecting sensitive information during chatbot use, driven by concerns and uncertainties about what happens to user data after sub- mission, including where it is stored, how long it is retained, how it is used in back-end processes such as model training and response generation, and the possibility of usersâ data be- ing sold to third parties. Before the study, seven students (P1, P4, P5, P6, P8, P9, P10) shared such concerns. For example, P4 discussed the difficulty of removing sensitive details once it is used to train models: â[Protecting sensitive info during CA use is] very important, because these providers potentially train on chats, making it very hard to remove the info.â and P1 elaborated on ambiguities around data use âitâs not very clear how this data could be used, unlike platforms like Google, where itâs obvious your data might be used for targeted ads or recommendations. With ChatGPT, thereâs no visible feed- back loop like that, so it doesnât feel like your data is being used in any particular way.â After the study, we observed this theme in the responses of seven students (P3, P4, P5, P6, P7, P8, P9), including P3, and P7, who had not mentioned it before the study. For instance, P7 discussed the possibility of training models on private data: âI know that they could use private data to train on and improve performance [...],â noting that this view resulted from participating in this study. 4.2.2 Unauthorized Access to Data Students described ways in which sensitive information shared with chatbots (like ChatGPT) could be accessed by 7 Table 2: Columns (LâR): RQ, identified Themes & Sub-Themes, the list of participants who articulated each theme before and after the study. Bold participant IDs denote those who discussed the theme after the study but not before. RQThemes & Sub-ThemesBefore StudyAfter Study RQ1.1. How students conceptualize sensitive information. ⢠Potentials for harm if leaked (4.1.1)P1, P2, P3, P5, P6, P7, P8P1, P2, P3, P7, P10 ⢠Restricted access (4.1.2) P1, P4, P5, P6, P7, P9, P10P3, P4, P5, P6, P7, P9, P10 ⢠Linkable to identity (4.1.3)P2, P3, P4, P8, P10P2, P3, P8 ⢠Context-based sensitivity (4.1.4)P5, P9P6, P9, P10 ⢠Security measures (4.1.5)P5P6 RQ1.2. Views on the importance of protecting sensitive info during chatbot use ⢠Unclear data lifecycle (4.2.1) P1, P4, P5, P6, P8, P9, P10P3, P4, P5, P6, P7, P8, P9 ⢠Unauthorized access to data (4.2.2)P2, P3, P4, P9, P10P2, P3, P4, P8, P9, P10 ⢠Potential consequences of data misuse (4.2.3)P1, P3, P6, P10P7, P10 ⢠Corp. distrust and lack of transparency (4.2.4)P2, P5, P8P3, P4, P6, P7 ⢠Self-disclosure needs (4.2.5)P1, P7- RQ1.3. What students think they can do to protect sensitive information during ChatGPT use. ⢠Withholding/anonymizing (4.3.1) ⢠Information exclusionP1-10P2, P3, P9, P10 ⢠Instance-based maskingP2, P4, P9 P1, P2, P3, P4, P5, P6, P7, P8, P9, P10 ⢠RetractingP4, P9P1, P2, P3, P4, P5, P6, P7, P8, P9 ⢠FakingP2P2, P3, P4, P7, P8, P9, P10 ⢠Generalizing-P3, P9 ⢠Using built-in privacy controls (4.3.2) ⢠Opting out of sharing contentP4, P9P4 ⢠Disabling memory-P2, P7 unauthorized parties to emphasize the importance of pro- tecting sensitive details. Before the study, five students (P2, P3, P4, P9, P10) articulated such concerns. For example, P9 shared worries that sensitive information could be accessed through targeted prompts, stating: âSince [sensitive details] are forever stored in the database, any other user interact- ing with ChatGPT could ask targeted questions to it to mali- ciously get that information. The chatbot not understanding the sensitive nature of this information might also provide it to the user.â Similarly, P2 raised concerns about unauthorized access through compromised accounts, noting: â[...] even if the chats are encrypted, if my ChatGPT login credentials are lost, somebody can access it and misuse it.â After the study, six students (P2, P3, P4, P8, P9, P10) described sim- ilar concerns. For instance, P4 discussed the importance of protections when opting in to data sharing for model training: âIf youâre opting for data sharing, itâs important to protect your sensitive information since it can be used for training and an attacker can prompt the model to share your info.â 4.2.3 Potential Consequences of Data Misuse Students emphasized the importance of protecting informa- tion because of the potentials for misuse and its serious con- sequences. Pre-test responses from four students (P1, P3, P6, P10) reflected such concerns. For example, P3 described ex- amples of such potential consequences, stating: âThat kind of information can be used for financial fraud, identity theft, and plagiarism (it already is and is scrutinized heavily in the AI ethics sphere). It can enable cybercrime and make it more prominent by giving easy access to nefarious par- ties.â Similarly, P1 emphasized the implications that could arise from behavioral analysis using sensitive information, saying: âThinking about it more, I do see how this kind of data could be a goldmine for behavioral analysis. These are not just search terms, they are complex conversations that reveal thought patterns, emotions, and personal struggles. That kind of information, if misused, could have serious implications.â In post-test responses, two students (P7, P10) discussed this themeâe.g., P7 elaborated on additional risks: âWith the amount of information a chatbot can collect about a person, lots of metadata and user preferences can be accessed, lead- ing to better targeted ads or even possibly surveillance by governmental figures if the data is sold or released to others.â 4.2.4 Corporate Distrust and Lack of Transparency Students find it important to protect sensitive information due to a lack of trust in chatbot owners [8]. They attributed this distrust to factors including ambiguity around data handling practices and past privacy violations. Before the study, three students (P2, P5, P8) expressed such perspectives. For ex- ample, P8 raised concerns about uncertainty in how data is handled, stating: âI donât personally know exactly where that information goes and how transparent OpenAI is about that,â and P5 highlighted distrust in large corporations and chatbot owners, noting: âAlthough it may be secure (as they say), it feels wrong to give my personal information to a big corpora- 8 tion. Many of which consistently break good privacy practices and sell information to third parties. This is more concerning when realizing that most chatbots are offered for free, which means they have to monetize some other way, either with the paid version or selling information.â After the study, four students (P3, P4, P6, P7) expressed similar concerns, none of whom had articulated this view prior to the study. For instance, P6 voiced concern that chatbot owners may prioritize profit over usersâ privacy, stating: âI donât believe we should be giving these models our information to allow them to become better [...]. We should always avoid giving information to corporations as they never have our best interests at heart.â 4.2.5 Self-Disclosure Needs Two students (P1, P7) emphasized the need for greater caution in protecting sensitive information during CA use, expressing concerns about a tendency (either in themselves or others) to disclose highly sensitive information to CAs that they would not typically share with friends or family, attributing this ten- dency to needs for self-disclosure (see below). None of the participants discussed this theme in their post-test responses. P7: âWhen Iâm feeling overwhelmed with CS/career stuff, I tend to type a long brain dump to feel like Iâm telling someone (even if itâs just ChatGPT). Iâve shared insecurities and fears with it thatâd be hard for me to even share with a close friend.â P1: âIâve noticed that people, including some of my friends and family, tend to overshare very personal things with chatbots, things they probably wouldnât say out loud or share with a friend. Sometimes they treat it like a therapist or use it to organize their thoughts. I donât usually do that myself, not because Iâm consciously guarding my privacy, but because Iâve never felt the need to use it that way.â 4.3What students think they can do to protect sensitive info during ChatGPT use (RQ1.3) 4.3.1 Withholding or Anonymizing Sensitive Info This theme covers approaches for protecting sensitive infor- mation before submitting them to the chatbot. Within this theme, we identified the following two sub-themes. Information Exclusion.This strategy involves withholding entire segments of data that users deem sensitive. Before the study, all participants mentioned this strategyâe.g., P8 stated: âKind of obvious, but donât put it in the chat! Beyond that, Iâm not super sure what can be done [...]â, similarly, P10 noted: âI think the best way is not to share any sensitive info when we are interacting with LLMs.â After the study, four students (P2, P3, P9, P10) described this strategy again. Instance-Based Anonymization. This strategy involves masking instances of sensitive information before submit- ting them to the chatbot and includes the Retracting, Faking, and Generalizing approaches. Before the study, only three students (P2, P4, P9) described instance-based anonymiza- tion: P4 and P9 mentioned the Retracting strategy, while P2 described Fakingâe.g., P9 noted: â[...] replace that info with a placeholder so that ChatGPT does not have the exact info.â After the study, all students discussed instance-based anonymization approaches. Specifically, all except P10 de- scribed Retracting; P2, P3, P4, P7, P8, P9, P10 discussed Faking; and P3 and P9 described Generalizing. For exam- ple, P2 emphasized Retracting and Faking: âI can retract the sensitive info or fake the data instead of providing actual sensitive info that belongs to peopleâ, and P3 highlighted the Faking and Generalizing strategies: âWe can provide fake info/generalized info that doesnât pinpoint to an individual.â 4.3.2 Using ChatGPTâs Built-in Privacy Controls This theme captures the use of features or settings built into ChatGPT to help protect sensitive information after submis- sion. Before the study, three students mentioned using tempo- rary chats (P1, P4, P9), two mentioned removing chat histo- ries (P1, P5), and two mentioned opting out of sharing content for model training (P4, P9). After the study (focusing only on approaches supported by our simulated chatbot), two students (P2, P7) mentioned disabling the chatbotâs memory, and one student (P4) mentioned opting out of content sharing. 4.4 Studentsâ Reflections on Changes in Their Privacy Thinking 4.4.1 Desire for User-Facing Privacy Tools in CAs This theme captures participantsâ interest in user-facing tools or features that support them in protecting sensitive infor- mation during chatbot interactions (P1, P2, P3, P4, P5, P6, P8, P9). For example, P6 emphasized that such features can help balance privacy with chatbot usefulness, noting: âBy having a UI feature that identifies and hides all our sensitive information given our input, itâs super useful to make sure the model is useful while still protecting ourselves.â Similarly, P5 discussed the practicality of such tools for their own chatbot use, particularly if they are chatbot-agnostic: âI think it was interesting to have a tool dedicated to censoring sensitive in- formation. I think if it existed in real life, I would be inclined to use it. However, it would have to either be chatbot-agnostic, meaning that it could be used with any chatbot, or it would have to be a chatbot I already use.â 4.4.2 Growing Privacy Awareness and Thinking Participants explained that participating in the study and in- teracting with the privacy panel prompted them to think more 9 about privacy and become more cautious about the data they submit to chatbots (P2, P3, P5, P7, P8, P9, P10). For example, P2 reflected on how their disclosure behavior may be changed after the study: âIâve used ChatGPT to write replies for emails or summarize different things before the study as well without really checking if it contains any private information. But after the study, I have started to think more before sending my prompts and removing or faking such data wherever possi- ble/necessary.â Similarly, P5 described how interacting with the privacy panel during the task sessions made them more aware of how much data they had been sharing with chatbots: â[...] it has opened my eyes to how much data I was feeding the chatbots without knowing it. I had never really thought of it before, but when I actually was tasked with censoring that data, I realized just how much of it I mightâve been leaking.â 4.5 Interface Design Features That Supported or Hindered Protection of Sensitive Infor- mation (RQ2) 4.5.1Intercepting Chatbot Interactions After Each Sub- mission of Sensitive Information We designed the privacy panel to automatically intercept in- teractions immediately after users attempt to send a message containing sensitive information and before it is sent to the API. Participants generally found this interception helpful, as it made the panel easy to access, intuitive to use, and mini- mally disruptive to their workflow (P2, P3, P4, P5, P6, P7, P9). For example, P2 shared that using the panel was easy because: âit automatically popped up every time there was any personal information in the prompt so it was intuitive and easy to access.â P6 and P9 emphasized the benefit of this design on raising awareness of sensitive content in their messages at moments when they could make context-based decisions (e.g., based on the given task) about how to handle the information. P7 described the benefit of integrating the panel into the existing workflow (e.g., avoiding navigation to other screens), stating: âIf this was a feature I had to go out of my way to interact with, I probably wouldnât think about what private information I might be sharing, and would just send the message straight away.â P7 also suggested improvements for situations in which users resize their browser window or use a split-screen view, noting that the panel may require au- tomatic resizing to enable simultaneous use of both the panel and the chatbot. Despite these benefits, P5 and P9 found the repeated interceptions sometimes tedious, especially when they did not consider the flagged information as truly sensi- tive, with P5 saying: â[...] there were a few things that made it more annoying, like how it popped up every time, even though [sharing] names arenât as much of a privacy risk personally.â 4.5.2Interface Terminology and Features for Under- standing the Panel The panel used Anonymizing as an umbrella term for three strategies labeled as Retracting, Faking, and Generalizing (Figure 1). Four participants (P4, P6, P7, P9) faced difficulty in distinguishing how the options work just by their labels, as P6 noted: âI wasnât sure about the exact differences be- tween anonymizing vs. retracting vs. generalizing vs. faking.â Among them, three (P4, P7, P9) shared that the panelâs locate and highlight feature helped understand how each strategy worked. P7 explains this: âThe feature to jump to the part of the text being anonymized/faked was very helpful [...] as it allowed me to follow what was being changed and how it was being changed according to which option I selected.â Despite its usefulness, the locate and highlight feature ap- peared to lack visibility as it was unused by four participants (P1, P2, P3, P6âexcluding P8 and P10 who used the panel minimally). P6, unaware of the featureâs presence, suggested adding such functionality to the panel: âMaybe an example being highlighted to show what was being edited in the text could be helpful?â We also noticed that three participants (P2, P6, P9) initially thought the panel automatically anonymized flagged information, indicating a need for clearer status sig- nifiers. For example, P2 misunderstood the âAnonymize allâ label and proposed renaming it to reduce confusion: â[...] I assumed itâs already done but later when I clicked it, I saw the options to fake or generalize or [retract] so, maybe like [using] âSelect an optionâ default value.â 4.5.3Quick and Low-Effort User Control Over Anonymization Participants frequently appreciated features of the panel that supported quick and low-effort anonymization of sensitive information (P1, P3, P5, P9), as P1 noted: âI liked the feature that allowed me to quickly retract private information like name, email, phone number.â They valued design elements that enabled their control over anonymization decisions, in- cluding the ability to use the panel optionally (P5), view all detected instances and apply anonymization actions individu- ally or in bulk (P2, P4, P6, P9), and track or restore changes as needed (P4, P9). For example, P4 shared: âI thought it was very helpful to show every instance of PII and adding an option to anonymize all, as well as restore all.â Some partic- ipants also suggested ways to further speed up the process, such as saving preferred anonymization actions for future prompts (P5, P9) or adding a universal âAnonymize Allâ but- ton to the panel that applies to all types and instances of information (P9). These reflections highlight a core challenge in interface design: balancing user control with efficiency, es- pecially when aiming to support informed and context-based privacy actions with minimal effort. 10 5 Discussion and Future Work 5.1 In-Context Support and Experiential Pri- vacy Learning Although pre-test responses showed that participants had some understanding of sensitive information, recognized the importance of protecting it during CA use, and were aware of several protective strategies, 8 of 10 showed no evidence of considering sensitive information protection during the first task session (when using the chatbot without the panel) and disclosed most embedded sensitive information. This pat- tern reflects the privacy paradox [22], where usersâ stated privacy attitudes do not align with their actual behaviors. In contrast, during the second task session (using the chatbot with the panel), participants began actively reasoning about what to protect and how, withholding or anonymizing most sensitive information before submission. The differences be- tween participant behaviors across sessions suggests that knowledge (e.g., what is sensitive, protective actions) and motivation (e.g., valuing privacy protection) may not be suffi- cient by themselves to lead users to taking protective actions. Instead, design features that facilitate in-context protective actions appear to encourage greater engagement with privacy- preserving practices. These elements (knowledge, incentives, and in-context action) are actually closely related to experien- tial learning, or learning by doing [16, 23] (see also models of sensemaking and decision-making [14,41]) Learning involves interpreting new information through the lens of prior knowl- edge [4], and experiential learning emphasizes the importance of exposure to new content, incentives for engagement, and opportunities for interaction (action and reflection) [16]. Be- low, we use this lens to interpret the potential impact of our panel components on changes in participantsâ perspectives from pre- to post-test responses. Anonymization Component (Figure 1, B): In the second task session, most participants engaged with the anonymiza- tion component, and post-study responses suggest that they learned about instance-based masking strategies (e.g., retract, fake). This component introduced potentially new strategies, supported users in applying them, and may have prompted re- flection on their effectsâe.g., by considering changes in chat- bot response quality. These supports were embedded within ongoing interactions, appearing right after each prompt sub- mission, allowing participants to consider contextual factors (e.g., tasks and goals) when making decisions. Integrating these features into the interaction flow may have facilitated learning about these privacy-protective strategies. FAQ Component (Figure 1, D): This component provided content related to conceptualizing sensitive information and the importance of privacy protection. During the second task session, only two participants opened the FAQ panel, and only one read parts of it. Not surprisingly, we observed minimal changes in participantsâ perspectives on sensitive information and the importance of privacy protection in the post-study responses. One possible explanation is that participants may have perceived the FAQs as static, text-heavy resources that were less directly integrated into or actionable within their immediate interactions. Future work could explore how in- formational components can be more tightly embedded in interaction flows and designed to support action. For example, the anonymization component (Figure 1, B) could include brief contextual prompts explaining why flagged information may be sensitive and why protecting it matters at that moment. 5.2UI/UX Design Decisions for Engagement and Learning in User-Facing Privacy Tools Interface Terminology and Language: In our study, al- though some participants found it difficult to distinguish be- tween the terms used for different instance-based masking strategies (i.e., anonymize, retract, fake, generalize), they were able to understand how each strategy worked through inter- action with the panelâby applying a strategy, locating, high- lighting, and undoing changes (Section 4.5.2). This suggests that interactive features can help compensate for unfamiliar or complex terminology. An alternative approach is to incor- porate contextual on-boarding techniques [33]âe.g., brief tooltip explanations shown on hover, or in-context pop-ups during initial use, could clarify new terms and features. Such approaches may support gradual, in-context learning and en- courage engagement. Regarding the FAQ component (Fig- ure 1, D), the limited engagement we observed may relate to how the label âFAQâ was interpreted. It may have signaled instructions about using the panel rather than content about privacy, as P2 noted: âI think the other features [of the panel] were pretty self-explanatory so, I didnât feel the need to read the details in the FAQs.â Prioritizing Features and Content: User engagement with interface elements can be influenced by how they are prioritized within the design. Users typically scan panels from top to bottom [7], and our participants followed this pattern: all read the warning message first, almost all engaged with the anonymization panel, some reviewed built-in privacy con- trols, and only two opened the FAQs (Figure 1). This ordering likely contributed to limited engagement with FAQs, espe- cially given usersâ time and effort constraints. Engagement may also depend on whether content appears in the primary interaction layer or is nested in secondary layers (e.g., addi- tional screens). Our participants appreciated that the panel was embedded within their workflow without requiring navi- gation to another page (Section 4.5.1); however, accessing the FAQs required opening a secondary pop-up, which may have further discouraged interaction. Future work can therefore ex- amine how to prioritize and surface privacy tool components, placing higher-priority features in the primary interaction layer and reducing reliance on secondary screens or nested content (e.g., through progressive disclosure [18, 32]). 11 5.3 Privacy Tradeoffs in CA Use In CA use, users weigh the costs and benefits of their actions against their goals. Advancing one objective may require com- promising another. Recognizing these trade-offs is essential for supporting users in balancing privacy with other chatbot use objectives. Below, we discuss two such trade-offs. Privacy Protection versus Chatbot Utility: Users often weigh whether (and how much) to withhold information to protect privacy while still providing sufficient context for high- quality chatbot responses (utility) [3, 43, 44]. How users navi- gate this tradeoff may depend on the perceived importance of each objective, their knowledge of potential actions, and the support available to carry them out. Without privacy features, users may prioritize chatbot utility over privacy, as observed in our first task session. Pre-test responses further suggest that participants were largely unaware of privacy-protective ac- tions beyond fully excluding segments of information (Table 2). However, such exclusion may be perceived as substan- tially reducing response quality and therefore avoided. When the panel introduced and supported instance-based masking strategies (which may have less impact on response quality) participants adopted these strategies. Nevertheless, some con- tinued to share sensitive details flagged by the panel, often citing their importance for the chatbot to generate high-quality responses. This suggests that future work should explore tools that better support users in balancing privacy and chatbot utilityâe.g., tools that can analyze the ongoing task (e.g., drafting emails) and provide guidance on what to share or withhold to preserve privacy while maintaining utility. Privacy Protection versus Convenience: UI/UX research has long emphasized the importance of efficiency (minimizing usersâ time and effort) in shaping tool adoption and engage- ment [11, 34]. This principle also applies to privacy tools, where convenience becomes an additional objective alongside privacy and chatbot utility [43]. Although our panel facili- tated protective actions (by detecting sensitive info, providing just-in-time awareness, and bulk anonymization), participants still expressed interest in features that would further reduce time and effort (Section 4.5.3). One direction is exploring au- tomation in anonymization based on user preferencesâe.g., automatically applying specific masking strategies to certain types of information. In our panel, shortcuts to built-in pri- vacy controls appeared each time the panel was displayed (Figure 1, C). While initially surfacing these controls pro- motes awareness, repeatedly displaying static elements may be unnecessary, as we observed some participants reopened the settings panel across multiple appearances simply to con- firm that nothing had changed (e.g., that the opt-out toggle remained disabled). Components that do not dynamically change (unlike the anonymization component; Figure 1, B) can therefore be minimized or hidden in later appearances to reduce redundancy and effort. 5.4 Learning Opportunities for Students Our findings on studentsâ perceptions of privacy in CAs (Sec- tions 4.1, 4.2, and 4.3) can inform opportunities to update studentsâ understanding of privacy in light of evolving AI capabilities. For example, although identifiable information is widely recognized as sensitive [5, 37], only five participants explicitly articulated this in their responses. Moreover, this un- derstanding may require further development, as modern AI tools (including generative CAs) can intensify identifiability risks through rapid, real-time aggregation of data from mul- tiple sources [26]. Similarly, usersâ views on the importance of privacy protection may benefit from greater awareness of risks introduced or exacerbated by modern AI systems (e.g., surveillance, intrusion, phrenology) [26]. Standalone educa- tional tools could also support deeper examination of usersâ privacy thinking, increasing awareness of past privacy inci- dents, and learning about protective behaviors [1, 2, 13, 15]. 6 Limitations We focused on sensitive information definition, the impor- tance of protecting privacy during CA use, and protective actions. Other relevant dimensions (e.g., understanding of CA privacy policies, awareness of past privacy incidents) were be- yond the scope of this study. Future work could examine these and explore how they may be enhanced through interactions with privacy tools. Our participants were CS undergraduate and masterâs students in the US. While this focus enabled a more controlled examination within a specific population, we do not intend to generalize our findings to other user groups (e.g., different ages or geographic contexts). Future research can replicate and extend this work with broader and more diverse populations to capture a wider range of privacy per- spectives and design needs. Finally, we examined short-term interactions with our privacy panel. Longitudinal studies are needed to assess the long-term effects of engaging with such tools during everyday CA use and to understand how repeated exposure may shape usersâ privacy perceptions over time. 7 Conclusion We examined how exposure to our just-in-time privacy no- tice panel during realistic chatbot use influenced participantsâ perceptions of privacy in CAs. We also analyzed the UI/UX design features of the panel that supported or hindered user- led protection of sensitive information during chatbot use. Our findings suggest that user-facing privacy tools have po- tentials in gradually and contextually encouraging users to engage with, reflect on, and learn about privacy during CA interactions. We further highlighted the critical role of UI/UX design in facilitating such engagement and the importance of accounting for usersâ privacy tradeoffs when designing privacy tools for CAs. 12 Acknowledgments References [1] AIAAIC. Lee Luda AI chatbot spouts offensive re- sponses. Online; AIAAIC Repository: AI, Algorithmic and Automation Incidents and Controversies, 2021. Ac- cessed: 2026-01-13. [2]AIAAIC. Google search indexes bard personal chats. Online; AIAAIC Repository: AI, Algorithmic and Au- tomation Incidents and Controversies, 2023. Accessed: 2026-01-13. [3] Mutahar Ali, Arjun Arunasalam, and Habiba Farrukh. Understanding usersâ security and privacy concerns and attitudes towards conversational AI platforms. In 2025 IEEE Symposium on Security and Privacy (SP), pages 298â316. IEEE, 2025. [4] Susan A Ambrose, Michael W Bridges, Michele DiP- ietro, Marsha C Lovett, and Marie K Norman. How learning works: Seven research-based principles for smart teaching. John Wiley & Sons, 2010. [5]Rahime Belen-Saglam, Jason R. C. Nurse, and Duncan Hodges. An investigation into the sensitivity of personal information and implications for disclosure: A UK per- spective. Frontiers in Computer Science, Volume 4 - 2022, 2022. [6]Virginia Braun and Victoria Clarke. Using thematic anal- ysis in psychology. Qualitative Research in Psychology, 3(2):77â101, 2006. [7]Michael D. Byrne. ACT-R/PM and menu selection: Applying a cognitive architecture to HCI. Interna- tional Journal of Human-Computer Studies, 55(1):41â 84, 2001. [8]Francisco Enrique Vicente Castro. Case studies on responsible computing: Speculative thinking and con- siderations in technology ecosystems. In Proceedings of the 26th ACM Annual Conference on Cybersecurity & Information Technology Education, SIGCITE â25, page 207â212. ACM, 2025. [9]Chaoran Chen, Daodao Zhou, Yanfang Ye, Toby Jia-Jun Li, and Yaxing Yao. Clear: Towards contextual llm- empowered privacy policy analysis and risk generation for large language model applications. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI â25, page 277â297. ACM, 2025. [10] Chun Jie Chong, Chenxi Hou, Zhihao Yao, and Seyed Mohammadjavad Seyed Talebi. Casper: Prompt sani- tization for protecting user privacy in web-based large language models. arXiv preprint arXiv:2408.07004, 2024. [11]Fred D Davis. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 13(3):319â340, 1989. [12]Emma EngstrĂśm, Kimmo Eriksson, Marie BjĂśrnstjerna, and Pontus Strimling. Global variations in online pri- vacy concerns across 57 countries. Computers in Human Behavior Reports, 9:100268, 2023. [13]Michael Feffer, Nikolas Martelaro, and Hoda Heidari. The AI incident database as an educational tool to raise awareness of AI harms. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mech- anisms, and Optimization, 2023. [14]Mohammad Hadi Nezhad, Francisco Castro, Beverly Woolf, and Ivon Arroyo. Math teachersâ in-class infor- mation needs and usage for effective design of class- room orchestration tools. In European Conference on Technology Enhanced Learning. Springer, 2024. [15] Mohammad Hadi Nezhad, Francisco Enrique Vicente Castro, Eugene Mak, Peter J Haas, Danielle Allessio, Leon Osterweil, Injila Rasul, Heather Conboy, and Ivon Arroyo. Embedding ethical awareness in computer science and AI education: The PEaRCE approach to responsible computing. In International Conference on Artificial Intelligence in Education, pages 135â149. Springer, 2025. [16]Knud Illeris. What do we actually mean by experien- tial learning? Human Resource Development Review, 6(1):84â95, 2007. [17]Jane Im, Ruiyi Wang, Weikun Lyu, Nick Cook, Hana Habib, Lorrie Faith Cranor, Nikola Banovic, and Florian Schaub. Less is not more: Improving findability and actionability of privacy controls for online behavioral advertising. In Proceedings of the 2023 CHI Confer- ence on Human Factors in Computing Systems, CHI â23. ACM, 2023. [18] IxDF. Progressive disclosure. Accessed: 2026-02-19. [19]Eric J Johnson, Suzanne B Shu, Benedict GC Dellaert, Craig Fox, Daniel G Goldstein, Gerald Häubl, Richard P Larrick, John W Payne, Ellen Peters, David Schkade, et al. Beyond nudges: Tools of a choice architecture. Marketing letters, 23(2):487â504, 2012. [20]Patrick Gage Kelley, Celestina Cornejo, Lisa Hayes, El- lie Shuo Jin, Aaron Sedley, Kurt Thomas, Yongwei Yang, and Allison Woodruff. There will be less privacy, of course: How and why people in 10 countries expect AI will affect privacy in the future. In SOUPS 2023, pages 579â603, 2023. 13 [21]Patrick Gage Kelley, Lorrie Faith Cranor, and Norman Sadeh. Privacy as part of the app decision-making pro- cess. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI â13, page 3393â3402, New York, NY, USA, 2013. Association for Computing Machinery. [22] Spyros Kokolakis. Privacy attitudes and privacy be- haviour: A review of current research on the privacy paradox phenomenon. Computers and Security, 64:122â 134, 2017. [23] David A Kolb. Experiential learning: Experience as the source of learning and development. FT press, 2014. [24]James Lappeman, Siddeeqah Marlie, Tamryn Johnson, and Sloane Poggenpoel. Trust and digital privacy: Will- ingness to disclose personal information to banking chat- bot services. Journal of Financial Services Marketing, 28(2), 2022. [25]Josephine Lau, Benjamin Zimmerman, and Florian Schaub. Alexa, are you listening? privacy perceptions, concerns and privacy-seeking behaviors with smart speakers. Proceedings of the ACM on human-computer interaction, 2(CSCW):1â31, 2018. [26]Hao-Ping (Hank) Lee, Yu-Ju Yang, Thomas Serban Von Davier, Jodi Forlizzi, and Sauvik Das. Deepfakes, phrenology, surveillance, and more! A taxonomy of AI privacy risks. In Proceedings of the 2024 CHI Confer- ence on Human Factors in Computing Systems, CHI â24. ACM, 2024. [27]Yao Li, Alfred Kobsa, Bart P Knijnenburg, and MH Car- olyn Nguyen. Cross-cultural privacy prediction. Pro- ceedings on Privacy Enhancing Technologies, 2017. [28] Lisa Mekioussa Malki et al. âHoovered up as a data pointâ: Exploring privacy behaviours, awareness, and concerns among UK users of LLM-based conversational agents. In Proceedings on Privacy Enhancing Technolo- gies. ACM, 2025. [29] Nuno Martins and Daniel BrandĂŁo. Advances in De- sign and Digital Communication I: Proceedings of the 5th International Conference on Design and Digital Communication, Digicom 2021, November 4â6, 2021, Barcelos, Portugal, volume 19. Springer Nature, 2021. [30]Niloofar Mireshghallah, Maria Antoniak, Yash More, Yejin Choi, and Golnoosh Farnadi. Trust no bot: Discov- ering personal disclosures in human-LLM conversations in the wild. arXiv preprint arXiv:2407.11438, 2024. [31]Mohammad Hadi Nezhad, Francisco Enrique Vicente Castro, and Ivon Arroyo. Understanding usersâ pri- vacy reasoning and behaviors during chatbot use to support meaningful agency in privacy. arXiv preprint arXiv:2601.18125, 2026. [32]Nielsen Norman Group. Progressive disclosure, 2006. Accessed: 2026-02-19. [33] Nielsen Norman Group. Onboarding tutorials vs. con- textual help.https://w.nngroup.com/articles/ onboarding-tutorials/, 2023. Accessed: 2026-02- 18. [34]Nielsen Norman Group. 10 usability heuristics for user interface design, 2024. Accessed: 2026-02-19. [35] Helen Nissenbaum. Privacy as contextual integrity. Washington Law Review, 79, 2004. [36]Florian Schaub, Bastian KĂśnings, and Michael Weber. Context-adaptive privacy: Leveraging context awareness to support privacy decision making. IEEE Pervasive Computing, 14(1):34â43, 2015. [37]Qiurong Song, Yanlai Wu, Rie Helene (Lindy) Hernan- dez, Yao Li, Yubo Kou, and Xinning Gui. Understanding usersâ perception of personally identifiable information. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI â25. ACM, 2025. [38] Christian StĂśhr, Amy Wanyu Ou, and Hans MalmstrĂśm. Perceptions and usage of AI chatbots among students in higher education across genders, academic levels and fields of study. Computers and Education: Artificial Intelligence, 7:100259, 2024. [39]Richard H Thaler and Cass R Sunstein. Nudge: Im- proving decisions about health, wealth, and happiness. Penguin, 2009. [40]Sigal Tifferet. Gender differences in privacy tendencies on social network sites: A meta-analysis. Computers in Human Behavior, 93:1â12, 2019. [41] Alyssa Friend Wise and Yeonji Jung. Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 2019. [42] Xing Zhang, Zhaoqian Li, Mingyang Zhang, Mingyue Yin, Zhangyu Yang, Dong Gao, and Hansen Li. Explor- ing AI chatbot usage behaviors and their association with mental health outcomes in chinese university stu- dents. Journal of Affective Disorders, 380:394â400, 2025. [43] Zhiping Zhang, Michelle Jia, Hao-Ping (Hank) Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. âItâs a fair gameâ, or is it? Examining how users navigate disclosure risks and benefits when using LLM-based conversational agents. In Proceed- ings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI â24. ACM, 2024. 14 [44]Jijie Zhou, Eryue Xu, Yaoyao Wu, and Tianshi Li. Re- scriber: Smaller-LLM-powered user-led data minimiza- tion for LLM-based chatbots. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI â25. ACM, 2025. A Supplementary Methodological Materials A.1 ChatGPT Interface Simulation Figure 2 shows the simulated settings panel for built-in pri- vacy controls: (left) disabling memory and (right) opting out of sharing content for model training. These panels were avail- able in both task sessions. A.2 User Task Details Table 3 shows detailed description of user tasks and the em- bedded sensitive information including their types, frequen- cies, and subjectsâi.e., who the information pertains or be- longs to. 1 Drawn from the Enron Email Corpus and modified as needed. 2 Drawn from a Ticket Classification Dataset and modified as needed. 15 Table 3: User Tasks. Columns (LâR): task assignment version (V), task description, types of sensitive info contained in each task (Sensitive Info), frequencyâof occurrencesâof each type (F), and Info Subjects (who the info pertains/belongs to). VTask DescriptionSensitive InfoFInfo Subject ATask 1: Email Analysis ⢠Step 1: Classifying seven emails 1 by their sentiments ⢠Step 2: Summarizing two separate email threads Name Email address Phone number Physical address 61 23 6 4 Others (e.g., employ- ees) ATask 2: Searching Housing Documents ⢠Step 1: Finding party restrictions from their housing contract (9-page doc) â˘Step 2: Finding maintenance information from the same hous- ing contract and a welcome letter (5-page doc) Name Email address Phone number Physical address SSN Date of birth 38 9 16 16 4 4 Self, Others(e.g., friend, home owner) ATask 3: Planning a Trip â˘Step 1: Finding timing conflicts between their travel itinerary (2-page doc) and semester schedule (1-page doc) â˘Step 2: Drafting an email to resolve the conflicts and provide their contact details for reservation purposes Name Email address Phone number Physical address Date of birth 5 1 3 7 2 Self, Others(e.g., friend) BTask 1: Customer Complaint Analysis â˘Step 1: Classifying seven customer complaints 2 by relevant departments â˘Step 2: Summarizing recommendations for the company from two separate groups of customer complaints Name Email address Phone number Physical address Date of birth 18 6 4 4 2 Others(e.g., cus- tomers) BTask 2: Searching Car & Housing Documents â˘Step 1: Finding costs and fees related to purchasing a leased car from their car lease contract (15-page doc) â˘Step 2: Finding information about utilities and responsibilities from their housing contract (9-page doc) Name Email address Phone number Physical address SSN Date of birth 23 4 10 12 4 4 Self, Others(e.g., friend, home owner) BTask 3: Planning Teeth Removal ⢠Step 1: Finding out-of-pocket expenses from their insurance document (6 pages) and an email from their doctor. â˘Step 2: Drafting an email to their doctor providing insurance details and asking if the insurance is accepted Name Email address Phone number Physical address Date of birth 8 2 2 1 5 Self, Others (e.g., doc- tor) 16 Figure 2: Simulated ChatGPT settings panels for built-in privacy controls: (left) Personalization panel for enabling or disabling memory, with a functional toggle and static buttons; (right) Data Control panel for opting in or out of content sharing for model training, with an interactive toggle. 17