Paper deep dive

Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy

Wooyoung Jung, Kahyun Jeon, Prosper Babon-Ayeng

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 129

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 7/21/2026, 2:12:53 AM

Summary

This study investigates the impact of user domain knowledge and AI literacy on human-AI collaboration within Large Language Model (LLM)-integrated Building Energy Management Systems (BEMS). Through a role-playing experiment with 85 participants interacting with OpenAI GPT-4o, the research analyzed how these factors influence interaction quality and energy-saving outcomes. Key findings indicate that while most users relied heavily on GPT's analytical capabilities with concise prompts, only the 'appliance identification rate' showed statistically significant differences driven by AI literacy rather than domain knowledge, suggesting LLMs may have an equalizing effect across expertise levels.

Entities (8)

OpenAI GPT-4o · software · 99%Building Energy Management System · system · 98%Large Language Model · technology · 97%AI Literacy · concept · 95%Domain Knowledge · concept · 95%Appliance Identification Rate · metric · 92%Kruskal-Wallis H test · method · 90%SCALE framework · framework · 88%

Relation Signals (5)

OpenAI GPT-4o → usedin → Building Energy Management System

confidence 95% · participants interacted with an advanced generative pre-trained transformer (OpenAI GPT-4o) ... that functioned as an LLM-integrated BEMS.

AI Literacy → influences → Appliance Identification Rate

confidence 90% · appliance identification rate ... showed statistically significant group differences ... driven by AI literacy rather than domain knowledge

LLMs → hasequalizingeffecton → expertise levels

confidence 88% · suggesting an equalizing effect of LLMs across expertise levels.

SCALE framework → analyzes → human-AI interactions

confidence 85% · analyzed using an analytical framework ... SCALE framework ... for analyzing the structure, depth, and quality of human-AI dialogues

Domain Knowledge → doesnotsignificantlyinfluence → Appliance Identification Rate

confidence 85% · driven by AI literacy rather than domain knowledge

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:This study aimed to comprehend how user domain knowledge and artificial intelligence (AI) literacy impact the effective use of human-AI interactive building energy management system (BEMS). While prior studies have investigated the potential of integrating large language models (LLMs) into BEMS or building energy modeling, very few studies have examined how user interact with such systems. We conducted a systematic role-playing experiment, where 85 human subjects interacted with an advanced generative pre-trained transformer (OpenAI GPT-4o). Participants were tasked with identifying the top five behavioral changes that could reduce home energy use with the GPT model that functioned as an LLM-integrated BEMS. Then, the collected prompt-response data and participant conclusions were analyzed using an analytical framework that hierarchically assessed and scored human-AI interactions and their home energy analysis approaches. Also, participants were classified into four groups based on their self-evaluated domain knowledge of building energy use and AI literacy, and Kruskal-Wallis H tests with post-hoc pairwise comparisons were conducted across 20 quantifiable metrics. Key takeaways include: most participants employed concise prompts (median: 16.2 words) and relied heavily on GPT's analytical capabilities; and notably, only 1 of 20 metrics, appliance identification rate, showed statistically significant group differences (p=0.037), driven by AI literacy rather than domain knowledge, suggesting an equalizing effect of LLMs across expertise levels. This study provides foundational insights into human-AI collaboration dynamics and promising development directions in the context of LLM-integrated BEMS and contributes to realizing human-centric LLM-integrated energy systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

127,944 characters extracted from source content.

Expand or collapse full text

1 Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy Wooyoung Jung 1,* Kahyun Jeon 2 Prosper Babon-Ayeng 1 1 The University of Arizona 2 Illinois Institute of Technology * Corresponding Author: wooyoung@arizona.edu Abstract This study aimed to comprehend how user domain knowledge and artificial intelligence (AI) literacy impact the effective use of human-AI interactive building energy management system (BEMS). While prior studies have investigated the potential of integrating large language models (LLMs) into BEMS or building energy modeling, very few studies have examined how user interact with such systems. We conducted a systematic role-playing experiment, where 85 human subjects interacted with an advanced generative pre-trained trans- former (OpenAI GPT-4o). Participants were tasked with identifying the top five behavioral changes that could reduce home energy use with the GPT model that functioned as an LLM-integrated BEMS. Then, the collected prompt-response data and participant conclusions were analyzed using an analytical framework that hierarchically assessed and scored human-AI interactions and their home energy analysis approaches. Also, participants were classified into four groups based on their self-evaluated domain knowledge of build- ing energy use and AI literacy, and Kruskal-Wallis H tests with post-hoc pairwise comparisons were con- ducted across 20 quantifiable metrics. Key takeaways include: most participants employed concise prompts (median: 16.2 words) and relied heavily on GPT’s analytical capabilities; and notably, only 1 of 20 metrics – appliance identification rate – showed statistically significant group differences (p=0.037), driven by AI literacy rather than domain knowledge, suggesting an equalizing effect of LLMs across expertise levels. This study provides foundational insights into human-AI collaboration dynamics and promising develop- ment directions in the context of LLM-integrated BEMS and contributes to realizing human-centric LLM- integrated energy systems. Keyword Building Energy Management System; Human-AI Interaction; Large Language Model; Domain Knowledge; AI Literacy. 1. Introduction Building energy use patterns are uniquely and stochastically shaped by occupants’ diverse interactions with appliances and building systems such as heating, ventilation, and air-conditioning (HVAC), lighting, and plumbing. Each occupant holds their own preferences and needs, which determine the operation, scheduling, and optimization of these systems – often balancing comfort, budgetary constraints and energy saving goals [1]. Despite this heterogeneity, at present, occupants receive limited feedback on their energy use. Monthly energy bills serve primarily as ‘financial consequences’ of their behavior, panelizing inefficient energy use without providing guidance or actionable insights for improvement. Even though the widespread installa- tion of smart meters has enabled utility-provided mobile apps to provide users hourly or daily tracking of aggregated energy consumption, achieving a representing progress in information accessibility (e.g., [2, 3]), users have limited opportunities to interaction with building energy management systems (BEMSs). Users are often confined to predefined software features that seldom offer interactive capabilities for knowledge acquisition or data analysis. For instance, occupants often hesitate to enroll in time-of-use (TOU) rate pro- grams – to encourage peak load reduction and load shifting through behavioral changes (e.g. [4-6]) – due to a lack of detailed, personalized information of how subscribing TOU rates may affect their specific en- ergy bills. In recent years, a few studies have proposed a large language model (LLM)-integrated BEMSs, which can address this long-lasting limitation [7-9]. These studies showed the potential of human-centered interactions 2 through their systems, where occupants can readily access and generate information from multiple home energy data sources, receive personalized recommendations for energy efficiency improvements or behav- ioral changes, and streamline control over appliances and building systems using natural language, as illus- trated in Figure 1 as Level #3. As their primary focus was to evaluate their systems’ feasibility, their exper- imental settings tested various user queries, covering a wide range of building energy management (BEM)- related tasks such as “How much energy did I use last month?” or “Set the temperature setpoint as 24°C”. The major limitation of these single-turn query-response trials is that they do not reveal how users actually collaborate with LLMs through iterative, multi-turn dialogues to build understanding and arrive at action- able energy-saving decisions. In practice, effective use of LLM-integrated BEMS requires users to interpret responses, refine their inquiries, evaluate suggested strategies against personal constraints, and synthesize information across multiple exchanges. Figure 1. Current and envisioned human building interactions in the context of BEMS We anticipate two primary challenges when these LLM-integrated BEMS are deployed in practice from the users’ perspective: (1) the complexity of building energy management, which involves multiple appliances and building systems (e.g., HVAC, lighting, plumbing) – each with distinct control mechanisms and engi- neering principles – operating within a system-of-systems structure. This complexity has further intensified in recent years, with utilities introducing new rate structures, such as TOU rates, to integrate renewable energy resources and promote demand response (DR) strategies; (2) the requirements for AI literacy, which extend beyond merely crafting well-structured prompts [10]. AI literacy comprises a set of competencies that enables users to critically evaluate AI technologies, communicate and collaborate effectively with AI systems, and utilize AI as a tool to achieve specific goals [11]. These competencies include providing suf- ficient detail, offering contextual framing (e.g., assigning GPT an appropriate expert role), and engaging in iterative questioning and reflection to improve the relevance, clarity, and personalization of the interaction. Accordingly, without adequately addressing these challenges, households may struggle to harness the full potential of LLM-integrated BEMS to access, interpret, and act on critical building energy management (BEM) information or may encounter system responses that are too generic, misaligned with household needs, or insufficiently contextualized to support informed energy decisions. Prior research in broader human-AI interaction contexts suggests that these two factors can significantly shape the quality of collaboration. Studies on AI-assisted decision-making have reported that users’ domain knowledge influences how they interpret, evaluate, and calibrate their reliance on AI-generated recommen- dations [12]. In addition, research on AI literacy has indicated that users’ ability to articulate effective prompts, engage in iterative dialogue, and critically evaluate AI outputs can affect the outcomes of human- AI collaboration [13]. However, whether and how these factors operate in the context of LLM-integrated BEMS, where users engage in multi-turn, goal-oriented conversations involving complex, domain-specific energy data, remains an open question. Consequently, this study aimed to examine human-AI interactions in the context of LLM-integrated BEMS. Data-infused homes (with smart home devices) Regular homes Ongoing transition Level #1: Manual interactions Level #2: Limited interactions through given features Level #3: Envisioned human-AI interactions User Level #1: Minimal interactions (e.g., monthly energy bills) Regular Home Data-Rich Home Level #2: Limited interactions via predefined program features (e.g., smart thermostats) Level #3 ( Envisioned ): Dynamic, user-responsive interactions tailored to individual needs and preferences (e.g., AI-integrated energy systems) Ongoing transition 3 Our research question was developed as follows: - Which characteristics – particularly domain knowledge and AI literacy – enable households to fully benefit from LLM-enhanced building energy management? To accomplish the research goal and address the research question, we designed a role-playing experiment in which participants were tasked with identifying five feasible behavioral changes to reduce household energy bills through interactions with an advanced generative pretrained transformer (GPT) model, with no time constraints (see Section 3 for more details). Our hypotheses were as follows: H1. Participants with limited domain knowledge may rely heavily on the LLM’s responses, potentially resulting in brief interactions. H2. Participants with strong domain knowledge may engage in extended interactions with the tool to validate LLM’s suggestions, ultimately arriving at reasonably accurate conclusions. H3. Participants paired with high AI literacy would compensate their limited domain knowledge by effectively interacting with GPT to extract relevant insights, formulate well-structured prompts, and critically assess AI-generated recommendations. H4. Excessively verbose GPT responses may negatively affect user agency and limit iterative dialogue, especially in participants with lower AI literacy. H5. Participants would perceive this LLM-based BEMS as useful and even educational, providing an accessible and engaging opportunity to enhance their understanding of home energy systems while simultaneously identifying feasible energy-saving strategies. Using a prescreening survey, we utilized a stratified sampling approach to recruit participants from different user groups based on their domain knowledge and AI literacy. A total of 85 participants conducted the role- playing experiment and then participated in a post-experiment survey. We then analyzed the interactions of 85 participants with an LLM tool – specifically, ChatGPT-4o by OpenAI – to investigate the relationships among domain knowledge, AI literacy, and interaction outcomes. The key contributions of this study are threefold: (1) a systematic empirical investigation of how user do- main knowledge and AI literacy shape human-AI interaction in LLM-integrated BEMS, using a 2×2 facto- rial design with 85 participants and non-parametric inferential statistical analyses; (2) the introduction of the SCALE framework and two complementary metric sets (interaction volume and conversational reason- ing) for analyzing the structure, depth, and quality of human-AI dialogues; and (3) evidence-based design implications for developing more effective LLM-integrated energy management platforms, including scaf- folded interaction designs, interpretive support features, and adaptive output mechanisms. 2. Literature Review 2.1. Use of Language Models in Home/Building Energy Management Pre-LLM. Despite several decades of natural language processing (NLP) development, its application in BEM remains limited, particularly for text data processing and analysis. One common use case is building code compliance checking. Fuchs [14] conducted a comprehensive and systematic review of this domain, covering 42 research articles published between 2000 and 2020. NLP was utilized to process building code documents, classify texts, extract key features and information. These outputs were integrated with model data extracted from building information models to assess compliances. In a similar context, Khanuja [15] used NLP to process energy efficiency measures data – often provided in non-standardized text formats – to ensure consistency and standardization. Lai et al. [16] and Hong et al. [17] used fundamental NLP tech- niques such as tokenization, stop word removal, and lemmatization to extract relevant information, includ- ing energy conservation measures. Olanrewaju et al. [18] applied NLP-based document similarity tech- niques – such as cosine similarity and term frequency-inverse document frequency (TF-IDF) [19] – to eval- uate technical manuals associated with green building certification systems, such as Leadership in Energy and Environmental Design). 4 The limited application in NLP have not significantly enhanced user interactions in BEMS, mainly enabling basic-level communication features. For example, Kim et al. [20] developed a task-oriented voice user interface (VUI), facilitating the interaction between users and their cloud-based eco-feedback and gaming platform called MySmartE. This VUI utilized Amazon’s Alexa Skills Kit, which enables developers to cre- ate conversational interfaces for appliances. Users could control their thermostats and review their energy savings using the VUI. However, as noted by the authors, the VUI was less frequently used than the visual user interface and was mainly employed for simple control tasks or information queries. Kumar et al. [21] presented an approach that allowed users to monitor indoor environmental quality (e.g., air temperature, humidity) through voice-activated controls. However, their system’s interaction remained unidirectional, limited to command execution, without supporting conversational exchanges through which users could inquire about their energy use patterns or receive contextual feedback. Post-LLM. Since the public release of LLM tools in early 2020s, a growing number of studies have applied these technologies to the field of BEMS. To date, the primary focus of these studies has been on feasibility assessment – evaluating whether LLMs possessed sufficient domain knowledge for BEMS (e.g., [22]) and demonstrating their potential in specific use cases such as automated energy waste detection [23], urban energy modeling [24], fault diagnosis [25-27], and energy management optimization [28] – or on technical integration strategies such as domain-specific fine-tuning and retrieval augmented generation (RAG) [29, 30]. These works have largely centered on system capabilities, including the types of questions LLMs can learn and answer, the accuracy of their energy-related outputs, and general recommendations for how such tools might be deployed in future smart home and building contexts. This trend persisted in the studies that focused on user aspects of LLM integration into BEMS. Specifically, He and Jazizadeh [7], [8] employed a pre-built platform, OpenAI Assistants Application Programming In- terface, to build their agentic system to incorporate tools (code interpreter, knowledge retrieval, and func- tion calling) and to assess the feasibility of enhancing building energy efficiency and management through LLM-integrated BEMS. They evaluated 95 queries involving information requests, analysis and sugges- tions, and connections to external devices utilizing various prompt types. Rey-Jouanchicot et al. [9] pro- posed an LLM-based smart home architecture that leveraged user preferences to proactively select device actions in response to home events, such as adjusting lighting when a user gets out of bed. Their system was evaluated on 11 predefined single-turn scenarios, where each home state triggered a single LLM-gen- erated action. In contrast, this study shifts the focus from system feasibility to the human and user side of LLM inte- gration into BEMS. While prior work has established that LLMs possess sufficient analytical capabilities for building energy applications, which demonstrates system-level competence does not ensure that users will effectively leverage these capabilities in practice. The quality of human-AI collaboration ultimately depends on how users interact with, interpret, and act upon AI-generated outputs. Rather than evaluating what LLMs can do, this research examines how actual users with varying backgrounds interact with a GenAI tool to explore BEM. Through a systematic human subject experiment and a multi-layered analysis of participant-GPT interactions, this study aims to provide insight into the practical dynamics of human-AI collaboration in the context of BEMS. 2.2. Domain Knowledge and AI Literacy in Human-AI Collaboration As noted in the Introduction, prior research in broader human-AI interaction contexts has identified domain knowledge and AI literacy as factors that can shape the quality of collaboration between users and AI sys- tems. This section reviews the empirical evidence underlying these two factors in greater detail. Domain knowledge. A growing body of research has examined how users’ prior knowledge of a task do- main affects their behavior when working with AI-assisted decision-support systems. Dikmen and Burns [12] investigated the role of domain knowledge in the context of AI-assisted peer-to-peer lending decisions and found that participants who possessed relevant domain knowledge exhibited more calibrated trust com- pared with those who lacked such knowledge. Nourani et al. [31] further demonstrated that domain experts 5 were able to dynamically adjust their trust based on the AI system’s observed performance, whereas novice users exhibited persistent over-reliance regardless of the system's accuracy. In a study on human-AI team performance, Inkpen et al. [32] reported that highly proficient users were generally able to discern when to follow AI recommendations and when to override them, contributing to improved complementary team performance. Conversely, Bansal et al. [33] revealed that without sufficient domain knowledge to inde- pendently assess the correctness of AI outputs, users tended to follow AI-generated explanations indiscrim- inately. Collectively, these findings suggest that domain knowledge can serve as a moderating factor in how users process, evaluate, and act upon AI-generated information, though the specific manifestation of this effect may vary across task contexts. AI literacy. In parallel, a separate line of research has examined how users’ familiarity with AI tools, par- ticularly their ability to construct effective prompts, engage in iterative refinement, and critically assess AI outputs [34, 35], influences the quality of human-AI collaboration. Knoth et al. [13] investigated prompt engineering behaviors among users with different levels of AI literacy and found that users with lower AI literacy tended to construct prompts that mimicked human-to-human instructions, whereas those with higher AI literacy employed more structured, iterative, and strategically refined prompting approaches. Kim et al. [36] observed a similar pattern in an educational setting: students with high AI literacy engaged in collaborative, context-rich interactions with GenAI and produced significantly better writing outcomes than those with low AI literacy, who relied on generic, one-directional prompts with minimal iteration. Liu et al. [37] further demonstrated that AI literacy was a significant predictor of job performance in workplace set- tings. Synthesis. While these studies provide converging evidence that both domain knowledge and AI literacy can shape human-AI interaction dynamics, a few important caveats should be noted. First, much of the evidence on domain knowledge originates from classification or decision-support tasks in which users make discrete judgments, such as accepting or rejecting a single AI recommendation. Similarly, findings on AI literacy have been drawn primarily from educational and workplace contexts. Whether these patterns hold in the context of LLM-integrated BEMS, where users engage in multi-turn, goal-oriented conversations to interpret complex, domain-specific energy data and arrive at actionable decisions, has not been empirically tested. Second, the interaction between these two factors remains underexplored; it is plausible that domain knowledge and AI literacy may compensate for, reinforce, or even diminish each other’s effects depending on the task context. These gaps motivate the present study’s research design, which systematically examines their individual and combined effects on user-LLM interaction in the context of BEM. 3. Methodology 3.1. Overview Grounded in the evidence reviewed in Section 2, we designed a controlled experiment to examine how these two factors influence user-LLM collaboration in the context of BEM. Specifically, we adopted a role- playing experimental design – an established methodology for studying human decision-making [38] – in which participants assumed the role of a homeowner and were tasked with identifying the top five feasible behavioral changes to reduce households energy bills through interactions with OpenAI’s ChatGPT-4o model. This approach enabled systematic manipulation of the two independent variables while preserving a realistic task context grounded in actual appliance-level energy data. Figure 2 presents a flowchart outlining the preparation, execution, and analysis phases of the study, with each component corresponding to the subsections that follow. This study was approved by the University of Arizona’s Institutional Review Board (IRB) in January, 2025 (STUDY00005726) as a minimal risk ex- periment, and data collection was conducted from February to May 2025. 6 Figure 2. Methodology: An overview 3.2. Pre-Experiment: Home Energy Use Data Selection and Analysis This part of the research aimed to establish a systematic analysis framework to identify home energy saving potential using appliance-level energy use data (Section 3.2.1) and create reference conclusions for use in the role-playing experiment (Section 3.2.2). To set up the experimental environment, we utilized a single home case as the foundation. This approach was chosen to maintain consistency and focus our analysis on participants’ interactions with the GPT model. 3.2.1. Energy Saving Potential Analysis Framework As our energy use analysis framework to draw reference conclusions, we developed a set of energy-saving potential metrics by revising and extending the user DR potential scoring approach shown in [39]. Specif- ically, our framework introduced two key improvements: (1) it differentiated between on-peak and off-peak timeframes and (2) it introduced two new metrics, appliance use flexibility and comfort association with appliance, to more comprehensively characterize appliance-level energy-saving potential, as detailed below: - Appliance Energy Use: We considered the mean and variance of energy consumption for an ap- pliance or circuit when activated. To assess energy-saving potential, a higher mean energy use com- bined with lower variance suggests greater potential for energy savings. - Appliance Use Frequency: This metric quantifies how often a household uses a specific appliance or circuit. It provides insight into whether usage can be reduced within two distinct timeframes (on- and off-peak). - Appliance Use Flexibility: This metric assesses how rigidly or flexibly an appliance or circuit is used. For example, when a user consistently operates an appliance – such as an oven – during a fixed time window (e.g., 11:30 am to 12:00 pm), it suggests a habitual routine that may be difficult to modify. This behavioral rigidity, particularly paired with high use frequency, indicates a low potential for energy savings. - Comfort Association with Appliance: This Boolean metric assesses whether a specific appliance influences user comfort (True or False). A behavioral change is considered feasible only if it reduces energy consumption without significantly impacting comfort or health. For example, altering a thermostat’s temperature setpoint by 5°F (2.8°C) might not be practical, as such changes could alter occupants’ thermal sensations and lead to thermal discomfort. The following equations are the details of our proposed metrics above. Table 1 shares the basic notations and their descriptions that we utilized to establish all metrics. Table 1. Basic notations and their descriptions Notation Description 푖 Appliance or circuit type 푡 Time within the day or timeframe (e.g., hour) 퐷 The total number of days in the dataset 7 푃 !,# Power consumption of appliance or circuit 푖 at time 푡 on day 푗 푇 ! Power threshold specific to the appliance or circuit 푖 A binary indicator function 푈 !,# (Equation (1)) to detect whether the appliance or circuit 푖 was in use at time 푡: 푈 !,# =% 1 0 푖푓 푃 !,# ≥푇 ! 푖푓 푃 !,# <푇 ! (1) In this study, we applied adaptive thresholds after examining the power consumptions of active and standby modes in each appliance and circuit. We referenced a couple of articles [40, 41] as our starting points and heuristically refined our thresholds. The hourly use frequency (퐹 !,$ ! ) was the sum of binary indicator values (푈 !,# ) for all 15-minute intervals within an hour (퐻 % ), as shown in Equation (2). 퐹 !,$ ! =0푈 !,# #∈$ ! (2) where 푡∈퐻 % refers to the four 15-minute intervals within hour 퐻 % . Then, we could calculate the normalized average hourly frequency over 퐷 days (퐹 3 !,$ ! '()* ) as in Equation (3). 퐹 3 !,$ ! '()* = 1 퐷 0 퐹 !,$ ! ,+ 푁 , +-. (3) where 푁 was the number of intervals per hour, which was four in our dataset. As noted, we considered the averaged frequency within a certain timeframe to enhance the behavioral use interpretability, as organized in Equation (4). 퐹 ! / = 1 |퐻 / | 0퐹 3 !,$ ! '()* $ ! ∈$ " (4) where 푝 was the timeframe (e.g., on-peak time from 4:00 pm to 9:00 pm). The flexibility was calculated through Equations (5) and (6). 휎 ! $ = ) 1 +퐻 $ +∙퐷 .. / 퐹 !,% ! ,& '()* −퐹 2 !,% " '()* 3 + % ! ∈% " - &./ (5) 휎 ! '()* = 휎 ! −휎 *!' 휎 *01 −휎 *!' (6) where 휎 *01 was the maximum standard deviation and 휎 *!' was the minimum standard deviation within the house. Lastly, the average and variance of power usage (Equations 7 and 8) were calculated. 푃 ! 023,$ = 1 |퐻 $ |∙퐷 ..푃 !,% ! ,& % ! ∈% " - &./ (7) 8 푃 ! 20),$ = ) 1 |퐻 $ |∙퐷 .. / 푃 !,% ! ,& −푃 ! 023,$ 3 + % ! ∈% " - &./ (8) These proposed metrics were discussed with an external subject expert for third-party adjudication, and their validity was confirmed after a series of discussions. Ultimately, we could pinpoint behavioral adjust- ments that can potentially lead to energy savings using these metrics, which were treated as reference solu- tions in our study. 3.2.2. Home Data Selection and Saving Strategies We had access to energy use and generation data from 30 homes in Austin, collected at 15-minute intervals at both circuit and appliance levels during July and August, a period when peak loads often occur, purchased from Pecan Street Inc. [42]. For this experiment, we chose the best case for our experiment based on the following criteria: (1) number and diversity of appliances used during the data collection period in a home and (2) complexity in identifying feasible behavioral changes for energy savings. Figure 3. TOU rate used in the role-playing experiment. We obtained the actual TOU rate data from the utility rate database [43] and utilized it for our experimental study. Figure 3 illustrates the TOU rate used in this study. Table 2 shows the analysis of the appliances and circuits used in the selected home using the proposed four metrics. Table 2. Appliance and circuit analysis of the selected house based on the proposed four metrics Among those with high or moderate energy-saving potential (as indicated in the last column of Table 2), we selected top five appliances and formulated corresponding behavioral changes to achieve energy savings, as follows. - HVAC unit: This appliance was the most frequently used in this house and had the second highest FrequencyFlexibilityFrequencyFlexibility EV charger6.430.0310.060.56FALSEH HVAC unit3.610.930.7310.18TRUEM or H Pool pump3.20.220.240.120.3FALSEH Electric water heater2.240.010.340.020.24TRUEM Oven1.710.000.320.020.41TRUEL or M Washing machine0.860.000.070.010.05FALSEM Dishwasher0.820.010.150.030.19FALSEL or M Bedroom0.730.080.160.010.07TRUEL or M Clothes dryer0.210.020.020.030.02FALSEL Note: The order is sorted based on the mean energy use and the remaining appliances and circuits were excluded because their energy-saving potentials were all ranked as low. L: Low, M: Moderate, H: High Appliance/Circuit type Mean energy use (kW) Off-peakOn-peakComfort association Energy saving potential 9 mean energy consumption value (3.61kW) – consequently, the highest energy driver of this house. Although it is closely associated with user comfort, several interventions can be viable, including: a. Co-conditioning with personal comfort systems (PCSs): As indicated in [44], using PCSs – which typically consume much less energy than HVAC units – in conjunction with HVAC units such as portable electric fans can expand the range of temperature setpoint selection. b. Adjusting the HVAC unit’s temperature setpoint: This intervention must be implemented carefully, as it does not guarantee user comfort. Hence, only a minimal temperature setback, e.g., 1-2°F or 0.6-1.1°C, should be considered to mitigate potential discomfort. c. Precooling strategy: If users are not sensitive to cooler temperatures during off-peak hours, they can precool their homes before on-peak hours. - Pool pump: This appliance showed a high mean energy consumption (3.2kW) and was used more frequently in both on and off-peak hours, compared with other high energy-consuming appliances except for the HVAC unit. In addition, it showed moderate flexibility values in both on- and off- peak hours. - EV charger: This appliance had the highest mean energy use (6.43kW) among all appliances and circuits in this house. Furthermore, it was used more frequently during on-peak hours and had high flexibility in both on- and off-peak hours. Lastly, this appliance is not associated with user comfort. - Dishwasher: This appliance showed a moderate energy consumption (0.82kW) and flexibility, its energy saving potential was rated as moderate and low as it was not associated with user comfort. - Electric water heater: This appliance showed a high mean energy consumption (2.24kW) and used more frequently during on-peak hours with moderate flexibility values in both timeframes. Despite its promise, its energy saving potential was assessed as moderate or low given its association with user comfort. These possible behavioral changes were set as our answers to the role-playing experiment. 3.3. Pre-Experiment: Subject Recruitment and Grouping This study advertised the human subject experiment, starting from Feb, 2025 to May, 2025 through emails, online advertisement (UA@Work), and personal invitations using authors’ professional networks. Inter- ested individuals were invited to complete a prescreening survey (provided in Appendix A) designed to assess their domain knowledge in home energy use and AI literacy levels for participant grouping. The prescreening survey measured two constructs through a 5-point Likert scale: Domain knowledge was assessed through questions that captured participants’ experiential and self-evaluated understanding of building energy, including their responsibility for utility bills, their familiarity with home energy use pat- terns. AI literacy was assessed through questions that captured both behavioral and self-evaluated aspects of GPT use such as the frequency and duration of GenAI tool use, self-evaluated expertise. These measures are self-reported, capturing participants’ perceived rather than objective competence levels. However, in human-AI interaction contexts, perceived competence is arguably the more behaviorally relevant construct, as users’ beliefs about their own knowledge and skills directly shape how they approach and engage with AI systems [45, 46]. This measurement approach is consistent with current practice in the field, where self- reported scales have been adopted to assess both domain knowledge [47, 48] and AI literacy [49, 50] in human-AI interaction research. Table 3. Four groups with different building energy knowledge and AI literacy levels AI Literacy Low High Domain Knowledge Low Group L Group LH High Group HL Group H Using a stratified sampling approach, we attempted to distinguish subjects into four groups (Table 3). The goal of our stratified sampling approach is to have a balanced distribution across all groups in this study. 10 3.4. Experiment 3.4.1. Experimental Sequence Figure 4. Experimental sequence from the participant point of view. Figure 4 illustrates the sequence of the designed role-playing experiment. To enhance accessibility, we conducted the experiment using a laptop (in-person) or the videotelephony software program Zoom (online). Participants were first briefed on their goal – reducing the household energy bill – and given sufficient time to review and understand the details of the experiment provided through the instruction document (see Appendix for more details). Participants were allowed to raise any questions, and the experimenter clarified their questions. Next, participants were granted control of the ChatGPT 4o model in a web-browser (the selection of the GPT model is provided in Appendix) as illustrated in Figure 5. At the beginning of the role- playing experiment, two datasets were uploaded in the prompt. The objective was that the GPT model could process and analyze data based on participants’ needs as if the GPT model was connected to energy data collected by appliance-level metering technologies. This step marked the temporary realization of an LLM- integrated BEMS, simulating how such a system could function to support data-driven energy analysis and decision-making. Then the experiment commenced. Participants were given limitless time to interact with the GPT tool to discuss and formulate their conclusions. When finished, the experimenter ensured that the chat remained within the OpenAI account and saved it for analysis. Lastly, participants were asked to take the post-experiment survey. Figure 5. Schematic of generative AI interface in the role-playing experiment 3.4.2. Participant Information Eligible participants were invited via email, and 85 individuals ultimately participated in the role-playing experiment out of the 114 who completed the pre-screening survey. Figure 6 together illustrate the diversity and balance of the study’s participant pool across two key dimensions: domain knowledge (building energy) and AI literacy. We classified participants into four groups based on median splits, revealing meaningful representation in each quadrant. This balanced distribution ensured that the study captured a full spectrum of user profiles, supporting robust comparisons and insights into how different types of users would engage with LLM-integrated BEMS. We provided participants’ demographic characteristics in Appendix. 11 Figure 6. Distribution of participants’ domain knowledge scores and AI literacy scores. 3.5. Post-Experient: Prompt-Response Data Analysis 3.5.1. Overview The prompt-response data were processed and analyzed through a systematic framework as shown in Figure 7. The goal was to establish a consistent and replicable scoring process that that could be applied across the collected dialogue data to meaningfully capture and represent the human-AI interaction dynamics. Figure 7. Prompt-response data analysis flowchart 3.5.2. GPT-based Structured Context-Aware Language Evaluation (SCALE) The proposed GPT-based SCALE framework leveraged the contextual understanding capabilities of GPT, specifically, GPT-4o-mini as our analytical and scoring engine, to determine whether specific concepts or metrics are meaningfully addressed within a conversation as illustrated in Figure 8. 12 Figure 8. The SCALE framework flowchart Rather than relying solely on keyword detection or pattern matching, this framework assesses the contextual relevance and clarity of user-GPT exchanges to classify whether a particular analytical element was actively considered. SCALE framework addressed a primary challenge in conversation analysis by offering a more accurate method for identifying substantive content. The framework’s multi-factor structure offered explainable confidence scores via four weighted factors with detailed justifications, as detailed in below: - Explicitness: This factor measured whether participants used appropriate domain terminology and demonstrate clear understanding of concepts. It drew on linguistic research examining directness and clarity in communication [51], and communication theory emphasizing the importance of ex- plicit information exchange in human-computer interaction contexts [52]. - Depth: This factor evaluated whether participants engaged at a surface semantic level (e.g., basic acknowledgement, simple responses without elaboration) or a substantive latent level (e.g., detailed reasoning, comprehensive understanding, thoughtful analysis with context). It built on cognitive processing research distinguishing between shallow and deep levels of information engagement [53, 54] and thematic analysis research that examines different levels of analytical engagement with data [55]. - Consideration: Drawing on consumer behavior research on consideration sets [56], this factor measured whether specific concepts or alternatives were actively incorporated into participants’ decision-making process, reflecting the breadth of their evaluative reasoning. - Evidence: This factor emphasized the importance of supporting information in credible communi- cation, building on discourse analysis research [57] that examines how speakers construct persua- sive and well-substantiated arguments through the use of supporting details, examples, and justifi- cations. Then, the confidence score is calculated using Equation (1). 퐶표푛푓푖푑푒푛푐푒 푠푐표푟푒= 휔 / ∙퐸푥푝푙푖푐푖푡푛푒푠+휔 + ∙퐷푒푝푡ℎ+휔 4 ∙퐶표푛푠푖푑푒푟푎푡푖표푛+휔 5 ∙퐸푣푖푑푒푛푐푒 4 (1) Where 휔 is the weight for each scoring factor. In this study, we applied the same weights to evenly consider all factors. This framework was implemented systematically through a chain-of-thought prompt template with scoring rubrics for reproducibility. These details are provided in Appendix. 3.5.3. User Engagement Evaluation The objective of this user engagement analysis was to assess the extent and style of participants’ interactions with the GPT model. Consequently, seven metrics were employed as follows: Category #1. Interaction Volume 13 The proposed three metrics in this category captured key aspects of user interaction volume. o Total conversation turns: This metric captured the total number of conversation turns – com- prising both user prompts and GPT responses – required for a participant to reach their conclu- sion. The underlying premise is that fewer turns may indicate a more efficient interaction, though it may suggest greater reliance on GPT’s analytical capabilities. o Average prompt length: This metric measured the average number of words per user prompt. It helps assess how much information or context participants provided to the GPT model throughout the conversation. Longer prompts may indicate more detailed or specific inquiries, reflecting greater user initiative or domain knowledge, while shorter prompts may suggest reli- ance on GPT to guide the interaction or formulate solutions. o Prompt-response ratio: This metric compared the number of user prompts to the number of GPT responses. It provides insights into the interactivity and balance of the conversation. A higher ratio may indicate more user-driven engagement and iterative querying, while a lower ratio may suggest that users relied more heavily on single, extended GPT responses or submitted fewer prompts overall. Category #2. Conversational Reasoning We defined four distinct reasoning metrics, building on decision support systems theory [58], which explores how interactive computer systems can enhance human decision-making processes through structured information gathering, alternative evaluation, and choice implementation. These metrics represented how participants interacted with GPT to find energy-saving interventions. The SCALE framework was applied for these metrics. o Information seeking: This metric captured directive speech acts aimed at obtaining information, clarification, or elaboration. This metric was crucial for understanding how participants actively acquired knowledge and clarified concepts, which serves as a foundation for more informed energy decisions. Examples were: Can you explain that in more detail? how does this work? or what are the specific steps to implement this? o Constraint articulation: This metric aimed to evaluate assertive speech acts that state personal limitations, boundaries, or situation constraints. Capturing this behavior revealed how well users personalize solutions and set realistic boundaries, reflecting their ability to contextualize recom- mendations. Examples were: I am sensitive to being hot, that is infeasible for my situation, or I do not want to change my mealtime. o Solution evaluation: This metric was to identify assertive speech acts that express judgments, assessments, or evaluations of proposed solutions. This metric highlighted the participant’s abil- ity to critically assess and engage with the AI-generated suggestions, a key indicator of cognitive involvement in decision-making. Examples were: The proposed method seems reasonable, I think the oven idea makes sense, electric vehicle charging at night is an excellent idea, or this would be effective for my situation. o Commitment expression: This metric was to measure commissive speech acts that express willingness to try solutions or commit to behavioral changes. This metric was essential for as- sessing whether the interaction led to an actionable behavioral commitment or confirmation of recommended behavioral changes, serving as a proxy for the user’s readiness or willingness to follow through with energy-saving actions suggested during the conversation. Examples were: I am willing to adjust my thermostat settings, I can reschedule when I run my dry washer, I will try using ceiling fans to comfort myself, or I will implement this change gradually. 3.5.4. Home Energy Analysis Evaluation In the home energy analysis, we investigated how both the participants and the GPT model interpreted and reasoned through the provided home energy data to arrive at actionable conclusions. The objective was to understand how users engaged with the data by applying their building energy knowledge and utilizing the model’s capabilities to identify feasible behavioral changes, reflecting a combination of human insight and AI support in the decision-making process. 14 In addition to energy-saving potential metrics introduced in Section 3.2, we incorporated additional three metrics to better capture participants’ reasoning processes and understanding of home energy analysis as presented in Table 4. Collectively, these metrics allowed us to assess how participants integrated practical considerations when forming their strategies. The SCALE framework was applied in this analysis. Table 4. Evaluation metrics used for home energy analysis and how they were identified in SCALE Metric Definition Examples Cost awareness Discussion of electricity bills, energy costs, utility rates, TOU pricing, cost savings, or financial impact of energy decisions Your electricity bill was $150 last month, Peak hours are more expensive, I want to reduce my energy bills, Please consider TOU rates in your calculations, Behavioral change Discussion of adjusting usage patterns, adopting energy-saving behaviors, modifying habits, or actionable steps to reduce energy consumption I suggest running the dishwasher at night, Your HVAC running times can be adjusted to off-peak hours, Let’s evaluate how I can change my energy usage patterns to reduce energy bills, I can’t change my energy use patterns for the refrigerator, Technical knowledge Discussion showing understand- ing of how appliances work, sys- tem operations, equipment speci- fications, or technical aspects of energy use Heat pumps are more efficient than traditional HVAC systems like furnaces, Smart thermostats can optimize energy use through scheduling and sensors, Variable-speed pumps adjust their speed based on demand, 3.6. Post-Experiment: User-GPT Conclusion Analysis To evaluate whether participants, through their interaction with the AI system, arrived at conclusions with reference solutions, a two-stage evaluation, informed by the LLM-as-a-judge paradigm [59], was conducted as shown in Figure 9. In the first stage, the LLM (Open AI GPT-4o-mini) was used to systematically com- pare each participant’s final energy-saving recommendations against a predefined set of reference solutions, following a CoT evaluation protocol that has been indicated to improve alignment with human judgements [60]. This comparison produced two distinct outcome metrics: (1) the appliance identification rate, defined as the proportion of the five target appliances that the participant correctly identified as candidates for en- ergy savings, and (2) the strategy alignment rate, defined as the proportion of the seven reference strategies that the participant’s recommendations substantially matched. In the second stage, the entire evaluation results from the LLM were manually cross-checked for three times as a validation step to ensure the relia- bility of the assessments. Figure 9. User-GPT conclusion analysis using reference conclusions and GPT 3.7. Post-Experiment: Survey Analysis The post-experiment survey, available in Appendix, was prepared: (1) to understand how participants would assess their own human-AI collaboration experiences (2) to assess changes in participants’ building energy knowledge and experience with the GPT model and (3) to gather participants’ perspectives on the benefits of utilizing GPT for managing home energy use. For the first goal, we posed questions about the role of their domain knowledge and prompt expertise during the role-playing experiment. For the second goal, we included a direct comparison between pre-screening and post-experiment responses by intentionally reusing the same questions from the prescreening survey – listed below. - Rate your understanding of home energy use and its patterns, 15 - Rate your understanding of how monthly energy bills are calculated, - Rate your expertise in using GenAI tools like ChatGPT. Regarding the third objective, we posed questions about whether they would consider using an AI-assisted BEM system in their homes and whether they believed such a system could support their learning and understanding of BEM. This line of questioning was intended to investigate both participants’ openness to future adoption and their perception of GPT as a learning companion in the field of home energy decision- making. 3.8. Individual, Group-Specific, and Overall Analysis and Comparison The analysis proceeded in two stages. First, we examined individual-level results across all participants to establish overall patterns, then conducted group-based comparisons to assess the influence of domain knowledge and AI literacy on interaction behaviors and outcomes. Four analytical dimensions were con- sidered: user engagement levels, home energy analysis, comparison between user-GPT conclusions and reference solutions, and post-experiment survey responses. Descriptive statistics, including means, standard deviations, and medians, were computed for each metric across the four participant groups to characterize general trends. To formally test for group differences, we employed non-parametric inferential tests, se- lected due to the non-normal distributions observed across most metrics and the unequal group sizes result- ing from the median-split classification. Kruskal-Wallis H tests [61] (degree of freedom = 3, α = .05) were first conducted to assess whether significant differences existed across the four groups. Mann-Whitney U tests [62] were then used to examine the main effects of each factor, domain knowledge (high vs. low) and AI literacy (high vs. low), independently, as well as all six pairwise group comparisons. Effect sizes were quantified using the rank-biserial correlation (r) [63], and Bonferroni correction [64] was applied to pair- wise comparisons to control for familywise error rate. 4. Results 4.1. Case-based Analysis To illustrate how the analysis framework captures distinct patterns of user-GPT interaction, we presented three representative cases selected from 85 participants. These cases were selected to represent contrasting interaction patterns along two dimensions: user engagement level and GPT performance quality. As shown in Table 5, the scoring criteria differentiated these cases across multiple metrics, revealing how variations in user input quality and interaction depth correspond to differences in both SCALE confidence scores and alignment with reference solutions. The full conversation logs are provided in Appendix H. Table 5. Results derived from the analysis framework using three representative conversations Category Metric Example #1 Example #2 Example #3 Participant Information Group L LH H Domain knowledge 2.71 2.57 4.43 AI literacy 1.25 3.75 3.75 Interaction Volume Total Conversation Turns 4.00 10 10 Average Prompt Length 8.50 17.00 54.6 Prompt Response Ratio 0.081 0.11 0.153 Conversational Reasoning Information Seeking 0.56 0.84 0.68 Constraint Articulation 0.00 0.00 0.68 Solution Evaluation 0.00 0.00 0.68 Commitment Expression 0.00 0.00 0.68 User GPT User GPT User GPT Home Energy Analysis Appliance energy use 0.00 0.74 0.00 0.74 0.48 0.94 Cost awareness 0.56 0.74 0.66 0.74 0.52 0.94 Behavioral change 0.56 0.74 0.84 0.82 0.76 0.94 Appliance use flexibility 0.00 0.44 0.74 0.58 0.48 0.74 Appliance use frequency 0.00 0.00 0.00 0.52 0.00 0.62 Comfort association with 0.00 0.44 0.00 0.00 0.66 0.70 16 appliance Technical knowledge 0.00 0.56 0.00 0.74 0.00 0.70 User-GPT conclusion Alignment rate with reference solutions 80% 20% 100% Example #1. This case showcases how a productive interaction can emerge even with minimal user en- gagement, provided GPT performs at a high level. As in Table 5, the interaction volume metrics reflected the participant’s minimal engagement: the conversation consisted of only four total turns with a notably low prompt-response ratio, indicating that GPT generated substantially more content than the participant. The conversational reasoning metrics further confirmed this pattern. The participant’s engagement was lim- ited to information seeking in the initial prompt, with no constraint articulation, solution evaluation, or commitment expression observed throughout the conversation. This indicates that the participant neither refined the scope of the analysis nor critically assessed GPT’s recommendations. Despite the limited engagement, the home energy analysis metrics revealed a clear asymmetry: GPT con- sistently scored higher than the participant across most dimensions, particularly in appliance energy use and technical knowledge, where the participant made no substantive contributions. Nonetheless, the user- GPT conclusions achieved an 80% alignment rate with reference solutions, with the only omission being a dishwasher intervention. This case demonstrates that GPT can produce reasonably accurate energy-saving interventions, even with limited user guidance, though the missed strategy suggests that more targeted prompting could have improved completeness. Example #2. This case demonstrates how increased interaction volume does not necessarily lead to better outcomes when user engagement lacks critical evaluation of GPT’s recommendations. In contrast to Exam- ple #1, the interaction volume metrics indicated a substantially more active conversation: 10 total turns with an average prompt length of 17.00 words. However, the conversational reasoning metrics revealed that this increased activity did not translate into deeper analytical engagement. While information seeking was rel- atively high (0.84), constraint articulation, solution evaluation, and commitment expression all remained at zero. In other words, the participant asked multiple follow-up questions, requesting TOU rate schedules, per-appliance savings estimates, and annual projections, but never questioned the validity of GPT's initial recommendations or attempted to refine them. Each subsequent prompt sought additional clarification on the same set of strategies rather than evaluating whether those strategies were appropriate. This lack of critical evaluation proved consequential. The home energy analysis metrics showed that GPT’s initial response contained a mismatch between its data analysis and its recommendations, a pattern reflected in the low user-GPT alignment rate of only 20% with reference solutions. Despite the participant’s relatively high AI literacy, the low domain knowledge appears to have limited their ability to recognize this mismatch. This case illustrates that interaction volume alone is an insufficient indicator of productive human-AI col- laboration; without substantive evaluation of AI-generated recommendations, additional turns may simply reinforce an initially suboptimal output. Example #3. This case showcases how high-quality user engagement can steer the conversation toward a fully aligned outcome, even when GPT’s initial recommendations required refinement. Although the inter- action volume was comparable to Example #2 in total conversation turns (10), the nature of engagement differed substantially. The average prompt length was 54.6 words (more than three times that of Example #2) and the prompt-response ratio was correspondingly higher. More critically, the conversational reasoning metrics distinguish this case from both preceding examples: all four dimensions were active throughout the conversation. This represents the only case among the three where the participant moved beyond infor- mation seeking to critically evaluate GPT’s suggestions, articulate personal constraints, and propose alter- native strategies. This deeper engagement directly shaped the trajectory of the conversation. The participant rejected GPT’s refrigerator optimization suggestion as infeasible, articulated thermal comfort constraints that precluded aggressive thermostat adjustments, proposed ceiling fans as an alternative cooling strategy, and assessed 17 the feasibility of reducing pool pump runtime by questioning its impact on water quality. These interactions prompted GPT to revise its initial recommendations, ultimately producing two new interventions that aligned with expert-judged solutions. The home energy analysis metrics reflected this collaborative dy- namic: the participant contributed meaningfully across most dimensions, particularly in behavioral change and comfort association, while GPT complemented these contributions with higher scores in appliance en- ergy use, cost awareness, and technical knowledge. The resulting user-GPT conclusions achieved a 100% alignment rate with expert-judged solutions. This case demonstrates that when participants actively evalu- ate, constrain, and redirect GPT's outputs, the collaborative process can yield outcomes that match expert- level recommendations. 4.2. Overall & Group-level Analysis This subsection presents the overall and group-level results across all 24 metrics, derived from the proposed analysis framework. Table 6 summarizes the descriptive statistics Table 6. Summary of metric scores by participant group with Kruskal-Wallis test results Category Metric L (n=28) LH (n=24) HL (n=12) H (n=21) p Interaction Volume Total Conversation Turns 13.21±14.20 12.88±10.40 13.42±11.96 10.90±8.05 0.94 Average Prompt Length 19.93±18.94 23.15±17.44 18.70±8.49 25.16±18.66 0.36 Prompt Response Ratio 0.10±0.07 0.14±0.12 0.14±0.12 0.14±0.11 0.790 Conversational Reasoning Information Seeking 0.70±0.12 0.67±0.18 0.68±0.08 0.69±0.10 0.80 Constraint Articulation 0.48±0.32 0.43±0.32 0.45±0.34 0.46±0.31 0.91 Solution Evaluation 0.41±0.32 0.47±0.30 0.39±0.30 0.40±0.33 0.85 Commitment Expression 0.29±0.35 0.25±0.35 0.25±0.32 0.16±0.27 0.58 User GPT User GPT User GPT User GPT User GPT Home Energy Analysis Appliance energy use 0.25± 0.31 0.76± 0.07 0.34± 0.30 0.76± 0.06 0.25± 0.32 0.73± 0.06 0.29± 0.29 0.77± 0.09 0.79 0.85 Cost awareness 0.63± 0.16 0.74± 0.06 0.56± 0.24 0.72± 0.15 0.64± 0.05 0.73± 0.06 0.65± 0.09 0.75± 0.07 0.55 0.80 Behavioral change 0.67± 0.17 0.77± 0.06 0.67± 0.17 0.75± 0.06 0.66± 0.10 0.75± 0.06 0.67± 0.09 0.75± 0.11 0.60 0.45 Appliance use flexibility 0.20± 0.28 0.54± 0.17 0.34± 0.30 0.55± 0.13 0.20± 0.30 0.54± 0.19 0.22± 0.27 0.49± 0.22 0.27 0.58 Appliance use frequency 0.00± 0.00 0.23± 0.27 0.00± 0.00 0.25± 0.28 0.04± 0.15 0.21± 0.26 0.00± 0.00 0.24± 0.28 0.11 0.93 Comfort association with appliance 0.27± 0.32 0.49± 0.26 0.28± 0.31 0.38± 0.29 0.28± 0.35 0.48± 0.17 0.33± 0.33 0.45± 0.24 0.87 0.46 Technical knowledge 0.02± 0.10 0.60± 0.29 0.00± 0.00 0.58± 0.31 0.00± 0.00 0.48± 0.36 0.05± 0.16 0.63± 0.28 0.32 0.87 User-GPT conclusion Overall alignment 0.65±0.13 0.71±0.13 0.63±0.12 0.67±0.13 0.22 Appliance identification rate 0.69±0.17 0.77±0.17 0.63±0.14 0.76±0.17 0.04 * Strategy alignment rate 0.59±0.17 0.62±0.17 0.52±0.15 0.57±0.12 0.17 Note: Mean ± standard deviation; * statistically significant Interaction Volume. Across all four groups, participants engaged in a comparable number of conversation turns with the AI system, averaging approximately 12.6 turns per session. Neither domain knowledge nor AI literacy produced meaningful differences in how much participants interacted with the system. Similarly, average prompt length and the prompt-response ratio, reflecting the verbosity and relative contribution of user input, did not differ across groups. These results suggest that participants across all competence profiles adopted broadly similar interaction patterns in terms of volume and balance of exchange, regardless of their 18 prior expertise. Conversational Reasoning. The four SCALE-based conversational reasoning dimensions showed no sig- nificant group differences. Information seeking scores were relatively uniform across groups, indicating that all participants, regardless of background, directed the conversation toward gathering relevant energy- related information at similar rates. Constraint articulation and solution evaluation scores were moderate across the board but exhibited high within-group variability, suggesting that individual differences in how participants framed constraints or critically assessed GPT-generated solutions far exceeded any group-level tendencies. Commitment expression remained the lowest-scoring dimension overall, reflecting that most participants did not explicitly commit to specific strategies during the conversation, a pattern consistent across all groups. Home Energy Analysis. On the user side, participants demonstrated moderate engagement with cost awareness and behavioral change, while energy consumption understanding remained relatively low with high variability. Use frequency and technical knowledge were near floor levels, indicating that participants rarely discussed appliance usage patterns or technical details in their prompts. None of the seven user-side dimensions differed significantly across groups. On the GPT side, ChatGPT consistently produced responses with high scores on energy consumption, cost awareness, and behavioral change, regardless of which group the participant belonged to. The uniformity of GPT-side scores across groups confirms that GPT’s analytical output was largely driven by its own re- sponse generation tendencies rather than being shaped by differences in user input. Notably, the gap be- tween user- and GPT-side scores was most pronounced for energy consumption and technical knowledge, suggesting that ChatGPT compensated for areas where participants contributed the least. User-GPT Conclusion Analysis. The conclusion analysis evaluated whether participants' final energy-sav- ing recommendations aligned with expert-validated reference solutions. The overall alignment score did not differ significantly across groups. However, the appliance identification rate was the only metric across all 24 measures to reach statistical significance. Groups with low AI literacy identified target appliances at higher rates than groups with high AI literacy. This counterintuitive pattern suggests that participants with greater AI experience may have relied more heavily on GPT’s direction without independently verifying appliance-level conclusions, whereas less AI-literate participants may have drawn more on their own rea- soning to confirm or supplement GPT's recommendations. The strategy alignment rate showed a similar directional trend but did not reach significance. Post-experiment survey results, which capture participants’ subjective perceptions of the LLM-integrated BEMS are presented in Section 4.3 alongside the corresponding hypothesis evaluations. 4.3. Hypothesis Evaluation In this subsection, we assessed our hypotheses, presented in Introduction. H1. Participants with limited domain knowledge may rely heavily on the LLM’s capabilities, potentially resulting in brief interactions. Supported. The tendency of relying on LLM’s analytical capabilities to draw their conclusions was found across all participant groups regardless of their domain knowledge levels. Total conver- sation turns did not differ significantly across groups (H = 0.42, p = .936), nor did average prompt length (p = .358) or prompt-response ratio (p = .790). Participants with low domain knowledge (Groups LH and L) averaged 12.88 and 13.21 turns, respectively, comparable to or slightly above the high domain knowledge groups (H: 10.90, HL: 13.42). Conversational reasoning scores were similarly uniform, with no SCALE dimension showing significant group differences (all p > .57). This may indicate that participants, irrespective of domain expertise, viewed the LLM as a suffi- ciently efficient problem-solver [47, 65] – reducing their perceived need to engage in deeper, iter- ative dialogue or to interrogate the model’s outputs critically or to interrogate the model’s outputs critically. Rather than validating or expanding on GPT’s suggestions, many treated the GPT model 19 more as a static information provider than a dynamic thought partner. This behavior suggested a broader cognitive framing of LLMs as answer engines rather than collaborative reasoning tools, especially when task goals seem achievable through direct prompting. It may also reflect a possible overestimation of GPT’s analytical capabilities [66], where participants, even those with high do- main knowledge, defaulted to passive use rather than applying their expertise to guide, refine, or challenge the GPT model’s responses. It is also worth noting that participants did not have direct access to the raw energy data during the conversation — the datasets were uploaded to ChatGPT, which performed all numerical analysis. Without independent tools to verify GPT's calculations, participants across all groups were structurally positioned to rely on the model's analytical outputs, which may partially explain the uniform reliance pattern observed. This behavioral pattern underscored a critical challenge in human-AI collaboration: effective use of LLMs requires not only prompt engineering skills or domain knowledge in both users and GPT, but also a shift in user mindset – from passive querying to active co-construction [67]. H2. Participants with strong domain knowledge may engage in extended interactions with the tool to validate LLM’s suggestions, ultimately arriving at reasonably accurate conclusions. Not Supported. Interestingly, participants with high domain knowledge did not consistently en- gage in longer or more complex interactions – no statistical difference in the interaction volume or conversational reasoning metrics – despite their presumed capacity to critically evaluate and refine the LLM’s outputs, compared to their less knowledgeable counterparts. This counterintuitive result can be explained as follows: First, participants with strong domain expertise might have assumed that GPT was already equipped with sufficient knowledge, leading them to trust its responses with- out feeling the need to probe further – a pattern consistent with automation complacency . Second, these participants may have approached the task with confidence in their own judgment, using GPT primarily as a confirmation tool rather than as a collaborator [68]. In such cases, their goal may not have been to challenge or extend the model’s analysis, but rather to verify that it aligned with their existing understanding [69]. Moreover, high-knowledge participants might have found GPT’s re- sponses to be “good enough” within the constraints of the task, reducing their motivation to test alternative perspectives or explore edge cases [70]. This aligns with the observed pattern of relatively short interactions even among expert users, and challenges the assumption that domain expertise necessarily translates into deeper or more interac- tive use of AI tools. H3. Participants paired with strong AI literacy would compensate their limited domain knowledge by effectively interacting with GPT to extract relevant insights, formulate well-structured prompts, and critically assess AI-generated recommendations. Partially Supported. AI literacy was the only participant factor that produced a statistically sig- nificant effect on any outcome metric (Table 6). Notably, Group LH achieved the highest alignment rate (77%) among all four groups, outperforming groups with high domain knowledge. This sug- gests that strong AI literacy can partially offset limited technical expertise by enabling users to better engage with, interpret, and refine GPT’s suggestions. However, this compensatory effect did not manifest through the specific mechanisms predicted by H3. None of the interaction volume or conversational reasoning metrics differed significantly by AI literacy. Therefore, it indicates that high AI literacy participants did not formulate notably longer prompts, engage in more conversation turns, or demonstrate measurably higher levels of constraint articulation or solution evaluation. This compensation appears to have operated not through how much or how explicitly participants interacted with GPT, but rather through how effectively they interpreted and acted upon GPT’s outputs [67, 71]. We therefore consider that this hypothesis is partially supported, since a 77% appliance 20 identification rate still falls short of ideal collaboration outcomes. If Group H had successfully collaborated with GPT in this role-playing experiment – actively leveraging their domain knowledge to shape, question, and refine GPT’s outputs – we could have better observed the full extent to which the knowledge gap could be compensated through user-GPT collaboration. The fact that Group H did not outperform LH reinforces the interpretation from H2: domain expertise alone did not translate into deeper analytical engagement with the LLM. H4. Excessively verbose GPT responses may negatively affect user agency and limit iterative dialogue, especially in participants with lower AI literacy. Supported. The prompt-response ratio across all 85 participants was notably low (median: 0.08) and this indicates that GPT responses were roughly 12 times longer than participant prompts. This asymmetry suggests that the sheer volume of GPT-generated content might have overwhelmed par- ticipants – making the interaction feel more like passively receiving information from AI than col- laboratively working toward optimized solutions [72, 73]. The density and breadth of responses may have imposed a cognitive load that interfered with participants’ ability to engage in critical reasoning or iterative inquiry, ultimately reducing the likelihood of deeper collaboration and shared decision-making regardless of their domain knowledge. This tendency was observed across all four groups. In other words, the structure and delivery of the AI’s output to user prompts are critical factors that shape user engagement. This finding carries important design implications for LLM-based BEMS. Effective structure and delivery of LLM outputs should encourage clarity, accessibility, and opportunities for users to re- spond, reflect, and refine their energy-saving strategies in collaboration with the AI. One potential approach is offering adaptive output modes, which allow users to choose between detailed, com- prehensive explanations and more concise, modular responses that invite follow-up questions. Such configurability would accommodate diverse use preferences while preserving opportunities for the iterative dialogue. H5. Participants would perceive this LLM-based analysis platform for HEM as useful and educational, providing a valuable opportunity to enhance their understanding while exploring feasible energy- saving strategies. Supported. The post-experiment survey results strongly support this prediction. Six survey items captured participants’ subjective evaluations of their interaction with GPT during the role-playing experiment, organized into two categories: self-evaluation of the experience and self-assessed hu- man-AI collaboration (Figure 10). Figure 10. Self-assessed human-AI collaboration 21 Regarding self-evaluation of the experience, responses to “difficulty in understanding ChatGPT answers” were overwhelmingly positive across all participant groups, with the majority disagreeing with the statement. Only five participants (5.9%) chose either agree or strongly agree with this statement. Responses to the second question (improving knowledge of home energy use, patterns, and saving potential) revealed strong perceived learning benefits. For example, 91.7% of partici- pants in Group LH, the highest rate across all participant groups, either agreed or strongly agreed with the statement, and 71.4% of Group H shared the same view. Participant perceptions of GPT’s capability to deliver expert-level support were similarly favorable. Interestingly, 81.8% of partici- pants with high building energy knowledge either agreed or strongly agreed with this statement while only 73.1% of participants with low building energy knowledge shared the same view. This gap suggests that participants with higher domain knowledge might have had clearer expectations or benchmarks for expert-level reasoning, making them more receptive to GPT’s ability to deliver technical insight and more confident in recognizing its expert-like contributions [74]. Regarding self-assessed human-AI collaboration, responses to the first option (I fully utilized ChatGPT’s capabilities) showed a moderate level of agreement across all groups. Agreement rates ranged from 39.3% in Group L to 70.8% in Group LH, suggesting that participants with higher AI literacy felt more capable of leveraging the tool’s full potential. The “I wish I knew some tech- niques to enhance ChatGPT’s responses” question revealed a strong, shared desire for better prompt engineering knowledge even by the high AI literacy group, who 77.8% expressed this sentiment. Group L showed the highest agreement (96.5%), indicating a widespread recognition, especially among novice users, of the gap between knowing how to use GPT and knowing how to employ it effectively. The last question (ChatGPT helped me arrive at better answers in the role-playing ex- periment) revealed an interesting trend across participant groups. The lowest agreement rate of 67.9% was found from Group L while the highest agreement rate of 85.7% was found from Group H. These findings underscore the promise of LLM-based BEM platforms as dual-purpose tools, capa- ble of supporting decision-making while also fostering energy literacy. This perception is further corroborated by the pre-post comparison of participants' self-reported assessments (Figure 11), which demonstrates a consistent upward shift in understanding of home energy use, understanding of energy bills, and AI literacy following the role-playing experiment. Taken together, these results suggest that the envisioned platform was not only perceived as a helpful assistant for immediate task completion but also served as a resource for ongoing learning and empowerment in navigating the increasingly complex energy landscape at home. Figure 11. Overall trends of self-assessed domain knowledge and GenAI tool expertise before and after the role- playing experiment. 5. Discussion This research, addressing core research questions in the interdisciplinary field of integrating AI with BEMS, left several discussion points. 22 Behavioral Tendency in User Prompts As analyzed through user engagement interaction volume metrics, most participants tended to utilize simple prompts to initiate their interactions with GPT. This brevity reflects a broader behavioral pattern where users might have expected GPT to deliver complete, accurate, and comprehensive answers without asking for detailed or iterative inputs – potentially handing over the responsibility of providing the context to GPT. This behavioral tendency suggests that participants may have assumed GPT would infer or fill in contextual gaps on their behalf, which ultimately constrained the depth and interactivity of the conversation. Ultimately, this prompt approach consequently limited deeper exploration or iterative reasoning, suggesting that even prompt-savvy users might default to passive engagement when the task feels goal-oriented or when early responses from GPT seem satisfactory. Notably, this observed behavior may reflect how many people en- gage in real-world scenarios, asking a single, task-oriented question and accepting the first plausible re- sponse without further probing, thereby the interactive and co-creative potential of human-AI collaboration [75]. Key Factor in Human-AI Collaboration: Domain Knowledge Recent research has demonstrated that while prompt engineering skills can significantly enhance the quality of LLM outputs, domain knowledge, incorporated into the LLM model, plays a critical role in formulating meaningful prompts and evaluating responses [76]. Currently, various techniques are available in LLMs to enhance performance on specific topics or applications such as parameter-efficient fine-tuning, RAG, and agentic frameworks. Even though these techniques could enhance LLM’s analytical performance on home energy use data, we intentionally chose not to implement them in this study in order to concentrate more on observing the baseline capabilities of general-purpose LLMs when interacting with participants in a naturalistic, un-augmented setting. This decision allowed us to better isolate and assess the impact of user characteristics – particularly, domain knowledge and AI literacy – on the success of user-GPT collaboration, without confounding the results with system-level enhancements. Previous studies also indicated that domain experts who acquire basic prompting skills achieve better results than AI-guided prompt engineering without domain knowledge [77]. Aligning with this observation that users require sufficient domain knowledge for high performance, this study discovered that in a human-AI collaboration context, users must also engage with AI at a deeper cognitive level – by critically interpreting AI-generated outputs, questioning assumptions, and iteratively refining conclusions through reasoning. This level of engagement enables the collaboration to move beyond surface-level exchanges and toward more accurate, context-aware, personalized, and reference outcomes. AI Literacy as the Differentiating Factor in Outcome Quality While much of the existing literature on human-AI collaboration has emphasized domain knowledge and prompt engineering skills as primary determinants of effective LLM use, the findings of this study suggest that AI literacy may also play a consequential role in shaping collaboration outcomes. Among the 20 metrics examined, the only statistically significant group difference was in the appliance identification rate, and this difference was driven by AI literacy. This distinction carries important implications. If AI literacy primarily enhances how users process and act upon AI outputs rather than how they formulate inputs, then interventions aimed at improving human-AI collaboration should focus not only on teaching users to write better prompts but also on developing their capacity for critical evaluation of AI-generated content. Tankelevitch et al. [67] have similarly argued that generative AI imposes unique metacognitive demands on users. Specifically, it requires them to assess the relevance, completeness, and accuracy of outputs that may appear fluent and authoritative regardless of their actual quality. The present findings lend empirical support to this argument: the gap between high- and low-AI-literacy participants manifested not in their ability to extract information from GPT, but in their ability to discern what to do with it. Role of GPT’s Analytical Performance in Human-AI Collaboration GPT’s analytical performance was not a targeted variable under investigation in this research; however, it 23 emerged as a critical factor influencing the quality of human-AI collaboration, especially in shaping user- GPT conclusions. The majority of participants, even those with 100% alignment rates, relied heavily on GPT’s (vanilla) analytical capabilities for interpreting appliance-level home energy data rather than offering explicit guidance or specifying tailored computational approaches. Consequently, GPT’s surface-level anal- ysis was often sufficient to identify the most salient energy-saving strategies, such as those involving HVAC and pool pump usage (each with >94% success rates), but fell short of uncovering less obvious, high-impact opportunities. In other words, if an LLM (or advanced LLM framework) shows strong contextual reasoning and analytical capabilities tailored to BEMS, it could enhance the productivity and effectiveness of human- AI collaborations by supporting deeper, more personalized, and data-driven exchanges. Our decision to use a general-purpose LLM in our experiment helped reveal the limitations of relying solely on default model behavior as our key takeaway. Design Implications for LLM-Integrated BEMS The behavioral patterns and outcome differences observed in this study point toward several design con- siderations for LLM-integrated BEMSs. First, the pervasive passivity observed across all participant groups, and it suggests that users may benefit from scaffolded interaction designs. Rather than presenting a blank text input field and expecting users to formulate effective queries independently, platforms could incorpo- rate guided prompting mechanisms such as structured question templates, suggested follow-up queries, or progressive disclosure interfaces that encourage users to refine their initial requests [78, 79]. Such scaffold- ing could be particularly valuable for users with lower domain knowledge, who may lack the vocabulary or conceptual framing to articulate specific analytical needs. Second, the finding that AI literacy was the key differentiator in outcome quality suggests that platforms should invest in interpretive support features. These might include confidence indicators for AI-generated recommendations, side-by-side comparisons of alternative strategies, or summary dashboards that help users evaluate and prioritize AI suggestions rather than passively accepting them [33, 80]. Such features would lower the metacognitive barrier for users with less AI experience, effectively embedding some of the evaluative capacity that high-AI-literacy participants brought to the task organically. Finally, given that GPT’s surface-level analysis was sufficient for identify- ing salient energy-saving strategies but fell short on less obvious opportunities, platforms targeting profes- sional or advanced use cases should consider incorporating domain-specific analytical layers [28, 81]. While the present study intentionally used baseline GPT to isolate user-level factors, real-world deploy- ments would benefit from augmenting the LLM's analytical depth to better match the expectations of do- main-knowledgeable users. 6. Conclusion This study investigated how user domain knowledge and AI literacy shape human-AI interactions in an LLM-integrated BEMS. Using a 2×2 factorial design with 85 participants interacting with a GPT model to anlayze residential energy data, we assessed both the process of interaction through interaction volume and conversational reasoning metrics based on the SCALE framework and the quality of outcomes, measured by appliance identification and strategy alignment accuracy. The key contributions and findings of this research are as follows: - LLM-assisted analysis produced remarkably uniform interaction patterns across expertise levels. Across 19 process-level metrics spanning interaction volume and conversational reasoning, no statistically significant differences were found among the four participant groups. Neither do- main knowledge nor AI literacy significantly altered how participants interacted with GPT, includ- ing conversation length, prompt complexity, or the types of reasoning expressed. This suggests that LLM tools may act as equalizers, flattening expertise-based differences in how users engage with analytical tasks. While this points to the accessibility and democratizing potential of LLM-inte- grated platforms, it also raises concerns about underutilization of domain expertise in human-AI collaboration. - AI literacy emerged as the key differentiator in outcome quality. The only statistically 24 significant group difference was in the appliance identification rate (Kruskal-Wallis H = 8.49, p = .037), and this difference was driven by AI literacy (Mann-Whitney U, p = .009) rather than domain knowledge (p = .446). Participants with higher AI literacy accomplished more accurate conclusions, suggesting that AI literacy operates primarily at the stage of output interpretation and critical evaluation rather than input formulation. - The SCALE framework was introduced as a structured analytical tool for identifying and cat- egorizing specific conversational concepts within human-AI interactions. Its five dimensions offer a systematic lens for assessing user reasoning quality. Its adaptable architecture supports broader application in future studies involving dialog-based AI systems. - Two complementary metric sets were proposed to assess user prompts and engagement levels. Interaction volume metrics captured the quantitative aspects of dialogue, while conversational rea- soning metrics outlined user reasoning depth and interaction quality. Together, these metrics ena- bled a multi-layer assessment of how participants engaged with GPT to achieve their intended out- comes. - Participants perceived the platform as useful and educational. Post-experiment survey results indicated that the majority of participants found the LLM-assisted energy analysis valuable, and self-assessed knowledge levels in energy use and energy bills increased significantly from pre- to post-experiment, confirming the platform's educational potential even in a single-session interac- tion. Beyond its research contributions, this study represents a foundational step toward realizing effective and scalable LLM-integrated BEMS that can empower a wide range of users to make informed, personalized, and context-sensitive energy decisions in their daily lives. Importantly, such systems can offer great oppor- tunities for households, but also for utilities and policymakers. For utilities, where energy consumption and generation data are collected at scale, LLM-integrated BEMSs can support synthesizing energy use patterns at granular levels and facilitate the development of new and more customized DR programs – realizing more effective peak demand management. Additionally, by benefitting from LLMs’ ability to interpret trends and patterns in historical and real-time data, utilities could better segment customer groups, tailor communications for maximum program participation, and offer education opportunities. For policymakers, these systems offer a powerful means to promote energy literacy and evaluate the impact of policy inter- ventions on energy use by analyzing dialogues between the systems and users – specifically, how users interpret, engage with, and respond to the interventions. Such insights can help policymakers identify gaps and refine programs to be more inclusive and effective across diverse populations. We acknowledge the limitations of this study. First, the experiment was designed for participants to act as if they were the household members making energy-saving decisions for the assigned home, as outlined in the experiment instruction. However, they were not the actual residents of the home and therefore lacked firsthand experience with its energy use patterns and behavioral routines. This gap may have limited their ability to question, reflect upon, or refine during their exchanges with GPT, potentially leading to reduced interaction volume and fewer opportunities for reasoning. Second, this the experimental design provided a single interaction session, which may have restricted participants’ ability to experiment with prompt varia- tions or build more advanced interactions over time. This one-shot format may have discouraged explora- tory prompting and reinforced the use of short, directive inputs, meaning that the observed passivity in user behavior may partly reflect the constraints of the study protocol rather than participants' inherent interaction tendencies. Third, domain knowledge and AI literacy were assessed through self-reported survey items; hence, the inherent subjectivity of this evaluation approach should be acknowledged. Also, we divided them based on median values in two dimensions, which may have obscured differences among them. However, as noted above, perceived competence could be the more relevant construct in this human-AI interaction context as users’ beliefs about their own capabilities directly influence their interaction behaviors with AI systems. Additionally, while standardized AI literacy instruments have been proposed in general 25 educational contexts [50], no validated scale currently exists that jointly captures domain knowledge in building energy management and AI literacy specific to LLM-based conversational tools. The survey items used in this study were therefore designed to reflect the constructs most relevant to this experimental context, and future work could benefit from developing and validating domain-specific measurement instruments. Fourth, this study relied on a single household energy dataset with a specific appliance configuration, cli- mate context, and TOU rate structure. This design choice was intentional: providing all participants with identical data ensured that observed differences in interaction behaviors could be attributed to participant characteristics, especially domain knowledge and AI literacy, rather than variation in the underlying energy data. However, we acknowledge that the specific energy analysis outcomes, such as which appliances were identified and the nature of the energy-saving recommendations generated, may be influenced by the sali- ence of particular appliances (e.g., HVAC, pool pump, EV charger) and the characteristics of the dataset used. Fifth, we employed ChatGPT-4o as our engine but given the rapid pace of advancements in this field, responses generated by more advanced LLMs may differ. Nonetheless, our core contribution remains valid, informative, and robust. One technical limitation worth noting was our use of JSON files in data processing. Finally, although JSON files facilitated structured data extraction and analysis, they prevented us from cap- turing and analyzing the visualizations that participants used to interpret the energy use data during the experiment. We decided not to overcomplicate our analysis by assessing these visual outputs as textual interactions could explain what was visualized. However, we acknowledge that incorporating visualization data could offer richer insights into how visual aids influence user reasoning, strategy development, and overall engagement with LLM. To continue our contribution to realizing LLM-integrated BEMS, several directions for future research are planned. First, we will explore additional analysis of our dataset by incorporating participants’ demographic information in the pre-screening survey (age, education level, gender, and ethnicity) to examine whether demographic heterogeneity contributes to differences in interactions within LLM-integrated BEMS. Sec- ond, we plan to investigate the potential of RAG or agentic frameworks in LLM-integrated BEMS, which can supplement LLMs with relevant external information (e.g., policy or rate updates) and can empower LLMs to autonomously plan and execute multistep tasks, guiding users through complex energy analyses. Third, future research can recruit actual homeowners to interact with the LLM-integrated BEMS about their own energy consumption over time to better simulate resident knowledge and promote intrinsic motivation, and address researchers to better capture the full range of human-AI interaction behaviors and assess whether sustained engagement leads to deeper collaboration and improved outcomes over time. Acknowledgment This material is based upon work supported by Salt River Project under grant SRP UA-08 24-25 and the National Science Foundation under grant #2519054. We would like to appreciate our SRP advisor, Victor J. Berrios, who provided external evaluation of our methodological approach. In addition, we appreciate all participants to the experiment. Declaration of generative AI and AI-assisted technologies in the writing process During the preparation of this work the authors used ChatGPT 4o and Claude Sonnet 4 in order to proofread sentences and write Python codes to generate figures. After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication. Appendix A. Pre-screening survey This pre-screening survey is to understand your background information, knowledge about building energy, and experiences with large language model (LLM) tools like ChatGPT. We will collect your email address to schedule the actual experiment. We plan to conduct experiments using the stratified sampling approach based on the provided information. Background 26 Q1: What is your age range? 1) 18-24 2) 25-29 3) 30-34 4) 35-39 5) 40-44 6) 45-49 7) 50-54 8) 55-59 9) 60 and above Q2: Which gender do you identify yourself? 1) Male 2) Female 3) Non-binary / third gender 4) Prefer not to say Q3: Are you of Hispanic, LatinX, or Spanish origin? 1) No 2) Yes Q4: How would you describe your ethnicity? 1) White 2) Black or African American 3) American Indian or Alaska Native 4) Asian 5) Native Hawaiian or Pacific Islander 6) From multiple races 7) Some other races (please specify) ____________________ 8) Prefer not to disclose Q5: What is the highest level of education you have completed or the highest degree you have received? 1) Less than high school degree 2) High school degree or equivalent 3) Some college but no degree 4) Associate degree 5) Bachelor degree 6) Master degree 7) Doctoral degree Understanding of Building Energy Q6: Are you responsible for paying the monthly utility bills (e.g., electricity, gas, water) of your residence? 1) Yes 2) No Q7: Please rate your understanding of home energy use and its patterns 1) No understanding 2) Basic understanding 3) Moderate understanding 4) Good understanding 5) Complete understanding Q8: Please rate your understanding of a building system and appliance from the perspective of energy consumption. No understanding Basic understanding Moderate understanding Good understanding Complete understanding 27 HVAC systems Lighting systems Building envelope (e.g., (walls, windows, roof, floors) Appliances (e.g., comput- ers, monitors, refrigera- tors, etc.) Q8: Are you familiar with time-of-use rates? 1) Definitely not 2) Probably not 3) Probably yes 4) Definitely yes Experience in GenAI Q9: How often do you use Generative Artificial Intelligence (GenAI) tools such as ChatGPT, Claude, or Gemini in a typical day? 1) Never 2) Rarely (a few times a week) 3) Sometimes (a few times a day) 4) Frequently (several times a day) 5) Constantly (multiple times an hour) Q10: How long have you been using these GenAI tools? 1) less than a month 2) 1-3 months 3) 4-6 months 4) 7-12 months 5) 1-2 year(s) 6) More than 2 years Q11: Please rate your level of expertise in using GenAI tools like ChatGPT. 1) Beginner 2) Lower intermediate 3) Intermediate 4) Upper intermediate 5) Expert Q12: Please rank your familiarity with prompt engineering 1) Not at all familiar 2) Slightly familiar 3) Somewhat familiar 4) Moderately familiar 5) Extremely familiar B. Experiment Instruction In this experiment, you will play a role-playing game. Role: - You are a homeowner in Austin, Texas (Hot and Humid) and currently collecting circuit-, appliance- level power use data (a 15-min interval) for two months (July and August). Below is the list of circuits and appliances that you have data. 28 o Appliances: HVAC unit, electric vehicle charger, washing machine, dishwasher, clothes dryer, microwave, oven 1, oven 2, pool pump, cooktop, refrigerator, and electric water heater o Circuits: Bathroom, bedroom, living room, and kitchen. - You are responsible for this house’s utility bills and currently participating in a time-of-use (TOU) rate (more details below). o Time-of-use (TOU) rate: an electricity pricing structure where the cost of electricity varies de- pending on the time of day you use it. The figure below shows your TOU rate during summer and what you are using in your house. (Figure 3 is presented here) Goal: - Your goal is to find the most effective five behavioral changes to reduce your energy bills through a conversation with ChatGPT. Specifically, please interact with ChatGPT to identify potential five behav- ior-based energy-saving strategies of your house using the collected data. Caveat & Restriction: - You also own other appliances in your house (you can imagine your house), but their energy use was not measured and collected. - The house energy data reflects your normal energy consumption behavior. - The behavioral changes should not significantly compromise your comfort or normal routines (e.g., de- laying meals by two hours from your regular schedule or disruptions to your thermal comfort). - Behavioral changes do not mean that you intend to adopt new technology. When the experimenter gives the signal, you may begin your interaction with ChatGPT. Post Survey: - After the role-playing experiment, you will take the post survey using Qualtrics and this will be the last step for you! C. GPT Model Selection Process Among high-performance GPT models, in Table A2, available during the time when the experimental study was conducted. We selected the OpenAI GPT 4o model for our experiment due to its superior multimodal capabilities, processing efficiency, and improved contextual understanding compared to other versions. Given that participants needed to analyze energy consumption data and provide calculation-based insights dynamically, GPT-4o’s reasoning ability using user-provided datasets made it the most suitable selection. Additionally, GPT-4o provided lower latency and rapid responsiveness that ensured seamless and smoother user experiences, especially in a real-time role-playing interaction based on screen-sharing or remote-con- trol environment while minimizing the technical disruptions. Table A1. GPT models available during the experiment Model Release date Description Ref GPT 4o May 13, 2024 A flagship multimodal model optimized for complex reasoning, creativity, and high-performance capabilities. In particular, it excels in data analytics of user-provided data, providing real-time seamless interaction. [82] GPT 4o- mini Jul 18, 2024 A lightweight and faster version of GPT-4o, designed for efficiency and cost-effectiveness. However, it has reduced capacity for highly complex reasoning tasks, and multimodal scalability. [83] GPT o1- mini Sep 12, 2024 A compact and highly efficient variant of GPT-o1, ideal for quick re- sponses and lightweight application. However, it is not optimized for deep reasoning, large context handling, and real-time analytical interac- tion. [84] D. Experiment Participant Demographic Information The study cohort was demographically well-balanced, with participants representing a broad mix of age groups, educational backgrounds, ethnicities, and gender, as shown in Figure A1. This diversity enhanced 29 the generalizability of the study and supported a comprehensive understanding of how users with varied backgrounds interact with LLM-integrated BEMS. We did not include this information since they were not included in the analysis. Figure A1. Participant demographic information E. SCALE Framework: Prompt and Rubric Figure A2. Prompt for the SCALE framework We developed a prompt, as shown in Figure A2, to implement the SCALE framework. The entire code for this GPT-based SCALE framework is shared in the following link: 38 18 11 9 5 1 3 0 10 20 30 40 50 60 70 18-2425-2930-3435-3940-4445-4960 and above Number of responses Age group 44 40 1 0 10 20 30 40 50 60 70 FemaleMalePrefer not to say Number of responses Gender 7 5 22 15 21 12 0 5 10 15 20 25 30 35 High school degree or equivalent Associate degree Some college but no degree Bachelor degree Master degree Doctoral degree Number of responses Education 32 24 14 3 5 7 0 5 10 15 20 25 30 35 AsianWhiteBlack or African American Multiple races Some other races Prefer not to disclose Number of responses Ethnicity 19 66 0 10 20 30 40 50 60 70 YesNo Number of responses Hispanic 30 - https://github.com/humanbuildingsynergy/HUBS_GPT-based_SCALE_Framework. For the reproducibility of the SCALE framework, we developed scoring rubrics as organized in Table A3. Table A2. Four scoring factors in the proposed context-aware multi-factor scoring framework. Factor #1: Explicitness Description How directly and clearly the concept is mentioned using appropriate terminology Scoring criteria for user prompts 1.0: Domain terminology used correctly in context 0.8: General energy terms used appropriately in context 0.6: Related terms that imply concept understanding 0.4: Indirect references requiring some inference 0.2: Vague connection or unclear terminology 0.0: No mention or connection Scoring criteria for GPT responses 1.0: Precise domain terminology with technical accuracy (kWh, TOU rates, HVAC) 0.8: Appropriate technical terms with clear explanations 0.6: General energy terminology used correctly 0.4: Basic energy terms with some accuracy 0.2: Imprecise or unclear terminology 0.0: No meaningful terminology used Factor #2: Depth Description Quality and authenticity of engagement with the concept Scoring criteria for user prompts 1.0: Deep contextual engagement with detailed reasoning, constraints, or comprehensive understanding 0.8: Substantial contextual engagement with clear reasoning or thoughtful analysis 0.6: Moderate contextual engagement with some reasoning or understanding 0.4: Basic contextual engagement with minimal reasoning or surface-level understanding 0.2: Shallow engagement with little reasoning or very limited understanding 0.0: No meaningful engagement demonstrated Scoring criteria for GPT responses 1.0: Comprehensive multi-faceted analysis examining multiple dimensions of the concept 0.8: Thorough analysis exploring several aspects or implications of the concept 0.6: Moderate analysis covering key aspects with reasonable detail 0.4: Basic analysis touching on main points with limited development 0.2: Superficial analysis with minimal exploration or development 0.0: No meaningful analytical processing demonstrated Factor #3: Consideration Description Whether the concept was meaningfully present in the conversation Scoring criteria for user prompts 1.0: Concept clearly influences user’s decisions, preferences, or thinking 0.8: Concept is actively considered in relation to user’s situation 0.6: Concept is acknowledged and shows contextual relevance 0.4: Concept is mentioned with minimal contextual connection 0.2: Concept is barely acknowledged or referenced 0.0: Concept is not considered in user’s thinking Scoring criteria for GPT responses 1.0: Concept is fully integrated into analysis and recommendations 0.8: Concept is clearly incorporated into response strategy 0.6: Concept is meaningfully addressed in the analysis 0.4: Concept is mentioned but not well integrated 0.2: Concept is briefly touched upon 0.0: Concept is not considered in the response Factor #4: Evidence Description Quality and authenticity of supporting evidence provided Scoring criteria for user prompts 1.0: Contextual examples, specific constraints, or experimental details 0.8: Clear situational context or constraints with details 0.6: General contextual examples or reasonable situational context 0.4: Some contextual information or basic examples 0.2: Minimal supporting contextual information 0.0: No contextual evidence or examples provided 31 Scoring criteria for GPT responses 1.0: Multiple high-quality sources: quantitative data + domain expertise + specific exam- ples 0.8: Strong primary source with additional supporting information 0.6: Solid single source with reasonable supporting details 0.4: Basic source material with minimal additional support 0.2: Weak or limited source material with little support 0.0: No credible supporting information provided F. Energy Saving Strategy Comparison Figure A3 presents the prompt used to compare two sets of strategies drawn from human-AI interactions and expert judgments. Figure A3. Prompt to guide how two sets of strategies (user-GPT vs. reference) should be compared G. Post-Experiment Survey This post survey aims to understand any changes in your knowledge of home energy use, your expertise in GenAI tools like ChatGPT, and your perspectives on both the benefits of GenAI and its potential for managing home energy use. Q1: Please rate your understanding of home energy use and its patterns after interactions with ChatGPT. 1) No understanding 2) Basic understanding 3) Moderate understanding 4) Good understanding 5) Advanced understanding Q2: Please rate your understanding of how monthly energy bills are calculated. 1) No understanding 2) Basic understanding 3) Moderate understanding 4) Good understanding 5) Advanced understanding Q3: How much do you agree with the following statement: I found it difficult to understand ChatGPT’s answers because of my limited knowledge regarding home energy. 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 32 5) Strongly agree Q4: How much do you agree with the following statement: My interactions with ChatGPT have improved my knowledge of home energy use, its pattern, and the potential savings achievable through behavioral changes. 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 5) Strongly agree Q5: Please rate your level of expertise in using GenAI tools like ChatGPT after your role-playing experiment 1) Beginner 2) Lower intermediate 3) Intermediate 4) Upper intermediate 5) Expert Q6: How much do you agree with the following statement: I fully utilized ChatGPT's capabilities. 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 5) Strongly agree Q7: I wish I knew some techniques to enhance ChatGPT's responses. 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 5) Strongly agree Q8: How much do you agree with the following statement: Through my interactions with ChatGPT, I was able to arrive at better answers in the role-playing experiment 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 5) Strongly agree Q9: How much do you agree with the following statement: With ChatGPT, I could incorporate expert-level knowledge/analysis into my answers. 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 5) Strongly agree Q10: How much do you agree with the following statement: If I could manage my home energy usage, patterns and bills through interactions with ChatGPT – similar to the role-play experiment – I would do so. 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 5) Strongly agree Q11: How much do you agree with the following statement: If I could use GenAI tools like ChatGPT for my home 33 energy use, I think my understanding of home energy use and its patterns will improve consistently and continu- ously. 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 5) Strongly agree H. Representative Conversation Examples. Figures from A4 to A6 illustrate the overview of the conversations analyzed in Section 4.1. Figure A4. Minimal user engagement, but high GPT performance in a participant-GPT collaboration Figure A5. Low to moderate user engagement and low GPT performance in a participant-GPT collaboration. Prompt #1: Initial Request Help me to find the most effective five behavioral changes to my energy bills this is good Prompt #2: Suggestion Acceptance •Analyzed real-time energy data - hourly averages of appliances •Cross-referenced consumption patterns with TOU rate schedule •Considered Austin's hot/humid climate factors, •Examined hourly usage variations across appliances •Prioritized changes that don't affect comfort or routines • Initial five recommendations : Splitting pool pump runtime, Pre- cool home strategy, EV charger scheduling, water heater optimization,ceiling fan + setpoint adjustment Response #1: Initial Recommendations Response #2:Follow-up suggestions •Offered follow-ups: show time-series plots, estimate monthly savings, build custom schedules, help communicate these strategies to others Comprehensive analysis No follow-up questions or clarification request - Minimal user engagement 80% alignment with expert-judged solutions Note: Blue texts: Interpretation of user engagement and GPT responses User prompts reflect participant’s exact wording, while GPT responses are presented as concise summaries due to their length and complexity. Note: Blue texts: Interpretation of user engagement and GPT responses User prompts reflect participant’s exact wording, while GPT responses are presented as concise summaries due to their length and complexity. This figure used an icon from Flaticon.com Prompt #1: Initial Request Please provide me 5 behavioral changes I can make to reduce my energy bill. follow up question: what are the low TOU rates periods for optimal saving? what hours specifically should I use my hair dryer or turn on the pool pump?" Prompt #2: Clarification Request •Analyzed energy usage data, TOU rates • Initial five recommendations: shifting bathroom and bedroom energy use, manage kitchen appliances, optimizing lighting and entertainment in the living room, unplug electronics in standby mode in key areas, adjust pool pump operation Response #1: Initial Recommendations Response #2:Clarification •Provided the TOU schedule and identified lowest rate periods •Hair dryer: Early morning before 8AM or after 8PM •Pool pump: Midnight to 8AM (overnight optimal) How much money will each of the 5 behavioral changes save me in my energy bill? Prompt #3: Clarification RequestResponse #3:Clarification •Calculated energy usage shifts to lower TOU periods •Applied rate differences between peak/low periods •Provided detailed breakdown table for each change What would be the total savings in 1 year if I utilized these behavioral changes? Prompt #4: Clarification RequestResponse #4:Clarification •Calculated estimated total savings: $147.75 Finally, please summarize the 5 behavioral changes, including the times of use Prompt #5: Final ConfirmationResponse #5:Clarification Final five recommendations provided: The same strategies mentioned in the first response above. Mismatch between analysis and recommendations - Suboptimal GPT performance (20% of alignment rate with expert solutions) Not evaluating or refining initial recommendations Note: Blue texts: Interpretation of user engagement and GPT responses User prompts reflect participant’s exact wording, while GPT responses are presented as concise summaries due to their length and complexity. 34 Figure A6. High-quality, efficient user engagement and GPT’s appropriate performance in a participant-GPT col- laboration. References 1. H. Zhu, X. Lian, Y. Liu, Y. Zhang, and Z. Li. (2023). Consideration of occupant preferences and habits during the establishment of occupant-centric buildings: A critical review. Energy and Buildings. 280: p. 112720.♢doi:♢https://doi.org/10.1016/j.enbuild.2022.112720. 2. Tucson Electric Power (TEP). TEP mobile app. n.d. [cited 2024 12/12]; Available: https://w.tep.com/mobileapp/. 3. Salt River Project (SRP). SRP mobile apps. n.d. [cited 2024 12/24]; Available: https://w.srpnet.com/account/mobile-apps. 4. U.S. Department of Energy. Demand Response and Time-Variable Pricing Programs. n.d. [cited 2025 07/22]; Available: https://w.energy.gov/femp/demand-response-and-time-variable-pricing-programs. 5. Tucson Electric Power (TEP). Time-of-Use. n.d. [cited 2025 07/22]; Available: https://w.tep.com/time-of-use/. 6. Salt River Project (SRP). SRP Time-of-Use Price Plan (TOU). n.d. [cited 2025 07/22]; Available: https://w.srpnet.com/price-plans/residential-electric/time-of-use. 7. T. He and F. Jazizadeh. LLM-based Building Energy Management Assistants. in Computing in Civil Engineering 2024. (2024). Pittsburgh, PA, USA. 8. T. He and F. Jazizadeh. (2025). Context-aware LLM-based AI Agents for Human-centered Energy Management Systems in Smart Buildings. arXiv preprint arXiv:2512.25055. 9. J. Rey-Jouanchicot, A. Bottaro, E. Campo, J.-L. Bouraoui, N. Vigouroux, and F. Vella. (2024). Leveraging Large Language Models for enhanced personalised user experience in Smart Homes. arXiv preprint arXiv:2407.12024. 10. OpenAI. Prompt Engineering. n.d. [cited 2024 12/12]; Available: 35 https://platform.openai.com/docs/guides/prompt-engineering. 11. T.K.F. Chiu, Z. Ahmad, M. Ismailov, and I.T. Sanusi. (2024). What are artificial intelligence literacy and competency? A comprehensive framework to support them. Computers and Education Open. 6: p. 100171.♢doi:♢https://doi.org/10.1016/j.caeo.2024.100171. 12. M. Dikmen and C. Burns. (2022). The effects of domain knowledge on trust in explainable AI and task performance: A case of peer-to-peer lending. International Journal of Human-Computer Studies. 162: p. 102792.♢doi:♢https://doi.org/10.1016/j.ijhcs.2022.102792. 13. N. Knoth, A. Tolzin, A. Janson, and J.M. Leimeister. (2024). AI literacy and its implications for prompt engineering strategies. Computers and Education: Artificial Intelligence. 6: p. 100225.♢doi:♢https://doi.org/10.1016/j.caeai.2024.100225. 14. S. Fuchs, Natural language processing for building code interpretation: systematic literature review report. (2021), The University of Auckland: Auckland. 15. A. Khanuja, Leveraging Natural Language Processing and Large Language Models to Understand, Categorize, and Standardize Building Energy Efficiency Measures. (2024), University of Cincinnati: Cincinnati. 16. Y. Lai, S. Papadopoulos, F. Fuerst, G. Pivo, J. Sagi, and C.E. Kontokosta. (2022). Building retrofit hurdle rates and risk aversion in energy efficiency investments. Applied Energy. 306: p. 118048.♢doi:♢https://doi.org/10.1016/j.apenergy.2021.118048. 17. S. Hong, J. Kim, and E. Yang. (2022). Automated Text Classification of Maintenance Data of Higher Education Buildings Using Text Mining and Machine Learning Techniques. Journal of Architectural Engineering. 28(1): p. 04021045.♢doi:♢https://doi.org/10.1061/(ASCE)AE.1943-5568.0000522. 18. O.I. Olanrewaju, W.I. Enegbuma, and M. Donn. (2024). Operational, embodied and whole life cycle assessment credits in green building certification systems: Desktop analysis and natural language processing approach. Building and Environment. 258: p. 111569.♢doi:♢https://doi.org/10.1016/j.buildenv.2024.111569. 19. K. Park, J.S. Hong, and W. Kim. (2020). A Methodology Combining Cosine Similarity with Classifier for Text Classification. Applied Artificial Intelligence. 34(5): p. 396– 411.♢doi:♢https://doi.org/10.1080/08839514.2020.1723868. 20. H. Kim, S. Ham, M. Promann, H. Devarapalli, G. Bihani, T. Ringenberg, V. Kwarteng, I. Bilionis, J.E. Braun, J.T. Rayz, L. Raymond, T. Reimer, and P. Karava. (2022). MySmartE – An eco-feedback and gaming platform to promote energy conserving thermostat-adjustment behaviors in multi-unit residential buildings. Building and Environment. 221: p. 109252.♢doi:♢https://doi.org/10.1016/j.buildenv.2022.109252. 21. S. Kumar, S. Benedict, and S. Ajith. Application of Natural Language Processing and IoTCloud in Smart Homes. in 2019 2nd International Conference on Intelligent Communication and Computational Techniques (ICCT). (2019). 22. J. Lu, X. Tian, C. Zhang, Y. Zhao, J. Zhang, W. Zhang, C. Feng, J. He, J. Wang, and F. He. (2024). Evaluation of large language models (LLMs) on the mastery of knowledge and skills in the heating, ventilation and air conditioning (HVAC) industry. Energy and Built Environment.♢doi:♢https://doi.org/10.1016/j.enbenv.2024.03.010. 23. C. Zhang, J. Zhang, Y. Zhao, and J. Lu. (2024). Automated data mining framework for building energy conservation aided by generative pre-trained transformers (GPT). Energy and Buildings. 305: p. 113877.♢doi:♢https://doi.org/10.1016/j.enbuild.2023.113877. 24. S. Choi and S. Yoon. (2024). GPT-based data-driven urban building energy modeling (GPT-UBEM): Concept, methodology, and case studies. Energy and Buildings. 325: p. 115042.♢doi:♢https://doi.org/10.1016/j.enbuild.2024.115042. 25. S. Chen, X. Liang, Y. Liu, X. Li, X. Jin, and Z. Du. (2025). Customized large-scale model for human- AI collaborative operation and maintenance management of building energy systems. Applied Energy. 393: p. 126169.♢doi:♢https://doi.org/10.1016/j.apenergy.2025.126169. 36 26. J. Zhang, C. Zhang, J. Lu, and Y. Zhao. (2025). Domain-specific large language models for fault diagnosis of heating, ventilation, and air conditioning systems by labeled-data-supervised fine-tuning. Applied Energy. 377: p. 124378.♢doi:♢https://doi.org/10.1016/j.apenergy.2024.124378. 27. C. Zhang, J. Lu, and Y. Zhao. (2024). Generative pre-trained transformers (GPT)-based automated data mining for building energy management: Advantages, limitations and the future. Energy and Built Environment. 5(1): p. 143–169.♢doi:♢https://doi.org/10.1016/j.enbenv.2023.06.005. 28. M. Liu, L. Zhang, J. Chen, W.-A. Chen, Z. Yang, L.J. Lo, J. Wen, and Z. O’Neill. (2025). Large language models for building energy applications: Opportunities and challenges. Building Simulation. 18(2): p. 225–234.♢doi:♢https://doi.org/10.1007/s12273-025-1235-9. 29. J. Hwang and S. Yoon. (2025). AI agent-based indoor environmental informatics: Concept, methodology, and case study. Building and Environment. 277: p. 112879.♢doi:♢https://doi.org/10.1016/j.buildenv.2025.112879. 30. S. Yoon, J. Song, and J. Li. (2025). Ontology-enabled AI agent-driven intelligent digital twins for building operations and maintenance. Journal of Building Engineering. 108: p. 112802.♢doi:♢https://doi.org/10.1016/j.jobe.2025.112802. 31. M. Nourani, J. King, and E. Ragan. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. in Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. (2020). 32. K. Inkpen, S. Chappidi, K. Mallari, B. Nushi, D. Ramesh, P. Michelucci, V. Mandava, L.H. Vepřek, and G. Quinn. (2023). Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making. ACM Trans. Comput.-Hum. Interact. 30(5): p. Article 71.♢doi:♢https://doi.org/10.1145/3534561. 33. G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M.T. Ribeiro, and D. Weld, Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance, in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. (2021), Association for Computing Machinery: Yokohama, Japan. p. Article 81. 34. M.C. Laupichler, A. Aster, J. Schirch, and T. Raupach. (2022). Artificial intelligence literacy in higher and adult education: A scoping literature review. Computers and Education: Artificial Intelligence. 3: p. 100101.♢doi:♢https://doi.org/10.1016/j.caeai.2022.100101. 35. B. Wang, P.-L.P. Rau, and T. Yuan. (2023). Measuring user competence in using artificial intelligence: validity and reliability of artificial intelligence literacy scale. Behaviour & information technology. 42(9): p. 1324–1337.♢doi:♢https://doi.org/10.1080/0144929X.2022.2072768. 36. J. Kim, S. Yu, S.-S. Lee, and R. Detrick. (2025). Students’ prompt patterns and its effects in AI-assisted academic writing: Focusing on students’ level of AI literacy. Journal of Research on Technology in Education: p. 1–18.♢doi:♢https://doi.org/10.1080/15391523.2025.2456043. 37. X. Liu, L. Zhang, and X. Wei. (2025). Generative Artificial Intelligence Literacy: Scale Development and Its Effect on Job Performance. Behavioral Sciences. 15(6): p. 811.♢doi:♢https://doi.org/10.3390/bs15060811. 38. H. Aguinis and K.J. Bradley. (2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational research methods. 17(4): p. 351– 371.♢doi:♢https://doi.org/10.1177/1094428114547952. 39. M. Afzalan and F. Jazizadeh. (2019). Residential loads flexibility potential for demand response using energy consumption patterns and user segments. Applied Energy. 254: p. 113693.♢doi:♢https://doi.org/10.1016/j.apenergy.2019.113693. 40. A. Chakraborty and A. Pfaelzer. (2011). An overview of standby power management in electrical and electronic power devices and appliances to improve the overall energy efficiency in creating a green world. Journal of Renewable and Sustainable Energy. 3(2).♢doi:♢https://doi.org/10.1063/1.3558870. 41. J. Ross and A. Meier, Whole-house measurements of standby power consumption, in Energy efficiency in household appliances and lighting. (2006), Springer. p. 278–285. doi: https://doi.org/10.1007/978-3- 37 642-56531-1_33. 42. Pecan Street Inc. Better tech, better decisions, a better world – we’re making it happen. 2025 [cited 2025 07/07]; Available: https://w.pecanstreet.org/. 43. OpenEI. Utility Rate Database. n.d. [cited 01/31 2025]; Available: https://openei.org/wiki/Utility_Rate_Database. 44. R. Rawal, M. Schweiker, O.B. Kazanci, V. Vardhan, Q. Jin, and L. Duanmu. (2020). Personal comfort systems: A review on comfort, energy, and economics. Energy and Buildings. 214: p. 109858.♢doi:♢https://doi.org/10.1016/j.enbuild.2020.109858. 45. A. Bandura. (1997). Self-efficacy: The exercise of control. Personnel Psychology. 50(3): p. 801–804. 46. D.R. Compeau and C.A. Higgins. (1995). Computer self-efficacy: Development of a measure and initial test. MIS quarterly. 19(2): p. 189–211.♢doi:♢https://doi.org/10.2307/249688. 47. J.M. Logg, J.A. Minson, and D.A. Moore. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes. 151: p. 90– 103.♢doi:♢https://doi.org/10.1016/j.obhdp.2018.12.005. 48. S. Gaube, H. Suresh, M. Raue, A. Merritt, S.J. Berkowitz, E. Lermer, J.F. Coughlin, J.V. Guttag, E. Colak, and M. Ghassemi. (2021). Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ digital medicine. 4(1): p. 31.♢doi:♢https://doi.org/10.1038/s41746-021-00385-9. 49. T. Lintner. (2024). A systematic review of AI literacy scales. npj Science of Learning. 9(1): p. 50.♢doi:♢https://doi.org/10.1038/s41539-024-00264-4. 50. D.T.K. Ng, W. Wu, J.K.L. Leung, T.K.F. Chiu, and S.K.W. Chu. (2024). Design and validation of the AI literacy questionnaire: The affective, behavioural, cognitive and ethical approach. British Journal of Educational Technology. 55(3): p. 1082–1104.♢doi:♢https://doi.org/10.1111/bjet.13411. 51. N. Baumgarten, B. Meyer, and D. Özçetin. (2008). Explicitness in translation and interpreting: A critical review and some empirical evidence (of an elusive concept). Across Languages and Cultures. 9(2): p. 177–203.♢doi:♢https://doi.org/10.1556/Acr.9.2008.2.2. 52. H.H. Clark and S.E. Brennan, Grounding in communication. Perspectives on socially shared cognition. (1991): American Psychological Association. 53. F.I. Craik and R.S. Lockhart. (1972). Levels of processing: A framework for memory research. Journal of verbal learning and verbal behavior. 11(6): p. 671–684. 54. F.I. Craik. (2002). Levels of processing: Past, present... and future? Memory. 10(5-6): p. 305– 318.♢doi:♢https://doi.org/10.1080/09658210244000135. 55. V. Braun and V. Clarke, Thematic analysis, in Encyclopedia of quality of life and well-being research. (2024), Springer. p. 7187–7193. doi: 56. A.D. Shocker, M. Ben-Akiva, B. Boccara, and P. Nedungadi. (1991). Consideration set influences on consumer decision-making and choice: Issues, models, and suggestions. Marketing Letters. 2(3): p. 181– 197.♢doi:♢https://doi.org/10.1007/BF00554125. 57. B. Johnstone and J. Andrus, Discourse analysis. (2024): John Wiley & Sons. 58. P.G. Keen, Decision support systems: a research perspective. Decision support systems: Issues and challenges: Proceedings of an international task force meeting. (1980). 23–44. 59. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, and E. Xing. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems. 36: p. 46595–46623. 60. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu. (2023). G-eval: NLG evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. 61. P.E. McKight and J. Najab. (2010). Kruskal‐wallis test. The corsini encyclopedia of psychology: p. 1– 1.♢doi:♢https://doi.org/10.1002/9780470479216.corpsy0491. 62. P.E. McKnight and J. Najab. (2010). Mann‐whitney U test. The Corsini encyclopedia of psychology: p. 1–1.♢doi:♢https://doi.org/10.1002/9780470479216.corpsy0524. 63. E.E. Cureton. (1956). Rank-biserial correlation. Psychometrika. 21(3): p. 287– 38 290.♢doi:♢https://doi.org/10.1007/BF02289138. 64. T.J. VanderWeele and M.B. Mathur. (2019). Some desirable properties of the Bonferroni correction: is the Bonferroni correction really so bad? American journal of epidemiology. 188(3): p. 617– 618.♢doi:♢https://doi.org/10.1093/aje/kwy250. 65. R. Parasuraman and V. Riley. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors. 39(2): p. 230–253.♢doi:♢https://doi.org/10.1518/001872097778543886. 66. Z. Buçinca, M.B. Malaya, and K.Z. Gajos. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proc. ACM Hum.-Comput. Interact. 5(CSCW1): p. Article 188.♢doi:♢https://doi.org/10.1145/3449287. 67. L. Tankelevitch, V. Kewenig, A. Simkute, A.E. Scott, A. Sarkar, A. Sellen, and S. Rintel, The Metacognitive Demands and Opportunities of Generative AI, in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. (2024), Association for Computing Machinery: Honolulu, HI, USA. p. Article 680. 68. C. Rastogi, Y. Zhang, D. Wei, K.R. Varshney, A. Dhurandhar, and R. Tomsett. (2022). Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making. Proc. ACM Hum.-Comput. Interact. 6(CSCW1): p. Article 83.♢doi:♢https://doi.org/10.1145/3512930. 69. R. Fogliato, S. Chappidi, M. Lungren, P. Fisher, D. Wilson, M. Fitzke, M. Parkinson, E. Horvitz, K. Inkpen, and B. Nushi, Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging, in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. (2022), Association for Computing Machinery: Seoul, Republic of Korea. p. 1362–1374. 70. H.A. Simon. (1956). Rational choice and the structure of the environment. Psychological review. 63(2): p. 129.♢doi:♢https://doi.org/10.1037/h0042769. 71. D.T.K. Ng, J.K.L. Leung, K.W.S. Chu, and M.S. Qiao. (2021). AI Literacy: Definition, Teaching, Evaluation and Ethical Issues. Proceedings of the Association for Information Science and Technology. 58(1): p. 504–509.♢doi:♢https://doi.org/10.1002/pra2.487. 72. K.Z. Gajos and L. Mamykina, Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning, in Proceedings of the 27th International Conference on Intelligent User Interfaces. (2022), Association for Computing Machinery: Helsinki, Finland. p. 794–806. 73. J. Sweller. (1988). Cognitive load during problem solving: Effects on learning. Cognitive science. 12(2): p. 257–285.♢doi:♢https://doi.org/10.1016/0364-0213(88)90023-7. 74. J. Kruger and D. Dunning. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of personality and social psychology. 77(6): p. 1121–1134. 75. A. Chatterji, T. Cunningham, D.J. Deming, Z. Hitzig, C. Ong, C.Y. Shan, and K. Wadman, How people use chatgpt. (2025), National Bureau of Economic Research. 76. Z. Xue, L. Li, S. Tian, X. Chen, P. Li, L. Chen, T. Jiang, and M. Zhang, Domain Knowledge is All You Need: A Field Deployment of LLM-Powered Test Case Generation in FinTech Domain, in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. (2024), Association for Computing Machinery: Lisbon, Portugal. p. 314–315. 77. A. Bradford, W. Li, L. Gerard, and M.C. Linn, Comparing Expert and ChatGPT-authored Guidance Prompts, in Proceedings of the Eleventh ACM Conference on Learning @ Scale. (2024), Association for Computing Machinery: Atlanta, GA, USA. p. 388–392. 78. J.D. Zamfirescu-Pereira, R.Y. Wong, B. Hartmann, and Q. Yang, Why Johnny Can’t Prompt: How Non- AI Experts Try (and Fail) to Design LLM Prompts, in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. (2023), Association for Computing Machinery: Hamburg, Germany. p. Article 437. 79. Q.V. Liao and J.W. Vaughan. (2023). Ai transparency in the age of llms: A human-centered research roadmap. arXiv preprint arXiv:2306.01941. 10. 80. Y. Zhang, Q.V. Liao, and R.K.E. Bellamy, Effect of confidence and explanation on accuracy and trust 39 calibration in AI-assisted decision making, in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. (2020), Association for Computing Machinery: Barcelona, Spain. p. 295–305. 81. Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, and H. Wang. (2023). Retrieval- augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997. 2(1). 82. A. Hurst, A. Lerer, A.P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, and A. Radford. (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276. 83. Z. Xie and C. Wu. (2024). Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities. arXiv preprint arXiv:2410.11190. 84. OpenAI. OpenAI o1-mini. 2024 [cited 2025 01/17]; Available: https://openai.com/index/openai-o1- mini-advancing-cost-efficient-reasoning/.

Instant research discovery