Paper deep dive
SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts
Qingsong Zou, Zhi Yan, Zhiyao Xu, Kuofeng Gao, Jingyu Xiao, Yong Jiang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 1:28:07 PM
Summary
The paper introduces SmartBench, the first benchmark dataset for evaluating Large Language Models (LLMs) in detecting anomalous device states and behavioral contexts within smart home environments. It distinguishes between context-independent anomalies (static snapshots) and context-dependent anomalies (temporal sequences). The authors evaluate 13 mainstream LLMs, finding that even state-of-the-art models like Claude-Sonnet-4.5 struggle with anomaly detection, achieving only 66.1% accuracy on context-independent tasks and 57.8% on context-dependent tasks, highlighting significant gaps in LLM capabilities for smart home security and reliability.
Entities (8)
Relation Signals (8)
SmartBench → contains → Context-Independent Anomalies
confidence 95% · containing both normal and anomalous device states as well as normal and anomalous device state transition contexts.
SmartBench → contains → Context-Dependent Anomalies
confidence 95% · containing both normal and anomalous device states as well as normal and anomalous device state transition contexts.
SmartBench → designedfor → Anomaly Detection
confidence 95% · SmartBench, which is the first smart home dataset designed for LLMs... to enhance the anomaly detection capabilities
SmartBench → evaluates → LLMs
confidence 95% · We evaluate 13 mainstream LLMs on this benchmark.
Claude-Sonnet-4.5 → achievesaccuracy → 57.8%
confidence 90% · performs even worse on context-dependent anomalies, with an accuracy of only 57.8%
Claude-Sonnet-4.5 → achievesaccuracy → 66.1%
confidence 90% · Claude-Sonnet-4.5 achieves only 66.1% detection accuracy on context-independent anomaly categories
GPT-5 → usedfor → Sample Generation
confidence 85% · We use GPT-5 to generate samples from these scenario descriptions.
GPT-5-mini → usedfor → Compliance Checking
confidence 85% · Specifically, we use GPT-5-mini to automatically examine each sample for compliance with the above criteria.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Due to the strong context-awareness capabilities demonstrated by large language models (LLMs), recent research has begun exploring their integration into smart home assistants to help users manage and adjust their living environments. While LLMs have been shown to effectively understand user needs and provide appropriate responses, most existing studies primarily focus on interpreting and executing user behaviors or instructions. However, a critical function of smart home assistants is the ability to detect when the home environment is in an anomalous state. This involves two key requirements: the LLM must accurately determine whether an anomalous condition is present, and provide either a clear explanation or actionable suggestions. To enhance the anomaly detection capabilities of next-generation LLM-based smart home assistants, we introduce SmartBench, which is the first smart home dataset designed for LLMs, containing both normal and anomalous device states as well as normal and anomalous device state transition contexts. We evaluate 13 mainstream LLMs on this benchmark. The experimental results show that most state-of-the-art models cannot achieve good anomaly detection performance. For example, Claude-Sonnet-4.5 achieves only 66.1% detection accuracy on context-independent anomaly categories, and performs even worse on context-dependent anomalies, with an accuracy of only 57.8%. More experimental results suggest that next-generation LLM-based smart home assistants are still far from being able to effectively detect and handle anomalous conditions in the smart home environment. Our dataset is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.06636v1
- Canonical: https://arxiv.org/abs/2603.06636v1
Trouble viewing inline? Open PDF directly →
Full Text
79,711 characters extracted from source content.
Expand or collapse full text
SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts Qingsong Zou ∗ Tsinghua Shenzhen International Graduate School Shenzhen, China zouqs21@mails.tsinghua.edu.cn Zhi Yan ∗ Jilin University Jilin, China yanzhi2422@mails.jlu.edu.cn Zhiyao Xu Tsinghua Shenzhen International Graduate School Shenzhen, China xu-zy25@mails.tsinghua.edu.cn Kuofeng Gao Tsinghua Shenzhen International Graduate School Shenzhen, China gkf24@mails.tsinghua.edu.cn Jingyu Xiao † The Chinese University of Hong Kong Hong Kong, China jyxiao@link.cuhk.edu.hk Yong Jiang Tsinghua Shenzhen International Graduate School Pengcheng Laboratory Shenzhen, China jiangy@sz.tsinghua.edu.cn Abstract Due to the strong context-awareness capabilities demonstrated by large language models (LLMs), recent research has begun exploring their integration into smart home assistants to help users manage and adjust their living environments. While LLMs have been shown to effectively understand user needs and provide appropriate re- sponses, most existing studies primarily focus on interpreting and executing user behaviors or instructions. However, a critical function of smart home assistants is the ability to detect when the home environment is in an anomalous state. This involves two key requirements: the LLM must accurately determine whether an anomalous condition is present, and provide either a clear explanation or actionable suggestions. To enhance the anomaly detection capabilities of next-generation LLM-based smart home assistants, we introduce SmartBench, which is the first smart home dataset designed for LLMs, containing both normal and anomalous device states as well as normal and anoma- lous device state transition contexts. We evaluate 13 mainstream LLMs on this benchmark. The experimental results show that most state-of-the-art models cannot achieve good anomaly detection performance. For example, Claude-Sonnet-4.5 achieves only 66.1% detection accuracy on context-independent anomaly categories, and performs even worse on context-dependent anomalies, with an accuracy of only 57.8%. More experimental results suggest that next-generation LLM-based smart home assistants are still far from being able to effectively detect and handle anomalous conditions in the smart home environment. Our dataset is publicly available at https://github.com/horizonsinzqs/SmartBench. ∗ The first two authors have equal contribution. † Jingyu Xiao is the corresponding author. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n CCS Concepts • Security and privacy→Human and societal aspects of se- curity and privacy. Keywords Smart Homes, Large Language Models, Anomaly Detection. ACM Reference Format: Qingsong Zou, Zhi Yan, Zhiyao Xu, Kuofeng Gao, Jingyu Xiao, and Yong Jiang. 2026. SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts. In . ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/n.n 1 Introduction Smart home assistants play an increasingly important role in mod- ern residential life. By leveraging user habits, historical interactions, and physical parameters of the home, these assistants enable auto- mated control of smart home devices (i.e. IoT devices) or provide actionable recommendations, helping users better regulate and manage their living environment [22,23,40,41]. Notably, anomaly conditions within a smart home can arise from various factors such as user misoperation or device malfunction. In addition, the rapid growth in the diversity and adoption of IoT devices has also made them attractive targets for attackers [3,8,30,49]. As a result, detect- ing anomaly conditions within the home environment has become a critical task for smart home systems [11, 16, 33, 35, 37, 39, 47]. Due to the strong understanding and generalization capabilities demonstrated by large language models (LLMs) across various do- mains, many recent studies have focused on integrating LLMs into smart home assistants to provide more intelligent and personalized services for smart home users [12,24,34,42,43]. For example, King et al. [25] introduced Sasha, which leverages LLMs to translate users’ loosely expressed instructions into personalized device oper- ations. Yu et al. [44] proposed IoTGPT, an LLM-based smart home agent designed to execute IoT commands in a reliable, efficient, and personalized manner. Gao et al. [12] developed a no-code smart home system powered by LLMs, which automatically generates trigger-action programs for IoT devices based on user requests. arXiv:2603.06636v1 [cs.LG] 24 Feb 2026 Conference’17, July 2017, Washington, DC, USAQingsong Zou et al. (a) Conflicting climate devices(b) Unsafe device state (c) Device left running for a long time(d) Device malfunction Figure 1: Examples of anomaly conditions in a smart home. These studies highlight the strong potential of LLMs in this domain. However, most focus on interpreting user behaviors or commands and overlook a core capability of smart home assistants: perceiving anomalous conditions in the home environment. In prac- tice, many signals may indicate an anomalous state. As shown in Figure 1(a), an air conditioner set to cooling while the heater is on may reflect user error or a power-draining attack; the assistant should warn the user and recommend which device to adjust based on the indoor temperature. Figure 1(b) presents another case. If a smart door or window lock is unlocked while the user is away, the assistant should notify the user of a security risk. Likewise, if the kitchen faucet is left running for an extended period (Figure 1(c)), or a humidifier runs while the humidity sensor continuously drops (Figure 1(d)), the assistant should alert the user to potential misop- eration or device malfunction. Identifying these anomalies is challenging because their causes vary and often require jointly considering multiple details. For in- stance, the anomalies in Figures 1(a) and 1(b) can be detected from device states at a single time point, whereas those in Figures 1(c) and 1(d) require tracking one or more devices over time. Moreover, users typically expect not only alerts but also actionable recommenda- tions, or at least a localization of the anomaly [11,39]. Thus, simply detecting the presence of an anomaly is insufficient. The diversity of IoT devices and application scenarios further complicates detec- tion, explanation, and attribution. Consequently, enabling LLMs to identify smart home anomalies and produce accurate explanations is critical for advancing smart home systems. To bridge this gap, we introduce a new comprehensive evaluation benchmark: SmartBench, which is the first smart home dataset designed for LLMs, containing both normal and anomalous device states as well as normal and anomalous device state transition contexts. Specifically, we construct two types of in-home anomalies: context-independent and context-dependent, covering 15 anomaly categories and totaling 4,400 samples. Each context-independent sample contains the states or readings of 62 IoT devices at a specific point in time. Each context-dependent sample consists of a sequence of device actions (state transitions) with corresponding timestamps, with sequence lengths ranging from 20 to 4732. All anomalous samples are also annotated with the criteria used to determine the anomaly, which can be used to further evaluate the reliability of LLM-based anomaly detection. By introducing SmartBench, we aim to provide an effective evaluation tool for developing the next generation of smart home assistants that are capable of accurately understanding, detecting, and dealing with complex anomalous states in smart home environments, thereby ensuring a safer and more reliable experience for users. As a conclusion, our main contributions are as follows: • To the best of our knowledge, we are the first to investigate the ability of LLMs to detect anomalous states in smart home environments. •We propose SmartBench, a benchmark that includes a diverse set of benign and anomalous in-home scenarios. It covers a wide range of anomaly types, including both context-independent and context-dependent, and a rich variety of device actions. •Our experiments on 13 mainstream LLMs show that almost all models perform poorly on this benchmark. Further analysis re- veals that even when a model is able to recognize the presence of an anomaly, it often fails to correctly analyze the underlying cause. This indicates that LLMs are still a long way from achiev- ing reliable detection and handling of anomalous conditions in home environments. 2 Related Work 2.1 Smart Home Assistant on LLM A smart home assistant is expected to support multiple capabilities simultaneously, including understanding and executing user com- mands, perceiving the indoor environment and detecting anomalies, and modeling user behaviors to provide personalized suggestions. Traditional machine learning and deep learning methods often struggle to support such multi-task requirements. Motivated by the strong performance of LLMs across domains, recent work has explored incorporating LLMs into smart home assistants to handle complex tasks in a unified manner [6,24,25, 31,38,44]. Civitarese et al. [6] propose ADL-LLM for recognizing activities of daily living from sensor data. King et al. [25] develop Sasha to interpret complex and ambiguous user instructions. Yu et al. [44] introduce IoTGPT for efficient execution of personalized IoT commands. Polo et al. [31] present an LLM-based architecture for context-aware interaction in smart environments. These studies demonstrate the potential of LLMs in enhancing the intelligence and automation of smart homes. However, they do not discuss the ability of LLMs to detect anomalous conditions within the smart home environment, leaving a significant gap between current approaches and the expected fully integrated smart home assistant. 2.2 Anomaly Detection in Smart Homes Anomaly detection has long been a central problem in smart home systems. Because smart home devices act as both sensors and ac- tuators, they can directly affect the physical home environment (e.g., temperature, humidity, and accessibility). Anomalies arising from user misoperations [30] or malicious attacks can therefore create serious security risks [3,10]. Accordingly, when a device state changes, it is important to assess its environmental impact and monitor for anomalous conditions. SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral ContextsConference’17, July 2017, Washington, DC, USA Most prior work uses machine learning or deep learning to model user–device interactions for anomaly detection [4,11,16,33,39, 46]. For instance, Rieger et al. [33] propose an autoencoder-based approach to detect previously unseen anomalies. Xiao et al. [39] introduce SmartGuard, an unsupervised framework that detects anomalies from noisy long-context behavior sequences. Chi et al. [4] and Gu et al. [16] infer user–device interactions from network traffic or wireless signals and then detect anomalies based on these interactions. Although effective, such methods typically stop at detection and rarely provide actionable guidance after an anomaly is found, making them more suitable as auxiliary tools than as an smart assistant. With the recent push to integrate LLMs into smart home assis- tants, emerging work has started to explore LLMs for this task [21]. For example, Zeng et al. [45] propose an LLM-based IoT security assistant that incorporates CoT to improve understanding of vul- nerabilities and threats. Chowdhury et al. [5] develop a security- focused Q&A system to help users address common IoT issues, and Dong et al. [9] introduce ChatIoT to analyze and disseminate IoT security and threat intelligence. However, these works either focus on device firmware and net- work traffic, overlooking the physical environment and human factors within the smart home, or function solely as Q&A bots with- out discussing the LLM’s ability to detect and analyze anomalous conditions occurring in real smart home environments. 3 SmartBench 3.1 Task Definition Assume a smart home environment where a set of IoT devices (denoted as퐷=푑 1 ,푑 2 ,푑 3 , . . .) is deployed within a specific loca- tion(denoted as퐿=푙 1 ,푙 2 ,푙 3 , . . .). At any given time, each device is in a corresponding state푆 푑 = 푠 1 ,푠 2 ,푠 3 , . . .(e.g., a smart lock can be in an "unlocked" or "locked" state depending on whether the unlock or lock function was called). Additionally, some specific IoT devices or sensors can read physi- cal environmental parameters퐸=푇 (temperature),퐻 (humidity), 퐴푄 (air quality),퐿 (light),푊 (weather),퐷 (day of week), . . .. These parameters can be accessed by the smart home assistant at any time. Typically, indoor environmental parameters readings in a smart home are collected periodically. For context-independent anomalies, the LLM is required to determine the set of potentially anomalous devices퐷 error based on the device state snapshot at a specific time, denoted as푆 푡 = 푠 푑,푡 ,푠 푑,푡 ,푠 푑,푡 , . . .,푑 ∈ 퐷, along with the current environmental pa- rameters퐸 푡 =푇 푡 ,퐻 푡 ,퐴푄 푡 ,퐵 푡 . In addition, the LLM must provide a clear rationale, denoted as푅, for identifying each anomaly. The overall input-output mapping is defined as: [푆 푡 ,퐸 푡 ] → 퐷 error ,푅,(1) if no anomaly is present, the output should simply be normal. For context-dependent anomalies, the LLM must analyze the sequence of device state transitions, and the temporal evolution of environmental parameters to determine whether an anomaly has occurred. In addition, we include two high-level information 퐼: a brief household profile (e.g., single adult office worker, family on vacation, night shift worker, etc.) and the duration of the this segments for the LLM to make more informed decisions. Let each environmental parameter at a specific time be treated as a special type of device state푠 푒,푡 , where푒 ∈ 퐸. Let푠 푥,푡 denotes the state of device푥at time푡,푙 푥 represents the room where device푥is located, and 푡 denote the timestamp. Given a sequence of state transitions over a time window퐴= (푡 1 ,푥 1 ,푠 푥 1 ,푡 1 ,푙 푥 ),(푡 2 ,푥 2 ,푠 푥 2 ,푡 2 ,푙 푥 ),(푡 3 ,푥 3 ,푠 푥 3 ,푡 3 ,푙 푥 ), . . ., where 푥 ∈ 퐷∪푃, 푠 푥,푡 ∈ 푆 푥 , the LLM is expected to infer the potentially anoma- lous device set퐷 error and provide a rationale푅for the detected anomaly. The corresponding input-output mapping is: [퐴,퐼] → 퐷 error ,푅,(2) if no anomaly is detected, the output should be normal. 3.2 Dataset Collection To systematically evaluate LLM-based smart-home anomaly de- tection, we define two anomaly types: context-independent and context-dependent. Context-independent anomalies are snapshots of device states and environmental parameters at a given time, whereas context-dependent anomalies are timestamped sequences of device state transitions, requiring understanding of temporal semantics and causal constraints. Because collecting data from real smart-home development envi- ronments is costly, we build our dataset via a three-stage pipeline: normal sample construction, compliance checking, and anomaly sample construction. Figure 2 illustrates this process. 3.2.1 Normal sample construction. We first adopt public smart- home datasets collected from real environments as the source of normal behavior. These datasets must be organized as time series of (timestamp,device,state,location) tuples. In particular, we use the dataset from [33] as our primary normal-behavior source. For context-independent samples, we define a discrete state space for each device type (e.g., an air conditioner can be in cooling, dehumidifying, or off mode) and use a fixed-length time window. At the end of each window, we aggregate all device states and environmental parameters into a global snapshot, which forms one context-independent normal sample. For context-dependent samples, we split the time series into continuous segments with varying window sizes. We downsample overly long segments by removing states from devices that con- tribute little contextual semantics. We also attach two high-level descriptors: a brief household profile (e.g., single adult office worker, family on vacation, night shift worker) and the segment duration to support more informed LLM judgments. Each segment plus these descriptors constitutes a context-dependent normal sample. However, open-source datasets that satisfy our requirements are scarce, so we also use a virtual smart-home environment. We main- tain a set of device categories, each with a predefined discrete state space, and use common daily routines or household scenarios as generation units. Each scenario is specified in standardized natural language and then mapped to device state-transition sequences.For example, a “returning home from work” scenario may be described as: a single adult office worker arrives home at around 18:30, un- locks the door, turns on the living room lights, and sequentially activates the air conditioner, humidifier, and television. The entire event takes place within 10 minutes. Conference’17, July 2017, Washington, DC, USAQingsong Zou et al. LLM Generation Context Compression Sub-sampling Time Windowing & Aggregation Global State Snapshot Sequence Segmentation Context- Independent Normal Samples Context- Dependent Normal Samples Context- Independent Anomalous Samples Context- Dependent Anomalous Samples Yes No Compliant? Normal Samples Anomaly Samples Flagged & Manual Correction Stage 1: Normal Sample Construction Stage 3: Anomaly Sample Construction Stage 2: Sample Compliance Validation Public Dataset Public Datasets (Operation, Timestamp) Time Series VirtualSmart Home Daily Scenarios Device Categories, State Spaces, Operations Context-dependent Anomaly Library (Sequence) AnomalyInjection Controlled Edits Context-independent Anomaly Library (Snapshot) ACON HeaterON Minimal Edits GPT5-Mini Auto-Check Temporal Consistency Device Operation Validity Semantic Consistency Final Normal Samples Figure 2: Dataset collection process of SmartBench. We use GPT-5 to generate samples from these scenario descrip- tions. For context-independent samples, we provide the device list and allowed states and ask GPT-5 to produce the resulting end- of-event device-state snapshot. For context-dependent samples, we prompt GPT-5 to generate (timestamp,device,state,location) sequences and the accompanying high-level descriptors. Notably, some context-dependent samples can span long dura- tions and become too lengthy for LLMs. Limited context windows and the quadratic cost of self-attention hinder long-sequence pro- cessing, often causing the “lost-in-the-middle” effect and degrading anomaly detection performance [19, 26]. To address this, we design a context compression strategy. Specif- ically, for the device state transition sequence of context-dependent samples, we merge or remove consecutive, repetitive, and seman- tically uninformative device state transitions, such as repeated readings of unchanged environmental parameters, gradual air con- ditioner temperature adjustments, or frequent channel switching on a television. In these cases, we retain only the initial and final states in the sequence and delete other actions. For any sequence of continuous device state transitions, a change in the state of another device or an environmental parameter during that period may alter the semantic interpretation of the original state transitions within the context. Our strategy follows a minimal semantic impact prin- ciple, meaning that we ensure that the compressed sequences do not alter the original semantic integrity. 3.2.2 Sample compliance validation. LLM-generated samples may include hallucinations, leading to unrealistic or logically inconsis- tent content. For context-independent samples, such errors typically appear as implausible environmental parameter values or references to non-existent devices or states. we address these issues with vali- dation functions that check these elements. For context-dependent samples, further validation is needed to verify the semantics of device state transition sequences. To ensure each synthetic sample is consistent with its scenario description and free of contextual contradictions, we conduct a compliance-checking procedure for every generated context-dependent sample, based on three criteria: (1)Device state transition validity: Each device state transition must match the device type and its defined state space, avoiding non-existent actions or disallowed state transitions. (2)Temporal consistency: Timestamps and their intervals must be reasonable and align with the time description in the scenario. (3)Semantic consistency: The sequence of device state transitions in the sample must align with the corresponding scenario de- scribed in natural language. We leverage the semantic understanding capabilities of LLMs to assist with this validation. Specifically, we use GPT-5-mini to automatically examine each sample for compliance with the above criteria. Any sample that violates one or more criteria is flagged as non-compliant and then will be manually corrected based on suggestions provided by the model. Appendix A.3 presents the prompt we used during this process. After the LLM review, we conducted a manual review of 200 ran- domly selected synthetic context-dependent samples. The results showed that 197 out of 200 samples (98.5%) were compliant. Among them, 39 samples were flagged by GPT-5-mini as needing revision. All of these passed the manual review after being corrected. These results demonstrate the effectiveness of GPT-5-mini for compliance checking and confirm that the final dataset contains high-quality samples that realistically reflect smart home scenarios. 3.2.3 Anomaly sample construction. Collecting anomaly samples in real-world environments is costly and difficult to scale. For in- stance, simulating device malfunctions may require firmware mod- ifications, and long-context sequences typically demand extended manual intervention and observation. To address this, we choose to simulate virtual anomalies to represent anomalous conditions. We first define a category library of context-independent anom- alies (i.e., an anomaly pattern library), where each category cor- responds to a state combination that have physical/semantic con- straints. For each anomaly type, we select normal samples contain- ing the relevant target state variables and apply minimal edits to transform them into anomalies. For example, in the case shown in Figure 1(a), we identify samples that include both air conditioner and heater state variables, and then set both to the "on" state simul- taneously to construct a typical conflicting state anomaly. For context-dependent anomalies, we also begin by defining an anomaly category library and manually author a collection of anomalous sequence fragments for each category. These fragments are then inserted into normal sequences to form anomalous samples. SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral ContextsConference’17, July 2017, Washington, DC, USA It is important to note that the insertion process follows temporal and semantic consistency constraints, rather than random injection. Each anomaly fragment must be coherent with its preceding context and must not introduce explicit contradictions. For example, after a device failure occurs, its normal executable actions should no longer appear, as this would break narrative consistency. To this end, we define four edit operations for controlled anomaly injection: •Insertion: Introduce a small number of key device state transi- tions at specific timestamps in the original sequence to explicitly introduce new evidence of anomalies, without altering the main context of the scenario. •Deletion: Remove transitions according to the target anomaly mechanism, interrupting an otherwise normal causal chain. • Temporal Shift: Keep the order unchanged, but shift key transi- tions to later timestamps to create temporal inconsistencies. • Value Adjustment: Maintain the event structure while modify- ing key observed values under constraints, thereby introducing abnormal deviations in magnitude, trend, or correlations. Without altering the overall scenario background, we apply these four types of edit operations to a small number of key events in a controlled manner, thereby constructing anomalous samples that are paired one-to-one with normal samples. These anomalies are designed to be indistinguishable from normal behavior without proper context, requiring temporal and semantic reasoning for accurate detection. 3.3 Data Statistic and Analysis In total, our dataset contains 4,400 samples. Based on whether temporal context is required for detection, we divide the samples into two categories: context-independent and context-dependent. A context-independent sample includes the states of several devices and corresponding environmental parameters at a specific point in time. A context-dependent sample consists of a sequence of (times- tamp, device, device state, location) pairs. Examples of the data formats for both types of samples are provided in Appendix A.1. DM 30.0% SH 28.8% PV 16.2% CA 10.6% EC 9.6% OM 4.8% (a) Context-Independent DM 12.0% CT 22.7% SA 9.2% SH 4.3% CA 4.0% ND 4.0% C 13.7% OA 29.2% EA 1.0% (b) Context-Dependent Figure 3: Anomaly types and percentages of SmartBench. To ensure balanced evaluation, we maintain a 1:1 ratio of nor- mal to anomalous samples. During the generation of normal sam- ples by LLMs, we prioritized physical environments and time pe- riods that were underrepresented in the open-source dataset, in order to better balance the overall distribution of the dataset. For the context-independent task, we include six types of anomalies: device_malfunction (DM), safety_hazard (SH), physical_violation (PV), compound_anomaly (CA), environmental_conflict (EC), and occupancy_mismatch (OM). For the context-dependent task, we include nine types of anomalies: device_malfunction (DM), causal_temporal (CT), statistical_anomaly (SA), safety_hazard (SH), climate_anomaly (CA), network_disconnect (ND), control_conflict (C), occupancy_anomaly (OA), and energy_anomaly (EA). Figure 3 presents the context-independent and context-dependent anomaly categories and percentage distribution in SmartBench. We measure the size of context-independent samples by count- ing the number of characters in each sample. For context-dependent samples, due to their longer context, we quantify size by the number of (timestamp,device,state,location) tuples, that is, the length of the device state transition sequence in the sample. General statistics about the dataset are summarized in Table 1 (#: Count, CI: context- independent sample, CD: context-dependent sample, Anom: anom- aly, Atype: anomaly type). Table 1: General statistics of SmartBench. StatisticsDescriptionStatisticsDescription Time of Day morning, midday, afternoon, night Room types basement, bathroom, bedroom, entrance, garage, hallway, kitchen, living_room Season spring, summer, fall, winter Environment Parameter temperature, humidity, light, motion, weather, smoke, gas, moisture day_of_week, is_weekend #Occupancy29#Duration15 #Device155#State274 #CI Normal1000#CD Normal1200 #CI Anom1000#CD Anom1200 #CI Atype50#CD Atype60 Min CI Size2953Min CD Length20 Max CI Size4836Max CD Length4732 Avg. CI Size3419.7Avg. CD Length400.5 Overall, SmartBench features a wide range of device types, state types, environment parameters and diverse smart home scenarios. The lengths of context-independent and context-dependent samples also vary widely in distribution. Most importantly, it includes a rich set of anomaly patterns and covers a broad type of normal and anomalous behaviors in smart home environments, making it a comprehensive benchmark for evaluating LLM-based assistants. Appendix A.4 provides detailed lists from SmartBench, including the types of devices and device states, as well as the full library of context-independent and context-dependent anomaly categories. 3.4 Evaluation Metrics To evaluate the ability of LLMs to detect anomalous states in smart homes, we adopt following key metrics: F1 score, Anomaly Location Score, and Attribution Consistency Score. Accuracy (Acc), Recall (R), Preicision (P) and F1 score (F1). We formulate anomaly detection as a binary classification task and evaluate LLMs using standard metrics. Accuracy measures the fraction of correctly classified samples. Precision (P) is the fraction of predicted anomalies that are true anomalies, while Recall (R) is the fraction of true anomalies correctly detected. We report the F1 score as the harmonic mean of P and R: 퐹1= 2푃푅/(푃 + 푅). False positive rate (FPR). FPR is a key metric in anomaly detection [33,39]. Excessive false alarms can severely harm user experience. We define FPR as the proportion of samples predicted as anomalous that are actually normal. A high FPR therefore indicates low reliability, even when overall accuracy seems acceptable. Conference’17, July 2017, Washington, DC, USAQingsong Zou et al. Anomaly location score (AL Score). Identifying whether a sample is anomalous is only a basic requirement for a smart as- sistant. To evaluate a model’s ability to pinpoint the cause of an anomaly, we introduce the anomaly location score. For each anomaly sample, we define a ground truth evidence device set퐷 퐴 , representing the devices whose current states or readings are responsible for the anomaly. The model is expected to output a predicted set퐷 푃 , which should ideally include as many true anomaly devices as possible while minimizing the inclusion of normal devices. Inspired by the Jaccard similarity coefficient [20], which is commonly used in data science to measure the similarity between two sets, we define the anomaly location score as: 퐴퐿 푆푐표푟푒= |퐷 퐴 ∩ 퐷 푃 | |퐷 퐴 ∪ 퐷 푃 | .(3) This metric evaluates the overlap between the predicted and ground truth anomaly devices. Attribution consistency score (AC Score). Being able to locate an anomaly does not necessarily mean the model has correctly analyzed its cause. To assess this deeper level of reasoning, we introduce a new metric: the attribution consistency score. For each anomaly sample, we manually maintain a ground truth explanation푅, which clearly describes the underlying cause of the anomaly. During inference, the model is required to provide its own explanation푅 푃 for any sample it classifies as anomalous. Following the LLM-as-a-Judge paradigm [18,27,48], we use GPT-5- mini to evaluate the semantic similarity between푅and푅 푃 . Specif- ically, GPT-5-mini is prompted to rate the similarity on a 1-to-5 scale, where a higher score indicates greater alignment between the model’s reasoning and the ground truth. Details of the evaluation prompt can be found in the Appendix A.2. By computing these metrics, we can progressively evaluate a smart home assistant’s ability to detect anomalous conditions within the home. A model with high F1 and low FPR can reliably signal that an anomaly exists, but users may still need to manually identify the issue. A high anomaly location score further helps by pinpointing likely involved devices, substantially narrowing the search space, though users must still diagnose the cause and decide on corrective actions. Only models with high attribution consistency provide all of these capabilities, accurately explaining the anomaly’s cause and thus enabling effective mitigation suggestions. 4 Experiments 4.1 Setting Models. We select a range of mainstream open-source and close- source LLMs for evaluation. Specifically, from the open-source models, we include meta-LLaMA-3.1-8B-Instruct (llama-8b), meta- LLaMA-3.1-70B-Instruct (llama-70b), meta-LLaMA-3.1-405B-Instruct (llama-405b) [15], DeepSeek-R1 (deepseek-r1) [17], and DeepSeek- V3 (deepseek-v3) [7]. From the close-source models, we select GPT-5 (gpt-5) [36], GPT-5-mini (gpt-5-mini) [28], Gemini-2.5-Pro (gemini- 2.5) [13], Gemini-3-Pro-preview (gemini-3) [14], Claude-4-Sonnet- 20250514 (claude-4) [2], Claude-4.5-Sonnet-20250929 (claude-4.5) [1], qwen-8b (qwen-8b), and qwen-32b (qwen-32b) [32]. We set temperature to 0 for all models. For the context- independent task, we set max_output_token to 12k for thinking models and 8k for non-thinking models. For the context-dependent task, we use 12k and 6k, respectively. 4.2 Research Questions In this section, we present a comprehensive analysis aimed at ad- dressing the following five research questions. RQ1: How well does LLM detect anomalous states in smart homes? RQ2: Can LLMs ana- lyze the underlying causes of anomalies? RQ3: How does model size affect anomaly detection performance? RQ4: What is the impact of context compression on model performance? RQ5: Can few-shot learning help improve anomaly detection capabilities? 4.3 Main Results Table 2 presents the evaluation results of different LLMs on various anomaly types in SmartBench. Several key observations can be drawn from these results. Overall, LLMs struggle to effectively detect anomalies in smart home environments. In the context-independent task, the best-performing model is gemini-3, achieving an F1 score of 79.3%, while the worst is qwen-8b, with only 46.2%. Although most models reach an F1 score above 60%, this still reflects limited detection ca- pability in practical terms. In the context-dependent task, the best result comes from gemini-2.5, with an F1 score of 67.6%. However, it also exhibits an extremely high FPR of 78.2%, indicating that it tends to classify normal samples as anomalous. Such a high FPR would severely impact user experience, making real-world deployment difficult. All other models achieve F1 scores no higher than 67.0% in the context-dependent task. This level of performance is clearly inadequate to meet user expectations for next-generation smart home assistants. The radar visualization (Figure 4) offers a clearer DM SH PV CA EC OM 20 40 60 80 100 Context-Independent DM CT SA SH CAND C OA EA 20 40 60 80 100 Context-Dependent gpt-5 gpt-5-mini gemini-3 gemini-2.5 claude-4.5 claude-4 qwen-32b qwen-8b ds-r1 ds-v3 Figure 4: Performance of LLMs on different anomaly tasks. illustration of LLM performance across different anomaly types, as measured by the F1 score. In context-independent tasks, most models exhibit relatively balanced detection performance across anomaly subtypes, though the majority of F1 scores remain below 80%. In context-dependent tasks, some models display slight vari- ability across anomaly types. For instance, claude-4 performs poorly on occupancy_anomaly (OA) and safety_hazard (SH) compared to its performance on other types. Similarly, deepseek-v3 achieves its best performance on device_ malfunction (DM) but performs worst on OA. Overall, most models maintain relatively stable performance SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral ContextsConference’17, July 2017, Washington, DC, USA Table 2: The main results of different LLMs on SmartBench. Bold indicates the best performance. Context-IndenpendentContext-Dependent PrecisionRecallF1 scoreFPRAL ScorePrecsionRecallF1 scoreFPRAL Score gemini-374.2%85.2% 79.3%29.7% 0.49157.4%79.8%66.8%59.2%0.347 gemini-2.564.5% 85.6%73.5%47.2%0.39753.8% 91.0% 67.6%78.2% 0.365 claude-4.563.9%74.0%68.6%41.8%0.31959.6%59.0%59.3%40.0%0.257 claude-473.8%50.7%60.1%18.0%0.23267.3%44.5%53.6% 21.7%0.247 deepseek-r175.8%68.5%72.0%21.9%0.36552.2%83.7%64.3%76.5%0.261 deepseek-v383.4%37.1%51.3%7.4%0.17953.9%51.3%52.6%43.8%0.170 gpt-592.6%68.9%79.0% 5.5%0.41668.8%48.8%57.1%22.2%0.251 gpt-5-mini68.5%76.9%72.5%35.3%0.36360.9%68.8%64.6%44.2%0.252 qwen-3-32b53.1%83.1%64.8%73.3%0.18951.0%80.0%62.3%77.0%0.185 qwen-3-8b52.4%41.3%46.2%37.5%0.05253.3%61.7%57.2%54.0%0.105 across anomaly categories, with GPT and Gemini series generally outperforming other families. Models perform better on context-independent anomaly detection than on context-dependent tasks. Across the ten LLMs we test, the average F1 score on context-independent samples is 66.7%, with an average AL score of 0.300. In contrast, on context- dependent tasks, the averages drop to 60.5% and 0.221, respectively. This performance gap may be attributed to the shorter input length of context-independent samples, which reduces the cognitive load on the LLM. With less irrelevant information to process, the models are less affected by the “lost-in-the-middle” effect and can more easily focus on identifying problematic device states. All LLMs fail to accurately localize the anomalies they detected. In the context-dependent task, even the best-performing model, gemini-2.5, achieves an AL Score of only 0.365. The worst performance come from the qwen-3 series, with AL Scores of just 0.185 and 0.105, respectively. These results indicate that, for most of the anomalies detected, the models are unable to correctly identify the responsible devices, often misclassifying normal device state transitions as anomalous. On average, the ten LLMs achieve only 0.221 AL Score in the context-dependent task. Even in context- independent setting, the best AL Score achieved is only 0.491, un- derscoring the limited anomaly localization ability of current LLMs in smart home environments. Findings 1: LLMs fail to achieve satisfactory performance on anomaly detection tasks in smart home environments. They are unable to effectively identify the presence of anomalies or accurately locate the source of the anomalies. 4.4 Anomaly Attribution Capability of LLMs LLMs are expected to provide reasonable explanations for why a sample is normal or anomalous. This is essential for enabling smart home assistants to deliver accurate descriptions of anomalies and offer appropriate recommendations to users. To evaluate the anomaly attribution capability of LLMs, we randomly sample 100 context-independent and 100 context-dependent anomaly samples from our dataset. For each, we compare the LLM’s explanation with the human-annotated ground truth. Specifically, we use gpt-5- mini to score the quality of the LLMs’ reasoning, and compute the average attribution consistency score as described in Section 3.4. The results are presented in Figure 5. gemini-3 gemini-2.5 claude-4.5 claude-4 ds-r1 ds-v3 gpt-5 gpt-5-mini qwen-32b qwen-8b 1 2 3 4 5 AC Score Context_Independent AC Score Context_Dependent AC Score 0.0 0.2 0.4 0.6 0.8 1.0 Valid Rate CI Valid Rate CD Valid Rate Figure 5: Anomaly attribution capability of LLMs. We consider an attribution to be valid if its consistency score is greater than or equal to 4. However, even under this threshold, most LLMs are only able to provide reasonable explanations for less than 70% of the anomalies. Only Gemini-3 and Gemini-2.5 reach acceptable attribution rates of 74% and 72% respectively for context- independent anomalies. Performance on context-dependent anom- alies is noticeably worse. Even the best-performing model achieves only a 3.19 average attribution consistency score, falling short of the threshold for acceptable explanation quality. Findings 2: Even the state-of-the-art LLMs still struggle to generate coherent and accurate explanations for anomalies in smart home environments, and therefore remain insufficient for supporting users in effectively resolving anomalous conditions. 4.5 The Effects of Model Sizes Table 3: Performance across models of different sizes. Context-IndependentContext-Dependent PRF1PRF1 qwen-8b34.3%24.0%28.2%47.6%41.6%44.4% qwen-32b 48.8%80.0%60.6%45.2%79.1%57.5% llama-8b 30.8%8.0%12.7%50.0%16.7%25.0% llama-70b42.9%18.0%25.4%0.0%0.0%0.0% llama-405b 57.1%32.0%41.0%75.0%37.5%50.0% Conference’17, July 2017, Washington, DC, USAQingsong Zou et al. As model size increases, we observe performance improvements on both context-independent and context-dependent tasks. This trend is evident in both the qwen-3 and llama-3.1 series. Gener- ally, larger models are expected to yield better performance. An exception is llama-70b, which achieves an F1 score of 0 on the context-dependent task, as it classifies all samples as normal. How- ever, even for large-scale models such as llama-405b, the F1 scores remain low-only 41.0% and 50% on the two tasks, respectively. This indicates that scaling alone is insufficient to overcome the chal- lenges posed by anomaly detection in smart home environments. Simply increasing model size is unlikely to resolve the limitations in reasoning and attribution required in this field. Findings 3: As model scale increases, LLMs generally detect anomalies better, but scaling parameters alone does not yield the expected performance improvement. 4.6 The Effects of Context Compression As mentioned in Section 3.2, our context-dependent samples un- dergo compression prior to evaluation. In this subsection, we aim to compare model performance on pre- and post-compression sam- ples. Specifically, we randomly select 50 context-dependent samples with an original sequence length of at least 10k tokens (estimated using cl100k_base [29]). We then inject random anomaly fragments into both the pre-compression and post-compression versions of 25 of these samples (injection strategy is consistent across the pre- compression and post-compression versions of each sample). Table 4: The performance of LLMs on compressed samples. AccuracyF1 scoreToken RawCCRawCCRawCC claude-4.560.0%60.0%66.7%66.7%5.58M2.18M claude-4 62.0%76.0%64.2%77.8%5.58M2.17M deepseek-r164.0%68.0%59.1%63.6%5.40M2.16M deepseek-v346.0%44.0%62.9%53.3%5.33M2.12M gemini-346.0%50.0%62.0%64.8%10.90M3.12M gemini-2.554.0%54.0%68.5%68.6%10.75M2.99M gpt-5 64.0%66.0%69.0%72.1%5.15M2.10M gpt-5-mini72.0%72.0%74.1%74.1%5.11M2.05M Each sample is then evaluated by the LLM both before and after compression. Table 4 presents the comparison of anomaly detec- tion performance and token usage across models (C: context- compressed inputs). All LLMs show a decrease of over 50% in token usage after compression. Moreover, nearly all models maintain their anomaly detection performance, with only deepseek-r1 exhibiting a slight performance drop. However, while context compression significantly reduces token overhead, we find that it offers limited improvement in anomaly detection performance. Findings 4: Context compression offers limited improvement. We still need to explore additional strategies to further enhance the anomaly detection capabilities of LLMs in smart home. 4.7 The Effects of ICL We randomly select 50 context-independent and 50 context- dependent samples from SmartBench to evaluate the impact of 0.00 0.25 0.50 0.75 1.00 F1 Score w/o ICLw/ ICL (a) Context-Independent gemini-3 gemini-2.5 claude-4.5 claude-4 ds-r1 ds-v3 gpt-5 gpt-5-mini qwen-32b qwen-8b 0.00 0.25 0.50 0.75 1.00 F1 Score (b) Context-Dependent Figure 6: The effects of ICL on LLMs’ performance. in-context learning (ICL) on model F1 scores. As shown in Figure 1 6, most models demonstrate improved anomaly detection perfor- mance after introducing ICL. On the context-dependent task, the average F1 score increases by 7.5%, with gpt-5 showing the largest improvement at 21.4%. Only qwen-8B experiences a performance drop, possibly due to its relatively small parameter size. For the context-independent task, the average F1 score increases by 9.8%. However, we also note that even with ICL, the average F1 score across all models reached only 80.8% on the context-independent task and 71.2% on the context-dependent task. This indicates that a significant performance gap remains between current capabilities and the demands of real-world deployment, highlighting the need for further optimization and improvement. Findings 5: Even with ICL, a significant performance gap remains between current capabilities and the demands of real- world deployment. 5 Conclusion To advance anomaly detection in LLM-based smart-home assistants, we propose SmartBench, the first smart-home dataset tailored for LLMs. It includes both normal and anomalous device-state snap- shots and state-transition contexts, covering context-independent and context-dependent anomalies with 4,400 samples. For each anomalous sample, we also provide the rationale for its label, en- abling evaluation of classification accuracy and anomaly localiza- tion. We evaluate 13 state-of-the-art LLMs on SmartBench and find that most perform poorly. Moreover, even when a model detects an anomaly, it often fails to correctly explain its cause. These results suggest that current LLM-based smart-home assistants remain un- reliable for residential anomaly detection. Overall, SmartBench and our evaluation establish a foundation for improving the real-world robustness of LLM-based smart-home assistants. SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral ContextsConference’17, July 2017, Washington, DC, USA References [1]Anthropic. 2025. Claude Sonnet 4.5 System Card. https://w.anthropic.com/ claude-sonnet-4-5-system-card System card published Oct 2025. [2]Anthropic. 2025. System Card: Claude Opus 4 & Claude Sonnet 4.https: //w.anthropic.com/claude-4-system-card [3]Haotian Chi, Chenglong Fu, Qiang Zeng, and Xiaojiang Du. 2022. Delay Wreaks Havoc on Your Smart Home: Delay-based Automation Interference Attacks. In 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022. IEEE, 285–302. https://doi.org/10.1109/SP46214.2022.9833620 [4]Haotian Chi, Qi Ma, Yuwei Wang, Jing Yang, and Haijun Geng. 2025. IoTBy- stander: A Non-Intrusive Dual-Channel-Based Smart Home Security Monitoring Framework. Applied Sciences 15, 9 (2025), 4795. [5]Hafijul Hoque Chowdhury, Riad Ahmed Anonto, Sourov Jajodia, Suryadipta Majumdar, and Md Shohrab Hossain. 2025. Identifying and Addressing User- level Security Concerns in Smart Homes Using" Smaller" LLMs. arXiv preprint arXiv:2509.19485 (2025). [6] Gabriele Civitarese, Michele Fiori, Priyankar Choudhary, and Claudio Bettini. 2025. Large language models are zero-shot recognizers for activities of daily living. ACM Transactions on Intelligent Systems and Technology 16, 4 (2025), 1–32. [7]DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] doi:10.48550/arXiv.2412.19437 [8] Shuaike Dong, Zhou Li, Di Tang, Jiongyi Chen, Menghan Sun, and Kehuan Zhang. 2020. Your Smart Home Can’t Keep a Secret: Towards Automated Fingerprinting of IoT Traffic. In Proceedings of ASIA CCS. [9] Ye Dong, Yan Lin Aung, Sudipta Chattopadhyay, and Jianying Zhou. 2025. Cha- tiot: Large language model-based security assistant for internet of things with retrieval-augmented generation. arXiv preprint arXiv:2502.09896 (2025). [10]Chenglong Fu, Qiang Zeng, Haotian Chi, Xiaojiang Du, and Siva Likitha Valluru. 2022. IoT Phantom-Delay Attacks: Demystifying and Exploiting IoT Timeout Behaviors. In 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2022, Baltimore, MD, USA, June 27-30, 2022. IEEE, 428–440. https://doi.org/10.1109/DSN53405.2022.00050 [11]Chenglong Fu, Qiang Zeng, and Xiaojiang Du. 2021. HAWatcher: Semantics- Aware Anomaly Detection for Appified Smart Homes. In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, Michael D. Bailey and Rachel Greenstadt (Eds.). USENIX Association, 4223–4240. https://w.usenix. org/conference/usenixsecurity21/presentation/fu-chenglong [12]Yi Gao, Kaijie Xiao, Fu Li, Weifeng Xu, Jiaming Huang, and Wei Dong. 2024. ChatIoT: Zero-code generation of trigger-action based IoT programs. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, 3 (2024), 1–29. [13]Gemini Team. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reason- ing, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv:2507.06261 [cs.CL] doi:10.48550/arXiv.2507.06261 [14] Google DeepMind. 2025. Gemini 3 Pro: Model Card. https://storage.googleapis. com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf Published/- Model Release: November 2025. [15] Andrea Grattafiori et al.2024.The Llama 3 Herd of Models. arXiv:2407.21783 [cs.CL] doi:10.48550/arXiv.2407.21783 [16] Tianbo Gu, Zheng Fang, Allaukik Abhishek, Hao Fu, Pengfei Hu, and Prasant Mohapatra. 2020. IoTGaze: IoT Security Enforcement via Wireless Context Analysis. In 39th IEEE Conference on Computer Communications, INFOCOM 2020, Toronto, ON, Canada, July 6-9, 2020. IEEE, 884–893. doi:10.1109/INFOCOM41043. 2020.9155459 [17]D. Guo et al.2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501. 12948 [18]Fan Huang, Haewoon Kwak, Kunwoo Park, and Jisun An. 2024. ChatGPT rates natural language explanation quality like humans: But on which scales? arXiv preprint arXiv:2403.17368 (2024). [19]Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang, Hao Chen, Shupeng Li, and Penghao Zhao. 2024. Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey. arXiv:2311.12351 [cs.CL] https://arxiv.org/abs/2311. 12351 [20]Paul Jaccard. 1908. Nouvelles Recherches Sur la Distribution Florale. Bulletin de la Societe Vaudoise des Sciences Naturelles 44 (01 1908), 223–70. doi:10.5169/seals- 268384 [21]Saeid Jamshidi, Negar Shahabi, Amin Nikanjam, Kawser Wazed Nafi, Foutse Khomh, and Carol Fung. 2025. The role of Large Language Models in IoT security: A systematic review of advances, challenges, and opportunities. Internet of Things (2025), 101735. [22]Yunhan Jack Jia, Qi Alfred Chen, Shiqi Wang, Amir Rahmati, Earlence Fernandes, Zhuoqing Morley Mao, and Atul Prakash. 2017. ContexloT: Towards Providing Contextual Integrity to Appified IoT Platforms. In 24th Annual Network and Distributed System Security Symposium, NDSS 2017, San Diego, California, USA, February 26 - March 1, 2017. The Internet Society. [23]David Schlangen Jonathan Jordan, Sherzod Hakimov. 2025. Plant in Cupboard, Orange on Rably, Inat Aphone. Benchmarking Incremental Learning of Situation and Language Model using a Text-Simulated Situated Environment. In The 28th International Conference of Text, Speech and Dialogue (TSD2025). [24]Evan King, Haoxiang Yu, Sangsu Lee, and Christine Julien. 2023. " Get ready for a party": Exploring smarter smart spaces with help from large language models. arXiv preprint arXiv:2303.14143 (2023). [25]Evan King, Haoxiang Yu, Sangsu Lee, and Christine Julien. 2024. Sasha: creative goal-oriented reasoning in smart homes with large language models. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, 1 (2024), 1–38. [26]Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. [27]Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. ChatGPT as a Factual Inconsistency Evaluator for Text Summarization. arXiv:2303.15621 [cs.CL] https: //arxiv.org/abs/2303.15621 [28]OpenAI. GPT-5 mini. 2025-11-26. Technical report, OpenAI, August 2025c. https: //openai.com/index/gpt-5-mini/ [29]OpenAI. 2022. tiktoken: Fast BPE tokeniser for use with OpenAI’s models. https: //github.com/openai/tiktoken Includes the cl100k_base encoding. [30]Muslum Ozgur Ozmen, Xuansong Li, Andrew Chu, Z Berkay Celik, Bardh Hoxha, and Xiangyu Zhang. 2022. Discovering IoT physical channel vulnerabilities. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 2415–2428. [31]Aurora Polo-Rodríguez, Laura Fiorini, Erika Rovini, Filippo Cavallo, and Javier Medina-Quero. 2025. Enhancing smart environments with context-aware chat- bots using large language models. arXiv preprint arXiv:2502.14469 (2025). [32] Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] doi:10. 48550/arXiv.2505.09388 [33] Phillip Rieger, Marco Chilese, Reham Mohamed, Markus Miettinen, Hossein Fereidooni, and Ahmad-Reza Sadeghi. 2023. ARGUS: Context-Based Detection of Stealthy IoT Infiltration Attacks. In Proceedings of the 32nd USENIX Conference on Security Symposium (Anaheim, CA, USA) (SEC ’23). USENIX Association, USA, Article 241, 18 pages. [34] Gyuhyeon Seo, Jungwoo Yang, Junseong Pyo, Nalim Kim, Jonggeun Lee, and Yohan Jo. 2025. SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents. arXiv:2509.24282 [cs.CL] https://arxiv.org/abs/ 2509.24282 [35] Amit Kumar Sikder, Leonardo Babun, Hidayet Aksu, and A. Selcuk Uluagac. 2019. Aegis: A Context-Aware Security Framework for Smart Home Systems. In Proceedings of the 35th Annual Computer Security Applications Conference (San Juan, Puerto Rico, USA) (ACSAC ’19). Association for Computing Machinery, New York, NY, USA, 28–41. doi:10.1145/3359789.3359840 [36] Aaditya Singh et al.2026. OpenAI GPT-5 System Card. arXiv:2601.03267 [cs.CL] https://arxiv.org/abs/2601.03267 [37]Jincheng Wang, Zhuohua Li, Mingshen Sun, Bin Yuan, and John C. S. Lui. 2023. IoT Anomaly Detection Via Device Interaction Graph. In 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Network, DSN 2023, Porto, Portugal, June 27-30, 2023. IEEE, 494–507. https://doi.org/10.1109/DSN58367. 2023.00053 [38]Yuxiang Wang, Siwen Wang, Haowei Han, Ao Wang, Boya Liu, Yong Zhao, Chengbo Wu, Bin Zhu, Bin Qin, Xiaokai Zhou, et al.2025. DevPiolt: Operation Recommendation for IoT Devices at Xiaomi Home. arXiv preprint arXiv:2511.14227 (2025). [39]Jingyu Xiao, Zhiyao Xu, Qingsong Zou, Qing Li, Dan Zhao, Dong Fang, Ruoyu Li, Wenxin Tang, Kang Li, Xudong Zuo, Penghui Hu, Yong Jiang, Zixuan Weng, and Michael R. Lyu. 2024. Make Your Home Safe: Time-aware Unsupervised User Behavior Anomaly Detection in Smart Homes via Loss-guided Mask. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Barcelona, Spain) (KDD ’24). Association for Computing Machinery, New York, NY, USA, 3551–3562. doi:10.1145/3637528.3671708 [40]Jingyu Xiao, Qingsong Zou, Qing Li, Dan Zhao, Kang Li, Wenxin Tang, Runjie Zhou, and Yong Jiang. 2023. User device interaction prediction via relational gated graph attention network and intent-aware encoder. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. 1634–1642. [41] Jingyu Xiao, Qingsong Zou, Qing Li, Dan Zhao, Kang Li, Zixuan Weng, Ruoyu Li, and Yong Jiang. 2023. I know your intent: Graph-enhanced intent-aware user device interaction prediction via contrastive learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 3 (2023), 1–28. [42] Zhiyao Xu, Dan Zhao, Qingsong Zou, Qing Li, Yong Jiang, Yuhang Wang, and Jingyu Xiao. 2025. Semantic-aware Graph-guided Behavior Sequences Generation with Large Language Models for Smart Homes. arXiv preprint arXiv:2508.03484 (2025). [43]Zhiyao Xu, Dan Zhao, Qingsong Zou, Jingyu Xiao, Yong Jiang, Zhenhui Yuan, and Qing Li. 2025. Synthetic User Behavior Sequence Generation With Large Language Models for Smart Homes. IEEE Internet of Things Magazine 8, 6 (2025), Conference’17, July 2017, Washington, DC, USAQingsong Zou et al. 17–23. doi:10.1109/MIOT.2025.3575803 [44]Chaerin Yu, Chihun Choi, Sunjae Lee, Hyosu Kim, Steven Y Ko, Young-Bae Ko, and Sangeun Oh. 2026. Leveraging LLMs for Efficient and Personalized Smart Home Automation. arXiv preprint arXiv:2601.04680 (2026). [45]Mingfei Zeng, Ming Xie, Xixi Zheng, Chunhai Li, Chuan Zhang, and Liehuang Zhu. 2025. Large Language Model-driven Security Assistant for Internet of Things via Chain-of-Thought. IEEE Internet of Things Journal (2025). [46]Wei Zhang, Yan Meng, Yugeng Liu, Xiaokuan Zhang, Yinqian Zhang, and Haojin Zhu. 2018. Homonit: Monitoring smart home apps from encrypted traffic. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. 1074–1088. [47]Dan Zhao, Qing Li, Qingsong Zou, Jingyu Xiao, Kaidong Wu, Ruoyu Li, Jianhui Lv, Yong Jiang, and Keqin Li. 2025. Security and privacy in smart homes: Challenges and latest developments. In Advances in the Internet of Things. CRC Press, 36–55. [48]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36 (2023), 46595–46623. [49]Qingsong Zou, Qing Li, Ruoyu Li, Yucheng Huang, Gareth Tyson, Jingyu Xiao, and Yong Jiang. 2023. Iotbeholder: A privacy snooping attack on user habitual behaviors from smart home wi-fi traffic. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 1 (2023), 1–26. SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral ContextsConference’17, July 2017, Washington, DC, USA A Appendix A.1 Examples of Context-Independent and Context-Dependent Samples. device_states: Living Room AC: cooling, Front Door: locked, Living Room Heater: heating, Living Roon Curtain: open, Indoor Humidity: 46, Indoor Temperature: 21.7, Front Door Montion: no motion, ... ... ; external_environment_context: Weather: sunny, Time: 15:32, Outdoor Temperature: 27.8, day_of_week: Tuesday, ... ... ; label: anomaly; evidence_devices: [AC, Heater]; reasoning: Indoor and outdoor temperatures are normal. The weather is sunny. The AC is detected to be in cooling mode, and the heater is in heating mode. Both devices are located in the living room. The two devices are conflicted with each other. It may be necessary to turn one of them off. Figure 7: An example of the context-independent sample. Figure 7 presents an example of a context-independent anomaly sample. This type of sample includes the states of several smart home devices along with indoor environmental parameters. Addi- tionally, it contains several non-environmental contextual parame- ters that may be useful for LLMs during reasoning. In this case, the air conditioner is set to cooling mode while the heater is operating in heating mode at the same time. As these two functions conflict, we manually label this sample as anomaly. Since we cannot determine the user’s personalized comfort temperature, the potentially anomalous device could be either the air conditioner or the heater. The rationale for labeling this sample as anomalous is also included in the sample for reference. duration_days:1, occupancy_type: single_adult_work_from_home, events: timestamp: 2025-12-01T07:15:06, device: kitchen_faucet, status: on, room: kitchen , timestamp: 2025-12-01T07:17:30, device: bathroom_light, status: off, room: bathroom , ... ... timestamp: 2025-12-01T08:27:30, device: livingroom_hvac, status: off, room: livingroom , timestamp: 2025-12-01T08:41:28, device: kitchen_faucet, status: on, room: kitchen , timestamp: 2025-12-01T08:49:56, device: entrance_door_sensor, status: motion, room: entrance , ... ... timestamp: 2025-12-01T10:17:30, device: livingroom_temp_sensor, status: 21, room: livingroom, timestamp: 2025-12-01T10:11:28, device: kitchen_faucet, status: on, room: kitchen , ... ... ; label: anomaly; evidence_devices: [kitchen_faucet]; reasoning: At 2025-12-01T07:15:06, the kitchen faucet was detected as being turned on, and shortly afterward, the user turned off the bathroom lights. At 2025-12-01T08:27:30, the user turned off the living room heating. At 2025-12-01T08:49:56, the entrance sensor detected movement, possibly indicating that the user had left home. However, at 2025-12-01T08:41:28, the kitchen faucet was still shown as being on. At 2025-12- 01T10:11:28, the kitchen faucet was still shown as being on. The kitchen faucet was left on for an extended period, possibly due to the user forgetting to turn it off. Figure 8: An example of the context-dependent sample. Figure 8 presents an example of a context-dependent anomaly sample. This type of sample mainly consists of a sequence of en- tries in the form of (timestamp, device, device state, room) and high-level descriptions. From this sequence, it can be observed that while other devices operate normally, the kitchen’s leak sensor indicates that the faucet has been continuously turned on. This likely indicates that the user forgot to turn it off or that the faucet is even malfunctioning. As a result, the evidence device is identified as the faucet, and the corresponding rationale for the anomaly is provided at the end of the sample. Conference’17, July 2017, Washington, DC, USAQingsong Zou et al. A.2 The prompt used to evaluate the explanations for anomalies. We follow the LLM-as-a-Judge paradigm and use the following prompt to instruct GPT-5-mini to assess the semantic similarity between the ground truth explanation푅and the model-generated explanation푅 푃 . To clarify the evaluation criteria and improve scor- ing consistency, we adopt a 5-point rating scale, with detailed de- scriptions for each level as shown in Figure 9. A.3 The Prompt Used for Sample Compliance Validation Figure 10 illustrates the prompt we used to guide gpt-5-mini in performing compliance checks on generated samples. In brief, we provide the sample to be evaluated and instruct the LLM not to modify it directly, but rather to determine whether modifications are necessary and offer corresponding suggestions. A.4 Taxonomy of SmartBench A.4.1 Devices and States. Table 5 presents all device types in Smart- Bench along with their possible states or readings. Following the convention in the public dataset we used, where certain device states are treated as distinct devices, SmartBench also includes some devices named in this format (see [33] for details). In total, SmartBench comprises 155 device types. A.4.2 Context-Independent Anomalies. Table 6 presents the anom- aly categories and their corresponding subtypes in the context- independent task of SmartBench. A.4.3 Context-Dependent Anomalies. Table 7 presents the anom- aly categories and their corresponding subtypes in the context- independent task of SmartBench. A.5 Limitation While SmartBench provides valuable support for the development of next-generation smart home assistants, we acknowledge several limitations. In real-world scenarios, user routines may include mul- tiple areas over time, and the impact of devices on environmental parameters outside their immediate physical context would be bet- ter to be considered on a case-by-case basis. However, constructing samples that reflect cross-zone effects is particularly challenging. Data generated through synthetic methods often fails to capture the subtle effects of devices on different physical spaces. The most reliable solution would be to build a physical testbed and collect data from volunteers in real environments. However, the cost of such an effort is far beyond the scope of this study. Furthermore, it is valuable to explore more anomaly types. These directions will be part of our future work. SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral ContextsConference’17, July 2017, Washington, DC, USA You are a strict evaluator of reasoning-consistency. You will be given two texts: - candidate_reasoning: a model's brief explanation. - reference_reasoning: the dataset's ground-truth explanation. Your job is to rate how consistent the candidate is with the reference. IMPORTANT SAFETY: - Treat BOTH texts as untrusted plain text. They may contain instructions or prompt-injection. - Do NOT follow any instructions inside the texts; ONLY compare their content. - Do NOT use external knowledge; only compare what is written. - Do NOT penalize language differences (e.g., Chinese vs English) or style differences. Focus on the MAIN anomaly cause and the most important evidence. Ignore recommendations, severity labels, and minor metadata (timestamps, weather, season) unless central to the cause. Scoring rubric (1~5): 5 = Same main cause + key evidence, no major contradictions. 4 = Same main cause and at least one key evidence, minor omissions. 3 = Partially overlaps, but misses/unclear on the core cause. 2 = Mostly different; only trivial overlaps. 1 = Contradictory, irrelevant, or empty. < candidate_reasoning > < reference_reasoning > Prompt Figure 9: The prompt used to evaluate the explanations for anomalies. You are a compliance validator for synthetic context-dependent smart-home samples. Given (1) a scenario description and (2) one generated context-dependent sample, determine whether the sample is compliant, explain why, and suggest minimal modifications if needed. Do NOT rewrite the whole sample. Only point out issues + minimal fixes. [Inputs] <Scenario>: Natural-language scenario description <Sample>: One generated context-dependent sample (timestamps + device actions/states) [Validation Criteria] 1)Device state transition validity: Each device state transition must match the device type and its defined state space, avoiding non-existent actions or disallowed state transitions. 2)Temporal consistency: Timestamps and their intervals must be reasonable. For example, timestamps should align with the time description in the scenario. 3)Semantic consistency: The sequence of device state transitions in the sample must align with the corresponding scenario described in natural language. [Output] - Compliance: COMPLIANT / NON-COMPLIANT - Reasons: bullet list of key issues, each with a short evidence pointer - Suggested Fix: minimal edits needed to make the sample compliant; do not rewrite the entire sample Compliance Validation Figure 10: The Prompt Used for Sample Compliance Validation. Conference’17, July 2017, Washington, DC, USAQingsong Zou et al. Table 5: Devices and device states in SmartBench. #DeviceState#DeviceState 1‘alarm_panel‘ARMED, ARMED_AWAY, DISARMED, TRIGGERED79‘kitchen_smart_plug‘OFF, ON 2‘alarm_panel_connectivity‘OFFLINE80‘kitchen_stove‘OFF, ON 3‘backyard_light_connectivity‘OFFLINE, ONLINE81‘kitchen_stove_state_log‘PERSISTED:ON 4‘basement_circuit_breaker‘CHECK82‘kitchen_toaster‘OFF, ON 5‘basement_hvac‘COOLING, HEATING, OFF83‘kitchen_under_cabinet_light‘OFF, ON 6‘basement_light‘OFF, ON84‘living_room_blinds‘OPEN 7‘basement_water_heater‘OFF, ON85‘living_room_ceiling_light‘OFF, ON, UNKNOWN 8‘bathroom_exhaust_fan‘OFF, ON86‘living_room_co_detector‘ALERT 9‘bathroom_exhaust_fan_state_log‘PERSISTED:ON87‘living_room_floor_lamp‘OFF, ON 10‘bathroom_faucet‘OFF, ON88‘living_room_glass_break_sensor‘ALERT 11‘bathroom_heater‘OFF, ON89‘living_room_hvac‘COOLING, HEATING, IDLE, OFF, ON 12‘bathroom_heater_state_log‘PERSISTED:ON90‘living_room_light‘OFF, ON, ON:100%:NEUTRAL, TRIGGER 13‘bathroom_hvac‘COOLING, HEATING, OFF91‘living_room_light_connectivity‘OFFLINE, ONLINE 14‘bathroom_light‘OFF, ON, TIMEOUT92‘living_room_light_state_log‘PERSISTED:ON 15‘bathroom_light_state_log‘PERSISTED:ON93‘living_room_motion_light_automation‘FAILURE_START:DISABLED 16‘bathroom_motion_sensor‘ MOTION, MOTION_DETECTED, NO_MOTION, TIMEOUT 94‘living_room_motion_sensor‘ ALERT, MOTION, MOTION_DETECTED, NO_MOTION, UNAVAILABLE 17‘bathroom_motion_sensor_state_log‘PERSISTED:NO_MOTION95‘living_room_motion_sensor_state_log‘PERSISTED:NO_MOTION 18‘bathroom_washer‘OFF, ON96‘living_room_occupancy_sensor‘OCCUPIED, UNOCCUPIED 19‘bedroom_computer‘OFF, ON97‘living_room_reading_lamp‘OFF 20‘bedroom_hvac‘COOLING, HEATING, OFF98‘living_room_smart_plug‘OFF, ON 21‘bedroom_light‘OFF, ON99‘living_room_switch‘COMMAND 22‘bedroom_motion_sensor‘MOTION, NO_MOTION100‘living_room_thermostat_connectivity‘OFFLINE, ONLINE 23‘bedroom_smart_speaker‘COMMAND101‘living_room_thermostat_mode‘COOL, HEAT, OFF 24‘bedroom_thermostat_connectivity‘OFFLINE, ONLINE102‘living_room_tv‘OFF, ON 25‘bedroom_window_sensor‘CLOSED, OPEN103‘living_room_tv_state_log‘PERSISTED:ON 26‘calendar_event‘ BIRTHDAY PARTY, DAILY_VENTILATION_HABIT, FAMILY VACATION - BEACH TRIP, MOVING DAY, NEWBORN_CARE_PERIOD, OVERNIGHT_GUESTS_WEEKENDS, SHIFT_WORK_STARTED, VACATION_TRAVEL_14D, WORK_SCHEDULE_NIGHT_SHIFT 104‘living_room_vacuum‘OFF, ON 27‘cloud_alarm_panel‘ARMED_AWAY105‘living_room_wall_sconce‘ON 28‘cloud_living_room_hvac‘ECO106‘living_room_window_sensor‘CLOSED, OPEN 29‘cloud_living_room_light‘OFF107‘maintenance_time_window‘STRUCTURED 30‘device_registry‘ NEW_DEVICE:EV_CHARGER, REGISTRY_CONFIRMED:EV_CHARGER, THERMOSTAT_INSTALL:DAY_0 108‘master_bedroom_bedside_lamp‘OFF, ON, TIMEOUT 31‘energy_manager‘DEMAND_RESPONSE, PEAK_SHAVING_ACTIVE109‘master_bedroom_ceiling_light‘OFF, ON 32‘entrance_alarm_panel‘TRIGGERED110‘master_bedroom_motion_sensor‘ MOTION_DETECTED, NO_MOTION, UNKNOWN 33‘entrance_door_sensor‘CLOSED, OPEN, UNAVAILABLE, UNKNOWN111‘master_bedroom_smart_plug‘OFF, ON 34‘entrance_door_sensor_connectivity‘OFFLINE, ONLINE112‘master_bedroom_window_sensor‘CLOSED, OPEN, TIMEOUT 35‘entrance_light‘OFF, ON113‘notification_system‘PUSH_RECEIVED 36‘entrance_motion_sensor‘MOTION, MOTION_DETECTED, NO_MOTION114‘office_ceiling_light‘OFF, ON 37‘entrance_security_camera‘ OFFLINE, RECORDING, RECORDING_ON, STANDBY, UNKNOWN 115‘office_desk_lamp‘OFF, ON 38‘entrance_smart_lock‘LOCKED, UNLOCKED116‘office_motion_sensor‘ MOTION_DETECTED, NO_MOTION, UNAVAILABLE, UNKNOWN 39‘entrance_smart_lock_connectivity‘OFFLINE, ONLINE117‘office_smart_plug‘OFF, ON 40‘entrance_smart_lock_state_log‘LOCK_CMD118‘pets‘NONE 41‘ev_charger_schedule‘SCHEDULE:23:00-07:00;RATED_KW:7.2119‘phone_gps_tracker‘LOCATION:SHANGHAI 42‘garage_door_connectivity‘OFFLINE, ONLINE120‘scene_controller‘MOVIE_MODE, SCENE:GOOD_NIGHT 43‘garage_door_sensor‘CLOSED, OPEN, TIMEOUT121‘security_camera_connectivity‘OFFLINE, ONLINE 44‘garage_door_sensor_connectivity‘OFFLINE, ONLINE122‘shed_sensor_connectivity‘OFFLINE, ONLINE 45‘garage_ev_charger‘OFF, ON123‘smart_home_app‘SYNCED 46‘garage_light‘OFF, ON124‘smart_hub‘OFFLINE, ONLINE, RESTART 47‘garage_motion_sensor‘ MOTION, MOTION_DETECTED, NO_MOTION, UNAVAILABLE, UNKNOWN 125‘smart_hub_command_log‘UNKNOWN_COMMAND 48‘garage_sensor_connectivity‘OFFLINE, ONLINE126‘system_mode‘VACATION_MODE:ON 49‘geofence_event‘APPROACHING_HOME;ETA_MIN:30;HVAC_MODE:PREHEAT127‘thermostat_system‘MODE:ECO_AWAY, MODE:LEARNING 50‘guest_bedroom_ceiling_light‘OFF, ON, TIMEOUT128‘voice_assistant‘FAILURE_START:OFFLINE 51‘guest_bedroom_motion_sensor‘MOTION_DETECTED, NO_MOTION129‘wifi_router‘OFFLINE, ONLINE 52‘hallway_light‘OFF, ON130‘zone_controller‘ZONE_ACTIVATE:LIVING_ROOM_LIGHTS 53‘hallway_motion_sensor‘MOTION, NO_MOTION131‘basement_power_meter‘2000W 54‘hallway_motion_sensor_connectivity‘OFFLINE, ONLINE132‘basement_thermostat‘15-18°C 55‘home_profile‘ NEWBORN:TRUE, PET:CAT, WORK_SCHEDULE:EARLY_SHIFT_STARTED, WORK_SCHEDULE:NIGHT_SHIFT_WORKER, WORK_SCHEDULE:REMOTE_FULL_TIME 133‘bathroom_power_meter‘2000W 56‘hvac_diagnostics‘STRUCTURED134‘bathroom_thermostat‘22-25°C 57‘hvac_failure_prediction‘RISK_HIGH135‘bedroom_power_meter‘64-2000W 58‘hvac_history‘STRUCTURED136‘bedroom_thermostat‘17.1-24°C 59‘ip_camera_front‘FAILURE_START:OFFLINE137‘entrance_door_angle‘0-69° 60‘kitchen_coffee_maker‘OFF, ON138‘entrance_door_sensor_battery‘28-100% 61‘kitchen_dishwasher‘OFF, ON139‘garage_thermostat‘18°C 62‘kitchen_door_sensor‘CLOSED, OPEN140‘grid_voltage‘206V 63‘kitchen_door_sensor_connectivity‘OFFLINE, ONLINE141‘kitchen_power_meter‘0-9000W 64‘kitchen_fridge‘CLOSED, OPEN142‘kitchen_smart_plug_power‘0-1582W 65‘kitchen_hvac‘COOLING, HEATING, OFF143‘kitchen_thermostat‘20-25°C 66‘kitchen_kettle‘OFF, ON144‘kitchen_wifi_rssi‘-85-60DBM 67‘kitchen_light‘OFF, ON145‘living_room_light_power_meter‘60W 68‘kitchen_light_state_log‘PERSISTED:ON146‘living_room_power_meter‘0-3000W 69‘kitchen_microwave‘OFF, ON147‘living_room_smart_plug_power‘0-162W 70‘kitchen_microwave_state_log‘PERSISTED:ON148‘living_room_thermostat‘18.1-25°C 71‘kitchen_motion_connectivity‘OFFLINE, ONLINE149‘living_room_tv_power‘0-180W 72‘kitchen_motion_sensor‘ MOTION, MOTION_DETECTED, NO_MOTION, TIMEOUT, UNAVAILABLE, UNKNOWN 150‘living_room_window_opening_angle‘0-42° 73‘kitchen_motion_sensor_connectivity‘OFFLINE, ONLINE151‘main_power_meter‘120-18000W 74‘kitchen_motion_sensor_state_log‘PERSISTED:NO_MOTION152‘master_bedroom_smart_plug_power‘0-118W 75‘kitchen_oven_plug‘POWER_OFF, POWER_ON153‘master_bedroom_window_opening_angle‘0-29° 76‘kitchen_range_hood‘OFF, ON154‘office_smart_plug_power‘0-147W 77‘kitchen_range_hood_state_log‘PERSISTED:ON155‘smart_hub_command_latency‘120-348MS 78‘kitchen_smart_lock‘LOCKED, UNLOCKED SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral ContextsConference’17, July 2017, Washington, DC, USA Table 6: Context-Independent Anomaly Types Anomaly TypeDescriptionSubtype device_malfunction Device or sensor malfunction, offline status, or abnormal reporting, or inconsistencies between “control/display status” and telemetry readings, physical location, or other supporting evidence. actuator_position_anomaly, alarm_false_trigger, camera_offline, configuration_conflict, invalid_categorical_state, leak_sensor_false_positive, leak_sensor_suspicious, lock_door_state_inconsistent, sensor_unavailable, telemetry_state_conflict safety_hazard High-risk safety events or protection failures related to smoke, gas, water leakage, door/window security, or surveillance, which may lead to fire, explosion, intrusion, or property loss. alarm_sensor_state_conflict, garage_access_risk, gas_detected, open_entry_while_away, security_disabled_with_activity, security_system_disabled, security_system_disabled_with_activity, smoke_detected, smoke_false_alarm_or_sensor_fault, surveillance_disabled, surveillance_disabled_with_activity, unlocked_entry_while_away, unlocked_entry_with_activity, water_leak_detected, window_left_open_in_bad_weather physical_violation Out-of-range, missing, or improperly formatted values, or values that contradict basic physical laws or environmental facts (e.g., humidity readings inconsistent with signs of leakage, or abnormal temperature differences between adjacent rooms). humidity_invalid_value, humidity_out_of_range, humidity_physics_conflict, temperature_invalid_value, temperature_out_of_range, temperature_spatial_inconsistency, temperature_unrealistic_indoor, thermal_distribution_anomaly compound_anomaly Multiple high-risk signals occur concurrently within the same time window, forming compound risks such as fire, intrusion, or utility-related hazards (with overlapping risks that are harder to interpret). fire_signature, intrusion_signature, multiple_safety_alerts, security_posture_failure, water_electric_risk_signature environmental_conflict The on/off status, mode, or target temperature of the HVAC system is inconsistent with ambient conditions such as indoor temperature or open-window ventilation, leading to conflicting control objectives or unreasonable adjustment directions. cooling_target_conflict, cooling_temperature_conflict, heating_target_conflict, heating_temperature_conflict, hvac_not_responding, hvac_running_while_off, hvac_status_mode_mismatch, hvac_window_conflict occupancy_mismatch Occupancy profiling (e.g., at-home or away status) conflicts with activity evidence such as motion sensors, access control logs, or power consumption patterns, indicating potential labeling errors or suggesting abnormal behavior or intrusion risks. activity_signature_with_no_occupants, device_active_while_away, motion_with_no_occupants, multi_room_activity_with_no_occupants Table 7: Context-Dependent Anomaly Types Anomaly TypeDescriptionSubtype device_malfunction Device or sensor failure, disconnection, or performance degradation, manifested as loss of controllability, lack of reporting, or reported readings/states that contradict other sensor data or power consumption patterns observed during the same time period. device_failure, parameter_drift, parameter_spike, sensor_stuck, sensor_drift, thermostat_failure, device_aging, component_cascade_failure, predictive_failure_risk causal_temporal Anomalous causal and temporal relationships between events, such as expected events not occurring, events happening out of order, significantly delayed responses, or durations/frequencies deviating from normal patterns. causal_break, causal_delay, duration_anomaly, sequence_violation, timing_anomaly, periodicity_anomaly, frequency_anomaly, period_anomaly statistical_anomaly The data significantly deviates from historical patterns or similar scenarios in terms of overall level, fluctuation amplitude, or the synchronization between key indicators. correlation_anomaly, baseline_deviation, point_anomaly, contextual_anomaly, collective_anomaly, spurious_event, distribution_shift, apparent_anomaly_mostly_seasonal safety_hazard High-risk safety incidents related to water leakage, intrusion, security breaches, device compromise, or electricity theft, which may pose direct threats to personal or property safety. water_leak, hidden_water_leak, collective_security_trigger, security_issue, unauthorized_entry, progressive_device_compromise, power_theft_upstream climate_anomaly Anomalies in HVAC system cooling/heating/ventilation and zone control, such as malfunctions, overheating, ventilation issues, mode conflicts, or imbalanced zone regulation. hvac_failure, excess_heat, ventilation_anomaly, hvac_mode_conflict, hvac_zone_conflict, hvac_zone_imbalance network_disconnect Anomalies in network links, gateways, or integration channels, leading to communication interruptions/fluctuations, state desynchronization, or cross-zone synchronization failures. network_failure, hub_offline, wireless_interference, network_topology_anomaly, state_sync_failure, zone_sync_failure, integration_failure, sensor_fusion_anomaly common_sense_conflict The event stream exhibits clear inconsistencies between control mode/state flags and actual device status or power consumption changes, or displays phenomena such as frequent short-term device toggling, partial execution of scenes, or long-term failure to achieve intended control objectives. rule_conflict, scene_mode_conflict, scene_partial_failure, coordination_conflict, feedback_loop, security_mode_conflict, energy_management_conflict occupancy_anomaly Occupancy profiles (e.g., home, away, sleep) are inconsistent with behavioral evidence such as actions, access control, or power consumption, or there are anomalous activities such as spontaneous device activation without clear triggers. occupant_incapacitation, unexpected_activity, occupancy_contradiction, ghost_device_activation energy_anomaly Anomalous fluctuations in power consumption, discrepancies between different meters or sub-metering statistics, or economic anomalies (e.g., cost increases despite expected savings), deviating from normal energy usage patterns. power_fluctuation, energy_discrepancy, economic_anti_optimization