Paper deep dive
Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI
David Joohun Kim, Daniyal Anjum, Bonny Banerjee, Omar Abbasi
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 4:31:26 AM
Summary
The paper introduces the Selective Attention System (SAS), an on-device implementation of Sequential Device-Addressed Routing (SDAR). SAS addresses the challenge of device-addressed speech detection in multi-speaker environments under strict pre-ASR edge constraints (latency <150ms, footprint <20MB). By modeling the task as a sequential decision process over interaction history rather than an utterance-local classification, SAS achieves high performance (F1=0.95 with audio+video fusion) and demonstrates that causal interaction history is critical for resolving ambiguous utterances.
Entities (4)
Relation Signals (3)
Selective Attention System → implements → Sequential Device-Addressed Routing
confidence 100% · present the Selective Attention System (SAS), an on-device implementation that instantiates this formulation.
Selective Attention System → runson → ARM Cortex-A
confidence 95% · SAS runs fully on-device on ARM Cortex-A class hardware
Sequential Device-Addressed Routing → addresses → Device-Directed Speech Detection
confidence 90% · SDAR is not merely a new model for DDSD; it is an alternative formulation of the decision problem
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We study device-addressed speech detection under pre-ASR edge deployment constraints, where systems must decide whether to forward audio before transcription under strict latency and compute limits. We show that, in multi-speaker environments with temporally ambiguous utterances, this task is more effectively modelled as a sequential routing problem over interaction history than as an utterance-local classification task. We formalize this as Sequential Device-Addressed Routing (SDAR) and present the Selective Attention System (SAS), an on-device implementation that instantiates this formulation. On a held-out 60-hour multi-speaker English test set, the primary audio-only configuration achieves F1=0.86 (precision=0.89, recall=0.83); with an optional camera, audio+video fusion raises F1 to 0.95 (precision=0.97, recall=0.93). Removing causal interaction history (Stage~3) reduced F1 from 0.95 to 0.57+/-0.03 in the audio+video configuration under our evaluation protocol. Among the tested components, this was the largest observed ablation effect, indicating that short-horizon interaction history carries substantial decision-relevant information in the evaluated setting. SAS runs fully on-device on ARM Cortex-A class hardware (<150 ms latency, <20 MB footprint). All results are from internal evaluation on a proprietary dataset evaluated primarily in English; a 5-hour evaluation subset may be shared for independent verification (Section 8.8).
Tags
Links
- Source: https://arxiv.org/abs/2604.08412v1
- Canonical: https://arxiv.org/abs/2604.08412v1
Trouble viewing inline? Open PDF directly →
Full Text
74,111 characters extracted from source content.
Expand or collapse full text
Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI David Joohun Kim Daniyal Anjum Bonny Banerjee Omar Abbasi Attention Labs contact@attentionlabs.ai Abstract We study device-addressed speech detection under pre-ASR edge deployment constraints, where systems must decide whether to forward audio before transcription under strict latency and compute limits. We show that, in multi-speaker environments with temporally ambiguous utterances, this task is more effectively modelled as a sequential routing problem over interaction history than as an utterance-local classification task. We formalise this as Sequential Device-Addressed Routing (SDAR) and present the Selective Attention System (SAS), an on-device implementation that instantiates this formulation. On a held-out 60-hour multi-speaker English test set, the primary audio-only configuration achieves F1 = 0.86 (preci- sion = 0.89, recall = 0.83); with an optional camera, audio+video fusion raises F1 to 0.95 (precision = 0.97, recall = 0.93). Removing causal interaction history (Stage 3) reduced F1 from 0.95 to 0.57± 0.03 in the audio+video configuration under our evaluation protocol. Among the tested components, this was the largest observed ablation effect, indicating that short-horizon interaction history carries substantial decision-relevant information in the evaluated setting. SAS runs fully on-device on ARM Cortex-A class hardware (<150 ms latency, <20 MB footprint). All results are from internal evalua- tion on a proprietary dataset evaluated primarily in English; a 5-hour evaluation subset may be shared for independent verification (Section 8.8). Keywords: sequential device-addressed routing· SDAR· device-directed speech detection· causal interaction- state estimation· edge inference· beamforming· temporal context· on-device voice AI 1. Introduction Every ambient voice AI system faces a version of the cocktail party problem [1]: when multiple people are present in a room, which speaker is addressing the device? As voice interfaces move toward continuous, multi- speaker environments in homes, vehicles, clinical settings, and spatial computing, explicit device- addressed detection becomes an increasingly impor- tant component of robust voice system design. Existing components address partial aspects of the routing problem but do not provide a complete real- time decision rule for device-addressed routing. Voice activity detection (VAD) detects speech presence but carries no addressee signal. Wake-word detection ap- proximates intent by requiring a fixed trigger phrase. Wake-word systems achieve high precision (>99%) un- der controlled conditions but impose an explicit inter- action contract: every utterance must be preceded by a trigger phrase, breaking conversational flow and requiring speakers to context-switch between human- directed and device-directed speech. In multi-speaker environments, the trigger phrase itself may occur in non-device-directed speech, and in multi-device set- tings (e.g., two robots in the same room), wake-word routing becomes ambiguous since all devices share the same trigger. Speaker diarization identifies who spoke but not to what. None of these components provides a reliable decision rule for whether an utterance should be forwarded downstream. The Selective Attention System (SAS) addresses this as an explicit pre-ASR inference step, operat- ing as a three-stage on-device cascade: an acous- tic geometry front-end that suppresses off-axis inter- ference, a lightweight utterance-level classifier that scores device-directed probability, and a session-aware temporal context stage that conditions routing on in- teraction history. Together these stages produce a bi- nary routing decision within 150 ms on ARM Cortex- A class hardware [8]. Device-addressed detection in multi-speaker environments depends strongly on tem- poral interaction modelling under the evaluated con- straints, and the three-stage architecture reflects this requirement directly. We use SAS to refer to the full routing system throughout; where needed, we distinguish the learned inference stages from surrounding pipeline compo- nents by stage number. 1 arXiv:2604.08412v1 [cs.SD] 9 Apr 2026 At scale, routing errors translate directly into un- necessary inference cost and degraded interaction re- liability, making upstream gating highly valuable in production voice pipelines. Every false trigger that clears the gate reaches ASR, LLM inference, and TTS, consuming the full downstream pipeline on audio that should have been discarded; a precise upstream gate reduces this unnecessary downstream inference. We describe the deployment constraints, system design, and internal evaluation results. Core con- tribution. We make two linked contributions. First, we formalise a deployment-specific setting for device-addressed detection: pre-ASR routing un- der causal, bounded-memory, edge-constrained infer- ence. We refer to this setting as Sequential Device- Addressed Routing (SDAR), in which the system de- cides whether to forward, suppress, or abstain us- ing current evidence, short-horizon interaction his- tory, and deployment-specific cost tradeoffs. Second, we present SAS as a compact on-device implementa- tion of this formulation and show, through ablation on our internal evaluation set, that causal interaction history produced the largest observed gain among the tested components. Contributions. 1. Problem reframing: We formalise pre-ASR device-addressed routing under edge constraints as a sequential decision problem (SDAR), explicitly incorporating bounded memory and asymmetric downstream cost. 2. System instantiation: We present SAS, a com- pact on-device architecture that decomposes rout- ing into spatial filtering, utterance-level evidence extraction, and causal interaction-state estima- tion. 3. Ablation evidence: On our internal multi- speaker evaluation set, removing the temporal- context stage caused the largest performance drop among the tested components, suggesting that short-horizon interaction history carries substan- tial decision-relevant information in this setting. Our goal is not to claim state-of-the-art performance across all DDSD settings, but to identify and opera- tionalise the decision structure induced by pre-ASR edge deployment. Device-directed speech detection has been studied extensively as an utterance-level clas- sification problem [11, 19, 22, 23]. Recent work has extended DDSD with multimodal LLMs [12, 13, 15] and streaming on-device architectures [24, 25]. How- ever, prior DDSD work spans post-ASR systems that exploit transcripts or decoder states, on-device acoustic detectors for streaming false-trigger mitiga- tion, and follow-up systems that incorporate prior- query context. Our distinction is not that prior DDSD lacks these ingredients entirely, but that we study their combination in a specific deployment setting: pre-ASR routing under causal, bounded- memory, edge-constrained inference. Neither formu- lation captures the deployment problem studied here, in which the system must make a pre-ASR routing decision causally, under bounded latency and mem- ory, with asymmetric downstream cost. Under this view, SDAR is not merely a new model for DDSD; it is an alternative formulation of the decision prob- lem induced by edge deployment constraints, which we show is empirically better aligned with system be- haviour under the evaluated conditions. Specifically, we show that when inference must operate causally, latency and memory are bounded, and downstream cost is asymmetric, the problem is naturally framed as a sequential decision process over interaction state (SDAR). Under this formulation, prior DDSD systems typically operate on a more restricted version of the problem in which interaction history is either unavail- able or incorporated only through post-ASR features. 2. Sequential Device-Addressed Routing (SDAR): Problem Formulation 2.1 Device-addressed detection Let x t denote a streaming audio frame at time t. The problem assigns a label y t ∈ 0, 1, 2 at each decision step: y t = 0 silent or no active speech y t = 1 speech, addressed to a person y t = 2 speech, addressed to the device The system outputs a tuple (y t , c t ): predicted class and confidence score c t ∈ [0, 1]. Audio is forwarded downstream only when y t = 2 and c t ≥ τ. When c t < τ, the system abstains (fail-closed behaviour). Pipeline-level vs. module-level task. The three- class label set above describes the full pipeline-level routing decision. Within this pipeline, SAS operates on VAD-positive segments only: an external VAD handles the silent/non-silent distinction (y t = 0 vs. y t ∈ 1, 2), and Stage 2 performs binary classi- fication (device-directed vs. non-device-directed) on speech segments, modulated by Stage 3’s temporal context. Evaluation metrics in Section 8 are com- puted over the SAS module’s binary routing decisions on VAD-positive segments unless otherwise noted. This makes τ a tunable operating point: operators shift it to trade false-trigger rate against miss rate for their environment. Internal evaluation figures are reported at τ = 0.70, the crossover point at which precision and recall are approximately balanced (97% 2 and 93% respectively in the primary A+V configura- tion; 89% and 83% in audio-only fallback). A naive two-class decomposition (VAD followed by an utterance-local binary device-directed classi- fier) suffers from two compounding failure modes: (1) VAD false-accepts non-speech and forwards it to the secondary classifier; (2) a binary classifier oper- ating without temporal interaction history lacks the context required to resolve addressee ambiguity in extended multi-party conversation. SAS retains an external VAD for speech segmentation but addresses these failure modes by conditioning the routing deci- sion on causal interaction history (Stage 3), not on the current utterance alone. For pre-ASR edge deployment, device-addressed de- tection is not well-posed as an utterance-local classi- fication problem. The deployment task is instead to make a causal routing decision over a partially ob- served interaction process, in which the addressee of the current utterance may be undecidable from the current utterance alone and recoverable only from short-horizon interaction history. Why utterance-local formulations are insuffi- cient. Consider utterances such as “turn that on,” “do it again,” or “what did you say.” These are acous- tically and lexically indistinguishable across addressee classes. Formally, there exists a set of utterances X t for which P (y = device | X t ) ≈ P (y = person | X t ). For these utterances, a classifier restricted to X t may be unable to resolve addressee reliably; the missing information can instead come from short-horizon in- teraction history H t . In the evaluated setting, com- bining current-utterance evidence with short-horizon interaction history provides a practical representation for this routing decision, which motivates SDAR as a causal decision process over interaction state. We formalise this deployment task as Sequential Device-Addressed Routing (SDAR). At each step t, the system observes utterance ev- idence X t and bounded interaction history H t , and selects an action: a t ∈forward, suppress, abstain. The action is taken under three constraints: (1) no ac- cess to future context; (2) bounded latency and mem- ory compatible with edge deployment; and (3) asym- metric downstream cost, since forwarding a non- device-directed utterance incurs avoidable ASR, LLM, and TTS computation. The latent variable of interest is whether the current conversational state licences a device-directed interpretation. We define Sequen- tial Device-Addressed Routing (SDAR) as the appli- cation of cost-sensitive sequential decision theory to the device-addressed routing problem. The routing action is selected under: a t ∼ π(X t , H t ; τ, C), where τ is a deployment-specific operating threshold and C = c fwd /c miss is a cost ratio between false for- wards and missed device-directed turns. In practice, τ is selected on a held-out validation split to approx- imate this tradeoff rather than derived as a closed- form optimum. We report τ = 0.70 and τ = 0.82 as two illustrative operating points corresponding to different cost regimes. The threshold τ may be se- lected based on deployment-specific cost ratios. The label y t ∈0, 1, 2 is an internal intermediate; the de- ployment output is the routing action. Formulations restricted to X t alone are insufficient under these con- ditions: it ignores interaction state that is decision- relevant but not recoverable from the current utter- ance in isolation. On the evaluation set described in Section 8 and under the tested model class, methods restricted to utterance-local features reach approximately 0.57 ± 0.03 F1. Because the evaluation set was constructed to include temporally ambiguous utterances (over- lapping speech, rapid turn-taking, ambiguous follow- ups), this figure characterises performance under con- ditions where interaction history is most decision- relevant. On corpora with fewer multi-turn ambigui- ties, the gap between utterance-local and interaction- history-aware methods may be smaller. Utterances such as “turn that on,” “what did you say,” and “yeah do it again” are acoustically indistinguishable across addressee classes; the addressee can only be resolved through interaction history. This observation is con- sistent with prior findings on temporal context in ad- dressee detection [4,25]; our contribution is its opera- tionalisation under pre-ASR edge constraints. Verifi- cation on independent datasets is needed (Table 6). The SDAR formulation defines the minimal infor- mation required to resolve device-addressed routing under causal and latency constraints. Interpretation as a partially observed decision process. SDAR can be viewed as a partially ob- served decision process in which the true conver- sational regime (device-engaged, person-directed, or transition) is latent and must be inferred from stream- ing observations. The system receives observations X t and maintains a bounded-memory belief over in- teraction state using H t , selecting routing actions un- der asymmetric cost. This interpretation connects the routing problem to the broader literature on POMDP- based spoken dialog management, where the dialog state is hidden and actions are chosen from a main- tained belief state. The key deployment-specific con- straint is that belief maintenance must be causal, bounded in memory, and executable within the la- 3 tency budget of edge hardware. 2.2 The routing gate abstraction SAS functions as a routing gate: its pipeline-level output is binary (forward or discard). An exter- nal VAD provides speech/non-speech segmentation; SAS’s learned classifier then determines whether the speech segment is device-directed. This can replace wake-word detection in fully open-mic deployments or complement it in hybrid configurations (Section 10), supplying the addressee routing function absent from standard VAD and unaddressed by existing pipeline components. It operates as a pre-ASR routing layer that integrates without modifying downstream com- ponents. Section 5 situates SAS within the full voice AI stack. 3. Related Work For a systematic review of acoustic addressee detec- tion methods, see Siegert et al. [22], whose PRISMA- style review screened 1,581 studies and retained 23, establishing the field’s research base. 3.1 Device-directed speech detection Device-directed speech detection (DDSD) classi- fies whether a spoken utterance is addressed to a voice assistant or is background/side speech. The problem was first framed as “learning when to lis- ten” by Shriberg et al. [19], who distinguished system-addressed from human-addressed speech in multiparty dialog. Mallidi et al. [11] introduced the modern DDSD formulation, combining acous- tic LSTM embeddings, ASR decoder features, and character embeddings to achieve 5.2% EER on far- field voice-controlled devices, and motivated wake- word-free follow-up queries. Subsequent work im- proved utterance-level classification through acous- tic and ASR-decoder features [23] and streaming on- device architectures [24,25]. Recent work has applied large language models to DDSD. Wagner et al. [12] combine acoustic, lex- ical, and ASR-decoder signals in a multimodal LLM, achieving 6.5% EER with a 1.5B-parameter GPT-2 model. Palaskar et al. [13] introduce Fusion Low Rank Adaptation (FLoRA) for efficient multimodal adap- tation, achieving performance parity with full fine- tuning while training only 1–5% of parameters. Wag- ner et al. [15] unify voice-trigger detection, DDSD, and ASR in a single∼8B-parameter model (SELMA). Chi et al. [16] apply knowledge distillation to com- press a 79M-parameter teacher to a ∼5M-parameter on-device student with∼22% average EER reduction. Prior DDSD work spans a range of settings: post- ASR systems that exploit transcripts or decoder fea- tures [11, 12, 15], on-device acoustic detectors for streaming false-trigger mitigation [24,25], and follow- up DDSD that models prior-query context [14]. Our distinction is the combination of pre-ASR routing po- sition, causal interaction-state estimation over short- horizon history, and bounded-memory edge deploy- ment under strict latency constraints. These differ- ences limit direct metric comparison; we focus on ar- chitectural and deployment characteristics (Table 10). Accordingly, our contribution is not a like-for-like benchmark claim against prior DDSD systems, but a deployment-specific formulation and reference archi- tecture that makes that formulation operational on- device. 3.2 Temporal context in addressee detection The necessity of conversational context for ad- dressee detection has independent support. Rudovic et al. [14] show that modelling the previous user query reduces false alarms by 20–40% in follow-up conver- sation DDSD, compared to modelling each utterance in isolation. Kong et al. [4] demonstrate that extend- ing temporal context from the current utterance to a 10-second window improves egocentric addressee de- tection (Talking-To-Me) from 59.5% to 67.2% mAP, with performance degrading beyond 15 seconds due to irrelevant history. Shriberg et al. [20] show that tem- poral and spectral dimensions of speaking style carry addressee signal independent of lexical content. These findings converge on SAS’s architectural assumption: the causal interaction-state estimator (Stage 3) recov- ers decision-relevant information that is absent from any single utterance. The ablation in Table 6 confirms this, consistent with the performance ceiling reported for utterance-local DDSD classifiers [23]. 3.3 Egocentric and multimodal addressee detec- tion The Ego4D Talking-To-Me (TTM) benchmark [2] (leading result: 71% mAP [6]) and multimodal ad- dressee detection work by Tsai et al. [21] address struc- turally different problems: different addressee types, modalities, hardware targets, and latency constraints. SAS evaluation figures are not directly comparable to TTM mAP; the TTM literature is cited as research context only. Beyond task formulation, practical eval- uation on Ego4D is infeasible for SAS: TTM defines “talking to me” as human-to-human addressee detec- tion, whereas SAS targets human-to-device routing— a different decision boundary. Additionally, Ego4D provides monaural audio only, with no direction-of- arrival metadata, precluding evaluation of Stage 1 (beamforming). Manual inspection of the Ego4D la- bels also revealed annotation inconsistencies in a sub- set of segments, further limiting its suitability as an external benchmark for this task. 4 3.4 Prosodic features in addressee classification Device-directed speech exhibits measur- able prosodic differences from person-directed speech [9, 10]. Users addressing a device tend to adopt a louder, slower, more deliberate delivery with higher mean fundamental frequency (F 0 ) and increased pitch range relative to casual conversation, a pattern termed hyperarticulation [9]. Shriberg et al. [20] show that rhythm and vocal-effort cues are effective for addressee detection without ASR or dialog context. Krishna et al. [17] confirm that even minimal prosodic features (pitch, voicing, jitter, shimmer) provide complementary signal for DDSD, improving false-accept rate by 8.4% when fused with verbal cues, and that modality dropout dur- ing training makes fusion models robust to missing modalities at inference. Stage 2 of SAS exploits these patterns directly, capturing temporal and spectral modulations of the speech signal. 3.5 VAD, turn-taking, and edge deployment Lightweight VAD systems demonstrate that audio classification is achievable under edge compute con- straints [7]. VAD detects speech presence but pro- vides no addressee signal. SAS is designed as a com- plement to VAD, adding the addressee inference step that VAD alone cannot provide. Turn-taking research establishes that gaps below approximately 200 ms are perceived as natural; gaps above this threshold introduce perceptible hesita- tion [8]. This provides the human-factors basis for the 150 ms latency target. Cornell et al. [26] study DDSD degradation during device playback and report a 56% false-reject reduction through implicit acoustic echo cancellation, a complementary approach to the threshold adjustment recommended in Section 9. 4. SAS: A Reference On-Device Implementa- tion of SDAR 4.1 Deployment constraints Four hard constraints govern the design: 1. End-to-end decision latency under 150 ms 2. Runtime footprint under 20 MB 3. ARM Cortex-A deployment without GPU or NPU baseline requirement 4. Audio-only at baseline; optional camera input where available On the reference platform (ARM Cortex-A72), audio-only end-to-end decision latency is under 55 ms (median 38 ms, p95 51 ms); audio+video end-to-end latency is under 150 ms (median 105 ms, p95 142 ms). Where an NPU is available, classifier inference can be offloaded further. Total runtime footprint is under 20 MB. Per-stage latency and memory breakdowns are available to the supplementary materials. 4.2 Three-stage architecture SAS instantiates the SDAR formulation as three sequentially gated components: (1) acoustic geom- etry for spatial filtering, (2) utterance-level classifi- cation for local evidence extraction, and (3) causal interaction-state estimation for sequential disam- biguation. This decomposition separates what can be inferred from the current signal from what must be in- ferred from short-horizon conversational state. Each component is necessary; removing any one results in substantial performance degradation (Table 6). The architecture is sequential: Stage 2 is invoked only on audio that Stage 1 has not already rejected on spa- tial grounds, and Stage 3 modulates Stage 2’s output only for frames that have cleared Stage 2’s internal confidence floor. This cascade structure concentrates inference budget on genuinely ambiguous frames. 4.2.1 Stage 1: Acoustic geometry A beamforming front-end localises the dominant speech source relative to the microphone array and suppresses off-axis interference (television au- dio, adjacent-room speech, HVAC). Spatial features inform Stage 2’s classifier. Beamforming is signal- processing only (no learned parameters); it requires a minimum of two microphones; single-microphone de- ployments bypass this stage and operate on Stages 2 and 3 only (Section 9). 4.2.2 Stage 2: Utterance-level classification A lightweight 1D-convolutional classifier operating directly on 64-dimensional log-mel filterbank features (25 ms frames, 10 ms hop), without requiring tran- scripts, language models, or language-specific prepro- cessing, estimates the probability that the current ut- terance is device-directed. The model consists of four convolutional blocks (each: 1D convolution, batch normalisation, ReLU, max-pool) followed by a sin- gle GRU layer and a sigmoid output head, trained as a binary classifier (device-directed vs non-device- directed) with cross-entropy loss. The three-class formulation (y t ∈ 0, 1, 2) is resolved at the sys- tem level: VAD provides the silent/non-silent distinc- tion, and Stage 2 handles the device-directed vs. non- device-directed decision. Stage 2 exploits the prosodic patterns described in Section 3 (Section 3.4): device- directed speech exhibits elevated F 0 , reduced speak- ing rate, and increased energy contour relative to person-directed speech [9,10,17,20]. The standard deployment variant (≈435 K parame- ters, ≈520 KB INT8-quantized) targets ARM Cortex- A class hardware and is the basis for all headline evaluation figures reported here. The self-contained 5 footprint suits platform reference designs without ex- ternal dependencies. INT8 post-training quantisation reduces active weight size by 4× with <0.5 F1-point degradation across tested variants. Where a camera is present, skeletal and gaze di- rection features extracted via a lightweight pose esti- mation model (CPU or NPU inference) are fused at the Stage 2 output layer, yielding the primary A+V configuration (F1 = 0.95). The pose estimation stage is optional; without it, the system operates in audio- only fallback mode (F1 = 0.86). The Stage 2 model was trained on the training par- tition of the 600-hour corpus described in Section 8, covering annotated multi-speaker audio with explicit adversarial coverage of overlapping speech and device- addressed interactions. The training set includes both proprietary recordings and publicly available multi- speaker corpora (AMI Meeting Corpus, LibriMix, and internal multi-speaker collections). Training followed a curriculum schedule: the model was first trained on simple two-speaker turn-taking interactions to establish a stable decision boundary, then progres- sively exposed to more complex conditions (overlap- ping speech, three- and four-speaker sessions, adver- sarial follow-ups). Specific composition ratios and per- source sample counts are available upon request. 4.2.3 Stage 3: Causal interaction-state estimation Stage 3 is a small causal Transformer operating over a rolling window of the previous N Stage 2 out- put tuples. Each tuple comprises the Stage 2 con- fidence score, VAD state, and inter-utterance time delta. The context window is fixed at 8 seconds of interaction history, sufficient to capture turn-taking, interruption, re-engagement, and follow-up behaviour. Performance peaks at approximately 8 seconds; be- yond 12 seconds, irrelevant history degrades Stage 3 accuracy, consistent with the degradation reported by Kong et al. [4] beyond 15 seconds in egocen- tric addressee detection. The model predicts at the last timestep, producing a scalar multiplicative prior α t ∈ [0, 1] that modulates the Stage 2 confidence score before threshold comparison: c ′ t = α t ·c (S2) t . Total pa- rameter count: ≈85 K of the system’s 520 K parame- ters. The mechanism is strictly causal: context expires naturally as new tuples arrive without requiring ex- plicit detection of speaker transitions. Its role is to estimate whether the current conversational state li- cences a device-directed interpretation of an otherwise ambiguous utterance. Interpretation as interaction-state estimation. Empirically, Stage 3 behaves like a compact con- text modulator over conversational regime (device- engaged, bystander conversation, or transition). Since α t ∈ [0, 1] is multiplicative, Stage 3 modu- lates routing by controlling how much of Stage 2’s confidence is preserved: following a confirmed device- directed turn, α t rises toward 1.0, preserving Stage 2’s score for subsequent ambiguous utterances; during sustained person-directed interaction, α t decays to- ward 0, suppressing them. The mechanism can only attenuate or preserve confidence, not amplify it be- yond Stage 2’s output. This behaviour is consistent with interaction-state estimation, though it does not by itself prove recovery of a uniquely identifiable la- tent state. No explicit long-horizon session-reset mechanism is used in the reported evaluation; all reported results use only the bounded rolling context described above. The architecture is specified at sufficient detail for independent reimplementation and verification of the ablation results in Table 6; trained weights are not released but the functional interface is fully defined. 4.3 Fail-closed routing Below τ, SAS abstains and no audio is forwarded. False triggers from background conversation are ex- pensive in production pipelines; the fail-closed gate eliminates these at source. The threshold is config- urable per deployment. At τ = 0.70, internal eval- uation (A+V) yields 97% precision and 93% recall; raising τ toward 0.85 increases precision at the cost of recall, suitable for LLM-backed pipelines with high per-query cost. 4.4 On-device data handling Audio the system rejects never leaves the device. The routing decision runs entirely on-device; no audio is transmitted until the gate passes. Model weights stay on the device; no cloud dependency is required for inference. 5. Voice Pipeline Integration 5.1 Position in the stack A complete ambient voice AI pipeline contains five sequential stages: audio capture, pre-processing (VAD), addressee routing (SAS), downstream intelli- gence (ASR → LLM → TTS), and output. Table 1 describes each stage’s role. SAS operates as a modular layer between VAD and ASR without touching anything downstream. Down- stream stages receive only device-directed audio; they do not need to know the gate exists. SAS can there- fore be integrated into any existing voice pipeline without modifying the ASR, LLM, or TTS compo- nents. 6 Table 1. Voice AI pipeline stages. SAS occupies the pre- ASR routing layer, a position currently unaddressed by standard pipeline components. StageFunctionOn-device VADSpeech presence detectionYes SASAddressee routingYes ASRTranscriptionOptional LLM / NLU Intent and responseOptional TTS / Output Response renderingOptional 6. Computational Efficiency 6.1 Motivating calculation This section presents illustrative deployment-level es- timates, not production measurements. The down- stream savings are dominated by the low device- directed base rate, not classifier-specific performance; any pre-ASR gate with reasonable precision yields comparable reductions. In the evaluated ambient deployments, approx- imately 8% of VAD-positive segments are device- directed. Wake-word detection is one gating ap- proach, but it requires a trigger phrase and imposes a rigid per-utterance interaction contract. The eco- nomics below apply to any pre-ASR gate, including SAS deployed alongside or in place of wake words; a structural comparison with wake-word detection is in Section 10. 6.2 Baseline assumptions • Environment: residential smart speaker, daytime active hours. • Speech density: ≈100 VAD-positive segments per hour. • Device-directed fraction: ≈8% of VAD-positive segments are assumed to be genuine device interac- tions, an illustrative ambient estimate. The held- out evaluation set has a higher device-directed frac- tion among VAD-positive segments (≈12%; Sec- tion 8); operators should calibrate this figure to their deployment. • ASR cost (on-device or cloud speech API): ≈200 ms CPU time per segment on-device; cloud API costs are additive. • Cloud LLM cost: $0.01 per call (representative mid-range pricing). 6.3 Gated pipeline behaviour At τ = 0.70 with internal false-trigger rate 2.1%, SAS forwards: N fwd = (100× 0.08) | z device-directed ×0.93 + (100× 0.92) |z non-device ×0.021 ≈ 7.4 + 1.9 = 9.3 segments/hr Under the stated assumptions, this represents a 90.7% reduction in ASR calls relative to ungated operation. Sensitivity to the device-directed fraction is approximately linear: at 15% device-directed (e.g., active command sessions), the reduction is approxi- mately 83%. Table 2 summarises the savings. Interpretation. The magnitude of this reduction is primarily determined by the base rate of device- directed speech rather than any specific classifier. The role of SAS is not to create this reduction, but to realise it reliably under multi-speaker ambiguity without requiring explicit wake-word invocation. The classification overhead incurred by SAS is re- covered in downstream compute savings under all evaluated ambient conditions. The breakeven point requires a device-directed fraction exceeding approxi- mately 85%, not observed in any tested environment. Table 2. Estimated pipeline savings at τ = 0.70 (100 VAD segments/hr, 8% device-directed). Figures are illus- trative. MetricNo gate With SAS Saving ASR calls / hr1009.3 90.7% ASR CPU-time / hr20 s1.9 s 90.5% LLM calls / hr1009.3 90.7% LLM cost / hr$1.00$0.09 91.0% False triggers / hr921.9 97.9% Missed genuine / hr00.6 — SAS overhead: median 38 ms per VAD segment (≈3.8 s CPU/hr at 100 segments/hr); the 18.1 s of ASR CPU saved per hour exceeds the gate cost by≈4.8×. Relative reductions in downstream inference calls are grounded in measured false-trigger rates; absolute cost values depend on deployment-specific parameters (e.g., LLM pricing tier). 7. Deployment Contexts SAS has been evaluated on smart speaker (fixed geometry), connected home audio, compan- ion robot (Reachy Mini 1 ), and multi-agent XR configurations without context-specific retraining or environment-specific acoustic tuning. Smart-speaker and companion-robot deployments are directly cov- ered by the evaluation conditions in Section 8. Auto- motive, hearable (single-microphone, 5 MB budget), and industrial deployments fall outside the evaluated acoustic range (28–85 dBA, RT60< 0.6 s) and are dis- cussed in Section 11. 1 Integration code: https://github.com/attentionlabs/ robot-engage 7 8. Internal Evaluation Reporting conventions. All performance metrics in this section are measured on the held-out test set described below; no estimates or projections are included. The primary metric is F1 on device- addressed class (class 2), macro-averaged across ses- sions. Because the majority of target platforms (smart speakers, hearables, embedded devices) lack cameras, audio-only at τ = 0.70 is reported as the pri- mary deployment configuration throughout this sec- tion. Audio+video results at τ = 0.70 are reported as an upper-bound configuration where camera hard- ware is available. Every table and figure specifies its modality and threshold. 8.1 Dataset and scope Evaluation is conducted on a held-out test set (60 hours) drawn from a 600-hour proprietary multi- speaker corpus collected across real-world residential and office environments. The dataset is constructed to explicitly cover conditions under which utterance- local classification fails, including overlapping speech, rapid turn-taking, and ambiguous follow-up utter- ances. Approximately 20–40% of device-directed ut- terances in the test set are conversational or open- ended (e.g., questions one would pose to a general- purpose AI rather than imperative voice-assistant commands), making them temporally ambiguous in the sense that their addressee cannot be resolved from the utterance alone. Annotation used a two-labeler forced-choice protocol: each utterance segment was in- dependently labeled as silent (0), person-directed (1), or device-directed (2), with disagreements resolved by a third labeler. Inter-annotator agreement statis- tics (Cohen’s κ per class) and per-class confusion rates are documented in the supplementary materials. Class distribution in the held-out set (across all seg- ments including silence): approximately 34% silent, 58% person-directed, and 8% device-directed (approx- imately 4.8 hours of device-directed speech out of 60 hours total). Among VAD-positive (non-silent) seg- ments only, the device-directed fraction is approxi- mately 12%. The limited volume of device-directed test data means that per-condition estimates (e.g., 4-speaker high-noise) rest on a small number of ses- sions; the bootstrap confidence intervals reported be- low quantify this uncertainty. The primary 60-hour test set is proprietary, inher- ently limiting full external validation. To support in- dependent scrutiny, we provide a fully specified eval- uation protocol (class definitions, thresholding proce- dure, model architecture, and scoring methodology) sufficient for third-party replication on comparable multi-speaker datasets. The same ablation behaviour is observed consis- tently across both the full dataset and the evaluation subset, confirming that the reported gains are driven by task formulation and architecture (Sections 2–4), not dataset scale. Hardware. ARM Cortex-A72 at stock clock speeds on standard embedded Linux; also tested on Reachy Mini and microphone configurations from lap- top stereo to 4-mic circular arrays. Environment. Natural room acoustics, no acous- tic treatment; typical office and living-room condi- tions. Noise floor range: 28–85 dBA. Sessions. 1–4 speakers present. One speaker ad- dresses the device; remaining speakers engage in un- scripted side conversation. Speaker counts refer to the number of people present and potentially active in the session; in groups of four, simultaneous overlapping speech from all participants is rare. The evaluated condition reflects natural turn-taking with occasional overlap, not continuous four-way crosstalk. Language. The SAS architecture is language- agnostic: it operates on acoustic and prosodic features without lexical or transcript input. The primary eval- uation corpus is English. Preliminary informal test- ing on non-English speech suggests comparable perfor- mance (expected 86–89% macro F1 across languages), but extensive multi-language evaluation has not yet been conducted. Modality. The deployment baseline reported throughout this section is audio-only (F1 = 0.86), re- flecting the majority of target platforms (smart speak- ers, hearables, embedded devices). Audio+video (F1 = 0.95) is reported as an upper-bound configu- ration for devices with camera hardware. The cas- cade architecture allows each stage to operate in- dependently when upstream modalities are unavail- able, analogous to the modality-dropout strategies employed in multimodal DDSD systems [13, 17] but achieved through cascade design rather than training- time augmentation. Single-microphone audio-only evaluation is reported in Section 9. Metric. F1 score on class 2 (device-addressed), macro-averaged across sessions. F1 is used rather than mAP because the deployment task is a binary routing decision, not ranked retrieval. Statistical reliability. To quantify result sta- bility, we compute bootstrap confidence intervals over sessions (1,000 resamples). For the primary audio+video configuration, F1 = 0.95 with a 95% confidence interval of ±0.02 across resampled ses- sion subsets; for audio-only fallback, F1 = 0.86± 0.02. The ablation results exhibit similar stability: the no-temporal-context condition yields F1 = 0.57± 0.03 across all resamples. Inter-annotator agreement 8 across the held-out set is Cohen’s κ = 0.82 for device- directed vs non-device-directed classification, indicat- ing strong agreement under ambiguous multi-speaker conditions. 8.2 Results Table 3. SAS evaluation summary at τ = 0.70. Preci- sion, recall, and F1 are session-macro averages over VAD- positive segments. Audio-only is the primary deployment configuration; audio+video is an upper-bound configura- tion for devices with camera hardware. MetricValue F1 (audio-only, primary)0.86 Precision (audio-only)0.89 Recall (audio-only)0.83 F1 (audio+video)0.95 Precision (A+V)0.97 Recall (A+V)0.93 Worst-case session F1 † 0.88 Avg. precision (all τ)0.87–0.90 False-trigger rate, baseline2.1% False-trigger rate, TV-active7.8% Operating threshold τ0.70 Decision latency<150 ms Runtime footprint<20 MB Hardware baselineARM Cortex-A72 GPU / NPU requiredNo Network requiredNo † Four speakers present, heavy background noise. Table 4. Per-class confusion matrix at τ = 0.70 (audio- only, primary deployment configuration). Rows are true labels; columns are predicted actions. Values are percent- ages of each true class. Pred. Sil. Pred. Pers. Pred. Dev. True Silent96.23.10.7 True Person-dir.2.493.83.8 True Device-dir.1.115.983.0 The dominant aggregate error mode is person-directed speech misclassified as device-directed (3.8%). Because Table 3 reports session-macro metrics while this matrix aggregates per-segment outcomes, the values should not be compared one-for-one. The second-largest error is missed device-directed speech classified as person-directed (15.9%), which drives the recall gap. 8.3 Per-speaker-count breakdown Table 5 disaggregates performance by number of speakers present. The degradation pattern is mono- tonic: each additional speaker degrades recall more than precision, consistent with the hypothesis that increased cross-talk makes it harder to capture device- directed turns while not appreciably increasing false triggers from non-device speech. F1 is consistent across held-out speakers and record- ing sessions; no per-speaker adaptation or enrolment is required. Table 5. Performance by number of speakers present at τ = 0.70 (audio+video). Speakers present Precision Recall F1 10.99 0.97 0.98 20.98 0.95 0.97 30.97 0.92 0.94 40.94 0.88 0.91 Condition mean0.97 0.93 0.95 1234 0.86 0.88 0.9 0.92 0.94 0.96 0.98 1 P = 0.94 P–R gap F1 = 0.91 R = 0.88 Simultaneous speakers Score ( τ = 0 . 70) Precision F1 Recall Figure 1. Precision, F1, and recall as a function of num- ber of speakers present at τ = 0.70. Shaded region shows the precision-recall gap, which widens with speaker count: cross-talk degrades recall faster than precision. 8.4 Precision-recall curve Figure 2 plots the full precision-recall curve from τ = 0.56 to τ = 0.85. Average precision across the full threshold range is estimated at 0.87–0.90. Two oper- ating points are highlighted: τ = 0.70 for standard deployments (F1 = 0.95, FTR 2.1%) and τ = 0.82 for high-media environments (F1 = 0.92, TV-FTR 3.4%). Full per-threshold precision, recall, F1, and false-trigger-rate values are available in the supple- mentary materials. 9 0.8000.8250.8500.8750.9000.9250.950 Recall 0.88 0.90 0.92 0.94 0.96 0.98 1.00 Precision τ=0.85 τ=0.75 τ=0.63 τ=0.56 AP ≈0.88 τ = 0.82 F1 = 0.92 TV-FTR 3.4% τ = 0.70 F1 = 0.95 FTR 2.1% Figure 2. Precision-recall curve across operating thresh- olds τ ∈ [0.56, 0.85]. Filled square (■): τ = 0.70 (standard). Filled diamond (♦): τ = 0.82 (high-media). Shaded area: AP ≈ 0.88. 8.5 Noise-floor and speaker-count interaction Figure 3 presents F1 across three noise-floor bands and four speaker counts. The single-speaker, low- noise cell (F1 = 0.99) is the easiest evaluated con- dition; the four-speaker, high-noise cell (F1 = 0.88) is the worst characterised, corresponding to the worst- case figure in Table 3. The four-speaker, high-noise condition contains approximately 1–2 hours of device- directed audio; per-condition estimates should be in- terpreted with this limited sample size in mind. 8.6 Ablation as Evidence for the SDAR Formula- tion Video modality contribution. The gap between audio+video (F1 = 0.95) and audio-only (F1 = 0.86) is 9 F1 points. Table 7 stratifies this gap across the noise-floor × speaker-count conditions from Figure 3, reporting audio-only and A+V F1 in each cell. The video contribution is smallest in the single- speaker, low-noise condition (∆ = +0.03) where au- dio alone is near ceiling, and largest in the four- speaker, high-noise condition (∆ = +0.14), where gaze and pose features disambiguate addressee in the conditions that degrade acoustic-only classification most. This confirms that camera hardware provides the greatest marginal value precisely where audio-only performance is weakest. Stage 1 degradation is concentrated in sessions with off-axis interference: F1 on TV-active sessions drops from 0.91 to 0.74 when beamforming is removed, while F1 on sessions without competing audio sources drops only from 0.97 to 0.94. Table 6. Stage ablation at τ = 0.70 (audio+video configuration). One stage removed per row; all others held fixed. ConfigurationF1∆F1 Full model (SAS)0.95— No Stage 1 (no beamforming)0.81 −0.14 No Stage 2 (no classifier)0.74 (±0.02) −0.21 No Stage 3 (no temporal ctx)0.57 (±0.03) −0.38 Rows ordered by ascending impact. In the “No Stage 2” condition, Stage 3 receives a fixed uninformative confidence score (0.5) in place of Stage 2’s output; in the “No Stage 3” condition, the Stage 2 score is compared directly against τ without temporal modulation. Stage 3 removal produces a 38-point F1 drop (bootstrap p < 0.001), exceeding the combined effect of removing Stages 1 and 2. Under this dataset and ablation protocol, removing Stage 3 caused the largest performance drop among the tested components. This suggests that short-horizon interaction history carries substantial decision-relevant information beyond the current utterance in the evaluated multi-speaker setting. This result characterises the evaluated interaction regime rather than establishing a universal bound; the magnitude of this effect is expected to vary with the prevalence of temporally ambiguous utterances and conversational overlap. Model-free temporal baselines (sliding-window majority vote, exponential moving average over Stage 2 scores) recover only a small fraction of this gap. A representative simple rule—forward if two or more of the last four utterances scored above 0.5—yields approximately 3–5 points higher precision but 3–5 points lower recall, producing a more conservative system that misses quick one-off commands. These comparisons suggest that simple temporal smoothing alone does not explain the observed gain. Table 7. Video modality contribution (∆F1 = A+V mi- nus audio-only) at τ = 0.70, stratified by noise floor and speaker count. Audio-only baselines range from 0.96 (1- speaker, low-noise) to 0.74 (4-speaker, high-noise); see Fig- ure 3. Noise floor1 spkr 2 spkr 3 spkr 4 spkr Low (28–50 dBA) +0.03 +0.05 +0.08 +0.10 Med (50–70 dBA) +0.04 +0.07 +0.09 +0.12 High (70–85 dBA) +0.06 +0.09 +0.11 +0.14 8.7 Threshold selection τ = 0.70 was selected on a held-out validation split of 2.3 hours, stratified to preserve the speaker-count and noise-floor distribution of the full corpus, sepa- rate from the 60-hour test set, prior to any evaluation on the test set. The test set was not inspected until threshold selection was locked. The supplementary materials provide threshold tuning guidance and per- environment operating point recommendations; the operating threshold is a per-environment function, not a single global value. 10 Simultaneous speakers Low (28–50 dBA) – Medium (50–70 dBA) – High (70–85 dBA) – Noise floor 0.990.980.970.94 0.980.970.930.91 0.950.930.900.88 1 speaker2 speakers3 speakers4 speakers 1.00 0.98 0.96 0.94 0.92 0.90 0.88 F1 score Figure 3. F1 heatmap: noise floor× speakers present at τ = 0.70. Each cell reports macro-averaged F1 across held-out sessions under that condition. 0.50.60.70.80.91 Full model Full model (SAS) Stage 1 removed (no beamforming) Stage 2 removed (no classifier) Stage 3 removed (no temp. context) 0.95 0.81−0.14 0.74−0.21 0.57−0.38 F1 score (τ= 0.70) Figure 4. Ablation study: F1 when one stage is removed at τ = 0.70. Rows ordered by ascending impact. Error bars show the reported range for Stage 2 (0.74± 0.02) and Stage 3 (0.57± 0.03). Stage 3 (temporal context) is the dominant contributor; its removal reduces F1 by −0.38 (∆F1). The reported results characterise performance un- der the evaluated distribution of speaker counts, noise conditions, and interaction patterns; they should be interpreted as evidence of behaviour under these con- ditions, not as a universal bound on device-addressed detection performance. 8.8 Evaluation limitations Reproducibility and independent verifica- tion. All headline figures are from internal evaluation on a proprietary dataset evaluated primarily in English, without independent au- dit. This is the most significant limitation of this report. Until the SAS-Bench-5h sub- set or a comparable independent corpus pro- duces concordant results, the reported met- rics should be treated as indicative rather than established. To support independent verification, we provide a fixed 5-hour evaluation subset (SAS- Bench-5h) with per-segment labels, scoring scripts, and ARM inference binaries. All reported metrics fol- low a fully specified evaluation protocol (data schema, labelling procedure, scoring methodology, threshold selection), enabling reproduction on any comparable multi-speaker corpus. No hyperparameters or thresh- olds were modified after test-set inspection. Language scope. All formal evaluation sessions are English-language. The architecture is language- agnostic by design (no lexical or transcript features), and preliminary testing on non-English speech sug- gests comparable performance (Section 11). However, extensive multi-language evaluation has not been con- ducted; formal per-language results will be reported once sufficient per-language sample sizes are available (Section 11). Acoustic scope. Generalisation to highly rever- berant spaces (RT60> 0.6 s), noise floors outside the tested range, or more than four speakers present is not characterized here. Hardware variability. Latency figures are from a specific ARM Cortex-A72 configuration. Thermal throttling, OS scheduling variance, and driver differ- ences on other ARM platforms will produce different observed latencies. Reporting gaps. This report does not include detailed per-condition confusion matrices beyond the summary in Table 4, latency distribution histograms (only summary statistics are given), longitudinal de- ployment data, or detailed per-utterance error analy- sis beyond the categorical failure modes in Section 9. These are available upon request on request. Structural limitations and remediation. Three limitations of this evaluation are structural rather than incidental. (1) The evaluation is internal and unaudited. (2) Multi-language performance has not been formally characterised, though the language-agnostic architecture and preliminary test- ing suggest comparable cross-language performance (Section 8); formal multi-language evaluation will be reported once per-language sample sizes are sufficient (Section 11). (3) The Stage 3 causal interaction-state mechanism is described at the architectural level sufficient for independent replication of the ablation results; full implementation details, including the session-boundary reset mechanism, are available upon request. 11 9. Known Failure Modes Observed failure modes fall into three structural categories. Signal ambiguity: cases where device- directed and non-device-directed speech are acousti- cally indistinguishable (e.g., television dialogue with interrogative prosody); these require higher thresh- olds or auxiliary signals. Context saturation: cases where interaction history becomes unreliable (e.g., >4 simultaneous speakers), degrading the interaction- state estimator. Distribution shift: cases where fea- ture distributions differ from training (e.g., untested acoustic environments), leading to potential misclas- sification. Illustrative failure cases. (1) A television char- acter asks “what did you say?” with interrogative prosody; the system may incorrectly forward due to strong prosodic similarity to device-directed speech. (2) A user addresses the device, then immediately turns to another person and says “do it again”; with- out sufficient temporal decay, residual interaction state may cause incorrect forwarding. (3) Two speak- ers talk simultaneously while a device-directed com- mand occurs during overlap; degraded signal separa- bility may cause the system to suppress the genuine command. Television and media audio. The most com- mon false-trigger class is television dialogue with in- terrogative prosody (direct questions addressed to on- screen characters) whose acoustic and prosodic fea- tures closely resemble device-directed speech. In TV- active sessions, false-trigger rate rises from 2.1% to 7.8% at τ = 0.70. Raising τ to 0.82 reduces this to 3.4% at a 6-point recall cost. Operators in high-media environments should use τ = 0.82 as their baseline op- erating point. The degradation from media playback is a known challenge in DDSD systems; Cornell et al. [26] report a 56% false-reject reduction through implicit acoustic echo cancellation during device play- back, a complementary approach to the threshold ad- justment recommended here. More than four speakers present. Above four speakers present, the temporal context buffer accu- mulates ambiguous turn-taking patterns that degrade Stage 3 classification. In preliminary internal test- ing with five speakers present, F1 drops below 0.75. Formal characterisation of five- and six-speaker con- ditions is ongoing. Non-English and accented speech. The SAS architecture operates on acoustic and prosodic fea- tures without lexical input, making it language- agnostic by design. The primary evaluation was conducted on North American and British English; preliminary informal testing on non-English speech suggests comparable performance (Section 8), but this has not been formally characterised. Tonal lan- guages (Mandarin, Thai) and languages with differ- ent device-directed speech registers may exhibit dif- ferent prosodic feature distributions, though relative rather than absolute prosodic features (Section 11) is expected to mitigate this. Formal per-language eval- uation is described in Section 11. Single-microphone fallback. Stage 1 beamform- ing requires a minimum of two microphones. On single-microphone devices, Stage 1 is bypassed and the system operates on Stages 2 and 3 only. Single- microphone evaluation on the held-out test set yields F1 = 0.84 (precision 0.88, recall 0.81) at τ = 0.70, a reduction of 11 F1 points relative to the multi-mic baseline. Full single-mic precision-recall curves are in the supplementary materials. A purpose-built hearables model at a 5 MB param- eter budget constitutes a separate development effort; estimated baseline F1 under those constraints is ap- proximately 60% (Section 11). Session boundary and context reset. If the Stage 3 timeout is set too long, carry-over context from a prior user can suppress legitimate device- directed classifications from a new user. The appro- priate timeout depends on deployment context and is covered in the supplementary materials. 10. Discussion 10.1 Comparison with VAD-only routing Table 8 compares SAS against baseline configura- tions evaluated on the held-out set. The VAD-only row uses measured recall (≈0.99): energy-threshold VAD misses low-energy speech in quiet conditions where SAS can still detect speech activity via visual features (lip movement). Precision reflects the ambi- ent device-directed fraction (≈8%). The remaining rows use the same internal evaluation protocol as the headline figures. Table 8. Comparison with routing baselines at τ = 0.70. The VAD-only row is a reference floor, not a competing system: VAD provides no addressee signal and is included only to illustrate the base-rate-driven precision bound. All other rows from the held-out set. SystemPRF1 VAD only (Silero)≈0.08 ≈0.99 ≈0.15 S1+S2 only (no ctx) a 0.55 (±.04) 0.55 (±.04) 0.57 (±.03) SAS audio-only0.890.830.86 SAS audio+video0.970.930.95 a Stage 3 removed; corresponds to the ablation condition in Table 6. All SAS figures at τ = 0.70. Under the evaluated protocol, SAS audio+video achieves a≈38 point F1 advantage over the strongest 12 utterance-local baseline. This gap is measured on a dataset constructed to include temporally ambigu- ous utterances; the margin may narrow on corpora with fewer multi-turn ambiguities. To verify that the confidence score c t supports principled thresh- old selection across deployments, we assess calibra- tion on the held-out set. After binning predictions into 10 equal-width confidence intervals, SAS exhibits well-separated score distributions for device-directed and non-device speech, with the majority of device- directed utterances scoring above 0.80 and the major- ity of non-device utterances scoring below 0.30. This separation ensures that operators tuning τ per envi- ronment are making decisions on a meaningful prob- ability scale, not merely finding an empirically ade- quate cutpoint on an arbitrary score. Reliability dia- grams and expected calibration error figures are avail- able in the supplementary materials. 10.2 Comparison with wake-word detection Wake-word detection imposes a fixed UX contract: a mandatory trigger phrase at the start of every in- teraction, a known false-positive rate on phonetically adjacent speech, and a mandatory trigger-phrase in- teraction overhead per interaction [27]. These are in- herent tradeoffs of a simple, deterministic activation scheme. SAS operates under different constraints: addressee-conditioned routing accuracy, no manda- tory trigger phrase, and sub-perceptual latency. Both systems output a continuous confidence score; the distinction lies in what triggers inference and how the score is computed. The cost is a configurable threshold that must be tuned per deployment. Pair- ing SAS with a wake-word detector pushes precision higher than either system alone; deployed without wake words, it supports natural group conversation with no per-utterance trigger requirement. Table 9 provides a structural comparison. A direct F1 com- parison on the same test set is not possible: the eval- uation corpus was collected without a wake-word pro- tocol, so no wake-word triggers exist in the ground truth and a wake-word detector cannot be meaning- fully scored against it. 10.3 Structural comparison with DDSD systems Table 10 compares SAS against published DDSD systems on architectural dimensions. Direct met- ric comparison is not possible due to differences in datasets, task formulations, and evaluation protocols; the table highlights structural differences in pipeline position, compute requirements, and temporal mod- elling. On comparability. Prior DDSD systems are typ- ically evaluated in single-speaker or post-ASR set- tings, where addressee ambiguity is limited or re- Table 9. Structural comparison with wake-word detec- tion. Property Wake-wordSAS (τ = 0.70) TriggerFixed phraseAny device-directed speech LatencyLow (processing)<150 ms (end-to-end) False pos. Phonetic similarity Tunable via τ OutputConfidence scoreConfidence score UXTrigger requiredNatural, unprompted speech Footprint Low<20 MB, ARM Cortex-A solved through lexical or decoder-derived features. Under these conditions, utterance-local classification is often sufficient. The SDAR setting studied here im- poses stricter constraints: (1) no access to transcripts or ASR decoder states, (2) multi-speaker overlap with ambiguous addressee cues, and (3) causal, bounded- memory inference under latency constraints. Under this constraint set, utterance-local methods degrade substantially (≈0.57 F1 in our evaluation). The com- parison is therefore not between models on the same problem, but between different formulations of the routing decision. Table 10. Structural comparison with published DDSD systems. Most listed systems exploit ASR transcripts or decoder features; SAS operates pre-ASR without tran- script access. Metrics are not comparable across systems (see text). System PositionASR Ctx ParamsHW [11]Post-ASRYes No 16 MGPU [12]Post-ASRYes No 0.1–1.5 BA100 [15]ReplacesYes No∼8 B16× A100 [16]Post-VADNo No∼5 MOn-device SAS Pre-ASRNo Yes∼520 KARM 10.4 Hybrid deployment with wake-word detec- tion In production voice pipelines, the first interaction is typically initiated by a wake word, with subsequent follow-up queries handled without re-invocation [11, 14]. SAS supports this hybrid configuration: the wake-word detector handles the initial trigger with high precision, and SAS takes over for follow-up rout- ing using its temporal context stage to determine whether subsequent speech is a continuation of the de- vice interaction or a return to person-directed conver- sation. This mirrors the deployment model described in [14], where modelling the initial query improves follow-up classification by 20–40%. 13 10.5 Generalisation beyond voice interfaces The SDAR formulation applies to any system that must decide whether to act under partial observability and asymmetric cost. Examples include multi-agent robotics (determining which agent a command is di- rected to), in-vehicle multi-occupant systems (resolv- ing which seat initiated a command), AR/XR sys- tems (resolving user intent in shared environments), and continuous sensing systems that must gate down- stream inference. Device-addressed routing is there- fore one instance of a broader class of causal routing problems over interaction state. 11. Future Work 11.1 Formal multi-language evaluation The SAS architecture is language-agnostic by de- sign, operating on acoustic and prosodic features without lexical input. Preliminary testing on non- English speech suggests expected performance of 86– 89% macro F1 across languages. Formal multi- language evaluation across target deployment mar- kets (Mandarin, Japanese, German, Spanish, French, Hindi) is planned to validate this expectation with sta- tistically rigorous per-language sample sizes. Cross- language robustness may be further improved via speaker-normalised pitch dynamics: replacing abso- lute F 0 elevation features with relative pitch-contour shifts within each speaker’s baseline, isolating intent- correlated prosodic variation that remains stable across tonal and non-tonal languages. 11.2 Automotive acoustic profile Automotive in-cabin voice falls outside the evalu- ated acoustic range. Key challenges include broad- band road noise, multi-occupant proximity, and vari- able RT60. An automotive-specific evaluation dataset is under development in partnership with OEM inte- gration teams. 11.3 NPU-optimised model variants An NPU-optimised variant using TFLite delegate and ONNX Runtime inference paths is under devel- opment, targeting effective latency below 60 ms and power consumption below 10 mW on representative NPU-enabled SoCs. This would extend the deploy- ment range to hearable-class hardware with stricter power budgets. 11.4 Hearables and single-microphone Single-microphone, sub-5 MB deployment presents a distinct modelling challenge: Stage 1 is unavailable and Stage 3’s parameter budget must be compressed by approximately 6×. Preliminary estimates place baseline F1 at approximately 0.60 under these con- straints, but no controlled evaluation exists. It is un- clear whether the SDAR formulation’s emphasis on temporal context remains the dominant factor when spatial filtering is unavailable, or whether the relative contribution of stages shifts under single-channel in- put. 11.5 Multimodal teacher-student distillation The gap between the primary A+V configuration (0.95 F1) and audio-only fallback (0.86 F1) is cur- rently addressed only by graceful degradation. Multi- modal teacher-student distillation could narrow this gap: train a strong audio+video teacher on both la- belled and large-scale unlabelled multi-speaker record- ings, then distil not only the final routing probabilities but also the teacher’s intermediate interaction-state representations into an audio-only student. With modality-dropout augmentation during distillation, the student can learn to infer visual-style interaction cues (gaze dynamics, turn-taking patterns) purely from audio prosody and rhythm. Knowledge dis- tillation techniques such as those in [16], which re- port compressing 79M-parameter ASR encoders into ∼5M-parameter on-device students, provide a start- ing point for this approach; however, the training data and evaluation protocol for [16] are not pub- licly available, limiting independent assessment of their reported compression ratios and accuracy. The expected outcome is a meaningful narrowing of the audio-only gap, though achieving 90%+ macro F1 un- der a sub-1 MB deployment constraint (corresponding to well under 1M parameters) remains an open re- search question. 11.6 Explicit causal belief-state tracking in Stage 3 The current Stage 3 is a learned attention-based modulator over a rolling context window. Replacing or augmenting it with an explicit causal belief-state tracker (e.g., a compact recurrent state-space model) would maintain a low-dimensional latent state rep- resenting the current “conversation regime” (device- engaged, bystander-social, or transition). A learned soft boundary detector (triggered by prosodic reset cues, long gaps, or speaker-change signals) would address the carry-over suppression failure mode de- scribed in Section 9 and improve worst-case session F1 beyond four speakers present. Saliency-weighted con- text pruning—replacing chronological expiration with an interaction-relevance score—would preserve high- confidence device-directed history while discarding ir- relevant background chatter in high-density environ- ments. 14 11.7 Media-aware rejection pathway Television and media audio is the dominant produc- tion false-trigger source (Section 9). The current mit- igation (raising τ) trades recall for precision. Adding a parallel media-rejection head in Stage 2, trained to separate device-directed speech from TV, news, and podcast audio with similar prosody, would address this directly. Reference-synchronous gating—cross- correlating the input signal against the on-device au- dio output buffer—could apply a media-suppression penalty to utterances with high periodic similarity to system playback, maintaining low false-trigger rates without a recall-degrading threshold shift. Where de- vice playback state is available, a lightweight implicit acoustic-echo-cancellation path (inspired by [26]) can fuse a low-latency reference signal into Stages 1/2, suppressing false triggers without recall cost. 12. Conclusion Under the evaluated conditions, the central chal- lenge in pre-ASR device-addressed routing is not ut- terance classification alone but causal estimation of interaction state: removing temporal context reduces F1 by 38 points, exceeding the effect of removing spa- tial filtering or utterance-level classification individ- ually. This supports our claim: under edge deploy- ment constraints, device-addressed detection, under the evaluated pre-ASR multi-speaker conditions, is more effectively modelled as sequential routing over interaction state than as isolated utterance classifica- tion. By formalising this as SDAR and implementing it within strict on-device constraints (sub-150 ms la- tency, sub-20 MB footprint, ARM CPU only), the pri- mary audio-only configuration achieves F1 = 0.86, ris- ing to 0.95 where camera hardware is available. Under illustrative ambient assumptions (8% device-directed base rate), pre-ASR gating reduces downstream infer- ence by approximately 90%, a saving dominated by the low base rate. All results are from internal eval- uation on a dataset evaluated primarily in English; the language-agnostic architecture shows promising preliminary cross-language results (Section 8), pend- ing formal multi-language evaluation. The modular, pipeline-compatible design makes SAS reusable across voice-enabled product categories. Independent verifi- cation is supported through the SAS-Bench-5h evalu- ation subset. Limitations and Disclosure All reported results are from internal evaluation on a proprietary dataset without independent third- party auditing. The test set was not used during model development or threshold selection. Evaluation conditions, per-speaker breakdowns, ablation details, and known limitations are documented in Sections 8– 9. Computational-efficiency figures (Section 6) are illustrative. Formal evaluation sessions are English- language; the architecture is language-agnostic with promising preliminary cross-language results, pending formal per-language evaluation (Section 9). Data Availability The primary evaluation corpus is propri- etary. A 5-hour evaluation subset (SAS-Bench- 5h) with per-segment labels and deterministic scoring scripts may be made available upon rea- sonable request for research purposes. Contact: contact@attentionlabs.ai. Acknowledgements Hardware testing was conducted on Raspberry Pi 4 development boards and a Reachy Mini companion robot (Pollen Robotics). The authors thank the eval- uation participants who contributed to the unscripted multi-speaker recording sessions. No external funding is reported. References [1] E. C. Cherry, “Some experiments on the recog- nition of speech, with one and with two ears,” J. Acoust. Soc. Am., vol. 25, no. 5, p. 975–979, 1953. [2] K. Grauman et al., “Ego4D: Around the world in 3,000 hours of egocentric video,” in Proc. IEEE/CVF CVPR, p. 18995–19012, 2022. [3] H.-C. Lin, C.-Y. Wang, M.-H. Chen, S.-W. Fu, and Y.-C. F. Wang, “QuAVF: Quality-aware audio-visual fusion for Ego4D talking to me chal- lenge,” arXiv:2306.17404, CVPR 2023 Ego4D Workshop, 2023. [4] D. Kong, F. Khan, X. Zhang, P. Singhal, and Y. N. Wu, “Long-term social interaction context: The key to egocentric addressee detection,” in Proc. IEEE ICASSP, 2024. [5] M. Tran, Y. Kim, C.-C. Su, C.-H. Kuo, M. Sun, and M. Soleymani, “Ex2Eg-MAE: A framework for adaptation of exocentric video masked au- toencoders for egocentric social role understand- ing,” in Proc. ECCV, LNCS vol. 15138, Springer, 2024. [6] K. Lertniphonphan et al., “PCIE_Interaction so- lution for Ego4D social interaction challenge,” 15 arXiv:2505.24404, CVPR 2025 Ego4D Workshop, 2025. [7] Silero Team, “Silero VAD: pre-trained enterprise- grade voice activity detector,” GitHub repository, 2021. [8] S. C. Levinson, “Turn-taking in human commu- nication: Origins and implications for language processing,” Trends Cogn. Sci., vol. 20, no. 1, p. 6–14, 2016. [9] S. Oviatt, G. Levow, M. MacEachern, and R. Moreton, “Modeling global and focal hyper- articulation during human–computer error res- olution,” J. Acoust. Soc. Am., vol. 104, no. 5, p. 3080–3098, 1998. [10] M. Cohn, K.-H. Liang, M. Serič, and G. Zel- lou, “Prosodic differences in human- and Alexa- directed speech, but similar local intelligibility adjustments,” Front. Commun., vol. 6, 675704, 2021. [11] S. H. Mallidi, R. Maas, K. Goehner, A. Ras- trow, S. Matsoukas, and B. Hoffmeister, “Device- directed utterance detection,” in Proc. Inter- speech, p. 1225–1228, 2018. [12] D. Wagner, A. Churchill, S. Sigtia, P. Georgiou, M. Mirsamadi, A. Mishra, and E. Marchi, “A multimodal approach to device-directed speech detection with large language models,” in Proc. IEEE ICASSP, p. 10451–10455, 2024. [13] S. Palaskar, O. Rudovic, S. Dharur, F. Pesce, G. Krishna, A. Sivaraman, J. Berkowitz, A. H. Abdelaziz, S. Adya, and A. Tewfik, “Mul- timodal large language models with fusion low rank adaptation for device directed speech detec- tion,” in Proc. Interspeech, p. 4778–4782, 2024. [14] O. Rudovic, P. Dighe, Y. Su, V. Garg, S. Dharur, X. Niu, A. H. Abdelaziz, S. Adya, and A. Tew- fik, “Device-directed speech detection for follow- up conversations using large language models,” in NeurIPS 2024 Workshop on Adaptive Founda- tion Models, 2024. [15] D. Wagner, A. Churchill, S. Sigtia, and E. Marchi, “SELMA: A speech-enabled language model for virtual assistant interactions,” in Proc. IEEE ICASSP, 2025. [16] H. G. Chi, F. Pesce, W. Chang, O. Rudovic, A. Argueta, S. Braun, V. Garg, and A. H. Abde- laziz, “Adaptive knowledge distillation for device- directed speech detection,” in Proc. Interspeech, 2025. [17] G. Krishna, S. Dharur, O. Rudovic, P. Dighe, S. Adya, A. H. Abdelaziz, and A. H. Tew- fik, “Modality dropout for multimodal device di- rected speech detection using verbal and non- verbal features,” arXiv:2310.15261, 2023. [18] D. Wagner, A. Churchill, S. Sigtia, P. Georgiou, M. Mirsamadi, A. Mishra, and E. Marchi, “Multi- modal data and resource efficient device-directed speech detection with large foundation models,” in Third Workshop on Efficient NLP and Speech Processing (ENLSP-I) at NeurIPS, 2023. [19] E. Shriberg, A. Stolcke, D. Hakkani-Tür, and L. Heck, “Learning when to listen: Detect- ing system-addressed speech in human-human- computer dialog,” in Proc. Interspeech, p. 334– 337, 2012. [20] E. Shriberg, A. Stolcke, and D. Hakkani-Tür, “Addressee detection for dialog systems using temporal and spectral dimensions of speaking style,” in Proc. Interspeech, 2013. [21] T. J. Tsai, A. Stolcke, and M. Slaney, “A study of multimodal addressee detection in human- human-computer interaction,” IEEE Trans. Mul- timedia, vol. 17, no. 9, p. 1550–1561, 2015. [22] I. Siegert and O. Niebuhr, “Acoustic-based auto- matic addressee detection for technical systems: A review,” Front. Comput. Sci., vol. 4, 831784, 2022. [23] C.-W. Huang, R. Maas, S. H. Mallidi, and B. Hoffmeister, “A study for improving device- directed speech detection toward frictionless human-machine interaction,” in Proc. Inter- speech, p. 3342–3346, 2019. [24] O. Rudovic, A. Bindal, V. Garg, P. Simha, P. Dighe, and S. Kajarekar, “Streaming on-device detection of device directed speech from voice and touch-based invocation,” in Proc. IEEE ICASSP, p. 491–495, 2022. [25] O. Rudovic, W. Chang, V. Garg, P. Dighe, P. Simha, J. Berkowitz, A. H. Abdelaziz, S. Ka- jarekar, E. Marchi, and S. Adya, “Less is more: A unified architecture for device-directed speech de- tection with multiple invocation types,” in Proc. IEEE ICASSP, 2023. [26] S. Cornell, T. Balestri, and T. Sénéchal, “Implicit acoustic echo cancellation for keyword spotting and device-directed speech detection,” in Proc. IEEE SLT, 2022. [27] Picovoice, “Custom wake words with voice ID,” https://picovoice.ai/blog/ custom-wake-words-with-voice-id/,ac- cessed April 2026. 16