Paper deep dive

Multi-System Cognitive Steering for Generative Communication

Eric Lacosse, Pablo Tano, Daniel C. McNamee

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 46

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/15/2026, 5:14:23 AM

Summary

The paper introduces 'Cognitive Steering,' a framework for Augmentative and Alternative Communication (AAC) that uses Large Language Models (LLMs) to expand highly abbreviated text inputs into fluent sentences by grounding them in multimodal context. The authors demonstrate that providing context such as environmental referents, emotional states, and communication motives significantly improves expansion accuracy (up to 88% top-5) compared to no-context baselines. They further reveal that different context types require distinct AI control methods: physical objects are best handled via explicit text prompting, while abstract factors like emotion are more effectively managed through activation steering (altering internal model activations).

Entities (9)

Cognitive Steering · framework · 98%Augmentative and Alternative Communication · domain · 95%Large Language Models · technology · 95%Activation Steering · technique · 92%Champalimaud Research · organization · 90%Cognitive Misalignment · concept · 90%Image-Chat · dataset · 90%Gemini-3-pro · model · 85%OLMo-2-13B-Instruct · model · 85%

Relation Signals (6)

Cognitive Steering → improves → Abbreviation Expansion Accuracy

confidence 95% · aligning frontier models with selected environmental referents... achieves up to ~88% top-5 expansion accuracy

Cognitive Steering → uses → Large Language Models

confidence 95% · We propose Cognitive Steering, a framework that can be leveraged to reconstruct user intent by grounding Large Language Models (LLMs) in multimodal context

Cognitive Steering → applies → AI control methods

confidence 92% · applying AI control methods... different types of context require distinct AI control techniques

Activation Steering → isusedfor → Emotion

confidence 90% · abstract factors like emotion may be more effectively managed by directly altering the AI’s internal activations... activation steering

Explicit Text Prompts → isusedfor → Physical Objects

confidence 90% · physical objects in a scene are best provided through explicit text prompts

Cognitive Misalignment → iscausedby → Sparse Inputs

confidence 88% · sparse inputs are inherently ambiguous; without sufficient context, AI models frequently misinterpret these abbreviations

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:A fundamental bottleneck for translating thoughts to speech is the high physical cost of motor input forcing a strict tradeoff between communication speed and expressivity. Generative AI offers a promising solution by expanding highly abbreviated text input into fluent sentences, significantly reducing the motor burden on the user. However, sparse inputs are inherently ambiguous; without sufficient context, AI models frequently misinterpret these abbreviations and generate outputs that fail to align with the user's true cognitive intent. We propose \textit{Cognitive Steering}, a framework that can be leveraged to reconstruct user intent by grounding Large Language Models (LLMs) in multimodal context and applying AI control methods. We demonstrate that aligning frontier models with selected environmental referents, emotional states, and communication motives on simulated abbreviation benchmarks achieves up to $\sim$88\% top-5 expansion accuracy, doubling no-context baselines. We also reveal that different types of context require distinct AI control techniques: while physical objects in a scene are best provided through explicit text prompts, abstract factors like emotion may be more effectively managed by directly altering the AI's internal activations. These findings suggest a hybrid architecture for future interfaces that leverage Cognitive Steering to help users communicate both quickly and accurately, recruiting multiple AI control methods simultaneously to perfectly capture their intended communication goals.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

42,292 characters extracted from source content.

Expand or collapse full text

Multi-System Cognitive Steering for Generative Communication Conference: BiAlign Workshop 2026 CHI; April 15, 2026; Barcelona, SpainProceedings of the BiAlign Workshop 2026 CHI, April 15, 2026, Barcelona, Spain Eric Lacosse Note: shared contribution Affiliation: Champalimaud Centre for Restorative Neurotechnology, Champalimaud Research, Lisbon, Portugal email: eric.lacosse@research.fchampalimaud.org , Pablo Tano Affiliation: Champalimaud Research, Lisbon, Portugal email: pablo.tano@research.fchampalimaud.org and Daniel C. McNamee Affiliation: Champalimaud Research, Champalimaud Centre for Restorative Neurotechnology, Lisbon, Portugal email: daniel.mcnamee@ email: research.fchampalimaud.org 2026 Abstract. A fundamental bottleneck for translating thoughts to speech is the high physical cost of motor input forcing a strict tradeoff between communication speed and expressivity. Generative AI offers a promising solution by expanding highly abbreviated text input into fluent sentences, significantly reducing the motor burden on the user. However, sparse inputs are inherently ambiguous; without sufficient context, AI models frequently misinterpret these abbreviations and generate outputs that fail to align with the user’s true cognitive intent. We propose Cognitive Steering, a framework that can be leveraged to reconstruct user intent by grounding Large Language Models (LLMs) in multimodal context and applying AI control methods. We demonstrate that aligning frontier models with selected environmental referents, emotional states, and communication motives on simulated abbreviation benchmarks achieves up to ∼ 88% top-5 expansion accuracy, doubling no-context baselines. We also reveal that different types of context require distinct AI control techniques: while physical objects in a scene are best provided through explicit text prompts, abstract factors like emotion may be more effectively managed by directly altering the AI’s internal activations. These findings suggest a hybrid architecture for future interfaces that leverage Cognitive Steering to help users communicate both quickly and accurately, recruiting multiple AI control methods simultaneously to perfectly capture their intended communication goals. Keywords: Alignment, Steering, Augmentative and Alternative Communication, Large Language Models, Brain Computer Interfaces 1. Introduction Augmentative and Alternative Communication (AAC) systems for users with severe speech and motor impairment, such as those with Amyotrophic Lateral Sclerosis (ALS), suffer from a fundamental bottleneck: they require limited high-cost motor input to express high-bandwidth cognitive intent. Although humans can speak ∼ 150 words per minute (Anumanchipalli et al., 2019), eye-gaze typing interfaces restrict transmission to roughly ten words per minute (Majaranta and Räihä, 2002). Generative AI offers a transformative opportunity to bridge this gap by expanding low-bandwidth inputs into fluent natural language (Fig. 1A) (Hoag et al., 2004). Recent systems allow users to transmit highly compressed signals—extreme abbreviations like “h-w-t-g-o-a-p” which are expanded into full sentences (“he wanted to go out and play”) (Cai et al., 2024). We view the user’s abbreviated input not as incomplete data, but as a compressed signal—a “blurred pattern”—that the AI must use to reconstruct the user’s intent (Lupyan and Arcas, 2026). Large Language Models (LLMs), much like human cognition, possess a capacity to reconstruct user intent from such minimal inputs based on contextual cues and grounding (Nour et al., 2025). In human communication, this reconstructive capacity is largely governed by pragmatics, i.e., the ability to derive intended meaning beyond literal syntax and semantics by relying on shared context, environmental constraints, and conversational maxims (Grice, 1975; Sperber and Wilson, 1986). Figure 1. (A) The AAC Pareto Frontier. Traditional AAC devices force a tradeoff (Hoag et al., 2004) between expressivity (Communication Expressivity, y-axis) and efficiency (Communication Efficiency, x-axis). An example of Mechanism 1 is letter-by-letter typing, which captures nuance but is slow; and an example of Mechanism 2 is preset phrases, which are fast but potentially generic. We propose that Cognitive Alignment (CA)—achieved by grounding LLM generation on human intentionality—allows the frontier to be broken as illustrated. By successfully expanding compressed inputs (abbreviations) into the intended utterances, users will better achieve high expressivity without sacrificing efficiency. (B) Keyboard expansion and typing based on eye-gaze control. A combination of bi-gram and tri-gram allows the selection of letters via a keyboard expansion through directional gestures performed by eye movement. Additional high-frequency common words can be placed for easy access on the keyboard for positional gaze detection. (C) Contextual Steering. The CCI achieves CA by injecting multimodal context (visual ego-centric view, saliency, audio) and pragmatic elements (referents, motive of communication, emotion) into the LLM inference process. A test-time intervention mechanism (e.g. prompting or steering) uses these signals to steer the LLM, ensuring the abbreviation “A G U O T” is expanded into the contextually relevant utterances that steered closer to the intended phrase like (“A GOOD USE OF TIME") or (“ALWAYS GET UP ON TIME"). However, relying on LLMs introduces a critical Cognitive Alignment (CA) challenge: without sufficient context, models frequently misinterpret ambiguous user intent (Subramonyam et al., 2024). For example, a user entering an abbreviation like “h-w-t-g-o-a-p” may intend “he wanted to go out and play”, yet in a different context, the same compressed input could be expanded into the unintended but plausible alternative, e.g., “how we think governs our actual perception”. Hence, while abbreviation expansion significantly reduces motor load, it introduces a problem: Cognitive Misalignment (CM). We functionally define CM as the difference between the user’s intended message and the AI’s interpretation of the user’s intent. This may happen for many reasons, e.g., lack of theory of mind or misinterpreted user intent (Subramonyam et al., 2024), leading the AI to unexpected behavior. This misalignment between human and AI may be best addressed by treating the LLM not as a digital replica of human cognition, but as a distinct form of intelligence (Bratton, ; Hayles, 2025). While cognitive science has extensively mapped the pragmatic cues humans rely on to resolve ambiguity (Clark, 1996), we lack a comparable understanding of the requirements that enable LLMs to perform similar feats (Ying et al., 2025). There is no guarantee of an isomorphism between the signals that ground human understanding and those that appropriately influence the model’s behavior. An LLM might successfully resolve pragmatic ambiguities that are completely opaque to, or divorced from, human intuition (Sreedharan et al., 2024; Zahedi et al., 2022). Conversely, an LLM’s output might not reflect information that might seem obvious to a human observer/user. Therefore, rather than imposing an anthropocentric understanding onto these systems, we must empirically interrogate them with minimal assumptions (Sharkey et al., 2025). This challenge necessitates the analysis presented in this study. In order to build a capable and robust Cognitive Communication Interface (CCI) for humans and AIs (Fig. 1B), we must identify the signals required to achieve CA so the model’s behavior outputs what the user intended without costly interventions. Here, we analyze how different contextual information sources, particularly those from commercially available eye-tracking interfaces (Baumann and Dierkes, 2023), can be leveraged to reconstruct abbreviated user inputs across both synthetically generated and real-world datasets, by exploiting pragmatic elements of the context for CA (Fig. 1C). We demonstrate that achieving better CA may require distinct technical interventions at test-time inference. While discrete semantic grounding benefits from explicit prompting, high-level intent, like those shaped through latent factors like emotion, can be effectively managed through targeted model interventions of its internal activations at test-time, i.e., activation steering (Turner et al., 2024), which has been shown to shift the recruitment of latent model concepts (Bigelow et al., 2025). We believe these results will help inform the future engineering and design decisions needed for next-gen CCIs that are bidirectionally aligned between humans and AIs (Shen et al., 2025). 2. Methods. Our Generated Dataset. Evaluating whether LLMs can resolve ambiguities as a consequence of Cognitive Misalignment requires the use of benchmarks that systematically vary the amounts of pragmatically relevant contexts under which utterances are produced. Here, we focus on investigating the impact of four pragmatic factors (Fig. 1C): (1) Motive of Communication (MOC): why the user is speaking and what functional goal are they aiming to achieve? For example, is the user requesting, sharing, or informing (Tomasello, 2010). (2) Explicitness: how concretely has the user expressed themselves? The gradient of underdetermination theory (Recanati, 2005) defines three levels, namely elliptical (i.e., fragmentary, high-context productions such as single words or gapped constructions), underspecified (i.e., syntactically complete sentences using pronouns or deictic expressions such as “that one” or “over there”), and explicit (i.e., full noun phrases with disambiguating attributes). (3) Referents: what are the concrete entities (objects, people, locations) in the physical world that a speaker is talking about. (4) Emotions: what is the speaker’s latent affective state (e.g., frustration, sadness, neutrality), which acts as a global modulator on the utterance. Our data generation pipeline proceeds through four sequential stages designed to construct each evaluation datum with complete and separate control over these variables. First, a scenario configuration is sampled from a structured ontology of over 235,000 permutations of environmental, embodied, and social factors. These factors include real-world domain (e.g., medical, office), visual complexity (how cluttered a scene is), attentional states (e.g., joint, disengaged, face-to-face), ego posture (from where the first-person view is taken from), and interlocutor dynamics (the relation between user and interlocutor, epistemic and social). This scenario configuration constrains what a speaker could plausibly say and why with considerable diversity across the generated set. Second, referential grounding uses a prompted LLM to populate each generated scenario with a target object (the semantic anchor of the utterance) and a controlled set of distractor objects in the scene. Third, visual scene generation translates the seeded scenario into structured prompts for a text-to-image model (Gemini nano-banana), producing egocentric scene images that preserve the camera perspective, object composition, and atmospheric cues implied by the scenario factors. Finally, utterance generation independently and uniformly samples values for the three other pragmatic factors—MOC, Explicitness, and Emotion—then prompts an LLM to produce a naturalistic spoken utterance that is grounded in the scene and shaped by these sampled factors. The utterance is then reduced to an abbreviation (as explained in the next paragraph), yielding the compressed input signal that the LLM under evaluation must expand back into the intended sentence. Image-Chat dataset. To complement our generated benchmark with an external dataset acquired from human users, we adapt Image-Chat (Shuster et al., 2020), a large-scale dataset of grounded human dialogues about images. In Image-Chat, crowdworkers were each assigned a personality trait (e.g. adventurous, sentimental) and asked to converse about a photograph from the YFCC100m corpus; alternating between Speaker A and Speaker B. We analyze only the first two dialogue interactions. We selected 150 random conversations from the dataset, where both the opening (Speaker A) and response (Speaker B) utterances contain two to seven words, filtering out entries containing slang, multi-sentence turns, grammatical errors, and pejoratives. The original dataset does not include any description of the image, so we sent the corresponding YFCC100m image to a multimodal LLM (Gemini-3-Flash) to get a scene-description from a first-person point of view, and a list of all distinct visible objects, people, and notable items. Abbreviation encoding. We simulate AAC-style compressed input by reducing each word in a target utterance to its first letter. High-frequency common words (a, the, is, to, it, are, that, not, you, and) are given verbatim in square brackets (e.g. [the] b [is] r h). The AAC keyboard can include these common words in the interface (Fig. 1B). Context layers. To systematically test which context information improves abbreviation expansion, we define a hierarchy of context layers that incrementally enrich the prompt. For the generated dataset, the layers are: L0 (abbreviation only), L1 (+ communicative intent and motive of communication), L2 (+ speaker emotion and explicitness), L3 (+ target and distractor referents), and L4 (+ scene description). An additional ablation condition (L4∖L2L4 L2) provides referents and scene description but withholds speaker emotions. For the Image-Chat dataset, the layers are: L0 (abbreviation only; for Speaker B, the preceding turn is always included), L1 (+ speaker personality), L2 (+ interlocutor personality), L3 (+ objects visible in the image), and L4 (+ full scene description). We also included the condition L4∖L2L4 L2. Frontier model evaluation via structured prompting. For each context layer, we construct a structured prompt containing the abbreviation, a token-by-token constraint specification (indicating which positions are fixed words vs. first-letter hints), and the context features corresponding to that layer. The prompt instructs the model (Gemini-3-Pro) to return five ranked candidate expansions. For the Image-Chat dataset, separate system prompts are used for Speaker A (opening turn, no conversational history) and Speaker B (responding to the preceding turn, which is included in the prompt). In both datasets, we implement a modular template design for prompts that ensures that the only difference between context conditions is the presence or absence of specific context blocks, keeping all other prompt text identical. Activation steering To compare prompting-based alignment with test-time residual intervention, i.e. activation steering (Zou et al., 2025; Turner et al., 2024; Rimsky et al., 2024; Cao et al., ; Liu et al., 2025), we use OLMo-2-13B-Instruct (OLMo et al., 2025). For each evaluation case, we compute a steering vector by running two forward passes, one with the L0 baseline prompt and one with the context-enriched prompt, and taking the difference of the residual-stream activations at the last token position at the target layer: ℓ=ℓcontext[-1]−ℓbaseline[-1]v_ =h_ ^context[-1]-h_ ^baseline[-1]. We use the last token since it aggregates the full contextual representation immediately prior to generation. During generation, this vector is added to the baseline residual at the target layer(s) with a strength α: ℓ′=ℓ+αℓh_ =h_ +α\,v_ . We evaluate two steering configurations: steering into layers 15, 20, and 25 with strength α=0.5α=0.5; and steering into layers 30, 35, and 38 with strength α=1.0α=1.0. Because OLMo-2 is less reliable than the frontier model at unconstrained generation, all decoding is further guided by a per-case regex constraint (Willard and Louf, 2023) that enforces the correct word count, preserves bracketed words verbatim, and requires each unknown word to start with the specified letter. Evaluation metrics. We evaluate the degree to which an LLM’s reconstructed output recovers the user’s intended utterance from an abbreviation signal, using three complementary metrics, each computed at both Top-1 (single best candidate) and Top-5 (best among five candidates). The Top-5 condition simulates the selection-from-list interaction paradigm common in AAC interfaces, where users choose among system-generated candidates rather than accepting a single prediction (Cai et al., 2024). Lexical Accuracy performs a word-level match; BLEU measures the degree of structural and lexical overlap between the reconstruction and the ground truth, is computed with add-one smoothing over n-grams of order 1 through min⁡(4,|sentence|) (4, ), with a brevity penalty (Papineni et al., 2002). LLM Judge asks the OLMo-2-13B-Instruct instance whether the candidate and reference convey equivalent meaning, sentiment, and intent in the current context (full scene description), approximating a holistic pragmatic acceptability judgment. We consider LLM-Judged equivalency the most complete measure of accuracy, so we refer to it simply as ‘accuracy’ in the rest of the paper, unless otherwise specified. 3. Results. Figure 2. Frontier LLMs achieve up to ∼88% 88\% top-5 expansion accuracy when aligned to pragmatic context. (A) Image-Chat dataset. (Top) Illustration of the 4 layers. L0 includes no context information and simply asks the LLM to expand the abbreviated sentence. L1 includes the emotion of Speaker A (first dialogue line) and L2 further includes the emotion of Speaker B (second dialogue line). L3 includes referents and L4 the full scene description. (Bottom) Performance for Speaker A (first row) and Speaker B (second row) under different context layers (L0–L4), and in the emotion ablated condition L4∖L2L4 L2. The y-axis represents Accuracy, and bars denote Top-1 and Top-5 performance. (B) Generated dataset. (Top) L0 is as in panel A. L1 includes the Motive of Communication (MOC), and L2 includes the speaker emotion and explicitness level. L3 includes target and distractor referents, and L4 the scene description. (Bottom) Performance across context layers and in the emotion ablated condition L4∖L2L4 L2. The y-axis represents Accuracy, and bars denote Top-1 and Top-5 performance. Statistical significance was assessed using paired Wilcoxon signed-rank tests (one-sided) with Holm–Bonferroni correction for multiple comparisons. * p < .05, ** p < .01, *** p < .001.Panel A shows the performance on the Image-Chat dataset across different context layers (L0 to L4) and an ablated condition, with bar charts for Speaker A and Speaker B. Panel B shows performance on the generated dataset across the same context layers and ablation condition. Bar charts display accuracy metrics with error bars. Aligned frontier models expand abbreviations with up to 88% accuracy (Fig. 2) Frontier LLMs have an exceptional ability to reconstruct a sentence given a corrupted version (Lupyan and Arcas, 2026). We find that this capability extends to abbreviation expansion: given only the user’s abbreviation, frontier models can often infer the intended full phrase, even when the abbreviation occurs in the first or second turn of a dialogue. This inference is not purely lexical. Many abbreviations are compatible with multiple plausible expansions, and disambiguation requires pragmatic grounding (e.g., who/what is being referred to, the speaker’s affect, and the desired communicative intent). To test whether such grounding is necessary, we provide the model with controlled access to context elements (Fig. 1C) via structured prompting (Fig. 3A), and evaluate performance as these elements are included or ablated. Throughout, we use Gemini-3-Pro as our frontier model, and defer comparisons to alternative alignment mechanisms in open-source models to the next subsection. In the Image-Chat dataset (Fig. 2A), the first dialogue line (Speaker A; first row) is the most underdetermined setting: there is no conversational history, so the model must rely on extra-linguistic cues to resolve ambiguous shorthand. Exploiting the context, the LLM interpreter achieves 73% abbreviation-expansion performance (see L3, top-5 LLM-judged equivalency, from now on we refer to this performance metric for the analysis), doubling the performance of the no-context baseline (L0). A Friedman test confirmed a significant effect of context layer on decoding performance (χ2(4)=23.98χ^2(4)=23.98, p<.001p<.001). Only the step L2 to L3 showed significant improvement for top-5 performance, consistent with the hypothesis that abbreviation expansion in situ is often a reference-resolution problem. However, referents alone are not sufficient for peak performance: ablating the speaker’s emotions (L4∖L2L4 L2) produces a consistent drop (Δ=−.05 =-.05, W=72,385W=72,385, p<.001p<.001, r=.15r=.15; one-sided Wilcoxon signed-rank), indicating that affective state contributes information that the model uses to select between competing expansions (e.g., neutral vs. frustrated or urgent formulations). By contrast, the second dialogue line (Speaker B; second row) is partially constrained by the immediately preceding turn, and therefore admits higher performance even without explicit context features (compare L0 between Speaker A and B). Nevertheless, adding referents and emotions in the prompt still yields an additional ∼ 13% top-5 LLM-judged gain over the baseline, and this gain largely disappears when emotions are removed (L4∖L2L4 L2, Δ=−.09 =-.09, W=3,906W=3,906, p<.001p<.001, r=.33r=.33; one-sided Wilcoxon signed-rank), suggesting that affect provides complementary signal rather than redundant information. The same pattern holds in the generated dataset (Fig. 2B). When supplied with pragmatic context, the frontier LLM reconstructs the intended phrase from the abbreviation with 88%88\% top-5 accuracy, even in the absence of any prior conversational content. Since top-5 corresponds to a realistic AAC interaction loop in which the user selects from candidate expansions, these results imply that context-dependent alignment can materially improve usability: it increases the chance that the intended message appears among a small set of options, while preserving the efficiency benefits of abbreviated input. A Friedman test confirmed a significant effect of context layer on decoding performance (χ2(4)=126.67χ^2(4)=126.67, p<.001p<.001). Ablating the MOC, emotion and explicitness (L4∖L2L4 L2) significantly lowers performance over the full L4 (Δ=−.170 =-.170, W=240.0W=240.0, p<.001p<.001, r=.65r=.65). Figure 3. Cognitive Steering - Prompting vs. test-time steering for alternative context layers. (A) Structured prompting is used to provide pragmatic context features (e.g., MOC, emotions, explicitness, referents) to the model at inference time. (B) Steering procedure: a context-direction vector is added to the residual stream of a chosen layer with varying strength (Steering 1–2), enabling alignment without modifying the prompt. Steering 1 steers layers 15, 20, and 25; Steering 2 steers layers 30, 35, and 38 (OLMo-2-13B-Instruct has 40 layers (OLMo et al., 2025)). (C) Top-5 LLM-judged similarity for alternative steering approaches (bars) on the generated dataset benchmark, for L3 (MOC, emotions, explicitness) and L4 (+Referents) context layer. The y-axis represents Top-5 LLM-judged similarity. Statistical significance was assessed using paired Wilcoxon signed-rank tests (one-sided) with Holm–Bonferroni correction for multiple comparisons. * p < .05, ** p < .01, *** p < .001. The efficiency of alignment approaches depends on context type (Fig. 3) . We found that the efficiency of alternative approaches for Cognitive Alignment depends on which dimension of the context is being considered (e.g. emotions vs. referents). Latent steering (Turner et al., 2024; Vu and Nguyen, 2025; Nguyen et al., 2025; Liu et al., 2025; Banayeeanzade et al., 2025; Wu et al., 2025; Zou et al., 2023) is a promising approach for alignment, an alternative to prompting, by which a vector is added to the residual activity of a target layer(s) at test-time, in order to influence the model’s behavior. Evaluating steering in proprietary models is not possible as it requires accessing intermediate activations during inference, so for this analysis we use the open-source OLMo-2-13B-Instruct model (OLMo et al., 2025), evaluated on the generated dataset. This model achieves significantly lower performance than the frontier model (compare with Fig. 2B), but it allows us to compare test-time intervention approaches for different context types. We compare prompting (Fig. 3A) with steering on early or later layers (Steering 1 and 2 in Fig. 3B). We found that aligning the L3 context layer in Fig. 2B benefits from different approaches than aligning to L4 in Fig. 2B (Fig. 3C). Alignment to MOC, Emotions and Explicitness (L3) benefits from steering early layers, which performs similarly to prompting. In contrast, alignment to referents (L4) is best achieved by prompting, or by steering a late layer. Intuitively, MOC, affect, and explicitness largely shape how an utterance should be produced. Steering early layers can act like a subtle prior: it nudges the model into the appropriate communicative mode without overriding the semantic or lexical content. In contrast, resolving referents is primarily about what the utterance is about: selecting the correct entities and filling in missing arguments from the context. Directly providing referents in the prompt is highly effective because it makes the relevant entities explicit in the model’s immediate context, similarly to decisively shifting token-level logits toward the intended entity using late-layer steering. 4. Discussion. We presented a novel multi-system Cognitive Steering framework for generative communication that relies on test-time intervention methods. We investigated structured context ablations for its use in LLMs. By treating user input as a compressed signal to be reconstructed, we demonstrate that LLMs have the potential to enable high-bandwidth communication from extremely limited motor inputs (e.g., selecting abbreviations). However, this capability is contingent upon solving the problem of Cognitive Misalignment whereby the LLM lacks the necessary context to resolve user intention. Our results suggest that solving misalignment requires grounding the LLM in highly specific pragmatic contexts, e.g., visual attention, emotional state, and communicative intent, that human listeners naturally utilize. With appropriate context, like that available from commercially wearable eye-trackers—those with world-cameras and microphones to catch and diarize conversations—we demonstrate that frontier LLMs can reconstruct intended utterances from highly compressed abbreviations with up to 88% top-5 accuracy (Fig. 2). In particular, we found that (visual) referents–information readily available via the same eye-tracking hardware used for typing–are the most significant driver of disambiguation (Fig. 2). This suggests a promising direction for future work where eye-gaze interfaces do not merely act as cursors for input, but as semantic sensors and selectors, actively logging attended objects to engineer context (Bulling et al., 2010). While visual context (referents and scene descriptions) provided the most significant boost in lexical accuracy (Fig. 2), the small but consistent performance drop observed in the emotion-ablated condition (L4∖L2L4 L2) suggests that semantic grounding (“what acts upon what”) can be paired with pragmatic grounding (“how and why the speaker feels”) to further narrow the search space of plausible expansions. This aligns with theories of human pragmatics suggesting that intent recognition is often distinct from reference resolution (Tomasello, 2010). For the user, this suggests that sensing the user’s latent mental state may further improve expansion accuracy. Our comparison of structured prompting versus activation steering (Fig. 3) suggests that different dimensions of communication map to different intervention points in the model architecture. Specific objects in a scene benefit from explicit prompting or late-layer steering. Conversely, abstract features like MOC and Emotion were effectively managed via early-layer steering. While observed in a specific open-source model, this supports the view that early layers process high-level style and global constraints, while later layers resolve specific token identities, though this needs verification in more capable frontier models before generalizing to all pretrained transformers. This dissociation suggests a path toward hybrid CCIs: using discrete prompts to ground the model in the physical world (via object detection) while using steering vectors to modulate the “voice” and intent of the output. Furthermore, treating in-context learning (prompting) as evidence accumulation and steering as a shift in prior beliefs provides a unified Bayesian framework for precisely predicting how these combined interventions will influence the model’s final generated output (Bigelow et al., 2025). Such an architecture is reflected in dual systems theory of human brain function whereby the fast, implicit System 1 and flexible, explicit System 2 divide the algorithmic labor of controlling thought and behavior (Dolan and Dayan, 2013). A deeper understanding of this analogy may open further opportunities for bidirectional alignment between AIs and humans. 5. Limitations and Future Work. (1) Due to the closed nature of frontier models (Gemini-3-Pro), our activation steering experiments were restricted to open-source models (OLMo-2-13B-Instruct (OLMo et al., 2025)), which show significantly lower baseline performance. (2) While we investigated controlled ablation of context factors, our evaluation relies on synthetic data and conversations from crowdworkers, which may not accurately reflect the linguistic patterns, errors, or true cognitive intent of individuals, especially those with severe motor and speech impairments using AAC devices. We need to deploy a real system under realistic conditions and to investigate whether simulated model performance reflects this. (3) Our simple first-letter abbreviation method assumes perfect spelling and consistent encoding, which may not hold true for real-world AAC users who often employ idiosyncratic abbreviations, make typographical errors, or may introduce a significant cognitive load. (4) While frontier model results appear very promising, achieving the latency required for fluent conversation remains an engineering challenge given a gesture-based interface, e.g., Fig. 1B. (5) While our results show a benefit to including emotional context, reliably capturing a user’s latent mental state in real-time AAC systems remains a significant technical and practical challenge. (6) We did not include n-gram baselines in our comparisons, which could further contextualize the performance gains of our LLM-based approach. (7) Finally, the introduction of AI that infers intent raises ethical questions regarding agency (Jordan, 2025; Institute, 2024). Preserving this agency is paramount: future systems must be designed to speak for the user, never over them (Thacker, 2020). 6. Conclusion. Generative AI transforms AAC into a cognitive bidirectional alignment task (Shen et al., 2025), involving a tradeoff between communication efficiency and expressivity exists. We conclude that by coupling low-bandwidth input with rich specific context and through applying the appropriate AI control (e.g., steering mechanisms), we can construct interfaces deeply attuned to the cognitive reality of the user thus overcoming Cognitive Misalignment and transform minimal input into maximum expression. 7. Acknowledgments We would especially like to thank Elizaveta Gurina for her feedback on this manuscript. E.L., P.T., D.M. thank the Champalimaud Foundation and the Portuguese Recovery and Resilience Plan (project number 62) for funding. Google Research Credits 423033108. References Anumanchipalli et al. (2019) G. K. Anumanchipalli, J. Chartier, and E. F. Chang Speech synthesis from neural decoding of spoken sentences. Nature 568 (7753), p. 493–498. External Links: ISSN 1476-4687, Document Cited by: §1. Banayeeanzade et al. (2025) A. Banayeeanzade, A. N. Tak, F. Bahrani, A. Bolourani, L. Blas, E. Ferrara, J. Gratch, and S. P. Karimireddy Psychological Steering in LLMs: An Evaluation of Effectiveness and Trustworthiness. arXiv. External Links: 2510.04484, Document Cited by: §3. Baumann and Dierkes (2023) C. Baumann and K. Dierkes Neon Accuracy Test Report. Technical report Pupil Labs. External Links: Document Cited by: §1. Bigelow et al. (2025) E. Bigelow, D. Wurgaft, Y. Wang, N. Goodman, T. Ullman, H. Tanaka, and E. S. Lubana Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering. arXiv. External Links: 2511.00617, Document Cited by: §1, §4. [5] B. Bratton After Alignment | Antikythera. Note: https://afteralignment.antikythera.org/ Cited by: §1. Bulling et al. (2010) A. Bulling, J. A. Ward, H. Gellersen, and G. Tröster Eye movement analysis for activity recognition using electrooculography. IEEE transactions on pattern analysis and machine intelligence 33 (4), p. 741–753. Cited by: §4. Cai et al. (2024) S. Cai, S. Venugopalan, K. Seaver, X. Xiao, K. Tomanek, S. Jalasutram, M. R. Morris, S. Kane, A. Narayanan, R. L. MacDonald, E. Kornman, D. Vance, B. Casey, S. M. Gleason, P. Q. Nelson, and M. P. Brenner Using large language models to accelerate communication for eye gaze typing users with ALS. Nature Communications 15 (1), p. 9449 (en). External Links: ISSN 2041-1723, Link, Document Cited by: §1, §2. [8] Y. Cao, T. Zhang, B. Cao, Z. Yin, L. Lin, F. Ma, and J. Chen Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization. Cited by: §2. Clark (1996) H. H. Clark Using Language. Cambridge University Press (en). External Links: ISBN 978-0-521-56745-9 Cited by: §1. Dolan and Dayan (2013) R. J. Dolan and P. Dayan Goals and Habits in the Brain. Neuron 80 (2), p. 312–325. External Links: ISSN 0896-6273, Link, Document Cited by: §4. Grice (1975) H. P. Grice Logic and conversation. In Speech acts, p. 41–58. Cited by: §1. Hayles (2025) N. K. Hayles Bacteria to AI: Human Futures with Our Nonhuman Symbionts. University of Chicago Press (en). Note: Google-Books-ID: hZEVEQAAQBAJ External Links: ISBN 978-0-226-83746-8 Cited by: §1. Hoag et al. (2004) L. A. Hoag, J. L. Bedrosian, K. F. McCoy, and D. E. Johnson Trade-offs between informativeness and speed of message delivery in augmentative and alternative communication. Journal of speech, language, and hearing research: JSLHR 47 (6), p. 1270–1285. External Links: ISSN 1092-4388, Document Cited by: Figure 1, §1. Institute (2024) C. Institute On the Noble Uses of AI. Note: https://blog.cosmos-institute.org/p/on-the-noble-uses-of-ai?utm_medium=ios&triedRedirect=true Cited by: §5. Jordan (2025) M. I. Jordan A Collectivist, Economic Perspective on AI. arXiv. Note: arXiv:2507.06268 [cs] External Links: Link, Document Cited by: §5. Liu et al. (2025) Z. Liu, Z. Xu, G. Dou, X. Yuan, Z. Tan, R. Poovendran, and M. Jiang Steering Multimodal Large Language Models Decoding for Context-Aware Safety. arXiv. External Links: 2509.19212, Document Cited by: §2, §3. Lupyan and Arcas (2026) G. Lupyan and B. A. y. Arcas The unreasonable effectiveness of pattern matching. arXiv. Note: arXiv:2601.11432 [cs] External Links: Link, Document Cited by: §1, §3. Majaranta and Räihä (2002) P. Majaranta and K. Räihä Twenty years of eye typing: systems and design issues. In Proceedings of the 2002 Symposium on Eye Tracking Research & Applications, ETRA ’02, New York, NY, USA, p. 15–22. External Links: Document, ISBN 978-1-58113-467-4 Cited by: §1. Nguyen et al. (2025) D. V. Nguyen, H. M. Vu, N. Y. Pham, L. Zhang, and T. M. Nguyen Activation Steering with a Feedback Controller. arXiv. Note: arXiv:2510.04309 [cs] External Links: Link, Document Cited by: §3. Nour et al. (2025) M. M. Nour, D. C. McNamee, I. Fradkin, and R. J. Dolan Charting trajectories of human thought using large language models. arXiv. External Links: 2509.14455, Document Cited by: §1. OLMo et al. (2025) T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, N. Lambert, D. Schwenk, O. Tafjord, T. Anderson, D. Atkinson, F. Brahman, C. Clark, P. Dasigi, N. Dziri, A. Ettinger, M. Guerquin, D. Heineman, H. Ivison, P. W. Koh, J. Liu, S. Malik, W. Merrill, L. J. V. Miranda, J. Morrison, T. Murray, C. Nam, J. Poznanski, V. Pyatkin, A. Rangapur, M. Schmitz, S. Skjonsberg, D. Wadden, C. Wilhelm, M. Wilson, L. Zettlemoyer, A. Farhadi, N. A. Smith, and H. Hajishirzi 2 OLMo 2 Furious. arXiv. External Links: 2501.00656, Document Cited by: §2, Figure 3, §3, §5. Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W. Zhu Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, P. Isabelle, E. Charniak, and D. Lin (Eds.), Philadelphia, Pennsylvania, USA, p. 311–318. External Links: Link, Document Cited by: §2. Recanati (2005) F. Recanati Deixis and Anaphora. In Semantics versus Pragmatics, Z. G. Szabo (Ed.), p. 0. External Links: ISBN 978-0-19-925152-0, Link, Document Cited by: item 2. Rimsky et al. (2024) N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner Steering Llama 2 via Contrastive Activation Addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 15504–15522. External Links: Document Cited by: §2. Sharkey et al. (2025) L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimersheim, A. Ortega, J. Bloom, S. Biderman, A. Garriga-Alonso, A. Conmy, N. Nanda, J. Rumbelow, M. Wattenberg, N. Schoots, J. Miller, E. J. Michaud, S. Casper, M. Tegmark, W. Saunders, D. Bau, E. Todd, A. Geiger, M. Geva, J. Hoogland, D. Murfet, and T. McGrath Open Problems in Mechanistic Interpretability. arXiv. External Links: 2501.16496, Document Cited by: §1. Shen et al. (2025) H. Shen, T. Knearem, R. Ghosh, K. Alkiek, K. Krishna, Y. Liu, Z. Ma, S. Petridis, Y. Peng, L. Qiwei, S. Rakshit, C. Si, Y. Xie, J. P. Bigham, F. Bentley, J. Chai, Z. Lipton, Q. Mei, R. Mihalcea, M. Terry, D. Yang, M. R. Morris, P. Resnick, and D. Jurgens Position: Towards Bidirectional Human-AI Alignment. arXiv. Note: arXiv:2406.09264 [cs] External Links: Link, Document Cited by: §1, §6. Shuster et al. (2020) K. Shuster, S. Humeau, A. Bordes, and J. Weston Image-Chat: Engaging Grounded Conversations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, p. 2414–2429. External Links: Document Cited by: §2. Sperber and Wilson (1986) D. Sperber and D. Wilson Relevance: communication and cognition. Vol. 142, Harvard University Press Cambridge, MA. Cited by: §1. Sreedharan et al. (2024) S. Sreedharan, A. Kulkarni, and S. Kambhampati Explainable Human-AI Interaction: A Planning Perspective. arXiv. Note: arXiv:2405.15804 [cs] External Links: Link, Document Cited by: §1. Subramonyam et al. (2024) H. Subramonyam, R. Pea, C. L. Pondoc, M. Agrawala, and C. Seifert Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces. arXiv. Note: arXiv:2309.14459 [cs] External Links: Link, Document Cited by: §1. Thacker (2020) J. Thacker The age of ai: artificial intelligence and the future of humanity. HarperChristian+ ORM. Cited by: §5. Tomasello (2010) M. Tomasello Origins of human communication. MIT press. Cited by: item 1, §4. Turner et al. (2024) A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid Steering Language Models With Activation Engineering. arXiv. External Links: 2308.10248, Document Cited by: §1, §2, §3. Vu and Nguyen (2025) H. M. Vu and T. M. Nguyen Angular Steering: Behavior Control via Rotation in Activation Space. arXiv. External Links: 2510.26243, Document Cited by: §3. Willard and Louf (2023) B. T. Willard and R. Louf Efficient Guided Generation for Large Language Models. arXiv. External Links: 2307.09702, Document Cited by: §2. Wu et al. (2025) Z. Wu, Q. Yu, A. Arora, C. D. Manning, and C. Potts Improved Representation Steering for Language Models. arXiv. External Links: 2505.20809, Document Cited by: §3. Ying et al. (2025) L. Ying, K. M. Collins, L. Wong, I. Sucholutsky, R. Liu, A. Weller, T. Shu, T. L. Griffiths, and J. B. Tenenbaum On Benchmarking Human-Like Intelligence in Machines. arXiv. Note: arXiv:2502.20502 [cs] External Links: Link, Document Cited by: §1. Zahedi et al. (2022) Z. Zahedi, S. Sreedharan, and S. Kambhampati A Mental-Model Centric Landscape of Human-AI Symbiosis. arXiv. Note: arXiv:2202.09447 [cs] External Links: Link, Document Cited by: §1. Zou et al. (2023) A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks Representation Engineering: A Top-Down Approach to AI Transparency. arXiv. Note: arXiv:2310.01405 External Links: Link Cited by: §3. Zou et al. (2025) A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks Representation Engineering: A Top-Down Approach to AI Transparency. arXiv. Note: arXiv:2310.01405 [cs] External Links: Link, Document Cited by: §2.

Instant research discovery