Paper deep dive
Scalable Semantic Steering of Embedding Projections
Wei Liu, Eric Krokos, Kirsten Whitley, Rebecca Faust, Chris North
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/7/2026, 10:44:38 AM
Summary
This paper introduces a scalable semantic steering method for embedding projections that shifts semantic computation from per-item to group-level processing. By generating hybrid semantic prototypes from a single LLM call per group, the method propagates analyst intent through embedding-space operations without retraining. Evaluated on the LitCovid corpus and an image dataset, it achieves alignment quality comparable to per-item LLM steering while reducing LLM calls and computational cost by over three orders of magnitude.
Entities (10)
Relation Signals (7)
Scalable Semantic Steering Method → evaluatedon → LitCovid Corpus
confidence 95% · On a 5K-document LitCovid corpus, our method achieves global alignment comparable to per-item LLM steering while reducing LLM calls by over three orders of magnitude.
Scalable Semantic Steering Method → reduces → Computational Cost
confidence 95% · reducing LLM calls by over three orders of magnitude.
Scalable Semantic Steering Method → achieves → Global Alignment
confidence 90% · achieves global alignment comparable to per-item LLM steering
Scalable Semantic Steering Method → uses → Hybrid Semantic Prototype
confidence 90% · A single LLM call generates structured profiles for all groups, which are embedded and combined with seed centroids to form hybrid semantic prototypes.
Hybrid Semantic Prototype → combines → Seed Centroids
confidence 88% · combined with seed centroids to form hybrid semantic prototypes.
Scalable Semantic Steering Method → extendsto → Stanford-40 Actions
confidence 85% · An image case study shows that the same prototype-based mechanism extends to multimodal embeddings.
Scalable Semantic Steering Method → utilizes → Soft Assignment
confidence 85% · using embedding-space soft assignment, abstention, and alignment-scaled updates before reprojection.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Low-dimensional projections support interactive visual analysis of high-dimensional data embeddings, but their structure often does not align with analyst-defined semantic relationships. Recent LLM-augmented semantic steering methods address this gap by externalizing analyst intent from user-defined groups of seed examples, but they propagate intent through per-item LLM reasoning, causing LLM calls and cost to grow linearly with collection size. We propose a scalable semantic steering method that shifts semantic computation from individual items to user-defined groups. A single LLM call generates structured profiles for all groups, which are embedded and combined with seed centroids to form hybrid semantic prototypes. The method then propagates intent without retraining, using embedding-space soft assignment, abstention, and alignment-scaled updates before reprojection. On a 5K-document LitCovid corpus, our method achieves global alignment comparable to per-item LLM steering while reducing LLM calls by over three orders of magnitude. An image case study shows that the same prototype-based mechanism extends to multimodal embeddings. These results suggest that group-level representations can make semantic steering more practical for larger embedding collections.
Tags
Links
- Source: https://arxiv.org/abs/2607.03978v1
- Canonical: https://arxiv.org/abs/2607.03978v1
Trouble viewing inline? Open PDF directly →
Full Text
35,547 characters extracted from source content.
Expand or collapse full text
Scalable Semantic Steering of Embedding Projections Wei Liu * Virginia Tech Eric Krokos † Department of Defense Kirsten Whitley ‡ Department of Defense Rebecca Faust § Tulane University Chris North ¶ Virginia Tech Figure 1: Prototype-based semantic steering on a text (LitCovid) dataset and an image (Human Actions) dataset. In each pair, the baseline projection (left) is reorganized into a more semantically coherent layout (right) using five user-selected seed examples per group (circled). Our method propagates analyst intent across the full collection through embedding-space operations driven by a single LLM call. ABSTRACT Low-dimensional projections support interactive visual analysis of high-dimensional data embeddings, but their structure often does not align with analyst-defined semantic relationships. Recent LLM- augmented semantic steering methods address this gap by exter- nalizing analyst intent from user-defined groups of seed examples, but they propagate intent through per-item LLM reasoning, caus- ing LLM calls and cost to grow linearly with collection size. We propose a scalable semantic steering method that shifts semantic computation from individual items to user-defined groups. A single LLM call generates structured profiles for all groups, which are em- bedded and combined with seed centroids to form hybrid semantic prototypes. The method then propagates intent without retraining, using embedding-space soft assignment, abstention, and alignment- scaled updates before reprojection. On a 5K-document LitCovid corpus, our method achieves global alignment comparable to per- item LLM steering while reducing LLM calls by over three orders of magnitude. An image case study shows that the same prototype- based mechanism extends to multimodal embeddings. These re- sults suggest that group-level representations can make semantic steering more practical for larger embedding collections. Index Terms: Semantic Steering, Semantic Interaction, Embed- ding Projections, Large Language Models, Semantic Prototypes. 1 INTRODUCTION Low-dimensional projections of high-dimensional embeddings are widely used to support visual analysis of text and image collections, enabling analysts to explore semantic structure through spatial or- ganization [10, 13]. However, projection structure is largely de- termined by the underlying embedding model and dimensionality * e-mail: wliu3@vt.edu, ORCID: 0009-0009-6340-8912 † e-mail: ericpkrokos@gmail.com, ORCID: 0000-0003-1350-5297 ‡ e-mail: visual.tycho@gmail.com, ORCID: 0000-0003-1356-326X § e-mail: rfaust1@tulane.edu, ORCID: 0000-0002-7640-1287 ¶ e-mail: north@vt.edu, ORCID: 0000-0002-8786-7103 reduction technique, which may not align with the semantic rela- tionships that analysts intend to examine for a given task [11, 1]. Semantic interaction (SI) [6] addresses this gap by allowing ana- lysts to reshape projections through spatial actions such as grouping or repositioning items. More recent work augments SI with large language models (LLMs) to externalize analyst intent as natural- language semantics rather than as implicit model parameters, prop- agating it across the dataset through item-level reasoning [12, 16]. While effective, this design has a fundamental scalability bottle- neck: semantic propagation is performed per item. Given a small set of seed examples, an LLM is invoked for every remaining item to evaluate its relationship to the analyst’s intent and generate item- level augmentations. The cost grows linearly with collection size. On a 5,000-item corpus, a single steering interaction issues over 5,000 LLM calls and costs roughly $32 under current pricing [17] (Section 4.3). As datasets grow, this design becomes increasingly impractical for interactive use. We make a structural observation: in SI, analysts often express intent at the group level, not at the item level. Users specify a few example groupings, and the relationships of interest live between these groups. This observation is reinforced by two parallel de- velopments outside the visual analytics community. In few-shot learning, a class can be represented by a prototype computed as the centroid of a small support set [25]. Recent vision–language work shows that textual class descriptions can serve as semantic pro- totypes [21, 20], and that combining text-derived prototypes with support-set centroids further strengthens them [18, 8]. Translated to SI, these insights suggest that semantic intent can be externalized once per group as a hybrid prototype: the seed centroid for data- grounded local geometry, and the LLM-generated profile embed- ding for language-articulated abstraction that generalizes beyond seeds. Intent can then be propagated entirely through embedding- space operations, removing the need for per-item LLM reasoning. Building on this insight, we propose a scalable semantic steer- ing method based on group-level semantic prototypes. From user- defined groups, a single LLM call generates structured semantic profiles for all groups jointly. Each profile is embedded and com- bined with its seed centroid into a hybrid prototype. Items are softly assigned to prototypes via embedding similarity and updated through alignment-scaled blending before reprojection. The entire pipeline issues one LLM call regardless of dataset size; the method arXiv:2607.03978v1 [cs.HC] 4 Jul 2026 operates purely in embedding space and extends naturally to multi- modal data through encoders such as CLIP [21] without retraining. This work makes the following contributions: (1) A scalable method for LLM-augmented semantic steering that shifts semantic computation from per-item reasoning to group- level abstraction, reducing LLM cost from O(N) to O(1). (2) A hybrid prototype design fusing seed centroids with LLM- generated group-profile embeddings, propagated through adaptive soft assignment and alignment-scaled embedding updates. (3) Evidence on a 5K text corpus that the method achieves simi- lar alignment quality to per-item LLM steering at over 1000× lower cost, with a case study showing the method extends to images. 2 RELATED WORK Semantic Interaction and Projection Steering. SI lets analysts shape projection layouts through spatial interactions such as group- ing or repositioning [6, 23, 7]. Such signals are typically translated into geometric constraints, feature reweighting, or learned distance metrics [2, 24, 5], and more recent frameworks extend SI to deep neural encoders by fine-tuning embeddings [1, 11]. While effective, intent is encoded implicitly through model parameters, requiring per-interaction optimization that scales poorly to larger collections. LLM-Based Steering with Explicit Semantics. Recent work uses LLMs to externalize analyst intent as natural-language semantics. Oliveira et al. [16] steer projections via zero-shot classification against user-specified categories, requiring the full label set in ad- vance. Liu et al. [12] remove this requirement by deriving “cluster cards” from grouped examples, but issue a per-item LLM call for selective augmentation. ModalChorus [29] aligns multimodal em- beddings via spatial drag operations and backend fine-tuning. A complementary line of work uses LLMs to describe existing clus- ters, leaving the layout unchanged [3]. These steering methods re- quire per-item LLM reasoning or iterative model fine-tuning, creat- ing a fundamental scalability bottleneck for interactive use. Prototype-Based Representations. Two lines of work inform our design. Prototypical Networks [25] represent each class as the mean embedding of its support examples for few-shot classifica- tion. In vision–language and text models, class names or descrip- tions can themselves act as prototypes in a shared embedding space [21, 9, 20], and recent few-shot methods strengthen prototypes by incorporating textual information alongside support-set centroids [18, 19, 8]. These advances target classification; we adapt the pro- totype formulation to semantic steering, fusing each user-defined group’s seed examples and description into a hybrid prototype. 3 METHOD We propose a scalable semantic steering method that replaces per- item LLM reasoning with group-level semantic prototypes. Given a small number of analyst-defined groups, our approach externalizes semantic intent in a single LLM call and propagates it across the entire collection through embedding-space operations. The pipeline is modality-agnostic and applies to both text and image collections. 3.1 Overview and Problem Setup Let D =d i N i=1 denote a collection of items (documents or im- ages) with embeddingsz i ∈R d produced by a pretrained encoder. A baseline projection is obtained by applying a dimensionality re- duction method (e.g., UMAP [14]) to z i . During an interaction, the analyst selects a small number of items and organizes them into K groups G k K k=1 , each expressing a semantic relationship of in- terest. Our goal is to reshape the projection to reflect the analyst’s intent through group-level semantic abstraction, with LLM usage independent of N. All embeddings are L 2 -normalized; normalize(·) denotes renormalization after vector combinations. Figure 2 illus- trates the pipeline: we generate group profiles (§3.2), construct hybrid prototypes (§3.3), propagate intent through soft assignment Figure 2: Scalable semantic steering pipeline. From user-defined groups (A), a single LLM call generates structured profiles for all groups jointly (B). Profiles are embedded and combined with seed centroids to form hybrid prototypes; items are softly assigned to pro- totypes via embedding similarity, with low-similarity items left unas- signed (C). Embeddings are updated with alignment-scaled blending and reprojected (D). Only stage B involves an LLM call, making the total LLM cost independent of dataset size. with abstention (§3.4), and update embeddings before reprojection (§3.5). 3.2 Group Profile Generation We issue one LLM call that returns profiles for all K groups—each profile contains a name, a short description, and inclusion and ex- clusion criteria. This adopts the cluster card formulation from prior LLM-augmented steering work [12], but it is the only LLM call our method requires. Each profile is then encoded with the same en- coder used for items, yielding a profile embedding p profile k ∈R d per group. For image collections, we use a multimodal encoder (e.g., CLIP [21]) to embed text profiles into the shared image-text space, enabling cross-modal similarity in subsequent stages. 3.3 Hybrid Semantic Prototypes Each group is represented by a hybrid prototype that com- bines two complementary signals.The seed centroid p seed k = normalize 1 |G k | ∑ i∈G k z i grounds the prototype in concrete exam- ples provided by the analyst, preserving local geometry of the orig- inal embedding space. The profile embedding p profile k injects high- level semantic abstraction articulated in natural language, which generalizes beyond the specific seed instances. The hybrid proto- type is their convex combination: p k = normalize γ p seed k +(1− γ) p profile k ,(1) where γ ∈ [0, 1] controls the trade-off. Setting γ = 1 recovers seed- only steering; γ = 0 uses only the LLM-articulated semantics. We use γ = 0.5 by default and study its effect in Section 4. 3.4 Soft Assignment with Adaptive Threshold We propagate semantic intent to the full collection by computing similarities between items and prototypes, s ik = cos(z i , p k ). Instead of a global threshold, we compute a per-group adaptive threshold that calibrates each prototype against the analyst’s seeds. Let D + k = G k be the seeds of group k and D − k = S j̸=k G j the seeds of all other groups. We define b k = 1 2 ̄s + k + ̄s − k , where ̄s + k and ̄s − k are the mean similarities of in-group and out-of-group seeds to p k , respectively. This threshold automatically locates a midpoint between what the analyst considers “in” and “out” for each group. Soft weights are then computed as w ik = σ(τ(s ik − b k )),(2) where σ(·) denotes the logistic sigmoid function and τ controls sharpness (τ = 20 by default). To preserve selectivity, we apply two filters. First, an adap- tive top-k mask K i = k : w ik ≥ ρ· max j w i j retains only groups whose weight is close to the maximum (with ρ = 0.85). This al- lows borderline items to remain associated with multiple prototypes while assigning most items primarily to a single group. Second, an abstain gate leaves items unchanged when max k w ik < t none , mir- roring the abstention mechanism in per-item LLM steering [12] but using embedding similarity rather than language reasoning. 3.5 Alignment-Scaled Update and Reprojection For each non-abstained item, we compute a steering target as the weight-normalized combination of its assigned prototypes: m i = normalize ∑ k∈K i w ik p k ∑ k∈K i w ik .(3) The updated embedding is an alignment-scaled interpolation be- tween the original embedding and the steering target: z ′ i = normalize (1− λ i ) z i + λ i m i ,λ i = α· max k w ik ,(4) where α∈ [0, 1] is a global steering strength and max k w ik scales the blend by per-item prototype alignment. Unlike prior work that applies a fixed blending coefficient uniformly across all steered items [12], our update strength is per-item: items with strong align- ment to a prototype are pulled more strongly toward the correspond- ing target, while ambiguous items move only slightly. Abstained items use λ i = 0. The updated embeddings z ′ i are reprojected with the same dimensionality reduction method as the baseline. 4 EXPERIMENTS We evaluate whether group-level semantic prototypes can match the steering quality of per-item LLM reasoning at substantially lower cost. 4.1 Setup Dataset. We use a 5,000-article subset of the LitCovid cor- pus [4], evenly distributed across four categories: Case Report, Di- agnosis, Prevention, and Treatment (1,250 articles each). Embeddings and projection. Text documents are embedded using text-embedding-3-small [17] and projected to 2D with UMAP [14] (cosine distance, n neighbors =30, min dist=0.1). Simulated interaction. To enable controlled and reproducible evaluation, we simulate analyst interaction by sampling 5 seed items per ground-truth category and treating them as user-defined groups. Reference labels are used only for evaluation and are never exposed to the LLM or the steering pipeline. All results are aver- aged over 3 random seed selections and reported as mean± std. Metrics.We report three groups of metrics relative to the unsteered baseline. (1) Alignment: ∆Sil measures global sepa- ration using a scaled silhouette score (Sil = 2s, where s is the standard silhouette [22]; Sil ≈ 1 is treated as ideal, indicating well-separated but not overly compact groups [11]). ∆NC mea- sures local coherence as the average fraction of same-group items among each point’s k = 10 nearest neighbors in the 2D projection [12]. (2) Assignment behavior: top-1 accuracy from arg max k w ik ; top-2 set-hit accuracy (whether the true group appears among the two highest-weighted prototypes); abstain rate (fraction with max k w ik < t none ). Accuracy is reported as full (counting abstained items as incorrect) and assigned (computed over non-abstained items only). (3) Cost: total LLM calls and dollar cost, computed under GPT-5.4 pricing [17]. Compared methods. We use the unsteered projection as a baseline and compare four steering configurations: per-item LLM steering [12], which performs LLM reasoning for each non-seed item, and three variants of our method—profile-only (γ=0), hy- brid (γ=0.5), and seed-only (γ=1). Since α ours upper-bounds the item-specific blend strength, whereas per-item LLM steering ap- plies a single coefficient across items, we set the per-item LLM steering coefficient to our method’s mean effective blend strength over steered items for a fair comparison. Table 1: Alignment quality on LitCovid (N = 4,999 † , mean± std over 3 random repetitions). ∆Sil and ∆NC are reported relative to the un- steered baseline (Sil=0.435, NC=0.779). Accuracy: full (assigned) ‡ . Method∆Sil↑∆NC↑Acc top-1↑Acc top-2↑ Per-item LLM [12] § 0.259± 0.0150.071± 0.0030.80 (0.90)– Ours (seed-only)0.248± 0.0330.010± 0.0120.64 (0.78)0.77 (0.94) Ours (profile-only)0.190± 0.0350.003± 0.0040.71 (0.74)0.87 (0.91) Ours (hybrid)0.302± 0.0700.023± 0.0060.70 (0.79)0.83 (0.94) † One article excluded due to non-parseable LLM responses in the per- item baseline on some seeds. ‡ “Full” counts abstained items as incorrect; “assigned” computes over non-abstained items only. Abstain rates: 0.110 (per-item), 0.115 (ours, hybrid). Per-item steering produces hard assignments and has no top-2. § Per-item LLM uses blending coefficient ̄ λ eff = 0.49± 0.02 (see §4.1) Implementation. All LLM calls use gpt-5.4 [17] with temper- ature 0. By default, we use γ=0.5, τ=20, ρ=0.85, and t none =0.20, and α ours = 0.9. 4.2 RQ1: Does group-level prototype steering match the alignment quality of per-item LLM steering? Table 1 reports alignment results on LitCovid. Our hybrid prototype achieves the strongest global alignment (∆Sil = 0.302), comparable to per-item LLM steering (0.259) despite issuing no per-item LLM calls. This suggests that a small number of group-level prototypes carry enough semantic information to globally reorganize a 5,000- document collection. The seed-only variant achieves lower but still comparable global alignment (0.248), while profile-only is notice- ably weaker (0.190), indicating that data-grounded structure and language-mediated abstraction contribute complementary signals; we examine this further in Section 4.4. Per-item steering retains an advantage in local consistency (∆NC = 0.071 vs. 0.023). This gap reflects a structural differ- ence between the two approaches: per-item reasoning produces a unique augmentation per document, allowing each to shift along a document-specific direction, whereas prototype-based steering pulls all documents in a group toward a shared target. The lat- ter strengthens between-group separation but reduces within-group differentiation. A similar pattern appears in assignment behavior: per-item reaches higher top-1 accuracy on assigned items (0.90 vs. 0.79), but our top-2 set-hit accuracy (0.94) approaches per-item’s top-1, indicating that disagreements primarily concern ambiguity between adjacent groups rather than misclassification (abstain rates are comparable: 0.110 vs. 0.115). Overall, group-level prototypes match per-item LLM steering on global alignment, with a con- trolled trade-off on within-group local structure. 4.3 RQ2: How much computational cost is reduced? A central motivation for our method is that semantic reasoning need not be performed for every item. By externalizing semantic intent once per group rather than per item, the cost of LLM reasoning becomes independent of dataset size. Table 2 quantifies this difference on LitCovid. At N = 5K, per- item LLM steering issues 5,001 reasoning calls and costs $32.35 per run, while our method issues a single LLM call (the group pro- file generation) at $0.025 per run, a 1,294× reduction in cost. Per- item cost scales linearly with N while ours remains constant; the gap widens at larger scales. This shift has direct implications for interactive use. Per-item LLM steering becomes increasingly im- practical as datasets grow, both in monetary cost and in the latency incurred by thousands of sequential or batched LLM calls. Our method, in contrast, requires only one LLM call regardless of N; subsequent stages reduce to embedding similarity and matrix opera- tions. Together with the alignment results in Section 4.2, these find- ings indicate that group-level abstraction enables semantic steering to operate at an interactive scale. Table 2: Computational cost comparison. Per-item steering scales as O(N); ours issues one LLM call regardless of N. N = 5K values are measured as the mean across 3 runs; other rows are projected. Per-item LLM [12]Ours Ncallstokens (in / out)$callstokens (in / out)$ratio 500 † 5010.80M / 80K3.2116.4K / 0.59K0.025128× 1K † 1,0011.61M / 161K6.4516.4K / 0.59K0.025258× 2K † 2,0013.24M / 322K12.9216.4K / 0.59K0.025517× 5K5,0018.10M / 807K32.3516.4K / 0.59K0.0251294× † Projected; per-item cost scales as N× per-call cost (measured at N = 5K). GPT-5.4 pricing: $2.50/M input, $15/M output [17]. 4.4 RQ3: What roles do seed and profile signals play? Our prototype design combines two complementary signals: the centroid of seed examples (γ = 1) and the embedding of the LLM- generated group profile (γ = 0). Table 1 reports three variants. The seed-only variant grounds prototypes directly in the analyst’s chosen examples, preserving local structure of the original embed- ding space. It achieves moderate global alignment (∆Sil = 0.248) but the lowest top-1 accuracy (0.64), suggesting that seed cen- troids alone do not generalize beyond the immediate neighborhood of the chosen examples. The profile-only variant relies entirely on language-mediated abstraction: the LLM articulates the shared meaning of each group, and this articulation drives prototype place- ment. It achieves the lowest global alignment (∆Sil = 0.190) but the lowest abstain rate (0.051) and competitive top-1 accuracy on assigned items (0.74), indicating that abstract semantic descriptions allow many documents to be assigned, but lack the data grounding needed to reorganize the projection at scale. The hybrid variant (γ = 0.5) combines both signals and achieves the strongest global alignment (∆Sil = 0.302). The two signals are complementary: seed centroids provide data-grounded local geometry, while profile embeddings provide language-articulated abstraction that general- izes beyond the seed neighborhood. Neither alone reproduces the hybrid’s performance, supporting our design choice of combining both signals instead of relying on either one alone. 5 CASE STUDIES We illustrate prototype-based steering on the two datasets in Figure 1, focusing on how user-defined groups reshape the projection and on how the method generalizes to both text and images. Text: LitCovid. The LitCovid baseline projection (Figure 1, left pair) places articles primarily by the geometry of their language- model embeddings. The four COVID-19 aspects overlap heavily in the central region: Prevention partially separates at the top, but Diagnosis, Treatment, and Case Report remain intermixed, mak- ing aspect structure difficult to read off the layout. Given five seed articles per aspect, our hybrid approach (γ = 0.5) reorganizes the layout into four spatially separated regions consistent with the an- alyst’s intent. Treatment articles consolidate into a single cohesive cluster on the left, and Diagnosis and Case Report (two aspects the baseline conflates) separate into distinct neighborhoods. The visual reorganization corresponds to the hybrid row of Table 1 and is qual- itatively comparable to per-item LLM steering [12], obtained here without any per-item LLM inference. Images: Stanford-40 Actions. To examine whether group-level abstraction generalizes beyond text, we apply the same pipeline to image embeddings on a 12-category subset of Stanford-40 Ac- tions [28] (2,583 images across 12 action categories). Image em- beddings come from CLIP ViT-B/32 [21], and group profiles are generated by GPT-5.4 [17] in vision mode using a JSON schema similar to that used in the text pipeline. UMAP uses n neighbors = 15; all other settings match the text pipeline. The baseline CLIP projection (Figure 1, right pair) forms a single connected manifold in which most categories overlap in the central region. After steering, the layout reorganizes into a set of well- separated regions aligned with the user-defined groups. Several categories that were heavily entangled in the baseline, such as wav- ing hands and brushing teeth, form distinct neighborhoods in the steered projection, while a smaller residual mixed region remains where visually similar actions (e.g., texting and pouring liquid) are not fully resolved. Quantitatively, the steered projection achieves ∆Sil = 0.20, ∆NC = 0.05, and top-1 accuracy = 0.70 and top-2 accuracy = 0.83 on non-seed images. These results suggest that the method extends naturally to other modalities with a compatible joint embedding space and a group-level profiling mechanism. 6 DISCUSSION From per-item reasoning to group-level abstraction. Our re- sults suggest that the scalability bottleneck in LLM-augmented se- mantic steering is partly a mismatch in granularity. Prior meth- ods [12, 16] reason about each item because projections display in- dividual points, but analysts often express intent by selecting and grouping examples [1, 7]. By shifting semantic reasoning from items to user-defined groups, our method aligns the computational unit with the interaction unit. The LLM externalizes intent once as group profiles; the method then uses the resulting prototypes to propagate that intent through embedding-space operations. Overview steering versus local refinement. Prototype steering is best viewed as a scalable overview-level steering mechanism for exploratory sensemaking. Because items are moved toward shared group-level prototypes, the method strengthens between-group sep- aration but provides less item-specific adaptation than per-item LLM steering [12]. This makes the method suitable for reorganiz- ing a projection around a small number of semantic concepts and obtaining a clearer overview of how a collection aligns with analyst intent. Tasks requiring fine-grained local neighborhoods or subtle within-group distinctions may benefit from subsequent local inter- actions or per-item refinement. The generated profiles also provide inspectable group-level semantics, although prototype assignments do not provide item-level rationales as in per-item LLM steering. Limitations and future work. Our quantitative evaluation uses simulated groups derived from ground-truth labels, but real analyst- selected groups may be noisier, more ambiguous, or less represen- tative [26]. Performance also depends on profile quality: incor- rect or overly broad profiles can bias prototype construction, a risk that hybrid fusion and abstention mitigate but do not eliminate. Unlike constrained or label-guided projection methods [27, 15], our method externalizes analyst-defined seed groups as language- articulated semantic profiles and uses prototype-based embedding adjustments to steer projections without retraining. Future work should evaluate robustness under noisy or ambiguous seeds, com- pare against additional semantic-steering baselines [1, 27], extend to broader multimodal datasets, and study interactive inspection and refinement of prototypes with human analysts [7]. 7 CONCLUSION We presented a scalable semantic steering method that shifts LLM reasoning from individual items to user-defined groups. By gener- ating group-level semantic profiles once and combining them with seed centroids into hybrid prototypes, the method propagates an- alyst intent through embedding-space operations without repeated LLM calls or retraining. The LitCovid results show that this group- level formulation can achieve global alignment comparable to per- item LLM reasoning while substantially reducing the cost of se- mantic propagation. The image case study further suggests that the same prototype-based mechanism can generalize beyond text. These findings indicate that group-level representations provide a practical path toward scalable LLM-augmented semantic steering. SUPPLEMENTAL MATERIALS Supplemental materials are available on OSF at https://osf.io/ vr4e9/. They include additional LitCovid projection comparisons, parameter sensitivity results, and the generated LitCovid group pro- files. ACKNOWLEDGMENTS This research was supported by industry, government, and institute members of the NSF SHREC Center, which was founded in the IUCRC program of the National Science Foundation. ChatGPT (OpenAI) was used for language editing and writing refinement of the manuscript. The authors take full responsibility for the content of the final manuscript. REFERENCES [1] Y. Bian and C. North. DeepSI: Interactive deep learning for seman- tic interaction. In Proceedings of the 26th International Conference on Intelligent User Interfaces, IUI ’21, p. 197–207. Association for Com- puting Machinery, New York, NY, USA, 2021. doi: 10.1145/3397481 .3450670 1, 2, 4 [2] E. T. Brown, J. Liu, C. E. Brodley, and R. Chang. Dis-function: Learn- ing distance functions interactively. In 2012 IEEE Conference on Vi- sual Analytics Science and Technology (VAST), p. 83–92, 2012. doi: 10.1109/VAST.2012.6400486 2 [3] R. Buchm ̈ uller, D. Collaris, L. Meng, and A. Chatzimparmpas. Lan- gLasso: Interactive cluster descriptions through LLM explanation, 2026. doi: 10.48550/arXiv.2601.10458 2 [4] Q. Chen, A. Allot, R. Leaman, R. Islamaj, J. Du, L. Fang, K. Wang, S. Xu, Y. Zhang, P. Bagherzadeh, et al. Multi-label classification for biomedical literature: an overview of the BioCreative VII Lit- Covid track for COVID-19 literature topic annotations. Database, 2022:baac069, 2022. doi: 10.1093/database/baac069 3 [5] M. Dowling, J. Wenskovitch, J. Fry, S. Leman, L. House, and C. North. SIRIUS: Dual, symmetric, interactive dimension reduc- tions. IEEE Transactions on Visualization and Computer Graphics, 25(1):172–182, 2019. doi: 10.1109/TVCG.2018.2865047 2 [6] A. Endert, P. Fiaux, and C. North. Semantic interaction for visual text analytics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, p. 473–482. Association for Computing Machinery, New York, NY, USA, 2012. doi: 10.1145/ 2207676.2207741 1, 2 [7] A. Endert, S. Fox, D. Maiti, S. Leman, and C. North. The semantics of clustering: analysis of user-generated spatializations of text docu- ments. In Proceedings of the International Working Conference on Ad- vanced Visual Interfaces, AVI ’12, p. 555–562. Association for Com- puting Machinery, New York, NY, USA, 2012. doi: 10.1145/2254556 .2254660 2, 4 [8] D. Goswami, S. Magistri, G. M. van de Ven, B. Twardowski, A. D. Bagdanov, T. Tuytelaars, and J. van de Weijer. Cross-modal prototype alignment and mixing for training-free few-shot classification, 2026. doi: 10.48550/arXiv.2603.24528 1, 2 [9] J. Hong, J. Park, D. Kim, S. Choi, B. Son, and J. Kang. Empowering sentence encoders with prompting and label retrieval for zero-shot text classification, 2023. doi: 10.48550/arXiv.2212.10391 2 [10] Z. Huang, D. Witschard, K. Kucher, and A. Kerren. VA + embeddings STAR: A state-of-the-art report on the use of embeddings in visual analytics. Computer Graphics Forum, 42(3):539–571, 2023. doi: 10. 1111/cgf.14859 1 [11] J. Lin, R. Faust, and C. North. ImageSI: Semantic interaction for deep learning image projections. In 2024 IEEE Visualization and Vi- sual Analytics (VIS), p. 91–95, 2024. doi: 10.1109/VIS55277.2024. 00026 1, 2, 3 [12] W. Liu, E. Krokos, K. Whitley, R. Faust, and C. North.LLM- augmented semantic steering of text embedding projection spaces. In Proceedings of the 2026 International Conference on Advanced Vi- sual Interfaces, AVI ’26. Association for Computing Machinery, New York, NY, USA, 2026. doi: 10.1145/3811427.3811450 1, 2, 3, 4 [13] W. Liu, C. North, and R. Faust. Visualizing spatial semantics of dimensionally reduced text embeddings, 2024. doi: 10.48550/arXiv .2409.03949 1 [14] L. McInnes, J. Healy, and J. Melville. UMAP: Uniform manifold approximation and projection for dimension reduction, 2020. doi: 10. 48550/arXiv.1802.03426 2, 3 [15] L. Meng, S. van den Elzen, N. Pezzotti, and A. Vilanova. Class- constrained t-sne: Combining data features and class probabili- ties. IEEE Transactions on Visualization and Computer Graphics, 30(1):164–174, 2024. doi: 10.1109/TVCG.2023.3326600 4 [16] A. A. Oliveira, M. Espadoto, R. Hirata, Jr., R. M. Cesar, Jr., and A. C. Telea. Creating user-steerable projections with interactive semantic mapping, 2025. doi: 10.48550/arXiv.2506.15479 1, 2, 4 [17] OpenAI.OpenAI API Documentation. https://platform. openai.com/docs, 2026. Accessed: 2026-04-23. 1, 3, 4 [18] J. J. P, K. Palanisamy, Y.-W. Chao, X. Du, and Y. Xiang. Proto-CLIP: Vision-language prototypical network for few-shot learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), p. 2594–2601, 2024. doi: 10.1109/IROS58592.2024. 10801660 1, 2 [19] F. Pahde, M. Puscas, T. Klein, and M. Nabi. Multimodal prototypical networks for few-shot learning. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), p. 2643–2652, 2021. doi: 10.1109/WACV48630.2021.00269 2 [20] S. Pratt, I. Covert, R. Liu, and A. Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classifica- tion. In 2023 IEEE/CVF International Conference on Computer Vi- sion (ICCV), p. 15645–15655, 2023. doi: 10.1109/ICCV51070.2023 .01438 1, 2 [21] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervi- sion. In M. Meila and T. Zhang, eds., Proceedings of the 38th Inter- national Conference on Machine Learning, vol. 139 of Proceedings of Machine Learning Research, p. 8748–8763. PMLR, 18–24 Jul 2021. 1, 2, 4 [22] P. J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65, 1987. doi: 10.1016/0377-0427(87)90125-7 3 [23] D. Sacha, L. Zhang, M. Sedlmair, J. A. Lee, J. Peltonen, D. Weiskopf, S. C. North, and D. A. Keim. Visual interaction with dimensional- ity reduction: A structured literature analysis. IEEE Transactions on Visualization and Computer Graphics, 23(1):241–250, 2017. doi: 10. 1109/TVCG.2016.2598495 2 [24] J. Z. Self, M. Dowling, J. Wenskovitch, I. Crandell, M. Wang, L. House, S. Leman, and C. North. Observation-level and parametric interaction for high-dimensional data analysis. ACM Trans. Interact. Intell. Syst., 8(2), June 2018. doi: 10.1145/3158230 2 [25] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few- shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, p. 4080–4090. Curran Associates Inc., Red Hook, NY, USA, 2017. 1, 2 [26] J. Wenskovitch, M. Dowling, and C. North. Toward addressing am- biguous interactions and inferring user intent with dimension reduc- tion and clustering combinations in visual analytics. ACM Trans. In- teract. Intell. Syst., 14(1), Jan. 2024. doi: 10.1145/3588565 4 [27] J. Xia, L. Huang, W. Lin, X. Zhao, J. Wu, Y. Chen, Y. Zhao, and W. Chen. Interactive visual cluster analysis by contrastive dimension- ality reduction. IEEE Transactions on Visualization and Computer Graphics, 29(1):734–744, 2023. doi: 10.1109/TVCG.2022.3209423 4 [28] B. Yao, X. Jiang, A. Khosla, A. L. Lin, L. Guibas, and L. Fei-Fei. Human action recognition by learning bases of action attributes and parts. In 2011 International Conference on Computer Vision, p. 1331–1338, 2011. doi: 10.1109/ICCV.2011.6126386 4 [29] Y. Ye, S. Xiao, X. Zeng, and W. Zeng.ModalChorus: Visual probing and alignment of multi-modal embeddings via modal fusion map. IEEE Transactions on Visualization and Computer Graphics, 31(1):294–304, 2025. doi: 10.1109/TVCG.2024.3456387 2