Paper deep dive
From High Recall to High Utility: Dataset-Adaptive Post-Processing of LLM-Generated Customer Intents
Mahesh Viswanathan, Joan Rossello, Leticia Fernandes, Paul Mutawe
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/8/2026, 6:47:48 AM
Summary
This paper presents a dataset-adaptive post-processing architecture for Customer Intent Extraction (CIE) that transforms high-recall, noisy LLM-generated candidate intents into stable, traceable, and reusable enterprise intelligence. The system separates recall-oriented extraction from utility-oriented reduction, employing source-specific preprocessing, semantic clustering (using UMAP and HDBSCAN), and constrained language-model aggregation to deduplicate and consolidate intents while preserving provenance and explainability. Two downstream applications are described: Machine-Generated Intents for cold-start recommendations and intent-guided semantic retrieval for mapping customer intents to business outcomes.
Entities (9)
Relation Signals (8)
Customer Intent Extraction â isdevelopedby â Cisco Systems
confidence 95% · Authors' affiliation: Cisco Systems, Inc.
Customer Intent Extraction â uses â Large Language Models
confidence 95% · Large language models can extract useful signals from heterogeneous enterprise data... We present a dataset-adaptive post-processing architecture developed for Customer Intent Extraction (CIE)
Post-Processing Architecture â supports â Machine-Generated Intents
confidence 92% · We also describe two downstream applications: Machine-Generated Intents... and intent-guided semantic retrieval
Post-Processing Architecture â uses â HDBSCAN
confidence 90% · followed by Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN)-style density-based clustering
Post-Processing Architecture â uses â Sentence-BERT
confidence 90% · Sentence-level semantic representations are computed using the Sentence-BERT family of siamese encoders
Post-Processing Architecture â uses â UMAP
confidence 90% · the adaptive clustering policy uses Uniform Manifold Approximation and Projection (UMAP) [6] for dimensionality reduction
End-to-End Plans â isinputto â Customer Intent Extraction
confidence 85% · Source-specific preprocessing first isolates evidence from multimodal plans... E2E plans are multimodal presentation documents
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models can extract useful signals from heterogeneous enterprise data, but high-recall extraction often produces outputs that are duplicated, uneven in granularity, semantically overlapping, or too numerous for downstream systems and human reviewers to use effectively. We present a dataset-adaptive post-processing architecture developed for Customer Intent Extraction (CIE), where unstructured customer language is transformed into stable, traceable intent units. The approach separates recall-oriented extraction from utility-oriented reduction. Source-specific preprocessing first isolates evidence from multimodal plans, sparse operational records, and structured opportunity data. Candidate intents are then standardized and deduplicated, optionally enriched with metadata for embedding computation, represented in a shared semantic vector space, and grouped using a clustering strategy selected according to the candidate set's characteristics. Cluster-level keywords provide an explainability layer, while singleton reassignment requires agreement between embedding and keyword similarity. Finally, constrained language-model aggregation produces one concise intent per cluster without introducing unsupported concepts, and the resulting unit retains provenance, clustering, embedding, and generation metadata. This treats post-processing not as cosmetic cleanup, but as a semantic reduction layer converting high-recall LLM outputs into reusable enterprise intelligence. We also describe two downstream applications: Machine-Generated Intents, which infer likely objectives for customers lacking direct evidence from peer customers with similar profiles, and intent-guided semantic retrieval and mapping, which uses the stable intent as a query against a downstream decision space, illustrated here by mapping customer intents to business outcomes.
Tags
Links
- Source: https://arxiv.org/abs/2610.09039v1
- Canonical: https://arxiv.org/abs/2610.09039v1
Trouble viewing inline? Open PDF directly â
Full Text
61,238 characters extracted from source content.
Expand or collapse full text
From High Recall to High Utility: Dataset-Adaptive Post-Processing of LLM-Generated Customer Intents Mahesh Viswanathan Affiliation: Cisco Systems, Inc., San Jose, CA, USA mahviswa, joanrb, leticiaf, pmutawe@cisco.com Joan Rossello Affiliation: Cisco Systems, Inc., San Jose, CA, USA mahviswa, joanrb, leticiaf, pmutawe@cisco.com Leticia Fernandes Affiliation: Cisco Systems, Inc., San Jose, CA, USA mahviswa, joanrb, leticiaf, pmutawe@cisco.com Affiliation: Cisco Systems, Inc., San Jose, CA, USA mahviswa, joanrb, leticiaf, pmutawe@cisco.com Paul Mutawe Affiliation: Cisco Systems, Inc., San Jose, CA, USA mahviswa, joanrb, leticiaf, pmutawe@cisco.com Abstract Large language models can extract useful signals from heterogeneous enterprise data, but high-recall extraction often produces outputs that are duplicated, uneven in granularity, semantically overlapping, or too numerous for downstream systems and human reviewers to use effectively. We present a dataset-adaptive post-processing architecture developed in the context of Customer Intent Extraction (CIE), where unstructured customer language is transformed into stable, traceable intent units. The approach deliberately separates recall-oriented extraction from utility-oriented reduction. Source-specific preprocessing first isolates evidence from multimodal plans, sparse operational records, and structured opportunity data. Candidate intents are then standardized and deduplicated, optionally enriched with metadata for embedding computation, represented in a shared semantic vector space, and grouped using a clustering strategy selected according to characteristics of the candidate set. Cluster-level keywords provide an explainability layer, while singleton reassignment requires agreement between embedding and keyword similarity. Finally, constrained language-model aggregation produces one concise intent per cluster without introducing unsupported concepts, and the resulting unit retains provenance, clustering, embedding, and generation metadata. The resulting architecture treats post-processing not as cosmetic cleanup, but as a semantic reduction layer that converts high-recall LLM outputs into reusable enterprise intelligence. Beyond the extraction pipeline itself, we describe two downstream applications of the resulting intent representation: Machine-Generated Intents, which infer likely objectives for customers lacking direct evidence by drawing on peer customers with similar profiles, and intent-guided semantic retrieval and mapping, which uses the stable intent as a query against a downstream decision space, illustrated here by mapping customer intents to business outcomes. Keywords: customer intent extraction, large language models, information extraction, semantic clustering, embeddings, post-processing, explainable AI, enterprise AI 1 Introduction Enterprise systems increasingly rely on large language models (LLMs) to recover useful information from documents, operational records, conversations, and other forms of unstructured or semi-structured data [1]. In customer-facing settings, however, useful information is rarely expressed as a clean label or a single self-contained sentence. A customer objective may be distributed across presentation tables, free-form notes, key performance indicators (KPIs), opportunity records, next-step fields, and product metadata. Different customers may describe the same objective in different words, while superficially similar statements can refer to materially different needs. Customer Intent Extraction (CIE) was developed to bridge this gap between natural customer language and the structured representations required by analytics, recommendations, and customer-success workflows. In CIE, a customer intent is an intermediate representation of what the customer is trying to achieve: it combines an overarching business strategy with a specific product, technology, domain, or operational goal. A useful intent must be goal-oriented, grounded in source evidence, appropriately scoped, and reusable beyond the source artifact from which it was inferred. The central engineering challenge is not simply to prompt an LLM to âsummarizeâ customer data. It is to identify what counts as intent, recover the evidence that supports it, preserve context, and consolidate repeated or overlapping statements without erasing distinctions that matter. A raw model response may sound plausible, but a production intent layer must also produce outputs that are stable, traceable, explainable, and structured for downstream consumption. This work rests on a deliberate asymmetry between extraction and post-processing. The early stages are optimized for recall: when the relevant evidence is not known in advance, candidate objectives should be captured broadly enough that important signals are unlikely to be lost. The post-processing stages are optimized for utility: the resulting candidate set is standardized, deduplicated, semantically grouped, checked for ambiguity, and reduced to a smaller set of stable intent units. The system therefore accepts redundancy early in the pipeline in exchange for a controlled and auditable reduction later. The primary contribution of this article is a post-processing architecture for transforming high-recall LLM-generated candidate intents into a smaller set of stable, explainable, and reusable intent units. Specifically, we describe: 1. A high-recall-to-high-utility design in which broad candidate extraction is followed by a dedicated semantic reduction stage; 2. A dataset-adaptive post-processing pipeline that combines deterministic standardization, optional metadata-aware semantic representation, and clustering strategies selected according to characteristics of the candidate set; 3. A conservative approach to ambiguity and outliers, in which singleton intents are reassigned only when multiple semantic signals provide sufficient evidence; 4. Constrained cluster-level aggregation that reduces redundancy while preserving the distinct information contained in the underlying intents; and 5. A traceable output representation that retains cluster membership, source provenance, semantic descriptors, and model and processing metadata alongside each consolidated intent. The remainder of the article focuses on the construction of the intent intelligence layer and, in particular, the post-processing architecture. Later sections describe two applications of the resulting intent representation: Machine-Generated Intents (MGI), which infer candidate intents for customers using peer and metadata information, and intent-guided semantic retrieval and mapping, in which an intent is used to retrieve and rank relevant downstream entities. Business-outcome mapping is presented as one concrete implementation of this broader retrieval pattern. Evaluation and the broader engineering implications of this architecture are then discussed as separate concerns. 2 Literature Review This article sits at the intersection of LLM-based information extraction, semantic representation, clustering, and explainable enterprise AI. The technical components used by the systemâembedding models, cosine similarity, hierarchical clustering, dimensionality reduction, density-based clustering, keyword generation, and constrained language-model synthesisâare established techniques. The contribution described here is the coordinated use of these components as a reduction layer after deliberately high-recall extraction, with the clustering workflow selected according to characteristics of the candidate set and with uncertainty and provenance preserved explicitly. 2.1 Intent Extraction and Business Language Understanding Extracting actionable intent from natural language has been studied extensively outside the enterprise setting. Corbeil et al. [2] build an intent landscape directly from real-world conversational corpora using extractive question-answering transformers, addressing a problem close to ours: recovering a structured objective space from heterogeneous, freeform customer language rather than from a fixed label set. Manias et al. [3] apply large language models to intent extraction in a different domain, translating natural-language network operator intents into structured configuration for 5G core networks; their work shares our motivation of using an LLM to bridge unstructured intent expression and a structured downstream representation, though the target representation there is a deterministic network configuration rather than a reusable semantic unit. Liu et al. [4] infer commercial intention from user behavior for sponsored search ad retrieval, illustrating a parallel high-volume, business-facing setting in which inferred intent becomes the input to a retrieval pipeline, similar in spirit to our intent-guided semantic retrieval and mapping application. At the systems level, Cheng et al. [1] describe how large language models were deployed across Meituanâs customer interaction systems to jointly improve customer satisfaction and reduce operating cost, reinforcing the broader premise of this article: production customer-facing LLM systems succeed or fail on data engineering, representation, and evaluation discipline rather than on model choice alone. None of these works, however, treat the extracted intent itself as a versioned, provenance-carrying data asset that is deliberately over-generated and then reduced; that framing is the specific contribution of CIEâs post-processing layer. 2.2 Semantic Representation, Clustering, and Deduplication The embedding, clustering, and deduplication components used in the post-processing pipeline build on established techniques. Sentence-level semantic representations are computed using the Sentence-BERT family of siamese encoders [5], which made cosine-similarity comparison between short texts computationally practical at the scale required by CIEâs candidate-intent volumes. For larger candidate sets, the adaptive clustering policy uses Uniform Manifold Approximation and Projection (UMAP) [6] for dimensionality reduction followed by Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN)-style density-based clustering [7], chosen because, unlike centroid-based methods, it tolerates clusters of uneven size and density and exposes outliers explicitly rather than forcing every point into a group. The standardization stageâs exact-deduplication logic is a deterministic special case of a broader problem: semantic deduplication of large, noisy text collections, studied by Abbas et al. [8] in the context of curating web-scale training corpora. CIEâs contribution here is not any one of these components but their coordinated, dataset-adaptive application as a post-extraction reduction layer, with the selection between clustering families made explicit and recorded rather than fixed in advance. 2.3 Constrained Generation, Faithfulness, and Evaluation Because both the keyword-extraction and constrained-aggregation stages rely on an LLM, the pipeline inherits the LLMâs well-documented tendency to introduce unsupported content. Huang et al. [9] provide a taxonomy of hallucination in large language models that motivates the explicit factual constraint placed on the aggregation generator: aggregation is restricted to information already present in the clusterâs member intents rather than treated as open-ended synthesis. On the evaluation side, using a capable LLM as an automatic judge of open-ended generation quality follows the practice formalized by Zheng et al. [10]; this article treats that practice as one input among several, alongside subject-matter expert (SME) review, rather than as a sufficient validation step on its own. 2.4 Retrieval-Based Intent-to-Outcome Mapping and Predictive Intent Generation The intent-guided semantic retrieval and mapping application uses a bi-encoder for high-recall candidate retrieval followed by a cross-encoder for precision re-ranking. This retrieve-fast, rerank-smart pattern is the same one studied by Geigle et al. [11] for cross-modal retrieval, and it generalizes the same architecture introduced for sentence encoders in [5]: an efficient, independently computed representation narrows the candidate space before a more expensive, jointly computed representation refines the final ranking. Machine-Generated Intents, which infer likely objectives for a customer from peer customersâ known intents, is a content-based instance of the cold-start recommendation problem: Viljanen et al. [12], working in a game-recommendation setting, address the same underlying challenge of producing useful recommendations for entities with no direct interaction history by relying on profile-level feature similarity, which is the same mechanism MGI applies at the customer level. Finally, this work extends the authorsâ broader interest in coordinating multiple AI components into auditable, production-grade enterprise systems: DAWN [13], the authorsâ prior work on a framework for orchestrating distributed AI agents, shares with the architecture described here the practice of wrapping every model invocation in an explicit, versioned, and inspectable structure so that a system built from stochastic components can still produce outputs an enterprise can trust, trace, and reprocess. 3 Problem Definition and Customer Intent 3.1 Definition CIE treats intent as a reusable data asset rather than a one-time model output. An intent is a statement that represents the customerâs strategic objective and the specific goal associated with a product, technology, domain, or operational context. It is not equivalent to an activity, a next step, or a product mention. For example, âuploaded the latest architecture deckâ describes a completed activity, while ârun an adoption workshopâ describes a next step. âCatalyst Center deploymentâ identifies a topic or technology. By contrast, âimprove network visibility to identify performance issues earlierâ expresses a desired operational outcome with enough context to be actionable as an intent. A statement such as âimprove visibilityâ may be a candidate intent but remains underspecified until product, domain, or other evidence is available. This distinction matters because the same lexical surface form may hide different objectives. Visibility in network operations can refer to performance monitoring and faster troubleshooting, whereas visibility in a security context can refer to threat detection, user activity, or compliance reporting. Customer intent extraction is therefore not adequately described as a fixed-label classification problem. 3.2 Design desiderata A stable intent unit is designed around seven properties: Faithful (the final statement contains only concepts supported by member intents and source evidence); Traceable (the raw candidates and source references remain available); Explainable (cluster keywords, representative members, centroids, and clustering metadata reveal how the output was formed); Consistent (canonical metadata and constrained generation reduce avoidable variation); Uncertainty-aware (unusual or weakly supported candidates can remain independent rather than being forced into a group); Versioned (model, prompt, embedding, and clustering configurations are recorded); and Product-consumable (the representation is predictable enough to support downstream systems). Figure 1: End-to-end operating model. Source-specific preparation preserves evidence, while a common post-processing layer transforms high-recall candidate intents into stable outputs. Figure 1 shows the end-to-end operating model. Stability does not imply immutable wording. Models, prompts, thresholds, and taxonomies will evolve. Stability means that downstream systems receive a consistent semantic object with enough provenance to explain, compare, and reprocess the representation as the implementation changes. 4 Intent Extraction 4.1 Why recall is favored early When the relevant evidence is unknown in advance, optimizing the first inference step for apparent precision can silently discard useful objectives. Customer evidence may appear in strategic-priority tables, initiative descriptions, KPIs, operational notes, or multiple fragments across a document. CIE therefore separates two objectives that are often combined: an extraction stage that maximizes evidence and candidate coverage, and a post-processing stage that maximizes usability and reduces semantic redundancy. The cost of this choice is predictable. The raw candidate set tends to contain exact duplicates, paraphrases, uneven levels of granularity, vague statements, and repeated evidence from different sources. The post-processing layer is designed specifically to absorb this cost. 4.2 Source-specific context preparation The preprocessing problem differs substantially by source. Applying one generic preprocessing strategy to every source would either remove useful evidence or pass excessive noise through to the LLM. 4.2.1 End-to-End plans End-to-End (E2E) plans are multimodal presentation documents in which meaning is often carried by layout as much as by text. Tables, merged cells, headings, and spatial relationships can determine which customer priority is associated with which initiative or product. SMEs identify target chapters containing high-value intent evidence. A PowerPoint (PPTX) parser is used to extract corresponding table content into structured JSON, followed by manual validation. This produces a reference representation against which multimodal extraction can be assessed, including keys, cell values, table structure, and merged-cell behavior. The objective at this stage is not to write the intent. It is to produce a compact and faithful evidence package that allows the intent extractor to reason over the relevant customer context without carrying the entire source presentation into the prompt. 4.2.2 Customer Success Console Customer Success Console (CS Console) data presents the opposite problem: the source is already fielded, but it is extremely sparse and repetitive. Many records do not include an in-scope product, free-form fields can be operationally uninformative, supporting fields are frequently empty, and exact or near-exact duplicates are common. The preprocessing pipeline therefore applies deterministic filters before LLM inference: records are restricted to non-empty in-scope product values, exact duplicates are removed, and regex rules reject known low-signal patterns such as meeting counts, document uploads, and similar administrative updates. In the dataset used during development, this reduced 300,398 records from 1,252 customers to 16,854 records from 1,138 customers, i.e., roughly five to six percent of the original records while preserving most customer coverage, as shown in Figure 2. Figure 2: Data reduction and coverage for E2E Plans and CS Console sources. For each source, the chart reports the total number of raw records considered, the number retained as valid candidate intents after preprocessing, and the resulting coverage percentage, illustrating that aggressive filtering removes the large majority of raw records while preserving most customer coverage. Table 1: Source-specific preparation before common intent extraction. Dimension E2E plans CS Console Sales Data Sources Primary form Slides, tables, merged cells, structured narrative Large sets of fielded records and free-form entries Related structured tables containing intent, opportunity, and product information Main failure mode Loss of layout and table relationships Sparse fields, low-signal updates, duplicates, missing product context Relevant context distributed across tables and inconsistent product representations Core preparation Target-chapter selection, multimodal table extraction, structured validation Product-scope filtering, regex rejection, empty-field handling, exact deduplication Table joins, product mapping, feature engineering, schema normalization Output Compact structured context package Filtered records grouped by customer and product Structured intent/opportunity record with enriched product context 4.2.3 Structured opportunity sources Table 1 summarizes these differences across sources. Structured sales data sources require a different form of preparation: the challenge is less about recovering presentation layout or filtering a single sparse table and more about assembling related information distributed across multiple datasets. Sales data source preparation, for example, combines an intent source table, opportunity line items, and product hierarchy information. Feature engineering maps product context to each candidate and produces a structured record that can be normalized into the common CIE schema. 4.3 Raw Intent Extraction and Representation Once relevant context has been isolated, a source-aware LLM extractor generates candidate intents. Different prompt patterns are used for sources with different evidence structure. E2E prompts operate on structured plan context, while CS Console prompts operate on filtered record context and can include few-shot examples that demonstrate the expected intent style. The extractor is instructed to recover the customer objective supported by the evidence rather than summarize every detail. Its output is explicitly treated as a candidate, not a final intent. The same underlying goal can therefore appear multiple times when supported by different records or sources. Each candidate preserves both its generated statement and its provenance. A practical raw intent object contains the intent text, source identifiers, trace pointers, a model-predicted product value, a canonical product value after standardization, customer or account scope, and generation metadata such as prompt version, model version, and timestamp. This provenance becomes essential during deduplication, debugging, reprocessing, and evaluation. Formally, let a candidate set be I=(xi,mi)i=1N,I=\(x_i,m_i)\_i=1^N, (1) where xix_i is the candidate intent text and mim_i is a metadata object containing source, product, customer scope, and generation information. The post-processing layer maps I to a smaller collection of clusters C=C1,âŠ,CK,Kâ€N,C=\C_1,âŠ,C_K\, K†N, (2) and generates one consolidated intent aka_k for each final cluster CkC_k. 5 Dataset-Adaptive Post-Processing The post-processing pipeline consists of eight logical steps: standardization, context padding, embedding generation, clustering strategy selection, clustering, keyword extraction and normalization, outlier handling, and constrained aggregation. These stages are intentionally modular so that individual components can be evaluated, tuned, or replaced without changing the output contract. 5.1 Standardization and exact deduplication Standardization creates a comparable input set before any semantic operation. Whitespace and casing are normalized, unusable or empty items are rejected, and exact duplicate text is collapsed [8]. When duplicate strings carry different source or product predictions, the records are merged rather than discarded so that provenance is preserved. Table 2: Post-processing stages and their engineering purpose. Step Stage Purpose Key output 1 Standardization Remove formatting variation, exact duplicates, empty items, and metadata inconsistency Clean candidate intent objects 2â3 Context padding and embeddings Enrich short/ambiguous text for embedding only and map candidates to a common semantic vector space Padded representation, unchanged display text, embedding and model version 4â5 Adaptive clustering Select and apply an interpretable/scalable grouping strategy according to candidate-set characteristics Cluster assignments, centroids, method and parameters 6 Keyword extraction Describe shared cluster meaning and provide an explainability signal Normalized cluster keyword set 7 Outlier handling Preserve unique intents or cautiously repair uncertain singleton assignments Updated clusters plus retained singletons 8 Constrained aggregation Produce one faithful, non-redundant statement per cluster Combined intent plus generation metadata Table 2 summarizes all eight stages. Product metadata is similarly mapped to approved canonical keys. Variants, abbreviations, or model predictions can be reconciled using maintained mappings or semantic similarity against known names and aliases. The original predictions are still retained for debugging. This stage is deliberately deterministic. Exact problems should be solved with exact logic: an LLM is not required to trim whitespace, detect identical strings, or enforce a known product mapping. Reserving probabilistic methods for genuinely semantic decisions reduces cost and improves interpretability. 5.2 Context padding and semantic representation Short intents can be semantically unstable in isolation. For example, âimprove visibilityâ may refer to application performance, network operations, threat detection, user activity, or compliance. For small or semantically sparse candidate sets, the system can create an embedding-only representation by appending selected metadata such as product, topic, domain, or customer scope. Let pip_i denote the text used for embedding. If a padding condition is satisfied, pi=xiâgâĄ(mi)p_i=x_i g(m_i); otherwise, pi=xip_i=x_i, where gâĄ(mi)g(m_i) serializes only the selected contextual metadata and â denotes concatenation. Crucially, pip_i does not replace the original customer-facing text xix_i. The padded representation is an internal semantic scaffold used only for embedding computation. A common embedding model fΞf_Ξ is then applied: ei=fΞâ(pi).e_i=f_Ξ(p_i). (3) Semantic proximity is measured using cosine similarity, sâĄ(i,j)=eiâ€âejâ„eiâ„ââ„ejâ„.s(i,j)= e_i e_j e_i e_j . (4) The embedding model and version are stored because changes to the embedding space can alter similarity distributions and cluster boundaries. 5.3 Adaptive clustering rather than one algorithm for every scale The number of candidate intents can vary substantially by customer, product, and source. A clustering method that is transparent for ten statements can be inefficient or unstable for hundreds. The initial CIE implementation therefore selects among materially different clustering families according to candidate-set size. Figure 3: Dataset-size-based clustering policy used in the initial implementation. Thresholds are tunable. The operational policy, shown in Figure 3, is that Gmethodâ(N)G_method(N) is a pairwise similarity graph when Nâ€20N†20, hierarchical agglomerative clustering when 20<N<5020<N<50, and dimensionality reduction followed by density-based clustering when Nâ„50Nâ„ 50. These values are engineering starting points, not universal scientific constants. They are configurable and should be tuned using historical data, cluster purity, singleton rate, total cluster count, runtime, and human review quality. The central idea is the adaptive selection itself, together with recording the selected method and parameters for reproducibility. For small sets, pairwise cosine similarities are computed and edges are created when similarity exceeds a calibrated threshold. Connected components become clusters. The method is highly inspectable but can be sensitive to threshold choice and single-link chaining. For medium-sized sets, hierarchical agglomerative clustering provides a global view of group structure while remaining relatively easy to inspect. Cosine distance and a linkage rule construct a dendrogram, which is partitioned using a tuned distance threshold. For larger sets, the implementation uses dimensionality reduction via UMAP [6] followed by density-based clustering (HDBSCAN) [7]. This approach can identify clusters of different sizes and densities while also flagging isolated intents as potential outliers, making it more suitable for larger candidate sets. 5.4 Cluster-level keywords as an explainability layer After clustering, an LLM generates semantic descriptors for the shared themes in each cluster. Singleton clusters receive keywords from their individual statement. Raw keywords are normalized for casing and punctuation, deduplicated, and stripped of irrelevant tokens; domain synonyms can optionally be merged using maintained dictionaries or controlled AI assistance. Keywords are not a replacement for the combined intent. They serve three roles. First, they allow a reviewer to inspect why a group of statements was considered coherent. Second, they provide an independent semantic signal during outlier handling. Third, they give the aggregation step an explicit set of concepts to preserve. Keyword quality can be evaluated along at least two dimensions: coverage, or whether the descriptors capture the content of the member intents; and distinctiveness, or whether neighboring clusters receive meaningfully different descriptors. 5.5 Outlier handling and visible uncertainty Clustering creates ambiguity as well as structure. Treating every singleton as an error can discard legitimate low-frequency customer objectives, whereas forcing every item into a cluster creates false certainty. Figure 4: Keyword extraction, outlier handling and constrained intent aggregation in the post-processing pipeline. CIE therefore uses a conservative reassignment policy for singleton intents, illustrated in Figure 4. Let u denote a singleton intent and let CkC_k denote a candidate destination cluster. Each intent i in CkC_k has an embedding vector eie_i. The centroid of the cluster, ckc_k, is calculated as the mean of the embeddings of its member intents: ck=1|Ck|ââiâCkei.c_k= 1|C_k| _iâ C_ke_i. (5) The semantic similarity between the singleton and the cluster is then measured using cosine similarity between the singleton embedding eue_u and the cluster centroid: seâ(u,k)=cosâĄ(eu,ck),s_e(u,k)= (e_u,c_k), (6) where seâ(u,k)s_e(u,k) is the embedding-similarity score. A second score, swâ(u,k)s_w(u,k), measures semantic similarity between the keywords associated with the singleton and those associated with the cluster. The singleton is reassigned to cluster CkC_k only when both scores exceed their respective thresholds: seâ(u,k)â„Ïeâ§swâ(u,k)â„Ïw,s_e(u,k)â„ _e\ \ s_w(u,k)â„ _w, (7) where Ïe _e and Ïw _w are the minimum embedding- and keyword-similarity thresholds. If several clusters satisfy both conditions, the strongest-supported cluster is selected. If none satisfy them, the singleton remains as an independent intent. The use of two signals is intentionally conservative. The system accepts that the correct representation may be âunique or uncertainâ rather than manufacturing a false match. Outlier rate, reassignment rate, and human acceptance of reassigned examples are useful signals for threshold calibration. 5.6 Constrained aggregation The final step converts each cluster into one concise statement. A language model receives the cluster members and, optionally, the normalized keywords. The generation instruction is narrow: remove repetition, preserve unique supported concepts, and do not introduce information not present in the cluster [9]. For example, a cluster containing âimprove application visibility and monitoring for operations,â âgain deeper insights into application performance,â and âimprove application monitoring for operational efficiencyâ can be aggregated into a statement such as âimprove application visibility, monitoring, operational efficiency, and performance insights.â The objective is aggregation rather than creative summarization. If KkK_k is the normalized keyword set for cluster CkC_k, the final representation can be written abstractly as ak=GÏâ(Ck,Kk,R),a_k=G_Ï(C_k,K_k;R), (8) where GÏG_Ï is the generation model and R is a set of factual constraints requiring the output to remain within the information supported by CkC_k. Figure 5: Illustrative example of the recall-utility operating model. The color coding shows the aggregation of similar intent units and an outlier (yellow) added to an existing cluster (brown). 5.7 The Stable Intent Unit The output of post-processing is more than a sentence. Each cluster is stored as a traceable semantic object containing: a cluster identifier; the cluster centroid; a reference intent, typically a member closest to the centroid; the complete member-intent list; normalized keywords; the canonical product; the combined intent; the clustering method and parameters; embedding and LLM versions; and source references. This schema insulates downstream consumers from model-level implementation changes. A retrieval or recommendation system can consume the stable combined intent, while investigators can inspect the source candidates, keywords, clustering method, and model versions when a result needs to be explained or reproduced. 5.8 Illustrative Example A worked example from the post-processing design starts with 15 free-text candidate insights spanning cybersecurity, collaboration, application monitoring, data-center operations, and one application-outage statement. Because N=15N=15, the adaptive policy selects pairwise similarity clustering. The initial clustering produces four multi-member groups and one singleton. The groups correspond to cybersecurity, collaboration platform modernization, application monitoring, and data-center operations. The singleton describes reducing the impact of application outages on business operations and service delivery. Cluster-level keywords are then generated. The application-monitoring cluster, for example, is described by terms such as application visibility, application monitoring, performance insights, operational efficiency, and application security. The singleton is described by application outages, service delivery, business operations, and operational resilience. The singleton is compared against each existing cluster using both centroid similarity and keyword similarity. In the scoring process, the application-monitoring cluster provides the strongest agreement (embedding similarity 0.84 and keyword similarity 0.79), while the remaining clusters are substantially weaker. The singleton is therefore reassigned to that cluster, as shown in Figure 5. Constrained aggregation then produces four final intent statements: (1) improve application visibility, monitoring, and performance insights; (2) strengthen security posture and resilience against cyber threats; (3) build reliable, secure, high-availability data-center operations; and (4) modernize the collaboration platform for reliability, analytics, and integration. This example reduces 15 overlapping and variably worded statements to four explainable outputs while retaining the substantive themes of the candidate set. It is intended to illustrate the mechanics of the method rather than serve as a quantitative benchmark. 6 Applications of the Intent Intelligence Layer Once customer objectives have been represented as stable intent units, the representation can be reused beyond the extraction pipeline itself. In CIE, this enables a shift from identifying what a customer has explicitly expressed to using the accumulated intent corpus as a reusable knowledge layer. Two applications illustrate this transition: Machine-Generated Intents, which infer plausible objectives where direct evidence is sparse, and intent-guided semantic retrieval and mapping, which uses an intent to identify relevant entities in a downstream decision space. 6.1 Machine-Generated Intents Machine-Generated Intents (MGI) extend intent coverage to customers for whom direct source evidence is unavailable or incomplete, but for whom sufficient customer metadata and peer information exist. Instead of extracting an objective from an E2E plan, CS Console record, or another source artifact, MGI uses the intents already observed for comparable customers as evidence from which candidate objectives can be inferred. This makes MGI a content-based instance of the cold-start recommendation problem [12], in which direct interaction or intent history is missing and profile-level information is used instead to identify relevant evidence from better-observed entities. The current development flow deliberately separates peer selection from intent inference. Peer identification is treated as a transparent retrieval problem based on business attributes rather than as a learned semantic-neighbor problem: SME-defined rules select donor customers through deterministic filters and rangesâfor example, matching market segment while constraining annual recurring revenue to an appropriate rangeâwith attributes such as industry, geography, and customer segment available as additional context. This makes the peer-selection stage directly inspectable and allows the business criteria defining a relevant peer group to evolve independently of the semantic intent-processing models. The resulting peers are then mapped to the existing intent corpus so that only customers with known intent evidence contribute candidate statements. Figure 6 summarizes this flow. Figure 6: Current development flow for Machine-Generated Intents. Deterministic peer-selection criteria identify relevant donor customers, whose observed intents are subsequently post-processed and tailored to the target customer. SME feedback provides an iterative validation loop for both peer relevance and inferred intents. Once donor customers have been selected, their known intents pass through a semantic reduction stage. Related candidate intents are clustered and merged, while distinct or weakly supported objectives can remain visible rather than being forced into a dominant theme. The resulting representations are then tailored using metadata belonging to the target customer rather than the donor customers. Candidate importance can additionally be informed by the amount of peer support behind a semantic intentâfor example, whether an equivalent objective occurs across several selected peersâproviding a natural signal for prioritization without treating frequency alone as evidence of correctness. This separation is important architecturally. Deterministic business rules answer the relatively explainable question of which customers constitute useful evidence, while semantic clustering and constrained generation address the more ambiguous question of what that evidence implies for the target customer. SME feedback can therefore operate at both levels: reviewing whether the peer-selection rules produce sensible donor populations and whether the inferred intents remain plausible, relevant, and appropriately tailored. More broadly, MGI demonstrates how a normalized intent corpus can become a knowledge source rather than merely the output of an extraction system. 6.2 Intent-Guided Semantic Retrieval and Mapping A second application treats the stable intent as an intermediate representation between a userâs expressed goal and the actions, resources, products, services, or other entities that may help satisfy it. Once intent has been represented consistently, it can serve as a semantic query against a structured downstream decision space: a home-buying intent could be matched to properties, financing options, or services; a commercial intent could be matched to products, offers, or next-best actions; and an enterprise customer intent could be mapped to capabilities, adoption objectives, or business outcomes. Intent therefore becomes more than an extracted signal: it provides a reusable semantic interface between unstructured human goals and the structured actions available to an organization. We refer to this broader pattern as intent-guided semantic retrieval and mapping. Related patterns appear in other domains. Manias et al. [3] translate natural-language network intent into structured configuration, while Liu et al. [4] use inferred commercial intention to drive ad retrieval at web scale. These examples illustrate that the downstream target need not be another piece of natural language: once intent has been normalized into a stable semantic object, it can provide the query layer for a range of structured decision spaces. Within CIE, intent-to-business-outcome mapping is one concrete implementation. Each customer intent is associated with product context, and the candidate outcome space is first restricted to outcomes defined for that product. This deterministic scoping reduces the retrieval space, avoids cross-product false positives, and prevents the semantic models from being asked to distinguish candidates that are operationally irrelevant to the intent. Retrieval then follows a two-stage bi-encoder/cross-encoder architecture. A bi-encoder embeds intents and candidate outcomes independently into a shared semantic space and uses cosine similarity to retrieve a high-recall candidate set. Because outcome representations can be computed once and reused, this stage provides efficient retrieval over a comparatively large candidate space. A cross-encoder subsequently evaluates only the shortlisted intentâoutcome pairs jointly, capturing richer pairwise interactions and producing a more precise final ranking. Table 3 summarizes the respective roles, strengths, and limitations of the bi-encoder, cross-encoder, and resulting hybrid architecture. Table 3: Comparison of bi-encoder, cross-encoder, and hybrid approaches for semantic intent-to-outcome retrieval. Approach Role and Characteristics Strengths Weaknesses Bi-Encoder Used for initial candidate retrieval. Intent and outcome texts are encoded independently, enabling efficient vector-similarity search over large candidate spaces. Model selection was informed by Massive Text Embedding Benchmark (MTEB) sentence-similarity benchmarks [14]. Fast, scalable, and allows outcome embeddings to be precomputed and reused. Lower ranking precision because intentâoutcome interactions are not evaluated jointly. Cross-Encoder Applied only to shortlisted candidates. Each intentâoutcome pair is evaluated jointly, allowing richer semantic interaction and more accurate ranking. High semantic accuracy and stronger pairwise ranking. More computationally expensive because each candidate pair must be evaluated separately. Hybrid (chosen) Uses the bi-encoder for high-recall candidate retrieval and the cross-encoder for final re-ranking of the shortlisted candidates. Combines efficient retrieval with high-precision semantic ranking. Introduces slightly greater pipeline complexity than either model used independently. The hybrid design therefore concentrates the more computationally expensive pairwise reasoning where it provides the most value, following the same efficient-retrieval-then-precise-rerank pattern studied by Geigle et al. [11] and building on the sentence-encoder paradigm of Reimers and Gurevych [5]. Candidate selection is treated as a ranking problem rather than as probability estimation. Neither cosine similarity nor the cross-encoder score is assumed to be a calibrated probability, making a single fixed cutoff brittle across products and candidate distributions. Instead, the ranked bi-encoder similarities are first examined for changes between consecutive candidates. A sufficiently large score drop provides evidence that semantic relevance is beginning to degrade and can therefore be used to restrict the candidate set before final ranking. As shown in Figure 7, the cross-encoder can reorder the shortlisted outcomes when pairwise semantic evaluation identifies a more plausible ranking. The final output ordering follows this cross-encoder ranking, while the displayed relative similarity values originate from the bi-encoder and are adjusted, where necessary, to remain monotonically decreasing with the final rank. The number of returned outcomes is also bounded to prevent noisy long tails; in the current implementation, no more than five business outcomes are returned. Figure 7: Illustration of hybrid ranking and candidate-selection logic. Bi-encoder similarity provides efficient candidate retrieval and score-gap information used to trim the candidate set before reranking, while the cross-encoder determines the final semantic ranking of the shortlisted outcomes. Table 4 details the governing parameters. Table 4: Scoring and thresholding rules applied during intent-to-outcome candidate selection and ranking. Mechanism Rule Score-gap detection Ranked bi-encoder similarities are scanned for a sufficiently large drop between consecutive candidates; a detected drop is treated as evidence that semantic relevance is beginning to degrade and is used to restrict the candidate set before reranking, rather than relying on a single fixed similarity cutoff. Maximum outcomes returned No more than five business outcomes are returned per customer intent, bounding the output to prevent noisy long tails. Final ordering Determined by the cross-encoder ranking over the shortlisted candidates, not by the bi-encoder similarity used for initial retrieval. Displayed similarity scores Derived from the bi-encoder scores and adjusted, where necessary, to decrease monotonically with the final cross-encoder rank, so that displayed values remain consistent with the output ordering. The hybrid design was selected empirically against bi-encoder-only, cross-encoder-only, and zero-shot classification alternatives. Table 5 summarizes the overall performance on the SME-annotated evaluation set using F1 at rank 5 (F1@5) and Mean Reciprocal Rank (MRR) as the primary ranking-quality metrics, alongside latency. Table 5: Overall comparison of intent-to-outcome mapping approaches. Higher is better for F1 and MRR; lower is better for latency. Approach F1@5 MRR Latency (s) Bi-encoder 51.09 0.76 0.056 Cross-encoder 54.82 0.87 0.098 Hybrid 54.82 0.87 0.060 Zero-shot classifier 47.00 0.85 1.213 The hybrid pipeline matched the strongest F1@5 and MRR performance, with inference latency close to the bi-encoder-only approach. The cross-encoder-only configuration produced similar ranking quality at higher latency, while the zero-shot classifier was substantially slower. Because the SME annotations were not exhaustive, these measurements are used comparatively to guide architecture selection rather than as an absolute estimate of production accuracy. The resulting application illustrates the broader value of the intent intelligence layer: the same normalized objective that supports consolidation and analysis can also function directly as a query for downstream retrieval, recommendation, and decision-support systems. 7 Evaluation and Feedback Loops CIE treats evaluation as part of the architecture rather than as a final acceptance test. Because the pipeline contains deterministic transformations, semantic models, generative components, and human-facing outputs, no single metric is sufficient. Evaluation therefore combines repeatability analysis, automated model-based evaluation, and SME review, with different forms of evidence applied to different stages of the system. For multimodal E2E extraction, one concern is whether repeated executions produce a stable representation of the same source material. A repeatability experiment was conducted by generating the predicted structured output for the same sampled E2E plan across five runs and comparing the extracted table columns and cells using embedding similarity. The experiment produced a mean similarity of 95.19%95.19\% with a standard deviation of 1.12%1.12\%, with individual runs ranging from 94.07%94.07\% to 96.31%96.31\%. These results indicate relatively low run-to-run variation for the sampled document. Importantly, repeatability measures stability rather than correctness: a consistently reproduced output may still omit or misinterpret source information, so this measure is considered alongside ground-truth and SME-based evaluation. Automatic evaluation was also applied to intent extraction from E2E-sourced evidence, using an independent LLM as a judge, following the LLM-as-a-judge methodology of Zheng et al. [10]. This targets a different stage of the pipeline than the repeatability experiment above: rather than the fidelity of structured context recovery, it assesses the candidate intents generated from that context. Gemini 2.5 Flash [15] was used as the evaluator at temperature zero, with each metric averaged across ten runs. Table 6 summarizes the reported results. Table 6: Automatic evaluation of intent extraction from E2E-sourced evidence, using an independent LLM judge. Values are the mean and standard deviation reported across ten evaluation runs. Metric Mean Standard deviation Interpretation Faithfulness 93.5% 7.9% Consistency of the generated response with the retrieved source context. Faithfulness vs. ground truth 66.1% 31.9% Faithfulness when evaluated relative to the manually established reference output. Semantic similarity 48.5 10.7 Semantic resemblance between the generated output and the reference answer. Answer correctness 37.4% 14.4% Combined assessment of semantic and factual agreement with the reference answer. The difference between high context faithfulness and the lower ground-truth-based metrics is informative. It suggests that an output can remain well grounded in the context supplied to the model while still differing materially from a manually defined reference representation. In document-processing pipelines, this distinction is useful because errors can originate either in the generation stage itself or earlier, through incomplete or imperfect extraction of the source context. The relatively high variance of the ground-truth faithfulness measure also reinforces the need to consider aggregate automatic scores together with example-level inspection rather than as a single acceptance threshold. Human evaluation provides an additional layer of evidence. During a pilot evaluation of intent extraction, SMEs reviewed inferred intents for five customers and supplied positive or negative feedback at the intent level. Across these customers, average intent-extraction accuracy was 78%78\%. A separate outcome-benchmarking evaluation collected feedback for 20 customers spanning four products. SMEs judged all recommended business outcomes to be relevant to their respective customers, while 81.6%81.6\% of the selected peers were judged relevant. The lower peer-relevancy result was attributed primarily to occasional inaccuracies in the customer industry information used during peer grouping. Table 7 summarizes these results. Table 7: Summary of SME-based evaluation results. Evaluation target Evaluation scope Result Interpretation Intent extraction Five customers 78% average accuracy SMEs assessed whether individual inferred intents were supported and relevant for each customer. Outcome relevancy 20 customers across four products 100% SMEs confirmed that all recommended business outcomes were relevant to the respective customers. Peer relevancy 20 customers across four products 81.6% Most selected peers were judged relevant; remaining errors were associated primarily with inaccuracies in customer industry metadata used for peer grouping. These evaluations reflect the broader principle that different stages require different forms of evidence. Multimodal context extraction can be evaluated against manually validated structured ground truth using cell/key similarity, edit distance, structural correctness, and repeatability. Intent extraction can be assessed for faithfulness to retrieved context, semantic and factual correctness against SME-validated references, and stability across repeated runs. Post-processing requires additional cluster-focused measures such as purity, singleton rate, cluster count, outlier reassignment acceptance, keyword coverage and distinctiveness, aggregation faithfulness, and information preservation. Downstream applications introduce further task-specific measures, such as peer relevance for MGI and ranking quality for intent-to-outcome retrieval. The development process also showed that SME feedback itself requires controls. Directly written SME intents did not always align with the source artifact, product scope, or the operational intent guidelines. A later validation round, in which SME-provided statements were rewritten according to the intent guidelines and then reviewed, produced substantially better qualitative agreement. This motivates a feedback design in which SMEs validate evidence-grounded candidate representations rather than acting as an assumed infallible source of truth. Taken together, these evaluation layers make feedback part of the systemâs iterative development loop. Automated metrics provide repeatable signals at scale, repeatability testing exposes instability in stochastic components, and SME review tests whether technically plausible outputs remain meaningful in the business context. Evaluation is therefore used not only to quantify final performance, but also to identify which component of the pipeline should be refined when failures occur. 8 Discussion Three engineering lessons emerge from the work. First, customer intent extraction is primarily a data and representation problem rather than only a model-selection problem. Large models can produce fluent candidates, but much of the practical quality improvement comes from source-specific filtering, structured context recovery, canonical metadata, semantic grouping, and constrained aggregation. The post-processing representation determines whether model outputs are reusable. Second, deterministic and probabilistic techniques should be assigned to the problems they solve best. Regex filtering, exact deduplication, and canonical mappings handle explicit data-quality rules. Embeddings and clustering handle semantic relatedness. LLMs are most useful where language understanding, semantic descriptor generation, or constrained synthesis is required. Treating every transformation as an LLM call would be harder to debug, more expensive, and less reproducible. Third, ambiguity should remain visible. A singleton is not automatically an error; it may encode a rare but legitimate customer objective. Similarly, a short candidate may be ambiguous in isolation but meaningful when paired with known product or domain metadata. The architecture therefore adds context for semantic computation without overwriting the original text, and it allows uncertain items to remain independent. More broadly, the system demonstrates a reusable pattern for LLM-based information extraction beyond customer intent. Any workflow that deliberately extracts a high-recall set of candidate insights from large heterogeneous corpora can face the same downstream problem: semantic duplication, inconsistent granularity, noise, and an output volume that is difficult to consume. A reduction layer that preserves provenance and uncertainty can therefore be treated as a first-class architectural component rather than a final formatting step [13]. 9 Limitations The current article describes a system architecture and development evidence rather than a completed controlled benchmark. The adaptive clustering thresholds were selected as conservative operational starting points and are not universal boundaries. They should be recalibrated for new embedding models, candidate-set distributions, and business domains. The large-set path also inherits sensitivity from dimensionality reduction and density-based clustering, while the small-set graph path can exhibit chaining when similarity thresholds are too permissive. Keyword generation and constrained aggregation still depend on an LLM. Although the prompt explicitly restricts generation to the information present in cluster members, factuality must be guardrailed and evaluated rather than assumed. Similarly, metadata padding can improve representation of underspecified text but may also overemphasize a product or domain label if the metadata itself is incorrect. Finally, the article currently focuses on English-language enterprise data and on customer-intent use cases within the described environment. Broader validation is required before claiming that the same thresholds, prompts, or clustering choices transfer unchanged to other domains. 10 Conclusion Customer intent extraction is not a single inference call. It is a controlled transformation from heterogeneous customer evidence to reusable business intelligence. The architecture described in this paper deliberately separates high-recall extraction from high-utility post-processing: it first preserves potentially relevant objectives, then applies deterministic normalization, metadata-aware semantic representation, adaptive clustering, explainable keyword generation, conservative outlier handling, and constrained aggregation. The resulting stable intent unit is both concise and auditable. Downstream systems receive a normalized statement, while the underlying member intents, source references, clustering decisions, keywords, and model versions remain available for explanation and reprocessing. This design makes semantic reduction a first-class component of enterprise LLM pipelines and provides a foundation for predictive intent generation, intent-guided semantic retrieval and mapping, and quality evaluation. References [1] X. Cheng, K. Zeng, Z. Cao, L. Dai, W. Gao, F. Han, A. Jian, F. Hong, W. Hu, Z. Huang, D. Kong, J. Leng, Z. Liao, P. Liu, J. Lin, X. Ma, J. Ruan, J. Song, X. Tan, R. Xiao, W. Yu, W. Zhan, H. Zhang, C. Zhou, H. Zhou, et al. (2025) Higher satisfaction, lower cost: a technical report on how LLMs revolutionize Meituanâs intelligent interaction systems. External Links: 2510.13291, Link Cited by: §1, §2.1. [2] J. Corbeil, M. T. Li, and H. A. Ghavidel (2022) Building the intent landscape of real-world conversational corpora with extractive question-answering transformers. External Links: 2208.12886, Link Cited by: §2.1. [3] D. M. Manias, A. Chouman, and A. Shami (2024) Towards intent-based network management: large language models for intent extraction in 5G core networks. In Proceedings of the International Conference on the Design of Reliable Communication Networks (DRCN), External Links: 2403.02238, Link Cited by: §2.1, §6.2. [4] T. Liu, Z. Wang, M. Qin, Z. Lu, X. Chen, Y. Yang, and P. Shu (2025) Real-time ad retrieval via LLM-generative commercial intention for sponsored search advertising. External Links: 2504.01304, Link Cited by: §2.1, §6.2. [5] N. Reimers and I. Gurevych (2019) Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: 1908.10084, Link Cited by: §2.2, §2.4, §6.2. [6] L. McInnes, J. Healy, and J. Melville (2018) UMAP: uniform manifold approximation and projection for dimension reduction. External Links: 1802.03426, Link Cited by: §2.2, §5.3. [7] L. McInnes and J. Healy (2017) Accelerated hierarchical density clustering. External Links: 1705.07321, Link Cited by: §2.2, §5.3. [8] A. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos (2023) SemDeDup: data-efficient learning at web-scale through semantic deduplication. External Links: 2303.09540, Link Cited by: §2.2, §5.1. [9] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu (2023) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. External Links: 2311.05232, Link Cited by: §2.3, §5.6. [10] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2306.05685, Link Cited by: §2.3, §7. [11] G. Geigle, J. Pfeiffer, N. Reimers, I. VuliÄ, and I. Gurevych (2022) Retrieve fast, rerank smart: cooperative and joint approaches for improved cross-modal retrieval. Transactions of the Association for Computational Linguistics (TACL). External Links: 2103.11920, Link Cited by: §2.4, §6.2. [12] M. Viljanen, J. Vahlo, A. Koponen, and T. Pahikkala (2020) Content based player and game interaction model for game recommendation in the cold start setting. External Links: 2009.08947, Link Cited by: §2.4, §6.1. [13] Z. Aminiranjbar, J. Tang, Q. Wang, S. Pant, and M. Viswanathan (2025) DAWN: designing distributed agents in a worldwide network. IEEE Access 13. External Links: Document Cited by: §2.4, §8. [14] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers (2023) MTEB: massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), External Links: 2210.07316, Link Cited by: Table 3. [15] Gemini Team (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. External Links: 2507.06261, Link Cited by: §7.