Paper deep dive

Filling the Gaps: Selective Knowledge Augmentation for LLM Recommenders

Jaehyun Lee, Sanghwan Jang, SeongKu Kang, Hwanjo Yu

Year: 2026Venue: arXiv preprintArea: cs.IRType: PreprintEmbeddings: 74

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 4/10/2026, 4:11:07 AM

Summary

The paper introduces KnowSA_CKP, a training-free framework for LLM-based recommender systems that addresses the 'knowledge gap' problem—the uneven distribution of item knowledge in LLMs. Instead of uniform augmentation, which wastes context and can degrade performance, KnowSA_CKP uses Comparative Knowledge Probing (CKP) to estimate an LLM's internal knowledge of items based on collaborative patterns. It then selectively injects external information only for items where the model lacks sufficient knowledge, improving both recommendation accuracy and context efficiency.

Entities (4)

KnowSA_CKP · framework · 100%LLM · technology · 100%Comparative Knowledge Probing · methodology · 98%Knowledge Gap Problem · problem · 95%

Relation Signals (3)

Comparative Knowledge Probing → iscomponentof → KnowSA_CKP

confidence 100% · we propose KnowSA_CKP (Knowledge-aware Selective Augmentation with Comparative Knowledge Probing)

KnowSA_CKP → mitigates → Knowledge Gap Problem

confidence 95% · we propose KnowSA_CKP (Knowledge-aware Selective Augmentation with Comparative Knowledge Probing) to mitigate the knowledge gap problem.

KnowSA_CKP → improves → Recommendation Accuracy

confidence 90% · KnowSA_CKP requires no fine-tuning step, and consistently improves both recommendation accuracy and context efficiency

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) have recently emerged as powerful training-free recommenders. However, their knowledge of individual items is inevitably uneven due to imbalanced information exposure during pretraining, a phenomenon we refer to as knowledge gap problem. To address this, most prior methods have employed a naive uniform augmentation that appends external information for every item in the input prompt. However, this approach not only wastes limited context budget on redundant augmentation for well-known items but can also hinder the model's effective reasoning. To this end, we propose KnowSA_CKP (Knowledge-aware Selective Augmentation with Comparative Knowledge Probing) to mitigate the knowledge gap problem. KnowSA_CKP estimates the LLM's internal knowledge by evaluating its capability to capture collaborative relationships and selectively injects additional information only where it is most needed. By avoiding unnecessary augmentation for well-known items, KnowSA_CKP focuses on items that benefit most from knowledge supplementation, thereby making more effective use of the context budget. KnowSA_CKP requires no fine-tuning step, and consistently improves both recommendation accuracy and context efficiency across four real-world datasets.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

73,325 characters extracted from source content.

Expand or collapse full text

Filling the Gaps: Selective Knowledge Augmentation for LLM Recommenders Jaehyun Lee 0009-0002-1232-542X Pohang University of Science and TechnologyPohangSouth Korea jminy8@postech.ac.kr , Sanghwan Jang 0009-0000-9856-491X Pohang University of Science and TechnologyPohangSouth Korea s.jang@postech.ac.kr , SeongKu Kang 0000-0001-5528-1426 Korea UniversitySeoulSouth Korea seongkukang@korea.ac.kr and Hwanjo Yu 0000-0002-7510-0255 Pohang University of Science and TechnologyPohangSouth Korea hwanjoyu@postech.ac.kr (2018) Abstract. Large language models (LLMs) have recently emerged as powerful training-free recommenders. However, their knowledge of individual items is inevitably uneven due to imbalanced information exposure during pretraining, a phenomenon we refer to as knowledge gap problem. To address this, most prior methods have employed a naive uniform augmentation that appends external information for every item in the input prompt. However, this approach not only wastes limited context budget on redundant augmentation for well-known items but can also hinder the model’s effective reasoning. To this end, we propose (Knowledge-aware Selective Augmentation with Comparative Knowledge Probing) to mitigate the knowledge gap problem. the LLM’s internal knowledge by evaluating its capability to capture collaborative relationships and selectively injects additional information only where it is most needed. By avoiding unnecessary augmentation for well-known items, on items that benefit most from knowledge supplementation, thereby making more effective use of the context budget. no fine-tuning step, and consistently improves both recommendation accuracy and context efficiency across four real-world datasets. Our code will be made publicly available upon publication. Recommender System, Large Language Models, Knowledge Gap †copyright: acmlicensed†journalyear: 2018†doi: .†conference: ; June 03–05, 2018; Woodstock, NY†isbn: 978-1-4503-X-X/2018/06†ccs: Information systems Recommender systems 1. Introduction Recommender systems play a crucial role in helping users navigate the ever-growing landscape of digital content and products. Classical approaches such as matrix factorization (Rendle et al., 2012) rely heavily on historical interaction data to infer user preferences. However, these methods struggle in cold-start scenarios involving users or items with limited or no historical interactions, which severely restricts the system’s ability to generalize. To overcome these challenges, recent studies have explored the use of large language models (LLMs) as knowledge-rich recommenders (Hou et al., 2024; Kim et al., 2024; Luo et al., 2024; Liao et al., 2023; Ren et al., 2024). Pretrained on vast corpora, LLMs possess broad semantic understanding and factual knowledge about a wide range of entities and domains. This allows them to serve as powerful training-free recommenders without additional costly fine-tuning steps (Liang et al., 2024; Hou et al., 2024; Yue et al., 2023). Despite their potential, one critical challenge in using LLMs for recommendation is the imbalance in their parametric knowledge, a phenomenon we refer to as the knowledge gap. As LLMs are pretrained using texts from the web, the amount of information they acquire about each item is inherently biased toward popular items with higher visibility online (Mallen et al., 2022; Yu et al., 2023; Kang and Choi, 2023), leaving the model with insufficient knowledge of long-tail items. However, in the recommendation context, popularity alone serves as an imperfect indicator for this knowledge gap. This is because an effective recommendation requires more than merely knowing details about individual items; it demands capturing their collaborative patterns, such as co-consumption behaviors in user histories and item-item relationships. Even among items with similar popularity, the extent to which an LLM acquire their relational context can differ substantially—one item might be associated with rich interaction patterns, while another exists in isolation. Due to this uneven knowledge, LLMs often over-rely on knowledge-rich items and fail to provide personalized recommendations. Despite its critical impact, little effort has been made to directly quantify and resolve this knowledge gap. While existing studies have not explicitly addressed this knowledge gap, many efforts can be seen as indirect attempts to alleviate it. One straightforward approach is model-level adaptation, where the LLM is fine-tuned using user-item interaction data to directly learn collaborative patterns (Bao et al., 2023; Zhang et al., 2025, 2024b; Liao et al., 2023; Harte et al., 2023). However, this approach incurs substantial computational costs and risks degrading the model’s general capabilities, such as instruction following and explainability (Hu et al., 2022; Yang et al., 2024a). Consequently, recent research has increasingly focused on prompt-level augmentation, which enriches input prompts with item metadata or external knowledge in a training-free manner (Lin et al., 2024b; Kim et al., ; Kim et al., 2025; Zheng et al., 2024). However, there remains substantial room for improvement in this direction. Most existing methods adopt uniform augmentation, indiscriminately adding information for all items in the input prompt, regardless of how much the model already knows about each item. This approach is suboptimal; it not only wastes the limited context budget by adding redundant information for known items, while simultaneously increasing the risk of performance degradation as LLMs struggle to interpret essential signals within excessively long contexts (Liu et al., 2023). To resolve this inefficiency, one might consider adopting adaptive retrieval strategies from the general NLP domain (Jiang et al., 2023; Su et al., 2024), which dynamically determine when to retrieve external information. However, directly applying these techniques to recommendation presents critical challenges. Existing adaptive methods monitor queries sequentially during generation to detect knowledge deficiency. Such an inference-time approach is ill-suited for recommendation scenarios, where minimizing inference latency. Moreover, LLMs struggle to accurately discriminate knowledge gaps for multiple items simultaneously within a complex prompt. To overcome these limitations, we propose a selective augmentation framework grounded in offline knowledge estimation. By pre-computing the knowledge necessity for each item, we can efficiently inject external information only where it is needed without incurring inference overhead. This naturally raises the question of how to quantify the degree of knowledge that an LLM possesses for each item. We begin by analyzing various knowledge proxies used to approximate the model’s knowledge, ranging from heuristics like item popularity (Wang et al., 2025) to advanced signals such as generation likelihood (Shi et al., 2023) and consistency metric (Lin et al., 2024d; Yao et al., 2025). However, directly applying these general-purpose methods to recommendations yields suboptimal results. As detailed in our analysis section 3.2.2, these proxies correlate poorly with recommendation accuracy because they primarily assess only semantic familiarity (e.g., knowing what an item is). They do not effectively capture the collaborative relationship between the user’s history and the target item (e.g., co-consumption pattern), which is essential for recommendation. These observations highlight the need for a recommendation-tailored knowledge scoring strategy that accounts for both semantic and collaborative aspects, enabling more targeted and effective prompt augmentation. In this work, we propose (Knowledge-aware Selective Augmentation with Comparative Knowledge Probing), a training-free framework designed to mitigate item-level knowledge gaps in LLM-based recommendation. designed to (1) estimate the LLM’s knowledge for each item, and (2) selectively inject additional information only where it is most needed. To this end, we introduce a new knowledge scoring strategy, called CKP, which quantifies the model’s capability to comparatively rank items based on collaborative patterns. Guided by this score, we employ a personalized augmentation strategy that selectively enriches knowledge-poor items with relevant reference anchors to bridge the semantic gap, effectively activating latent knowledge already encoded in the model. Unlike prior methods that uniformly augment all items, more efficient use of the context budget by focusing on items that benefit most from supplementation. The paper makes the following key contributions: • Problem. We formally present the necessity and difficulty of resolving the knowledge gap problem in LLM-based recommendation, which remains less explored in the previous literature. • Analysis. We provide a comprehensive analysis of various knowledge proxies for estimating an LLM’s knowledge in recommendation tasks, shedding light on the design of an effective solution. • Algorithm. We propose , which is equipped with new knowledge scoring and augmentation strategies tailored to recommendation tasks. As a plug-and-play framework, it can be flexibly applied to various LLMs. • Experiments. Extensive experiments show that improves recommendation quality in both accuracy and diversity, with negligible additional latency. 2. Preliminaries Notations. Let the dataset be represented as =(,ℐ,,ℋ)D=(U,I,T,H), where U, ℐI, T, and ℋH represent the sets of users, items, item attribute texts, and user interaction histories, respectively. For each user u∈u , the interaction sequence is denoted as Hu=(i1u,i2u,…,i|Hu|u)∈ℋH^u=(i^u_1,i^u_2,…,i^u_|H^u|) , where |Hu||H^u| is the length of user u’s interaction sequence, and iku∈ℐi^u_k is the k-th item user u interacted with chronologically. Each item i∈ℐi is associated with textual features, represented as (ti,ai)∈(t^i,a^i) , where tit^i is the item’s title and aia^i are additional textual attributes (e.g., genre, developer, and description). Recommendation with an LLM ranker. We adopt the standard two-stage recommendation (Covington et al., 2016; Kang and McAuley, 2019), consisting of candidate generation followed by ranking. In the first stage, a small set of candidate items Cu=i1u,i2u,…,imuC^u=\i_1^u,i_2^u,…,i_m^u\ is retrieved from the full item set ℐI using lightweight models, where m≪|ℐ|m |I|. In the second stage, we employ LLMs as training-free rankers (Hou et al., 2024; Wei et al., 2021), which orders the candidate items based on its parametric knowledge and the input prompt. The prompt is constructed using the user’s history HuH^u, the candidate set CuC^u, and their associated features from T. Problem Definition. We follow the two-stage recommendation, where the LLM is leveraged as a training-free ranker. Each item is represented by its title as the simplest form of input, and can be further augmented using additional attributes (Lin et al., 2024a, b). Our goal is to develop a plug-and-play framework that mitigates the knowledge gap problem by selectively augmenting input prompts. We seek to estimate the LLM’s internal knowledge for each item and selectively inject additional information only where it is most needed. By avoiding redundant augmentation for well-known items, the framework can utilize the context budget more effectively by allocating it to items that benefit most from knowledge supplement. Figure 1. Recommendation performance change with uniform augmentation for all items. Indiscriminate augmentation for all items can rather hurt performance. (a) Spearman correlation (b) Performance Alignment across Knowledge Levels Figure 2. Alignment between knowledge scores and recommendation quality on the A-Beauty dataset. (a) Spearman correlation between Recall@1 and knowledge scores obtained from five methods. (b) Average Recall@1 across four quantile bins grouped by knowledge scores at the item (left) and user (right) levels. Group 1 represents the lowest knowledge score bin, and Group 4 is the highest. The proposed knowledge scoring method consistently aligns with recommendation performance across all perspectives. 3. Analysis: Knowledge Scoring for LLM-based Recommendation Before presenting our framework, we empirically investigate the nature of the knowledge gap in LLM-based recommendation. Specifically, we address two fundamental questions: (1) Does simply injecting external information for all items resolve the knowledge deficit? (2) If selective augmentation is needed, can we rely on existing knowledge proxies to diagnose what the model knows? 3.1. The Paradox of Uniform Augmentation We examine the impact of uniform augmentation, where full item attributes are indiscriminately added for every item in the input prompt. Figure 1 illustrates the performance difference between uniform augmentation and no augmentation (i.e., each item is represented by its title, the simplest input form) across four datasets. Despite the richer input, we observe notable performance drops on three datasets, suggesting that uniform augmentation is not universally beneficial and may even degrade recommendation accuracy. Since uniform augmentation indiscriminately appends information even for items the model already knows, it leads to two critical issues. First, LLMs face known difficulties in interpreting information within extended contexts (Liu et al., 2023; Hou et al., 2024). Consequently, augmented knowledge is often overlooked, making it difficult for the model to effectively utilize the provided information even for less-known items. Second, the excessive prompt length substantially increases inference costs (e.g., API fees) and latency. These observations underscore the necessity of selective augmentation: injecting information only where the model’s internal knowledge is insufficient. 3.2. The Misalignment of Existing Proxies To implement selective augmentation, it is essential to estimate the degree of knowledge that an LLM possesses for each item, a process we refer to as knowledge scoring. We begin by evaluating whether existing proxies, ranging from simple heuristics to advanced adaptive RAG metrics, can serve as reliable indicators for this purpose. 3.2.1. Existing Proxies We consider four categories of proxies: • Popularity refers to the item frequency in the interaction dataset. This is a common heuristic for item exposure, similar to Wikipedia pageviews used in general domains (Mallen et al., 2022; Wang et al., 2025). • Pre-training Data Detection estimates whether an item was present in the LLM’s pre-training corpus (Carlini et al., 2021; Zhang et al., 2024a; Shi et al., 2023). We adopt Min-K% (Shi et al., 2023), which computes the average log-probability over the bottom k%k\% of tokens in the generated item title, conditioned on the target domain (e.g., In the movie domain, the title is:). • Uncertainty measures the model’s internal confidence in LLM predictions (Fomicheva et al., 2020; Duan et al., 2024; Fadeeva et al., 2024; Lin et al., 2024d; Xia et al., 2025). We utilize EigValLaplacian (Lin et al., 2024d), which computes the sum of Laplacian eigenvalues from a weighted graph constructed based on the semantic similarity of multiple sampled responses (e.g., descriptions generated for the movie ”Titanic”). • Adaptive Retrieval Score represents the self-awareness of the LLM regarding its need for external information in Adaptive RAG research (Jiang et al., 2023; Su et al., 2024; Jeong et al., 2024; Yao et al., 2025). We employ the scoring strategy from SeaKR (Yao et al., 2025), which measures the consistency of internal states across multiple responses generated from a query requesting an item description, where a lower score reflects a higher need for retrieval. 3.2.2. Empirical Observations To serve as a reliable indicator of LLM’s competence for recommendation, the proxy should be closely correlated with the recommendation performance. We report the result with Qwen-2.5-7B with no prompt augmentation (i.e., each item is represented by its title, the simplest input form). Similar tendencies are observed with other models. Our findings reveal two key observations: O1. Existing proxies correlate weakly with ranking performance. We first evaluate whether the proxies align with actual recommendation quality by computing the Spearman correlation between each proxy score and the model’s recall (Recall@1). For a comprehensive analysis, we analyze the correlations at both item- and user-levels.111For item-level, we compute the correlation across all test items. For user-level, we compute the average proxy score over the items in each user’s history and correlate it across all users. The results are presented in Figure 2(a). Interestingly, we observe that the simplest heuristic, popularity, exhibits the highest correlation, whereas sophisticated, recent techniques yield weak correlations. • Popularity: As a heuristic derived solely from dataset statistics, popularity acts as an external signal that does not necessarily align with the LLM’s actual exposure during pretraining. Therefore, it cannot capture the full complexity of the model’s internal knowledge, underscoring the need for a model-centric estimation that directly reflects the LLM’s prediction behaviors. • Min-K%: Since this method relies on token-level likelihoods of item titles, it is highly susceptible to surface-level biases. Item titles are typically short and lack sufficient context (Liang et al., 2024), which leads to two major biases: (i) Lexical Bias, where high likelihood may merely reflect familiarity with common words (e.g., ”New York”) rather than genuine understanding of the item, and (i) Length Bias, where longer titles tend to receive lower total likelihoods due to the greater accumulation of negative log-probabilities, thereby penalizing those items regardless of the model’s knowledge. • EigValLaplacian and SeaKR: Since these methods rely on generating item descriptions, they are inherently sensitive to prompt design. Furthermore, the excessive length of generated descriptions incurs prohibitive computational costs, limiting scalability across large item catalogs. Most critically, LLMs frequently exhibit unwarranted confidence by producing fluent descriptions even for completely unknown concepts (Amayuelas et al., 2023), which yields unreliable signals as the model fails to distinguish between genuine knowledge and hallucinated familiarity. Critically, all of these proxies focus on measuring item-specific knowledge in isolation. Consequently, this approach fails to capture collaborative patterns such as co-consumption behaviors that are critical for recommendation. Furthermore, given that the target domain knowledge represents a tiny fraction of the LLM’s vast open-world knowledge, querying it without sufficient context often fails to activate the relevant information. O2. Existing proxies fail to reflect group-level differences in model competence. For a more in-depth understanding, we group target items and users into four quantile bins based on their knowledge scores and compute the average Recall@1 within each group. A reliable proxy should exhibit a monotonic increase in performance across groups (i.e., higher knowledge scores → higher recall). As shown in Figure 2(b), the existing proxies yield inconsistent performance across bins, failing to reveal a meaningful relationship between knowledge and performance. In contrast, our proposed method (CKP) demonstrates a clear monotonic increase, confirming its effectiveness as a reliable knowledge indicator. Summary. Our analysis shows that (1) uniform augmentation is suboptimal for mitigating the knowledge gap and improving recommendation quality, and (2) existing proxies used in other domains are not suitable for knowledge scoring in recommendation tasks. 4. Methodology We introduce , a selective augmentation framework with comparative knowledge probing. It consists of two main stages: (1) knowledge scoring that estimates the LLM’s knowledge on each item for recommendation (section 4.1), and (2) selective augmentation that enriches the knowledge for lesser-known items (section 4.2). 4.1. Knowledge Scoring for Recommendation Our goal is to develop a knowledge scoring strategy tailored to LLM-based recommendations. We adopt a likelihood-based approach, inferring the LLM’s knowledge based on the likelihood it assigns to an item given interaction contexts. 4.1.1. Overview For each item t, we construct its interaction context windows tW_t from interaction histories ℋH. Each window w∈tw _t is a sequence of items preceding t, providing its contexts. Ideally, the knowledge score should be aggregated over the entire set tW_t to fully capture the item’s contextual patterns. However, as this entails excessive computational costs, we employ a popularity-stratified sampling strategy to obtain a representative subset ~t⊂t W_t _t. Specifically, we partition tW_t into three quantile bins based on the average popularity of items in each window, and then uniformly sample from each bin. This strategy ensures computational efficiency while preserving a balanced coverage of diverse interaction patterns. For each window, we build a prompt wP_w and measure the LLM’s generation preference as a conditional probability, PLLM(t∣w)P_LLM(t _w). The item-level knowledge score K(t)K(t) is defined as the average of these probabilities over all sampled windows: (1) K(t):=1|t~|∑w∈t~PLLM(t∣w)K(t):= 1| W_t| _w∈ W_tP_LLM(t _w) The key challenges lie in the design of: (i) the prompt wP_w, including both instruction formulation and output structure, and (i) the likelihood function PLLM(t∣w)P_LLM(t _w), particularly how to extract a reliable knowledge proxy from the generated output. Our design is guided by three desiderata for recommendation-tailored scoring: (1) Interaction-based contextualization: The scoring should reflect the LLM’s behavior conditioned on specific interaction contexts, rather than on the coarse-grained conditions (e.g., domain). (2) Ranking-oriented scoring: As recommendation is inherently a ranking task that orders probable items, the scoring should capture the LLM’s ability to assign appropriate ranks. (3) Robustness to surface-level bias: The scoring should be robust against the inherent generation biases of LLMs (e.g., lexical preference, output length), as discussed in section 3.2.2. Guided by these desiderata, we introduce two knowledge scoring methods: a baseline Direct Knowledge Probe (DKP), and a more advanced Comparative Knowledge Probe (CKP). 4.1.2. Naive Approach: Direct Knowledge Probe (DKP) As the simplest instantiation, DKP estimates the generation probability of the item title, given its interaction window. The prompt wDKPP^DKP_w includes a window w and an instruction for next item prediction, e.g., “Given w\w\, the next item is:”. The likelihood is obtained by aggregating the token probabilities of the item title y=(y1,…,yL)y=(y_1,…,y_L): (2) PLLM(t∣wDKP)=exp⁡(∑j=1Llog⁡P(yj∣wDKP,y<j))P_LLM(t ^DKP_w)= ( _j=1^L P(y_j ^DKP_w,y_<j) ) Substituting PLLMP_LLM into Eq. 1 yields the item-level knowledge score for DKP. While DKP satisfies the first desideratum by leveraging user interaction context, it fails the others: it considers each item separately, making it less aligned with the ranking-oriented nature of recommendation. Also, it remains susceptible to surface-level biases, which can inflate scores for item titles containing verbose or frequently occurring tokens. To overcome these limitations, we introduce our proposed method, CKP. 4.1.3. Comparative Knowledge Probe (CKP) Our key idea to meet the remaining two desiderata is to reframe the scoring task as a relative comparison problem based on content-neutral identifiers. Fine-grained comparison set. For each item t, we construct a comparison set C(t)C(t), consisting of t and several distractor items. The LLM will be instructed to rank this set given the interaction window w. Compared to querying each item independently, this approach is better aligned with the ranking nature of recommendation. To achieve this, we employ a hybrid sampling strategy that combines random and semantic distractors. First, we include n random distractors (randC_rand), sampled uniformly from the item set ℐI. These serve as easy cases, providing item diversity to ensure the model maintains the capability to discriminate the target from irrelevant noise. (3) rand(t)∼UniformSample(ℐ∖(t∪w),n)C_rand(t) (I (\t\∪ w),\,n ) Second, we include m semantic distractors (semC_sem)—items that are semantically similar to t but not valid for the current context. These serve as hard negatives designed to mitigate the model’s tendency to overestimate its knowledge. By providing these plausible alternatives, we force the model to demonstrate fine-grained reasoning: it allows the model to concentrate probability on the target only when it possesses sufficient discriminative knowledge to reject these semantically similar but contextually incorrect items. They are selected based on the cosine similarity of text embeddings e, derived from item titles and attributes:222As the simplest choice, we use Sentence-BERT (Reimers and Gurevych, 2019). (4) sem(t)=Top−mj∈ℐ∖(t∪w)(cos⁡(t,j))C_sem(t)=Top-m_j (\t\∪ w) ( (e_t,e_j) ) The final set is C(t)=t∪rand(t)∪sem(t)C(t)=\t\ _rand(t) _sem(t). This hybrid design balances item diversity and ranking difficulty, providing comparison sets that are comprehensive and challenging. Identifier-based top‑1 estimation. Given the comparison set, a standard scoring method is to instruct the LLM to generate a ranked list of item titles provided in the prompt. The probability of placing the target item t (i.e., the true next item) near the top of the list can then serve as its knowledge score. However, such an approach has two limitations. First, as discussed earlier, it is susceptible to surface-level biases arising from the token composition of item titles. Second, the ranking process itself suffers from a structural flaw due to the autoregressive nature of generation: items ranked later in the list are selected from a smaller remaining pool, leading to inflated probabilities simply because fewer alternatives remain.333For instance, in a set of three items, the probability of the last-ranked item is calculated from a pool of only one remaining option, artificially inflating its score We propose a simple yet effective solution by re-designing both the prompting scheme—the instruction and output structure—and the likelihood function. Specifically, we randomly shuffled the elements in C(t)C(t)444This randomization mitigates positional bias (Hou et al., 2024), ensuring that the model’s selection is driven by content rather than the order of presentation. and assign content-neutral identifiers (e.g., [A], [B]) to each item. These identifiers are decoupled from the original item titles, effectively removing surface-level biases while remaining computationally efficient. Then, we instruct the LLM to pinpoint the single most preferred item from C(t)C(t), instead of generating a full ranking. The likelihood is computed based on the top-1 selection probability, thereby avoiding the spurious artifacts of full ranking. The prompt wCKPP^CKP_w includes a window w, the comparison set C(t)C(t), and an instruction such as ”Choose a single identifier of the most preferred item”. The LLM output for this prompt would be an index (e.g.,[A]) for an item in C(t)C(t). Then, with the output logit values zi∈ℝz_i for each i∈C(t)i∈ C(t), we define the top-1 selection likelihood from the C(t)C(t) based on the list-wise ranking model (Cao et al., 2007): (5) PLLM(t∣wCKP)=ϕ(zt)ϕ(zt)+∑j∈C(t)∖tϕ(zj)P_LLM(t ^CKP_w)= φ(z_t)φ(z_t)+ _j∈ C(t) \t\φ(z_j) where ϕ(⋅)φ(·) is an increasing function, we adopt the exponential function. CKP effectively measures recommendation-tailored knowledge by explicitly prompting for top-1 selection. It also satisfies all three desiderata: conditioning on w for contextualization, selecting the top-1 from C(t)C(t) for ranking alignment, and using context-neutral identifiers for robustness against surface-level biases. Replacing PLLMP_LLM in Eq.1 derives the knowledge score of CKP. Table 1. An example prompt used for calculating knowledge scores. All prompts are prefixed with a domain-specific system instruction (e.g., ”You are a helpful assistant for [DOMAIN] recommendations.”). Metric Input Template DKP The user’s interaction history is as follows: [HISTORY (Item Titles] The next item is: [TARGET (Item Title)] CKP Your task is to recommend the top-1 item from the candidate set based on the user’s purchase history. You must only respond with the single identifier of the recommended item. PURCHASED ITEMS: [HISTORY (Item Titles)] CANDIDATE ITEMS: [COMPARISON SET (ID, Item Titles)] Candidate [ 4.2. Knowledge-aware Selective Augmentation We now enhance the LLM-based recommenders via selective knowledge augmentation. Given the task setup (section 2), our focus lies in answering two key questions: what to augment (section 4.2.1) and with which information (section 4.2.2), to maximize recommendation accuracy. The final prompting process is provided in section 4.2.3. 4.2.1. What to augment: Augmentation Priority Score While our knowledge score effectively reveals the LLM’s competence, we note that other factors should also be considered when prioritizing which items to augment. In recommendation tasks, it is well known that an item’s importance varies with factors such as its recency in the user history and its popularity (Petrov and Macdonald, 2024; Abbattista et al., 2024; Kang and McAuley, 2018); lacking knowledge about such items can have a greater negative impact on recommendation accuracy. By consolidating these factors, we define the Augmentation Priority Score (APS) for each item i as: (6) APS(i)=(1−k(i))⋅f(i)⋅r(i)APS(i)=(1-k(i))· f(i)· r(i) This score consists of three components.555Each component is log-transformed and min-max normalized to match the scale. The first term (1−k(i))(1-k(i)) represents the LLM’s knowledge deficiency, where k(i)k(i) is the normalized value of K(i)K(i). The second term f(i)f(i) represents the interaction frequency, prioritizing statistically prominent items. Finally, the recency score r(i)=exp⁡(−λ⋅pos(i))r(i)= (-λ·pos(i)) assigns higher weights to more recent interactions, where pos(i)pos(i) is the item’s reverse chronological position (i.e., the most recent item has pos(i)=0pos(i)=0) and λ is a decay parameter. This APS will be used to decide items to be augmented, which will be further explained in section 4.2.3. 4.2.2. With which information: Reference Matching Score After deciding the augmentation targets, we need to enrich each with information most beneficial for recommendation. A natural starting point is the textual attributes of each item in T. Moreover, we note that items can be better understood in relation to others. Specifically, semantically or behaviorally related items may provide valuable signals that complement the target item’s standalone information. We refer to such items as reference items and incorporate their information for augmentation as well. The reference items are obtained via Reference Matching Score (RMS), which quantifies the suitability of each item r∈ℐ∖(t∪Hu∪Cu)r (\t\∪H^u∪C^u) as a knowledge-supporting reference for a target item t: (7) RMS(t,r)=k(r)⋅s(t,r)⋅c(t,r)RMS(t,r)=k(r)· s(t,r)· c(t,r) This score also consists of three components, each properly normalized. First, the normalized knowledge score k(r)k(r) prioritizes items that are well understood by the LLM. Second, the semantic similarity s(t,r)s(t,r) is measured as the cosine similarity between their text embeddings, as used in Eq. 4. Third, the co-consumption score c(t,r)c(t,r) captures behavioral proximity, computed as the co-occurrence frequency in the interaction histories.666c(t,r)=freq(t,r)freq(t)⋅freq(r)c(t,r)= freq(t,r) freq(t)· freq(r), where freq(t,r) is the co-occurrence count within a sliding window of size 2, and freq(i) is the total number of windows containing item i. Our RMS design ensures that the reference items are semantically and behaviorally related to the target item, while also being well understood by the LLM. Note that we do not augment the full details of these reference items. Instead, we guide the LLM by providing minimal cues through their titles. Since the LLM already possesses sufficient knowledge about them, this can activate the relevant information from its parameters, allowing the LLM to naturally reference it without consuming much input context budget. Context-aware Variant. We acknowledge that the proposed RMS is inherently context-agnostic; it produces the same set of references for a given target item t, regardless of the user’s specific history. However, since items possess multifaceted attributes, the ideal reference may vary depending on the user’s specific context. To address this, we explore a context-aware variant that personalizes the reference selection process by incorporating the user’s sequential interaction patterns. Specifically, we modulate the RMS score using the user-item similarity from the sequential retriever (e.g., SASRec (Kang and McAuley, 2018)) already employed for candidate generation. This approach leverages existing latent representations without requiring a separate encoder. Formally, the score is adjusted as RMSctx=RMS(t,r)⋅cos⁡(u,r)RMS_ctx=RMS(t,r)· (h^u,e^r), where uh^u and re^r are the embeddings from the retrieval model. Input: Original prompt with HuH^u and CuC^u, item textual attributes ai\a^i\, normalized knowledge scores k(i)k(i) Output: Augmented prompt 1 20.3em 3Compute APS(i)APS(i) for each item i∈Hu∪Cui∈ H^u∪ C^u (Eq.6) 4Select top-KaugK_aug items with highest APS as targets ℐaugI_aug 5foreach t∈ℐaugt _aug do 6 Add textual attributes at\a^t\ to the prompt 7 Identify top-KrefK_ref reference items using RMS(t,r)(t,r) (Eq.7) 8 Add titles of reference items to the prompt Algorithm 1 Selective Knowledge Augmentation 4.2.3. Final Prompt Construction The detailed augmentation process is presented in Algorithm 1. The number of augmentation targets (KaugK_aug) is empirically set, considering resource constraints such as GPU memory and the LLM’s context budget. For reference items, we find that a small number (Kref≤3K_ref≤ 3) is typically sufficient. A conceptual example of the augmentation is provided below: USER HISTORY: [”The Witcher 3”, ”Elden Ring”, …, ”Salt and Sanctuary”] CANDIDATE ITEMS: [”Dark Souls”, …, ”Hollow Knight”] AUGMENTATION TARGET: ”Salt and Sanctuary” AUGMENTED INFORMATION: - Textual Attributes: ”A 2D action role-playing game that combines fast, brutal combat with richly developed RPG mechanics.” - Reference Items: [”Blasphemous”, ”Dead Cells”] In sum, based on the knowledge scores, injects additional information—item attributes and related reference items—where it is most needed. This enables the LLM to utilize its context budget more effectively by allocating more capacity to those that benefit most from knowledge supplementation. Remarks: inference efficiency of . Although multiple factors such as knowledge scores and co-consumption score, these values are precomputed and stored offline. Thus, they can be accessed via simple lookup operations without introducing meaningful runtime overhead. In practice, negligible additional inference latency. Notably, compared to uniform augmentation, it reduces the total inference latency by avoiding unnecessary input tokens (section 5.3.4). Table 2. Overall performance (Recall@1). Each subcolumn shows results with Random (R) and SASRec (S) candidates. Red color indicates no improvement over No Augment. The best and second-best results are marked in bold and underlined, respectively. Dataset LLM No Augment Uniform-Meta Uniform-Wiki SelectiveAcc SelectivePop SelectiveMinK SelectiveEigV SelectiveSeaKR SelectiveSelf Improv. (%) R S R S R S R S R S R S R S R S R S R S R S A-Beauty Llama-8B 0.141 0.075 0.181 0.070 0.093 0.073 0.174 0.221 0.161 0.212 0.191 0.199 0.182 0.207 0.166 0.191 0.098 0.110 0.233 0.246 22.0 11.3 Mistral-7B 0.098 0.084 0.074 0.069 0.074 0.068 0.065 0.074 0.072 0.097 0.075 0.100 0.102 0.079 0.060 0.071 0.044 0.038 0.113 0.119 10.8 19.0 Qwen-7B 0.170 0.111 0.207 0.099 0.179 0.133 0.209 0.140 0.216 0.138 0.199 0.141 0.206 0.146 0.201 0.150 0.128 0.091 0.267 0.166 23.6 10.7 Qwen-32B 0.295 0.218 0.293 0.213 0.348 0.237 0.351 0.226 0.329 0.218 0.361 0.221 0.349 0.217 0.339 0.226 0.271 0.216 0.407 0.257 12.7 8.4 A-Gift Llama-8B 0.062 0.044 0.063 0.048 0.050 0.040 0.156 0.085 0.129 0.085 0.157 0.100 0.145 0.078 0.147 0.077 0.074 0.046 0.167 0.110 6.4 10.0 Mistral-7B 0.084 0.049 0.101 0.037 0.058 0.051 0.096 0.048 0.081 0.042 0.077 0.039 0.058 0.049 0.058 0.050 0.031 0.022 0.114 0.059 12.9 18.0 Qwen-7B 0.095 0.056 0.077 0.058 0.090 0.051 0.096 0.063 0.098 0.054 0.088 0.070 0.090 0.070 0.091 0.069 0.099 0.061 0.112 0.083 13.1 18.6 Qwen-32B 0.165 0.090 0.155 0.082 0.149 0.057 0.156 0.106 0.153 0.102 0.155 0.104 0.157 0.094 0.153 0.103 0.138 0.093 0.175 0.119 6.1 14.4 ML-1M Llama-8B 0.133 0.067 0.104 0.051 0.049 0.055 0.125 0.058 0.136 0.061 0.095 0.071 0.134 0.053 0.105 0.049 0.110 0.053 0.152 0.088 11.8 23.9 Mistral-7B 0.089 0.038 0.109 0.033 0.048 0.053 0.047 0.043 0.101 0.045 0.117 0.052 0.113 0.045 0.103 0.038 0.034 0.022 0.125 0.065 6.8 22.6 Qwen-7B 0.139 0.039 0.132 0.065 0.073 0.065 0.154 0.021 0.149 0.054 0.140 0.058 0.128 0.032 0.153 0.018 0.143 0.021 0.168 0.070 9.1 7.7 Qwen-32B 0.181 0.037 0.129 0.049 0.125 0.040 0.164 0.029 0.172 0.031 0.163 0.031 0.169 0.035 0.167 0.031 0.173 0.043 0.201 0.054 11.0 10.2 Steam Llama-8B 0.078 0.042 0.063 0.041 0.086 0.039 0.111 0.029 0.122 0.054 0.095 0.066 0.111 0.030 0.141 0.032 0.097 0.048 0.152 0.072 7.8 9.1 Mistral-7B 0.059 0.033 0.051 0.030 0.057 0.046 0.056 0.041 0.071 0.055 0.057 0.044 0.073 0.027 0.068 0.019 0.028 0.023 0.077 0.067 5.5 21.8 Qwen-7B 0.108 0.043 0.059 0.050 0.061 0.046 0.133 0.034 0.127 0.043 0.112 0.057 0.119 0.035 0.119 0.030 0.119 0.028 0.148 0.063 11.3 10.5 Qwen-32B 0.121 0.035 0.073 0.044 0.114 0.044 0.117 0.045 0.123 0.042 0.126 0.043 0.120 0.047 0.117 0.042 0.101 0.045 0.150 0.053 19.0 12.8 5. Experiments 5.1. Experimental Setup Table 3. Statistics of the datasets used in our experiments. Avg. Len denotes the average sequence length of users. Dataset #Users #Items #Inter. Avg. Len Attributes A-Beauty 4884 3948 16973 3.50 title, brand A-Gift 3392 834 13503 4.01 title, brand, category ML-1M 6040 3416 999611 161.85 title, genre Steam 25859 4038 327097 12.93 title, genre, developer, specs Datasets. We use four public datasets: Amazon-Beauty (A-Beauty) (Ni et al., 2019), Amazon-Gift Cards (A-Gift) (Ni et al., 2019), ML-1M (Harper and Konstan, 2015), and Steam (Kang and McAuley, 2018). For all datasets, we filter out users and items with fewer than five interactions, and items with missing textual features. The interactions are sorted chronologically to form historical sequences. The statistics for each preprocessed dataset are summarized in Table 3. Baselines. We compare various augmentation strategies, categorized into three groups. (a) No Augment denotes the default setup, where each item is represented by its title. (b) Uniform Augmentation indiscriminately enriches all items, the most widely used approach in the literature. Specifically, we test two variants: Uniform-Meta (Attributes) and Uniform-Wiki (Wikipedia descriptions). (c) Selective Augmentation prioritizes augmentation for certain items. For fair comparison, all methods augment target items with item attributes and reference item titles. We implement baselines corresponding to the proxies analyzed in section 3.2.1: SelectivePop (Popularity), SelectiveMinK (Pre-training Detection via Min-K% (Shi et al., 2023)), SelectiveEigV (Uncertainty via EigValLaplacian (Lin et al., 2024d)), and SelectiveSeaKR (Adaptive RAG via SeaKR (Yao et al., 2025)). We also include SelectiveAcc, which uses Recall@@1 from the candidate generation stage.777In the two-stage recommendation (section 2), the accuracy of the first-stage model can serve as a natural proxy. In this work, we use item-level Recall@1 of SASRec (Kang and McAuley, 2018). Additionally, we introduce SelectiveSelf, a prompting-based baseline that operates via a two-stage inference without offline scoring, explicitly asking the LLM to identify unfamiliar items in the input prompt to apply selective augmentation. Lastly, denotes the proposed frameworks, utilizing CKP. Evaluation setup. We adopt the standard leave‑one‑out protocol (Hou et al., 2024; Kang and McAuley, 2018), where the last item in each user sequence serves as the ground‑truth item. The evaluation task is to rank a candidate set of 20 items (1 ground-truth with 19 negatives) under two different settings: (i) Random (R), with negatives from uniform sampling over unseen items (Kim et al., 2024; Wang et al., 2025), and (i) SASRec (S), with negatives from the top-19 predictions of a SASRec trained on each dataset (Hou et al., 2024). Following (Kim et al., 2024), we report Recall@@1 of the ranking list from LLMs. Implementation details. We evaluate our framework on Llama-3.1-8B (Grattafiori et al., 2024), Mistral-7B-v0.3 (37), Qwen2.5-7,32B (40) and GPT-4o (Hurst et al., 2024). We follow the prompting scheme from (Ren et al., 2024). An example prompt is provided in Table 4. For fair comparison, we fix the history, candidates, and prompt template across all methods; only the content of the auxiliary information varies by augmentation strategy. For each dataset, we randomly sample 1,500 users for testing. The remaining data is split into training and validation sets (9:1) for SASRec training, knowledge scoring, and hyperparameter tuning. User histories are truncated to the recent 50 interactions. For hyperparameters, we tune the model in certain ranges as follows: (1) For SASRec, embedding dimension ∈64,128∈\64,128\ and batch size ∈64,128,256∈\64,128,256\. (2) For CKP, random (n) and semantic (m) distractors ∈0,1,2∈\0,1,2\. (3) For APS, recency decay λ∈0.7,0.4,0.1λ∈\0.7,0.4,0.1\ and augmentation targets Kaug∈2,3,5,10,20,40K_aug∈\2,3,5,10,20,40\. (4) For RMS, reference items Kref∈1,2,3K_ref∈\1,2,3\. Table 4. Structure of the recommendation prompt. INSTRUCTION: Your task is to recommend 20 games to a specific user from a candidate item set. PURCHASED ITEMS: [”The Witcher 3”, ”Elden Ring”, …, ”Salt and Sanctuary”] CANDIDATE ITEMS: [”A”: ”Dark Souls”, …, ”T”: ”Hollow Knight”] AUXILIARY INFORMATION: [”title”: ”Salt and Sanctuary”, ”description”: ”A 2D action role-playing game …”, ”title of similar game”: ”Blasphemous”] OUTPUT: [”A”, ”C”, …, ”T”] 5.2. Overall Performance Table 2 shows performance across four datasets. outperforms all baselines across all datasets and LLMs. First, uniform augmentation does not guarantee performance improvement. They often fails to outperform the ’No Augment’ (red in table), likely because indiscriminately adding all metadata can exceed the LLM’s effective context budget, leading to information overload that obscures critical information. Also, naive selective augmentation strategies relying on heuristics show inconsistent results across datasets and models, indicating that single-view proxies are not robust. This again supports the necessity of a tailored augmentation strategy for recommendation. Finally, robust and superior performance in all settings. By leveraging CKP for accurate knowledge estimation and guiding augmentation via APS and RMS, it provides targeted, effective enhancements beyond simple heuristics. Figure 3. Average Recall@1 across four quantile bins grouped by knowledge score (CKP) on A-Beauty dataset. Results of Qwen-7B with random candidates. Performance by knowledge levels. We further analyze performance across users with varying levels of knowledge. We group test users into four quantile groups based on the average CKP scores of items in the interaction history, with Group 1 representing the most knowledge-poor users. Figure 3 shows that achieves the highest accuracy across all groups. Notably, the performance gap between uniform augmentation is most pronounced in Groups 1–2. This suggests that effectively bridges the knowledge gap through selective augmentation guided by knowledge scores. Further, it confirms that our targeted augmentation is highly effective in supplementing knowledge for users where LLMs have limited competence. 5.3. Study of 5.3.1. Comparison with Frontier Models One might attribute the poor performance of the SelectiveSelf baseline to the limited reasoning capabilities of open-source models. To investigate this, we conducted a comparative experiment using a frontier model, GPT-4o. As shown in Table 5, remarkably, even for GPT-4o, the self-asking strategy resulted in a performance drop compared to the No Augment baseline, whereas Qwen-32B equipped with KnowSACKP achieved the highest performance. This suggests that accurately diagnosing knowledge gaps for numerous distinct items (up to 70 items, including 50 history and 20 candidate items) within a single prompt is an inherently difficult task, even for advanced LLMs. These findings underscore the importance of a scoring strategy designed to diagnose knowledge gaps in the context of recommendation. Table 5. Performance comparison on A-Beauty dataset. We investigate whether a stronger LLM can effectively perform selective augmentation via self-asking. Improv. denotes the relative improvement in Recall compared to the No Augment baseline. Model Method R S Improv. GPT-4o No Augment 0.316 0.265 - Uniform-Meta 0.349 0.269 +10.4% SelectiveSelf 0.300 0.251 -5.1% Qwen-32B No Augment 0.295 0.218 - Uniform-Meta 0.293 0.213 -0.7% KnowSACKP (Ours) 0.407 0.257 +38.0% 5.3.2. Ablation Study Table 6 presents the results of various ablations. First, we analyze CKP, our knowledge scoring strategy. Removing relative comparison, which replaces CKP with DKP, largely degrades performance, and removing semantic distractors leads to a slight drop—validating our comparative probing design. Next, we assess our augmentation mechanisms. For both APS and RMS, removing each proposed component results in performance degradation, with a particularly severe drop observed when the knowledge score is excluded, which supports the validity of our design. The necessity of APS is particularly evident on ML-1M, a dataset with long historical interactions. The severe performance drop of ‘w/o APS’ highlights its critical role; prioritizing augmentations is essential in long contexts to ensure effective knowledge injection and to mitigate potential misinterpretations by LLMs (Liu et al., 2023). For RMS, removing reference items entirely (‘w/o RMS’) degrades performance. Notably, co-consumption score proves to be vital, showing the importance of reflecting behavioral collaborative patterns. Table 6. Ablation study on the components of the proposed method using Qwen-7B with random candidates. Method Variant A-Beauty A-Gift ML-1M KnowSACKP (ours) 0.267 0.112 0.168 Ablations on CKP w/o Relative Comparison 0.214 0.081 0.139 w/o Semantic Distractors 0.219 0.110 0.151 Ablations on APS w/o APS (Uniform Aug. w/ ref. items) 0.194 0.072 0.087 w/o Knowledge Score 0.213 0.095 0.151 w/o Interaction Frequency 0.216 0.109 0.153 w/o Recency Score 0.225 0.110 0.160 Ablations on RMS w/o RMS (Textual attributes only) 0.223 0.087 0.123 w/o RMS (Wikipedia description only) 0.079 0.032 0.063 w/o Knowledge Score 0.188 0.095 0.146 w/o Semantic Similarity 0.215 0.107 0.149 w/o Co-consumption Score 0.169 0.109 0.135 w/o SASRec Contextualization 0.233 0.114 0.167 5.3.3. Analysis on Long-tail Coverage and Bias To assess how our framework mitigates the knowledge gap for less-known items, we examine the coverage of long-tail items. As a metric, we employ Long-tail Coverage (LTC@K) (Abdollahpouri et al., 2019), which measures the average fraction of long-tail items—defined as those in the bottom 80% of popularity—appearing in a user’s top-K recommendation list.888LTC@K=1|Utest|∑u∈Utest|RuK∩ILT|/KLTC@K= 1|U_test| _u∈ U_test|R^K_u∩ I_LT|/K, where ILTI_LT is the long-tail item set, and RuKR_u^K is the top-K list for user u. A higher LTC@K value indicates better coverage of long-tail items. Figure 4 shows that selective augmentation generally improves long-tail coverage compared to uniform augmentation. While a slight gain in LTC@K over other selective baselines, it achieves significantly higher recommendation accuracy in Table 2. This shows that a superior balance, enhancing recommendation diversity without sacrificing accuracy. Furthermore, we investigate whether this improvement stems from an artificial bias, where the LLM over-recommends augmented items simply because they are enriched with extra knowledge. We analyze the Top-1 prediction frequency of the augmentation targets (ℐaugI_aug) within the candidate set. Interestingly, compared to ’No Augment’, this frequency from 393 to 375 on A-Beauty using Qwen-7B. This indicates that our augmentation provides clarifying knowledge, enhancing the model’s ability to discern and reject inappropriate candidates, rather than simply boosting their ranks. Figure 4. Long-tail Coverage (LTC@10) on A-Beauty (left) and Steam (right), using Qwen-7B with random candidates. 5.3.4. Efficiency Analysis We analyze the computational efficiency of both offline preparation and online inference perspectives. First, we address the scalability of the offline precomputation phase. While calculating knowledge scores, CKP, for all item windows appears computationally intensive, our popularity-stratified sampling strategy (section 4.1) drastically reduces this cost.999For instance, on the ML-1M dataset, the number of sampled windows accounts for only 1% of the total windows. As shown in Table 7, our strategy consistently accelerates the scoring process across all datasets. This computational efficiency ensures that precomputing knowledge scores is practically feasible, allowing them to be stored offline. Second, regarding online inference, we evaluate the average number of input tokens per user history and the corresponding inference latency. Since the augmentation factors are precomputed, they can be accessed with negligible runtime overhead. Table 8 shows that the ’Uniform-Meta.’ strategy incurs substantial overhead; indiscriminately adding all attributes significantly increases both token count and latency.101010Note that latency does not increase linearly with input tokens; inference time is dominated by output decoding, while input processing is parallelized on GPUs (Yang et al., 2024b). In contrast, reduces this overhead while still achieving the highest accuracy, as shown in Table 2. 5.3.5. Hyperparameter Analysis We provide analysis to guide the hyperparameter selection: the number of augmented items (KaugK_aug) and the number of reference items (KrefK_ref). Both hyperparameters should be chosen with consideration for resource constraints, such as GPU memory and the LLM’s context budget. In our experiments, the number of augmented items used is approximately 10 on average across datasets. Figure 5 reports the results for Qwen-7B on Steam, with similar trends observed across other settings. We observe that performance improves as KaugK_aug increases, but saturates beyond a certain point. This indicates that while providing supplementary information is beneficial, excessive augmentation can exceed the LLM’s effective context budget. For KrefK_ref, a small number is generally sufficient, with the best performance achieved at 22. 6. Related Work LLM-based Recommendation. To improve the LLM-based recommendation (Hou et al., 2024; Kim et al., 2024; Luo et al., 2024), two major strategies have emerged. The first is model-level adaptation, which fine-tunes LLMs using interaction data. (Bao et al., 2023; Zhang et al., 2025, 2024b; Lin et al., 2024c; Bao et al., 2025; Chen et al., 2025) incorporate recommendation-specific objectives into the fine-tuning process. (Kim et al., 2024) combines lightweight adapters with frozen LLMs, significantly reducing training costs. Some methods (Kim et al., ; Liu et al., 2024) use auxiliary data such as reviews to enhance personalization. However, fine-tuning LLMs remains resource-intensive compared to conventional recommenders. Also, interaction data for fine-tuning is sparse and skewed toward popular items, limiting the model’s generalization. Another emerging direction is prompt-level adaptation, which enriches input prompts with additional information (Lin et al., 2024b; Kim et al., ; Zheng et al., 2024; Wang et al., 2025). For example, (Kim et al., ) utilizes review summaries to provide rich information for users and items, while (Wang et al., 2025) uses knowledge graphs to supplement missing knowledge. These approaches allow LLMs to supplement incomplete knowledge without fine-tuning. However, most existing methods adopt uniform augmentation—applying information to all items indiscriminately, which overlooks the inherent knowledge gaps in LLMs. We propose an effective strategy to improve training-free LLM recommenders by selectively enriching items lacking sufficient knowledge. Figure 5. Effect of KaugK_aug (left) and KrefK_ref (right) in KnowSACKPKnowSA_CKP. Table 7. Computational cost of CKP scoring. We compare the processing time (seconds) between full vs. sampled windows. Mem denotes GPU memory usage. Dataset TimeFull (s) TimeSampled (s) Speedup Mem (GB) A-Beauty 903 395 2.3×2.3× 17.7 A-Gift 711 124 5.7×5.7× 17.8 ML-1M 61,362 602 101.9×101.9× 17.8 Steam 18,442 927 19.9×19.9× 19.2 Table 8. Efficiency comparison of different augmentation strategies on Qwen-7B. Overhead denotes the percentage increase in tokens and latency compared to ’No Augment’. Uniform-Meta Dataset Tokens Overhead (%) Latency Overhead (%) Tokens Overhead (%) Latency Overhead (%) A-Beauty +33.0% +7.8% +19.0% +5.6% A-Gift +87.4% +6.9% +54.1% +5.7% ML-1M +85.6% +11.6% +29.3% +5.5% Steam +414.3% +14.2% +99.0% +4.1% Knowledge Probing for LLMs. A core of our framework is to accurately estimate how much an LLM knows about each item. We categorize existing approaches relevant to this goal into three groups. First, Pre-training Data Detection (PDD) aims to determine if a text was seen during pre-training (Carlini et al., 2021; Zhang et al., 2024a; Shi et al., 2023; Zhou et al., 2024). PDD methods typically rely on generation likelihood, assuming that pre-seen texts will yield higher probabilities (Shi et al., 2023; Zhang et al., 2024a; Zhou et al., 2024). Second, Uncertainty Estimation (UE) focuses on quantifying the confidence in model prediction (Xia et al., 2025; Lin et al., 2024d). UE methods range from latent information-based metrics (e.g., entropy) (Fomicheva et al., 2020; Duan et al., 2024) to consistency-based measures (Lin et al., 2024d; Fadeeva et al., 2024); for instance, (Lin et al., 2024d) computes the eigenvalues of a Laplacian graph constructed from the semantic similarity of sampled responses. Third, Adaptive Retrieval approaches (Yao et al., 2025; Jiang et al., 2023; Su et al., 2024; Jeong et al., 2024) dynamically decide when to retrieve external documents. Approaches include monitoring generation probabilities (Jiang et al., 2023; Su et al., 2024), leveraging internal model states (Yao et al., 2025), or employing external classifiers (Jeong et al., 2024) to predict retrieval needs. While these approaches have proven their efficacy in general NLP tasks, they are suboptimal for recommendation. The core issue is that they assess item-specific knowledge in isolation, thereby overlooking the collaborative patterns—the relationship between the user’s history and the target item—essential for recommendation, as detailed in our analysis (section 3.2.2). To address this, we propose CKP, a new knowledge scoring method that evaluates the LLM’s knowledge in a comparative setting conditioned on user interaction contexts. This approach effectively mitigates surface-level biases and enables a more recommendation-aligned estimation of the model’s actual knowledge. 7. Conclusion We first provide in-depth analysis showing the necessity and challenges of addressing the knowledge gap in LLM-based recommendation. Motivated by the analysis, we introduce , which mitigates the gap through selective knowledge augmentation. two main components: (1) Comparative Knowledge Probing, which estimates the LLM’s knowledge of each item, and (2) Selective Augmentation, which enriches knowledge for lesser-known items. Extensive experiments show that improves both recommendation accuracy and long-tail coverage, while also reducing the input context consumption of LLMs. Future work may explore leveraging external sources such as knowledge graphs to further enhance augmentation. Acknowledgments This work was supported by the NRF grant funded by the MSIT (No. RS-2024-00335873), the IITP grant funded by the MSIT (No.RS-2019-I191906, Artificial Intelligence Graduate School Program(POSTECH)), Korea Innovation Foundation (INNOPOLIS) grant funded by the Korea government (MSIT) (No. RS-2025-25449754). This work was also supported by ICT Creative Consilience Program through the IITP grant funded by the MSIT (IITP-2026-RS-2020-I201819) and Basic Science Research Program through the NRF funded by the Ministry of Education (NRF-2021R1A6A1A03045425). References D. Abbattista, V. W. Anelli, T. Di Noia, C. Macdonald, and A. V. Petrov (2024) Enhancing sequential music recommendation with personalized popularity awareness. In Proceedings of the 18th ACM Conference on Recommender Systems, p. 1168–1173. Cited by: §4.2.1. H. Abdollahpouri, R. Burke, and B. Mobasher (2019) Managing popularity bias in recommender systems with personalized re-ranking. arXiv preprint arXiv:1901.07555. Cited by: §5.3.3. A. Amayuelas, K. Wong, L. Pan, W. Chen, and W. Wang (2023) Knowledge of knowledge: exploring known-unknowns uncertainty with large language models. arXiv preprint arXiv:2305.13712. Cited by: 3rd item. K. Bao, J. Zhang, W. Wang, Y. Zhang, Z. Yang, Y. Luo, C. Chen, F. Feng, and Q. Tian (2025) A bi-step grounding paradigm for large language models in recommendation systems. ACM Transactions on Recommender Systems 3 (4), p. 1–27. Cited by: §6. K. Bao, J. Zhang, Y. Zhang, W. Wang, F. Feng, and X. He (2023) Tallrec: an effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, p. 1007–1014. Cited by: §1, §6. Z. Cao, T. Qin, T. Liu, M. Tsai, and H. Li (2007) Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, p. 129–136. Cited by: §4.1.3. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. (2021) Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), p. 2633–2650. Cited by: 2nd item, §6. J. Chen, C. Gao, S. Yuan, S. Liu, Q. Cai, and P. Jiang (2025) Dlcrec: a novel approach for managing diversity in llm-based recommender systems. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, p. 857–865. Cited by: §6. P. Covington, J. Adams, and E. Sargin (2016) Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, p. 191–198. Cited by: §2. J. Duan, H. Cheng, S. Wang, A. Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu (2024) Shifting attention to relevance: towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5050–5063. Cited by: 3rd item, §6. E. Fadeeva, A. Rubashevskii, A. Shelmanov, S. Petrakov, H. Li, H. Mubarak, E. Tsymbalov, G. Kuzmin, A. Panchenko, T. Baldwin, et al. (2024) Fact-checking the output of large language models via token-level uncertainty quantification. arXiv preprint arXiv:2403.04696. Cited by: 3rd item, §6. M. Fomicheva, S. Sun, L. Yankovskaya, F. Blain, F. Guzmán, M. Fishel, N. Aletras, V. Chaudhary, and L. Specia (2020) Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics 8, p. 539–555. Cited by: 3rd item, §6. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.1. F. M. Harper and J. A. Konstan (2015) The movielens datasets: history and context. Acm transactions on interactive intelligent systems (tiis) 5 (4), p. 1–19. Cited by: §5.1. J. Harte, W. Zorgdrager, P. Louridas, A. Katsifodimos, D. Jannach, and M. Fragkoulis (2023) Leveraging large language models for sequential recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, p. 1096–1102. Cited by: §1. Y. Hou, J. Zhang, Z. Lin, H. Lu, R. Xie, J. McAuley, and W. X. Zhao (2024) Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval, p. 364–381. Cited by: §1, §2, §3.1, §5.1, §6, footnote 4. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §1. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. (2024) Gpt-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §5.1. S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. Park (2024) Adaptive-RAG: learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), p. 7036–7050. External Links: Link, Document Cited by: 4th item, §6. Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig (2023) Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 7969–7992. Cited by: §1, 4th item, §6. C. Kang and J. Choi (2023) Impact of co-occurrence on factual knowledge of large language models. arXiv preprint arXiv:2310.08256. Cited by: §1. W. Kang and J. McAuley (2018) Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), p. 197–206. Cited by: §4.2.1, §4.2.2, §5.1, §5.1, footnote 7. W. Kang and J. McAuley (2019) Candidate generation with binary codes for large-scale top-n recommendation. In Proceedings of the 28th ACM international conference on information and knowledge management, p. 1523–1532. Cited by: §2. [24] J. Kim, H. Kim, H. Cho, S. Kang, B. Chang, J. Yeo, and D. Lee Review-driven personalized preference reasoning with large language models for recommendation. corr, abs/2408.06276, 2024. doi: 10.48550. arXiv preprint ARXIV.2408.06276. Cited by: §1, §6, §6. S. Kim, H. Kang, S. Choi, D. Kim, M. Yang, and C. Park (2024) Large language models meet collaborative filtering: an efficient all-round llm-based recommender system. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 1395–1406. Cited by: §1, §5.1, §6. S. Kim, G. Lee, K. Kim, J. Yoo, and K. Shin (2025) ItemRAG: item-based retrieval-augmented generation for llm-based recommendation. arXiv preprint arXiv:2511.15141. Cited by: §1. Y. Liang, L. Yang, C. Wang, X. Xu, P. S. Yu, and K. Shu (2024) Taxonomy-guided zero-shot recommendations with llms. arXiv preprint arXiv:2406.14043. Cited by: §1, 2nd item. J. Liao, S. Li, Z. Yang, J. Wu, Y. Yuan, and X. Wang (2023) Llara: aligning large language models with sequential recommenders. CoRR. Cited by: §1, §1. J. Lin, R. Shan, C. Zhu, K. Du, B. Chen, S. Quan, R. Tang, Y. Yu, and W. Zhang (2024a) Rella: retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation. In Proceedings of the ACM Web Conference 2024, p. 3497–3508. Cited by: §2. X. Lin, W. Wang, Y. Li, F. Feng, S. Ng, and T. Chua (2024b) Bridging items and language: a transition paradigm for large language model-based recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 1816–1826. Cited by: §1, §2, §6. X. Lin, W. Wang, Y. Li, S. Yang, F. Feng, Y. Wei, and T. Chua (2024c) Data-efficient fine-tuning for llm-based recommendation. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, p. 365–374. Cited by: §6. Z. Lin, S. Trivedi, and J. Sun (2024d) Generating with confidence: uncertainty quantification for black-box large language models. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §1, 3rd item, §5.1, §6. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2023) Lost in the middle: how language models use long contexts. arXiv preprint arXiv:2307.03172. Cited by: §1, §3.1, §5.3.2. Q. Liu, X. Wu, Y. Wang, Z. Zhang, F. Tian, Y. Zheng, and X. Zhao (2024) Llm-esr: large language models enhancement for long-tailed sequential recommendation. Advances in Neural Information Processing Systems 37, p. 26701–26727. Cited by: §6. S. Luo, B. He, H. Zhao, W. Shao, Y. Qi, Y. Huang, A. Zhou, Y. Yao, Z. Li, Y. Xiao, et al. (2024) Recranker: instruction tuning large language model as ranker for top-k recommendation. ACM Transactions on Information Systems. Cited by: §1, §6. A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi (2022) When not to trust language models: investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511. Cited by: §1, 1st item. [37] (2023) Mistral 7b. ArXiv abs/2310.06825. External Links: Link Cited by: §5.1. J. Ni, J. Li, and J. McAuley (2019) Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 188–197. Cited by: §5.1. A. Petrov and C. Macdonald (2024) RSS: effective and efficient training for sequential recommendation using recency sampling. ACM Transactions on Recommender Systems 3 (1), p. 1–32. Cited by: §4.2.1. [40] (2024) Qwen2.5 technical report. ArXiv abs/2412.15115. External Links: Link Cited by: §5.1. N. Reimers and I. Gurevych (2019) Sentence-bert: sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. Cited by: footnote 2. X. Ren, W. Wei, L. Xia, L. Su, S. Cheng, J. Wang, D. Yin, and C. Huang (2024) Representation learning with large language models for recommendation. In Proceedings of the ACM Web Conference 2024, p. 3464–3475. Cited by: §1, §5.1. S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme (2012) BPR: bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618. Cited by: §1. W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer (2023) Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789. Cited by: §1, 2nd item, §5.1, §6. W. Su, Y. Tang, Q. Ai, Z. Wu, and Y. Liu (2024) DRAGIN: dynamic retrieval augmented generation based on the real-time information needs of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), p. 12991–13013. External Links: Document Cited by: §1, 4th item, §6. S. Wang, W. Fan, Y. Feng, X. Ma, S. Wang, and D. Yin (2025) Knowledge graph retrieval-augmented generation for llm-based recommendation. arXiv preprint arXiv:2501.02226. Cited by: §1, 1st item, §5.1, §6. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2021) Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. Cited by: §2. Z. Xia, J. Xu, Y. Zhang, and H. Liu (2025) A survey of uncertainty estimation methods on large language models. arXiv preprint arXiv:2503.00172. Cited by: 3rd item, §6. H. Yang, Y. Zhang, J. Xu, H. Lu, P. A. Heng, and W. Lam (2024a) Unveiling the generalization power of fine-tuned large language models. arXiv preprint arXiv:2403.09162. Cited by: §1. Y. Yang, L. Jiao, and Y. Xu (2024b) A queueing theoretic perspective on low-latency llm inference with variable token length. In 2024 22nd International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), p. 273–280. Cited by: footnote 10. Z. Yao, W. Qi, L. Pan, S. Cao, L. Hu, L. Weichuan, L. Hou, and J. Li (2025) SeaKR: self-aware knowledge retrieval for adaptive retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 27022–27043. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1, 4th item, §5.1, §6. Q. Yu, J. Merullo, and E. Pavlick (2023) Characterizing mechanisms for factual recall in language models. arXiv preprint arXiv:2310.15910. Cited by: §1. Z. Yue, S. Rabhi, G. d. S. P. Moreira, D. Wang, and E. Oldridge (2023) Llamarec: two-stage recommendation using large language models for ranking. arXiv preprint arXiv:2311.02089. Cited by: §1. W. Zhang, R. Zhang, J. Guo, M. de Rijke, Y. Fan, and X. Cheng (2024a) Pretraining data detection for large language models: a divergence-based calibration method. arXiv preprint arXiv:2409.14781. Cited by: 2nd item, §6. Y. Zhang, K. Bao, M. Yan, W. Wang, F. Feng, and X. He (2024b) Text-like encoding of collaborative information in large language models for recommendation. arXiv preprint arXiv:2406.03210. Cited by: §1, §6. Y. Zhang, F. Feng, J. Zhang, K. Bao, Q. Wang, and X. He (2025) Collm: integrating collaborative embeddings into large language models for recommendation. IEEE Transactions on Knowledge and Data Engineering. Cited by: §1, §6. Z. Zheng, W. Chao, Z. Qiu, H. Zhu, and H. Xiong (2024) Harnessing large language models for text-rich sequential recommendation. In Proceedings of the ACM Web Conference 2024, p. 3207–3216. Cited by: §1, §6. B. Zhou, Z. Wang, L. Wang, H. Wang, Y. Zhang, K. Song, X. Sui, and K. Wong (2024) DPDLLM: a black-box framework for detecting pre-training data from large language models. In Findings of the Association for Computational Linguistics ACL 2024, p. 644–653. Cited by: §6.

Instant research discovery