Paper deep dive
SPADE: Escaping the Popularity-Similarity Frontier to Measure Serendipitous Recommendations
Tobias Vente, Maarten Peirsman, Noah Daniëls, Hannu Toivonen, Bart Goethals
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/28/2026, 4:22:57 AM
Summary
The paper introduces SPADE (Serendipitous Pareto Distance Evaluation), a novel offline metric for measuring serendipity in recommender systems. Unlike existing metrics that isolate historical similarity or global popularity, SPADE maps items into a two-dimensional space defined by these factors to construct a user-specific Pareto frontier. The serendipity score is calculated as the average Euclidean distance of correctly recommended items from this frontier, effectively penalizing items that are either too popular or too similar to the user's history while rewarding relevant, unexpected discoveries.
Entities (17)
Relation Signals (14)
Tobias Vente → affiliatedwith → University of Antwerp
confidence 95% · email: tobias.vente@uantwerpen.be ... ADREM Data Lab, University of Antwerp
SPADE → evaluatedon → CiteULike
confidence 95% · We evaluate SPADE across five datasets from distinct domains: CiteULike
SPADE → evaluatedon → Netflix
confidence 95% · We evaluate SPADE across five datasets from distinct domains: ... and Netflix
SPADE → evaluatedon → MovieLens 1M
confidence 95% · We evaluate SPADE across five datasets from distinct domains: ... MovieLens-1M
SPADE → uses → Pareto frontier
confidence 95% · SPADE maps all items into a two-dimensional space to directly calculate a user-specific Pareto frontier
SPADE → comparedwithalgorithm → SLIM
confidence 90% · We compute SPADE against recommendations predicted by five baseline algorithms: ... SLIM
SPADE → comparedwithalgorithm → ItemKNN
confidence 90% · We compute SPADE against recommendations predicted by five baseline algorithms: ... ItemKNN
SPADE → comparedwithalgorithm → EASE
confidence 90% · We compute SPADE against recommendations predicted by five baseline algorithms: EASE
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recommender systems engineer serendipity to foster active exploration and break predictable consumption cycles. The problem with existing offline beyond-accuracy metrics is that they often either isolate historical similarity or global popularity. We aim to design an evaluation metric that examines similarity, popularity, and actual user relevance. To achieve this, we introduce SPADE (Serendipitous Pareto Distance Evaluation). SPADE maps all items into a two-dimensional space to directly calculate a user-specific Pareto frontier of maximally popular and historically similar items. The final serendipity score is then computed by averaging the minimum Euclidean distance from this boundary strictly for the correctly recommended test-set items. Evaluating SPADE across five datasets and five baseline algorithms confirms its effectiveness; our results show that the metric successfully prevents algorithms from exploiting beyond-accuracy measures with irrelevant or non-personalized recommendations, reliably isolating serendipitous discoveries.
Tags
Links
- Source: https://arxiv.org/abs/2609.31164v1
- Canonical: https://arxiv.org/abs/2609.31164v1
Trouble viewing inline? Open PDF directly →
Full Text
41,494 characters extracted from source content.
Expand or collapse full text
SPADE: Escaping the Popularity-Similarity Frontier to Measure Serendipitous Recommendations Tobias Vente Maarten Peirsman Noah Daniëls Hannu Toivonen Bart Goethals Abstract Recommender systems engineer serendipity to foster active exploration and break predictable consumption cycles. The problem with existing offline beyond-accuracy metrics is that they often either isolate historical similarity or global popularity. We aim to design an evaluation metric that examines similarity, popularity, and actual user relevance. To achieve this, we introduce SPADE (Serendipitous Pareto Distance Evaluation). SPADE maps all items into a two-dimensional space to directly calculate a user-specific Pareto frontier of maximally popular and historically similar items. The final serendipity score is then computed by averaging the minimum Euclidean distance from this boundary strictly for the correctly recommended test-set items. Evaluating SPADE across five datasets and five baseline algorithms confirms its effectiveness; our results show that the metric successfully prevents algorithms from exploiting beyond-accuracy measures with irrelevant or non-personalized recommendations, reliably isolating serendipitous discoveries. †copyrightyear: 2026†copyright: Copyright © 2026 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (C BY 4.0).†venue: FRAME’26: Methodology First - Rethinking Research Assessment in RecSys Workshop, September 28, 2026, Minneapolis, Minnesota, USA†email: tobias.vente@uantwerpen.be†url: https://tobiasvente.github.io/†email: maarten.peirsman@uantwerpen.be†email: noah.daniels@uantwerpen.be†email: hannu.toivonen@helsinki.fi†url: http://w.cs.helsinki.fi/hannu.toivonen/†email: bart.goethals@uantwerpen.be†url: https://w.bartgoethals.com/†address: ADREM Data Lab, University of Antwerp, Middelheimlaan 1, 2020 Antwerpen, Belgium†address: Department of Computer Science, PO Box 68, FI-00014 University of Helsinki, Finland†address: Froomle, Post X, Borsbeeksebrug 34, 2600 Antwerpen, Belgium 1 Introduction Recommender systems excel at predicting user preferences, yet they often struggle to deliver what users never knew they needed Ge et al. (2010); Ziarani and Ravanmehr (2021b). When systems strictly optimize for predictive accuracy, they risk trapping users in isolated information silos and continuously narrowing their exposure to diverse content Nguyen et al. (2014). To overcome this filter bubble effect and ensure long-term user retention, researchers engineer serendipity into these systems Xi et al. (2025). Serendipity acts as a counterweight, fostering active exploration and breaking predictable consumption cycles Ge et al. (2010). While dictionary definitions of serendipity emphasize pure chance and luck, in the context of recommender systems, it is treated instead as an explicit engineering objective. Serendipity is defined as the intersection of surprise, relevance, and novelty in recommendations Maksai et al. (2015a); Kaminskas and Bridge (2016a); Kotkov et al. (2018). Introducing serendipity does not mean serving random items; rather, it involves delivering suggestions that appear as fortunate, random discoveries to the user, while in fact being purposefully selected by the algorithm Maksai et al. (2015a); Kaminskas and Bridge (2016a); Kotkov et al. (2018). For example, recommending the next obvious blockbuster or a direct sequel almost guarantees high acceptance, but it fails to deliver a surprising discovery Ziarani and Ravanmehr (2021b). In contrast, recommending an item from an unexpected yet relevant genre achieves serendipity Jenders et al. (2015). The degree of serendipity in a recommendation relies on two distinct dimensions: its similarity to the target user’s historical interactions and its global popularity across the system Kaminskas and Bridge (2016a); Zhang et al. (2012). While suggesting highly similar or globally dominant items is a safe strategy to ensure baseline relevance, it fundamentally compromises both unexpectedness and novelty Kaminskas and Bridge (2016a); Zhang et al. (2012). A recommendation that too closely mirrors a user’s past interactions fails to generate surprise, just as a universally popular item is highly unlikely to provide a novel discovery. Consequently, achieving serendipity requires navigating away from these high-probability zones to find items that are relevant, yet neither popular nor similar to a user’s past interactions. Unfortunately, established serendipity metrics struggle to capture this dynamic due to their narrow focus. They tend to evaluate serendipity by isolating just one dimension of the recommendation space. For example, some approaches measure unexpectedness purely as the content-based distance between recommendations and a user’s history Shani and Gunawardana (2010); De Pessemier et al. (2014); Wang et al. (2018). Other metrics focus entirely on global patterns. These include evaluating serendipity as a deviation from primitive popularity baselines Murakami et al. (2007); Ge et al. (2010), relying on item co-occurrence probabilities Kaminskas and Bridge (2014); Kaminskas and Bridge (2016b), or measuring the time saved in discovering an item Kawamae (2015). While existing serendipity metrics capture distinct facets of serendipity, they share a flaw: they do not assess both similarity and popularity simultaneously. By ignoring either similarity or popularity, isolated metrics often reward mainstream items simply because they are new to the user, or reward highly predictable items just because they are globally rare, or inadvertently reward obscure discoveries that hold no actual relevance to the user. Failing to penalize these false positives can inflate the measured serendipity, exposing the need for a metric that strictly optimizes for both dimensions. To address this gap, we introduce SPADE (Serendipitous Pareto Distance Evaluation), a novel offline beyond-accuracy metric that quantifies serendipity by jointly evaluating historical similarity and global popularity. Within this two-dimensional space, we construct a user-specific Pareto frontier of items that maximize both global reach and alignment with a user’s history. SPADE measures serendipity as the Euclidean distance from an algorithm’s correctly predicted items to the closest point on the Pareto front. By defining SPADE and evaluating it against established beyond accuracy metrics, we investigate whether this structural approach can successfully measure serendipity. Ultimately, SPADE provides researchers with a standard, quantifiable method to determine whether an algorithm is merely retreating to predictable and popular content or if it is actively delivering serendipitous discoveries. To validate SPADE, we execute three research tasks: formalization, offline evaluation, and comparative analysis. First, we define SPADE’s mathematical framework, detailing the construction of the Pareto frontier and the calculation of distances. Then, by comparing SPADE against commonly used beyond-accuracy metrics (Novelty, Co-Occurrence, and Primitivity) across five baseline algorithms on five widely used datasets, we demonstrate how it isolates serendipitous items that traditional metrics misclassify. The code used to execute the experiments is publicly available in our GitHub11 1 https://github.com/Maarten11/SPADE repository and further contains documentation to ensure the reproducibility of our experiments 2 Related Work While the importance of serendipity in recommender systems is widely recognized, existing evaluation metrics largely approach the concept through isolated dimensions. A review of the literature reveals three primary methodologies for quantifying serendipity and unexpectedness offline. However, because these approaches rely on a narrow focus (prioritizing either user history, primitive baselines, or temporal distance), they struggle to capture the necessary dual dynamic of historical similarity and global popularity, leaving them vulnerable to false positives. The most frequent approach to measuring serendipity calculates the distance between a recommended item and the user’s historical profile, utilizing either content features or collaborative filtering (CF) metrics. Initially outlined by Shani and Gunawardana Shani and Gunawardana (2010), content-based distance measurements have been widely adapted to evaluate how far a new recommendation deviates from a user’s known preferences De Pessemier et al. (2014); Jelassi et al. (2015); Wang et al. (2018). Variations of this metric, which calculate the difference between recommendations and user profiles through slightly different formulas, have been used to evaluate user exploration Chen et al. (2021) and multi-objective diversity Sá et al. (2022). Similarly, other researchers measure unexpectedness by calculating CF-based distances, frequently employing Point-wise Mutual Information (PMI) or Normalized PMI (NPMI) to evaluate the statistical co-occurrence between a candidate item and the specific items in a user’s historical profile Kaminskas and Bridge (2014); Kaminskas and Bridge (2016b); Kaya and Kaleli (2022). While effective at measuring personalization, these profile-centric metrics are entirely blind to global item popularity. By focusing solely on the user’s history, they risk rewarding highly popular items simply because they are absent from a specific user’s profile, falsely categorizing a highly predictable global trend as a serendipitous discovery. A second prominent approach operationalizes serendipity by comparing a proposed algorithm’s outputs against those of a “primitive” recommender system. Originally introduced by Murakami et al. Murakami et al. (2007) and famously formalized by Ge et al. Ge et al. (2010), this metric conceptualizes serendipity as the proportion of relevant items that are unexpected. Unexpectedness is defined by the items the primitive baseline fails to recommend. This method has been widely adopted across various domains, including news recommendations Maksai et al. (2015b), serendipity-oriented greedy algorithms Kotkov et al. (2017), fuzzy inference systems Babu et al. (2015), research paper recommendations Nishioka et al. (2019); Nishioka et al. (2020), offline data processing Ren et al. (2015), and deep neural networks Adamopoulos and Tuzhilin (2014); Ziarani and Ravanmehr (2021a). The fundamental flaw of this approach is its heavy reliance on the chosen baseline. Because the primitive system can be based on either global popularity or historical similarity (recommending the same content), the resulting serendipity score is ambiguous. It cannot guarantee that an item is both a long-tail discovery and a taste-broadening deviation. A more specialized metric defines serendipity through the lens of time. Introduced by Kawamae Kawamae (2010), this approach measures the difference between the time an algorithm recommends an item and the time it would theoretically take a user to find that item on their own without the system’s help. An item with a large time difference is considered highly serendipitous because it implies the user would not have naturally discovered it in the near future. This metric has been primarily used to evaluate real-time recommendations Kawamae (2015); Wang et al. (2018). While conceptually interesting, this temporal approach requires extensive, time-stamped historical logs of organic user discovery, which are rarely available. Furthermore, it measures the symptom of serendipity (delayed discovery) rather than the underlying structural causes—the item’s relationship to the user’s taste and the global market. While our work introduces the Pareto front as an evaluation tool, the concept itself is well-established in recommender systems literature, primarily within the domain of multi-objective optimization Lin et al. (2019); Ribeiro et al. (2014). Because recommender systems often face competing goals—such as maximizing accuracy while maintaining catalog coverage, diversity, or fairness—researchers frequently employ Pareto frontiers to find optimal trade-offs during model training Li and Tuzhilin (2024); Ribeiro et al. (2014). For instance, algorithms are often designed to push recommendations toward a Pareto-optimal boundary where accuracy cannot be further improved without sacrificing diversity Xiao et al. (2017); Peng et al. (2024). Critically, the existing literature uses Pareto frontiers almost exclusively as an optimization objective for generating recommendations rather than as a post-hoc evaluation metric. Ultimately, the current literature lacks a unified metric that evaluates serendipity as a function of both user-level similarity and global popularity. Because existing metrics ignore this crucial trade-off, they lack a reliable mechanism to penalize the false positives that occur at the extremes of either dimension. 3 SPADE (Serendipitous Pareto Distance Evaluation) To evaluate serendipity, every candidate item i for a user profile U is mapped into a two-dimensional space defined by its global popularity and its similarity to the user’s profile. We calculate a user-specific Pareto front of items that maximize both of these dimensions. Each item on this front is either the most popular item among items that are at least as similar or the most similar item among items that are at least as popular. We posit that these Pareto-optimal items are the least serendipitous because they dominate either popularity or similarity. Conversely, items farther from the Pareto front are considered more serendipitous, as they are simultaneously less popular and less similar to items on the front (fig. 1). To compute the final SPADE score, we calculate the average distance from the Pareto front only for the correctly recommended items (i.e., those in the user’s test set). 3.1 Defining the Dimensions For this Pareto formulation to be well-posed, the two dimensions must represent a genuine trade-off. It is therefore critical to choose a similarity measure that is not positively correlated with popularity. If a positively correlated measure such as cosine similarity is used Lin et al. (2024), the most popular items will inherently also score as the most similar. Geometrically, this causes the candidate items to cluster along a single diagonal, and the Pareto front collapses into a single point (or a very narrow band) in the extreme upper-right of the space. When the front collapses in this way, there is no longer a boundary of trade-offs, and measuring distance to the front becomes meaningless. We therefore require opposing measures to scatter items across the space, forcing a wide, distinct curve that represents the maximum possible trade-off between popularity and similarity. Popularity (x-axis): We define the popularity of a candidate item i by its interaction frequency, normalized to a range of [0,1][0,1]. During training, the occurrence nin_i of each item is counted and divided by the maximum item count across the entire item set I: pop(i)=nimaxk∈Inkpop(i)= n_i _k∈ In_k Similarity (y-axis): To satisfy the independent requirement, we utilize unnormalized pointwise mutual information (PMI). The PMI of two items is calculated based on the fraction of users who interacted with either and both items as follows: PMI(i,j)=lognnninnjn=logn1nninjPMI(i,j)= n_ijn n_in n_jn= n_ij 1nn_in_j where nin_i and njn_j indicate the number of users who have interacted with items i and j respectively, n_ij represents the number of users who interacted with both items, and n indicates the total number of users. It is well known that PMI is negatively correlated with popularity Bouma (2009), making it ideal for our use case. We intentionally avoid Normalized PMI Kaminskas and Bridge (2014); Kaminskas and Bridge (2016b), as the normalization process reintroduces positive correlation with popularity. To counteract PMI’s sensitivity to noise for rare items, we apply empirical Bayes smoothing Robbins (1992). This dampens the effect of low counts and shifts PMI towards 0 when evidence for the item pair is small. We also clip the similarities to be strictly non-negative, as negative similarities predominantly represent noise. The formula for our smoothed Positive PMI (PPMI) is: PPMIα(i,j)=max0,logn+α1nninj+αPPMI_α(i,j)= \0, n_ij+α 1nn_in_j+α \ We fix the smoothing strength at α=1α=1 in our experiments. To compute the overall similarity between a candidate item i and the user profile U, we aggregate the pairwise similarities by taking the sum over all items j in the user’s history: sim(i,U)=∑j∈UPPMIα(i,j)sim(i,U)= _j∈ UPPMI_α(i,j) 3.2 Space Normalization and Distance Calculation Because popularity and profile similarity naturally operate on different scales, calculating a distance metric would result in the larger-scaled axis dominating the space. To ensure both dimensions contribute equally, we apply Min-Max scaling to the profile similarities for each user, projecting the similarity axis into the same [0,1][0,1] range as the popularity axis. We denote this scaled similarity as sim~(i,U) sim(i,U). With both axes bounded in [0,1][0,1], we formally define the user-specific Pareto front UP_U. An item p belongs to UP_U if there exists no other item k such that both pop(k)>pop(p)pop(k)>pop(p) and sim~(k,U)>sim~(p,U) sim(k,U)> sim(p,U) hold true. Finally, the SPADE score of a candidate item i is calculated as the minimum Euclidean distance i to the Pareto front UP_U: SPADE(i,U)=minp∈U(pop(i)−pop(p))2+(sim~(i,U)−sim~(p,U))2SPADE(i,U)= _p _U (pop(i)-pop(p))^2+( sim(i,U)- sim(p,U))^2 A higher SPADE score indicates that an item is located further away from the Pareto-optimal boundaries of popularity and similarity, effectively quantifying its serendipitous value. The score for a user’s recommendation RUR_U is calculated as the average SPADE score over all correctly recommended items (i.e., the items in user U’s test set). To calculate the SPADE score for an algorithm, we average the scores over all the users in the test set. 4 Experimental Setup Our experiments demonstrate the effectiveness of SPADE in evaluating recommendation serendipity relative to popular serendipity metrics. We evaluate SPADE using baseline recommendation algorithms and common dataset configurations reported in the recommender systems literature, ensuring that SPADE’s performance can be applied to general recommender system research. Our analysis focuses on the top-N recommendation task, specifically examining how accurately the metric isolates genuinely unexpected items from easily predictable false positives based entirely on offline data. Datasets: We evaluate SPADE across five datasets from distinct domains: CiteULike Wang et al. (2013), MIND Wu et al. (2020), MovieLens-1M Harper and Konstan (2015), MovieLens-20M Harper and Konstan (2015), and Netflix22 2 https://w.kaggle.com/datasets/netflix-inc/netflix-prize-data. Table 1 shows statistical information on the datasets. Dataset # Interactions # Users # Items Avg.#Int. per user Avg.#Int. per item Sparsity CiteULike 200,251 5,550 15,439 36.08 12.97 99.77% MIND* 1,086,939 10,000 33,220 108.69 32.72 99.67% ML-1M 574,385 6,038 3,125 95.13 183.80 96.96% ML-20M* 730,034 10,000 11,361 73.00 64.26 99.36% Netflix* 1,234,880 10,000 15,093 123.49 81.82 99.18% Table 1: Statistical information on the datasets used in our experiments. Datasets marked with (*) have been subsampled to 10,000 users to keep within hardware limitations. Algorithms: We compute SPADE against recommendations predicted by five baseline algorithms: EASE, SLIM, Item-based k-Nearest Neighbors (ItemKNN), Popularity, and Random. We use the algorithm implementations from the Recpack library Michiels et al. (2022) and perform a grid search to tune the hyperparameters33 3 More details, like the search spaces, can be found in our GitHub repository: https://github.com/Maarten11/spade. Preprocessing Following common practice, we perform 5-core filtering across all datasets, pruning the data to ensure that each item contains at least 5 interactions Barkan et al. (2021); Melchiorre et al. (2022). For the most intensive data sets, MIND, MovieLens-20M and Netflix, we subsample the user populations due to hardware memory limitations, capping them at 10,000 users. Beyond-Accuracy Metrics: To evaluate the effectiveness of SPADE against existing standards, our experimental setup uses three established baseline metrics that align directly with the methodologies discussed in section 2. First, to represent the profile-centric evaluation approaches, we measure Co-Occurrence. This metric operationalizes collaborative filtering (CF) distance between a candidate item and a user’s historical profile, using Point-wise Mutual Information (PMI) to assess statistical unexpectedness. Second, to capture the comparative baseline methodology, we evaluate Primitivity. Following the framework established by Ge et al., this metric calculates serendipity as the proportion of recommendations that deviate from the outputs of a “primitive” system (which, for our experiments, is defined as a standard popularity-based recommender). Finally, we include Novelty, calculated as the inverse global popularity of the recommended items. Because profile-distance metrics (like Co-Occurrence) are often blind to global trends, Novelty serves as the traditional proxy for uncovering long-tail items, allowing us to independently isolate the global popularity dimension that SPADE structurally integrates. Pipeline: For our evaluation, we rely on the strong generalization data-split scenario44 4 https://recpack.froomle.ai/generated/recpack.scenarios.StrongGeneralization.html#recpack.scenarios.StrongGeneralization utilizing an 80/20 train-test split, whereas 20% of the train set is used as validation data. We evaluate both standard recommendation performance and all included metrics at cut-offs K∈1,2,3,5,10,20,30K∈\1,2,3,5,10,20,30\. Hardware: All experiments and evaluations were executed on an Apple Silicon M5 Pro with 64 GB of RAM. 5 Results & Discussion In this section, we empirically evaluate the efficacy of the SPADE metric at both the user and dataset levels. We first provide a qualitative validation of its underlying mechanics through specific user case studies, followed by a comprehensive cross-domain quantitative analysis comparing SPADE against popular beyond-accuracy metrics. 5.1 Qualitative Validation: The Mechanics of SPADE To illustrate the functionality and practical value of the SPADE metric, we examine the recommendation space for two distinct users who show serendipitous item interactions (User 1874 and User 2186). For both users, we visualize the item space by mapping normalized similarity to the user’s historical profile on the y-axis and global item popularity on the x-axis. In these visualizations, candidate items are represented by blue dots, predictions generated by the EASE baseline are marked in green, and ground truth interactions from the test set are denoted by red crosses. The Pareto front—representing the optimal trade-off between similarity and popularity—is delineated by a solid line. Figure 1: The y-axis denotes normalized item similarity to the user’s historical profile, and the x-axis denotes global item popularity. Candidate items are represented by blue dots, EASE baseline predictions in green, and ground truth interactions (test set) by red crosses. The solid line indicates the Pareto front maximizing similarity and popularity. More serendipitous items are in the bottom-left quadrant. These relevant but unexpected items lie furthest from the Pareto front and therefore receive the highest serendipity scores under the SPADE metric. In both cases, we observe a ground truth interaction that deviates significantly from the user’s typical behavior, falling far outside the high-similarity, high-popularity cluster near the Pareto front. Case Study 1 - User 1874: For User 1874, one test-set item stands out in the bottom-left quadrant of the plot in fig. 1: the 1997 Sci-Fi movie Retroactive. This user’s historical interactions are dominated by Drama, Crime, and Thriller films, with Sci-Fi movies accounting for only 9% of their viewing history. Correspondingly, the item exhibits a low similarity score of 0.46 and an exceptionally low popularity score of 0.005. Despite being neither popular globally nor aligned with the user’s established preferences, the user still interacted with it, proving its underlying relevance. Because this item lies the furthest distance from the Pareto front, SPADE identifies it as serendipitous and assigns it the highest serendipity score. This demonstrates SPADE’s ability to reward the successful recommendation of deeply serendipitous, relevant items. Case Study 2 - User 2186: User 2186 provides a contrasting example of serendipity that is even more extreme but equally valid. The standout ground truth item here is the 2000 crime movie The Way of the Gun. While this user primarily consumes Action and Adventure, Crime movies constitute 11.6 % of their interaction history. This item yields a similarity score of 0.38 and a popularity score of 0.05. If a recommender system were to successfully predict it, SPADE would award it the highest serendipity score for this user, acknowledging that the recommendation pushes the boundaries of their typical consumption habits while remaining relevant. Ultimately, these examples validate the core premise of SPADE: by measuring distance from the Pareto front, the metric successfully identifies and rewards items that break away from obvious popularity and historical-similarity biases, capturing serendipity. 5.2 Cross-Domain Empirical Evaluation Having established the foundational mechanics of the SPADE metric at the user level, we extend our evaluation to a cross-domain quantitative analysis to assess its robustness compared to traditional beyond-accuracy metrics. Figure 2: Performance comparison of baseline recommendation algorithms (EASE, SLIM, ItemKNN, and Popularity) across varying recommendation list sizes (K∈1,2,3,5,10,20,30K∈\1,2,3,5,10,20,30\). The subplots depict the evaluation trajectories for accuracy (NDCG) and beyond-accuracy metrics (Primitivity, Co-Occurrence Surprise, Novelty, and SPADE) as the cutoff threshold increases. Dataset Algorithm NDCGK Primitivity CoOccurence Novelty SPADE Netflix EASE 0.315685 4.844650 0.103907 1.168139 0.127368 Netflix SLIM 0.280544 4.544350 0.100814 1.110615 0.117351 Netflix ItemKNN 0.190461 3.705450 0.085582 0.993255 0.061379 Netflix Popularity 0.109781 0.000000 0.076752 0.783255 0.029467 Netflix Random 0.001734 5.498050 0.214283 inf 0.004001 ML-1M EASE 0.334812 4.396189 0.095179 1.094719 0.101459 ML-1M SLIM 0.299857 3.864540 0.089278 0.971579 0.085457 ML-1M ItemKNN 0.258572 3.752361 0.085493 1.013478 0.068455 ML-1M Popularity 0.149624 0.000000 0.068181 0.687975 0.030897 ML-1M Random 0.007858 5.476885 0.179450 2.941511 0.014116 MIND EASE 0.528621 5.281100 0.160480 1.738990 0.234498 MIND SLIM 0.468111 5.185100 0.156472 1.632805 0.210349 MIND ItemKNN 0.360481 5.016600 0.151883 1.917122 0.194649 MIND Popularity 0.063749 0.000000 0.116312 1.038346 0.036538 MIND Random 0.000617 5.498997 0.281789 inf 0.003692 ML-20M EASE 0.313071 4.349293 0.116474 1.230299 0.131990 ML-20M SLIM 0.277540 4.074596 0.114782 1.169247 0.115773 ML-20M ItemKNN 0.238583 4.194949 0.108970 1.134467 0.101111 ML-20M Popularity 0.127909 0.000000 0.087457 0.783185 0.031791 ML-20M Random 0.001366 5.496224 0.221319 inf 0.002831 CiteULike EASE 0.277414 5.330776 0.348068 3.494691 0.406407 CiteULike SLIM 0.193589 5.174007 0.324404 3.157389 0.340890 CiteULike ItemKNN 0.239903 5.459838 0.383043 3.975786 0.402518 CiteULike Popularity 0.011624 0.000903 0.253401 2.244291 0.015532 CiteULike Random 0.001361 5.496300 0.386524 4.097694 0.007649 Table 2: Cross-domain evaluation of baseline recommendation models (EASE, SLIM, ItemKNN, and Popularity) at a top-K cutoff of 10. The table details ranking accuracy alongside beyond-accuracy metrics to illustrate the trade-off between relevance and serendipitous discovery across the different datasets. Traditional metrics, such as Novelty, Primitivity, and Co-Occurrence, evaluate recommended items irrespective of their actual relevance to the user. Consequently, an algorithm can mathematically maximize traditional beyond-accuracy metric scores by simply recommending random, irrelevant items. The empirical evaluation clearly illustrates this vulnerability, e.g., on the MIND dataset, the Random algorithm yields a near-zero ranking accuracy (NDCG of 0.0006), yet achieves very high Primitivity (5.4989) and Co-Occurrence (0.281789) scores (table 2, fig. 2). In contrast, SPADE ensures relevance by calculating serendipity scores only for predicted items included in the test set. By accurately identifying random recommendations as unconstrained noise, SPADE successfully penalizes this lack of relevance and assigns a proportionally low serendipity score of 0.0036 (table 2, fig. 2). This strict relevance constraint also resolves a critical blind spot when evaluating high-performing models. While traditional metrics, e.g., show that EASE on ML-1M achieves both high ranking accuracy (NDCG of 0.3348) and high novelty (1.09, table 2), looking at these metrics in isolation cannot verify whether the novel recommendations are actually the ones the user found relevant. Finally, SPADE successfully passes sanity checks by confirming that the non-personalized Random and Popularity baselines yield minimal serendipitous value. True serendipity inherently requires recommending relevant items that avoid both high similarity to the user’s history and high global popularity. This expectation is perfectly mirrored by the Popularity and Random baselines, which consistently yield the lowest SPADE scores across all evaluated datasets (e.g., 0.0036 for Random on MIND, and 0.0294 for Popularity on Netflix). Conclusion In conclusion, SPADE provides a robust, empirical framework for algorithm evaluation in recommender systems. By mathematically grounding serendipity in test-set relevance, SPADE prioritizes relevance and penalizes highly personalized or popular recommendations. Consequently, SPADE enables researchers to reliably distinguish genuinely serendipitous recommendations from those that simply exploit isolated beyond-accuracy metrics. References Adamopoulos and Tuzhilin (2014) P. Adamopoulos and A. Tuzhilin On unexpectedness in recommender systems: or how to better expect the unexpected. ACM Transactions on Intelligent Systems and Technology (TIST) 5 (4), p. 1–32. Cited by: §2. Babu et al. (2015) L. D. Babu E. D. Raj et al. Evaluation of similarity based recommender using fuzzy inference system. In 2015 International Conference on Computing and Network Communications (CoCoNet), p. 576–581. Cited by: §2. Barkan et al. (2021) O. Barkan, R. Hirsch, O. Katz, A. Caciularu, and N. Koenigstein Anchor-based collaborative filtering. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, p. 2877–2881. Cited by: §4. Bouma (2009) G. J. Bouma Normalized (pointwise) mutual information in collocation extraction. External Links: Link Cited by: §3.1. Chen et al. (2021) M. Chen, Y. Wang, C. Xu, Y. Le, M. Sharma, L. Richardson, S. Wu, and E. Chi Values of user exploration in recommender systems. In Proceedings of the 15th acm Conference on recommender systems, p. 85–95. Cited by: §2. De Pessemier et al. (2014) T. De Pessemier, S. Dooms, and L. Martens Comparison of group recommendation algorithms. Multimedia tools and applications 72 (3), p. 2497–2541. Cited by: §1, §2. Ge et al. (2010) M. Ge, C. Delgado-Battenfeld, and D. Jannach Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ’10, New York, NY, USA, p. 257–260. External Links: ISBN 9781605589060, Link, Document Cited by: §1, §1, §2. Harper and Konstan (2015) F. M. Harper and J. A. Konstan The movielens datasets: history and context. ACM Trans. Interact. Intell. Syst. 5 (4). External Links: ISSN 2160-6455, Link, Document Cited by: §4. Jelassi et al. (2015) M. N. Jelassi, S. Ben Yahia, and E. Mephu Nguifo Towards more targeted recommendations in folksonomies. Social Network Analysis and Mining 5 (1), p. 68. Cited by: §2. Jenders et al. (2015) M. Jenders, T. Lindhauer, G. Kasneci, R. Krestel, and F. Naumann A serendipity model for news recommendation. In Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz), p. 111–123. Cited by: §1. Kaminskas and Bridge (2014) M. Kaminskas and D. Bridge Measuring surprise in recommender systems. In Proceedings of the workshop on recommender systems evaluation: dimensions and design (Workshop programme of the 8th ACM conference on recommender systems), Cited by: §1, §2, §3.1. Kaminskas and Bridge (2016a) M. Kaminskas and D. Bridge Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Trans. Interact. Intell. Syst. 7 (1). External Links: ISSN 2160-6455, Link, Document Cited by: §1, §1. Kaminskas and Bridge (2016b) M. Kaminskas and D. Bridge Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Transactions on Interactive Intelligent Systems (TiiS) 7 (1), p. 1–42. Cited by: §1, §2, §3.1. Kawamae (2010) N. Kawamae Serendipitous recommendations via innovators. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, p. 218–225. Cited by: §2. Kawamae (2015) N. Kawamae Real time recommendations from connoisseurs. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, p. 537–546. Cited by: §1, §2. Kaya and Kaleli (2022) T. Kaya and C. Kaleli A novel top-n recommendation method for multi-criteria collaborative filtering. Expert Systems with Applications 198, p. 116695. Cited by: §2. Kotkov et al. (2018) D. Kotkov, J. A. Konstan, Q. Zhao, and J. Veijalainen Investigating serendipity in recommender systems based on real user feedback. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC ’18, New York, NY, USA, p. 1341–1350. External Links: ISBN 9781450351911, Link, Document Cited by: §1. Kotkov et al. (2017) D. Kotkov, J. Veijalainen, S. Wang, T. A. Majchrzak, P. Traverso, K. Krempels, and V. Monfort A serendipity-oriented greedy algorithm for recommendations. Cited by: §2. Li and Tuzhilin (2024) P. Li and A. Tuzhilin Deep pareto reinforcement learning for multi-objective recommender systems. arXiv preprint arXiv:2407.03580. Cited by: §2. Lin et al. (2024) S. Lin, C. Gao, J. Chen, S. Zhou, B. Hu, Y. Feng, C. Chen, and C. Wang How do recommendation models amplify popularity bias? an analysis from the spectral perspective. arXiv preprint arXiv:2404.12008. Cited by: §3.1. Lin et al. (2019) X. Lin, H. Chen, C. Pei, F. Sun, X. Xiao, H. Sun, Y. Zhang, W. Ou, and P. Jiang A pareto-efficient algorithm for multiple objective optimization in e-commerce recommendation. In Proceedings of the 13th ACM Conference on recommender systems, p. 20–28. Cited by: §2. Maksai et al. (2015a) A. Maksai, F. Garcin, and B. Faltings Predicting online performance of news recommender systems through richer evaluation metrics. In Proceedings of the 9th ACM Conference on Recommender Systems, RecSys ’15, New York, NY, USA, p. 179–186. External Links: ISBN 9781450336925, Link, Document Cited by: §1. Maksai et al. (2015b) A. Maksai, F. Garcin, and B. Faltings Predicting online performance of news recommender systems through richer evaluation metrics. In Proceedings of the 9th ACM Conference on Recommender Systems, p. 179–186. Cited by: §2. Melchiorre et al. (2022) A. B. Melchiorre, N. Rekabsaz, C. Ganhör, and M. Schedl Protomf: prototype-based matrix factorization for effective and explainable recommendations. In Proceedings of the 16th ACM Conference on Recommender Systems, p. 246–256. Cited by: §4. Michiels et al. (2022) L. Michiels, R. Verachtert, and B. Goethals Recpack: an (other) experimentation toolkit for top-n recommendation using implicit feedback data. In Proceedings of the 16th ACM Conference on Recommender Systems, p. 648–651. Cited by: §4. Murakami et al. (2007) T. Murakami, K. Mori, and R. Orihara Metrics for evaluating the serendipity of recommendation lists. In Annual conference of the Japanese society for artificial intelligence, p. 40–46. Cited by: §1, §2. Nguyen et al. (2014) T. T. Nguyen, P. Hui, F. M. Harper, L. Terveen, and J. A. Konstan Exploring the filter bubble: the effect of using recommender systems on content diversity. In Proceedings of the 23rd International Conference on World Wide Web, W ’14, New York, NY, USA, p. 677–686. External Links: ISBN 9781450327442, Link, Document Cited by: §1. Nishioka et al. (2019) C. Nishioka, J. Hauk, and A. Scherp Towards serendipitous research paper recommender using tweets and diversification. In International Conference on Theory and Practice of Digital Libraries, p. 339–343. Cited by: §2. Nishioka et al. (2020) C. Nishioka, J. Hauke, and A. Scherp Influence of tweets and diversification on serendipitous research paper recommender systems. PeerJ Computer Science 6, p. e273. Cited by: §2. Peng et al. (2024) K. Peng, M. Raghavan, E. Pierson, J. Kleinberg, and N. Garg Reconciling the accuracy-diversity trade-off in recommendations. In Proceedings of the ACM Web Conference 2024, p. 1318–1329. Cited by: §2. Ren et al. (2015) Y. Ren, G. Li, and W. Zhou A survey of recommendation techniques based on offline data processing. Concurrency and Computation: Practice and Experience 27 (15), p. 3915–3942. Cited by: §2. Ribeiro et al. (2014) M. T. Ribeiro, N. Ziviani, E. S. D. Moura, I. Hata, A. Lacerda, and A. Veloso Multiobjective pareto-efficient approaches for recommender systems. ACM Transactions on Intelligent Systems and Technology (TIST) 5 (4), p. 1–20. Cited by: §2. Robbins (1992) H. E. Robbins An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, p. 388–394. Cited by: §3.1. Sá et al. (2022) J. Sá, V. Queiroz Marinho, A. R. Magalhães, T. Lacerda, and D. Goncalves Diversity vs relevance: a practical multi-objective study in luxury fashion recommendations. In Proceedings of the 45th International ACM SIGIR Conference on research and development in information retrieval, p. 2405–2409. Cited by: §2. Shani and Gunawardana (2010) G. Shani and A. Gunawardana Evaluating recommendation systems. In Recommender systems handbook, p. 257–297. Cited by: §1, §2. Wang et al. (2018) C. Wang, Z. Deng, J. Lai, and P. S. Yu Serendipitous recommendation in e-commerce using innovator-based collaborative filtering. IEEE transactions on cybernetics 49 (7), p. 2678–2692. Cited by: §1, §2, §2. Wang et al. (2013) H. Wang, B. Chen, and W. Li Collaborative topic regression with social regularization for tag recommendation. In IJCAI, Cited by: §4. Wu et al. (2020) F. Wu, Y. Qiao, J. Chen, C. Wu, T. Qi, J. Lian, D. Liu, X. Xie, J. Gao, W. Wu, et al. Mind: a large-scale dataset for news recommendation. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 3597–3606. Cited by: §4. Xi et al. (2025) Y. Xi, M. Weng, W. Chen, C. Yi, D. Chen, G. Guo, M. Zhang, J. Wu, Y. Jiang, Q. Liu, Y. Yu, and W. Zhang Bursting filter bubble: enhancing serendipity recommendations with aligned large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ’25, New York, NY, USA, p. 5059–5070. External Links: ISBN 9798400714542, Link, Document Cited by: §1. Xiao et al. (2017) L. Xiao, Z. Min, Z. Yongfeng, G. Zhaoquan, L. Yiqun, and M. Shaoping Fairness-aware group recommendation with pareto-efficiency. In Proceedings of the eleventh ACM conference on recommender systems, p. 107–115. Cited by: §2. Zhang et al. (2012) Y. C. Zhang, D. Ó. Séaghdha, D. Quercia, and T. Jambor Auralist: introducing serendipity into music recommendation. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM ’12, New York, NY, USA, p. 13–22. External Links: ISBN 9781450307475, Link, Document Cited by: §1. Ziarani and Ravanmehr (2021a) R. J. Ziarani and R. Ravanmehr Deep neural network approach for a serendipity-oriented recommendation system. Expert Systems with Applications 185, p. 115660. Cited by: §2. Ziarani and Ravanmehr (2021b) R. J. Ziarani and R. Ravanmehr Serendipity in recommender systems: a systematic literature review. Journal of Computer Science and Technology 36 (2), p. 375–396. Cited by: §1, §1.