Paper deep dive

Decomposing Staleness in Recommender Systems: A Dual-Filter Framework for Supersession and Decay

Di Bai, Feng Han, Zhenwei Tang, Jintao Liu, Luoshu Wang, Jialu Liu

Year: 2026Venue: arXiv preprintArea: cs.IRType: PreprintEmbeddings: 55

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/22/2026, 3:02:37 AM

Summary

The paper introduces SDF (Supersession-Decay Filtering), a system deployed in Google Discover to address content staleness in recommender systems. SDF decomposes staleness into two mechanisms: supersession (where new items render old ones obsolete) and relevance decay (where an item's value diminishes over time). It employs a relational staleness model using knowledge distillation from an LLM to detect supersession between item pairs, and a Predicted Traffic Ratio (PTR) model that uses multimodal content to forecast relevance decay. SDF prunes stale candidates upstream of ranking, reducing serving costs and significantly decreasing user-filed staleness reports by 54.9% over two years.

Entities (8)

Google Discover · platform · 99%SDF · system · 98%Relevance Decay · concept · 95%supersession · concept · 95%PTR Model · model · 92%Relational Staleness Model · model · 92%Knowledge Distillation · technique · 90%LLM · technology · 90%

Relation Signals (7)

SDF → deployedin → Google Discover

confidence 99% · SDF (Supersession-Decay Filtering), a staleness filtering system fully deployed in Google Discover

Relational Staleness Model → detects → supersession

confidence 95% · a relational staleness model that detects supersession between item pairs

PTR Model → forecasts → Relevance Decay

confidence 95% · a predicted traffic ratio (PTR) model that forecasts relevance decay from the item's content

SDF → targets → supersession

confidence 95% · SDF targets both mechanisms with complementary filters... a relational staleness model that detects supersession

SDF → targets → Relevance Decay

confidence 95% · SDF targets both mechanisms... and a predicted traffic ratio (PTR) model that forecasts relevance decay

SDF → reduces → staleness reports

confidence 93% · user-filed staleness reports (in-product user feedback) declined by 54.9% relative to the pre-deployment baseline

Relational Staleness Model → usestechnique → Knowledge Distillation

confidence 90% · we leverage a large language model (LLM) pipeline for synthetic data generation and apply knowledge distillation to a student model

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Stale recommendations are a pervasive challenge and a leading source of user complaints on large-scale content platforms. Items lose relevance through two primary mechanisms: supersession, where emerging updates render prior coverage stale, and relevance decay, where an item's informational value naturally diminishes over its lifecycle. Traditional countermeasures serve as crude proxies: age cutoffs poorly reflect actual relevance loss, while engagement heuristics rely on lagging signals, broadly exposing users to stale content before the system adapts. We present SDF (Supersession-Decay Filtering), a staleness filtering system fully deployed in Google Discover, a personalized recommendation feed with hundreds of millions of daily and billions of monthly active users. SDF targets both mechanisms with complementary filters, each powered by a learned model: a relational staleness model that detects supersession between item pairs, and a predicted traffic ratio (PTR) model that forecasts relevance decay from the item's content, trained on lifetime visit traffic. Applied via disjunction upstream of the ranking stage, SDF prunes stale candidates, measurably reducing downstream serving costs. Online experiments demonstrate that these filters significantly reduce the prevalence of stale content while improving user engagement. Over a two-year production deployment, user-filed staleness reports (in-product user feedback) declined by 54.9% relative to the pre-deployment baseline, establishing SDF as a robust and scalable paradigm for resolving content staleness at industrial scale.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

51,343 characters extracted from source content.

Expand or collapse full text

Decomposing Staleness in Recommender Systems: A Dual-Filter Framework for Supersession and DecayConference: Proceedings of the 35th ACM International Conference on Information and Knowledge Management; November 7–11, 2026; Rome, Italy.Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 7–11, 2026, Rome, ItalyDOI: 10.1145/3799682.3840082ISBN: 979-8-4007-2539-5/2026/11CCS: Information systems Recommender systemsCCS: Information systems Document filteringCCS: Information systems Novelty in information retrieval Di Bai Affiliation: Google LLC , Mountain View , USA email: vivianbai@google.com OrcID: 0009-0009-1813-4844 , Feng Han Affiliation: Google LLC , Mountain View , USA email: bladehan@google.com OrcID: 0009-0009-5584-4562 , Zhenwei Tang Affiliation: Google LLC , Toronto , Canada email: lilvjosephtang@google.com OrcID: 0000-0002-8742-9146 , Jintao Liu Affiliation: Google LLC , Mountain View , USA email: liujintao@google.com OrcID: 0009-0005-1325-0142 , Luoshu Wang Affiliation: Google LLC , Mountain View , USA email: luoshu@google.com OrcID: 0009-0009-6681-545X and Jialu Liu Affiliation: Google LLC , New York , USA email: jialu@google.com OrcID: 0000-0002-8721-8656 2026; © c Abstract. Stale recommendations are a pervasive challenge and a leading source of user complaints on large-scale content platforms. Items lose relevance through two primary mechanisms: supersession, where emerging updates render prior coverage stale, and relevance decay, where an item’s informational value naturally diminishes over its lifecycle. Traditional countermeasures serve as crude proxies: age cutoffs poorly reflect actual relevance loss, while engagement heuristics rely on lagging signals, broadly exposing users to stale content before the system adapts. We present SDF (Supersession-Decay Filtering), a staleness filtering system fully deployed in Google Discover, a personalized recommendation feed with hundreds of millions of daily and billions of monthly active users. SDF targets both mechanisms with complementary filters, each powered by a learned model: a relational staleness model that detects supersession between item pairs, and a predicted traffic ratio (PTR) model that forecasts relevance decay from the item’s content, trained on lifetime visit traffic. Applied via disjunction upstream of the ranking stage, SDF prunes stale candidates, measurably reducing downstream serving costs. Online experiments demonstrate that these filters significantly reduce the prevalence of stale content while improving user engagement. Over a two-year production deployment, user-filed staleness reports (in-product user feedback) declined by 54.9% relative to the pre-deployment baseline, establishing SDF as a robust and scalable paradigm for resolving content staleness at industrial scale. Keywords: Recommender Systems, Content Staleness, Content Filtering, Knowledge Distillation †c-license: by 1. Introduction Recommender systems are inherently time-sensitive. They surface content against an evolving notion of relevance, and when this evaluation lags reality, items become stale: they continue to surface despite no longer reflecting the current state of the underlying topic. In large-scale recommender systems like Google Discover, staleness is a pervasive challenge and a primary source of user dissatisfaction. Staleness is not a monolithic problem. Items lose relevance through distinct mechanisms that dictate both how staleness manifests and how it must be detected. This work addresses two primary dimensions: Supersession. An item abruptly becomes stale when a newly arriving item advances the underlying topic. For example, an early bulletin reporting that “Queen Elizabeth I is under medical supervision” was superseded within hours by the announcement of her passing; similarly, an early report that “Ohtani is weighing offers from multiple teams” was rendered stale the moment his signing with the Dodgers was announced. The staleness signal here is relational rather than absolute. Because only updates that contradict, supersede, or shift the narrative render an older item stale, detection requires comparing item pairs, not just by publication time (Figure 1, top). Relevance decay. An item naturally loses informational value over its lifecycle, driven by its own content rather than supersession. A guide to “tonight’s meteor shower” is highly useful for one evening before its relevance sharply drops, and a heartwarming local clip naturally fades without any follow-up. The staleness signal here is intrinsic to the item, tied directly to its intended lifespan (Figure 1, bottom). Figure 1. Two staleness mechanisms. Top: Supersession. Each new update on the Ohtani signing renders prior coverage stale. Bottom: Relevance decay. Time-sensitive items naturally lose value and become stale as they near the end of their useful lifespan, without any superseding counterpart.Two illustrations of content staleness. The top panel is a supersession timeline in which successive news items about Shohei Ohtani's team signing are placed along a left-to-right time axis, with arrows marking that each later item renders the earlier coverage stale. The bottom panel shows user-visit-over-time curves illustrating relevance decay: a time-sensitive item spikes sharply and then goes stale within a day, while a slower-decaying item rises and falls over a longer period before going stale, each marked stale where its traffic tails off. Traditional countermeasures address neither aspect effectively. Time-based ranking demotes rather than removes stale items, forcing users to still scroll past them. Hand-tuned age cutoffs are coarse and heavily rely on publication time; a long cutoff still serves stale items, while a short one drops useful evergreen content. Engagement-based heuristics, such as click-through rate floors, introduce structural lag: the system only learns an item is stale by repeatedly exposing users to a degraded experience. We address both mechanisms with SDF (Supersession-Decay Filtering), a unified staleness filtering system deployed in Google Discover, a personalized feed of open-web content in the Google Search app, serving hundreds of millions of daily and billions of monthly active users. While downstream ranking systems remain engagement-based and trend-aware, SDF acts as a proactive filtering layer. It prunes fundamentally stale content before it enters the computation-heavy ranking stage, providing a refined candidate set. SDF accomplishes this by decomposing the problem into two complementary components: Relational staleness filter for supersession. To capture this relational signal, we develop a pairwise classification framework with rationale supervision that determines whether a new arrival renders an existing item stale. Because manual annotation at scale is prohibitively expensive, we leverage a large language model (LLM) pipeline for synthetic data generation and apply knowledge distillation to a student model for scalable serving. Intrinsic staleness filter for relevance decay. To capture intrinsic decay, we introduce the predicted traffic ratio (PTR) model. Using an item’s multimodal content as input and trained on lifetime visit traffic, PTR forecasts relevance decay over future horizons. This component proactively filters items nearing the end of their lifespan, freeing serving capacity for fresher and more valuable content. SDF combines the two filters via disjunction upstream of the ranking stage, simultaneously improving user experience and serving efficiency. Across a two-year production deployment in Google Discover, SDF reduced user-filed staleness reports (in-product user feedback) by 54.9% (supersession by 64.0%, decay by 34.4%), validating the dual-filter framework for staleness filtering in recommender systems and broader content applications. 2. Related Work Staleness modeling in recommender systems. Staleness in recommender systems has primarily been studied indirectly through popularity decay rather than direct detection. One paradigm uses time as an explicit model feature (TimeSVD++ (Koren 2009)), with later work applying personalized time-decay to click-through rate (Yoneda et al. 2019) or tuning recency against relevance (Chakraborty et al. 2019). Another characterizes content lifecycles empirically: news shelf-life curves from social-media activity (Castillo et al. 2014), popularity trajectory prediction (Figueiredo et al. 2014), category-level lifecycle behavior (Gulla et al. 2016), and popularity dynamics (Wu and Huberman 2007; Yang and Leskovec 2011; Wu et al. 2023). Both lines share limitations for the staleness-filtering task: they require observed engagement, which lags, so stale content may still be served, and they model decay at coarse rather than per-item granularities. SDF’s PTR is designed for this task: inspired by data-quality currency metrics (Heinrich and Klier 2015), it predicts content-level per-item lifetime traffic ratios from multimodal content before engagement is observed. SDF deploys PTR as an upstream filter to prune stale candidates before they consume heavy compute in downstream serving, where compute costs scale at industrial volume. A separate line, document-level novelty detection (Bernstein and Zobel 2005; Soboroff and Harman 2005; Ghosal et al. 2022; Zhang et al. 2002), asks whether an incoming document adds new information to a corpus; SDF’s relational filter addresses the inverse problem of whether a new item renders an earlier one stale by direct pairwise relational judgment. Industrial practices on staleness and corpus filtering. Industrial recommender systems typically combine static age cutoffs with engagement-based heuristics for candidate management, particularly to control content freshness. This pattern recurs across diverse platforms: YouTube’s multi-funnel fresh-content stack (Wang et al. 2023) uses age and interaction-count thresholds for fresh-content slotting; Kuaishou’s cold-start pipeline (Chen et al. 2025) applies fixed exposure thresholds for cold-start gating; Twitter/X’s open-source algorithm (Twitter Engineering Team 2023) applies post-ranking heuristic filters, with audits documenting significant popularity amplification (Bouchaud et al. 2023; Ye et al. 2025). Despite the recurring pattern, content-level staleness modeling remains under-explored in industry. The reliance on heuristic pipelines leaves stale items to consume substantial downstream compute, representing meaningful efficiency headroom that proactive filtering can reclaim. SDF realizes this efficiency through two complementary filters. As a self-contained module that refines the candidate set, SDF presents a generalizable framework applicable to other industrial recommender systems. 3. Methodology SDF’s two filters, targeting supersession and relevance decay, are composed by disjunction. Section 3.1 formalizes both tasks; Sections 3.2 and 3.3 detail each model; Section 3.4 describes composition and serving. Figure 2. SDF end-to-end workflow. A candidate item dtd_t (arriving at time t) is evaluated in parallel. Top: Relational staleness filter for supersession. A student model predicts pairwise staleness, yielding Ssupersession(dt)S_supersession(d_t). Bottom: Intrinsic staleness filter for relevance decay. A multimodal model predicts per-horizon traffic ratios from the item’s content, yielding Sdecay(dt,t′)S_decay(d_t;t ) for decision time t′t . The two outputs are OR-fused into S⁡(dt,t′)S(d_t;t ); items flagged (S⁡(dt,t′)=1S(d_t;t )=1) are filtered, while the rest remain eligible for recommendation and pass to downstream serving.Block diagram of the SDF end-to-end workflow with two parallel branches evaluating a candidate item. The top branch is the relational staleness filter: a student model classifies whether newer topic-related items render the candidate stale, yielding a supersession indicator. The bottom branch is the intrinsic staleness filter: the item's text, image, and video are combined in a multimodal encoder and passed to a multi-task PTR network that predicts a series of per-horizon traffic ratios; these are then combined into a decay indicator at the current decision time. The two indicators are OR-fused into a joint staleness indicator that determines whether the item stays eligible for recommendation or is excluded from the candidate set. 3.1. Problem Formulation Notation. We use the following notation: • d: an item in the Discover feed. • ℛ⁡(d)R(d): items sharing the major topic with d (e.g., the same event or entity). • A: the candidate set from which the downstream ranker selects. SDF maintains A continuously, adding arriving items and removing those that become stale through either of two mechanisms. The relational staleness filter targets supersession: d is removed once a newer topic-related arrival renders it stale. The intrinsic staleness filter targets relevance decay: d is removed when its content-based prediction indicates it has decayed past its useful lifespan. The downstream ranker consumes only A, so removed items are never served. Task 1: Relational staleness detection. Learn a pairwise predictor (1) f⁡(d,d′)∈0,1f(d,d )∈\0,1\ that takes an existing item d and a newly arriving item d′∈ℛ⁡(d)d (d), and predicts whether d′d renders d stale. Once f⁡(d,d′)=1f(d,d )=1 for some d′d , we deem d stale and remove it from A. This task addresses the supersession aspect of staleness. Task 2: Intrinsic staleness detection. For a candidate item dtd_t arriving in the system at time t and a forward horizon τ (an elapsed time from arrival), learn a content-based predictor (2) g⁡(dt,τ)∈[0,1]g(d_t,τ)∈[0,1] that estimates the fraction of dtd_t’s expected lifetime traffic accumulating within [t,t+τ][t,t+τ]. At a later decision time t′>t >t, let τ=t′−tτ=t -t be dtd_t’s elapsed age. We remove dtd_t from A when its predicted lifetime traffic share at this age reaches a threshold θ, i.e., g⁡(dt,τ)≥θg(d_t,τ)≥θ, indicating dtd_t is predicted to have accumulated most of its lifetime traffic, with further serving yielding diminishing returns. This task addresses the relevance decay aspect of staleness. SDF fusion. Let ℛ>t(dt)⊆ℛ⁡(dt)R_>t(d_t) (d_t) denote items related to dtd_t that arrived after dtd_t. We formalize the two removal rules at decision time t′t : (3) Ssupersession(dt)=[∃d′∈ℛ>t(dt):f(dt,d′)=1]S_supersession(d_t)=1\! [\,∃\,d _>t(d_t):\;f(d_t,d )=1\, ] (4) Sdecay(dt;t′)=[g(dt,τ)≥θ],τ=t′−tS_decay(d_t;\,t )=1\! [\,g(d_t,τ)≥θ\, ], τ=t -t Their OR-fusion gives the joint staleness indicator S⁡(dt,t′)S(d_t;\,t ): (5) S⁡(dt,t′):=Ssupersession(dt)∨Sdecay(dt,t′)S(d_t;\,t ):=S_supersession(d_t)\, \,S_decay(d_t;\,t ) The candidate dtd_t remains in A when S⁡(dt,t′)=0S(d_t;\,t )=0. When either filter fires, S⁡(dt,t′)=1S(d_t;\,t )=1 and dtd_t is removed from A. 3.2. Relational Staleness Detection To realize the pairwise predictor f of Equation 1 under production resource constraints, we distill knowledge from an LLM teacher into a compact student model. The pipeline has two stages: synthesizing a balanced labeled pair corpus from the LLM teacher, then fine-tuning the student model on those labels (Figure 2). Synthetic data generation. Supervised learning would require a large labeled dataset, but manual curation is infeasible: each pair demands careful reading of two items. We therefore synthesize the training set in two steps: selecting which item pairs to label, and obtaining labels for them from an LLM teacher (Ding et al. 2023; He et al. 2024; Tan et al. 2024). Sampling. Curating the unlabeled pairs is non-trivial along two axes. The first is intra-pair similarity: if it is too high, the items are likely near-duplicates with equivalent information; if too low, they are unrelated and their staleness relation is trivial. Valuable positive pairs, where one item supersedes the other, live in a narrow mid-similarity band. The second is content redundancy: hot topics generally have more similar items, so without pre-filtering we risk generating many redundant pairs of low marginal value, even when each pair individually sits in the right similarity band. Algorithm 1 addresses both axes: it clusters items to limit redundancy, and upweights pairs in optimal similarity ranges, so positive pairs, despite comprising a small fraction of the candidate population, are over-represented to produce a balanced dataset. Algorithm 1 Generate unlabeled item pairs 1: Input: A set of items D 2: Output: A set of item pairs P 3: Cluster D and generate clusters C 4: for cluster c∈c do 5: select ckc^k diverse representative items from c, k∝|c|k |c| 6: end for 7: ℐ←⋃cckI← _cc^k 8: Similarity-score-to-weight map M=[σ1,σ2),w1;…M=\[ _1, _2),w_1;\,…\ 9: for d1∈ℐd_1 do 10: for d2∈ℐd_2 do 11: generate pair (d1,d2)(d_1,d_2) if d1≠d2d_1≠ d_2 and from different clusters 12: compute similarity σ for the pair; look up weight w←M⁡(σ)w← M(σ) 13: end for 14: end for 15: Generate P by weight-sampling over candidate pairs Labeling. LLMs can adapt to tasks by learning from in-context demonstrations (Brown et al. 2020; Wei et al. 2022). Given an unlabeled pair ⟨d,d′⟩ d,d and K demonstrations ⟨d(j),d′(j),sj,ej⟩j=1K\ d^(j),d (j),s_j,e_j \_j=1^K for the relational staleness task, where sjs_j is the class label and eje_j is a rationale, we format them into a prompt and ask the LLM to decode an output sequence containing both the class and the rationale. Mirroring conventional human annotation, where multiple raters’ independent judgments are aggregated by majority vote, we draw multiple independent samples from the LLM and aggregate them by majority voting (Wang et al. 2022) (Figure 2). Because each annotation carries both a class and a rationale, we apply two-stage voting: first over classes, then over the candidate rationales aligned with the winning class via a second LLM pass. The second stage matters because a rationale can be wrong even when the associated class is right. Model training. With class and rationale annotations per pair, we follow the knowledge distillation paradigm for LLMs (Xu et al. 2024), transferring the teacher’s pairwise staleness judgments into a compact student model via supervised fine-tuning on the synthesized labels. The student consumes a seed prompt: a fixed task prefix concatenated with the existing item d and the newly arriving item d′d . Conditioned on the seed prompt, the student decodes the rationale followed by the class in the format ‘RATIONALE. Therefore, CLASS’, patterned after Chain-of-Thought (CoT) decoding (Wei et al. 2022); the rationale is brief and supervised directly from the teacher (Hsieh et al. 2023) rather than emerging at inference time. We additionally incorporate Natural Language Inference (NLI) (Bowman et al. 2015) into training. NLI asks a model to judge the relationship between a hypothesis and a premise, structurally analogous to judging whether an arriving item supersedes an existing one, and offers a large, human-curated corpus of such pair-relational judgments. Our synthesized staleness labels are teacher-bounded and constrained by the pair-sampling distribution of Algorithm 1, whereas NLI exposes the student to a broader and independently curated distribution of pair-comparison examples, strengthening its ability to judge relations between two text inputs. 3.3. Intrinsic Staleness Detection The predicted traffic ratio (PTR) model realizes g⁡(dt,τ)g(d_t,τ) from Equation 2 using the content of dtd_t as input. Deployed as the intrinsic staleness filter, it complements the relational staleness filter by capturing items whose staleness arises from intrinsic decay. Training target and data. We take t to be dtd_t’s arrival time, identified with its indexing-pipeline annotation time, closely approximating its publication time. We use the user-visit distribution of an item as a natural proxy for its relevance decay: a time-sensitive item collects most of its lifetime visits within a short window after publication, while an evergreen item accrues visits more uniformly across its full lifespan. The predicted traffic ratio of an item dtd_t at horizon τ is the share of its lifespan visit traffic that has accumulated between t and t+τt+τ: (6) PTRτ(dt)=#visits in [t,t+τ]#visits over the item’s full lifespanPTR_τ(d_t)= \#\,visits in [t,\,t+τ]\#\,visits over the item's full lifespan and the model is trained so that g⁡(dt,τ)≈PTRτ(dt)g(d_t,τ) _τ(d_t). Two practical specializations adapt this conceptual definition to production use in Discover. First, the denominator is operationalized as a fixed 3030-day reference window, which covers the lifespan of most content in Discover. Second, the horizon τ is sampled at a small, human-interpretable set of offsets, τ∈12h, 1d, 3d, 5d, 7d, 10d, 14dτ\!∈\!\12h, 1d, 3d, 5d, 7d, 10d, 14d\, chosen to align with familiar content lifecycles: breaking news at the hour scale, daily reporting at 11 to 33 days, weekly content at 55 to 77 days, and slower-decaying pieces out to two weeks. A PTR7dPTR_7d close to 1.01.0 indicates the item is predicted to collect the bulk of its lifetime traffic within a week of publication and is unlikely to draw further traffic thereafter; a low value indicates an item still drawing traffic at that horizon. PTR’s training labels are derived from items’ lifetime visit traffic, sourced from anonymized user activity logs. We use search-click logs rather than raw page-view logs. Although the latter are higher-volume, they are biased by publisher promotion, making it difficult to separate time-sensitive and evergreen distributions. Search clicks instead reflect explicit, timely user intent: each click is an active user selection signaling user-perceived relevance rather than passive attention, aligning PTR’s training signal with the relevance-decay phenomenon it models. PTR labels are computed by dividing the visits within each horizon by the 30-day total, yielding labels non-decreasing in τ. We filter out items falling below a minimum-traffic floor to keep training labels statistically meaningful. The resulting dataset spans multiple languages and formats, giving the model coverage across Discover’s serving surfaces. Model architecture and filtering. We instantiate g⁡(dt,τ)g(d_t,τ) as a multimodal multi-task network with early fusion (intrinsic staleness in Figure 2). On the input side, text, image, and video tokens of the item are combined as a single input to a shared encoder, initialized from an internal multimodal pre-trained backbone, producing a joint item representation. Horizon-specific dense feed-forward prediction heads (one per horizon) are applied to this shared representation, yielding the full set of per-horizon predictions in a single forward pass. The network is trained end-to-end with mean-squared-error loss summed across horizons. Equation 4 expresses the decay decision at an idealized single horizon τ matching the item’s current age t′−t -t. In practice, g is trained at a fixed discrete set of horizons T rather than as a continuous function of τ. The training labels are monotonically non-decreasing in τ by construction, and g⁡(dt,τ)g(d_t,τ) tracks this monotonicity across T in practice. Equation 4 thus specializes to evaluating g at the largest trained horizon already elapsed by decision time t′t : (7) Sdecay(dt;t′)=[maxτ∈(g(dt,τ)⋅[t′>t+τ])≥θ]S_decay(d_t;\,t )=1\! [\, _τ (g(d_t,τ)·1[\,t >t+τ\,] )≥θ\, ] The inner indicator masks horizons not yet elapsed; the max over the masked g values isolates the largest elapsed horizon’s prediction. The rule fires when that prediction meets the threshold θ, i.e., the item has captured at least a fraction θ of its lifetime traffic and further serving offers diminishing value. 3.4. SDF Composition and Serving Both filters rely on unpersonalized, item-level signals: neither f nor g depends on the requesting user or any context other than time. They compute off the user-request path at a high enough cadence that cached outputs remain current, so applying Equation 5 at serving time reduces to a lookup against the request timestamp t′t , with no model inference at serving time. SDF evaluates millions of new items per day for staleness. PTR runs per-item on content alone, while the relational staleness filter runs pairwise classification, making it the compute-heavier of the two. To balance coverage with throughput, the relational staleness filter runs in two complementary modes. The online mode, optimized for labeling latency, subscribes to incoming items and labels time-sensitive ones in near real time, using a lightweight pre-filter to drop easy negatives (e.g., unrelated pairs), plus neighbor retrieval spanning different time buckets to prevent near-duplicate items from dominating. The batch mode, optimized for throughput, backfills the long tail and items missed by the online mode, extending a stale label to other items in the same cluster for broader coverage. 4. Experiments We evaluate SDF along three axes: per-filter offline evaluation (Section 4.1), online A/B testing under a continuous prevalence-delta framework that drives launch decisions in Discover (Section 4.2), and the two-year deployment outcome on user-filed reports and serving-cost savings (Section 4.3). 4.1. Offline Evaluation 4.1.1. Relational Staleness Filter Setup. Following internal institutional policies, we use PaLM 2-L (Anil et al. 2023) as the LLM annotator. We manually annotated 300 balanced pairs as a development golden set for prompt tuning. This set is held out and disjoint from the few-shot demonstrations provided in the LLM prompt. We iteratively tuned the prompt’s instructions and demonstrations until PaLM 2-L reached >90%>\!90\% precision and recall on this golden set. The tuned annotator draws 7 independent samples per pair and aggregates them by majority voting, producing labels for 1,100,133 examples (880,027 / 109,951 / 110,155 train/val/test). The annotator’s residual error rate is on par with audited noise in widely used benchmarks, e.g., ∼6% \!6\% on MMLU (Hendrycks et al. 2020; Gema et al. 2025). We instantiate the student model of Section 3.2 as T5 (Raffel et al. 2020), fine-tuned from a publicly available 11B checkpoint. We train with batch size 128, learning rate 1e−31e-3, 20k steps, and 1k warm-up steps. Hyper-parameters are tuned on the validation set, and metrics are reported on the test set using Accuracy (Acc.), Precision (Prec.), Recall (Rec.), and F1, with F1 as the primary metric. Results. Table 1 ablates the NLI auxiliary training and CoT-style rationale supervision. We label a configuration Disc (discriminative) when the student is trained with class-only supervision, and Gen (generative) when training also supervises the student on the teacher’s rationale as a CoT-style signal. NLI improves F1 in both regimes, consistent with the pair-relational transfer motivated in Section 3.2. The absolute margin is modest, but we report this ablation in full so that the design choice and its empirical support remain available to researchers and practitioners working on similar applied pipelines. Gen outperforms Disc both with and without NLI, indicating that supervising the student on the teacher’s rationale at training time, rather than relying on a chain-of-thought to emerge at inference, helps independently of the auxiliary NLI training. The adopted Gen + NLI configuration achieves the best offline performance. We further validate the deployed model on a held-out 550-pair human-rated golden test set sampled from Discover with balanced positives and negatives, disjoint from the development and training sets, achieving 83.5%83.5\% precision and %95\% recall. Table 1. Relational staleness offline evaluation. Disc (discriminative): class-only supervision; Gen (generative): CoT-style rationale supervision; NLI: auxiliary pair-relational training. Per column, bold marks the best value and underline the second best. Acc. Prec. Rec. F1 Disc 83.71 83.68 85.15 84.41 Disc + NLI 83.86 84.25 84.65 84.45 Gen 83.37 81.31 88.14 84.59 Gen + NLI 84.16 84.31 85.28 84.79 Case studies. Table 2 shows CoT-style decoded outputs from the relational staleness filter, illustrating a practical advantage of the generative formulation: each prediction is accompanied by a model-decoded rationale that exposes why the arriving item renders the existing one stale, or why it does not. Article R: The Oakland Athletics are looking to continue their winning streak against the struggling Texas Rangers… Article A: Rangers hope to have stabilized following series victory over Oakland. The Texas Rangers came away with a series victory… Output: Article R is about the Oakland Athletics’ goal of winning a series against the Texas Rangers. Article A reports on the outcome of that series, making Article R outdated. Therefore, Stale. Article R: Fixed rate cliff looms over Australian housing market. New CoreLogic report sheds light on emerging risk… Article A: CoreLogic has released a new report unpacking how the housing market has evolved in the three years since COVID was declared a global pandemic.… Output: Both articles discuss the Australian housing market. Article R focuses on the fixed rate cliff, while Article A provides an overview of the market’s performance over the past three years. Since Article A does not mention the fixed rate cliff, it does not make Article R stale. Therefore, Fresh. Table 2. CoT-style decoded outputs from the relational staleness filter. “Article R” is the relevant existing item; “Article A” is the arriving new item. 4.1.2. Intrinsic Staleness Filter Architectural ablation: late vs. early fusion. We compare a late-fusion baseline (text, image, and video processed by separate per-modality encoders whose outputs are concatenated and fed into per-horizon dense heads) against the deployed early-fusion design of Section 3.3 (text, image, and video tokens combined as a single input to a shared multimodal encoder before the prediction heads). Both variants are trained on identical data with matched optimizer and capacity budgets. On a curated, balanced pool of items with editorially labeled time-sensitivity, we classify each item as time-sensitive iff g⁡(dt,7d)≥θg(d_t,7d)≥θ. The deployed early-fusion model reaches %80\% accuracy, vs. 76%76\% for the late-fusion baseline. The four-point gap supports our early-fusion choice: jointly processing modalities outperforms per-modality late concatenation. Stale-item classification. A pairwise held-out evaluation analogous to the relational staleness classifier’s is not well-defined for PTR, since decay expresses itself over an item’s lifetime traffic curve rather than against a fixed pair-level label. We therefore evaluate PTR directly as a stale-item classification task, matching its deployment as a binary classifier at threshold θ. From a held-out pool labeled by 5 independent human raters per item, we evaluate the PTR rule of Equation 7 at θ=0.9θ\!=\!0.9, chosen from the operating range 0.85,0.9,0.95,0.98\0.85,0.9,0.95,0.98\ to keep precision high with adequate coverage. We sweep the rating-majority cut at three tiers: Majority ≥3≥\!3/5, Strong ≥4≥\!4/5, and Unanimous 5/5 raters agreeing whether the item is stale, with items lacking strong agreement excluded from the pool at each cut. All metrics rise as the cut tightens (Figure 3): tighter agreement shrinks the ground-truth set to the clearest cases, which the fixed classifier identifies most reliably. F1 rises from 78.07%78.07\% at the Majority cut to 87.50%87.50\% at the Unanimous cut; precision exceeds %88\% across all cuts. Grouped bar chart of PTR stale-item classification quality, reporting accuracy, precision, recall, and F1 at three rating-agreement cuts: Majority (at least 3 of 5 raters), Strong (at least 4 of 5), and Unanimous (5 of 5). Within each metric the bars increase as agreement tightens; F1 rises from about 78 percent at the Majority cut to about 88 percent at the Unanimous cut, and precision rises from about 88 percent to about 97 percent. Figure 3. PTR stale-item classification quality at varying rating-majority cuts; the deployed classifier (θ=0.9θ\!=\!0.9) is held fixed. All metrics rise because tighter agreement isolates higher-confidence cases.Grouped bar chart of PTR stale-item classification quality, reporting accuracy, precision, recall, and F1 at three rating-agreement cuts: Majority (at least 3 of 5 raters), Strong (at least 4 of 5), and Unanimous (5 of 5). Within each metric the bars increase as agreement tightens; F1 rises from about 78 percent at the Majority cut to about 88 percent at the Unanimous cut, and precision rises from about 88 percent to about 97 percent. 4.2. Online A/B Testing Continuous prevalence-delta framework. User-filed staleness reports are too sparse to clear the statistical confidence bar in any single A/B test arm, so we measure online staleness by rating items sampled weekly from each A/B test’s control and experiment arms across 4 consecutive weeks. Sampling is based on each item’s view delta between arms, restricted to items above a minimum view delta and weighted by delta magnitude: promoted items received more views in the experiment arm than in the control arm, while demoted items received fewer. This differential sampling concentrates the rating budget on items the experiment arm actually moved. Each sampled item, paired with a representative view time, is rated by 5 independent human raters on an ordinal scale from Not at all stale to Completely stale, binarized at a fixed cutoff; an item is stale when at least 3 of 5 raters rate it stale. Experiment metrics collapse supersession and decay staleness into a single composite metric for launch interpretability. After rating, we compute the stale rate (fraction of stale items) separately for the promoted and demoted sets. The launch metric is the prevalence delta, the difference between these two rates: (8) Δstale=rpromoted−rdemoted _stale=r_promoted-r_demoted where rXr_X is the stale rate in set X. A negative, statistically significant Δstale _stale means stale content is more prevalent in the demoted set than the promoted set, indicating the experiment arm reduces staleness; a positive value indicates the reverse. Per-week ratios are averaged for launch reporting. Filter A/B results. Table 3 compares SDF and its two component filters individually alongside a baseline of two heuristics commonly used in industry, age cutoffs and an engagement cliff, in the initial locale experiment. Configurations are tuned to be engagement-neutral; trivially aggressive cuts could reduce staleness much further but at the cost of substantially shrinking the candidate set and degrading user engagement, which we explicitly rule out. Within this engagement-neutral constraint, each SDF filter individually drives a statistically significant negative Δstale _stale, while the full SDF achieves the strongest reduction. This indicates that the two filters target largely distinct staleness mechanisms and work well together. Table 3. Prevalence delta of SDF and its component filters individually, alongside an age-cutoff + engagement-cliff baseline. Negative Δstale _stale means the promoted set is less stale than the demoted set. For Δstale _stale, bold marks the best result. Filter Promoted Demoted Δstale _stale Baseline 4.00±0.86%4.00± 0.86\% 4.10±0.87%4.10± 0.87\% −0.10±1.22%-0.10± 1.22\% Relational 1.68±0.81%1.68± 0.81\% 5.87±2.11%5.87± 2.11\% −4.19±2.25%-4.19± 2.25\% Intrinsic 7.90±1.67%7.90± 1.67\% 11.00±1.94%11.00± 1.94\% −3.10±2.56%-3.10± 2.56\% SDF 4.33±0.89%4.33± 0.89\% 11.24±2.04%11.24± 2.04\% −6.91±2.23%-6.91± 2.23\% Long-term joint-holdback experiment. A two-month joint-holdback experiment, in which both SDF filters are disabled for a held-out user population, quantifies the user-side impact alongside the staleness movement. Compared to the holdback, SDF drives a −0.16±0.07%-0.16± 0.07\% reduction in user dismissal rate, a +0.13±0.04%+0.13± 0.04\% lift in feed diversity, and a +0.26±0.18%+0.26± 0.18\% lift in feed positive engagement; all ± ranges report 95% confidence intervals. The pattern is consistent with long-term system health: users dismiss fewer items, feed diversity ticks up, and positive engagement lifts, despite the reduction in candidate volume from filtering. 4.3. Online Deployment User-reported staleness over a 2-year window. We monitor user-filed staleness reports over a two-year window of SDF deployment, covering successive model upgrades and locale expansion, with a stable user base (Figure 4). Reports are filed by selecting stale from a quality-issue menu. Each is then categorized as supersession or decay. We track both 7-day and 28-day rolling averages; the 28-day view is the official monitoring metric, smoothing weekly and event-driven variance. Compared to the start-of-window pre-deployment baseline, total reports fell by 54.9%54.9\%, supersession by 64.0%64.0\%, and decay by 34.4%34.4\%, each primarily driven by its corresponding filter, consistent with SDF’s architectural decomposition. Figure 4. User-filed staleness reports over the two-year SDF deployment in Discover, by mechanism: total (blue), supersession (red), decay (yellow). Top: 7-day rolling average. Bottom: 28-day rolling average.Two line charts of user-filed staleness report volumes over the two-year SDF deployment, each plotting three curves: total reports (blue), supersession reports (red), and decay reports (yellow), against time on the horizontal axis. The top chart shows the 7-day rolling average and the bottom chart the 28-day rolling average. All three curves trend downward over the window, with the total and supersession curves declining most, indicating fewer staleness reports after deployment. Wins and losses. We present representative production cases that illustrate SDF’s wins and losses. • Wins. The relational staleness filter correctly flags pairwise news supersession: “House GOP leaders float new plan” (existing) is flagged stale by “House Republicans failed to agree on a spending plan” (arriving), and “Mazda 3 IPM launching soon?” (existing) by “Mazda 3 prices revealed” (arriving). PTR correctly drops single-day items (Geminid meteor-shower guide) after their event window, retains multi-day items (Phuket food-festival guide) during their relevance period, and leaves evergreen items (NYT health-coach explainer) unfiltered. • Losses. (i) The relational staleness filter’s pairwise judgment can be ambiguous on certain items such as political commentary, where content tends to mix opinion and fact and no single item cleanly supersedes another. (i) PTR misses a long tail of gradually-decaying items: their predicted traffic ratio never reaches the threshold, and some become moderately stale without being caught. (i) A substantial fraction of residual user-filed staleness reports corresponds to content the user has already seen on Discover or elsewhere. Such user-perceived staleness lies outside SDF’s supersession/decay scope. Serving-cost savings. SDF prunes candidates, avoiding heavy downstream workloads such as dense scoring. Monitored over Q1 2026, the relational staleness filter saves 4.76%4.76\% in serving CPU and TPU hours, while the intrinsic staleness filter saves an additional 3.98%3.98\%. Across a continuously updating corpus, these efficiency gains translate directly into proportional cost reductions. Continuing development. SDF has been deployed across a two-year window marked by rapid LLM advances. We have been progressively upgrading the relational staleness model with newer LLM releases; closing the gap to the latest LLM capabilities within serving’s latency and cost budgets remains an ongoing effort. In parallel, we are actively augmenting PTR with multimodal LLM reasoning and extending it toward continuous-horizon lifespan representation learning. 5. Conclusion Recommender systems contend with continuously shifting relevance. We present SDF, which reframes staleness as a joint property of an item and its environment, with two complementary dimensions: a relational dimension capturing supersession by newer items, and an intrinsic dimension capturing relevance decay over the item’s own lifecycle. We address the relational dimension with an LLM-distilled pairwise model and the intrinsic dimension with a multimodal predicted traffic ratio model, composing the two via OR-fusion to check content-serving eligibility. Across a two-year deployment in Google Discover, a personalized recommendation feed serving hundreds of millions of daily and billions of monthly active users, SDF reduced user-filed staleness reports by 54.9%54.9\%, eased user dissatisfaction, and enhanced positive engagement. Beyond Discover, SDF establishes the supersession-decay paradigm as a principled, scalable, and widely applicable approach to staleness in recommender systems and broader content applications. GenAI Usage Disclosure The authors acknowledge the use of Google Gemini to assist with the refinement of the manuscript. All AI-assisted revisions were manually reviewed by the authors, who assume full responsibility for the accuracy and integrity of the final work. References (1) Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403 (2023). Bernstein and Zobel (2005) Yaniv Bernstein and Justin Zobel. 2005. Redundant documents and search effectiveness. In Proceedings of the 14th ACM international conference on Information and knowledge management. 736–743. Bouchaud et al. (2023) Paul Bouchaud, David Chavalarias, and Maziyar Panahi. 2023. Crowdsourced audit of Twitter’s recommender systems. Scientific Reports 13, 1 (2023), 16815. Bowman et al. (2015) Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 conference on empirical methods in natural language processing. 632–642. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. Castillo et al. (2014) Carlos Castillo, Mohammed El-Haddad, Jürgen Pfeffer, and Matt Stempeck. 2014. Characterizing the life cycle of online news stories using social media reactions. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 211–223. Chakraborty et al. (2019) Abhijnan Chakraborty, Saptarshi Ghosh, Niloy Ganguly, and Krishna P Gummadi. 2019. Optimizing the recency-relevance-diversity trade-offs in non-personalized news recommendations. Information Retrieval Journal 22, 5 (2019), 447–475. Chen et al. (2025) Gaode Chen, Ruina Sun, Yinjie Jiang, Tianxiang Li, Yinlong Dai, Qifan Shi, Xiaojie Qin, Jialong Fu, Piaoyi Chen, Ronggeng Huang, et al. 2025. A Cold-start Recommendation System at Kuaishou Designed from the Short-video Perspective. In Companion Proceedings of the ACM on Web Conference 2025. 124–132. Ding et al. (2023) Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 11173–11195. Figueiredo et al. (2014) Flavio Figueiredo, Marcos André Gonçalves, and Jussara M Almeida. 2014. Improving the effectiveness of content popularity prediction methods using time series trends. arXiv preprint arXiv:1408.7094 (2014). Gema et al. (2025) Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. 2025. Are we done with mmlu?. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 5069–5096. Ghosal et al. (2022) Tirthankar Ghosal, Tanik Saikh, Tameesh Biswas, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Novelty detection: A perspective from natural language processing. Computational Linguistics 48, 1 (2022), 77–117. Gulla et al. (2016) Jon Atle Gulla, Cristina Marco, Arne Dag Fidjestøl, Jon Espen Ingvaldsen, and Özlem Özgöbek. 2016. The Intricacies of Time in News Recommendation.. In UMAP (Extended Proceedings). He et al. (2024) Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. Annollm: Making large language models to be better crowdsourced annotators. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track). 165–190. Heinrich and Klier (2015) Bernd Heinrich and Mathias Klier. 2015. Metric-based data quality assessment—Developing and evaluating a probability-based currency metric. Decision Support Systems 72 (2015), 82–96. Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020). Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023. 8003–8017. Koren (2009) Yehuda Koren. 2009. Collaborative filtering with temporal dynamics. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 447–456. Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67. Soboroff and Harman (2005) Ian Soboroff and Donna Harman. 2005. Novelty detection: the trec experience. In Proceedings of human language technology conference and conference on empirical methods in natural language processing. 105–112. Tan et al. (2024) Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 930–957. Twitter Engineering Team (2023) Twitter Engineering Team. 2023. Twitter’s Recommendation Algorithm (Open-Source Release). Twitter Engineering Blog and GitHub repositories twitter/the-algorithm and twitter/the-algorithm-ml. https://blog.x.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm Wang et al. (2023) Jianling Wang, Haokai Lu, Sai Zhang, Bart Locanthi, Haoting Wang, Dylan Greaves, Benjamin Lipshitz, Sriraj Badam, Ed H Chi, Cristos J Goodrow, et al. 2023. Fresh Content Needs More Attention: Multi-funnel Fresh Content Recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5082–5091. Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837. Wu and Huberman (2007) Fang Wu and Bernardo A Huberman. 2007. Novelty and collective attention. Proceedings of the National Academy of Sciences 104, 45 (2007), 17599–17601. Wu et al. (2023) Leilei Wu, Lingling Yi, Xiao-Long Ren, and Linyuan Lü. 2023. Predicting the popularity of information on social platforms without underlying network structure. Entropy 25, 6 (2023), 916. Xu et al. (2024) Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116 (2024). Yang and Leskovec (2011) Jaewon Yang and Jure Leskovec. 2011. Patterns of temporal variation in online media. In Proceedings of the fourth ACM international conference on Web search and data mining. 177–186. Ye et al. (2025) Jinyi Ye, Luca Luceri, and Emilio Ferrara. 2025. Auditing political exposure bias: Algorithmic amplification on Twitter/X during the 2024 US presidential election. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. 2349–2362. Yoneda et al. (2019) Takeshi Yoneda, Shunsuke Kozawa, Keisuke Osone, Yukinori Koide, Yosuke Abe, and Yoshifumi Seki. 2019. Algorithms and system architecture for immediate personalized news recommendations. In IEEE/WIC/ACM International Conference on Web Intelligence. 124–131. Zhang et al. (2002) Yi Zhang, Jamie Callan, and Thomas Minka. 2002. Novelty and redundancy detection in adaptive filtering. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval. 81–88.

Instant research discovery