Paper deep dive
Back to All-Entity Ranking: Sampler-Dependent Evaluation in Continuous-Time Dynamic Graphs
Minwoo Yu, Young-guk Ha
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/1/2026, 2:28:19 AM
Summary
This paper investigates the impact of negative sampling strategies on the evaluation of Continuous-Time Dynamic Graphs (CTDGs). It demonstrates that sampled-negative metrics (like MRR with K=20) are unstable and can lead to incorrect model rankings compared to all-entity ranking. The authors argue that non-uniform negative distributions and finite candidate sets distort the Bayes-optimal ranking, often conflating pair-history membership signals with actual predictive performance. They recommend all-entity ranking over a fixed catalog as the primary evaluation method for CTDG benchmarks to ensure reliable architecture comparisons.
Entities (9)
Relation Signals (6)
All-Entity Ranking → recommendedfor → CTDG Evaluation
confidence 95% · We therefore recommend all-entity ranking as the primary evidence for architecture comparisons on CTDG benchmarks with an enumerable, fixed destination catalog.
LastFM → usedin → CTDG Evaluation
confidence 95% · Across six models on LastFM, MOOC, Reddit, and Wikipedia...
Negative Sampling → causesinstabilityin → Model Rankings
confidence 92% · even a finite candidate set drawn uniformly can destabilize model rankings and measured module effects.
MRR → affectedby → Candidate Set Size
confidence 90% · The measured effect of the same module also changes in magnitude and direction with the candidate-set size and training objective.
CRAFT → uses → Pair History
confidence 90% · CRAFT uses the candidate destination as a query to compress the source’s recent history through cross-attention.
Negative Sampling → confounds → Pair History Signals
confidence 88% · Under random candidates, these two signals often point toward the correct answer in the same direction, so sampled MRR alone cannot identify their respective contributions.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Next-destination prediction in continuous-time dynamic graphs (CTDGs) commonly ranks an observed interaction against sampled negative destinations. The resulting score is conditional on both the negative distribution and the number of candidates chosen by the researcher. We show that a non-uniform negative distribution changes the Bayes-optimal ranking, while even a finite candidate set drawn uniformly can destabilize model rankings and measured module effects. Time-varying source-destination history membership and model operations that use this information directly transmit the sampler's influence to the evaluation score. We examine this mechanism using a factorial evaluation of repeated and new positives against seen and unseen negatives, a minimal scorer based solely on pair-history membership, and controlled representation interventions. Across six models on LastFM, MOOC, Reddit, and Wikipedia, at least one model pair changes relative order between the expected Uniform-20 metric and the full catalog on three of the four datasets. The measured effect of the same module also changes in magnitude and direction with the candidate-set size and training objective. These results establish that model-superiority and ablation conclusions from sampled-negative benchmarks are conditional on the stated candidate configuration. All-entity ranking evaluates every destination in a fixed catalog, eliminating negative-selection freedom and sampling variation while retaining the original CTDG scorer. We therefore recommend all-entity ranking as the primary evidence for architecture comparisons on CTDG benchmarks with an enumerable, fixed destination catalog.
Tags
Links
- Source: https://arxiv.org/abs/2607.27861v1
- Canonical: https://arxiv.org/abs/2607.27861v1
Trouble viewing inline? Open PDF directly →
Full Text
50,996 characters extracted from source content.
Expand or collapse full text
Back to All-Entity Ranking: Sampler-Dependent Evaluation in Continuous-Time Dynamic Graphs Minwoo Yu and Young-guk Ha The authors are with the Smart Computing Laboratory, Department of Computer Science and Engineering, Konkuk University, Seoul 05029, Republic of Korea (e-mail: snowypainter@konkuk.ac.kr; ygha@konkuk.ac.kr).Young-guk Ha is the corresponding author. Abstract Next-destination prediction in continuous-time dynamic graphs (CTDGs) commonly ranks an observed interaction against sampled negative destinations. The resulting score is conditional on both the negative distribution and the number of candidates chosen by the researcher. We show that a non-uniform negative distribution changes the Bayes-optimal ranking, while even a finite candidate set drawn uniformly can destabilize model rankings and measured module effects. Time-varying source–destination history membership and model operations that use this information directly transmit the sampler’s influence to the evaluation score. We examine this mechanism using a factorial evaluation of repeated and new positives against seen and unseen negatives, a minimal scorer based solely on pair-history membership, and controlled representation interventions. Across six models on LastFM, MOOC, Reddit, and Wikipedia, at least one model pair changes relative order between the expected Uniform-20 metric and the full catalog on three of the four datasets. The measured effect of the same module also changes in magnitude and direction with the candidate-set size and training objective. These results establish that model-superiority and ablation conclusions from sampled-negative benchmarks are conditional on the stated candidate configuration. All-entity ranking evaluates every destination in a fixed catalog, eliminating negative-selection freedom and sampling variation while retaining the original CTDG scorer. We therefore recommend all-entity ranking as the primary evidence for architecture comparisons on CTDG benchmarks with an enumerable, fixed destination catalog. I Introduction A continuous-time dynamic graph (CTDG) observes an interaction stream ei=(ui,vi,ti)e_i=(u_i,v_i,t_i) in chronological order and predicts the next destination from the history available at the current time. A common evaluation ranks the observed destination v+v^+ together with sampled negative destinations [14, 12, 6]. The evaluator draws K negative destinations from v1−,…,vK−∼q(v∣u,t,ℋt),v_1^-,…,v_K^- q(v u,t,H_t), (1) where q is the probability distribution that selects a negative destination given source u, time t, and past history ℋtH_t, and K is the number of negatives compared with one positive. Random-20, for example, uses a q that draws destinations uniformly from a fixed candidate space and sets K=20K=20. This study considers CTDG next-destination prediction in which source and destination roles are distinct and the destination catalog defined by the training prefix is shared at test time. The JODIE interaction datasets Wikipedia, Reddit, MOOC, and LastFM follow this setting. Open-world graphs with rapidly growing node sets, homogeneous graphs in which source and destination roles are interchangeable, and tasks with event-specific eligibility require separate risk-set definitions. The Dynamic Graph Benchmark (DGB) established that uniform random negatives contain many easy edges and that historical and inductive negatives alter both model performance and model order [12]. The Temporal Graph Benchmark (TGB) and subsequent evaluation studies likewise reported changes in difficulty and ranking under different candidate constructions [6, 13, 3]. In recommendation, sampled metrics have been shown theoretically and empirically to disagree with full-item rankings and potentially fail to preserve the relative order of models [7, 2]. We connect this established evaluation discrepancy to time-dependent pair history and architecture conclusions in CTDGs. Deriving the Bayes-optimal score for sampled ranking shows that a non-uniform q, whose probability varies across candidates, orders destinations by p+(v∣x)/q(v∣x)p_+(v x)/q(v x). A uniform q over a fixed catalog preserves the Bayes-optimal full-catalog order, but MRR measured with finite K observes only a nonlinear partial view of that order. Consequently, the same checkpoint and scorer can yield different MRR values, model rankings, and module effects as q and K change. In CTDGs, this phenomenon interacts with time-varying pair membership. Pair membership is the binary indicator of whether the current source and a candidate destination have interacted previously. Recent models jointly represent the histories of two nodes or directly inject recurrence history for the current pair into the scoring path [18, 20, 17, 19]. In a small random candidate set, a positive is often a previously observed pair whereas the negatives are previously unseen pairs. Coarse separation of the seen and unseen groups can then produce a high rank, while the ability to choose the actual destination within either group remains conflated in a single MRR. We examine this connection in four steps. First, we derive the Bayes-optimal ranking from the posterior over a sampled candidate set and distinguish uniform from non-uniform q. Second, we compare the optimal coefficient of a minimal scorer based only on pair-history membership with the coefficient learned from data. Third, a fully crossed 2×22× 2 evaluation of repeated/new positives and seen/unseen negatives, together with an orthogonal representation projection, separates subgroup separation from within-group ranking. Fourth, using identical score vectors from six CTDG models, we compute sampled MRR for K∈1,5,20,50,100K∈\1,5,20,50,100\ and MRR over the entire fixed training catalog. The principal empirical result is that model ranking and measured module effect depend on q and K. On LastFM, expected Uniform-20 ranks CRAFT above DyGFormer, whereas the full catalog ranks DyGFormer above CRAFT. The ordering of the three leading models also changes on MOOC. On Wikipedia, the expected gain of CRAFT-R over CRAFT moves from +0.0037+0.0037 with 20 negatives to −0.0266-0.0266 over the full catalog. The former has a seed-paired confidence interval containing zero, while the latter is negative. Beyond the arithmetic decline in MRR caused by enlarging a candidate set, the conclusion about which model or design component is superior can therefore change. This finding also bears directly on causal explanations of sampled-negative improvements. CRAFT attributes the advantage of candidate-to-history cross-attention to learning compatibility between the destination and the source history [17]. Yet the same operation that queries history with candidate identity also conveys whether the source–destination pair has appeared before. Under random candidates, these two signals often point toward the correct answer in the same direction, so sampled MRR alone cannot identify their respective contributions. Our membership controls and representation interventions isolate this alternative path, while all-entity ranking enables architecture comparison without relying on a sampled candidate composition. • Theory. A non-uniform q changes the Bayes target, whereas uniform finite-K evaluation destabilizes model and module comparisons. • Mechanism. Time-varying pair membership and pair-history operations amplify this effect. • Recommendation. Architecture comparisons on CTDG next-destination benchmarks with an enumerable fixed catalog should use all-entity ranking as their primary evidence. I Related Work I-A CTDG Link Prediction and Negative Sampling TGN applies binary cross-entropy to observed edges and sampled negative edges [14]. DGB organized random, historical, and inductive negatives within a unified dynamic link-prediction setting and reported that the simple EdgeBank baseline performs strongly against easy random negatives [12]. DyGLib applied these three distributions to multiple CTDG architectures [18], while TGB provided an MRR-based evaluation comparing one positive with multiple negatives [6]. TGB 2.0 compared random 1-vs-1000, destination-aware 1-vs-1000, and 1-vs-all evaluation, demonstrating the effect of the candidate space on reported results [3]. The Stanford JODIE datasets Wikipedia, Reddit, MOOC, and LastFM remain widely used in CTDG research. The original JODIE evaluation compared the next item with every item in the dataset and reported MRR and Recall@10 for Reddit, Wikipedia, and LastFM [8]. These interactions were subsequently reformulated widely as sampled-negative binary link prediction. RepeatMixer and TPNet evaluated recurrence information and pairwise temporal features on the four datasets using random, historical, and inductive AP/AUC [21, 10]. Yang et al. compared samplers, aggregators, and memory modules for temporal graph neural networks at scale using 1:11:1 training, 1:91:9 validation, and 1:491:49 test candidates [16]. BandRank optimized listwise MRR and Hits@5 over query-specific candidate sets [9]. These studies refine the units used to compare CTDG architectures, and each conclusion is conditional on its stated candidate construction. I-B Sampled and Full-Catalog Metrics Recommendation research has analyzed the discrepancy between sampled and full-catalog metrics directly. Krichene and Rendle showed that a sampled metric may fail to preserve the relative ordering of models under the exact metric even in expectation [7]. Dallmann et al. reported that model orders obtained for sequential recommenders under uniform and popularity sampling do not consistently agree with full ranking [2]. We connect this general result to time-dependent pair membership, pair-history operations, and sampler-specific optimal reward directions in CTDGs. All-entity evaluation also has a clear lineage in CTDGs. DyRep computed a rank by replacing the test destination with every other entity [15], and JODIE compared the ground-truth item with every item [8]. Following this lineage, we apply contemporary CTDG scorers to every destination in a fixed candidate space. The original scorer is retained, while the freedom to choose q and K is removed from architecture and module comparisons. I-C CTDG Models Using Pair History TGN updates node-specific memory and combines the representations of two nodes in the final link predictor [14]. GraphMixer similarly encodes each node’s temporal neighbors independently before predicting a link from the two representations [1]. DyGFormer computes neighbor co-occurrence between the historical sequences of the source and destination [18]. TNCN and NAT use temporal common neighbors or joint neighborhood relations in a pair representation [20, 11]. CRAFT uses the candidate destination as a query to compress the source’s recent history through cross-attention. CRAFT-R augments this representation with the number of previous occurrences of the current source–destination pair [17]. Link History Aggregation (LHA) in TAMI is an adapter that separately retains recent interactions of the current pair; in our experiments, it is attached to DyGFormer [19]. These models provide expressive representations of pair compatibility and recurrence. They also create direct paths through which sampler-induced differences in seen/unseen composition can reach the score. I-D Mechanistic Attribution of the Membership-Aligned Component CRAFT presents learnable node embeddings and target-aware matching as two key requirements for future-link prediction. The former provides persistent node identities, while the latter directly computes compatibility between a destination and the source behavior history through candidate-to-history cross-attention. CRAFT-R additionally incorporates pair recurrence counts in settings with many seen edges. CRAFT supports this design through sampled-negative performance and comparisons with existing backbones augmented by learnable embeddings [17]. When candidate identity forms the query and past destination identities form the keys, cross-attention can encode both semantic compatibility and an identity match with a historical pair. If a random candidate set frequently contains seen positive pairs and unseen negative pairs, the two signals induce the same ranking direction. Sampled MRR then observes their combined effect. Our 2×22× 2 membership control and representation intervention trace the membership-aligned component of CRAFT’s predictive performance. Residual ranking performance after removing membership-aligned separation captures the remaining predictive signals jointly. I The Ranking Target of Sampled-Negative Evaluation I-A Problem Setup Let x=(u,t,ℋt)x=(u,t,H_t) denote the context at time t, p+(v∣x)p_+(v x) the conditional distribution of the observed destination, and q(v∣x)q(v x) the negative distribution. Construct C=v0,…,vKC=\v_0,…,v_K\ from one positive and K independent negatives, with an equal prior probability that each position contains the positive. The posterior probability that candidate j is positive is Pr(J=j∣C,x) (J=j C,x) =p+(vj∣x)∏i≠jq(vi∣x)∑ℓ=0Kp+(vℓ∣x)∏i≠ℓq(vi∣x) = p_+(v_j x) _i≠ jq(v_i x) _ =0^Kp_+(v_ x) _i≠ q(v_i x) =p+(vj∣x)/q(vj∣x)∑ℓ=0Kp+(vℓ∣x)/q(vℓ∣x). = p_+(v_j x)/q(v_j x) _ =0^Kp_+(v_ x)/q(v_ x). (2) Equation (2) shows that this posterior is the normalized value of p+(v∣x)/q(v∣x)p_+(v x)/q(v x). The Bayes-optimal ordering maximizes expected reciprocal rank when the candidate-generating distribution is known exactly. Placing candidates in descending posterior order achieves this maximum. A standard log-score inducing the same order is ℓq∗(x,v)=logp+(v∣x)−logq(v∣x). _q^*(x,v)= p_+(v x)- q(v x). (3) Thus, the optimal score for sampled ranking is the log-density ratio between the observed-destination and negative distributions. This result agrees with noise-contrastive estimation, in which the optimal classifier separating data from noise learns a density ratio [5]. Proper-scoring rule theory formalizes the property that a log score elicits the stated probability distribution truthfully [4]. For sampling without replacement, let Q be the joint probability of the negative set. The posterior becomes Pr(J=j∣C,x)∝p+(vj∣x)Q(C∖vj∣x,vj). (J=j C,x) p_+(v_j x)Q(C \v_j\ x,v_j). (4) In (4), the product ∏qΠ q for independent sampling is replaced by the actual sampler Q; the candidate-generation rule remains part of the Bayes-optimal ordering. I-B Uniform Finite-K and Non-Uniform Samplers For a fixed catalog ℛR, the uniform sampler is qunif(v∣x)=1/|ℛ|q_unif(v x)=1/|R|. Its optimal score is ℓunif∗(x,v)=logp+(v∣x)+log|ℛ|. _unif^*(x,v)= p_+(v x)+ |R|. (5) The second term in (5) is constant across candidates. The population-optimal ordering under uniform sampling therefore equals the optimal full-catalog ordering. Sampled MRR, however, observes only a random subset of the full order. Averaging a nonlinear rank metric across scorers can make not only its absolute value but also relative model order and module gain depend on K [7, 2]. Under uniform sampling without replacement, the expected metric can be computed exactly without a particular negative draw. Let r be the positive’s full-catalog rank and N the catalog size. The number X of the K negatives scoring above the positive follows X∼Hypergeom(N−1,r−1,K),RRK=11+X.X (N-1,r-1,K), _K= 11+X. (6) Its expected reciprocal rank is [RRK∣r,N]=(NK+1)−(N−rK+1)r(N−1K).E[R_K r,N]= NK+1- N-rK+1r N-1K. (7) Equation (7) determines expected finite-K MRR solely from each event’s full rank. It removes evaluation-seed variance and separates changes caused by a particular candidate draw from those caused by the finite-K metric transformation. At K=N−1K=N-1, the expectation becomes 1/r1/r, equal to the all-entity reciprocal rank. For historical, source-history, or popularity samplers, q(v∣x)q(v x) varies by candidate, so the term −logq(v∣x)- q(v x) in (3) directly affects the order. The support of a source-history sampler is suppqhist(⋅∣x)=ℋu(t)∖v+.suppq_hist(· x)=H_u(t) \v^+\. (8) For a new positive in (8), the positive lies outside the support while every negative lies inside it. Pair-history membership then separates the labels perfectly. A non-uniform sampler consequently determines not only candidate difficulty but also the optimal ordering rewarded by evaluation. I-C Optimal Coefficient of Pair Membership Define pair membership as M(u,v,t)=[(u,v)∈ℋt].M(u,v,t)=1[(u,v) _t]. (9) Equation (9) indicates whether source u and destination v have interacted at least once before time t. Consider the minimal scorer sβ(M)=b+βMs_β(M)=b+β M, which uses only this indicator. Let p1p_1 and q1q_1 be the probabilities that M=1M=1 among positives and negatives, respectively. The optimal seen/unseen score difference is βq∗=logit(p1)−logit(q1). _q^*=logit(p_1)-logit(q_1). (10) Equation (10) sets the optimal reward to the difference in log odds between the seen rate of true positives and that induced among negatives by the sampler. Under uniform random sampling, q1=|ℛseen(u,t)|/|ℛ|q_1=|R_seen(u,t)|/|R|; if p1>q1p_1>q_1, then βq∗>0 _q^*>0. This reflects useful recurrence in the data: positive events are more concentrated on historical pairs than their catalog prevalence would suggest. Under source-history sampling, q1=1q_1=1, and in the presence of new positives, βq∗→−∞ _q^*→-∞. This limit expresses the opposite perfect separation created by the sampler support. The concern is therefore not recurrence itself. Rather, coarse seen/unseen separation can account for a large share of MRR in a small candidate set and obscure the ability to select a relation within the seen or unseen subgroup. IV Controlled Evaluation and the All-Entity Protocol IV-A 2×22× 2 Pair-Membership Evaluation Fully crossing the historical membership of positive and negative pairs produces the four conditions in Table I. Repeated–Seen is recurrent-link ranking: selecting the relation that recurs now among relations observed previously. New–Unseen is novel-link ranking: selecting the actual positive among relations being formed with the source for the first time. In both conditions, positive and negative membership agree, so membership alone cannot separate the labels. TABLE I: Evaluation design crossing positive and negative pair membership. Seen negative Unseen negative Repeated positive Recurrent-link ranking Seen-subgroup separation New positive Unseen-subgroup separation Novel-link ranking In Repeated–Unseen, the positive pair has appeared before; in New–Seen, the positive pair appears for the first time. These two conditions diagnose whether a model assigns higher scores to the seen or unseen subgroup. An unseen negative forms a new pair with the current source, but its destination node must already have appeared before t. We apply the same checkpoint to the same positive events for which all four conditions can be constructed, changing only the negative composition. The two off-diagonal cells use 20 negatives to diagnose membership direction. For the primary diagonal evaluation, a repeated positive is compared with every seen destination of its source, while a new positive is compared with every unseen destination in the fixed catalog. This all-seen/all-unseen decomposition removes finite-K sampling from the diagonal cells as well. IV-B Pair-Membership Representation Intervention The pre-scorer representation of CRAFT-R, which uses pair history explicitly, is zuv(t)=[hu∣v(t);ev(t);ruv(t)],z_uv(t)=[h_u v(t);e_v(t);r_uv(t)], (11) combining a candidate-conditioned source representation, a candidate embedding, and a pair-recurrence representation. Define the difference between the mean representations of seen and unseen training pairs as a=μseen−μunseen.a= _seen- _unseen. (12) We remove this direction using the orthogonal projection P⟂=I−aa⊤a⊤a,zuv⟂(t)=P⟂zuv(t).P_ =I- a a a, z _uv(t)=P_ z_uv(t). (13) Equation (13) removes only the component parallel to a and preserves its orthogonal complement. This is a controlled intervention for tracing the representation path behind an observed score change, rather than a new prediction method. As controls, we also remove a random direction of equal dimensionality and a direction computed after shuffling the membership labels, thereby measuring the effect of removing an arbitrary single dimension. IV-C Fixed-Catalog All-Entity Ranking The primary risk set ℛR is fixed to all destination identities that occur at least once in the chronological training prefix. We exclude the source itself and, for test events whose positive belongs to ℛR, evaluate the model’s existing score for every destination. Full-catalog MRR averages the reciprocal rank of the observed destination within the entire catalog across test events. Hits@10 is computed from the same complete order. We assign average rank to score ties. All-entity ranking requires neither a separate intensity head nor probability calibration. Any CTDG model returning a source–destination score can be evaluated using the same checkpoint and scorer. One complete score vector per event yields both the full-catalog and sampled results. Sampled MRR uses the first K∈1,5,20,50,100K∈\1,5,20,50,100\ elements of a uniform random permutation excluding the positive. This nested construction shares candidates across different values of K. The primary K-sweep applies Eq. (7) to each event’s full rank to compute the expected MRR under uniform sampling. Actual nested draws are retained as a secondary check against the expectation curve. For a stateful model, we score every candidate from the same current memory and update memory exactly once with the observed positive interaction. This procedure prevents leakage caused by candidate-specific state changes. A secondary sensitivity analysis forms a time-available catalog from destinations that have appeared at least once before each event. IV-D Comparison Quantities We measure agreement between the sampled and full-catalog model orders using Kendall’s τ. A value of τ=1τ=1 means that every model pair has the same relative order, while τ=−1τ=-1 means that every order is reversed. We measure the effect of the pair-recurrence module in CRAFT-R as GR(K)=MRRK(CRAFT-R)−MRRK(CRAFT).G_R(K)=MRR_K(CRAFT-R)-MRR_K(CRAFT). (14) The magnitude and sign of GR(K)G_R(K) report the change recorded by evaluation when the module is added to the same backbone. V Experiments V-A Setup Table I summarizes the common training and evaluation settings. TABLE I: Training and evaluation settings. Item Setting Datasets LastFM, MOOC, Reddit, Wikipedia Event range Most recent 32,768 events of each dataset Temporal split Training prefix 85%, test suffix 15% History length 320 Training epochs 3 Seeds 7, 17, 27 Primary training objective BCE, one uniform random negative per positive Objective sensitivity CRAFT/CRAFT-R retrained with BPR Models CRAFT, CRAFT-R, DyGFormer, DyGFormer+LHA, TGN, GraphMixer Primary catalog All destinations appearing in the training prefix Sampled evaluation K∈1,5,20,50,100K∈\1,5,20,50,100\, nested uniform candidates Catalog coverage LastFM 1.000, MOOC 1.000, Reddit 0.996, Wikipedia 0.987 Reported values Mean and standard deviation over three seeds We compare CRAFT, CRAFT-R, DyGFormer, DyGFormer+LHA, TGN, and GraphMixer. TGN and GraphMixer encode each node history separately and combine the two nodes only in the final scorer. DyGFormer uses co-occurrence between the two histories. In CRAFT, the candidate enters as the cross-attention query over source history, and CRAFT-R adds pair recurrence. The LHA condition attaches TAMI’s pair-history adapter to DyGFormer. All models are trained with BCE using one random negative; parameters are frozen during evaluation. We report the mean and standard deviation over three seeds. This comparison is a unified-objective architecture study that matches training conditions across architectures. We additionally train CRAFT and CRAFT-R with the BPR objective used in the original CRAFT study, separating training-objective sensitivity of module gain from stability of the membership mechanism. V-B Reward Direction Predicted by the Minimal Scorer Table I tests Eq. (10) under paired-random and source-history conditions on the same events. The analytic β∗β^* and directly learned β agree on every dataset under random sampling. Under source-history sampling, the coefficient moves to a large negative value. The positive random coefficient reflects recurrence in the data, where positive events concentrate on seen pairs. The negative source-history coefficient reflects opposite perfect separation created by the sampler support. TABLE I: Analytic and learned values for the minimal pair-membership scorer. Dataset Events p1p_1 Random q1q_1 β∗β^* Random β History β LastFM 18,366 .816 .122 3.464 3.464 -8.606 Wikipedia 971 .719 .093 3.211 3.211 -8.732 Reddit 880 .828 .062 4.290 4.290 -8.590 MOOC 10,195 .607 .344 1.078 1.078 -8.902 V-C 2×22× 2 Results Models that directly transmit pair history show large amplitudes in the two subgroup-separation conditions. CRAFT-R obtains Repeated–Unseen MRR of 0.957, 0.979, 0.987, and 0.888 on LastFM, Wikipedia, Reddit, and MOOC, respectively. When only the positive is unseen (New–Seen), the corresponding values are 0.059, 0.050, 0.048, and 0.508. DyGFormer+LHA likewise records Repeated–Unseen MRR of 0.974–0.996 and New–Seen MRR of 0.048–0.050 on the first three datasets. TGN and GraphMixer respond in the same direction, but with smaller amplitudes. The membership-matched diagonal provides separate relation-selection results. CRAFT-R’s recurrent-link MRR is 0.404, 0.431, 0.303, and 0.667 on LastFM, Wikipedia, Reddit, and MOOC, while its novel-link MRR is 0.345, 0.303, 0.493, and 0.782. Subgroup separation and within-subgroup ranking, which a single sampled MRR conflates, can therefore be read separately. Removing the membership direction in Eq. (13) reduces Repeated–Unseen from 0.957 to 0.246 on LastFM, 0.979 to 0.201 on Wikipedia, 0.987 to 0.182 on Reddit, and 0.888 to 0.489 on MOOC. New–Seen moves from 0.059 to 0.657, 0.050 to 0.539, 0.048 to 0.772, and 0.508 to 0.940, respectively. The changes in novel-link MRR are −0.011,−0.070,−0.016,+0.004-0.011,-0.070,-0.016,+0.004. The intervention primarily changes the preference between seen and unseen subgroups rather than selection among new relations. Figure 1 displays the four CRAFT-R conditions as a matrix. Rows distinguish repeated from new positive pairs, columns distinguish seen from unseen negative pairs, and each cell reports MRR. Every cell uses 20 negatives. The off-diagonal cells reverse the direction in which pair-history membership identifies the answer and serve as diagnostics. Figure 1: Pair-membership 2×22× 2 MRR for CRAFT-R. Rows identify the positive pair and columns identify the negative pair by historical membership. All four cells are directional diagnostics using 20 negatives. The primary diagonal evaluation uses the exhaustive rankings in Table IV. All-seen compares a repeated positive with every destination previously encountered by its source. All-unseen compares a new positive with every unseen destination in the fixed catalog. CRAFT-R’s all-unseen MRR is .0145 on LastFM, .5648 on MOOC, .1063 on Reddit, and .0453 on Wikipedia. TABLE IV: Membership-controlled all-seen and all-unseen MRR. CRAFT CRAFT-R Dataset All-seen All-unseen All-seen All-unseen LastFM .1939 .0193 .2041 .0145 MOOC .6253 .5780 .6247 .5648 Reddit .8715 .0967 .8549 .1063 Wikipedia .8923 .0534 .8202 .0453 V-D Mechanistic Attribution of the Membership-Aligned Component CRAFT explains candidate-to-history cross-attention through fine-grained compatibility between a destination and source behavior history [17]. That explanation and pair-membership lookup predict the same sampled-negative outcome: learnable candidate identity can match keys in source history, and CRAFT-R passes the pair recurrence count directly to the scorer. The 2×22× 2 results distinguish these explanations. Holding CRAFT-R parameters and event history fixed while changing only membership composition yields Repeated–Unseen MRR of 0.888–0.987 and New–Seen MRR of 0.048–0.508. Orthogonally projecting out the membership direction strongly reverses both diagnostics, whereas New–Unseen novel-link MRR changes by only −0.070-0.070 to +0.004+0.004, depending on the dataset. The component localized by the intervention therefore primarily separates the seen and unseen subgroups rather than expressing compatibility among unseen candidates. This result reproduces CRAFT’s strong sampled MRR while giving its origin a narrower interpretation. Cross-attention is a powerful candidate-conditioned scorer, and sampled MRR jointly rewards semantic matching and identity-based pair membership. Membership-matched Repeated–Seen and New–Unseen rankings, together with all-entity ranking, measure residual ranking performance after membership-aligned separation is removed. V-E Expected Uniform-20 and Full-Catalog Model Comparison Table V reports MRR under expected Uniform-20 and the full catalog using the same checkpoint, scorer, and test event. Reciprocal rank declining as the candidate set grows is arithmetically expected; we therefore distinguish this absolute decline from changes in relative model order and module gain. TABLE V: Expected Uniform-20 MRR and MRR over the complete fixed training catalog (mean ± standard deviation). Dataset Model Expected-20 Full Dataset Model Expected-20 Full LastFM CRAFT .4283±.0112 .0636±.0065 MOOC CRAFT .7627±.0046 .4625±.0066 CRAFT-R .6892±.0002 .1590±.0029 CRAFT-R .7593±.0053 .4568±.0079 DyGFormer .4161±.0015 .0811±.0100 DyGFormer .7565±.0079 .4686±.0073 DyGFormer+LHA .6380±.0071 .1139±.0064 DyGFormer+LHA .6804±.0085 .3835±.0103 TGN .3969±.0165 .0557±.0106 TGN .2831±.0241 .0933±.0105 GraphMixer .4488±.0042 .0922±.0027 GraphMixer .5128±.0060 .2681±.0059 Reddit CRAFT .8308±.0117 .4919±.0612 Wikipedia CRAFT .8948±.0028 .7366±.0100 CRAFT-R .8744±.0011 .6637±.0006 CRAFT-R .8985±.0007 .7101±.0058 DyGFormer .8022±.0029 .4771±.0205 DyGFormer .8283±.0085 .5784±.0094 DyGFormer+LHA .8647±.0010 .6357±.0070 DyGFormer+LHA .8522±.0028 .6510±.0102 TGN .6892±.0024 .2204±.0094 TGN .6668±.0248 .2160±.0521 GraphMixer .7088±.0008 .2561±.0009 GraphMixer .7455±.0048 .3160±.0101 Expected Uniform-20 places CRAFT above DyGFormer on LastFM, whereas the full catalog places DyGFormer and GraphMixer above CRAFT. On MOOC, the leading three models are CRAFT, CRAFT-R, and DyGFormer under expected Uniform-20, but DyGFormer, CRAFT, and CRAFT-R under the full catalog. The Reddit order is preserved. On Wikipedia, CRAFT and CRAFT-R exchange positions. Most of the six-model ordering remains stable; the changes concentrate in particular model pairs. V-F Rank and Module Effect Across K TABLE VI: Kendall’s τ between the expected uniform finite-K model order and the all-entity order. Dataset K=1K=1 K=5K=5 K=20K=20 K=50K=50 K=100K=100 LastFM .733 .867 .867 .867 1.000 MOOC .733 .733 .733 1.000 1.000 Reddit 1.000 1.000 1.000 1.000 1.000 Wikipedia 1.000 1.000 .867 .867 .867 TABLE VII: Effect of CRAFT-R relative to CRAFT, GR(K)G_R(K). Dataset K=1K=1 K=5K=5 K=20K=20 K=50K=50 K=100K=100 Full LastFM +.0591 +.1696 +.2608 +.2693 +.2442 +.0955 MOOC -.0001 -.0009 -.0034 -.0052 -.0057 -.0057 Reddit +.0036 +.0160 +.0436 +.0724 +.0982 +.1718 Wikipedia -.0019 -.0014 +.0037 +.0084 +.0106 -.0266 In Table VI, LastFM and MOOC fail to preserve some model-pair orders at small K. Wikipedia remains at τ=0.867τ=0.867 for K=20,50,100K=20,50,100. In Table VII, Wikipedia has small positive point estimates at K=20,50,100K=20,50,100, followed by a decrease under the full catalog. The sign remains positive on Reddit, but the effect size ranges from +0.0034+0.0034 at K=1K=1 to +0.1718+0.1718 over the full catalog. A module conclusion from sampled ablation is thus conditional on q and K. In particular, the Wikipedia result prevents attribution of the sampled point estimate to a universal effect of repeat-time encoding. Because the module is fixed and only the candidate protocol changes, the measured gain reflects an interaction between recurrence encoding and candidate composition. Table VIII reports uncertainty for the CRAFT-R minus CRAFT MRR difference computed on matched seeds and test events. The seed-paired t interval represents variation across three training seeds. The time-block paired bootstrap groups chronologically ordered test events into blocks of 128 and resamples the same blocks jointly for both models and all seeds. This interval represents test-sequence uncertainty conditional on trained checkpoints; the seed-paired result is the primary generalization interval. The Wikipedia Expected-20 gain is +0.0038+0.0038, with a seed interval containing zero. The all-entity gain is −0.0266-0.0266, and both intervals are below zero. The sampled condition provides a small positive point estimate with training-seed uncertainty, whereas the all-entity condition shows a consistent decrease. TABLE VIII: CRAFT-R minus CRAFT MRR differences and 95% confidence intervals computed on matched seeds and test events. Dataset–Protocol MRR difference Seed-paired t CI Time-block bootstrap CI LastFM–Expected-20 +.2608 [.2267, .2950] [.2460, .2756] LastFM–All-entity +.0955 [.0845, .1065] [.0862, .1056] MOOC–Expected-20 -.0034 [-.0269, .0200] [-.0061, -.0010] MOOC–All-entity -.0057 [-.0269, .0155] [-.0107, -.0008] Reddit–Expected-20 +.0436 [.0093, .0779] [.0400, .0471] Reddit–All-entity +.1718 [-.0135, .3571] [.1616, .1813] Wikipedia–Expected-20 +.0038 [-.0036, .0112] [-.0004, .0080] Wikipedia–All-entity -.0266 [-.0526, -.0005] [-.0367, -.0170] V-G Training-Objective Sensitivity Table IX reports CRAFT and CRAFT-R retrained with the BPR objective used by CRAFT. On LastFM and Reddit, the magnitude of the module effect varies substantially with K under both BCE and BPR. On MOOC and Wikipedia, the objective also changes its sign. In particular, the Expected-20 +0.0037+0.0037 and all-entity −0.0266-0.0266 observed for the Wikipedia BCE checkpoints become +0.0345+0.0345 and +0.0629+0.0629, respectively, for the BPR checkpoints. The sign change under BCE therefore does not generalize as an objective-independent property of the CRAFT architecture. Within the same objective and checkpoint, however, both objectives preserve the finding that K and the risk set alter the measured module effect. TABLE IX: CRAFT-R minus CRAFT MRR by training objective. Expected-20 is the exact expectation of uniform 20-negative evaluation. BCE BPR Dataset Expected-20 All-entity Expected-20 All-entity LastFM +.2608 +.0955 +.2696 +.0983 MOOC -.0034 -.0057 +.0147 +.0212 Reddit +.0436 +.1718 +.1035 +.3431 Wikipedia +.0037 -.0266 +.0345 +.0629 Table X applies the same 2×22× 2 diagnostic to the BPR checkpoints. CRAFT-R obtains .932–1.000 for Repeated–Unseen and .048–.338 for New–Seen across the four datasets. The large contrast has the same direction under BCE and BPR, so the mechanistic observation that the pair-history path conveys membership separation is stable across training objectives. TABLE X: CRAFT-R membership-diagnostic MRR by training objective. R–U denotes Repeated–Unseen and N–S denotes New–Seen. BCE BPR Dataset R–U N–S R–U N–S LastFM .957 .059 .978 .049 MOOC .888 .508 .932 .338 Reddit .987 .048 1.000 .048 Wikipedia .979 .050 1.000 .048 Figure 2 connects the module effects from Table VII across candidate-set sizes. The horizontal axis gives the number of uniform random negatives and the full catalog; the vertical axis gives the MRR difference between CRAFT-R and CRAFT, with values above the dashed line favoring CRAFT-R. Effect size changes substantially with K on LastFM and Reddit. On Wikipedia, small positive point estimates at sampled K=20,50,100K=20,50,100 move to a negative full-catalog effect. The curve visualizes how the measured effect of an unchanged module varies continuously with candidate count and why a sampled ablation conclusion need not represent the full-catalog conclusion. Figure 2: CRAFT-R minus CRAFT MRR as a function of the expected number of negative candidates under uniform sampling. The dashed line marks zero module effect. V-H Representation Controls and Risk-Set Sensitivity TABLE XI: Full-catalog MRR after removing representation directions from CRAFT-R. Dataset Base Membership Random Shuffled LastFM .1590 .0738 .1588 .1570 MOOC .4568 .3278 .4600 .4605 Reddit .6637 .0416 .6640 .6537 Wikipedia .7101 .0405 .7206 .7121 In Table XI, removing a random direction or a shuffled-label direction preserves the baseline result. Removing the membership direction produces consistently different outcomes on all four datasets. The controls distinguish removal of an arbitrary representation dimension from intervention on a representation axis carrying pair membership. Model order is identical under the fixed training catalog and the time-available catalog. Their MRR differences are zero on LastFM and MOOC, 0.0002–0.0005 on Reddit, and 0.0020–0.0054 on Wikipedia. V-I Evaluation Cost Figure 3 reports all-entity scoring cost as catalog size grows for the same CRAFT checkpoint. With 128 test events and candidate batches of 4096, time per event increases nearly linearly from 0.13–0.14 ms for approximately 20 candidates to 4.94–5.69 ms for 822–991 candidates. Candidate batching makes peak GPU memory saturate near 200 MB. Exact ranking is therefore directly computable at the catalog scale studied here; this result does not represent the cost of direct evaluation for million-entity catalogs. Figure 3: All-entity scoring time and peak GPU memory versus fixed-catalog size. We evaluate 128 test events from each CRAFT checkpoint in candidate batches of 4096; points show the mean over three seeds. VI Discussion and Conclusion A sampled-negative CTDG score is jointly produced by the model scorer and the candidate protocol. A non-uniform q changes the Bayes-optimal ordering through a density ratio. Uniform q over a fixed catalog preserves the population-optimal order, but finite K observes only part of the complete ranking and does not reliably preserve model order or module gain. This distinction decomposes the familiar observation that “random negatives are easy” into two precise evaluation effects. Pair recurrence is useful information in the actual data-generating process. When positive events concentrate on seen pairs relative to their prevalence in a uniform catalog, the optimal membership coefficient is positive. The evaluation issue arises when this coarse subgroup separation accounts for a large share of MRR in a small candidate set and obscures destination selection within the seen and unseen groups. A support-restricted sampler such as source-history can create perfect separation in the opposite direction. The 2×22× 2 evaluation and representation intervention separate subgroup separation, recurrent-link ranking, and novel-link ranking, and show how pair-history operations transmit this difference to the score. This distinction supports a mechanistic attribution of CRAFT’s performance. CRAFT’s candidate-to-history cross-attention computes semantic compatibility and a match with historical pair identity along the same path. Under random candidates, both signals frequently identify the answer in the same direction, so high sampled MRR measures their combined effect. Crossing membership composition and removing the corresponding representation axis show that pair membership accounts for the large amplitude of the subgroup diagnostics. Membership-matched and all-entity rankings provide controlled conditions measuring residual ranking performance after membership-aligned separation. The lower absolute MRR of the full catalog relative to expected Uniform-20 is an arithmetic consequence of having more candidates. Changes in comparative conclusions occur for particular model pairs and modules. At least one model pair changes relative order on LastFM, MOOC, and Wikipedia. The CRAFT-R module effect on Wikipedia has a small positive sampled point estimate with seed-level uncertainty, but a consistently negative full-catalog value under the unified BCE objective. With BPR retraining, both sampled and all-entity effects are positive. The training objective changes the scores supplied by an architecture, while the candidate protocol changes the comparison problem used to measure those scores. Architecture superiority and ablation effects supported by sampled-negative results are therefore reproducible conditional evidence when q and K are stated together. The measured effect of repeat-time encoding is accordingly an interaction between the module and candidate protocol rather than a fixed property of the module alone. Fixed-training-catalog all-entity ranking applies directly to any existing CTDG model that provides source–destination scores. It eliminates freedom in selecting negatives and removes sampling variance, measuring one CTDG destination-prediction problem specified by the catalog definition and its coverage. Historical, inductive, and source-history candidates remain useful diagnostics for dissecting model behavior. Placing all-entity results as the primary architecture and module comparison, with sampled protocols reported as secondary analyses that explicitly state q, K, and support, stabilizes interpretation. This recommendation applies to benchmarks with an exactly enumerable fixed catalog. At large scale, reporting retrieval recall and approximate-ranking error against the same fixed all-entity target preserves the evaluation question. All-entity ranking fixes the evaluation target by removing candidate selection at test time. The negative sampler and loss function used during training remain part of the learned model specification. The BPR sensitivity results demonstrate the need to report the training objective together with the evaluation protocol. Acknowledgements The authors have no acknowledgements to declare. Funding This research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Data availability This study uses the public Wikipedia, Reddit, MOOC, and LastFM temporal interaction datasets distributed with JODIE. The raw data are available from the Stanford SNAP JODIE repository. The accompanying materialization pipeline records the source metadata and produces the chronological splits used in this study. Code availability The code for data materialization, training, sampled-candidate diagnostics, 2×22× 2 controls, fixed-catalog all-entity evaluation, and result reproduction is publicly available at https://github.com/SnowyPainter/goodctdg-public. References [1] W. Cong, S. Zhang, J. Kang, B. Yuan, H. Wu, X. Zhou, H. Tong, and M. Mahdavi (2023) Do we really need complicated model architectures for temporal networks?. In International Conference on Learning Representations, Cited by: §I-C. [2] A. Dallmann, D. Zoller, and A. Hotho (2021) A case study on sampling strategies for evaluating neural sequential item recommendation models. In Proceedings of the 15th ACM Conference on Recommender Systems, p. 505–514. External Links: Document Cited by: §I, §I-B, §I-B. [3] J. Gastinger, S. Huang, M. Galkin, E. Loghmani, A. Parviz, F. Poursafaei, J. Danovitch, E. Rossi, I. Koutis, H. Stuckenschmidt, R. Rabbany, and G. Rabusseau (2024) TGB 2.0: a benchmark for learning on temporal knowledge graphs and heterogeneous graphs. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Document Cited by: §I, §I-A. [4] T. Gneiting and A. E. Raftery (2007) Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102 (477), p. 359–378. External Links: Document Cited by: §I-A. [5] M. Gutmann and A. Hyvärinen (2010) Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, p. 297–304. Cited by: §I-A. [6] S. Huang, F. Poursafaei, J. Danovitch, M. Fey, W. Hu, E. Rossi, J. Leskovec, M. Bronstein, G. Rabusseau, and R. Rabbany (2023) Temporal graph benchmark for machine learning on temporal graphs. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Document Cited by: §I, §I, §I-A. [7] W. Krichene and S. Rendle (2020) On sampled metrics for item recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 1748–1757. External Links: Document Cited by: §I, §I-B, §I-B. [8] S. Kumar, X. Zhang, and J. Leskovec (2019) Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 1269–1278. External Links: Document Cited by: §I-A, §I-B. [9] Y. Li, Y. Xu, X. Lin, W. Zhang, and Y. Zhang (2025) Ranking on dynamic graphs: an effective and robust band-pass disentangled approach. In Proceedings of the ACM Web Conference, External Links: Document Cited by: §I-A. [10] X. Lu, L. Sun, T. Zhu, and W. Lv (2024) Improving temporal link prediction via temporal walk matrix projection. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Document Cited by: §I-A. [11] Y. Luo and P. Li (2022) Neighborhood-aware scalable temporal network representation learning. In Learning on Graphs Conference, p. 1–18. Cited by: §I-C. [12] F. Poursafaei, S. Huang, K. Pelrine, and R. Rabbany (2022) Towards better evaluation for dynamic link prediction. In Advances in Neural Information Processing Systems, Vol. 35. External Links: Document Cited by: §I, §I, §I-A. [13] R. Romero, T. De Bie, and J. Lijffijt (2023) New perspectives on the evaluation of link prediction algorithms for dynamic graphs. arXiv preprint arXiv:2311.18486. External Links: Document Cited by: §I. [14] E. Rossi, B. Chamberlain, F. Frasca, D. Eynard, F. Monti, and M. Bronstein (2020) Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637. External Links: Document Cited by: §I, §I-A, §I-C. [15] R. Trivedi, M. Farajtabar, P. Biswal, and H. Zha (2019) DyRep: learning representations over dynamic graphs. In International Conference on Learning Representations, External Links: Link Cited by: §I-B. [16] Y. Yang, H. Zhou, R. Kannan, and V. Prasanna (2024) Towards ideal temporal graph neural networks: evaluations and conclusions after 10,000 gpu hours. Proceedings of the VLDB Endowment 18 (4), p. 956–969. External Links: Document Cited by: §I-A. [17] L. Yi, R. Lei, F. Mo, Y. Zheng, Z. Wei, and Y. Ye (2025) Future link prediction without memory or aggregation. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §I, §I, §I-C, §I-D, §V-D. [18] L. Yu, L. Sun, B. Du, and W. Lv (2023) Towards better dynamic graph learning: new architecture and unified library. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Document Cited by: §I, §I-A, §I-C. [19] Z. Yu, J. Wu, Z. Wu, S. Zhong, W. Su, C. Lee, and W. Zhuo (2025) TAMI: taming heterogeneity in temporal interactions for temporal graph link prediction. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §I, §I-C. [20] X. Zhang, Y. Wang, X. Wang, and M. Zhang (2024) Efficient neural common neighbor for temporal graph link prediction. In arXiv preprint arXiv:2406.07926, External Links: Document Cited by: §I, §I-C. [21] T. Zou, Y. Mao, J. Ye, and B. Du (2024) Repeat-aware neighbor sampling for dynamic graph learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, External Links: 2405.17473 Cited by: §I-A. Additional Reproducibility Information. Every sampled-K and full-catalog result is derived from one score vector computed per event. Training stochasticity is independent across seeds. Within a seed, the candidates for different K form nested prefixes of one fixed random permutation. The complete experiment comprises 72 training and evaluation runs: four datasets, six models, and three seeds. For paired-random and source-history comparisons, both conditions use only events for which 20 source-history negatives can be constructed. The 2×22× 2 evaluation uses the same positive events for which all four conditions are available. The numbers of repeated and new events are 14,979 and 3,387 on LastFM; 698 and 273 on Wikipedia; 729 and 151 on Reddit; and 6,106 and 3,981 on MOOC. -A Derivation of the Uniform Finite-K Expectation If r−1r-1 catalog candidates score above the positive, the probability mass function in Eq. (6) gives [RRK∣r,N]=1(N−1K)∑x(r−1x)(N−rK−x)x+1.E[R_K r,N]= 1 N-1K _x r-1x N-rK-xx+1. (15) Applying (r−1x)/(x+1)=(rx+1)/r r-1x/(x+1)= rx+1/r and simplifying the sum with Vandermonde’s identity yields (NK+1)−(N−rK+1)/r(N−1K)\ NK+1- N-rK+1\/\r N-1K\, giving Eq. (7). We use average rank for ties. Let a and b be the numbers of candidates scoring strictly above and equal to the positive, and let H and T denote the corresponding numbers included in a sample. The tie-aware reciprocal rank is 1/(1+H+T/2)1/(1+H+T/2), implemented as ∑h,t(ah)(bt)(N−1−a−bK−h−t)(N−1K)11+h+t/2. _h,t ah bt N-1-a-bK-h-t N-1K\, 11+h+t/2. (16) Only three of 352,458 model–event instances across the 72 BCE runs contain ties (0.00085%).