Paper deep dive

Entity Resolution in Practice: Lessons from a Self-Serve Pipeline

Kaushik Pavani, Ganga Aluri, Pravin Jadhav, Neeraj Prasad, Kiran Sanka

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 35

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 8/4/2026, 10:24:38 AM

Summary

This paper presents lessons from building a self-serve entity resolution (ER) pipeline, highlighting three key findings: (1) no single matching algorithm dominates across all datasets, necessitating an automatic tournament to select the best matcher (DeepMatcher, LightGBM, or GAT) per dataset; (2) precision and recall require separate optimization strategies, with hard rule-based vetoes for precision and diverse candidate retrieval (HNSW ensembles and identifier blocking) for recall; and (3) transitive closure clustering can cause silent merge errors from false positives, requiring a 'verified merge' approach that actively re-verifies cross-group merges to prevent mega-cluster formation.

Entities (10)

Entity Resolution · concept · 98%Cora · dataset · 95%MusicBrainz 200K · dataset · 95%NCV · dataset · 95%DeepMatcher · model · 92%LightGBM · model · 92%GAT · model · 90%Verified Merge · algorithm · 90%Standard Operating Procedure · methodology · 88%HNSW · algorithm · 85%

Relation Signals (10)

Entity Resolution → evaluatedon → MusicBrainz 200K

confidence 95% · We evaluate on six deduplication benchmarks... MB 200K

Entity Resolution → evaluatedon → NCV

confidence 95% · NCV denotes the 5M-record benchmark

DeepMatcher → performsbeston → Cora

confidence 90% · DeepMatcher wins on Cora

LightGBM → performsbeston → NCV

confidence 90% · LightGBM wins on ... NCV

Entity Resolution → usesmethodology → Standard Operating Procedure

confidence 90% · a structured YAML specification—a Standard Operating Procedure (SOP)—encodes matching logic

Entity Resolution → hascomponent → DeepMatcher

confidence 85% · DeepMatcher wins on Cora, Geo Settlements, and MB 200K

Entity Resolution → hascomponent → LightGBM

confidence 85% · LightGBM wins on Restaurants, DBLP-Scholar, and NCV

Entity Resolution → hascomponent → GAT

confidence 85% · GAT wins none

Entity Resolution → usesalgorithm → HNSW

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We built and evaluated a self-serve entity resolution (ER) system on six benchmarks spanning 864 to 5M records, and three lessons emerged that are absent from existing ER literature. (1) No single matching algorithm wins everywhere - a self-serve pipeline cannot predict its next dataset, so we recommend training several algorithm families per dataset and letting an automatic bake-off pick the winner. (2) Precision and recall need separate fixes, not a shared threshold - precision needs hard rule-based vetoes, recall needs more diverse candidate retrieval. (3) One false-positive link can silently merge unrelated entities - assuming "A matches B" and "B matches C" implies "A matches C" lets a single bad link chain hundreds of records together, so every cross-group merge must be actively re-verified. We hope these lessons save practitioners the months of dead-end experiments that led us to them.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

34,061 characters extracted from source content.

Expand or collapse full text

Entity Resolution in Practice: Lessons from a Self-Serve Pipeline Kaushik Pavani Ganga Aluri Pravin Jadhav Neeraj Prasad Kiran Sanka Abstract We built and evaluated a self-serve entity resolution (ER) system on six benchmarks spanning 864 to 5 M records, and three lessons emerged that are absent from existing ER literature. (1) No single matching algorithm wins everywhere—a self-serve pipeline cannot predict its next dataset, so we recommend training several algorithm families per dataset and letting an automatic bake-off pick the winner. (2) Precision and recall need separate fixes, not a shared threshold—precision needs hard rule-based vetoes, recall needs more diverse candidate retrieval. (3) One false-positive link can silently merge unrelated entities—assuming “A matches B” and “B matches C” implies “A matches C” lets a single bad link chain hundreds of records together, so every cross-group merge must be actively re-verified. We hope these lessons save practitioners the months of dead-end experiments that led us to them. I Introduction ID Name Phone Address City True R1 Sakura Sushi 503-0147 42 Oak St Portland A R2 Sakura Sushi Bar 503-0147 42 Oak St Portland A R3 Sakura Sushi — — — ? R4 Sakura Sushi 206-9283 8 Elm Ave Seattle B R5 Sakura 206-9283 8 Elm Ave Seattle B x Entity A (Portland) x Bridge record (sparse) x Entity B (Seattle) Figure 1: A preview of a typical failure mode in ER systems, and one of the lessons we discuss in this paper. Consider the task of deduplicating restaurant records so that one cluster represents one physical location (a single Sakura Sushi at 42 Oak St in Portland) rather than a brand across cities. Five similarly named records arrive (R1–R5). Most pairwise matchers will accept both R1↔ 3 and R3↔ 4: the sparse bridge R3 has nothing in any populated field that disagrees with either group. The pipeline then chains the two correct-looking links together, silently merging the Portland and Seattle locations into one cluster—an incorrect answer under the per-location definition above. We study this in §VI and propose two fixes: (a) a verified-merge clustering step, and (b) a sparsity-aware confidence threshold for records with few populated fields. Entity resolution (ER)—identifying records that refer to the same real-world entity—is foundational to data integration [6, 8]. Recent LLM-based approaches [20, 7] can match records zero-shot but cost hundreds of dollars per million pairs and embed matching logic in opaque weights. PLM-based systems [15, 16] are cheap at inference but require thousands of labeled pairs per domain and offer no audit trail. Neither approach works for organizations running dozens of ER tasks under changing requirements, strict auditability, and tight cost budgets. We built a system to bridge this gap. The key idea is simple: a structured YAML specification—a Standard Operating Procedure (SOP)—encodes matching logic as inspectable, versionable configuration. An LLM teacher conditioned on the SOP labels candidate pairs; those labels train a lightweight matcher via distillation at orders-of-magnitude lower cost. The SOP simultaneously prompts the teacher, structures its output, and serves as the audit trail. This paper is not a systems paper. It is a practitioner’s guide organized around three failure modes we encountered while evaluating this pipeline on six benchmarks spanning four orders of magnitude (864 to 5 M records)—failure modes absent from existing ER literature: L1. No single matcher wins; let them compete (§IV). A tournament over three canonical architectures (DeepMatcher, LightGBM, GAT) auto-selects the best per dataset. Under a data-scarce regime (≤ 10K training records), DeepMatcher and LightGBM each win 3/6 benchmarks on Pair-F1; GAT wins none. L2. Precision and recall need separate toolkits (§V). Hard vetoes for precision, blocking ensemble diversity for recall. No single threshold can optimize both. L3. One false positive can collapse your clusters (§VI). Transitive closure creates silent mega-clusters; verified merge clustering with active cross-cluster verification recovers recall safely. I The Framework Our system follows the standard block → match → cluster ER architecture [6, 19]. Three design choices motivate the lessons below. (1) Domain knowledge lives in an SOP, not weights. What counts as a match is a business decision—e.g., two food-court tenants share one phone number but are different entities—and cannot be inferred from data without examples no organization possesses. We encode such rules in a versioned YAML SOP that serves three roles: LLM teacher prompt, distillation signal (per-field similarity assessments richer than a binary label), and audit trail. A full SOP example is in Appendix B. (2) Retrieval and matching are separate components. A blocker encoder (Siamese fine-tuning, contrastive loss) optimizes for recall; a matcher optimizes for precision. Training them separately avoids the tension inherent in a single end-to-end model. The matcher consumes blocker embeddings plus schema-driven features (Jaro-Winkler, exact match, transposition detection) and is selected via a tournament over three canonical families (Table I). The full pipeline—onboarding, training, and inference—is detailed in Appendix A with the architecture diagram (Figure 3) and model equations. (3) Per-dataset tuning is mandatory. The pipeline exposes ∼ 40 hyperparameters whose optimal values depend on duplicate density, field sparsity, schema width, and scale, so we tune per dataset. For each experiment we obtain a strong baseline configuration using both Optuna TPE Bayesian search [1] and an LLM autoresearch agent [11] and iterate from there; the lessons below hold whichever search method produced the configuration. TABLE I: Matcher roster: one from each principal family. Input: E=embeddings, A=attribute features, G=graph structure. Matcher Family Input DeepMatcher MLP [16] E+A LightGBM GBDT [12] A GAT GNN [4] E+G I Experimental Setup Datasets. We evaluate on six deduplication benchmarks spanning five domains (Table I) and four orders of magnitude in scale. NCV denotes the 5M-record benchmark of Saeedi et al. [24]; it contains only generic structured fields used for ER evaluation—no behavioral, financial, or sensitive attributes. No proprietary, customer, or industry data is used anywhere; all experiments are reproducible from the cited public benchmarks. TABLE I: Benchmark datasets. |ℛ||R|=records, |||S|=schema fields, |||C|=ground-truth clusters, Sp. = fraction of pairs with missing fields. Citations: [17, 22, 23, 24]. Dataset Domain |ℛ||R| |||S| |||C| Sp. Restaurants Restaur. 864 5 752 2% Cora Biblio. 1,879 17 182 68% Geo Settl. Geogr. 3,054 3 820 11% DBLP-Sch. Biblio. 66,879 4 61,604 5% MB 200K Music 193,750 8 100,000 31% NCV Record 5,000,000 4 3,500,840 3% Splits and protocol. We split at the entity level—no record from the same entity appears in both training and test—and cap training and validation at 10K records each, regardless of dataset size, to reflect real-world deployment where labeling requires domain expertise. This yields heavily skewed ratios: Cora uses a conventional 42/13/45 split (1.9K records), but MusicBrainz 200K trains on just 5% (10K of 194K records). This design is deliberate: a method requiring abundant labels is impractical for self-serve deployment. We use a commercially available frontier LLM as teacher, all-MiniLM-L6-v2 (d=384d=384) as base encoder, and fixed seed 42. Pair-F1 is the primary metric throughout [16]; purity is reported as a secondary metric to distinguish over-splitting from over-merging. IV Lesson 1: Which Matcher Wins Depends on the Dataset Claim. No single matcher architecture dominates across ER problems. A tournament that trains three canonical architectures and auto-selects the winner eliminates a key human decision point. Evidence. Table I shows tournament results across all six benchmarks. No single family dominates, and the winning architecture changes character across datasets. TABLE I: Tournament leaderboard: Pair-F1 on held-out test (≤ 10K training records). Winner in bold. Purity in parentheses. †margin <0.001<0.001. Dataset DM LGBM GAT Winner Restaur. 0.948 (.99) 0.969 (1.0) 0.748 (.99) LGBM Cora 0.968 (.98) 0.891 (.98) 0.809 (.89) DM Geo Set. 0.979 (.99) 0.960 (.99) 0.964 (.99) DM DBLP-Sch. 0.160 (1.0) 0.541 (.94) 0.239 (1.0) LGBM MB 200K 0.964 (1.0) 0.948 (.99) 0.833 (.95) DM NCV 0.992 (1.0) 0.993 (1.0) 0.989 (1.0) LGBM† Score: DM 3/6, LGBM 3/6, GAT 0/6. Why the winner changes. Each winner reflects structural properties of its dataset. DeepMatcher wins on Cora, Geo Settlements, and MB 200K—datasets where field-level attention and soft semantic similarity matter (sparse attributes with OCR noise, paraphrase equivalence, subtle variant spellings). LightGBM wins on Restaurants, DBLP-Scholar, and NCV—datasets that are either small (the embedding tower lacks signal to fine-tune meaningfully) or dominated by structured identifier fields where exact-match and Jaro-Winkler features suffice. GAT wins nothing: at our 10K training cap, no dataset provides enough connected-component structure for 2-hop graph attention to outperform attribute-based methods, and GAT also suffers a train/test graph mismatch when the k-N graph at inference is built on a much larger test split. Self-serve systems cannot pick in advance. The winning architecture changes with dataset size, schema sparsity, and entity density—properties not known before running the data. A fixed “always DeepMatcher” policy loses on Restaurants, DBLP-Scholar, and NCV; “always LightGBM” loses on Cora, Geo Settlements, and MB 200K. The tournament costs nothing extra—all three matchers share the same training pairs and embeddings—and removes a decision point that would otherwise require dataset-specific expertise. Cost and latency. The teacher-student paradigm makes the tournament practical: the LLM teacher labels once during training; the distilled matcher handles all inference. The teacher costs ∼ $450/1M pairs at ∼ 2 s per pair; the tournament-winning matcher costs $12/1M pairs—a 37.5×37.5× cost reduction. LightGBM winners reach 222–263K pairs/sec on CPU; DeepMatcher winners run at 5–10K pairs/sec including SBERT inference. Practitioner guidance. Always run the tournament. The winner is also a diagnostic: LightGBM winning indicates a small or identifier-heavy dataset; DeepMatcher winning indicates soft similarity matters; GAT winning indicates a large, densely co-referent dataset (rare at scale). None of these conditions can be reliably predicted from schema inspection alone—only the data reveals which signal type dominates. V Lesson 2: Precision and Recall Break at Different Stages Claim. Precision and recall fail at structurally different points in the pipeline, and the common instinct—tune the matcher threshold—cannot fix either. V-A Root causes Recall is lost before the matcher runs. A pair never retrieved is lost forever—no threshold adjustment recovers it. Two retrieval failures dominate. (i) Embedding retrievers miss surface variants. Embedding similarity collapses “J. Smith” and “John Smith,” but OCR artifacts (“Heuslein”/“Hauslein”) or heavy abbreviations push genuine matches apart; HNSW’s M parameter leaves coverage gaps that compound at scale. (i) Embedding retrievers operate in a single modality. Exact categorical identifiers and structured codes produce no useful gradient in the embedding space; two records sharing an identical identifier but with variant text are never nominated. Precision fails because sparse records look like everything. A record with only one populated field has nothing to disagree with; it scores high against every other record sharing that field. A sparse record becomes a bridge: it matches above threshold against two unrelated dense clusters, and transitive closure chains them into one. This is not a matcher bug; it is the geometry of the problem. V-B Fix: separate tools per stage For recall: diversify the retriever. We compose two structurally different retrieval strategies and union their outputs. Strategy 1—HNSW ensemble for embedding-space coverage: an ensemble of HNSW indices with diverse (M,_ℎ)(M,ef\_search) configurations, ens=⋃i=1NKNNk(;Mi,i).C_ens= _i=1^N\,KNN_k(E;\,M_i,ef_i). (1) On MusicBrainz, a single M=16M=16 index misses 67 true pairs (97.7% recall); the ensemble recovers 11 of them (+0.4+0.4 p). Strategy 2—Identifier-based blocking for non-semantic matches: a lightweight exact-match inverted index over identifier fields, final=HNSW∪IDC_final=C_HNSW _ID. On NCV, identifier blocking recovers 7 true-positive pairs the HNSW ensemble missed entirely (+0.3+0.3 p); on DBLP-Scholar (no identifier fields), it contributes nothing—each strategy activates only where needed (Table IV). TABLE IV: Blocker recall (%) at k=20k=20. †No identifier fields; +ID contributes 0 additional pairs. Dataset Single Ensemble +ID Blk. DBLP-Scholar 100.0 100.0 100.0† MB 200K 97.7 98.1 98.1† NCV 97.7 97.8 98.1 For precision: hard rules on top of soft classifiers. A learned matcher is a function of its training distribution; production data drifts. A model that achieved 99% pairwise precision on validation can degrade when field-population rates shift—and in ER the cost is not a noisy prediction but a permanently merged cluster that downstream consumers inherit. Customers also treat certain rules as non-negotiable (“different phone number means different restaurant”), and no amount of retraining can guarantee a soft classifier will never violate them. We layer three deterministic guardrails on top of the matcher. (1) Sparsity-aware thresholds. Training data is typically balanced by entity size, but production has a long tail of sparse records with one or two populated fields. A global threshold over-accepts these pairs. We bin candidate pairs by the number of shared populated fields and learn a separate threshold per bin, with monotonicity enforced (θb≥θb+1 _b≥ _b+1): sparser pairs require higher confidence. This improves purity by +8.4+8.4 p on MusicBrainz and +1.5+1.5 p on NCV; no-op on Restaurants (full fields). (2) Per-field hard vetoes. For identifier fields, a hard rule zeroes the match probability when both records have the field populated but similarity falls below a field-specific floor: y^=0if∃f∈:both_havef∧simf<τf. y=0\;\;if\;\;∃ f \!:\;both\_have_f _f< _f. (2) This improves purity by +3.6+3.6 p on MusicBrainz and +32.8+32.8 p on Restaurants—datasets where identifier conflicts between genuinely different entities are common. Fields designated no_override encode unconditional business rules that no learned model can bypass. (3) Evidence and fast-path gates. Two additional cheap gates compose with the sparsity threshold and veto: an evidence gate rejects pairs that share too few populated fields for any matcher to be reliable, and a fast-path gate short-circuits pairs with very high confidence (≥99.5%≥ 99.5\%) that no hard rule contradicts. TABLE V: Precision ablation: cluster purity (%) as mechanisms are added incrementally. Baseline=tournament-winning matcher with global threshold. Bold=best per dataset. Dataset Baseline +Sparsity +Vetoes Restaurants 51.6 53.1 84.4 Cora 95.9 98.0 98.1 Geo Settl. 98.3 98.7 98.5 DBLP-Sch. 95.4 99.8 99.5 MB 200K 77.7 86.1 89.7 NCV 97.4 98.9 98.6 Practitioner guidance. Diagnose before tuning. If your largest clusters contain records from different entities: precision problem—add field vetoes, tighten sparsity thresholds. If singleton clusters should have been merged: recall problem—increase HNSW M, add ensemble indices, check blocking coverage. Fix retrieval gaps at the retrieval layer; lowering the matcher threshold cannot recover pairs the retriever never nominated. VI Lesson 3: One False Positive Can Collapse Your Clusters Claim. Connected components (C) clustering—the standard post-matching step in ER [5, 26]—works well when matchers are well-calibrated but fails when they are not. Center-based clustering avoids error propagation but under-merges. Active cross-cluster verification recovers recall without cascading false merges. Transitivity is an assumption, not a guarantee. ML matchers are not inherently transitive [3, 2]: a matcher may declare ⟨ri,rj⟩ r_i,r_j and ⟨rj,rk⟩ r_j,r_k as matches while ⟨ri,rk⟩ r_i,r_k is a non-match—a logically inconsistent triple that C resolves by merging all three. When the matcher’s false-positive rate is non-trivial, a single borderline edge propagates through Union-Find and chains unrelated clusters into mega-clusters. Figure 2 shows the cascade on three “Sakura Sushi” records: the matcher never directly scores r1↔r3r_1\! \!r_3, yet C merges Portland and Seattle into one cluster. ID Name Phone Addr City True R1 Sakura Sushi 503-0147 42 Oak St Portland A R2 Sakura Sushi — — — A R3 Sakura Sushi 206-9283 8 Elm Ave Seattle B Pair Score C outcome R1 ↔ R2 0.91 Merged (correct) R2 ↔ R3 0.88 Merged (correct) R1 ↔ R3 — Never scored ⇒ FP Figure 2: Transitive closure failure on a 3-record subset of Figure 1. The sparse bridge R2 lets C chain the Portland and Seattle entities without ever directly comparing R1 to R3. Verified merge (§VI) forces the missing comparison and blocks the merge. Verified merge. We replace blind transitive closure with a two-stage procedure. Stage 1 (Center assignment): each record joins the cluster of its single highest-scoring neighbor above threshold—no edges propagate. Stage 2 (Verified merge): for each Stage-1 cluster pair connected by at least one above-threshold edge in the original candidate set, (a) select up to k=3k=3 representatives per cluster closest to the centroid; (b) score all cross-cluster representative pairs through the matcher with hard vetoes enabled—generating direct pairwise evidence the blocking stage may never have produced; (c) if any cross-cluster pair triggers a veto or scores below threshold, block the merge. A single piece of negative evidence is sufficient—this asymmetry prevents error propagation. For connected components of 3+ Stage-1 clusters, we verify all (n2) n2 cluster pairs independently to prevent transitivity from re-entering through the merge pass itself. TABLE VI: Clustering ablation. Pair-F1, Adjusted Rand Index (ARI), and pairwise precision per dataset. Baseline=center-based clustering [9]. +Transitivity=connected components [5]. +Verification=verified merge. Bold=best per dataset. Baseline +Transitivity +Verification Dataset F1 ARI Prec. F1 ARI Prec. F1 ARI Prec. Restaurants 1.000 .900 1.000 1.000 .900 1.000 1.000 .900 1.000 Cora .321 .286 .986 .885 .853 .914 .876 .779 .994 Geo Settl. .889 .804 .994 .771 .831 .720 .972 .956 .989 DBLP .424 .426 .411 .371 .464 .312 .477 .475 .444 MB 200K .540 .577 .509 .000 .000 .000 .277 .356 .229 NCV .667 .668 .500 .002 .002 .001 .667 .647 .500 Reading the ablation. Table VI shows the three regimes. On Restaurants (clean, small) all three algorithms reach Pair-F1 = 1.0=\,1.0: the matcher is so well-calibrated that transitivity adds no false links. On Cora (99.7% validation precision) transitivity provides the largest gain—0.321 → 0.885 F1—by recovering multi-hop links the baseline fragments; verification is comparable (0.876) with higher precision. On Geo Settlements verification dominates: F1 0.889 → 0.972, while raw transitivity hurts (0.889 → 0.771) as geographically similar but distinct settlements get chained. On MB 200K and NCV transitivity catastrophically collapses F1 (0.540 → 0.000 and 0.667 → 0.002 respectively), as common field values chain unrelated records into mega-clusters. Verification preserves the baseline on NCV (0.667) and reduces damage on MB 200K (0.277): when the underlying matcher’s precision is too low (0.509), even verification gates cannot save it. The lesson. Transitivity is not a free lunch. It helps when the matcher is well-calibrated (Cora, Restaurants) and catastrophically hurts when the false-positive rate is high (MB 200K, NCV). The key predictor is baseline precision: above 0.9 transitivity is safe; below 0.5 it creates mega-clusters. Verified merge provides a safety net across all regimes and produces the best Pair-F1 on 4/6 datasets (strictly best on Geo Settlements and DBLP, tied for best on Restaurants and NCV). Its cost is modest: O(k2)O(k^2) additional matcher calls per candidate cluster pair, with k=3k=3 by default. VII Related Work Classical and PLM-based ER. The field traces from Fellegi-Sunter [8] through Magellan [13] to PLM-based systems. Ditto [15] achieved 29% F1F_1 improvement via BERT fine-tuning; Paganelli et al. [18] analyzed how BERT representations serve entity matching, and ZeroER [30] extends the paradigm to the unsupervised setting. Thirumuruganathan et al. [27] document a 40% F1F_1 drop from data heterogeneity—a direct motivator for our L1. Graph-based ER. HierGAT [31] and GraphER [10] encode relational structure with attention and differential dependencies respectively; Saeedi et al. [21] use graph metrics to drive cluster repair with active LLM feedback. These methods operate inside the matcher and so do not address the cross-stage cascades exposed by L2 and L3. LLM-based ER and distillation. Peeters and Bizer [20] report GPT-4 outperforms transferred PLMs by 40–68% zero-shot; Fan et al. [7] and Wang et al. [29] investigate cost-effective and selection-based ER paradigms. Wadhwa et al. [28] and Steiner et al. [25] distill LLM reasoning into smaller open-weight matchers, similar in spirit to our teacher / student split. Our work differs in conditioning the teacher on an inspectable SOP and in adding hard-rule safeguards (§V) and verified merge (§VI) that the teacher itself does not perform. SOP-driven agents. Agent-S [14] automates SOP execution; SOP-Bench [32] shows even GPT-4o achieves 30–70% on complex SOPs. We are, to our knowledge, the first to apply SOP-driven automation to entity resolution. VIII Limitations Our study has a few limitations that bound the scope of its conclusions. Public datasets by design. We deliberately restrict all experiments to six public benchmarks so that every result is fully reproducible and no proprietary or customer information is disclosed. Accordingly, we report standard accuracy metrics (Pair-F1, purity, and Adjusted Rand Index) rather than production outcomes such as business impact or robustness to live distribution drift. Data-scarce regime. We cap training and validation at 10K records each to reflect self-serve deployments where labeling requires scarce domain expertise. Some findings are specific to this budget—for example, “GAT never wins the tournament” should be read as “under a 10K-label budget,” since a graph-based matcher could plausibly overtake the other families given abundant labels. Limited roster and single configuration. The tournament covers three canonical families (DeepMatcher, LightGBM, GAT) and excludes LLM-based or cross-encoder matchers at inference. We use a single teacher LLM, base encoder (all-MiniLM-L6-v2), and seed (42), so we report point estimates rather than variance or significance tests. Verified merge has a precision floor. Verified merge is a safety net, not a cure: when baseline pairwise precision is very low (e.g. MusicBrainz 200K at ≈ 0.51), its gates reduce but do not prevent cluster collapse. Human effort is not quantified. SOP construction relies on a domain expert refining an LLM-drafted specification over a few iterations; we characterize this effort qualitatively rather than in person-hours. IX Conclusion We deployed entity resolution at scale on six benchmarks and walked away with three findings that we wish someone had handed us at the start. First, no single matching algorithm wins across datasets (§IV). In a small per-dataset bake-off across three canonical families, DeepMatcher and LightGBM each took the top spot on three benchmarks and GAT never won. Any team that commits to one architecture therefore loses on at least a third of the datasets it has yet to see, and the bake-off pays for itself many times over. Second, precision and recall fail at different stages and need different mechanisms (§V). Moving the matcher score threshold trades one off against the other and solves neither. We improve precision only by adding hard rule-based vetoes that the matcher cannot learn from data, and we improve recall only by running several diverse blocking strategies in parallel. Third, one false-positive link can silently merge unrelated entity groups because transitive closure compounds matcher errors (§VI). We repeatedly saw single low-evidence links chain hundreds of records that shared nothing in common into one giant cluster. Our verified-merge step—which re-runs the matcher on representative pairs across every candidate merge before committing— produces the best Pair-F1 on four of the six datasets and avoids the catastrophic precision collapse that transitive closure inflicts on the two largest. None of these three mechanisms is individually novel, but adopting all three as defaults rather than as escalation paths is what changed our production error profile. We hope the recipes here give other teams a shorter route to the same outcome. References [1] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama (2019) Optuna: a next-generation hyperparameter optimization framework. Cited by: §I. [2] Anonymous (2025) TransClean: finding false positives in multi-source entity matching under real-world conditions via transitive consistency. arXiv preprint arXiv:2506.04006. Cited by: §VI. [3] D. Baas, M. Dastani, and A. Feelders (2021) Exploiting transitivity constraints for entity matching in knowledge graphs. In arXiv preprint arXiv:2104.12589, Cited by: §VI. [4] S. Brody, U. Alon, and E. Yahav (2022) How attentive are graph attention networks?. In ICLR, Cited by: TABLE I. [5] V. Christophides, V. Efthymiou, T. Palpanas, G. Papadakis, and K. Stefanidis (2021) An overview of end-to-end entity resolution for big data. ACM Computing Surveys 53 (6), p. 127:1–127:42. Cited by: TABLE VI, §VI. [6] V. Christophides, V. Efthymiou, T. Palpanas, G. Papadakis, and K. Stefanidis (2021) An overview of end-to-end entity resolution. ACM Computing Surveys 54 (6), p. 1–42. Cited by: §I, §I. [7] Y. Fan, J. Li, S. Liu, and T. Rekatsinas (2024) Cost-effective in-context learning for entity resolution. In Proc. ICDE, Cited by: §I, §VII. [8] I. P. Fellegi and A. B. Sunter (1969) A theory for record linkage. J. American Statistical Association 64 (328), p. 1183–1210. Cited by: §I, §VII. [9] I. Hassanzadeh, M. A. Saeed, and A. Khodaei (2009) Swoosh: a generic approach to entity resolution. In VLDB, Cited by: TABLE VI. [10] J. Hu, M. Bewong, S. Kwashie, Y. Zhang, V. Nofong, J. Wondoh, and Z. Feng (2025) GraphER: when GDD meets GNN for entity resolution on property graphs. Information Systems 132, p. 102517. Cited by: §VII. [11] A. Karpathy (2026) Autoresearch: LLM-driven autonomous research loop. Note: ://github.com/karpathy/autoresearch Cited by: §I. [12] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu (2017) LightGBM: a highly efficient gradient boosting decision tree. In NeurIPS, p. 3146–3154. Cited by: TABLE I. [13] P. Konda et al. (2016) Magellan: toward building entity matching management systems. PVLDB 9 (12), p. 1197–1208. Cited by: §VII. [14] W. Li et al. (2025) Agent-S: LLM agentic workflow to automate standard operating procedures. arXiv:2503.15520. Cited by: §VII. [15] Y. Li, J. Li, Y. Suhara, A. Doan, and W.-C. Tan (2020) Deep entity matching with pre-trained language models. PVLDB 14 (1), p. 50–60. Cited by: §I, §VII. [16] S. Mudgal et al. (2018) Deep learning for entity matching: a design space exploration. In Proc. SIGMOD, p. 19–34. Cited by: §I, TABLE I, §I. [17] F. Naumann and M. Herschel (2010) HPI repeatability datasets for entity resolution: Restaurants, Cora, and DBLP-Scholar. Note: ://hpi.de/naumann/projects/repeatability/datasets.html Cited by: TABLE I. [18] M. Paganelli, D. Tiano, and F. Guerra (2024) Analyzing how BERT performs entity matching. VLDB Journal 33, p. 1–25. Cited by: §VII. [19] G. Papadakis, D. Skoutas, E. Thanos, and T. Palpanas (2020) Blocking and filtering techniques for entity resolution: a survey. ACM Computing Surveys 53 (2), p. 1–42. Cited by: §I. [20] R. Peeters and C. Bizer (2025) Entity matching using large language models. In Proc. EDBT, p. 338–350. Cited by: §I, §VII. [21] A. Saeedi, M. Hosseinzadeh, and E. Rahm (2025) Graph metrics-driven record cluster repair meets LLM-based active learning. ACM JDIQ 17 (2), p. 1–28. Cited by: §VII. [22] A. Saeedi, E. Peukert, and E. Rahm (2017) Using link discovery to enrich DBpedia with equivalent entity relationships. In Proc. ADBIS, Cited by: TABLE I. [23] A. Saeedi, E. Peukert, and E. Rahm (2018) Comparative evaluation of distributed clustering approaches for entity resolution. In Proc. EDBT, p. 181–192. Cited by: TABLE I. [24] A. Saeedi, E. Peukert, and E. Rahm (2018) Scalable matching and clustering of entities with FAMER. Complex Systems Informatics and Modeling Quarterly (CSIMQ) (16), p. 61–83. Cited by: §I, TABLE I. [25] M. Steiner, R. Peeters, and C. Bizer (2024) Fine-tuning large language models for entity matching. arXiv:2409.08185. Cited by: §VII. [26] R. C. Steorts, R. Hall, and S. E. Fienberg (2022) (Almost) all of entity resolution. Science Advances 8 (12). Cited by: §VI. [27] S. Thirumuruganathan et al. (2025) Heterogeneity in entity matching. arXiv:2508.08076. Cited by: §VII. [28] S. Wadhwa, L. Hawkins, C. Agrawal, B. C. Wallace, and A. Agrawal (2024) Learning from natural language explanations for generalizable entity matching. arXiv:2406.09330. Cited by: §VII. [29] T. Wang, Y. Zhang, and S. Roth (2025) Match, compare, or select? an investigation of LLMs for entity matching. In Proc. COLING, p. 89–110. Cited by: §VII. [30] R. Wu, S. Chaba, S. Sawlani, X. Chu, and S. Thirumuruganathan (2020) ZeroER: entity resolution using zero labeled examples. In Proc. SIGMOD, p. 1149–1164. Cited by: §VII. [31] D. Yao, Y. Gu, G. Cong, H. Jin, and X. Lv (2022) Entity resolution with hierarchical graph attention networks. In Proc. SIGMOD, p. 429–442. Cited by: §VII. [32] M. Yin et al. (2025) SOP-Bench: complex industrial SOPs for evaluating LLM agents. arXiv:2506.08119. Cited by: §VII. Appendix A Framework Details Figure 3 shows the full pipeline: onboarding (SOP construction, blocking, LLM labeling—all human-in-the-loop, top row), training (blocker and matcher distillation plus tournament, top row right), and inference (blocking, matching, clustering, audit—bottom row). LLMs touch only the human-facing stages; matching and clustering run entirely on lightweight distilled models. RecordsSOPConstructionCandidateBlockingLLMLabelingBlockerTrainingMatcherTrainingTournamentTeacherLLMNewRecordBlockingMatchingClusteringAuditTrailpairslabeled pairsencodermatchersSOPlabelsencodercandidatesscoresclustersbest matcherrefine SOPretrainTRAININFEROnboarding (human-in-the-loop)Training (automated) Figure 3: Full ER pipeline. LLM-driven stages (yellow, orange) involve humans; cost-sensitive matching/clustering (teal) run on lightweight distilled models. Solid arrows: data flow. Faded arrows: feedback loops (SOP refinement and matcher retraining). Why SOPs (extended). There is no universal definition of an “entity.” Consider two restaurant listings sharing phone, address, and city but with different names (e.g. Sakura Sushi and Thai Orchid at the same food-court address): a generic ER system says match (three strong fields agree); a domain expert says no—multiple tenants share one phone line. The rule “same name + same phone = match; phone alone is insufficient” cannot be inferred from data without examples no organization possesses. An LLM drafts an initial SOP from the schema; a domain expert refines it in 3–5 iterations. SOPs vary substantially across benchmarks—even two bibliographic datasets (Cora, DBLP-Scholar) require structurally different SOPs. Pipeline phases (extended). Onboarding (human-in-the-loop). The SOP-conditioned LLM teacher labels candidate pairs surfaced by approximate k-N blocking, producing per-field similarity assessments, a confidence score, and natural-language evidence; a domain expert reviews low-confidence labels and refines the SOP (typically 2–3 rounds). Training (automated). Labeled pairs train the blocker encoder and matcher independently; the tournament evaluates all matchers on held-out validation pairs. Inference. An ensemble of blockers (multiple HNSW indices plus identifier-based blocking) generates candidates that the tournament-winning matcher scores with safeguard layers (§V); verified merge (§VI) then clusters records using conservative direct assignment followed by verified cross-cluster merging. Appendix B SOP Excerpt Figure 4 shows a SOP excerpt for restaurant matching, illustrating how field importance, acceptable variations, and decision boundaries are encoded as inspectable YAML. sop: version: "1.2" domain: restaurant_matching field_hierarchy: critical: [name, phone] high: [address] medium: [city, cuisine] low: [zipcode] tolerances: name: - type: abbreviation - type: typo # max_edit_dist: 2 phone: - type: formatting decision_rules: match: ">=2 critical agree, 0 conflict" review: "1 conflict + >=2 high agree" non_match: ">=2 critical conflict" Figure 4: SOP excerpt for restaurant matching.

Instant research discovery