Paper deep dive
OpenSanctions Pairs: Large-Scale Entity Matching with LLMs
Chandler Smith, Magnus Sesodia, Friedrich Lindenberg, Christian Schroeder de Witt
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/20/2026, 2:27:19 PM
Summary
This paper introduces OpenSanctions Pairs, a large-scale entity matching benchmark derived from real-world international sanctions data, containing 755,540 labeled pairs across 293 heterogeneous sources. The study benchmarks a production rule-based matcher (nomenklatura RegressionV1) against various open- and closed-source Large Language Models (LLMs) in zero- and few-shot settings. Results show that off-the-shelf LLMs, particularly GPT-4o (98.95% F1) and DeepSeek-R1-Distill-Qwen-14B (98.23% F1), substantially outperform the rule-based baseline (91.33% F1). The authors conclude that pairwise matching performance is approaching a practical ceiling, suggesting future efforts should focus on pipeline components like blocking and clustering rather than further optimizing pairwise matching models.
Entities (8)
Relation Signals (7)
OpenSanctions Pairs → contains → 755,540 labeled pairs
confidence 98% · The dataset contains 755,540 labeled pairs spanning 293 heterogeneous sources
nomenklatura RegressionV1 → achievesf1score → 91.33%
confidence 97% · production rule-based baseline (91.33% F1)
GPT-4o → achievesf1score → 98.95%
confidence 97% · reaching up to 98.95% F1 (GPT-4o)
DeepSeek-R1-Distill-Qwen-14B → achievesf1score → 98.23%
confidence 96% · 98.23% F1 with a locally deployable open model (DeepSeek-R1-Distill-Qwen-14B)
OpenSanctions Pairs → derivedfrom → international sanctions aggregation
confidence 95% · derived from real-world international sanctions aggregation and analyst deduplication
LLMs → outperforms → nomenklatura RegressionV1
confidence 95% · Off-the-shelf LLMs substantially outperform the production rule-based baseline
DSPy MIPROv2 → optimizes → LLM prompts
confidence 90% · DSPy MIPROv2 prompt optimization yields consistent but modest gains
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We release OpenSanctions Pairs, a large-scale entity matching benchmark derived from real-world international sanctions aggregation and analyst deduplication. The dataset contains 755,540 labeled pairs spanning 293 heterogeneous sources across 31 countries, with multilingual and cross-script names, noisy and missing attributes, and set-valued fields typical of compliance workflows. We benchmark a production rule-based matcher (nomenklatura RegressionV1 algorithm) against open- and closed-source LLMs in zero- and few-shot settings. Off-the-shelf LLMs substantially outperform the production rule-based baseline (91.33\% F1), reaching up to 98.95\% F1 (GPT-4o) and 98.23\% F1 with a locally deployable open model (DeepSeek-R1-Distill-Qwen-14B). DSPy MIPROv2 prompt optimization yields consistent but modest gains, while adding in-context examples provides little additional benefit and can degrade performance. Error analysis shows complementary failure modes: the rule-based system over-matches (high false positives), whereas LLMs primarily fail on cross-script transliteration and minor identifier/date inconsistencies. These results indicate that pairwise matching performance is approaching a practical ceiling in this setting, and motivate shifting effort toward pipeline components such as blocking, clustering, and uncertainty-aware review. Code available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2603.11051v1
- Canonical: https://arxiv.org/abs/2603.11051v1
Trouble viewing inline? Open PDF directly →
Full Text
44,786 characters extracted from source content.
Expand or collapse full text
OpenSanctions Pairs: Large-Scale Entity Matching with LLMs Chandler Smith * 1 Magnus Sesodia * 1 Friedrich Lindenberg 2 Christian Schroeder de Witt 1 Abstract We release OpenSanctions Pairs, a large-scale en- tity matching benchmark derived from real-world international sanctions aggregation and analyst deduplication. The dataset contains 755,540 la- beled pairs spanning 293 heterogeneous sources across 31 countries, with multilingual and cross- script names, noisy and missing attributes, and set-valued fields typical of compliance workflows. We benchmark a production rule-based matcher (nomenklatura RegressionV1 algorithm) against open- and closed-source LLMs in zero- and few- shot settings. Off-the-shelf LLMs substantially outperform the production rule-based baseline (91.33% F1), reaching up to 98.95% F1 (GPT- 4o) and 98.23% F1 with a locally deployable open model (DeepSeek-R1-Distill-Qwen-14B). DSPy MIPROv2 prompt optimization yields con- sistent but modest gains, while adding in-context examples provides little additional benefit and can degrade performance. Error analysis shows complementary failure modes: the rule-based sys- tem over-matches (high false positives), whereas LLMs primarily fail on cross-script transliter- ation and minor identifier/date inconsistencies. These results indicate that pairwise matching per- formance is approaching a practical ceiling in this setting, and motivate shifting effort toward pipeline components such as blocking, cluster- ing, and uncertainty-aware review. Code avail- able athttps://github.com/chansmi/ OSINT_entity_resolution 1. Introduction Entity matching (EM) determines whether two records re- fer to the same real-world entity. The task is also known as entity resolution, record linkage, field matching, and * Equal contribution 1 Department of Engineering Science, Uni- versity of Oxford, Oxford, UK 2 OpenSanctions. Correspondence to: Chandler Smith<chandler.smith@eng.ox.ac.uk>. Preliminary work. Under review by the International Conference on Machine Learning (ICML). Do not distribute. deduplication, with the latter typically referring to match- ing records within the same dataset and record linkage to matching across different datasets (Fellegi & Sunter, 1969; Christen, 2012). EM is a core component of data integration pipelines, as matching errors propagate directly to down- stream entity graph construction and analysis (Fellegi & Sunter, 1969; Getoor & Machanavajjhala, 2012). Existing EM benchmarks are largely drawn from e- commerce and bibliographic domains and typically rely on a small number of datasets, often as few as two, with closely aligned schemas, fixed field sets, and relatively standard- ized entity representations (K ̈ opcke et al., 2010a;b; Mudgal et al., 2018b; Peeters et al., 2023). As a result, they fail to reflect deployment settings in which entity matching is performed across tens to hundreds of datasets with hetero- geneous schemas, overlapping but non-identical fields, and substantial data quality issues such as missing values, in- consistencies, and noise. In these settings, entities may be represented using aliases or transliterations and appear in non-Latin scripts, e.g., Cyrillic or Chinese characters, mak- ing high-precision matching significantly more challenging. Sanctions data provides a politically and operationally im- portant example of this setting. International authorities publish overlapping lists of sanctioned individuals and orga- nizations, and financial institutions must screen customers against these lists to avoid regulatory penalties and reputa- tional damage, with compliance failures resulting in fines of hundreds of millions of dollars (Sterling & Meijer, 2018). Effective entity matching is critical for both regulatory com- pliance and detecting sanctions evasion, where individuals obscure ownership through shell companies and name vari- ations across jurisdictions (Montenarh & Marsden, 2024). Despite this practical importance, sanctions data are typi- cally proprietary or access-restricted, and to our knowledge no public benchmark for pairwise entity matching on sanc- tions data has been published. To address this gap, we present the first systematic study of pairwise EM techniques on large-scale sanctions data in collaboration with OpenSanctions 1 . We aggregate 293 datasets from 31 countries, with human analysts performing pairwise matching (OpenSanctions, 2021a). From this pro- cess, we extract 755,540 labeled entity pairs covering over 1 https://opensanctions.org 1 arXiv:2603.11051v1 [cs.IR] 24 Feb 2026 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs one million unique entities. Crucially, these labels reflect ex- pert human decisions under incomplete evidence, mirroring real-world EM practice rather than definitive ground truth. Our experiments reveal that methods based on large lan- guage models (LLMs) achieve F1 scores up to 98.95%, compared to 91.33% for the legacy rule-based matcher pre- viously used in production. This performance approaches observed human labeling consistency on the dataset, indi- cating diminishing returns from further optimization of pair- wise matching models. We therefore argue that future work should prioritize other components of the entity resolution pipeline, including blocking, clustering, provenance model- ing, and uncertainty handling (Getoor & Machanavajjhala, 2012; Christophides et al., 2020). Our contributions: 1.We release and document a large-scale EM dataset derived from real sanctions OSINT data, an order of magnitude larger than existing benchmarks and sub- stantially more heterogeneous. 2. We establish baselines comparing legacy rule-based matching with LLM-based approaches, demonstrating that LLMs reliably replicate expert decisions. 3.We analyze failure modes to identify where future re- search effort is most productive beyond pairwise match- ing. 2. Related Work Datasets A variety of datasets have been developed to evaluate entity matching methods. Standard benchmarks include product datasets such as Abt/Buy and Amazon/- Google Products, as well as bibliographic datasets such as DBLP/ACM and DBLP/Scholar (Rahm & Leipzig, 2019; Mudgal et al., 2018b; K ̈ opcke et al., 2010a;b). These typi- cally involve pairwise matching between two sources with clean, structured, and monolingual attributes. More recent benchmarks such as WDC Products increase scale and diffi- culty, but remain focused on e-commerce settings with lim- ited source heterogeneity. Across domains, existing datasets are characterized by few sources, scalar attributes, and lim- ited variation in scripts or provenance, and thus do not reflect the complexity of multi-source, multilingual environments. Our dataset differs along several dimensions. It operates in a sanctions OSINT setting with entities aggregated from 293 heterogeneous sources across 31 countries. Names and attributes span multiple scripts and transliteration stan- dards, and entities are represented by set-valued and time- dependent attributes rather than single records. At 755,540 labeled pairs, it is substantially larger than most prior bench- marks. We provide a quantitative comparison in Table 1. Entity Matching MethodsEM originated in probabilistic formulations developed for administrative record linkage (Fellegi & Sunter, 1969; Elmagarmid et al., 2007). Early work established the core pipeline of blocking, matching, and clustering, introducing methods to reduce candidate comparison costs (Hern ́ andez & Stolfo, 1995; Getoor & Machanavajjhala, 2012; O’Hare et al., 2018; Papadakis et al., 2020). Neural approaches subsequently improved robustness to noisy and text-heavy data by learning attribute representations directly, with systems such as Magellan demonstrating gains over rule-based methods (Konda et al., 2016; Mudgal et al., 2018a). Pretrained language mod- els further advanced EM by encoding entire entity pairs as sequences and fine-tuning transformer-based classifiers (Li et al., 2020; Ornstein, 2025). Parallel work has ex- plored probabilistic and Bayesian formulations that explic- itly model uncertainty and matching constraints (Steorts et al., 2015; Sadinle, 2016). The most recent line of work applies large language models to entity matching via prompting, in-context learning, and fine-tuning, often achieving strong performance with mini- mal task-specific adaptation (Narayan et al., 2022; Tang et al., 2022; Peeters & Bizer, 2023; Peeters, 2025; Fan et al., 2023; Li et al., 2024a;b; Steiner et al., 2025; Wang et al., 2025). Most closely related to our work, Peeters et al. (2024) show that LLM-based methods are more robust to out-of-distribution entities than fine-tuned pretrained lan- guage models. Efficient variants using smaller language models have also been proposed (Zhang et al., 2025). For broader context, surveys provide overviews of classical and neural approaches (Elmagarmid et al., 2007; Barlaug & Gulla, 2021). Beyond predictive performance, recent work has examined explainability for entity matching: Barlaug (2022) propose LEMON, a model-agnostic method that produces dual explanations for matches and non-matches and introduces attribution potential to help users understand decision boundaries. Recent work has also applied NLP methods specifically to sanctions screening, demonstrating improvements over traditional fuzzy matching approaches (Kim & Yang, 2024). 3. Dataset 3.1. Background International sanctions are imposed by authorities including the UN Security Council, US Treasury (OFAC), EU Foreign Service, and UK HMT (Felbermayr et al., 2020). These lists overlap substantially, and the same individual may appear on multiple lists under different name transliterations, aliases, or with varying metadata. OpenSanctions addresses this fragmentation by aggregating 293 source datasets from 31 countries into a unified database. As part of this integration, human analysts perform pairwise deduplication to identify 2 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs Table 1.Comparison with existing entity matching benchmarks. #Matches refers to the number of positive pairwise matches; * denotes deduplicated entities, not pairwise matches. Dataset#Entities#Matches#Sources#Unique FieldsDomain Fodors-Zagat (Mudgal et al., 2018a;b)94611026Restaurant iTunes-Amazon (Mudgal et al., 2018a;b)53913228Music Walmart-Amazon (Mudgal et al., 2018a;b)10,24296225E-commerce Abt-Buy (Rahm & Leipzig, 2019)1,081 + 1,0921,09724E-commerce Amazon-GoogleProducts (Rahm & Leipzig, 2019)1,363 + 3,2261,30024E-commerce DBLP-ACM (Rahm & Leipzig, 2019)2,614 + 2,2942,22424Bibliographic DBLP-Scholar (Rahm & Leipzig, 2019)2,616 + 64,2635,34724Bibliographic WDC Products (Peeters et al., 2023)11,7152,162 ∗ 3,2595E-commerce MusicBrainz20k (Rahm & Leipzig, 2019)19,37516,250510Music OpenSanctions (ours)1,002,093581,149293131Sanctions when records from different sources refer to the same real- world entity. 3.2. Construction and Analysis The dataset consists of 755,540 pairwise entity matching judgements, derived from OpenSanctions’ deduplication workflow (OpenSanctions, 2025; 2021b). Construction fol- lows a two-stage process: 1.Blocking: An inverted index generates candidate pairs based on shared properties (name fragments, identi- fiers, phone numbers). This reduces theO(n 2 )com- parison space while remaining tolerant to spelling vari- ations via character n-grams and transliteration han- dling. 2.Manual Review: Analysts review candidate pairs via a text-based interface, comparing entities side-by-side. For ambiguous cases, external research supplements the decision. Each judgement links to a canonical entity ID, enabling transparent audit trails and reversibility. Each record follows the schema: "left": ...entity..., "right": ...entity..., "judgement": "positive|negative" The resulting corpus covers 1,002,093 unique entities across schemas including Person (320K), Company (48K), and Organization (47K). Each entity contains up to 132 property fields (names, aliases, birth dates, nationalities, addresses, identifiers), with entities averaging 1.32 source datasets and 1.11 aliases. The dataset exhibits 76.9% positive (match) and 23.1% negative (non-match) labels. The imbalance reflects the blocking strategy that prioritizes likely matches. Importantly, these are human analyst judgements prioritiz- ing precision over recall, not definitive ground truth. 4. Pairwise Entity Matching LetDdenote a structured dataset withnentities, where each entitye i ∈Dis represented by a set of attribute-value pairs: e i =(a j , V i,j ) 1≤j≤m . Here,a j denotes thej-th attribute (e.g.,name,birthDate), andV i,j is the corresponding value set, which may be empty, singleton, or multi-valued. This set-valued representation is essential for sanctions data, where entities commonly have multiple aliases, nationalities, or addresses. 4.1. Task Definition Given two entitiese, e ′ ∈D, predict whether they refer to the same real-world entity. We define a matching function: M :D×D →0, 1(1) whereM(e, e ′ ) = 1indicates a positive match (the entities represent the same real-world object) andM(e, e ′ ) = 0 indicates a negative match. 5. Experiments and Analysis 5.1. Baselines We compare three method categories aligned with Table 2: 1.Rule-Based Baseline: We use the RegressionV1 algo- rithm from OpenSanctions’nomenklaturaframe- work currently used to assist human experts (Open- Sanctions, 2025). This method computes 18 features spanning name similarity (token overlap, Levenshtein distance, phonetic matching), date comparison (exact match, year match, disjoint years), identifier overlap (passport, tax ID), and demographic consistency (coun- try, gender). A pre-trained logistic regression model outputs match probabilities. 2. Open-Source LLMs: We choose Llama-3.1-8B- Instruct and DeepSeek-R1-Distill-Qwen-14B, with the 3 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs "id": "pk-cnic-3520114139885", "caption": "Khalid Mehmood", "datasets": ["pk_proscribed"], "properties": "name": ["Khalid Mehmood"], "fatherName": ["Muhammad Khan"], "idNumber": ["3520114139885"], "address": ["LAHORE, Punjab"], "country": ["pk"] ... ̸= "id": "pk-cnic-3710502620181", "caption": "Khalid Mehmood", "datasets": ["pk_proscribed"], "properties": "name": ["Khalid Mehmood"], "fatherName": ["Haji Muhammad"], "idNumber": ["3710502620181"], "address": ["ATTOCK, Punjab"], "country": ["pk"] ... Figure 1. Example entity pair illustrating the difficulty of pairwise matching. These two individuals on Pakistan’s proscribed persons list share the exact same name, country, and sanctions program, yet they are different people with distinct national ID numbers and fathers. The Nomenklatura baseline assigned this pair a 0.98 match score, near the maximum possible, demonstrating how name-based matching fails for common names without unique identifiers. latter a DeepSeek-distilled model built on a Qwen2.5- 14B backbone. Their local deployability, absence of usage-based costs, and strong reproducibility align well with OpenSanctions’ values and use case. 3. Proprietary LLMs: We compare seven closed-source models spanning different model scales ranging from GPT-5 Nano, a smaller and more lightweight model, to GPT-5.2 Pro and Claude Opus 4.5, both frontier- scale models, all accessed via API. This comparison provides a reference point for maximal achievable per- formance. 5.2. Evaluation Protocol Dataset Sampling From the full corpus of 755,540 la- beled pairs, we construct label-stratified samples of both 10,000 and 1,000 pairs for development and evaluation. We primarily use the 10,000 pair set, but in cases where prompt optimization or model inference was prohibitively expen- sive, the 1,000 pair set was used. All sampling uses a fixed random seed of 42 for reproducibility. For thenomenklaturaRegressionV1 baseline, we use 2,000 pairs to tune the threshold (which we find to be 0.15), and evaluate on the remaining 8,000. For Llama-3.1-8B, we use 200 pairs for prompt optimiza- tion, and evaluated on 800 pairs for 0-shot, 2- and 4-shot (optimized), and on 9800 pairs for 0- and 8-shot (optimized). For DeepSeek-R1-Distill-Qwen-14B, we optimize on 200 pairs and evaluate on 3,680 pairs (due to runtime con- straints), compared to 1,960 pairs in the unoptimized case. For GPT-5 Nano, and Claude Opus 4.5, we use 200 pairs for prompt optimization, and evaluate on the remaining 9,800. For GPT-5.2 Pro, we did not do any prompt optimization and instead evaluated on 800 test pairs, with reasoning effort set to medium. Metrics We report standard binary classification metrics: accuracy, precision, recall, and F1 score. Given the class im- balance (76.9% positive) and the asymmetric costs of false positives versus false negatives in compliance settings, we emphasize F1 as the primary metric. We include a confusion matrix to highlight the relative strengths and weaknesses between models. 5.3. Implementation Details Local Model Inference. All open-source model weights are obtained from Hugging Face using the transformers library. Inference is performed using tensor parallelism across an 8x46GB NVIDIA A40 GPU Cluster or 8x128GB AMD MI250X. For locally served models, we enforce structured outputs via constrained decoding with the Outlines library to ensure that generated outputs conform to the expected schema. For deterministic decoding, we set temperature≤ 0.01and dosample=False(greedy decoding). Maximum new tokens is set to 512. Input sequences are truncated to 4,096 tokens when necessary, though the longest entity pair in our dataset (an entity with 304 aliases) requires approximately 2,000 tokens, well within this limit. API Model Inference. API-based models are accessed us- ing LightLLM 2 , or directly through the model provider’s API. Structured JSON outputs are used to extract class de- cisions in a consistent format. For all models, we set the temperature to 0 to ensure deterministic decoding and re- producibility. GPT-5.2 Pro supports explicit reasoning ef- 2 https://github.com/ModelTC/LightLLM 4 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs fort configuration, which setreasoning="effort": "medium". Optimization.Prompts are optimized using DSPy MIPROv2 (DSPy, n.d.), which performs Bayesian optimiza- tion over the space of instruction variants and demonstration subsets. The optimizer is initialized with the same manually- designed conflict-focused prompt 5.4. MIPROv2 generates instruction variants by prompting the target LLM to propose alternative phrasings, then evaluates each candidate on the 200-pair development set. We configure MIPROv2 to eval- uate 15 candidate programs, each consisting of a distinct (instruction, demonstrations) combination. Each candidate includes up to 8 in-context demonstrations from the devel- opment set. We set the number of optimization trials to approximately1.5×the number of candidates (∼22–23 tri- als). Instruction variants are generated with temperature 1to encourage diversity in proposed phrasings. Example optimization prompts are provided in Appendix A. 5.4. Default Prompt Our baseline prompt was manually designed around a conflict-detection framing, achieving 94.95% F1 on GPT- 5 Nano. The key insight is framing entity resolution as contradiction detection rather than similarity matching. System Prompt. You are an expert entity resolution system. Your task is to determine if two entity records refer to the same real- world entity. Primary task: Identify CONFLICTS, not similarities. - Name variations (transliterations, nicknames, titles) are common - Missing fields are normal - absence of data is NOT evidence of difference - Same entity often appears across multiple sources with variations Decision Process: 1. Look for CONTRADICTORY evidence ( different dates, conflicting IDs, incompatible attributes) 2. If NO contradictions found -> POSITIVE ( same entity) 3. Only NEGATIVE if explicit conflicts exist The DEFAULT is POSITIVE unless you find proof of difference. User Prompt Template. Determine if these two entity records refer to the same real-world entity. === Entity A === entity_a === Entity B === entity_b Classify as: - POSITIVE: Same entity (no conflicts found ) - NEGATIVE: Different entities (explicit conflicts exist) Respond with a JSON object containing: - "classification": "positive" or "negative " - "reasoning": Brief explanation focusing on conflicts or lack thereof Field Definitions.The optimized prompt uses a structured input/output format with the following field semantics: Field PrefixDescription Entity A:First entity record with names, dates, IDs, and other attributes Entity B: Second entity record with names, dates, IDs, and other attributes Reasoning:Brief explanation of decision, focusing on conflicts or lack thereof Classification: positive if same entity,negativeif different entities 5.5. Results Table 2 presents the main results. We highlight several key findings: LLMs Achieve Human-Expert Performance. LLM- based approaches reach near-ceiling performance (99% F1), substantially outperforming the rule-based baseline (91.33% F1) currently used in production. This indicates that a task traditionally requiring significant manual effort and domain expertise can be can be automated with minimal human intervention. Among open-source models, DeepSeek-14B performs best, reaching 98.23% F1 and closely approaching the performance of proprietary LLMs such as GPT-5.2 Pro (98.75% F1). Prompt Optimization Provides Minor Gains. Auto- matic prompt optimization via MIPROv2 yields consistent improvements over unoptimized prompts. For Llama-3.1- 8B, the optimized zero-shot configuration achieves 95.94% F1, compared to 94.05% F1 with an unoptimized zero-shot prompt, representing a gain of 1.9 F1 points. These gains suggest that the underlying decision rule required for the task is already present in the model, and that performance improvements can arise from reducing instruction ambigu- 5 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs Table 2.Main results. Configurations marked “(opt)” use DSPy MIPROv2 prompt optimization. Best results in Accuracy and F1 are shown in bold. See section 5.2 for the sample sizes used. ModelConfigAcc.↑F1↑Prec.↑Recall↑ Rule-Based Baseline nomenklatura/RegressionV1—85.4591.3384.4699.42 Open-Source Models Llama-3.1-8B0-shot90.4094.0589.8498.67 Llama-3.1-8B0-shot (opt)93.8895.9497.9893.97 Llama-3.1-8B2-shot (opt)93.0095.4694.8596.08 Llama-3.1-8B4-shot (opt)93.2595.6494.7296.57 Llama-3.1-8B8-shot (opt)93.8895.9497.9893.97 DeepSeek-R1-Distill-Qwen-14B0-shot96.5397.7696.2499.33 DeepSeek-R1-Distill-Qwen-14B0-shot (opt)97.3198.2398.6097.86 Proprietary Models GPT-3.5 Turbo0-shot91.8494.4998.1691.10 GPT-4o0-shot98.3898.9598.7899.11 GPT-5 Nano0-shot92.4095.2491.7798.99 GPT-5 Nano0-shot (opt)92.7695.4692.2198.94 GPT-5.2 Pro0-shot97.7598.5398.3798.69 GPT-5.2 Pro4-shot98.0798.7598.7598.75 GPT-5.2 Pro8-shot97.7998.5698.4698.65 Claude 3 Haiku0-shot88.0092.6886.7799.44 Claude 3.7 Sonnet0-shot96.2597.5099.0096.05 Claude Opus 4.50-shot93.2695.4599.1192.05 Claude Opus 4.50-shot (opt)93.5495.6599.1392.41 ity (Figure 2). However, the optimization appears to have little effect on proprietary models. Prompt Optimization and Few-Shot Learning Don’t Mix. When in-context examples are introduced, even un- der prompt optimization for the corresponding few-shot configuration, we do not observe further improvements. Llama-3.1-8B with optimized 2-shot, 4-shot, and 8-shot prompts underperforms the optimized zero-shot baseline (95.46%, 95.64%, and 95.94% vs. 95.94% F1), while a 4-shot configuration yields only negligible changes for GPT- 5-Nano. This pattern suggests that, despite being jointly optimized, in-context demonstrations introduce inductive biases that compete with the instruction-level decision strat- egy learned during prompt optimization, leading to reduced performance. Model Size vs. Speed Trade-off.DeepSeek-14B slightly outperforms Llama-3.1-8B (98.23% vs. 95.94% F1) but requires approximately 3×more inference time per example (∼70s vs.∼25s). For deployment at scale, Llama-3.1-8B offers a favorable accuracy-throughput trade-off. Performance vs. Model Release Date. Figure 3 plots unoptimized (0-shot) F1 against model release date for all evaluated systems. The rule-based Nomenklatura baseline, fixed at 91.3% F1, serves as a static reference. In early 2023, the first LLMs we evaluate (GPT-3.5 Turbo, Claude 3 Haiku) achieved 92–94% F1, only marginally above the rule-based system. By mid-2024, newer architectures (GPT-4o, Llama- 3.1-8B, DeepSeek-R1-Distill-Qwen-14B) reached 94–99% F1, and by late 2025 the best proprietary models (GPT- 5.2 Pro, Claude 3.7 Sonnet) approach 98–99% F1. The trendline (linear regression) across all LLM results shows a steady upward trajectory, widening the gap over the static rule-based baseline from roughly 2 F1 points in early 2023 to over 7 points by late 2025. This trend reinforces that pairwise matching performance is approaching a practical ceiling, and that further gains are more likely to come from improvements to other pipeline components such as block- ing, clustering, and uncertainty-aware review, rather than from additional pairwise model tuning. 5.6. Failure Modes Table 3 presents confusion matrices revealing distinct failure patterns across methods. Rule-Based: High False Positive Rate. The Nomen- klatura baseline yields high recall but produces many false positives. This reflects OpenSanctions’ operational prior- ity: in compliance screening, missing a sanctioned entity is costlier than flagging a false match for human review. The low threshold (0.15) ensures all true matches are captured at the expense of precision. 6 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs Before Optimization (Manual Design) System: You are an expert entity resolution system. Your task is to determine if two entity records refer to the same real-world entity. Primary task: Identify CONFLICTS, not similarities. - Name variations (transliterations, nicknames, titles) are common - Missing fields are normal - absence of data is NOT evidence of difference - Same entity often appears across multiple sources with variations Decision Process: 1. Look for CONTRADICTORY evidence (different dates, conflicting IDs, incompatible attributes) 2. If NO contradictions found -> POSITIVE (same entity) 3. Only NEGATIVE if explicit conflicts exist The DEFAULT is POSITIVE unless you find proof of difference. User: Determine if these two entity records refer to the same real-world entity. === Entity A === entity a === Entity B === entityb Classify as POSITIVE (same entity) or NEGATIVE (different entities). Respond with JSON: "classification": ..., "reasoning": ... After Optimization (MIPROv2) Instructions: Given two entity records, determine if they refer to the same real-world entity. Analyze for contradictions in attributes like dates, IDs, or other details. Consider common variations in names and missing data. Classify as ’positive’ if no conflicts are found, otherwise ’negative’. Provide a clear reasoning for your classification. Input: Entity A: entity a Entity B: entityb Output: Reasoning: reasoning Classification: positive|negative Figure 2.Prompt comparison showing the full manual prompt (left, 170 words) versus the MIPROv2-optimized prompt (right, 50 words). The optimizer discovered a more concise framing while preserving the conflict-detection principle. Table 3. Confusion matrix for the test results.DeepSeek- Qwen=DeepSeek-R1-Distill-Qwen-14B. All 0-shot, not opti- mized. MethodTP%TN%FP%FN% Nomenklatura0.7660.0880.1410.004 Llama-3.1-8B0.7590.1450.0860.010 DeepSeek-Qwen0.7580.2080.0300.005 GPT-3.5 Turbo0.7010.2180.0130.068 GPT-4o0.7620.2220.0090.007 GPT-5 Nano0.7610.1630.0680.008 GPT-5.2 Pro0.7560.2210.0130.010 Claude 3 Haiku0.7590.1200.1160.004 Claude 3.7 Sonnet0.7310.2320.0070.030 Claude Opus 4.50.7080.2250.0060.061 LLM Challenges: Transliteration and Off-by-One Er- rors. LLM-based methods exhibit two primary failure modes. First, cross-script transliteration differences (e.g., Arabic, Cyrillic, and Latin) can obscure genuine matches when name variation is substantial. Second, minor discrep- ancies in dates or identifiers (“off-by-one” errors) can trigger incorrect negative predictions: the conflict-focused prompt may overemphasize small inconsistencies that arise from data entry noise rather than from distinct entities. Viewed differently, this behavior suggests that LLMs can also serve as effective detectors of underlying data quality issues. Complementary Failure Modes. Rule-based and LLM methods exhibit complementary error patterns: Nomen- klatura over-predicts matches while LLMs occasionally under-predict on hard cases involving transliteration. An en- semble approach using Nomenklatura for initial high-recall filtering followed by LLM re-ranking could leverage these complementary strengths. 6. Limitations Our approach has several limitations. The evaluation labels reflect expert judgments under incomplete evidence rather than definitive ground truth, so model-label disagreements may in some cases reflect labeling uncertainty rather than prediction errors. The unique characteristics of sanctions data, including structured identifiers, cross-jurisdictional name variations, and compliance-oriented schemas, may limit generalizability to other entity matching domains such as e-commerce or bibliographic deduplication. Our evaluation also has practical constraints. Due to compu- tational cost and resource limitations, we evaluated models on samples of varying size, with some models tested on as 7 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs Nov 2022 Feb 2023 May 2023 Aug 2023 Nov 2023 Feb 2024 May 2024 Aug 2024 Nov 2024 Feb 2025 May 2025 Aug 2025 Nov 2025 Feb 2026 Model Release Date 88 90 92 94 96 98 100 F1 Score (%) Nomenklatura RegressionV1 (91.3%) GPT-3.5 Turbo (94.5%) GPT-4o (99.0%) GPT-5 Nano (95.2%) GPT-5.2 Pro (98.5%) Claude 3 Haiku (92.7%) Claude 3.7 Sonnet (97.5%) Claude Opus 4.5 (95.5%) Llama-3.1-8B (94.0%) DeepSeek-R1- Distill-Qwen-14B (97.8%) Entity Resolution F1 vs. Model Release Date (0-shot, unoptimized) Rule-based OpenAI Anthropic Open-source Rule-based baseline Trend (LLMs) Figure 3.Unoptimized (0-shot) F1 score versus model release date for all evaluated models. The dashed grey line indicates the rule-based Nomenklatura baseline (91.3% F1); the blue dashed trendline is a linear fit across all LLM models. Over three years, LLM performance has risen from 92–94% F1 (early 2023) to 98–99% F1 (late 2024–2025), widening the gap over the static rule-based system from∼2 F1 points to over 7. few as 800 pairs; while all samples are label-stratified and drawn from the same distribution, smaller samples carry greater statistical uncertainty. LLM performance depends on prompt design. Our MIPROv2 prompt optimization was computationally intensive, which constrained the training sample and may have limited potential gains; the resulting prompts may also require tuning for other datasets or use cases. Specifically, from a sample ofn=10,000, we set aside200 examples for MIPROv2 optimization. In one instance (gpt- 4o), this quantity of prompt optimization yielded no im- provements over the baseline prompts, highlighting the need for additional MIPROv2 samples. Additionally, our evaluation focuses on batch deduplica- tion rather than real-time screening; the latency of current LLMs may be prohibitive for transaction-level matching where sub-second response times are required. This is not a failure of the benchmark, but a limitation of the potential implementation. Finally, while LLM explanations may hallucinate rationales, grounding predictions in the full entity context mitigates this risk. Inference cost remains non-trivial but is offset by the one-time nature of deduplication, viable open-source alternatives, and declining compute costs. 7. Conclusion We introduced OpenSanctions Pairs, the first large-scale, public entity matching benchmark derived from real-world international sanctions data. With over 755K labeled pairs spanning 293 heterogeneous sources, the dataset captures the multilingual, multi-schema, and noisy conditions under which entity matching systems are deployed in practice. Our experiments show that off-the-shelf LLMs substan- tially outperform a production-grade rule-based system and achieve performance close to observed human expert con- sistency. This result indicates that pairwise entity matching in this setting is no longer the primary bottleneck, and that a task previously requiring significant manual effort and domain expertise can be reliably automated within a human- in-the-loop workflow. These findings motivate a shift in research focus beyond isolated pairwise classification. Future work should ad- dress the broader entity resolution pipeline, including auto- mated blocking, clustering, and uncertainty-aware decision making, where human judgment and system design remain critical. While prompt optimization yields consistent but modest gains, further improvements are likely to come from integrating LLMs more deeply into pipeline-level decisions rather than refining pairwise prompts alone. 8 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs Impact Statement This work releases a large, public benchmark for entity matching derived from real-world international sanctions data, helping bridge the gap between existing academic benchmarks and operational compliance settings. By evalu- ating LLMs on a multilingual and heterogeneous task, the study supports more realistic assessment of entity resolution methods and highlights where automated systems can assist expert analysts. The dataset and analysis are intended to en- courage safer, human-in-the-loop deployment by clarifying model limitations and reducing unnecessary false positives in compliance workflows. References Barlaug, N. Lemon: Explainable entity matching, 2022. URL https://arxiv.org/abs/2110.00516. Barlaug, N. and Gulla, J. A. Neural networks for entity matching: A survey. ACM Trans. Knowl. Discov. Data, 15 (3), April 2021. ISSN 1556-4681. doi: 10.1145/3442200. URL https://doi.org/10.1145/3442200. Christen, P.Data Matching:Concepts and Tech- niques for Record Linkage, Entity Resolution, and Duplicate Detection.Springer Berlin Heidelberg, 2012.ISBN 9783642311642.doi:10.1007/ 978-3-642-31164-2. URLhttp://dx.doi.org/ 10.1007/978-3-642-31164-2. Christophides, V., Efthymiou, V., Palpanas, T., Papadakis, G., and Stefanidis, K. End-to-end entity resolution for big data: A survey, 2020. URLhttps://arxiv.org/ abs/1905.06397. DSPy.Miprov2 — dspy api optimizer.https: //dspy.ai/api/optimizers/MIPROv2/,n.d. URLhttps://dspy.ai/api/optimizers/ MIPROv2/. Accessed: 2026-01-29. Elmagarmid, A. K., Ipeirotis, P. G., and Verykios, V. S. Duplicate record detection: A survey. IEEE Trans. on Knowl. and Data Eng., 19(1):1–16, January 2007. ISSN 1041-4347. Fan, M., Han, X., Fan, J., Chai, C., Tang, N., Li, G., and Du, X. Cost-effective in-context learning for en- tity resolution: A design space exploration, 2023. URL https://arxiv.org/abs/2312.03987. Felbermayr, G., Kirilakha, A., Syropoulos, C., Yalcin, E., and Yotov, Y. V. The global sanctions data base. European Economic Review, 129:103561, 2020. ISSN 0014-2921. doi: https://doi.org/10.1016/j.euroecorev.2020.103561. URLhttps://w.sciencedirect.com/ science/article/pii/S0014292120301914. Fellegi, I. P. and Sunter, A. B.A theory for record linkage.Journal of the American Sta- tisticalAssociation,64(328):1183–1210,1969. doi:10.1080/01621459.1969.10501049.URL https://w.tandfonline.com/doi/abs/ 10.1080/01621459.1969.10501049. Getoor, L. and Machanavajjhala, A. Entity resolution: Theory, practice & open challenges. Proceedings of the VLDB Endowment, 5:2018–2019, 08 2012. doi: 10.14778/2367502.2367564. Hern ́ andez, M. A. and Stolfo, S. J. The merge/purge prob- lem for large databases. In Proceedings of the 1995 ACM SIGMOD international conference on Management of data - SIGMOD ’95, SIGMOD ’95, p. 127–138. ACM Press, 1995. doi: 10.1145/223784.223807. URLhttp: //dx.doi.org/10.1145/223784.223807. Kim, S. and Yang, S. Accuracy improvement in financial sanction screening: is natural language processing the solution? Frontiers in Artificial Intelligence, 7, Novem- ber 2024. ISSN 2624-8212. doi: 10.3389/frai.2024. 1374323. URLhttp://dx.doi.org/10.3389/ frai.2024.1374323. Konda, P., Das, S., Suganthan G. C., P., Doan, A., Ardalan, A., Ballard, J. R., Li, H., Panahi, F., Zhang, H., Naughton, J., Prasad, S., Krishnan, G., Deep, R., and Raghavendra, V. Magellan: toward building entity matching manage- ment systems. Proc. VLDB Endow., 9(12):1197–1208, August 2016. ISSN 2150-8097. doi: 10.14778/2994509. 2994535.URLhttps://doi.org/10.14778/ 2994509.2994535. K ̈ opcke, H., Thor, A., and Rahm, E. Evaluation of en- tity resolution approaches on real-world match prob- lems. Proceedings of the VLDB Endowment, 3(1–2): 484–493, September 2010a. ISSN 2150-8097. doi: 10.14778/1920841.1920904. URLhttp://dx.doi. org/10.14778/1920841.1920904. K ̈ opcke, H., Thor, A., and Rahm, E. Learning-based ap- proaches for matching web data entities. IEEE Inter- net Computing, 14(4):23–31, July 2010b. ISSN 1089- 7801. doi: 10.1109/mic.2010.58. URLhttp://dx. doi.org/10.1109/MIC.2010.58. Li, H., Feng, L., Li, S., Hao, F., Zhang, C. J., and Song, Y. On leveraging large language models for enhancing entity resolution: A cost-efficient approach, 2024a. URL https://arxiv.org/abs/2401.03426. Li, H., Li, S., Hao, F., Zhang, C. J., Song, Y., and Chen, L.Booster: Leveraging large language models for enhancing entity resolution. In Companion Proceed- ings of the ACM Web Conference 2024, W ’24, 9 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs p. 1043–1046, New York, NY, USA, 2024b. Associ- ation for Computing Machinery. ISBN 9798400701726. doi: 10.1145/3589335.3651245. URLhttps://doi. org/10.1145/3589335.3651245. Li, Y., Li, J., Suhara, Y., Doan, A., and Tan, W.-C. Deep entity matching with pre-trained language models. Pro- ceedings of the VLDB Endowment, 14(1):50–60, Septem- ber 2020. ISSN 2150-8097. doi: 10.14778/3421424. 3421431. URLhttp://dx.doi.org/10.14778/ 3421424.3421431. Montenarh, J. and Marsden, S. Unmasking the oligarchs – using open source data to detect sanctions violations. Journal of Economic Criminology, 3:100055, March 2024. ISSN 2949-7914. doi: 10.1016/j.jeconc.2024. 100055. URLhttp://dx.doi.org/10.1016/j. jeconc.2024.100055. Mudgal, S., Li, H., Rekatsinas, T., Doan, A., Park, Y., Krishnan, G., Deep, R., Arcaute, E., and Raghaven- dra, V. Deep learning for entity matching: A design space exploration. In Proceedings of the 2018 Interna- tional Conference on Management of Data, SIGMOD ’18, p. 19–34, New York, NY, USA, 2018a. Associa- tion for Computing Machinery. ISBN 9781450347037. doi: 10.1145/3183713.3196926. URLhttps://doi. org/10.1145/3183713.3196926. Mudgal, S., Li, H., Rekatsinas, T., Doan, A., and Raghavan, P.Datasets for the deepmatcher pa- per.https://github.com/anhaidgroup/ deepmatcher/blob/master/Datasets.md, 2018b. Used in SIGMOD 2018 paper “Deep Learning for Entity Matching”; accessed 2026-01-28. Narayan, A., Chami, I., Orr, L., Arora, S., and R ́ e, C. Can foundation models wrangle your data?, 2022. URL https://arxiv.org/abs/2205.09911. OpenSanctions.Deduplicationworkflow, November112021a.URLhttps:// w.opensanctions.org/articles/ 2021-11-11-deduplication/.Accessed: 2026-01-13. OpenSanctions.Howwededuplicatecompa- niesandpeopleacrossdatasources,2021b. URLhttps://w.opensanctions.org/ articles/2021-11-11-deduplication/. Accessed: 2025-01-15. OpenSanctions. Nomenklatura: Framework and command- line tools for integrating followthemoney data streams from multiple sources.https://github.com/ opensanctions/nomenklatura/ , 2025.Ac- cessed: 2026-01-20. OpenSanctions.Matcher training data, 2025.URL https://w.opensanctions.org/docs/ opensource/pairs/. Accessed: 2025-01-15. Ornstein, J. T. Probabilistic record linkage using pretrained text embeddings. Political Analysis, p. 1–12, 2025. doi: 10.1017/pan.2025.10016. O’Hare,K.,Jurek-Loughrey,A.,andCampos, C. d.A Review of Unsupervised and Semi- supervised Blocking Methods for Record Linkage, p. 79–105.Springer International Publishing, November 2018.ISBN 9783030018726.doi: 10.1007/978-3-030-01872-64.URLhttp://dx. doi.org/10.1007/978-3-030-01872-6_4. Papadakis, G., Skoutas, D., Thanos, E., and Palpanas, T. A survey of blocking and filtering techniques for entity resolution, 2020. URLhttps://arxiv.org/abs/ 1905.06167. Peeters, R. Entity Matching using Deep Neural Networks: From Discriminative Pre-trained Language Models to Generative Large Language Models. PhD thesis, Univer- sity of Mannheim, 2025. URLhttps://madoc.bib. uni-mannheim.de/69425/. Peeters, R. and Bizer, C. Using chatgpt for entity match- ing, 2023. URLhttps://arxiv.org/abs/2305. 03423. Peeters, R., Der, R. C., and Bizer, C. Wdc products: A multi- dimensional entity matching benchmark, 2023. URL https://arxiv.org/abs/2301.09521. Peeters, R., Steiner, A., and Bizer, C. Entity matching using large language models, 2024. URLhttps://arxiv. org/abs/2310.11244. Rahm, E. and Leipzig, D. G.Benchmark datasets for entity resolution, 2019.URLhttps://dbs. uni-leipzig.de/research/projects/ benchmark-datasets-for-entity-resolution . Dataset collection released 2010–2019; accessed 2026- 01-28. Sadinle, M. Bayesian estimation of bipartite matchings for record linkage, 2016. URLhttps://arxiv.org/ abs/1601.06630. Steiner, A., Peeters, R., and Bizer, C. Fine-tuning large language models for entity matching, 2025. URLhttps: //arxiv.org/abs/2409.08185. Steorts, R. C., Hall, R., and Fienberg, S. E.A bayesian approach to graphical record linkage and de- duplication, 2015.URLhttps://arxiv.org/ abs/1312.4645. 10 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs Sterling,T. and Meijer,B. H.Dutch bank ingfined$900millionforfailingtospot money laundering,September 4 2018.URL https://w.reuters.com/article/ us-ing-groep-settlement-money-laundering-idUSKCN1LK0PE/. Accessed: 2026-01-13. Tang, J., Zuo, Y., Cao, L., and Madden, S. Generic entity resolution models. In NeurIPS 2022 First Table Represen- tation Workshop, 2022. URLhttps://openreview. net/forum?id=tRkVo1jMas. Wang, T., Chen, X., Lin, H., Chen, X., Han, X., Sun, L., Wang, H., and Zeng, Z. Match, compare, or select? an investigation of large language models for entity match- ing. In Rambow, O., Wanner, L., Apidianaki, M., Al- Khalifa, H., Eugenio, B. D., and Schockaert, S. (eds.), Proceedings of the 31st International Conference on Com- putational Linguistics, p. 96–109, Abu Dhabi, UAE, January 2025. Association for Computational Linguis- tics. URLhttps://aclanthology.org/2025. coling-main.8/. Zhang, Z., Groth, P., Calixto, I., and Schelter, S. ANY- MATCH – efficient zero-shot entity matching with a small language model. In Workshop on Preparing Good Data for Generative AI: Challenges and Approaches, 2025. URLhttps://openreview.net/forum? id=Nees1OD5td. 11 OpenSanctions Pairs: Large-Scale Entity Matching with LLMs A. Prompts This appendix contains the prompts used for LLM-based entity resolution experiments. We present both our manually- designed default prompt and the DSPy MIPROv2-optimized prompt that achieved state-of-the-art performance. A.1. Optimized Prompt (DeepSeek-R1-Distill-Qwen-14B) This prompt was automatically generated by DSPy MIPROv2 optimization, achieving 98.23% F1 with DeepSeek-R1-Distill- Qwen-14B. The optimizer discovered a more concise framing that retains the conflict-detection principle. Instructions. Given two entity records, determine if they refer to the same real-world entity. Analyze for contradictions in attributes like dates, IDs, or other details. Consider common variations in names and missing data. Classify as ’positive’ if no conflicts are found, otherwise ’negative’. Provide a clear reasoning for your classification. A.2. Structured Output Schema We use JSON structured output to ensure consistent parsing across all LLM experiments: "type": "object", "properties": "classification": "type": "string", "enum": ["positive", "negative"] , "reasoning": "string" , "required": ["classification", "reasoning"] A.3. Entity Formatting Entities are serialized as pretty-printed JSON, preserving the original schema and properties structure from the dataset: "schema": "Person", "properties": "name": ["John Smith", "J. Smith"], "birthDate": ["1965-03-15"], "country": ["US"], "nationality": ["American"] 12