Paper deep dive
Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement
Rafael Macalaba, Aivin V. Solatorio, Patrick Michael Brock, Olivier Dupriez
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/16/2026, 8:03:30 AM
Summary
This paper presents a weakly supervised framework for extracting dataset mentions from unstructured documents related to forced displacement and Fragile, Conflict, and Violence (FCV) settings. The approach utilizes a lightweight model (GLiNER) trained on general research literature to generate candidate mentions, which are then refined by a large language model (GPT-4o) to validate and correct boundaries. The refined annotations, supplemented by synthetic data, are used to fine-tune the lightweight model. The resulting model achieves 74.1% precision and 70.5% recall on a gold-standard benchmark of 1,706 passages.
Entities (10)
Relation Signals (6)
Weakly Supervised Framework â produces â Dataset Mentions
confidence 95% ¡ Extracting Dataset Mentions in Forced Displacement and FCV Documents
Weakly Supervised Framework â targets â FCV Documents
confidence 95% ¡ We present a weakly supervised framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents
GLiNER â usedby â Weakly Supervised Framework
confidence 95% ¡ A lightweight model trained on general research literature generates candidate dataset mentions... GLiNER... a lightweight local model previously trained for dataset- mention extraction
GPT-4o â usedby â Weakly Supervised Framework
confidence 95% ¡ a frontier large language model (LLM) reviews in context... We therefore use GPT-4o as a context-aware annotation judge.
World Bank â produces â Data Resources
confidence 90% ¡ organizations such as the World Bank... produce and support surveys, administrative registries, and other data resources
UNHCR â produces â Data Resources
confidence 90% ¡ organizations such as... the United Nations High Commissioner for Refugees (UNHCR)... produce and support surveys
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Development and humanitarian organizations produce and support surveys, administrative registries, and other data resources to inform research, policy, and operations, yet systematically identifying where these datasets are referenced remains difficult. Such references are dispersed across research papers, project documents, humanitarian reports, and other unstructured text, limiting both the ability to trace data use and to identify potential gaps in data availability or dissemination. We present a weakly supervised framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents without first constructing a large manually labeled training corpus. A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents, which a frontier large language model (LLM) reviews in context, validating or rejecting candidates and correcting their extraction boundaries. The resulting annotations are supplemented with targeted synthetic and contrastive examples and used to fine-tune the lightweight model for large-scale extraction. We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. Across the full benchmark, the model achieves 74.1\% precision and 70.5\% recall at the mention level; among passages containing dataset references, precision reaches 89.5\%. At the passage level, the model achieves 88.2\% accuracy and 88.6\% specificity in distinguishing passages with dataset references from those without them. These results demonstrate a practical approach for constructing domain-specific supervision when labeled data are limited, and provide a technical foundation for larger-scale analysis of data use and potential gaps in the displacement data landscape.
Tags
Links
- Source: https://arxiv.org/abs/2609.12107v1
- Canonical: https://arxiv.org/abs/2609.12107v1
Trouble viewing inline? Open PDF directly â
Full Text
76,773 characters extracted from source content.
Expand or collapse full text
Working Paper EXTRACTING DATASET MENTIONS IN FORCED DIS- PLACEMENT AND FCV DOCUMENTS: A WEAKLY SU- PERVISED FRAMEWORK WITH LLM-BASED LABEL REFINEMENT Rafael Macalaba â , Aivin V. Solatorio â , & Olivier Dupriez Development Data Group Office of the World Bank Group Chief Statistician The World Bank 1818 H Street N.W., Washington, 20433 District of Columbia, USA rmacalaba, asolatorio, odupriez@worldbank.org Patrick Michael Brock World Bank UNHCR Joint Data Center UN City Marmorvej 51, 2100 Copenhagen, Denmark brock@unhcr.org ABSTRACT Development and humanitarian organizations produce and support surveys, ad- ministrative registries, and other data resources to inform research, policy, and operations, yet systematically identifying where these datasets are referenced re- mains difficult. Such references are dispersed across research papers, project doc- uments, humanitarian reports, and other unstructured text, limiting both the ability to trace data use and to identify potential gaps in data availability or dissemi- nation. We present a weakly supervised framework for adapting dataset extrac- tion to forced displacement and Fragile, Conflict, and Violence (FCV) documents without first constructing a large manually labeled training corpus. A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents, which a frontier large language model (LLM) reviews in context, validating or rejecting candidates and correcting their extrac- tion boundaries. The resulting annotations are supplemented with targeted syn- thetic and contrastive examples and used to fine-tune the lightweight model for large-scale extraction. We evaluate the resulting model on an independent gold- standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. Across the full benchmark, the model achieves 74.1% pre- cision and 70.5% recall at the mention level; among passages containing dataset references, precision reaches 89.5%. At the passage level, the model achieves 88.2% accuracy and 88.6% specificity in distinguishing passages with dataset ref- erences from those without them. These results demonstrate a practical approach for constructing domain-specific supervision when labeled data are limited, and provide a technical foundation for larger-scale analysis of data use and potential gaps in the displacement data landscape. * GitHub/HF: @avsolatorio, avsolatorio@gmail.com â GitHub: @rafmacalaba, rafael.macalaba@yahoo.com 1 arXiv:2609.12107v1 [cs.CL] 10 Sep 2026 Working Paper 1INTRODUCTION Empirical research, policy development, and operational decision-making in Fragile, Conflict, and Violence (FCV) and forced-displacement settings can draw on household surveys, administrative registries, socioeconomic assessments, and other specialized data resources. However, systemat- ically tracking where these resources are referenced remains difficult. Dataset citation practices are inconsistent, and references are not always captured through standardized citations or persistent identifiers (Buneman et al., 2021; Piwowar & Vision, 2013; Silvello, 2018). This creates a particular opportunity for development and humanitarian organizations that produce, finance, and disseminate data in data-scarce environments. Datasets produced or supported by orga- nizations such as the World Bank, the United Nations High Commissioner for Refugees (UNHCR), and the Joint Data Center on Forced Displacement (JDC) may appear across research papers, policy documents, project documents, humanitarian reports, footnotes, and other unstructured text rather than through consistent bibliographic citations (Mooney & Newton, 2012). Identifying these refer- ences at scale can serve two complementary purposes. First, it can reveal where existing data re- sources are being taken up in research, policy, and operations, providing evidence about their reach and contribution to evidence-informed analysis and decision-making. Second, when combined with information on what data exist, where they are available, and which populations and topics they cover, patterns of use and non-use can help identify potential gaps in data production, accessibility, or dissemination. Such evidence can therefore support not only retrospective monitoring of data in- vestments, but also prospective decisions about where additional investment in data production and dissemination may be most valuable. Automated information extraction offers a practical solution. Recent models have improved the abil- ity to identify entities and structured information from unstructured text (NuMind, 2024; Hussain et al., 2023), while generalist named-entity recognition models such as GLiNER (Zaratiana et al., 2023) provide a flexible foundation for identifying specialized entity types. The main challenge in applying these methods to FCV and forced-displacement documents is the limited availability of domain-specific labeled data. Models trained on research literature may not transfer directly to spe- cialized documents, while manually constructing a large training corpus would require substantial expert annotation. We address this problem by using an existing dataset-extraction model as a source of weak super- vision. A lightweight model previously trained to identify dataset mentions in general research literature (Macalaba & Solatorio, 2026) is first used to generate candidate mentions from unlabeled FCV and forced-displacement documents. A frontier large language model (LLM) then reviews these candidates in context, validates or rejects them, and corrects their extraction boundaries. The resulting annotations are supplemented with targeted synthetic and contrastive examples and used to fine-tune the lightweight model for the target domain. This approach uses the LLM to construct higher-quality training data while retaining a lightweight model for large-scale inference. We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. The evaluation considers both whether the model correctly extracts dataset mentions and whether it correctly distinguishes passages con- taining dataset references from those containing none. The paper makes three contributions. First, we introduce a weakly supervised framework that com- bines an existing extraction model, LLM-assisted annotation refinement, and targeted synthetic data to construct domain-specific training data when labeled examples are scarce. Second, we show how this supervision can be transferred to a lightweight local model for large-scale dataset extraction. Third, we evaluate the resulting model on an independent, heterogeneous gold-standard benchmark covering multiple types of FCV and forced-displacement documents. Reliable extraction of dataset references is one technical building block for constructing a broader picture of the displacement data landscape. A textual reference does not by itself establish how a dataset was used, whether it influenced a decision or outcome, or whether the absence of a reference indicates a genuine data gap. The present study therefore evaluates the extraction capability rather than attempting to estimate the full landscape of displacement data use. Its benchmark is designed to test transfer across heterogeneous research, humanitarian, and operational document types, not to constitute a representative corpus of all policy, research, and operational materials relevant to 2 Working Paper forced displacement. Applying the framework to that broader substantive objective would addition- ally require systematic corpus construction, displacement-relevance filtering, dataset resolution and linkage, and contextual classification of how identified data are used. We return to these extensions in future work. 2RELATED WORK 2.1DATASET CITATION AND DISCOVERABILITY Inconsistent data citation practices are a long-standing barrier to tracking dataset use. Mooney & Newton (2012) document the absence of a stable citation convention for datasets analogous to that for scholarly publications, and Silvello (2018) formalizes the requirements a data citation system must satisfy to support discovery, reuse, and credit attribution. Buneman et al. (2021) extend this to the citation graph itself, showing how linking data citations to publication citations can improve dataset discoverability at scale. Piwowar & Vision (2013) provide empirical evidence that datasets with persistent identifiers are reused and cited more often, underscoring the practical value of sys- tematically tracking references for organizations, such as the World Bank and UNHCR, that invest heavily in data production. Our work complements this literature: rather than proposing a citation standard, we extract dataset references as they actually appear in unstructured text, which remains the dominant mode of reference in development and humanitarian documents today. The relationship between citation practice and observed data use also motivates a potential down- stream application of the extraction framework. More standardized and persistent references may make datasets easier both for users to discover and for organizations to trace across downstream doc- uments. The present study does not test whether improved citation practice causes greater data use. However, applying the extractor at scale could enable future empirical analysis of whether datasets with stronger documentation, persistent identifiers, standardized citation guidance, or greater ac- cessibility are subsequently referenced across a wider range of research, policy, and operational documents. Such analysis could provide an additional evidence base for investments in data docu- mentation and dissemination. 2.2AUTOMATED DATASET MENTION EXTRACTION A growing line of work applies information extraction methods directly to the dataset-mention iden- tification problem. Heddes et al. (2021) apply a transformer-based named-entity-recognition tagger to detect dataset names in scientific articles, and Hussain et al. (2023) combine transformer-based question-answering and named-entity-recognition models within a similar extraction framework applied to the same underlying task. Younes & Scherp (2023) directly compare named-entity- recognition and question-answering approaches for extracting previously unseen dataset names, finding that question-answering formulations generalize better to datasets absent from training data â motivating our own use of an LLM, rather than a fixed tagger, for candidate adjudication. Beyond individual research contributions, Potok (2022) describes âShow US the Data,â a multi-agency U.S. government pilot applying AI methods to improve the discoverability of public datasets referenced in the literature, illustrating that automated dataset-mention tracking is also an active institutional priority and not solely a research problem. Our own prior work builds a dataset-extraction model for research literature using synthetic and LLM-assisted data (Solatorio et al., 2025) and evaluates it at scale on World Bank Policy Research Working Papers (Macalaba & Solatorio, 2026; 2025). The present paper extends this line of work in a new direction: rather than improving extraction on re- search literature, we adapt an existing research-literature model to a substantially different document distribution â humanitarian, operational, and FCV documents â where dataset references are sparser, less standardized, and more often expressed alongside project titles and administrative systems. This domain-transfer setting is the primary methodological focus of the present work. 2.3WEAK SUPERVISION AND LLM-ASSISTED ANNOTATION Our candidate-generation-then-refinement design draws on the weak-supervision paradigm formal- ized by Ratner et al. (2017), in which noisy, programmatic, or model-generated labels are combined and denoised rather than manually curated from scratch. Where Snorkel-style systems typically 3 Working Paper combine multiple labeling functions probabilistically, we instead use a single upstream model to generate candidates and a frontier LLM to adjudicate them individually in context â a design in the spirit of LLM-based data programming and self-training approaches such as Self-Instruct (Wang et al., 2022), in which a strong model is used to generate and filter training signal for a down- stream task. This also relates to active learning (Settles, 2009), in that both approaches aim to concentrate limited annotation effort where it is most informative; our approach substitutes LLM adjudication for a human oracle at the candidate-review step, while still relying on a rigorously human-constructed gold-standard set for final evaluation. This paper applies this LLM-refined weak-supervision pipeline to the specific problem of transferring dataset-mention extraction into humanitarian and FCV document collections. 2.4DATA USE IN FORCED DISPLACEMENT AND FCV SETTINGS The scale and persistence of forced displacement make reliable, accessible, and policy-relevant data a strategic priority for organizations operating in these settings (UNHCR, 2025). The World Bankâ UNHCR Joint Data Center on Forced Displacement (JDC) Strategy 2024â2027 emphasizes four complementary priorities: systematic inclusion of forcibly displaced and stateless populations in national statistics; targeted production of high-quality data and timely analysis to inform policy and programs; data innovation to improve the quality, timeliness, and accessibility of data; and opera- tionalizing data and evidence to strengthen solutions to forced displacement (World BankâUNHCR Joint Data Center on Forced Displacement, 2024). The World Bank Groupâs FCV Strategy 2026â 2030 similarly emphasizes earlier anticipation of FCV risks, differentiated engagement across FCV settings, a stronger One World Bank Group approach centered on jobs, and strengthened operational tools and partnerships (World Bank Group, 2026). UNHCRâs data transformation strategy (UN- HCR, 2020) further emphasizes the role of data systems, technology, and responsible data use in humanitarian settings. These strategic priorities complement the practical and governance challenges of producing and using data in fragile and displacement-affected settings. Hoogeveen & Pape (2020) document the practical work involved in producing reliable survey data in fragile settings, while the Inter-Agency Standing Committeeâs operational guidance (Inter-Agency Standing Committee, 2023) sets out gov- ernance principles for humanitarian data. Beyond data production itself, however, organizations that produce, fund, and disseminate these resources also stand to benefit from systematic visibility into where and how the resulting data are subsequently referenced. Such visibility has potential value beyond accountability or retrospective monitoring. Evidence on where displacement-relevant data are referenced can help characterize the reach of existing data resources across research, policy, and operational processes. When combined with information on what data are available and the populations and topics they cover, patterns of observed use and non-use may also help identify potential gaps in data production, accessibility, documentation, or dissemination. This paper develops one technical component needed for such analysis: scalable extraction of dataset references from heterogeneous FCV and forced-displacement documents. 3METHODOLOGY Our goal is to adapt dataset extraction to FCV and forced-displacement documents when domain- specific labeled training data are scarce. Rather than first constructing a large manually annotated training corpus, we use an existing dataset-extraction model to generate weak annotations from unlabeled documents and a large language model (LLM) to refine them. We supplement these annotations with targeted synthetic examples and use the combined data to fine-tune a lightweight extraction model for the target domain. The framework consists of four stages. First, a lightweight model previously trained to extract dataset mentions from research literature identifies candidate mentions in unlabeled FCV and forced- displacement documents. Second, a frontier LLM reviews these candidates in their original context, removes invalid predictions, and corrects extraction boundaries. Third, targeted synthetic and con- trastive examples expand the training data beyond the cases observed in the available documents. Finally, the annotations from real documents and the synthetic examples are combined to fine-tune the lightweight model for large-scale extraction. 4 Working Paper This design separates training-data construction from inference. The existing extractor provides an efficient source of weak supervision, while the LLM is used selectively to improve annotation quality. Synthetic generation expands coverage of difficult or underrepresented cases. The resulting supervision is then transferred to a lightweight local model that can be applied independently at scale, without requiring LLM verification at inference time. 3.1WEAKLY SUPERVISED CANDIDATE GENERATION The first stage identifies possible dataset references in previously unlabeled FCV and forced- displacement documents. Rather than manually reviewing every passage, we use an existing ex- traction model to locate text spans that may refer to datasets. These predictions serve as weak annotations: they reduce the annotation search space but are not assumed to be correct. PDF documents are processed page by page using a standardized text-extraction util- ity.The extracted text is divided into page-level chunks and passed to GLiNER2 (ai4data/datause-extraction), a lightweight local model previously trained for dataset- mention extraction using general research publications and World Bank Policy Research Working Papers (Zaratiana et al., 2023; Macalaba & Solatorio, 2026). The model therefore provides an ini- tial dataset-extraction capability but has not yet been adapted to the FCV and forced-displacement documents considered here. At this stage, the objective is to maximize candidate coverage rather than precision. We therefore apply a permissive prediction threshold of Ď = 0.15, retaining uncertain predictions that would nor- mally be discarded at a higher inference threshold. This produces an intentionally broad candidate set, which the LLM-assisted refinement stage then narrows and corrects. For each candidate, the model returns the predicted text span, character offsets, and initial classi- fication. These predictions are retained together with their surrounding source text and passed to the LLM-assisted refinement stage. The local model therefore acts as a screening mechanism rather than a final annotator: it identifies where potential dataset references occur, while the subsequent stage determines whether each candidate should be retained, corrected, or discarded. 3.2LLM-ASSISTED ANNOTATION REFINEMENT The weak annotations generated in the first stage are useful for locating potential dataset references, but benefit from a second, context-aware pass before serving as domain-specific training data. We therefore use GPT-4o as a context-aware annotation judge. Each candidate is evaluated together with its surrounding source text according to a predefined annotation protocol. The refinement stage performs two functions that are directly relevant to the extraction task: 1. Candidate validation: determine whether the predicted span refers to a structured data resource, such as a survey, database, administrative registry, census, microdata collection, or other identifiable dataset; and 2. Boundary correction: adjust the predicted span when the initial extraction contains too much or too little text. Candidates referring instead to projects, organizations, analytical methods, narrative publications, qualitative methods, document headings, or administrative tools that do not themselves constitute data resources are excluded. For example, consider the following passage from the World Bankâs Jordan Emergency Food Secu- rity Project Project Appraisal Document: âExisting food insecurity levels are particularly high among Jordanâs refugee population. According to the most recent mobile Vulnerability Assessment and Mapping (mVAM) com- pleted by the World Food Program (WFP) in Jordan, 7 percent of Jordanian households were found to be food insecure as of February 2021...â The first-stage model identifies mobile Vulnerability Assessment and Mapping (mVAM) as a candidate dataset reference. The LLM retains the candidate because the surround- ing text indicates that mVAM refers to a structured data resource used to measure food insecurity. 5 Working Paper In contrast, consider the following passage from the UNHCR Mindanao Forced Displacement An- nual Report: âThe Annual Mindanao Displacement Dashboard aims to provide a starting point for in- formation and analysis that can help protection agencies, policy makers and other stake- holders concerning instances of forced displacement...â At the permissive candidate-generation threshold, the first-stage model identifies Annual Mindanao Displacement Dashboard as a possible dataset reference. The LLM rejects the candidate because the surrounding text indicates that it is an information product rather than an underlying structured dataset. The LLM response is constrained through OpenAIâs structured output interface using a Pydantic schema. This ensures that validation decisions, corrected spans, and associated annotations are returned in a consistent machine-readable format. The complete response schema, system prompt, and inclusion and exclusion rules are provided in Appendix A.1. This stage converts an initial, broad set of model predictions into a cleaner set of domain-specific annotations without requiring the LLM to search every passage in the document collection from scratch. Because the refinement stage operates on candidates surfaced in the first stage, the overall pipelineâs recall depends on the coverage of the initial candidate generator; we discuss this design choice further in Section 6. 3.3ANNOTATION SCHEMA Dataset references vary substantially in how precisely they identify the underlying data resource. To provide a consistent annotation target, validated mentions are classified into three levels of speci- ficity: ⢠named: a formally named survey, database, registry, or other data resource with a rec- ognizable title or acronym, such as the Demographic and Health Survey, LSMS, or World Development Indicators; ⢠descriptive: a specific description of a data resource that does not use a unique formal name, such as survey of Javanese farm households; and ⢠vague: a general reference to a data resource without enough information to identify a particular dataset, such as administrative data or utility records. The annotation schema also supports additional metadata fields, including the dataset acronym, publisher, geography, reference population, and publication year when these are available in the surrounding text. These fields provide structured information about the identity and scope of a referenced data resource, but they are not part of the extraction evaluation reported in this paper. The full annotation schema and validation rules are provided in Appendix A.1. 3.4TARGETED SYNTHETIC DATA GENERATION The LLM-refined annotations provide domain-specific examples drawn from real documents, but their coverage is naturally bounded by both the available document collection and the candidates surfaced by the initial model. We therefore supplement these annotations with synthetic examples. The purpose of synthetic generation is not to reproduce the distribution of the underlying corpus, but to broaden the training data with additional dataset types, linguistic forms, and underrepresented cases. We use few-shot prompting with GPT-4o-mini to generate passages containing dataset refer- ences. Each prompt provides annotated examples that demonstrate the expected extraction format and is guided by selected dataset names, dataset types, and contextual patterns for which additional examples are desired. These include FCV-relevant resources such as the Socio-Economic Insights Survey (SEIS), Multi-Sector Needs Assessment (MSNA), and WFPâs mobile Vulnerability Assessment and Mapping (mVAM), as well as generic expressions such as rainfall data. 6 Working Paper We use more than 100 prompt templates spanning Development Economics, Humanitarian and Pro- tection, Global Health, Food Security and Agriculture, Education, and Climate and Environment. Varying both the dataset seeds and the contexts in which they appear increases the diversity of the generated examples while retaining control over their annotations. Generated records are constrained using a Pydantic schema and subjected to automated validation before inclusion in the training corpus. The generated text must contain the specified dataset and metadata attributes verbatim; dataset spans cannot include standalone publication years; and spans cannot terminate in incomplete grammatical fragments such as trailing prepositions or verbs. Ex- amples that fail these checks are discarded. Additional generation details and the domain-specific variable banks are reported in Appendix A.2. 3.4.1CONTRASTIVE EXAMPLES Synthetic generation is also used to create contrastive examples. The purpose of these examples is to expose the model to cases in which similar expressions should receive different extraction labels depending on their context. For example, LSMS should be extracted when it refers to the Living Standards Measurement Survey used as a data resource, but not when LSMS team refers to the people conducting fieldwork. Simi- larly, household consumption survey may identify a data resource, whereas we conducted household surveys describes a data-collection activity rather than the resulting dataset. Table 1 illustrates these cases. Table 1: Illustrative contrastive synthetic examples. Context TypeExample SentenceTarget SpanLabel Dataset mentionThe analysis incorporates the Living Stan- dards Measurement Survey (LSMS) to calculate consumption aggregates. LivingStandards Measurement Survey named Non-dataset useThe LSMS team coordinated the field oper- ations and trained the local enumerators. NoneEmpty Dataset mentionWe gathered the household consump- tion survey data across three displacement camps. household consump- tion survey descriptive Non-dataset useWe conducted household surveys to assess local needs and immediate protection risks. NoneEmpty These examples encourage the model to use surrounding context rather than relying only on familiar dataset names, acronyms, or data-related expressions. 3.5FINE-TUNING AND FINAL EXTRACTION The final training corpus combines two sources of supervision: annotations derived from real FCV and forced-displacement documents through weakly supervised candidate generation and LLM- assisted refinement, and targeted synthetic examples generated through few-shot prompting. The combined corpus is used to fine-tune GLiNER2 for dataset extraction in the target domain. This final step transfers the supervision produced during training-data construction into the lightweight extraction model. The frontier LLM is therefore not part of the deployed extraction pipeline: once fine-tuned, the local model can process new document collections independently. For final extraction, the fine-tuned model is applied at a confidence threshold of Ď = 0.40. Its per- formance is evaluated against a separate gold-standard holdout set of 1,706 text chunks. The holdout data are not used to construct the weak annotations, refine candidates, generate synthetic examples, or fine-tune the model. Section 4 describes the evaluation corpus and benchmark construction in detail. 7 Working Paper 4DATA AND EVALUATION DESIGN The document collections in this study constitute a technical validation corpus, rather than a repre- sentative corpus from which to characterize the overall landscape of displacement data use. Docu- ments were selected to expose the extraction framework to heterogeneous settings in which dataset references differ in frequency, form, and surrounding terminology. The resulting evaluation there- fore addresses whether the extraction method transfers across document types; it does not estimate the prevalence or distribution of data use across research, policy, and operations. A full application to mapping the displacement data landscape would require a separately designed corpus with explicit coverage of the relevant evidence ecosystemâfor example, academic research, government strategies and policy documents, humanitarian assessments, and development opera- tionsâas well as a mechanism for identifying displacement-relevant passages within documents whose overall subject may be broader. These corpus-construction and relevance-classification tasks are outside the scope of the present technical evaluation. 4.1DOCUMENT COLLECTIONS The corpus comprises 93 documents purposively assembled from four document collections cho- sen to provide variation in document format, substantive context, and the explicitness with which datasets are referenced. The collections span settings ranging from formal research publications, where datasets are often identified explicitly, to humanitarian and operational documents, where ref- erences may be more implicit or embedded in narrative, administrative, and monitoring language. The resulting composition is intended to expose the extraction framework to heterogeneous forms of dataset mention rather than to constitute a representative sample of the broader forced-displacement and FCV evidence ecosystem. ⢠UNHCR/ReliefWeb reports (43 documents): Humanitarian reports and field-level up- dates covering displacement, protection needs, and conditions affecting displaced popula- tions. This collection provides a relatively heterogeneous reporting environment in which references to surveys, assessments, registries, and other data resources may be sparse, im- plicit, or expressed without formal citation conventions. A larger number of documents was therefore included to obtain sufficient coverage of these less standardized forms of dataset reference. ⢠SEIS and humanitarian briefs (25 documents): Socio-Economic Insights Surveys (SEIS) and related analytical briefs focused on the socioeconomic conditions of displaced populations. These documents provide an analytically oriented setting in which datasets and empirical findings are more central to the text than in field-level humanitarian report- ing, while remaining less standardized than formal research publications. The collection also includes documents identified by subject-matter experts as thematically relevant to forced displacement. ⢠World Bank Policy Research Working Papers (14 documents): Research papers in which datasets are typically identified explicitly as part of empirical analysis. This col- lection serves as the most citation-explicit document type in the benchmark and is also closest to the source domain of the initial extraction model, which was trained primarily on research-oriented documents of this kind. It therefore provides a useful comparison for assessing how performance changes as the model is applied to less structured and less citation-standardized domains. ⢠World Bank Project Appraisal Documents (11 documents): Operational documents containing narrative text, indicators, tables, project and administrative terminology, and monitoring frameworks. These documents represent a high-ambiguity setting in which project titles, administrative systems, reports, monitoring platforms, and other information products may resemble dataset references. Their inclusion is particularly important for evaluating whether the model can distinguish genuine data sources from dataset-adjacent entities in operational text. More broadly, project documents are relevant to the down- stream objective of examining where and how data resources are referenced in the design and implementation of development operations. 8 Working Paper Candidates were chunked, pre-extracted, LLM-judged, then human-adjudicated with agreement- based filtering (conflicts reviewed manually), and splits are document-disjoint to prevent leakage. 4.2GOLD-STANDARD EVALUATION SET The final model is evaluated against a manually annotated holdout benchmark of 1,706 text chunks. Of these, 545 contain at least one annotated dataset reference and 1,161 contain no dataset reference. The large share of negative passages reflects an important feature of large-scale extraction: most document passages do not contain dataset references, so the model must learn both when to extract and when to return nothing. Table 2 summarizes the benchmark. Table 2: Composition of the gold-standard holdout benchmark. Document CollectionTotal ChunksWith DatasetWithout Dataset World Bank PADs1297122 World Bank PRWPs25967192 UNHCR / ReliefWeb16922147 SEIS / Baseline Holdout1,149449700 Total1,7065451,161 Dataset prevalence varies substantially across the benchmark. Approximately 39% of SEIS/baseline passages contain a dataset reference, compared with about 5% of PAD passages, allowing us to evaluate the model under both data-rich and data-sparse conditions. 4.3EVALUATION METRICS We evaluate the final model at a prediction confidence threshold of Ď = 0.40. Because the task in- volves identifying both the presence and the exact textual location of dataset references, we evaluate performance at two levels: dataset-mention extraction and passage-level classification. 4.3.1DATASET-MENTION EXTRACTION At the mention level, a prediction is considered correct when it sufficiently overlaps with an anno- tated dataset reference. We measure overlap using the Jaccard similarity between the sets of tokens in the predicted span, S 1 , and the annotated span, S 2 : J(S 1 ,S 2 ) = |W 1 ⊠W 2 | |W 1 | +|W 2 |â|W 1 ⊠W 2 | ,(1) where W 1 and W 2 are the sets of tokens contained in the predicted and annotated spans, respectively. We consider a prediction to match the annotation when J(S 1 ,S 2 )⼠0.5. We report precision and recall. Precision measures the proportion of predicted dataset mentions that correspond to valid annotated references, while recall measures the proportion of annotated dataset mentions recovered by the model. We also report the F 0.5 -score, which gives greater weight to precision than recall. For downstream landscape analysis, false-positive references could distort es- timates of both observed data use and apparent gaps in use, making precision particularly important. 4.3.2PERFORMANCE ON PASSAGES WITH AND WITHOUT DATASET MENTIONS We report mention-level performance under two complementary settings. First, we evaluate the model across the complete holdout benchmark, including passages both with and without dataset references. This represents the modelâs expected behavior when processing complete document collections. Second, we report extraction performance on passages that contain at least one annotated dataset ref- erence. This isolates the modelâs ability to identify datasets when they are actually present. Compar- ing the two settings helps distinguish errors in dataset identification from false predictions generated on passages that contain no datasets. 9 Working Paper 4.3.3PASSAGE-LEVEL CLASSIFICATION Finally, we evaluate whether the model can correctly determine when a passage contains any dataset reference. For this analysis, each text chunk is treated as a binary classification problem. A passage is predicted as positive if the model extracts at least one dataset mention and negative if it extracts none. We report accuracy, recall, F 1 -score, and specificity. Specificity, or the true negative rate, measures the proportion of passages without dataset references for which the model correctly returns no ex- traction. This metric is particularly important for operational documents, where dataset mentions are sparse and most text consists of administrative or project-related content. Together, the mention-level and passage-level evaluations capture two requirements of the intended system: accurately identifying dataset references when they occur and avoiding spurious extractions when they do not. 5RESULTS We evaluate the model produced by the semi-supervised framework on two complementary capabil- ities. First, we assess whether it can correctly extract dataset mentions and their textual boundaries. Second, we assess whether it can distinguish passages that contain dataset references from those that do not. Together, these evaluations capture the two capabilities required for large-scale dataset detection: finding dataset references when they occur and avoiding spurious extractions when they do not. 5.1DATASET MENTION EXTRACTION Table 3 reports mention-level extraction performance on the 1,706-chunk gold-standard holdout benchmark at a confidence threshold of Ď = 0.40. Across the full benchmark, the model correctly identifies 620 dataset mentions, producing an overall precision of 74.1% and recall of 70.5%. When the evaluation is restricted to passages containing at least one annotated dataset reference, precision increases to 89.5%, while recall remains 70.5%. This result indicates that, once operating within passages where datasets are present, nearly nine out of ten extracted mentions correspond to an annotated dataset reference. Table 3: Dataset mention extraction performance at a confidence threshold of Ď = 0.40. Evaluation CorpusChunks (Pos/Neg) Evaluation SetTPFPFNPrecisionRecall World Bank PADs129 (7 / 122)All Chunks511331.2%62.5% Positive Only503100.0%62.5% World Bank PRWPs259 (67 / 192)All Chunks68253773.1%64.8% Positive Only6893788.3%64.8% UNHCR / ReliefWeb169 (22 / 147)All Chunks23141362.2%63.9% Positive Only2331388.5%63.9% SEIS / Baseline Holdout1,149 (449 / 700) All Chunks52416720775.8%71.7% Positive Only5246120789.6%71.7% Overall1,706 (545 / 1,161) All Chunks62021726074.1%70.5% Positive Only6207326089.5%70.5% The positive-only results are notably consistent across document types. Precision reaches 88.3% for World Bank Policy Research Working Papers, 88.5% for UNHCR/ReliefWeb documents, and 89.6% for the SEIS/baseline collection. The PAD subset yields 100% precision on positive pas- sages, reflecting strong performance on the small number of annotated mentions available for that collection. Recall ranges from 62.5% to 71.7% across the four collections. 10 Working Paper These results demonstrate that the domain-adapted model extracts dataset references with strong precision across substantially different document types. The gap between the all-chunk and positive- only results reflects the separate task of determining when no dataset reference is present at all, which we examine next as a standalone detection problem. 5.2DATASET DETECTION We therefore evaluate the model separately as a passage-level detector. A passage is considered positive when the model extracts at least one dataset reference and negative when it returns no extraction. Across the full benchmark, the model correctly identifies 475 of the 545 passages containing dataset references and correctly returns no extraction for 1,029 of the 1,161 passages without dataset refer- ences. This corresponds to an overall detection accuracy of 88.16%, recall of 87.16%, and specificity of 88.63%. Table 4: Passage-level dataset detection performance. Document CollectionChunksTP / FP / FN / TNAccuracyF 1 Specificity World Bank PRWPs25957 / 15 / 10 / 17790.3%82.0%92.2% UNHCR / ReliefWeb16919 / 11 / 3 / 13691.7%73.1%92.5% World Bank PADs1294 / 11 / 3 / 11189.1%36.4%91.0% SEIS / Baseline Holdout1,149395 / 95 / 54 / 60587.0%84.1%86.4% Detection performance is stable across the four document collections. Accuracy ranges from 87.0% to 91.7%, while specificity ranges from 86.4% to 92.5%. This is a meaningful result given that 1,161 of the 1,706 passages in the benchmark contain no dataset reference at all: the model performs well not only in finding passages that contain datasets, but also in filtering out the much larger set of passages where no extraction should be made. The PAD results merit a different interpretation because of the strong class imbalance in this subset. Only 7 of 129 PAD passages contain an annotated dataset reference. The model correctly identifies four of these passages and correctly rejects 111 of the 122 negative passages, producing high accu- racy (89.1%) and specificity (91.0%); the F 1 -score (36.4%) is more sensitive to the small absolute number of positive examples in this subset than to the modelâs underlying behavior. This illustrates the value of reporting specificity and accuracy alongside F 1 when evaluating rare-event extraction in operational documents. 5.3PERFORMANCE ACROSS DOCUMENT TYPES The holdout benchmark deliberately spans research, humanitarian, and operational documents that differ substantially in structure, writing style, and the ways in which dataset references are expressed. This heterogeneity provides a test of whether the model has learned dataset-reference patterns that generalize across document types rather than patterns specific to a single collection. The clearest evidence of cross-document performance comes from extraction precision on pas- sages containing dataset references. Despite substantial differences across collections, precision remains close to 90% for PRWPs, UNHCR/ReliefWeb documents, and the SEIS/baseline collec- tion. Passage-level specificity is similarly stable, exceeding 86% in every collection. Together, these results suggest that the model transfers effectively across the research and humanitarian document types represented in the benchmark while retaining a strong ability to reject passages that do not contain dataset references. The diversity of the benchmark also provides some evidence that the learned extraction capability is not tied to a single document genre. However, the benchmark does not include the broader range of government strategies, national policy documents, and other policy-oriented materials that would be needed to establish transfer to policy corpora. The present results therefore support the plausibility of such transfer, but direct evaluation on these document types remains necessary. Operational documents present a particularly demanding setting because they contain many named entities and expressions that resemble dataset references without actually referring to datasets. To 11 Working Paper better characterize this challenge, we examine candidates generated from PADs during the weak- supervision stage. At the permissive candidate-generation threshold of Ď = 0.15, 145 of 203 candi- dates are rejected during LLM-assisted refinement. Project and program titles constitute the largest category of rejected candidates, accounting for 50.3%, followed by administrative software and monitoring systems at 17.2%. Other rejected candidates include publications and standardized doc- ument headings. This analysis identifies a recurring source of ambiguity in operational documents: distinguishing genuine dataset references from dataset-adjacent entities such as project titles, administrative sys- tems, publications, and standardized headings. The contrastive training examples described in Sec- tion 3.4.1 are designed specifically to expose the model to these difficult negative cases. The results therefore highlight both the importance of negative-example construction for domain adaptation and a concrete area in which further expansion of the training data may improve performance on opera- tional documents. 5.4SUMMARY OF EVALUATION RESULTS Overall, the evaluation provides evidence that the semi-supervised framework can adapt a general dataset-extraction model to specialized FCV and forced-displacement documents without requir- ing a large manually labeled training corpus. The resulting model achieves 89.5% precision when extracting mentions from passages containing datasets and identifies whether a passage contains a dataset with 88.16% accuracy across the complete holdout benchmark. These results also point toward clear next steps. Mention-level recall of approximately 70% sug- gests room to expand candidate coverage in the weak-supervision stage, and data-sparse operational documents remain the setting most likely to benefit from additional contrastive training examples that sharpen the boundary between datasets and project titles or administrative systems. Together, the results demonstrate that the framework produces a practical, deployable dataset detector from limited domain-specific supervision, with a clear path to further gains. 6DISCUSSION This study shows that a general dataset-extraction model can be adapted to specialized FCV and forced-displacement documents without first constructing a large manually labeled training corpus. By combining weakly supervised candidate generation, LLM-assisted annotation refinement, and targeted synthetic data, the framework produces a domain-adapted local model that performs con- sistently across research, humanitarian, and operational documents. The results are particularly strong when dataset references are present. Precision reaches 89.5% on positive passages and remains close to 90% across the major document collections. At the same time, evaluation on the complete benchmark shows why extraction performance should be assessed on more than just passages known to contain datasets. Most passages contain no dataset reference, making the ability to return no extraction a central part of the task. The modelâs 88.6% specificity indicates that it distinguishes dataset-bearing passages from other content well, and the framework provides a clear structure â contrastive training examples â for continuing to improve on this. 6.1ADAPTING EXTRACTION MODELS WITH LIMITED LABELED DATA A central contribution of the framework is how it constructs domain-specific supervision. Rather than treating predictions from the general-domain model as final labels, we use them as weak an- notations that identify likely dataset references. A frontier LLM then reviews these candidates in context, removes invalid predictions, corrects extraction boundaries, and applies a consistent anno- tation schema. Synthetic generation provides additional examples for difficult or underrepresented cases. The resulting annotations are then used to fine-tune a lightweight model for large-scale ex- traction. This division of labor allows each model to serve a different purpose: the general model provides broad candidate generation, the LLM provides context-sensitive refinement, and the fine-tuned local model provides scalable inference. The approach is particularly valuable when manually annotating 12 Working Paper a large domain-specific corpus would be costly, but an existing model already provides a reasonable starting point. Importantly, the framework does not eliminate the need for human judgment. Instead, it shifts expert effort away from exhaustive passage-by-passage annotation toward defining annotation rules, identifying important error cases, and constructing a rigorous gold-standard evaluation set â a more scalable use of scarce domain expertise. 6.2LEARNING WHAT NOT TO EXTRACT The evaluation also highlights a distinctive feature of this domain: learning what constitutes a dataset is only part of the task. The model must also learn to distinguish datasets from other entities that look similar. This is most visible in operational documents. Project titles, administrative systems, monitoring tools, publications, and standardized headings can share many surface characteristics with dataset names. The PAD analysis in Section 5 shows that these ânear-missâ entities account for a substantial share of filtered candidates during LLM-assisted refinement. This suggests that continued gains may come as much from expanding negative and contrastive examples as from adding more positive dataset mentions. In specialized document collections, ex- plicitly teaching the model the boundary between the target entity and plausible alternatives is a particularly effective lever. This insight directly motivated the contrastive synthetic examples used in our training procedure and is a natural focus for continued development of the framework. 6.3FROM DATASET DETECTION TO MAPPING THE DISPLACEMENT DATA LANDSCAPE The value of dataset-mention extraction extends beyond counting references. When extracted men- tions are linked to identifiable data resources and interpreted alongside information on data avail- ability and substantive information needs, patterns of observed use and non-use can contribute to a broader picture of the displacement data landscape. Repeated references can provide evidence that particular data resources are reaching downstream research, policy, or operational processes. Con- versely, limited observed use of an existing and relevant dataset may warrant investigation of barriers related to accessibility, documentation, discoverability, timeliness, or dissemination. Where impor- tant information needs are not adequately served by existing data resources, the same landscape analysis may instead point to potential gaps in data production. Such interpretations cannot be made from dataset mentions alone. The absence of an observed ref- erence does not establish that a dataset was unavailable, unused, or insufficient, particularly when the underlying document corpus is not designed to be representative. Moving from mention extrac- tion to substantive landscape analysis would therefore require additional components: systematic corpus construction spanning research, policy, humanitarian, and operational materials; classifica- tion of displacement relevance at the document or passage level; resolution and linkage of extracted mentions to identifiable datasets; contextual classification of how those datasets are used; and com- parison with inventories of available data and relevant information needs. An important distinction in this broader framework is between the relevance of a dataset and the relevance of its use. A national household survey, for example, may include displaced populations but be referenced in a policy document only for national-level findings unrelated to displacement. Conversely, the same survey may be used specifically to analyze outcomes among refugees or inter- nally displaced populations. Establishing displacement relevance therefore requires examining the context in which a dataset is referenced, rather than inferring relevance from the identity or coverage of the dataset alone. The present study establishes one technical component of this broader system: scalable identifica- tion of dataset references across heterogeneous documents. Combined with the additional compo- nents described above, this capability could contribute to systematic analysis of where displacement- relevant data are reaching research, policy, and operational processes, where existing data resources may face barriers to uptake, and where unmet information needs may motivate additional data pro- duction. 13 Working Paper 6.4FUTURE DIRECTIONS Several directions could further improve and validate the extraction framework itself. The training procedure combines LLM-refined annotations from real documents with synthetic and contrastive examples. The consistent precision observed on positive passages across document types (Sec- tion 5), together with the candidate-filtering analysis in Section A.4, suggests that exposure to dif- ficult negative cases may be particularly important for distinguishing genuine dataset references from project titles, administrative systems, and other dataset-adjacent entities in operational docu- ments. Targeted ablation studies could quantify the contribution of each source of supervision. Ex- panding candidate coverage during weak supervision, for example through an additional candidate- generation model or method, could help improve mention-level recall. Measuring agreement be- tween LLM-assisted and expert annotations on a held-out sample would provide further evidence on the reliability of the refinement stage, while expanding the operational-document benchmark would strengthen evaluation in settings where dataset references are relatively sparse and ambiguous. A second direction is to evaluate how well the extraction capability transfers beyond the docu- ment types represented in the current benchmark. In particular, direct evaluation on government strategies, national policy documents, and other policy-oriented corpora would be necessary before making broader claims about performance across the displacement evidence ecosystem. Such eval- uation could also test document- and passage-level methods for identifying displacement relevance, including cases in which broader national datasets are used specifically to produce evidence about displaced populations. The principal longer-term direction is to integrate these capabilities into a broader displacement- data landscape pipeline. This would entail constructing a systematic application corpus, resolving extracted mentions to identifiable datasets, classifying the context and substantive role of each ref- erence, and linking observed references to inventories of available data. Such an application could then examine relationships between data availability and observed uptake and investigate whether apparent gaps are more consistent with limitations in data production or with barriers related to accessibility, documentation, discoverability, timeliness, or dissemination. These extensions would move the framework from identifying dataset references toward a more systematic evidence base for monitoring the reach of existing data investments and informing future decisions about data production and dissemination. 7CONCLUSION Tracking dataset references across large document collections is difficult when citation practices are inconsistent and domain-specific labeled data are scarce. This paper presents a weakly supervised framework for adapting dataset extraction to FCV and forced-displacement documents without re- quiring a large manually annotated training corpus. The approach combines candidate generation from an existing extraction model, LLM-assisted annotation refinement, targeted synthetic and con- trastive examples, and fine-tuning of a lightweight local extraction model. Evaluation on an independent gold-standard benchmark demonstrates that the resulting model can identify dataset references across heterogeneous research, humanitarian, and operational documents while maintaining strong precision on passages containing dataset mentions. The results also high- light a distinctive challenge of domain adaptation in this setting: dataset references must be dis- tinguished from semantically similar entities such as project titles, administrative systems, publi- cations, and other information products. The candidate-refinement analysis suggests that exposure to such difficult negative cases is an important component of the adaptation process and motivates further evaluation through targeted ablation studies. More broadly, reliable dataset extraction provides a technical foundation for examining the dis- placement data landscape at a scale that would be difficult to achieve through manual review alone. The present study establishes one component of that broader capability: scalable identification of dataset references across heterogeneous documents. Combined with systematic corpus construc- tion, displacement-relevance classification, dataset resolution and linkage, contextual classification of how data are used, and information on available data resources, this capability could support anal- ysis of where displacement-relevant data are reaching research, policy, and operational processes, where existing resources may face barriers to uptake, and where unmet information needs may in- 14 Working Paper dicate potential gaps in data production. Such analysis would extend the value of dataset-reference extraction beyond monitoring the downstream reach of existing data investments toward informing future decisions about data production, accessibility, documentation, and dissemination. OPEN SOURCE, ETHICS, DISCLOSURE, AND DISCLAIMER CodeandModelAvailability.Thefine-tunedextractionmodel,modelcard, inferenceexamples,andsupportingdocumentationarepubliclyavailableat https://huggingface.co/collections/ai4data/data-use-displacement. Ethics Statement. This study analyzes publicly available research papers, project documents, and humanitarian reports. It does not process individual-level records or personally identifiable informa- tion about displaced or vulnerable populations; all analyses are conducted at the level of document text. The framework is intended to help data-producing organizations systematically identify where surveys, administrative data, and other data resources are referenced across research, policy, and op- erational documents. Because automated extraction is imperfect, human review remains appropriate when model outputs are used for consequential monitoring, reporting, or evaluation. Disclosure of AI Use. AI tools were used as part of both the research methodology and manuscript preparation. As described in the paper, OpenAI models and open-source models were used for tasks including candidate verification, synthetic-data generation, and model development. Microsoft Copilot and ChatGPT were also used to improve the clarity and readability of the manuscript. All methodological decisions, analyses, interpretations, and conclusions remain the responsibility of the authors. Disclaimer. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Recon- struction and Development/World Bank and its affiliated organizations, the Executive Directors of the World Bank, or the governments they represent. REFERENCES Peter Buneman, Dennis Dosso, Matteo Lissandrini, and Gianmaria Silvello. Data citation and the citation graph. Quantitative Science Studies, 2(4):1399â1422, 2021. doi: 10.1162/qss a00166. URL https://doi.org/10.1162/qss_a_00166. Jenny Heddes, Pim Meerdink, Miguel Pieters, and Maarten Marx. The automatic detection of dataset names in scientific articles. Data, 6(8):84, 2021. doi: 10.3390/data6080084. URL https: //w.mdpi.com/2306-5729/6/8/84. Johannes Hoogeveen and Utz Pape (eds.). Data Collection in Fragile States: Innovations from Africa and Beyond. Palgrave Macmillan, Cham, 2020. doi: 10.1007/978-3-030-25120-8. URL https://doi.org/10.1007/978-3-030-25120-8. Tayyaba Hussain, Muhammad Usman Akram, and Anum Abdul Salam. A novel data extraction framework using natural language processing (defnlp) techniques. SSRN, July 2023. URL https://ssrn.com/abstract=4524837. Inter-AgencyStandingCommittee.IASCoperationalguidanceon dataresponsibilityinhumanitarianaction,2023.URL https:// interagencystandingcommittee.org/operational-response/ iasc-operational-guidance-data-responsibility-humanitarian-action. Rafael Macalaba and Aivin V. Solatorio. AI for monitoring and classifying data used in research literature, 2026. URL https://arxiv.org/abs/2605.30582. RafaelSevillaMacalabaandAivinSolatorio.Trackingdatasetuseat scalewithAI:Howsyntheticdatahelpsovercomedatascarcity,Novem- ber2025.URL https://blogs.worldbank.org/en/opendata/ tracking-dataset-use-at-scale-with-ai--how-synthetic-data-helps-. 15 Working Paper Hailey Mooney and Mark P. Newton. The anatomy of a data citation: Discovery, reuse, and credit. Journal of Librarianship and Scholarly Communication, 1(1):eP1035, 2012.doi: 10.7710/2162-3309.1035. URL https://doi.org/10.7710/2162-3309.1035. NuMind.Nuextract:Aframeworkforstructureddataextrac- tion.NuMindBlog,2024.URL https://numind.ai/blog/ nuextract-a-foundation-model-for-structured-extraction. Heather A. Piwowar and Todd J. Vision. Data reuse and the open data citation advantage. PeerJ, 1: e175, 2013. doi: 10.7717/peerj.175. URL https://peerj.com/articles/175. Nancy Potok. Show US the data. Harvard Data Science Review, 4(2), 2022. doi: 10.1162/99608f92. 9d13ba15. URL https://doi.org/10.1162/99608f92.9d13ba15. Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R Ě e. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endow- ment, 11(3):269â282, 2017. Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009. Gianmaria Silvello. Theory and practice of data citation. Journal of the Association for Information Science and Technology, 69(1):6â20, 2018. doi: 10.1002/asi.23917. URL https://arxiv. org/abs/1706.07976. Aivin V. Solatorio, Rafael Macalaba, and James Liounis. Large language models and synthetic data for monitoring dataset mentions in research papers, 2025. URL https://arxiv.org/abs/ 2502.10263. UNHCR. UNHCR data transformation strategy 2020â2025. Technical report, United Nations High Commissioner for Refugees, 2020. URL https://w.unhcr.org/us/media/ data-transformation-strategy-2020-2025. UNHCR. Global trends: Forced displacement in 2024. Technical report, United Nations High Commissioner for Refugees, 2025. URL https://w.unhcr.org/sites/default/ files/2025-06/global-trends-report-2024.pdf. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, and Daniel Khashabi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022. World BankâUNHCR Joint Data Center on Forced Displacement.Strategy 2024â2027. Technical report, World BankâUNHCR Joint Data Center on Forced Displacement, 2024. URL https://w.jointdatacenter.org/wp-content/uploads/2024/ 04/JDC-Strategy2024-27.pdf. WorldBankGroup.Aworldbankgroupstrategyforengaginginfragility, conflictandviolenceaffectedsettings(2026â2030).Technicalreport,World BankGroup,Washington,DC,2026.URL https://w.worldbank. org/en/topic/fragilityconflictviolence/publication/ a-wbg-strategy-for-engaging-in-fcv-settings-2026-2030. Yousef Younes and Ansgar Scherp. Question answering versus named entity recognition for ex- tracting unknown datasets. IEEE Access, 11:92775â92787, 2023. doi: 10.1109/ACCESS.2023. 3309148. URL https://doi.org/10.1109/ACCESS.2023.3309148. Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. Gliner: Generalist model for named entity recognition using bidirectional transformer. arXiv preprint, 2023. URL http: //arxiv.org/abs/2311.08526. 16 Working Paper AAPPENDIX A.1A. LLM-ASSISTED ANNOTATION REFINEMENT This section provides additional implementation details for the LLM-assisted annotation refinement described in Section 3. Candidate dataset spans produced by the weakly supervised candidate gen- erator are passed to GPT-4o together with their surrounding context. The LLM determines whether each candidate is a valid dataset reference and returns its decision through OpenAIâs structured outputs API. The annotation rules are specified in two complementary ways. A Pydantic response schema con- strains the structure of the returned annotation, while a system prompt defines what constitutes a valid dataset reference and the rules used to distinguish datasets from other entities. A.1.1PYDANTIC RESPONSE FORMAT The response from the LLM judge is constrained using the Pydantic schema shown in Figure 1. The structured response format ensures that validation decisions and associated annotations can be parsed consistently and incorporated into the training-data construction pipeline. Figure 1: Pydantic response schema used for LLM-assisted annotation refinement. A.1.2LLM JUDGE SYSTEM PROMPT The system prompt used to evaluate candidate dataset mentions is reproduced below. 17 Working Paper System Prompt You are an expert data annotator and LLM judge. Your task is to evaluate whether a can- didate text span extracted from a sentence is a valid âData Mentionâ based on the strict guidelines below. Core Principle A valid data mention must refer directly to a structured data object or resource (e.g., a database, survey, file, registry, microdata, census, or indicator index). ⢠Any citation, quote, or reference to data being used, relied upon, or quoted (e.g., in external reports, policy strategies, or cited literature) is a valid data mention. ⢠Vague or generic references to data resources (e.g., ârepeated survey dataâ, âtime-varying data on productionâ, âmicrodataâ, âhousehold dataâ, âadministrative recordsâ) are valid vague data mentions. Exclusions (Flag as Invalid) 1. Author calculations and self-attributions (e.g., âown elaborationâ, âSAG estimatesâ). 2. Geographical or political entities on their own (e.g., âGhanaâ, âSomaliaâ). 3. Model configurations, statistical methodologies, or analytical frameworks (e.g., âpro- bit modelâ, âsensitivity analysisâ). 4. Technical hardware, sensors, or datums (e.g., âsensorâ, âWGS84â). 5. Document structure and layout titles (e.g., âTable 1â, âFigure 3â, âthis reportâ). 6. Qualitative research methodologies on their own (e.g., âFocus Group Discussionsâ, âKey Informant Interviewsâ). 7. Bare organization names in opinion, actor, or decision roles (e.g., âWorld Bankâ, âUNHCRâ) unless combined with a data noun. 8. Activity, process, or recommendation references (e.g., âdata collection and surveysâ). 9. Written reports, narrative publications, bulletins, policy briefs, and response plans (e.g., âWorld Development Report 2022â, âHumanitarian Response Planâ) when cited as background literature rather than an active structured database. 10. PDF page-layout artifacts, running headers or footers, and repeating document or project titles (e.g., âCosta Rica Results in Education (P181174)â, âZambia Agribusi- ness and Trade Projectâ). 11. Table cells and structured form fields that merely list data sources without prose nar- rative. Only evaluate mentions that appear in complete prose sentences. Strict Specificity Taxonomy ⢠named: The dataset has a proper name, acronym, or formal registry or branding title (e.g., âDemographic and Health Surveyâ, âDHSâ, âLSMSâ). ⢠descriptive: The dataset is described in a specific way but does not use a proper name. It must contain at least two identifying details, such as who collected it, the population or geography covered, when it was collected, or how it was produced. ⢠vague: Generic data terms or variables without specific identifying details (e.g., âsurvey dataâ, âadministrative recordsâ). A.2B. SYNTHETIC DATA GENERATION Synthetic examples supplement the LLM-refined annotations obtained from real documents. As de- scribed in Section 3, generation uses few-shot prompting to produce additional examples of dataset references across different domains and contexts. This section provides additional detail on the domain variable banks, validation rules, and contrastive examples used in this process. A.2.1DOMAIN VARIABLE BANKS To increase variation in the generated examples, the generation procedure draws from six domain- specific banks containing dataset names and related data resources. These banks steer generation toward terminology relevant to development, humanitarian, and FCV applications. 18 Working Paper Table 5: Domain variable banks used to steer synthetic example generation. DomainIllustrative Data Resources and Terms Development EconomicsLogistics Performance Index; CPIA; Enterprise Surveys; Doing Busi- ness; Household Income and Expenditure Survey; LSMS; Poverty As- sessment Humanitarian & ProtectionMulti-Sector Needs Assessment (MSNA); UNHCR Refugee Registry; Protection Monitoring; mVAM; Displacement Tracking Matrix; Liveli- hoods Survey Global HealthDHS; Multiple Indicator Cluster Surveys (MICS); STEPwise approach to surveillance; National Health Accounts; Malaria Indicator Survey Food Security & Agricul- ture Integrated Food Security Phase Classification (IPC); Comprehensive Food Security and Vulnerability Analysis (CFSVA); Food Consumption Score EducationPISA; TIMSS; PIRLS; EGRA; EGMA; SACMEQ; Education Manage- ment Information System (EMIS); learning assessment database Climate & EnvironmentEM-DAT; global climate models; NOAA satellite data; IPCC emissions database; water monitoring indicators A.2.2VALIDATION OF SYNTHETIC EXAMPLES Generated examples are subjected to automated checks before inclusion in the training corpus. These checks enforce three requirements: 1. The generated passage must contain the requested dataset and associated metadata at- tributes verbatim. 2. Extracted dataset spans must not include standalone publication years. 3. Extracted spans must not terminate in incomplete grammatical fragments, such as trailing prepositions or verbs (e.g., âofâ, âforâ, or âfromâ). Examples that fail these checks are excluded from the training corpus. A.2.3CONTRASTIVE EXAMPLES Synthetic generation is also used to construct contrastive examples. These examples are designed to distinguish dataset references from similar expressions whose interpretation changes with con- text. The objective is to reduce reliance on surface features such as capitalization, acronyms, or the presence of terms such as âsurvey.â Table 6 provides illustrative examples. Table 6: Illustrative contrastive examples used during synthetic-data generation. ContextExample SentenceTarget SpanLabel DatasetThe analysis incorporates the Living Standards Measurement Survey (LSMS) to calculate con- sumption aggregates. LivingStandards Measurement Survey named Non-datasetThe LSMS team coordinated field operations and trained local enumerators. NoneEmpty DatasetWe analyzed the household consumption survey data across three displacement camps. household consump- tion survey descriptive Non-datasetWe conducted household surveys to assess imme- diate protection needs. NoneEmpty The contrastive examples are intended to teach the model that the presence of a familiar dataset name or data-related expression is not sufficient for extraction. The surrounding context must indicate that the expression refers to a data resource. 19 Working Paper A.3C. ADDITIONAL EVALUATION DETAILS This section provides additional detail on the evaluation procedure described in Section 4. We evaluate the final fine-tuned model at both the mention level and the passage level. A.3.1MENTION-LEVEL SPAN MATCHING Mention-level evaluation compares each predicted dataset span with the gold-standard annotation using token-level Jaccard similarity. For a predicted span S p and gold span S g , similarity is defined as J(S p ,S g ) = |W p ⊠W g | |W p ⪠W g | ,(2) where W p and W g denote the sets of tokens in the predicted and gold-standard spans, respectively. A prediction is considered a match when J(S p ,S g )⼠0.5.(3) This criterion allows minor differences in extraction boundaries while requiring substantial overlap with the annotated dataset mention. A.3.2PASSAGE-LEVEL DETECTION We additionally evaluate whether the model correctly determines if a text chunk contains at least one dataset reference. A passage is classified as positive if the model extracts one or more dataset mentions and negative if it returns no extraction. This provides a complementary evaluation of the modelâs ability to distinguish dataset-bearing pas- sages from passages containing no dataset reference, which is important when applying the extractor to large document collections. A.4D. ANALYSIS OF FILTERED CANDIDATE MENTIONS The weak-supervision stage deliberately uses a permissive prediction threshold of Ď = 0.15 to prioritize candidate recall. Consequently, the candidate generator surfaces both potential dataset references and other entities that share textual characteristics with dataset names. These predictions are subsequently evaluated during LLM-assisted annotation refinement. This behavior is particularly visible in operational documents, where project titles, administrative systems, monitoring tools, document headings, and information products often appear as capitalized multi-word expressions or acronyms. To characterize these cases, Table 7 reports the 15 most fre- quent candidate strings filtered out during annotation refinement in the FCV operational documents. 20 Working Paper Table 7: Top 15 most frequent candidate strings filtered out during LLM-assisted annotation refine- ment in FCV operational documents. RankCandidate StringFreq.Primary Reason for Exclusion 1Iraq COVID-19 Vaccination Project25Project title, not a structured data resource. 2FormalEmploymentCreation Project 23Project title repeated as part of the document struc- ture. 3Municipal Services Improvement Project... 18Project title, not a structured data resource. 4PROJECT FINANCING DATA5Document structure header, not a dataset. 5INSTITUTIONAL DATA4Section title within an operational reporting template. 6SYSTEMATICOPERATIONS RISK-RATING TOOL 3Risk-assessment framework or tool, not a dataset. 7Procurement Risk Assessment & Mgmt System 3Administrative system (PRAMS), not a dataset. 8Municipal Services Improvement Project 3Project name in an administrative context. 9Emergency Food Security Project3Operational project title, not a structured data re- source. 10Project Development Objective Indi- cators 2Monitoring framework terminology rather than an identifiable dataset. 11Systematic Tracking of Exchanges in Procurement 2Administrative procurement platform (STEP), not a dataset. 12TKYB ESMS2Operational environmental and social management system. 13EU Needs Assessment2Narrative publication or report rather than an under- lying dataset. 14official monthly data1Generic expression rejected under the annotation cri- teria in its document context. 15Livelihoods Survey Findings1Narrative publication or report rather than an under- lying dataset. The filtered candidates are dominated by meaningful but out-of-scope entities rather than arbitrary strings. In particular, project and program titles recur frequently in operational documents and often have the same surface features as named datasets. Administrative systems and monitoring tools present a similar pattern. This illustrates why candidate generation and annotation refinement serve different purposes in the framework: the first stage is designed to surface plausible candidates broadly, while the second determines whether those candidates actually refer to structured data re- sources. A.4.1ILLUSTRATIVE CONTEXT EXAMPLES Candidate strings alone do not always provide enough information to determine whether an ex- pression refers to a dataset. Table 8 illustrates how valid dataset references and filtered candidates appear in context. The examples show why surrounding text is useful for distinguishing structured data resources from projects, administrative platforms, qualitative methods, and publications. 21 Working Paper Table 8: Illustrative contexts for valid dataset references and filtered candidate mentions. StatusCandidate EntityContext Sentence ValidDemographic Health Surveyâ...estimates are derived using microdata from the De- mographic Health Survey conducted in the region.â ValidMSNA Poland 2023 surveyâThe MSNA Poland 2023 survey serves as the pri- mary dataset for profiling refugee household expendi- ture.â ValidZUS dataâWe cross-reference these registration figures with monthly administrative ZUS data on active social ben- efits.â FilteredIraqCOVID-19Vaccination Project âUnder the Iraq COVID-19 Vaccination Project, the government procured vaccines to support the immu- nization campaign.â FilteredSystematic Tracking of Ex- changes in Procurement âAll procurement processes must be logged in the Systematic Tracking of Exchanges in Procurement platform.â FilteredFocus Group DiscussionsâQualitative findings from the Focus Group Discus- sions are used to supplement our statistical results.â FilteredLivelihoods Survey FindingsâAccording to the Livelihoods Survey Findings pub- lished by the agency, agricultural yields have de- clined.â The examples reinforce the role of context-aware refinement. Terms such as âSurvey,â âAssessment,â or an institutional acronym provide useful signals during candidate generation, but they do not by themselves establish that the expression refers to a dataset. The refinement stage instead evaluates the role of the candidate in the surrounding text. 22