Paper deep dive

WRAP++: Web discoveRy Amplified Pretraining

Jiang Zhou, Yunhao Wang, Xing Wu, Tinghao Yu, Feng Zhang

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 60

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 4/10/2026, 3:44:20 AM

Summary

WRAP++ (Web discoveRy Amplified Pretraining) is a framework that enhances LLM pretraining by moving beyond single-document rephrasing to cross-document knowledge discovery. By leveraging web hyperlinks to identify high-confidence relational motifs (dual-links and co-mentions), WRAP++ synthesizes joint QA pairs that require multi-hop reasoning. This approach amplifies data scale combinatorially and improves factual knowledge recoverability, as demonstrated by significant performance gains on the SimpleQA benchmark using OLMo-based models.

Entities (5)

OLMo · language-model · 100%SimpleQA · benchmark · 100%WRAP++ · framework · 100%FineWiki · dataset · 95%Qwen3 · language-model · 95%

Relation Signals (3)

WRAP++ → improvesperformanceon → SimpleQA

confidence 100% · On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches

WRAP++ → usesdataset → FineWiki

confidence 100% · We instantiate WRAP++ on Wikipedia... Specifically, we use the English subset of FineWiki

WRAP++ → utilizesmodel → Qwen3

confidence 95% · We use Qwen3-30B-A3B-Instruct-FP8 as our instruction-tuned generator

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operate at the single-document level, rewriting individual web pages in isolation. This confines synthesized examples to intra-document knowledge, missing cross-document relationships and leaving facts with limited associative context. We propose WRAP++ (Web discoveRy Amplified Pretraining), which amplifies the associative context of factual knowledge by discovering cross-document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, WRAP++ discovers high-confidence relational motifs including dual-links and co-mentions, and synthesizes QA that requires reasoning across both documents. This produces relational knowledge absent from either source document alone, creating diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia, we amplify ~8.4B tokens of raw text into 80B tokens of cross-document QA data. On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches and exhibit sustained scaling gains, underscoring the advantage of cross-document knowledge discovery and amplification.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

59,376 characters extracted from source content.

Expand or collapse full text

April 10, 2026 WRAP++: WEB DISCOVERY AMPLIFIED PRETRAINING Jiang Zhou, Yunhao Wang, Xing Wu † , Tinghao Yu, Feng Zhang ABSTRACT Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operate at the single-document level, rewriting individual web pages in isolation. This confines synthesized examples to intra-document knowl- edge, missing cross-document relationships and leaving facts with limited asso- ciative context. We propose WRAP++ (Web discoveRy Amplified Pretraining), which amplifies the associative context of factual knowledge by discovering cross- document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, WRAP++ discovers high-confidence rela- tional motifs including dual-links (A↔ B) and co-mentions (A→ E ← B with A→ B), and synthesizes QA that requires reasoning across both documents. This produces relational knowledge absent from either source document alone, creating diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia, we amplify ∼8.4B tokens of raw text into 80B tokens of cross-document QA data. On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches and exhibit sus- tained scaling gains, underscoring the advantage of cross-document knowledge discovery and amplification. 1INTRODUCTION Synthetic data has become an increasingly important component of large language model (LLM) pretraining. WRAP (Maini et al., 2024) showed that rephrasing noisy web text into QA format can improve pretraining, and later systems scaled this recipe substantially: Nemotron-C (Su et al., 2025) produced∼2 trillion synthetic tokens from Common Crawl, Phi-4 (Abdin et al., 2024) used 40% synthetic data in pretraining, and Qwen3 (Yang et al., 2025) incorporated synthetic data into its training pipeline. However, this progress has been explored mainly along intra-document axes—rephrasing strategy, generator model, and source quality (Nguyen et al., 2025; Niklaus et al., 2026)—varying how a sin- gle document is rewritten without changing what is synthesized. Because many facts are distributed across multiple documents, this single-document paradigm confines the model to limited associative context for each fact, which ultimately hinders knowledge recoverability. This limitation motivates a shift toward cross-document synthesis: bringing multiple facts into a shared context to learn relational knowledge jointly. However, moving to cross-document synthesis is non-trivial. A na ̈ ıve approach of randomly pairing documents yields little improvement over single-document baselines (see §4.1), as forcing an LLM to synthesize joint QA from unrelated texts produces fabricated connections and low-quality data. Thus, the document selection mechanism is critical: cross-document synthesis only succeeds when the paired documents contain genuinely related facts. Web hyperlinks provide a broad relevance signal by encoding human-curated judgments of impor- tance (Zhou et al., 2022). For example, the Wikipedia pages of composers Hans Zimmer and Ludwig G ̈ oransson are topologically linked through shared collaborations with director Christopher Nolan. While single-document synthesis might only extract isolated facts (e.g., ”G ̈ oransson won an Oscar † Corresponding Author. Correspondence to ucaswu@tencent.com. 1 arXiv:2604.06829v2 [cs.CL] 9 Apr 2026 WRAP++ (OURS) — WEB DISCOVERY AMPLIFIED PRETRAINING Web Corpus Fine Wiki 6.7M articles 1Topological Relation Discovery AB Dual-Link AB Co-Mention E 2Cross-Doc Joint QA Synthesis Entity Pairs (A, B) ➕ Doc A Doc B SINGLE-DOCUMENT REWRITING Cross-Doc QA Pretraining Corpus I Q: What films earned Nolan's original Tenet composer and his replacement their second Oscars? A: Hans Zimmer turned down Tenet for Dune, which earned his second Oscar (after The Lion King). His replacement, Ludwig Göransson, later won his second Oscar for Oppenheimer (after Black Panther). Therefore, the answer is Dune and Oppenheimer. 80B Source tokensSynthesized 8.4B ~10× amplification G F E B C A D HJ X denote a page Cross-Doc Dependency Factual Chaining Omniscient Internalize Göransson won Best Score for Oppenheimer. Zimmer scored multiple Nolan films... 5.4B Q: What award did Göransson win for Oppenheimer? A: Best Original Score at the Academy Awards. Synthesized tokens LLM rewrites one document into QA Limited to intra-doc knowledge denote a relation Figure 1: Overview of the WRAP++ pipeline. Unlike single-document WRAP, which rewrites individual documents, WRAP++ discovers cross-document entity relationships from web topology and amplifies them into pretraining data through joint QA synthesis. for Oppenheimer”), WRAP++ pairs these connected documents to synthesize multi-hop relational QA. As illustrated in Figure 1, the model is forced to explicitly reason across both texts—deducing that Zimmer left Tenet to score Dune (his second Oscar), leading to G ̈ oransson’s hiring and sub- sequent second Oscar for Oppenheimer. This explicit relational supervision provides vital disam- biguation cues and diverse retrieval paths absent from single-document synthesis, saving the LLM from having to implicitly infer these complex connections from massive unstructured text. Based on this insight, we propose WRAP++ (Web discovRy Amplified Pretraining), which ef- fectively extends the synthetic data paradigm from single-document rewriting to cross-document discovery and amplification. Amplifying Associative Context via Relation Discovery. We discover high-confidence relational motifs from web hyperlinks (Zhou et al., 2022)—specifically dual-links (A ↔ B) and co-mentions (A → E ← B with A → B). Rather than simply concatenating these documents, we feed the discovered pairs to an instruction-tuned LLM generator subjected to three strict synthesis con- straints: enforcing Cross-Document Dependency to mandate joint reasoning, requiring Explicit Factual Chaining to decode multi-hop logical paths, and ensuring Omniscient Internalization by forbidding local document attribution. This process produces genuinely new relational knowl- edge (comparisons, contrasts, bridging facts) that creates diverse retrieval paths to the same facts. Furthermore, because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis achieves a∼10× data amplification—scaling a fixed 8.4B-token source corpus into 80B tokens of cross-document QA data—consistently improving the knowledge recoverability of the model. We instantiate WRAP++ on Wikipedia and amplify ∼8.4B tokens of raw text into 80B tokens of cross-document QA data—compared to only ∼5.4B tokens from single-document WRAP. On the SimpleQA benchmark (Wei et al., 2024), OLMo-based models at both 7B and 32B scales trained with WRAP++ data substantially outperform all single-document baselines, and WRAP++ demon- strates a more favorable scaling trajectory than single-document approaches. Our contributions are three-fold: 2 1. We propose WRAP++, a framework that extends single-document rewriting into topology- guided relation discovery and joint QA synthesis, amplifying the associative context of factual knowledge. 2. We instantiate WRAP++ on Wikipedia to synthesize 80B tokens of cross-document QA data, demonstrating that combinatorial relation discovery enables data amplification far beyond single-document synthesis. 3. We show on SimpleQA with OLMo-based 7B and 32B models that WRAP++ substantially outperforms single-document baselines, exhibiting a favorable scaling trajectory. 2METHOD: WRAP++ WRAP++ is a framework that transitions synthetic pretraining data from single-document rewrit- ing to topology-guided cross-document discovery and amplification. The framework consists of two core stages: Topological Relation Discovery (§2.2) and Joint QA Synthesis (§2.3). Figure 1 illustrates the overall pipeline. 2.1PROBLEM FORMULATION AND GRAPH ABSTRACTION LetD = d 1 ,d 2 ,...,d N denote a large-scale web corpus consisting of N documents. The inter- document references (e.g., hyperlinks) naturally induce a directed knowledge graph G = (V,E), where each vertex v i ∈ V corresponds to a document d i , and a directed edge e i,j ∈ E exists if d i explicitly references d j . Since each document in our corpus describes a single entity, we use “entity pair” and “document pair” interchangeably throughout this paper. Conventional single-document synthesis operates solely on the local context of v i , limiting the model’s exposure to isolated facts. In contrast, WRAP++ leverages the topological structure of G to discover genuine semantic dependencies across documents, bringing related knowledge into a shared synthesis context to amplify the associative context of factual knowledge. 2.2TOPOLOGICAL RELATION DISCOVERY A na ̈ ıve approach of pairing random documents from D forces the synthesis model to hallucinate spurious connections. To ensure the semantic validity of cross-document synthesis, we discover high-confidence relational motifs directly fromG. We focus on two topological structures that pro- vide strong inductive biases for relational reasoning: Dual-link Motif. Two documents u and v form a dual-link relationship if they mutually reference each other. Formally, a dual-link pair (u,v) satisfies the bidirectional constraint: e u,v ∈E ∧ e v,u ∈E(1) This mutual dependency typically indicates a strong, foundational semantic correlation (e.g., a no- table director and their magnum opus, or a scientist and their core discovery). Discovering this motif ensures the underlying entity pair is highly coupled. Co-mention Motif. Documents u and v share a co-mention relationship if they both reference a common structural hub E, while maintaining a direct link between themselves. Formally, the triplet (u,v,E) satisfies: e u,E ∈E ∧ e v,E ∈E ∧ e u,v ∈E(2) The shared structural context E imposes implicit analogical, hierarchical, or comparative relation- ships (e.g., two competing theories cited in the same survey article). This motif explicitly encour- ages the subsequent synthesis model to generate relational knowledge that contrasts and compares the related entities, thereby amplifying their associative context. 2.3CROSS-DOCUMENT JOINT QA SYNTHESIS Given a discovered document pair (d u ,d v ) connected by a valid topological motif, we employ an instruction-tuned LLM generator M θ to synthesize a set of composite QA instances S u,v = 3 (q i ,c i ,a i ) K i=1 , where q i is the question, c i is the intermediate factual chain, and a i is the final answer. The generation process is conditioned on a structured prompt P and the concatenated document context: S u,v ∼M θ (d u ⊕ d v ,P)(3) To amplify the associative context of the synthesized data and prevent the generator from degrading to shallow single-document summarization, P enforces three functional constraints on the output space: • Strict Cross-Document Dependency: The generated question q i must have high entropy given only one document. Deriving the correct answer a i must strictly require logical premises from both d u and d v , ensuring the synthesis produces genuinely new relational knowledge rather than merely rephrasing isolated facts. • Explicit Factual Chaining: Before outputting a i , the generator must explicitly decode the traversal path c i . By articulating the necessary facts extracted from both documents and linking them step-by-step, the pretraining model internalizes multi-hop knowledge struc- tures, effectively creating diverse associative entry points to the underlying facts. • Omniscient Internalization: The generator is strictly prohibited from attributing facts to the local context (e.g., avoiding “According to Passage A”). It must output universally valid statements. This ensures the synthesized data serves as parametric world knowledge rather than context-dependent reading comprehension exercises. 3EXPERIMENTS 3.1EXPERIMENTAL SETUP Synthesis Data. In principle, WRAP++ is a general framework applicable to any text corpus containing hyperlinks. In this work, we instantiate it on Wikipedia because of its rich link structure and its widespread use in prior work on synthetic rewriting methods (Maini et al., 2024; Su et al., 2025). Specifically, we use the English subset of FineWiki (Penedo, 2025) as our base corpus (D), which contains approximately 8.4B tokens. We parse the hyperlinks in FineWiki to construct the directed inter-document graphG used for topological relation discovery. Synthesis Model. We use Qwen3-30B-A3B-Instruct-FP8 as our instruction-tuned generatorM θ . The prompt is designed to enforce the strict cross-document dependency and explicit factual chain- ing constraints described in §1, thereby encouraging high-quality relational QA generation. The full prompt template is provided in Appendix D. Data Scale. Topological relation discovery substantially amplifies the data scale beyond individual documents. The dual-link motif yields highly coupled entity pairs that produce∼3B tokens of cross- document QA data. Incorporating the co-mention motif broadens coverage, bringing the combined WRAP++ dataset to∼82.7B tokens. Training Models. To assess the effect of cross-document synthetic data on parametric knowledge, we continue pretraining from the OLMo-3 stage-1 last checkpoint at both the 7B and 32B scales for one epoch. We choose OLMo because it is fully open-source and provides publicly released checkpoints throughout training, making it a suitable platform for controlled continued-pretraining experiments. 3.2EVALUATION SETUP Benchmark. We evaluate on SimpleQA (Wei et al., 2024), a knowledge-intensive benchmark de- signed to measure short-form factual accuracy while minimizing sensitivity to formatting heuristics. Most SimpleQA questions can be answered directly from Wikipedia-derived knowledge, making it a natural testbed for studying factual knowledge acquisition under our setup. 4 Table 1: SimpleQA results after 1-epoch continued training on OLMo-3-7B and OLMo-3-32B using different data recipes. The metric reported is the empirical pass@128 rate (%). Data RecipeOLMo-3-7B OLMo-3-32B Pretrained Base34.7642.35 + WRAP39.5544.43 + Extended WRAP43.6947.91 + WRAP++49.1353.97 Metric. We use pass@128 as our primary metric, defined as the empirical probability that at least one of 128 sampled responses contains the correct fact. Our goal is to measure knowledge recover- ability rather than only top-1 answer accuracy. In this setting, pass@128 is useful because it probes whether the correct factual association can be elicited from the model under sampling, even when it is not the single most likely surface form. We therefore treat it as a more sensitive indicator of parametric knowledge recoverability during continued pretraining than pass@1. Baselines. To isolate the benefit of cross-document synthesis, we compare WRAP++ against two closely related single-document baselines derived from the same FineWiki corpus: 1. WRAP (∼5.4B tokens): Standard single-document QA synthesis following the original WRAP recipe (Maini et al., 2024), representing the typical yield of intra-document fact extraction. 2. Extended WRAP (∼17.4B tokens): An expanded single-document synthesis utilizing additional prompting strategies (e.g., exhaustive extraction) to push the limits of single- document scaling. These baselines also illustrate the data-scaling constraint of single-document synthesis: on the same FineWiki corpus, standard WRAP yields only∼5.4B tokens and Extended WRAP reaches∼17.4B tokens, both well below the∼82.7B-token scale of WRAP++. 3.3MAIN RESULTS Table 1 presents the main results of 1-epoch continued training across different synthesis recipes. We highlight two principal findings. WRAP++ substantially outperforms single-document baselines. Across both model scales, continued pretraining with WRAP++ yields substantially higher pass@128 on SimpleQA compared to all single-document approaches (+9.5 p on 7B, +9.8 p on 32B over WRAP; +5.4 p on 7B, +6.1 p on 32B over Extended WRAP). This advantage reflects two complementary factors. First, cross- document synthesis produces higher-quality relational knowledge per token: at a matched budget of ∼8B tokens, WRAP++ already outperforms Extended WRAP by +2.48 p (detailed in §4.5), confirming a genuine quality advantage independent of data scale. Second, the combinatorial nature of relation discovery amplifies this quality advantage to a far larger data space (∼80B tokens) that single-document methods cannot access, yielding further gains as training progresses (Figure 2). Surpassing the single-document scaling bottleneck. Single-document methods face an inherent data bottleneck: the finite number of extractable facts within an individual page. While Extended WRAP attempts to push this limit through exhaustive extraction (reaching∼17.4B tokens), it ulti- mately depletes the source material. The resulting diminishing returns (+4.1 p on 7B, +3.7 p on 32B over standard WRAP) suggest information saturation under the single-document paradigm. In contrast, because the number of valid cross-document entity pairs grows combinatorially, WRAP++ amplifies the same FineWiki source corpus into∼80B tokens of relational knowledge—a data space fundamentally inaccessible to single-document methods. Single-document methods cannot close this gap simply by training longer, since their source material is already exhausted. We analyze the resulting scaling dynamics in detail next. 5 010.0B20.0B30.0B40.0B50.0B60.0B70.0B80.0B Tokens 0.350 0.375 0.400 0.425 0.450 0.475 0.500 0.525 0.550 Pass@k (empirical) SimpleQA pass@128 vs Token OLMo3-32B-WRAP++ OLMo3-32B-exWRAP OLMo3-32B-WRAP OLMo3-7B-WRAP++ OLMo3-7B-exWRAP OLMo3-7B-WRAP Figure 2: SimpleQA pass@128 vs. training tokens. Single-document recipes (WRAP and Ex- tended WRAP) reach a data bottleneck early, limiting further knowledge acquisition. In contrast, the combinatorial nature of WRAP++ allows it to scale effectively up to 80B tokens, improving performance without obviously plateauing. 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 k (log scale) 0.1 0.2 0.3 0.4 0.5 Pass@ k (unbiased) (a) OLMo3-7B 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 k (log scale) (b) OLMo3-32B 10.0B 20.0B 30.0B 40.0B 50.0B 60.0B 70.0B 80.0B Training Tokens Figure 3: Evolution of pass@k performance during training. The curves illustrate the unbiased pass@k of OLMo-3-7B (a) and OLMo-3-32B (b) on SimpleQA (k ∈ [1, 128] in log scale). The color gradient (from light to dark blue) tracks the accumulation of consumed WRAP++ tokens (from 10B to 80B). The strictly monotonic upward shift across all values of k indicates robust, unsaturated knowledge internalization. 3.4SCALING AND TRAINING DYNAMICS Figure 2 plots pass@128 as a function of training tokens consumed. The trajectories confirm the scaling bottleneck discussed above: single-document recipes plateau early, whereas WRAP++ maintains a steady upward trend all the way to 80B tokens without obvious saturation, demonstrating that the combinatorial data space opened by relation discovery translates into sustained knowledge gains. To further dissect how this scaling translates into knowledge recoverability, we track the evolution of the pass@k curves throughout training. Figure 3 visualizes the unbiased SimpleQA pass@k for both models. As training progresses (indicated by the light-to-dark blue gradient representing the 80B token influx), the curves exhibit a monotonic upward shift across all values of k. Crucially, this improvement spans the entire logarithmic k-spectrum. The persistent lift at small k (the leftmost regions of the curves) shows that the model’s top-ranked answers increasingly contain the correct fact, reflecting higher precision. Simultaneously, the parallel gains at larger k indicate a broader and more robust set of associative retrieval paths to the same knowledge. 6 Table 2: Ablation results across different design choices (OLMo-3-7B, ∼8B tokens). We report SimpleQA pass@128. WRAP++ (default) uses topological relation discovery, combined topolo- gies, joint QA synthesis, and the Qwen3-30B-A3B synthesis model. Ablation AxisVariantpass@128 Pairing Strategy (§4.1) Random entity pairing43.46 Topological relation discovery45.11 Relation Topology (§4.2) Dual-link only44.24 Co-mention only44.36 Combined45.11 Synthesis Method (§4.3) Raw concatenation (no QA)35.43 QA with source documents prepended38.93 Joint QA synthesis45.11 Synthesis Model Scale (§4.4) Qwen3-30B-A3B45.11 Qwen3-235B-A22B47.70 4ABLATIONS AND ANALYSIS We conduct extensive ablations to validate each component of WRAP++. Due to experimental costs, unless otherwise noted, all ablation experiments use OLMo-3-7B continued pretraining with ∼8B tokens and report the results in terms of SimpleQA pass@128. 4.1NECESSITY OF TOPOLOGICAL RELATION DISCOVERY We explore whether the topological relation discovery is essential, or whether randomly pairing Wikipedia pages would suffice. Table 2 (Pairing Strategy rows) shows a clear performance drop (from 45.11 to 43.46) when entities are paired randomly rather than via dual-link or co-mention re- lations. Qualitatively, random pairing forces the synthesis model to fabricate relationships between unrelated entities, producing factually incorrect comparisons and superficial connections. This con- firms that principled relation discovery—specifically, dual-link and co-mention motif discovery—is important for high-quality cross-document synthesis. 4.2TOPOLOGY COMPARISON: DUAL-LINK VS. CO-MENTION We explore the contribution of each relation type at a matched token budget of∼8B. As shown in Table 2 (Relation Topology rows), both topologies provide strong relational signal at this budget. Co-mention retains a slight edge over dual-link (44.36 vs. 44.24), while their combination yields the best overall performance (45.11). This suggests that bidirectional links and shared structural context capture yet complementary aspects of cross-document knowledge. 4.3NECESSITY AND FORMAT OF QA SYNTHESIS We explore the optimal data format for learning cross-document relationships by comparing our joint QA synthesis against two alternatives: (1) raw concatenation of related documents (no QA), and (2) prepending source documents to the synthesized QA pairs. Table 2 shows raw concatena- tion performs only marginally above the pretrained base (35.43 vs. 34.8), indicating that explicit synthesis is essential to convert document proximity into learnable relational knowledge. Moreover, prepending source documents to QA pairs causes a notable performance drop (38.93), likely by al- lowing the model to superficially copy answers rather than parametrically internalizing them. Thus, joint QA synthesis provides the most effective format for amplifying associative context. 4.4EFFECT OF SYNTHESIS MODEL SCALE We explore the effect of synthesis model scale on WRAP++ quality by comparing two synthesis models of different scales: Qwen3-30B-A3B (3B active parameters) and Qwen3-235B-A22B (22B active parameters). Table 2 (Synthesis Model Scale rows) shows that the larger model produces higher-quality cross-document QA, leading to better downstream pass@128. In practice, the choice 7 Table 3: Performance comparison of WRAP++ mixed with other single-document strategies. Each mixture contains∼8B tokens total. Single-Document Component Proportion of WRAP++ in Mixture 0% (Baseline Only)50% (1:1 Mix) Pretrained Model (No further training)34.76 Raw FineWiki39.2341.80 Distill38.1241.59 Extract Knowledge38.6943.35 Knowledge List38.6042.03 WRAP++ (100%, Ours)45.11 Table 4: Integrating WRAP++ into OLMo-3-7B 100B-token mid-training. We report SimpleQA pass@128 and the average over 12 general benchmarks (including MMLU Redux, HellaSwag, etc. See Appendix F for the full list of tasks). SettingSimpleQA Gen. Avg pass@128(12 tasks) Pretrained Base34.7657.79 + Midtrain (100B)34.7468.24 + WRAP++ Mix (100B)37.5868.16 between synthesis models involves a cost–quality tradeoff: the larger model is preferable when generation budget is not the bottleneck, while the smaller model enables broader coverage at lower compute cost. 4.5COMPARISON WITH OTHER SINGLE-DOCUMENT STRATEGIES We contextualize the performance of WRAP++ against other representative single-document rephrasing strategies Su et al. (2025) applied to the identical FineWiki source corpus, including: (a) Distill—rewriting into cleaner, more concise prose while preserving information; (b) Extract Knowledge—extracting key factual statements and discarding redundancy; (c) Knowledge List— outputting structured knowledge in list format. Table 3 presents both the isolated performance (0% and 100%) and the mixing dynamics (50% blending) at a strictly restricted budget of∼8B tokens. When evaluated in isolation at the 8B-token budget, pure WRAP++ (45.11) substantially outperforms all listed single-document baselines. This margin at a restricted data scale reveals an important dynamic in pretraining efficiency: discovery- driven synthesis already yields stronger knowledge recoverability, even before exploiting its data amplification headroom. When blending WRAP++ with other strategies in a 1:1 ratio, we observe a clear uplift effect: injecting WRAP++ into any weaker baseline consistently improves upon its 0% counterpart (e.g., Knowledge List rises from 38.60 to 42.03), further highlighting the advantage of discovery-driven synthesis. 4.6INTEGRATION WITH OLMO-3 MID-TRAINING DATA We further explore whether WRAP++ data can be integrated into a realistic mid-training pipeline without harming general capabilities. We augment OLMo-3’s 100B-token mid-training mixture with 6B tokens of WRAP++ data and train for the full schedule. As a baseline, we train on the original OLMo-3 mid-training mixture under identical conditions. Following Niklaus et al. (2026), we eval- uate both SimpleQA and the average performance across 12 general tasks (detailed in Appendix F). Table 4 shows that adding WRAP++ data yields a meaningful improvement on SimpleQA (+2.9 points pass@128) while maintaining a comparable general-benchmark average (68.16 vs. 68.24). Notably, WRAP++ explicitly enhances knowledge-intensive tasks, yielding clear gains on MMLU Redux (+1.28, detailed in Appendix F). This demonstrates that WRAP++ integrates cleanly into full-scale mid-training, preserving broad capabilities while explicitly strengthening the model’s gen- 8 eral knowledge foundation. Moreover, as the mid-training budgets of leading models grow toward the trillion-token regime (Yang et al., 2025; Zeng et al., 2026), the 80B-token scale of WRAP++ suggests strong potential for integration into future large-scale training pipelines. 5RELATED WORK Synthetic Data for LLM Pretraining. WRAP (Maini et al., 2024) established synthetic rephras- ing as a practical pretraining paradigm, showing that rewriting web documents into cleaner QA- style text with instruction-tuned models can accelerate pretraining by ∼3×. Subsequent work has expanded this design space along three main axes. First, on rephrasing strategy, Nemotron-C (Su et al., 2025) extracts QA pairs and knowledge lists, REWIRE (Nguyen et al., 2025) introduces guided rewriting with explicit quality criteria, and later work explores additional target formats such as tutorials, FAQs, and mathematical reformulations (Maini et al., 2025; Niklaus et al., 2026). Second, on generator model, studies spanning models from 270M to 27B parameters suggest that moderate-scale models (∼1B–4B) already produce rephrasings competitive with much larger gen- erators (Maini et al., 2024; Niklaus et al., 2026). Third, on source data quality, rephrasing can upcycle low-quality web text (Nguyen et al., 2025), although higher-quality source documents still tend to yield stronger downstream performance (Niklaus et al., 2026). A cross-cutting question concerns how synthetic and original data should be combined, since synthetic-only training often improves factual recall at the expense of broader capabilities, making mixture design important in practice (Maini et al., 2024; Niklaus et al., 2026). Despite this progress, existing methods all syn- thesize from single documents in isolation. WRAP++ differs from this entire line of work by intro- ducing cross-document knowledge discovery and amplification: instead of rewriting one document at a time, it discovers relational structure from web topology and jointly synthesizes training exam- ples from related entity pairs, explicitly modeling relational knowledge that prior single-document approaches leave untapped. 6CONCLUSION We presented WRAP++, a framework that amplifies the associative context of factual knowledge by discovering cross-document relationships from web topology and synthesizing joint QA over related entity pairs. By mining relational motifs (dual-links and co-mentions) from Wikipedia hyperlinks, WRAP++ creates training data with richer relational structure and more diverse retrieval paths than single-document rewriting. On SimpleQA, WRAP++ substantially outperforms single-document approaches at 7B and 32B scales, with a favorable scaling trajectory up to 80B tokens. REFERENCES Marah Abdin, Jyoti Aneja, Harkirat Behl, S ́ ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL https://arxiv.org/abs/2412. 08905. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI conference on artificial intelligence, 34(05):7432–7439, 2020. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec 9 Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161, 2019. Aryo Pradipta Gema et al. Are we done with mmlu? arXiv preprint arXiv:2407.03120, 2024. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling, 2024. URL https:// arxiv.org/abs/2401.16380. Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, et al. Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining. arXiv preprint arXiv:2508.10975, 2025. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. Thao Nguyen, Yang Li, Olga Golovneva, Luke Zettlemoyer, Sewoong Oh, Ludwig Schmidt, and Xian Li.Recycling the web: A method to enhance pre-training data quality and quantity for language models. In Second Conference on Language Modeling, 2025. URL https: //openreview.net/forum?id=lkjhBdz3rn. Joel Niklaus, Guilherme Penedo, Hynek Kydlicek, Elie Bakouch, Lewis Tunstall, Ed Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, and Thomas Wolf. The synthetic data playbook: Generating trillions of the finest tokens, 2026. Team Olmo, :, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heine- man, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Shashank Gupta, Taira Anderson, Teng Xiao, Tyler Murray, Tyler Romero, Vic- toria Graf, Akari Asai, Akshita Bhagia, Alexander Wettig, Alisa Liu, Aman Rangapur, Chloe Anastasiades, Costa Huang, Dustin Schwenk, Harsh Trivedi, Ian Magnusson, Jaron Lochner, Ji- acheng Liu, Lester James V. Miranda, Maarten Sap, Malia Morgan, Michael Schmitz, Michal Guerquin, Michael Wilson, Regan Huff, Ronan Le Bras, Rui Xin, Rulin Shao, Sam Skjons- berg, Shannon Zejiang Shen, Shuyue Stella Li, Tucker Wilde, Valentina Pyatkin, Will Mer- rill, Yapei Chang, Yuling Gu, Zhiyuan Zeng, Ashish Sabharwal, Luke Zettlemoyer, Pang Wei Koh, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. Olmo 3, 2025. URL https: //arxiv.org/abs/2512.13961. Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 1470–1480, 2015. 10 Guilherme Penedo.Finewiki, 2025.URL https://huggingface.co/datasets/ HuggingFaceFW/finewiki.Source:WikimediaEnterpriseSnapshotAPI (https://api.enterprise.wikimedia.com/v2/snapshots). Text licensed under C BY-SA 4.0 with attribution to Wikipedia contributors. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-C: Transforming Common Crawl into a refined long-horizon pretraining dataset. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2459–2475, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/ 2025.acl-long.123. URL https://aclanthology.org/2025.acl-long.123/. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019. URL https://arxiv.org/ abs/1811.00937. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Py- atkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 Furious, 2024. URL https://arxiv.org/abs/2501.00656. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/abs/2411.04368. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 3472–3483, 2019. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chengxing Xie, Cunxiang Wang, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Lan Luo, Ke Zhan, Enrui Hu, Xinyu Zhang, Hao Jiang, Zhao Cao, Fan Yu, Xin Jiang, Qun Liu, and Lei Chen. Hyperlink-induced pre-training for passage retrieval in open-domain question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7135–7146, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.493. URL https://aclanthology.org/2022.acl-long.493/. 11 ALIMITATION Our experiments instantiate WRAP++ on Wikipedia, which is a clean and entity-centric corpus. Extending to noisier web corpora (e.g., Common Crawl), where hyperlinks include advertisements, navigation elements, and low-quality references, will require additional filtering heuristics; we are actively exploring this direction. BTRAINING DETAILS Architecture. We use the OLMo-3 architecture (Olmo et al., 2025), a decoder-only transformer with the Dolma-2 tokenizer (vocabulary size padded to a multiple of 128). Experiments are con- ducted at two scales: OLMo-3-7B (7 billion parameters) and OLMo-3-32B (32 billion parame- ters). We utilize the olmo-core training framework (Team OLMo et al., 2024), which provides a highly optimized and reproducible infrastructure for large-scale distributed training. All models use FlashAttention-2 (Dao, 2023) as the attention backend. Continued Pretraining. All experiments initialize from official OLMo-3 pretrained checkpoints: step 1,413,814 for 7B and step 679,000 for 32B. We load only the model weights and optimizer state (no trainer state) and continue pretraining on synthetic data mixtures for 1 epoch. Table 5 summarizes the hyperparameters for each configuration. Table 5: Continued pretraining hyperparameters. All experiments use a linear decay schedule (no warmup, decaying to 0) and SkipStepAdamW (Olmo et al., 2025) with β 1 =0.9, β 2 =0.95, weight decay 0.1 (embedding weights excluded), max gradient norm 1.0, and auxiliary z-loss with multi- plier 10 −5 . Parameters are stored in bfloat16 and gradients are reduced in float32 via HSDP. HyperparameterOLMo-3-7B (8B) OLMo-3-7B (80B)OLMo-3-32B Sequence length8,1928,1928,192 Global batch size (tokens) ∼2M (2 21 )∼2M (2 21 )∼4M (4× 2 20 ) Peak learning rate2.07× 10 −5 2.07× 10 −5 2.07× 10 −5 LR scheduleLinear→ 0Linear→ 0Linear→ 0 Warmup steps000 Training steps4,00040,00010,000 Training tokens∼8B∼80B∼80B Precisionbfloat16bfloat16bfloat16 Data parallelHSDP (block wrap)HSDP (block wrap)HSDP (full wrap, shard 64) Activation ckpt.FFN-onlyFFN-onlyBudget (50%) GPUs256× H20256× H201024× H20 The learning rate of 2.07× 10 −5 is inherited from the official OLMo-3 mid-training recipe. We use zero warmup steps because the optimizer state is loaded from the pretrained checkpoint, ensuring stable training from the first step. For the OLMo-3-7B scale, training on∼8B tokens using 256 H20 GPUs takes approximately 4.5 hours with a Model FLOPs Utilization (MFU) of ∼65%. For the larger OLMo-3-32B scale, training on∼80B tokens using 1,024 H20 GPUs requires approximately 1 day and 20 hours. These benchmarks demonstrate the efficiency and scalability of our training pipeline on modern hardware. Data Format. All synthetic QA data is formatted as plain text with “Question:” and “Answer:” delimiters, consistent with prior WRAP work. For cross-document QA, the synthesized output directly states facts without referencing source passages, ensuring the model internalizes them as parametric knowledge rather than reading comprehension signals. CEVALUATION DETAILS In-Context Learning Setup. Because our continued pretraining experiments operate on base models (OLMo-3) that have not undergone instruction tuning, these models cannot reliably fol- low zero-shot formatting directives. To accurately probe their parametric knowledge, we adopt a 5-shot in-context learning protocol for all evaluations, including both SimpleQA and the 12 general 12 benchmarks. Specifically, for each evaluation instance, we prepend the prompt with five demon- stration question-answer pairs. For SimpleQA, these demonstrations are sampled directly from the SimpleQA dataset; to ensure strict evaluation integrity and prevent data contamination, any exam- ples used as few-shot demonstrations are explicitly excluded from the active evaluation set during inference. Decoding and Sampling Parameters. We adopt distinct decoding strategies tailored to the nature of each benchmark. For SimpleQA, to compute the pass@k metric (where n = 128), we employ nucleus sampling with a temperature of 0.6 and top-p of 0.95 to provide a diverse distribution for knowledge recoverability analysis. In contrast, for the 12 general benchmarks, we use greedy decoding (temperature 0.0) to ensure deterministic and reproducible outputs across all model com- parisons. These parameters are held constant across all model scales and data recipes to ensure a fair evaluation. Unbiased Estimation of pass@k. While the pass@k metric intuitively represents the probability of generating at least one correct answer within k attempts, empirically estimating this by drawing exactly k samples yields high variance. To achieve a more stable and unbiased estimate, we adopt the methodology introduced by Chen et al. (2021). For each evaluation instance, we generate n total samples (n ≥ k) and determine the number of correct responses, c. The unbiased estimator for pass@k is then computed as: pass@k = 1− n−c k n k (4) where · · denotes the binomial coefficient. In our experiments, we generate n = 128 samples per question. This formulation efficiently leverages all n generated samples to calculate the expected pass rate for any evaluation budget k ≤ n, thereby reducing variance without requiring repeated sampling passes. DSYNTHESIS PROMPT TEMPLATES Single-Document WRAP Prompt (Baseline). Following Maini et al. (2024), we use the standard QA-style prompt for single-document synthesis: Convert the following paragraph into a conversational format with multiple tags of ‘Question:’ followed by ‘Answer:’. WRAP++ Cross-Document Joint QA Prompt. For cross-document synthesis, we provide the full text of both related entities and apply the following instruction. The prompt enforces three critical constraints: cross-document dependency, explicit factual chaining, and omniscient internal- ization (see §2.3). You are an expert data generator for language model pretraining. Below are two related Wikipedia passages: [Passage A] text a [Passage B] text b Task: 1) Generate high-quality synthetic QA pairs that REQUIRE information from BOTH Passage A and Passage B to answer. 2) The Answer MUST begin with a step-by-step reasoning process. This reasoning must explicitly bridge facts from both passages. 3) Do not use external knowledge. 4) CRITICAL CONSTRAINT: The generated QA pair will be used to train a model WITHOUT these passages provided as context. Therefore, you MUST act as an omniscient AI stating absolute 13 facts from your own inherent knowledge. - DO NOT use any attribution phrases like ‘According to Passage A’, ‘Passage B mentions’, ‘As stated in the text’, or ‘Based on the provided documents’. - State the facts directly and confidently. Output format (strict): Question: [Insert a complex question that bridges facts from both passages] Answer: [Acting as an omniscient AI, directly state all necessary factual premises from both passages, and logically synthesize them to derive the conclusion.] Therefore, [State the final, concise answer.] Synthesis Hyperparameters. All QA synthesis uses Qwen3-30B-A3B-Instruct (FP8) as the gen- erator with temperature 0.7, top-p = 0.8, and a maximum output length of 32,768 tokens. For the synthesis model scale ablation (§4.4), we additionally use Qwen3-235B-A22B-Instruct. ERELATION DISCOVERY STATISTICS Table 6: Statistics of the hyperlink relations discovered from FineWiki (English Wikipedia). StatisticValue Source corpusFineWiki (English) Raw corpus tokens∼8.4B Wikipedia articles processed∼6.7M Dual-link pairs (A↔ B)∼9.6M Co-mention pairs (A→ E ← B, A→ B) ∼232M Dual-link synthesized tokens∼3B Co-mention synthesized tokens∼79.7B Total WRAP++ tokens∼82.7B Single-doc WRAP tokens (baseline)∼5.4B Extended WRAP tokens (baseline)∼17.4B Table 6 reports the relation discovery and synthesis statistics. The dual-link motif yields a relatively small but high-precision set of ∼9.6M entity pairs, while the co-mention motif provides a much larger pool of ∼232M pairs, enabling substantial combinatorial expansion. The total synthesized corpus of ∼82.7B tokens is approximately 15× larger than single-document WRAP on the same source, illustrating the amplification advantage of relation-driven cross-document synthesis. FMID-TRAINING EVALUATION BENCHMARKS The 12 general tasks used for evaluation in Section 4.6 are categorized as follows: • General Knowledge: ARC (Clark et al., 2018), MMLU Redux (Gema et al., 2024) • Reading Comprehension: SQuAD v2 (Rajpurkar et al., 2018), DROP (Dua et al., 2019) • Reasoning: OpenBookQA (Mihaylov et al., 2018), CSQA (Talmor et al., 2019) • Natural Language Understanding: WinoGrande (Sakaguchi et al., 2021), PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019) • Math: GSM8K (Cobbe et al., 2021) • Table Understanding: WikiTableQuestions (Pasupat & Liang, 2015), TriviaQA (Joshi et al., 2017) Table 7 provides per-benchmark results for the mid-training integration experiment described in Section 4.6. 14 Table 7: Per-benchmark breakdown for mid-training integration (OLMo-3-7B, full 100B-token schedule, 3-shot cloze format). Accuracy (%) is reported for all tasks except SQuAD v2 and DROP, which use token-level F1. CategoryBenchmarkBase Midtrain WRAP++ Mix General Knowledge ARC77.4784.9885.07 MMLU Redux60.3065.3066.58 Reading Comprehension SQuAD v242.2748.5949.99 DROP40.5166.5165.36 Reasoning OpenBookQA76.4085.2082.40 CSQA72.1574.2072.97 Language Understanding WinoGrande53.2865.1162.98 PIQA73.9974.5474.92 HellaSwag59.8370.1573.12 MathGSM8K38.3679.5377.79 Table & Trivia WikiTableQ36.5543.7444.46 TriviaQA62.4261.0162.23 Average57.7968.2468.16 70.0 72.5 75.0 77.5 80.0 82.5 85.0 Score (%) ARC 50 55 60 65 70 75 Score (%) CSQA 10 20 30 40 50 60 70 Score (%) DROP 45 50 55 60 65 70 75 80 Score (%) GSM8K 40 50 60 70 Score (%) HellaSwag 54 56 58 60 62 64 66 68 Score (%) MMLU Redux 68 70 72 74 76 78 80 82 84 Score (%) OpenBookQA 62 64 66 68 70 72 74 76 78 Score (%) PIQA 0 20B40B60B80B 100B 45 50 55 60 Score (%) SQuAD v2 0 20B40B60B80B 100B 45 50 55 60 Score (%) TriviaQA 0 20B40B60B80B 100B 30 32 34 36 38 40 42 44 46 Score (%) WikiTableQ 0 20B40B60B80B 100B 54 56 58 60 62 64 Score (%) WinoGrande Per-Benchmark Score Trends During Mid-Training (WRAP++ Mix, 100B Tokens) Figure 4: Per-benchmark score trajectories during mid-training with the WRAP++ Mix (100B to- kens) on OLMo-3-7B. Each subplot tracks one of the 12 evaluation tasks over the course of train- ing. All benchmarks exhibit a clear upward trend, with knowledge-intensive tasks (e.g., HellaSwag, GSM8K, CSQA) showing the most pronounced gains. GADDITIONAL ABLATION: CO-MENTION WITH THREE DOCUMENTS In the main co-mention motif (A→ E ← B while A→ B), we use the two target entities A and B as input to joint QA synthesis. A natural extension is to also include the bridging page E as a third input document, potentially providing additional shared context. We use a dedicated 3-document prompt template that instructs the synthesis model to generate QA requiring facts from all three passages. Surprisingly, including the bridging page E does not improve—and slightly hurts—performance (Table 8). We hypothesize that the bridging page introduces distracting context: since E typically links to many entities, its content is broad and may divert the synthesis model from focusing on the 15 Table 8: Co-mention synthesis using 2 vs. 3 input documents (OLMo-3-7B,∼8B tokens, SimpleQA pass@8). Co-mention VariantSimpleQA pass@8 2-doc (entities A, B only)15.74 3-doc (entities A, B + bridge E)15.42 specific relationship between A and B. The 2-document formulation used in WRAP++ strikes a better balance between relational grounding and synthesis focus. HSYNTHESIZED QA DATA STATISTICS We report comprehensive statistics of the synthesized WRAP++ dataset to characterize the length distributions and data composition. Statistics are computed over the full corpus of 240,658,065 QA instances across 24,224 JSONL files, processed in parallel using 256 workers. Character-level and word-level lengths are measured on the raw synthesized text after extracting the “Question:” and “Answer:” fields from each record. Dataset Composition. Table 9 summarizes the overall dataset composition by relation type. Table 9: Composition of the WRAP++ synthesized QA dataset by relation type. Relation Type QA Instances Proportion Co-mention231,292,95496.1% Dual-link9,365,1113.9% Total240,658,065100.0% Length Distributions. Table 10 reports the distributional statistics of the synthesized QA text. Questions are concise (median 203 characters, 32 words), while answers are substantially longer (median 1,386 characters, 212 words), reflecting the explicit factual chaining constraint that requires step-by-step reasoning before stating the final conclusion. The overall QA length is concentrated in the 1,000–2,000 character range (56.1% of all instances), with 27.6% in the 2,000–5,000 range and 16.0% in the 500–1,000 range. Fewer than 0.3% of instances fall outside the 500–5,000 character window, indicating a well-controlled generation process. Figure 5 visualizes the question and answer length distributions at both character and word levels. 01000200030004000 Length (characters) 0.000 0.002 0.004 0.006 0.008 0.010 Density (Question) Character-Level Length Question (med=203) Answer (med=1,386) 0100200300400500600 Length (words) 0.00 0.02 0.04 0.06 0.08 Density (Question) Word-Level Length Question (med=32) Answer (med=212) 0.0000 0.0001 0.0002 0.0003 0.0004 0.0005 0.0006 Density (Answer) 0.000 0.001 0.002 0.003 0.004 Density (Answer) Question vs. Answer Length Distributions Figure 5: Question vs. answer length distributions at character level (left) and word level (right). Questions are tightly concentrated around a median of 203 characters (32 words), while answers exhibit a broader, right-skewed distribution with a median of 1,386 characters (212 words), reflecting the explicit factual chaining required by the synthesis prompt. 16 Table 10: Length statistics of WRAP++ synthesized QA data (aggregated over all 240.7M in- stances). “Chars” denotes character count; “Words” denotes whitespace-delimited token count. FieldMean Std Min P5 P25 Median P75 P95 Character-level Question2074345143177203233283 Answer1,4246241395639381,3861,8172,463 QA (combined)1,6516372427661,1611,6102,0512,716 Word-level Question33772328323745 Answer217911692149212272364 Comparison Across Relation Types. Table 11 compares the QA length characteristics between the two relation motifs. Dual-link instances produce slightly shorter answers (median 1,210 vs. 1,394 characters), likely because mutual references tend to encode more focused bilateral relation- ships, whereas co-mention pairs often involve broader categorical or analogical connections that require more elaboration. Despite these differences, both subsets maintain similar question lengths and overall distributional shape. Figure 6 overlays the QA length histograms for both relation types, and Figure 7 provides a violin plot comparison across three length dimensions. 010002000300040005000 QA Length (characters) 0.0000 0.0001 0.0002 0.0003 0.0004 0.0005 0.0006 0.0007 Density Synthesized QA Length Distribution by Relation Type Co-mention (n=231.3M, 96.1%) Dual-link (n=9.4M, 3.9%) Figure 6: Synthesized QA length distribution by relation type. Both subsets exhibit a similar uni- modal shape, with the co-mention distribution (blue) shifted slightly rightward relative to dual-link (red), consistent with the longer factual chains required to bridge co-mentioned entities. Table 11: Median QA lengths (characters) by relation type. Word-level medians are shown in paren- theses. Relation TypeQuestionAnswerQA Total Co-mention204 (32 words)1,394 (213 words)1,618 Dual-link197 (32 words)1,210 (189 words)1,428 Overall203 (32 words)1,386 (212 words)1,610 Source Document Lengths. The input Wikipedia passages exhibit substantial length variation. The first passage (texta) has a median length of 4,458 characters (P5–P95: 519–27,963), while the second passage (textb) is generally longer with a median of 9,575 characters (P5–P95: 943– 50,004, where 50,004 indicates truncation at the maximum context window). This asymmetry arises because co-mention pairs order documents by the directed edge A → B, where B (the referenced entity) tends to be a more prominent article. Figure 8 visualizes this distributional asymmetry. 17 Co-mentionDual-link 100 200 300 400 500 Characters Question Length Co-mentionDual-link 0 1000 2000 3000 4000 5000 Characters Answer Length Co-mentionDual-link 0 100 200 300 400 500 600 700 800 Words Answer Word Count Length Comparison: Co-mention vs. Dual-link Co-mentionDual-link Figure 7: Violin plot comparison of question length, answer length, and answer word count between co-mention and dual-link subsets. Black horizontal lines indicate medians. Both motifs produce similarly distributed questions, while co-mention answers are moderately longer, reflecting the ad- ditional elaboration needed to bridge indirectly related entities. 01000020000300004000050000 Document Length (characters) 0.00000 0.00002 0.00004 0.00006 0.00008 0.00010 0.00012 Density Source Document Length Distribution Passage A (med=4690) Passage B (med=9456) Figure 8: Source document length distributions for the two input passages. Passage A (the referenc- ing entity) is typically shorter (median≈4,700 chars), while Passage B (the referenced entity) tends to be longer and more prominent (median ≈9,500 chars), with a visible mass accumulation at the 50K truncation boundary. QA Length Bucketed Distribution. Table 12 and Figure 9 provide a bucketed view of the com- bined QA and answer length distributions. The synthesis process produces a unimodal distribution with the majority of instances in the 1,000–2,000 character range. No instances have empty answers, and fewer than 0.01% of answers are shorter than 200 characters, confirming that the explicit factual chaining constraint effectively prevents degenerate outputs. IQUALITATIVE ANALYSIS Example: WRAP vs. WRAP++ Synthesis. Source entity A: Ludwig G ̈ oransson (Swedish com- poser, film score artist) Source entity B: Oppenheimer (2023 film directed by Christopher Nolan) 18 <200 200 500 500 1K 1K 2K 2K 5K 5K 10K >10K 0 10 20 30 40 50 Percentage (%) 16.0% 56.1% 27.6% QA Total Length (chars) <200 200 500 500 1K 1K 2K 2K 5K 5K 10K >10K 0 10 20 30 40 50 Percentage (%) 2.8% 25.5% 55.0% 16.7% Answer Length (chars) Length Distribution (Bucketed, Full Dataset: 240.7M instances) Figure 9: Bucketed length distributions computed over the full dataset of 240.7M instances. Left: combined QA length; right: answer-only length. The 1K–2K character bucket dominates both dis- tributions (>55%), with a secondary concentration in the 2K–5K range for QA and the 500–1K range for answers. Extreme lengths (>5K or <200) are negligible. Table 12: Bucketed length distributions for the combined QA text and answer text (character-level). Length Range (chars) QA (combined)Answer only Count%Count% [0, 200)00.02,479 <0.01 [200, 500)456,3990.26,771,1772.8 [500, 1,000)38,609,77616.061,411,37725.5 [1,000, 2,000)135,127,24956.1132,242,23555.0 [2,000, 5,000)66,336,55727.640,114,78816.7 [5,000, 10,000)99,207 <0.189,225 <0.1 [10,000,∞)28,877 <0.126,784 <0.1 As shown in Table 13, the WRAP++ output creates richer associative context: it contrasts with a commonly confused entity (Zimmer), provides cross-film comparisons, and generates reverse- direction queries—all contributing to more robust knowledge encoding. 19 Table 13: Qualitative comparison between single-document WRAP and cross-document WRAP++ synthesis for the Ludwig G ̈ oransson and Oppenheimer example. WRAP (single-document QA on entity A)WRAP++ (cross-document QA on entities A+B) Q: Who is Ludwig G ̈ oransson? A: Ludwig G ̈ oransson is a Swedish composer known for film scores. Q: What films has G ̈ oransson scored? A: He has scored Black Panther, Tenet, and Oppenheimer. Q: What films earned Nolan’s original Tenet composer and his replacement their second Oscars? A: Hans Zimmer turned down Tenet for Dune, which earned his second Oscar (after The Lion King). His replacement, Ludwig G ̈ oransson, later won his second Oscar for Oppenheimer (after Black Panther). Therefore, the answer is Dune and Oppenheimer. 20

Instant research discovery