Paper deep dive
Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics
Luca Foppiano
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/23/2026, 1:59:48 AM
Summary
This paper analyzes the 'unit bias' in web-PDF corpus statistics, demonstrating that counting documents versus counting tokens yields vastly different insights due to extreme text mass concentration. Using the CC-MAIN-2021-31-PDF-UNTRUNCATED corpus, the authors show that while long documents are a small fraction of the count, they hold the majority of tokens. The study highlights that Common Crawl's truncation cap (1 MiB, later 5 MiB) disproportionately destroys text content, with significant portions of truncated files being unrecoverable by standard parsers. The authors recommend reporting corpus statistics in both documents and tokens to accurately reflect data quality and loss.
Entities (8)
Relation Signals (7)
CC-MAIN-2021-31-PDF-UNTRUNCATED â hassize â 7.9M documents, 32.6B tokens
confidence 98% · On CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M web PDFs, 32.6B tokens)...
CC-MAIN-2021-31-PDF-UNTRUNCATED â hasginicoefficient â 0.807
confidence 95% · 3.02% of text-bearing documents hold half the tokens (Gini 0.807)
Common Crawl â implements â Truncation Cap
confidence 95% · Common Crawl caps the payload it stores per record.
Truncation Cap â affects â CC-MAIN-2021-31-PDF-UNTRUNCATED
confidence 94% · Common Crawlâs truncation cap: it affected 23.06% of documents and 63.08% of the text.
PyMuPDF â recovers â 11.4% of truncated text
confidence 92% · PyMuPDF recovers 11.4% of the tokens... of that text
PDFium â recovers â 1.4% of truncated text
confidence 92% · PDFium recovers 1.4%... of that text
TeX â produces â PDFs
confidence 90% · The PDFs produced by a TeX toolchain are 1.66% of documents and 4.05% of the text.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:PDF corpora advertise their size in tokens but compute every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) per document, and none decomposes its token total. The two units diverge sharply. On CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M web PDFs, 32.6B tokens), 3.02% of text-bearing documents hold half the tokens (Gini 0.807); documents over 50 pages are 5.00% of the corpus but 53.53% of its text. The PDFs produced by a TeX{} toolchain are 1.66% of documents and 4.05% of the text. The clearest casualty is Common Crawl's truncation cap: it affected 23.06% of documents and 63.08% of the text. Reconstructing the truncated files and extracting both versions, two widely used libraries recover 11.4% and 1.4% of that text; between 72% and 97% of affected documents yield nothing; roughly 55--62% of the corpus's text is lost. Under the 5 MiB cap adopted in March 2025, 30.19% of tokens would still be truncated, and recovery on those documents rises only from 3.3% to 13.2%. We recommend that corpus statistics be reported in both units: documents and tokens.
Tags
Links
- Source: https://arxiv.org/abs/2608.16390v1
- Canonical: https://arxiv.org/abs/2608.16390v1
Trouble viewing inline? Open PDF directly â
Full Text
21,450 characters extracted from source content.
Expand or collapse full text
Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics Luca Foppiano Common Crawl Foundation luca@commoncrawl.org Abstract PDF corpora advertise their size in tokens but compute every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) per document, and none decomposes its to- ken total. The two units diverge sharply. On C-MAIN-2021-31-PDF-UNTRUNCATED(7.9M web PDFs, 32.6B tokens), 3.02% of text- bearing documents hold half the tokens (Gini 0.807); documents over 50 pages are 5.00% of the corpus but 53.53% of its text. The PDFs produced by a T E X toolchain are 1.66% of doc- uments and 4.05% of the text. The clearest casualty is Common Crawlâs truncation cap: it affected 23.06% of documents and 63.08% of the text. Reconstructing the truncated files and extracting both versions, two widely used libraries recover 11.4% and 1.4% of that text; between 72% and 97% of affected documents yield nothing; roughly 55â62% of the corpusâs text is lost. Under the 5 MiB cap adopted in March 2025, 30.19% of tokens would still be truncated, and recovery on those documents rises only from 3.3% to 13.2%. We recom- mend that corpus statistics be reported in both units: documents and tokens. 1 Introduction PDFs have become a first-class source of pretrain- ing text (Poznanski et al., 2025; Montalvo and Wightman, 2024). In one current open data pool, PDF-derived text amounts to roughly 4.3T tokens. 1 FinePDFs alone is described as âabout 3 trillion tokens across 475 million documentsâ (KydlĂ Ë cek et al., 2025). Those corpora are not naive about tokens: they headline them. What none of them does is com- pute a rate in tokens, or break a token total down. FinePDFs routes 368.8M of 1.29B files to OCR and 1 Marinâsdatakitsource registry: FinePDFs 1,186.5B English plus 1,353.4B across 18 non-English subsets, and dolma4pdfs1,804.0B over 137,132,279 documents.https: //github.com/marin-community/marin recovers 53.5% of its truncated files by re-fetching; CCpdf reports per-document success rates (Turski et al., 2023); PDFA gives three totals (2,159,432 documents, 18M pages, 9.7B tokens) but no drop rates in any unit (Montalvo and Wightman, 2024). No PDF corpus we are aware of decomposes its token total by length or quality. This would be harmless if documents were in- terchangeable. They are not. The web-text com- munity already recognises this: Nemotron-C re- ports a token-weighted yield, â+57.4% more high- quality tokensâ (Su et al., 2025), and PDFs have far more extreme length skew than HTML. FinePDFs itself observes that its documents have a ââŒ5.3k character median (about 2Ăother corpora)â with a â95th percentileâŒ68kâ against ââŒ11â13k else- where,â and then reports every rate per document anyway. We measure what those rates become when weighted by text. Our contributions: 1.the first token-weighted characterisation of a web-PDF corpus, with a direct document- versus-token comparison of every headline statistic (§4.1â4.3); 2.a concentration result: text mass in web PDFs is far more skewed than document counts sug- gest (Gini 0.807); 3.the first measurement of what Common Crawlâs truncation cap costs in text: 23.06% of documents but 63.08% of tokens, at most 11.4% of it recoverable, and 30.19% still lost at the new 5 MiB limit (§4.4â4.6); 4. released code. 2 Related Work Totals in tokens, rates in documents.FinePDFs (KydlĂ Ë cek et al., 2025) routes documents between a text path (Docling) and a GPU OCR path with a learned classifier reporting F1 0.71 on the OCR Preprint. Under review. arXiv:2608.16390v1 [cs.CL] 17 Aug 2026 class, and quantifies every stage in files. CCpdf (Turski et al., 2023) tabulates ânumber of docu- ments per processing step and languageâ and re- ports no token count for its own corpus. PDFA (Montalvo and Wightman, 2024), derived from the same corpus we study, filters per document, dis- carding files over 100 MB or slower than 500 ms to render, i.e. selecting on size, and publishes no drop rates. GovScape (Huang et al., 2026) notes that âfor some pages, no corresponding text representa- tion is found embedded within the PDF. . . these pages are currently excludedâ without quantifying the exclusion in any unit. Extraction benchmarks (Ouyang et al., 2025; Poznanski et al., 2025) mea- sure fidelity on documents that parsed and report no coverage at all. Truncation is named but never quantified. Common Crawl caps the payload it stores per record.PDFs are the format worst affected: for the crawlCC-MAIN-2023-06they were 0.8% of successfully fetched records but 11.85% of 88 TiB of WARC storage (PDF Association, 2023). CCpdf stated the consequence plainly in 2023: the crawlerâs 1 MB cap is âquite high for HTML pages, but unfortunately rather low for PDF filesâ (Turski et al., 2023). Their workaround was to re-download the truncated documents from origin; to our knowl- edge nobody since has quantified the effect. FromCC-MAIN-2025-13(March 2025) the cap rose from 1 MiB to 5 MiB, reported as +13% fetched content (403 TiBâ455 TiB) (Common Crawl Foundation, 2025). The Common Crawl Foundation reported the ef- fect of the change on truncation by MIME type: PDFs fell from 25.7% to 6.8%, against 2.25% to 0.14% for all types and 2.2% to 0.04% for HTML. Even after the change PDFs are truncated at roughly 49Ăthe rate of content generally. What is not reported, before or after, is what that costs in text. 3 Data and Method Corpus.C-MAIN-2021-31-PDF-UNTRUNCATED (Digital Corpora et al., 2021) contains every PDF found in Common CrawlCC-MAIN-2021-31, with payloads that Common Crawl truncated refetched whole from origin. It ships five metadata tables covering 8.3M URLs. Becausefile_nameis the post-SHA-256 identity, we deduplicate to one row per file, giving 7,932,654 unique documents and 32,570,135,761 tokens. Shapedoc %tok %ratio report/thesis (>50 p)4.4249.7111.24Ă article-shaped (4â30 p)31.6426.870.85Ă long (31â50 p)3.219.082.83Ă landscape (slides)14.087.760.55Ă 2â3 p21.203.760.18Ă 1 page (flyer/form)24.962.700.11Ă Table 1: The same corpus counted two ways. Cat- egories are mutually exclusive and orientation takes precedence over page count, so a long landscape doc- ument is counted as slides; the row therefore covers portrait documents only. Counting purely by page count, documents over 50 pages are 5.00% of the cor- pus and 53.53% of its text; 0.49% of documents with unparseable page metadata are omitted. Theunit. Textmassistakenfrom tika_eval_num_tokens,a token count al- ready published as part of the corpus metadata (Digital Corpora et al., 2021) (Apache Tika 2.8.0, tesseract disabled), a whitespace/ICU count rather than an LLM tokenizerâs. We claim only proportions, which is all the argument requires. The field is right-censored at 10,000,000, but only two documents in the corpus reach that cap. Truncation. Theprovenancetable recordscc_truncated,fetched_status, fetched_lengthand the WARC byte range per file (c_truncated=âlengthâcoincides exactly withfetched_status=âREFETCHED_SUCCESSâ). For truncated records the WARC record length pins toâŒ1.049 MB, so the cap is directly observed, and the re-fetched originals give the true size of every document Common Crawl stored only as a fragment: a pairing no other public corpus offers. 4 Results 4.1 Text mass is extremely concentrated Table 1 gives the page-count distribution in both units. Counting by page count alone regardless of orientation, documents over 50 pages are 5.00% of the corpus and 53.53% of its text: a document in that tail carries about eleven times an average doc- umentâs text. We use this threshold for the concen- tration claim here and in §4.2. The mirror image: the 46.2% of documents with three pages or fewer contribute 6.46% of the text. Over the 7,292,093 text-bearing documents, 3.02% hold half the to- kens and 15.54% hold 80%; the Gini coefficient of tokens across documents is 0.807. A corpus described by document count as docs %tokens % Truncated by C (1 MiB)23.0663.08 still truncated at 5 MiB5.9430.19 Table 2: Common Crawlâs truncation cap, priced in both units, over the 7,932,878 files with provenance records. âmostly flyers and formsâ is, by text, a corpus of long documents. Both descriptions are arithmeti- cally correct. 4.2The concentration is not an artifact of one extractor Since the skew could be a property of the extractor rather than of the corpus, we validate the distribu- tion against page counts, which the corpus authors extracted with Poppler, an independent tool mea- suring an entirely different quantity. Page counts reproduce the same concentration: the Gini coeffi- cient of pages across documents is 0.767 against 0.807 for tokens, 3.98% of documents hold half the pages against 3.02% for tokens, and documents over 50 pages account for 54.27% of all pages against 53.53% of all tokens, a difference of 0.74 percentage points. Two tools measuring two dif- ferent quantities give the same answer: a shared extraction bias cannot produce this agreement. 4.3 The scholarly share more than doubles Classifying theproducer/creatorstrings into toolchain families, a T E X toolchain accounts for 1.66% of documents but 4.05% of tokens (2.43Ă). PowerPoint runs the other way, 2.24% of docu- ments against 0.82% of tokens (0.37Ă), and Mi- crosoft Word is 24.95% of documents against 19.37% of tokens. The conclusion that this corpus is not primarily scholarly can be inferred using both units, but its magnitude is off by a factor of 2.4 in the unit a language model consumes. 4.4 Truncation: 23% of documents, 63% of the text Common Crawl truncated 1,829,061 of this cor- pusâs documents at the 1 MiB cap in force at the time, corresponding to 23.06% of them. Those doc- uments hold 63.08% of the corpusâs text (Table 2). Truncated documents carry 2.74Ă the mean token mass; their median true size is 2.62 MB against 190 KB for the rest. Applying the current 5 MiB cap to the corpus, 5.94% of documents and 30.19% of tokens would still be truncated. The March 2025 change there- fore recovers roughly half of the exposure and leaves the other half in place. Our document- level counterfactual is consistent with Common Crawlâs own post-change measurement of 6.8% on 2025 crawls 2 , computed on a population five years younger; the token figure has no published counterpart. In bytes, the 1 MiB cap kept 1.738 TiB of these filesâ 9.449 TiB total, discarding 81.6%; storing them whole would have cost 7.712 TiB of addi- tional archive. The 5 MiB cap lands between the two: it would keep 5.106 TiB (46.0% discarded), so the March 2025 change spends 3.368 TiB of that 7.712 TiB and leaves 4.344 TiB behind. At either cap, byte loss overstates text loss: 81.6% of bytes against 63.08% of tokens at 1 MiB, 46.0% against 30.19% at 5 MiB, because PDF bytes are largely images and embedded fonts. The byte figure prices the archive; the token figure prices the corpus. 4.5 Truncation destroys text rather than trimming it To measure what is actually lost in truncation we reconstruct what Common Crawl held, cut- ting each of this corpusâs 1,829,061 truncated documents at 1 MiB, and extracting both ver- sions. Whether a fragment is recoverable may be a property of the parser rather than of the file, so we run two independent engines, PyMuPDF (MuPDF) (Artifex Software, Inc., 2025) and PDFium (Chromiumâs) (pypdfium2 team, 2025), and report on the 1,225,130 documents both tools successfully processed, and whose whole version yields text. PyMuPDF recovers 11.4% of the tokens and 20.7% of the pages; PDFium recovers 1.4% and 1.9%, a gap of 8.4Ăon identical input. The two agree to within 1.7% on the intact versions of those same documents, so the divergence is specific to damaged input and is not a general difference in extraction quality. There is therefore no single ârecovery rateâ for a truncated PDF: the quantity is not defined until the extractor is named. The shape of the failure differs as sharply as its size, and both shapes are invisible to a pipeline that counts successful parses. PyMuPDF opens 98.9% of truncated files, because MuPDF rebuilds a missing cross-reference table, but the page con- tent streams lie beyond the cut, so the document 2 https://commoncrawl.org/blog/march-2025-cra wl-archive-now-available opens, reports a page count, and returns no text: 72.4% yield nothing at all. PDFium attempts no such reconstruction and refuses the file outright, failing to open 96.7% of the same documents; its zero-yield rate of 96.8% is almost exactly its open- failure rate. One extractor logs these as successes with empty output, the other as parse errors. The gap is not spread evenly across the corpus. A linearized PDF (Adobe âFast Web Viewâ) places its first page and a cross-reference table at the front of the file so that a partial copy is still renderable; 40.7% of the documents here are linearized. Under PyMuPDF those files recover 1.1% of their tokens against 18.5% for the non-linearized rest, while un- der PDFium it is 1.1% against 1.5%. Note that the gap persists when files of similar size are compared, so it is not a size effect. The difference between the two engines therefore lives in the non-linearized population, and the Fast Web View structure, de- signed to keep partial files usable, is the one from which least is recovered. Applying the recovered fractions to the exposure in Table 2, roughly 55% to 62% of this corpusâs total text is destroyed by the 1 MiB cap, the range spanning the two extractors. That product multi- plies an exposure counted in Tika tokens by a re- covery counted in whitespace tokens. Since every truncated document has both counts, we re-weight each by its Tika count instead: the corpus figure moves by 0.42 points under PyMuPDF and by zero under PDFium. The two counts agree closely per document (median ratio 1.01â1.02) and diverge only for scripts without whitespace word separa- tors: a property of the corpus, not of truncation. One caveat does remain: we count tokens rather than reading them, and a token count cannot distin- guish text that a repair path recovered cleanly from reconstruction artefact, so ârecovers moreâ must not be read as ârecovers betterâ. 4.6 The 5 MiB cap recovers little of what it still exposes Raising the cap to 5 MiB (Table 2) leaves 30.19% of tokens exposed; we tested whether they are recoverable, since every document truncated at 5 MiB is also truncated at 1 MiB and only the cut point moves between the runs. On the 316,174 documents PyMuPDF processed at both caps, recovery rises from 3.3% to 13.2% of tokens, and 21,203 documents (6.7%) go from yielding no text at all to yielding some. PDFium moves from 0.1% to 2.4% on the same documents. A five-fold larger prefix therefore multiplies recov- ered text roughly four-fold, but still leaves 77.6% of these documents (97.0% under PDFium) yielding no text. The change halves the exposure; what it leaves behind remains almost entirely unreadable. The benefit is also concentrated immediately above the cap (Table 3, Appendix A): 74% of the rescued documents lie between 5 and 10 MiB, while above 25 MiB the larger cap is worth 2â3 percentage points. This is what a fixed prefix must do, 5 MiB is half of a 10 MiB file and a twentieth of a 100 MiB one, and it holds under both engines. Raising the cap further has sharply diminishing re- turns per additional TiB of archive: the documents that dominate the remaining token mass are those a larger fixed prefix helps least. Both effects reflect the cut rather than the run: to- ken counts from the whole documents agree exactly for all but 0.036% of paired documents under both engines (a shared, harness-imposed deadline). Re- covery is not monotone, however: 800 documents (0.25%) yield text at 1 MiB and none at 5 MiB, so repair is sensitive to where a file stops, but the effect is 26Ărarer than the reverse and does not disturb the aggregate. 5 Conclusion Counted in documents and counted in text, the stud- ied corpusCC-MAIN-2021-31-PDF-UNTRUNCATED is two different corpora: the units diverge by up to 11Ăper category, and Common Crawlâs cap cost 23% of documents but 55â62% of the text. Corpus statistics should therefore be reported in both units; every rate in this literature (coverage, filter drop, OCR routing) is per-document or unreported. Size- based filters must be priced in text, because size is where the text is: PDFAâs 100 MB and 500 ms cuts, FinePDFsâ router, and the cap itself all se- lect on it. For Common Crawl, §4.4â4.6 price the 1 MiBâ5 MiB change: it halves the exposure but recovers little of what it still truncates; the remain- ing exposure calls for a size-aware fetch policy for large PDFs, not a higher cap. Limitations Every figure describing corpus composition and truncation exposure is computed over all 7.9M doc- uments from the shipped metadata. The recov- ery figures cover the 1.2M truncated documents both extractors successfully processed, 67% of the 1.83M Common Crawl truncated. The shortfall is concentrated in ten of our sixty-four PyMuPDF shards, which produced nothing because of techni- cal errors in the extractor (e.g. out of memory, hang- ing); since shards partition the corpus by archive index, the loss is arbitrary with respect to docu- ment content, and per-shard recovery rates vary by about a percentage point. It is nonetheless not a random sample, and we report no confidence inter- vals. The metadata is third-party and derived (Dig- ital Corpora et al., 2021), andproducer/creator strings are self-reported with 8.08% unclassified, so the T E X share is a lower bound in both units. page_sizeis recorded for the first page only, mak- ing the landscape category approximate. We study a single crawl snapshot, from before the policy change, and the only one for which re-fetched orig- inals exist; the 5 MiB figure is therefore a counter- factual computed on 2021 file sizes rather than a measurement of a current crawl. Finally, the two extractors we compare are both text-layer parsers; an OCR pipeline would fail differently again, and we do not measure one. Acknowledgement We thank DFKI (Deutsches Forschungszentrum fĂŒr KĂŒnstliche Intelligenz GmbH) for supporting this work with computation and storage. Data and Code availability The data was collected from the resource provided by Digital Corpora et al. (2021). The code is avail- able athttps://github.com/lfoppiano/c-w acky-pdf. References Artifex Software, Inc. 2025. PyMuPDF. Version 1.28.2; Python bindings for MuPDF. Common Crawl Foundation. 2025. March 2025 crawl archive now available.https://commoncrawl.or g/blog/march-2025-crawl-archive-now-avail able. Truncation limit raised from 1 MiB to 5 MiB at C-MAIN-2025-13. Digital Corpora, NASA JPL, and DARPA SafeDocs. 2021. C-MAIN-2021-31-PDF-UNTRUNCATED (SAFEDOCS).https://digitalcorpora.org/c orpora/file-corpora/c-main-2021-31-pdf-u ntruncated/. Ying-Hsiang Huang, Claire Gong, Shreya Shaji, Alison Yan, Leslie Harka, Albert Du, Anjali Gopal, Samuel J Klein, Shannon Zejiang Shen, Mark Phillips, Trevor Owens, Kyle Deeds, and Benjamin Charles Germain Lee. 2026. Govscape: A public multimodal search system for 70 million pages of government pdfs. Preprint, arXiv:2511.11010. Hynek KydlĂ Ë cek, Guilherme Penedo, and Leandro Von Werra. 2025. Finepdfs: Liberating 3t of the finest tokens from pdfs. Pablo Montalvo and Ross Wightman. 2024. PDFA: pdfa-eng-wds.https://huggingface.co/datas ets/pixparse/pdfa-eng-wds. Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, et al. 2025.Om- nidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24838â24848. IEEE. PDF Association. 2023. New large-scale pdf corpus now publicly available.https://pdfa.org/new -large-scale-pdf-corpus-now-publicly-ava ilable. Reporting WARC storage and truncation shares for C-MAIN-2023-06. Jake Poznanski, Aman Rangapur, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. 2025. olmocr: Unlocking trillions of tokens in pdfs with vision lan- guage models. arXiv preprint arXiv:2502.18443. pypdfium2 team. 2025. pypdfium2. Version 5.12.1; Python bindings for PDFium. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Nemotron-c: Transforming common crawl into a refined long-horizon pretraining dataset. In Proceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 2459â2475. MichaĆ Turski, Tomasz StanisĆawek, Karol Kaczmarek, PaweĆ Dyda, and Filip Grali Ì nski. 2023. CCpdf: Building a high quality corpus for visually rich docu- ments from web crawl data. In Document Analysis and Recognition â ICDAR 2023. ArXiv:2304.14953. A Recovery by document size Table 3 breaks the paired recovery measurement of §4.6 down by document size. recovered sizedocs1 MiB5 MiBgain 5â10 MiB179,8104.88%22.44%+17.6 10â25 MiB99,8312.62%8.97%+6.4 25â50 MiB25,7001.64%4.47%+2.8 50â100 MiB8,6831.36%3.28%+1.9 >100 MiB2,1501.01%2.96%+2.0 Table 3: What raising the cap buys, by document size (PyMuPDF, gain in percentage points). The same 316,174 documents are cut at both points; the benefit is concentrated immediately above the cap.