Paper deep dive
SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking
Michael J. Bommarito
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/4/2026, 2:59:03 AM
Summary
The paper introduces SHELF (Synthetic Harness for Evaluating LLM Fitness), a Python-based system for generating controlled bibliographic benchmark data using labelled taxonomies and writing specifications. The dataset contains 62,899 model-written documents based on Library of Congress vocabularies, designed to test classification, clustering, retrieval, and pair classification tasks. The study evaluates various methods including TF, TF-IDF, BM25, and dense encoders, finding that subject classification performs well (0.8887 F1) while genre-form classification and clustering remain challenging. SHELF allows for independent variation of bibliographic facets and generation of post-cutoff documents to prevent contamination, though it notes that scores do not directly estimate production accuracy.
Entities (8)
Relation Signals (7)
Michael J. Bommarito II → authored → SHELF
confidence 95% · SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking Michael J. Bommarito II
SHELF → evaluates → TF-IDF
confidence 95% · We compare TF, TF-IDF, BM25, popular encoders... each method appears only on tasks that support it.
SHELF → evaluates → BM25
confidence 95% · We compare TF, TF-IDF, BM25, popular encoders... each method appears only on tasks that support it.
SHELF → uses → Library of Congress
confidence 95% · This first release contains 62,899 model-written documents based on Library of Congress vocabularies
SHELF → compareswith → LCSHBench
confidence 90% · Comparisons with LCSHBench and Project Gutenberg show that model rankings transfer more reliably than absolute scores
SHELF → compareswith → Project Gutenberg
confidence 90% · Comparisons with LCSHBench and Project Gutenberg show that model rankings transfer more reliably than absolute scores
SHELF → iscomparedto → MTEB
confidence 85% · The Massive Text Embedding Benchmark (MTEB) [18]... does not include bibliographic tasks. Here enters SHELF...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Libraries and archives manage large collections with limited staff and computing budgets, yet common benchmarks do not systematically test their bibliographic work. They need to know which methods work for their tasks and what those methods require to run. SHELF, the Synthetic Harness for Evaluating LLM Fitness, addresses this gap. It is a Python system that turns labelled taxonomies, writing specifications, and a generation budget into controlled benchmark data and evaluation tasks. This first release contains 62,899 model-written documents based on Library of Congress vocabularies, with tasks for classification, clustering, retrieval, pair classification, and instruction retrieval. We compare TF, TF-IDF, BM25, popular encoders, and, on subject classification only, zero-shot decoders; each method appears only on tasks that support it. Subject classification reaches 0.8887, while genre-form classification reaches only 0.2605, and several pair and clustering tasks remain near chance. Sparse methods remain competitive on classification, while TF-IDF is the fastest measured arm in the subject timing experiment. SHELF also varies bibliographic facets independently and can generate new, verifiably unseen documents after a model's training cutoff. Comparisons with LCSHBench and Project Gutenberg show that model rankings transfer more reliably than absolute scores, but SHELF scores do not estimate accuracy on production catalogue data. We release all source code and data under permissive licenses on GitHub and Hugging Face.
Tags
Links
- Source: https://arxiv.org/abs/2609.03047v1
- Canonical: https://arxiv.org/abs/2609.03047v1
Trouble viewing inline? Open PDF directly →
Full Text
72,091 characters extracted from source content.
Expand or collapse full text
SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking Michael J. Bommarito I 1 michael.bommarito@gmail.com September 4, 2026 Abstract Libraries and archives manage large collections with limited staff and computing budgets, yet common benchmarks do not systematically test their bibliographic work. They need to know which methods work for their tasks and what those methods require to run. SHELF, the Synthetic Harness for Evaluating LLM Fitness, addresses this gap. It is a Python system that turns labelled taxonomies, writing specifications, and a generation budget into controlled benchmark data and evaluation tasks. This first release contains 62,899 model-written documents based on Library of Congress vocabularies, with tasks for classification, clustering, retrieval, pair classification, and instruction retrieval. We compare TF, TF-IDF, BM25, popular encoders, and, on subject classification only, zero-shot decoders; each method appears only on tasks that support it. Subject classification reaches 0.8887, while genre-form classification reaches only 0.2605, and several pair and clustering tasks remain near chance. Sparse methods remain competitive on classification, while TF-IDF is the fastest measured arm in the subject timing experiment. SHELF also varies bibliographic facets independently and can generate new, verifiably unseen documents after a model’s training cutoff. Comparisons with LCSHBench and Project Gutenberg show that model rankings transfer more reliably than absolute scores, but SHELF scores do not estimate accuracy on production catalogue data. We release all source code and data under permissive licenses on GitHub and Hugging Face. 1 Introduction Benchmarks should help users choose a model for their tasks, but that choice becomes difficult when no bench- mark covers their work or when a leaderboard score does not predict performance under their constraints. Libraries and archives face both of these issues. They need to know which methods can classify, cluster, retrieve, and pair documents by subject, genre, geography, audi- ence, or register, and what each method requires to run. And they need to do this across large collections, typically with limited staff and computing budgets. Yet common benchmarks do not systematically test bibliographic work, so libraries lack evidence that is both task-relevant and cost-aware. At the same time, bibliographic tasks are useful mea- sures of general fitness. A model that cannot classify, cluster, retrieve, or pair documents by subject, genre, ge- ography, audience, or register likely has practical limita- tions that an aggregate score may hide. In particular, a model that cannot distinguish what a document is about from how it is written is likely to make serious mistakes when drafting or summarizing. While success on these 1 Portions of this work were prepared with assistance from large lan- guage models. The author is solely responsible for all content, including any errors or omissions. tasks does not establish general fitness, failure can identify limits that matter well beyond bibliographic work. The Massive Text Embedding Benchmark (MTEB) [18], the best-known benchmark in this area, measures model performance broadly, yet it does not include bibliographic tasks. Across all 1,451 definitions inmteb2.20.4, there is no Library of Congress, Dewey, MARC, or cataloguing task. 2 Here enters SHELF, the Synthetic Harness for Evaluat- ing LLM Fitness. SHELF generates bibliographic bench- mark data when real labelled text is scarce, expensive, or contested. The software takes a labelled taxonomy, writ- ing specifications, and a generation budget, then writes labelled documents and builds evaluation tasks. The current release includes one dataset made with this system. It uses Library of Congress vocabularies and varies subject, genre, audience, register, geography, and length. We ask which sparse methods, encoders, and decoders can support the resulting bibliographic tasks, although not every method supports every task. Every benchmark score in this paper comes from this released dataset. The only exceptions are the external validation 2 Checked by searching the name, description, type, and domains of every task definition on 29 August 2026. Two names mislead. LccSentimentClassificationuses the Leipzig Corpora Collection, not the Library of Congress classification, and the one task matching “cata- logue” retrieves beverage marketing documents. 1 arXiv:2609.03047v1 [cs.CL] 2 Sep 2026 experiments in Section 6, which show why we do not apply its scores to other datasets. Library cataloguing makes this a harder problem, but also a real one. SHELF uses 21 broad Library of Congress Classification classes and 133 genre and form labels. Cat- aloguers do not always assign the same subject headings to the same work, as the LCSHBench concordance study later quantifies [27]. Two projects have begun to fill that gap with real cata- logue records [8,27]. Those records carry a further prob- lem, though: what a cataloguer assigned is bound up with who catalogued it, when, and in what language. Genre and subject also move together in natural collections, so a corpus of real records cannot tell you whether a model reads the subject or reads the register. SHELF therefore generates documents instead of col- lecting existing catalogue records. Each document is writ- ten to a specified subject, genre, audience, register, and other bibliographic facets. This control lets us vary facets separately, even when they often occur together in nat- ural collections. SHELF also varies the writing model, prompt style, and sampling settings, so the source of each document is measured rather than hidden. The released dataset contains 62,899 documents from 25 models and records how each document was gener- ated. Its factorial subset contains 18,345 documents from 15 models and four prompt styles, covering all 60 model– prompt pairings. Temperature ranges from 0.6 to 1.2, and nucleus-sampling probability ranges from 0.85 to 1.0. Re- searchers can use this subset when an analysis requires tighter control over the writing model and bibliographic facets. Section 3 describes the subset and the split proto- col. All released SHELF documents were generated in or after November 2025, and every embedding checkpoint evaluated here predates them, so the documents are out of sample for the embedding panel. This claim does not cover the later decoder models in Section 4.6. Future models may be trained on this public release, but SHELF can then generate another dataset after those models’ training cutoffs while preserving the same labels and task design. The new documents will be verifiably unseen, although generation after a cutoff does not remove source bias. The current evaluation turns those controlled docu- ments into five kinds of task: classification, clustering, retrieval, pair classification, and instruction retrieval. To- gether, they cover subject, topic, genre, audience, register, and geography. Table 2 lists every task variant, metric, and evaluated method family, while the 25 configurations span sparse methods, early sentence encoders, and recent small and mid-size embedding models. The results separate easy labels from hard ones. Subject classification reaches 0.8887, while classification across 133 genre forms reaches 0.2605. Subject retrieval reaches 0.7104, but genre-form retrieval reaches 0.1173. Register and audience pairs remain near chance. Sparse methods compete with dense encoders in classification and trail them in retrieval. The separate tasks show which formu- lations and label spaces current embeddings can use. Machine-written data still require an external check, so we compare the subject tasks with Project Gutenberg passages and LCSHBench catalogue records. SHELF pro- duces similar model rankings, but absolute scores do not transfer between the corpora. We use this result to bound the benchmark claim, not to define the benchmark: the re- leased dataset supports model comparison and diagnosis, not estimates of production accuracy. In sum, this paper makes five contributions: 1. A system that turns a labelled taxonomy, controlled writing specifications, and a generation budget into bibliographic benchmarking data and evaluation tasks. The released code can write a fresh dataset after a future training cutoff (Section 3). 2.An evaluation dataset of 62,899 documents from 25 models. Of these, 20,367 are specification-backed, including an 18,345-document generator-balanced factorial component, so the writing model is a re- ported axis rather than a nuisance (Section 3). 3.An evaluation spanning five task formulations and six bibliographic facets, with stored results for 25 model configurations (Sections 4 and 4.4). 4.Measurements of task difficulty, label leakage, model headroom, and failed task designs (Sections 4.5, 5 and 5.2). 5.A bounded external check of subject scores and model rankings against two natural corpora (Sec- tions 6 and 6.2). 2 Related work 2.1 Automated subject cataloguing Predicting a classification from a subject heading is not a new task; Frank and Paynter described it in 2004 [12]. An- nif provides an open-source system for automated subject indexing, and SemEval-2025 Task 5 evaluated subject tag- ging over roughly 100,000 English and German catalogue records [8]. The nearest neighbour to this paper is LCSHBench [27], released in June 2026: 22,346 books in 15 languages from the Harvard, Columbia, and Princeton catalogues, admit- ted only when two agencies agreed. Its concordance study over 465,187 works is the strongest published statement 2 of how far cataloguers disagree — 93.3% share a concept- level heading, but only 39.4% assign identical sets. That line of work assigns headings to real records at ex- treme label scale. We measure how well a representation separates classes under a design that fixes the confounds. The questions differ, so LCSHBench is useful here as a test corpus rather than as a competitor. We use it that way in Sections 6 and 6.2. 2.2 Embedding benchmarks MTEB [18] and the Benchmarking Information Retrieval collection (BEIR) [29] have shaped the field. The Massive Multilingual Text Embedding Benchmark (MMTEB) [11] extended it to more than 500 tasks and more than 250 languages. HUME measured human performance on 16 MTEB datasets and found only a small gap between the human average and the best model [9]. The Embedder’s Dilemma compares 26 embedding models from 118M to 14B parameters with ten language models and finds that the best aggregate scores are close, while cost and task- level strengths differ sharply [10]. Our smaller encoder set does not test that frontier, a limit stated in Section 7. 2.3 Domain-specific benchmarks FinMTEB [28] finds that general rank does not reliably predict finance-task rank. CaseHOLD [42] likewise builds a domain task because existing legal tasks did not expose the intended gap. Raji and colleagues [20] warn against benchmarks that claim to measure broad constructs. Bean and colleagues [1] review 445 benchmarks and find recurring weaknesses in construct validity. Bowman and Dahl [5] include reli- able annotation among their criteria. SHELF does not yet meet that criterion, as we acknowledge in Section 7. 2.4 Model-generated evaluation data Generating a corpus for evaluation is less common than generating training examples. E5-Mistral [32] trained on hundreds of thousands of model-written examples and achieved strong benchmark results without labelled data. PhantomWiki [14] generates a fresh, factually con- sistent corpus on demand rather than using a fixed dataset. YourBench [24] takes the opposite route and generates evaluation sets from a user’s own documents with lan- guage models, which makes the corpus cheap to refresh and leaves its label semantics dependent on the generator. Gill and colleagues [13] provide a direct warning. They wrote model versions of two reading-comprehension datasets and found them valid by the annotation guide- lines yet easier for models than the human-written origi- nals. They call for direct tests of the practice. Section 6 provides one, and its transfer results support part of their criticism. A second objection is source bias: neural retrievers rank model-written text above equivalent human text, by a wide margin [6]. Our corpus is written by machine throughout, so source bias may affect the reported model rankings by an unknown amount. We return to this in Section 7. Majurski and Matuszek [17] show that written bench- marks grounded in real documents can reproduce rank- ings from human-curated benchmarks. Section 6.2 tests that claim here. 2.5 Contamination Synthetic generation is often defended as a guard against contamination, and that defence is weaker than it looks. GSM1k [40] showed accuracy drops of up to eight points on a fresh test set built to match an old one. Work on rewritten samples [38] showed that a model trained on such a test set stays undetectable by n-gram overlap, which defeats the usual decontamination check. We claim only what timing buys. The released SHELF documents postdate the evaluated embedding check- points, and the system can generate new records after a future model’s training cutoff. These properties protect against direct overlap, but neither removes source bias. The unlikely combinations in the factorial design do some- thing different: they test whether a method can separate facets that usually move together. 2.6 Instruction-conditioned retrieval Instruction-following retrieval is covered by FollowIR [36] and NevIR [35]. FollowIR finds that existing retrieval mod- els often fail to use detailed instructions; many instruction- aware retrieval models score below zero on its paired measure. NevIR finds bi-encoder and sparse systems at or below random performance. CoDeR [39] measures the fraction of queries whose top-kresults contain evidence that vio- lates a stated constraint. Together, this work leaves a clear gap. Cataloguing benchmarks use real records at large label scale, while general embedding benchmarks cover many tasks but not bibliographic work. Model-generated benchmarks show what controlled data can provide, but also why source bias and contamination claims require care. SHELF joins these lines without replacing them. It uses controlled generation to separate bibliographic facets, compares sparse methods and encoders across compatible tasks, and adds a subject- only decoder comparison. It then checks subject rankings against natural corpora, so those comparisons bound what the synthetic scores can mean. That combination—control 3 inside SHELF and calibration outside it—is the basis of the evaluation that follows. 3 Corpus and evaluation protocol 3.1 Corpus composition and splits Theallsubset released on Hugging Face contains 62,899 documents and is the corpus used for the main classi- fication, retrieval, clustering, and headroom results. It combines 42,532 documents generated in late 2025 with 20,367 documents generated in August 2026. The two components share the same principal label spaces but dif- fer in provenance. The original records predate stored specifications and retain their stratified document-level 60/20/20 splits. The later records carry specifications and use stratified 60/20/20 splits grouped byspec_id. The pooled configuration preserves each document’s source split. Consequently, the specification-level leakage guar- antee applies to the later component, not to the pooled corpus as a whole. All reported runs use seed 42 and body text only. The generator-balancedv0_4_corecomponent con- tains 18,345 of the later documents. Claims that depend on factorial control or generator balance name that compo- nent explicitly; claims reported onalluse the full 62,899- document corpus. Each such document begins as a specification: a subject class, a genre and form, a set of topics, an audience, a register, a geographic region, and a target length. The specification is hashed, and that hash is the document’s spec_id. Two properties follow from this. Every writing model receives the same specification, so a difference between models is a difference in writing rather than in assignment. And every document that shares aspec_idis a separate version of one brief, so those versions can be kept together when the corpus is split. Splitting at the document level puts near-duplicate ver- sions of the same brief on both sides of the split. In a pilot corpus of 600 specifications, each written by eight generators, 598 specifications straddled a split that way. Grouping on spec_id leaves none straddling. 3.2 Independence by construction The facets are drawn independently. A natural corpus can- not guarantee this, because subject, genre, and register are typically correlated. SHELF instead contains combi- nations unlikely to occur in most collections: a techni- cal puzzle set about agricultural conservation and quan- tum mechanics, professional sermons about networks and biotechnology, and casual maps about ecology and geol- SliceDocs Mean Median SD Min 5th 95th Max SHELF, all62,899612326 782134 2,369 6,203 SHELF, v0_4_core18,345548341 685532 1,916 4,451 Gutenberg transfer3,01648044992 184 401671896 LCSHBench transfer4,92414095 146219392 3,388 Table 1: Words per document, counted on whitespace over the body text of every document in each slice. 5th and 95th are percentiles. ogy. These are real examples from the generator-balanced slice v0_4_core, not invented illustrations. That slice is also balanced across writing models. The measured association between writing model and label is negligible: Cramér’sV = 0.016for subject class and 0.027for genre category. Across the wider v0.4 genera- tion, which adds single-generator supplementary docu- ments, subject-class association rises to0.037, and any claim that requires independent generation should use thev0_4_coresubset. The implication is that if genre and subject usually move together, a model that reads only the register will score well on a subject task, and the benchmark cannot tell which it read. 3.3 Scale and generation models The balanced component holds documents from 15 current-generation models across 11 laboratories — An- thropic, OpenAI, Google, Alibaba, DeepSeek, Zhipu, Moonshot, MiniMax, Meta, Mistral, and xAI — with the largest single model at 9.24%. For comparison, E5-Mistral used one model and Cosmopedia [2] used one model. The aggregate corpus is not generator-balanced: its largest writing model supplies 47.7% of the documents. Thus scale describes the aggregate corpus, while gener- ator balance describes the factorial component. We name the component when a result depends on that balance. A third slice,transfer_gutenberg, holds 3,016 Project Gutenberg passages. These passages are human-written and human-catalogued. We use them only to test whether model rankings from SHELF carry over to natural text. We do not include them in the main corpus, because doing so would erase the distinction that the transfer test is meant to measure. Table 1 summarises document length. SHELF is right- skewed by design: the eight target-length categories run from a median of 18 words for micro to 2,985 for extended, and the short and medium categories together hold nearly half the corpus. The shortest documents are one-line micro-length forms such as prayers and greeting cards; 28 documents have ten words or fewer. The three corpora also differ in shape. Gutenberg passages are fixed-size extracts that cluster between 400 and 700 words, while LCSHBench records are catalogue metadata with a median 4 of 95 words. The 200-word truncation in Section 6 controls for that difference. 3.4 Quality control Every document was checked before it entered the re- leased corpus. The checks required a parseable response in the requested language and length range. They also rejected refusals, near-duplicates, documents that missed their assigned topics, and documents that stated their own catalogue labels. The first version of the label check flagged 15 to 20% of the documents, but most of those flags were false. It treated any use of a label as self-labelling, so ordinary phrases in the text could trigger it. We narrowed the check to places where a label was presented as a label, such as a heading, a term followed by a colon, or a taxonomy code in parentheses. The revised check flagged 2.48% of the earlier corpus and less than 0.5% of the newer corpus. One generation failure was more serious. For one model, 14.6% of calls returned reasoning tokens but no document, mostly on requests for short text. We increased the output-token allowance and removed every affected record. Both corrections changed which documents en- tered the release. Researchers who reproduce this work should check how each provider handles reasoning to- kens, thinking budgets, and output-token limits, espe- cially when requesting short documents. 3.5 Evaluation protocol Table 2 is the complete task inventory for this paper. The first five rows are released SHELF benchmark formula- tions. The final row is a separate decoder comparison on one classification task. Panel size changes with the formulation. Classifica- tion and clustering use 25 configurations. Retrieval also uses 25, replacing term frequency with BM25. Instruction retrieval uses the same 25 retrieval-compatible methods. The released pair configurations come from an earlier 21- model sweep. The cross-corpus analysis counts 24 distinct embedding models, or 25 for pairs, after accounting for task-specific methods. The masking analysis uses the 21 models for which every required arm was run. The model comparison uses frozen encoders. For classi- fication, we fit scikit-learn 1.8.0 logistic regression on the training embeddings, with class-balanced weights, a maxi- mum of 1,000 iterations, seed 42, and otherwise default set- tings. We report macro-F 1 on the test split. For retrieval, test documents are queries and the training and validation documents form the search corpus; rankings use cosine similarity and are scored with nDCG@10. For clustering, we applyk-means to normalised test embeddings withk set to the number of labels and report adjusted Rand index. Pair tasks use cosine similarity and report AUC, which does not require choosing a decision threshold. Exact checkpoint names, package versions, corpus checksums, and per-task result files are in the evaluation repository at https://github.com/mjbommar/shelf-benchmark. 4 Benchmark results We evaluate 25 configurations on SHELF classification and clustering tasks. The panel includes lexical methods and encoders from 22M to 596M parameters. Four en- coders were released in 2025; one appears at two context lengths, so they account for five configurations. We re- port macro-F 1 for classification and adjusted Rand index for clustering. An adjusted Rand index near zero means that the clusters are no better than a random partition. We keep the two metrics separate because they measure different tasks. 4.1 Evaluated methods The lexical baselines are term frequency and TF-IDF, both reduced with truncated SVD. BM25 [22] is included only on tasks that support its scoring interface. The encoder panel begins with BERT [7], RoBERTa [16], and DistilBERT [23], used as frozen backbones with mean pooling. It also includes MiniLM [33], MPNet [25], the BGE [37], E5 [31], and GTE [15] families, GTR-T5 [19], and Instructor [26]. This range lets us compare early sentence encoders with models trained more directly for embedding and retrieval. The panel also includes two masked-language models from the OGBert family, trained on the OpenGloss syn- thetic dictionary corpus [3]. We label the published 110M model and the smaller 36M model as OGBert 110M and OGBert 36M. Their model cards document the training data and checkpoints. 3 Four encoders in the panel were released in 2025: Granite-small-r2 at 48M parameters, GTE-ModernBERT at 149M [34], EmbeddingGemma at 308M [30], and Qwen3-Embedding at 596M [41].We run GTE- ModernBERT with context limits of 2,048 and 8,192 tokens to measure the effect of that limit (Section 7). The largest of these newer encoders has 596M parameters; the largest older encoder in the panel has 335M. Thus, the panel remains a comparison among small and mid-size encoders; it does not compare them with frontier-scale models. We do not add the query or document prefixes rec- ommended by individual model cards. Every encoder receives the same unprompted input. This keeps the pro- 3 https://huggingface.co/mjbommar/ogbert-110m-baseand https://huggingface.co/mjbommar/ogbert-v1-mlm. 5 FormulationShort descriptionTask variantsMetricEvaluated methods Classificationpredict one labelsubject; genre category; genre form; registermacro-F 1 TF; TF-IDF; encoders Clusteringrecover label groupssubject; genre form; register; geographyARITF; TF-IDF; encoders Retrievalrank documents for a labelsubject; genre category; genre formnDCG@10TF-IDF; BM25; encoders Pair classificationscore a document pairsame subject; same topic; topic overlap; same genre form; same register; same audience AUCTF-IDF; BM25; encoders Instruction retrieval match one facet and differ on an- other same subject/different form; same topic/different subject; same form/different subject; same audience/different register nDCG@10TF-IDF; BM25; encoders Decoder referencepredict a label without SHELF ex- amples subject onlymacro-F 1 four decoder models Table 2: Tasks evaluated in this paper. Not every method supports every formulation. ARI is the adjusted Rand index. tocol consistent, but it may understate models that depend on those prefixes and may affect their order in the tables. 4.2 Classification The word-frequency baseline is competitive. TF-IDF leads the register task, comes third on the 21-class subject task, and sits mid-table on genre form. BEIR established that BM25 is hard to displace in zero-shot retrieval [29], and FinMTEB reported bag-of-words beating every dense model on financial semantic similarity [28]. Our results reproduce that pattern. Model size buys less than expected. Models at 335M parameters and above occupy ranks two, four, seven, and fifteen on the subject task. Three sit behind a sparse baseline with no parameters to speak of. 4.3 Clustering Geography and register do not separate the current mod- els: their best scores are 0.0068 and 0.0678. Genre form separates them only weakly, with a best score of 0.0921. Subject clustering is the only useful leaderboard among the four, reaching 0.5152 and spreading the field. 4.4Retrieval, pair classification, and in- structions The remaining released formulations rank or score docu- ments rather than assign a label. Tables 5 to 7 report them in turn. 4.4.1 Retrieval Genre form is as hard to retrieve as it is to classify: the best model reaches 0.1173, against 0.7104 for subject. Clas- sification and retrieval both identify genre form as the harder label space. RoBERTa leads category retrieval despite ranking last on the subject and genre-form classification tasks. Be- cause raw RoBERTa lacks sentence-level similarity train- ing, we treat this as a task-specific result rather than evi- dence of generally stronger representations [21]. BM25 ranks 23rd, 24th and 25th of 25 on the three retrieval tasks. TF-IDF leads the register classification task and ranks third on subject (Table 3), so sparse strength is task-specific. 4.4.2 Pair classification Each pair task is a balanced binary decision: 2,000 posi- tive and 2,000 negative pairs. We report AUC because it measures how well cosine similarity orders positive above negative pairs without selecting a threshold on the test set. BM25 has no embedding, so its pair score is the mean of the BM25 score of each document used as a query against the other. Table 6 summarises the six tasks. Models distinguish pairs by subject, topic, and topic overlap, with median AUC from 0.69 to 0.71. They distin- guish genre form less reliably, at 0.62, while register and audience remain close to chance. Four thousand pairs are not four thousand independent observations, because they are built from far fewer doc- uments and each document can appear in several pairs. We therefore bootstrap documents rather than individ- ual pairs. When a document is resampled, we include every pair that contains it. Rebuilding the same-subject task on the pooled corpus and bootstrapping that way gives 0.8309 with a 95% document-clustered interval of [0.8093, 0.8511] for the best model. The 0.8474 in Table 6 comes from the released pair configurations, which are mined from the original cor- pus component; the 0.8309 is the same task mined from the aggregate corpus. The two corpora produce different pair sets and different scores. The table uses the released configurations because readers can download them. 4.4.3 Instruction retrieval Instructions anchored on subject are followed to a useful degree; instructions anchored on form or audience are not. This resembles FollowIR, where many instruction-aware retrieval models scored below zero on the paired measure. The present task does not isolate instruction following from the strength of the underlying label signal. 6 ModelSubject (21)Category (14)Genre form (133)Register (8) EmbeddingGemma-300M † 0.88870.79880.26050.6013 Qwen3-Embedding-0.6B † 0.88230.71560.19330.5347 TF-IDF+SVD0.86860.73420.17350.6358 BGE-large0.86630.73950.20690.5525 GTE-ModernBERT-8k † 0.86550.75890.22690.5291 GTE-ModernBERT-2k † 0.86460.75750.22810.5301 E5-large0.85120.75210.20000.5882 Granite-small-r2 † 0.85100.67130.17100.4505 E5-base0.84920.74610.21060.5658 GTE-base0.84690.71290.18370.4964 BGE-base0.84450.71790.20220.5329 MPNet0.84280.65060.14180.4629 GTE-small0.83860.70720.17620.4608 BGE-small0.83420.70730.19600.4735 GTR-T5-large0.83000.70560.16340.5158 E5-small0.82490.71760.17930.5243 Instructor-base0.82040.71940.17860.5173 GTR-T5-base0.80790.65820.14670.4785 TF+SVD0.80480.64500.12880.5103 MiniLM-L60.80150.62280.14760.4374 BERT0.79280.69130.14810.5983 DistilBERT0.78110.67820.13020.5784 OGBert 110M0.74490.61780.09510.5281 OGBert 36M0.69130.56050.08200.4846 RoBERTa0.67150.61420.07340.5200 Table 3: Classification, macro-F 1 , on the pooled corpus. Ordered by the 21-class subject task. Label counts are in the column heads. Rows marked† are 2025 encoders; GTE-ModernBERT appears twice, at 2,048 and 8,192 tokens on identical weights. TaskBest modelBest Median SubjectQwen3-Embedding-0.6B0.51520.3439 Genre form EmbeddingGemma-300M0.09210.0484 RegisterRoBERTa0.06780.0110 GeographyGTR-T5-base0.00680.0025 Table 4: Best model, best score, and median adjusted Rand index for each clustering task over 25 configurations on the pooled corpus. Zero is chance. These are single-run values at seed 42. The rank-agreement analysis in Section 6.2 uses each model’s median over five seeds instead, because clustering is seed-sensitive: the subject task’s best and median become 0.5247 and 0.3456 under that measure rather than 0.5152 and 0.3439. 4.5 Task difficulty and headroom The subject task has 21 classes, while the genre-form task has 133. We score all 25 configurations on both tasks, using the same corpus and documents. This lets us ask two separate questions: how much harder is genre form, and does it change the order of the models? The two relevant columns appear in Table 3. 4.5.1 Differences between tasks Genre form is much harder. The best score falls from 0.8887 on subject to 0.2605 on genre form. Every model TaskBest modelBest MedianBM25 SubjectQwen3-Embed-0.6B 0.71040.5988 0.4533 Genre form E5-base0.11730.0947 0.0585 CategoryRoBERTa0.49680.4247 0.2704 Table 5: Best model, best score, panel median, and BM25 nDCG@10 for each retrieval task over 25 configurations on the pooled corpus. “Qwen3-Embed-0.6B” abbreviates Qwen3- Embedding-0.6B. loses between 0.60 and 0.70 points, and the mean falls from 0.8226 to 0.1698. This is not an artifact of unseen classes. All 133 forms appear in every split, the rarest form in the test split has 25 documents, and the same documents carry both labels. 4.5.2 Agreement in model rankings The difficulty changes, but the model order mostly does not. Across the 24 distinct models, the two tasks have a rank correlation of 0.815, with a bootstrap interval of [0.58, 0.94]. A practitioner choosing among these models gets roughly the same order from either task. 7 TaskBestMedian Same subject0.84740.6925 Same topic0.81630.7054 Topic overlap0.82820.7133 Same genre form0.65790.6174 Same register0.57180.5329 Same audience0.55190.5331 Table 6: Pair classification AUC over 21 models, including BM25, on the released pair configurations, which use the original cor- pus component described in Section 3 rather than the aggregate corpus used elsewhere in this section. Chance is 0.5. InstructionBestMedian Same subject, different form0.54960.4724 Same topic, different subject0.20300.1771 Same form, different subject0.07640.0439 Same audience, different register0.06800.0507 Table 7: Instruction-conditioned retrieval, nDCG@10. Each task asks for documents matching one facet while differing on another. 4.5.3 Interpreting the genre-form task Genre form is useful because it remains difficult, not be- cause it produces a new model order. The top twelve subject scores span only 0.046, while every genre-form score remains low. A nearly saturated subject task can hide how much work remains; the genre-form task cannot. Its best score is only 0.2605. An early partial run gave a different impression. The first nine completed models had a rank correlation of 0.450, but that group was small and skewed toward cheap models; none had more than 109M parameters. After all models finished, the correlation rose to 0.815. The partial estimate had an interval that included zero and did not support a claim. We did not archive that run, so we do not report its interval. 4.6 Zero-shot decoder comparison Cataloguers have not annotated a SHELF sample, so the benchmark has no human agreement rate or expert ceiling. We add a separate comparison with four decoder language models, which generate text one token at a time. We ask each model to assign a subject class to each of the 12,504 test documents. These results are a separate zero-shot reference, not part of the 25-configuration main panel. They do not enter the rank analysis in Section 6.2. The decoder models postdate the original public SHELF release and may have encountered it during training. Their results do not carry the temporal out-of-sample guarantee that applies to the embedding panel. 4.6.1 Classification results The class names contain useful information. A uniform guess over 21 classes has an expected macro-F 1 of 0.048. Every decoder scores well above that value and predicts all 21 classes. Without labelled SHELF examples, the models can connect document text with the meanings of the class names. This result does not show whether the models en- countered Library of Congress labels during pretraining. The supervised probe scores higher, but the comparison is not like for like. The best decoder reaches 0.5860 against 0.8887 for the best encoder, and that encoder has 0.30B parameters. The encoder arm fits a classifier on 37,795 labelled documents, while the decoder arm receives no labelled SHELF examples. In this experiment, the small supervised encoder scores higher than all four zero-shot decoders. The decoder results give a reference for the tested models and prompts, not a general zero-shot floor. 4.6.2 Prompt and generator effects Prompt selection affects the scores. We selected the prompt and context limit for each model on the valida- tion split. Across the tested settings, macro-F 1 changed by more than 0.4 for one model, and different prompts worked best for different models. Using the chat template improved mean macro-F 1 by a factor of two to three over raw completion. The table reports each model’s best vali- dation setting. The repository contains the full validation grid. We find no consistent own-family advantage. Models in the GPT and OpenAI generator groups wrote 68.4% of the test split, so a GPT judge is largely reading text from its own family. Own-family accuracy differs from other-family accuracy by+0.0006,−0.0533,+0.0065, and+0.0271for the four judges. The direction is not con- sistent. The pooled test split has twelve stored generator groups. Legacy metadata separatesgptandopenai, so these groups are not the eleven laboratories counted for the balanced component in Section 3. Across the twelve groups, the six pairwise rank correlations between judges range from 0.622 to 0.902. The judges broadly agree about which groups of documents are harder, but these descrip- tive results do not identify the cause. 4.6.3 Measured inference rates Accuracy alone does not determine which method to de- ploy. We measured local inference on the first 1,000 test documents. The neural inputs were capped at no more than 512 tokens from each model’s tokenizer. The lexical 8 ModelParametersMacro-F 1 Accuracy Qwen3.5-0.8B0.87B0.38220.3912 Gemma-4-E2B5.12B0.45570.4959 Qwen3.5-2B2.27B0.50750.5263 GPT-5.6-lunaundisclosed0.58600.6068 Best encoder, supervised0.30B0.8887 Table 8: Decoders on 21-way subject classification, zero-shot, over the same 12,504 test documents as Table 3. The encoder row is a supervised linear probe fitted on 37,795 labelled documents; the decoders see no labelled SHELF examples. The rows therefore do not compare representation quality: one method has a fitted classifier and the other does not. model read the whole document. We discarded a warm- up pass and excluded model loading and classifier fitting. The rates come from one run, so they describe this har- ness and machine rather than each model’s best possible throughput. The lexical baseline is strong on this task. TF-IDF with a logistic head processes 2,271 documents per second on CPU and scores 0.8686. It trails the best encoder by 0.0201 macro-F 1 and has nineteen times its measured processing rate. For supervised subject classification, TF-IDF is a practical baseline when a system does not need reusable embeddings for other tasks. The encoders show an accuracy–rate tradeoff. EmbeddingGemma-300M improves macro-F 1 by 0.0872 over MiniLM-L6. Its measured processing rate is about one sixth as high. The table reports the tradeoff but does not assign a dollar or energy cost to it. Parameter count alone does not determine processing rate. Gemma-4-E2B has 5.12B weights but processes 20.3 documents per second in this harness. Qwen3.5-0.8B has 0.87B weights and processes 11.5. Architecture, imple- mentation, and batching also affect processing rate. This comparison does not isolate their effects or predict rates on another system. Input length changes the rate. Qwen3.5-0.8B processes 9.6 documents per second at its frozen 2,048-token limit, compared with 11.5 at the 512-token cap. Qwen3.5-2B processes 7.2 whole documents per second, compared with 11.2 at the cap. The macro-F 1 values in the table come from the frozen configurations. The 512-token rates for these two models therefore do not describe the exact configurations that produced those scores. 5 Benchmark diagnostics The aggregate scores do not show what the models read or where a task fails. We test the most important sources of misleading performance before turning to external val- idation. 5.1 Surface label signal SHELF documents often repeat words from their own labels. This surface signal can make a task easier without making the underlying representation better. 5.1.1 Measuring label terms Count how often a document contains the exact name of its own label. Reported alone, the number has no clear meaning. A document about birds will contain the word “birds”, and it should; a corpus scoring zero would be a strange corpus. The number becomes readable when a natural corpus is measured the same way. Real documents do name their own subjects, about 6% of the time at matched length. SHELF does it about three times as often. That excess is consistent with the higher within-corpus score, but this count does not estimate how much of the score difference it causes. 5.1.2 Removing the label terms The count alone cannot tell us how much these terms affect a score, so we run two deletion tests. The masked arm removes every occurrence of a document’s own label terms. Those terms are broader than the subject-class name counted in Table 10: they cover the subject-class name, the document’s topic terms, and its genre form and category names, each with a frozen variant list of the plu- ral and every token of five or more characters. The sham arm removes the same number of random tokens from the same document. Comparing the two separates the effect of removing label words from the effect of shortening the text. Every document loses the same token count in the masked and sham arms. We check the count per doc- ument, not only in the mean. Masking can affect only documents that contain their own label terms. That is 86.3% of SHELF, 52.3% of Gutenberg, and 24.1% of LCSH- Bench, so the test has less power on LCSHBench than on SHELF. 9 ArmModelMacro-F 1 Docs/sAmortized ms/doc Lexical, CPUTF-IDF + logistic0.86862270.90.44 Encoder + probeMiniLM-L60.8015715.91.40 Encoder + probeBGE-small0.8342379.32.64 Encoder + probeEmbeddingGemma-300M0.8887119.28.39 Decoder, 0-shotQwen3.5-0.8B0.382211.586.92 Decoder, 0-shotGemma-4-E2B0.455720.349.24 Decoder, 0-shotQwen3.5-2B0.507511.289.72 Decoder, 0-shotGPT-5.6-luna (API)0.58609.4106 Table 9: Reported test macro-F 1 and observed processing rates. Accuracy uses the full test evaluation. Local rates use 1,000 documents on one host: TF-IDF on CPU, encoders on an RTX 4070 Ti SUPER in batches of 64, and local decoders on that GPU one document at a time. Rate measurements cap neural inputs at 512 model-specific tokens; TF-IDF read the whole document. The macro-F 1 values use each method’s frozen test configuration, which uses 2,048 tokens for Qwen3.5-0.8B and whole documents for Qwen3.5-2B. The API row uses concurrent requests over the full test run. The final column is elapsed time divided by document count, not single-request latency. CorpusFull textFirst 200 words Gutenberg (natural)9.0%6.1% SHELF23.4%19.1% Table 10: How often the subject-class name appears in its own document. Both corpora are running prose, and the right column cuts every document to its first 200 words so length cannot drive the comparison. Counted exhaustively over all 65,915 documents, so the figures are exact rather than sampled. The number of removed tokens also differs because the corpora carry different metadata. Masking removes an average of 9.13 tokens from SHELF, 1.89 from Gutenberg, and 0.63 from LCSHBench. We therefore compare each masked score with its own sham rather than comparing masked scores across corpora. Removing label terms costs about 3.4 macro-F 1 points on SHELF and 1.2 on Gutenberg, while the sham moves neither score materially. Retrieval behaves the same way, at−0.0396masked against+0.0008sham on SHELF. Label-term availability therefore contributes measurably to SHELF’s absolute scores, more than it does in Guten- berg. Masking changes the rankings only slightly. Corre- lation between masked and unmasked model order is 0.987 on SHELF, 0.988 on Gutenberg, and 0.983 on LC- SHBench. Removing random tokens changes none of the orders: the sham correlation is 1.000[1.00, 1.00]. On SHELF, the difference between the masked and sham cor- relations is−0.013, with an interval of[−0.063, +0.000]. The cross-corpus rankings also remain similar after mask- ing. SHELF’s agreement is 0.831 with Gutenberg and 0.748 with LCSHBench, compared with 0.870 and 0.765 before masking. Label-term availability raises scores but has little effect on model order. 5.1.3 Changes between generations SHELF exists in two generations built with different prompts and gates. Their difference can be measured, but it cannot be assigned to one change in the pipeline. Genre-form repetition fell by 81% and is now rare. Topic repetition fell less and remains common. 5.1.4 Topic leakage and mitigation The cause is in the prompt. Genre and subject reach the writing model as descriptions — “spatial representations, geographic visualizations” rather than “Maps”. Topics reach it as the exact strings. This difference accounts for the pattern in Table 12. We tried describing topics too and rejected it before spending anything, because only 382 of 1,983 topic de- scriptions come from real scope notes. Another 1,546 are built from position in the subject tree and pick the wrong sense: “Information” resolves to a topic within criminal procedure, “Cloud computing” to distributed data process- ing, “Security” to investments. Writing from those would cut the leak by writing about the wrong subject. What does work is an instruction. Telling the model to develop the subject without using the topic words as labels cuts the leak from 82.2% to 4.4% on one model and from 61.5% to 23.1% on another. Removing the topic words has a cost, though. We asked a blind judge, one that did not know which prompt had produced each document, whether the document still cov- ered its assigned topics: 543 judgements across two gen- erators. With the plain instruction, topic coverage fell by 11.7 points, with an interval from−19.5to−3.4. When 10 CorpusUnmaskedMaskedSham SHELF0.79780.7641 (−0.0337)0.7974 (−0.0004) Gutenberg0.43180.4203 (−0.0115)0.4325 (+0.0007) LCSHBench0.52480.5012 (−0.0235)0.5264 (+0.0016) Table 11: Subject classification, macro-F 1 , averaged over the 21 distinct models measured in the masking arms. Masking deletes the label terms; the sham deletes as many random tokens from the same documents. The sham column is the control: what a document loses by being shortened at all. GenerationSubject classTopicsGenre form First21.8%76.7%7.3% Second13.4%45.0%1.4% Table 12: Label names appearing in their own documents, by generation, every document truncated to 200 words. the instruction also supplied a trustworthy gloss of the topic, coverage fell by 6.5 points, with an interval from −13.7to+1.2. The plain instruction therefore trades leakage for coverage. The gloss version may avoid that trade, but its interval includes zero, so we cannot say. The released generation keeps the original prompt. We did not measure coverage by cosine similarity be- tween the document and the topic string. That measure falls whenever the treatment removes the string itself, even if the document still covers the topic. A blind judge- ment of meaning does not depend on the words the treat- ment removes. 5.2 Weak and failed tasks Several tasks and construction choices failed their in- tended checks. These failures define where the benchmark is informative and where it is not. Weak clustering tasks. Adjusted Rand index is cor- rected for chance. Subject clustering separates the model panel, with a median of 0.3439 and a best score of 0.5152. Genre is much weaker, with a best score of 0.0921. Reg- ister peaks at 0.0678, and geographic clustering peaks at 0.0068. The geographic task does not support model comparison. A geographic-label defect. In the first generation, 76.4% of documents carrying two geographic tags had tags from different regions. This was an artefact of inde- pendent sampling rather than a property of the text. The second generation samples compatible tags and checks the result against a shuffled-label control. A rewrite increased label leakage.We tested a more discursive writing style as a way to reduce lexical predic- tion. It raised exact copying of label terms from 29.3% to 35.5%. TF-IDF macro-F 1 on a 20-subclass pilot task also rose from 0.639 to 0.724. We rejected the rewrite. Suppressing topic words reduced coverage. A prompt instruction cut topic leakage, but blind judging found lower topic coverage. A gloss-based variant re- duced that loss, although its interval included zero. The released generation keeps the original prompt because the study does not support a free improvement in both measures. Thresholded pair scores were misleading.The first pair-task implementation chose a threshold on test labels and reportedF 1 . For genre form, register, and audience, the median model then produced the all-positive result: F 1 = 0.667with accuracy 0.500. The benchmark now uses AUC, which measures pair ordering without fitting a test-set threshold. Run order affected the last decimal places.Unsorted embedding caches and variable thread counts caused score drift. The evaluator now sorts texts, fixes thread settings, and records run metadata. The measured drift is small relative to the reported effects, but older result files do not all carry the same provenance fields. The release preserves those files and names the result set used for each analysis. 6 External validation 6.1 Transfer of absolute scores SHELF is designed to compare models on controlled bib- liographic tasks. Its scores are not intended to predict accuracy on a library’s records. We test that boundary with 21-class subject classification on Project Gutenberg passages and 4,924 English LCSHBench records. We fit TF-IDF with logistic regression on each corpus and test it on all three. Macro-F 1 is averaged across the 21 classes. This probe has no prior language training, so its transfer gap measures the corpora rather than model memory. It uses the full TF-IDF matrix and a grouped 70/30 split on SHELF. The 0.8686 baseline in Table 3 in- 11 stead compresses TF-IDF to 256 dimensions and uses the benchmark’s stored 60/20/20 split, so the two in-domain scores are not expected to match. The diagonal measures how easy each corpus is within itself. SHELF reaches 0.8796 where Gutenberg reaches 0.5101 and LCSHBench 0.5559, so SHELF is a much easier classification problem than either natural corpus. That gap is consistent with the label-term excess measured in Section 5. Scores fall in every transfer direction. A probe trained on SHELF scores 0.8796 on SHELF but 0.3193 on Guten- berg, and a probe trained on Gutenberg scores 0.2823 on SHELF. Corpus size does not explain this: shrinking all three corpora to the same 3,016 documents leaves SHELF at 0.8052 on its own documents and 0.2341 on Gutenberg. Nor does document length, which differs sharply across the three corpora (Table 1): with every document cut to its first 200 words, SHELF remains the easiest corpus to clas- sify within itself, at 0.8521 against 0.4478 for Gutenberg and 0.5522 for LCSHBench. Generated text is not the only cause. The two human- written corpora do not transfer to each other either: a probe trained on Gutenberg scores 0.2135 on LCSHBench, the lowest cell in Table 13. Book passages and catalogue records differ in form and in cataloguing practice, and that difference matters even when people wrote both. Read by column, SHELF is the better off-domain train- ing source for both natural corpora. On LCSHBench records, a probe trained on SHELF scores 0.4078 where one trained on Gutenberg scores 0.2135; on Gutenberg passages, 0.3193 against 0.2800 from LCSHBench. SHELF and LCSHBench share Library of Congress vocabulary that literary prose does not. Part of the advantage is size: with SHELF cut to 3,016 documents, its score on Guten- berg falls to 0.2341. The conclusion is still narrow: a SHELF score describes performance on SHELF, not the accuracy to expect in production. 6.2 Transfer of model rankings Benchmark scores need not transfer for model order to remain useful. We rank 24 distinct embedding models on the subject task in SHELF, Gutenberg, and LCSHBench. In- tervals bootstrap models because the question is whether the order survives a different model panel. SHELF gives similar subject-model rankings to both natural corpora, and both intervals exclude zero. The agreement is lower than the 0.958 correlation between the natural corpora. The intervals overlap, so the table does not establish a difference between those correlations. We then test whether the ranking agreement holds across four subject-task formulations. Classification agree- ment was already known at preregistration, on the smaller model panel available then:ρ = 0.877against Gutenberg and0.777against LCSHBench. The panel has since grown, which is why Table 14 reports slightly different values for the same comparison. Before running retrieval, cluster- ing, or pairs, we froze the decision rule for the resulting four-formulation claim: agreement requiresρ≥ 0.6with a lower interval bound above zero, and at least three of four formulations must pass against both natural corpora. The dated preregistration record is distributed with the evaluation repository. 4 All four formulations pass against both corpora. A pre- registered stability check also supports the clustering row. Across five seeds, the median pairwise rank correlations are 0.991 on SHELF, 0.932 on Gutenberg, and 0.962 on LCSHBench. One of the ten Gutenberg seed pairs is 0.892, below the 0.90 floor, but the preregistered rule applies the median and all three corpora pass. The capable-only sensitivity analysis removes both OG- Bert models and RoBERTa without sentence-specific train- ing. All eight comparisons still pass. Its weakest cell is subject classification against LCSHBench at 0.655, with an interval of[0.22, 0.93]. The four formulations reuse labels, documents, and representations. They show that agreement is not confined to one formulation; they are not four independent validations. 7 Scope and limitations SHELF supports comparison among the evaluated models on its released tasks. Its controlled facets also support di- agnosis: a model can be tested on subject, genre, audience, or register without relying on the correlations found in catalogue records. The following limits bound that use. 7.1Reference points and external validity Cataloguers have not annotated a SHELF sample, so the scores have no human agreement rate or expert ceiling. The benchmark shows how models differ on its labels, but not whether a remaining error would matter to a cata- loguer. The zero-shot decoder comparison in Section 4.6 is a machine reference, not a substitute for human review. That decoder comparison is also narrow. It covers sub- ject classification only, uses four models, and gives the decoders no labelled example while the encoders are fit- ted on 37,795. It does not test genre form, where encoder scores are lowest. We therefore do not know whether prompted decoders would close that gap. All SHELF documents are model-written, and neural retrievers can prefer model-written text [6]. Using several writing models does not measure or remove that bias. A 4 https://github.com/mjbommar/shelf-benchmark/blob/master /docs/PREREGISTRATION.md 12 Train\ testSHELFGutenbergLCSHBench SHELF 0.8796 [0.88, 0.88] 0.3193 [0.29, 0.34] 0.4078 [0.38, 0.43] Gutenberg 0.2823 [0.28, 0.29] 0.5101 [0.48, 0.54] 0.2135 [0.19, 0.23] LCSHBench 0.4458 [0.44, 0.45] 0.2800 [0.25, 0.31] 0.5559 [0.53, 0.58] Table 13: Subject classification, macro-F 1 . Rows are training corpora and columns are test corpora. Intervals resample test documents. PairSpearman95% intervalKendall SHELF vs Gutenberg0.870[0.61, 0.98]0.761 SHELF vs LCSHBench0.765[0.43, 0.96]0.645 Gutenberg vs LCSHBench0.958[0.86, 0.99]0.841 Table 14: Rank agreement between corpora over 24 distinct embedding models. Intervals use 2,000 bootstrap resamples of the model set. matched study of human and model-written text is still needed. The external comparison covers subject labels only be- cause Gutenberg and LCSHBench do not provide the other SHELF facets at comparable coverage. Agreement on sub- ject classification, retrieval, clustering, and pairs does not validate the genre, audience, register, geography, or in- struction tasks. The transfer matrix also uses one lexical probe. Its scores do not carry between the three corpora, but another classifier may lose a different amount. 7.2 Corpus and split provenance The main results pool 42,532 original records with document-level splits and 20,367 later records with specifi- cation IDs. The grouped-split guarantee therefore applies only to the later component. We cannot test whether ver- sions of one original brief appear on both sides of a split. A complete core-only sweep under grouped splits would show how much this uncertainty affects the headline re- sults. The pooled corpus is also not balanced across writing models: its largest model supplies 47.7% of the documents. Comparisons among writing models must use the bal- anced factorial component described in Section 3. The released pair tasks come from the original corpus compo- nent, as stated in Table 6. 7.3 Models, tasks, and implementation The evaluated encoders range from 22M to 596M parame- ters. This includes recent small and mid-size models, but not the billion-parameter systems at the top of general embedding leaderboards [10]. A larger panel may produce a different order. The evaluator omits the query and document prefixes recommended by individual model cards. One input for- mat makes the comparison consistent, but it may favour models that need less prompting. These rankings apply to the unprompted protocol, not to each model’s best configuration. Several tasks do not separate the current models. Ge- ographic clustering and the register and audience pair tasks remain at or near chance. They work as negative controls, but not as useful leaderboards. Document length differs across models and corpora. Many encoders cut text at 512 tokens, while SHELF documents often exceed that limit.Running GTE- ModernBERT at 2,048 and 8,192 tokens changed the mean across 32 task–corpus cells by−0.0006. Longer context did not help that model, but this experiment does not test the step from 512 tokens or any other encoder. Finally, the timing results in Table 9 come from one run and have no repeated-trial intervals. TF-IDF uses CPU, encoders use GPU batches, local decoders process one document at a time, and the hosted model uses con- current requests on undisclosed hardware. TF-IDF reads whole documents while neural inputs have token limits. These are observed processing rates for one harness and machine, not hardware-neutral measures of latency, cost, or energy. 13 FormulationGutenbergLCSHBench Classification0.870 [0.61, 0.98]0.765 [0.43, 0.96] Retrieval0.813 [0.59, 0.91]0.898 [0.75, 0.96] Clustering0.916 [0.78, 0.97]0.950 [0.85, 0.98] Pairs0.907 [0.74, 0.96]0.942 [0.82, 0.98] Table 15: Spearman rank agreement by subject-task formulation, with 95% model-bootstrap intervals. Classification, retrieval, and clustering use 24 models; pairs use 25. 8 Conclusion SHELF tests bibliographic work that general leaderboards do not isolate. This release contains 62,899 documents and tasks for classification, clustering, retrieval, pair classifica- tion, and instruction retrieval. The results show why those tasks should remain separate. Current methods classify broad subjects well, yet struggle with fine genre forms, register, audience, and several clustering tasks. Sparse methods remain strong classifiers, while dense encoders lead retrieval. In the timing experiment, TF-IDF has a near- best subject score and the highest measured processing rate. The controlled data make these differences easier to diagnose. SHELF can vary subject, genre, audience, reg- ister, and writing model independently, and its factorial subset includes combinations unlikely to occur in most natural collections. But control does not make synthetic scores equivalent to production accuracy. Label terms make SHELF somewhat easier, some tasks remain near chance, and the corpus has no human catalogue judge- ment as a reference. The comparisons with Project Gutenberg and LCSH- Bench sharpen that boundary. Absolute scores change sharply across corpora, while subject-model rankings are more stable. SHELF can therefore help choose and com- pare methods on its released tasks, but it cannot predict accuracy on a library’s catalogue. A future study with cat- aloguer annotations and a broader encoder panel would test both limits directly. SHELF is a system as well as a fixed release. It can generate new documents after a future model’s training cutoff while preserving the labels and task design. We release the corpus [4], source code, per-task results, and scripts under permissive licences on Hugging Face and at https://github.com/mjbommar/shelf-benchmark. Disclosures and limitations Data availability The dataset is available on Hugging Face. Evalua- tion code and stored per-task results are available on GitHub. Funding none to declare. Competing interests none to declare. AI assistance Portions of this work were prepared with assistance from large language models. The author is solely responsible for all content, including any errors or omissions. Limitations The limits on interpretation, including the absence of a human reference and a frontier-scale encoder, are stated in Section 7. References [1]Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, et al. Measuring what matters: Con- struct validity in large language model benchmarks. In NeurIPS Datasets and Benchmarks Track, pages 19868–19949, 2025. doi: 10.52202/085713-0590. URL https://doi.org/10.52202/085713-0590. [2]Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cos- mopedia, February 2024. URLhttps://huggingfac e.co/datasets/HuggingFaceTB/cosmopedia. [3]Michael J. Bommarito. OpenGloss: A synthetic ency- clopedic dictionary and semantic knowledge graph, 2025. URLhttps://arxiv.org/abs/2511.18622. verified 2026-09-02. [4] Michael J. Bommarito. SHELF: A synthetic harness for multi-task bibliographic benchmarking.https: //huggingface.co/datasets/mjbommar/SHELF, 2026. Dataset, C-BY-4.0. [5]Samuel R. Bowman and George E. Dahl. What will it take to fix benchmarking in natural language under- standing? In NAACL-HLT, pages 4843–4855, 2021. 14 doi: 10.18653/v1/2021.naacl-main.385. URLhttps: //aclanthology.org/2021.naacl-main.385/. [6] Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, Gang Wang, and Jun Xu. Neural retrievers are biased towards LLM- generated content. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 526– 537, 2024. doi: 10.1145/3637528.3671882. URL https://arxiv.org/abs/2310.20501. [7]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, pages 4171–4186, 2019. doi: 10.18653/v1/N19-1423. URLhttps://aclanthology .org/N19-1423/. [8]Jennifer D’Souza, Sameer Sadruddin, Holger Israel, Mathias Begoin, and Diana Slawig. SemEval-2025 task 5: LLMs4Subjects - LLM-based automated sub- ject tagging for a national technical library’s open- access catalog. In Proceedings of the 19th International Workshop on Semantic Evaluation, pages 2570–2583, 2025. URLhttps://aclanthology.org/2025.seme val-1.328/. [9] Adnan El Assadi, Isaac Chung, Roman Solomatin, Niklas Muennighoff, and Kenneth Enevoldsen. HUME: Measuring the human-model performance gap in text embedding tasks. In International Con- ference on Learning Representations, 2026.URL https://openreview.net/forum?id=rcmfu1ydAf. [10]Adnan El Assadi, Niklas Muennighoff, and Jinhyuk Lee. The embedder’s dilemma: LLMs are better, but at what cost? In Conference on Language Modeling, 2026. URL https://arxiv.org/abs/2608.12875. [11]Kenneth Enevoldsen, Isaac Chung, Imene Ker- boua, Márton Kardos, Ashwin Mathur, David Stap, et al. MMTEB: Massive multilingual text embedding benchmark. In International Conference on Learning Representations, 2025. URLhttps://arxiv.org/ab s/2502.13595. [12] Eibe Frank and Gordon W. Paynter. Predicting library of congress classifications from library of congress subject headings. Journal of the American Society for Information Science and Technology, 55 (3):214–227, 2004. doi: 10.1002/asi.10360. URL https://doi.org/10.1002/asi.10360. [13]Alexander Gill, Abhilasha Ravichander, and Ana Marasović. What has been lost with synthetic eval- uation? In Findings of the Association for Compu- tational Linguistics: EMNLP 2025, pages 9902–9945, 2025. doi: 10.18653/v1/2025.f indings-emnlp.526. URLhttps://aclanthology.org/2025.findings -emnlp.526/. [14]Albert Gong, Kamil ̇ e Stankeviči ̄ ut ̇ e, Chao Wan, An- mol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P. Gomes, and Kilian Q. Weinberger. PhantomWiki: On-demand datasets for reasoning and retrieval evaluation. In International Conference on Machine Learning, 2025. URLhttps://arxiv.or g/abs/2502.20377. [15]Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learn- ing, 2023. URLhttps://arxiv.org/abs/2308.032 81. [16] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A ro- bustly optimized BERT pretraining approach, 2019. URL https://arxiv.org/abs/1907.11692. [17]Michael Majurski and Cynthia Matuszek. Grounding synthetic data evaluations of language models in unsupervised document corpora. arXiv preprint, 2025. URL https://arxiv.org/abs/2505.08905. [18]Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the ACL, pages 2014–2037, 2023. doi: 10.18653/v1/2023.eacl-main.148. URLht tps://aclanthology.org/2023.eacl-main.148/. [19]Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In EMNLP, pages 9844–9855, 2022. doi: 10.18653/v1/20 22.emnlp-main.669. URLhttps://aclanthology.o rg/2022.emnlp-main.669/. [20] Inioluwa Deborah Raji, Emily M. Bender, Amanda- lynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world bench- mark. In NeurIPS Datasets and Benchmarks Track, 2021. URL https://arxiv.org/abs/2111.15366. [21] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In EMNLP-IJCNLP, pages 3982–3992, 2019. doi: 10.1 8653/v1/D19-1410. URLhttps://aclanthology.o rg/D19-1410/. 15 [22]Stephen Robertson and Hugo Zaragoza. The prob- abilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3 (4):333–389, 2009. doi: 10.1561/1500000019. URL https://doi.org/10.1561/1500000019. [23] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019. URL https://arxiv.org/abs/1910.01108. [24]Sumuk Shashidhar, Clémentine Fourrier, Alina Lo- zovskia, Thomas Wolf, Gokhan Tur, and Dilek Hakkani-Tür. YourBench: Easy custom evaluation sets for everyone, 2025. URLhttps://arxiv.org/ abs/2504.01833. [25] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. MPNet: Masked and permuted pre-training for language understanding. In NeurIPS, 2020. URL https://arxiv.org/abs/2004.09297. [26]Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction-finetuned text embeddings. In Findings of ACL, pages 1102–1121, 2023. doi: 10.186 53/v1/2023.f indings-acl.71. URLhttps://aclantho logy.org/2023.findings-acl.71/. [27] Kwok Leong Tang.LCSHBench: A multilin- gual, consensus-grounded benchmark for library of congress subject heading assignment, 2026. URL https://arxiv.org/abs/2606.04382. [28] Yixuan Tang and Yi Yang. FinMTEB: Finance mas- sive text embedding benchmark. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 3620–3638, 2025. doi: 10.18653/v1/2025.emnlp- main.179. URLhttps: //aclanthology.org/2025.emnlp-main.179/. [29]Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS Datasets and Benchmarks Track, 2021. URLhttps://datase ts-benchmarks-proceedings.neurips.c/paper/ 2021/hash/65b9eea6e1c6b9f0cd2a47751a186f -Abstract-round2.html. [30] Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Pa- nyam, et al.EmbeddingGemma: Powerful and lightweight text representations, 2025. URLhttps: //arxiv.org/abs/2509.20354. [31]Liang Wang, Nan Yang, Xiaolong Huang, Binx- ing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly- supervised contrastive pre-training, 2022.URL https://arxiv.org/abs/2212.03533. [32]Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In ACL, pages 11897–11916, 2024. doi: 10.18653/v1/20 24.acl-long.642. URLhttps://aclanthology.org /2024.acl-long.642/. [33]Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In NeurIPS, 2020. URL https://arxiv.org/abs/2002.10957. [34]Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, et al. Smarter, better, faster, longer: A modern bidi- rectional encoder for fast, memory efficient, and long context finetuning and inference. In ACL, pages 2526–2547, 2025. doi: 10.18653/v1/2025.acl-long.127. URLhttps://aclanthology.org/2025.acl-long. 127/. [35]Orion Weller, Dawn Lawrie, and Benjamin Van Durme. NevIR: Negation in neural information retrieval. In EACL, pages 2274–2287, 2024. doi: 10.18653/v1/2024.eacl- long.139. URLhttps: //aclanthology.org/2024.eacl-long.139/. [36]Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. FollowIR: Evaluating and teaching information retrieval models to follow instructions. In NAACL-HLT, pages 11926–11942, 2025. doi: 10.18653/v1/2025.naacl-long.597. URL https://aclanthology.org/2025.naacl-long.59 7/. [37]Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muen- nighoff, Defu Lian, and Jian-Yun Nie. C-Pack: Packed resources for general chinese embeddings. In ACM SIGIR, pages 641–649, 2024. doi: 10.1145/3626772.36 57878. URL https://arxiv.org/abs/2309.07597. [38]Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language mod- els with rephrased samples, 2023. URLhttps: //arxiv.org/abs/2311.04850. [39]Xingkun Yin, Xuebin Tang, and Hongyang Du. CoDeR: Local constraint-compatible retrieval be- 16 yond semantic similarity, 2026. URLhttps://ar xiv.org/abs/2606.13204. [40] Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Sum- mer Yue. A careful examination of large language model performance on grade school arithmetic, 2024. URL https://arxiv.org/abs/2405.00332. [41]Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URLhttps://arxiv.org/abs/2506 .05176. [42] Lucia Zheng, Neel Guha, Brandon R. Anderson, Pe- ter Henderson, and Daniel E. Ho. When does pre- training help? assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ le- gal holdings. In International Conference on Ar- tificial Intelligence and Law, pages 159–168, 2021. doi: 1 0 . 1 1 4 5 / 3 4 6 2 7 5 7 . 3 4 6 6 0 8 8.URLhttps: //doi.org/10.1145/3462757.3466088. 17