Paper deep dive
Characterising AI Models for Cataloguing
Miguel Arana-Catania, Neil Jefferies
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/18/2026, 2:31:05 PM
Summary
This study evaluates the performance of various Large Language Models (LLMs), including OpenAI GPT-4.1 variants, GPT-5, and Gemini 2.5 Flash-Lite, in automating the cataloguing of digital collections. Using a dataset of 87 scanned dissertations from the Bodleian Libraries, the authors compared AI-generated metadata against human-curated ground truth using Jaccard similarity, semantic similarity, and BLEU scores. The research highlights that GPT-4.1-mini generally achieved the best performance, while also identifying non-intuitive prompt sensitivities and format-dependent variations in AI output quality.
Entities (14)
Relation Signals (8)
Neil Jefferies → authored → Characterising AI Models for Cataloguing
confidence 95% · Miguel Arana-Catania 1 and Neil Jefferies 1
Miguel Arana-Catania → authored → Characterising AI Models for Cataloguing
confidence 95% · Miguel Arana-Catania 1 and Neil Jefferies 1
Bodleian Libraries → owned → Global Dissertations collection
confidence 95% · The dataset used in this project was drawn from the Bodleian Libraries’ Global Dissertations collection
Gemini 2.5 Flash Lite → developedby → Google
confidence 90% · The Gemini models [14–16]
GPT-4.1 Mini → developedby → OpenAI
confidence 90% · The OpenAI GPT models [10–13]
GPT-4.1 Mini → performedbest → Cataloguing Task
confidence 90% · It can be seen that GPT-4.1-mini obtains the top results in most of the categories
GPT-5 → performedsecondbest → Cataloguing Task
confidence 85% · followed by GPT-5
MARC → producedbestresults → Cataloguing Task
confidence 85% · In summary, MARC seems to produce the best results, followed by JSON, and then by BIBFRAME.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The creation of digital collections involves not only the digitisation of content, but also the creation of catalogue records for it. This often-overlooked task requires slow and costly expert manual work. In this project, we have evaluated the application of AI models to this task, comparing different implementations and models. This work includes a qualitative and quantitative evaluation of the experiments carried out, as well as recommendations on the use of AI models that go beyond the specific use case.
Tags
Links
- Source: https://arxiv.org/abs/2607.11353v1
- Canonical: https://arxiv.org/abs/2607.11353v1
Trouble viewing inline? Open PDF directly →
Full Text
36,904 characters extracted from source content.
Expand or collapse full text
Characterising AI Models for Cataloguing Journal Title X(X):1–7 ©The Author(s) 2026 Reprints and permission: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/ToBeAssigned w.sagepub.com/ SAGE Miguel Arana-Catania 1 and Neil Jefferies 1 Abstract The creation of digital collections involves not only the digitisation of content, but also the creation of catalogue records for it. This often-overlooked task requires slow and costly expert manual work. In this project, we have evaluated the application of AI models to this task, comparing different implementations and models. This work includes a qualitative and quantitative evaluation of the experiments carried out, as well as recommendations on the use of AI models that go beyond the specific use case. Keywords cataloguing, llms, multimodal, digital collections Introduction The broad objective of this work at the Bodleian Libraries was to build in-house understanding and expertise in AI tools based on Large Language Models (LLMs), with a particular focus on cataloguing-related tasks. In this project, this has been approached from two main angles: 1. Understand the performance of various AI and other automated tools in creating or extracting metadata from a variety of sources, in comparison with the benchmark records from the previous activity. 2. Understand the broader behaviour of AI tools that may influence future deployment, such as their reliability, consistency, and dependency on prompt construction. AI technologies have been applied in diverse ways to the task of library cataloguing and classification. [1, 2] provide comprehensive reviews of these explorations. In this regard, prior to our project, [3] evaluated the generation of subject headings using human-generated metadata from the works as input; [4] used LLMs for topic classification using a controlled vocabulary; [5] worked with university theses, using AI to identify the ‘college’ field in the metadata record; [6] applied traditional machine learning techniques to the generation of Dewey Decimal Classification classes; [7] used machine learning to classify documents based on their images; [8] fine-tuned LLMs to generate specific metadata fields such as title, language or date; and [9] used LLMs to generate Library of Congress Classification classes. Whilst these previous works offer interesting insights into the use of AI for specific tasks within the cataloguing process, our project aims for a much more ambitious goal. In this work, we employ an end-to-end approach using AI models for the entire cataloguing process. Our experiments employ scanned images of the content as input and produce the complete metadata record as output, in the appropriate encoding format (MARC, BIBTEX, JSON), which includes not only the content of the relevant fields but also the correct formatting code. Methodology In this section, we describe the methodology used in this article. In brief, this has involved applying various LLMs to a collection of digitised document images with the aim of automating the creation of catalogue records for these documents. Below, we present details of the data used, as well as the experimental and evaluation methodology. Dataset The dataset used in this project was drawn from the Bodleian Libraries’ Global Dissertations collection that had been newly scanned. This, mainly 16th & 17th Century European, dataset had the advantage that it was very unlikely that the texts had previously been made available online, so the AI algorithms under test would not have been trained on them. The results obtained would thus be an accurate reflection of cataloguing performance rather than the retrieval of information already embedded in the AI model. A card catalogue for the collection was also scanned and used to provide a human-generated ground truth for evaluation purposes. The corpus consisted of 87 dissertations, with a median of 16 page images per item. The total number of images analysed was 1886. Preprocessing The image preprocessing involved conversion from archival TIFF to JPEG and resizing them to a maximum horizontal resolution of 1600 pixels, maintaining the original aspect ratio. This was done to reduce the AI inference’s 1 University of Oxford, UK Corresponding author: Miguel Arana-Catania, University of Oxford, Broad Street, Oxford, OX1 3BG, UK Email: humd0244@ox.ac.uk Prepared using sagej.cls [Version: 2017/01/17 v1.20] arXiv:2607.11353v1 [cs.CL] 13 Jul 2026 2Journal Title X(X) computational load. The resizing was performed using the Lanczos filter of the Pillow library. AI inference The AI models used were the following: • OpenAI GPT: 4.1-nano, 4.1-mini, 4.1, 5. • Gemini: 2.5 Flash-Lite. The OpenAI GPT models [10–13] and the Gemini models [14–16] are large language models based on the Transformer architecture [17]. This architecture uses the concept of attention [18] to contextualise the use of words in sentences. While large language models are models that are just designed to predict next words in sentences, since the transformer development they have quickly dominated all natural language processing tasks [19, 20]. Originally developed to be used with language, in recent years their use has been extended to images [21, 22], allowing the development of multimodal LLMs able to use images and text as inputs. This multimodality allowed them to be used for tasks such as the one implemented in this project. The Gemini models are based on the mixture-of-experts transformer architecture [23–25]. For the complete set of images, the number of equivalent tokens processed was: 7.2M (GPT-4.1-nano), 4.7M (GPT- 4.1-mini), 2M (GPT-4.1), 1.7M (GPT-5), and 2.9M (Gemini 2.5 Flash-Lite). The prompts used for the models consisted of 3 strings concatenated together. The first part defined the AI persona, the data, and the task to be carried out. There were 2 versions of this first part (called “multiple” and “single”), designed to use either the full set of record images or just the first image containing the title page. The second part of the prompt included hints given to the models to facilitate the task. There were 2 versions of this second part (called “simple” and “advanced”), including or not references to the use of authoritative names and catalogue records. The third part of the prompt was used to specify the output format (“MARC”, “JSON”, “BIBFRAME”). In total, considering all possible combinations, there were 12 variations of the prompts. A total of 60 experiments were carried out, applying for each of the 5 models the 12 versions of the prompts. After applying the 60 experiments to each of the 87 records, we obtained a total of 5220 outputs. Quantitative Evaluation Ground Truth To evaluate the results of the experiments, we used the manually produced catalogue cards for each dissertation. Each of the dissertation folders also contained a scanned image of the corresponding card. The extracted content of the card was compared with the outputs produced by the AI models from the dissertation page images. Extracting card content used a similar AI-driven process to the one described above for the dissertations as whole. However, this is a much simpler task since a card only contains human-curated catalogue information, and it is already largely segregated by the card layout. This use-case primarily tests handwriting recognition and the sorting of data into the correct fields, for a sample with very neat handwriting and easily distinguishable data characteristics (title, author, data, language and place). AI tools were expected to (and did) perform well. The OpenAI GPT-4.1 model was applied to the card images to extract their output and generate the different formats for use in the evaluation. EvaluationTo evaluate the correspondence between the ground truth and the generated outputs, we used 3 metrics: • Jaccard similarity. • Semantic similarity. • BLEU score. The Jaccard similarity [26] evaluates the overlap between the sets of words of the ground truth and the output. It is calculated as the number of words in the intersection of the sets divided by the number of words in the union of the sets. The semantic similarity [27] was implemented using the SentenceTransformer library * [28] with the ‘all-MiniLM- L6-v2’ model to calculate the embedding representation of ground truth and output, and then calculate the cosine product between both as the final metric score. The embedding representation of a text is a numerical representation of a text produced using an AI model. Each numerical representation can be understood as a vector. The cosine similarity evaluates how similar the vectors are by evaluating how different the directions they point to are. The more different the texts, the more their numerical vector representations will diverge. Unlike the previous metrics, which evaluate the similarity between literal words, these vectors produce a semantic representation of the texts as a whole, independently of the words used. The BLEU score [29] was implemented using the sacreBLEU library † [30]. This score computes a similarity score based on N-gram precision and output length. Using these methods, we are able to evaluate more than 5000 outputs produced, which would have been impossible to do using a completely manual evaluation. In addition to these main evaluations, we conducted a brief analysis using a subset of 13 dissertations that had been recently re-catalogued more thoroughly. While this only evaluates part of the full 87 record dataset and one of the output formats (MARC), this comparison is against what could be considered a “gold-standard" human-sourced record. Qualitative Evaluation The data fields from the experiments were also extracted from their respective formats and loaded into Excel spreadsheets for visual inspection. While the quantitative analysis can identify broad performance characteristics of the algorithms under test, it does not provide an insight into the nature of the variance between results, and will statistically suppress occasional “hallucinations". This part of the evaluation therefore focuses on the character and consistency of variations rather than outright performance at scale as an indication of the nature of any intervention or monitoring that might be necessary. ∗ https://sbert.net/ † https://github.com/mjpost/sacrebleu Prepared using sagej.cls 3 Results The experiments carried out have been evaluated both quantitatively and qualitatively. We present these results below. Quantitative Evaluation The results of the experiments are presented in Tables 1, 2, and 3. Table 1. Results using MARC format ModelpromptimagesJaccard Sem. BLEU GPT-4.1advanced multiple 0.1920.793 0.485 GPT-4.1advanced single0.1820.652 0.476 GPT-4.1-miniadvanced multiple 0.3240.794 0.738 GPT-4.1-miniadvanced single0.3360.807 0.749 GPT-4.1-nanoadvanced multiple 0.2490.787 0.650 GPT-4.1-nanoadvanced single0.2560.771 0.671 GPT-5advanced multiple 0.3220.804 0.681 GPT-5advanced single0.3270.802 0.712 GPT-4.1simplemultiple 0.2500.816 0.577 GPT-4.1simplesingle0.2470.802 0.582 GPT-4.1-minisimplemultiple 0.3260.811 0.742 GPT-4.1-minisimplesingle0.3410.823 0.755 GPT-4.1-nanosimplemultiple 0.2590.799 0.647 GPT-4.1-nanosimplesingle0.2850.816 0.690 GPT-5simplemultiple 0.3110.813 0.693 GPT-5simplesingle0.3010.823 0.640 Gemini-2.5-FL advanced multiple 0.2650.796 0.611 Gemini-2.5-FL advanced single0.2300.767 0.537 Gemini-2.5-FL simplemultiple 0.2740.812 0.619 Gemini-2.5-FL simplesingle0.2450.798 0.600 Table 2. Results using JSON format ModelpromptimagesJaccard Sem. BLEU GPT-4.1advanced multiple 0.1380.722 0.156 GPT-4.1advanced single0.1490.705 0.169 GPT-4.1-miniadvanced multiple 0.3500.837 0.527 GPT-4.1-miniadvanced single0.3490.744 0.532 GPT-4.1-nanoadvanced multiple 0.2850.796 0.464 GPT-4.1-nanoadvanced single0.2980.785 0.484 GPT-5advanced multiple 0.3450.823 0.493 GPT-5advanced single0.3530.847 0.523 GPT-4.1simplemultiple 0.2790.762 0.383 GPT-4.1simplesingle0.3120.760 0.445 GPT-4.1-minisimplemultiple 0.3370.821 0.485 GPT-4.1-minisimplesingle0.3560.721 0.528 GPT-4.1-nanosimplemultiple 0.2850.800 0.449 GPT-4.1-nanosimplesingle0.3030.800 0.483 GPT-5simplemultiple 0.3290.816 0.468 GPT-5simplesingle0.3340.825 0.487 Gemini-2.5-FL advanced multiple 0.3150.717 0.469 Gemini-2.5-FL advanced single0.3140.682 0.479 Gemini-2.5-FL simplemultiple 0.3060.718 0.437 Gemini-2.5-FL simplesingle0.3030.690 0.482 Additionally, we present the average of the previous results averaged by model, in Tables 4, 5, and 6. It can be seen that GPT-4.1-mini obtains the top results in most of the categories, followed by GPT-5. In particular, Table 3. Results using BIBFRAME format ModelpromptimagesJaccard Sem. BLEU GPT-4.1advanced multiple 0.1830.601 0.456 GPT-4.1advanced single0.1730.407 0.478 GPT-4.1-miniadvanced multiple 0.2060.837 0.591 GPT-4.1-miniadvanced single0.2350.860 0.614 GPT-4.1-nanoadvanced multiple 0.1470.818 0.438 GPT-4.1-nanoadvanced single0.1730.823 0.462 GPT-5advanced multiple 0.2490.807 0.541 GPT-5advanced single0.2520.816 0.567 GPT-4.1simplemultiple 0.2020.662 0.486 GPT-4.1simplesingle0.1990.465 0.518 GPT-4.1-minisimplemultiple 0.2090.882 0.602 GPT-4.1-minisimplesingle0.2440.896 0.615 GPT-4.1-nanosimplemultiple 0.1400.809 0.419 GPT-4.1-nanosimplesingle0.1720.845 0.453 GPT-5simplemultiple 0.2640.850 0.540 GPT-5simplesingle0.2620.846 0.545 Gemini-2.5-FL advanced multiple 0.0460.600 0.188 Gemini-2.5-FL advanced single0.0870.686 0.308 Gemini-2.5-FL simplemultiple 0.0250.597 0.125 Gemini-2.5-FL simplesingle0.0750.621 0.242 Table 4. Results using MARC format - Average by model JaccardSemanticBLEU GPT-4.10.2180.7660.530 GPT-4.1-mini 0.3320.8090.746 GPT-4.1-nano0.2620.7930.664 GPT-50.315 0.8100.681 Gemini-2.5-FL0.2530.7930.592 Table 5. Results using JSON format - Average by model JaccardSemanticBLEU GPT-4.10.2190.7370.288 GPT-4.1-mini 0.3480.7810.518 GPT-4.1-nano0.2930.7950.470 GPT-50.340 0.8280.493 Gemini-2.5-FL0.3090.7020.467 Table 6. Results using BIBFRAME format - Average by model JaccardSemanticBLEU GPT-4.10.1890.5340.484 GPT-4.1-mini0.223 0.8690.605 GPT-4.1-nano0.1580.8240.443 GPT-50.2570.8300.548 Gemini-2.5-FL0.0580.6260.216 the latter seems to obtain better results for the semantic metric, while the former produces better results for Jaccard and BLEU. In Tables 7, 8, and 9 are presented the results averaged by prompt category. Table 7. Results using MARC format - Average by prompt JaccardSemanticBLEU advanced0.2680.7770.631 simple0.284 0.8110.654 multiple 0.277 0.8020.644 single0.2750.7860.641 Prepared using sagej.cls 4Journal Title X(X) Table 8. Results using JSON format - Average by prompt JaccardSemanticBLEU advanced0.2900.7660.430 simple0.314 0.7710.465 multiple0.297 0.7810.433 single0.3070.7560.461 Table 9. Results using BIBFRAME format - Average by prompt JaccardSemanticBLEU advanced0.1750.7250.464 simple0.179 0.7470.454 multiple0.167 0.7460.439 single0.1870.7260.480 The “simple” version of the prompt is, in most of the cases, producing better results than the “advanced” one. The single-image case works better than using all the images, for BIBFRAME and JSON, and in any case presents very close results to the multiple-image case, while drastically reducing the computational cost. Looking at the output formats, the three formats produce very similar results with respect to the semantic metric. Regarding Jaccard similarity, MARC and JSON produce better results than BIBFRAME. Considering the BLEU score, MARC obtains better results than JSON and BIBFRAME. In summary, MARC seems to produce the best results, followed by JSON, and then by BIBFRAME. As we mentioned in the previous section, we conducted an additional evaluation using a subset of 13 manually created MARC records. Those evaluation results are presented in Table 10. Table 10. Results using MARC format and the human-created records ModelpromptimagesJaccard Sem. BLEU GPT-4.1advanced multiple 0.1440.562 0.157 GPT-4.1advanced single0.1320.577 0.132 GPT-4.1-miniadvanced multiple 0.1110.552 0.092 GPT-4.1-miniadvanced single0.1340.578 0.115 GPT-4.1-nanoadvanced multiple 0.0960.518 0.072 GPT-4.1-nanoadvanced single0.1200.515 0.094 GPT-5advanced multiple 0.1290.552 0.122 GPT-5advanced single0.1380.573 0.112 GPT-4.1simplemultiple 0.1270.589 0.131 GPT-4.1simplesingle0.1390.558 0.133 GPT-4.1-minisimplemultiple 0.1110.561 0.100 GPT-4.1-minisimplesingle0.1380.567 0.113 GPT-4.1-nanosimplemultiple 0.0900.448 0.072 GPT-4.1-nanosimplesingle0.0930.502 0.080 GPT-5simplemultiple 0.1120.564 0.097 GPT-5simplesingle0.1070.508 0.082 Gemini-2.5-FL advanced multiple 0.1080.548 0.093 Gemini-2.5-FL advanced single0.1100.489 0.118 Gemini-2.5-FL simplemultiple 0.1050.583 0.100 Gemini-2.5-FL simplesingle0.1180.598 0.116 In this evaluation, GPT-4.1 obtains the best results, followed closely by GPT-4.1-mini for the Jaccard and semantic metrics, and with a larger difference for the BLEU metric. The “advanced” version of the prompt obtains a better result for Jaccard and BLEU, but not for the semantic metric. The same happens for the single-image cases. Qualitative Evaluation Manual inspection of the outputs of the various experiments identified a number of characteristic idiosyncrasies for the various AI tools that appeared regardless of the model used. These are noteworthy in that they are not generally the type of behaviour that would be expected of a human cataloguer. We did not encounter “hallucinations", where the model deviated completely from the task at hand, but more subtle variations which are harder to detect as part of a quality assurance process. Any single record from the test dataset exhibits many of these behaviours. As an illustration, we will present extracts from the outputs of the following experiments, which involve the same item (shelfmark: Diss. D 1,2), same input image (title page only), same prompt (simple instructions), same model (GPT-4.1), but different output formats requested (MARCXML, JSON and BIBFRAME). This has not been specially chosen to illustrate these points but was simply the first item processed in our tests. MARCXML Abbreviated extract from full record < d a t a f i e l d t a g = " 1 0 0 " i n d 1 = " 1 " i n d 2 = " " > < s u b f i e l d c o d e = " a " > F a g e l , F r a n c o i s , < / s u b f i e l d > < s u b f i e l d c o d e = " e " > a u t h o r . < / s u b f i e l d > < / d a t a f i e l d > < d a t a f i e l d t a g = " 2 4 5 " i n d 1 = " 1 " i n d 2 = " 0 " > < s u b f i e l d c o d e = " a " > S p e c i m e n j u r i s p u b l i c i i n a u g u r a l e d e g u a r a n t i a f o e d e r u m , < / s u b f i e l d > < / d a t a f i e l d > < d a t a f i e l d t a g = " 3 0 0 " i n d 1 = " " i n d 2 = " " > < s u b f i e l d c o d e = " a " > 1 v o l u m e ( u n p a g e d ) < / s u b f i e l d > < s u b f i e l d c o d e = " c " > 22 cm < / s u b f i e l d > < / d a t a f i e l d > < d a t a f i e l d t a g = " 5 0 2 " i n d 1 = " " i n d 2 = " " > < s u b f i e l d c o d e = " b " > D o c t o r o f Laws ( J u r i s U t r i u s q u e D o c t o r ) , < / s u b f i e l d > < s u b f i e l d c o d e = " c " > L e i d e n U n i v e r s i t y , < / s u b f i e l d > < s u b f i e l d c o d e = " d " > 1 7 5 9 < / s u b f i e l d > < / d a t a f i e l d > < d a t a f i e l d t a g = " 7 1 0 " i n d 1 = " 2 " i n d 2 = " " > < s u b f i e l d c o d e = " a " > U n i v e r s i t e i t L e i d e n . < / s u b f i e l d > < s u b f i e l d c o d e = " e " > d e g r e e g r a n t i n g i n s t i t u t i o n . < / s u b f i e l d > < / d a t a f i e l d > JSON Full record " T i t l e " : " S p e c i m e n J u r i s P u b l i c i I n a u g u r a l e d e G u a r a n t i a F o e d e r u m " , " A u t h o r " : " F r a n c o i s F a g e l " , " D a t e o f P u b l i c a t i o n " : " 1 7 5 9 " , " P l a c e o f P u b l i c a t i o n " : " L u g d u n i B a t a v o r u m " , Prepared using sagej.cls 5 " L a n g u a g e " : " L a t i n " , " D e g r e e " : " D o c t o r a t u s i n U t r i u s q u e J u r e ( D o c t o r o f Law ) " , " D e g r e e g r a n t i n g i n s t i t u t i o n " : " A c a d e m i a Lugduno − B a t a v a ( L e i d e n U n i v e r s i t y ) " , " S u b j e c t C l a s s i f i c a t i o n " : " I n t e r n a t i o n a l l a w − T r e a t i e s ; Law − N e t h e r l a n d s − H i s t o r y − 18 t h c e n t u r y " BIBFRAME Abbreviated extract from full record < b f : P e r s o n > < r d f s : l a b e l > F r a n c o i s F a g e l < / r d f s : l a b e l > < b f : l a b e l > F a g e l , F r a n c o i s < / b f : l a b e l > < / b f : P e r s o n > <! − − Work e n t i t y f o r t h e d i s s e r t a t i o n −> < b f : Work> < b f : t i t l e >< b f : T i t l e > < b f : m a i n T i t l e > S p e c i m e n j u r i s p u b l i c i i n a u g u r a l e d e G u a r a n t i a F oe de ru m < / b f : m a i n T i t l e > < / b f : T i t l e > < / b f : t i t l e > < b f : c o n t r i b u t i o n >< b f : C o n t r i b u t i o n > < b f : a g e n t r d f : r e s o u r c e = " h t t p : / / e x a m p l e . o r g / p e r s o n / f a g e l − f r a n c o i s " / > < b f : r o l e >< b f : R o l e > < r d f s : l a b e l > a u t h o r < / r d f s : l a b e l > < / b f : R o l e > < / b f : r o l e > < / b f : C o n t r i b u t i o n > < / b f : c o n t r i b u t i o n > < b f : s u b j e c t >< b f : T o p i c r d f : a b o u t = " h t t p : / / i d . l o c . g o v / a u t h o r i t i e s / s u b j e c t s / s h 8 5 1 3 0 6 1 1 " > < r d f s : l a b e l > T r e a t i e s < / r d f s : l a b e l > < / b f : T o p i c > < / b f : s u b j e c t > < b f : s u b j e c t >< b f : T o p i c r d f : a b o u t = " h t t p : / / i d . l o c . g o v / a u t h o r i t i e s / s u b j e c t s / s h 8 5 1 9 1 2 2 7 " > < r d f s : l a b e l > I n t e r n a t i o n a l law < / r d f s : l a b e l > < / b f : T o p i c > < / b f : s u b j e c t > <! − − D e g r e e i n f o r m a t i o n −> < b f : d i s s e r t a t i o n >< b f : D i s s e r t a t i o n > < b f : d e g r e e >< b f : D e g r e e > < r d f s : l a b e l > D o c t o r o f Law ( J u r i s D o c t o r , D o c t o r a t u s ) < / r d f s : l a b e l > < / b f : D e g r e e > < / b f : d e g r e e > < b f : g r a n t i n g I n s t i t u t i o n > < b f : O r g a n i z a t i o n r d f : a b o u t = " h t t p : / / i d . l o c . g o v / a u t h o r i t i e s / n ames / n 7 9 0 1 8 1 7 3 " > < r d f s : l a b e l > U n i v e r s i t a s Lugduno − B a t a v a ( L e i d e n U n i v e r s i t y ) < / r d f s : l a b e l > < / b f : O r g a n i z a t i o n > < / b f : g r a n t i n g I n s t i t u t i o n > < / b f : D i s s e r t a t i o n > < / b f : d i s s e r t a t i o n > < / b f : Work> <! − − I n s t a n c e e n t i t y f o r t h e p h y s i c a l d i s s e r t a t i o n −> < b f : I n s t a n c e > < b f : i n s t a n c e O f r d f : r e s o u r c e = " h t t p : / / e x a m p l e . o r g / work / f a g e l − g u a r a n t i a − f o e d e r u m " / > < b f : p r o v i s i o n A c t i v i t y >< b f : P r o v i s i o n A c t i v i t y > < b f : p l a c e >< b f : P l a c e > < r d f s : l a b e l > L u g d u n i B a t a v o r u m [ L e i d e n , N e t h e r l a n d s ] < / r d f s : l a b e l > < / b f : P l a c e > < / b f : p l a c e > < b f : d a t e > 1 7 7 0 < / b f : d a t e > < / b f : P r o v i s i o n A c t i v i t y > < / b f : p r o v i s i o n A c t i v i t y > < / b f : I n s t a n c e > Non-intuitive Prompt Sensitivity By comparing these records we can see that changing the output format has a non- intuitive effect on the way that information is extracted from the page and formatted. In MARCXML, the dissertation title is rendered in lower case with an initial capital, in JSON the major words are capitalised and in BIBFRAME only the primary topic is capitalised. In the original, the entire title is printed in upper case. MARCXML data also seems to include trailing punctuation for some reason. Data Synthesis In the MARCXML record, datafield tag 300 corresponds to physical information about the original object. The GPT model has synthesised a plausible but completely fictitious dimension of 22cm since the digitised images submitted for scanning have had the scale ruler cropped out. The prompt did not include physical information in the required MARCXML fields. In the BIBFRAME record, all the authoritative links to Library of Congress resources were either fictitious (https://id.loc.gov/authorities/subjects/sh85191227.html) or incorrect(http://id.loc.gov/authorities/names/n79018173 refers to a person rather than Leiden University), although of the correct form to pass casual inspection. Date “Blindness" In the BIBFRAME record, the change in required output format causes the algorithm to fail to recognise the date correctly, generating 1770 rather than the (correct) value of 1759 seen in the MARCXML and JSON outputs. Incorrect extraction of dates seemed to be particularly prevalent. Translation In addition to failures, the models also exhibited some emergent behaviours that were not part of the prompted tasks, but might be potentially useful or counterproductive depending on circumstances. The source documents were primarily in Latin but, in many cases, the algorithms helpfully expanded and translated terms into modern languages, albeit somewhat at random. In the source document for the records above, the academic institution involved is referenced by the terms “Lugduni Batvorum" and “Acad. Lugd. Bat." which was correctly interpreted as Leiden University in all three cases. However, the MARCXML record also references “Universiteit Lieden" in one place - which is linguistically appropriate for the institution but confusing given that English is used elsewhere. In BIBFRAME we see the incorrect resynthesised pseudo- Latinate “Universitas Lugduno-Batava", whereas “Academia Lugduno-Batava (Leiden University)" in the JSON records, expands the Latin abbreviation correctly. Inconsistency An overriding observation about these behaviours is that they do not appear consistently, as can be seen from analysing just a single record in the test dataset. A prompt/output format that performs well for one item may not be optimal for another, ostensibly similar, item. The plausibility of many of these variant results makes error detection difficult. Another type of inconsistency encountered in the project is the tendency for the algorithms to stray from the tasks specified in the prompt. Data is falsified in an attempt to generate fuller MARCXML and BIBFRAME records despite prompts that indicate precisely which fields are required. Conversely, unrequested translations revealed a potentially useful capability. Discussion The qualitative and quantitative assessments of the experi- ments described above have led to the following points for discussion and recommendations regarding the use of AI models. Prepared using sagej.cls 6Journal Title X(X) Larger Models and More Data is not Always Better The project tested several models from OpenAI as well as some from other vendors. The quantitative comparison of the outputs of the various algorithms found that GPT-4.1-mini performed marginally better than either GPT-4.1 or GPT-5.1. Secondly, there is no benefit in using all the scanned images instead of just the title page, and in some cases doing so makes the results worse. Larger and newer models do not necessarily perform better depending on use-case. Smaller models are con- siderably cheaper, and more environmentally-friendly, than larger models. There is no guarantee that newer models will perform better or as well as older models. This should be considered when building workflows that are dependent on particular models or tools. Complex Tasks Generating modern digital records is more complex than simple transcription, involving the use of authoritative references, controlled vocabularies, etc., which proved too complex for a single prompt despite being well within notional context windows for the algorithms concerned. Complex tasks need to be broken down into simpler tasks, some of which may be suitable for automation using automated AI (or other) tools. This is probably best accomplished by an expert in the field, in conjunction with a business analyst. Emergent Behaviours and Hallucinations Tools could randomly perform additional tasks that were not specified in the prompt. In some cases, these were “hallucinations” such as synthesising physical data where none existed to conform to the algorithm’s interpretation of record completeness. At other times, it revealed a potentially useful capability, such as translating the Latin name of a degree-awarding institution into modern English or Dutch. Identifying such cases and redefining tasks and prompts to minimise “hallucinations” and leverage helpful emergent behaviours is an essential part of operationalising tool use. Prompt Sensitivity “Prompt engineering” is more of an art than a science, based on empirical evidence of what seems to work rather than a deep understanding of the internal mechanics of LLMs. Crafting effective prompts thus requires a certain amount of trial-and-error with realistic test datasets to get optimal performance. Our testing revealed that performance is not only dependent on prompt construction in readily understandable ways – for example, overly complex task specifications are not always helpful (more complex and detailed prompting showed improvements in some metrics and regressions in others) – but also in unexpected ways – changing the required output format for metadata causes the model to handle dates less reliably. In quantitative terms, MARC seems to produce the best results, followed by JSON, and then by BIBFRAME. At an individual record level, changing the output format has a significant non-intuitive impact on how data is extracted from the page images and processed into metadata records. A prompt that works well for one model does not necessarily work well for another model, so any time a model is changed or upgraded, a re-engineering effort is required to ensure performance is not compromised. This is a potentially non-trivial cost because of rapid model evolution. Inconsistency Both manual and AI processes are likely to result in a certain level of error. However, the nature of the errors generated by LLM-based tools does differ qualitatively from human errors in that they are less consistent and logical. Whereas human errors are frequently systematic, the appearance of hallucinations and helpful emergent behaviours can vary on a record-by-record basis, which makes error checking and correction more difficult either by manual or algorithmic means. Since correction is relatively expensive, the goal in operationalising LLM-based tools should be to first minimise the inconsistency in output to acceptable levels. Conclusions This workstream was undertaken as a research activity to build in-house understanding of LLM-based tools. The use case was processing digitised documents and catalogue cards with a view to automatically generating catalogue records suitable for use in Library Systems. This revealed a number of characteristics of these AI tools which should be borne in mind when considering their operational implementation. In addition to the qualitative analysis of the results and the recommendations developed based on it, in this article we have presented a quantitative analysis of the experiments carried out, which underpin these recommendations and highlight practical aspects in the implementation of LLMs in use cases similar to that presented in this project. Overall, LLM-based tools showed promise but require careful, use-case specific implementations assisted by domain experts, the acceptance of an unavoidable level of inconsistency/inaccuracy and the expectation that ongoing maintenance will be required. References [1] Mahmud MR.Ai in automating library cataloging and classification. Library Hi Tech News 2024; . [2] Engel JY, Do DT, Salem B et al. Artificial intelligence in library cataloging: A review of literature. Journal of Library Metadata 2025; 25(4): 261–276. [3] Osmani I. Machine learning for metadata creation in libraries: An exploratory study of ai tools for subject cataloguing and classification, 2025. [4] Martorana M, Kuhn T, Stork L et al.Zero-shot topic classification of column headers: Leveraging llms for metadata enrichment. In Knowledge Graphs in the Age of Language Models and Neuro-Symbolic AI. IOS Press, 2024. p. 52–66. [5] González-Espinoza A, Jebbia D and Lan H.Metadata augmentation using nlp, machine learning and ai-chatbots: A comparison. Journal of Library Metadata 2025; : 1–17. Prepared using sagej.cls 7 [6] Golub K, Suominen O, Mohammed AT et al. Automated dewey decimal classification of swedish library metadata using annif software. Journal of Documentation 2024; 80(5): 1057–1079. [7] Lorang E, Soh LK, Liu Y et al. Digital libraries, intelligent data analytics, and augmented description: A demonstration project, 2020. [8] Busch L, Tebernum D and Velarde G.Exploring llm capabilities in extracting dcat-compatible metadata for data cataloging. arXiv preprint arXiv:250705282 2025; . [9] Song X, Luo P, Thomale J et al. Comparative analysis of large language models’ performance in book classification tasks using library of congress classification system. Journal of Information Science 2026; : 01655515261425547. [10] Radford A, Narasimhan K, Salimans T et al.Improving language understanding by generative pre-training, 2018. [11] Radford A, Wu J, Child R et al.Language models are unsupervised multitask learners. OpenAI blog 2019; 1(8): 9. [12] Brown T, Mann B, Ryder N et al. Language models are few-shot learners. Advances in neural information processing systems 2020; 33: 1877–1901. [13] Achiam J, Adler S, Agarwal S et al. Gpt-4 technical report. arXiv preprint arXiv:230308774 2023; . [14] Team G, Anil R, Borgeaud S et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:231211805 2023; . [15] Team G, Georgiev P, Lei VI et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:240305530 2024; . [16] Comanici G, Bieber E, Schaekermann M et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:250706261 2025; . [17] Vaswani A, Shazeer N, Parmar N et al. Attention is all you need. Advances in neural information processing systems 2017; 30. [18] Galassi A, Lippi M and Torroni P.Attention in natural language processing. IEEE transactions on neural networks and learning systems 2020; 32(10): 4291–4308. [19] Zhao WX, Zhou K, Li J et al. A survey of large language models. arXiv preprint arXiv:230318223 2023; 1(2): 1–124. [20] Yang J, Jin H, Tang R et al. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data 2024; 18(6): 1–32. [21] Dosovitskiy A, Beyer L, Kolesnikov A et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929 2020; . [22] Han K, Wang Y, Chen H et al.A survey on vision transformer.IEEE transactions on pattern analysis and machine intelligence 2022; 45(1): 87–110. [23] Shazeer N, Mirhoseini A, Maziarz K et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:170106538 2017; . [24] Fedus W, Zoph B and Shazeer N.Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research 2022; 23(120): 1–39. [25] Riquelme C, Puigcerver J, Mustafa B et al. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems 2021; 34: 8583–8595. [26] Niwattanakul S, Singthongchai J, Naenudorn E et al. Using of jaccard coefficient for keywords similarity. In Proceedings of the international multiconference of engineers and computer scientists, volume 1. p. 380–384. [27] Chandrasekaran D and Mago V.Evolution of semantic similarity—a survey. Acm Computing Surveys (Csur) 2021; 54(2): 1–37. [28] Reimers N and Gurevych I. Sentence-bert: Sentence embed- dings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, p. 3982–3992.URL https://arxiv.org/abs/1908. 10084. [29] Papineni K, Roukos S, Ward T et al.Bleu: a method for automatic evaluation of machine translation.In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. p. 311–318. [30] Post M.A call for clarity in reporting bleu scores.In Proceedings of the third conference on machine translation: Research papers. p. 186–191. Prepared using sagej.cls