Paper deep dive
Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training
Lukas Borggren, Jenny Kunz, Marco Kuhlmann
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to address this limitation is to specialise existing models through additional training on target-domain corpora. In this work, we investigate such continued pre-training for adapting large language models to Swedish journalism, using a high-quality dataset that we curate from millions of news articles. To evaluate the adaptation efficacy, we also construct a novel domain-specific benchmark that covers six editorial tasks. Through full and parameter-efficient fine-tuning across two model sizes, we find that continued pre-training yields benefits in the target domain, but only when paired with experience replay to mitigate forgetting. We observe consistent enhancements in the models' generation quality and factual knowledge, but not their proficiency in discriminative tasks. Exploring a training-free method to facilitate instruction following, we see further improvements, but exclusively for models trained with low-rank adaptation. Crucially, we demonstrate the importance of targeted evaluation in the adaptation process, as an existing Swedish benchmark largely fails to capture the models' in-domain performance gains.
Tags
Links
- Source: https://arxiv.org/abs/2608.30609v1
- Canonical: https://arxiv.org/abs/2608.30609v1
Trouble viewing inline? Open PDF directly â
Full Text
84,369 characters extracted from source content.
Expand or collapse full text
Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training Lukas Borggren Affiliation: Linköping University Affiliation: Bonnier News Email: firstname.lastname@liu.se Jenny Kunz Affiliation: Linköping University Marco Kuhlmann Affiliation: Linköping University Abstract Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to address this limitation is to specialise existing models through additional training on target-domain corpora. In this work, we investigate such continued pre-training for adapting large language models to Swedish journalism, using a high-quality dataset that we curate from millions of news articles. To evaluate the adaptation efficacy, we also construct a novel domain-specific benchmark that covers six editorial tasks. Through full and parameter-efficient fine-tuning across two model sizes, we find that continued pre-training yields benefits in the target domain, but only when paired with experience replay to mitigate forgetting. We observe consistent enhancements in the modelsâ generation quality and factual knowledge, but not their proficiency in discriminative tasks. Exploring a training-free method to facilitate instruction following, we see further improvements, but exclusively for models trained with low-rank adaptation. Crucially, we demonstrate the importance of targeted evaluation in the adaptation process, as an existing Swedish benchmark largely fails to capture the modelsâ in-domain performance gains. 1 Introduction News text has historically been near-ubiquitous in natural language processing via resources like the WSJ Penn Treebank (Marcus et al., 1993), Reuters corpora (Lewis et al., 2004), and CNN/Daily Mail (Hermann et al., 2015). In the current paradigm of generative large language models (LLMs), complex problems like fake news detection (Tong et al., 2025) and personalised headline generation (Wan et al., 2026) are studied. However, journalism has rarely been treated as a target domain for LLM adaptation; to the best of our knowledge, merely one study has investigated it (Yao et al., 2024). Other work has created foundational LLMs for journalism, but focuses on pre-training from scratch for high-resource languages (Wang et al., 2023; Wu et al., 2023). Such training demands vast amounts of compute and data, posing a barrier to model development for lower-resource languages. Continued pre-training (CPT) on specialised corpora is commonly used to adapt general-purpose LLMs to domains like finance (Li et al., 2023), law (Colombo et al., 2024), and biomedicine (Labrak et al., 2024). It is similarly effective for improving models on resource-constrained languages, including the Scandinavian languages Danish Zhang et al. (2025), Norwegian (Samuel et al., 2025), Icelandic (Gogoulou et al., 2024), and Faroese (Kunz et al., 2026). For Swedish, the remaining major Scandinavian language, CPT has been explored, but with limited methodological details (AI Sweden, 2024), or for relatively small models (Glocker et al., 2025; Kunz, 2026b). Instead, efforts have mainly targeted from-scratch pre-training (Norlund and Stenbom, 2021; Ekgren et al., 2022; Ekgren et al., 2024; Kalpakchi and Boye, 2023; Silo AI, 2024). This paper aims to jointly address the understudied adaptation of LLMs to journalism and Swedish. By initially curating a high-quality corpus from millions of Swedish news articles, we can subsequently specialise models via CPT. Addressing the scarcity of evaluation resources, we introduce a domain-specific benchmark covering six editorial tasks to gauge the effectiveness of adaptation. Our results show that CPT increases modelsâ generation capabilities and parametric knowledge, but hurts their performance on discriminative tasks. Moreover, we find the data-mixture composition to be crucial for successful adaptation, observing negative results when omitting experience replay. In our experiments, low-rank adaptation is competitive with full fine-tuning, and drastically more compatible with instruct vectors to improve instruction following. Notably, an existing Swedish benchmark fails to reflect the broadly positive in-domain effects of CPT, illustrating the need for targeted evaluation. Overall, we demonstrate that CPT is a viable approach for adapting LLMs to Swedish journalism, with the potential to enable publishers to develop and retain control over their editorial tools. We also provide insights and a blueprint for researchers and practitioners seeking to specialise models for other languages and domains. 2 Background and Related Work CPT is a canonical approach to enhancing a modelâs knowledge and capabilities in some target area (Gururangan et al., 2020). A core challenge is to adapt a model without largely losing its original abilities, that is, to avoid catastrophic forgetting (Wu et al., 2025). 2.1 Data Curation Experience replay (Chaudhry et al., 2019) is commonly combined with CPT, mitigating forgetting by including a proportion as small as 1% of previously seen examples during training (Scialom et al., 2022). Additionally, replaying data may even enhance target-domain performance (Ibrahim et al., 2024; Parmar et al., 2024; Ke et al., 2025). If a base modelâs pre-training distribution is unknown, replay data like code and English text are typically sampled from openly available corpora (Chen et al., 2023; Etxaniz et al., 2024; Dorkin et al., 2026). The choice of such proxy data influences CPT efficacy, with higher-quality sources yielding larger in-domain gains (Wang et al., 2025). Training on a well-curated subset rather than the full dataset can improve results further (Xie et al., 2024; Guo et al., 2025b; Nag et al., 2025). For pre-training in general, deduplicating data increases efficiency, reduces data leakage, causes less memorisation, and can ultimately boost performance (Lee et al., 2022; Penedo et al., 2024). 2.2 Constrained Learning A complementary remedy for catastrophic forgetting is to restrain CPT to avoid excessive overwriting of knowledge (Guo et al., 2025a). This regularising effect can be achieved by architecturally constraining parameter updates (Ke et al., 2022; Ke et al., 2023; Nayak et al., 2026) or carefully configuring learning rate dynamics (Gupta et al., 2023; Parmar et al., 2024). Parameter-efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) (Hu et al., 2022) may be used to this end (Nag et al., 2025; Pezeshkpour and Hruschka, 2025; Kunz et al., 2026). Functionally restricting updates, LoRA shifts the balance of CPT to reduce forgetting at the expense of learning (Biderman et al., 2024; Elhady et al., 2025), sometimes to the extent of obstructing learning altogether (Tejaswi et al., 2024). Upscaling methods like LLaMA Pro (Wu et al., 2024), SOLAR (Kim et al., 2024), and SCALE (Lee et al., 2026) serve a similar purpose by introducing parameters to expand a modelâs width or depth, while keeping its pre-trained backbone frozen during CPT. 2.3 Instruction Following Naively performing CPT can effectively erase a modelâs instruction-following capabilities (Jindal et al., 2024). To counteract this, a CPT corpus may be augmented with instruction-like or reading-comprehension data (Shi and Lipani, 2023; Cheng et al., 2024a; Cheng et al., 2024b; RodrĂguez et al., 2025). Given the scarcity of specialised instruction data, several training-free methods instead combine a CPT-adapted model with an instruction-tuned version of its base model, relying on either model merging (Siriwardhana et al., 2024; Ueda et al., 2026) or model editing akin to task vectors (Ilharco et al., 2023). The latter subtracts the original pre-trained model from its instruction-tuned descendant in parameter space and adds the resulting instruct vector to the adapted model (Huang et al., 2024; Lin et al., 2025; Tanwar et al., 2025). 2.4 Targeted Evaluation The effectiveness of CPT is commonly assessed using existing domain-specific benchmarks (Hirano and Imajo, 2024; Ying et al., 2025; Luo et al., 2026). For journalism, although general evaluation frameworks exist (Nishal et al., 2024), previous work relies on either English financial benchmarks (Wu et al., 2023) or Chinese tasks (Wang et al., 2023; Li et al., 2024; Yao et al., 2024). In Swedish, there exist merely three evaluation tasks related to journalism: lead paragraph generation (Monsen and Jönsson, 2021), as well as summarisation and search engine-optimised (SEO) headline generation (Eide, 2024). For Swedish in general, the only existing benchmark collections are the Swedish portion of EuroEval (Nielsen, 2023) and Superlim (Berdicevskis et al., 2023), the latter primarily catered to encoder models. 3 Method Drawing on work reviewed in Section 2, we create a domain-specific corpus and evaluation benchmark, and develop a blueprint for CPT adaptation. 3.1 BonCorpus: Compiling Pre-Training Data We source the pre-training corpus from 21.7M Swedish articles published by Bonnier News publications from 1991 up until 2026. The publications include daily, evening, financial, and local newspapers, as well as lifestyle, sports, and trade magazines. Although individual texts are generally of high quality, the collection as a whole is noisy in several respects. For example, older articles originally stored in now-obsolete formats have been imperfectly migrated over time, leaving artefacts such as links in the plain-text body. Moreover, publications frequently cross-publish content, resulting in many near-duplicate articles. To alleviate these issues, we implement an extensive pre-processing pipeline. For a detailed description, see Appendix A. First, we parse and aggregate articles to standardise their format across publications. We then apply Unicode normalisation and resolve encoding inconsistencies before using regular expressions to remove extraneous text spans. To discard low-quality articles, we develop a broad set of filters based on text statistics, fastText (Joulin et al., 2017) language-identification scores, and metadata such as tags and article types. Thereafter, we perform exact and fuzzy deduplication, using MinHash LSH (Broder, 1997) to identify candidate pairs for the latter. After calculating and thresholding the Levenshtein distance for each pair, we construct a graph with the retained pairs as edges and their constituent articles as nodes. We compute the graphâs connected components to group similar articles and finally select only the longest one from each group. The full pipeline reduces the document count by nearly a third, yielding BonCorpus: 14.6M articles totalling 8.5B tokens under the tokeniser of our base model family, which is specified in Section 4. 3.2 BonEval: Constructing Evaluation Tasks Due to the dearth of evaluation resources for Swedish journalism, we construct the BonEval benchmark by repurposing editorial content into evaluation tasks. See Appendix B for more details on this process. BonEval comprises six tasks spanning three categories representative of editorial LLM application: generative tasks entailing text production, discriminative tasks targeting information extraction, and knowledge-intensive tasks involving factual accuracy. The six tasks are: 1. Headline: headline generation based on the remainder of the article text. 2. Lead: lead-paragraph generation based on the article body. 3. Summary: bullet-point summarisation of the full article text. 4. Topic: multi-class news topic classification of the full article text. The 19 labels extend the top-level terms in the Media Topics taxonomy (36). 5. Entity: salient named entity recognition (NER) on the full article text. Unlike standard NER, only persons, organisations, and locations central to the article are targeted. 6. Quiz: multiple-choice question answering based on general-knowledge quizzes. They span 15 categories and range from highly specific Swedish themes to globally known subjects. For all tasks except Quiz, we source evaluation examples from BonCorpus and subsequently exclude the corresponding articles from the training data. To minimise potential overlap with our base modelsâ original pre-training data, while still covering major quadrennial events such as elections and sports championships, we only consider articles published from 2022 onward. For Headline, Lead, Topic, and Entities, we use stratified sampling to obtain 8,192 examples per task, approximately balanced across publications. Summary and Quiz each come from a single publication, and using all available data results in 100 and 11,960 examples, respectively. 3.3 BonLM: Adapting Models To create the domain-specialised BonLM, we perform CPT with BonCorpus as the primary training data. However, because the corpus exclusively comprises Swedish editorial content, it introduces a marked distribution shift relative to existing modelsâ pre-training data. To mitigate forgetting and potentially improve adaptation, we therefore employ experience replay to construct several data mixtures. Specifically, we sample code and English text from Common Corpus, a high-quality collection of uncopyrighted and openly licensed data (Langlais et al., 2026). Although editorial text in pre-training corpora can improve general language-modelling performance (de la Rosa et al., 2025), it is unknown whether general-domain text can improve performance on editorial tasks. To investigate this, we also sample Swedish text from Common Corpus for our mixtures. Omitting the article-specific parsing and cleaning steps, we apply our pre-processing pipeline to all replay data. We curate the data mixtures by diluting BonCorpus with every possible combination of the three replay types. Guided by prior work (Fujii et al., 2024; Messmer et al., 2025; Samuel et al., 2025), we sample each included type to constitute 10% of the tokens in the resulting mixture. This yields eight mixtures of varying sizes, labelled by concatenating abbreviations for their constituents: BonCorpus (B), code (C), English (E), and Swedish (S). The full BonCorpus accounts for 100% of the tokens in the smallest mixture, B, and 70% in the largest, BCES. We also study the impact of PEFT methods on CPT by comparing LoRA and LLaMA Pro against full fine-tuning (FFT). Finally, we explore the effects of adding an instruct vector (IV) to each model. 4 Experimental Setup Following Chen et al. (2025), we conduct initial experiments with smaller models to derive a training recipe. As starting points for CPT, we use the 3B and 8B base models from Mistralâs Ministral 3 family (Liu et al., 2026). We defer details of our training setup to Appendix C. 4.1 Evaluation We assess the effectiveness of CPT with BonEval. To validate our benchmark, we compare its results to those obtained on three existing tasks. These supplementary tasks are similar to ours but smaller in scale: article summarisation and SEO headline generation (Eide, 2024), as well as question answering about Sweden-related facts (Kunz, 2026a). Additionally, we evaluate models on seven Swedish tasks from EuroEval: SweReC for sentiment classification, SUC 3.0 for NER, ScaLA for linguistic acceptability, MultiWikiQA for reading comprehension, SweDN for summarisation, MMLU for knowledge, and HellaSwag for common-sense reasoning (EuroEval, 2026). Among these, SweDN partially overlaps with BonCorpus. For comparability, we implement BonEval following EuroEvalâs methodology, using few-shot prompting, multiple evaluation rounds, and the same primary metrics (Nielsen et al., 2025). Specifically, we evaluate Headline, Lead, and Summary with chrF3++ (PopoviÄ, 2017); Topic and Quiz with Matthews correlation coefficient (MCC); and Entity with micro-averaged F1 score. See Appendix D for implementation details. 4.2 Initial Experiments For each of the eight data mixtures, we run CPT on Ministral 3 3B for 8,000 steps, approximating one full epoch over B; the other mixtures are larger and thus subsampled. Based on the average BonEval score across tasks, we select the best-performing mixture for subsequent experiments. Thereafter, we perform CPT with LoRA and LLaMA Pro, configuring each method to approximately render equal numbers of trainable parameters. For the resulting models, we examine the effect of adding unweighted IVs. Since LLaMA Pro alters the model architecture, there is no straightforward way to apply IV, and preliminary experimentation yielded poor outcomes. We also explored adding analogously computed reasoning vectors, but they generally underperformed their instruct counterparts. 4.3 Final Training Model Average Ministral 3 3B Base 39.73 + BonCorpus (B) 36.98 + Code (BC) 39.99 + English (BE) 40.01 + Swedish (BS) 40.06 + Code + English (BCE) 40.01 + Code + Swedish (BCS) 41.34 + English + Swedish (BES) 39.02 + Code + English + Swedish (BCES) 41.48 LoRA 41.32 LLaMA Pro 41.17 Full fine-tuning (FFT) + instruct vector (IV) 39.07 LoRA + IV 42.27 Table 1: Average BonEval scores from 3B experiments. The best mixture (BCES) is used for the results in the last four rows. Underline denotes section-best and bold overall best. See full results in Table 11, Appendix E. Model SweReC SUC 3.0 ScaLA MultiWikiQA SweDN MMLU HellaSwag Average GPT-SW3 6.7B 10.44 26.72 9.84 69.47 30.62 2.92 2.05 21.72 Llama SW3 8B 79.90 36.02 8.81 72.22 32.23 17.02 5.64 35.98 Apertus 8B 79.11 45.74 34.34 74.05 33.39 41.54 32.54 48.67 Ministral 3 8B Base 80.19 66.44 51.70 75.71 35.30 58.10 43.68 58.73 Instruct 77.53 60.51 50.62 74.28 35.31 56.22 57.27 58.82 Reasoning 77.05 65.21 47.77 79.44 31.16 49.96 64.27 59.26 BonLM 8B FFT 80.01 50.64 39.61 70.53 31.94 30.62 17.66 45.86 FFT + IV 79.07 45.84 22.54 70.04 36.03 20.09 12.48 40.87 LoRA 80.55 66.86 59.85 69.25 35.03 53.34 36.82 57.39 LoRA + IV 78.27 62.27 54.24 70.05 36.27 47.18 52.06 57.19 Table 2: Results for BonLM 8B and baselines on Swedish EuroEval tasks. Micro-averaged F1 is reported for SUC 3.0, token-averaged F1 for MultiWikiQA, and chrF3++ for SweDN. For all other tasks, MCC is reported. Underline denotes section-best and bold overall best. To obtain BonLM 8B, we run CPT on Ministral 3 8B using the best recipe from the initial experiments. We set the number of training steps to the equivalent of four epochs, a duration shown to be effective for CPT in prior work (Muennighoff et al., 2023; Xue et al., 2023; Guo et al., 2025b). Thereafter, we evaluate the resulting model on all tasks and compare it with similarly sized baselines: the base, instruct, and reasoning versions of Ministral 3 8B; GPT-SW3 6.7B, pre-trained from scratch on Scandinavian-language data (Ekgren et al., 2024); Llama SW3 8B, adapted to the Scandinavian languages via CPT (AI Sweden, 2024); and Apertus 8B, pre-trained on multilingual data (HernĂĄndez-Cano et al., 2026). 5 Results and Analysis The main experimental results are presented in Tables 1, 2, and 3, with additional details provided in Appendix E. 5.1 Impact of Data-Mixture Composition Table 1 reports average BonEval scores from the initial 3B experiments. Overall, CPT improves upon the base model for all mixtures except B and BES. B exhibits the greatest degradation, likely reflecting catastrophic forgetting caused by the mixtureâs homogeneous, BonCorpus-only composition. Conversely, BCES includes all text types and achieves the highest score, narrowly outperforming BCS. These observations corroborate the findings reviewed in Section 2.1 that replaying code and English text enhances target-domain performance. Furthermore, the results suggest that replaying general-domain text benefits performance on editorial tasks. The task-level breakdown in Table 11, Appendix E, show that adding Swedish text to any mixture consistently raises scores on Lead and especially Quiz. We surmise that the gain on Quiz arises because Common Corpus includes knowledge-dense sources such as Swedish Wikipedia, which cover detailed facts that complement those typically found in news articles. For the remaining experiments, we perform CPT with the BCES mixture. 5.2 PEFT and Instruct-Vector Compatibility Among the PEFT methods in Table 1, LoRA slightly outperforms LLaMA Pro despite having fewer trainable parameters, as detailed in Appendix C.1. Although this result diverges from the findings of Wu et al. (2024), later work has reported better target-domain adaptation with LoRA than with LLaMA Pro (Lee et al., 2026). Adding an IV to the LoRA model further improves performance, yielding the highest average BonEval score. By contrast, adding an IV to the FFT model impairs performance, contrary to prior work described in Section 2.3. To examine LoRAâs regularising effect, we compute the mean relative Euclidean distance between the adapted and base-model parameters, obtaining 0.08 for LoRA and 0.18 for FFT. We hypothesise that this smaller drift partly explains LoRAâs compatibility with IV, as the resulting model remains closer in parameter space to the model that originally underwent instruction tuning. Why IV is counterproductive for the FFT model remains unclear. Because prior studies have focused primarily on Llama and Qwen models, IV compatibility may vary across model families, but further investigation is needed to confirm this. Given that FFT outperforms PEFT without IV, and LoRA with IV performs best overall, we train one BonLM 8B variant using each method. Model BonEval Supplementary Headline Lead Summary Topic Entity Quiz Average SvD AB Facts GPT-SW3 6.7B 17.70 22.78 36.11 0.49 27.50 6.00 18.43 15.96 33.87 1.69 Llama SW3 8B 19.35 29.13 40.51 68.28 26.17 61.30 40.79 18.57 36.17 23.54 Apertus 8B 18.61 23.09 40.93 70.95 22.81 61.05 39.57 21.09 40.07 18.38 Ministral 3 8B Base 18.35 25.79 42.49 72.84 29.97 60.12 41.59 20.08 41.88 20.62 Instruct 29.28 32.03 46.27 71.93 23.20 57.76 43.41 30.41 45.10 25.21 Reasoning 21.36 27.68 47.93 71.06 20.67 52.91 40.27 11.86 42.86 19.27 BonLM 8B FFT 21.58 29.88 43.76 72.40 28.34 68.86 44.14 21.81 39.80 40.82 FFT + IV 24.32 31.24 43.26 42.28 28.38 61.98 38.58 22.79 40.29 32.41 LoRA 21.29 31.07 44.35 68.93 29.81 71.36 44.47 23.85 43.60 42.24 LoRA + IV 29.76 35.01 49.88 70.54 23.99 66.95 46.02 30.14 47.18 38.56 Table 3: Results for BonLM 8B and baselines on BonEval and supplementary tasks Svenska Dagbladet (SvD) SEO Headline, Aftonbladet (AB) Summary, and SwedishFacts. Micro-averaged F1 is reported for Entity, and MCC for Topic, Quiz, and SwedishFacts. For all other tasks, chrF3++ is reported. 5.3 Mixed Results on EuroEval The results for BonLM 8B and the baselines on the Swedish EuroEval tasks are shown in Table 2. Ministral Base outperforms the three other baselines across all tasks. It also surpasses its post-trained descendants on all tasks except reading comprehension, summarisation, and common-sense reasoning. On average, both BonLMs underperform their base model, plausibly due to CPT-induced catastrophic forgetting. Similarly, LoRAâs substantial outperformance of FFT may stem from the methodâs regularising effect described in Section 2.2. For individual tasks, CPT causes drastic regressions on the LLM-translated MMLU and HellaSwag, and a more modest decline on the LLM-synthesised MultiWikiQA. Contrastively, BonLM LoRA narrowly beats the base model on the remaining tasks, all of which are natively Swedish. 5.4 Improved Performance on BonEval Table 3 displays the results on BonEval and the supplementary tasks for BonLM 8B and the baselines. Again, Ministral Base generally outperforms the other baselines, although Llama SW3 and Apertus achieve higher scores on Headline and Quiz. BonLM FFT and LoRA both outperform all baselines on average and on most individual tasks. Unlike in the initial 3B experiments and in prior work (Biderman et al., 2024; Tejaswi et al., 2024), LoRA is generally superior to FFT. The effect of IV mirrors that observed for the 3B models, and LoRA + IV yields the highest average score. At the task level, CPT consistently improves the generative Headline, Lead, and Summary but slightly degrades the discriminative Topic and Entity. We conjecture that classification and NER are comparatively domain-agnostic, as there may be no uniquely journalistic way to perform them. If so, retaining the base modelâs general capabilities likely benefits performance on these tasks. As with the 3B models, Quiz exhibits the largest absolute gains from CPT. This suggests that news articles contain culturally rich information and that LLMs can be imbued with such knowledge through CPT. The supplementary-task results support the validity of BonEval, with CPT improving the base modelâs generative abilities and factual knowledge. A minor discrepancy is Ministral Instructâs strong performance on SEO headline generation. More broadly, Table 3 reveals a consistent trend: instruction tuning increases generation quality. Across all five generative tasks, Instruct outperforms Base, and LoRA + IV surpasses LoRA. The modest results for Reasoning are expected since BonEval does not target reasoning capabilities. Nevertheless, prolonged post-training of Ministral notably reduces scores on Topic, Entity, and Quiz. 5.5 Value of Targeted Evaluation In summary, CPT improves overall performance on BonEval but degrades it on EuroEval. We partly attribute this disparity to BonLM acquiring domain-specific capabilities at the expense of general ones. For example, CPT on predominantly editorial content unsurprisingly decreases the modelsâ common-sense reasoning performance, as measured by HellaSwag. However, machine-translated benchmarks can exhibit cultural biases (Singh et al., 2025), overestimate target-language performance (Kuulmets and Fishel, 2023), and miss nuances captured by native evaluations (Chen et al., 2024). This raises the possibility that the LLM-synthesised portion of EuroEval does not accurately reflect a modelâs proficiency in Swedish. On knowledge tasks specifically, CPT drives substantial gains on the native Quiz and SwedishFacts but causes regressions on the translated MMLU. Importantly, EuroEval does not fully capture the in-domain benefits of CPT evident in BonEval, such as stronger generative capabilities and greater factual knowledge. This underscores the importance of language- and domain-specific benchmarks for guiding model adaptation. 6 Conclusion and Future Work This paper presents the first comprehensive study of adapting LLMs to Swedish journalism. By detailing the curation of BonCorpus, which spans 35 years of editorial content, we highlight the challenges involved in processing ostensibly clean articles. We further introduce BonEval, a domain-specific benchmark comprising six editorial tasks. Our results reaffirm that CPT is effective for domain adaptation, but only when coupled with carefully constructed experience-replay mixtures. Across two model sizes, CPT consistently improves performance on generative and knowledge-intensive tasks but not on discriminative tasks. We also find that IV compatibility depends on the adaptation method: IV generally benefits models trained with LoRA but not those trained with FFT. Crucially, our findings demonstrate the importance of integrating targeted evaluation into the adaptation process, as EuroEvalâs Swedish tasks fail to reflect the broader in-domain trends revealed by BonEval. We conclude that journalism is a promising domain for LLM research and application. In future work, we plan to investigate the downstream impact of CPT by fine-tuning BonLM for editorial use cases such as headline generation and stylistic correction. We also recognise the limitations of automatic evaluation and aggregate metrics. Going forward, we intend to collaborate with journalists to design human-evaluation protocols that better assess the real-world utility of LLMs in newsrooms. Limitations For business and legal reasons, we cannot release proprietary data, code, or models, which reduces the reproducibility of our study. Computational constraints also restrict our adaptation experiments to a single model family, limiting the generalisability of our findings. Because Ministral already has relatively strong Swedish-language capabilities, experiments with another model family could yield different conclusions. Moreover, because the original pre-training data are undisclosed, we cannot assess potential overlap with BonCorpus or BonEval, nor its effects on our results. We also provide limited insight into the downstream effects of CPT since we do not fine-tune BonLM for specific editorial applications. Finally, our exclusive reliance on automatic evaluation precludes any assertive claims about real-world utility. Ethical Considerations Because our research artefacts will not be released, the general risk of misuse by external actors is limited. However, BonLM is intended for internal deployment as part of Bonnier Newsâ editorial tools. It would be subject to strict usage policies but nonetheless affect the work of reporters and editors and, ultimately, the news presented to the public. Although adapting existing LLMs is more efficient than training them from scratch, it still requires substantial computational resources, as detailed in Appendix C. Yet we view a strong base model as an investment that can improve computational efficiency in future work. Finally, copyright is an existential concern for publishers, and we have carefully considered it throughout the data-sourcing process. Acknowledgments This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation (KAW), and by TrustLLM funded by Horizon Europe GA 101135671. Computational resources were provided on the Berzelius system funded by KAW and operated by the National Academic Infrastructure for Supercomputing in Sweden (NAISS). We are grateful to Hans Hjelm at Bonnier News for his support and guidance throughout the project. References AI Sweden (2024) AI Sweden AI-Sweden-Models/Llama-3-8B. Note: Model repositoryaccessed: 2026-06-16 External Links: Link Cited by: §1, §4.3. Berdicevskis et al. (2023) A. Berdicevskis, G. Bouma, R. Kurtz, F. Morger, J. Ăhman, Y. Adesam, L. Borin, D. DannĂ©lls, M. Forsberg, T. Isbister, A. Lindahl, M. Malmsten, F. Rekathati, M. Sahlgren, E. Volodina, L. Börjeson, S. Hengchen, and N. Tahmasebi Superlim: A Swedish Language Understanding Evaluation Benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, p. 8137â8153. External Links: Document Cited by: §2.4. Biderman et al. (2024) D. Biderman, J. Portes, J. J. G. Ortiz, M. Paul, P. Greengard, C. Jennings, D. King, S. Havens, V. Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research. External Links: Link Cited by: §C.1, §2.2, §5.4. Broder (1997) A. Z. Broder On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997, p. 21â29. External Links: Document Cited by: §3.1. Chaudhry et al. (2019) A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. S. Torr, and M. Ranzato On Tiny Episodic Memories in Continual Learning. Computing Research Repository arXiv:1902.10486. Note: version 4 External Links: Document Cited by: §2.1. Chen et al. (2025) J. Chen, Z. Chen, J. Wang, K. Zhou, Y. Zhu, J. Jiang, Y. Min, W. X. Zhao, Z. Dou, J. Mao, Y. Lin, R. Song, J. Xu, X. Chen, R. Yan, Z. Wei, D. Hu, W. Huang, and J. Wen Towards Effective and Efficient Continual Pre-training of Large Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 5779â5795. External Links: Document Cited by: §4. Chen et al. (2024) P. Chen, S. Yu, Z. Guo, and B. Haddow Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, United States, p. 9706â9726. External Links: Document Cited by: §5.5. Chen et al. (2023) Z. Chen, A. H. Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami, A. Sallinen, A. Sakhaeirad, V. Swamy, I. Krawczuk, D. Bayazit, A. Marmet, S. Montariol, M. Hartley, M. Jaggi, and A. Bosselut MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. Computing Research Repository arXiv:2311.16079. Note: version 1 External Links: Document Cited by: §2.1. Cheng et al. (2024a) D. Cheng, Y. Gu, S. Huang, J. Bi, M. Huang, and F. Wei Instruction Pre-Training: Language Models are Supervised Multitask Learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, United States, p. 2529â2550. External Links: Document Cited by: §2.3. Cheng et al. (2024b) D. Cheng, S. Huang, and F. Wei Adapting Large Language Models via Reading Comprehension. In Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.3. Colombo et al. (2024) P. Colombo, T. Pires, M. Boudiaf, R. Melo, D. Culver, E. Malaboeuf, G. Hautreux, J. Charpentier, and M. Desa SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain. In Advances in Neural Information Processing Systems, Vol. 37, p. 129672â129695. External Links: Document Cited by: §1. de la Rosa et al. (2025) J. de la Rosa, V. Mikhailov, L. Zhang, F. Wetjen, D. Samuel, P. Liu, R. Braaten, P. MĂŠhlum, M. B. Birkenes, A. Kutuzov, T. Enstad, H. C. FarsethĂ„s, S. A. Brygfjeld, J. A. Gulla, S. Oepen, E. Velldal, W. Ăstgulen, L. Ăvrelid, and A. S. Myhre The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies, Tallinn, Estonia, p. 544â560. External Links: Link Cited by: §3.3. Ding et al. (2024) H. Ding, Z. Wang, G. Paolini, V. Kumar, A. Deoras, D. Roth, and S. Soatto Fewer Truncations Improve Language Modeling. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235, p. 11030â11048. External Links: Link Cited by: Appendix C. Dorkin et al. (2026) A. Dorkin, T. Purason, E. Kalbaliyev, H. Kuulmets, M. Ojastu, M. FiĆĄel, T. AlumĂ€e, E. Aedmaa, K. Kruusmaa, and K. Sirts EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training. Computing Research Repository arXiv:2603.02041. Note: version 1 External Links: Document Cited by: §2.1. Eide (2024) S. Eide Schibsted Text Tasks: Introducing New Datasets for Norwegian and Swedish Language Models. Note: Blog postaccessed: 2026-06-16 External Links: Link Cited by: §2.4, §4.1. Ekgren et al. (2022) A. Ekgren, A. Cuba Gyllensten, E. Gogoulou, A. Heiman, S. Verlinden, J. Ăhman, F. Carlsson, and M. Sahlgren Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish. In Proceedings of the Thirteenth Conference on Language Resource and Evaluation, p. 3509â3518. External Links: Document Cited by: §1. Ekgren et al. (2024) A. Ekgren, A. Cuba Gyllensten, F. Stollenwerk, J. Ăhman, T. Isbister, E. Gogoulou, F. Carlsson, J. Casademont, and M. Sahlgren GPT-SW3: An Autoregressive Language Model for the Scandinavian Languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, p. 7886â7900. External Links: Document Cited by: §1, §4.3. Elhady et al. (2025) A. Elhady, E. Agirre, and M. Artetxe Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 32174â32186. External Links: Document Cited by: §2.2. Etxaniz et al. (2024) J. Etxaniz, O. Sainz, N. Perez, I. Aldabe, G. Rigau, E. Agirre, A. Ormazabal, M. Artetxe, and A. Soroa Latxa: An Open Language Model and Evaluation Suite for Basque. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 14952â14972. External Links: Document Cited by: §2.1. EuroEval (2026) EuroEval EuroEval Datasets: Swedish. Note: Dataset documentationaccessed: 2026-06-16 External Links: Link Cited by: §4.1. Fujii et al. (2024) K. Fujii, T. Nakamura, M. Loem, H. Iida, M. Ohi, K. Hattori, H. Shota, S. Mizuki, R. Yokota, and N. Okazaki Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities. In First Conference on Language Modeling, External Links: Link Cited by: §3.3. Glocker et al. (2025) K. Glocker, K. Kukk, R. Oji, M. Bollmann, M. Kuhlmann, and J. Kunz Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation. Computing Research Repository arXiv:2512.10772. Note: version 1 External Links: Document Cited by: §1. Gogoulou et al. (2024) E. Gogoulou, T. Lesort, M. Boman, and J. Nivre Continual Learning Under Language Shift. In Text, Speech, and Dialogue: 27th International Conference, p. 71â84. External Links: Document Cited by: §1. Guo et al. (2025a) H. Guo, F. Zeng, F. Zhu, J. Wang, X. Wang, J. Zhou, H. Zhao, W. Liu, S. Ma, X. Zhang, and C. Liu Continual Learning for Generative AI: From LLMs to MLLMs and Beyond. Computing Research Repository arXiv:2506.13045. Note: version 4 External Links: Document Cited by: §2.2. Guo et al. (2025b) Y. Guo, J. Fu, H. Zhang, and D. Zhao Efficient Domain Continual pretraining by Mitigating the Stability Gap. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 32850â32870. External Links: Document Cited by: §2.1, §4.3. Gupta et al. (2023) K. Gupta, B. ThĂ©rien, A. Ibrahim, M. L. Richter, Q. G. Anthony, E. Belilovsky, I. Rish, and T. Lesort Continual Pre-Training of Large Language Models: How to re-warm your model?. In Workshop on Efficient Systems for Foundation Models, External Links: Link Cited by: §2.2. Gururangan et al. (2020) S. Gururangan, A. MarasoviÄ, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith Donât Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, p. 8342â8360. External Links: Document Cited by: §2. Hermann et al. (2015) K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom Teaching Machines to Read and Comprehend. In Advances in Neural Information Processing Systems, Vol. 28, p. 1693â1701. External Links: Link Cited by: §1. HernĂĄndez-Cano et al. (2026) A. HernĂĄndez-Cano, A. HĂ€gele, A. H. Huang, A. Romanou, A. Solergibert, B. PĂĄsztor, B. Messmer, D. Garbaya, E. F. Äurech, I. Hakimi, J. G. Giraldo, M. Ismayilzada, N. Foroutan, S. Moalla, T. Chen, V. SabolÄec, Y. Xu, M. Aerni, B. AlKhamissi, I. A. Marinas, M. H. Amani, M. Ansaripour, I. Badanin, H. Benoit, E. Boros, N. J. Browning, F. Bösch, M. Böther, N. Canova, C. Challier, C. Charmillot, J. Coles, J. M. Deriu, A. Devos, L. Drescher, D. Dzenhaliou, M. Ehrmann, D. Fan, S. Fan, S. Gao, M. Gila, M. Grandury, D. Hashemi, A. M. Hoyle, J. Jiang, M. Klein, A. Kucharavy, A. Kucherenko, F. LĂŒbeck, R. Machacek, T. I. Manitaras, A. Marfurt, K. Matoba, S. Matrenok, H. Mendonça, F. R. Mohamed, S. Montariol, L. Mouchel, S. Najem-Meyer, J. Ni, G. Oliva, M. Pagliardini, E. Palme, A. Panferov, L. Paoletti, M. Passerini, I. Pavlov, A. Poiroux, K. Ponkshe, N. Ranchin, J. Rando, M. Sauser, J. Saydaliev, M. Sayfiddinov, M. Schneider, S. Schuppli, M. Scialanga, A. Semenov, K. Shridhar, R. Singhal, A. Sotnikova, A. Sternfeld, A. K. Tarun, P. Teiletche, J. Vamvas, X. Yao, H. Zhao, A. Ilic, A. Klimovic, A. Krause, C. Gulcehre, D. Rosenthal, E. Ash, F. TramĂšr, J. VandeVondele, L. Veraldi, M. Rajman, T. C. Schulthess, T. Hoefler, A. Bosselut, M. Jaggi, and I. Schlag Apertus: Democratizing Open and Compliant LLMs for Global Language Environments. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, p. 46877â46955. External Links: Document Cited by: §4.3. Hirano and Imajo (2024) M. Hirano and K. Imajo Construction of Domain-Specified Japanese Large Language Model for Finance Through Continual Pre-Training. In 2024 16th IIAI International Congress on Advanced Applied Informatics, p. 273â279. External Links: Document Cited by: §2.4. Hsu et al. (2025) P. Hsu, Y. Dai, V. Kothapalli, Q. Song, S. Tang, S. Zhu, S. Shimizu, S. Sahni, H. Ning, Y. Chen, and Z. Wang Liger-Kernel: Efficient Triton Kernels for LLM Training. In Championing Open-source Development in ML Workshop, External Links: Link Cited by: Appendix C. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: Low-Rank Adaptation of Large Language Models. In Tenth International Conference on Learning Representations, External Links: Link Cited by: §2.2. Huang et al. (2024) S. Huang, P. Li, Y. Hsu, K. Chen, Y. T. Lin, S. Hsiao, R. Tsai, and H. Lee Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 10943â10959. External Links: Document Cited by: §2.3. Ibrahim et al. (2024) A. Ibrahim, B. ThĂ©rien, K. Gupta, M. L. Richter, Q. G. Anthony, E. Belilovsky, T. Lesort, and I. Rish Simple and Scalable Strategies to Continually Pre-train Large Language Models. Transactions on Machine Learning Research. External Links: Link Cited by: §2.1. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing Models with Task Arithmetic. In Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2.3. International Press Telecommunications Council (2026) International Press Telecommunications Council IPTC NewsCodes: Media Topics. Note: Controlled vocabularyaccessed: 2026-06-16 External Links: Link Cited by: item 4. Jindal et al. (2024) I. Jindal, C. Badrinath, P. Bharti, L. Vinay, and S. D. Sharma Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMs. Computing Research Repository arXiv:2410.10739. Note: version 1 External Links: Document Cited by: §2.3. Joulin et al. (2017) A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Valencia, Spain, p. 427â431. External Links: Document Cited by: §3.1. Kalpakchi and Boye (2023) D. Kalpakchi and J. Boye SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish. Computing Research Repository arXiv:2304.13994. Note: version 3 External Links: Document Cited by: §1. Ke et al. (2025) Z. Ke, Y. Ming, X. Nguyen, C. Xiong, and S. Joty Demystifying Domain-adaptive Post-training for Financial LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, p. 31033â31059. External Links: Document Cited by: §2.1. Ke et al. (2023) Z. Ke, Y. Shao, H. Lin, T. Konishi, G. Kim, and B. Liu Continual Pre-training of Language Models. In Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2.2. Ke et al. (2022) Z. Ke, Y. Shao, H. Lin, H. Xu, L. Shu, and B. Liu Adapting a Language Model While Preserving its General Knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, p. 10177â10188. External Links: Document Cited by: §2.2. Kim et al. (2024) S. Kim, D. Kim, C. Park, W. Lee, W. Song, Y. Kim, H. Kim, Y. Kim, H. Lee, J. Kim, C. Ahn, S. Yang, S. Lee, H. Park, G. Gim, M. Cha, H. Lee, and S. Kim SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), Mexico City, Mexico, p. 23â35. External Links: Document Cited by: §2.2. Kundu et al. (2024) A. Kundu, R. D. Lee, L. Wynter, R. K. Ganti, and M. Mishra Enhancing Training Efficiency Using Packing with Flash Attention. Computing Research Repository arXiv:2407.09105. Note: version 6 External Links: Document Cited by: Appendix C. Kunz et al. (2026) J. Kunz, I. N. Debess, and A. Simonsen Family Matters: Language Transfer and Merging for Adapting Small LLMs to Faroese. Computing Research Repository arXiv:2510.00810. Note: version 2 External Links: Document Cited by: §1, §2.2. Kunz (2026a) J. Kunz A Diagnostic Benchmark for Sweden-Related Factual Knowledge. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, p. 5328â5334. External Links: Document Cited by: §4.1. Kunz (2026b) J. Kunz Preferences for Idiomatic Language are Acquired Slowly â and Forgotten Quickly: A Case Study on Swedish. Transactions of the Association for Computational Linguistics 14, p. 1266â1285. External Links: Document Cited by: §1. Kuulmets and Fishel (2023) H. Kuulmets and M. Fishel Translated Benchmarks Can Be Misleading: the Case of Estonian Question Answering. In Proceedings of the 24th Nordic Conference on Computational Linguistics, TĂłrshavn, Faroe Islands, p. 710â716. External Links: Link Cited by: §5.5. Labrak et al. (2024) Y. Labrak, A. Bazoge, E. Morin, P. Gourraud, M. Rouvier, and R. Dufour BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, p. 5848â5864. External Links: Document Cited by: §1. Langlais et al. (2026) P. Langlais, P. Chizhov, C. Arnett, C. R. Hinostroza, M. Nee, E. K. Jones, I. Girard, D. Mach, A. Stasenko, and I. P. Yamshchikov Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training. In Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §3.3. Lee et al. (2026) J. Lee, J. Choi, B. Hwang, J. Choo, B. Kim, J. Yi, J. Lee, D. Jung, J. Park, K. Park, and S. Jung SCALE: Upscaled Continual Learning of Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2026, San Diego, California, United States, p. 41006â41020. External Links: Document Cited by: §2.2, §5.2. Lee et al. (2022) K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, p. 8424â8445. External Links: Document Cited by: §2.1. Lewis et al. (2004) D. D. Lewis, Y. Yang, T. G. Rose, and F. Li RCV1: A New Benchmark Collection for Text Categorization Research. Journal of Machine Learning Research 5, p. 361â397. External Links: Link Cited by: §1. Li et al. (2023) J. Li, Y. Bian, G. Wang, Y. Lei, D. Cheng, Z. Ding, and C. Jiang CFGPT: Chinese Financial Assistant with Large Language Model. Computing Research Repository arXiv:2309.10654. Note: version 2 External Links: Document Cited by: §1. Li et al. (2024) M. Li, M. Chen, B. Tang, S. ShengbinHou, P. Wang, H. Deng, Z. Li, F. Xiong, K. Mao, C. Peng, and Y. Luo NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalism. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 9993â10014. External Links: Document Cited by: §2.4. Lin et al. (2025) P. Lin, R. Balasubramanian, F. Liu, N. Kandpal, and T. Vu Efficient Model Development through Fine-tuning Transfer. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, p. 2617â2636. External Links: Document Cited by: §2.3. Liu et al. (2026) A. H. Liu, K. Khandelwal, S. Subramanian, V. Jouault, A. Rastogi, A. SadĂ©, A. Jeffares, A. Jiang, A. Cahill, A. Gavaudan, A. Sablayrolles, A. HĂ©liou, A. You, A. Ehrenberg, A. Lo, A. Eliseev, A. Calvi, A. Sooriyarachchi, B. Bout, B. RoziĂšre, B. D. Monicault, C. Lanfranchi, C. Barreau, C. Courtot, D. Grattarola, D. Dabert, D. d. l. Casas, E. Chane-Sane, F. Ahmed, G. Berrada, G. Ecrepont, G. Guinet, G. Novikov, G. Kunsch, G. Lample, G. Martin, G. Gupta, J. Ludziejewski, J. Rute, J. Studnia, J. Amar, J. Delas, J. S. Roberts, K. Yadav, K. Chandu, K. Jain, L. Aitchison, L. Fainsin, L. Blier, L. Zhao, L. Martin, L. Saulnier, L. Gao, M. Buyl, M. Jennings, M. Pellat, M. Prins, M. PoirĂ©e, M. Guillaumin, M. Dinot, M. Futeral, M. Darrin, M. Augustin, M. Chiquier, M. Schimpf, N. Grinsztajn, N. Gupta, N. Raghuraman, O. Bousquet, O. Duchenne, P. Wang, P. v. Platen, P. Jacob, P. Wambergue, P. Kurylowicz, P. R. Muddireddy, P. Chagniot, P. Stock, P. Agrawal, Q. Torroba, R. Sauvestre, R. Soletskyi, R. Menneer, S. Vaze, S. Barry, S. Gandhi, S. Waghjale, S. Gandhi, S. Ghosh, S. Mishra, S. Aithal, S. Antoniak, T. L. Scao, T. Cachet, T. S. Sorg, T. Lavril, T. N. Saada, T. Chabal, T. Foubert, T. Robert, T. Wang, T. Lawson, T. Bewley, T. Bewley, T. Edwards, U. Jamil, U. Tomasini, V. Nemychnikova, V. Phung, V. MaladiĂšre, V. Richard, W. Bouaziz, W. Li, W. Marshall, X. Li, X. Yang, Y. E. Ouahidi, Y. Wang, Y. Tang, and Z. Ramzi Ministral 3. Computing Research Repository arXiv:2601.08584. Note: version 1 External Links: Document Cited by: §4. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled Weight Decay Regularization. In Seventh International Conference on Learning Representations, External Links: Link Cited by: Appendix C. Luo et al. (2026) Y. Luo, J. Zhang, S. Fan, K. Yang, M. Hong, Y. Wu, M. Qiao, and Z. Nie BioMedGPT: An Open Multimodal Large Language Model for BioMedicine. IEEE Journal of Biomedical and Health Informatics 30 (2), p. 981â992. External Links: Document Cited by: §2.4. Marcus et al. (1993) M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics 19 (2), p. 313â330. External Links: Link Cited by: §1. Messmer et al. (2025) B. Messmer, V. SabolÄec, and M. Jaggi Enhancing Multilingual LLM Pretraining with Model-Based Data Selection. In Advances in Neural Information Processing Systems, Vol. 38, p. 70037â70073. External Links: Document Cited by: §3.3. Monsen and Jönsson (2021) J. Monsen and A. Jönsson A Method for Building Non-English Corpora for Abstractive Text Summarization. In CLARIN Annual Conference Proceedings, p. 82â85. External Links: Link Cited by: §2.4. Muennighoff et al. (2023) N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. Raffel Scaling Data-Constrained Language Models. In Advances in Neural Information Processing Systems, Vol. 36, p. 50358â50376. External Links: Document Cited by: §4.3. Nag et al. (2025) A. Nag, S. Chakrabarti, A. Mukherjee, and N. Ganguly Efficient Continual Pre-training of LLMs for Low-resource Languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), Albuquerque, New Mexico, United States, p. 304â317. External Links: Document Cited by: §2.1, §2.2. Nayak et al. (2026) N. S. Nayak, K. Killamsetty, L. Han, A. Bhandwaldar, P. Chanda, K. Xu, O. Silkin, M. Eyceoz, H. Wang, A. Pareja, and A. Srivastava Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning. In Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2.2. Nielsen et al. (2025) D. S. Nielsen, K. Enevoldsen, and P. Schneider-Kamp Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies, Tallinn, Estonia, p. 561â572. External Links: Link Cited by: Appendix D, §4.1. Nielsen (2023) D. S. Nielsen ScandEval: A Benchmark for Scandinavian Natural Language Processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics, TĂłrshavn, Faroe Islands, p. 185â201. External Links: Link Cited by: §2.4. Nishal et al. (2024) S. Nishal, C. Li, and N. Diakopoulos Domain-Specific Evaluation Strategies for AI in Journalism. Computing Research Repository arXiv:2403.17911. Note: version 1 External Links: Document Cited by: §2.4. Norlund and Stenbom (2021) T. Norlund and A. Stenbom Building a Swedish Open-Domain Conversational Language Model. In Proceedings of the 23rd Nordic Conference on Computational Linguistics, Reykjavik, Iceland (Online), p. 357â366. External Links: Link Cited by: §1. Parmar et al. (2024) J. Parmar, S. Satheesh, M. Patwary, M. Shoeybi, and B. Catanzaro Reuse, Donât Retrain: A Recipe for Continued Pretraining of Language Models. Computing Research Repository arXiv:2407.07263. Note: version 1 External Links: Document Cited by: §2.1, §2.2. Penedo et al. (2024) G. Penedo, H. KydlĂÄek, L. B. Allal, A. Lozhkov, M. Mitchell, C. Raffel, L. Von Werra, and T. Wolf The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems, Vol. 37, p. 30811â30849. External Links: Document Cited by: §A.3, §2.1. Pezeshkpour and Hruschka (2025) P. Pezeshkpour and E. Hruschka Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMsâ Domain-Specific Insight Learning?. Computing Research Repository. Note: version 1 External Links: Document Cited by: §2.2. PopoviÄ (2017) M. PopoviÄ chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, Copenhagen, Denmark, p. 612â618. External Links: Document Cited by: §4.1. Rajbhandari et al. (2020) S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He ZeRO: Memory optimizations Toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, p. 1â16. External Links: Document Cited by: Appendix C. RodrĂguez et al. (2025) P. RodrĂguez, S. P. SuĂĄrez, P. Gamallo, and S. S. Docio Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A Galician Case Study. In Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria, p. 4622â4637. External Links: Document Cited by: §2.3. Samuel et al. (2025) D. Samuel, V. Mikhailov, E. Velldal, L. Ăvrelid, L. G. G. Charpentier, A. Kutuzov, and S. Oepen Small Languages, Big Models: A Study of Continual Training on Languages of Norway. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies, Tallinn, Estonia, p. 573â608. External Links: Link Cited by: §1, §3.3. Scialom et al. (2022) T. Scialom, T. Chakrabarty, and S. Muresan Fine-tuned Language Models are Continual Learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, p. 6107â6122. External Links: Document Cited by: §2.1. Shah et al. (2024) J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. In Advances in Neural Information Processing Systems, Vol. 37, p. 68658â68685. External Links: Document Cited by: Appendix C. Shi and Lipani (2023) Z. Shi and A. Lipani Donât Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner. In Advances in Neural Information Processing Systems, Vol. 36, p. 5827â5849. External Links: Document Cited by: §2.3. Silo AI (2024) Silo AI Viking 7B/13B/33B: Sailing the Nordic Seas of Multilinguality. Note: Blog postaccessed: 2026-06-16 External Links: Link Cited by: §1. Singh et al. (2025) S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, S. Ruder, W. Ko, A. Bosselut, A. Oh, A. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 18761â18799. External Links: Document Cited by: §5.5. Siriwardhana et al. (2024) S. Siriwardhana, M. McQuade, T. Gauthier, L. Atkins, F. F. Neto, L. Meyers, A. Vij, T. Odenthal, C. Goddard, M. MacCarthy, and J. Solawetz Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation. Computing Research Repository arXiv:2406.14971. Note: version 1 External Links: Document Cited by: §2.3. Tanwar et al. (2025) E. Tanwar, D. Nathani, W. Y. Wang, and T. Chakraborty Understanding the Effects of Domain Finetuning on LLMs. Computing Research Repository arXiv:2510.09359. Note: version 1 External Links: Document Cited by: §2.3. Tejaswi et al. (2024) A. Tejaswi, N. Gupta, and E. Choi Exploring Design Choices for Building Language-Specific LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, United States, p. 10485â10500. External Links: Document Cited by: §2.2, §5.4. Tong et al. (2025) Z. Tong, Y. Gu, H. Liu, Q. Liu, S. Wu, H. Shi, and X. Zhang Generate First, Then Sample: Enhancing Fake News Detection with LLM-Augmented Reinforced Sampling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 24276â24290. External Links: Document Cited by: §1. Ueda et al. (2026) K. Ueda, F. Portet, H. Suwa, and K. Yasumoto Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, p. 10114â10129. External Links: Document Cited by: §2.3. Wan et al. (2026) J. Wan, S. Touileb, L. Steskal, and L. Ăvrelid Personalizing News Headlines with Retrieval-Augmented Generation. In Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual, San Diego, California, United States, p. 55â67. External Links: Document Cited by: §1. Wang et al. (2025) S. Wang, Y. Xie, B. Ding, J. Gao, and Y. Zhang Language Adaptation of Large Language Models: An Empirical Study on LLaMA2. In Proceedings of the 31st International Conference on Computational Linguistics, Abu Dhabi, United Arab Emirates, p. 7195â7208. External Links: Link Cited by: §2.1. Wang et al. (2023) Z. Wang, Z. Lu, B. Jin, and H. Deng MediaGPT : A Large Language Model For Chinese Media. Computing Research Repository arXiv:2307.10930. Note: version 2 External Links: Document Cited by: §1, §2.4. Wu et al. (2024) C. Wu, Y. Gan, Y. Ge, Z. Lu, J. Wang, Y. Feng, Y. Shan, and P. Luo LLaMA Pro: Progressive LLaMA with Block Expansion. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 6518â6537. External Links: Document Cited by: §2.2, §5.2. Wu et al. (2023) S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann BloombergGPT: A Large Language Model for Finance. Computing Research Repository arXiv:2303.17564. Note: version 2 External Links: Document Cited by: §1, §2.4. Wu et al. (2025) T. Wu, T. Vu, L. Luo, and G. Haffari Continual Learning of Large Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, Suzhou, China, p. 16â17. External Links: Document Cited by: §2. Xie et al. (2024) Y. Xie, K. Aggarwal, and A. Ahmad Efficient Continual Pre-training for Building Domain Specific Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, p. 10184â10201. External Links: Document Cited by: §2.1. Xue et al. (2023) F. Xue, Y. Fu, W. Zhou, Z. Zheng, and Y. You To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis. In Advances in Neural Information Processing Systems, Vol. 36, p. 59304â59322. External Links: Document Cited by: §4.3. Yao et al. (2024) S. Yao, Q. Ke, K. Li, Q. Wang, and J. Hu News GPT: A Large Language Model for Reliable and Hallucination-Controlled News Generation. In Proceedings of the 2024 3rd International Symposium on Robotics, Artificial Intelligence and Information Engineering, p. 113â119. External Links: Document Cited by: §1, §2.4. Ying et al. (2025) Y. Ying, G. Zhang, C. Danxin, C. Du, G. Yue, S. Jiang, J. Liang, Y. Fu, H. Hu, and Y. Xiao Data-Efficient Selection via Grammatical Complexity in Continual Pre-training of Domain-Specific LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, p. 22055â22069. External Links: Document Cited by: §2.4. Zhang et al. (2025) M. Zhang, M. MĂŒller-Eberstein, E. Bassignana, and R. v. d. Goot SnakModel: Lessons Learned from Training an Open Danish Large Language Model. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies, Tallinn, Estonia, p. 812â825. External Links: Link Cited by: §1. Appendix A Training Data This section describes selected stages of the pre-processing pipeline in greater detail. A.1 Cleaning We clean the articles by omitting extraneous fields during source-format parsing and by applying regular-expression-based filters. In particular, we remove text spans that are peripheral to the main article content, such as links, advertisements, reader notices, and author bylines. Additionally, we remove artefacts erroneously retained as plain text, such as JavaScript code and HTML tags. By contrast, we preserve elements that provide additional context, including fact boxes, tables, and lists. A.2 Filtering To remove low-quality or otherwise unsuitable articles, we apply a broad set of text- and metadata-based filters to the raw corpus. We design these filters using a combination of domain knowledge and exploratory analysis. When articles of a known category cannot be identified from metadata alone, we instead develop text-based statistics and regular expressions to detect them. By analysing articles flagged as candidates for removal, we iteratively tune the filters to balance false positives and false negatives. Additionally, by examining articles with extreme values for various textual properties, such as unusually high newline counts, we identify and remove previously unknown categories of low-quality content. For example, some general text-quality filters require each article to âą have a headline; âą have body text; âą contain between 2 and 2,000 newline characters, inclusive, counting those following the headline or lead paragraph; âą contain at least 20 words; âą contain at least 17 distinct words; âą contain at least 200 characters; âą have a typeâtoken ratio of at least 0.2; and âą be classified as Swedish by fastText11 1 https://hf.co/facebook/fasttext-language-identification with a score greater than 0.9. Although articles are sourced from Swedish publications, some are written in other languages to serve specific readerships; using the fastText filter, we remove content in languages such as English, Arabic, and Russian. The largest category of excluded material consists of template-based articles that are automatically generated from structured data, typically covering sports results, real estate transactions, and weather forecasts. In addition, we discard large numbers of roundup articles consisting primarily of links to previously published content on a particular topic. Other article types we remove include republished press releases and public records, as well as personal notices such as birthday greetings, wedding announcements, and obituaries. A.3 Deduplication We perform document-level deduplication on normalised text obtained by lowercasing and removing punctuation and diacritics. Following Penedo et al. (2024), we construct MinHash signatures using 112 hash functions and apply LSH for candidate generation, but lower the similarity threshold to 0.25 to increase recall. For each candidate pair, we compute the normalised Levenshtein distance and treat the articles as duplicates if it is less than 0.2. Appendix B Evaluation Data When constructing BonEval, we apply task-specific filters to obtain suitable and representative evaluation examples. Across all tasks sampled from BonCorpus, we exclude articles longer than 1,024 tokens. For each task, we apply the following additional criteria: 1. Headline: Headlines contain 10â120 characters, and input texts contain more than 200 characters. We exclude headlines beginning with âLIST: âŠâ or similar prefixes. 2. Lead: Leads contain 30â500 characters, and input texts contain more than 200 characters. Each lead must also be shorter than half its corresponding body text. We exclude leads containing bullet points of any kind. 3. Summary: Summaries contain 200â800 characters and comprise three to five bullet points. We normalise their formatting. 4. Topic: Articles are originally labelled with one to five topics, exactly one of which is top-level and serves as the target. To simplify the underlying hierarchical multi-label classification problem, which contains thousands of topics, we restrict the label set to the 19 top-level topics shown in Table 8. 5. Entity: Articles have one to five entity labels, each of which is mentioned within the first 90% of the text. We exclude labels likely to reflect annotation errors, such as external news outlets derived from attribution phrases like âaccording to âŠâ. Additionally, for geographically nested location labels, we retain only the more specific one; for example, we keep Stockholm rather than Sweden when both are present. 6. Quiz: Each question has exactly four answer choices. We discard quizzes with an English-language theme or that rely on images or audio. Because the quizzes date back to 2009, we also attempt to exclude strongly time-dependent content, including quizzes in the category âCurrent affairsâ and questions whose answers change over time, such as âHow many prime ministers has the United Kingdom had?â. The quizzes span the following 15 categories: (a) Words and language (b) Current affairs (removed) (c) Around Sweden (d) Around the world (e) Animals and nature (f) Science (g) Food and drink (h) Sport and games (i) Books (j) Film and TV (k) Music (l) Culture (m) Politics and society (n) History (o) Miscellaneous Hyperparameter Value ÎČ1 _1 0.9 ÎČ2 _2 0.95 ϔΔ 1e-8 Weight decay 0.1 Schedule Cosine Linear warmup 10% Max. learning rate 5e-5 Min. learning rate 5e-6 Dropout 0.0 Gradient clipping 1.0 Sequence length 16,384 Global batch size 64 Precision bfloat16 Table 4: Training configuration for all experiments. Appendix C Training Setup Across all experiments, we use NVIDIA DGX H200 nodes and consume a total of 6,500 GPU-hours. We use the AdamW optimiser (Loshchilov and Hutter, 2019) and the hyperparameters reported in Table 4. To improve computational and memory efficiency, we use DeepSpeed ZeRO-2 (Rajbhandari et al., 2020), FlashAttention-3 (Shah et al., 2024), and Liger kernels (Hsu et al., 2025). Depending on the model size and number of GPUs, we use activation checkpointing and gradient accumulation as needed to achieve the target global batch size. Because article lengths vary substantially, we pack examples up to the maximum sequence length so that each optimisation step is based on roughly the same number of tokens. To minimise truncation, which can degrade model performance, we use a best-fit-decreasing packing strategy (Ding et al., 2024). As the packed sequences still vary slightly in length, we flatten each batch into a single sequence to eliminate padding inefficiencies (Kundu et al., 2024). C.1 PEFT Configurations We configure LoRA and LLaMA Pro to have comparable numbers of trainable parameters for the 3B model: 395M and 466M, respectively. For LLaMA Pro, we insert one additional layer after every sixth layer of the base model. Following Biderman et al. (2024), we apply LoRA to all transformer modules with a rank of 256, an α of 512, and no dropout. Appendix D Evaluation Setup We implement BonEval and run the evaluations with the EuroEval library. For tasks outside BonEval, we rely on EuroEvalâs existing implementations. To parse model outputs, we use log probabilities for Topic and Quiz and structured generation for Entity. We evaluate each model over 10 runs, randomly sampling few-shot examples from the training split for each run. During development, we evaluate on validation splits rather than test splits. Following Nielsen et al. (2025), we heuristically determine the number of few-shot examples for each task based on context length. Rather than targeting 1,000 tokens, we select the number of examples such that no input exceeds 4,096 tokens. We cap this number at 12, guided by the existing task implementations.22 2 https://github.com/EuroEval/EuroEval/blob/e630b62/src/euroeval/tasks.py The resulting number of few-shot examples for each task is reported in Table 5. If IV is applied to a model, we consider it instruction-tuned. Task prompts for base and instruction-tuned models are shown in Tables 6 and 7, respectively; English translations are provided in Tables 8 and 9, respectively. For Quiz, we use EuroEvalâs default multiple-choice prompts. Task Train Val Test Few-Shot Headline 1,024 256 8,192 3 Lead 1,024 256 8,192 3 Summary 32 32 100 3 Topic 1,024 256 8,192 3 Entity 1,024 256 8,192 3 Quiz 1,024 256 11,960 12 Table 5: Number of examples for BonEval tasks. Task Prompt Prefix Prompt Template Headline Följande Ă€r artiklar med tillhörande rubriker. Artikel: lead + body Rubrik: headline Lead Följande Ă€r artiklar med tillhörande ingresser. Artikel: body Ingress: lead Summary Följande Ă€r artiklar med tillhörande sammanfattningar i punktform. Artikel: article Sammanfattning: summary Topic Följande Ă€r artiklar med tillhörande frĂ„gor och svar om deras Ă€mnen. Artikel: article FrĂ„ga: Vilket Ă€mne beskriver bĂ€st artikelns innehĂ„l? Svarsalternativ: a. Arbete b. Brott, lag och rĂ€t c. Ekonomi, nĂ€ringsliv och finans d. HĂ€lsa och sjukvĂ„rd e. Klimat och miljö f. Konflikter och krig g. Kultur och nöje h. Kungligt i. Livsstil och fritid j. Olyckor och katastrofer k. Personligt l. Politik m. Religion och tro n. SamhĂ€lle o. Skola och utbildning p. Sport q. Trafik och fordon r. Vetenskap och teknik s. VĂ€der Svar: a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s Entity Följande Ă€r artiklar med tillhörande namngivna entiteter i JSON-format som Ă€r centrala för innehĂ„llet. Artikel: article Namngivna entiteter: "person": [...], "plats": [...], "organisation": [...] Quiz Följande Ă€r flervalsfrĂ„gor (med svar). FrĂ„ga: question Svarsalternativ: a. choice_a b. choice_b c. choice_c d. choice_d Svar: a|b|c|d Table 6: BonEval prompts for base models. Task Instruction Prompt Headline Artikel: lead + body Skriv en rubrik till artikeln ovan. Lead Artikel: body Skriv en ingress till artikeln ovan. Summary Artikel: article Skriv en sammanfattning i punktform av artikeln ovan. Topic Artikel: article FrĂ„ga: Vilket Ă€mne beskriver bĂ€st artikelns innehĂ„l? Svarsalternativ: a. Arbete b. Brott, lag och rĂ€t c. Ekonomi, nĂ€ringsliv och finans d. HĂ€lsa och sjukvĂ„rd e. Klimat och miljö f. Konflikter och krig g. Kultur och nöje h. Kungligt i. Livsstil och fritid j. Olyckor och katastrofer k. Personligt l. Politik m. Religion och tro n. SamhĂ€lle o. Skola och utbildning p. Sport q. Trafik och fordon r. Vetenskap och teknik s. VĂ€der Besvara frĂ„gan ovan med a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r eller s, och inget annat. Entity Artikel: article Identifiera de namngivna entiteter som Ă€r centrala för artikelns innehĂ„l. Svara i JSON-format med nycklarna person, plats och organisation, dĂ€r vĂ€rdena Ă€r listor över de namngivna entiterna av den typen, precis som de förekommer i artikeln. Quiz FrĂ„ga: question Besvara frĂ„gan ovan med a, b, c eller d, och inget annat. Table 7: BonEval prompts for instruction-tuned models. Task Prompt Prefix Prompt Template Headline The following are articles with associated headlines. Article: lead + body Headline: headline Lead The following are articles with associated leads. Article: body Lead: lead Summary The following are articles with associated bullet-point summaries. Article: article Summary: summary Topic The following are articles with associated questions and answers about their topics. Article: article Question: Which topic best describes the content of the article? Choices: a. Labour b. Crime, law and justice c. Economy, business and finance d. Health e. Environment f. Conflict, war and peace g. Arts, culture, entertainment and media h. Royalty i. Lifestyle and leisure j. Disaster, accident and emergency incident k. Human interest l. Politics and government m. Religion n. Society o. Education p. Sport q. Traffic and vehicles r. Science and technology s. Weather Answer: a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s Entity The following are articles with associated named entities in JSON format that are central to the content. Article: article Named entities: "person": [...], "location": [...], "organisation": [...] Quiz The following are multiple choice questions (with answers). Question: question Choices: a. choice_a b. choice_b c. choice_c d. choice_d Answer: a|b|c|d Table 8: Translated BonEval prompts for base models. Task Instruction Prompt Headline Article: lead + body Write a headline to the above article. Lead Article: body Write a lead to the above article. Summary Article: article Write a bullet-point summary of the above article. Topic Article: article Question: Which topic best describes the content of the article? Choices: a. Labour b. Crime, law and justice c. Economy, business and finance d. Health e. Environment f. Conflict, war and peace g. Arts, culture, entertainment and media h. Royalty i. Lifestyle and leisure j. Disaster, accident and emergency incident k. Human interest l. Politics and government m. Religion n. Society o. Education p. Sport q. Traffic and vehicles r. Science and technology s. Weather Answer the above question by replying with a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, or s, and nothing else. Entity Article: article Identify the named entities central to the content of the article. Respond in JSON format with the keys person, location, and organisation, where the values are lists of the named entities of that type, exactly as they appear in the article. Quiz Question: question Answer the above question by replying with a, b, c or d, and nothing else. Table 9: Translated BonEval prompts for instruction-tuned models. Appendix E Complete Evaluation Results Complete evaluation results for the 3B experiments are presented in Tables 10 and 11, while those for BonLM 8B and the baselines are presented in Tables 12 and 13. Model SweReC SUC 3.0 ScaLA MultiWikiQA SweDN MMLU HellaSwag Average Ministral 3 3B Base 77.61 55.79 38.86 75.41 34.70 49.04 26.02 51.06 Instruct 75.23 39.66 33.50 70.94 32.66 42.32 48.18 48.93 Reasoning 74.17 44.48 29.58 78.11 22.62 26.10 35.57 44.37 Data Mixtures B 78.22 31.41 28.22 65.83 30.77 21.24 5.80 37.35 BC 75.09 41.52 28.30 71.09 32.34 24.70 10.03 40.44 BE 75.68 39.46 22.84 72.28 33.12 24.07 8.21 39.38 BCE 78.62 47.04 25.50 72.41 33.58 26.78 7.20 41.59 BS 77.30 41.92 25.04 69.16 35.11 26.31 13.80 41.24 BCS 79.60 47.24 35.09 68.33 34.94 27.31 11.74 43.46 BES 79.25 45.77 33.25 69.28 34.99 28.61 10.26 43.06 BCES 78.85 47.60 32.31 70.43 33.46 28.90 11.49 43.29 PEFT Methods LoRA 77.39 55.46 33.50 75.18 34.37 44.46 20.30 48.67 LLaMA Pro 78.04 54.17 34.09 74.73 34.48 47.22 20.92 49.09 Instruct Vectors B + IV 75.23 27.13 7.33 54.50 27.85 7.58 4.21 29.12 BC + IV 77.10 36.87 9.77 72.08 32.24 10.97 6.24 35.04 BE + IV 74.11 33.89 10.23 72.28 30.79 9.69 4.28 33.61 BCE + IV 77.63 50.36 12.82 69.90 33.24 16.57 9.44 38.57 BS + IV 77.83 44.04 15.27 71.25 33.74 13.42 9.40 37.85 BCS + IV 74.58 42.88 14.63 72.20 35.87 13.26 6.58 37.14 BES + IV 69.12 40.72 26.07 69.68 32.18 17.40 7.82 37.57 BCES + IV 75.71 44.60 23.91 72.87 34.03 17.49 9.38 39.71 LoRA + IV 78.31 46.00 32.76 71.57 34.79 37.71 34.86 48.00 Table 10: Results from 3B experiments on Swedish EuroEval tasks. Micro-averaged F1 is reported for SUC 3.0, token-averaged F1 for MultiWikiQA, and chrF3++ for SweDN. For all other tasks, MCC is reported. Underline denotes section-best and bold overall best. Model BonEval Supplementary Headline Lead Summary Topic Entity Quiz Average SvD AB Facts Ministral 3 3B Base 18.62 26.24 42.28 70.39 29.80 51.07 39.73 19.94 40.72 15.81 Instruct 26.38 31.28 45.87 71.80 23.15 47.80 41.05 27.96 44.15 16.64 Reasoning 21.19 28.80 46.08 68.45 23.49 33.66 36.94 14.70 39.59 10.15 Data Mixtures B 20.61 25.37 39.42 52.49 23.68 60.30 36.98 17.56 34.15 31.25 BC 20.48 22.63 40.83 67.61 28.40 59.97 39.99 20.02 37.92 30.61 BE 20.54 25.03 41.04 67.59 26.38 59.46 40.01 18.97 38.26 30.10 BS 19.81 27.28 42.87 62.87 26.11 61.41 40.06 19.98 38.20 31.65 BCE 20.48 25.86 40.77 65.33 29.05 58.58 40.01 18.92 40.19 32.02 BCS 20.98 26.74 43.42 68.35 27.65 60.89 41.34 19.76 40.21 33.74 BES 20.26 27.63 40.80 60.92 23.73 60.80 39.02 17.73 40.30 30.78 BCES 20.73 26.83 42.24 70.44 29.28 59.40 41.48 20.46 39.48 32.93 PEFT Methods LoRA 19.95 27.24 43.98 69.32 30.47 56.97 41.32 20.95 41.12 24.74 LLaMA Pro 20.24 26.47 43.05 71.85 30.30 55.09 41.17 19.97 40.10 23.57 Instruct Vectors B + IV 23.15 28.41 39.57 1.48 23.66 43.48 26.62 17.89 29.10 14.56 BC + IV 25.61 29.06 41.34 47.30 25.66 48.36 36.22 25.74 38.25 18.26 BE + IV 25.24 32.62 40.86 42.60 21.61 44.31 34.54 24.85 39.14 18.27 BCE + IV 24.99 32.76 43.20 46.74 25.42 49.19 37.05 19.87 40.91 25.88 BS + IV 25.38 30.79 41.77 7.64 23.93 54.35 30.64 20.88 35.57 25.97 BCS + IV 24.00 31.79 44.56 16.38 23.34 49.76 31.64 23.92 41.66 22.28 BES + IV 23.48 31.34 41.06 31.18 21.38 51.23 33.28 20.35 38.32 24.38 BCES + IV 25.09 31.86 44.72 55.34 26.06 51.36 39.07 20.08 40.75 27.66 LoRA + IV 26.06 31.55 48.20 69.33 25.10 53.36 42.27 30.14 45.60 22.45 Table 11: Results from 3B experiments on BonEval and supplementary tasks Svenska Dagbladet (SvD) SEO Headline, Aftonbladet (AB) Summary, and SwedishFacts. Micro-averaged F1 is reported for Entity, and MCC for Topic, Quiz, and SwedishFacts. For all other tasks, chrF3++ is reported. Underline denotes section-best and bold overall best. Model SweReC SUC 3.0 ScaLA GPT-SW3 6.7B 10.44 ± 5.52 / 13.17 ± 4.04 26.72 ± 2.58 / 20.29 ± 2.65 9.84 ± 1.72 / 50.83 ± 3.01 Llama SW3 8B 79.90 ± 0.86 / 75.36 ± 1.95 36.02 ± 2.16 / 26.67 ± 2.22 8.81 ± 2.26 / 53.34 ± 0.97 Apertus 8B 79.11 ± 0.42 / 75.56 ± 1.40 45.74 ± 2.02 / 31.99 ± 2.86 34.34 ± 4.28 / 63.41 ± 4.61 Ministral 3 8B Base 80.19 ± 0.89 / 77.61 ± 1.68 66.44 ± 2.73 / 56.13 ± 5.14 51.70 ± 3.18 / 72.75 ± 2.57 Instruct 77.53 ± 1.40 / 77.72 ± 1.60 60.51 ± 1.80 / 37.49 ± 3.12 50.62 ± 1.62 / 73.58 ± 1.53 Reasoning 77.05 ± 1.23 / 78.58 ± 0.96 65.21 ± 2.35 / 46.84 ± 5.92 47.77 ± 2.11 / 73.38 ± 1.13 BonLM 8B FFT 80.01 ± 1.19 / 78.68 ± 0.83 50.64 ± 2.54 / 39.69 ± 4.46 39.61 ± 3.35 / 68.07 ± 2.49 FFT + IV 79.07 ± 1.51 / 79.96 ± 0.90 45.84 ± 3.33 / 30.85 ± 3.45 22.54 ± 2.08 / 52.49 ± 3.10 LoRA 80.55 ± 0.71 / 79.68 ± 1.10 66.86 ± 2.40 / 57.70 ± 4.66 59.85 ± 2.35 / 79.35 ± 1.45 LoRA + IV 78.27 ± 0.99 / 79.74 ± 0.72 62.27 ± 1.73 / 37.58 ± 3.26 54.24 ± 2.12 / 76.68 ± 1.16 Model MMLU HellaSwag GPT-SW3 6.7B 2.92 ± 0.78 / 24.38 ± 0.74 2.05 ± 1.26 / 25.91 ± 0.80 Llama SW3 8B 17.02 ± 1.24 / 37.27 ± 0.87 5.64 ± 1.44 / 29.15 ± 1.02 Apertus 8B 41.54 ± 1.04 / 55.81 ± 0.85 32.54 ± 1.80 / 48.61 ± 1.52 Ministral 3 8B Base 58.10 ± 0.90 / 68.44 ± 0.67 43.68 ± 3.19 / 56.16 ± 2.74 Instruct 56.22 ± 1.32 / 66.93 ± 1.01 57.27 ± 1.22 / 67.88 ± 0.92 Reasoning 49.96 ± 1.61 / 62.39 ± 1.20 64.27 ± 1.57 / 73.00 ± 1.27 BonLM 8B FFT 30.62 ± 0.94 / 47.93 ± 0.74 17.66 ± 2.17 / 37.31 ± 1.66 FFT + IV 20.09 ± 2.11 / 39.49 ± 2.08 12.48 ± 3.46 / 32.92 ± 3.22 LoRA 53.34 ± 0.91 / 64.80 ± 0.63 36.82 ± 2.96 / 50.75 ± 2.62 LoRA + IV 47.18 ± 1.50 / 59.86 ± 1.16 52.06 ± 1.60 / 63.80 ± 1.20 Model MultiWikiQA SweDN GPT-SW3 6.7B 69.47 ± 1.95 / 54.28 ± 2.52 30.62 ± 1.09 / 32.89 ± 1.26 Llama SW3 8B 72.22 ± 1.74 / 56.16 ± 2.15 32.23 ± 0.86 / 33.14 ± 1.03 Apertus 8B 74.05 ± 1.99 / 55.93 ± 2.66 33.39 ± 0.85 / 35.03 ± 0.98 Ministral 3 8B Base 75.71 ± 2.89 / 59.71 ± 3.45 35.30 ± 1.13 / 37.30 ± 1.45 Instruct 74.28 ± 1.75 / 56.32 ± 2.07 35.31 ± 0.20 / 38.54 ± 0.20 Reasoning 79.44 ± 1.93 / 64.01 ± 2.54 31.16 ± 1.28 / 37.39 ± 1.12 BonLM 8B FFT 70.53 ± 1.40 / 52.92 ± 1.31 31.94 ± 1.85 / 32.89 ± 2.11 FFT + IV 70.04 ± 1.89 / 56.36 ± 2.21 36.03 ± 0.83 / 38.20 ± 1.05 LoRA 69.25 ± 2.27 / 51.67 ± 2.68 35.03 ± 1.14 / 36.84 ± 1.31 LoRA + IV 70.05 ± 2.09 / 50.23 ± 2.61 36.27 ± 0.33 / 39.57 ± 0.36 Table 12: Results for BonLM 8B and baselines on Swedish EuroEval tasks, showing primary and secondary metrics with 95% confidence intervals. MCC / macro-averaged F1 are reported for SweRec and ScaLA, micro-averaged F1 without / with MISC for SUC 3.0, MCC / accuracy for MMLU and HellaSwag, token-averaged F1 / exact match for MultiWikiQA, and chrF3++ / chrF4++ for SweDN. Model Headline Lead Summary GPT-SW3 6.7B 17.70 ± 0.98 / 17.85 ± 1.06 22.78 ± 1.79 / 23.36 ± 2.02 36.11 ± 1.93 / 37.17 ± 2.13 Llama SW3 8B 19.35 ± 0.97 / 19.41 ± 1.01 29.13 ± 1.33 / 29.86 ± 1.39 40.51 ± 2.08 / 40.90 ± 2.30 Apertus 8B 18.61 ± 0.88 / 18.66 ± 0.92 23.09 ± 2.42 / 23.22 ± 2.59 40.93 ± 2.23 / 41.60 ± 2.45 Ministral 3 8B Base 18.35 ± 0.74 / 18.40 ± 0.77 25.79 ± 2.22 / 26.07 ± 2.41 42.49 ± 3.06 / 43.01 ± 3.44 Instruct 29.28 ± 0.21 / 31.02 ± 0.29 32.03 ± 0.80 / 33.19 ± 1.04 46.27 ± 0.45 / 48.74 ± 0.43 Reasoning 21.36 ± 1.50 / 21.58 ± 1.58 27.68 ± 1.71 / 27.92 ± 1.88 47.93 ± 1.39 / 49.31 ± 1.68 BonLM 8B FFT 21.58 ± 1.25 / 21.69 ± 1.30 29.88 ± 2.55 / 30.44 ± 2.75 43.76 ± 3.09 / 44.20 ± 3.36 FFT + IV 24.32 ± 1.68 / 24.56 ± 1.77 31.24 ± 1.49 / 31.71 ± 1.69 43.26 ± 2.30 / 43.53 ± 2.49 LoRA 21.29 ± 0.91 / 21.38 ± 0.94 31.07 ± 1.90 / 32.01 ± 2.14 44.35 ± 2.94 / 45.03 ± 3.30 LoRA + IV 29.76 ± 0.56 / 30.73 ± 0.66 35.01 ± 0.38 / 36.79 ± 0.62 49.88 ± 0.84 / 51.32 ± 1.05 Model Topic Entity Quiz GPT-SW3 6.7B 0.49 ± 0.20 / 0.28 ± 0.06 27.50 ± 1.45 / 27.50 ± 1.45 6.00 ± 1.11 / 28.72 ± 1.03 Llama SW3 8B 68.28 ± 0.76 / 47.73 ± 0.78 26.17 ± 0.64 / 26.17 ± 0.64 61.30 ± 0.69 / 70.83 ± 0.54 Apertus 8B 70.95 ± 0.61 / 51.75 ± 0.84 22.81 ± 2.58 / 22.81 ± 2.58 61.05 ± 0.62 / 70.74 ± 0.49 Ministral 3 8B Base 72.84 ± 0.76 / 59.84 ± 1.36 29.97 ± 0.96 / 29.97 ± 0.96 60.12 ± 0.56 / 69.96 ± 0.43 Instruct 71.93 ± 1.02 / 59.39 ± 1.25 23.20 ± 0.28 / 23.20 ± 0.28 57.76 ± 0.55 / 68.08 ± 0.43 Reasoning 71.06 ± 1.17 / 55.46 ± 1.20 20.67 ± 1.24 / 20.67 ± 1.24 52.91 ± 0.43 / 64.50 ± 0.33 BonLM 8B FFT 72.40 ± 0.58 / 53.92 ± 1.28 28.34 ± 1.07 / 28.34 ± 1.07 68.86 ± 0.33 / 76.59 ± 0.25 FFT + IV 42.28 ± 9.30 / 29.82 ± 6.42 28.38 ± 0.82 / 28.38 ± 0.82 61.98 ± 0.62 / 71.34 ± 0.51 LoRA 68.93 ± 0.77 / 57.49 ± 0.95 29.81 ± 0.57 / 29.81 ± 0.57 71.36 ± 0.35 / 78.49 ± 0.27 LoRA + IV 70.54 ± 1.53 / 58.21 ± 1.48 23.99 ± 0.27 / 23.99 ± 0.27 66.95 ± 0.61 / 75.09 ± 0.47 Table 13: Results for BonLM 8B and baselines on BonEval, showing primary and secondary metrics with 95% confidence intervals. Micro-averaged F1 without / with MISC are reported for Entity, MCC / macro-averaged F1 for Topic, and MCC / accuracy for Quiz. For all other tasks, chrF3++ / chrF4++ is reported.