Paper deep dive
sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak
Marek Šuppa, Ivan Vykopal, Andrej Ridzik, Kristián Sopkovič, Natália Kňažeková, Jaroslav Kopčan, Miroslav Blšták, Viktória Ondrejová, Daniel Hládek, Michal Gregor, Martin Tamajka, Marián Šimko
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/8/2026, 7:01:07 AM
Summary
The paper introduces sk-bench, a native-first benchmark for evaluating Large Language Models in Slovak, addressing the lack of comprehensive evaluation for this under-resourced language. It comprises 30 datasets across 10 skill categories, including 11 newly introduced or packaged resources like IFEval-SK and Chiby/SKJ1. The study evaluates 55 open- and closed-weight models, finding that proprietary APIs outperform the best open models by 12.6 points. Key findings include that translation preserves model rankings but blurs performance distinctions, continued pretraining in Slovak can degrade performance (as seen with Qwen3-14B) unless repaired by instruction tuning, and test-time reasoning significantly boosts scores for larger models.
Entities (15)
Relation Signals (12)
sk-bench → evaluates → Slovak
confidence 100% · We present sk-bench, a native-first Slovak benchmark
sk-bench → includes → IFEval-SK
confidence 95% · Eleven resources are introduced... including IFEval-SK
sk-bench → includes → Chiby
confidence 95% · native Chiby/SKJ1 resources
sk-bench → includes → SKJ1
confidence 95% · native Chiby/SKJ1 resources
Qwen3-14B → performancedegradedby → continued_slovak_pretraining
confidence 95% · For Qwen3-14B, continued Slovak pretraining lowers the overall score by 13.9 points.
IFEval-SK → adaptedfrom → M-IFEval
confidence 90% · we translated the M-IFEval... templates to Slovak
sk-bench → builton → LightEval
confidence 90% · sk-bench builds on LightEval (Habib et al., 2023)
test_time_reasoning → improves → Qwen3-14B
confidence 90% · Test-time reasoning improves scores by 8.5 to 12.5 points for models of 9B and above.
instruction_tuning → repairs → Qwen3-14B
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation. We present sk-bench, a native-first Slovak benchmark with 30 datasets (33 scored task variants) across ten skill categories. Eleven resources are introduced or first packaged for generative-LLM evaluation, including IFEval-SK with Slovak-adapted instruction checkers and native Chiby/SKJ1 resources for Slovak grammar and morphology. We evaluate 55 open- and closed-weights models under one harness. The best open model trails proprietary APIs by 12.6 points. Model rankings are similar for native and translated closed-form data ($\rho\geq0.98$), though translation separates the strongest models less well. By contrast, human-authored and LLM-generated QA questions rank models differently ($\rho=0.72$). For Qwen3-14B, continued Slovak pretraining lowers the overall score by 13.9 points. A small instruction set restores three quarters of that loss. Test-time reasoning improves scores by 8.5 to 12.5 points for models of 9B and above. Together, these findings suggest four design lessons for other under-resourced languages: use native data where translation fails, plan instruction repair after language adaptation, enable test-time reasoning before scaling up, and avoid overinvesting in target-language prompts. We release the data and code at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2610.09152v1
- Canonical: https://arxiv.org/abs/2610.09152v1
Trouble viewing inline? Open PDF directly →
Full Text
196,794 characters extracted from source content.
Expand or collapse full text
sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak Marek Šuppa †thanks: Correspondence: marek@suppa.sk Affiliation: Comenius University in Bratislava, Slovakia Ivan Vykopal Affiliation: Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia github.com/slovak-nlp/sk-bench huggingface.co/collections/slovak-nlp/sk-bench Andrej Ridzik Affiliation: Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia github.com/slovak-nlp/sk-bench huggingface.co/collections/slovak-nlp/sk-bench Kristián Sopkovič Affiliation: Technical University of Košice, Slovakia Natália Kňažeková Affiliation: Comenius University in Bratislava, Slovakia Jaroslav Kopčan Affiliation: Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia github.com/slovak-nlp/sk-bench huggingface.co/collections/slovak-nlp/sk-bench Miroslav Blšták Affiliation: Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia github.com/slovak-nlp/sk-bench huggingface.co/collections/slovak-nlp/sk-bench Viktória Ondrejová Affiliation: Cisco Systems Daniel Hládek Affiliation: Technical University of Košice, Slovakia Michal Gregor Affiliation: Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia github.com/slovak-nlp/sk-bench huggingface.co/collections/slovak-nlp/sk-bench Martin Tamajka Affiliation: Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia github.com/slovak-nlp/sk-bench huggingface.co/collections/slovak-nlp/sk-bench Marián Šimko Affiliation: Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia github.com/slovak-nlp/sk-bench huggingface.co/collections/slovak-nlp/sk-bench Affiliation: Brno University of Technology, Czechia Abstract Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation. We present sk-bench, a native-first Slovak benchmark with 30 datasets (33 scored task variants) across ten skill categories. Eleven resources are introduced or first packaged for generative-LLM evaluation, including IFEval-SK with Slovak-adapted instruction checkers and native Chiby/SKJ1 resources for Slovak grammar and morphology. We evaluate 55 open- and closed-weights models under one harness. The best open model trails proprietary APIs by 12.6 points. Model rankings are similar for native and translated closed-form data (ρ≥0.98ρ≥ 0.98), though translation separates the strongest models less well. By contrast, human-authored and LLM-generated QA questions rank models differently (ρ=0.72ρ=0.72). For Qwen3-14B, continued Slovak pretraining lowers the overall score by 13.9 points. A small instruction set restores three quarters of that loss. Test-time reasoning improves scores by 8.5 to 12.5 points for models of 9B and above. Together, these findings suggest four design lessons for other under-resourced languages: use native data where translation fails, plan instruction repair after language adaptation, enable test-time reasoning before scaling up, and avoid overinvesting in target-language prompts. We release the data and code at github.com/slovak-nlp/sk-bench. 1 Introduction Figure 1: Overview of the sk-bench design: ten skill categories, 30 evaluation tasks. Chips with a gold fill, heavy gold border and a leading star (⋆ ) are the 11 resources introduced or first packaged for generative-LLM evaluation by sk-bench, listed first within each panel. Plain chips with a hairline border are the 19 pre-existing datasets we integrate into the harness. Novelty (the chip itself) and provenance (the N / MT / MT∗ / G badge at the right: native Slovak, translated from English, translated then adapted with a custom Slovak registry, LLM-generated over a Slovak source) are independent of each other—MultiWikiQA-SK is generated but pre-existing, Autoškola is native and new. Tasks with bilingual templates are run with both English and Slovak prompts. IFEval-SK and Slohry are Slovak-only. These ten skill categories organise the datasets. The results tables instead use six reporting groups defined by answer format (Table 1, §6). Slovak, a fusional, morphologically rich West Slavic language spoken by roughly five million people, is not short of NLP resources: the contemporary Slovak National Corpus passes one billion tokens SNK (2025), a Slovak-centric encoder (Pikuliak et al., 2022) has existed since 2021, and the skLEP benchmark (Suppa et al., 2025) covers nine encoder-style NLU tasks (POS, two NER datasets, NLI, RTE, QA, sentiment, hate-speech, and STS). Yet Slovak remains under-served in two respects: (a) generative-LLM evaluation under a unified harness: an LLM practitioner picking between, say, a 70 B-class open model and a closed API for a Slovak deployment cannot get zero-shot, identically prompted scores for both across the skills LLMs are used for; and (b) Slovak coverage in the multilingual benchmarks that shape model development: XNLI (Conneau et al., 2018) and XCOPA (Ponti et al., 2020) omit Slovak, and HELM (Liang et al., 2023) and BIG-bench (Srivastava et al., 2023) provide minimal Slovak-specific signal. Multilingual suites that do include Slovak rely mostly on machine translation (Lai et al., 2023; Singh et al., 2025), which inherits source-language bias and culturally Western-centric content (Singh et al., 2025) and leaves exploitable translation artefacts in the data (Artetxe et al., 2020). We address (a) and (b) by combining native Slovak data with selected translated proxies under a single generative-LLM harness. Research questions. We use sk-bench to address three empirical questions about Slovak LLMs (§6). Their answers inform the cross-language lessons in §7 and Appendix A. RQ1 (data provenance). For the same skill, how much do native, machine-translated, and LLM-generated evaluation data disagree about which LLM is best in Slovak? We separate translation from question generation (§6.1). RQ2 (language adaptation). What does adapting an instruction-tuned model to Slovak cost, and can the loss be repaired? We compare a Slovak continued-pretraining checkpoint with its base and add a Slovak instruction-tuning step across three bases (§6.2). RQ3 (test-time reasoning). Does test-time reasoning transfer to Slovak, and which skills does it move? We evaluate nine models in both standard and reasoning modes (§6.3). We make four contributions: (1) sk-bench, a native-first benchmark for generative-LLM evaluation in Slovak (§3.2): 30 tasks across ten skill categories, run under one harness against 55 contemporary open- and closed-weights models, with a unified Slovak leaderboard (§6). (2) Eleven new Slovak evaluation resources: These did not exist as LLM evaluations before sk-bench: five we built from scratch (authored, annotated, or constructed by us), four we curated from public institutional sources whose items existed but where no dataset did, and two are existing expert resources first packaged for generative-LLM evaluation. All eleven are marked with a star (★ ) in Figure 1, which separates them from the nineteen datasets we integrate, and itemised by tier in Table 10. Two could not be built by translation: IFEval-SK, whose Slovak instruction-checker registry we re-implemented from scratch (Appendix L), and the paired Chiby/SKJ1 grammaticality probe (Dobeš and Bednár, 2026). The native exam sets (driving law, admission tests, school subjects) are topically selective, but their professionally validated gold labels make them the native counterpart to translated MMLU in RQ1 (§6.1). (3) A reproducible harness and open release. The harness (§4) scores open-weights and closed (API) models through one generate-and-match pipeline. We release the prompts, run scripts, and per-task breakdowns for reproducible follow-up work (§5.4). (4) Four transferable design lessons (L1–L4) for under-resourced-language benchmarks, each backed by a Slovak result: translation preserves closed-form rankings but blurs the frontier and diverges on QA, adaptation costs Qwen3-14B 13.9 points that instruction tuning largely repairs, and reasoning adds 8.5–12.5 points at ≥ 9B (§7, Appendix A). 2 Related Work Slovak NLP benchmarks. The skLEP benchmark (Suppa et al., 2025) bundled nine NLU tasks (POS, two NER datasets, NLI, RTE, QA, sentiment, hate-speech, STS) into a single GLUE-style suite and is the closest precedent to sk-bench. We reuse several skLEP subtasks under the LLM harness and extend its encoder-style scope to multi-choice reasoning, summarisation, instruction-following, and Slovak professional and school exams. SK-QuAD (Hládek et al., 2023) provides ∼ 91 k manually annotated extractive QA pairs. WikiGoldSK (Suba et al., 2023) contributes a gold-standard NER set. Slovak summarisation has been studied on SME-Sum (Suppa and Adamec, 2020) and SlovakSum (Ondrejová and Suppa, 2024). Sentiment classification has a longer history (Pecar et al., 2019), and the SentiSK and ToxicSK datasets (Sokolová et al., 2025) underpin the corresponding tasks in our suite. The recent Trojsten benchmark (Zahradník and Suppa, 2025) evaluates LLMs on Slovak high-school STEM competition problems but is single-domain. SkMTEB (Šuppa et al., 2026), a Slovak text-embedding benchmark rather than a generative evaluation, introduced the natively annotated NLI-SK-annotated dataset that we reuse in our suite. Table 6 (App. A) summarises how sk-bench differs from skLEP and the Slavic, Central-European, and multilingual benchmarks discussed below. Slavic and Central European benchmarks. The closest analogues are BenCzechMark (Fajcik et al., 2025) for Czech; KLEJ (Rybak et al., 2020) and LEPISZCZE (Augustyniak et al., 2022) for Polish; RussianSuperGLUE (Shavrina et al., 2020) and the LLM-oriented MERA (Fenogenova et al., 2024) for Russian; HuLU (Ligeti-Nagy et al., 2024) for Hungarian; and SI-NLI (Klemen et al., 2024) together with the SloBench platform (CJVT and University of Ljubljana, 2024) for Slovene. sk-bench borrows the multi-skill, multi-metric design of BenCzechMark and the GLUE-lineage benchmark design of KLEJ/HuLU but specialises the tasks, prompts, and metrics for Slovak. Single-language native benchmarks. sk-bench belongs to a family of single-language, natively-sourced LLM benchmarks that pair a curated task suite with a unified harness: C-Eval for Chinese (Huang et al., 2023), IndoMMLU for Indonesian (Koto et al., 2023), ArabicMMLU for Arabic (Koto et al., 2024), and BenCzechMark (Fajcik et al., 2025) for the fellow West Slavic Czech. What sk-bench adds is the paired provenance contrast (§3.1): measuring a skill from natively authored, machine-translated, and LLM-generated data lets us test whether a translated proxy ranks models reliably for that skill (§6.1). No native-first generative-LLM suite of this breadth previously existed for Slovak. Multilingual benchmarks containing Slovak. Slovak is one of the 122 variants in Belebele (Bandarkar et al., 2024), one of the 26 languages in Okapi’s translated MMLU/ARC/HellaSwag (Lai et al., 2023), and one of 300+ in MultiWikiQA (Smart, 2025). INCLUDE (Romanou et al., 2025) is a non-translated multilingual benchmark; sk-bench shares its preference for native data. Global MMLU (Singh et al., 2025) documents cultural bias in purely translated MMLU. MEGAVERSE (Ahuja et al., 2024) (22 datasets, 83 languages) is a precedent for our open/closed side-by-side comparison. MultiBLiMP (Jumelet et al., 2026) covers Slovak grammaticality natively rather than by translation, but is restricted to subject–verb agreement minimal pairs generated automatically from Universal Dependencies and UniMorph. Chiby and SKJ1 instead pair naturalistic Slovak error data with expert-authored items across nine error types. Closest to our RQ1 in design is MultiLoKo (Hupkes and Bogoychev, 2025), which contrasts locally-sourced with translated data across 31 languages. sk-bench makes that contrast per skill within one language. Evaluation methodology. sk-bench builds on LightEval (Habib et al., 2023) and the evaluation-harness lineage of HELM (Liang et al., 2023), the EleutherAI lm-evaluation-harness (Gao et al., 2023), the Open LLM Leaderboard (Fourrier et al., 2024), GLUE/SuperGLUE (Wang et al., 2019b; Wang et al., 2019a), BIG-bench (Srivastava et al., 2023), and XTREME (Hu et al., 2020). For instruction-following, we adapt IFEval (Zhou et al., 2023) to Slovak via the multilingual M-IFEval (Dussolle et al., 2025). 3 The sk-bench Task Suite Figure 1 summarises the suite. The full per-dataset catalogue (Table 10) appears in Appendix C. The suite combines three kinds of data: native Slovak data that tests language-specific competence, Slovak domain data whose labels come from professional or educational settings, and translated or generated proxies that cover skills for which no native Slovak benchmark exists. We keep these provenance types explicit because RQ1 (§6.1) tests whether a model’s Slovak ranking is stable across them. 3.1 Design Principles sk-bench is organised around a paired provenance contrast per skill: wherever possible the same skill is measured from two differently-sourced datasets, so that provenance can be varied while the skill is held fixed and a model’s Slovak ranking can be compared across the two (Table 11, Appendix D). The new native datasets fill the native side where none existed, which enables the analysis in §6.1. §Limitations lists what the suite does not cover (dialogue, agentic tool use, code, multimodality, and dialects). 3.2 Inclusion Criteria We selected datasets according to four criteria, in priority order: (i) native Slovak data preferred over translation, (i) publicly available redistribution-friendly licence, (i) usable test split of at least 100 examples, (iv) automatic, reproducible scoring. We call the result native-first: we prioritise native Slovak evaluation data and admit translated datasets only when no native equivalent exists for the skill of interest (e.g. MMLU-style multi-choice reasoning). Datasets failing (i)–(iv) are noted but excluded from the main results. 3.3 The Skill Categories The full per-dataset catalogue (Table 10, Appendix C) lists every task with its size, provenance, skill, and licence. Per-task descriptions and metrics are in Appendix D. We group the ten skill categories by provenance. Native Slovak data. Most of the suite is native. Classification uses SentiSK and MovieReview (sentiment), ToxicSK and skLEP-hate (toxicity and hate speech), and the native NLI-SK-annotated set. Question answering uses the human-authored skLEP-QA task from skLEP (Suppa et al., 2025), built on SK-QuAD (Hládek et al., 2023). Professional and school knowledge comes from five native exam sets (Financial Exams, Autoškola, e-testy-sk, Slovak-tests, and Scio admission tests) plus the Slovak Trivia set. Grammar and morphology, which translation cannot produce, are tested by Chiby and SKJ1 (Dobeš and Bednár, 2026) (grammaticality), Slohry (inflectional morphology), and Word Analogies derived from the Slovak Conceptual Dictionary (Blšták, 2025). Summarisation uses SlovakSum and SME-Sum, each down-sampled to 1,500 (Appendix J) and scored with ROUGE, METEOR, and chrF/chrF++ against single human references. IFEval-SK measures verifiable instruction following: we translated the M-IFEval (Dussolle et al., 2025) templates to Slovak and re-implemented the programmatic checkers against Slovak word boundaries, casing, and punctuation. OneRuler-SK probes long-context retrieval over Slovak text. Translated and generated proxies. Where no native resource of comparable breadth exists, we admit translated proxies: Okapi-MMLU-SK and Global-MMLU-Lite (subject knowledge), and ARC-SK, SlovakCOPA, Belebele-SK, and TruthfulQA-SK (reasoning and commonsense). For QA we add MultiWikiQA-SK (Smart, 2025), an existing published dataset whose questions were generated by Gemini 1.5 Pro from the November 2023 Slovak Wikipedia and then rephrased by the same model in a second pass to blunt surface word-matching. It is released under C BY-NC-SA 4.0. Because its generator belongs to a proprietary model family that we also evaluate, we check for generator self-preference in Appendix H.3 and find none. These form the translated and generated sides of the RQ1 contrasts (§6.1). 4 Evaluation Harness sk-bench wraps LightEval’s Pipeline (Habib et al., 2023) so the same prompt, few-shot policy, and scoring logic apply to every model: open weights are served with vLLM (Kwon et al., 2023) and closed models queried through LiteLLM (BerriAI, 2024). Every task is scored by text generation followed by string matching against the gold answer (or the allowed labels): we generate up to a per-task token budget (32 tokens for classification and multi-choice, 512 for QA and summarisation, 4096 for IFEval), normalise, and compute the metric. Open and closed models are therefore scored identically. The only remaining difference is decoding compliance (some closed APIs ignore seed or apply post-training that temperature=0 does not override). We hand-write Slovak instruction templates per task category and parallel English templates as a robustness check. Interpreting free-form output. Because every task is scored from generated text, variation in output format and length is a source of measurement error. A fixed-order parser strips chat-template artefacts before matching, per-task token budgets are identical across providers, and refusals and unresolvable outputs are scored incorrect rather than dropped. This rarely matters for standard models (a multiple-choice answer is a median of two tokens), only for small models in reasoning mode, which exhaust the budget inside the trace before emitting a parseable answer (§6.3). Appendices E and M give the parsing rules, the language-mismatch policy, per-task output-length statistics, and the templates. Metrics and aggregation. We use accuracy for classification and multi-choice tasks; multilingual quasi-exact-match (QEM) and quasi-F1 (QF1) (Habib et al., 2023) for QA; ROUGE (Lin, 2004), chrF/chrF++ (Popović, 2015; Popović, 2017), and METEOR (Banerjee and Lavie, 2005) for summarisation; and the prompt- and instruction-level strict/loose accuracies of Zhou et al. (2023) for IFEval-SK. The main sk-bench score is the unweighted mean of the reporting-group averages, in the spirit of GLUE/SuperGLUE (Wang et al., 2019b; Wang et al., 2019a). Within-group aggregation is by task macro-average, not by example count, to prevent the 13,062-item Okapi-MMLU-SK from dominating its group against the 500-item SlovakCOPA. Model Knowledge & reasoning Reading & QA Classification Generation Instruction following Grammar & morphology Overall <<2B Granite 4.0-H 1B 39.8 ± 0.6 28.2 ± 0.6 50.1 ± 7.8 14.8 ± 0.9 14.9 6.0 ± 0.8 25.6 ± 1.8 Qwen3 1.7B 35.0 ± 5.5 30.9 ± 2.8 52.4 ± 7.8 12.8 ± 0.9 23.5 12.3 ± 1.9 27.8 ± 2.4 2–9B Ministral 3 8B 60.4 ± 1.3 39.2 ± 6.2 70.6 ± 4.4 13.5 ± 0.8 24.0 37.1 ± 1.0 40.8 ± 2.3 Ministral 3 8B (R) 54.9 ± 1.9 37.8 ± 4.0 61.9 ± 7.2 13.1 ± 1.1 39.9 32.1 ± 2.8 39.9 ± 2.4 9–32B Qwen3.5 27B 81.2 ± 0.2 55.7 ± 6.3 79.5 ± 3.4 14.9 ± 0.4 36.1 64.6 ± 0.8 55.3 ± 1.8 Gemma 4 31B 78.8 ± 0.7 53.1 ± 8.7 82.4 ± 0.7 14.4 ± 0.5 30.2 66.7 ± 2.1 54.3 ± 1.3 32–200B Llama 3.3 70B 72.1 ± 0.5 53.9 ± 4.0 79.4 ± 0.2 21.4 ± 0.5 35.4 56.4 ± 1.0 53.1 ± 1.0 gpt-oss 120B 73.7 ± 1.0 53.2 ± 6.3 79.0 ± 0.5 13.5 ± 0.4 74.9 65.3 ± 1.0 60.0 ± 1.1 >>200B Qwen3.5 397B-A17B 83.4 ± 0.5 55.5 ± 8.2 80.6 ± 0.8 14.6 ± 0.2 37.8 72.3 ± 3.0 57.4 ± 1.7 GLM-5.1 80.6 ± 0.4 61.2 ± 6.9 82.6 ± 1.0 14.2 ± 0.2 35.9 70.8 ± 1.3 57.6 ± 1.2 Proprietary (size undisclosed) Gemini 3.1 Pro 92.4 ± 0.2 71.3 ± 4.5 82.6 ± 0.4 14.8 ± 0.0 87.2 87.6 ± 0.4 72.6 ± 0.9 GPT-5.5 89.2 ± 0.3 67.3 ± 6.8 81.6 ± 0.3 15.5 ± 0.3 86.4 86.1 ± 0.3 71.0 ± 1.2 Claude Opus 4.6 87.2 ± 0.5 58.6 ± 10.2 83.4 ± 0.4 15.4 ± 0.2 80.6 83.6 ± 0.5 68.1 ± 2.0 Table 1: No open-weights model reaches the proprietary APIs on Slovak. Main sk-bench leaderboard (Slovak, 0-shot): top two open-weights models per size tier and top three proprietary APIs. Cells are the unweighted mean of the group’s per-task primary metrics (×100× 100). Best and second-best are over all 55 models, so a mark appears only where a shown model holds the record. Overall averages the six groups. Grey ± is the standard deviation across the three prompt templates. Full leaderboard and per-task scores: Tables 14 and 15. 5 Experimental Setup 5.1 Models Evaluated We evaluate 55 models: 49 open-weight (46 base checkpoints plus three we instruction-tune on Slovak data, below) and 6 proprietary, grouped into four parameter tiers (≤ 2 B, 2–10 B, 10–70 B, ≥ 70 B) spanning 350 M to 1.1 T parameters. Per-tier membership is in the leaderboard (Table 1). The full catalogue with sizes, licences, and revisions is in Appendix F. The open weights cover the IBM Granite, Qwen3/3.5/3.6, Gemma, Llama, and Mistral/Ministral families; the European-centric TildeOpen (Bergmanis et al., 2026), EuroLLM (Martins et al., 2024), Teuken (Ali et al., 2024), Bielik, and Apertus models; the Aya, GPT-OSS, Command-A, GLM, MiniMax, and Kimi checkpoints; and two Slovak-adapted bases (below). The proprietary models are two each from the GPT-5 (OpenAI, 2026) (GPT-5.4 Mini, GPT-5.5), Gemini 3.1 (Google DeepMind, 2026) (Pro, Flash Lite), and Claude 4.6 (Anthropic, 2026) (Opus, Sonnet). Slovak instruction tuning. We fine-tuned the base models slovak-nlp/mistral-sk-7b, slovak-nlp/Qwen3-14B-sk, and TildeAI/TildeOpen-30b-64k (Mistral-SK, Qwen3-SK, and TildeOpen-SK below) on the Slovak split of the TaCo Alpaca data (Upadhayay and Behzadan, 2023) (machine-translated Stanford Alpaca (Taori et al., 2023)) to create instruction-tuned counterparts. 5.2 Inference, Decoding, and Prompt Language Open-weights models run on H200 141 GB and A100 40 GB nodes (served as described in §4). The default protocol is zero-shot with greedy decoding (temperature=0, fixed seed=1234 where supported). Per-task token budgets are kept identical across providers. The one exception is reasoning mode, where the budget is 8192 tokens. The released results use the Slovak instruction templates. The harness also ships parallel English templates for each task. 5.3 Few-Shot Policy The main results are zero-shot. We additionally ran every task with a few-shot counterpart at 3-shot (random exemplars from each task’s training or validation split) under both Slovak and English prompts. The effect is analysed in §6.4 (Table 8). 5.4 Reproducibility The harness, prompts, model configuration, decoding settings, and per-example raw model outputs are released with the benchmark. Software dependencies are pinned in the released harness (pyproject.toml), and each Hugging Face dataset is loaded at a fixed revision. Closed-weights API calls record the model snapshot and the call date. Generation is greedy (temperature=0), so the main results carry no generation stochasticity for open-weights models. Crediting the component datasets. sk-bench integrates existing Slovak resources alongside the datasets we introduce, and aggregation of this kind can redirect citations from the original authors to the wrapper. To preserve credit for the original authors, we follow the practice established by Universal Dependencies, where the collection as a whole is citable but every component keeps its own authorship and its own citation. The release ships a per-component BibTeX entry for each integrated dataset, so a user of any subset can cite its original authors. Every dataset card names the original authors first, ahead of any sk-bench attribution. The release README asks users to cite the datasets they actually use, not only sk-bench. In the paper itself, every integrated dataset is cited in its task description (§D), with its licence and provenance recorded in the dataset catalogue (Table 10). Closed-API non-determinism. We query closed-API models at temperature=0. We did not run repeated calls per (model, task). The three Slovak and three English prompt variants instead give a per-model spread of scores that bounds prompt-induced variation. 6 Results and Analysis Proprietary APIs lead the Slovak leaderboard and the best open-weights model trails them by 12.6 points. Among open models, scale is a weak predictor of Slovak ability. Table 1 is the main sk-bench leaderboard: the top two models per size tier (and the top three proprietary APIs) scored on six reporting groups (classification, knowledge & reasoning, reading & QA, generation, grammar & morphology, and instruction following), plus an overall macro-average over the six. The full 55-model leaderboard and per-task scores are in Appendix H (Tables 14 and 15), with a thinking-mode breakdown alongside. Reporting groups are defined by answer format and cut across the ten skill categories of Figure 1: grammar & morphology comprises the two production tasks (Word Analogies, Slohry), while the Chiby and SKJ1 grammaticality judgements are selection tasks and are scored under reading & QA and knowledge & reasoning respectively. Figure 2: Open-weights Slovak scores plateau past ∼ 20B: overall sk-bench score vs. parameter count (log scale). The frontier (blue) flattens past gpt-oss 20B/Qwen3.5 27B—no larger open model, up to the 1.1T Kimi K2.6, beats gpt-oss 120B (60.0). The dashed line marks the best closed model (Gemini 3.1 Pro, 72.6). A fully-labelled version is in Appendix H (Figure 7). Gemini 3.1 Pro leads at 72.6 overall, ahead of GPT-5.5 (71.0) and Claude Opus 4.6 (68.1). The strongest open-weights model is gpt-oss 120B at 60.0. Mid-sized open models are nonetheless competitive: Qwen3.5 27B (55.3), Gemma 4 31B (54.3), and gpt-oss 20B (54.1) come within a few points of the largest (>>200B) open models (45.0–57.6), and the Pareto frontier flattens past gpt-oss 20B with no larger model beating gpt-oss 120B (Figure 2). The open–closed gap is also uneven across skills: it is widest on instruction following (best closed 87.2 vs. best open 74.9) and smallest, even reversed, on summarisation, but the reversal comes from the metric. Under single-reference ROUGE-L, Llama 3.3 70B (21.4) tops the group ahead of GPT-5.5 (15.5) and Gemini 3.1 Pro (14.8), yet Llama falls to 13th under chrF (28.6, behind both at 29.0), and under METEOR the best summariser is Claude Opus 4.6. The generation column therefore ranks models under one metric and says little about summarisation quality (§Limitations). 6.1 RQ1: Where Does Data Provenance Reorder Rankings? Within parameter band Contrast Overall <<2B 2–9B 9–32B 32–200B >>200B Prop. top-10 NLI (N vs MT) 0.99 0.93 0.99 0.98 0.96† 0.90† 0.94† 10/10 MCQ (N vs MT) 0.98 0.93 0.96 0.98 0.93† 0.40† 0.71† 7/10 QA (N vs G) 0.72 0.17 0.43 0.74 0.64† 0.80† 0.94† 6/10 Table 2: Translation is a safe ranking proxy for closed-form Slovak skills, but question provenance is not. Spearman ρ between the native and translated/generated sk-bench leaderboards—overall, within parameter-size bands, and as top-10 rank overlap—over all 55 models (zero-shot, Slovak prompts, accuracy for NLI/MCQ, QEM for QA). Overall 95% bootstrap CIs: NLI [0.97,1.00][0.97,1.00], MCQ [0.94,0.99][0.94,0.99], QA [0.57,0.83][0.57,0.83]. † band has n≤7n≤ 7, so its within-band ρ is high-variance (wide CIs) and read only qualitatively. N=native, MT=translated, G=LLM-generated. Translation preserves the coarse ordering of models on closed-form tasks, but how the QA questions are produced reorders models substantially, so native data pays off on some skills more than others, and a paired contrast shows which. We test this with three skill-level dataset contrasts that compare provenance while holding the broad skill fixed (Table 2): along the translation contrast, native NLI-SK-annotated vs. translated skLEP-nli, and native e-testy-sk vs. translated Okapi-MMLU-SK; and along the question-generation contrast, human-authored skLEP-QA vs. LLM-generated MultiWikiQA-SK. For each we report the Spearman rank correlation between the two leaderboards, overall and within parameter-size bands, plus top-10 rank overlap. The per-model native-vs-translated scatter is in Appendix A (Figure 4). A single correlation over a pool spanning 350 M to 1.1 T parameters is dominated by the between-tier ordering, so the within-band values and the top-10 overlap are more informative. Translation is a safe ranking proxy for closed-form skills. For NLI the native and translated leaderboards are near-identical (overall ρ=0.99ρ=0.99, 95% bootstrap CI [0.97,1.00][0.97,1.00], identical top-10) and stay highly correlated in every band we can estimate. For multiple choice, translation is likewise safe overall (ρ=0.98ρ=0.98 [0.94,0.99][0.94,0.99]), but the two top-10s share only 7 of 10 models, suggesting translated MMLU separates the strongest models less reliably. Our frontier bands (n≤6n≤ 6) are too small to estimate this divergence reliably. Limits of the dataset-level contrast. The two translation-contrast rows pair differently-sourced datasets covering the same skill rather than the same items in two languages, so they mix content differences with provenance. We report them at the dataset level because that is the choice a benchmark builder faces: build a native set or adopt a translated suite such as Okapi-MMLU (if a native set exists, one uses it). An item-matched control would isolate translation per se, and is constructible from data we release, since Belebele-SK and Global-MMLU-Lite are parallel across Slovak and English. We leave it to future work. Question provenance reorders QA rankings. Human-authored skLEP-QA and LLM-generated MultiWikiQA-SK agree only at ρ=0.72ρ=0.72 [0.57,0.83][0.57,0.83] (top-10 overlap 6/10), and the disagreement is largest below the frontier (ρ=0.17ρ=0.17 and 0.430.43 at <<2B and 2–9B, rising to 0.740.74 at 9–32B). For Slovak QA, how the questions are produced moves model rankings substantially more than translation of multiple-choice items does. Two mechanisms plausibly separate QA from the closed-form skills. First, the formats differ: NLI is format-robust closed classification over a fixed label set, whereas open QA is free-text generation scored by quasi-exact-match and token F1, and is far more exposed to format and surface variation. Second, MultiWikiQA-SK’s questions were deliberately rephrased to reduce surface word overlap, which fits the divergence being largest below the frontier, where models lean most on word-overlap shortcuts. The per-task scores for both sides of every contrast are in Appendix H (Table 15). 6.2 RQ2: What Does Slovak Adaptation Cost, and Can It Be Repaired? By “cost” we mean capability lost relative to the base model, not training compute (App. B). The cost of adaptation. Adaptation is expensive and the loss is largely repairable: continued Slovak pretraining trades broad capability for a narrow grammar gain, and a small Slovak instruction set then recovers about three quarters of it. We evaluate one adapted/base pair whose members are both in the sweep, slovak-nlp/Qwen3-14B-sk (Qwen3-SK) against its Qwen/Qwen3-14B base. A single pair cannot support a population-level claim. We read it as a case study isolating the adaptation step from base architecture (though not from other post-training differences). Per-group and per-task scores for both models are in Appendix H. For this pair, Slovak adaptation lowers the overall score from 47.4 to 33.5: it helps only the grammar group (+6.8+6.8) while degrading knowledge (−23.7-23.7), reading (−25.2-25.2), and classification (−30.6-30.6). Instruction following falls by only −6.4-6.4, far less than knowledge, reading, or classification, so adaptation mainly costs content and comprehension, not instruction following. Figure 5 (App. A) plots the full base-adapted-repaired arc per group. Repairing it with instruction tuning. Three of our open models are Slovak base LMs that we further tuned on a small Slovak Alpaca-style instruction set, giving a base/tuned pair at 7B, 14B, and 30B (§5, Table 3, App. A). The effect of this single tuning step is strongly base-dependent and rises monotonically with the capability of the base: the weakest base (Mistral-SK, 19.0 overall) loses 4.0 points, the mid base (TildeOpen-SK, 31.9) is essentially flat (+0.8+0.8), and the strongest (Qwen3-SK, 33.5) gains +10.3+10.3. The only consistent effect is on instruction following, which the data targets: instruction following improves for two bases (+13.2+13.2, +11.5+11.5) and is unchanged for the third, and free-form generation moves little. Every other group swings by up to 30 points in both directions depending on the base (Figure 6, App. A): Qwen3-SK trades 19.8 grammar points for large gains in knowledge (+26.6+26.6), reading, and classification, whereas TildeOpen-SK does the reverse (+30.7+30.7 grammar, −29.2-29.2 classification). A small Alpaca set thus reliably teaches the instruction format but does not add knowledge or reasoning, so whether it nets a gain depends on how much capability the base already has to expose. For the Qwen lineage, Alpaca instruction-tuning recovers most of the adaptation loss, back to 43.8 (about three-quarters of the 13.9-point deficit, though still 3.6 below the original Qwen3-14B). Instruction tuning therefore mostly re-exposes capability the base already has instead of adding new Slovak capability, which is also why it helps least on the bases with the least capability to recover. 6.3 RQ3: Does Test-Time Reasoning Transfer to Slovak? Test-time reasoning gives large, consistent gains in Slovak for models of 9B or more, mostly in instruction following, and it backfires below that size, so its benefit depends on capacity. Nine models expose an optional reasoning (“thinking”) mode and were evaluated in both settings under an 8,192-token generation cap: eight Qwen variants from 0.6B to the 35B-A3B mixture-of-experts (MoE), plus Gemma 4 31B (Table 4, App. A, with full per-task scores in Appendix H, Table 16). Every model of 9B or more gains +8.5+8.5 to +12.5+12.5 overall, and the gain is highly concentrated: instruction following (IFEval-SK) improves for every model in the set—by +7+7 to +54+54 points, and by at least +21+21 above 9B, the largest and most consistent effect—while grammar & morphology and knowledge rise at scale and free-form generation (summarisation ROUGE-L) is flat or slightly negative everywhere (≤+0.2≤+0.2). The gains therefore carry over to a morphologically rich Slavic language. Reasoning can also reduce scores. The two smallest Qwen3.5 models (0.8B, 2B) regress sharply (−16.8-16.8 and −9.7-9.7 overall), while the older Qwen3 0.6B/1.7B at the same sizes improve, so capacity, not size alone, decides the outcome. The regression is a format failure: these models exhaust the budget inside the trace without emitting a parseable answer, so reasoning helps only once a model can close its trace within budget (App. A). InstructionGrammarGeneration0010102020303040405050group score Δ from reasoningQwen3.5 9BQwen3.5 27BQwen3.6 27BGemma 4 31BQwen3.6 35B-A3B Figure 3: Test-time reasoning transfers to Slovak, and concentrates on instruction following. Per-group sk-bench change (Slovak, 0-shot) from thinking mode for the five ≥ 9B models, each gaining +8.5+8.5 to +12.5+12.5 overall. The effect is flat on free-form generation. 6.4 Robustness: Prompt Language, Few-Shot, Context Length Three perturbations leave both the overall score and the model ordering largely intact. The per-group tables and their discussion are in Appendix A. Prompt language (L4): re-running the sweep with English rather than Slovak prompts, on every task that has an English template, moves the overall score by just +0.4+0.4 (Table 5). Grammar & morphology is the only sizable shift (−2.6-2.6, where Slovak helps). Few-shot exemplars: 3-shot raises the overall score only modestly across the 52 models with few-shot results (Slovak +2.0+2.0, English +1.6+1.6, Table 8), and unevenly—it helps classification and reading & QA, where exemplars pin down the answer format, but slightly lowers knowledge and reasoning, echoing the instruction-tuning result of §6.2: demonstrations teach the answer format but add no knowledge. Long context: on OneRuler-SK’s matched Slovak and English haystacks, Slovak retrieval declines only mildly from 4k to 16k (58.1→54.458.1→ 54.4) and tracks English throughout, so there is no Slovak-specific length cliff up to 16k. Scores vary across models far more than across lengths (Table 7). 7 Conclusion sk-bench is a native-first benchmark for Slovak. It covers ten skills with 30 datasets, eleven of which are introduced or first packaged here. Several of them, such as IFEval-SK, Chiby, and SKJ1, could not have been built by translation. We evaluate 55 open- and closed-weights models under one harness and release all per-task scores (§6, Appendix H). The best open model still trails proprietary APIs by 12.6 points. Building sk-bench taught us four design lessons for other under-resourced languages (Appendix A). L1: Spend native data where translation fails. Translation preserves the overall ranking on closed-form skills (ρ≥0.98ρ≥ 0.98) but separates the strongest models less well (top-10 overlap 7 of 10 for multiple choice). Native data matters most for grammar and morphology, which translation cannot supply, and for QA, where LLM-generated questions rank models differently from human-authored ones (ρ=0.72ρ=0.72). L2: Plan an instruction-repair step after language adaptation. Continued Slovak pretraining raised Qwen3-14B’s grammar score by 6.8 points but cut its overall score from 47.4 to 33.5, a familiar case of catastrophic forgetting (French, 1999; Luo et al., 2023). A small instruction set recovers three quarters of the loss, but only for bases with capability left to recover. L3: Enable test-time reasoning before scaling up. It adds 8.5 to 12.5 points for models of 9B and above, mostly in instruction following, but backfires on the smallest models (Qwen3.5 0.8B and 2B lose 16.8 and 9.7 points), which exhaust their budget before answering. L4: Do not over-invest in target-language prompts. They barely move scores in our case: switching every template to English changes the overall score by only +0.4+0.4 points, and grammar, the one group that shifts noticeably, by −2.6-2.6. We therefore recommend that benchmarks for under-resourced languages include at least one native paired contrast per skill and re-implement instruction checkers natively instead of translating them. Limitations Coverage. sk-bench focuses on textual single-turn evaluation in standard written Slovak. It does not currently cover dialogue, multi-turn agentic tool use, multimodality, code generation, or dialectal/colloquial variants of Slovak. Translated tasks. Several tasks (Okapi-MMLU-SK, ARC-SK, TruthfulQA-SK, Belebele-SK, SlovakCOPA, IFEval-SK) are translations from English source data. They inherit cultural and topical bias from English-speaking contexts (Singh et al., 2025). We retain them because no native Slovak counterpart of equivalent breadth exists, but caution against treating them as measurements of Slovak knowledge per se. Single-reference summarisation. SlovakSum and SME-Sum each provide one human reference per article. ROUGE and chrF under-penalise paraphrastic-but-correct summaries and over-penalise stylistic divergence. We treat the resulting numbers as relative rankings of summarisation quality rather than absolute measures. Test-set contamination. We have not measured or certified the absence of any sk-bench test item from the training corpora of the models we evaluate. A new dataset does not imply unseen content, so we distinguish three layers at which leakage can occur. Benchmark-format leakage is exposure to the exact prompt and gold answer in sk-bench form. This format is new for all eleven resources introduced or first packaged here. Answer-key leakage is exposure to the question–answer pairs of the underlying item banks. These have been public for years for the exam sets (Autoškola, Scio, e-testy-sk, Slovak-tests). Source-document leakage is exposure to the raw text behind the task, which is public in most cases: Chiby is built over Slovak Wikipedia revisions and Word Analogies over the Slovak Conceptual Dictionary. The tasks least exposed on all three layers are therefore those whose gold signal we authored ourselves: the Chiby annotations and the IFEval-SK checkers. Assembling a fresh benchmark reduces exposure of the exact test split but not exposure to the underlying public material, so scores on the previously released datasets (NLI-SK-annotated, skLEP, SK-QuAD, SlovakSum, SME-Sum, ARC, MMLU, TruthfulQA), particularly for closed models whose training data is undisclosed, should not be read as contamination-free measurements of Slovak ability. Appendix G gives the per-dataset breakdown. Aggregation. The sk-bench score weights all reporting groups equally and uses task macro-averaging within each group. This prevents large translated tasks (Okapi-MMLU-SK) from dominating their group, but it remains contestable: an example-weighted score would yield different rankings. We release per-task and per-group breakdowns (Appendix H) so readers can re-aggregate under their own weighting. Reproducibility caveats. Closed-weights APIs can change without notice, so we cannot guarantee bit-identical reproducibility of those numbers later. Open-weights results are reproducible given the released harness, the released checkpoints, and the same vLLM version. Ethical Considerations The datasets aggregated by sk-bench were originally released under their respective licences (catalogued in Appendix C). We re-distribute only what those licences allow. Tasks whose licence forbids redistribution are run from the original source. We use every artefact solely for non-commercial research evaluation, consistent with the intended use of the benchmark and evaluation resources it derives from. The sk-bench artefacts we release (the new datasets, the harness, and the per-task model outputs) are likewise intended for research evaluation. Components derived from non-commercial (C BY-NC∗) or research-only sources inherit those restrictions and must not be used commercially. Hate-speech and toxicity tasks contain material that may be offensive. We mark these tasks as such in the released metadata and recommend appropriate IRB-style review for any follow-up annotation work. The Slovak Wikipedia, news, and exam content underlying several tasks is publicly available. We add no personally-identifying information beyond what is already public. The user-generated sentiment and toxicity tasks (SentiSK, ToxicSK) consist of public social-media comments used as released by their original authors. We perform no re-identification and add no annotations linking text to individuals. Use of AI assistants. AI assistants were used to help generate the result tables and figures, to script parts of the evaluation harness and analysis, and, as the authors are not native English speakers, to improve the grammar and language of the manuscript. The benchmark design, experiments, analyses, and conclusions are the authors’ own and were verified by the authors. LLM use within the benchmark itself (LLM-generated MultiWikiQA-SK and the LLM-based annotation of Autoškola) is marked by the dataset provenance labels and documented in the dataset appendices. Acknowledgements This work was partially funded by the Ministry of Education, Research, Development and Youth of the Slovak Republic under the project KEGA 049TUKE-4/2024; by VEGA 1/0685/26; by the Slovak Research and Development Agency under the project APVV-22-0414; by the European Union under the project lorAI – Low Resource Artificial Intelligence, GA No. 101136646; and by CLARIN-SK, the Slovak national consortium of the CLARIN ERIC research infrastructure. It was also partially funded by the EU NextGenerationEU through the Recovery and Resilience Plan for Slovakia under the project No. 09I02-03-V01-00029. Part of the research results was obtained using the computational resources procured in the national project National competence centre for high performance computing (project code: 311070AKF2), funded by the European Regional Development Fund, EU Structural Funds Informatization of Society, Operational Program Integrated Infrastructure, as well as the computational resources of the supercomputer PERUN, operated by the Supercomputing Centre at the Technical University of Košice (TUKE), Slovakia, with the support of the European Union from the funds of the Recovery and Resilience Plan of the Slovak Republic within the framework of project No. 17I03-04-P03-00001, Development and design of a supercomputer for the National Supercomputing Center. References Ahuja et al. (2024) Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2024. MEGAVERSE: benchmarking large language models across languages, modalities, models and tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 2598–2637. Association for Computational Linguistics. Ali et al. (2024) Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Rutmann, Charvi Jain, Max Lübbering, Daniel Steinigen, Johannes Leveling, Katrin Klug, Jasper Schulze Buschhoff, Lena Jurkschat, Hammam Abdelwahab, Benny Jörg Stein, Karl-Heinz Sylla, Pavel Denisov, Nicolo’ Brandizzi, Qasid Saleem, and 22 others. 2024. Teuken-7b-base & teuken-7b-instruct: Towards european llms. Preprint, arXiv:2410.03730. Anthropic (2026) Anthropic. 2026. System Card: Claude Sonnet 4.6. https://w-cdn.anthropic.com/bbd8ef16d70b7a1665f14f306e88b53f686a75.pdf. Accessed: 2026-05-25. Apertus et al. (2025) Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan García Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, and 84 others. 2025. Apertus: Democratizing open and compliant llms for global language environments. Preprint, arXiv:2509.14233. Artetxe et al. (2020) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674–7684. Association for Computational Linguistics. Augustyniak et al. (2022) Łukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymczak, Arkadiusz Janz, Piotr Szymański, Marcin Wątroba, Mikołaj Morzy, Tomasz Jan Kajdanowicz, and Maciej Piasecki. 2022. This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. Bandarkar et al. (2024) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 749–775. Association for Computational Linguistics. Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. Bergmanis et al. (2026) Toms Bergmanis, Martins Kronis, Ingus Jānis Pretkalniņš, Dāvis Nicmanis, Jeļizaveta Jelinska, Roberts Rozis, Rinalds Vīksna, and Mārcis Pinnis. 2026. Tildeopen llm: Leveraging curriculum learning to achieve equitable language representation. Preprint, arXiv:2603.08182. BerriAI (2024) BerriAI. 2024. LiteLLM: A unified interface for LLM apis. https://github.com/BerriAI/litellm. Accessed: 2026-05-25. Blšták (2025) Miroslav Blšták. 2025. Slovak conceptual dictionary. Preprint, arXiv:2512.00579. CJVT and University of Ljubljana (2024) CJVT and University of Ljubljana. 2024. SloBench: The Slovene language technology evaluation platform. https://slobench.cjvt.si. Accessed: 2026-04-16. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457. Cohere et al. (2025) Team Cohere, :, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, and 211 others. 2025. Command a: An enterprise-ready large language model. Preprint, arXiv:2504.00698. Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2475–2485. Association for Computational Linguistics. Csaki et al. (2024) Zoltan Csaki, Bo Li, Jonathan Lingjie Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. SambaLingo: Teaching large language models new languages. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL), pages 1–21, Miami, Florida, USA. Association for Computational Linguistics. Dang et al. (2024) John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261. Dobeš and Bednár (2026) Marek Dobeš and Peter Bednár. 2026. SKJ1: Slovak grammar correction multi-choice set. Hugging Face dataset. To appear as “SKJ1 - Benchmark for Slovak Language Capabilities of Large Language Models”, TCL 2026. Accessed: 2026-04-20. Dussolle et al. (2025) Antoine Dussolle, Andrea Cardeña Díaz, Shota Sato, and Peter Devine. 2025. M-IFEval: Multilingual instruction-following evaluation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6176–6191, Albuquerque, New Mexico. Association for Computational Linguistics. Fajcik et al. (2025) Martin Fajcik, Martin Docekal, Jan Dolezal, Karel Ondrej, Karel Beneš, Jan Kapsa, Pavel Smrz, Alexander Polok, Michal Hradis, Zuzana Neverilova, Ales Horak, Radoslav Sabol, Michal Stefanik, Adam Jirkovsky, David Adamczyk, Petr Hyner, Jan Hula, and Hynek Kydlicek. 2025. BenCzechMark : A Czech-Centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism. Transactions of the Association for Computational Linguistics, 13:1068–1095. Fenogenova et al. (2024) Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton A. Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Anastasia Minaeva, Denis Dimitrov, Alexander Panchenko, and Sergey Markov. 2024. MERA: A comprehensive LLM evaluation in Russian. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9920–9948. Association for Computational Linguistics. Fourrier et al. (2024) Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open LLM leaderboard v2. Hugging Face Space https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard. Accessed: 2026-04-28. French (1999) Robert M. French. 1999. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135. Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2023. A framework for few-shot language model evaluation. Garabík (2026) Radovan Garabík. 2026. Slovenská wikipédia ako zdroj dát pre korpus chýb. Kultúra slova. In print. Google DeepMind (2026) Google DeepMind. 2026. Gemini 3.1 Pro Model Card. https://deepmind.google/models/model-cards/gemini-3-1-pro/. Accessed: 2026-05-07. Granite Team, IBM (2024) Granite Team, IBM. 2024. Granite 3.0 language models. Technical report, IBM. Grattafiori et al. (2024) Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783. Habib et al. (2023) Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval: A lightweight framework for llm evaluation. Accessed: 2026-02-15. Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300. Hládek et al. (2025) Daniel Hládek, Kristián Sopkovič, Ján Staš, Zuzana Sokolová, and Matúš Pleva. 2025. Financial question-answering dataset for slovak language model evaluation. Journal of Linguistics/Jazykovedný časopis, 76(1):247–257. Hládek et al. (2023) Daniel Hládek, Ján Staš, Jozef Juhár, and Tomáš Koctúr. 2023. Slovak dataset for multilingual question answering. IEEE Access, 11:32869–32881. Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research, pages 4411–4421. PMLR. Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems (NeurIPS): Datasets and Benchmarks Track, volume 36, pages 62991–63010. Hupkes and Bogoychev (2025) Dieuwke Hupkes and Nikolay Bogoychev. 2025. Multiloko: a multilingual local knowledge benchmark for llms spanning 31 languages. Preprint, arXiv:2504.10356. Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. Jumelet et al. (2026) Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2026. MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal pairs. Transactions of the Association for Computational Linguistics, 14:193–216. Klemen et al. (2024) Matej Klemen, Aleš Žagar, Jaka Čibej, and Marko Robnik-Šikonja. 2024. SI-NLI: A Slovene natural language inference dataset and its evaluation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 14859–14870. ELRA and ICCL. Koto et al. (2023) Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12359–12374. Koto et al. (2024) Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640. Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611–626. ACM. Kňažeková and Šuppa (2025) Natália Kňažeková and Marek Šuppa. 2025. NLI-SK-annotated: A native Slovak natural language inference dataset. Hugging Face dataset. Accessed: 2026-02-20. Lai et al. (2023) Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - System Demonstrations, Singapore, December 6-10, 2023, pages 318–327. Association for Computational Linguistics. Liang et al. (2023) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, and 31 others. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research. Featured Certification, Expert Certification, Outstanding Certification. Ligeti-Nagy et al. (2024) Noémi Ligeti-Nagy, Gergő Ferenczi, Enikő Héja, László János Laki, Noémi Vadász, Zijian Győző Yang, and Tamás Váradi. 2024. HuLU: Hungarian language understanding benchmark kit. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 8360–8371. ELRA and ICCL. Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics. Luo et al. (2023) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. Preprint, arXiv:2308.08747. Martins et al. (2024) Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. EuroLLM: Multilingual language models for Europe. Preprint, arXiv:2409.16235. MiniMax et al. (2025) MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, and 71 others. 2025. Minimax-01: Scaling foundation models with lightning attention. Preprint, arXiv:2501.08313. Ociepa et al. (2025) Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, and Remigiusz Kinas. 2025. Bielik 11b v2 technical report. Preprint, arXiv:2505.02410. Ondrejová and Šuppa (2024) Viktória Ondrejová and Marek Šuppa. 2024. Can LLMs handle low-resource dialects? a case study on translation and common sense reasoning in šariš. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 130–139, Mexico City, Mexico. Association for Computational Linguistics. Ondrejová and Suppa (2024) Viktoria Ondrejová and Marek Suppa. 2024. SlovakSum: A large scale Slovak summarization dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 14916–14922. ELRA and ICCL. OpenAI et al. (2025) OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. OpenAI (2026) OpenAI. 2026. GPT-5.5 System Card. https://deploymentsafety.openai.com/gpt-5-5. Accessed: 2026-05-10. Pecar et al. (2019) Samuel Pecar, Marián Simko, and Mária Bieliková. 2019. Improving sentiment classification in Slovak language. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, BSNLP@ACL 2019, Florence, Italy, August 2, 2019, pages 114–119. Association for Computational Linguistics. Pikuliak et al. (2022) Matúš Pikuliak, Štefan Grivalský, Martin Konôpka, Miroslav Blšták, Martin Tamajka, Viktor Bachratý, Marián Simko, Pavol Balážik, Michal Trnka, and Filip Uhlárik. 2022. SlovakBERT: Slovak masked language model. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 7156–7168. Association for Computational Linguistics. Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association for Computational Linguistics. Popović (2015) Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. Popović (2017) Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. Romanou et al. (2025) Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed A. Haggag, Snegha A., Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, and 38 others. 2025. INCLUDE: evaluating multilingual language understanding with regional knowledge. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Rybak et al. (2020) Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Ireneusz Gawlik. 2020. KLEJ: comprehensive benchmark for Polish language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1191–1201. Association for Computational Linguistics. Sainz et al. (2024) Oscar Sainz, Iker García-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, and 1 others. 2024. Data contamination report from the 2024 CONDA shared task. In Proceedings of the 1st Workshop on Data Contamination (CONDA), pages 41–56, Bangkok, Thailand. Association for Computational Linguistics. Shavrina et al. (2020) Tatiana Shavrina, Alena Fenogenova, Anton A. Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. RussianSuperGLUE: A Russian language understanding evaluation benchmark. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4717–4726. Association for Computational Linguistics. Singh et al. (2025) Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics. Smart (2025) Dan Saattrup Smart. 2025. Multiwikiqa: A reading comprehension benchmark in 300+ languages. Preprint, arXiv:2509.04111. SNK (2025) SNK. 2025. Slovak National Corpus (prim-11.0). https://korpus.sk. Accessed: 2025-04-11. Ľ. Štúr Institute of Linguistics, Slovak Academy of Sciences. Sokolová et al. (2025) Zuzana Sokolová, Maroš Harahus, Daniel Hládek, and Ján Staš. 2025. Annotated slovak datasets for toxicity, hate speech, and sentiment analysis. Journal of Linguistics/Jazykovedný časopis, 76(1):279–289. Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, and 1 others. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res., 2023. Štefánik et al. (2023) Michal Štefánik, Marek Kadlčík, Piotr Gramacki, and Petr Sojka. 2023. Resources and few-shot learners for in-context learning in Slavic languages. In Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023), pages 94–105, Dubrovnik, Croatia. Association for Computational Linguistics. Suba et al. (2023) David Suba, Marek Suppa, Jozef Kubik, Endre Hamerlik, and Martin Takac. 2023. WikiGoldSK: Annotated dataset, baselines and few-shot learning experiments for Slovak named entity recognition. In Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023), pages 138–145, Dubrovnik, Croatia. Association for Computational Linguistics. Suppa and Adamec (2020) Marek Suppa and Jergus Adamec. 2020. A summarization dataset of Slovak news articles. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 6725–6730. European Language Resources Association. Suppa et al. (2025) Marek Suppa, Andrej Ridzik, Daniel Hládek, Tomáš Javůrek, Viktoria Ondrejová, Kristína Sásiková, Martin Tamajka, and Marián Šimko. 2025. skLEP: A Slovak general language understanding benchmark. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 26716–26743. Association for Computational Linguistics. Šuppa et al. (2026) Marek Šuppa, Andrej Ridzik, Daniel Hládek, Natália Kňažeková, and Viktória Ondrejová. 2026. SkMTEB: Slovak massive text embedding benchmark and model adaptation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 45597–45628. Association for Computational Linguistics. Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca. Team et al. (2025a) Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025a. Gemma 3 technical report. Preprint, arXiv:2503.19786. Team et al. (2025b) GLM-4.5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, and 152 others. 2025b. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. Preprint, arXiv:2508.06471. Team et al. (2025c) Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, and 181 others. 2025c. Kimi k2: Open agentic intelligence. Preprint, arXiv:2507.20534. Upadhayay and Behzadan (2023) Bibek Upadhayay and Vahid Behzadan. 2023. Taco: Enhancing cross-lingual transfer for low-resource languages in llms through translation-assisted chain-of-thought processes. Preprint, arXiv:2311.10797. Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261–3275. Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Yong et al. (2023) Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina. 2023. BLOOM+1: Adding language support to BLOOM for zero-shot prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11682–11703, Toronto, Canada. Association for Computational Linguistics. Zahradník and Suppa (2025) Adam Zahradník and Marek Suppa. 2025. Trojsten benchmark: Evaluating LLM problem-solving in Slovak STEM competition problems. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35106–35121, Suzhou, China. Association for Computational Linguistics. Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911. Appendix A Lessons for Under-Resourced-Language Benchmark Builders This appendix expands the four lessons of §7, each with the Slovak result and per-model table behind it, and ends with two recommendations for benchmark builders. L1: Spend the native-data budget where translation fails. On closed-form skills translation preserves the coarse ordering of models but separates them less well at the frontier, where a leaderboard matters most. Native and translated NLI agree on the entire top-10, whereas native and translated multiple choice agree on only 7 of 10, so translated MMLU separates the strongest models less reliably than a native exam set does (§6.1, Table 2). Pooled over all 55 models the two contrasts look equally safe (ρ=0.99ρ=0.99 for NLI, ρ=0.98ρ=0.98 for multiple choice, per-model scatter in Figure 4), but a single correlation over a pool spanning 350 M to 1.1 T parameters is dominated by between-tier ordering and so overstates how interchangeable the two leaderboards are. Our frontier bands hold at most six or seven models, so the divergence within them is noisy and only qualitative († in Table 2). Native data pays off most where translation is weak or impossible: on question answering, where human-authored and LLM-generated questions rank models very differently (ρ=0.72ρ=0.72), and on grammar and morphology, which translation cannot produce at all. A native paired contrast is how a new language finds out which skills those are. 005050100100005050100100native NLItranslated NLINLI (ρ=0.99ρ=0.99)005050100100005050100100native examstranslated MMLUMultiple choice (ρ=0.98ρ=0.98)005050100100005050100100human skLEP-QAgenerated WikiQAQA (ρ=0.72ρ=0.72) Figure 4: RQ1 data-provenance contrasts: each point is one of the 55 models, scored on the native source (x) and its translated or LLM-generated counterpart (y). The dashed line is y=xy=x. Points hug the diagonal for NLI and multiple choice (translation preserves the ranking) but scatter off it for QA (human-authored vs. generated questions disagree), concentrated among weaker models. Per-tier ρ and top-10 overlap are in Table 2. L2: Budget an instruction-repair step after language adaptation. Continued training is known to degrade previously learned behaviour (French, 1999; Luo et al., 2023), and the cost of adapting LLMs to new languages is an active research topic (Yong et al., 2023; Csaki et al., 2024). We measure how large this effect is for a low-resource language and what it costs to repair. Continued pretraining on Slovak lowers an instruction-tuned model’s overall score from 47.447.4 to 33.533.5 (Qwen3-14B, §6.2), and the loss is concentrated in classification (−30.6-30.6), reading (−25.2-25.2) and knowledge (−23.7-23.7) against a grammar gain (+6.8+6.8): adaptation helps Slovak grammar but hurts content and comprehension. Instruction following falls by only −6.4-6.4. A small Slovak instruction set then recovers roughly three quarters of the loss (→43.8→\!43.8, per-base scores in Table 3), but only for bases that retain enough capability. A repair step should therefore be planned from the start. Overall Δ by group Lineage Base +Alp. Δ Know. Class. Instr. Gram. Mistral-SK 7B 19.0 14.9 -4.0 -1.1 -16.9 -0.4 +4.9 Qwen3-SK 14B 33.5 43.8 +10.3 +26.6 +16.5 +13.2 -19.8 TildeOpen-SK 30B 31.9 32.8 +0.8 -0.6 -29.2 +11.5 +30.7 Table 3: Effect of Slovak Alpaca-style instruction-tuning on three base models (Slovak, 0-shot), sorted by size. Base and +Alp. are the overall macro-average of the base Slovak LM and its instruction-tuned counterpart. The Δ columns are tuned −- base for the overall score and the four most affected reporting groups (Knowledge & reasoning, Classification, Instruction following, Grammar & morphology). The net effect rises with base capability—negative for the weakest base (in bold), flat for the mid one, clearly positive for the strongest—while instruction following (the tuned skill) improves or holds for all three and the other groups swing in both directions. Figure 5: The Qwen3-14B lineage: base, after Slovak continued pretraining, and after the subsequent Slovak instruction-tuning step, per reporting group (overall score beside each label). Adaptation trades broad capability for a single grammar gain. Instruction tuning recovers knowledge, reading, and instruction following, and reverses the grammar gain. Scores are the six group averages from Table 14. Know.Read.Class.Gen.Instr.Gram.Mistral-SK 7BKnow.Read.Class.Gen.Instr.Gram.TildeOpen-SK 30BKnow.Read.Class.Gen.Instr.Gram.Qwen3-SK 14Boriginal (Qwen only)Slovak base++ Alpaca Figure 6: RQ2: per-reporting-group scores (Slovak, 0-shot, axes 00–100100) of three Slovak base checkpoints before (Slovak base) and after (++Alpaca) a small Slovak instruction-tuning step. For Qwen3-SK, whose upstream multilingual model (Qwen3-14B) was also evaluated, the dashed polygon shows that released model before Slovak adaptation. The upstream bases of Mistral-SK and TildeOpen-SK are not in the sweep. The effect varies by base and by group: Alpaca broadly lifts Qwen3-SK (knowledge, reading, classification, instruction-following) at the cost of grammar, is mixed for TildeOpen-SK, and mostly shrinks the weakest (Mistral-SK), and Slovak adaptation itself drops Qwen3-14B before Alpaca partly restores it (§6.2, with per-group deltas in Table 3). L3: Turn on test-time reasoning before increasing model size. Reasoning mode yields large, consistent gains at ≥ 9B (+8.5+8.5 to +12.5+12.5 overall, §6.3, per-model deltas in Table 4), mostly in instruction following and grammaticality. The gains hold in Slovak, but only above a capacity floor: below it, small models exhaust their generation budget before emitting an answer. Overall Δ by group Model Std Think Δ Instr. Gram. Gen. Qwen3 0.6B 21.5 27.2 +5.7 +19.6 +1.1 -0.3 Qwen3.5 0.8B† 24.0 7.2 -16.8 +10.1 +3.6 -9.4 Qwen3 1.7B 27.8 40.0 +12.2 +23.8 +10.2 -1.0 Qwen3.5 2B† 29.4 19.8 -9.7 +7.1 +12.5 -10.8 Qwen3.5 9B 47.9 56.4 +8.5 +21.2 +28.8 -3.5 Qwen3.5 27B 55.3 64.3 +8.9 +35.8 +11.6 -0.9 Qwen3.6 27B 54.2 65.6 +11.4 +43.2 +8.6 -0.4 Gemma 4 31B 54.3 66.0 +11.7 +53.8 +8.2 +0.2 Qwen3.6 35B-A3B 52.2 64.7 +12.5 +44.1 +13.8 -0.5 Table 4: Effect of test-time reasoning (“thinking” mode), Slovak, 0-shot, for the models evaluated in both modes (sorted by size). Std and Think are the overall macro-average. The Δ columns are thinking −- standard for the overall score and for the three most affected reporting groups (Instruction following, Grammar & morphology, Generation). Reasoning runs under an 8,192-token generation cap. The two smallest models (†) exhaust it inside the reasoning trace without emitting a parseable answer, so their answer-extraction groups collapse while instruction and grammar still improve (negative overall Δ in bold). Full per-task thinking scores are in Table 16. Why small models regress under reasoning. Under the token cap the two smallest Qwen3.5 models exhaust their budget inside the reasoning trace without emitting a parseable answer, so their multiple-choice accuracy falls to ∼ 1% (vs. 42% in standard mode), well below random. This pulls down every group scored by answer extraction, while instruction following and grammar, scored over the whole generation, still rise (§6.3). Thinking mode also raises the generation budget (8,192 tokens vs. the per-task standard caps of 32–4,096), but those caps already exceed each task’s answer length, so the extra budget goes to the reasoning trace, not the answer. The gains also land on short-answer skills (grammar, knowledge), where a longer budget alone cannot help. A budget-matched control is left to future work. L4: Do not over-invest in target-language prompts for scoring. Re-running the sweep with English prompts moves the overall score by only +0.4+0.4 (Table 5). The prompt language matters for grammar tasks and little elsewhere. This concerns the instructions only: the test items themselves must still be Slovak. Reporting group SK EN Δ Knowledge & Reasoning 58.1 58.3 +0.1 Reading & QA 39.4 41.0 +1.6 Classification 61.3 61.1 -0.2 Generation 13.2 13.1 -0.1 Grammar & Morphology 33.5 30.9 -2.6 Overall 49.9 50.3 +0.4 Table 5: Prompt-language effect: mean Slovak- vs. English-prompt score per reporting group, macro-averaged across all evaluated models over the tasks available in both languages (IFEval-SK and Slohry are Slovak-only and excluded). Δ=EN−SK =EN-SK. A positive value means the English prompt scores higher. Full per-task English scores are in Table 17. skLEP BCM MERA HuLU INCL sk-bench Language sk cs ru hu multi sk Generative LLM eval ✓ ✓ ✓ ✓ Native paired contrast ✓ Open + closed, one harness ∼ ✓ Grammar & morphology ∼ ✓ Instruction-following ∼ ✓ Table 6: How sk-bench differs from the closest Slavic, Central-European, and multilingual LLM benchmarks. The lower block lists the design choices that set sk-bench apart. Cells mark what each benchmark actually provides (∼ = partial). BenCzechMark (BCM) and MERA score by log-likelihood, which closed APIs do not expose, so neither ranks open- and closed-weights models on one leaderboard. sk-bench’s generate-and-match scoring does. INCL = INCLUDE. Of sk-bench’s 30 tasks, 11 are resources introduced or first packaged here (Table 10). We do not tabulate the same count for the other suites, whose new-versus-integrated split we cannot establish uniformly from their papers. Recommendations for benchmark builders. (i) Ship at least one native paired contrast per skill, so you can measure whether translation is safe for your language and skill (Table 6). (i) Re-implement instruction and verifier checkers natively rather than translating them: Slovak word boundaries, casing, and diacritics break the original IFEval checker on its own examples (Appendix L). Long-context robustness. Table 7 reports the OneRuler-SK results by context length referenced in §6.4. Because the Slovak and English haystacks are matched in length and run on the same models, the Slovak−-English gap at a fixed length separates a language penalty from the model’s context-window limit. Two things follow. First, there is no Slovak-specific length cliff up to 16k: macro-averaged over all 55 models, Slovak retrieval declines only mildly from 4k to 16k (→54.458.1\!→\!55.6\!→\!54.4), and the gap to English does not widen with length; it narrows and slightly reverses (−2.4/−0.7/+1.1-2.4/-0.7/+1.1). Long-context retrieval over Slovak text thus depends on the model, not the language, consistent with L4. Second, scores vary across models, not lengths: the six proprietary APIs stay above 99%99\% at every length, a few models do degrade with length (e.g. Kimi K2.6, →23.163.7\!→\!23.1 from 4k to 16k), and several base models—including the Slovak-adapted ones—score near zero already at 4k. That floor is the same answer-not-emitted format failure seen for the smallest reasoning models (§6.3), not a length effect. Because 16k is the longest length we evaluate, we cannot locate a breaking point for the capable models, which stay flat across this range. Finding where Slovak long-context retrieval breaks down would need 32k–128k haystacks, which we leave to future work. Context SK EN Δ 4,096 58.1 60.5 -2.4 8,192 55.6 56.4 -0.7 16,384 54.4 53.4 +1.1 Table 7: OneRuler-SK by context length: mean retrieval accuracy (×100× 100, 0-shot) over the RULER-style subtasks at 4,096 / 8,192 / 16,384 tokens, macro-averaged across all evaluated models, for the matched Slovak and English haystacks. Because SK and EN share the same model and the same length, Δ=SK−EN =SK-EN isolates the language effect from the context-window limit. The gap does not widen with length—it closes and slightly reverses by 16k—so up to 16k there is no Slovak-specific long-context penalty. 16k is the longest length evaluated. Few-shot vs. zero-shot. Table 8 gives the full 3-shot vs. zero-shot breakdown by reporting group and prompt language behind the summary in §6.4. The effect is small and preserves the ranking: the four-group overall rises by only +2.0+2.0 (Slovak) and +1.6+1.6 (English), so zero-shot scores do not noticeably understate any model. The gains come from classification and reading & QA, where a few examples fix the output format and label set; knowledge & reasoning drops slightly, since examples add no facts and only use up context. Slovak and English prompts behave almost identically, except on grammar & morphology, where Slovak examples help slightly (+1.1+1.1) and English ones hurt (−1.6-1.6). (The table lists all six reporting groups. Generation and instruction following were run zero-shot only, so they appear at their 0-shot value with no 3-shot entry, and the “Overall” row spans the four paired groups, so it is not directly comparable to the six-group overall of the main leaderboard.) As with Alpaca tuning (§6.2) and reasoning (§6.3), examples teach a model how to answer, not what to know. Zero-shot is therefore a fair default, and few-shot is worth its context cost mainly on fixed-label tasks with strict output formats. Slovak English Reporting group 0-shot 3-shot 0-shot 3-shot Knowledge & reasoning 60.6 59.4 61.2 60.4 Reading & QA 29.4 32.6 30.7 33.6 Classification 61.9 66.8 61.3 67.0 Grammar & morphology 35.7 36.8 28.6 27.0 Overall 46.9 48.9 45.4 47.0 Table 8: Zero-shot vs. 3-shot, macro-averaged over the 52 models with few-shot results, for the four reporting groups of the main leaderboard (Table 1) that have a few-shot counterpart. Generation (summarisation) and instruction following were run zero-shot only and are therefore omitted. The Overall row is the macro-average over these four groups. Few-shot helps most on classification and reading & QA, slightly hurts knowledge & reasoning, and moves the overall little. The pattern is the same under Slovak and English prompts. Appendix B Costs As shown in Table 9, our evaluation consumed 5,354 GPU hours (1,454 on the cluster with A100 40 GB GPUs and 3,900 on the cluster with H200 141 GB GPUs) and 15,500 CPU hours across 764,079 prompts across all open-source models. Proprietary API costs ranged from $102.40 (Gemini 3.1 Flash Lite) to $3,833.67 (Gemini 3.1 Pro), with GPT-5.4 Mini at $286.96, GPT-5.5 at $2,761.67, and Claude Sonnet 4.6 and Claude Opus 4.6 at $1,560.83 and $1,353.67, respectively. The Claude Sonnet 4.6 figure aggregates both its zero-shot and few-shot runs, whereas Claude Opus 4.6, like Gemini 3.1 Pro and GPT-5.5, was evaluated in the zero-shot setting only. Inference times for open-source models ranged from 1.84 to 80.83 hours, scaling with model size. Model Input Tokens [M] Completion Tokens [M] Total Tokens [M] Cost Evaluation Time [hours] Open-source Models ibm-granite/granite-4.0-h-350m 457.03 7.27 464.29 N/A 4.17 ibm-granite/granite-4.0-350m 457.03 10.83 467.85 N/A 1.97 Qwen/Qwen3-0.6B 439.36 14.13 453.49 N/A 2.49 Qwen/Qwen3.5-0.8B 369.88 13.04 382.92 N/A 3.16 google/gemma-3-1b-it 361.46 8.33 369.79 N/A 2.22 ibm-granite/granite-4.0-h-1b 458.30 7.39 465.69 N/A 5.04 ibm-granite/granite-4.0-1b 458.30 8.31 466.62 N/A 3.09 Qwen/Qwen3-1.7B 439.36 17.38 456.74 N/A 3.04 utter-project/EuroLLM-1.7B-Instruct 332.54 12.50 345.05 N/A 2.03 ibm-granite/granite-3.2-2b-instruct 506.76 20.90 527.67 N/A 5.12 Qwen/Qwen3.5-2B 369.88 12.82 382.70 N/A 3.96 meta-llama/Llama-3.2-3B-Instruct 405.58 10.80 416.39 N/A 3.36 CohereLabs/tiny-aya-global 602.03 13.82 615.85 N/A 3.60 CohereLabs/tiny-aya-water 602.03 13.74 615.77 N/A 3.65 openGPT-X/Teuken-7B-instruct-v0.6 341.66 18.52 360.17 N/A 9.75 slovak-nlp/mistral-sk-7b 449.54 12.29 461.82 N/A 7.87 meta-llama/Llama-3.1-8B-Instruct 405.58 11.07 416.65 N/A 5.91 mistralai/Ministral-3-8B-Instruct-2512 372.43 15.44 387.87 N/A 2.07 mistralai/Ministral-3-8B-Reasoning-2512 372.43 46.72 419.15 N/A 2.34 CohereLabs/aya-expanse-8b 359.57 22.26 381.83 N/A 6.25 utter-project/EuroLLM-9B-Instruct-2512 340.57 12.83 353.40 N/A 6.52 Qwen/Qwen3.5-9B 369.88 9.41 379.29 N/A 9.73 speakleash/Bielik-11B-v3.0-Instruct 456.55 15.94 472.50 N/A 12.11 mistralai/Ministral-3-14B-Instruct-2512 372.43 14.39 386.82 N/A 1.84 mistralai/Ministral-3-14B-Reasoning-2512 372.43 530.61 903.03 N/A 12.65 Qwen/Qwen3-14B 439.36 9.81 449.17 N/A 2.73 slovak-nlp/Qwen3-14B-sk 439.36 42.09 481.45 N/A 16.33 openai/gpt-oss-20b 402.44 85.76 488.20 N/A 3.32 utter-project/EuroLLM-22B-Instruct-2512 340.57 9.84 350.41 N/A 4.92 Qwen/Qwen3.5-27B 369.89 9.85 379.74 N/A 12.55 Qwen/Qwen3.6-27B 369.89 8.85 378.73 N/A 11.38 TildeAI/TildeOpen-30b 446.01 9.37 455.38 N/A 7.13 TildeAI/TildeOpen-30b-64k 306.24 7.96 314.20 N/A 5.91 google/gemma-4-31B-it 364.16 7.97 372.13 N/A 8.39 CohereLabs/aya-expanse-32b 359.57 16.99 376.56 N/A 5.80 ibm-granite/granite-4.0-h-small 462.76 9.20 471.96 N/A 4.14 Qwen/Qwen3.6-35B-A3B 369.89 9.40 379.29 N/A 7.01 meta-llama/Llama-3.3-70B-Instruct 405.58 8.89 414.47 N/A 7.52 swiss-ai/Apertus-70B-Instruct-2509 411.40 24.64 436.03 N/A 9.87 CohereLabs/c4ai-command-a-03-2025 742.28 19.38 761.66 N/A 4.67 openai/gpt-oss-120b 402.44 67.54 469.98 N/A 6.52 Qwen/Qwen3.5-397B-A17B 369.89 9.90 379.79 N/A 10.54 mistralai/Mistral-Large-3-675B-Instruct-2512 372.43 19.59 392.02 N/A 7.65 MiniMaxAI/MiniMax-M2.7 438.59 836.75 1275.34 N/A 80.83 zai-org/GLM-5.1-FP8 399.11 8.14 407.25 N/A 9.05 moonshotai/Kimi-K2.6 414.52 14.44 428.96 N/A 6.66 Mistral-SK 7B (Alpaca) 456.42 28.10 484.52 N/A 3.01 Qwen3-SK 14B (Alpaca) 439.36 12.88 452.24 N/A 4.37 TildeOpen-SK 30B (Alpaca) 314.65 13.29 327.94 N/A 6.39 Thinking Versions Qwen/Qwen3-0.6B 435.63 602.11 1037.74 N/A 18.76 Qwen/Qwen3.5-0.8B 368.36 6119.13 6487.49 N/A 213.20 Qwen/Qwen3-1.7B 435.63 793.35 1228.98 N/A 67.53 Qwen/Qwen3.5-2B 368.96 3019.48 3388.44 N/A 310.53 Qwen/Qwen3.5-9B 368.35 2380.40 2748.75 N/A 393.67 Qwen/Qwen3.5-27B 368.36 1764.76 2133.12 N/A 136.75 Qwen/Qwen3.6-27B 368.36 808.09 1176.45 N/A 87.62 google/gemma-4-31B-it 366.46 456.12 822.58 N/A 89.84 Qwen/Qwen3.6-35B-A3B 368.36 865.91 1234.27 N/A 57.95 Proprietary Models Gemini 3.1 Flash Lite 358.41 9.13 367.53 $102.40 — Gemini 3.1 Pro* 166.26 577.82 744.08 $3,833.67 — GPT-5.4 Mini 357.55 9.48 367.02 $286.96 — GPT-5.5* 167.70 128.53 296.23 $2,761.67 — Claude Sonnet 4.6 387.62 26.53 414.15 $1,560.83 — Claude Opus 4.6* 184.45 17.26 201.71 $1,353.67 — Table 9: Resource consumption and evaluation time across models on the sk-bench benchmark. Token counts are shown in millions (M). * denotes models evaluated only in the zero-shot setting due to API constraints. Evaluation time is measured in hours. “—” indicates data not available or not applicable. Appendix C Full Dataset Catalogue Table 10 lists every dataset in sk-bench with its evaluation size, provenance, target skill, licence, and a one-line description. It expands on the overview in Figure 1. Table 10: Dataset catalogue for sk-bench. |test|| test| is the evaluation test-set size used by the released harness. Prov.: N=Slovak-native, MT=translated from English, MT∗^\!*=translated and adapted with a custom Slovak instruction registry, G=generated over Slovak source documents. Resources marked ★ are the eleven introduced or first packaged with sk-bench, at three levels of contribution: built from scratch (IFEval-SK, Chiby, Slohry, Slovak Trivia, OneRuler-SK), newly curated from public institutional sources (Autoškola, e-testy-sk, Slovak-tests, Scio), and existing expert resources first packaged for generative-LLM evaluation (SKJ1, Word Analogies). Licence is the licence declared on the source artefact (“Mixed” = the bundled release carries per-task licences; “—” = none declared). Dataset |test|| test| Prov. Skill Licence Description SentiSK 4,677 N sentiment C BY-SA 4.0 Slovak user-text sentiment corpus used in binary and 3-class settings. Neutral examples are excluded from the binary score. MovieReview 2,008 N sentiment — Slovak film- and series-review sentiment with labels derived from user star ratings (Štefánik et al., 2023), included as a second native binary sentiment source. See Appendix D.3. skLEP-sentiment 1,042 N sentiment Mixed The sentiment subtask from skLEP, retained as a small validation cross-check against the larger sentiment resources. ToxicSK 881 N safety classification C BY-SA 3.0 Slovak toxicity-detection corpus scored as yes/no classification. skLEP-hate 1,319 N safety classification Mixed skLEP hate-speech subtask, paired with ToxicSK to separate toxicity from explicit hate-speech labelling. NLI-SK-annotated 597 N NLI — Human-annotated Slovak 3-way entailment dataset after filtering the non-NLI “Other” category. skLEP-NLI 5,004 MT NLI Mixed Translated 3-way NLI data from skLEP, used as the translated counterpart to NLI-SK-annotated. skLEP-RTE 1,660 MT RTE Mixed Separate translated binary textual-entailment dataset from GLUE RTE, included as a second entailment task alongside skLEP-NLI. skLEP-QA 9,583 N extractive QA Mixed Slovak Wikipedia/news question-answering data, evaluated with normalised exact match and token F1. MultiWikiQA-SK 4,997 G open QA C BY-NC-SA 4.0 LLM-generated Slovak Wikipedia QA pairs, included to contrast human-authored and generated prompts over the same broad source domain. SlovakSum 1,500 N summarisation — Large Slovak news summarisation corpus. sk-bench uses an evaluation-sized down-sample from 20,662 items with the original human reference summaries. SME-Sum 1,500 N summarisation — Slovak news summarisation dataset based on SME articles. sk-bench uses an evaluation-sized down-sample from 8,001 items. Belebele-SK 900 MT reading comprehension C BY-SA 4.0 Human-translated Slovak slice of the parallel Belebele multiple-choice reading-comprehension benchmark. ARC-SK 1,169 MT science reasoning C BY-NC 4.0 Okapi translation of ARC Challenge science questions, used as a compact translated reasoning benchmark. Okapi-MMLU-SK 13,062 MT knowledge reasoning — Slovak translation of MMLU-style multi-subject questions across 57 subjects, the largest translated multi-choice component in the suite. TruthfulQA-SK 778 MT truthfulness C BY-NC 4.0 Slovak translation of TruthfulQA multiple-choice questions designed to elicit common false beliefs. SlovakCOPA 500 MT causal reasoning C BY-NC 4.0 Slovak version of COPA, testing plausible causes and effects in short everyday scenarios. e-testy-sk ★ 444 N school exams Other Slovak high-school examination item bank covering multiple school subjects. Slovak-tests ★ 343 N Slovak-language exams — Subset of native school-exam questions that specifically test Slovak grammar, orthography, and vocabulary. Financial Exams 1,334 N professional exams C BY-SA 4.0 Slovak financial-sector certification questions, testing domain-specific regulatory and procedural knowledge. Autoškola ★ 701 N professional exams C BY 4.0 Slovak driving-licence exam questions, included for law- and culture-specific practical knowledge. Chiby ★ 3,432 N grammar C BY-SA 4.0 Grammaticality classification data derived from Slovak Wikipedia revisions and manually filtered for spelling, diacritics, and capitalisation errors. SKJ1 ★ 100 N grammar — Slovak grammar-correction multiple-choice set from TUKE-UUI. Scio ★ 1,785 N admission tests — Slovak Scio admission-test questions spanning biology, mathematics, chemistry, and general knowledge. Slohry ★ 4,004 N grammar C BY 4.0 Slovak word-game items testing lexical and morphological competence. Slovak Trivia ★ 353 N world knowledge — Native Slovak trivia questions on culture, geography, and general knowledge. Word Analogies ★ 2,651 N analogy — Slovak word-analogy completion items testing lexical-semantic relations. Global-MMLU-Lite 400 MT knowledge reasoning Apache 2.0 Curated Slovak slice of the Global-MMLU-Lite multilingual knowledge benchmark. IFEval-SK ★ 537 MT∗^\!* instruction following — Slovak adaptation of IFEval with translated prompts and programmatic checkers re-implemented for Slovak word boundaries, casing, and punctuation. OneRuler-SK ★ 9,000 N long context — Slovak long-context reading-comprehension suite probing retrieval and reasoning over extended Slovak text. Appendix D Task Descriptions Skill Native side Contrast side NLI NLI-SK-annotated skLEP-NLI (MT) Multiple-choice e-testy-sk ★ , Okapi-MMLU-SK (MT) knowledge Slovak-tests ★ Question answering skLEP-QA (human) MultiWikiQA-SK (G) Grammar & morph. Chiby ★ , SKJ1 ★ , translation cannot Slohry ★ produce these Table 11: The paired provenance contrasts that structure the suite. Each row holds a skill fixed and varies how the evaluation data was produced. ★ marks a resource introduced or first packaged with sk-bench. MT = machine-translated, G = LLM-generated. The grammar row has no contrast side: translation cannot produce these items, so they must be built natively. §6.1 measures how far the rankings disagree in each row. Table 10 lists each dataset’s size, provenance, and skill. This appendix describes, for every task in the leaderboard (Table 1), what the model is asked to do, how it is scored, where the data comes from, and why the task is in the suite. Resources marked ★ are introduced or first packaged with sk-bench. Several have full curation cards in Appendix J. Prompt templates are in Appendix M, and throughout the paper a dataset name is a hyperlink to its entry here. D.1 Knowledge & Reasoning ARC-SK. Multiple-choice grade-school science questions. The model selects one option, scored by accuracy. Okapi machine translation of ARC-Challenge, a small translated reasoning set (Clark et al., 2018; Lai et al., 2023). Okapi-MMLU-SK. Four-option multiple choice spanning MMLU’s 57 academic subjects, scored by accuracy. This is the largest translated multi-choice component. With ARC-SK it is the translated side of the multi-choice provenance contrast (RQ1) against the native e-testy-sk/Slovak-tests (Hendrycks et al., 2020; Lai et al., 2023). Global-MMLU-Lite. Four-option multiple choice, scored by accuracy. A small curated slice of a multilingual knowledge benchmark, kept as an independent check on Okapi-MMLU-SK (Singh et al., 2025). TruthfulQA-SK. Multiple choice in the MC1 setting (single-true-answer multiple choice)—one true answer among several plausible-sounding false ones, scored by accuracy. Probes susceptibility to common misconceptions rather than recall (Lin et al., 2022; Lai et al., 2023). Scio ★ . Single-answer multiple choice, scored by accuracy. Items parsed from Scio admission-test papers covering general study prerequisites alongside biology, chemistry, and mathematics. Curated by the authors: the parsing, cleaning, and answer-key alignment are ours, and no external annotators were involved (Appendix I.1). Financial Exams. Five-option single-answer multiple choice, scored by accuracy. Parsed from the publicly released financial-advisor certification exams of the National Bank of Slovakia and tagged by topic (insurance, capital markets, deposits, loans, pensions) and difficulty (Hládek et al., 2025). Autoškola ★ . Multiple choice, scored by accuracy. Questions from the theory part of the official Slovak driving-licence test across 35 topic categories, with the official answer key as gold. Assembled by the authors from that official question bank with LLM assistance. The gold labels are the official answer key, not model output (Appendix I.2). SlovakCOPA. Binary forced choice between two alternatives for the more plausible cause or effect of a premise, scored by accuracy (chance =50%=50\%). A standard-Slovak translation of COPA (Ondrejová and Šuppa, 2024). SKJ1 ★ . Five-option (A–E) multiple choice, scored by accuracy. Slovak grammar-correction items spanning nine error types—gender agreement, inflection, reflexive sa/si, Bohemisms, Anglicisms, and others. The items are expert-authored by the team behind the SKJ1 dataset (Dobeš and Bednár, 2026) and adopted from that work. We recruited no experts ourselves, and sk-bench’s contribution is the generative-LLM wrapping—prompts, parsing, and scoring (Appendix I.3). Slovak Trivia ★ . Single-answer multiple choice, scored by accuracy. Native questions on Slovak culture, geography, and general knowledge, derived from short articles. Constructed by the authors for sk-bench. No external annotators were involved. D.2 Reading & QA Belebele-SK. Four-option multiple-choice reading comprehension over a short passage, scored by accuracy. The human-translated Slovak slice of the parallel Belebele set, so scores are comparable across its 122 languages (Bandarkar et al., 2024). Chiby ★ . Binary grammaticality judgement: decide whether a sentence drawn from Slovak Wikipedia revisions is correct or carries a spelling/diacritic/capitalisation error, scored by accuracy. Annotated from Chiby v0.4 by three native Slovak speakers who are co-authors of this work, after cosine-similarity pairing and language filtering (full procedure in Appendix K) (Garabík, 2026). Translation cannot produce such pairs. MultiWikiQA-SK. Free-text answer generation grounded in a Wikipedia passage, scored by case- and diacritic-normalised exact match and token F1. Questions were generated by an LLM (Gemini 1.5 Pro) over the 2023 Wikipedia dump (Smart, 2025) and then rephrased by the same model in a second pass to reduce surface word overlap with the passage. It is the generated side of the QA contrast (RQ1) against the human-authored skLEP-QA (Suppa et al., 2025) task (built on SK-QuAD (Hládek et al., 2023)). e-testy-sk ★ . High-school exam items from the e-testy.sk portal across nine subjects, reported in two formats: single-answer multiple choice (ETest-M, accuracy) and free-text questions (ETest-Q, normalised exact match / token F1). The items were public, but not as a dataset: the parsing, cleaning, and answer-key alignment are ours. It is the native multiple-choice counterpart to Okapi-MMLU-SK. Slovak-tests ★ . School and standardised-exam items (maturita, entrance exams, Monitor 9) focused on Slovak language and literature, reported as SKTest-M (multiple choice, accuracy) and SKTest-Q (free-text, normalised exact match / token F1). As with e-testy-sk, the items were public, but not as a dataset: the parsing, cleaning, and answer-key alignment are ours. OneRuler-SK ★ . Long-context retrieval and reasoning, scored by exact match against a gold string (the absent-needle case answers žiadne). A Slovak adaptation of ONERULER built over an 18th-century Slovak novel as the haystack with needle words drawn from Wiktionary, spanning needle-in-a-haystack and common-word-extraction tasks at 4k–16k tokens. The haystack, the needle placement, and the gold strings were constructed by the authors. The gold answer is determined by construction, so no annotation round was required. skLEP-QA. Extractive free-text QA over Slovak Wikipedia/news, scored by normalised exact match and token F1. The question-answering subtask of skLEP (Suppa et al., 2025) is built on the human-authored, crowd-annotated SK-QuAD (Hládek et al., 2023), and forms the human side of the QA contrast against generated MultiWikiQA-SK. The main text and the per-task tables of Appendix H refer to the evaluated task as skLEP-QA. D.3 Classification NLI-SK-annotated. Three-way entailment (entailment / neutral / contradiction), scored by accuracy. Hypotheses are hand-written. Premises are drawn from SlovakSum titles, hand-written items on Slovak traditions and geography, and image captions. Labels are assigned by three native Slovak speakers using unanimous or majority agreement (Kňažeková and Šuppa, 2025). The evaluated subset includes both. The resource was first announced in SkMTEB (Šuppa et al., 2026). It forms the native side of the NLI contrast (RQ1) against translated skLEP-NLI (Suppa et al., 2025). skLEP-NLI. Three-way entailment, scored by accuracy. The skLEP NLI subtask, machine-translated from XNLI with native post-editing—the translated counterpart in the NLI provenance contrast (Suppa et al., 2025). skLEP-RTE. Binary entailment, scored by accuracy. A separate machine-translated dataset from the GLUE RTE corpus, testing entailment versus non-entailment (Suppa et al., 2025). SentiSK. User-comment sentiment, reported both 3-class (Senti-3) and binary (Senti-2, neutral dropped), scored by accuracy. Slovak Facebook comments manually annotated at TU Košice. The 3-class set skews negative (Sokolová et al., 2025). skLEP-sentiment. Binary sentiment, scored by accuracy. The skLEP sentiment subtask (from the Reviews3 corpus), a small validation cross-check against the larger sentiment resources (Suppa et al., 2025). MovieReview. Binary sentiment over Slovak film and series reviews, scored by accuracy. Labels derive from user star ratings, making it a second native binary sentiment set from a different domain (Štefánik et al., 2023). ToxicSK. Binary toxicity detection (toxic / non-toxic), scored by accuracy. Slovak Facebook comments manually annotated at TU Košice, class-balanced between the two labels (Sokolová et al., 2025). skLEP-hate. Binary hate-speech detection, scored by accuracy. The skLEP hate-speech subtask (from the Slovak Hate Speech Database), paired with ToxicSK to separate general toxicity from explicit hate (Suppa et al., 2025). D.4 Generation SlovakSum. Abstractive news summarisation, with a generated summary scored by ROUGE-L against a single human reference. A length-stratified down-sample of a multi-source Slovak news corpus that preserves the original length distribution (Appendix J) (Ondrejová and Suppa, 2024). SME-Sum. Abstractive summarisation of SME.sk news articles, with the article perex as the reference summary, scored by ROUGE-L. A length-stratified down-sample of SME-Sum (Appendix J) (Suppa and Adamec, 2020). D.5 Grammar & Morphology Word Analogies ★ . Open-ended completion: given a labelled relation and three terms of an analogy, the model produces the missing fourth, scored by exact match. Probes lexical-semantic relations. The relation inventory and term pairs are taken from the Slovak Conceptual Dictionary (Blšták, 2025). sk-bench’s contribution is the generative-LLM wrapping—prompts, parsing, and scoring. Slohry ★ . Inflectional-morphology production across four sub-tasks—verb conjugation, adjective/adverb comparison, noun declension, and grammatical-form identification. The model generates the requested form, scored by exact match against a single gold answer. 4,004 items targeting Slovak morphological competence, generated by the authors from the standard Slovak paradigms, so the gold form follows from the paradigm rather than from an annotation judgement. D.6 Instruction Following IFEval-SK ★ . Verifiable instruction following: each prompt carries machine-checkable constraints (length, casing, keywords, format, response language) verified by programmatic checkers. Scoring uses prompt-level strict accuracy. A Slovak adaptation of M-IFEval whose 25 instruction types were re-implemented for Slovak word boundaries, casing, and punctuation (Appendix L) (Zhou et al., 2023; Dussolle et al., 2025). Both the Slovak templates and the checker registry are ours. Verification is programmatic, so no annotators were involved. Appendix E Scoring and Parsing Details This appendix describes how raw model outputs are parsed and normalised before scoring. Output parsing. The free-text parser proceeds in this fixed order: (i) strip leading/trailing whitespace and chat-template artefacts (“The answer is”, “Odpoveď’ :”, …); (i) apply Unicode NFC normalisation; (i) for multi-choice tasks, scan for the first occurrence of an isolated answer letter (A–Z) optionally followed by ), ., or whitespace and return that letter; (iv) for fixed-vocabulary classification, lowercase under the Slovak locale (preserving diacritics — č/c, í/i, š/s pairs are semantically distinct in Slovak) and match the first allowed label substring. If neither rule fires, the example is counted as incorrect for the main score. Classification metric calculation. We use two classification metrics. The conservative metric counts an answer as correct only when exactly one valid class label appears in the response, so an answer naming several labels scores zero even if one of them is correct; this penalises hedging. The multi-label metric, for tasks with several correct labels, extracts the set of labels in the response and requires it to equal the gold set exactly. Invalid outputs and refusals. Refusals (“Som AI a nemôžem…”), safety wrappers, and any output that the parser cannot resolve are counted as incorrect for the main score. Slovak normalisation for free-form tasks. For QA we apply NFC normalisation, Slovak-locale case folding (without diacritic stripping), and the QEM/QF1 token splitter from LightEval. We do not stem because Slovak inflection makes stemming brittle. Quasi-exact match and token-level quasi-F1 then compare the normalised answer with the gold answer. Summarisation metrics. ROUGE-1/2/L is computed with the rouge_score package (default tokeniser, no stemming). chrF/chrF++ and METEOR are computed through LightEval. Absolute scores are treated as ordinal. Token budgets. A single per-task generation budget is applied identically across providers (32 tokens for short-answer classification, 512 for QA and summarisation, 4096 for IFEval). Because Slovak tokenisation differs by model, this is not the same character or word budget across checkpoints. Output length and format failures. The harness’s per-task token logs show that output length is not a problem for almost any model. On multiple-choice tasks the graded answer is a median of 2 tokens, well within the 32-token budget, and across standard (non-reasoning) models the mean generation is only 8–37 tokens, none of which ever reaches the 8,192-token ceiling. Long, budget-exhausting generations happen only in reasoning mode: at least one generation reaches the ceiling on 88.6% of MiniMax M2.7’s evaluation runs and 46.3% of Ministral 3 14B-Reasoning’s, against 0% for every standard model. Even in reasoning mode we grade only the answer and not the reasoning trace — gpt-oss 20B averages 112 tokens per generation, of which only about 14 are the answer we score. Format failures that affect scores are thus concentrated in reasoning mode on small models (§6.3): when they exhaust the budget, their multiple-choice accuracy falls to about 1% against 42% in standard mode, because no answer is emitted, not because their reasoning got worse. Appendix F Model Catalogue Table 12 lists the evaluated open-weights models with their parameter count, licence, reported knowledge or pretraining cutoff where published (“—” otherwise), language coverage, the Hugging Face commit SHA pinned at evaluation time, and a reference to its family’s technical report. References are at the family level (e.g. all Qwen3.x variants cite the Qwen3 report). Our Slovak-adapted and fine-tuned models have no separate report and are described in §5. We record a reported knowledge or pretraining cutoff wherever the model card or technical report states one. Nine of the catalogued checkpoints do. The remaining entries are “—” because no cutoff is published. Cutoff-versus-dataset-date reasoning about possible contamination is therefore available for a minority of our models, which is why Appendix G documents dataset availability instead. Model Params. Licence Reported cutoff # Langs HF SHA Ref. ibm-granite/granite-4.0-h-350m 350M Apache 2.0 — 12 3b17b717b8f2f5d305b0a92c1491e239aeda19c8 (Granite Team, IBM, 2024) ibm-granite/granite-4.0-350m 350M Apache 2.0 — 12 bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2 (Granite Team, IBM, 2024) Qwen/Qwen3-0.6B 0.6B Apache 2.0 — 100+ c1899de289a04d12100db370d81485cdf75e47ca (Yang et al., 2025) Qwen/Qwen3.5-0.8B 0.8B Apache 2.0 — 201 2fc06364715b967f1860aea9cf38778875588b17 (Yang et al., 2025) google/gemma-3-1b-it 1B gemma — 140+ dcc83ea841ab6100d6b47a070329e1ba4cf78752 (Team et al., 2025a) ibm-granite/granite-4.0-h-1b 1B Apache 2.0 — 12 d18cca4c121edb87d022116d281ce212c9136f57 (Granite Team, IBM, 2024) ibm-granite/granite-4.0-1b 1B Apache 2.0 — 12 6a7381ba1f54d684f508d991aeb7dc580157103 (Granite Team, IBM, 2024) Qwen/Qwen3-1.7B 1.7B Apache 2.0 — 100+ 70d244c86ccca08cf5af4e1e306ecf908b1ad5e (Yang et al., 2025) utter-project/EuroLLM-1.7B-Instruct 1.7B Apache 2.0 — 35 a25c7fa65fc2a644e6270b8940dbe295b51da681 (Martins et al., 2024) ibm-granite/granite-3.2-2b-instruct 2B Apache 2.0 — 12 641593c3b25bec0b1efe9f0f7d7a67f7243f86a3 (Granite Team, IBM, 2024) Qwen/Qwen3.5-2B 2B Apache 2.0 — 201 15852e8c16360a2fea060d615a32b45270f8a8fc (Yang et al., 2025) meta-llama/Llama-3.2-3B-Instruct 3B llama3.2 — 8 0cb88a4f764b7a12671c53f0838cd831a0843b95 (Grattafiori et al., 2024) CohereLabs/tiny-aya-global 3.35B C-BY-NC-4.0 — 67 b2d281bab0beaecb3418d0c0dc53eaba90574011 (Dang et al., 2024) CohereLabs/tiny-aya-water 3.35B C-BY-NC-4.0 — 67 3696a27d9f455538e50887778122c898b6370b8 (Dang et al., 2024) openGPT-X/Teuken-7B-instruct-v0.6 7B C-BY-NC-4.0 09/2023 24 eb3cb8fdd396037c7029e3b2087f54c415086ac (Ali et al., 2024) slovak-nlp/mistral-sk-7b 7B Apache 2.0 — 1 089497ae0d72018e9591895c54f774bf0b4a83a4 — meta-llama/Llama-3.1-8B-Instruct 8B llama3.1 12/2023 8 0e9e39f249a16976918f6564b8830bc894c89659 (Grattafiori et al., 2024) mistralai/Ministral-3-8B-Instruct-2512 8B Apache 2.0 — 11 dcfc440ed367bc62945a537e9e31473406535292 (Jiang et al., 2023) mistralai/Ministral-3-8B-Reasoning-2512 8B Apache 2.0 — 11 6c2c4c82a395598a424f3a17555ae0c19e3f5b (Jiang et al., 2023) CohereLabs/aya-expanse-8b 8B C-BY-NC-4.0 — 23 5062468bf9bc0c6035fd64e06274333ec127d980 (Dang et al., 2024) utter-project/EuroLLM-9B-Instruct-2512 9B Apache 2.0 — 35 1acaaeb8d9e8d9b5e3bf0240aacd32fc11e4213 (Martins et al., 2024) Qwen/Qwen3.5-9B 9B Apache 2.0 12/2025 201 c202236235762e1c871ad0ccb60c8e5ba337b9a (Yang et al., 2025) speakleash/Bielik-11B-v3.0-Instruct 11B Apache 2.0 — 33 e0e273ce65377e91790567115d9b841911f3864 (Ociepa et al., 2025) mistralai/Ministral-3-14B-Instruct-2512 14B Apache 2.0 — 11 1e4bed9a74c1d8af713d9e2545d69020ced05dc (Jiang et al., 2023) mistralai/Ministral-3-14B-Reasoning-2512 14B Apache 2.0 — 11 eaabe4241136f11758e111c5b6b8bae5dde0b031 (Jiang et al., 2023) Qwen/Qwen3-14B 14B Apache 2.0 — 100+ 40c069824f4251a91eefaf281ebe4c544efd3e18 (Yang et al., 2025) slovak-nlp/Qwen3-14B-sk 14B Apache 2.0 — 1 61bc40572b2fa4720e75a7a9de457151aefce396 — openai/gpt-oss-20b 20B Apache 2.0 06/2024 N/A 6cee5e81e83917806bbde320786a8fb61efebee (OpenAI et al., 2025) utter-project/EuroLLM-22B-Instruct-2512 22B Apache 2.0 — 35 38196362cf70aebbc19d1acb9ebc7c18ef81d1b2 (Martins et al., 2024) Qwen/Qwen3.5-27B 27B Apache 2.0 12/2025 201 fc05daec18b0a78c049392ed2e771dde82bdf654 (Yang et al., 2025) Qwen/Qwen3.6-27B 27B Apache 2.0 — 201 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9 (Yang et al., 2025) TildeAI/TildeOpen-30b 30B C-BY-4.0 — 32 faea2654cb208f13c0f84702b82c3113400bdd4a (Bergmanis et al., 2026) TildeAI/TildeOpen-30b-64k 30B C-BY-4.0 — 32 e4c6ad4d87756665566e8e3fd9a23d3a413f2489 (Bergmanis et al., 2026) google/gemma-4-31B-it 31B Apache 2.0 01/2025 35+ fcf2302760ae9c6e528a8dbba9d636e56848237 (Team et al., 2025a) CohereLabs/aya-expanse-32b 32B C-BY-NC-4.0 — 23 b306ea27e360683b50c005d7fcbad6a242317910 (Dang et al., 2024) ibm-granite/granite-4.0-h-small 32B Apache 2.0 — 12 b8c0982bab7fde4eb48110f5a069527c008fab39 (Granite Team, IBM, 2024) Qwen/Qwen3.6-35B-A3B 35B Apache 2.0 — 201 995ad96eacd98c81ed38be0c5b274b04031597b0 (Yang et al., 2025) meta-llama/Llama-3.3-70B-Instruct 70B llama3.3 12/2023 8 6f6073b423013f6a7d4d9f39144961bfbfbc386b (Grattafiori et al., 2024) swiss-ai/Apertus-70B-Instruct-2509 70B Apache 2.0 — 1000 ec382cbace709945d642a54f4448411df05f793b (Apertus et al., 2025) CohereLabs/c4ai-command-a-03-2025 111B C-BY-NC-4.0 — 23 066daaa11d57404b07c650d1068936d31431979 (Cohere et al., 2025) openai/gpt-oss-120b 120B Apache 2.0 06/2024 N/A b5c939de8f754692c1647ca79fbf85e8c1e70f8a (OpenAI et al., 2025) Qwen/Qwen3.5-397B-A17B 397B Apache 2.0 12/2025 201 8472618112abcbd45acbcdc58436aff4233c23f7 (Yang et al., 2025) mistralai/Mistral-Large-3-675B-Instruct-2512 675B Apache 2.0 — 11 5b3d32a0a147594527f38cb136a8390eaa82222 (Jiang et al., 2023) MiniMaxAI/MiniMax-M2.7 230B other — N/A d494266a4affc0d2995ba1fa35c8481cbd84294b (MiniMax et al., 2025) zai-org/GLM-5.1-FP8 754B MIT — 2 f396cf805182f4ca10fa675e1a99815b3ca384db (Team et al., 2025b) moonshotai/Kimi-K2.6 1.1T other — N/A 7eb5002f6aadc958aed6a9177b7ed26b94011b (Team et al., 2025c) Fine-Tuned Models TildeOpen-SK 30B (Alpaca) 30B C-BY-4.0 — 2 N/A — Qwen3-SK 14B (Alpaca) 14B Apache 2.0 — 2 N/A — Mistral-SK 7B (Alpaca) 7B Apache 2.0 — 2 N/A — Table 12: Summary of benchmarked models. The Ref. column cites each model’s family technical report. “—” indicates models for which no technical report is available, including our own Slovak-adapted/fine-tuned models as well as community Slovak models sourced from Hugging Face. Appendix G Data Availability and Contamination As noted in the Limitations, leakage can happen at three levels: benchmark-format leakage (the exact prompt and gold answer in sk-bench form), answer-key leakage (the question–answer pairs of the underlying item bank), and source-document leakage (the raw text behind the task). The latter two correspond to annotation and raw text contamination in the sense of Sainz et al. (2024), while benchmark-format leakage is specific to newly packaged suites. No contamination measurement was performed for this paper, and none of the numbers we report should be read as contamination-free. Instead, we document availability: when the source material was published, whether an evaluation split over it had been released before, and whether the underlying source is public. Comparing dataset dates with model cutoffs is less useful: the model catalogue (Table 12) records checkpoint revisions and reported cutoffs, but most models have no published cutoff, so the comparison is possible for only a minority of the 55 models. Group (examples) Source published Split previously released? Underlying source public? Newly packaged here (Chiby, Slohry, OneRuler-SK, Scio, Autoškola, e-testy-sk, Slovak-tests, Word Analogies, Slovak Trivia) 2024–2026 No — evaluation split first assembled for sk-bench Often yes (public exams, Wikipedia, dictionary) Previously released native (NLI-SK-annotated, SK-QuAD, skLEP, SlovakSum, SME-Sum) 2020–2026 Yes Yes Translated proxies (Okapi-MMLU-SK, ARC-SK, TruthfulQA-SK, Belebele-SK, SlovakCOPA) English 2018–2022, Slovak 2023–2024 Yes Yes (English originals ubiquitous) LLM-generated (MultiWikiQA-SK) 2025, over a 2023 Wikipedia dump Yes Yes (Wikipedia) Table 13: Data availability by group. “Source published” is when the underlying material became available, not when sk-bench packaged it. “Split previously released” asks whether an evaluation split over that material was already circulating before this work. No contamination measurement was performed. The table documents exposure opportunity, not exposure. We draw only a narrow conclusion from Table 13. The newly packaged tasks are the only ones whose exact split was not previously circulating, and even for those the source material may still have appeared in training. The previously released native group and the translated proxies warrant the most caution, particularly for closed models whose training data is undisclosed and whose knowledge cutoff is not published. Readers who want to compare a specific model against a specific cutoff should use the released per-task breakdowns (Appendix H) to restrict the leaderboard to the first row of the table, which is the subset for which benchmark-format leakage can be excluded. Appendix H Per-Task Detailed Results This appendix gives the complete results behind the main leaderboard (Table 1, which shows the top two open models per size tier and top three proprietary APIs): Table 14 is the full reporting-group leaderboard for all 55 models, and the remaining tables give the per-task scores. All cells are the task’s primary metric (×100× 100) with Slovak prompts, 0-shot. Best and second-best per column are marked. Table 15 gives every model on every task and Table 16 repeats it for thinking-mode models. The per-reporting-group averages used in the body leaderboard (Table 1) are computed directly from these per-task scores. Figure 7: Open-weights size–score Pareto frontier, with every frontier model labelled. Overall sk-bench score against parameter count (log scale). The blue line is the frontier, the best score reachable at or below each size. Gains are steep up to ∼ 9B, then the frontier flattens: past gpt-oss 20B and Qwen3.5 27B no open model beats gpt-oss 120B (60.0), and the 675B–1.1T MoEs sit below it. The dashed line marks the best closed-weights model (Gemini 3.1 Pro, 72.6). Table 14: Full sk-bench leaderboard: all 55 models by reporting group, grouped by size tier (Slovak, 0-shot). Each cell is the unweighted mean of that group’s per-task primary metrics (×100× 100), with best and second-best per column. Overall is the unweighted mean of the six group averages. The small grey ± value is the standard deviation across the three prompt templates. The body leaderboard (Table 1) shows the top two open models per size tier and top three proprietary APIs. Per-task scores are in Table 15. Model Knowledge & reasoning Reading & QA Classification Generation Instruction following Grammar & morphology Overall <<2B Granite 4.0 350M 27.5 ± 3.1 10.1 ± 0.6 13.8 ± 10.3 9.7 ± 3.4 10.8 1.8 ± 1.3 12.3 ± 1.9 Granite 4.0-H 350M 25.5 ± 5.7 9.4 ± 0.9 27.3 ± 10.1 11.7 ± 2.0 15.3 1.3 ± 0.0 15.1 ± 3.2 Qwen3 0.6B 33.1 ± 5.2 20.9 ± 3.5 43.3 ± 7.5 10.6 ± 0.2 16.8 4.2 ± 1.3 21.5 ± 1.6 Qwen3.5 0.8B 33.4 ± 2.8 26.0 ± 0.3 53.5 ± 7.8 9.7 ± 0.2 14.3 7.0 ± 2.5 24.0 ± 1.3 Gemma 3 1B 35.1 ± 2.0 19.9 ± 3.0 54.4 ± 4.1 11.8 ± 1.0 12.3 3.0 ± 0.8 22.7 ± 0.6 Granite 4.0 1B 34.5 ± 0.3 27.6 ± 1.0 41.8 ± 10.9 13.2 ± 0.2 17.1 5.2 ± 2.4 23.2 ± 1.9 Granite 4.0-H 1B 39.8 ± 0.6 28.2 ± 0.6 50.1 ± 7.8 14.8 ± 0.9 14.9 6.0 ± 0.8 25.6 ± 1.8 EuroLLM 1.7B 35.2 ± 1.9 19.2 ± 0.6 40.3 ± 6.9 13.4 ± 0.2 14.7 8.4 ± 1.4 21.9 ± 1.4 Qwen3 1.7B 35.0 ± 5.5 30.9 ± 2.8 52.4 ± 7.8 12.8 ± 0.9 23.5 12.3 ± 1.9 27.8 ± 2.4 2–9B Granite 3.2 2B 38.2 ± 0.2 22.5 ± 2.1 51.7 ± 2.5 8.7 ± 0.1 13.8 11.6 ± 0.9 24.4 ± 0.2 Qwen3.5 2B 43.4 ± 1.9 34.2 ± 4.2 56.9 ± 6.9 11.2 ± 0.8 18.8 12.2 ± 1.0 29.4 ± 1.6 Llama 3.2 3B 34.7 ± 0.8 25.8 ± 2.7 46.2 ± 2.4 14.8 ± 1.1 14.0 12.5 ± 2.4 24.6 ± 0.7 Tiny-Aya Global 50.7 ± 1.1 34.6 ± 5.5 73.9 ± 2.3 10.6 ± 0.3 26.8 19.6 ± 0.4 36.0 ± 1.6 Tiny-Aya Water 50.5 ± 1.1 34.9 ± 5.5 74.3 ± 1.7 10.8 ± 0.1 27.6 20.4 ± 1.0 36.4 ± 1.5 Mistral-SK 7B 22.6 ± 4.1 19.3 ± 1.0 34.6 ± 11.4 16.2 ± 0.8 14.2 6.9 ± 2.3 19.0 ± 2.0 Mistral-SK 7B (Alpaca) 21.5 ± 1.6 12.6 ± 1.3 17.7 ± 7.6 12.0 ± 0.8 13.8 11.8 ± 2.6 14.9 ± 1.2 Teuken 7B 39.8 ± 2.4 26.1 ± 1.6 59.1 ± 3.7 10.3 ± 0.3 15.3 15.4 ± 2.9 27.7 ± 2.0 Aya Expanse 8B 44.5 ± 3.1 25.8 ± 4.9 59.0 ± 8.7 12.3 ± 0.8 15.1 14.8 ± 2.2 28.6 ± 3.0 Llama 3.1 8B 47.8 ± 2.0 41.4 ± 4.3 60.9 ± 4.9 15.3 ± 0.6 19.6 25.8 ± 0.6 35.1 ± 2.2 Ministral 3 8B 60.4 ± 1.3 39.2 ± 6.2 70.6 ± 4.4 13.5 ± 0.8 24.0 37.1 ± 1.0 40.8 ± 2.3 Ministral 3 8B (R) 54.9 ± 1.9 37.8 ± 4.0 61.9 ± 7.2 13.1 ± 1.1 39.9 32.1 ± 2.8 39.9 ± 2.4 9–32B EuroLLM 9B 55.8 ± 1.3 42.7 ± 8.4 74.6 ± 1.4 14.9 ± 0.7 22.7 45.2 ± 0.4 42.7 ± 2.0 Qwen3.5 9B 69.7 ± 0.7 50.4 ± 5.2 77.0 ± 2.1 15.0 ± 0.2 31.1 44.4 ± 2.4 47.9 ± 1.4 Bielik 11B 66.5 ± 1.0 47.1 ± 3.9 74.6 ± 7.4 12.6 ± 0.6 30.4 46.9 ± 0.5 46.3 ± 2.4 Ministral 3 14B 63.8 ± 0.9 34.6 ± 4.8 55.9 ± 17.1 14.2 ± 1.3 18.4 36.5 ± 0.8 37.2 ± 3.1 Ministral 3 14B (R) 29.8 ± 13.4 16.2 ± 6.4 16.8 ± 19.7 0.2 ± 0.2 34.3 25.2 ± 2.5 20.4 ± 2.4 Qwen3 14B 67.1 ± 1.1 49.1 ± 6.5 74.3 ± 8.7 16.2 ± 0.4 34.1 44.0 ± 1.3 47.4 ± 2.4 Qwen3-SK 14B 43.4 ± 21.7 23.9 ± 8.4 43.7 ± 17.8 11.6 ± 0.6 27.7 50.8 ± 5.1 33.5 ± 10.2 Qwen3-SK 14B (Alpaca) 70.0 ± 0.7 45.6 ± 3.5 60.2 ± 8.3 15.0 ± 0.5 41.0 30.9 ± 1.1 43.8 ± 2.5 gpt-oss 20B 67.4 ± 0.5 47.6 ± 4.8 76.9 ± 1.2 13.0 ± 0.5 62.4 57.2 ± 0.8 54.1 ± 0.9 EuroLLM 22B 63.9 ± 1.6 41.6 ± 6.5 75.0 ± 5.7 15.2 ± 0.7 26.4 49.2 ± 3.1 45.2 ± 2.0 Qwen3.5 27B 81.2 ± 0.2 55.7 ± 6.3 79.5 ± 3.4 14.9 ± 0.4 36.1 64.6 ± 0.8 55.3 ± 1.8 Qwen3.6 27B 79.8 ± 0.5 59.7 ± 3.7 70.7 ± 10.7 14.9 ± 0.4 35.0 65.0 ± 1.8 54.2 ± 2.8 TildeOpen 30B 45.1 ± 2.6 21.5 ± 1.6 8.8 ± 0.5 3.4 ± 2.1 16.8 6.3 ± 4.3 17.0 ± 0.5 TildeOpen 30B-64k 58.0 ± 0.6 40.4 ± 3.3 62.8 ± 8.9 14.5 ± 1.9 15.6 0.2 ± 0.4 31.9 ± 2.3 TildeOpen-SK 30B (Alpaca) 57.4 ± 2.1 34.2 ± 3.2 33.6 ± 2.7 13.3 ± 0.9 27.2 30.9 ± 1.9 32.8 ± 0.8 Gemma 4 31B 78.8 ± 0.7 53.1 ± 8.7 82.4 ± 0.7 14.4 ± 0.5 30.2 66.7 ± 2.1 54.3 ± 1.3 Aya Expanse 32B 61.5 ± 0.9 41.9 ± 8.5 76.9 ± 2.5 14.2 ± 0.2 24.2 35.7 ± 1.6 42.4 ± 2.4 Granite 4.0-H Small 67.3 ± 0.3 42.4 ± 6.3 71.9 ± 6.2 16.4 ± 0.3 33.9 50.0 ± 1.3 47.0 ± 2.4 32–200B Qwen3.6 35B-A3B 77.2 ± 0.4 55.4 ± 7.0 71.8 ± 9.9 14.5 ± 0.5 32.8 61.4 ± 2.5 52.2 ± 2.8 Apertus 70B 57.5 ± 1.2 43.3 ± 6.1 69.7 ± 3.8 13.2 ± 0.5 27.9 54.8 ± 3.6 44.4 ± 1.6 Llama 3.3 70B 72.1 ± 0.5 53.9 ± 4.0 79.4 ± 0.2 21.4 ± 0.5 35.4 56.4 ± 1.0 53.1 ± 1.0 Command A 64.9 ± 3.0 32.4 ± 3.7 38.4 ± 5.2 12.4 ± 0.5 26.4 42.8 ± 6.5 36.2 ± 1.9 gpt-oss 120B 73.7 ± 1.0 53.2 ± 6.3 79.0 ± 0.5 13.5 ± 0.4 74.9 65.3 ± 1.0 60.0 ± 1.1 >>200B MiniMax M2.7 78.0 ± 1.1 51.5 ± 4.6 78.4 ± 1.1 13.5 ± 0.1 44.7 76.6 ± 0.3 57.1 ± 1.2 Qwen3.5 397B-A17B 83.4 ± 0.5 55.5 ± 8.2 80.6 ± 0.8 14.6 ± 0.2 37.8 72.3 ± 3.0 57.4 ± 1.7 Mistral Large 3 675B 77.3 ± 0.2 49.3 ± 10.6 65.3 ± 5.6 11.8 ± 0.8 20.9 45.3 ± 5.7 45.0 ± 0.9 GLM-5.1 80.6 ± 0.4 61.2 ± 6.9 82.6 ± 1.0 14.2 ± 0.2 35.9 70.8 ± 1.3 57.6 ± 1.2 Kimi K2.6 80.6 ± 0.3 52.7 ± 7.9 78.0 ± 7.9 14.7 ± 0.4 31.8 76.2 ± 0.3 55.7 ± 3.0 Proprietary (size undisclosed) Gemini 3.1 Pro 92.4 ± 0.2 71.3 ± 4.5 82.6 ± 0.4 14.8 ± 0.0 87.2 87.6 ± 0.4 72.6 ± 0.9 GPT-5.5 89.2 ± 0.3 67.3 ± 6.8 81.6 ± 0.3 15.5 ± 0.3 86.4 86.1 ± 0.3 71.0 ± 1.2 Claude Opus 4.6 87.2 ± 0.5 58.6 ± 10.2 83.4 ± 0.4 15.4 ± 0.2 80.6 83.6 ± 0.5 68.1 ± 2.0 Gemini 3.1 Flash-Lite 85.7 ± 0.6 54.8 ± 11.2 82.2 ± 0.7 14.0 ± 0.3 82.7 80.8 ± 0.5 66.7 ± 2.4 Claude Sonnet 4.6 84.7 ± 0.3 54.0 ± 9.6 81.6 ± 1.9 15.3 ± 0.3 78.0 75.8 ± 0.6 64.9 ± 1.9 GPT-5.4 mini 78.2 ± 0.5 58.0 ± 9.1 79.0 ± 1.1 15.2 ± 0.4 80.1 73.0 ± 2.2 63.9 ± 1.5 Model ARC MMLU G-MMLU TruthQA SCIO FinExam Driving COPA SKJ1 Trivia Belebele Chiby WikiQA ETest-M ETest-Q SKTest-M SKTest-Q OneRuler skLEP-QA NLI skLEP-NLI skLEP-RTE Senti-3 Senti-2 skLEP-Sent MovieRev Toxic skLEP-Hate SlovakSum SME-Sum IFEval Analogies Slohry Avg <<2B Granite 4.0 350M 26.5 27.7 26.7 21.9 25.9 25.9 33.2 48.7 12.7 26.1 30.0 0.9 3.6 19.4 0.2 25.4 1.3 7.2 3.0 7.3 7.1 5.2 28.5 22.0 23.8 25.6 3.1 1.4 9.8 9.6 10.8 0.3 3.2 15.9 Granite 4.0-H 350M 24.1 26.6 25.6 13.8 21.8 22.3 32.2 50.5 15.3 22.4 26.7 2.7 1.7 20.0 0.2 26.9 0.8 0.5 4.9 4.0 3.9 14.9 43.3 67.4 10.3 52.5 41.2 8.2 10.8 12.6 15.3 0.0 2.6 19.0 Qwen3 0.6B 25.9 25.9 26.2 65.3 21.4 31.0 61.1 49.4 3.3 21.1 34.2 26.8 1.2 20.9 0.5 29.0 1.5 56.0 17.6 23.4 21.2 36.1 28.3 75.8 33.8 58.6 56.3 56.6 11.6 9.7 16.8 2.6 5.9 28.9 Qwen3.5 0.8B 41.1 33.1 35.8 28.5 28.7 34.5 54.7 50.1 2.7 24.6 47.1 43.9 14.9 25.4 0.9 36.8 1.3 50.5 13.4 24.7 22.8 56.7 67.5 85.2 57.6 69.0 57.7 40.2 10.9 8.5 14.3 7.3 6.7 33.2 Gemma 3 1B 27.3 28.4 32.8 52.8 23.5 35.0 70.0 51.8 2.7 26.9 34.0 34.7 13.7 21.7 2.6 35.9 2.1 13.9 20.1 22.2 24.2 51.7 65.2 81.5 82.0 73.1 56.1 33.3 12.0 11.7 12.3 2.0 4.0 32.2 Granite 4.0 1B 41.5 38.2 39.4 19.2 35.2 36.1 47.3 51.8 2.3 33.6 61.4 51.0 8.5 28.5 1.6 40.2 2.1 41.2 13.9 34.2 34.9 60.8 30.8 59.2 30.9 47.9 27.4 50.6 11.9 14.5 17.1 4.0 6.5 31.0 Granite 4.0-H 1B 49.1 40.6 44.8 18.6 35.4 42.3 43.1 51.1 41.3 31.6 54.6 49.8 6.1 32.1 1.3 40.9 2.7 49.4 17.1 27.1 24.7 63.5 38.4 55.5 90.3 70.4 40.7 40.0 13.3 16.3 14.9 3.4 8.6 35.1 EuroLLM 1.7B 25.9 24.3 24.4 45.6 22.4 18.8 57.1 50.9 57.3 24.8 27.9 50.0 17.3 19.0 5.5 27.5 3.8 5.6 16.2 14.7 17.8 50.2 19.3 24.4 74.8 51.1 46.5 63.9 13.1 13.8 14.7 6.9 9.9 28.7 Qwen3 1.7B 40.2 31.1 36.1 37.7 19.6 40.0 62.0 51.1 14.0 18.2 52.5 50.4 15.2 22.8 2.7 34.8 2.4 80.2 17.2 14.9 15.7 47.0 55.3 70.6 71.5 70.6 61.3 65.0 12.7 13.0 23.5 6.9 17.8 35.6 2–9B Granite 3.2 2B 41.1 38.2 39.7 14.4 30.1 47.5 49.4 55.7 38.3 28.0 55.1 24.2 3.9 29.1 1.4 39.7 1.5 33.2 14.1 11.0 11.8 44.7 57.6 76.4 93.3 85.1 41.4 44.1 9.5 7.8 13.8 9.5 13.7 33.5 Qwen3.5 2B 60.6 44.5 51.6 22.9 41.1 47.6 53.1 55.6 20.7 36.2 71.1 56.9 21.1 33.9 4.4 44.8 3.9 47.8 23.5 45.4 46.6 72.5 56.3 75.2 44.8 54.2 63.9 53.3 11.8 10.5 18.8 7.1 17.2 40.0 Llama 3.2 3B 40.1 38.1 43.2 11.0 33.9 45.6 37.9 49.1 15.7 32.9 52.4 18.3 23.6 29.5 9.3 40.5 3.2 34.1 21.2 23.2 23.0 49.7 64.7 85.7 46.1 69.3 32.1 21.7 13.9 15.6 14.0 12.0 12.9 32.2 Tiny-Aya Global 64.9 49.0 54.2 46.2 40.9 58.5 67.8 63.7 20.3 41.4 78.4 50.6 12.4 37.0 8.8 53.2 4.9 54.9 11.6 61.5 61.2 74.7 55.6 83.4 91.7 83.4 77.3 76.0 11.4 9.9 26.8 11.5 27.7 47.6 Tiny-Aya Water 66.1 49.6 54.6 43.4 41.3 59.7 68.0 62.6 17.7 41.9 79.0 50.7 11.7 35.9 9.5 53.8 4.3 57.4 11.4 63.2 63.6 74.6 55.4 83.2 92.7 83.0 75.4 77.5 11.5 10.1 27.6 13.8 26.9 47.8 Mistral-SK 7B 23.8 22.9 26.6 25.3 20.6 24.0 31.1 34.6 3.0 14.4 32.1 43.3 19.7 17.2 17.1 24.8 3.8 0.0 15.7 22.6 20.4 46.2 25.0 23.4 50.8 35.1 50.7 37.5 13.5 18.8 14.2 13.8 0.0 23.4 Mistral-SK 7B (Alpaca) 14.3 12.3 16.2 30.2 10.8 15.6 46.9 57.3 4.7 7.0 22.7 5.5 13.3 11.0 0.7 17.0 2.0 35.6 5.8 18.8 20.2 30.0 11.1 15.2 12.2 22.6 8.5 21.3 12.8 11.1 13.8 13.8 9.8 16.7 Teuken 7B 49.9 34.9 39.7 32.6 23.4 54.4 66.0 57.5 6.7 33.5 66.6 53.8 4.2 26.9 4.3 39.7 4.2 17.9 16.8 22.4 20.3 60.2 59.7 79.8 90.2 80.6 46.7 71.9 10.6 10.0 15.3 9.6 21.1 37.3 Aya Expanse 8B 61.4 46.5 51.2 36.5 39.3 54.6 58.0 33.5 29.0 35.2 78.7 15.5 16.8 32.7 6.5 52.6 3.6 13.2 12.5 36.5 38.7 59.2 47.8 65.9 88.3 87.7 48.5 57.9 12.2 12.5 15.1 14.8 14.7 38.7 Llama 3.1 8B 56.5 47.9 53.9 36.4 41.2 58.3 53.3 58.1 36.3 35.9 74.1 50.1 35.1 40.5 13.8 49.7 8.5 68.3 32.6 34.6 32.2 57.5 54.7 75.9 90.8 86.7 53.0 63.1 14.7 16.0 19.6 24.5 27.1 45.5 Ministral 3 8B 78.6 59.7 64.9 45.4 53.4 68.7 71.5 67.3 47.0 47.0 85.2 54.2 9.8 50.2 12.7 61.2 6.2 58.0 15.4 54.7 50.0 74.7 51.6 72.4 96.2 89.5 81.5 65.0 13.1 14.0 24.0 32.3 41.9 52.0 Ministral 3 8B (R) 70.6 52.8 57.9 45.2 43.7 59.8 63.8 51.1 67.0 36.5 78.7 38.5 10.8 46.1 13.1 54.5 5.6 74.1 19.2 46.8 46.0 65.4 47.4 67.4 87.8 77.9 46.9 71.7 12.9 13.2 39.9 28.0 36.2 47.8 9–32B EuroLLM 9B 75.7 54.5 62.5 40.2 45.4 63.2 69.1 70.9 30.3 46.1 82.5 58.4 19.8 45.7 16.7 63.0 8.1 63.1 27.3 57.7 57.0 78.3 68.4 88.8 87.1 81.6 81.7 70.6 13.9 16.0 22.7 40.4 50.0 53.2 Qwen3.5 9B 88.7 67.6 72.2 64.8 61.3 73.9 78.3 81.0 61.0 47.9 89.3 61.6 18.4 62.0 20.2 65.8 12.9 83.2 39.8 65.9 68.3 83.5 60.1 82.5 95.4 90.9 69.1 77.1 13.6 16.3 31.1 42.1 46.7 60.4 Bielik 11B 83.7 61.3 68.4 66.7 53.5 73.5 75.4 85.9 45.3 51.7 89.9 59.3 7.0 55.7 25.4 66.2 16.4 86.7 17.3 66.2 68.2 80.4 50.8 72.3 81.0 86.2 90.4 76.3 12.9 12.3 30.4 41.9 51.8 57.9 Ministral 3 14B 82.3 62.5 66.2 51.0 51.0 72.9 73.0 68.7 61.7 48.9 87.3 58.4 5.7 57.9 13.1 63.5 4.8 4.2 16.3 58.3 56.8 77.6 23.8 40.7 49.6 46.3 78.6 71.7 13.4 15.0 18.4 36.7 36.2 47.6 Ministral 3 14B (R) 48.0 28.9 33.4 25.0 22.6 12.6 39.3 35.3 24.7 28.2 52.5 11.7 0.4 35.3 3.2 40.0 0.6 0.0 2.4 15.4 14.6 26.2 11.6 15.0 17.8 8.2 28.8 13.6 0.2 0.2 34.3 11.3 39.2 20.6 Qwen3 14B 85.7 65.5 65.1 63.1 59.4 70.9 79.6 76.5 59.7 45.2 88.7 63.4 23.3 58.9 16.0 62.5 10.1 85.1 33.5 50.0 54.6 83.4 58.1 76.6 94.3 90.0 84.3 77.4 14.2 18.2 34.1 41.4 46.5 58.6 Qwen3-SK 14B 39.1 33.3 35.0 52.3 18.5 69.9 31.3 79.0 55.0 20.8 88.7 0.1 27.5 23.3 4.6 40.6 5.5 1.0 23.7 21.9 29.1 54.1 39.9 54.7 38.5 66.4 25.8 62.7 11.9 11.2 27.7 49.2 52.3 36.2 Qwen3-SK 14B (Alpaca) 86.7 66.2 68.9 70.7 59.3 71.0 81.9 80.5 65.0 50.1 89.7 63.5 10.7 58.7 18.4 67.6 9.9 83.9 7.6 68.0 67.4 82.8 32.6 51.0 15.3 59.0 87.3 78.0 14.5 15.5 41.0 29.3 32.5 54.1 gpt-oss 20B 88.1 72.6 74.8 58.7 67.1 65.5 72.8 83.3 56.7 34.4 89.6 62.6 14.8 61.5 15.6 65.0 9.4 85.1 24.6 71.0 72.1 80.7 56.1 72.5 96.1 87.3 82.1 73.9 12.5 13.5 62.4 42.5 71.9 60.5 EuroLLM 22B 80.6 59.2 68.0 58.2 48.4 69.2 76.9 77.5 47.3 53.4 83.4 53.9 12.2 56.3 16.0 66.5 9.9 54.0 22.0 54.4 50.9 78.8 61.4 85.8 94.6 88.6 86.2 74.3 14.2 16.2 26.4 44.9 53.6 55.9 Qwen3.5 27B 94.5 80.0 84.9 76.8 69.1 83.2 87.8 90.5 82.7 62.1 93.7 69.0 16.6 73.6 27.8 77.4 18.2 83.3 41.3 70.1 68.3 79.9 61.3 80.0 97.3 93.6 86.5 78.9 13.7 16.2 36.1 54.7 74.5 67.4 Qwen3.6 27B 93.7 79.6 83.3 77.8 69.3 83.6 86.9 88.5 76.7 58.1 94.0 67.4 34.4 71.9 39.7 76.1 21.6 83.3 48.8 29.6 27.9 85.9 62.5 83.5 97.1 93.6 81.2 75.1 13.7 16.2 35.0 54.6 75.4 65.6 TildeOpen 30B 57.7 42.1 47.5 45.2 27.4 46.5 74.4 32.1 51.3 26.4 66.0 17.4 10.1 32.6 9.6 51.3 2.4 0.0 4.4 6.6 1.7 13.3 1.4 0.2 0.2 1.1 39.7 14.7 4.2 2.6 16.8 12.6 0.0 23.0 TildeOpen 30B-64k 69.4 51.4 59.9 42.4 41.3 60.6 81.0 58.5 62.7 52.9 74.3 52.4 41.7 41.8 43.5 64.0 22.1 0.0 23.7 46.5 42.8 66.4 60.6 70.4 89.1 59.2 60.8 69.1 13.2 15.8 15.6 0.5 0.0 47.1 TildeOpen-SK 30B (Alpaca) 72.6 50.8 59.1 43.5 44.2 58.5 76.7 71.3 49.0 48.7 77.4 24.8 6.9 43.6 19.4 64.1 9.5 43.5 18.5 38.4 35.4 63.4 8.3 10.5 3.7 20.7 48.7 73.4 13.4 13.2 27.2 20.1 41.7 39.4 Gemma 4 31B 94.7 79.6 86.9 78.7 63.4 84.5 87.9 85.9 66.3 59.7 94.1 66.9 13.1 70.7 27.9 77.7 14.8 83.3 29.6 81.2 82.6 84.5 60.3 75.9 97.1 93.3 88.3 78.1 13.8 15.0 30.2 54.0 79.4 66.7 Aya Expanse 32B 78.0 59.4 63.9 55.3 48.2 68.9 71.0 81.1 43.3 45.7 88.7 57.8 17.8 43.9 13.3 63.3 4.9 65.4 22.3 67.6 69.5 75.1 57.6 77.5 94.7 91.0 87.6 71.8 13.7 14.7 24.2 32.8 38.7 54.8 Granite 4.0-H Small 84.2 65.6 73.8 63.4 56.5 69.3 74.3 83.5 54.3 47.7 86.1 53.5 15.0 57.2 13.8 66.3 8.1 56.9 24.9 46.7 48.8 82.7 55.1 76.4 95.7 89.1 79.7 72.8 14.7 18.1 33.9 44.2 55.9 56.6 32–200B Qwen3.6 35B-A3B 92.3 76.3 81.5 76.0 65.8 80.7 83.5 85.5 76.3 53.7 92.5 69.8 23.7 69.7 31.0 75.6 14.4 83.4 38.9 40.2 36.1 85.5 55.6 77.8 97.3 93.0 84.6 76.4 13.6 15.3 32.8 51.2 71.6 63.7 Apertus 70B 71.3 53.5 62.8 43.2 45.8 67.8 67.1 73.0 43.0 47.6 78.8 50.7 19.2 45.7 17.2 60.2 12.5 80.5 24.8 47.2 42.6 70.4 54.8 74.8 95.9 89.2 79.6 72.9 13.2 13.2 27.9 48.7 60.8 53.2 Llama 3.3 70B 87.5 72.3 80.6 66.0 59.7 76.2 83.8 87.3 57.0 50.2 91.6 62.2 29.1 63.6 33.3 69.4 16.2 81.2 38.1 68.2 72.3 81.2 57.6 82.0 96.9 92.7 85.1 79.0 16.1 26.6 35.4 47.2 65.6 64.0 Command A 88.0 68.2 72.6 65.3 57.2 78.1 79.4 19.7 61.7 59.0 92.3 7.0 4.0 53.2 10.4 67.7 3.6 50.5 2.7 17.3 20.3 59.0 31.0 29.7 36.3 35.9 47.7 68.8 11.6 13.2 26.4 49.6 36.1 43.1 gpt-oss 120B 90.6 77.9 81.0 65.9 71.6 74.4 76.0 83.9 64.7 51.4 92.2 65.0 14.6 71.0 27.9 71.7 17.5 90.9 28.4 72.8 75.8 83.2 57.8 75.6 96.4 89.8 84.8 75.3 12.9 14.2 74.9 50.5 80.2 65.5 >>200B MiniMax M2.7 93.6 82.9 86.9 73.1 71.7 81.6 83.5 83.1 63.3 59.9 93.0 59.5 21.4 73.1 42.4 76.0 22.8 34.7 40.6 72.6 75.4 81.7 57.2 71.7 96.8 92.4 83.5 74.4 13.0 14.1 44.7 63.0 90.2 65.9 Qwen3.5 397B-A17B 94.9 82.8 90.1 80.8 73.6 87.8 86.3 90.1 79.0 68.7 94.6 72.2 9.9 78.0 33.9 80.5 19.9 83.4 27.2 78.6 77.2 82.5 61.7 81.3 96.3 94.0 78.2 75.2 13.5 15.8 37.8 61.7 82.8 68.8 Mistral Large 3 675B 90.8 77.0 83.7 72.7 65.4 82.2 83.9 89.6 64.3 63.2 93.0 61.8 16.7 62.0 24.8 76.8 11.1 73.2 24.1 71.9 68.7 83.0 28.8 35.8 85.3 47.4 88.2 78.2 11.2 12.3 20.9 58.0 32.6 58.7 GLM-5.1 92.3 78.7 86.2 77.3 68.6 84.8 87.7 89.8 76.3 63.9 94.0 66.5 28.0 74.3 40.8 78.7 22.0 99.3 46.7 76.9 78.1 85.0 61.9 85.9 96.6 92.5 87.9 78.6 13.5 15.0 35.9 59.7 82.0 69.9 Kimi K2.6 93.1 78.7 84.8 78.6 64.7 87.0 85.7 90.2 81.0 62.4 94.0 65.2 23.4 73.4 38.1 77.6 18.3 45.4 39.2 60.2 57.3 84.1 60.8 82.8 97.0 93.1 88.6 77.9 13.8 15.7 31.8 62.6 89.8 66.6 Proprietary (size undisclosed) Gemini 3.1 Pro 97.0 91.5 94.0 94.4 82.0 96.3 98.3 91.5 84.3 94.3 95.8 74.4 31.8 88.7 73.7 89.6 37.5 100.0 49.9 82.5 85.2 85.8 62.2 77.3 97.8 95.7 83.6 73.8 13.9 15.7 87.2 76.8 98.5 78.8 GPT-5.5 96.5 91.2 92.9 84.8 81.1 96.0 96.1 77.8 83.3 92.4 95.0 74.5 18.3 87.9 67.6 88.8 35.6 99.9 38.2 79.7 81.3 83.7 61.3 76.0 98.3 95.2 85.1 74.0 14.1 16.9 86.4 74.1 98.0 76.4 Claude Opus 4.6 94.8 88.2 91.5 91.4 74.3 93.2 91.6 90.8 80.3 76.2 94.8 73.2 16.8 79.0 33.9 83.9 14.8 99.8 31.2 83.2 85.3 80.9 59.9 79.7 97.4 94.8 90.3 78.8 14.6 16.1 80.6 71.4 95.7 73.6 Gemini 3.1 Flash-Lite 94.8 84.2 89.9 81.2 71.1 89.7 91.5 90.7 84.7 79.7 93.9 75.0 11.1 80.1 17.8 85.5 9.9 99.8 20.4 81.2 81.7 82.8 61.3 78.5 97.4 94.1 85.9 76.5 13.5 14.5 82.7 67.9 93.7 71.6 Claude Sonnet 4.6 94.0 85.3 89.6 80.6 73.2 89.6 88.5 90.9 82.0 73.4 93.3 68.8 14.7 59.9 32.6 81.9 12.3 100.0 22.3 80.6 83.0 80.1 60.3 81.3 95.4 91.2 85.1 77.7 14.4 16.1 78.0 63.1 88.5 70.5 GPT-5.4 mini 91.3 74.7 82.2 70.6 64.0 84.3 87.0 90.6 78.0 59.3 92.0 71.0 19.3 72.5 32.4 80.4 17.9 99.9 37.0 68.2 67.5 81.4 58.8 82.5 96.7 91.5 86.4 77.7 13.9 16.5 80.1 60.4 85.5 68.8 Table 15: sk-bench results across all tasks. Slovak (SK), 0-shot. Each cell is the task’s primary metric (×100× 100): accuracy for multiple-choice and classification, quasi-exact-match (qem) for open QA, ROUGE-L for summarisation, and prompt-level strict accuracy for IFEval. Best and second-best per column. Avg is the unweighted mean of the columns shown. Model ARC MMLU G-MMLU TruthQA SCIO FinExam Driving COPA SKJ1 Trivia Belebele Chiby WikiQA ETest-M ETest-Q SKTest-M SKTest-Q OneRuler skLEP-QA NLI skLEP-NLI skLEP-RTE Senti-3 Senti-2 skLEP-Sent MovieRev Toxic skLEP-Hate SlovakSum SME-Sum IFEval Analogies Slohry Avg <<2B Qwen3 0.6B 39.1 34.5 33.2 29.3 30.6 40.4 53.7 53.1 7.0 26.7 46.7 46.1 10.3 26.8 2.9 31.1 2.4 49.3 18.7 36.9 38.6 64.9 35.1 47.2 79.5 65.5 54.8 31.8 11.2 9.6 36.3 4.5 6.1 33.5 Qwen3.5 0.8B 7.1 2.8 2.3 0.6 2.5 1.7 3.6 1.9 0.0 0.3 1.2 0.1 0.0 4.1 0.1 2.3 0.8 0.5 0.0 2.5 3.2 0.3 2.9 4.5 15.2 6.1 3.9 2.5 0.3 0.2 24.4 2.3 18.9 3.6 Qwen3 1.7B 67.5 51.6 53.9 36.9 49.5 49.6 59.4 68.8 22.0 31.4 72.0 51.1 25.1 42.4 14.8 43.7 5.6 93.2 23.5 54.8 57.2 76.1 49.3 63.4 92.0 81.1 71.2 68.7 12.0 11.7 47.3 22.2 22.8 48.2 2–9B Qwen3.5 2B 42.9 31.2 25.8 10.9 23.9 24.7 31.9 13.5 1.3 4.9 29.9 18.8 21.3 18.5 1.4 13.5 1.4 26.9 23.5 20.6 18.2 23.8 20.3 26.2 62.6 41.7 26.4 24.3 0.5 0.3 25.9 23.7 25.7 21.4 9–32B Qwen3.5 9B 93.5 80.9 84.6 68.8 74.9 77.1 82.1 82.0 63.7 50.8 92.6 46.0 26.8 69.6 25.0 69.0 11.5 80.4 39.8 71.8 72.0 75.2 53.8 66.4 95.5 90.3 73.0 71.2 10.5 12.4 52.3 59.8 86.5 63.9 Qwen3.5 27B 95.8 86.6 89.4 79.9 77.9 86.1 88.8 84.3 77.0 66.4 94.7 64.7 25.6 75.6 32.9 79.0 19.5 96.0 44.9 80.3 79.8 83.9 60.0 73.9 97.1 94.2 85.5 75.7 12.9 15.0 71.9 63.0 89.5 71.1 Qwen3.6 27B 95.7 86.7 89.2 80.0 78.3 84.1 84.1 83.6 74.7 64.4 95.0 69.2 38.4 76.1 50.3 79.8 25.5 98.4 42.2 78.1 78.7 83.7 60.1 76.9 97.3 93.9 86.1 75.3 13.4 15.7 78.2 61.3 85.9 72.1 Gemma 4 31B 96.1 86.9 90.8 80.1 78.8 87.0 89.9 82.3 74.7 64.1 94.8 66.9 15.9 76.7 30.9 82.1 17.2 100.0 33.5 80.1 82.2 83.7 60.8 74.5 97.1 94.0 87.6 75.4 13.8 15.5 84.0 62.1 87.7 71.1 32–200B Qwen3.6 35B-A3B 93.5 84.8 88.8 78.7 76.2 81.5 84.1 82.3 70.7 62.5 93.9 69.6 31.6 72.7 43.6 78.9 25.6 96.7 39.4 75.2 75.0 84.0 59.7 77.0 97.0 93.1 85.4 75.0 13.1 14.9 76.9 63.6 86.9 70.7 Table 16: sk-bench results for thinking-mode models. Slovak (SK), 0-shot. Each cell is the task’s primary metric (×100× 100): accuracy for multiple-choice and classification, quasi-exact-match (qem) for open QA, ROUGE-L for summarisation, and prompt-level strict accuracy for IFEval. Best and second-best per column. Avg is the unweighted mean of the columns shown. H.1 English-prompt results The released leaderboard uses the Slovak instruction templates, but the harness ships a parallel English template for every task. We re-ran the full sweep with the English prompts. For the tasks that have an English counterpart (IFEval-SK and Slohry are Slovak-only by construction) Table 17 gives the full English per-task scores. The per-reporting-group Slovak-vs-English comparison and its discussion are in the main text (§6.4, Table 5). Model ARC MMLU G-MMLU TruthQA SCIO FinExam Driving COPA SKJ1 Trivia Belebele Chiby WikiQA ETest-M ETest-Q SKTest-M SKTest-Q OneRuler skLEP-QA NLI skLEP-NLI skLEP-RTE Senti-3 Senti-2 skLEP-Sent MovieRev Toxic skLEP-Hate SlovakSum SME-Sum Analogies Avg <<2B Granite 4.0 350M 26.3 27.0 25.4 7.5 27.7 24.7 19.0 50.3 0.0 26.5 32.0 50.3 6.4 20.6 0.6 28.1 1.3 16.8 8.5 34.4 31.2 53.8 46.6 74.8 59.9 70.8 46.7 72.7 8.2 9.0 0.9 29.3 Granite 4.0-H 350M 25.5 27.8 25.9 7.6 24.3 25.3 25.7 50.7 48.3 22.9 28.7 50.2 6.3 22.4 0.6 20.2 0.8 24.7 14.3 25.2 23.7 44.5 62.5 87.4 30.4 65.4 47.3 67.0 11.1 15.1 0.3 30.1 Qwen3 0.6B 29.9 29.5 29.0 30.2 23.8 32.6 65.5 48.5 7.7 24.1 39.6 50.7 8.1 24.7 0.3 30.5 1.1 52.3 18.6 32.3 29.6 52.5 65.9 66.8 46.2 55.7 53.3 28.0 12.4 12.5 2.0 32.4 Qwen3.5 0.8B 40.5 32.8 37.8 48.9 27.6 37.1 60.5 50.6 0.0 26.1 53.5 51.4 19.5 25.8 1.0 37.6 1.7 37.3 15.0 31.6 28.7 60.0 64.8 85.8 57.2 67.2 64.4 67.6 12.4 11.9 1.1 37.3 Gemma 3 1B 32.0 31.3 36.9 39.8 26.7 35.6 65.8 51.5 1.7 28.6 37.1 52.0 14.5 24.1 1.7 35.9 2.2 20.1 15.2 44.4 43.3 65.3 66.7 87.2 60.1 72.5 63.6 46.8 10.1 10.2 1.3 36.3 Granite 4.0 1B 40.9 36.5 39.3 14.2 32.6 38.6 33.4 52.7 10.3 35.1 62.3 52.0 16.9 27.3 1.4 39.1 2.0 65.2 20.3 47.6 45.3 72.9 38.6 73.8 75.8 78.5 51.1 72.7 12.2 14.1 5.6 39.0 Granite 4.0-H 1B 49.5 40.7 43.0 21.0 34.8 42.1 46.2 56.9 40.0 29.2 58.1 51.9 19.0 30.5 1.0 40.9 2.0 56.3 20.0 48.1 43.4 72.0 45.6 52.1 94.2 81.7 53.7 72.7 13.3 16.9 5.2 41.4 EuroLLM 1.7B 27.3 25.4 29.2 16.0 27.1 20.9 29.8 49.8 64.7 25.3 29.3 50.0 15.5 21.1 3.1 26.2 3.8 12.6 14.2 28.6 29.5 49.2 66.2 88.0 43.6 64.7 44.2 67.5 8.4 7.0 3.9 32.0 Qwen3 1.7B 40.6 29.0 32.4 40.1 19.4 34.9 66.4 54.7 16.7 25.6 56.9 51.5 21.8 27.0 2.5 35.9 2.7 79.0 24.8 32.0 28.6 75.3 50.5 70.1 59.4 77.6 63.8 42.8 13.2 14.3 4.8 38.5 2–9B Granite 3.2 2B 40.8 37.6 39.8 18.9 30.1 46.7 45.6 57.4 41.7 27.8 59.5 51.5 16.8 29.8 2.6 38.8 2.1 46.6 23.2 62.7 65.6 76.6 53.6 81.9 87.6 84.9 65.3 74.5 10.0 8.4 9.0 43.1 Qwen3.5 2B 60.3 44.5 52.8 29.3 41.8 45.3 59.5 53.1 11.3 34.7 73.5 56.5 17.5 35.8 4.1 46.8 5.5 38.9 25.6 40.0 37.1 79.1 48.4 63.5 71.2 68.0 65.0 72.5 13.5 16.2 2.5 42.4 Llama 3.2 3B 38.2 37.0 44.2 9.5 33.5 43.5 29.4 53.8 32.0 34.3 53.7 52.1 29.7 28.0 6.1 39.5 3.1 32.4 16.2 28.5 29.1 55.4 45.5 67.8 49.6 55.5 47.6 65.5 14.0 15.5 6.0 35.4 Tiny-Aya Global 64.7 48.8 53.9 45.9 39.8 57.5 67.8 65.3 22.3 41.1 78.4 52.8 15.5 37.3 8.1 52.6 5.2 49.2 14.8 42.7 38.2 58.2 47.7 75.3 86.0 80.5 60.3 75.8 12.9 12.7 10.7 45.9 Tiny-Aya Water 65.2 49.5 53.8 41.8 40.7 57.7 68.0 65.1 19.7 42.1 79.4 52.0 14.7 37.7 7.8 52.2 5.3 60.5 13.9 36.5 33.7 61.4 47.9 74.5 87.6 81.8 53.8 75.5 13.0 13.0 11.6 45.7 Mistral-SK 7B 21.3 18.7 20.0 9.3 16.2 21.1 40.8 25.9 1.7 13.1 22.9 27.0 17.7 12.8 13.0 20.3 4.3 0.0 16.4 7.4 5.9 14.8 17.8 28.5 10.1 36.8 26.1 9.4 8.5 10.6 4.0 16.2 Mistral-SK 7B (Alpaca) 18.9 13.6 18.2 32.6 10.7 12.4 54.1 47.0 4.7 5.8 26.5 0.4 14.2 10.3 0.6 19.2 1.8 40.5 5.8 0.4 0.7 0.0 0.2 0.2 0.0 0.3 0.3 0.1 12.7 10.9 4.9 11.9 Teuken 7B 48.7 35.1 40.8 34.9 22.9 52.9 71.8 53.1 2.0 33.1 66.3 45.6 3.8 30.8 3.7 45.6 3.9 37.9 9.1 24.0 24.1 59.0 40.7 54.5 86.3 84.4 46.9 72.7 10.2 9.3 4.4 37.4 Aya Expanse 8B 62.0 48.0 51.5 38.5 40.7 53.1 58.3 72.3 28.0 34.3 80.6 24.0 42.7 33.6 8.8 52.1 3.9 49.3 24.6 47.4 55.2 75.4 42.0 60.6 82.1 76.9 79.9 77.5 12.5 13.7 10.9 46.5 Llama 3.1 8B 55.5 47.7 55.1 32.5 40.7 55.6 52.7 56.8 39.7 31.3 75.7 42.0 38.0 37.6 9.6 49.5 8.4 77.9 23.8 20.7 18.2 51.1 32.6 49.1 77.2 61.1 63.9 62.9 14.7 15.7 8.1 42.1 Ministral 3 8B 78.6 60.3 66.0 43.5 52.9 59.3 72.8 65.8 48.3 46.1 85.6 54.1 13.7 50.4 12.3 62.1 6.3 35.4 15.9 43.0 42.2 77.4 53.4 75.2 92.0 87.6 61.2 73.6 13.2 14.7 31.5 51.4 Ministral 3 8B (R) 71.5 52.0 59.7 60.2 48.5 39.1 69.9 57.7 28.7 36.5 80.5 52.9 13.1 43.3 11.8 49.7 4.8 90.0 18.4 49.1 49.4 76.3 55.9 80.6 78.9 76.6 49.1 72.7 12.6 13.4 21.2 49.2 9–32B EuroLLM 9B 73.8 52.9 59.6 41.3 43.5 61.2 73.0 68.1 36.3 42.1 79.1 55.5 19.6 45.1 20.5 62.1 12.9 47.5 24.5 54.7 52.5 81.5 60.1 84.7 86.7 82.3 79.2 74.2 13.5 14.9 30.9 52.7 Qwen3.5 9B 88.5 67.6 71.8 60.9 60.7 73.1 78.9 82.4 60.7 47.6 88.9 63.9 22.2 62.5 26.6 66.9 15.3 84.5 38.5 74.5 71.5 85.1 46.6 60.0 89.4 88.8 74.2 76.5 13.5 16.0 39.2 61.2 Bielik 11B 82.2 61.3 69.2 58.4 54.7 72.5 73.9 82.2 52.0 52.9 89.6 58.7 22.8 54.9 30.8 65.4 18.9 86.5 31.8 76.3 78.4 81.9 58.5 76.0 95.9 92.4 84.6 77.4 13.0 12.6 38.0 61.4 Ministral 3 14B 82.1 63.4 67.3 53.3 55.8 70.3 75.9 71.8 67.7 49.1 87.6 52.9 11.8 59.7 14.2 63.8 5.5 0.0 16.7 49.2 45.1 79.8 19.7 31.3 36.0 35.7 78.8 74.0 13.4 15.1 37.6 47.9 Ministral 3 14B (R) 57.2 39.0 43.3 9.6 31.5 42.2 20.6 52.8 37.7 26.9 43.7 22.1 3.2 30.8 8.4 35.7 2.0 0.6 5.3 42.0 41.9 66.8 16.5 10.4 22.1 20.3 57.1 63.3 0.9 0.5 21.5 28.3 Qwen3 14B 83.8 64.7 66.2 58.5 58.7 67.9 79.6 75.7 63.3 46.8 88.7 60.5 28.3 57.8 21.6 62.0 11.8 85.8 37.8 47.2 52.8 83.7 47.4 71.9 89.1 88.7 76.5 76.8 13.9 16.6 37.0 58.7 Qwen3-SK 14B 85.4 64.3 68.2 72.2 55.2 73.6 86.2 69.3 53.7 52.5 71.9 61.7 40.1 61.6 23.4 64.5 12.7 32.4 24.1 56.1 51.1 27.8 26.5 49.1 65.2 63.2 64.7 74.6 12.4 10.9 48.8 52.4 Qwen3-SK 14B (Alpaca) 86.1 65.6 69.5 68.9 59.1 70.1 83.1 83.9 66.7 49.6 89.6 20.6 11.7 58.3 17.9 67.5 11.1 76.1 7.7 46.2 45.7 50.0 12.8 4.3 28.3 9.3 3.7 5.5 14.2 14.4 32.0 42.9 gpt-oss 20B 88.5 73.6 74.8 53.2 67.6 64.5 73.4 83.3 59.0 34.6 90.0 65.1 28.2 62.3 24.2 65.6 12.9 85.7 33.2 74.0 76.4 83.6 53.9 70.9 95.7 87.9 75.6 74.1 12.8 13.7 40.9 61.3 EuroLLM 22B 78.9 56.9 65.6 58.4 46.9 66.9 81.0 76.1 50.7 49.8 82.3 62.3 14.9 54.8 19.9 66.1 12.0 64.9 28.6 68.0 72.1 83.4 57.1 82.0 94.6 88.3 81.1 78.8 14.4 16.3 38.6 58.4 Qwen3.5 27B 94.3 80.0 84.8 74.9 69.1 82.3 86.1 87.5 80.0 61.0 93.4 68.9 21.6 72.2 33.9 76.7 19.0 83.3 42.6 29.7 29.3 85.0 44.4 64.3 91.1 89.6 81.0 79.0 13.9 16.4 51.6 64.1 Qwen3.6 27B 94.0 79.5 84.4 77.1 69.0 82.5 87.2 86.5 73.3 57.1 93.8 68.6 33.0 70.6 39.6 75.9 21.0 83.3 49.4 27.6 26.4 85.3 41.5 56.6 85.1 77.3 79.1 76.4 13.7 16.3 51.2 63.3 TildeOpen 30B 65.2 46.7 54.9 47.9 41.2 53.0 75.1 46.8 18.0 43.4 68.4 0.0 17.3 36.5 25.5 56.4 8.5 0.0 10.3 16.2 17.1 28.9 28.6 31.1 30.9 27.3 49.0 45.2 1.2 0.8 11.1 32.4 TildeOpen 30B-64k 71.5 52.0 61.3 46.5 43.8 64.4 82.6 63.1 62.7 54.1 76.6 51.8 43.9 40.3 52.6 66.0 24.4 0.0 24.2 40.8 38.2 55.3 46.1 56.8 91.9 67.2 48.1 69.1 7.7 7.5 20.0 49.4 TildeOpen-SK 30B (Alpaca) 68.4 45.7 51.7 37.7 41.0 58.6 70.1 72.2 53.0 48.7 73.5 0.2 10.8 40.0 22.6 61.4 10.5 38.0 16.3 5.0 1.8 9.1 0.5 0.1 0.0 0.0 0.0 1.9 13.2 12.5 20.2 28.5 Gemma 4 31B 94.6 78.5 85.4 74.9 62.4 83.4 86.9 83.3 68.7 56.6 93.9 63.6 11.8 66.7 32.1 77.3 19.5 83.3 33.8 83.8 84.3 85.4 58.9 77.7 96.4 93.3 81.2 76.9 13.9 15.3 50.0 66.9 Aya Expanse 32B 78.0 60.0 64.5 57.1 49.0 69.4 70.7 79.9 41.0 45.0 90.7 44.8 22.2 44.3 14.3 63.7 6.7 49.6 27.1 58.4 58.2 76.4 40.1 59.0 84.9 89.1 82.0 76.2 14.5 16.5 30.4 53.7 Granite 4.0-H Small 83.6 64.5 72.5 62.2 55.5 66.9 69.9 77.9 59.3 46.4 86.7 58.5 25.2 56.5 16.5 66.7 9.2 75.0 26.4 72.4 72.4 84.2 47.5 62.8 94.1 88.6 73.7 73.6 14.6 18.1 43.6 58.9 32–200B Qwen3.6 35B-A3B 92.6 75.8 80.9 74.6 66.6 79.4 81.7 82.9 74.0 50.9 92.3 68.3 34.6 69.2 36.1 75.5 15.5 83.3 43.3 27.6 27.0 83.4 39.8 58.0 91.9 90.0 81.5 76.5 13.9 15.9 46.4 62.2 Apertus 70B 71.3 53.5 63.2 44.2 45.9 66.4 70.1 65.6 51.0 48.8 83.0 49.9 22.5 45.7 14.6 61.2 12.2 80.1 21.2 63.0 60.6 76.6 49.0 69.3 93.3 88.8 51.7 71.8 13.5 13.8 37.9 53.5 Llama 3.3 70B 86.7 71.5 79.2 63.6 59.4 73.8 81.6 83.6 48.7 53.4 91.3 58.5 33.9 62.3 33.1 69.2 18.2 82.1 41.0 69.0 73.4 79.7 51.7 72.1 93.9 91.0 83.1 77.3 16.2 24.3 45.4 63.5 Command A 81.7 57.7 64.2 60.4 42.6 76.1 74.5 24.9 59.7 54.1 92.0 0.1 8.3 47.3 5.0 60.4 3.9 72.5 8.4 50.5 38.4 64.4 6.1 7.8 26.5 12.5 17.8 30.0 10.2 10.9 43.8 39.1 gpt-oss 120B 91.6 78.8 80.6 63.0 72.2 73.3 77.8 80.1 65.7 52.6 91.4 66.6 20.3 70.6 36.0 72.4 21.0 91.0 36.0 76.8 79.1 85.8 55.2 71.1 96.1 89.9 79.3 74.8 12.9 14.4 47.9 65.3 >>200B MiniMax M2.7 93.6 82.2 86.7 71.1 71.8 80.8 82.6 82.9 60.7 60.3 92.9 61.0 24.9 73.2 46.9 75.7 24.1 34.3 42.5 74.7 77.8 82.1 55.8 69.5 96.4 92.5 78.7 74.5 13.1 14.5 62.5 65.8 Qwen3.5 397B-A17B 95.0 82.8 89.6 79.8 73.4 87.6 86.3 90.1 82.3 68.5 94.7 70.0 14.0 77.0 45.0 80.6 25.8 83.3 31.8 64.8 67.4 82.6 45.2 57.2 85.2 84.0 72.9 75.4 13.7 16.1 57.7 67.1 Mistral Large 3 675B 90.6 76.7 83.5 71.9 64.8 79.7 84.2 85.7 62.0 61.9 92.4 59.7 15.0 61.2 22.3 75.7 12.6 10.4 21.6 70.0 69.0 67.5 28.4 42.8 73.5 47.1 78.5 76.4 12.9 15.0 54.6 57.0 GLM-5.1 91.7 76.3 84.4 75.5 61.5 82.8 86.4 79.6 74.7 61.1 92.6 65.7 34.8 71.4 39.6 77.0 18.3 83.2 45.7 77.3 73.8 85.1 61.0 80.7 96.0 92.1 86.5 77.4 13.4 15.2 55.4 68.3 Kimi K2.6 92.6 79.1 85.7 78.1 67.7 85.8 85.7 90.9 79.3 64.3 94.7 63.3 37.1 74.4 44.1 78.8 23.1 0.7 44.1 65.3 61.4 84.2 58.2 77.8 96.4 92.6 83.3 77.8 14.2 16.6 57.5 66.3 Proprietary (size undisclosed) Gemini 3.1 Pro 96.9 91.3 94.8 93.9 81.5 96.2 98.0 91.6 82.3 93.6 96.0 75.1 35.8 88.3 69.7 89.7 38.1 100.0 52.3 84.1 85.6 86.4 62.2 77.3 97.6 96.1 82.1 73.3 14.1 16.0 75.7 77.9 GPT-5.5 96.4 91.3 92.7 84.0 81.1 96.4 96.1 82.5 82.0 93.5 94.2 74.1 22.7 87.5 70.5 88.8 35.6 99.9 43.0 80.4 80.9 83.7 60.7 75.3 97.9 95.2 80.7 73.8 14.3 17.0 71.8 75.6 Gemini 3.1 Flash-Lite 94.7 83.8 88.9 79.3 70.7 89.5 92.4 90.3 84.7 77.6 94.2 71.8 17.6 80.8 26.6 85.5 18.3 99.8 31.0 83.4 82.7 83.9 60.5 80.8 97.5 94.4 81.0 75.4 13.6 15.1 64.1 71.3 Claude Opus 4.6 93.8 86.8 90.5 89.8 71.6 92.6 91.3 87.9 83.0 78.5 94.4 71.5 14.1 70.6 23.7 81.3 13.4 100.0 33.3 81.6 83.0 80.2 57.8 72.7 97.8 94.3 85.5 79.3 14.9 16.8 66.8 70.9 GPT-5.4 mini 91.1 75.0 82.4 69.9 63.4 84.2 87.1 90.1 79.3 58.5 91.7 69.9 26.0 72.7 34.7 80.2 20.2 99.7 38.9 74.4 68.8 81.2 57.7 83.2 95.3 91.5 78.9 76.3 13.8 16.9 56.7 68.1 Claude Sonnet 4.6 93.0 84.0 87.6 78.6 71.5 88.5 88.2 90.1 80.3 73.0 91.2 66.2 14.1 58.9 28.2 79.4 14.3 100.0 19.8 72.5 74.1 81.3 54.1 73.8 95.3 87.0 78.1 76.9 14.6 16.4 62.9 67.5 Table 17: sk-bench results across all tasks. English (EN) prompts, 0-shot — the English-template counterpart of Table 15. IFEval-SK and Slohry are Slovak-only by construction and omitted. Each cell is the task’s primary metric (×100× 100), with best and second-best per column. Avg is the unweighted mean of the columns shown. H.2 Few-shot (3-shot) results The main results are zero-shot. We also ran the applicable tasks 3-shot under both prompt languages (§6.4, Table 8); summarisation and instruction following were run zero-shot only. For the 52 models run 3-shot, Table 18 gives the Slovak 3-shot per-task scores and Table 19 their English-prompt counterpart, parallel to the 0-shot Tables 15 and 17. Model ARC MMLU G-MMLU Scio Driving COPA SKTest-M SKTest-Q skLEP-QA skLEP-NLI skLEP-RTE Senti-3 Senti-2 skLEP-Sent MovieRev Toxic skLEP-Hate Analogies Slohry Avg <<2B Granite 4.0 350M 24.6 27.8 27.1 23.9 24.9 49.1 26.2 1.0 10.1 24.9 42.2 16.1 21.2 36.8 48.0 41.8 62.9 0.5 1.5 26.9 Granite 4.0-H 350M 24.3 25.4 26.0 24.4 59.0 49.9 30.7 1.5 9.4 26.9 49.0 67.9 85.6 22.8 58.8 31.1 44.6 0.1 0.4 33.6 Qwen3 0.6B 35.5 31.8 32.7 26.5 36.5 43.5 33.9 1.1 15.1 35.7 45.2 47.0 84.6 51.8 59.1 54.9 46.9 0.7 8.8 36.4 Qwen3.5 0.8B 40.1 33.6 34.2 26.1 32.3 50.5 29.8 2.1 11.6 36.6 57.2 51.6 78.1 76.2 73.7 54.6 33.6 4.1 9.0 38.7 Gemma 3 1B 28.2 30.0 33.0 25.0 65.0 51.5 33.1 1.8 17.0 35.4 58.6 49.9 61.5 91.5 80.1 54.1 47.8 0.6 3.4 40.4 Granite 4.0 1B 44.2 39.8 42.9 35.1 59.1 56.8 42.1 3.2 22.8 47.5 65.8 55.1 71.5 67.8 65.8 50.6 57.0 1.6 12.3 44.3 Granite 4.0-H 1B 53.9 43.6 44.6 36.3 59.3 51.8 42.1 0.6 21.3 45.2 69.7 37.1 67.3 93.8 81.2 65.2 57.8 0.8 9.6 46.4 EuroLLM 1.7B 26.2 26.6 26.7 25.8 49.6 50.2 26.9 4.8 20.4 30.1 50.2 50.3 87.7 22.6 58.7 45.0 70.2 5.9 24.4 37.0 Qwen3 1.7B 56.4 41.1 42.0 30.1 65.9 63.1 41.1 2.8 20.1 47.9 68.7 28.6 63.3 92.0 80.0 66.6 54.1 2.7 12.4 46.3 2–9B Granite 3.2 2B 42.4 39.6 38.7 29.5 51.0 60.1 37.2 2.9 21.0 37.5 67.0 58.6 72.6 91.7 86.5 79.5 47.1 4.2 7.5 46.0 Qwen3.5 2B 60.6 43.4 45.8 36.0 57.2 53.2 41.5 3.6 24.8 42.7 68.7 45.0 66.9 40.3 67.6 62.8 60.5 2.8 18.2 44.3 Llama 3.2 3B 13.0 9.1 7.3 10.6 12.8 41.0 18.3 2.5 17.2 29.2 45.1 45.7 62.2 29.0 50.1 52.4 48.7 5.9 4.8 26.6 Tiny-Aya Global 67.3 50.7 54.2 41.1 65.4 69.8 54.4 4.6 20.4 65.7 74.5 49.4 81.0 94.4 89.7 87.4 75.9 7.5 19.5 56.5 Tiny-Aya Water 68.3 51.4 54.8 41.4 64.5 71.1 54.3 4.1 20.1 66.2 76.0 45.3 80.0 92.4 89.7 88.0 77.6 8.1 22.6 56.6 Mistral-SK 7B 37.2 29.3 37.9 30.6 56.2 57.4 31.7 11.9 28.0 35.2 58.1 25.6 50.3 94.8 91.3 47.9 71.9 14.4 47.6 45.1 Mistral-SK 7B (Alpaca) 19.2 7.2 12.2 5.0 5.4 54.6 4.9 1.5 6.9 8.8 10.2 0.3 50.4 17.6 9.2 37.1 40.5 0.2 18.6 16.3 Teuken 7B 53.6 42.8 49.8 34.8 68.0 60.0 47.0 5.5 25.2 37.2 58.2 46.6 58.5 75.2 69.4 46.7 73.5 0.1 1.4 44.9 Aya Expanse 8B 53.3 27.4 34.0 20.7 42.8 28.5 29.2 2.4 25.9 28.3 68.0 32.8 50.0 67.9 77.9 77.9 45.0 1.8 1.7 37.6 Llama 3.1 8B 51.0 39.9 50.8 34.2 60.7 64.8 49.6 9.0 30.6 44.9 67.2 55.8 70.0 87.3 80.0 73.5 66.9 20.9 23.7 51.6 Ministral 3 8B 70.8 54.3 54.7 47.6 41.4 34.2 51.4 14.1 27.1 46.1 73.9 43.1 54.0 91.6 86.8 79.5 59.6 31.4 64.8 54.0 Ministral 3 8B (R) 22.9 12.7 5.8 8.8 35.0 0.5 26.6 6.0 21.5 16.0 13.2 36.4 24.2 81.0 80.9 34.8 40.1 22.5 45.2 28.1 9–32B EuroLLM 9B 75.2 54.3 60.5 41.4 79.9 73.2 60.1 9.1 42.2 66.2 75.8 58.1 83.8 95.9 89.9 90.5 73.0 30.5 63.6 64.4 Qwen3.5 9B 88.4 69.7 70.2 59.5 80.5 87.3 66.6 15.1 47.5 75.6 83.9 59.1 82.3 96.9 91.2 86.9 77.2 30.3 70.9 70.5 Bielik 11B 84.5 62.8 67.9 53.2 78.6 89.4 67.1 13.3 25.9 75.4 82.2 63.8 77.5 96.2 92.6 90.7 76.9 27.6 38.7 66.5 Ministral 3 14B 84.1 62.2 66.2 49.2 70.8 83.3 61.0 14.1 33.5 57.8 78.4 54.2 69.9 94.3 85.4 85.4 67.1 28.6 75.4 64.2 Ministral 3 14B (R) 68.3 39.3 28.8 20.0 39.6 70.3 38.3 1.8 14.7 24.2 64.6 10.4 1.9 17.1 16.4 19.1 37.9 1.6 53.2 29.9 Qwen3 14B 86.5 67.8 69.1 59.3 84.4 83.7 65.6 16.0 45.9 77.2 75.9 52.9 71.9 97.2 91.9 87.2 75.9 39.9 67.7 69.3 Qwen3-SK 14B 73.5 41.7 44.0 24.0 3.9 18.9 46.7 16.2 24.0 52.9 71.2 5.6 7.6 12.3 29.3 1.2 21.1 48.5 0.2 28.6 Qwen3-SK 14B (Alpaca) 88.0 69.7 68.4 59.4 86.9 86.9 68.9 8.3 11.5 78.1 81.0 57.7 74.4 96.1 91.7 90.0 77.6 18.9 59.9 67.0 gpt-oss 20B 86.3 71.8 71.9 64.1 72.2 84.0 62.4 7.3 25.5 75.3 82.4 57.1 77.1 96.3 88.3 85.4 75.1 25.2 73.5 67.4 EuroLLM 22B 81.0 61.3 64.1 48.4 81.8 88.6 66.3 13.3 42.0 72.9 75.8 59.5 84.2 95.9 91.3 73.2 62.9 33.5 64.8 66.4 Qwen3.5 27B 94.9 82.1 85.7 69.6 88.6 90.6 78.2 23.4 51.8 82.3 82.9 62.9 83.3 97.6 94.4 90.4 80.0 50.7 86.0 77.6 Qwen3.6 27B 95.1 81.7 84.4 67.8 88.7 89.9 77.5 25.6 53.0 80.4 85.3 63.2 87.6 97.4 94.4 87.6 75.8 51.1 85.4 77.5 TildeOpen 30B 36.4 31.4 21.2 23.3 63.9 66.9 41.8 2.5 1.8 18.9 31.5 66.3 45.5 72.8 62.1 54.8 35.3 13.2 46.1 38.7 TildeOpen 30B-64k 77.2 57.6 65.6 47.6 82.1 89.2 71.7 28.9 35.7 54.9 74.7 66.6 80.5 97.8 94.8 80.1 71.2 44.4 85.4 68.7 TildeOpen-SK 30B (Alpaca) 75.3 50.6 52.7 39.2 80.8 89.4 66.9 15.0 26.5 46.5 65.5 56.0 76.0 46.5 76.6 84.6 70.8 6.8 54.1 56.8 Gemma 4 31B 94.7 81.9 86.7 72.1 89.0 84.6 77.7 27.6 50.0 85.0 85.6 62.5 83.1 97.4 94.5 91.3 78.8 51.8 86.3 77.9 Aya Expanse 32B 77.8 61.4 65.6 50.6 78.0 84.3 64.3 7.6 32.6 74.7 78.7 49.6 75.0 95.8 92.6 89.4 72.6 15.2 6.4 61.7 Granite 4.0-H Small 85.8 67.7 75.8 56.6 80.0 88.1 68.1 14.1 37.9 77.7 82.4 59.9 73.9 96.0 91.8 89.1 74.7 30.4 65.6 69.2 32–200B Qwen3.6 35B-A3B 93.1 78.6 83.1 66.5 85.8 88.7 75.5 21.4 52.7 78.9 82.8 61.1 85.7 96.9 93.6 83.7 76.4 47.6 84.0 75.6 Apertus 70B 77.1 58.5 67.4 47.8 83.0 90.3 63.0 17.8 32.5 61.8 69.3 53.8 71.9 95.2 91.7 84.7 66.9 40.3 75.3 65.7 Llama 3.3 70B 88.3 73.1 77.0 58.4 83.5 89.1 69.3 21.0 44.2 78.0 82.2 59.4 84.6 97.4 94.0 89.9 77.6 41.7 74.2 72.8 Command A 37.6 24.0 14.8 22.2 1.9 18.9 19.9 2.5 2.3 37.6 23.3 8.5 8.1 23.9 33.3 26.2 29.1 42.4 5.0 20.1 gpt-oss 120B 91.2 78.0 79.4 68.3 78.4 84.5 71.7 20.3 34.0 78.5 84.6 58.5 80.8 96.3 78.9 86.2 72.1 44.4 83.3 72.1 >>200B MiniMax M2.7 93.2 83.1 87.2 72.5 84.5 86.1 76.0 27.6 44.7 79.6 83.5 60.0 79.4 97.1 93.6 86.3 75.3 57.0 92.1 76.8 Qwen3.5 397B-A17B 95.4 84.9 89.2 72.9 88.1 91.7 80.3 29.4 45.7 83.2 85.6 61.2 81.7 97.5 94.5 85.8 75.8 60.4 92.8 78.7 Mistral Large 3 675B 91.6 78.9 84.2 63.2 87.8 92.3 76.3 26.6 38.8 76.4 84.0 44.8 49.2 87.5 59.8 91.3 79.1 52.0 86.4 71.1 GLM-5.1 92.6 81.2 86.5 68.0 86.5 92.1 79.6 32.6 50.8 80.1 85.5 60.6 89.8 96.7 93.7 90.4 77.9 57.2 91.1 78.6 Kimi K2.6 93.8 82.4 88.8 72.5 88.4 93.3 78.8 29.7 53.6 83.8 83.4 62.0 86.9 97.3 94.1 91.1 77.2 60.6 92.1 79.5 Proprietary (size undisclosed) Claude Sonnet 4.6 94.2 82.7 – – 88.8 92.7 82.2 – 42.0 84.3 85.5 59.5 83.1 97.7 94.7 88.8 77.4 63.5 – 81.1 Gemini 3.1 Flash-Lite 94.9 85.7 90.7 71.9 92.0 93.7 85.2 21.1 46.5 84.4 86.4 63.0 80.2 97.8 94.7 91.0 78.0 66.7 94.1 79.9 GPT-5.4 mini 91.1 75.3 80.8 61.8 89.7 90.5 79.4 30.3 44.6 75.0 80.6 59.9 86.1 97.5 91.6 90.7 72.5 46.5 91.0 75.5 Table 18: sk-bench results across all tasks. Slovak (SK), 3-shot — the few-shot counterpart of Table 15, restricted to the tasks and models that were run with 3-shot exemplars (random exemplars from each task’s train/validation split). Summarisation, instruction following, and the other tasks run zero-shot only are omitted, as are the models without few-shot results. Each cell is the task’s primary metric (×100× 100), with best and second-best per column. Avg is the unweighted mean of the columns shown. Model ARC MMLU G-MMLU Scio Driving COPA SKTest-M SKTest-Q skLEP-QA skLEP-NLI skLEP-RTE Senti-3 Senti-2 skLEP-Sent MovieRev Toxic skLEP-Hate Avg <<2B Granite 4.0 350M 26.3 28.0 29.2 25.3 22.2 49.9 28.6 1.3 11.4 34.2 50.3 12.7 70.0 90.1 70.3 46.4 72.7 0.2 37.2 Granite 4.0-H 350M 27.1 27.6 28.1 28.9 31.3 50.6 30.7 1.1 11.1 31.0 55.1 55.1 87.3 64.1 60.0 47.0 72.5 0.2 39.4 Qwen3 0.6B 36.8 34.1 34.3 29.6 30.7 53.3 31.8 2.1 15.5 36.7 55.2 44.1 81.0 83.0 67.6 55.7 63.7 0.4 42.0 Qwen3.5 0.8B 43.3 35.0 37.6 28.6 48.7 51.9 31.6 1.7 14.6 36.7 60.4 48.7 76.6 89.6 74.7 57.9 71.5 0.6 45.0 Gemma 3 1B 30.5 29.9 33.6 24.0 67.9 52.4 35.8 2.2 18.1 40.3 61.5 50.6 64.4 91.5 78.0 72.7 67.4 0.9 45.7 Granite 4.0 1B 45.5 40.2 43.3 34.5 48.9 58.9 39.8 3.5 23.0 57.5 71.6 51.2 61.8 92.6 83.8 72.7 72.8 1.9 50.2 Granite 4.0-H 1B 53.5 43.2 46.2 36.2 68.5 53.7 43.9 1.4 21.0 56.0 72.9 45.8 65.1 93.6 83.0 55.4 72.6 0.3 50.7 EuroLLM 1.7B 27.5 27.1 27.8 27.4 17.2 50.2 26.7 4.3 19.5 35.8 52.5 58.0 50.3 94.1 82.5 38.7 71.7 5.1 39.8 Qwen3 1.7B 55.8 41.7 44.1 37.7 63.5 62.6 39.4 4.2 27.1 56.1 70.1 39.1 71.3 90.7 83.1 68.0 58.0 3.3 50.9 2–9B Granite 3.2 2B 42.5 40.7 42.7 31.7 50.6 59.8 37.9 2.2 29.0 62.5 75.4 60.7 73.9 93.3 86.6 74.0 76.0 3.5 52.4 Qwen3.5 2B 64.2 46.0 47.2 40.2 56.9 56.1 44.7 4.6 31.3 47.2 74.2 46.2 71.6 83.5 83.3 77.0 74.0 1.5 52.8 Llama 3.2 3B 11.1 14.5 9.3 9.6 14.6 50.3 18.0 2.2 15.7 35.4 48.9 38.5 58.1 40.3 53.9 58.4 61.1 5.2 30.3 Tiny-Aya Global 66.6 51.3 55.3 39.3 71.1 61.1 53.8 6.0 23.9 66.8 73.3 49.8 66.5 91.1 86.6 81.5 76.3 5.6 57.0 Tiny-Aya Water 67.9 52.0 55.7 40.8 70.5 63.5 54.4 6.0 23.1 66.6 72.3 51.1 72.0 92.4 89.1 82.6 77.1 5.9 57.9 Mistral-SK 7B 36.4 28.9 38.0 29.0 52.9 52.8 31.1 12.3 26.8 35.6 53.5 23.1 73.2 88.0 90.5 43.5 68.2 13.5 44.3 Mistral-SK 7B (Alpaca) 20.0 5.1 9.3 4.3 1.5 50.6 2.8 2.2 6.7 0.8 0.5 0.4 16.1 15.1 0.2 1.2 3.8 0.2 7.8 Teuken 7B 54.9 43.3 51.3 34.2 70.0 72.7 47.8 6.6 22.8 52.8 66.3 33.4 51.3 63.8 77.0 71.5 73.6 1.0 49.7 Aya Expanse 8B 64.2 50.5 54.6 40.7 60.5 77.1 54.5 6.2 33.2 64.4 76.7 26.1 44.4 63.2 55.5 83.6 78.0 3.5 52.0 Llama 3.1 8B 43.5 32.7 42.5 28.8 25.4 65.6 32.6 6.4 26.2 46.8 67.5 41.9 41.0 65.0 56.6 60.0 50.7 10.1 41.3 Ministral 3 8B 79.9 59.5 63.7 47.3 57.6 72.9 57.5 13.2 25.7 56.4 70.9 54.3 67.1 93.9 88.9 86.3 71.0 33.2 61.1 Ministral 3 8B (R) 33.7 18.2 16.6 13.3 60.9 0.3 30.4 1.8 21.1 37.8 52.2 61.7 74.8 59.8 72.9 54.4 43.0 30.0 37.9 9–32B EuroLLM 9B 78.0 55.3 61.1 45.0 77.6 77.7 63.0 14.8 36.2 73.5 81.2 56.4 71.0 97.1 91.2 90.0 76.5 23.6 65.0 Qwen3.5 9B 89.1 69.6 70.4 60.2 81.4 87.3 67.7 16.7 46.1 78.6 83.0 54.8 57.2 95.2 90.8 85.4 77.2 31.8 69.0 Bielik 11B 84.9 64.3 69.7 55.2 79.0 88.4 68.1 19.9 43.0 80.0 81.5 62.7 75.8 96.7 93.2 89.7 78.3 34.2 70.3 Ministral 3 14B 81.5 62.6 65.8 51.1 74.4 85.1 60.0 12.2 31.3 62.9 80.5 53.9 59.9 95.1 85.7 82.7 75.5 39.1 64.4 Ministral 3 14B (R) 73.8 43.5 26.8 32.8 60.3 82.2 37.1 1.4 4.2 20.1 54.8 14.0 3.9 29.4 42.7 17.2 48.0 8.8 33.4 Qwen3 14B 87.1 67.5 68.8 59.1 85.9 83.8 63.9 16.5 45.5 81.8 82.7 49.2 76.9 96.5 91.1 85.2 77.7 38.3 69.9 Qwen3-SK 14B 83.7 54.5 62.7 31.4 66.4 3.5 64.1 17.5 34.1 62.9 57.7 11.4 12.6 15.4 29.0 60.4 60.7 47.1 43.1 Qwen3-SK 14B (Alpaca) 88.1 69.6 67.9 60.9 87.1 87.6 69.1 13.7 13.0 57.5 49.5 21.7 53.9 14.7 12.2 43.1 45.5 18.3 48.5 gpt-oss 20B 88.4 72.6 73.8 64.7 71.2 79.3 65.3 12.5 31.0 77.2 84.1 55.3 76.7 97.0 88.3 81.5 75.0 31.6 68.1 EuroLLM 22B 80.1 59.7 64.3 50.1 79.6 84.7 67.7 19.2 39.2 73.9 82.6 58.5 77.8 97.1 92.8 88.3 76.9 33.6 68.1 Qwen3.5 27B 95.2 82.4 86.2 68.9 88.1 89.7 77.7 25.4 52.2 81.0 85.7 57.1 82.8 97.8 94.6 89.1 78.1 50.5 76.8 Qwen3.6 27B 94.7 81.9 84.2 68.5 88.1 87.8 76.2 25.8 53.2 77.9 86.3 60.5 84.8 98.0 94.7 86.7 75.5 50.6 76.4 TildeOpen 30B 56.1 33.9 38.5 30.1 76.1 51.9 44.6 2.4 1.8 13.5 21.2 34.5 31.8 56.2 33.9 49.7 21.8 16.2 34.1 TildeOpen 30B-64k 77.6 56.8 63.1 48.1 82.0 84.5 70.5 28.0 34.6 45.5 64.7 64.3 79.3 98.2 94.2 83.0 70.2 46.0 66.2 TildeOpen-SK 30B (Alpaca) 70.7 44.3 43.7 33.9 79.4 85.1 63.9 12.2 24.3 36.5 30.0 27.8 19.6 34.6 16.9 4.1 17.3 10.2 36.4 Gemma 4 31B 94.8 82.1 86.1 71.3 89.5 83.7 77.9 29.7 51.5 85.2 85.6 61.8 84.3 97.4 94.9 90.4 77.4 51.8 77.5 Aya Expanse 32B 79.6 63.2 67.8 51.6 78.4 86.1 65.4 12.2 43.6 79.8 82.0 41.9 73.7 94.3 92.5 90.8 79.2 16.9 66.6 Granite 4.0-H Small 86.2 67.5 75.2 56.1 77.3 88.1 67.7 13.7 38.1 80.4 80.1 54.9 69.0 96.4 91.2 88.6 75.8 32.3 68.8 32–200B Qwen3.6 35B-A3B 93.2 78.7 82.9 66.7 87.6 88.0 75.4 21.7 52.6 80.4 83.5 53.9 77.4 97.4 93.4 85.7 76.7 49.8 74.7 Apertus 70B 77.0 59.3 67.8 50.3 82.9 88.2 66.0 15.8 27.5 66.8 76.5 39.2 69.1 94.2 91.2 60.2 62.8 39.9 63.0 Llama 3.3 70B 88.1 73.5 78.8 59.2 83.4 88.7 70.4 21.3 44.6 78.5 80.8 50.0 62.4 95.6 93.4 90.0 78.1 43.8 71.2 Command A 1.2 11.0 7.2 6.8 0.5 5.2 14.6 4.2 10.1 20.8 35.6 22.9 36.9 32.6 30.8 35.7 27.5 45.9 19.4 gpt-oss 120B 91.5 77.7 80.8 67.2 77.0 82.7 67.6 20.6 36.2 80.4 85.5 57.7 79.9 96.4 83.6 83.6 74.4 47.2 71.7 >>200B MiniMax M2.7 93.3 82.9 86.2 73.3 85.8 83.5 76.5 28.9 46.3 81.1 82.3 60.9 79.5 97.2 93.8 84.4 75.3 59.7 76.2 Qwen3.5 397B-A17B 95.4 85.1 90.5 73.1 89.1 93.5 81.0 32.2 47.8 85.1 84.7 62.5 83.5 97.6 95.0 83.7 76.0 60.5 78.7 Mistral Large 3 675B 92.0 74.2 82.7 59.9 84.6 91.5 74.5 26.8 39.1 77.6 83.5 49.2 57.6 89.5 68.9 89.0 79.9 53.4 70.8 GLM-5.1 92.5 81.4 85.5 68.4 86.5 92.7 79.4 32.2 49.6 82.5 85.3 62.4 89.6 97.2 93.5 91.1 78.1 57.7 78.1 Kimi K2.6 94.0 82.6 88.8 72.6 87.4 92.9 78.1 32.5 52.7 85.8 84.0 59.5 78.9 97.4 93.8 89.1 78.3 61.9 78.4 Proprietary (size undisclosed) Claude Sonnet 4.6 93.9 81.4 – – 90.1 92.4 83.1 – 45.5 81.8 85.0 58.2 84.4 97.6 94.6 87.4 77.5 64.4 81.1 Gemini 3.1 Flash-Lite 94.6 85.6 90.4 72.0 91.7 92.8 85.1 33.5 50.3 84.3 86.5 63.1 83.8 98.0 95.4 90.0 77.2 66.3 80.0 GPT-5.4 mini 90.7 75.3 83.0 61.1 89.7 90.3 79.4 29.1 44.6 74.4 79.7 60.9 86.9 97.6 91.3 90.2 76.4 46.7 74.9 Table 19: sk-bench results across all tasks. English (EN) prompts, 3-shot — the English-template, few-shot counterpart of Table 15, restricted to the tasks and models run with 3-shot exemplars. Summarisation, instruction following, the Slovak-only tasks (IFEval-SK, Slohry), and the other tasks run zero-shot only are omitted, as are the models without few-shot results. Each cell is the task’s primary metric (×100× 100), with best and second-best per column. Avg is the unweighted mean of the columns shown. H.3 Generator self-preference check One task in the suite has LLM-generated questions: MultiWikiQA-SK’s questions were generated by Gemini 1.5 Pro over the November 2023 Slovak Wikipedia dump, then rephrased in a second pass by the same model to reduce surface word overlap with the passage. This could favour the generator’s own model family. We check this against the human-authored task in the same reporting group, skLEP-QA: if the generator family were favoured, Gemini models would rank higher on the generated set than on human-authored skLEP-QA. They do not. Across the 55 models, Gemini 3.1 Pro ranks 1st on human-authored skLEP-QA and 4th on the Gemini-generated set, and Gemini 3.1 Flash-Lite drops from 30th to 38th—both rank lower on the generated set, the opposite of self-preference. The models that gain most are small and older ones (e.g. Mistral-SK 7B, +26+26 places), which suggests that weaker models rely on surface word matching, not that the generator is favoured. The low agreement between human-authored and LLM-generated QA in RQ1 (ρ=0.72ρ=0.72, §6.1) thus reflects how the generated questions can be answered, not which model generated them. Appendix I Collection and Curation of New Datasets For each dataset introduced with sk-bench whose provenance warrants more than the one-line summary in Appendix D, this section records how the items were obtained, who curated them, and whether human annotators were involved. Chiby has its own curation card in Appendix K and IFEval-SK’s instruction registry is described in Appendix L. I.1 Scio Collection. Parsed by the authors from Scio admission-test papers, covering general study prerequisites alongside biology, chemistry, and mathematics. Curation. The parsing, cleaning, and answer-key alignment are ours. No external annotators were involved. Licence. No licence is declared on the source artefact, so the catalogue (Table 10) records “—”. I.2 Autoškola Collection. The official Slovak driving-licence theory questions, spanning 35 topic categories, with the official answer key as gold. Curation. An LLM assisted in constructing the dataset. The gold labels themselves are the official answer key, not model output. This use of an LLM inside the benchmark is also flagged in the Ethical Considerations section and carries a provenance label in the catalogue. I.3 SKJ1 Collection. Adopted from the SKJ1 dataset (Dobeš and Bednár, 2026): the items are expert-authored by the team behind that dataset, and we did not recruit experts ourselves. Curation. sk-bench’s contribution is the generative-LLM wrapping—the prompts, the answer parsing, and the scoring. Licence. No licence is declared on the source artefact, so the catalogue (Table 10) records “—”. Appendix J Summarisation Datasets Downsampling To keep evaluation affordable while preserving the length distribution, we applied a stratified downsampling procedure based on text length. For each dataset (SlovakSum and SME-Sum), we divided the full range of document lengths into 10 equal-width bins and allocated the target sample size of 1,500 examples per split proportionally according to each bin’s representation in the original data. This keeps rare long documents in the sample, which uniform random sampling would tend to miss. Any rounding remainder from the proportional allocation was distributed to the largest bins, with final samples drawn without replacement within each bin. The resulting subsets preserve the distributional statistics (median, interquartile range, and mean) while reducing computational costs. Appendix K Chiby-classification: Dataset Curation To construct the dataset chiby-classification, we preprocessed and annotated the dataset Chiby v0.4 Garabík (2026), which is based on revisions of Slovak Wikipedia. The preprocessing was carried out in several steps. First, the data were decompressed and cleaned by removing tags from sentences, and errors were extracted into new columns for annotation purposes. The error tags belong to one of seven categories: unclassified, spelling, typographical, punct, lexicosemantic, diacritics, and capitalisation. Of the seven revision types present in the original dataset, three were selected: spelling, diacritics, and capitalisation. Sentence pairs not containing any of the selected error types were removed. The following sentence pairs were then excluded: • Pairs containing more than one type of error. • Pairs in which the sentence had fewer than 55 characters or more than 300 characters. • Pairs containing a URL (http, w). • Pairs that were parsed incorrectly, identified as sentences not beginning with a capital letter or not ending with a full stop, exclamation mark, or question mark. Sentence similarity was then checked with cosine similarity to confirm correct pairing between the original and revised sentences. The threshold was selected empirically through manual inspection of pairs produced by several candidate thresholds, including 0.90 and 0.95, and 0.99 was found to best preserve correctly paired sentences while filtering out mismatched pairs. Only pairs with a cosine similarity score above 0.99 were retained. Finally, the langdetect library11 1 https://pypi.org/project/langdetect/ was used to verify that the sentences were in Slovak, those identified as being in another language were removed. After preprocessing, 67% of pairs contained spelling errors, 23% diacritics errors, and 10% capitalisation errors. A total of 2,400 sentence pairs were then randomly sampled to match this distribution: 1,608 from the spelling category, 552 from the diacritics category, and 240 from the capitalisation category. The dataset was annotated by three native Slovak speakers, all of whom hold a university-level degree and are co-authors of this work. Each sentence pair was annotated by a single annotator, who determined whether the first sentence (original) was correct, the second sentence (revised) was correct, or neither. As each pair was reviewed by one annotator only, inter-annotator agreement was not computed. Where a grammatical error was minor, it was corrected in the revised sentence and the pair was retained. Pairs for which neither sentence was deemed correct and the error could not be straightforwardly corrected were discarded (see Section K.1 for full annotation instructions). After annotation, 1,716 sentence pairs remained and were randomly shuffled. Sentences were labelled 0 if incorrect and 1 if correct, resulting in a total of 3,432 rows in the dataset. This dataset is released under the Creative Commons Attribution-ShareAlike 4.0 International (C BY-SA 4.0)22 2 https://creativecommons.org/licenses/by-sa/4.0/deed.en licence, consistent with the licence of the Chiby v0.4 dataset. K.1 Annotation Instructions You will be shown a series of sentence pairs. Each pair consists of an original sentence and a revised sentence. For each pair, read both sentences and assign one of the following labels: • original – the original sentence is correct as written (the revision was unnecessary or introduces an error). • revision – the revised sentence is the correct one (it fixes an error present in the original). • fixed – both sentences contain an error, but you as the annotator have corrected the revised sentence and provided the correct version. • neither – both sentences are incorrect and the error cannot be straightforwardly corrected. When in doubt, ask yourself:Is either sentence fully correct on its own? If yes, label accordingly. If neither sentence is correct but you can see a clear correction, write the corrected version and label it fixed. If the error is too complex to correct, label it neither. Please note: • You do not need to identify or describe what type of error is present – only whether the sentence is correct or not. • When using the fixed label, make sure to provide your corrected version of the sentence. • If you label a pair as neither, it will be discarded from the final dataset. Only use this label when the error is too complex or unclear to correct in a simple revision. • Each sentence pair is reviewed by you alone, so take your time and read carefully. Appendix L IFEval-SK: Slovak Instruction Registry Size and splits. 537 prompts. Adaptation procedure. For each English instruction type in Zhou et al. (2023), we (i) translated the instruction template, and (i) re-implemented the programmatic checker against Slovak word boundaries, casing rules, and punctuation. Appendix M Prompt Library Most tasks have three prompt variants in both Slovak and English to measure prompt sensitivity (§5). IFEval-SK uses one Slovak prompt per item, and Slohry is also Slovak-only. Each box below pairs the Slovak (left) and English (right) form of one variant. slots mark the fields filled in at run time. Typographic quotation marks are reproduced as in the released templates. M.1 NLI — binary NLI — binary Prompt 1 SLOVAK Aký je vzťah medzi nasledujúcimi dvomi vetami? Vyplýva druhá veta z prvej alebo jej protirečí? Odpovedz iba slovom „vyplýva“ alebo „protirečí“. Premisa: sentence1 Hypotéza: sentence2 Odpoveď: ENGLISH What is the relationship between the two sentences? Does the first sentence entail or contradict the second sentence? Answer only with the word "entailment" or "contradiction". Premise: sentence1 Hypothesis: sentence2 Answer: NLI — binary Prompt 2 SLOVAK Urči, či hypotéza vyplýva z danej premisy. Odpovedz iba slovom „áno“ alebo „nie“. Premisa: sentence1 Hypotéza: sentence2 Odpoveď: ENGLISH Determine whether the premise entails the hypothesis. Answer only "yes" or "no". Premise: sentence1 Hypothesis: sentence2 Answer: NLI — binary Prompt 3 SLOVAK Napíšem dve vety, ktorým hovorím premisa a hypotéza. Hypotéza môže z premisy buď vyplývať alebo jej protirečiť. Vyhodnoť vzťah týchto viet a odpovedz buď slovom „vyplýva“ alebo „protirečí“. Premisa: sentence1 Hypotéza: sentence2 Odpoveď: ENGLISH I will write two sentences, which I call the premise and the hypothesis. The premise may entail the hypothesis or contradict it. Determine the relationship between these sentences and answer with either "entailment" or "contradiction". Premise: sentence1 Hypothesis: sentence2 Answer: M.2 NLI — 3-class NLI — 3-class Prompt 1 SLOVAK Aký je vzťah medzi dvomi vetami? Druhá veta z prvej vyplýva, protirečí jej alebo je neutrálna? Odpovedz iba slovom „vyplýva“, „protirečí“ alebo „neutrálny“. Premisa: sentence1 Hypotéza: sentence2 Odpoveď: ENGLISH What is the relationship between the two sentences? Does the first sentence entail, contradict, or is neutral towards the second sentence? Answer only "entailment", "contradiction", or "neutral". Premise: sentence1 Hypothesis: sentence2 Answer: NLI — 3-class Prompt 2 SLOVAK Napíšem dve vety, ktorým hovorím premisa a hypotéza. Hypotéza môže z premisy vyplývať, alebo jej protirečiť. Pokiaľ ani nevyplýva, ani neprotirečí, tak je neutrálna. Vyhodnoť, či hypotéza z premisy „vyplýva“, alebo jej „protirečí“, alebo je vzťah „neutrálny“. Premisa: sentence1 Hypotéza: sentence2 Odpoveď: ENGLISH I will write two sentences, which I call the premise and the hypothesis. The premise may entail the hypothesis or contradict it. If it neither entails nor contradicts the hypothesis, then it is neutral. Determine the relationship as "entailment", "contradiction" or "neutral". Premise: sentence1 Hypothesis: sentence2 Answer: NLI — 3-class Prompt 3 SLOVAK Úloha: Urči, či hypotéza vyplýva z premisy (vyplýva), či je s ňou v spore (protirečí) alebo ani jeden z týchto vzťahov (neutrálny). Premisa: sentence1 Hypotéza: sentence2 Odpoveď: ENGLISH Task: Determine whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). Premise: sentence1 Hypothesis: sentence2 Answer: M.3 Sentiment — binary Sentiment — binary Prompt 1 SLOVAK Urči sentiment nasledujúceho textu. Odpovedz iba slovom „negatívny“ alebo „pozitívny“. Text: text Sentiment považujem za: ENGLISH Determine the sentiment of the following text. Answer only "negative" or "positive". Text: text I consider the sentiment to be: Sentiment — binary Prompt 2 SLOVAK Rozhodni, či je sentiment nasledujúceho textu negatívny alebo pozitívny. Odpovedz iba slovom „negatívny“ alebo „pozitívny“. Text: text Odpoveď: ENGLISH Determine whether the sentiment of the following text is negative or positive. Answer only "negative" or "positive". Text: text Answer: Sentiment — binary Prompt 3 SLOVAK Klasifikuj sentiment do kategórií: negatívny/pozitívny. Text: text Sentiment: ENGLISH Classify sentiment into the following categories: negative/positive. Text: text Sentiment: M.4 Sentiment — 3-class Sentiment — 3-class Prompt 1 SLOVAK Urči sentiment nasledujúceho textu. Odpovedz iba slovom „negatívny“, „neutrálny“ alebo „pozitívny“. Text: text Sentiment považujem za: ENGLISH Determine the sentiment of the following text. Answer only "negative", "neutral", or "positive". Text: text I consider the sentiment to be: Sentiment — 3-class Prompt 2 SLOVAK Rozhodni, či sentiment nasledujúceho textu je negatívny, neutrálny alebo pozitívny. Odpovedz iba slovom „negatívny“, „neutrálny“ alebo „pozitívny“. Text: text Odpoveď: ENGLISH Determine whether the sentiment of the following text is negative, neutral, or positive. Answer only with "negative", "neutral", or "positive". Text: text Answer: Sentiment — 3-class Prompt 3 SLOVAK Klasifikuj sentiment do kategórií: negatívny/neutrálny/pozitívny. Text: text Sentiment: ENGLISH Classify sentiment into the following categories: negative/neutral/positive. Text: text Sentiment: M.5 Hate speech — binary Hate speech — binary Prompt 1 SLOVAK Urči, či zadaný výrok obsahuje nenávistné prejavy. Odpovedz len slovom „áno“ alebo „nie“. Výrok: text Odpoveď: ENGLISH Determine whether the given claim includes hate speech. Answer only "yes" or "no". Claim: text Answer: Hate speech — binary Prompt 2 SLOVAK Obsahuje nasledujúci text nenávistné prejavy? Odpovedz áno/nie. Text: text Odpoveď: ENGLISH Does the following text contain hate speech? Answer yes/no. Text: text Answer: Hate speech — binary Prompt 3 SLOVAK Rozhodni, či nasledujúci text obsahuje známky nenávistného prejavu (hate speech). Odpovedz iba slovom „áno“ alebo „nie“. Text: text Odpoveď: ENGLISH Determine whether the following text contains signs of hate speech. Respond only with the word "yes" or "no". Text: text Answer: M.6 Toxicity — binary Toxicity — binary Prompt 1 SLOVAK Urči, či daný text obsahuje toxické prejavy. Odpovedz len „áno“ alebo „nie“. Text: text Odpoveď: ENGLISH Determine whether the given text includes toxic speech. Answer only "yes" or "no". Text: text Answer: Toxicity — binary Prompt 2 SLOVAK Je nasledujúci text toxický? Odpovedz len „áno“ alebo „nie“. Text: text Odpoveď: ENGLISH Is the following text toxic? Answer only "yes" or "no". Text: text Answer: Toxicity — binary Prompt 3 SLOVAK Zisti, či v texte nižšie nájdeš akúkoľvek formu toxického obsahu. Napíš „áno“, ak sa v texte vyskytuje toxický obsah, inak „nie“. Text: text Odpoveď: ENGLISH Check the text below to see if it contains any form of toxic content. Write "yes" if the text contains toxic content, otherwise "no". Text: text Answer: M.7 Summarisation Summarisation Prompt 1 SLOVAK text Napíš zhrnutie vyšie uvedeného textu s využitím maximálne 50 slov. Výstup musí byť v slovenčine. Zhrnutie: ENGLISH text Write a summary for the text above with a maximum of 50 words. The output has to be in Slovak. Summary: Summarisation Prompt 2 SLOVAK Sumarizuj nasledujúci text v slovenčine do maximálne 50 slov. Text: text Súhrn: ENGLISH Summarize the following text in Slovak into a maximum of 50 words. Text: text Summary: Summarisation Prompt 3 SLOVAK Prečítaj si nasledujúci text a stručne ho sumarizuj v slovenčine, s využitím maximálne 50 slov. Text: text Zhrnutie: ENGLISH Read the following text and summarize it briefly in Slovak, using no more than 50 words. Text: text Summary: M.8 Completion Completion Prompt 1 SLOVAK Doplň pokračovanie textu výberom jednej z nasledujúcich možností. Odpovedz iba použitím písmena správnej odpovede (A, B, C, ..., end_letter). Text: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Complete the sentence by selecting one of the following options. Answer only by selecting the letter corresponding to the correct answer (A, B, C, ..., end_letter). Text: question Choices: A) option1 B) option2 C) option3 ... Answer: Completion Prompt 2 SLOVAK Úloha: Vyber správne pokračovanie nasledujúceho textu z možností A, B, C, ..., end_letter. Odpovedz len písmenom správnej odpovede. Text: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Task: Select the correct continuation of the following text from the options: A, B, C, ..., end_letter. Answer only with the letter of the correct answer. Text: question Choices: A) option1 B) option2 C) option3 ... Answer: Completion Prompt 3 SLOVAK Zvoľ najvhodnejšiu možnosť na dokončenie nasledujúceho textu. Napíš len písmeno správnej odpovede. Text: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Choose the best option to complete the following text. Write only the letter of the correct answer. Text: question Choices: A) option1 B) option2 C) option3 ... Answer: M.9 Multiple choice — single answer Multiple choice — single answer Prompt 1 SLOVAK Vyber správnu odpoveď A, B, C, ..., end_letter pre zadanú otázku. Pre každú otázku existuje len jedna správna odpoveď. Odpovedz len písmenom správnej možnosti. Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Choose the correct answer A, B, C, ..., end_letter for the given question. There is only one correct answer for each question. Answer only with the letter of the correct option. Question: question Choices: A) option1 B) option2 C) option3 ... Answer: Multiple choice — single answer Prompt 2 SLOVAK Úloha: Vyber správnu odpoveď A, B, C, ..., end_letter na nasledujúcu otázku. Odpovedz len písmenom správnej možnosti. Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Task: Select the correct answer A, B, C, ..., end_letter for the following question. Answer only with the letter of the correct option. Question: question Choices: A) option1 B) option2 C) option3 ... Answer: Multiple choice — single answer Prompt 3 SLOVAK Predložím ti otázky a možné odpovede A, B, C, ..., end_letter a tvojou úlohou je vybrať správnu odpoveď. Odpovedz len písmenom správnej možnosti. Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH I will present you with questions and possible answers A, B, C, ..., end_letter, and your task is to select the correct answer. Answer only with the letter of the correct option. Question: question Choices: A) option1 B) option2 C) option3 ... Answer: M.10 Multiple choice with context — single answer Multiple choice with context — single answer Prompt 1 SLOVAK Vyber správnu odpoveď A, B, C, ..., end_letter pre zadanú otázku len na základe uvedeného kontextu. Pre každú otázku existuje len jedna správna odpoveď. Odpovedz len písmenom správnej možnosti. Kontext: context Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Choose the correct answer A, B, C, ..., end_letter for the given question only based on the given context. There is only one correct answer for each question. Answer only with the letter of the correct option. Context: context Question: question Choices: A) option1 B) option2 C) option3 ... Answer: Multiple choice with context — single answer Prompt 2 SLOVAK Úloha: Vyber správnu odpoveď A, B, C, ..., end_letter na nasledujúcu otázku len na základe uvedeného kontextu. Odpovedz len písmenom správnej možnosti. Kontext: context Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Task: Select the correct answer A, B, C, ..., end_letter for the following question only based on the given context. Answer only with the letter of the correct option. Context: context Question: question Choices: A) option1 B) option2 C) option3 ... Answer: Multiple choice with context — single answer Prompt 3 SLOVAK Predložím ti otázky a možné odpovede A, B, C, ..., end_letter a tvojou úlohou je vybrať správnu odpoveď len na základe uvedeného kontextu. Odpovedz len písmenom správnej možnosti. Kontext: context Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH I will present you with questions and possible answers A, B, C, ..., end_letter, and your task is to select the correct answer only based on the given context. Answer only with the letter of the correct option. Context: context Question: question Choices: A) option1 B) option2 C) option3 ... Answer: M.11 Multiple choice — multiple answers Multiple choice — multiple answers Prompt 1 SLOVAK Vyber všetky správne odpovede A, B, C, ..., end_letter pre zadanú otázku. Pre každú otázku môže byť jedna alebo viac správnych odpovedí. Odpovedz zoznamom písmen správnych odpovedí oddelených čiarkami. Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Choose all the correct answers A, B, C, ..., end_letter for the given question. There may be one or more correct answers for each question. Answer by listing the letters of the correct answers separated by commas. Question: question Choices: A) option1 B) option2 C) option3 ... Answer: Multiple choice — multiple answers Prompt 2 SLOVAK Úloha: Vyber všetky správne odpovede A, B, C, ..., end_letter na nasledujúcu otázku. Odpovedz zoznamom písmen správnych odpovedí oddelených čiarkami. Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH Task: Select all the correct answers A, B, C, ..., end_letter for the following question. Answer by listing the letters of the correct answers. Question: question Choices: A) option1 B) option2 C) option3 ... Answer: Multiple choice — multiple answers Prompt 3 SLOVAK Predložím ti otázky a možné odpovede A, B, C, ..., end_letter a tvojou úlohou je vybrať všetky správne odpovede. Odpovedz zoznamom písmen správnych odpovedí oddelených čiarkami. Otázka: question Možnosti: A) option1 B) option2 C) option3 ... Odpoveď: ENGLISH I will present you with questions and possible answers A, B, C, ..., end_letter, and your task is to select all the correct answers. Answer by listing the letters of the correct answers. Question: question Choices: A) option1 B) option2 C) option3 ... Answer: M.12 Question answering — with context Question answering — with context Prompt 1 SLOVAK Odpovedz na nasledujúcu otázku priamo len na základe uvedeného kontextu, bez vysvetľovania, v slovenčine. Kontext: context Otázka: question Odpoveď: ENGLISH Answer the following question directly only based on the given context, without explanation, in Slovak. Context: context Question: question Answer: Question answering — with context Prompt 2 SLOVAK Úloha: Nájdi odpoveď na nasledujúcu otázku len na základe uvedeného kontextu. Odpoveď musí byť v slovenčine. Kontext: context Otázka: question Odpoveď: ENGLISH Task: Find the answer to the question only based on the given context. The answer has to be in Slovak. Context: context Question: question Answer: Question answering — with context Prompt 3 SLOVAK Kontext: context Odpovedz na nasledujúcu otázku v slovenčine: question Odpoveď: ENGLISH Context: context Answer the following question in Slovak: question Answer: M.13 Question answering — without context Question answering — without context Prompt 1 SLOVAK Odpovedz na nasledujúcu otázku priamo, bez vysvetľovania, v slovenčine. Otázka: question Odpoveď: ENGLISH Answer the following question directly, without explanation, in Slovak. Question: question Answer: Question answering — without context Prompt 2 SLOVAK Úloha: Nájdi odpoveď na nasledujúcu otázku. Odpoveď musí byť v slovenčine. Otázka: question Odpoveď: ENGLISH Task: Find the answer to the question. The answer has to be in Slovak. Question: question Answer: Question answering — without context Prompt 3 SLOVAK Odpovedz na nasledujúcu otázku v slovenčine: question Odpoveď: ENGLISH Answer the following question in Slovak: question Answer: M.14 Chiby — grammaticality Chiby — grammaticality Prompt 1 SLOVAK Skontroluj, či je daná veta gramaticky správna alebo nesprávna. Veta nie je napísaná správne pokiaľ obsahuje pravopisné chyby (napríklad chýbajúca diakritika, písanie veľkých a malých písmen), morfologické chyby (napríklad skloňovanie) alebo lexikálne chyby (napríklad čechizmy). Musí byť napísaná gramaticky správne. Ignoruj medzery navyše, nemeň čiarku za bodkočiarku a ani nepoužívaj dlhú pomlčku (---). Daj pozor na správne využitie pomlčky (--) a spojovníka (-). Ak je veta nesprávna napíš „nie“ a ak je veta správna napíš „áno“. Veta: sentence Odpoveď:" ENGLISH Check whether the sentence is grammatically correct or incorrect. A sentence is not written correctly if it contains spelling errors (such as missing diacritics or incorrect capitalisation), morphological errors (such as incorrect declension), or lexical errors (such as Czechisms). It must be written grammatically correctly. Ignore extra spaces, do not change commas to semicolons, and do not use an em dash (---). Pay attention to the correct use of the en dash (--) and the hyphen (-). If the sentence is incorrect, write "no", if the sentence is correct, write "yes". Sentence: sentence Answer: Chiby — grammaticality Prompt 2 SLOVAK Tvojou úlohou je napísať, či je veta gramaticky správna (odpovedz slovom „áno“) alebo nesprávna (odpovedz slovom „nie“). Ignoruj medzery navyše a nemeň čiarku za bodkočiarku. Nepoužívaj dlhú pomlčku (---). Používaj iba pomlčku (--) a spojovník (-). Veta: sentence Odpoveď: ENGLISH Your task is to indicate whether each sentence is grammatically correct (answer "yes") or incorrect (answer "no"). Ignore extra spaces and do not change commas to semicolons. Do not use an em dash (---). Use only an en dash (--) or a hyphen (-). Sentence: sentence Answer: Chiby — grammaticality Prompt 3 SLOVAK Veta je gramaticky správna ak neobsahuje žiadne typy chýb (napríklad pravopisné, morfologické, štylistické, lexikálne a iné). Do chýb nerátaj medzery navyše a nezamieňaj čiarku s bodkočiarkou. Nepoužívaj dlhú pomlčku (---). Používaj iba pomlčku (--) a spojovník (-). Pokiaľ je veta správna, napíš „áno“ a pokiaľ je nesprávna, napíš „nie“ . Veta: sentence Odpoveď: ENGLISH A sentence is grammatically correct if it contains no errors (such as spelling, morphological, stylistic, lexical or other errors). Do not consider extra spaces to be errors, and do not swap comma for a semicolon. Do not use the em dash (---). Use only the en dash (--) or the hyphen (-). If the sentence is correct, write "yes", if it is incorrect, write "no". Sentence: sentence Answer: M.15 SKJ1 — grammar correction SKJ1 — grammar correction Prompt 1 SLOVAK Skontroluj správnosť nasledujúcej vety a vyber možnosť, v ktorej sa nachádza správny tvar (letter_string). Odpovedz len písmenom správnej možnosti. Veta: sentence Možnosti: A) option1 B) option2 C) option3 D) option4 E) option5 Odpoveď: ENGLISH Check the correctness of the following sentence and select the option with the correct form (letter_string). Answer only with the letter of the correct option. Sentence: sentence Choices: A) option1 B) option2 C) option3 D) option4 E) option5 Answer: SKJ1 — grammar correction Prompt 2 SLOVAK Vyber gramaticky správnu verziu nasledujúcej vety z možností: letter_string. Odpovedz len písmenom správnej možnosti. Veta: sentence Možnosti: A) option1 B) option2 C) option3 D) option4 E) option5 Odpoveď: ENGLISH Select the grammatically correct version of the following sentence from the options: letter_string. Answer only with the letter of the correct option. Sentence: sentence Choices: A) option1 B) option2 C) option3 D) option4 E) option5 Answer: SKJ1 — grammar correction Prompt 3 SLOVAK Tvojou úlohou je vybrať gramaticky správny tvar nasledujúcej vety. Odpovedz len písmenom správnej možnosti: letter_string. Veta: sentence Možnosti: A) option1 B) option2 C) option3 D) option4 E) option5 Odpoveď: ENGLISH Your task is to choose the grammatically correct version of the following sentence. Answer only with the letter of the correct option: letter_string. Sentence: sentence Choices: A) option1 B) option2 C) option3 D) option4 E) option5 Answer: M.16 COPA — causal reasoning COPA — causal reasoning Prompt 1 SLOVAK Vzhľadom na predpoklad „premise“, a skutočnosť, že hľadáme príčinu tohto predpokladu, ktorá hypotéza sa zdá byť pravdepodobnejšia? Hypotéza 1: choice1 Hypotéza 2: choice2 Odpovedz len „1“ alebo „2“. Odpoveď: ENGLISH Given the premise "premise", and that we are looking for the cause of this premise, which hypothesis seems more plausible? Hypothesis 1: choice1. Hypothesis 2: choice2. Answer only with "1" or "2". Answer: COPA — causal reasoning Prompt 2 SLOVAK Na základe predpokladu „premise“ a na to, že nás zaujíma, čo sa stalo v dôsledku tohto predpokladu, ktorá hypotéza sa javí ako pravdepodobnejšia? Hypotéza 1: choice1 Hypotéza 2: choice2 Odpovedz len „1“ alebo „2“. Odpoveď: ENGLISH Given the premise "premise", and that we are wondering what happened as a result of this premise, which hypothesis seems more plausible? Hypothesis 1: choice1. Hypothesis 2: choice2. Answer only with "1" or "2". Answer: COPA — causal reasoning Prompt 3 SLOVAK Je uvedený textový popis situácie „premise“ a dva úryvky textu s popisom „choice1“ a „choice2“. Rozhodni, ktorý z týchto dvoch úryvkov je dôsledkom opísanej situácie. Odpovedz jedným číslom „1“ alebo „2“, bez ďalšieho textu. Odpoveď: ENGLISH A text description of the situation "premise" and two text fragments of the description "choice1" and "choice2" are given. Decide which of the two fragments is a consequence of the described situation? Answer with one number "1" or "2", without any additional text. Answer: M.17 Word analogies Word analogies Prompt 1 SLOVAK Doplň slovo tak, aby sa analogicky hodilo k predchádzajúcim trom slovám. Napíš iba chýbajúce slovo, nič iné. concept1 - concept2 concept3 - ? Odpoveď: ENGLISH Fill in the word so that it fits analogically with the preceding three words. Output only the missing word, nothing else. concept1 - concept2 concept3 - ? Answer: Word analogies Prompt 2 SLOVAK Doplňte analógiu. Na základe vzťahu medzi prvým párom slov uplatnite rovnaký vzťah a nájdite chýbajúce slovo. Napíšte iba chýbajúce slovo, nič iné. concept1 ↔ concept2 concept3 ↔ ? Odpoveď: ENGLISH Complete the analogy. Given the relationship between the first pair of words, apply the same relationship to find the missing word. Output only the missing word, nothing else. concept1 ↔ concept2 concept3 ↔ ? Answer: Word analogies Prompt 3 SLOVAK Dostanete slovnú analógiu v nasledujúcom formáte: A - B C - ? Vašou úlohou je zistiť vzťah medzi slovami A a B a rovnaký vzťah uplatniť na slovo C, čím získate chýbajúce slovo. Napíšte iba chýbajúce slovo, nič iné. concept1 - concept2 concept3 - ? Odpoveď: ENGLISH You are given a word analogy in the following format: A - B C - ? Your task is to identify the relationship between A and B, then apply that same relationship to C to produce the missing word. Output only the missing word, nothing else. concept1 - concept2 concept3 - ? Answer: