Paper deep dive
ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks
Jan-Philipp Schmidt
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/26/2026, 5:26:30 PM
Summary
ActuBench is a multi-agent LLM pipeline designed for the automated generation and evaluation of actuarial assessment items aligned with the International Actuarial Association (IAA) Education Syllabus. The system utilizes four specialized LLM roles: Agent A (drafting), Agent B (distractor construction), Agent C (independent verification), and an Auxiliary Agent (summarization and labeling). The pipeline employs a bounded one-shot repair loop driven by Agent C's verification results to ensure quality. The research evaluates 50 language models across two benchmarks: a 100-item Multiple-Choice Question (MCQ) set and a 100-item open-ended set scored by an LLM judge. Key findings highlight the necessity of multi-agent verification, the high performance of locally-hosted open-weights models (like Gemma 4 and Cerebras-hosted 120B models) on the cost-performance Pareto front, and the divergence in model rankings between MCQ and LLM-as-Judge evaluation modes.
Entities (10)
Relation Signals (7)
ActuBench â alignedwith â International Actuarial Association
confidence 100% ¡ aligned with the International Actuarial Association (IAA) Education Syllabus.
ActuBench â uses â Agent B
confidence 100% ¡ one agent drafts items, one constructs distractors...
ActuBench â uses â Agent C
confidence 100% ¡ a third independently verifies both stages and drives bounded one-shot repair loops
ActuBench â uses â Agent A
confidence 100% ¡ The pipeline separates four LLM roles by adapter: one agent drafts items...
ActuBench â uses â Auxiliary Agent
confidence 100% ¡ and a cost-optimized auxiliary agent handles Wikipedia-note summarization
Agent C â verifies â Agent B
confidence 100% ¡ Agent C then verifies those and, on failure, triggers a single-shot Agent B repair
Agent C â verifies â Agent A
confidence 100% ¡ Agent C then verifies the item and, on a failure verdict, Agent A produces a single-shot repair
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present ActuBench, a multi-agent LLM pipeline for the automated generation and evaluation of advanced actuarial assessment items aligned with the International Actuarial Association (IAA) Education Syllabus. The pipeline separates four LLM roles by adapter: one agent drafts items, one constructs distractors, a third independently verifies both stages and drives bounded one-shot repair loops, and a cost-optimized auxiliary agent handles Wikipedia-note summarization and topic labelling. The items, per-model responses and complete leaderboard are published as a browsable web interface at this https URL, allowing readers and practitioners to inspect individual items without a repository checkout. We evaluate 50 language models from eight providers on two complementary benchmarks -- 100 empirically hardest multiple-choice items and 100 open-ended items scored by an LLM judge -- and report three headline findings. First, multi-agent verification is load-bearing: the independent verifier flags a majority of drafted items on first pass, most of which the one-shot repair loop resolves. Second, locally-hosted open-weights inference sits on the cost-performance Pareto front: a Gemma~4 model running on consumer hardware and a Cerebras-hosted 120B open-weights model dominate the near-zero-cost region, with the latter within one item of the top of the leaderboard. Third, MCQ and LLM-as-Judge rankings differ meaningfully: the MCQ scaffold inflates the performance ceiling, and Judge-mode evaluation is needed to discriminate at the frontier.
Tags
Links
- Source: https://arxiv.org/abs/2604.20273v1
- Canonical: https://arxiv.org/abs/2604.20273v1
Trouble viewing inline? Open PDF directly â
Full Text
61,765 characters extracted from source content.
Expand or collapse full text
ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks Jan-Philipp Schmidt TH KĂśln, Institut fĂźr Versicherungswesen (ivwKĂśln) jan-philipp.schmidt@th-koeln.de April 23, 2026 Abstract We present ActuBench, a multi-agent LLM pipeline for the automated generation and evaluation of advanced actuarial assessment items aligned with the International Actuarial Association (IAA) Education Syllabus. The pipeline separates four LLM roles by adapter: one agent drafts items, one constructs distractors, a third independently verifies both stages and drives bounded one-shot repair loops, and a cost-optimized auxiliary agent handles Wikipedia- note summarization and topic labelling. The items, per-model responses and complete leaderboard are published as a browsable web interface athttps://actubench.de/en/, allowing readers and practitioners to inspect individual items without a repository checkout. We evaluate 50 language models from eight providers on two complementary benchmarks â 100 empirically hardest multiple-choice items and 100 open-ended items scored by an LLM judge â and report three headline findings. First, multi-agent verification is load-bearing: the independent verifier flags a majority of drafted items on first pass, most of which the one-shot repair loop resolves. Second, locally-hosted open-weights inference sits on the costâperformance Pareto front: a Gemma 4 model running on consumer hardware and a Cerebras-hosted 120B open-weights model dominate the near-zero-cost region, with the latter within one item of the top of the leaderboard. Third, MCQ and LLM-as-Judge rankings differ meaningfully: the MCQ scaffold inflates the performance ceiling, and Judge-mode evaluation is needed to discriminate at the frontier. 1 Introduction Actuarial science is one of the most technically demanding professional domains: it requires precise mathematical reasoning under stated assumptions, deep familiarity with insurance and financial theory, sensitivity to regulatory frameworks, and the ability to trade off computation, approximation and model risk. Internationally, the body of knowledge expected of a qualified actuary is codified by the International Actuarial Association (IAA) Education Syllabus, 1 which covers life insurance, non-life insurance, pensions, health insurance, enterprise risk management, and actuarial data science through more than a hundred learning objectives. These objectives underwrite the qualification examinations of national actuarial bodies such as the Deutsche Aktuarvereinigung (DAV), the Society of Actuaries (SOA) and the Institute and Faculty of Actuaries (IfoA). Constructing advanced assessment material for this body of knowledge â whether multiple- choice items, computational problems, or open-ended reasoning tasks â is expensive. Skilled subject-matter experts must craft questions that are unambiguous, internally consistent, grounded in current knowledge, and calibrated for difficulty; the process is slow even for a single examination committee. Meanwhile, language models have made rapid progress on general and domain- specific reasoning benchmarks [11, 15, 16], raising the question of how capable the current frontier 1 https://actuaries.org/app/uploads/2025/06/20250525_EducationSyllabus_Final.pdf 1 arXiv:2604.20273v1 [cs.AI] 22 Apr 2026 is on actuarial tasks specifically and what it would take to measure that capability reproducibly. Existing actuarial-LLM studies either survey use-cases without benchmarking [1], target a narrow quantitative subdomain [9], or benchmark Chinese-language insurance knowledge [26]; to our knowledge no English-language LLM benchmark aligned with the IAA Education Syllabus exists. We address this with ActuBench: a multi-agent LLM pipeline that generates IAA-aligned assessment items from learning objectives and Wikipedia-grounded notes, verifies them with an independent verifier agent, and evaluates language models on the resulting pool in two complementary modes. Our contributions are fivefold. 1.A multi-agent generation pipeline with an independent verifier. Four LLM roles are distinguished by adapter: Agent A drafts items, Agent B constructs distractors, Agent C independently verifies both, and a cost-optimized auxiliary agent handles Wikipedia-note summarization and topic-label assignment. Agent C never contributes generated content; it drives bounded repair loops triggered by its verdicts. This verifier-first role separation is the pipelineâs principal methodological differentiator (section 3.4). 2.A dual evaluation protocol: MCQ and LLM-as-Judge. We evaluate each model both in a classical four-option MCQ setting and in an open-ended setting where a judge LLM scores the evaluateeâs free-text response against the known correct answer. The two views produce meaningfully different rankings of the same models (section 5.3). 3.A live, web-accessible benchmark viewer athttps://actubench.de/en/. The 100 MCQ items, 100 Judge items, and every model response are browsable and searchable through a web UI, allowing readers and practitioners to inspect individual items and per-model answers without a GitHub checkout. 4. An empirical first evaluation of 50 language models. We evaluate 50 models spanning eight providers â Anthropic, Google, OpenAI, xAI, DeepSeek, Mistral, Cohere, and open-weights hosted endpoints including locally-run Gemma â on 100 empirically hardest MCQ items and 100 open-ended Judge items (section 4.3). 5. Evidence that locally-hosted open-weights inference is on the costâperformance Pareto front. A Cerebras-hosted open-weights 120B-parameter model reaches 97 % MCQ accuracy on the hardest items at near-zero cost; a Gemma 4 model running on a consumer GPU via Ollama reaches 85 % at zero marginal cost. At the paid end of the scale, per-answer frontier accuracy can be obtained at an order-of-magnitude lower cost than from the flagship reasoning-mode variants (section 5.1). The remainder of the paper is structured as follows. Section 2 reviews related work. Section 3 describes the generation pipeline. Section 4 specifies the two benchmark modes and the item- selection protocol. Section 5 reports empirical results along three axes: costâperformance, sector-level accuracy, and MCQ-vs-Judge divergence. Section 6 discusses reasoning-mode variants, Agent-C repair statistics, and implications for practical LLM use in actuarial settings. Section 7 and section 8 close. 2 Background and Related Work ActuBench sits at the intersection of six research threads: automatic question generation, LLM benchmark design, LLM-as-judge evaluation, multi-agent LLM pipelines, benchmark contamination, and the still-thin literature on LLMs in actuarial science. 2 2.1 Automatic Question Generation Automatic question generation (AQG) has been studied for over two decades. Early work on assessment-oriented AQG used syntactic transformations on parse trees combined with statistical ranking, exemplified by Heilman and Smith [10]âs seminal factual-QG system. Neural sequence-to-sequence models extended this line to learnt question generators, with Du, Shao, and Cardie [5] training a seq2seq model on sentence-question pairs from SQuAD (optionally conditioned on paragraph context). Kurdi et al. [14] provide the authoritative systematic review of AQG for educational purposes across rule-based, template, statistical and neural eras. With instruction-tuned LLMs, generating domain multiple-choice questions with correct answers, plausible distractors and rationales became feasible at scale; Elkins et al. [8] evaluated InstructGPT-generated educational questions in a teacher-expert study and found high relevance, grammaticality and answerability, with most items requiring only minor or no edits. That result sharpens the remaining quality-control question: not whether LLMs can draft plausible items, but whether automated verification can catch the residual factual and difficulty-calibration errors without teacher review. ActuBenchâs Agent-A/Agent-B split with an independent Agent-C verifier is a direct answer to that question. 2.2 LLM Benchmarks A broad family of LLM benchmarks covers general academic knowledge (MMLU [15]), math- ematical reasoning (MATH [11], DROP [7]), and cross-disciplinary graduate-level reasoning (HLE [16]). Domain-specific benchmarks include FinQA for numerical reasoning on financial documents [3] and various medical-licensing datasets. A recurring concern across this literature is that static benchmarks saturate quickly as models improve, motivating designs that retain only the items current models most frequently miss [16]; our empirically-hardest-100 construction (section 4.3) adopts this idea. 2.3 LLM-as-Judge Evaluation The LLM-as-judge paradigm was popularized by Zheng et al. [25] (MT-Bench and Chatbot Arena), who demonstrated that capable LLMs, particularly GPT-4, can approximate human preference scoring on open-ended tasks at above 80 % agreement. Kim et al. [13] trained Prometheus, an open-source 13-billion-parameter fine-tuned evaluator LLM, to produce fine- grained rubric-grounded verdicts with reference answers. Critical follow-up work documented systematic biases: Wang et al. [21] show that LLM judges exhibit pronounced positional bias in the sense that simply reordering candidate responses in the prompt can flip the quality ranking. For domains where a ground-truth correct answer exists (as in actuarial exam items), the judgeâs task is narrower than in preference-style evaluation: the judge must decide whether an extracted answer matches a known correct answer rather than rank competing responses. ActuBenchâs judge prompt (section 4.2) enforces this framing explicitly. 2.4 Multi-Agent LLM Pipelines Multi-agent LLM systems partition a task across roles with distinct prompts and, increasingly, distinct underlying models. Hong et al. [12] (MetaGPT) formalizes role specialization for software-engineering tasks; Wu et al. [22] (AutoGen) provides a domain-agnostic framework for multi-agent LLM conversation, with demonstrations across mathematics, retrieval-augmented generation, optimization, chess and coding. Chen et al. [2] demonstrate emergent collaboration in open-ended problem solving. Du et al. [6] introduce multi-agent debate as a mechanism to reduce hallucinations and improve factuality; the empirical pattern they report â agents that individually err but converge to correct answers under mutual critique â implicitly supports the more general verification-generation asymmetry we rely on in section 3.4. ActuBench 3 applies the same role-specialization principle but restricts the scope to a single, closed content- generation task: one adapter drafts, another audits distractors, a third independently verifies. The restriction yields a well-scoped, bounded-cost pipeline in which each agentâs prompt is short and each verdict auditable â distinct from the open-ended negotiation of debate-style systems. 2.5 Benchmark Contamination A fast-growing literature documents data contamination as a central threat to static LLM bench- marks: popular benchmark items often appear in pre-training corpora and model performance on them may not reflect genuine generalization [4, 19, 24, 27]. Our design addresses this in two ways: the pipeline can generate fresh items on demand from new learning objectives, and the item pool released athttps://actubench.de/en/is a curated subset rather than a static dump. 2.6 Actuarial AI Actuarial AI has historically focused on classical machine learning on tabular and time-series data: Richmanâs surveys [17, 18] and WĂźthrich and Merz [23]âs textbook treat deep learning for mortality, reserving and non-life pricing, while Troxler and Schelldorfer [20] demonstrate transformer-based NLP for claims descriptions. LLM-specific studies have only recently ap- peared. Balona [1] is framed as a practitioner survey of LLM use-cases in insurance rather than an evaluation benchmark. Hao et al. [9] build quantitative QA pairs from reinsurance training materials and fine-tune Llama-2 variants; their scope is narrow and their item set non-public. The concurrent Chinese-language CUFEInse benchmark [26] evaluates eleven LLMs on 14,430 insurance-operations questions spanning theory, industry, safety, agents and logical rigor. CUFEInse and ActuBench are complementary rather than overlapping: CUFEInse targets insurance operations on Chinese-language industry content at large scale, while ActuBench targets IAA-syllabus-aligned actuarial reasoning on English-language content with a smaller but methodologically controlled item set and an independent-verifier generation pipeline. To our knowledge, ActuBench is the first English-language actuarial LLM benchmark aligned with the IAA Education Syllabus. 3 Generation Pipeline 3.1 Overview ActuBench processes each assessment item through a linear sequence of stages, each of which persists its output to the database before the next stage begins. Figure 1 shows the full workflow. Four distinct LLM roles are distinguished, each bound to its own adapter and potentially its own underlying model: â˘Agent A drafts content (keyword extraction, item draft, and â when required â item repair). Agent A is typically the strongest and most expensive adapter in the pipeline, and in the generation run used for this paper it was bound to a reasoning-mode model (claude-sonnet-4-6:thinking). â˘Agent B specializes in distractor design and distractor repair. Agent B does not need the deeper reasoning of Agent A â the task is more constrained and creative rather than multi-step â so in the generation run used here it was bound deliberately to a non-reasoning adapter (claude-sonnet-4-6). ⢠Agent C is the independent verifier. It performs two dedicated verification stages (once on the item stem and correct answer, once on the four-option ensemble) and never contributes 4 generated content. Agent C is bound to its own adapter, allowing a separate, potentially more critical model to be used for verification (here o3:reasoning). ⢠The auxiliary agent handles all cost-sensitive ancillary work: summarizing Wikipedia extracts into structured notes, and later assigning topic labels over the actuarial sectors. This adapter is always a cheap, fast model (heregpt-5.4-mini); the cost savings of keeping these two stages off the primary generator add up across hundreds of items. Generation proceeds top-to-bottom in Figure 1. A learning objective from the International Actuarial Association (IAA) Education Syllabus is turned into a Wikipedia search query by Agent A; relevant article excerpts are fetched through the MediaWiki API; the auxiliary agent summarizes them into bullet-point notes; Agent A uses the notes plus the learning objective to draft an item. Agent C then verifies the item and, on a failure verdict, Agent A produces a single-shot repair that is re-checked. Agent B then constructs three distractors; Agent C verifies those and, on failure, triggers a single-shot Agent B repair with its own re-check. Finally the auxiliary agent assigns topic labels to the finished item and the item enters the evaluation pool. IAA learning objectiveKeywordsWikipedia fetch Wiki notesItem draft Verify item Repair Re-check Distractors Verify distractors Repair Topic labels Re-checkBenchmark ok fail ok fail â Agent A (keywords, item draft, repair)â Agent B (distractors, repair)â Agent C (verification) â Auxiliary (wiki notes, topic labels)â External API Figure 1: ActuBench generation pipeline. Four LLM roles plus an external Wikipedia API cooperate to produce one assessment item. Agent A drafts (blue); Agent B builds distractors (teal); Agent C verifies both independently (gold); a cost-optimized auxiliary handles note summarization and topic labelling (green). Solid arrows are the primary flow; dashed red-labeled arrows are repair branches taken only on verification failure. 3.2 Agent A Agent A carries the two stages that benefit most from strong underlying reasoning: item drafting and repair. Each stage emits a structured JSON output that is parsed and cached. Keyword extraction. Given an IAA learning objective (code, textual description, subject and topic), Agent A produces a small JSON object with primary search terms, synonyms and abbreviations, disambiguation exclusions, and between one and three candidate search strings. Requiring explicit disambiguation-guard terms (e.g., âexclude: option pricing in financeâ when the objective is on actuarial option costs) sharpens downstream Wikipedia retrieval measurably. Item drafting. From the learning-objective text, the sampled difficulty archetype, and the Wikipedia notes produced by the auxiliary agent, Agent A produces the question stem, the single correct answer, and a detailed rationale justifying why that answer is correct. The prompt enforces several hard constraints: English-language only, single best answer, no multiple defensible answers under reasonable interpretation, explicit statement of any assumption needed for uniqueness, and no reliance on parametric knowledge that cannot be traced to the Wikipedia notes. The correct answer occupies a fixed internal slot by construction; randomization into an MCQ-presentation order is done only at evaluation time. 5 Repair. When Agent C returns a non-ok verdict on the item, Agent A is re-invoked with the item draft and Agent Câs failure report, and produces a revised item that attempts to resolve the flagged issue. This is a single-shot repair, not a negotiation: the revised item then goes through a re-check, and if that also fails the item is persisted with a failure flag but not discarded. 3.3 Agent B Agent B specializes in distractor design. Given the question stem, the correct answer with its rationale, and the Wikipedia notes, Agent B produces three distractors. Each distractor is required to be incorrect given the same notes and assumption set that make A correct, individually plausible to a learner at the target difficulty, and distinguishable from the other distractors. Each distractor is accompanied by a short free-text rationale describing the specific misconception or computational slip it is meant to probe. This prompt design follows the distractor-as-misconception-probe tradition in educational assessment [10, 14], adapted to the LLM-generation setting. On Agent Câs failure verdict for the distractor ensemble, Agent B is re-invoked once with the failure report and produces a revised set, which is then re-checked. 3.4 Agent C â Independent Verification Agent C is the principal methodological differentiator of ActuBench. It performs two verification stages â one on the item stem and correct answer, one on the full four-option ensemble â and returns an ok/fail verdict with a structured failure report. Unlike Agents A and B, Agent C never contributes generated content; it only consumes a completed draft. A separate LLM adapter is bound to Agent C so that a different model can be used for verification than for generation. This role and adapter separation was driven by two converging observations. First, there is evidence that LLMs can be more accurate at judging a candidate solution than at producing one: Zheng et al. [25] show that strong LLMs agree with human judges on open-ended tasks at above 80 %, and the convergence pattern documented in multi-agent debate [6] â where individually-erring agents converge to correct answers under mutual critique â is consistent with the same asymmetry. This is the same asymmetry exploited in retrieval- augmented generation, multi-agent debate, and outcome-supervised training: it is usually easier to recognize a flaw than to avoid one. For item construction, the implication is direct: having the same agent both draft an item and declare it correct is prone to shared blind spots, while routing verification through a model that has not seen the draft-in-progress breaks this shared-failure mode. Empirically, Agent C flags more than three in five items Agent A produced in our run as requiring item-level repair (section 6.2), which is substantially higher than any self-verification rate we observed in preliminary experiments. Second, verification is a well-scoped task with structured output, for which smaller or more reasoning-oriented models often outperform larger general-purpose generators in our domain. Decoupling the verifier adapter from the generator lets us pair a strong generator with an efficient verifier, and lets us swap either component without restructuring the pipeline. The role separation is analogous to the generatorâcritic split in multi-agent coding frameworks [12, 22], specialized here to content verification on a single generation task rather than to open-ended problem solving. Item verification. Given the learning objective, the difficulty archetype, the Wikipedia notes, the question stem, the correct answer, and the correct-answer rationale, Agent C returns a verdictokâ true, falsewith a structured failure report onok=false. The report enumerates: ambiguity (can another interpretation of the stem make another answer defensible?), factual error (does the correct answer conflict with the notes?), and underspecification (is a required assumption missing?). This report is both stored in the database and passed to Agent Aâs repair stage as the prompt context. 6 Distractor verification. Given all four options with their rationales, Agent C checks that the distractors are non-duplicative, individually plausible, individually incorrect under the itemâs stated assumption set, and probe distinct misconceptions. On failure, the report identifies which distractor(s) are problematic and for what reason, and is consumed by Agent Bâs repair stage to produce a revised set. Repair loop structure. Each verification stage admits at most one repair iteration. The re-check that follows each repair is final: if it still fails, the item is persisted with the failure flag and continues to the topic-label stage. This bounded-retry design trades a modest yield loss (about 15 % of items in our run were still flagged after repair; see section 6.2) for a predictable upper bound on per-item cost. 3.5 Auxiliary Agent The auxiliary agent handles two cost-sensitive ancillary stages: Wikipedia-note summarization (after the MediaWiki fetch) and topic-label classification (after the item is finalized). The auxiliary adapter is always bound to a cheap, fast model: the two stages are well-structured and do not benefit from frontier reasoning, and the savings compound over hundreds of generated items. Note summarization produces a fixed bullet-list schema grouping extract content into named facts with provenance; topic labelling assigns each item zero or more of six actuarial- sector labels (life, non-life, AFIR-ERM, ADS-AI, health, pension). The labels are not mutually exclusive â an item on longevity-linked non-life hybrid products may carry both Life and Non-life â and feed the sector-level analysis in section 5.2. 4 Benchmark Methodology We evaluate language models on the items produced by the pipeline of section 3 in two com- plementary modes. The two benchmarks operate on disjoint item sets drawn from the same generation pipeline: 100 items for the MCQ benchmark and 100 different items for the Judge benchmark, so that no model sees the same item in both evaluation modes. 4.1 MCQ Evaluation In MCQ mode, the evaluated model is shown the question stem together with the four options AâD and is instructed to respond with a single letter. A deterministic per-run pseudo-random shuffle decides, for each item, which of the four options receives label A, B, C, or D at presentation time (the correct answer is stored internally in a fixed slot and never appears in the same letter position across runs). The shuffle map is persisted with every answer row for later analysis. The prompt is deliberately terse to minimize format-following confounds: System: You are taking a multiple-choice actuarial exam. Select the single best answer. Respond with ONLY the letter A, B, C, or D â no explanation, no punctuation. User: Question: stem A) . . . B) . . . C) . . . D) . . . The response is parsed by extracting the first token inA, B, C, D; if no such token is present the answer is recorded as unparseable and counted as incorrect. Token counts and per-call cost are computed from the providerâs reported usage and the pricing table at call time. 4.2 Judge Evaluation Judge mode removes the multiple-choice scaffold entirely. The evaluated model sees only the question stem and is asked to answer in free text, showing its reasoning and then stating its final answer: 7 Evaluatee system: You are taking an actuarial exam. Answer the following question as accurately and concisely as possible. Show your reasoning briefly, then state your final answer clearly. A second LLM â the judge â then assesses the free-text response against the known correct answer. The judge prompt returns a structured JSON verdict with four fields: the final answer extracted from the response, a short reasoning note, a yes/no correctness verdict, and a 0â100 confidence score (the evaluateeâs self-reported confidence where available, 100 otherwise). The judgeâs instruction restricts its scope to comparing the extracted answer to the correct answer on meaningful differences, accepting small numerical tolerances, and explicitly forbids re-solving the problem or arguing for an alternative answer. This design follows the LLM-as-judge paradigm established in [13, 25] while tightening it for a domain in which the correct answer is authoritative rather than preferential. The two benchmarks produce different views of the same competence. MCQ mode tests recognition-plus-elimination â a model that cannot derive the correct answer but can identify implausible options may still succeed. Judge mode tests derivation from scratch: the model must produce a correct answer without the four-option scaffold, and a capable judge then checks it. We report both and contrast them in section 5.3. 4.3 Item Selection for the Benchmarks The generation pipeline produces items across five hard difficulty archetypes â quantitative calculation, assumption sensitivity, conceptual inversion, edge case / boundary and multi- step logic â sampled with a weight schedule that favors quantitative calculation. Pre-tests showed that items drawn from easier archetypes (terminology precision, consistency checks) are solved nearly uniformly by frontier models and therefore have little discriminative value; we therefore exclude those archetypes from the evaluation pool. Pre-tests further showed that the quantitative-calculation archetype is consistently the most discriminating among the five hard archetypes. For this study we construct two benchmarks of one hundred items each, disjoint by construc- tion. MCQ benchmark (100 items). We first generated an internal pool of 200 items spanning the five hard archetypes. From that pool we retain the 100 items on which the collective accuracy across the evaluated models was lowest â the empirically-hardest half. This empirically- hardest-Nconstruction is a standard move in benchmark design [15, 16] because static frontier benchmarks saturate quickly under LLM progress: retaining only the items the communityâs current models most frequently miss preserves discriminative power at the top. The resulting set contains 46 quantitative-calculation items, 17 conceptual-inversion items, 13 assumption- sensitivity items, 13 edge-case items and 11 multi-step-logic items. Cross-model mean accuracy on the retained items ranges from 7.8 % (hardest) to 94.1 % (easiest). Judge benchmark (100 items). The Judge benchmark comprises 100 items drawn from a separate pool of approximately 400 items generated by the same pipeline. We restrict the Judge benchmark to the quantitative-calculation archetype because the pre-tests cited above identified it as the most discriminating archetype among the five hard categories, and because the open-ended format most directly rewards genuine derivation on computational tasks. The Judge items are disjoint from the MCQ items. 4.4 Model Selection We evaluate 50 models spanning eight providers: Anthropic, Google, OpenAI, xAI, DeepSeek, Mistral, Cohere, and an open-weights family of hosted endpoints (Groq, Cerebras) plus a locally-hosted Ollama runtime. Where a provider exposes both a standard and a reasoning-mode variant of the same underlying model (e.g.claude-opus-4-6vs.claude-opus-4-6:thinking, 8 ProviderModel$/1M in $/1M out Type anthropic claude-opus-4-6:thinking5.00025.000 thinking openaigpt-5-mini0.1251.000 dense openaio3:reasoning2.0008.000 thinking openaio4-mini:reasoning1.1004.400 thinking cerebrasgpt-oss-120bâ open-source googlegemini-3.1-pro-preview2.00012.000 dense googlegemini-2.5-pro:thinking1.25010.000 thinking groqopenai/gpt-oss-120b0.1500.600 open-source openaigpt-51.25010.000 dense openaio3-mini:reasoning1.1004.400 thinking anthropic claude-sonnet-4-6:thinking3.00015.000 thinking groqopenai/gpt-oss-20b0.0750.300 open-source openaigpt-5-nano0.0500.400 dense xaigrok-3-mini:reasoning0.3000.500 thinking anthropic claude-sonnet-4-63.00015.000 dense googlegemini-2.5-pro1.25010.000 dense googlegemma-4-31b-it0.1300.380 open-source xaigrok-3-mini0.3000.500 dense anthropic claude-opus-4-2025051415.00075.000 dense coherecommand-a-reasoning-08-20252.50010.000 thinking googlegemini-3-flash-preview0.5003.000 dense xaigrok-43.00015.000 dense anthropic claude-opus-4-65.00025.000 dense ollamagemma4:latest0.0000.000 open-source anthropic claude-sonnet-4-20250514:thinking3.00015.000 thinking googlegemma-4-26b-a4b-it0.0800.350 open-source anthropic claude-sonnet-4-202505143.00015.000 dense googlegemini-2.5-flash0.1500.600 dense googlegemini-2.5-flash:thinking0.1500.600 thinking deepseekdeepseek-reasoner:reasoning0.2800.420 thinking googlegemini-3.1-flash-lite-preview0.2501.500 dense groqmeta-llama/llama-4-scout-17b-16e-instruct0.1100.340 open-source openaigpt-5.42.50015.000 dense openaigpt-5.21.75014.000 dense anthropic claude-haiku-4-5-202510011.0005.000 dense openaigpt-4.12.0008.000 dense xaigrok-33.00015.000 dense coherecommand-a-03-20252.50010.000 dense deepseekdeepseek-chat0.2800.420 dense groqllama-3.3-70b-versatile0.5900.790 open-source openaigpt-5.4-mini0.7504.500 dense mistralmistral-large-25120.5001.500 dense mistralmistral-large-latest0.5001.500 dense openaigpt-4.1-mini0.4001.600 dense openaigpt-4.1-nano0.1000.400 dense mistralmistral-small-latest0.1500.600 open-source openaigpt-5.4-nano0.2001.250 dense groqqwen/qwen3-32b0.2900.590 open-source groqllama-3.1-8b-instant0.0500.080 open-source cerebrasllama3.1-8b0.1000.100 open-source Table 1: Evaluated models. Per-million-token prices (USD) are the providerâs published rates at the time the benchmark was run. Type: dense for standard decoder models, thinking for reasoning-mode variants, open-source for models whose weights are publicly released. Five additional models are excluded due to adapter errors (section C). gemini-2.5-provs.gemini-2.5-pro:thinking), we include both to support the paired com- parison in section 6.1. Five further models were benchmarked but are excluded from all main tables and figures because adapter-level errors caused every answer to be recorded as unparseable; they are listed in section C. Table 1 summarizes the included set. Anthropic Claude is among the evaluated models and is also the tool used to draft this manuscript (see Acknowledgments); we disclose this dual role rather than excluding Claude from the evaluation, because its relative position on the benchmark is of independent scientific interest. 4.5 Reproducibility Two artefacts support replication. First, the 100 MCQ items and 100 Judge items used here, together with every benchmark answer (presented options, shuffle map, raw response, extracted answer, token counts, cost), are published as a browsable interface athttps://actubench.de/ en/. Readers can inspect any item and the responses each model gave to it without installing any software. Second, every number in this paper is derived from a single frozen snapshot of the generation database, so that re-running the reporting pipeline reproduces every plot and table deterministically. The generation and benchmark-runner code are available from the author on request. 9 5 Results All numbers in this section come from the frozen data snapshot described in section 4.5 and the 50 included models of section 4.4. Throughout this section, âMCQ accuracyâ refers to accuracy on the 100 MCQ items, and âJudge accuracyâ to accuracy on the 100 Judge items. 5.1 CostâPerformance Landscape Figure 2 shows each model as a numbered, provider-coloured point on a costâaccuracy plane. The horizontal axis is the total USD cost of running the full MCQ benchmark (log scale, spanning six orders of magnitude from near-zero for locally-hosted inference up to a few US dollars for the strongest reasoning-mode variants). The vertical axis is MCQ accuracy. Hollow markers are reasoning-mode variants; filled markers are standard models. Model names are listed in the legend under the figure to keep the plot area uncluttered; the Pareto relationship can be read off the scatter by eye. Three features of the landscape stand out. First, the accuracy ceiling is crowded: four models âclaude-opus-4-6:thinking,gpt-5-mini,o3:reasoningando4-mini:reasoningâ reach the top MCQ accuracy of 98 % (98/100 items correct), yet their total-run costs span a full order of magnitude, from about nine US cents up to over $1.50 for the same accuracy. On hard but structurally closed multiple-choice items, per-answer frontier performance can be obtained much more cheaply from the right non-flagship model than from the flagship. Second, two models strictly dominate at the zero-cost end of the scale: a hosted open-weights 120B-parameter model from Cerebras at 97 % MCQ accuracy at an extremely low cost (effectively a fraction of a US cent for the full 100-item run), and a locally-hosted Gemma 4 model running on consumer hardware at 85 % accuracy and zero marginal cost. The Cerebras endpoint lies within one item of the top of the leaderboard; the Ollama case shows that a medium-sized open-weights model running on consumer hardware achieves accuracy within 13 percentage points of the best paid option. Third, the space between the Pareto-dominant points at the zero-cost end and the top- accuracy cluster is thin. A reader might be tempted to interpret this as reasoning-mode inference being wasteful, but that overstates the case: the MCQ task as posed (recognition plus elimination) is not the hardest test these models can be given, and the compression at the ceiling is partly an artefact of it. The Judge-mode results below (section 5.3) decompress the top of the scale. 5.2 Accuracy by Actuarial Sector Each item in the MCQ benchmark is tagged by the auxiliary agent (section 3.5) with zero or more actuarial-sector labels: life, non-life, AFIR-ERM (actuarial approach for financial risks and enterprise risk management), ADS-AI (actuarial data science and AI), health, and pension. Labels are not mutually exclusive. Table 2 reports the mean accuracy per sector, pooled over all 50 included models. At the top of the ranking the sector-level accuracies are near-uniform: the five strongest models score within a couple of items of 100 % on every sector. Differentiation appears in the full distribution over all 50 models. Mean accuracy is highest on ADS-AI and lowest on non-life; the most statistically robust figure is AFIR-ERM, which carries the largest item count inside the benchmark and is the best estimate of how frontier and near-frontier LLMs handle advanced enterprise-risk and capital-management content. The uneven sector counts per MCQ benchmark are a direct artefact of the sampling over IAA learning objectives; a future study that rebalances the sample could sharpen the smaller-sample estimates. 10 10 â2 10 â1 10 0 Total benchmark-run cost (USD, log scale) 40 50 60 70 80 90 100 Accuracy (%) 1234 6 78910 111213 14 1516 1718 19 20 21 22 23 2526 27 28 29 30 313233 34 3536 37 38 39 4041 424344 45 46 47 48 49 50 Anthropic OpenAI Open / Cerebras / Groq Google xAI Cohere DeepSeek Mistral # Provider Model# Provider Model 1⌠Anthclaude-opus-4-6:thinking8⢠Openopenai/gpt-oss-120b 11⌠Anthclaude-sonnet-4-6:thinking12⢠Openopenai/gpt-oss-20b 15⢠Anthclaude-sonnet-4-624⢠Opengemma4:latest 19⢠Anthclaude-opus-4-2025051432⢠Openmeta-llama/llama-4-scout-17b-16e-instruct 23⢠Anthclaude-opus-4-640⢠Openllama-3.3-70b-versatile 25⌠Anthclaude-sonnet-4-20250514:thinking 48⢠Openqwen/qwen3-32b 27⢠Anthclaude-sonnet-4-2025051449⢠Openllama-3.1-8b-instant 35⢠Anthclaude-haiku-4-5-2025100150⢠Openllama3.1-8b 20⌠Cohere command-a-reasoning-08-20252⢠OAIgpt-5-mini 38⢠Cohere command-a-03-20253⌠OAIo3:reasoning 30⌠DSeekdeepseek-reasoner:reasoning4⌠OAIo4-mini:reasoning 39⢠DSeekdeepseek-chat9⢠OAIgpt-5 6⢠Googgemini-3.1-pro-preview10⌠OAIo3-mini:reasoning 7⌠Googgemini-2.5-pro:thinking13⢠OAIgpt-5-nano 16⢠Googgemini-2.5-pro33⢠OAIgpt-5.4 17⢠Googgemma-4-31b-it34⢠OAIgpt-5.2 21⢠Googgemini-3-flash-preview36⢠OAIgpt-4.1 26⢠Googgemma-4-26b-a4b-it41⢠OAIgpt-5.4-mini 28⢠Googgemini-2.5-flash44⢠OAIgpt-4.1-mini 29⌠Googgemini-2.5-flash:thinking45⢠OAIgpt-4.1-nano 31⢠Googgemini-3.1-flash-lite-preview47⢠OAIgpt-5.4-nano 42⢠Mistrmistral-large-251214⌠xAIgrok-3-mini:reasoning 43⢠Mistrmistral-large-latest18⢠xAIgrok-3-mini 46⢠Mistrmistral-small-latest22⢠xAIgrok-4 5⢠Opengpt-oss-120b37⢠xAIgrok-3 Figure 2: Costâperformance landscape on the MCQ benchmark. Each point is one evaluated model, coloured by provider; filled markers are standard models, hollow markers are reasoning- mode variants. Numbered points are identified in the legend above. 11 Sector#Items Mean accuracy ADS-AI230.870 Health20.790 Life240.777 Pension50.756 AFIR-ERM410.748 Non-life120.733 Table 2: Mean accuracy per actuarial sector on the MCQ benchmark, pooled over the 50 included models. Item counts per sector are uneven because each item may carry multiple labels and because the IAA syllabus is itself unevenly weighted across sectors. Small-sample sectors (health, pension) should be read with a correspondingly wider confidence band. 5.3 MCQ vs. Judge Divergence To contrast the two evaluation modes we plot each modelâs MCQ accuracy (on the 100-item MCQ benchmark) against its Judge accuracy (on the 100-item Judge benchmark) in fig. 3. The two item sets are disjoint by construction but both come from the same generation pipeline, with the Judge pool restricted to the most discriminating difficulty archetype (section 4.3). 0.00.20.40.60.81.0 MCQ accuracy 0.0 0.2 0.4 0.6 0.8 1.0 Judge accuracy MCQ vs. Judge (Ď=0.68,Ď=0.50, N=39) y = x Figure 3: Per-model accuracy on the 100-item MCQ benchmark (x-axis) versus the 100-item Judge benchmark (y-axis). The dotted line isy=x; points below the diagonal are models that lose accuracy when the four-option scaffold is removed. SpearmanĎand KendallĎover the N = 39 commonly evaluated models are reported in the figure title. 12 The picture is more nuanced than an informal âJudge is harderâ hypothesis would suggest. At the top of the MCQ ranking the two benchmarks diverge sharply: the best MCQ score is 98 %, the best Judge score 87 %, and several frontier models lose on the order of 10â15 p when the four-option scaffold is removed. For lower-tier models the sign of the gap frequently reverses: a number of smaller or cheaper models gain 10â20 p on Judge relative to their MCQ score. When a model cannot reliably solve the underlying computation, MCQ-style elimination exposes that more cleanly than the Judgeâs looser numerical-tolerance matching does, so the Judge can end up being the more forgiving instrument at the low end of the scale. The finding restated: the MCQ scaffold inflates the performance ceiling. At the top of the benchmark, where every strong model is near 100 % on MCQ, Judge mode produces a more spread-out ranking with a ceiling closer to 87 %; at the bottom of the benchmark, the judgeâs tolerant matching can be more forgiving than MCQâs one-letter-only scoring. The rank correlations reported in fig. 3 reflect exactly this: moderate agreement, not identity, between the two benchmarksâ orderings. Practically, this means MCQ-only actuarial benchmarks should not be read as a ranking of genuine reasoning capacity on open-ended problems. Judge-mode evaluation is a sharper instrument near the frontier and a differently-shaped instrument further down; both views are needed. 6 Discussion 6.1 Reasoning-Mode vs. Standard Variants Six provider/model pairs in our evaluation expose both a standard decoder variant and an explicit reasoning-mode (âthinkingâ) variant built on the same base model. Figure 4 shows the paired comparison on the MCQ benchmark. The aggregate picture is modest: across the six pairs, the mean accuracy gain from enabling the reasoning mode is about three and a half percentage points, at an average inference-cost multiple around two and a half. The distribution is skewed: one pair âclaude-opus-4-6versusclaude-opus-4-6:thinkingâ moves from 85 % to 98 % accuracy (+13 p) at roughly 4.7Ăthe cost; most other pairs move by one to four percentage points. One pair shows a slight regression (gemini-2.5-flash:thinkingscores 2 p below its non-thinking sibling on this set). Two implications follow. First, reasoning mode is not free performance: on a well-constructed hard-MCQ benchmark it delivers a few percentage points of accuracy per factor of two-to-three in cost, with high variance across model families. Second, the cost-aware reader of section 5.1 should not equate âreasoning modeâ with âPareto-dominantâ; the two Pareto-dominant models at the zero-cost end of the scale in our study are both standard open-weights models (gpt-oss-120b on Cerebras and Gemma 4 via Ollama). Where reasoning mode does dominate is at the frontier ceiling: five of the top six scores on the MCQ benchmark come from reasoning-mode variants. 6.2 Agent-C Repair Statistics Of the items Agent C reviewed during generation of the MCQ benchmark pool, roughly three in five were flagged on first pass at the item level (item stem + correct answer) and just under half were flagged at the distractor-ensemble level. These rates are substantially higher than any self-verification rate we observed when the same adapter that drafted an item also verified it in preliminary experiments â consistent with the literature on the verification-generation asymmetry [6, 25]. An in-distribution generator is a poor auditor of its own output. Repair is largely effective. Of the items flagged at the item level, roughly three-quarters pass the re-check after a single-shot Agent A repair and proceed normally; the remainder â around one in six items overall â are persisted with a failure flag. The distractor-level loop behaves comparably: most flagged distractor sets are repaired on a single Agent B pass, a minority persist the failure flag. In aggregate, a pipeline that discarded repair failures rather than marking and 13 0.840.860.880.900.920.940.960.981.00 Accuracy google/gemini-2.5-flash anthropic/claude-sonnet-4-20250514 anthropic/claude-opus-4-6 xai/grok-3-mini google/gemini-2.5-pro anthropic/claude-sonnet-4-6 non-thinking thinking Figure 4: Accuracy of six reasoning-mode (red) variants against their standard-decoder siblings (grey), on the MCQ benchmark. The dumbbell is ordered by non-thinking accuracy. The x-axis is zoomed to the region where the paired variants actually sit. Differences are typically small (one to four percentage points); the Anthropic Opus pair is the outlier at +13 p. keeping them would lose a non-trivial fraction of its output. The design decision to keep-and-flag rather than discard was made for dataset completeness and is re-visitable for studies that require perfectly verified subsets only. 6.3 Implications for Practical Actuarial LLM Use Three practical conclusions follow from our results. For multiple-choice or structured decision support, a small or mid-tier general-purpose model (for examplegpt-5-miniat roughly ten US cents per hundred answers, or a hosted open-weights endpoint at near-zero cost) delivers accuracy within two percentage points of the flagship reasoning variants on the hardest actuarial items we could construct. The extra money spent on the most expensive reasoning-mode models buys little additional accuracy on this task format. For open-ended reasoning and derivation, the picture reverses: the MCQ ranking compresses models at the top, while Judge-mode scoring reveals a roughly 15 p gap between the strongest frontier reasoning models (about 85 % Judge accuracy) and merely strong non-reasoning models (about 70 %). When the task actually requires deriving an answer rather than selecting one, reasoning-mode or frontier-generation models are cost-justified. For local deployment, the observation that Gemma 4 running via Ollama on a single consumer GPU reaches 85 % accuracy on the MCQ benchmark is material. A team that prefers â for data-governance, cost-control or latency reasons â to run inference locally is not paying a large accuracy penalty on structured actuarial knowledge; on open-ended reasoning the penalty is larger, and a hybrid architecture (local for MCQ-style classification, hosted frontier for derivation) is likely the right trade-off. 7 Limitations and Future Work Five limitations of the present study deserve explicit note. Statistical power. Each of our two benchmarks contains 100 items. A 5-point accuracy difference between two models at 90 % mean accuracy has a Wilson 95 % confidence interval of roughlyÂą6 p, which means small rank differences in the middle of our tables are not statistically 14 decisive. The MCQ-benchmark design choice of empirically-hardest-100 trades sample size for discriminative power at the ceiling; a larger study that re-ran the pipeline to produce, say, 400 items per benchmark would tighten the mid-table estimates at proportionately higher evaluation cost. Factual grounding via Wikipedia. The generation pipeline (section 3.2) uses Wikipedia as its sole factual anchor. This is a deliberate choice: Wikipedia is reproducibly fetchable, well-understood as a distribution, and its extracts can be versioned. But it under-covers two important regions of the IAA syllabus: non-English regulatory specifics (Solvency I technical standards, national actuarial standards of practice) and highly specialized practice literature (professional-body notes, consulting-firm methodology papers). Items that would require those sources either degrade to Wikipedia-adjacent paraphrase or are not generated at all. Language. The current generation pipeline and both benchmarks are English-only; Wikipedia is accessed on the English edition. A German-language ActuBench would need a German note- summarization adapter and would produce a different distribution of items on the same syllabus. We consider this the most concrete near-term extension, given that the DAV examination ecosystem the work motivates is itself German-language. Judge-model bias. The LLM-as-judge design imports known judge-side biases: position, verbosity, and stylistic preferences of the judge model can influence verdicts independently of correctness [21]. For closed-ended actuarial items with ground-truth answers the exposure is smaller than for preference-style evaluation, but it is not zero. A robustness check that varies the judge model and reports inter-judge agreement would strengthen the Judge-mode claims of section 5.3; we regard this as the most important methodological follow-up. Contamination risk. Although our pipeline can generate fresh items on demand and the item subset we release is curated, the generation pipeline itself is LLM-driven and its prompts and Wikipedia sources are public. Any generated item released publicly risks eventual inclusion in future pre-training corpora. The live-website release model of section 4.5 slows but does not prevent this. A long-running ActuBench evaluation would rotate its public subset periodically. 8 Conclusion We have presented ActuBench, a multi-agent LLM pipeline for the automated generation and evaluation of advanced actuarial assessment items. The pipelineâs principal methodological move is an independent verifier agent: separate adapters draft, construct distractors, and verify; each verdict is structured and auditable; one-shot repair loops make a substantial fraction of flagged items usable without human intervention. The verifier catches first-pass problems in a majority of drafted items at the stem level and in close to half at the distractor level, most of which are subsequently repaired by the one-shot loop. We evaluated fifty language models spanning eight providers on two complementary bench- marks â 100 empirically hardest multiple-choice items and 100 open-ended items scored by an LLM judge. Three findings are our headline empirical contribution. At the top of the MCQ benchmark, four models share the ceiling at 98 % accuracy while their inference costs span an order of magnitude: flagship accuracy on closed-form items is cheaper than it looks. A hosted open-weights 120B model and a locally-run Gemma 4 hold two Pareto-dominant positions at near-zero marginal cost. And MCQ-mode and Judge-mode do not rank models identically: removing the four-option scaffold shifts the benchmarkâs ceiling downward by about 10â15 p and reveals cleaner differentiation among the strongest reasoning-mode models. The items and per-model responses are browsable athttps://actubench.de/en/; the pipeline code is available from the author on request. We hope the combination of independent verification on the generation side, a ground-truth LLM-as-judge mode on the evaluation side, and a curated public item subset makes ActuBench useful as both a practical evaluation tool for the actuarial community and an empirical reference for the broader LLM-benchmark literature. 15 References [1] Caesar Balona. âActuaryGPT: Applications of Large Language Models to Insurance and Actuarial Workâ. British Actuarial Journal 2024. SSRN 4543652, first posted 17 Aug 2023. url: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4543652. [2]Weize Chen et al. âAgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorsâ 2023. arXiv:2308.10848. url:https://arxiv.org/abs/2308. 10848. [3]Zhiyu Chen et al. âFinQA: A Dataset of Numerical Reasoning over Financial Dataâ 2021. arXiv: 2109.00122. url: https://arxiv.org/abs/2109.00122. [4]Chunyuan Deng et al. âInvestigating Data Contamination in Modern Benchmarks for Large Language Modelsâ 2023. arXiv:2311.09783. url:https://arxiv.org/abs/2311.09783. [5]Xinya Du, Junru Shao, and Claire Cardie. âLearning to Ask: Neural Question Generation for Reading Comprehensionâ 2017. arXiv:1705.00106. url:https://arxiv.org/abs/ 1705.00106. [6]Yilun Du et al. âImproving Factuality and Reasoning in Language Models through Multia- gent Debateâ 2023. arXiv: 2305.14325. url: https://arxiv.org/abs/2305.14325. [7]Dheeru Dua et al. âDROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphsâ 2019. arXiv:1903.00161. url:https://arxiv.org/abs/ 1903.00161. [8] Sabina Elkins et al. âHow Useful are Educational Questions Generated by Large Language Models?â 2023. arXiv: 2304.06638. url: https://arxiv.org/abs/2304.06638. [9] Yilin Hao et al. âUtilizing Large Language Models (LLMs) for Quantitative Reasoning- Intensive Tasks within the (Re)Insurance Sectorâ. Annals of Actuarial Science 2025. doi: 10.1017/S1748499525100079. url: https://doi.org/10.1017/S1748499525100079. [10]Michael Heilman and Noah A. Smith. âGood Question! Statistical Ranking for Question Generationâ. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). Asso- ciation for Computational Linguistics, 2010, p. 609â617. url:https://aclanthology. org/N10-1086/. [11]Dan Hendrycks et al. âMeasuring Mathematical Problem Solving With the MATH Datasetâ 2021. arXiv: 2103.03874. url: https://arxiv.org/abs/2103.03874. [12]Sirui Hong et al. âMetaGPT: Meta Programming for a Multi-Agent Collaborative Frame- workâ 2023. arXiv: 2308.00352. url: https://arxiv.org/abs/2308.00352. [13] Seungone Kim et al. âPrometheus: Inducing Fine-grained Evaluation Capability in Lan- guage Modelsâ 2023. arXiv: 2310.08491. url: https://arxiv.org/abs/2310.08491. [14] Ghader Kurdi et al. âA Systematic Review of Automatic Question Generation for Educa- tional Purposesâ. International Journal of Artificial Intelligence in Education 30.1 2020, p. 121â204. doi:10.1007/s40593-019-00186-y. url:https://doi.org/10.1007/ s40593-019-00186-y. [15]Humza Naveed et al. âA Comprehensive Overview of Large Language Modelsâ 2024. arXiv: 2307.06435. url: https://arxiv.org/abs/2307.06435. [16]Long Phan, Alice Gatti, Ziwen Han, et al. âHumanityâs Last Examâ 2025. arXiv:2501. 14249. url: https://arxiv.org/abs/2501.14249. [17] Ronald Richman. âAI in Actuarial Scienceâ. SSRN Electronic Journal 2018. doi:10. 2139/ssrn.3218082. url:https://papers.ssrn.com/sol3/papers.cfm?abstract_ id=3218082. 16 [18]Ronald Richman. âAn AI Vision for the Actuarial Professionâ. Casualty Actuarial Society E-Forum (Summer 2024) 2024. SSRN 4758296, prize-winning essay. url:https://papers. ssrn.com/sol3/papers.cfm?abstract_id=4758296. [19]Oscar Sainz et al. âNLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmarkâ 2023. arXiv:2310.18018. url:https://arxiv. org/abs/2310.18018. [20]Andreas Troxler and JĂźrg Schelldorfer. âActuarial Applications of Natural Language Processing Using Transformers: Case Studies for Using Text Features in an Actuarial Contextâ 2022. arXiv: 2206.02014. url: https://arxiv.org/abs/2206.02014. [21]Peiyi Wang et al. âLarge Language Models are not Fair Evaluatorsâ 2023. arXiv:2305. 17926. url: https://arxiv.org/abs/2305.17926. [22]Qingyun Wu et al. âAutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Frameworkâ 2023. arXiv:2308.08155. url:https://arxiv.org/abs/2308. 08155. [23]Mario V. WĂźthrich and Michael Merz. Statistical Foundations of Actuarial Learning and its Applications. 1st ed. Springer Actuarial. Springer Cham, 2023. doi:10.1007/978-3- 031-12409-9. url: https://link.springer.com/book/10.1007/978-3-031-12409-9. [24] Cheng Xu et al. âBenchmark Data Contamination of Large Language Models: A Surveyâ 2024. arXiv: 2406.04244. url: https://arxiv.org/abs/2406.04244. [25] Lianmin Zheng et al. âJudging LLM-as-a-Judge with MT-Bench and Chatbot Arenaâ 2023. arXiv: 2306.05685. url: https://arxiv.org/abs/2306.05685. [26]Hua Zhou et al. âDesign, Results and Industry Implications of the Worldâs First Insurance Large Language Model Evaluation Benchmark (CUFEInse)â 2025. arXiv:2511.07794. url: https://arxiv.org/abs/2511.07794. [27]Kun Zhou et al. âDonât Make Your LLM an Evaluation Benchmark Cheaterâ 2023. arXiv: 2311.01964. url: https://arxiv.org/abs/2311.01964. A Key Prompts This appendix reproduces the core prompts used by Agents A, B and C, and by the auxiliary agent. All prompts request JSON-only output; validation and caching are handled by the pipeline. For space, we quote the prompt instructions in condensed form; the full templates, including structured output schemas, are available from the author on request. Agent A â item drafting. Given the IAA learning objective, the sampled difficulty archetype, and the Wikipedia notes assembled by the auxiliary agent, Agent A is instructed to produce the question stem, the correct answer, and its rationale. Hard constraints: English only; a single best answer under all reasonable interpretations; explicit statement of any assumption needed for uniqueness; no reliance on parametric knowledge that cannot be traced back to the notes. The difficulty archetype is passed as a short natural-language description (e.g., âQuantitative calculation: multi-step computations where each sub-step admits a characteristic errorâ) so the model calibrates style and depth accordingly. Agent B â distractor construction. Agent B is shown the stem, the correct answer with its rationale, and the Wikipedia notes. It is asked for three distractors, each with a short free-text rationale stating the specific misconception or computational slip the distractor is meant to probe, and each satisfying individual plausibility, collective distinctness, and individual incorrectness under the stemâs stated assumptions. Agent C â item verification. Agent C receives the learning objective, the Wikipedia notes, the stem, the correct answer, and the rationale, but not the identity of Agent A or 17 the drafting prompt. It returns a structured verdict ok: bool, issues: [type: factual_- error|ambiguity|underspecification|difficulty_misalignment, description: str]. On ok=false, Agent Aâs repair stage is re-invoked with the full issue report. Agent C â distractor verification. Agent C receives all four options with their rationales. It returns the same verdict schema, with issue types specialized to distractor design: duplicate_- with_A, duplicate_with_other, not_plausible, inadvertently_correct. Auxiliary agent â Wikipedia notes. Given a Wikipedia extract, the auxiliary agent produces a fixed bullet-list schema grouping the content into named facts and their provenance. The same adapter is re-used for topic-label assignment at the end of the pipeline. Judge-mode prompts. The evaluatee sees a minimal system prompt (âYou are taking an actuarial exam. Answer as accurately and concisely as possible. Show your reasoning briefly, then state your final answer clearly.â) and the bare stem. The judge sees the original question, the evaluateeâs free-text response, and the known correct answer, and returns a JSON verdict comprising the extracted final answer, a short reasoning note, a yes/no correctness decision, and a 0â100 confidence score. B Example Items Two example items from the MCQ benchmark are shown below. Option A is always the correct answer in the stored item; at evaluation time the four options are shuffled per run and the shuffle map is persisted (section 4.1). Example 1 â Quantitative Calculation Learning objective 4.2.2: Apply techniques for creating new distributions: multiplication by a constant, raising to a power. Stem. A random variableXfollows a log-normal distribution with parametersÎź= 1.2 and Ď 2 = 0.09, meaningln(X)âź Normal(Îź,Ď 2 ). Define a new random variableY= 4X 2 . Given ln(4)â 1.386, what are the parameters (Îź Y ,Ď 2 Y ) of the log-normal distribution of Y ? ⢠A (correct): Îź Y = 3.786, Ď 2 Y = 0.36. ⢠B: Îź Y = 2.4, Ď 2 Y = 0.36. ⢠C: Îź Y = 3.786, Ď 2 Y = 0.18. ⢠D: Îź Y = 5.172, Ď 2 Y = 0.36. Rationale. Sinceln(X)âź Normal(1.2,0.09), we haveln(Y) =ln(4)+2ln(X)âź Normal(ln(4)+ 2¡1.2,4¡0.09) =Normal(3.786,0.36). Distractors B and D test incorrect handling of the additive constant; C tests incorrect variance scaling (multiplying Ď 2 by 2 rather than by 4). Example 2 â Conceptual Inversion Learning objective 2.1.5: Apply the term structure of interest rates to modelling cash flows. Stem. An analyst values a single, certain cash flow using standard present-value discounting with the appropriate spot rate from the government yield curve (no embedded options, no credit adjustments). She finds that the present value of a $50,000 payment due in 18 months is $50,750. What condition must exist in the current interest-rate environment for this result to be internally consistent with standard discounting theory? ⢠A (correct): The 18-month spot rate must be negative. â˘B: The 18-month spot rate must be positive, because a positive rate naturally produces a slight premium over the face amount. â˘C: The yield curve must be inverted, meaning long-term spot rates are lower than short-term spot rates. 18 â˘D: The 18-month implied forward rate must be negative, but the 18-month spot rate may remain positive. Rationale. Under standard discounting,PV=FV/(1+r) t . HerePV > FV, so (1+r) 1.5 < 1, which requiresr <0. Distractor B inverts the direction of the PV-FV relationship; C confuses the shape of the curve with the sign of the level; D confuses forward and spot rates. C Excluded Models Five models were run in the benchmark but are excluded from all main tables and figures because adapter-level errors caused every answer to be recorded as unparseable (zero accuracy on all items of the MCQ pool): ⢠anthropic/claude-opus-4-20250514:thinking ⢠groq/qwen/qwq-32b:reasoning ⢠groq/deepseek-r1-distill-llama-70b:reasoning ⢠mistral/magistral-medium-2509:reasoning ⢠mistral/ministral-3-8b-2512 In all five cases the zero accuracy reflects the adapterâs inability to retrieve a parseable letter from the response (thinking-mode outputs exceeding the token budget, or unexpected response envelopes). These are adapter-implementation issues, not capability measurements; a future pipeline update should either fix the adapters or skip the affected models cleanly. Acknowledgments â LLM-assistance disclosure The manuscript was drafted with the assistance of Anthropic Claude (Opus 4.7, 1M-context). All research design, empirical interpretations and final text decisions are the responsibility of the author. Claude is additionally one of the evaluated models in this study (section 4.4); we disclose this dual role rather than excluding Claude from the evaluation, because its relative position on the benchmark is of independent scientific interest. 19