Paper deep dive
Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science
Benjamin Wilcox, Dawei Gao, Pradeeban Kathiravelu, Douglas Causey, Kewei Sha, Yunhe Feng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/8/2026, 7:45:10 AM
Summary
This paper introduces ArcticQA, a dataset of 194 evidence-grounded multiple-choice questions derived from primary Arctic research, and ArcticAbstain, a paired benchmark designed to evaluate Large Language Models' (LLMs) ability to abstain from answering when no valid option is present. The study evaluates eight models from the Gemini, Claude, and ChatGPT families, finding significant variation in baseline abstention rates (0.0% to 63.0%) and an average increase in abstention of 5.05 percentage points when the correct answer is removed. The work highlights the need to jointly evaluate abstention frequency and responsiveness to answer availability in scientific QA.
Entities (9)
Relation Signals (7)
ArcticAbstain → usesdataset → ArcticQA
confidence 98% · Using ArcticQA, we develop ArcticAbstain, a paired benchmark...
ArcticQA → createdby → Benjamin Wilcox
confidence 95% · We introduce ArcticQA... Benjamin Wilcox Affiliation: University of North Texas
ArcticQA → derivedfrom → Primary Arctic Research
confidence 95% · ArcticQA, a dataset of 194 questions derived from primary Arctic research
ArcticAbstain → evaluates → LLM Abstention
confidence 95% · ArcticAbstain evaluates LLM abstention on Arctic scientific questions
Gemini → evaluatedin → ArcticAbstain
confidence 90% · We evaluate eight models from the Gemini, Claude, and ChatGPT families... on 194 Arctic scientific questions
Claude → evaluatedin → ArcticAbstain
confidence 90% · We evaluate eight models from the Gemini, Claude, and ChatGPT families... on 194 Arctic scientific questions
ChatGPT → evaluatedin → ArcticAbstain
confidence 90% · We evaluate eight models from the Gemini, Claude, and ChatGPT families... on 194 Arctic scientific questions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) should abstain from scientific multiple-choice questions when no option is valid, but frequent abstention alone does not demonstrate sensitivity to answer availability. We introduce ArcticQA, a dataset of 194 questions derived from primary Arctic research, with automated checks of answer support and distractor contradiction against source evidence. We further develop ArcticAbstain, a paired benchmark comparing answer-present and answer-absent conditions, with the correct answer replaced by a distractor in the latter and an explicit abstention option in both. We evaluate eight models from the Gemini, Claude, and ChatGPT families at high reasoning effort, with three trials per condition, yielding 9,312 recorded responses. Answer-present abstention rates range from 0.0% to 63.0%, whereas replacing the correct answer increases abstention by 5.05 percentage points on average. These findings highlight substantial baseline differences and the need to evaluate abstention frequency and responsiveness jointly. The dataset and benchmark are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.09446v1
- Canonical: https://arxiv.org/abs/2610.09446v1
Trouble viewing inline? Open PDF directly →
Full Text
34,806 characters extracted from source content.
Expand or collapse full text
Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science Benjamin Wilcox Affiliation: University of North Texas, Denton, TX Email: benjamin.wilcox@unt.edu Dawei Gao Affiliation: University of North Texas, Denton, TX Email: dawei.gao@unt.edu Pradeeban Kathiravelu Affiliation: University of Alaska Anchorage, Anchorage, AK Email: pkathiravelu@alaska.edu Douglas Causey Affiliation: University of Alaska Anchorage, Anchorage, AK Email: dcausey@alaska.edu Kewei Sha Affiliation: University of North Texas, Denton, TX Email: kewei.sha@unt.edu Yunhe Feng Affiliation: University of North Texas, Denton, TX Email: yunhe.feng@unt.edu Abstract Large language models (LLMs) should abstain from scientific multiple-choice questions when no option is valid, but frequent abstention alone does not demonstrate sensitivity to answer availability. We introduce ArcticQA, a dataset of 194 questions derived from primary Arctic research, with automated checks of answer support and distractor contradiction against source evidence. We further develop ArcticAbstain, a paired benchmark comparing answer-present and answer-absent conditions, with the correct answer replaced by a distractor in the latter and an explicit abstention option in both. We evaluate eight models from the Gemini, Claude, and ChatGPT families at high reasoning effort, with three trials per condition, yielding 9,312 recorded responses. Answer-present abstention rates range from 0.0% to 63.0%, whereas replacing the correct answer increases abstention by 5.05 percentage points on average. These findings highlight substantial baseline differences and the need to evaluate abstention frequency and responsiveness jointly. The dataset and benchmark are available at https://github.com/BenWilcox8/arctic-qa. 1 Introduction Large language models (LLMs) could help researchers navigate and synthesize the multidisciplinary literature of Arctic science. Their usefulness, however, depends on whether they can provide scientifically supported answers and recognize when they should abstain. Arctic research spans interconnected physical, ecological, and human systems, and its findings often depend on specific locations, seasons, populations, and measurement conditions. Moreover, field observations are unevenly distributed across the region: prior research has documented substantial spatial biases in terrestrial Arctic sampling and scientific citation (Metcalfe et al., 2018). In this setting, an answer that is scientifically plausible in general may be incorrect within the scope of a particular question. Such errors can distort the interpretation of published findings and propagate into subsequent scientific analyses, making abstention an important component of trustworthy Arctic question answering (QA). LLM abstention has already been investigated in other scientific disciplines. For example, Wen et al. (2024) study abstention in biomedical and computer-science QA and show that behavior varies with the model, question type, and available context. Broader evaluations, including Abstain-QA and AbstentionBench, also reveal persistent difficulties in distinguishing questions that warrant an answer from those that warrant abstention (Madhusudhan et al., 2025; Kirichenko et al., 2025). However, to the best of our knowledge, no dedicated benchmark has systematically evaluated LLM abstention on scientific questions grounded in primary Arctic research. Existing results therefore leave unresolved whether LLMs can appropriately withhold answers when the available choices conflict with the specific conditions and findings of Arctic studies. Addressing this gap requires an evaluation resource that preserves the scope of Arctic scientific evidence. Geographic variation, seasonal dependence, and differences in study populations or measurement protocols complicate the construction of both correct answers and demonstrably incorrect alternatives. A finding from one setting cannot automatically establish what is valid in another. These considerations make benchmark curation particularly demanding: questions must state the relevant conditions, answers must be traceable to source findings, and distractors must be evaluated within the same scope. A dedicated Arctic resource is therefore necessary to examine abstention against explicit scientific evidence and to make the basis of the evaluation inspectable. To support trustworthy evaluation, we introduce ArcticQA, an evidence-grounded QA dataset constructed from primary Arctic research. For each item, we identify a clearly defined finding and its supporting evidence, then formulate a standalone question that preserves the conditions needed to interpret that finding. Separate automated checks assess whether the designated answer is supported by the source and whether each distractor is contradicted within the question’s stated scope. Checking distractors explicitly is essential because an option’s absence from a paper does not establish that it is false. Building on prior work in paper-grounded and automatically generated scientific QA (Dasigi et al., 2021; Wan et al., 2024), ArcticQA provides item-level evidence and validation checks to support transparent assessment of Arctic scientific answers. Using ArcticQA, we develop ArcticAbstain, a paired benchmark centered on the following question: How does an LLM’s abstention behavior change when a scientifically valid answer is removed from the available options? We operationalize abstention as selecting an explicit abstention option instead of a substantive answer. In the answer-present condition, each question includes one designated correct answer, three distractors, and the abstention option. In the matched answer-absent condition, the correct answer is replaced with a fourth distractor, while the question and the remaining options are retained. The underlying scientific question remains answerable from its source; the manipulation changes whether a valid answer appears among the choices. Under this protocol, abstaining in the answer-present condition constitutes false abstention, whereas selecting a substantive option in the answer-absent condition constitutes false commitment. Comparing the two conditions measures sensitivity to answer availability relative to a model’s baseline tendency to abstain, which a raw abstention rate alone cannot establish. We evaluate eight models from the Gemini, Claude, and ChatGPT families on 194 Arctic scientific questions at high reasoning effort. Each model answers both conditions three times, yielding 9,312 recorded responses. We analyze abstention alongside the accuracy of non-abstaining responses. Answer-present abstention rates range from 0.0% to 63.0% across models, while replacing the correct answer increases abstention by 5.05 percentage points on average. These results reveal substantial variation in baseline abstention and a comparatively limited average change after answer replacement under our evaluation protocol. They underscore the need to evaluate both a model’s willingness to answer Arctic scientific questions and its responsiveness to the absence of a valid option. Our main contributions are as follows: • ArcticQA: an evidence-grounded Arctic QA dataset. We construct questions from primary Arctic research, retain item-level supporting evidence, and apply separate automated checks to answers and distractors to support trustworthy, inspectable evaluation. • ArcticAbstain: a paired benchmark for Arctic scientific abstention. We vary answer availability within matched question pairs to measure false abstention, false commitment, and changes in abstention relative to baseline behavior. • An empirical evaluation of eight LLMs. We conduct repeated trials across three model families, characterizing differences in abstention behavior and sensitivity to answer replacement on Arctic scientific questions. 2 Related Work Work on selective generation asks whether a model can estimate uncertainty and withhold an unreliable answer. Self-evaluation prompts can improve selective generation Ren et al. (2023), while multi-model collaboration can expose knowledge gaps before a response is produced Feng et al. (2024a). Abstain-QA evaluates willingness to answer across answerable and unanswerable prompts Madhusudhan et al. (2025). AbstentionBench broadens this analysis across types of unanswerable questions and reports that reasoning-oriented training can increase forced problem solving Kirichenko et al. (2025). These studies motivate our high-reasoning-effort setting, but their conclusions cannot be assumed to transfer directly to source-bounded Arctic scientific questions. Scientific abstention has also been studied through controlled context changes and external verification. Context-perturbation experiments alter the evidence available to a model and observe how its answer or abstention changes Wen et al. (2024). Abstention-aware scientific reasoning systems can decompose claims and check them against retrieved evidence or natural-language-inference models Abdaljalil et al. (2026). Scientific claim-verification datasets such as SciFact-Open likewise connect claims to evidence retrieval and verification Wadden et al. (2022). These systems evaluate the reliability of a composite process. ArcticQA instead evaluates the response of the tested model to a self-contained multiple-choice item, without giving the model a retrieval or verification pipeline. Scientific question-answering resources cover difficult reasoning and questions anchored in research papers. GPQA targets graduate-level questions in biology, physics, and chemistry, while MMLU-Pro expands difficult multi-task multiple-choice evaluation Rein et al. (2023); Wang et al. (2024). Other work builds information-seeking questions around research papers or generates scientific QA datasets with fine-grained checks Dasigi et al. (2021); Wan et al. (2024). Round-trip consistency can filter synthetic question-answer pairs by testing whether an answer can be reconstructed Alberti et al. (2019). Decontextualization work formalizes the requirement that text remain understandable outside its original document Choi et al. (2021). ArcticQA combines these concerns by deriving each question from bounded source evidence. The final question must also stand alone for a reader who does not have the paper. Automatic multiple-choice construction introduces a separate risk: a plausible distractor can be partly correct, true under another scope, or equivalent to the answer. Prior work therefore treats distractor generation and distractor evaluation as distinct tasks Yu et al. (2024); Byun and Choi (2025); Feng et al. (2024b). Iterative critique can improve generated questions Yao et al. (2025), but automated refinement and model judging can preserve shared errors or introduce judge bias Zheng et al. (2023); Benedetto et al. (2025). ArcticQA responds with per-option source checks, deterministic rejection rules, bounded revision paths, and an explicitly non-gold machine-acceptance label. 3 ArcticQA Dataset We introduce ArcticQA, a question-answering dataset grounded in specific findings from primary Arctic research. Reliable abstention evaluation requires both a supported correct answer and demonstrably incorrect distractors. We therefore develop a construction pipeline that links each target answer to supporting source evidence and verifies each distractor within the question’s stated scope. The pipeline comprises paper selection, finding extraction and question generation, QA validation, and distractor generation and validation. 3.1 Paper selection and eligibility We identified 84,829 papers through keyword searches using common Arctic research terms in Semantic Scholar11 1 https://w.semanticscholar.org/. Metadata filtering reduced this set to 4,420 candidate papers, whose full texts were subsequently assessed for eligibility. We defined geographic eligibility using a latitude threshold of 66.56∘66.56 N for terrestrial studies and a predefined list of eligible Arctic marine regions for marine studies. This list remained fixed throughout the eligibility assessment process. 3.2 Finding extraction and question generation For each eligible paper, an LLM reads the full text and identifies the primary finding together with its supporting textual evidence. Once validated, the finding is held fixed as the target answer. A separate LLM writer then formulates a standalone question that asks specifically for this answer while preserving the scope and conditions of the original finding. This answer-first approach, followed by answer reconstruction, draws on the roundtrip-consistency principle of Alberti et al. (2019). We adapt scientific-paper QA generation (Wan et al., 2024) by restricting generation to one clearly scoped finding per paper. 3.3 QA validation We validate each QA pair through blinded answer reconstruction followed by a separate verification step. The reconstructor LLM receives the question, its context, and the source evidence, but not the proposed answer. It produces a reconstructed answer, supporting evidence, and alternative answers. A separate verifier compares the proposed and reconstructed answers and assesses their consistency with the source. Specifically, it checks whether the evidence entails the proposed answer, whether the question and answer preserve the finding’s scope and claim type, and whether the question’s context is sufficient without disclosing the answer. A QA pair proceeds to distractor generation only after passing all checks. This combination of reconstruction and entailment checking is informed by the complementary factual-consistency signals identified by Fabbri et al. (2022). 3.4 Distractor generation and validation For each validated QA pair, we construct four distractors through separate candidate-generation and verification stages. This staged design draws on prior work on automated distractor generation and filtering (Yu et al., 2024; Byun and Choi, 2025). A writer LLM proposes four to six candidates by modifying numerical values, categories, directions of change, scope, or entities. The writer does not verify its own outputs; each candidate undergoes a separate verification step. To be retained, a candidate must be contradicted by specific source evidence within the question’s stated scope. It is rejected if a reasonable interpretation of the paper could support it as a correct answer. Absence from the source alone does not establish falsity, and validity is assessed under the conditions specified by the question, including the relevant location and time. We retain four candidates that satisfy these criteria. These checks emphasize distractor validity, which prior work distinguishes from plausibility (Feng et al., 2024b), and make the basis for rejecting each option traceable to the source evidence. 4 ArcticAbstain Benchmark ArcticAbstain evaluates LLM abstention on Arctic scientific questions using two matched multiple-choice conditions: answer-present and answer-absent. In the former, the correct answer is included among the options; in the latter, it is replaced by a validated distractor. Comparing responses across these conditions measures how abstention changes when a correct answer is no longer available for selection. 4.1 Paired abstention protocol Each pair shares the same question wording and three validated distractors. Both items also include the explicit option “I abstain from answering,” consistent with prior work that allows LLMs to reject all candidate answers during self-evaluation (Ren et al., 2023). Each item contains five options, with their order randomized to reduce systematic effects of option position. 4.1.1 Answer-present condition The answer-present item contains the gold answer, three distractors, and the abstention option. Under the benchmark’s scoring protocol, selecting the gold answer is correct, selecting a distractor is an incorrect answer, and selecting the abstention option constitutes false abstention, or over-abstention. This condition quantifies how often a model abstains despite the presence of a valid answer. 4.1.2 Answer-absent condition We replace the gold answer with a fourth validated distractor, retaining the same question, three shared distractors, and abstention option. Because no candidate answer is valid within the question’s stated scope, abstention is the only response scored as correct. Selecting any distractor constitutes false commitment, indicating failure to abstain when no valid answer is available. The underlying question remains answerable from the source paper in both conditions; only the available answer options change. Together, the paired conditions distinguish false abstention from false commitment and quantify the change in abstention relative to the model’s answer-present baseline behavior. 5 Experiments 5.1 Experimental setup We evaluate eight models from three families: two Gemini models, three Claude models, and three ChatGPT models. All models are configured to use high reasoning effort. Each question from ArcticQA is evaluated under the paired answer-present and answer-absent conditions defined in ArcticAbstain. Each model answers each condition three times through separate calls, yielding six responses per model and 48 responses across all eight models for each base question. The analysis includes 194 questions with all 48 planned response records, for a total of 9,312 recorded responses. Invalid responses are excluded from metric calculations; thus, the number of recorded responses is distinguished from the number of valid responses used in the metric denominators for reported results. 5.2 Evaluation metrics We evaluate answer selection and abstention in ArcticAbstain using six metrics from prior abstention research Feng et al. (2024a); Wen et al. (2025); Wen et al. (2024). For each model, we partition its responses across all evaluated questions and trials into five mutually exclusive categories. In the answer-present condition, N1N_1, N2N_2, and N3N_3 denote the numbers of correct answers, incorrect answers, and abstentions, respectively. In the answer-absent condition, N4N_4 and N5N_5 denote the numbers of incorrect answers and abstentions, respectively. Because no substantive option is correct in the answer-absent condition, every non-abstaining response in that condition is counted in N4N_4. Under the benchmark’s answer-availability criterion, N3N_3 represents false abstentions and N5N_5 represents appropriate abstentions. The total number of evaluated responses is N=∑i=15NiN= _i=1^5N_i. Abstention accuracy (ACC) measures overall success in providing correct answers or abstaining when necessary: ACC=N1+N5N.ACC= N_1+N_5N. (1) It credits correct answers when a valid option is present and abstentions when all substantive options are incorrect, thereby jointly evaluating answer correctness and abstention behavior. Abstention precision measures the proportion of abstentions that occur when no valid answer option is available: Precisionabs=N5N3+N5.Precision_abs= N_5N_3+N_5. (2) A higher value indicates that a greater proportion of the model’s abstentions are appropriate under the evaluation protocol. Following the definition summarized by Wen et al. (2025), abstention recall is computed as Recallabs=N5N2+N4+N5.Recall_abs= N_5N_2+N_4+N_5. (3) This metric measures the proportion of appropriate abstentions among the combined set of appropriate abstentions and incorrect substantive answers. Its denominator includes errors from both conditions: N2N_2 captures incorrect selections despite the availability of a valid option, and N4N_4 captures incorrect selections when no valid option is available. Thus, this definition penalizes incorrect answering in both conditions and differs from recall restricted to identifying answer-absent cases. We summarize abstention precision and recall using their harmonic mean, the abstention F1-score: F1abs=2PrecisionabsRecallabsPrecisionabs+Recallabs.F1_abs= 2\,Precision_abs\,Recall_absPrecision_abs+Recall_abs. (4) The abstention rate measures how frequently the model abstains across both conditions: AbstentionRate=N3+N5N.Abstention\ Rate= N_3+N_5N. (5) This rate describes the model’s overall tendency to withhold answers. Because it includes both false and appropriate abstentions, a higher abstention rate does not necessarily indicate better overall abstention performance. Finally, reliable accuracy (R-Acc) measures answer accuracy conditional on the model selecting a substantive option: R-Acc=N1N1+N2+N4.R-Acc= N_1N_1+N_2+N_4. (6) Its denominator includes all non-abstaining responses across both conditions. R-Acc therefore characterizes the correctness of committed answers and should be interpreted alongside the abstention rate to account for how frequently the model chooses to answer. 5.3 Aggregation and statistical analysis For the aggregate metric comparison, we compute each metric separately for the three trials and report the median across trials in Table 2. For the paired condition analysis in Table 1, we pool responses across all three trials to estimate each model’s condition-specific abstention rates. The paired shift is the answer-absent rate minus the answer-present rate, expressed in percentage points (p). We estimate 95% confidence intervals for these shifts using 10,000 bootstrap resamples of questions with a random seed of 7. Each resample retains both conditions and all three trials for every selected question, preserving the dependence among responses associated with that question. For each model, a question-level sign-flip permutation test assesses whether abstention changes after the correct answer is replaced. We apply Holm correction across the eight model-specific tests to control the family-wise error rate and assess significance at an adjusted p<0.05p<0.05. 5.4 Results and analysis 5.4.1 Variation in baseline abstention Table 1 shows substantial variation in answer-present abstention, ranging from 0.0% to 63.0%. ChatGPT 5.6 Sol and ChatGPT 5.6 Terra almost always select a substantive option, with abstention rates of 0.2% and 0.0%, respectively. The Gemini models abstain on 6.0%–6.4% of answer-present responses, whereas the Claude models abstain on 36.1%–48.5%. ChatGPT Astra has the highest rate, at 63.0%. The large differences among the three ChatGPT models show that model-family membership alone does not characterize abstention behavior in this evaluation. Model Present abst. Absent abst. Shift [95% CI] (p) Permutation p Holm p Gemini 3.8 Flash 6.4% 10.3% +3.9[0.6, 7.4]+3.9\;[0.6,\,7.4] 0.037 0.187 Gemini 3.7 Flash 6.0% 8.6% +2.6[−0.1, 5.4]+2.6\;[-0.1,\,5.4] 0.063 0.198 Claude Fable 5.1 36.1% 46.9% +10.8[5.7, 16.1]+10.8\;[5.7,\,16.1] 0.00010 0.0008 Claude Opus 5 41.3% 47.5% +6.2[0.9, 11.6]+6.2\;[0.9,\,11.6] 0.031 0.187 Claude Sonnet 5 48.5% 51.0% +2.5[−1.7, 6.9]+2.5\;[-1.7,\,6.9] 0.331 0.331 ChatGPT Astra 63.0% 74.1% +11.1[5.7, 16.6]+11.1\;[5.7,\,16.6] 0.00015 0.0011 ChatGPT 5.6 Sol 0.2% 2.1% +1.9[0.3, 4.0]+1.9\;[0.3,\,4.0] 0.049 0.198 ChatGPT 5.6 Terra 0.0% 1.5% +1.5[0.3, 3.3]+1.5\;[0.3,\,3.3] 0.065 0.198 Table 1: Condition-specific abstention rates pooled across three trials for 194 questions. Shifts are answer-absent minus answer-present rates, in percentage points (p). The 95% confidence intervals use question-level bootstrap resampling. Permutation tests use question-level paired differences; Holm-adjusted p-values account for the eight model-specific tests. 5.4.2 Response to correct-answer removal Replacing the correct answer with a distractor increases the abstention point estimate for every model. The increases range from 1.5 to 11.1 percentage points, with a mean increase of 5.05 percentage points across the eight models. These shifts are modest relative to the 63.0% range in answer-present abstention. Figure 1 visualizes each model’s paired rates. Claude Fable 5.1 and ChatGPT Astra exhibit the largest increases, at 10.8 and 11.1 percentage points, respectively. Both changes remain statistically significant after Holm correction, with adjusted p-values of 0.0008 and 0.0011. The remaining six models do not meet the adjusted significance threshold. Thus, the point estimates consistently indicate increased abstention after answer removal, while the model-specific tests provide evidence of a change for two models after correction for multiple comparisons. Figure 1: Abstention rates under the answer-present and answer-absent conditions for the eight models. Each segment connects a model’s two rates, pooled across three trials. Labels report the answer-absent minus answer-present shift in percentage points. 5.4.3 Accuracy and abstention must be interpreted jointly Table 2 reports median metrics across the three trials. ChatGPT Astra has the highest median ACC (0.541), abstention F1-score (0.618), and R-Acc (0.540), together with the highest overall abstention rate (0.686). However, its 63.0% answer-present abstention rate also indicates frequent abstention when a valid option is available. Its comparatively high conditional answer accuracy therefore coexists with substantial false abstention. All reported R-Acc medians exceed 0.125, the expected value for uniform random selection among the four substantive options under equally represented conditions: 12×14=0.125 12× 14=0.125. The other seven models have R-Acc medians between 0.261 and 0.379. These descriptive comparisons do not establish statistically significant differences between models. For example, ChatGPT 5.6 Sol has an R-Acc of 0.291, numerically close to the Gemini models’ values of 0.303 and 0.306, but a much lower abstention rate of 0.011. Conditional answer accuracy and the tendency to abstain therefore capture distinct aspects of performance, and both are needed to assess reliability on Arctic scientific questions. Model ACC Abst. rate F1_abs R-Acc Gemini 3.8 Flash 0.329 0.084 0.134 0.303 Gemini 3.7 Flash 0.326 0.073 0.113 0.306 Claude Fable 5.1 0.458 0.415 0.465 0.379 Claude Opus 5 0.438 0.444 0.459 0.359 Claude Sonnet 5 0.408 0.498 0.464 0.303 ChatGPT Astra 0.541 0.686 0.618 0.540 ChatGPT 5.6 Sol 0.298 0.011 0.029 0.291 ChatGPT 5.6 Terra 0.266 0.008 0.021 0.261 Table 2: Metrics pooled across the three trials for eight models evaluated at high reasoning effort on 194 Arctic scientific questions. Each trial’s metrics combine the answer-present and answer-absent conditions. 6 Conclusion We introduced ArcticQA, a dataset grounded in primary Arctic research, and ArcticAbstain, a paired benchmark for evaluating LLM abstention under answer-present and answer-absent conditions. Across eight models, answer-present abstention rates ranged from 0.0% to 63.0%, while removing the correct answer increased abstention by 5.05 percentage points on average. These findings highlight the distinction between a model’s tendency to abstain and its responsiveness to answer availability. Reliable evaluation should therefore consider false abstention, false commitment, and the accuracy of non-abstaining responses jointly. Our dataset and benchmark provide a foundation for assessing and improving LLM reliability in Arctic scientific question answering. 7 Limitations ArcticAbstain is a geographically bounded case study whose coverage reflects discovery queries, full-text availability, and model-guided processing order. It is not representative of all Arctic research or other scientific domains. Dataset construction relies on model-generated content and automated validation. No sample was reviewed by domain experts; thus, machine_accepted_unverified denotes automated acceptance, not expert-verified ground truth. The residual error rate remains unmeasured. The evaluation covers 194 questions, eight models, a single prompt, one explicit abstention option, and high reasoning effort. The paired multiple-choice task assesses rejection of invalid option sets rather than spontaneous abstention in open-ended scientific work. The results do not establish training exposure, explain observed response policies, or demonstrate generalization beyond the evaluated setting. References Abdaljalil et al. (2026) S. Abdaljalil, E. Serpedin, and H. Kurban Knowing when not to answer: abstention-aware scientific reasoning. arXiv (Cornell University). External Links: Document Cited by: §2. Alberti et al. (2019) C. Alberti, D. Andor, E. Pitler, J. Devlin, and M. Collins Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 6168–6173. External Links: Document, Link Cited by: §2, §3.2. Benedetto et al. (2025) L. Benedetto, S. Taslimipoor, and P. Buttery A survey on automated distractor evaluation in multiple-choice tasks. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications, p. 55–69. External Links: Document, Link Cited by: §2. Byun and Choi (2025) G. Byun and J. D. Choi D-GEN: automatic distractor generation and evaluation for reliable assessment of generative models. In Findings of ACL, p. 3316–3349. External Links: Document, Link Cited by: §2, §3.4. Choi et al. (2021) E. Choi, J. Palomaki, M. Lamm, T. Kwiatkowski, D. Das, and M. Collins Decontextualization: making sentences stand-alone. Transactions of the Association for Computational Linguistics 9, p. 447–461. External Links: Document, Link Cited by: §2. Dasigi et al. (2021) P. Dasigi, K. Lo, I. Beltagy, A. Cohan, N. A. Smith, and M. Gardner A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of NAACL-HLT, p. 4599–4610. External Links: Document, Link Cited by: §1, §2. Fabbri et al. (2022) A. Fabbri, C. Wu, W. Liu, and C. Xiong QAFactEval: improved QA-based factual consistency evaluation for summarization. In Proceedings of NAACL-HLT, p. 2587–2601. External Links: Document, Link Cited by: §3.3. Feng et al. (2024a) S. Feng, W. Shi, Y. Wang, W. Ding, V. Balachandran, and Y. Tsvetkov Don’t hallucinate, abstain: identifying llm knowledge gaps via multi-llm collaboration. External Links: 2402.00367, Link Cited by: §2, §5.2. Feng et al. (2024b) W. Feng, J. Lee, H. McNichols, A. Scarlatos, D. Smith, S. Woodhead, N. Ornelas, and A. Lan Exploring automated distractor generation for math multiple-choice questions via large language models. In Findings of NAACL, p. 3067–3082. External Links: Document, Link Cited by: §2, §3.4. Kirichenko et al. (2025) P. Kirichenko, M. Ibrahim, K. Chaudhuri, and S. J. Bell AbstentionBench: reasoning llms fail on unanswerable questions. arXiv.org. External Links: Document Cited by: §1, §2. Madhusudhan et al. (2025) N. Madhusudhan, S. T. Madhusudhan, V. Yadav, and M. Hashemi Do LLMs know when to NOT answer? investigating abstention abilities of large language models. In Proceedings of the 31st International Conference on Computational Linguistics, Abu Dhabi, UAE, p. 9329–9345. External Links: Link Cited by: §1, §2. Metcalfe et al. (2018) D. B. Metcalfe, T. D. G. Hermans, J. Ahlstrand, M. Becker, M. Berggren, R. G. Björk, M. P. Björkman, D. Blok, N. Chaudhary, C. Chisholm, A. T. Classen, N. J. Hasselquist, M. Jonsson, J. A. Kristensen, B. B. Kumordzi, H. Lee, J. R. Mayor, J. Prevéy, K. Pantazatou, J. Rousk, R. A. Sponseller, M. K. Sundqvist, J. Tang, J. Uddling, G. Wallin, W. Zhang, A. Ahlström, D. E. Tenenbaum, and A. M. Abdi Patchy field sampling biases understanding of climate change impacts across the Arctic. Nature Ecology & Evolution 2 (9), p. 1443–1448. External Links: Document, Link Cited by: §1. Rein et al. (2023) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof q&a benchmark. External Links: arXiv:2311.12022 Cited by: §2. Ren et al. (2023) J. Ren, Y. Zhao, T. Vu, P. J. Liu, and B. Lakshminarayanan Self-evaluation improves selective generation in large language models. External Links: 2312.09300, Link Cited by: §2, §4.1. Wadden et al. (2022) D. Wadden, K. Lo, B. Kuehl, A. Cohan, I. Beltagy, L. L. Wang, and H. Hajishirzi SciFact-Open: towards open-domain scientific claim verification. In Findings of EMNLP, p. 4719–4734. External Links: Document, Link Cited by: §2. Wan et al. (2024) Y. Wan, Y. Liu, A. Ajith, C. Grazian, B. Hoex, W. Zhang, C. Kit, T. Xie, and I. Foster SciQAG: a framework for auto-generated science question answering dataset with fine-grained evaluation. arXiv preprint arXiv:2405.09939. External Links: Document, Link Cited by: §1, §2, §3.2. Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen MMLU-pro: a more robust and challenging multi-task language understanding benchmark. External Links: 2406.01574, Link Cited by: §2. Wen et al. (2024) B. Wen, B. Howe, and L. L. Wang Characterizing LLM abstention behavior in science QA with context perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, p. 3437–3450. External Links: Document, Link Cited by: §1, §2, §5.2. Wen et al. (2025) B. Wen, J. Yao, S. Feng, C. Xu, Y. Tsvetkov, B. Howe, and L. L. Wang Know your limits: a survey of abstention in large language models. External Links: 2407.18418, Link Cited by: §5.2, §5.2. Yao et al. (2025) Z. Yao, A. Parashar, H. Zhou, W. S. Jang, F. Ouyang, Z. Yang, and H. Yu MCQG-SRefine: multiple choice question generation and evaluation with iterative self-critique, correction, and comparison feedback. In Proceedings of NAACL-HLT, p. 10728–10777. External Links: Document, Link Cited by: §2. Yu et al. (2024) C. Yu, W. H. Lee, L. T. Ke, S. Guo, and Y. Fan Automating true-false multiple-choice question generation and evaluation with retrieval-based accuracy differential. In Proceedings of the 17th International Natural Language Generation Conference, p. 198–212. External Links: Document, Link Cited by: §2, §3.4. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track, External Links: Document, Link Cited by: §2.