Paper deep dive

From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options

Obed Junias, Maria Leonor Pacheco

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 91

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 8/14/2026, 5:15:35 AM

Summary

The paper introduces a structured framework for compound answer reasoning in Large Language Models (LLMs) that addresses failures in logical composition. The method decomposes compound options (connected by AND, OR, NEITHER/NOR) into atomic answers, elicits contrastive evidence for each atom, and uses an operator-constrained Integer Linear Program (ILP) to compose calibrated scores into a final prediction. This approach significantly improves Macro-F1 on Logical-CommonsenseQA and a new benchmark, Logical-SATA, particularly for complex operators like NEITHER/NOR.

Entities (11)

Integer Linear Program · algorithm · 99%LLM · model · 99%Maria Leonor Pacheco · author · 99%Obed Junias · author · 99%University of Colorado Boulder · organization · 99%Logical-CommonsenseQA · benchmark · 98%Logical-SATA · benchmark · 97%AND · logicaloperator · 95%NEITHER/NOR · logicaloperator · 95%OR · logicaloperator · 95%SATA-Bench · benchmark · 95%

Relation Signals (9)

Maria Leonor Pacheco → affiliatedwith → University of Colorado Boulder

confidence 99% · Maria Leonor Pacheco Affiliation: University of Colorado Boulder

Obed Junias → affiliatedwith → University of Colorado Boulder

confidence 99% · Obed Junias Affiliation: University of Colorado Boulder

Logical-SATA → derivedfrom → SATA-Bench

confidence 95% · Logical-SATA, a reading-comprehension benchmark derived from SATA-Bench

Structured Compositional Reasoning Framework → evaluateson → Logical-CommonsenseQA

confidence 95% · We evaluate on LOGICAL-COMMONSENSEQA... Our framework improves Macro-F1 from 48.3 to 77.0

Structured Compositional Reasoning Framework → evaluateson → Logical-SATA

confidence 95% · introduce LOGICAL-SATA... Our framework improves... from 47.0 to 75.6 on LOGICAL-SATA

Structured Compositional Reasoning Framework → uses → Integer Linear Program

confidence 95% · An operator-constrained integer linear program then composes the calibrated scores into a single prediction.

Structured Compositional Reasoning Framework → handlesoperators → AND

confidence 90% · We study compound options connected by And, Or, and Neither/Nor

Structured Compositional Reasoning Framework → handlesoperators →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on LOGICAL-COMMONSENSEQA and introduce LOGICAL-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

89,350 characters extracted from source content.

Expand or collapse full text

From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options Obed Junias Affiliation: University of Colorado Boulder Email: obed.junias@colorado.edu Maria Leonor Pacheco Affiliation: University of Colorado Boulder Email: maria.pacheco@colorado.edu Abstract Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by And, Or, and Neither/Nor, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on Logical-CommonsenseQA and introduce Logical-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated Logical-CommonsenseQA split and from 47.0 to 75.6 on Logical-SATA, with the largest gains on Neither/Nor. 1 Introduction Large language models (LLMs) perform well across a wide range of tasks (Brown et al. 2020; Ouyang et al. 2022), but systematic evaluations reveal persistent weaknesses in their logical reasoning (Parmar et al. 2024). These failures are not uniform across logical operators. For example, Junias and Pacheco 2026 evaluate composition over compound answer options and report a graded pattern, with performance strongest on conjunction, weaker on disjunction, and collapsing on negated compositions. That the difficulty tracks the operator rather than the content suggests it stems from the way logical possibilities are represented and combined, not from missing knowledge. As illustrated in Figure 1, mental-model theories predict exactly this ordering in humans. They propose that people reason by constructing representations of situations compatible with a logical expression rather than applying formal proof rules (Johnson-Laird et al. 1992), so difficulty depends on the number and structure of the possibilities that must be maintained (Klauer 1997; Meiser et al. 2001; Neys 2006; DeWall et al. 2008). A conjunction (A∧B)(A B) can often be held single joint possibility, whereas a disjunction (A∨B)(A B) requires the alternatives to be maintained and compared (García-Madruga et al. 2001). Negation increases difficulty, as the reasoner must represent the original proposition while tracking that it is rejected (Macbeth et al. 2014; Khemlani et al. 2014). Neither/Nor is the extreme case, combining both demands. These studies do not imply that LLMs reason as humans do, but models show the same signature, degrading on disjunctive compositions (Khalid et al. 2025; Hoveyda et al. 2026; Junias and Pacheco 2026) and failing to revise affirmative predictions once a proposition is negated (García-Ferrero et al. 2023; Kassner and Schütze 2020; Ravichander et al. 2022; She et al. 2023). Figure 1: Mental-model representation of the possibilities associated with and, or, and neither/nor. Composition is therefore a distinct burden, not a byproduct of comprehension. Standard prompting, however, fuses the two, as the model must assess each atomic proposition and combine them under the operator in a single pass. This produces a compositionality gap, where a model solves the component subproblems correctly yet fails to combine them (Press et al. 2023). It also leaves no way to diagnose which step broke down, and it cannot enforce the composition, since a model asked to satisfy hard constraints in free generation can silently violate them. A common response to this gap is to make intermediate structure explicit, through chain-of-thought and decomposed prompting, entailment trees, or contrastive judgments over opposing candidates (Wei et al. 2022; Khot et al. 2023; Dalvi et al. 2021; Liusie et al. 2024). These produce richer intermediate evidence, but the combination step remains an unconstrained generation. Neuro-symbolic approaches instead delegate inference to an external solver, first translating the natural language problem into a formal representation (Pan et al. 2023; Olausson et al. 2023; Ye et al. 2023). This enforces the composition, but shifts the burden onto auto-formalization, and the solver is only as reliable as the translation it receives. In this paper, we study compound answer reasoning as a setting for isolating logical composition. This requires no translation, as the logical structure is already explicit in the answer options. What remains is to produce reliable intermediate evidence, as in the decomposition methods above, and to combine it under constraints that cannot be violated. Figure 2: Overview of the proposed framework. Each compound answer option is decomposed into two atomic answers and its explicit operator. The LLM scores local evidence for each atom, and an operator-constrained ILP combines these scores to select exactly one answer option under the corresponding operator semantics. We present a framework that does this, decomposing each option into its atomic answers, eliciting contrastive evidence for each, and composing that evidence under the constraints imposed by the operators. For every unique atomic answer, we construct paired positive and negative hypotheses stating that the answer is supported or not supported by the context. Atoms shared across options are scored once, so a proposition receives a single judgment wherever it appears. The LLM scores both hypotheses, and their contrast forms the evidence for that answer, so the model is never asked to judge a compound option as a whole. The resulting scores are calibrated and passed to an operator-constrained integer linear program (ILP), which jointly infers the assignment of each atomic answer and selects exactly one compound option. Figure 2 provides an overview of the proposed framework. To evaluate our framework, we use two benchmarks containing compound answer options connected by And, Or, and Neither/Nor: Logical-CommonsenseQA (Junias and Pacheco 2026), a commonsense reasoning benchmark, and Logical-SATA, a reading comprehension benchmark which we construct from SATA-Bench (Xu et al. 2025). The two require different kinds of atomic evidence, and we improve substantially on both, most sharply on the operator our account identifies as hardest: on Neither/Nor, macro-F1 rises from 14.0 to 76.8 on Logical-CommonsenseQA and from 12.6 to 73.4 on Logical-SATA. In sum, we contribute: (1) a structured framework that elicits contrastive evidence for individual atomic answers and combines them through operator-constrained ILP inference; (2) a relative calibration method, which scores each atom by both its confidence and its standing among the other atoms; (3) Logical-SATA, a new reading-comprehension benchmark for compound answer reasoning; (4) an evaluation across two distinct benchmarks, showing the largest gains on the operators that degrade most under standard prompting. The datasets11 1 ojayy/logical-csqa, ojayy/logical-sata and code22 2 github.com/obedjunias19/structured-compositional-reasoning are publicly available. 2 Related Work Logical and Compositional Reasoning with LLMs Logical-reasoning benchmarks such as ProofWriter (Tafjord et al. 2021), LogicNLI (Tian et al. 2021), FOLIO (Han et al. 2024), ReClor (Yu et al. 2020), and LogiQA (Liu et al. 2020) assess whether models can derive valid conclusions given facts, rules, premises, or constraints. Other datasets, including ConjNLI (Saha et al. 2020), CONDAQA(Ravichander et al. 2022), SCoNE (She et al. 2023), and the NOT benchmark (García-Ferrero et al. 2023) isolate logical phenomena such as conjunction, disjunction and negation. These evaluations show that logical performance remains sensitive to inference structure, linguistic formulation, and negation (Parmar et al. 2024). One strategy to address this is to decompose reasoning into intermediate steps, making them explicit or dividing a complex problem into simpler subproblems (Wei et al. 2022; Zhou et al. 2022; Khot et al. 2023). EntailmentBank organizes explanations as trees of multi-premise entailment steps (Dalvi et al. 2021), and DecompNLI provides a systematic framework for evaluating the validity of decomposed textual inferences (Weir et al. 2024). Multi- and Compound-Answer Benchmarks Most multiple-choice reasoning benchmarks require selecting a single correct answer type and score each candidate as a whole (Talmor et al. 2019; Bisk et al. 2019; Sap et al. 2019; Clark et al. 2018; Hendrycks et al. 2020). Consequently, they do not test whether a model can evaluate several atomic answers and combine them under an explicit logical operator. Multi-answer question-answering benchmarks relax the assumption that each question has only one correct response. MultiRC evaluates reading comprehension questions for which several candidate answers may be correct, while RoMQA requires models to recover multiple valid answers supported by evidence distributed across passages (Khashabi et al. 2018; Zhong et al. 2023). SATA-Bench more directly studies the select-all-that-apply format across several domains, where each option is evaluated independently and the model must identify the complete set of correct choices (Xu et al. 2025). These benchmarks evaluate multi-answer selection, but they do not place explicit boolean operators within the candidate answers. Logical-CommonsenseQA instead places explicit boolean operators within the candidate answers (Junias and Pacheco 2026). We extend the same operator-based structure to paragraph-based reading comprehension through Logical-SATA, which constructs compound options from the independently annotated answers in SATA-Bench. Its construction is described in Section 4. Confidence Elicitation and Contrastive Judgments Structured inference requires local scores that represent the model’s evidence for each atomic decision. Prior work obtains such scores from token probabilities, true–false self-assessment, repeated generation, or verbalized confidence (Jiang et al. 2021; Kadavath et al. 2022; Tian et al. 2023; Pauk and Pacheco 2026). These approaches differ in whether they require access to model probabilities and in how closely their reported confidence corresponds to empirical correctness. Other work studies comparative and contrastive judgments, where models evaluate competing candidates or opposing interpretations rather than assigning an isolated score to one statement. Such comparisons have shown advantages over pointwise evaluation in some natural-language evaluation and question-answering settings (Fortier-Dubois and Rosati 2023; Liusie et al. 2024; Yao and Yang 2026). This line of work motivates eliciting separate evidence for positive and negative interpretations of an atomic answer before combining those judgments through structured inference. Neuro-Symbolic Methods and Structured Inference Neuro-symbolic methods increasingly use LLMs to translate NLP problems into formal representations processed by deterministic solvers (Pan et al. 2023; Ye et al. 2023; Olausson et al. 2023). Earlier work combines uncertain model predictions under symbolic constraints. DRaiL provides a general framework for integrating neural scorers with relational rules and global inference (Zhang et al. 2016; Pacheco and Goldwasser 2021). Particularly relevant to question answering, Pujari and Goldwasser 2019 combine per-option machine-comprehension scores with NLI-based relations between answer choices and use ILP inference to obtain consistent predictions. Other approaches use satisfiability-based inference to reconcile model beliefs, answer compatibility relations, or generated explanations (Kassner et al. 2021; Mitchell et al. 2022; Jung et al. 2022). More recent work combines prompted local predictions with combinatorial inference and studies confidence elicitation, calibration, and structured learning in this setting (Mehta et al. 2024; Pauk and Pacheco 2026). Our framework is closer to this line of work than to full-problem autoformalization: the context and atomic answers remain in natural language, while the explicitly provided boolean operators determine how the local evidence is composed. 3 Framework Overview In this section, we present a framework for compound answer reasoning in which an LLM supplies local atomic evidence and a structured inference layer performs the logical composition. We parse each option into its atomic answers and operator (Sec. 3.2), elicit and calibrate evidence for each atomic answer in isolation (Sec. 3.3–Sec. 3.5), and defer composition to an integer linear program that combines this evidence under the operator semantics (Sec. 3.6) Fig. 2 gives an overview. 3.1 Task Formulation Each instance consists of a context C=(P,q)C=(P,q), where q is a question and P is an optional paragraph needed to answer it, together with four candidate answer options =A1,A2,A3,A4A=\A_1,A_2,A_3,A_4\. Unlike standard multiple choice, each option is compound. That is, it contains two atomic answers joined by an explicit boolean operator, Ai=ai(1)∘iai(2),∘i∈And,Or,Neither/Nor. gatheredA_i=a_i^(1) _i\;a_i^(2),\\[4.0pt] _i∈\ And, Or, Neither/Nor\. gathered For the question What would you use to write on a whiteboard?, one option might be a marker And chalk and another a marker And a dry-erase pen. Both contain the atomic answer a marker, but only the second is valid. Validity therefore depends on two separable things: whether each atomic answer holds in the context, and how the operator combines them. Let ϕ∘:0,12→0,1 _ :\0,1\^2→\0,1\ denote the composition rule for operator ∘ , applied to the statuses of two atomic answers: ϕ∘(ρ1,ρ2)=ρ1∧ρ2,∘=Andρ1∨ρ2,∘=Or¬ρ1∧¬ρ2,∘=Neither/Nor. _ ( _1, _2)= cases _1 _2,& = And\\[2.0pt] _1 _2,& = Or\\[2.0pt] _1 _2,& = Neither/Nor. cases That is, And requires both atoms to hold, Or requires at least one, and Neither/Nor requires that neither does. In the example, a marker and a dry-erase pen both hold while chalk does not, so ϕAnd _ And returns 1∧0=01 0=0 for the first option and 1∧1=11 1=1 for the second. Every instance is constructed so that exactly one option is valid under the gold statuses of its atoms. This is what makes the task diagnostic, as a model cannot succeed by scoring options independently, since the correct answer is determined jointly by the atomic statuses and the operators applied to them. The task is to predict the index i∗i^* of that option. Written this way, the atomic statuses carry the content of the task and ϕ∘ _ carries its logic. 3.2 Option Decomposition We deterministically parse each option into the triplet Ai=(ai(1),∘i,ai(2))A_i=(a_i^(1), _i,a_i^(2)) and collect the atomic answers appearing anywhere in the instance: C=⋃i=14ai(1),ai(2).U_C= _i=1^4 \a_i^(1),a_i^(2) \. Because CU_C collects atomic answers rather than options, an atom occurring in several options appears in it once. In our running example, a marker contributes a single element even though it occurs in two options, and any status later assigned to it applies to both. The elements of CU_C, not the compound options, are the units the rest of this sections operates on. 3.3 Contrastive Hypothesis Construction To supply evidence about the status of an atomic answer a∈Ca _C, we construct a pair of opposing natural language hypotheses conditioned on C: hC+(a) h_C^+(a) :a satisfies context C, : a satisfies context C, hC−(a) h_C^-(a) :a does not satisfy context C. : a does not satisfy context C. For the whiteboard example, the pair for chalk asserts that chalk is, and is not, something you would use to write on a whiteboard. Eliciting evidence for both members of the pair makes the result a comparison between two readings of the same atom. Prior work on comparative and contrastive evaluation finds such judgments more reliable (Liusie et al. 2024). At this stage, we do not determine which element in the pair is correct. We only produce the opposing statements that will be evaluated in the next steps. 3.4 Confidence Elicitation Next, we estimate the model’s local evidence for the two hypotheses associated with each atomic answer. Following prior work on prompt-based structured prediction (Pauk and Pacheco 2026), we present hC+(a)h_C^+(a) and hC−(a)h_C^-(a) as choices A and B within a single prompt and ask the model which is more plausible. Let ℓC+(a) _C^+(a) and ℓC−(a) _C^-(a) denote the log probabilities of the first answer tokens corresponding to the two choices. The raw evidence scores are their normalization over the two alternatives: sC,raw±(a)=exp⁡(ℓC±(a))exp⁡(ℓC+(a))+exp⁡(ℓC−(a))s_C,raw^±(a)= \! ( _C^±(a) ) \! ( _C^+(a) )+ \! ( _C^-(a) ) (1) The two scores lie in [0,1][0,1] and sum to one, so either determines the other. Because both hypotheses appear in the same prompt, the resulting score is a comparison between two readings of a rather than an isolated judgment about one of them. These are the model’s local evidence for a, obtained before any logical constraint is applied. We also evaluated three alternatives, which differ only in how support for the two hypotheses is obtained: independent true–false scoring, generation sampling, and verbalized confidence. Paired multiple choice provides the strongest evidence on our experiments (Sec. 5.1). We describe the alternatives in Appendix C. 3.5 Score Calibration Equation 1 gives the model’s relative preference between the two hypotheses, not the probability that a is correct. Since these scores are later combined across the atomic answers of an instance, they need to be comparable to one another, and a raw preference of 0.90.9 need not carry the same weight for one atom as for another. We therefore calibrate them against the gold atomic status. Because the positive and negative scores sum to one, calibrating the positive score determines the negative one, sC,cal−(a)=1−sC,cal+(a)s_C,cal^-(a)=1-s_C,cal^+(a). All calibrators are fit on atomic examples from the training set. We evaluate two standard post-hoc calibration methods. Platt scaling fits a logistic transformation of the raw positive score to the gold atomic label (Platt 1999). Isotonic calibration instead fits a non-decreasing non-parametric mapping, without assuming a sigmoid relationship between the score and correctness (Zadrozny and Elkan 2002). Relative Calibration Platt scaling and isotonic calibration adjust each atomic score based only on its absolute value. However, exactly one option in an instance is valid, so which atoms are selected depends on how they stand relative to the others present. Two atoms scored 0.910.91 and 0.860.86 support different conclusions depending on whether the remaining atoms sit near 0.90.9 or near 0.20.2, and an independent mapping cannot distinguish these cases. We therefore introduce relative calibration, which supplies both a score’s magnitude and its standing within the instance. For each atomic answer a, we construct the feature vector fC(a)=[logit⁡(sC,raw+(a))zC(a)rankC⁡(a)sC,max+−sC,raw+(a)],f_C(a)= bmatrixlogit\! (s_C,raw^+(a) )\\ z_C(a)\\ rank_C(a)\\ s_C, ^+-s_C,raw^+(a) bmatrix, where zC(a)z_C(a) is the within-instance standardized score, rankC⁡(a)rank_C(a) is its rank among the atomic scores, and sC,max+s_C, ^+ is the highest positive score in the instance. A logistic regression maps these features to the calibrated positive score: sC,cal+(a)=σ⁡(⊤fC(a)+b),s_C,cal^+(a)=σ\! (w f_C(a)+b ), with w and b learned from the training set. 3.6 Globally Constrained Inference The confidence elicitation and calibration stages produce continuous local evidence for each atomic answer, with no reference to the operators. Following prompt-based structured prediction (Mehta et al. 2024; Pauk and Pacheco 2026), we combine these scores under global constraints that encode the logical structure of the compound answer options using Integer Linear Programming (ILP). Going forward, we write sC±(a)s_C^±(a) for the calibrated scores sC,cal±(a)s_C,cal^±(a). The uncalibrated variant we report in Sec. 5 substitutes sC,raw±(a)s_C,raw^±(a) throughout. Decision Variables We formulate a binary ILP over two sets of variables. For each atomic answer a∈Ca _C, ya∈0,1y_a∈\0,1\ is its inferred status, with ya=1y_a=1 when a satisfies C. Similarly, for each compound option Ai=ai(1)∘iai(2)A_i=a_i^(1) _i\;a_i^(2), xi∈0,1x_i∈\0,1\ is its inferred validity, taking the value of 1 when the statuses assigned to its two atomic answers satisfy the operator ∘i _i and 0 otherwise. Because CU_C holds each atomic answer once, an atom occurring in several options has a single variable, and the status it receives applies to all of them. Given that our task formulation requires exactly one compound option to be valid, the unique option for which xi=1x_i=1 is returned as the final prediction. We collect all these variables into ∈0,1|C|y∈\0,1\^|U_C| and ∈0,14x∈\0,1\^4. Operator Constraints The validity of each compound option Ai=ai(1)∘iai(2)A_i=a_i^(1) _i\;a_i^(2) must equal the composition rule of Sec. 3.1 applied to the inferred statuses of its atoms, xi=ϕ∘i(yai(1),yai(2)),x_i= _ _i\! (y_a_i^(1),\,y_a_i^(2) ), which we encode exactly with linear inequalities. Writing y1y_1 and y2y_2 for yai(1)y_a_i^(1) and yai(2)y_a_i^(2), respectively: And:xi≤y1,xi≤y2,xi≥y1+y2−1Or:xi≥y1,xi≥y2,xi≤y1+y2NNor:xi≤1−y1,xi≤1−y2,xi≥1−y1−y2 array[]l And:&x_i≤ y_1,\ \ x_i≤ y_2,\\ &x_i≥ y_1+y_2-1\\[5.0pt] Or:&x_i≥ y_1,\ \ x_i≥ y_2,\\ &x_i≤ y_1+y_2\\[5.0pt] NNor:&x_i≤ 1-y_1,\ \ x_i≤ 1-y_2,\\ &x_i≥ 1-y_1-y_2 array The Neither/Nor constraints are the complement of the Or constraints, as ¬(y1∨y2) (y_1 y_2) requires that every assignment satisfying the disjunction be excluded. Since for our task exactly one option is valid by construction (Sec. 3.1), we additionally require ∑i=14xi=1 _i=1^4x_i=1. Objective Among the assignments satisfying these constraints, we select the one carrying the strongest total evidence by optimizing: max, _y,\,x ∑a∈C[sC+(a)ya+sC−(a)(1−ya)] _a _C [s_C^+(a)\,y_a+s_C^-(a)\,(1-y_a) ] s.t. .t. (,)∈ℱC (y,x) _C where ℱCF_C denotes the set of feasible assignments, that is, the pairs (,)∈0,1|C|×0,14(y,x)∈\0,1\^|U_C|×\0,1\^4 that satisfy all constraints. This objective scores only the atomic assignment, where the first term (sC+s_C^+) contributes the supporting evidence, while the second (sC−s_C^-) contributes the opposing evidence (Sec. 3.3). The constraints then determine which option follows from it. The predicted answer is the unique index ı with xı^=1x_ =1. 4 Benchmark Datasets We evaluate our framework on two benchmarks that follow the compound answer reasoning task formulation in Section 3.1 but differ in the source of their atomic labels: contextual commonsense plausibility and reading-comprehension ratings. Logical-CommonsenseQA evaluates the interaction between commonsense judgment and logical composition (Junias and Pacheco 2026). Each instance contains a commonsense question and four compound answer options, each joining two atomic answers under a logical operator. For Where do you see tiny bottles of shampoo when away from home?, one option is hotels And gym showers. It contains 19,996 instances (11,996 train / 6,000 dev / 2,000 test), evenly distributed across four settings: three in which all four options share the same operator (And, Or, Neither/Nor), and a Mixed setting in which operators may differ across the options of an instance. The test set is further divided into human-validated (HV) and non-validated (NV) subsets of 1,000 each. Logical-SATA is constructed from the human-labeled training partition of SATA-Bench (Xu et al. 2025), which uses a select-all-that-apply format in which multiple answer choices may be correct for a paragraph-based reading-comprehension question. We pair these annotated answers into compound options of the same form as Logical-CommonsenseQA, two atomic answers joined by a logical operator. We first remove duplicate source instances and retain questions containing at least two correct and at least three incorrect answers, which is the minimum needed to build one valid option and three distractors. This yields 1,390 eligible source questions, of which 1,350 are selected to obtain balanced training, development, and test splits. For each source question, we partition the original choices into correct and incorrect atomic answers and construct valid and invalid compound option pools according to operator semantics. We construct operator-specific instances (And, Or, Neither/Nor), in which the gold option and distractors are sampled from the corresponding pools. We also construct Mixed instances, in which candidates from all three operators are combined into a single pool before sampling one valid option and three distractors. The resulting dataset contains 5,400 instances (2,400 train / 1,000 dev / 2,000 test), evenly distributed across the operator settings. Appendix D includes additional details and examples for both benchmarks, as well as a schema summarizing the construction of Logical-SATA. 5 Experiments Experimental Settings All experiments use Llama-3.1-8B-Instruct (Dubey et al. 2024), hereafter Llama-8B, and temperature 0.7, with results averaged over five runs. We fit calibrators on the training set and report results on the test sets. We report Macro-F1, and Brier score and log loss for atomic calibration quality. Parameters and other implementation details are included in App. A. 5.1 Main Results Tables 1 and 2 compare our structured-inference framework with direct LLaMA-8B prompting under zero- through three-shot prompting and zero-shot chain-of-thought prompting. We report Macro-F1 on the human-validated (HV) split of Logical-CommonsenseQA and the Logical-SATA test set. All results are reported as mean ± standard deviation over five runs. Results on the non-validated (NV) test split for Logical-CommonsenseQA are provided in App B. Results using alternative scoring strategies are reported in App C.4. Direct Prompting vs. Structured Inference Structured inference substantially outperforms direct prompting across both benchmarks. On the Logical-CommonsenseQA-HV split, macro-F1 for the strongest direct-prompting configuration is 48.3, whereas our paired multiple-choice evidence with globally constrained inference achieves 75.8, an improvement of 27.5 points. Relative calibration further increases the performance to 77.0. The same pattern holds for Logical-SATA benchmark, where paired multiple-choice structured inference obtains 72.2 macro-F1, compared to 47.0 for the strongest direct-prompting baseline. These results provide evidence that explicitly separating atomic evaluation from logical composition can reduce the compositionality gap observed under direct compound-answer prediction. Performance Across Logical Operators The gains from structured inference are concentrated on Or, Neither/Nor, and Mixed, with the largest improvement on Neither/Nor. The strongest direct-prompting baseline reaches only 14.0 macro-F1 on Logical-CommonsenseQA and 12.6 on Logical-SATA, whereas paired multiple-choice structured inference raises these to 75.1, and 71.9, respectively, and relative to 76.8 and 73.4. This recovery suggests that models retain useful evidence about the individual atomic answers even when they fail to combine two negative judgments correctly during direct compound-answer prediction. On And, gains are smaller: 70.8 to 72.4 on Logical-CommonsenseQA and 70.9 to 73.6 on Logical-SATA. These results indicate that explicit logical composition is most beneficial when the final decision requires handling alternatives, jointly rejecting atomic answers, or applying different operators across candidate options. Method And Or N Mix All Direct prompting 0-shot 68.2±1.168.2^± 1.1 55.7±2.455.7^± 2.4 14.0±1.814.0^± 1.8 47.2±2.047.2^± 2.0 46.5±1.346.5^± 1.3 1-shot 70.8±0.670.8^± 0.6 54.4±1.954.4^± 1.9 8.7±0.88.7^± 0.8 43.5±1.143.5^± 1.1 44.5±0.444.5^± 0.4 2-shot 64.2±2.164.2^± 2.1 53.3±3.753.3^± 3.7 9.4±1.59.4^± 1.5 41.1±3.541.1^± 3.5 42.2±1.442.2^± 1.4 3-shot 57.4±3.157.4^± 3.1 52.9±3.052.9^± 3.0 8.5±0.88.5^± 0.8 40.9±2.240.9^± 2.2 40.1±1.040.1^± 1.0 CoT (0-shot) 70.1±1.770.1^± 1.7 62.3±0.762.3^± 0.7 13.9±0.713.9^± 0.7 46.1±2.746.1^± 2.7 48.3±0.548.3^± 0.5 Structured inference Paired MC 72.4±0.072.4^± 0.0 85.2±0.085.2^± 0.0 75.1±0.075.1^± 0.0 70.3±0.070.3^± 0.0 75.8±0.075.8^± 0.0 + Platt 72.0±0.072.0^± 0.0 84.4±0.084.4^± 0.0 75.6±0.075.6^± 0.0 73.7±0.273.7^± 0.2 76.4±0.176.4^± 0.1 + Isotonic 72.6±0.372.6^± 0.3 83.6±0.283.6^± 0.2 75.2±0.075.2^± 0.0 71.5±0.471.5^± 0.4 75.7±0.275.7^± 0.2 + Relative 71.8±0.271.8^± 0.2 84.0±0.084.0^± 0.0 76.8±0.076.8^± 0.0 75.2±0.075.2^± 0.0 77.0±0.177.0^± 0.1 Table 1: Macro-F1 on the human-validated (HV) split of Logical-CommonsenseQA. N = Neither/Nor. Method AND OR N MIX All Direct prompting 0-shot 64.4±1.264.4^± 1.2 58.7±1.658.7^± 1.6 12.1±1.012.1^± 1.0 38.6±0.938.6^± 0.9 44.0±0.644.0^± 0.6 1-shot 70.9±1.170.9^± 1.1 60.8±0.660.8^± 0.6 9.3±1.09.3^± 1.0 36.4±1.836.4^± 1.8 44.4±0.844.4^± 0.8 2-shot 70.9±1.370.9^± 1.3 60.4±1.660.4^± 1.6 8.9±1.38.9^± 1.3 38.0±1.238.0^± 1.2 44.8±0.544.8^± 0.5 3-shot 70.3±0.870.3^± 0.8 59.2±0.959.2^± 0.9 9.9±1.29.9^± 1.2 38.1±1.238.1^± 1.2 44.6±0.444.6^± 0.4 CoT (0-shot) 69.8±1.469.8^± 1.4 65.7±2.065.7^± 2.0 12.6±0.812.6^± 0.8 38.5±0.738.5^± 0.7 47.0±0.847.0^± 0.8 Structured inference Paired MC 73.6±0.073.6^± 0.0 82.2±0.082.2^± 0.0 71.9±0.071.9^± 0.0 60.9±0.060.9^± 0.0 72.2±0.072.2^± 0.0 + Platt 74.8±0.074.8^± 0.0 82.4±0.082.4^± 0.0 73.0±0.073.0^± 0.0 70.4±0.070.4^± 0.0 75.2±0.075.2^± 0.0 + Isotonic 74.6±0.074.6^± 0.0 83.7±0.083.7^± 0.0 73.0±0.073.0^± 0.0 69.7±0.069.7^± 0.0 75.3±0.075.3^± 0.0 + Relative 74.4±0.074.4^± 0.0 82.2±0.082.2^± 0.0 73.4±0.073.4^± 0.0 72.1±0.072.1^± 0.0 75.6±0.075.6^± 0.0 Table 2: Macro-F1 on the Logical-SATA test set. Effects of Calibration Calibration improves both the reliability of the atomic evidence scores and downstream compound prediction. As shown in Table 3, Platt scaling, isotonic calibration, and relative calibration all reduce atomic Brier score and log loss on both benchmarks, with relative calibration performing best on both metrics. This indicates that incorporating within-instance information improves the calibration of the local evidence scores. The reduction is proportionally larger for log loss than for Brier score, particularly on Logical-SATA, indicating that the raw scores are overconfident rather than merely out of order. The downstream gains are smaller than the atomic improvements, since calibration changes the final prediction only when it alters the relative evidence among competing feasible assignments. Relative calibration’s largest downstream gains occur in the Mixed setting, where Macro-F1 increases by 4.9 on Logical-CommonsenseQA, and 11.2 points on Logical-SATA. When all four options share an operator, a systematic bias in the atomic scores shifts them equally and it largely cancels. In Mixed, options impose opposing demands, since And and Or require atoms to be accepted while Neither/Nor requires them to be rejected, so the same bias favors one operator over another. Platt scaling and isotonic calibration apply a single global mapping and cannot correct this, whereas relative calibration’s features are defined against the other atoms in the instance. Error Analysis When the inference layer is provided with gold atomic statuses, accuracy reaches 1.001.00 on both benchmarks. This follows from construction, as each instance has exactly one valid option, and the ILP encodes the operator semantics exactly. The informative quantity is thus how many atomic errors survive composition. Atomic accuracy is 0.830 on Logical-CommonsenseQA-HV and 0.824 on Logical-SATA, against a compound accuracy of 0.758 and 0.723. Qualitative analysis reveals different sources of atomic error across the benchmarks. On Logical-CommonsenseQA, errors often involve broad interpretations of open-ended commonsense questions or insufficient attention to modifiers such as uncommon. On Logical-SATA, the model often selects the label matching a passage’s main topic, rejecting other labels that also apply. The logical operators then determine how these errors propagate: Or can tolerate an incorrect atomic judgment when another atomic answer remains supported, whereas And and Neither/Nor can be invalidated by a single incorrect assignment. Mixed instances are especially sensitive because the same atomic assignment can affect options governed by different operators. A full analysis and examples are provided in Appendix C.5. Benchmark Method Brier ↓ Log loss ↓ LSATA Uncalibrated 0.1896 0.8141 Platt 0.1493 0.4567 Isotonic 0.1483 0.4531 Relative 0.1449 0.4438 LCQA-HV Uncalibrated 0.1919 0.6368 Platt 0.1681 0.5076 Isotonic 0.1680 0.5145 Relative 0.1464 0.4567 Table 3: Atomic calibration results on LSATA and the LCQA’s human-validated split. Brier score is the mean squared error between the predicted probability that an atomic answer is supported and its gold binary status. Log loss is the negative log-likelihood of that status under the predicted probability. Lower is better for both, but log loss penalizes confident errors far more heavily. 6 Conclusions and Future Work We study reasoning over compound answer options, where two atomic answers are joined by And, Or, or Neither/Nor. Direct prediction requires evaluating the atomic answers and composing them in a single step. We propose a framework that separates these stages by eliciting local evidence for opposing hypotheses about each atomic answer and combining it through globally constrained inference. We also introduce relative calibration, which incorporates how each atomic score compares with the other scores in the same instance. Across Logical-CommonsenseQA and Logical-SATA, structured inference substantially outperforms direct prompting, with the largest improvements on Neither/Nor, and relative calibration performs best overall, with largest gains on Mixed. More broadly, these results suggest that some failures on logical reasoning tasks reflect difficulties in composing local judgments, rather than only the absence of relevant knowledge. Future work could extend the framework to options with more than two atomic answers, to structures such as implication, exclusive disjunction, and nested expressions, and to settings where the atomic answers and operators must be extracted from less structured text. A further direction is to replace inference with a probabilistic formulation, allowing atomic uncertainty to propagate to the compound prediction rather than being discarded at assignment time, and yielding a distribution over options instead of a single choice. Evaluating the framework across additional model families and reasoning benchmarks would help determine how broadly the compositionality gap generalizes. Limitations Our evaluation uses a single model, Llama-3.1-8B-Instruct, and two benchmarks with explicit binary operators over pairs of atomic answers. The results therefore do not establish that the same gains will hold for other model families, larger models, longer logical expressions, or operators such as implication and exclusive disjunction. The benchmarks also enforce exactly one valid compound option, whereas real tasks may permit multiple valid answers or no valid answer. The framework depends on the quality of the atomic evidence supplied to the inference layer. Errors in commonsense interpretation, passage grounding, or source annotations can therefore propagate to the final prediction even when the logical constraints are applied correctly. Logical-CommonsenseQA may additionally contain questions with several plausible commonsense interpretations, while Logical-SATA inherits the label definitions and domain coverage of SATA-Bench. Finally, the reported results may be sensitive to prompt design, calibration data, and the choice of confidence-elicitation strategy. Ethical Considerations This work uses publicly available data and does not involve human-subject data collection. Nevertheless, the benchmarks and model outputs may reflect biases, ambiguities, or annotation errors present in their source datasets. The proposed framework improves consistency under explicit logical constraints, but it does not by itself guarantee that the underlying atomic judgments are correct. It should therefore not be interpreted as providing reliable logical guarantees for high-stakes applications such as medical, legal, or financial decision-making. We used generative AI assistance in accordance with the ACL Policy on Publication Ethics. Its use was limited to language editing and compression, manuscript organization, LaTeX formatting, and figure preparation. All output was reviewed, verified and further edited by the authors. References Bisk et al. (2019) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Dalvi et al. (2021) Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entailment trees. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 7358–7370. DeWall et al. (2008) C Nathan DeWall, Roy F Baumeister, and EJ Masicampo. 2008. Evidence that logical reasoning depends on conscious processing. Consciousness and Cognition, 17(3):628–645. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others. 2024. The llama 3 herd of models. Fortier-Dubois and Rosati (2023) Etienne Fortier-Dubois and Domenic Rosati. 2023. Using contradictions improves question answering systems. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 827–840, Toronto, Canada. Association for Computational Linguistics. García-Ferrero et al. (2023) Iker García-Ferrero, Begoña Altuna, Javier Alvez, Itziar Gonzalez-Dios, and German Rigau. 2023. This is not a dataset: A large negation benchmark to challenge large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 8596–8615. García-Madruga et al. (2001) Juan A García-Madruga, Sergio Moreno, Nuria Carriedo, Francisco Gutiérrez, and Philip N Johnson-Laird. 2001. Are conjunctive inferences easier than disjunctive inferences? a comparison of rules and models. The Quarterly Journal of Experimental Psychology Section A, 54(2):613–632. Han et al. (2024) Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, and 16 others. 2024. FOLIO: Natural language reasoning with first-order logic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22017–22031, Miami, Florida, USA. Association for Computational Linguistics. Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Hoveyda et al. (2026) Mohanna Hoveyda, Jelle Piepenbrock, Arjen P. de Vries, Maarten de Rijke, and Faegheh Hasibi. 2026. Orlog: Resolving complex queries with llms and probabilistic reasoning. In Advances in Information Retrieval: 48th European Conference on Information Retrieval, ECIR 2026, Delft, The Netherlands, March 29 – April 2, 2026, Proceedings, Part I, page 98–114, Berlin, Heidelberg. Springer-Verlag. Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977. Johnson-Laird et al. (1992) Philip N Johnson-Laird, Ruth M Byrne, and Walter Schaeken. 1992. Propositional reasoning by model. Psychological review, 99(3):418. Jung et al. (2022) Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1266–1279, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Junias and Pacheco (2026) Obed Junias and Maria Leonor Pacheco. 2026. LOGICAL-COMMONSENSEQA: A benchmark for logical commonsense reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 746–758, San Diego, California, United States. Association for Computational Linguistics. Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Thomas Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova Dassarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. Language models (mostly) know what they know. ArXiv, abs/2207.05221. Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7811–7818. Kassner et al. (2021) Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. 2021. BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8849–8861, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Khalid et al. (2025) Irtaza Khalid, Amir Masoud Nourollah, and Steven Schockaert. 2025. Large language and reasoning models are shallow disjunctive reasoners. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8843–8869, Vienna, Austria. Association for Computational Linguistics. Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252–262, New Orleans, Louisiana. Association for Computational Linguistics. Khemlani et al. (2014) Sangeet Khemlani, Isabel Orenes, and Philip N Johnson-Laird. 2014. The negations of conjunctions, conditionals, and disjunctions. Acta Psychologica, 151:1–7. Khot et al. (2023) Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. Preprint, arXiv:2210.02406. Klauer (1997) Karl Christoph Klauer. 1997. Working memory involvement in propositional and spatial reasoning. Thinking & Reasoning, 3(1):9–47. Liu et al. (2020) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3622–3628. International Joint Conferences on Artificial Intelligence Organization. Main track. Liusie et al. (2024) Adian Liusie, Vatsal Raina, Yassir Fathullah, and Mark Gales. 2024. Efficient LLM comparative assessment: A product of experts framework for pairwise comparisons. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6835–6855, Miami, Florida, USA. Association for Computational Linguistics. Macbeth et al. (2014) Guillermo Macbeth, Eugenia Razumiejczyk, María C Crivello, Claudia Bolzán, Carolina I Pereyra Girardi, and Guillermo Campitelli. 2014. Mental models for the negation of conjunctions and disjunctions. Mehta et al. (2024) Maitrey Mehta, Valentina Pyatkin, and Vivek Srikumar. 2024. Promptly predicting structures: The return of inference. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 112–130, Mexico City, Mexico. Association for Computational Linguistics. Meiser et al. (2001) Thorsten Meiser, Karl Christoph Klauer, and Birgit Naumer. 2001. Propositional reasoning and working memory: the role of prior training and pragmatic content. Acta Psychologica, 106(3):303–327. Mitchell et al. (2022) Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1754–1768, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Neys (2006) Wim De Neys. 2006. Dual processing in reasoning: Two systems but one reasoner. Psychological science, 17(5):428–433. Olausson et al. (2023) Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, and Roger Levy. 2023. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5153–5176, Singapore. Association for Computational Linguistics. Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. Pacheco and Goldwasser (2021) Maria Leonor Pacheco and Dan Goldwasser. 2021. Modeling content and context with deep relational learning. Transactions of the Association for Computational Linguistics, 9:100–119. Palta et al. (2024) Shramay Palta, Nishant Balepur, Peter Rankel, Sarah Wiegreffe, Marine Carpuat, and Rachel Rudinger. 2024. Plausibly problematic questions in multiple-choice benchmarks for commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3451–3473, Miami, Florida, USA. Association for Computational Linguistics. Pan et al. (2023) Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3806–3824, Singapore. Association for Computational Linguistics. Parmar et al. (2024) Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13679–13707. Pauk and Pacheco (2026) Matt Pauk and Maria Leonor Pacheco. 2026. Mapping the course for prompt-based structured prediction. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3483–3508, Rabat, Morocco. Association for Computational Linguistics. Platt (1999) John Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Press et al. (2023) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore. Association for Computational Linguistics. Pujari and Goldwasser (2019) Rajkumar Pujari and Dan Goldwasser. 2019. Using natural language relations between answer choices for machine comprehension. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4010–4015, Minneapolis, Minnesota. Association for Computational Linguistics. Ravichander et al. (2022) Abhilasha Ravichander, Matt Gardner, and Ana Marasović. 2022. Condaqa: A contrastive reading comprehension dataset for reasoning about negation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8729–8755. Saha et al. (2020) Swarnadeep Saha, Yixin Nie, and Mohit Bansal. 2020. Conjnli: Natural language inference over conjunctive sentences. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8240–8252. Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China. Association for Computational Linguistics. She et al. (2023) Jingyuan S. She, Christopher Potts, Samuel R. Bowman, and Atticus Geiger. 2023. ScoNe: Benchmarking negation reasoning in language models with fine-tuning and in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1803–1821, Toronto, Canada. Association for Computational Linguistics. Tafjord et al. (2021) Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621–3634, Online. Association for Computational Linguistics. Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. Tian et al. (2021) Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021. Diagnosing the first-order logical reasoning ability through LogicNLI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3738–3747, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442, Singapore. Association for Computational Linguistics. Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. Weir et al. (2024) Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, and Benjamin Van Durme. 2024. Enhancing systematic decompositional natural language inference using informal logic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9458–9482, Miami, Florida, USA. Association for Computational Linguistics. Xu et al. (2025) Weijie Xu, Shixian Cui, Xi Fang, Chi Xue, Stephanie Eckman, and Chandan K Reddy. 2025. Sata-bench: Select all that apply benchmark for multiple choice questions. arXiv preprint arXiv:2506.00643. Yao and Yang (2026) Liang Yao and Yang Yang. 2026. Large language models are contrastive reasoners. Expert Systems with Applications, 301:130407. Ye et al. (2023) Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2023. Satlm: Satisfiability-aided language models using declarative prompting. Advances in Neural Information Processing Systems, 36:45548–45580. Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.04326. Zadrozny and Elkan (2002) Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694–699. Zhang et al. (2016) Xiao Zhang, Maria Leonor Pacheco, Chang Li, and Dan Goldwasser. 2016. Introducing DRAIL – a step towards declarative deep relational learning. In Proceedings of the Workshop on Structured Prediction for NLP, pages 54–62, Austin, TX. Association for Computational Linguistics. Zhong et al. (2023) Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettlemoyer. 2023. RoMQA: A benchmark for robust, multi-evidence, multi-answer question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7055–7067, Singapore. Association for Computational Linguistics. Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and 1 others. 2022. Least-to-most prompting enables complex reasoning in large language models. In The eleventh international conference on learning representations. Appendix Appendix A Implementation Details All experiments use Llama-3.1-8B-Instruct (Dubey et al. 2024). Direct-prompting baselines use zero- through three-shot prompting and an additional zero-shot chain-of-thought baseline on both benchmarks. Atomic confidence scores are generated at a temperature of 0.70.7. Direct-prompting and structured-inference results are averaged over five runs. Generation sampling uses five generations per atomic answer, and all experiments use a random seed of 4242. For Logical-SATA, calibration uses the full training set of 2,4002,400 instances. For Logical-CommonsenseQA, we sample 2,4002,400 training instances, balanced across the four logical settings, to match the Logical-SATA calibration-set size. Global inference is performed using Gurobi Optimizer 13.0.2. Experiments are run on an NVIDIA A100 GPU. Appendix B Additional Logical-CommonsenseQA Results Table 4 reports Macro-F1 on the non-validated Logical-CommonsenseQA split. Method And Or N Mix Overall Direct prompting 0-shot 62.5±1.762.5^± 1.7 57.2±3.357.2^± 3.3 13.7±2.013.7^± 2.0 45.1±1.345.1^± 1.3 44.9±0.944.9^± 0.9 1-shot 68.4±1.168.4^± 1.1 59.0±1.359.0^± 1.3 7.2±1.67.2^± 1.6 42.4±0.642.4^± 0.6 44.3±0.544.3^± 0.5 2-shot 63.3±1.263.3^± 1.2 52.6±1.452.6^± 1.4 7.8±1.67.8^± 1.6 43.5±1.243.5^± 1.2 41.9±0.441.9^± 0.4 3-shot 56.7±0.956.7^± 0.9 51.1±1.851.1^± 1.8 7.3±2.37.3^± 2.3 43.1±2.643.1^± 2.6 39.7±1.739.7^± 1.7 CoT (0-shot) 66.8±2.866.8^± 2.8 65.3±2.165.3^± 2.1 11.7±1.911.7^± 1.9 48.6±1.248.6^± 1.2 48.3±0.748.3^± 0.7 Structured inference Paired MC 73.7±0.273.7^± 0.2 85.1±0.085.1^± 0.0 77.9±0.277.9^± 0.2 67.9±0.067.9^± 0.0 76.2±0.176.2^± 0.1 + Platt 74.8±0.474.8^± 0.4 85.1±0.085.1^± 0.0 77.6±0.077.6^± 0.0 68.2±0.068.2^± 0.0 76.5±0.176.5^± 0.1 + Isotonic 74.9±0.274.9^± 0.2 86.0±0.086.0^± 0.0 76.4±0.076.4^± 0.0 68.0±0.268.0^± 0.2 76.4±0.076.4^± 0.0 + Relative 74.0±0.274.0^± 0.2 85.1±0.085.1^± 0.0 78.0±0.078.0^± 0.0 74.3±0.074.3^± 0.0 77.9±0.177.9^± 0.1 Table 4: Macro-F1 on the non-validated (NV) split of Logical-CommonsenseQA. N = Neither/Nor. B.1 Full Accuracy Results Tables 5 and 6 report accuracy results for the direct-prompting and structured-inference configurations. And Or N Mixed Overall Method NV HV NV HV NV HV NV HV NV HV Direct prompting 0-shot 62.6±1.862.6^± 1.8 68.1±1.068.1^± 1.0 57.3±3.357.3^± 3.3 55.6±2.755.6^± 2.7 14.5±2.114.5^± 2.1 14.6±1.914.6^± 1.9 45.4±1.345.4^± 1.3 47.5±2.247.5^± 2.2 44.9±1.044.9^± 1.0 46.5±1.446.5^± 1.4 1-shot 68.5±1.268.5^± 1.2 70.8±0.670.8^± 0.6 59.4±1.359.4^± 1.3 54.5±2.154.5^± 2.1 7.4±1.67.4^± 1.6 8.8±0.88.8^± 0.8 42.4±0.642.4^± 0.6 43.8±0.943.8^± 0.9 44.4±0.444.4^± 0.4 44.5±0.544.5^± 0.5 2-shot 62.8±0.862.8^± 0.8 63.0±2.463.0^± 2.4 52.8±1.552.8^± 1.5 53.3±3.453.3^± 3.4 8.6±1.58.6^± 1.5 10.2±1.210.2^± 1.2 43.0±1.243.0^± 1.2 40.9±3.440.9^± 3.4 41.8±0.441.8^± 0.4 41.8±1.441.8^± 1.4 3-shot 56.9±0.956.9^± 0.9 56.6±3.056.6^± 3.0 50.8±1.750.8^± 1.7 52.5±3.152.5^± 3.1 8.4±2.48.4^± 2.4 9.4±0.69.4^± 0.6 42.1±2.742.1^± 2.7 40.6±2.240.6^± 2.2 39.5±1.739.5^± 1.7 39.8±0.939.8^± 0.9 CoT (0-shot) 66.5±2.866.5^± 2.8 69.9±1.869.9^± 1.8 65.0±2.265.0^± 2.2 62.3±0.762.3^± 0.7 12.1±2.012.1^± 2.0 14.3±0.814.3^± 0.8 48.4±1.248.4^± 1.2 45.8±2.545.8^± 2.5 48.0±0.848.0^± 0.8 48.1±0.548.1^± 0.5 Structured inference Paired MC 73.9±0.273.9^± 0.2 72.4±0.072.4^± 0.0 85.2±0.085.2^± 0.0 85.2±0.085.2^± 0.0 77.8±0.277.8^± 0.2 75.2±0.075.2^± 0.0 68.0±0.068.0^± 0.0 70.4±0.070.4^± 0.0 76.2±0.176.2^± 0.1 75.8±0.075.8^± 0.0 + Platt calibration 75.0±0.475.0^± 0.4 72.0±0.072.0^± 0.0 85.2±0.085.2^± 0.0 84.4±0.084.4^± 0.0 77.6±0.077.6^± 0.0 75.6±0.075.6^± 0.0 68.4±0.068.4^± 0.0 73.8±0.273.8^± 0.2 76.6±0.176.6^± 0.1 76.4±0.176.4^± 0.1 + Isotonic calibration 75.1±0.275.1^± 0.2 72.5±0.372.5^± 0.3 86.1±0.086.1^± 0.0 83.5±0.283.5^± 0.2 76.4±0.076.4^± 0.0 75.2±0.075.2^± 0.0 68.2±0.268.2^± 0.2 71.7±0.371.7^± 0.3 76.5±0.076.5^± 0.0 75.7±0.275.7^± 0.2 + Relative calibration 74.2±0.274.2^± 0.2 71.8±0.271.8^± 0.2 85.2±0.085.2^± 0.0 84.0±0.084.0^± 0.0 78.0±0.078.0^± 0.0 76.8±0.076.8^± 0.0 74.4±0.074.4^± 0.0 75.2±0.075.2^± 0.0 78.0±0.178.0^± 0.1 77.0±0.177.0^± 0.1 Table 5: Accuracy on the non-validated (NV) and human-validated (HV) subsets of Logical-CommonsenseQA. Method And Or N Mixed Overall Direct prompting 0-shot 62.1±1.462.1^± 1.4 57.4±1.657.4^± 1.6 12.0±0.912.0^± 0.9 37.3±0.937.3^± 0.9 42.2±0.742.2^± 0.7 1-shot 70.9±1.170.9^± 1.1 61.2±0.661.2^± 0.6 9.2±1.09.2^± 1.0 36.7±1.636.7^± 1.6 44.5±0.844.5^± 0.8 2-shot 70.8±1.370.8^± 1.3 60.9±1.660.9^± 1.6 8.9±1.38.9^± 1.3 38.4±1.338.4^± 1.3 44.8±0.544.8^± 0.5 3-shot 70.2±0.870.2^± 0.8 59.7±0.859.7^± 0.8 9.8±1.29.8^± 1.2 38.4±1.138.4^± 1.1 44.5±0.444.5^± 0.4 CoT (0-shot) 67.0±1.467.0^± 1.4 63.1±2.163.1^± 2.1 12.4±0.912.4^± 0.9 37.1±0.637.1^± 0.6 44.9±0.844.9^± 0.8 Structured inference Paired MC 73.6±0.073.6^± 0.0 82.4±0.082.4^± 0.0 72.0±0.072.0^± 0.0 61.0±0.061.0^± 0.0 72.3±0.072.3^± 0.0 + Platt calibration 74.8±0.074.8^± 0.0 82.6±0.082.6^± 0.0 73.0±0.073.0^± 0.0 70.4±0.070.4^± 0.0 75.2±0.075.2^± 0.0 + Isotonic calibration 74.6±0.074.6^± 0.0 83.8±0.083.8^± 0.0 73.0±0.073.0^± 0.0 69.8±0.069.8^± 0.0 75.3±0.075.3^± 0.0 + Relative calibration 74.4±0.074.4^± 0.0 82.4±0.082.4^± 0.0 73.4±0.073.4^± 0.0 72.2±0.072.2^± 0.0 75.6±0.075.6^± 0.0 Table 6: Accuracy on the Logical-SATA test set. Appendix C Alternative Confidence Elicitation Strategies In addition to paired multiple-choice confidence, we evaluate three alternative strategies that differ in how they obtain support for the positive and negative hypotheses. Each method produces non-negative support values rC+(a),rC−(a),r_C^+(a),\,r_C^-(a), which we normalize to obtain raw evidence scores: sC,raw+(a) s_C,raw^+(a) =rC+(a)rC+(a)+rC−(a), = r_C^+(a)r_C^+(a)+r_C^-(a), sC,raw−(a) s_C,raw^-(a) =rC−(a)rC+(a)+rC−(a). = r_C^-(a)r_C^+(a)+r_C^-(a). C.1 Independent True–False Confidence We evaluate the positive and negative hypotheses independently using separate prompts. For each hypothesis, we extract the probability assigned to the True answer token: rC+(a) r_C^+(a) =Pr⁡(True∣C,hC+(a)), = \! ( True C,h_C^+(a) ), rC−(a) r_C^-(a) =Pr⁡(True∣C,hC−(a)). = \! ( True C,h_C^-(a) ). The two independently obtained values are then normalized to form relative evidence scores. C.2 Generation Sampling We use the same paired prompt as in the main method but estimate confidence through repeated stochastic generation rather than token probabilities. We sample the model N times and parse each generation gng_n as selecting either the positive hypothesis, represented by A, or the negative hypothesis, represented by B. The support values are their empirical selection frequencies: rC+(a) r_C^+(a) =1N∑n=1N[gn=A], = 1N _n=1^NI[g_n= A], rC−(a) r_C^-(a) =1N∑n=1N[gn=B]. = 1N _n=1^NI[g_n= B]. Because every valid generation selects one of the two alternatives, these values already sum to one. The sampling parameters are reported in Appendix A. C.3 Verbalized Confidence We evaluate the positive and negative hypotheses in separate model calls and ask the model to report a numerical confidence score between 00 and 1010. The parsed responses define rC+(a)r_C^+(a) and rC−(a)r_C^-(a), which are normalized to obtain the corresponding raw evidence scores. Unlike the other strategies, verbalized confidence relies on the model’s self-reported numerical judgment rather than answer-token probabilities or repeated selections. C.4 Comparison of Confidence Elicitation Strategies We compare four strategies for eliciting local evidence: paired multiple-choice confidence, independent true–false confidence, generation sampling, and verbalized confidence. Results are reported as mean ± standard deviation over five runs. Method AND OR N MIX Overall NV F1 HV F1 NV F1 HV F1 NV F1 HV F1 NV F1 HV F1 NV F1 HV F1 Paired MC 73.7±0.273.7± 0.2 72.4±0.072.4± 0.0 85.1±0.085.1± 0.0 85.2±0.085.2± 0.0 77.9±0.277.9± 0.2 75.1±0.075.1± 0.0 67.9±0.067.9± 0.0 70.3±0.070.3± 0.0 76.2±0.176.2± 0.1 75.8±0.075.8± 0.0 Independent T/F 74.7±0.074.7± 0.0 70.8±0.070.8± 0.0 81.6±0.081.6± 0.0 80.8±0.080.8± 0.0 75.6±0.075.6± 0.0 73.6±0.073.6± 0.0 66.3±0.066.3± 0.0 70.3±0.070.3± 0.0 74.6±0.074.6± 0.0 73.9±0.073.9± 0.0 Generation sampling 70.7±1.770.7± 1.7 69.5±0.869.5± 0.8 84.3±0.684.3± 0.6 81.1±1.181.1± 1.1 71.5±2.071.5± 2.0 69.1±2.769.1± 2.7 66.1±0.866.1± 0.8 67.2±0.667.2± 0.6 73.2±1.073.2± 1.0 71.7±0.771.7± 0.7 Verbalized confidence 52.5±2.652.5± 2.6 49.7±2.649.7± 2.6 68.5±1.568.5± 1.5 66.1±1.466.1± 1.4 54.7±4.354.7± 4.3 46.4±2.846.4± 2.8 51.3±2.851.3± 2.8 51.5±3.551.5± 3.5 56.8±1.556.8± 1.5 53.5±0.753.5± 0.7 Table 7: Macro-F1 for alternative confidence elicitation strategies on the non-validated (NV) and human-validated (HV) subsets of Logical-CommonsenseQA. N denotes Neither/Nor. Method AND OR N MIX Overall Paired MC 73.6±0.073.6± 0.0 82.2±0.082.2± 0.0 71.9±0.071.9± 0.0 60.9±0.060.9± 0.0 72.2±0.072.2± 0.0 Independent T/F 70.3±0.070.3± 0.0 78.6±0.078.6± 0.0 68.6±0.068.6± 0.0 49.7±0.049.7± 0.0 66.8±0.066.8± 0.0 Generation sampling 63.1±1.063.1± 1.0 77.7±1.977.7± 1.9 65.5±0.565.5± 0.5 61.5±0.561.5± 0.5 66.9±0.666.9± 0.6 Verbalized confidence 41.6±1.241.6± 1.2 55.7±1.055.7± 1.0 41.6±2.541.6± 2.5 53.1±0.953.1± 0.9 48.0±0.848.0± 0.8 Table 8: Macro-F1 for alternative confidence elicitation strategies on the Logical-SATA test set. As shown in Tables 7 and 8, paired multiple-choice confidence achieves the strongest overall performance on both benchmarks. It improves overall Macro-F1 over independent true–false confidence from 74.6 to 76.2 on Logical-CommonsenseQA-NV, from 73.9 to 75.8 on Logical-CommonsenseQA-HV, and from 66.8 to 72.2 on Logical-SATA. This pattern suggests that directly contrasting the positive and negative hypotheses within the same prompt provides more useful local evidence than evaluating them independently. Generation sampling performs below paired multiple-choice confidence overall and exhibits greater variability because its evidence scores are estimated from stochastic generations. It nevertheless slightly outperforms paired multiple choice in the Mixed setting on Logical-SATA. Verbalized confidence performs substantially worse across both benchmarks, indicating that self-reported numerical confidence provides less reliable evidence for globally constrained inference than answer-token probabilities or repeated model selections. C.5 Detailed Error Analysis We analyze errors at two levels. First, we examine the semantic causes of incorrect atomic judgments, asking why the model assigns high or low confidence to the constructed hypotheses. Second, we examine how the resulting atomic judgments propagate through the logical operators and global inference constraints to produce the final prediction. This separation helps distinguish errors arising from semantic understanding from those arising through logical composition. C.5.1 Semantic Causes of Atomic Errors Among the inspected Logical-CommonsenseQA errors, we first observe cases in which the model assigns high confidence to incorrect atomics. For example, for the question What could have a hot handle?, the model assigns high confidence to plastic container and glass jar, despite the metal cookware alternatives being more strongly supported by ordinary commonsense. These cases reflect direct errors in atomic scoring. We also find several cases in which a question admits multiple plausible commonsense interpretations. Open-ended terms such as "might", "could", "where", and "may" encourage the model to consider a broad range of possibilities. In these cases, the model may assign high confidence to atomics outside the gold set, indicating a difference between its interpretation of the question and the interpretation represented by the annotations. Since commonsense judgments can be context-dependent, ambiguity in question interpretation may contribute to some of these errors (Palta et al. 2024). Among the inspected Logical-SATA errors, we find several cases in which the model assigns low confidence to labels that are supported by the document. Unlike Logical-CommonsenseQA, where multiple commonsense interpretations may be plausible, these errors often involve difficulty identifying all labels that apply to the given paragraph. In particular, the model may recognize the document’s general subject while failing to recover a broader or secondary label. This indicates that some atomic errors in Logical-SATA result from the model not identifying the full set of applicable labels. Across both benchmarks, we also observe cases where the model judges whether an atomic is generally plausible while giving insufficient weight to modifiers or relations expressed in the question. This suggests that some atomic errors arise when contextual conditions are not fully preserved during hypothesis scoring. Table 9 presents representative examples of these semantic error patterns. Benchmark Input context Observed atomic behavior Interpretation LCQA What could have a hot handle? The model assigns high confidence to plastic container and glass jar. The model incorrectly accepts atomics that are less strongly supported than the metal cookware alternatives. LCQA Where do you see tiny bottles of shampoo when away from home? The model assigns high confidence to hotels, vacation rentals, cruise shops, and gym showers. The open-ended wording may support several possible locations. LCQA What is an uncommon side effect of drinking alcohol? The model assigns high confidence to frequent restroom visits, although the question asks specifically for an uncommon effect. The model appears to judge the general plausibility of the effect while giving insufficient weight to the modifier "uncommon". LSATA A biomedical article on mitochondrial protein import that mentions yeast and mouse. The model assigns low confidence to the label Organisms. The model fails to connect explicit textual evidence to a broader applicable label. LSATA An article describing a product launch and providing information about the company. The model recognizes the product launch but assigns low confidence to company description. The model identifies the main event while missing a secondary applicable label. Table 9: Representative semantic atomic errors in Logical-CommonsenseQA (LCQA) and Logical-SATA (LSATA). The examples include direct atomic scoring errors, broad interpretations of open-ended questions, insufficient attention to modifiers, and failures to identify applicable document labels. C.5.2 Logical Propagation of Atomic Errors We next examine how incorrect atomic judgments cascade through the logical structure of the options to affect the final prediction. To illustrate this interaction, Table 10 compares instances based on the same question but constructed using different operators, showing how logical composition can tolerate or amplify the underlying scoring errors. The comparison shows that the same type of atomic error can have different consequences depending on the operator. In the or construction, the false-positive atomics are tolerated because the gold option remains supported by its disjuncts. In the and construction, the additional false-positive atomics support a competing conjunction and lead to an incorrect prediction. In the Mixed construction, the same false positives both invalidate the gold neither option and support a competing and option. This behavior is also reflected in the broader error sample. and errors arise when one or both required conjuncts are rejected, whereas or errors occur when the model rejects all atomics that could support the gold option. In contrast, neither/nor errors arise when at least one member of the gold pair is incorrectly accepted. Mixed instances are more complex because one atomic judgment can influence options with different operators. Because each benchmark instance contains exactly one correct option, global inference enforces the exact-one constraint. However, the model’s local evidence scores may make either no option or multiple options logically valid. In these cases, the solver must then adjust some atomic assignments so that exactly one option remains valid. Which assignments change depends on both their confidence scores and their roles across the options. As a result, the final inferred assignments may differ from the model’s local preferences even when the logical constraints are correctly enforced. Overall, compound-level errors usually originate in the model’s atomic judgments, while the logical operators and global constraints determine how those errors affect the final prediction. Or can tolerate some incorrect atomic assignments, whereas And and Neither/Nor may be invalidated by a single error. In Mixed instances, one atomic assignment can influence options governed by different operators, making error propagation more complex. Question: What could have a hot handle? Framing Gold option Prediction Relevant p+p^+ scores Logical consequence Or metal saucepan OR baking tray Gold option metal saucepan: 0.981; baking tray: 0.644; plastic container: 0.917; glass jar: 0.877 The gold option remains valid because at least one of its disjuncts is retained. The false-positive atomics do not prevent the correct prediction. And cast iron skillet AND baking tray metal saucepan AND plastic container cast iron skillet: 0.977; baking tray: 0.644; metal saucepan: 0.981; plastic container: 0.917 The extra positive atomics support a competing conjunction, causing the gold conjunction to lose under the uniqueness constraint. Mixed NEITHER glass jar NOR plastic container baking tray AND plastic container glass jar: 0.877; plastic container: 0.917; baking tray: 0.644 The false positives invalidate the gold Neither/Nor option, while plastic container also supports the competing And option. Table 10: Different effects of atomic scoring errors across logical constructions based on the same Logical-CommonsenseQA question. Appendix D Benchmark Structure and Examples Figures 3 and 4 present the structure and construction of the benchmarks. Figure 3: Structure of Logical-CommonsenseQA instances. Operator-specific instances use the same operator across all four options, whereas Mixed instances may contain different operators. Figure 4: Construction of Logical-SATA from SATA-Bench. Each source instance provides a paragraph, a reading-comprehension question, and independently annotated correct and incorrect atomic answers. Pairs of atomic answers are combined according to the semantics of And, Or, and Neither/Nor. Each source question produces one item for each operator-specific setting and one Mixed item. Figures 5 and 6 present representative instances from the four logical settings. The values in parentheses are ground-truth binary labels for the atomic answers. QID 133: And Question. Helium, magnesium, and sulfur are likely to be found where in a school? A. art studio (0) And physical education (0) B. science lab (1) And materials workshop (1) Gold C. materials workshop (1) And art studio (0) D. science lab (1) And physical education (0) Gold composition: 1∧1=11 1=1 QID 604: Or Question. What do horses do to get energy? A. graze on pasture (1) Or rest in stables (0) Gold B. play in fields (0) Or nibble on flowers (0) C. play in fields (0) Or rest in stables (0) D. nibble on flowers (0) Or rest in stables (0) Gold composition: 1∨0=11 0=1 QID 1223: Neither/Nor Question. Where is confetti thrown from the rooftops? A. Neither festive parades (1) Nor isolated parks (0) B. Neither public events (1) Nor business functions (0) C. Neither public events (1) Nor isolated parks (0) D. Neither indoor gatherings (0) Nor isolated parks (0) Gold Gold composition: ¬0∧¬0=1 0 0=1 QID 1666: Mixed Question. Sarah took poison by accident. She found it in the cabinet and thought that it was what? A. vitamin energy tonic (0) And sports hydration drink (0) B. herbal cold remedy (1) And sports hydration drink (0) C. herbal cold remedy (1) Or sports hydration drink (0) Gold D. child’s flavored syrup (1) And sports hydration drink (0) Gold composition: 1∨0=11 0=1 The same (1/0)(1/0) pattern is invalid under And. Figure 5: Representative Logical-CommonsenseQA instances covering the And, Or, Neither/Nor, and Mixed settings. Values in parentheses denote ground-truth binary labels for the atomic answers. The highlighted option is the unique gold option. QID 718: And Source domain. Biomedimal MeSH classification Article. Role of LRRK2 in the regulation of dopamine receptor trafficking. Question. Given the above article, which MeSH root categories can be assigned to it? A. Geographicals (0) And Disciplines and Occupations (0) B. Anthropology, Education, Sociology, and Social Phenomena (0) And Technology, Industry, and Agriculture (0) C. Chemicals and Drugs (1) And Diseases (1) Gold D. Geographicals (0) And Named Groups (0) Gold composition: 1∧1=11 1=1 QID 158: Or Source domain. Science reading comprehension Passage excerpt. Magnets exert forces on certain materials. Unlike ordinary contact forces, magnetic forces can act without the magnet touching the object. Question. How are magnetic forces different from other forces? A. Other forces cause a change called motion (0) Or Magnets produce force by touching (0) B. All forces are essentially the same because they use touching (0) Or Other forces cause a change called motion (0) C. All forces are essentially the same because they use touching (0) Or Magnets produce force by touching (0) D. Magnetic force can produce motion without actual contact with certain materials (1) Or Other forces cause a change called motion (0) Gold Gold composition: 1∨0=11 0=1 QID 96: Neither/Nor Source domain. Literary reading comprehension Passage excerpt. “Femme” describes a dangerously attractive woman who uses her sexual attractiveness to lure men to their downfall. Madame White Snake is depicted as bewitchingly beautiful. Question. What does “femme” refer to? A. Neither a bewitchingly beautiful woman (1) Nor all women (0) B. Neither all women (0) Nor a true lover (0) Gold C. Neither a sexually seductive woman (1) Nor a true lover (0) D. Neither a sexually seductive woman (1) Nor a bewitchingly beautiful woman (1) Gold composition: ¬0∧¬0=1 0 0=1 QID 1095: Mixed Source domain. EU legal-document classification Document. Council Decision of 7 March 2011 appointing members and alternates of the Advisory Committee for the Coordination of Social Security Systems for Ireland, France, and the Netherlands. Question. What concepts does the above document include? A. Neither advisory committee (EU) (1) Nor Maghreb (0) B. advisory committee (EU) (1) Or social security (1) Gold C. Neither advisory committee (EU) (1) Nor social security (1) D. special polymer (0) Or uncultivated land (0) Gold composition: 1∨1=11 1=1 Option C contains the same atomics but is invalid under Neither/Nor: ¬1∧¬1=0. 1 1=0. Figure 6: Representative Logical-SATA instances from four source domains, covering the And, Or, Neither/Nor, and Mixed settings. Values in parentheses denote ground-truth binary labels for the atomic answers. The highlighted option is the unique gold option. Passage and document excerpts are shortened for readability. Appendix E Prompt Templates We present prompt templates used in our experiments. Each template retains the task description, instance-specific inputs, principal instructions, and required output format. E.1 Hypothesis Construction For each atomic answer, the model constructs a positive hypothesis hC+(a)h_C^+(a), stating that the atomic answer satisfies the constraint, and a negative hypothesis hC−(a)h_C^-(a), stating that it does not. Logical-CommonsenseQA You create two hypothesis statements for an atomic answer against the commonsense question. Input Question: question Atomic statement: atomic Instructions • Use the complete atomic statement exactly as written. • Do not simplify or replace the atomic statement. • Construct two logically opposing hypotheses. • H+ must state that the atomic satisfies the question constraints. • H- must state that the atomic does not satisfy the question constraints. • Do not determine which hypothesis is correct. Output Return valid JSON containing the fields ’H+’ and ’H-’. Logical-SATA You create two hypothesis statements for an atomic answer against the question, grounded in a reading-comprehension passage. Input Passage: paragraph Question: question Atomic statement: atomic Instructions • Use the complete atomic statement exactly as written. • Do not simplify or replace the atomic statement. • Construct two logically opposing hypotheses. • H+ must state that the atomic satisfies the question constraints • H- must state that the atomic does not satisfy the question constraints. • Do not determine which hypothesis is correct. Output Return valid JSON containing the fields ’H+’ and ’H-’. E.2 Paired Multiple-Choice Confidence The paired multiple-choice prompt jointly presents the positive and negative interpretations of an atomic answer. The model selects the interpretation that is correct with respect to the context and constraint. Logical-CommonsenseQA You evaluate which of two competing hypotheses about an atomic answer is correct with respect to a required commonsense constraint. Input Question: question Atomic statement: atomic_statement Option A: H_plus Option B: H_minus Instructions • Judge whether the atomic answer fulfills that requirement. • Judge the atomic answer independently of the other answer options. • Select exactly one option. Output Return only the single letter A or B. Answer: Logical-SATA You evaluate which of two competing hypotheses about an atomic answer is correct with respect to a required reading-comprehension constraint. Input Passage: paragraph Question: question Atomic statement: atomic_statement Option A: H_plus Option B: H_minus Instructions • Determine whether the atomic answer fulfills that requirement as described in the passage. • Judge the atomic answer independently of the other answer options. • Select exactly one option. Output Return only the single letter A or B. Answer: E.3 Independent True–False Confidence Independent true–false confidence evaluates the positive and negative hypotheses in separate model calls. Logical-CommonsenseQA You determine whether a hypothesis about an atomic answer is true with respect to a required commonsense constraint. Input Question: question Atomic statement: atomic_statement Hypothesis: H_plus or H_minus Instructions • Determine whether the supplied hypothesis is true. • Judge the atomic answer against the constraint on its own terms. Output Return only True or False. Answer: Logical-SATA You determine whether a hypothesis about an atomic answer is true with respect to a reading-comprehension constraint. Input Passage: paragraph Question: question Atomic statement: atomic_statement Hypothesis: H_plus or H_minus Instructions • Determine whether the supplied hypothesis is true according to the passage. • Judge the atomic answer against the passage and constraint on its own terms. Output Return only True or False. Answer: Separate calls are made for hC+(a)h_C^+(a) and hC−(a)h_C^-(a). E.4 Generation Sampling Generation sampling uses the same benchmark-specific templates as paired multiple-choice confidence. You evaluate which of two competing hypotheses about an atomic answer is correct with respect to the required question constraints. Input [Passage: paragraph] Question: question Atomic statement: atomic_statement Option A: H_plus Option B: H_minus Instructions • Judge whether the atomic answer fulfills the specific requirement expressed by the constraint. • Judge the atomic answer independently of the other answer options. • Select exactly one option. Output Return only the single letter A or B. Answer: The passage field is included for Logical-SATA and omitted for Logical-CommonsenseQA. Rather than extracting answer-token probabilities, we sample five responses and compute the positive and negative evidence scores from the empirical frequencies of A and B. E.5 Verbalized Confidence Verbalized confidence evaluates the positive and negative hypotheses in separate calls and asks the model to report a numerical confidence value. Logical-CommonsenseQA You report your confidence that a hypothesis about an atomic answer is true with respect to the required commonsense question constraints. Input Question: question Atomic statement: atomic_statement Hypothesis: H_plus or H_minus Instructions • Evaluate whether the supplied hypothesis is true. • Judge the atomic answer against the constraint on its own terms. Output Return one integer from 00 to 1010, where 00 indicates complete confidence that the hypothesis is false and 1010 indicates complete confidence that it is true. Confidence: Logical-SATA You report your confidence that a hypothesis about an atomic answer is true with respect to a reading-comprehension question constraint. Input Passage: paragraph Question: question Atomic statement: atomic_statement Hypothesis: H_plus or H_minus Instructions • Evaluate whether the supplied hypothesis is true according to the passage. • Judge the atomic answer against the passage and constraint on its own terms. Output Return one integer from 00 to 1010, where 00 indicates complete confidence that the hypothesis is false and 1010 indicates complete confidence that it is true. Confidence: Separate calls are made for hC+(a)h_C^+(a) and hC−(a)h_C^-(a). E.6 Representative Demonstrations The full prompts contain fixed demonstrations. We show one representative example from each benchmark. Logical-CommonsenseQA Question: Where can you see a mountain in your own home? Atomic statement: a window facing the mountains Option A: A window facing the mountains is a place from inside a home where someone can see a mountain. Option B: A window facing the mountains is not a place from inside a home where someone can see a mountain. Answer: A Logical-SATA Passage: The Legal Aid Society was barred from returning to its headquarters near the World Trade Center site because of environmental concerns. Question: Why could the Legal Aid Society not return to its original headquarters? Atomic statement: contamination risk from the nearby World Trade Center site Option A: Contamination risk from the nearby World Trade Center site was the reason the Legal Aid Society could not return to its original headquarters. Option B: Contamination risk from the nearby World Trade Center site was not the reason the Legal Aid Society could not return to its original headquarters. Answer: A

Instant research discovery