Paper deep dive

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

Kawai Chung, Chunkit Chan, Yuxuan Liu, Yauwai Yim, Xi Yang, Wenbin Hu, Huihao Jing, Chiu Yeung Tang, Changxuan Fan, Ki Sen Hung, Tsun On Kwok, Haoran Li

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 97

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven generative baselines on 11,117 English text questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second across all six capabilities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show distinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM selection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate expansion reveals distinct selection and output-validity problems. Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

93,361 characters extracted from source content.

Expand or collapse full text

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models Kawai ChungChunkit ChanYuxuan LiuYauwai Yim Xi YangWenbin HuHuihao JingChiu Yeung Tang Changxuan FanKi Sen HungTsun On KwokHaoran Li Department of Computer Science and Engineering The Hong Kong University of Science and Technology Code kwchungac,ckchancc,hlibt@cse.ust.hk Abstract Dialogue and collaborative assistants repeat- edly judge what a partner knows, believes, or intends. Typed decision models offer a po- tential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven genera- tive baselines on 11,117 English text questions from nine ToM benchmarks across six capabil- ity groups. TypeSafe Jev reaches 73.2% source- balanced accuracy including native controls and ranks first or second across all six capabil- ities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show dis- tinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM se- lection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate ex- pansion reveals distinct selection and output- validity problems. Related-question cues ac- count for most of Jev’s downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task- specific confidence calibration, candidate de- sign, and output validation as integration prior- ities. 1 Introduction Theory of Mind (ToM) is the ability to attribute mental states, such as beliefs, knowledge, and in- tentions, to others and use them to interpret or pre- dict behavior (Premack and Woodruff, 1978). Ad- vances in the language and reasoning capabilities of large language models (LLMs) have brought in- creasing attention to their ability to reason about others’ perspectives (Strachan et al., 2024, Chan et al., 2026). Useful interactive responses depend on the recipient’sperspective, including missed in- formation and outdated beliefs. Knowledge access Beliefs Goals, desires & intentions Emotions Social pragmatics Mental-state application Benchmark BigToM OpenToM ToMBench SimpleToM FANToM ToMATO CogToM Hi-ToM ToMi Jev-ToM Figure 1: Coverage of evaluated subsets and their union (✓: present,×: absent). Agents use mental-state inferences for action selection and response generation. Collaborative agents update explicit belief states from observa- tions for planning (Li et al., 2023). Dialogue agents use generative LLMs, including compact models, to infer participants’ mental states from conversa- tion and condition responses on those inferences (Hwang et al., 2026). For example, ToM-agent tracks a partner’s beliefs, desires, and intentions in simulated persuasion dialogues, such as attempts to encourage a donation (Yang et al., 2025). It main- tains candidate interpretations, revises them using observed and predicted replies, and conditions its next response on the candidate with the highest confidence. The authors identify repeated LLM calls and long prompts as sources of waiting time and cost (Yang et al., 2025). Repeated mental-state selection motivates testing Jev for candidate-based judgments in dialogue and collaboration, with can- didate generation and response composition left to the surrounding system. Such intermediate judg- ments can have finite alternatives, even when final responses require open-ended language. Typed decision models offer a direct interface for bounded judgments (Wang, 2026, Deußer et al., 1 arXiv:2610.18610v1 [cs.CL] 13 Oct 2026 2026). Given context, a question, and candidates, TypeSafe Jev (hereafter Jev) returns a selection and candidate probabilities as structured outputs (Type- Safe AI, 2026). Its design targets fast, inexpen- sive decisions (Almeida, 2026). Structured selec- tion makes Jev a promising candidate for bounded ToM judgments currently performed by generative LLMs. Fulfilling this role requires accurate mental- state judgments at low time and cost. Fast, structured output alone does not establish suitability for this role. A high overall score can conceal differences between tracking knowledge, attributing beliefs or emotions, and applying men- tal states to behavior. Moreover, an agent needs to know when a selection is reliable and whether supplied reasons alter the choice or break the re- quired output format. This raises a research ques- tion: How competitive and reliable is Jev as a bounded ToM decision component? We present Jev-ToM, a systematic framework for evaluating Jev’s ToM capabilities. To the best of our knowledge, this is the first dedicated eval- uation of Jev’s ToM capabilities across multiple benchmarks organized into explicitly defined capa- bility groups. We compare five typed decision mod- els and seven generative baselines on 11,117 ques- tions from nine existing ToM benchmarks across six operational capability groups, as shown in Fig- ure 1. We examine confidence-based answer se- lection, interference from supplied reasons, and output validity, alongside observed response time and reference-price cost. Our results show that competitive accuracy can coexist with benchmark-dependent confidence er- rors, while supplied reasons can improve or disrupt selections depending on their attachment and the evaluated materials. The evaluation identifies confi- dence calibration, candidate sensitivity, and output validity as priorities for integration testing. Our contributions are threefold: • We compare Jev and generative LLMs across ToM tasks, measuring accuracy, reference cost, and observed response time. • We diagnose Jev’s reliability through confidence- based selection and candidate interventions, sep- arating selection errors from output-validity fail- ures obscured by aggregate accuracy. •We investigate supplied reasons and mental-state feedback, testing reason attachment and sepa- rating the contributions of related-question cues and predicted states. BenchmarkQuestionsFamiliesOptions BigToM1,200502 OpenToM2,5331492/3 ToMBench7303232/4 SimpleToM8911772 FANToM1,805642 ToMATO1,396404 CogToM2,1541,2154 Hi-ToM1543015 ToMi2542202 Total11,1172,2682/3/4/15 Table 1: Evaluated benchmark subsets. 2 Related Work ToM agents and benchmarks. Mental-state judgments support cooperative planning (Li et al., 2023), dialogue perspective tracking (Qiu et al., 2024, Yang et al., 2025), response selection (Hwang et al., 2026), and action prediction (Mu et al., 2026). These uses motivate evaluating inter- mediate mental-state judgments in agents. ToM benchmarks differ in what successful judgment requires: controlled belief attribution and higher- order perspectives (beliefs about another person’s beliefs) (Le et al., 2019, Gandhi et al., 2023, Wu et al., 2023), conversational information asymme- try (Kim et al., 2023), or diverse psychological states and social situations (Xu et al., 2024, Chen et al., 2024, Tong et al., 2026, Shinoda et al., 2025). SimpleToM separates mental-state inference from behavior prediction and judgment (Gu et al., 2026), while NegotiationToM studies inference in human negotiations (Chan et al., 2024). This variety makes scope and aggregation choices central to interpret- ing a ToM score. Jev interfaces and evaluation. Typed decision interfaces expose choices, scores, and probabilities for software to consume (Almeida, 2026, TypeSafe AI, 2026). Mnemon uses Jev to evaluate retrieved records while LLM plans searches and composes responses (Wang, 2026). This division motivates a similar component role for mental-state selection. Broad Jev evaluations already examine task perfor- mance, calibration, and resource use (Deußer et al., 2026), alongside annotation and rubric assessment (Ibrahim and Zaki, 2026, Rao and Callison-Burch, 2026). Adversarial and option-definition studies establish that structured outputs can remain sensi- tive to request content (Hu et al., 2026, Sun et al., 2026). JevOut also includes ToMBench when test- ing contextual additions that redirect decisions (Xu 2 et al., 2026). We extend these studies by connect- ing cross-benchmark ToM profiles with confidence- based selection, bidirectional changes under sup- plied reasons, and output validity. 3 Experiments To assess breadth and reliability in bounded ToM selection, we compare performance on fixed an- swer choices and examine confidence, supplied reasons, and resource use in separate subsets of models and questions. 3.1 Theory of Mind Ability Categories We adapt Abilities in Theory of Mind Space (ATOMS) to group questions by the mental states they concern (Beaudoin et al., 2020). Knowledge access covers information available through ob- servation and communication (Kim et al., 2023). Beliefs concern representations of the world and others’ perspectives (Gandhi et al., 2023, Wu et al., 2023). Goals, desires, and intentions group re- lated preferences and planned actions for cross- source comparison (Xu et al., 2024). Emotions covers feelings (Shinoda et al., 2025), while social pragmatics groups socially situated and non-literal meaning (Chen et al., 2024). CogToM’s native Per- cept items remain in benchmark totals but outside capability groups pending finer mapping. Mental-state application adds a different di- mension: using a perspective to predict, select, or judge behavior (Gu et al., 2026). A question ask- ing what someone believes concerns attribution, whereas one asking how that belief affects their action concerns application. Application can over- lap the state categories because it describes a task demand. Figure 1 also shows unequal source cover- age: beliefs draw on eight benchmarks, while social pragmatics draws on two. These profiles compare models over the specified question groups. 3.2 Dataset Selection and Harmonization We assemble a common evaluation from nine ex- isting ToM benchmarks with English contexts, ref- erence answers, and discrete candidate choices (Gandhi et al., 2023, Le et al., 2019, Wu et al., 2023, Kim et al., 2023, Xu et al., 2024, Chen et al., 2024, Tong et al., 2026, Shinoda et al., 2025, Gu et al., 2026). Table 1 summarizes the evaluated subsets. We use normal-length discrete OpenToM, short- context discrete FANToM, English ToMBench, and the ToMi subset accompanying SimToM (Wilf et al., 2024). The collection consists of fixed devel- opment or previously used subsets, not newly col- lected questions. Appendix B.2 gives their scope. Source-specific adapters convert each question into a shared record containing source and ques- tion identifiers, a question family, context, question, candidate texts, and a reference answer. Families group related questions or scenario variants using source identifiers or shared contexts, so a BigToM family can span multiple story variants. We re- tain the source contexts without truncation and pre- serve task-specific candidate meanings. Candidate counts remain unequal, ranging from two to fifteen. Harmonization therefore standardizes the evalua- tion record while retaining the original decision task. We check that fields are nonempty, candidates are distinct, and references identify an offered an- swer. Of the initial 11,167 questions, we exclude 50 BigToM items for every model because their stories omit the subsequent event required by the reference. The resulting 11,117 questions and each question’s candidate order are frozen across pri- mary models. References are retained for scoring and withheld from model inputs. Source and family identifiers preserve the connection between stan- dardized records and related judgments. We then map native task, ability, or mental-state labels to the six capability groups. ToMBench ap- plication additionally follows the requested behav- ior prediction, selection, or judgment. All 730 ToMBench question targets underwent an agent- assisted post-evaluation review without item-level model predictions. This was not independent hu- man validation. A question may enter multiple capability groups but counts once in its benchmark total. Appendix B.1 provides the mapping rules. 3.3 Evaluation Metrics Strict scoring requires a valid output and correct se- lection. Typed probabilities must cover exactly the offered choices, lie in[0, 1], sum to one, and give maximal probability to the selected choice, within numerical tolerances. Generated answers must re- solve unambiguously to a candidate. Parsing and validation rules are detailed in Appendix B.3. Let z mi = 1when modelmreturns a valid answer matching the reference for itemi, and0otherwise, including invalid, unsupported, or failed responses. For benchmarks, letD s containN s questions. Ex- act match (EM, %) and source-balanced overall 3 GPT-3.5-1106 Jev-1.13 GPT-5.6-Luna GPT-4o-mini DeepSeek-V4-Flash GPT-4.1-nano Llama-3.1-8B Open-Jev-2B Valen SmolLM3-3B Tiny-Jev OpenJev-0.6B 0 50 100 Answer accuracy (%) 76.1 73.6 72.6 71.6 63.3 55.3 54.9 43.9 39.1 36.5 35.7 23.4 Jev-1.13 GPT-3.5-1106 GPT-5.6-Luna DeepSeek-V4-Flash GPT-4o-mini GPT-4.1-nano Llama-3.1-8B Valen Open-Jev-2B Tiny-Jev SmolLM3-3B OpenJev-0.6B 0 50 100 68.4 66.9 63.5 55.3 54.3 51.8 48.2 44.7 43.6 40.2 34.8 31.8 Beliefs Jev-1.13 GPT-5.6-Luna GPT-3.5-1106 DeepSeek-V4-Flash GPT-4o-mini Llama-3.1-8B GPT-4.1-nano Open-Jev-2B Tiny-Jev SmolLM3-3B Valen OpenJev-0.6B 0 50 100 78.5 76.2 74.4 69.0 68.8 65.3 62.7 52.0 48.4 45.9 40.0 36.6 Jev-1.13 GPT-3.5-1106 GPT-5.6-Luna DeepSeek-V4-Flash GPT-4o-mini GPT-4.1-nano Llama-3.1-8B Open-Jev-2B SmolLM3-3B Tiny-Jev OpenJev-0.6B Valen 0 50 100 Answer accuracy (%) 84.8 79.7 79.5 77.5 74.8 73.0 65.7 59.1 56.2 55.2 43.5 41.0 Emotions DeepSeek-V4-Flash Jev-1.13 GPT-5.6-Luna GPT-3.5-1106 GPT-4o-mini Llama-3.1-8B GPT-4.1-nano OpenJev-0.6B Tiny-Jev Open-Jev-2B SmolLM3-3B Valen 0 50 100 89.9 89.1 84.9 82.5 80.1 74.8 74.4 56.4 53.8 52.5 51.3 50.1 DeepSeek-V4-Flash Jev-1.13 GPT-3.5-1106 Open-Jev-2B GPT-5.6-Luna Valen GPT-4.1-nano GPT-4o-mini Llama-3.1-8B OpenJev-0.6B SmolLM3-3B Tiny-Jev 0 50 100 70.7 70.4 65.6 65.0 62.7 62.3 60.3 60.2 59.0 58.8 53.6 43.1 Knowledge AccessGoals, Desires and Intentions Social PragmaticsMental-State Application Figure 2: Accuracy (%) for knowledge access, beliefs, goals/desires/intentions, emotions, social pragmatics, and mental-state application, averaged equally across contributing benchmarks. Panels sort models by accuracy, with typed models colored and generative baselines gray. Benchmark control questions and unmapped questions are excluded. EM are EM m,s = 100 N s X i∈D s z mi ,(1) EM m,overall = 1 9 9 X s=1 EM m,s .(2) For capabilityc, letD s,c contain its assigned questions from sourcesandS c the sources with nonempty D s,c . We compute EM m,c = 1 |S c | X s∈S c 100 |D s,c | X i∈D s,c z mi .(3) Overall EM includes the benchmarks’ original con- trol questions and unmapped questions, while ca- pability EM includes only assigned questions. An item may enter multiple capability groups but ap- pears once per benchmark, so overall EM is not the mean of the six capability scores. Equal benchmark weighting prevents larger datasets from dominating. Pooled accuracy instead weights questions equally where explicitly stated. Further aggregation details are provided in Appendix B.3. 3.4 Evaluated Models We evaluated 12 models: five typed decision mod- els and seven generative baselines. The typed models are TypeSafe Jev-1.13 (Almeida, 2026), Open-Jev-2B (ZefanCai, 2026), Valen-Preview- 0923 (Valen-Team, 2026), Tiny-Jev (lostargon, 2026), and OpenJev-0.6B (IamBusy, 2026). The generative baselines are SmolLM3-3B (Bakouch et al., 2025), Llama-3.1-8B-Instruct (Meta, 2024), GPT-4.1-nano (OpenAI, 2025), GPT-4o-mini (Ope- nAI, 2024), GPT-3.5-Turbo-1106 (OpenAI, 2023), GPT-5.6-Luna (OpenAI, 2026), and DeepSeek-V4- Flash-0731 (DeepSeek, 2026). Primary models receive the same contexts, ques- tions, and ordered candidates without reference labels. Typed interfaces encode the context as state and the question as instructions, returning a choice and candidate probabilities. Generative models re- ceive the same content and return an option label. Both outputs are mapped back to the original can- didates for strict scoring. Model-specific templates, 4 ModelBigToMOpenToMToMBenchSimpleToMFANToMToMATOCogToMHi-ToMToMiOverall Jev-1.1389.260.076.759.179.683.681.359.769.773.2 Open-Jev-2B 80.445.251.449.771.550.149.439.672.056.6 Valen74.938.146.075.148.632.739.433.157.949.5 Tiny-Jev73.934.041.520.971.254.239.930.547.245.9 OpenJev-0.6B 74.639.140.756.929.120.042.531.853.943.2 GPT-3.5-110686.961.875.155.277.681.678.155.872.471.6 GPT-5.6-Luna85.157.673.849.772.281.981.564.970.970.9 GPT-4o-mini85.253.867.057.775.375.172.242.967.366.3 DeepSeek-V4-Flash84.560.270.067.061.866.877.741.659.165.4 GPT-4.1-nano 78.147.857.564.167.571.866.535.163.461.3 Llama-3.1-8B76.748.459.965.358.862.665.118.263.857.6 SmolLM3-3B78.345.048.147.742.135.249.735.157.148.7 Table 2: Exact-match accuracy (%) on 11,117 questions, including native controls. Overall weights benchmarks equally. Invalid outputs count as errors. Bold/underlined values mark group best/second-best, with blue/green for typed/generative models. Excluding controls, Jev scores 69.1% and GPT-3.5 70.9%. decoding settings, and deployments are recorded in Appendix B. 4 Results and Analysis 4.1 Where Are Typed Models Competitive? Jev is competitive across all six capabilities. Figure 2 ranks Jev first or second in every capa- bility, with its largest lead over the strongest gen- erative baseline in emotions: 84.8% against GPT- 3.5’s 79.7%. Strong relative rankings still leave substantial errors in beliefs and mental-state appli- cation, Jev’s lowest-scoring capabilities. The other typed models remain below Jev but differ in their strengths. Open-Jev-2B leads this group in four capabilities, while Valen leads beliefs and OpenJev- 0.6B social pragmatics. Tiny-Jev exceeds OpenJev- 0.6B on emotions, reaching 55.2% against 43.5%, yet trails it on application. Application is the strongest capability for Open-Jev-2B, Valen, and OpenJev-0.6B, whereas Tiny-Jev favors emotions. Broad competitiveness is therefore a result for Jev, and the shared typed interface does not make the models interchangeable. Benchmark rankings can reverse. Table 2 shows why overall rank is insufficient for model selection. Removing native controls reverses Jev’s lead over GPT-3.5, leaving 69.1% against 70.9%. The comparison thus depends partly on which task demands enter the aggregate. Rankings also re- verse within a specific benchmark: Open-Jev-2B reaches 72.0% on ToMi against Jev’s 69.7%, de- spite trailing substantially overall. Appendices C.1 and C.2 provide capability breakdowns and control- free comparisons. Choosing a ToM component consequently requires matching its strengths to the intended tasks, rather than relying on a single ag- gregate score. 050100 SimpleToM OpenToM ToMBench FANToM BigToM CogToM ToMi ToMATO Hi-ToM 46.9 62.5 54.4 51.1 81.1 60.9 34.3 46.6 21.4 02040 34.9 29.0 9.8 7.3 4.5 4.3 2.3 1.8 0.0 Figure 3: Jev answer selection at confidence≥ 0.90: coverage (left, % of all questions) and retained error (right, % of retained answers). Both panels follow de- scending error rate. Hi-ToM retains 33 answers. 4.2 Can Confidence Identify Reliable Answers? Confidence supports selection unevenly across tasks. Figure 3 compares Jev’s answer selection at a descriptive selected-answer probability cutoff of 0.90 across all 11,117 primary responses. Cover- age is the fraction retained and retained error their error rate. Seven benchmarks fall below 10% re- tained error, but SimpleToM and OpenToM remain at 34.9% and 29.0%. Similar coverage can there- fore conceal very different reliability: ToMATO and SimpleToM both retain about 47% of answers, yet their retained errors are 1.8% and 34.9%. Con- versely, Hi-ToM’s zero observed errors concern only 33 retained answers. Appendices D.1 and D.2 provide counts and risk–coverage curves. The de- scriptive cutoff therefore needs task-specific vali- dation before use as an acceptance policy. 5 ConditionKeyedFoil Aligned reasons CW Shared correct reason C Neutral filler N No reasons – Shared wrong reason W Swapped reasons WC 0255075100 Answer accuracy (%) 99.0 97.8 96.8 96.4 83.7 25.3 0255075100 Answer accuracy (%) 71.0 69.4 17.4 10.0 9.3 16.1 Earlier CorrectEarlier Errors Figure 4: Fresh Jev answer accuracy on earlier correct and incorrect cases. C/W/N/–: reference/selected incor- rect/neutral/absent reasons. Dashed: fresh no-reason baseline. 4.3Intervention 1: How Do Supplied Reasons Affect Selection? Reason attachment can overturn an otherwise correct selection. We compare six conditions with questions, answer choices, and Jev settings fixed. The keyed reason supports the reference answer, and the foil reason supports a selected in- correct answer. Aligned reasons accompany their own answers, swapped reasons exchange these two attachments, and shared conditions repeat either reason across answers. No-reason and neutral-text controls complete the comparison. Jev selects only an answer on 249 historically correct and 1,554 historically incorrect questions, each in original and reversed orders. Figure 4 shows aligned rea- sons raising fresh accuracy on earlier-correct cases from 96.4% to 99.0%, while swapping lowers it to 25.3%. Neutral text preserves 96.8%, so extra text alone does not explain the loss. On earlier er- rors, aligned and shared keyed reasons reach 71.0% and 69.4% against a fresh 10.0% baseline. Ap- pendix E.2 details sampling and conditions. Re- covery can occur without option-specific support, yet misleading attachment can overturn a correct selection. Small net changes conceal opposing outcome changes. In a separate eight-model study, ma- terials pair existing questions with supplied rea- sons. Three coauthors assessed reason support and attachment on an initial 450-question sample, with methods and majority-reference results in Ap- pendix E.3. Models select bare answers or com- plete answer–reason pairs. Aligned pairs preserve each attachment, while crossing includes allK 2 combinations ofKanswers and reasons, including the keyed pair. Figure 5 compares the 1,803 preced- ing error-enriched materials, termed reused, with new reasons for 2,958 other existing questions. All gains use strict answer-component accuracy and equal benchmark weights. Aligned gains shrink 0204060 DeepSeek-V4-Flash Llama-3.1-8B GPT-4.1-nano GPT-4o-mini Open-Jev-2B GPT-5.6-Luna GPT-3.5-1106 Jev-1.13 +51.4+3.7 0204060 +42.2-0.4 AlignedCrossed 0 4 8 +6.96 -3.27 +3.69 +6.69 -7.08 -0.40 Answer accuracy change (p)Change contribution (p) Reused materialsNew materials Repair contribution Damage contribution Net Aligned ReasonsCrossed ReasonsRepair and Damage Figure 5: Strict answer-component accuracy changes for aligned and crossed reasons, ordered by aligned gains on new materials. Repair and Damage: Jev’s contributions (percentage points of all weighted outcomes) on new materials. Benchmarks receive equal weight. 6 on new materials for every model: Jev falls from 51.4 to 3.7 points, below Open-Jev-2B’s 8.3-point gain. Crossed gains are lower still for all eight models and negative for five. Jev’s Repair and Damage panel explains why the net effect is insuf- ficient. Against matched no-reason answers, re- pair changes errors to correct answers and damage loses previously correct answers. From aligned to crossed, repair stays near seven points while dam- age rises from 3.3 to 7.1. Thus, 13.8% of weighted outcomes change strict correctness despite a net −0.4points. Crossing also increases candidate count and repetition. Appendix E.4 details mate- rial selection and repair–damage scoring. Reason sensitivity must therefore be assessed against both the material selection and the underlying changes in correctness. Jev-1.13 78.21.62.210.47.5 GPT-5.6-Luna 77.61.84.416.2<0.1 GPT-3.5-1106 70.94.29.715.2<0.1 DeepSeek-V4-Flash 66.92.56.615.88.3 GPT-4o-mini 57.26.515.720.6<0.1 Open-Jev-2B 50.63.513.532.4<0.1 GPT-4.1-nano 43.88.517.529.90.2 Llama-3.1-8B 35.713.023.827.30.2 Both correct Answer only Reason only Both wrong Invalid / failed Figure 6: Crossed-pair outcomes on all 4,761 materi- als, averaging both candidate orders and benchmarks equally. “Both correct” is strict joint accuracy, and “only” identifies the correct component. Failures stay in the denominator. Rows sort by both-correct share, with tints on a 0–100% scale. Keyed reasons can accompany wrong answers. Figure 6 separates answer/reason correctness from output validity in the same crossed responses, using benchmark-equal outcome shares. Strict joint ac- curacy requires valid output with both components matching their keys. Every model selects only the keyed reason more often than only the keyed an- swer. For Llama, the shares are 23.8% and 13.0%, showing that selecting supplied support does not en- sure a correct answer. Similar joint scores also con- ceal different failure profiles: Jev and Luna achieve 78.2% and 77.6% both-correct outcomes, but their invalid or failed shares are 7.5% and less than 0.1%, respectively. Open-Jev-2B rarely fails validation, yet its keyed-reason/wrong-answer share reaches −20−10010 +7.9 +2.3 −13.9 −4.6 7075808590 81.084.3 76.482.4 Reattached − original (8) 16 candidates GPT-3.5Jev-1.13 StrictSaved choice (post hoc) Joint accuracy (%) 16 ℒ original 8 candidates Original 8 candidates Candidate Changes Strict vs Saved Figure 7: Joint answer–reason accuracy on 108 ques- tions (one per family) from CogToM, ToMATO, and ToMBench, with two orders and equal benchmark weights. Candidate Changes: strict changes from the original eight-pair set after reattachment or expansion to sixteen. Strict vs Saved: post-hoc scoring reads choices despite invalid probabilities without repairing them. 13.5%. Appendix E.6 details probabilityvalidation and output diagnostics. Reliable pair selection re- quires checking semantic choices as well as the output contract. Candidate expansion affects both selection and output validity.To separate attachment from can- didate expansion, we test 108 questions, one per family, using an eight-pair baseline that repeats four answers and four reasons twice. Reattachment changes only pairs with neither reference compo- nent. Expansion instead includes all sixteen com- binations and changes repetition. Figure 7 shows reattachment helping both models, while expansion reduces accuracy, including a 13.9-point GPT-3.5 loss despite valid outputs. Jev’s strict accuracy falls from 81.0% to 76.4%. Post-hoc saved-choice scor- ing reads selections despite invalid probabilities without repairing them. Its smaller decline widens the strict-to-saved gap from 3.2 to 6.0 points, distin- guishing selection loss from additional validation failures. Appendix E.5 gives the protocol. Even when the correct pair remains available, expanding the candidates can impair both selection and output validity. 4.4 Intervention 2: Does Mental-State Feedback Improve Action Prediction? Question cues account for most of Jev’s gain. Table 3 compares five typed models on 400 behavior-prediction questions from families out- side the primary sample. Each target shares a story 7 $0.01$0.03$0.1$0.3$1 OpenRouter reference cost (US$ / 1,000 questions) 55 60 65 70 75 80 Benchmark-averaged accuracy (%) 3001,0003,00010,000 Mean response time (ms) 55 60 65 70 75 80 Benchmark-averaged accuracy (%) TypeSafe Jev DeepSeek-V4-Flash GPT-3.5-1106 GPT-4.1 nano GPT-5.6 Luna Llama-3.1-8B GPT-4o mini Open-Jev-2B Figure 8: Accuracy versus OpenRouter reference cost (left) and observed deployment time with differing concurrency (right). Shading marks lower accuracy with higher cost (left) or longer response time (right) than Jev. Model / taskOriginal Question only Predicted state Flipped state Jev-1.1378.5089.2593.0086.25 Open-Jev-2B66.5064.0064.7561.75 Valen62.7560.5059.7560.00 Tiny-Jev48.5052.5051.5052.50 OpenJev-0.6B61.5055.0051.2551.00 Jev: BigToM92.0098.0097.5097.00 Jev: SimpleToM65.0080.5088.5075.50 Table 3: Accuracy (%) on 400 additional questions from families outside the primary sample, equally weighting 200 BigToM and 200 SimpleToM targets. The last rows show Jev by source. Its same-template fixed-No replay matches SimpleToM predicted feedback at 88.5%. with a belief/awareness question.We compare fresh original controls with question-only cues sup- plying the related question, predicted-state feed- back adding the same model’s answer, and flipped- state feedback supplying the other binary answer, which need not be wrong. Question cues account for 10.75 of Jev’s 14.5-point gain, with a further 3.75 points from adding the predicted answer. That extra gain concentrates on SimpleToM, where a post-hoc offline fixed-No replay of saved calls also reaches 88.5% and reproduces every predicted- feedback selection. BigToM changes little across the three additions. Tiny-Jev favors question-only, while predicted feedback reduces accuracy for Open-Jev-2B, Valen, and OpenJev-0.6B relative to their original controls. Appendix F.2 details the replay and fixed-versus-predicted comparisons. Related-question cues can improve action selection, but the recorded gains do not establish a benefit from instance-specific mental-state predictions. 4.5 What Are the Cost–Time Tradeoffs? Jev improves the tradeoff against nano and mini. Figure 8 compares eight deployments on 11,117 questions, with seven priced using frozen Open- Router reference tariffs. At US$0.0295 per 1,000 questions, Jev costs 32.7% less than nano and 56.3% less than mini, with 11.7- and 6.7-point accuracy advantages. Lower cost alone does not favor Jev universally: DeepSeek offers US$0.0138 at 65.4% accuracy, giving Jev 7.6 more points at 2.14 times the cost. DeepSeek’s longer observed response time also shows that lower cost need not mean less waiting. Jev’s separate timing run records 73.0% accuracy and a 282 ms mean response time, the highest accu- racy and shortest mean in this cohort. Its mean is 83.9% below nano and 82.3% below mini. Open- Jev-2B follows at 464 ms with 56.6% accuracy and appears only in the time comparison because it lacks a matching tariff. Appendix G gives settings and full values. Observed timings reflect differing hardware, concurrency, and retry settings. Shorter decision calls could reduce waiting in assistants making repeated sequential judgments, but the over- all benefit depends on the cost and time of gener- ating candidates and composing responses, which this experiment does not measure. 5 Conclusion Jev-ToM shows that Jev combines competitive bounded ToM selection with favorable reference- cost and observed-time comparisons in the eval- uated deployments. Benchmark-dependent con- fidence errors and sensitivity to supplied reasons show why accuracy alone is insufficient for adop- tion. The results prioritize task-specific confidence 8 calibration, candidate design, and output validation before testing Jev within complete assistants. Limitations Language and task coverage. Our evaluation is limited to English, text-based questions with supplied candidates. The nine benchmarks cover several forms of ToM judgment, but their results do not establish generalization to other languages, cul- tural settings, or multimodal social situations. The six capability groups summarize the included tasks and are not an exhaustive measure of ToM. Pos- sible training exposure to the public benchmarks also cannot be excluded. Component-level evaluation. We evaluate Jev as a bounded decision component, without testing its integration into a complete interactive assistant. Candidate generation, partner-state updates across turns, and interactive task success are therefore outside our empirical claims. The confidence and reason analyses are exploratory, and the descriptive 0.90 cutoff is not a validated acceptance policy. The observed accuracy and resource advantages concern the evaluated tasks and deployments, so their value in a complete system depends on the surrounding components. Ethical Considerations This study uses existing benchmark situations, model outputs, and annotations by three postgrad- uate students at English-medium universities with prior experience annotating ToM benchmarks. An- notation procedures are provided in Appendix E.3. It does not infer sensitive attributes of real users. Social-decision components can nevertheless harm users when predictions about beliefs or intentions are treated as facts. Applications should distinguish observed information from inferred states and vali- date the consequences of their errors. Our results do not justify autonomous consequential decisions about individuals. Dataset and model redistribution remains subject to source terms, with the planned release described in Appendix I. To support reproducibility, we plan to release a versioned package containing evaluation materi- als, subset and family identifiers, candidate-order mappings, per-question predictions and available probability vectors, de-identified annotation-stage records, and executable scoring and family-level resampling code. The package will cover the rea- son interventions and matched-candidate analyses, with source attribution and redistribution governed by the original licenses. Appendix I describes the planned release. We used GPT-6.1-Sol to generate and separately review the reason materials. These supplied rea- sons are experimental candidate text, not expla- nations produced by the evaluated models. Con- struction and prompts are documented in Appen- dices E.1 and J, respectively. Language-model as- sistance also supported the question-target mapping audit, analysis-code preparation, and manuscript revision. The authors remain responsible for the methods, claims, and final publication. References Diogo Almeida. 2026. Introducing System One Models & Jev. Bushra Asseri and Abdulaziz Asseri. 2026. Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV. arXiv preprint arXiv:2609.36399. Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Noua- mane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guil- herme Penedo, Hugo Larcher, Mathieu Morlon, Vaib- hav Srivastav, Joshua Lochner, and 4 others. 2025. SmolLM3: smol, multilingual, long-context rea- soner. Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H. Beauchamp. 2020. Systematic Re- view and Inventory of Theory of Mind Measures for Young Children. Frontiers in Psychology, 10:2905. Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang, Weiqi Wang, and Yangqiu Song. 2024. Ne- gotiationToM: A benchmark for stress-testing ma- chine theory of mind on negotiation surrounding. In Findings of the Association for Computational Lin- guistics: EMNLP 2024, pages 4211–4241, Miami, Florida, USA. Association for Computational Lin- guistics. Chunkit Chan, Yauwai Yim, Hongchuan Zeng, Zhiy- ing Zou, Xinyuan Cheng, Zhifan Sun, Zheye Deng, Kawai Chung, Yuzhuo Ao, Yixiang Fan, Cheng Ji- ayang, Ercong Nie, Ginny Y. Wong, Helmut Schmid, Hinrich Schütze, Simon See, and Yangqiu Song. 2026. XToM: Exploring the multilingual theory of mind for large language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17681–17716, San Diego, California, United States. Association for Computational Linguistics. 9 Han Chen and Yingrui Li. 2026. Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces. arXiv preprint arXiv:2609.37470. Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. 2024. ToMBench: Benchmarking theory of mind in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 15959–15983, Bangkok, Thailand. Association for Computational Linguistics. Kawai Chung, Chunkit Chan, Yauwai Yim, Yuxuan Liu, Haochen Shi, Weiqi Wang, Qing Zong, Tianshi Zheng, Yixuan Fu, Kai Chung Wong, Hao Liang, Yi- fan Gao, Xi Yang, Janet Hui-wen Hsiao, and Yangqiu Song. 2026. MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoning. arXiv preprint arXiv:2607.26465. Version 2. DeepSeek. 2026. DeepSeek-V4-Flash Update. Release announcement, 31 July 2026. Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa. 2026. Evaluating and Benchmarking the System One Model Jev. arXiv preprint arXiv:2609.37647. Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, and Zhongyu Wei. 2026. General Decision Mod- els: Benchmarking and Insights Beyond Jev. arXiv preprint arXiv:2610.03935. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gersten- berg, and Noah D. Goodman. 2023. Understand- ing Social Reasoning in Language Models with Lan- guage Models. arXiv preprint arXiv:2306.15448. Yuling Gu, Oyvind Tafjord, Hyunwoo Kim, Jared Moore, Ronan Le Bras, Peter Clark, and Yejin Choi. 2026. SimpleToM: Exposing the Gap between Ex- plicit ToM Inference and Implicit ToM Application in LLMs. In International Conference on Learning Representations. Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, and Leo Yu Zhang. 2026. JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models. arXiv preprint arXiv:2609.31142. EunJeong Hwang, Yuwei Yin, Giuseppe Carenini, Peter West, and Vered Shwartz. 2026. Infusing Theory of Mind into Socially Intelligent LLM Agents. In Findings of the Association for Computational Lin- guistics: ACL 2026, pages 11327–11360, San Diego, California, United States. Association for Computa- tional Linguistics. IamBusy. 2026. OpenJev-0.6B model card. Hugging Face. Hazem Ibrahim and Yasir Zaki. 2026. Evaluating Deci- sion Models for Text Annotation in Computational Social Science. arXiv preprint arXiv:2609.24574. Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. 2023. FANToM: A benchmark for stress-testing ma- chine theory of mind in interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14397–14413, Singapore. Association for Computational Linguis- tics. Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5872–5877, Hong Kong, China. Association for Computational Linguistics. Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. 2023. Theory of Mind for Multi-Agent Col- laboration via Large Language Models. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 180–192, Sin- gapore. Association for Computational Linguistics. Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. 2026.JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. arXiv preprint arXiv:2609.26550. Version 5. Yida Lin. 2026. PACT: Pairwise-Anchored Calibrated Tuning for Single-Token Typed Decisions. arXiv preprint arXiv:2609.35865. lostargon. 2026. Tiny-Jev model card. Hugging Face. Meta. 2024. Introducing Llama 3.1: Our most capable models to date. Chunjiang Mu, Ya Zeng, Qiaosheng Zhang, Kun Shao, Chen Chu, Hao Guo, Danyang Jia, Zhen Wang, and Shuyue Hu. 2026. Adaptive Theory of Mind for LLM-based Multi-Agent Coordination. Proceedings of the AAAI Conference on Artificial Intelligence, 40(35):29608–29616. OpenAI. 2023. New models and developer products announced at DevDay. OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence. OpenAI. 2025. Introducing GPT-4.1 in the API. OpenAI. 2026. GPT-5.6: Frontier intelligence that scales with your ambition. David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515–526. Shuwen Qiu, Mingdian Liu, Hengli Li, Song-Chun Zhu, and Zilong Zheng. 2024. MindDial: Enhancing Con- versational Agents with Theory-of-Mind for Com- mon Ground Alignment and Negotiation. In Pro- ceedings of the 25th Annual Meeting of the Special 10 Interest Group on Discourse and Dialogue, pages 746–759, Kyoto, Japan. Association for Computa- tional Linguistics. Delip Rao and Chris Callison-Burch. 2026. Jev vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places. arXiv preprint arXiv:2609.29769. Version 2. Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majum- dar. 2023. Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners. In Proceedings of the 7th Conference on Robot Learn- ing, volume 229 of Proceedings of Machine Learning Research, pages 661–682. PMLR. Kazutoshi Shinoda, Nobukatsu Hojo, Kyosuke Nishida, Saki Mizuno, Keita Suzuki, Ryo Masumura, Hiroaki Sugiyama, and Kuniko Saito. 2025. ToMATO: Ver- balizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind. arXiv preprint arXiv:2501.08838. James W. A. Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, Michael S. A. Graziano, and Cristina Becchio. 2024. Testing theory of mind in large language models and humans. Nature Human Be- haviour. Yu Sun, Junhao Xu, Jiajia Shi, and Zijin Yang. 2026. Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It. arXiv preprint arXiv:2609.26758. Haibo Tong, Zeyang Yue, Feifei Zhao, Erliang Lin, Lu Jia, Ruolin Chen, Yinqian Sun, Qian Zhang, and Yi Zeng. 2026. CogToM: A comprehensive theory of mind benchmark inspired by human cognition for large language models. In Proceedings of the 64th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 31383–31425, San Diego, California, United States. Association for Computational Linguistics. TypeSafe AI. 2026. Jev: Introduction. Technical docu- mentation. Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. arXiv preprint arXiv:2302.08399. Valen-Team. 2026. Valen-Preview-0923 model card. Hugging Face. Guangren Wang. 2026.Mnemon: Raw Records, Fast Judgments, Slow Thoughts. arXiv preprint arXiv:2609.36059. Alex Wilf, Sihyun Lee, Paul Pu Liang, and Louis- Philippe Morency. 2024. Think twice: Perspective- taking improves large language models’ theory-of- mind capabilities. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 8292–8308, Bangkok, Thailand. Association for Computational Linguistics. Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yu- long Chen, and Naihao Deng. 2023. Hi-ToM: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 10691–10706, Singapore. Association for Computational Linguistics. Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. 2024. OpenToM: A comprehensive bench- mark for evaluating theory-of-mind reasoning capa- bilities of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8593–8623, Bangkok, Thailand. Association for Computational Linguistics. Zixiang Xu, Zirui Song, Chiyu Zhang, Xiuying Chen, Xi Liu, Xiyang Hu, and Yue Zhao. 2026. JevOut: Natural Context Can Flip Decision Models. arXiv preprint arXiv:2609.30243. Bo Yang, Jiaxian Guo, Yusuke Iwasawa, and Yutaka Matsuo. 2025. Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflection. arXiv preprint arXiv:2501.15355. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems, volume 36. ZefanCai. 2026. Open-Jev-2B model card. Hugging Face. Jike Zhong, Ming Li, and Yuxiang Lai. 2026. When the Right Answer Is Missing: An Arithmetic- Dependent Rejection Bottleneck in Jev.arXiv preprint arXiv:2609.39496. Appendix A Extended Related Work A.1 ToM benchmarks and application ToM evaluations differ in the evidence available to a character and the reasoning required of a model. ToMi controls answer-space regularities that al- lowed earlier tasks to be solved through dataset biases (Le et al., 2019). BigToM uses causal tem- plates for belief and action questions, while Hi- ToM extends perspective tracking to recursively nested beliefs (Gandhi et al., 2023, Wu et al., 2023). Small alterations to otherwise similar tasks can also expose failures concealed by average success (Ull- man, 2023). Task structure and perspective depth therefore matter when interpreting a ToM score. 11 Broader benchmarks add psychological diversity and conversational information asymmetry. Open- ToM incorporates personality, preferences, inten- tions, and physical and psychological states (Xu et al., 2024). ToMBench organizes diverse social- cognitive tasks and abilities, while CogToM draws on paradigms inspired by human cognition (Chen et al., 2024, Tong et al., 2026). FANToM asks related questions about participants with unequal access to conversational information (Kim et al., 2023). ToMATO uses role-playing models’ private thoughts to construct questions about beliefs, inten- tions, desires, emotions, and knowledge (Shinoda et al., 2025). NegotiationToM studies beliefs, de- sires, and intentions in human negotiations (Chan et al., 2024). These constructions motivate report- ing benchmark-specific results alongside capability groups. SimpleToM distinguishes mental-state inference from behavior prediction and judgment, exposing a gap between explicit attribution and implicit appli- cation (Gu et al., 2026). SimToM instead filters a story to information available to the queried charac- ter before answering a mental-state question (Wilf et al., 2024). Our application category follows the requested judgment, and the feedback interven- tion separates the contribution of a related question from that of a predicted mental state. A.2 Jev evaluation and decision reliability TypeSafe introduces Jev as a model mapping un- structured state and typed questions to choices, scores, or probabilities (Almeida, 2026, TypeSafe AI, 2026). Across 37 datasets, Deußer et al. (2026) report strong results on many tasks but weaker per- formance for some label sets and judgment set- tings. The computational social science evaluation of Ibrahim and Zaki (2026) finds substantial gaps from the best LLM annotators on many tasks de- spite lower measured costs. These findings favor evaluation of the intended decision domain and label space. Confidence routing depends on both the task and the fallback. Li et al. (2026) test accepting con- fident Jev verdicts and escalating uncertain cases, with gains weakening under style-adversarial in- puts and reference-free prose. In rubric judging, Rao and Callison-Burch (2026) find shared errors between Jev and generative judges that limit accu- racy gains from deferral. The annotation study like- wise reports task-dependent confidence behavior (Ibrahim and Zaki, 2026). Our confidence analysis examines retained ToM answers without assuming that a fallback would correct them. Well-formed outputs can still reflect unintended request cues. JevAdvBench compares single-edit attacks with clean and repeated calls (Hu et al., 2026). Sun et al. (2026) show that option-name polarity can redirect selections under unchanged definitions. JevOut optimizes natural-looking con- textual additions that redirect initially correct de- cisions, including on ToMBench (Xu et al., 2026). Our predefined reason attachments probe repair and damage without adversarial context search. Proba- bility Contracts distinguishes valid reports, coher- ent probabilities, and downstream decisions (Chen and Li, 2026). An arithmetic rejection study further separates selecting an available answer from recog- nizing an answer-absent candidate set (Zhong et al., 2026). Together, this work motivates checking se- mantic selection, numerical validity, and candidate coverage separately. JEVal extends general decision-model evalua- tion to bilingual domains, agent workflows, and social simulation, finding that fast local decisions can coexist with lower long-horizon success and ag- gregate prediction errors (Duan et al., 2026). It also introduces InnerJev through reasoning-to-readout self-distillation. PACT studies contrastive evidence, answer-code permutations, and calibration objec- tives for single-token decisions (Lin, 2026). These methods illustrate different routes to a decision in- terface rather than a common architecture for our five typed models. A.3 Multilingual and multimodal reasoning XToM constructs matched ToMi, FANToM, and NegotiationToM instances in English, Chinese, German, French, and Japanese through transla- tion and human refinement (Chan et al., 2026). It distinguishes factual understanding from mental- state inference and finds that strong multilingual language understanding does not ensure compara- ble ToM performance across languages. A sepa- rate Jev cultural-values audit reports response and confidence changes across persona and language conditions (Asseri and Asseri, 2026). Both make language and social framing relevant evaluation dimensions beyond our English-only comparison. MultivationBench studies motivations inferred from accumulated visual and textual narratives (Chung et al., 2026). Maslow’s expanded hierar- chy of needs and Reiss’s basic desires support four multi-label tasks pairing theoretical definitions with 12 context-specific motivation options. This design distinguishes recognizing a psychological category from grounding it in behavior as a story unfolds. Its results identify difficulties in fine-grained mo- tivation selection and correct judgments across a narrative. It complements belief-centered evalua- tion through visual grounding, multi-label motiva- tions, and evolving evidence, which our fixed text questions do not measure. A.4 Mental-state reasoning in agents Explicit belief representations support cooperative planning (Li et al., 2023). MindDial tracks speaker beliefs, predicted listener beliefs, and their differ- ences for common-ground alignment and negoti- ation (Qiu et al., 2024). ToMA connects mental- state generation with dialogue lookahead and in- teractive goal achievement (Hwang et al., 2026). Mnemon assigns record-level judgments to Jev while an LLM plans retrieval and composes re- sponses (Wang, 2026). These systems motivate evaluating repeated selection as a component along- side the surrounding interaction, as discussed in Appendix H. B Evaluation Protocol Table 4 summarizes the model cohorts, questions, candidate constructions, and aggregation for each comparison. Variants reuse source questions, so six conditions in two orders produce twelve re- sponses per question. Primary, reason, feedback, and resource comparisons retain their own frozen populations. B.1 Capability mapping Native task, ability, and mental-state labels deter- mine the six capability assignments in Table 5. Ap- plication additionally requires predicting, select- ing, or judging behavior, including communica- tive appropriateness. Belief attribution, emotional reactions, factual recall, and utterance-truth judg- ments alone are excluded from application. The 152 ToMBench application assignments overlap other capabilities. All 730 ToMBench targets un- derwent an agent-assisted post-evaluation review without item-level model predictions, rather than independent human validation. A question counts once in its benchmark total. B.2 Datasets and harmonization The common primary set contains 11,117 questions. We use normal-length discrete OpenToM, short- context discrete FANToM, English ToMBench, and the ToMi subset accompanying SimToM (Wilf et al., 2024), with fixed development or previously used subsets. Source adapters preserve contexts, references, candidate meanings, identifiers, and question families. Contexts are not truncated, and candidate order is frozen across models. From 11,167 initial questions, we exclude 50 BigToM percept-to-belieftrue-belief items whose sto- ries omit the subsequent event required by the ref- erence. Table 6 reports question, family, option, and context-length statistics. B.3 Input, validation, and aggregation Typed models receive the context asstate, the question asinstructions, and indexed candidate texts, returning a choice and probabilities. Gener- ative models receive the same content and return a candidate label. Strict scoring counts failed or unparseable outputs as wrong. Jev probabilities must cover exactly the offered choices, be finite and lie in[0, 1], sum to one within10 −4 , and as- sign the selected choice a probability within10 −6 of the maximum. Confidence selection uses the selected-answer probability. Exact instructions and an input–output example appear in Appendix J. Primary overall accuracy averages nine bench- mark scores equally, including native controls and unmapped questions. Capability scores average contributing benchmarks and exclude controls and unassigned items. In the eight-model reason com- parison, we average orders within questions, ques- tions within benchmarks, and benchmarks equally. The six-condition intervention instead pools ques- tions separately within each historical-correctness group. These weighting schemes remain distinct. B.4 Models and settings Table 7 records the twelve models and their primary settings. Models receive one source question per request without task-specific fine-tuning. The six- condition study uses Jev, the full pair study uses eight models, matched candidates use Jev and GPT- 3.5, and feedback uses five typed models. Resource settings are documented separately in Appendix G. B.5 Paired uncertainty estimates Within each benchmark, bootstrap sampling resam- ples source-defined families with replacement, re- taining all questions, paired outcomes, and candi- date orders together. Each draw recomputes the original statistic. We use 5,000 draws for primary, 13 material, and feedback comparisons, 2,000 for the six-condition study, and 10,000 for matched can- didates. Percentiles 2.5 and 97.5 define pointwise 95% intervals. Comparators remain fixed across draws. The estimates describe family-sampling variation, without adjustment for multiple compar- isons or comparator selection, and do not measure repeated-inference or deployment variability. Com- plete estimates remain in the reproduction materi- als. C Capability and Benchmark Results C.1 Capability breakdowns Tables 8 and 9 expand Figure 2 with benchmark– capability scores for all twelve models. Missing capability cells are omitted, while controls and unmapped questions enter only benchmark totals. This separates a model’s overall rank from its per- formance on the intended mental-state task. C.2 Sensitivity to native controls Table 10 removes 920 native controls: 600 BigToM, 132 FANToM, 30 Hi-ToM, and 158 ToMi questions. The remaining 10,197 questions retain all 505 unmapped CogToM items and equal benchmark weighting. Jev falls from 73.2% to 69.1%, while GPT-3.5 changes from 71.6% to 70.9%, reversing their order. Their paired difference is−1.79points [−3.48,−0.07]. ToMi contributes strongly to this reversal, with 50.0% for Jev and 83.3% for GPT- 3.5 on its 96 non-control belief questions. The es- timates use 5,000 within-benchmark family draws with seed 2026100401. ToMi retains 89 of its 220 families, and other benchmarks retain all families. The aggregate ranking consequently depends on whether native controls are included. D Confidence-based Selection D.1 Threshold and counts The descriptive threshold of 0.90 applies to the probability assigned to the selected answer on all 11,117 Jev primary responses. Coverage divides retained answers by all questions in a benchmark. Retained error divides incorrect retained answers by retained answers. Table 11 gives the underlying counts for Figure 3, including 33 retained Hi-ToM answers with no observed errors. The threshold is not a validated deployment policy. D.2 Risk–coverage curves Figure 9 extends the comparison across all ob- served thresholds. Equal probabilities are retained together, and empty selections have undefined er- ror. OpenToM and SimpleToM have similar full- coverage error, 40.0% and 40.9%, but retaining about half their answers yields 23.3% error at 48.8% coverage and 35.3% at 48.9% coverage, re- spectively. Confidence therefore separates errors more effectively in the former comparison even though both retain substantial error. E Reason Interference and Candidate Selection E.1 Material construction For the eight-model pair study, 4,761 materials comprise 1,803 reused questions with reasons and newly prepared reasons for 2,958 other existing questions. Generation and review use benchmark references to construct supporting reasons for orig- inal answers, including plausible but mistaken rea- sons for distractors. Review checks story support, queried perspective, attachment, and overt refer- ence leakage. This selects experimental materials rather than establishing human correctness. Generation and separate review use GPT-6.1- Sol with JSON output. Newly prepared materials specify temperature 0, at mostmax(1600, 200K) generation tokens forKoptions, and 1,600 review tokens. Only service failures permit repeat requests. The exact generation and review prompts appear in Appendix J.3. E.2 Six-condition answer selection Figure 4 uses the 1,803 retained reused materials, comprising 249 historically correct and 1,554 his- torically incorrect Jev answers. Material screening starts with 531 correct and 2,533 incorrect cases. Generation and initial screening retain 2,142 cases, and subsequent review retains 1,803. Historical cor- rectness defines the groups, while fresh no-reason calls provide their baselines. Each question retains its original answers un- der six conditions: no reasons, identical neutral filler, aligned reasons, swapped keyed/foil reasons, the keyed reason repeated on every answer, and the foil repeated on every answer. The foil sup- ports Jev’s earlier incorrect choice, or the first non- keyed option for earlier-correct questions. Swap- ping changes only the keyed and foil attachments. Original and reversed orders are pooled separately 14 within each historical group. Instructions and neu- tral text appear in Appendix J.2. Complete plotted values remain in the reproduction materials. E.3 Human annotation Three postgraduate coauthors at English-medium universities, with prior ToM annotation experi- ence, annotated in English as research collabora- tors without additional annotation payment. Writ- ten English–Chinese instructions and a label guide supported the procedure. Figure 10 preserves the complete interface excerpts. The initial sample contains 50 questions per benchmark, totaling 450 questions and 1,879 rea- sons, stratified by material origin and option count without model predictions. Each annotator initially assessed 270 questions. All three shared 90 ques- tions and 374 reasons, while 180 further questions had two annotations and 180 had one. A later answer-only stage collected 450, 450, and 270 judg- ments without adding reason labels. A follow-up covered all 249 earlier-correct intervention ques- tions with three human judgments each. Nineteen exact-material overlaps yield 680 unique questions across stages. Table 12 summarizes coverage and majority outcomes. Annotators first answered from the story, ques- tion, and ordered options, recording evidence before reasons were revealed. Later revisions were saved separately. References, model predic- tions, machine review, and material origin were hidden. Story-support labels wereSUPPORTED, CONTRADICTED,NOT_ESTABLISHED, orUNCERTAIN. Attachment labels wereYES,NO, orUNCERTAIN, asking whether the reason, if true, supported its answer. Non-SUPPORTED/YESjudgments required a note. Answer judgments also allowedMULTIPLE, NONE, or UNCERTAIN. For the initial two-annotator comparison on 374 shared reasons, story-support agreement was 85.56% (Cohen’sκ = 0.768) and attachment agreement 95.19% (κ = 0.833). The later answer- only stage had mean pairwise agreement of 82.47% on 270 questions shared by all three (Fleiss’κ = 0.751). On the 90 initially shared questions, all three endorsed the keyed reason and attachment on 75, while at least one endorsed an alternative answer–reason pair on 30. Agreement and endorse- ment do not establish correctness or complete adju- dication. The post-hoc follow-up analysis establishes an answer-category reference when at least two of three annotators choose the same single category. For the first two annotators it uses the revised an- swer after reason review, or the initial answer if unchanged. The third vote comes from the inde- pendent record saved before other judgments were shown. Non-single-category judgments remain out- side this subset. Of 249 questions, 219 have ma- jority references, including 211 unanimous cases, and all 219 match the benchmark categories. The remaining 30 stay in the annotation records. Table 12 retains both candidate orders and strict scoring, yielding 438 responses per condition on the majority subset. Aligned reasons raise accu- racy from 98.2% to 100.0%, while swapped rea- sons lower it to 24.9%. The paired changes are +1.8and−73.3points, with family-bootstrap in- tervals[0.5, 3.6]and[−77.6,−68.9]. Reason at- tachment therefore remains consequential on ques- tions whose answer categories receive human ma- jority support. The detailed annotation joins and exploratory R/K/S screens remain in the reproduc- tion materials. E.4 Material selection, repair, and damage For original answersC = (c 1 ,...,c K )and rea- sonsE = (e 1 ,...,e K ), the pair study compares bare answers, aligned pairs(c i ,e i ), and allK 2 crossed pairs. Each condition uses a fixed shuffled order and its reversal. Answer-component scoring checks the selected answer, while joint accuracy requires both keyed components and a valid output. Crossing changes length, repetition, and candidate count as well as attachment. Table 13 supplies absolute reused/new accura- cies for Figure 5, together with Jev’s paired repair and damage contributions. Repair changes a wrong no-reason outcome to a correct one, while dam- age changes a correct outcome to a wrong one. Both use all weighted outcomes, so the net change combines repair and signed damage, rather than conditioning on earlier errors or successes. On new materials, Jev’s aligned change is+3.69points [1.93, 5.51], while its crossed change is−0.40 [−3.03, 2.17]. Small observed changes remain de- scriptive, and an interval spanning zero does not establish equivalence. E.5 Matched candidates We select 40 accepted four-answer questions per source from CogToM, ToMATO, and ToMBench, one per family. Twelve families form the pilot and 108 questions the confirmation set. Every condition 15 uses a fixed shuffled order and its reversal with equal benchmark weights. Let(i,j)pair answer iand reasonj, with(0, 0)the reference pair. The eight-candidate sets are E orig =(0, 0), (0, 1), (1, 1), (1, 2), (2, 2), (2, 3), (3, 0), (3, 3), E reassign =(0, 0), (0, 1), (1, 2), (1, 3), (2, 1), (2, 3), (3, 0), (3, 2). Each answer and reason appears twice. Reattach- ment preserves the reference pair’s position, an- swer sequence, total length, and candidates with a reference component. Expansion compares the original eight pairs with all sixteen combinations. Four-pair controls use the original diagonal or (0, 0), (1, 2), (2, 3), (3, 1). Table 14 reports all six conditions. Saved-choice scoring reads choices despite invalid probabilities without repairing them and remains post-hoc. Jev’s reattachment change is+2.31points[−4.17, 8.80], and its expansion change is−4.63 [−10.19, 0.93]. GPT-3.5’s expan- sion change is−13.89 [−19.91,−7.87]. E.6 Semantic outcomes and output validity The five categories in Figure 6 partition requests into both components correct, only the answer cor- rect, only the reason correct, neither correct, and in- valid/failed output. Table 15 distinguishes probabil- ity failures from service failures on the full crossed cohort. Jev has 448 sum failures and three selected- maximum failures among 9,522 responses, while DeepSeek has 843 final HTTP 429 outcomes and four parsing failures. These pooled counts use a different denominator from the figure’s benchmark- equal shares. For example, one saved Jev response selects c3with probability0.9400000000000001, assigns 0.05toc5, and zero elsewhere. Its selection can be correct while its approximately0.99sum fails the numerical contract. All 448 sum failures differ from one by approximately0.01. A saved-response replay finds no change through sum tolerance10 −3 , whereas10 −2 admits most of these outputs. Strict results retain the original10 −4 /10 −6 tolerances. Complete candidate-count strata, tolerance replays, and response examples remain in the reproduction materials. F Mental-state Feedback F.1 Sample and conditions We sample 200 BigToM belief/action pairs and 200 SimpleToM awareness/behavior pairs from fami- lies outside the primary evaluation, without con- sulting references or predictions. BigToM uses 100 families with true- and false-belief variants, bal- anced for explicit initial-belief statements. Simple- ToM uses 200 families, 20 per scenario, with one story each and no behavior-judgment questions. The five typed models first answer the upstream question. Fresh Original controls answer the un- modified downstream question. Question-only adds the related question, Predicted-state also adds the same model’s selected answer text, and Flipped- state supplies the alternative binary answer, which need not be wrong. Conditions preserve the story prefix, downstream question, candidates, and order, changing only the suffix. Appendix J.4 gives its exact text. Table 16 reports all models and sources. BigToM and SimpleToM receive equal weight, and SimpleToM awareness references are uniformly No. F.2 Fixed-No content control Table 17 compares predicted feedback with a post- hoc fixed-No replay on all 200 SimpleToM targets. For each target, we select the archived predicted or flipped response whose supplied upstream answer is No, preserving the original attribution header and consulting no downstream correctness. Re- played inputs exactly match the saved originals. Jev obtains 177/200 correct answers, with every selected option identical to predicted feedback. A constant cue thus reproduces Jev’s recorded selec- tions on this subset. The predicted-minus-question- only change over both sources is+3.75points [1.75, 6.00], but this gain does not establish a bene- fit from instance-specific mental-state predictions. Full upstream-correctness strata remain in the re- production materials. G Cost and Response Time Table 18 joins the values in Figure 8 with serving settings for 11,117 questions per deployment. Ref- erence costs apply OpenRouter public input/output tariffs frozen on 1 October 2026, without batch or cache discounts, to all recorded attempts and request overhead. Totals include conservative al- lowances for missing token usage. Seven deploy- ments have tariffs, with GPT-3.5 using a historical 16 listing. Open-Jev-2B appears only in the timing panel. These estimates are reference prices rather than actual bills. Mean response time runs from evaluation- request start to final outcome, including commu- nication and transport retries but excluding queu- ing and model initialization. Jev and Open-Jev- 2B have separate serial timing passes with one at- tempt per question. Other deployments use concur- rent requests. Open-Jev-2B uses a local M4 Pro with 48 GiB, macOS 15.7.7, vllm-jev 0.1.0, Py- Torch 2.13.0, and Transformers 5.17.0. Llama and DeepSeek use DeepInfra with provider fallback disabled. Direct generative calls use temperature 0 and 32 output tokens. Luna uses low effort in new tool-disabled sessions without that hard cap. Its recorded usage includes additional session context whose complete text was not archived. Provider rate limits and hosted hardware were not recorded. Exact timeouts, attempt limits, output caps, tar- iffs, and missing-usage accounting remain in the reproduction materials. These serving conditions distinguish the observed deployment comparison from isolated architectural latency. H Potential of Jev Application The capability profiles and resource comparisons motivate testing Jev as a selector in dialogue and collaborative agents (Qiu et al., 2024, Li et al., 2023). An LLM could propose partner-state in- terpretations from dialogue history and generate responses while Jev selects among them (Yao et al., 2023). Emotions and social pragmatics are promis- ing starting points given Jev’s observed domain scores, while belief attribution and application leave more room for improvement. Candidates should distinguish the partner’s per- spective from the agent’s information and be re- vised as evidence changes. Evaluate candidate cov- erage separately from selection accuracy when a correct interpretation is present. Reason interfer- ence motivates checking attachment and candidate sensitivity, while task-specific calibration can guide acceptance and clarification (Ren et al., 2023). In- valid outputs need a fallback. These component checks support a proposed integration study, with dialogue quality and collaborative task success eval- uated in the complete system. I Data Access and Reproduction Source benchmarks and models retain their access terms. BigToM, ToMBench, FANToM, CogToM, and SimToM use MIT licences, Hi-ToM Apache- 2.0, SimpleToM ODC-By, OpenToM C BY-NC 4.0, and ToMATO the Meta Llama 3 Community License. We plan to release our added reason mate- rials with attribution under C BY 4.0. The versioned reproduction materials identify source questions and families, subsets, material versions, candidate orders and mappings, separate reference keys, predictions, and available probabil- ity vectors. Annotation stages distinguish initial answers, revisions after reason review, and adjudi- cation using de-identified annotator IDs. Scoring and family-resampling code will retain both orders together. Complete table values, uncertainty esti- mates, detailed annotation screens and joins, tol- erance replays, response examples, and execution settings are preserved separately from the concise appendix. The prepared anonymous review archive con- tains materials, parsed primary and feedback out- comes, and aggregate reason results. Per-question reason outputs and matched-candidate records are archived separately. Complete replay coverage and release of these supplementary materials remain planned. Aggregate appendix data do not substi- tute for those per-question records. 17 Supplementary Tables and Figures Study and cohortSampleTarget and candidate constructionAggregation Primary / 12 models11,117 questions, nine benchmarks K original answers, answer accuracyEqual benchmark weights Confidence / JevSame 11,117 primary questionsSelected-answer probability, coverage and retained error Within benchmark Six conditions / Jev 1,803 materials, split 249 / 1,554 by earlier correctness K answers with six reason conditions, two orders Pooled within historical group Reason pairs / 8 models4,761 materials: 1,803 reused and 2,958 new K answers, K aligned pairs, K 2 crossed pairs, two orders Equal benchmark weights Matched candidates / 2 models 108 questions, one per family, three benchmarks Joint selection among 4, 8, or 16 pairs, two orders Equal benchmark weights Feedback / 5 typed models 200 BigToM and 200 SimpleToM pairs, 300 families Upstream binary state, downstream selection under four suffix conditions Equal benchmark weights Resources / 8 deployments 11,117 questions per deploymentOriginal answer selection, reference cost and response time Benchmark-equal accuracy, mean request time Table 4: Overview of the main comparisons. Condition and order variants reuse questions.Kdenotes original answer count. Costs use seven priced deployments, and timing uses eight. The full pair study also records reason- only controls. BenchmarkKBGESACUNFamilies BigToM–400–200600–1,20050 OpenToM–2384149–2,533149 ToMBench78218145103186152–730323 SimpleToM297–594–891177 FANToM1446227–132–1,80564 ToMATO305289546256–1,39640 CogToM212440469331197–5052,1541,215 Hi-ToM–124–30–15430 ToMi–96–158–254220 Table 5: Benchmark-to-capability question counts. K/B/G/E/S/A denote knowledge, beliefs, goals/desires/intentions, emotions, social pragmatics, and application. C/U denote native controls and unmapped CogToM items. The 152 ToMBench application assignments overlap other capabilities. A dash indicates no assigned questions. BenchmarkQuestionsFamiliesOptionsWords: median BigToM1,20050269 OpenToM2,5331492/3196 ToMBench7303232/461 SimpleToM891177236 FANToM1,805642566 ToMATO1,396404539 CogToM2,1541,215476 Hi-ToM1543015298 ToMi254220259 Total11,1172,2682/3/4/15– Table 6: Evaluated-subset statistics. Words are whitespace-delimited units in the original state, excluding questions and options. 18 ModelCheckpoint or versionEvaluation settings Jev-1.13 typesafe/jev-1.13Native choice interface. Open-Jev-2B ZefanCai/Open-Jev-2BNative choice interface and checkpoint settings. Valen Valen-Team/Valen-Preview-0923Native choice interface and checkpoint settings. Tiny-Jev lostargon/Tiny-JevNative choice interface and checkpoint settings. OpenJev-0.6B IamBusy/OpenJev-0.6BNative choice interface and checkpoint settings. SmolLM3-3B HuggingFaceTB/SmolLM3-3B ,revision a07c9a04f16 Non-thinking greedy decoding, 32 output tokens, batch size four, 16,384-token input limit. GPT-4.1-nano gpt-4.1-nano-2025-04-14Temperature 0, 32 output tokens. GPT-4o-mini gpt-4o-mini-2024-07-18Temperature 0, 32 output tokens. GPT-3.5-1106 gpt-3.5-turbo-1106Temperature 0, 32 output tokens. GPT-5.6-Luna gpt-5.6-lunaLow effort, tool use disabled, no 32-token hard cap. Llama-3.1-8B meta-llama/llama-3.1-8b-instructTemperature 0, 32 output tokens. DeepSeek-V4- Flash deepseek/deepseek-v4-flash-0731Non-thinking mode, temperature 0, 32 output tokens. Table 7: Model versions and primary evaluation settings. All models receive one question per request without task-specific fine-tuning. CapabilityN Jev 1.13 GPT-3.5 1106 GPT-5.6 Luna GPT-4o mini DeepSeek V4-Flash GPT-4.1 nano Llama-3.1 8B Open-Jev 2BValen SmolLM3 3BTiny-Jev OpenJev 0.6B Cross-source capability atlas Knowledge access2,338 73.676.172.671.663.355.354.943.939.136.535.723.4 Beliefs4,178 68.466.963.554.355.351.848.243.644.734.840.231.8 Goals, desires and intentions 1,309 78.574.476.268.869.062.765.352.040.045.948.436.6 Emotions690 84.879.779.574.877.573.065.759.141.056.255.243.5 Social pragmatics383 89.182.584.980.189.974.474.852.550.151.353.856.4 Mental-state application946 70.465.662.760.270.760.359.065.062.353.643.158.8 BigToM Beliefs400 74.568.265.067.064.262.561.557.556.262.055.853.8 Mental-state application200 93.593.092.084.587.560.063.076.562.060.559.553.5 Control600 97.597.396.297.797.094.591.397.091.795.290.895.5 All1,200 89.286.985.185.284.578.176.780.474.978.373.974.6 OpenToM Beliefs2,384 60.162.557.954.161.048.248.645.638.245.233.139.1 Goals, desires and intentions149 57.749.753.049.047.741.645.640.335.642.348.339.6 All2,533 60.061.857.653.860.247.848.445.238.145.034.039.1 ToMBench Knowledge access78 53.865.451.355.143.624.429.526.923.125.69.019.2 Beliefs218 80.383.981.770.667.961.561.057.849.150.040.435.8 Goals, desires and intentions145 75.971.773.163.470.357.265.545.538.643.435.235.2 Emotions103 74.869.968.068.068.961.254.460.245.652.454.448.5 Social pragmatics186 83.974.278.069.983.965.169.953.858.156.554.355.4 Mental-state application152 73.765.865.155.369.157.257.253.342.146.139.539.5 All730 76.775.173.867.070.057.559.951.446.048.141.540.7 SimpleToM Knowledge access297 89.289.687.291.290.265.082.518.959.934.71.73.7 Mental-state application594 44.138.031.040.955.463.656.765.282.754.230.583.5 All891 59.155.249.757.767.064.165.349.775.147.720.956.9 Table 8: Capability exact match (%), part 1. Cross-source rows average contributing benchmarks, while source rows pool assigned questions. Controls and unassigned items enter only All. Overlapping application assignments do not sum to benchmark totals. Bold marks the highest point estimate. 19 CapabilityN Jev 1.13 GPT-3.5 1106 GPT-5.6 Luna GPT-4o mini DeepSeek V4-Flash GPT-4.1 nano Llama-3.1 8B Open-Jev 2BValen SmolLM3 3BTiny-Jev OpenJev 0.6B FANToM Knowledge access1,44681.382.677.482.063.070.360.777.649.349.178.432.2 Beliefs22773.135.224.221.639.638.328.217.632.24.412.37.0 Control13272.094.797.795.586.486.490.997.069.730.393.934.1 All1,80579.677.672.275.361.867.558.871.548.642.171.229.1 ToMATO Knowledge access30577.079.381.071.859.371.159.743.328.926.652.819.7 Beliefs28982.779.280.376.168.968.960.649.133.228.052.922.1 Goals, desires and intentions54686.184.884.277.868.173.865.854.435.039.255.317.2 Emotions25687.180.179.772.370.771.561.750.431.644.955.123.8 All1,39683.681.681.975.166.871.862.650.132.735.254.220.0 CogToM Knowledge access21266.563.766.058.060.445.842.052.834.446.736.842.5 Beliefs44076.672.770.763.472.756.158.635.739.535.030.528.6 Goals, desires and intentions46994.591.394.584.989.878.084.267.850.758.654.854.4 Emotions33192.489.190.984.392.786.481.066.845.671.356.258.0 Social pragmatics19794.490.991.990.495.983.879.751.342.146.253.357.4 Unassigned50567.164.675.259.261.253.946.730.725.742.819.627.7 All2,15481.378.181.572.277.766.565.149.439.449.739.942.5 Hi-ToM Beliefs12450.050.060.537.137.133.117.738.733.933.933.930.6 Control30 100.080.083.366.760.043.320.043.330.040.016.736.7 All15459.755.864.942.941.635.118.239.633.135.130.531.8 ToMi Beliefs9650.083.367.744.831.245.849.046.975.019.862.537.5 Control15881.665.872.881.075.974.172.887.347.579.738.063.9 All25469.772.470.967.359.163.463.872.057.957.147.253.9 Table 9: Capability exact match (%), part 2, using the same model order and scoring as Table 8. Missing source– capability combinations are omitted. ModelAll questionsControl-free GPT-3.5-110671.6270.87 Jev-1.1373.2169.08 GPT-5.6-Luna70.8568.55 GPT-4o-mini66.2961.59 DeepSeek-V4-Flash65.4060.22 GPT-4.1-nano61.3157.14 Llama-3.1-8B57.6454.03 Open-Jev-2B56.6051.64 Valen49.5349.47 Tiny-Jev45.9345.92 SmolLM3-3B48.7042.65 OpenJev-0.6B43.1938.86 Table 10: Benchmark-equal exact match with all 11,117 questions and after removing 920 native controls. Control- free scores retain 10,197 questions, including unmapped items. Rows descend by control-free accuracy. SourceQuestionsKeptWrong BigToM1,20097344 OpenToM2,5331,583459 ToMBench73039739 SimpleToM891418146 FANToM1,80592367 ToMATO1,39665112 CogToM2,1541,31157 Hi-ToM154330 ToMi254872 Pooled11,1176,376826 Table 11: Counts for Jev selection at probability≥ 0.90. Coverage uses all questions, while retained error uses Kept. Native controls remain included. 20 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) BigToM 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) OpenToM 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) ToMBench 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) SimpleToM 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) FANToM 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) ToMATO 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) CogToM 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) Hi-ToM 020406080100 Coverage (%) 0 20 40 60 80 100 Error (%) ToMi Figure 9: Jev risk–coverage curves on the same 11,117 primary questions as the main confidence analysis. Coverage is the percentage of all benchmark questions retained, and error is the percentage of retained answers that are incorrect. Each point retains all answers at or above one observed selected-answer probability, keeping ties together. Orange markers denote the 0.90 threshold. Annotation coverage / follow-up outcomeQuestions Initial sample / targeted follow-up / overlap450 / 249 / 19 Combined unique questions680 Majority reference / unanimous / no majority219 / 211 / 30 Majority references matching benchmark categories219/219 ConditionAccuracy (%)∆ vs. no reasons (p)95% CI for ∆ Full group, original references: 249 questions, 202 source families No reasons96.39— Aligned reasons99.00+2.61[0.40, 5.00] Swapped reasons25.30−71.08[−75.41,−66.87] Human-majority subset: 219 questions, 180 source families No reasons98.17— Aligned reasons100.00+1.83[0.47, 3.56] Swapped reasons24.89−73.29[−77.62,−68.92] Table 12: Annotation coverage and Jev intervention results under benchmark and human-majority references. The 19 overlaps count once. Majority references require at least two matching single-category votes, with unanimous cases included. Both orders yield 498 full-group and 438 majority-subset responses per condition. Accuracy is pooled (%), differences are paired percentage points, and intervals use 2,000 within-benchmark family-bootstrap draws. 21 ModelMaterialsNo reasonsAlignedCrossed Jev-1.13Reused22.9374.3565.10 Jev-1.13New88.1091.7987.70 GPT-5.6-LunaReused36.3170.0567.19 GPT-5.6-LunaNew81.7988.9086.15 GPT-3.5-1106Reused36.3672.3560.52 GPT-3.5-1106New82.2188.3881.58 DeepSeek-V4-FlashReused30.6760.1055.83 DeepSeek-V4-FlashNew66.8780.9577.49 GPT-4o-miniReused30.0460.2747.08 GPT-4o-miniNew78.7787.1974.37 Open-Jev-2BReused32.9151.5237.23 Open-Jev-2BNew63.4971.7765.32 GPT-4.1-nanoReused33.3351.9039.99 GPT-4.1-nanoNew67.8976.5859.19 Llama-3.1-8BReused32.6650.5841.44 Llama-3.1-8BNew58.6268.5853.44 Jev, new materialsRepair (p)Damage (p)Net change (p) Aligned6.96-3.27+3.69 Crossed6.69-7.08-0.40 Table 13: Absolute answer-component accuracy on reused/new reason materials and Jev repair/damage contributions on new materials. Accuracies average benchmarks equally. Reused/new contain 1,803/2,958 existing questions. Repair and signed damage use all weighted outcomes. ConditionJev strictJev saved choiceGPT-3.5 strictJev invalid Answers only71.3072.2271.762 Four original pairs85.6585.6579.630 Four reassigned pairs92.1392.5991.201 Eight original- attachment pairs81.0284.2676.3910 Eight reassigned pairs83.3387.0484.268 All sixteen pairs76.3982.4162.5016 Table 14: Matched-candidate accuracy (%) on 108 questions in two orders, with equal benchmark weights. Jev invalid counts use 216 responses per row, and GPT-3.5 has no invalid outcomes. Answer-only measures answer accuracy, while pair rows measure joint accuracy. Post-hoc saved-choice scoring does not replace strict results. Final failure typeJevDeepSeek Probability sum4480 Selected choice below maximum30 Parsing / candidate ID04 HTTP 4290843 Invalid / failed451847 All crossed responses9,5229,522 Table 15: Final crossed-pair failures. Counts are pooled across two orders, while Figure 6 uses equal benchmark weights. Earlier failed transport attempts are not additional outcomes. 22 Initial Answer Reason Judgments Figure 10: Unfilled excerpts from the annotation interface. Initial Answer: story, question, options, and the initial-answer form, saved before reasons are revealed. Reason Judgments: label guide and the first option’s reason-judgment fields, repeated for every option. The bilingual interface is shown in English for readability. ModelTaskBelief/awarenessOriginalQuestionPredictedFlipped Jev-1.13BigToM action84.5092.0098.0097.5097.00 Jev-1.13SimpleToM behavior91.0065.0080.5088.5075.50 Open-Jev-2BBigToM action58.5073.0071.5067.5066.50 Open-Jev-2BSimpleToM behavior19.0060.0056.5062.0057.00 ValenBigToM action56.5060.5058.0058.0057.50 ValenSimpleToM behavior61.5065.0063.0061.5062.50 Tiny-JevBigToM action62.5055.0055.5054.5058.00 Tiny-JevSimpleToM behavior3.0042.0049.5048.5047.00 OpenJev-0.6BBigToM action54.5053.5051.0051.0050.50 OpenJev-0.6BSimpleToM behavior6.0069.5059.0051.5051.50 Table 16: Mental-state feedback accuracy (%) by model and source, with 200 native upstream questions and 200 action/behavior targets per source. The upstream belief/awareness score is measured once per model. The four downstream conditions use the same target questions. 23 ModelQuestionPredictedFixed No Changed choices Jev-1.131611771770 Open-Jev-2B11312411311 Valen1261231252 Tiny-Jev9997945 OpenJev-0.6B11810310318 Table 17: Correct answers out of 200 SimpleToM behavior targets for question-only cues, predicted feedback, and the same-template fixed-No replay. Changed choices compares fixed No with predicted feedback on all 200 targets. Every output is valid. The replay selects the existing predicted or flipped response whose supplied answer is No, retaining the original attribution header and including every target exactly once. Reference prices and costs ModelInputOutputTotalPer 1k TypeSafe Jev0.04200.00000.32840.0295 GPT-4.1 nano0.10000.40000.48760.0439 GPT-4o mini0.15000.60000.75170.0676 GPT-5.6 Luna0.20001.20009.99290.8989 Llama-3.1-8B0.05000.08000.24630.0222 DeepSeek-V4-Flash0.01081.28000.15350.0138 GPT-3.5-11061.00002.00004.92780.4433 Accuracy, observed time, and serving settings ModelEM (%)Mean (ms)Route / hardwareWorkersAttempts Jev-1.1373.03282.31OpenRouter, hardware NR11 GPT-3.5-110671.621389.13OpenRouter, hardware NR83 GPT-5.6-Luna70.857998.70OpenAI isolated session, hardware NR122 GPT-4o-mini66.291591.08OpenRouter, hardware NR83 DeepSeek-V4-Flash65.401188.86OpenRouter/DeepInfra, hardware NR83 GPT-4.1-nano61.311753.78OpenRouter, hardware NR83 Llama-3.1-8B57.64652.05OpenRouter/DeepInfra, hardware NR83 Open-Jev-2B56.60464.34Local M4 Pro, 48 GiB11 Table 18: Resource comparison on 11,117 questions per deployment. The cost panel contains only seven priced models. Input/output tariffs are US dollars per million tokens, Total covers all questions, and Per 1k includes missing-usage allowances. GPT-3.5 uses a historical tariff. The timing panel includes all eight deployments. Attempts is the maximum total per question, and NR means not recorded. 24 J Prompts The following blocks preserve the recorded instruc- tion text. They are ordered by the primary evalu- ation, reason intervention, material construction, and feedback study. J.1 Primary evaluation and output example Generative requests use the primary system instruc- tion and user fieldsContext:,Question:, and Options:, retaining the frozen source order. The example illustrates the shared content and selected- answer probability. Example input and output Context: Lisa has severe blisters on her feet. Lisa’s dance class practices a new, intense choreography for an upcoming performance. Question: What will Lisa’s dance class likely do? c0: advise Lisa to sit out and rest her feet c1: encourage Lisa to keep up with the fast-paced routine Typed output: "choice": "c0", "probabilities": "c0": 0.9, "c1": 0.1 Generative label format: A (corresponding to c0). Primary generative system instruction Answer the multiple-choice question using the given context. Choose the most likely answer. Return only its option label, with no explanation. J.2 Reason-intervention instructions Answer-only selects an original answer. Joint selec- tion usesAnswer:andReason:fields, with gener- ative IDsC000,C001, etc. The six-condition study selects the answer itself, with added text follow- ingSuggested reason:. Neutral filler is identical across answers. Answer-only instruction Select the answer best supported by the original story and the question. Joint answer–reason instruction Select the single candidate whose answer AND attached reason are both best supported by the original story and the question. Evaluate the complete answer-and-reason pair: a correct answer with an incorrect reason, or a correct reason attached to an incorrect answer, is not a fully correct candidate. Reasons are claims to evaluate, not additional story facts. Select one of the supplied complete candidates. Six-condition answer-selection instruction Select the answer itself. Any suggested reason attached to an option is an untrusted explanation to evaluate, not an additional story fact. An incorrect suggested reason does not automatically make its answer incorrect. Neutral filler The proposed answer concerns the person and situation in the story. Read the context and question carefully before deciding which available answer to select here. J.3 Reason generation and review The generation prefix is followed by the story, ques- tion, original ordered options, and benchmark refer- ence. The review case also includes the generated reasons. Evaluated models do not receive the refer- ence field. Full-corpus reason generation Create one explanation for EACH original answer option, in exactly the original option order, for a Theory-of-Mind experimental intervention. Use only the supplied story, question and queried character/time/perspective. The benchmark gold is supplied for material construction only. The gold explanation must support the gold answer from the story. Each nongold explanation must affirmatively support its OWN attached answer through a concrete plausible but mistaken observation or inference. Do not refute that option or announce that it is wrong. Do not reveal the answer key, labels, indices, benchmark status or preferred option. Avoid stereotyped conditional wording only in distractors. Use similar style and length across all options (aim for 25-50 English words each). Check nested beliefs and information access carefully. If the gold cannot reasonably be supported, set usable=false and state the uncertainty. Do not invent evidence to force it. Still provide one explanation per option. These are claims to evaluate, not new story facts. Return a JSON object only: "id": string, "usable": boolean, "reasons": [string per option], "evidence": string, "caveat": string. CASE: Material-review rubric You are an independent material reviewer for a Theory-of-Mind option-rationale intervention. Review the supplied original story, question, original answer options, benchmark gold answer and the generated explanations. You are checking whether this material is a sound experimental intervention, not whether the benchmark or Jev is right. Check every option, not only the gold answer: 1. The gold explanation must support the gold answer using the supplied story, queried person, time and perspective. Nested beliefs must address the requested mental-state holder and observer. Record material uncertainty when support is insufficient. Do not label the benchmark definitively defective. 25 Material-review rubric (continued) 2. Each nongold explanation is intentionally mistaken. It must affirmatively support its own attached candidate through a concrete erroneous inference or observation. Do not reject merely because it contradicts the story: deliberate error is required. Reject explanations that instead refute their attached answer, give the correct answer, explicitly acknowledge their own error, or fail to support their attached option. 3. Verify original option order, rationale count and answer mapping. Reject correctness leakage, an obvious label cue, or material that cannot implement the intended comparison. 4. Record minor length or style imbalance as warnings. Do not reject for a small deviation from the suggested 25–50 English words alone. Exclude only a concrete validity defect or unresolved material uncertainty. Do not repair, rewrite, or replace the original artifact. 5. Do not consult Jev outcomes, prior review judgments, historical success or the material generator’s identity. All judgments must use the case below. This is gold-informed machine review, not human annotation or a measurement of Jev’s internal reasoning. Return JSON with id, approved (boolean), issues (list of concrete rejection reasons), warnings (list of residual limitations). A rejection must have an issue. An approval must have no unresolved validity issue. Review the full case before deciding. J.4 Mental-state feedback suffixes Suffixes follow two newline characters. Newlines separate the question label, question, attribution, and selected answer text. Predicted and flipped conditions differ only in which upstream answer text is supplied, without references or probabilities. Question-only feedback suffix Related mental-state question: upstream_question Predicted-state and flipped-state suffix Related mental-state question: upstream_question Earlier answer selected by this model (may be incorrect): upstream_selected_option_text 26

Instant research discovery