Paper deep dive
CruxBench: A Benchmark of Information Discovery
Hui Dai, Lina Piao, Nick Merrill, Nadja Flechner, Ezra Karger, Haifeng Xu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/30/2026, 6:53:14 AM
Summary
The paper introduces CruxBench, a benchmark designed to evaluate Large Language Models' (LLMs) ability to perform information discovery by generating 'cruxes'—informative subquestions that help resolve complex forecasting problems. Unlike traditional benchmarks that measure answer accuracy, CruxBench measures the Value of Information (VOI) of generated questions by observing how much their resolution updates real-world beliefs in prediction markets. The study evaluates eight models on 293 forecasting questions, finding that VOI correlates strongly with general model capability (r=0.90), though even frontier models only narrowly outperform random baselines.
Entities (9)
Relation Signals (6)
CruxBench → usesmetric → Value of Information
confidence 98% · CruxBench, a benchmark that grades LLM-generated questions by their Value of Information (VOI)
CruxBench → evaluates → LLMs
confidence 97% · we introduce CruxBench... to evaluate this capability of information discovery
CruxBench → usesdatasource → Polymarket
confidence 95% · Our data comes from a forecasting platform... selected, highly liquid prediction markets.
Value of Information → measuredby → KL Divergence
confidence 94% · We measure the market-belief update around t_ij using KL divergence
CruxBench → correlateswith → Model Capability
confidence 92% · VOI correlates highly with independent measures of model capability (r=0.90)
Claude Opus 4.6 → performson → CruxBench
confidence 90% · Claude-Opus-4.6... only narrowly outperform the random-timing baseline
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many complex real-world tasks is identifying which questions are worth asking in the first place: decomposing a difficult problem into subquestions -- which we call cruxes -- whose answers provide key steps on the path toward solving the target problem. To evaluate this capability of information discovery, we introduce CruxBench, a benchmark that grades LLM-generated questions by their Value of Information (VOI): how much a model-proposed crux updates beliefs about a target forecasting question. CruxBench enjoys a rare combination of three key properties: it is (1) contamination-resistant by construction, since ground truth is generated by future world events; (2) open-ended, admitting unbounded and complex text-based submissions rather than one correct numeric answer; and (3) grounded, with informativeness measured against quantified changes in real-world beliefs. We evaluate a diverse set of eight models on 293 target forecasting questions and find that VOI correlates highly with independent measures of model capability (r=0.90) and captures cruxes' usefulness for answering target questions. However, information discovery remains challenging even for frontier LLMs, which only narrowly outperform a random-timing baseline.
Tags
Links
- Source: https://arxiv.org/abs/2609.35879v1
- Canonical: https://arxiv.org/abs/2609.35879v1
Trouble viewing inline? Open PDF directly →
Full Text
115,406 characters extracted from source content.
Expand or collapse full text
CruxBench: A Benchmark of Information Discovery Hui Dai , Lina Piao11footnotemark: 1 , Nick Merrill, Nadja Flechner, Ezra Karger, Haifeng Xu †thanks: Equal contributions. This research was supported by a grant from Coefficient Giving. The views expressed in this paper do not necessarily reflect the views of the Federal Reserve Bank of Chicago or the Federal Reserve System. Email: ameliadai, linap15, haifengxu@uchicago.edu Affiliation: The University of Chicago Affiliation: Forecasting Research Institute Affiliation: Federal Reserve Bank of Chicago Abstract Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many complex real-world tasks is identifying which questions are worth asking in the first place: decomposing a difficult problem into subquestions – which we call cruxes – whose answers provide key steps on the path toward solving the target problem. To evaluate this capability of information discovery, we introduce CruxBench, a benchmark that grades LLM-generated questions by their Value of Information (VOI): how much a model-proposed crux updates beliefs about a target forecasting question. CruxBench enjoys a rare combination of three key properties: it is (1) contamination-resistant by construction, since ground truth is generated by future world events; (2) open-ended, admitting unbounded and complex text-based submissions rather than one correct numeric answer; and (3) grounded, with informativeness measured against quantified changes in real-world beliefs. We evaluate a diverse set of eight models on 293 target forecasting questions and find that VOI correlates highly with independent measures of model capability (r=0.90r=0.90) and captures cruxes’ usefulness for answering target questions. However, information discovery remains challenging even for frontier LLMs, which only narrowly outperform a random-timing baseline. 1 Introduction Experts and decision makers often make decisions under uncertainty, relying implicitly or explicitly on probabilistic beliefs about future outcomes, especially over long time horizons. They may confront an ultimate question such as: Will the U.S. invade Cuba? Will there be a recession in the U.S.? And if so, over what time horizon? Answering such questions requires reasoning about a large, open-ended set of signals and intermediate events that may predict or affect the ultimate outcome. Developing accurate predictions of the ultimate outcome is therefore important, but an equally important task is to decompose the ultimate question into useful building blocks: subquestions, which we call cruxes, whose resolution can help experts and decision makers update their beliefs about the ultimate outcome efficiently and accurately [39, 37]. Consider the following example: Example. Consider the ultimate question: “Will there be a recession in the U.S. in the next two years?” An AI system might propose the following candidate cruxes: • “Will investment in AI data centers plateau or contract in the next six months?” • “Will the Democratic Party win a majority of seats in the House in the fall?” • “Will the U.S. economy show signs of weakness?” These cruxes differ in quality. The first corresponds to a concrete, economically relevant event whose resolution could directly and directionally update beliefs about the ultimate question. The second may also be informative, but its connection is more indirect, potentially operating through the relationship between political control and economic outcomes. The third, while intuitively related, is vague and difficult to operationalize or resolve. Figure 1: CruxBench pipeline overview. Given an ultimate question UiU_i and the information available at the start of evaluation window j, a generator proposes a crux QijQ_ij that is expected to resolve within the window. A separate resolver identifies the crux’s resolution time tijt_ij using evidence available during the window. We measure the market-belief update around tijt_ij using KL divergence and average across questions to obtain the generator’s VOI score. A good crux, then, is one that is specific, resolvable, and sufficiently informative that its resolution meaningfully shifts beliefs about the ultimate outcome. But how should we evaluate a model’s ability to discover such cruxes? Existing evaluations of large language models (LLMs) largely overlook this capability, with most benchmarks assessing performance on fixed question–answer pairs and static datasets that are vulnerable to training-data contamination. Such benchmarks can test whether a model produces the correct answer or prediction, but they do not capture the open-ended task of proposing new, decision-relevant questions. Forecasting benchmarks address part of this limitation by grounding evaluation in future events, but they still focus primarily on predictive accuracy rather than the discovery of informative intermediate signals [21, 48, 46]. We therefore build on forecasting as a contamination-resistant, real-world-grounded evaluation setting while shifting the target of evaluation from answering questions to discovering informative ones. We introduce CruxBench,11 1 Code and data are available at https://github.com/ai-prophet/cruxbench. a benchmark for evaluating the ability of LLMs to generate informative subquestions. Given an ultimate forecasting question, a model must propose candidate cruxes: intermediate questions whose resolution should update beliefs about the ultimate outcome. We evaluate these cruxes not through human judgment or textual plausibility, but through their value of information (VOI) [36, 20, 14]: the extent to which their resolution updates the belief about real-world forecasting markets. We evaluate a diverse set of 8 models across 6 model families on 293 ultimate forecasting questions, and find that the ability to generate informative cruxes correlates strongly with general model capability (Pearson r=0.90r=0.90, p<0.01p<0.01). Moreover, cruxes from higher-VOI models help external forecasters improve their predictions of the ultimate outcome on average. However, information discovery remains challenging even for frontier models: Claude-Opus-4.6, Gemini-3.1-Pro-Preview, and GPT-5.4 only narrowly outperform the random-timing baseline, whereas weaker models’ scores are not statistically distinguishable from it. CruxBench thus provides a forward-looking and practically relevant evaluation of information discovery, with substantial headroom for future models. Our contributions are summarized below: • CruxBench, a contamination-resistant, open-ended, and real-world-grounded benchmark that evaluates LLMs’ ability to discover informative forecasting subquestions (i.e., cruxes) using value of information. • A scalable pipeline for automatically generating, resolving, and evaluating cruxes using real-world belief updates from selected, highly liquid prediction markets. • The empirical validation that the capability of generating high-VOI cruxes correlates strongly with independent measures of model capability and improves the prediction of ultimate outcomes. 1.1 Additional discussion on related work LLMs in forecasting. There is growing interest in using LLMs to forecast future events. One line of work introduces benchmarks that measure how accurately models answer forecasting questions [21, 48, 43, 46]. Another develops methods to improve forecasting performance, including agentic systems [1, 27], fine-tuning on the binary outcomes of resolved events [19, 40, 41, 42, 8], and post-hoc calibration [10]. Although informative cruxes may surface within a model’s reasoning process, these works primarily evaluate forecasting accuracy rather than a model’s ability to select informative signals. Evaluating information. Earlier work measures information by its effect on uncertainty reduction or decision quality. Nelson [28] surveys the main families of measures for assessing the usefulness of a question: likelihood-ratio measures such as Bayesian diagnosticity [15]; entropy-based measures such as expected information gain [38, 24], which equals the expected KL divergence of the posterior from the prior [22] and coincides with VOI under the logarithmic scoring rule [36, 20, 14]; and impact, the absolute change in beliefs [44]. These measures are not interchangeable and can rank the same questions differently [28, 14]. In the forecasting setting in particular, Rosenberg et al. [37] note that likelihood-ratio metrics ignore the probability that the crux itself occurs, and adopt VOI to capture both the magnitude and the probability of belief updates. Whereas they use cruxes to study disagreement on a single ultimate question (AI existential risk) and rely on human experts’ estimates to compute VOI, we address a different question: benchmarking LLM-generated cruxes at scale. Automatic question generation and evaluation. A related line of work evaluates the questions LLMs ask while collaborating with humans, focusing on information gathering, proactive communication or user engagement [25, 49, 12]. These studies concern closed-world, simulated tasks such as flight-ticket booking, whereas CruxBench targets questions about future world events, whose resolutions are unknown at generation time and whose informativeness can support real-world decision-making under uncertainty. In forecasting, recent work uses LLMs to generate question–answer pairs from news, either to study temporal generalization [9] or to construct fine-tuning data [8, 42]. These pipelines are retrospective: given an event that has already occurred, the model reconstructs a forecasting question whose answer is known, so their evaluation centers on validity criteria such as whether a question is future-tense and resolvable. While Bosse et al. [6] study the automatic generation and evaluation of genuinely forward-looking forecasting questions, they also focus on validity. CruxBench pursues a different and more demanding goal: testing whether models can identify useful future information before the ultimate outcome is known, going beyond validity. 2 Quantifying crux quality via value of information (VOI) Our goal is to evaluate models’ ability to generate informative intermediate questions, or cruxes, whose resolutions can shift beliefs about the ultimate outcome. In this section, we describe how we use value of information to evaluate this capability at the model level. 2.1 Notation and task setup We refer to each long-horizon target as an ultimate question. Formally, for each i=1,…,Ni=1,…,N, the ultimate question is a binary random variable Ui∈0,1U_i∈\0,1\, where Ui=1U_i=1 denotes a Yes resolution and Ui=0U_i=0 denotes a No resolution. At generation time, the resolution of UiU_i is unknown. For each ultimate question UiU_i, each model generates J binary cruxes Qij∈0,1Q_ij∈\0,1\, for j=1,…,Jj=1,…,J, each of which should resolve before UiU_i and be informative about its eventual resolution. For notational simplicity, for an ultimate question U and one of its generated cruxes Q, we define: • p:=P(U=1)p:=P(U=1), the prior belief about U just before Q resolves; • ℙ′:=P(U=1∣Q)P :=P(U=1 Q), the random posterior belief about U given Q, whose randomness is inherited from Q; • p(1):=P(U=1∣Q=1)p^(1):=P(U=1 Q=1) and p(0):=P(U=1∣Q=0)p^(0):=P(U=1 Q=0), the two possible posteriors; • q:=P(Q=1)q:=P(Q=1), the probability that the crux Q resolves Yes. Then ℙ′P takes the value p(1)p^(1) with probability q and p(0)p^(0) with probability 1−q1-q, and the law of total probability gives p=qp(1)+(1−q)p(0)p=q\,p^(1)+(1-q)\,p^(0). When referring to a specific pair (Ui,Qij)(U_i,Q_ij), we attach the subscript ijij to these quantities (e.g., pijp_ij, pij′p _ij, ℙij′P _ij). Our data comes from a forecasting platform, where we observe the prior belief p and the realized posterior p′∈p(1),p(0)p ∈\p^(1),p^(0)\, i.e., the value taken by ℙ′P on the branch of Q that occurred. The counterfactual posterior and the crux probability q are unobserved. Our question is therefore: given the observed belief pairs (pij,pij′)(p_ij,p _ij), how can we quantify the informativeness of a generator’s cruxes QijQ_ij? 2.2 Value of information We assess a crux’s informativeness via the value of information (VOI), which captures how much knowing the resolution of the crux Q would reduce uncertainty about the ultimate question U. Throughout the paper, we adopt entropy as the canonical measure of uncertainty, H(x):=−xlogx−(1−x)log(1−x)H(x):=-x x-(1-x) (1-x) for x∈[0,1]x∈[0,1], with logarithms taken in base 2. VOI is then formally defined as the expected entropy reduction: VOI(Q,U):=H(p)−Q[H(ℙ′)]=H(p)−qH(p(1))−(1−q)H(p(0)).VOI(Q,U)\;:=\;H(p)-E_Q[H(P )]\;=\;H(p)-qH(p^(1))-(1-q)H(p^(0)). (1) Equivalently, (1) is the mutual information I(U,Q)I(U;Q), a natural measure of the information shared between two random variables. A crux has high VOI if its possible resolutions would, on average, move beliefs about U toward greater certainty. Population-level VOI. Ideally, we would evaluate (1) for every ultimate–crux pair (Ui,Qij)(U_i,Q_ij). However, two quantities are not directly observable in the real-world setting. First, we observe only one realized branch of the world: either Q=1Q=1 and posterior p(1)p^(1), or Q=0Q=0 and posterior p(0)p^(0), but never both. Since per-crux VOI is an expectation over both branches, it is not identifiable from one realized history. Second, the probability q=P(Q=1)q=P(Q=1) is itself unobserved and would have to be estimated, introducing additional noise. Instead, we target VOI at the population level: VOI:=(i,j)[VOI(Qij,Ui)]=(i,j)[H(pij)−Qij[H(ℙij′)]],VOI\;:=\;E_(i,j)\! [VOI(Q_ij,U_i) ]\;=\;E_(i,j)\! [H(p_ij)-E_Q_ij\! [H(P _ij) ] ], (2) where the outer expectation (i,j)E_(i,j) is over an ultimate–crux pair drawn at random from the population of such pairs, and the inner expectation QijE_Q_ij is over the resolution of the crux QijQ_ij. 2.3 Estimating VOI Let D denote the observed dataset of ultimate–crux pairs. For each (i,j)∈(i,j) , we observe the prior belief pijp_ij just before QijQ_ij resolves and the realized posterior belief pij′p _ij after QijQ_ij resolves. To estimate (2) from single-branch observations, we seek a per-sample function g:[0,1]2→ℝg [0,1]^2 depending only on the observable pair (p,p′)(p,p ) such that Q[g(p,ℙ′)]=VOI(Q,U).E_Q\! [g(p,P ) ]\;=\;VOI(Q,U). For any such g, the resulting estimator is unbiased for the population VOI in (2) (and consistent under standard sampling conditions, e.g., independently sampled ultimate questions): VOI^g:=1||∑(i,j)∈g(pij,pij′). VOI^g\;:=\; 1|D| _(i,j) g(p_ij,p _ij). Estimator 1: Change in entropy. A straightforward choice of g is the realized reduction in entropy, ΔH(p,p′):=H(p)−H(p′), H(p,p ):=H(p)-H(p ), with the corresponding unbiased estimator (see a proof in Appendix B) VOI^ΔH:=1||∑(i,j)∈ΔH(pij,pij′). VOI H\;:=\; 1|D| _(i,j) H(p_ij,p _ij). (3) Estimator 2: KL divergence. Another natural choice is the KL divergence of the realized posterior from the prior, viewing each as a Bernoulli distribution over U:22 2 This requires p,p′∈(0,1)p,p ∈(0,1). In practice, clipping can be applied to restrict the domain of p,p′p,p . KL(p,p′):=KL(Bern(p′)∥Bern(p))=p′logp′p+(1−p′)log1−p′1−p.KL(p,p )\;:=\;KL\! (Bern(p )\, \|\,Bern(p) )\;=\;p p p+(1-p ) 1-p 1-p. (4) Note that KL(p,p′)KL(p,p ) takes its arguments in the (prior, posterior) order of g, the reverse of their order inside KL(⋅∥⋅)KL(·\,\|\,·). This choice also yields an unbiased estimator of the population VOI (see a proof in Appendix B): VOI^KL:=1||∑(i,j)∈KL(pij,pij′). VOI^KL\;:=\; 1|D| _(i,j) KL(p_ij,p _ij). (5) Extension beyond binary outcomes. Although CruxBench considers binary questions, our framework can extend to multi-outcome ultimate questions by replacing Bernoulli beliefs with categorical distributions and using categorical KL divergence. We provide the full formulation in Appendix B. Remark 1 (Choice of estimators) Although both VOI^ΔH VOI H and VOI^KL VOI^KL are unbiased estimators of the same population VOI, they may differ substantially in finite samples. Throughout the paper, we use VOI^KL VOI^KL as our primary estimator for CruxBench for several reasons: it is nonnegative on every realized update, aligns with the interpretation of information as belief revision, and has a local first-order variance advantage in small- and moderate-update regimes. See more details of theoretical and empirical comparison in Appendix C. 3 CruxBench construction 3.1 Pipeline overview Figure 1 summarizes our end-to-end CruxBench pipeline for evaluating a model’s ability to generate informative cruxes. To evaluate information discovery from real-world data, we require a setting where belief updates (p,p′)(p,p ) can be observed over time at scale. Therefore, a natural starting point is to use prediction markets: platforms where users bet on the outcomes of real-world events, with the contract prices reflecting an aggregated probabilistic belief about each outcome. These prices are relatively well-calibrated estimates of event probabilities.33 3 The median Polymarket price has Brier =0.116=0.116 and AUC =0.894=0.894 on our dataset, substantially better than the strongest model baseline P(U)P(U) (Gemini-3.1-Pro-Preview: Brier =0.195=0.195, AUC =0.727=0.727). Each selected market question serves as one ultimate question UiU_i. Given a fixed set Ui\U_i\, we benchmark models in three steps. • Step 1: Crux generation. For each ultimate question UiU_i, we provide all evaluated models with the same fixed information set and prompt each to independently generate its own set of candidate cruxes Qij\Q_ij\. • Step 2: Crux resolution. Since generated cruxes are typically not themselves traded in prediction markets, we resolve them using a separate resolver agent. For each crux, the resolver assigns a binary outcome Qij∈0,1Q_ij∈\0,1\ and a resolution timestamp tijt_ij. • Step 3: VOI estimation. For each resolved crux, we read the prior pijp_ij and realized posterior pij′p _ij from the price history of UiU_i shortly before and after tijt_ij. Aggregating (pij,pij′)(p_ij,p _ij) across all of a model’s cruxes using VOI^KL VOI^KL (defined as (5)) yields the model’s benchmark score. 3.2 Construction details Crux generation. For each ultimate question UiU_i resolving at time TiresT_i^res, we partition the interval [max(t0,Tistart),Tires−τ][ (t_0,T_i^start),T_i^res-τ] into non-overlapping 30-day windows, where TistartT_i^start is the question’s start date, t0t_0 is a fixed cutoff, and τ=3τ=3 days is a buffer so the crux resolves before UiU_i’s own end-of-life price flows. We randomly sample J windows per question. For each sampled window [tjstart,tjend][t_j^start,\,t_j^end], the model is prompted to generate one crux QijQ_ij that resolves within that window, therefore having J cruxes per ultimate question. See Appendix M.1 for the generation prompt. To prevent data leakage, we set t0=September 1, 2025t_0=September 1, 2025, since all evaluated models report a knowledge cutoff or release date before this (Table 1), and disable web search during generation. Each model, instead, receives a standardized information set consisting of up to 10 relevant news articles published before the target window start time tjstartt_j^start, retrieved using the AskNews API.44 4 AskNews (https://asknews.app) provides quality-controlled articles with publication timestamps, allowing reliable date filtering. We avoid general search engines such as Google because their date restrictions are known to be unreliable. For example, pages may be updated after publication without a corresponding change to the displayed publication date [34]. This both supplies the recent context and ensures that all models generate cruxes from the same underlying information. However, because models may differ in training data and cutoff dates, this procedure does not fully eliminate potential information asymmetries. Our design should therefore be interpreted as controlling for, rather than completely removing, pretraining-based differences across models. Crux resolution. Each generated crux is resolved by a separate resolver agent, distinct from the evaluated models, with access to the AskNews API. The resolver returns whether the crux is resolvable; if so, it also returns a binary outcome Qij∈0,1Q_ij∈\0,1\ and a resolution timestamp tijt_ij at which sufficient evidence appears. We then use a separate timestamp-verification agent to verify resolution timestamps against the underlying public evidence and align cruxes across models that resolve on the same event. Finally, we validate the resolver against human annotations and find 92.6%92.6\% agreement. We measure this by randomly sampling about 20 questions from each model and checking the binary outcome and resolution time through (a) manual search, (b) verification against the resolution criteria, and (c) review of the resolver’s reasoning. The resolution-label accuracy is 96.9%96.9\%, and 95.7%95.7\% of correctly labeled cruxes also have the correct resolution date. See Appendix M.2 for the resolution prompts. VOI estimation. For each resolved crux, we estimate the prior and posterior beliefs about UiU_i from its market price history around the resolution time tijt_ij: p^ij,p^ij′=24-hour average “Yes” price of Ui before / after tij. p_ij,\, p _ij=24-hour average ``Yes′ price of U_i before / after t_ij. The benchmark score for a model is the empirical KL estimator from Section 2.3: VOI^KL:=1||∑(i,j)∈KL(p^ij,p^ij′). VOI^KL:= 1|D| _(i,j) KL( p_ij, p _ij). (6) 4 Model evaluations on CruxBench 4.1 Dataset We construct the benchmark from a snapshot of all Polymarket markets pulled on 2026-03-25 from its public APIs.66 6 https://gamma-api.polymarket.com/markets, https://clob.polymarket.com/prices-history We select resolved binary markets that are (i) topically suited to forecasting decomposition (excluding sports, crypto, and weather markets), and (i) liquid and uncertain enough over our evaluation period to admit nontrivial belief updates. We additionally deduplicate near-variants of the same event: a single Polymarket event often appears as several markets at different deadlines or thresholds (e.g., Fed cuts 25 bps vs. Fed cuts 50+ bps). For each remaining market we sample one active 30-day evaluation window uniformly at random. The resulting dataset contains ||=293|D|=293 ultimate questions, each paired with one window. That is, we choose J=1J=1 and generate one crux per ultimate. See Appendix D.1 for details on data filtering. For the price histories, we construct volume-aware prices that downweight raw prices with little nearby trading activity, as described in Appendix D.2. 4.2 Experimental setup We evaluate eight generators, spanning frontier and open-weight model families, on CruxBench using the pipeline in Section 3.77 7 Model access and decoding parameters can be found in Appendix F. Claude-Sonnet-4.6 serves as the resolver for all generators, and Claude-Sonnet-5 as the timestamp verifier. To ensure comparability across models, unresolvable cruxes are scored as KL(p^,p^′)=0KL( p, p )=0 rather than dropping them, as models could otherwise benefit from generating vague or non-resolvable questions. We also introduce two baselines. First, a random baseline, the average KL(p^,p^′)KL( p, p ) over all candidate hours in the active window. Equivalently, it is the expected score of a crux whose resolution time is chosen at random, capturing the average market information gain over time. Second, a hindsight oracle, the maximum KL(p^,p^′)KL( p, p ) over all candidate hours in the active window, representing the moment that retrospectively shows the largest information gain. Note that it serves as an upper bound rather than an attainable target, since the largest market move may reflect noise or unexpected events that a generator could not identify in advance. 4.3 Main findings CruxBench separates frontier models from weaker generators. As shown in Table 1, only the three frontier models (Claude-Opus-4.6, Gemini-3.1-Pro-Preview, and GPT-5.4) significantly outperform the random baseline, while the remaining models do not significantly outperform random timing (examples in Appendix E). At the same time, even the strongest generator remains nearly an order of magnitude below the hindsight oracle (0.1290.129). Although the oracle is a retrospective ceiling that no real generator may fully reach, this gap indicates substantial headroom. We say a crux is resolvable if its outcome can be unambiguously evaluated as true or false at a specific time by an LLM resolver. Most generators achieve high resolvability rates (Table 1), showing that they can produce well-formed, though not necessarily informative, cruxes. For typical resolvability failure modes, see Appendix L. These results suggest that crux generation is a nontrivial capability that current models do not uniformly possess. The task requires more than producing plausible and resolvable questions. Models must identify which near-term events are likely to occur and reason about how those events would update beliefs about the ultimate question. CruxBench therefore serves as a forward-looking target as model capabilities advance, measuring progress on a capability with direct value for human decision-making: surfacing informative cruxes that guide where attention should be focused. Table 1: Performance on CruxBench (n=293n=293). “Resolvable” reports the number and percentage of cruxes with resolvable=true. Random baseline is the per-window mean of VOI^KL VOI^KL over all hours in the active window (random-timing baseline); oracle is the per-window maximum (best-possible pivot in hindsight). Bold indicates the best-performing model. Stars denote uncorrected one-sided paired t-tests against the random baseline: ∗p<0.05^*\,p<0.05, p∗∗<0.01^**\,p<0.01. Cutoffs marked † use the model’s public release date as a proxy where an official training-data cutoff is not published. Model VOI^KL VOI^KL (bits) 95% CI # resolved (rate) Knowledge cutoff Claude-Opus-4.6 0.016∗ [0.009, 0.023] 279 (95.2%) Aug 2025 Gemini-3.1-Pro-Preview 0.014∗^* * [0.007, 0.020] 274 (93.5%) Jan 2025 GPT-5.4 0.013∗^* * [0.007, 0.019] 268 (91.5%) Aug 2025 DeepSeek-V3.1 0.012 [0.003, 0.020] 262 (89.4%) Aug 2025† Qwen3-235B-A22B 0.012 [0.005, 0.018] 271 (92.5%) Jul 2025† Llama-3.3-70B 0.010 [0.005, 0.016] 252 (86.0%) Dec 2023 Llama-3.1-8B-Instruct 0.010 [0.005, 0.015] 271 (92.5%) Dec 2023 GPT-3.5-Turbo 0.009 [0.005, 0.013] 275 (93.9%) Sep 2021 Random baseline 0.007 [0.006, 0.009] — — Oracle 0.129 [0.100, 0.157] — — 4.4 VOI reflects model capability Figure 2: VOI^KL VOI^KL correlates with model capability. (a) vs. Epoch AI capability score. (b) vs. each generator’s forecasting performance, measured by predicting P(U)P(U) at window start and scored by −Brier(P(U))-Brier(P(U)) (left) and AUC(P(U))AUC(P(U)) (right). Brier score is sign-flipped so that higher is better. Figure 2 shows that crux quality, as measured by VOI^KL VOI^KL, tracks both general and forecasting-specific measures of model capability. Panel (a) compares each generator’s score against the Epoch Capability Index (ECI) [13], a composite measure of general model capability that aggregates over 50 benchmarks. The two are strongly positively correlated (r=0.90r=0.90, p<0.01p<0.01). Panel (b) compares VOI against each generator’s own forecasting performance on the same ultimate questions: each generator predicts P(U)P(U) at window start using the information set described in Section 3.2, scored by negative Brier score [7] and AUC. Crux quality correlates strongly with both (r=0.76r=0.76 for −Brier-Brier, r=0.85r=0.85 for AUC). Together, these correlations support crux generation as a capability tied to both general model performance and forecasting skill. 4.5 Good cruxes improve downstream forecasting An important test of CruxBench is whether its scores predict downstream usefulness. Therefore, we examine if presented with a generator’s crux, does the crux improve a forecaster’s prediction of U, and is the improvement correlated with VOI? To test this, we evaluate a separate forecaster on each ultimate question at the window start date, with access to the same news articles available to the generator. We compare two setups: (i) a baseline where the forecaster predicts P(U)P(U) directly, and (i) crux-informed forecast P(U∣Q)P(U Q), where the forecaster predicts the ultimate given one generator’s crux Q and its eventual resolution. Thus, the only difference between the two is the information contributed by the resolved crux. We also note that this setup is intentionally not realistic: in practice, a forecaster at window start date would not have selective access to one future event’s outcome. We allow this leakage in order to study the contribution of a single crux’s resolution to predicting U. Results are averaged across three external forecasters: Claude-Haiku-4.5, Gemini-3-Flash-Preview, and GPT-5-Mini. Figure 3 shows that VOI^KL VOI^KL correlates positively with downstream forecasting performance: r=0.65r=0.65 for −Brier-Brier and r=0.68r=0.68 for AUC. The highest-VOI generators, Claude-Opus-4.6 and Gemini-3.1-Pro-Preview, produce the largest gains. For example, when given the resolved cruxes generated by Claude-Opus-4.6, the Brier score decreases from 0.198 to 0.176 and AUC increases from 0.690 to 0.730. This suggests that higher-VOI cruxes tend to be more useful to forecasters. Compared with the no-crux baseline, cruxes from all eight generators improve the Brier score, but not all improve AUC. This is plausible because small updates in the right direction can reduce squared error (better Brier score). AUC ignores probability magnitudes and depends only on ranking, so it improves only when cruxes discriminate Yes- from No-resolving ultimates. The top two models produce the largest gains in both Brier score and AUC, whereas the third-ranked model, GPT-5.4, shows only a modest Brier improvement and no AUC improvement. Manual inspection suggests that this reflects a difference in crux style: many of GPT-5.4’s high-KL88 8 We note that a crux’s KL is a realized belief update and should not be interpreted as that crux’s true VOI. The benchmark’s VOI scores are computed at the model level, as discussed in Section 2.2. cruxes are based on public announcements or expressions of intent rather than concrete actions or institutional commitments. Such cruxes can coincide with market movements but may provide weaker evidence about the ultimate outcome. Furthermore, VOI and downstream forecasting gains capture different aspects of crux quality: VOI measures the informativeness of a model’s generated cruxes, while downstream improvement additionally depends on whether the forecaster can effectively incorporate that crux signal. Therefore, although VOI is broadly predictive of downstream usefulness, models with similar VOI may still differ in how effectively their cruxes communicate actionable information. Figure 3: VOI^KL VOI^KL correlates with downstream forecasting utility for separate forecasters. For each generator, three forecasters (Claude-Haiku-4.5, Gemini-3-Flash-Preview, GPT-5-Mini) predict P(U∣Q)P(U Q) given the crux information; results are averaged across the three. 4.6 Cross-model similarity of generated cruxes Cruxes generated by different models for the same ultimate are often semantically similar, with mean pairwise cosine similarity ≈0.75≈ 0.75 (Figure 4(a)).99 9 Cosine similarities are computed on the concatenated question and resolution-criteria text, embedded by OpenAI’s text-embedding-3-small [30]. However, semantic similarity does not imply operational agreement. Among pairs with cos≥0.8 ≥ 0.8 in which both cruxes specify a numerical threshold, 84% use different numerical thresholds (Figure 11), and 24% receive different Yes/No resolutions (Figure 4(b)). These operational differences matter because thresholds and deadlines determine when a crux resolves and, consequently, whether it captures the relevant belief update. Take the stolen Louvre jewels case in Appendix E as an example. For the ultimate question of whether any of the jewels would be returned by December 31, Claude-Opus-4.6 asks whether French police would identify or arrest a suspect, while DeepSeek-V3.1 requires a suspect to be both arrested and charged. Although these two cruxes are semantically similar with cos=0.924 =0.924, they receive very different realized KL scores. When French police arrested two suspects on October 25, the market updated substantially, giving the first crux KL=0.135KL=0.135 bits. Formal charges came four days later, after the update had largely been priced in, so the stricter crux captures only KL=0.002KL=0.002 bits. Thus, even semantically similar cruxes can receive very different scores: informative crux generation requires not only identifying the right event, but also specifying an appropriate resolution criterion. Figure 4: Semantic similarity is only a weak proxy for crux equivalence. (a) Mean cosine similarity between cruxes generated by each pair of models for the same ultimate question (mean=0.75=0.75, n=293n=293). (b) Outcome disagreement, measured as 1−P(sameYes/Noresolution)1-P(same\ Yes/No\ resolution), and the median absolute difference in resolution dates, grouped by cosine-similarity of crux pairs. 5 Conclusion, limitations, and future directions This paper presents CruxBench, a benchmark for evaluating the ability of language models to generate informative intermediate questions for forecasting tasks. Unlike traditional benchmarks that evaluate answer accuracy or forecasting performance, CruxBench evaluates information discovery: the ability to identify which future events would most shift beliefs about an ultimate question. By grounding evaluation in prediction markets, CruxBench provides a scalable, open-ended, and contamination-resistant framework for studying crux generation. Across 293 forecasting questions, we find that (i) CruxBench scores align strongly with independent measures of model capability, with weaker models performing no better than a random baseline; (i) cruxes from higher-VOI models generally improve downstream forecasting performance; and (i) informativeness is often driven by differences in operationalization rather than semantic similarity alone. At the same time, the large gap between current models and the hindsight oracle suggests substantial room for improvement in models’ ability to identify pivotal uncertainties that drive ultimate outcomes. Another open question is to discover which of VOI^ΔH VOI H and VOI^KL VOI^KL, or any affine combination of the two whose weights sum to one (which remains unbiased), is the most statistically efficient estimator. Finally, we discuss potential limitations of CruxBench, which give rise to interesting future directions. First, while per-crux VOI would be the ideal target, recovering it requires accurate estimates of the event probability P(Q)P(Q) and the conditional probability P(U∣Q)P(U Q), for which we do not have a principled estimation approach. We therefore report only population-level estimates. Second, we rely on market prices as a proxy for event probabilities. Although prediction markets with high liquidity are generally well-calibrated, noise remains. Moreover, observed belief updates (p,p′)(p,p ) cannot be cleanly attributed to the crux alone: there may be unrelated price fluctuations and simultaneous information arrivals contributing to the measured update. Finally, our experiments are retrospective rather than live. This design allows us to evaluate many resolved questions at scale, but it also limits our ability to test the latest models due to the data contamination concern. While our framework can be naturally extended to live evaluation, we leave it to future work. Acknowledgments and Disclosure of Funding This work is supported in part by the AI2050 program at Schmidt Sciences (Grant G-24-66104). References [1] R. Alur, B. C. Stadie, D. Kang, R. Chen, M. McManus, M. Rickert, T. Lee, M. Federici, R. Zhu, D. Fogerty, et al. (2025) AIA forecaster: technical report. arXiv preprint arXiv:2511.07678. Cited by: §1.1. [2] Anthropic (2025) Introducing Claude Haiku 4.5. Note: https://w.anthropic.com/news/claude-haiku-4-5Accessed: 2026-05-06 Cited by: Table 5. [3] Anthropic (2026) Introducing Claude Opus 4.6. Note: https://w.anthropic.com/news/claude-opus-4-6Accessed: 2026-05-06 Cited by: Table 5. [4] Anthropic (2026) Introducing Claude Sonnet 4.6. Note: https://w.anthropic.com/news/claude-sonnet-4-6Accessed: 2026-05-06 Cited by: Table 5. [5] Anthropic (2026) Introducing Claude Sonnet 5. Note: https://w.anthropic.com/news/claude-sonnet-5Accessed: 2026-08-28 Cited by: Table 5. [6] N. I. Bosse, P. Mühlbacher, J. Wildman, L. Phillips, and D. Schwarz (2026) Automating forecasting question generation and resolution for AI evaluation. arXiv preprint arXiv:2601.22444. Cited by: §1.1. [7] G. W. Brier (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 (1), p. 1–3. Cited by: §4.4. [8] N. Chandak, S. Goel, A. Prabhu, M. Hardt, and J. Geiping (2026) Curating the future: a scalable recipe for training open-ended forecasters. In ICML, Cited by: §1.1, §1.1. [9] H. Dai, R. Teehan, and M. Ren (2025) Are LLMs prescient? a continuous evaluation using daily news as the oracle. In ICML, Cited by: §1.1. [10] H. Dai, R. Teehan, P. Torabian, and M. Ren (2026) Aligning LLMs with human uncertainty: a Beta-Bernoulli calibrator for LLM forecasting. arXiv preprint arXiv:2605.27668. Cited by: §1.1. [11] DeepSeek (2025) DeepSeek-V3.1 release. Note: https://api-docs.deepseek.com/news/news250821Accessed: 2026-05-06 Cited by: Table 5. [12] Y. R. Dong, T. Hu, Z. Hui, C. Zhang, I. Vulić, A. Bobu, and N. Collier (2026) Value of information: a framework for human–agent communication. In ACL, Cited by: §1.1. [13] Epoch AI (2026) Capabilities & Benchmarking. Note: Accessed: 22 Sep 2026 External Links: Link Cited by: §4.4. [14] A. Frankel and E. Kamenica (2019) Quantifying information and uncertainty. American Economic Review 109 (10), p. 3650–3680. Cited by: Appendix G, §1, §1.1. [15] I. J. Good (1950) Probability and the weighing of evidence. Charles Griffin, London. Cited by: §1.1. [16] Google (2025) Gemini 3 Flash model card. Note: https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Flash-Model-Card.pdfAccessed: 2026-05-06 Cited by: Table 5. [17] Google (2026) Gemini 3.1 Pro model card. Note: https://deepmind.google/models/model-cards/gemini-3-1-pro/Accessed: 2026-05-06 Cited by: Table 5. [18] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: Table 5. [19] D. Halawi, F. Zhang, C. Yueh-Han, and J. Steinhardt (2024) Approaching human-level forecasting with language models. In NeurIPS, Cited by: §1.1. [20] R. A. Howard (1966) Information value theory. IEEE Transactions on Systems Science and Cybernetics 2 (1), p. 22–26. Cited by: §1, §1.1. [21] E. Karger, H. Bastani, C. Yueh-Han, Z. Jacobs, D. Halawi, F. Zhang, and P. E. Tetlock (2025) ForecastBench: a dynamic benchmark of AI forecasting capabilities. In ICLR, Cited by: §1, §1.1. [22] S. Kullback and R. A. Leibler (1951) On information and sufficiency. The Annals of Mathematical Statistics 22 (1), p. 79–86. Cited by: §1.1. [23] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In SOSP, Cited by: Appendix F. [24] D. V. Lindley (1956) On a measure of the information provided by an experiment. The Annals of Mathematical Statistics 27 (4), p. 986–1005. Cited by: §1.1. [25] D. Mazzaccara, A. Testoni, and R. Bernardi (2024) Learning to ask informative questions: enhancing LLMs with preference optimization and expected information gain. In Findings of EMNLP, Cited by: §1.1. [26] Meta (2024) Llama 3.3 model card. Note: https://w.llama.com/docs/model-cards-and-prompt-formats/llama3_3/Accessed: 2026-05-06 Cited by: Table 5. [27] K. Murphy (2026) Agentic forecasting using sequential Bayesian updating of linguistic beliefs. arXiv preprint arXiv:2604.18576. Cited by: §1.1. [28] J. D. Nelson (2005) Finding useful questions: on Bayesian diagnosticity, probability, impact, and information gain. Psychological Review 112 (4), p. 979–999. Cited by: §1.1. [29] H. Ng, L. Peng, Y. Tao, and D. Zhou (2026) Price discovery and trading in modern prediction markets. SSRN Electronic Journal. External Links: Link Cited by: §A.1. [30] OpenAI Developers (2024) text-embedding-3-small. Note: https://developers.openai.com/api/docs/models/text-embedding-3-smallAccessed: 2026-05-06 Cited by: footnote 9. [31] OpenAI (2024) Introducing APIs for GPT-3.5 Turbo and Whisper. Note: https://openai.com/index/introducing-chatgpt-and-whisper-apis/Accessed: 2026-05-06 Cited by: Table 5. [32] OpenAI (2025) GPT-5 mini. Note: https://platform.openai.com/docs/models/gpt-5-miniModel documentation. Accessed: 2026-05-06 Cited by: Table 5. [33] OpenAI (2026) Introducing GPT-5.4. Note: https://openai.com/index/introducing-gpt-5-4/Accessed: 2026-05-06 Cited by: Table 5. [34] D. Paleka, S. Goel, J. Geiping, and F. Tramèr (2026) Pitfalls in evaluating language model forecasters. In ICLR, Cited by: footnote 4. [35] Qwen Team (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Table 5. [36] H. Raiffa and R. Schlaifer (1961) Applied statistical decision theory. Harvard Business School Publications, Division of Research, Graduate School of Business Administration, Harvard University. Cited by: §1, §1.1. [37] J. Rosenberg, E. Karger, Z. Jacobs, M. Hickman, A. Morris, H. Durland, O. Kuusela, and P. E. Tetlock (2025) Belief updating in AI-risk debates: exploring the limits of adversarial collaboration. Risk Analysis 45 (12), p. 4350–4366. Cited by: §1.1, §1. [38] C. E. Shannon (1948) A mathematical theory of communication. The Bell System Technical Journal 27 (3), p. 379–423. Cited by: §1.1. [39] P. E. Tetlock and D. Gardner (2016) Superforecasting: the art and science of prediction. Random House. Cited by: §1. [40] B. Turtel, D. Franklin, and P. Schoenegger (2025) LLMs can teach themselves to better predict the future. arXiv preprint arXiv:2502.05253. Cited by: §1.1. [41] B. Turtel, D. Franklin, K. Skotheim, L. Hewitt, and P. Schoenegger (2025) Outcome-based reinforcement learning to predict the future. arXiv preprint arXiv:2505.17989. Cited by: §1.1. [42] B. Turtel, P. Wilczewski, D. Franklin, and K. Skothiem (2026) Future-as-label: scalable supervision from real-world outcomes. arXiv preprint arXiv:2601.06336. Cited by: §1.1, §1.1. [43] Z. Wang, X. Zhou, Y. Yang, B. Ma, L. Wang, R. Dong, and A. Anwar (2025) OpenForecast: a large-scale open-ended event forecasting dataset. In COLING, Cited by: §1.1. [44] G. L. Wells and R. Lindsay (1980) On estimating the diagnosticity of eyewitness nonidentifications. Psychological Bulletin 88 (3), p. 776–784. Cited by: §1.1. [45] J. Wolfers and E. Zitzewitz (2004) Prediction markets. Journal of Economic Perspectives 18 (2), p. 107–126. Cited by: §A.2. [46] Q. Yang, S. Mahns, S. Li, A. Gu, J. Wu, and H. Xu (2026) LLM-as-a-Prophet: understanding predictive intelligence with Prophet Arena. In ICLR, Cited by: §1, §1.1. [47] W. J. Youden (1950) Index for rating diagnostic tests. Cancer 3 (1), p. 32–35. Cited by: §J.1. [48] Z. Zeng, J. Liu, S. Chen, T. He, Y. Liao, Y. Tian, J. Wang, Z. Wang, Y. Yang, L. Yin, et al. (2026) FutureX: an advanced live benchmark for LLM agents in future prediction. In ICLR, Cited by: §1, §1.1. [49] T. Zhang, P. Qin, Y. Deng, C. Huang, W. Lei, J. Liu, D. Jin, H. Liang, and T. Chua (2024) CLAMBER: a benchmark of identifying and clarifying ambiguous information needs in large language models. In ACL, Cited by: §1.1. Appendix Appendix A Theory assumptions and proofs A.1 Key assumptions Our empirical framework identifies value of information (VOI) from observed market data under several structural assumptions. These assumptions link observed price movements to belief updates and are necessary for unbiased estimation. (A1) Market prices represent beliefs. We assume prediction market prices correspond to probabilistic beliefs: p≈P(U=1),p′≈P(U=1∣Q).p≈ P(U=1), p ≈ P(U=1 Q). Why needed. Our estimators interpret (p,p′)(p,p ) as prior and posterior beliefs. Without this mapping, ΔH H and KL no longer estimate VOI. Why acceptable. Prediction markets are widely studied as information aggregation mechanisms and are generally well-calibrated in expectation, especially in liquid markets [29]. (A2) Martingale property. We assume posterior beliefs satisfy Q[ℙ′]=p.E_Q[P ]=p. Why needed. This condition is required for unbiasedness: Q[ΔH(p,ℙ′)]=Q[KL(p,ℙ′)]=VOI(Q,U).E_Q[ H(p,P )]=E_Q[KL(p,P )]=VOI(Q,U). Why acceptable. Under Bayesian updating with correctly specified beliefs, the prior equals the expectation of the posterior. Empirically, prediction markets approximately satisfy this calibration property, as seen below (Figure 5). Figure 5: Empirical verification of the martingale property. Each blue point is the mean rolling-window posterior pap_a within a pbp_b decile (±1SE± 1\,SE); dashed grey: 45° null E[pa∣pb]=pbE[p_a p_b]=p_b; red: OLS fit. On n=2,088n=2,088 crux-resolution pairs across 8 generators (W=24hW=24h), β^=1.00 β=1.00, α^=+0.0002 α=+0.0002, R2=0.96R^2=0.96, and decile-binned E[pa∣pb]E[p_a p_b] lies within 1.0p1.0p of the diagonal in every bin: Polymarket prices satisfy the martingale property in the mean. (A3) Attribution assumption. We assume that the observed belief update at p→p′p→ p around a crux’s resolution time t reflects the informational impact of that crux. Why needed. VOI is defined as the effect of observing Q. Without attributing the price change to Q, we cannot interpret (p,p′)(p,p ) as a realization of P(U∣Q)P(U Q). Why acceptable. We focus on short windows around clearly defined events, so that the dominant information arrival is plausibly driven by the crux’s resolution. While not perfectly causal, this approximation is standard in event-study settings. Furthermore, we average prices over the 24 hours before and after the resolution time, rather than relying on identification of the exact moment at which the market incorporates the information. (A4) Representative branch realization. We observe only one realized posterior p′∈p1,p0p ∈\p_1,p_0\. This realization is assumed to be drawn according to the true probability q=P(Q=1)q=P(Q=1). Why needed. Unbiased estimation requires: Q[g(p,ℙ′)]=VOI(Q,U).E_Q[g(p,P )]=VOI(Q,U). Selection bias in which branch is observed would violate this equality. Why acceptable. Crux resolution is determined by real-world events rather than researcher choice, making realized branches plausibly representative. (A5) Population-level exchangeability. We assume the collection of (i,j)(i,j) pairs has sufficient independence such that 1||∑g(pij,pij′)→[VOI(Q,U)]. 1|D|Σ g(p_ij,p _ij)\;→\;E[VOI(Q,U)]. Why needed. Per-crux VOI is not identifiable; aggregation is required for consistency. Why acceptable. Our dataset spans many independent forecasting questions, and VOI is normalized via entropy, making cross-question comparisons meaningful. These assumptions allow us to interpret observed price changes as noisy realizations of belief updates induced by crux resolution, enabling estimation of population-level VOI. A.2 Practical constraints Our empirical setup relies on several assumptions that may not hold perfectly in practice. These issues mainly affect how cleanly we can identify and measure VOI, potentially introducing bias or additional variance into our estimators. Market inefficiencies. Prediction market prices may not perfectly reflect true beliefs due to liquidity constraints, strategic trading, or risk premium [45]. In these cases, p may deviate from the true probability P(U=1)P(U=1), weakening the interpretation of price updates as belief updates and potentially biasing VOI estimates. Confounding information arrivals. Observed price movements may reflect multiple pieces of information arriving simultaneously rather than only the resolution of the crux. As a result, the observed update (p,p′)(p,p ) may capture the combined effect of several events instead of the isolated contribution of Q. Timing misalignment. Information may be incorporated into market prices before or after our measurement window. If markets react early, the measured update is attenuated because part of the information has already been priced in. If markets react late, unrelated future information may contaminate (p,p′)(p,p ). Non-event resolutions and weak signals. Many cruxes (64.2%) resolve to “No” simply because the specified event never occurs within the assigned time window. In these cases, the resolution corresponds to the expiration of the window rather than a discrete observable event. Although the absence of an event can still be informative, these resolutions often induce only small belief updates, making (p,p′)(p,p ) more sensitive to background market noise and unrelated fluctuations. Resolution errors. Our automated resolver pipeline may introduce timing or classification errors when using external information sources. These errors effectively add measurement noise to (p,p′)(p,p ), increasing estimator variance and potentially introducing bias if the errors are systematic. We partially mitigate these threats through aggregation across a large set of cruxes, and we provide the following robustness checks. Threat 1: the prior price satisfies the martingale property [p′∣p]=pE[p p]=p across t-test, regression, and decile-bias checks (Figure 5). Threat 2 is addressed indirectly by the cross-model crux convergence in Section 4.6. Threat 3: model rankings are relatively stable when the 24-hour belief window is varied over W∈1,…,48W∈\1,…,48\ hours, except at smaller WWs (Appendix H). Threat 4: we impute KL=0KL=0 for unresolvable cruxes (Table 1) to avoid selection bias toward easily resolvable questions (Appendix L). Threat 5: hand-validation against the LLM resolver gives 92.6%92.6\% overall agreement, 96.9%96.9\% resolution-label accuracy, and 95.7%95.7\% timestamp agreement among correctly labeled cruxes (Section 3.2). Appendix B Extension of the metrics and evaluations to non-binary outcomes Suppose now that the ultimate question takes values in 1,…,K\1,…,K\ with K≥2K≥ 2, and that the crux takes values in a finite set Q (in the main text, =0,1Q=\0,1\). Let π=(π1,…,πK)π=( _1,…, _K) denote the prior distribution over outcomes just before Q resolves, where πk:=P(U=k) _k:=P(U=k) and ∑k=1Kπk=1 _k=1^K _k=1. After the crux resolves, beliefs update to the random posterior distribution Π′=(Π1′,…,ΠK′) =( _1,…, _K) with Πk′:=P(U=k∣Q) _k:=P(U=k Q), the analogue of ℙ′P in the main text. For each possible crux realization a∈a , define πk(a):=P(U=k∣Q=a)andqa:=P(Q=a),π^(a)_k:=P(U=k Q=a) q_a:=P(Q=a), so that Π′=π(a) =π^(a) with probability qaq_a. By the law of total probability, π=∑a∈qaπ(a),i.e.,Q[Πk′]=πkfor every k.π= _a q_a\,π^(a), .e., _Q[ _k]= _k every k. (7) The entropy of a categorical distribution π is H(π):=−∑k=1KπklogπkH(π):=- _k=1^K _k _k, with the convention 0log0=00 0=0; for K=2K=2 it reduces to the binary entropy H(π1)H( _1) of the main text. The entropy-based value of information is then VOI(Q,U):=H(π)−Q[H(Π′)]=H(π)−∑a∈qaH(π(a)).VOI(Q,U)\;:=\;H(π)-E_Q[H( )]\;=\;H(π)- _a q_aH(π^(a)). Equivalently, VOI(Q,U)=I(U,Q)VOI(Q,U)=I(U;Q), the mutual information between the ultimate question and the crux. Let (π,π′)(π,π ) be an observed prior–posterior pair, where π′π is the realized value of Π′ . The two per-sample functions of the main text extend as follows: the entropy reduction is ΔH(π,π′):=H(π)−H(π′) H(π,π ):=H(π)-H(π ), and the natural analogue of the binary KL function is the categorical KL divergence, with arguments in the (prior, posterior) order of the main text: KL(π,π′):=KL(π′∥π)=∑k=1Kπk′logπk′πk,KL(π,π )\;:=\;KL\! (π \, \|\,π )\;=\; _k=1^Kπ _k π _k _k, with the additional convention 0log00=00 00=0. Since πk=0 _k=0 implies Πk′=0 _k=0 by (7), outcomes with zero prior probability contribute nothing to H, ΔH H or KLKL, and KL(π,Π′)KL(π, ) is always finite. Unbiasedness. Both choices are unbiased for VOI. For the entropy reduction, this is immediate from the definition of VOI: Q[ΔH(π,Π′)]=H(π)−Q[H(Π′)]=VOI(Q,U).E_Q\! [ H(π, ) ]\;=\;H(π)-E_Q[H( )]\;=\;VOI(Q,U). (8) For the KL divergence, since KL(π,π′)=−H(π′)−∑kπk′logπkKL(π,π )=-H(π )- _kπ _k _k and H(π)=−∑kπklogπkH(π)=- _k _k _k, we have the pointwise identity KL(π,π′)=ΔH(π,π′)−∑k=1K(πk′−πk)logπk.KL(π,π )\;=\; H(π,π )\;-\; _k=1^K(π _k- _k) _k. (9) The last term has zero mean under Q, because Q[Πk′]=πkE_Q[ _k]= _k by (7). Hence Q[KL(π,Π′)]=Q[ΔH(π,Π′)]=VOI(Q,U).E_Q\! [KL(π, ) ]\;=\;E_Q\! [ H(π, ) ]\;=\;VOI(Q,U). Therefore, averaging over observed prior–posterior pairs (πij,πij′)( _ij,π _ij) yields the estimators of the population VOI VOI^ΔH VOI H :=1||∑(i,j)∈ΔH(πij,πij′), := 1|D| _(i,j) H( _ij,π _ij), VOI^KL VOI^KL :=1||∑(i,j)∈KL(πij,πij′)=1||∑(i,j)∈∑k=1Kπij,k′logπij,k′πij,k. := 1|D| _(i,j) KL( _ij,π _ij)\;=\; 1|D| _(i,j) _k=1^Kπ _ij,k π _ij,k _ij,k. Here we assume, without loss of generality, that all ultimate questions have the same number of outcomes K: if UiU_i has fewer than K outcomes, we pad it with artificial outcomes that receive probability zero under both the prior and the posterior, which by the conventions above changes neither H nor KLKL. For K=2K=2 and =0,1Q=\0,1\, identifying the Yes resolution with outcome k=1k=1 gives π=(p,1−p)π=(p,1-p), π′=(p′,1−p′)π =(p ,1-p ), π(a)=(p(a),1−p(a))π^(a)=(p^(a),1-p^(a)) and q1=q_1=q. This recovers the estimators (3) and (5) of the main text, together with their unbiasedness. The same argument extends to continuous outcomes by replacing sums over outcomes with integrals: VOI(Q,U)=I(U;Q)=Q[KL(Π′∥π)]VOI(Q,U)=I(U;Q)=E_Q[KL( \,\|\,π)], with the KL divergence taken between densities, so the KL estimator remains unbiased. The entropy form continues to hold with differential entropies whenever they are finite. Appendix C Estimator choice: details KL is nonnegative and does not penalize movement toward uncertainty. A basic property of VOI is nonnegativity: observing additional information should not reduce the expected value of the decision problem. The KL estimator preserves this property at the realized-sample level: KL(p,p′)≥0.KL(p,p )≥ 0. By contrast, realized entropy reduction can be negative: ΔH(p,p′)<0 H(p,p )<0 if H(p′)>H(p).H(p )>H(p). This happens when the posterior moves closer to the uncertainty 1/21/2, even though the update may still be informative. For example, suppose we are initially confident that candidate A will win an election (prior p=0.9p=0.9), but a scandal breaks and we revise our belief to p′=0.5p =0.5. Then ΔH=−0.531 H=-0.531 while KL=0.737KL=0.737. The update is clearly informative, yet ΔH H assigns it negative value. More generally, ΔH H only rewards movement toward certainty, regardless of direction, whereas KL captures information content whether the update increases or decreases entropy. The latter is a closer match to the intuitive notion of crux usefulness. Relationship and variance. The entropy-reduction and KL estimators are related by the exact identity ΔH(p,p′)=KL(p,p′)+logp1−p(p′−p). H(p,p )=KL(p,p )+ p1-p(p -p). (10) Thus, entropy reduction differs from KL by an additional linear term in the belief update (p′−p)(p -p). Since this residual term has mean zero across branches, both estimators remain unbiased for VOI. However, it contributes additional finite-sample variance. Writing δ=p′−pδ=p -p, a Taylor expansion gives KL(p,p+δ)=δ22ln(2)p(1−p)+O(δ3),KL(p,p+δ)= δ^22 (2)p(1-p)+O(δ^3), (11) so KL begins at second order in the update size, whereas ΔH H contains an extra first-order component proportional to logp1−pδ. p1-p\,δ. Consequently, Var[ΔH(p,p′)]−Var[KL(p,p′)]=[logp1−p]2σδ2+O([|δ|3]),Var[ H(p,p )]-Var[KL(p,p )]= [ p1-p ]^2 _δ^2+O\! (E[|δ|^3] ), (12) where σδ2=Var(δ) _δ^2=Var(δ). The leading term is nonnegative, implying that KL has lower variance to leading order whenever posterior updates are sufficiently small. Intuitively, KL acts as a variance-reduced version of entropy reduction by removing the directional first-order sensitivity of the update. Estimator choice and variance simulation To compare the statistical efficiency of VOI estimators, we conduct a Monte Carlo study in which we simulate belief updates under a partially informative signal. In each setting, the signal reveals the true outcome with probability α and otherwise reflects unrelated noise, allowing us to vary informativeness from uninformative to fully revealing. Across these regimes, we compute VOI using both KL-divergence and entropy-difference formulations and compare their sampling variability across replications. This design isolates how estimator variance depends on signal informativeness and the underlying distribution of beliefs (see Appendix I for details). Across 121 simulation cells spanning different belief distributions and levels of informativeness α, we find that the KL estimator has lower variance in the majority of regimes (78 out of 121 cells, 64.5%), with a geometric mean variance ratio Var(H)/Var(KL)=1.247Var(H)/Var(KL)=1.247, indicating that KL is more variance-efficient on average. This advantage is most pronounced in low- and moderate-informativeness regimes (small α), where posterior updates are limited and signals are weakly informative (as seen in Figure 10 of Appendix I). Hence, we report VOI^KL VOI^KL as our headline benchmark metric, as it is the lower-variance estimator in the majority of regimes and especially in the low-information settings most relevant to our application (see Appendix J for more details). For completeness, Appendix G also reports results for VOI^ΔH VOI H, along with two additional belief-update metrics: a variance-based VOI metric (p′−p)2(p -p)^2 and the absolute update |p′−p||p -p|. Empirically, the latter two produce rankings broadly consistent with the KL-based measure (as is expected by the decomposition in Appendix G). Appendix D Dataset construction D.1 Data filtering Here we describe the filter from raw Polymarket dump to the final benchmark set summarized in Section 4.1. Table 2 shows the counts after filtering for each stage. Resolved binary markets. We retain only resolved markets with binary “Yes” or “No” options. We deduplicate markets by question text and require them to overlap with the evaluation period (≥t0=≥ t_0= 2025-09-01). Topic filters. We first exclude markets tagged as sports, crypto, or weather. We then apply keyword filters to remove remaining sports-, crypto-, and awards-related markets. These domains are often highly volatile or driven by short-term information that may be difficult to decompose into informative cruxes. We also drop questions whose text contains tweet or say (e.g., Will Trump say “Cat” in May 2026?), which are usually not substantive events and contain high uncertainty. We further drop the “any of the listed” type of markets, because we find that the listed resolution criteria are not well defined. We also drop a small set of markets requiring access to some AI-leaderboards, whose information is hard for the resolver to gather. Lifetime and liquidity filters. We require that the market lasts endDate−startDate≥60 endDate- startDate≥ 60 days. For liquidity, we require at least 10,000 USDC in cumulative trading volume at fetch time and price histories spanning at least 30 distinct days. After randomly selecting one evaluation window per market, we further require more than 1,000 USDC in observed trading volume within that window. Event-level deduplication. For the same events, Polymarket often has several markets around it (at different resolution dates or different numerical thresholds). To ensure diversity in our dataset, we randomly select one single market per Polymarket event ID. We also remove near-duplicates by date- and number-stripped normalization of the question text. Active-window selection. We drop windows where the market price is within 0.10.1 of 00 or 11 through the window (i.e., the event has effectively already resolved), and residual windows shorter than 7 days. For the final evaluation, we sample one window per ultimate question, so J=1J=1. Table 2: Per-stage retention markets from the raw Polymarket dump to the final dataset (||=293|D|=293). Stage Markets Raw Polymarket dump (2026-03-25) 108,101 price-history end ≥ 2025-09-01 87,124 has resolution 86,819 dedup by question text 84,043 UMA status = resolved 84,035 has Yes outcome 30,524 excluded tags (sports/crypto/weather) 25,179 excluded words (tweet, say) 23,111 lifetime ≥ 60 days 3,009 sports/crypto/awards keyword filter 1,414 price-history length ≥ 30 1,063 event-ID deduplication 519 normalized-question deduplication 505 description-phrase blocklist 492 has ≥1≥ 1 valid 30-day window 342 active window ≥ 7 days 336 selected-window volume >> 1,000 USDC 293 D.2 Volume-aware price history construction Polymarket’s reported price history tracks the midpoint between the best bid and ask quotes in the order book. Consequently, the reported price can change when traders update or cancel their quotes, even if no trade happens. To mitigate the influence of price movements unsupported by trades, we reconstruct the price history by assigning more weight to prices with more nearby trading volume. Let ptrawp_t^raw denote the raw hourly Yes price, and let VtV_t be the observed USDC trading volume within a ±12± 12-hour neighborhood of hour t. We define αt=min(1,Vt1000),pt=αtptraw+(1−αt)pt−1, _t= (1, V_t1000 ), p_t= _tp_t^raw+(1- _t)p_t-1, with p0=p0rawp_0=p_0^raw. Therefore, when nearby trading volume reaches 1,000 USDC, we trust the raw price. When no nearby trading is observed, we carry forward the previous adjusted price. For volumes between 0 and 1,000 USDC, ptp_t is a weighted average of the raw price and the previous adjusted price, with more weight placed on the raw price as volume increases. All metrics are computed using the volume-aware price history ptp_t. Appendix E Examples of cruxes E.1 Example 1: Stolen Louvre jewels Ultimate question: “Will any of the stolen Louvre jewels be returned by December 31?” Evaluation window: 2025-10-21 to 2025-11-20 Table 3: Cruxes generated for “Will any of the stolen Louvre jewels be returned by December 31?” with their realized KL(p^,p^′)KL( p, p ). Model Proposed Crux KL Gemini-3.1-Pro-Preview Will authorities announce the arrest or formal charging of at least one suspect in connection with the October 2025 Louvre jewel heist between Oct. 21 and Nov. 20, 2025? 0.1348 Claude-Opus-4.6 Will French police publicly identify or arrest at least one suspect in the Louvre crown jewels heist by Nov. 15, 2025? 0.1348 GPT-5.4 Will French authorities publicly announce at least one arrest or formal charging of a suspect in connection with the October 19, 2025 Louvre crown-jewels theft by 2025-11-20? 0.1348 Llama-3.3-70B Will the French authorities announce the recovery of at least one stolen Louvre jewel or the arrest of at least one suspect in connection with the heist between 2025-10-21 and 2025-11-20? 0.1348 Qwen3-235B-A22B Will French law enforcement announce the arrest of at least one suspect in connection with the Louvre crown jewels heist between Oct. 21, 2025, and Nov. 20, 2025? 0.1348 DeepSeek-V3.1 Will at least one suspect involved in the Louvre crown jewels heist be formally arrested and charged by Nov. 15, 2025? 0.0022 Llama-3.1-8B-Instruct Will the French authorities have recovered at least one of the stolen Louvre jewels by Nov. 15, 2025? 0.0002 GPT-3.5-Turbo Will any of the stolen Louvre jewels appear on the public market by Nov. 20, 2025? (This includes online listings or auction announcements.) 0.0002 Figure 6: Price history of the “stolen Louvre jewels” ultimate. The shaded band is the 30-day evaluation window. Markers show each crux’s resolution time, and fanned horizontally where several share a timestamp. Figure 6 shows that the ultimate market price rose from p≈0.24p≈ 0.24 to p≈0.43p≈ 0.43 on 2025-10-25, when French authorities announced the arrest of two suspects in the theft. From Table 3, five models ask for an announced arrest or charging, capturing this update and getting a high KL(p^,p^′)=0.135KL( p, p )=0.135. The stricter formulation from DeepSeek-V3.1, however, requires a suspect to be arrested and charged. Since formal charges came four days later, by which time the market had already priced them in, this crux scores a near-zero KL. Meanwhile, Llama-3.1-8B-Instruct and GPT-3.5-Turbo ask about the jewels being recovered or appearing on the public market, neither of which happened during the window, so both resolve No at their deadlines with KL≈0≈0. E.2 Example 2: NewJeans’ return Ultimate question: “Will NewJeans perform again in 2025?” Evaluation window: 2025-10-31 to 2025-11-30 Table 4: Cruxes generated for “Will NewJeans perform again in 2025?” with their realized KL(p^,p^′)KL( p, p ). Model Proposed Crux KL GPT-5.4 Will ADOR or NewJeans publicly announce between Oct. 31 and Nov. 30, 2025 that they have begun, resumed, or scheduled formal talks specifically aimed at restarting NewJeans group activities? 0.5023 Qwen3-235B-A22B Will ADOR announce new group activities for NewJeans between Oct. 31 and Nov. 30, 2025? 0.5023 Llama-3.3-70B Will NewJeans or their members make a public statement about their plans for 2025 between Oct. 31 and Nov. 30, 2025? 0.5023 Llama-3.1-8B-Instruct Will NewJeans’ members contradict their recent court statement on their relationship with ADOR by engaging in any form of public activity as NewJeans, including interviews, social-media posts, or public appearances, between Oct. 31 and Nov. 20, 2025? 0.5023 Claude-Opus-4.6 Will NewJeans, or their representatives, file a formal appeal of the October 30, 2025 Seoul Central District Court ruling by Nov. 30, 2025? 0.0539 Gemini-3.1-Pro-Preview Will ADOR, HYBE, or NewJeans officially announce any new public group activities, such as a music release, fan event, or live performance, between Oct. 31 and Nov. 30, 2025? 0.0007 DeepSeek-V3.1 Will ADOR announce any NewJeans concert dates or tour plans between Oct. 31 and Nov. 30, 2025? 0.0007 GPT-3.5-Turbo Will NewJeans hold an official live performance as a group before Nov. 30, 2025? 0.0007 Figure 7: Price history of the “NewJeans’ return” ultimate. The shaded band is the 30-day evaluation window. Markers show each crux’s resolution time and are fanned horizontally where several share a timestamp. Before the evaluation window, NewJeans had been engaged in a year-long contract dispute with ADOR, during which the members declared their contracts terminated and later suspended activities after a court prohibited independent work. Whether the group would signal a return to activities under ADOR is therefore highly informative for whether they would perform again in 2025. As shown in Table 4, GPT-5.4, Qwen3-235B-A22B, Llama-3.3-70B, and Llama-3.1-8B-Instruct all focus on whether NewJeans would resume group activities or make public statements about their future plans. The market movement supports these models’ choice of crux: the November 12 return announcement coincides with a sharp increase in the probability that NewJeans would perform again, resulting in high KL scores (Figure 7). Claude-Opus-4.6 instead asks whether the members will appeal the earlier court ruling, which resolves No and has a smaller market update. By contrast, Gemini-3.1-Pro-Preview, DeepSeek-V3.1, and GPT-3.5-Turbo require more specific downstream events such as concert plans. None occurs by their November 30 deadlines, so their cruxes resolve without a meaningful market reaction and receive KL≈0≈0. Therefore, we see that high-KL cruxes must identify not only the right event, but also the appropriate level of specificity. Appendix F Model access and decoding parameters Table 5 lists each model’s provider and exact model identifier. Open-weight generators (Llama-3.3-70B, DeepSeek-V3.1, Qwen3-235B-A22B) are accessed through the Together AI inference API1010 10 https://docs.together.ai/docs/serverless-models#chat-models, and Llama-3.1-8B-Instruct runs locally using vLLM [23] (on a single NVIDIA A100 80GB GPU). Across all providers we use a max_tokens budget of 16,000. We pass each provider’s default sampling parameters. Web search is disabled for all generators and forecasters. The resolver (Claude-Sonnet-4.6) is run with the same 16k-token budget and is given the AskNews search tool with a cap of 10 calls per crux. Table 5: Models used in our experiments: eight generators produce candidate cruxes, one resolver assigns binary outcomes and initial resolution timestamps, a verifier checks and aligns those timestamps, and three forecasters are used in the downstream forecasting experiment in Section 4.5. Role Model Provider Identifier Generator Claude-Opus-4.6 [3] Anthropic claude-opus-4-6 Generator Gemini-3.1-Pro-Preview [17] Google gemini-3.1-pro-preview Generator GPT-5.4 [33] OpenAI gpt-5.4 Generator DeepSeek-V3.1 [11] Together deepseek-ai/DeepSeek-V3.1 Generator Qwen3-235B-A22B [35] Together Qwen/Qwen3-235B-A22B-Instruct-2507-tput Generator Llama-3.3-70B [26] Together meta-llama/Llama-3.3-70B-Instruct-Turbo Generator Llama-3.1-8B-Instruct [18] vLLM (local) meta-llama/Llama-3.1-8B-Instruct Generator GPT-3.5-Turbo [31] OpenAI gpt-3.5-turbo-1106 Resolver Claude-Sonnet-4.6 [4] Anthropic claude-sonnet-4-6 Verifier Claude-Sonnet-5 [5] Anthropic claude-sonnet-5 Forecaster Claude-Haiku-4.5 [2] Anthropic claude-haiku-4-5-20251001 Forecaster Gemini-3-Flash-Preview [16] Google gemini-3-flash-preview Forecaster GPT-5-Mini [32] OpenAI gpt-5-mini Appendix G Additional metric results Table 6 reports results under the primary KL estimator, the alternative entropy-reduction estimator, and two additional belief-update metrics. The first additional metric is squared probability change, Δp2=(p′−p)2 p^2=(p -p)^2, which remains a valid VOI estimator but under the variance-based uncertainty function f(p)=p(1−p)f(p)=p(1-p) rather than entropy [14]. The second additional metric is absolute probability change, |Δp|=|p′−p|| p|=|p -p|. This metric is not a VOI estimator, but it provides an intuitive diagnostic of the magnitude of the observed market update, regardless of direction. Empirically, the metrics Δp2 p^2 and |Δp|| p| broadly track the VOI^KL VOI^KL ranking. By contrast, VOI^ΔH VOI H produces a noticeably different ordering, reflecting the finite-sample behavior discussed in Section C. Table 6: Additional metric results by model (n=293n=293). VOI^KL VOI^KL and VOI^ΔH VOI H are measured in bits. Bold indicates the best-performing model. Significance is relative to the random baseline: ∗p<0.05^*\,p<0.05, p∗∗<0.01^**\,p<0.01. Model VOI^KL VOI^KL 95% CI Rank VOI^ΔH VOI H 95% CI Rank |Δp|| p| 95% CI Rank Δp2 p^2 95% CI Rank Claude-Opus-4.6 0.016 ∗ [0.009, 0.023] 1 0.017 [0.006, 0.029] 1 0.027 ∗ [0.021, 0.034] 1 0.004 ∗^* * [0.002, 0.006] 1 Gemini-3.1-Pro-Preview 0.014 ∗^* * [0.007, 0.020] 2 0.011 [-0.001, 0.023] 3 0.024 ∗^* * [0.019, 0.030] 2 0.003 ∗^* * [0.001, 0.005] 2 GPT-5.4 0.013 ∗^* * [0.007, 0.019] 3 0.009 [-0.002, 0.020] 6 0.023 [0.017, 0.028] 3 0.003 [0.001, 0.005] 3 DeepSeek-V3.1 0.012 [0.003, 0.020] 4 0.014 [0.004, 0.025] 2 0.020 [0.014, 0.026] 5 0.003 [0.000, 0.005] 4 Qwen3-235B-A22B 0.012 [0.005, 0.018] 5 0.007 [-0.003, 0.017] 8 0.021 [0.016, 0.027] 4 0.003 [0.001, 0.004] 5 Llama-3.3-70B 0.010 [0.005, 0.016] 6 0.009 [-0.001, 0.019] 7 0.019 [0.014, 0.024] 7 0.002 [0.001, 0.004] 6 Llama-3.1-8B-Instruct 0.010 [0.005, 0.015] 7 0.010 [0.000, 0.021] 5 0.019 [0.015, 0.024] 8 0.002 [0.001, 0.003] 8 GPT-3.5-Turbo 0.009 [0.005, 0.013] 8 0.011 [0.001, 0.020] 4 0.019 [0.015, 0.024] 6 0.002 [0.001, 0.003] 7 Random baseline 0.007 [0.006, 0.009] — 0.009 [0.007, 0.010] — 0.019 [0.017, 0.021] — 0.002 [0.001, 0.002] — Oracle 0.129 [0.100, 0.157] — 0.199 [0.181, 0.217] — 0.131 [0.118, 0.145] — 0.031 [0.024, 0.038] — For small belief updates, let δ=p′−pδ=p -p. A Taylor expansion gives KL(p,p+δ)=δ22ln(2)p(1−p)+O(δ3),KL(p,p+δ)= δ^22 (2)p(1-p)+O(δ^3), showing that KL is locally proportional to squared price movement. Since both |Δp|| p| and Δp2 p^2 are monotone in the update size |δ||δ|, they often induce similar empirical rankings despite lacking the information-theoretic grounding of KL. Appendix H Averaging-window robustness for belief estimation For each resolved crux QijQ_ij with resolution time tijt_ij, let Pi(t)P_i(t) denote the “Yes” price of the ultimate market UiU_i at time t. For an averaging window of W hours, we estimate the prior and posterior beliefs as: p^ij(W)=1W∑h=1WPi(tij−h),p^ij′(W)=1W∑h=1WPi(tij+h). p_ij^(W)= 1W _h=1^WP_i(t_ij-h), p_ij (W)= 1W _h=1^WP_i(t_ij+h). Our main results use W=24W=24 hours. Sweeping W∈[1,48]W∈[1,48]h, Figures 8 and 9 show that the correlation between Epoch AI capability and each of VOI^KL VOI^KL, |Δp|| p|, and Δp2 p^2 remains relatively stable, and generator rankings are preserved except at the smaller WWs, where short-term price noise may affect the estimation. Figure 8: Different metrics vs. averaging window W. Dashed black curve is the per-hour random baseline. Figure 9: Window-size robustness across W∈[1,48]W∈[1,48]h. (a) Pearson r between each metric’s per-generator mean and the Epoch AI capability score (⋆ means p<0.05p<0.05). (b) Spearman ρ between the generator ranking at the reference W=24W=24h and at every other W. Appendix I Monte Carlo simulation details I.1 Overview We conduct a Monte Carlo study to compare the sampling variance of two VOI estimators: a KL-divergence and an entropy-difference. The goal is to evaluate their relative statistical efficiency under controlled belief-update regimes, where the informativeness of signals can be systematically varied. Each simulation cell corresponds to a fixed choice of belief distributions and signal informativeness (Fp,Fq,αF_p,F_q,α), and within each cell we estimate the sampling variance of each estimator across repeated replications (R). I.2 Data generating process In each replication, we simulate a dataset of size n consisting of ultimate questions and crux resolutions. The data are generated as Ui∼Bern(p),Xi∼Bern(q),Zi∼Bern(α),U_i (p),X_i (q),Z_i (α), where UiU_i is the ultimate question, XiX_i is an independent noise draw, and ZiZ_i determines whether the crux resolution reveals truth or noise. The crux resolution is defined as Qi=Uiif Zi=1,Xiif Zi=0.Q_i= casesU_i&if Z_i=1,\\ X_i&if Z_i=0. cases (13) Thus, with probability α, the crux resolution reveals the true outcome, and with probability 1−α1-α, it reflects unrelated noise. The parameter α∈[0,1]α∈[0,1] therefore controls the informativeness of the crux, ranging from pure noise (α=0α=0) to perfect information (α=1α=1). For each replication, the parameters p and q are drawn from distributions FpF_p and FqF_q, respectively. I.3 Posterior computation We compute posterior beliefs P(U=1∣Q)P(U=1 Q) in closed form via Bayes’ rule. The likelihoods are: P(Q=1∣U=1) P(Q=1 U=1) =α+(1−α)q, =α+(1-α)q, P(Q=1∣U=0) P(Q=1 U=0) =(1−α)q, =(1-α)q, P(Q=0∣U=1) P(Q=0 U=1) =(1−α)(1−q), =(1-α)(1-q), P(Q=0∣U=0) P(Q=0 U=0) =α+(1−α)(1−q). =α+(1-α)(1-q). Applying Bayes’ rule yields: P(U=1∣Q=1)=p[α+(1−α)q]p[α+(1−α)q]+(1−p)(1−α)q,P(U=1 Q=1)= p[α+(1-α)q]p[α+(1-α)q]+(1-p)(1-α)q, P(U=1∣Q=0)=p(1−α)(1−q)p(1−α)(1−q)+(1−p)[α+(1−α)(1−q)].P(U=1 Q=0)= p(1-α)(1-q)p(1-α)(1-q)+(1-p)[α+(1-α)(1-q)]. I.4 VOI estimators For each observation i, we compute two measures of information gain: KL-based estimator IGiKL=DKL(Bern(P(U=1∣Qi))∥Bern(p))IG_i^KL=D_KL (Bern(P(U=1 Q_i))\,\|\,Bern(p) ) Entropy-difference estimator IGiΔH=H(p)−H(P(U=1∣Qi)).IG_i H=H(p)-H(P(U=1 Q_i)). We aggregate within each replication: VOI^rKL VOI^KL_r =1n∑i=1nIGiKL, = 1n _i=1^nIG_i^KL, VOI^rΔH VOI H_r =1n∑i=1nIGiΔH. = 1n _i=1^nIG_i H. I.5 Simulation design We evaluate estimator performance across a grid of simulation settings: • Belief distributions: 11 combinations of (Fp,Fq)(F_p,F_q), including uniform, symmetric Beta (e.g., Beta(2,2)Beta(2,2), Beta(5,5)Beta(5,5)), skewed Beta (e.g., Beta(2,5)Beta(2,5), Beta(5,2)Beta(5,2)), and Jeffreys prior Beta(0.5,0.5)Beta(0.5,0.5), • Informativeness levels: α∈0,0.1,…,1α∈\0,0.1,…,1\, • Replications per cell: R=1000R=1000, • Sample size: n=200n=200. This yields a total of 121 simulation cells and 121,000 Monte Carlo replications, as seen in Figure 10. Figure 10: Where KL has lower Monte Carlo sampling variance than ΔH H. Heatmap over the swept (Fp,Fq,α)(F_p,F_q,α) grid: red cells are (Fp,Fq,α)(F_p,F_q,α) at which Var(KL^)<Var(ΔH^)Var( KL)<Var( H) (KL optimal); blue otherwise. Cell winners come from an F-ratio test on the two replication-level variance estimates and are stable at our R, though variance ratios sit within a few percent of 1. The KL advantage concentrates in the low-α regimes (α∈[0.1,0.6]α∈[0.1,0.6]). The boundary cases at α∈0,1α∈\0,1\ are degenerate — both estimators reduce to a constant. I.6 Variance estimation and comparison Within each simulation cell, we compute the sample variance of each estimator across replications: Var^KL Var_KL =Var(VOI^rKL), =Var( VOI^KL_r), Var^ΔH Var_ H =Var(VOI^rΔH). =Var( VOI H_r). We define a win for the KL estimator as: Var^KL<Var^ΔH, Var_KL< Var_ H, and the overall win rate as: WinRateKL=#cells where KL wins#cells.WinRate_KL= \#\cells where KL wins\\#\cells\. We also report the distribution of the variance ratio: Var^ΔHVar^KL, Var_ H Var_KL, including its median and geometric mean, to capture the magnitude of differences beyond binary comparisons. I.7 Results Across 121 simulation cells (Table 7): • KL has lower variance in 78 cells (64.5%), • Entropy has lower variance in 43 cells (35.5%), • Median variance ratio: 1.0311.031, • Geometric mean variance ratio: 1.2471.247. The advantage of KL is most pronounced in low-informativeness regimes (small α), where posterior updates are small. Entropy-difference estimators can outperform KL in a minority of regimes, particularly under skewed parameterizations or larger updates. I.8 Sanity checks The simulation satisfies key consistency checks: • α=0α=0: Q is independent of U, implying VOI≈0VOI≈ 0, • α=1α=1: Q=UQ=U, yielding maximal information, • VOI increases monotonically in α. I.9 Caveats The win-rate metric is binary and does not reflect the magnitude of variance differences. Additionally, variance estimates are subject to Monte Carlo noise, so near-ties may vary across random seeds. To address this, we report summary statistics of the variance ratio. Finally, the boundary cases α=0α=0 and α=1α=1 are degenerate, with both estimators exhibiting near-zero variance. Table 7: Per-combination win rate for KL. Fraction of α-values (per row nα=11n_α=11) at which Var(KL^)<Var(ΔH^)Var( KL)<Var( H), with median variance ratio Var(ΔH^)/Var(KL^)Var( H)/Var( KL). Under a two-sided sign test of H0:P(KLwins)=0.5H_0:P(KL\ wins)=0.5, the strongest rows (81.8%81.8\%) yield p≈0.065p≈ 0.065. Variance ratios cluster within [0.98,1.14][0.98,1.14], so per-combination effects are small. Across all 11 combinations, 99 favor KL (sign test, p≈0.065p≈ 0.065), indicating a directional advantage. Combination (Fp/FqF_p/F_q) nαn_α KL win rate (%) Median Var(H)/Var(KL) beta(2,2) / beta(2,2) 11 81.8 1.06 beta(2,5) / beta(5,2) 11 81.8 1.09 beta(5,2) / beta(2,5) 11 81.8 1.14 beta(5,5) / beta(5,5) 11 72.7 1.13 jeffreys / uniform 11 72.7 1.02 beta(2,5) / uniform 11 63.6 1.02 uniform / uniform 11 63.6 1.03 beta(5,2) / beta(5,2) 11 54.5 1.02 uniform / jeffreys 11 54.5 1.01 jeffreys / jeffreys 11 45.5 0.997 beta(2,5) / beta(2,5) 11 36.4 0.980 Appendix J Empirical calibration of signal informativeness In addition to the Monte Carlo analysis, we estimate the level of signal informativeness α implied by observed data. This provides an empirical calibration of the regimes studied in the simulation and helps assess whether the parameter values considered are realistic. J.1 Identification of α In the mixture model (as above in equation (13)), the conditional distribution of the crux resolution Q given the ultimate question U satisfies: P(Q=1∣U=1) P(Q=1 U=1) =α+(1−α)q, =α+(1-α)q, P(Q=1∣U=0) P(Q=1 U=0) =(1−α)q. =(1-α)q. Therefore, α=P(Q=1∣U=1)−P(Q=1∣U=0)=TPR−FPR.α=P(Q=1 U=1)-P(Q=1 U=0)=TPR-FPR. Thus, α is identified as the difference between the true positive rate and false positive rate (Youden’s J statistic [47]). Importantly, the dependence on q cancels, so α can be estimated directly from empirical crux resolutions Q and realized outcomes U without specifying the noise distribution. J.2 Empirical estimation We estimate the informativeness parameter α using observed crux resolutions. In the data, each market provides: • an ultimate question U∈0,1U∈\0,1\ (market resolution), and • a corresponding crux resolution Q∈0,1Q∈\0,1\ (crux resolution). This directly matches the mixture model, where Q is a binary signal about U. We therefore estimate: TPR TPR =ℙ(Q=1∣U=1)=#Q=1,U=1#U=1, =P(Q=1 U=1)= \#\Q=1,\,U=1\\#\U=1\, FPR FPR =ℙ(Q=1∣U=0)=#Q=1,U=0#U=0, =P(Q=1 U=0)= \#\Q=1,\,U=0\\#\U=0\, α α =TPR^−FPR^. = TPR- FPR. This estimator corresponds to Youden’s J statistic and provides a direct measure of how informative the crux is about the ultimate question, and using our empirical data we get Table 8. J.3 Results Pooling the 2×2 contingency of (U,Q)(U,Q) across all eight forecasting models (N=2,152N=2,152 resolvable cruxes) yields TPR^=0.324,FPR^=0.295,α^=0.030. TPR=0.324, FPR=0.295, α=0.030. The standard error from a two-sample test of proportions, SE^(α^)=TPR^(1−TPR^)/n++FPR^(1−FPR^)/n−=0.022 SE( α)= TPR(1- TPR)/n_++ FPR(1- FPR)/n_-=0.022 (n+=604n_+=604, n−=1,548n_-=1,548), gives a 95% Wald CI of [−0.014,0.074][-0.014,0.074] and p≈0.18p≈ 0.18. The pooled estimate is positive but indistinguishable from zero at the 5% level. Per-model estimates (Table 8) span α^∈[−0.060,0.123] α∈[-0.060,0.123]: only the three strongest models (Gemini-3.1-Pro-Preview, GPT-5.4, Claude-Opus-4.6) yield α^>0.11 α>0.11, and even these do not individually clear zero. This empirical calibration places the realistic operating regime of crux-style decomposition firmly in the low-α band of the simulation in Appendix I, supporting the paper’s focus on low-information settings when comparing VOI estimators. Table 8: Empirical estimate of crux informativeness, by model. α^=TPR−FPR α=TPR-FPR (Youden’s J); n+,n−n_+,n_- are the per-row counts of U=1U=1 and U=0U=0. Model α α TPR FPR n+n_+ n−n_- Gemini-3.1-Pro-Preview +0.123+0.123 0.4050.405 0.2820.282 7979 195195 GPT-5.4 +0.118+0.118 0.4440.444 0.3270.327 7272 196196 Claude-Opus-4.6 +0.113+0.113 0.4360.436 0.3230.323 7878 201201 GPT-3.5-Turbo +0.042+0.042 0.2220.222 0.1800.180 8181 194194 DeepSeek-V3.1 −0.014-0.014 0.1810.181 0.1950.195 7272 190190 Llama-3.3-70B −0.034-0.034 0.3800.380 0.4140.414 7171 181181 Llama-3.1-8B-Instruct −0.050-0.050 0.2600.260 0.3090.309 7777 194194 Qwen3-235B-A22B −0.060-0.060 0.2700.270 0.3300.330 7474 197197 Pooled (micro) +0.030+0.030 0.3240.324 0.2950.295 604 1,5481,548 Appendix K Cosine similarity and operationalization differences Figure 11 compares crux pairs along two operational dimensions: numeric threshold and source kind (structured data providers versus news outlets). It reports the within-pair match rate for each dimension by question cosine similarity. Threshold agreement remains low: even among near-paraphrases (cos>0.8 >0.8), only 16% of pairs use the same numeric cutoff. Source-kind agreement increases with semantic similarity, from 51% to 68%, but remains far from complete. Thus, even semantically similar cruxes often differ in both their exact criterion and the type of evidence used to resolve it. K.1 Source Kind Taxonomy For the operationalization analysis in Figure 11, we extract named sources from each crux’s resolution criteria via case-insensitive substring match against a fixed keyword list. Each named source is then mapped to one of two broad categories: structured and news. The source kind_match axis asks whether two cruxes share the same category, while source asks semantic similarity. Structured (quantitative / official data providers). • Reviews and entertainment data: Metacritic, Rotten Tomatoes, IMDb, Box Office Mojo, The-Numbers, Billboard, Spotify, Steam (incl. SteamDB, SteamCharts), Nielsen. • Macroeconomic data: FRED (St. Louis Fed), Federal Reserve / FOMC, Bureau of Labor Statistics, Bureau of Economic Analysis, U.S. Treasury. • Markets and exchanges: CME Group (incl. FedWatch), NYSE, NASDAQ, SEC (incl. EDGAR). • Crypto and prediction markets: CoinGecko, CoinMarketCap, Polymarket, Kalshi, PredictIt. • Government and public health: NOAA (incl. weather.gov, NCEI), FBI, CDC, WHO. • Reference: Wikipedia. News (journalistic outlets). New York Times, Wall Street Journal, Reuters, Bloomberg, Financial Times, Associated Press, CNN, BBC, The Guardian, Washington Post, CNBC, Forbes, Fortune, Axios, Politico, Business Insider, Yahoo Finance. A crux mentioning at least one structured source is labeled structured; one mentioning only news outlets is labeled news; and one mentioning neither is labeled none. When a crux mentions both, it is labeled structured, since structured providers are the more committal anchor for resolution. Figure 11: Operationalization-axis match rate by cosine similarity. For each question pair we record whether the two cruxes share the same numeric threshold (when both emit one) and the same source kind (Appendix K.1). Error bars are Wilson 95% CIs on each binomial proportion. Even at cos>0.8 >0.8, threshold agreement is only 0.160.16 and source-kind agreement only 0.680.68. Appendix L Unresolvable questions and their classification We define resolvability as whether the resolver can evaluate a crux as true or false at a specific time based on available evidence. We sample 50 unresolvable questions,1111 11 The 50 sampled questions are drawn to roughly match each model’s share of unresolvable questions in the full dataset. and identify three main failure modes: 1. Data availability failures. (OPEN50.0%)50.0\%)1212 12 Percentage of the 50 sampled questions in this category. Table 9 provides the per-model breakdown. The crux relies on data that is unstable, inaccessible, or unavailable at the required level of granularity, such as niche dashboards that are not retrievable, data not published at the required level of breakdown, or source-platform ambiguity. Example: “Will Phantom’s revenue exceed $200k on …” There are live dashboards like Flowscan, but news sources do not archive the fine-grained daily values required to resolve the question later. 2. Temporal misalignment. (18.0%18.0\%) The crux is tied to an event or data release that does not occur within the specified resolution window, either because the data release/a fixed event is scheduled, or delayed to after the window closes. Example: Delayed government reports or sports matches. Underlying event occurs within window but official data released after. 3. Observability failures. (32.0%32.0\%) The crux depends on events that are not reliably observable, typically because they involve unverifiable negatives, exhaustive enumeration, or events whose non-occurrence is not publicly recorded. Example: “Will at least two women appear on TIME covers between …” Yes resolves from observing two covers, while No requires exhaustively checking every cover in the window. We note that the majority (50%) of failures stem from data availability, which is expected as the resolver pipeline relies on news: if aggregate information is unavailable or the metric is niche, the resolver would fail. Table 9: Unresolvable cruxes by failure mode and model (n=50n=50). Counts of cruxes each model marked unresolvable, classified into three failure modes: data availability, temporal, and observability. Data availability dominates overall (25/50=50%25/50=50\%), but the distribution is model-dependent: Llama-3.3-70B and Qwen3-235B-A22B most often fail due to data availability, while Llama-3.1-8B-Instruct’s failures are predominantly temporal. Gemini-3.1-Pro-Preview and Claude-Opus-4.6 show a larger share of observability failures. Model Data avail. Temporal Observability Total Claude-Opus-4.6 1 0 2 3 Gemini-3.1-Pro-Preview 2 0 3 5 GPT-5.4 3 2 1 6 DeepSeek-V3.1 4 1 3 8 Qwen3-235B-A22B 5 0 1 6 Llama-3.3-70B 8 0 3 11 Llama-3.1-8B-Instruct 0 4 2 6 GPT-3.5-Turbo 2 2 1 5 All 25 9 16 50 Appendix M Prompts M.1 Crux generation prompt ⬇ You are a helpful assistant to forecasting research scientists. The goal is to generate subquestions / cruxes for a given ultimate question. TASK BACKGROUND: In forecasting, we often care about answering questions that are either too complex, too broad or too far in the future to answer directly. If the outcome of a question will only be known in 20 years, we would like to have indicators for which track the world is on, with respect to the outcome of the question. For example, if the ultimate question regards nuclear safety in 2050, we would like to have indicators in the near term which could be used as indicators for the ultimate question - such as the number of nuclear power plants in the world, the number of nuclear weapons in the world, the number of nuclear accidents in the world, etc. These indicators are called subquestions / cruxes. They are indicators that can be used to track the world’s progress towards the ultimate question. We want the subquestions to be grounded in current world events that pertain to the ultimate question. Ideal subquestions are such that they are cruxes for ultimate question disagreements; i.e if 2 forecasters disagree on the outcome of the ultimate question, they are more likely to agree on the outcome of the final question once the subquestion is resolved. TASK: - Generate subquestions that are meaningful early indicators of the ultimate question’s outcome – not restatements of the ultimate question with an earlier end date. GUIDELINES: Questions: - The question should be binary. This means that the outcome of the question should be either yes or no. The question should be formulated in such a way that forecasters answer with a probability. Hence, a question such as "will X increase or decrease" is not a good question. A better question is "Will X increase by at least 10 units before [specific date]?". A higher probability answer implies more likely ’yes’, a lower probability implies more likely ’no’. - Informative: Is this crux / subquestion likely to provide evidence for the ultimate outcome question that the user cares about? The question should have a high value of information - meaning that knowing the outcome of this question would update forecasters on their forecast of the ultimate outcome. - Good questions usually do not have an expected forecast very near 0 or 1, because then the forecasts will not be informative. This is a general guideline, not a rule: there may be exceptions to this. - Consider potential loopholes in the resolution that make the forecasts uninformative. I.e. if we are curious about the difference between 2 competing LLMs on a benchmark, and the benchmark becomes saturated, the question will be resolvable, but not informative (both competitors will have the same score of 100%). - The question should be a clear and concise question that is easy to understand. - The question should be grounded in current world events. For whatever question you want to ask, ensure you know the current state of that area to choose appropriate thresholds for the question. - All terms in the question and resolution criteria must be clearly defined, eliminating ambiguity. - Time-bounded: The subquestion end_date MUST be before the end date of the ultimate question. The subquestion end_date must also be AFTER the start date of the ultimate question (i.e. today’s date). Both the ultimate question end date and start date will be provided to you. Background information: - The background information should provide enough context to fully understand the question and why it is relevant. - The background information should include the current state of the area, including relevant thresholds for the question. For example if you are forecasting the number of nuclear weapons in the world, you should include the current number of nuclear weapons in the world. If you are forecasting the number of AI jobs, you should include the current number of AI jobs etc. These should influence the thresholds for the question. - The background information must provide citations. - Citations MUST include actual, clickable URLs (http/https). Do NOT output placeholder citation tokens like "citeturn5search5…". Inline the source URL next to claims or include them in a short Sources list. - At the end of Background information, add a short "Sources:" list with each item on its own line containing the source title (or site) and full URL. Resolution criteria: - Aim for tight resolution criteria. The resolution criteria should leave little room for discretion in deciding the resolution. As best you can, try to limit the scope for ex-post quarrels about what really happened, and who was right. - Define your terms. Questions of the sort "will X occur?" often hinge on how X is defined. It is therefore important to spell out your definitions with extra care. Don’t worry about being too pedantic here! - Be concrete. Try to specify precisely and in detail which steps should or shouldn’t be followed when resolving the question. Examples are helpful for making these instructions concrete. - Use authoritative sources, when possible. Good options are numerical data regularly published by a reliable publicly available source. Note that you should be sure that the sources will be available at the time of resolution, or otherwise you might want to specify alternative sources of information. - Consider and account for edge-cases. Try to imagine scenarios for which the resolution conditions fail to cleanly apply, or cases that are just on the edge of counting towards resolution. If such scenarios or edge-cases are plausible, you should clarify how the question should resolve when such events rear their head. - Consider fall-back criteria. When you have a resolution that should be easy to check assuming all goes well, try to handle also the case where all doesn’t go well. What if the data source you specified stops being published? Is there anything else odd that might happen to make the outcome unclear? - Consider delayed events: if the event is postponed beyond the resolution window, the question should be considered unresolvable – specify this explicitly in the resolution criteria. - Try to account for unknown unknowns. Think about how the resolution criteria behave when something you don’t expect happens anyway. - Well-researched: The question should be well-researched and grounded in current world events. It must provide high quality sources for the resolution criteria. The background information must provide enough context to fully understand the question and why it is relevant. Both the resolution criteria and the background information must provide citations. - For the Resolution criteria, include the specific data source(s) you propose to use and provide the full URLs. Avoid any placeholder tokens; always include explicit links. - Time-bounded: The question should specify when it will be resolved. MUST explicitly state the start date and end date of the subquestion, clearly defining the resolution window. For example: "This question resolves Yes if [condition] occurs between [start_date] and [end_date] (inclusive)." This ensures resolvers know the exact time window during which qualifying events must occur. ⬇ Ultimate question: ultimate_question Description: description Today’s date: start_date Ultimate question end date: end_date Generate n subquestion(s) whose start_date is today (start_date) and end_date is no later than period_end. IMPORTANT: - Every subquestion end_date MUST be between start_date and period_end. - The resolution_criteria for each subquestion MUST explicitly mention the subquestion’s start_date (start_date) and end_date, clearly stating the window during which qualifying events count toward resolution. For example: "This question resolves Yes if [condition] occurs between start_date and [end_date]." - DO NOT generate a subquestion that is just the ultimate question with an earlier deadline or tighter resolution criteria. A good subquestion asks about a different observable event that could be an indicator of the ultimate outcome. articles_block RETURN FORMAT: Return a JSON array of dictionaries. Each dictionary MUST contain: - question: the subquestion / crux - background_information: background information on the question. Include citations with explicit, clickable URLs and add a short Sources list at the end. - resolution_criteria: the resolution criteria. Must explicitly state the start_date and end_date defining the resolution window. You must provide links (full URLs) to the sources which you propose to use for the resolution criteria. Do not include placeholder citation tokens. - start_date: "start_date" (today’s date, MUST be exactly this value) - end_date: the date when the question will be resolved (in UTC). MUST be between start_date and period_end. - rationale: a rationale for why this question is relevant to the ultimate question Return ONLY the JSON array, no other text. M.2 Crux resolution prompts ⬇ You are a research assistant that determines whether forecasting questions have resolved Yes or No based on news articles. Use the asknews_search tool to find relevant news articles and determine the resolution. ⬇ Determine whether the following binary forecasting question has resolved Yes or No. Question: question Resolution criteria: resolution_criteria Start date: start_date End date: end_date You are a careful, high-precision resolution assistant. Your goal is to determine whether the question resolved Yes or No using explicit, verifiable evidence. IMPORTANT – resolution rules: - Only events that occurred BETWEEN start_dateT00:00:00Z and end_dateT23:59:59Z (inclusive) count. - Prefer precision over recall. If uncertain, return resolvable=false. - Do NOT guess, infer, or assume. Only use explicit evidence. resolvable=false if ANY of the following: - The underlying event or data is missing, unavailable, or not reported - The resolution criteria are ambiguous, contradictory, or unclear - The evidence is indirect, inferred, or does not exactly match the criteria - A reliable first-occurrence timestamp cannot be established - Conflicting evidence cannot be resolved - Any reason prevents a definitive Yes/No answer resolution=Yes: - The resolving event explicitly occurred within the window - If multiple qualifying events exist, use the FIRST occurrence - You MUST verify that no earlier qualifying occurrence exists in the window resolution=No: - If the condition became definitively impossible before end_date -> resolution_datetime = the exact datetime it became impossible. Always use the earliest date at which a definitive No conclusion is supported by explicit evidence. "Definitively impossible" means there is explicit, public evidence of a discrete event that forecloses the Yes outcome (e.g., a deadline passed, a vote failed, a candidate withdrew) – not merely that the probability appeared low. - If the window closed with no earlier definitive No moment -> resolution_datetime = end_date EVIDENCE REQUIREMENTS (ALL must be satisfied for resolvable=true): 1. Evidence directly matches the resolution criteria (no interpretation required) 2. Event is explicitly stated (not inferred) 3. Event is clearly within the window 4. A reliable timestamp exists (not approximate) 5. Source is high-quality OR multiple sources agree If any condition fails -> resolvable=false STRICT RULES FOR resolution_datetime: - Must be an exact ISO 8601 UTC datetime - No vague language ("around", "approximately", etc.) - If timestamp is not explicitly available or cannot be reliably determined -> resolvable=false Follow this procedure: STEP 1 – DECOMPOSE - Identify the exact condition required for Yes - Identify what observable evidence would confirm it - Identify key entities, thresholds, and timing constraints STEP 2 – TARGETED SEARCH - Generate multiple precise search queries including: * entity + metric + date * variations to find earliest occurrence * queries to disprove the event STEP 3 – EVENT DETECTION - Identify candidate events - Reject any that: * require inference * are indirect or approximate * do not clearly satisfy criteria STEP 4 – CRITERIA MATCHING - Confirm the event EXACTLY satisfies the resolution criteria - Reject partial or approximate matches STEP 5 – TIMESTAMP EXTRACTION - Extract earliest exact timestamp of occurrence - Reject if timestamp is vague or inferred STEP 6 – FIRST OCCURRENCE CHECK - Search specifically for earlier qualifying events - If you cannot confirm this is the first -> resolvable=false STEP 7 – VERIFICATION (CRITICAL) - Attempt to disprove your conclusion: * look for contradictory evidence * check similar events outside window * confirm correct interpretation - If contradiction exists -> resolvable=false STEP 8 – FINAL DECISION - Yes: event occurred within window (first occurrence) - No: condition never met within window - Otherwise: resolvable=false Double check before answering: 1. Evidence is direct, explicit, and within the window 2. No inference or assumptions were used 3. Timestamp is exact and reliable 4. FIRST occurrence condition is satisfied 5. No contradictory evidence exists 6. If any doubt remains -> resolvable=false Return STRICT JSON: "evidence": "step1_decompose": "<exact condition for Yes, observable evidence required, key entities/thresholds/timing>", "step2_search_queries": "<search queries used and what each was intended to find>", "step3_event_detection": "<candidate events identified and any rejected with reasons>", "step4_criteria_matching": "<confirmation or rejection of exact criteria match – include inline citations [source](url)>", "step5_timestamp_extraction": "<exact timestamp found, or why it could not be extracted>", "step6_first_occurrence_check": <whether an earlier qualifying occurrence was found>, "step7_verification": "<contradictory evidence checked, alternative interpretations considered>", "step8_final_decision": "<final reasoning tying all steps to the resolution>" , "resolvable": true or false, "resolution": "Yes" or "No" or "", "resolution_datetime": "<exact UTC datetime in ISO 8601, or empty if resolvable=false>", "source_urls": ["url1", "url2", …] ⬇ You are a resolution-time adjudicator for a forecasting benchmark. Several language models each proposed one crux (subquestion) for the same ultimate market question, and a resolver assigned each crux a Yes/No resolution and a timestamp. Resolver timestamps are noisy: the same real-world event sometimes received different timestamps for different models’ cruxes, and some ’No’ resolutions were stamped at an evidence time even though the answer was not yet determined. Your job is to adjudicate TIMESTAMPS ONLY. Never change a Yes/No label. RULES 1. Event grouping: identify which cruxes are resolved by the same underlying real-world event (across models). All cruxes in one event group must receive the IDENTICAL timestamp: the earliest time the resolving evidence plausibly became public, based on the evidence provided (event dates, article dates, official announcement times). Cruxes resolved by genuinely different events (e.g., an arrest vs. a later formal charging, when the crux wording requires the latter) belong to different groups even if close in time. 2. Yes resolutions: timestamp = the canonical public time of the resolving event (per its event group). 3. No resolutions: timestamp = the crux’s OWN deadline, at end_date T23:59:59Z (rule "no_deadline") – UNLESS an event strictly before the deadline made the No logically or institutionally irreversible, e.g. the subject was eliminated from the process, an authority issued a binding and final negative decision, or the only possible path to Yes was closed (rule "no_irreversible", timestamp = that event’s canonical time). Announcements of delay, postponement, low likelihood, or intent are NOT irreversible: the answer remains open until the deadline. Worked example: a crux asks "will NASA complete a rehearsal before Feb 16?"; on Feb 3 the first attempt is scrubbed and NASA announces a second rehearsal is needed and the launch is moving to March. This is rule "no_deadline" (stamp Feb 16): schedules can change again, so completion before Feb 16 remained possible – a delay announcement is evidence of No, not determination of No. Contrast: if that crux’s deadline were Feb 5 and completing another attempt within 2 days were physically/procedurally impossible, rule "no_irreversible" (stamp the scrub announcement time) would apply. Apply "no_irreversible" only when you can state WHAT made Yes impossible before the deadline, not merely unlikely. Two hard requirements for "no_irreversible": (a) NO HINDSIGHT: you must justify impossibility using only information available AT the candidate event’s time. That the Yes-event in fact never happened before the deadline (known from later evidence) is NOT an argument – every No crux trivially satisfies that. If your justification cites what happened afterwards, it is invalid; use "no_deadline". (b) BINDING CONSTRAINT: the impossibility must follow from a binding physical, legal, or procedural constraint with fixed dates (e.g., candidate eliminated from a completed vote; the only scheduled session already past; a law’s effective date). Repair timelines, agency scheduling projections, "requires X first" chains, and stated targets are expectations, not constraints – they can be revised, so they are NOT irreversible. 4. If the resolver’s existing timestamp already satisfies these rules, keep it. 5. Use UTC ISO format like 2026-02-03T17:00:00Z. Timestamps must lie within the crux’s [start_date, end_date 23:59:59] window. 6. TIMEZONE DISCIPLINE AND TIME PRECISION: source times are often stated in local timezones (ET, PT, CET, …) or as bare dates. Assign the event time by this ladder, and report which level you used in "time_precision": - "stated": an article or official source explicitly states the event time. Rationale must cite WHICH source (title or URL), QUOTE the stated time with its timezone, and SHOW the conversion to UTC (e.g., "NASA blog: ’scrubbed at 12:00 p.m. EST Feb 3’ -> 17:00 UTC"). - "convention": no explicit time, but the event has a documented standard release time (e.g., "BLS Employment Situation releases at 8:30 a.m. ET -> 12:30 UTC"). Name the convention in the rationale. - "date_only": only a date is determinable. Do NOT invent an hour: use T00:00:00Z of the earliest date the evidence was public, and say so in the rationale. Never present a guessed hour as stated. Return JSON only – an array with EXACTLY one entry per crux listed in the input: [ "model": "<model name as given>", "event_group": "<short label, e.g. ’wdr-scrub’ or ’deadline’>", "rule": "yes_event" | "no_deadline" | "no_irreversible", "adjudicated_datetime": "Y-M-DDTHH:M:SSZ", "time_precision": "stated" | "convention" | "date_only", "changed": true|false, "rationale": "1-3 sentences; for event-timestamped rules, cite the source and show the timezone conversion to UTC" ]