Paper deep dive
From Checker to Forecaster: Code-Owned Evaluation of Model-Generated Strategic Routes Under Delayed Ground Truth
Aleh Manchuliantsau
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/18/2026, 1:45:34 PM
Summary
The paper introduces RouteCast, a protocol for code-owned provisional forecasting of model-generated strategic routes when ground truth is delayed, censored, or private. In a retrospective pilot on 21 venture cases, RouteCast achieved an AUC of 0.756, outperforming a blind LLM judge (0.678) and matching an identity-exposed LLM judge (0.761). The study highlights that typed route decomposition did not significantly improve discrimination over the whole-packet score.
Entities (7)
Relation Signals (5)
RouteCast → achieves → AUC 0.756
confidence 95% · the whole-packet RouteCast score showed preliminary retrospective discrimination (AUC 0.756
RouteCast → isevaluatedon → YC 2012–2014
confidence 95% · The retrospective pilot uses YC 2012–2014 companies
RouteCast → handles → Delayed Ground Truth
confidence 90% · RouteCast instantiates this regime for model-generated typed strategic routes... ground truth is delayed, censored, or private
RouteCast → outperforms → Blind LLM Judge
confidence 90% · RouteCast score showed preliminary retrospective discrimination (AUC 0.756... while a blind LLM judge reached AUC 0.678
Identity-exposed LLM Judge → hasleakagerisk → RouteCast
confidence 80% · consistent with recognition- or outcome-related leakage risk
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast. RouteCast instantiates this regime for model-generated typed strategic routes: models propose candidate routes and structured factors; point-in-time evidence, reference classes, and deterministic transformations produce a provisional forecast-ranking; later outcomes evaluate the forecast. In a retrospective venture pilot on 21 binary-outcome cases (6 positive, 15 negative), the whole-packet RouteCast score showed preliminary retrospective discrimination (AUC 0.756, 95% CI [0.471,0.980]), while a blind LLM judge reached AUC 0.678 [0.419,0.897] and an identity-exposed LLM judge reached AUC 0.761 [0.515,0.944], consistent with recognition- or outcome-related leakage risk. A preregistered decomposition ablation on the same binary subset found that converting the identical inputs into typed staged routes was indistinguishable from the whole-packet score (Delta AUC = -0.144, 95% CI [-0.471,0.176]) and from a deterministic heuristic (Delta AUC = -0.089, 95% CI [-0.412,0.278]). The pilot establishes an auditable feasibility result and exposes failure modes; it does not establish prospective calibration, causal decision improvement, route-decomposition advantage, or cross-domain validity.
Tags
Links
- Source: https://arxiv.org/abs/2607.10972v1
- Canonical: https://arxiv.org/abs/2607.10972v1
Trouble viewing inline? Open PDF directly →
Full Text
43,012 characters extracted from source content.
Expand or collapse full text
From Checker to Forecaster: Code-Owned Evaluation of Model-Generated Strategic Routes Under Delayed Ground Truth Aleh Manchuliantsau Independent Researcher aleh.manchuliantsau@gmail.com (Version 1.0: July 2026) Abstract Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast. RouteCast instantiates this regime for model-generated typed strategic routes: models propose candidate routes and structured factors; point-in-time evidence, reference classes, and deterministic transformations produce a provisional forecast-ranking; later outcomes evaluate the forecast. In a retrospective venture pilot on 21 binary-outcome cases (6 positive, 15 negative), the whole-packet RouteCast score showed preliminary retrospective discrimination (AUC 0.756, 95% CI [0.471,0.980][0.471,0.980]), while a blind LLM judge reached AUC 0.678 [0.419,0.897][0.419,0.897] and an identity-exposed LLM judge reached AUC 0.761 [0.515,0.944][0.515,0.944], consistent with recognition- or outcome-related leakage risk. A preregistered decomposition ablation on the same binary subset found that converting the identical inputs into typed staged routes was indistinguishable from the whole-packet score (ΔAUC=−0.144 =-0.144, 95% CI [−0.471,0.176][-0.471,0.176]) and from a deterministic heuristic (ΔAUC=−0.089 =-0.089, 95% CI [−0.412,0.278][-0.412,0.278]). The pilot establishes an auditable feasibility result and exposes failure modes; it does not establish prospective calibration, causal decision improvement, route-decomposition advantage, or cross-domain validity. 1 Introduction Models increasingly generate strategic options rather than only factual answers. A founder asks which venture route to pursue, a lab asks which research direction might be worth a quarter, and a product team asks which roadmap bet should be made next. These tasks differ from factual question answering because the relevant outcome arrives later, is often censored or private, and may be observed only through a proxy. The evaluator must act before the world can grade the recommendation. Deterministic evaluation of model outputs has so far required either a contract checkable at evaluation time or feedback arriving within the loop. We study the complementary regime—delayed, censored, or private ground truth—in which deterministic code must issue a code-owned provisional forecast whose correctness is unknowable at scoring time. We instantiate this regime for a new evaluation object, competing model-generated typed strategic routes, and audit the protocol’s integrity properties on a retrospective pilot, including an identity-leakage contrast and a preregistered decomposition ablation reported as indistinguishable. The conceptual move is from checker to forecaster. Deterministic code is already used as a checker or control authority in software evaluation, runtime assurance, and evidence-gated control. In the regime studied here, however, correctness is not available at scoring time. Code can own the forecast only by issuing an auditable forecast-like ranking, preserving the information set available at decision time, and later exposing that ranking to outcome resolution. The retrospective pilot therefore asks whether the implementation survives obvious integrity failures. It is not a validation of prospective forecasting authority. Table 1: Feature comparison with representative close predecessors. System Outcome resolution Inputs to final rule Final rule Evaluation object GroundEval / CARE checkable now or in-loop observed traces or evidence gates deterministic check/gate agent behavior or policy DeLLMa delayed LLM verbal forecasts and LLM preference rankings analytic expected-utility maximization single action under uncertain states DIALECTIC delayed model scores or judgments model-owned or model-aggregated company, investment, or rationale RouteCast delayed, censored, or private frozen priors and admissibility-checked evidence, with model-mediated typing and extraction versioned deterministic forecast-ranking competing typed strategic routes The initial implementation domain is venture route selection. RouteCast ranks Wedge–Bridge–Vision trajectories from point-in-time packets. A route packet can contain a current wedge, typed transition claims, evidence, cost-to-learn, a staged-value calculation, and a binding transition to test. RouteCast is defined for competing typed strategic routes. The retrospective pilot evaluates one cohort through several analyses: a frozen whole-packet RouteCast score, blinded and identity-exposed LLM baselines, a deterministic heuristic, a bounded stability probe, and a preregistered typed-route decomposition ablation. The decomposition analysis is therefore a variant within the same pilot, not evidence from a separate pilot. The contributions are: 1. Regime formulation. We formalize code-owned provisional evaluation when correctness cannot be checked at scoring time. 2. Protocol and object. We instantiate the regime for competing typed strategic routes using point-in-time evidence, reference classes, deterministic ranking, and a binding-transition output. 3. Retrospective pilot. We report an identity-exposure contrast, preliminary whole-packet discrimination, and a preregistered decomposition ablation reported as indistinguishable. The scope is deliberately bounded. We do not claim that RouteCast has an established advantage over LLM judges, that its probabilities are calibrated, that typed decomposition improved ranking in the pilot, or that the protocol transfers to OSS, R&D, or product roadmaps. Those are prospective evaluation targets. 2 Problem Setting and Definitions Let t0t_0 denote the decision time. The available information set ℐt0I_t_0 contains only evidence admissible at that time: captured source text, timestamps, provenance, reference classes, and user-provided context. The outcome Y is unavailable, censored, private, or unresolved at t0t_0. At a later resolution time t1t_1, a frozen predicate or outcome mapping may reveal Y, a proxy for Y, or an unresolved status. WedgeBridgeVision(V)p1,U1p_1,U_1c1Ec_1^Ep2,U2p_2,U_2c2Ec_2^E next test: learning cost ℓb _b Figure 1: Typed strategic route. Each transition eie_i carries a provisional success probability pip_i, epistemic uncertainty UiU_i, and an execution cost ciEc_i^E. A selected binding transition can be tested at learning cost ℓi _i before full execution. The terminal node carries value V. A strategic route is a candidate path from a current state to a target state. A typed transition is one edge in that path with a source state, target state, actor, mechanism, required asset, evidence, threshold, kill condition, probability, cost, and, when applicable, resolution window. A model-owned proposal is any route, claim, decomposition, prior suggestion, or rationale authored by a model. A code-owned provisional forecast is a ranking or derived quantity produced by versioned code and data from ℐt0I_t_0, without permitting model prose to overwrite the final transformation. Three evaluation concepts must remain separate. Ranking performance asks whether later positives rank above later negatives, for example by AUC or top-bucket lift. Probability calibration asks whether forecast probabilities match observed frequencies, for example by Brier score, calibration curves, or expected calibration error with small-sample caveats [3, 11, 12]. Decision utility asks whether acting on the ranking improves outcomes or reduces cost relative to a counterfactual policy. The retrospective pilot mainly provides evidence about integrity, retrospective discrimination, and failure modes. It does not measure prospective calibration or causal utility. 3 Related Work 3.1 Proposal–Authority Separation and Runtime Assurance RouteCast does not introduce proposal–authority separation. The Simplex lineage and runtime-assurance literature separate a high-performance or untrusted controller from a trusted safety mechanism or fallback authority [19, 20, 10]. These works establish proposal–authority separation in settings where safety or correctness can be checked now, or where a trusted fallback/control mechanism operates in-loop. Similar generator–verifier and diagnosis–control patterns appear in contemporary agent systems. For example, auditable policy-adaptation work separates an LLM diagnostic layer from a deterministic control layer [22]. The shared principle is that a model may propose, diagnose, or structure information without owning the final authority. RouteCast uses that principle in a setting where the final authority cannot be a present-time checker. 3.2 Deterministic Computation Over Model-Derived Factors DeLLMa prompts an LLM for verbal state forecasts, converts them into a normalized state distribution, derives pairwise preferences from LLM rankings, fits an approximate utility function, and analytically maximizes expected utility [16]. RouteCast is therefore not distinguished by deterministic arithmetic alone. Its residual distinction is the evaluation object and input-governance protocol: model outputs may provide typed classifications, extracted factors, evidence candidates, and estimator runs, but frozen priors, admissibility-checked evidence records, and versioned transformations produce the route quantities and forecast-ranking that are later evaluated against transition outcomes. 3.3 Deterministic Checking and In-Loop Control GroundEval replaces LLM-as-judge scoring with deterministic, trace-based checks over stateful agent behavior, evidence access, and time-bounded constraints [9]. It works because the evaluation contract is checkable now: the evaluator can inspect what the agent fetched, cited, and was allowed to access. CARE keeps a non-LLM incumbent optimizer as the default path in scientific experimentation and uses an auditable evidence gate before authorizing a challenger policy [15]. It works because feedback arrives within the adaptation loop. RouteCast studies the complementary regime where correctness is unavailable at scoring time and the code-owned output must be graded later. 3.4 Delayed-Outcome and Venture-Evaluation Systems Several recent systems evaluate investments, ventures, or finance rationales under delayed outcomes. DIALECTIC uses multi-agent debate and numerical scores for startup evaluation, with backtesting on opportunities from five VC funds [1]. ValueBlindBench frames delayed-ground-truth financial-rationale evaluation as a pre-calibration metrology problem for LLM judges [4]. Strategic Foresight reports a fully prospective venture tournament over Kickstarter projects [7]. VCBench provides anonymized founder profiles for venture forecasting and explicitly addresses identity leakage [5]. Look-Ahead-Bench studies look-ahead bias in point-in-time financial LLM workflows [2]. SSFF combines machine-learning and LLM components for startup success forecasting [21]. These works motivate the delayed-ground-truth setting and provide important comparison points. They generally evaluate companies, founders, rationales, or single forecasts, often with model-owned scoring or model-derived or model-aggregated factors. RouteCast’s residual target is narrower: code-owned provisional ranking of competing typed strategic routes under delayed, censored, or private ground truth. 3.5 Residual Relationship to Prior Work RouteCast does not introduce deterministic decision arithmetic or delayed-outcome evaluation individually. Its residual contribution is their combination for competing typed strategic routes, with point-in-time provenance, frozen priors, admissibility-checked evidence, staged costs and values, and transition-level later resolution. The decision-analysis components draw on expected utility and statistical decision theory [18, 17], reference-class and outside-view reasoning [14], staged investment and real-options logic [6, 8], and value-of-information analysis [13]. 4 RouteCast Protocol RouteCast is a protocol for evidence-gated, code-owned provisional forecasting. The protocol has an authority boundary: models may propose route packets, assumptions, priors, and decompositions; evidence pipelines may produce admissible observations; versioned code and data own deterministic transformations and the forecast-ranking; later outcome resolution evaluates the forecast. LLM route proposals Point-in-time evidence Versioned code/data Code-owned forecast-ranking Later outcome resolution proposal only admissible observations forecast inputs provisional forecast evaluation decision-time information set Figure 2: RouteCast authority-boundary architecture. Models propose routes and factors; point-in-time evidence, versioned code, and frozen data produce a code-owned provisional forecast-ranking that is evaluated only after later outcome resolution. Code-owned does not mean objectively correct. It means the final transformation is explicit, reproducible, and not overwritten by model prose. A code-owned forecast-ranking can be wrong, miscalibrated, or overfit. The point is that the ranking can be audited, ablated, frozen, and later scored. Code-owned also does not mean model-free. Models may classify transitions, extract factors, propose evidence records, or contribute estimator runs. These outputs affect the computation only through typed and provenance-checked interfaces. The ownership claim is narrower: final numerical quantities and rankings are produced by a frozen deterministic pipeline, and model prose cannot directly overwrite them. Operationally, the protocol uses packetized provenance. A proposal packet records candidate routes and claims. An evidence packet records source excerpts, capture dates, trust tiers, corroborating and contradicting signals, and the point-in-time boundary. A ranking packet records code-computed quantities such as normalized scores, probabilities, costs, values, risk gates, and the selected binding transition. This packet structure makes leakage checks and future resolution possible. 5 Route Object and Forecasting A typed route is a sequence or graph of transitions: ei=(si,si+1,ai,mi,Ai,Ei,θi,ki,pi,Ui,ciE,ℓi,hi),e_i=(s_i,s_i+1,a_i,m_i,A_i,E_i, _i,k_i,p_i,U_i,c_i^E, _i,h_i), where sis_i and si+1s_i+1 are source and target states, aia_i is the actor, mim_i the mechanism, AiA_i the required asset, EiE_i the evidence set, θi _i the success threshold, kik_i the kill condition, pip_i the provisional transition probability, UiU_i epistemic uncertainty, ciEc_i^E execution or continuation cost, ℓi _i learning cost, and hih_i a resolution horizon when one is specified. pip_i is the provisional probability that transition eie_i will resolve positively under the information set available at decision time. UiU_i measures epistemic uncertainty around that estimate, for example disagreement or sensitivity across admissible estimators. It is not the transition’s failure probability 1−pi1-p_i. In the route object, pip_i enters route-value forecasting, while UiU_i helps identify which uncertainty should be tested next. UiU_i is not automatically subtracted from pip_i, and the present uncertainty measure is a heuristic rather than a calibrated confidence interval. When an implementation uses cross-family or cross-model disagreement, that disagreement is an implementation heuristic rather than a calibrated uncertainty estimator. The descriptors “code-computed” and “code-owned” refer to the deterministic transformation and final authority, not to model-independent provenance. The final pip_i is code-derived but may be model-mediated through transition typing and evidence extraction. UiU_i is a code-computed proxy derived partly from model-estimator disagreement. 5.1 Code-Owned Arithmetic and Gates Models may produce per-criterion rationales and structured sub-scores, but the aggregate arithmetic is code-owned. Criterion totals are computed as T=∑iwi⋅clip[0,100](si)∑iwi,T= _iw_i·clip_[0,100](s_i) _iw_i, over present criteria. Missing criteria are excluded from the denominator; model-stated totals are ignored. Eligibility and risk gates are applied before ranking and cannot be averaged away by a high narrative or value score; their definitions and thresholds are versioned and frozen before evaluation. 5.2 Evidence Reweighting The implemented protocol adjusts a prior from corroborating and contradicting evidence tiers: p(e)=clip[0,1](pprior(e)⋅Wreality(e)),p(e)=clip_[0,1] (p_prior(e)· W_reality(e) ), Wreality(e)=clip[0.3,1.5](1+0.5sat(C)−0.5sat(K)),W_reality(e)=clip_[0.3,1.5] (1+0.5sat(C)-0.5sat(K) ), where C and K are corroborating and contradicting evidence weights and sat(x)=∑ix(i)0.5i1+∑ix(i)0.5i.sat(x)= _ix_(i)0.5^i1+ _ix_(i)0.5^i. pprior(e)p_prior(e) is drawn from a frozen reference-class table or another versioned prior policy; it is not accepted directly from model prose. Where no calibrated prior exists, the value remains a preregistered proxy rather than a calibrated probability claim. In the saturation function, x(i)x_(i) denotes evidence weights sorted in descending order. If no admissible evidence is available, Wreality=1W_reality=1. Unverified evidence is excluded before computation. The rule is code-owned and saturating, so additional evidence has diminishing marginal effect. Its marginal contribution is not isolated by the retrospective pilot. 5.3 Staged Value, Execution Cost, and Cost-to-Learn Execution costs are computed bottom-up and floored by reference classes: ciE=max(∑jnijuij,creferencefloor(ei)).c_i^E= ( _jn_iju_ij,\;c_reference\ floor(e_i) ). Route value is folded as staged expected value: EV=−c1E+p1(−c2E+p2(⋯+pnV)).EV=-c_1^E+p_1 (-c_2^E+p_2 (·s+p_nV ) ). The terminal value proxy used by the pilot is V=TAM×αcapture×αdefensibility.V=TAM× _capture× _defensibility. This is a frozen ranking proxy, not an empirically calibrated company valuation. Its dominance in the decomposition ablation is reported as a construct-validity limitation. The staged-value formula is an implementation of standard staged decision and real-options reasoning, not a claim of mathematical novelty. The semantics are abandonment-aware: later execution costs are paid only if earlier transitions succeed. The learning cost ℓi _i is different from ciEc_i^E. It is the cost of an experiment or evidence-gathering action intended to reduce UiU_i or resolve the transition predicate before full execution. ℓi _i does not automatically enter the existing frozen expected-value formula unless the learning experiment itself is modeled as a route stage. 5.4 Binding Transition The binding transition is a VoI-inspired heuristic for the next uncertainty to test: b=argmaxiUiDi.b= _iU_iD_i. Here UiU_i is uncertainty and DiD_i is downstream stake. The current heuristic identifies a high-leverage uncertainty and reports its learning cost separately. A cost-aware value-of-information or knowledge-gradient selector belongs in prospective evaluation. The heuristic is a protocol output, not established decision utility. 5.5 Flat Probability Product The flat product ∏ipi _ip_i is retained only as a diagnostic because it ignores staged abandonment and is mechanically sensitive to route length. 6 Retrospective As-Of Pilot The pilot serves as an integrity audit and feasibility test, not as a confirmatory backtest. 6.1 Cohort and Outcome Mapping The retrospective pilot uses YC 2012–2014 companies with point-in-time descriptions reconstructed from public materials. The audit cohort contains 26 cases. Five cases with mixed outcome labels are retained in the case-level appendix but excluded from binary discrimination metrics. All reported AUC and lift estimates therefore use the 21-case binary subset: 6 positive and 15 negative. The sample is subsampled and outcome-balanced rather than prevalence-representative. It is also not cleanly held out from prior development, so the estimates should be treated as integrity and feasibility evidence rather than confirmatory performance. Heterogeneous later outcomes were mapped separately before binary aggregation. 6.2 Reconstruction and Blinding Point-in-time packets use descriptions and evidence intended to reflect what was admissible at the decision time, with Wayback verification where applicable. Later outcomes were stored separately from scoring inputs. For blinded packets, company names, domains, URLs, investor names, fame flags, and obvious brand cues were removed. Recognizability may remain from distinctive facts, and outcome raters may know later histories. The decomposition run used masked point-in-time inputs. Network access was disabled and regression-tested for that run, and each company was scored once under the preregistered rule. 6.3 Frozen Artifacts and Pilot Analyses The retrospective pilot contains a frozen whole-packet RouteCast score, a blind LLM judge with K=5K=5 correlated same-model passes, an identity-exposed LLM contrast, a simple deterministic heuristic, and a typed-route decomposition ablation. Score files were frozen before outcome join. The original whole-packet scores were frozen on 2026-07-07 before outcome labeling, with score-file SHA-256 beginning 4c4dff; the decomposition-ablation scores were frozen before outcome join at commit b0dbd8a, with SHA-256 beginning de4172cd. Both provisional score artifacts therefore predate this manuscript’s arXiv/public-disclosure package. The repository preserves the legacy field name crf_score in the frozen score artifact. The pilot asks four questions. Does the whole-packet code-owned score retain retrospective discrimination? Does identity exposure change apparent LLM-judge discrimination? Does typed decomposition add discrimination? What protocol weaknesses are exposed by the retrospective pilot? 7 Results Table 2: Retrospective pilot results on the 21-case binary subset. Raw scores are not cross-scorer comparable. Scorer N AUC ↑ 95% CI Top-Q lift ↑ Interpretation RouteCast whole-packet frozen score 21 0.756 [0.471, 0.980] 2.1 preliminary retrospective discrimination Blind LLM judge mean K=5K=5 21 0.678 [0.419, 0.897] 1.4 blinded LLM baseline Simple deterministic heuristic 21 0.700 [0.406, 0.948] 2.1 deterministic baseline Identity-exposed LLM judge 21 0.761 [0.515, 0.944] 1.4 leakage contrast only Note. Single-scorer CIs are company bootstraps with B=10,000B=10,000 and seed 20260710. The table reports discrimination, not calibration. Across the five correlated passes, blind-LLM AUC ranged from 0.650 to 0.706; RouteCast re-extraction remained at 0.756 in all five runs. The legacy frozen score field is crf_score. 7.1 Whole-Packet Analysis The whole-packet RouteCast score showed retrospective discrimination on the 21-case binary analysis subset: AUC 0.756, with a 95% company-bootstrap CI of [0.471,0.980][0.471,0.980], with top-quartile lift 2.1. The sample is too small for general claims, and raw scores are not comparable across scoring systems; AUC and lift use within-scorer rankings. 7.2 Identity-Exposure Contrast The blind LLM judge mean over K=5K=5 same-model passes reached AUC 0.678, with a 95% company-bootstrap CI of [0.419,0.897][0.419,0.897]. The identity-exposed LLM judge reached AUC 0.761 [0.515,0.944][0.515,0.944]. Identity exposure increased apparent discrimination in this experimental contrast. The result is consistent with recognition- or outcome-related leakage, but it does not establish that leakage caused the improvement. Distinctive masked facts may still permit recognition, so the blind result is also not leakage-free. The whole-packet RouteCast score and the exposed judge are not established as statistically different by this audit. A paired analysis would be needed for that claim, and the present paper does not include one. Retrospective discrimination is also not calibration. 8 Decomposition Ablation The preregistered decomposition ablation tested whether converting the same masked point-in-time packets into typed staged routes improved cohort ranking. The experiment was preregistered before outcome join at commit cb7625d. Scores were frozen before outcome join at commit b0dbd8a in eval/decomposed_backtest/scores_frozen.jsonl, with SHA-256 beginning de4172cd. Network access was disabled and regression-tested, each company was run once, and the reporting rule was fixed in advance: a tie or loss would be reported. Table 3: Decomposition ablation on the 21-case binary subset. Paired company bootstrap, B=10,000B=10,000, seed 20260710. Comparison Δ 95% CI Decomposed staged-NPV minus whole-packet RouteCast crf_score -0.144 [-0.471, 0.176] Decomposed staged-NPV minus deterministic heuristic -0.089 [-0.412, 0.278] Diagnostic: isolated chain probability product AUC 0.333 [0.092, 0.606] Against the whole-packet RouteCast score, ΔAUC=−0.144,95%CI=[−0.471,0.176]. =-0.144, 95\%\,CI=[-0.471,0.176]. Against the deterministic heuristic, ΔAUC=−0.089,95%CI=[−0.412,0.278]. =-0.089, 95\%\,CI=[-0.412,0.278]. Both intervals span zero. The verdict under the preregistered rule is: indistinguishable on this cohort. The experiment provides no evidence that decomposition improves discrimination beyond the whole-packet protocol on this cohort. The diagnostics are more useful than the point estimate. The isolated chain probability product had AUC 0.333 with 95% CI [0.092,0.606][0.092,0.606]. Because it ignores staged abandonment and is mechanically sensitive to route length, it can anti-rank later positives when ambitious chains contain more transitions. Staged-NPV discrimination was dominated by terminal value/TAM rather than by the transition probability chain. The first edge was labeled observed in 58% of generated chains, which suggests E0 inflation when a thin product description is treated as demonstrated traction. These diagnostics motivated protocol hardening and prospective evaluation rules; they are not evidence of calibrated edge forecasting. The null ablation limits the discrimination claim; it does not remove the value of typed transitions as preregistered resolution interfaces and tools for error localization. 9 What the Evidence Supports The pilot supports an identity-exposure warning and provides preliminary evidence of whole-packet retrospective discrimination on this cohort. It does not support a typed-decomposition advantage. It does not establish calibration, incremental value from evidence reweighting, or decision utility. Cross-domain transfer and scorer self-improvement remain untested. Discrimination and calibration are not interchangeable. AUC 0.756 [0.471,0.980][0.471,0.980] on a small retrospective cohort does not imply that the probability assigned to a transition is numerically calibrated, and the negative decomposition ablation does not imply that typed routes are useless. It only says they did not add ranking signal in this pilot implementation on this cohort. 10 Limitations Internal Validity. The retrospective pilot uses historical reconstruction. Packet construction involved human choices, outcome labels may compress heterogeneous trajectories, and point-in-time reconstruction can be contaminated by hindsight. Outcome raters may know later company histories. Masked facts can remain recognizable. Evidence-source selection and reference-class construction may also carry leakage or selection bias. Statistical Validity. The binary cohort has n=21n=21 cases with 6 positives. AUC estimates are unstable and confidence intervals are wide. Multiple comparisons are present, and paired uncertainty is not resolved for every contrast. There is no independent external test cohort. Future transition-level analyses must cluster by route because edges within a route are not independent samples. Construct Validity. Funding, traction, or persistence is not identical to route quality. Outcome labels compress pivots, acqui-hires, shutdowns, revenue, customer proof, and capital access. The decomposition ablation used one-sentence packets; its route chains are inferential reconstructions, not verified accounts of realized company plans. Terminal value/TAM dominance means staged NPV can recover a large-market signal rather than transition-level edge quality. External Validity. The pilot is venture-only, historical, public-data-limited, and affected by survivorship and censored failures. It does not show transfer to OSS, R&D, product roadmaps, or private operational settings. Market regimes and funding environments also change. System Validity. Evidence acquisition remains partly manual. Code-owned output can be wrong, priors can be misspecified, and private outcomes may never resolve. Evidence reweighting, binding-transition selection, and route-level probabilities require prospective calibration before they can be treated as forecast-quality claims. Model-mediated transition typing may select an inappropriately favorable reference class. Cross-family agreement may also understate uncertainty when model families share correlated blind spots. Prospective evaluation should therefore include transition-type sensitivity, source deduplication, reference-class coverage diagnostics, and non-model uncertainty baselines. 11 Prospective Protocol The Open Experiment is the planned implementation of the scientific protocol, not evidence that the retrospective pilot is calibrated. Cases should be registered prospectively, with frozen model and scorer versions, typed transitions created before outcomes, milestone predicates specified before resolution, and 30/90/180-day or domain-appropriate windows. Outcome resolution should be blinded where feasible. The protocol should evaluate both discrimination and calibration. Metrics include Brier score, calibration curves, expected calibration error with binning caveats, AUC, top-bucket lift, and decision-relevant utility measures where a defensible counterfactual exists. Confidence intervals for transition-level metrics should use a cluster bootstrap by route. Baselines should be credible enough to fail the method: a fitted logistic or gradient-boosted model on the same structured factors; a GroundEval-like contract baseline that checks evidence compliance; a CARE-like incumbent/challenger gate; a standard decision tree or flat-NPV baseline; a route-structure ablation; a reality-reweighting ablation; and a binding-transition comparison against a defensible value-of-information or knowledge-gradient baseline. Recognition should be measured as a covariate rather than treated as a binary nuisance. Prospective forecast calibration is the target of this program. 12 Discussion 12.1 From Checker to Forecaster A checker evaluates against an available contract. A forecaster issues a provisional forecast before correctness is observable. Later outcomes grade the forecast. The question is not whether code or a model scores better, but who owns the forecast-ranking when the world cannot yet grade it. RouteCast makes the ownership boundary explicit: the model may propose, but the provisional ranking is a versioned computation. A frozen milestone predicate turns today’s provisional forecast into a checkable contract at resolution time. 12.2 Versioned Learning Loop Resolved outcomes do not revise the frozen forecast. They enter a separate, versioned learning loop that measures forecast error and evaluates candidate updates to priors, weights, thresholds, evidence-reweighting parameters, and calibration mappings. The objective is to improve the calibration and discrimination of future forecasts. A new scorer version is promoted only when preregistered evaluation on held-out resolved cohorts shows improvement without material integrity regressions. 12.3 Adversarial Silence Internal red-teaming of the staged expected-value fold surfaced an adversarial omission pattern: deleting risky intermediate transitions from a route can raise its provisional score, because an unmodeled span pays neither cost nor probability. The same typed-transition structure that enables later resolution also makes this failure mode detectable as a deterministic coverage check over typed state records, without recourse to a model judge. A systematic treatment—including exploit measurement, a deterministic detector, and value-provenance coupling intended to close the exploit—is reserved for a companion paper. 12.4 Domain Extension OSS project selection, research-direction selection, and product-roadmap prioritization are proposed transfer tests. They are not established domains. Each would need its own point-in-time packet construction, outcome mapping, leakage audit, calibration analysis, and negative-result reporting. 13 Conclusion We studied code-owned provisional evaluation under delayed ground truth, a regime in which deterministic code cannot check correctness at scoring time and must instead issue a forecast-ranking. RouteCast instantiates this regime for typed strategic routes using point-in-time evidence and auditable computation. In a small retrospective pilot, the whole-packet score showed discrimination and identity exposure increased apparent LLM-judge performance, while a preregistered decomposition ablation was indistinguishable from the whole-packet protocol. Prospective transition-level resolution is required to evaluate calibration, decision utility, and cross-domain transfer. Competing interests The author is developing a commercial implementation of the protocol through Dynamic Resonance. References [1] J. Y. Bae, S. Malberg, J. Galang, A. Retterath, and G. Groh (2026-03) DIALECTIC: a multi-agent system for startup evaluation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), Rabat, Morocco, p. 711–727. Note: arXiv:2603.12274 [cs.MA], v1 submitted 2026-02-19; EACL Industry Track, March 25–27, 2026. https://aclanthology.org/2026.eacl-industry.53/ External Links: Document, 2603.12274 Cited by: §3.4. [2] M. Benhenda (2026) Look-ahead-bench: a standardized benchmark of look-ahead bias in point-in-time LLMs for finance. Note: arXiv:2601.13770 [cs.AI] External Links: 2601.13770, Document Cited by: §3.4. [3] G. W. Brier (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 (1), p. 1–3. Note: https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.xml Cited by: §2. [4] S. Chang, P. Zhu, and Y. Chen (2026) ValueBlindBench: agreement-gated stress testing of LLM-judged investment rationales before returns are observable. Note: arXiv:2604.25224 [cs.AI] External Links: 2604.25224, Document Cited by: §3.4. [5] R. Chen, J. Ternasky, A. S. Kwesi, B. Griffin, A. O. Yin, Z. Salifu, K. Amoaba, X. Mu, F. Alican, and Y. Ihlamur (2025) VCBench: benchmarking LLMs in venture capital. Note: arXiv:2509.14448 [cs.AI] External Links: 2509.14448, Document Cited by: §3.4. [6] R. G. Cooper (1990) Stage-gate systems: a new tool for managing new products. Business Horizons 33 (3), p. 44–54. External Links: Document Cited by: §3.5. [7] F. A. Csaszar, A. Peterson, and D. Wilde (2026) The strategic foresight of LLMs: evidence from a fully prospective venture tournament. Note: arXiv:2602.01684 [econ.GN] External Links: 2602.01684, Document Cited by: §3.4. [8] A. K. Dixit and R. S. Pindyck (1994) Investment under uncertainty. Princeton University Press, Princeton, NJ. External Links: Document Cited by: §3.5. [9] J. Flynt (2026) GroundEval: a deterministic replacement for LLM-as-Judge in stateful agent evaluation. Note: arXiv:2606.22737 [cs.AI] External Links: 2606.22737, Document Cited by: §3.3. [10] J. G. Fuller (2020) Run-time assurance: a rising technology. In 2020 IEEE/AIAA 39th Digital Avionics Systems Conference (DASC), p. 1–9. External Links: Document Cited by: §3.1. [11] T. Gneiting and A. E. Raftery (2007) Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102 (477), p. 359–378. External Links: Document Cited by: §2. [12] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 70, p. 1321–1330. Note: https://proceedings.mlr.press/v70/guo17a.html Cited by: §2. [13] R. A. Howard (1966) Information value theory. IEEE Transactions on Systems Science and Cybernetics 2 (1), p. 22–26. External Links: Document Cited by: §3.5. [14] D. Kahneman and D. Lovallo (1993) Timid choices and bold forecasts: a cognitive perspective on risk taking. Management Science 39 (1), p. 17–31. External Links: Document Cited by: §3.5. [15] G. Liu, W. Kong, Z. Wang, B. Zhang, B. Li, P. Zhang, and T. Shi (2026) CARE: controlling LLM-generated policies through auditable review of evidence in scientific experimentation. Note: arXiv:2606.14581 [cs.LG] External Links: 2606.14581, Document Cited by: §3.3. [16] O. Liu, D. Fu, D. Yogatama, and W. Neiswanger (2024) DeLLMa: decision making under uncertainty with large language models. Note: arXiv:2402.02392 [cs.AI] External Links: 2402.02392, Document Cited by: §3.2. [17] H. Raiffa and R. Schlaifer (1961) Applied statistical decision theory. Harvard Business School, Boston. Cited by: §3.5. [18] L. J. Savage (1954) The foundations of statistics. John Wiley & Sons, New York. Cited by: §3.5. [19] D. Seto, B. Krogh, L. Sha, and A. Chutinan (1998) The simplex architecture for safe online control system upgrades. In Proceedings of the 1998 American Control Conference, Vol. 6, p. 3504–3508. External Links: Document Cited by: §3.1. [20] L. Sha (2001) Using simplicity to control complexity. IEEE Software 18 (4), p. 20–28. External Links: Document Cited by: §3.1. [21] X. Wang, Y. Ihlamur, and F. Alican (2024) SSFF: investigating LLM predictive capabilities for startup success through a multi-agent framework with enhanced explainability and performance. Note: arXiv:2405.19456 [cs.AI] External Links: 2405.19456, Document Cited by: §3.4. [22] S. Zhong, Y. Su, and M. Witbrock (2026) Separating diagnosis from control: auditable policy adaptation in agent-based simulations with LLM-based diagnostics. Note: arXiv:2603.22904 [cs.AI] External Links: 2603.22904, Document Cited by: §3.1. Appendix A Supplementary Case-Level Pilot Table The full case-level pilot table is appendix-only and uses blinded IDs. Original company names and the blind-ID map are intentionally not shown in the paper. Appendix Table A1. Case-level blinded pilot detail. Original company names are withheld from the paper; blind IDs map to de-identified point-in-time packets. Raw scores are not cross-scorer comparable; AUC and lift use within-scorer rankings. Buckets: S+ strong-positive, + positive, ∼ mixed, - negative. The RouteCast score column preserves the legacy frozen crf_score field; the blind LLM column reports the mean over K=5K=5 same-model blinded passes. Blind ID Category RouteCast score ↑ within-scorer rank only Blind LLM mean score ↑ separate scale Heuristic score ↑ separate scale RouteCast predicted bucket Later outcome bucket Recog. risk BC01 consumer / search 36.6 54 33 neg - – BC02 hardware / robotics 45.4 58 69 mix - – BC03 real estate / analytics 47.2 46 69 mix - – BC04 SaaS / hiring 49.7 50 71 pos ∼ – BC05 consumer / e-commerce 42.5 57 67 mix - – BC06 developer tools / infrastructure 55.6 60 89 pos + yes BC07 consumer / social 39.2 60 43 neg - – BC08 SaaS / developer API 55.3 47 81 pos - yes BC09 developer tools / cloud 55.5 55 79 pos + yes BC10 developer tools / AI API 48.4 63 47 mix + – BC11 consumer / productivity 31.5 38 13 neg - – BC12 SaaS / adtech 45.0 52 67 mix ∼ – BC13 SaaS / developer services 47.6 52 59 mix - – BC14 healthcare / mental wellness 48.2 54 45 mix - – BC15 consumer / video 39.0 62 43 neg ∼ – BC16 SaaS / web design tools 53.0 57 67 pos S+ yes BC17 consumer / on-demand services 41.3 55 57 neg ∼ – BC18 consumer / local delivery 40.0 52 57 neg S+ yes BC19 SaaS / mobile tools 50.8 48 71 pos - – BC20 SaaS / careers 47.4 61 67 mix - – BC21 education 40.7 44 43 neg - – BC22 SaaS / communications API 57.8 60 69 pos - yes BC23 govtech / nonprofit software 51.0 62 69 pos - – BC24 security / identity 53.0 55 79 pos + – BC25 consumer / utility 40.7 50 43 neg - – BC26 consumer / marketplace 40.6 52 67 neg ∼ – Appendix B External Evaluation Tracks Appendix Table B1. Proposed external evaluation tracks. These tracks are transfer tests for the prospective protocol; they do not convert the retrospective pilot into evidence of cross-domain validity. Track Needed data owner/source First pilot Incentive Venture routes Accelerators, angel groups, venture studios, or founder communities with timestamped route packets and later outcomes. Blind point-in-time route ranking on an older resolved cohort, followed by a small prospective cohort. Better audit discipline for route selection without turning funding into the only outcome proxy. OSS/developer project selection Maintainers, developer platforms, package registries, or hackathon/program operators with public adoption traces. Point-in-time ranking of project directions using only repository, package, and community signals available before later adoption. Transparent evidence about which project bets earn durable use rather than short-lived attention. Novel research direction selection Labs, workshop organizers, OpenReview-style venues, or benchmark maintainers with timestamped proposals and later traces. Blind ranking of research directions before acceptance, reuse, citation, or benchmark uptake is known. Better evaluation of tractable novelty under noisy peer-review and reuse signals. Product roadmap prioritization Growth-stage product teams willing to share de-identified roadmap, feature, and metric histories. Retrospective point-in-time ranking of shipped and rejected feature bets, then preregistered prospective scoring. Decision support that reports uncertainty, negative evidence, and opportunity cost rather than persuasive roadmap prose.