Paper deep dive
Split the Labor: Separating Evidence Interpretation from Decision Aggregation
Zhelun Wu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them. We propose a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves. The separation also reveals a failure mode in how such systems combine, which we call count-scale drift. Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, the vote rule and the posterior order instances differently, and no threshold reconciles them. Pooling calibrated log-likelihood ratios addresses both problems. The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by counting positives, and additive multi-signal detectors. We then instantiate the principle twice on one longitudinal corpus, once after outcomes resolve and once before. The same partition helps in both, at different granularities: over reading in the first, over learning capacity in the second. There, a small sequence encoder on an easy auxiliary objective plus a tree ensemble carrying the censored survival loss reaches 0.921 AUPRC against 0.805 for a hand-crafted baseline. We separate what transfers from what must be re-estimated per domain, and state five predictions that would falsify the framework, three negative results, and which comparisons remain confounded.
Tags
Links
- Source: https://arxiv.org/abs/2608.14509v1
- Canonical: https://arxiv.org/abs/2608.14509v1
Trouble viewing inline? Open PDF directly →
Full Text
91,299 characters extracted from source content.
Expand or collapse full text
Split the Labor: Separating Evidence Interpretation from Decision Aggregation — A Design Principle, Its Failure Mode, and Two Instantiations Zhelun Wu Affiliation: Atlassian Email: zhelunwu@hotmail.com Abstract Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them. We propose a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves. The separation also reveals a failure mode in how such systems combine, which we call count-scale drift. Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, the vote rule and the posterior order instances differently, and no threshold reconciles them. Pooling calibrated log-likelihood ratios addresses both problems. The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by counting positives, and additive multi-signal detectors. We then instantiate the principle twice on one longitudinal corpus, once after outcomes resolve and once before. The same partition helps in both, at different granularities: over reading in the first, over learning capacity in the second. There, a small sequence encoder on an easy auxiliary objective plus a tree ensemble carrying the censored survival loss reaches 0.921 AUPRC against 0.805 for a hand-crafted baseline. We separate what transfers from what must be re-estimated per domain, and state five predictions that would falsify the framework, three negative results, and which comparisons remain confounded. 1 Introduction Decision-support systems increasingly ask language models to interpret unstructured evidence distributed across many partially informative and sometimes conflicting sources. The prevailing architecture concatenates them into one prompt. This is simple and often effective, but it has four costs. Provenance is lost, and recovering attribution after the fact is itself an open problem [30, 12]. Evidence quality is heterogeneous, so unweighted joint reading lets many uninformative sources dominate a few decisive ones. Long contexts degrade retrieval of mid-context information [26]. And a generative system asked for a conclusion supplies one regardless of support. These four failures share a cause. A monolithic reader is being asked to do two jobs whose requirements point in opposite directions. Interpretation rewards capacity, context, and the freedom to weigh nuance; combination rewards fixed arithmetic, comparability across instances, calibrated strength, and the discipline to return nothing. Asking one mechanism for both means each job is done at the other’s terms, and the interface between them — where provenance, reliability, and uncertainty would live — never gets designed at all. The principle. Partition the inference at that interface, then design the interface first. Concretely: read each source in isolation under a causal context constraint and require it to emit a fixed evidence tuple; then combine tuples with an explicit, auditable rule. Fixing the contract determines the rest. Four things follow. Reliability is declared rather than implied, so weighting becomes an estimation problem instead of a tuning problem. Provenance is structural rather than reconstructed. Failures are isolated to a single reading operation. And abstention becomes expressible: a rule over tuples can return no conclusion, where a generative reader asked for one will produce it anyway. The partition also generalizes past reading: in our second study the same cut is applied to learning capacity, with a small encoder taking the easy sub-problem and a tree ensemble the hard one. Why the combination step needs attention. Systems that do partition usually combine by summing weights and thresholding the sum. Section 4 shows that this is not a different rule from posterior thresholding. It is the same rule, evaluated at an operating point that drifts with the number of sources. The drift rate depends on parameters that are rarely estimated in practice. The result is stated generally enough to cover a class of rules used well beyond language models. It also comes with a diagnostic that can be computed on data these systems already log. None of the mechanisms we compose is new. Selective prediction with a reject option dates to Chow [5] with a modern treatment [10, 13]. Reliability-weighted aggregation of noisy annotators originates with Dawid and Skene [8, 38, 31]. Weight of evidence [15], opinion pooling [14], evidential combination [33], and noisy-OR [29] address combination from different starting points. Task decomposition [23] and cross-sample aggregation [36] are established prompting techniques. What is new is the composition rather than any mechanism: which cut to make, what contract to place at the cut, which combination rules that contract admits, and what fails when an unsuitable rule is used. • A design principle with an interface. Section 3 states the evidence tuple, the four axes along which a task determines its instantiation (evidence locality, reliability heterogeneity, dependence structure, resolution regime), and the conditions under which partitioning pays rather than costs. • Analytical. Section 4 names and characterizes count-scale drift (Proposition 2): threshold rules over unnormalized weight sums are posterior rules with an operating point linear in source count, biasing toward over-assertion on evidence-rich instances, worse for more reliable readers. Proposition 1 shows the additive corroboration bonus is decision-equivalent to rescaling the threshold. Pooling calibrated log-likelihood ratios removes both defects, is exact under stated conditions (Proposition 4). Proposition 3 shows the heterogeneous case is worse than drift alone: unless weights are proportional to per-bucket log-odds gain, no posterior threshold reproduces the vote rule at all. • Methodological. We do not use elicited model confidence, since it is poorly calibrated [21, 39, 18]. Instead, reliability buckets are constructed from observable properties of the reading operation: source length, and whether the reader returned a non-empty grounded rationale. Their sensitivity and false-positive rates are then estimated from data. We give a block-structured dependence discount so sources sharing an upstream origin do not accumulate as independent corroboration, and we pool over the reader’s confusion matrix rather than per-label rates, because our reader emits one label per source and per-label pooling is misspecified in that regime (Section 4.4). • Two instantiations, one object, two regimes. The same corpus is studied after resolution (Section 6) and before it (Section 7). What changes between them is only whether the label exists and whether future evidence is admissible; the partition pays in both, which is stronger evidence for the principle than either study alone. • Transfer, stated systematically. Section 8 maps the contract onto four other domains along the same four axes, and separates what is domain-neutral (the contract, the pooling arithmetic, the drift diagnostic) from what must be re-estimated (rates, blocks, thresholds). • A falsifiable programme. Section 10 states five predictions the framework makes and the experiment that discriminates each, with three negative results and an explicit register of confounds. One boundary is worth drawing early, because it determines how the paper should be read. The results are measurements of a running system whose combination step is the weighted vote our analysis criticizes (Section 4.1); the log-likelihood-ratio construction is derived, not deployed. So the empirical claims here concern the partition and the reading strategy, and the analytical claims concern the combination rule. Keeping those separate is what makes each usable: a measured architecture with a diagnosed defect is more actionable than an unmeasured architecture without one. 2 Related work Selective prediction. Chow’s reject option [5] formalizes the coverage/risk trade-off, developed in [10, 13]; it motivates our emission threshold, though we report no risk–coverage curve. Conformal prediction [1] offers distribution-free coverage guarantees and is a natural alternative to our threshold pair; we discuss why we did not adopt it in Section 9. Aggregating unreliable sources. Dawid and Skene [8] estimate per-annotator error rates from unlabeled data by EM; [38, 31] extend this. Our per-bucket rates are a coarsened Dawid–Skene model in which “annotator” identity is the reader’s confidence bucket rather than a person, which keeps the parameter count small at our label frequencies. Unlike Dawid–Skene we estimate supervised, which trades label efficiency for stability; the unsupervised variant is one of our baselines. Decomposition and attribution. [37, 23, 36] decompose reasoning but aggregate by majority vote over samples of one context, not over independent evidence sources. RAG [25] supplies sources and usually concatenates. Attribution metrics [30, 12, 28] and consistency-based detection [27] evaluate groundedness after generation; we make provenance a structural output. Survival analysis and tabular learning. We work in discrete time rather than in the continuous-time proportional-hazards tradition [6]; competing-risk formulations follow [11, 35]; neural approaches include [22, 24] and tree-based approaches [20]. That a boosted tree [3] carrying the survival loss beats an end-to-end neural model is consistent with trees remaining competitive on tabular data at moderate scale [17, 34]. 3 The evidence interface 3.1 The contract A reading operation returns ei=(Y^i,bi,ri,ai)e_i=( Y_i,b_i,r_i,a_i): a hypothesis over the label space, an ordinal reliability bucket, a rationale, and provenance. Each of the four fields removes a specific failure. The hypothesis makes the reader’s commitment explicit rather than leaving it embedded in prose. The bucket carries reliability as data, so combination weights can be estimated from outcomes instead of chosen by hand. It is an ordinal bucket rather than a probability, because elicited model confidence is poorly calibrated [21, 39, 18]. Our buckets are therefore built from observable properties of the reading operation itself. In the deployment studied here, two such properties are used: whether the source is long enough to be informative, and whether the reader returned a grounded rationale. The rationale field makes that judgement auditable, and is one of the properties the bucket is derived from. Provenance makes attribution structural, so no post-hoc attribution model is needed to answer why a label was asserted. The contract is deliberately weak about what a reader is. Anything that emits the tuple composes: a language model over text, a statistical detector over telemetry, a vision model over images, a human reviewer. What they must share is a calibration scale, not an architecture — which is the mechanism behind the transfer claims in Section 8. Figure 1 shows the partition and the interface it creates. s1s_1s2s_2sns_nread s1s_1read s2s_2read sns_ne1=(Y^1,b1,r1,a1)e_1=( Y_1,b_1,r_1,a_1)e2=(Y^2,b2,r2,a2)e_2=( Y_2,b_2,r_2,a_2)en=(Y^n,bn,rn,an)e_n=( Y_n,b_n,r_n,a_n)causal context LLR pooling αb,y,βb,y _b,y, _b,y rates κj _j dependence discount π(y∣z)π(y z) prior assertabstainrejectthe interfaceinterpretation: capacity, contextcombination: calibrated arithmetic Figure 1: The partition. Each source is read alone under a causal context constraint and must emit the same four-field evidence tuple; a fixed rule then combines tuples into a decision that includes abstention. The dashed line is the design surface: everything left of it can be replaced without touching the arithmetic, and everything right of it can be corrected without re-reading a source. 3.2 Four axes that determine an instantiation Given a task, four properties fix most design choices. 1. Evidence locality. Is a source interpretable alone, or only against its predecessors? This sets how much context each reading operation gets, and whether the context is causal or complete. 2. Reliability heterogeneity. How widely does source informativeness vary, and is that variation observable before reading? Wide and observable variation is what makes buckets worth carrying. 3. Dependence structure. Do sources share upstream origins? Blocks of shared origin must not accumulate as independent corroboration (Section 4.6). 4. Resolution regime. Has the outcome resolved? If so, an empirical prior is available and backfilling a record is meaningful; if not, the prior is unavailable and the output is a live read, not a committed conclusion. Our two studies differ on exactly one of these — the fourth — which is why we treat them as one experiment on the principle rather than two unrelated applications. 3.3 When partitioning pays Partitioning is not free: it forfeits joint reasoning across sources, adds an estimation problem for the rates, and requires the interface to be maintained as prompts evolve. Four conditions make it worthwhile. The source count varies enough that comparability across instances matters. Reliability is heterogeneous enough that weighting changes decisions. Provenance is an obligation rather than a convenience. And abstention has value, meaning a wrong committed answer costs more than no answer. Where sources are few, homogeneous, and uncontested, concatenation is the better architecture and this paper does not apply. 3.4 Formal setting An instance comprises sources =s1,…,snS=\s_1,…,s_n\ and metadata z. The target is a label set Y⊆Y . We formulate multi-label because our metrics are multi-label; a single argmax is inconsistent with reporting instance-level F1F_1. An asymmetry we must carry through the formalism. Ground truth is a set, but our deployed reader emits at most one label per source, |Y^i|≤1| Y_i|≤ 1, with a distinguished value for “insufficient evidence”. The multi-label target is therefore assembled entirely by aggregation. Two consequences follow, and both affect the aggregation. First, one source’s assertions across labels are mutually exclusive by construction, so they are negatively dependent given the label and cannot be pooled label-by-label as if independent (Section 4.4). Second, silence about y is never uninformative: a source that asserted a competing label was asked and answered otherwise. We keep the general notation Y^i Y_i because nothing in the aggregation requires |Y^i|≤1| Y_i|≤ 1, but every number we report comes from the single-label regime. Each source yields an evidence object ei=fθ(si,ci)=(Y^i,bi,ri,ai),e_i=f_θ(s_i,c_i)=( Y_i,\,b_i,\,r_i,\,a_i), (1) with cic_i the causally available context, Y^i Y_i the asserted labels, bi∈1,…,Bb_i∈\1,…,B\ an ordinal reliability bucket, rir_i the rationale, aia_i provenance. Per label y, an aggregator returns p^(y) p(y) and a decision in assert,reject,abstain\assert,reject,abstain\. 3.5 Independent reading Each source is read alone, with a bounded summary of causally prior sources and never future ones. This yields traceability (each assertion maps to one reading operation with rationale attached), bounded context (cost scales with source count, and late sources are not truncated because earlier ones consumed the window [26]), and failure isolation. The cost argument is arithmetic rather than measured: reading each source once is linear in source count, where concatenation grows super-linearly in prompt length. We report no wall-clock or token measurements and make no latency claim. The causal constraint is a deployment requirement: a system scoring a live instance cannot see future sources. Full-context reading is therefore an oracle reference only. Note that causal context induces dependence between evidence objects: reader i conditions on a summary of sources 1..i−11..i-1, so assertions are not conditionally independent given the label. This is the motivation for the dependence discount in Section 4.6, not an afterthought. 4 Combining evidence: a defect and its repair 4.1 The deployed rule Every number in Section 6 comes from the following rule, and we state it first so that the analysis below is read as criticism of a running system rather than as a description of one. Each source receives a weight from a fixed table. A source contributes zero if the reader returned no determination, and zero if the source is short and carries any determination other than an unresponsiveness label. It contributes 0.30.3 if it is short and carries that label, and 0.30.3 if it is long enough but its determination arrives without a grounded rationale. It contributes 1.01.0 if it is long enough and has one. A label is emitted when its summed weight reaches 1.01.0. Ties are broken toward the latest source, and both labels are surfaced if a tie survives. One label is treated asymmetrically: a generic unresponsiveness label is admitted on short sources, where nothing else is, and is then dropped post hoc whenever any other label survives threshold. So the emitted set is not simply the set above threshold, and the formalism of Section 3 does not capture this rule — a subsumption constraint over labels sits outside it. For resolved instances a single most-frequent label per source-count bucket is layered on as a prior fallback, fitted on the 129 calibration instances of Section 6.2. The weights and the threshold were set by hand and never fitted. Two properties of this rule matter for what follows. Its weights are bucket-dependent and were not chosen to be proportional to any log-odds quantity, so it is the heterogeneous case of Proposition 3 rather than the uniform case of Proposition 2: it inherits count-scale drift and adds a misweighting that no threshold can absorb. And its reliability buckets are constructed from observable properties of the reading operation — source length and rationale presence — not from elicited model confidence, which is the one design choice here we would keep unchanged. 4.2 Three defects in weighted voting A common rule assigns weights wiw_i and scores V(y)=∑iwi[y∈Y^i]V(y)= _iw_iI[y∈ Y_i], then adds a corroboration bonus λh(C(y))λ h(C(y)) where C(y)=∑i[y∈Y^i]C(y)= _iI[y∈ Y_i] counts asserting sources and h is increasing, e.g. h(C)=log(1+C)h(C)= (1+C). Proposition 1 (Redundancy of the corroboration bonus). If weights are uniform, wi=w>0w_i=w>0, then for any λ≥0λ≥ 0 and any strictly increasing h, the decision rule S(y)≥τ\S(y)≥τ\ with S(y)=V(y)+λh(C(y))S(y)=V(y)+λ h(C(y)) is identical to a threshold rule on the vote count alone: there exists τ′τ with S(y)≥τ=C(y)≥τ′\S(y)≥τ\=\C(y)≥τ \. Proof. Under uniform weights V(y)=wC(y)V(y)=w\,C(y), so S(y)=wC(y)+λh(C(y))=:ϕ(C(y))S(y)=w\,C(y)+λ h(C(y))=:φ(C(y)) where ϕ(c)=wc+λh(c)φ(c)=wc+λ h(c) is strictly increasing in c as a sum of strictly increasing and non-decreasing functions. A strictly increasing ϕφ on the integers admits a threshold correspondence: set τ′=minc∈ℤ≥0:ϕ(c)≥τ = \c _≥ 0:φ(c)≥τ\, with τ′=∞τ =∞ when the set is empty (the rule never asserts). Then ϕ(C)≥τ⇔C≥τ′φ(C)≥τ C≥τ . ∎ Remark 1. Proposition 1 is elementary — any strictly increasing function of a scalar statistic induces the same family of threshold rules — and we state it as a proposition only because the bonus term is widely implemented as though it changed the decision. The substantive content of this section is Proposition 2. So under uniform weights the bonus induces no decision that rescaling the threshold could not, and λ is unidentifiable from decisions alone. The bonus has an effect only when weights are heterogeneous, and it then trades weight against count with no principle fixing the exchange rate between them. Proposition 3 supplies that exchange rate. Once weights are set accordingly, the bonus has nothing left to do. Before the general statement, it helps to see the mechanism on a small case. Suppose three sources are read and two assert y. Now suppose six are read and the same two assert y. A vote rule cannot distinguish these cases: both have a vote score of two. A posterior can. In the second case, four sources were asked about y and declined it, which is evidence against. The vote rule therefore treats a case carrying more contrary evidence as identical to one carrying less. The proposition below makes this exact and gives the rate. Proposition 2 (A vote threshold is a posterior threshold that drifts with source count). Assume (i) every source is read for every label, so non-assertion is informed silence; (i) readers share bucket rates (α,β)(α,β) with α>βα>β; (i) assertions are conditionally independent given the label. Write A=logαβ>0A= αβ>0 and D=log1−α1−β<0D= 1-α1-β<0. For an instance with n sources of which k assert y, the exact posterior log-odds are logitPr[y∣ℰ]=logitπ(y)+kA+(n−k)D.logit [y ]=logitπ(y)+kA+(n-k)D. (2) Then for uniform weights the vote rule V=wk≥τ\V=wk≥τ\, with τ′=τ/wτ =τ/w, is exactly the posterior rule logitPr[y∣ℰ]≥T(n)\logit [y ]≥ T(n)\ where T(n)=logitπ(y)+τ′(A−D)+nD.T(n)=logitπ(y)+τ (A-D)+nD. (3) The vote rule is thus not a different rule from posterior thresholding; it is posterior thresholding at an operating point that falls linearly in n at rate |D|=|log1−α1−β||D|= | 1-α1-β | per additional source. Proof. Equation 2 is the weight-of-evidence decomposition [15]: each asserting source contributes A and each silent source contributes D. Regrouping, logitPr[y∣ℰ]=logitπ(y)+nD+k(A−D)logit [y ]=logitπ(y)+nD+k(A-D) with A−D>0A-D>0, so k≥τ′⇔logitPr[y∣ℰ]≥logitπ(y)+nD+τ′(A−D)=T(n)k≥τ [y ] π(y)+nD+τ (A-D)=T(n). ∎ We call this count-scale drift. A concrete case may help. Take α=0.9α=0.9, β=0.2β=0.2 and a prior of 0.20.2, so that each assertion is worth +1.50+1.50 in log-odds and each silence −2.08-2.08. Consider two instances that a vote rule cannot distinguish, since both have exactly two asserting sources. With three sources the posterior log-odds are −0.46-0.46, or a probability near 0.390.39. With six sources they are −6.69-6.69, or a probability near 0.0010.001. The vote score is identical in both cases; the posteriors differ by roughly a factor of 300300. Equation 3 is more useful than a bare incomparability claim in three ways. It identifies the direction of the drift (D<0D<0, so the bar falls as evidence accumulates). It gives the rate, which depends on a parameter that is rarely estimated in practice. The drift is larger for more reliable readers, since α→1α→ 1 sends |D|→∞|D|→∞. And it makes the problem a matter of degree rather than of kind. The practical question is then how widely n varies in a given deployment. Where n is nearly constant, the vote rule is a harmless reparameterization. Where n ranges over an order of magnitude, as it does here from single-source instances up to 25, it is not. Figure 2 plots the effective threshold against n at three reader reliabilities. 11223344556677885500−5-5−10-10−15-15−20-20number of sources consulted, nnimplied threshold (log-odds)calibrated rule: one threshold, flat in nnvote rule, (α,β)=(0.75,0.25)(α,β)=(0.75,0.25)vote rule, (α,β)=(0.95,0.05)(α,β)=(0.95,0.05) Figure 2: Count-scale drift (Proposition 2), plotted in posterior log-odds. A vote threshold is a posterior threshold, but not a fixed one: the bar it enforces falls linearly in the number of sources consulted, at rate log1−α1−β 1-α1-β. More reliable readers drift faster — at (α,β)=(0.95,0.05)(α,β)=(0.95,0.05) the effective bar drops by almost three log-odds per source. Curves are the exact values of Equation 3 at τ′=1τ =1 and prior 0.20.2; nothing is fitted. Remark 2 (Premise (i) is necessary). If readers fire only positively and silence is genuinely uninformative, then D=0D=0, T(n)T(n) is constant, and the count-scale defect vanishes; what remains is only the weighting question of Proposition 3. So the claim is not that vote counting is always wrong; it is wrong precisely when non-assertion carries information — which in our deployment it does, and strongly, because a single-label reader that named a competing label was asked about y and declined it (Section 4.4). Proposition 2 assumes homogeneous rates, which our deployment does not have: its weights are bucket-dependent by design. The heterogeneous case is the one that matters in practice, and it is worse. Proposition 3 (Heterogeneous buckets: misweighting on top of drift). Let bucket b have rates (αb,βb)( _b, _b) with αb>βb _b> _b, write Ab=logαbβbA_b= _b _b, Db=log1−αb1−βbD_b= 1- _b1- _b, and gb=Ab−Db>0g_b=A_b-D_b>0. Under premises (i) and (i) of Proposition 2, logitPr[y∣ℰ]=logitπ(y)+∑i=1nDbi+∑i:y∈Y^igbi.logit [y ]=logitπ(y)+ _i=1^nD_b_i+\!\! _i:\,y∈ Y_i\!\!g_b_i. (4) Consider the vote rule V≥τ\V≥τ\ with V=∑i:y∈Y^iwbiV= _i:y∈ Y_iw_b_i and wb>0w_b>0. (a) The vote rule agrees with some posterior threshold rule on every instance if and only if wb=cgbw_b=c\,g_b for a single constant c>0c>0: weights must be proportional to the per-bucket log-odds gain, not merely ordered like it. (b) When they are, the induced posterior threshold is T=logitπ(y)+∑iDbi+τ/cT=logitπ(y)+ _iD_b_i+τ/c, which varies across instances through the bucket composition of their sources — recovering Equation 3 when all buckets coincide. (c) When they are not, no threshold on the posterior reproduces the rule: the two orderings cross. Proof. Equation 4 is Equation 2 with per-source rates, regrouped as before. For (a), sufficiency: if wb=cgbw_b=c\,g_b then V=c∑i:y∈Y^igbi=c(logitPr[y∣ℰ]−logitπ(y)−∑iDbi)V=c _i:y∈ Y_ig_b_i=c(logit [y ]-logitπ(y)- _iD_b_i), so V≥τV≥τ iff the posterior log-odds exceed the T in (b). For necessity, suppose wb/gb>wb′/gb′w_b/g_b>w_b /g_b for some pair b,b′b,b . Then wb′/wb<gb′/gbw_b /w_b<g_b /g_b, so the open interval (wb′/wb,gb′/gb)(w_b /w_b,\;g_b /g_b) is non-empty and contains a rational m/m′m/m with m,m′m,m positive integers. Take instance A consisting of m asserting sources all in bucket b, and ℬB of m′m asserting sources all in bucket b′b , with no silent sources and a common prior. Then mgb<m′gb′m\,g_b<m \,g_b while mwb>m′wb′m\,w_b>m \,w_b : the posterior ranks ℬB above A and the vote ranks A above ℬB. Any τ∈(V(ℬ),V()]τ∈(V(B),V(A)] therefore asserts on the lower-posterior instance and abstains on the higher, which no monotone rule in the posterior does. This proves (c) and the necessity half of (a). ∎ The gap is large at realistic parameters. Take three buckets with (α,β)(α,β) equal to (0.55,0.35)(0.55,0.35), (0.75,0.25)(0.75,0.25) and (0.95,0.05)(0.95,0.05). The corresponding gains are g=0.82g=0.82, 2.202.20 and 5.895.89. Now apply hand-set weights of 0.30.3, 0.30.3 and 1.01.0, of the shape deployed systems use. The weights are correctly ordered, but their ratios to g range over a factor of 2.72.7. Sampling instances with up to eight sources, the emitted set and the withheld set overlap across roughly 1717 units of posterior log-odds. The rule is therefore not a noisy approximation to a posterior threshold; it is a different ordering. Corollary 1 (Scope of the result). Proposition 2 makes no reference to language models, and applies to any decision rule of the form “sum weights over the evidence that fired, emit if the sum exceeds a constant” where the number of evidence channels consulted varies across instances and non-firing is informative. That form includes score-summing rule engines in fraud and security triage, multi-test diagnostic panels scored by counting positives, and multi-signal anomaly detectors that add per-signal scores. In all of these, every channel is evaluated on every instance, so a channel that does not fire has declined rather than been omitted. The result does not automatically extend to retrieval pipelines. There an unretrieved passage is absent rather than silent, so D=0D=0 may hold and the drift need not appear; the misweighting of Proposition 3 still applies. Where the result does apply, the remedy is arithmetic rather than architectural: convert each channel’s contribution to a calibrated log-likelihood ratio before summing, then threshold the posterior. Corollary 2 (Directional bias). Thresholding V asserts at lower posterior on instances with many sources than on instances with few. Hence, at matched coverage, precision under a vote rule should decline in n, while precision under a rule that thresholds Equation 2 should not. Corollary 2 is a falsifiable prediction about the deployed rule, and it is testable without new inference, since the per-source evidence objects are already persisted: stratify instances by source count and compare per-bin precision against per-bin prevalence. We have not run it. We record one design constraint for whoever does: label prevalence may itself vary with n, so a prevalence trend would mimic or mask the predicted precision trend, and the comparison must be against the prevalence-adjusted expectation rather than a flat line. 4.3 Per-bucket reliability estimation For each label y and bucket b we estimate αb,y _b,y =Pr[y∈Y^i∣y∈Y,bi=b], = [\,y∈ Y_i y∈ Y,\;b_i=b\,], βb,y _b,y =Pr[y∈Y^i∣y∉Y,bi=b], = [\,y∈ Y_i y∉ Y,\;b_i=b\,], (5) smoothed with a Beta(1,1)(1,1) prior, with rare labels pooled into a fallback bucket. These are Dawid–Skene error rates [8] with buckets in place of annotator identities. Estimation data must be disjoint from evaluation data. These rates are not fitted in the deployed system, which uses the hand-set weights of Section 4.1; the only fitted component is the frequency prior π, estimated on the 129 calibration instances and evaluated on the disjoint 33 (Section 6.2). Were the rates fitted, the same partition would be required: our labels come from a single curated set, so estimating α,β,ρα,β,ρ and evaluating on the same instances would contaminate every figure. 4.4 Single-label readers need a confusion matrix, not a rate pair When |Y^i|≤1| Y_i|≤ 1 the reader is a multinomial channel, and its error structure is a confusion matrix rather than a pair of per-label rates. Pooling each label independently from (αb,y,βb,y)( _b,y, _b,y) alone treats “source i named a competing label y′y ” identically to “source i was silent”, when in fact the first is stronger evidence against y than the second. Define, per bucket, Mb(y′∣⋅)=Pr[Y^i=y′∣⋅,bi=b],y′∈∪∅,M_b(y ·)= [ Y_i=\y \ ·,\;b_i=b], y ∪\ \, (6) where ∅ denotes the reader’s “insufficient evidence” output and ⋅· is the conditioning event y∈Yy∈ Y or y∉Yy∉ Y. Source i contributes to label y ℓi(y)=logMbi(yi′∣y∈Y)Mbi(yi′∣y∉Y), _i(y)= M_b_i(y _i y∈ Y)M_b_i(y _i y∉ Y), (7) positive when yi′=y _i=y, typically negative when yi′y _i is a competing label, and near zero when yi′=∅y _i= is uninformative about y. A full matrix costs O(||2)O(|Y|^2) parameters per bucket, which our label frequencies do not support. We therefore coarsen the emission to three outcomes per target label: asserts y, asserts some other label, or abstains. This costs four parameters per (bucket, label), two free per conditioning event, against two for the rate pair. It preserves the distinction that matters. Equation 8 below is the special case in which the middle outcome is merged into silence. Neither variant produced our results: the deployed rule counts weighted assertions and treats every non-assertion, competing or silent, as a non-event, which is the coarsest of the three. 4.5 Log-likelihood-ratio pooling Source i contributes ℓi(y)=logαbi,yβbi,y,y∈Y^i,log1−αbi,y1−βbi,y,y∉Y^i, _i(y)= cases _b_i,y _b_i,y,&y∈ Y_i,\\[7.74998pt] 1- _b_i,y1- _b_i,y,&y∉ Y_i, cases (8) so informed silence is evidence against y rather than neutral. Pooled posterior log-odds are logitp^(y)=logitπ(y∣z)+∑j=1mκj∑i∈ℬjℓi(y),logit p(y)=logitπ(y z)+ _j=1^m _j\!\! _i _j\!\! _i(y), (9) where ℬ1,…,ℬmB_1,…,B_m partition sources into dependence blocks and κj _j discounts within-block accumulation (Section 4.6). Proposition 4 (Exactness). Suppose (i) assertions are conditionally independent given the label; (i) the rates αb,y,βb,y _b,y, _b,y are exact; (i) κj=1 _j=1; and (iv) the bucket assignment is ancillary, meaning Pr[bi=b∣y∈Y]=Pr[bi=b∣y∉Y] [b_i=b y∈ Y]= [b_i=b y∉ Y] for all b. Then Equation 9 equals the true posterior log-odds of y. Proof. The observation from source i is the pair (bi,[y∈Y^i])(b_i,I[y∈ Y_i]), whose likelihood factors as Pr[bi∣⋅]Pr[y∈Y^i∣⋅,bi] [b_i ·] [y∈ Y_i ·,b_i]. Under (iv) the first factor cancels in the ratio, leaving exactly ℓi(y) _i(y) of Equation 8; summing over conditionally independent sources and adding the log-prior-odds is Bayes’ rule [15]. ∎ Remark 3 (Premise (iv) is not free, and we likely violate it). Conditioning on the bucket discards the evidence carried by the bucket assignment. When buckets are not ancillary the exact posterior carries an additional term ∑ilogPr[bi∣y∈Y]Pr[bi∣y∉Y] _i [b_i y∈ Y] [b_i y∉ Y], and Equation 9 is a lower-variance approximation rather than an identity. Source length is plausibly ancillary; rationale presence is plausibly not, since a reader is likelier to produce a grounded rationale when a genuine reason exists — which makes a high bucket itself weak evidence for the label. The direction of the resulting error is then conservative: pooling understates the evidence for y. Estimating the two bucket-assignment distributions is cheap on the same calibration fold, and we regard the omission as an approximation to be measured rather than a licence. In words: start from the prior odds, add one term per source, and read off the result. A source that asserts y contributes a positive term, sized by how much more often its bucket asserts y correctly than incorrectly. A source that stays silent contributes a negative term. A source from an uninformative bucket contributes close to zero, without anyone having to choose a weight for it. Three consequences follow. Corroboration is automatic — two independent asserting sources contribute twice the log-evidence — so no bonus term and no λ is needed, and Proposition 1 does not apply. The output is a probability, so one threshold is comparable across instances with different source counts and different bucket compositions, addressing Proposition 2 and Proposition 3. The prior enters on the correct scale, so no mixing coefficient is needed; where the outcome has not resolved or prior contamination is a concern, the prior term is dropped and the pooled sum read as a likelihood ratio. 4.6 Block-structured dependence discount A single global discount penalizes genuinely independent sources in large instances. We instead partition sources into blocks sharing an upstream origin (same author, same document lineage, or adjacency under the causal-context chain) and discount within block: κj=11+ρ(|ℬj|−1), _j= 11+ρ\,(|B_j|-1), (10) with ρ∈[0,1)ρ∈[0,1) the average residual within-block correlation of assertions given the label. With ρ=0ρ=0 this reduces to independent pooling; as ρ→1ρ→ 1 each block contributes the evidence of a single source. We fit ρ by maximizing calibration-fold log-likelihood of the pooled posterior. We claim no exactness for this correction and it should not be read as one. It is a design-effect discount, in the spirit of an effective sample size: exact at the two endpoints, an interpolation in between, and not derived from any specified within-block dependence model. Its virtue is that it has one interpretable parameter fitted on held-out data, and it shrinks the pooled evidence toward the prior in both directions — a mostly silent block moves the posterior less far down, which is the intended behaviour and not a sign error. As with the rest of this section, ρ is unfitted here and the blocking rule untested; whether per-block discounting earns its complexity over a single global discount is open. 4.7 Relation to noisy-OR: a different link, not a special case Noisy-OR [29] is the other natural rule for this setting and is widely used for it, so the relationship is worth stating precisely. Under noisy-OR, each asserting source i is treated as an independent sufficient cause. Source i succeeds with probability qiq_i, and y holds if any source succeeds: p^(y)=1−∏i:y∈Y^i(1−qi),equivalentlylog(1−p^(y))=∑i:y∈Y^ilog(1−qi). p(y)=1-\!\! _i:\,y∈ Y_i\!\!(1-q_i), (1- p(y) )=\!\! _i:\,y∈ Y_i\!\! (1-q_i). (11) It is tempting — and we asserted as much in an earlier draft — to call this a special case of Equation 9. It is not. Evidence accumulates additively in different parameterizations: noisy-OR is additive in log(1−p^) (1- p), a complementary-log link, whereas Equation 9 is additive in logitp^logit p. The two families intersect only trivially. A counterexample suffices: with qi=0.5q_i=0.5 throughout, Equation 11 gives posterior log-odds 0,1.099,1.946,2.7080,1.099,1.946,2.708 for 1,2,3,41,2,3,4 asserting sources, whose successive increments 1.099,0.847,0.7621.099,0.847,0.762 are not constant, while Equation 9 increments by exactly log(α/β) (α/β) per asserting source; no choice of prior and rates reproduces the sequence. The substantive difference is saturation. Noisy-OR treats each assertion as a sufficient cause, so its increments shrink as the posterior approaches one and it can never be driven to certainty by volume alone — attractive when readers are unreliable in unmodelled ways. Log-odds pooling treats each assertion as a likelihood ratio, so increments are constant and k independent assertions can reach any confidence, which is correct under its premises and dangerous when they fail. Neither dominates: they encode different beliefs about what a source is. What Equation 9 does offer over Equation 11 is not generality but scope — it admits negative and competing evidence, false-positive rates, an explicit prior, and a dependence discount, none of which Equation 11 expresses. 4.8 Assumption ledger Table 1 collects every result in this section with the assumptions it needs and the deployment condition that voids it. We include it because the results are only useful to someone else if the conditions under which they fail are as legible as the statements. Table 1: Assumptions and failure conditions for the results of this section. Conditional independence given the label is assumed by every result except Proposition 1, and is the premise our own causal-context design violates by construction — which is what the dependence discount is for. Result Needs Fails when Proposition 1 (bonus is redundant) Uniform weights; h increasing; λ≥0λ≥ 0 Weights heterogeneous — then the bonus does change decisions, though with no principle fixing the exchange rate Proposition 2 (count-scale drift) Informed silence; homogeneous rates; conditional independence Positive-only firing (D=0D=0); or heterogeneous buckets, where Proposition 3 governs Proposition 3 (misweighting) Informed silence; conditional independence Weights already proportional to gbg_b; or one bucket only Corollary 1 (scope) Every channel consulted per instance Channels omitted rather than silent, as in retrieval Corollary 2 (precision declines in n) The above, plus prevalence not varying with n Prevalence trends with n, which must be measured, not assumed Proposition 4 (exactness) Conditional independence; exact rates; κ=1κ=1; ancillary buckets Any of the four; bucket ancillarity is the one we expect to fail (Remark 3) κj _j discount Nothing — it is a design-effect heuristic Always approximate away from ρ∈0,1ρ∈\0,1\; no exactness is claimed 4.9 Abstention as deployed Abstention in the running system is a single one-sided threshold: a label whose summed weight falls short of 1.01.0 is not emitted, and an instance with no label above threshold produces no committed determination. There is no reject band and no calibrated p^(y) p(y) to threshold, so the natural generalization of the proposal in this section — assert above τhi _hi, reject below τlo _lo, abstain between — is not what runs. We report no risk–coverage curve. That absence is not incidental, because the evaluation population excludes every instance whose ground truth is “no determinable reason” (Section 6.2), so the data needed to measure what abstention buys has been filtered away upstream of any curve we could draw. We therefore make no quantitative claim for the abstention mechanism at all, and treat the coverage question as open rather than as answered by an unreported experiment. 5 Experimental setup 5.1 Reader configuration The reader is a hosted instruction-tuned language model accessed through an internal gateway, at temperature 0.10.1 with a 4,000-token output cap and one sample per source: no self-consistency, so the per-source variance we would need to separate reader noise from aggregation error is not measured. The prompt is pinned to a dated version identifier and the aggregation is re-validated when that identifier changes. Context budgets were set from the corpus length distribution. The target source is capped at 55,000 characters, its 95th percentile being 54,718. The two immediately prior sources are capped at 35,000 characters each, roughly the 85th percentile per neighbour. Older prior sources enter only as summaries and key points, under per-field caps of 700 and 2,500 characters within a 12,000 character total. At most 25 sources are processed per instance. Execution uses 25 concurrent workers with up to three retries; the daily job reads a four-day rolling window and checkpoints for crash recovery. 5.2 Reproducibility The corpus is proprietary operational data and cannot be released, so external replication of the numbers is impossible by construction. Everything else can and should be released. That includes the prompt templates and their version identifier, the reader model and decoding settings, and the bucket-construction rule. It also includes the estimation code for α,β,ρ,πα,β,ρ,π, the fold assignment, the aggregation and evaluation code, and seeds. Finally it includes the metric definitions, particularly instance F1F_1, which is not micro-F1F_1. We also state the compute: one reader call per source, no fine-tuning, and a tree model trained in minutes on a single machine. Reader model, decoding settings, context budgets and concurrency are stated in Section 6’s configuration subsection; prompt templates, the fold assignment, and seeds are not yet included. A venue that requires a reproducibility checklist will accept a proprietary corpus and will not accept missing prompts. 5.3 Scope of the comparisons we report Both instantiations vary one axis and hold the rest fixed. Instantiation I varies how sources are read — jointly, independently, with or without causal context — while the aggregation rule stays the deployed weighted vote throughout. Instantiation I varies the model architecture and feature set while the reader stays fixed. No experiment in this paper varies the combination rule, so nothing here separates the aggregation proposals of Section 4 from the deployed vote, and we draw no such comparison. 5.4 Statistical protocol All confidence intervals are bootstrap intervals resampling instances, not rows [9]. This matters because the forecasting test set contains many snapshots per instance, so row-level resampling would understate variance by treating correlated snapshots as independent. Paired comparisons between configurations use the same instance resamples. 6 Instantiation I: the resolved regime 6.1 Task and metrics Given the full history attached to a resolved instance, identify which of a fixed reason set applies. The task is multi-label. We report micro-pooled precision and recall over (instance, label) pairs; instance F1F_1, computed per instance over predicted and true label sets then averaged across instances; and macro F1F_1 over labels. Instance F1F_1 is not the harmonic mean of pooled precision and recall, which is why those columns need not reconcile arithmetically. 6.2 Evaluation population, and what it excludes The curated pool comprises 1,000 labelled instances, of which 991 have attached sources and 162 survive eligibility filtering; the split is 129 instances for prior and rate estimation and 33 for evaluation, stratified by outcome. Two facts about that filter must be stated before any number is read. First, all differences in Table 2 rest on 33 instances. At that size a 0.05 gap in instance F1F_1 is not separable from sampling noise, and macro F1F_1 over long-tail labels is less stable still. We therefore present Table 2 as a descriptive case report: at this sample size the observed ordering of configurations should not be interpreted as statistically established. The three-decimal precision in the source material is retained only for traceability against the internal log. Second, the filter removes every instance whose gold label is “not determinable”, precisely the stratum on which abstaining is the correct output. Precision measured after removing it is therefore conditioned on a determinable reason existing, so this evaluation does not measure the selective-prediction benefit of abstention. A complete risk–coverage evaluation should be run on the unfiltered pool with abstention scored as a prediction of “not determinable”; that analysis remains future work. Reader selection contaminates the audit set. The reader model was chosen by F1F_1 on the “audit” golden set (0.560 across candidates). That set is therefore not an independent test set for the reader, and we report the “free-choice” set here. Both should be reported with the selection dependence stated, rather than only the one not used for selection. Label contamination guard. The structured outcome-reason field maintained by human operators is excluded from the reader’s prompt, so the reader cannot copy the label it is being scored against. We note this because it is the first thing to check and the easiest to omit. 6.3 Compared systems Internal variants: (1) prior only; (2) one-shot concatenation; (3) independent, no context; (4) independent, full context (oracle); (5) independent, causal context; (6) causal ++ prior fallback. All six are internal variants; we ran no external baseline. The comparisons a reviewer would reasonably require — self-consistency over one-shot samples [36], unsupervised Dawid–Skene [8], and a one-shot reader instructed to emit citations [12] — are absent, and Table 2 should be read as an internal design comparison rather than as a positioning against published methods. 6.4 Results Table 2: Reading strategies, aggregation held fixed. n=33n=33 evaluation instances (Section 6.2); bold marks the numerically best value in a column and implies no significance, since instance-level intervals at this n overlap for every pair. Instance F1F_1 is a tie between one-shot and the recommended configuration, and both are marked. †Oracle: uses future sources, not deployable, and not length-matched to the causal condition (see Section 6.5). Method Precision Recall Instance F1F_1 Macro F1F_1 Prior only 0.697 0.199 0.303 0.114 One-shot 0.645 0.607 0.614 0.425 Independent, no context 0.806 0.418 0.520 0.385 Independent, full context† 0.816 0.439 0.534 0.392 Independent, causal context 0.824 0.459 0.587 0.416 Causal ++ prior fallback 0.772 0.554 0.614 0.466 We do not report inter-annotator agreement on the curated set, instance-level bootstrap intervals, or the source-count distribution of the evaluation fold. The first is unavailable, the other two we did not compute; together with n=33n=33 this is why we treat the table as descriptive. Independent causal reading is the most precise configuration, consistent with joint reading letting weak sources dilute strong ones. Recall is lower; the prior fallback recovers most of it, raising macro F1F_1 from 0.416 to 0.466 while keeping precision above one-shot. We note what the table does not show: on instance F1F_1 the recommended configuration and plain one-shot concatenation are identical at 0.614. The case for independent reading in this study rests on precision, on per-source provenance, and on cost behaviour, not on aggregate F1F_1, and the abstract should not be read as claiming otherwise. 6.5 The oracle anomaly is confounded The oracle loses to causal context on instance F1F_1 (0.534 vs. 0.587) despite strictly more information. We flag this as confounded rather than explained: the two conditions differ in both the availability of future evidence and total context length, so at least two mechanisms are consistent with the result. Hindsight contamination would have the reader attribute to an early source a reason legible only later, inflating false positives. Context dilution would degrade attention to the source under evaluation [26]. A third possibility is that the effect is within noise, which the missing intervals cannot currently exclude. We ran no arm that separates them, so we assert no mechanism; a length-matched control that varies only the informativeness of the extra context would be the way to. What the comparison does support. All three mechanisms are consistent with an absence of penalty rather than a gain, and that weaker reading is well supported: on every metric in Table 2 the causal condition is at least as good as the oracle (precision 0.824 vs. 0.816, recall 0.459 vs. 0.439, instance F1F_1 0.587 vs. 0.534, macro F1F_1 0.416 vs. 0.392). The claim we make is therefore the negative one, which is also the one deployment needs: restricting the reader to information available at the time each source arrived — the only condition under which the system can run before an outcome is known — cost no measurable accuracy here. Whether the apparent advantage is real, and if so why, is what remains open. At n=33n=33 the third possibility is not a formality. A 0.053 difference in instance F1F_1 corresponds to roughly two instances changing their label sets. We would not report the anomaly at all except that it is counterintuitive enough that omitting it would be worse; it is stated here so that a larger curated set can confirm or dissolve it. 7 Instantiation I: the unresolved regime The object of study does not change here; its epistemic status does. The same instances, sources, and reader carry over, but the outcome has not occurred, so no label exists to attribute, no empirical prior is available, and future sources are not merely disallowed but nonexistent. Under the fourth axis of Section 3 this is the only coordinate that moves, and it moves the partition with it: what must be separated is no longer reading from combining but representation learning from censored timing estimation. The interface survives the change of granularity — the reader’s tuples enter this model as features — while the combination mechanism is replaced by one suited to the new sub-problem. If the principle were an artifact of the first task, it would not survive that substitution. 7.1 Formulation A binary “will it succeed” classifier conflates two negatives: instances that fail and instances merely unresolved. We use a discrete-time competing-risk hazard model [11, 24] with K=7K=7 buckets, at 7, 14, 30, 60, 90, 180 and 365 days. Bucket k emits a softmax triplet over \success, failure, survive\ conditional on survival to its start, giving 21 logits. Horizon probabilities chain: Pr[success by T]=∑k:tk≤T(∏j<khjsurv)hksucc. [success by T]= _k:t_k≤ T ( _j<kh^surv_j )h^succ_k. (12) Censored instances contribute only survival terms, so no synthetic label is imposed on unresolved cases. 7.2 Snapshots and leakage control Each source arrival creates a snapshot built only from then-available information; an instance with five sources yields five progressively better-informed rows. The duration label is remaining time to resolution, not total lifetime. Because days_since_last_source is identically zero on source-anchored rows, a model trained only on those rows would ignore staleness at inference, when it is nonzero. We insert synthetic midpoint rows carrying the preceding features with days_since_last_source=gap/2 days\_since\_last\_source=gap/2. Evaluation uses source-anchored rows only. 7.3 Capacity-partitioned hybrid A single-layer GRU [4] with 64 hidden units (≈ 25K parameters) reads the source sequence, trained on an easier auxiliary three-class objective (success/failure/censored) with class-weighted cross-entropy. Its per-timestep hidden states are concatenated with 17 hand-crafted features (11 numeric: cadence, staleness, counts, instance size; 6 categorical: 3 reader-extracted signals from Instantiation I, 3 structured metadata) and passed to a gradient-boosted tree [3] with multi-output trees over all 21 logits and a custom competing-risk negative log-likelihood handling censoring. The hypothesis: at median 2–3 sources per instance and ≈ 24K effective sequences, no single model should learn sequence representations and censored timing structure at once. 7.4 Metrics Ranking: AUPRC at each horizon, appropriate at our prevalence [7, 32]. This is our only primary metric. We report MAE in days as a secondary, descriptive quantity; under censoring and a resolution-filtered test set it is not defensible as a primary metric, and we do not use it to rank configurations. We compute neither time-dependent concordance [2, 19] nor the IPCW Brier score [16], which is a real gap for a survival model: without them we report discrimination at fixed horizons and nothing about calibration over time. 7.5 Setup 73,638 sources across 23,553 instances over twelve steady-state months. Training and data cutoffs are separated by three months. Train: 33,840 snapshots / 10,541 instances. Test: 69,070 snapshots / 11,928 instances. Prevalence at 90 days is 4.2% success, 6.4% failure. Population bookkeeping requires care because different analyses use different eligible subsets. Ranking metrics are reported over a resolved population of 14,524 instances plus 2,701 censored-known, drawn from the full scoring window rather than from the 11,928-instance test split; the length-stratified table below uses a further subset of 4,874. We therefore do not compare figures across these populations, and Section 11 records the remaining reconciliation work explicitly. 7.6 Ablation Table 3: Identical training data and held-out temporal split. Δ is change in AUPRCs_s vs. row 2. ‡Prevalence on the resolved population used for ranking metrics, not the unconditional 90-day prevalence of Section 7; this is the correct no-skill reference for every row here. §Row 7 is not feature-matched to row 6 and therefore does not isolate architecture, and no feature-matched neural arm was run. ¶Feature count unreconciled: the enumerated set is 11 numeric ++ 6 categorical =17=17, and row 6’s 81 =64+17=64+17 agrees, but this row is logged as 18 and rows 3–4 as 25 and 34; the offset propagates and must be fixed. As in Table 2, bold marks the numerically best value in a column and implies no significance: no column reports instance-level intervals. MAE is descriptive and is not bolded, since we do not use it to rank configurations. # Configuration Feat. AUPRCs_s AUPRCf_f F1F_1 MAE (d) Δ 0 Random (prevalence)‡ 0 0.185 0.171 0.270 80.9 −-0.620 1 Cadence only 5 0.505 0.327 0.428 53.3 −-0.300 2 Hand-crafted (baseline) 18¶ 0.805 0.667 0.700 50.6 — 3 ++ conversation quality 25 0.805 0.675 0.702 50.6 ++0.000 4 ++ all structured engagement 34 0.804 0.671 0.704 50.6 −-0.001 5 GRU ++ cadence 69 0.833 0.679 0.725 51.8 ++0.028 6 Hybrid 81 0.921 0.779 0.744 48.0 ++0.116 7 End-to-end neural§ 13/src 0.316 0.315 0.147 98.0 −-0.489 Four observations, with one important qualification. Reader-extracted signals dominate surface features. Cadence-only to hand-crafted adds 0.300 AUPRC, carried by the reader-extracted signals and structured metadata. Two of those categoricals are exposed to semantic leakage, so the attribution of this increment is unidentified (Section 7.7). Surface engagement metrics add nothing. Seven conversation-quality features add 0.000; sixteen structured engagement metrics are marginally negative. We read these as crude proxies for a signal the reader already captures semantically — an argument against expanding surface feature sets once a semantic reader is in place. Sequence dynamics subsume much of the hand-crafted signal. The encoder on cadence inputs alone beats the 18-feature baseline (row 5, 0.833, against row 2, 0.805); combining the two adds a further 0.088 (row 6, 0.921), so extracted content contributes residual rather than redundant signal. Rows 2 and 5 also differ in feature count, so this comparison is suggestive rather than clean. The end-to-end neural configuration underperforms, but the comparison is not yet controlled. Row 7 reaches 0.316 AUPRC with 98-day MAE, early-stopping at epoch 19. This is consistent with the capacity-partition hypothesis and with the tabular-learning literature [17, 34]. It does not yet establish the hypothesis, because row 7 is not feature-matched to row 6: it sees 13 raw per-source features with no cadence features on the survival head, against 81 for row 6. We therefore describe the capacity partition as a hypothesis consistent with a suggestive but confounded comparison, and we do not claim it as established. 7.7 Leakage audit Ranking metrics are computed on the resolved population, whose no-skill AUPRC is 0.185 (row 0 of Table 3); the headline 0.921 is a five-fold lift over that baseline, not over the unconditional 4.2% 90-day prevalence of Section 7, which is a different population. Point-in-time snapshot construction rules out temporal leakage but not semantic leakage. Two features are semantically exposed. Pipeline stage. The categorical stage is a human-maintained field that operators advance as an instance approaches resolution. Its value at a snapshot is available at that moment, but on late snapshots it is close to a deterministic announcement of the outcome. The reader’s own reason label. The categorical dominant_reason is the reader’s judgement of why this instance resolves as it does, extracted from source text that on late sources frequently states the resolution outright. Used as a forecasting feature, it may partially encode outcome information present in late-stage evidence rather than information available for genuine anticipation. Bound from the ablation. Both exposed features sit in the hand-crafted block, so rows 1 and 5 carry neither. The ablation also shows that strong ranking performance is achievable without either exposed feature: row 5, the sequence encoder on cadence inputs with no categoricals, reaches 0.833 AUPRC, compared with 0.805 for row 2, which includes both. This indicates that the headline performance cannot be attributed solely to those exposed features, although their exact incremental contribution remains unidentified: rows 2 and 5 differ in architecture and feature count, so this comparison does not isolate either feature’s marginal effect. The 0.088 increment from row 5 to row 6 is the gain associated with adding the hand-crafted block in the hybrid configuration; that block contains six numeric features, four unexposed categoricals, and the two exposed ones, so the contribution of the exposed pair within that increment is likewise not identified. The capacity-partition result rests on rows 2, 5 and 6. The encoder’s per-source inputs contain neither exposed field. What remains open. We ran no ablation dropping either feature and no fixed-lead-time evaluation. The +0.300+0.300 increment from row 1 to row 2, the step at which the categoricals enter, is therefore unidentified between semantic content and outcome encoding, as are the reported operating points. Prediction 5 (Section 10) is the test. The sequence-length trend in Table 4 carries the same ambiguity: instances with more sources are also further along. 7.8 Scaling with sequence length Accuracy rises monotonically with the number of sources observed at scoring time (Table 4): AUPRC on the success horizon climbs from 0.795 in the one-to-two-source bin to 0.940 at thirteen or more, a gain of 0.145, while the descriptive close-time error falls from roughly 59 to 39 days. Two readings of this are consistent with the data and we cannot separate them here. The sequence encoder may be extracting more from longer histories, which is what the capacity partition predicts; or instances with more sources may simply be closer to resolution, which is the identification problem raised in Section 7.7. Worth noting either way is the low end: even single-source instances score 0.795 against the 0.185 no-skill baseline on this population, so the model is not merely useless until evidence accumulates. Table 4: Accuracy as sequential evidence accumulates. The monotone trend is consistent with the sequence-encoder rationale, but it is not identified: instances with more sources are also further along, so proximity to resolution is an unmodelled competing explanation (Section 7.7). Instance counts in the upper bins are small and intervals are pending. The 1–2 bin count (2,830) coincides exactly with the population used for the operating-point table, which we flag as a probable provenance error to be checked rather than a coincidence. Sources observed Instances AUPRCs_s AUPRCf_f F1F_1 MAE (d) 1–2 2,830 0.795 0.622 0.699 59.4 3–4 1,043 0.802 0.680 0.696 51.9 5–7 571 0.827 0.697 0.718 42.5 8–12 304 0.862 0.788 0.765 38.5 13++ 126 0.940 0.883 0.849 39.0 7.9 Score stability On a 200-instance sample with at least five sources each, the median within-instance range of the 90-day success probability is 0.104 and the median standard deviation 0.030, with a P90 range of 0.860. Most instances are stable; the tail corresponds to genuine trajectory change. A between-source drop exceeding 0.20 is our operating threshold for flagging review. 7.10 Negative results Post-hoc calibration failed under temporal shift. Isotonic regression [40] fitted on the training split and applied to test degraded AUPRC by 0.009. The mechanism is base-rate shift across the temporal split: the calibrator learned a mapping that does not hold in the test period. An earlier internal explanation attributed the observed under-confidence to probability mass being distributed over seven buckets and three outcomes. The current analysis does not support that interpretation. The chained 90-day marginal is a probability and should be calibrated against realized frequencies; distributing mass across buckets does not by itself explain marginal miscalibration. The reported deciles — decile 5 predicting 0.02 against 9% realized — indicate genuine miscalibration of the marginal. The defensible conclusion is narrow: isotonic regression fitted across a temporal shift did not help, and we consume scores as rankings pending a shift-aware calibration attempt. We report no reliability diagram or expected calibration error per horizon, and we did not retry calibration on a temporally adjacent validation slice rather than the full training split. The negative result therefore covers one calibration method under one fitting protocol, and should not be read as evidence that the scores cannot be calibrated. Down-weighting synthetic midpoint rows did not help. No weight setting improved AUPRC, suggesting midpoints supply genuine staleness signal rather than noise. Pooling beat per-segment models. Splitting halves already-limited data for the weakest segments (≈ 3K training snapshots each), and the pooled model transfers patterns from high-volume segments. Per-segment AUPRC ranges 0.723–0.969, but the widest intervals fall on the smallest segments (20–86 instances) and we do not interpret those as genuine outperformance. 7.11 Deployment Per source arrival the system emits horizon probabilities for each competing outcome at 30 and 90 days, an expected days-to-resolution estimate, and the model version. Rows are keyed by (source, instance) so trajectory and attribution questions are answerable, with instance-level views built on top. Separate success and failure scores are maintained rather than one signed score, since an instance can be unlikely to resolve either way. Raw scores are stored rather than tiers, so thresholds move without redeployment. Operating points are conditioned on resolution. Threshold-level precision and recall in our internal reporting are computed over instances that resolved within the test window, excluding roughly 9,098 unresolved ones. Because the conditioning event is unknown at scoring time, these figures are conditional operating-point estimates rather than estimates of deployed precision and should not be quoted as headline accuracy. Against unconditional prevalence the conditioned top-tier figures are not attainable; against the resolved-population base rate they correspond to roughly a two-fold lift. We have not recomputed them on the full scored population, so we quote no operating point as a headline figure and state the conditioning wherever one appears. 8 Transfer The contract is domain-neutral by construction: it constrains what a reader emits, not what a reader is. Table 5 instantiates it along the four axes of Section 3 for five domains outside ours. Each was chosen because it already runs an aggregation rule of the form Corollary 1 describes. Table 5: Instantiating the contract along the four axes of Section 3. Each domain aggregates multiple partially reliable signals under an obligation to justify the conclusion, and in several of them the conventional method is to sum scores against a fixed cut — the form Corollary 1 shows to be a drifting posterior threshold. Domain Sources Reliability bucket from Dependence blocks Regime Clinical decision support Notes, labs, imaging reports, prior visits Modality and assay precision; report completeness Same encounter; same instrument; copied-forward text Both Security triage Detector alerts, host telemetry, threat intel Detector historical FPR; alert enrichment depth Shared sensor, shared rule lineage Unresolved Financial analysis Filings, transcripts, analyst notes Document class and recency; quantitative grounding Same issuer-period; syndicated language Both Evidence synthesis Published studies of varying quality Design and risk-of-bias tier; sample size Same cohort; same research group Resolved Industrial telemetry Per-signal detectors over a shared feature build Signal type; data-quality flag Shared upstream pipeline or reference series Unresolved Three things transfer without re-derivation. The contract transfers because it is a schema. The pooling arithmetic transfers because Equation 9 is Bayes’ rule under a stated independence structure, not a fitted model. And the drift diagnostic transfers because it needs only what these systems already log: per-instance channel counts, which channels fired, and outcomes. Three things do not, and treating them as if they did is the likely failure mode of adopting this work. Reliability rates αb,y,βb,y _b,y, _b,y are properties of a reader on a corpus and must be re-estimated whenever either changes, including on prompt revisions. The blocking rule is domain knowledge: what counts as shared upstream origin is a fact about the data pipeline, not a modelling choice. Thresholds encode the local cost of a wrong assertion against the cost of no answer, and no default is defensible across domains. The generalization claim is therefore precise: the decomposition and its arithmetic transfer; the constants never do. Two properties of the domains in Table 5 deserve note, because they are what make the framework worth the trouble there rather than merely applicable. Each has an obligation to show its basis — clinical, regulatory, or evidentiary — so provenance as a structural output is worth more than post-hoc explanation. And in each, no answer is a legitimate and sometimes preferred output, which is exactly what a generative reader cannot reliably produce and a rule over calibrated evidence can. 9 Discussion The two instantiations make one point at two granularities. In the resolved regime, splitting monolithic reading into per-source interpretation plus explicit aggregation improved precision and made provenance structural. In the unresolved regime, splitting a monolithic learning problem into representation learning plus censored timing estimation coincides with a large margin, on a comparison we have not controlled. The recurring structure is that the harder sub-problem in each case has a specialized solver — calibrated arithmetic in one, a censoring-aware boosted tree in the other — and that giving it to a general-purpose learner along with everything else wastes it. We should be precise about how much the two studies share. The second consumes the reader from Instantiation I as three categorical features, but not the aggregator: no LLR pooling, no abstention, no dependence discount enters the forecasting model. The unification is therefore architectural rather than demonstrated. The experiment that would demonstrate it would be to supply the pooled posterior p^(y) p(y) from Equation 9, with its abstention flag, as a feature to the forecasting model and to measure whether calibrated evidence outperforms raw extracted categories. Absent that, this paper contains two adjacent contributions sharing a corpus and a reader, and we would rather say so than assert a unification. The architecture separates two uncertainties that one-shot systems conflate. Source uncertainty — one ambiguous observation — surfaces as a low reliability bucket. Aggregate uncertainty — plausible observations conflicting — surfaces as a posterior in the abstention band. A system can abstain at either level and report which. Why not conformal prediction? Conformal methods [1] would give distribution-free coverage guarantees on our label sets, which our threshold pair does not. We did not adopt them because exchangeability fails across our temporal split — the same shift that defeated isotonic calibration — and because conformal sets do not by themselves supply the per-source attribution that motivates the architecture. Time-series conformal variants are the natural next step and we regard the absence as a limitation rather than a design choice. A last observation about why the partition keeps paying. Each half, once separated, can be improved by someone who does not understand the other. The reader can be swapped for a better model, or for a human, without touching the arithmetic; the arithmetic can be corrected — as Section 4 argues it should be — without re-reading a single source. Monolithic systems do not have that property, and it is worth more over a system’s life than any single margin reported here. 10 Falsifiable predictions A framework is more useful when it makes claims that could turn out to be false. We state five such claims below. Each is paired with the experiment that would discriminate it, and each can be run on data already collected or with one additional training run. 1. Count-scale drift is practically material, not merely formal. Under the deployed vote rule, precision should decline in source count relative to per-bin prevalence; under posterior thresholding it should not. Stratify the evaluation instances by source count and compare per-bin precision against per-bin prevalence. A flat profile would leave Proposition 2 valid but practically irrelevant at our distribution of n, and we would report it as such. 2. Calibrated pooling beats vote counting on the same tuples. Re-aggregate the stored evidence objects under majority vote, weighted vote, weighted vote with bonus, noisy-OR, unsupervised Dawid–Skene, and Equation 9. No new inference is required. If pooling does not improve macro F1F_1 or selective risk, the analytical argument survives but the engineering recommendation does not. 3. Abstention selects hard cases rather than discarding at random. On the unfiltered pool, with “no determinable reason” scored as a class, the risk–coverage curve should dominate a matched-coverage random abstention control. This is the prediction we most want tested, because the current evaluation population excludes the stratum it concerns (Section 6.2). 4. The capacity partition, not the feature set, drives the forecasting margin. Train the end-to-end neural model on the hybrid’s full feature set. If it closes most of the 0.921 versus 0.316 gap, the margin was features and the partition claim fails. 5. The reader-extracted increment is prediction, not outcome encoding. Re-run without pipeline stage, without the reader-extracted categoricals, and at fixed lead times of 30, 60, and 90 days. The +0.300+0.300 increment of Table 3 should survive at 60-day lead if it reflects semantic content, and collapse if it reflects outcome announcements (Section 7.7). The framework also makes one prediction we consider its most consequential and have no way to test here: that supplying the pooled posterior with its abstention flag as a feature to the forecasting model outperforms supplying raw extracted categories. That experiment is what would make the two instantiations one system rather than two applications of one principle. 11 Limitations The main empirical limitation concerns evaluation coverage in Instantiation I rather than any single number. Eligibility filtering for Instantiation I removed every instance whose gold label is “not determinable”, reducing 1,000 labelled instances to 162 and leaving 33 for evaluation (Section 6.2); the present evaluation therefore does not measure performance on the cases where abstaining is the correct output. At 33 instances no ordering among the rows of Table 2 is statistically supported, the oracle anomaly included. And the aggregation argument of Section 4 is analytical throughout: our results come from the deployed weighted vote, no experiment here varies the combination rule, so we characterize the defect analytically while empirical evaluation of the proposed repair remains future work. In Instantiation I the credit assigned to reader-extracted content is not identified. Pipeline stage and the reader’s own reason label may encode the outcome rather than predict it (Section 7.7), which exposes the +0.300+0.300 increment and the operating points. Strong ranking performance remains without either exposed feature: row 5 of Table 3 carries neither and reaches 0.833, while row 2 carries both and reaches 0.805. Because those rows differ in architecture and feature count, this comparison does not isolate either feature’s marginal contribution; it does, however, show that the ranking result does not depend solely on the exposed pair. The capacity-partition result is carried by rows 2, 5 and 6 rather than by a single exposed feature. Row 7 of Table 3 is separately not feature-matched to row 6, so that arm of the capacity-partition argument rests on a confounded comparison. The no-skill AUPRC of row 0 exceeds the unconditional 90-day prevalence quoted in Section 7 because the two are computed on different populations, the former conditioning on resolution; we have not reconciled the counts exactly, but the gap is one of conditioning rather than an inconsistency, and Δ may be read as lift over chance within the table. Several evaluation choices affect how the reported numbers should be interpreted. Threshold precision and recall are computed over resolved instances only, so they are conditional on resolution and are not directly comparable to deployed precision over all scored instances; ranking metrics over the scored evaluation population do not use those operating-point thresholds, which is why AUPRC is primary. Restricting the test set to instances resolving before the data cutoff over-represents fast-resolving ones, biasing MAE downward and distorting the horizon marginals — a second reason we treat MAE as descriptive. Multiple snapshots per instance make row-level variance estimates optimistic, so intervals must resample instances rather than rows. We report no calibration for either study: the deployed rule does not pool at all, Section 4.4 shows what a single-label reader would require, and in Instantiation I we report ranking without a proper scoring rule over time, so neither system emits a number we would defend as a probability. Consistent with that, summing per-instance probabilities into a quarterly aggregate produced absolute percentage errors ranging from single digits in the best quarter to well above 100% in two others; the model ranks, it does not estimate portfolios. Two limitations belong to the method rather than to our data. Calibrated pooling corrects for known per-bucket error rates, but it cannot correct a reader whose errors correlate with the label, since such error is absorbed into αb,y _b,y and reappears as signal. The rates themselves come from finite calibration data, and for rare labels the log-ratio is unstable as β→0β→ 0 while clipping it biases toward weaker evidence. Finally, everything here rests on one corpus in one domain: the mechanisms are domain-neutral, but the weights, thresholds, and capacity partition are fitted to this regime, and we report no human-in-the-loop outcome, so “decision support” names the system’s role rather than a measured effect on anyone’s decisions. 12 Broader impacts The deployment writes to records about identifiable employees’ work and surfaces risk flags on the deals they own, so it sits inside a performance management context whether or not it is used for one. Three properties matter and two are already enforced. The system writes only into empty fields and never overwrites a human entry; every written value carries a provenance flag marking it machine-generated; and every assertion links to the sources supporting it, so a person can contest a specific inference rather than a score. The property not yet established is the one that matters most: we have run no study of how these outputs affect the decisions people make, so we cannot claim the provenance is used, nor that flags are read as evidence rather than as verdicts. Automation bias would show up as exactly the pattern our abstention design is meant to prevent. The corpus consists of recorded human conversations processed under the recording consent and retention policy in force for that data; the reader sees transcript text and no participant identifiers beyond those already in the transcript, and no personal data is released with this work. We note that a system which explains why a deal was lost is also a system that attributes causes to named individuals’ conduct; label sets should be audited for reasons that are effectively performance judgements. 13 Conclusion We argued that evidence organization is an architectural decision rather than a prompt-construction detail, and that the decision to make is where to cut and what contract to place at the cut. Placing a four-field evidence tuple there determines both halves: reliability becomes estimable, provenance becomes structural, failures localize, and abstention becomes expressible. We then characterized the failure mode in how such systems usually combine. Count-scale drift makes a vote threshold into a posterior threshold that slides linearly in source count, at a rate set by parameters that are rarely estimated. We stated the result generally enough to cover a class of rules beyond language models, and showed that heterogeneous reliabilities make matters worse still, producing a different ordering rather than a drifting one. We then gave the arithmetic that addresses both. We then instantiated the principle on one corpus in both of its epistemic regimes, where it pays at two different granularities: over reading before resolution is known, and over learning capacity when it is not. The constants do not transfer and we have said which. What transfers is the cut, the contract, the pooling arithmetic, and a diagnostic computable on logs that most such systems already keep. Systems that must be inspected and contested need conclusions that arrive with their supporting evidence, its calibrated strength, and the standing option of no conclusion at all; that is a property of an interface, and interfaces are designed, not prompted. References Angelopoulos and Bates [2021] Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021. Antolini et al. [2005] Laura Antolini, Patrizia Boracchi, and Elia Biganzoli. A time-dependent discrimination index for survival data. Statistics in Medicine, 24(24):3927–3944, 2005. Chen and Guestrin [2016] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of KDD, 2016. Cho et al. [2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of EMNLP, 2014. Chow [1970] C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41–46, 1970. Cox [1972] D. R. Cox. Regression models and life-tables. Journal of the Royal Statistical Society, Series B, 34(2):187–220, 1972. Davis and Goadrich [2006] Jesse Davis and Mark Goadrich. The relationship between precision-recall and ROC curves. In International Conference on Machine Learning, 2006. Dawid and Skene [1979] A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, Series C, 28(1):20–28, 1979. Efron [1979] B. Efron. Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7(1):1–26, 1979. El-Yaniv and Wiener [2010] Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11:1605–1641, 2010. Fine and Gray [1999] Jason P. Fine and Robert J. Gray. A proportional hazards model for the subdistribution of a competing risk. Journal of the American Statistical Association, 94(446):496–509, 1999. Gao et al. [2023] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Proceedings of EMNLP, 2023. Geifman and El-Yaniv [2017] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, 2017. Genest and Zidek [1986] Christian Genest and James V. Zidek. Combining probability distributions: A critique and an annotated bibliography. Statistical Science, 1(1):114–135, 1986. Good [1950] I. J. Good. Probability and the Weighing of Evidence. Charles Griffin, 1950. Graf et al. [1999] Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Martin Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine, 18(17-18):2529–2545, 1999. Grinsztajn et al. [2022] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Advances in Neural Information Processing Systems, 2022. Guo et al. [2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. Harrell et al. [1996] Frank E. Harrell, Kerry L. Lee, and Daniel B. Mark. Multivariable prognostic models. Statistics in Medicine, 15(4):361–387, 1996. Ishwaran et al. [2008] Hemant Ishwaran, Udaya B. Kogalur, Eugene H. Blackstone, and Michael S. Lauer. Random survival forests. Annals of Applied Statistics, 2(3):841–860, 2008. Kadavath et al. [2022] Saurav Kadavath, Tom Conerly, Amanda Askell, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. Katzman et al. [2018] Jared L. Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. DeepSurv: Personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Medical Research Methodology, 18(1), 2018. Khot et al. [2023] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In International Conference on Learning Representations, 2023. Lee et al. [2018] Changhee Lee, William R. Zame, Jinsung Yoon, and Mihaela van der Schaar. DeepHit: A deep learning approach to survival analysis with competing risks. In AAAI Conference on Artificial Intelligence, 2018. Lewis et al. [2020] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, 2020. Liu et al. [2024] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024. Manakul et al. [2023] Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of EMNLP, 2023. Min et al. [2023] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of EMNLP, 2023. Pearl [1988] Judea Pearl. Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, 1988. Rashkin et al. [2023] Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. Measuring attribution in natural language generation models. Computational Linguistics, 49(4), 2023. Raykar et al. [2010] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297–1322, 2010. Saito and Rehmsmeier [2015] Takaya Saito and Marc Rehmsmeier. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10(3), 2015. Shafer [1976] Glenn Shafer. A Mathematical Theory of Evidence. Princeton University Press, 1976. Shwartz-Ziv and Armon [2022] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84–90, 2022. Wang et al. [2019] Ping Wang, Yan Li, and Chandan K. Reddy. Machine learning for survival analysis: A survey. ACM Computing Surveys, 51(6), 2019. Wang et al. [2023] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023. Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. Whitehill et al. [2009] Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, 2009. Xiong et al. [2024] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In International Conference on Learning Representations, 2024. Zadrozny and Elkan [2002] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of KDD, 2002.