Paper deep dive
Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement
Yuhe Wu, Rui Qian, Guangyu Wang, Yuran Chen, Yuanchao Zhu, Junjie Yang, Zhengheng Li, Jiulin Cai, Tianyi Zhang, Zihan Dong, Jiaxin Liu, Yujie Chen, Guang Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/24/2026, 4:47:54 AM
Summary
The paper introduces Tempered Evidence Fusion (TEF), a training-free method for measuring public value orientations from long social media posts using Large Language Models (LLMs). TEF addresses the issue of uneven evidence distribution by weighting sentence-level log-odds by their normalized information gain (entropy-based reliability), effectively discounting uncertain sentences while preserving decisive evidence. The authors also present MIND, a benchmark of 8,358 Chinese and English posts across six value dimensions, where TEF demonstrates superior accuracy and calibration compared to direct prediction and voting baselines.
Entities (11)
Relation Signals (11)
Tempered Evidence Fusion → outperforms → Majority Vote
confidence 95% · On MIND, TEF outperforms the strongest baseline among Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy points
Tempered Evidence Fusion → outperforms → Soft Vote
confidence 95% · On MIND, TEF outperforms the strongest baseline among Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy points
Tempered Evidence Fusion → outperforms → Direct
confidence 95% · On MIND, TEF outperforms the strongest baseline among Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy points
MIND → usedforevaluation → Tempered Evidence Fusion
confidence 95% · On MIND, TEF outperforms the strongest baseline... across five LLMs and two languages.
Tempered Evidence Fusion → uses → normalized information gain
confidence 93% · TEF... weights each sentence's log-odds by its normalized information gain, as derived from a generalized Bayesian posterior.
Tempered Evidence Fusion → uses → log-odds
confidence 93% · TEF... weights each sentence's log-odds by its normalized information gain
Tempered Evidence Fusion → evaluatedon → Qwen2.5-7B
confidence 90% · We evaluate three open-source models, Qwen2.5-7B... using... TEF
Tempered Evidence Fusion → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) are increasingly used to measure public value orientations from long social media posts, yet such posts often mix background, quotations, concessions, and only a few stance-bearing sentences. Existing approaches either ask the model to predict a document-level label directly, which can be overconfident, or aggregate sentence-level predictions by majority or soft voting, which treat uncertain and decisive sentences as equally informative. We formulate long-text value measurement as a decision-fusion problem and propose Tempered Evidence Fusion (TEF), a training-free rule that weights each sentence's log-odds by its normalized information gain, as derived from a generalized Bayesian posterior. This makes the fused score nearly vanish for uncertain sentences while preserving the Bayes-optimal weight of decisive evidence. We further introduce Multi-event Insight Network Dimensions (MIND), a benchmark of 8,358 Chinese and English posts spanning five years of public events and six value dimensions. On MIND, TEF outperforms the strongest baseline among Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy points and 4.6 macro-F1 points across five LLMs and two languages. MIND dataset and code are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.27165v1
- Canonical: https://arxiv.org/abs/2609.27165v1
Trouble viewing inline? Open PDF directly →
Full Text
31,323 characters extracted from source content.
Expand or collapse full text
COUNT EVIDENCE, NOT SENTENCES: TEMPERED EVIDENCE FUSION OF LLM JUDGMENTS FOR LONG-TEXT VALUE MEASUREMENT Yuhe Wu 1,∗ , Rui Qian 2,∗ , Guangyu Wang 1,∗ , Yuran Chen 3 , Yuanchao Zhu 4 , Junjie Yang 5 Zhengheng Li 6 , Jiulin Cai 7 , Tianyi Zhang 8 , Zihan Dong 9 , Jiaxin Liu 1 , Yujie Chen 10 , Guang Zhang 1,† 1 HKUST(GZ) 2 FDU 3 DUFE 4 UESTC 5 UMD 6 SEU 7 USTC 8 Independent 9 Georgia Tech 10 CUHK(SZ) ABSTRACT Large language models (LLMs) are increasingly used to measure public value orientations from long social media posts, yet such posts often mix background, quotations, concessions, and only a few stance-bearing sentences. Existing approaches either ask the model to predict a document-level label directly, which can be overconfident, or aggregate sentence-level predictions by majority or soft voting, which treat uncertain and decisive sentences as equally informative. We formulate long-text value measurement as a decision-fusion problem and propose Tempered Evidence Fusion (TEF), a training-free rule that weights each sentence’s log-odds by its normalized information gain, as derived from a generalized Bayesian posterior. This makes the fused score nearly vanish for uncertain sentences while preserving the Bayes-optimal weight of decisive evidence. We further introduce Multi-event Insight Network Dimensions (MIND), a benchmark of 8,358 Chinese and English posts spanning five years of public events and six value dimensions. On MIND, TEF outperforms the strongest baseline among Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy points and 4.6 macro-F1 points across five LLMs and two languages. MIND dataset and code are available at GitHub. Index Terms— Decision fusion, large language models, uncer- tainty, calibration, computational social science 1. INTRODUCTION Public value orientations are key latent variables in computational so- cial science, capturing how people position themselves on issues such as economic regulation, climate policy, or cultural change. Social media makes these orientations observable through text at scale [1,2]. Yet this scale, together with the subjectivity of human annotation, has motivated growing interest in using LLMs as measurement in- struments [3,4]. Existing LLM-based pipelines usually either ask the model to read the whole post and return a document-level la- bel [5,6,7], or split the post into sentences and aggregate sentence- level predictions by voting [8]. However, we observe that little research has examined how sentence-level LLM judgments should be fused when evidence is unevenly distributed across a long post. In Fig. 1, long social media posts often mix background information, quotations, and concessions, with only a few stance-bearing sentences. In such cases, direct predic- tion can collapse heterogeneous evidence into one overconfident label, while majority or soft voting can allow uncertain sentences to outvote a few decisive ones. This leads us to ask: How can sentence-level judgments be combined so that decisive evidence dominates the final decision while uncertain sentences are discounted? ∗ Equal contribution † Corresponding author Our Method: TEF! Existing Method 2 3 Direct Majority Vote Soft Vote Context Task: Measure value orientation Content:Targets are debated...... Cut emissions now....... Options: Change/Stability Segmant 1Segmant 2Segmant 3Segmant 4Full Context Low EntropyLow EntropyHigh EntropyHigh Entropy 1 Full Context Definitely Sta. S S SC 1 2 3 4 3:1 So Sta. Average P(S) < 50% So Sta. Aggregator 1 2 3 4 1 3 2 4 P(C) Totally > 50% So Change-Oriented! Log-Odds Accumulation Count Evidence，Not Sentences！ CSCSCS CS Fig. 1. Long-text value measurement requires counting informa- tive evidence rather than treating all sentences equally. LEFT: Di- rect prediction collapses mixed content into one label, while vot- ing can let many uncertain sentences outweigh a few decisive ones. RIGHT: TEF weights sentence log-odds by normalized information gain, discounting uncertain judgments so decisive evidence drives the document-level orientation. Bearing this in mind, we formulate long-text value measurement as a decision-fusion problem, where each sentence acts as a local detector that emits a soft decision. Under conditional independence and calibrated sentence posteriors, the Bayes-optimal document-level rule sums sentence log-likelihood ratios. This observation places existing voting rules on a common scale: majority voting keeps only the sign of each sentence judgment, and soft voting uses a bounded surrogate that underweights highly decisive sentences. At the same time, directly summing LLM logits is unreliable because next-token probabilities are only approximately calibrated, especially near the point of maximal uncertainty. To this end, we present Tempered Evidence Fusion (TEF), a training-free fusion rule that tempers each sentence’s log-odds by its normalized information gain, as derived from a generalized Bayesian posterior [9,10]. The resulting score nearly vanishes for uncertain sentences but preserves the Bayes-optimal weight of decisive evi- dence, yielding a soft censoring effect similar in spirit to censoring rules in distributed detection [11]. In Fig. 1 (right), TEF counts evi- dence rather than sentences, allowing a few informative sentences to determine the document-level orientation. Our contributions are threefold. (i) We cast long-text value mea- surement with LLMs as a decision-fusion problem and derive TEF from a tempered Bayesian posterior, showing that it suppresses uncer- tain sentences while preserving the weight of decisive evidence. (i) We introduce MIND, a five-year, cross-lingual benchmark of 8,358 posts over six value dimensions. (i) We show that TEF outperforms the strongest of the Direct, Majority Vote, and Soft Vote baselines by 4.5 accuracy points and 4.6 macro-F1 points on average across five LLMs and two languages, yielding better-calibrated estimates. yuheewuu,qiianruii@gmail.com, guangzhang@hkust-gz.edu.cn arXiv:2609.27165v1 [cs.CL] 22 Sep 2026 2. RELATED WORK LLM-based stance and ideology measurement has expanded from short-form texts such as tweets to longer, multi-issue documents [12, 13,14]. In parallel, the reliability of LLM confidence has been studied from multiple perspectives, including token-level probabili- ties [15,16], verbalized confidence [17,18], and aggregation methods beyond majority voting [19]. In signal processing, fusing local deci- sions is a classical problem [20]: log-linear opinion pools combine probabilistic judgments [21], the Chair–Varshney rule weights local decisions by their reliabilities [22], and censoring schemes transmit only observations with informative likelihood ratios [11]. Despite these connections, long-text measurement with LLMs still largely relies on equal-weight voting. In contrast, we formulate long-text value measurement as sentence-level decision fusion and derive an evidence-aware fusion rule, TEF, that discounts uncertain sentences while preserving the weight of decisive ones. 3. PROPOSED TEF Problem Formulation. We formulate long-text value measurement as a decision-fusion problem. The goal is to infer a latent social vari- able from a collection of textual segments whose evidential strength may vary substantially within the same document. LetXdenote the input space and letY = c 1 ,...,c K be the set of target measure- ment categories. For an observationX ∈X, the latent measurement variableYtakes values inY. We segment each long text intoN sentence-level units, X 7→D(X) =D 1 ,D 2 ,...,D N ,(1) whereD i denotes thei-th segment. This granularity preserves lo- cal semantic coherence while keeping inference computationally manageable. For each segmentD i , a language modelMreturns a probability distribution over the measurement categories, denoted byp i = M(D i ) = p (1) i ,...,p (K) i , where each entryp (k) i = P M (Y = c k | D i ) andp i ∈ ∆ K−1 . The distribution encodes both a local predictionˆy i = arg max k p (k) i and its uncertainty, measured by entropyH(p i ) = − P K k=1 p (k) i logp (k) i .Common aggregation rules discard part of this probabilistic evidence. Majority voting uses only the segment-level hard labels, while soft voting averages probability vectors with equal weight, ˆ Y MV = mode ˆy i N i=1 , ˆ Y SV = arg max c∈Y 1 N N X i=1 p (c) i . (2) Both rules treat all segments as equally reliable. This is problematic for scientific measurement from long texts, where decisive evidence may appear in only a few segments and many other segments may be ambiguous, rhetorical, or weakly relevant. TEF instead aggregates reliability-weighted evidence. Letw i = ω(p i ) ∈ [0, 1]be the reliability assigned to segmentD i , and letg : [0, 1] →Rbe a scoring transformation. The document-level measurement is ˆ Y = arg max c∈Y P N i=1 w i g p (c) i . The following paragraphs specifyω and g, which together define the fusion rule of TEF. Reliability Weighting. TEF instantiates the preceding framework with entropy-based reliability weighting and one-vs-rest log-odds evi- dence. Algorithm 1 gives the complete procedure. We use normalized information gain as the segment reliability,w i = 1− H(p i ) log K ∈ [0, 1]. This weight is close to one for concentrated distributions and close to zero for near-uniform distributions. Thus confident segments con- tribute strongly, while uncertain segments are softly censored. Evidence Aggregation. For each categoryc, TEF transforms the segment posterior into a one-vs-rest log-odds score,g p (c) i = log p (c) i 1−p (c) i . The aggregated evidence for category c is Score(c) = N X i=1 w i log p (c) i 1− p (c) i .(3) The final measurement outcome is ˆ Y = arg max c∈Y Score(c). In implementation, probabilities are clipped to[ε, 1− ε]before apply- ing the logit, and log-odds values are clipped to[−M,M ]. Unless otherwise stated, we setε = 10 −6 andM = 10. Unlike majority voting, TEF does not reducep i to a hard label before aggregation. Unlike soft voting, TEF does not assume that all segments are equally informative. It therefore preserves distributional information while reducing the impact of noisy or ambiguous segments. Algorithm 1 Tempered Evidence Fusion Require:TextX, modelM, categoriesY =c 1 ,...,c K , clip boundM, stability constant ε. Ensure: Measurement outcome ˆ Y 1: D 1 ,...,D N ←D(X)▷ Segment into sentences 2: Initialize Score(c)← 0 for all c∈Y 3: for i = 1 to N do 4:p i ←M(D i )▷ Segment-level distribution 5: H i ←− P K k=1 p (k) i logp (k) i ▷ Entropy 6: w i ← 1− H i / logK▷ Reliability weight 7: for c∈Y do 8: ̃p← clip(p (c) i ,ε, 1− ε) 9:ℓ← clip log ̃p 1− ̃p ,−M,M 10:Score(c)← Score(c) + w i ℓ 11: end for 12: end for 13: ˆ Y ← arg max c∈Y Score(c) 14: return ˆ Y Theoretical Analysis. Proposition 1 (Information Preservation). Since the hard label ˆy i = arg max k p (k) i is a deterministic function ofp i , the data pro- cessing inequality [23] givesI(Y ;p i )≥ I(Y ; ˆy i ). Thus, hard voting can discard reliability information even when two segments support the same category with different confidence. TEF follows this principle by retaining the full distributionp i instead of reducing each segment toˆy i . Its entropy-based weightw i distinguishes decisive evidence from ambiguous evidence, so two segments with the same hard label can contribute different amounts. The method therefore preserves the information discarded by majority voting while producing an additive document-level score. Proposition 2 (Tempered Bayesian Pooling). For a binary con- trast with equal priors, calibrated conditionally independent seg- ment posteriors satisfy the additive Bayes log-odds identity [24] log P (Y =c|D 1:N ) P (Y = ̄c|D 1:N ) = P N i=1 log P (Y =c|D i ) P (Y = ̄c|D i ) . Therefore, evidence is naturally accumulated on the log-odds scale rather than through hard counts or raw probability averaging. TEF uses the one-vs-rest transformg(p) = log p 1−p to imple- ment this additive evidence scale. Because LLM posteriors are ap- proximate, it tempers each log-odds contribution [9] withw i = (logK − H(p i ))/ logK, the normalized information gain from a uniform prior. Near-uniform segments therefore have little influence, while low-entropy stance-bearing segments retain a contribution close to the Bayesian additive statistic. This gives TEF its soft-censoring behavior, as uncertain sentences do not accumulate by sheer num- ber, while decisive sentences can determine the final document-level measurement outcome even when they are few. Stage I Data Collection Stage I Data Annotation Stage I Agent-Driven Data Evolution Round 1Data Source Social Media Discussion Forums Data Preprocessing Anonymization Selection Deduplication Round 2 Round 3 DimensionSubcategory Orientation Mixed Annotators Domin ExpertsSpecialized Annotators Consensors? Consensors? Distribution & Qualify Check If Imbalanced ： Balencing Agent If Balanced ： Synthetic Samples Critic Agent Semantic-Edit Agent Balancing Refinement Arbitration & Discard Re-EvaluateLow Confidence Revision Agree ≥ 2/3? Agree ≥ 2/3? Agree ≥ 2/3? Fail PassPass Pass Fail Arbitration AArbitration B Discard Generative Oversampling Verified News RSS Feeds Raw Samples Fail Fig. 2. Construction pipeline of MIND. Posts are collected from public sources, filtered for relevance, anonymized, and deduplicated, then annotated in three rounds for dimension, subcategory, and orientation before being balanced and refined by LLM agents. Revolutionism Reformism Globalism Isolationism Free-Market Interventionist Equality of Opportunity Equality of Outcome Ethical Liberalism Ethical Conservatism Individualism Collectivism Substantive Justice Procedural Justice Progressive Jurisprudence Conservative Jurisprudence Conservation Priority Development Priority Climate Actionism Climate Skepticism Inclusive Sharing IP Protection Technological Libertarianism Technological Regulation 621 374 1,531 477 452 602 224 206 316 549 385 832 1,152 694 193 192 201 163 427 121 621 196 631 602 Revolutionism Reformism Globalism Isolationism Free-Market Interventionist Equality of Opportun ity Equality of Outcome Ethical Liberalism Ethical Conservatism Individualism Collectivism Substantive Justice Procedural Justice Progressive Ju risprudence Conservative Jurisprudence Conservation Priority Deve lopmen t Priority Climate Ac tionis m Climate Skepticism Inclusive Sharing IP Protection Technological Libertarianism Technological Regulation 341 97 390 272 304 352 278 263 198 202 245 426 693 263 101 80 67 15 49 16 42 16 88 121 (a) Chinese (b) English Fig. 3. Statistics of MIND. Radial bars report the number of labels for each value orientation, with colors denoting dimensions and paired bars representing the opposing orientations of each subcategory. Pan- els (a) and (b) correspond to Chinese and English posts. 4. THE MIND BENCHMARK We construct the MIND dataset, a longitudinal, cross-lingual bench- mark for measuring public value orientations. The final corpus con- tains 8,358 posts, including 5,474 Chinese posts and 2,884 English posts over six value dimensions in Fig. 3. MIND is built through a three-stage pipeline. Stage I: Data Collection. We collect can- didate posts from X, Weibo, Reddit, Zhihu, discussion forums, and news feeds from January 2020 to January 2025, covering major public events such as the Russia–Ukraine war and the rise of gen- erative AI. For each value dimension, keyword search is anchored to high-search-volume events. Raw posts are filtered for relevance and opinion content, anonymized, de-duplicated, and screened by an automatic quality score. Stage I: Data Annotation. We adopt a three-round annotation framework to label the value dimension, subcategory, and orientation in sequence. Each round uses three annotators and accepts a label only when at least two agree. Failed or disputed cases enter arbitration, samples without a stable majority are discarded, and accepted samples pass distribution and quality checks before the evolution stage. Stage I: Agent-Driven Data Evolution. To address class imbalance and semantic ambiguity, we introduce an agent-driven evolution stage. When a category is underrepresented, a balancing agent generates synthetic samples through generative over- sampling. For low-confidence or ambiguous samples, a critic agent evaluates the alignment between text and label, and a semantic-edit agent revises the text to make the labeled orientation explicit. Only verified samples are retained in the final benchmark. 5. EXPERIMENTS Experiment Settings.We evaluate three open-source models, Qwen2.5-7B [25], LLaMA3-8B [26], and Qwen3-14B [27], and two proprietary models, DeepSeek-V3.2 [28] and GPT-4o-mini [29], using one shared prompt per language for TEF, Direct, Majority Vote, and Soft Vote. Each prompt specifies the dimension and its two posi- tions and requests a single answer token. We read the answer-token log-probabilities without sampling. Open-source models are served on RTX 3090 GPUs. We setε = 10 −6 andM = 10, and report dimension-level accuracy and macro-F1. Main Results. In Table 3, TEF has the highest accuracy and macro-F1 for every model in both languages, exceeding the strongest baseline by 4.5 accuracy and 4.6 macro-F1 points on average, so the gain comes from the fusion rule rather than from model-specific tuning. At the dimension level, TEF is the best or tied for best in 116 of 120 comparisons and never trails the strongest baseline by more than 2.6 points. Segmentation alone does not ensure improvement, since equal- weight voting, whether hard or soft, does not consistently improve on Direct. The gains of TEF concentrate where the baselines are weakest, on the smaller open-source models, on culture and politics in Chinese, and on environment in English, whereas on dimensions where Direct is already strong TEF matches it. Ablation Study. Table 4 evaluates TEF by removing one component at a time. On Qwen2.5-7B, both variants retain most of TEF’s average gain over Soft Vote; on DeepSeek-V3.2, however, both fall below Soft Vote in both languages on average. Their complementarity reflects their distinct roles: the log-odds transform emphasizes decisive evi- dence, while entropy weighting suppresses uncertain sentences. The largest accuracy drop occurs on DeepSeek-V3.2’s English environ- ment dimension, where removing entropy weighting or the log-odds transform reduces accuracy by 22.9 or 23.8 percentage points. Fig. 4. Calibration on Qwen2.5-7B. TEF achieves the lowest expected calibration error and overconfidence gap across both Chinese and English settings. Compared with Direct, Majority Vote, and Soft Vote, TEF aligns confidence with accuracy by tempering weak evidence. Table 3. Main results on MIND. Accuracy (Acc) and macro-F1 (F1) in percent for every model, language, and dimension. Cul., Eco., Soc., Pol., Tech., and Env. denote culture, economy, society, politics, technology, and environment. MV and SV denote Majority Vote and Soft Vote, and Ours denotes TEF. For each model, language, dimension, and metric, the best method is in bold and the second best is underlined. ChineseEnglish ModelMethodCul.Eco.Soc.Pol.Tech.Env.Cul.Eco.Soc.Pol.Tech.Env. AccF1AccF1AccF1AccF1AccF1AccF1AccF1AccF1AccF1AccF1AccF1AccF1 Qwen2.5-7B Direct44.644.244.644.574.473.255.254.172.1 66.382.681.854.053.447.2 47.173.569.756.6 53.6 62.9 56.2 70.664.6 MV44.4 43.7 43.4 41.3 64.0 64.0 50.5 50.3 67.6 64.7 71.6 71.553.1 52.2 50.543.6 49.7 49.7 60.953.3 62.9 58.1 51.4 48.9 SV43.5 42.8 43.2 40.6 65.7 65.6 51.8 51.5 69.966.373.4 73.353.2 52.5 49.0 41.8 50.8 50.8 60.8 53.763.461.355.0 52.3 Ours 72.1 69.9 47.7 45.1 81.5 80.6 68.3 62.3 72.8 66.5 85.1 84.1 62.5 58.2 59.4 59.4 79.6 77.3 67.8 66.4 64.6 61.6 80.7 67.2 LLaMA3-8B Direct63.5 58.236.4 35.9 73.870.7 60.656.570.9 64.980.8 79.562.953.441.1 40.874.864.149.9 48.9 54.952.6 59.656.6 MV59.9 51.8 36.7 36.5 68.9 68.6 55.9 54.1 66.5 61.9 74.4 73.954.9 48.9 47.8 35.3 60.2 59.7 56.550.454.954.448.6 47.6 SV60.8 53.6 37.337.073.1 72.954.3 52.8 70.064.1 78.277.555.4 49.0 48.136.5 62.2 61.5 54.6 46.8 52.0 51.7 50.5 48.5 Ours 65.9 59.9 38.1 38.0 82.7 82.0 64.4 62.4 70.9 65.1 78.277.563.3 56.2 55.8 53.5 80.5 77.6 67.5 63.4 58.9 58.7 78.0 70.5 Qwen3-14B Direct73.771.7 41.339.983.9 82.8 61.3 58.6 74.568.9 87.9 86.860.4 46.6 56.256.282.3 76.3 50.146.3 60.0 53.6 74.3 60.1 MV69.5 65.7 41.339.981.2 79.5 60.9 58.1 73.1 63.9 84.4 83.660.654.0 56.256.282.5 76.749.7 46.1 57.1 52.6 75.263.4 SV73.771.840.9 38.3 83.782.661.1 58.4 74.568.987.986.860.654.0 55.8 55.8 80.2 72.8 49.9 46.560.6 54.075.263.4 Ours 73.8 71.9 42.1 40.7 83.9 82.8 68.6 64.0 74.8 69.2 88.0 87.0 62.3 54.3 57.9 57.8 82.5 76.8 50.7 47.4 60.6 54.1 76.1 64.2 DeepSeek-V3.2 Direct68.766.0 39.6 39.4 79.8 78.8 62.3 59.8 73.767.1 86.4 85.060.259.949.5 49.5 79.276.955.7 51.257.1 52.2 52.3 50.4 MV68.5 66.338.6 38.5 78.5 77.5 61.9 61.6 71.5 66.6 83.6 82.659.0 57.9 57.7 58.577.2 74.9 61.849.2 57.1 54.5 54.1 52.7 SV68.5 65.8 40.039.880.279.262.662.273.3 67.685.984.959.8 59.6 58.158.0 79.1 76.8 62.7 51.258.355.551.4 49.7 Ours 75.6 73.5 42.4 40.5 83.9 82.7 62.8 62.3 74.4 68.0 86.4 85.4 63.7 60.3 58.8 58.8 83.9 80.3 62.7 55.4 60.0 58.7 74.3 66.4 GPT-4o-mini Direct63.9 53.8 42.239.182.0 81.2 67.4 63.7 74.961.883.682.360.947.149.842.081.777.054.4 53.3 63.455.279.8 68.7 MV64.0 54.442.238.9 80.2 78.8 67.765.3 73.0 60.4 82.0 80.159.8 45.7 49.840.3 79.7 75.6 56.155.561.7 53.9 81.768.1 SV64.354.3 42.0 38.3 81.3 79.967.765.2 74.2 61.0 85.9 84.360.8 46.8 50.3 42.081.3 76.8 55.8 54.5 62.3 54.9 80.7 68.5 Ours 74.0 71.2 42.9 41.1 81.4 79.967.8 65.3 75.3 68.9 85.9 84.4 64.2 57.5 50.3 42.8 83.9 79.7 57.5 56.7 64.6 56.6 83.5 72.4 Table 4. Ablation of TEF. Accuracy and macro-F1 (%) on Qwen2.5-7B and DeepSeek-V3.2; abbreviations as in Table 3. w/o Entropy keeps only the log-odds sum; w/o Log-Odds averages entropy-weighted probabilities instead of log-odds. Ours is the full TEF (bold). ModelLanguageMethod Cul.Eco.Soc.Pol.Tech.Env.Average AccF1AccF1AccF1AccF1AccF1AccF1AccF1 Qwen2.5-7B Chinese Ours72.1 69.9 47.7 45.1 81.5 80.6 68.3 62.3 72.8 66.5 85.1 84.1 71.3 68.1 w/o Entropy71.368.845.642.979.979.167.461.471.765.881.680.769.666.5 w/o Log-Odds71.869.646.243.580.379.567.261.172.266.281.580.669.966.8 English Ours62.5 58.2 59.4 59.4 79.6 77.3 67.8 66.4 64.6 61.6 80.7 67.2 69.1 65.0 w/o Entropy61.856.658.058.077.275.266.165.162.356.275.260.966.862.0 w/o Log-Odds61.857.058.558.477.175.166.165.063.456.776.161.767.262.3 DeepSeek-V3.2 Chinese Ours75.6 73.5 42.4 40.5 83.9 82.7 62.8 62.3 74.4 68.0 86.4 85.4 70.9 68.7 w/o Entropy68.165.439.239.079.678.762.461.972.767.185.784.868.066.2 w/o Log-Odds68.565.739.339.179.578.562.762.273.367.684.983.868.066.2 English Ours63.7 60.3 58.8 58.8 83.9 80.3 62.7 55.4 60.0 58.7 74.3 66.4 67.2 63.3 w/o Entropy58.958.557.156.978.576.362.050.553.751.451.449.760.357.2 w/o Log-Odds59.359.156.656.478.876.562.450.954.351.350.548.960.357.2 Table 1. Robustness Analysis. Average accuracy (%) over both lan- guages under the original, verbose, and minimal prompts. Strongest baseline is the best of Direct, MV, and SV per prompt; Ours is TEF. Model Strongest baselineOurs Original Verbose Minimal Original Verbose Minimal Qwen2.5-7B61.560.859.270.269.868.6 LLaMA3-8B60.860.058.267.066.465.3 Qwen3-14B67.266.564.868.467.966.8 DeepSeek-V3.265.064.362.769.168.567.5 GPT-4o-mini67.266.565.069.368.867.8 Table 2. Efficiency Analysis. Total tokens and cost over the twelve evaluation subsets and mean time to first token (TTFT) on RTX 3090 GPUs. Voting covers MV and SV, which issue the same queries. ModelMethodTokens (M)TTFT (ms)Cost ($) Qwen2.5-7B Direct0.523.70.09 Voting2.714.40.27 Ours2.715.80.28 LLaMA3-8B Direct0.628.60.10 Voting3.115.20.28 Ours3.117.30.30 Analysis. 1) Calibration. Because measurement outputs are thresh- olded or manually reviewed, document-level confidence should reflect correctness. In Fig. 4, Direct and both voting rules are overconfident on Qwen2.5-7B, whereas TEF achieves the lowest expected calibra- tion error [30,31] and overconfidence gap. Its confidence is therefore better suited to selective review. 2) Robustness. To test whether the gain is due to the fusion rule rather than the shared prompt, we rerun all methods with a verbose prompt that adds role background and annotation guidelines and with a minimal prompt that retains only the classification directive. In Table 1, TEF remains best under every prompt and degrades the least under either perturbation. 3) Efficiency. In Table 2, TEF uses the same single-token queries as the voting methods, so its token count is identical. Its time to first token differs by only about two milliseconds. Relative to Direct, sentence-level methods cost roughly three times as much, but TEF is the only one that consistently turns this overhead into a performance gain. 6. CONCLUSION We cast long-text value measurement as the fusion of soft sentence decisions and derived TEF, which tempers each sentence’s log-odds by its information gain. On MIND, TEF is more accurate than direct prediction and voting for five LLMs in two languages and better cali- brated on Qwen2.5-7B, and its gain survives changes of the prompt. 7. REFERENCES [1]Michael Conover, Jacob Ratkiewicz, Matthew Francisco, Bruno Gonc ̧alves, Filippo Menczer, and Alessandro Flammini, “Politi- cal polarization on Twitter,” in Proceedings of the International AAAI Conference on Web and Social Media, Barcelona, Spain, 2011, vol. 5, p. 89–96. [2] Byunghwee Lee, Rachith Aiyappa, Yong-Yeol Ahn, Haewoon Kwak, and Jisun An, “A semantic embedding space based on large language models for modelling human beliefs,” Nature Human Behaviour, vol. 9, no. 9, p. 1928–1940, 2025. [3] Nick Pangakis and Sam Wolken, “Keeping humans in the loop: Human-centered automated annotation with generative AI,” in Proceedings of the International AAAI Conference on Web and Social Media, 2025, vol. 19, p. 1471–1492. [4] Tunazzina Islam and Dan Goldwasser, “Uncovering latent argu- ments in social media messaging by employing LLMs-in-the- loop strategy,” in Findings of the Association for Computational Linguistics: NAACL 2025, 2025, p. 7397–7429. [5]Fabrizio Gilardi, Meysam Alizadeh, and Ma ̈ el Kubli, “ChatGPT outperforms crowd workers for text-annotation tasks,” Proceed- ings of the National Academy of Sciences, vol. 120, no. 30, p. e2305016120, 2023. [6]Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang, “Can large language models transform computational social science?,” Computational Linguistics, vol. 50, no. 1, p. 237–291, 2024. [7]Petter T ̈ ornberg, “Large language models outperform expert coders and supervised classifiers at annotating political social media messages,” Social Science Computer Review, vol. 43, no. 6, p. 1181–1195, 2024. [8]Dimitrios Tsirmpas, Ioannis Gkionis, Georgios Th. Papadopou- los, and Ioannis Mademlis, “Neural natural language processing for long texts: A survey on classification and summarization,” Engineering Applications of Artificial Intelligence, vol. 133, p. 108231, 2024. [9]Pier Giovanni Bissiri, Chris C. Holmes, and Stephen G. Walker, “A general framework for updating belief distributions,” Journal of the Royal Statistical Society Series B: Statistical Methodol- ogy, vol. 78, no. 5, p. 1103–1130, 2016. [10]Peter Gr ̈ unwald and Thijs van Ommen, “Inconsistency of Bayesian inference for misspecified linear models, and a pro- posal for repairing it,” Bayesian Analysis, vol. 12, no. 4, p. 1069–1103, 2017. [11]Constantino Rago, Peter Willett, and Yaakov Bar-Shalom, “Cen- soring sensors: A low-communication-rate scheme for dis- tributed detection,” IEEE Transactions on Aerospace and Elec- tronic Systems, vol. 32, no. 2, p. 554–568, 1996. [12] Rudra Ranajee Saha, Laks V. S. Lakshmanan, and Raymond T. Ng, “Stance detection with explanations,” Computational Linguistics, vol. 50, no. 1, p. 193–235, 2024. [13] Abu Ubaida Akash, Ahmed Fahmy, and Amine Trabelsi, “Can large language models address open-target stance detection?,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, p. 971–985. [14]Prasanta Bhattacharya, Hong Zhang, Yiming Cao, Wei Gao, Brandon Siyuan Loh, Joseph J. P. Simons, and Liang Ze Wong, “Rethinking stance detection: A theoretically-informed research agenda for user-level inference using language models,” arXiv preprint arXiv:2502.02074, 2025. [15] Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neu- big, “How can we know when language models know? On the calibration of language models for question answering,” Trans- actions of the Association for Computational Linguistics, vol. 9, p. 962–977, 2021. [16]Saurav Kadavath et al., “Language models (mostly) know what they know,” arXiv preprint arXiv:2207.05221, 2022. [17]Katherine Tian et al., “Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, 2023, p. 5433–5442. [18]Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, and Anirudha Majumdar, “A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions,” ACM Computing Surveys, vol. 58, no. 3, p. 1–38, 2025. [19]Rui Ai, Yuqi Pan, David Simchi-Levi, Milind Tambe, and Haifeng Xu, “Beyond majority voting: LLM aggregation by leveraging higher-order information,”arXiv preprint arXiv:2510.01499, 2025. [20]Robert R. Tenney and Nils R. Sandell, “Detection with dis- tributed sensors,” IEEE Transactions on Aerospace and Elec- tronic Systems, vol. AES-17, no. 4, p. 501–510, 1981. [21]Christian Genest and James V. Zidek, “Combining probabil- ity distributions: A critique and an annotated bibliography,” Statistical Science, vol. 1, no. 1, p. 114–135, 1986. [22]Z. Chair and Pramod K. Varshney, “Optimal data fusion in multiple sensor detection systems,” IEEE Transactions on Aerospace and Electronic Systems, vol. AES-22, no. 1, p. 98– 101, 1986. [23]Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, Wiley, 2nd edition, 2005. [24]Pramod K. Varshney, Distributed Detection and Data Fusion, Springer, New York, NY, 1997. [25] Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, et al., “Qwen2.5 technical report,” 2024, arXiv:2412.15115. [26] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al., “The Llama 3 herd of models,” 2024, arXiv:2407.21783. [27]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al., “Qwen3 technical report,” 2025, arXiv:2505.09388. [28]DeepSeek-AI, “DeepSeek-V3.2: Pushing the frontier of open large language models,” 2025, arXiv:2512.02556. [29]OpenAI, “GPT-4o mini: Advancing cost-efficient intelligence,” OpenAI blog, July 2024. [30]Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht, “Obtaining well calibrated probabilities using Bayesian binning,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2015, p. 2901–2907. [31]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning, 2017, p. 1321–1330.