Paper deep dive
Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth
Jackson Eshbaugh, Jorge Silveyra
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/11/2026, 2:39:54 AM
Summary
The paper introduces a calibration-first framework for generating supervision scores without ground-truth labels or a shared annotation space. It aligns heterogeneous, subset-specific scorers using a synthetic ordinal reference space before fusing their outputs. The method is evaluated on Ames Housing, Breast Cancer Wisconsin, and Wine Quality datasets, showing it outperforms uncalibrated averaging and often matches or exceeds individual scorers, with significant improvements over baselines in specific domains.
Entities (11)
Relation Signals (9)
Jorge Silveyra → affiliatedwith → Lafayette College
confidence 95% · Jorge Silveyra Affiliation: Department of Computer Science Lafayette College
Jackson Eshbaugh → affiliatedwith → Lafayette College
confidence 95% · Jackson Eshbaugh Affiliation: Department of Computer Science Lafayette College
Calibration-First Framework → evaluatedon → Ames Housing
confidence 95% · Across three benchmark datasets... Ames Housing... our framework consistently outperforms uncalibrated averaging
Calibration-First Framework → evaluatedon → Breast Cancer Wisconsin
confidence 95% · Performance relative to sample-dependent baselines varies by domain... on Breast Cancer Wisconsin
Calibration-First Framework → evaluatedon → Wine Quality
confidence 95% · Performance relative to sample-dependent baselines varies by domain... on Wine Quality.
Calibration-First Framework → uses → Synthetic Ordinal Reference Space
confidence 95% · Our framework aligns subset-specific scorers using a synthetic ordinal reference space before fusion.
Calibration-First Framework → outperforms → Uncalibrated Averaging
confidence 90% · our framework consistently outperforms uncalibrated averaging
Calibration-First Framework → relatedto →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce a calibration-first framework that produces supervision scores without access to ground-truth labels or a shared annotation space. Our framework aligns subset-specific scorers using a synthetic ordinal reference space before fusion. This reference space is constructed from ordered calibration features that represent the latent concept, providing a common scale on which otherwise incomparable scorer outputs can be aligned. Because our calibration procedure uses the reference space rather than training samples, it is independent of the training set's empirical distribution. Across three benchmark datasets, our framework consistently outperforms uncalibrated averaging and achieves higher primary-metric point estimates on the evaluation metrics than the best individual scorer. Performance relative to sample-dependent baselines varies by domain, with absolute differences below 0.02 on Ames Housing and below 0.01 on Breast Cancer Wisconsin and Wine Quality. After Bonferroni correction, differences remain significant for all three comparisons on Ames Housing and one on Breast Cancer Wisconsin. Additionally, we show that using fewer calibration levels per feature can closely approximate higher-resolution results at substantially lower computational cost. Together, these results support our framework as a viable approach to construct supervision scores when neither ground-truth labels nor a shared annotation space is available.
Tags
Links
- Source: https://arxiv.org/abs/2610.09525v1
- Canonical: https://arxiv.org/abs/2610.09525v1
Trouble viewing inline? Open PDF directly →
Full Text
75,562 characters extracted from source content.
Expand or collapse full text
Align Before You Combine 1.0 2026 Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth Jackson Eshbaugh Affiliation: Department of Computer Science Lafayette College Easton, PA, USA Email: eshbaugj@lafayette.edu Jorge Silveyra Affiliation: Department of Computer Science Lafayette College Easton, PA, USA Email: silveyj@lafayette.edu Abstract We introduce a calibration-first framework that produces supervision scores without access to ground-truth labels or a shared annotation space. Our framework aligns subset-specific scorers using a synthetic ordinal reference space before fusion. This reference space is constructed from ordered calibration features that represent the latent concept, providing a common scale on which otherwise incomparable scorer outputs can be aligned. Because our calibration procedure uses the reference space rather than training samples, it is independent of the training set’s empirical distribution. Across three benchmark datasets, our framework consistently outperforms uncalibrated averaging and achieves higher primary-metric point estimates on the evaluation metrics than the best individual scorer. Performance relative to sample-dependent baselines varies by domain, with absolute differences below 0.02 on Ames Housing and below 0.01 on Breast Cancer Wisconsin and Wine Quality. After Bonferroni correction, differences remain significant for all three comparisons on Ames Housing and one on Breast Cancer Wisconsin. Additionally, we show that using fewer calibration levels per feature can closely approximate higher-resolution results at substantially lower computational cost. Together, these results support our framework as a viable approach to construct supervision scores when neither ground-truth labels nor a shared annotation space is available. 1 Introduction Heterogeneous evidence sources are often used to train machine learning systems, yet they frequently lack ground-truth labels or a shared annotation space. Current approaches, such as weak supervision and multimodal fusion, typically assume that supervision sources operate within a common target space. This assumption is problematic when supervision is derived from heterogeneous subset-specific scorers that operate on different evidence representations and incomparable numerical scales. In such settings, each source may act as a noisy semantic observer of an underlying latent concept, where the resulting signals may exist in different scales and in different representational spaces. Consequently, the outputs cannot be directly compared and require calibration before meaningful fusion can occur. In this document, we introduce a calibration-first framework for supervision without ground truth that aligns subset-specific scorers using a synthetic ordinal reference space before fusion. Our framework uses this reference space as a semantic calibration mechanism. By evaluating subset-specific scorers relative to this shared reference space, heterogeneous evidence sources can be aligned onto a common semantic scale prior to fusion. The resulting aligned scores are then fused into a supervision score. Ultimately, this approach enables supervision from heterogeneous sources where neither labeled data nor a shared annotation space is available. To assess the viability of our framework, we conducted a case study in a residential energy-retrofit application—where text-derived and structured scorers operate on different numerical scales—alongside benchmarks from the housing and medical domains. Our contributions are: (1) a calibration-first framework for supervision without ground truth; (2) a synthetic ordinal reference space for aligning heterogeneous scorers prior to fusion; and (3) formal properties and empirical evidence demonstrating ordinal preservation and improved supervision quality. 2 Related Work Given the diverse scope of this problem and to contextualize the landscape in which our work lies, we review three related research areas that address different challenges. Specifically, (1) weak supervision focuses on learning from imperfect or noisy labels, (2) multimodal learning addresses the integration of heterogeneous data representations, and (3) latent construct research studies how unobservable concepts can be inferred from indirect sources. Modern weak supervision research has given rise to programmatic weak supervision (PWS), which combines multiple distant supervision approaches through noisy labeling functions and latent label modeling [26]. Current PWS involves users encoding domain knowledge into noisy, explicitly engineered labeling functions that collectively enable latent-label estimation without direct supervision and generally assumes these functions produce values within a shared target space [22, 6, 4, 25]. Recent work has extended the PWS paradigm by constructing probabilistic supervision from multiple imperfect diagnostic tests or expert assessments through latent class modeling [24], and by exploring constraint-based alternatives to latent generative label models [1]. These approaches still assume supervision sources operate within a common label space and focus on recovering latent labels rather than aligning heterogeneous scoring functions defined on incomparable semantic scales. A related line of research, multimodal machine learning, concentrates on integrating heterogeneous sources such as text, structured measurements, and simulation outputs. Most multimodal systems assume ground-truth supervision and treat modalities as complementary predictive features aligned to a shared target, combining them through fusion strategies to improve downstream performance [18, 3, 17]. Recent developments have also explored calibration mechanisms for combining heterogeneous information sources, demonstrating that alignment of subset-specific signals prior to fusion can improve downstream retrieval and reasoning performance [2]. These models are generally trained using observable ground truth targets. A third area of work is related to measuring latent constructs—constructs that are not directly observable. In fields such as education, psychology, and other social sciences, psychometrics is used to measure latent constructs, such as intelligence, personality traits, or educational proficiency. Classical test theory and Item Response Theory (IRT) model observed responses as imperfect indicators of an underlying latent variable and provide mechanisms for placing heterogeneous observations onto a common measurement scale [13, 11, 21]. Construct validity emphasizes that meaningful measurement requires a theoretically justified relationship between observed indicators and the latent construct being measured [9, 7]. The existing literature provides several foundations for the problem considered in this work, but none directly address supervision construction from heterogeneous evidence sources when no common target space or ground-truth labels are available. Specifically, our work expands beyond prior research in three key ways: 1. Beyond a Shared Space: Existing weak supervision frameworks assume that supervision sources already operate within a shared target space. In contrast, we construct a common semantic scale on which heterogeneous subset-specific scorers can be meaningfully compared. 2. Beyond Multimodal Prediction: Multimodal machine learning typically assumes an observable target variable and uses modalities as complementary predictive features. Our goal is not to improve prediction through fusion but to construct supervision itself, aligning subset-specific observations when no ground-truth target is available. 3. Beyond Latent Construct Estimation: Psychometrics addresses the problem of measuring latent constructs through the use of multiple imperfect indicators and provides mechanisms for placing heterogeneous observations onto a common measurement scale. Instead, our approach constructs an external reference structure composed of synthetic examples enabling supervision without ground-truth labels or a pre-existing measurement instrument. We address the challenge of constructing supervision from heterogeneous subset-specific scorers that operate on incomparable semantic scales in the absence of ground-truth labels using a calibration-first framework that uses a synthetic semantic reference space to align heterogeneous scorers before fusion. 3 Framework Our framework consist of two operational phases: (1) calibration using a synthetic ordinal reference space and (2) scoring and fusion of unlabeled instances11 1 Our code is available on GitHub: https://github.com/Lafayette-EshbaughSilveyra-Group/calibration-first. These phases are depicted in Figure 3. We first introduce the necessary preliminaries before describing each phase. subfigure [t]0.48 subfigure[t]0.48 Figure 1: Calibration. Subset-specific scorers are evaluated over the synthetic ordinal reference space to estimate normalization statistics (μj,σj)( _j, _j). Figure 2: Scoring and fusion. Scores from a new unlabeled instance are standardized using the normalization statistics and fused into the supervision score κ. Figure 3: Calibration-first supervision framework. During calibration, subset-specific scorers are evaluated over a synthetic ordinal reference space to estimate normalization statistics. During inference, scores for a new unlabeled instance are standardized using these statistics and fused into a calibrated supervision score κ. 3.1 Preliminaries 3.1.1 Notation and Assumptions We assume that the latent concept is characterized by n calibration features, contained in the set ℱF. For each feature i∈ℱi , let its set of possible values be denoted by Ai=a∣bi⪯a⪯ci,A_i=\a b_i a c_i\, where bib_i and cic_i denote the minimum and maximum values for feature i, respectively, and ⪯ denotes the feature’s natural ordering. The Cartesian product =∏i=1nAiA= _i=1^nA_i defines the ambient space of all possible configurations of the calibration features, where each element of A is an ordered n-tuple specifying one value for every calibration feature. Any observed dataset satisfies ⊆D , the synthetic ordinal reference space S satisfies S⊆S , and any new unlabeled instance satisfies x∈x . Another foundational assumption of the framework is that each calibration feature is monotonically related to the latent concept of interest. Although observations of these features through different evidence sources may be noisy, incomplete, or expressed on incomparable scales, movement along the ordered levels of any calibration feature corresponds to consistent movement along the latent concept. Under this assumption, the synthetic reference space induces a meaningful ordinal structure over the latent concept, and subset-specific observations can be interpreted as noisy evidence of an instance’s position within that structure. The monotonicity assumption defines a validity condition of the framework. If it is violated, the induced ordinal reference structure need not correspond to the latent concept, and no semantic validity guarantee is made for the resulting supervision score. 3.1.2 Subset-Specific Scorers It is necessary to estimate an instance’s position along the latent concept. The framework uses a collection of subset-specific scorers to determine this from heterogeneous observations. Each subset U⊆ℱU is selected a priori to represent a coherent portion of the latent concept. Its corresponding scorer is a function: FU:∏i∈UAi→ℝF_U: _i∈ UA_i A subset may correspond to a conventional data modality, such as text, images, or simulation output, but may also consist of a coherent group of related features or a concept-specific view of a shared evidence source. Accordingly, multiple features may be evaluated jointly by one scorer, and multiple scorers may operate on the same underlying observation to estimate different aspects of the latent concept. Scorers may be machine learning-based or heuristic. A machine learning scorer uses a learned model, such as a decision tree, multilayer perceptron, or large language model, to map the evidence associated with its designated feature subset to a scalar estimate. A heuristic scorer deterministically computes a scalar estimate using a predefined function derived from expert knowledge or application-specific rules, such as linear scoring rules, deterministic scaling functions, or rule-based systems. These scorers are used throughout both operational phases of the framework. During calibration, each scorer is evaluated over the evidence generated from the synthetic ordinal reference space introduced in Section 3.2.1 to estimate scorer-specific calibration statistics (Figure 3). During inference, the same scorers are applied to new unlabeled instances, and their outputs are standardized and fused to produce the final supervision score (Figure 3). The framework accommodates any combination of machine learning or heuristic scorers, provided that each produces a real-valued output whose ordering is consistent with the latent concept. To ensure meaningful aggregation across subset-specific scorers, scorer outputs must also share a common directionality, such that larger values consistently indicate greater correspondence to the latent concept or vice versa. 3.2 Operational Phases Given the monotonicity assumption and a collection of subset-specific scorers, the framework proceeds in two operational phases. The first evaluates each scorer over evidence generated from a synthetic ordinal reference space to estimate scorer-specific calibration statistics. The second applies the same scorers to previously unseen instances, standardizes their outputs using the corresponding calibration statistics, and fuses the standardized scores to produce a supervision score. 3.2.1 Calibration via a Synthetic Reference Space Calibration is performed using a synthetic ordinal reference space that defines a shared semantic reference for the latent concept across all subset-specific scorers. Because S is defined independently of the observed dataset, calibration is anchored to a domain-defined semantic range rather than to the empirical distribution of a particular sample. Beginning with the n calibration features introduced in Section 3.1.1, each feature is discretized into k ordered levels spanning an observable range. These levels must preserve the assumed monotonic relationship between the feature and the latent concept, but need not be uniformly spaced. Collectively, they define the synthetic ordinal reference space used for calibration. For a given application, the calibration levels may be obtained by partitioning each calibration feature into k ordered bins spanning its observable range. Let ℒ=L1,L2,…,LnL=\L_1,L_2,…,L_n\ denote the collection of calibration-level sets, where each Li=ℓi1,ℓi2,…,ℓikL_i=\ _i1, _i2,…, _ik\ contains the k ordered calibration levels associated with feature i. Each calibration-level set is a subset of the corresponding feature domain; that is, Li⊆AiL_i A_i. Calibration levels may be defined using domain knowledge, bin midpoints, empirical means, or other application-specific procedures. The framework is agnostic to this choice, requiring only that the resulting levels preserve the monotonic ordering assumed between the calibration feature and the latent concept. The synthetic ordinal reference space is constructed as the Cartesian product of the calibration-level sets: S=∏i=1nLi.S= _i=1^nL_i. By construction, S⊆S . Each element s∈Ss∈ S represents a calibration point corresponding to a particular combination of ordinal feature levels. Collectively, these points span the ordinal range of the latent concept and provide a shared reference space for aligning scorer outputs. For each calibration point, the evidence required by each subset-specific scorer is constructed from the corresponding feature configuration. Depending on the scorer, this evidence may take the form of a conventional modality, a derived representation, or a coherent group of calibration features. The resulting evidence is then evaluated using the same subset-specific scorer that will later be applied to unlabeled instances. This ensures that each scorer’s calibration statistics are estimated from outputs produced by the scorer itself over its corresponding evidence representation. In the worst case, calibration requires evaluating all |S|=kn|S|=k^n reference points with each of the m subset-specific scorers to estimate scorer-specific calibration statistics (μj,σj)( _j, _j), where j∈1,…,mj∈\1,…,m\ indexes the scorers. If multiple calibration points produce the same evidence representation for a scorer, the corresponding evaluation may be cached and reused, reducing the number of distinct scorer evaluations without changing the reference distribution. If scorer j has evaluation cost cjc_j, the worst-case total calibration cost scales as O(kn∑j=1mcj).O (k^n _j=1^mc_j ). This cost is incurred only once during calibration to estimate the normalization statistics. Thereafter, scoring a new instance requires evaluating each subset-specific scorer once, yielding a complexity of O(∑j=1mcj).O ( _j=1^mc_j ). The choice of k governs a tradeoff between calibration resolution and computational cost. Larger values of k produce a finer discretization of the calibration features, potentially yielding more representative normalization statistics, but increase the size of the reference space exponentially. In practice, k should be chosen to adequately resolve the ordinal structure of the latent concept while remaining computationally tractable given the cost of evaluating each scorer. These design choices result in a calibration procedure that is not necessarily tied to a particular dataset. By constructing the reference space with this approach, we obtain the following benefits: • Comparability: A calibrated value has the same intended interpretation across datasets. • Stability: Changes in sample composition do not themselves change the calibration mapping. • Transferability: Calibration learned from the synthetic reference space can be reused on new datasets rather than recomputed from each empirical distribution. • Cleaner separation: Changes in supervision scores reflect scorer responses to examples, rather than changes in the distribution used to normalize the scorers. 3.2.2 Supervision Score Fusion & Normalization Once the calibration phase is complete, a new unlabeled instance x∈x is evaluated by each subset-specific scorer, producing raw scores sj(x)j=1m,\s_j(x)\_j=1^m, where j∈1,…,mj∈\1,…,m\ indexes the scorers. Directly combining these raw scores risks privileging scorers with larger numerical ranges. Each score is therefore standardized using its corresponding calibration statistics μj _j and σj _j. The standardized output of scorer j is given by Equation 1, where μj _j and σj _j are obtained as described in Section 3.2.1. s^j(x)=sj(x)−μjσj s_j(x)= s_j(x)- _j _j (1) This transformation ensures that each subset-specific scorer’s contribution to the fused signal reflects its relative position within the corresponding reference distribution rather than its original numerical scale. Standardization therefore aligns differences in scorer location and positive linear scale while preserving the ordering of scorer outputs. Because the transformation has positive slope 1/σj1/ _j when σj>0 _j>0, scorers exhibiting zero variance across the calibration space are excluded from fusion because they provide no discriminative information regarding the latent concept. The ordinal information provided by any monotone scorer is therefore unchanged by standardization. This property is formalized in Proposition A.1, proved in Appendix A. The fused supervision score κ(x)κ(x) is computed as the arithmetic mean of the standardized scorer outputs according to Equation 2. κ(x)=1m∑j=1ms^j(x).κ(x)= 1m _j=1^m s_j(x). (2) In the absence of ground truth, we adopt equal weighting across the predefined subset-specific scorers as a neutral default because no empirical basis exists for estimating scorer reliability or assigning differential confidence to individual scorers. The calibration step removes differences in numerical scale, allowing each scorer to contribute equally according to its position within its own reference distribution. The set of subset-specific scorers is defined before calibration and held fixed during inference so that the weighting induced by the fusion rule remains consistent. Proposition A.3, proved in Appendix A, shows that the fused supervision score ranks instances according to aggregate calibrated support across scorers. Together with Proposition A.1, this establishes that calibration preserves scorer-specific ordinal information while fusion combines that information into a single supervision score. 4 Evaluation We evaluate the proposed framework through cross-domain validation and application-specific analyses. The cross-domain experiments evaluate the resulting supervision scores against held-out labels, while the application-specific experiments examine whether the framework exhibits the behavior characterized by Propositions A.1 and A.3. Given the framework’s conceptual relationship to latent-variable measurement, we additionally compare it with traditional psychometric methods in Appendix E. 4.1 Cross-Domain Validation We evaluate the performance correspondence of calibration-first fusion with held-out targets relative to several established and natural comparison baselines. Specifically, we carry out this assessment across three datasets from unrelated domains: Ames Housing [10], a regression benchmark; Breast Cancer Wisconsin [23], a medical classification benchmark; and Wine Quality [8], a physicochemical benchmark. For each dataset, we compare against empirical z-scoring, unsupervised PCA, Borda rank averaging, uncalibrated averaging, and the best individual subset-specific scorer’s value. For each dataset, we defined four heuristic subset-specific scorers whose feature domains are disjoint. Following Section 3.2.1, we constructed the synthetic ordinal reference space with k=5k=5 levels for each of the benchmark’s features. Full scorer definitions and calibration ranges are provided in Appendix C. Table 1 summarizes the results, with 95% confidence intervals for the primary metrics constructed using B=50,000B=50,000 bootstrap resamples. Pairwise method comparisons were conducted using paired bootstrap resampling, with the same resampled observations used for all methods within each replicate. To maintain a family-wise confidence level of at least 95% across three pairwise comparisons, we constructed Bonferroni-adjusted 98.33% confidence intervals. Method Primary [95% CI] Δ=κ−method =κ-method [95% paired CI] Bonf. [98.33% CI] Housing Calibrated Fusion (κ) 0.890[0.876, 0.903]0.890\,[0.876,\,0.903] — — Empirical z-Score Avg. 0.908[0.895, 0.919]0.908\,[0.895,\,0.919] −0.0172[−0.0225,−0.0122]-0.0172\,[-0.0225,\,-0.0122] [−0.0238,−0.0111][-0.0238,\,-0.0111] Borda Rank Averaging 0.905[0.893, 0.916]0.905\,[0.893,\,0.916] −0.0148[−0.0214,−0.0085]-0.0148\,[-0.0214,\,-0.0085] [−0.0230,−0.0072][-0.0230,\,-0.0072] Unsupervised PCA (PC1) 0.909[0.896, 0.919]0.909\,[0.896,\,0.919] −0.0181[−0.0233,−0.0131]-0.0181\,[-0.0233,\,-0.0131] [−0.0245,−0.0120][-0.0245,\,-0.0120] Uncalibrated Average 0.859[0.840, 0.877]0.859\,[0.840,\,0.877] +0.0311[0.0143, 0.0485]+0.0311\,[0.0143,\,0.0485] [0.0109, 0.0524][0.0109,\,0.0524] Best Single Scorer 0.81500.8150 — — Cancer Calibrated Fusion (κ) 0.984[0.976, 0.991]0.984\,[0.976,\,0.991] — — Empirical z-Score Avg. 0.983[0.974, 0.990]0.983\,[0.974,\,0.990] +0.0017[0.0004, 0.0030]+0.0017\,[0.0004,\,0.0030] [0.0002, 0.0034][0.0002,\,0.0034] Borda Rank Averaging 0.984[0.975, 0.992]0.984\,[0.975,\,0.992] +0.0002[−0.0034, 0.0041]+0.0002\,[-0.0034,\,0.0041] [−0.0041, 0.0051][-0.0041,\,0.0051] Unsupervised PCA (PC1) 0.987[0.978, 0.993]0.987\,[0.978,\,0.993] −0.0024[−0.0049,−0.0001]-0.0024\,[-0.0049,\,-0.0001] [−0.0055, 0.0003][-0.0055,\,0.0003] Uncalibrated Average 0.957[0.940, 0.972]0.957\,[0.940,\,0.972] +0.0268[0.0140, 0.0414]+0.0268\,[0.0140,\,0.0414] [0.0115, 0.0449][0.0115,\,0.0449] Best Single Scorer 0.97050.9705 — — Quality Calibrated Fusion (κ) 0.556[0.520, 0.590]0.556\,[0.520,\,0.590] — — Empirical z-Score Avg. 0.546[0.510, 0.582]0.546\,[0.510,\,0.582] +0.0093[0.0012, 0.0176]+0.0093\,[0.0012,\,0.0176] [−0.0005, 0.0194][-0.0005,\,0.0194] Borda Rank Averaging 0.549[0.513, 0.584]0.549\,[0.513,\,0.584] +0.0069[−0.0050, 0.0189]+0.0069\,[-0.0050,\,0.0189] [−0.0077, 0.0215][-0.0077,\,0.0215] Unsupervised PCA (PC1) 0.546[0.509, 0.581]0.546\,[0.509,\,0.581] +0.0097[−0.0061, 0.0257]+0.0097\,[-0.0061,\,0.0257] [−0.0094, 0.0293][-0.0094,\,0.0293] Uncalibrated Average 0.192[0.142, 0.240]0.192\,[0.142,\,0.240] +0.3641[0.3173, 0.4112]+0.3641\,[0.3173,\,0.4112] [0.3062, 0.4221][0.3062,\,0.4221] Best Single Scorer 0.47570.4757 — — Table 1: Cross-domain validation on the primary metric. Point estimates are reported with percentile-bootstrap 95% confidence intervals (B=50,000)(B=50,000). Pairwise intervals report Δ=κ−method =κ-method, with positive values favoring calibrated fusion. Bonferroni-adjusted 98.33% confidence intervals control the family-wise error rate across the three datasets separately for each baseline. The primary metric is Spearman’s ρ for Ames Housing and Wine Quality and AUROC for Breast Cancer Wisconsin. Additional metrics are available in Appendix C. Across all three domains, calibrated fusion improves over uncalibrated averaging on the primary metric. Moreover, Bonferroni-adjusted paired bootstrap confidence intervals exclude zero in each case, and achieves higher primary-metric point estimates than the best individual scorer. Furthermore, our results indicate that for empirical z-scoring, Borda rank averaging, and unsupervised PCA, performance is domain dependent. On Ames Housing, calibrated fusion is less than 0.020.02 lower than all three metrics, with all differences remaining distinguishable from zero after Bonferroni correction. On Breast Cancer Wisconsin, calibrated fusion is higher than empirical z-score averaging after correction by 0.00170.0017, while no adjusted difference is detected relative to Borda rank averaging or PCA. On Wine Quality, no adjusted difference is detected between calibrated fusion and any of the three metrics. In general, these results provide a clear indication that our methodology is a viable approach, given that no statistically significant differences were observed for the majority of the metrics. For the remaining metrics, the observed differences were relatively small. 4.2 Empirical Illustration of Framework Properties To exemplify our framework, we label 247 synthetic homes generated using the pipeline proposed by [12]. In this setting, the latent concept is residential energy retrofit priority, and the synthetic ordinal reference space is defined by four calibration features: HVAC heating COP, HVAC cooling COP, roof R-value, and wall R-value. Because no ground truth labels are available in this synthetic dataset, we aim to approximate the latent concept as a source of supervision using our framework. Four subset-specific scorers are employed. Each home inspection note is evaluated by an LLM to produce separate HVAC and insulation retrofit-need scores. Two heuristic scorers operate on the structured calibration features: an HVAC scorer jointly evaluates heating and cooling COP, while an insulation scorer jointly evaluates roof and wall R-values. For this dataset, the calibration reference space consists of all 625625 combinations formed by selecting one of the five ordered levels for each of the four calibration features (54)(5^4). These levels were selected using domain knowledge to span a broad range of residential efficiency conditions while preserving the expected ordinal relationship between each feature and retrofit need. The four scorer outputs are standardized and fused to produce concept-specific HVAC and insulation scores and an overall supervision score κ, with larger values indicating greater retrofit need. Additional implementation details, including the exact calibration values and their interpretation, are provided in Appendix D. We perform a sensitivity analysis to verify that each scorer responds monotonically to its designated input, and that calibration preserves this monotonicity. The inputs to one subset-specific scorer are varied across five ordered efficiency levels while the inputs to the remaining scorers are held fixed at the neutral third level. Level 1 represents the least efficient condition, while Level 5 represents the most efficient condition. Table 2 reports the averages of standardized scorer outputs in which negative values indicate retrofit need below the calibration-space mean, while positive values indicate retrofit need above the mean. Across all four experiments, the subset-specific supervision scores did not increase as efficiency improved, indicating that increasing efficiency did not produce an increase in estimated retrofit need. The calibrated scorer outputs also preserved every ordering expressed by their corresponding raw outputs. Across the complete calibration space, no ordinal preservation failures occurred in the 195,000195,000 pairwise comparisons evaluated for each scorer22 2 The equal scores observed between HVAC levels 4 and 5 and between insulation levels 1 and 2 reflect ties in the corresponding LLM outputs rather than ordering reversals introduced by calibration.. Hence, the ordinal preservation guarantee in Proposition A.1 is satisfied empirically for all four subset-specific scorers over the complete calibration space. Scorer Level 1 Level 2 Level 3 Level 4 Level 5 / HVAC 1.10 0.43 0.26 -0.57 -0.57 Structured / HVAC 1.26 0.76 0.26 -0.24 -0.74 Text / Insulation 0.94 0.94 -0.09 -0.24 -0.53 Structured / Insulation 0.91 0.41 -0.09 -0.59 -1.09 Table 2: Concept-specific supervision scores under scorer-level sensitivity analysis. Larger values indicate greater retrofit need. Evidence Structured Evidence HVAC Insulation 1 Level 1 2.10 1.94 Level 1 Level 5 0.10 -0.06 Level 5 Level 1 0.43 0.47 Level 5 Level 5 -1.57 -1.53 Table 3: Concept-specific supervision scores. Level 1 is least efficient and Level 5 is most efficient. Additionally, we evaluate the consistency of the outputs of subset-specific scorers in order to validate that their results reflect the quality of their inputs. This process consists of comparing four extreme combinations of the inputs in both formats: text-derived and structured evidence. In Table 3, we observe that jointly inefficient evidence produced the highest retrofit-priority signal for both concepts, while jointly efficient evidence produced the lowest. The cases with combined conditions fell between these endpoints and were asymmetric given that the calibrated text-derived and structured contributions differed in magnitude. These results are consistent with Proposition A.3. 4.3 Calibration Resolution Sensitivity Analysis The parameter k determines both the resolution of the synthetic calibration space and its computational cost. We evaluate whether dense calibration grids are necessary by measuring the tradeoff between calibration resolution and agreement with a high-resolution reference space. Specifically, we evaluate k∈2,3,5,7,9,11,13k∈\2,3,5,7,9,11,13\ against a k=15k=15 reference space. The k=15k=15 space was constructed by defining 15 ordered values for each calibration feature through interpolation between the predefined physical anchor values described in Appendix B. Lower-resolution spaces were constructed as nested subsets of this reference grid by selecting approximately equally spaced ordinal levels. This procedure preserves the endpoints and maintains comparable coverage of the feature ranges while reducing the number of calibration points evaluated. The selected levels and corresponding physical values are provided in Appendix B. As resolution increases, agreement with the k=15k=15 reference improves, while the number of calibration points grows exponentially, from 1616 points at k=2k=2 to 50,62550,625 points at k=15k=15. Despite this growth, the induced rankings stabilize at substantially lower resolutions. A resolution of k=7k=7 recovers a Spearman correlation of 0.99390.9939 with the k=15k=15 reference, while k=9k=9 achieves 0.99870.9987 correlation and 96.7%96.7\% top-10%10\% overlap using only 12.9%12.9\% of the reference calibration points. Figure 4 illustrates the resulting tradeoff between calibration cost and agreement. Increasing from k=9k=9 to k=15k=15 improves Spearman correlation by only 0.00130.0013, while increasing EnergyPlus computation time from approximately 2.62.6 hours to nearly 2020 hours. Thus, for the Synthetic Homes case study, k=9k=9 provides a near-reference agreement while avoiding the computational expense of the full reference grid. These results suggest that intermediate calibration resolutions may provide a favorable balance between computational cost and agreement with higher-resolution references. Figure 4: Spearman ρ against k=15k=15 vs EnergyPlus compute time for all tested values of k. 5 Conclusion In this work, we showed that heterogeneous evidence sources without ground-truth labels or a shared annotation space can be combined into a common supervision signal using our calibration-first framework. In a residential energy retrofit case study, the implementation exhibited the predicted monotonic behavior and remained stable across calibration resolutions. Additionally, we examined the tradeoff between computational cost and reference space resolution, finding that lower-resolution grids closely approximate higher-resolution results while substantially reducing computation. Across three benchmark datasets, calibrated fusion consistently outperformed uncalibrated averaging on the primary metric, with differences remaining distinguishable from zero after Bonferroni correction, and achieved higher primary-metric point estimates than the best individual scorer. Relative to empirical z-scoring, Borda rank averaging, and unsupervised PCA, performance was domain-dependent: calibrated fusion was within 0.02 of all three baselines on Ames Housing, with each difference remaining distinguishable from zero after correction. On Breast Cancer Wisconsin, one adjusted difference was detected, and on Wine Quality, no adjusted differences were detected. These findings suggest that our framework performs competitively with empirical z-scoring, Borda rank averaging, and unsupervised PCA. Unlike these metrics, our framework estimates scorer-specific calibration statistics from an externally defined synthetic ordinal reference space rather than from the observed sample. As a consequence, the proposed calibration procedure supports score comparability across cohorts, stability under changes in sample composition, transferability across datasets, and clearer separation between scorer behavior and the distribution of the observed sample. Our framework, similarly to other weak-supervision approaches, relies on domain knowledge, which is encoded through the selection and ordering of calibration features. Because these choices determine the structure of the calibration space, future work should examine how sensitive the resulting supervision is the assumptions made during its construction. Additional directions include incorporating scorer reliability into the fusion procedure and developing principled criteria for selecting calibration resolutions. Ultimately, supervision without ground truth requires more than combining heterogeneous signals; it requires making them comparable. Calibration-first fusion provides this missing step by aligning scorer outputs to a common scale before fusion. Acknowledgments The authors extend gratitude to Lafayette College for supporting this work by funding it through the EXCEL Scholars program and providing computational resources with the Firebird high performance computing cluster. This work also used Jetstream2 at Indiana University through allocation PHY240196 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program [5], which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. References [1] V. Agrawal, R. Pukdee, N. Balcan, and P. K. Ravikumar (2025) Learning from weak labelers as constraints. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 93153–93193. External Links: Link Cited by: §2. [2] A. Bacellar (2026) Calibrated fusion for heterogeneous graph-vector retrieval in multi-hop qa. Note: arXiv preprint arXiv:2603.28886 External Links: Document Cited by: §2. [3] T. Baltrusaitis, C. Ahuja, and L. Morency (2019) Multimodal machine learning: a survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (2), p. 423–443. External Links: ISSN 1939-3539, Link, Document Cited by: §2. [4] B. Boecking, W. Neiswanger, E. Xing, and A. Dubrawski (2021) Interactive weak supervision: Learning useful heuristics for data labeling. In International Conference on Learning Representations, Cited by: §2. [5] T. J. Boerner, S. Deems, T. R. Furlani, S. L. Knuth, and J. Towns (2023) ACCESS: advancing innovation: nsf’s advanced cyberinfrastructure coordination ecosystem: services & support. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good, PEARC ’23, New York, NY, USA, p. 173–176. External Links: ISBN 9781450399852, Link, Document Cited by: §5. [6] S. R. Cachay, B. Boecking, and A. Dubrawski (2021) End-to-end weak supervision. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), Cited by: §2. [7] L. A. Clark and D. Watson (2019) Constructing validity: New developments in creating objective measuring instruments.. Psychological Assessment 31 (12), p. 1412–1427. External Links: ISSN 1939-134X, 1040-3590, Document Cited by: §2. [8] P. Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis (2009) Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems 47 (4), p. 547–553. Cited by: §4.1. [9] L. J. Cronbach and P. E. Meehl (1955) Construct validity in psychological tests.. Psychological Bulletin 52 (4), p. 281–302. External Links: ISSN 1939-1455, 0033-2909, Document Cited by: §2. [10] D. De Cock (2011) Ames, Iowa: Alternative to the Boston Housing Data as an End of Semester Regression Project. Journal of Statistics Education 19 (3), p. 8. External Links: ISSN 1069-1898, Document Cited by: §4.1. [11] S. E. Embretson and S. P. Reise (2025) Item Response Theory: Foundations for Psychologists and Social Scientists. 2 edition, Routledge, New York. External Links: Document, ISBN 978-1-315-72655-7 Cited by: §2. [12] J. Eshbaugh, C. Tiwari, and J. Silveyra (2026) Synthetic homes: a multimodal generative ai pipeline for residential building data generation under data scarcity. Machine Learning with Applications 25, p. 100959. External Links: ISSN 2666-8270, Document Cited by: §4.2. [13] R. M. Furr and V. R. Bacharach (2008) Item Response Theory and Rasch Models. In Psychometrics: An Introduction, p. 314–334. External Links: ISBN 978-1-4129-2760-4 Cited by: §2. [14] L. R. Goldberg (1992) The development of markers for the Big-Five factor structure.. Psychological Assessment 4 (1), p. 26–42. External Links: ISSN 1939-134X, 1040-3590, Document Cited by: §E.1. [15] E. O. W. Kirkegaard and S. Jensen (2025) Psychometric analysis of the multifactor general knowledge test. OpenPsych. External Links: Document Cited by: §E.1. [16] K. Lee and M. C. Ashton (2004) Psychometric Properties of the HEXACO Personality Inventory. Multivariate Behavioral Research 39 (2), p. 329–358. External Links: ISSN 0027-3171, 1532-7906, Document Cited by: §E.1. [17] P. P. Liang, A. Zadeh, and L. Morency (2024) Foundations & trends in multimodal machine learning: principles, challenges, and open questions. ACM Computing Surveys 56 (10), p. 1–42. External Links: ISSN 1557-7341, Link, Document Cited by: §2. [18] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng (2011) Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML), Bellevue, Washington, USA, p. 689–696. Cited by: §2. [19] OpenPsychometrics (2019) Multifactor general knowledge test (mgkt). Note: https://openpsychometrics.org/tests/MGKT/Accessed 2026-06-15 Cited by: §E.1. [20] OpenPsychometrics (2019) OpenPsychometrics. Note: https://openpsychometrics.orgAccessed 2026-06-15 Cited by: §E.1. [21] G. Rasch (1980) Probabilistic models for some intelligence and attainment tests. Expanded ed edition, University of Chicago Press, Chicago. External Links: ISBN 978-0-226-70553-8 978-0-226-70554-5, LCCN BF431 .R338 1980 Cited by: §2. [22] A. Ratner, S. H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. Ré (2017) Snorkel: rapid training data creation with weak supervision. Proceedings of the VLDB Endowment 11 (3), p. 269–282. External Links: ISSN 2150-8097, Link, Document Cited by: §2. [23] W. Wolberg, O. Mangasarian, N. Street, and W. Street (1993) Breast cancer wisconsin (diagnostic). Note: UCI Machine Learning RepositoryDOI: https://doi.org/10.24432/C5DW2B Cited by: §4.1. [24] H. Yazbeck, J. Assaf, T. M. Lietman, J. D. Keenan, J. Jackson, J. Kalpathy-Cramer, J. P. Campbell, X. Song, and T. K. Redd (2026) Latent Supervision: A Method for Improved Performance and Calibration of Machine Learning Classification Models in Ophthalmology. Ophthalmology Science 6 (5), p. 101151. External Links: ISSN 26669145, Document Cited by: §2. [25] P. Yu, T. Ding, and S. H. Bach (2022) Learning from multiple noisy partial labelers. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, G. Camps-Valls, F. J. R. Ruiz, and I. Valera (Eds.), Proceedings of Machine Learning Research, Vol. 151, p. 11072–11095. Cited by: §2. [26] J. Zhang, C. Hsieh, Y. Yu, C. Zhang, and A. Ratner (2022) A Survey on Programmatic Weak Supervision. arXiv. External Links: 2202.05433, Document Cited by: §2. Appendix A Propositions & Proofs Proposition A.1 (Ordinal Preservation). Let h denote the latent concept, and let sjs_j be a subset-specific scorer such that h(a)<h(b) sj(a)<sj(b)h(a)<h(b) s_j(a)<s_j(b) for any instances a,ba,b. Let s^j s_j denote the standardized scorer obtained using calibration statistics with σj>0 _j>0. Then h(a)<h(b) s^j(a)<s^j(b).h(a)<h(b) s_j(a)< s_j(b). Thus, calibration preserves any ordinal structure in sjs_j that is consistent with the latent concept. Proof A.2. By assumption, h(a)<h(b)h(a)<h(b) implies sj(a)<sj(b)s_j(a)<s_j(b). The calibrated score is s^j(x)=sj(x)−μjσj. s_j(x)= s_j(x)- _j _j. Since σj>0 _j>0, this transformation is affine with positive slope 1/σj1/ _j. Therefore sj(a)<sj(b) sj(a)−μjσj<sj(b)−μjσj,s_j(a)<s_j(b) s_j(a)- _j _j< s_j(b)- _j _j, so s^j(a)<s^j(b) s_j(a)< s_j(b). Hence h(a)<h(b) s^j(a)<s^j(b)h(a)<h(b) s_j(a)< s_j(b). This result means that calibration neither creates nor destroys ordinal information. If subset-specific scorers are monotone observations of the latent concept, then calibrated scores preserve the latent ordering while expressing each score relative to the same reference distribution. Proposition A.3 (Fusion Consistency). Let κ(x)=1m∑j=1ms^j(x).κ(x)= 1m _j=1^m s_j(x). For any two instances a and b, define Δj=s^j(b)−s^j(a). _j= s_j(b)- s_j(a). Then κ(b)>κ(a)κ(b)>κ(a) if and only if ∑j:Δj>0Δj>∑j:Δj<0|Δj|. _j: _j>0 _j> _j: _j<0| _j|. Thus, the fused supervision score ranks instances according to aggregate calibrated support across subset-specific scorers. Proof A.4. Let Δj=s^j(b)−s^j(a) _j= s_j(b)- s_j(a) represent the calibrated difference for subset-specific scorer j. Let A=j∣Δj>0A=\j _j>0\ be the set of scorers that favor b, and let B=j∣Δj<0B=\j _j<0\ be the set of scorers that favor a. Define Δκ=κ(b)−κ(a)=∑j=1mΔjm. _κ=κ(b)-κ(a)= _j=1^m _jm. Since m>0m>0, Δκ>0⇔∑j=1mΔj>0. _κ>0 _j=1^m _j>0. Partitioning the sum into positive and negative contributions yields ∑j=1mΔj=∑j∈AΔj−∑j∈B|Δj|. _j=1^m _j= _j∈ A _j- _j∈ B| _j|. Therefore, Δκ>0⇔∑j∈AΔj>∑j∈B|Δj|. _κ>0 _j∈ A _j> _j∈ B| _j|. Hence fusion ranks b>ab>a if and only if the aggregate calibrated support for b exceeds the aggregate calibrated support for a. Appendix B Calibration Space Construction & Resolution Analysis This appendix provides additional details regarding the construction of the synthetic calibration space and the selection of calibration resolution. The calibration space defines the shared reference used to align subset-specific scorers prior to fusion. Because the number of calibration points grows exponentially with the number of calibration features and the number of ordinal levels selected for each feature, we also evaluate the computational tradeoff associated with increasing calibration resolution. B.1 Selection of Calibration Levels Each calibration feature is represented using an ordinal scale containing k ordered levels. Calibration levels are selected to span the feasible range of each feature while preserving the ordinal relationships between levels. For the Synthetic Homes case study, the calibration features are HVAC heating COP, HVAC cooling COP, roof R-value, and wall R-value. Each feature range is discretized independently, with lower values representing less favorable conditions and higher values representing more favorable conditions. For the main experiments, calibration levels are selected as approximately evenly spaced points across the feasible range of each calibration feature. This procedure provides a uniform ordinal reference space while avoiding arbitrary concentration of calibration points in specific regions of the feature space. For the resolution sensitivity analysis, the highest-resolution calibration space (k=15k=15) serves as the reference space. Lower-resolution calibration spaces are constructed as nested subsets of this reference grid by selecting approximately equally spaced ordinal levels. This preserves the endpoints and maintains similar coverage of the underlying feature ranges while reducing the number of calibration points evaluated. Table 4 reports the selected ordinal levels for each evaluated value of k. The corresponding physical parameter values used to construct the calibration space are provided in Table 5. Selected levels =2k=2 1, 15 k=3k=3 1, 8, 15 k=5k=5 1, 5, 8, 11, 15 k=7k=7 1, 3, 6, 8, 10, 13, 15 k=9k=9 1, 3, 5, 6, 8, 10, 11, 13, 15 k=11k=11 1, 2, 4, 5, 7, 8, 9, 11, 12, 14, 15 k=13k=13 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15 k=15k=15 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 Table 4: Selected ordinal levels from the k=15k=15 master calibration grid for each calibration resolution. Wall R Roof R Heating COP Cooling COP 1 4 10 0.7 1 2 5 13.333 0.7333 1.3333 3 6 16.667 0.7667 1.6667 4 7 20 0.8 2 5 8.5 22.5 0.825 2.25 6 10 25 0.85 2.5 7 11.5 27.5 0.875 2.75 8 13 30 0.9 3 9 14.75 32.5 0.9125 3.125 10 16.5 35 0.925 3.25 11 18.25 37.5 0.9375 3.375 12 20 40 0.95 3.5 13 23.333 43.333 0.9667 3.6667 14 26.667 46.667 0.9833 3.8333 15 30 50 1 4 Table 5: Physical values associated with the k=15k=15 calibration levels. B.2 Calibration Resolution and Computational Cost The number of calibration points increases exponentially with calibration resolution. Because the calibration space varies four features independently, the total number of calibration points is |S|=k4,|S|=k^4, where k denotes the number of ordinal levels assigned to each calibration feature. Consequently, increasing calibration resolution provides finer coverage of the reference space but requires additional scorer evaluations and, in the Synthetic Homes case study, additional EnergyPlus simulations. Figure 5 illustrates the growth in calibration points as a function of k. Although increasing resolution improves agreement with the highest-resolution reference space, the computational cost grows rapidly. Figure 5: Growth in the number of calibration points with calibration resolution k. The calibration space grows as k4k^4 because four calibration features are varied independently. B.3 Resolution Agreement and Diminishing Returns To evaluate the effect of calibration resolution on the resulting supervision signal, each lower-resolution calibration space is compared against the k=15k=15 reference space. Agreement is evaluated using rank correlation and differences between the resulting supervision scores. Table 6 reports the full results across the evaluated calibration resolutions, including calibration-space size, computation time, Spearman correlation, and top-10% overlap with the k=15k=15 reference. Points Runtime Spearman ρ Top-10% Overlap =5k=5 625 0.24h 0.9871 88.56 % k=7k=7 2401 0.94h 0.9939 92.42 % k=9k=9 6561 2.56h 0.9987 96.66 % k=15k=15 50625 19.76h 1.0000 100.00 % Table 6: Sensitivity results from selected k-values. Figure 6 shows the relationship between calibration resolution and Spearman correlation with the k=15k=15 reference. Agreement improves rapidly at lower resolutions before approaching saturation at higher resolutions. Figure 6: Spearman correlation between supervision scores generated at each calibration resolution and the k=15k=15 reference space. While increasing k improves agreement, these improvements require increasingly larger calibration spaces. Figure 4 illustrates the tradeoff between agreement with the reference space and EnergyPlus computation time. Increasing the resolution beyond moderate values provides only marginal improvements in rank preservation while requiring substantially greater computational expense. B.4 Implications for Calibration Resolution Selection The results demonstrate that increasing calibration resolution improves agreement with the highest-resolution reference space, but the marginal benefit decreases as the number of calibration points increases. Therefore, the appropriate value of k depends on the computational resources available and the precision required for a given application. Future work may investigate automated approaches for selecting calibration resolution, analogous to elbow-based methods used in clustering. Such approaches could identify a point at which additional calibration points provide insufficient improvement relative to their computational cost. Appendix C Cross-Domain Validation Details This appendix provides the scorer definitions and synthetic reference spaces used for the cross-domain validation experiments in Section 4.1. In all three datasets, the target variable was removed before supervision construction and used only for post hoc evaluation. The feature groups were selected a priori based on semantic correspondence among the dataset variables and their expected monotonic relationship with the latent concept. Calibration endpoints were manually specified to span low-to-high values for each feature. Neither the feature groupings nor the endpoint values were selected using the held-out target labels. C.1 Ames Housing The latent concept is residential property value. Four heuristic scorers represent complementary aspects of the concept: sscale(x) s_scale(x) =xGrLivArea+xTotalBsmtSF2, = x_GrLivArea+x_TotalBsmtSF2, (3) squality(x) s_quality(x) =xOverallQual+xOverallCond2, = x_OverallQual+x_OverallCond2, (4) smodernization(x) s_modernization(x) =xYearBuilt+xYearRemodAdd2, = x_YearBuilt+x_YearRemodAdd2, (5) sgarage(x) s_garage(x) =xGarageCars+xGarageArea2. = x_GarageCars+x_GarageArea2. (6) Larger values of each scorer indicate greater expected property value. The four ordered calibration dimensions and their endpoint values are shown in Table 7. Within each dimension, all associated features are varied jointly across five linearly spaced levels. Their Cartesian product produces 54=6255^4=625 synthetic calibration points. dimension Feature Level 1–Level 5 scale GrLivArea 600–3500 TotalBsmtSF 300–1800 and condition OverallQual 2–10 OverallCond 3–9 and modernization YearBuilt 1920–2010 YearRemodAdd 1950–2010 capacity GarageCars 0–4 GarageArea 0–1000 Table 7: Calibration dimensions and endpoint values used for Ames Housing. The SalePrice target is removed before scorer construction. Missing numerical values are replaced using feature medians. Although categorical features are ordinally encoded during preprocessing, the four scorers use only the numerical features listed above. C.2 Breast Cancer Wisconsin The latent concept is malignancy risk. The original dataset target is reoriented so that malignant cases are assigned one and benign cases are assigned zero. This target is withheld during supervision construction. Four heuristic scorers are defined as subset-specific means: sg(x)=1|ℱg|∑i∈ℱgxi,g∈size,shape,texture,worst,s_g(x)= 1|F_g| _i _gx_i, g∈\size,shape,texture,worst\, (7) where each feature set ℱgF_g comprises the corresponding variables listed in Table 8. Larger values indicate greater estimated malignancy risk. As in the Ames experiment, the features associated with each calibration dimension are varied jointly across five linearly spaced levels, yielding 54=6255^4=625 synthetic reference points. dimension Feature Level 1–Level 5 Mean radius 7–28 Mean perimeter 45–190 Mean area 150–2500 irregularity Mean compactness 0.02–0.35 Mean concavity 0–0.45 Mean concave points 0–0.20 and smoothness Mean texture 9–40 Mean smoothness 0.05–0.16 Mean symmetry 0.10–0.30 -case measurements Worst radius 8–36 Worst perimeter 50–250 Worst area 200–4300 Worst concavity 0–1.25 Worst concave points 0–0.30 Table 8: Calibration dimensions and endpoint values used for Breast Cancer Wisconsin. C.3 Wine Quality The latent concept is wine quality. Four heuristic scorers are defined using monotonic physicochemical properties: sacidity(x) s_acidity(x) =−xvolatile acidity+xcitric acid2, = -x_volatile acidity+x_citric acid2, (8) sadditives(x) s_additives(x) =−xchlorides+xsulphates2, = -x_chlorides+x_sulphates2, (9) spreservatives(x) s_preservatives(x) =−xfree SO2−xtotal SO22, = -x_free SO_2-x_total SO_22, (10) sprofile(x) s_profile(x) =xalcohol−xdensity2. = x_alcohol-x_density2. (11) Larger values indicate higher expected quality. Each of the four corresponding calibration dimensions is varied jointly across five linearly spaced levels, yielding 54=6255^4=625 synthetic calibration points. The calibration endpoints are shown in Table 9. Target quality labels are removed prior to supervision construction and used solely for post hoc evaluation. dimension Feature Level 1–Level 5 Volatile acidity 1.58–0.12 Citric acid 0.0–1.0 Chlorides 0.611–0.012 Sulphates 0.33–2.0 Free sulfur dioxide 72.0–1.0 Total sulfur dioxide 289.0–6.0 Alcohol 8.0–14.9 Density 1.00369–0.99007 Table 9: Calibration dimensions and endpoint values used for Wine Quality. C.4 Calibration, Fusion, and Baselines For each scorer j, the reference mean and population standard deviation are computed over the corresponding 625 synthetic scorer outputs. Observed scores are standardized using these reference statistics and averaged with equal weight to obtain κ. The uncalibrated baseline is the direct arithmetic mean of the four raw scorer outputs. We additionally compare against empirical z-score averaging, Borda rank averaging, and the first principal component obtained using unsupervised PCA. Each raw scorer is also evaluated individually. Ames Housing and Wine Quality are evaluated primarily using Spearman correlation, with Kendall correlation and top-10% overlap reported as secondary and additional metrics. Breast Cancer Wisconsin is evaluated primarily using AUROC, with average precision and Spearman correlation with malignancy status reported as secondary and additional metrics. Primary results and paired comparisons are reported in Table 1; the complete secondary and additional results are reported in Table 10. Method Secondary Additional Housing Calibrated Fusion (κ) 0.7197 73.97% Empirical z-Score Avg. 0.7444 78.08% Borda Rank Averaging 0.7359 73.29% Unsupervised PCA (PC1) 0.7456 78.08% Uncalibrated Average 0.6890 73.29% Best Single Scorer 0.6365 68.49% Cancer Calibrated Fusion (κ) 0.9770 0.8111 Empirical z-Score Avg. 0.9743 0.8084 Borda Rank Averaging 0.9774 0.8108 Unsupervised PCA (PC1) 0.9812 0.8151 Uncalibrated Average 0.9456 0.7662 Best Single Scorer 0.9616 0.7880 Quality Calibrated Fusion (κ) 0.4403 44.38% Empirical z-Score Avg. 0.4336 42.50% Borda Rank Averaging 0.4339 46.25% Unsupervised PCA (PC1) 0.4349 38.75% Uncalibrated Average 0.1511 20.62% Best Single Scorer 0.3718 34.38% Table 10: Secondary and additional cross-domain validation metrics. For Ames Housing and Wine Quality, the secondary metric is Kendall’s τ and the additional metric is top-10% overlap. For Breast Cancer Wisconsin, the secondary metric is average precision and the additional metric is Spearman correlation with malignancy status. Bold values indicate the highest observed value within each dataset and metric. Finally, Table 11 reports the primary-metric performance of each individual subset-specific scorer. Subset-Specific Scorer Metric Value Housing Physical scale Spearman ρ 0.8150 Age and modernization 0.6814 Garage capacity 0.6496 Quality and condition 0.6197 Cancer Worst-case features AUROC 0.9705 Wisconsin Size 0.9391 Shape irregularity 0.9325 Texture and smoothness 0.7775 Quality Profile Spearman ρ 0.4757 Additives 0.3911 Acidity 0.3244 Preservatives 0.1724 Table 11: Primary validation performance of each subset-specific scorer across all three evaluation datasets. Appendix D Synthetic Homes Implementation Details This appendix describes the implementation of the proposed framework for the Synthetic Homes case study presented in Section 4.2. D.1 Calibration Features and Subset-Specific Scorers The latent concept is residential retrofit priority. We characterize this concept using four calibration features associated with two building-system categories: HVAC heating COP, HVAC cooling COP, roof R-value, and wall R-value. Increasing HVAC COP and insulation R-value correspond to increasing building efficiency and, consequently, decreasing retrofit need. The implementation employs four subset-specific scorers associated with two feature subsets. Two text-derived scorers are produced by an LLM that evaluates a single home inspection note and returns separate estimates of HVAC and insulation retrofit need. Two structured scorers are heuristic functions operating on the corresponding numerical feature subsets. The structured HVAC scorer jointly evaluates heating and cooling COP, while the structured insulation scorer jointly evaluates roof and wall R-values. We denote the four raw scorer outputs by stext,HVAC, s_text,HVAC, stext,insulation, s_text,insulation, sstructured,HVAC, s_structured,HVAC, sstructured,insulation. s_structured,insulation. All four outputs are oriented so that larger values indicate greater retrofit need. D.2 Synthetic Ordinal Reference Space Each calibration feature is assigned five ordered efficiency levels, shown in Table 12. Their Cartesian product yields 54=6255^4=625 calibration points. Feature Increasing Efficiency ⟶ heating COP 0.7 0.8 0.9 0.95 1.0 HVAC cooling COP 1.0 2.0 3.0 3.5 4.0 Roof R-value 10 20 30 40 50 Wall R-value 4 7 13 20 30 Table 12: Calibration levels used to construct the Synthetic Homes ordinal reference space. The calibration values were selected using domain knowledge rather than estimated from the empirical distribution of the Synthetic Homes dataset. Their purpose is to define an interpretable reference space spanning clearly inefficient, intermediate, and highly efficient conditions while preserving the expected monotonic relationship between each feature and retrofit need. They should not be interpreted as building-code thresholds, normative retrofit recommendations, or representative quantiles of the existing housing stock. The values are held fixed across the experiments to provide a common reference distribution for estimating scorer-specific calibration statistics. D.3 Whole-Home Inspection Notes For each calibration point, the heating and cooling COP levels are collapsed into a single HVAC note level. Similarly, the wall and roof R-value levels are collapsed into a single insulation note level. For two constituent levels a,b∈1,…,5a,b∈\1,…,5\, the corresponding note level is ℓ(a,b)=⌊a+b2+12⌋, (a,b)= a+b2+ 12 , (12) which applies half-up rounding to their mean. For example, constituent levels 2 and 3 produce note level 3. One HVAC component and one insulation component are selected using the resulting levels and concatenated to form a single whole-home inspection note. The text components used in the experiments are shown in Table 13. HVAC Component Insulation Component 1 The home has an aging central HVAC system showing substantial exterior corrosion and poor operating efficiency. Attic insulation is minimal, with exposed joists visible throughout, causing significant heat loss. 2 The HVAC system appears to be in working condition but is an older standard-efficiency model. No signs of added insulation were observed in the basement ceiling, suggesting potential energy inefficiency. 3 The home has a functioning central HVAC system of moderate age and efficiency, with no major defects observed but no recent high-efficiency upgrades. Walls appear to be adequately insulated based on construction year, though no upgrades were observed. 4 The HVAC system was recently replaced with a standard high-efficiency model and is expected to operate efficiently. Blown-in insulation is present in the attic to a depth of approximately 10 inches, providing good thermal resistance. 5 A state-of-the-art HVAC system with smart thermostats and variable-speed compressors was recently installed, maximizing energy efficiency. High-performance spray foam insulation was installed throughout the walls, attic, and basement, providing maximum energy efficiency. Table 13: Text components used to construct home inspection notes at each ordinal efficiency level. The complete note presents the insulation component followed by the HVAC component. The LLM receives this home note in a single request and returns two scores in the interval [0,1][0,1]: one for HVAC retrofit need and one for insulation retrofit need. The model is instructed to evaluate the two categories separately using all relevant evidence in the complete note. Because each text component depends only on a collapsed concept-level condition, multiple points in the 625625-point structured reference space produce the same textual realization. The five HVAC conditions and five insulation conditions yield 52=255^2=25 distinct home notes. LLM outputs are cached by the complete note, so each distinct note is evaluated once and its two outputs are reused for every calibration point that produces the same note. The repeated outputs nevertheless retain their multiplicity in the 625625-point calibration distribution. D.4 Structured Scorers For a numerical feature q with ordered calibration anchors v1<v2<⋯<v5,v_1<v_2<·s<v_5, the implementation maps q onto an efficiency level e(q)∈[1,5]e(q)∈[1,5]. Values at or below v1v_1 are assigned efficiency level 1, and values at or above v5v_5 are assigned efficiency level 5. Between adjacent anchors viv_i and vi+1v_i+1, the efficiency level is obtained by linear interpolation: e(q)=i+q−vivi+1−vi,vi≤q≤vi+1.e(q)=i+ q-v_iv_i+1-v_i, v_i≤ q≤ v_i+1. (13) The corresponding feature-specific retrofit-need value is r(q)=6−e(q),r(q)=6-e(q), (14) so that increasing efficiency produces decreasing retrofit need. The structured HVAC score is the mean of the heating- and cooling-specific retrofit-need values: sstructured,HVAC(x)=r(qheat(x))+r(qcool(x))2.s_structured,HVAC(x)= r(q_heat(x))+r(q_cool(x))2. (15) The structured insulation score is defined analogously: sstructured,insulation(x)=r(qroof(x))+r(qwall(x))2.s_structured,insulation(x)= r(q_roof(x))+r(q_wall(x))2. (16) At the five calibration anchors, each feature-specific retrofit-need value is respectively 5,4,3,2,15,4,3,2,1. The interpolation permits the same scorers to operate on values that fall between the predefined calibration anchors. No EnergyPlus energy-consumption output or held-out retrofit label is used to construct these structured scores. D.5 Text Scorer The text scorer uses GPT-4.1-mini with temperature set to zero. It receives the complete home inspection note and is instructed to return a JSON object containing stext,HVAC∈[0,1]andstext,insulation∈[0,1].s_text,HVAC∈[0,1] s_text,insulation∈[0,1]. A score of zero indicates that the corresponding retrofit is judged definitely unnecessary, while a score of one indicates that it is judged definitely necessary. Intermediate values represent partial need or uncertainty. The response is parsed and validated to ensure that both required fields are present and fall within the specified interval. D.6 Calibration and Fusion Each scorer is evaluated over all 625625 calibration points. For subset-specific scorer j, the implementation computes the mean μj=1625∑x∈sj(x) _j= 1625 _x s_j(x) and population standard deviation σj=1625∑x∈(sj(x)−μj)2. _j= 1625 _x (s_j(x)- _j )^2. A scorer output for an evaluated instance is standardized as s^j(x)=sj(x)−μjσj. s_j(x)= s_j(x)- _j _j. (17) The text-derived and structured outputs are first fused within each building-system category: κHVAC(x) _HVAC(x) =12(s^text,HVAC(x)+s^structured,HVAC(x)), = 12 ( s_text,HVAC(x)+ s_structured,HVAC(x) ), (18) κinsul(x) _insul(x) =12(s^text,insul(x)+s^structured,insul(x)). = 12 ( s_text,insul(x)+ s_structured,insul(x) ). (19) The overall supervision signal is the equal-weight mean across all four calibrated scorers: κ(x)=14(CLOSEs^text,HVAC(x)+s^structured,HVAC(x)OPEN+s^text,insul(x)+s^structured,insul(x)), splitκ(x)= 14 (& s_text,HVAC(x)+ s_structured,HVAC(x)\\ &+ s_text,insul(x)+ s_structured,insul(x) ), split (20) which equivalently evaluates to κ(x)=12(κHVAC(x)+κinsul(x))κ(x)= 12 ( _HVAC(x)+ _insul(x) ). Larger values of κHVAC _HVAC, κinsulation _insulation, and κ indicate greater estimated retrofit need. Because these quantities are formed from standardized outputs, negative values indicate retrofit need below the relevant calibration-space mean rather than negative physical retrofit need. The same note-construction procedure and scoring functions used to estimate the calibration statistics are applied during inference. Each runtime scorer output is therefore standardized using the reference distribution generated by that same scorer. Appendix E Exploratory Psychometric Analysis To further investigate potential connections between calibration-first supervision construction and latent-variable measurement, we conducted an exploratory analysis on several psychometric datasets. Unlike the primary validation experiments in Section 4.1, these datasets do not contain naturally heterogeneous evidence sources operating on incompatible scales. Instead, they consist of multiple subscales designed to measure related psychological or cognitive constructs. Consequently, these experiments should be interpreted as exploratory evidence rather than as direct validation of the proposed framework. Our goal was to evaluate whether calibrated fusion can recover latent structure extracted directly from item-level responses. For each dataset, domain or subscale scores were treated as evidence sources and fused using the procedure described in Section 3. The resulting fused score κ was then compared against the first principal component (PC1) computed from the complete item-response matrix. E.1 Datasets We considered three datasets, all sourced from the Open Psychometrics data repository [20]. BIG5. The BIG5 personality inventory contains 50 items organized into five domains: Extraversion, Neuroticism, Agreeableness, Conscientiousness, and Openness. Domain scores were treated as evidence sources and fused into a single score κ. The BIG5 dataset consists of responses to the IPIP Big Five Factor Markers questionnaire [14]. HEXACO. The HEXACO personality inventory [16] contains 240 items organized into six domains: Honesty–Humility, Emotionality, Extraversion, Agreeableness, Conscientiousness, and Openness. Domain scores were again treated as evidence sources and fused into κ. MGKT. The Multifactor General Knowledge Test (MGKT) [19, 15] contains objective knowledge questions spanning multiple domains of general knowledge. Domain-level scores were treated as evidence sources and fused into κ. For each dataset, the fused score was compared against the first principal component extracted from the underlying item-level response matrix. E.2 Results Table 14 summarizes the results. Method Spearman ρ with Item-Level PC1 5 Calibrated Fusion (κ) 0.615 HEXACO Calibrated Fusion (κ) 0.252 MGKT Calibrated Fusion (κ) 0.987 Table 14: Agreement between calibrated fusion and the dominant latent factor extracted from item-level responses. Several observations emerge. First, MGKT exhibits an extremely strong correspondence between κ and the dominant latent factor extracted from item responses, with a Spearman correlation of 0.9870.987. This suggests that calibrated fusion almost perfectly reproduces the ranking implied by item-level latent structure. Second, BIG5 demonstrates a moderate relationship between κ and PC1. This is unsurprising because the Big Five framework was designed to represent several distinct personality dimensions rather than a single latent trait. Third, HEXACO exhibits substantially weaker agreement. HEXACO was explicitly developed as a multidimensional personality framework containing conceptually distinct domains, and therefore a single fused score is not expected to align strongly with a one-dimensional latent representation. E.3 Discussion These findings are consistent with established psychometric theory. Instruments characterized by a strong dominant latent factor should exhibit stronger agreement between fused scores and latent-variable estimates. Conversely, instruments designed to measure multiple relatively independent constructs should exhibit weaker agreement because no single latent dimension adequately summarizes the entire instrument. An important observation is that calibrated fusion and uncalibrated averaging produced nearly identical results across all three psychometric datasets. Unlike the Ames Housing and Breast Cancer Wisconsin experiments, the psychometric evidence sources were already measured on highly comparable scales and used similar response formats. As a result, little scale mismatch existed between evidence sources, leaving relatively little work for the calibration procedure to perform. This behavior should not be interpreted as a failure of calibration. Rather, it demonstrates that the method behaves sensibly when sources are already aligned and that calibration becomes most important when heterogeneous evidence sources operate on substantially different scales. Overall, these experiments provide exploratory evidence that calibration-first fusion is connected to latent-variable modeling. In particular, the MGKT results suggest that domain-level fusion can recover latent structure similar to that obtained from item-level analysis. However, because the psychometric evidence sources were already expressed on comparable scales, these experiments do not directly evaluate the primary contribution of this paper—calibration across heterogeneous information sources—and are therefore presented as supplementary evidence rather than as primary validation.