Paper deep dive
WhiteCon: Semi-Supervised Domain Adaptation Regression Through Whitening Transform and Dual Consistency
Se Jin Sim, Seoung Bum Kim
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Domain adaptation is crucial for addressing distributional shifts that degrade model performance across domains. While most existing research has centered on classification, semi-supervised domain adaptation regression (SSDAR) for continuous-output tasks remains largely unexplored, particularly in practical scenarios with limited labeled target data. To address this gap, we propose semi-supervised domain adaptation regression through whitening transform and dual consistency (WhiteCon), which combines domain-specific whitening transform (DWT) and dual consistency regularization to enhance training stability and domain adaptation. DWT reduces the variance of the model parameters by transforming the feature covariance matrix into an identity matrix, thus stabilizing training under ordinary least squares assumptions. In addition, variance consistency regularization, as part of dual consistency regularization, aligns the variances of weak, strong, and mixup-augmented features to improve resilience against augmentation-induced perturbations. Empirical evaluations on various benchmark datasets under SSDAR settings demonstrate that the proposed WhiteCon achieves state-of-the-art performance compared to existing methods, effectively addressing domain shifts in regression tasks. The code for WhiteCon is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.34078v1
- Canonical: https://arxiv.org/abs/2609.34078v1
Trouble viewing inline? Open PDF directly β
Full Text
44,705 characters extracted from source content.
Expand or collapse full text
WhiteCon: Semi-Supervised Domain Adaptation Regression Through Whitening Transform and Dual Consistency Se Jin Sim [0009β0000β6028β1690] and Seoung Bum Kim [0000β0002β2205β8516] School of Industrial and Management Engineering Korea University, Seoul, Republic of Korea ssj259, sbkim1@korea.ac.kr Abstract. Domain adaptation is crucial for addressing distributional shifts that degrade model performance across domains. While most existing research has centered on classification, semi-supervised domain adaptation regression (SSDAR) for continuous-output tasks remains largely unexplored, particularly in practical scenarios with limited labeled target data. To address this gap, we pro- pose semi-supervised domain adaptation regression through whitening transform and dual consistency (WhiteCon), which combines domain-specific whitening transform (DWT) and dual consistency regularization to enhance training stabil- ity and domain adaptation. DWT reduces the variance of the model parameters by transforming the feature covariance matrix into an identity matrix, thus stabi- lizing training under ordinary least squares assumptions. In addition, variance consistency regularization, as part of dual consistency regularization, aligns the variances of weak, strong, and mixup-augmented features to improve resilience against augmentation-induced perturbations. Empirical evaluations on various benchmark datasets under SSDAR settings demonstrate that the proposed White- Con achieves state-of-the-art performance compared to existing methods, effec- tively addressing domain shifts in regression tasks. The code for WhiteCon is available at https://github.com/sejin-sim/WhiteCon. Keywords: semi-supervised domain adaptation, regression, feature whitening, consistency regularization, variance alignment. 1 Introduction Domain adaptation is a critical area of research that addresses performance degradation caused by distribution shifts between different domains. This topic has recently gained substantial attention in various fields, such as computer vision, natural language pro- cessing, and medical data analysis [1-3]. When a model trained on a source domain is applied to a target domain, significant differences in data distributions across domains can degrade the model's generalization performance [4, 5]. This issue is particularly challenging when labeled data are limited, emphasizing the need for domain adaptation that can reduce distribution gaps to ensure reliable performance in diverse settings [6]. 2 Fig. 1. Illustration of the OLS problem in SSDAR using WhiteCon. ο represents the parameters of the regressor. (a) Domain shift between source and target data under OLS. (b) WhiteCon mit- igates the domain shift in SSDAR under OLS. Most existing research on domain adaptation has focused on classification problems, leading to the development of various methods for reducing domain differences. Some methods, such as statistical moment alignment, aim to minimize distributional discrep- ancies [7, 8]. Another effective strategy, adversarial training, uses a minimax game be- tween feature extractors and domain classifiers to reduce domain discrepancies, yield- ing promising results [9, 10]. Although some classification methods can be extended to regression, their reliance on class probability distributions is poorly suited to continu- ous outputs, creating a significant challenge for regression-based domain adaptation [11]. In response to these challenges, unsupervised domain adaptation regression (UDAR) methods have been proposed [12, 13]. UDAR methods aim to reduce domain discrep- ancies by using only labeled source data and unlabeled target data, making them par- ticularly useful when labeled target data is scarce. Several methods, including singular value decomposition and distribution matching in feature space, have been used to align source and target domains [11, 14]. However, even small amounts of labeled target data significantly improve performance and are often feasible to acquire [13, 15]. As a result, semi-supervised domain adaptation (SSDA) integrates small amounts of labeled target data with labeled source and unlabeled target data [16]. Similar to unsu- pervised domain adaptation (UDA), SSDA methods for classification are also challeng- ing to apply directly to regression tasks because they often rely on entropy measures from class prediction probabilities, which do not align well with continuous regression outputs. The earliest proposed semi-supervised domain adaptation regression (SSDAR) method [17] aligned statistical moments and used graph Laplacians for unlabeled data, while recent method [18] used domain-specific regressors for invariant risks and adver- sarial learning to prevent the model from distinguishing between source and target dis- tributions. Although these methods can be applied to some extent without specific re- gression assumptions, they rely on classification-based approaches, which may limit their effectiveness in addressing the challenges of regression tasks. To address these challenges, it is crucial to develop a method that directly considers the intrinsic proper- ties of regression, rather than simply adapting classification techniques. Fig. 1 illustrates the challenges posed by domain shifts in SSDAR, highlighting the distributional gap between source and target domains. To tackle these challenges, we introduce semi-supervised domain adaptation regression through whitening transform and dual consistency (WhiteCon). WhiteCon effectively reduces domain shift and ad- dresses the unique challenges of regression tasks by combining domain-specific whit- ening transform (DWT) and dual consistency regularization. The first component, DWT, eliminates correlations between features by transform- ing the feature covariance matrix into an identity matrix. Building upon the success of DWT in classification tasks [19], to our knowledge we are the first to apply it to regres- sion tasks, providing a mathematical justification under ordinary least squares (OLS) assumptions. Unlike classification where DWT only aids feature alignment, in regres- sion DWT improves model performance through regressor parameter variance reduc- tion. Second, we introduce dual consistency regularization, which enforces consistency across both predictions and feature representations. Specifically, the variance con- sistency regularization enforces variance alignment among features extracted from weakly, strongly, and mixup-augmented samples of unlabeled target data. By matching the variances of strong and mixup augmentations to those of weak augmentations, the model achieves consistency across augmentation intensities and improves robustness. In summary, our method combines the variance-reducing effect of DWT with the ro- bustness provided by dual consistency regularization, creating an effective method for SSDAR. The contributions of this study are summarized as follows: (1) To the best of our knowledge, this study presents the first application of DWT to regression tasks with mathematical justification under the OLS framework. We demonstrate that DWT reduces the variance of regression parameters, thus improving task performance and providing theoretical support for the effective- ness of the proposed method. (2) We introduce feature variance consistency regularization that aligns feature variances across different augmentations to improve robustness in SSDAR. This variance alignment promotes consistent feature distributions, reducing sensitivity to perturbations and enhancing generalization. Unlike existing meth- ods that rely on complex formulations or focus on only one aspect inde- pendently, our approach is simple and addresses both feature and prediction consistency simultaneously in an intuitive manner. (3) We propose WhiteCon, a simple yet effective framework that combines DWT and dual consistency regularization for SSDAR. Our method achieves state-of- the-art performance on various benchmark datasets and demonstrates its supe- riority through extensive evaluations, including ablation studies, regressor pa- rameter variance analysis, varying proportions of labeled target data, and fea- ture visualizations. 4 2 Related work 2.1 Domain Adaptation for Classification UDA has been widely studied, particularly for classification tasks. One common ap- proach, moment matching, aims to reduce distributional discrepancies by aligning the statistical moments of distributions [7, 8]. Maximum mean discrepancy (MMD) [20], a prominent technique, measures divergence by comparing their features in a reproducing kernel Hilbert space. However, MMD focuses on aligning input feature distributions but may overlook the complex dependencies between input features and continuous targets in regression tasks. Another approach is adversarial learning, inspired by generative adversarial net- works [21], which produces domain-invariant representations by training features that are difficult to distinguish between domains [9]. This method minimizes distributional differences through a competitive process with a domain discriminator [10]. Domain- adversarial neural networks (DANN) [6] align distributions through a gradient reversal layer. However, when domain discrepancies are substantial or highly complex, domain- invariant representations alone may not suffice to reduce the gap effectively. The availability of labeled target data has increased interest in SSDA [22]. Saito et al. [13] demonstrated that UDA methods often struggled in SSDA settings. Various semi-supervised methods have applied consistency regularization, which aligns model predictions across augmented views of the same data to enhance stability and perfor- mance [23]. However, while effective for classification within SSDA, consistency reg- ularization's effectiveness in regression remains unproven. Most SSDA methods for classification rely on computing the entropy of class pre- diction probabilities using softmax [13, 24]. This reliance limits their applicability to regression tasks, which require models capable of handling continuous value predic- tions rather than class probability distributions. 2.2 Domain Adaptation for Regression Cortes and Mohri [25] provided a theoretical analysis of domain adaptation for regres- sion. Over time, several methods have been introduced for this task, often relying on importance weighting in non-deep learning models or focusing on creating invariant representations of features [18]. Recently, UDAR methods were proposed to address regression using only labeled source and unlabeled target data. Chen et al. [11] proposed representation subspace distance (RSD) for domain adaptation regression, which uses singular value decompo- sition to produce orthogonal bases while preserving feature scale. However, our empir- ical analysis reveals that the relationship between feature scale and performance was not always consistent. Nejjar et al. [14] proposed domain adaptation regression by aligning the inverse Gram matrices (DARE-GRAM) based on the closed-form OLS problem. However, deriving the pseudo-inverse matrix required specifying the low- rank property as a hyperparameter, potentially leading to excessive hyperparameter tun- ing. Wu et al. [26] proposed distribution-informed neural networks that build distribu- tion-aware relationships using neural tangent kernel theory, but assume infinite-width networks that may not hold in practice. Dhaini et al. [27] proposed dictionary learning for subspace mapping, though performance depends on dictionary size and initializa- tion. SSDAR methods aim to address realistic scenarios where a small amount of labeled target data is available for model training. Singh and Chakraborty [17] proposed deep domain adaptation for regression (DeepDAR), the first SSDAR method using deep learning. They used MMD for distribution alignment and graph Laplacians for unla- beled target data. However, MMD only aligns feature distributions without ensuring consistent regression outputs, creating a weak connection to regression performance. Li et al. [18] proposed learning invariant representations and risks (LIRR) for both re- gression and classification, using domain-specific predictors and DANN. While theo- retically grounded, LIRR's joint optimization prevents it from addressing regression- specific parameter stability under domain shift. 3 Proposed Methods 3.1 Preliminaries Roy et al. [19] proposed DWT for reducing the distribution gap between source and target domains by eliminating feature correlations in unsupervised domain adaptation classification (UDAC). Unlike batch normalization, DWT applies batch whitening to transform the feature covariance matrix into an identity matrix. This ensures that fea- tures from both domains are mapped into a spherical distribution, which helps the model generalize across both domains. Let ο represent the original feature vectors, and ο ο denote the decorrelated feature by applying a whitening matrix ο. The whitening transformation is as follows: ο ο =ο ( οβο Μ ) , (1) where ο Μ is the mean vector of the features, and ο is computed from the covariance matrix Ξ£= ( οβο Μ )( οβο Μ ) ο . Using the Cholesky decomposition, the covariance ma- trix Ξ£ is decomposed as Ξ£=ο ο , where ο is a lower triangular matrix. The whitening matrix is then obtained as ο=ο ο . By applying ο, the covariance matrix of the whit- ened feature ο ο becomes the identity matrix ο, as shown below: Covο ο ο=ο ο ο ο ο = ο ( οβο Μ )( οβο Μ ) ο ο ο =οΞ£ο ο =ο ο ο ο (ο ο ) ο =ο. (2) Consequently, DWT removes correlations between features and aligns the distributions between the source and target domains, improving generalization and reducing the im- pact of domain shifts. Let ο = ( ο ο ο ,ο ο ο ) ο ο represent a set of labeled data from the source domain, where ο ο ο denotes the continuous values associated with samples ο ο ο . Similarly, let ο ο = ο(ο ο ο )ο ο ο represent a set of unlabeled data from the target domain. Additionally, let 6 ο ο =(ο ο ο ,ο ο ο ) ο ο represent a small set of labeled data in the target domain, where οβͺο and typically οβͺο, indicating that only a small amount of target data is la- beled. A key challenge is domain shift, where the source domain distribution ο ( ο ο ) differs from the target domain distribution ο (ο,ο) ο, i.e., ο ( ο ο ) β ο (ο,ο) ο. This domain shift complicates model performance in the target domain, necessitating adap- tation to address the distribution gap. Given the data, the feature extractor ο(β) transforms the source and target samples into features. The regressor ο(β), implemented as a fully connected layer, uses these features to generate predictions for both domains, denoted as ο ο = ο(ο(ο ο )) and ο ο = ο(ο(ο ο )). To optimize the model using the labeled samples from both the source and target domains, the supervised loss ο ο is defined as follows: ο ο = ο ο ο ο ο β( ο ο ο βο ο ο ) ο ο ο + ο ο β ο ο ο βο ο ο ο ο ο ο ο, (3) where ο and ο represent the numbers of labeled samples in the source and target do- mains, respectively. 3.2 Motivations Once features ο are extracted, predicting the target ο using regressor parameters ο can be formulated as an OLS problem [28]: ο=ο+ο, (4) where ο is a random error. We construct our regressor ο(β) as a single linear layer. This simplifies the mapping from features to predictions, clarifying the OLS problem. Under the OLS assumption, the variance of the parameter ο can be expressed as follows: Varο ο ο=ο ο ( ο ο ο ) ο , (5) where ο ο represents the variance of the error ο. As shown in Equation (2), when fea- tures ο are whitened using DWT in the feature extractor, the equation is modified as follows: Varο ο ο=ο ο ο ο ο ο ο ο ο =ο ο ο, (6) where ο ο ο indicates that the variance of the regressor parameters is reduced. This re- duction in parameter variance can help improve the performance of the regression tasks [29]. These equations are applicable to SSDAR where a small amount of labeled target data is available for model training. Furthermore, a previous study [11] has shown that batch normalization negatively impacts domain adaptation regression, with better per- formance observed when it is disabled. Fig. 2. Overview of the proposed framework, WhiteCon. DWT removes correlations between features and reduces the variance of regression model parameters. The proposed loss ο ο trains the model to ensure that the variances of strongly and mixup-augmented features (Var ο and Var ο ο ) align more closely with the variance of weakly augmented features Var ο ο , introducing consistency regularization for the augmented unlabeled features and enhancing model robust- ness. 3.3 Overview of the Proposed Method Our method combines DWT and dual consistency regularization loss ο ο to address SSDAR. ο ο consists of prediction consistency regularization loss ο ο and feature variance consistency regularization ο ο . The total loss L ο is given by: ο ο =ο ο + ο ο = ο ο + ο ο + ο ο . (7) Fig. 2 presents an overview of our proposed framework, WhiteCon. The framework takes source and target samples and applies weak, strong, and mixup augmentations to the unlabeled target samples. These augmented samples pass through a feature extractor with DWT, which removes feature correlations and stabilizes training by reducing the variance of regressor parameters. The loss ο ο aligns the variances of strongly and mixup-augmented features with those of weakly augmented features, while ο ο en- forces consistency across predictions from these augmentations. 3.4 Dual Consistency Regularization As discussed in Section 2.1, consistency regularization has proven effective in domain adaptation. Inspired by the semi-supervised regression approach of Sim et al. [30], we apply weak, strong, and mixup augmentations to unlabeled target samples ο ο to ob- tain weak ο ο ο , strong ο ο and mixup ο ο ο augmented unlabeled target samples. Here, ο ο ο is a linear combination of weak and strong augmented samples. Prediction Consistency Regularization. After applying these augmentations, the fea- ture extractor ο(β), which includes DWT, transforms the augmented unlabeled target samples into feature representations ο ο ο ο , ο ο ο and ο ο ο ο , respectively. Each of 8 these feature representations belongs to β οΓο , where ο is the feature dimensionality. Using the regressor ο(β), we obtain predictions ο ο ο , ο ο and ο ο ο . We train the model to make the strongly augmented prediction ο ο and the mixup prediction ο ο ο close to the weakly augmented prediction ο ο ο as the pseudo-label. The predic- tion consistency regularization loss ο ο is defined as follows: ο ο =ο ο ο ο ο β ο ο ο ο βο ο ο ο ο ο ο + ο ο β ο ο ο ο βο ο ο ο ο ο ο ο ο, (8) where ο ο is a hyperparameter for ο ο . Incorporating elements of a semi-supervised re- gression approach, ο ο promotes consistent predictions across different augmenta- tions, which is important in SSDAR settings with limited labeled target data. Feature Variance Consistency Regularization. Unlike classification tasks, regression tasks predict continuous values for unlabeled data, making it challenging to apply threshold-based functions for pseudo-label refinement as in classification [31]. There- fore, introducing additional consistency beyond prediction consistency regularization is critical [30]. Previous studies proposed feature consistency across augmented unla- beled targets through complex calculations, such as simultaneously regulating variance, invariance, and covariance, or contrastive learning [32]. To simplify and enhance con- sistency regularization, we use feature variance from augmentations. The variance of the weakly augmented features Var ο ο is calculated as follows: Var ο ο = ο ο β ο ο ο ο ο βο ο ο ο ο , ο ο (9) where ο ο ο ο ββ οΓο represents the weakly augmented features across ο samples and ο feature dimensions, and ο ο ο ββ οΓο is the mean vector of these features. The var- iance is computed along each feature dimension, resulting in a variance vector with the same dimensionality ο as the feature matrix. Similar to Var ο ο in Equation (9), the variances for strongly and mixup-augmented features (Var ο and Var ο ο ) are also computed along each feature dimension. The model is trained to align the variances of the strongly Var ο and mixup-augmented features Var ο ο with that of the weakly augmented features Var ο ο . The feature consistency regularization loss ο ο is ex- pressed as follows: ο ο =ο ο ο ο ο β οVar ο ο ο βVar ο ο ο ο ο + ο ο β οVar ο ο ο βVar ο ο ο ο ο ο ο, (10) where ο ο is a hyperparameter for ο ο . This variance alignment promotes consistent feature distributions across augmentations, reducing sensitivity to perturbations and en- hancing generalization. Mean absolute error (MAE) is used for ο ο because it handles differences evenly and prevents large errors from disproportionately affecting the train- ing, leading to more stable variance alignment [33]. Unlike existing methods that focus on either prediction or feature consistency independently, our dual consistency regular- ization simultaneously addresses both aspects in a unified and straightforward manner. Fig. 3. Sample images from the three benchmark datasets, BIWI, QMUL, and MPI3D. 4 Experiments 4.1 Datasets We used three benchmark datasets to address the SSDAR problem: BIWI [34] and QMUL [35] for head pose estimation, and MPI3D [36] for object position estimation. Sample images from each dataset are shown in Fig. 3. To satisfy the SSDAR setting, we randomly selected a small proportion (5%) of the target data as labeled target sam- ples for training. Weak augmentation consisted of random cropping and Gaussian blur (excluding angle-altering transforms such as flips), and strong augmentation followed the FixMatch protocol. BIWI. This dataset contains 5,874 images of 6 female (F) subjects and 9,804 images of 14 male (M) subjects, captured while turning their heads. We evaluated the dataset under two cases: MβF and FβM, predicting yaw, pitch, and roll angles. QMUL. The dataset consists of 1,504 images of 8 female (F) subjects and 3,595 images of 40 male (M) subjects, also captured while turning their heads. We evaluated two cases: MβF and FβM, predicting yaw and pitch angles. MPI3D. This dataset contains 1,036,800 3D object samples across three domains: real (RL), realistic (RC), and toy (T). The dataset has been used in UDAR settings [11, 14], and we adapted it for SSDAR, evaluating six cases: RLβRC, RLβT, RCβRL, RCβT, TβRL, and TβRC, predicting the vertical and horizontal axes. 4.2 Experimental Setup Implementation Details. For the backbone model, we used ResNet50, which is widely used in domain adaptation regression and is suitable for the input image size [11, 14]. The source and target labels were scaled to the range [0, 1] using min-max normaliza- tion. The AdamW optimizer was used with a learning rate of 0.001 and a batch size of 48. The model was trained for 50 epochs, with a linear ramp-up for ο ο and ο ο over the first 20 epochs for stability. The optimal hyperparameters ο ο and ο ο were selected based on the highest validation score. An NVIDIA RTX 4090 GPU was used for all the experiments. 10 Compared Methods. We adapted the following methods as mentioned in Section 2 to the SSDAR setting for comparison: S+T, MMD, DANN, RSD, DARE-GRAM, Deep- DAR, LIRR, and Full_T. S+T is a supervised learning method that only uses labeled source data and a small amount (5%) of labeled target data. Full_T is a model trained with the full set (100%) of labeled target data. We included UDAC methods (MMD and DANN) that can be applied to regression tasks, UDAR methods (RSD and DARE- GRAM), and SSDAR methods (DeepDAR and LIRR). Methods that could not be im- plemented under the same experimental conditions were excluded from the comparison [26, 27]. To apply UDA methods to the SSDAR setting, we incorporated ο ο to train the model using the labeled target samples [18, 23]. 4.3 Results BIWI. Table 1 shows that WhiteCon achieved the best performance with an average ο ο of 0.86 and MAE of 4.71. Performance differences were more pronounced in FβM (WhiteCon: ο ο 0.87 vs. RSD: ο ο 0.80), while MβF results were comparable across methods (ο ο : 0.83β0.85). WhiteCon outperformed the second-best method, RSD (av- erage ο ο : 0.82 (β0.04), MAE: 5.73 (+0.99)). QMUL. WhiteCon achieved the best performance in both FβM and MβF cases, with an average ο ο of 0.83 and MAE of 9.35 (Table 1). Notably, most baseline methods showed performance similar to S+T (within 0.01 ο ο difference), suggesting saturation of existing approaches on QMUL. WhiteCon achieved meaningful gains over the sec- ond-best method, MMD (average ο ο : 0.79 (β0.04), MAE: 10.52 (+1.17)). MPI3D. As shown in Table 2, WhiteCon achieved the highest overall performance across all cases, with an average ο ο of 0.90 and MAE of 2.04. While scenarios with RL and RC as source domains achieved higher performance, the TβRL and TβRC cases proved challenging for comparative methods. Nevertheless, WhiteCon demonstrated robust performance even under these difficult conditions. The second-best method, DARE-GRAM (average ο ο : 0.74 (β0.16), MAE: 3.78 (+1.74)), lagged substantially behind, representing the largest performance gap observed across the three datasets. 4.4 Analyses Ablation Study. To evaluate the contribution of each component, we removed each module individually across three datasets (Table 3). While the most influential compo- nent varied across datasets, removing ο ο (w/o ο ο ) consistently led to the largest per- formance degradation, confirming its critical role in enhancing overall performance. Removing DWT (w/o DWT) or ο ο (w/o ο ο ) also led to performance drops of similar degree, indicating that both modules provide complementary contributions of compa- rable importance. These findings demonstrate that WhiteConβs superior performance relies on the integration of all three components, with ο ο being the most important factor in general. Table 1. Performance comparison of the proposed method and comparative methods on BIWI and QMUL in terms of ο ο and MAE. Result is reported as mean Β± standard deviation with three different random seeds. Bold and underline indicate the best and the second-best results. The Wilcoxon rank-sum test is used to verify significant differences between the best and second- best average results, and is noted by p-value (*: p-value < 0.05). Method BIWI QMUL FβM MβF Average FβM MβF Average ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ S+T MMD DANN RSD DARE-GRAM DeepDAR LIRR WhiteCon (ours) Full_T Table 2. Performance comparison of the proposed method and comparative methods on MPI3D in terms of ο ο and MAE. Result is reported as mean Β± standard deviation across two regression outputs with three different random seeds. Bold and underline indicate the best and the second- best results. The Wilcoxon rank-sum test is used to verify significant differences between the best and second-best average results, and is noted by p-value (*: p-value < 0.05). Method RLβRC RLβT RCβRL RCβT TβRL TβRC Average ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ ο ο β MAEβ S+T MMD DANN RSD DARE-GRAM DeepDAR LIRR WhiteCon (ours) Full_T Table 3. Contribution of each component (DWT, ο ο , ο ο ) to the overall performance of WhiteCon on BIWI, QMUL, and MPI3D datasets in terms of MAE. Result is reported as mean Β± standard deviation with three different random seeds. Bold indicates the component with the largest contribution to performance on each dataset, and underline denotes the second most in- fluential component. Method DWT ο ο ο ο BIWI Average QMUL Average MPI3D Average MAE MAE MAE Proposed P P P w/o DWT P P w/o ο ο P P w/o ο ο P P 12 Fig. 4. Comparison of regressor parameter variance for QMUL FβM. The ο-axis represents the different methods, the ο-axis shows parameter variance values. Fig. 5. Performance comparison with increasing proportions of labeled target data on QMUL FβM. The ο-axis shows the proportions of labeled target data; the ο-axis shows ο ο . The dashed line indicates the upper bound (Full_T) with 100% labeled target data. Fig. 6. Feature visualization results of comparison methods on MPI3D RLβRC. The color bar represents label values. Variance of Regressor Parameters. To empirically validate the mathematical justifi- cation presented in Section 3.2, we conducted experiments to measure parameter vari- ance across methods (Fig. 4). WhiteCon achieved the lowest variance of 1.273Γ10β»β΄, compared to 1.438Γ10β»β΄β1.515Γ10β»β΄ for comparison methods. These results are con- sistent with Section 3.2, suggesting that lower parameter variance contributes to im- proved regression performance. Notably, while Full_T achieved high performance de- spite higher variance due to the absence of domain shift, among domain adaptation methods, WhiteCon both minimized variance and achieved the highest performance. Increasing Proportions of Labeled Target. To verify WhiteCon's robustness across varying amounts of labeled target data, we conducted experiments with 5%, 10%, 20%, and 30% proportions. Fig. 5 shows that WhiteCon consistently outperformed compari- son methods across all proportions on QMUL FβM. Remarkably, WhiteCon with 30% labeled data (ο ο 0.91) exceeded Full_T trained on 100% data (ο ο 0.88). Feature Visualization. To examine feature alignment in the regression context, where effective alignment should show continuous patterns along label values, we visualized features (from the feature extractor ο(Β·)) on MPI3D RLβRC using t-distributed sto- chastic neighbor embedding in Fig. 6. (a) S+T showed features largely unaligned with label values. While (d) RSD and (f) DeepDAR demonstrated improved alignment, con- fusion between label values remained, with dark colors scattered among bright colors. In contrast, (h) WhiteCon showed the clearest alignment, effectively grouping features with similar label values and demonstrating its superior ability to reduce the domain gap. 5 Conclusion In this study, we proposed WhiteCon, a method for addressing SSDAR problems, com- bining DWT and dual consistency regularization to reduce domain discrepancies. DWT improves model performance under OLS assumptions by transforming the feature co- variance matrix into an identity matrix, thus reducing the variance of regression param- eters. Variance consistency regularization aligns feature variances across augmenta- tions, improving model robustness. Experimental results on multiple benchmark da- tasets demonstrate that WhiteCon achieves state-of-the-art performance, confirming its effectiveness. Future work will extend WhiteCon to underexplored modalities such as time series and tabular data, which are widely used in real-world regression applications but remain understudied in SSDAR, requiring the design of modality-specific augmen- tation and adaptation strategies. Acknowledgments This research was supported by Brain Korea 21 FOUR, the Ministry of Science and ICT (MSIT) in Korea under the ITRC support program supervised by the Institute for Information Communication Technology Planning and Evaluation (IITP-2026-RS- 2020-0-01749), and the National Research Foundation of Korea grant funded by the Korea government (RS-2022-00144190). 14 References 1. Das A, Rabby ASA, Kowsar I, Rahman F. A deep learning-based unified solution for character recognition. In: Proceedings of the International Conference on Pattern Recognition, ICPR, p. 1671-7 (2022) 2. Pandey V, Panwar N, Kumbhar A, Roy P, Iwamura M. Enhanced cross-task EEG classi- fication: domain adaptation with EEGNet. In: Proceedings of the International Conference on Pattern Recognition, ICPR, p. 354β69 (2024) 3. Chen Y, Ouyang X, Zhu K, Agam G. Semi-supervised dual-domain adaptation for semantic segmentation. In: Proceedings of the International Conference on Pattern Recognition, ICPR, p. 230-7 (2022) 4. Wu X, Du C, Zhang H, Liu J, Zhang D, Zou H. Unsupervised domain adaptation for cross- device iris liveness detection model transfer. In: Proceedings of the International Confer- ence on Pattern Recognition, ICPR, p. 256-72 (2024) 5. Seraj MS, Chakraborty S. Multi-source deep domain adaptation for deepfake detection. In: Proceedings of the International Conference on Pattern Recognition, ICPR, p. 127-42 (2024) 6. Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, et al. Domain- adversarial training of neural networks. Journal of Machine Learning Research 17(59):1- 35 (2016) 7. Long M, Cao Y, Wang J, Jordan M. Learning transferable features with deep adaptation networks. In: Proceedings of the International Conference on Machine Learning, ICML, p. 97-105 (2015) 8. Long M, Zhu H, Wang J, Jordan MI. Deep transfer learning with joint adaptation networks. In: Proceedings of the International Conference on Machine Learning, ICML, p. 2208-17 (2017) 9. Hoffman J, Tzeng E, Park T, Zhu J-Y, Isola P, Saenko K, et al. Cycada: Cycle-consistent adversarial domain adaptation. In: Proceedings of the International Conference on Machine Learning, ICML, p. 1989-98 (2018) 10. Tzeng E, Hoffman J, Saenko K, Darrell T. Adversarial discriminative domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, p. 7167-76 (2017) 11. Chen X, Wang S, Wang J, Long M. Representation subspace distance for domain adapta- tion regression. In: Proceedings of the International Conference on Machine Learning, ICML, p. 1749-59 (2021) 12. Shu R, Bui H, Narui H, Ermon S. A DIRT-T Approach to unsupervised domain adaptation. In: Proceedings of the International Conference on Learning Representations, ICLR (2018) 13. Saito K, Kim D, Sclaroff S, Darrell T, Saenko K. Semi-supervised domain adaptation via minimax entropy. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, p. 8050-8 (2019) 14. Nejjar I, Wang Q, Fink O. DARE-GRAM: Unsupervised domain adaptation regression by aligning inverse gram matrices. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, p. 11744-54 (2023) 15. Hanneke S, Kpotufe S. On the value of target data in transfer learning. In: Proceedings of the International Conference on Neural Information Processing Systems, NeurIPS (2019) 16. French G, Mackiewicz M, Fisher M. Self-ensembling for visual domain adaptation. In: Pro- ceedings of the International Conference on Learning Representations, ICLR (2017) 17. Singh A, Chakraborty S. Deep domain adaptation for regression. Development and Analy- sis of Deep Learning Architectures. 2020. 18. Li B, Wang Y, Zhang S, Li D, Keutzer K, Darrell T, et al. Learning invariant representations and risks for semi-supervised domain adaptation. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR, p. 1104-13 (2021) 19. Roy S, Siarohin A, Sangineto E, Bulo SR, Sebe N, Ricci E. Unsupervised domain adapta- tion using feature-whitening and consensus loss. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR, p. 9471-80 (2019) 20. Long M, Zhu H, Wang J, Jordan MI. Unsupervised domain adaptation with residual transfer networks. In: Proceedings of the International Conference on Neural Information Pro- cessing Systems, NeurIPS (2016) 21. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial nets. In: Proceedings of the International Conference on Neural Information Processing Systems, NeurIPS (2014) 22. Li L, Zhang Z. Semi-supervised domain adaptation by covariance matching. IEEE Trans- actions on Pattern Analysis and Machine Intelligence 41(11):2724-39 (2018) 23. Yu Y-C, Lin H-T. Semi-supervised domain adaptation with source label adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, p. 24100-9 (2023) 24. Li J, Li G, Shi Y, Yu Y. Cross-domain adaptive clustering for semi-supervised domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, p. 2505-14 (2021) 25. Cortes C, Mohri M. Domain adaptation in regression. In: Proceedings of the International Conference on Algorithmic Learning Theory, ALT, p. 308-23 (2011) 26. Wu J, He J, Wang S, Guan K, Ainsworth E. Distribution-informed neural networks for domain adaptation regression. In: Proceedings of the International Conference on Neural Information Processing Systems, NeurIPS (2022) 27. Dhaini M, Berar M, Honeine P, Van Exem A. Unsupervised domain adaptation for regres- sion using dictionary learning. Knowledge-Based Systems 267:110439 (2023) 28. Goldberger AS. Econometric Theory. 1964. 29. Reddy TA, Andersen K. An evaluation of classical steady-state off-line linear parameter estimation methods applied to chiller performance data. Hvac&R Research 8(1):101-24 (2002) 30. Sim S, Bae J, Kim SB. Robust semi-supervised regression for vehicle interior noise predic- tion. IEEE Access 12:60-72 (2024) 31. Dai W, Li X, Cheng K-T. Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, p. 7304-13 (2023) 32. Bardes A, Ponce J, Lecun Y. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In: Proceedings of the International Conference on Learning Rep- resentations, ICLR (2022) 33. Kwon B, Son H. Accurate path loss prediction using a neural network ensemble method. Sensors 24(1):304 (2024) 34. Fanelli G, Dantone M, Gall J, Fossati A, Gool L. Random forests for real time 3D face analysis. International Journal of Computer Vision 101(3):437β58 (2013) 35. Sherrah J, Gong S. Fusion of perceptual cues for robust tracking of head pose and position. Pattern Recognition 34(8):1565-72 (2001) 36. Gondal MW, Wuthrich M, Miladinovic D, Locatello F, Breidt M, Volchkov V, et al. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. In: Proceedings of the International Conference on Neural Information Processing Systems, NeurIPS (2019) Appendix for WhiteCon: Semi-Supervised Domain Adaptation Regression Through Whitening Transform and Dual Consistency Se Jin Sim [0009β0000β6028β1690] and Seoung Bum Kim [0000β0002β2205β8516] School of Industrial and Management Engineering Korea University, Seoul, Republic of Korea ssj259, sbkim1@korea.ac.kr Fig. A1. Feature scale comparison across methods on BIWI FβM. The ν₯-axis represents the different methods and the ν¦-axis shows Frobenius norm values A-1. Feature Scale. RSD argued based on experiments that maintaining a feature scale similar to supervised learning before domain adaptation yields optimal performance in domain adaptation regression. To investigate RSD's assertion about the importance of feature scale in do- main adaptation regression, we conducted feature scale analysis using Frobenius norm. The Frobenius norm, an extension of the L2 norm for matrices, quantifies the scale of feature representations by taking the square root of the sum of the squared elements across all entries. Fig. A1 presents a feature scale comparison of methods on BIWI FβM. Although WhiteCon achieved the highest overall performance (as shown in Ta- ble 1), RSD showed a feature scale of 20.91, which is closest to that of S+T at 19.76. Additionally, MMD, which achieved the second-highest performance, had the second- largest feature scale of 26.42 among the methods. This result contrasts with RSDβs as- sertion that optimal performance in domain adaptation regression is achieved when the feature scale is close to S+T, suggesting that a similar feature scale is not a strict re- quirement for achieving high performance. 2 Table A1. Training (source, labeled target, and unlabeled target), validation, and testing data splits across three benchmark datasets. Dataset Case Source Target Labeled (νΎ%) Unlabeled Validation Testing BIWI FβM 3,447 50 (5%) 2,500 100 2,131 100 (10%) 200 (20%) 300 (30%) 1,000 (100%) MβF 5,624 35 (5%) 1,500 70 1,247 70 (10%) 140 (20%) 210 (30%) 700 (100%) QMUL FβM 1,504 44 (5%) 1,349 89 1,215 89 (10%) 179 (20%) 268 (30%) 896 (100%) MβF 2,500 15 (5%) 600 30 569 30 (10%) 61 (20%) 91 (30%) 305 (100%) MPI3D RL β RC 3,000 50 (5%) 2,000 100 10,437 RL β T 100 (10%) 10,348 RC β RL 200 (20%) 10,405 RC β T 300 (30%) 10,348 T β RL 1,000 (100%) 10,405 T β RC 10,437 A-2. Datasets Settings We used three benchmark datasets: BIWI and QMUL for head pose estimation, and MPI3D for object position estimation, under SSDAR settings. Table A1 shows the number of training, validation, and testing samples for each dataset in each case. We used 10% of the labeled target data for validation, which was based on SSDA methods for classification [1-3]. To simulate different proportions of labeled target data, denoted as K%, we used values including 5%, 10%, 20%, and 30%, as shown in Fig. 5. Table A2. Details of hyperparameter search for each method. Optimal values were selected based on validation performance. MMD Coefficient of MMD loss 0.0005, 0.0001, 0.001, 0.01 DANN Coefficient of DANN loss 0.001, 0.01, 0.1, 1 RSD Coefficient of RSD loss 0.0001, 0.001 Coefficient of bases mismatch penalization loss 0.001, 0.01, 0.1 DARE-GRAM Coefficient of angle loss 0.001, 0.005, 0.05, 0.5, 0.1, 1 Coefficient of trade-off loss 0.00001, 0.0001, 0.0005, 0.001, 0.01 DeepDAR Coefficient of MMD loss 0.001, 0.01, 0.1, 1, 10 Coefficient of semi-supervised loss 0.001, 0.01, 0.1, 1, 10 LIRR Coefficient of DANN loss 0.01, 10, 100 WhiteCon (ours) Coefficient of prediction consistency regularization loss (ν 1 ) 0.1, 0.2, 0.3, 0.4, 0.5 Coefficient of feature variance consistency regularization loss (ν 2 ) 0.1, 0.2, 0.3, 0.4, 0.5 Fig. A2. MAE results from WhiteCon hyperparameter search across three benchmark datasets. The ν₯-axis represents ν 1 , the ν¦-axis represents ν 2 , and the ν§-axis represents MAE. A-3. Hyperparameter Search We conducted extensive hyperparameter optimization for all compared methods to en- sure fair evaluation. Table A2 details the hyperparameter ranges examined for each method, with optimal values selected based on validation performance. For WhiteCon, we systematically explored ν 1 and ν 2 values in the range 0.1, 0.2, 0.3, 0.4, 0.5. Fig. A2 shows the hyperparameter search results for WhiteCon. We conducted experiments on a representative case from each of the three benchmark datasets to determine optimal 4 parameters. Based on MAE performance, the optimal hyperparameters were identified as BIWI (0.1, 0.1), QMUL (0.3, 0.5), and MPI3D (0.1, 0.1). A-4. Augmentation The images from all benchmarks were resized to 224 Γ 224 pixels. All comparison methods were trained using weakly augmented images. Following the augmentation guidelines for image angle prediction from Hu et al. [4], our weak augmentation con- sisted of random cropping and Gaussian blur, while excluding transformations that could alter the angle, such as flips. The strong augmentation strategy was identical to the one used in FixMatch [5], a widely-used semi-supervised method. As mentioned in Section 3.4, the mixup augmentation is a linear combination of weakly and strongly augmented samples. References 1. Li J, Li G, Shi Y, Yu Y. Cross-domain adaptive clustering for semi-supervised domain ad- aptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, p. 2505-14 (2021) 2. Saito K, Kim D, Sclaroff S, Darrell T, Saenko K. Semi-supervised domain adaptation via minimax entropy. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, p. 8050-8 (2019) 3. Yu Y-C, Lin H-T. Semi-supervised domain adaptation with source label adaptation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, p. 24100-9 (2023) 4. Hu H-C, Wu X, Liu H, Wei T-R, Wu H-T. Full-range head pose geometric data augmenta- tions. arXiv preprint arXiv:2408.01566 (2024) 5. Sohn K, Berthelot D, Carlini N, Zhang Z, Zhang H, Raffel CA, et al. FixMatch: Simplifying semi-supervised learning with consistency and confidence. In: Proceedings of the Interna- tional Conference on Neural Information Processing Systems, NeurIPS (2020)