Paper deep dive

A Hybrid Tsallis-Polarization Impurity Measure for Decision Trees: Theoretical Foundations and Empirical Evaluation

Edouard Lansiaux, Idriss Jairi, Hayfa Zgaya-Biau

Year: 2026Venue: arXiv preprintArea: stat.MLType: PreprintEmbeddings: 37

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 7/20/2026, 11:22:35 PM

Summary

The paper introduces the Integrated Tsallis Combination (ITC), a hybrid impurity measure for decision trees that combines normalized Tsallis entropy with an exponential polarization component. The authors provide theoretical foundations, including proofs of concavity and boundary conditions, and conduct an empirical evaluation on seven benchmark datasets comparing 23 impurity measures. Results show that while simple parametric measures like Tsallis (alpha=0.5) achieve the highest accuracy, ITC variants offer competitive performance with strong theoretical guarantees and O(K) computational complexity.

Entities (10)

Integrated Tsallis Combination · algorithm · 98%decision tree · model · 95%Tsallis entropy · mathematical-concept · 95%Gini Impurity · algorithm · 92%Shannon Entropy · algorithm · 92%Breast Cancer Wisconsin Dataset · dataset · 90%Digits dataset · dataset · 90%Friedman Test · statistical-method · 90%IRIS Dataset · dataset · 90%Wine Dataset · dataset · 90%

Relation Signals (9)

Integrated Tsallis Combination → combines → Tsallis entropy

confidence 95% · ITC combines normalized Tsallis entropy with an exponential polarization component through a convex combination

Integrated Tsallis Combination → evaluatedon → Digits dataset

confidence 95% · The Digits dataset... tests multi-class performance.

Integrated Tsallis Combination → evaluatedon → IRIS Dataset

confidence 95% · Through an extensive comparative evaluation on seven benchmark datasets... The Iris dataset... are classic benchmarks

Integrated Tsallis Combination → evaluatedon → Wine Dataset

confidence 95% · The Iris dataset... and Wine dataset... are classic benchmarks

Integrated Tsallis Combination → evaluatedon → Breast Cancer Wisconsin Dataset

confidence 95% · The Breast Cancer Wisconsin dataset... represents a real-world medical diagnosis problem.

Integrated Tsallis Combination → hascomplexity → O(K)

confidence 95% · computational efficiency O(K)-making it a rigorous, generalizable alternative

Gini Impurity → comparedwith → Shannon Entropy

confidence 90% · The two most widely adopted measures—Gini impurity [4] and Shannon entropy [5]—exhibit well-documented limitations.

Friedman Test → usedfor →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We introduce the Integrated Tsallis Combination (ITC), a hybrid impurity measure for decision tree learning that combines normalized Tsallis entropy with an exponential polarization component. While many existing measures sacrifice theoretical soundness for computational efficiency or vice versa, ITC provides a mathematically principled framework that balances both aspects. The core innovation lies in the complementarity between Tsallis entropy's information-theoretic foundations and the polarization component's sensitivity to distributional asymmetry. We establish key theoretical properties-concavity under explicit parameter conditions, proper boundary conditions, and connections to classical measures-and provide a rigorous justification for the hybridization strategy. Through an extensive comparative evaluation on seven benchmark datasets comparing 23 impurity measures with five-fold repetition, we show that simple parametric measures (Tsallis $\alpha=0.5$) achieve the highest average accuracy ($91.17\%$), while ITC variants yield competitive results ($88.38-89.16\%$) with strong theoretical guarantees. Statistical analysis (Friedman test: $\chi^2=3.89$, $p=0.692$) reveals no significant global differences among top performers, indicating practical equivalence for many applications. ITC's value resides in its solid theoretical grounding-proven concavity under suitable conditions, flexible parameterization ($\alpha$, $\beta$, $\gamma$), and computational efficiency $O(K)$-making it a rigorous, generalizable alternative when theoretical guarantees are paramount. We provide guidelines for measure selection based on application priorities and release an open-source implementation to foster reproducibility and further research.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

35,432 characters extracted from source content.

Expand or collapse full text

A Hybrid Tsallis–Polarization Impurity Measure for Decision Trees: Theoretical Foundations and Empirical Evaluation Dr Edouard Lansiaux Emergency Department, Lille University Hospital, France Lille University, France edouard1.lansiaux@chu-lille.fr Dr Idriss Jairi LIRMM Montpellier University, France idriss.jairi@umontpellier.fr Pr Hayfa Zgaya-Biau CRISTAL UMR CNRS 9189 Lille University, France hayfa.zgaya-biau@univ-lille.fr Abstract We introduce the Integrated Tsallis Combination (ITC), a hybrid impurity measure for decision tree learning that combines normalized Tsallis entropy with an exponential polarization component. While many existing measures sacrifice theoretical soundness for computational efficiency or vice versa, ITC provides a mathematically principled framework that balances both aspects. The core innovation lies in the complementarity between Tsallis entropy’s information-theoretic foundations and the polarization component’s sensitivity to distributional asymmetry. We establish key theoretical properties—concavity under explicit parameter conditions, proper boundary conditions, and connections to classical measures—and provide a rigorous justification for the hybridization strategy. Through an extensive comparative evaluation on seven benchmark datasets comparing 23 impurity measures with five-fold repetition, we show that simple parametric measures (Tsallis α=0.5α=0.5) achieve the highest average accuracy (91.17%), while ITC variants yield competitive results (88.38–89.16%) with strong theoretical guarantees. Statistical analysis (Friedman test: χ2=3.89χ^2=3.89, p=0.692p=0.692) reveals no significant global differences among top performers, indicating practical equivalence for many applications. ITC’s value resides in its solid theoretical grounding—proven concavity under suitable conditions, flexible parameterization (α, β, γ), and computational efficiency O(K)O(K)—making it a rigorous, generalizable alternative when theoretical guarantees are paramount. We provide guidelines for measure selection based on application priorities and release an open-source implementation to foster reproducibility and further research. Keywords: Decision Trees, Impurity Measures, Tsallis Entropy, Polarization Metrics, Machine Learning Interpretability, Hybrid Measures, Theoretical Guarantees 1 Introduction Decision trees remain a cornerstone of machine learning due to their interpretability, non-parametric nature, and strong performance across diverse domains [1, 2]. Their construction relies critically on impurity measures to determine optimal splits at each node, making the choice of impurity criterion fundamental to model accuracy, interpretability, and generalization [3]. Despite decades of research, the quest for an impurity measure that simultaneously offers theoretical soundness, practical sensitivity, and computational efficiency continues. The two most widely adopted measures—Gini impurity [4] and Shannon entropy [5]—exhibit well-documented limitations. Gini impurity, while computationally efficient, shows reduced sensitivity to small class imbalances and can yield suboptimal splits in multi-class settings [6]. Shannon entropy provides stronger information-theoretic foundations but may lead to overfitting due to excessive sensitivity to probability variations in distribution tails [7]. These trade-offs have motivated numerous generalizations, including parametric families such as Rényi entropy [8] and Tsallis entropy [9], as well as practical modifications like polarization indices [10] and distance-based measures [11]. However, these approaches typically excel in specific aspects while compromising others. A fundamental challenge lies in simultaneously optimizing three competing objectives: theoretical soundness (concavity, boundary conditions, generalization), practical sensitivity to distributional characteristics, and computational efficiency. Most existing measures prioritize one or two objectives at the expense of the others, leading to suboptimal real-world performance where all three dimensions are important. In this work, we propose the Integrated Tsallis Combination (ITC). Our main contributions are fourfold. First, we introduce a novel hybrid formulation combining normalized Tsallis entropy with exponential polarization through a convex combination, and we provide a rigorous mathematical analysis demonstrating the complementary nature of the two components. Second, we establish fundamental theoretical properties of ITC, including concavity under explicit parameter conditions, proper boundary conditions, symmetry, and connections to classical measures. Third, we conduct an extensive comparative evaluation comparing 23 impurity measures across seven diverse datasets, with five-fold repetition and rigorous statistical testing. Fourth, we provide practical implementation guidelines, parameter optimization recommendations, and an open-source implementation to facilitate adoption and extension by the research community. The remainder of this paper is organized as follows. Section 2 reviews related work and identifies limitations of existing approaches. Section 3 presents the ITC measure, its theoretical properties, and the justification for hybridization. Section 4 describes the experimental setup and reports empirical results, including statistical analysis and computational benchmarks. Section 5 interprets the findings, discusses implications, and provides usage guidelines. Section 6 concludes and outlines future research directions. 2 Background and Related Work 2.1 Classical Impurity Measures Decision tree induction algorithms such as CART [1] and C4.5 [2] typically employ one of three classical impurity measures. For a K-class classification problem with probability vector =(p1,…,pK) p=(p_1,…,p_K), the Gini impurity is defined as G()=1−∑i=1Kpi2G( p)=1- _i=1^Kp_i^2, and it remains the default in many implementations due to its computational efficiency [4]. The Shannon entropy (or information gain) is given by H()=−∑i=1Kpilog⁡piH( p)=- _i=1^Kp_i p_i, forming the basis for algorithms prioritizing information-theoretic purity. The misclassification rate M()=1−maxi⁡piM( p)=1- _ip_i is conceptually simple but often produces suboptimal splits due to its insensitivity to distribution changes that do not affect the majority class [12]. Despite their widespread adoption, these classical measures exhibit fundamental limitations. Gini impurity’s quadratic nature reduces sensitivity to small probability variations, potentially missing splits that could improve generalization [6]. Shannon entropy’s logarithmic computations increase computational overhead and can lead to overfitting in the presence of noisy data [7]. The misclassification rate, while simple, is non-concave and may yield degenerate splits. 2.2 Parametric Generalizations To address these limitations, researchers have proposed parametric families that generalize classical measures and offer tunable sensitivity. Tsallis entropy [9] is defined for α>0α>0, α≠1α≠ 1 as Tα()=(1−∑i=1Kpiα)/(α−1)T_α( p)=(1- _i=1^Kp_i^α)/(α-1), with the limits T1()=H()T_1( p)=H( p) (Shannon entropy) and T2()=G()T_2( p)=G( p) (Gini impurity). The parameter α controls the measure’s sensitivity: α<1α<1 emphasizes minority classes, while α>1α>1 emphasizes majority classes. Rényi entropy [8] provides another parametric generalization: Rα()=11−αlog∑i=1KpiαR_α( p)= 11-α _i=1^Kp_i^α, which also includes Shannon entropy as α→1α→ 1. Recent work has explored the use of these parametric measures in decision trees, demonstrating that appropriate parameter selection can improve performance on imbalanced datasets [14, 15]. 2.3 Probabilistic Divergences and Distance-Based Measures Another line of research applies probabilistic divergences to quantify the difference between the observed class distribution and a reference distribution (typically uniform). The Kullback–Leibler (KL) divergence DKL(∥)=∑i=1Kpilog⁡(pi/ui)D_KL( p u)= _i=1^Kp_i (p_i/u_i) measures information loss when approximating p by uniformity, where =(1/K,…,1/K) u=(1/K,…,1/K). The Jensen–Shannon (JS) divergence DJS(∥)=12DKL(∥)+12DKL(∥)D_JS( p u)= 12D_KL( p m)+ 12D_KL( u m), with =(+)/2 m=( p+ u)/2, provides a symmetric and bounded alternative. Distance-based measures, such as Hellinger distance and energy distance [11], have also been investigated for split selection, particularly in the context of imbalanced data. 2.4 Polarization and Hybrid Approaches Recent work has introduced polarization-based impurity measures that explicitly capture distributional asymmetry [10]. The polarization index is typically defined as P()=∑i=1K|pi−p¯|rP( p)= _i=1^K|p_i- p|^r, with p¯=1/K p=1/K and r>0r>0. These measures are sensitive to deviations from uniformity and can complement information-theoretic criteria. Hybrid measures combining multiple perspectives have shown promise [13], but often lack rigorous theoretical justification or systematic evaluation. Our analysis identifies several gaps in the current literature, including the sensitivity–robustness trade-off, the theoretical–practical gap, the lack of orthogonal dimension coverage, parameter sensitivity issues, and challenges with imbalanced data. The ITC measure proposed in this work directly addresses these gaps by providing a principled hybrid framework with proven theoretical properties under appropriate conditions, comprehensive empirical validation, and practical implementation guidelines. 3 The ITC Impurity Measure 3.1 Mathematical Formulation ITC combines normalized Tsallis entropy with an exponential polarization component through a convex combination, leveraging the complementary strengths of both components while mitigating their individual limitations. The normalized Tsallis component is defined as Tαnorm()=Tα()/TαmaxT_α^norm( p)=T_α( p)/T_α^max, where Tα()T_α( p) is the standard Tsallis entropy and the maximum value at the uniform distribution is Tαmax=(1−K1−α)/(α−1)T_α^max=(1-K^1-α)/(α-1). This normalization ensures Tαnorm()∈[0,1]T_α^norm( p)∈[0,1] for all p, providing consistent scaling across different numbers of classes. We introduce a novel polarization measure with exponential sensitivity: Pβ()=∑i=1K|pi−p¯|⋅exp⁡(−β⋅|pi−p¯|)P_β( p)= _i=1^K|p_i- p|· (-β·|p_i- p|), where p¯=1/K p=1/K and β>0β>0 controls the sensitivity decay rate. The exponential term provides adaptive sensitivity: high for small deviations from uniformity and decreasing for larger deviations. The normalized polarization component is Pβnorm()=1−Pβ()/PβmaxP_β^norm( p)=1-P_β( p)/P_β^max, with maximum value Pβmax=2(K−1)K⋅(1−e−β)P_β^max= 2(K-1)K·(1-e^-β), yielding Pβnorm()∈[0,1]P_β^norm( p)∈[0,1] with 0 at pure nodes and 11 at the uniform distribution. The maximum is achieved at a pure distribution (e.g., p1=1p_1=1, pi=0p_i=0 for i>1i>1) because this maximizes each term |pi−p¯||p_i- p|. The complete ITC measure combines both components through a convex combination: ITCα,β,γ()=γ⋅Tαnorm()+(1−γ)⋅Pβnorm()ITC_α,β,γ( p)=γ· T_α^norm( p)+(1-γ)· P_β^norm( p), where γ∈[0,1]γ∈[0,1] balances the theoretical and practical components. This formulation preserves the mathematical properties of both components while enabling adaptive behavior across different distribution characteristics. 3.2 Theoretical Foundation for Hybridization The selection of Tsallis entropy and polarization components is motivated by their complementary nature. Tsallis entropy measures uncertainty through a power-law transformation of probabilities, making it sensitive to the overall shape of the distribution. In contrast, the polarization component focuses on deviations from uniformity, capturing asymmetry. Empirically, we observe that the information gains produced by the two components are only weakly correlated across candidate splits (Pearson correlation r≈0.2r≈ 0.2 on average), indicating that they capture different aspects of impurity. 3.3 Theoretical Properties We establish several important theoretical properties for ITC. For the polarization component, note that the function f(x)=xe−βxf(x)=xe^-β x is concave on [0,2/β][0,2/β]. Since |pi−1/K|≤1−1/K|p_i-1/K|≤ 1-1/K, a sufficient condition for the concavity of each term is β≤2/(1−1/K)β≤ 2/(1-1/K). A conservative condition independent of K is β≤2β≤ 2. Theorem 1 (Boundary Conditions). For any K≥2K≥ 2, α>0α>0, β>0β>0, γ∈[0,1]γ∈[0,1], ITC satisfies purity (zero if and only if pi=1p_i=1 for some i), uniformity (maximized at pi=1/Kp_i=1/K for all i), and symmetry (invariant under permutation of class labels). These properties follow directly from the corresponding properties of the normalized Tsallis and polarization components. Theorem 2 (Concavity). For α>0α>0 and β≤2β≤ 2, ITC()ITC( p) is concave in p. Tsallis entropy is known to be concave for all α>0α>0 (a standard result in information theory). The polarization term f(x)=xe−βxf(x)=xe^-β x is concave on [0,/2β][0,/ 2β]. Since |pi−1/K|≤1−1/K|p_i-1/K|≤ 1-1/K, concavity holds whenever β≤21−1Kβ≤ 21- 1K. Thus, β≤2β≤ 2 is a sufficient K-independent condition. A convex combination of concave functions is concave, hence ITC is concave under the stated condition. Concavity ensures non-negative impurity reduction at each split, a desirable property for decision tree construction. Remark 1. For β>2β>2, concavity is not guaranteed theoretically. However, numerical evaluations on dense grids of the probability simplex for β up to 55 and γ in [0.2,0.6][0.2,0.6] did not reveal any violation of concavity for the combined ITC measure. Theorem 3 (Connection to Classical Measures). ITC recovers classical impurity measures up to normalization in the following limits: limα→2,γ→1ITC()=G() _α→ 2,γ→ 1ITC( p)=G( p) (Gini impurity), limα→1,γ→1ITC()=H() _α→ 1,γ→ 1ITC( p)=H( p) (Shannon entropy), and limγ→0ITC()=Pβnorm() _γ→ 0ITC( p)=P_β^norm( p) (pure polarization). Theorem 4 (Computational Complexity). For K classes, ITC computation requires O(K)O(K) operations per split evaluation, matching the complexity of Gini impurity and Shannon entropy. Both ITC and Shannon have O(K)O(K) arithmetic complexity. However, Shannon requires logarithmic evaluations, which are typically more expensive than the exponential–linear operations in ITC in practice. 3.4 Parameter Optimization and Sensitivity Analysis We performed an extensive grid search over α∈[0.1,3.0]α∈[0.1,3.0], β∈[1,10]β∈[1,10], and γ∈[0,1]γ∈[0,1] across all datasets, evaluating over 2,000 parameter combinations. The optimal parameters maximizing average accuracy were α∗=2.0α^*=2.0, β∗=4.5β^*=4.5, and γ∗=0.4γ^*=0.4. The empirically optimal β∗=4.5β^*=4.5 lies outside the conservative concavity-guaranteed region (β≤2)(β≤ 2), illustrating the trade-off between theoretical guarantees and empirical performance. Sensitivity analysis reveals a broad performance plateau around these values, indicating robustness to small parameter variations. For example, varying α by ±0.3± 0.3 changes accuracy by less than 0.5%0.5\% on average. 4 Experimental Evaluation 4.1 Experimental Setup We selected seven datasets to ensure diversity in sample size, feature dimensionality, number of classes, and class balance. The Iris dataset (150 samples, 4 features, 3 classes) [18] and Wine dataset (178 samples, 13 features, 3 classes) [19] are classic benchmarks in pattern recognition. The Breast Cancer Wisconsin dataset (569 samples, 30 features, 2 classes) [20] represents a real-world medical diagnosis problem. The Digits dataset (1797 samples, 64 features, 10 classes) [21] tests multi-class performance. Additionally, we generated three synthetic datasets using scikit-learn’s ‘make_classification‘ function: Binary Balanced (1000 samples, 10 features, 2 classes, equal proportions), Binary Imbalanced (1000 samples, 10 features, 2 classes, 30:70 ratio), and Multiclass-4 (1500 samples, 12 features, 4 classes), designed to test specific characteristics such as class imbalance and multi-class behavior. We implemented 23 impurity measures across six categories: classical measures (Gini, Shannon entropy, misclassification rate), parametric measures (Rényi with α=0.5,2.0α=0.5,2.0, Tsallis with α=0.5,1.3,2.0α=0.5,1.3,2.0, normalized Tsallis with α=1.3α=1.3, Kumaraswamy–Charlier), probabilistic divergences (cross-entropy, KL divergence, Jensen–Shannon divergence), distance-based measures (Hellinger distance, energy distance), specialized measures (polarization index with α=3.5α=3.5, Bregman divergences for squared and entropy), and hybrid measures (ITC standard with α=2.0,β=4.5,γ=0.4α=2.0,β=4.5,γ=0.4, ITC with α=1.3α=1.3, ITC with α=1.7α=1.7, Shannon–polarization, Tsallis–Hellinger). All measures were implemented within a unified CART framework [22] to ensure fair comparison. Experiments were conducted using scikit-learn’s ‘DecisionTreeClassifier‘ modified to accept custom impurity measures, with parameters ’min_samples_split=2’ and ’max_depth=20’. We performed five-fold stratified cross-validation repeated five times with different random seeds, resulting in 25 runs per dataset per measure. All experiments were conducted on a workstation with Intel Xeon Gold 5218 CPU (2.30 GHz) and 128 GB RAM, using Python 3.12, NumPy 1.26, SciPy 1.11, and scikit-learn 1.4. 4.2 Overall Performance Results Table 1 presents the top-performing impurity measures by average accuracy across all datasets. Simple parametric measures, particularly Tsallis with α=0.5α=0.5, achieve the highest empirical performance. ITC variants occupy ranks 7, 10, and 12 with accuracies ranging from 88.38% to 89.16%, demonstrating competitive yet not superior performance. Table 1: Top 10 Impurity Measures by Average Accuracy (across all datasets) Rank Measure Accuracy 95% CI 1 Tsallis (α=0.5α=0.5) 0.9117 [0.8936, 0.9298] 2 Rényi (α=0.5α=0.5) 0.9085 [0.8890, 0.9280] 3 Shannon–Polarization 0.9064 [0.8875, 0.9253] 4 Shannon 0.9057 [0.8874, 0.9239] 5 Tsallis (α=2.0α=2.0) 0.9020 [0.8823, 0.9217] 5 Gini 0.9020 [0.8823, 0.9217] 7 ITC (α=1.3,β=4.5,γ=0.4α=1.3,β=4.5,γ=0.4) 0.8916 [0.8707, 0.9125] 8 Kumaraswamy 0.8899 [0.8682, 0.9116] 8 Tsallis (α=1.3α=1.3) 0.8899 [0.8682, 0.9116] 10 ITC (α=1.7,β=4.5,γ=0.4α=1.7,β=4.5,γ=0.4) 0.8859 [0.8637, 0.9080] Table 2 provides detailed performance metrics, including standard deviation, stability rating, relative improvement over Gini, and Cohen’s d effect size. Tsallis α=0.5α=0.5 shows a small positive effect (d=0.458) relative to Gini, while ITC variants exhibit small negative effects (d=-0.104 to -0.361), indicating modest practical differences. Table 2: Comprehensive Performance Metrics for Selected Measures Measure Accuracy Std Dev Stability vs Gini Cohen’s d Tsallis (α=0.5α=0.5) 0.9117 0.009 High +1.08% +0.458 Rényi (α=0.5α=0.5) 0.9085 0.010 High +0.72% +0.262 Shannon–Polarization 0.9064 0.010 High +0.49% +0.161 Shannon 0.9057 0.009 High +0.41% +0.154 Tsallis (α=2.0α=2.0) 0.9020 0.010 High 0.00% 0.000 Gini 0.9020 0.010 High – – ITC (α=1.3α=1.3) 0.8916 0.011 Moderate –1.15% –0.104 Tsallis (α=1.3α=1.3) 0.8899 0.011 Moderate –1.34% –0.075 ITC (α=1.7α=1.7) 0.8859 0.011 Moderate –1.79% –0.361 ITC (standard) 0.8838 0.012 Moderate –2.02% –0.340 4.3 Statistical Significance Analysis We conducted a Friedman test to assess global differences among the 23 measures. The test yielded χ2=3.8886χ^2=3.8886 with 22 degrees of freedom and p=0.6917p=0.6917, indicating no statistically significant global differences at α=0.05α=0.05. This suggests that, overall, the measures perform similarly across datasets. To visualize pairwise comparisons, we constructed a critical difference diagram following the Nemenyi post-hoc test (Figure 1). Although the global test is not significant, the diagram illustrates the relative rankings and the extent of overlap among measures. ITC variants are positioned in the middle ranks and are not separated from the top performers by the critical distance, consistent with the absence of global significance. 01122334455667788991010Tsallis0.5Renyi0.5Shannon-PolShannonTsallis2.0GiniITC1.3KumaraswamyTsallis1.3ITC1.7Average Rank Figure 1: Average ranks of the top 10 impurity measures. The horizontal axis represents the mean rank across datasets. ITC variants lie in the middle range, overlapping with many top measures. 4.4 Computational Performance Table 3 summarizes the benchmark duration, and Table 4 reports average training times per metric for each dataset. ITC’s computational cost is comparable to Gini, confirming its O(K)O(K) efficiency. Table 3: Benchmark Phase Durations Phase Duration Percentage Individual Metrics Benchmark 98 min 65.3% Hybrid Comparison Analysis 16 min 10.7% Sensitivity Analysis 21 min 14.0% Statistical Testing 15 min 10.0% Total 150 min 100% Table 4: Average Training Time per Metric by Dataset (5 runs) Dataset Samples Features Classes Time/Metric (s) Iris 150 4 3 8.0 Wine 178 13 3 91.3 Breast Cancer 569 30 2 232.1 Digits 1797 64 10 173.4 Binary Balanced 1000 10 2 134.0 Binary Imbalanced 1000 10 2 360.9 Multiclass-4 1500 12 4 486.0 4.5 Hybridization Effectiveness Table 5 compares all hybrid measures evaluated. Shannon–Polarization, a simpler hybrid, achieves the highest accuracy among hybrids, outperforming ITC variants. This suggests that theoretical sophistication does not guarantee empirical superiority; simpler combinations may suffice for many applications. Table 5: Hybrid Measure Performance Comparison Hybrid Combination Accuracy Rank vs Best Hybrid Shannon–Polarization 0.9064 3 – ITC (α=1.3α=1.3) 0.8916 7 –1.63% ITC (α=1.7α=1.7) 0.8859 10 –2.26% ITC (standard) 0.8838 12 –2.49% Tsallis–Hellinger 0.8113 20 –10.49% 5 Discussion 5.1 Interpreting the Empirical Results Our comprehensive benchmark yields several important insights. First, simple parametric measures, particularly Tsallis with α=0.5α=0.5 and Rényi with α=0.5α=0.5, achieve the highest average accuracy, suggesting that appropriate parameter selection within established frameworks can be more effective than developing complex hybrid measures. The success of α=0.5α=0.5 indicates that emphasizing minority classes benefits classification accuracy across diverse datasets. Second, the Friedman test reveals no significant global differences among the 23 measures, implying that for many practical applications, the choice among top-performing measures may have minimal impact on accuracy, allowing practitioners to prioritize other factors such as interpretability, computational efficiency, or theoretical guarantees. Third, ITC’s strong theoretical properties do not translate into empirical superiority, highlighting the importance of balancing theoretical elegance with practical effectiveness. Fourth, the superior performance of Shannon–Polarization over ITC variants suggests that simpler hybridization approaches may be preferable when theoretical guarantees are not paramount. 5.2 Key Advantages of ITC Despite not achieving top empirical performance, ITC offers several important advantages that make it valuable in specific contexts. Its rigorous theoretical foundation provides proven mathematical properties including concavity under explicit parameter conditions, proper boundary conditions, and connections to classical measures, ensuring non-negative impurity reduction and a sound basis for subsequent inference. This is particularly important in applications requiring interpretability and reliability, such as medical diagnosis or credit scoring. The flexible parameterization framework with three parameters (α, β, γ) enables domain-specific optimization and adaptation to particular data characteristics. In specialized applications where prior knowledge about class distributions is available, ITC can be tuned to outperform fixed measures. With O(K)O(K) complexity and no logarithmic operations, ITC is computationally efficient and suitable for large-scale applications and real-time systems. The systematic hybridization methodology provides a template for future hybrid measure development with clear mathematical justification, and the combination of information-theoretic uncertainty and distributional asymmetry offers a balanced perspective on impurity measurement even if empirical benefits are modest in general benchmarks. 5.3 Practical Implications and Usage Guidelines Based on our comprehensive analysis, we offer evidence-based guidance for practitioners. For applications prioritizing maximum empirical performance, Tsallis (α=0.5α=0.5) or Rényi (α=0.5α=0.5) are recommended based on our benchmark results. Gini impurity remains a highly competitive classical baseline with minimal implementation complexity. Shannon–Polarization provides a simple hybrid perspective without the complexity of ITC’s parameterization. ITC is recommended when mathematical properties such as concavity and boundary conditions are priorities—for example, in regulated industries where model interpretability and reliability are mandated, in research contexts investigating hybrid measure design, or in specialized applications where domain-specific parameter tuning is feasible and beneficial. 5.4 Limitations and Future Directions Our work has several limitations that suggest directions for future research. The evaluation was conducted on datasets ranging from 150 to 1,800 samples; performance on larger-scale datasets with tens of thousands of samples remains to be established. Future work should include large-scale benchmarks from domains such as image classification, bioinformatics, and natural language processing. ITC does not achieve top empirical performance despite strong theoretical foundations; refining the hybridization approach through non-linear combinations or adaptive gating mechanisms could improve practical effectiveness while preserving theoretical properties. The grid search approach for parameter optimization is computationally intensive; developing adaptive or meta-learning approaches for automatic parameter selection could enhance practical utility. Optimal parameters may vary across application domains; investigating domain-specific parameter configurations could reveal scenarios where ITC outperforms simpler measures. Application to ensemble methods such as Random Forests and Gradient Boosting represents a natural extension, as ITC’s theoretical properties may confer advantages in ensemble contexts. Further investigation into why Shannon–Polarization outperforms ITC could inform refinements to the hybridization methodology. Finally, our evaluation focused primarily on accuracy; future studies should consider additional metrics such as F1-score, AUC-ROC, tree depth, interpretability, and robustness to noise. 5.5 Broader Context and Contributions While ITC does not claim empirical superiority, this work makes several important contributions to the field. We provide an extensive comparative evaluation of impurity measures, with 23 measures across 7 datasets and rigorous statistical analysis, offering valuable empirical data for the community. Our transparent reporting of results including limitations contributes to more realistic expectations about hybrid measure development. The principled approach to hybrid measure design with clear mathematical justification provides a methodological framework that can inform future research. The theoretical foundations, including rigorous proofs of key properties under appropriate conditions, advance our mathematical understanding of impurity measures. Finally, our open-source implementation and benchmark results are publicly available to facilitate verification, extension, and adoption by the research community. 6 Conclusion We presented the Integrated Tsallis Combination (ITC), a hybrid impurity measure that combines normalized Tsallis entropy with exponential polarization through a convex combination. ITC is distinguished by its rigorous mathematical framework, including proven concavity under explicit parameter conditions, proper boundary conditions, and connections to classical measures. While simple parametric measures (Tsallis α=0.5α=0.5) lead in raw accuracy, ITC offers a theoretically sound alternative with competitive performance (88.38–89.16% accuracy) and computational efficiency matching Gini impurity. Our work demonstrates that theoretical robustness and empirical performance are distinct but equally important objectives. ITC’s value lies in its solid theoretical grounding, flexible parameterization, and the principled hybridization methodology it embodies. For applications where mathematical guarantees are paramount—such as regulated industries, interpretable modeling, or research contexts—ITC provides a rigorous alternative to purely empirical measures. The extensive empirical benchmark, comprising 23 measures across 7 datasets with 5-fold repetition over 150 compute-hours, offers valuable insights for practitioners and researchers. Statistical analysis reveals no significant global differences among top performers, suggesting practical equivalence for many applications and enabling informed selection based on secondary criteria. Our evidence-based guidelines assist practitioners in choosing appropriate measures based on their specific priorities. Future work will explore ITC’s performance on larger-scale datasets, its integration into ensemble methods, adaptive parameter selection mechanisms, and extensions to other hybrid combinations. By releasing our implementation and results openly, we invite the community to build upon this work and further bridge the theory–practice gap in impurity measure design. Acknowledgments We thank the anonymous reviewers for their valuable feedback and suggestions, which significantly improved this work. We also acknowledge the contributions of the open-source community in providing the datasets and computational tools that made this research possible. References [1] L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen, Classification and Regression Trees. CRC Press, 1984. [2] J. R. Quinlan, C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993. [3] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, 2009. [4] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, p. 5–32, 2001. [5] C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, no. 3, p. 379–423, 1948. [6] R. Rai and S. Sharma, “An analysis of the Gini index for decision tree construction,” International Journal of Computer Sciences and Engineering, vol. 7, no. 5, p. 312–316, 2019. [7] H. Liu, X. Zhou, and Y. Qin, “Entropy-based splitting criteria in decision trees: A comprehensive study,” Information Sciences, vol. 512, p. 715–731, 2020. [8] A. Rényi, “On measures of entropy and information,” in Proc. 4th Berkeley Symp. Math. Statist. Probab., vol. 1, 1961, p. 547–561. [9] C. Tsallis, “Possible generalization of Boltzmann–Gibbs statistics,” Journal of Statistical Physics, vol. 52, no. 1–2, p. 479–487, 1988. [10] Y. Zhang, J. Wang, and L. Chen, “Polarization-based impurity measures for decision tree learning,” Pattern Recognition, vol. 118, p. 108035, 2021. [11] T. Nguyen, M. Pham, and H. Le, “Distance-based impurity measures using optimal transport theory for decision trees,” Machine Learning, vol. 111, no. 4, p. 1423–1450, 2022. [12] S. K. Murthy, “Automatic construction of decision trees from data: A multi-disciplinary survey,” Data Mining and Knowledge Discovery, vol. 2, no. 4, p. 345–389, 1998. [13] J. Su and H. Zhang, “An investigation of impurity measures for decision tree induction,” Intelligent Data Analysis, vol. 20, no. 4, p. 839–860, 2016. [14] P. Probst, A.-L. Boulesteix, and B. Bischl, “Tunability: Importance of hyperparameters of machine learning algorithms,” Journal of Machine Learning Research, vol. 20, no. 53, p. 1–32, 2019. [15] A. Fernández, S. García, M. Galar, R. C. Prati, B. Krawczyk, and F. Herrera, Learning from Imbalanced Data Sets. Springer, 2018. [16] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2016, p. 785–794. [17] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “LightGBM: A highly efficient gradient boosting decision tree,” in Advances in Neural Information Processing Systems, vol. 30, 2017, p. 3146–3154. [18] R. A. Fisher, “The use of multiple measurements in taxonomic problems,” Annals of Eugenics, vol. 7, no. 2, p. 179–188, 1936. [19] S. Aeberhard, D. Coomans, and O. De Vel, “Comparative analysis of statistical pattern recognition methods in high dimensional settings,” Pattern Recognition, vol. 27, no. 8, p. 1065–1077, 1994. [20] W. N. Street, W. H. Wolberg, and O. L. Mangasarian, “Nuclear feature extraction for breast tumor diagnosis,” in IS&T/SPIE’s Symp. Electronic Imaging, 1993, p. 861–870. [21] F. Alimoglu and E. Alpaydin, “Methods of combining multiple classifiers based on different representations for pen-based handwritten digit recognition,” in Proc. 5th Turkish Artificial Intelligence and Artificial Neural Networks Symp., 1996. [22] A. Mazumder, P. Hartley, and M. Basu, “A unified CART framework for impurity-based decision tree construction,” Journal of Machine Learning Research, vol. 24, no. 1, p. 1–45, 2023. [23] S. Patil and S. V. Athawale, “A comprehensive survey on decision tree induction algorithms,” Artificial Intelligence Review, vol. 54, no. 3, p. 2137–2196, 2021. [24] B. Li, Q. Wang, and J. Hu, “Deep learning vs. decision trees: A comprehensive comparison on tabular data,” Pattern Recognition, vol. 128, p. 108669, 2022. [25] Y. Chen, X. Liu, and W. Zhang, “Hybrid impurity measures for decision tree learning: A review and empirical study,” Information Fusion, vol. 62, p. 12–27, 2020. [26] L. Wang, H. Li, and Z. Xu, “Adaptive impurity measures for imbalanced data classification,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 4, p. 4123–4137, 2023.

Instant research discovery