Paper deep dive
KGroups: A Versatile Univariate Max-Relevance Min-Redundancy Feature Selection Algorithm for High-dimensional Biological Data
Malick Ebiele, Malika Bendechache, Rob Brennan
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/31/2026, 2:44:44 AM
Summary
The paper introduces KGroups, a novel univariate filter feature selection (FFS) algorithm that functions as a univariate version of the Min-Redundancy Max-Relevance (mRMR) method. By utilizing clustering for feature selection rather than incremental search, KGroups achieves predictive performance comparable to multivariate mRMR while being significantly faster (up to 821 times) on high-dimensional biological datasets. The algorithm is parameterizable and supports various relevance estimators, offering a more efficient alternative to traditional mRMR and KBest methods.
Entities (5)
Relation Signals (3)
KGroups → isa → Univariate mRMR
confidence 95% · In this paper, we propose a new univariate mRMR called KGroups
KGroups → outperforms → KBest
confidence 95% · KGroups outperforms KBest.
KGroups → extends → SCSIG
confidence 90% · KGroups extends the Supervised Cosine Similarity Information Gain (SCSIG) proposed by Dubey and Saxena.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper proposes a new univariate filter feature selection (FFS) algorithm called KGroups. The majority of work in the literature focuses on investigating the relevance or redundancy estimations of feature selection (FS) methods. This has shown promising results and a real improvement of FFS methods' predictive performance. However, limited efforts have been made to investigate alternative FFS algorithms. This raises the following question: how much of the FFS methods' predictive performance depends on the selection algorithm rather than the relevance or the redundancy estimations? The majority of FFS methods fall into two categories: relevance maximisation (Max-Rel, also known as KBest) or simultaneous relevance maximisation and redundancy minimisation (mRMR). KBest is a univariate FFS algorithm that employs sorting (descending) for selection. mRMR is a multivariate FFS algorithm that employs an incremental search algorithm for selection. In this paper, we propose a new univariate mRMR called KGroups that employs clustering for selection. Extensive experiments on 14 high-dimensional biological benchmark datasets showed that KGroups achieves similar predictive performance compared to multivariate mRMR while being up to 821 times faster. KGroups is parameterisable, which leaves room for further predictive performance improvement through hyperparameter finetuning, unlike mRMR and KBest. KGroups outperforms KBest.
Tags
Links
- Source: https://arxiv.org/abs/2603.28417v1
- Canonical: https://arxiv.org/abs/2603.28417v1
Trouble viewing inline? Open PDF directly →
Full Text
72,695 characters extracted from source content.
Expand or collapse full text
Under-review KGroups: A Versatile Univariate Max-Relevance Min-Redundancy Feature Selection Algorithm for High-dimensional Biological Data Malick Ebiele 1,3* , Malika Bendechache 2,3 and Rob Brennan 1,3 1* Computer Science, University College Dublin, Dublin, Dublin, Ireland. 2 Computer Science, University of Galway, Galway, Galway, Ireland. 3 ADAPT Centre, Dublin, Dublin, Ireland. *Corresponding author(s). E-mail(s): malick.ebiele@adaptcentre.ie; Contributing authors: malika.bendechache@universityofgalway.ie ; rob.brennan@ucd.ie; Abstract This paper proposes a new univariate filter feature selection (FFS) algorithm called KGroups. The majority of work in the literature focuses on investigating the relevance or redundancy estimations of feature selection (FS) methods. This has shown promising results and a real improvement of FFS methods’ predictive performance. However, limited efforts have been made to investigate alternative FFS algorithms. This raises the following question: how much of the FFS meth- ods’ predictive performance depends on the selection algorithm rather than the relevance or the redundancy estimations? The majority of FFS methods fall into two categories: relevance maximisation (Max-Rel, also known as KBest) or simul- taneous relevance maximisation and redundancy minimisation (mRMR). KBest is a univariate FFS algorithm that employs sorting (descending) for selection. mRMR is a multivariate FFS algorithm that employs an incremental search algo- rithm for selection. In this paper, we propose a new univariate mRMR called KGroups that employs clustering for selection. Extensive experiments on 14 high- dimensional biological benchmark datasets showed that KGroups achieves similar predictive performance compared to multivariate mRMR while being up to 821 times faster. KGroups is parameterisable, which leaves room for further predictive performance improvement through hyperparameter finetuning, unlike mRMR and KBest. KGroups outperforms KBest. Keywords: Feature Selection, Clustering-based Filter Method, Univariate Max-Relevance Min-Redundancy (mRMR), Data Mining, Knowledge Discovery 1 arXiv:2603.28417v1 [cs.LG] 30 Mar 2026 Under-review JEL Classification: C02 , C38 , C45 , C55 MSC Classification: 68T01 , 68T20 1 Introduction Feature selection (FS) methods in machine learning (ML) and statistical analysis have existed since at least the 1990s with an ever increasing interest [1]. They define ways of selecting a subset of features from the original dataset while maintaining a competitive predictive performance. Feature selection methods are important in tackling computational cost (in terms of training time) and resource requirements [2]. Today, they are also important for addressing the energy consumption, efficiency, monetary cost of training, and the sustainability of Artificial Intelligence (AI) models [3–5]. There are three primary type of FS methods: filters, wrappers, and embedded. Wrappers are considered the most predictively performant, while filters are considered the most computationally efficient. Filters are also considered the most generalisable to other models (models they were not initially trained for; see Table 1 in section 2 below) [1, 2, 6]. With the rise of interest in green AI initiatives, filter feature selection (FFS) methods are receiving more attention from researchers and practitioners because they align with sustainable AI objectives [7–12]. This can also be explained by the fact that FFS methods have only one drawback to address (lag in predictive performance) compared to two drawbacks for wrappers (computational inefficiency and failure to generalise to other models). FFS methods have been widely applied to feature selection and dimensionality reduction tasks in heterogeneous domains including but not limited to biology, healthcare, marketing, information retrieval, human activity recognition [13–18, 18–22]. FS methods can be classified into two groups: value-based and non-value-based. Value-based FS methods first assign a relevance or a redundancy value to each feature then perform the selection using either or both of the assigned relevance and redundancy values. Max-Rel, mRMR, and KGroups are all value-based FFS. Non-value-based FS techniques perform the selection task without assigning any values to the features; they are in majority wrappers (see Table 1 in the background section). This paper focuses on value-based FFS methods because they are computationally efficient, have only one drawback, and can be integrated with explainable AI [23]. Value-based FFS estimate features relevance using a similarity measure from i) information theory (e.g. mutual information also known as information gain), i) statistics (e.g. F-value also known as F-statistics), or i) machine learning models’ built-in feature ranking methods also known as embedded FS techniques. Once a relevance value is assigned to each of the feature, the features are sorted in descending order of relevance and the topkare selected (thus the names Max-Rel, SelectKBest or KBest). As to multivariate FFS methods, a redundancy estimator is needed in addition to the relevance estimator. They consist of maximising the difference between or the quotient of the relevance 2 Under-review Rel(F i ) by the redundancyRed(F i ;S) of the candidate featureF i with respect to the already selected featuresSusing an incremental search algorithm (see section 3, particularly Eqs. (5) and (7)). These methods are referred to as mRMR meaning Min-Redundancy Max-Relevance. This paper proposed a new versatile univariate algorithm, called KGroups, that simultaneously maximises the relevance and minimises the redundancy using clustering. In other words, KGroups is a univariate mRMR. First, a relevance value is assigned to each feature. Then, the features are clustered intokgroups using binning. Finally, the feature with the maximum relevance value from each cluster is selected (for more details, see section 4 below). There are been previous attempts to use binning-based clustering or discretisation for FS and dimensionality reduction [15,24–27]. KGroups extends the Supervised Cosine Similarity Information Gain (SCSIG) proposed by Dubey and Saxena[15]. SCSIG was originally designed for cosine similarity-based binning and mutual information-based selection (see section 4 for more details). The paper addresses the following research question: To what extent can a univariate Max-Relevance Min-Redundancy filter feature selection algo- rithm improve the performance of feature selection methods in terms of classification accuracy and computational cost? To answer the research question above, we performed extensive experimental analysis of KGroups with competing algo- rithms (KBest, and multivariate mRMR) on 14 high-dimensional biological benchmark datasets and reported their respective average 5-fold cross validation accuracy score and execution time (see section 6). This paper has the following contributions: (1) Design and implementation of KGroups, a versatile univariate mRMR algorithm that competes with the state of the art multivariate mRMR FFS algorithm while being up to 821 times faster. (2) Extension of the SCSIG [15] FFS method into a flexible algorithm that support any type of relevance estimator. KGroups is parameterisable and introduces the concept of tie-breaking; unlike SCSIG. (3) Comprehensive Analysis of KGroups, highlighting its strengths, weaknesses, and potentials. The remainder of the paper is as follows. The next section defines the cores concepts of FFS methods. In section 3, we present the related work. Section 4 and 5 describes our proposed method and the experimental setting, respectively. The experimental results and discussion are presented in section 6. The limitations and potentials of the KGroups algorithm is presented in section 7. Finally, the concluding remarks and future work is highlighted in section 8. 2 Background In this section, we provide the definitions of the core concepts of the Feature Selection (FS) domain. 2.1 Feature Selection Methods There are three primary types of FS methods: filters, wrappers, and embedded [1,2, 6,28]. Table 1 below gives a comparative analysis of the three primary types of FS in terms of relative predictive performance, computational cost, and generalisation 3 Under-review [of the selected features] to other models. It also provides a brief description of each method and examples. Table 1: Comparative analysis of the primary types of feature selection methods (adapted from [6]) TypePredic- tive Perfor- mance Compu- tational Cost Genera- lisation to other models Example Methods Short Description Filters LowLowHighSelectKBest; SelectPer- centile - Perform feature selection at pre- processing without relying on ML model - Use statistics, information theory, or geometry to measure relevance and redundancy - Model-agnostic approach WrappersHighHighLowForward Selection; Backward Selection; Recursive Selection; Exhaustive Selection - Integrate variable selection into train- ing process - Evaluate subsets based on predictive performance - Can use SHAP values, permutation importance - Assigned values can be negative Embedded Medium MediumMediumGiniand Information Gain Impor- tance - Use ML models with built-in feature selection - Tied to specific models (e.g. tree- based models) - Assign importance values during training - Non-relevant features often assigned zero 2.2 Filter Feature Selection Methods According to Table 1, FFS methods have only one shortcoming which is their low predictive performance compared to wrapper and embedded methods. In this paper, we propose a value-based FFS algorithm that achieves similar average 5-fold cross validation accuracy compared to competing algorithms across 14 benchmark datasets (see Tables 4 and 5), three heterogenous relevance estimators (see Eqs. (1)-(3) below), and 6 popular classifiers (see section 5.1 for more details about the classification models). 4 Under-review For relevance estimation, this paper employed mutual information, also known as information gain (I), F-value aka F-statistics (F), and Random Forest Gini importance (Gini), also known as Mean Decrease in Impurity (MDI); see Eqs. (1), (2), and (3), respectively. I(X,Y ) = I(Y,X) = X x i ∈X X y i ∈Y P(x i ,y i ) log P(x i ,y i ) P(x i )P(y i ) (1) F-value = Explained variance Unexplained variance = Between-sample variability Within-sample variability ,(2) TheFcalculation involves many steps which are too long to present here. The formula above (Eq. (2)) is just illustrative. A comprehensive and detailed explanation can be found in Siegel [29]. Gini x j = MDI x j = 1 T T X t=1 X nodes n split on x j p(n) [ I(n)− p left I(left)− p right I(right) ] (3) In the Gini importance formula (Eq. (3)) ,Tis the number of trees in the forest;n the node in treetthat splits on featurej;p(n) the proportion of samples reaching noden;I(n) the impurity of noden(in terms of the Gini index);p left andp right are the proportions of samples going to the left and right child, respectively; and I(left), I(right) the impurity of the left and right children, respectively. The Gini index is defined asG= 1− X c j ∈Y p(c j ) 2 , withYthe set of unique classes inYand p(c j ) the probability of samples belonging to class c j in the node or dataset. 3 Related Work In this section, we review the existing FFS literature. Before doing so, we provide an overview of the FFS methods as found in literature in Fig. 1. This figure illustrates the key steps and decision points involved in current FFS approaches. Section 3.1 below gives the general formulation of the state of the art FFS algorithms (Max-Rel and mRMR). The vast majority of the FFS methods found in the literature is an instance of one of these general formulation shown in Eqs. (4)-(7); independently from their respective relevance and redundancy estimators they use. 3.1 General formulation of Max-Rel and mRMR The equations (4) and (5) are the general formulation of the Max-Rel and mRMR feature selection problems.Fis the set of all features,Sis the set of selected features, F the set of non-selected features, andF i is the candidate feature.Rel(F i ) is the relevance value ofF i andRed(F i ,S) is the redundancy value ofF i with respect to the already selected features. 5 Under-review The objective of Max-Rel is to addF i toSif and only ifRel(F i ) is the maximum relevance value of all the features inF . To do so, Max-Rel uses a descending sorting algorithm and return the top k of the features with k ≤|F|. mRMR, on the other hand, aims to addF i toSif and only ifF i maximises the difference or quotient between its relevanceRel(F i ) and its redundancyRed(F i ,S) with respect to the already selected features. The first formulation by Battiti[30] includes a redundancy regulation parameterβ ∈[0,1]. This parameterβis set to 1 in the subsequent studies, starting from Ding and Peng[31].β= 1 is the standard in recent mRMR-based FFS methods (Eq. (6)). Ding and Peng[31]proposed another reformulation of mRMR by replacing the subtraction operation between relevance and redundancy by a division operation (see Eq. (7)). More details on mRMR-based methods are provided in the section 3.3. max F i ∈F Rel(F i )(4) max F i ∈F Rel(F i )− β× Red(F i ;S)(5) max F i ∈F Rel(F i )− Red(F i ;S)(6) max F i ∈F Rel(F i ) Red(F i ;S) (7) FFS methods can be classified into two main groups: univariate and multivariate. This classification mainly focus on the number of variable involved in the actual selection algorithm i.e. sorting or an incremental search algorithm i.e. sequential forward search. For instance, a feature selection method using random forest Gini importance (an embedded FS method) can be classified as univariate FFS even though the relevance estimation involves many features and their interactions [2]. On the other hand, multivariate FFS incorporate an estimation of feature interaction (redundancy or complementarity) and uses a search algorithm such as the sequential forward search (a greedy search algorithm) to select the optimal feature subset [30–33]. Figure 1 below give an overview of FFS methods on a tabular data as found in the literature. 3.2 Univariate FFS This is the most used family of FFS methods because they are the most computational efficient and easily adaptable from one relevance estimator to another [4,34,35]. Descending sorting is the only search algorithm used for Max-Rel feature selection methods. Even though they are considered less predictively performant than multivariate FFS methods (mRMR), variance-based Max-Rel has been shown to outperform mRMR and embedded FS methods (including random forest Gini importance, Permutation importance, and Boosting importance from the eXtreme Gradient Boosting model) in a systematic benchmark study on high-dimensional biological datasets [34]. This kind of studies are missing in the extant literature, especially in recent years. These results are not surprising but highlight the challenges of the FS task where every single 6 Under-review Fig. 1: Overview of FFS methods on a tabular data as found in the literature. decision can positively or negatively impact the expected outcome. FS challenges and gaps are discussed in section 3.4 below. 3.3 Multivariate FFS The first Min-Redundancy Max-Relevance (mRMR) FFS method was introduced by Battiti[30]and called mutual information-based feature selection (MIFS). It uses mutual information for both relevance and redundancy estimation (see Eq. (8)). Battiti[30]used an incremental search algorithm (a greedy algorithm) to solve the objective function shown in Eq. (8). max X i ∈F I(X i ;Y )− β× X X j ∈S I(X i ;X j ),(8) where Rel(X i ) = I(X i ;Y ) and Red(X i ;S) = P X j ∈S I(X i ;X j ). Recent mRMR-based FFS methods used one of the two formulation proposed by Ding and Peng[31]shown in Eqs. (6) and (7). They also proposed a modified version of the mutual information-based redundancy estimation such asRed(X i ;S) = 1 |S| P X j ∈S I(X i ;X j ). Ding and Peng[31]also proposed a new relevance and redundancy estimator using F-value and Pearson correlation coefficient, respectively. They also used an incremental search algorithm to solve the objective functions of their proposed 7 Under-review mRMR-based FFS methods. The four variants proposed by Ding and Peng[31]are shown in Eqs. (9)-(12). max X i ∈F I(X i ;Y )− 1 |S| X X j ∈S I(X i ;X j )(9) max X i ∈F I(X i ;Y ) 1 |S| P X j ∈S I(X i ;X j ) (10) max X i ∈F F-value(X i ;Y )− 1 |S| X X j ∈S ρ(X i ;X j )(11) max X i ∈F F-value(X i ;Y ) 1 |S| P X j ∈S ρ(X i ;X j ) (12) ρ (X i ;X j ) = cov(X i ;X j ) σ X i σ X j is Pearson correlation coefficient; wherecov(X i ;X j ) is the covariance ofX i andX j ;σ X i andσ X j are the standard deviations ofX i andX j , respectively. In the extant literature, most efforts are made on finding alternative relevance or redundancy estimators [19,36–42]. For instance, in a recent study [43], the authors proposed a relevance estimator using the maximum information coefficient (MIC) and a redundancy estimator using the wavelet similarity, max X i ∈F MIC(X i ;Y )− 1 |S| X X j ∈S WTSim(X j ;X i ),(13) where Rel(X i ) = MIC(X i ;Y ) = max |X i |,|Y|<B I(X i ;Y ) log 2 ( min(|X i |,|Y|) ) ,(14) Red(X i ;S) = 1 |S| X X j ∈S WTSim(X j ;X i ) = 1 |S| X X j ∈S Similarity(hash(X j ),hash(X i )), (15) B =|F| 0.6 , andSimilarity(hash(X j ),hash(X i ))is the similarity of the features at different scales after wavelet decomposition. For more details, please refer to Yuan et al. [43]. Another example is found in Yuan et al.[44]where the objective function is defined as max X i ∈F SU(X i ;Y )− 1 |S| X x j ∈S Sim normal (hash(X j ),hash(X i )),(16) 8 Under-review where Rel(X i ) = SU(X i ;Y ) = 2· I(X i ;Y ) H(X i ) + H(Y ) , and(17) Red(X i ;S) = 1 |S| X x j ∈S Sim normal (hash(X j ),hash(X i )).(18) SU is the symmetry uncertainty, andSim normal (hash(X j ),hash(X i )) is the nor- malised similarity betweenX i andX j after a cosine transform (like a Fourier transform used in signal processing); for more details, please refer to [44]. Zhao et al.[16]proposed three new extensions of mRMR by introducing random forest Gini importance as relevance estimator and the randomized dependence coef- ficient (RDC) as a [non-linear] redundancy estimator. The first extension consists of replacingρ(X i ;X j ) byρ rdc (X i ;X j ) in Eq. (12). The second consists of replacing F-value(X i ;Y) by random forest Gini importance in Eq. (12). The third and last extension consists of substitutingF-value(X i ;Y) by random forest Gini importance andρ(X i ;X j ) byρ rdc (X i ;X j ) in Eq. (12). The randomized dependence coefficient ρ rdc (X i ;X j ) is defined as the largest canonical correlation between the non-linear pro- jections ofX i andX j ; for more details, please refer to Lopez-Paz et al.[45]. Ihianle et al. [33]also proposed four new extensions of mRMR. The first two consists of replacing ρ(X i ;X j ) bydCor(X i ;X j ) in Eqs. (11) and (12).dCor(X i ;X j ) = dCov(X i ,X j ) √ dVar(X i ) dVar(X j ) is the distance correlation coefficient betweenX i andX j . It is a statistical measure that evaluates the interdependence between two random variables by considering both linear and nonlinear relationships [46]. The last two consists of replacingρ(X i ;X j ) byMIC(X i ;X j ) in Eqs. (11) and (12). There are no dedicated relevance or redundancy estimators; meaning, any relevance estimator can be employed as a redundancy estimator, and vice versa. For instance, Yuan et al.[43]employedMIC as a relevance estimator while Ihianle et al. [33] used it as a redundancy estimator. Some studies include conditionality in the estimation of the relevance or redundancy [41,47,48]. For instance, Zhang and Gao[48]redefined mRMR as shown in Eq. (19) below by including conditional mutual information and conditional entropy terms in the redundancy estimator. In a more recent study [49], the authors also defined redundancy in terms of a conditional mutual information as shown in Eq. (20). They redefined the relevance estimator as followRel(X i ) = I(X i ;Y ) H(Y ) . Given that both, the relevance and redundancy estimators, have the same denominator, by multiplying Eq. (20) byH(Y) one gets Eq. (21). Given the common denominator, their redefinition can be reduced to Eq. (21) below. max X i ∈F I(X i ;Y )− 1 |S| X X j ∈S I(X i ;X j )− X X j ∈S I(X j ;Y|X i ) H(Y|X i ) + H(Y|X i ,X j ) (19) 9 Under-review max X i ∈F I(X i ;Y ) H(Y ) − 1 |S| X X j ∈S 1− I(X j ;Y | X i ) + I(X i ;Y | X j ) 2H(Y ) (20) max X i ∈F I(X i ;Y )− H(Y )− 1 2|S| X X j ∈S I(X j ;Y | X i ) + I(X i ;Y | X j ) (21) All those mRMR-based methods found in the literature still used the original incremental search algorithm algorithm (or a variant) to select subsequent features from F . That makes all of them computational expensive, and therefore, they generalised poorly when|F|increases considerably [50] (also see section 6.3, particularly Fig. 5 and Table 8). Another observation is that none of them investigate both formulae of mRMR (shown in Eqs. (6) and (7)) despite studies showing that it is very difficult to know a priori which one will yield the optimal subset S ∗ from a given feature set F [35, 51]. To address mRMR-based methods computational requirements, Ramírez-Gallego et al.[50]proposed two new implementations of mRMR using graphics processing units (GPUs) for parallel computing, and Apache Spark for distributed comput- ing. Another approach could be to employ less computationally expensive relevance and redundancy estimators. For instance, mRMR using F-value and Pearson correlation is faster than using mutual information for both relevance and redundancy because mutual information is more expensive to estimate than F-value and Pearson correlation [16] (also see Fig. 5). Dynamic programming could be another option to speed up mRMR-based FFS methods. 3.4 Challenges of the FS task FS is a mature domain that has been developing for decades. Despite its maturity, there are still a lot of challenges to address and questions to answer. For instance, given an optimal feature subsetS ∗ , any feature added to or removed fromS ∗ can worsen its predictive performance by multiple points. This is becauseS ∗ predictive performance is determined by individual or non-contextual relevance of each feature as well as interactive or contextual relevance (i.e. redundancy and complementarity) [2]. While the non-contextual relevance is constant, the contextual relevance changes every timeSis modified. In theory, the only way to guarantee to select the optimal subsetS ∗ is through exhaustive search wrappers. Exhaustive search is a very complex and time-consuming process because it expands exponentially with respect to the number of features [1,52]; meaning there are exactly 2 |F| − 1 possible combinations of features 1 . Exhaustive search is not practical for high-dimensional datasets [2, 52]. As to FFS, it is a exploratory task meaning one needs to test multiple relevance (and redundancy in the case of multivariate selection) estimators. It also means testing 1 −1 because the empty set∅has to be removed from the power setP(F) ofF. The power setP(F) of the set F is the set of all possible subsets of F, including the empty set ∅ and F itself. 10 Under-review multiple values ofk. This exploratory approach gives a higher chance of getting a feature subsetSthat achieves a high predictive performance, but with higher computational cost. For instance, in their famous paper, Guyon and Elisseeff[2]propose a check list on how to perform such exploration. In this paper, we propose KGroups, a new versatile univariate mRMR algorithm that can easily be applied to any type of relevance estimator i.e. supervised (such as mutual information or F-value), unsupervised (such as entropy or variance), embedded (such as random forest Gini importance). KGroups is fast and and parameterisable. KGroups is as fast as KBest and significantly faster than its multivariate counterpart of mRMR. 4 Proposed Method This section provides a detailed description of the proposed univariate FFS algorithm based on clustering, KGroups. A binning clustering approach is chosen over classical ML clustering techniques (which will be explored in future studies) because it is an effective value-based univariate grouping method, computationally efficient, and parameterisable (see section 6). Equation (22) is the general formulation of the KGroups algorithm. It consists of maximising the Rel(F i ) within each feature cluster created using binning. max b j ∈f(j;α) Rel(F), subject to Rel(F)≤ b j ,(22) where f(j;α) = min Rel(F) + (max Rel(F)− min Rel(F))× u α j (23) andu j = j k ,j= 1,2,..,k; andα >0, called power, is a real number used as the bin size regulation hyperparameter (preferably α∈ [0.05; 3]). KGroups has three main steps: i) relevance estimation, i) feature clustering based on their relevance values, and i) actual feature selection. Each step is described below. 1.Relevance Estimation. Like Max-Rel and mRMR, one has to select a relevance estimator and calculate the relevanceRel(F i ) of each feature in the dataset. Any relevance estimator from the state of the art can be used (see section 3). 2.Feature Clustering. This step is built upon the last one and uses the relevance values to cluster the features intokgroups. Here, each featureF i is assigned to the clusterC j if and only ifRel(F i )∈]f(j−1;α);f(j;α)]; noting thatf(0;α) = min Rel(F) (see Eq. (22)). Once the feature clustering is complete, we proceed to the actual feature selection step. 3.Actual Feature Selection. The goal here is to select the feature with the maximum relevance valueRel(F i ) within each clusterC j forj= 1,2,..,k. In practice, there is often more that one feature with the highest relevance value within a given cluster. There are many ways to address this issue. One is to return the first or last feature within the cluster with the highest relevance value. Another way is to randomly 11 Under-review select one or to just select all of the them. The first three approaches (select first, last or one randomly) have one major drawback which is reproducibility. If the features are shuffled then the resulting feature will be totally different. The second issue is explainability; why is that feature preferred over the others. As to selecting all the features with the highest relevance value, it fails to effective reduce the number of features when the the distribution of the relevance values is skewed. To address all of the issues highlighted in the above paragraph, we introduced the concept of tie-breaking. It is a sequential process which consists of using additional relevance estimators, one at a time, until either all the ties are broken or the list of tie-breakers is exhausted. If all the ties are broken before the exhaustion of the list of tie-breakers, the process stops. On the other hand, if the list is exhausted and some ties are not broken, the remaining features are returned. This concept of tie-breaking adds a layer of robustness to the KGroups algorithm; at the cost of a little extra computation time. 5 Experimental Setup The experiments are designed to evaluate the three feature selection algorithms i.e. KGroups (ours), mRMR, and KBest, ability to extract low-dimensional data from high-dimensional datasets, the robustness across classifiers of the low-dimensional data extraction, and the time complexity of the respective algorithms. Our experiments employed 14 high-dimensional biological feature selection bench- mark datasets (see Table 2), 6 widely used classifiers (with 4 highly reported in the FS literature i.e. KNeighbors, RandomForest, GaussianNB, and LinearSVC; see section 5.1 below), and 3 relevance estimators (mutual information, F-value, and Gini importance). The experimental configuration includes KBest (Max-Rel), the two variants of mRMR (shown in Eqs. (6) and (7)), and seven variants of KGroups (forα ∈ 0.3; 0.5; 0.7; 1.0; 1.3; 1.5; 1.7; see Eqs. (22) and Eq. (23)) for mutual information and F-value relevance estimators. As to Gini importance relevance estimator, only one variant (the quotient variant proposed by Zhao et al.[16]; see Eq. (7)) of mRMR is available in public Python packages 2 [53], thus only this one has been included. We performed hyperparameter finetuning for KGroups and mRMR algorithms. For KGroups, we finetuned the power hyperparameterα∈0.3; 0.5; 0.7; 1.0; 1.3; 1.5; 1.7. For mRMR, we finetuned the operator betweenRel(F i ) andRed(F i ,S); meaning between the difference (mRMR_D, see Eq. (6)) and quotient (mRMR_Q, see Eq. (7)) formulations of mRMR. KGroups is more suitable for hyperparameter finetuning because it is fast compared to mRMR; in fact, on average (across the three rele- vance estimators) one KGroups variant is 122.8 times faster than one of mRMR (see section 6.3 for more details, especially Fig. 5 and Table 8). In other words, the time complexity of 123 variants of KGroups (123 different values of the the power hyperparameterα) is equivalent to one of mRMR (mRMR_D or mRMR_Q). KBest has only one variant (no hyperparameters). 2 https://feature-engine.trainindata.com/en/latest/user_guide/selection/MRMR.html 12 Under-review 5.1 Classifiers The 6 classifiers used arek-Nearest Neighbors Classifier (KNeighbors), Gaussian Naive Bayes Classifier (GaussianNB), Random Forest Classifier (RandomForest), eXtreme Gradient Boosting Classifier (XGB), Multi-Layer Perceptron Classifier (MLP), and Linear Support Vector machine Classifier (LinearSVC). The classifiers are available in popular machine learning packages (XGB in XGBoost [54] package and the rest in the scikit-learn [55] package). The classifiers are used with their default parameters and no hyper-parameter finetuning was performed. The short name of the classifiers between parenthesis has been automatically extracted from their respective Python packages. These short names are used when referencing the classifiers in the rest of this paper. KNeighbors, RandomForest, GaussianNB, and LinearSVC have been chosen because they have been used in recent feature selection papers [15,43,44] and have been highly reported in the last two decades in FS literature. Another reason for selecting GaussianNB is because it is a probabilistic model. These are all widely used for classification tasks and are considered effective. XGB has been chosen because it is a robust tree-based model (compared to DecisionTree also known as classification and regression trees or CART) and MLP because it is a [simple] neural network (Feed Forward Network or FFS). These reasons aligned with the justifications given by Ebiele et al.[52]for selecting classifiers with different architectures and underlying learning principles. Choosing these classifiers will allow us to study the robustness of the FS methods and algorithms. It also simplifies the comparison with other FS methods. 5.2 Datasets The 14 high-dimensional biological benchmark datasets employed in the experiments are from Li et al.[56]’s feature selection repository. Li et al.[56]’s paper is a highly cited systematic review of feature selection literature which identified a total of 29 feature selection benchmark datasets including biological, text, face image, and hand-written image datasets. The 14 datasets selected for our experiments are all the biological benchmark datasets available on Li et al.[56]’s feature selection repository. In terms of tasks, there are binary and multiclass classification datasets with the maximum number of classes being 11 by the dataset Carcinom. The datasets with the lowest and highest dimensions are lung_discrete and GLI_85, respectively. In terms of sample size, these are all small datasets with the number of entries ranging from 50 to 203. SMK_CAN_187 is the largest dataset in terms of number of entries and features (dimensionality) combined with 187 entries and 19993 features. All these details about the datasets are summarised in Table 2 below. The datasets are sorted alphabetically by name (case sensitive sorting). This is very typical with tabular biological datasets, they are usually small size and high-dimensional [34,43,56]. This type of datasets (small size and high-dimensional) are not eligible for deep neural networks (DNNs) because of their size. Additionally, Grinsztajn et al.[57]have demonstrated that tree-based model (e.g. RandomForest, XGB) outperform DNNs (including Transformer models) on tabular data, with much less computational cost, due to irregular patterns in the target function, uninformative features, and non rotationally-invariant data where linear combinations of features 13 Under-review misrepresent the information. The failure of DNNs on tabular data has been noticed before Grinsztajn et al.[57]and gave birth to an entirely new research area called tabular deep learning. This new research area aims to stimulate efforts to design and implement DNNs architecture specialised on learning from tabular data. Table 2: The list of benchmark datasets used in our experiments. Name#Instances #Features #Classes ALLAML7271292 CLL_SUB_111111113403 Carcinom174918211 GLI_8585222832 GLIOMA5044344 Prostate_GE10259662 SMK_CAN_187187199932 TOX_17117157484 colon6220002 leukemia7270702 lung20333125 lung_discrete733257 lymphoma9640269 nci96097129 5.3 Training Setup and Evaluation For k ∈ [2, 100], each classifier is trained using 5-fold cross validation. The classifiers have been evaluated using the accuracy metric for ease of comparison to related work. It is defined as: Accuracy = TP+TN TP+TN+FP+FN (see confusion matrix in Table 3)(24) The classifiers were trained on a shared high performance computing (HPC) node running a Red Hat Enterprise Linux (RHEL) 9.5 machine with 356 GB RAM and an Intel Xeon Gold 6152 CPU (30.25 MB L3 Cache and 2.1 up to 3.7 GHz Frequency). The datasets have been scaled using the scikit-learn Standard scaler before training. For KGroups, cosine similarity has been employed as tie-breaker for Mutual information and Mutual information has been employed as tie-breaker for F-value and Gini importance. 6 Experimental Results and Discussion The experimental results are shown and discussed below for a set of 3 FS related tasks. KGroups, KBest, and mRMR algorithms have been compared on low-dimension feature extraction, their robustness across heterogeneous ML models, and execution 14 Under-review Table 3: Confusion matrix. Actual Positive (P)Negative (N) Predicted Positive (P)True Positive (TP)False Positive (FP) Negative (N)False Negative (FN)True Negative (TN) time. To make comparison easier to understand and follow, we performed a pairwise comparision of the three FFS algorithms; meaning KGroups versus KBest, then KGroups verus mRMR for the low-dimension feature extraction task and the robustness across heterogeneous ML models of the extraction. 6.1 Low-dimension Feature Extraction The low-dimension feature extraction task consists of selecting a subset ofkfeatures from the original feature set such thatk ≪|F|; here we chosek= 100. Therefore, for k ∈[2,100], we trained 6 different classifiers using 5-fold cross validation and recorded the average and standard deviation accuracy score. Table 4 displays the best overall experimental configuration per relevance estimator and dataset as a result of systematic search. The results reported below are the best configurations among 594 possible results per variant per dataset (594 = 99 possible value ofk ×6 classifiers). For mRMR and KGroups, the variant (after hyperparameter finetuning) with the highest accuracy per dataset is reported. KGroups versus KBest: KGroups with Gini importance relevance estimator got the best configuration on 7 datasets. KGroups with mutual information estimator got the best configuration on 9 datasets. KGroups with F-value relevance estimator got the best configuration on 10 datasets. KGroups and KBest tied on 4, 3, and 1 datasets, for Gini importance, mutual information, and F-value relevance estimator, respectively. Across the three relevance estimators, KGroups achieved best configuration on 11 datasets overall, tying with KBest on 4 datasets (7 excluding ties). KBest achieved best configuration on 2 datasets excluding ties. Both, KGroups and KBest failed to achieved the best configuration on the TOX_171 dataset. KGroups versus mRMR: KGroups with Gini importance relevance estimator got the best configuration on 9 datasets. KGroups with mutual information estimator got the best configuration on 5 datasets. KGroups with F-value relevance estimator got the best configuration on 2 datasets. KGroups and mRMR tied on 2, 2, and 4 datasets, for Gini importance, mutual information, and F-value relevance estimator, respectively. Across the three relevance estimators, KGroups achieved best configuration on 6 datasets overall, tying with mRMR on 2 datasets (4 excluding ties). mRMR achieved best configuration on 7 datasets excluding ties. Both, KGroups and mRMR failed to achieved the best configuration on the TOX_171 dataset. Figures 2-4 below show the distribution of the number of features selected by the best configurations per datasets and per FS algorithms (and their respective variants). Overall, KGroups (and its variants) select fewer features than the two mRMR variants and KBest. This is particularly visible for Gini importance and F-value relevance estimators. As to the mutual information relevance estimator, some of KGroups variants 15 Under-review occasionally select more features than at least one mRMR variants or KBest. This is noticeable with the colon dataset on Fig. 3. However, there are no instances where the two mRMR variants and KBest select fewer features than all the variants of KGroups. It is worth mentioning that the distribution of the number of selected features by KGroups variants for F-value relevance estimator on the nci9 dataset is abnormal and will be discussed in section 7 below. Table 4: Best overall configurations of KBest and KGroups across classifiers. The highest accuracy per relevance estimator excluding ties (Win) is in bold. Tied highest accuracy scores (Draw) areunderlined. The star ( ∗ ) indicates best configuration per dataset. a indicates abnormally low accuracy. Gini importanceMutual info.F-value DatasetAll feat.KBest KGroupsKBest KGroupsKBest KGroups ALLAML95.91100.00*100.00*98.57 100.00*98.57 100.00* CLL_SUB_11173.8288.26*88.2284.6684.6671.1579.21 Carcinom95.4794.8296.54*95.3895.3992.5277.01 GLIOMA78.0098.00*96.0090.0094.0088.0090.00 GLI_8584.6295.2998.82*96.4798.82*95.2997.65 Prostate_GE89.6897.0096.1096.0597.05*95.1097.05* SMK_CAN_18770.1876.4978.61*76.4975.4375.3974.82 TOX_171 91.15*83.0984.2969.0373.6670.2076.00 colon84.7490.0091.7993.3395.00*90.1393.59 leukemia97.73100.00*100.00*100.00*100.00*100.00*100.00* lung94.1097.0498.01*97.5297.5494.6096.55 lung_discrete89.5593.1495.90*91.7190.4893.1494.38 lymphoma91.3895.84*95.84*95.7994.7987.5394.79 nci952.22 a 78.33*78.33*48.33 a 66.6778.33*23.33 a Win / Draw3 / 47 / 42 / 39 / 33 / 110 / 1 6.2 Robustness of Feature Extraction The FS task involves many choices that can each impact the resulting feature subset considerably as discussed in Guyon and Elisseeff[2]. One of those choices is the classifier used; another is the number featureskto be returned (this can be set manually or systematically given a range of possible values ofkas we did). We examined 6 popular classifiers, each trained independently and recorded the best configurations for KBest, mRMR and KGRoups using the three relevance estimators (Gini importance, mutual information and F-value). Calculating the average (and standard deviation) accuracy of the top configurations per classifier allows us to study the robustness of the feature selection algorithms. High average accuracy means the algorithm is reliable in returning feature subset with the highest accuracy independently of the classifier. Low standard deviation indicates high convergence across classifiers; another sign of reliability. Tables 6 and 7 highlight the results of the studies of the robustness of the three feature selection algorithms included in this paper. 16 Under-review Table 5: Best overall configurations of mRMR and KGroups across classifiers. The highest accuracy per relevance estimator excluding ties (Win) is in bold. Tied highest accuracy scores (Draw) areunderlined. The star ( ∗ ) indicates best configuration per dataset. a indicates abnormally low accuracy. Gini importanceMutual info.F-value DatasetAll feat.mRMR KGroupsmRMR KGroupsmRMR KGroups ALLAML95.91100.00*100.00*100.00*100.00*100.00*100.00* CLL_SUB_11173.8291.86*88.2290.0484.6685.5779.21 Carcinom95.4795.9896.5496.55*95.3995.9777.01 GLIOMA78.0094.0096.00*90.0094.0088.0090.00 GLI_8584.6296.4798.82*97.6598.82*97.6597.65 Prostate_GE89.6896.0596.1098.00*97.0597.0597.05 SMK_CAN_18770.1876.4978.6173.8175.4381.28*74.82 TOX_17191.15*85.9584.2977.7873.6680.1376.00 colon84.7490.0091.7993.4695.00*93.3393.59 leukemia97.73100.00*100.00*100.00*100.00*100.00*100.00* lung94.1097.0498.0198.02*97.5497.5296.55 lung_discrete89.5594.5795.90*91.8190.4894.5794.38 lymphoma91.3898.95*95.8496.8494.7997.8994.79 nci952.22 a 76.6778.3346.67 a 66.6785.00*23.33 a Win / Draw3 / 29 / 27 / 25 / 28 / 42 / 4 Fig. 2: Distribution of number of features selected by KBest, mRMR, and KGroups using Gini importance relevance estimator and KNeighbors, GaussianNB, Random- Forest, XGB, MLP, and LinearSVC classification models. 17 Under-review Fig. 3: Distribution of number of features selected by KBest, mRMR, and KGroups using mutual information relevance estimator and KNeighbors, GaussianNB, Ran- domForest, XGB, MLP, and LinearSVC classification models. Fig. 4: Distribution of number of features selected by KBest, mRMR, and KGroups using F-value relevance estimator and KNeighbors, GaussianNB, RandomForest, XGB, MLP, and LinearSVC classification models. 18 Under-review KGroups versus KBest: KGroups with Gini importance relevance estimator got the best configuration on 12 datasets. KGroups with mutual information estimator got the best configuration on 13 datasets. KGroups with F-value relevance estimator got the best configuration on 11 datasets. KGroups and KBest have not tied on any of the three relevance estimators. Across the three relevance estimators, KGroups achieved best configuration on 13 datasets (excluding ties). Both, KGroups and KBest failed to achieved the best configuration on the TOX_171 dataset. KGroups versus mRMR: KGroups with Gini importance relevance estimator got the best configuration on 8 datasets. KGroups with mutual information estimator got the best configuration on 7 datasets. KGroups with F-value relevance estimator got the best configuration on 5 datasets. KGroups and mRMR tied on 2, 0, and 0 datasets, for Gini importance, mutual information, and F-value relevance estimator, respectively. Across the three relevance estimators, KGroups achieved best configuration on 7 datasets overall, tying with mRMR on 2 datasets (5 excluding ties). mRMR achieved best configuration on 7 datasets excluding ties. Another thing noticeable when looking at the boxplots on Figs. 2-4 above is their difference in range (max− min, excluding outliers). In general, KGroups variants boxplots are narrower than KBest and the two variants of mRMR across the three relevance estimators, especially for Gini importance and F-value. In other words, the numbers of keys features selected by KGroups vary little from one classifier to another. This is a additional sign of robustness of the KGroups algorithm. 6.3 Execution Time In this section, we study the time complexity of the three feature selection algorithms during the feature extraction using their respective Central Processing Unit (CPU) execution time. CPU time is the total duration any of the CPU cores spends actively executing instructions for a specific task or program. This excludes system interruptions, time spent waiting for input or output operations, network requests, or the system entering a low-power or idle state. CPU time is equivalent for sequential, parallel or distributed tasks or programs. For our experiments, we recorded both CPU and Wall times, and they are equivalent. However, the CPU time has been chosen over Wall time to be reported below for the reasons mentioned above. Figure 5 and Table 8 below display the CPU time in seconds of three FS algorithms across the three relevance estimators. Time complexity of mRMR variants are equivalent so are KGroups variants, for a given relevance and redundancy estimators. The time complexity of KBest and KGroups variants is equivalent for different relevance estimators e.g. Gini importance, mutual information, and F-value (see Fig. 5 below). However, mRMR time complexity heavily depends on the time complexity of the redundancy estimator. That is why the time complexity of mRMR is the same for Gini importance and F-value but dramatically different for mutual information. In fact, time complexity of mRMR is on average 8.2 times slower for mutual information compared to Gini importance and F-value. These results aligned with the one reported by Zhao et al. [16] (see section 3.3). Table 8 shows the CPU execution time in seconds of the three algorithms per dataset and relevance estimator. Overall, KGroups is slightly faster that KBest and 19 Under-review Table 6: Average±standard deviation of the best configurations of KBest and KGroups (KG) across classifiers. The highest average accuracy per relevance estimator excluding ties (Win) is in bold. Tied highest average accuracy scores (Draw) areunderlined. The star ( ∗ ) indicates best configuration per dataset. a indicates abnormally low accuracy. Gini import.Mutual info.F-value DatasetAll feat. KBestKGKBestKGKBestKG ALLAML87.8898.3699.56*97.43 99.3396.98 98.33 ±7.31±1.37±1.09±2.21±1.12±2.03±1.41 CLL_SUB_11165.3985.2885.99*78.96 80.0065.62 73.36 ±6.38±2.50±1.68±4.41±2.64±4.58±4.19 Carcinom85.1386.8890.41*87.26 89.8281.93 69.05 ±7.27±9.89±6.06±10.71±6.61±14.66±10.44 GLIOMA62.6790.3390.67*85.67 87.6778.67 84.67 ±16.54±6.74±2.42±3.44±4.97±6.41±3.72 GLI_8577.6394.90 96.2794.9096.86*93.72 95.10 ±8.39±0.60±0.89±1.42±0.61±0.96±0.88 Prostate81.3495.23 94.6195.4195.72*94.60 95.42 ±11.16±1.29±1.03±0.79±0.81±0.51±0.80 SMK_CAN_18767.5274.5975.68*71.66 73.4374.83 72.80 ±2.57±1.44±0.80±3.08±1.25±0.67±1.33 TOX_171 78.94*77.50 76.8366.47 69.4166.49 71.21 ±7.46±3.91±2.56±2.70±2.10±2.62±2.85 colon77.1187.93 90.1191.2691.90*89.10 91.48 ±6.14±1.61±0.94±1.68±2.01±0.80±1.31 leukemia87.8899.5699.78*99.0899.78*98.8999.78* ±6.98±0.69±0.54±1.69±0.54±1.00±0.54 lung 84.0295.7296.47*94.98 95.9792.03 93.75 ±15.75±1.69±0.65±2.91±1.48±4.93±4.29 lung_discrete77.5888.3589.02*87.11 86.7987.52 87.70 ±15.61±7.97±5.92±6.74±4.39±6.02±7.31 lymphoma75.0686.70 89.1589.13 89.1582.9089.62* ±19.38±11.29±7.27±8.62±5.69±6.91±6.07 nci943.33 a 68.33 70.5641.11 a 57.5071.39*22.50 a ±8.80±9.25±6.97±8.21±7.66±5.91±1.39 Win/Draw2/0 12/01/0 13/03/0 11/0 20 Under-review Table 7: Average±standard deviation of the best configurations of mRMR and KGroups (KG) across classifiers. The highest average accuracy per relevance estimator excluding ties (Win) is in bold. Tied highest average accuracy scores (Draw) areunderlined. The star ( ∗ ) indicates best configuration per dataset. a indicates abnormally low accuracy. Gini import.Mutual info.F-value DatasetAll feat. mRMRKGmRMRKGmRMRKG ALLAML87.8898.1699.56*99.10 99.3398.13 98.33 ±7.31±1.45±1.09±0.70±1.12±2.11±1.41 CLL_SUB_11165.3988.12*85.9985.14 80.0081.05 73.36 ±6.38±2.75±1.68±3.35±2.64±5.67±4.19 Carcinom85.1390.41*90.41*90.22 89.8289.82 69.05 ±7.27±5.96±6.06±8.84±6.61±10.71±10.44 GLIOMA62.6788.0090.67*83.00 87.6783.33 84.67 ±16.54±5.66±2.42±7.77±4.97±4.32±3.72 GLI_8577.6394.70 96.2794.7196.86*95.69 95.10 ±8.39±2.07±0.89±2.44±0.61±1.92±0.88 Prostate81.3494.92 94.6195.74*95.7295.25 95.42 ±11.16±0.95±1.03±0.79±0.81±1.14±0.80 SMK_CAN_18767.5274.22 75.6869.96 73.4378.59*72.80 ±2.57±1.76±0.80±2.23±1.25±2.25±1.33 TOX_171 78.9480.81*76.8374.27 69.4175.68 71.21 ±7.46±4.00±2.56±2.77±2.10±2.38±2.85 colon77.1188.22 90.1191.8691.90*90.90 91.48 ±6.14±1.78±0.94±2.01±2.01±1.99±1.31 leukemia87.8899.78*99.78*99.1099.78*98.8999.78* ±6.98±0.54±0.54±1.65±0.54±1.00±0.54 lung 84.0295.5696.47*96.06 95.9795.80 93.75 ±15.75±1.76±0.65±1.68±1.48±2.15±4.29 lung_discrete77.5888.98 89.0287.89 86.7989.70*87.70 ±15.61±6.46±5.92±5.16±4.39±6.31±7.31 lymphoma75.0691.31*89.1590.69 89.1590.18 89.62 ±19.38±8.96±7.27±6.93±5.69±8.96±6.07 nci943.33 a 70.00 70.5640.00 a 57.5076.11*22.50 a ±8.80±10.06±6.97±7.82±7.66±8.41±1.39 Win/Draw4/28/27/07/09/05/0 21 Under-review significantly faster than mRMR. One can notice that the execution time per dataset of KGroups and KBest changes little across the three relevance estimator; confirming that the choice of the relevance estimator has limited effects on the time complexity of these two algorithms. However, the choice of the relevance and particularly of the redundancy estimators has a high impact on the time complexity of mRMR algorithm. Fig. 5: Boxplots of the execution time in seconds of KBest, mRMR, and KGroups. 7 KGroups Limitations and Potentials In this section, we are going to discuss the limitations and potentials of the KGroups algorithm. 7.1 KGroups Limitations Based on initial exploratory analysis of the distribution of the feature relevance values (independently of the relevance estimator), KGroups works better when the range is relatively large and the feature relevance values are fairly distributed across the range (not too many empty groups or clusters). For instance, the F-value relevance estimation of the nci9 dataset yielded only two non-empty clusters fork ∈[2,100]. As a result, KGroups manages to select no more than two features during the low-dimension feature extraction task (see Fig. 4) and performed poorly (see Tables 4-7). Further study will be required to determine the exact or a more accurate estimation of the limits. However, for now, we can state that KGroups has two main limitations: the range and distribution of the feature relevance values. The second limitation seems to be the most important. One way to address it is to simply change the relevance estimator. 22 Under-review Table 8: Execution time in seconds of the three FFS algorithms, KGroups (KG, on the left), KBest (KB, on the right), and mRMR (everything in between) per relevance estimator. The lowest execution time per relevance estimator is in bold. The last letter indicates the variant of mRMR: D for Difference and Q for Quotient. The other letters indicate the relevance and redundancy estimators e.g. RFCQ for RandomForest Pearson Correlation Quotient, MID for mutual information (for both relevance and redundancy) Difference, and FCQ for F-value Pearson Correlation Quotient. Gini import.Mutual info.F-value DatasetKG RFCQ KBKG MID MIQ KBKG FCD FCQ KB ALLAML180 5,952 186186 52,448 52,432 192184 5,878 5,864 176 CLL_SUB_111234 9,600 275253 94,996 94,983 271233 9,434 9,425 245 Carcinom552 8,307 917752 96,504 94,879 74628 8,004 7,998 660 GLIOMA196 3,757 188204 31,009 30,729 206189 3,695 3,691 193 GLI_85193 18,635 239204 168,077 167,806 237203 18,261 18,257 191 Prostate194 5,136 205202 49,347 48,072 209195 5,050 5,045 196 SMK_CAN_187259 17,323 378277 209,722 199,394 308273 16,945 17,019 260 TOX_171281 5,163 343309 58,694 56,592 320290 4,962 4,958 300 colon181 1,860 174190 14,564 14,644 185187 1,794 1,790 184 leukemia176 5,945 172178 59,081 55,889 183176 5,849 5,853 168 lung276 3,287 324311 36,602 37,106 307270 3,096 3,086 290 lung_disc208612 191216 2,505 2,828 205214 538 534 204 lymphoma231 3,517 221239 33,646 33,130 251235 3,424 3,436 226 nci9216 7,869 217232 81,428 76,530 274178 7,849 7,778 209 7.2 KGroups Potentials Despite the limitations discussed in the above section, KGroups has two major advan- tages: speed and parameterisation; this makes KGroups suitable for hyperparameter finetuning. From a limited brute force hyperparameter finetuning performed in this study, we demonstrated that KGroups predictive performance can be improved from that procedure. Therefore, KGroups potentials can be divided into two: predictive performance optimisation and research opportunities. Predictive performance optimisation. If predictive performance is the main concern, then one can do a more complete brute force hyperparameter finetuning for α∈]0; 3] with a step of 0.1 or 0.05, for example. One would have to train 30 and 60 variants of KGroups, respectively. Instead of a fixed step value of 0.1 or 0.05, one can use dynamic steps with larger values when close to 1 and smaller values when close to 0 or above 2. One can also finetune the bin size smoothing hyperparameter, a boolean variable. This boolean variable has two effects: (a) reduces the bins size of low relevance values for very smallαand increase the bins size of low relevance values for largeα(see Fig. 6) and (b) converts fixed step into dynamic step values. This will raise the number of variants of KGroups to 60 and 120, respectively. This whole process is still computationally cheaper than one variant of mRMR; as aforementioned one variants of mRMR is equivalent to 123 variants of KGroups in terms of time complexity. KGroups predictive performance can also be improve further by trying a handful of tie-breakers for additional computational cost. 23 Under-review Research opportunities. Instead of a brute force approach of hyperparameter finetuning, researchers can investigate more efficient ways to achieve competing predic- tive performance with less computation. For instance, the choice of the tie-breaker(s) and the value(s) ofαaccording to the distribution of the relevance estimator. Finding smarter and cheaper ways of making those two choices can make a huge difference. Figure 6 below shows the impact ofαand the smoothing variable on the bins size fork= 20. Mutual information relevance estimator on the ALLAML dataset is used as example. On the top row is displayed mutual information (α= 1, on the left) and cosine similarity (the tie-breaker, on the right) relevance values. One can notice that the distribution mutual information relevance values is right-skewed (but with no empty clusters); however, the distribution of the cosine similarity relevance values is normally distributed. Forα= 1, all the bins of the histogram have the same size. On the bottom row is shown the distribution of smoothed and non-smoothed bins forα= 0.5 andα= 1.5, respectively, from left to right. Forα <1, more features with high relevance values are selected. Inversely, more features with low relevance values are selected for α > 1. Fig. 6: Impact ofαand the smoothing variable on the bins size fork= 20; example of mutual information relevance estimator on the ALLAML dataset. 8 Conclusion In this paper, we proposed KGroups, a new versatile univariate Max-Relevance Min- Redundancy algorithm that can easily be applied to any type of relevance estimator. 24 Under-review KGroups is fast (up to 821 times faster that multivariate mRMR; 123 times faster on average) and parameterisable. This make it suitable for hyperparameters finetuning. Based on intensive and systematic experimentations, we demonstrated KGroups superiority compared to KBest. we also showed that a full hyperparameters finetuning of mRMR is equivalent to a limited brute force hyperparameters finetuning of KGroups in terms of predictive performance measured in average 5-fold cross validation accuracy. Next, we highlighted and discussed KGroups limitations and potentials. For future work, we aim to investigate ways to address KGroups limitations and harness its potentials. We also want to extend the experiments performed in this paper to additional datasets, especially to other types or modalities of data, such as images. Supplementary information. The entire project, from the code files to the raw and processed experimental results, is released and publicly available on GitHub at https://github.com/malick-jaures/KGroups_Bio.git for ease of reproduction and to stimulate further research in this research area. Acknowledgements. This research was conducted with the financial support of Taighde Eíreann – Research Ireland under Grant Agreement No. 13/RC/2106_P2 at the ADAPT Centre at University College Dublin. References [1] Theng, D., Bhoyar, K.K.: Feature selection techniques for machine learning: a survey of more than two decades of research. Knowledge and Information Systems 66(3), 1575–1637 (2024) https://doi.org/10.1007/s10115-023-02010-5 . Accessed 2024-07-30 [2]Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of machine learning research 3(Mar), 1157–1182 (2003). Accessed 2025-02-13 [3]Verdecchia, R., Cruz, L., Sallou, J., Lin, M., Wickenden, J., Hotellier, E.: Data- Centric Green AI An Exploratory Empirical Study. In: 2022 International Conference on ICT for Sustainability (ICT4S), p. 35–45 (2022). https://doi.org/ 10.1109/ICT4S55073.2022.00015 . https://ieeexplore.ieee.org/document/9830097/ Accessed 2025-02-10 [4]Omar, R., Muccini, H.: Energy Efficiency of AI-Powered Components: A Com- parative Study of Feature Selection Methods. In: 2024 10th International Conference on ICT for Sustainability (ICT4S), p. 99–108 (2024). https://doi.org/ 10.1109/ICT4S64576.2024.00019 . https://ieeexplore.ieee.org/document/10805260/ Accessed 2025-02-10 [5] Cottier, B., Rahman, R., Fattorini, L., Maslej, N., Owen, D.: The rising costs of training frontier AI models. arXiv (2024). https://doi.org/10.48550/arXiv.2405. 21015 . http://arxiv.org/abs/2405.21015 Accessed 2025-02-10 25 Under-review [6]Jiao, R., Nguyen, B.H., Xue, B., Zhang, M.: A Survey on Evolutionary Mul- tiobjective Feature Selection in Classification: Approaches, Applications, and Challenges. IEEE Transactions on Evolutionary Computation, 1–1 (2023) https: //doi.org/10.1109/TEVC.2023.3292527 . Accessed 2024-07-30 [7]Pedrycz, W.: Towards green machine learning: challenges, opportunities, and developments. Journal of Smart Environments and Green Computing 2(4), 163– 174 (2022) https://doi.org/10.20517/jsegc.2022.16 . Accessed 2025-02-15 [8]Tabbakh, A., Al Amin, L., Islam, M., Mahmud, G.M.I., Chowdhury, I.K., Mukta, M.S.H.: Towards sustainable AI: a comprehensive framework for Green AI. Dis- cover Sustainability 5(1), 408 (2024) https://doi.org/10.1007/s43621-024-00641-4 . Accessed 2025-02-15 [9] Bolón-Canedo, V., Morán-Fernández, L., Cancela, B., Alonso-Betanzos, A.: A review of green artificial intelligence: Towards a more sustainable future. Neuro- computing 599, 128096 (2024) https://doi.org/10.1016/j.neucom.2024.128096 . Accessed 2025-02-15 [10]Salehi, S., Schmeink, A.: Data-Centric Green Artificial Intelligence: A Survey. IEEE Transactions on Artificial Intelligence 5(5), 1973–1989 (2024) https://doi. org/10.1109/TAI.2023.3315272 . Accessed 2025-02-10 [11] Järvenpä, H., Lago, P., Bogner, J., Lewis, G., Muccini, H., Ozkaya, I.: A Synthesis of Green Architectural Tactics for ML-Enabled Systems. In: Proceed- ings of the 46th International Conference on Software Engineering: Software Engineering in Society. ICSE-SEIS’24, p. 130–141. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3639475.3640111 . https://dl.acm.org/doi/10.1145/3639475.3640111 Accessed 2025-02-10 [12]Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Aga, F., Huang, J., Bai, C., Gschwind, M., Gupta, A., Ott, M., Melnikov, A., Candido, S., Brooks, D., Chauhan, G., Lee, B., Lee, H.-H., Akyildiz, B., Balandat, M., Spisak, J., Jain, R., Rabbat, M., Hazelwood, K.: Sustainable AI: Environmental Implications, Challenges and Opportunities. Proceedings of Machine Learning and Systems 4, 795–813 (2022). Accessed 2025-02-10 [13]Buś, S., Jędrzejewski, K., Guzik, P.: Using Minimum Redundancy Maximum Relevance Algorithm to Select Minimal Sets of Heart Rate Variability Parame- ters for Atrial Fibrillation Detection. Journal of Clinical Medicine 11(14), 4004 (2022) https://doi.org/10.3390/jcm11144004 . Publisher: Multidisciplinary Digital Publishing Institute. Accessed 2025-08-19 [14]Doewes, A., Swasono, S.E., Harjito, B.: Feature selection on Human Activ- ity Recognition dataset using Minimum Redundancy Maximum Relevance. In: 2017 IEEE International Conference on Consumer Electronics - Taiwan (ICCE- TW), p. 171–172 (2017). https://doi.org/10.1109/ICCE-China.2017.7991050 . 26 Under-review https://ieeexplore.ieee.org/abstract/document/7991050 Accessed 2025-08-19 [15]Dubey, V.K., Saxena, A.K.: A Cosine-Similarity Mutual-Information Approach for Feature Selection on High Dimensional Datasets. J. Inf. Technol. Res. 10(1), 15–28 (2017) https://doi.org/10.4018/JITR.2017010102 . Accessed 2025-02-18 [16]Zhao, Z., Anand, R., Wang, M.: Maximum Relevance and Minimum Redun- dancy Feature Selection Methods for a Marketing Machine Learning Platform. In: 2019 IEEE International Conference on Data Science and Advanced Ana- lytics (DSAA), p. 442–452 (2019). https://doi.org/10.1109/DSAA.2019.00059 . https://ieeexplore.ieee.org/abstract/document/8964172 Accessed 2025-08-19 [17] Atef, M., Ouf, S., Seoud, W., Gabr, M.I.: A novel approach using explainable prediction of default risk in peer-to-peer lending based on machine learning models. Neural Computing and Applications (2025) https://doi.org/10.1007/ s00521-025-11489-8 . Accessed 2025-08-19 [18]Billah, M., Waheed, S.: Minimum redundancy maximum relevance (mRMR) based feature selection from endoscopic images for automatic gastrointestinal polyp detection. Multimedia Tools and Applications 79(33), 23633–23643 (2020) https://doi.org/10.1007/s11042-020-09151-7 . Accessed 2025-08-19 [19] Radovic, M., Ghalwash, M., Filipovic, N., Obradovic, Z.: Minimum redundancy maximum relevance feature selection approach for temporal gene expression data. BMC Bioinformatics 18(1), 9 (2017) https://doi.org/10.1186/s12859-016-1423-9 . Accessed 2025-08-19 [20]Ma, X.-H., Chen, Z.-G., Liu, J.-M.: Wavelength selection method for near-infrared spectroscopy based on Max-Relevance Min-Redundancy. Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy 310, 123933 (2024) https: //doi.org/10.1016/j.saa.2024.123933 . Accessed 2025-08-19 [21]Shirzad, M.B., Keyvanpour, M.R.: A feature selection method based on minimum redundancy maximum relevance for learning to rank. In: 2015 AI & Robotics (IRANOPEN), p. 1–5 (2015). https://doi.org/10.1109/RIOS.2015.7270735 . https://ieeexplore.ieee.org/abstract/document/7270735 Accessed 2025-08-19 [22]Huang, M., Sun, L., Xu, J., Zhang, S.: Multilabel Feature Selection Using Relief and Minimum Redundancy Maximum Relevance Based on Neighborhood Rough Sets. IEEE Access 8, 62011–62031 (2020) https://doi.org/10.1109/ACCESS.2020. 2982536 . Accessed 2025-08-19 [23] Cheng, X.: A Comprehensive Study of Feature Selection Techniques in Machine Learning Models. Social Science Research Network, Rochester, NY (2024). https://doi.org/10.2139/ssrn.5154947 . https://papers.ssrn.com/abstract= 5154947 Accessed 2025-08-29 27 Under-review [24]Liu, X., Xue, J.: Spatial Cluster Analysis by the Bin-Packing Problem and DNA Computing Technique. Discrete Dynamics in Nature and Soci- ety 2013(1), 891428 (2013) https://doi.org/10.1155/2013/891428 . _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1155/2013/891428. Accessed 2025-07- 29 [25]Muthukumaran, S., Mary, A.V.A., Valli, M.: A Novel Approach to Data Dis- cretization Using Clustering Techniques in Mining of High Dimensional Data (2016) [26]Sanders, J.S., Fabian, A.C.: Adaptive binning of X-ray galaxy cluster images. Monthly Notices of the Royal Astronomical Society 325(1), 178–186 (2001) https://doi.org/10.1046/j.1365-8711.2001.04410.x . Accessed 2025-07-29 [27] Zhang, Y., Wu, J., Cai, J.: Compact Representation for Image Classification: To Choose or to Compress? In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, p. 907–914. IEEE, Columbus, OH, USA (2014). https: //doi.org/10.1109/CVPR.2014.121 . https://ieeexplore.ieee.org/document/6909516 Accessed 2025-09-01 [28]Moslemi, A.: A tutorial-based survey on feature selection: Recent advancements on feature selection. Engineering Applications of Artificial Intelligence 126, 107136 (2023) https://doi.org/10.1016/j.engappai.2023.107136 . Accessed 2024-07-30 [29] Siegel, A.F.: Chapter 15 - ANOVA: Testing for Differences Among Many Samples and Much More. In: Siegel, A.F. (ed.) Practical Business Statistics (Seventh Edition), p. 469–492. Academic Press, ???(2016).https://doi.org/10.1016/B978-0-12-804250-2.00015-8. https://w.sciencedirect.com/science/article/pii/B9780128042502000158 Accessed 2025-05-13 [30]Battiti, R.: Using Mutual Information for Selecting Features in Supervised Neural Net Learning. Neural Networks, IEEE Transactions on 5, 537–550 (1994) https: //doi.org/10.1109/72.298224 [31]Ding, C., Peng, H.: MINIMUM REDUNDANCY FEATURE SELECTION FROM MICROARRAY GENE EXPRESSION DATA. Journal of Bioinformat- ics and Computational Biology 03(02), 185–205 (2005) https://doi.org/10.1142/ S0219720005001004 . Accessed 2025-09-05 [32]Li, C., Luo, X., Qi, Y., Gao, Z., Lin, X.: A new feature selection algorithm based on relevance, redundancy and complementarity. Computers in Biology and Medicine 119, 103667 (2020) https://doi.org/10.1016/j.compbiomed.2020.103667 . Accessed 2025-08-19 [33] Ihianle, I.K., Machado, P., Owa, K., Adama, D.A., Otuka, R., Lotfi, A.: Minimising redundancy, maximising relevance: HRV feature selection for stress classification. 28 Under-review Expert Systems with Applications 239, 122490 (2024) https://doi.org/10.1016/j. eswa.2023.122490 . Accessed 2025-08-19 [34]Bommert, A., Welchowski, T., Schmid, M., Rahnenführer, J.: Benchmark of filter methods for feature selection in high-dimensional gene expression survival data. Briefings in Bioinformatics 23(1), 354 (2022) https://doi.org/10.1093/bib/bbab354 . Accessed 2025-07-16 [35]Bugata, P., Drotar, P.: On some aspects of minimum redundancy maximum relevance feature selection. Science China Information Sciences 63(1), 112103 (2019) https://doi.org/10.1007/s11432-019-2633-y . Accessed 2025-08-19 [36] Jo, I., Lee, S., Oh, S.: Improved Measures of Redundancy and Relevance for mRMR Feature Selection. Computers 8(2), 42 (2019) https://doi.org/10.3390/ computers8020042 . Publisher: Multidisciplinary Digital Publishing Institute. Accessed 2025-08-19 [37]Che, J., Yang, Y., Li, L., Bai, X., Zhang, S., Deng, C.: Maximum relevance minimum common redundancy feature selection for nonlinear data. Informa- tion Sciences 409-410, 68–86 (2017) https://doi.org/10.1016/j.ins.2017.05.013 . Accessed 2025-08-19 [38] Senawi, A., Wei, H.-L., Billings, S.A.: A new maximum relevance-minimum multicollinearity (MRmMC) method for feature selection and ranking. Pat- tern Recognition 67, 47–61 (2017) https://doi.org/10.1016/j.patcog.2017.01.026 . Accessed 2025-09-08 [39]Mandal, M., Mukhopadhyay, A.: An Improved Minimum Redundancy Maximum Relevance Approach for Feature Selection in Gene Expression Data. Procedia Tech- nology 10, 20–27 (2013) https://doi.org/10.1016/j.protcy.2013.12.332 . Accessed 2025-08-19 [40] Yang, J., Zhu, Z., He, S., Ji, Z.: Minimal-redundancy-maximal-relevance feature selection using different relevance measures for omics data classification. In: 2013 IEEE Symposium on Computational Intelligence in Bioinformatics and Compu- tational Biology (CIBCB), p. 246–251 (2013). https://doi.org/10.1109/CIBCB. 2013.6595417 . https://ieeexplore.ieee.org/abstract/document/6595417 Accessed 2025-08-19 [41]Lin, D., Tang, X.: Conditional Infomax Learning: An Integrated Framework for Feature Extraction and Fusion. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) Computer Vision – ECCV 2006, p. 68–82. Springer, Berlin, Heidelberg (2006). https://doi.org/10.1007/11744023_6 [42] Kwak, N., Choi, C.-H.: Input feature selection for classification problems. IEEE Transactions on Neural Networks 13(1), 143–159 (2002) https://doi.org/10.1109/ 72.977291 . Accessed 2025-09-08 29 Under-review [43]Yuan, G., Li, X., Qiu, P., Zhou, X.: Feature selection method based on wavelet similarity combined with maximum information coefficient. Information Sciences 699, 121801 (2025) https://doi.org/10.1016/j.ins.2024.121801 . Accessed 2025-05- 09 [44]Yuan, G., Zhai, Y., Tang, J., Zhou, X.: CSCIM_fs: Cosine similarity coefficient and information measurement criterion-based feature selection method for high- dimensional data. Neurocomputing 552, 126564 (2023) https://doi.org/10.1016/j. neucom.2023.126564 . Accessed 2025-02-18 [45]Lopez-Paz, D., Hennig, P., Schölkopf, B.: The Randomized Dependence Coefficient. In: Advances in Neural Information Pro- cessing Systems, vol. 26. Curran Associates, Inc., ??? (2013). https://proceedings.neurips.c/paper/2013/hash/aab3238922bcc25a6f606eb525ffdc56- Abstract.html Accessed 2025-09-08 [46] Székely, G.J., Rizzo, M.L.: Brownian distance covariance. The Annals of Applied Statistics 3(4), 1236–1265 (2009) https://doi.org/10.1214/09-AOAS312 . Publisher: Institute of Mathematical Statistics. Accessed 2025-10-09 [47]Zhang, P., Gao, W., Hu, J., Li, Y.: A conditional-weight joint relevance metric for feature relevancy term. Engineering Applications of Artificial Intelligence 106, 104481 (2021) https://doi.org/10.1016/j.engappai.2021.104481 . Accessed 2025-04-28 [48] Zhang, P., Gao, W.: Feature selection considering Uncertainty Change Ratio of the class label. Applied Soft Computing 95, 106537 (2020) https://doi.org/10. 1016/j.asoc.2020.106537 . Accessed 2025-09-08 [49]Wang, Y., Li, X., Ruiz, R.: Feature Selection With Maximal Relevance and Minimal Supervised Redundancy. IEEE Transactions on Cybernetics 53(2), 707–717 (2023) https://doi.org/10.1109/TCYB.2021.3139898 . Accessed 2025-08-28 [50]Ramírez-Gallego, S., Lastra, I., Martínez-Rego, D., Bolón-Canedo, V., Benítez, J.M., Herrera, F., Alonso-Betanzos, A.: Fast-mRMR: Fast Minimum Redundancy Maximum Relevance Algorithm for High-Dimensional Big Data. International Journal of Intelligent Systems 32(2), 134–152 (2017) https://doi.org/10.1002/ int.21833 . _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/int.21833. Accessed 2025-09-01 [51]Berrendero, J.R., Cuevas, A., Torrecilla, J.L.: The mRMR variable selec- tion method: a comparative study for functional data. Journal of Statistical Computation and Simulation (2016). Publisher: Taylor & Francis. Accessed 2025-08-19 [52] Ebiele, M., Bendechache, M., Ward, M., Geary, U., Byrne, D., Creagh, D., Brennan, R.: The Impact of Data Valuation on Feature Importance in Classification Models. 30 Under-review In: Fortino, G., Kumar, A., Swaroop, A., Shukla, P. (eds.) Proceedings of Third International Conference on Computing and Communication Networks vol. 917, p. 601–617. Springer, Singapore (2024). https://doi.org/10.1007/978-981-97-0892-5_ 47 . https://link.springer.com/10.1007/978-981-97-0892-5_47 Accessed 2025-05-08 [53]Galli, S.: Feature-engine: A Python package for feature engineering for machine learning. Journal of Open Source Software 6(65), 3642 (2021) https://doi.org/10. 21105/joss.03642 . Accessed 2025-10-15 [54]Chen, T., Guestrin, C.: XGBoost: A Scalable Tree Boosting System. In: Proceed- ings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery And Data Mining. KDD ’16, p. 785–794. ACM, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939785 . event-place: San Francisco, California, USA. http://doi.acm.org/10.1145/2939672.2939785 [55] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011) [56]Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R.P., Tang, J., Liu, H.: Feature Selection: A Data Perspective. ACM Comput. Surv. 50(6), 94–19445 (2017) https://doi.org/10.1145/3136625 . Accessed 2025-05-07 [57] Grinsztajn, L., Oyallon, E., Varoquaux, G.: Why do tree-based models still outperform deep learning on typical tabular data? In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22, p. 507–520. Curran Associates Inc., Red Hook, NY, USA (2022) 31