Paper deep dive
Neural Network Conversion of Machine Learning Pipelines
Man-Ling Sung, Jan Silovsky, Man-Hung Siu, Herbert Gish, Chinnu Pittapally
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/27/2026, 1:38:55 AM
Summary
This paper explores the conversion of non-neural machine learning pipelines, specifically those using random forest classifiers, into neural network (NN) student models via knowledge distillation. The authors evaluate this approach on 100 OpenML tasks, demonstrating that student NNs can effectively mimic teacher random forests, and investigate methods for automatic hyper-parameter selection for the student models.
Entities (5)
Relation Signals (3)
Knowledge Distillation → usedtotrain → Neural Network
confidence 95% · The goal of knowledge distillation is to transfer knowledge acquired by teacher to a student
Neural Network → mimics → Random Forest
confidence 90% · Our results show that for the majority of the tasks, the student NN can indeed mimic the teacher
OpenML → providesdatafor → Machine Learning Model
confidence 90% · OpenML (Open Machine Learning), founded by Vanschoren et al. (2014), is a platform for sharing datasets, ready-to-use models, and problems in machine learning.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Transfer learning and knowledge distillation has recently gained a lot of attention in the deep learning community. One transfer approach, the student-teacher learning, has been shown to successfully create ``small'' student neural networks that mimic the performance of a much bigger and more complex ``teacher'' networks. In this paper, we investigate an extension to this approach and transfer from a non-neural-based machine learning pipeline as teacher to a neural network (NN) student, which would allow for joint optimization of the various pipeline components and a single unified inference engine for multiple ML tasks. In particular, we explore replacing the random forest classifier by transfer learning to a student NN. We experimented with various NN topologies on 100 OpenML tasks in which random forest has been one of the best solutions. Our results show that for the majority of the tasks, the student NN can indeed mimic the teacher if one can select the right NN hyper-parameters. We also investigated the use of random forest for selecting the right NN hyper-parameters.
Tags
Links
- Source: https://arxiv.org/abs/2603.25699v1
- Canonical: https://arxiv.org/abs/2603.25699v1
Trouble viewing inline? Open PDF directly →
Full Text
19,300 characters extracted from source content.
Expand or collapse full text
JMLR: Workshop and Conference Proceedings 1:1–7, 2018ICML 2018 AutoML Workshop Neural Network Conversion of Machine Learning Pipelines Man-Ling Sungsammi.sung@raytheon.com Jan Silovskyjan.silovsky@raytheon.com Man-hung Siuman-hung.siu@raytheon.com Herbert Gishherb.gish@raytheon.com Chinnu Pittapallychinnu.pittapally@raytheon.com Raytheon BBN Technologies, 10 Moulton Street, Cambridge, MA 02138 Abstract Transfer learning and knowledge distillation has recently gained a lot of attention in the deep learning community. One transfer approach, the student-teacher learning, has been shown to successfully create “small” student neural networks that mimic the performance of a much bigger and more complex “teacher” networks. In this paper, we investigate an extension to this approach and transfer from a non-neural-based machine learning pipeline as teacher to a neural network (N) student, which would allow for joint optimization of the various pipeline components and a single unified inference engine for multiple ML tasks. In particular, we explore replacing the random forest classifier by transfer learning to a student N. We experimented with various N topologies on 100 OpenML tasks in which random forest has been one of the best solutions. Our results show that for the majority of the tasks, the student N can indeed mimic the teacher if one can select the right N hyper-parameters. We also investigated the use of random forest for selecting the right N hyper-parameters. Keywords: transfer learning, student-teacher learning, neural network 1. Introduction Our goal is to explore the use of neural networks as replacements for ML pipeline, or a por- tion of these pipeline. We will accomplish this replacement by having the neural network (N) learning from the original pipeline. This goes beyond the more usual motivation in employing the student teacher paradigm, which goes from large N to smaller, more de- ployable ones, e.g., Ba and Caruana (2014); Hinton et al. (2015). While smaller and more deployable are welcome characteristics, we have additional considerations. One considera- tion is that our converted components may be part of a larger network, and chaining various converted components to form a larger neural network will simplify the joint optimization of all parts of our system. Mapping various systems into neural networks can have several additional benefits. Specialized hardware, such as GPUs can enhance performance and a neural network may have better generalization performance than the original systems. Moreover, we expect our systems to operate in dynamic environments and having a unified approach to these changes can enhance the capabilities of these more deployable systems. In particular we can use standard methods for regularizing these networks, which ties in with generalization capabilities as well as methods for adapting the networks to changing conditions. In this work, our focus is on converting other classifiers into N classifiers. We under- stand that a neural network is not necessarily the best classifier in all situations, especially © 2018 M.-L. Sung, J. Silovsky, M.-h. Siu, H. Gish & C. Pittapally. arXiv:2603.25699v1 [cs.LG] 26 Mar 2026 Sung Silovsky Siu Gish Pittapally in small data problems. However, our goal is not to exceed the performance of the teacher but rather attempt to match it. In some situations it may be necessary to rely mainly on the function approximation capabilities of neural networks and at other times we may need to train the neural network using methods of data augmentation. By data augmentation we envision modeling of the feature space based on the original training data and generating new samples based on this model. The samples, coupled with the labels provided by the teacher provide additional training for the neural network. In some cases, the teacher brings more to the knowledge transfer problem than just generating classification responses. In some particular cases, we can have knowledge of the decision boundary in some form as well as metadata regarding the structure of the particular classifier. For example, for random forest classifiers it has been shown (Sethi (1990),Biau et al. (2016)) that they can be restructured as multi-layered neural networks. Additionally, Wang et al. (2017) shows how to create random forest inspired neural network architectures. In our current work, these characteristics are not exploited. In the following, we discuss in greater detail the student-teacher approach, which for us is conversion to a neural network, followed by a discussion of our experimental results. 2. Neural Network Conversion 2.1. Student-Teacher Knowledge Distillation Learning The goal of knowledge distillation is to transfer knowledge acquired by ”teacher” to a ”student” such that the student can perform as well or better than the teacher. Typically, the teacher is a complex system either with large number of parameters and/or an ensemble of classifiers while the student is relatively smaller to enable efficient inference, e.g. in Hinton et al. (2015). For Inductive Transfer Learning defined in Pan and Yang (2010), consider training data T =(x 1 , y 1 ), . . . , (x n , y n ) where x’s are the input features with corresponding labels y’s. The teacher model M is trained using T . M can generate the labels of a new training set T ′ =(x ′ 1 , ˆy 1 ), . . . , (x ′ n , ˆy n ) whereˆy is the set of label posteriors generated by M .x ′ in T ′ andx in T can be different. In Li et al. (2014),x ′ includes additional unlabeled data and in Cui et al. (2017),x ′ andx are generated by different feature extractors. 2.2. Student-Teacher over Different System Types In typical distillation framework, both teacher and student are neural networks, or as in Tyukina et al. (2017), transfer across very specific systems is possible by tapping into the internal states of the teacher systems. However, the student-teacher formulation in Sec- tion 2.1 can be generalized to distill between two different system types with the following considerations. 1. Trainability: The student system can be trained using label posteriors ˆy. 2. Feature Handling: The student system can process the type of input feature x. 3. Student Complexity: The student system should have enough capacity to learn the decision boundaries of the teacher system. For example, using a linear classifier as a 2 Neural Network Conversion of Machine Learning Pipelines student will not be able to mimic the classification decision of a deep neural network classifier. Other than the particular type of student system (i.e. neural networks or random forests), the hyper-parameters of the student and the amount of available training can have signifi- cant impact on distillation effectiveness. 2.3. Training Data for Student The quality of the transfer depends heavily on the amount of training data available and the complexity of the student model. As noted above, the student can be trained with a different data set from those used for training the teacher. While it can be difficult or expensive to obtain manually annotated data, the annotation needed for student training, ˆy, can easily be generated using the teacher model. Thus, extending T ′ only involves obtaining more x ′ . This can be accomplished by 1) collecting more unlabeled data which is feasible for many problems; 2) Using T to estimate the input feature distribution, P (x), and then sampling from it. Where P (x) can be estimated using either parametric models, such as GMM, or non-parametric models, such as KNN or any kernel-based distribution estimators; 3) Assuming P (x) to be a uniform distribution and sampling from it. Such P (x) can be suboptimal as discussed in Scholkopf et al. (2012) but can be useful as a smoothing function. 2.4. Initial Approach In this paper we focus on a set of random forest teachers and our ability to match the random forest performance with N classifiers. We selected random forest classifiers based on their reputation for providing the best performance on a wide range of problems and also their widespread use. We perform this exploration on a standard set of problems provided by OpenML. In addition to student-teacher performance comparisons, we also investigate ways to determine the best choice of neural network architecture and hyper-parameters to employ on particular problems. 3. Experiments 3.1. OpenML OpenML (Open Machine Learning), founded by Vanschoren et al. (2014), is a platform for sharing datasets, ready-to-use models, and problems in machine learning. It provides cross-language APIs that facilitates the reproduction and comparison of different machine learning architectures. There are 4 main organization groups, 1. Data: collection of data sets available for definition of ML problems; 2. Task: a formulation of a ML problem and specification of evaluation criteria; 3. Flow: describes a particular solution as a composition of primitives/modules performing various tasks - e.g. feature extraction, normalization, classification, etc. 4. Run: describes particular configuration of a flow, most importantly, hyperparameters of individual primitives. Hence, multiple Runs can be associated with identical Flow and yield different performance. 3 Sung Silovsky Siu Gish Pittapally 3.2. Experimental setup First, we identified a Flow, Olson (2017), employing random forest as the backend clas- sifier, which was evaluated for many tasks. The flow we found was composed of three sklearn primitives: preprocessing.imputation.Imputer, decomposition.pca.PCA and ensem- ble.forest.RandomForestClassifier. Next, we selected 100 Tasks based on best Runs. A Student system was built simply by substituting the random forest (RF) classifier by Multi-Layer Perceptron (MLP) 1 . For each Task, we used identical set of 600 different configurations of MLPs acting as different Students. Table 1 tabulates the parameters modified in our configurations. For parameters not listed, sklearn’s MLP defaults are used. By bottleneck, we refer to the middle layer in systems having 3 or more layers and the relative size presented in Table 1 is relative to the standard layer size in the networks 2 . LayersNodes in layerRel. bottleneck sizeActivationInit. learning rate 1,2,3,4,510,25,100,200,4000.2, 0.5, 1.0relu, tanh1e-2,1e-3,1e-4,1e-5 Table 1: Overview of different configurations of MLP Students 3.3. Student-Teacher Knowledge Transfer In this study, we focused on knowledge transfer using the original training inputs. Thus, the Student model M ′ is trained with training data T ′ =(x 1 , ˆy 1 ), . . . , (x n , ˆy n ), where ˆy = M (x). The OpenML experiments are designed as 10-fold cross-validation and we followed this experimental setup. This means that for each task, 10 different RF Teachers were trained and the knowledge transfer applied independently for 10 MLP Students with a particular configuration (one of the 600). The final task accuracy is then simply an average over the 10 folds. Fig. 1 illustrates the performance difference of the random forest Teachers and the MLP Students. The best performing MLP configuration is considered for each task. Over all tasks, 55% of Students perform equally well or better than Teacher. On average, the performance of Students is worse by 2.66%. In terms of the median, the Students perform as well as the Teachers (0.01% better). The shift between the average and median is caused by few outliers as shown in the right side of the figure. We plan to further investigate why MLP performs so poorly on the few outliers. For some tasks, the Student surprisingly outperforms the Teacher by a larger margin. We attribute this partly to natural statistical variations, and partly to the fact that RF par- titions the feature space in rectangular regions while MLP has smoother decision boundary which may fit certain problems better. 1. We relied on sklearn’s implementation of MLP classifier Pedregosa et al. (2011) 2. A N with 3 layers, 100 nodes per layer and relative bottleneck size of 0.5, has (100,50,100) nodes in its hidden layers 4 Neural Network Conversion of Machine Learning Pipelines Figure 1: Histogram on differences between RF and MLP accuracies on 100 tasks 3.4. Student Versatility and Complementarity Having large number of MLP student configurations (hereafter, we refer to these student configurations as Students) is impractical and we expect many to have similar performance across Tasks. It is desirable to keep only a smaller set of complementary Students, i.e. student configurations with high performance across many Tasks. Fig. 2 depicts how varying the number of Student candidates affect the performance across Tasks. Candidate sets of each size were formed by removing the systems with least contribution to the overall performance. We found that the single best system 3 turns out to be very versatile across Tasks as it performs only 0.9% worse on average compared to the choice of the best Student out of the full inventory of 600 Students. However, as shown in the figure, picking from 20 Students reduces the gap by half to 0.45%. 3.5. Automatic Student Selection While we can rely on cross-validation experiments to select the best Student, it may still not be feasible to train multiple Students in some practical applications. Ideally we would be able to automatically select the best Student candidate based on characteristics of the Data, Task and the Teacher. We carried out a set of experiments using random forest for selecting the best student candidate. Intuitively, the complexity of selecting the best Student grows with the number of Students candidates and the complexity is further accentuated by the limited number of training samples (100 samples corresponding to the 100 Tasks). The RF system for automatic Student selection was trained with a 10-fold cross-validation over the Tasks. As input features to this system, we used metadata characterizing the datasets as pro- vided by OpenML (Ope (2018)). We excluded features corresponding to performance of other reference classifiers, e.g. nearest neighbor. As a result, our input feature vectors were formed by 74 coefficients reflecting various dataset qualities and quantities. Fig. 2 shows the comparison of choice of the best candidate from the set of a particular size with the automatic choice done by the random forest. We conclude that the automatic Student selection fails to select the best Student candidates. Our reasoning is that the metadata provided by OpenML for dataset characterization are not suitable for automatic system selection and the performance is also affected by the small number of samples available. 3. A DNN with two hidden layers (400,400), relu activation function and initial learning rate of 1e-2 5 Sung Silovsky Siu Gish Pittapally Figure 2: All accuracies are computed using cross-validation. The top line ”MLP” is ob- tained by selecting the best (out of 600) Student per task. The ”SubsetMLP” and ”AutoRF” curves show the performance of selecting from a subset of size N. 4. Conclusions and Future Work There are multiple benefits in being able to represent machine learning pipelines for various datasets and tasks in a unified framework based on neural networks. In this work, we first laid out a solution for conversion of generic machine learning pipelines into neural networks. We view the conversion as a multi-stage process where parts of the original pipeline are first converted separately before joint optimization can be done. We then focused on the conversion of the back-end classifier represented by random forest into a N. We showed that NNs learned employing the student-teacher concept performed generally as well as the original random forests, with a few outliers. While NNs with many different configurations were initially considered, we showed that the number of N configurations can be significantly reduced without harming the performance. Finally, we investigated the possibility of using a random forest for automatic selection of the best N configuration based on the characteristics of the data. In contrast to using a single best configuration, this automatic selection leads to only a marginal improvement for very small sets of Students and the performance deteriorates as the number of Students grows. We attribute this mainly to the lack of relevant information in the metadata which is used as input to the automatic selection system, and the lack of training samples. Experimental work presented in this paper represents just an initial step in our effort and many aspects of our proposed solution will have to be further investigated in the future, such as substitution of various parts of generic ML pipelines (including feature extraction or transformation), augmentation of training data, end-to-end joint optimization and automatic selection of the best N configuration for substitution. Acknowledgement This work is sponsored by the Air Force Research Laboratory (AFRL) and DARPA. 6 Neural Network Conversion of Machine Learning Pipelines References List of openml data attributes and measure, 2018. URL https://w.openml.org/search? type=measure. Accessed: 2018-05-21. Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pages 2654–2662, 2014. G. Biau, E. Scornet, and J. Welbl.Neural random forests.In arXiv preprint arXiv:1604.07143, 2016. Jia Cui, Brian Kingsbury, Bhuvana Ramabhadran, George Saon, Tom Sercu, Kartik Au- dhkhasi, Abhinav Sethyand Markus Nussbaum-Thom, and Andrew Rosenberg. Knowl- edge distillation across ensembles of multilingual models for low-resource languages. In ICASSP, 2017. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In arXiv, 2015. Jinyu Li, Rui Zhao, Jui-Ting Huang, and Yifan Gong. Learning small-size dnn with output- distribution-based criteria. In Interspeech, 2014. Randal Olson. Openml flow id: 5909, 2017. URL https://w.openml.org/f/5909. Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2010. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blon- del, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. Bernhard Scholkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In ICML, 2012. I. K. Sethi. Entropy nets: from decision trees to neural networks. Proceedings of the IEEE, 78(10):1605–1613, 1990. Ivan Yu Tyukina, Alexander N. Gorbana, Konstantin I. Sofeikova, and Ilya Romanenko. Knowledge transfer between artificial intelligence systems. In arXiv, 2017. Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49–60, 2014. Suhang Wang, Charu Aggarawal, and Huan Liu. Using a random forest to inspire a neural network and improving on it. In SIAM International Conference on Data Mining, 2017. 7