Paper deep dive

Cohort Organized Learning: Clustering Through Agreement

Finn Henry O'Shea, Maria Elena Monzani

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 83

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 87%

Last extracted: 7/9/2026, 6:35:22 AM

Summary

The paper introduces Cohort Organized Learning (CoOL), an unsupervised clustering method that partitions unlabeled data through mutual agreement among a cohort of differentiable neural network observers. Unlike traditional algorithms, CoOL avoids explicit distance or similarity matrix computations, instead leveraging Expectation Maximization (EM) to reconcile cluster labels across observers. Training minimizes cross-entropy loss combined with determinant-based regularization to prevent degenerate solutions. The framework is demonstrated on synthetic data and the MNIST dataset, highlighting its applicability to high-dimensional data without predefined similarity metrics.

Entities (10)

Cohort Organized Learning · method · 95%Differentiable Observers · model-component · 90%Expectation-Maximization · algorithm · 90%SLAC National Accelerator Laboratory · organization · 90%Stanford University · organization · 90%Agreement Rate · metric · 85%Determinant Regularization · technique · 85%Reliability Matrix · mathematical-concept · 85%Cross-Entropy Loss · loss-function · 80%MNIST Dataset · dataset · 80%

Relation Signals (10)

Maria Elena Monzani → affiliatedwith → Stanford University

confidence 90% · Maria Elena Monzani1,2 ... 2Kavli Institute for Particle Astrophysics and Cosmology, Stanford University

Finn H. O’Shea → affiliatedwith → SLAC National Accelerator Laboratory

confidence 90% · Finn H. O’Shea1,∗ ... 1SLAC National Accelerator Laboratory

Cohort Organized Learning → employs → Differentiable Observers

confidence 90% · CoOL functions by having a user-defined number of observers estimate the cluster identity of the samples in collaboration with each other.

Cohort Organized Learning → uses → Expectation-Maximization

confidence 90% · CoOL modifies this approach, it uses EM and differentiable observers capable of learning to force the observers to agree on the cluster identity of each of the samples.

Cohort Organized Learning → applies → Determinant Regularization

confidence 85% · To regularize against this, we add a term to the loss proportional to the absolute value of the determinant of each of the observers.

Cohort Organized Learning → avoids → explicit distance/similarity computations

confidence 85% · a method for partitioning data that does not rely on distance or similarity matrix computations between samples in the objective function, but rather uses agreement.

Cohort Organized Learning → monitors → Agreement Rate

confidence 85% · During training, we monitor two quantities: the agreement rate and the number of unique labels produced by CoOL.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:In this article we describe Cohort Organized Learning (CoOL), a method for clustering data without explicit distance or similarity computations. Herein, we will describe CoOL, derive the gradients determined by expectation maximization to train the networks, show how to monitor convergence during training and evaluate the clusters after training, and discuss a series of examples and use cases. We also discuss CoOL's limitations and future prospects on related tasks. Because CoOL uses neural networks to estimate the clusters, it can be used to cluster any data that can be made compatible and we illustrate this on vector data and images.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

82,889 characters extracted from source content.

Expand or collapse full text

Cohort Organized Learning: Clustering Through Agreement Finn H. O’Shea1,∗ 0000-0003-2398-7381 Maria Elena Monzani1,2 0000-0002-8254-5308 1SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, CA 94025-7015, USA 2Kavli Institute for Particle Astrophysics and Cosmology, Stanford University, Stanford, CA 94305-4085 USA ∗Author to whom any correspondence should be addressed. foshea@slac.stanford.edu Abstract In this article we describe Cohort Organized Learning (CoOL), a method for clustering data without explicit distance or similarity computations. Herein, we will describe CoOL, derive the gradients determined by expectation maximization to train the networks, show how to monitor convergence during training and evaluate the clusters after training, and discuss a series of examples and use cases. We also discuss CoOL’s limitations and future prospects on related tasks. Because CoOL uses neural networks to estimate the clusters, it can be used to cluster any data that can be made compatible and we illustrate this on vector data and images. keywords: deep learning, clustering, expectation maximization †articletype: Paper 1 Introduction The modern scientific process produces enormous amounts of data, and most of that data is unlabeled because there is not enough human-hours to sort through it, to say nothing of the cost. One of the contributors to this prodigious data production is the high dimensionality (i.e. number) of signals recorded during routine operation. Examples of high-dimensional tools in physics include tokamaks [38, 2, 7], dark matter detectors [1, 5], particle accelerators [51, 46], telescopes [27, 22], and collider detectors [52, 6]. These tools, and many more, each have a trove of potentially useful information that is too costly to organize using conventional methods. In this article, we resort to clustering, which can be used to partition unlabeled data or to better understand its structure [29]. As a fundamental strategy for organizing data, it is used in practically every field that produces data [28, 44, 64]. All clustering algorithms, deep-learning included, rely on some measure of distance (or similarity) between the samples111See Section 3 for examples of how distance and similarity are used in deep clustering.. Herein, we add another strategy by introducing Cohort Organized Learning (CoOL), a method for partitioning data that does not rely on distance or similarity matrix computations between samples in the objective function, but rather uses agreement. Several differentiable observers (neural networks in our case) use expectation maximization (EM) [13] to reconcile their cluster labels for the samples and learn from each other to partition the samples as a cohort. Herein, we will describe CoOL, derive the gradients determined by EM to train the observers, show how to monitor convergence during training without labels, describe how to evaluate the success of clustering, and discuss a series of examples using the MNIST handwritten digits dataset [33]. We are not interested in images per se or learning representations therefrom, so we do not spend any effort attempting to surpass state of the art performance on these datasets. Along the way, we will discuss some of CoOL’s limitations, as we understand them, and how variations on CoOL might be used to solve other problems. 2 Description of CoOL CoOL functions by having a user-defined number of observers estimate the cluster identity of the samples in collaboration with each other. The mathematical basis for this collaboration is the expectation maximization (EM) [13] work of Dawid and Skene [12]. In that work, the authors find estimates for the unknown class identity of samples by reconciling the (fixed) labels created by a number of experts. In doing so, they find estimates for the proportion of samples that belong to each class and the “error-rate matrix” (what we will call the reliability matrix) for each observer. CoOL modifies this approach, it uses EM and differentiable observers capable of learning to force the observers to agree on the cluster identity of each of the samples. In other words, the different observers learn to collaboratively cluster the samples as a cohort. The procedure is straightforward, a diagram is given in Figure 1. First, the user defines the differentiable observers, the number of clusters those observers should use, and hyperparameters such as learning rate and weight decay 222We do not use weight decay in the work presented here.. Second, the observers are shown the samples (one epoch is showing all the samples to the observers) and each observer produces a label for each sample. Third, those labels are used to perform one or more steps of the EM-algorithm. Finally, the results of the EM-algorithm are used to compute a loss for back propagation. Figure 1: Diagram of data flow through CoOL. The solid lines show relationships that allow back propagation. For our example here, we use the scikit-learn function make_classification to make a 2-dimensional, 3-class, 64-sample clustering problem. The classes are approximately balanced with a single cluster per class. The feature space is shown in Figure 2. We use 5 neural networks, each with a single hidden layer of 50 units with leaky ReLU activation [62]. All layers are fully-connected. The output of the networks is a softmax function with three outputs. Despite the low dimensionality of the example, we do not attempt to demonstrate superlative performance compared to, say, classical algorithms like DBSCAN [45], OPTICS [4], or spectral clustering [59]. Rather, we aim to describe CoOL and how to use it, and in doing so we can introduce some definitions and concepts useful to understanding how it functions. In this example we do not standardize the features of the input data. This is because CoOL uses the structure of the data to find a partition of it and changing that structure by scaling the different dimensions is contra our goals. See Figure 14.5 and the surrounding text of [23] for a similar argument also in the context of clustering. We note that this particular problem is actually quite difficult. There are several green diamonds that occupy the space of the orange squares. There is a single orange square separated from the others by the entire blue circle cluster. There are also two blue squares in the lower left that are separated from the others by two green diamonds. For these reasons, we do not expect any clustering algorithm to achieve 100% accuracy. Figure 2: Data for clustering. The different colors and shapes indicate class membership of the samples. The samples are fed into all 5 neural networks as a 64-sample batch. A key requirement of CoOL is simultaneity of samples for all the networks, i.e. each network must be shown the same samples at the same time. It is not possible to, for example, reorder the samples for one of the networks. In this example, the samples shown to each network are identical and the networks are also identical (but not copies). This is a convenience, the samples shown to each network need not be the same, they just need to be related by simultaneity. We discuss this as further work in Section 6. Figure 3: Agreement rate (blue, solid line and circles) and number of unique labels (orange, dashed line) during training on the clustering problem shown in Figure 2. The top plot shows the case when learning is regularized by the determinants of the error rate matrices of the neural network outputs. The bottom plot shows the case without this regularization. The networks are trained for 2000 epochs with a learning rate of 10−410^-4 and no weight decay using the Adam optimizer [30]. During training, we monitor two quantities: the agreement rate and the number of unique labels produced by CoOL. Both of these are shown in Figure 3. Agreement rate is the fraction of samples where all of the observers agree on which clusters the samples belong to. The number of unique labels is the number of different labels that the networks produce for the samples that they all agree on. Figure 4: Results of clustering by CoOL. The different colors and shapes indicate cluster membership of the samples. The filled shapes show samples that were correctly clustered together while the unfilled shapes show samples that were incorrectly clustered together. For the latter, the color represents the cluster CoOL assigned the sample to while the shape represents the true cluster identity. In this work we attempted to use the results of the EM-algorithm to estimate the “true” cluster for each sample and use cross-entropy to train the observers as if they were supervised. We see in the bottom plot of Figure 3, that this was not successful. The most expedient route for the observers to agree on the cluster labels for each sample is to agree that all of the samples belong to a single cluster. This is both a strong attractor during optimization and not useful. This degenerate case has an easy-to-identify signature in that the reliability matrix for each of the observers has only a single column with non-zero elements. To regularize against this, we add a term to the loss proportional to the absolute value of the determinant of each of the observers. See Sections 4 and 7.1 for more details on this term. As shown in the top plot of Figure 3, this regularization helps CoOL learn clusters for the samples. The results of the clustering by CoOL are shown in Figure 4. As predicted due to the difficulty of the problem, some of the samples were incorrectly clustered. However, we see that CoOL has done a reasonable job of clustering the samples. The discrimination boundaries are clearly non-linear. It also does not appear to have maximized the margins between the clusters [11]; see the two blue empty squares to the lower right of the filled orange squares. Nevertheless, CoOL has done a good job finding the clusters without any kind of distance- or similarity-based hyperparameter. To generate labels using CoOL we show the samples to each of the networks to obtain a probability distribution over the possible output labels. We then say that a network’s assignment is the maximum probability label. It is then possible to reconcile the labels among the networks for each sample using any of the strategies for solving the label correspondence problem given in Section 3. In practice and for the work here, we find that all the networks agree on the label, so such reconciliation is not necessary. Because of this, we simply take the output of the first model as the predicted labels. Note that these labels are not related to the “true” labels in the dataset, should any exist, except through the mapping made by CoOL. It is possible for the number of output labels to be different from the number of true labels. We cover this topic in Section 5.2. 3 Related work One way to view CoOL is as a clustering algorithm. While most algorithms, including deep-learning based algorithms, rely on some notion of distance (with DBSCAN as a classical example [45]) or similarity/affinity between samples (with spectral clustering as a classical example [59]), CoOL relies on the self-expressiveness of the data. This idea is similar to the one described in [14], but we do not assume anything about how the samples relate to each other or the subspaces (manifolds) they inhabit. An expansive survey of deep clustering is available in [65]. Distance-based methods have been extended to deep learning typically by using auto-encoders to transform the data into a lower dimensional latent space. This strategy is common in the deep clustering literature [19, 15, 40, 34, 43, 49]. These methods require some definition of closeness, density, or similarity in the reduced dimensionality (latent) space. Deep learning similarity/affinity-based methods have also been implemented [37, 47, 63]. These kinds of algorithms can scale poorly to large sample sizes in general because the affinity computations scale as the number of samples squared unless special steps are taken. Recent strategies tend to focus on approximating the affinity computations to avoid the unappealing scaling. Two examples are SpectralNet, which approximates the affinity matrix using mini-batch sampling [47], and Deep Embedded Subspace Clustering, which treats the bases of the affinity function as parameters to be learned [9]. Others have used ensembles of networks, trained independently, to compute the similarity of samples through consensus of those networks [20]. Another way to view CoOL is as a label reconciliation algorithm or ensemble algorithm for clustering. A comprehensive survey of classical ensemble algorithms is given in [58], the most relevant are a series of works by Topchy [54, 56, 55]. Importantly, these works attempt to solve the label correspondence problem333The label correspondence problem is the task of associating labels from different labeling strategies with each other. once a number of different models have produced cluster labels, whereas here we use learnable labels from a cohort of trainable observers to estimate a set of cluster labels. The authors of [18] use expectation maximization to perform image segmentation. Therein, they train multiple neural networks in parallel to simultaneously learn representations of image objects and the assignment of pixels to the different objects, which they call clusters. EM has also been used to learn the parameters for stochastic neural networks and a mixture of experts [39], to train neural networks to learn a mixture model [53, 35], and to train a text encoder while simultaneously cleaning noisy labels [10]. CoOL shares a passing similarity with methods that learn about data by comparing two unlabeled streams of data, such as deep canonical correlation analysis [3], maximum covariance analysis [8] (and its deep version [36]), and coincidence anomaly detection [26]. 4 Learning In this section we adapt the machinery of expectation maximization [13], as described by Dawid and Skene [12] and shown diagrammatically in Figure 1, to the problem of unsupervised clustering. We will discuss the process as batch-based, where each batch uses the whole dataset, and constitutes an epoch. However, all of the formulae to come depend on a sum over the samples, so they can easily be used in a distributed data parallel fashion or in mini-batches. We assume K randomly initialized observers, which are any differentiable way to estimate the probabilities of J possible classes. We further assume that there are I samples in our batch. The output of observer k for sample i and class j is given by sij(k)s_ij^(k), and we have ∑jsij(k)=1 _js_ij^(k)=1. The first thing we need is an estimate for the probability of each sample being a member of each class, which we compute as the mean over all the observers, Tij(0)=1K∑k=1Ksij(k).T_ij^(0)= 1K _k=1^Ks_ij^(k). (1) Where the superscript (0)(0) indicates that this is the first estimate. Following Dawid and Skene, we can then compute the proportion of samples that belong to each class as pj=1I∑i=1ITij(t)p_j= 1I _i=1^IT_ij^(t) (2) for any iteration, t, of EM. We will suppress the superscript (t)(t) for all variables other than Tij(t)T_ij^(t) because they are all estimated the same way given Tij(t)T_ij^(t), whereas Tij(0)T_ij^(0) is estimated differently from later values for Tij(t)T_ij^(t). To compute what we call the reliability matrices for each observer and remain consistent with the work of Dawid and Skene, we need a label for each sample made by each observer rather than a probability distribution over all possible labels. To do that, we draw a single sample from the distribution represented by each observer’s output for each sample. In doing so, we create a binary matrix, nij(k)n_ij^(k), with a single 1 in each row, for each observer that is the same shape as Eq. 1. We can then compute the reliability matrix for each observer as πjl(k)=π~jl(k)∑l=1Jπ~jl(k) _jl^(k)= π_jl^(k) _l=1^J π_jl^(k) (3) where we define π~jl(k)=∑i=1ITij(t)nil(k) π_jl^(k)= _i=1^IT_ij^(t)n_il^(k) (4) for any iteration, t, of EM. We can then use Eq. 2.5 of [12] to compute a new estimate for Tij(1)T_ij^(1) as Tij(1)=pj∏k=1K∏l=1J(πjl(k))nil(k)∑q=1Jpq∏k=1K∏l=1J(πql(k))niq(k)T_ij^(1)= p_j _k=1^K _l=1^J ( _jl^(k) )^n_il^(k) _q=1^Jp_q _k=1^K _l=1^J ( _ql^(k) )^n_iq^(k) (5) and repeat the process again. The above formulae can be re-used for each iteration of EM. The number of parameters to be estimated by EM is very small, KJ2+JKJ^2+J per iteration. This will require hundreds of iterations to reach the size of a small neural network, and here we focus on only a single iteration. In other words, memory consumption is driven by gradient computation in the observers, not the EM parameters. The goal for training is to minimize the negative log-likelihood function along with the regularization term L=∑k=1KL(k)=∑k=1K(−α[∑i=1I∑c=1J(yi=c)log⁡(sic(k))]−λ|det(π~(k))|).L= _k=1^KL^(k)= _k=1^K (-α [Σ _i=1^IΣ _c=1^JI(y_i=c) (s_ic^(k)) ]-λ| ( π^(k))| ). (6) Where (yi=c)I(y_i=c) is the indicator variable and equals one when the label for sample i is c and zero otherwise. The labels come from sampling Tij(t)T_ij^(t) at the end of the EM process to produce a single label for each sample, similar to how nij(k)n_ij^(k) is generated for each observer. We did this to achieve a model-driven balance between exploration of possible labels and exploitation of labels that networks agree on. When all of the networks strongly agree on the sample label, that sample label is very likely to be chosen. When the networks are indifferent to the label, any label can be chosen. This allows the algorithm to explore the label space. The weight parameters, α and λ, allow us to change the importance of the two terms. The first term in the parenthesis is the usual cross-entropy loss function for supervised learning and the second term is the regularization of the determinants of the observer’s un-normalized reliability matrix. We perform ablations studies over these parameters in Appendix 7.3. We simply note here that λ is a much more important parameter than α or the choice of the negative log-likelihood as the training function. For an output m of a particular observer v on sample h the derivative of the first term is given by ∂L∂shm(v)|λ=0=−α(yh=m)shm(v). . ∂ L∂ s_hm^(v) |_λ=0=-α I(y_h=m)s_hm^(v). (7) Thus, the gradient for most of the shm(v)s_hm^(v) are zero except for the outputs which have been selected through the sampling of Tij(t)T_ij^(t). In addition, the gradients are only large when a label is chosen for which shm(v)s_hm^(v) is small. This matches our intuition about what should be happening during learning. When all of the observers have long agreed on the label for a sample, their outputs are near 1 and the gradients are modest. However, an observer which assigns a small probability to that label will see a correspondingly large gradient. This disagreement dynamic also happens for the determinant term of Eq. 6, which we discuss in Appendix 7.1. The derivative of the second term is given by ∂L∂shm(v)|α=0=−λ∑k=1Ksign(det(π~(k)))[∑r,q=1J(−1)r+qdet(π¯~rq(k))∂π~rq(k)∂shm(v)],=−λ∑k=1Ksign(det(π~(k)))K[∑q=1J(−1)m+qdet(π¯~mq(k))nhq(k)], split . ∂ L∂ s_hm^(v) |_α=0&=-λ _k=1^K sign( ( π^(k))) [ _r,q=1^J(-1)^r+q ( π_rq^(k)) ∂ π_rq^(k)∂ s_hm^(v) ],\\ &=-λ _k=1^K sign( ( π^(k)))K [ _q=1^J(-1)^m+q ( π_mq^(k))n_hq^(k) ], split (8) where π¯~mq(k) π_mq^(k) is the minor matrix of π~(k) π^(k) with row m and column q removed; the second line comes from combining Equations 1 and 4 and substituting for π~rq(k) π_rq^(k) in the first line. This gradient does not depend on v, the identity of the observer that produced the estimate shm(v)s_hm^(v), and sums over k, all of the observers. Because of this, the gradient from this term is the same for all the observers. In other words, the observers share the same gradient that is a kind of weighted average over the gradients of all the individual observers. Unlike the “selective” gradient shown in Eq. 7, and perhaps hard to tell, the gradient term in Eq. 8 is usually non-zero for all samples, h, and outputs, m. This sharing of gradients may help get the observers out of local minima or saddle points. As each observer is randomly initialized, it will start in a different part of the, potentially enormous, parameter space. From there they share gradients and learn until they have agreed on the class label for most of the samples, i.e. the algorithm only converges when the parameters for all of the observers produce a nearly identical set of labels for all the samples. Because the observers all start out in a different part of parameter space, we do not expect (and have not observed) them converging to the same location in parameter space. Therefore, in order to agree on the label of the training samples, we expect that the observers will use the strongest features to partition the samples and that these partitions will generalize to samples not seen during training. We provide evidence for this in Section 5.1. We note that det(π~(k)) ( π^(k)) is largest when the samples are equally partitioned among the classes. This is not generally a problem, as shown in Section 5.2, because we will use grouping to get more similarity information on the samples. Why use π~rq(k) π_rq^(k) in Equation 8 instead of πrq(k) _rq^(k)? Taking the derivative of Equation 6 with π substituted for π~ π, we find ∂L∂shm(v)|α=0=−λ∑k=1Ksign(det(π(k)))K(∑ℓ=1Jπ~mℓ(k))2[∑q=1J(−1)m+qdet(π¯mq(k))∑i=1ITim(0)(nhq(k)−niq(k))], . ∂ L∂ s_hm^(v) |_α=0=-λ _k=1^K sign( (π^(k)))K( _ =1^J π_m ^(k))^2 [ _q=1^J(-1)^m+q ( π_mq^(k)) _i=1^IT_im^(0) (n_hq^(k)-n_iq^(k) ) ], (9) where we have made the same assumptions used in Equation 8. From the properties of determinants and using Equation 3 it is straightforward to see that sign(det(π(k)))=sign(det(π~(k))) sign( (π^(k)))= sign( ( π^(k))) and det(π¯mq(k))=det(π¯~mq(k))∏w=1,w≠mJ∑u=1Jπ~wu(k). ( π_mq^(k))= ( π_mq^(k)) _w=1,w≠ m^J _u=1^J π_wu^(k). (10) We note that the denominator in this equation is positive semi-definite and does not depend on q, so it can be moved outside the sum over q. The terms Tim(0)T_im^(0) are also positive semi-definite. All of these facts combined mean that the term in in Equation 9 proportional to nhq(k)n_hq^(k) always points in the same direction as Equation 8 and is always much smaller in magnitude. However, the term proportional to niq(k)n_iq^(k) can be larger, and cause the gradients of Equations 8 and 9 to point in different directions. We have found that Equation 8 leads to larger determinants of the π(k)π^(k) matrices and much better partitioning of the samples. A comparison of the evolution of the determinant of reliability for the two different regularization strategies is given in Figure 5. We leave further exploration of this facet of CoOL, including the effect of using more than one iteration of EM (implied by the use of Equation 1), to future work. 4.1 Understanding Convergence Without Labels In a typical clustering problem, we will not have access to labels or be able to see the clustering in low dimension as we did in Figure 4 to evaluate the progress of CoOL. In order to determine the success of CoOL, defined as converging to an agreed-upon partition of most of the samples, we monitor five attributes: (1) loss, (2) agreement rate, (3) the number of unique classes, (4) the determinants of the π(k)π^(k) matrices, and (5) the Shannon Equitability Index. The loss tells us how well CoOL is learning to cluster the samples, lower is better and saturation indicates there is nothing left to learn. We saw in Figure 3 that agreement rate, the fraction of the samples on which all the observers agree on the label, grows while CoOL is learning, while the number of unique classes will drop if CoOL ends up in a degenerate solution. Typically, on a successful run of CoOL, the number of unique classes never varies from the number defined by the user and the agreement rate gets well above 95%. Figure 5: (top) Loss as function of epoch when learning the problem shown in Figure 2 using Equation 8 (solid lines) and Equation 9 (dashed lines). (middle) det(π(k)) (π^(k)) for the 5 observers trained. (bottom) Shannon Equitability Index (SEI) for the 5 observers trained. Both runs resulted in agreement rate above 98% and 3 classes. However, the observers trained using Equation 9 put 48 of the 64 samples into a single cluster. In Figure 5 we see the evolution of the other attributes for a successful run (solid lines) and a run that was not successful (dashed lines). It is clear from this figure that the determinant of the reliability matrix (π(k)π^(k)) grows for all the observers when the observers are learning to cluster the samples, but the same is not true for the less successful run. The determinants might not get to one, even on a successful run, because a small number of samples in the “wrong” part of the reliability matrix can dramatically reduce the value of the determinant. This problem is worse with a larger number of clusters. The Shannon Equitability Index (SEI) is a measure of how equally each class is represented in a dataset, it is given by EH=−∑n=1Npnlog⁡(pn)log⁡(N),E_H=- _n=1^Np_n (p_n) (N), (11) where N is the number of classes and pnp_n is the proportion of samples in class n. EHE_H takes on the value of 0 when there is only one class in the dataset and 1 when the dataset is equally partitioned between all the classes. Equal partition of the samples is the most favored outcome of learning with Equation 6. This is because such a partition maximizes the size of the π~(k) π^(k). Between Figures 3 and 5 the difference between a successful run of CoOL and several failure modes is clear. In a successful run: the loss will continue to grow in magnitude; the agreement rate will approach unity; the number of unique classes will equal the number of allowed clusters; the determinant of the reliability matrices will grow and saturate; and the SEI will approach 1 for all observers. The lack of any of these features is an indication that learning was not successful. We reiterate that none of these metrics requires knowledge of the “true” cluster identity of the samples, if that information even exists. 5 MNIST Example Herein, we show CoOL clustering the MNIST digits. We chose this example because the data is readily available from any number of sources and is typically used as a (nearly) first example for both deep learning packages and deep learning itself. Throughout this section we will refer to problems by the number of classes that CoOL will have to cluster. The digits used during any experiment are counted up starting from zero to the number of classes less one. As an example, when we say that an experiment uses 5 classes, we mean that it uses digits 0 through 4. We will show both where CoOL succeeds and where it fails. We emphasize here that we focus on image datasets because they are accessible, CoOL is not limited to working with images. 5.1 Clustering MNIST digits Figure 6: Plot of accuracy versus agreement rate for various hyperparameter settings on the MNIST digit problem. The different markers indicate the learning rate used: circles mean 10−510^-5, squares mean 10−410^-4, and triangles mean 10−310^-3. Filled markers indicate that a weight decay of 10−410^-4 was used while empty markers mean no weight decay was used. For this hyperparameter scan we used 288 samples per class. As is usually the case with any deep learning problem, the first thing we do is perform a hyperparameter scan. In this scan, we vary the number of digits, learning rate, and weight decay factor, and keep α=λ=1α=λ=1. We train for 2000 epochs using the Adam optimizer and 288 samples per class. Following Section 4.1, we judge the success of any given run by the agreement rate and the number of unique classes the observers use. In Figure 6, we compare the agreement rate with the accuracy of the clusters. To compute accuracy, we assign to each cluster the true label that is most common in that cluster and then each sample in that cluster with that label is counted as a correct prediction while samples with a different label are counted as incorrect predictions. We see that agreement rate (among the observers) is a fairly good predictor of accuracy, i.e. accuracy tends to be high when the agreement rate is high. The exception is when the observers do not partition the samples into all of the desired classes, but this outcome is easily spotted, as shown in Section 2. It also appears that weight decay is not useful. It may be the case that gradient sharing sufficiently guards against overfitting. Figure 7: Determinant of the π(k)π^(k) matrix for all 5 observers during select runs of CoOL on a varying number of digits with a varying number of classes (shown in the legends). (left) shows a linear scale and (right) shows a log scale. For completeness, these plots show training through 5000 epochs regardless of when early stopping occurred. It is clear from Figure 6, that the agreement rate (and the accuracy) both fall as the number of desired digits/classes increases. In Figure 7 we plot |det(π(k))|| (π^(k))| (see Equation 3) for each of the 5 observers during runs of CoOL with varying number of digits and classes. From this point on, we train for 5000 epochs using 1000 samples per class. The 9-class scenario is not converging because two of the observers are learning poorly (their determinants are stuck at or near zero), while the other three observers seem to learn for a little while. The 7-class scenario shows determinants that are no longer growing at a value of about 0.6. We will see in Appendix 7.3 that CoOL performs better on the higher-class-count experiments with smaller values of λ. It may be the case that a change in consensus label for a sample improves |π(k)||π^(k)| for some of the observers, but diminishes the same for other observers and this causes a contention. The smaller shared gradients required for learning the higher-class-count clusters suggests that the region of convergence is smaller and learning weight decay strategies might be useful. We leave investigation of this issue for future work and note that in Section 5.2.1 we discuss how we can use grouping to handle clustering that is not limited by the number of clusters assumed during an individual run. In Table 1 we provide the agreement rate, accuracy, normalized mutual information (NMI) [60], and adjusted rand index (ARI) [25] at the end of training. The latter three are computed on a hold-out set. We stop training CoOL when the agreement rate reaches 99.5% or epoch 5000 is reached, which ever is first. All experiments are performed five times; we show the mean and standard deviation of all the quantities. We train two different architectures using CoOL: one based on convolutional neural networks (CoOL-Conv) and another that uses a fully-connected network (CoOL-Enc).444CoOL-Enc uses the (not pre-trained) IDEC encoder architecture [19]. For comparison, we also show the performance of related algorithms: Improved Deep Embedded Clustering (IDEC) [19], inTerpretable nEuraL cLustering (TELL) [43], SpectralNet [47], and k-Means [29]. All of these algorithms are trained using the parameters and architectures that were used to train them on MNIST in their respective publications, except for k-Means, where the default parameters in scikit-learn are used [42]. The algorithms are trained from scratch except for IDEC for which weights are available that were pre-trained on the entire MNIST dataset [61]. IDEC has the highest accuracy across all the datasets. This is hardly surprising given the extensive pretraining of the autoencoder that was available. Such pretraining might not be available in other contexts, as there might not be enough data, but IDEC serves as a point of comparison for the other methods. The second highest accuracy for each of the cases belongs to CoOL-Conv (3-classes) and TELL (5-, 7-, and 9-classes). CoOL-Conv is third in both the 5- and 7-class cases, performing nearly as well as TELL, while TELL is third in the 3-class case. SpectralNet appears to be consistently miss-clustering the hold-out set because the NMI and ARI are fair to good, but the accuracy is not. The performance of the fully-connected version of CoOL is modest. It is interesting that the performance of K-Means is better as the number of classes is increased (in terms of all metrics) and the same is true for SpectralNet (in terms of NMI and ARI). Because the datasets get larger with larger class counts, both because there are more classes and and there are more samples overall, this suggests that these algorithms benefit from more data and/or class diversity. CoOL appears to do best when the class count is modest, which was foreshadowed in Figures 6 and 7. For these runs, we set α=λ=1α=λ=1 as suggested default parameters, in order to make a fair comparison with the other methods, for which we took the parameters used during learning on MNIST in their respective publications. In Appendix 7.3 we show that performance on the higher class-count problem can be dramatically improved by changing the λ parameter while monitoring the agreement rate. In addition, we discuss the computational performance of CoOL in Appendix 7.2. Table 1: Comparison between different deep clustering models on the MNIST dataset. The columns show the number of classes used from the MNIST dataset. For each model, we show the accuracy on a hold-out set (acc.), normalized mutual information (NMI), and adjusted rand index (ARI). In addition, the CoOL models show the agreement rate. Each experiment was run 5 times to generate a mean and standard deviation as shown. 3-MNIST 5-MNIST 7-MNIST 9-MNIST CoOL-Conv agr. rate 0.995± 0.000 0.995± 0.000 0.959± 0.007 0.001± 0.002 acc. 0.957± 0.060 0.834± 0.080 0.697± 0.120 0.259± 0.018 NMI 0.869± 0.121 0.719± 0.101 0.654± 0.077 0.141± 0.028 ARI 0.892± 0.138 0.692± 0.130 0.589± 0.101 0.073± 0.017 IDEC acc. 0.989± 0.001 0.965± 0.001 0.885± 0.001 0.827± 0.001 NMI 0.941± 0.004 0.901± 0.002 0.842± 0.001 0.764± 0.001 ARI 0.967± 0.003 0.916± 0.002 0.800± 0.001 0.699± 0.001 TELL acc. 0.874± 0.013 0.864± 0.018 0.752± 0.020 0.617± 0.014 NMI 0.711± 0.017 0.695± 0.028 0.576± 0.023 0.509± 0.019 ARI 0.684± 0.024 0.692± 0.032 0.541± 0.032 0.420± 0.021 SpectralNet acc. 0.383± 0.288 0.106± 0.141 0.092± 0.062 0.131± 0.110 NMI 0.577± 0.118 0.680± 0.140 0.769± 0.037 0.856± 0.005 ARI 0.486± 0.165 0.573± 0.234 0.671± 0.073 0.865± 0.006 CoOL-Enc agr. rate 0.997± 0.001 0.996± 0.001 0.645± 0.440 0.046± 0.048 acc. 0.817± 0.116 0.658± 0.125 0.436± 0.165 0.238± 0.112 NMI 0.604± 0.174 0.507± 0.111 0.336± 0.189 0.124± 0.114 ARI 0.610± 0.203 0.452± 0.150 0.253± 0.142 0.077± 0.074 K-Means acc. 0.312± 0.004 0.484± 0.002 0.537± 0.073 0.638± 0.021 NMI 0.267± 0.005 0.475± 0.001 0.482± 0.027 0.532± 0.004 ARI 0.158± 0.007 0.350± 0.001 0.376± 0.037 0.443± 0.008 5.2 Other Scenarios In this section we demonstrate CoOL’s performance beyond the somewhat idealized cases of the previous sections. 5.2.1 The Wrong Number of Clusters Up until this point and for convenience of the exposition, we have run CoOL on datasets where we knew the number of clusters ahead of time. Typically, this information is not known ahead of time and we now turn to the matter of understanding the clustering without this information. In Section 4.1 we demonstrated a method for monitoring convergence without cluster labels, so the labels are not necessary to determine whether or not CoOL has converged to a partition of the samples. There are still several outstanding questions: (1) how can we be assured that the clusters are not random and (2) how do we deal with the arbitrary labels given to the clusters by CoOL? The crux of the problem of the second question is that on any two runs of CoOL, the label given to a sample need not be consistent. For example, a sample might be put in cluster 0 for the first run, cluster 3 in the second run, and cluster 2 on the third run. In fact, we need not even assume the same number of clusters for the different runs. The solution begs the question about the nature of clustering. Clustering only works, no matter the algorithm, if some set of the samples are more similar to each other than they are to some other set of samples. Therefore, if CoOL is clustering in some sensible and consistent way, more-similar samples will be clustered together on each successive run of CoOL, even though the arbitrary label is changing for each run. To understand the consistency of the clustering, we assign each sample to a “group” that is simply the concatenation of the cluster assignments given on the various runs, in order. To continue the example sample described earlier, it would have group (0, 3, 2). It is reasonable to ask, at this point, which cluster to use for a given run if all of the observers in that run do not agree on the cluster. The easiest thing to do is to separate those samples from the others. As we saw in Section 5, we can train until the number of disagreements is small, so this is hardly a hindrance to the method. We use this strategy here, but note that the fact that observers cannot agree on a sample’s cluster is information that can be exploited by the user to investigate, for example, anomalies. Figure 8: The fraction of groups containing the given number of samples when the features of the MNIST digits are shuffled on a per-sample basis. The distribution is very close to random assignment of the samples to the groups. With a group label for each sample, we can immediately answer the question as to random assignment. If the assignments are random, then the group labels would also be random and the number of members in each group would follow a binomial distribution with mean given by I/(∏r=1RCr)I/( _r=1^RC_r), where I is the number of samples, CrC_r is the number of clusters used for run r, and R is the number of runs. In Figure 8 we show the distribution of the number of samples per group when we shuffle the features of each sample. The distribution is quite similar to random assignment. Table 2: Table of all the groups with at least 10 samples found when clustering 5 digits into 4 classes using CoOL. The grouping produced 64 groups total. Group is the group name for a set of samples; count is the number of samples in that group; label is the most common label in the group; labels is a list of all the labels found in the group; and consistency is the fraction of the group samples that have the label given in the label column. A consistency of 1 means that all of the group samples have the same label. Group count label labels consistency (1, 3, 3, 3, 3) 877 0 (0) 1.000 (3, 2, 0, 2, 0) 774 4 (0 2 3 4) 0.995 (0, 0, 2, 1, 2) 764 1 (1 2) 0.995 (2, 1, 1, 0, 1) 411 3 (2 3) 0.968 (2, 1, 0, 0, 0) 234 2 (2) 1.000 (2, 1, 1, 1, 2) 135 2 (1 2 3) 0.948 (1, 1, 1, 0, 1) 103 3 (2 3) 0.990 (0, 0, 2, 2, 2) 79 1 (1) 1.000 (1, 3, 1, 0, 1) 50 3 (3) 1.000 (2, 2, 0, 0, 0) 44 2 (2) 1.000 (3, 2, 0, 0, 0) 15 2 (2) 1.000 (2, 1, 1, 0, 2) 14 2 (2 3) 0.857 (3, 1, 1, 0, 1) 14 3 (3) 1.000 (2, 1, 0, 0, 2) 11 2 (2) 1.000 (2, 3, 0, 0, 0) 10 2 (2) 1.000 (3, 1, 1, 2, 1) 10 3 (3) 1.000 To demonstrate the grouping process we ran CoOL 5 times. The hyperparameters were the same for each run: 5 observers, 4 clusters, 5 digits, 1000 samples per digit, 2000 epochs, learning rate of 10−410^-4, no weight decay, and the Adam optimizer. We then group the samples as just described and show the ten largest groups in Table 2. We see immediately that the groups are not random as we would expect, in that case, nearly 10 samples per group distributed over 1024 groups. Instead, there are only 64 groups that contain any samples at all. We see that CoOL has easily separated nearly 90% of the zeros from the rest of the samples. Performance on digits 1 and 4 is modestly inferior. In contrast, CoOL is regularly clustering digits 2 and 3 together into smaller groups instead of two separate larger groups. These group names are themselves meaningful as we would expect that samples from two group names with many matching digits will be more similar to each other than with a sample from a group with a totally different group name. For example, the groups (2, 2, 0, 0, 0) and (3, 2, 0, 0, 0) are both small groups of digit 2. The fact that several groups of digits 2 and 3 begin with the sequence (2, 1) or, even, (2, 1, 1) suggests that those digits look similar to CoOL. The previous examples assume the number of clusters is very close to the true number of clusters and, under this constraint, we found that the clusters or groups typically represent digits. In Appendix 7.4 we investigate what happens when the true number of clusters is much larger than the number of clusters assumed by CoOL. We find that CoOL does not focus on digits, but rather large scale features of digits. The largest group appears to be narrow digits that slope from the lower left to the upper right. The second largest group appears to be digits with large loops. The third and fourth groups, which are very close together in group-space being labeled (1, 0, 2, 3, 1) and (0, 0, 2, 3, 1), look both quite similar as boxes with tails down to the lower right and distinct in the patterns which fill in the legs of the box. All of this is to say that CoOL has found a way to cluster and group 10 digits using only 4 clusters and those groups look reasonably consistent both within those groups and with nearby groups (and contrastive to more distant groups). We now return to the two questions we asked at the beginning of this section. The cluster assignments from CoOL do not appear to be random, which answers our first question. We have also provided a method, i.e. grouping, for understanding how consistently CoOL clusters similar samples together. Grouping provides a framework for a user to understand the cluster assignments in the data CoOL is trained on, but CoOL can give a cluster assignment for samples it was not trained on [21], we turn to what happens in that case next. 5.2.2 Unseen Digits In Section 5.1 we saw that CoOL performs similarly as well on a hold-out set of samples of the digits it was trained on. This can be seen in a different way in Figure 9, where we see that the clustering of samples not used for training or validation is quite good (these samples are from digits that were used during training). The bottom plot of Figure 9 shows what happens when CoOL is shown samples from digits it was not trained on. Figure 9: (top) The agreement rate for all digits (and samples) CoOL was trained or validated on. (bottom) The agreement and disagreement rate for samples and digits not used during training. The digits to the left of the dashed vertical line were used for training. It is clear that digits that were not learned have much higher disagreement rate than those that were in the training set. This demonstrates another useful feature of CoOL. Because the clustering generalizes well, CoOL can be used to detect distributional drift of the input data. In other words, CoOL can be used to detect changes in the data over time without understanding anything about the clusters. This might be used to detect when new data is out of distribution (OOD) from a training set [50]. A recent example of this problem in the context of radiation management in a super-conducting accelerator can be found in [16]. As a hypothetical distributional shift, suppose we monitor the disagreement rate as a function of time during a process that we have recently trained CoOL to cluster. The disagreement rate is given by Δ≐td+udt+u=δt1+δuδtut1+ut. t_d+u_dt+u= _t 1+ _u _t ut1+ ut. (12) Where t is the number of train-like samples and tdt_d is the number of train-like samples that CoOL disagrees on; u and udu_d are defined similarly for the “unlike” samples, i.e. those samples unlike the ones seen during training, and δt=td/t _t=t_d/t (δu=ud/u _u=u_d/u) is the fraction of train-like (unlike) samples that are disagreed upon. The ratio of unlike samples to train-like samples is ut=Δ−δtδu−Δ. ut= - _t _u- . (13) If we assume, as seems reasonable from Figure 9, that δt=0.01 _t=0.01 and δu=0.1 _u=0.1, and take Δ=2δt =2 _t, we find u/t=u/t= 1/8. Thus, a modest number of OOD samples will cause a doubling in the overall disagreement rate if the new samples are well out of distribution from the training samples. Further exploration of this topic is context dependent and is the subject of future work. 6 Conclusions and Future Work We have described CoOL, a method for clustering data while simultaneously training differentiable models to do the same with end-to-end learning. CoOL can cluster data with any dimensionality compatible with the models. On the experiments we ran, CoOL returned reasonable clusters. Along the way, we gave formulae for the gradients passed back to the models and a method for monitoring convergence without ground truth labels. We have also shown that, even in the absence of interpretable clusters, a trained set of models can be used to detect distributional drift by monitoring the disagreement rate between the models. Throughout this work, we have noted several areas for future work. The question on what is the largest number of clusters that can be used with CoOL remains open, as does how many iterations of EM can be used productively. In addition, different methods for selecting nil(k)n_il^(k) are an open topic of research, especially less stochastic ones. We anticipate that there are other, potentially superior, ways to regularize learning. For example, the entropy of the partition of the samples should have similar properties as the determinant of the reliability matrix [48], maximizing the normalized mutual information might also offer some benefits [60]. We further expect that having a relatively small number of labeled samples will allow semi-supervised training of the networks. Another task is to use simultaneity to cluster different representations of the same set of concepts. For example, three of the observers could be shown images while 2 others are shown sounds related to those pictures. In this example, clustering in one input type might be easier than the other, and so simultaneity might be used to improve clustering in a difficult space, or simply to associate related sounds and images with the same cluster labels without supervision. This is possible because CoOL can be used on any data type with an associated differentiable model. Of course, we need not limit the number of input types to two or to images and sound. 7 Appendices 7.1 Properties of the Determinant Near Convergence Assume that we have K observers that agree on I−1I-1 of the samples, but there is one observer, K, that estimates a small probability that sample I is a different class from all the other observers’ estimates. Further assume that these observers classify a fraction, β, of the I−1I-1 samples as class 1 and 1−β1-β as class 0. All the observers classify sample I as certainly class 0 except observer K, which assigns probably α that this sample is class 1. Under these assumptions, we find that T^ij(0)=δj,ci T^(0)_ij= _j,c_i for the first I−1I-1 samples and T^I0(0)=1−α/K T^(0)_I0=1-α/K (T^I1(0)=α/K T^(0)_I1=α/K) for the remaining sample. At this point, it is straightforward to compute the EM parameters. First, we find Ip0=(1−β)(I−1)+1−α/KIp_0=(1-β)(I-1)+1-α/K and Ip1=β(I−1)+α/KIp_1=β(I-1)+α/K. For the K−1K-1 certain observers (k∈1,…,K−1k∈\1,…,K-1\), the un-normalized reliability matrix is π~(k)=((1−β)(I−1)+1−α/K0α/Kβ(I−1)). π^(k)= pmatrix(1-β)(I-1)+1-α/K&0\\ α/K&β(I-1) pmatrix. (14) For the uncertain observer, there are two possibilities. If the sampling of the observer’s outputs returns class zero (nI0(K)=1n_I0^(K)=1 and nI1(K)=0n_I1^(K)=0), then the reliability matrix is identical to that shown in Eq. 14. On the other hand, if that sampling returns class 1 (nI0(K)=0n_I0^(K)=0 and nI1(K)=1n_I1^(K)=1), the reliability matrix is π~(K)=((1−β)(I−1)1−α/K0β(I−1)+α/K). π^(K)= pmatrix(1-β)(I-1)&1-α/K\\ 0&β(I-1)+α/K pmatrix. (15) In either of these matrices, and their normalized forms, there are only two free parameters, β and α, the latter of which represents observer K’s uncertainty about which class sample I belongs to. We take the derivative of the determinants of Eqs. 14 and 15 with respect to an arbitrary parameter of one of the observers which we will call xrx_r where the subscript r references the observer number. We note that the parameter α depends only on observer K, while β depends on all of the observers. For Eq. 14 the derivative is ∂xr(detπ~(k))=(I−1)[(1−2β)(I−1)+1−αK]∂β∂xr−β(I−1)K∂α∂xrδrK ∂ x_r( π^(k))=(I-1) [(1-2β)(I-1)+1- αK ] ∂β∂ x_r- β(I-1)K ∂α∂ x_r _rK (16) and the derivative of Eq. 15 is ∂xr(detπ~(K))=(I−1)[(1−2β)(I−1)−αK]∂β∂xr+(1−β)(I−1)K∂α∂xrδrK. ∂ x_r( π^(K))=(I-1) [(1-2β)(I-1)- αK ] ∂β∂ x_r+ (1-β)(I-1)K ∂α∂ x_r _rK. (17) We can see that the determinant acts as expected on the α term. When the sampling of observer K’s outputs chooses class 0, maximizing the determinant decreases α. However, when the sampling of observer K’s outputs chooses class 1, maximizing the determinant increases α. The expected coefficient for the ∂α/∂xr∂α/∂ x_r term is α(1−β)(I−1)K−(1−α)β(I−1)K=I−1K(α−β).α (1-β)(I-1)K-(1-α) β(I-1)K= I-1K(α-β). (18) This expectation tells us that when α is greater than β the former tends to increase. In other words, when observer K assigns a probability (α) that sample I is class 1 that is larger than the fraction of samples that all observers agree are class 1 (β), that probability tends to increase. We now normalize the rows of Eqs. 14 and 15 such that they sum to 1 before computing the derivative. For Eq. 14 this yields ∂xr(detπ(k))=I−1KI2p12[α∂β∂xr−β∂α∂xrδrK] ∂ x_r( π^(k))= I-1KI^2p_1^2 [α ∂β∂ x_r-β ∂α∂ x_r _rK ] (19) while for Eq. 15 this yields ∂xr(detπ(K))=I−1I2p02[−(1−αK)∂β∂xr+(1−β)K∂α∂xrδrK]. ∂ x_r( π^(K))= I-1I^2p_0^2 [-(1- αK) ∂β∂ x_r+ (1-β)K ∂α∂ x_r _rK ]. (20) These equations have similar properties, with respect to the ∂α/∂xr∂α/∂ x_r term, as the un-normalized case. However, the expected coefficient of that term is not as easy to parse. We find this expectation value is I−1KI2p02p12[α(1−β)p12−(1−α)βp02]. I-1KI^2p_0^2p_1^2 [α(1-β)p_1^2-(1-α)β p_0^2 ]. (21) To simplify this equation, we assume β=1/2β=1/2, in which case p0≈p1≈1/2p_0≈ p_1≈ 1/2 as well. This yields an expectation that is 4/I24/I^2 times smaller than Eq. 18 under the same assumptions. The further motivates our use of the un-normalized reliability matrix in Section 4. 7.2 Computational Performance of CoOL Here we investigate the computational performance of CoOL. We train the convolutional version of CoOL on an NVidia A100 GPU and record the time it takes to reach 99.5% agreement rate on the validation set or 5000 epochs, which ever comes first. We measure the memory reserved using the pytorch function max_memory_reserved that returns the maximum GPU memory managed by the caching allocator during training [41]. In addition to the batch-learning version shown in Section 5, we also train a mini-batch version where each epoch is split into 1000 sample batches. Every 25 epochs we compute the accuracy, NMI, and ARI on both the training and validation set. We fix the number of classes, J, at 5. We vary the number of networks, K, between the values 3, 5, 7, and 9. We also vary the number of samples per class, I/JI/J, between the values 100, 300, 1000, 3000. Each experiment is repeated 3 times. Figure 10: Statistics for training time and memory consumption when training CoOL with convolutional layers. In the top row we show time to complete training and in the bottom row we show memory allocated by pytorch during training [41]. The left column shows batch training and the right column shows mini-batch training. The colored curves indicate the number of networks, K, used to learn 5 classes of the MNIST dataset as indicated in the legends, where the slope of the curves is also given. The black dash-dot lines with diamond markers shows the performance of TELL and the black dotted lines with x markers shows the performance of SpectralNet. The results of these parameter sweeps are shown in Figure 10. We see that, as expected from Section 4, memory consumption increases linearly as a function of the number of samples during the batch training scheme. The memory allocated is dominated by the gradients needed for backpropagation in the neural networks themselves, not in the EM portion, which is a tiny fraction of the total parameters. The training time is also linear in the number of samples. Using a mini-batch version places an upper boundary on memory consumption and also reduces training time. The latter is a common benefit to mini-batch learning [17]. In Figure 11 we see that the number of networks, K, does not effect the performance of CoOL on hold-out data. Figure 11: Accuracy of the CoOL-trained convolutional networks on a hold-out set of 5 MINST digits as a function of the number of networks. Training is stopped when the agreement rate reaches 99.5% or 5000 epochs, which ever comes first. Training IDEC on the datasets used here takes a trivial amount of time. However, the network used was pretrained extensively on the entire MNIST dataset wherein “[e]ach layer [of the autoencoder] is pretrained for 50000 iterations with a dropout rate of 20%. The entire deep autoencoder is further finetuned for 100000 iterations without dropout” [61]. Using the same hardware as the CoOL cases and pre-loading the data on to the GPU, we find that 100 epochs of pretraining takes 96.9± 1.8 seconds. Thus, the 100000 epochs of finetuning alone would take 1615 minutes, far longer than any of the runs shown in Figure 10. The training time and memory consumption for TELL are similar to CoOL, as is its performance. On the other hand, SpectralNet trains faster and uses less memory because it approximates the affinity matrix in order to enable “its scalability and thus allows one to cluster large datasets that are prohibitive for standard spectral clustering” [47]. However, SpectralNet was the poorest performer in terms of accuracy (see Table 1). 7.3 Ablation Study Herein we investigate the learning properties of CoOL by making some critical changes to the algorithm. We vary the values of λ and α, defined in Section 4. We substitute KL-divergence for the cross-entropy term in Eq. 6. And, finally, we use a pretrained backbone to replace the randomly initialized networks. Figure 12: Accuracy of the CoOL-trained convolutional networks on a hold-out set of MINST images as a function of the weight of the determinant term, λ. The box shows the settings used to produce Table 1. The dashed lines show the accuracy of a random guess model, where the colors match the colors shown in the legend. The first ablation we make is over λ. We train on the MNIST dataset in the same fashion of section 5, fix α=1α=1, and train until agreement reaches 99.5% or 5000 epochs, which ever comes first. We perform each experiment 3 times. A plot of hold-out set accuracy versus λ is shown in Figure 12. The performance depends on the selection of λ, but the peaks are broad so that very accurate selection of lambda is unnecessary. In other words, a very coarse scan, perhaps a decade or two per step, appears sufficient to identify the region of best performance. As usual, we monitor both agreement rate and the number of unique classes predicted. It is easy to tell the difference between successful and unsuccessful runs. Successful runs have smoothly increasing agreement rate that begins improving within a few hundred epochs of start and a stable number of unique classes predicted and unstable runs are missing one or both of these features. Using the λ values associated with the highest agreement rate, we vary α over many decades, including α=0α=0. We find that α has no impact at all on the 3- and 5-class cases, and a modest, ∼10% 10\%, effect on the higher class cases. In Figure 13 we compare the training time, measured in epochs, and the accuracy if we replace the negative log-likelihood loss used in Section 4 with the Kullback–Leibler divergence. We treat the model outputs as the approximating distribution and T (see Equations 1 and 5) as the “true” distribution. We also disable this term in the loss function by taking α as 0 (disable) or 1 (enable) and do the same for the negative log-likelihood loss. Training is stopped when the agreement rate reaches 99.5% or 5000 epochs, which ever comes first and we run all experiments 3 times. This figure shows that the loss term does not change the learning as all of the results (for a particular number of classes) are clustered much more tightly than the error bars. Figure 13: Accuracy of the CoOL-trained convolutional networks on a hold-out set of MINST images as a function of the number of epochs. Training is stopped when the agreement rate reaches 99.5% or 5000 epochs, which ever comes first. The empty symbols represent models trained using Kullback–Leibler divergence without sampling and the full symbols are models trained using negative log-likelihood loss as in Section 4. The blue symbols are for learning 5 digits and the red symbols are for 7 digits. The circles represent α=0α=0 and the squares represent α=1α=1. Table 3: Performance of CoOL on the CIFAR10 dataset. The models are selected by highest agreement rate among all models that produce the correct number of unique labels. The columns show the mean and standard deviation of 3 runs for each dataset for all the indicated quantities. The CoOL model converged only once on the 7-CIFAR dataset (thus, the lack of standard deviation) and never on the 9-CIFAR dataset. 3-CIFAR10 5-CIFAR10 7-CIFAR10 9-CIFAR10 CoOL agr. rate 0.999± 0.001 0.998± 0.001 0.207 CoOL acc. 0.497± 0.021 0.311± 0.013 0.221 CoOL NMI 0.079± 0.019 0.055± 0.002 0.039 CoOL ARI 0.079± 0.017 0.043± 0.001 0.024 CoOL-ResNet50 agr. rate 1.000± 0.000 1.000± 0.000 1.000± 0.000 0.893± 0.005 CoOL-ResNet50 acc. 0.468± 0.072 0.329± 0.007 0.263± 0.031 0.235± 0.008 CoOL-ResNet50 NMI 0.078± 0.029 0.068± 0.007 0.088± 0.008 0.093± 0.006 CoOL-ResNet50 ARI 0.082± 0.034 0.053± 0.006 0.061± 0.009 0.055± 0.009 To investigate the effect of a pre-trained backbone on CoOL’s performance, we switch to the CIFAR-10 dataset [31]. This is a much more difficult clustering problem. We select the classes and samples in the same way as the MNIST experiments. To replicate what a user might do with a dataset with unknown labels, we design a network that appears to learn via increasing agreement rate (available in the code that accompanies this paper). We also mean-subtract and scale the input channel data such that all the input values lie in [-1, 1]. For the pre-trained backbone we use the ResNet-50 network available in the torchvision package [24, 57] and train only the final connections to the clustering output. The input data is scaled in the same way that the data was during pre-training. For both models, we train in the same way as we did for the MNIST experiments while scanning λ from 10−910^-9 to 10610^6 in steps of 10. Model selection is done by first discarding any models that do not achieve the desired number of classes on output, and then selecting the model with the highest agreement rate. This procedure can be used on any dataset without labels. Performance of the selected models is shown in Table 3. We see that the arbitrary network that we created is partially successful at separating the different classes (in terms of accuracy). However, it does not do well on the experiments with larger number of classes in that it seldom converges in the allowed number of epochs. This serves as an indication that the models used are likely under-powered for the clustering task at hand. The pre-trained feature-extractor and clustering head is much more successful as it converges on all data sets and out-performs a random guess model. As pre-trained models are rare outside of well studied data types, such as the images we use here, we merely comment that pre-trained models are a good starting place, if one is available. Summarizing the results of this appendix, including some details we uncovered in Section 2, the following is a strategy for getting the best performance out of CoOL on new datasets without labels: (1) fix learning rate to 10−410^-4 and do not use weight decay; (2) scan λ over many decades and choose the value that maximizes agreement rate; (3) scan α over many decades and look for the value that maximizes agreement rate further still (It is also fine to take α=0α=0.); (4) train on a large number of classes (say, 7 to 9) and check to see that the determinants of their outputs are smoothly increasing, if they are not, the networks might be under-powered for the clustering task. 7.4 Groups of MNIST digits Herein, we cluster all 10 digits of the MNIST handwritten digit dataset using only 4 clusters. We train for 2000 epochs using 5 observers, 1000 samples per digit, a learning rate of 10−410^-4, and the Adam optimizer. This training is run 5 times to generate the groups shown in Table 4. At first look, it appears that CoOL has not done anything sensible as, unlike Table 2, the groups all contain multiple digits and have relatively poor consistency. However, the task of clustering the 10 digits into 4 clusters cannot possibly partition the samples into the digits. Let us see what CoOL has found, instead. Table 4: Table of 10 largest groups found when clustering 10 digits into 4 classes using CoOL. The grouping produced 276 groups total. Group is the group name for a set of samples; count is the number of samples in that group; label is the most common label in the group; labels is a list of all the labels found in the group; and consistency is the fraction of the group samples that have the label given in the label column. A consistency of 1 means that all of the group samples have the same label. Group count label labels consistency (2, 2, 0, 1, 0) 1228 1 (0 1 2 3 4 5 6 7 8 9) 0.683 (3, 3, 3, 0, 3) 856 0 (0 2 3 4 5 6 7 8 9) 0.849 (1, 0, 2, 3, 1) 479 7 (1 3 4 7 8 9) 0.697 (0, 0, 2, 3, 1) 424 4 (3 4 7 8 9) 0.491 (0, 1, 1, 0, 2) 354 6 (0 2 3 5 6 8) 0.949 (1, 0, 2, 2, 1) 336 9 (2 4 7 9) 0.426 (3, 0, 3, 3, 2) 226 2 (0 2 3 4 9) 0.969 (1, 1, 1, 2, 3) 216 3 (1 3 4 5 8 9) 0.542 (1, 1, 1, 0, 3) 195 3 (2 3 4 5 8 9) 0.800 (3, 3, 3, 0, 2) 181 6 (0 2 3 4 6) 0.420 (1, 3, 3, 0, 3) 165 4 (3 4 5 7 8 9) 0.339 By summing all the images on a per-group basis, we can plot composite images of the “mean” image for each group. This is shown in Figure 14. We see that the first six groups contain images of digits that look like 1, 0, 7, 4, 6, and 9 (from left to right in the figure). We also see that the three groups that are near each other make digits that look very similar to each other. Group (1, 0, 2, 3, 1) looks more like a 7, group (0, 0, 2, 3, 1) looks more like a 4, and group (1, 0, 2, 2, 1) looks more like a 9, but all three composite images look very similar is well. Figure 14: Composite images for the 6 largest groups shown in Table 4. We include this appendix to demonstrate that guessing the correct number of semantic labels in the data set is not necessary to get useful information (in this case digit-like representations as shown in Figure 14) from CoOL, or, indeed, any clustering algorithm. The authors wish to acknowledge useful conversations with Eric Darve and Daniel Ratner. SLAC National Accelerator Laboratory is supported by the U.S. Department of Energy, Office of Science under Contract No. DE-AC02-76SF00515. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. FHO was responsible for the conceptualization, formal analysis, investigation, methodology, software, validation, visualization, and writing. MEM was responsible for funding acquisition, project administration, resources and supervision. The MNIST dataset is available here [33] and the CIFAR-10 dataset is available here [32]. The authors retrieved the data through torchvision [57]. References [1] J. Aalbers, D. S. Akerib, et al. (2024-08) The Data Acquisition System of the LZ Dark Matter Detector: FADR. arXiv (en). Note: arXiv:2405.14732 [physics]Comment: 18 pages, 24 figures External Links: Link, Document Cited by: §1. [2] G. Abla, G. Heber, D. P. Schissel, D. Robinson, L. Abadie, A. Wallander, and S. M. Flanagan (2014) ITERDB—the data archiving system for iter. Fusion Engineering and Design 89 (5), p. 536–541. Note: Proceedings of the 9th IAEA Technical Meeting on Control, Data Acquisition, and Remote Participation for Fusion Research External Links: ISSN 0920-3796, Document, Link Cited by: §1. [3] G. Andrew, R. Arora, J. Bilmes, and K. Livescu (2013) Deep Canonical Correlation Analysis. In Proceedings of the 30th International Conference on Machine Learning, Vol. 28(3), p. 1247–1255 (en). Cited by: §3. [4] M. Ankerst, M. M. Breunig, H. Kriegel, and J. Sander (1999-06) OPTICS: Ordering Points to Identify the Clustering Structure. ACM SIGMOD Record 28 (2), p. 49–60 (en). External Links: Document Cited by: §2. [5] E. Aprile (2024) The xenon program for dark matter direct detection. Nuclear Physics B 1003, p. 116463. Note: Special Issue of Nobel Symposium 182 on Dark Matter External Links: ISSN 0550-3213, Document, Link Cited by: §1. [6] J.C. Bernauer, C.T. Dean, et al. (2023-02) Scientific computing plan for the ECCE detector at the Electron Ion Collider. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 1047, p. 167859 (en). External Links: ISSN 01689002, Link, Document Cited by: §1. [7] P. Brans How to manage 2 petabytes of new data every day. Note: https://w.iter.org/node/20687/how-manage-2-petabytes-new-data-every-dayAccessed: 2025-12-09 Cited by: §1. [8] C. S. Bretherton, C. Smith, and J. M. Wallace (1992) An intercomparison of methods for finding coupled patterns in climate data. Journal of Climate 5 (6), p. 541 – 560. External Links: Document, Link Cited by: §3. [9] J. Cai, J. Fan, W. Guo, S. Wang, Y. Zhang, and Z. Zhang (2022-06) Efficient Deep Embedded Subspace Clustering. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, p. 21–30 (en). External Links: Link, Document Cited by: §3. [10] J. Chen, R. Zhang, J. Xu, C. Hu, and Y. Mao (2023-11) A Neural Expectation-Maximization Framework for Noisy Multi-Label Text Classification. IEEE Transactions on Knowledge and Data Engineering 35 (11), p. 10992–11003 (en). External Links: ISSN 1041-4347, 1558-2191, 2326-3865, Link, Document Cited by: §3. [11] C. Cortes and V. Vapnik (1995-09) Support-vector networks. Machine Learning 20 (3), p. 273–297 (en). External Links: ISSN 0885-6125, 1573-0565, Link, Document Cited by: §2. [12] A. P. Dawid and A. M. Skene (1979) Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics 28 (1), p. 20 (en). External Links: ISSN 00359254, Link, Document Cited by: §2, §4, §4. [13] A. P. Dempster, N. M. Laird, and D. B. Rubin (1977-09) Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society Series B: Statistical Methodology 39 (1), p. 1–22 (en). External Links: ISSN 1369-7412, 1467-9868, Link, Document Cited by: §1, §2, §4. [14] E. Elhamifar and R. Vidal (2013-02) Sparse Subspace Clustering: Algorithm, Theory, and Applications. arXiv (en). Note: arXiv:1203.1005 [cs] External Links: Link, Document Cited by: §3. [15] B. Gao, Y. Yang, H. Gouk, and T. M. Hospedales (2019-10) Deep clustering with concrete k-means. arXiv (en). Note: arXiv:1910.08031 [cs] External Links: Link Cited by: §3. [16] S. Goldenberg, K. Ahammed, A. Carpenter, J. Li, R. Suleiman, and C. Tennant (2025-04) Data-driven gradient optimization for field emission management in a superconducting radio-frequency linac. Physical Review Accelerators and Beams 28 (4), p. 044603 (en). External Links: ISSN 2469-9888, Link, Document Cited by: §5.2.2. [17] I. Goodfellow, Y. Bengio, and A. Courville (2016) Deep learning. MIT Press. Note: http://w.deeplearningbook.org Cited by: §7.2. [18] K. Greff, S. v. Steenkiste, and J. Schmidhuber (2017-11) Neural Expectation Maximization. arXiv (en). Note: arXiv:1708.03498 [cs]Comment: Accepted to NIPS 2017 External Links: Link, Document Cited by: §3. [19] X. Guo, L. Gao, X. Liu, and J. Yin (2017-08) Improved Deep Embedded Clustering with Local Structure Preservation. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne, Australia, p. 1753–1759 (en). External Links: ISBN 978-0-9992411-0-3, Link, Document Cited by: §3, §5.1, footnote 4. [20] D. Gupta and R. Ramjee (2020) UNSUPERVISED CLUSTERING USING PSEUDO-SEMI- SUPERVISED LEARNING. In Proceedings of 8th International Conference on Learning Representations, Addis Ababa, Ethiopia (en). External Links: Link Cited by: §3. [21] W. L. Hamilton, R. Ying, and J. Leskovec (2018-09) Inductive Representation Learning on Large Graphs. arXiv (en). Note: arXiv:1706.02216 [cs]Comment: Published in NIPS 2017; version with full appendix and minor corrections External Links: Link, Document Cited by: §5.2.1. [22] S. Hasan, M. Allaux, A. Biland, F. Gillardo, H. Li, M. Linhoff, E. Lyard, M. Pierre, V. Savchenko, and R. Walter (2025-07) Prototyping a Bulk Data Management System for CTAO with Rucio. In Proceedings of 39th International Cosmic Ray Conference, Vol. 501, Geneva, Switzerland, p. 672 (en). External Links: Document Cited by: §1. [23] T. Hastie, R. Tibshirani, and J. Friedman (2017) The elements of statistical learning: data mining, inference, and prediction. Springer. Note: 2nd edition Cited by: §2. [24] K. He, X. Zhang, S. Ren, and J. Sun (2015) Deep residual learning for image recognition. CoRR abs/1512.03385. External Links: Link, 1512.03385 Cited by: §7.3. [25] L. Hubert and P. Arabie (1985-12) Comparing partitions. Journal of Classification 2 (1), p. 193–218 (en). External Links: ISSN 0176-4268, 1432-1343, Link, Document Cited by: §5.1. [26] R. Humble, Z. Zhang, F. O’Shea, E. Darve, and D. Ratner (2024-09) Coincident learning for unsupervised anomaly detection of scientific instruments. Machine Learning: Science and Technology 5 (3), p. 035036 (en). External Links: ISSN 2632-2153, Link, Document Cited by: §3. [27] Z. Ivezic, S. M. Kahn, et al. (2019-03) LSST: from Science Drivers to Reference Design and Anticipated Data Products. The Astrophysical Journal 873 (2), p. 111 (en). Note: arXiv:0805.2366 [astro-ph]Comment: 57 pages, 32 color figures, version with high-resolution figures available from https://w.lsst.org/overview External Links: ISSN 0004-637X, 1538-4357, Link, Document Cited by: §1. [28] A. K. Jain, M.N. Murty, and P. J. Flynn (1999-09) Data Clustering: A Review. ACM Computing Surveys 31 (3), p. 264–323. External Links: Link, Document Cited by: §1. [29] G. James, D. Witten, T. Hastie, and R. Tibshirani (2013) An introduction to statistical learning: with applications in r. Springer. Cited by: §1, §5.1. [30] D. P. Kingma and J. Ba (2017) Adam: a method for stochastic optimization. External Links: 1412.6980, Link Cited by: §2. [31] A. Krizhevsky and G. Hinton (2009) Learning multiple layers of features from tiny images. Technical report Technical Report 0, Technical report, University of Toronto, University of Toronto, Toronto, Ontario. External Links: Link Cited by: §7.3. [32] A. Krizhevsky, V. Nair, and G. Hinton Note: https://w.cs.toronto.edu/~kriz/cifar.html, Accessed: 2025-12-16 External Links: Link Cited by: §7.4. [33] Y. LeCun, C. Cortes, and C. J. C. Burges (1998) The MNIST Database of Handwrit- ten Digits. External Links: Link Cited by: §1, §7.4. [34] Z. Li, Y. Chen, Y. LeCun, and F. T. Sommer (2022-01) Neural Manifold Clustering and Embedding. arXiv (en). Note: arXiv:2201.10000 [cs] External Links: Link Cited by: §3. [35] D. Liu, M. T. Vu, S. Chatterjee, and L. K. Rasmussen (2020-05) Neural Network based Explicit Mixture Models and Expectation-maximization based Learning. arXiv (en). Note: arXiv:1907.13432 [cs]Comment: IJCNN 2020 External Links: Link, Document Cited by: §3. [36] S. Luo, W. Yuan, E. Adelson, A. G. Cohn, and R. Fuentes (2018) ViTac: feature sharing between vision and tactile sensing for cloth texture recognition. External Links: 1802.07490, Link Cited by: §3. [37] J. Lv, Z. Kang, X. Lu, and Z. Xu (2021) Pseudo-supervised Deep Subspace Clustering. IEEE Transactions on Image Processing 30, p. 5252–5263 (en). Note: arXiv:2104.03531 [cs] External Links: ISSN 1057-7149, 1941-0042, Link, Document Cited by: §3. [38] B.B. McHarg, J.R. Burruss, J. Freeman, C.T. Parker, J. Schachter, and D.P. Schissel (2002-06) DIII-D data management. Fusion Engineering and Design 60 (3), p. 305–309 (en). External Links: ISSN 09203796, Link, Document Cited by: §1. [39] S.-K. Ng and G.J. McLachlan (2004-05) Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass Classification. IEEE Transactions on Neural Networks 15 (3), p. 738–749 (en). External Links: ISSN 1045-9227, Link, Document Cited by: §3. [40] C. Niu, H. Shan, and G. Wang (2022) SPICE: Semantic Pseudo-labeling for Image Clustering. IEEE Transactions on Image Processing 31, p. 7264–7278 (en). Note: arXiv:2103.09382 [cs] External Links: ISSN 1057-7149, 1941-0042, Link, Document Cited by: §3. [41] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer (2017) Automatic differentiation in pytorch. In NIPS-W, Cited by: Figure 10, §7.2. [42] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay (2011) Scikit-learn: machine learning in Python. Journal of Machine Learning Research 12, p. 2825–2830. Cited by: §5.1. [43] X. Peng, Y. Li, I. W. Tsang, H. Zhu, J. Lv, and J. T. Zhou (2022-04) XAI Beyond Classification: Interpretable Neural Clustering. arXiv (en). Note: arXiv:1808.07292 [cs]Comment: 28 pages External Links: Link Cited by: §3, §5.1. [44] A. Saxena, M. Prasad, A. Gupta, N. Bharill, O. P. Patel, A. Tiwari, M. J. Er, W. Ding, and C. Lin (2017-12) A review of clustering techniques and developments. Neurocomputing 267, p. 664–681 (en). External Links: ISSN 09252312, Link, Document Cited by: §1. [45] E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu (2017-09) DBSCAN Revisited, Revisited: Why and How You Should (Still) Use DBSCAN. ACM Transactions on Database Systems 42 (3), p. 1–21 (en). External Links: ISSN 0362-5915, 1557-4644, Link, Document Cited by: §2, §3. [46] N. Schwarz, S. Veseli, and D. Jarosz (2019) Data management at the advanced photon source. Synchrotron Radiation News 32 (3), p. 13–18. External Links: Document Cited by: §1. [47] U. Shaham, K. Stanton, H. Li, B. Nadler, R. Basri, and Y. Kluger (2018-04) SpectralNet: Spectral Clustering using Deep Neural Networks. arXiv (en). Note: arXiv:1801.01587 [stat]Comment: Added citations. Accepted to ICLR 2018 External Links: Link Cited by: §3, §5.1, §7.2. [48] C. E. Shannon (1948) A mathematical theory of communication. The Bell System Technical Journal 27 (3), p. 379–423. External Links: Document Cited by: §6. [49] J. Svirsky and O. Lindenbaum (2024-06) Interpretable Deep Clustering for Tabular Data. arXiv (en). External Links: Document Cited by: §3. [50] N. Tagasovska and D. Lopez-Paz (2019-12) Single-Model Uncertainties for Deep Learning. In Proceedings of 33rd Conference on Neural Information Processing Systems, Vancouver, Canada (en). Cited by: §5.2.2. [51] J. Thayer, Z. Chen, R. Claus, D. Damiani, C. Ford, M. Dubrovin, V. Elmir, W. Kroeger, X. Li, S. Marchesini, V. Mariani, R. Melcchiori, S. Nelson, A. Peck, A. Perazzo, F. Poitevin, C. P. O’Grady, J. Otero, O. Quijano, M. Shankar, M. Uervirojnangkoorn, R. Veraldi, M. Weaver, C. Weninger, S. Yamajala, C. Wang, and C. H. Yoon (2024) Massive Scale Data Analytics at LCLS-I. EPJ Web of Conferences 295, p. 13002 (en). External Links: ISSN 2100-014X, Link, Document Cited by: §1. [52] The ATLAS Collaboration (2009) Expected performance of the atlas experiment - detector, trigger and physics. External Links: 0901.0512, Link Cited by: §1. [53] D. Tissera, K. Vithanage, R. Wijesinghe, A. Xavier, S. Jayasena, S. Fernando, and R. Rodrigo (2022-09) Neural mixture models with expectation-maximization for end-to-end deep clustering. Neurocomputing 505, p. 249–262 (en). External Links: ISSN 09252312, Link, Document Cited by: §3. [54] A. Topchy, A.K. Jain, and W. Punch (2003) Combining multiple weak clusterings. In Third IEEE International Conference on Data Mining, Melbourne, FL, USA, p. 331–338 (en). External Links: ISBN 978-0-7695-1978-4, Link, Document Cited by: §3. [55] A. Topchy, A.K. Jain, and W. Punch (2005-12) Clustering ensembles: models of consensus and weak partitions. IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (12), p. 1866–1881 (en). External Links: ISSN 0162-8828, Link, Document Cited by: §3. [56] A. Topchy, A. K. Jain, and W. Punch (2004-04) A Mixture Model for Clustering Ensembles. In Proceedings of the 2004 SIAM International Conference on Data Mining, p. 379–390 (en). External Links: ISBN 978-0-89871-568-2 978-1-61197-274-0, Link, Document Cited by: §3. [57] TorchVision maintainers and contributors (2016) TorchVision: pytorch’s computer vision library. GitHub. Note: https://github.com/pytorch/vision Cited by: §7.3, §7.4. [58] S. Vega-Pons and J. Ruiz-Shulcloper (2011-05) A SURVEY OF CLUSTERING ENSEMBLE ALGORITHMS. International Journal of Pattern Recognition and Artificial Intelligence 25 (03), p. 337–372 (en). External Links: ISSN 0218-0014, 1793-6381, Link, Document Cited by: §3. [59] U. Von Luxburg (2007-12) A tutorial on spectral clustering. Statistics and Computing 17 (4), p. 395–416 (en). External Links: ISSN 0960-3174, 1573-1375, Link, Document Cited by: §2, §3. [60] I. H. Witten and E. Frank (2005) Data mining: practical machine learning tools and techniques. 2nd edition, Morgan Kaufmann. Cited by: §5.1, §6. [61] J. Xie, R. Girshick, and A. Farhadi (2016-05) Unsupervised Deep Embedding for Clustering Analysis. arXiv (en). Note: arXiv:1511.06335 [cs]Comment: icml2016 External Links: Link, Document Cited by: §5.1, §7.2. [62] B. Xu, N. Wang, T. Chen, and M. Li (2015-11) Empirical Evaluation of Rectified Activations in Convolutional Network. arXiv (en). Note: arXiv:1505.00853 [cs] External Links: Link, Document Cited by: §2. [63] J. Yang, D. Parikh, and D. Batra (2016-06) Joint Unsupervised Learning of Deep Representations and Image Clusters. arXiv (en). Note: arXiv:1604.03628 [cs]Comment: 19 pages, 11 figures, 14 tables, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) External Links: Link, Document Cited by: §3. [64] H. Yin, A. Aryani, S. Petrie, A. Nambissan, A. Astudillo, and S. Cao (2024-01) A Rapid Review of Clustering Algorithms. arXiv (en). External Links: Document Cited by: §1. [65] S. Zhou, H. Xu, Z. Zheng, J. Chen, Z. li, J. Bu, J. Wu, X. Wang, W. Zhu, and M. Ester (2022-06) A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions. arXiv (en). Note: arXiv:2206.07579 [cs]Comment: Github Repo: https://github.com/zhoushengisnoob/DeepClustering External Links: Link, Document Cited by: §3.

Instant research discovery