Paper deep dive
Cluster Validation Indices as Self-Supervised Objectives for Text Representation Learning
Kishor Kumar Bhaumik, Nicolas Roque dos Santos, Neil Shah, Jia Chen, Evangelos E. Papalexakis
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-supervised fine-tuning refines the embedding space of a pretrained language encoder without labels. However, the commonly used approaches are computationally expensive. Specifically, contrastive learning-based methods need multiview data and in-batch negative examples, while negative-free approaches require auxiliary graphs/networks. An interesting question arises: can self-supervised fine-tuning be done without relying on either additional negatives or graph data? To answer this question, we introduce SilK (Silhouette-guided K-means), which trains on a Cluster Validation Index, an internal measure of cluster quality without using labels. SilK clusters the corpus and then regresses a simplified silhouette toward a target value. Each document is compared only against the k cluster centroids, never against other documents, so the method needs no augmentation, no negative pairs and one view per document. On BERT-base, SilK trains 1.46x faster per epoch than the fastest baseline we evaluate and uses 45.4% less peak GPU memory than the leanest one. Under frozen-encoder linear probing, SilK stays competitive with the best baselines on three downstream tasks.
Tags
Links
- Source: https://arxiv.org/abs/2610.04830v1
- Canonical: https://arxiv.org/abs/2610.04830v1
Trouble viewing inline? Open PDF directly →
Full Text
25,524 characters extracted from source content.
Expand or collapse full text
Cluster Validation Indices as Self-Supervised Objectives for Text Representation Learning Kishor Kumar Bhaumik Nicolas Roque dos Santos Neil Shah Jia Chen Evangelos E. Papalexakis Abstract Self-supervised fine-tuning refines the embedding space of a pretrained language encoder without labels. However, the commonly used approaches are computationally expensive. Specifically, contrastive learning-based methods need multiview data and in-batch negative examples, while negative-free approaches require auxiliary graphs/networks. An interesting question arises: can self-supervised fine-tuning be done without relying on either additional negatives or graph data? To answer this question, we introduce SILK (Silhouette-guided K-means), which trains on a Cluster Validation Index, an internal measure of cluster quality without using labels. SILK clusters the corpus and then regresses a simplified silhouette toward a target value. Each document is compared only against the k cluster centroids, never against other documents, so the method needs no augmentation, no negative pairs and one view per document. On BERT-base, SILK trains 1.46×1.46× faster per epoch than the fastest baseline we evaluate and uses 45.4%45.4\% less peak GPU memory than the leanest one. Under frozen-encoder linear probing, SILK stays competitive with the best baselines on three downstream tasks. Our code is available at https://github.com/Kishor-Bhaumik/SILK. Index Terms: Self-supervised learning, cluster validation index, silhouette, BERT, representation learning. †address: 1University of California, Riverside 2Pinterest 1 Introduction and Related Work Text embeddings underpin retrieval, semantic search, clustering, and retrieval-augmented generation, where the evidence supplied to a language model is selected by the geometry of the embedding space [12]. Pretrained encoders such as BERT [4] carry rich contextual information, but masked language modeling, in general, does not force text embeddings to align with geometrical information, i.e., the resulting space is anisotropic and separates latent semantic categories poorly [5]. Self-supervised fine-tuning addresses this by reshaping the space with unlabeled text. Contrastive approaches such as SimCSE [6] and DiffCSE [3] build two views of each sentence via dropout masks or edit-based auxiliary tasks and contrast them against in-batch negatives, with later work [19] refining the augmentations and the sampling. Introducing negatives adds computational cost: pairwise comparison is quadratic in the batch and semantically related texts drawn as negatives are actively pushed apart. Negative-free methods instead prevent collapse architecturally, through feature decorrelation, asymmetry, or momentum encoders [7]; SCD [10] combines a self-contrastive term with a decorrelation penalty over two dropout views. Collapse avoidance then becomes an extra component to design and tune. This suggests a question that predates deep representation learning: can a classical clustering criterion serve directly as the self-supervised objective? Cluster Validation Indices (CVIs) score a partition by intra-cluster compactness and inter-cluster separation with no ground truth. Among these indices, the silhouette index [16] encodes exactly the preference contrastive learning expresses through pairs; same-cluster points close, different-cluster points far—but does so without constructing any pairs. CVIs have been used for five decades to evaluate clusterings. Using a CVI itself as a differentiable training loss is recent [18]. In the literature, clustering results have been used to guide training neural networks in self-supervised settings. For instance, DeepCluster [2] trains a CNN on k-means pseudolabels and graph methods use clustering to select positives or to optimize cluster quality directly [18]. CARL-G [18] trains a GNN with a simplified-silhouette loss, and uses a transductive GCN over a fixed graph generating all node embeddings in a single forward pass. However, neither method is directly applicable to text corpora as documents are independent sequences without explicit graph structure and the activation memory of a Transformer makes a full-corpus computation graph infeasible for corpora of any size. Therefore, it is worth investigating whether CVI-based learning transfers to text data. Figure 1: Overview of SILK. Documents x1,…,xnx_1,…,x_n are encoded by BERT and projected. k-means is fit on the resulting embeddings and the silhouette loss pulls each document toward its assigned centroid while pushing it from the nearest competing one. That CVI loss is backpropagated into BERT. Unlike contrastive methods, no augmented views or negative pairs are constructed. In light of this, we propose SILK (Silhouette-guided K-means). SILK first embeds an unlabeled corpus of n documents and applies k-means to obtain corpus-level centroids. It then optimizes a simplified silhouette, which replaces distances to other documents by distances to the k centroids. The computational cost therefore drops from (n2)O(n^2) pairwise distances to (nk)O(nk) embedding-to-centroid distances, with k≪nk n. To run this on a Transformer we separate the scope of the clustering from the scope of the computation graph, refreshing centroids once per epoch and accumulating gradients across batches into a single optimizer step. Our contributions are summarized below. • CVI loss. SILK optimizes cluster quality directly, contrasting each document against k corpus-level centroids instead of against other documents. It uses one view per document and requires no augmented views or negative pairs. • Feasible training scheme. A silhouette loss needs the whole corpus clustered but a Transformer can backpropagate only one batch at a time. We cluster the corpus once per epoch and accumulate gradients across batches, so peak activation memory scales with the batch rather than with the corpus. • Lower training cost. With a single view and no in-batch negatives, per-epoch time and peak GPU memory fall below every baseline we evaluate, while linear-probe accuracy stays competitive with the strongest. 2 Proposed Method Figure 1 shows the method. Documents are encoded by BERT, projected, and clustered with k-means. The CVI loss on that clustering is backpropagated to fine-tune the encoder. 2.1 Setup Given an unlabeled corpus =x1,…,xnD=\x_1,…,x_n\ of n documents, let fθf_θ be BERT-base with [CLS] pooling and gϕ:ℝ768→ℝdg_φ:R^768\!→\!R^d a 2-layer MLP projector with d≪768d 768. Let i=fθ(xi)h_i=f_θ(x_i) be the encoder output. The embedding of document i is i=gϕ(fθ(xi))‖gϕ(fθ(xi))‖2,i=1,…,n,z_i= g_φ(f_θ(x_i))\|g_φ(f_θ(x_i))\|_2, 20.00003pti=1,…,n, (1) Normalization keeps a few high-variance directions from dominating the distances, which matters because BERT features are anisotropic [5]. The projector is used only during fine-tuning. At evaluation we discard gϕg_φ and use the 768768-dimensional encoder features [7, 18]. Let =1,…,kC=\C_1,…,C_k\ be a k-means partition of ii=1n\z_i\_i=1^n, jc_j the centroid of jC_j, i+c_i^+ the centroid of the cluster containing iz_i and i−c_i^- the nearest other centroid. 2.2 Silhouette-Guided Objective The silhouette index [16] asks, for each point, whether it lies closer to its own cluster than to the nearest competing one, comparing the mean distance to fellow members (compactness) against the mean distance to the nearest other cluster (separation). This is the geometric preference contrastive learning encodes through pairs, obtained without pairs or labels. The computational cost grows quadratically with the number of documents, (n2)O(n^2), because all pairwise distances must be calculated and retained for backpropagation, which is untenable for a Transformer on tens of thousands of documents. We therefore use the simplified silhouette [8, 18], which replaces each set of points by the centroid summarizing it: a′(i)=‖i−i+‖2,b′(i)=‖i−i−‖2,a (z_i)=\|z_i-c_i^+\|_2, 20.00003ptb (z_i)=\|z_i-c_i^-\|_2, (2) where a′a is how far a document has drifted from its own centroid and b′b how close it has come to its nearest rival. The per-document score is s′(i)=b′(i)−a′(i)maxa′(i),b′(i)∈[−1,1],s (z_i)= b (z_i)-a (z_i) \a (z_i),b (z_i)\∈[-1,1], (3) near 11 when the document sits far nearer its own centroid than any other and negative when it would be better placed elsewhere. Each document is compared against k centroids rather than n−1n-1 neighbours, giving (nkd)O(nkd) time and (nk)O(nk) memory. What does optimizing Eq. (3) do to the encoder? In the common case a′<b′a <b we have s′=1−a′/b′s =1-a /b , whose partials are ∂s′/∂a′=−1/b′∂ s /∂ a =-1/b and ∂s′/∂b′=a′/b′2∂ s /∂ b =a /b 2. Raising the score therefore pulls the document toward its own centroid and pushes it from the nearest competing one. The push is weighted by a′/b′2a /b 2, so a document already tight on its centroid is repelled weakly, while one caught between two clusters is repelled strongly. The objective contrasts each document against a small set of corpus-level summaries rather than against individual documents, which is precisely why no negative sampling is required. We do not maximize the mean score. A silhouette of 11 is reached only when every cluster collapses to a point; since k-means inevitably misassigns some documents, maximization drives those onto the wrong centroid and destroys the within-cluster variation a downstream classifier needs [18]. We regress toward a bounded target τ∈[−1,1]τ∈[-1,1] where the outer sum runs over the NBN_B mini-batches of an epoch: ℒcvi=1NB∑ℬ|τ−1|ℬ|∑i∈ℬs′(i)|,L_cvi= 1N_B _B |τ- 1|B| _i s (z_i) |, (4) 2.3 Clustering under a Bounded Computation Graph CARL-G evaluates its objective in one forward pass per epoch [18], which a transductive GCN permits because all n node embeddings live in a single computation graph. However, Transformer activations scale as (nLTd)O(nLTd) in documents, layers, tokens, and width, placing a full-corpus graph orders of magnitude beyond device memory. Stepping per mini-batch is no remedy either: k-means would then be fit to a few hundred randomly drawn documents, leaving the centroids of Eq. (2) sample-dependent and the loss batch-local. We decouple the two scopes. An inference-mode pass builds no graph and produces the embeddings for GPU k-means [9], initialized with k-means++++ [1] in the first epoch and warm-started from the previous centroids thereafter, so the loss target does not jump when the representation has barely moved. A second pass recomputes embeddings with gradients enabled, evaluates Eq. (4) against the frozen centroids, and frees each batch’s graph before building the next; the accumulated gradients give one optimizer step per epoch. Every batch is scored against the same frozen centroids, so this step is computed under one consistent clustering, while peak activation memory falls from (n)O(n) to (|ℬ|)O(|B|). We pay one extra forward pass and the graph shrinks by a factor of n/|ℬ|n/|B| where |ℬ||B| is the batch size. This is what lets a silhouette objective run on a Transformer. 3 Experiments 3.1 Setup Protocol. All methods fine-tune bert-base-uncased on the unlabeled 20 Newsgroups [11] all subset (∼ 18,846 documents), with headers, footers and quotes removed. We fine-tune on documents rather than sentences so the corpus carries the topical structure a clustering objective can exploit. Afterwards we discard the projector, freeze the encoder and fit a linear classifier on its [CLS] features. No supervised gradient reaches the encoder, so the probe measures the geometry of the learned space rather than the capacity of downstream fine-tuning. Baselines and tasks. We compare against raw BERT (no fine-tuning), SimCSE [6] and DiffCSE [3] (contrastive), and SCD [10] (negative-free, non-contrastive). We probe on TREC-6 [13] (6 classes), Snippets [15] (8 classes), and StackOverflow [20] (20 classes). These span question types, web snippets, and technical short text, and none matches the fine-tuning domain, so the probe also measures transfer. All methods, including the baselines, are fine-tuned for 20 epochs. Hyperparameters. SILK uses target τ=0.5τ=0.5, a 2-layer MLP projector with output d=128d=128, and a maximum sequence length of 128128 tokens. The batch size is 128128. We train with AdamW at learning rate 10−510^-5 and weight decay 10−410^-4 for 2020 epochs under a cosine schedule with 1010 warmup epochs. GPU k-means runs for at most 100100 assignment and update steps and stops earlier if it converges. The first epoch tries 1010 initializations; later epochs warm-start from the previous centroids and use a single trial. We report the mean and standard deviation over 55 seeds, each covering both fine-tuning and linear probing. The number of clusters is chosen per downstream dataset by validation-split linear-probe accuracy (k=150k=150, 120120, and 200200; see Sec. 3.4). The probe trains a single linear layer with AdamW (lr 10−210^-2, weight decay 10−410^-4, batch 512512) for 1515 epochs, and we report final-epoch test accuracy. SimCSE DiffCSE SCD SILK Avoids negative pairs ✗ ✗ ✓ ✓ Augmentation-free ✗ ✗ ✗ ✓ Single view per document ✗ ✗ ✗ ✓ No extra anti-collapse term ✓ ✗ ✗ ✓ Table 1: What each objective requires. SILK satisfies all four properties by contrasting against centroids instead of documents. Method TREC-6 Snippets StackOverflow BERT 79.04±0.4079.04± 0.40 71.13±0.5071.13± 0.50 72.76±0.3072.76± 0.30 SimCSE 90.64±0.3090.64± 0.30 80.70±0.4080.70± 0.40 79.16±0.5079.16± 0.50 DiffCSE 88.15±0.6088.15± 0.60 80.67±0.3080.67± 0.30 76.18±0.4076.18± 0.40 SCD 89.15±0.5089.15± 0.50 79.35±0.4079.35± 0.40 77.28±0.6077.28± 0.60 SILK (ours) 92.47±0.4092.47± 0.40 82.04±0.3082.04± 0.30 78.24±0.5078.24± 0.50 Table 2: Linear-probe test accuracy (%, mean ± std over 5 seeds, each covering both fine-tuning and probing) on frozen features. All methods fine-tuned on 20 Newsgroups. 3.2 Downstream Accuracy Table 2 reports linear-probe accuracy. SILK improves over the strongest baseline by 1.831.83 points on TREC-6 and 1.341.34 on Snippets, and trails SimCSE by 0.920.92 on StackOverflow. Every fine-tuning method improves substantially on raw BERT, confirming that the pretrained space is poorly suited to linear separation and that the gains come from reshaping it rather than from the probe. The StackOverflow result is the informative one. It has 2020 classes of technical short text, the furthest of the three from the Usenet discussion corpus SILK was fine-tuned on, and it is also where a clustering objective has least to work with: with few tokens per document, the centroids defining a′a and b′b are estimated from sparse, overlapping topics. A loss built on cluster structure should give ground where that structure is weak, and it does. This also marks the boundary of where the method applies. 3.3 Training Cost Table 3 reports per-epoch time and peak memory under matched conditions. SILK trains 1.46×1.46× faster per epoch than SCD, the fastest baseline, and uses 45.4%45.4\% less memory than SimCSE, the leanest—despite spending an extra forward pass on clustering every epoch. The saving is structural rather than incidental. Contrastive baselines hold two views of every document in the graph and materialize a |ℬ|×|ℬ||B|×|B| similarity matrix, so their negatives live inside the batch. SILK keeps one view and contrasts against k centroids that are external to the batch and fixed for the epoch. This is also why the two respond differently to batching: shrinking the batch alters the InfoNCE [14] objective by removing negatives, whereas for SILK it changes only how finely the same corpus-level gradient is accumulated. Method Time/epoch (min) Peak GPU mem (GB) SimCSE 3.017 21.823 DiffCSE 4.200 33.766 SCD 2.578 22.536 SILK 1.766 11.917 Table 3: Self-supervised training cost per epoch on an RTX A6000, batch size 128, FP32. Time includes the corpus embedding pass and k-means as well as the gradient update. Memory is the mean per-epoch peak of the PyTorch allocator; linear probing is excluded. All methods run for the same number of epochs (20), so per-epoch cost is proportional to total cost. 3.4 Number of Clusters A natural guess is to set k to the number of downstream classes. We instead treat k as a hyperparameter and, for each downstream dataset, train separate models across a range of k. Figure 2 plots validation-split linear-probe accuracy for that search, and Table 2 uses the k at each peak. Accuracy climbs steeply well past the class count, and on all three tasks the best k is roughly an order of magnitude larger. Beyond that point the gains stop and the curves turn back down. Both ends of the range are easy to read. Too few clusters force unrelated topics onto a shared centroid, so a′a is minimized by pulling together documents that should stay apart. Too many split each topic into near-duplicates, leaving b′b set by a neighbour that differs from the document’s own cluster in name only, and the separation term stops carrying signal. The same preference for large k appears in clustering-based self-supervision for vision and graphs [2, 18], which suggests the clusters work as a partition of the representation space rather than a stand-in for the label set. In practice the region around each peak is broad, so k only needs to be set to the right order of magnitude. Figure 2: Validation-split linear-probe accuracy against the number of clusters k. Each point is a separate training run. The k used in Table 2 is the peak of this same validation-split search. The best k exceeds the downstream class count by an order of magnitude on all three tasks. 4 Limitations and Future Work We used a small corpus (∼ 18.8k documents) for clustering in our experiments. On a much larger corpus mini-batch or streaming k-means [17] would take its place. The centroids would then be approximate and whether the gains survive that approximation is the first thing to test. The method also depends on k, which must be chosen per corpus. Finally, the objective may be applied beyond text modality. The same recipe should carry over to images and other modalities. We leave that for future work. 5 Conclusion We presented SILK in this paper, which fine-tunes a language encoder using a Cluster Validation Index as its training loss. Each document is compared against cluster centroids alone and a single optimizer step per epoch is enough. To run it on a Transformer we clustered at the epoch level and accumulated gradients across batches, which keeps memory bounded while every update uses one consistent clustering. SILK trains faster and uses less memory than every baseline we evaluate and stays competitive on linear-probe accuracy. Cluster validation indices have long been used to judge representations; these results suggest they can also be used to learn them. Acknowledgements. The work was partially supported by the Agriculture and Food Research Initiative Competitive Grant no. 2020-69012-31914 from the USDA National Institute of Food and Agriculture, and by the National Science Foundation under grant no. 2431569, grant 2524228, and the CREST Center for Multidisciplinary Research Excellence in CyberPhysical Infrastructure Systems (MECIS) grant no. 2112650. The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agencies. Compliance with Ethical Standards. This work used publicly available benchmark datasets containing no personally identifiable information, and required no human or animal subjects. The authors declare no conflicts of interest. References [1] D. Arthur S. Vassilvitskii et al. (2007) K-means++: the advantages of careful seeding. In Soda, Vol. 7, p. 1027–1035. Cited by: §2.3. [2] M. Caron, P. Bojanowski, A. Joulin, and M. Douze (2018) Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), p. 132–149. Cited by: §1, §3.4. [3] Y. Chuang, R. Dangovski, H. Luo, Y. Zhang, S. Chang, M. Soljačić, S. Li, W. Yih, Y. Kim, and J. Glass (2022) DiffCSE: difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 4207–4218. Cited by: §1, §3.1. [4] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §1. [5] K. Ethayarajh (2019) How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 55–65. Cited by: §1, §2.1. [6] T. Gao, X. Yao, and D. Chen (2021) Simcse: simple contrastive learning of sentence embeddings. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 6894–6910. Cited by: §1, §3.1. [7] J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. (2020) Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33, p. 21271–21284. Cited by: §1, §2.1. [8] E. R. Hruschka, L. N. de Castro, and R. J. G. B. Campello (2004) Evolutionary algorithms for clustering gene-expression data. In Fourth IEEE International Conference on Data Mining (ICDM’04), p. 403–406. External Links: Document Cited by: §2.2. [9] J. Johnson, M. Douze, and H. Jégou (2019) Billion-scale similarity search with gpus. IEEE transactions on big data 7 (3), p. 535–547. Cited by: §2.3. [10] T. Klein and M. Nabi (2022) SCD: self-contrastive decorrelation of sentence embeddings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 394–400. Cited by: §1, §3.1. [11] K. Lang (1995) Newsweeder: learning to filter netnews. In Machine learning proceedings 1995, p. 331–339. Cited by: §3.1. [12] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, p. 9459–9474. Cited by: §1. [13] X. Li and D. Roth (2002) Learning question classifiers. In Coling 2002: The 19th international conference on computational linguistics, Cited by: §3.1. [14] A. v. d. Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §3.3. [15] X. Phan, L. Nguyen, and S. Horiguchi (2008) Learning to classify short and sparse text & web with hidden topics from large-scale data collections. In Proceedings of the 17th international conference on World Wide Web, p. 91–100. Cited by: §3.1. [16] P. J. Rousseeuw (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics 20, p. 53–65. Cited by: §1, §2.2. [17] D. Sculley (2010) Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web, p. 1177–1178. Cited by: §4. [18] W. Shiao, U. S. Saini, Y. Liu, T. Zhao, N. Shah, and E. E. Papalexakis (2023) Carl-g: clustering-accelerated representation learning on graphs. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 2036–2048. Cited by: §1, §1, §2.1, §2.2, §2.2, §2.3, §3.4. [19] X. Wu, C. Gao, L. Zang, J. Han, Z. Wang, and S. Hu (2022) Esimcse: enhanced sample building method for contrastive learning of unsupervised sentence embedding. In Proceedings of the 29th International Conference on Computational Linguistics, p. 3898–3907. Cited by: §1. [20] J. Xu, P. Wang, G. Tian, B. Xu, J. Zhao, F. Wang, and H. Hao (2015) Short text clustering via convolutional neural networks. In Proceedings of the 1st workshop on vector space modeling for natural language processing, p. 62–69. Cited by: §3.1.