Paper deep dive
D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentation
Elena Bueno-Benito, Mariella Dimiccoli
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/8/2026, 4:17:58 AM
Summary
The paper introduces D-CLOT, a method for unsupervised action segmentation that addresses the 'representation-prototype inconsistency' found in existing Optimal Transport (OT) based methods like CLOT. D-CLOT employs a graph-constrained module to preserve local neighborhood geometry and periodically re-anchors action prototypes to stabilized representation geometry. Two variants are proposed: D-CLOT, which updates prototypes using k-means, and D-CLOT_B, which updates them as OT barycenters. The method demonstrates improved performance on benchmarks like YTI, FS-Eval, and Assembly101.
Entities (11)
Relation Signals (10)
D-CLOT → buildson → CLOT
confidence 95% · To address this issue, we build on the recently introduced CLOT... D-CLOT extends this refinement to the action prototypes
D-CLOT → evaluatedon → Assembly101
confidence 95% · We further establish the first unsupervised action-segmentation baseline on Assembly101
D-CLOT → usescomponent → Graph-constrained module
confidence 95% · Specifically, we introduce a graph-constrained module that regularizes the OT-refined frame and segment representations
D-CLOT → usescomponent → Action-embedding refinement
confidence 95% · An action-embedding refinement step then periodically re-anchors the prototypes to this stabilized representation geometry.
D-CLOT → evaluatedon → YTI
confidence 90% · Across five established benchmarks... gains of up to +12.7 F1 and +10.2 mIoU (YTI)
D-CLOT → evaluatedon → FS-Eval
confidence 90% · activity-level gains of up to +8.9 F1 (FS-Eval)
Action-embedding refinement → purpose → Re-anchor prototypes
confidence 90% · periodically re-anchors the prototypes to this stabilized representation geometry.
Graph-constrained module → purpose → Preserve local neighborhood geometry
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Optimal transport (OT) has emerged as an effective framework for unsupervised action segmentation. Yet, in existing OT-based methods, the latent action prototypes that define the OT costs are not re-estimated from the refined frame geometry. Instead, they evolve solely through gradients from the pseudo-label loss. We identify this \emph{representation--prototype inconsistency} as a central bottleneck, particularly around ambiguous transitions and for short or infrequent actions. To address this issue, we build on the recently introduced CLOT, which refines frame embeddings based on estimated segment embeddings, and further re-estimates the action prototypes from the refined frame embeddings. Specifically, we introduce a graph-constrained module that regularizes the OT-refined frame and segment representations by preserving the local neighborhood geometry of the encoder output. An action-embedding refinement step then periodically re-anchors the prototypes to this stabilized representation geometry. We study two instantiations that share the same backbone, graph module, and objective: D-CLOT updates the prototypes using $k$-means, whereas D-CLOT$_{B}$ updates them as OT barycenters weighted by the refined transport plan, yielding an assignment-aware prototype update consistent with the current transport geometry. Across five established benchmarks, both variants improve segment-level quality over CLOT, with per-video gains of up to $+12.7$ F1 and $+10.2$ mIoU (YTI) and activity-level gains of up to $+8.9$ F1 (FS-Eval). We further establish the first unsupervised action-segmentation baseline on Assembly101, a procedural and substantially more fine-grained benchmark than those commonly used in prior work. Extensive ablations and sensitivity analyses demonstrate that the two refinement mechanisms are complementary and robust.
Tags
Links
- Source: https://arxiv.org/abs/2608.05877v1
- Canonical: https://arxiv.org/abs/2608.05877v1
Trouble viewing inline? Open PDF directly →
Full Text
90,080 characters extracted from source content.
Expand or collapse full text
D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentation Elena Bueno-Benito, Mariella Dimiccoli The authors are with the Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Barcelona, Spain (e-mail: ebueno, mdimiccoli@iri.upc.edu). Abstract Optimal transport (OT) has emerged as an effective framework for unsupervised action segmentation. Yet, in existing OT-based methods, the latent action prototypes that define the OT costs are not re-estimated from the refined frame geometry. Instead, they evolve solely through gradients from the pseudo-label loss. We identify this representation–prototype inconsistency as a central bottleneck, particularly around ambiguous transitions and for short or infrequent actions. To address this issue, we build on the recently introduced CLOT, which refines frame embeddings based on estimated segment embeddings, and further re-estimates the action prototypes from the refined frame embeddings. Specifically, we introduce a graph-constrained module that regularizes the OT-refined frame and segment representations by preserving the local neighborhood geometry of the encoder output. An action-embedding refinement step then periodically re-anchors the prototypes to this stabilized representation geometry. We study two instantiations that share the same backbone, graph module, and objective: D-CLOT updates the prototypes using k-means, whereas D-CLOTB updates them as OT barycenters weighted by the refined transport plan, yielding an assignment-aware prototype update consistent with the current transport geometry. Across five established benchmarks, both variants improve segment-level quality over CLOT, with per-video gains of up to +12.7+12.7 F1 and +10.2+10.2 mIoU (YTI) and activity-level gains of up to +8.9+8.9 F1 (FS-Eval). We further establish the first unsupervised action-segmentation baseline on Assembly101, a procedural and substantially more fine-grained benchmark than those commonly used in prior work. Extensive ablations and sensitivity analyses demonstrate that the two refinement mechanisms are complementary and robust. 111Code, checkpoints, and V-JEPA2 features for Breakfast and Assembly101 will be released. I Introduction Temporal action segmentation assigns an action label to each frame of an untrimmed video and underlies downstream tasks such as activity recognition and procedural video understanding [17, 37, 31]. Although fully supervised methods achieve the strongest performance, they require costly frame-level annotations [21, 52], motivating weakly supervised and fully unsupervised alternatives [57, 24, 54]. Unsupervised action segmentation aims to discover recurring actions and their temporal boundaries without manual labels. Classical approaches first learn frame representations and then group them into action segments through clustering or temporal modeling [27, 50, 30]. Figure 1: Overview of the proposed D-CLOT framework. Given precomputed frame embeddings X from untrimmed videos, the CLOT loop refines frame- and segment-level representations through three OT-based alignment stages, producing frame embeddings F, segment embeddings S, refined frame embeddings RF_R, and their associated pseudo-labels. D-CLOT extends this refinement to the action prototypes A, by computing their shift on a stabilized representation geometry, obtained through a graph regularization term. The blue boxes indicate learnable variables. A recent and effective line of work casts this joint problem as optimal transport (OT), aligning frame representations to latent action embeddings through a soft transport plan that doubles as a pseudo-label for self-training [28, 55, 3]. ASOT [55] removed the need for a known action order by enforcing temporal consistency directly in the transport problem. HVQ [44] improved short-action detection via hierarchical vector quantization. VASOT [3] added cross-video correspondences as regularisation. CLOT [9] unifies these threads through a multi-level cyclic mechanism: frame embeddings are aligned to action prototypes, a parallel decoder infers segment embeddings from the aligned sequence, and the resulting segment structure is propagated back to the frame domain via cross-attention before a final OT alignment. This cyclic formulation leaves an asymmetry unaddressed: while frame and segment representations are iteratively refined, the action embeddings A that define the transport costs at every stage are not re-estimated from this refined geometry. They are initialised once via k-means and thereafter evolve only through the classification gradient, so as OT refinement re-organises the frame representation space, the prototypes meant to organise it can fall behind, a representation–prototype inconsistency. This is most damaging around ambiguous temporal transitions, where refined embeddings of adjacent frames from different actions overlap, and for short or infrequent actions, whose limited transport mass is easily absorbed by dominant modes. Moreover, since the refinement stages optimise purely for the OT-induced classification loss, nothing prevents them from introducing associations between frames that were not neighbours in the original encoder geometry. We address this problem with D-CLOT, which leaves the CLOT backbone and its three-stage OT objective intact, while introducing two successive operations. First, it stabilizes the refined frame and segment representations through a graph-constrained regularizer that encourages their local neighbourhood structure to remain consistent with that of the encoder output. Second, it periodically re-anchors the action-embedding to this stabilized representation geometry, thereby mitigating the drift that can arise when they are updated solely through gradient descent. The proposed refinement is general and admits multiple prototype-update strategies. We study two instantiations that share the same backbone, graph module, and objective. D-CLOT refreshes A by applying k-means to the pooled stabilized representations, providing a lightweight, assignment-agnostic correction. In contrast, D-CLOTB updates each prototype as the OT barycenter of the stabilized representations, weighted by the soft mass of the refined transport plan, yielding an assignment-aware update consistent with the current transport geometry. Since the two variants differ only in their prototype re-estimation strategy, their comparison isolates the effect of assignment awareness on prototype refinement. We also broaden the empirical scope of the closed-loop OT paradigm as we move beyond the cooking and instructional domain with a new evaluation on Assembly101 [41], a large-scale, fine-grained procedural-activity dataset. We define a controlled subset over 15 toy-assembly categories and two camera views, and extract V-JEPA 2 features for it. With 1111–4242 fine-grained actions per activity, far more than Breakfast (BF) [26], YouTube Instructions (YTII)[2], 50Salads (FS) [45], or Desktop Assembly (DA)[28], this subset is a considerably harder testbed for unsupervised action segmentation. We evaluate D-CLOT on five datasets under activity- and video-level Hungarian matching, with an ablation isolating the graph module at the frame and refined-frame levels and the action-embedding refinement step, across both instantiations. The contributions of this paper are as follows: • We identify a representation-prototype inconsistency in CLOT: the latent action embeddings that define the OT costs are not re-estimated as the refined frame and segment representations evolve. • We propose an action-embedding refinement step that periodically re-estimates the latent prototypes from the refined frame embeddings RF^R and blends the resulting estimates with the current prototypes. Together with the graph constraint described below, this closes the representation–prototype loop without modifying the CLOT backbone or its training objective. • We introduce a graph-constrained regularization module that stabilizes the OT-refined frame and segment representations by preserving the local neighborhood structure of the encoder’s feature space. • We study two instantiations of the proposed refinement: D-CLOT, which uses a k-means-based prototype update, and D-CLOTB, which updates prototypes as OT barycenters weighted by the refined transport plan. Both variants achieve state-of-the-art performance on YTII [2], FS [45], DA [28], and BF [26]. • We establish the first unsupervised action-segmentation baseline on Assembly101 [41], a large-scale, procedural, fine-grained testbed. The following section reviews related work. Section I introduces the proposed approach and details its two instantiations. Section IV describes the validation protocol and discusses experimental results. Finally, Section V concludes the paper. I Related works I-A Temporal action segmentation Fully supervised action segmentation methods remain the most reliable but require costly data annotations [23, 5, 6, 36]. To improve scalability and practicality, research has increasingly shifted towards weakly-supervised [35, 43, 54, 56, 8] and unsupervised approaches [10, 18, 27, 15, 28, 30, 33, 40, 48, 50, 55, 44, 3], aiming to reduce reliance on labeled data while maintaining competitive segmentation performance. As the estimated clusters lack semantic labels, the evaluation process requires finding the Hungarian correspondence between the clusters and the actual action classes. The Hungarian matching can be performed for video-level segmentation [1, 10, 19, 33, 40], activity-level segmentation [18, 27, 28, 30, 48, 50, 55, 3], or for a global scope across an entire set of videos [18, 27, 30, 8]. Depending on the hierarchical level used, methods aim to improve segmentation through these correspondences. Unsupervised video-level action segmentation focuses on processing individual videos independently, without relying on predefined activity categories. Existing approaches can be broadly categorized into representation learning methods and clustering-based methods. Representation learning approaches aim to learn robust action features before applying a clustering algorithm. LSTM+AL [1] predicts future frames and assigns segmentation boundaries based on prediction errors, while TSA [10] introduces a contrastive learning framework that employs a triplet selection strategy. Clustering-based approaches directly segment videos using similarity metrics. While clustering has been underexplored in action segmentation, recent work on TW-FINCH [40] incorporates temporal proximity alongside semantic similarity for improved clustering. Similarly, ABD [19] detects action boundaries by measuring adjacent frame similarities, and OTAS [33] enhances boundary detection by incorporating object-centric features. Unsupervised activity-level action segmentation traditionally follows a two-step pipeline: first, learning action representations in a self-supervised manner, and then clustering the learned embeddings, typically assuming a predefined number of clusters. Classical methods strongly rely on temporal regularization to model the sequential nature of activities [27]. This idea has been further refined through encoder-decoder architectures, incorporating either visual reconstruction losses (VTE) [50] or discriminative embedding losses (UDE) [46] to enhance clustering performance. Other methods have framed the problem as a self-supervised learning task, where action prototypes are discovered via auxiliary classification objectives [18, 30]. For instance, CAD [18] introduced a framework that identifies action prototypes using an activity classification task, while ASAL [30] proposed a method that distinguishes between valid and invalid action orderings based on shuffled segment predictions. Recently, OT has emerged as a powerful tool for jointly learning action representations and pseudo-labels through self-training, enabling effective feedback between representation learning and clustering, while directly optimizing for action segmentation. TOT [28] introduced a temporal OT formulation that generates pseudo-labels from predicted cluster assignments. However, this approach assumes fixed action ordering across all videos and uniform label assignment, contradicting the natural variability and long-tailed distribution of action labels. UFSA [48] addressed these limitations by incorporating frame- and segment-level cues from transcripts, allowing for action permutations within activities and non-uniform label assignments. However, it still requires prior knowledge of an estimated action order to infer the segmentation. To overcome this constraint, ASOT [55] introduced an OT-based method capable of producing temporally consistent segmentations without any prior assumptions about action order. This makes it particularly suited for both pseudo-labeling and decoding. However, ASOT enforces a strong structural prior, limiting its ability to detect short-duration actions, which are critical in many real-world applications. More recently, HVQ [44] tackled this issue by using a hierarchical vector quantization-based approach, significantly improving short-action detection. However, its learned codebook representing action classes has limited generalization capabilities compared to ASOT, since it lacks explicit feedback between the representation and clustering. In a related direction, VASOT [3] extends the OT-based paradigm by jointly addressing video alignment and action segmentation, leveraging cross-video correspondences to regularize the learned action representations. To jointly address ASOT’s structural rigidity and HVQ’s lack of representation-clustering feedback, CLOT [9] introduced a closed-loop OT framework with a multi-level cyclic feature learning mechanism, in which frame and segment embeddings are iteratively refined through cross-attention and reconciled through three coupled OT problems. I-B Optimal transport for structured prediction. OT has become a key framework in machine learning for measuring distributional discrepancies, particularly in unsupervised clustering and representation learning [55, 28, 48, 9, 3]. Unlike traditional probability metrics, Wasserstein distance preserves the geometric structure of distributions, making it well-suited for structured prediction tasks [39, 29]. However, its computational complexity limits its scalability. To address this, projection-based OT (POT) methods, such as Sliced Wasserstein (SW) distance, efficiently approximate OT by projecting high-dimensional distributions onto lower-dimensional subspaces. SW distance has been successfully applied to point-cloud processing, color transfer, Gaussian Mixture Model learning, and domain adaptation, making it a practical alternative for large-scale applications [25, 38]. I-C Graph-constrained regularization and subspace clustering Subspace clustering assumes that high-dimensional samples lie in a union of low-dimensional subspaces. Under the self-expressiveness principle, each sample is reconstructed from other samples, yielding a coding matrix C with X=XCX=XC and diag(C)=0diag(C)=0, from which an affinity matrix is built for spectral clustering [49]. SSC [20] promotes sparse affinities, LSR [34] enforces dense, grouping-consistent codings, and later work adds temporal structure, noise robustness, and outlier modelling [32]. Such self-expressive models remain sensitive to the input representation: noisy or ambiguous features corrupt the learned affinity. Graph-constrained regularization mitigates this by letting an auxiliary representation adapt to the clustering objective while preserving the local neighbourhood geometry [22]. In human motion segmentation: Dimiccoli et al. [16] anchor jointly learned features and affinities to the original neighbourhood structure, while TVSH [53] imposes a graph total-variation penalty that keeps the embedding smooth within segments yet sharp at boundaries and refines it from the segmentation output. These methods refine a static representation and derive the clustering from it. We differ in two respects: 1) we graph-regularize OT-refined frame features to obtain reliable geometry for updating latent action prototypes, and 2) we close the representation–prototype loop, maintaining their mutual consistency throughout refinement. I-D Prototype and barycentric refinement Prototype refinement is widely used in unsupervised representation learning to reduce representation drift and improve cluster consistency. Methods such as DeepCluster [12] periodically update prototypes with k-means, while SwAV [13] integrates prototype assignments into online training through balanced clustering constraints. However, these methods are not designed for temporal action segmentation, where prototypes define the assignment geometry across temporally structured and duration-imbalanced actions. D-CLOT updates A using k-means centroids, providing a simple prototype correction aligned with the current representation space. D-CLOTB instead updates A as barycenters of the refined features weighted by the OT transport plan [14, 7]. This barycentric update is assignment-aware, since each frame contributes according to its soft OT mass, making the refined prototypes consistent with the current transport geometry. Figure 2: Diagram of the D-CLOT architecture. As in CLOT, an MLP encoder with feature dispatching produces frame embeddings F, a parallel decoder estimates segment embeddings S, and a cross-attention module propagates segment-level information back to the frame domain, yielding refined frame embeddings RF_R. Three OT problems are solved at the frame, segment, and refined-frame levels, producing the pseudo-label transport plans T, ST_S, and RT_R, respectively. D-CLOT further introduces a graph-constrained module that regularizes the refined representation by preserving the local neighborhood structure of the encoded feature space.Then the regularized representation is used to explicitly update the action embeddings A through an action-prototype refinement step. Two update variants are considered: a K-means refresh on RF_R and an OT barycentric update using both ~R F_R and the refined transport plan ~R T_R. The resulting update closes the representation–prototype loop by feeding the refined action embeddings back into the OT-based CLOT pipeline. I Methodology We introduce D-CLOT, illustrated in Fig 2, a graph-constrained action-refinement framework for unsupervised temporal action segmentation. Given a set of untrimmed videos, the goal is to infer frame-level action assignments without using frame-level labels, transcripts, or action-order constraints during training. The proposed method builds upon CLOT [9], which establishes a cyclic interaction between frame-level and segment-level representations through three optimal transport (OT) problems, and extends it by regularizing the refined frame geometry and explicitly updating the latent action embeddings from this refined representation. Sections I-B and I-E recall the CLOT backbone and OT formulation for completeness; Sections I-C and I-D introduce the graph-constrained regularization and action-embedding refinement modules, which constitute the contributions of this extension. I-A Problem Formulation Let :=Vbb=1BV:=\V^b\_b=1^B be a collection of B untrimmed activity videos. Each video VbV^b is a sequence of precomputed frame descriptors b=[1b;…;Nbb]∈ℝNb×DX^b=[x_1^b;…;x_N_b^b] ^N_b× D, where NbN_b is the number of sampled frames and D the input feature dimension. No frame-level labels, transcripts, or action-order constraints are available during training. Given a predefined number of latent action classes K, the objective is to infer ^b=[y^1b,…,y^Nbb] y^b=[ y^b_1,…, y^b_N_b], y^ib∈1,…,K y^b_i∈\1,…,K\, for every video b. The latent action space is parameterised by a shared action-embedding matrix =[1,…,K]⊤∈ℝK×dA=[a_1,…,a_K] ^K× d, where k∈ℝda_k ^d is the prototype of the k-th latent action, constrained to the unit sphere (‖k‖2=1\|a_k\|_2=1). Importantly, A is shared across all videos of an activity and must therefore capture action structure consistently across subjects, durations, and orderings. For readability, we omit the video superscript b whenever unambiguous. I-B Backbone D-CLOT retains the CLOT architecture as its backbone, mapping frame descriptors ∈ℝN×DX ^N× D into three progressively structured representations: frame embeddings F, segment embeddings S, and refined frame embeddings RF_R. It comprises an MLP encoder with feature dispatching, a query-based parallel segment decoder, and a cross-attention refinement module. Encoder with feature dispatching. The encoder maps X into frame embeddings =norm(MLP())∈ℝN×dF=norm(MLP(X)) ^N× d, where d is the latent dimension and norm(⋅)norm(·) denotes ℓ2 _2 normalisation. A feature-dispatching mechanism then injects information from the latent prototypes A, conditioning each frame on its affinity to the prototype space. For each frame embedding if_i and prototype embedding ka_k, the affinity is ϕ(k,i)=σ(bϕ+sϕk⊤i‖k‖2‖i‖2),φ(a_k,f_i)=σ\! (b_φ+s_φ\, a_k f_i\|a_k\|_2\,\|f_i\|_2 ), (1) where σ(⋅)σ(·) is the sigmoid and bϕ,sϕb_φ,s_φ are learnable scalars, and the embedding is updated through a prototype-conditioned residual i′=i+1K∑k=1Kϕ(k,i)k.f _i=f_i+ 1K _k=1^Kφ(a_k,f_i)\,a_k. (2) Each frame thus aggregates information from semantically related prototypes while preserving its descriptor through the residual. We write =norm(i′i=1N)F=norm(\f _i\_i=1^N) for the dispatched representation. The resulting frame embeddings are aligned with the prototypes by a temporally consistent OT problem over a frame-to-cluster cost matrix C, producing a soft frame-to-action assignment. Parallel segment decoder. A transformer decoder maps F into segment embeddings ∈ℝK′×dS ^K × d. Rather than autoregressive decoding [6], it uses a parallel strategy adapted from action anticipation [21], inferring all K′K segment hypotheses jointly and avoiding the error accumulation of sequential decoders. A set of learnable queries ∈ℝK′×dQ ^K × d, each a latent segment prototype [11], interact through self-attention to model inter-segment dependencies and through cross-attention with the encoded frame sequence to gather frame-level evidence, with K′≤K ≤ K. As at the frame level, the decoder is coupled with a temporally consistent OT problem over a segment-to-cluster cost matrix SC_S. Cross-attention refinement. The final component propagates segment-level information back to the frame domain. Given the frame embeddings F and the segment embeddings S, the refinement module computes a frame-to-segment attention map and updates the frame representation as R=+softmax(⊤τ⋅d)F_R=F+softmax\! ( FS τ· d )S (3) where τ controls the sharpness of the alignment, so each frame aggregates its most relevant segment embeddings, combining local evidence with higher-level temporal context. The refined embeddings RF_R are aligned with the prototype space through a third OT problem over a frame-to-segment-to-cluster cost matrix RC_R. The backbone thus forms a cyclic pipeline: frames are organised w.r.t. prototypes, segments are decoded from the frame sequence, and segment-level structure is projected back onto the frames. D-CLOT retains this pipeline and extends it as described next. I-C Graph-Constrained representation regularization Although RF_R incorporates segment-level information via cross-attention, the refined geometry may still inherit noise from the initial feature space, particularly near action boundaries and for rare or short-duration actions. Since OT pseudo-labels are sensitive to local feature geometry, when embeddings of temporally adjacent frames from different actions overlap, the transport plan assigns inconsistent soft labels that propagate errors into subsequent iterations. Therefore, we introduce a graph-constrained regulariser that encourages an auxiliary representation to preserve the local neighbourhood structure of the encoder anchor encF_enc, while still allowing the CLOT objective to reorganise the space according to the latent action structure. Graph affinity. Given a representation ∈ℝN×dZ ^N× d and a validity mask ∈0,1Nm∈\0,1\^N that zeros out padded frames, we define the normalised pairwise affinity =()∈ℝ+N×NG=S(Z) ^N× N_+ as Gij=wij∑(p,q)∈ℰwpq,wij=mimjexp(−1−cos(i,j)h), G_ij= w_ij _(p,q) w_pq,\,\;\;\;w_ij=m_i\,m_j\, \! (- 1- (z_i,z_j)h ), (4) where cos(⋅,⋅) (·,·) is the cosine similarity on ℓ2 _2-normalised rows, ℰE is the set of valid frame pairs, and h>0h>0 is a bandwidth. Diagonal entries are zero; optionally, a top-k sparsification retains the k largest weights per row, followed by symmetrisation. The matrix is finally normalised to unit total mass, so G encodes the local neighbourhood geometry of Z as an empirical joint distribution over frame pairs [22]. Graph-constrained loss. Let encF_enc, the pre-dispatching encoder output, be a fixed anchor capturing the geometry of the input space with 0=(enc)G^0=S(F_enc) be the affinity of the fixed anchor, and and =()G=S(Z) that of an auxiliary representation ∈,RZ∈\F,\,F_R\. The regulariser minimises the cross-entropy between the two induced affinity distributions, ℒG(enc,)=−∑i,jGij0log(Gij+ϵ),L_G(F_enc,Z)=- _i,jG^0_ij\, \! (G_ij+ε ), (5) where 0G^0 and G are computed from the auxiliary representation via (4) and ϵ>0ε>0 is a numerical stabiliser. The anchor encF_enc is detached from the computation graph, so gradients flow only through Z. Minimising (5) discourages the OT refinement stages from forming inter-action connections absent in the encoder geometry; because cross-entropy is strictly convex in the affinity distribution, the fidelity term admits a unique minimiser. By default, the fidelity term is imposed as a soft penalty added to the CLOT objective (Section I-F), leaving F and RF_R as the representations passed to the OT solver. I-D Action-embedding refinement The graph module stabilises the refined frame geometry, but the action embeddings A are still updated only through gradient descent on the CLOT objective. As the OT stages reorganise the feature space, A can fall out of alignment with the stabilised geometry it is meant to organise. We therefore introduce an epoch-level action-embedding refinement that periodically re-anchors A to the graph-stabilised representation. This leaves the CLOT architecture and the per-iteration gradient updates untouched: it is an outer loop that proposes an updated prototype set and blends it into A. To preserve cluster identity and avoid abrupt shifts, both variants apply a conservative, normalised blend rather than a hard replacement. Given a proposal A, the update is ←norm((1−β)+β^),A \! ((1-β)\,A+β\, A ), (6) with blend factor β∈(0,1]β∈(0,1] and row-wise ℓ2 _2 normalisation. The two configurations differ only in how the proposal A is formed, and correspondingly in how the graph constraint of Section I-C is imposed: D-CLOT uses an assignment-agnostic k-means proposal on penalty-regularised features, while D-CLOTB uses an assignment-aware barycentric proposal on an explicitly graph-stabilised representation. D-CLOT: k-means refinement. In the first variant, we define ℱ=(Rb)i:b=1,…,B,i=1,…,NbF= \(f_R^b)_i:b=1,…,B,\;i=1,…,N_b \ as the refined frame features pooled over all videos of the activity. The video index b reappears here by necessity: the prototypes K are shared across the activity and are therefore re-estimated from the entire collection rather than a single video. The proposal is ^←KMeans(ℱ,K). A\;←\;KMeans\! (F,\,K ). (7) blended into A via (6). This update is assignment-agnostic: it uses no transport plan and no assumption beyond K. Its stability therefore rests entirely on the graph penalty (5) having kept RF_R close to the encoder geometry, a dependence confirmed by the ablation in Section IV-D. D-CLOTB: barycentric refinement. The barycentric proposal requires, for each frame, both a stabilised feature and a matched soft assignment. Rather than reading the penalty-regularised RF_R, this configuration enforces the graph constraint (5) through an explicit auxiliary variable. We solve a short inner problem ~R=argminλGℒG(enc,)+λa‖−R‖F2, F_R= _V\; _G\,L_G(F_enc,V)+ _a\, \|V-F_R \|_F^2, (8) where ∥⋅∥F\|·\|_F is the Frobenius norm, approximated by a few ℓ2 _2-normalised projected-gradient steps and used in place of RF_R for decoding and OT. The two terms play opposing roles: the fidelity term pulls the neighbourhood geometry of V towards the encoder anchor, while the alignment term keeps V close to the CLOT refined representation, so that λG/λa _G/ _a directly controls how much the stabilised representation is allowed to depart from the OT-driven one. During training, the inner steps are differentiably unrolled, so gradients reach the network; at inference, they run without backpropagating to the network parameters. This yields an explicit graph-stabilised representation ~R F_R and its refined transport plan ~R T_R. Each prototype is then updated as the OT barycenter of the stabilised features, weighted by their soft assignment mass. Collecting the stabilised features (~Rb)i( f_R^b)_i and the assignment entries (~Rb)i,k( T_R^b)_i,k over the epoch, we define the total soft mass of action k as πk=∑b∑i(~Rb)i,k _k= _b _i( T_R^b)_i,k and set ^k=norm(∑b∑i(~Rb)i,k(~Rb)iπk),πk>ε,k,otherwise, A_k= casesnorm\! ( _b _i( T_R^b)_i,k\,( f_R^b)_i _k ),& _k> ,\\[6.02773pt] a_k,&otherwise, cases (9) before blending via (6), where ε>0 >0 guards the denominator against actions that receive no transport mass in a given epoch, and it is fixed to 10−810^-8. Unlike the k-means update, the barycentric update is assignment-aware: each frame contributes to a prototype in proportion to its OT confidence, so the prototype tracks the current transport geometry, and frames with low mass contribute negligibly. This is what protects the long tail. A rare action whose frames are assigned with low confidence exerts a correspondingly small pull on its prototype, so the prototype is neither dragged towards a dominant mode nor displaced by ambiguous boundary frames. Both variants share the same backbone, the same graph-fidelity term (5), and the same blended, normalised update (6). They differ in how the constraint is applied and in the resulting prototype proposal: D-CLOT imposes the graph term as a penalty and refreshes A by k-means over the refined features, whereas D-CLOTB enforces it through an explicit re-optimisation and updates A by an assignment-aware barycenter of the resulting ~R F_R. Comparing them therefore isolates the contribution of assignment-awareness while holding everything else fixed. I-E Optimal transport formulation For completeness we recall the unbalanced OT formulation of ASOT [55], which underlies each of the three OT problems (,S,RT,T_S,T_R) in the pipeline; we refer to [55, 9] for the full derivation. KOT. Given histograms μ=1NNμ= 1N1_N and ν=1KKν= 1K1_K and a cost matrix ∈ℝ+N×KC ^N× K_+, the Kantorovich problem [47] seeks the minimum-cost coupling ∗=min∈ℱKOT(,),ℱKOT(,):=⟨,⟩,T^*= _T \ F_KOT(C,T), _KOT(C,T):= ,T , (10) over the transportation polytope :=∈ℝ+N×K:K=μ,⊤N=νT:=\T _+^N× K:T1_K=μ,\ T 1_N=ν\. Here T is a soft assignment between frames and actions. GW. The Gromov-Wasserstein term encodes structural priors such as temporal consistency by comparing intra-domain costs v∈ℝN×NC^v ^N× N (frames) and a∈ℝK×KC^a ^K× K (actions): ℱGW(v,a,):=∑i,k∈[N]j,l∈[K]L(Cikv,Cjla)TijTkl,F_GW(C^v,C^a,T):=\!\! _ subarrayci,k∈[N]\\ j,l∈[K] subarray\!\!L(C^v_ik,C^a_jl)\,T_ij\,T_kl, (11) with L:ℝ×ℝ→ℝL:R×R penalising deviations between cost entries. The two terms are fused as ℱFGW:=αℱGW+(1−α)ℱKOTF_FGW:=α\,F_GW+(1-α)\,F_KOT for α∈[0,1]α∈[0,1]. Unbalanced relaxation. The action (column) marginal is relaxed into a soft KL penalty, allowing the long-tailed action distributions typical of untrimmed video: min∈pℱFGW(,)+λDKL(⊤N∥ν), _T _p\ F_FGW(C,T)+λ\,D_KL\! (T 1_N\, \|\,ν ), (12) where p:=∈ℝ+N×K:K=μT_p:=\T _+^N× K:T1_K=μ\ keeps only the frame marginal, and λ>0λ>0 controls how strongly the action marginal ⊤NT 1_N is pulled towards ν: larger λ favours uniform usage, smaller λ permits long-tailed distributions. Cost matrices. The stage-wise costs use sw,v,a\C^sw,C^v,C^a\ for the frame and refined-frame problems and k,s,a\C^k,C^s,C^a\ for the segment problem. Except for swC^sw, they follow [55]: v,aC^v,C^a penalise assigning temporally adjacent frames (|i−k|≤Nr,i≠k|i-k|≤ N_r,\ i≠ k) to different actions (j≠lj≠ l), with no penalty outside the radius NrN_r or when adjacent frames share an action. The visual cost is ijk=i⊤j‖i‖‖j‖−ρZij,Zij=|iN−jK|,C^k_ij= x_i a_j\|x_i\|\,\|a_j\|-ρ\,Z_ij, Z_ij= | iN- jK |, (13) where Z is the temporal prior of [28, 55] regularising the coupling towards a banded-diagonal structure, weighted by ρ≥0ρ≥ 0. Sliced-Wasserstein cost. For swC^sw we complement the cosine cost with the Sliced-Wasserstein (SW) distance [25, 38]. The p-Wasserstein distance between measures μ,νμ,ν is Wp(μ,ν)=(infγ∈Π(μ,ν)∫ℝd×ℝdℓp(x,y)γ(x,y))1/p,W_p(μ,ν)= ( _γ∈ (μ,ν) _R^d×R^d ^p(x,y)\,dγ(x,y) )^1/p, (14) estimated empirically by projecting onto M random directions θmm=1M\ _m\_m=1^M on d−1S^d-1, SWDp(i,j)=(1M∑m=1Mℓ(Rθm#i,Rθm#j))1/p,SWD_p(x_i,a_j)= ( 1M _m=1^M (R_ _m\#x_i,R_ _m\#a_j ) )^1/p, (15) with ℓ the quadratic loss. We use p=1p=1, reducing the SW distance to M one-dimensional OT problems with closed-form solutions [39], and set ijsw=1+SWD(i,j)−ijk.C_ij^sw=1+SWD(x_i,a_j)-C^k_ij. (16) The SW cost is used at the frame and refined-frame levels; the simplified variant omits it, using kC^k alone. I-F Training objective and inference We train in a fully unsupervised manner by minimising the cross-entropy between predicted assignment probabilities and OT pseudo-labels at all three levels. For video b, the frame-to-action probabilities are Pijb=softmax(b⊤τ)ijP^b_ij=softmax( F^bA τ)_ij with temperature τ>0τ>0, and the masked cross-entropy is ℒ(,)=−1B∑b=1B∑i=1N∑j=1KTijblogPijb,,L(T,P)=- 1B _b=1^B _i=1^N _j=1^KT^b_ij P^b_ij,, (17) The segment and refined-frame probabilities S,RP_S,P_R are defined analogously against S,RT_S,T_R, giving the three-stage CLOT loss ℒCLOT=ℒ(,)+ℒ(S,S)+ℒ(R,R)L_CLOT=L(T,P)+L(T_S,P_S)+L(T_R,P_R). The graph regulariser enters the objective only under the penalty mechanism (D-CLOT), giving the per-iteration loss ℒD−CLOTtrain=ℒCLOT+λGFℒG(enc,)+λGRℒG(enc,R),L^train_D-CLOT=L_CLOT+ _G^F\,L_G(F_enc,F)+ _G^R\,L_G(F_enc,F_R), (18) with λGF,λGR≥0 _G^F, _G^R≥ 0 weighting the frame- and refined-frame-level regularisation. Under the auxiliary-variable mechanism (D-CLOTB) the fidelity term is instead consumed inside the inner problem (8), and the outer objective reduces to ℒCLOTL_CLOT evaluated on the graph-stabilised representations, ~ F and ~R F_R. At inference, we obtain RF_R. Under the auxiliary-variable mechanism, this representation is graph-stabilised via the same inner steps used during training, without backpropagation. The final segmentation solves the refined OT problem between RF_R (or ~R F_R) and A, assigning each frame y^i=argmaxk(R)i,k, y_i= _k(T_R)_i,k, with RT_R replaced by ~R T_R for D-CLOTB. IV Experiments IV-A Experimental setting Implementation details The backbone is unchanged from CLOT [9] (Section I-B). Optimisation uses Adam with weight decay 10−410^-4 and a learning rate around 10−310^-3. Action embeddings A are initialised by k-means, and the number of action clusters K is set to the ground-truth action count of each dataset, consistent with prior work [44, 55, 3, 9]. Videos are sampled to 256256 frames per training iteration, except on Assembly101 [41], where the longer average duration motivates a budget of 512512 frames. Training alternates gradient-based optimisation of (18) with periodic re-estimation of A: the prototypes are refreshed by k-means (7) for D-CLOT or by the OT barycenter (9) for D-CLOTB, and blended into the current prototypes via (6) rather than replaced outright. Neither module introduces learnable parameters, so both variants have the same parameter count as CLOT. Detailed hyperparameter settings are reported in supp.mat. Datasets and Features. We evaluate our approach on five video datasets covering cooking, instructional, and assembly-oriented procedural activities. Following standard practice in unsupervised action segmentation, we rely on pre-extracted visual representations to ensure a fair comparison with prior work [48, 27, 28, 44, 55, 18]. Table I reports the principal statistics of each dataset. • Breakfast (BF) [26] contains approximately 1,7001,700 videos of subjects preparing breakfast dishes, organized into 1010 activities and 4848 fine-grained action classes (e.g., cracking an egg or pouring flour). Clip duration ranges from about 3030 s to several minutes. We use the standard Fisher-vector encoding of improved dense trajectories (IDT) [51]. • YouTube Instructions (YTI) [2]comprises 150150 instructional videos from 55 activities, each lasting roughly two minutes. It is particularly challenging because background frames dominate the footage (approximately 75%75\%). We represent each frame by the concatenation of HOF descriptors and VGG16 conv5 features [42]. TABLE I: Statistics of datasets used in the experiments. Background means the % of background frames in a dataset. Statistic BF YTII FS (mid) FS (eval) DA A101 #Videos 1,712 149 50 50 76 200 Avg. #Frames/video 2,097.49 516.60 11,551.90 11,551.90 778.49 13,047.81 Minimum 130 95 7,555 7,555 611 4,429 Maximum 9,741 2,320 18,143 18,143 1,154 55,248 Feature Dim. 64 3,000 64 64 512 1,408 #Activities (V) 10 5 1 1 1 15 Avg. #Actions/video 5 9 18 17 22 15 Background 12.1% 61.9% 14.4% 14.4% 2.7% 42.1% • 50 Salads (FS) [45] includes 5050 salad-preparation videos totaling 4.54.5 hours. We report results under the two standard annotation granularities: Mid, with 1919 action classes, and Eval, with 1212 coarser classes. We use Fisher vector features extracted from IDT [51]. • Desktop Assembly (DA) [28] contains 7676 videos, each of about 1.51.5 min, in which subjects assemble a desktop setup following a fixed procedure of 2222 temporally ordered actions. We use the features released by Tran et al. [48]. • Assembly101 [41] is a large-scale multi-view dataset for toy assembly and disassembly. The full dataset comprises 4,3214,321 videos, with an average duration of approximately 7.17.1 minutes and a total duration of about 513513 hours. It covers 1515 toy categories and provides annotations for 202202 coarse action classes. Since the full benchmark contains a large number of videos, viewpoints, and fine-grained action instances, we define a controlled subset tailored to unsupervised action segmentation. Specifically, we use the coarse action annotations, restrict the data to the assembly task, and evaluate the two locally available views, view1 and view2. The resulting subset contains 200200 view-specific videos from the local train–validation split and spans all 1515 toy categories. The number of observed coarse actions per toy category ranges from 1111 to 4242, from jackhammer to excavator. We extract V-JEPA2 features [4] and treat each view-specific recording as an independent video instance222The pretrained weights used in this work are available at huggingface.co/facebook/vjepa2-vitl-fpc64-256.. TABLE I: Comparisons of action segmentation performance obtained by applying the Hungarian matching per video and at the activity-level ”Full” on the Breakfast [26], Youtube Instr. [2], 50Salads [45] and Desktop Assembly [28] benchmarks. The highest accuracy is in bold, and the second highest is underlined. D-CLOT refreshes action embeddings with K-means on refined frame embeddings, while D-CLOTB updates action embeddings as barycenters of graph-constrained refined features. Methods Breakfast YTI 50Salads (Mid) 50Salads (Eval) DA MoF F1 mIoU MoF F1 mIoU MoF F1 mIoU MoF F1 mIoU MoF F1 mIoU Per video TWF* [40] 62.7 49.8 42.3 56.7 48.2 - 66.8 56.4 48.7 71.7 - - 73.3 67.7 57.7 ABD* [19] 64.0 52.3 - 67.2 49.2 - 71.8 - - 71.2 - - - - - OTAS* [33] 67.9 - - 65.7 - - 72.4 - - 73.5 - - - - - TSA* [10] (kmeans) 63.7 58.0 53.3 59.7 55.3 - - - - - - - - - - ASOT [55] 63.3 53.5 35.9 71.2 63.3 47.8 64.3 51.1 33.4 64.5 58.9 33.0 73.4 68.0 47.6 CLOT [9] 66.3 55.9 37.1 69.3 60.8 48.2 69.4 63.8 45.0 64.6 69.7 42.5 73.5 75.2 52.4 D-CLOT 65.7 58.1 38.6 74.9 73.5 58.4 70.5 70.0 49.2 67.9 71.9 43.7 73.7 79.0 54.9 D-CLOTB 67.5 59.0 38.6 71.6 65.9 49.7 69.4 66.2 46.1 68.7 75.6 44.7 74.5 79.2 55.2 Full CTE [27] 41.8 26.4 - 39.0 28.3 - 30.2 - - 35.5 - - 47.6 44.9 - VTE [50] 48.1 - - - 29.9 - 24.2 - - 30.6 - - - - - UDE [46] 47.4 31.9 - 43.8 29.6 - - - - 42.2 34.4 - - - - ASAL [30] 52.5 37.9 - 44.9 32.1 - 34.4 - - 39.2 - - - - - TOT [28] 47.5 31.0 - 40.6 30.0 - 31.8 - - 47.4 42.8 - 56.3 51.7 - TOT+ [28] 39.0 30.3 - 45.3 32.9 - 34.3 - - 44.5 48.2 - 58.1 53.4 - UFSA [48] 52.1 38.0 - 49.6 32.4 - 36.7 30.4 - 55.8 50.3 - 65.4 63.0 - ASOT [55] 56.1 38.3 18.6 52.9 35.1 24.7 46.2 37.4 24.9 59.3 53.6 30.1 70.4 68.0 45.9 HVQ [44] 54.4 39.7 - 50.3 35.1 - - - - - - - - - - VASOT [3] 57.5 39.0 18.8 53.2 35.7 25.2 47.2 41.3 26.1 60.6 57.4 34.5 70.9 75.1 49.3 CLOT [9] 60.1 40.1 18.5 54.4 36.7 23.4 50.6 46.6 31.4 59.4 63.2 38.8 68.8 72.6 48.1 D-CLOT 59.4 40.5 19.0 55.7 37.9 27.3 51.0 47.8 32.8 62.7 65.3 39.1 68.7 76.4 50.5 D-CLOTB 58.5 41.3 19.1 54.9 36.8 26.4 50.8 47.0 32.0 64.9 72.1 40.4 69.7 76.2 51.8 TABLE I: Unsupervised action segmentation results on Assembly101 [41]. Results are reported using V-JEPA2 features [4] over 15 toy-category activities, with Hungarian matching computed at the activity level and per video. Assembly101 Method Video-level Activity-level Avg. MoF F1 mIoU MoF F1 mIoU CLOT [9] 53.1 42.2 21.8 26.1 11.5 4.8 26.6 D-CLOT 49.5 54.7 37.9 24.8 15.3 10.3 32.1 D-CLOTB 49.0 56.5 40.4 24.7 16.2 9.5 32.7 Metrics. We follow standard evaluation protocols for unsupervised action segmentation [17]. Since predicted labels are unordered clusters, they are aligned with ground-truth action labels using Hungarian matching. We report two complementary settings: video-level matching, where the alignment is computed independently for each video [40, 19, 10], and activity-level matching, where a single alignment is computed over all videos of the same activity category [48, 27, 28, 50, 44, 55]. For Assembly101 [41], where the proposed benchmark is defined over selected toy categories, we analogously report toy-level matching over the selected subset. We report three standard metrics: Mean over Frames (MoF), which measures frame-wise accuracy and is sensitive to dominant actions; F1 Score [27], which evaluates segment-level agreement; and mean Intersection over Union (mIoU), which averages class-wise overlap and provides a more balanced assessment under class imbalance. Since MoF is less sensitive to short and rare actions, F1 and mIoU are the primary metrics for evaluating this failure mode. IV-B State-of-the-art comparisons Table I compares D-CLOT and D-CLOTB against prior unsupervised methods under both per-video and activity-level Hungarian matching. Both differ only in how the action prototypes are re-estimated once the frame and segment representations have been refined: D-CLOT applies k-means to the stabilized embeddings (assignment-agnostic), whereas D-CLOTB updates each prototype as an OT barycenter weighted by the refined transport plan (assignment-aware). Per-video matching. D-CLOT attains the best F1 and mIoU on YTI and FS-Mid, improving over CLOT by +12.7+12.7 F1 and +10.2+10.2 mIoU on YTI, and by +6.2+6.2 F1 and +4.2+4.2 mIoU on FS-Mid. D-CLOTB provides the strongest gains on FS-Eval and DA, increasing F1 by +5.9+5.9 and +6.3+6.3 points, respectively, and mIoU by +2.2+2.2 and +4.3+4.3 points. On YTI, both variants also improve MoF, so the gain there is not a trade-off but a uniform improvement. Both variants surpass methods that train directly on the target videos (TWF*, ABD*, OTAS*, TSA*) on most metrics, without requiring any per-video fine-tuning. Full matching. The same pattern holds under the harder activity-level protocol, and on F1 and mIoU both variants improve on all OT-based baselines across every dataset. D-CLOT leads on YTI (+1.3+1.3 MoF, +1.2+1.2 F1, +3.9+3.9 mIoU over CLOT) and FS-Mid (+0.4+0.4 MoF, +1.2+1.2 F1, +1.4+1.4 mIoU), while D-CLOTB is best on FS-Eval (+5.5+5.5 MoF, +8.9+8.9 F1, +1.6+1.6 mIoU) and attains the top DA mIoU, alongside D-CLOT’s best DA F1. On Breakfast, both variants trade a small MoF decrease for higher segment-level scores: D-CLOTB improves F1 by +1.2+1.2 and mIoU by +0.6+0.6, while D-CLOT yields smaller gains (+0.4+0.4 F1, +0.5+0.5 mIoU) with a smaller MoF drop (−0.7-0.7). This trade-off is expected: MoF is dominated by long-duration actions, whereas F1 and mIoU reward class-balanced, segment-level agreement, precisely what prototype re-anchoring improves near ambiguous transitions and for short or infrequent actions. Overall, the proposed variants achieve the best or second-best results on most datasets and metrics. The choice between variants is dataset-dependent: D-CLOT tends to benefit datasets with sharper cluster structure (YTI, FS-Mid), while the assignment-aware update of D-CLOTB is more effective on datasets with more procedurally consistent, imbalanced action durations (BF, FS-Eval, DA). TABLE IV: Ablation Study on the four datasets: Breakfast [26], YTI [2], 50Salads [45] and Desktop Assembly [28]. GC denotes the graph-constrained term, applied at the frame level (F), the refined-frame level (FRF_R), or both; A A denotes the periodic re-estimation of the prototypes. The top row (CLOT [9]) is the baseline. Within each variant block, the highlighted row is the full model (GC at both levels and refinement); the other rows toggle one factor at a time. Rows labelled ”only GC in ⋅·” activate the graph term at the indicated level(s) without refining the prototypes, thus isolating GC alone. Rows labelled ”w/o GC ⋅·” keep refinement active but remove the graph term at the indicated level. Both full models improve over CLOT on most metrics, and the full D-CLOTB is the only configuration that surpasses all its ablated variants on every dataset and metric. Breakfast YTI 50Salads (Mid) 50Salads (Eval) DA MoF F1 mIoU MoF F1 mIoU MoF F1 mIoU MoF F1 mIoU MoF F1 mIoU CLOT [9] 60.1 40.1 18.5 54.4 36.7 23.4 50.6 46.6 31.4 59.4 63.2 38.8 68.8 72.6 48.1 D-CLOT w/o A A only GC in F 57.5 39.4 17.9 51.8 36.0 26.9 51.2 46.3 31.3 49.5 57.1 36.4 60.5 67.5 41.7 shared by both variants only GC in FRF_R 58.3 40.2 18.3 50.8 34.8 25.8 50.6 47.5 32.4 48.3 56.4 36.0 65.8 68.8 45.0 only GC in F,FRF,\,F_R 58.1 39.6 18.1 50.1 34.7 25.1 51.9 45.3 30.1 48.2 56.2 36.1 58.3 64.1 40.3 D-CLOT w/o GC 58.3 39.8 17.9 53.9 36.2 25.4 46.3 42.8 29.4 50.1 58.1 36.1 58.8 59.2 38.7 w/o GC in FRF_R 58.5 40.3 18.0 54.6 36.8 26.2 49.7 47.5 32.0 51.9 59.4 37.3 59.2 61.8 39.6 w/o GC in F 59.5 41.0 18.8 54.4 37.7 26.3 50.6 47.0 32.2 54.9 66.6 38.9 67.0 68.9 46.7 Full 59.4 40.5 19.0 55.7 37.9 27.3 51.0 47.8 32.8 62.7 65.3 39.1 68.7 76.4 50.5 D-CLOTB w/o GC 57.2 40.6 18.6 51.1 34.9 23.8 49.2 43.7 30.5 48.5 58.0 35.9 59.0 63.6 40.0 w/o GC in FRF_R 42.0 32.1 18.4 53.0 36.1 25.3 41.6 38.7 25.2 47.8 57.6 35.0 65.2 68.9 44.8 w/o GC in F 56.3 39.5 18.1 53.8 36.2 25.7 47.0 44.0 28.5 54.9 66.6 39.0 68.1 68.7 45.7 Full 58.5 41.3 19.1 54.9 36.8 26.4 50.8 47.0 32.0 64.9 72.1 40.4 70.4 78.2 51.8 , Figure 3: Qualitative ablation of the D-CLOT components on the prototype geometry and the resulting segmentation. For a representative YTI video we show, top, the first two principal components of the refined frame embeddings RF_R (dots, coloured by ground-truth action) together with the action prototypes A (diamonds) and, bottom, the ground-truth (GT) and predicted (pred) frame labels. The four panels ablate the two modules: (i) the CLOT baseline; (i) D-CLOT without prototype re-estimation (w/o A A); (i) D-CLOT without the graph constraint (w/o GC); and (iv) the full D-CLOT model. In CLOT, two prototypes collapse onto the same region and a spurious action (blue) is predicted where none exists in the ground truth. The graph constraint restores temporally coherent boundaries but leaves the spurious prototype active; prototype re-estimation relocates the prototypes towards their supporting frame clusters. Only the full model, combining both, removes the spurious assignment and recovers a segmentation that closely matches the ground truth. Colours denote action classes. IV-C Assembly101: a fine-grained procediral benchmark Table I highlights the difficulty of our Assembly101 subset. It contains the longest videos on average (13,04813,048 frames), the largest activity diversity (1515 toy categories with 1111–4242 actions each), and substantial background (42.1%42.1\%). Together with its multi-view recordings and V-JEPA2 features, these properties make it more challenging than existing cooking and instructional benchmarks. Table I reports the first unsupervised action-segmentation results on Assembly101. Both D-CLOT and D-CLOTB improve substantially over CLOT in F1 and mIoU at the video level, at the cost of a small MoF drop, again reflecting the MoF/F1–mIoU trade-off under the long-tailed, fine-grained action distribution of this benchmark (1111–4242 actions per toy category). Activity-level performance is markedly lower for every method, confirming that Assembly101 is a harder benchmark than the cooking and instructional datasets, owing to its larger action vocabulary and multi-view structure. D-CLOTB obtains the best overall average, indicating that graph-constrained action refinement generalises beyond the cooking and instructional domains to large-scale, fine-grained assembly video. Our method provides the strongest first baseline, but the scores indicate substantial room for future work. (a) Frame graph weight λGF _G^F (b) Refine graph weight λGR _G^R (c) Graph bandwidth h (d) Blend factor β (e) k-means refinement (f) Barycentric refinement Figure 4: Sensitivity analysis on F1. Each panel varies one hyperparameter while keeping the remaining configuration fixed. Colours denote datasets and line styles denote D-CLOT and D-CLOTB. IV-D Ablation study Table IV isolates the contribution of the three proposed components: graph regularization on the frame embeddings (F~ F), graph regularization on the refined frame embeddings (F~R F_R), and action-embedding refinement (A). These components play complementary roles. The graph on F~ F preserves local visual neighborhoods before segment-level propagation, whereas the graph on F~R F_R regularizes a representation already enriched by cross-attention and therefore acts more directly on temporal coherence. The results show that action refinement is most effective when applied to a graph-stabilized representation. Updating A alone is unstable for D-CLOT, since prototypes are re-anchored to an unconstrained refined space. Conversely, graph regularization alone yields dataset-dependent gains, but does not consistently match the full configuration. Overall, the full model improves over CLOT on most dataset/metric pairs, especially in F1 and mIoU, showing that geometry stabilization and prototype refinement address different aspects of the representation–prototype mismatch. A few ablated configurations outperform the full D-CLOT model on metrics. These cases reflect metric-specific trade-offs rather than a failure of the full design. On FS-Mid, removing A slightly increases MoF, but reduces both F1 and mIoU, consistent with MoF’s bias toward dominant long-duration actions. On FS-Eval, using F~R F_R and A without F~ F improves F1, but causes a large MoF drop and slightly lower mIoU, indicating boundary-level gains at the expense of frame coverage. In contrast, D-CLOTB shows a more stable pattern: its full configuration dominates all partial variants across datasets and metrics. This supports the assignment-aware barycentric update as a more robust refinement mechanism. Fig. 3 illustrates the complementary roles of graph regularization (GC) and action refinement. CLOT disrupts the action order, while GC recovers more coherent temporal boundaries. However, GC still assigns frames to a spurious action prototype, shown by the blue segment. Updating A corrects this prototype mismatch: in the full D-CLOT model, the blue segment disappears, indicating that any frame no longer selects the corresponding prototype. Thus, GC mainly regularizes temporal structure, whereas action refinement corrects the prototype-to-frame assignment. On this video, D-CLOT improves MoF from 0.6630.663 to 0.9390.939, F1 from 0.3330.333 to 1.0001.000, and mIoU from 0.3480.348 to 0.8830.883. The PCA plot supports this interpretation. Stable prototypes, such as the light-green one, remain almost unchanged, while the dark-green prototype moves substantially toward the relevant frame cluster. This displacement reduces the influence of the dark-blue prototype and resolves the erroneous assignment. The proximity between the light-blue and dark-orange prototypes also explains the observed confusion in ordering and boundaries in CLOT. Overall, the figure shows that D-CLOT improves segmentation by jointly correcting temporal structure and prototype geometry. IV-E Sensitivity analysis We analyse the sensitivity of D-CLOT and D-CLOTB with respect to the main hyperparameters introduced by the graph-constrained stabilisation and action-refinement modules in Fig. 4. Since temporal action segmentation is highly sensitive to over-segmentation and boundary fragmentation, we report F1 in the main paper and provide the corresponding MOF and mIoU plots in the supplementary material. Compared with MOF, which can be biased towards frequent or long-duration actions, and compared with mIoU, it is less brittle under the noisy pseudo-labels inherent to unsupervised training. F1 provides a more diagnostic measure of segmental quality. Graph fidelity weights (λGF _G^F, λGR _G^R). On DA, F1 peaks sharply at λGF=0.05 _G^F=0.05 and λGR=0.1 _G^R=0.1 for both variants: a moderate amount of graph regularisation stabilises the refined frame geometry, while excessive regularisation over-constrains it, as expected from (5). D-CLOTB is smoother and less sharply peaked than D-CLOT, particularly with respect to λGF _G^F. On YTI, F1 varies little: with 62% background frames, there is limited geometry to anchor to, so the modules act as a stabiliser rather than a strong amplifier, the intended fallback behaviour, not a shortcoming. FS-Eval sits in between, but the barycentric variant benefits from the refined graph weight, though this gain requires controlled strength to avoid over-smoothing. Graph bandwidth (h). On DA, small h collapses the affinity graph to near-diagonal and the term stops regularising, while a large h flattens it and discards local structure, per (4). YTI stays flat, consistent with a stabilisation role rather than a performance lever under weak temporal regularity. FS-Eval is mixed: flat for D-CLOT, but a 10-point swing for D-CLOTB, again reflecting the barycentric variant’s greater ability to exploit graph structure when it is informative. Blend factor (β). On DA, moderate blending helps both variants: the transport plan is reliable, so re-anchoring prototypes captures real structure. On YTI, a small blending is best, with only mild degradation as β grows; this is not a failure of the update but confirmation that the bounded, convex blend of (6) degrades gracefully under unreliable transport mass. FS-Eval is less consistent between variants: D-CLOT peaks at the largest β tested, D-CLOTB barely moves. Stabilisation budget (k-means iterations, inner steps S). D-CLOT is robust to its k-means budget (7): F1 does not change on YTI, and our default is within 0.1 point of the best value on DA. Therefore, sharper prototype relocation is beneficial here without requiring careful tuning. D-CLOTB’s inner-step count S (8) matters more. The three datasets peak sharply at S=2S=2, our default. In general, more steps pull the representation harder toward the encoder anchor, which helps denoise YTI’s weaker signal but risks over-smoothing FS-Eval’s already coarse, 12-class boundaries. We fix S=2S=2, which matches DA and YTI exactly and balances the other dataset complementary preferences. Across datasets, the sensitivity curves demonstrate that graph-constrained action refinement is robust to moderate hyperparameter variations. No parameter causes a substantial performance degradation, and the best results generally arise from intermediate graph regularization and conservative prototype updates. This suggests that refinement improves the geometry of frame representations without excessively constraining OT-based action discovery. D-CLOT is preferable when stronger k-means prototype relocation is beneficial, whereas D-CLOTB yields smoother, assignment-aware updates through its auxiliary graph-stabilized representation. The lower sensitivity observed on YTI indicates that, for noisier videos pooled across activities, refinement primarily stabilizes learning rather than substantially amplifying performance. This behavior is particularly evident for β: the bounded update in (6) limits prototype changes on YTI while exploiting clearer structure on DA. Overall, these results support the selected default configuration and show that the improvements do not depend on narrowly tuned hyperparameters. Figure 5: Example segmentation from DA[28] (top) and FS (Eval)[45] (bottom). We report the ground-truth (GT), the results of CLOT, and the result in ours two variants, D-CLOT and D-CLOTB. IV-F Qualitative results. Figure 5 compares the temporal segmentations produced by CLOT, D-CLOT, and D-CLOTB on representative videos from 50Salads and Desktop Assembly. On FS-Eval, CLOT introduces spurious classes in the early–middle region and over-merges the central structure into a single block, losing the fine transitions. D-CLOT removes them and restores cleaner, more coherent segments, while D-CLOTB recovers short, minority actions and thin boundaries that the other rows miss. On DA, CLOT preserves the global action order but drifts boundaries and drops several short steps. D-CLOT tightens these boundaries, and D-CLOTB yields the closest alignment to GT, respecting the procedural order and recovering the short intermediate segments. Re-anchoring the prototypes to the graph-stabilized geometry thus fixes the spurious-class and boundary-merging failures of CLOT, and the assignment-aware update preserves exactly the short, imbalanced actions that drive F1/mIoU while barely affecting MoF. V Discussion and Conclusion We introduced D-CLOT, a graph-constrained extension of CLOT for unsupervised temporal action segmentation. The proposed framework addresses the mismatch between progressively refined frame representations and latent action prototypes by jointly regularizing the representation geometry and periodically updating the prototypes. We investigated two refinement strategies: a k-means update and an assignment-aware optimal transport barycentric update, while preserving the original CLOT formulation. Extensive experiments on five benchmarks demonstrate consistent improvements over CLOT, with substantial gains in segment-level quality, particularly in F1 and mIoU. The ablation analysis shows that graph-based regularization and prototype refinement are complementary: graph regularization promotes temporally coherent representations, whereas prototype refinement maintains alignment between latent actions and the evolving feature space. Among the proposed variants, the assignment-aware barycentric refinement consistently provides the most robust performance across datasets. Beyond improving the state of the art, we present the first unsupervised temporal action segmentation benchmark on Assembly101, highlighting the challenges posed by fine-grained actions, viewpoint variability, and procedural diversity. These results suggest that continuously adapting latent prototypes to an evolving representation space is a key ingredient for scalable unsupervised temporal action segmentation, and motivate future work on richer prototype learning and more expressive latent action models. Acknowledgments This work was supported by grant PRE2020-094714, the project PID2019-110977GA-I00 and the project PID2023-151351NB-I00, funded by Ministerio de Ciencia e Innovación (MCIN)/ Agencia Estatal de Investigación (AEI) /10.13039/501100011033, by European Social Fund (ESF) Investing in your future and by ERDF, UE. It was also supported by an Alexander von Humboldt (AvH) fellowship for experienced researchers funded by the AvH Foundation. References [1] S. N. Aakur and S. Sarkar (2019) A perceptual prediction framework for self supervised event segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §I-A, §I-A. [2] J. Alayrac, P. Bojanowski, N. Agrawal, J. Sivic, I. Laptev, and S. Lacoste-Julien (2016) Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4575–4583. Cited by: Figure 10, 4th item, §I, 2nd item, TABLE I, TABLE IV, TABLE IV. [3] A. S. Ali, S. A. Mahmood, M. Saeed, A. Konin, M. Z. Zia, and Q. Tran (2025) Joint self-supervised video alignment and action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10807–10818. Cited by: §I, §I-A, §I-A, §I-A, §I-B, §IV-A, TABLE I. [4] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. Robert Hogan, D. Dugas, P. Bojanowski, V. Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y. Li, X. Ma, S. Chandar, F. Meier, Y. LeCun, M. Rabbat, and N. Ballas (2025) V-jepa 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. Cited by: 5th item, TABLE I. [5] E. Bahrami, G. Francesca, and J. Gall (2023) How much temporal long-term context is needed for action segmentation?. In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §I-A. [6] N. Behrmann, S. A. Golestaneh, Z. Kolter, J. Gall, and M. Noroozi (2022) Unified fully and timestamp supervised temporal action segmentation via sequence to sequence translation. In Proceedings of the European Conference on Computer Vision (ECCV), Cited by: §I-A, §I-B. [7] J. Benamou, G. Carlier, M. Cuturi, L. Nenna, and G. Peyré (2015) Iterative Bregman Projections for Regularized Transportation Problems. SIAM Journal on Scientific Computing 37 (2), p. A1111–A1138. External Links: Document Cited by: §I-D. [8] E. Bueno-Benito and M. Dimiccoli (2024) 2by2: weakly-supervised learning for global action segmentation. In International Conference on Pattern Recognition, p. 380–395. Cited by: §I-A, §I-A. [9] E. Bueno-Benito and M. Dimiccoli (2025) CLOT: closed loop optimal transport for unsupervised action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10719–10729. Cited by: Figure 6, Figure 6, Appendix B, §I, §I-A, §I-B, §I-E, §I, §IV-A, TABLE I, TABLE I, TABLE I, TABLE IV, TABLE IV, TABLE IV. [10] E. Bueno-Benito, B. Tura, and M. Dimiccoli (2023) Leveraging triplet loss for unsupervised action segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, Cited by: §I-A, §I-A, §I-A, §IV-A, TABLE I. [11] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020) End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV), p. 213–229. Cited by: §I-B. [12] M. Caron, P. Bojanowski, A. Joulin, and M. Douze (2018) Deep Clustering for Unsupervised Learning of Visual Features. In Proceedings of the European Conference on Computer Vision (ECCV), p. 132–149. Cited by: §I-D. [13] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin (2020) Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Advances in Neural Information Processing Systems, Vol. 33, p. 9912–9924. Cited by: §I-D. [14] M. Cuturi and A. Doucet (2014) Fast Computation of Wasserstein Barycenters. In Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 32, p. 685–693. Cited by: §I-D. [15] C. Dias and M. Dimiccoli (2018) Learning event representations by encoding the temporal context. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, p. 0–0. Cited by: §I-A. [16] M. Dimiccoli, L. Garrido, G. Rodriguez-Corominas, and H. Wendt (2021) Graph constrained data representation learning for human motion segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 1460–1469. Cited by: §I-C. [17] G. Ding, F. Sener, and A. Yao (2023) Temporal action segmentation: an analysis of modern techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (2), p. 1011–1030. Cited by: §I, §IV-A. [18] G. Ding and A. Yao (2022) Temporal action segmentation with high-level complex activity labels. IEEE Transactions on Multimedia 25, p. 1928–1939. Cited by: §I-A, §I-A, §I-A, §IV-A. [19] Z. Du, X. Wang, G. Zhou, and Q. Wang (2022) Fast and unsupervised action boundary detection for action segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §I-A, §I-A, §IV-A, TABLE I. [20] E. Elhamifar and R. Vidal (2013) Sparse subspace clustering: algorithm, theory, and applications. IEEE transactions on pattern analysis and machine intelligence 35 (11), p. 2765–2781. Cited by: §I-C. [21] D. Gong, J. Lee, M. Kim, S. J. Ha, and M. Cho (2022) Future transformer for long-term action anticipation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3052–3061. Cited by: §I, §I-B. [22] P. Goyal and E. Ferrara (2018) Graph embedding techniques, applications, and performance: a survey. Knowledge-Based Systems 151, p. 78–94. Cited by: §I-C, §I-C. [23] F. Huang, X. Chen, W. Wu, and W. Ma (2025) Not all samples are equal: boosting action segmentation via selective incremental learning. Engineering Applications of Artificial Intelligence 147, p. 110334. Cited by: §I-A. [24] J. Huang, R. Cai, J. Fang, H. Zhu, B. Xu, W. Chen, Z. Li, and S. Gao (2026-06) Hierarchical action learning for weakly-supervised action segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 6054–6064. Cited by: §I. [25] S. Kolouri, K. Nadjahi, U. Simsekli, R. Badeau, and G. Rohde (2019) Generalized sliced wasserstein distances. Advances in Neural Information Processing Systems (NeurIPS) 32. Cited by: §I-B, §I-E. [26] H. Kuehne, A. Arslan, and T. Serre (2014) The language of actions: recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: Figure 10, 4th item, §I, 1st item, TABLE I, TABLE IV, TABLE IV. [27] A. Kukleva, H. Kuehne, F. Sener, and J. Gall (2019) Unsupervised learning of action classes with continuous temporal embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §I, §I-A, §I-A, §I-A, §IV-A, §IV-A, TABLE I. [28] S. Kumar, S. Haresh, A. Ahmed, A. Konin, M. Z. Zia, and Q. Tran (2022) Unsupervised action segmentation by joint representation learning and online clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 20174–20185. Cited by: Figure 9, 4th item, §I, §I, §I-A, §I-A, §I-A, §I-B, §I-E, Figure 5, Figure 5, 4th item, §IV-A, §IV-A, TABLE I, TABLE I, TABLE I, TABLE IV, TABLE IV. [29] K. Lee, W. Xu, F. Fan, and Z. Tu (2018) Wasserstein introspective neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3702–3711. Cited by: §I-B. [30] J. Li and S. Todorovic (2021) Action shuffle alternating learning for unsupervised action segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §I, §I-A, §I-A, §I-A, TABLE I. [31] K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, et al. (2024) Mvbench: a comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22195–22206. Cited by: §I. [32] Y. Li, J. Zhou, X. Zheng, J. Tian, and Y. Y. Tang (2019) Robust subspace clustering with independent and piecewise identically distributed noise modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8720–8729. Cited by: §I-C. [33] Y. Li, Z. Xue, and H. Xu (2024) OTAS: unsupervised boundary detection for object-centric temporal action segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 6437–6446. Cited by: §I-A, §I-A, §I-A, TABLE I. [34] C. Lu, H. Min, Z. Zhao, L. Zhu, D. Huang, and S. Yan (2012) Robust and Efficient Subspace Segmentation via Least Squares Regression. In Proceedings of the European Conference on Computer Vision (ECCV), p. 347–360. External Links: Document Cited by: §I-C. [35] Z. Lu and E. Elhamifar (2022) Set-supervised action learning in procedural task videos via pairwise order consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19903–19913. Cited by: §I-A. [36] Z. Lu and E. Elhamifar (2024) FACT: frame-action cross-attention temporal modeling for efficient action segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §I-A. [37] Z. Lu and E. Elhamifar (2025) Multi-modal few-shot temporal action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 14106–14116. Cited by: §I. [38] K. Nguyen and N. Ho (2023) Energy-based sliced wasserstein distance. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 18046–18075. Cited by: §I-B, §I-E. [39] J. Rabin, G. Peyré, J. Delon, and M. Bernot (2012) Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision: Third International Conference, SSVM, p. 435–446. Cited by: §I-B, §I-E. [40] M. S. Sarfraz, N. Murray, V. Sharma, A. Diba, L. V. Gool, and R. Stiefelhagen (2021) Temporally-weighted hierarchical clustering for unsupervised action segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §I-A, §I-A, §I-A, §IV-A, TABLE I. [41] F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao (2022) Assembly101: a large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 21096–21106. Cited by: 5th item, §I, 5th item, §IV-A, §IV-A, TABLE I. [42] K. Simonyan and A. Zisserman (2015) Very deep convolutional networks for large-scale image recognition. In 3rd international conference on learning representations (ICLR 2015), Cited by: 2nd item. [43] Y. Souri, M. Fayyaz, L. Minciullo, G. Francesca, and J. Gall (2022) Fast weakly supervised action segmentation using mutual consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §I-A. [44] F. Spurio, E. Bahrami, G. Francesca, and J. Gall (2025) Hierarchical vector quantization for unsupervised action segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 6996–7005. Cited by: §I, §I-A, §I-A, §IV-A, §IV-A, §IV-A, TABLE I. [45] S. Stein and S. J. McKenna (2013) Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, p. 729–738. Cited by: Figure 9, 4th item, §I, Figure 5, Figure 5, 3rd item, TABLE I, TABLE IV, TABLE IV. [46] S. Swetha, H. Kuehne, Y. S. Rawat, and M. Shah (2021) Unsupervised discriminative embedding for sub-action learning in complex activities. In 2021 IEEE International Conference on Image Processing (ICIP), p. 2588–2592. Cited by: §I-A, TABLE I. [47] M. Thorpe (2018) Introduction to optimal transport. Notes of Course at University of Cambridge. Cited by: §I-E. [48] Q. Tran, A. Mehmood, M. Ahmed, M. Naufil, A. Zafar, A. Konin, and M. Z. Zia (2024) Permutation-aware activity segmentation via unsupervised frame-to-segment alignment. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision, p. 6412–6422. Cited by: §I-A, §I-A, §I-A, §I-B, 4th item, §IV-A, §IV-A, TABLE I. [49] R. Vidal (2011) Subspace clustering. IEEE Signal Processing Magazine 28 (2), p. 52–68. Cited by: §I-C. [50] R. G. VidalMata, W. J. Scheirer, A. Kukleva, D. D. Cox, and H. Kuehne (2021) Joint visual-temporal embedding for unsupervised learning of actions in untrimmed sequences. In IEEE Winter Conference on Applications of Computer Vision (WACV), p. 1237–1246. Cited by: §I, §I-A, §I-A, §I-A, §IV-A, TABLE I. [51] H. Wang and C. Schmid (2013) Action recognition with improved trajectories. In Proceedings of IEEE/CVF international conference on computer vision, Cited by: 1st item, 3rd item. [52] T. Wang and S. Todorovic (2026-03) Timestamp query transformer for temporal action segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 5016–5025. Cited by: §I. [53] Z. Xing and W. Zhao (2026) Temporal visual semantics-induced human motion understanding with large language models. Cited by: §I-C. [54] A. Xu and W. Zheng (2024) Efficient and effective weakly-supervised action segmentation via action-transition-aware boundary alignment. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §I, §I-A. [55] M. Xu and S. Gould (2024) Temporally consistent unbalanced optimal transport for unsupervised action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14618–14627. Cited by: Appendix B, §I, §I-A, §I-A, §I-A, §I-B, §I-E, §I-E, §I-E, §IV-A, §IV-A, §IV-A, TABLE I, TABLE I. [56] R. Zhang, S. Wang, Y. Duan, Y. Tang, Y. Zhang, and Y. Tan (2023) HOI-aware adaptive network for weakly-supervised action segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: §I-A. [57] Z. Zhao, R. Ghoddoosian, I. Dwivedi, N. Agarwal, and B. Dariush (2025-06) Pose-aware weakly-supervised action segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition Workshops, p. 97–107. Cited by: §I. This supplementary material provides additional evidence supporting the robustness and practicality of D-CLOT and D-CLOTB. It examines how consistently the methods improve individual videos, rather than relying only on dataset-level averages (App. A). It also reports complete hyperparameter settings (App. B), additional sensitivity results (App. C), computational-cost measurements (App. D), and further qualitative examples (App. E). Together, these analyses clarify where the gains come from, how stable they are, and the additional cost of each variant. Appendix A Where Do the Gains Come From? Figure 6 analyses where the aggregate gains come from, moving from dataset means to the per-video distribution. A mass above it denotes videos that improve, and a mass below it denotes regressions. This view distinguishes a broad, population-level effect from one carried by a few sequences, which dataset averages alone cannot reveal. On FSeval, the entire distribution shifts upward: for both variants, the boxes lie predominantly above the zero line in MoF and in F1 for D-CLOT/D-CLOTB. The gains are therefore not driven by a few favourable sequences but reflect a consistent, population-level effect, and D-CLOTB shows the stronger shift, indicating that the assignment-aware barycentric update yields a more systematic alignment between frame representations and action prototypes. YTI exhibits a complementary regime. Its distributions stay concentrated around zero but develop pronounced positive tails, producing sizeable average gains despite lower per-video win rates: the refinements do not improve every sequence, but where they help, the gains outweigh the occasional regressions. The contrast between the two datasets thus reveals two operating modes, a stable and broadly distributed improvement on FSeval and a selective but high-impact correction on YTI, the latter consistent with YTI’s shorter, more heterogeneous videos and its independently trained activity-specific checkpoints, which introduce greater per-video variability. Figure 6: Per-video improvement over CLOT [9] for MoF and F1 on FS-Eval (n=50n=50) and YTI (n=149n=149, pooled over its five activities). For each video, Δ=variant−CLOT =variant-CLOT is computed separately for D-CLOT and D-CLOTB; each box shows the median, interquartile range, and whiskers of Δ , with per-video values overlaid as jittered points. Mass above the dashed Δ=0 =0 line marks videos that improve over CLOT, mass below it marks regressions. Appendix B Hyperparameter Details D-CLOT and D-CLOTB share the same backbone and optimisation protocol, inherited unchanged from the underlying OT-based encoder: we use the Adam optimiser with a weight decay of 10−410^-4, and, following [55, 9], set the number of action prototypes K to the ground-truth number of actions per activity category/dataset. The temporal-consistency radius r of the base OT solver is fixed at 0.020.02 for all datasets, except for Breakfast, where the longer and more finely segmented videos benefit from a wider window (r=0.04r=0.04). The frame/action affinity graph used by the graph-constrained modules is sparsified to its 3232 nearest neighbours throughout, except Assembly101 (1616), whose substantially longer, multi-view videos favour a sparser graph. For D-CLOTB, the auxiliary alignment weight is fixed at λa=1.0 _a=1.0 for all datasets except Breakfast (λa=0.25 _a=0.25), where a lighter alignment term better tolerates the noisier pseudo-labels typical of longer, unstructured activities. Beyond this shared backbone, D-CLOT and D-CLOTB introduce a small number of graph-regularisation and prototype-update hyperparameters: the frame and refined graph weights λGF _G^F and λGR _G^R, the graph bandwidth h, and the blend factor β. Rather than treat these as free parameters tuned independently per dataset, we select them from the robust, wide-margin operating region identified by the sensitivity analysis in Sec.IV-E and in App. C which show performance to be stable under moderate perturbation of each of these values; the per-dataset choices used to produce our reported hyperparameters are given in Tables V and VI for D-CLOT and D-CLOTB, respectively. TABLE V: Hyperparameter settings used for D-CLOT across datasets. FS (Mid), FS (Eval) and A101 denote the 50 Salads Mid, Eval splits, and Assembly101, respectively. BF YTI FS (Mid) FS (Eval) DA A101 λGF _G^F 0.10 0.10 0.05 0.05 0.05 0.03 λGR _G^R 0.10 0.15 0.10 0.20 0.10 0.05 Graph bandwidth h 0.05 0.05 0.05 0.03 0.05 0.08 Blend factor β 0.25 0.02 0.05 0.15 0.10 0.25 K-means iterations 20 20 20 5 10 20 Epochs 25 15 60 20 110 35 TABLE VI: Hyperparameter settings used for D-CLOTB across datasets. FS (M), FS (E) and A101 denote the 50 Salads Mid, Eval splits, and Assembly101, respectively. BF YTI FS (Mid) FS (Eval) DA A101 λGF _G^F 0.01 0.05 0.10 0.05 0.05 0.03 λGR _G^R 0.15 0.10 0.10 0.10 0.10 0.05 Graph bandwidth h 0.02 0.08 0.05 0.02 0.08 0.08 Blend factor β 0.05 0.25 0.25 0.02 0.05 0.25 Barycentric inner steps S 2 2 2 2 2 2 Epochs 50 30 35 20 90 35 Appendix C Sensitive Analysis This section complements Section IV-E with MoF and mIoU for the same datasets. MoF and mIoU peak at the same hyperparameter value as F1 in almost every case. A few FS-Eval configurations shift the MoF optimum by one grid step and under 1.51.5 points, with F1 and mIoU unaffected. This is consistent with FS-Eval’s coarse 1212-class structure, where MoF depends on coverage of a few long segments and is sensitive to the handful of boundary frames a one-step change reassigns, whereas F1 and mIoU are not. No hyperparameter trades one metric off against another, so the F1 gains in the main text are not an artefact of over- or under-segmentation (a) Frame graph weight λGF _G^F (b) Refine graph weight λGR _G^R (c) Graph bandwidth h (d) Blend factor β (e) k-means refinement (f) Barycentric refinement Figure 7: Sensitivity analysis on mIoU. Each panel varies one hyperparameter while keeping the remaining configuration fixed. Colours denote datasets and line styles denote D-CLOT and D-CLOTB. (a) Frame graph weight λGF _G^F (b) Refine graph weight λGR _G^R (c) Graph bandwidth h (d) Blend factor β (e) k-means refinement (f) Barycentric refinement Figure 8: Sensitivity analysis on mof. Each panel varies one hyperparameter while keeping the remaining configuration fixed. Colours denote datasets and line styles denote CLOT2 and CLOTB2^2_B. Appendix D Computacional cost and practicality D-CLOT and D-CLOTB retain the same 92.5K trainable parameters as CLOT. Their improvements therefore do not rely on additional model capacity. Table VII reports training time, inference latency, and peak GPU memory on the Breakfast coffee activity. All methods use the same encoder, OT configuration, batch size, and hardware. D-CLOT adds 2.06 s per training epoch due to the periodic K-means prototype update. This operation is used only during training, and its inference latency remains comparable to CLOT. D-CLOTB adds 0.47 s per epoch. Its barycentric re-estimation costs only 26 ms every ten epochs. At inference, the inner refinement increases latency by 8.2 ms per video. Peak memory remains below 82 MB for all variants. Overall, the proposed closed loops introduce limited computational overhead and require no additional trainable parameters. D-CLOT preserves the inference cost of CLOT, while D-CLOTB provides a stronger assignment-aware refinement at a moderate additional cost. TABLE VII: Training cost of the CLOT family on the Breakfast coffee activity (K=7K=7, 167 training videos, B=2B=2, and T=256T=256 sampled frames per video). Latency is averaged over 60 iterations after 15 warm-up iterations on a single RTX 4090. The refinement column reports its cost and update period. Method Params GFLOPs Iter. (ms) Memory (MB) Refinement Epoch (s) CLOT 92.5 0.036 61.7±0.861.7± 0.8 70.9 – 5.02 D-CLOT 92.5 0.077 63.8±0.663.8± 0.6 75.5 1,915 ms / epoch 7.08 D-CLOTB 92.5 0.077 73.2±1.173.2± 1.1 81.2 26 ms / 10 epochs 5.49 Appendix E More qualitative results Figure 9 provides additional qualitative comparisons across a broader set of videos. The results reinforce the trends observed in the main paper: D-CLOT and D-CLOTB better preserve temporal continuity, recover short and ambiguous actions, and reduce both over-segmentation and boundary drift relative to CLOT. In particular, the barycentric refinement of D-CLOTB yields more coherent action segments in challenging transition regions. The same behaviour appears on BF and YTI in Figure 10: both variants track the long dominant actions well, but D-CLOT tends to drop the short terminal segments (the trailing blue on BF, the closing teal on YTI), whereas D-CLOTB recovers them and preserves the GT ordering more faithfully. This is consistent with the quantitative trend: D-CLOTB preserves exactly the short, imbalanced actions that drive F1/mIoU while barely affecting the long-action-dominated MoF. Figure 9: Example segmentation from DA[28] (top) and FS (Eval)[45] (bottom). We report the ground-truth (GT), the results of CLOT, and the result in ours two variants, D-CLOT and D-CLOTB. Figure 10: Example segmentation from Breakfast[26] (top) and YTI[2] (bottom), We report the ground-truth (GT), the results of CLOT, and the result in ours two variants, D-CLOT and D-CLOTB.