Paper deep dive
Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning
Lingyu Shen, Wei Tang, Fakhri Karray, Min-Ling Zhang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-instance partial-label learning (MIPL) addresses inexact supervision in both the instance and label spaces, which can be applied to video classification. However, bag-level labels do not explicitly supervise the correspondence between candidate classes and temporal evidence. We propose {\ours}, which couples label disambiguation with temporal evidence allocation through a joint class--time assignment. Occupancy-regularized spherical matching associates contextualized video features while learning nonuniform temporal mass and discouraging excessive concentration. During training, candidate-restricted inference recomputes the assignment within the candidate label set. A dual-marginal KL projection then constructs a structured teacher that incorporates momentum-refined class beliefs while preserving the proposal's temporal occupancy. A single plan-level KL objective aligns the full-space predictor with this teacher. Our analysis characterizes when candidate re-solving differs from masking and shows that, under the stated construction, the joint objective decomposes into class-marginal and class-conditional temporal supervision. We construct VCMIPL benchmarks from Breakfast, DoTA, and FineAction using model-generated candidate labels and evaluate the method across four feature representations. Extensive experimental results demonstrate that PIVOTMIPL outperforms existing MIPL algorithms in both effectiveness and efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2610.06234v1
- Canonical: https://arxiv.org/abs/2610.06234v1
Trouble viewing inline? Open PDF directly →
Full Text
91,829 characters extracted from source content.
Expand or collapse full text
Joint Class–-Time Learning for Video Classification with Multi-Instance Partial-Label Learning Lingyu Shen †thanks: These authors contributed equally to this work. Affiliation: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China Wei Tang†footnotemark: Affiliation: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates Fakhri Karray Affiliation: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates Min-Ling Zhang †thanks: Corresponding author. Email: zhangml@seu.edu.cn Affiliation: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China Abstract Multi-instance partial-label learning (MIPL) addresses inexact supervision in both the instance and label spaces, which can be applied to video classification. However, bag-level labels do not explicitly supervise the correspondence between candidate classes and temporal evidence. We propose PivotMipl, which couples label disambiguation with temporal evidence allocation through a joint class–time assignment. Occupancy-regularized spherical matching associates contextualized video features while learning nonuniform temporal mass and discouraging excessive concentration. During training, candidate-restricted inference recomputes the assignment within the candidate label set. A dual-marginal KL projection then constructs a structured teacher that incorporates momentum-refined class beliefs while preserving the proposal’s temporal occupancy. A single plan-level KL objective aligns the full-space predictor with this teacher. Our analysis characterizes when candidate re-solving differs from masking and shows that, under the stated construction, the joint objective decomposes into class-marginal and class-conditional temporal supervision. We construct VCMIPL benchmarks from Breakfast, DoTA, and FineAction using model-generated candidate labels and evaluate the method across four feature representations. Extensive experimental results demonstrate that PivotMipl outperforms existing MIPL algorithms in both effectiveness and efficiency. 1 Introduction Learning to classify videos from weak annotations requires identifying both the true label and its supporting temporal evidence. Video-level labels do not specify which frames or clips support the target class within a sequence that may also contain background, transitions, and related activities, resulting in inexact supervision in the input space (Zhang et al., 2025b). At the class level, candidate pseudolabeling retains several plausible classes from model predictions to generate the candidate label set (Zhang et al., 2024). It contains the true label and several false-positive labels, which does not specify the true label and provides inexact supervision in the output space (Wang et al., 2025). Multi-instance partial-label learning (MIPL) captures both forms of inexact supervision by representing each example as a bag of instances paired with a candidate label set containing its true label and several false-positive labels (Tang et al., 2024c). We formulate this setting as video classification with MIPL (VCMIPL), where each bag is an ordered sequence of frame or clip features and its supervision is a candidate label set. Existing methods offer several ways to learn from inexact supervision. Attention-based MIPL couples bag representation learning with momentum-based label disambiguation (Tang et al., 2023). In partial-label learning, prototype-guided disambiguation (Wang et al., 2022b; Jin et al., 2025) and transport-based label refinement (Wang et al., 2022a; Gu et al., 2025) exploit feature geometry and assignment constraints, respectively. Weakly-supervised video classification jointly learns classification and temporal instance selection from explicit video-level labels (Wang et al., 2017). However, in VCMIPL, temporal evidence must be inferred while the true class remains ambiguous. Although instance selection and label disambiguation can interact during learning, refining bag-level class probabilities alone does not determine how temporal evidence is allocated to each candidate class. Distinct class–time assignments can yield identical class probabilities, even when they assign the same total mass to every temporal position. This motivates learning from a structured class–time target that combines refined class beliefs with temporal allocation inferred within the candidate set. In this paper, we propose a VCMIPL method named PivotMipl, i.e., Plan Inference via Variable-Occupancy Transport for Multi-Instance Partial-Label learning, which represents each video through a class–prototype–time assignment. A temporal encoder produces contextualized features that are matched to multiple prototypes per class on the unit sphere. Temporal occupancy, the mass assigned to each position, is inferred jointly with the assignment, with regularization discouraging excessive concentration. During training, we re-solve the assignment within the candidate label set to obtain a proposal and refine its class marginal through momentum accumulation. A KL projection then constructs the closest teacher to this proposal subject to the refined class marginal and the proposal’s temporal occupancy. A single plan-level KL objective aligns the full-space predictor with the structured teacher. At test time, prediction uses only the full-space assignment, requiring neither candidate labels nor teacher construction. We construct three VCMIPL benchmarks based on Breakfast (Kuehne et al., 2014), DoTA (Yao et al., 2023), and FineAction (Liu et al., 2022) datasets, using four feature representations. Extensive experimental results on the benchmarks show that PivotMipl achieves the highest mean accuracy and balanced accuracy in 10 of 12 configurations, averaging gains of 3.16%3.16\% and 3.26%3.26\% over the strongest baseline per configuration, respectively. And the accuracy is significantly higher than the strongest baseline in 8 of 12 configurations. Ablations support the contributions of occupancy regularization, adaptive temporal allocation, joint class–time supervision, dual-marginal teacher construction, multiple prototypes, and temporal encoding. Gains persist in 10 configurations against AgopMipl (Cao et al., 2026), which is the strongest baseline, with the same temporal encoder. Further analyses show favorable accuracy–training-time trade-offs in several configurations and stable performance over the tested regularization ranges on three representative configurations. Our contributions are threefold: First, we introduce a structured framework for VCMIPL that couples candidate-restricted temporal allocation with label disambiguation through a joint assignment and a teacher constrained by class and temporal marginals. Second, we analyze the inference and supervision mechanisms, characterizing the relationship between candidate re-solving and masking, the concentration control induced by occupancy regularization, and the effective supervision provided by the joint objective. Third, we construct three VCMIPL benchmarks with model-generated candidate label sets with four feature representations, and evaluate the framework against representative MIPL methods on the benchmarks. 2 Related work 2.1 Multi-Instance Partial-Label Learning MIPL learns from bags with ambiguous candidate labels, addressing inexact supervision in both instance and label spaces. Tang et al. (2024c) introduced instance-level probabilistic disambiguation, whereas Tang et al. (2023) used attention-based bag representations and momentum-refined label beliefs. Subsequent methods exploit complementary candidate- and non-candidate-label information (Tang et al., 2024b), attention and prediction margins (Tang et al., 2024a), probabilistic generative modeling of latent label distributions (Yang et al., 2024), and instance- and bag-level dependencies and heterogeneity for computational efficiency (Yang et al., 2025). Recent approaches incorporate candidate-aware propensity scoring and deconfounding (Luo et al., 2026), feature and label propagation over bag- and instance-level graphs (Zhu et al., 2026), feature geometry based on average gradient outer products (Cao et al., 2026), aggregation uncertainty through evidential learning (Jie et al., 2026), and prediction calibration (Tang et al., 2026). However, our focus is on the specific coupling of candidate-restricted temporal occupancy, historical class beliefs, and a teacher constrained by class and temporal marginals. 2.2 Weakly Supervised Video Learning Weakly supervised video learning infers temporal evidence from video-level labels. UntrimmedNets represents videos as temporal segments and jointly learns segment selection and video-level classification (Wang et al., 2017). Later methods model intra- and inter-segment dependencies (He et al., 2022), organize evidence into proposals (Ren et al., 2023), and infer salient snippet features from variations between neighboring time steps (Yun et al., 2024). PseudoFormer progressively converts weak predictions into more reliable pseudo supervision for temporal localization (Liu and Liu, 2025), while multimodal large language models provide semantic priors (Zhang et al., 2025a). These methods generally assume unambiguous video labels. Our setting additionally requires identifying the true class within a candidate set and linking it to supporting temporal evidence. 3 Methodology 3.1 Problem Formulation and Overview Let =(i,Si)i=1NtrainD=\(X_i,S_i)\_i=1^N_train denote a MIPL training dataset of NtrainN_train videos. Each video is represented as an ordered multi-instance bag i=(i1,…,ini)∈ℝni×dX_i=(x_i1,…,x_in_i) ^n_i× d, where nin_i is the number of instances and d is the feature dimension. Each video has a single target label Yi∈=1,…,CY_i =\1,…,C\, but is annotated only with a nonempty candidate label set Si⊆S_i containing YiY_i. We adopt collective bag classification, allowing multiple temporal positions to jointly support the target class without requiring every raw frame to belong to that class. The framework of PivotMipl is shown in Figure 1. A feature-aware temporal encoder produces contextualized instance representations. Occupancy-regularized spherical transport matches these representations to multiple prototypes per class, yielding an actor plan over the full label space with adaptive temporal occupancy while discouraging excessive concentration. During training, we re-solve the transport problem within the candidate label set to obtain a proposal and refine its class marginal through momentum accumulation. We then construct a structured teacher plan that matches the refined class marginal while preserving the proposal’s temporal occupancy. A plan-level KL objective aligns the actor with the detached teacher, jointly supervising class assignments and class-conditional temporal distributions. Figure 1: The framework of PivotMipl 3.2 Temporal Encoder The temporal encoder maps a multi-instance bag i∈ℝni×dX_i ^n_i× d to contextualized features i∈ℝni×dZ_i ^n_i× d. A local difference module processes frame-level ResNet (He et al., 2016) and DINOv3 (Siméoni et al., 2025) features, while clip-level SlowFast (Feichtenhofer et al., 2019) and VideoMAEv2 (Wang et al., 2023) features bypass this module because they already represent local clips. Both paths then use bidirectional Mamba-2 (Dao and Gu, 2024) and a gated residual adapter. We denote row-wise affine maps by Affa(H)=HWa⊤+ba⊤Aff_a(H)=HW_a +1b_a , with learnable weights and biases, and apply layer normalization (LN) over the feature dimension. The input projection maps the features from d-dimensional to dhd_h-dimensional channels: i(0)=GELU(Affin(LN(i))),i(0)∈ℝni×dh.H_i^(0)=GELU (Aff_ in(LN(X_i)) ), _i^(0) ^n_i× d_h. (1) Here AffinAff_ in maps d-dimensional input channels to dhd_h-dimensional hidden channels. Local Temporal Differences. For frame-level features, we first smooth the projected sequence with a three-position moving average, then compute backward and forward differences: ^it(0)=13(i,t−1(0)+it(0)+i,t+1(0)),Δit−=^it(0)−^i,t−1(0),Δit+=^i,t+1(0)−^it(0). h_it^(0)= 13 (h_i,t-1^(0)+h_it^(0)+h_i,t+1^(0) ), _it^-= h_it^(0)- h_i,t-1^(0), _it^+= h_i,t+1^(0)- h_it^(0). (2) Here it(0)h_it^(0) denotes the t-th row of i(0)H_i^(0). The moving average uses replicate padding at both sequence boundaries. We set Δi1−= _i1^-=0 and Δini+= _in_i^+=0; when ni=1n_i=1, both difference vectors are zero. We concatenate the two difference sequences along the channel dimension, denoted by [⋅;⋅][·;·], project them from 2dh2d_h-dimensional channels to dhd_h-dimensional ones, and apply two convolutional branches: i=AffΔ([i−;i+]),i(j)=SiLU(δj(i)),(δ1,δ2)=(1,2).D_i=Aff_ ([ _i^-; _i^+]), _i^(j)=SiLU (C_ _j(D_i) ), ( _1, _2)=(1,2). (3) δjC_ _j contains a depthwise temporal convolution with kernel size of 55 and dilation δj _j, followed by a 1×11× 1 pointwise convolution for channel mixing. The two branches have receptive fields of 55 and 99 positions over iD_i, respectively, and each produces an output of shape ni×dhn_i× d_h. Here SiLU(x)=xσ(x)SiLU(x)=xσ(x), where σ is the sigmoid function. We fuse the branch outputs and form a gated motion residual: i _i =SiLU(LN(AffM([i(1);i(2)]))), =SiLU (LN(Aff_M([U_i^(1);U_i^(2)])) ), (4) i _i =σ(Affg([i(0);i])),ℳ(i(0))=i(0)+Dropout0.1(i⊙i). =σ (Aff_g([H_i^(0);M_i]) ), (H_i^(0))=H_i^(0)+Dropout_0.1(G_i _i). Both AffMAff_M and AffgAff_g map 2dh2d_h channels to dhd_h. The gate iG_i weights the input-dependent motion representation iM_i at each temporal position and hidden channel. The input to the bidirectional encoder is ~i(0)=ℳ(i(0)) H_i^(0)=M(H_i^(0)) for frame-level features and ~i(0)=i(0) H_i^(0)=H_i^(0) for clip-level features. Bidirectional Temporal Context. We apply L bidirectional Mamba-2 blocks in sequence: i(L)=ℬ(L)∘⋯∘ℬ(1)(~i(0)).H_i^(L)=B^(L) ·s ^(1)( H_i^(0)). (5) Each block processes the sequence in forward and reverse temporal order. The backward output is restored to the original order, and the two outputs are combined using a feature-dependent directional gate before a feed-forward residual update. The output adapter projects the hidden features back to d-dimensional channels and adds a gated residual to the input: i=i+σ(γ)Dropout(Affout(LN(i(L)))).Z_i=X_i+σ(γ)Dropout (Aff_ out(LN(H_i^(L))) ). (6) Here AffoutAff_ out maps dhd_h-dimensional channels to d-dimensional ones, and γ∈ℝγ is a learnable scalar. The identity path preserves the input features, while the residual incorporates temporal context. Therefore, the output positions are contextualized features rather than independent raw snippets. 3.3 Occupancy-Regularized Spherical Transport Spherical Matching And Occupancy Regularization. Each class has K learnable prototypes ck∈ℝdp_ck ^d. For nonzero features and prototypes, define the unit-normalized vectors ¯it=it/‖it‖2 z_it=z_it/\|z_it\|_2 and ¯ck=ck/‖ck‖2 p_ck=p_ck/\|p_ck\|_2. The matching cost is half the squared geodesic distance on the unit sphere: Cickt=12arccos2(¯ck⊤¯it).C_ickt= 12 ^2( p_ck z_it). (7) For any nonempty admissible class set ⊆A , define Δi _i^A as the unit simplex over (c,k,t)∈×1,…,K×1,…,ni(c,k,t) ×\1,…,K\×\1,…,n_i\. With temporal occupancy mit=∑c∈,kΠicktm_it= _c ,k _ickt, our inference operator is ε,λ(i;)=argmin∈Δi∑c∈,k,tΠicktCickt+ε∑c∈,k,tΠickt(logΠickt−1)+λ2(ni∑tmit2−1), splitT_ ,λ(C_i;A)= _ ∈ _i^A \& _c ,k,t _icktC_ickt+ _c ,k,t _ickt( _ickt-1)\\ &+ λ2 (n_i _tm_it^2-1 ) \, split (8) where ε>0 >0 and λ≥0λ≥ 0. Here, “transport” refers to a regularized assignment with freely optimized class and temporal marginals. Let iu_i be the uniform temporal distribution, with uit=1/niu_it=1/n_i. The occupancy penalty can be written as Rocc(i)=χ2(i∥i)=eD2(i∥i)−1=ni∑tmit2−1,R_ occ(m_i)=χ^2(m_i\|u_i)=e^D_2(m_i\|u_i)-1=n_i _tm_it^2-1, (9) where χ2χ^2 and D2D_2 denote the Pearson χ2χ^2 divergence and the order-2 Rényi divergence, respectively. This penalty discourages excessive mass on a few temporal positions without enforcing uniform occupancy. It controls concentration but does not identify action locations. More information about the concentration properties can be found in Appendix A.5. Factorized Inference. For fixed temporal occupancy, the conditional minimization over classes and prototypes admits a Gibbs solution. Define it=∑c∈,ke−Cickt/ε,ρickt=e−Cickt/εit,fit=−εlogit.Z_it^A= _c ,ke^-C_ickt/ , _ickt^A= e^-C_ickt/ Z_it^A, f_it^A=- _it^A. (10) Substituting Πickt=mitρickt _ickt=m_it _ickt^A into Eq. (8) and dropping terms independent of im_i gives i⋆=argmini∈Δni⟨i,i⟩+εDKL(i∥i)+λ2(ni∥i∥22−1),m_i = _m_i∈ _n_i \ _i,f_i^A + D_ KL(m_i\|u_i)+ λ2 (n_i\|m_i\|_2^2-1 ) \, (11) where Δni _n_i is the unit simplex in ℝniR^n_i. The remaining problem is to optimize an nin_i-dimensional occupancy vector, with its normalization multiplier determined via scalar root-finding. The optimal plan, full-space actor plan, and actor class marginal are, respectively, Πickt⋆=mit⋆ρickt,iA=ε,λ(i,),qicA=∑k,tΠicktA. _ickt =m_it _ickt^A, _i^A=T_ ,λ(C_i;Y), q_ic^A= _k,t _ickt^A. (12) Occupancy Solver. For λ>0λ>0, the occupancy can be expressed via the normalization multiplier ξi _i: mit(ξi)=ελniW(λεe(ξi−fit)/ε),gi(ξi)=∑tmit(ξi)−1,m_it( _i)= λ n_iW\! ( λ e^( _i-f_it^A)/ ), g_i( _i)= _tm_it( _i)-1, (13) where W is the principal Lambert-W function, defined by W(x)eW(x)=xW(x)e^W(x)=x. We choose the multiplier to satisfy gi(ξi)=0g_i( _i)=0. The described solver initializes ξi(0)=−εlog∑tuitexp(−fit/ε) _i^(0)=- _tu_it (-f_it^A/ ) and applies Newton updates: ξi(r+1)=ξi(r)−gi(ξi(r))∑tmit(ξi(r))ε+λnimit(ξi(r)). _i^(r+1)= _i^(r)- g_i( _i^(r)) _t m_it( _i^(r)) +λ n_im_it( _i^(r)). (14) When λ=0λ=0, the solution instead takes the form mit⋆=exp(−fit/ε)/∑sexp(−fis/ε)m_it = (-f_it^A/ )/ _s (-f_is^A/ ). This case requires a separate branch as Eq. (13) contains 1/λ1/λ, and entropy regularization remains active. Appendix A.1 derives these expressions and establishes the uniqueness of the positive optimum. We compute fitf_it^A using log-sum-exp and evaluate the conditional assignments in the log domain. For λ>0λ>0, the Lambert-W term is evaluated directly from its log argument: letting hit=log(λ/ε)+(ξi−fit)/εh_it= (λ/ )+( _i-f_it^A)/ , we solve wit+logwit=hitw_it+ w_it=h_it for wit=W(ehit)w_it=W(e^h_it) by Newton iteration, avoiding explicit construction of ehite^h_it. We then update the scalar normalization multiplier by Newton iteration and recover mit=εwit/(λni)m_it= w_it/(λ n_i). The occupancy solve is performed in double precision, and we monitor both its normalization and KKT stationarity residuals. For λ=0λ=0, we directly use the closed-form softmax solution. Class Belief Update. During training, we construct a proposal on the candidate label set SiS_i using the full-space actor’s current costs and temperature: ~i=ε,λ(sg(i),Si),q~ic=∑k,tΠ~ickt,m~it=∑c∈Si,kΠ~ickt. _i=T_ ,λ(sg(C_i);S_i), q_ic= _k,t _ickt, m_it= _c∈ S_i,k _ickt. (15) Here, sgsg denotes stop-gradient. We recompute the partition functions and free energies over candidate classes and apply the same occupancy solver to obtain ~i m_i. Therefore, this restriction determines class support and can change temporal allocation. Re-solving generally differs from globally masking and renormalizing the actor. Global masking also changes temporal mass, but it need not optimize the restricted occupancy-regularized objective. Appendix A.2 gives the precise equivalence conditions, including the λ=0λ=0 boundary. The method uses re-solving rather than a fixed-time conditional renormalization. Initialize qic(0)=[c∈Si]/|Si|q_ic^(0)=1[c∈ S_i]/|S_i|. At epoch e, combine the stored belief with the current proposal marginal: qic(e)=μqic(e−1)+(1−μ)q~ic,μ∈[0,1),c∈Si,q_ic^(e)=μ q_ic^(e-1)+(1-μ) q_ic, μ∈[0,1),\ c∈ S_i, (16) and set qic(e)=0q_ic^(e)=0 for c∉Sic∉ S_i. The EMA averages class marginals, not full plans or network parameters. Each training bag is visited once per epoch. At its visit in epoch e, the stored qi(e−1)q_i^(e-1) is combined with the current detached proposal marginal to form qi(e)q_i^(e); this refined belief is used immediately for teacher construction and is stored after the optimizer step for the next epoch. 3.4 Candidate-Restricted Teacher Construction The teacher incorporates historical class mass while retaining the proposal’s current temporal occupancy. We obtain it through the KL projection on the candidate support: iT _i^T =argmin≥0DKL(∥~i), = _ ≥ 0D_ KL( \| _i), (17) subject to∑k,tΠickt to _k,t _ickt =qic(e)(c∈Si),∑c∈Si,kΠickt=m~it(t=1,…,ni). =q_ic^(e) (c∈ S_i), _c∈ S_i,k _ickt= m_it (t=1,…,n_i). Let M~ict=∑kΠ~ickt M_ict= _k _ickt denote the proposal’s class–time matrix. The teacher has the multiplicative form ΠicktT=αicΠ~icktβit _ickt^T= _ic _ickt _it. Starting from positive scaling factors, we alternate the updates αic←qic(e)∑tM~ictβit,βit←m~it∑c∈SiM~ictαic. _ic← q_ic^(e) _t M_ict _it, _it← m_it _c∈ S_i M_ict _ic. (18) Each update enforces one target marginal; both constraints hold at convergence. We perform the marginal scaling in the log domain using log-sum-exp denominators. Computing the scaling factors requires only the |Si|×ni|S_i|× n_i class–time matrix. The full teacher is then recovered by multiplying each proposal entry by αicβit _ic _it, with the same factors applied across prototypes. We extend the teacher with zeros outside SiS_i to obtain ^iT _i^T. The training objective is ℒ=1Ntrain∑iℒiplan,ℒiplan=DKL(sg(^iT)∥iA).L= 1N_train _iL_i^plan, _i^plan=D_KL\! (sg( _i^T)\| _i^A ). (19) Under the same-cost construction and exact teacher projection, the prototype-conditional distributions of the actor and teacher coincide on the candidate support. Therefore, the plan-level objective reduces to ℒiplan=DKL(iT∥iA)+∑c:qicT>0qicTDKL(PiT(t∣c)∥PiA(t∣c)),L_i^plan=D_KL(q_i^T\|q_i^A)+Σ _c:q_ic^T>0q_ic^TD_KL\! (P_i^T(t c)\|P_i^A(t c) ), (20) where qicTq_ic^T and qicAq_ic^A are the teacher and actor class marginals, respectively, and PiT(t∣c)P_i^T(t c) and PiA(t∣c)P_i^A(t c) denote their class-conditional temporal distributions obtained after marginalizing over the prototype dimension. Thus, beyond class-marginal alignment, the joint objective explicitly supervises the class-conditional temporal allocation. A complete derivation and the corresponding current-point gradient equivalence are provided in Appendix A.3. The loss uses the full-space actor without renormalizing it over SiS_i, thereby penalizing mass assigned to non-candidate classes. Detaching the entire target-construction pipeline gives ∇θℒiplan=−∑c∈Si,k,tΠicktT∇θlogΠicktA. _θL_i plan=-Σ _c∈ S_i,k,t _ickt^T _θ _ickt^A. (21) The actor gradient propagates through both temporal occupancy and Gibbs conditional assignments to the matching costs, prototypes, and temporal encoder. We differentiate through the unrolled actor solver, and no gradients pass through proposal inference, EMA updates, or teacher projection. The resulting supervision acts on class–time associations parameterized by multiple prototypes. Under the stated same-cost construction, the actor and teacher have identical conditional distributions over prototypes given class and time on the candidate support. The prototype-conditional KL term is therefore zero, although prototypes still affect the class–time distribution and can receive gradients through it. The full KL decomposition, the corresponding gradient equivalence at the teacher construction point, and boundary cases are established in Appendix A.3–A.4. 3.5 Joint Class–Time Learning and Prediction Algorithm 3.5 summarizes the per-bag training procedure. Training uses one complete video bag per optimization step: we compute one plan-level loss, perform one backward pass and gradient-clipped optimizer update, and then write the detached refined class belief back to the training dataset. Since bags are processed individually, no temporal padding is introduced. At test time, we compute only iZ_i, the matching costs, and the full-space actor, and predict y^i=argmax∑k,tc∈ΠicktA y_i= _c _k,t ^A_ickt. Algorithm 1: PivotMipl training and prediction 1 Initialize parameters and qic(0)=[c∈Si]/|Si|q_ic^(0)=1[c∈ S_i]/|S_i|. 2 For each epoch e and each training bag (Xi,Si)(X_i,S_i): 3 Encode valid ordered positions and compute iC_i. 4 Solve full-space actor iA=(i,) _i^A=T(C_i;Y). 5 Solve proposal ~i=(sg(i),Si) _i=T(sg(C_i);S_i). 6 Update class belief using Eq. (16). 7 Scale the proposal to (i(e),~i)(q_i^(e), m_i) using Eq. (18). 8 Extend teacher by zero outside SiS_i and detach it. 9 Backpropagate the actor KL, clip the global gradient norm, step the optimizer, and store the detached refined belief. 10 At test time, return argmax∑k,tcΠicktA _c _k,t _ickt^A without candidates. 4 Experiments 4.1 Experimental Setup Benchmark Construction We construct three VCMIPL benchmarks, i.e., Breakfast-MIPL, DoTA-MIPL, and FineAction-MIPL, from the Breakfast (Kuehne et al., 2014), DoTA (Yao et al., 2023), and FineAction (Liu et al., 2022) datasets, respectively. For FineAction, we retain single-label videos and exclude classes with fewer than 50 videos. The resulting subsets contain 1,989 Breakfast videos (10 classes), 4,577 DoTA videos (9 classes), and 10,675 FineAction videos (78 classes). Appendix B reports candidate-set and feature statistics. We use ResNet (He et al., 2016) and DINOv3 (Siméoni et al., 2025) for frame-level features, and SlowFast (Feichtenhofer et al., 2019) and VideoMAEv2 (Wang et al., 2023) for clip-level features. To reduce the annotation cost, we generate candidate labels using a VideoMAEv2 recognizer equipped with a TimeSformerHead in MMAction2 (MMAction2 Contributors, 2020). We fine-tune the recognizer on four folds and generate predictions for the held-out fold, repeating this procedure for all five folds. Fine-tuning uses 30 epochs on DoTA and 100 epochs on both Breakfast and FineAction. Following Zhang et al. (2024), we first select labels for each video in descending order of confidence until their cumulative confidence reaches 95%95\%. A selected label is retained only if the video also ranks among the top 50%50\% of videos by confidence for that label. Comparative Methods We compare PivotMipl with seven MIPL baselines: DeMipl (Tang et al., 2023), EliMipl (Tang et al., 2024b), MiplMa (Tang et al., 2024a), ProMipl (Yang et al., 2024), PsMipl (Luo et al., 2026), DualG (Zhu et al., 2026), and AgopMipl (Cao et al., 2026). We exclude MiplGp (Tang et al., 2024c), FastMipl (Yang et al., 2025), and UAMipl (Jie et al., 2026) because their released implementations require all instance-level features to be loaded into memory, making them impractical for these video benchmarks under the available memory budget. Implementation Details PivotMipl is implemented in PyTorch and trained for 40 epochs on an NVIDIA H100 GPU using AdamW with (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999) and zero weight decay. The learning rates are 5×10−35× 10^-3 for the prototypes and 5×10−45× 10^-4 for the temporal encoder. We evaluate PivotMipl over the same five predefined folds used for cross-fitting the candidate-label recognizer. We report classification accuracy (ACC) and balanced accuracy (BACC) as mean ± standard deviation over five folds, where BACC is the mean recall over ground-truth classes represented in each test fold. More implementation details are provided in Appendix B.3. Table 1: Classification accuracy on the benchmark datasets. Results are reported as (mean ± std %). The best result in each column is bolded, and the second-best results are underlined. ∙ /∘ indicates whether PivotMipl is statistically superior/inferior to the corresponding method under a two-sided paired t-test at p<0.05p<0.05. Dataset Method DINOv3 VideoMAEv2 ResNet SlowFast Breakfast -MIPL DeMipl 11.4%± 0.4%∙11.4\%\, ±\,0.4\%\, 25.7%± 10.0%∙25.7\%\, ±\,10.0\%\, 10.1%± 0.7%∙10.1\%\, ±\,0.7\%\, 11.0%± 0.5%∙11.0\%\, ±\,0.5\%\, EliMipl 27.8%± 0.8%∙27.8\%\, ±\,0.8\%\, 52.2%± 1.9%∙52.2\%\, ±\,1.9\%\, 26.1%± 1.7%∙26.1\%\, ±\,1.7\%\, 11.3%± 0.1%∙11.3\%\, ±\,0.1\%\, MiplMa 33.9%± 1.5%∙33.9\%\, ±\,1.5\%\, 63.6%± 2.1%∙63.6\%\, ±\,2.1\%\, 30.5%± 2.4%∙30.5\%\, ±\,2.4\%\, 11.3%± 0.1%∙11.3\%\, ±\,0.1\%\, ProMipl 62.2%¯± 1.2%∙ 62.2\%\, ±\,1.2\%\, 83.8%± 3.4%∙83.8\%\, ±\,3.4\%\, 43.7%¯± 1.2%∙ 43.7\%\, ±\,1.2\%\, 41.6%± 1.1%∙41.6\%\, ±\,1.1\%\, PsMipl 47.3%± 5.0%∙47.3\%\, ±\,5.0\%\, 70.3%± 7.3%∙70.3\%\, ±\,7.3\%\, 11.0%± 1.4%∙11.0\%\, ±\,1.4\%\, 33.0%± 0.6%∙33.0\%\, ±\,0.6\%\, DualG 45.7%± 3.2%∙45.7\%\, ±\,3.2\%\, 64.9%± 4.4%∙64.9\%\, ±\,4.4\%\, 34.3%± 2.0%∙34.3\%\, ±\,2.0\%\, 14.4%± 1.6%∙14.4\%\, ±\,1.6\%\, AgopMipl 46.3%± 4.2%∙46.3\%\, ±\,4.2\%\, 87.6%± 4.3%87.6\%\, ±\,4.3\% 33.3%± 3.1%∙33.3\%\, ±\,3.1\%\, 50.4%¯± 2.5%∙ 50.4\%\, ±\,2.5\%\, PivotMipl 66.3%± 1.0%66.3\%\, ±\,1.0\% 86.1%¯± 4.5% 86.1\%\, ±\,4.5\% 52.3%± 3.4%52.3\%\, ±\,3.4\% 55.9%± 1.8%55.9\%\, ±\,1.8\% DoTA -MIPL DeMipl 45.6%± 1.3%∙45.6\%\, ±\,1.3\%\, 68.1%± 0.7%∙68.1\%\, ±\,0.7\%\, 42.6%± 0.4%∙42.6\%\, ±\,0.4\%\, 41.2%± 1.6%∙41.2\%\, ±\,1.6\%\, EliMipl 45.8%± 1.6%∙45.8\%\, ±\,1.6\%\, 68.3%± 1.0%∙68.3\%\, ±\,1.0\%\, 43.4%± 0.5%∙43.4\%\, ±\,0.5\%\, 41.7%± 1.0%∙41.7\%\, ±\,1.0\%\, MiplMa 46.8%± 1.1%∙46.8\%\, ±\,1.1\%\, 69.2%± 0.9%∙69.2\%\, ±\,0.9\%\, 43.7%± 1.0%∙43.7\%\, ±\,1.0\%\, 41.8%± 1.1%∙41.8\%\, ±\,1.1\%\, ProMipl 49.2%± 1.8%∙49.2\%\, ±\,1.8\%\, 72.5%± 1.5%72.5\%\, ±\,1.5\% 45.9%¯± 2.2%∙ 45.9\%\, ±\,2.2\%\, 48.0%¯± 0.4% 48.0\%\, ±\,0.4\% PsMipl 49.8%¯± 0.9%∙ 49.8\%\, ±\,0.9\%\, 74.1%¯± 0.9% 74.1\%\, ±\,0.9\% 44.6%± 1.3%∙44.6\%\, ±\,1.3\%\, 26.0%± 13.0%∙26.0\%\, ±\,13.0\%\, DualG 46.1%± 2.6%∙46.1\%\, ±\,2.6\%\, 69.6%± 0.8%∙69.6\%\, ±\,0.8\%\, 44.3%± 1.2%∙44.3\%\, ±\,1.2\%\, 43.6%± 0.9%∙43.6\%\, ±\,0.9\%\, AgopMipl 49.7%± 1.3%49.7\%\, ±\,1.3\% 69.9%± 1.8%∙69.9\%\, ±\,1.8\%\, 45.2%± 0.7%∙45.2\%\, ±\,0.7\%\, 47.0%± 0.1%∙47.0\%\, ±\,0.1\%\, PivotMipl 52.7%± 2.0%52.7\%\, ±\,2.0\% 74.7%± 1.8%74.7\%\, ±\,1.8\% 49.6%± 1.2%49.6\%\, ±\,1.2\% 48.8%± 0.6%48.8\%\, ±\,0.6\% FineAction -MIPL DeMipl 41.5%± 1.6%∙41.5\%\, ±\,1.6\%\, 52.1%± 1.6%∙52.1\%\, ±\,1.6\%\, 21.7%± 1.8%∙21.7\%\, ±\,1.8\%\, 50.2%± 2.1%∙50.2\%\, ±\,2.1\%\, EliMipl 42.0%± 1.0%∙42.0\%\, ±\,1.0\%\, 49.6%± 2.7%∙49.6\%\, ±\,2.7\%\, 22.9%± 2.0%∙22.9\%\, ±\,2.0\%\, 49.3%± 1.7%∙49.3\%\, ±\,1.7\%\, MiplMa 55.4%± 1.4%∙55.4\%\, ±\,1.4\%\, 68.3%± 0.5%∙68.3\%\, ±\,0.5\%\, 35.7%± 3.0%∙35.7\%\, ±\,3.0\%\, 57.2%± 2.3%∙57.2\%\, ±\,2.3\%\, ProMipl 80.7%± 1.5%∙80.7\%\, ±\,1.5\%\, 92.0%± 3.4%92.0\%\, ±\,3.4\% 69.7%± 1.4%∙69.7\%\, ±\,1.4\%\, 82.4%± 2.2%∙82.4\%\, ±\,2.2\%\, PsMipl 60.5%± 1.4%∙60.5\%\, ±\,1.4\%\, 66.2%± 1.5%∙66.2\%\, ±\,1.5\%\, 43.0%± 2.0%∙43.0\%\, ±\,2.0\%\, 32.7%± 14.9%∙32.7\%\, ±\,14.9\%\, DualG 58.1%± 1.4%∙58.1\%\, ±\,1.4\%\, 74.2%± 0.8%∙74.2\%\, ±\,0.8\%\, 46.4%± 1.3%∙46.4\%\, ±\,1.3\%\, 63.9%± 0.8%∙63.9\%\, ±\,0.8\%\, AgopMipl 82.2%¯± 1.3%∙ 82.2\%\, ±\,1.3\%\, 96.3%± 0.5%∘96.3\%\, ±\,0.5\%\, 71.1%¯± 1.0%∙ 71.1\%\, ±\,1.0\%\, 84.2%¯± 2.2%∙ 84.2\%\, ±\,2.2\%\, PivotMipl 88.5%± 1.0%88.5\%\, ±\,1.0\% 94.0%¯± 1.4% 94.0\%\, ±\,1.4\% 76.2%± 1.2%76.2\%\, ±\,1.2\% 88.3%± 1.0%88.3\%\, ±\,1.0\% 4.2 Main Experimental Results Table 1 shows that PivotMipl achieves the highest mean ACC in 10 of the 12 dataset–feature configurations, with an average gain of 3.16% over the strongest baseline in each configuration. Beyond the overall improvement, the performance pattern supports the central motivation of our framework. Most competing MIPL methods ultimately resolve ambiguity through bag-level class supervision, without explicitly retaining which temporal evidence supports each candidate class. In contrast, PivotMipl preserves this class–time association throughout disambiguation, so that the plan-level objective constrains both the class belief and its supporting temporal allocation. This distinction is particularly evident with the frame-level DINOv3 and ResNet representations, where PivotMipl consistently ranks first across all three datasets. These representations leave temporal dependencies largely unresolved before MIPL learning, making explicit evidence allocation especially useful for separating informative instances from background or class-irrelevant content. The main exceptions occur in VideoMAEv2, where strong baselines already achieve highly competitive performance. A plausible explanation is that its clip-level representations already encode substantial local spatiotemporal context, reducing the remaining ambiguity that can be resolved by downstream temporal restructuring. Nevertheless, PivotMipl is inferior to AgopMipl in only one configuration while being significantly superior to the strongest competing method in eight, suggesting that its advantage primarily stems from coupling label disambiguation with temporal evidence allocation rather than from a particular feature backbone. 4.3 Ablation Study We evaluate six ablation variants across all 12 dataset–feature configurations using Fold 1 ACC at epoch 40. Table 2 presents the individual results. Moreover, a summary of the ablations across twelve dataset–feature configurations is provided in Appendix C.3.1. A1 removes occupancy regularization by setting λ=0λ=0, while retaining entropy regularization and learned temporal occupancy. A2 fixes temporal occupancy to the uniform distribution, mit=1/nim_it=1/n_i. A3 replaces the joint KL objective with a class-level KL objective. A4 replaces the dual-marginal teacher projection with class-marginal-only rescaling, removing the constraint that preserves the proposal’s temporal occupancy. A5 uses a single prototype per class, i.e., K=1K=1. A6 removes the temporal encoder. Table 2 shows that every ablation reduces mean ACC across the 12 configurations. Notably, A1 and A2 reduce ACC in 10 and 9 configurations, favoring regularized yet adaptive temporal allocation. A3 and A4 degrade performance across all configurations, even with explicit class–time supervision and occupancy-preserving teacher construction. A5 and A6 each reduce ACC in 11 configurations, supporting multiple prototypes and temporal contextualization. Temporal encoding has the largest average impact, while the joint objective and dual-marginal teacher deliver the most consistent gains. Appendix C.1 reports the BACC results, with PivotMipl achieving the highest mean performance in 1010 of the 1212 configurations. To assess the role of temporal encoding, In Appendix C.3.2, we compare PivotMipl with AgopMipl, for which we adopt the same temporal encoder as PivotMipl. PivotMipl outperforms this baseline in 1010 of the 1212 configurations, suggesting that the encoder alone does not account for the observed gains. Appendix C.2 reports favorable trade-offs between accuracy and computational efficiency across most evaluated settings. Appendix C.4 shows limited performance variation over the tested ranges of the main transport hyperparameters. Table 2: Ablation study of PivotMipl. Dataset Feature PivotMipl A1 A2 A3 A4 A5 A6 Breakfast -MIPL DINOv3 66.4%¯ 66.4\% 62.6%62.6\% 72.1%72.1\% 66.1%66.1\% 64.8%64.8\% 64.3%64.3\% 57.1%57.1\% VideoMAEv2 77.4%¯ 77.4\% 77.8%77.8\% 77.1%77.1\% 70.3%70.3\% 67.1%67.1\% 72.6%72.6\% 75.6%75.6\% ResNet 52.2%¯ 52.2\% 42.6%42.6\% 54.4%54.4\% 51.4%51.4\% 50.3%50.3\% 47.9%47.9\% 35.4%35.4\% SlowFast 53.9%53.9\% 48.4%48.4\% 53.4%53.4\% 52.6%52.6\% 51.6%51.6\% 53.6%¯ 53.6\% 49.9%49.9\% DoTA -MIPL DINOv3 54.3%54.3\% 50.3%50.3\% 51.7%51.7\% 53.5%¯ 53.5\% 52.6%52.6\% 51.3%51.3\% 49.3%49.3\% VideoMAEv2 72.8%¯ 72.8\% 72.0%72.0\% 70.9%70.9\% 72.2%72.2\% 72.7%72.7\% 73.0%73.0\% 72.7%72.7\% ResNet 50.9%50.9\% 48.1%48.1\% 49.8%49.8\% 50.5%¯ 50.5\% 50.4%50.4\% 47.2%47.2\% 47.5%47.5\% SlowFast 48.4%48.4\% 46.4%46.4\% 47.8%¯ 47.8\% 45.9%45.9\% 47.1%47.1\% 46.5%46.5\% 43.9%43.9\% FineAction -MIPL DINOv3 90.0%90.0\% 89.7%¯ 89.7\% 86.3%86.3\% 79.0%79.0\% 86.3%86.3\% 85.7%85.7\% 89.4%89.4\% VideoMAEv2 92.4%92.4\% 93.7%93.7\% 96.3%¯ 96.3\% 88.7%88.7\% 89.1%89.1\% 91.1%91.1\% 96.7%96.7\% ResNet 78.1%78.1\% 75.5%75.5\% 71.9%71.9\% 72.6%72.6\% 77.6%¯ 77.6\% 75.4%75.4\% 75.8%75.8\% SlowFast 90.0%90.0\% 80.7%80.7\% 86.5%86.5\% 84.9%84.9\% 86.9%86.9\% 88.0%¯ 88.0\% 64.3%64.3\% 5 Conclusion We presented PivotMipl, a structured framework coupling candidate-label disambiguation with temporal evidence allocation for video multi-instance partial-label learning. Occupancy-regularized spherical transport matches contextualized features to multiple prototypes per class, learning nonuniform temporal assignments while discouraging excessive concentration. Candidate-restricted inference and dual-marginal KL projection construct a teacher that integrates momentum-refined class beliefs while preserving the proposal’s temporal occupancy. A single plan-level KL objective aligns the full-space predictor with this teacher. Our analysis characterizes when candidate re-solving differs from masking and establishes, under the proposed construction, the objective’s decomposition into class-marginal alignment and class-conditional temporal supervision. Across Breakfast-MIPL, DoTA-MIPL, and FineAction-MIPL with four feature representations, PivotMipl achieves the highest mean ACC and BACC in 10 of 12 configurations. Ablations support the joint objective and dual-marginal teacher, while encoder-matched comparisons suggest gains beyond temporal encoding. These findings support resolving label ambiguity through explicit class–time associations rather than class probabilities alone. References Cao et al. (2026) N. Cao, X. Zhao, and T. Zhang Enhanced multi-instance partial label learning via average gradient outer product. In Proceedings of the 43rd International Conference on Machine Learning, Cited by: §1, §2.1, §4.1. Dao and Gu (2024) T. Dao and A. Gu Transformers are SSMs: generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, p. 10041–10071. Cited by: §3.2. Feichtenhofer et al. (2019) C. Feichtenhofer, H. Fan, J. Malik, and K. He SlowFast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 6201–6210. Cited by: §B.2, §3.2, §4.1. Gu et al. (2025) Y. Gu, B. Chen, Z. Chen, Z. Qiao, X. Luo, J. Luo, Z. Xiao, W. Ju, and M. Zhang MATE: masked optimal transport with dynamic selection for partial label graph learning. Artificial Intelligence 348, p. 104396. Cited by: §1. He et al. (2022) B. He, X. Yang, L. Kang, Z. Cheng, X. Zhou, and A. Shrivastava ASM-Loc: action-aware segment modeling for weakly-supervised temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13915–13925. Cited by: §2.2. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. Cited by: §B.2, §3.2, §4.1. Jie et al. (2026) G. Jie, F. Wang, G. Song, L. Lin, Y. Yu, and Q. Zheng Uncertainty-aware multi-instance partial-label learning via evidential deep model. Neurocomputing 701, p. 134578. Cited by: §2.1, §4.1. Jin et al. (2025) Y. Jin, Y. Wang, and X. Chu NPC: negative prototypical contrasting for label disambiguation of partial label learning. Journal of Computer Science and Technology 40 (5), p. 1386–1400. Cited by: §1. Kuehne et al. (2014) H. Kuehne, A. Arslan, and T. Serre The language of actions: recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 780–787. Cited by: §B.1, §1, §4.1. Liu et al. (2022) Y. Liu, L. Wang, Y. Wang, X. Ma, and Y. Qiao FineAction: a fine-grained video dataset for temporal action localization. IEEE Transactions on Image Processing 31, p. 6937–6950. Cited by: §B.1, §1, §4.1. Liu and Liu (2025) Z. Liu and Y. Liu Bridge the gap: from weak to full supervision for temporal action localization with PseudoFormer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8711–8720. Cited by: §2.2. Luo et al. (2026) H. Luo, Y. Zhang, Z. Zhou, W. Liu, and M. Li Propensity scoring for multi-instance partial-label learning. In Web and Big Data (APWeb-WAIM 2025), p. 239–255. Cited by: §2.1, §4.1. MMAction2 Contributors (2020) MMAction2 Contributors OpenMMLab’s next generation video understanding toolbox and benchmark. Note: https://github.com/open-mmlab/mmaction2 Cited by: §B.1, §4.1. Ren et al. (2023) H. Ren, W. Yang, T. Zhang, and Y. Zhang Proposal-based multiple instance learning for weakly-supervised temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2394–2404. Cited by: §2.2. Siméoni et al. (2025) O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haziza, L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts, A. Vedaldi, J. Tolan, J. Brandt, C. Couprie, J. Mairal, H. Jégou, P. Labatut, and P. Bojanowski DINOv3. External Links: 2508.10104 Cited by: §B.2, §3.2, §4.1. Tang et al. (2024a) W. Tang, Y. Yang, Z. Wang, W. Zhang, and M. Zhang Multi-instance partial-label learning with margin adjustment. In Advances in Neural Information Processing Systems 37, p. 26331–26354. Cited by: §2.1, §4.1. Tang et al. (2026) W. Tang, Y. Yang, W. Zhang, and M. Zhang Calibratable disambiguation loss for multi-instance partial-label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §2.1. Tang et al. (2023) W. Tang, W. Zhang, and M. Zhang Disambiguated attention embedding for multi-instance partial-label learning. In Advances in Neural Information Processing Systems 36, p. 56756–56771. Cited by: §1, §2.1, §4.1. Tang et al. (2024b) W. Tang, W. Zhang, and M. Zhang Exploiting conjugate label information for multi-instance partial-label learning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, p. 4973–4981. Cited by: §2.1, §4.1. Tang et al. (2024c) W. Tang, W. Zhang, and M. Zhang Multi-instance partial-label learning: towards exploiting dual inexact supervision. Science China Information Sciences 67 (3), p. 132103. Cited by: §1, §2.1, §4.1. Wang et al. (2022a) H. Wang, M. Xia, Y. Li, Y. Mao, L. Feng, G. Chen, and J. Zhao SoLar: sinkhorn label refinery for imbalanced partial-label learning. In Advances in Neural Information Processing Systems 35, p. 8104–8117. Cited by: §1. Wang et al. (2022b) H. Wang, R. Xiao, Y. Li, L. Feng, G. Niu, G. Chen, and J. Zhao PiCO: contrastive label disambiguation for partial label learning. In International Conference on Learning Representations, Cited by: §1. Wang et al. (2023) L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao VideoMAE V2: scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14549–14560. Cited by: §B.2, §3.2, §4.1. Wang et al. (2017) L. Wang, Y. Xiong, D. Lin, and L. Van Gool UntrimmedNets for weakly supervised action recognition and detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 6402–6411. Cited by: §1, §2.2. Wang et al. (2025) W. Wang, D. Wu, J. Wang, G. Niu, M. Zhang, and M. Sugiyama Realistic evaluation of deep partial-label learning algorithms. In International Conference on Learning Representations, Cited by: §1. Yang et al. (2024) Y. Yang, W. Tang, and M. Zhang ProMIPL: a probabilistic generative model for multi-instance partial-label learning. In 2024 IEEE International Conference on Data Mining (ICDM), p. 560–569. Cited by: §2.1, §4.1. Yang et al. (2025) Y. Yang, W. Tang, and M. Zhang Fast multi-instance partial-label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 22038–22046. Cited by: §2.1, §4.1. Yao et al. (2023) Y. Yao, X. Wang, M. Xu, Z. Pu, Y. Wang, E. Atkins, and D. J. Crandall DoTA: unsupervised detection of traffic anomaly in driving videos. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (1), p. 444–459. Cited by: §B.1, §1, §4.1. Yun et al. (2024) W. Yun, M. Qi, C. Wang, and H. Ma Weakly-supervised temporal action localization by inferring salient snippet-feature. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 6908–6916. Cited by: §2.2. Zhang et al. (2024) J. Zhang, Q. Wei, F. Liu, and L. Feng Candidate Pseudolabel Learning: enhancing vision-language models by prompt tuning with unlabeled data. In Proceedings of the 41st International Conference on Machine Learning, p. 60004–60020. Cited by: §B.1, §1, §4.1. Zhang et al. (2025a) Q. Zhang, J. Fang, R. Yuan, X. Tang, Y. Qi, K. Zhang, and C. Yuan Weakly supervised temporal action localization via dual-prior collaborative learning guided by multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24139–24148. Cited by: §2.2. Zhang et al. (2025b) Q. Zhang, Y. Qi, X. Tang, R. Yuan, X. Lin, K. Zhang, and C. Yuan Rethinking pseudo-label guided learning for weakly supervised temporal action localization from the perspective of noise correction. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 10085–10093. Cited by: §1. Zhu et al. (2026) Z. Zhu, K. Tang, S. Feng, Y. Tang, H. Wang, G. Lyu, C. Peng, and Y. Sun Dual graph disambiguation for multi-instance partial-label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 29223–29231. Cited by: §2.1, §4.1. Appendix A Structural Properties and Proofs We analyze the optimization problems defined in the paper, rather than the convergence of network training. Throughout this appendix, one bag has n≥1n≥ 1 valid temporal positions, finite costs CcktC_ckt, K≥1K≥ 1 prototypes per class, and a nonempty admissible class set A. We fix ε>0 >0, λ≥0λ≥ 0, and ut=1/nu_t=1/n. Padding positions are excluded from this domain. All plans below are exact optima unless numerical approximations are explicitly discussed. Derivative statements additionally assume that the actor is differentiable at the parameter point under consideration. These properties do not imply recovery of the true label, correct temporal localization, or generalization of the learned network. A.1 Factorization and Uniqueness Write Πckt=mtρckt _ckt=m_t _ckt, where ∑tmt=1 _tm_t=1 and ∑c∈,kρckt=1 _c ,k _ckt=1. For any positive mtm_t, conditional minimization over ρ⋅t _· t gives Zt=∑c∈,ke−Cckt/ε,ρckt=e−Cckt/εZt,ft=−εlogZt.Z_t^A= _c ,ke^-C_ckt/ , _ckt^A= e^-C_ckt/ Z_t^A, f_t^A=- Z_t^A. (22) Indeed, its cost plus conditional entropy equals ft+εDKL(ρ⋅t∥ρ⋅t)f_t^A+ D_KL( _· t\|ρ^A_· t). The full objective therefore reduces, up to a constant independent of m, to Gλ(m)=⟨f,m⟩+ε∑tmtlog(mt/ut)+λ2R(m),R(m)=n∑tmt2−1,m∈Δn.G_λ(m)= f,m + _tm_t (m_t/u_t)+ λ2R(m), R(m)=n _tm_t^2-1, m∈ _n. (23) This is an optimization over an n-dimensional vector subject to one simplex constraint; only its normalization multiplier is a scalar root variable. Proposition 1 (Unique Interior Optimum). For finite f, the objective in Eq. (23) has a unique minimizer m⋆m with mt⋆>0m_t >0 for every valid t. Its necessary and sufficient optimality conditions are ft+εlog(mt⋆/ut)+λnmt⋆=ξ,∑tmt⋆=1.f_t+ (m_t /u_t)+λ nm_t =ξ, _tm_t =1. (24) Proof. The continuous extension of xlog(x/u)x (x/u) at zero is strictly convex on [0,∞)[0,∞). For positive arguments, this follows from its second derivative 1/x1/x; if one argument is zero, for y>0y>0 and a∈(0,1)a∈(0,1), (ay)log(ay/u)=aylog(y/u)+ayloga<aylog(y/u)(ay) (ay/u)=ay (y/u)+ay a<ay (y/u). Thus, the entropy is strictly convex on the whole simplex. The other terms are linear or convex, and compactness gives existence and uniqueness. A boundary point with mt=0m_t=0 can be perturbed by transferring δ>0δ>0 mass from a positive component to t. Its objective difference is εδlogδ+O(δ)<0 δ δ+O(δ)<0 for small δ, so it cannot minimize the objective. At the interior optimum, differentiating the Lagrangian and absorbing its common entropy constant into ξ gives Eq. (24). Strict convexity makes these conditions sufficient. For n=1n=1, the unique feasible point is already positive. ∎ For λ>0λ>0, the principal Lambert-W function yields mt(ξ)=ελnW(λεe(ξ−ft)/ε),H(ξ)=∑tmt(ξ)−1.m_t(ξ)= λ nW\! ( λ e^(ξ-f_t)/ ), H(ξ)= _tm_t(ξ)-1. (25) Implicit differentiation of Eq. (24) at fixed f gives H′(ξ)=∑tmt(ξ)ε+λnmt(ξ)>0.H (ξ)= _t m_t(ξ) +λ nm_t(ξ)>0. (26) The sum in H tends to zero as ξ→−∞ξ→-∞ and to infinity as ξ→∞ξ→∞, so the root is unique. It obeys the finite bracket mintft+λ≤ξ⋆≤maxtft+λ. _tf_t+λ\ ≤\ ξ \ ≤\ _tf_t+λ. (27) To see this, the left side of Eq. (24), as a function of mtm_t, is strictly increasing and equals ft+λf_t+λ at mt=utm_t=u_t. At the lower bracket, each mt(ξ)≤utm_t(ξ)≤ u_t, and at the upper bracket, each mt(ξ)≥utm_t(ξ)≥ u_t. Eq. (27) also provides a valid bracket for a safeguarded Newton–bisection solver. Our implementation instead uses the fixed-iteration Newton procedure described in Appendix B.3, without bisection. If all ftf_t are equal, the bracket is a single exact root. The case λ=0λ=0 must be handled separately: mt⋆=e−ft/ε∑se−fs/ε,ξ⋆=−εlog∑tute−ft/ε.m_t = e^-f_t/ _se^-f_s/ , ξ =- _tu_te^-f_t/ . (28) It retains entropic smoothing. Substitution of zero into the expression with 1/λ1/λ in Eq. (25) is invalid. A.2 Candidate-Restricted Re-solving and Global Masking Let ΠA ^A solve the problem over all classes, and let Π~ solve it over a nonempty candidate set S, using exactly the same C,ε,λ,n,uC, ,λ,n,u. Define the globally masked actor plan and its retained conditional mass by at=∑c∈S,kρcktA>0,Q=∑tmtAat>0,ΠcktR=[c∈S]ΠAcktQ,mtR=mtAatQ.a_t= _c∈ S,kρ^A_ckt>0, Q= _tm_t^Aa_t>0, ^R_ckt= 1[c∈ S] ^A_cktQ, m_t^R= m_t^Aa_tQ. (29) All comparisons with Π~ extend its noncandidate entries by zero. Thus global masking generally changes temporal occupancy. This operation differs both from masking only a class probability vector and from renormalizing the conditional class distribution separately at each time while explicitly retaining mAm^A. Proposition 2 (When Re-solving Differs From Masking). If λ=0λ=0, then Π~=ΠR = ^R. If λ>0λ>0, then Π~=ΠR = ^R if and only if ata_t is constant over valid temporal positions. Proof. The restricted conditional distribution is ρ~ckt=ρcktA/at ρ_ckt=ρ^A_ckt/a_t for c∈Sc∈ S, and f~t=ftA−εlogat f_t=f_t^A- a_t. The conditional distribution of ΠR ^R is exactly ρ~ ρ, so only the occupancies can differ. When λ=0λ=0, Eq. (28) and Eq. (22) imply ΠAckt∝e−Cckt/ε ^A_ckt e^-C_ckt/ globally; restricting and normalizing this Gibbs distribution is the restricted optimum. For λ>0λ>0, insertion of mRm^R into the restricted stationarity expression gives f~t+εlog(mtR/ut)+λnmtR=ξA−εlogQ+λn(mtR−mtA). f_t+ (m_t^R/u_t)+λ nm_t^R=ξ^A- Q+λ n(m_t^R-m_t^A). (30) If ΠR ^R is optimal, the right side must be constant in t. Since λn>0λ n>0, mtR−mtAm_t^R-m_t^A must be constant; its sum is zero, so mR=mAm^R=m^A. Positivity of mAm^A and Eq. (29) imply at=Qa_t=Q for all t. Conversely, constant ata_t gives mR=mAm^R=m^A, making Eq. (30) constant. The unique interior optimum established above then implies ΠR=Π~ ^R= . ∎ Consequently, re-solving enforces the occupancy-regularized optimality conditions within the candidate domain. Its distinction from global masking does not follow merely from the fact that temporal occupancy changes. The result assumes the uniform prior and exact objectives stated above; it does not assert equivalence for modified priors, regularizers, or solvers. Why Retain The Association Between Class And Time? As an illustration, consider two generic class–time distributions, U=(0.40.10.10.4),V=(0.10.40.40.1).U= pmatrix0.4&0.1\\ 0.1&0.4 pmatrix, V= pmatrix0.1&0.4\\ 0.4&0.1 pmatrix. (31) Both have class and temporal marginals (0.5,0.5)(0.5,0.5), yet associate each class with a different time. Their class-marginal KL is zero, while DKL(U∥V)=0.6log4>0D_ KL(U\|V)=0.6 4>0. This example concerns distributions, not measured video predictions or an asserted actor–teacher pair. It shows why marginal agreement alone leaves the class–time association unspecified. In our construction, proximity to the proposal selects a joint target among the plans that satisfy both marginals. A.3 Teacher projection and the effective supervision space Let q be the refined candidate-class marginal and m~ m the current proposal occupancy. Finite costs and ε>0 >0 give a positive proposal on S×1,…,K×1,…,nS×\1,…,K\×\1,…,n\. The stated initialization and EMA with μ∈[0,1)μ∈[0,1) give qc>0q_c>0 on S. The KL projection with class marginal q and temporal marginal m~ m has a unique solution. A strictly positive feasible point exists, for example qcm~t/Kq_c m_t/K; the entropy boundary argument ensures that the minimizing plan is positive on this domain. Its Lagrange stationarity condition gives ΠcktT=αcβtΠ~ckt,αc,βt>0. ^T_ckt= _c _t _ckt, _c, _t>0. (32) Although the factors have a reciprocal scaling ambiguity, the plan is unique. With M~ct=∑kΠ~ckt M_ct= _k _ckt, alternating marginal scaling is αc←qc∑tM~ctβt,βt←m~t∑c∈SM~ctαc. _c← q_c _t M_ct _t, _t← m_t _c∈ S M_ct _c. (33) These updates need only the |S|×n|S|× n class–time matrix. They can also be implemented in the log domain. If a target class has exactly zero mass in an alternative initialization, remove that row and apply the argument on the positive target support; extend it by zeros afterwards. Assume that actor and proposal are constructed at the same parameter point θ0 _0, using the same costs, prototypes, features, and ε , and that the teacher correction depends only on (c,t)(c,t). Then rk|c,tA=rk|c,tT=e−Cckt/ε∑k′e−Cck′t/ε(c∈S).r^A_k c,t=r^T_k c,t= e^-C_ckt/ _k e^-C_ck t/ (c∈ S). (34) The temporal multiplier in the actor, the restricted partition function in the proposal, and the teacher scaling in Eq. (32) all cancel when conditioning on (c,t)(c,t). Proposition 3 (KL and Current-point Gradient Equivalence). Set MctA/T=∑kΠcktA/TM^A/T_ct= _k ^A/T_ckt and extend teacher plans and marginals with zeros outside S (denoted by hats). Under the conditions just stated, DKL(Π^T∥ΠA) D_KL( ^T\| ^A) =DKL(M^T∥MA), =D_KL( M^T\|M^A), (35) =DKL(qT∥qA)+∑c:qcT>0qcTDKL(PT(t∣c)∥PA(t∣c)), =D_KL(q^T\|q^A)+ _c:q_c^T>0q_c^TD_KL (P_T(t c)\|P_A(t c) ), (36) where qcT=∑tM^ctTq_c^T= _t M^T_ct is the actual teacher class marginal. If this teacher is frozen, the three-dimensional and two-dimensional losses also have equal first derivatives with respect to actor parameters at θ=θ0θ= _0. Proof. The KL chain rule applied to (c,t)(c,t) and then to k gives DKL(Π^T∥ΠA)=DKL(M^T∥MA)+∑c,t:M^ctT>0M^ctTDKL(r⋅|c,tT∥r⋅|c,tA).D_KL( ^T\| ^A)=D_KL( M^T\|M^A)+ _c,t: M^T_ct>0 M^T_ctD_KL(r^T_· c,t\|r^A_· c,t). (37) Eq. (34) makes the last term zero. Applying the same chain rule to class and time in M proves Eq. (36). For the derivative statement, denote the teacher built at θ0 _0 by T0T_0 and hold it fixed while differentiating the actor. The difference between the two loss gradients is −∑c,tM^ctT0∑krk|c,tT0∇θlogrk|c,tA(θ).- _c,t M^T_0_ct _kr^T_0_k c,t _θ r^A_k c,t(θ). (38) At θ0 _0, rT0=rA(θ0)r^T_0=r^A( _0), so every inner sum is ∑k∇θrk|c,tA(θ0)=∇θ1=0 _k _θr^A_k c,t( _0)= _θ1=0. This proves local first-order equivalence despite stopping gradients through the teacher; equality of values alone would not justify that derivative conclusion. ∎ The extra supervision beyond the class marginal is therefore the class-conditional temporal distribution, parameterized by multi-prototype geometry. Prototype parameters still influence ∑ke−Cckt/ε _ke^-C_ckt/ and generally receive nonzero gradients. There is no additional nonzero prototype-conditional KL term under these conditions. The result does not extend to stale teachers reused after actor updates, different teacher costs or temperatures, prototype-dependent corrections, or higher derivatives of a frozen-teacher loss. For an approximate multiplicative projection, conditional preservation still holds, but Eq. (36) uses its actual marginal qTq^T, which need not equal the requested q until projection residuals vanish. A.4 Teacher Projection and Class–Time Supervision Proposition 4 (No Momentum And No Occupancy Penalty). Under the same-cost and exact-solution assumptions, let μ=0μ=0 and λ=0λ=0. Then the teacher equals the candidate proposal and DKL(Π^T∥ΠA)=−logQ,Q=∑c∈SqcA.D_KL( ^T\| ^A)=- Q, Q= _c∈ Sq_c^A. (39) The stopped-teacher gradient at its construction point also equals the gradient of −logQ- Q. Proof. When μ=0μ=0, the refined marginal is the proposal class marginal. The proposal already satisfies both teacher constraints and has zero KL distance to itself, so the uniqueness of the projection implies ΠT=Π~ ^T= . The masking proposition with λ=0λ=0 gives Π^T=[c∈S]ΠA/Q ^T=1[c∈ S] ^A/Q. The log ratio on its support is the constant −logQ- Q, proving the value identity. At the teacher construction point, differentiation with that teacher fixed gives −∑c∈S,k,tΠcktAQ∇θlogΠcktA=−1Q∇θ∑c∈S,k,tΠcktA=∇θ[−logQ].- _c∈ S,k,t ^A_cktQ _θ ^A_ckt=- 1Q _θ _c∈ S,k,t ^A_ckt= _θ[- Q]. (40) No derivative through the teacher is used. ∎ This is a boundary configuration, not an identity for the full method with positive momentum or occupancy regularization. It motivates a controlled candidate-mass baseline without asserting its relative empirical performance. A.5 Effective-Support Control by Occupancy Regularization Define Neff(m)=1/∑tmt2N_eff(m)=1/ _tm_t^2, so that 1≤Neff(m)≤n1≤ N_eff(m)≤ n and R(m)=nNeff(m)−1=χ2(m∥u)=exp(D2(m∥u))−1.R(m)= nN_eff(m)-1=χ^2(m\|u)= (D_2(m\|u))-1. (41) Proposition 5 (Fixed-geometry Monotonicity). For fixed f,ε>0,nf, >0,n, R(mλ⋆)R(m_λ ) is nonincreasing and Neff(mλ⋆)N_eff(m_λ ) is nondecreasing in λ≥0λ≥ 0. Moreover, mλ⋆→um_λ → u as λ→∞λ→∞. Proof. Write F(m)=⟨f,m⟩+εDKL(m∥u)F(m)= f,m + D_KL(m\|u). For λ2>λ1≥0 _2> _1≥ 0, optimality at the two coefficients gives F(m1)+λ12R(m1) F(m_1)+ _12R(m_1) ≤F(m2)+λ12R(m2), ≤ F(m_2)+ _12R(m_2), F(m2)+λ22R(m2) F(m_2)+ _22R(m_2) ≤F(m1)+λ22R(m1). ≤ F(m_1)+ _22R(m_1). Adding them yields (λ2−λ1)(R(m2)−R(m1))≤0( _2- _1)(R(m_2)-R(m_1))≤ 0. Eq. (41) gives the support statement. Since R(u)=0R(u)=0, comparison with the feasible u gives, for λ>0λ>0, 0≤R(mλ⋆)≤2[F(u)−minm∈ΔnF(m)]λ.0≤ R(m_λ )≤ 2[F(u)- _m∈ _nF(m)]λ. (42) The numerator is finite by compactness. As R(m)=n‖m−u‖22R(m)=n\|m-u\|_2^2, the bound implies convergence to u. At λ=0λ=0, Eq. (28) gives the entropy-smoothed Gibbs occupancy. ∎ This statement holds with geometry fixed. During learning f changes, so it does not establish monotonic support across training updates or monotonic accuracy in λ. Effective support measures mass concentration, not semantic diversity or true-action coverage; overly diffuse mass can also assign weight to the background around a short action. A.6 Differentiability and Computational Complexity For an exact interior solution, set wt=(ε/mt⋆+λn)−1>0w_t=( /m_t +λ n)^-1>0. Differentiating Eq. (24) gives dmt=wt(dξ−dft)dm_t=w_t(dξ-df_t), and ∑tdmt=0 _tdm_t=0 implies dξ=(∑twtdft)/(∑twt)dξ=( _tw_tdf_t)/( _tw_t). Hence ∂m⋆∂f=−diag(w)+ww⊤∑twt. ∂ m ∂ f=-diag(w)+ w _tw_t. (43) This matrix is symmetric and negative semidefinite: for any vector v, its quadratic form is the negative weighted variance −∑twtvt2+(∑twtvt)2/∑twt≤0- _tw_tv_t^2+( _tw_tv_t)^2/ _tw_t≤ 0. Its null direction 1 reflects invariance to adding a common constant to f. The formula also covers λ=0λ=0, yielding the usual softmax Jacobian with a negative inverse-temperature factor. A Jacobian–vector product costs O(n)O(n) without materializing an n×n× n matrix. Differentiation of the complete actor must additionally include ∂ft/∂Cckt=ρckt∂ f_t/∂ C_ckt= _ckt and the conditional Gibbs derivative; retaining only the occupancy path would change the gradient. For feature dimension d, the dense spherical matching costs O(CKnd)O(CKnd) operations and O(CKn)O(CKn) storage when scores are materialized. Given these scores, the conditional distributions and free energies cost O(CKn)O(CKn). With JAJ_A and JSJ_S scalar root iterations, the actor and proposal occupancy solves cost O((JA+JS)n)O((J_A+J_S)n), in addition to the O(|S|Kn)O(|S|Kn) restricted reductions. With I marginal-scaling iterations, projection on M~ M costs O(I|S|n)O(I|S|n); a three-dimensional teacher, if materialized, costs O(|S|Kn)O(|S|Kn) storage. These counts exclude the temporal encoder and depend on the requested numerical precision. The scalar normalization root does not remove the class–prototype matching cost. For numerical evaluation, define ht=log(λ/ε)+(ξ−ft)/εh_t= (λ/ )+(ξ-f_t)/ and let ωt=W(eht) _t=W(e^h_t). Rather than explicitly forming ehte^h_t, we evaluate ωt _t from ωt+logωt=ht _t+ _t=h_t using stable Newton updates and recover mt=εωt/(λn)m_t= _t/(λ n). The implementation uses eight fixed Lambert-W updates per evaluation and twelve Newton iterations for the normalization root, with the latter performed in double precision. The λ=0λ=0 branch uses the closed-form softmax solution. An approximate solver should report its mass residual |∑tmt−1|| _tm_t-1|, stationarity residual maxt|ft+εlog(mt/ut)+λnmt−ξ| _t|f_t+ (m_t/u_t)+λ nm_t-ξ|, and both teacher marginal residuals ‖∑k,tΠcktT−qc‖∞\| _k,t ^T_ckt-q_c\|_∞ and ‖∑c,kΠcktT−m~t‖∞\| _c,k ^T_ckt- m_t\|_∞. Appendix B Dataset construction and implementation details B.1 More Information on Benchmark Construction We construct VCMIPL benchmarks from Breakfast (Kuehne et al., 2014), DoTA (Yao et al., 2023), and FineAction (Liu et al., 2022). Table 3 summarizes the video counts, class counts, and candidate-set sizes. Each example consists of an ordered video bag and a candidate label set containing its ground-truth class. For FineAction, we retain videos with a single annotated target class and exclude classes containing fewer than 50 videos. The resulting benchmarks contain 1,989 videos from 10 classes for Breakfast, 4,577 videos from 9 classes for DoTA, and 10,675 videos from 78 classes for FineAction. In the output space, we generate candidate labels using a VideoMAEv2 recognizer equipped with a TimeSformerHead in MMAction2 (MMAction2 Contributors, 2020). Candidate probabilities are obtained through five-fold cross-fitting: the recognizer is fine-tuned using the exact video labels from four folds and predicts the remaining fold. Repeating this procedure produces an out-of-fold prediction for every video. The recognizer is fine-tuned for 30 epochs on DoTA and 100 epochs on both Breakfast and FineAction. Following the sample-wise confidence accumulation and class-wise selection principles of CPL (Zhang et al., 2024), we combine two criteria. Let i=(pi1,…,piC)p_i=(p_i1,…,p_iC) denote the out-of-fold class-probability vector for video i, and let ci1,…,ciCc_i1,…,c_iC order the classes by decreasing confidence. The sample-wise candidate set contains the smallest number of classes whose cumulative probability reaches 95%95\%: Ai=ci1,…,ciℓi,ℓi=minℓ∈1,…,C:∑j=1ℓpi,cij≥0.95.A_i=\c_i1,…,c_i _i\, _i= \ ∈\1,…,C\: _j=1 p_i,c_ij≥ 0.95 \. (44) For each class c, let BcB_c denote the indices of the top 50%50\% of videos in the class-wise ranking population, ordered by their confidence for that class. Intersecting the two selections gives S~i=Ai∩c∈:i∈Bc. S_i=A_i∩\c :i∈ B_c\. (45) To satisfy the standard MIPL assumption, we include the ground-truth class whenever it is absent: Si=S~i∪Yi.S_i= S_i∪\Y_i\. (46) Consequently, the final candidate set is nonempty and satisfies Yi∈SiY_i∈ S_i by construction. This guarantee applies to the corrected candidate sets, rather than to the recognizer’s predictions before correction. Exact labels are used to train the candidate recognizer and perform this inclusion step during benchmark construction; downstream MIPL learning uses only the resulting candidate sets as supervision. Table 3: Video-level statistics of the constructed MIPL benchmarks. Dataset Videos Classes Mean |Si||S_i| Range |Si||S_i| Breakfast 1,989 10 3.3896 1–9 DoTA 4,577 9 3.2733 1–8 FineAction 10,675 78 11.5592 2–64 B.2 Feature Representations and Dataset Statistics ResNet (He et al., 2016) and DINOv3 (Siméoni et al., 2025) provide frame-level features, whereas SlowFast (Feichtenhofer et al., 2019) and VideoMAEv2 (Wang et al., 2023) provide clip-level features. These representations yield twelve dataset–feature configurations spanning different temporal granularities and feature dimensions. Table 4 reports the complete instance statistics. Total instances denotes the number of extracted frame or clip features across all videos; maximum, minimum, and mean refer to the number of instances per video. Dimension denotes the width of each feature vector. Within each dataset, the four representations share the video, class, and candidate-set statistics in Table 3. The fine-tuning budgets in the dataset headers refer to the candidate recognizer, not to the individual feature extractors. Table 4: Instance and feature statistics for all twelve dataset–feature configurations. Feature Total instances Maximum Minimum Mean Dimension Breakfast: 1,989 videos, 10 classes, 100 fine-tuning epochs Candidate size: 1–9, mean 3.3896 ResNet 4,103,316 9,745 186 2,063.00 2,048 DINOv3 4,103,316 9,745 186 2,063.00 4,096 SlowFast 1,026,577 2,437 47 516.13 2,304 VideoMAEv2 1,026,577 2,437 47 516.13 768 DoTA: 4,577 videos, 9 classes, 30 fine-tuning epochs Candidate size: 1–8, mean 3.2733 ResNet 469,756 300 26 102.63 2,048 DINOv3 469,756 300 26 102.63 4,096 SlowFast 119,188 75 7 26.04 2,304 VideoMAEv2 119,188 75 7 26.04 768 FineAction: 10,675 videos, 78 classes, 100 fine-tuning epochs Candidate size: 2–64, mean 11.5592 ResNet 37,723,340 64,644 36 3,533.80 2,048 DINOv3 37,723,340 64,644 36 3,533.80 4,096 SlowFast 9,518,494 47,372 9 891.66 2,304 VideoMAEv2 9,518,494 47,372 9 891.66 768 B.3 More Implementation Settings We adopt a fixed five-fold evaluation protocol for all three benchmarks. In each fold, four predefined folds are used for training and the remaining fold for testing, and the reported ACC and BACC are averaged over the five held-out folds. Candidate labels are generated in an out-of-fold manner using the same fold partition: the candidate recognizer is fine-tuned on four folds and applied to the held-out fold, and this procedure is repeated until every video receives an out-of-fold prediction. The downstream MIPL models are then trained using only the resulting candidate label sets as supervision. PivotMipl is optimized for 40 epochs with one complete video bag per optimization step, without temporal padding, and the same training protocol is used across all feature representations unless otherwise specified. Ablation studies are conducted on Fold 1 and evaluated at epoch 40. Detailed optimization, architecture, transport, and teacher-projection settings are summarized in Table 5. Table 5: Experimental settings and configurable implementation defaults of PivotMipl. Setting Value or implementation Framework / hardware PyTorch / NVIDIA H100 Optimizer AdamW; (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999); weight decay 00 Learning rates Prototypes: 5×10−35× 10^-3; temporal encoder: 5×10−45× 10^-4 Gradient clipping Global ℓ2 _2 norm clipped to 5.05.0 Training budget 4040 epochs Motion module Average-filter kernel 33; convolution kernel 55; dilations 1,21,2; motion-residual dropout 0.10.1 Candidate recognizer budget DoTA: 3030 epochs; Breakfast and FineAction: 100100 epochs Main evaluation Five folds; mean and standard deviation of ACC and BACC Ablation evaluation Fold 1 ACC at epoch 40 Learning-rate schedule Constant; no warm-up or scheduler Batch size One complete video bag per optimization step Prototypes per class Breakfast: K=4K=4; DoTA: K=8K=8; FineAction: K=12K=12 Class momentum μ=0.9μ=0.9 Transport regularization Entropic coefficient ε=0.07 =0.07; occupancy coefficient λ=0.01λ=0.01 Temporal encoder Hidden dimension 512512; 44 bidirectional Mamba-2 blocks Feed-forward network Expansion factor 44; intermediate dimension 20482048 Mamba-2 configuration State dimension 128128; convolution width 44; expansion factor 22 Mamba-2 computation Head dimension 6464; chunk size 256256; memory-efficient path enabled Teacher projection At most 120120 IPFP iterations; marginal tolerance 10−610^-6 Base random seed 123123, with fold-dependent offsets Appendix C Additional Experimental Results C.1 Balanced Accuracy Results Table 6 further confirms the advantage of PivotMipl under class-balanced evaluation. PivotMipl achieves the highest mean BACC, the mean recall over ground-truth classes represented in each test fold, in 10 of the 12 dataset–feature configurations, with an average gain of 3.26% over the strongest baseline. Notably, the improvement pattern closely follows that of ACC, indicating that the gains are not primarily driven by better prediction on dominant classes, but extend across the class distribution. PivotMipl also ranks first in all four DoTA-MIPL configurations, where the substantially lower BACC values make balanced recognition particularly challenging. The only exceptions again occur with VideoMAEv2, suggesting that the reduced headroom observed in the ACC results persists under class-balanced evaluation. Table 6: Balanced accuracy on the benchmark datasets. Results are reported as (mean ± std %). The best result in each column is bolded, and the second-best results are underlined. Dataset Method DINOv3 VideoMAEv2 ResNet SlowFast Breakfast -MIPL DeMipl 10.2%± 0.4%10.2\%\, ±\,0.4\% 24.4%± 10.2%24.4\%\, ±\,10.2\% 10.6%± 0.6%10.6\%\, ±\,0.6\% 10.0%± 0.0%10.0\%\, ±\,0.0\% EliMipl 27.9%± 0.8%27.9\%\, ±\,0.8\% 52.4%± 2.1%52.4\%\, ±\,2.1\% 26.0%± 1.9%26.0\%\, ±\,1.9\% 10.0%± 0.0%10.0\%\, ±\,0.0\% MiplMa 34.2%± 1.5%34.2\%\, ±\,1.5\% 63.8%± 2.0%63.8\%\, ±\,2.0\% 30.8%± 2.6%30.8\%\, ±\,2.6\% 10.0%± 0.0%10.0\%\, ±\,0.0\% ProMipl 61.9%¯± 1.1% 61.9\%\, ±\,1.1\% 83.3%± 3.5%83.3\%\, ±\,3.5\% 43.7%¯± 1.0% 43.7\%\, ±\,1.0\% 41.4%± 1.1%41.4\%\, ±\,1.1\% PsMipl 47.7%± 4.9%47.7\%\, ±\,4.9\% 70.1%± 7.1%70.1\%\, ±\,7.1\% 10.9%± 1.4%10.9\%\, ±\,1.4\% 32.9%± 0.5%32.9\%\, ±\,0.5\% DualG 45.2%± 2.7%45.2\%\, ±\,2.7\% 65.1%± 4.1%65.1\%\, ±\,4.1\% 34.7%± 2.2%34.7\%\, ±\,2.2\% 13.8%± 1.7%13.8\%\, ±\,1.7\% AgopMipl 46.0%± 4.3%46.0\%\, ±\,4.3\% 87.2%± 4.3%87.2\%\, ±\,4.3\% 33.3%± 3.3%33.3\%\, ±\,3.3\% 50.1%¯± 2.4% 50.1\%\, ±\,2.4\% PivotMipl 66.0%± 1.0%66.0\%\, ±\,1.0\% 85.2%¯± 4.3% 85.2\%\, ±\,4.3\% 52.2%± 3.3%52.2\%\, ±\,3.3\% 55.8%± 1.8%55.8\%\, ±\,1.8\% DoTA -MIPL DeMipl 24.8%± 0.7%24.8\%\, ±\,0.7\% 42.7%± 0.5%42.7\%\, ±\,0.5\% 23.2%± 0.4%23.2\%\, ±\,0.4\% 18.2%± 2.6%18.2\%\, ±\,2.6\% EliMipl 24.8%± 0.8%24.8\%\, ±\,0.8\% 42.9%± 0.8%42.9\%\, ±\,0.8\% 23.9%± 0.6%23.9\%\, ±\,0.6\% 21.2%± 1.7%21.2\%\, ±\,1.7\% MiplMa 25.8%± 0.6%25.8\%\, ±\,0.6\% 43.2%± 0.5%43.2\%\, ±\,0.5\% 23.9%± 0.5%23.9\%\, ±\,0.5\% 21.5%± 3.4%21.5\%\, ±\,3.4\% ProMipl 32.7%¯± 1.6% 32.7\%\, ±\,1.6\% 58.0%± 1.5%58.0\%\, ±\,1.5\% 28.8%¯± 2.0% 28.8\%\, ±\,2.0\% 28.8%¯± 0.8% 28.8\%\, ±\,0.8\% PsMipl 30.0%± 0.9%30.0\%\, ±\,0.9\% 58.3%¯± 1.4% 58.3\%\, ±\,1.4\% 26.2%± 0.5%26.2\%\, ±\,0.5\% 14.9%± 4.7%14.9\%\, ±\,4.7\% DualG 26.0%± 0.7%26.0\%\, ±\,0.7\% 44.1%± 0.6%44.1\%\, ±\,0.6\% 24.9%± 0.8%24.9\%\, ±\,0.8\% 24.2%± 0.9%24.2\%\, ±\,0.9\% AgopMipl 28.6%± 0.9%28.6\%\, ±\,0.9\% 51.4%± 3.1%51.4\%\, ±\,3.1\% 26.1%± 1.0%26.1\%\, ±\,1.0\% 27.5%± 1.0%27.5\%\, ±\,1.0\% PivotMipl 33.8%± 1.5%33.8\%\, ±\,1.5\% 63.2%± 3.5%63.2\%\, ±\,3.5\% 31.8%± 1.2%31.8\%\, ±\,1.2\% 31.8%± 0.7%31.8\%\, ±\,0.7\% FineAction -MIPL DeMipl 36.0%± 1.8%36.0\%\, ±\,1.8\% 48.8%± 2.8%48.8\%\, ±\,2.8\% 18.3%± 1.3%18.3\%\, ±\,1.3\% 44.6%± 2.1%44.6\%\, ±\,2.1\% EliMipl 34.8%± 1.6%34.8\%\, ±\,1.6\% 44.0%± 2.7%44.0\%\, ±\,2.7\% 18.9%± 2.0%18.9\%\, ±\,2.0\% 44.7%± 1.4%44.7\%\, ±\,1.4\% MiplMa 49.6%± 2.0%49.6\%\, ±\,2.0\% 61.1%± 0.3%61.1\%\, ±\,0.3\% 30.6%± 2.7%30.6\%\, ±\,2.7\% 51.4%± 2.2%51.4\%\, ±\,2.2\% ProMipl 79.0%± 1.3%79.0\%\, ±\,1.3\% 91.0%± 3.5%91.0\%\, ±\,3.5\% 66.5%± 1.6%66.5\%\, ±\,1.6\% 80.2%± 2.0%80.2\%\, ±\,2.0\% PsMipl 57.2%± 1.1%57.2\%\, ±\,1.1\% 63.1%± 1.6%63.1\%\, ±\,1.6\% 39.4%± 1.8%39.4\%\, ±\,1.8\% 29.2%± 14.1%29.2\%\, ±\,14.1\% DualG 53.7%± 0.6%53.7\%\, ±\,0.6\% 66.3%± 1.2%66.3\%\, ±\,1.2\% 40.3%± 1.1%40.3\%\, ±\,1.1\% 57.4%± 0.7%57.4\%\, ±\,0.7\% AgopMipl 81.7%¯± 1.1% 81.7\%\, ±\,1.1\% 95.2%± 1.2%95.2\%\, ±\,1.2\% 67.5%¯± 1.6% 67.5\%\, ±\,1.6\% 82.0%¯± 3.0% 82.0\%\, ±\,3.0\% PivotMipl 86.3%± 1.5%86.3\%\, ±\,1.5\% 92.0%¯± 1.7% 92.0\%\, ±\,1.7\% 73.6%± 1.1%73.6\%\, ±\,1.1\% 85.3%± 1.4%85.3\%\, ±\,1.4\% C.2 Efficiency Analysis We also evaluate the efficiency of PivotMipl, presented in Figure 2. Figure 2 compares classification accuracy against training time. PivotMipl exhibits a favorable accuracy–efficiency trade-off across most dataset–feature configurations; the exceptions occur in the VideoMAEv2 settings where AgopMipl can achieve a Pareto-superior operating point. Figure 2: Accuracy–training-time trade-off across the twelve dataset–feature configurations. Each panel corresponds to one dataset–feature setting and compares PivotMipl with the MIPL baselines in terms of mean classification accuracy and total training time over five folds. Higher accuracy and shorter training time are preferred. C.3 Ablation Analysis C.3.1 Ablation Summaries Table 7 shows that the gains of PivotMipl are distributed across multiple components rather than dominated by a single design choice. Replacing the plan-level objective with class-level KL (A3) or removing the temporal-marginal constraint from the teacher (A4) degrades performance in all 12 configurations, highlighting joint class–time supervision and dual-marginal projection as the most structurally consistent contributors. Removing the temporal encoder (A6) causes the largest average drop, while the strong performance gap retained against encoder-matched baselines indicates that temporal contextualization alone does not explain the full advantage. Finally, A1, A2, and A5 show that occupancy regularization, adaptive temporal allocation, and multiple prototypes each provide complementary gains, supporting the need to model both temporal concentration and intra-class variation. Table 7: Summary of the ablations across twelve dataset–feature configurations. Variant Modification Full-model wins Mean gain A1 No occupancy regularization 10/12 +3.25%+3.25\% A2 Uniform temporal occupancy 9/12 +0.72%+0.72\% A3 Class-level KL 12/12 +3.26%+3.26\% A4 Class-marginal-only teacher 12/12 +2.53%+2.53\% A5 Single prototype per class 11/12 +2.52%+2.52\% A6 No temporal encoder 11/12 +5.77%+5.77\% C.3.2 Analysis Of The Effectiveness Of The Temporal Encoder Table 8: Accuracy for AgopMipl, AgopMipl augmented with the same temporal encoder architecture as PivotMipl, and PivotMipl. The best and second-best values within each row are shown in bold and underlined, respectively. Dataset Feature AgopMipl AgopMipl ++ Temporal Encoder PivotMipl Breakfast DINOv3 46.4%46.4\% 53.1%¯ 53.1\% 66.4%66.4\% VideoMAEv2 79.3%79.3\% 78.1%¯ 78.1\% 77.4%77.4\% ResNet 31.4%31.4\% 45.4%¯ 45.4\% 52.2%52.2\% SlowFast 50.6%50.6\% 53.1%¯ 53.1\% 53.9%53.9\% DoTA DINOv3 49.2%¯ 49.2\% 48.4%48.4\% 54.3%54.3\% VideoMAEv2 69.5%69.5\% 69.9%¯ 69.9\% 72.8%72.8\% ResNet 46.0%46.0\% 48.6%¯ 48.6\% 50.9%50.9\% SlowFast 46.9%46.9\% 47.6%¯ 47.6\% 48.4%48.4\% FineAction DINOv3 85.5%85.5\% 86.8%¯ 86.8\% 90.0%90.0\% VideoMAEv2 96.7%¯ 96.7\% 97.2%97.2\% 92.4%92.4\% ResNet 73.1%73.1\% 75.4%¯ 75.4\% 78.1%78.1\% SlowFast 87.9%87.9\% 89.2%¯ 89.2\% 90.0%90.0\% To examine whether adding temporal encoding alone can reproduce the performance advantage of PivotMipl, we equip AgopMipl, the strongest baseline, with the same temporal encoder architecture. We compare the original AgopMipl, its encoder-augmented variant, and PivotMipl. The result of the above shows PivotMipl retains higher ACC in 10 configurations, with an average advantage of 2.83%2.83\% over the encoder-augmented baseline. Together with A6, this comparison suggests that temporal encoding contributes to performance, but adding the same encoder architecture to AgopMipl does not by itself account for the observed advantage of PivotMipl across most configurations. C.4 Parameter Sensitivity Analysis We examine the sensitivity of PivotMipl to the two transport-specific hyperparameters, the entropic coefficient ε and the occupancy coefficient λ in Eq. (8), using three representative Fold-1 configurations: Breakfast–ResNet, DoTA–VideoMAEv2, and FineAction–SlowFast. As shown in Figure 3, performance varies only moderately over the evaluated ranges for both parameters, despite substantial differences in dataset scale and feature representation. The two parameters play complementary roles in the transport inference: ε controls the sharpness of the class–prototype–time assignment, whereas λ regulates the concentration of the temporal marginal. A very small ε favors sharper matching, while increasing it produces smoother assignments; similarly, increasing λ discourages excessive concentration of mass on a small number of temporal positions. The relatively broad performance plateaus therefore suggest that PivotMipl does not depend on a narrowly tuned balance between these two effects. Instead, the structured teacher and plan-level supervision remain effective under a range of assignment smoothness and temporal occupancy regularization strengths. This robustness is particularly desirable in VCMIPL, where sequence length, temporal redundancy, and feature granularity vary substantially across datasets. Figure 3: Sensitivity of PivotMipl to the entropic coefficient ε and occupancy coefficient λ on three representative dataset–feature configurations. Appendix D Limitations Our study focuses on offline VCMIPL within a fixed label space. Although the experiments span multiple datasets and feature representations, they do not establish robustness to distribution shifts or evolving class vocabularies. Assessing how class–time associations transfer across such changes remains a direction for future work. Moreover, the current formulation uses bidirectional temporal context and normalizes assignments over the observed video. Extending it to strictly online prediction would require causal contextualization and incremental assignment updates that account for incomplete temporal evidence.