Paper deep dive
Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation
Duy Tran Thanh, Yeejin Lee, Byeongkeun Kang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This work addresses the challenge of open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), which aim to recognize both predefined and unseen object categories without exhaustive human annotations. Existing methods often suffer from noisy pseudo-masks, limited visual-textual grounding, and difficulty handling synonyms or out-of-vocabulary (OOV) words. To overcome these challenges, we propose a multimodal framework that leverages pre-trained vision-language models for automatic pseudo-label generation, CLIP-guided synonym filtering, and GPT-based caption reconstruction. In our target-vocabulary-assisted pseudo-labeling setting, the framework first constructs pseudo segmentation masks, descriptive captions, and semantically aligned synonym sets using Grounded SAM, LLaVA, and CLIP, providing multimodal supervision without manual annotation. We then enhance visual-textual alignment through three complementary training objectives: an extended grounding loss that incorporates visually grounded synonyms, a semantic consistency loss, and a generative caption reconstruction loss. Extensive experiments on the COCO dataset demonstrate that the proposed method consistently outperforms previous state-of-the-art approaches under this protocol, achieving substantial improvements on both OVIS and OSPS benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2608.11681v1
- Canonical: https://arxiv.org/abs/2608.11681v1
Trouble viewing inline? Open PDF directly →
Full Text
68,751 characters extracted from source content.
Expand or collapse full text
Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic SegmentationJournal: Neurocomputing Duy Tran Thanh Email: duy.tranthanh@seoultech.ac.kr Affiliation: Department of Electronic Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul, 01811, South Korea Yeejin Lee Email: yeejinlee@seoultech.ac.kr Affiliation: Department of Electrical and Information Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul, 01811, South Korea Byeongkeun Kang Email: byeongkeunkang@cau.ac.kr Corresponding author: Corresponding author. Affiliation: School of Electrical and Electronics Engineering, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, South Korea Abstract This work addresses the challenge of open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), which aim to recognize both predefined and unseen object categories without exhaustive human annotations. Existing methods often suffer from noisy pseudo-masks, limited visual-textual grounding, and difficulty handling synonyms or out-of-vocabulary (OOV) words. To overcome these challenges, we propose a multimodal framework that leverages pre-trained vision-language models for automatic pseudo-label generation, CLIP-guided synonym filtering, and GPT-based caption reconstruction. In our target-vocabulary-assisted pseudo-labeling setting, the framework first constructs pseudo segmentation masks, descriptive captions, and semantically aligned synonym sets using Grounded SAM, LLaVA, and CLIP, providing multimodal supervision without manual annotation. We then enhance visual-textual alignment through three complementary training objectives: an extended grounding loss that incorporates visually grounded synonyms, a semantic consistency loss, and a generative caption reconstruction loss. Extensive experiments on the COCO dataset demonstrate that the proposed method consistently outperforms previous state-of-the-art approaches under this protocol, achieving substantial improvements on both OVIS and OSPS benchmarks. Keywords: Open-vocabulary instance segmentation , Open-set panoptic segmentation , Vision-language model , Visual-textual alignment , Pseudo-label generation. 1 Introduction Instance segmentation is a fundamental task in computer vision that aims to simultaneously detect, segment, and classify individual object instances, supporting a wide range of applications in robotics, autonomous driving, surveillance, and medical imaging. Although deep neural networks have achieved significant advancements in this domain, most existing approaches remain limited to detecting and segmenting objects from a predefined set of categories and require expensive pixel-level annotations for training (34). To mitigate these challenges, recent studies have explored zero-shot and weakly supervised segmentation frameworks that aim to recognize novel categories with minimal human supervision (17; 16; 12; 2). However, weakly supervised methods still require image-level or bounding-box annotations for all categories (16), whereas zero-shot approaches (17; 1; 47) rely only on textual embeddings such as word vectors. Nevertheless, zero-shot methods are typically constrained by limited generalization ability and often struggle to capture fine-grained visual details when applied to complex real-world datasets. To overcome these limitations, researchers have explored open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), which integrate large-scale vision-language models (VLMs) with flexible semantic representations (45; 13; 35; 37; 29; 4; 15; 3; 20; 40; 7; 26). In these frameworks, models are trained with strong supervision for base categories and weak supervision from image-caption pairs for novel ones, enabling broader generalization to unseen classes. XPM (13) is one of the earliest frameworks, employing a teacher-student paradigm in which the teacher model generates pseudo-masks by aligning the visual features of object regions with word embeddings extracted from image captions. The student model is then trained on these pseudo-labels while estimating annotation noise to improve robustness against imperfect supervision. However, such methods typically rely on strong supervision for base classes and weak supervision from image-caption pairs for novel ones, leading to a bias toward base categories. To alleviate this issue, Mask-free OVIS (35) removes the need for manual annotations by training a Mask R-CNN model with pseudo-masks generated by pre-trained vision-language models. Despite these advances, both XPM (13) and Mask-free OVIS (35) suffer from noisy pseudo-masks caused by inaccurate visual-textual alignments, limiting segmentation accuracy. CGG (37) addresses this problem by introducing caption grounding and caption generation losses. The grounding loss aligns object nouns in captions with their corresponding image regions while the generation loss enhances contextual understanding. However, existing open-vocabulary instance segmentation methods still face several fundamental challenges. First, their utilization of captions is typically limited, resulting in insufficient visual-textual grounding. Second, these models struggle to handle synonyms and out-of-vocabulary (OOV) words, thereby constraining their ability to generalize to novel categories and diverse linguistic expressions. Lastly, some approaches rely entirely on noisy pseudo-masks (35), often leading to suboptimal segmentation accuracy. To address these challenges, we propose a multimodal framework that leverages pre-trained vision-language models for automatic pseudo-label generation, CLIP-guided synonym filtering, and GPT-based caption reconstruction. In our target-vocabulary-assisted pseudo-labeling setting, target novel-category names are used as text prompts for Grounded SAM during pseudo-mask generation. The framework integrates an automated pipeline built upon Grounded SAM (32) and LLaVA (24) to generate pseudo segmentation masks, descriptive captions, and semantically aligned synonym sets, thereby enhancing visual-language alignment without requiring any additional manual annotations. The generated pseudo masks, captions, and synonyms are then jointly utilized with the labeled base dataset to train the model. Furthermore, we extend existing classification and mask losses by introducing an enhanced grounding loss, a semantic consistency loss, and a generative caption reconstruction loss. Together, these components enable more robust and semantically coherent visual-textual alignment under the target-vocabulary-assisted protocol, improving generalization across both seen and target novel categories. Rather than proposing a fundamentally new segmentation architecture, this work focuses on improving open-vocabulary generalization through target-vocabulary-assisted multimodal pseudo-labeling and training-time visual-textual supervision. We adopt a Mask2Former-based segmenter as a strong base architecture and introduce complementary supervision signals generated from pretrained vision-language models. Specifically, Grounded SAM provides pseudo segmentation masks using target novel-category prompts, LLaVA generates descriptive pseudo captions, and CLIP filters visually grounded synonym candidates. These multimodal pseudo-labels are then used with synonym-aware grounding, semantic consistency, and generative caption reconstruction losses to improve robustness to vocabulary variation and unseen categories. The main contributions of this paper are summarized as follows: 1. We introduce an automated multimodal pipeline that leverages pre-trained vision-language models to generate pseudo segmentation masks, descriptive captions, and semantically aligned synonym sets, providing additional multimodal supervision without manual annotations under a target-vocabulary-assisted pseudo-labeling protocol. 2. We propose a semantic consistency loss and an extended grounding loss that leverage both predefined category names and their visually grounded synonyms to improve generalization and robustness to vocabulary variations. 3. We introduce a GPT-based generative caption reconstruction loss that enhances visual-textual reasoning by reconstructing masked captions conditioned on visual features. 4. We demonstrate that the proposed method consistently outperforms previous state-of-the-art approaches on both OVIS and OSPS benchmarks using the COCO dataset under the target-vocabulary-assisted evaluation protocol. 2 Related Works 2.1 Open-Vocabulary Instance and Panoptic Segmentation Recent surveys have provided comprehensive overviews of open-vocabulary learning and segmentation. 38 review open-vocabulary learning in relation to zero-shot learning, open-set recognition, and out-of-distribution detection, and summarize recent progress in open-vocabulary detection and segmentation. Similarly, 48 discuss image segmentation in the foundation-model era, highlighting the growing role of large-scale vision-language and promptable segmentation models. Together, these surveys indicate a clear trend from fixed-vocabulary supervised segmentation toward weak image-text supervision, pseudo-label generation, and foundation-model-based visual-textual alignment. 13 introduced one of the earliest frameworks for open-vocabulary instance segmentation, which adopts a teacher-student paradigm. In this framework, the teacher model generates pseudo-masks by aligning the visual features of object regions with the word embeddings of objects extracted from image captions. The student model is then trained using these pseudo-masks while simultaneously estimating the noise levels of the pseudo-mask annotations to improve robustness against imperfect supervision. Since open-vocabulary methods are typically trained with strong human supervision for base classes and weak supervision from image-caption pairs for novel categories (13), they tend to exhibit a bias toward base categories. To mitigate this issue, 35 proposed the Mask-free OVIS framework, which eliminates the need for manual mask annotations for both base and novel classes. Their approach trains a Mask R-CNN architecture using pseudo-masks generated from a pre-trained vision-language model and image-caption pairs. However, both XPM (13) and Mask-free OVIS (35) rely on pseudo-masks generated by mapping object regions to words in captions. Consequently, inaccurate visual-textual alignments can produce noisy pseudo-masks, ultimately limiting segmentation accuracy. To address this limitation, 37 proposed the CGG framework, which introduces a novel caption grounding loss and a caption generation loss. The grounding loss is computed using only object nouns in captions to prevent matching non-visible words to image regions. Meanwhile, the caption generation loss complements the grounding loss by encouraging the model to learn richer contextual representations from image-caption pairs. Different from previous works (13; 35; 37), we propose a unified framework that integrates additional grounding, semantic consistency, and generative caption reconstruction losses based on automatically generated pseudo-captions and CLIP-filtered synonyms, to enhance visual-textual alignment. While CGG (37) relies on restricted noun-based grounding, our framework provides richer supervision by automatically generating diverse captions using LLaVA, extracting semantically related words using CLIP, and leveraging them for grounding and semantic consistency objectives. Moreover, by incorporating a GPT-driven caption reconstruction loss, our model captures fine-grained contextual semantics beyond object-level grounding. 2.2 Mask-Classification and Universal Segmentation Architectures Mask-classification-based architectures have become strong foundations for modern segmentation. MaskFormer (6) reformulates segmentation as a mask classification problem by predicting a set of binary masks with corresponding class labels, rather than assigning labels independently to each pixel. Mask2Former (5) further extends this framework with masked attention and provides a unified architecture for semantic, instance, and panoptic segmentation. Owing to its strong performance and flexibility, Mask2Former (5) has been widely adopted as a backbone or baseline architecture in subsequent segmentation studies. Recent studies have also explored query learning and clustering-based formulations for segmentation. 36 introduce instance-unique and transformation-equivariant queries for query-based instance segmentation, while CLUSTSEG (22) formulates diverse segmentation tasks as a unified neural clustering process. These works highlight the importance of query-based learning and unified formulations for diverse segmentation tasks. Recent methods have further explored unified and generalist segmentation frameworks. X-Decoder (50) introduces a generalized decoding framework that connects pixel-level segmentation outputs and language tokens within a shared semantic space. FreeSeg (30) studies universal and open-vocabulary segmentation by integrating dense prediction with flexible language representations. OMG-Seg (21) further investigates whether a single model can handle a broad range of segmentation tasks, including image segmentation, video segmentation, open-vocabulary segmentation, prompt-driven segmentation, and interactive segmentation. These methods are relevant to our work because they demonstrate the importance of flexible segmentation architectures for handling diverse categories and tasks. Our method is related to these architectures because it adopts a Mask2Former-style query-based segmenter. However, unlike methods that introduce a new decoder or segmentation backbone, our work focuses on improving open-vocabulary generalization through multimodal pseudo-labeling and training-time visual-textual supervision. 2.3 Foundation Models for Open-Vocabulary and Promptable Segmentation Foundation models have recently played an important role in open-vocabulary and promptable segmentation. SAM (18) provides strong promptable mask generation and demonstrates strong transfer ability across diverse segmentation scenarios. Grounded SAM (32) combines open-set object localization with mask prediction by integrating Grounding DINO (25) and SAM (18), enabling the segmentation of arbitrary text-specified objects. Open-Vocabulary SAM (44) further integrates SAM and CLIP for simultaneous interactive segmentation and recognition over a large vocabulary. ODISE (42) leverages diffusion and vision-language representations for open-vocabulary panoptic segmentation. These works demonstrate the strong potential of foundation models for scalable mask generation, promptable segmentation, and open-vocabulary recognition. Unlike methods that mainly use foundation models as segmentation or recognition components, our framework uses them primarily to generate complementary supervision signals for training. Specifically, Grounded SAM (32) provides pseudo segmentation masks, LLaVA (24) generates descriptive pseudo captions, and CLIP (31) filters visually grounded synonym candidates. During inference, these auxiliary modules are removed, and the model retains only the Mask2Former-based segmentation components and CLIP text embeddings. Table 1: Conceptual and technical differences between CGG and the proposed method. Both methods use a Mask2Former-based segmentation architecture, while our method differs in multimodal pseudo-label construction and synonym-aware supervision. Component CGG (37) Ours Base architecture Mask2Former-based Mask2Former-based Inference-time modules Segmenter + text embeddings Segmenter + text embeddings Caption supervision Original captions Original captions + LLaVA-generated pseudo captions Grounding target Object nouns from captions Object nouns, novel words, and visually grounded synonyms Novel-class pseudo-mask supervision Not explicitly generated Grounded-SAM-based pseudo masks Synonym handling Not explicitly modeled CLIP-based visual synonym filtering Semantic consistency No explicit synonym-category consistency Synonym-category semantic consistency loss Caption objective Caption generation GPT-based masked caption reconstruction 2.4 Language-Supervised Visual-Textual Alignment for Segmentation Language-conditioned segmentation methods align visual regions, pixels, or mask embeddings with text representations to recognize categories beyond a fixed vocabulary. OpenSeg (10) and related vision-language segmentation approaches exploit image-text supervision to learn open-vocabulary dense prediction. PGSeg (46) studies weakly open-vocabulary semantic segmentation using only image-text pairs and introduces prototypical knowledge to guide visual grouping and group-text alignment. CLIPSelf (39) analyzes the gap between global CLIP image representations and local region representations, and adapts CLIP-based vision transformers for open-vocabulary dense prediction without requiring region-text pairs. These methods demonstrate the importance of language supervision and region-language alignment for generalizing to unseen categories. Our method follows this general direction but differs in two aspects. First, instead of relying only on predefined class names or original captions, we generate pseudo captions and extract visually grounded synonyms to enrich the training vocabulary. Second, we explicitly introduce synonym-aware grounding, semantic consistency, and caption reconstruction losses to strengthen visual-textual alignment at the region and caption levels. This design enables the model to better handle linguistic variations and out-of-vocabulary expressions in open-vocabulary instance and open-set panoptic segmentation. Figure 1: Overall framework of MCCF. The proposed framework consists of three stages: multimodal pseudo-label construction, training with multimodal supervision, and lightweight inference. First, MCCF constructs pseudo masks, LLaVA-generated pseudo captions, and visually grounded pseudo synonyms. These pseudo labels are then used to train a Mask2Former-based segmenter with standard segmentation losses and the proposed visual-textual objectives, including synonym-aware grounding, semantic consistency, and GPT-based caption reconstruction losses. At inference time, auxiliary pseudo-label generation modules are removed. 3 Method 3.1 Problem Formulation Following previous works (13; 37), during training, we are given a training set bD_b consisting of images I and their ground-truth annotations Y for base classes bC_b (i.e., b=(i,i)i=1NbD_b=\( I_i,Y_i)\_i=1^N_b) where NbN_b denotes the number of images in bD_b. Each ground-truth annotation iY_i includes pairs of an instance mask and its class label for objects belonging to bC_b in i I_i. In addition, another dataset cD_c is provided containing images I and their image-level captions CcC^c (i.e., c=(i,Cic)i=1NcD_c=\( I_i,C^c_i)\_i=1^N_c) where NcN_c represents the number of images in cD_c. We denote the set of object nouns extracted from the captions as caption classes cC_c. Since captions contain a wider variety of words than the limited set of base class names, the number of caption classes |c||C_c| is larger than the number of base classes |b||C_b|. The trained model is expected to predict accurate instance masks and class labels for objects belonging not only to the base classes bC_b but also to novel categories nC_n. The novel class set nC_n includes all categories outside bC_b, encompassing both caption classes cC_c and entirely unseen categories during training. For classification, we employ text embeddings of class names extracted from the pre-trained CLIP text encoder (31). 3.2 Baseline Method Following CGG (37), we adopt Mask2Former (5) as our baseline segmentation architecture because of its high accuracy and ability to handle diverse segmentation tasks including panoptic, instance, and semantic segmentation. As in Mask2Former (5), our baseline architecture consists of three main components: a backbone, a pixel decoder, and a transformer decoder. The pixel decoder generates per-pixel embeddings, while the transformer decoder, followed by an MLP head, produces multimodal embeddings and mask embeddings. Final mask predictions are obtained by computing the product between the per-pixel embeddings and the mask embeddings. Unlike CGG (37), which employs BERT (9) for textual embeddings, we use the CLIP text encoder to extract text embeddings, enabling better alignment between visual and textual representations in a shared multimodal space. Class predictions are computed as the dot product between the multimodal embeddings from the transformer decoder and the text embeddings. As illustrated in the inference stage of Figure 1, during inference, we retain only the Mask2Former components (5) and the pre-trained text embeddings of all classes, including both base and novel categories. The auxiliary modules used during training are excluded at inference time to ensure efficient prediction. Table 1 summarizes the conceptual and technical differences between CGG and the proposed method. While both methods adopt a Mask2Former-based segmentation architecture, they differ in how visual-textual supervision is constructed and exploited during training. Specifically, our method extends CGG by introducing LLaVA-generated pseudo captions, Grounded-SAM-based pseudo masks, CLIP-based synonym filtering, synonym-category embedding alignment, and GPT-based caption reconstruction, while retaining a comparable inference-time structure based on the segmentation model and text embeddings. 3.3 Proposed Method Figure 1 provides an overview of the proposed MCCF framework. The framework is organized into three stages. First, we construct multimodal pseudo labels by combining pseudo masks, LLaVA-generated pseudo captions, and visually grounded pseudo synonyms. Second, these pseudo labels are used to train a Mask2Former-based segmenter with both standard segmentation losses and the proposed visual-textual objectives, including synonym-aware grounding, semantic consistency, and GPT-based caption reconstruction. Finally, during inference, the auxiliary pseudo-label generation modules are removed, and the model performs prediction using only the trained segmenter and CLIP text embeddings. This design enables MCCF to exploit rich multimodal supervision during training while maintaining a lightweight inference pipeline. 3.3.1 Multimodal Pseudo-label Generation and Refinement To enhance visual-language alignment without requiring any additional manual annotations, we propose an automated multimodal pipeline that generates pseudo segmentation masks, descriptive captions, and semantically aligned synonym sets using pre-trained vision-language models. The automatically generated pseudo-labels provide both pixel-level and language-level supervision, which are subsequently leveraged during training to improve generalization to unseen categories. As the entire procedure relies only on pre-trained modules and predefined processing steps, it is executed only once before training. Pseudo Mask and Caption Generation. To generate pseudo masks, we employ Grounded SAM (32), which integrates Grounding DINO (25) for object detection and SAM (18) for segmentation, as illustrated in Figure 2. Specifically, given an image I, Grounding DINO detects objects belonging to novel classes nC_n based on their text labels, producing bounding boxes, confidence scores, and class predictions. These bounding boxes are then refined by SAM into pixel-level pseudo segmentation masks psd M^psd. As a result, high-quality pseudo-mask annotations are automatically generated for novel categories without any human supervision. Next, we use LLaVA (Large Language and Vision Assistant) (24) to generate pseudo captions CpsdC^psd describing the detected objects, as shown in Figure 2. LLaVA combines CLIP-ViT (31) for visual encoding and Vicuna (8) for language generation. Given the class labels CLASS_LABELS predicted by Grounding DINO (25), we embed them into a structured prompt designed to encourage LLaVA to produce detailed, diverse, and semantically rich descriptions using synonyms or paraphrased expressions. The prompt is defined as follows: "There is/are CLASS_LABELS in the image. Describe their appearance, position, and quantity in detail and accurately. Instead of using the words in CLASS_LABELS, try to use synonyms or creative descriptions to convey their identity." Figure 2: Multimodal pseudo-label construction. Grounding DINO and SAM generate pseudo masks, LLaVA generates pseudo captions, and CLIP-based multimodal filtering selects visually grounded pseudo synonyms from candidate nouns extracted from the pseudo captions. Blue boxes denote frozen pretrained modules, green boxes denote non-learnable processing steps, pink boxes indicate generated pseudo labels, and white boxes denote inputs or intermediate outputs. Multimodal Filtering-based Pseudo Synonym Generation. Given the pseudo captions generated by LLaVA, we first extract nouns and noun phrases and identify candidate words that do not belong to the pre-defined class vocabulary, as illustrated in Figure 2. Then, we propose a CLIP-based multimodal filtering strategy to select visually grounded synonym candidates among these undefined words. Specifically, for each undefined word wundefw^undef extracted from the pseudo captions, we encode it using the CLIP text encoder to obtain its textual embedding t. For the visual representation, we crop and mask the input image I using the pseudo masks psd M^psd generated by SAM for novel categories. The masked image is computed as mask=⊙psd I^mask= I M^psd, where ⊙ denotes element-wise multiplication. These masked images are then encoded using the CLIP image encoder to obtain visual embeddings v. Then, we compute the multimodal similarity between the textual embedding iundef t_i^undef of the i-th word and the visual embedding jmask v_j^mask of the j-th masked image using cosine similarity: sij=(iundef)Tjmask∥iundef∥∥jmask∥,s_ij= ( t_i^undef)^T v_j^mask t_i^undef v_j^mask , (1) where ∥⋅∥ · denotes the Euclidean norm. The similarity score quantifies the semantic and visual alignment between textual concepts and localized visual regions. To retain only the most relevant and visually consistent synonym candidates, we apply a filtering strategy that combines top-1 ranking with a similarity threshold. A candidate synonym wsynw^syn is preserved if its similarity score exceeds a threshold of τ=0.4τ=0.4 and ranks as the most similar word for the corresponding visual instance. This criterion suppresses noisy or ambiguous terms while retaining synonyms that are semantically and visually consistent with masked images for novel categories. The threshold τ=0.4τ=0.4 is empirically selected to balance the retention of visually relevant synonym candidates and the suppression of noisy or weakly grounded words. We use top-1 selection together with this threshold to keep only the most visually consistent synonym for each pseudo instance. 3.3.2 Training We leverage the pseudo segmentation masks psd M^psd, pseudo captions CpsdC^psd, and pseudo synonyms wpsdw^psd generated for the dataset cD_c from Section 3.3.1, in addition to the labeled training set bD_b for base classes, during training. We adopt the classification and mask losses from (37; 5), enhance the grounding loss from (37) using the pseudo synonyms, and further introduce a semantic consistency loss and a generative caption reconstruction loss. The overall training framework is illustrated in Figure 3. Classification Loss. We employ a cross-entropy loss for the classification loss ℒclsL_cls to align visual and textual embeddings using bD_b and cD_c with pseudo labels. The logits are obtained from the dot product between the multimodal embeddings f produced by the transformer decoder (with an MLP head) and the text embeddings t extracted from the CLIP text encoder, as shown in Figure 3. Formally, the classification loss ℒclsL_cls is defined as: ℒcls=−∑iiln(i),L_cls=- _i y_i ( p_i), (2) where i indexes the classes, y denotes the one-hot encoded class label of an instance, and i p_i represents the predicted probability for class i, obtained by applying a softmax to the dot product between f and t. Figure 3: Training framework of MCCF. The Mask2Former-based segmenter is trained with pseudo masks, class names, pseudo synonyms, and captions. In addition to standard classification and mask losses, MCCF introduces synonym-aware grounding, semantic consistency, and GPT-based caption reconstruction losses. Blue boxes denote frozen pretrained modules, yellow boxes denote trainable network components, green boxes denote non-learnable processing steps, orange boxes denote loss functions, and white boxes denote inputs or intermediate outputs. Mask Loss. We adopt the mask loss ℒmaskL_mask from (5), which comprises a mask classification loss ℒmask-clsL_mask-cls, a pixel-wise binary cross-entropy loss ℒceL_ce, and a Dice loss ℒdiceL_dice. The total mask loss ℒmaskL_mask is computed using bD_b and cD_c with pseudo labels and is formulated as: ℒmask=λmask-clsℒmask-cls+λceℒce+λdiceℒdice,L_mask= _mask-clsL_mask-cls+ _ceL_ce+ _diceL_dice, (3) where λmask-cls _mask-cls, λce _ce, and λdice _dice are weighting coefficients that balance the contribution of each term. Grounding Loss. We adopt the grounding loss from (37), which aims to learn visual-textual alignment by maximizing the similarity between the visual and textual embeddings of matched pairs. However, while CGG (37) computes this loss using only object nouns extracted from the given captions in cD_c, we extend it to include both object nouns and their synonyms obtained from the given captions CcC^c and the generated pseudo captions CpsdC^psd. This extension enables the model to capture a broader range of semantic relationships between visual regions and language expressions. Specifically, we compute cosine similarities between the visual embeddings f from the transformer decoder (with an MLP head) and the text embeddings nov t^nov and syn t^syn for novel class names and their synonyms, respectively, using the CLIP text encoder. Then, the grounding loss is designed to maximize similarity for matched visual–text pairs while minimizing it for mismatched ones. The overall grounding loss ℒgrL_gr is defined as follows: ℒgr=ℒgr-nov(,novel)+ℒgr-syn(,syn),L_gr=L_gr-nov( f, t^novel)+L_gr-syn( f, t^syn), (4) where ℒgr-novL_gr-nov and ℒgr-synL_gr-syn denote the grounding losses for novel-category nouns and their synonyms, respectively. By optimizing this objective, the model learns a unified visual-text embedding space that effectively associates image regions with diverse textual concepts, thereby enhancing zero-shot recognition and robustness to vocabulary variations. Semantic Consistency Loss. This loss aims to enforce consistency among semantically equivalent words, particularly novel class names and their synonyms, within a shared multimodal embedding space. Although synonyms such as “plane,” “airplane,” and “aircraft” convey the same meaning, their embeddings may occupy different positions in the feature space, leading to inconsistent predictions. To mitigate this issue, we project novel class names wnovw^nov and their corresponding synonyms wsynw^syn into the text embedding space using the CLIP text encoder, yielding nov t^nov and syn t^syn, respectively. We then compute the distance between these paired embeddings, weighted by the visual embedding f obtained from the transformer decoder to emphasize visually relevant regions. The semantic consistency loss ℒconsL_cons is defined as follows: ℒcons=iT(inov−isyn)2.L_cons=\ f_i^T( t^nov_i- t^syn_i)\^2. (5) This loss encourages consistent representations for words with equivalent meanings, enhancing robustness to linguistic variation and improving generalization to unseen categories. Generative Caption Reconstruction Loss. This loss aims to enhance the model’s ability to understand and reason over fine-grained visual-textual relationships by incorporating a generative language component. Unlike the grounding loss, which focuses on maximizing the similarity between matched visual and textual embeddings, this loss encourages the model to reconstruct complete captions conditioned on visual features. Given a caption C=[c1,c2,…,cn]C=[c_1,c_2,…,c_n] from either the original captions CcC^c or the generated pseudo captions CpsdC^psd, a subset of tokens is randomly replaced with the mask token [MASK], resulting in a masked caption C~ C. The masked caption C~ C is then fed into a GPT-based generative module as the query sequence, while the image features F extracted from the backbone of the segmentation network are used as keys and values in the cross-attention layers. This design enables the GPT module to leverage visual information when reconstructing the missing textual tokens. The generative caption reconstruction loss ℒreconL_recon is computed by maximizing the likelihood of correctly predicting each masked token conditioned on the preceding tokens, the masked caption, and the corresponding image features. Formally, the loss is defined as: ℒrecon=−∑i=1nlogp(c^i∣ci−1,C~,),L_recon=- _i=1^n p( c_i c_i-1, C, F), (6) where p(c^i∣ci−1,C~,)p( c_i c_i-1, C, F) denotes the conditional probability of predicting token c^i c_i given the previous token ci−1c_i-1, the partially masked caption C~ C, and the image feature map F. Total Loss. The total loss ℒtotalL_total is defined as the weighted sum of the five previously introduced components: the classification loss ℒclsL_cls, mask loss ℒmaskL_mask, grounding loss ℒgrL_gr, semantic consistency loss ℒconsL_cons, and generative caption reconstruction loss ℒreconL_recon. ℒtotal=ℒcons+λclsℒcls+λmaskℒmask+λgrℒgr+λreconℒrecon,L_total=L_cons+ _clsL_cls+ _maskL_mask+ _grL_gr+ _reconL_recon, (7) where λcls _cls, λmask _mask, λgr _gr, and λrecon _recon are hyperparameters that balance the contribution of each loss term. In all experiments, we set λcls _cls, λmask _mask, λgr _gr, and λrecon _recon to 2, 5, 2, and 2, respectively. 4 Experiments and Results 4.1 Experimental Setting Datasets. We conduct experiments on the COCO dataset (23) for both open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS). For OVIS, following previous works (47; 13; 37; 35), we split the 65 object classes into 48 base classes and 17 novel classes, where pixel-level annotations are provided only for the base classes during training. The 17 novel categories are bus, dog, cow, elephant, umbrella, tie, skateboard, cup, knife, cake, couch, keyboard, sink, scissors, airplane, cat, and snowboard. For OSPS, following (14; 41; 37), we construct three unknown-category settings with 5%, 10%, and 20% unknown thing classes. In the 5% setting, car, cow, pizza, and toilet are treated as unknown classes. In the 10% setting, boat, tie, zebra, and stop sign are additionally excluded. In the 20% setting, dining table, banana, bicycle, cake, sink, cat, keyboard, and bear are further excluded. These categories are removed from the labeled training set and evaluated as unknown thing classes. Metric and Evaluation Protocol. For OVIS, we use the mask-based mean Average Precision (mAP) at an IoU threshold of 0.5. Following (45; 13; 37; 35), we evaluate under two settings: constrained and generalized. In the constrained setting, the trained model is tested on images containing either base or novel classes. In the generalized setting, evaluation is conducted on images containing both base and novel classes. The generalized setting is more challenging due to inherent class bias, which tends to favor the base classes. For OSPS, we report Panoptic Quality (PQ) and Segmentation Quality (SQ), following (37). Table 2: Quantitative comparison of open-vocabulary instance segmentation. Bold and underlined values denote the best and second-best scores, respectively. † indicates CGG re-trained using the same Grounded-SAM-based novel-class pseudo-mask annotations as our method while keeping the original CGG training losses. Method Constrained Generalized Base Novel Base Novel OVR+OMP (2) 31.3 14.1 30.5 8.3 SB (1) 41.6 20.8 41.0 16.0 BA-RPN (47) 41.8 20.1 41.3 15.4 Soft-Teacher (43) 41.8 14.8 41.5 9.6 Unbiased-Teacher (27) 41.8 15.1 41.4 9.8 OVR-RCNN (45) 42.0 20.9 41.6 17.1 XPM (13) 42.4 24.0 41.5 21.6 Mask-free OVIS (35) 36.7 27.4 28.7 25.0 CGG (37) 46.8 29.5 46.0 28.4 CGG† 48.0 45.1 47.7 43.6 MCCF (Ours) 47.8 51.6 47.4 50.4 Implementation Details. We adopt CLIP embeddings (31) as the shared representation space for the classification head, image encoder, and text encoder. For text processing, we employ the BPE tokenizer (33) from GPT-2 to tokenize captions and map discrete tokens to continuous embeddings using GPT-2’s pre-trained embedding layer, following (49). Unlike (37), which uses an LVIS-specific class parser to extract object nouns, we adopt the NLTK parser for word extraction to preserve lexical diversity and capture a richer semantic vocabulary in open-vocabulary settings. For OVIS, we retain the top 100 queries as model outputs, corresponding to the highest-confidence object predictions. For OSPS, we follow (14; 41; 37) by prioritizing mask predictions for thing classes and assigning the remaining background regions to stuff categories. The models are trained on two GPUs with a mini-batch size of four using the AdamW optimizer (28) with a weight decay of 0.0001. Following (5; 37), we apply random cropping during both pre-training and training. Our training procedure consists of two stages, following (37; 45). In the first stage, we perform class-agnostic pretraining using bD_b and cD_c with pseudo mask labels without incorporating captions. In the second stage, the model is fine-tuned with all proposed loss functions using bD_b and cD_c along with the complete set of pseudo labels. For the GPT-based caption reconstruction module, we use captions from both the original COCO captions and the LLaVA-generated pseudo captions. During training, a subset of caption tokens is randomly replaced with the [MASK] token. The masked caption embeddings are fed into the GPT-based decoder as the query sequence, while image features extracted from the segmentation backbone are used as keys and values in the cross-attention layers. The decoder then reconstructs the masked tokens conditioned on visual features. This reconstruction objective is used only during training and is optimized in the second training stage together with the segmentation, grounding, and semantic consistency losses. For pseudo-label noise handling, we use CLIP-based multimodal filtering to remove weakly grounded synonym candidates. Specifically, each candidate synonym extracted from LLaVA-generated pseudo captions is compared with the corresponding masked visual region using CLIP similarity. A candidate synonym is retained only when its similarity score exceeds τ=0.4τ=0.4 and ranks within the top-1 candidates for the corresponding visual instance. This strategy suppresses ambiguous or visually inconsistent words while preserving synonyms that are semantically and visually aligned with the pseudo masks. Pseudo-label vocabulary. In our main pseudo-labeling setting, we use the target novel-category vocabulary as text prompts for Grounding DINO (25) to generate pseudo annotations for novel categories. Therefore, this setting should be interpreted as a target-vocabulary-assisted pseudo-labeling protocol rather than a fully category-agnostic open-vocabulary protocol. Since this protocol can provide stronger pseudo masks than caption-only pseudo-labeling, we additionally introduce a controlled baseline, CGG†, to separate the effect of pseudo-mask quality from the effect of the proposed training objectives. CGG† is trained with the same Grounded-SAM-based novel-class pseudo-mask annotations as ours while retaining the original CGG loss functions (37). Thus, the comparison with CGG† provides a more controlled evaluation of the proposed synonym-aware grounding, semantic consistency, and caption reconstruction objectives. Table 3: Quantitative comparison of open-set panoptic segmentation. The "Unk." column denotes the proportion of unknown classes among all classes. Bold and underlined values indicate the best and second-best scores, respectively. ∗ indicates that the scores are averaged across unknown classes, whereas EOPSN (14) and Dual (41) treat all unknown categories as a single class. The superscripts "Th" and "St" denote thing and stuff classes, respectively. Method Unk. Known Unknown (%) PQThPQ^Th SQThSQ^Th PQStPQ^St SQStSQ^St PQThPQ^Th SQThSQ^Th EOPSN (14) 20 45.0 80.3 28.2 71.2 11.3 73.8 Dual (41) 45.0 80.6 28.1 70.1 21.4 79.1 CGG (37) 48.4 82.3 34.4 81.1 36.5∗ 78.0∗ MCCF (ours) 47.1 81.5 33.9 80.2 54.5∗ 85.3∗ EOPSN (14) 10 44.5 80.6 28.4 71.8 17.9 76.8 Dual (41) 45.0 80.7 27.8 72.2 24.5 79.9 CGG (37) 49.2 82.8 34.6 81.2 41.6∗ 82.6∗ MCCF (ours) 48.4 82.3 34.0 80.8 53.1∗ 83.7∗ EOPSN (14) 5 44.8 80.5 28.3 73.1 23.1 74.7 Dual (41) 45.1 80.9 28.1 73.1 30.2 80.0 CGG (37) 50.2 83.1 34.3 81.5 45.0∗ 85.2∗ MCCF (ours) 49.8 82.9 33.8 81.3 52.3∗ 83.0∗ 4.2 Results Quantitative Results for OVIS. Table 2 presents quantitative comparisons between the proposed method and prior works (37; 35; 13; 45; 27; 43; 47; 1; 2) on the open-vocabulary instance segmentation task. Compared with the original CGG (37), our method improves novel-class AP by 22.1 and 22.0 points under the constrained and generalized settings, respectively. For the base classes, our method achieves improvements of 1.0 and 1.4 AP points in the constrained and generalized settings, respectively. To further isolate the effect of the proposed supervision strategy, we additionally report CGG†, where CGG is re-trained using the same Grounded-SAM-based novel-class pseudo-mask annotations as our method while retaining the original CGG losses. Compared with CGG†, MCCF improves novel-class AP by 6.5 and 6.8 points in the constrained and generalized settings, respectively. These results indicate that the proposed synonym-aware grounding, semantic consistency, and caption reconstruction objectives provide additional gains beyond stronger pseudo-mask annotations. A detailed ablation study on each proposed objective is further provided in Table 4. (a) CGG (b) Ours Figure 4: Qualitative comparison of open-vocabulary instance segmentation results between the proposed method and the previous state-of-the-art method, CGG (37). Quantitative Results for OSPS. Table 3 presents quantitative comparisons between the proposed method and previous works (37; 41; 14) on the open-set panoptic segmentation task. The results show that our method improves unknown-class panoptic segmentation performance over the previous state-of-the-art method (37) across most settings and metrics. Specifically, our method achieves absolute improvements of 18.0, 11.5, and 7.3 PQ points for unknown classes under the 20%, 10%, and 5% unknown settings, respectively. For known classes, our method slightly underperforms CGG (37). This suggests that the proposed training strategy shifts the model toward better recognition of unknown categories, with a small trade-off on known categories. Nevertheless, the substantial gains on unknown classes indicate that the proposed multimodal pseudo-labeling and visual-textual alignment objectives are effective for open-set panoptic segmentation. Figure 5: Qualitative comparison of open-set panoptic segmentation results between the proposed method (second row) and the previous state-of-the-art method, CGG (37) (first row). Figure 6: Additional qualitative results for OVIS and OSPS. The first and second rows correspond to OVIS and OSPS examples, respectively. The captions below the images are pseudo captions generated by LLaVA. Qualitative Results. Figure 4 presents qualitative comparisons between the proposed method and the previous state-of-the-art method, CGG (37), on open-vocabulary instance segmentation. In the first row, our method successfully segments knives belonging to unknown classes, whereas CGG (37) fails to detect them. Additionally, CGG (37) incorrectly identifies a cup as an umbrella. In the second row, CGG (37) misclassifies computer mice and laptops as keyboards, while our method accurately predicts a keyboard from novel categories. Furthermore, our method correctly segments the couch in the third row and classifies the cows in the last row. Figure 5 presents qualitative comparisons between the proposed method and the previous state-of-the-art method, CGG (37), on open-set panoptic segmentation. In the first column, our method accurately segments the road, whereas CGG misclassifies it as grass. Moreover, our method more precisely identifies bananas in the second column and better distinguishes the stuff category in the last column. Figure 6 presents additional qualitative results of the proposed MCCF framework. The first row shows open-vocabulary instance segmentation (OVIS) examples, while the second row shows open-set panoptic segmentation (OSPS) examples. The captions shown below each example are pseudo captions generated by LLaVA. These examples illustrate that MCCF can localize and segment objects described in the generated captions, demonstrating the usefulness of LLaVA-generated pseudo captions as multimodal supervision for both instance-level and panoptic segmentation settings. Figure 7: Failure cases in OVIS. The examples show typical errors caused by over-segmentation, cluttered scenes, partial occlusion, low illumination, and ambiguous object boundaries. We present representative failure cases in Figure 7 to further analyze the limitations of the proposed pseudo-labeling strategy. In cluttered scenes, the model may also produce inaccurate masks or confuse neighboring objects, such as sinks, cups, and keyboards, due to partial occlusion, low illumination, or ambiguous object boundaries. Addressing these failure cases with more robust pseudo-label validation and boundary refinement remains an important direction for future work. Table 4: Ablation study on open-vocabulary instance segmentation. Method Novel AP Baseline (CGG†) ℒgrL_gr ℒconsL_cons ℒreconL_recon Constrained Generalized ✓ 45.1 43.6 ✓ ✓ 49.5 47.8 ✓ ✓ ✓ 50.9 49.3 ✓ ✓ ✓ ✓ 51.6 50.4 4.3 Ablation Studies In Table 4, we present an ablation study evaluating the effectiveness of each proposed objective under the open-vocabulary instance segmentation setting. We begin with a CGG-based baseline trained using the same Grounded-SAM-based novel-class pseudo-mask annotations as our method, which corresponds to CGG† in Table 2. This baseline retains the original CGG losses and achieves AP scores of 45.1 and 43.6 for novel categories under the constrained and generalized settings, respectively. Replacing the original CGG grounding loss with our extended grounding loss ℒgrL_gr in Eq. (4) improves the performance to 49.5 and 47.8. Adding the semantic consistency loss ℒconsL_cons further enhances the results to 50.9 and 49.3. Finally, introducing the generative caption reconstruction loss ℒreconL_recon yields the best performance, achieving AP scores of 51.6 and 50.4 under the constrained and generalized settings, respectively. We further analyze the effect of CLIP-based synonym filtering on the full MCCF model in Table 5. When CLIP-based filtering is removed, LLaVA-generated synonyms are used without visual verification, and the model achieves 47.1 and 44.0 AP on novel categories under the constrained and generalized settings, respectively. In contrast, applying CLIP-based filtering improves the performance to 51.6 and 50.4 AP, demonstrating that visually grounded synonym filtering effectively suppresses noisy language candidates and improves open-vocabulary generalization. Table 5: Effect of CLIP-based synonym filtering on open-vocabulary instance segmentation. Method Constrained Novel Generalized Novel MCCF w/o CLIP filtering 47.1 44.0 MCCF w/ CLIP filtering 51.6 50.4 Computational Overhead. We further analyze the inference-time computational overhead of the proposed method. Table 6 compares the CGG baseline and our model in terms of learnable parameters and GFLOPs. Our model increases the number of parameters from 35.6M to 38.4M and GFLOPs from 227.5 to 232.9, corresponding to relative increases of 7.9% and 2.4%, respectively. These results show that the proposed method introduces only modest additional inference cost. It is also worth noting that the auxiliary pseudo-label generation components, including Grounded SAM, LLaVA, and CLIP-based synonym filtering, are used only during pseudo-label construction and training, and are not required during inference. Table 6: Inference-time computational overhead comparison with the CGG baseline. Method Parameters (M) GFLOPs CGG (37) 35.6 227.5 MCCF (Ours) 38.4 232.9 5 Conclusion This paper presents a multimodal framework for open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), designed to enhance visual-textual alignment and generalization to unseen categories without requiring manual annotations. In our target-vocabulary-assisted pseudo-labeling setting, target novel-category names are used as prompts during pseudo-mask generation. Under this protocol, the proposed approach leverages pre-trained vision-language models to generate pseudo segmentation masks, descriptive captions, and semantically aligned synonym sets, thereby providing rich multimodal supervision. In addition, we introduced a semantic consistency loss and an extended grounding loss that incorporate both predefined category names and visually grounded synonyms to improve robustness to vocabulary variations. A GPT-based generative caption reconstruction loss was further proposed to strengthen fine-grained visual-textual reasoning by reconstructing masked captions conditioned on visual features. Extensive experiments on the COCO dataset demonstrate that our method consistently outperforms previous state-of-the-art approaches on both OVIS and OSPS benchmarks under the target-vocabulary-assisted protocol, validating the effectiveness of our multimodal pseudo-labeling and alignment strategy. Rather than replacing existing segmentation architectures with a new decoder or backbone, our results show that open-vocabulary instance and open-set panoptic segmentation can be substantially improved by enriching training supervision with target-vocabulary-assisted multimodal pseudo-labels and synonym-aware visual-textual alignment objectives. Limitation and Future Work. While the proposed framework demonstrates strong performance, computational resource constraints prevented training on large-scale datasets such as LVIS (11) and Open Images (19). Future work will explore large-scale training and computationally efficient learning strategies to further enhance open-vocabulary generalization and scalability. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgments This research was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government(MSIT) (No. RS-2023-00252434). References Bansal et al. (2018) A. Bansal, K. Sikka, G. Sharma, R. Chellappa, and A. Divakaran Zero-shot object detection. In Proceedings of the European conference on computer vision (ECCV), p. 384–400. Cited by: §1, §4.2, Table 2. Biertimpel et al. (2021) D. Biertimpel, S. Shkodrani, A. S. Baslamisli, and N. Baka Prior to segment: foreground cues for weakly annotated classes in partially supervised instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2824–2833. Cited by: §1, §4.2, Table 2. Che and Nguyen (2026) H. Che and V. Nguyen FA-seg: a fast and accurate diffusion-based method for open-vocabulary segmentation. Neurocomputing 660, p. 131844. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Chen et al. (2026) L. Chen, Q. Yang, K. Ding, Z. Li, G. Shen, F. Li, Q. Cao, and S. Xiang Efficient redundancy reduction for open-vocabulary semantic segmentation. Neurocomputing 665, p. 132229. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Cheng et al. (2022) B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar Masked-attention mask transformer for universal image segmentation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 1280–1289. External Links: Document Cited by: §2.2, §3.2, §3.2, §3.3.2, §3.3.2, §4.1. Cheng et al. (2021) B. Cheng, A. Schwing, and A. Kirillov Per-pixel classification is not all you need for semantic segmentation. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.), Vol. 34, p. 17864–17875. External Links: Link Cited by: §2.2. Cheng et al. (2025) S. Cheng, J. Huang, X. Wang, L. Huang, and Z. Wei Image–text aggregation for open-vocabulary semantic segmentation. Neurocomputing 630, p. 129702. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Chiang et al. (2023) W. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. Vicuna: an open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) 2 (3), p. 6. Cited by: §3.3.1. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 4171–4186. Cited by: §3.2. Ghiasi et al. (2022) G. Ghiasi, X. Gu, Y. Cui, and T. Lin Scaling open-vocabulary image segmentation with image-level labels. In Computer Vision – ECCV 2022, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner (Eds.), Cham, p. 540–557. External Links: ISBN 978-3-031-20059-5 Cited by: §2.4. Gupta et al. (2019) A. Gupta, P. Dollar, and R. Girshick Lvis: a dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5356–5364. Cited by: §5. Hoang et al. (2025) C. M. Hoang, Y. Lee, and B. Kang Generalized class discovery in instance segmentation. Proceedings of the AAAI Conference on Artificial Intelligence 39 (4), p. 3491–3499. Cited by: §1. Huynh et al. (2022) D. Huynh, J. Kuen, Z. Lin, J. Gu, and E. Elhamifar Open-vocabulary instance segmentation via robust cross-modal pseudo-labeling. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 7010–7021. Cited by: §1, §1, §2.1, §2.1, §2.1, §2.1, §3.1, §4.1, §4.1, §4.2, Table 2. Hwang et al. (2021) J. Hwang, S. W. Oh, J. Lee, and B. Han Exemplar-based open-set panoptic segmentation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1175–1184. Cited by: §4.1, §4.1, §4.2, Table 3, Table 3, Table 3, Table 3. Jiang et al. (2026) Y. Jiang, S. Qiang, W. Li, H. Zhao, and Y. Liang OV-kfa: open-vocabulary object detection via key feature alignment. Neurocomputing 659, p. 131790. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Kim et al. (2025a) D. M. Kim, S. Lee, and B. Kang Completely weakly supervised class-incremental learning for semantic segmentation. Pattern Recognition Letters 196, p. 16–23. External Links: ISSN 0167-8655 Cited by: §1. Kim et al. (2025b) H. Kim, B. Kang, and Y. Lee Generalized zero-shot learning for point cloud segmentation with evidence-based dynamic calibration. Proceedings of the AAAI Conference on Artificial Intelligence 39 (4), p. 4248–4256. Cited by: §1. Kirillov et al. (2023) A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. Girshick Segment anything. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , p. 3992–4003. External Links: Document Cited by: §2.3, §3.3.1. Kuznetsova et al. (2020) A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, et al. The open images dataset v4: unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision 128 (7), p. 1956–1981. Cited by: §5. Li et al. (2025) H. Li, J. Su, D. Zhou, and M. Cao MarIns3D: an open-vocabulary 3d instance segmentation model with mask refinement. Neurocomputing 651, p. 131018. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Li et al. (2024) X. Li, H. Yuan, W. Li, H. Ding, S. Wu, W. Zhang, Y. Li, K. Chen, and C. C. Loy OMG-seg: is one model good enough for all segmentation?. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 27948–27959. External Links: Document Cited by: §2.2. Liang et al. (2023) J. C. Liang, T. Zhou, D. Liu, and W. Wang CLUSTSEG: clustering for universal segmentation. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 20787–20809. External Links: Link Cited by: §2.2. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft coco: common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, p. 740–755. Cited by: §4.1. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 34892–34916. External Links: Link Cited by: §1, §2.3, §3.3.1. Liu et al. (2025a) S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang Grounding dino: marrying dino with grounded pre-training for open-set object detection. In Computer Vision – ECCV 2024, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Cham, p. 38–55. External Links: ISBN 978-3-031-72970-6 Cited by: §2.3, §3.3.1, §3.3.1, §4.1. Liu et al. (2025b) W. Liu, J. Lou, X. Wang, W. Zhou, J. Cheng, and X. Yang Physically-guided open vocabulary segmentation with weighted patched alignment loss. Neurocomputing 614, p. 128788. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Liu et al. (2021) Y. Liu, C. Ma, Z. He, C. Kuo, K. Chen, P. Zhang, B. Wu, Z. Kira, and P. Vajda Unbiased teacher for semi-supervised object detection. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §4.2, Table 2. Loshchilov (2019) I. Loshchilov Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §4.1. Ping et al. (2026) M. Ping, J. Ye, S. Cui, J. Zhang, Z. Hu, D. Zhou, P. Xu, and Y. Tian SPSRL: open-vocabulary semantic segmentation with spatial prior and semantic relation learning. Neurocomputing 677, p. 133088. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Qin et al. (2023) J. Qin, J. Wu, P. Yan, M. Li, R. Yuxi, X. Xiao, Y. Wang, R. Wang, S. Wen, X. Pan, and X. Wang FreeSeg: unified, universal and open-vocabulary image segmentation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 19446–19455. External Links: Document Cited by: §2.2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. Cited by: §2.3, §3.1, §3.3.1, §4.1. Ren et al. (2024) T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan, Z. Zeng, H. Zhang, F. Li, J. Yang, H. Li, Q. Jiang, and L. Zhang Grounded sam: assembling open-world models for diverse visual tasks. External Links: 2401.14159, Link Cited by: §1, §2.3, §2.3, §3.3.1. Sennrich et al. (2016) R. Sennrich, B. Haddow, and A. Birch Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), K. Erk and N. A. Smith (Eds.), Berlin, Germany, p. 1715–1725. Cited by: §4.1. Tran et al. (2024) D. D. T. Tran, B. Kang, and Y. Lee MSTA3D: multi-scale twin-attention for 3d instance segmentation. In Proceedings of the 32nd ACM International Conference on Multimedia, M ’24, New York, NY, USA, p. 1467–1475. External Links: ISBN 9798400706868 Cited by: §1. VS et al. (2023) V. VS, N. Yu, C. Xing, C. Qin, M. Gao, J. C. Niebles, V. M. Patel, and R. Xu Mask-free ovis: open-vocabulary instance segmentation without manual mask annotations. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 23539–23549. External Links: Document Cited by: §1, §1, §1, §2.1, §2.1, §2.1, §4.1, §4.1, §4.2, Table 2. Wang et al. (2022) W. Wang, J. Liang, and D. Liu Learning equivariant segmentation with instance-unique querying. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 12826–12840. External Links: Document, Link Cited by: §2.2. Wu et al. (2023) J. Wu, X. Li, H. Ding, X. Li, G. Cheng, Y. Tong, and C. C. Loy Betrayed by captions: joint caption grounding and generation for open vocabulary instance segmentation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , p. 21881–21891. Cited by: §1, §1, §2.1, §2.1, Table 1, §3.1, §3.2, §3.2, §3.3.2, §3.3.2, Figure 4, Figure 5, §4.1, §4.1, §4.1, §4.1, §4.1, §4.1, §4.1, §4.1, §4.2, §4.2, §4.2, §4.2, §4.2, Table 2, Table 3, Table 3, Table 3, Table 6. Wu et al. (2024a) J. Wu, X. Li, S. Xu, H. Yuan, H. Ding, Y. Yang, X. Li, J. Zhang, Y. Tong, X. Jiang, B. Ghanem, and D. Tao Towards open vocabulary learning: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (7), p. 5092–5113. External Links: Document Cited by: §2.1. Wu et al. (2024b) S. Wu, W. Zhang, L. Xu, S. Jin, X. Li, W. Liu, and C. C. Loy CLIPSelf: vision transformer distills itself for open-vocabulary dense prediction. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.4. Wu et al. (2025) X. Wu, Z. Xu, Q. Qian, and B. Huang GCD-net: global consciousness-driven open-vocabulary semantic segmentation network. Neurocomputing 636, p. 129982. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Xu et al. (2022) H. Xu, H. Chen, L. Liu, and Y. Yin Two-stage decision improves open-set panoptic segmentation. In BMVC, Vol. 2, p. 7. Cited by: §4.1, §4.1, §4.2, Table 3, Table 3, Table 3, Table 3. Xu et al. (2023) J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello Open-vocabulary panoptic segmentation with text-to-image diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 2955–2966. External Links: Document Cited by: §2.3. Xu et al. (2021) M. Xu, Z. Zhang, H. Hu, J. Wang, L. Wang, F. Wei, X. Bai, and Z. Liu End-to-end semi-supervised object detection with soft teacher. In Proceedings of the IEEE/CVF international conference on computer vision, p. 3060–3069. Cited by: §4.2, Table 2. Yuan et al. (2025) H. Yuan, X. Li, C. Zhou, Y. Li, K. Chen, and C. C. Loy Open-vocabulary sam: segment and recognize twenty-thousand classes interactively. In Computer Vision – ECCV 2024, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Cham, p. 419–437. External Links: ISBN 978-3-031-72775-7 Cited by: §2.3. Zareian et al. (2021) A. Zareian, K. D. Rosa, D. H. Hu, and S. Chang Open-vocabulary object detection using captions. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 14388–14397. Cited by: §1, §4.1, §4.1, §4.2, Table 2. Zhang et al. (2023) F. Zhang, T. Zhou, B. Li, H. He, C. Ma, T. Zhang, J. Yao, Y. Zhang, and Y. Wang Uncovering prototypical knowledge for weakly open-vocabulary semantic segmentation. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 73652–73665. External Links: Link Cited by: §2.4. Zheng et al. (2021) Y. Zheng, J. Wu, Y. Qin, F. Zhang, and L. Cui Zero-shot instance segmentation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 2593–2602. Cited by: §1, §4.1, §4.2, Table 2. Zhou et al. (2024) T. Zhou, W. Xia, F. Zhang, B. Chang, W. Wang, Y. Yuan, E. Konukoglu, and D. Cremers Image segmentation in foundation model era: a survey. External Links: 2408.12957, Link Cited by: §2.1. Zhu et al. (2022) Z. Zhu, Y. Wei, J. Wang, Z. Gan, Z. Zhang, L. Wang, G. Hua, L. Wang, Z. Liu, and H. Hu Exploring discrete diffusion models for image captioning. arXiv preprint arXiv:2211.11694. Cited by: §4.1. Zou et al. (2023) X. Zou, Z. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuan, N. Peng, L. Wang, Y. J. Lee, and J. Gao Generalized decoding for pixel, image, and language. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 15116–15127. External Links: Document Cited by: §2.2.