Paper deep dive

VL-AcneSeg: A Vision-Language Framework for Region-Aware Acne Lesion Segmentation

Sukju Oh, Soo Ick Cho, Dae Hun Suh, Sukkyu Sun

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 102

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Acne assessment is crucial for clinical decision-making, yet traditional grading and counting are subjective and fail to account for lesion size. While area-based assessment has emerged as a promising alternative, acne segmentation has continued to rely on general-purpose architectures. To address this gap, we propose VL-AcneSeg, a multimodal framework for acne lesion segmentation that leverages CLIP and region-level text prompts to incorporate spatial priors, enabling lesions to be localized across the whole face. Because region-level prompts indicate which facial areas contain lesions, we report a single global prompt, which requires no such information, as our primary setting. On our internal clinical dataset, VL-AcneSeg achieves a Dice score of 0.5082 and an IoU of 0.3407 under this protocol, the highest among all compared methods, including recent vision-language segmentation methods that are themselves given region-level prompts; region-level prompting raises these to 0.5296 and 0.3602. Moreover, lesion area measurements derived from our segmentation correlate with IGA scores at a level comparable to expert annotations (Pearson r = 0.719 versus 0.658). Notably, our framework maintains consistent performance across external validation datasets, performing reliably even on uncontrolled smartphone images without requiring additional training or fine-tuning. By pairing a protocol that requires no lesion-location information with area-based severity estimation, this work provides a foundation for objective acne assessment outside the clinic. Our implementation is publicly available at: this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

98,504 characters extracted from source content.

Expand or collapse full text

VL-AcneSeg: A Vision-Language Framework for Region-Aware Acne Lesion Segmentation Sukju Oh Soo Ick Cho Dae Hun Suh and Sukkyu Sun †thanks: ©˜2026 IEEE. Accepted for publication in the IEEE Journal of Biomedical and Health Informatics. Personal use of this material is permitted; all other uses require permission from IEEE.†thanks: This research was supported by the “Regional Innovation System & Education (RISE)” through the Seoul RISE Center, funded by the MOE (Ministry of Education) and the Seoul Metropolitan Government, and conducted in collaboration with Inskinlab Co., Ltd. (https://inskinlab.com/) (2026-RISE-01-007-05). This work was also supported by the Artificial Intelligence Convergence Innovation Human Resources Development supervised by the MSIT (Ministry of Science and ICT) and the IITP (Institute for Information & Communications Technology Planning & Evaluation) (IITP-2026-RS-2023-00254592). The authors thank Dong Hyo Kim, MD and Ji Won Lee, MD for the lesion annotations produced in the course of the study reported in [1]. (Corresponding author: Sukkyu Sun.)†thanks: Ethical Approval: This study was approved by the Institutional Review Board of Seoul National University Hospital (No. 2510-149-1690) and conducted in accordance with the Declaration of Helsinki.†thanks: Sukju Oh and Sukkyu Sun are with the Department of Computer Science and Artificial Intelligence, Dongguk University, Seoul 04620, Republic of Korea (e-mail: dhtjrwn119@dgu.ac.kr; sukkyu.sun@dgu.ac.kr).†thanks: Soo Ick Cho is with InSkin Lab Inc., Seoul 06193, Republic of Korea (e-mail: sooickcho@inskinlab.com).†thanks: Dae Hun Suh is with the Department of Dermatology, Seoul National University College of Medicine, and also with the Department of Dermatology, Seoul National University Hospital, Seoul 03080, Republic of Korea (e-mail: daehun@snu.ac.kr). Abstract Acne assessment is crucial for clinical decision-making, yet traditional grading and counting are subjective and fail to account for lesion size. While area-based assessment has emerged as a promising alternative, acne segmentation has continued to rely on general-purpose architectures. To address this gap, we propose VL-AcneSeg, a multimodal framework for acne lesion segmentation that leverages CLIP and region-level text prompts to incorporate spatial priors, enabling lesions to be localized across the whole face. Because region-level prompts indicate which facial areas contain lesions, we report a single global prompt, which requires no such information, as our primary setting. On our internal clinical dataset, VL-AcneSeg achieves a Dice score of 0.5082 and an IoU of 0.3407 under this protocol, the highest among all compared methods, including recent vision-language segmentation methods that are themselves given region-level prompts; region-level prompting raises these to 0.5296 and 0.3602. Moreover, lesion area measurements derived from our segmentation correlate with IGA scores at a level comparable to expert annotations (Pearson r = 0.719 versus 0.658). Notably, our framework maintains consistent performance across external validation datasets, performing reliably even on uncontrolled smartphone images without requiring additional training or fine-tuning. By pairing a protocol that requires no lesion-location information with area-based severity estimation, this work provides a foundation for objective acne assessment outside the clinic. Our implementation is publicly available at: https://github.com/sukjuoh/VL-AcneSeg Index Terms: Acne Segmentation, Vision-Language Model, Medical Imaging. Input Prompt: “acne lesion on the left cheek” (a) Ground Truth (b) Localization (c) Refinement Fig. 1: Conceptual framework of VL-AcneSeg. Given a regional text prompt, VL-AcneSeg localizes potential lesion sites through CLIP-based semantic alignment (Middle) and then performs refinement to produce the segmentation mask (Right). I Introduction Acne vulgaris is one of the most prevalent chronic skin disorders worldwide [2], ranking among the top three most common skin conditions and affecting up to 85% of adolescents and young adults during their lifetime [3, 4, 5]. Because it predominantly involves the facial region, where lesions are highly visible, acne can lead to considerable psychosocial stress [6]. Acne lesions are broadly categorized into non-inflammatory types (e.g., comedones), which can be removed through simple extraction, and inflammatory types (e.g., papules, pustules, nodules), which often require pharmacological treatment depending on severity. Since inflammatory lesions are more directly tied to disease progression and therapeutic decisions, developing automated tools that can precisely quantify them is a critical priority for objective acne management. Severity is assessed in clinical practice by global grading or by lesion counting, and each has its own limitation. Global grading compares a patient’s presentation against standardized reference cases [7] and carries the inter- and intra-observer variability that entails [8, 9]. Counting individual lesions by type and region is more quantitative [10, 11] and has been automated by recent detection methods [12, 13, 1], but it records a lesion as present or absent, so a small papule and a large confluent nodule contribute equally [14, 15]. Area-based assessment is a promising alternative because it reflects lesion extent as well as lesion number. Gazeau et al. combined lesion areas with lesion-specific severity scores into an overall index, offering a more objective basis for severity assessment [16]. Despite this potential, acne segmentation itself has received limited attention, largely due to the scarcity of annotated data and the inherent difficulty of the task [17, 18]. Pixel-level annotation of numerous small lesions requires dermatological expertise and is both costly and time-consuming. In addition, variations in skin tone, illumination, and imaging conditions introduce substantial appearance variability, while lesions often have blurred boundaries, irregular shapes, and subtle contrasts with surrounding skin, with confounding factors such as moles, scars, and pigmentation further complicating discrimination. Reflecting these challenges, previous studies on acne segmentation have typically relied on general-purpose architectures and conventional training strategies, often applied to cropped patches around lesions rather than full-face images [19, 20, 21]. While such approaches simplify the task, they limit the ability to capture the global distribution of lesions, which is essential for area-based severity assessment. Even full-face models [16, 22] have introduced little segmentation methodology tailored to the specific challenges of acne. Vision-language models offer a way forward, since they allow simple textual cues—such as the approximate facial region in which lesions are present—to be supplied as spatial priors for segmentation. Existing vision-language segmentation methods, however, are built around a different question: they use text to specify which class to segment, and their alignment relies on the contrast between classes, which is absent when there is a single lesion class. What acne does offer is a face whose anatomy is stable and nameable, so text can instead specify where the target may appear. Such region-level descriptions carry only coarse spatial information at inference time, yet can narrow where the model searches for lesions across the face. Accordingly, we propose VL-AcneSeg, a multimodal framework that utilizes region-level text prompts to guide the segmentation of inflammatory lesions. As illustrated in Fig. 1, the model aligns these textual prompts with visual features via CLIP to identify potential lesion sites, which are then refined into precise segmentation masks. Under region-level prompting the design additionally yields a severity estimate for each facial region. Our contributions are as follows: • We recast text-guided segmentation for a setting in which open-vocabulary alignment does not apply. Existing paradigms use text to specify which class to segment, which carries little information when there is a single lesion class; we instead use it to specify where the target may appear, so that the text channel supplies a spatial rather than a categorical constraint. • We adapt the patch-wise similarity paradigm to acne through tiled high-resolution CLIP encoding, layer-targeted V-V attention and CLS-token-guided cross-attention, so that the alignment operates at the resolution the lesions demand. • We report a deployment-realistic protocol that requires no reference-derived information, evaluate it on external clinical and smartphone datasets without retraining, and show that the resulting lesion areas track Investigator’s Global Assessment severity at the level of the reference annotations. I Related Work I-A Acne Lesion Segmentation Acne lesion segmentation has been explored only to a limited extent compared to other dermatological conditions. Existing studies often rely on cropped image patches rather than full-face images, limiting their ability to capture the global lesion distribution [19, 20, 21], and typically adopt standard architectures such as U-Net without acne-specific design [22]. Gazeau et al. [16] employed a conventional segmentation backbone within their area-based severity framework, but their focus was on the evaluation methodology rather than the segmentation model itself. Moreover, absolute segmentation metrics reported in this domain remain modest across studies, reflecting the intrinsic difficulty of distinguishing acne from visually similar skin features such as PIH, scars, and pores [22, 16]. A systematic review covering 2017–2025 screened 345 articles and identified 29 eligible studies, the majority addressing severity grading or lesion counting rather than pixel-level segmentation [18]. Pixel-level acne segmentation has therefore received limited attention, and the methods that do address it were not designed for the setting that area-based assessment requires: the whole face at once, with lesions that are numerous and easily confused with the marks they leave behind. I-B Open-Vocabulary and Text-Guided Segmentation Vision-language models such as CLIP [23] and ALIGN [24] learn joint image–text representations that support open-vocabulary understanding and zero-shot transfer [25, 26, 27]. This has motivated their use in dense prediction, through transformer-based fusion [28, 29, 30] and open-vocabulary paradigms such as OVSeg [31] and ODISE [32], which match class-agnostic region proposals with text embeddings. However, these paradigms face significant practical and logical hurdles in acne segmentation. For ROI-based methods, if a mask generator can successfully propose an ROI for a specialized single-class lesion, the subsequent text-alignment step becomes effectively redundant. Furthermore, the visual features within an acne lesion are often too localized and subtle to provide sufficient discriminative information for meaningful alignment, while random masking strategies like MaskCLIP [33] fail to maintain the structural integrity of such small-scale lesions during training. Moreover, the lack of distinct positive-negative pairs in localized skin imagery makes it difficult to establish a viable contrastive learning framework. More recently, CAT-Seg [34] introduced a patch-wise cosine similarity paradigm that formulates segmentation as a direct alignment between CLIP’s visual and textual embeddings, with subsequent extensions such as SED [35] and ESC-Net [36] refining this formulation for natural-image open-vocabulary settings; we compare against SED directly, whereas ESC-Net couples the cost volume with SAM, which enters our comparison as a separate fine-tuned baseline. The most fundamental distinction between our approach and CAT-Seg lies in the role of text: CAT-Seg consumes class-level prompts that specify what object to segment, whereas VL-AcneSeg consumes region-level prompts that specify where the target may appear. Building on this distinction, our framework repurposes the alignment itself: the text channel supplies a spatial constraint rather than a categorical one, which extends the formulation to dense single-class segmentation, a setting where prior open-vocabulary methods struggle for want of inter-class contrast. The architectural components that follow are adaptations required to make this alignment operate at the resolution the task demands. A complementary line of work prompts CLIP with visual rather than textual references: PBIP [37] constructs class prototypes from a training image bank and uses them as image prompts for weakly supervised histopathological segmentation. Such prototype prompts encode appearance, whereas the region-level prompts used here encode anatomical location, and the two forms of guidance are therefore complementary. I-C Text-Guided Segmentation Model in the Medical Domain In the medical domain, text has mainly been used to identify the target structure. LViT [38] integrates BERT [39] text embeddings into a hybrid CNN-Transformer framework, while VLM-based approaches such as MedCLIP-SAM [40], SegICL [41], and OMT-SAM [42] combine pretrained text encoders with segmentation frameworks for zero-shot or prompt-based segmentation of anatomical structures. More recent studies have begun to exploit the spatial content of clinical text explicitly. TVE-Net [43] converts the location descriptions in radiology reports into a “text view” through a hand-designed positional probability function, assigning a lesion probability to each image region so that textual spatial information is injected into the segmentation network. STPNet [44] instead retrieves multi-scale textual descriptions from a curated medical text repository during training and thereby removes the need for text input at inference, while EviVLM [45] introduces evidential learning to quantify and mitigate the modality gap between image and text representations. Our framework shares with TVE-Net the premise that textual location information can serve as a spatial prior, but realises it differently. TVE-Net maps free-text radiology reports to a probability map through a hand-designed positional function, whereas VL-AcneSeg keeps the prior inside the joint embedding space: fixed-template region prompts are encoded by the CLIP text encoder and aligned with patch embeddings through cosine similarity, so that the prior is a learned visual–textual correspondence rather than a predefined geometric mapping, and no narrative report is required. These advances have moreover addressed radiology, where lesions are comparatively large and few in number; to our knowledge text-guided segmentation has not been applied to acne, whose lesions are small, numerous and distributed across the whole face. Fig. 2: Facial regions were defined using Mediapipe Face Mesh [46], and lesion masks were assigned to each region according to their centroid location. Region-wise masks and text prompts (“acne lesion on the region”) were then generated to provide spatial guidance for segmentation. Fig. 3: Overview of VL-AcneSeg. Overlapping tiles are encoded by CLIP and reassembled into a unified feature map, whose patch-wise cosine similarity with the text embeddings yields the localization map; intermediate vision features and the CLS token provide additional guidance. A Swin Transformer, a CNN bottleneck and a decoder then refine this map into the lesion mask. I Method I-A Data Preparation We collected standardized clinical photographs from 258 patients (38.4% male; mean age 22.7 ± 5.97 years) who visited the Department of Dermatology, Seoul National University Hospital, using Canon EOS 550D and Nikon D7100 cameras. All participants were of Korean ethnicity, and Fitzpatrick phototype was not routinely recorded. The same cohort was described previously in the context of automated lesion detection and counting [1]; the present study is the first to derive pixel-level segmentation masks from these data. A total of 1,213 images carry 20,699 annotated acne lesions. The lesion inventory originates from the study of this cohort reported in [1], where two dermatology residents independently marked each lesion. For the present study these annotations were converted into pixel-level masks with LabelMe [47], and a board-certified dermatologist reviewed every image, corrected the delineations and adjudicated the final set. The reference standard is therefore fixed by a single senior reader rather than by agreement among readers. The annotations cover five lesion types (closed and open comedones, papules, nodules/cysts, and pustules). Because clinical acne severity assessment is based primarily on inflammatory lesions, and because comedones are frequently indistinguishable at the pixel level in standardized clinical photographs, this study targets inflammatory lesions only (papules, nodules/cysts, and pustules); patients without any inflammatory lesion were excluded, leaving 222 patients, 821 images and 4,655 annotated lesion instances. To generate region masks and text prompts, each facial image was divided into anatomical regions (forehead, cheeks, nose, and chin) using MediaPipe Face Mesh landmarks [46]. As illustrated in Fig. 2, the center coordinate of each lesion mask was used to determine its corresponding facial region, and the mask was assigned accordingly. For every region containing one or more lesions, an individual text prompt was generated following a fixed template format, “acne lesion on the region.” For example, if acne lesions were present on the forehead, chin, and neck, three separate prompts (“acne lesion on the forehead,” “acne lesion on the chin,” and “acne lesion on the neck”) were produced. These prompts indicate the approximate location of lesions across facial regions. I-B Model Overview Our goal is to segment acne lesions from facial images by leveraging textual descriptions of facial regions as spatial priors. Given an input image and corresponding region-level text prompts, our model outputs a pixel-wise segmentation mask indicating the locations of acne lesions. The overall architecture is illustrated in Fig. 3. The input image is divided into tiles, encoded by the CLIP vision encoder and reassembled into a visual feature map, whose patch-wise cosine similarity with the CLIP text embeddings yields a localization map that is then passed to a Swin Transformer. Intermediate visual features and the CLS tokens provide additional guidance, and a cross-attention module conditioned on the text embeddings refines the representation before the mask decoder, so that region-level cues are combined with visual features in a patch-aligned manner. I-C Localization: High-Resolution Features and Patch-wise Similarity Acne lesions are typically small and have blurry boundaries, making high-resolution input crucial for precise segmentation. However, the CLIP vision encoder Φ (ViT-L/14@336px) does not support inputs larger than 336×336336× 336. To overcome this limitation, we divide each input image X∈ℝ3×H×WX ^3× H× W into overlapping tiles of size k×k× k (k=336k=336) and process each tile individually through Φ . The tiling layout follows from two constraints rather than from a search: the tile size is fixed at 336336 by the encoder, and clinical facial photographs have a 3:43:4 width-to-height ratio, so tiling the image with square patches requires four rows and three columns. With the longer side resampled to 10241024, the smallest such layout is the 4×34× 3 configuration used here; coarser layouts reduce the pixel density per tile, while the next admissible refinement quadruples the number of CLIP forward passes for tiles that each cover less anatomical context. For each tile Pi,jP_i,j, we obtain the CLS token and patch embeddings: [zi,jcls,Zi,j]=Φ⁡(Pi,j),[z_i,j^cls,Z_i,j]= (P_i,j), (1) where Zi,j∈ℝc×h×wZ_i,j ^c× h× w is the token grid for the (i,j)(i,j)-th tile (h=w=24,c=1024h=w=24,\;c=1024). All tile embeddings are then reassembled into a single feature map S by placing them at their corresponding spatial locations and averaging the overlapping regions using a Gaussian weight window G: S=∑i,jZi,j⊙Gi,j∑i,jGi,j+ϵ,S= _i,jZ_i,j G_i,j _i,jG_i,j+ε, (2) where ⊙ denotes element-wise multiplication, Gi,jG_i,j is a 2D Gaussian kernel of size h×wh× w centered on tile (i,j)(i,j) with standard deviation σ=h/4σ=h/4 that downweights border patches to reduce stitching artifacts at tile boundaries, and ϵ=10−6ε=10^-6 is a small constant ensuring numerical stability. Finally, the stitched feature map S is upsampled to a fixed resolution and normalized at each spatial location: Snorm(x,y)=S⁡(x,y)‖S⁡(x,y)‖2.S_norm(x,y)= S(x,y)\|S(x,y)\|_2. (3) Lesions are localized by aligning visual features with the text prompts (Fig. 1 (b)). The NTN_T region prompts Tkk=1NT\T_k\_k=1^N_T are passed through the CLIP text encoder Γ : tk=Γ(Tk)∈ℝCk=1NT.\t_k= (T_k) ^C\_k=1^N_T. (4) Let S∈ℝC×H×WS ^C× H× W denote the visual feature map extracted from the image encoder Φ . Before computing similarity, both the visual features S and text embeddings tkt_k are normalized along the channel dimension. For each text embedding tkt_k, we compute a patch-wise cosine similarity map with the visual features as: Mk(h,w)=Sh,w⊤tk‖Sh,w‖2‖tk‖2,M_k(h,w)= S_h,w t_k\|S_h,w\|_2\|t_k\|_2, (5) yielding a set of similarity maps: M=Mkk=1NT∈ℝH×W×NT,M=\M_k\_k=1^N_T ^H× W× N_T, (6) each encoding the alignment between visual patches and one region-level description. I-D Feature Enhancement CLIP [23] aligns visual and textual representations through a contrastive objective, but its attention is biased toward large and salient regions and is often dominated by a few tokens, which suppresses the fine-grained local features on which small lesions depend—a limitation also noted in AnomalyCLIP [48]. To address this issue, we adopt the V-V attention mechanism introduced by AnomalyCLIP, but redesign its deployment strategy for fine-grained lesion segmentation. Whereas AnomalyCLIP applies V-V attention across multiple intermediate layers for global anomaly detection, we apply it only at the final feature extraction layer. The reason is that V-V attention replaces query–key routing with value–value affinity, so each token is re-expressed as a weighted average of tokens with similar values. Applied once, this suppresses the few dominant tokens that otherwise absorb the attention mass; applied repeatedly, the same averaging acts as a low-pass filter over the token grid, and the patch-aligned spatial structure that dense prediction depends on is progressively smoothed away. Anomaly detection tolerates this because it requires only a coarse image-level score, whereas lesions a few pixels across do not. Table I confirms the distinction empirically. Specifically, let V∈ℝN×dV ^N× d denote the value embeddings extracted from this target layer of the CLIP vision encoder, where N is the number of tokens and d is the embedding dimension. We compute a token-token affinity matrix through a scaled dot product between the value vectors: Avv=Softmax(VV⊤d),A_v=Softmax ( V d ), (7) and use it to refine the visual features: V′=AvvV.V =A_vV. (8) The refined features are then passed through the original output projection of that layer and added to the original layer output to form the final feature map. Fine-grained local information diminishes with layer depth, so we additionally take the feature map of the 12th CLIP layer as guidance and fuse it before the bottleneck (Section I-E). We further utilize CLS tokens to inject coarse spatial information derived from the image tiles. Since each image is divided into 4×34× 3 tiles and individually processed by the CLIP vision encoder, we obtain one CLS token cn∈ℝCc_n ^C for each tile n=1,…,Nn=1,…,N. Given the normalized text embeddings tkk=1NT\t_k\_k=1^N_T , we compute the cosine similarity between all CLS tokens and text embeddings to produce tile-text similarity weights: wn,k=exp⁡(cn⊤tk)∑n′=1Nexp⁡(cn′⊤tk).w_n,k= (c_n t_k) _n =1^N (c_n t_k). (9) These weights are then used to compute a weighted combination of CLS tokens for each text embedding, representing the pooling operation (P) illustrated in Fig. 3: c¯k=∑n=1Nwn,kcn. c_k= _n=1^Nw_n,kc_n. (10) The resulting ck¯ c_k encodes spatial cues indicating which tiles are semantically aligned with each text prompt. Finally, ck¯ c_k is concatenated with the corresponding text embedding tkt_k, and this combined representation is used as the query for the cross-attention module. I-E Refinement and Mask Decoding The patch-wise similarity maps M∈ℝB×NT×H×WM ^B× N_T× H× W are first passed through a convolution layer to introduce a channel dimension, resulting in M′=Conv(M)∈ℝ(B×NT)×CM×H×WM =Conv(M) ^(B× N_T)× C_M× H× W. In parallel, the original CLIP vision encoder feature map S is replicated and concatenated with M′M to form the input feature FinF_in. This feature map is processed by a Swin Transformer with W-MHSA and SW-MHSA blocks to capture long-range spatial dependencies (Fig. 3). The guidance feature FguideF_guide, taken from the 12th layer of the CLIP vision encoder, is passed through a convolution layer, replicated NTN_T times and concatenated with FswinF_swin before the bottleneck: Fbottle_in=Concat[Fswin,RepeatNT(Fguide)],F_bottle\_in=Concat[F_swin,Repeat_N_T(F_guide)], (11) where Fguide∈ℝB×Cg×H′×W′F_guide ^B× C_g× H × W . The bottleneck refines this representation to integrate spatial similarity cues with low-level visual features: Fbottle=Bottleneck(Fbottle_in)∈ℝ(B×NT)×Cb×H′×W′.F_bottle=Bottleneck(F_bottle\_in) ^(B× N_T)× C_b× H × W . (12) The weighted CLS token c¯k c_k and the text embedding tkt_k form the query Q: Q=Concat[c¯k,tk]k=1NT∈ℝNT×Cq.Q=\Concat[ c_k,t_k]\_k=1^N_T ^N_T× C_q. (13) which is broadcast to Q′∈ℝ(B×NT)×Cq×H′×W′Q ^(B× N_T)× C_q× H × W and interacts with the refined visual features through cross-attention: Ffused=CrossAttn(Q′,Fbottle,Fbottle).F_fused=CrossAttn(Q ,F_bottle,F_bottle). (14) The mask decoder produces the segmentation mask Y Y: the fused feature FfusedF_fused is processed through Multi-Head Self-Attention (MHSA) and Layer Normalization (LN), then concatenated with the similarity map M′M to anchor the predictions to the initial localization results. The final mask is produced through learned upsampling via a Convtranspose layer followed by a convolution: Y^=Conv(CLOSEConvTranspose(CLOSEOPENOPENConcat(LN(MHSA(Ffused)),M′))). split Y=Conv(&ConvTranspose(\\ &Concat(LN(MHSA(F_fused)),M ))). split (15) I-F Loss Function We supervise the network with two complementary losses: a global segmentation loss over the full-face mask, and a region-specific loss for each regional prediction. For the global segmentation, we adopt the weighted IoU and weighted BCE losses from PraNet [49], which emphasize hard pixels by upweighting challenging regions: ℒtotal=ℒwIoU(Y^,Y)+ℒwBCE(Y^,Y).L_total=L_wIoU( Y,Y)+L_wBCE( Y,Y). (16) For region-specific supervision, we apply focal loss [50] (with α=0.75α=0.75) to each regional prediction Y^k Y_k associated with a text prompt TkT_k: ℒregion=1NT∑k=1NTℒfocal(Y^k,Yk),L_region= 1N_T _k=1^N_TL_focal( Y_k,Y_k), (17) where YkY_k is the ground-truth mask for the k-th facial region and NTN_T is the number of region-level prompts. This term supervises the similarity map M directly, encouraging alignment between visual patches and their regional prompts, while the focal weighting emphasizes small or ambiguous lesions. The final objective combines both terms: ℒ=0.6ℒtotal+0.4ℒregion.L=0.6L_total+0.4L_region. (18) IV Experiments IV-A Setup We evaluated our model on three datasets. The first is the internal test set described in Section I-A. The internal dataset was split into training, validation, and test sets at an 8:1:1 ratio on a patient-wise basis to prevent data leakage across multiple images from the same subject; the split was defined on the full cohort of 258 patients, and after excluding patients without inflammatory lesions the realized split comprised 190, 16 and 16 patients. Investigator’s Global Assessment (IGA) scores were not recorded at acquisition and were assigned retrospectively at the visit level by three board-certified dermatologists, for the internal test set only, since IGA is used solely for the clinical correlation analysis (Section IV-G) and never for model training or selection. The 58 internal test images correspond to 22 visits from 16 patients, of which 3, 7, 8 and 4 were graded 1 to 4 on the standard five-point scale and none as grade 0; the grading protocol is given in Section S1 of the Supplementary Material. The other two datasets were derived from the AI Hub Korean Skin Condition Measurement Dataset11 1 https://w.aihub.or.kr, developed to support facial skin analysis for Korean individuals. TABLE I: Composition of the internal splits and the two external test sets, counted over the inflammatory lesion masks used in this study. The full annotated cohort comprises 258 patients, 1,213 images and 20,699 lesions across five types. The internal dataset was divided patient-wise, and each external image originates from a distinct subject. Split Patients Images Lesions Internal (Seoul National University Hospital) Train 190 703 3,814 Validation 16 60 494 Test 16 58 347 Total 222 821 4,655 External (AI Hub) Controlled 71 71 171 Real-world 56 56 101 From this collection we constructed two external test sets, one for a controlled imaging setting and one for an uncontrolled, real-world setting. Because the collection contains many images without visible inflammatory lesions, candidates were screened with an acne detection model trained on the public ACNE04 dataset [51] and then sampled at random; the dermatologist who annotated the internal set reviewed the sampled images and excluded those without confirmed inflammatory acne, yielding 71 controlled-setting and 56 real-world images. The screening model predicts bounding boxes rather than masks and its outputs were used only for candidate selection, never in the annotations or in any evaluation. Pixel-level masks were newly constructed for the present study by the same board-certified dermatologist who adjudicated the internal set, so both sets share one reference reader; unlike the internal set, no prior box inventory was available here, so the lesions were located as well as delineated. Text prompts were generated as described in Section I-A, and the composition of every split is summarized in Table I. The two subsets differ from the internal cohort in lesion burden and probe different kinds of imaging shift: the controlled subset was acquired with digital cameras under a setup comparable to the internal cohort, whereas the real-world subset was acquired with smartphones. The screening procedure, the resulting sampling bias and the available metadata are detailed in Section S1 of the Supplementary Material. IV-B Evaluation Metrics We used the Dice coefficient and Intersection over Union (IoU): Dice=2|Y∩Y^||Y|+|Y^|,Dice= 2|Y∩ Y||Y|+| Y|, (19) IoU=|Y∩Y^||Y∪Y^|.IoU= |Y∩ Y||Y∪ Y|. (20) We further report pixel-wise Precision and Recall, together with lesion-level counts per image. A predicted connected component is counted as a true positive (TP/img) if it overlaps a reference lesion in at least one pixel, and as a false positive (FP/img) otherwise. Specificity is not reported: background pixels dominate facial images, so it exceeds 0.999 for every method and carries no information. Let Ω denote the set of valid image pixels, Y the ground-truth lesion mask and Y Y the predicted mask. The pixel-wise counts are defined as TP=|Y∩Y^|TP=|Y∩ Y|, FP=|Y^∖Y|FP=| Y Y|, FN=|Y∖Y^|FN=|Y Y|, and TN=|Ω|−|Y∪Y^|TN=| |-|Y∪ Y|, giving Precision=TPTP+FP,Precision= TPTP+FP, (21) Recall=TPTP+FN.Recall= TPTP+FN. (22) All metrics are aggregated over the entire test set by accumulating TPTP, FPFP, FNFN, and TNTN across images before computing the ratios (micro-averaging). Lesion-level recall, precision and F1 restricted to the smallest quartile of lesions (below 500 px2), together with a threshold-free comparison based on the area under the precision–recall curve, are reported for all methods in Table S5 of the Supplementary Material. Zero-padded regions (Section IV-C) are excluded from Ω , so the denominator is the same effective area for every method. IV-C Implementation and Training Protocol The model was implemented in PyTorch and trained on an NVIDIA A6000 GPU (48 GB). We used the AdamW [52] optimizer with a learning rate of 2×10−42× 10^-4 for the main network and 2×10−62× 10^-6 for the CLIP encoders, for 40 epochs with a batch size of 4. The optimizer’s betas were set to (0.9, 0.999). Baselines used the default hyperparameters of their original papers with no additional search, and were initialized from the pretrained weights those implementations provide—ViT-B/16 for TransUNet, Swin Transformer for Swin-UNet, MiT for SegFormer, ImageNet ResNet encoders for PSPNet, DeepLabV3+ and CE-Net, VMamba for Swin-UMamba† , and DeiT-Base [53] with Bio_ClinicalBERT [54] for EviVLM—while U-Net, U-Net++ and nnU-Net were trained from random initialization. SAM was initialized from the pretrained ViT-L checkpoint, with its image encoder and mask decoder fully fine-tuned and the prompt encoder left unused, as no point or box prompts were supplied. Every model, including ours, used the identical patient-wise 8:1:1 split and no data augmentation. For each model we selected the checkpoint with the highest validation IoU, and fixed the binarization threshold at 0.5. CAT-Seg and SED share our input resolution and 4×34× 3 tiling, so the three CLIP-based methods are compared at matched resolution. All others take the image resized to a height of 1024, zero-padded along the width where the architecture requires a fixed input shape, with padded regions excluded from the loss and the evaluation. IV-D Results TABLE I: Quantitative comparison of acne lesion segmentation on the internal and external datasets. Shaded rows (∗ ) are vision-language methods given region-level prompts derived from the reference masks, an oracle setting; CAT-Seg and SED use the same tiling scheme and input resolution as the proposed method, so that the CLIP-based methods are compared at matched resolution. All remaining rows, including VL-AcneSeg (Global), require no reference-derived information. TP/img and FP/img are the numbers of predicted lesion components per image that do or do not overlap a reference lesion; neither is ranked. Point estimates are computed over all test-set pixels; 95% confidence intervals are obtained by patient-level cluster bootstrap (1,000 resamples). Best values are in bold and second-best are underlined. Model Backbone Internal Data External Data (Evaluation Only) (Training & Evaluation) Controlled Setting (1) Real-world Setting (2) IoU Dice Precision Recall TP/img FP/img IoU Dice Precision Recall TP/img FP/img IoU Dice Precision Recall TP/img FP/img VL-AcneSeg (Region)∗ ViT-L 0.3602 [0.309, 0.399] 0.5296 [0.472, 0.570] 0.5202 0.5393 4.29 2.48 0.4232 [0.376, 0.470] 0.5948 [0.546, 0.640] 0.6110 0.5794 1.64 0.60 0.3044 [0.2476, 0.3483] 0.4667 [0.3969, 0.5166] 0.4882 0.4471 1.50 0.82 SED [35]∗ ConvNeXt-L 0.3182 [0.255, 0.373] 0.4828 [0.406, 0.544] 0.6291 0.3917 4.09 3.74 0.3070 [0.263, 0.355] 0.4698 [0.416, 0.524] 0.7509 0.3419 1.97 1.51 0.1650 [0.121, 0.229] 0.2833 [0.215, 0.373] 0.6365 0.1822 0.98 0.32 CAT-Seg [34]∗ ViT-L 0.2909 [0.226, 0.374] 0.4507 [0.368, 0.545] 0.5832 0.3673 3.09 1.45 0.2287 [0.202, 0.255] 0.3722 [0.336, 0.407] 0.4101 0.3407 1.70 4.44 0.1300 [0.098, 0.160] 0.2300 [0.179, 0.276] 0.2050 0.2622 0.82 0.46 EviVLM [45]∗ ViT-B 0.3290 [0.261, 0.385] 0.4952 [0.414, 0.556] 0.5289 0.4654 4.57 5.72 0.2584 [0.213, 0.311] 0.4106 [0.351, 0.474] 0.7417 0.2839 1.00 0.24 0.2487 [0.182, 0.312] 0.3984 [0.308, 0.476] 0.5134 0.3254 0.66 0.82 VL-AcneSeg (Global) ViT-L 0.3407 [0.282, 0.392] 0.5082 [0.440, 0.563] 0.4938 0.5238 3.69 2.43 0.3814 [0.345, 0.425] 0.5522 [0.513, 0.597] 0.5011 0.6149 1.83 0.94 0.2671 [0.213, 0.313] 0.4216 [0.352, 0.477] 0.3277 0.5908 1.23 0.93 SAM [55] ViT-L 0.3216 [0.251, 0.409] 0.4867 [0.401, 0.581] 0.5601 0.4303 3.21 1.66 0.3764 [0.3306, 0.4284] 0.5469 [0.4969, 0.5999] 0.7064 0.4462 1.54 0.64 0.2546 [0.1568, 0.2919] 0.4058 [0.2710, 0.4519] 0.3452 0.4923 0.91 1.34 U-Net [56] CNN 0.1418 [0.107, 0.167] 0.2483 [0.193, 0.286] 0.1490 0.7443 4.86 19.07 0.1771 [0.1549, 0.1980] 0.3009 [0.2682, 0.3305] 0.1857 0.7931 2.13 13.34 0.0863 [0.0620, 0.1148] 0.1590 [0.1167, 0.2060] 0.0884 0.7869 1.43 9.82 PSPNet [57] CNN 0.1802 [0.134, 0.212] 0.3053 [0.236, 0.350] 0.2337 0.4402 2.33 1.21 0.3368 [0.3015, 0.3724] 0.5038 [0.4633, 0.5427] 0.4157 0.6394 1.79 1.43 0.1199 [0.0814, 0.1599] 0.2141 [0.1506, 0.2757] 0.1375 0.4826 0.91 1.62 DeepLabV3+ [58] CNN 0.1807 [0.118, 0.229] 0.3061 [0.211, 0.373] 0.2592 0.3737 2.22 4.71 0.2603 [0.2058, 0.3185] 0.4131 [0.3413, 0.4832] 0.5224 0.3416 1.23 0.56 0.1681 [0.1018, 0.2344] 0.2878 [0.1848, 0.3798] 0.4078 0.2224 0.52 0.38 U-Net++ [59] CNN 0.1575 [0.103, 0.221] 0.2721 [0.186, 0.362] 0.3802 0.2118 3.34 11.29 0.1986 [0.1627, 0.2342] 0.3315 [0.2798, 0.3796] 0.4420 0.2650 1.71 4.37 0.2517 [0.1817, 0.3309] 0.4022 [0.3076, 0.4973] 0.3455 0.4822 1.39 3.88 CE-Net [60] CNN 0.2215 [0.161, 0.278] 0.3626 [0.277, 0.434] 0.2836 0.5028 3.34 4.31 0.1248 [0.0810, 0.1922] 0.2220 [0.1499, 0.3224] 0.1495 0.4308 1.81 8.06 0.1234 [0.0748, 0.1900] 0.2197 [0.1392, 0.3194] 0.1306 0.6915 1.57 7.93 SegFormer [61] Transformer 0.2047 [0.146, 0.263] 0.3399 [0.254, 0.417] 0.2865 0.4177 3.90 10.83 0.1328 [0.1119, 0.1538] 0.2345 [0.2013, 0.2667] 0.1537 0.4974 1.79 17.21 0.0222 [0.0142, 0.0317] 0.0434 [0.0281, 0.0614] 0.0233 0.3072 0.88 16.38 TransUNet [62] Transformer 0.2624 [0.189, 0.352] 0.4157 [0.317, 0.520] 0.5210 0.3459 3.09 2.36 0.1360 [0.0937, 0.1845] 0.2394 [0.1713, 0.3115] 0.3155 0.1928 0.93 0.60 0.0328 [0.0097, 0.0794] 0.0634 [0.0191, 0.1471] 0.0426 0.1245 0.23 1.57 Swin-UNet [63] Transformer 0.2089 [0.150, 0.264] 0.3456 [0.262, 0.418] 0.3266 0.3670 3.12 7.53 0.2005 [0.1563, 0.2476] 0.3341 [0.2703, 0.3969] 0.4446 0.2676 1.19 3.79 0.1454 [0.0737, 0.2346] 0.2539 [0.1373, 0.3800] 0.2390 0.2709 0.73 3.84 nnU-Net [64] CNN 0.1909 [0.142, 0.242] 0.3206 [0.248, 0.390] 0.2311 0.5234 4.09 17.97 0.2079 [0.177, 0.240] 0.3442 [0.301, 0.387] 0.2927 0.4177 1.53 3.50 0.0454 [0.017, 0.084] 0.0869 [0.033, 0.155] 0.0575 0.1779 0.39 4.62 Swin-UMamba† [65] Mamba 0.2877 [0.207, 0.355] 0.4468 [0.343, 0.524] 0.3329 0.6793 4.98 7.84 0.3278 [0.295, 0.364] 0.4937 [0.456, 0.534] 0.4933 0.4941 1.69 2.13 0.2552 [0.188, 0.326] 0.4066 [0.316, 0.492] 0.3566 0.4729 0.75 11.68 Fig. 4: Qualitative comparison across the three datasets: (A) internal, (B) external controlled, and (C) external smartphone images. The left column shows the reference annotation on the full face; the green box marks the region enlarged in the remaining columns, so that lesions a few pixels across remain visible at print size. Predicted masks are overlaid in red, and both prompting conditions of the proposed method are shown. A comparison including all baselines and both prompting conditions is given in Figs. S3–S5 of the Supplementary Material. As summarized in Table I, VL-AcneSeg attains the highest IoU and Dice scores in every evaluation setting. While traditional architectures like U-Net achieve high recall, they tend to be indiscriminate and often fail to distinguish acne lesions from visually similar conditions such as scars, post-inflammatory hyperpigmentation (PIH), and rashes. As shown in Fig. 4, baseline models frequently misclassify these non-acne conditions as lesions, whereas VL-AcneSeg correctly identifies true acne through vision-language alignment. Because the vision-language baselines are all supplied with region-level prompts, they are compared against VL-AcneSeg under the same regional prompting condition, whereas the remaining methods, which take no text input, are compared against the global-prompt configuration; in both cases the difference is assessed by paired patient-level cluster bootstrap, with the two-sided p value taken as twice the proportion of resamples in which the difference lies on the opposite side of zero from the point estimate. Every baseline other than SAM differs significantly: p≤0.005p≤ 0.005 for the convolutional, Transformer and Mamba architectures, and 0.013, 0.034 and 0.048 for CAT-Seg, SED and EviVLM. SAM is the one method from which the proposed model is not distinguished (p = 0.58). We therefore retrained it and the global-prompt configuration three times with different seeds. IoU was 0.3460 ± 0.0049 against 0.3219 ± 0.0052 and Dice 0.5141 ± 0.0054 against 0.4871 ± 0.0065; the variation across seeds is about five times smaller than the gap between them, and the ordering held every time. The remaining configurations were trained once, and Table I reports that run. The direction of every comparison favours VL-AcneSeg and the ordering is preserved across all three datasets, although with 16 test patients the intervals around the smaller differences remain wide; these are reported in Table S4 of the Supplementary Material. On external datasets, VL-AcneSeg maintains consistent performance across both controlled and real-world settings. Most baseline models show notable degradation, with Transformer-based architectures being particularly vulnerable to distribution shifts compared to CNN-based models, likely due to their weaker inductive biases [66]. The gap becomes more pronounced in the real-world smartphone setting, where CAT-Seg and SED degrade substantially and fall below several unimodal baselines, indicating that general-purpose VLM architectures do not readily transfer to domain-shifted clinical environments. VL-AcneSeg, by contrast, retains stable segmentation under unpredictable lighting and varying facial angles. IV-E Ablation Study We evaluated the contributions of VL-AcneSeg’s modules and hyperparameters through three studies: (1) architectural components, (2) image patch resolutions, and (3) text prompt strategies. Results on the external datasets are included in Table I, and the corresponding patch-resolution comparison is given in Table S3 of the Supplementary Material. IV-E1 Effectiveness of Architectural Components TABLE I: Ablation study on the model components of VL-AcneSeg. Confidence intervals and p values are obtained by paired cluster bootstrap against the Full Model, at the patient level for the internal dataset and at the image level for the external datasets. Specificity exceeds 0.999 everywhere and is omitted. FP lesions/img counts predicted lesion components matching no reference lesion. Best values are in bold and second-best are underlined. Metrics Full Model w/o cls w/o v-v w/o region all-layer token attention mask loss v-v Internal IoU 0.3602 [0.309, 0.399] 0.3563 [0.294, 0.398] 0.3420 [0.290, 0.380] 0.3510 [0.290, 0.394] 0.2832 [0.215, 0.336] Dice 0.5296 [0.472, 0.570] 0.5254 [0.458, 0.568] 0.5097 [0.451, 0.552] 0.5196 [0.454, 0.565] 0.4414 [0.357, 0.505] Precision 0.5202 0.5149 0.4936 0.4935 0.3700 Recall 0.5393 0.5363 0.5268 0.5486 0.5449 p-value — 0.554 0.057 0.526 <<0.001 TP lesions/img 4.29 3.49 3.52 3.55 4.38 FP lesions/img 2.48 1.90 2.27 2.09 3.57 External (Controlled) IoU 0.4232 [0.376, 0.470] 0.4064 [0.364, 0.452] 0.4020 [0.359, 0.449] 0.4097 [0.366, 0.455] 0.3314 [0.294, 0.374] Dice 0.5948 [0.546, 0.640] 0.5780 [0.534, 0.622] 0.5735 [0.528, 0.620] 0.5813 [0.536, 0.625] 0.4979 [0.454, 0.544] Precision 0.6110 0.6107 0.5879 0.5947 0.4588 Recall 0.5794 0.5486 0.5597 0.5684 0.5442 p-value — 0.119 0.114 0.332 <<0.001 TP lesions/img 1.64 1.57 1.27 1.07 1.68 FP lesions/img 0.60 0.47 0.67 0.45 0.78 External (Real-world) IoU 0.3044 [0.246, 0.351] 0.2873 [0.190, 0.365] 0.2819 [0.218, 0.343] 0.2905 [0.216, 0.352] 0.1627 [0.114, 0.217] Dice 0.4667 [0.395, 0.520] 0.4464 [0.320, 0.535] 0.4398 [0.358, 0.511] 0.4502 [0.355, 0.520] 0.2798 [0.204, 0.356] Precision 0.4882 0.4571 0.3785 0.4269 0.1965 Recall 0.4471 0.4361 0.5248 0.4763 0.4860 p-value — 0.573 0.346 0.520 <<0.001 TP lesions/img 1.50 1.34 0.95 0.83 1.55 FP lesions/img 0.82 0.77 0.56 0.48 0.93 As shown in Table I, the Full Model attains the highest IoU and Dice on all three datasets. Removing any single component lowers performance consistently but by a margin that the present test sets do not resolve individually, indicating that the CLS token, V-V attention and the region mask loss act in a complementary manner rather than through one dominant factor. The placement of V-V attention, by contrast, is clearly resolved: applying it across all layers, as in the original formulation, degrades performance on every dataset (p<0.01p<0.01) and raises the number of false positive lesions per image from 2.48 to 3.57 internally and from 0.82 to 0.93 in the real-world setting, consistent with the view that repeated value–value mixing erases the fine spatial detail on which small lesions depend. IV-E2 Impact of Image Patch Resolution TABLE IV: Ablation study on different image patch resolutions for CLIP-based processing in VL-AcneSeg. Best values are in bold and second-best are underlined. Metrics 4×34× 3 patches 3×23× 2 patches 2×12× 1 patches IoU 0.3602 0.3194 0.2700 Dice 0.5296 0.4841 0.4253 Precision 0.5202 0.4308 0.4070 Recall 0.5393 0.5524 0.4452 TP lesions/img 4.29 3.67 3.14 FP lesions/img 2.48 2.72 1.71 We tested tiling configurations of 4×34× 3, 3×23× 2, and 2×12× 1 patches. As summarized in Table IV, while the 3×23× 2 configuration yields the highest Recall, it suffers from low Precision due to its inability to distinguish acne from scars. In contrast, 4×34× 3 patches provide the most balanced results. Localization maps for the three configurations are shown in Fig. S1 of the Supplementary Material, where finer granularity yields the detailed spatial priors needed to discriminate true lesions from confounding skin features. Furthermore, qualitative evidence of precise localization in unconstrained smartphone environments is provided in Fig. S2 of the Supplementary Material. IV-E3 Role of the Text Prompt TABLE V: Where the performance originates. Each row removes one element of the vision–language pipeline. Unseen synonyms and Shuffled text prompt keep the CLIP text encoder but substitute synonyms absent from training or pair each image with the prompts of another image, testing whether the prompt is read as a reference to a particular area. Learnable embedding and One-hot region encoding replace the text embedding with a non-linguistic region code, separating language from spatial conditioning. CLIP vision encoder only discards the text branch altogether, and w/o CLIP removes the CLIP encoders as well, so that the interval between the two isolates the contribution of the pretrained visual representation. The last four settings were retrained. Prompt representation IoU Dice Precision Recall Internal Regional text prompt 0.3602 [0.309, 0.399] 0.5296 [0.472, 0.570] 0.5202 0.5393 Unseen synonyms 0.3422 [0.291, 0.392] 0.5100 [0.451, 0.563] 0.5272 0.4938 Shuffled text prompt 0.3039 [0.251, 0.356] 0.4661 [0.402, 0.525] 0.5425 0.4085 Learnable embedding 0.3565 [0.301, 0.399] 0.5256 [0.463, 0.570] 0.4654 0.6037 One-hot region encoding 0.3111 [0.260, 0.357] 0.4745 [0.413, 0.526] 0.4313 0.5274 CLIP vision encoder only 0.2358 [0.184, 0.270] 0.3816 [0.311, 0.426] 0.3642 0.4007 w/o CLIP (Swin only) 0.1694 [0.117, 0.219] 0.2897 [0.210, 0.359] 0.2047 0.4957 External (Controlled) Regional text prompt 0.4232 [0.376, 0.470] 0.5948 [0.546, 0.640] 0.6110 0.5794 Unseen synonyms 0.3724 [0.337, 0.414] 0.5427 [0.504, 0.585] 0.6476 0.4671 Shuffled text prompt 0.2170 [0.171, 0.267] 0.3567 [0.292, 0.421] 0.5597 0.2617 Learnable embedding 0.4292 [0.393, 0.474] 0.6006 [0.565, 0.643] 0.6246 0.5784 One-hot region encoding 0.3732 [0.335, 0.417] 0.5435 [0.502, 0.588] 0.4885 0.6125 CLIP vision encoder only 0.2354 [0.205, 0.272] 0.3811 [0.340, 0.427] 0.3638 0.4002 w/o CLIP (Swin only) 0.2150 [0.181, 0.251] 0.3539 [0.307, 0.401] 0.2725 0.5049 External (Real-world) Regional text prompt 0.3044 [0.248, 0.348] 0.4667 [0.397, 0.517] 0.4882 0.4471 Unseen synonyms 0.2708 [0.199, 0.334] 0.4262 [0.332, 0.500] 0.4509 0.4041 Shuffled text prompt 0.2435 [0.175, 0.290] 0.3916 [0.298, 0.449] 0.4685 0.3364 Learnable embedding 0.2900 [0.227, 0.347] 0.4496 [0.369, 0.515] 0.5289 0.3910 One-hot region encoding 0.2734 [0.212, 0.329] 0.4294 [0.350, 0.495] 0.3864 0.4831 CLIP vision encoder only 0.1470 [0.107, 0.188] 0.2563 [0.194, 0.316] 0.2447 0.2692 w/o CLIP (Swin only) 0.0180 [0.008, 0.035] 0.0353 [0.016, 0.067] 0.0186 0.3475 To test whether the spatial prior rests on the semantics of the anatomical terms or on memorised region identifiers, we prepared two clinical synonyms for each of the fifteen regions and at inference replaced every prompt by one of the two at random, without retraining or adapting the model (Table S1 of the Supplementary Material). Accuracy is largely preserved: internal IoU moves from 0.3602 to 0.3422, with precision rising slightly and recall falling, a shift towards more conservative predictions rather than a loss of spatial grounding. Randomly pairing each image with the prompts belonging to another image is by contrast costly. On the controlled set IoU falls from 0.4232 to 0.2170, close to the level obtained without CLIP at all; internally the drop is more moderate, from 0.3602 to 0.3039. The prompt is therefore followed as a reference to a particular area rather than treated as an undifferentiated conditioning signal, although how much the correct reference is worth varies between the two datasets. Alternative prompt contents, adding a severity level or a lesion count to the region name, are compared in Table S2. Read from the bottom, Table V separates where the performance originates. Removing CLIP altogether leaves internal IoU at 0.1694 and real-world IoU at 0.0180, precision collapsing while recall is preserved: without the pretrained visual representation the model no longer distinguishes active lesions from post-inflammatory hyperpigmentation, pores and scars. Restoring the CLIP vision encoder alone recovers 0.2358, and adding region conditioning a further 0.3602, so both contribute substantially, while the visual representation is what sustains performance under domain shift. Among the conditioning schemes, a fixed one-hot index reaches only 0.3111 whereas a learnable per-region vector matches the text prompts (0.3565 against 0.3602): what matters is not that a region is named but that its code can be aligned with the patch features, which an orthogonal index cannot be. Either source yields one vector per region that enters the same cosine-similarity localization, so with a fixed set of fifteen areas the two are mechanistically equivalent and differ only in that the text encoder also resolves terms outside that vocabulary—a distinction that becomes practical only where the vocabulary is open or prompts arrive as free-form clinical language. (a) GT (b) Forehead (c) Right cheek (d) Right temple (e) Frontal region (f) Malar area (g) Lateral forehead Fig. 5: Anatomical region discrimination through text-guided spatial priors. Top row: region names used during training. Bottom row: anatomical terms never seen during training, paired column-wise with the region above. Fig. 5 shows how prompts steer spatial attention. For distinct areas such as the forehead (b) and cheek (c) the response is regionally dominant, whereas in the narrow temple region (d) the model falls back on the broader concept of an acne lesion. The bottom row probes terms absent from training: single-word clinical synonyms transfer, with “frontal region” (e) concentrating over the forehead and “malar area” (f) isolating the mid-face while suppressing the forehead response seen in (b) and (e). The compositional expression “lateral forehead” (g) instead yields a diffuse map, indicating that a spatial modifier is not fully composed with the anatomical noun. Fig. 6: Segmentation accuracy under prompt perturbation on the internal test set. The 58 test images carry 165 lesion-bearing (positive) region prompts and 705 lesion-free (negative) regions. Each pool was shuffled once with a fixed seed, and the perturbation level denotes the fraction of that pool that is withheld (FN) or added (FP); the perturbed sets are nested. When every prompt of an image is withheld, it falls back to the single global prompt, marked by the dashed line. GT Original Result Adversarial Result GT Ori.: “acne on lip; chin” (actual) Adv.: “acne on lip; left cheek” (remove chin, add left cheek) GT Ori.: “acne on left cheek; chin” (actual) Adv.: “acne on forehead; chin” (remove left cheek, add forehead) Fig. 7: Model response under adversarial prompting. Prompts for lesion-bearing regions are omitted and misleading prompts are issued for lesion-free regions. We further quantified the response to imperfect prompts (Fig. 6), withholding prompts for regions that do contain lesions and issuing prompts for regions that do not. The two behave asymmetrically. Withholding degrades accuracy steadily, from 0.3602 to 0.3100 IoU at 40% perturbation, and falls below the global-prompt baseline of 0.3407 at roughly 15%. Over-prompting is far better tolerated, declining only to 0.3438 at the same level and remaining above that baseline throughout the range examined. The asymmetry itself is informative: that omission costs more than over-prompting means the model attends less to regions it has not been told about, which is the behaviour the mechanism is intended to produce. Where the shuffled-prompt condition shows that the model follows the region it is given, the omission curve shows that it looks less closely at regions it is not given. The two error types are not symmetric in the pipeline either: a spurious region enters the localization stage as a candidate and can still be rejected during refinement, whereas a region that never enters it cannot be recovered downstream. Two adversarial cases in Fig. 7 illustrate both sides: lesions are still localized where prompts are withheld, since visual evidence remains, and under misleading prompts candidates appear only where textual guidance and visual evidence coincide. In practical terms a prompt source must therefore achieve high recall over lesion-bearing regions, while spurious regions carry comparatively little cost. IV-F Failure Case Analysis (A) (B) GT Pred GT Pred Blue: PIH/scars/pores (FP) Orange: missed lesions (FN) Blue: PIH cluster (FP) Orange: missed papules (FN) Fig. 8: Failure case analysis of VL-AcneSeg. Red regions denote acne masks. Blue boxes mark false positives, predominantly PIH, scars and prominent pores; orange boxes mark false negatives, typically small or low-contrast lesions. The residual errors follow a consistent pattern (Fig. 8). In (A) the cheek and chin carry active lesions, post-inflammatory hyperpigmentation and faded scars together, and the model both marks the residual features as lesions (blue boxes) and misses small papules embedded in densely pigmented skin (orange boxes). In (B) small inflammatory lesions are interspersed with widespread red marks, and the same two errors recur: subtle active lesions are overlooked and isolated non-acne marks elsewhere on the face are labelled as acne. Both modes arise from the same ambiguity, since active lesions and their residue share colour, shape and texture in clinical photographs. Confusions of this kind, rather than gross localization failures, account for most of the remaining error. IV-G Clinical Implementation TABLE VI: Correlation between the predicted lesion-to-face area ratio and IGA severity on the internal test set (22 visits from 16 patients). p values compare each coefficient with that of the expert annotations using a test for dependent correlations sharing one variable; confidence intervals are obtained by patient-level cluster bootstrap. Source of area ratio Pearson Spearman r [95% CI] p ρ [95% CI] p Expert annotation (GT) 0.658 [0.441, 0.807] — 0.710 [0.541, 0.793] — VL-AcneSeg (region) 0.719 [0.477, 0.848] 0.430 0.684 [0.485, 0.799] 0.719 VL-AcneSeg (global) 0.695 [0.442, 0.833] 0.621 0.695 [0.465, 0.843] 0.868 EviVLM 0.640 [0.361, 0.796] 0.839 0.660 [0.392, 0.809] 0.598 CAT-Seg 0.650 [0.407, 0.797] 0.924 0.685 [0.479, 0.812] 0.737 SED 0.630 [0.279, 0.808] 0.770 0.645 [0.372, 0.827] 0.526 SAM 0.650 [0.320, 0.820] 0.932 0.572 [0.299, 0.759] 0.187 nnU-Net 0.579 [0.179, 0.793] 0.435 0.530 [0.203, 0.776] 0.125 Swin-UMamba† 0.587 [0.155, 0.801] 0.501 0.510 [0.151, 0.766] 0.116 PSPNet 0.690 [0.373, 0.863] 0.738 0.571 [0.323, 0.755] 0.218 U-Net 0.630 [0.336, 0.795] 0.722 0.628 [0.376, 0.809] 0.372 U-Net++ 0.630 [0.291, 0.844] 0.798 0.549 [0.200, 0.790] 0.254 TransUNet 0.610 [0.293, 0.782] 0.623 0.610 [0.383, 0.771] 0.244 DeepLabV3++ 0.600 [0.221, 0.808] 0.589 0.496 [0.189, 0.728] 0.085 CE-Net 0.590 [0.229, 0.779] 0.512 0.553 [0.240, 0.778] 0.225 SegFormer 0.540 [0.182, 0.753] 0.286 0.527 [0.179, 0.780] 0.219 Swin-UNet 0.400 [0.003, 0.617] 0.057 0.462 [0.110, 0.722] 0.102 To assess clinical relevance beyond pixel-level metrics, we correlated the lesion-to-face area ratio of each model with Investigator’s Global Assessment (IGA) severity on the internal test set, pooling the views of a visit so that each of the 22 visits contributes one observation (Table VI). The area ratio derived from VL-AcneSeg tracks IGA at the level of the expert annotations themselves (Pearson 0.719 versus 0.658; Spearman 0.684 versus 0.710); the direction of this comparison reverses between the two coefficients and neither difference is significant (p = 0.430 and 0.719), so the predicted and reference ratios cannot be distinguished in their agreement with IGA on this sample. Comparing methods, no coefficient differs significantly from that of the reference annotations at this sample size. On Spearman, however, the six highest values belong to the reference annotations and to the five vision-language configurations, with the convolutional and Transformer architectures below them even where their pixel-level accuracy is comparable. Within this group the global prompt, our primary setting, correlates as well as the region-level one, as expected for a measure defined over the whole face; region-level prompts instead yield a separate estimate for each facial area. A high coefficient alone, however, does not imply an accurate area estimate: U-Net reaches a moderate value because its inflated areas still scale with lesion burden, whereas VL-AcneSeg reaches a comparable coefficient with precision and recall balanced, so its estimates track severity without a systematic offset. These results support the premise that lesion area measured from a segmentation can serve as a quantitative severity measure. This is also where an area-based measure differs from counting. A count treats every lesion as one unit irrespective of extent, so it saturates precisely where severity is greatest: confluent papules that have merged into a plaque contribute the same value as a few discrete ones, and a lesion that shrinks under treatment without resolving contributes the same value throughout. An area ratio varies continuously with both, which is the property area-based assessment relies on. A Dice score near 0.53 is modest by the standards of organ segmentation, but the quantity entering the severity measure is the aggregate lesion area rather than the boundary of each lesion, and a Dice score at this level is sufficient for that area to track IGA as closely as the reference annotations do. The difficulty of the task is also reflected in human performance: in a reader study on this cohort, eight dermatologists and residents reached a median lesion-detection F1 of 0.31 for inflammatory lesions from images alone [1]. Applications requiring precise per-lesion delineation would need higher accuracy than we report here. IV-H Deployment Considerations The framework runs in two settings. Under the global prompt no region information is required, which is why we report it as the primary setting. Under region-level prompting a clinician or the patient indicates which facial areas carry lesions—a coarse judgement, not the location of individual lesions—which raises accuracy and additionally gives a separate estimate for each area. The perturbation analysis indicates how accurate such prompts need to be. Withholding prompts for lesion-bearing regions falls below the global-prompt baseline at roughly 15% omission, whereas issuing prompts for lesion-free regions stays above it throughout the range examined. In practice, therefore, over-inclusive prompting is safe, whereas leaving affected regions unmarked is what degrades the result. TABLE VII: Computational cost, measured on a single NVIDIA A6000 at the input resolution used throughout. For VL-AcneSeg, all regions denotes prompts issued for all fifteen facial regions; the tiled CLIP encoding is shared across prompts. Params (M) FLOPs (G) Time (ms) VL-AcneSeg (global prompt) 432 1,417 858 VL-AcneSeg (all regions) 432 1,676 1,536 SAM (ViT-L) 312 1,312 594 EviVLM 130 567 176 Swin-UMamba† 27 25 64 U-Net 17 428 74 VL-AcneSeg is slower than most baselines (Table VII), which follows from the tiled high-resolution encoding on which the accuracy also depends. In absolute terms, however, one image takes 858 ms under the global prompt and at most 1.54 s with all fifteen regions, so cost is not a constraint at either setting. The tiled encoding accounts for roughly 83% of the computation but is shared across prompts, so each additional region prompt adds only 18.5 GFLOPs, or about 48 ms. IV-I Limitations and Future Directions Several limitations should be acknowledged. First, region-level prompting presupposes knowledge of which facial areas contain lesions; we therefore report the global prompt, which requires none, as our primary setting. With a closed set of fifteen regions the pretrained text prior is also not fully exploited: unseen synonyms are resolved (Table V), but that capacity makes no difference to a deployment in which the vocabulary is fixed in advance. Second, all internal and external data represent Korean populations, so the evidence does not extend to other ethnicities, skin tones or imaging devices, and claims of worldwide applicability would be unwarranted. This constraint is not particular to our work: the two public acne datasets are likewise drawn from a single ethnic group, and no published acne algorithm has yet undergone prospective clinical validation [18]. Fitzpatrick phototype was not recorded in the clinical dataset and is not distributed with the external collection, so performance could not be stratified by skin type; nor could it be stratified by severity, since the 22 graded visits leave strata too small for stable estimates. The same sample size widens the intervals around the smaller between-method differences, so their magnitude is estimated less precisely than their direction. Constructing pixel-level acne datasets requires substantial dermatological effort, and evaluation on multi-ethnic, multi-site data remains the most important direction for establishing broader applicability. Third, Dice and IoU remain moderate relative to other medical segmentation tasks, and the residual error is dominated by the ambiguity between active lesions and their own residue described in Section IV-F. Fourth, the correlation with IGA does not by itself show that the method is ready for clinical use; establishing that would require a prospective study against expert assessment. Finally, inter-annotator agreement was not quantified. The annotation followed an adjudication design in which a single board-certified dermatologist fixed the final label set, so agreement between readers is not the quantity that defines the reference standard here; the underlying lesion markings were moreover produced for an earlier study [1], and the drafts preceding adjudication were not retained as separate outputs. Quantifying observer variability on this task would help establish the practical ceiling for pixel-level acne segmentation and remains an objective for future work. Several of these limitations suggest concrete next steps. The confusion is between an active lesion and its own residue, which are hard to separate in the spatial domain because they share colour, shape and texture; features that expose properties the RGB image does not make explicit, such as a frequency-domain representation, may provide evidence that spatial appearance alone does not. Our annotations distinguish five lesion types, and we plan to extend the model to a multi-class objective, which would also allow papules, pustules and nodules to be segmented jointly. Prompts that vary in what they describe—lesion type, extent or a clinician’s own phrasing—would put the text encoder to work, and are a natural direction once multi-class supervision is in place. More broadly, the reformulation itself is not specific to acne: wherever lesions are small and numerous but distributed over an anatomy that is stable and nameable, text can supply the same kind of spatial constraint. V Conclusion We introduced VL-AcneSeg, a multimodal framework for inflammatory acne lesion segmentation. Where open-vocabulary methods use text to specify which class to segment, we use it to specify where the target may appear, so that the text channel supplies a spatial rather than a categorical constraint—a formulation suited to a task with a single lesion class distributed over a nameable anatomy. Because region-level prompts presuppose knowledge of lesion locations, we report a single global prompt as the primary setting and treat region-level prompting as an oracle upper bound. Across internal and external datasets VL-AcneSeg attains the highest Dice and IoU among the compared methods, including vision-language baselines that are themselves given region-level prompts, and retains this under domain shift where general-purpose architectures transfer less well. Controlled substitutions indicate that the prompt is read as a reference to a particular area rather than as an undifferentiated signal, although on a closed region vocabulary a learned region vector is mechanistically equivalent to a text embedding. The lesion areas the model produces track clinical IGA severity as closely as the expert annotations do, supporting area-based severity assessment as an alternative to lesion counting. We believe VL-AcneSeg provides a practical foundation for automated acne management. Conflict of Interests The authors declare that they have no conflict of interest. References [1] D. H. Kim, S. Sun, S. I. Cho, H. Kong, J. W. Lee, J. H. Lee, and D. H. Suh (2023) Automated facial acne lesion detecting and counting algorithm for acne severity evaluation and its utility in assisting dermatologists. American Journal of Clinical Dermatology 24 (4), p. 649–659. Cited by: §I, §I-A, §IV-G, §IV-I, VL-AcneSeg: A Vision-Language Framework for Region-Aware Acne Lesion Segmentation. [2] A. Grada, S. Muddasani, A. B. Fleischer Jr, S. R. Feldman, and G. M. Peck (2022) Trends in office visits for the five most common skin diseases in the united states. The Journal of Clinical and Aesthetic Dermatology 15 (5), p. E82. Cited by: §I. [3] J. K. Tan and K. Bhate (2015) A global perspective on the epidemiology of acne. British Journal of Dermatology 172 (S1), p. 3–12. Cited by: §I. [4] M. Law, A. Chuh, A. Lee, and N. Molinari (2010) Acne prevalence and beyond: acne disability and its predictive factors among chinese late adolescents in hong kong. Clinical and experimental dermatology 35 (1), p. 16–21. Cited by: §I. [5] K. Bhate and H. Williams (2013) Epidemiology of acne vulgaris. British Journal of Dermatology 168 (3), p. 474–485. Cited by: §I. [6] S. Y. Park, M. Y. Park, D. H. Suh, H. H. Kwon, S. Min, S. J. Lee, W. J. Lee, M. W. Lee, H. H. Ahn, H. Kang, et al. (2016) Cross-sectional survey of awareness and behavioral pattern regarding acne and acne scar based on smartphone application. International Journal of Dermatology 55 (6), p. 645–652. Cited by: §I. [7] S. I. Cho, J. H. Yang, and D. H. Suh (2021) Analysis of trends and status of physician-based evaluation methods in acne vulgaris from 2000 to 2019. The Journal of Dermatology 48 (1), p. 42–48. Cited by: §I. [8] C. Beylot, M. Chivot, M. Faure, H. Pawin, F. Poli, J. Revuz, N. Auffret, D. Moyse, B. Dréno, and G. E. A. (GEA) (2010) Inter-observer agreement on acne severity based on facial photographs. Journal of the European Academy of Dermatology and Venereology 24 (2), p. 196–198. Cited by: §I. [9] T. Agnew, G. Furber, M. Leach, and L. Segal (2016) A comprehensive critique and review of published measures of acne severity. The Journal of clinical and aesthetic dermatology 9 (7), p. 40. Cited by: §I. [10] I. H. Bae, J. H. Kwak, C. H. Na, M. S. Kim, B. S. Shin, and H. Choi (2024) A comprehensive review of the acne grading scale in 2023. Annals of Dermatology 36 (2), p. 65. Cited by: §I. [11] J. K. Tan, K. Fung, and L. Bulger (2006) Reliability of dermatologists in acne lesion counts and global assessments. Journal of cutaneous medicine and surgery 10 (4), p. 160–165. Cited by: §I. [12] Q. T. Huynh, P. H. Nguyen, H. X. Le, L. T. Ngo, N. Trinh, M. T. Tran, H. T. Nguyen, N. T. Vu, A. T. Nguyen, K. Suda, et al. (2022) Automatic acne object detection and acne severity grading using smartphone images and artificial intelligence. Diagnostics 12 (8), p. 1879. Cited by: §I. [13] J. Wang, C. Wang, Z. Wang, A. H. Hounye, Z. Li, M. Kong, M. Hou, J. Zhang, and M. Qi (2023) A novel automatic acne detection and severity quantification scheme using deep learning. Biomedical Signal Processing and Control 84, p. 104803. Cited by: §I. [14] D. Thiboutot, A. Longenecker, D. Canfield, and S. V. Patwardhan (2021) Parametric acne severity (pas) score and lesion counts from multi-modality facial image analysis correlates strongly with investigator assessment.. Journal of drugs in dermatology: JDD 20 (6), p. 642–647. Cited by: §I. [15] M. H. Gold, A. Bhatia, A. Kaur, M. Doucette, and A. Kothare (2022) Picture-based acne lesion counts: a validation study to assess accuracy and reliability of acne lesion counts via photography. Journal of Cosmetic Dermatology 21 (12), p. 6965–6975. Cited by: §I. [16] L. Gazeau, H. Nguyen, Z. Nguyen, M. Lebedeva, T. Nguyen, T. To, J. Le Digabel, J. Filiol, G. Josse, C. Perlis, et al. (2024) AcneAI: a new acne severity assessment method using digital images and deep learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 68–78. Cited by: §I, §I, §I-A. [17] M. Moncho-Santonja, S. Aparisi-Navarro, B. Defez, and G. Peris-Fajarnés (2023) Segmentation of acne vulgaris images techniques: a comparative and technical study. Applied Sciences 13 (10), p. 6157. Cited by: §I. [18] D. O. Traini, G. Palmisano, C. Guerriero, and K. Peris (2025) Artificial intelligence in the assessment and grading of acne vulgaris: a systematic review. Journal of Personalized Medicine 15 (6), p. 238. Cited by: §I, §I-A, §IV-I. [19] N. Yadav, S. M. Alfayeed, A. Khamparia, B. Pandey, D. N. Thanh, and S. Pande (2022) HSV model-based segmentation driven facial acne detection using deep learning. Expert Systems 39 (3), p. e12760. Cited by: §I, §I-A. [20] M. S. Junayed, M. B. Islam, and N. Anjum (2022) A transformer-based versatile network for acne vulgaris segmentation. In 2022 Innovations in Intelligent Systems and Applications Conference (ASYU), p. 1–6. Cited by: §I, §I-A. [21] S. Kim, H. Yoon, and J. Lee (2024) Semi-supervised facial acne segmentation using bidirectional copy–paste. Diagnostics 14 (10), p. 1040. Cited by: §I, §I-A. [22] S. Kim, C. Lee, G. Jung, H. Yoon, J. Lee, and S. Yoo (2023) Facial acne segmentation based on deep learning with center point loss. In 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS), p. 678–683. Cited by: §I, §I-A. [23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §I-B, §I-D. [24] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, p. 4904–4916. Cited by: §I-B. [25] K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022) Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 16816–16825. Cited by: §I-B. [26] Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, et al. (2022) Regionclip: region-based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 16793–16803. Cited by: §I-B. [27] J. Jeong, Y. Zou, T. Kim, D. Zhang, A. Ravichandran, and O. Dabeer (2023) Winclip: zero-/few-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19606–19616. Cited by: §I-B. [28] H. Ding, C. Liu, S. Wang, and X. Jiang (2022) VLT: vision-language transformer and query generation for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (6), p. 7900–7916. Cited by: §I-B. [29] Z. Yang, J. Wang, Y. Tang, K. Chen, H. Zhao, and P. H. Torr (2022) Lavt: language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 18155–18165. Cited by: §I-B. [30] Z. Wang, Y. Lu, Q. Li, X. Tao, Y. Guo, M. Gong, and T. Liu (2022) Cris: clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11686–11695. Cited by: §I-B. [31] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu (2023) Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 7061–7070. Cited by: §I-B. [32] J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello (2023) Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2955–2966. Cited by: §I-B. [33] C. Zhou, C. C. Loy, and B. Dai (2022) Extract free dense labels from clip. In European conference on computer vision, p. 696–712. Cited by: §I-B. [34] S. Cho, H. Shin, S. Hong, A. Arnab, P. H. Seo, and S. Kim (2024) Cat-seg: cost aggregation for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4113–4123. Cited by: §I-B, TABLE I. [35] B. Xie, J. Cao, J. Xie, F. S. Khan, and Y. Pang (2024) Sed: a simple encoder-decoder for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 3426–3436. Cited by: §I-B, TABLE I. [36] M. Lee, S. Cho, J. Lee, S. Yang, H. Choi, I. Kim, and S. Lee (2025) Effective sam combination for open-vocabulary semantic segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 26081–26090. Cited by: §I-B. [37] Q. Tang, L. Fan, M. Pagnucco, and Y. Song (2025) Prototype-based image prompting for weakly supervised histopathological image segmentation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 30271–30280. Cited by: §I-B. [38] Z. Li, Y. Li, Q. Li, P. Wang, D. Guo, L. Lu, D. Jin, Y. Zhang, and Q. Hong (2023) Lvit: language meets vision transformer in medical image segmentation. IEEE transactions on medical imaging 43 (1), p. 96–107. Cited by: §I-C. [39] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §I-C. [40] T. Koleilat, H. Asgariandehkordi, H. Rivaz, and Y. Xiao (2024) Medclip-sam: bridging text and image towards universal medical image segmentation. In International conference on medical image computing and computer-assisted intervention, p. 643–653. Cited by: §I-C. [41] L. Shen, F. Shang, X. Huang, Y. Yang, H. Huang, and S. Xiang (2024) SegICL: a multimodal in-context learning framework for enhanced segmentation in medical imaging. arXiv preprint arXiv:2403.16578. Cited by: §I-C. [42] W. Zhang, Z. Zhang, M. He, and J. Ye (2025) Organ-aware multi-scale medical image segmentation using text prompt engineering. arXiv preprint arXiv:2503.13806. Cited by: §I-C. [43] L. Fang, X. Li, Y. Xu, F. Zhang, and C. Zhang (2025) Driven by textual knowledge: a text-view enhanced knowledge transfer network for lung infection region segmentation. Medical Image Analysis 103, p. 103625. Cited by: §I-C. [44] D. Shan, Z. Li, Y. Li, Q. Li, J. Tian, and Q. Hong (2025) STPNet: scale-aware text prompt network for medical image segmentation. IEEE Transactions on Image Processing 34, p. 3169–3180. Cited by: §I-C. [45] Q. Pan, Z. Li, G. Yang, Q. Yang, and B. Ji (2025) Evivlm: when evidential learning meets vision language model for medical image segmentation. IEEE Transactions on Medical Imaging. Cited by: §I-C, TABLE I. [46] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C. Chang, M. G. Yong, J. Lee, et al. (2019) Mediapipe: a framework for building perception pipelines. arXiv preprint arXiv:1906.08172. Cited by: Fig. 2, Fig. 2, §I-A. [47] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman (2008) LabelMe: a database and web-based tool for image annotation. International journal of computer vision 77 (1), p. 157–173. Cited by: §I-A. [48] Q. Zhou, G. Pang, Y. Tian, S. He, and J. Chen (2024) Anomalyclip: object-agnostic prompt learning for zero-shot anomaly detection. Vol. 2024. Cited by: §I-D. [49] D. Fan, G. Ji, T. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao (2020) Pranet: parallel reverse attention network for polyp segmentation. In International conference on medical image computing and computer-assisted intervention, p. 263–273. Cited by: §I-F. [50] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár (2017) Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, p. 2980–2988. Cited by: §I-F. [51] X. Wu, N. Wen, J. Liang, Y. Lai, D. She, M. Cheng, and J. Yang (2019) Joint acne image grading and counting via label distribution learning. In 2019 IEEE/CVF international conference on computer vision (ICCV), p. 10641–10650. Cited by: §IV-A. [52] I. Loshchilov and F. Hutter (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §IV-C. [53] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou (2021) Training data-efficient image transformers & distillation through attention. In International conference on machine learning, p. 10347–10357. Cited by: §IV-C. [54] E. Alsentzer, J. Murphy, W. Boag, W. Weng, D. Jindi, T. Naumann, and M. McDermott (2019) Publicly available clinical bert embeddings. In Proceedings of the 2nd clinical natural language processing workshop, p. 72–78. Cited by: §IV-C. [55] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, et al. (2023) Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4015–4026. Cited by: TABLE I. [56] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, p. 234–241. Cited by: TABLE I. [57] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia (2017) Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2881–2890. Cited by: TABLE I. [58] L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), p. 801–818. Cited by: TABLE I. [59] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang (2018) Unet++: a nested u-net architecture for medical image segmentation. In International workshop on deep learning in medical image analysis, p. 3–11. Cited by: TABLE I. [60] Z. Gu, J. Cheng, H. Fu, K. Zhou, H. Hao, Y. Zhao, T. Zhang, S. Gao, and J. Liu (2019) Ce-net: context encoder network for 2d medical image segmentation. IEEE transactions on medical imaging 38 (10), p. 2281–2292. Cited by: TABLE I. [61] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo (2021) SegFormer: simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems 34, p. 12077–12090. Cited by: TABLE I. [62] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou (2021) Transunet: transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306. Cited by: TABLE I. [63] H. Cao, Y. Wang, J. Chen, D. Jiang, X. Zhang, Q. Tian, and M. Wang (2022) Swin-unet: unet-like pure transformer for medical image segmentation. In European conference on computer vision, p. 205–218. Cited by: TABLE I. [64] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein (2021) NnU-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18 (2), p. 203–211. Cited by: TABLE I. [65] J. Liu, H. Yang, H. Zhou, L. Yu, Y. Liang, Y. Yu, S. Zhang, H. Zheng, and S. Wang (2025) Swin-umamba†: adapting mamba-based vision foundation models for medical image segmentation. IEEE Transactions on Medical Imaging 44 (10), p. 3898–3908. External Links: Document Cited by: TABLE I. [66] Z. Wang, Y. Bai, Y. Zhou, and C. Xie (2022) Can cnns be more robust than transformers?. arXiv preprint arXiv:2206.03452. Cited by: §IV-D. Supplementary Material S1 Dataset Details S1-A Internal cohort The internal cohort comprises 258 patients (38.4% male; mean age 22.7 ± 5.97 years) who visited the Department of Dermatology, Seoul National University Hospital between February and July 2020. Photographs were acquired with Canon EOS 550D and Nikon D7100 cameras under standardized conditions. Fitzpatrick phototype was not routinely recorded in the clinical dataset; all participants were of Korean ethnicity, a population in which phototypes I and IV predominate. S1-B IGA grading protocol Investigator’s Global Assessment scores were assigned retrospectively at the visit level: three board-certified dermatologists independently reviewed all images acquired at a given visit and graded that visit on the standard five-point scale (0, clear; 1, almost clear; 2, mild; 3, moderate; 4, severe). The final grade was determined by majority voting; when the three graders assigned three different grades, the case was resolved through joint discussion until consensus was reached. Among the 22 visits in the internal test set, 3, 7, 8 and 4 visits were graded as IGA 1, 2, 3 and 4 respectively, and no visit was graded as IGA 0. S1-C External subset construction The AI Hub Korean Skin Condition Measurement Dataset includes 13,936 facial images, 84,688 skin condition measurement records and 125,424 labeled data entries. Because the collection contains many images without visible inflammatory lesions, we first applied an acne detection model trained on the public ACNE04 dataset as a screening step. A deliberately low confidence threshold (0.3) was used so that the screening favoured recall, retaining images with even weak detection responses. From this candidate pool we randomly sampled 100 images captured with digital cameras in a controlled environment and 100 images captured with smartphones under uncontrolled conditions. Each sampled image was then reviewed by the dermatologist who adjudicated the internal set, and those without confirmed inflammatory acne were excluded, yielding the final external test sets of 71 and 56 images. Consistent with the permissive screening threshold, only 71% and 56% of the sampled images were confirmed to contain inflammatory acne. A residual sampling bias nevertheless remains, restricted to images whose lesions the screening model failed to detect even at this low threshold, and this should be considered when interpreting the external validation results. S1-D External imaging conditions and metadata Because the AI Hub collection was compiled for general facial skin analysis rather than from an acne patient population, the two external subsets differ from the internal cohort in lesion burden. They also differ in the type of imaging shift they probe. The controlled subset was acquired with digital cameras under a standardized setup comparable to that of the internal cohort and therefore mainly reflects a shift in population and acquisition site rather than in imaging device. The real-world subset was acquired with smartphone cameras and includes images taken with both front- and rear-facing cameras, introducing additional variability in resolution, field of view and colour processing. The AI Hub distribution does not specify the individual camera models, and it provides neither participant demographics nor Fitzpatrick phototypes; these characteristics therefore cannot be reported for the external subsets. S2 Text Prompt Details TABLE S1: Anatomical synonyms used in the unseen-prompt experiment. Each training region name was replaced at inference by one of the two alternatives, chosen at random; none appears in training. Seen (training) Unseen 1 Unseen 2 forehead supraorbital area frontal region left cheek left malar region left buccal area right cheek right malar region right buccal area chin mentum mental region nose nasal region nasal bridge left temple left pterion area left temporal region right temple right pterion area right temporal region glabella interciliary space intercilium left eyebrow left supraorbital ridge left superciliary arch right eyebrow right supraorbital ridge right superciliary arch left eye left periorbital region left orbital area right eye right periorbital region right orbital area lip labial region vermilion area upper lip upper labial region infranasal area neck cervical region anterior cervical area TABLE S2: Ablation study on different text prompt strategies for VL-AcneSeg in internal data. Best values are in bold and second-best are underlined. Metrics Region Prompt Level Prompt Num Prompt IoU 0.3602 0.3521 0.3594 Dice 0.5296 0.5208 0.5288 Precision 0.5202 0.4889 0.4940 Recall 0.5393 0.5572 0.5688 Table S1 lists the anatomical synonyms used in the unseen-prompt experiment reported in the main paper; none of the alternatives appears during training. Table S2 compares three prompt contents on the internal dataset: Region supplies spatial guidance only, Level adds a qualitative severity descriptor, and Num adds an exact lesion count per region. Adding count descriptors raises recall by widening the search space but lowers precision, reflecting a trade-off between spatial guidance and quantitative prompt engineering. S3 Localization under Different Patch Resolutions (a) GT (b) 4×34× 3 (c) 3×23× 2 (d) 2×12× 1 (e) Global Prompt Fig. S1: Ablation study on patch granularity and regional textual cues. Finer patch division improves localization precision. The model still localizes lesions under a global text prompt alone (e). Localization maps for the three tiling configurations evaluated in Section IV-E of the main paper. Coarser tiling spreads activation over confounding skin features, whereas the finer division keeps the response on the lesions themselves. S4 Qualitative Analysis of Localization in Smartphone Image (a) Ground Truth (b) Full Model (c) Global Prompt Fig. S2: Localization heatmaps in a smartphone environment. (b) The full model identifies lesion-suspected areas with high precision and sharpness. (c) Although the global text prompt configuration exhibits broader localization extending beyond facial boundaries, it still captures the actual lesion sites. Figure S2 shows localization maps for a smartphone image under the two prompting conditions. With region-level prompts the response is concentrated on the lesion sites. With a global prompt the activation is broader and occasionally extends beyond the facial boundary, but the lesion sites are still recovered, so the model remains usable where region information is unavailable. S5 Patch Resolution on External Datasets TABLE S3: Effect of patch tiling resolution on the two external datasets. Component-wise ablations on the external datasets are reported in the main paper. Best values are in bold and second-best are underlined. Ablation Model External Datasets (Evaluation Only) Controlled Setting (1) Real-world Setting (2) IoU Dice Precision Recall TP/img FP/img IoU Dice Precision Recall TP/img FP/img VL-AcneSeg (Full Model) 0.4232 0.5948 0.6110 0.5794 1.64 0.60 0.3044 0.4667 0.4882 0.4471 1.50 0.82 3×23× 2 Patches 0.3812 0.5520 0.5664 0.5383 1.83 0.79 0.2443 0.3927 0.3787 0.4077 1.11 0.66 2×12× 1 Patches 0.3019 0.4638 0.4593 0.4683 1.58 0.77 0.1213 0.2164 0.1354 0.5383 0.89 1.43 Table S3 shows that the gain from finer tiling observed on the internal dataset also holds on both external datasets. S6 Confidence Intervals for Between-Method Differences TABLE S4: Differences in IoU and Dice between VL-AcneSeg and each baseline on the internal test set, with 95% confidence intervals from paired patient-level cluster bootstrap over the 16 test patients. A positive value favours the proposed method. Vision-language baselines receive region-level prompts and are therefore compared against the region-level configuration; the remaining methods take no text input and are compared against the global prompt. Baseline Δ [95% CI] p Δ [95% CI] p Vision-language methods, compared against VL-AcneSeg (region) EviVLM ++0.031 [0.001, 0.069] 0.048 ++0.034 [0.001, 0.080] 0.046 CAT-Seg ++0.069 [0.012, 0.136] 0.013 ++0.079 [0.011, 0.166] 0.015 SED ++0.042 [0.003, 0.097] 0.034 ++0.047 [0.003, 0.112] 0.038 Remaining methods, compared against VL-AcneSeg (global) SAM ++0.019 [−-0.037, 0.078] 0.580 ++0.021 [−-0.041, 0.091] 0.592 Swin-UMamba† ++0.053 [0.025, 0.076] 0.005 ++0.061 [0.018, 0.085] 0.005 TransUNet ++0.078 [0.021, 0.144] 0.005 ++0.092 [0.023, 0.182] 0.005 CE-Net ++0.119 [0.065, 0.170] <<0.001 ++0.146 [0.076, 0.211] <<0.001 Swin-UNet ++0.132 [0.091, 0.170] <<0.001 ++0.163 [0.110, 0.217] <<0.001 SegFormer ++0.136 [0.090, 0.177] <<0.001 ++0.168 [0.109, 0.228] <<0.001 nnU-Net ++0.150 [0.110, 0.185] <<0.001 ++0.188 [0.136, 0.234] <<0.001 PSPNet ++0.161 [0.132, 0.185] <<0.001 ++0.203 [0.172, 0.230] <<0.001 DeepLabV3++ ++0.160 [0.131, 0.194] <<0.001 ++0.202 [0.164, 0.253] <<0.001 U-Net++ ++0.183 [0.121, 0.236] <<0.001 ++0.236 [0.158, 0.311] <<0.001 U-Net ++0.199 [0.165, 0.229] <<0.001 ++0.260 [0.225, 0.292] <<0.001 The main paper reports confidence intervals for each method separately, which do not indicate whether two methods differ, since the same patients contribute to both estimates. Table S4 therefore reports the difference itself, resampled in pairs so that the correlation between the two methods is preserved. Every difference is significant at the 0.05 level except that of SAM, whose interval is the only one to include zero; as reported in the main paper, that comparison was examined further by retraining both configurations with three seeds. The width of the intervals reflects the size of the test set: the differences against the conventional architectures are estimated to within a few hundredths of an IoU point, whereas those against the vision-language baselines, where the margin is smaller, are correspondingly less precise. S7 Additional Evaluation Metrics This section reports two metrics that address aspects the main comparison leaves open. The first is small-lesion performance at the lesion level. Acne lesions vary widely in size, and the pixel-level metrics of the main paper are dominated by the larger ones simply because they contribute more pixels; a method could therefore score well while missing most of the small lesions. We isolate the smallest quartile and count lesions rather than pixels, so that each lesion carries equal weight. The second is the area under the precision–recall curve. All results in the main paper use a binarization threshold of 0.5, fixed for every method without per-model tuning, and AUPRC is computed from the probability maps before that threshold is applied, so it indicates whether the reported ordering depends on that choice. TABLE S5: Additional metrics on the internal test set. Small lesions are those in the lowest quartile of the lesion-area distribution over the whole annotated set, corresponding to an area below 500 px2; recall, precision and F1 are computed over these lesions at the lesion level. AUPRC is the area under the precision–recall curve, computed from the predicted probability maps and therefore independent of the binarization threshold. Rows are ordered by small-lesion F1. Best values are in bold and second-best are underlined. Model Small lesions (≤500≤ 500 px2) AUPRC Recall Precision F1 VL-AcneSeg (region) 0.494 0.451 0.472 0.4138 VL-AcneSeg (global) 0.492 0.432 0.460 0.4068 SAM 0.457 0.432 0.444 0.3400 SED 0.431 0.426 0.428 0.4167 EviVLM 0.453 0.405 0.428 0.4037 CAT-Seg 0.358 0.466 0.405 0.3922 DeepLabV3++ 0.381 0.392 0.386 0.3958 CE-Net 0.420 0.275 0.332 0.1655 Swin-UMamba† 0.554 0.214 0.309 0.3963 PSPNet 0.503 0.208 0.294 0.3261 U-Net++ 0.531 0.194 0.284 0.1582 Swin-UNet 0.333 0.247 0.284 0.1882 TransUNet 0.506 0.182 0.268 0.1745 nnU-Net 0.420 0.169 0.241 0.3249 U-Net 0.840 0.060 0.112 0.3582 SegFormer 0.136 0.083 0.103 0.1840 Small lesions. All quantities in this block are computed at the lesion level, over connected components rather than pixels. Recall alone is misleading here. U-Net recovers 84.0% of the small lesions but only 6.0% of what it marks is one, and the same pattern holds for Swin-UMamba† , U-Net++, TransUNet and PSPNet, all of which reach recall comparable to or above the proposed method at less than half its precision. CAT-Seg sits at the other extreme, with the highest precision of any method and the lowest recall among the vision-language ones. Both configurations of VL-AcneSeg are in the upper range on both quantities, which is why they lead on F1, and the global-prompt configuration exceeds every vision-language baseline even though those baselines receive region-level prompts. Threshold independence. Both configurations of the proposed method fall in the upper range, so the advantage reported in the main paper does not depend on the choice of threshold. The ordering below them differs from the pixel-level one, since AUPRC also reflects how well a model ranks pixels at thresholds it is never evaluated at. The top group is tightly packed, so AUPRC should be read as separating the stronger methods from the weaker ones rather than as distinguishing between them. S8 Comprehensive Qualitative Comparison of Acne Segmentation Figs. S3–S5 extend the qualitative comparison of the main paper to all baselines, on the internal, external controlled and external smartphone datasets. Each figure enlarges a single facial region so that lesions a few pixels across remain visible, and includes both prompting conditions of the proposed method. Fig. S3: Comprehensive qualitative comparison on the internal dataset (Part I). The left panel shows the input with the reference annotation; the green box marks the region enlarged in the remaining panels. Predicted masks are overlaid in red. Fig. S4: Comprehensive qualitative comparison on the external controlled dataset (Part I). Fig. S5: Comprehensive qualitative comparison on the external smartphone dataset (Part I).

Instant research discovery