Paper deep dive

CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

Seungyong Lee, Hyun Jun Jang, Sangoh Kim, Sungjoon Park

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 87

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 7/13/2026, 6:18:37 AM

Summary

The paper introduces CtrlVTON, a controllable virtual try-on framework that reformulates VTO as an image editing problem to enable precise user control over garment size, style, and spatial placement. It addresses instance-level segmentation challenges through VIP-SAM, which solves the novel Visual-Instance-Prompt Segmentation (VIP-Seg) task. The framework achieves state-of-the-art results on fashion benchmarks and introduces VITON-HD-edit, a new benchmark and data pipeline for instance-level segmentation and mask-controllable try-on.

Entities (10)

CtrlVTON · framework · 95%VIP-SAM · model · 95%Diffusion Transformer · architecture · 90%VIP-Seg · task · 90%Virtual Try-On · task · 90%VITON-HD-edit · dataset · 90%COCO-20 i · dataset · 85%Image Editing · task · 85%LoRA · technique · 85%PASCAL-5 i · dataset · 85%

Relation Signals (9)

VIP-SAM → solves → VIP-Seg

confidence 95% · We define and solve Visual-Instance-Prompt Segmentation via VIP-SAM

CtrlVTON → uses → VIP-SAM

confidence 95% · VIP-SAM is the prerequisite that unlocks the controllable VTO framework

Authors → created → VITON-HD-edit

confidence 90% · We publicly release VITON-HD-edit, a benchmark built from this pipeline

CtrlVTON → employs → Diffusion Transformer

confidence 90% · We fine-tune a pre-trained image-editing diffusion transformer

CtrlVTON → extends → CtrlVTON-base

confidence 90% · CtrlVTON extends CtrlVTON-base with pixel-level spatial control by training a lightweight LoRA adapter

CtrlVTON → reformulates → Virtual Try-On

confidence 90% · recasts try-on as an image editing problem rather than an inpainting problem

VIP-SAM → evaluatedon → COCO-20 i

confidence 85% · achieves state-of-the-art performance on standard category-level benchmarks [44,50]

VIP-SAM → evaluatedon → PASCAL-5 i

confidence 85% · achieves state-of-the-art performance on standard category-level benchmarks [44,50]

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn -- its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

86,102 characters extracted from source content.

Expand or collapse full text

CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation Seungyong Lee 1† , Hyun Jun Jang 1 , Sangoh Kim 1,2‡ , and Sungjoon Park 1†,* 1 NXN Labs 2 KAIST seungyong@nxn.ai, hjang@nxn.ai, tkddh1109@kaist.ac.kr, sungjoon@nxn.ai https://github.com/nxnai/CtrlVTON Fig. 1: (a) CtrlVTON-base is a baseline image-editing model that enables semantic control via task tokens (full_swap / partial_swap / add) over multiple garment classes. (b) CtrlVTON enables fine-grained spatial control through hand-drawn masks (yellow), supporting both single- and multi-garment try-on. Abstract. Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn— its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel- level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity. Keywords: Virtual Try-On· Controllable Image Generation· Visual-Prompt Segmentation 1 Introduction The fashion and e-commerce industries have long sought to bridge the gap between how a garment appears online and how it looks when worn. Virtual try-on (VTO) addresses this need by synthesizing a photorealistic † Equal contribution. ‡ Work done during internship at NXN Labs. ∗ Corresponding author. arXiv:2607.09362v1 [cs.CV] 10 Jul 2026 2S. Lee et al. image of a person wearing the garment, allowing customers to visualize how they look without physically trying it on. Recent diffusion-based methods have substantially improved photorealism and garment fi- delity [10,12,31,37,43,74], making VTO commercially viable. Despite this progress, current VTO methods share a fundamental limitation: they allow users limited control over how a garment should be worn, includ- ing size (e.g. loose or fitted), style (e.g. tucked in or untucked, zipped or unzipped), and spatial placement (e.g. spatial position, layering). To enable controllability, we start by recasting VTO as an image-editing problem rather than an inpainting problem. This reformulation avoids the well-known failure modes of inpainting-based VTO methods (Sec. B of Supp.). The resulting model, CtrlVTON-base, handles diverse garment categories (tops, bottoms, dresses, shoes, bags) and garment display formats (flatlay, on-person, in-the-wild). It also supports two scenarios that conventional VTO methods struggle with: garment layering (adding an item over the outfit) and selective garment switching (replacing only a specific item). We then notice that even in the editing framework, segmentation masks can be used a pixel-level interface for spatial control. By extending CtrlVTON-base with this capability, we obtain CtrlVTON (Sec. 4), which not only matches the strongest VTO systems on image quality, but also enables fine-grained control over garment style, size, and placement via segmentation masks. The models rely critically on our data preparation pipeline, which requires an automatic, scalable way to segment each reference garment within the person image. We formalize this segmentation problem as a new task, visual-instance-prompt segmentation (VIP-Seg): given a support image (e.g., a flatlay garment), locate the same instance in a query image (e.g., a person wearing it). This task is distinct from the category-level visual-reference-prompt segmentation (VRP-Seg) studied in VRP-SAM [54] and related works [41,56,58,59]. The model must identify a specific instance under same-category distractors, heavy occlusion, and non-rigid deformation between studio flatlay and on-body imagery. Our model, VIP-SAM, achieves state-of-the-art performance on both our purpose-built fashion benchmark and standard category-level benchmarks [44,50] repurposed for instance-level segmentation. In summary, we make the following contributions: 1. We introduce the VIP-Seg task. Our model, VIP-SAM, achieves state-of-the-art results on our fashion domain dataset as well as standard benchmarks repurposed for our application. 2. We propose CtrlVTON, a framework that enables pixel-level control over garment style, size, and placement via segmentation masks. CtrlVTON handles diverse garment categories and garment display formats, and unifies standard garment swap, layering, selective switching, and multi-garment try-on within one framework. 3. We develop a data pipeline that synthesizes (person, garment, person-with-different-garment) triplets along with their corresponding masks. We publicly release VITON-HD-edit (Sec. 4.2), a benchmark built from this pipeline that supports image-editing VTO, mask-controllable VTO, and instance-level visual-prompt segmentation. 2 Related Work 2.1 Virtual Try-On Image-based virtual try-on (VTO) aims to synthesize a photorealistic image of a person wearing a given query garment. Early methods geometrically warped the garment features and used GAN for render- ing [21, 36, 55, 64, 66]. However, these methods struggled with complex poses, fine textures, and occlusion. Diffusion-based VTO substantially improved realism and detail fidelity. Initially, diffusion-based VTO uti- lized latent diffusion and parallel UNets [43,74]. Subsequent methods refined the formulation with semantic- correspondence mechanisms, improved conditioning, and architectural simplification [10, 12, 31, 72]. More recent approaches adopt Diffusion Transformer (DiT) backbones, exploiting stronger generative priors of- fered by large-scale pretrained models [13,15,19,28,37]. Many of these methods are based on image inpainting [10, 12–14, 28, 31, 37, 72]. Although inpainting enables the user to specify which region to edit, the inpainting mask is also the origin of the well-known difficulties with complex poses, occlusions, and identity drift. CtrlVTON3 Fig. 2: Segmentation model architectures. (a) VRP-SAM trains extra modules to extract a visual prompt from support and query images; this prompt is fed to SAM’s mask decoder in place of the usual spatial prompts (point, box, mask) (b, c). VIP-SAM with the SAM (ViT) encoder in (b) and the SAM2 (Hiera) encoder in (c). In each variant, adapters inject support-image features into the encoder after every block. The output is decoded into a mask by a module trained from scratch. These difficulties are resolved by recent editing-based methods [15, 19], which remove the reliance on inpainting masks. Another line of work extends VTO from supporting a single reference garment to multiple garments [13,14, 73]. At the same time, large proprietary image editing models [3, 4,17,35, 45], though not designed for VTO, already match or surpass methods designed for VTO. Although these image editing models removed the limitations of the inpainting models, they also lost the spatial control. The user is left with no way to specify how a garment is worn. M&M VTO [73] and PromptDresser [32] come close by offering coarse text-guided layout for multi-garment outfits, but cannot express precise spatial placement and do not support layered garments. CtrlVTON closes this gap by accept- ing segmentation masks within the editing framework, recovering the inpainting model’s pixel-level spatial control while avoiding its other limitations. 2.2 Controllable Image Editing Existing controllable image editing methods provide spatial control through various interfaces. Structural- cue conditioning [52,68] generates the whole image from edges, depth, or drag points. Layout-based meth- ods [40, 57, 60] operate on bounding boxes or other coarse primitives, with multi-subject variants such as MS-Diffusion [57] pairing each box with a separate reference image. Mask-based inpainting [29, 49, 63, 75] provides precise spatial control through masks, with text prompts specifying the content within. Reference- guided composition [8,65] places a reference image at a user-specified location, reproducing its appearance with high fidelity. None of these interfaces, however, simultaneously supports reference-image fidelity, user- specified spatial layout, and identity preservation of the input person—the combination VTO requires. 2.3 Visual-Reference-Prompt Segmentation The Segment Anything Model (SAM) [34] is a foundation model for interactive segmentation, producing class- agnostic masks from spatial prompts (points, boxes, or coarse masks). SAM2 [48] extends this paradigm to video, propagating object identities across frames through a memory module. Both are restricted to spatial prompts and offer no mechanism for specifying a target instance through a separate reference image. Follow- up work fills this gap with visual reference prompting: given a query and an annotated support image, the model segments the referenced object in the query. Training-free methods like PerSAM [69] and Matcher [41] prompt SAM with feature similarities derived from pretrained foundation models. Training-based methods like VRP-SAM [54] and ProSAM [56] train a dedicated encoder that converts the reference into a SAM- compatible prompt. More recently, SAM3 [5] extends SAM with text and image-exemplar prompts (positive or negative bounding boxes) for concept-level segmentation. Architectural differences aside, these methods (with the exception of SAM3) rely on feature-level corre- spondence between reference and query, and are evaluated at the category-level on splits like Pascal-5 i [50] 4S. Lee et al. and COCO-20 i [44], or open-vocabulary benchmarks like LVIS [20] and SA-Co [5]. This conflates instances of the same category, whereas our setting demands instance-level discrimination: singling out a specific garment among same-category items (e.g. layered shirts of similar color). VIP-SAM is designed to solve this problem. 3 Visual-Instance-Prompt Segmentation (VIP-Seg) Task definition. We define VIP-Seg as follows: given a support image (I S ) of an object (e.g., a studio flatlay of a particular garment), segment precisely that instance in a query image (I Q ) containing it (e.g., a person wearing that garment). This task is distinct from VRP-Seg: the objective is to segment the exact same instance, not just any object of the same class, even in the presence of same-class distractors (e.g., a shirt worn over another shirt of similar color). Rather than generalizing to novel classes, we focus on identifying the same instance across scenes, under same-category distractors, heavy occlusion, and non-rigid deformation between the studio flatlay and the on-person image, typical of product detail page (PDP) images in e-commerce. Why prior architectures struggle. To see why instance-level discrimination is hard for prior methods, consider VRP-SAM’s architecture in Fig. 2(a). The query image is encoded by a frozen SAM backbone, while the support image is processed in parallel to produce a visual reference prompt for the mask decoder. For this to succeed, two conditions must hold: the frozen image encoder must disambiguate all relevant objects, and the matching modules (prompt encoder and mask decoder) must establish correspondence between objects in the support and query images. Both conditions are difficult to meet when garments are layered or share similar color. Empirically, VRP-SAM and similar methods struggle on exactly these cases, see Fig. 5. VIP-SAM. We inject reference features into the backbone at earlier stages, so that the query encoder is conditioned on the support image from the outset rather than only at the prompt level. The query image is processed by the SAM backbone (ViT) or the SAM2 backbone (Hiera), giving rise to two variants shown in Fig. 2 (b) and (c). The support image is processed by interchangeable encoders such as ResNet-50 [22], DINOv2 [46], or DINOv3 [53]. Importantly, the query image features are conditioned on the support image features by cross-attention adapters at intermediate stages of the SAM or SAM2 encoders. Training data. We construct a fashion segmentation dataset for VIP-Seg by collecting (garment, person) image pairs across diverse garment categories. For each pair we annotate the region the garment occupies on the person image, yielding a (garment, person, mask) triplet. The full dataset comprises roughly 10K human-labeled masks. Bridge to CtrlVTON. VIP-SAM is the prerequisite that unlocks the controllable VTO framework in Sec. 4. Training CtrlVTON to learn spatial control requires accurate, garment-instance-level masks, yet existing segmentation methods cannot reliably isolate the garment on the person that matches a given visual reference (Fig. 5). VIP-SAM fills this gap, providing the precise masks needed to train CtrlVTON. 4 Controllable Virtual Try-On CtrlVTON combines the strengths of two VTO paradigms surveyed in Sec. 2: mask-conditioned inpainting and mask-free editing. From editing models we adopt full-image conditioning: rather than erasing and refilling a region, CtrlVTON is conditioned on the entire input image. Under the right training scheme, this enables selective information transfer: pose, identity, other garments, accessories, and background flow through from the input, while only the targeted region is modified to match the new garment. From inpainting models we adopt mask conditioning, which provides local spatial control by specifying exactly where the new garment should appear. 4.1 From Inpainting to Editing VTO aims to produce a target person image p depicting the input person wearing the garment shown in a reference image g ref . Inpainting-based VTO methods cast this as a masked-image completion problem (p masked ,g ref )→ p, where p masked is obtained by erasing the garment region of p. CtrlVTON5 Fig. 3: Overview of training samples. We illustrate both single-garment and multi-garment data as well as the notation used throughout this work. Top: target person p, synthetic reference person p ref , and the reference garment(s) g ref (or g (1..K) ref in the multi-garment case). Bottom: the corresponding masks M p , M p ref , and M g ref provide spatial control. In the multi-garment setting, each garment mask M (k) g ref and its corresponding region in M p are rendered in the same color, enabling simultaneous per-garment control. Editing formulation. As discussed in Sec. 2 (see also Sec. B of Supp. for examples), the inpainting for- mulation requires a fragile trade-off between mask tightness and identity preservation: an undersized mask leaks residual garment pixels that corrupt the output, while an oversized mask erases identity cues the model must then hallucinate. Even worse, the output is further biased by the shape of the mask and the contextual bias outside the mask (e.g., shadows). We avoid this trade-off by casting VTO as an image editing problem (p ref ,g ref ) → p, where p ref is a reference person image depicting the same person, pose, and background as p but wearing a different garment than g ref . Under this formulation the model is free to modify the entire image but learns to preserve identity, pose, and background from the reference image p ref because of how data is prepared. Training therefore requires triplets (p ref ,g ref ,p), which no public dataset provides. We describe how such a dataset can be constructed in Sec. 4.2. 4.2 Data Curation Training the editing model requires triplets (p,p ref ,g ref ). Source pairs (p,g ref ) are readily available from existing data sources, but p ref is not, so we synthesize p ref as described below. For controllability, we use the three masks (M p , M p ref , M g ref ) corresponding to the garment regions in the three images. Thus, each training instance is the tuple p, p ref , g ref , M p , M p ref , M g ref . Fig. 3 shows sample training instances for both single- and multi-garment scenarios. The remainder of this section describes how these images and masks are constructed. Source pairs (p,g ref ). We assemble person–garment pairs from three complementary sources—public VTO datasets, commercial datasets licensed from fashion retailers, and in-house datasets—spanning diverse gar- ment categories (tops, bottoms, full-body garments, shoes, bags) and garment image formats (flatlay, on- person, in-the-wild). A detailed breakdown of the training corpus by garment cardinality (single- vs. multi- garment samples), garment category, and task-token assignment is given in Sec. C.1 of Supp. Masks M p and M g ref . M p is the mask of g ref in p, obtained by querying VIP-SAM (Sec. 3) with g ref as the visual prompt. M g ref is the mask of the reference garment in g ref , obtained via dichotomous image segmentation [42] for flatlay images and via VIP-SAM for human-worn images. Synthesizing p ref . Inspired by the synthetic-data construction pipelines used to train recent image-editing models [51,62], we synthesize p ref with off-the-shelf image generation models [2,3,35,37]. Given (p,M p ,g unpaired ), where g unpaired is a different garment that the person could plausibly wear, an inpainting system is used to produce p ref by filling the masked region of p with g unpaired , while preserving pose, identity, and background. See Sec. C.2 of Supp. for full details. Extracting M p ref . M p ref is the mask of the garment in p ref that must be replaced by g ref . For training data, it is obtained by querying VIP-SAM with g unpaired as the visual prompt. 6S. Lee et al. Fig. 4: Overview of CtrlVTON. For the base model, we fine-tune an image-editing DiT backbone on triplets (p, p ref , g ref ). The controllability extension (red-dashed region) takes as input the masks M p , M p ref , and M g ref , which are channel-wise concatenated with the tokens of the corresponding images. It is implemented as a LoRA adapter on top of the frozen base model. Quality control. For each source pair, we synthesize four to five candidates and select one via a three-stage process: VLM-based screening, contour-based filtering using the VIP-SAM masks M p ref and M p to detect silhouette leakage, and final review by three annotators (Sec. C.3 of Supp.). VITON-HD-edit. Applying the data preparation pipeline to the full VITON-HD test set (2,032 im- ages) yields VITON-HD-edit, a public benchmark supporting image-editing VTO, VIP-Seg, and spatially- controllable VTO. 4.3 Model We train two editing models that differ in the granularity of control. CtrlVTON-base is trained on triplets (p, p ref , g ref ) under semantic-level control. Given p ref and g ref , together with a garment-class token and a task token that specify the garment type and try-on operation, the model generates the final image p. CtrlVTON extends CtrlVTON-base with pixel-level spatial control by training a lightweight LoRA adapter on top of CtrlVTON-base, which additionally consumes the three masks M p , M p ref , and M g ref . The full architecture of both models is summarized in Fig. 4; we now describe each in turn. CtrlVTON-base. We fine-tune a pre-trained image-editing diffusion transformer [3] on triplets (p ref ,g ref ,p). To express the full range of try-on operations, the model is conditioned on two discrete semantic tokens. The garment-class token (τ cls ) ∈ upper, lower, full, shoes, bag specifies the type of g ref . The task token (τ task ) ∈ full_swap, partial_swap, add specifies how existing garments in p ref should be handled: full_swap replaces all garments of the matching class with g ref ; partial_swap replaces only a single garment of that class (e.g., switching just the inner shirt while keeping the outer jacket); and add preserves existing garments and places g ref on top (e.g., layering a jacket over a shirt). Together, the garment-class and task tokens enable the base model to support the garment swapping, selective switching, and layering scenarios introduced in Sec. 1. CtrlVTON CtrlVTON-base is an editing model that provides semantic-level control over VTO: the two discrete tokens specify which try-on operation to perform, while the model itself determines where the new garment should appear on the body. CtrlVTON extends the base model with pixel-level spatial control by conditioning on the three masks M p , M p ref , and M g ref as additional inputs to the network. It is implemented by training a LoRA adapter on top of the frozen base model, with the masks provided as additional inputs (Fig. 4). Inference-time mask. During training, M p is extracted by VIP-SAM from (p,g ref ). During inference, M p is provided by the user. A typical workflow first runs CtrlVTON-base to generate an initial try-on result, extracts the corresponding garment mask using VIP-SAM, and then edits the mask to adjust garment style, fit, or placement before running CtrlVTON. This procedure keeps the inference-time masks aligned with CtrlVTON7 the training distribution. M g ref is extracted by BEN2 [42] or VIP-SAM as in training. M p ref can be an all- white mask (no spatial constraint), an all-black mask (no replacement, as in add), or a SAM-derived or hand-crafted mask when finer control is desired. Mask injection. Each mask is spatially aligned with the corresponding image by construction, so we inject it via channel-wise concatenation in the latent space of the DiT. Let z X ∈ R H×W×C and z M X ∈ R H×W×C denote the VAE-encoded latents of image X and its mask M X , respectively. For each of the three image inputs we form an augmented latent ̃z X = z X ∥ z M X , X ∈p, p ref , g ref ,(1) where∥ denotes channel-wise concatenation. This doubles the channel dimension of the latents while leaving the H × W token grid untouched, so neither the attention cost nor the conditioning-token count grows; we compare against the token-wise injection alternative [33] in Sec. E of Supp. Training objective. The base model is a flow-matching DiT that parameterizes the velocity field v θ (x t ,t| c) [3]. We freeze θ and train LoRA parameters ∆θ attached to the linear projection blocks of both the M- DiT and Single-DiT blocks. We optimize the standard flow-matching objective with augmented conditioning ̃c = (z M p , ̃z p ref , ̃z g ref ,τ cls ,τ task ): L ctrl = E x 1 ∼p data x 0 ∼N(0,I) t∼U[0,1] h v θ+∆θ x t ,t| ̃c − (x 1 − x 0 ) 2 2 i , (2) where x t = (1− t)x 0 + tx 1 and x 1 is the latent of the target p. Extension to multiple garments. In practical applications, outfits often consist of multiple garments, so we extend the model to jointly condition on multiple reference garments. No architectural change is required: the model now ingests multiple reference garmentsg (1) ref ,...,g (K) ref instead of just one. To preserve per-garment controllability, we color-code the conditioning masks: each garment is assigned a distinct RGB color, shared between M p and its corresponding M g (k) ref . These RGB masks are channel-wise concatenated just as in the single-garment case. The model is trained in a single stage on the combined single- and multi-garment dataset. All experiments in Sec. 5 use the multi-garment model. 5 Experiments 5.1 VIP-SAM We evaluate VIP-SAM on our fashion segmentation benchmark and on the standard category-level bench- marks COCO-20 i [44] and PASCAL-5 i [50]. Since our goal is to evaluate VIP-Seg, we modify the evaluation Table 1: VIP-Seg results: mIoU (FB-IoU). Note that VIP-SAM (ViT-B/ResNet-50) significantly outperforms VRP- SAM (ViT-B/ResNet-50). MethodQuery Enc Support Enc Fashion-val Fashion-test COCO-20 i PASCAL-5 i PerSAM [69] ViT-HViT-H52.3 (70.2) 47.5 (67.3) 22.6 (56.5) 45.5 (66.9) Matcher [41] ViT-HDINOv2-L 56.8 (73.2) 52.9 (70.8) 50.7 (72.9) 64.1 (78.4) VRP-SAM [54] ViT-BResNet-50 91.3 (95.0) 91.4 (95.1) 48.5 (71.9) 56.8 (71.6) VRP-SAM [54] ViT-HResNet-50 94.4 (96.8) 94.3 (96.8) 59.4 (78.0) 68.7 (80.5) ProSAM [56] ViT-HResNet-50 94.6 (96.9) 93.8 (96.5) 56.8 (76.6) 69.3 (80.8) VIP-SAMViT-BResNet-5095.5 (97.4)95.3 (97.3)62.3 (79.5)69.4 (80.8) VIP-SAMHiera-B+ResNet-5095.8 (97.6)95.4 (97.4)63.7 (80.1)72.5 (83.1) VIP-SAMHiera-LResNet-5096.5 (98.0)95.8 (97.6)67.2 (82.6)76.6 (85.9) VIP-SAMHiera-LDINOv3-H97.2 (98.4)96.6 (98.1)74.0 (86.4)79.1 (87.6) 8S. Lee et al. Fig. 5: Qualitative comparison of visual-reference segmentation methods. Each column shows a different method applied to the same inputs. Existing methods often include surrounding garments or the entire body, whereas VIP-SAM consistently isolates only the queried instance. protocol on COCO-20 i and PASCAL-5 i by training and evaluating on the same category, rather than on held-out classes. Across all three evaluations, VIP-SAM achieves state-of-the-art performance (Tab. 1), con- firming that early-stage feature injection benefits instance-level segmentation. In Fig. 5, we show examples of layered and similar-textured garments where VRP-SAM-style late feature matching methods fail, whereas VIP-SAM correctly segments the queried instance. Further details on training, evaluation, and compute resources are in Sec. A of Supp. 5.2 CtrlVTON Experimental Setup Datasets. We evaluate on four public benchmarks, each chosen to probe a distinct capability of CtrlVTON. For single-garment VTO we use VITON-HD [9] together with a clothes-only subset of OmniTry Bench [15] containing 2,250 samples. The clothes-only restriction enables fair comparison with prior single-garment VTO methods. For multi-garment try-on, where the model must dress a person using several reference garments, we use DressCode-MR [13] and Garments2Look [24]. For single-garment mask control, we use VITON-HD- edit (introduced in Sec. 4.2). Because no public benchmark is currently available for multi-garment mask control, we report only qualitative results in that setting. Baselines. For single-garment VTO we compare against open-weight inpainting-based VTO models (IDM- VTON [10], CatVTON [12], Leffa [72], Voost [37], CORAL [33]) and editing-based VTO models (Any2AnyTryon [19], OmniTry [15]). For multi-garment VTO we compare against three open-weight models: OmniTry and Fast- Fit [13] transfer reference garments onto the input person, while BootComp [11] generates a new person image from the references. For mask-controllable try-on (Sec. 5.4), no comparable open-weight baseline is available, so we benchmark against the strongest proprietary image-editing models: Nano Banana Pro [17], GPT Image 1.5 [45], Seedream 4.5 [4], and FLUX.2 [pro] [35]. Metrics. We report three families of metrics. All VTO evaluation in this paper follows the unpaired protocol: transferring a garment that the person is not originally wearing. This precludes the use of metrics such as SSIM [61] and LPIPS [70], which require ground-truth target images that are unavailable for unpaired evaluation. Distribution-level scores such as FID [23] remain computable against the real-image distribution, CtrlVTON9 Fig. 6: Illustration of the mask-adherence evaluation. The generated mask M gen (re-extracted via VIP-SAM) is compared against the input control mask M p under IoU (region overlap), d Hu (global shape via Hu invariants), and d H (worst-case boundary deviation via symmetric Hausdorff distance). but are known to be poorly aligned with try-on quality: they capture global statistics while overlooking instance-level errors such as distorted textures or misplaced patterns [15, 39]. We therefore rely on the following instance-level criteria (further details in Sec. F of Supp.). (a) Garment-fidelity metrics. Following OmniTry [15], we measure the cosine similarity between the reference garment and the generated garment crop. This is calculated across two embedding spaces: M-DINO [6], which captures local geometry, and M-CLIP-I [47], which represents global semantics. VIP-SAM (Sec. 3) provides the per-instance garment crops this protocol requires. (b) VLM-as-judge. We adopt a VLM-as-judge framework using Gemini 3.0 Flash [16], building on recent VTO-specific adaptations [7,33,39]. We define three complementary metrics: GTC (Garment Transfer Con- sistency) measures local and global fidelity to the reference garment; PBC (Person-Background Consistency) measures the preservation of identity, pose, held belongings, and background; and PR (Physical Realism) measures whether the garment appears physically plausible on the body (drape, occlusion, lighting consis- tency). These metrics refine or follow established rubrics from CORAL [33] and T-Stars-Tryon 1.0 [7]. (c) Mask-adherence metrics. For evaluating spatial-controllability (Sec. 5.4), we extract the generated mask M gen from the output with VIP-SAM and compare it to the input control mask M p along three comple- mentary axes (Fig. 6): IoU (region overlap), the Hu moment distance d Hu [25, 67] (global shape), and the symmetric Hausdorff distance d H [27,67] (worst-case boundary). 5.3 Single- and Multi-Garment Virtual Try-On We evaluate CtrlVTON-base on three settings: single-garment, multi-garment, and task-token control. The same checkpoint is used for all three (Sec. 4.3). Single-garment. Tab. 2 presents results on VITON-HD and OmniTry Bench. For qualitative comparison, see Fig. 7; further examples are in Fig. 15 of Supp. CtrlVTON-base outperforms inpainting-based baselines and is competitive with, or surpasses, recent editing-based models on most metrics. Table 2: Evaluation of single-garment VTO on VITON-HD and OmniTry Bench. Methods are grouped by formulation (Inpainting vs. Editing). Best per column is bold, second-best is underlined; the row highlighted in light blue is our method. VITON-HDOmniTry Bench Formulation MethodM-DINO↑ M-CLIP-I↑ GTC↑ PBC↑ PR↑ M-DINO↑ M-CLIP-I↑ GTC↑ PBC↑ PR↑ Inpainting IDM-VTON [10]0.71980.8134 3.6798 4.0876 4.08820.58760.7989 3.0124 3.3251 3.2845 CatVTON [12]0.68210.8076 3.6543 4.0432 3.9644 0.57440.7906 2.9856 3.2942 3.1423 Leffa [72]0.66870.7423 3.2987 3.6543 3.7582 0.52340.7345 2.6543 3.0214 2.9567 Voost [37]0.72540.8089 3.6587 4.0521 4.0023 0.57980.7956 3.0012 3.3105 3.2144 CORAL [33]0.72980.8054 3.6612 4.0398 3.8134 0.58210.7923 2.9945 3.2876 3.1021 Editing Any2Any Tryon [19] 0.70980.7521 3.3234 3.6987 3.8872 0.53980.7398 3.1245 3.4128 3.3102 OmniTry [15]0.74210.88234.08764.58764.0187 0.69950.8560 3.65423.87653.7421 CtrlVTON-base0.80540.88454.20574.73014.37530.72820.85514.14014.54394.3121 10S. Lee et al. Fig. 7: Qualitative comparison on single-garment VTO. Only CtrlVTON-base faithfully renders the refer- ence garment while preserving the person’s hairstyle and personal items (phone, watch, and shoulder bag). Additional examples are provided in Fig. 15 of Supp. Multi-garment. Tab. 3 presents results on DressCode-MR and Garments2Look (qualitative comparison in Fig. 16 of Supp.). CtrlVTON-base surpasses methods using architectures designed for multi-garment (Fast- Fit), sequential single-garment inference (OmniTry), or person regeneration from references (BootComp). Table 3: Evaluation of multi-garment VTO on DressCode-MR and Garments2Look. Best per column is bold, second-best is underlined; the row highlighted in light blue is our method. “–” denotes that PBC is not applicable to BootComp, which synthesizes a new person image from the references rather than preserving an input one. ∗ Garments2Look does not provide an unpaired split, so we construct one by randomly pairing persons with garments within the test set. DressCode-MRGarments2Look ∗ MethodM-DINO↑ M-CLIP-I↑ GTC↑ PBC↑ PR↑ M-DINO↑ M-CLIP-I↑ GTC↑ PBC↑ PR↑ BootComp [11]0.54120.7023 2.9845–3.1245 0.53120.7124 2.8456–2.9874 OmniTry [15]0.59870.7456 3.2543 4.0123 3.4218 0.55430.7321 3.1254 3.89423.3421 FastFit [13]0.65120.82763.75434.11233.92760.60120.78543.41283.6543 3.5218 CtrlVTON-base0.67120.84563.89874.68564.15560.65890.83983.87434.62214.1370 Task-token control. The standard VTO benchmarks above target only a single task (full_swap), so they do not cover the full expressivity of CtrlVTON-base. In addition to full_swap, which replaces every existing garment of the matching class with the reference, we also use partial_swap, which replaces a single user-selected garment while preserving the rest, and add, which keeps every existing garment in place and layers the reference on top. Fig. 8 shows the model’s response to all three task tokens. full_swap replaces every existing garment of the matching class with the reference, partial_swap replaces a single user-selected garment while preserving the rest, and add keeps every existing garment in place and layers the CtrlVTON11 reference on top. Switching the task token is the only change between these outputs, demonstrating that a single token-based interface unifies full swap, selective swap, and layering. More qualitative and quantitative evaluations of token-following consistency are reported in Sec. F.3 of Supp. Fig. 8: Effect of the task token on CtrlVTON-base results. For the same (person, garment) pair, switching only the task token yields qualitatively different try-ons: full_swap replaces every upper-class garment with the reference; partial_swap replaces a single upper garment; add keeps all existing garments and layers the reference on top. 5.4 Mask-Controllable Try-On To evaluate CtrlVTON’s ability to faithfully place garments via user-provided masks, we benchmark our method on the VITON-HD-edit dataset against four proprietary editing models [4, 17, 35, 45]. For a fair comparison, each proprietary baseline receives p, g ref , and M p as three reference images, together with a text prompt that explains the role of each reference and describes the desired final image. Tab. 4 presents the headline result: CtrlVTON achieves substantially better spatial control (IoU, d Hu , d H ) than the proprietary baselines by a wide margin while remaining competitive on fidelity metrics. While proprietary models maintain high fidelity, they exhibit a notable lack of spatial adherence, often failing to align the garment with the provided mask (Fig. 9). Table 4: Evaluation of single-garment mask-controllable VTO on VITON-HD-edit. Best per column is bold, second-best is underlined; the row highlighted in light blue is our method. MethodIoU↑d Hu ↓d H ↓ M-DINO↑ M-CLIP-I↑ GTC↑ PBC↑ PR↑ Nano Banana Pro [17] 0.871 0.0044 35.460.8256 0.9087 4.2856 4.1287 4.4877 GPT Image 1.5 [45]0.811 0.0074 53.28 0.78540.88874.0876 3.5743 4.2910 Seedream 4.5 [4]0.865 0.0039 41.99 0.81650.90654.2721 4.6276 4.3788 FLUX.2 [pro] [35]0.8730.0053 38.20 0.79430.90344.2654 4.65894.4912 CtrlVTON0.9610.002226.050.82120.90524.27734.83524.4219 Fig. 10 surveys the controllability enabled by our system: spatial control over fit and placement, unified support for layering and selective swap, and per-garment spatial control via color-coded masks in multi- garment settings. The mask interface gives users direct control over the degree and spatial detail of styling decisions. This level of precision is difficult to convey via text alone: how far a shirt is tucked in, where a zipper stops, how a sleeve is rolled, or how oversized the fit should be. The mask interface thereby establishes VTO as an interactive styling tool rather than a one-shot outfit generator. 12S. Lee et al. Fig. 9: Qualitative comparison of mask-controllable try-on against proprietary editing models. Both single-garment and multi-garment scenarios are covered. Each output is overlaid with per-image mask-adherence metrics (IoU, d Hu , d H ), averaged across masks for multi-garment cases. Control masks are hand-drawn by the authors. Fig. 19 of the appendix overlays M p on each generated image, allowing direct visual inspection of spatial agreement. 6 Conclusion We presented CtrlVTON, a framework that treats controllability as a first-class objective in virtual try-on. VIP-SAM re-identifies a specific garment across flatlay, on-person, and in-the-wild images, yielding the masks necessary for mask-conditioned VTO systems. CtrlVTON recasts try-on as an editing problem over triplets (p ref ,g ref ,p), sidestepping the structural limitations of the inpainting formulation while adding pixel- precise spatial control through a lightweight mask-conditioning LoRA. The resulting model offers a unified framework for garment swapping, layering, and multi-garment composition. To facilitate future research on editing-based and spatially-controllable VTO, we release VITON-HD-edit as a public testbed for both tasks. CtrlVTON13 Fig. 10: Fine-grained control via mask conditioning. Given the same garment and person reference, CtrlVTON generates diverse outputs by varying only the input mask. (Top) Styling of a single garment: tucking, buttoning, and sleeve length. (Middle) Layered styling of an outer garment over an inner one: zipping, tucking, and sizing. (Bottom) Joint styling of a full outfit including top, bottom, and footwear: buttoning, cuffing, and sizing. Inset masks (right of each output) visualize the spatial conditioning and are hand-drawn by the authors. The label below each output describes the authors’ intent. Additional examples are provided in Fig. 18 in the appendix. 14S. Lee et al. References 1. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025) 2. Black Forest Labs: FLUX. https://github.com/black-forest-labs/flux (2024) 3. Black Forest Labs, Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dockhorn, T., English, J., English, Z., Esser, P., Kulal, S., Lacey, K., Levi, Y., Li, C., Lorenz, D., Müller, J., Podell, D., Rombach, R., Saini, H., Sauer, A., Smith, L.: FLUX.1 Kontext: Flow matching for in-context image generation and editing in latent space (2025), https://arxiv.org/abs/2506.15742 4. ByteDance Seed Team: Seedream 4.0: Toward next-generation multimodal image generation. arXiv preprint arXiv:2509.20427 (2025) 5. Carion, N., Gustafson, L., Hu, Y.T., et al.: SAM 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719 (2025) 6. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV (2021) 7. Chen, M., Chen, Z., Du, Y., Gao, Z., Hu, T., Lan, J., Lin, C., Shen, Y., Wang, X., Wang, Z., et al.: Tstars-Tryon 1.0: Robust and realistic virtual try-on for diverse fashion items. arXiv preprint arXiv:2604.19748 (2026) 8. Chen, X., Huang, L., Liu, Y., Shen, Y., Zhao, D., Zhao, H.: AnyDoor: Zero-shot object-level image customization. In: CVPR (2024) 9. Choi, S., Park, S., Kang, M., Choo, J.: VITON-HD: High-resolution virtual try-on via misalignment-aware nor- malization. In: CVPR (2021) 10. Choi, Y., Kwak, S., Lee, K., Choi, H., Shin, J.: Improving diffusion models for authentic virtual try-on in the wild. In: ECCV (2024) 11. Choi, Y., Kwak, S., Yu, S., Choi, H., Shin, J.: Controllable human image generation with personalized multi- garments. In: CVPR (2025) 12. Chong, Z., Dong, X., Li, H., Zhang, S., Zhang, W., Zhang, X., Zhao, H., Jiang, D., Liang, X.: CatVTON: Concatenation is all you need for virtual try-on with diffusion models. In: ICLR (2025) 13. Chong, Z., Lei, Y., Zhang, S., He, Z., Wang, Z., Zhang, X., Dong, X., Wu, Y., Jiang, D., Liang, X.: FastFit: Accelerating multi-reference virtual try-on via cacheable diffusion models. arXiv preprint arXiv:2508.20586 (2025) 14. Deria, A., et al.: MuGa-VTON: Multi-garment virtual try-on via diffusion transformers with prompt customiza- tion. arXiv preprint arXiv:2508.08488 (2025) 15. Feng, Y., Zhang, L., Cao, H., Chen, Y., Feng, X., Cao, J., Wu, Y., Wang, B.: OmniTry: Virtual try-on anything without masks. In: NeurIPS (2025) 16. Google DeepMind: Gemini 3 Flash. Google DeepMind model page (2025), https://deepmind.google/models/ gemini/flash/ 17. Google DeepMind: Introducing Nano Banana Pro: Gemini 3 Pro Image. Google blog post (2025), https://blog. google/technology/ai/nano-banana-pro/ 18. Güler, R.A., Neverova, N., Kokkinos, I.: DensePose: Dense human pose estimation in the wild. In: CVPR (2018) 19. Guo, H., Zeng, B., Song, Y., Zhang, W., Zhang, C., Liu, J.: Any2AnyTryon: Leveraging adaptive position em- beddings for versatile virtual clothing tasks. In: ICCV (2025) 20. Gupta, A., Dollár, P., Girshick, R.: LVIS: A dataset for large vocabulary instance segmentation. In: CVPR (2019) 21. Han, X., Wu, Z., Wu, Z., Yu, R., Davis, L.S.: VITON: An image-based virtual try-on network. In: CVPR (2018) 22. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016) 23. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS (2017) 24. Hu, J., Cheng, Z., Wong, W., Zou, X.: Garments2look: A multi-reference dataset for high-fidelity outfit-level virtual try-on with clothing and accessories. arXiv preprint arXiv:2603.14153 (2026) 25. Hu, M.K.: Visual pattern recognition by moment invariants. IRE Trans. Inf. Theory 8(2), 179–187 (1962) 26. Huang, L., Wang, W., Wu, Z.F., Shi, Y., Dou, H., Liang, C., Feng, Y., Liu, Y., Zhou, J.: In-context lora for diffusion transformers. arXiv preprint arXiv:2410.23775 (2024) 27. Huttenlocher, D.P., Klanderman, G.A., Rucklidge, W.J.: Comparing images using the Hausdorff distance. IEEE TPAMI 15(9), 850–863 (1993) 28. Jiang, B., Hu, X., Luo, D., He, Q., Xu, C., Peng, J., Zhang, J., Wang, C., Wu, Y., Fu, Y.: FitDiT: Advancing the authentic garment details for high-fidelity virtual try-on (2024), https://arxiv.org/abs/2411.10499 29. Ju, X., Liu, X., Wang, X., Bian, Y., Shan, Y., Xu, Q.: BrushNet: A plug-and-play image inpainting model with decomposed dual-branch diffusion. In: ECCV (2024) 30. Khirodkar, R., Bagautdinov, T., Martinez, J., Zhaoen, S., James, A., Selednik, P., Anderson, S., Saito, S.: Sapiens: Foundation for human vision models. In: ECCV (2024) 31. Kim, J., Gu, G., Park, M., Park, S., Choo, J.: StableVITON: Learning semantic correspondence with latent diffusion model for virtual try-on. In: CVPR (2024) CtrlVTON15 32. Kim, J., Jin, H., Park, S., Choo, J.: Promptdresser: Improving the quality and controllability of virtual try-on via generative textual prompt and prompt-aware mask. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 16026–16036 (2025) 33. Kim, J., Shin, Y., Jin, S., Chung, D., Nam, J., Kim, T., Park, J., Kang, H., Kim, S.: Coral: Correspondence alignment for improved virtual try-on. arXiv preprint arXiv:2602.17636 (2026) 34. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollar, P., Girshick, R.: Segment anything. In: ICCV (2023) 35. Labs, B.F.: FLUX.2: Frontier Visual Intelligence. https://bfl.ai/blog/flux-2 (2025) 36. Lee, S., Gu, G., Park, S., Choi, S., Choo, J.: High-resolution virtual try-on with misalignment and occlusion- handled conditions. In: ECCV (2022) 37. Lee, S., gi Kwak, J.: Voost: A unified and scalable diffusion transformer for bidirectional virtual try-on and try-off. In: ACM SIGGRAPH Asia (2025) 38. Levin, E., Fried, O.: Differential diffusion: Giving each pixel its strength. In: Computer Graphics Forum. vol. 44. Wiley Online Library (2025) 39. Li, J., Chen, T., Jiang, S., Wang, W., Luo, J., Wu, C.: OpenVTON-Bench: A large-scale high-resolution benchmark for controllable virtual try-on evaluation. arXiv preprint arXiv:2601.22725 (2026) 40. Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: GLIGEN: Open-set grounded text-to-image generation. In: CVPR (2023) 41. Liu, Y., Zhu, M., Li, H., Chen, H., Wang, X., Shen, C.: Matcher: Segment anything with one shot using all-purpose feature matching. In: ICLR (2024) 42. Meyer, M., Spruyt, J.: Ben: Using confidence-guided matting for dichotomous image segmentation. arXiv preprint arXiv:2501.06230 (2025) 43. Morelli, D., Baldrati, A., Cartella, G., Cornia, M., Bertini, M., Cucchiara, R.: LaDI-VTON: Latent diffusion textual-inversion enhanced virtual try-on. In: ACM M (2023) 44. Nguyen, K., Todorovic, S.: Feature weighting and boosting for few-shot segmentation. In: ICCV (2019) 45. OpenAI: The new ChatGPT Images is here. https://openai.com/index/new-chatgpt-images-is-here/ (2025) 46. Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning robust visual features without supervision. TMLR (2024) 47. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML (2021) 48. Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.Y., Girshick, R., Dollár, P., Feichtenhofer, C.: SAM 2: Segment anything in images and videos. In: ICLR (2025) 49. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022) 50. Shaban, A., Bansal, S., Liu, Z., Essa, I., Boots, B.: One-shot learning for semantic segmentation. In: BMVC (2017) 51. Sheynin, S., Polyak, A., Singer, U., Kirstain, Y., Zohar, A., Ashual, O., Parikh, D., Taigman, Y.: Emu Edit: Precise image editing via recognition and generation tasks. In: CVPR (2024) 52. Shi, Y., Xue, C., Liew, J.H., Pan, J., Yan, H., Zhang, W., Tan, V.Y.F., Bai, S.: DragDiffusion: Harnessing diffusion models for interactive point-based image editing. In: CVPR (2024) 53. Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, C., Mairal, J., Jégou, H., Labatut, P., Bojanowski, P.: DINOv3. arXiv preprint arXiv:2508.10104 (2025) 54. Sun, Y., Chen, J., Zhang, S., Zhang, X., Chen, Q., Zhang, G., Ding, E., Wang, J., Li, Z.: VRP-SAM: SAM with visual reference prompt. In: CVPR (2024) 55. Wang, B., Zheng, H., Liang, X., Chen, Y., Lin, L., Yang, M.: Toward characteristic-preserving image-based virtual try-on network. In: ECCV (2018) 56. Wang, X., Sebastian, C., He, W., Ren, L.: Prosam: Enhancing the robustness of sam-based visual reference segmentation with probabilistic prompts. In: ICCV (2025) 57. Wang, X., Fu, S., Huang, Q., He, W., Jiang, H.: MS-Diffusion: Multi-subject zero-shot image personalization with layout guidance. In: ICLR (2025) 58. Wang, X., Wang, W., Cao, Y., Shen, C., Huang, T.: Images speak in images: A generalist painter for in-context visual learning. In: CVPR (2023) 59. Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., Huang, T.: SegGPT: Towards segmenting everything in context. In: ICCV (2023) 16S. Lee et al. 60. Wang, X., Darrell, T., Rambhatla, S.S., Girdhar, R., Misra, I.: InstanceDiffusion: Instance-level control for image generation. In: CVPR (2024) 61. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600–612 (2004) 62. Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., ming Yin, S., Bai, S., Xu, X., Chen, Y., Chen, Y., Tang, Z., Zhang, Z., Wang, Z., Yang, A., Yu, B., Cheng, C., Liu, D., Li, D., Zhang, H., Meng, H., Wei, H., Ni, J., Chen, K., Cao, K., Peng, L., Qu, L., Wu, M., Wang, P., Yu, S., Wen, T., Feng, W., Xu, X., Wang, Y., Zhang, Y., Zhu, Y., Wu, Y., Cai, Y., Liu, Z.: Qwen-Image technical report (2025), https://arxiv.org/abs/2508.02324 63. Xie, S., Zhang, Z., Lin, Z., Hinz, T., Zhang, K.: SmartBrush: Text and shape guided object inpainting with diffusion model. In: CVPR (2023) 64. Xie, Z., Huang, Z., Dong, X., Zhao, F., Dong, H., Zhang, X., Zhu, F., Liang, X.: GP-VTON: Towards general purpose virtual try-on via collaborative local-flow global-parsing learning. In: CVPR (2023) 65. Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Wen, F.: Paint by example: Exemplar-based image editing with diffusion models. In: CVPR (2023) 66. Yang, H., Zhang, R., Guo, X., Liu, W., Zuo, W., Luo, P.: Towards photo-realistic virtual try-on by adaptively generating↔preserving image content. In: CVPR (2020) 67. Yang, L., Liu, Y., Li, Y., Bai, X., Lu, H.: FitControler: Toward fit-aware virtual try-on. arXiv preprint arXiv:2512.24016 (2025) 68. Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV (2023) 69. Zhang, R., Jiang, Z., Guo, Z., Yan, S., Pan, J., Dong, H., Qiao, Y., Gao, P., Li, H.: Personalize segment anything model with one shot. In: ICLR (2024) 70. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018) 71. Zheng, C., Lan, Y., Wang, Y.: Lanpaint: Training-free diffusion inpainting with asymptotically exact and fast conditional sampling. TMLR (2025) 72. Zhou, Z., Liu, S., Han, X., Liu, H., Ng, K.W., Xie, T., Cong, Y., Li, H., Xu, M., Pérez-Rua, J.M., Patel, A., Xiang, T., Shi, M., He, S.: Learning flow fields in attention for controllable person image generation. In: CVPR (2025) 73. Zhu, L., Li, Y., Liu, N., Peng, H., Yang, D., Kemelmacher-Shlizerman, I.: M&M VTO: Multi-garment virtual try-on and editing. In: CVPR (2024) 74. Zhu, L., Yang, D., Zhu, T., Reda, F., Chan, W., Saharia, C., Norouzi, M., Kemelmacher-Shlizerman, I.: TryOn- Diffusion: A tale of two UNets. In: CVPR (2023) 75. Zhuang, J., Zeng, Y., Liu, W., Yuan, C., Chen, K.: A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. In: ECCV (2024) CtrlVTON17 Supplementary Material A VIP-SAM: Resource Comparison We complement the segmentation accuracy results in Tab. 1 of the main paper with a comparison of the resource footprint of each method. Tab. 5 shows peak training memory, peak inference memory, FLOPs, total parameter count, and the number of trainable parameters. Three observations stand out. First, VIP- SAM (ViT-B/ResNet-50) requires substantially higher training resource than VRP-SAM (ViT-B/ResNet-50) because gradient must be computed for the layers inside the ViT backbone, but the inference resource is comparable. From architectural point of view, these two configurations are directly comparable, and the results in Tab. 1 of the main text support the validity of our design choices. Second, VIP-SAM (especially Hiera variants) achieves higher accuracy with a substantially smaller resource footprint (memory, FLOPs, parameters) than the SAM-ViT-H-based baselines: even our largest variant (Hiera-L + DINOv3-H) has inference memory comparable to PerSAM/Matcher while reaching state-of-the-art accuracy. Third, despite training more parameters than VRP-SAM/ProSAM, our smaller variants use less inference memory and fewer FLOPs, owing to the lighter Hiera backbones. Table 5: Resource comparison for visual-reference segmentation methods. “–” denotes metrics that are not applicable: PerSAM and Matcher are training-free. MethodTrain Mem (GiB) Inference Mem (GiB) FLOPs (×10 12 ) Tot Params (M) Train Params (M) PerSAM [69] (ViT-H / ViT-H)–6.4711.9641– Matcher [41] (ViT-H / DINOv2-L)–7.658.07945– VRP-SAM [54] (ViT-B/ResNet-50)14.24.071.481181.59 VRP-SAM [54] (ViT-H/ResNet-50)18.57.056.476661.59 ProSAM [56] (ViT-H/ResNet-50)19.17.056.476661.72 VIP-SAM (ViT-B / ResNet-50)42.44.873.7716149.0 VIP-SAM (Hiera-B+ / ResNet-50)20.12.781.4312835.9 VIP-SAM (Hiera-L / ResNet-50)30.53.302.6327438.5 VIP-SAM (Hiera-L / DINOv3-H)32.26.504.431,09037.4 B Inpainting Failure Modes: Visual Examples In Sec. 1 and Sec. 4.1 of the main paper, we claimed that inpainting-based VTO is fundamentally limited by the mask itself. We justify this claim through Fig. 11, which shows two examples and four annotated failure regions. The first three regions show that the generated output is highly sensitive to the size and shape of the mask; the last shows that the output is also entangled with the context outside the mask. C Data Pipeline Details C.1 Data Sources and Proportions The person–garment pairs used to train CtrlVTON are drawn from three complementary sources: (i) publicly available VTO datasets, (i) commercial datasets licensed from fashion retailers, and (i) in-house datasets collected specifically for this work. A coarse breakdown of source-level proportions and garment-category coverage is reported in Table 6. We further break down the corpus along three axes that are directly relevant to the capabilities evaluated in Sec. 5 of the main paper: the single- vs. multi-garment split, the distribution over garment categories, and the distribution over task tokens (Sec. 4.3 of the main paper). Table 7 reports these statistics. A multi- garment sample is labeled mixed in the per-sample task-token breakdown when its constituent garments are assigned more than one distinct task token (e.g., one garment tagged full_swap while another in the same sample is tagged add); this case has no analogue in the per-garment breakdown, where every garment carries exactly one task token by construction. 18S. Lee et al. Fig. 11: Failure modes of inpainting-based VTO. Each example shows the input person, the masked person fed to the inpainting model, the reference garment, and the inpainted output. Red: an undersized mask leaves residual pixels of the original garment (blue jacket sleeve); the inpainting model treats them as context and renders the new garment incorrectly. Green: enlarging the mask to avoid the above issue erases identity information (pose, face, hair, skin and tattoos, held items like bags or phones), which the inpainting model must then hallucinate (cf. Fig. 7, rows 2–3). Cyan: the output is biased by the shape of the mask itself. A trapezoidal erasure region cues the inpainting model to synthesize a skirt even though the reference shows pants. Magenta: the output is also biased by context outside the mask. The shadow of the original skirt on the floor remains visible and the inpainting model harmonizes the new garment with that shadow, again producing a skirt rather than the reference pants. These examples illustrate why mask-completion is inadequate for VTO: it is heavily influenced by mask quality and the context, both of which are difficult to control. Table 6: Coarse composition of the CtrlVTON training corpus. SourceShare Public VTO datasets32% Licensed commercial datasets 62% In-house datasets6% C.2 Masking Strategies for Synthetic p ref To obtain synthetic images for our training data, we utilize inpainting models [2,37] and editing models [3,35] with training-free inpainting methods [38, 71]. We will refer to both methods as an inpainting system for convenience. The mask supplied to the inpainting system controls how much of the original image is allowed to change. A fixed strategy is insufficient because each image requires different trade-offs between context preservation, garment-shape variation, and localization. We therefore use three masking strategies, summarized below in order from our most frequently used (default) to the most surgical (exception). Box mask (default). Used as the default for the majority of the corpus. Starting from a garment-agnostic body region produced by off-the-shelf human segmentation models [18,30], we extract the axis-aligned bound- ing box of the garment region, and exclude face, hair, hands, and held items. The box’s extent is independent of the original garment’s silhouette, so the inpainting system has the freedom to render the new garment in a wide range of shapes, producing the highest garment-shape diversity in the synthetic data. CtrlVTON19 Table 7: Composition of the CtrlVTON training corpus by garment cardinality, garment category, and task-token assignment. Garment cardinality (per sample) Single-garment 51% Multi-garment 49% Garment category (per garment) Upper41% Lower24% Full18% Bag7% Shoes10% Task token (per garment) full_swap 68% partial_swap 18% add14% Task token (per sample) full_swap 70% partial_swap 11% add7% Mixed12% Loose mask. Used when free-form variation is unsafe, e.g., the background contains complex structure that should not change, or the person is holding accessories (bags, jewelry) that must be preserved. We again build a garment-agnostic mask using human segmentation models [18,30], but keep its segmentation contour rather than converting it to a bounding box. This confines the edit to the body silhouette while allowing natural garment-shape variation. Tight mask. Used to synthesize data for operations that demand precise spatial control—most importantly the partial_swap task token (Sec. 4.3 of the main paper), where only a single specific garment among several should be modified. The tight mask is the per-instance VIP-SAM mask of the original garment, dilated by a small margin. This concentrates the edit in the exact region intended for change and leaves the rest of the body untouched. We generate at least one candidate per strategy and use the VLM + human screening described below to pick the best p ref for each image. C.3 Quality-Control Protocol For every source pair (p,g ref ) we synthesize four to five candidate p ref images using inpainting systems with different masking strategies. The candidates then pass through a three-stage funnel: a VLM-based screen, an automatic silhouette-leakage filter, and a final human review that selects the best candidate. Stage 1: VLM-based screen. A vision–language model is prompted to answer four yes/no questions comparing each candidate p ref to p: (1) Is the person identity preserved? (2) Is the pose preserved? (3) Is the background preserved? (4) Is the modification confined to a garment region? A candidate must receive “yes” on all four questions to proceed to the next stage. Stage 2: Silhouette-leakage filter. When fine-tuning a pre-trained editing model for VTO, it tends to preserve the silhouette of p ref . Therefore, training on pairs (p,p ref ) with near-identical silhouettes reinforces the model’s tendency to ignore the shape of g ref (Fig. 17). In our case, since both VIP-SAM masks M p and M p ref are easily obtained, we use them directly in the screening process. Let A and B denote the two binary masks, and let ∂A,∂B be their 1-pixel-thick boundary point sets. Given the tolerance τ px we define the directional contour match fractions f A→B = p∈ ∂A : d(p,∂B)≤ τ |∂A| , f B→A = p∈ ∂B : d(p,∂A)≤ τ |∂B| , (3) where d(p,S) = min q∈S ∥p−q∥ 2 is the Euclidean distance from p to the closest point in S, computed efficiently via a distance transform. We then define the contour match fraction as CMF(A,B) = max f A→B , f B→A , (4) and discard any candidate whose CMF(M p ,M p ref ) exceeds a threshold. Intuitively, CMF is the largest fraction of one mask’s contour that lies within τ pixels of the other mask’s contour; a high value means the edited and original garments share most of their silhouette. We use this contour-based score rather than mask IoU 20S. Lee et al. because in the case of partial silhouette overlap (e.g., identical sleeves and hem with a different neckline), IoU remains only moderate even though most of the contour is leaking, see Fig. 12 (d). The maximum in CMF makes the score symmetric and robust to one mask being a strict subset of the other. CMF is also distinct from the symmetric Hausdorff distance d H used in our mask-adherence metrics (Appendix F.4): d H captures the worst-case pointwise deviation between two contours, whereas CMF captures the bulk fraction of contour that is within the tolerance, which is what we need to detect silhouette leakage. Fig. 12: Examples illustrating the contour match fraction (CMF) filter and its advantage over IoU. Each row shows a (p, p ref ) pair together with the two VIP-SAM masks M p and M p ref , and reports both IoU(M p , M p ref ) and CMF(M p , M p ref ). Pairs marked fail (high CMF) are discarded because their silhouettes are nearly identical: training on them would teach the model to ignore g ref and copy the reference contour. Case (d) shows an example that an IoU-based filter would miss. Stage 3: Human review. Three annotators inspect every candidate that survives Stages 1 and 2. Their job is twofold: catch failure cases the automatic filters missed (e.g., subtle identity drift, shadow inconsistency, implausible garment proportions) and select the best candidate per source pair by overall physical plausibility. The top candidate becomes the p ref used for that training instance; all others are discarded. D Training Details Visual-Instance-Prompt Segmentation Our training configuration mostly follows VRP-SAM’s since our innovation is in the architecture. We randomly split the fashion dataset into train, val, and test sets according to the ratio 85 : 7.5 : 7.5. On this dataset, we use the same lr scheduler and loss as VRP-SAM, but change batch size to 16 and train for 100 epochs. Note that 16 is the maximum batch size that fits on an L40 GPU for ViT-B / ResNet-50 configuration. For fairness, we use the same configuration when training VRP-SAM and ProSAM on this dataset. When training on the COCO-20 i and PASCAL-5 i datasets, we use the splits (i.e. the partition of classes into training and held-out) provided by VRP-SAM and report the average across splits. Unlike standard few- shot segmentation, we evaluate on the same classes used during training rather than the held-out classes, so that the evaluation becomes similar to our task—finding the same instance—rather than generalization to novel classes. We train all models (VIP-SAM, VRP-SAM, ProSAM) for 50 epochs, matching VRP-SAM’s original configuration. Interestingly, ProSAM outperforms VRP-SAM on the PASCAL-5 i dataset but not on the COCO-20 i dataset. Upon closer examination, we find that images in COCO-20 i often contain multiple same-category CtrlVTON21 instances of varying size and shape. A simple proxy for this is the number of connected components per annotation mask (this can both overestimate and underestimate the object count: adjacent objects merge into a single component, while individual objects can fragment into several). After filtering small mask components, which are usually just noise, we find that on average, a mask in COCO-20 i contains 1.91 components per mask vs 1.77 in PASCAL-5 i . This is telling since the trick introduced in ProSAM pushes the support image embedding into a flat region of the loss landscape, which acts like a regularizer. When the support image contains multiple visually distinct objects of the same class, the resulting embedding becomes an "average" over them. While this regularization helps generalization to novel classes, it can hurt when evaluation classes match training classes. This is especially true when the support image contains multiple instances: while ProSAM is forced to average over them, an unregularized embedding could instead collapse onto a single instance, which is what instance-level identification needs. CtrlVTON-base CtrlVTON-base is obtained by full-parameter fine-tuning of FLUX.2 Klein [3] on the triplets (p ref ,g ref ,p). Training runs at ∼1MP input resolution with a global batch size of 128. The total compute budget is approximately 20 H200-days (i.e., 20 NVIDIA H200 GPUs for approximately one day of continuous training). E Mask Injection Strategies: Channel-Wise vs. Token-Wise In Sec. 4.3 of the main paper, we concatenated the mask tokens channel-wise with their corresponding image tokens. A natural alternative is to concatenate them along the token dimension instead, treating the mask tokens as additional reference tokens as in IC-LoRA [26] and CORAL [33]. Here we justify our choice of channel-wise over token-wise concatenation. Design rationale. Channel-wise concatenation has two properties that are desirable in our setting. First, every mask is spatially aligned with its corresponding image by construction: M p shares the same pixel grid as p, M p ref as p ref , and M g ref as g ref . Concatenating along the channel dimension preserves this alignment for free. Token-wise concatenation instead treats each mask as an additional reference block, forcing the network to re-discover the spatial correspondence through attention. Second, the cost of self-attention is quadratic in the number of tokens, so concatenating three full-resolution mask tokens inflates both compute and memory at every attention layer, which is particularly inefficient for high resolution images. Empirical comparison. We compare the two concatenation strategies under the same training budget on the VITON-HD-edit benchmark, and report trainable parameter count, inference throughput, peak inference VRAM, and mask-adherence and perceptual metrics. Table 8: Channel-wise vs. token-wise mask injection. The two variants start from the same base checkpoint and are trained for the same number of steps. Injection scheme Trainable params (M) Inference speed (it/s) VRAM (GB) IoU↑d Hu ↓ d H ↓ M-DINO↑ M-CLIP-I↑ GTC↑ Token-wise5720.3862.650.8925 0.0034 31.84 0.78210.8987 4.1985 Channel-wise5720.9143.470.96100.002226.050.82120.90524.2773 F Evaluation Metrics: Details In this section, we present additional details on garment-fidelity, VLM-as-judge, and mask-adherence, which were summarized in Sec. 5.2 of the main paper (a)–(c). F.1 Garment-fidelity Metrics (M-DINO, M-CLIP-I) We adopt the garment-fidelity metrics of OmniTry [15], which are designed for the mask-free editing setting where no per-instance ground-truth image is available. For each generated try-on image we crop the gar- ment region using its VIP-SAM mask (Sec. 3 of the main paper), apply white-background normalization to 22S. Lee et al. remove surrounding context, and compare the result with an identically processed reference garment in two embedding spaces. For each encoder we report the cosine similarity, which lies in [−1, 1], higher being better: – M-DINO: features from a self-supervised ViT trained with DINO [6] capture fine-grained local structure, so this score is sensitive to structural details of garment parts. – M-CLIP-I: features from the CLIP [47] image encoder are aligned with semantic concepts, so this score primarily reflects category-level coherence. Empirically, M-DINO scores tend to be lower than M-CLIP-I on the same outputs because the DINO embedding penalizes geometric variation that the CLIP embedding is largely invariant to. The two metrics are therefore most informative when read together: a method that scores well on M-CLIP-I but poorly on M-DINO preserves the garment category but not its precise local structure, whereas a method strong on both is faithful at both the category and structural levels. F.2 VLM-as-Judge: GTC / PBC / PR We use Gemini 3.0 Flash [16] as a judge to score try-on outputs along three VTO quality criteria. Re- cent work has shown that traditional distributional metrics such as FID inadequately capture VTO-specific quality, motivating VLM-based evaluation protocols [7, 33, 39]. CORAL [33] introduces Garment Transfer Consistency (GTC) and Fit Pose Coherence (FPC) to separately assess garment fidelity and wearing plau- sibility. OpenVTON-bench [39] decomposes try-on quality into five criteria (background, identity, texture, shape, realism), arguing that a single aggregate score hides failures specific to individual criteria. Tstars- Tryon 1.0 [7] organizes the criteria into two stages: a garment-aware stage that evaluates identity consistency and garment fidelity given the reference garment as context, and a garment-agnostic stage that evaluates background preservation and physical realism without the reference garment, isolating these aspects from garment-induced bias. Following the broader VLM-as-judge literature, we adopt three criteria to evaluate VTO quality: – GTC (Garment Transfer Consistency): how faithfully the reference garment is reproduced on the person, from local details (prints, buttons, zippers, pockets) to global properties (texture, color, silhouette). – PBC (Person-Background Consistency): how well the input person and surroundings are preserved, including identity (face, hair, skin, tattoos), pose, body shape, personal items (bags, phones, watches, jewelry), and background. – PR (Physical Realism): whether the garment is worn in a physically plausible manner, including drape and folds, contact with the body, consistent occlusion with other garments and accessories, and light- ing/shadow consistency. Each metric is scored on a [0, 5] float scale, where 5 means “no perceivable issues” and 0 means “severe failure on every aspect.” The grader is instructed to evaluate strictly: every visible artifact, mismatch, or implausibility deducts points, and a perfect score is reserved for outputs that withstand close scrutiny. The exact prompt we send to the model is below. VLM-as-judge prompt (verbatim) You are a strict evaluator of virtual try-on results. You will be shown the following images: [INPUT_PERSON] - the original photograph of the person. [REFERENCE_GARMENT] - the garment(s) that the person should be wearing (could be a single item or a set of multiple items). [GENERATED_IMAGE] - the model’s output. Score the GENERATED_IMAGE along three orthogonal axes. Each score is a float in [0, 5]. Be STRICT: 5 is reserved for outputs with no perceivable issues; deduct points for any visible artifact, mismatch, or implausibility. GTC (Garment Transfer Consistency, [0, 5]): Does the rendered garment match the REFERENCE_GARMENT in BOTH fine details (prints, buttons, zippers, logos, text) AND global CtrlVTON23 properties (texture, color, silhouette)? - (Multi-garment): Check if ALL provided reference garments are present and individually accurate. Deduct points if any item is missing, simplified, or shows color bleeding from other items. PBC (Person-Background Consistency, [0, 5]): Are the input person and surroundings preserved by the edit? - Identity & Body: Face, hair, skin tone, tattoos, and proportions. - Non-target Areas: Preservation of held belongings (bags, phones) and original garments NOT meant to be changed (e.g., shoes, hats). - Background: Scene integrity, lighting, and lack of warping. PR (Physical Realism, [0, 5]): Is the garment worn in a physically plausible way? - Drape and folds consistent with gravity and body contour. - Correct contact: No floating fabric or clipping through skin. - (Multi-garment): Check the interaction between items. Outerwear must correctly cover inner layers; tops should be naturally tucked in or layered over bottoms without unrealistic merging or clipping. - Lighting and shadows on the new garment(s) match the scene. Return your answer as a single JSON object, no extra text: "GTC": <float>, "PBC": <float>, "PR": <float> Reproducibility and the Choice of a Proprietary VLM Judge Using Gemini 3.0 Flash [16] as the VLM-as-judge introduces a reproducibility limitation inherent to proprietary VLM models: the underlying model can change or be deprecated by its provider without notice, and its internals cannot be independently audited. However, this limitation is not unique to our work: concurrent VTO evaluation work relies on proprietary VLM judges as well, with OpenVTON-Bench [39] and CORAL [33] using Gemini 2.0 Flash and GPT-5, respectively. Open-weight VLM models currently lack the discriminative power that reliable VTO evaluation requires. As an illustration, we repeat our GTC/PBC/PR evaluation with an open-weight VLM judge, Qwen3-VL-8B-Instruct [1], on the same five methods evaluated on VITON-HD-edit in Tab. 4 of the main paper (the four proprietary editing models and CtrlVTON). Table 9 reports, for each method, the mean over five independent scoring runs together with the 95% confidence interval. The confidence intervals overlap substantially across methods on GTC and PR, and even where a gap appears (e.g., PBC) it spans only a few hundredths of a point—far too little to reflect the differences we know exist between these methods. In practice, Qwen3-VL-8B-Instruct cannot reliably rank methods at this quality level. Utilizing a proprietary model for VLM-as-judge is required to assess VTO output at retail-grade image quality. Table 9: VLM-as-judge scores from an open-weight model (Qwen3-VL-8B-Instruct [1]) on VITON-HD-edit, using the same GTC/PBC/PR system prompt as Appendix F.2. Each cell reports the mean ± 95% confidence interval over 5 independent runs. Compare against the proprietary-VLM (Gemini 3.0 Flash) scores for the same methods in Tab. 4 of the main paper: the confidence intervals here overlap heavily despite the real quality differences that Tab. 4 reveals. MethodGTC↑PBC↑PR↑ Nano Banana Pro [17] 4.782± 0.031 4.911± 0.047 4.902± 0.041 GPT Image 1.5 [45] 4.803± 0.044 4.869± 0.033 4.921± 0.041 Seedream 4.5 [4]4.741± 0.052 4.933± 0.021 4.858± 0.043 FLUX.2 [pro] [35]4.789± 0.037 4.902± 0.041 4.928± 0.037 CtrlVTON4.790± 0.0185.000± 0.0004.970± 0.009 F.3 Task-Following Consistency (TFC) for Token Control GTC/PBC/PR are designed to be task-agnostic: they measure properties of the output that should hold regardless of which task token (τ task ) was used. To quantitatively evaluate adherence to the task-token 24S. Lee et al. (Sec. 5.3 of the main paper, “Task-token adherence”), we additionally define a fourth, task-specific criterion used only for the evaluation in Tab. 10. TFC (Task-Following Consistency, [0, 5]): how faithfully the output realizes the operation specified by the task token, evaluated separately for each token: – full_swap: every garment of the matching class in the input is replaced with the reference; no original garment of that class remains. – partial_swap: exactly one garment of the matching class is replaced with the reference; all other garments, including same-class items, remain unchanged. – add: all original garments are preserved; the reference is added as an additional layer on top of the existing outfit. The grader is shown the input person, reference garment, generated output, and the intended task token, and returns a strict TFC score under the same protocol as Appendix F.2. We are not aware of any open- weight VTO system that exposes operation-level control via discrete tokens, so a head-to-head comparison is not possible; Tab. 10 therefore reports CtrlVTON-base scores in isolation. The takeaway is that the model achieves consistently high TFC across all three operations on OmniTry-Bench, indicating that the token interface is reliable, while GTC remains stable across tokens—token-level control does not trade off against garment fidelity. The exact prompt we send to the model is below. Fig. 13: Qualitative results of task-token control on OmniTry-Bench. The figure is organized into three panels, one per task token: full_swap (left), partial_swap (middle), and add (right). full_swap replaces every garment of the matching class with the reference. partial_swap replaces only the targeted item while leaving the rest of the outfit intact (e.g., the inner top is swapped while the outer jacket is preserved). add keeps every existing garment in place and layers the reference on top (e.g., a blazer layered over a tee). Table 10: Task-token following on OmniTry-Bench (clothes-only subset). For each sample, we run CtrlVTON- base under the three task tokens, holding all other inputs fixed; only the task token varies between rows. TFC is the task-specific metric defined above; GTC is the same garment-fidelity metric as in Tab. 2 of the main paper. Scores are on [0, 5] scale, higher is better. Task tokenTFC↑ GTC↑ full_swap4.9234 4.1401 partial_swap 4.5872 4.1356 add4.7156 4.1287 Average4.7421 4.1348 CtrlVTON25 VLM-as-judge prompt for TFC (verbatim) [Same preamble and image definitions as the GTC/PBC/PR prompt.] You will also be given a TASK TOKEN that specifies the intended operation: [TASK_TOKEN] - one of: full_swap, partial_swap, add. Score the GENERATED_IMAGE on the following axis. The score is a float in [0, 5]. Be STRICT: 5 is reserved for outputs with no perceivable issues under detailed inspection; deduct points for any deviation from the intended operation, however small. TFC (Task-Following Consistency, [0, 5]): Does the output faithfully realize the operation specified by the TASK_TOKEN? full_swap: - Every garment of the matching class in INPUT_PERSON is replaced with the REFERENCE_GARMENT. - No original garment of that class remains. partial_swap: - Exactly one garment of the matching class is replaced with the REFERENCE_GARMENT. - All other garments, including other same-class items, remain unchanged. add: - All original garments in INPUT_PERSON are preserved. - The REFERENCE_GARMENT is added as an additional layer on top of the existing outfit. Return your answer as a single JSON object, no extra text: "TFC": <float> F.4 Mask-Adherence Metrics (IoU, d Hu , d H ) For each generated try-on image we re-extract the garment mask M gen via VIP-SAM (Sec. 3 of the main paper) and compare it against the input control mask M p under three complementary criteria—region overlap, global shape, and boundary deviation (illustrated in Fig. 19). Intersection over Union (IoU). IoU measures the overlap between the control mask and the generated garment region: IoU(M p ,M gen ) = |M p ∩ M gen | |M p ∪ M gen | .(5) Higher is better. IoU captures region-level agreement but is insensitive to localized boundary failures and shape distortions that preserve overall area. Hu moment distance d Hu . To capture global shape agreement beyond mere overlap, we adopt the definition of Hu-moment from FitControler [67]. For a binary mask I(x,y), the central moments are μ pq = X x,y (x− ̄x) p (y− ̄y) q I(x,y),(6) where ( ̄x, ̄y) is the centroid. The moments are normalized by area as η pq = μ pq μ r 00 , r = p+q 2 + 1,(7) from which the seven translation-, scale-, and rotation-invariant Hu invariants Φ(I) = [φ 1 ,...,φ 7 ] [25] can be obtained (e.g., φ 1 = η 20 + η 02 ). We then report the Euclidean distance between the invariant vectors of the control mask and the re-extracted mask: d Hu (M p ,M gen ) = Φ(M p )− Φ(M gen ) 2 .(8) Lower is better. 26S. Lee et al. Symmetric Hausdorff distance d H . Hu moments are useful for describing global shape but are insensitive to localized failures such as a sleeve protruding well beyond the requested mask. Following [27, 67], we additionally report the symmetric Hausdorff distance between contours defined by point sets A = ∂M p and B = ∂M gen : d H (A,B) = max sup a∈A inf b∈B d(a,b), sup b∈B inf a∈A d(b,a) , (9) with d(·,·) the Euclidean distance in pixels. Lower is better. Together with IoU and d Hu , this gives a balanced view of region, global shape, and worst-case boundary fidelity. F.5 Supplementary evaluation of mask-adherence metrics using SAM3 Both the garment-fidelity metrics (Appendix F.1) and the mask-adherence metrics above rely on VIP-SAM to extract garment crops and re-extract M gen from generated images. Because VIP-SAM is also used throughout the training-data pipeline (Sec. 4.2 of the main paper) to construct M p and M g ref , evaluating with VIP-SAM raises a concern of self-referential evaluation: strong scores could in principle reflect that CtrlVTON has overfit to VIP-SAM’s particular notion of a garment mask, or that some information leaks from training into the evaluation metric through the shared segmenter, rather than reflecting genuine spatial controllability. To rule this out, we re-run inference-time mask extraction and metric computation on VITON-HD-edit using an independent segmenter, SAM3 [5], prompted with a text description of the garment rather than a visual prompt—so that neither the segmenter nor the prompting modality overlaps with anything used at training time. Table 11 reports the resulting scores for all five methods on VITON-HD-edit. Compared to the VIP-SAM-based scores in Tab. 4 of the main paper, all five methods shift only slightly under SAM3, but critically, the ranking is unchanged: CtrlVTON remains far ahead of every proprietary baseline on all three spatial-control metrics, so the conclusion of Tab. 4 of the main paper is unaffected by the choice of segmenter. We also observed a small number of cases where SAM3 extracts a garment mask that differs noticeably from the corresponding VIP-SAM mask; Fig. 14 shows that in these cases, CtrlVTON’s output still closely follows whichever mask it is given. Taken together, the table and the figure indicate that CtrlVTON’s spatial controllability is not an artifact of overfitting to VIP-SAM’s specific masks, nor of train–test leakage through the evaluation metric itself. Table 11: Re-evaluation of single-garment mask-controllable VTO on VITON-HD-edit using SAM3 (text-prompted) instead of VIP-SAM for both inference-time mask extraction and metric computation. Compare against Tab. 4 of the main paper, which uses VIP-SAM throughout. MethodIoU↑d Hu ↓d H ↓ M-DINO↑ M-CLIP-I↑ Nano Banana Pro [17] 0.858 0.0049 37.12 0.82010.9043 GPT Image 1.5 [45]0.798 0.0081 55.67 0.77980.8829 Seedream 4.5 [4]0.852 0.0043 43.87 0.81020.9008 FLUX.2 [pro] [35]0.861 0.0058 40.15 0.78890.8981 CtrlVTON0.95890.003522.690.81930.8941 CtrlVTON27 Fig. 14: CtrlVTON generalizes across segmentation models. These examples illustrate the cases where VIP- SAM and SAM3 disagree on the garment mask. For each example (row), we extract the garment mask from the ground-truth image using two different segmentation models—VIP-SAM and SAM3, shown inset with a yellow border in the last two columns—and feed each mask to CtrlVTON alongside the same garment and person references. Row 1: the two masks disagree on the garment’s hemline length; CtrlVTON follows each mask faithfully, generating a correspondingly shorter or longer hem rather than a fixed length learned from training. Row 2: VIP-SAM’s mask captures the garment reference correctly, while SAM3’s text-prompted mask captures the cardigan; CtrlVTON again follows whichever mask it receives, producing markedly different outputs. In both cases, the generated output tracks the given mask rather than a shape prior learned from VIP-SAM’s masks during training, showing that CtrlVTON’s spatial controllability transfers to masks from a segmentation model it never saw at training time. 28S. Lee et al. G Additional Qualitative Results This section collects qualitative comparisons that did not fit in the main paper. Each figure compares along a specific axis; the caption gives the full context. Fig. 15: Additional single-garment VTO results. Comparisons across diverse garment types, person images, and reference representations, complementing Fig. 7. CtrlVTON29 Fig. 16: Additional multi-garment VTO results. Each row shows multiple reference garments tried on a sin- gle person, comparing CtrlVTON-base (which composes all garments in a single forward pass) with multi-garment baselines. FastFit and BootComp natively accept multiple garments, while FitDit and OmniTry are run sequentially, applying one garment at a time. 30S. Lee et al. Fig. 17: Effect of data curation on try-on quality. Comparison between a baseline (left) and our CtrlVTON- base trained on curated data (right). The baseline exhibits the silhouette-leakage artifact described in Appendix C.3: the synthesized garment’s outline follows the silhouette of the garment worn by the reference person, rather than that of the reference garment (rows 1, 3). It also produces physically implausible results, such as unnatural garment fit and body contact (row 2). CtrlVTON-base, trained on triplets that pass our three-stage quality-control funnel, resolves both issues and renders g ref faithfully. This demonstrates that a well-curated supervision set is sufficient to turn the same backbone into a high-quality VTO model. CtrlVTON31 Fig. 18: Additional mask-controllable try-on results. Each row shows multiple outputs from the same (person, garment) pair generated by varying only the input control mask, complementing Fig. 10 of the main paper. 32S. Lee et al. Fig. 19: Visualization of mask-following metrics. Each output is overlaid with the input control mask M p and the re-extracted mask M gen (obtained via VIP-SAM), with per-image IoU, d Hu , and d H printed alongside. Visually large discrepancies translate into low IoU and high d H , while subtle global-shape mismatches are reflected in d Hu . The overlays confirm that CtrlVTON respects the requested layout while proprietary baselines don’t. CtrlVTON33 Fig. 20: Representative failure cases of our two models. (a) CtrlVTON-base is guided only by the semantic tokens (garment-class, task), so it can render the reference garment in a stylistically or physically implausible way, e.g., a one-piece bodysuit left untucked over pants, or a top whose fabric appears torn at the torso as if split by the body. (b) CtrlVTON is highly responsive to the user-provided control mask, which is also its main weakness. A poorly drawn mask can cause the model to ignore the person reference and synthesize the garment in arbitrary regions, or conversely to extend the garment beyond the masked area. Output quality therefore depends directly on the quality of the input mask.

Instant research discovery