Paper deep dive
ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On
Yuhan Li, Zhiyu Jin, Yifan Tong, Wenxiang Shang, Benlei Cui, Xuanhong Chen, Hangcheng Zhu, Bingbing Ni
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Virtual footwear try-on (VFTON), a critical yet underexplored area in virtual try-on (VTON), aims to synthesize faithful try-on results given diverse footwear and model images while maintaining 3D consistency and texture authenticity. Unlike conventional garment-focused VTON methods, VFTON presents unique challenges due to (1) Data Scarcity, which arises from the difficulty of perfectly matching product shoes with models wearing the identical ones, (2) Viewpoint Misalignment, where the target foot pose and source shoe views are always misaligned, leading to incomplete texture information and detail distortion, and (3) Background-induced Color Distortion, where complex material of footwear interacts with environmental lighting, causing unintended color contamination. To address these challenges, we introduce MVShoes, a multi-view shoe try-on dataset consisting of 7305 well-annotated image triplets, covering diverse footwear categories and challenging try-on scenarios. Furthermore, we propose a dual-stream DiT architecture, ShoeFit, designed to mitigate viewpoint misalignment through Multi-View Conditioning with 3D Rotary Position Embedding, and alleviate background-induced distortion using the LayeredRefAttention which leverages background features to modulate footwear latents. The proposed framework effectively decouples shoe appearance from environmental interferences while preserving high-quality texture detail through decoupled denoising and conditioning branches. Extensive quantitative and qualitative experiments demonstrate that our method substantially improves rendering fidelity and robustness under challenging real-world product shoes, establishing a new benchmark in high-fidelity footwear try-on synthesis.
Tags
Links
- Source: https://arxiv.org/abs/2610.18325v1
- Canonical: https://arxiv.org/abs/2610.18325v1
Trouble viewing inline? Open PDF directly →
Full Text
68,647 characters extracted from source content.
Expand or collapse full text
ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On Yuhan Li Email: melodious@sjtu.edu.cnhttps://colorful-liyu.github.io/ShoeFit/ Zhiyu Jin Email: nibingbing@sjtu.edu.cnhttps://colorful-liyu.github.io/ShoeFit/ Yifan Tong Affiliation: Alibaba Group Wenxiang Shang Affiliation: Alibaba Group Benlei Cui Affiliation: Alibaba Group Xuanhong Chen, Hangcheng Zhu, Bingbing Ni Affiliation: Shanghai Jiao Tong University Affiliation: Alibaba Group Abstract Virtual footwear try-on (VFTON), a critical yet underexplored area in virtual try-on (VTON), aims to synthesize faithful try-on results given diverse footwear and model images while maintaining 3D consistency and texture authenticity. Unlike conventional garment-focused VTON methods, VFTON presents unique challenges due to (1) Data Scarcity, which arises from the difficulty of perfectly matching product shoes with models wearing the identical ones, (2) Viewpoint Misalignment, where the target foot pose and source shoe views are always misaligned, leading to incomplete texture information and detail distortion, and (3) Background-induced Color Distortion, where complex material of footwear interacts with environmental lighting, causing unintended color contamination. To address these challenges, we introduce MVShoes, a multi-view shoe try-on dataset consisting of 7305 well-annotated image triplets, covering diverse footwear categories and challenging try-on scenarios. Furthermore, we propose a dual-stream DiT architecture, ShoeFit, designed to mitigate viewpoint misalignment through Multi-View Conditioning with 3D Rotary Position Embedding, and alleviate background-induced distortion using the LayeredRefAttention which leverages background features to modulate footwear latents. The proposed framework effectively decouples shoe appearance from environmental interferences while preserving high-quality texture detail through decoupled denoising and conditioning branches. Extensive quantitative and qualitative experiments demonstrate that our method substantially improves rendering fidelity and robustness under challenging real-world product shoes, establishing a new benchmark in high-fidelity footwear try-on synthesis. † * Equal contributions. †Work done during an internship at Alibaba. ‡Corresponding author. Figure 1: VFTON presents three challenges, while ShoeFit gives a successful solution. 1 Introduction Virtual Footwear Try-On (VFTON) involves synthesizing product shoes onto human foot images, preserving their geometric structure and material properties, based on input images of both the source product and the target foot. VFTON offers significant commercial potential and academic value, with Nike’s annual footwear sales reaching 33,427 million dollars [48], while shoes, as complex 3D multi-view objects [57], present unique challenges in research on controllable generation. Unfortunately, this field has not been sufficiently explored within the community. There are three main challenges hindering the advancement of VFTON. (1) Data Scarcity: Due to the prevalence of footwear with similar appearances but subtle variations, as in Fig. 1 (a-1), such as slight differences in shoelaces, heels, and stripes, it is challenging to accurately match product shoes with models wearing the identical ones. Further exacerbating this issue, companies tend to keep data related to shoe try-ons confidential due to its clear commercial value [75, 22]. As a result, there is currently no available open-source dataset for footwear try-ons. (2) Viewpoint Misalignment: Unlike flat-lay 2D garment VTON, where both the target garment and human pose are captured from aligned front-facing perspectives, as in Fig. 1 (a-2), virtual footwear try-on inherently involves mismatched viewpoints between in-shop product images (source view) and target human poses (target view). Specifically, information needed by the target view is often not provided in the given product image due to 3D constraints, creating an information-deficient scenario that results in detail distortion and artifacts [42, 56]. (3) Background-induced Color Distortion: Unlike the diffuse reflection properties of knitted fabrics in clothing, which ensure visual color largely remains unaffected by the background, shoes often exhibit complex reflective properties that interact with ambient lighting [35], making them more susceptible to visual interference from background colors, as in Fig. 1 (a-3). This interaction between appearance and background leads to color contamination from surroundings and distorted material rendering in synthesized results. The above issues are first identified and explicitly proposed in this paper, which make it impossible for VFTON to share the same pipeline as VTON, thus presenting VFTON with unique challenges and research value. To tackle these challenges, we introduce MVShoes, the first multi-view footwear try-on dataset. It comprises 7305 annotated high-resolution image triplets, each containing two product shoe images from different views and a corresponding try-on result, as shown in Fig. 2 (a). The dataset covers a diverse range of shoe categories and try-on scenarios, supported by a comprehensive and rigorous data cleaning pipeline to ensure accurate matching of identical product shoes with their human images. Furthermore, we propose ShoeFit, a customized dual-image-stream DiT [51] architecture for VFTON, which generates high-fidelity try-on results through decoupled denoising and conditioning branches. ShoeFit is also equipped with two key improvements: (1) Multi-View Conditioning, incorporating reference concatenation [13] and corresponding 3D Rotary Position Embedding (3D RoPE). While a single source product view often fails to provide complete shoe texture features required for the target try-on view, the try-on results can significantly benefit from multiple source views, effectively reducing uncertainty in target surface rendering and minimizing texture distortion. (2)LayeredRefAttention Module. LayeredRefAttention leverages background features to modulate shoe features [31], decoupling the intrinsic shoe appearance from the surroundings. Additionally, the subsequent denoising attention suppresses background-related computations, effectively mitigating background color contamination. This layer-aware attention mechanism significantly enhances robustness in real-world footwear try-on scenarios. Our contributions are summarized as follows: • We formally define the VFTON problem, reporting the unique issues of Data Scarcity, Viewpoint Misalignment and Background-induced color distortion across real-world scenarios. • We curate the first shoe try-on dataset, containing 7305 pairs of high-quality multi-view garment-model samples, paving the way for subsequent footwear try-on studies. • We introduce a dual-stream architecture with Multi-View Conditioning and LayeredRefAttention modules to address the challenges in VFTON. • Both quantitative and qualitative experiments validate the superiority of our approach. 2 Related Works Image-based Virtual Try-on. The virtual try-on task is concerned with synthesizing images of a person donning the designated garment with appropriate fit [24], while retaining salient characteristics of the original garment and person, given a pair of images depicting a person and a target garment. To execute this task, numerous works [17, 29, 36, 4, 14, 23] have utilized Generative Adversarial Networks (GANs) [20] with two-stage strategy [37, 19, 66]: (1) warping the clothing to the desired shape [8, 40] and (2) fusing the deformed clothing via try-on generator based on GAN. As significant progress in Text-to-Image diffusion models [25, 47, 28, 11] is witnessed in recent years, some works [9, 21, 45, 32, 38] have been motivated to incorporate pre-trained diffusion models [55, 52] as priors into virtual try-on task. Most recently, OOTDiffusion [65], IDM [12], and MMTryon [75] achieve garment feature extraction with a parallel U-Net and feed them through self-attention for enhanced integration. CatVTON [13] concatenates the garment and person images along the spatial dimension and feed them into a single UNet. Unfortunately, these methods serve for fabric clothing, intrinsically lack support for faithful try-on to 3D multi-view shoes with various materials, exhibiting severe color contamination from the background. Figure 2: We illustrate examples of the MVShoes dataset in (a) and data processing pipeline in (b). Controllable Diffusion Models. To attain conditional control in diffusion models, T2I-Adapter [46] and IP-Adapter [69] incorporate additional trainable modules to fuse condition features. Additionally, some investigations utilize various prompt engineering techniques [39, 68, 73] and implement cross-attention constraints [10, 64, 30, 74] to facilitate more controllable generation. Recently, the release of FLUX [5] and SD3 [18] has sparked renewed interest in DiT architectures [51] with subsequent conditional frameworks such as Flux.1 Fill [6] and Flux.1 Redux [7] achieving inpainting and object injection capabilities through enhanced encoder designs. However, these methods remain constrained by coarse-grained conditioning mechanisms, which fail to ensure generation fidelity—particularly in VFTON tasks with high geometric variability and stringent fidelity requirements. 3 MVShoes Dataset We collect the first footwear try-on dataset which contains a total of 7305 annotated high-resolution image triplets, each consisting of two product shoe images from different views, and a corresponding try-on result. The dataset features diverse shoe categories and includes foot-specific pose landmarks and shoes-agnostic masks to facilitate precise alignment. 3.1 MVShoes Collection All images are sourced from the Internet and include a diverse range of model scenarios such as full-body models, half-body models, foot close-ups, as well as a comprehensive variety of shoe categories, effectively meeting practical application needs. Initially, we collected approximately 25,000 raw shoe product image sets, each consisting of 2 to 5 images per shoe, including standalone product images and images of people wearing the shoes from different perspectives. Data Processing. We first employ Qwen-VL [3] to differentiate between product footwear images (without humans) and try-on images featuring human models. Subsequently, we use the CLIP model [53] to extract image features and set a similarity threshold to eliminate duplicate images within the same raw image set. Next, we segment the shoe region in each image, match the DINO [43] features and assess their similarity to filter those data pairs with different footwear colors and styles. This process results in the formation of preliminary human-footwear try-on triplets, followed by manual filtering to eliminate any errors or oversights. We then remove visually blurred images and compile statistics on shoe categories and try-on scenarios. Finally, following the method proposed in DWPose [67] and Grounding DINO [43], we extract foot poses and shoe-agnostic masks. This process culminates in the creation of high-resolution, category-comprehensive try-on triplets. The data processing pipeline is illustrated in Fig. 2 (b). Figure 3: The category distribution of MVShoes. 3.2 Dataset Statistic Rich Scenes. In conventional VTON tasks [37, 15], model images typically consist of full-body or upper-body views. However, given that footwear occupies a relatively small proportion of the body in VFTON tasks, full-body and half-body models are less effective in showcasing shoe details. To address this, we simulate the distribution of footwear model scenes observed on e-commerce platforms and curate 4 model scenarios: top-down foot close-up, horizontal foot close-up, half-body model, and full-body model, as illustrated in Fig. 3, where horizontal foot close-ups constitute the largest proportion. In such cases, accurate preservation of side details is particularly challenging, especially when compounded by viewpoint misalignment. Diverse Footwear Categories. We also prioritize the comprehensiveness of footwear categories. By annotating shoe attributes [2], we establish a two-level product category label set and ensure that its distribution aligns with everyday use, as depicted in Fig. 3. Notably, we observe that environmental lighting often affects the visual appearance of footwear, where reflections introduce background color contamination, particularly in categories such as boots, leather shoes, and high heels. 4 Method Figure 4: We illustrate existing and our DiT framework comparison in (a)-(b), and heatmaps visualization for component validation of Multi-view Conditioning and LayeredRefAttention in (c)-(d). An overview of the ShoeFit is presented in Fig. 4 (b). Given a human image X and target shoe images CI(n)C_I^(n), ShoeFit is aimed to generate an authentic try-on image. The backbone of ShoeFit employs the FLUX.1-Lite [5], with a customized dual-image-stream architecture in Sec. 4.1: denoising image stream and a copy of it as as conditioning branch to replace text modality. The features of human images and multi-view shoes are then integrated through Reference Concatenation and LayeredRefAttention modules, which are described in Sec. 4.2 and 4.3. 4.1 Dual-image-stream DiT Framework The original FLUX [5] is a text-to-image model composed of a series of stacked M-DiT blocks. Earlier works [60, 61] directly concatenate Image/Text condition tokens with noisy image tokens such as [X;CI;CT][X;C_I;C_T], relying on a single denoising stream to finish condition extraction and denoising, while the other deal with text modality, as shown in Fig. 4(a). However, for VFTON, the generated try-ons are primarily determined by the given shoe images rather than limited semantic prompts. Thus, we remove the text stream from FLUX, achieving approximately 40% parameter savings, including those for T5 [54] and CLIP models [53]. Instead, we replicate a portion of the denoising stream’s architecture and weights to initialize the conditioning image stream, establishing a dual-image-stream framework. This design decouples feature preservation of shoe details from the high-quality generation process, allocating additional effective model capacity to enhance overall fidelity performance compared to direct token concatenation approaches [60, 61]. 4.2 Multi-view Conditioning In VFTON, the information about shoes from the try-on view remains incomplete due to view mismatch and 3D-to-2D projection occlusion [63]. The inherent misalignment between source product images and target human images leads to unfaithful and physically implausible try-on results. Observing that e-commerce platforms often provide multiple-view product images (e.g., front, side, top views), we introduce a multi-view shoe image dataset and propose a multi-view conditional injection mechanism via Reference Concatenation and 3D RoPE. The additional perspectives comprehensively encompass more aspects of the shoe, as illustrated in Fig. 4 (c), significantly reducing uncertainty during generation and delivering high-fidelity and commercially viable synthetic results. Reference Concatenation. Inspired by CatVTON [13], we adopt a simple yet effective multi-view injection strategy: concatenate multi-view conditioning shoe images along the spatial dimension as: CIcat=[CI1,CI2,…CIN]∈ℝb×N⋅l×cC_I^cat=[C_I^1,C_I^2,...C_I^N] ^b× N· l× c, where b s the batch size, l denotes the number of image tokens per conditioning image, and c is the feature channel. The concatenated multi-view latent CIcatC_I^cat is then fed into the conditioning image stream for feature extraction and interaction with the target human image. Concurrently, the denoising stream receives 4 concatenated components with 16+16+1+1616+16+1+16 channels: the noisy human latents XtX_t, the latent shoe-agnostic masked image E(Xmasked)E(X_masked), the resized agnostic mask XmX_m, and the foot pose landmarks E(Xpose)E(X_pose), where E(⋅)E(·) represents VAE [33] encoding. The effectiveness of Reference Concatenation stems from the global attention interactions in attention layers. Intuitively, during the attention, each Query patch in the human image accesses and matches the most semantically relevant Key-Value pairs from the full set of concatenated multi-view product features, as illustrated in Fig. 4 (c). As a result, our method ensures consistent alignment between all regions in the target pose and their corresponding details in the product images, thereby achieving high-fidelity virtual try-on. 3D RoPE. However, naive Reference Concatenation introduces position misalignment in the relative positional relation of conditioning images. FLUX employs an essentially 2D RoPE scheme [58] to encode spatial relative relationships along the height and width dimensions, where the inner product in attention depends solely on the token embeddings xm,xnx_m,x_n and their relative distances m−nm-n: ⟨fq(xm,m),fk(xn,n)⟩=g(xm,xn,m−n). f_q(x_m,m),f_k(x_n,n) =g(x_m,x_n,m-n). (1) When multi-view conditioning images are naively concatenated, discontinuous semantic jumps emerge at the concatenation boundaries, where tokens with small relative distances may correspond to semantically distant regions (i.e., different views of the footwear). To address this issue, we extend the 2D RoPE to three dimensions. Specifically, EposID=concat[EposIDn;EposIDh;EposIDw]∈ℝH×W×3E_posID=concat[E_posID^n;E_posID^h;E_posID^w] ^H× W× 3, where the first channel EposIDnE_posID^n serves as a index mask to separate different conditioning views, while the second and third channels EposIDh,wE_posID^\h,w\ encode spatial positions within each image along the height and width dimensions, respectively. Within each channel of EposIDn,h,wE_posID^\n,h,w\, frequency encoding is performed given a base frequency of θ=10000θ=10000. ωd=1θ2d/D,for d=0,1,…,D2−1,vn,h,w=EposIDn,h,w⋅ωd∈ℝH×W×Dn,Dh,Dw2, _d= 1θ^2d/D,for d=0,1,..., D2-1, v_ \n,h,w \=E_posID \n,h,w \· _d ^H× W× \D_n,D_h,D_w \2, (2) where hype-parameters Dn,Dh,DwD_n,D_h,D_w is predefined as 16,56,5616,56,56. Subsequently, we obtain positional rotation matrices R1DR_1D for each of the n,h,wn,h,w channels internally. By concatenating these matrices along D dimension, we construct 3D RoPE matrix R3D∈ℝH×W×(Dn+Dh+Dw)2×2×2R_3D ^H× W× (D_n+D_h+D_w)2× 2× 2 as follow: R1Dn,h,w=[cos(vn,h,wCLOSE−sin(vn,h,w)sin(vn,h,w)cos(vn,h,w)],R3D=concat[R1Dn;R1Dh;R1Dw]R_1D \n,h,w \= bmatrixcos(v_ \n,h,w \&-sin(v_ \n,h,w \)\\ sin(v_ \n,h,w \)&cos(v_ \n,h,w \) bmatrix, R_3D=concat[R_1D^n;R_1D^h;R_1D^w] (3) 4.3 LayeredRefAttention Module We observe that the appearance of the synthesized shoes is highly susceptible to contamination from the background colors of the conditioning images. This phenomenon arises from the complex reflective material of footwear interacting strongly with environmental lighting, termed Background-induced Color Distortion. Specifically, the issue arises when Query patches from the human image inadvertently match key-value pairs originating from the background regions of the conditioning image, as illustrated in Fig. 4 (d). A naive solution involves masking out background regions during attention, but this would neglect critical illumination cues embedded in the background, which are essential for accurately inferring the shoe’s intrinsic appearance and texture [35]. To address this challenge, we propose a layer-aware attention module called LayeredRefAttention to explicitly differentiate between background and foreground content in the conditioning images, as illustrated in Fig. 4 (b). By leveraging background features to modulate foreground shoe features, our method decouples the intrinsic shoe appearance from its background context [31]. Subsequent denoising stream attention computations are restricted to interactions between the foreground shoe content and the target human image, effectively eliminating background illumination effects while avoiding background color contamination. Specifically, for n∈1,…,Nn∈\1,...,N\ conditioning view CI(n)C_I^(n), we employ a Squeeze-and-Excitation (SE) block [27] followed by global average pooling over spatial dimensions to compute channel-specific weights: P(n)=AvgPool(SE(CI(n))∈ℝb×cCLOSE.P^(n)=AvgPool(SE(C_I^(n)) ^b× c. (4) Subsequently, we employ two linear layers F(⋅)F(·) to extract background modulation parameters and modulate foreground shoe features, filtering irrelevant environmental lighting and background reflection to ensure faithful material preservation of shoes as: βscale(n),βshift(n)=F(P(n)),Cfg(n)=(1+βscale(n))⋅(LN(CI(n)⊙Mfg(n)))+βshift(n), _scale^(n), _shift^(n)=F(P^(n)), C_fg^(n)=(1+ _scale^(n))·(LN(C_I^(n) M_fg^(n)))+ _shift^(n), (5) where LN(⋅)LN(·) means layer normalization and Mfg(n)M_fg^(n) represents the binary shoe masks for conditioning product image. The attention variables K,Vc,fg(n)\K,V\_c,fg^(n) extracted from N views of modulated foreground conditioning feature are concatenated with the corresponding Q,K,Vimg\Q,K,V\_img from human image for subsequent denosing-stream attention: QC,fg(n),KC,fg(n),VC,fg(n)=QKV(Cfg(n)),Q_C,fg^(n),K_C,fg^(n),V_C,fg^(n)=QKV(C_fg^(n)), (6) Attnimage=softmax(Qimg⋅concat(Kimg,KC,fg1,…,N,⊤)d)⋅concat(Vimg,VC,fg1,…,N).Attn_image=softmax( Q_img·concat(K_img;K_C,fg^1,...,N, ) d)·concat(V_img;V_C,fg^1,...,N). (7) Meanwhile, the conditioning-stream branch performs attention computation using unmasked conditioning latents as: QC(n),KC(n),VC(n)=QKV(LN(CI(n))),AttnC=softmax(QC1,…,N⋅KC1,…,N,⊤d)⋅VC1,…,N.Q_C^(n),K_C^(n),V_C^(n)=QKV(LN(C_I^(n))), Attn_C=softmax( Q_C^1,...,N· K_C^1,...,N, d)· V_C^1,...,N. (8) To clarify our contributions, we omit the distillation strength normalization and final residual connections in the attention layer of FLUX.1-Lite [5] that remain unchanged from the original implementation. In summary, LayeredRefAttention effectively enables material-background decoupling by modulating foreground shoe features to recover intrinsic color properties, resulting in texture-preserving footwear try-on synthesis, as illustrated in Fig. 4 (d). 5 Experiments Figure 5: Comparisons on the hard scenes, where ShoeFit delivers high-fidelity and rubost results. Table 1: Quantitative comparison on MVshoes. ShoeFit significantly surpasses all baselines. View Num. Method SSIM ↑ LPIPS ↓ FID ↓ KID ↓ DISTS ↓ 1-View FLUX.1-Fill [6] 0.725 0.201 24.44 2.221 0.1293 FLUX.1-Fill-Redux [7] 0.724 0.199 24.67 2.086 0.1274 OOTDiffusion-SDXL [65] 0.724 0.194 22.60 2.345 0.1114 CatVTON-Flux-Lite [13] 0.745 0.185 22.71 3.327 0.1179 ShoeFit-1V 0.772 0.159 21.07 0.944 0.1059 2-View CatVTON-Flux-Lite-2V [13] 0.756 0.171 22.64 3.318 0.1040 ShoeFit-2V 0.780 0.149 20.18 0.485 0.0941 Figure 6: Ablation study. We highlight the improvements by all three components of method in red. 5.1 Experimental Setup Implementation Details. We employ the FLUX.1-Lite [5] as the backbone and customize a dual-image-stream architecture as in Sec. 4.1. Our experiments are carried out on MVShoes at a resolution of 768×768768× 768, with 6305 pairs for training and 1000 pairs for testing. The model is trained for 77 days on 88 80GB-A100 GPUs with DeepSpeed [1] ZeRO-2 to reduce memory usage, at a batch size of 44. All parameters of DiT are trainable, using an AdamW optimizer [44] with a constant learning rate of 3e−53e-5. At inference time, we run ShoeFit on a single A10 GPU for 2525 steps. The data augmentation follows the same protocol as in StableVTON [32]. Please refer to Appendix for more details. Evaluation Protocols. Following evaluation in prior VTON works [45], we measure reconstruction accuracy by LPIPS [72], SSIM [62] and DISTS [16] in a paired setting given ground truth images, and authenticity of unpaired synthesized images by FID [50] and KID [59] without ground truth. Baselines. As a subtask of inpainting, we first select two widely used inpainting FLUX variants as baselines: FLUX.1-Fill [6] and FLUX.1-Fill-Redux [7]. The former involves concatenating product images with human images on the spatial dimension, similar to CatVTON [13]. The latter enhances the conditioning stream with SigLip [70] to incorporate product shoe images. Additionally, we compare ShoeFit to two state-of-the-art methods in the VTON field: OOTDiffusion [65] based on the SDXL [52] and CatVTON [13] built upon the FLUX.1-Lite. All models are implemented as originally deployed for single-view product shoe conditional generation. Considering CatVTON’s capability to easily extend to multi-image injection, we further train a version of CatVTON to simultaneously inject two product views, the same as ShoeFit, denoted as CatVTON-FLUX-Lite-2V. Please refer to Appendix for more implementation details. 5.2 Qualitative Results Fig. 5 provides a qualitative comparison between ShoeFit and the baselines on MVShoes, addressing challenges such as complex structures, background interference, and difficult target poses. For a fair comparison, we report the results of the single-view conditioning version of all methods. Notably, we observe that OOTDiffusion [65] tends to exhibit extreme color variations, while FLUX.1-Fill-Redux [7] and FLUX.1-Fill [6], possibly affected by the limitation of only LoRA [26] being trainable (r=64), often produce visually plausible results that do not strictly adhere to the input conditions. In contrast, ShoeFit accurately generates structural details of the shoes (rows 1 and 2), decouples environmental lighting (row 3), avoids background confusion (row 4), and demonstrates robustness to uncommon poses (row 5). Please refer to the Appendix for more results. 5.3 Quantitative Results For a fair comparison, we report the results of one/two product view conditioning generation separately in Tab. 1. FLUX.1-Fill-Redux [7] and FLUX.1-Fill [6] achieve the worst FID and DISTS scores, while OOTDiffusion [65] and CatVTON [13] show complementary performance in unpaired and paired evaluations. ShoesFit, benefiting from the additional specialized parameters provided by the Dual-image-stream DiT framework and background decoupling enabled by the LayeredRefAttention, significantly surpasses all baselines under both single-view and multi-view conditional generation. 5.4 Ablation Study To validate our technical contributions, we define a Vanilla Model that uses FLUX.1-Lite as the backbone, deploying both a text-stream and an image-stream as in Fig. 4 (a). Multi-view Conditioning and LayeredRefAttention Module are removed, using only one footwear image as conditioning. Dual-image-stream DiT Framework. Since the text-stream only provides high-level semantics, its utility is limited in VFTON, which demands high image fidelity. Therefore, we first remove the text-stream and replicate a portion of the denoising stream’s architecture and weights to initialize the conditioning image stream, effectively decoupling the conditional feature extraction from the denoising process, while keeping the Vanilla Model’s other settings unchanged(denoted as “+ D-i-s”). As shown in Tab. 2 and Fig. 6, the vanilla model often generates artifacts and incorrect structures facing high-frequency textures due to the limitation in effective network capacity and the weakness in coupling of conditional images and noise. This demonstrates that the Dual-image-stream DiT framework plays a fundamental role in enhancing the model’s expressive capabilities. LayeredRefAttention Module. Observing the visual contamination from background and environmental lighting, we further integrate the LayeredRefAttention into the “+ D-i-s” version, denoted as “+ D-i-s + LRA”, which effectively decouples the intrinsic appearance of the shoe from its surroundings. As illustrated in Tab. 2 and Fig. 6, the model without LayeredRefAttention often fails to accurately reproduce the natural colors of the shoe where environmental lighting or shadows of product images are prominent, resulting in outputs that are mixed with environmental light or shadows. Additionally, mismatches with background patches in Attention can introduce colors confusion from the background content, such as slippers being contaminated by floor colors in Column 3 of Fig. 6. In contrast, the one with "LRA" effectively avoids such color contamination, delivering superior try-on results. To investigate the efficacy of simple background masking, we conducted an ablation study utilizing standard attention while removing the background from the product shoe images. Visually, this naive approach exacerbates color artifacts, particularly for highly saturated hues. We hypothesize that this occurs because removing the background discards crucial illumination cues embedded in the product image, which are essential for the model to infer the shoe’s intrinsic material properties and texture. Given that the training set contains inherent color variations between in-studio product shots and on-foot images due to disparate lighting, forcing the model to operate without background context introduces lighting ambiguity. Consequently, the model struggles to maintain color constancy, leading to degraded try-on performance and reduced visual fidelity. Multi-view Conditioning System. While a single source product view often fails to capture the complete texture of footwear, try-on results can benefit from multiple source views, thereby effectively reducing uncertainty in generation. Thus, we introduce the Multi-view Conditioning pipeline using Reference Concatenation and 3D RoPE in the “+ D-i-s + LRA” version, denoted as “+ D-i-s + LRA + MC”. As demonstrated in Tab. 2 and Fig. 6, the model without “MC”, which uses only one product shoe image as input (yellow box), often generates unrealistic speculations and artifacts in unseen areas due to insufficient conditional information. In contrast, ShoeFit benefits from the Multi-view Conditioning system, producing details that are more faithful to the input across different views. Furthermore, we also conduct an ablation study upon 3D RoPE’s impact in Tab. 2, demonstrating the performance gain we achieve using the proposed RoPE without incurring any additional computational overhead. Please refer to the Appendix for more ablation results. Table 2: Quantitative ablation study of each component in Section of Method. Method SSIM ↑ LPIPS ↓ FID ↓ KID ↓ DISTS ↓ Vanilla Model 0.745 0.185 22.71 3.327 0.1179 + D-i-s 0.761 0.174 21.57 1.493 0.1038 + D-i-s + Standard Attn (w/ mask) 0.756 0.181 21.70 1.698 0.1056 + D-i-s + LRA (w/ mask) 0.772 0.159 21.07 0.944 0.1059 + D-i-s + LRA + MC (w/o 3D RoPE) 0.778 0.152 20.18 0.695 0.0971 + D-i-s + LRA + MC (Full) 0.780 0.149 20.18 0.485 0.0941 5.5 Inference Time and Memory Usage We evaluated the efficiency of our proposed enhancements by measuring inference time and memory footprint on an NVIDIA H20 GPU (with batch size of 1, 25 steps, and torch.bfloat16 precision). The results, presented in Table 3, reveal that the ShoeFit model with our Dual-image-stream (D-i-s) architecture substantially outperforms CAT-VTON in both speed and memory efficiency. This stems from removing the text stream and its computationally expensive T5 and CLIP encoders. Table 3: Comparison of inference time and GPU memory usage across different architectures. Method Inference Time per Image GPU Memory Usage Cat-VTON (image + text streams) 22.27 seconds 35.5 GB ShoeFit (Dual-image-stream, D-i-s) 17.44 seconds 22.0 GB ShoeFit + D-i-s + LayeredRefAttention 21.15 seconds 28.5 GB 6 Conclusion In conclusion, we formally define the VFTON problem and curate the first shoe try-on dataset, MVShoes, containing 7305 pairs of high-quality multi-view garment-model samples. Our proposed framework addresses the critical challenges of viewpoint misalignment and background-induced color distortion by leveraging a dual-stream architecture with Multi-View Conditioning and LayeredRefAttention modules. Through a comprehensive evaluation on the proposed MVShoes dataset, we demonstrate significant improvements in detail fidelity and texture consistency across diverse scenarios. This indicates that our method effectively mitigates background interference while preserving fine-grained shoe details across views, establishing a strong baseline for future VFTON research. Broader impacts. With the ability to synthesize images, arises the risk that ShoeFit might be used for inappropriate purposes such as producing media that breaches intellectual property rights or privacy norms. Because of these risks, we strongly advocate for the conscientious use of this technology. Limitation and future work. ShoeFit demonstrates substantial improvements in generating high-fidelity footwear try-on images across diverse scenes. However, certain limitations persist. Firstly, the model occasionally struggles with accurately rendering small logos and intricate text, particularly critical for e-commerce sellers. Secondly, we hope to introduce explicit 3D geometric and material priors in the future to achieve more robust multi-view representation and refined visual fidelity. Acknowledgements This work is supported by the Science and Technology Commission of Shanghai Municipality under research grant No. 25ZR1401187. References [1] DeepSpeed. Note: https://github.com/microsoft/DeepSpeed Cited by: §5.1. [2] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §3.2. [3] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou (2023) Qwen-vl: a versatile vision-language model for understanding, localization, text reading, and beyond. External Links: 2308.12966, Link Cited by: 1st item, Appendix B, §3.1. [4] S. Bai, H. Zhou, Z. Li, C. Zhou, and H. Yang (2022) Single stage virtual try-on via deformable attention flows. In European Conference on Computer Vision, Cited by: §2. [5] Black-forest-labs (2024) FLUX.1-dev. Note: https://github.com/black-forest-labs/flux External Links: Link Cited by: 3rd item, §C.1, §2, §4.1, §4.3, §4, §5.1. [6] Black-forest-labs (2024) FLUX.1-fill-dev. Note: https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev External Links: Link Cited by: §D.1, §2, §5.1, §5.2, §5.3, Table 1. [7] Black-forest-labs (2024) FLUX.1-redux-dev. Note: https://huggingface.co/black-forest-labs/FLUX.1-Redux-dev External Links: Link Cited by: §D.1, §2, §5.1, §5.2, §5.3, Table 1. [8] F. L. Bookstein (1989) Principal warps: thin-plate splines and the decomposition of deformations. IEEE Transactions on pattern analysis and machine intelligence. Cited by: §2. [9] M. Chen, X. Chen, Z. Zhai, C. Ju, X. Hong, J. Lan, and S. Xiao (2024) Wear-any-way: manipulable virtual try-on via sparse correspondence alignment. arXiv preprint. Cited by: §2. [10] M. Chen, I. Laina, and A. Vedaldi (2024) Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 5343–5353. Cited by: §2. [11] X. Chen, L. Huang, Y. Liu, Y. Shen, D. Zhao, and H. Zhao (2023) Anydoor: zero-shot object-level image customization. arXiv preprint arXiv:2307.09481. Cited by: §2. [12] Y. Choi, S. Kwak, K. Lee, H. Choi, and J. Shin (2024) Improving diffusion models for virtual try-on. arXiv preprint arXiv:2403.05139. Cited by: §2. [13] Z. Chong, X. Dong, H. Li, S. Zhang, W. Zhang, X. Zhang, H. Zhao, D. Jiang, and X. Liang (2024) Catvton: concatenation is all you need for virtual try-on with diffusion models. arXiv preprint arXiv:2407.15886. Cited by: §1, §2, §4.2, §5.1, §5.3, Table 1, Table 1. [14] A. Chopra, R. Jain, M. Hemani, and B. Krishnamurthy (2021) Zflow: gated appearance flow-based virtual try-on with 3d priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Cited by: §2. [15] M. Davide, F. Matteo, C. Marcella, L. Federico, C. Fabio, and C. Rita (2022) Dress code: high-resolution multi-category virtual try-on. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §3.2. [16] K. Ding, K. Ma, S. Wang, and E. P. Simoncelli (2020) Image quality assessment: unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence 44 (5), p. 2567–2581. Cited by: §5.1. [17] H. Dong, X. Liang, Y. Zhang, X. Zhang, X. Shen, Z. Xie, B. Wu, and J. Yin (2020) Fashion editing with adversarial parsing learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 8120–8128. Cited by: §2. [18] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. (2024) Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: §2. [19] Y. Ge, Y. Song, R. Zhang, C. Ge, W. Liu, and P. Luo (2021) Parser-free virtual try-on via distilling appearance flows. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Cited by: §2. [20] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. Advances in neural information processing systems. Cited by: §2. [21] J. Gou, S. Sun, J. Zhang, J. Si, C. Qian, and L. Zhang (2023) Taming the power of diffusion models for high-quality virtual try-on with appearance flow. In Proceedings of the 31st ACM International Conference on Multimedia, Cited by: §2. [22] H. Guo, B. Zeng, Y. Song, W. Zhang, C. Zhang, and J. Liu (2025) Any2AnyTryon: leveraging adaptive position embeddings for versatile virtual clothing tasks. arXiv preprint arXiv:2501.15891. Cited by: §1. [23] X. Han, X. Hu, W. Huang, and M. R. Scott (2019) Clothflow: a flow-based model for clothed person generation. In Proceedings of the IEEE/CVF international conference on computer vision, Cited by: §2. [24] X. Han, Z. Wu, Z. Wu, R. Yu, and L. S. Davis (2018) Viton: an image-based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §2. [25] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems. Cited by: §2. [26] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §D.1, §5.2. [27] J. Hu, L. Shen, and G. Sun (2018) Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 7132–7141. Cited by: §C.2, §4.3. [28] L. Huang, D. Chen, Y. Liu, Y. Shen, D. Zhao, and J. Zhou (2023) Composer: creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778. Cited by: §2. [29] Y. Jo and J. Park (2019) Sc-fegan: face editing generative adversarial network with user’s sketch and color. In Proceedings of the IEEE/CVF international conference on computer vision, p. 1745–1753. Cited by: §2. [30] X. Ju, A. Zeng, C. Zhao, J. Wang, L. Zhang, and Q. Xu (2023) Humansd: a native skeleton-guided diffusion model for human image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 15988–15998. Cited by: §2. [31] T. Karras, S. Laine, and T. Aila (2019) A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 4401–4410. Cited by: §1, §4.3. [32] J. Kim, G. Gu, M. Park, S. Park, and J. Choo (2024) StableVITON: learning semantic correspondence with latent diffusion model for virtual try-on. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §2, §5.1. [33] D. P. Kingma and M. Welling (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §C.1, §4.2. [34] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, et al. (2023) Segment anything. arXiv preprint arXiv:2304.02643. Cited by: 2nd item, Appendix B. [35] P. Kocsis, J. Philip, K. Sunkavalli, M. Nießner, and Y. Hold-Geoffroy (2024) Lightit: illumination modeling and control for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9359–9369. Cited by: §1, §4.3. [36] C. Lee, Z. Liu, L. Wu, and P. Luo (2020) Maskgan: towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5549–5558. Cited by: §2. [37] S. Lee, G. Gu, S. Park, S. Choi, and J. Choo (2022) High-resolution virtual try-on with misalignment and occlusion-handled conditions. In European Conference on Computer Vision, Cited by: §2, §3.2. [38] Y. Li, H. Zhou, W. Shang, R. Lin, X. Chen, and B. Ni (2024) AnyFit: controllable virtual try-on for any combination of attire across any scenario. arXiv preprint arXiv:2405.18172. Cited by: §2. [39] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee (2023) Gligen: open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22511–22521. Cited by: §2. [40] T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie (2017) Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §2. [41] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2022) Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §C.1, §C.1. [42] M. Liu, C. Xu, H. Jin, L. Chen, M. Varma T, Z. Xu, and H. Su (2023) One-2-3-45: any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems 36, p. 22226–22246. Cited by: §1. [43] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang (2024) Grounding dino: marrying dino with grounded pre-training for open-set object detection. External Links: 2303.05499, Link Cited by: 6th item, Appendix B, §3.1. [44] I. Loshchilov and F. Hutter (2018) Fixing weight decay regularization in adam. Cited by: §5.1. [45] D. Morelli, A. Baldrati, G. Cartella, M. Cornia, M. Bertini, and R. Cucchiara (2023) LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On. In Proceedings of the ACM International Conference on Multimedia, Cited by: §2, §5.1. [46] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, and Y. Shan (2024) T2i-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 4296–4304. Cited by: §2. [47] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen (2021) Glide: towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741. Cited by: §2. [48] INC. NIKE (2024) NIKE, inc. annual report. Note: https://w.sec.gov/Archives/edgar/data/320187/000032018724000044/nke-20240531.htm External Links: Link Cited by: §1. [49] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P. Huang, H. Xu, V. Sharma, S. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski (2023) DINOv2: learning robust visual features without supervision. Cited by: 4th item, Appendix B. [50] G. Parmar, R. Zhang, and J. Zhu (2022) On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §5.1. [51] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4195–4205. Cited by: §C.1, §1, §2. [52] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach (2023) Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. Cited by: §2, §5.1. [53] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, Cited by: 3rd item, Appendix B, §C.1, §3.1, §4.1. [54] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2023) Exploring the limits of transfer learning with a unified text-to-text transformer. External Links: 1910.10683, Link Cited by: §4.1. [55] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Cited by: §C.1, §2. [56] R. Shi, H. Chen, Z. Zhang, M. Liu, C. Xu, X. Wei, L. Chen, C. Zeng, and H. Su (2023) Zero123++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110. Cited by: §1. [57] Y. Shi, P. Wang, J. Ye, M. Long, K. Li, and X. Yang (2023) Mvdream: multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512. Cited by: §1. [58] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §C.1, §4.2. [59] J. Sutherland, M. Arbel, and A. Gretton (2018) Demystifying mmd gans. In International Conference for Learning Representations, Cited by: §5.1. [60] Z. Tan, S. Liu, X. Yang, Q. Xue, and X. Wang (2024) Ominicontrol: minimal and universal control for diffusion transformer. arXiv preprint arXiv:2411.15098. Cited by: §4.1. [61] H. Wang, J. Peng, Q. He, H. Yang, Y. Jin, J. Wu, X. Hu, Y. Pan, Z. Gan, M. Chi, et al. (2025) UniCombine: unified multi-conditional combination with diffusion transformer. arXiv preprint arXiv:2503.09277. Cited by: §4.1. [62] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. Cited by: §5.1. [63] Wikipedia (2025) 3D projection. Note: https://en.wikipedia.org/wiki/3D_projection External Links: Link Cited by: §4.2. [64] J. Xie, Y. Li, Y. Huang, H. Liu, W. Zhang, Y. Zheng, and M. Z. Shou (2023) Boxdiff: text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 7452–7461. Cited by: §2. [65] Y. Xu, T. Gu, W. Chen, and C. Chen (2024) OOTDiffusion: outfitting fusion based latent diffusion for controllable virtual try-on. arXiv preprint arXiv:2403.01779. Cited by: §D.1, §2, §5.1, §5.2, §5.3, Table 1. [66] H. Yang, R. Zhang, X. Guo, W. Liu, W. Zuo, and P. Luo (2020) Towards photo-realistic virtual try-on by adaptively generating-preserving image content. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Cited by: §2. [67] Z. Yang, A. Zeng, C. Yuan, and Y. Li (2023) Effective whole-body pose estimation with two-stages distillation. External Links: 2307.15880, Link Cited by: 5th item, Appendix B, §3.1. [68] Z. Yang, J. Wang, Z. Gan, L. Li, K. Lin, C. Wu, N. Duan, Z. Liu, C. Liu, M. Zeng, et al. (2023) Reco: region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14246–14255. Cited by: §2. [69] H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang (2023) Ip-adapter: text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721. Cited by: §2. [70] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. External Links: 2303.15343, Link Cited by: §D.1, §5.1. [71] L. Zhang, A. Rao, and M. Agrawala (2023) Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3836–3847. Cited by: §D.1. [72] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, Cited by: §5.1. [73] T. Zhang, Y. Zhang, V. Vineet, N. Joshi, and X. Wang (2023) Controllable text-to-image generation with gpt-4. arXiv preprint arXiv:2305.18583. Cited by: §2. [74] X. Zhang, D. Song, P. Zhan, Q. Chen, Z. Xu, W. Luo, K. Zhang, and A. Liu (2024) BooW-vton: boosting in-the-wild virtual try-on via mask-free pseudo data training. arXiv preprint arXiv:2408.06047. Cited by: §2. [75] X. Zhang, E. Lin, X. Li, Y. Luo, M. Kampffmeyer, X. Dong, and X. Liang (2024) MMTryon: multi-modal multi-reference control for high-quality fashion generation. arXiv preprint arXiv:2405.00448. Cited by: §1, §2. • In Section A, we provide a more detailed discussion about the limitations of ShoeFit. • In Section B, we provide additional details about the MVShoes datasets, including data processing and data statistics. • In Section C, we provide preliminary knowledge about the FLUX [5] model, as well as how LayeredRefAttention modules are applied in the Single Stream DiT Block. • In Section D, we provide additional details about the experimental setup, including baseline training, data augmentation, and hyperparameters. • In Section E, we present a multitude of ShoeFit generated images, including more ablation results, more comparisons with the baselines, with additional results displayed in challenging scenarios. Figure 7: Similar to the common constraints faced by existing generative models, ShoeFit occasionally struggles to accurately render small logos and intricate text due to their small portion in images and high-frequency variations. Appendix A Limitations and Future Work We propose ShoeFit, a dual-stream DiT framework that addresses the critical challenges of viewpoint misalignment and background-induced color distortion in VFTON by Multi-View Conditioning and LayeredRefAttention modules. However, certain limitations persist. Firstly, similar to the common constraints faced by existing generative models, the model occasionally struggles to accurately render small logos and intricate text due to their small portion in images and high-frequency variations. We illustrate this limitation in Fig. 7. These small logos are particularly critical for e-commerce sellers as they often convey brand information that consumers care about. Therefore, we plan to develop a detailed supplementary conditioning method in future research to pre-detect and enhance the injection of such patterns, fundamentally addressing this issue. Secondly, we aim to introduce explicit 3D geometric and material priors in the future to achieve more robust multi-view representation and refined visual fidelity. Appendix B MVShoes Dataset Supplement In the main body of the paper, we provide a brief description of the data processing pipeline and a rough visual presentation of the dataset statistics in terms of Rich Scenes and Diverse Footwear Categories due to space constraints. In this section, we offer more detailed information on the data processing pipeline to ensure reader comprehension. Additionally, we present the specific distribution statistics of MVShoes in a quantitative table format. Data Processing Details. Given a dataset comprising raw images of various shoes and human models, we employ Qwen2.5-VL [3], which operates with 7 billion parameters, to distinguish between footwear images and human model images. The prompting framework utilized is as follows: “Analyze the provided image and determine whether it depicts a model image or a shoe image. A model image is defined as one that portrays a human model’s lower body or legs adorned with shoes, while a shoe image solely comprises images of shoes without any human presence. Assign a value of ‘1’ if it qualifies as a model image and ‘0’ if it does not. ” Following this classification, we extract image features from the shoe images using the CLIP [53] model, implementing a similarity threshold of 0.9 to effectively eliminate duplicate images within the dataset. Subsequently, we perform segmentation by SAM [34] to isolate the shoe regions in each image, aligning the DINO [49] features to filter out shoes-model data pairs exhibiting inner similarities greater than 0.8. This procedure facilitates the initial construction of human-footwear try-on triplets, which are then subjected to manual filtering to address any potential errors or oversights. Additionally, we exclude visually blurred images and compile statistical analyses pertaining to shoe categories and try-on scenarios. In accordance with methodologies established in DWPose [67] and GroundingDINO [43], we extract foot poses and shoe masks, ultimately resulting in the generation of high-resolution, category-comprehensive try-on triplets. All models referenced above are open-source, and their respective URLs are provided as follows: • Qwen2.5-VL-7B [3]: https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct • SAM [34]: https://github.com/facebookresearch/segment-anything • CLIP [53]: https://github.com/openai/CLIP • DINO [49]: https://github.com/facebookresearch/dinov2 • DWPose [67]: https://github.com/IDEA-Research/DWPose • GroundingDINO [43]: https://github.com/IDEA-Research/GroundingDINO Dataset Statistics. The dataset features diverse shoe categories and rich human scenes. We present the quantitative distribution statistics of MVShoes in Tab. 4 and Tab. 5. We also provide more visual samples from MVShoes datasets in Fig. 8. Pose Estimation for Shoe Try-On. Traditional open-source pose estimation methods (e.g., OpenPose, DWPose) in VITON pipeline, struggle to capture lower-body/foot poses in VFTON, which often only include partial leg shots or full lower-body views. To address this, we fine-tune the dwpose model on specially collected datasets containing diverse lower-body and close-up leg poses. This enables our model to generate accurate pose descriptions for various leg postures, significantly enhancing its generalization capability for shoe-related applications. Generalizable Foot Mask Design. Given the diversity of shoe types (e.g., sneakers, boots, sandals) and their varying leg coverage, we propose a leg-pose-aware mask generation strategy. Unlike VTON’s body-centric masks, our method combines leg pose information with bounding box expansion from shoe detection to create adaptive masks. This allows seamless cross-shoe category try-on capabilities, for example, a model wearing long boots or flip-flops can be transformed into sneakers while maintaining anatomical alignment and realistic occlusion. As illustrated in the first row in Fig. 11, this approach lays the foundation for robust, universal shoe try-on systems. However, our mask generation strategy is not without its limitations. A failure case is presented in the second row of Fig. 5(b). Here, the region of the rear shoe is both heavily occluded and minimally visible, which leads to the failure of our keypoint and shoe detection modules. Consequently, this rear shoe area is erroneously classified as background, and only the front foot is designated as the inpainting region. This incomplete mask ultimately results in an unsuccessful virtual try-on. We contend that this is an inherent challenge for all methods reliant on an explicit masking strategy, and developing more robust solutions to handle such severe occlusion cases remains an important direction for our future work. Shoe Expansion Phenomenon. The volumetric expansion of footwear observed during the try-on process is a naturally occurring phenomenon. In real-world datasets, it is common to find that shoes with stiffer tongues tend to bulge outwards when worn, resulting in a slight expansion of the shoe’s overall volume. This effect is illustrated in Fig. 5, where the third column clearly shows a prominent bulge in the tongue area, leading to body expansion. This expansion occurs stochastically in real-world data pairs, as exemplified in the third row of Fig. 1(a), where the shoe upper in the second column exhibits a slightly thicker contour. Consequently, by effectively fitting to the training data distribution, our model learns to replicate this real-world phenomenon, accurately rendering the corresponding deformations inherent to the try-on process. Table 4: Shoe Category Statistics. We report the sample count for each subcategory, with its proportion indicated in parentheses. Primary Categories Subcategory Casual Shoes: 3158 (43.23%) Lifestyle Casual Shoes: 803 (10.99%) Sneakers: 1428 (19.55%) Canvas Shoes: 141 (1.93%) Children Casual Shoes: 99 (1.36%) Men Casual Shoes: 687 (9.40%) Athletic Shoes: 558 (7.64%) Running Shoes: 322 (4.41%) Basketball Shoes: 121 (1.66%) Training Shoes: 45 (0.62%) Football Shoes: 70 (0.96%) High-Heel Shoes: 895 (12.25%) Classical High-heel Shoes: 340 (4.65%) Ladies Casual Shoes: 401 (5.49%) Mary Jane Shoes: 154 (2.11%) Boots: 1045 (14.31%) Ankle Boots: 416 (5.69%) Chelsea Boots: 56 (0.77%) High Boots: 153 (2.09%) Snow Boots: 183 (2.51%) Martin Boots: 237 (3.24%) Sandals: 279 (3.82%) Flip Flops: 183 (2.51%) Beach Sandals: 36 (0.49%) Strap Sandals: 60 (0.82%) Dress Shoes: 389 (5.33%) Loafers: 209 (2.86%) Formal Leather Shoes: 180 (2.46%) Slippers: 981 (13.43%) Clogs: 82 (1.12%) Thong Slippers: 18 (0.25%) House Slippers: 881 (12.06%) Table 5: Human scene statistics. Scene Types Number of samples Percentage Top-down Foot 1719 23.53% Horizontal Foot 4330 59.27% Half-body Model 833 11.40% Full-body Model 423 5.79% Figure 8: We provide more visual samples from MVShoes datasets. Appendix C Method Supplement C.1 Preliminary FLUX.1 Our ShoeFit is an extension of Stable Diffusion 3 [55] and FLUX.1 [5], which are the most commonly used text-to-image diffusion models based on Flow Matching[41] and DiT[51]. FLUX.1 [5] employs a variational autoencoder [33] (VAE) that consists of an encoder ℰE and a decoder D to enable image representations in the latent space. It is also equipped with rotary positional embeddings (RoPE)[58] and denoise-text-stream attention layers to improvement performance. FLUX.1 implements an actual two-dimensional RoPE scheme for encoding spatial positions in the latent space: ωd=1θ2d/D,forD=0,1,…,D/2−1, _d= 1θ^2d/D,forD=0,1,...,D/2-1, (9) where θ is typically set to 10000. The position encoding applies a rotation matrix: [cos(ωd⋅)−sin(ωd⋅)sin(ωd⋅)cos(ωd⋅)] bmatrix ( _d·pos)&- ( _d·pos)\\ ( _d·pos)& ( _d·pos) bmatrix (10) This rotation is applied to query and key vectors in the attention mechanism, enabling the model to capture relative positional relationships in the latent space. Flow Matching Flow matching [41] aligns the flow of information between noise ϵε and data distributions by optimizing a velocity field utu_t, which progressively converts noise into data over time. This technique ensures that the generative model maps the noise distribution to the actual data distribution in a structured manner. The text encoders [53] τθ _θ are employed to deal with the given text prompt y. The flow matching loss is defined as follows: ℒ=Et,pt(z|ϵ),p(ϵ),y[‖vΘ(z,t,τθ(y))−ut(z|ϵ)‖2].L=E_t,p_t(z|ε),p(ε),y [ \|v_ (z,t, _θ(y))-u_t(z|ε) \|^2 ]. (11) In this context, vΘ(z,t,τθ(y))v_ (z,t, _θ(y)) signifies the conditional velocity field determined by the weights of the neural network, while ut(z|ϵ)u_t(z|ε) represents the vector field created by the model to delineate the probabilistic trajectory between the noise and actual data distributions. The symbol E stands for the expectation, which involves either integration or summation over time t, latent variables z, conditions y, and noise ϵε. This expectation computes the mean of the squared differences for all conditions, ensuring that the model’s performance is evaluated over numerous instances to yield a dependable estimate of its generative capability. C.2 LayeredRefAttention in Single-stream DiT Block The original FLUX is a text-to-image model composed of a series of stacked M-DiT (double-stream) blocks, followed by a series of stacked single-stream DiT blocks. Due to space constraints, the framework of the LayeredRefAttention module shown in the main text is its structure within the double-stream DiT blocks. Here, we provide how it is used within a single-stream DiT block in Fig. 10. Similar to the way in double-stream blocks, we employ a Squeeze-and-Excitation (SE) block [27] followed by global average pooling over spatial dimensions to compute channel-specific weights: P(n)=AvgPool(SE(CI(n))∈ℝb×cCLOSE.P^(n)=AvgPool(SE(C_I^(n)) ^b× c. (12) Subsequently, we employ two linear layers F(⋅)F(·) to extract background modulation parameters and modulate foreground shoe features, filtering irrelevant environmental lighting and background reflection to ensure faithful material preservation of shoes as: βscale(n),βshift(n)=F(P(n)),Cfg(n)=(1+βscale(n))⋅(LN(CI(n)⊙Mfg(n)))+βshift(n), _scale^(n), _shift^(n)=F(P^(n)), C_fg^(n)=(1+ _scale^(n))·(LN(C_I^(n) M_fg^(n)))+ _shift^(n), (13) where LN(⋅)LN(·) means layer normalization and Mfg(n)M_fg^(n) represents the binary shoe masks for conditioning product image. The subsequent operations are then performed as in regular single-stream DiT blocks. Figure 9: More visual results of the ablation study. We highlight the improvements by all three components of the method in red. Best viewed when zoomed in. Figure 10: We provide how LayeredRefAttention is used within a single-stream DiT block. Appendix D Implementation Supplement D.1 Baseline Training Details We here provide detailed descriptions of the training processes for the baseline models, focusing specifically on the training configurations. All models are trained on MVShoes at a resolution of 768×768768× 768, utilizing 6305 pairs for training and 1000 pairs for testing. For Flux.1 Fill [6], we implement the training and inference pipeline by concatenating footwear images with human model images. In the trainable components, we apply LoRA [26] with a rank of 64 to all attention modules in the model and a Flux ControlNet [71] comprising 6 single layers and 6 double layers, injecting the poses at every denoising step. The model is trained for 6 days on 8 80GB-A100 GPUs using DeepSpeed ZeRO-2, with a batch size of 4. We utilize the AdamW optimizer, setting a constant learning rate of 3e-5 for training, and operate the model on a single A100 GPU for 25 steps during inference. For Flux.1 Redux [7], we adopt the same training settings as those used for FLUX.1-Fill. Additionally, we incorporate two linear layers for projecting SigLip [70] features of the product shoe images, training these two layers concurrently with the attention LoRAs and ControlNet. The parameters of SigLip remain frozen throughout the training process. We retain the same inference settings as employed in FLUX.1-Fill. For OOTDiffusion [65], we execute the training and inference pipelines based on the official code. We also implement ControlNet to facilitate pose injection. The model is trained for 6 days on 4 80GB-A100 GPUs with DeepSpeed ZeRO-2, at a batch size of 8. During inference, the model is run on a single A10 GPU for 25 steps. For CatVTON-Flux-Lite-2V, we maintain the text stream for FLUX-lite and concatenate footwear images with human model images. The model undergoes training for 7 days on 12 80GB-A100 GPUs utilizing DeepSpeed ZeRO-2, with a batch size of 2. All other training and inference configurations align with those of ShoeFit-2V. D.2 Data Augmentation We have implemented data augmentation techniques that could potentially enhance the model’s generalization ability as well as its fidelity performance. Specifically, the data augmentation operations include (a) horizontal flipping of images, (b) resizing footwear and human figures through padding (up to 10% of the image size), (c) randomly adjusting the image’s hue within a range of -5 to +5, and (d) randomly adjusting the image’s contrast within a specified range (between 0.8 and 1.2 times the original contrast). Each of these operations occurs independently with a 50% probability. Moreover, these operations are simultaneously applied to both the footwear and model images. D.3 LayeredRefAttention Hyperparameters In the LayeredRefAttention layer, we primarily introduced a new Linear layer and an SE block. The Linear layer for the foreground follows the same dimensions as other linear layers within the module, which is the hidden size. The SE block takes input features with the same dimensions of hidden size, and internally, we use a compression rate of reduction=4reduction=4 for SE Block, as Linear(channels,channels/4)Linear(channels,channels/4), ReLU()ReLU(), Linear(channels/4,channels)Linear(channels/4,channels). Appendix E More visual results Fig. 9 provides more results about the ablation study. We highlight the improvements by all three components of the method in red. Fig. 11 provides more results on MVShoes for comparisons between the baselines and ShoeFit. For a fair comparison, we report the results of the single-view conditioning version of all methods. Our method substantially improves rendering fidelity and robustness under challenging real-world product shoes, establishing a new benchmark in high-fidelity footwear try-on synthesis. Fig. 12 provides more results on MVShoes for inspection to demonstrate that ShoeFit synthesizes high-fidelity and detail-faithful try-on results. Fig. 13 details two key limitations of our model. First, in the early training phase, the model has not yet mastered spatial reasoning, resulting in structurally distorted outputs (row (a)). Second, generating precise masks for tall footwear remains a challenge. As shown in the row (b), inaccurate masks for items like boots can cause significant visual artifacts or fitting errors. Fig. 14 illustrate more results on out-of-distribution (OOD) samples not present in the MVShoes dataset. The high-quality try-on results demonstrate the model’s notable stability and strong generalization capabilities. Figure 11: More visual comparisons on the MVShoes by ShoeFit. Best viewed when zoomed in. Figure 12: More visual results on the MVShoes by ShoeFit. Best viewed when zoomed in. Figure 13: More visual results on two primary failure modes of our method: (a) structural failure observed during early training stages, and (b) artifacts arising from inaccurate masks in long footwear scenarios. Figure 14: More visual results on out-of-distribution (OOD) samples not present in the MVShoes dataset. The high-quality try-on results demonstrate the model’s notable stability and strong generalization capabilities.