Paper deep dive
Giraffe: A Mapping Architecture from Hidden Text Representations to Visual Embeddings for Efficient Graphic Design
Nejla Ghaboosi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/26/2026, 5:14:13 AM
Summary
The paper introduces Giraffe, a novel architecture that maps hidden text representations from multimodal large language models (MLLMs) to visual embeddings using a single [IMG] token per image. This approach addresses the limitation of previous methods that require multiple tokens per image, which increases input length and hinders complex graphic design generation. Giraffe employs two shallow MLP blocks (a main block and an assisting block) trained with six distinct loss functions, allowing for efficient inference by omitting the assisting block. The system demonstrates strong performance in both text-to-design and image-to-design generation tasks, producing visually coherent and stylistically consistent designs.
Entities (10)
Relation Signals (7)
Giraffe → enables → Graphic Design
confidence 95% · Strong performance is demonstrated in both image-to-design and text-to-design generation tasks.
Giraffe → uses → CLIP ViT-L/14
confidence 95% · The architecture employs... mapping... to the embedding space of visual models, such as CLIP ViT-L/14
Giraffe → consistsof → MLP Block
confidence 92% · The proposed mapping architecture... is composed of two shallow MLP blocks.
Giraffe → outputsto → FLUX
confidence 90% · Finally, real images are produced by passing z^v to FLUX with CLIP VIT-L/14 IP Adapter.
Giraffe → replaces → Multiple Tokens
confidence 90% · using a single [IMG] token per image... addressing... methods often represent each image using multiple specialised tokens
Gemma3 4B → usedin → Giraffe
confidence 90% · In this experiment, the Gemma3 4B model [20] is used as the language model.
GILL → uses → Stable Diffusion
confidence 85% · GILLMapper translates... into the embedding space of Stable Diffusion’s... text encoder.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) have made significant progress in understanding and interpreting mul- timedia content. However, their ability to generate me- dia remains limited. Recent approaches have attempted to bridge this gap by translating the hidden representations of token sequences into the embedding space of visual models or directly into raw image data. However, these methods often represent each image using multiple specialised to- kens which significantly increases the input length. This be- comes a major limitation for tasks such as graphic design generation where the output typically involves a seamless blend of thousands of tokens across text, multiple images, and layout information. To address this challenge, a novel architecture is proposed that maps hidden token represen- tations to the embedding space of visual models, such as CLIP ViT-L/14, using a single [IMG] token per image. The architecture employs two shallow MLP blocks, each with a separate compression module followed by a shared expan- sion module, trained with six distinct loss functions. One block aids the other during training and is omitted during inference, resulting in a lightweight solution. Strong perfor- mance is demonstrated in both image-to-design and text-to- design generation tasks.
Tags
Links
- Source: https://arxiv.org/abs/2608.23970v1
- Canonical: https://arxiv.org/abs/2608.23970v1
Trouble viewing inline? Open PDF directly →
Full Text
32,311 characters extracted from source content.
Expand or collapse full text
Giraffe: A Mapping Architecture from Hidden Text Representations to Visual Embeddings for Efficient Graphic Design Nejla Ghaboosi Affiliation: Canva Research Email: nejla@canva.com Abstract Multimodal large language models (MLLMs) have made significant progress in understanding and interpreting multimedia content. However, their ability to generate media remains limited. Recent approaches have attempted to bridge this gap by translating the hidden representations of token sequences into the embedding space of visual models or directly into raw image data. However, these methods often represent each image using multiple specialised tokens which significantly increases the input length. This becomes a major limitation for tasks such as graphic design generation where the output typically involves a seamless blend of thousands of tokens across text, multiple images, and layout information. To address this challenge, a novel architecture is proposed that maps hidden token representations to the embedding space of visual models, such as CLIP ViT-L/14, using a single [IMG] token per image. The architecture employs two shallow MLP blocks, each with a separate compression module followed by a shared expansion module, trained with six distinct loss functions. One block aids the other during training and is omitted during inference, resulting in a lightweight solution. Strong performance is demonstrated in both image-to-design and text-to-design generation tasks. 1 Introduction Multimodal large language models (MLLMs) have made remarkable advancements in tasks involving visual comprehension and understanding such as visual question answering (VQA), image captioning, and object grounding [21, 2, 13, 12, 22, 24]. However, these methods primarily focus on processing multimodal inputs but only generating textual outputs. This restricts their effectiveness in tasks that require generating visual content alongside text, such as graphic design generation. Graphic designs are composed of various multimodal elements including text, images, SVG shapes, colours, and spatial layout, all working together to convey meaning and visual appeal. Figure 1: Example results from the proposed architecture. The model generates visually coherent and stylistically consistent graphic designs from either text or image inputs. Recent approaches have explored the expansion of language models to accommodate the generation of multimodal outputs. In FROMAGe [10], image retrieval capabilities are introduced by augmenting a frozen language model with two trainable linear layers. The first linear layer maps the hidden representation of a [RET] token whilst the second maps the visual embedding of the corresponding image. These two embeddings are aligned through contrastive learning [14], allowing the model to associate textual queries with relevant visual content. Nonetheless, this approach falls short when a design requires an image with a specific subject, style, or colour theme that is not available in the media library, or when user lacks access to a sufficiently large media collection. GILL [9] extends the capabilities of FROMAGe by enabling media generation rather than just retrieval. This is achieved through a mapping network called GILLMapper - a transformer with both encoder and decoder components. GILLMapper translates the hidden representations of r [IMG] tokens, generated by the language model, into the embedding space of Stable Diffusion’s [17] text encoder. The alignment between the two embedding spaces is learned using a mean squared error (MSE) loss. During inference, the images are generated by passing the mapped embeddings through the diffusion model. Similarly, Emu [18] enables media generation by augmenting a language model with a module called Causal Transformer that transforms visual inputs into N [IMG] visual tokens. The model is trained to autoregressively predict these visual tokens, using an MSE loss between the predicted embeddings and the ground-truth tokens generated by the Causal Transformer. During inference, the generated visual tokens are decoded into images using a Stable Diffusion model that has been fine-tuned to explicitly condition on generated embeddings. In Chameleon [19], images are represented using 1024 discrete tokens derived from a codebook of size 8192, produced by a learned image tokenizer. These image tokens, along with text tokens, are used to train the model autoregressively. During inference, a learned de-tokenizer is used to reconstruct images from generated image tokens. Across all of these methods, a shared characteristic is the use of multiple visual tokens to represent images. In fact, [9] further investigates the impact of token count and finds that reducing the number of tokens generally leads to performance degradation as it produces shorter and less expressive inputs for the mapping network.This issue becomes particularly significant in graphic design generation tasks, where a single design may contain multiple images and is typically represented as a long, interleaved sequence of text and image tokens to capture elements such as text, images, SVG shapes, layout, and visual attributes. Consequently, representing each image with multiple tokens can cause the overall token sequence to become extremely long, often approaching or exceeding the model’s maximum token capacity. Moreover, the limited attention span of transformer-based architectures makes it challenging to capture long-range dependencies across such extended sequences, thereby constraining the model’s ability to produce coherent and consistent designs. Some approaches have explored generating images directly rather than relying solely on embeddings. For instance, Transfusion [26] leverages a VAE [8] to encode images into latent representations and then applies a diffusion process within this latent space. In this setup, text is modelled autoregressively using a transformer while image generation is guided by a diffusion-based objective. DreamLLM [3], on the other hand, enables direct image generation by using a frozen Stable Diffusion decoder that is guided by semantic queries produced by the language model. These queries are trained using Score Distillation Sampling (SDS) [15], allowing the model to align textual prompts with image generation targets. Like the aforementioned methods, both these approaches still rely on multi-vector or multi-token representations per image, which may limit their effectiveness in design generation tasks. Figure 2: The proposed Giraffe architecture overview. It enables mapping from hidden token representations to the embedding space of visual models, such as CLIP ViT-L/14, using only a single [IMG] token. This is achieved using two shallow MLP blocks where each includes a seperate compression module followed by a shared expansion module, trained with six distinct loss functions (left). The assisting MLP block aids the main block in learning the mapping and is ignored during inference (right), leading to a more lightweight solution. This paper proposes a novel architecture that facilitates mapping from hidden token representations of a language model to the embedding space of a visual model, like CLIP ViT-L/14 [16], using just a single [IMG] token per image. This enables the language model to create coherent graphic designs by integrating text and image tokens within long sequences. The proposed mapping architecture, termed Giraffe due to its L-shaped structure, is composed of two shallow MLP blocks. Each block features a unique compression module, followed by a shared expansion module that links the two. One block is mainly responsible to handle the mapping, while the other assists the main block during the training phase, playing an essential role in helping the primary block to reach its mapping goal. When it comes to inference, the assisting block is removed, leading to a more lightweight solution. Since the generated embeddings follow a known distribution, there is no need to fine-tune a diffusion model. An existing pretrained model such as FLUX [11] with CLIP VIT-L/14 IP Adapter [23] can be used directly to generate images. Extensive experiments highlight its strong performance in both text-to-design generation and image-to-design tasks. In the text-to-design generation task, it can be seen that the produced images align with the provided prompt while preserving the overall style, color palette, and semantics of the graphic design, leading to an aesthetically pleasing outcome. Likewise, in the image-to-design task, the generated graphic design closely mirrors the reference image. Collectively, the observed behavior indicates that the proposed architecture effectively captures all image-related information within a single [IMG] token. Fig. 1 shows an example for each task. It is important to note that while this paper focuses primarily on images, the proposed architecture is flexible and can be adapted for various media, including audio and video. 2 Method This section describes the graphic design representation used by the model, the proposed Giraffe architecture, the training objectives, and the inference process. 2.1 Graphic Design Structure A graphic design is composed of various elements, including text, images, and SVG shapes, which all work together to convey ideas and create visual harmony. Each element is defined by a set of properties that determine its structure and appearance. These include positional attributes such as top, left, width, height, and layering order, as well as visual characteristics like colour, transparency, and rotation. Certain attributes are also specific to particular element types. For example, text elements may include the textual content itself, font family, font size, and styling options such as bold or italic. By representing these element-specific properties in textual form, a graphic design can be represented as a long sequential structure of interleaved text and image components, where each xtx_t and xvx_v denotes a single text or image component, respectively. The inclusion of separate image components is necessary, as visual content cannot be fully captured through text alone. 2.2 Giraffe Architecture Following prior work [21, 13, 4, 25, 5], the visual embedding of each image xvx_v is extracted and projected into the same dimensional space as the word embeddings, using the following transformations: zv z_v =vϕ(xv)∈ℝd, =v_φ(x_v) ^d, (1) uv u_v =Wzv∈ℝm =Wz_v ^m , where W is a learnable linear mapping, vϕ(.)v_φ(.) is the pretrained and frozen visual encoder (ViT-L/14) prior to its projection head and m represents the input embedding dimension of the language model. Each xvx_v in the input sequence is then replaced with just one special [IMG] token and its corresponding word embedding is substituted with uvu_v. To train the model to understand and generate multimodal content, it is optimised with the standard next-token prediction objective. Specifically, the model is fine-tuned to predict each next token using the cross-entropy loss: ℒCE=−1M∑m=1M∑t=1TmlogPψ,W(wt(m)∣w<t(m)) _CE=- 1M _m=1^M _t=1^T_m P_ψ,W (w_t^(m) w_<t^(m) ) (2) , where M is the batch size, TmT_m is the length of the mmth input sequence, and ψ denotes the language model parameters. Each token wtw_t may be either a text token or the special [IMG] token. Finally, to enable image generation, a mapping network is integrated with the language model to decode each [IMG] token into its corresponding normalised visual embedding z~v z_v. z~v z_v is obtained by first clipping the original visual embedding zvz_v to remove extreme values, followed by applying the min-max-normalision technique to rescale the values into the range [-1, 1]. This mapping is learned using a multilayer perceptron (MLP) block, which comprises of a compression module fθ1f_ _1 followed by an expansion module fθs′f _ _s with SiLU activation function applied throughout both modules. However, the last layer in fθs′f _ _s uses Tanh as its activation function. The parameters of this block are optimised by minimising the mean square error (MSE) loss between its output and z~v z_v: ℒb1-MSE _b1-MSE =1N∑i=1N(z~^vi−z~vi)2, = 1N _i=1^N ( z_v_i- z_v_i )^2, (3) z~^vi z_v_i =fθs′(fθ1(hψ([IMG]i))) =f _ _s(f_ _1(h_ψ([IMG]_i))) , where hψ([IMG]i)h_ψ([IMG]_i) denotes the representation of the ithith [IMG] token from the final hidden layer of the language model and N represents the number of [IMG] tokens in the batch. Additionally, a cosine similarity loss is used to promote directional alignment between the predicted embeddings and the ground truth: ℒb1-COS _b1-COS =1N∑i=1N(1−gφ(z~^vi)⋅gφ(z~vi)‖gφ(z~^vi)‖‖gφ(z~vi)‖) = 1N _i=1^N (1- g_ ( z_v_i)· g_ ( z_v_i)\|g_ ( z_v_i)\|\|g_ ( z_v_i)\| ) (4) , where gφ(.)g_ (.) denotes the projection head of the pretrained and frozen ViT-L/14 visual encoder. gφ(.)g_ (.) is applied to both predicted and target embeddings to ensure compatibility in the representation space. While the proposed MLP block establishes the baseline mapping, it often struggles to learn the mapping effectively. To enhance this block’s ability to fully capture the transformation, an assisting MLP block is introduced, forming the Giraffe architecture named for its L-shaped structure. The assisting block features its own compression module fθ2′f _ _2 with SiLU as activation function but shares the expansion module fθs′f _ _s with the main MLP block. Functionally, it operates as an autoencoder where the input is the normalised visual embedding z~v z_v and the output is its approximation z¯vi z_v_i. Similarly, it is trained with two losses: ℒb2-MSE _b2-MSE =1N∑i=1N(z¯vi−z~vi)2, = 1N _i=1^N ( z_v_i- z_v_i )^2, (5) z¯vi z_v_i =fθs′(fθ2′(z~vi)) =f _ _s(f _ _2( z_v_i)) and, ℒb2-COS _b2-COS =1N∑i=1N(1−gφ(z¯vi)⋅gφ(z~vi)∥gφ(z¯vi)∥.∥gφ(z~vi∥). = 1N _i=1^N (1- g_ ( z_v_i)· g_ ( z_v_i)\|g_ ( z_v_i)\|.\|g_ ( z_v_i\| ). (6) To further assist fθ1f_ _1 in learning the mapping transformation, two infoNCE losses [14] are applied on the bottlenecks of the two MLP blocks: ℒNCE1 _NCE_1 =−1N∑i=1N(logesim(fθ1(z~vi),fθ2′(z~vi))/τ1M∑j=1Mesim(fθ1(z~vi),fθ2′(z~vj))/τ), =- 1N _i=1^N (log e^sim (f_ _1( z_v_i),f _ _2( z_v_i) )/τ 1M _j=1^Me^sim (f_ _1( z_v_i),f _ _2( z_v_j) )/τ ), (7) ℒNCE2 _NCE_2 =−1N∑i=1N(logesim(fθ1(z~vi),fθ2′(z~vi))/τ1M∑j=1Mesim(fθ1(z~vj),fθ2′(z~vi))/τ), =- 1N _i=1^N (log e^sim (f_ _1( z_v_i),f _ _2( z_v_i) )/τ 1M _j=1^Me^sim (f_ _1( z_v_j),f _ _2( z_v_i) )/τ ), where τ is a learnable temperature parameter. The total loss is calculated as: ℒtotal _total =λ1ℒb1-MSE+λ2ℒb1-COS = _1L_b1-MSE+ _2L_b1-COS (8) +λ3ℒb2-MSE+λ4ℒb2-COS + _3L_b2-MSE+ _4L_b2-COS +λ5(ℒNCE1+ℒNCE2)+λ6ℒCE, + _5(L_NCE_1+L_NCE_2)+ _6L_CE, where λ1 _1, λ2 _2, λ3 _3, λ4 _4, λ5 _5 and λ6 _6 are hyperparameters representing loss weights. The optimisation is carried out over parameters W, θ1 _1, θ2 _2, θs _s, and ψ. An overview of the architecture is shown in Fig. 2. 2.3 Inference At inference time, fθ2′f _ _2 is removed, resulting in a lightweight mapping network. Each predicted [IMG] token is mapped into its normalised visual embedding as follows: z~^v z_v =fθs′(fθ1(hψ([IMG]))), =f _ _s(f_ _1(h_ψ([IMG]))), (9) which is then denormalised to recover the predicted visual embedding zv z_v. During the autoregressive process, zv z_v is used directly instead of recomputing zvz_v via the visual encoder vϕ(.)v_φ(.) for each generated [IMG] token, enabling faster token generation. Finally, real images are produced by passing z^v z_v to FLUX with CLIP VIT-L/14 IP Adapter. The generated images are then placed within the specified bounding boxes to create the final graphic design. 3 Experiments To evaluate the proposed method, two experiments are carried out on text-to-design and image-to-design generation tasks. The model is trained on a proprietary dataset of 1,800,000 professionally created graphic designs. These graphic designs span a wide range of formats, including social media posts, banners, flyers, business cards, and logos. The dataset is split into training and validation subsets, with 90% used for training and 10% for validation. In the mapping network, fθ1(.)f_ _1(.) consists of layers with dimensions 1024−768−5121024-768-512, while fθ2′(.)f _ _2(.) has dimensions 1024−768−512−5121024-768-512-512, with the final layer in both serving as the bottleneck. fθs′(.)f _ _s(.) is composed of layers with dimensions 768−1024−1024768-1024-1024. The hyperparameters λ1 _1, λ2 _2, λ3 _3, λ4 _4, λ5 _5 and λ6 _6 are set to 1, 1, 1, 1, 0.1, and 1, respectively. 3.1 Text-to-design generation Table 1: FID score and CLIP cosine similarity for text-to-design and image-to-design tasks, respectively. Model Text-to-Design Image-to-Design (FID) (Cosine) Proposed 66.85 0.85±0.050.85± 0.05 Baseline 81.61 - Figure 3: Qualitative results for the text-to-design task. The generated graphic designs closely follow the prompts while remaining visually coherent, stylistically consistent, and of high overall quality. The model often produces complex designs with multiple images that interact harmoniously. Numbers in parentheses indicate the number of images used in each design. Figure 4: Qualitative results for the baseline model. Generated results have limited layout variation and less consistent visual style. Numbers in parentheses indicate the number of images used in each design. In this experiment, the Gemma3 4B model [20] is used as the language model. Training is carried out in mixed precision using the bfloat16 [1] format for 56,000 steps. Evaluation is conducted on a separate set of curated prompts, chosen to cover a wide range of cases. Each design sample is represented in JSON format, with floating-point bounding box coordinates rounded to the nearest integer. Furthermore, each design is accompanied by a textual description that captures its style, theme, and key details, for example, “A vibrant Instagram post announcing a summer smoothie sale with fresh fruit illustrations and bright colours”. For benchmarking, a baseline model is also trained without the mapping network, in which textual description of each image is generated instead and subsequently passed to FLUX for image generation. To evaluate the performance of the model, a comprehensive qualitative analysis is performed on the generated results. Figure 3 and Fig. 4 show sample outputs generated by the proposed architecture and the baseline model, respectively, for the same set of prompts. The number of images in each design is indicated in parentheses. As shown in these figures, the proposed method generates designs that are more visually appealing, with greater layout diversity and stronger stylistic and thematic coherence. Many of the generated designs include multiple images that work harmoniously in terms of color theme and style, as seen in prompts 3, 4, 9, and 12. In contrast, the baseline model often produces designs with minimal layout variation, featuring a background image, a central shape, and overlaid text. When the baseline generates designs containing multiple images, the images usually lack color harmony and appear visually not very relevant to the overall design. For instance, in prompt 7, the images generated by the baseline model do not exhibit consistent color or style, whereas the corresponding design produced by the proposed method maintains a coherent theme, with images that complement each other and enhance the overall design coherence. These limitations in the baseline model likely arise from representing each image solely through textual descriptions. The resulting increase in token length, together with the limited attention span of transformer-based architectures, makes it difficult to capture long-range dependencies and constrains the model’s ability to generate more complex designs. In addition, representing images as text alone makes it challenging to accurately capture their style and color theme. In contrast, the [IMG] token represents each image with a single token, allowing the model to focus on finer-grained details within its attention span. Moreover, the [IMG] token preserves the style and thematic information of an image more faithfully, enabling the proposed model to generate outputs that are more complex, visually coherent, stylistically consistent, and better aligned with the intended design. To further evaluate the performance of the proposed mapping network, the FID score [6] is computed by comparing the statistical distributions of the generated designs with those of real, professionally created designs from the training set. As shown in Tab. 1, the proposed method achieves a considerably lower FID score than the baseline model, indicating that it generates designs of higher quality and greater diversity, consistent with the findings of the qualitative study. 3.2 Image-to-design generation Figure 5: Qualitative results for the image-to-design task. In each pair, the left image shows the input generated by GPT-4o, and the right image shows the model’s output. In this experiment, the language model is replaced with a transformer with both encoder and decoder components, each consisting of 10 layers. The architecture has 500 million learnable parameters. Each design in the training set is first rendered, and the resulting image is divided into patches. These patch values are then concatenated with the CLIP embedding [16] of the rendered image and fed into the transformer’s encoder. The model is trained to generate the corresponding design as a sequence of text interleaved with [IMG] tokens, following the approach used in the previous experiment. However, since replacing the language model with a plain transformer and using a relatively small training set makes learning natural language from scratch challenging, the content of each textbox is replaced with the word “TEXT” to simplify the task. Training is performed in float32 precision for 187,000 steps. Evaluation is conducted on a small set of graphic designs generated from curated prompts using GPT-4o [7]. To evaluate the model’s performance, both quantitative and qualitative studies are conducted. Cosine similarity is computed between the CLIP embedding of each original input image and the generated design to measure how closely the generated design matches the input. As shown in Tab. 1, the cosine similarity of CLIP embeddings is 0.85±0.050.85± 0.05, indicating strong adherence to the original images. Figure 5 illustrates qualitative examples produced by the trained model. In each pair, the left image is the GPT-4o generated input, and the right image is the model’s output. For easier assessment, the original text content is manually added to the generated text boxes. While minor inconsistencies in details are observed, the outputs generally align closely with the inputs. These results demonstrate that a single [IMG] token per image captures substantial semantic, stylistic, and color information. They also suggest that knowledge transferred from a pretrained language model has minimal effect on the model’s ability to map [IMG] tokens to CLIP embeddings. However, a pretrained language model is still required for generating the text content. 4 Conclusion This paper introduces Giraffe, a novel architecture that maps hidden token representations from a language model to the embedding space of a visual model using a single [IMG] token per image. This approach allows language models to generate graphic designs, which would normally require very long sequences of interleaved text and image tokens. The mapping architecture consists of two shallow MLP blocks, each with a unique compression module followed by a shared expansion module that connects the blocks. The main block performs the mapping, while the second block assists in training to help the main block learn the mapping more effectively. During inference, the assisting block is removed, resulting in a lightweight solution. Experiments on a text-to-design task show that the architecture effectively captures the semantics, style, and color of each image, producing more complex designs with greater visual harmony than when images are represented solely through textual descriptions. The results also emphasize the importance of using a single token per image to achieve higher diversity and complexity in the generated graphic designs. Similarly, experiments on an image-to-design task demonstrate that the proposed architecture can capture semantics, style, and colour information with just one [IMG] token per image. References [1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. (2016) Tensorflow: large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467. Cited by: §3.1. [2] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, p. 23716–23736. Cited by: §1. [3] R. Dong, C. Han, Y. Peng, Z. Qi, Z. Ge, J. Yang, L. Zhao, J. Sun, H. Zhou, H. Wei, et al. (2023) Dreamllm: synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499. Cited by: §1. [4] C. Eichenberg, S. Black, S. Weinbach, L. Parcalabescu, and A. Frank (2022) Magma–multimodal augmentation of generative models through adapter-based finetuning. In Findings of the association for computational linguistics: EMNLP 2022, p. 2416–2428. Cited by: §2.2. [5] H. Fei, S. Wu, H. Zhang, T. Chua, and S. Yan (2024) Vitron: a unified pixel-level vision llm for understanding, generating, segmenting, editing. Advances in neural information processing systems 37, p. 57207–57239. Cited by: §2.2. [6] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30. Cited by: §3.1. [7] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. (2024) Gpt-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §3.2. [8] D. P. Kingma and M. Welling (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §1. [9] J. Y. Koh, D. Fried, and R. R. Salakhutdinov (2023) Generating images with multimodal language models. Advances in Neural Information Processing Systems 36, p. 21487–21506. Cited by: §1, §1. [10] J. Y. Koh, R. Salakhutdinov, and D. Fried (2023) Grounding language models to images for multimodal inputs and outputs. In International Conference on Machine Learning, p. 17283–17300. Cited by: §1. [11] B. F. Labs (2024)FLUX.1-dev(Website) Note: Accessed: 2025-06-10 External Links: Link Cited by: §1. [12] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §1. [13] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: §1, §2.2. [14] A. v. d. Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §1, §2.2. [15] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall (2022) Dreamfusion: text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988. Cited by: §1. [16] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1, §3.2. [17] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695. Cited by: §1. [18] Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang (2023) Emu: generative pretraining in multimodality. arXiv preprint arXiv:2307.05222. Cited by: §1. [19] C. Team (2024) Chameleon: mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818. Cited by: §1. [20] G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. (2025) Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: §3.1. [21] M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill (2021) Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34, p. 200–212. Cited by: §1, §2.2. [22] P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang (2022) Ofa: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International conference on machine learning, p. 23318–23340. Cited by: §1. [23] XLabs-AI (2025)Flux-ip-adapter-v2(Website) Note: Accessed: 2025-06-10 External Links: Link Cited by: §1. [24] Z. Yang, Z. Gan, J. Wang, X. Hu, F. Ahmed, Z. Liu, Y. Lu, and L. Wang (2022) Unitab: unifying text and box outputs for grounded vision-language modeling. In European Conference on Computer Vision, p. 521–539. Cited by: §1. [25] J. Zhang, M. Khayatkhoei, P. Chhikara, and F. Ilievski (2023) Towards perceiving small visual details in zero-shot visual question answering with multimodal llms. arXiv preprint arXiv:2310.16033. Cited by: §2.2. [26] C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy (2024) Transfusion: predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039. Cited by: §1.