Paper deep dive
Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation
Nikolai Röhrich, Isabell Hans, Felix Krause, Björn Ommer
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/17/2026, 5:15:46 AM
Summary
The paper introduces Concept Guidance (CoG), a training-free method for text-to-image diffusion models that enables precise, concept-specific control by leveraging per-layer mutual information. CoG identifies concept-relevant layers, skips them to generate negative predictions, and uses performance-weighted extrapolation to guide denoising, improving local coherence (e.g., text, hands) and aesthetics across models like PixArt-alpha, SD3, and FLUX.1-dev without additional training or external models.
Entities (10)
Relation Signals (8)
Nikolai Röhrich → affiliatedwith → LMU Munich
confidence 95% · Affiliation: LMU Munich, Germany
Concept Guidance → appliedto → Stable Diffusion 3
confidence 95% · We demonstrate performance increases across various targets and popular models like ... SD3
Concept Guidance → appliedto → PixArt-alpha
confidence 95% · We demonstrate performance increases across various targets and popular models like PixArt-alpha
Concept Guidance → appliedto → FLUX.1 [Dev]
confidence 95% · We demonstrate performance increases across various targets and popular models like ... FLUX.1-dev
Concept Guidance → improves → Text-to-Image Diffusion Models
confidence 95% · CoG ... enables precise, out-of-the-box guidance for T2I models and increases T2I generation performance
Concept Guidance → uses → Mutual Information
confidence 92% · We introduce a novel notion of concept-wise mutual information ... to find large, concept-dependent differences between individual layers
Concept Guidance → integrateswith → Classifier-Free Guidance
confidence 90% · CoG seamlessly integrates with CFG, allowing users to effortlessly steer both global prompt alignment and targeted semantics
Concept Guidance → usestechnique →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2608.14172v1
- Canonical: https://arxiv.org/abs/2608.14172v1
Trouble viewing inline? Open PDF directly →
Full Text
64,084 characters extracted from source content.
Expand or collapse full text
Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation Nikolai Röhrich† Thanks: Corresponding author: n.roehrich@campus.lmu.de Affiliation: LMU Munich, Germany Affiliation: Konrad Zuse School of Excellence in Reliable AI (relAI), Germany Isabell Hans† Affiliation: LMU Munich, Germany Affiliation: Konrad Zuse School of Excellence in Reliable AI (relAI), Germany Felix Krause† and Björn Ommer Affiliation: CompVis @ LMU Munich, Germany Affiliation: CompVis @ LMU Munich, Germany Affiliation: Munich Center for Machine Learning (MCML), Germany Affiliation: Munich Center for Machine Learning (MCML), Germany Abstract Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer’s concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-α, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance. Keywords: Text-to-Image Generation Diffusion Models Guidance 00footnotetext: Equal contribution. 1 Introduction Text-to-image (T2I) diffusion models have achieved remarkable quality in generating images from natural language descriptions [34, 37, 9, 12]. By sampling a random latent and iteratively refining it, these models gradually transform the latent into a coherent image that aligns with a given text prompt. Two fundamental challenges persist, particularly in one-shot generation. First, T2I models lack reliability in tasks requiring precise local coherence. This is perhaps most apparent in outputs involving text, where models frequently produce misspellings or hallucinated characters [8], and in images involving human hands, which often exhibit incorrect finger counts, distortions, or implausible geometry [32]. Figure 1: Concept Guidance (CoG) improves arbitrary quantifiable concepts across a variety of T2I models. CoG enables models to recover from classical failure modes while also delivering better generative quality for abstract concepts. Second, Classifier-Free Guidance (CFG) [16] – the de facto standard guidance mechanism in T2I diffusion – provides no fine-grained control over the generation process. In CFG, generation is guided by performing a conditional and an unconditional forward pass and by extrapolating beyond the conditional noise prediction. This technique effectively controls global prompt alignment, but it entirely lacks the ability to control fine-grained semantic details and tends to underperform for complex prompts due to its global nature [26, 49]. Prominent attempts to address these limitations fall into three broad categories, each with significant drawbacks (complementary directions, such as refined guidance rules, are discussed in Section 2): (1) Fine-tuning methods like ControlNet [53] or T2I-Adapters [31] can impose spatial control but require extensive per-concept training and large datasets. (2) Gradient-based methods [11, 3] can guide generation towards a target but are computationally expensive. (3) Inference-time intervention, e.g. by manipulating cross-attention maps [14, 23], offers more flexibility, but often requires careful model-specific tuning. Our method, Concept Guidance (CoG), in contrast, is an out-of-the-box, concept-specific guidance mechanism for T2I models that requires no training, external models, gradients, or reverse-engineering of model internals. By introducing a novel notion of per-layer, per-concept mutual information, we demonstrate the varying degrees of influence that layers in T2I models have on the generation of specific semantic concepts (see Figure 3), and we exploit this information by reinforcing the influence of concept-relevant layers (see Figure 2). We first propose a framework to measure the layer-wise target performance by using target-specific metrics. Then, we generate predictions with relevant layers skipped and guide by extrapolating the standard prediction away from those predictions, using performance-based weighting. This selectively amplifies the most concept-relevant layers, allowing for precise guidance. Additionally, CoG seamlessly integrates with CFG, allowing users to effortlessly steer both global prompt alignment and targeted semantics. We make the following contributions: 1. We propose a novel method of measuring the mutual information between diffusion network layers and target concepts, and thereby show that the properties of specific layers can be exploited for precise, per-concept guidance. We demonstrate that layers that work well for skip-based guidance are precisely the layers responsible for generating relevant concepts. 2. In light of our layer analysis, we present Concept Guidance, a concept-specific guidance method based on layer skipping. CoG is a simple yet effective method that enables precise, out-of-the-box guidance for T2I models and increases T2I generation performance for arbitrary (measurable) concepts. 3. Through extensive experiments, we demonstrate that Concept Guidance consistently outperforms Classifier-Free Guidance and alternative guidance mechanisms based on layer-skipping. Across all tested models, we achieve an average single-target performance increase of 8.1%8.1\% compared to CFG. 2 Related Work Training-Free Guidance via Model Perturbations Diffusion sampling is commonly steered with classifier guidance [11] which leverages gradients from an external classifier to steer generation, or classifier-free guidance (CFG) [16], which increases condition adherence by contrasting conditional and unconditional predictions. Recent work revisits guidance rules to reduce CFG artifacts, e.g., via adaptive projection (APG) [38] or manifold-constrained guidance (CFG++) [10], and to obtain CFG-like behavior without special unconditional training [39]. Autoguidance [22] instead contrasts the model with a weakened version of itself, e.g., an earlier training checkpoint, thus requiring access to training artifacts. A complementary training-free line constructs a “weak” model at inference time by perturbing the generator itself, including attention-maps [17, 18, 1], processed tokens [36], or self-guidance derived from the model’s own dynamics [25]. Layer skipping is another perturbation method widely exposed in modern diffusion pipelines, e.g., Stable Diffusion 3 [42], yet it remains underexplored as a controllable handle. Spatiotemporal Skip-Guidance (STG) uses a fixed single-layer skip to improve video quality [21]. Our work is closest in spirit to perturbation-based training-free guidance, but differs by making skip perturbations concept-dependent and by using weighted combinations of multiple layers rather than a fixed skip. Concept-Specific Control Beyond Prompting Beyond prompt engineering, concept-specific control is often achieved by adding learnable components: low-rank adapters (LoRA) [19] enable parameter-efficient updates and support per-concept modules such as concept sliders [13], while other approaches train auxiliary controllers (e.g., adapters) for new conditioning modalities [31]. Relatedly, some methods train lightweight predictors or readout heads on frozen diffusion features and backpropagate through them during sampling to enforce targets [28]. A different family uses external guidance objectives at inference time—either via gradients from arbitrary guidance functions [3] or via energy-based losses built from off-the-shelf predictors [51], with newer formulations addressing proxy unreliability through trust-region style sampling [20]. In contrast, our goal is a practical mechanism for concept-specific improvement within an off-the-shelf generator: we require no training and no external predictors during denoising. Concept Localization and Interpretability A growing body of work probes where and how diffusion models represent text-conditioned concepts. Attention-centric analyses and interventions treat cross-attention as the primary locus of word-region binding [14, 7], and attribution methods such as DAAM derive token-to-pixel maps from cross-attention aggregation [46]. Separately, feature-level probing reveals that semantic correspondences emerge in intermediate layers and vary strongly with depth [45, 29]. Information-theoretic perspectives quantify prompt–image dependence more robustly than raw attention, using MI-style decompositions [24, 52] and applying information-theoretic objectives to improve alignment [48]. Mechanistic and causal approaches further localize attribute-relevant components for model editing [5, 4] or component attribution [33], and recent work identifies “vital” layers in transformer backbones for training-free editing [2]. Our work connects these threads by turning per-concept layer specialization into an actionable control signal: we localize concept-relevant layers through targeted interventions and use the resulting layers to guide generation. 3 Method 3.1 Preliminaries T2I Diffusion Text-to-Image diffusion models [34, 37, 12] generate images aligned with a text prompt by iteratively denoising a randomly sampled latent into a noise-free image [15, 41]. This reverse process is trained by adding noise to image samples from the target distribution during the forward process. Formally, given an image x0x_0, noisy latents xtx_t at timestep t are given by q(xt∣x0)=(xt,α¯tx0,(1−α¯t)I),q(x_t x_0)=N(x_t; α_tx_0,(1- α_t)I), (1) where α¯t α_t controls the noise level. Parameters θ are updated based on the distance between predicted noise ϵθ(xt,t,c) _θ(x_t,t,c) and actual noise ϵε, thus solving minθx0,ϵ,t,c‖ϵ−ϵθ(xt,t,c)‖2. _θ _x_0,ε,t,c\|ε- _θ(x_t,t,c)\|^2. (2) Classifier Guidance Classifier Guidance [11] was introduced to steer unconditional diffusion models towards a distribution aligned with a desired mode y. Given a classifier pϕ(y∣xt)p_φ(y x_t), guidance is achieved by modifying the process to favor samples the classifier considers more likely. The distribution is adjusted as pθ(xt−1∣xt,y)∝pθ(xt−1∣xt)pϕ(y∣xt),p_θ(x_t-1 x_t,y) p_θ(x_t-1 x_t)\,p_φ(y x_t), (3) which corresponds to adding a correction to the model’s score estimate based on the gradients of the external classifier. In practice, this results in ∇logpθ(xt∣y)=∇logpθ(xt)+λ∇logpϕ(y∣xt),∇ p_θ(x_t y)=∇ p_θ(x_t)+λ\,∇ p_φ(y x_t), (4) where λ is a scalar guidance strength controlling the influence of the classifier on the generation process. The classifier gradients ∇logpϕ(y∣xt)∇ p_φ(y x_t) can be interpreted as the direction of strongest label-alignment in the latent space, y→ y. Classifier-Free Guidance Classifier-Free Guidance [16] steers diffusion generation towards a target y specified by a text prompt c, without requiring a classifier. Instead, the model is trained to perform both unconditional and text-conditioned generation. Then, guidance is achieved by extrapolating from the unconditional prediction ϵθ(xt,t,∅) _θ(x_t,t, ) beyond the conditional prediction ϵθ(xt,t,c) _θ(x_t,t,c): ϵ~θ(xt,t,c)=(1−λ)ϵθ(xt,t,∅)+λϵθ(xt,t,c), ε_θ(x_t,t,c)=(1-λ) _θ(x_t,t, )+λ _θ(x_t,t,c), (5) where λ≥1λ≥ 1 controls the strength of conditioning. Similar to Classifier Guidance, this process can be interpreted as determining and reinforcing the direction of strongest condition alignment in the latent space y→ y. 3.2 Concept Guidance Figure 2: Concept Guidance precisely approximates the target direction by combining per-layer skip predictions using concept-relevant layers (α,β,γα,β,γ). It computes individual skip-layer noise predictions (ϵ[θ∖α],ϵ[θ∖β],ϵ[θ∖γ] _[θ α], _[θ β], _[θ γ]) and extrapolates over them to precisely estimate the target direction. Classifier Guidance is precise in guiding towards specific targets, while Classifier-Free Guidance is simple and effective for increasing text-to-image alignment. We introduce Concept Guidance to combine the best of both worlds: highly usable, target-specific guidance in the latent space of diffusion models. We exploit the fact that given a target y, different layers of T2I models are particularly impactful w.r.t. y. To approximate the noise-space direction of highest target performance y→ y, we find such layers and precisely adjust their influence (see Figure 4). Mutual Information Analysis We hypothesize to find consistent, concept-dependent patterns of distributed layer responsibilities in T2I models that could be exploited for concept-specific guidance. To validate our intuition and to provide insight into why guiding with skipped layers works [21, 42], we analyze the mutual information (MI) between each layer and a given concept. We adapt the work of Wang et al. [48], where MI is used to locate layers that are responsible for general text-to-image alignment. Wang et al. [48] formulate their notion as the expected difference between conditional and unconditional predictions: I(x,c)=t,ϵκt‖ϵθ(xt,t,c)−ϵθ(xt,t,∅)‖2,I(x,c)=E_t,ε _t \| _θ(x_t,t,c)- _θ(x_t,t, ) \|^2, (6) where κt _t scales the contribution of each timestep, reflecting information flow at that denoising stage. To extend this framework for our method, we introduce per-layer MI by computing Equation 6 with a single skipped layer: I(x,c,i)=t,ϵκt‖ϵ[θ∖i](xt,t,c)−ϵ[θ∖i](xt,t,∅)‖2,I(x,c,i)=E_t,ε _t \| _[θ i](x_t,t,c)- _[θ i](x_t,t, ) \|^2, (7) where we denote the conditional noise prediction with layer i skipped as ϵ[θ∖i](xt,t,c) _[θ i](x_t,t,c), or simply ϵ[θ∖i] _[θ i]. We then define per-concept, per-layer MI as the difference in MI given a positive and a negative text prompt c and c∖yc y that differ only in the presence of the target y. That is, the mutual information between the target concept y and images generated while skipping layer i is given by I(x,y,i)=I(x,c,i)−I(x,c∖y,i).I(x,y,i)=I(x,c,i)-I(x,c y,i). (8) (a) FLUX.1-dev (b) PixArt-α Figure 3: Different concepts concentrate in different layers, enabling concept-specific layer selection for guidance. Bar plots report per-layer mutual information in (a) FLUX.1-dev and (b) PixArt-α, revealing strong, concept-dependent variation. Layer-Skipping We skip concept-relevant layers by redefining the residual mapping as an identity function for each skipped layer, similar to STG [21]: Res(zl)=fl(zl)+zl,Res¯(zl)=id(zl)=zl, (z_l)=f_l(z_l)+z_l, Res(z_l)=id(z_l)=z_l, (9) where zlz_l denotes the feature representation at the l-th layer, and flf_l represents the nonlinear transformation in layer l. In Res¯ Res, the block output is set equal to its input, thus bypassing flf_l. This preserves information flow while preventing additional perturbations, allowing for controlled modulation. Locating Layer-Directions in Latent Space The fundamental intuition behind our method is to decompose layer-wise predictions into a target direction y→ y and a residual r→ r. Let did_i be the noise prediction with layer i amplified, then di=αiy→+r→i,d_i= _i y+ r_i, (10) where αi≥0 _i≥ 0, and rir_i satisfies ⟨ri,y⟩=0 r_i,y =0. Then, the idea is to extract information about the magnitude of αi _i by profiling the effectiveness of single-layer skip-guidance. For each layer i, we generate a set of images by using the skipped prediction ϵ[θ∖i] _[θ i] as a negative direction to guide away from, similar to the unconditional noise prediction in Equation 5. We measure the target performance pip_i of these resulting images. Given this performance observation, the per-layer performance can be represented as a sum of αi _i plus a noise component ϵi _i: pi=αi+ϵi,p_i= _i+ _i, (11) where we assume that ϵii≥1\ _i\_i≥ 1 are i.i.d. with [ϵi]=0E[ _i]=0, and that ϵi _i is independent of (αi,r→i)( _i, r_i). Observing performances then yields a per-layer impact distribution for a given concept. The computational cost of this procedure scales linearly with the number of layers and the number of samples, i.e., (L⋅N)O(L· N) for L layers and N samples per layer. Notably, profiling is performed only once per model and concept. We make our layer analysis and code available at https://github.com/CompVis/concept_guidance. Pseudocode is provided in Algorithms 2 and 1. Figure 4: Concept Guidance Workflow. Left: In a one-time profiling stage, each layer is skipped individually, generations are scored, and top-k layers are selected. Right: At inference, per-layer skipped predictions are aggregated into a performance-weighted negative prediction; CoG then extrapolates from beyond the standard prediction. Performance-Weighted Multi-Layer Guidance To approximate y→ y, we compute predictions for the top-k best performing layers and weigh their impact on the overall prediction using their observed performance pip_i. Let K be the set of best-performing layers, then we compute an individual negative noise prediction with layer i skipped for all i∈Ki∈ K. Each prediction is weighed by the performance term pip_i relative to the target performance without layer-skipping p∅p_ . Specifically, the weight ωi _i for the prediction with layer i skipped is given by: ωi=pi−p∅. _i=p_i-p_ . (12) Figure 5: Optimizing aesthetics with CoG improves perceptual quality. Side-by-side examples compare CFG vs. CoG. Thus, each layer contributes to the guidance process only insofar as it improves target performance compared to standard generation. The final negative noise prediction is then given by a weighted mean over all k noise predictions: ϵneg=∑i∈Kωi⋅ϵ[θ∖i]∑i∈Kωi. _neg= _i∈ K _i· _[θ i] _i∈ K _i. (13) Finally, we extrapolate ϵneg _neg beyond the standard noise prediction ϵθ _θ: ϵCoG=(1−λ)ϵneg+λϵθ, _CoG=(1-λ) _neg+λ _θ, (14) where ϵθ _θ is given by CFG according to Equation 5, and λ≥1λ≥ 1 controls the guidance strength. Thus, CoG integrates seamlessly with CFG and allows easily combining general prompt adherence with target-specific guidance. 4 Experiments Figure 6: CoG improves fine-grained local coherence for hard failure cases like visible text and hands. Side-by-side examples compare CFG vs. CoG. 4.1 Setup We evaluate our method on multiple models with varying architecture, size, and output quality, such as PixArt-α α [9], Stable Diffusion 3 [12, 42], Stable Diffusion 3.5 [43] and FLUX.1-dev [6]. As targets we choose text and hand generation since text-to-image diffusion models notoriously struggle with those concepts. To provide evidence that CoG also extends to more abstract concepts we choose general aesthetics. We evaluate with EasyOCR [47], a pretrained model from MediaPipe [27] and a pretrained model for evaluating aesthetics [40] based on CLIP embeddings [35]. Furthermore, we investigate if improving a target concept with CoG creates a trade-off for overall image quality using HPSv3 [30], which aligns strongly with human preferences. Throughout, all guidance methods are evaluated on identical prompts, seeds, and scheduler settings, so reported improvements correspond to paired comparisons. Beyond these metrics, we show that CoG also generalizes to concepts without a hand-crafted metric by using a vision-language model as the scoring judge (Appendix C), and we release our per-model, per-concept layer configurations (Table 6). 4.2 Layer Analysis We derive insights on layer-concept interactions from computing per-layer, per-concept MI according to Equation 8. Evaluating MI across all tasks for FLUX.1-dev and PixArt-α (see Figure 3 (a) and (b)) reveals that (i) there exist large concept-dependent differences, which supports our claim that networks distribute concept-specific knowledge non-uniformly across layers and (i) we find that layers with high per-concept MI are often located in the middle of the network. To validate our findings, we evaluate correlations between per-layer MI and the measured layer performance on the target metric, and find very high correlations in the range of [0.692,0.956][0.692,0.956]. This strongly reinforces our intuition that performance-weighted, multi-skip Concept Guidance works by extrapolating the influence of the layers that are most responsible for the generation of target concepts. 4.3 Qualitative Results CoG consistently improves visual quality across all concepts. We present comparisons of CFG and CoG in Figure 5 and Figure 6. We further provide extensive uncurated comparisons of CoG and several baselines in Figures 14 and 15. Table 1: Concept Guidance achieves better performance across measured tasks. † does not generate visible text. Model Method Text (↑ ) Hands (↑ ) Aesthetics (↑ ) PixArt-α† CFG - 0.357 6.667 CoGsingle - 0.496 6.709 CoGmulti - 0.509 ▲ 42% 6.722 ▲ 0.8% SD3 CFG 0.476 0.651 6.197 CoGsingle 0.483 0.674 6.333 CoGmulti 0.490 ▲ 2.9% 0.684 ▲ 5.0% 6.348 ▲ 2.4% SD3.5 CFG 0.580 0.646 6.276 CoGsingle 0.598 0.662 6.408 ▲ 2.1% CoGmulti 0.610 ▲ 5.2% 0.679 ▲ 5.1% 6.368 FLUX.1-dev CFG 0.481 0.672 6.448 CoGsingle 0.499 0.655 6.631 CoGmulti 0.510 ▲ 6.1% 0.675 ▲ 0.5% 6.657 ▲ 3.2% Table 2: Concept Guidance can be applied to multiple concepts at once. † does not generate visible text. Text (↑ ) + Hands (↑ ) Text (↑ ) + Aesthetics (↑ ) Hands (↑ ) + Aesthetics (↑ ) Model Method Text Hands Text Aes. Hands Aes. PixArt-α† CFG - - - - 0.330 6.217 CoGsingle - - - - 0.252 6.231 CoGmulti - - - - 0.334 6.260 SD3 CFG 0.353 0.423 0.416 6.213 0.631 5.922 CoGsingle 0.354 0.502 0.396 6.222 0.720 5.915 CoGmulti 0.356 0.503 0.424 6.282 0.745 5.938 SD3.5 CFG 0.498 0.620 0.637 5.974 0.330 6.217 CoGsingle 0.493 0.655 0.634 5.861 0.252 6.260 CoGmulti 0.508 0.693 0.642 5.893 0.334 6.231 FLUX.1-dev CFG 0.346 0.554 0.377 6.606 0.668 6.205 CoGsingle 0.354 0.494 0.418 6.628 0.677 6.215 CoGmulti 0.377 0.538 0.431 6.657 0.710 6.301 Complex Compositional Tasks. Concept Guidance improves tasks that require fine-grained structural coherence and correct object interactions. For human hands, CoG visibly reduces common artifacts and produces more plausible results, particularly in complex scenarios where hands interact with other objects (Figure 6 d). When generating visible text, CoG consistently generates text more faithful to the prompt. In some cases, CoG even successfully renders complete and correct text where CFG fails to produce any readable output (Figure 6 b). General Concepts. We find that Concept Guidance also improves concept-specific alignment beyond notorious failure cases of T2I models. When optimizing for the more general concept of aesthetics, CoG yields both general and prompt-specific enhancements. Overall, we find that CoG produces images with improved lighting, contrast, and compositional symmetry (Figure 5 c). Moreover, CoG adapts stylistic elements to the prompt. Human faces exhibit sharper features, and healthier skin tones (Figure 5 a). Architectural scenes appear more modern, clean, and luxurious (Figure 5 b). Lastly, natural landscapes display greater visual diversity, for instance, through a richer variety of vegetation (Figure 5 d). 4.4 Quantitative Results We evaluate CoG with multiple performance-weighted layer skips (CoGmulti) using four models and three different target concepts, and compare our method against Classifier-Free Guidance [16] and Concept Guidance with only a single, target-optimized skipped layer (CoGsingle). Across all settings and models, CoGmulti consistently outperforms both CFG and CoGsingle (see Table 2): Concept Guidance achieves an average performance increase of 8.1%8.1\% compared to CFG, and at its best, a 42%42\% increase for hand generation with PixArt-α. We also find consistent improvements across models. CoG achieves an average target performance increase of 3.3%3.3\% for FLUX.1-dev [6], 3.5%3.5\% for SD3 [42], 3.9%3.9\% for SD3.5 [43], and 22%22\% for PixArt-α [9]. Regarding different tasks, CoG yields stable but moderate increases for aesthetics, and higher increases for tasks that require high local coherence. Specifically, CoG increases target performance by 2.0%2.0\% for aesthetics, by 4.7%4.7\% for text, and by 13.4%13.4\% for hands. We suspect localized tasks are especially sensitive to guidance accuracy. Our intuition is that CoG’s improvements are caused by a better-aligned update direction. Figure 7: Multi-target Concept Guidance simultaneously improves multiple concepts on FLUX.1-dev compared to CFG. Combined Targets. We then evaluate CoG on multiple targets at once. Here, we only use layers for skipping that increase performance for all targets, and choose those that yield the highest combined gain. We find that while synergies between targets vary, CoG again consistently outperforms CFG (see Table 2). Using only a single layer for guidance is often worse than CFG, indicating that weighted skipping of multiple layers is essential for optimizing several targets. We show qualitative results in Figure 7, where CoG jointly resolves common failure cases. Table 3: Evaluation Metrics. Comparison against APG [38] and PAG [1]. (a) SD3 & SD3.5 Model Method Aesth. Hands Text SD3PixArt-α CFG 6.358 0.664 0.477 CFG+APG [38] 6.524 0.590 0.396 PAG [1] 6.354 0.687 0.481 CoG 6.494 0.707 0.491 CoG+APG 6.600 0.746 0.410 SD3.5PixArt-α CFG 6.293 0.475 0.504 CFG+APG [38] 6.462 0.535 0.463 PAG [1] 6.200 0.352 0.491 CoG 6.344 0.618 0.514 CoG+APG 6.499 0.733 0.476 (b) PixArt & FLUX Model Method Aesth. Hands Text PixArt-α CFG 6.622 0.388 – CFG+APG [38] 7.096 0.399 – PAG [1] 6.606 0.401 – CoG 6.658 0.437 – CoG+APG 7.143 0.499 – FLUX.1-dev CFG 6.585 0.727 0.608 CFG+APG [38] 6.650 0.747 0.472 PAG [1] 6.652 0.626 0.629 CoG 6.659 0.754 0.650 CoG+APG 6.573 0.738 0.586 Table 4: Auxiliary Metrics. Model Method KDD↓ CLIP↑ LPIPS↑ PixArt-α CFG 1.589 31.05 0.649 CoGtext – – – CoGhands 2.127 30.81 0.584 CoGaesth. 1.679 30.98 0.655 SD3PixArt-α CFG 0.869 31.97 0.635 CoGtext 0.961 31.77 0.650 CoGhands 0.967 31.77 0.645 CoGaesth. 0.955 31.69 0.653 Table 5: Human Preference Scores favor CoG over CFG on SD3.5 [43]. Especially when optimizing for aesthetics, CoG yields preferred images. Text (↑ ) Hands (↑ ) Aesthetics (↑ ) Mean (↑ ) CFG [16] 8.991 6.298 7.832 7.707 CoGmulti 8.945 6.552 8.277 7.925 Win Rate 48.4% 62.6% 74.6% 61.87% Stronger Baselines. We further compare CoG against two state-of-the-art training-free guidance methods: Adaptive Projected Guidance (APG) [38] and Perturbed-Attention Guidance (PAG) [1]. As APG and PAG improve general guidance behavior while CoG contributes a concept-aware guidance direction, the two are complementary and can be combined (CoG+APG). As shown in Table 3 (a, b), CoG improves over CFG in every setting, and either CoG or CoG+APG is best overall in every cell. Complementarity is clearest for aesthetics and hands, where CoG+APG wins, while CoG alone is better for text. Quality and Diversity. We emphasize that CoG does not aim to improve general generation quality, but rather provides systematic, concept-aware inference-time steering; we therefore report auxiliary metrics to characterize the trade-off between general and concept-specific quality. We use Kernel DINO Distance (KDD) instead of FID, as FID is poorly aligned with perceptual quality for state-of-the-art models [44, 50], as well as CLIP-Score and LPIPS (Table 5). CLIP-Score remains close to CFG, LPIPS shows no systematic diversity collapse, and KDD reflects the expected fidelity/control trade-off. Consistently, the HPSv3 scores reported below measure overall generation quality while steering toward a concept, rather than concept quality itself. Human Preference Comparison. We use HPSv3 [30] as a preference-based proxy to probe perceptual trade-offs under target optimization and present absolute values and win-rates vs. CFG in Table 5. We explicitly do not optimize Concept Guidance for HPSv3: we keep the standard, per-target settings and evaluate 5,0005,000 samples per target on SD3.5 [43]. CoG is preferred for Hands (62.6%) and Aesthetics (74.6%), indicating that CoG is preferred by humans, especially when optimizing aesthetics. Text is near parity, indicating a minor trade-off. 4.5 Ablation Studies Here we summarize three ablations; the detailed discussion, figures (Figures 10 and 10) and tables are provided in Appendix D. (i) Per-concept layers: selecting skipped layers per concept, rather than using a single fixed layer as in STG [21], improves target performance by up to 24.5%24.5\% and on average 5.6%5.6\% (Table 7). (i) Number of skipped layers: performance improves with k up to a sweet spot around k=2k=2–33, with most of the gain already obtained from a single relevant layer, while larger k eventually introduces interference between layer directions (Figure 10, Table 9); the additional inference cost is therefore opt-in (Section E.3). (i) Aggregation strategy: performance-weighted aggregation of separate per-layer predictions outperforms both Naive single-pass multi-skipping (as in the HuggingFace SD3 pipeline [42]) and Uniform weighting (Figure 10, Table 9), suggesting that popular diffusion pipelines could be improved by incorporating our method. Finally, CoG is robust to the choice of guidance scale λ (Table 11). Limitations. Concept Guidance applies to concepts with a meaningful scoring signal: either an automatic metric or a VLM-based judge (Appendix C). Layer profiling is performed once per model/concept pair and should not be assumed to transfer across backbones, so rankings must be recomputed for new models (we release our configurations in Table 6 to avoid this cost for the models we study). At inference, CoG adds one noise prediction per skipped layer, increasing latency with k (Section E.3). In practice, a single layer already yields most of the benefit. Finally, CoG steers a targeted concept rather than improving general generation quality resulting in a potential trade-off. 5 Conclusion We introduced Concept Guidance, a simple, general, and effective mechanism for precise latent control in text-to-image diffusion models. By identifying and amplifying the influence of concept-specific layers, CoG solves persistent, well-known failure cases of standard guidance–like misspelled text and malformed hands – but can also optimize for more general concepts like overall aesthetics. Concept Guidance’s key strength is its usability: it is a plug-and-play component that integrates seamlessly with CFG, requires no training, gradients, or external models, and can be added to any existing pipeline with minimal modification. Acknowledgements This work has been supported by the Horizon Europe project ELLIOT (GA No. 101214398), the German Federal Ministry for Economic Affairs and Energy within the project “NXT GEN AI METHODS – Generative Methoden für Perzeption, Prädiktion und Planung”, the project “GeniusRobot” (01IS24083) funded by the Federal Ministry of Research, Technology and Space (BMFTR), and the BMWE ZIM-project (No. K5785001LO4) “conIDitional LoRA”. The authors gratefully acknowledge the Gauss Center for Supercomputing for providing compute through the NIC on JUWELS/JUPITER at JSC and the HPC resources supplied by the NHR@FAU Erlangen. Furthermore, this work was partially supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Research, Technology and Space. References [1] D. Ahn, H. Cho, J. Min, W. Jang, J. Kim, S. Kim, H. H. Park, K. H. Jin, and S. Kim (2024) Self-rectifying diffusion sampling with perturbed-attention guidance. In European Conference on Computer Vision, p. 1–17. Cited by: §2, §4.4, Table 3, Table 3, Table 3, Table 3, Table 3. [2] O. Avrahami, O. Patashnik, O. Fried, E. Nemchinov, K. Aberman, D. Lischinski, and D. Cohen-Or (2024) Stable flow: vital layers for training-free image editing. arXiv preprint arXiv:2411.14430. External Links: Link Cited by: §2. [3] A. Bansal, H. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein (2023) Universal guidance for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 843–852. Cited by: §1, §2. [4] S. Basu, K. Rezaei, P. Kattakinda, R. Rossi, C. Zhao, V. Morariu, V. Manjunatha, and S. Feizi (2024) On mechanistic knowledge localization in text-to-image generative models. arXiv preprint arXiv:2405.01008. External Links: Link Cited by: §2. [5] S. Basu, N. Zhao, V. Morariu, S. Feizi, and V. Manjunatha (2023) Localizing and editing knowledge in text-to-image generative models. arXiv preprint arXiv:2310.13730. External Links: Link Cited by: §2. [6] Black Forest Labs, (2024) FLUX.1-dev. Note: https://huggingface.co/black-forest-labs/FLUX.1-dev Cited by: §E.4, §4.1, §4.4. [7] H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or (2023) Attend-and-excite: attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826. External Links: Link Cited by: §2. [8] J. Chen, Y. Huang, T. Lv, L. Cui, Q. Chen, and F. Wei (2023) Textdiffuser: diffusion models as text painters. Advances in Neural Information Processing Systems 36, p. 9353–9387. Cited by: §1. [9] J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, et al. (2023) Pixart-alpha: fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426. Cited by: §E.4, §1, §4.1, §4.4. [10] H. Chung, J. Kim, G. Y. Park, H. Nam, and J. C. Ye (2024) CFG++: manifold-constrained classifier free guidance for diffusion models. arXiv preprint arXiv:2406.08070. Cited by: §2. [11] P. Dhariwal and A. Nichol (2021) Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, p. 8780–8794. Cited by: §1, §2, §3.1. [12] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. (2024) Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: §1, §3.1, §4.1. [13] R. Gandikota, J. Materzyńska, T. Zhou, A. Torralba, and D. Bau (2024) Concept sliders: lora adaptors for precise control in diffusion models. In European Conference on Computer Vision, p. 172–188. Cited by: §2. [14] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or (2022) Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626. Cited by: §1, §2. [15] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §3.1. [16] J. Ho and T. Salimans (2022) Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. Cited by: Figure 8, §E.4, Table 10, Figure 14, Figure 15, Appendix F, §1, §2, §3.1, §4.4, Table 5. [17] S. Hong, G. Lee, W. Jang, and S. Kim (2023) Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 7462–7471. Cited by: §2. [18] S. Hong (2024) Smoothed energy guidance: guiding diffusion models with reduced energy curvature of attention. Advances in Neural Information Processing Systems 37, p. 66743–66772. Cited by: §2. [19] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §2. [20] W. Huang, Y. Jiang, T. Van Wouwe, and C. K. Liu (2024) Constrained diffusion with trust sampling. arXiv preprint arXiv:2411.10932. External Links: Link Cited by: §2. [21] J. Hyung, K. Kim, S. Hong, M. Kim, and J. Choo (2025) Spatiotemporal skip guidance for enhanced video diffusion sampling. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 11006–11015. Cited by: Appendix D, Table 7, Table 7, Table 7, Table 7, §E.4, Figure 14, Figure 15, Appendix F, §2, §3.2, §3.2, §4.5. [22] T. Karras, M. Aittala, T. Kynkänniemi, J. Lehtinen, T. Aila, and S. Laine (2024) Guiding a diffusion model with a bad version of itself. Advances in Neural Information Processing Systems 37, p. 52996–53021. Cited by: §2. [23] J. Kim, E. Esmaeili, and Q. Qiu (2025) Text embedding is not all you need: attention control for text-to-image semantic alignment with text self-attention maps. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 8031–8040. Cited by: §1. [24] X. Kong, O. Liu, H. Li, D. Yogatama, and G. V. Steeg (2024) Interpretable diffusion via information decomposition. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. [25] T. Li, W. Luo, Z. Chen, L. Ma, and G. Qi (2024) Self-guidance: boosting flow and diffusion generation on their own. arXiv preprint arXiv:2412.05827. External Links: Link Cited by: §2. [26] N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum (2022) Compositional visual generation with composable diffusion models. In European conference on computer vision, p. 423–439. Cited by: §1. [27] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C. Chang, M. G. Yong, J. Lee, et al. (2019) Mediapipe: a framework for building perception pipelines. arXiv preprint arXiv:1906.08172. Cited by: §4.1. [28] G. Luo, T. Darrell, O. Wang, D. B. Goldman, and A. Holynski (2023) Readout guidance: learning control from diffusion features. arXiv preprint arXiv:2312.02150. External Links: Link Cited by: §2. [29] G. Luo, L. Dunlap, D. H. Park, A. Holynski, and T. Darrell (2023) Diffusion hyperfeatures: searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems 36, p. 47500–47510. Cited by: §2. [30] Y. Ma, X. Wu, K. Sun, and H. Li (2025) Hpsv3: towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 15086–15095. Cited by: §4.1, §4.4. [31] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, and Y. Shan (2024) T2i-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 4296–4304. Cited by: §1, §2. [32] S. Narasimhaswamy, U. Bhattacharya, X. Chen, I. Dasgupta, S. Mitra, and M. Hoai (2024) Handiffuser: text-to-image generation with realistic hand appearances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2468–2479. Cited by: §1. [33] Q. H. Nguyen, H. Phan, and K. D. Doan (2024) Unveiling concept attribution in diffusion models. arXiv preprint arXiv:2412.02542. External Links: Link Cited by: §2. [34] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen (2021) Glide: towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741. Cited by: §1, §3.1. [35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §4.1. [36] J. Rajabi, S. Mehraban, S. Sadat, and B. Taati (2025) Token perturbation guidance for diffusion models. arXiv preprint arXiv:2506.10036. External Links: Link Cited by: §2. [37] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695. Cited by: §1, §3.1. [38] S. Sadat, O. Hilliges, and R. M. Weber (2024) Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations, Cited by: §2, §4.4, Table 3, Table 3, Table 3, Table 3, Table 3. [39] S. Sadat, M. Kansy, O. Hilliges, and R. M. Weber (2024) No training, no problem: rethinking classifier-free guidance for diffusion models. arXiv preprint arXiv:2407.02687. External Links: Link Cited by: §2. [40] C. Schuhmann (2022) LAION-aesthetics. Note: LAION Blog External Links: Link Cited by: §4.1. [41] J. Song, C. Meng, and S. Ermon (2020) Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Cited by: §3.1. [42] Stability AI, (2024) Stable diffusion 3 medium. Note: https://huggingface.co/stabilityai/stable-diffusion-3-medium Cited by: Appendix D, §E.4, Figure 14, Figure 15, Appendix F, §2, §3.2, §4.1, §4.4, §4.5. [43] Stability AI, (2024) Stable diffusion 3.5 large. Note: https://huggingface.co/stabilityai/stable-diffusion-3.5-large Cited by: §E.4, §4.1, §4.4, §4.4, Table 5, Table 5. [44] G. Stein, J. Cresswell, R. Hosseinzadeh, Y. Sui, B. Ross, V. Villecroze, Z. Liu, A. L. Caterini, E. Taylor, and G. Loaiza-Ganem (2023) Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems 36, p. 3732–3784. Cited by: §4.4. [45] L. Tang, M. Jia, Q. Wang, C. P. Phoo, and B. Hariharan (2023) Emergent correspondence from image diffusion. Advances in neural information processing systems 36, p. 1363–1389. Cited by: §2. [46] R. Tang, L. Liu, A. Pandey, Z. Jiang, G. Yang, K. Kumar, P. Stenetorp, J. Lin, and F. Ture (2022) What the daam: interpreting stable diffusion using cross attention. arXiv preprint arXiv:2210.04885. External Links: Link Cited by: §2. [47] D. Vedhaviyassh, R. Sudhan, G. Saranya, M. Safa, and D. Arun (2022) Comparative analysis of easyocr and tesseractocr for automatic license plate recognition using deep learning algorithm. In 2022 6th International Conference on Electronics, Communication and Aerospace Technology, p. 966–971. Cited by: §4.1. [48] C. WANG, G. Franzese, A. Finamore, M. Gallo, and P. Michiardi (2025) Information theoretic text-to-image alignment. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2, §3.2. [49] Q. Wu, Y. Liu, H. Zhao, T. Bui, Z. Lin, Y. Zhang, and S. Chang (2023) Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 7766–7776. Cited by: §1. [50] J. Yang, Z. Geng, X. Ju, Y. Tian, and Y. Wang (2026) Representation fr\ ’echet loss for visual generation. arXiv preprint arXiv:2604.28190. Cited by: §4.4. [51] J. Yu, Y. Wang, C. Zhao, B. Ghanem, and J. Zhang (2023) FreeDoM: training-free energy-guided conditional diffusion model. arXiv preprint arXiv:2303.09833. External Links: Link Cited by: §2. [52] R. Zawar, S. Dewan, P. Saxena, Y. Chang, A. Luo, and Y. Bisk (2024) DiffusionPID: interpreting diffusion via partial information decomposition. arXiv preprint arXiv:2406.05191. External Links: Link Cited by: §2. [53] L. Zhang, A. Rao, and M. Agrawala (2023) Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, p. 3836–3847. Cited by: §1. Supplementary Material Appendix A Layer Indices and Weights To facilitate reproducibility and allow future work to apply Concept Guidance without the computational cost of layer profiling, we provide the exact configurations used in our experiments. For each model and target concept, we list the set of top-k layer indices K identified as most impactful, along with their corresponding importance weights ωi _i. As defined in Section 3.2, the weights represent the performance gain of the skipped layer relative to the baseline. Table 6: Layer Configurations. Top-k skipped layers (K) and their corresponding weights (Ω ) for all evaluated models and target concepts. Model Task k Skipped Layers (K) Weights (Ω ) PixArt-α Human Hands [4] [3, 23, 8, 18] [1.00, 0.88, 0.73, 0.63] Aesthetics [3] [9, 20, 5] [1.00, 0.28, 0.12] SD3 Visible Text [3] [7, 11, 4] [1.00, 0.55, 0.45] Human Hands [4] [9, 6, 5, 10] [1.00, 0.48, 0.45, 0.21] Aesthetics [3] [6, 7, 8] [1.00, 0.71, 0.70] SD3.5 Visible Text [2] [15, 10] [1.00, 0.49] Human Hands [2] [7, 18] [1.00, 0.98] Aesthetics [1] [4] [1.00] FLUX.1-dev Visible Text [3] [11, 3, 9] [1.00, 0.85, 0.74] Human Hands [3] [9, 12, 15] [1.00, 0.81, 0.78] Aesthetics [3] [15, 14, 13] [1.00, 0.18, 0.09] Appendix B Convergence Analysis Empirically, as demonstrated in Section 4.5, we find that CoG approximates the latent space direction of highest target performance y→ y more precisely for a larger k, at least until a certain threshold. Here, we formalize our intuition and prove that under idealized assumptions, CoG predictions converge to y→ y as the number of skipped layers goes to infinity. In practice, the number of skippable layers is of course bounded by the model architecture. Lemma 1 Let di=αiy→+rid_i= _i y+r_i be the noise prediction of layer i, where y→ y is the unit ground-truth direction, αi _i represents signal strength, and rir_i is a zero-mean residual vector. Let K be the set of k layers selected by CoG. The normalized CoG estimator y^k=∑i∈di‖∑i∈di‖ y_k= _i d_i \| _i d_i \| converges to the true direction y→ y as the number of selected layers k→∞k→∞, provided that the expected signal strength of selected layers is positive, i.e. [αi∣i∈]=μ>0E[ _i i ]=μ>0. Proof Let sk=1k∑i∈dis_k= 1k _i d_i be the mean vector of the selected layers. Substituting the decomposition di=αiy→+rid_i= _i y+r_i into the sum, we have: sk=(1k∑i∈αi)y→+(1k∑i∈ri).s_k= ( 1k _i _i ) y+ ( 1k _i r_i ). We analyze the asymptotic behavior of the two terms on the right-hand side as k→∞k→∞ by applying the Law of Large Numbers (LLN): 1. Signal Term: The coefficient of y→ y is the sample mean of the signal strengths αi _i. By the LLN, this converges to the conditional expectation of α: 1k∑i∈αi→k→∞μ. 1k _i _i k→∞μ. 2. Residual Term: The second term is the sample mean of the independent residual vectors rir_i. Since [ri]=E[r_i]=0, the LLN dictates: 1k∑i∈ri→k→∞. 1k _i r_i k→∞0. Combining these results, the unnormalized estimator converges to the scaled ground truth: sk→k→∞μy→.s_k k→∞μ y. Since μ>0μ>0, the magnitude ‖sk‖\|s_k\| converges to μ‖y→‖=μ\| y\|=μ. Finally, the normalized estimator y^k y_k satisfies: y^k=sk‖sk‖→k→∞μy→μ=y→. y_k= s_k\|s_k\| k→∞ μ yμ= y. Appendix C Optimizing Arbitrary Concepts with VLMs. While we are bound to concepts for which valid metrics exist when reporting results in the experiment section, we find that using vision-language models (VLMs) as judges for layer-finding makes Concept Guidance available for arbitrary concepts, even when no valid metrics exist. We demonstrate the broad applicability of Concept Guidance by optimizing for the following additional targets (Figure 8): • Symmetry and Geometric Regularity: We optimize for images whose main subject displays strong structural symmetry and an overall regular geometric arrangement. This property is relevant across portraits, architecture, and design, yet it is difficult to measure. • Background Separation: We target images where the subject is clearly separated and visually dominant. Subject-background separation is a key component in product photography, close-up shots, and portraits. • Ukiyo-e Style: We optimize for images that match the visual characteristics of traditional Japanese ukiyo-e woodblock prints. This shows that Concept Guidance can target highly specific artistic styles. Figure 8: Comparison of Classifier-Free Guidance [16] and Concept Guidance for additional targets where layers are profiled using a VLM (InternVL3-14B). Appendix D Additional Ablation Results This section provides the detailed ablation studies summarized in Section 4.5. Leveraging Concept-Wise Layers To measure the effect of leveraging different layers for guidance per concept, we compare CoG against STG [21], where a fixed single layer is skipped based on overall generation quality, regardless of the given concept. While STG is a method for video generation guidance, we port the approach to text-to-image generation by choosing the best overall layer based on FID scores. To demonstrate the effect achieved by per-concept information alone, we do not leverage performance-weighted multi-skipping, i.e. we use CoGsingle only. We find that incorporating per-layer, per-concept knowledge when choosing the skipped layer yields significant performance increases of up to 24.5%24.5\%, and an average increase of 5.6%5.6\% (Table 7). Table 7: Leveraging per-concept layer information yields significant gains over guiding with a fixed layer as is the case in Spatio-Temporal Skip-Guidance. Model Method Text (↑) Hands (↑) Aesthetics (↑) PixArt-α STG [21] - 0.384 6.502 CoGsingle - 0.478 ▲ 24.48% 6.606 ▲ 1.6% SD3 STG [21] 0.441 0.662 6.359 CoGsingle 0.462 ▲ 4.8% 0.664 ▲ 0.3% 6.359 ▲ 0.0% SD3.5 STG [21] 0.531 0.540 6.297 CoGsingle 0.564 ▲ 6.2% 0.603 ▲ 11.7% 6.344 ▲ 0.7% FLUX.1-dev STG [21] 0.490 0.595 6.724 CoGsingle 0.500 ▲ 2.0% 0.643 ▲ 8.1% 6.865 ▲ 2.1% Number of Skipped Layers The idea behind CoG is to approximate the direction of highest target performance y→ y more accurately for a larger k, which we also demonstrate theoretically in our convergence analysis. However, we also expect that in practice, there is a tradeoff between increased accuracy and negative synergies between multiple noise predictions for a larger k. In particular, we find that while multiple layers can each be beneficial, their resulting predictions may diverge. Thus, skipping too many layers with CoG can result in guiding with possibly conflicting noise predictions. To find the optimal tradeoff, we evaluate models with up to 5 skipped layers, and measure their impact on task-specific metrics. As shown in Figure 10 and Table 9, performance indeed generally improves with more skipped layers up to a certain point, with the optimal tradeoff being usually achieved around 2–3 skipped layers. Notably, significant gains are already achieved by skipping a single, concept-relevant layer. Table 8: Performance-weighted aggregation is key for Concept Guidance. We compare skipping all layers in the same pass (Naive) equal weighting of multiple predictions (Uniform), and performance-based weighting (CoG). Model Metric Guidance Method Naive Uniform CoG (ours) PixArt-α Hands (↑ ) 0.481 0.482 0.509 Aesthetics (↑ ) 6.470 6.714 6.722 SD3 Text (↑ ) 0.453 0.477 0.490 Hands (↑ ) 0.571 0.669 0.684 Aesthetics (↑ ) 6.009 6.338 6.348 SD3.5 Text (↑ ) 0.575 0.591 0.610 Hands (↑ ) 0.664 0.677 0.679 Aesthetics (↑ ) 6.321 6.368 6.361 FLUX.1-dev Text (↑ ) 0.483 0.478 0.510 Hands (↑ ) 0.159 0.666 0.675 Aesthetics (↑ ) 6.628 6.607 6.657 Table 9: Increasing k trades off gains vs. interference, with a clear sweet spot. We sweep k (top-k improving layers); settings with fewer than five improving layers omit larger-k results. Across architectures, the optimum is approximately k=3k=3. Model Metric Number of Layers 1 2 3 4 5 PixArt-α† Hands (↑ ) 0.473 0.484 0.502 0.535 0.501 Aesthetics (↑ ) 6.680 6.709 6.729 - - SD3 Text (↑ ) 0.428 0.445 0.455 0.444 0.452 Hands (↑ ) 0.632 0.666 0.653 0.675 0.704 Aesthetics (↑ ) 6.194 6.257 6.259 6.191 6.169 SD3.5 Text (↑ ) 0.536 0.582 0.554 0.539 0.548 Hands (↑ ) 0.627 0.692 0.684 0.649 0.665 Aesthetics (↑ ) 6.376 6.295 6.309 6.291 6.295 FLUX.1-dev† Text (↑ ) 0.481 0.478 0.509 0.491 0.487 Hands (↑ ) 0.657 0.667 0.670 0.655 0.641 Aesthetics (↑ ) 6.652 6.654 6.662 - - Multiple Layers and Contribution Weighting To demonstrate the benefit of computing weighted averages of per-layer noise predictions, we compare our method against two additional baselines. For Uniform skip-guidance, separate noise predictions are computed per layer but are not weighted based on performance. In Naive skip-guidance, multiple layers are skipped within the same forward pass. Notably, Naive corresponds to the current implementation in the HuggingFace SD3 pipeline [42]. We hypothesize that Naive leads to degraded images due to strong manifold distortions induced by skipping several layers simultaneously. Moreover, we expect that without weighting, models underperform due to the lack of directional control on the noise manifold. Our experiments provide quantitative (Table 9) and qualitative (Figure 10) evidence that separate noise predictions are necessary and that contribution based weighting improves performance. Notably, our findings indicate that popular diffusion pipelines could be significantly improved by incorporating our method of computing separate predictions and combining them through weighted aggregation. Figure 9: Guidance Method Ablation. Figure 10: Number of Layers Ablation. Appendix E Implementation Details E.1 Code Availability Our implementation of Concept Guidance, together with the layer profiling code and the per-model, per-concept layer configurations reported in Table 6, is publicly available at https://github.com/CompVis/concept_guidance. E.2 Computational Resources For all experiments, models, and tasks, we use nodes of four NVIDIA A100 GPUs with 80 GBs of VRAM. E.3 Inference and Profiling Cost Concept Guidance introduces a one-time, offline profiling stage and a small online inference overhead. For profiling, we use N=100N=100 prompts per concept. For example, for SD3 with L=24L=24 layers this amounts to N×L=2400N× L=2400 generations and completes in ∼4 4 hours on a single A100. Profiling is performed only once per model/concept pair (Algorithms 2 and 1), and we release the resulting layer configurations (Table 6) so that CoG can be applied directly, without any profiling, for the models and concepts we study. At inference, the cost scales linearly with the number of skipped layers k, since each skipped layer requires one additional noise prediction (Table 10). The additional cost per skipped layer ranges from 0.40.4 s for PixArt-α to 12.412.4 s for FLUX.1-dev, i.e. it grows with model size; relative to a single CFG pass this corresponds to roughly 10%10\% for PixArt-α and up to nearly a full additional pass for FLUX.1-dev. As shown in Table 2, significant gains are already obtained with a single skipped layer (CoGsingleCoG_single), so the overhead is opt-in and can be tuned to the desired level of concept control. Table 10: Inference time (s) on an A100 GPU. Model CFG [16] CoGsingle (k=1) CoGmulti (k=2) +Sec / Layer PixArt-α 3.5 3.9 4.3 0.4 Stable Diffusion 3 4.4 6.2 8 1.8 Stable Diffusion 3.5 5.9 7.7 9.5 1.8 FLUX.1-dev 12.9 25.3 37.7 12.4 E.4 Guidance Scales Classifier-Free Guidance For CFG [16], we use the standard guidance scale per model. For Flux.1-dev [6], the default guidance scale is 3.53.5, for PixArt-α [9], it is 4.54.5, and for both Stable Diffusion 3 and 3.5 [42, 43], the default scale is 7.07.0. Concept Guidance To measure individual layer performance with residual skipping, we use a guidance scale of 2.02.0, as used in single-skip residual STG [21]. For CoG, we sweep over guidance scales ranging from 1.251.25 to 3.003.00 in 0.250.25 steps, and find that a guidance scale between 2.02.0 and 2.52.5 generally works best to achieve the results reported in Table 2. Full results are reported in Table 11. Table 11: CoG Guidance Scale Sweep. For most models and tasks, the optimal guidance scale falls in the range between 2.02.0 and 2.52.5. †PixArt-α does not generate any visible text. Model Metric Guidance Scale (λ) 1.25 1.50 1.75 2.00 2.25 2.50 2.75 3.00 PixArt-α† Text (↑ ) - - - - - - - - Hands (↑ ) 0.396 0.373 0.418 0.504 0.503 0.496 0.492 0.468 Aesthetics (↑ ) 6.647 6.678 6.695 6.723 6.716 6.759 6.740 6.734 SD3 Text (↑ ) 0.482 0.472 0.486 0.489 0.506 0.481 0.480 0.474 Hands (↑ ) 0.602 0.631 0.660 0.670 0.640 0.662 0.605 0.629 Aesthetics (↑ ) 6.169 6.198 6.250 6.248 6.257 6.260 6.217 6.241 SD3.5 Text (↑ ) 0.561 0.569 0.581 0.600 0.595 0.606 0.602 0.598 Hands (↑ ) 0.603 0.628 0.636 0.634 0.653 0.633 0.645 0.628 Aesthetics (↑ ) 6.324 6.337 6.349 6.419 6.388 6.378 6.351 6.356 FLUX.1-dev Text (↑ ) 0.502 0.509 0.521 0.515 0.524 0.521 0.515 0.503 Hands (↑ ) 0.642 0.666 0.695 0.617 0.617 0.620 0.635 0.638 Aesthetics (↑ ) 6.663 6.711 6.766 6.753 6.789 6.779 6.785 6.815 E.5 Conditional Prompts To ensure optimal diversity of conditional prompts, we generate prompts with different large language models. Specifically, we encourage LLMs to cover a wide range of prompts with respect to length, complexity and general theme. For instance, in text generation, we make sure that prompts contain a varying count of text instances, and that text within these instances has varying complexity. For text specifically, we find that strong models like FLUX.1-dev seldom struggle with prompts containing very simple text, thus we slightly adjust the complexity to model strength for evaluation. We provide sample prompts from our datasets for text (Figure 11), hands (Figure 12) and aesthetics (Figure 13). Figure 11: Prompt Samples for Text Generation. Figure 12: Prompt Samples for Hand Generation. Figure 13: Prompt Samples for Aesthetics. E.6 Algorithms We implement CoG according to Algorithm 1 and layer profiling according to Algorithm 2. Notably, once layers have been found using layer profiling, CoG can be added into existing pipelines by exchanging the layer-wise forward with a conditional forward that returns the input if for the current layer i, i∈Ki∈ K, according to Equation 9, and implementing guidance with the weighted negative noise prediction according to Equations 13 and 14. Algorithm 1 Concept Guidance (CoG) Step 1: xt,t,c,,Ω,CoG Scale λx_t,t,c,K, ,CoG Scale λ, CFG Scale γ 2: ϵunc,ϵcond←ϵθ(xt,t,∅),ϵθ(xt,t,c) _unc, _cond← _θ(x_t,t, ), _θ(x_t,t,c) 3: ϵθ←ϵunc+γ(ϵcond−ϵunc) _θ← _unc+γ( _cond- _unc) ⊳ Standard CFG 4: ϵsum←0,W←0 _sum← 0,W← 0 5: for i∈i do ⊳ Aggregate Skipped Predictions 6: ϵskip←ϵ[θ∖i](xt,t,c) _skip← _[θ i](x_t,t,c) 7: ϵsum←ϵsum+ωi⋅ϵskip _sum← _sum+ _i· _skip 8: W←W+ωiW← W+ _i 9: end for 10: ϵneg←ϵsum/W _neg← _sum/W 11: return (1−λ)ϵneg+λϵθ(1-λ) _neg+λ _θ ⊳ Apply CoG Algorithm 2 CoG Layer Profiling 1: Model θ, Prompts C, Metric Φ(⋅) (·), Top-k count 2: for c∈c do ⊳ Compute Baseline Performance 3: x0←Sample(θ,c)x_0 (θ,c) 4: p∅←Φ(x0)p_ ← (x_0) 5: end for 6: p∅←p∅/||p_ ← p_ /|C| 7: ℒ←[ ]L←[ ] ⊳ Initialize Impact List 8: for layer i∈1,…,Li∈\1,…,L\ do ⊳ Compute Layer Impact 9: for c∈c do 10: x0←Sample(θ∖i,c)x_0 ( _ i,c) 11: pi←Φ(x0)p_i← (x_0) 12: end for 13: pi←pi/||p_i← p_i/|C| 14: ωi←max(0,pi−p∅) _i← (0,p_i-p_ ) 15: Append (i,ωi)(i, _i) to ℒL 16: end for 17: Sort ℒL by ωi _i descending ⊳ Select Top Layers 18: ←i∣(i,ωi)∈ℒ[:k]K←\i (i, _i) [:k]\ 19: Ω←ωi∣i∈ ←\ _i i \ 20: return ,ΩK, Appendix F Uncurated Samples Complementing the qualitative results in Figures 5 and 6, we show uncurated samples for the typical failure case of text generation and for the general task of generating more aesthetic images. For both tasks, we choose random seeds and generate images with Classifier-Free Guidance [16], Naive Skip-Guidance (as found in SD3 [42]), Spatio-Temporal Skip-Guidance [21], and Concept Guidance, using the same seed for each method. Uncurated samples generated with Flux.1-dev are shown in Figure 14 for text, and in Figure 15 for aesthetics. Figure 14: Uncurated Comparison of Classifier-Free Guidance (CFG) [16], Naive Skip-Guidance as found in Stable Diffusion 3 [42], Spatio-Temporal Skip Guidance [21] and Concept Guidance for Text Generation (FLUX.1-dev). Figure 15: Uncurated Comparison of Classifier-Free Guidance (CFG) [16], Naive Skip-Guidance as found in Stable Diffusion 3 [42], Spatio-Temporal Skip Guidance [21] and Concept Guidance for Aesthetics (FLUX.1-dev).