Paper deep dive
OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models
Ali Aliev, Kamil Garifullin, Nikolay Yudin, Vera Soboleva, Alexander Molozhavenko, Ivan Oseledets, Aibek Alanov, Maxim Rakhuba
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/10/2026, 3:09:56 AM
Summary
OrthoFuse is a training-free method for merging orthogonal style and concept adapters in diffusion models. By leveraging the Riemannian manifold structure of Group-and-Shuffle (GS) orthogonal matrices, the method approximates geodesics between adapters and applies a spectra restoration transform to maintain generation quality, outperforming existing LoRA-based fusion techniques.
Entities (5)
Relation Signals (3)
GS-orthogonal matrices → forms → Riemannian manifold
confidence 95% · The set of GS-orthogonal matrices forms a smooth manifold.
OrthoFuse → merges → GS-orthogonal matrices
confidence 95% · our technique to merge two GS orthogonal matrices is capable of uniting concept and style features
OrthoFuse → improvesupon → LoRA
confidence 90% · Our method outperforms existing state-of-the-art LoRA-based approaches
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training data to adapt the model to a narrow task. However, there is an open question: how to combine several adapters tuned for different tasks into one which is able to yield adequate results on both tasks? Specifically, merging subject and style adapters for generative models remains unresolved. In this paper we seek to show that in the case of orthogonal fine-tuning (OFT), we can use structured orthogonal parametrization and its geometric properties to get the formulas for training-free adapter merging. In particular, we derive the structure of the manifold formed by the recently proposed Group-and-Shuffle ($\mathcal{GS}$) orthogonal matrices, and obtain efficient formulas for the geodesics approximation between two points. Additionally, we propose a $\text{spectra restoration}$ transform that restores spectral properties of the merged adapter for higher-quality fusion. We conduct experiments in subject-driven generation tasks showing that our technique to merge two $\mathcal{GS}$ orthogonal matrices is capable of uniting concept and style features of different adapters. To the best of our knowledge, this is the first training-free method for merging multiplicative orthogonal adapters. Code is available via the $\href{this https URL}{link}$.
Tags
Links
- Source: https://arxiv.org/abs/2604.05183v1
- Canonical: https://arxiv.org/abs/2604.05183v1
Trouble viewing inline? Open PDF directly →
Full Text
85,963 characters extracted from source content.
Expand or collapse full text
OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models Ali Aliev 111Equal contribution. ,1,222Correspondence: alievali0278@gmail.com, garifullin.k@hse.ru Kamil Garifullin 111Equal contribution. ,1,2,3,222Correspondence: alievali0278@gmail.com, garifullin.k@hse.ru Nikolay Yudin 1 Vera Soboleva 1,2,3 Alexander Molozhavenko 1 Ivan Oseledets 3 Aibek Alanov 1,2,3 Maxim Rakhuba 1 1HSE University, 2FusionBrain Lab, 3AXXX Abstract In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training data to adapt the model to a narrow task. However, there is an open question: how to combine several adapters tuned for different tasks into one which is able to yield adequate results on both tasks? Specifically, merging subject and style adapters for generative models remains unresolved. In this paper we seek to show that in the case of orthogonal fine-tuning (OFT), we can use structured orthogonal parametrization and its geometric properties to get the formulas for training-free adapter merging. In particular, we derive the structure of the manifold formed by the recently proposed Group-and-Shuffle (GS) orthogonal matrices, and obtain efficient formulas for the geodesics approximation between two points. Additionally, we propose a spectra restoration transform that restores spectral properties of the merged adapter for higher-quality fusion. We conduct experiments in subject-driven generation tasks showing that our technique to merge two GS orthogonal matrices is capable of uniting concept and style features of different adapters. To the best of our knowledge, this is the first training-free method for merging multiplicative orthogonal adapters. Code is available via the link: https://github.com/ControlGenAI/OrthoFuse. Figure 1: Overview of the proposed method OrthoFuse. By considering GS orthogonal adapters as elements of the manifold, we are able to draw curves between them to fuse adapters into one with common features of object and style in particular proportion. To facilitate the generation quality, we analyze the spectrum of the orthogonal blocks inside GS representation and propose a specific curve on the manifold, aiming to preserve the distribution of blocks’ eigenvalues. 1 Introduction The impressive generative abilities of diffusion models [25, 17] have fueled increasing interest in subject-driven generation [27, 8, 16] and stylization tasks [31]. Typically, such tasks are addressed by fine-tuning pre-trained models using a small dataset corresponding to a specific concept, which can be a particular object, individual, or artistic style. While current approaches made it possible to handle subject- and style-oriented generation independently, a critical challenge remains unresolved: generating images combining a user-defined subject with a user-defined artistic style. However, addressing this problem would significantly expand the capabilities of generative models and decrease training costs, offering a new level of control and creative freedom to users. Despite the promising advancements in merging LoRA [13] adapters, current research has yet to explore methods for combining multiplicative orthogonal adapters, recently introduced in [24]. At the same time, orthogonal adapters have been shown to deliver more stable training dynamics and reduce the risk of overfitting, making them a particularly appealing option for personalization and stylization tasks. Another notable benefit of orthogonal fine-tuning is that, compared to LoRA, multiplicative orthogonal adapters are able to preserve the spectral and Frobenius norm of the layer by design, which is hard to achieve in the case of LoRA. This unique property enables the seamless fusion of orthogonal adapters without any concern regarding differences in their magnitudes. However, the absence of studies focusing on the fusion of orthogonal adapters represents a notable gap in this domain, presenting a promising research direction for advancing more robust and effective subject-style synthesis in generative models. In this work, we introduce a novel training-free approach for merging orthogonal adapters built upon the foundations of GS-orthogonal parametrization proposed in [11]. By studying the properties of this matrix class, we demonstrate that they form a Riemannian manifold. This structure allows us to approximate geodesics between two orthogonal adapters and explore other meaningful curves on the manifold, ultimately facilitating the construction of an optimally merged adapter. Additionally, we examine the spectral properties of the resulting merged adapters and establish a strong correlation between the spectral distribution and the merging quality. A naive geodesic approach tends to produce a more “compressed-to-one” spectral distribution, which diminishes the expressiveness of the outputs by limiting both concept representation and style preservation. To overcome this issue, we propose applying the Cayley transform to the resulting curve, thereby restoring the spectral distribution and achieving a superior, optimized merge and generation quality (Figure 1). Overall our key contributions are as follows: • We investigate the set of GS-orthogonal matrices and show this set forms a Riemannian manifold. • Using these theoretical insights, we propose a method to approximate geodesics between GS-orthogonal matrices. • Through extensive experiments, we show that our training-free approach effectively fuses parameter-efficient orthogonal adapters, successfully combining both style and concept patterns. Our method outperforms existing state-of-the-art LoRA-based approaches as well as straightforward joint orthogonal fine-tuning. • To the best of our knowledge, we are the first to propose a training-free merging method specifically designed for orthogonal multiplicative adapters. 2 Related Work LoRA vs Orthogonal Fine-tuning. LoRA-based methods [13, 20, 30] have become the standard approach for parameter-efficient fine-tuning. However, prior work [24] shows that additive low-rank updates may distort neuron relationships, which are important for preserving generative semantics in diffusion models. To address this, orthogonal fine-tuning methods introduce multiplicative updates that preserve weight geometry via orthogonal parametrizations. While the original block-diagonal construction [24] is efficient, it limits interactions across parameter groups. To improve expressiveness, [11] propose combining multiple orthogonal blocks with permutations, enabling richer transformations while retaining computational efficiency. Adapter Merging. Merging independently trained adapters is commonly studied in the context of LoRA. Early approaches rely on simple weighted averaging [28], while more recent methods introduce either learnable or structured composition strategies. Training-based approaches such as MoLe [33] and ZipLoRA [29] learn how to combine multiple LoRA adapters via additional optimization, using layer-wise gating (MoLe) or fine-grained column-wise mixing coefficients (ZipLoRA). B-LoRA [7] leverages architectural modularity by selectively training specific components (e.g., attention layers) to better separate and combine content and style. In contrast, training-free methods such as K-LoRA [22] select adapters based on weight statistics, avoiding retraining but remaining sensitive to scale inconsistencies between independently trained LoRAs. Test-time and Representation-level Methods. Training-free personalization can also be achieved by modifying the generation process at inference time. For instance, RB-Modulation [26] steers reverse diffusion dynamics using reference-based objectives, without relying on adapter parametrizations or explicit adapter merging. In contrast, our work focuses on combining pre-trained adapters directly in parameter space via closed-form fusion. Subject–style separation is often approached via feature-level control mechanisms (e.g., StyleAligned [12], StyleDrop [32]), which guide generation by modulating internal representations. In contrast, OrthoFuse exploits the geometric structure of orthogonal weight manifolds, enabling composition directly in parameter space and avoiding scale inconsistencies of LoRA-based methods. Our method is also related to optimization on matrix manifolds [9, 2]; however, applying Riemannian geometry to merge orthogonal adapters for diffusion models remains largely unexplored. 3 Preliminaries 3.1 Diffusion Model Fine-tuning Fine-tuning based personalized image generation is a process of adapting the model’s weights to generate user-defined concept. To link a new concept to a unique text token and class name, for example “sks dog”, the model εθ _θ undergoes fine-tuning on a limited dataset of concept images C=xi=1NC=\x\_i=1^N, optimizing the following objective: minθp,t,z=ℰ(x),x∈C,ε[‖ε−εθ(t,zt,p)‖22], _θE_p,t,z=E(x),x , [ \| - _θ(t,z_t,p) \|_2^2 ], (1) where ℰE denotes the encoder that maps an image x to its latent representation z=ℰ(x)z=E(x). In the case of orthogonal fine-tuning [11], we exclusively optimize the multiplicative adapter A, which is GS-orthogonal matrix, while keeping the original model weights W fixed. As a result, the updated weights are defined as: W∗=AW^*=AW. 3.2 GS-orthogonal matrices Let us start with a definition of the GS-orthogonal matrix, which we use as the main building block of our orthogonal adapter in our work. We introduce a simplified definition that is used in our experiments, for the more general case see [11]. Recall also that a square matrix A is called orthogonal if it satisfies A⊤A=IA A=I. Definition 3.1 ([11]). An n×n× n orthogonal matrix A is called GS(PL,P,PR)(P_L,P,P_R)-orthogonal matrix with block size b×b× b if it can be represented in the following form: A=PLLPRPR,A=P_LLPRP_R, (2) where L=diag(L1,L2,…,Lnb)L=diag(L_1,L_2,…,L_ nb) and R=diag(R1,R2,…,Rnb)R=diag(R_1,R_2,…,R_ nb) are block-diagonal matrices with Li,Ri∈ℝb×bL_i,R_i ^b× b, and PL,P,PRP_L,P,P_R are permutation matrices. Theorem 1 shows that without the loss of generality we may assume that the diagonal blocks of L and R are set to be orthogonal. Theorem 1 ([11]). Let A be any orthogonal matrix from GS(PL,P,PR)(P_L,P,P_R). Then, A admits PL(LPR)PRP_L(LPR)P_R representation with the matrices L,RL,R consisting of orthogonal blocks. A certain choice of the permutation matrices PL,P,PRP_L,P,P_R depends on the application. For example, we set PR=IP_R=I and PL=P⊤P_L=P , which ensures that the identity matrix PLIPIPR=IP_LIPIP_R=I belongs to our class — a property vital for initialization. In this paper P is chosen to be the so-called perfect shuffle [10, 5], which maximizes the number of non-zero elements in the matrix [11]. Following [5], we denote perfect shuffle permutations as P(b,n)P_(b,n). Applying permutation P(b,n)P_(b,n) can be interpreted as the following procedure: firstly, it reshapes a length-n vector into a b×nbb× nb matrix in row-major order. Then, this matrix is transposed and flattened back into a vector (again in row-major order). For simplicity of notation, we replace GS(P(b,n)⊤,P(b,n),I)(P_(b,n) ,P_(b,n),I) with GS where it is unambiguous. We also note that this representation resembles Monarch matrices [5], which, however, did not consider orthogonality constraints. In this paper, we maintain block orthogonality during fine-tuning via the Cayley transform: for every block B of block-diagonal L or R the operation B=(I−K)−1(I+K),K⊤=−K.B=(I-K)^-1(I+K), K =-K. yields an orthogonal matrix from the special orthogonal group SO(N)SO(N) of orthogonal matrices with the determinant equal to 11. 3.3 Riemannian geometry An n-dimensional topological manifold ℳM is a space that is locally homeomorphic to ℝnR^n. It means that for every point p∈ℳp , there exists an open neighborhood U∋pU p and a homeomorphism φ:U→V :U→ V, where V⊂ℝnV ^n is open; the pair (U,φ)(U, ) is called a chart. A smooth manifold is a topological manifold equipped with an atlas (a collection of charts covering ℳM) such that all transition maps ψ∘φ−1ψ ^-1 between overlapping charts are smooth. This additional structure allows for defining smooth functions, curves, and maps on the manifold. A Riemannian manifold (ℳ,g)(M,g) is a smooth manifold endowed with a Riemannian metric g, which is a smoothly varying family of inner products gpg_p on each tangent space (denoted TpℳT_pM). This metric makes it possible to measure lengths of tangent vectors, angles between tangent vectors, and lengths of curves. By integrating the length of curves, one obtains an intrinsic distance function dg:ℳ×ℳ→ℝd_g:M×M , turning ℳM into a metric space. On a Riemannian manifold, geodesics generalize the notion of straight lines from Euclidean geometry. A geodesic is a curve γ:(−ε,ε)→ℳγ:(- , ) that is locally length-minimizing: for any two sufficiently close points on γ, the curve realizes the shortest path between them, as measured by dgd_g. A Lie group is a smooth manifold that is also a group, in which the group operations of multiplication and inversion are smooth maps. For more details see [19, 2, 1]. It is also a well-known fact (see, e.g., [19]) that some matrix classes, like orthogonal group O(N)O(N), special orthogonal group SO(N)=A∈O(N)∣det(A)=1⊆ℝN×NSO(N)=\A (N) (A)=1\ ^N× N and the set of fixed-rank matrices are smooth manifolds. This allows for leveraging their geometric properties in optimization tasks. For instance, one can interpret neural network weights as points on a manifold and apply Riemannian optimization methods [2] for efficient training. 4 Method Figure 2: Distribution of the eigenvalues of orthogonal fine-tuning adapters before and after merging. (Left): eigenvalues of LC1L_C^1. (Center): eigenvalues of LS1L_S^1. (Right): eigenvalues of B(0.5)B(0.5) (orange) and of BOrthoFuse(0.5)B_OrthoFuse(0.5) (blue). Eigenvalues are calculated for the query GS-orthogonal adapter of StableDiffusion-XL model. In this work, we aim to merge two GS-orthogonal adapters so that we obtain an adapter from the same structured class. To achieve this without additional training, we establish the geometry of GS-orthogonal matrices class. The following theorem allows us to treat GS-orthogonal matrices as manifold elements, which is our key theoretical result. Theorem 2. The set of GS(PL,P,PR)(P_L,P,P_R)-orthogonal matrices forms a smooth manifold. Proof. See Appendix A. ∎ Theorem 2 provides a way to connect manifold elements via interpretable curves. We show that, with the right choice of the curve, we can obtain a gradual transfer between adapters, allowing us to mix concept and style in the desired proportion. This curve will serve as the reasonable approximation of the local minimizing geodesics between two points. Now, assume that we have two GS-orthogonal matrices: AC=P⊤LCPRC,AS=P⊤LSPRS,A_C=P L_CPR_C, A_S=P L_SPR_S, (3) where ACA_C and ASA_S are weight update matrices trained on a certain concept and style respectively. One may assume that the local minimizing geodesics between two GS-orthogonal matrices is just a block-wise interpolation between the corresponding diagonal blocks of (LC,LS)(L_C,L_S) and (RC,RS)(R_C,R_S). However, the GS-orthogonal manifold exhibits a more complicated structure, and the local minimizing geodesics between two manifold points is resource-intensive to calculate. Fortunately, orthogonal fine-tuning yields matrices whose diagonal blocks lie close to the identity matrix. This empirical observation, which was also reported in [24], allows us to conclude that block-wise geodesics interpolation precisely approximates the exact local minimizing geodesics on the GS-orthogonal manifold. See Appendix B for more details. Now, let us see the procedure of connecting blocks in more detail. On SO(n)SO(n), a geodesic starting at BCB_C is B(t)=BCexp(tΩ)B(t)=B_C (t ) with Ω skew-symmetric (see [4, 2, 6, 18]). To make the geodesic reach BSB_S at t=1t=1 for an arbitrary pair of corresponding blocks BC,BS∈SO(n)B_C,B_S (n), we set Ω=−log(BS⊤BC) =- (B_S B_C). Thus we obtain the well‑known formula B(t)=BCexp(−t⋅log(BS⊤BC)),B(t)=B_C (-t· (B_S B_C)), (4) where t∈[0,1]t∈[0,1] and exp and log denote matrix exponent and matrix logarithm functions respectively. In practice, B(t)B(t) can be computed using the eigendecomposition BS⊤BC=UΛU∗B_S B_C=U U^* (since orthogonal matrix is always diagonalizable): B(t)=BCUexp(−t⋅log(Λ))U∗,B(t)=B_CU (-t· ( ))U^*, (5) which reduces to matrix functions that are separately applied to eigenvalues, and matrix multiplications that are highly efficient on GPUs. The fact that we apply our fusing operation for ACA_C and ASA_S block-wise plays a vital role in the resulting algorithm efficiency due to cubical time scaling of eigendecomposition. We empirically observe that merging orthogonal adapters draws the eigenvalues of the resulting matrix closer to 11 compared to the original components (see Figure 2). Since these eigenvalues control the orthogonal adapter’s rotation power, their convergence toward unity weakens the intended stylistic modifications, making it closer to an identity layer transform. To address this attenuation of style and to proactively enhance the adapter’s effect, we propose a spectra restoration procedure. A candidate for spectra restoration is a rotation of the eigenvalues on the complex unit circle. This operation corresponds to multiplying the complex phase of each eigenvalue by a scalar factor. For eigenvalues lying on a unit sphere near the point 11, the phase can be extracted by taking the logarithm, yielding a value in a subsegment i⋅(−α,α)⊂i⋅(−π,π)i·(-α,α)⊂ i·(-π,π) providing that initial orthogonal matrix is close to I. Formally, we propose the following approach: Brotated(t)=exp(η(t)log(B(t))),B_rotated(t)= (η(t) (B(t))), (6) where η(t)η(t) is smooth phase multiplier satisfying η(0)=η(1)=1,andη(1/2)=η0,η(0)=η(1)=1, η(1/2)= _0, (7) with η0 _0 being a hyperparameter. The condition η(0)=η(1)=1η(0)=η(1)=1 is to ensure that we restore initial adapters on the boarder. Based on our ablation studies (see Appendix I), we empirically found that a suitable solution consists of η0=2 _0=2 and a second-order polynomial η(t)η(t) satisfying property (7): η(t)=1+4t(1−t).η(t)=1+4t(1-t). (8) Figure 3: Ablation of the fusion parameter t from (7). When t=0t=0, the merged adapter reduces to a pure concept adapter, preserving identity with no stylization. When t=1t=1, the merged weights correspond to a pure style adapter. Intermediate values produce a continuous fusion curve between concept preservation and style strength, with t=0.6t=0.6 yielding the most balanced trade-off. The only problem with such an approach is that it is computationally demanding and requires matrices to be diagonalized for every t by contrast to (4). To improve computational efficiency, while staying close to (6), we do two approximation steps. First of all, we use the following straightforward propositions. Proposition 1. For a matrix B(t)∈SO(N)B(t) (N) the following equality holds: log(B(t))=B(t)−B(t)⊤2+(‖B(t)−I‖23), (B(t))= B(t)-B(t) 2+O (\|B(t)-I\|_2^3 ), as B(t)B(t) tends to I. Proof. See Appendix C. ∎ Proposition 2. exp(tK)=(I−t2K)−1(I+t2K)+(t3), (tK)= (I- t2K )^-1 (I+ t2K )+O(t^3), as t→0t→ 0. Proof. See [10, Chapter 11.3.1] (Padé Approximation Method for Exponent for p=q=1p=q=1). ∎ As a result, we arrive at the following proposition yielding a hardware-efficient way to apply spectra restoration. Proposition 3. Let η(t)=1+4t(1−t)η(t)=1+4t(1-t) and det(B(t)+I)≠0det(B(t)+I) =0. Then for BOrthoFuse(t)=(I−η(t)4(B(t)−B(t)⊤))−1⋅(I+η(t)4(B(t)−B(t)⊤)), splitB_OrthoFuse(t)&= (I- η(t)4(B(t)-B(t) ) )^-1·\\ ·& (I+ η(t)4(B(t)-B(t) ) ), split (9) it holds that BOrthoFuse(t)=Brotated(t)+(‖B(t)−I‖23),B_OrthoFuse(t)=B_rotated(t)+O (\|B(t)-I\|_2^3 ), as B(t)→IB(t)→ I and where BrotatedB_rotated is defined in (6) Proof. See Appendix D. ∎ To sum up, in our merging procedure we add a specific transformation for each diagonal block B(t)B(t) merged via local minimizing geodesics from (4). As a result, the spectrum of the resulting blocks is modified using (9) with η(t)η(t) defined in (8). 4.1 Practical Implementation Our method operates on top of any diffusion model whose layers can be fine-tuned with orthogonal adapters. Source of adapters. Both style and concept adapters are trained using the orthogonal parametrization described in Section 3.2: using the same notation as in (2), we express concept ACA_C and style ASA_S adapters as follows: AC=P(b,n)⊤LCP(b,n)RC,AS=P(b,n)⊤LSP(b,n)RS, split&A_C=P_(b,n) L_CP_(b,n)R_C,\\ &A_S=P_(b,n) L_SP_(b,n)\,R_S, split (10) where LC=diag(LC(1),…,LC(nb)),RC=diag(RC(1),…,RC(nb)),LS=diag(LS(1),…,LS(nb)),RS=diag(RS(1),…,RS(nb)). splitL_C=diag(L_C^(1),…,L_C^( nb)),\ &R_C=diag(R_C^(1),…,R_C^( nb)),\\ L_S=diag(L_S^(1),…,L_S^( nb)),\ &R_S=diag(R_S^(1),…,R_S^( nb)). split (11) Goal. Given ACA_C and ASA_S, the task is to construct a fused adapter A(t)A(t) controlled by the fusion parameter t∈[0,1]t∈[0,1], such that A(t)A(t) belongs to GS-orthogonal matrices class and contains a mixture of features extracted from ACA_C and ASA_S in particular proportion. Algorithm. The merge operation is performed independently for each pair of blocks that are either (LC(i),LC(i))(L_C^(i),L_C^(i)) or (RC(i),RS(i))(R_C^(i),R_S^(i)). The merging procedure consists of two steps: 1. Block-wise geodesic interpolation. Since the adapters consist of independent orthogonal blocks, the merging operation is performed block by block. For each pair of blocks (BC(i),BS(i))(B_C^(i),B_S^(i)) we compute their fused version using the block-wise geodesic interpolation defined in (5): B~(i)(t)=BC(i)Uexp(−t⋅log(Λ))U⊤, B^(i)(t)=B_C^(i)U (-t· ( ))U , (12) where BS⊤BC=UΛU⊤B_S B_C=U U . This produces an intermediate block corresponding to the fusion level t along the geodesic between the concept and style transformations. 2. Eigenvalue rotation. After going along the geodesic, we apply the eigenvalue rotation operation used during fine-tuning, described in (9): B(i)(t)=(I−η(t)4(B~(i)(t)−B~(i)(t)⊤))−1⋅(I+η(t)4(B~(i)(t)−B~(i)(t)⊤)). splitB^(i)(t)&= (I- η(t)4( B^(i)(t)- B^(i)(t) ) )^-1·\\ ·& (I+ η(t)4( B^(i)(t)- B^(i)(t) ) ). split (13) As a result, B(i)(t)B^(i)(t) serves as a final block of the merged style and concept adapter. Additionally, we implement an accelerated version of this algorithm that merges two GS-orthogonal adapters in under one second. Details are provided in Appendix K, where we also include pseudocode for both the OrthoFuse algorithm and its accelerated variant. Table 1: Quantitative comparison of OrthoFuse and baseline methods. Style Sim measures style fidelity using CLIP similarity with the reference style image. CLIP and DINO concept metrics evaluate semantic consistency with the original concept. The geometric mean combines style and concept metrics to summarize overall trade-off between style fidelity and concept preservation. Method Style Sim CLIP DINO Geo. Mean (Style, DINO) Geo. Mean (Style, CLIP) Merging time Training-based Joint training 0.48 0.79 0.67 0.57 0.62 1.5 hours ZipLoRA r=8r=8 0.49 0.74 0.55 0.52 0.6 4 minutes ZipLoRA r=64r=64 0.49 0.76 0.64 0.56 0.61 Training-free K-LoRA r=8r=8 0.46 0.76 0.55 0.5 0.59 <1<1 sec K-LoRA r=64r=64 0.49 0.76 0.56 0.52 0.61 OrthoFuse 0.61 0.68 0.51 0.56 0.64 <1<1 sec 5 Experiments Figure 4: Qualitative comparisons. We present images generated by OrthoFuse alongside those created using the compared baselines. OrthoFuse strikes an ideal balance between concept and style, preserving both the concept and style. 5.1 Datasets To evaluate the effectiveness of OrthoFuse, we used a diverse set of styles and object concepts. The style adapters were trained on styles from StyleDrop [31] and from the artistic style collection used in K-LoRA [22], covering both classical and contemporary artworks. The concept adapters were trained on object concepts sampled from the DreamBooth dataset. This setup allows us to assess OrthoFuse on a wide variety of visual domains and to validate its ability to preserve both stylistic and semantic attributes during fusion. 5.2 Experimental Details All experiments were conducted using the SDXL [23] model as the base model. Additional results for FLUX [3] are reported in Appendix E. Each adapter was trained separately. Concept adapters were fine-tuned using 44-55 images per subject, while style adapters were trained on a single style reference image. All adapters were trained with the number of blocks set to 3232. Our ablation study in Figure 3 demonstrates that the value t=0.6t=0.6 achieves the most robust fusion behavior across a wide range of style-concept combinations, providing stable style transfer while maintaining strong identity consistency. Unless stated otherwise, we report results obtained with t=0.6t=0.6. Additional insights regarding the impact of varying t can be found in the Appendix I, where we provide a more detailed examination of the effects of this parameter. Additionally, a comparison of generations obtained after block-wise geodesic interpolation alone versus after the subsequent eigenvalue rotation step is provided in Appendix F, illustrating the effect of the rotation on stylistic fidelity and concept preservation. 5.3 Evaluation Metrics We evaluate OrthoFuse using both semantic and stylistic similarity metrics. To assess concept preservation, we compute the CLIP similarity and DINO similarity between original and generated images. To evaluate style fidelity, we calculate the CLIP similarity between the generated images and the reference style image. This metric quantifies how well the artistic characteristics of the target style are transferred during the fusion process. 5.4 Quantitative Comparisons We compare OrthoFuse with three representative baselines: K-LoRA [22], ZipLoRA [29] and Joint Orthogonal Adapters Training (Joint). For a fair comparison, we evaluated these methods using two different LoRA ranks: (1) Rank r=8r=8, as in the K-LoRA [22] original paper; (2) Rank r=64r=64, chosen as the rank for original ZipLoRA [29] method and to match the number of parameters used in our orthogonal adapters. Note that a LoRA of rank 3232 roughly corresponds to an orthogonal adapter with the number of blocks set to 6464 in terms of parameter count. For the quantitative analysis, we used 66 concepts from the DreamBooth dataset and 1212 styles from StyleDrop and K-LoRA. This resulted in 72 concept-style combinations; for each, we generated 10 images per pair for evaluation. Table 1 reports the quantitative results averaged over all concept-style pairs. OrthoFuse achieves the highest CLIP style similarity, demonstrating the strongest style transfer among all compared methods. While its concept preservation metrics (CLIP and DINO similarity) are slightly lower than those of the best-performing baseline, this behavior is expected because strong stylization always moves the generation away from the original concept images. Additionally, the geometric mean of style and concept similarity is highest for OrthoFuse, indicating the best overall balance between style fidelity and concept retention. The Joint baseline, which trains an orthogonal adapter for both concept and style at the same time, achieves the highest concept-preservation scores (CLIP and DINO); however, we found out that it can favor the concept and ignore the style in several cases, as reflected in its low style similarity. Moreover, this method is training-based and requires additional fine-tuning for each concept-style pair, whereas OrthoFuse is entirely training-free. Compared to K-LoRA and ZipLoRA, OrthoFuse provides significantly stronger style transfer while maintaining competitive concept consistency, offering a more stable and reliable fusion across diverse concept–style combinations. 5.5 Qualitative Results Figure 4 presents qualitative comparisons between OrthoFuse and the baseline fusion methods. OrthoFuse consistently finds a balance between style transfer and concept preservation, maintaining the semantic identity of the concept while accurately reflecting the target artistic style. In contrast, baseline methods exhibit clear limitations. ZipLoRA and K-LoRA often produce artifacts or fail to transfer stylistic features faithfully, particularly for challenging styles, and their results depend heavily on the specific concept–style pair, making them unstable across different combinations. Joint preserves the concept very well but struggles to apply the target style effectively, resulting in weaker stylization. Overall, OrthoFuse generates visually coherent compositions with consistent textures, lighting, and stylistic patterns, even for difficult style-concept combinations. These qualitative observations align with the quantitative findings, confirming that OrthoFuse achieves the most balanced integration of concept and style, producing images that are both semantically faithful and aesthetically rich. Additional qualitative results are provided in the Appendix G. 5.6 User Study Table 2: User study. Comparison of our method with K-LoRA and ZipLoRA on all images used in the quantitative evaluation. 65 participants, 1460 pairwise comparisons. Question Our vs K-L. Our vs Z. Concept Preserv. (Q1) 48% vs 52% 54% vs 46% Style Transfer (Q2) 77% vs 23% 83% vs 17% Overall Preference (Q3) 67% vs 33% 76% vs 24% To complement automatic metrics and account for their known limitations in evaluating concept–style trade-offs, we conducted a user study comparing our method with K-LoRA and ZipLoRA. Participants were asked three questions: Q1 evaluated Concept Preservation, Q2 assessed Style Transfer, and Q3 measured Overall Preference. Full protocol details and question wording are provided in the Appendix J. We collected responses from 65 participants, resulting in 1,460 pairwise comparisons across all images used in the quantitative evaluation, with half of the comparisons performed against K-LoRA and the other half against ZipLoRA. In each trial, participants compared the results of two methods applied to the same concept–style pair and selected the preferred image according to the given criterion. The results are summarized in Table 2, where each value denotes the percentage of participants preferring our method over the baseline. The study shows that while K-LoRA achieves slightly better concept preservation, our method is strongly preferred in terms of style transfer. ZipLoRA, in contrast, is outperformed by our method in both concept preservation and style transfer. Q3 further shows that participants favor our results by a substantial margin over both K-LoRA and ZipLoRA. Overall, the user study confirms that our approach produces images that better satisfy perceptual expectations of stylized concept generation. 6 Conclusion We introduce OrthoFuse, the first training-free method for orthogonal adapter merging. Our approach substantially improves style transfer fidelity while maintaining highly competitive concept preservation, striking a robust balance between the two. By leveraging structured orthogonal parametrization and manifold-based geodesic approximations, our framework unites adapters tuned for different tasks into a single fused adapter without additional training. Extensive experiments in subject-driven generation tasks demonstrate that OrthoFuse outperforms existing fusion techniques, achieving superior style transfer while maintaining semantic consistency of the concept. Although some trade-offs between concept preservation and style fidelity remain, OrthoFuse establishes a robust, efficient, and principled foundation for multi-adapter fusion in diffusion models, enabling high-quality generation across diverse concept–style combinations. Acknowledgments The work was supported by the grant for research centers in the field of AI provided by the Ministry of Economic Development of the Russian Federation in accordance with the agreement 000000C313925P4E0002 and the agreement with HSE University № 139-15-2025-009. The calculations were performed in part through the computational resources of HPC facilities at HSE University [15]. References [1] V. V. G. A. L. Onishchik (1993) Lie groups and lie algebras i: foundations of lie theory; lie transformation groups. 1 edition, Encyclopaedia of Mathematical Sciences №20, Springer-Verlag Berlin Heidelberg. External Links: ISBN 354061222X; 9783540612223; 364257999X; 9783642579998 Cited by: Appendix A, Appendix A, §3.3, Proposition 9. [2] P.-A. Absil, R. Mahony, and R. Sepulchre (2008) Optimization algorithms on matrix manifolds. Princeton University Press, Princeton, NJ. External Links: ISBN 978-0-691-13298-3 Cited by: §2, §3.3, §3.3, §4. [3] black-forest-labs (2024) FLUX.1. Note: https://github.com/black-forest-labs/flux Cited by: §5.2. [4] N. Boumal (2023) An introduction to optimization on smooth manifolds. Cambridge University Press. External Links: ISBN 9781009166157 Cited by: §4. [5] T. Dao, B. Chen, N. S. Sohoni, A. Desai, M. Poli, J. Grogan, A. Liu, A. Rao, A. Rudra, and C. Ré (2022) Monarch: expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning, p. 4690–4721. Cited by: Appendix B, §3.2, Lemma 1. [6] A. Edelman, T. A. Arias, and S. T. Smith (1998) The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications 20 (2), p. 303–353. External Links: Document Cited by: §4. [7] Y. Frenkel, Y. Vinker, A. Shamir, and D. Cohen-Or (2024) Implicit style-content separation using b-lora. In European Conference on Computer Vision, p. 181–198. Cited by: §2. [8] R. Gal, Y. Alaluf, Y. Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or (2022) An image is worth one word: personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618. Cited by: §1. [9] S. Gallot, D. Hulin, and J. Lafontaine (2004) Riemannian geometry. 3 edition, Springer-Verlag, Berlin. Cited by: Definition B.1, Definition B.2, Appendix B, Appendix B, Appendix B, §2, Proposition 7, Proposition 8. [10] G. H. Golub and C. F. V. Loane (2013) Matrix computations. 4 edition, Johns Hopkins University Press. Cited by: Appendix B, §3.2, §4, Lemma 1. [11] M. Gorbunov, N. Yudin, V. Soboleva, A. Alanov, A. Naumov, and M. Rakhuba (2024) Group and shuffle: efficient structured orthogonal parametrization. Advances in neural information processing systems 37, p. 68713–68739. Cited by: §1, §2, §3.1, §3.2, §3.2, Definition 3.1, Remark 2, Theorem 1. [12] A. Hertz, A. Voynov, S. Fruchter, and D. Cohen-Or (2024) Style aligned image generation via shared attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4775–4785. Cited by: §2. [13] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §1, §2. [14] G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi (2023) Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §K.3. [15] P. Kostenetskiy, R. Chulkevich, and V. Kozyrev (2021) HPC resources of the higher school of economics. In Journal of Physics: Conference Series, Vol. 1740, p. 012050. Cited by: Acknowledgments. [16] N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J. Zhu (2023) Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1931–1941. Cited by: §1. [17] B. F. Labs (2024) FLUX. Note: https://github.com/black-forest-labs/flux Cited by: §1. [18] J. M. Lee and opt_learn (2019) Equation for geodesic in manifold of orthogonal matrices. Note: Mathematics Stack Exchange, https://math.stackexchange.com/q/3265705 Cited by: §4. [19] J. M. Lee (2012) Introduction to smooth manifolds. 2 edition, Springer. External Links: Link, ISBN 978-1-4419-9982-5 Cited by: Appendix A, Appendix A, Appendix A, §K.3, Appendix B, Appendix B, Definition B.1, Appendix B, Appendix B, Appendix B, §3.3, §3.3, Proposition 4, Proposition 5, Proposition 6, Proposition 9. [20] S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen (2024) Dora: weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, Cited by: §2. [21] S. Mataigne, P. Absil, and N. Miolane (2025) On the approximation of the riemannian barycenter. In International Conference on Geometric Science of Information, p. 12–21. Cited by: §H.1. [22] Z. Ouyang, Z. Li, and Q. Hou (2025) K-lora: unlocking training-free fusion of any subject and style loras. arXiv preprint arXiv:2502.18461. Cited by: §2, §5.1, §5.4. [23] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach SDXL: improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, Cited by: §5.2. [24] Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng, Z. Liu, D. Zhang, A. Weller, and B. Schölkopf (2023) Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems 36, p. 79320–79362. Cited by: §1, §2, §2, §4. [25] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695. Cited by: §1. [26] L. Rout, Y. Chen, N. Ruiz, A. Kumar, C. Caramanis, S. Shakkottai, and W. Chu (2024) Rb-modulation: training-free personalization of diffusion models using stochastic optimal control. arXiv preprint arXiv:2405.17401. Cited by: §2. [27] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman (2023) Dreambooth: fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22500–22510. Cited by: §1. [28] Low-rank adaptation for fast text-to-image diffusion fine-tuning External Links: Link Cited by: §2. [29] V. Shah, N. Ruiz, F. Cole, E. Lu, S. Lazebnik, Y. Li, and V. Jampani (2024) Ziplora: any subject in any style by effectively merging loras. In European Conference on Computer Vision, p. 422–438. Cited by: §2, §5.4. [30] V. Soboleva, A. Alanov, A. Kuznetsov, and K. Sobolev (2025) T-lora: single image diffusion model customization without overfitting. arXiv preprint arXiv:2507.05964. Cited by: §2. [31] K. Sohn, L. Jiang, J. Barber, K. Lee, N. Ruiz, D. Krishnan, H. Chang, Y. Li, I. Essa, M. Rubinstein, et al. (2023) Styledrop: text-to-image synthesis of any style. Advances in Neural Information Processing Systems 36, p. 66860–66889. Cited by: §1, §5.1. [32] K. Sohn, N. Ruiz, K. Lee, D. C. Chin, I. Blok, H. Chang, J. Barber, L. Jiang, G. Entis, Y. Li, et al. (2023) Styledrop: text-to-image generation in any style. arXiv preprint arXiv:2306.00983. Cited by: §2. [33] X. Wu, S. Huang, and F. Wei (2024) Mixture of lora experts. arXiv preprint arXiv:2404.13628. Cited by: §2. Supplementary Material Appendix A Smoothness In this section, we prove that the set of GS orthogonal matrices form a smooth manifold using a slightly different notation which is more convenient for these purposes. To do this, let us define additional objects we are going to use further. For i=1,…,mi=1,…,m let BiB_i be a block-diagonal matrix with kik_i orthogonal blocks of size bi×bib_i× b_i and let PiP_i for i=1,…,m−1i=1,…,m-1 be an arbitrary fixed permutation matrix. Based on these notation, we define a set of orthogonal matrices morthA_m^orth: morth=A|A=BmPm−1…B1. split&A^orth_m= \A\; |\;A=B_mP_m-1… B_1 \. split These set of matrices is a significantly more general set than GS orthogonal matrices, considered in Section 3.2 due to the arbitrariness of m, the choice of every permutation matrix and the choice of the number of blocks in each BiB_i. For this set we provide our main technical contribution. Theorem 3. 2orthA^orth_2 is a submanifold in O(N)O(N) and in GLN(ℝ)GL_N(R). Proof. To prove this, let us firstly define groups ℬiB_i: ℬi=O(n1)×⋯×O(nki)B_i=O(n_1)×…×O(n_k_i) (14) and their Cartesian product G=ℬ2×ℬ1G=B_2×B_1. Note that G is a Lie group as it is a Cartesian product of two Lie groups. Additionally, G is compact because it is closed (being defined by the system of closed polynomial equations AA⊤=IAA =I) and bounded (each element of the Cartesian product is bounded by the square root of the matrix size in the Frobenius norm). Note that the orthogonality condition implies that the transpose operation preserves the block-diagonal structure and orthogonality. It also gives us an opportunity to define Lie group action: G=ℬ2×ℬ1↷MG=B_2×B_1 M where M is a manifold. In our case we can choose M=O(N)M=O(N) or M=GLN(ℝ)M=GL_N(R) and take x=P1∈Mx=P_1∈ M. The rule of action is defined as follows: g⋅x:ℬ2×ℬ1×M→M:B2xB1⊤.g· x:B_2×B_1× M→ M:B_2xB_1 . (15) Let us prove that it is really action by definition: • g⋅x∈Mg· x∈ M because orthogonal matrices are closed under multiplication. • (g′g)⋅x=B2′B2x(B1′B1)⊤=B2′B2xB1⊤(B1′)⊤=g′⋅(g⋅x).(g g)· x=B _2B_2x(B _1B_1) =B _2B_2xB _1(B _1) =g ·(g· x). • e⋅x=I2⋅x⋅I1⊤=x⋅I1⊤=x.e· x=I_2· x· I_1 =x· I_1 =x. • The action is smooth as the composition of polynomial (hence smooth) matrix operations. Thus, by definition ([1, 19]), this is indeed a Lie group action. Using the fact that the orbit of a compact Lie group action on a manifold (either GLN(ℝ)GL_N(R) or O(N)O(N)) yields submanifold (see [1, Theorem 2.3] or [19, Corollary 21.6 & Problem 21-17]), we obtain the desired result. In addition to that, using the relation concerning group orbits we get: OrbG(x)≅G/StabG(x),Orb_G(x) G/Stab_G(x), where OrbG(x)=g⋅P1=B2P1B1⊤∣(B2,B1)∈ℬ2×ℬ1, splitOrb_G(x)=\g· P_1=B_2P_1B_1 (B_2,B_1) _2×B_1\, split (16) StabG(P1)=(B2,B1)∈ℬ2×ℬ1|B2P1B1⊤=P1⇔B2=P1B1P1⊤ splitStab_G(P_1)&= \(B_2,B_1) . . _2×B_1\, |\,\\ &B_2P_1B_1 =P_1 .B_2=P_1B_1P_1 \ split (17) and ≅ denotes diffeomorphism (see the proof in, e.g., [19]). ∎ Remark 1. We note that Theorem 3 is more general than our particular use case: diagonal blocks of BiB_i can be of different size and belong to UU (unitary matrices), SOSO (special orthogonal matrices), or SUSU (special unitary matrices). Moreover, the theorem statement is true for any matrix P1P_1 taken from the manifold M (not necessarily a permutation matrix). Since we have shown that 2orthA^orth_2 is a submanifold, multiplying on the left (or right) by a fixed permutation matrix is a smooth diffeomorphism, and therefore the image PLAPR∣A∈2orth\\,P_LAP_R A ^orth_2\,\ is also a submanifold for any fixed permutation matrices PL,PRP_L,P_R. Appendix B GS-orthogonal matrices merging The main problem in GS-orthogonal matrices merging is that the orbit of the action is diffeomorphic to a homogeneous space. It means that the same matrix in manifold M can be obtained in several ways. Here Perfect Shuffle permutation plays an important role: on the one hand, it provides good mixing, and on the other hand, its similarity transformation allows us to rather easily compute the stabilizer, i.e., obtain a description of the orbit. For the latter, we recall the result from [5, 10]. Lemma 1 ([5, 10]). Let P be a Perfect Shuffle permutation matrix. For any diagonal matrix D of size k1×k1k_1× k_1 and any matrix M of size b1×b1b_1× b_1, the following equation holds: P(D⊗M)P⊤=M⊗D.P(D M)P =M D. From Lemma 1, it follows that when k1≥b2k_1≥ b_2 the stabilizer is discrete. Consequently, by [19, Theorem 21.17, Theorem 21.18], the discrete stabilizer implies that the orbit attains its maximal possible dimension: dimOrbG(P1)=dimGStabG(P1)==dimG−dimStabG(P1)≤dimG−0=dimG. split& _G(P_1)= GStab_G(P_1)=\\ ~&= G- _G(P_1)≤ G-0= G. split (18) The following lemma formally clarifies the desired result. Lemma 2. Stabilizer is discrete when k1≥b2k_1≥ b_2. Proof. To prove that stabilizer is discrete, we need to solve B2=P1B1P1⊤,B_2=P_1B_1P_1 , (19) where B2=diag(B21,B22,…,B2k2),B2i∈O(b2)B_2=diag(B_2^1,B_2^2,…,B_2^k_2), B_2^i (b_2) (20) Let us denote C=P1B1P1⊤.C=P_1B_1P_1 . In terms of index notation, block-diagonal property for B2B_2 means that (B2)ij≠0⟹⌊i−1b2⌋=⌊j−1b2⌋(B_2)_ij≠ 0 i-1b_2 = j-1b_2 Now consider an element (B2l)i′j′(B_2^l)_i j within a block of B2B_2, with local indices i′,j′∈1,…,b2i ,j ∈\1,…,b_2\ and global indices i=(l−1)b2+i′i=(l-1)b_2+i , j=(l−1)b2+j′j=(l-1)b_2+j . According to Lemma 1, to be non-zero, (i,j)(i,j)-th of B2B_2 must satisfy i≡j(modk1)i≡ j k_1, and thus i′≡j′(modk1)i ≡ j k_1. In the case k1≥b2k_1≥ b_2 the condition i′≡j′(modk1)i ≡ j k_1 for i′,j′∈1,…,b2i ,j ∈\1,…,b_2\ holds only when i′=j′i =j since |i′−j′|<b2≤k1|i -j |<b_2≤ k_1. Thus, the matrix B2lB_2^l can have nonzero elements only on the diagonal. Since B2lB_2^l is orthogonal, it must be a diagonal orthogonal matrix. The diagonal elements of such a matrix are ±1± 1. This set is finite and, as any stabilizer, forms a subgroup of G [19]. The same result can be obtained for the case of blocks from SOSO. ∎ For the cases where k1≥b2k_1≥ b_2 does not hold, the stabilizer (in general) is non-discrete. Remark 2. Note that the Perfect Shuffle permutation is not the only matrix yielding maximum dimension under the condition k1≥b2k_1≥ b_2. However, Perfect Shuffle is optimal from the dense matrix formation perspective, as it minimizes the number of nonzero elements in the resulting matrix [11]. For these practical reasons, we employ Perfect Shuffle in our structured representation. Now, knowing the structure of the manifold, we can find the geodesics. To do this, we need some additional theory from [9, 19]. Definition B.1 ([9, 19]). Let M and M~ M be two manifolds. A map p:M~→Mp: M→ M is a smooth covering map if: • p is smooth and surjective, • for every point x∈Mx∈ M, there exists a neighborhood U of x in M such that p−1(U)p^-1(U) is a disjoint union ⋃i∈IUi _i∈ IU_i of open subsets of M~ M, and for each i∈Ii∈ I, the restriction p:Ui→Up:U_i→ U is a diffeomorphism. Definition B.2 ([9]). Let (M,g)(M,g) and (M~,h)( M,h) be two Riemannian manifolds. A map p:M~→Mp: M→ M is a Riemannian covering map if: • p is a smooth covering map, • p is a local isometry. While the formal definitions of isometry and local isometry are available in [9], they will not be essential for our subsequent development. Proposition 4 ([19], Proposition 21.28). Every discrete subgroup of a Lie group is a closed Lie subgroup of dimension zero. Proposition 5 ([19], Proposition 21.34). For each n≥1n≥ 1, the Lie groups SO(n),U(n),SU(n)SO(n),U(n),SU(n) are connected. Proposition 6 ([19], Theorem 21.29). If G is a connected Lie group and Γ⊆G G is a discrete subgroup, then G/ΓG/ is a smooth manifold and the quotient map π:G→G/Γπ:G→ G/ is a smooth normal covering map. Proposition 7 ([9], Proposition 2.18). Let p:N→Mp:N→ M be a smooth covering map. For any Riemannian metric g on M, there exists a unique Riemannian metric h on N such that p is a Riemannian covering map. Proposition 8 ([9], Proposition 2.81). Let p:(N,h)→(M,g)p:(N,h)→(M,g) be a Riemannian covering map. The geodesics of (M,g)(M,g) are the projections of the geodesics of (N,h)(N,h), and the geodesics of (N,h)(N,h) are the liftings of those of (M,g)(M,g). Proposition 9 ([1, 19]). The following diagram is commutative: GGG/StabG(P1)G/Stab_G(P_1)OrbG(P1)⊆SO(N)Orb_G(P_1) (N)π πφP1 _P_1f f where π:G→G/Hπ:G→ G/H is surjective map that sends each element g∈Gg∈ G to its corresponding coset gHgH (H denotes StabG(P1)Stab_G(P_1)): π(g)=gHπ(g)=gH. φP1:G→OrbG(P1) _P_1:G _G(P_1)\, is the map defined by action φP1(g)=g⋅P1=B2P1B1⊤ _P_1(g)=g· P_1=B_2P_1B_1 . Finally, f:G/H→OrbG(P1)f:G/H _G(P_1)\, is the diffeomorphism defined by the rule: f(gH):=g⋅P1f(gH):=g· P_1. Proof. Commutativity of the diagram means that for every g∈Gg∈ G, the following identity holds: φP1(g)=f(π(g)). _P_1(g)=f(π(g)). Verification: By the definition of π: π(g)=gHπ(g)=gH. By the definition of f: f(π(g))=f(gH)=g⋅P1f(π(g))=f(gH)=g· P_1. By the definition of φP1 _P_1: φP1(g)=g⋅P1 _P_1(g)=g· P_1. Thus, φP1(g)=f(π(g)) _P_1(g)=f(π(g)) for all g∈Gg∈ G, which proves the commutativity of the diagram. The map f is a diffeomorphism and is well‑defined; see, for example, [19]. ∎ Now let us combine these statements for the case k1≥b2k_1≥ b_2. In this case, the stabilizer forms a subgroup. Therefore, using Propositions 4, 5, the fact that the Cartesian product of connected Lie groups is connected, and Proposition 6, we conclude that π:G→G/Hπ:G→ G/H is a smooth covering map (since smooth normal covering map satisfies a stronger condition than a smooth covering map (see [19])), where H=StabG(P1)H=Stab_G(P_1). Next, we show that the action map φP1 _P_1 is also a smooth covering map. To do this, it is sufficient to show, by Proposition 9, that the composition of the diffeomorphism f and the smooth covering map π is a smooth covering map. Since π is a smooth covering, for any x¯∈G/H x∈ G/H there exists an evenly covered open neighborhood U⊂G/HU⊂ G/H such that π−1(U)=⨆i∈IViπ^-1(U)= _i∈ IV_i, where each Vi⊂GV_i⊂ G is open and the restriction π|Vi:Vi→Uπ|_V_i:V_i→ U is a diffeomorphism. Because f is a diffeomorphism, W=f(U)W=f(U) is an open neighborhood of f(x¯)f( x) in OrbG(P1)Orb_G(P_1). Then: φP1−1(W)=π−1(f−1(W))=π−1(U)=⨆i∈IVi. _P_1^-1(W)=π^-1(f^-1(W))=π^-1(U)= _i∈ IV_i. For each i, the restriction φP1|Vi=f∘π|Vi _P_1|_V_i=f π|_V_i is a diffeomorphism Vi→WV_i→ W, as it is a composition of two diffeomorphisms. Thus, φP1 _P_1 satisfies the exact definition of a smooth covering map. Finally, let g be the standard Riemannian metric on SO(N)SO(N) induced by the Frobenius inner product. As OrbG(P1)Orb_G(P_1) is an embedded submanifold, it inherits a Riemannian metric gorbg_orb. According to Proposition 7, there exists a unique Riemannian metric h on G such that φP1:(G,h)→(OrbG(P1),gorb) _P_1:(G,h)→(Orb_G(P_1),g_orb) is a Riemannian covering map. By Proposition 8, exact geodesics on the orbit are projections of exact geodesics in (G,h)(G,h). Such metric need not necessarily be identical across blocks, nor necessarily coincide with the Frobenius norm. Nevertheless, we can perform a block-wise connection (uniform across blocks) using formula (4) and empirically verify that the resulting curve exhibits nearly constant velocity in the Frobenius norm, consistent with the constant‑speed property of geodesics (see, [9, Definition 2.77]), thereby supporting its interpretation as a meaningful geodesic approximation with respect to the Frobenius norm. These findings are visually confirmed in Figure 5. Note that another natural approach is to follow a geodesic in the ambient space SO(N)SO(N); however, this is computationally expensive, inefficient, and generally does not stay within the GS manifold. Nevertheless, we empirically demonstrate that the resulting block-wise curve closely approximates a geodesic in the ambient space SO(N)SO(N) (see Figure 6). Remark 3. Here and below we use the matrix logarithm, which is not always defined. In practice, however, we work with matrices whose diagonal blocks are sufficiently close to the identity, for which the logarithm is well defined (see Lemma 3 for a proof). Figure 5: Velocity of geodesics measured in the Frobenius norm: comparison of the exact SO(N)SO(N) geodesic, the block‑wise orthogonal approximation, and the block‑wise Cayley space approximation. Figure 6: Relative error (in Frobenius norm) between the blockwise geodesic (geodesic projection from G) and the geodesic in M, as well as the computation time of the two methods. Observe that also that to obtain the group action we added transposition on the right, but we are given matrices in the form L1PR1=L1P(R1⊤)⊤L_1PR_1=L_1P(R_1 ) and L2PR2=L2P(R2⊤)⊤L_2PR_2=L_2P(R_2 ) , and that’s why we need to make additional transposition after connecting R1⊤R_1 and R2⊤R_2 . Below we prove that this approach is equivalent to directly combining the matrix blocks without introducing additional transposition. Proposition 10. For A1,A2∈SO(n)A_1,A_2 (n) let γA1,A2(t)=A1exp(tlog(A1⊤A2)),t∈[0,1], __A_1,A_2(t)=A_1 (t (A_1^\! \!A_2) ), 28.80008ptt∈[0,1], be the standard geodesic joining A1A_1 to A2A_2. Construct the analogous geodesic between the transposed matrices, γA1⊤,A2⊤(t)=A1⊤exp(tlog(A1A2⊤)), __A_1^\! \!,A_2^\! \!(t)=A_1^\! \! (t (A_1A_2^\! \!) ), and transpose the result. Then γA1,A2(t)=[γA1⊤,A2⊤(t)]⊤. __A_1,A_2(t)= [ __A_1^\! \!,A_2^\! \!(t) ]^\! \!. Proof. [γA1⊤,A2⊤(t)]⊤=exp(tlog(A1A2⊤))⊤A1. [ __A_1^\! \!,A_2^\! \!(t) ]^\! \!= (t (A_1A_2^\! \!) )^\! \!A_1. Now let’s use (expM)⊤=exp(M⊤)( M)^\! \!= (M^\! \!) and (logM)⊤=log(M⊤)( M)^\! \!= (M^\! \!) (this follows from the absolute convergence of the Taylor series): [γA1⊤,A2⊤(t)]⊤=exp(tlog(A2A1⊤))A1. [ __A_1^\! \!,A_2^\! \!(t) ]^\! \!= \! (t (A_2A_1^\! \!) )A_1. Let Ω=log(A1⊤A2)∈(n) = (A_1^\! \!A_2)∈ so(n). Since A1exp(Ω)A1⊤=A1(∑k=0∞Ωk!)A1⊤=∑k=0∞1k!A1ΩkA1⊤=∑k=0∞1k!(A1ΩA1⊤)k=exp(A1ΩA1⊤). split&A_1 ( )A_1 =A_1 ( _k=0^∞ ^kk! )A_1 \\ =& _k=0^∞ 1k!A_1 ^kA_1 = _k=0^∞ 1k!(A_1 A_1 )^k\\ =& (A_1 A_1 ). split A similar matrix equation is true for the logarithm function: for A1∈SO(N)A_1 (N) we have A1log(X)A1⊤=log(A1XA1⊤).A_1 (X)A_1 = (A_1XA_1 ). Then A2A1⊤=A1exp(Ω)A1⊤=exp(A1ΩA1⊤).A_2A_1^\! \!=A_1 ( )A_1^\! \!= (A_1 A_1^\! \!). Thus log(A2A1⊤)=A1ΩA1⊤=A1log(A1⊤A2)A1⊤. (A_2A_1^\! \!)=A_1 A_1^\! \!=A_1 (A_1^\! \!A_2)A_1^\! \!. Finally, we obtain: γA1⊤,A2⊤(t)=exp(tlog(A2A1⊤))A1==exp(tA1log(A1⊤A2)A1⊤)A1==A1exp(tlog(A1⊤A2))A1⊤A1==A1exp(tlog(A1⊤A2))=γA1,A2(t). split _A_1 ,A_2 (t)=& \! (t (A_2A_1^\! \!) )A_1=\\ =& (tA_1 (A_1^\! \!A_2)A_1^\! \! )A_1=\\ =&A_1 (t (A_1^\! \!A_2) )A_1^\! \!A_1=\\ =&A_1 \! (t (A_1^\! \!A_2) )= __A_1,A_2(t). split (21) ∎ Appendix C Proof of Proposition 1 Proof. Utilizing eigendecomposition of B(t)B(t) B(t)=UΛU∗,Λ=diag(x1+iy1,…,xn+iyn),B(t)=U U^*, =diag(x_1+iy_1,…,x_n+iy_n), (22) satisfying |xi|2+|yi|2=1|x_i|^2+|y_i|^2=1. Since the eigenvalues of B(t)B(t) lie on a unit sphere, we can express them as eiϕie^i _i for each i=1,…,ni=1,…,n. Then, for the principle branch of the logarithm, we have log(B(t))=Ulog(Λ)U∗==Ulog(diag(eiϕ1,…eiϕn))U∗==Udiag(iϕ1,…iϕn)U∗. split (B(t))=&~U ( )U^*=\\ =&~U (diag(e^i _1,… e^i _n))U^*=\\ =&~Udiag(i _1,… i _n)U^*. split (23) Now let us consider B(t)−B(t)⊤2 B(t)-B(t) 2. Using eigendecomposition, we have B(t)−B(t)⊤2=Udiag(iy1,…iyn)U∗. B(t)-B(t) 2=Udiag(iy_1,… iy_n)U^*. (24) Subtracting from (23) the final form of (24), we obtain log(B(t))−B(t)−B(t)⊤2==Udiag(i(ϕ1−y1),…i(ϕn−yn))U∗ split& (B(t))- B(t)-B(t) 2=\\ =&~Udiag (i( _1-y_1),… i( _n-y_n) )U^* split (25) Using that yi=sinϕiy_i= _i, and assuming that yiy_i is close to 0 (which is satisfied in practice), we can utilize Taylor expansion for sin function and finally obtain Udiag(i(ϕ1−y1),…,i(ϕn−yn))U∗==Udiag((y13),…,(yn3))U∗==(‖diag(y1,…,yn)‖23), split&~Udiag (i( _1-y_1),…,i( _n-y_n) )U^*=\\ =&~Udiag (O(y_1^3),…,O(y_n^3) )U^*=\\ =&~O (\|diag(y_1,…,y_n)\|_2^3 ), split (26) where the latter matrix belongs to (‖B(t)−I‖23)O (\|B(t)-I\|_2^3 ). Indeed, ‖B(t)−I‖2==‖diag(x1−1+iy1,…,xn−1+iyn)‖22==∥diag(1−y12−1+iy1,…,1−yn2−1+iyn)∥2⩾1/2‖diag(y1,…,yn)‖2 split&\|B(t)-I\|_2=\\ =&~\|diag(x_1-1+iy_1,…,x_n-1+iy_n)\|_2^2=\\ =&~\|diag( 1-y_1^2-1+iy_1,…,\\ &\ 1-y_n^2-1+iy_n)\|_2 \\ &~1/2\|diag(y_1,…,y_n)\|_2 split (27) for small enough maxi|yi| _i|y_i|, which completes the proof. ∎ Appendix D Proof of Proposition 3 Proof. Set skew-symmetric matrix K(t):=B(t)−B(t)⊤2K(t):= B(t)-B(t) 2. From proposition 1 we have: logB(t)=K(t)+(‖B(t)−I‖23). B(t)=K(t)+O (\|B(t)-I\|_2^3 ). Hence, BRotated(t)=exp(η(t)logB(t))==exp(η(t)K(t)+(‖B−I‖23)). split&B_Rotated(t)= (η(t) B(t) )=\\ ~&= (η(t)K(t)+O(\|B-I\|_2^3) ). split (28) In last equation we used that η(t)η(t) has a closed form 1+4t(1−t)1+4t(1-t), which is bounded on [0, 1]. Using smoothness of the matrix exponential we get: BRotated(t)=exp(η(t)K(t))+(‖B(t)−I‖23).B_Rotated(t)= (η(t)K(t))+O (\|B(t)-I\|_2^3 ). From Proposition 2, with η(t)η(t) we obtain the following: exp(η(t)K(t))=(I−η(t)2K(t))−1(I+η(t)2K(t))++(‖η(t)K(t)‖23). split& (η(t)K(t))= (I- η(t)2K(t) )^-1 (I+ η(t)2K(t) )+\\ ~&+O (\|η(t)K(t)\|_2^3 ). split (29) As ‖η(t)K(t)‖2=(‖K(t)‖2)=(‖B(t)−I‖2)\|η(t)K(t)\|_2=O(\|K(t)\|_2)=O(\|B(t)-I\|_2), we obtain exp(η(t)K(t))=BOrthoFuse(t)+(‖B(t)−I‖23). (η(t)K(t))=B_OrthoFuse(t)+O (\|B(t)-I\|_2^3 ). Combining all the results, we get BOrthoFuse(t)=BRotated(t)+(‖B(t)−I‖23).B_OrthoFuse(t)=B_Rotated(t)+O (\|B(t)-I\|_2^3 ). ∎ Appendix E Additional Results on FLUX Figure 7: Qualitative results of OrthoFuse merging on the FLUX model. Each row shows generations produced from different prompts after merging a style adapter and a concept adapter. OrthoFuse maintains consistent concept preservation and style fidelity across prompts. We further evaluate OrthoFuse on the FLUX model to verify that our merging procedure can be applied to different model architectures. Figure 7 shows qualitative generations obtained after merging a style adapter and a concept adapter in FLUX. Each row corresponds to a distinct style-concept pair, while each column shows outputs for different text prompts applied to the same merged adapters. The results demonstrate that OrthoFuse produces stable and coherent merges across a diverse set of style–concept combinations. Even under varying prompts, the merged adapters consistently preserve the underlying concept while expressing the intended style. Appendix F Necessity of Eigenvalue Rotation for High-Quality Merging Figure 8: Comparison of block-wise geodesic interpolation and OrthoFuse merging trajectories. At t=0.6t=0.6, OrthoFuse achieves near-ideal style transfer while preserving the target concept. All images were generated with the prompt: “A <<concept>> << superclass>> in jungle in <<style>> style”. OrthoFuse method combines block-wise geodesic interpolation with spectra restoration, which can be considered as a specific eigenvalue rotation along the unit sphere, preserving orthogonality (see Section 4 for more details). While block-wise geodesics provide a natural and accurate approximation of the real local minimizing geodesic in practice, we observe that fusing GS orthogonal adapters with the help of block-wise geodesics only is insufficient for achieving high-quality semantic merging in diffusion models. Specifically, geodesic approximation via block-wise interpolation tends to drift away from the target concept and often fails to consistently align the style transformation across blocks. Figure 8 illustrates this effect by comparing the merging trajectories obtained with block-wise geodesic approximation and our full OrthoFuse procedure. At intermediate interpolation levels – most clearly at t=0.6t=0.6 – the block-geodesic trajectory produces partially fused images where the transferred style is incomplete and the underlying concept begins to degrade. In contrast, OrthoFuse maintains both style fidelity and concept integrity, demonstrating that eigenvalue rotation plays a critical role in stabilizing the latent path and preventing semantic collapse. All images in Figure 8 were generated with the prompt: “A <<concept>> <<superclass>> in jungle in <<style>> style.” These results confirm that spectra restoration operation is not merely an auxiliary refinement but an essential operation for producing coherent and high-quality merges. Appendix G Additional Results on SDXL Figure 9: Qualitative results of OrthoFuse merging on SDXL. Rows correspond to different prompts; columns show generations obtained from different style–concept adapter pairs. OrthoFuse yields coherent style–concept merges across both prompts and adapter combinations. We additionally provide extended qualitative results on the SDXL backbone to complement the evaluations in the main paper. Figure 9 presents generations obtained after merging a style adapter and a concept adapter within SDXL. In this visualization, each row corresponds to a different text prompt, while each column shows outputs for distinct style-concept adapter pairs applied to the same prompt. Across all prompts and adapter configurations, OrthoFuse consistently achieves coherent style–concept integration, demonstrating strong concept preservation and stable expression of the intended style. Appendix H Ablation study on other merging methods H.1 Low-rank adapter merging Figure 10: Comparison of OrthoFuse (orthogonal adapters merging) with merging low-rank (LoRA) adapters. Merging low-rank adapters results in noticeably weaker performance compared to OrthoFuse, even when both approaches are tuned to use approximately the same number of trainable parameters. We attribute this gap to the scale mismatch inherent to low-rank adapters, which makes their merging substantially more difficult. All images were generated with the prompt: “a <<concept>> dog in the jungle in <<style>> style”. To highlight the applicability of orthogonal fine-tuning and extend the experimental scale, we provide a similar adapter merging experiment but for a fixed-rank manifold: we train low-rank adapters with LoRA for style and concept and try to merge them as fixed-rank manifold elements. Assume that we aim to merge two low-rank matrices XCX_C and XSX_S which are low-rank weight updates for an arbitrary model layer: XC=UCVC⊤,XS=USVS⊤,X_C=U_CV_C , X_S=U_SV_S , (30) where UC,US∈ℝn×r,VC,VS∈ℝm×rU_C,U_S ^n× r,V_C,V_S ^m× r. In the case of low-rank adapters, we aim to minimize the following objective: for t∈[0,1]t∈[0,1] we seek to optimize t⋅dℳr2(Xt,XS)+(1−t)⋅dℳr2(Xt,XC)→minrk(Xt)=r,t· d^2_M_r(X_t,X_S)+(1-t)· d^2_M_r(X_t,X_C)→ _rk(X_t)=r, (31) where dℳr(⋅,⋅)d_M_r(·,·) denotes the distance along the manifold e.g. the shortest curve between two points along the manifold. In our implementation, we replace the distance inside the manifold with the help of the Frobenius norm. Such a substitution is inspired by [21], which proposes certain theoretical guarantees on the closeness of such an approximation when the optimization is done with the change of the exact distance to its upper bound. Having replaced the manifold distance with the Frobenius norm, we obtain the following minimization: t‖Xt−XS‖F2+(1−t)‖Xt−XC‖F2→minrk(Xt)=r.t\|X_t-X_S\|_F^2+(1-t)\|X_t-X_C\|_F^2→ _rk(X_t)=r. (32) This problem appears to be solved efficiently via ALS algorithm. Indeed, for the current low-rank approximation of Xt=UtVt⊤X_t=U_tV_t we are able to alternately update its skeleton factors with short recurrent formulas: • V-step: QR-decomposing U=QURU=Q_UR and Xt=QUV^X_t=Q_U V, we rewrite the task to the following one: ‖V^‖F2−2⟨V^⊤,QU(tXS+(1−t)XC)⟩→minV^\| V\|_F^2-2 V ,Q_U(tX_S+(1-t)X_C) → _ V (33) Taking the gradient by V V, it gives us the update for V: 2V^−2(QU⊤(tXC+(1−t)XS))⊤=0⇒V^=(tXC+(1−t)XS)⊤QU split&2 V-2 (Q_U (tX_C+(1-t)X_S ) ) =0 \\ & V=(tX_C+(1-t)X_S) Q_U split (34) • U-step: in a similar to V-step manner, one can obtain the following update rule for U: considering QR-decomposition of V=QVRV=Q_VR and Xt=U^QV⊤X_t= UQ_V we need to solve the same optimization problem ‖U^‖F2−2⟨U^⊤,QV(tXS+(1−t)XC)⟩→minU^,\| U\|_F^2-2 U ,Q_V(tX_S+(1-t)X_C) → _ U, (35) from which we immediately obtain U^=(tXC+(1−t)XS)⊤QV. U=(tX_C+(1-t)X_S) Q_V. (36) It is worth mentioning that this problem can be easily generalized to the case of several low-rank adapters. In this case, in V-step and U-step one need to replace the term (tXC−(1−t)XS)(tX_C-(1-t)X_S) with the weighted sum of the corresponding low-rank adapters. To validate this approach, in Figure 10 we report the empirical performance of the proposed merging method and compare it with OrthoFuse side by side. It can be observed that merging method applied to low-rank adapters performs worse than for GS orthogonal adapters failing to preserve concept pattern and style fidelity. To make the comparison fair, both methods were tuned using approximately the same number of parameters in corresponding parameter-efficient adapters. H.2 Orthogonal adapter merging via multiplication Figure 11: Direct Merging via Multiplication. The result of directly merging orthogonal adapters, accomplished through the multiplication of two GS orthogonal matrices, exhibits limitations in style preservation, struggles to maintain color consistency, and has a negative impact on concept fidelity. Figure 12: Ablation of the fusion parameter η0 _0. In order to additionally explain the motivation to take into account the geometry of the GS orthogonal manifold, we try the evaluate the most trivial way to merge orthogonal adapters together by multiplying their weight updates. On Figure 11 we report images obtained by multiplying orthogonal weight updates of the concept and style respectively. It can be seen that such an approach to fuse orthogonal adapters fails to preserve both style and concept patterns, which shows that for orthogonal adapters we need a more complicated approach which explicitly treats the structure of both orthogonal adapters. Appendix I Ablation study of Fusion Parameters In this appendix, we present ablation studies analyzing the impact of two key fusion parameters, η0 _0 and t, on the performance of our proposed method. I.1 Ablation of η0 _0 Figure 12 illustrates the results of the ablation study for the fusion parameter η0 _0: • When η0=0 _0=0, the merged adapter collapses to the identity matrix, leading to no concept or style blending. • At η0=1 _0=1, the model reduces to block-wise geodesic interpolation. • The optimal performance is observed around (η0≈2 _0≈ 2), while larger values tend to degrade performance. This analysis emphasizes the importance of appropriately selecting the fusion parameter η0 _0 to achieve the desired balance between concept and style. I.2 Ablation of t Figure 13 presents the results of the ablation study for the fusion parameter t: • At t=0t=0, the method maximizes image similarity and minimizes style similarity. • As t increases, image similarity decreases while style similarity increases, with the maximum style similarity achieved at t≈0.8t≈ 0.8. • Notably, the stylistic effects exhibited at t=0.8t=0.8 are stronger than those observed at t=1t=1. This is because, for t<1t<1, the eigenvalue transformation is applied, which can amplify the stylistic components. When t=1t=1, this transformation is disabled to recover the original style adapter from (7), which can make the result at t=0.8t=0.8 appear stylistically stronger by comparison. This behavior indicates the trade-off between image and style similarity, underscoring the significance of fine-tuning parameter t for optimal performance. Figure 13: Ablation of the fusion parameter t. Appendix J User Study In this appendix, we provide further details about the user study conducted to evaluate the effectiveness of our proposed method compared to K-LoRA and ZipLoRA. To address the limitations of automatic metrics in assessing concept–style trade-offs, we designed a user study involving 65 participants. This study resulted in 1,460 pairwise comparisons across all images used in the evaluation. Participants were asked to compare images generated by different methods applied to the same concept–style pair and respond to the following questions: Q1: Which image better captures the features of the style? Evaluate whether the style is recognizable through visual characteristics (colors, textures, brush strokes, lines, etc.); can we say that the concept is genuinely represented in this style, rather than just slightly altered? Q2: Which image better preserves the concept? Assess how well the original object (concept) is maintained; is it recognizable (shape, proportions, structure), and are important details retained? Please disregard any changes in pose. Q3: Which method, in general, handled the task of style transfer to the concept better? Assess the overall result of the style transfer: • Does it create the impression that the concept is naturally executed in the given style? • How well does the style harmonize with the object? If you do not see a difference regarding any question or are uncertain about your choice, please select ”not sure.” Appendix K OrthoFuse Implementation Details This section provides implementation details of the OrthoFuse algorithm used to construct the fused adapter A(t)A(t) from independently trained concept and style adapters. As described in Section 3.2, both adapters are represented via block structures. In algorithms below we denote concept and style corresponding weight matrices (DC(i),DS(i))(D_C^(i),D_S^(i)). Importantly, (DC(i),DS(i))(D_C^(i),D_S^(i)) are weight matrices which are used to build skew-hermitian matrices for a subsequent Cayley transform application. All merging operations are then performed independently on each corresponding pair of blocks. K.1 Full OrthoFuse: Geodesic Block Interpolation The full OrthoFuse method performs interpolation along the geodesic in the orthogonal group for each block. For every pair (BC(i),BS(i)) (B_C^(i),B_S^(i)) we compute: B~(i)(t)=Geodesic(BC(i),BS(i),t). B^(i)(t)=Geodesic (B_C^(i),B_S^(i),t ). (37) In practice, the geodesic is computed via: 1. Conversion to skew-symmetric generators using the Cayley parameterization; 2. Spectral decomposition of BS⊤BCB_S B_C; 3. Logarithmic interpolation in the Lie algebra; 4. Exponential map back to the orthogonal group The corresponding pseudocode for a single block is shown below. Algorithm 1 OrthoFuse merging 1:DC,DSD_C,D_S. 2:KC=DC−DC⊤2;K_C= D_C-D_C 2; KS=DS−DS⊤2K_S= D_S-D_S 2; 3:BC=torch.linalg.solve((I−KC)(I+KC));B_C= torch.linalg.solve((I-K_C)(I+K_C)); BS=torch.linalg.solve((I−KS)(I+KS))B_S= torch.linalg.solve((I-K_S)(I+K_S)); 4:Λ,U=torch.linalg.eig(BS⊤BC) ,U= torch.linalg.eig(B_S B_C); 5:Λlog=log(Λ).imag⋅i _log= ( ). imag· i; 6:Bt=BCtorch.linalg.matrix_exp(−t⋅UΛlogU∗).realB_t=B_C torch.linalg.matrix\_exp(-t· U _logU^*). real; 7:return BtB_t. The postprocessing step is defined as follows. Algorithm 2 OrthoFuse postprocess 1:BtB_t. 2:η=1+4t(1−t)η=1+4t(1-t) 3:Q=ηBt/2Q=η B_t/2; 4:Qskew=Q−Q⊤2Q^skew= Q-Q 2; 5:Q=torch.linalg.solve((I−Qskew)(I+Qskew))Q= torch.linalg.solve((I-Q^skew)(I+Q^skew)); 6:return Q. Overall, the full OrthoFuse procedure is defined as: OrthoFuse = OrthoFuseMerging + OrthoFusePostprocess. K.2 Accelerated OrthoFuse (Merge Inside Cayley Space) We also implement a computationally efficient approximation. Instead of performing geodesic interpolation in O(k)O(k), we interpolate directly in the Cayley parameter space: Dmerge(i)=tDC(i)+(1−t)DS(i)D^(i)_merge=tD_C^(i)+(1-t)D_S^(i) (38) The merged skew-symmetric block is then mapped to the orthogonal group via the Cayley transform. Algorithm 3 OrthoFuse: merge inside Cayley space 1:DC,DSD_C,D_S. 2:Dmerge=tDC+(1−t)DSD_merge=tD_C+(1-t)D_S 3:Kmerge=Dmerge−Dmerge⊤2K_merge= D_merge-D_merge 2; 4:Bt=torch.linalg.solve((I−Kmerge)(I+Kmerge))B_t= torch.linalg.solve((I-K_merge)(I+K_merge)); 5:return BtB_t. Figure 14: Comparative analysis of OrthoFuse and its accelerated version. Using identical concept and style adapters and the same fusion parameter t, both methods produce visually indistinguishable results. The accelerated version removes the eigendecomposition step while preserving identity and style fidelity. We note, however, that the two methods are not strictly identical; for example, in the first row, the dog’s right paw in the OrthoFuse result is not fully placed on the surfboard, whereas in FastOrthoFuse it is. The full accelerated pipeline is therefore: FastOrthoFuse = MergeInsideCayleySpace + OrthoFusePostprocess. K.3 Theoretical Justification of Accelerated OrthoFuse First, we show that the matrix logarithm is correctly defined. Lemma 3. Assume orthogonal matrices BS,BC∈SO(n)B_S,B_C (n) satisfy ‖BS−I‖2≤ε<1\|B_S-I\|_2≤ <1 and ‖BC−I‖2≤ε<1\|B_C-I\|_2≤ <1. Then the matrix logarithm log(BS⊤BC) (B_S B_C) is well-defined. Proof. Using the triangle inequality and submultiplicativity, we bound the spectral norm of the difference: ‖BS⊤BC−I‖2 \|B_S B_C-I\|_2 ≤‖BS⊤−I‖2+‖BC−I‖2≤2ε. ≤\|B_S -I\|_2+\|B_C-I\|_2≤ 2 . Since BS⊤BCB_S B_C is orthogonal, its eigenvalues lie on the unit circle. If −1-1 were an eigenvalue, the distance to the identity I would be at least |−1−1|=2|-1-1|=2. Hence, whenever 2ε<22 <2, the matrix BS⊤BCB_S B_C has no eigenvalues equal to −1-1. ∎ Now we make use of the following auxiliary lemma. Lemma 4. For sufficiently small matrices X and Y with ‖X‖2,‖Y‖2=(ε)\|X\|_2,\|Y\|_2=O( ), the matrix logarithm of their exponential product is given by: log(exp(X)exp(Y))=X+Y+12[X,Y]+(ε3) ( (X) (Y))=X+Y+ 12[X,Y]+O( ^3) Proof. This follows from well-known (see, for example, [19]) Baker-Campbell-Hausdorff formula. By submultiplicativity, the norm of the commutator satisfies ‖[X,Y]‖2≤2‖X‖2‖Y‖2=(ε2)\|[X,Y]\|_2≤ 2\|X\|_2\|Y\|_2=O( ^2). Consequently, nested commutators such as [X,[X,Y]][X,[X,Y]] inherently possess a higher order of smallness (ε3)O( ^3). Similarly arguing by induction, it is straightforward to establish that each subsequent nesting of the commutator increases the order of smallness. ∎ Finally, we prove that geodesics can be approximated by a connection in the space of skew‑symmetric matrices. Proposition 11. Let BC,BS∈SO(n)B_C,B_S (n) be orthogonal matrices parameterized by skew-symmetric matrices DC,DS∈(n)D_C,D_S∈ so(n) via the Cayley transform: BC=Cayley(DC)B_C=Cayley(D_C) and BS=Cayley(DS)B_S=Cayley(D_S), where ‖DC‖2,‖DS‖2=(ε)\|D_C\|_2,\|D_S\|_2=O( ). The Cayley transform of their linearly interpolated generators approximates the exact Riemannian geodesic B(t)=BCexp(−tlog(BS⊤BC))B(t)=B_C (-t (B_S B_C)) up to a third-order error: B(t)=Cayley((1−t)DC+tDS)+(ε3)B(t)=Cayley((1-t)D_C+tD_S)+O( ^3) Proof. Recall that the Cayley transform matches the matrix exponential up to the second order: Cayley(K)=exp(K)+(ε3)Cayley(K)= (K)+O( ^3). We rewrite our endpoints as BC=exp(DC)+(ε3)B_C= (D_C)+O( ^3) and BS⊤=Cayley(−DS)=exp(−DS)+(ε3)B_S =Cayley(-D_S)= (-D_S)+O( ^3). Applying Lemma 4, we approximate the logarithm term: log(BS⊤BC) (B_S B_C) =log(exp(−DS)exp(DC)) = ( (-D_S) (D_C)) =DC−DS−12[DS,DC]+(ε3) =D_C-D_S- 12[D_S,D_C]+O( ^3) Substituting this into the geodesic equation yields B(t)=exp(DC)exp(V)+(ε3)B(t)= (D_C) (V)+O( ^3), where the exponent is defined as V=−t(DC−DS)+t2[DS,DC]V=-t(D_C-D_S)+ t2[D_S,D_C]. We apply Lemma 4 a second time to combine these into a single exponential exp(Z) (Z), where Z=DC+V+12[DC,V]Z=D_C+V+ 12[D_C,V]. Given that 12[DC,V]=12[DC,−t(DC−DS)]+(ε3) 12[D_C,V]= 12[D_C,-t(D_C-D_S)]+O( ^3), we expand Z: Z Z =DC−t(DC−DS)+t2[DS,DC] =D_C-t(D_C-D_S)+ t2[D_S,D_C] +12[DC,−t(DC−DS)]+(ε3) + 12[D_C,-t(D_C-D_S)]+O( ^3) Due to the anti-symmetry of the Lie bracket, the second-order commutators perfectly cancel each other out: 12[DC,−t(DC−DS)]=t2[DC,DS]=−t2[DS,DC] 12[D_C,-t(D_C-D_S)]= t2[D_C,D_S]=- t2[D_S,D_C] This exact cancellation reduces the exponent to Z=(1−t)DC+tDSZ=(1-t)D_C+tD_S. Therefore, the geodesic simplifies to B(t)=exp((1−t)DC+tDS)+(ε3)B(t)= ((1-t)D_C+tD_S)+O( ^3). Applying the Padé equivalence exp(Z)=Cayley(Z)+(ε3) (Z)=Cayley(Z)+O( ^3) once more concludes the proof. ∎ Another way to think about this is as follows. After training the adapters using skew‑symmetric matrices, we obtain their final representations and then apply the Cayley transform (a retraction) to obtain orthogonal matrices. Thus, performing linear interpolation in the skew‑symmetric parameter space corresponds to mixing weights — a concept reminiscent of task arithmetic (see, e.g., [14]). K.4 Computational Considerations The computational bottleneck of the full OrthoFuse algorithm is the eigendecomposition step: Λ,U=torch.linalg.eigh(QS⊤QC). ,U= torch.linalg.eigh(Q_S Q_C). (39) For the matrix sizes used in our adapters, this operation accounts for approximately 90% of the total merging time. In PyTorch, this routine is not efficiently parallelized in our setting and dominates the wall-clock runtime. The accelerated variant completely removes the eigendecomposition. All remaining operations (matrix multiplications, linear solves, and matrix exponentials) are efficiently parallelized, leading to a substantial speedup. In practice, the fast version performs adapter merging in under one second while nonaccelerated version works in 90 seconds. K.5 Empirical Observation Despite the simplification, the accelerated version produces images that are visually almost indistinguishable from those obtained using the full geodesic interpolation (see Figure 14). In our experiments, we observe no meaningful degradation in identity preservation or style transfer quality, while achieving a significant reduction in computational cost. Appendix L Limitations While OrthoFuse is training-free, it requires adapters to be in GS-orthogonal form. Most community adapters are standard LoRAs; applying our method directly to them would need a projection, which risks losing the information encoded in the original LoRA weights. Extending our fusion to LoRA weights is a promising future direction.