Paper deep dive
HyperFix: Combinatorial Nonlinear Correction for Task Vector Merging
Hyo Seo Kim, Ren Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Task vectors enable model merging without joint retraining. In practice, the subset of task vectors to be merged may vary, but many existing methods use scalar tuning for a particular subset, requiring repeated tuning across subsets and restricting task vector merging to linear rescaling. We therefore formulate merging across varying task subsets as a combinatorial correction problem and introduce HyperFix, a lightweight hypernetwork that predicts subset-conditioned nonlinear corrections in weight space. Trained once on singleton, pair, and triple subsets from a task bank, HyperFix generalizes to larger subsets without per-subset optimization. Our local perturbation analysis bounds the residual correction beyond linear merging and motivates learning it from small task updates. Experiments across diverse benchmarks show that HyperFix outperforms existing task vector merging methods while reducing tuning cost.
Tags
Links
- Source: https://arxiv.org/abs/2608.11499v1
- Canonical: https://arxiv.org/abs/2608.11499v1
Trouble viewing inline? Open PDF directly →
Full Text
55,219 characters extracted from source content.
Expand or collapse full text
HyperFix: Combinatorial Nonlinear Correction for Task Vector Merging Hyo Seo Kim Ren Wang Abstract Task vectors enable model merging without joint retraining. In practice, the subset of task vectors to be merged may vary, but many existing methods use scalar tuning for a particular subset, requiring repeated tuning across subsets and restricting task vector merging to linear rescaling. We therefore formulate merging across varying task subsets as a combinatorial correction problem and introduce HyperFix, a lightweight hypernetwork that predicts subset-conditioned nonlinear corrections in weight space. Trained once on singleton, pair, and triple subsets from a task bank, HyperFix generalizes to larger subsets without per-subset optimization. Our local perturbation analysis bounds the residual correction beyond linear merging and motivates learning it from small task updates. Experiments across diverse benchmarks show that HyperFix outperforms existing task vector merging methods while reducing tuning cost. Introduction Task vectors represent the parameter differences between pretrained and task-specific fine-tuned weights. Task arithmetic (Ilharco et al. 2023) linearly combines these task vectors to integrate task-specific capabilities without expensive joint retraining. This approach has demonstrated strong performance across vision (Ortiz-Jimenez, Favero, and Frossard 2023), language (Zhang et al. 2025; Zeng et al. 2025), and multimodal domains (Huang et al. 2024). Despite these advances, many existing task vector merging methods are developed and evaluated for a single fixed task subset, with scalar tuning for that subset. However, different downstream needs may require different subsets of task-specific capabilities, highlighting the need for merging methods that can accommodate varying target subsets. In particular, the tuned scalar cannot be reliably reused because it is specific to a particular subset, as illustrated in Fig. 1(a). As diverse subsets need to be considered, repeated per-subset tuning is required, resulting in a combinatorial tuning cost. Moreover, these merging methods are restricted to linear scaling of the merged task vector, limiting their ability to capture nonlinear interactions among tasks. Figure 1: Comparison of the existing merging approach and HyperFix. (a) Existing merging requires separate scalar tuning for each task subset and can only rescale the linearly merged task vector. (b) HyperFix uses a hypernetwork to predict subset-conditioned correction, avoiding per-subset tuning and enabling nonlinear correction. We therefore view task vector merging as a combinatorial correction problem, where the goal is to develop a shared mechanism that constructs a subset-conditioned merged model for any selected subset of task vectors, rather than optimizing a model for a single fixed subset. To address this problem, we introduce HyperFix, a framework that predicts subset-conditioned nonlinear corrections, as illustrated in Fig. 1(b). Specifically, HyperFix uses a lightweight hypernetwork, trained on small subsets and applied to larger subsets, to generate the corrections. Our theoretical analysis explains why corrections learned from small subsets can generalize to larger ones. Under mild local smoothness conditions, nonlinear interaction effects can be approximated using low-order subsets. Consistent with this analysis, our experimental results demonstrate that HyperFix consistently outperforms merging baselines across all subset sizes on eight image classification benchmarks and three CLIP (Radford et al. 2021) architectures. By avoiding repeated per-subset scalar tuning, HyperFix reduces tuning cost by up to 82.6%. Our main contributions are as follows: • We view task vector merging across varying subsets as a combinatorial correction problem and introduce HyperFix, a hypernetwork-based framework that learns a shared mapping from task-subset embeddings to weight corrections that capture nonlinear task interactions. • We theoretically show that nonlinear interactions can be approximated using low-order subsets, explaining generalization from small to larger subsets. • Across diverse benchmarks, HyperFix outperforms merging baselines across all subset sizes while reducing tuning cost dramatically. Related Work Model Merging Model merging aims to combine multiple models fine-tuned from a shared foundation model into a single model that preserves their task-specific capabilities. A fundamental approach is Task Arithmetic (Ilharco et al. 2023), which represents task-specific adaptations as task vectors and merges them through linear addition. Subsequent methods improve task vector merging through masking and magnitude-based selection (Yadav et al. 2023; Yu et al. 2024; Kim, Han, and Choe 2025), geometry-aware constraints (Sun et al. 2025; Porrello et al. 2026), variational formulations (Zhang et al. 2025), and low-rank decomposition of task matrices (Gargiulo et al. 2025). Adaptive coefficient-based methods such as AdaMerging (Yang et al. 2024) optimize linear merging coefficients for a given set of target tasks. In PEFT settings, alignment-based merging of LoRA adapters has also been explored (Panariello et al. 2025). Despite these advances, most existing methods construct a merged model separately for each target task subset and do not learn a shared merging function across subsets. In contrast, HyperFix learns a shared merging function that predicts subset-conditioned nonlinear corrections. Hypernetwork-based Model Editing Hypernetworks (Ha, Dai, and Le 2017; Krueger et al. 2017; Ratzlaff and Fuxin 2019) are secondary neural networks that generate or modulate the parameters of a target network. In model editing, hypernetworks produce parameter updates conditioned on editing requests. MeG (Wan et al. 2025) uses a diffusion-based hypernetwork to generate dynamic weight neurons, enabling large-scale knowledge updates without directly modifying internal model weights, unlike approaches such as MEMIT (Meng et al. 2023) and MALMEN (Tan, Zhang, and Fu 2024). LoRA.rar (Shenaj et al. 2025) uses a hypernetwork to predict merging coefficients for combining subject and style LoRAs in real time, reducing computation compared with optimization-based methods. Unlike these hypernetwork-based approaches, HyperFix neither generates full model weights nor predicts scalar merging coefficients. Instead, it predicts a structured low-rank nonlinear correction on top of the linearly merged task vector, conditioned on a permutation-invariant representation of the selected task subset. Method Figure 2: Overview of HyperFix. Given a target task subset S, HyperFix constructs the merged task vector τS _S and uses the subset embedding zSz_S to predict a subset-conditioned correction ΔS _S, yielding θS=θ0+τS+ΔS _S= _0+ _S+ _S. Left: HyperFix is trained on task subsets with |S|≤3|S|≤ 3 (singleton, pair, triple) using KL-based distillation from the corresponding single-task teachers. Right: HyperFix generalizes to larger task subsets (|S|>3|S|>3), where the learned correction is applied without additional optimization. In this section, we present HyperFix, a hypernetwork-based correction framework for task vector merging (Figure 2). We view merging across varying task subsets as a combinatorial correction problem, where the goal is to learn a shared correction function. HyperFix augments linear merging with a subset-conditioned correction generated by a nonlinear hypernetwork. Learned from small task subsets, the shared correction function is applied without additional optimization to larger subsets from the same task bank. We first motivate the need for subset-conditioned corrections and then describe the subset embedding, hypernetwork parameterization, training objective, and inference procedure. Motivation Linear merging assumes that linearly combining task-specific parameter updates in weight space is sufficient to retain the capabilities of the selected tasks in a single model. Let θ0 _0 denote the pretrained model parameters and θi _i the fine-tuned parameters for task i, and define the task vector as τi=θi−θ0 _i= _i- _0. For a subset of tasks S, linear merging produces a merged task vector τS _S τS=1|S|∑i∈Sτi. _S= 1|S| _i∈ S _i. (1) This yields the merged parameters θ0+τS _0+ _S. HyperFix operates on top of this merged task vector and can also be applied to other merging rules, such as TIES (Yadav et al. 2023). However, this additive assumption ignores potential nonlinear interactions between task vectors. In practice, merging multiple task vectors can introduce non-additive interference, leading to performance degradation as the subset size increases. To capture interactions beyond linear merging, we introduce a subset-conditioned correction term θS=θ0+τS+ΔS, _S= _0+ _S+ _S, (2) where ΔS _S captures structured interaction effects that depend on the specific combination of tasks in S. Importantly, these effects are not universal offsets but vary with the alignment and conflict structure among task vectors in the subset. The construction of ΔS _S is detailed in the following subsection. HyperFix: Combinatorial Nonlinear Correction To model interaction effects beyond linear merging, we require a compact representation of a task subset. Conditioning directly on raw task vectors would entail operating in the full encoder parameter space, which contains tens of millions of dimensions (e.g., CLIP ViT-B/32), making learning computationally expensive and difficult to optimize. Instead, we construct a low-dimensional task-level representation that captures relationships between task vectors. Subset embedding. We summarize pairwise relationships between task vectors using a task-level Gram matrix Gij=⟨τi,τj⟩,G_ij= _i, _j , (3) where the inner product is taken over all encoder parameters. The Gram matrix is computed once from the task vectors, mean-centered across tasks, and kept fixed during HyperFix training. Let N be the number of tasks and G∈ℝN×NG ^N× N the corresponding Gram matrix. We define the task embedding zi∈ℝNz_i ^N as the i-th row of G zi=Gi:.z_i=G_i:. (4) For a subset S, we define the subset embedding zS=1|S|∑i∈Szi.z_S= 1|S| _i∈ Sz_i. (5) The Gram matrix provides pairwise relationship features among task vectors in weight space. Averaging the corresponding task-level embeddings yields a permutation-invariant representation of the interaction structure within a subset. This embedding, therefore, encodes interaction statistics rather than raw parameters, allowing the hypernetwork to focus on modeling residual nonlinear effects. Structured low-rank correction via hypernetwork. Given the subset embedding zSz_S, HyperFix uses a hypernetwork hϕh_φ to predict a structured low-rank correction ΔS=hϕ(zS). _S=h_φ(z_S). (6) For each encoder weight matrix Wℓ∈ℝdℓ×mℓW_ ^d_ × m_ , HyperFix predicts low-rank factors Uℓ∈ℝdℓ×rU_ ^d_ × r and Vℓ∈ℝmℓ×rV_ ^m_ × r, forming a LoRA-style update ΔWℓ=UℓVℓ⊤ W_ =U_ V_ , where the rank is set to r=4r=4 in all experiments. The low-rank parameterization improves structural efficiency while allowing expressive corrections distributed across layers. The hypernetwork is implemented as a two-layer MLP with a hidden dimension of 512 and GELU activation. The MLP outputs a single concatenated vector that is partitioned and reshaped into the per-layer factors (Uℓ,Vℓ)(U_ ,V_ ), thereby producing all encoder updates in a single forward pass. The resulting correction augments the linear merge in Eq. (2). Training objective. Our primary training objective is to optimize the hypernetwork parameters by aligning the predictive distributions of the merged model θS _S with those of the corresponding single-task models. Specifically, we employ KL-based knowledge distillation. During training, the image encoder and all task-specific classification heads are frozen, and only the HyperFix parameters are optimized. We train HyperFix on subsets of size one to three (singleton, pair, and triple) to learn correction patterns while keeping training computationally tractable. For each task i∈Si∈ S, we use the corresponding single-task model θi _i as a reference and compare its predictions with those of the merged model θS _S. Let pi(⋅∣x)p_i(· x) and qi(⋅∣x)q_i(· x) denote the predictive distributions induced by θi _i and θS _S, respectively, for task i and input x. We minimize the average KL divergence LS=1|S|∑i∈SKL(pi(⋅∣x)∥qi(⋅∣x)).L_S= 1|S| _i∈ SKL\! (p_i(· x)\;\|\;q_i(· x) ). (7) For each i∈Si∈ S, the KL divergence is computed on mini-batches sampled from task i’s dataset, and the loss is averaged across tasks in S. Inference. At inference time, no additional optimization is required. Given a subset S, we compute zSz_S, predict ΔS _S via Eq. (6), and construct θS _S using Eq. (2). The merged model is then directly evaluated. Theoretical Analysis In this section, we provide a theoretical analysis of HyperFix, explaining why training on task subsets of size at most three is sufficient for generalization to larger subsets not used during training. 11 1 Our theoretical results are local and perturbative in nature. We analyze behavior in a neighborhood of the pretrained model θ0 _0 under small task updates, where empirical loss landscapes are often locally smooth and well-conditioned in practice. The assumptions below are not intended to globally characterize deep networks, but to isolate the mechanisms governing nonlinear interaction effects near θ0 _0. Detailed proofs are provided in the Supplementary. Preliminaries and Notation We adopt the notation introduced in the Method section. Recall that linear merging produces parameters θ0+τS _0+ _S (Eq. 1), and HyperFix augments this with a correction term as in Eq. 2. For analysis, define the subset-averaged loss LS(θ):=1|S|∑i∈SLi(θ).L_S(θ):= 1|S| _i∈ SL_i(θ). (8) Let θS⋆ _S denote a local stationary point of LSL_S near θ0 _0 (i.e., ∇LS(θS⋆)=0∇ L_S( _S )=0). The ideal residual correction beyond linear merging is ΔS⋆:=θS⋆−(θ0+τS). _S:= _S -( _0+ _S). (9) Finally, let z¯:=1N∑i=1Nzi,σz2:=1N∑i=1N|zi−z¯|2 z:= 1N _i=1^Nz_i, _z^2:= 1N _i=1^N|z_i- z|^2 (10) denote the mean and variance of the task embeddings, respectively. Assumptions We introduce the assumptions used in our analysis. A1 (Local smoothness). Each task loss Li(θ)L_i(θ) is three-times continuously differentiable in a neighborhood N of θ0 _0, and ‖∇2Li(θ)‖op≤H,‖∇3Li(θ)‖op≤M,\|∇^2L_i(θ)\|_op≤ H, \|∇^3L_i(θ)\|_op≤ M, (11) for all θ∈θ . A2 (Local conditioning). For any subset S, we assume that LSL_S admits a locally well-conditioned Hessian in a neighborhood of θ0 _0, i.e., ∇2LS(θ)⪰μI,∇^2L_S(θ) μ I, (12) for some μ>0μ>0 in N. A3 (Small task updates). Task vectors satisfy ‖τi‖≤ρ\| _i\|≤ρ for all i, where ρ is sufficiently small so that θ0+τS∈ _0+ _S . A4 (Low-order representability). The dominant component of the ideal correction ΔS⋆ _S admits a smooth representation in a permutation-invariant subset embedding space ΔS⋆=g(zS)+η(S),‖η(S)‖≤ε, _S=g(z_S)+η(S), \|η(S)\|≤ , (13) where g is LgL_g-Lipschitz. Low-Order Interaction Generalization Theorem 1 (Nonlinear remainder). Under Assumptions A1–A3, ‖ΔS⋆‖≤1μ(‖∇LS(θ0)‖+Hρ),\| _S\|≤ 1μ (\|∇ L_S( _0)\|+Hρ ), (14) and the nonlinearity beyond the first-order Hessian term satisfies ∇LS(θ0+τS)=∇LS(θ0)+∇2LS(θ0)τS+RS,∇ L_S( _0+ _S)=∇ L_S( _0)+∇^2L_S( _0) _S+R_S, (15) where ‖RS‖≤M2ρ2\|R_S\|≤ M2ρ^2. Hence the parameter effect induced by this nonlinear remainder is bounded by ‖(∫01∇2LS(θ0+τS+tΔS⋆)t)−1RS‖≤M2μρ2. \| ( _0^1∇^2L_S( _0+ _S+t _S)\,dt )^-1R_S \|≤ M2μρ^2. (16) Interpretation. The first inequality bounds the overall correction needed beyond linear merging. It is controlled by (i) local conditioning (1/μ1/μ), (i) the task-update scale ρ, and (i) the gradient bias at θ0 _0. The gradient term represents baseline mismatch between the pretrained model and the task mixture. It does not scale with task-update magnitude nor reflect higher-order interactions. The second inequality isolates the nonlinear interaction component: under bounded third derivatives, the part not captured by the first-order Hessian term (instead captured by the curvature-driven remainder) shrinks quadratically with ρ (O(ρ2)O(ρ^2)). Thus, when task vectors are small, the curvature-induced component of the correction remains controlled. This result motivates learning a compact residual correction. Theorem 1 establishes that the residual correction beyond linear merging is controlled in magnitude and dominated by low-order interaction terms under small task updates. In particular, the nonlinear remainder scales quadratically with the update size. This suggests that the dominant interaction structure is smooth and locally well-behaved in the subset embedding space. We now formalize how learning such low-order interactions from small task subsets supports generalization to larger subsets not used during training. Theorem 2 (Low-order interaction generalization). Let hϕh_φ be a hypernetwork trained on task subsets with |S|≤3|S|≤ 3 to minimize |S|≤3‖hϕ(zS)−ΔS⋆‖2E_|S|≤ 3\|h_φ(z_S)- _S\|^2. Then for any subset size m≥2m≥ 2, |S|=m‖hϕ(zS)−ΔS⋆‖≤ε+Lgσzm,E_|S|=m\|h_φ(z_S)- _S\|≤ + L_g _z m, (17) where σz2 _z^2 bounds the variance of the task embeddings ziz_i across tasks. The 1m 1 m term arises from the concentration of the empirical subset embedding zSz_S around its expectation as the subset size increases. Interpretation. HyperFix generalizes from small task subsets because task interactions are locally smooth and dominated by low-order effects. If each task update τi _i is small, and the loss landscape around the pretrained model θ0 _0 is smooth, then the deviation from linear merging arises primarily from pairwise and triple interactions. Higher-order interactions decay rapidly with the magnitude of task updates. Furthermore, the subset embedding zSz_S is an empirical average of task interaction statistics. As the subset size increases, this average becomes more stable due to the concentration of measure. Since HyperFix learns a smooth mapping from zSz_S to correction parameters, it can extrapolate from small task subsets (|S|≤3|S|≤ 3) to larger task subsets. This explains why training up to triples is sufficient. Experiments In this section, we evaluate HyperFix under the combinatorial correction setting through three key questions. First, can a correction function trained only on small task subsets generalize to larger subsets while maintaining performance? Second, do the predicted corrections meaningfully adapt to the selected task subset, rather than acting as a fixed global adjustment? Third, can HyperFix replace repeated per-subset tuning with a shared training procedure, thereby reducing computational cost while maintaining competitive or superior performance? Setup Combinatorial correction setting. We evaluate the combinatorial correction problem on eight image classification benchmarks: Cars (Krause et al. 2013), DTD (Cimpoi et al. 2014), EuroSAT (Helber et al. 2019), GTSRB (Stallkamp et al. 2011), MNIST (LeCun 1998), RESISC45 (Cheng, Han, and Lu 2017), SUN397 (Xiao et al. 2016), and SVHN (Netzer et al. 2011). Given task vectors τii=1N\ _i\_i=1^N, the goal is to learn a shared correction function such that, for any subset S⊆1,…,NS \1,…,N\, the merged model performs well on all tasks in S. HyperFix is trained only on subsets with |S|≤3|S|≤ 3 (singleton, pair, and triple) and evaluated on both subset sizes used during training (|S|=2,3|S|=2,3) and larger subset sizes not used during training (|S|=4,…,8|S|=4,…,8). For each subset size |S||S|, we evaluate all (8|S|) 8|S| possible subsets and report their average performance. Baselines. We evaluate both weight-space (Ilharco et al. 2023) and tangent-space (Ortiz-Jimenez, Favero, and Frossard 2023) task vectors with the same merging strategies. Mean (Wortsman et al. 2022) performs uniform averaging of task vectors without additional scaling. Sum ++ Scalar (Ilharco et al. 2023) sums task vectors and selects a scalar coefficient separately for each subset from 21 evenly spaced values in [0,1][0,1] to maximize the average normalized validation accuracy. TIES ++ Scalar (Yadav et al. 2023) first resolves sign conflicts using TIES merging and then applies the same scalar search. In contrast, HyperFix augments linear merging with a subset-conditioned nonlinear correction and requires no per-subset tuning. We evaluate Mean ++ HyperFix and TIES ++ HyperFix, where the learned correction is applied on top of the corresponding linear merge rule. Training details. We conduct experiments using pretrained CLIP models with ViT-B/32, ViT-B/16, and ViT-L/14 backbones (Radford et al. 2021). Main results are reported on ViT-B/32, with additional results for ViT-B/16 and ViT-L/14 provided in the Supplementary. During training, all encoder and task-specific classification head parameters are frozen, and only the hypernetwork parameters are optimized. Training is performed for 10 epochs over all singleton, pair, and triple task subsets (|S|∈1,2,3|S|∈\1,2,3\), with random shuffling at each epoch and a batch size of 128 for each task. For each subset, the hypernetwork is optimized by averaging the logit-level KL distillation losses from the corresponding single-task models, using temperature T=2.0T=2.0. We optimize the hypernetwork using AdamW with learning rate 1×10−41× 10^-4 and weight decay 0.10.1. Gradients are clipped to 1.0. The hypernetwork is a two-layer MLP with hidden dimension 512 and predicts low-rank updates with rank r=4r=4 for each encoder weight matrix. Task vectors are computed from independently fine-tuned models on the training split. HyperFix is trained on the validation split, which is also used to tune the scalar coefficients for the scalar-based baselines, and all reported results are evaluated on the held-out test split. All experiments are conducted on a single NVIDIA GH200 GPU. Metrics. We report normalized accuracy following prior work (Ilharco et al. 2023), defined as the task accuracy divided by the accuracy of the corresponding single-task fine-tuned model. For each subset, normalized accuracy is averaged over all tasks in the subset, and we report the mean and standard deviation across all subsets of the same size. We additionally report absolute accuracies in the Supplementary, where we observe consistent method rankings and performance trends across subset sizes. Generalization Across Task Subsets Method |S|=2|S|=2 |S|=3|S|=3 |S|=4|S|=4 |S|=5|S|=5 |S|=6|S|=6 |S|=7|S|=7 |S|=8|S|=8 Avg. Standard Fine-Tuning Mean 95.7± 1.4 89.6± 2.2 84.3± 2.6 80.1± 2.5 77.0± 2.1 74.6± 1.6 72.9± 0.0 82.0 Sum ++ Scalar 96.4± 1.1 92.1± 1.4 88.2± 1.7 84.7± 1.7 81.7± 1.4 79.2± 1.3 77.0± 0.0 85.6 TIES ++ Scalar 97.6± 1.1 94.4± 1.3 91.0± 1.6 87.8± 1.6 85.0± 1.6 82.6± 1.3 80.9± 0.0 88.5 Mean ++ HyperFix 97.7± 1.7 96.3± 1.8 95.1± 1.6 94.0± 1.4 93.2± 1.1 92.6± 0.8 92.0± 0.0 94.4 TIES ++ HyperFix 98.6± 1.3 97.9± 1.2 97.0± 1.2 96.0± 1.1 94.9± 1.0 93.9± 0.7 92.9± 0.0 95.9 Tangent-Space Fine-Tuning Mean 96.8± 1.4 92.5± 1.9 88.8± 2.1 85.8± 2.0 83.4± 1.7 81.4± 1.3 79.8± 0.0 86.9 Sum ++ Scalar 97.8± 1.3 95.4± 1.5 92.9± 1.5 90.8± 1.3 88.8± 1.1 87.2± 0.9 85.7± 0.0 91.2 TIES ++ Scalar 97.5± 1.4 94.5± 1.5 91.9± 1.5 89.7± 1.3 88.0± 1.1 86.4± 0.9 85.0± 0.0 90.4 Mean ++ HyperFix 99.1± 1.1 98.1± 1.0 97.0± 0.9 96.0± 0.8 95.0± 0.7 94.1± 0.6 93.2± 0.0 96.1 TIES ++ HyperFix 98.9± 1.3 98.3± 1.2 97.4± 1.1 96.6± 1.0 95.7± 0.8 94.9± 0.6 94.1± 0.0 96.5 Table 1: Generalization performance of HyperFix on task subsets. We evaluate merging performance across subset sizes |S|=2,…,8|S|=2,…,8 after training only on small task subsets (singleton, pair, and triple). Results are averaged over all (8|S|) 8|S| subsets for each |S||S| and reported as normalized accuracy (%). We compare methods using standard fine-tuning (Ilharco et al. 2023) (top) and tangent-space fine-tuning (bottom) (Ortiz-Jimenez, Favero, and Frossard 2023). Avg. reports the average normalized accuracy across all subset sizes. Training on small task subsets enables generalization to larger subsets. Table 1 reports normalized accuracy across all subset sizes. While Mean performs competitively for small task subsets (|S|=2|S|=2), its performance degrades sharply as more tasks are merged, dropping from 95.7 % to 72.9 % under standard fine-tuning. Although Scalar tuning mitigates this degradation, it still exhibits a substantial performance drop as |S||S| increases. In contrast, trained only on task subsets of size at most three, HyperFix maintains high performance on |S|=4,…,8|S|=4,…,8 without additional tuning. Under standard fine-tuning, Mean ++ HyperFix improves the |S|=8|S|=8 performance from 72.9 % (Mean) to 92.0 %, surpassing both Sum ++ Scalar (77.0 %) and TIES ++ Scalar (80.9 %), despite these methods performing per-subset scalar tuning (with TIES additionally resolving sign conflicts). TIES ++ HyperFix further reaches 92.9 %. Notably, the performance gap between linear merging baselines and HyperFix widens as |S||S| increases, suggesting that non-additive task interactions become increasingly important for larger task subsets. The same trend holds under tangent-space fine-tuning. Mean ++ HyperFix consistently outperforms all linear merging baselines across every subset size. This consistency across both fine-tuning regimes indicates that the gains of HyperFix arise from modeling structured task interactions rather than from a specific task-vector construction. We confirm that the performance gains persist across backbone scales, including ViT-B/16 and ViT-L/14 (see Supplementary). The reported standard deviations reflect variation across all (8|S|) 8|S| task subsets. When |S|=8|S|=8, there is only a single subset, resulting in zero standard deviation by definition. Objective |S|=2|S|=2 |S|=3|S|=3 |S|=4|S|=4 |S|=5|S|=5 |S|=6|S|=6 |S|=7|S|=7 |S|=8|S|=8 Avg. CE 95.0± 3.6 93.6± 2.9 92.1± 2.5 90.8± 2.1 89.6± 1.7 88.7± 1.2 87.8± 0.0 91.1 KL (ours) 97.7± 1.7 96.3± 1.8 95.1± 1.6 94.0± 1.4 93.2± 1.1 92.6± 0.8 92.0± 0.0 94.4 Table 2: Effectiveness of training objective for combinatorial correction. We compare cross-entropy (CE) training with logit-level KL distillation for Mean ++ HyperFix. We report normalized accuracy (%) averaged over all task subsets for each subset size. Avg. further averages results across |S|=2,…,8|S|=2,…,8. KL uses logit-level distillation from the corresponding single-task fine-tuned models, while CE uses ground-truth labels. All other hyperparameters are identical. KL consistently improves performance, particularly for larger task subsets not used during training (|S|≥4|S|≥ 4). KL distillation improves generalization across task subsets. We compare two training objectives for Mean ++ HyperFix under identical data usage and optimization settings: (i) KL distillation from the corresponding single-task fine-tuned models, and (i) supervised CE on ground-truth labels from the same training task subsets. Table 2 shows that KL consistently yields stronger performance across all subset sizes, with increasingly larger gains as |S||S| grows. For example, at |S|=8|S|=8, KL achieves 92.0% compared to 87.8% for CE (+4.2 percentage points). Overall, KL improves the average normalized accuracy from 91.1% to 94.4% (+3.3 percentage points). These results indicate that distillation provides a smoother and more structured learning signal for predicting parameter corrections than direct supervision with ground-truth labels alone. |S|max|S|_ 1 2 3 4 5 6 7 8 Acc. 82.0 90.4 94.4 95.3 95.7 95.9 95.9 95.9 Gain – +8.4 +4.0 +0.9 +0.4 +0.2 +0.0 +0.0 Table 3: Ablation on maximum training subset size. HyperFix is trained only on subsets with size |S|≤|S|max|S|≤|S|_ , where |S|max|S|_ denotes the maximum subset size used during training. Reported values are averaged across subset sizes |S|=2,…,8|S|=2,…,8, after averaging over all task subsets at each size. Performance improves as |S|max|S|_ increases from 1 to 3, while additional gains beyond |S|max≥4|S|_ ≥ 4 are limited, consistent with our theory that dominant interaction effects are captured by low-order subsets. Ablation on maximum training subset size. Table 3 analyzes how the maximum subset size |S|max|S|_ (i.e., the largest subset size included during HyperFix training) affects merging performance for Mean ++ HyperFix. Training only on single-task subsets (|S|max=1|S|_ =1) yields 82.0%. Including pairwise subsets (|S|max=2|S|_ =2) improves performance to 90.4% (+8.4 points), indicating that including pair subsets provides substantial additional supervision. Extending training to triple subsets (|S|max=3|S|_ =3) further increases performance to 94.4% (+4.0 points), already achieving strong generalization to unseen larger subsets up to |S|=8|S|=8. Beyond this point, additional exposure to higher-order subsets yields only marginal gains, increasing from 94.4% to at most 95.9% (+1.5 points overall). This saturation suggests that training on low-order subsets captures most of the correction patterns needed for larger-subset generalization. These empirical findings are consistent with Theorem 1 and Theorem 2, which together indicate that nonlinear remainders are controlled in magnitude and that once dominant low-order interactions are learned, higher-order effects contribute diminishing additional benefit. Subset-Specificity of Corrections Figure 3: Magnitude of corrections across subset sizes. We show the distribution of the ℓ2 _2 norm of the correction ΔS _S for each subset size, computed over all (8|S|) 8|S| task subsets. Boxes indicate interquartile ranges, center lines denote medians, and whiskers show the 5th–95th percentile range. Across all subset sizes, the median remains stable. Method Acc. (%) Mean 95.7± 1.4 Sign-flipped 93.3± 4.6 Shuffled 95.4± 3.2 HyperFix (Ours) 97.7± 1.7 Table 4: Dependence on the subset embedding. We report normalized accuracy (%) for |S|=2|S|=2 under different inputs. Replacing zSz_S with shuffled or sign-flipped embeddings degrades performance, indicating that the predicted correction meaningfully depends on the subset representation. Magnitude of corrections. To better understand the behavior of the hypernetwork, we analyze the ℓ2 _2 norm of the predicted correction ΔS _S across subset sizes |S||S| for Mean ++ HyperFix. For each |S|∈2,…,8|S|∈\2,…,8\, we compute ‖ΔS‖2\| _S\|_2 over all (8|S|) 8|S| task combinations and examine its distribution. As shown in Figure 3, the distribution of ‖ΔS‖2\| _S\|_2 remains well-controlled across subset sizes. The median remains at a comparable scale across subset sizes. This empirical behavior is consistent with the residual bound discussed in the Theoretical Analysis section, which predicts that the magnitude of the nonlinear correction is controlled by the local update scale under smoothness assumptions. Dependence on the subset embedding. We analyze how the predicted correction depends on the subset embedding zSz_S under the Mean ++ HyperFix setting in Table 4. Given the subset embedding zSz_S for a subset S, we consider two perturbations while keeping the base merge τS _S fixed: (i) shuffled: replace zSz_S with zS~z_ S from a randomly sampled S~≠S S≠ S, and (i) sign-flipped: replace zSz_S with −zS-z_S. Using the correct embedding yields the highest accuracy (97.7%), improving over the Mean (95.7%) by +2.0 points. In contrast, perturbing the input embedding degrades performance, sign-flipping reduces accuracy to 93.3%, and shuffling yields 95.4%, both close to or below the Mean. These results indicate that the predicted correction meaningfully depends on the subset embedding rather than acting as a fixed offset independent of the selected task subset. Training Cost and Computational Efficiency Although scalar-based merging requires no additional training, it relies on validation time optimization. For each task subset S, scalar tuning evaluates K candidate coefficients, each requiring evaluation over all tasks in S. Aggregated over all task subsets up to a maximum size |S|max|S|_ , the cumulative validation cost becomes ∑j=2|S|max(Nj)jK _j=2^|S|_ NjjK. In our eight task setting with K=21K=21, this results in 21,33621,336 validation model evaluations before testing. In contrast, HyperFix shifts this optimization to a single amortized training phase. We train the hypernetwork once, using only task subsets with |S|≤3|S|≤ 3, corresponding to ∑j=13(Nj) _j=1^3 Nj training subsets, after which the learned correction function is fixed and applied to all subsets without any per-subset optimization. For a fair comparison, HyperFix is trained on the validation split, using the same data employed for scalar coefficient search. As the number of tasks increases, the number of possible task subsets grows combinatorially, whereas the training cost of HyperFix scales only with subsets up to size three. To quantify the efficiency gain, we measure the wall-clock time required to construct and evaluate the merged model under identical hardware and data settings. All experiments are conducted on a single NVIDIA GH200 (120GB HBM3) GPU with CUDA 12.4. For the full task subset with |S|=8|S|=8, scalar tuning requires 1634.26 seconds due to validation search over K=21K=21 candidate coefficients. In contrast, HyperFix requires only 284.16 seconds without any per-subset search, reducing the total execution time by 82.6%. This empirical gap directly reflects the elimination of repeated validation-time optimization in HyperFix. Conclusion We introduced combinatorial correction as the problem of learning a shared correction function over the combinatorial space of task subsets. We showed that linear task vector merging faces both representational and scalability limitations as the subset size increases. Although trained only on small task subsets, HyperFix generalizes to larger subsets not used during training without per-subset optimization, consistently improving performance across diverse benchmarks. These results demonstrate the importance of modeling nonlinear task interactions for scalable and generalizable task vector merging, opening a path toward more principled and general model merging frameworks. References Cheng, Han, and Lu (2017) Cheng, G.; Han, J.; and Lu, X. 2017. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10): 1865–1883. Cimpoi et al. (2014) Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; and Vedaldi, A. 2014. Describing Textures in the Wild. In CVPR. Gargiulo et al. (2025) Gargiulo, A. A.; Crisostomi, D.; Bucarelli, M. S.; Scardapane, S.; Silvestri, F.; and Rodola, E. 2025. Task singular vectors: Reducing task interference in model merging. In CVPR. Ha, Dai, and Le (2017) Ha, D.; Dai, A. M.; and Le, Q. V. 2017. HyperNetworks. In ICLR. Helber et al. (2019) Helber, P.; Bischke, B.; Dengel, A.; and Borth, D. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7): 2217–2226. Huang et al. (2024) Huang, B.; Mitra, C.; Arbelle, A.; Karlinsky, L.; Darrell, T.; and Herzig, R. 2024. Multimodal task vectors enable many-shot multimodal in-context learning. In NeurIPS. Ilharco et al. (2023) Ilharco, G.; Ribeiro, M. T.; Wortsman, M.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A. 2023. Editing models with task arithmetic. In ICLR. Kim, Han, and Choe (2025) Kim, H. S.; Han, D.; and Choe, J. 2025. NegMerge: Sign-Consensual Weight Merging for Machine Unlearning. In ICML. Krause et al. (2013) Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013. 3D Object Representations for Fine-Grained Categorization. In ICCVW, 554–561. Krueger et al. (2017) Krueger, D.; Huang, C.-W.; Islam, R.; Turner, R.; Lacoste, A.; and Courville, A. 2017. Bayesian hypernetworks. LeCun (1998) LeCun, Y. 1998. The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/. Meng et al. (2023) Meng, K.; Sharma, A. S.; Andonian, A. J.; Belinkov, Y.; and Bau, D. 2023. Mass-Editing Memory in a Transformer. In ICLR. Netzer et al. (2011) Netzer, Y.; Wang, T.; Coates, A.; Bissacco, A.; Wu, B.; and Ng, A. Y. 2011. Reading Digits in Natural Images with Unsupervised Feature Learning. In NeurIPSW. Ortiz-Jimenez, Favero, and Frossard (2023) Ortiz-Jimenez, G.; Favero, A.; and Frossard, P. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. In NeurIPS, 66727–66754. Panariello et al. (2025) Panariello, A.; Marczak, D.; Magistri, S.; Porrello, A.; Twardowski, B.; Bagdanov, A. D.; Calderara, S.; and van de Weijer, J. 2025. Accurate and Efficient Low-Rank Model Merging in Core Space. In NeurIPS. Porrello et al. (2026) Porrello, A.; Buzzega, P.; Dangel, F.; Sommariva, T.; Salami, R.; Bonicelli, L.; and Calderara, S. 2026. Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature. In ICLR. Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML. Ratzlaff and Fuxin (2019) Ratzlaff, N.; and Fuxin, L. 2019. HyperGAN: A Generative Model for Diverse, Performant Neural Networks. In ICML. Shenaj et al. (2025) Shenaj, D.; Bohdal, O.; Ozay, M.; Zanuttigh, P.; and Michieli, U. 2025. Lora.rar: Learning to merge loras via hypernetworks for subject-style conditioned image generation. In ICCV. Stallkamp et al. (2011) Stallkamp, J.; Schlipsing, M.; Salmen, J.; and Igel, C. 2011. The German traffic sign recognition benchmark: a multi-class classification competition. In IJCNN, 1453–1460. IEEE. Sun et al. (2025) Sun, W.; Li, Q.; Li, B.; Wang, W.; and Geng, Y. 2025. Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts. In ICLR. Tan, Zhang, and Fu (2024) Tan, C.; Zhang, G.; and Fu, J. 2024. Massive Editing for Large Language Models via Meta Learning. In ICLR. Wan et al. (2025) Wan, W.; Lao, Q.; Xie, Z.; Wu, H.; Lin, R.; Lin, L.; and Wang, K. 2025. Massive Editing for Large Language Models Based on Dynamic Weight Generation. Wortsman et al. (2022) Wortsman, M.; Ilharco, G.; Gadre, S. Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A. S.; Namkoong, H.; Farhadi, A.; Carmon, Y.; Kornblith, S.; et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML. Xiao et al. (2016) Xiao, J.; Ehinger, K. A.; Hays, J.; Torralba, A.; and Oliva, A. 2016. Sun database: Exploring a large collection of scene categories. IJCV, 119: 3–22. Yadav et al. (2023) Yadav, P.; Tam, D.; Choshen, L.; Raffel, C. A.; and Bansal, M. 2023. Ties-merging: Resolving interference when merging models. In NeurIPS, 7093–7115. Yang et al. (2024) Yang, E.; Wang, Z.; Shen, L.; Liu, S.; Guo, G.; Wang, X.; and Tao, D. 2024. AdaMerging: Adaptive Model Merging for Multi-Task Learning. In ICLR. Yu et al. (2024) Yu, L.; Yu, B.; Yu, H.; Huang, F.; and Li, Y. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML. Zeng et al. (2025) Zeng, F.; Guo, H.; Zhu, F.; Shen, L.; and Tang, H. 2025. Robustmerge: Parameter-efficient model merging for mllms with direction robustness. In NeurIPS. Zhang et al. (2025) Zhang, B.; Du, Y.; Zhen, X.; and Shao, L. 2025. Variational Task Vector Composition. In NeurIPS. Supplementary Material This supplementary material provides further experimental results and theoretical proof details. Further Experimental Results Results on Additional Backbones. To verify that the benefits of HyperFix are not specific to a single backbone scale, we additionally evaluate on CLIP ViT-B/16 and ViT-L/14. Table 5 shows that the overall trend observed on ViT-B/32 remains consistent across larger backbones. Linear merging methods degrade as the subset size increases, whereas Mean + HyperFix maintains much stronger and more stable performance across all subset sizes. For example, on ViT-B/16, the performance of Mean drops from 96.4% at |S|=2|S|=2 to 77.5% at |S|=8|S|=8, while Mean + HyperFix retains 92.7% at |S|=8|S|=8 and improves the overall average from 85.0% to 95.1%. Similar behavior is observed on ViT-L/14, where HyperFix again achieves the best average performance and remains stable as more tasks are merged. Each configuration is trained once. Reported means and standard deviations are computed across all (8|S|) 8|S| task subsets. Table 5: Generalization performance across additional backbones. Results are averaged over all (8|S|) 8|S| subsets and reported as normalized accuracy (%). Values after ± denote the standard deviation across subsets. Avg. reports the average normalized accuracy across subset sizes |S|=2,…,8|S|=2,…,8. Backbone Method |S|=2|S|=2 |S|=3|S|=3 |S|=4|S|=4 |S|=5|S|=5 |S|=6|S|=6 |S|=7|S|=7 |S|=8|S|=8 Avg. ViT-B/16 Mean 96.4± 1.6 91.2± 2.5 86.7± 2.8 83.3± 2.6 80.9± 2.1 79.0± 1.5 77.5± 0.0 85.0 Sum + Scalar 97.4± 1.1 94.0± 1.5 90.7± 1.6 87.7± 1.6 85.1± 1.5 83.0± 1.2 81.2± 0.0 88.4 Mean + HyperFix 98.3± 1.1 97.0± 1.4 95.8± 1.4 94.8± 1.3 94.0± 1.1 93.3± 0.8 92.7± 0.0 95.1 ViT-L/14 Mean 98.0± 1.0 94.7± 1.8 91.7± 1.9 89.2± 1.8 87.2± 1.5 85.6± 1.2 84.4± 0.0 90.1 Sum + Scalar 98.8± 0.8 97.3± 1.0 95.6± 1.0 94.0± 1.1 92.3± 0.9 90.8± 0.7 89.7± 0.0 94.1 Mean + HyperFix 98.9± 0.7 97.9± 0.8 97.0± 0.9 96.1± 0.8 95.4± 0.6 94.8± 0.5 94.3± 0.0 96.4 Table 6: Absolute accuracy across task subsets. Results are averaged over all (8|S|) 8|S| subsets and reported as absolute accuracy (%). Values after ± denote the standard deviation across subsets. Avg. reports the average absolute accuracy across subset sizes |S|=2,…,8|S|=2,…,8. Backbone Method |S|=2|S|=2 |S|=3|S|=3 |S|=4|S|=4 |S|=5|S|=5 |S|=6|S|=6 |S|=7|S|=7 |S|=8|S|=8 Avg. Standard Fine-Tuning ViT-B/32 Mean 86.6± 7.1 81.1± 5.0 76.2± 3.9 72.3± 3.0 69.3± 2.3 67.1± 1.6 65.4± 0.0 74.0 Sum + Scalar 87.3± 7.1 83.5± 4.8 80.0± 3.4 76.8± 2.6 74.1± 2.0 71.7± 1.4 69.8± 0.0 77.6 TIES + Scalar 88.2± 6.9 85.4± 4.9 82.4± 3.6 79.5± 2.8 77.0± 2.2 74.8± 1.7 73.1± 0.0 80.1 Mean + HyperFix 88.4± 7.7 87.2± 6.0 86.2± 4.8 85.3± 3.8 84.6± 2.9 84.0± 2.0 83.5± 0.0 85.6 TIES + HyperFix 89.1± 7.3 88.6± 5.6 87.8± 4.5 86.9± 3.6 86.0± 2.8 85.2± 1.9 84.3± 0.0 86.8 Tangent-Space Fine-Tuning Mean 84.5± 6.6 80.9± 5.4 77.7± 4.5 75.1± 3.6 72.9± 2.8 71.2± 2.0 69.7± 0.0 76.0 Sum + Scalar 85.3± 6.4 83.3± 4.9 81.3± 4.0 79.4± 3.2 77.8± 2.5 76.4± 1.8 75.1± 0.0 79.8 TIES + Scalar 85.1± 6.4 82.6± 4.9 80.4± 3.9 78.5± 3.1 77.0± 2.4 76.0± 1.7 74.5± 0.0 79.1 Mean + HyperFix 86.5± 6.7 85.6± 5.1 84.7± 4.0 83.9± 3.2 83.0± 2.5 82.2± 1.7 81.5± 0.0 83.9 TIES + HyperFix 86.3± 6.8 85.8± 5.2 85.1± 4.1 84.4± 3.2 83.7± 2.4 83.0± 1.6 82.3± 0.0 84.4 ViT-B/16 Mean 89.3± 6.1 84.6± 5.0 80.4± 4.2 77.3± 3.4 74.9± 2.6 73.1± 1.8 71.7± 0.0 78.8 Sum + Scalar 90.2± 5.9 87.2± 4.3 84.2± 3.4 81.4± 2.7 79.1± 2.2 77.1± 1.5 75.5± 0.0 82.1 Mean + HyperFix 91.0± 5.9 89.8± 4.8 88.8± 3.9 87.9± 3.2 87.1± 2.5 86.5± 1.7 86.0± 0.0 88.2 ViT-L/14 Mean 92.4± 4.8 89.3± 4.2 86.5± 3.4 84.1± 2.8 82.2± 2.2 80.7± 1.6 79.6± 0.0 85.0 Sum + Scalar 93.1± 4.6 91.8± 3.4 90.2± 2.6 88.6± 2.1 87.1± 1.6 85.7± 1.1 84.6± 0.0 88.7 Mean + HyperFix 93.2± 4.8 92.3± 3.7 91.4± 3.0 90.7± 2.4 90.0± 1.9 89.4± 1.3 89.0± 0.0 90.9 Absolute Accuracy on Task Subsets. While the main paper reports normalized accuracy, we additionally report absolute accuracy. Table 6 summarizes the results across all task subsets. The overall trends are consistent with the normalized accuracy results. Linear merging methods degrade substantially as the subset size increases, whereas HyperFix maintains much stronger performance across all subset sizes. For example, under standard fine-tuning on ViT-B/32, Mean decreases from 86.6% at |S|=2|S|=2 to 65.4% at |S|=8|S|=8, while Mean + HyperFix retains 83.5%. TIES + HyperFix further improves this to 84.3%, consistently outperforming scalar-based baselines. A similar pattern holds under tangent-space fine-tuning, where HyperFix variants achieve the strongest performance across all subset sizes. The larger standard deviations in absolute accuracy mainly arise from differences in the performance of the single-task fine-tuned models across datasets. Since normalized accuracy is divided by the corresponding single-task performance, it reduces this cross-dataset variation and exhibits smaller variance. Overall, these results confirm that the improvements of HyperFix persist when evaluated using absolute accuracy. Table 7: Full results for maximum training subset size. HyperFix is trained on subsets with size |S|≤|S|max|S|≤|S|_ . Reported values are the average normalized accuracy (%) across all (8|S|) 8|S| task subsets. |S|max|S|_ |S|=2|S|=2 |S|=3|S|=3 |S|=4|S|=4 |S|=5|S|=5 |S|=6|S|=6 |S|=7|S|=7 |S|=8|S|=8 Avg. 1 95.7± 1.4 89.6± 2.2 84.3± 2.6 80.1± 2.5 77.0± 2.1 74.6± 1.5 72.8± 0.0 82.0 2 98.0± 0.7 95.3± 1.2 92.4± 1.4 89.8± 1.4 87.5± 1.2 85.6± 0.9 84.0± 0.0 90.4 3 97.7± 1.7 96.3± 1.8 95.1± 1.6 94.0± 1.4 93.2± 1.1 92.6± 0.8 92.0± 0.0 94.4 4 97.6± 1.7 96.5± 1.8 95.7± 1.7 95.0± 1.4 94.5± 1.2 94.2± 0.8 93.8± 0.0 95.3 5 98.1± 1.3 96.9± 1.5 96.0± 1.5 95.3± 1.3 94.9± 2.2 94.6± 0.8 94.3± 0.0 95.7 6 97.6± 2.2 97.6± 2.2 96.0± 1.7 95.5± 1.4 95.1± 1.1 94.8± 0.8 94.6± 0.0 95.9 7 97.9± 1.5 96.9± 1.6 96.1± 1.5 95.6± 1.3 95.2± 1.1 94.9± 0.8 94.7± 0.0 95.9 8 97.8± 1.8 96.8± 1.7 96.1± 1.6 95.6± 1.3 95.2± 1.9 94.9± 0.8 94.7± 0.0 95.9 Full Results for the Training Subset Size Ablation. In the main paper, we report the average normalized accuracy across all subset sizes to analyze the effect of the maximum training subset size |S|max|S|_ . For completeness, Table 7 provides the full performance across subset sizes. Consistent with the trends in the main paper, increasing the training subset size from |S|max=1|S|_ =1 to |S|max=3|S|_ =3 leads to substantial improvements across most subset sizes. For example, the average accuracy increases from 82.0% to 94.4%. This suggests that learning pairwise and triple task interactions plays an important role in modeling nonlinear task vector merging. Beyond |S|max≥4|S|_ ≥ 4, additional gains become relatively small. This saturation indicates that a large part of the interaction effects can already be captured by pairwise and triple task interactions. Theory Proof Details Proof of Theorem 1: A bound on the residual correction magnitude Lemma 1 (Residual correction is controlled by local geometry). Under the assumptions in the main paper, the ideal correction ΔS⋆ _S satisfies ‖ΔS⋆‖≤1μ‖∇LS(θ0+τS)‖.\| _S\|\;≤\; 1μ\,\|∇ L_S( _0+ _S)\|. (18) Moreover, local smoothness gives ‖∇LS(θ0+τS)‖≤‖∇LS(θ0)‖+H‖τS‖.\|∇ L_S( _0+ _S)\|≤\|∇ L_S( _0)\|+H\| _S\|. (19) Since ‖τS‖≤ρ\| _S\|≤ρ, we further have ‖∇LS(θ0)‖+H‖τS‖≤‖∇LS(θ0)‖+Hρ.\|∇ L_S( _0)\|+H\| _S\|≤\|∇ L_S( _0)\|+Hρ. (20) In the common case where ‖∇LS(θ0)‖\|∇ L_S( _0)\| is small (e.g., θ0 _0 is a good shared initializer), this yields ‖ΔS⋆‖=O(ρ)\| _S\|=O(ρ). Proof. By definition, ∇LS(θS⋆)=0∇ L_S( _S )=0 and θS⋆=θ0+τS+ΔS⋆ _S = _0+ _S+ _S . Define the averaged Hessian along the segment from θ0+τS _0+ _S to θS⋆ _S by AS:=∫01∇2LS(θ0+τS+tΔS⋆)t.A_S:= _0^1∇^2L_S ( _0+ _S+t _S )\,dt. (21) The mean-value form of Taylor’s theorem then gives ∇LS(θS⋆)=∇LS(θ0+τS)+ASΔS⋆.∇ L_S( _S )=∇ L_S( _0+ _S)+A_S _S . (22) Since ∇LS(θS⋆)=0∇ L_S( _S )=0, it follows that ASΔS⋆=−∇LS(θ0+τS).A_S _S =-∇ L_S( _0+ _S). (23) By local stability, each Hessian inside the integral has a minimum eigenvalue at least μ, hence the averaged Hessian is also ⪰μI μ I and is invertible with operator norm at most 1/μ1/μ. Therefore, ‖ΔS⋆‖≤1μ‖∇LS(θ0+τS)‖.\| _S\|≤ 1μ\|∇ L_S( _0+ _S)\|. (24) For the second inequality, apply Taylor’s theorem to ∇LS∇ L_S at θ0 _0: ∇LS(θ0+τS) ∇ L_S( _0+ _S) =∇LS(θ0) =∇ L_S( _0) (25) +(∫01∇2LS(θ0+tτS)dt)τS. + ( _0^1∇^2L_S( _0+t _S)\,dt ) _S. Taking norms and using ‖∇2LS(⋅)‖op≤H\|∇^2L_S(·)\|_op≤ H (since it averages task Hessians bounded by H), ‖∇LS(θ0+τS)‖≤‖∇LS(θ0)‖+H‖τS‖.\|∇ L_S( _0+ _S)\|≤\|∇ L_S( _0)\|+H\| _S\|. (26) Finally, ‖τS‖≤1|S|∑i∈S‖τi‖≤ρ\| _S\|≤ 1|S| _i∈ S\| _i\|≤ρ. ∎ Proof of Theorem 1: Why higher-order interactions decay as O(ρ2)O(ρ^2) Lemma 2 (Third-order Taylor remainder bound). Under the smoothness assumptions, for any u with ‖u‖≤2ρ\|u\|≤ 2ρ, ∇LS(θ0+u)=∇LS(θ0)+∇2LS(θ0)u+RS(u),∇ L_S( _0+u)=∇ L_S( _0)+∇^2L_S( _0)\,u+R_S(u), (27) where the remainder satisfies ‖RS(u)‖≤M2‖u‖2.\|R_S(u)\|≤ M2\|u\|^2. (28) Proof. This is the standard integral remainder form for the gradient: ∇LS(θ0+u) ∇ L_S( _0+u) =∇LS(θ0)+∇2LS(θ0)u =∇ L_S( _0)+∇^2L_S( _0)u (29) +∫01[∇2LS(θ0+tu)−∇2LS(θ0)]udt. + _0^1 [∇^2L_S( _0+tu)-∇^2L_S( _0) ]u\,dt. Using the mean value form for Hessians with bounded third derivative, ‖∇2LS(θ0+tu)−∇2LS(θ0)‖op≤Mt‖u‖\|∇^2L_S( _0+tu)-∇^2L_S( _0)\|_op≤ M\,t\|u\|, we obtain ‖RS(u)‖≤∫01Mt‖u‖⋅‖u‖t=M2‖u‖2.\|R_S(u)\|≤ _0^1M\,t\|u\|·\|u\|\,dt= M2\|u\|^2. (30) ∎ Lemma 3 (Residual contains only low-order effects up to O(ρ2)O(ρ^2)). Assume additionally that ‖∇LS(θ0)‖\|∇ L_S( _0)\| is small (or treat it as a constant bias term). Let u=τSu= _S. Then ‖∇LS(θ0+u)‖≤‖∇LS(θ0)‖+‖∇2LS(θ0)‖op‖u‖+M2‖u‖2≤‖∇LS(θ0)‖+Hρ+M2ρ2. gathered\|∇ L_S( _0+u)\|\\ ≤\|∇ L_S( _0)\|+\|∇^2L_S( _0)\|_op\|u\|+ M2\|u\|^2\\ ≤\|∇ L_S( _0)\|+Hρ+ M2ρ^2. gathered (31) Consequently, by Lemma 1, ‖(∫01∇2LS(θ0+τS+tΔS⋆)t)−1RS‖≤M2μρ2. \| ( _0^1∇^2L_S( _0+ _S+t _S)\,dt )^-1R_S \|≤ M2μρ^2. (32) Proof. Apply Lemma 2 with u=τSu= _S and ‖u‖≤ρ\|u\|≤ρ. Then combine with Lemma 1. ∎ Interpretation. The above shows (conservatively) that when task vectors are small, and the loss is smooth, the needed correction is dominated by low-order Taylor terms. Any interaction effects that require higher-order derivatives become rapidly smaller with ρ. Proof of Theorem 2: Why a hypernetwork conditioned on zSz_S can generalize HyperFix predicts ΔShyper=hϕ(zS) ^hyper_S=h_φ(z_S) with zS=1|S|∑i∈Sziz_S= 1|S| _i∈ Sz_i. The key property is that zSz_S is size-invariant and averages per-task interaction descriptors. Lemma 4 (Stability of the set embedding under increasing subset size). Assume the vectors ziz_i have bounded second moment under the (empirical) task distribution: ‖zi−z‖2≤σz2E\|z_i-Ez\|^2≤ _z^2. If a subset S of size m is sampled by drawing tasks approximately i.i.d. from this distribution, then ‖zS−z‖≤σzm.E\|z_S-Ez\|≤ _z m. (33) Proof. Since zSz_S is the average of m samples, its variance scales as 1/m1/m: ‖zS−z‖2≤σz2/mE\|z_S-Ez\|^2≤ _z^2/m. Jensen’s inequality gives ‖zS−z‖≤‖zS−z‖2≤σz/mE\|z_S-Ez\|≤ E\|z_S-Ez\|^2≤ _z/ m. ∎ Lemma 5 (Why larger subsets are not harder in embedding space). Under Assumption A4, if g is LgL_g-Lipschitz, then ‖ΔS⋆−g(z)‖≤ε+Lgσzm.E\| _S-g(Ez)\|≤ + L_g _z m. (34) Proof. By the triangle inequality and Assumption A4, ‖ΔS⋆−g(z)‖≤‖g(zS)−g(z)‖+‖η(S)‖≤Lg‖zS−z‖+ε. gathered\| _S-g(Ez)\|\\[1.0pt] ≤\|g(z_S)-g(Ez)\|+\|η(S)\|\\[1.0pt] ≤ L_g\|z_S-Ez\|+ . gathered (35) Taking expectations and applying Lemma 4 yields the result. ∎