Paper deep dive
On-the-fly Weight Generation: A Hypernetwork Proof of Concept on ARC-1D
Fabio J. Fehr, Philip Torr
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/2/2026, 9:40:36 AM
Summary
This paper presents a proof-of-concept for on-the-fly weight generation using hypernetworks to create specialized, compact models from few-shot demonstrations on the ARC-1D task. The authors demonstrate that a hypernetwork can infer task-specific transformations from input-output pairs and generate the complete parameters of a tiny specialist model. These generated specialists achieve performance comparable to individually trained models, form a structured weight space that supports generalization to unseen transformations, and benefit from the removal of explicit task identifiers.
Entities (6)
Relation Signals (4)
ARC-1D → usedforevaluation → Hypernetwork
confidence 96% · Using ARC-1D as a controlled testbed, we show that individual transformations can be represented by tiny specialist models, and that a hypernetwork can generate their parameters from context.
Hypernetwork → generatesparametersfor → Canon Transformer
confidence 95% · a hypernetwork can generate their parameters from context... a three-layer Canon Transformer... with its parameters are generated anew from each support set
Hypernetwork → enables → On-the-fly Weight Generation
confidence 94% · Together, these results provide a proof of concept that few-shot task context can be compiled on-the-fly into compact executable model parameters
Task Identifier → removalimproves → Generalization
confidence 90% · In both settings, removing explicit task identifiers improves generalisation beyond the training transformations.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:General-purpose models can adapt to many tasks from context, while specialised models can execute individual functions with less capacity. Yet obtaining such specialists requires task-specific training or adaptation. We ask whether they can instead be generated directly from a few demonstrations. Using ARC-1D as a controlled testbed, we show that individual transformations can be represented by tiny specialist models, and that a hypernetwork can generate their parameters from context. The generated parameters form a structured weight space, while the resulting specialists show partial compositional generalisation and generalisation to transformations not seen during training. In both settings, removing explicit task identifiers improves generalisation beyond the training transformations. Together, these results provide a proof of concept that few-shot task context can be compiled on-the-fly into compact executable model parameters, and that the resulting weight space can support reuse and generalisation beyond known functions.
Tags
Links
- Source: https://arxiv.org/abs/2610.00820v1
- Canonical: https://arxiv.org/abs/2610.00820v1
Trouble viewing inline? Open PDF directly →
Full Text
68,806 characters extracted from source content.
Expand or collapse full text
On-the-fly Weight Generation: A Hypernetwork Proof of Concept on ARC-1D Fabio J. Fehr University of Oxford Philip Torr University of Oxford Code Abstract General-purpose models can adapt to many tasks from context, while specialised models can execute individual functions with less capacity. Yet obtaining such specialists requires task-specific training or adaptation. We ask whether they can instead be generated directly from a few demonstrations. Using ARC-1D as a controlled testbed, we show that individual transformations can be represented by tiny specialist models, and that a hypernetwork can generate their parameters from context. The generated parameters form a structured weight space, while the resulting specialists show partial compositional generalisation and generalisation to transformations not seen during training. In both settings, removing explicit task identifiers improves generalisation beyond the training transformations. Together, these results provide a proof of concept that few-shot task context can be compiled on-the-fly into compact executable model parameters, and that the resulting weight space can support reuse and generalisation beyond known functions. 1 Introduction In-context learning has shown that large general-purpose models can infer task-specific behaviour from examples at inference time, without explicit gradient-based adaptation [Brown et al., 2020, Min et al., 2022, Dong et al., 2024]. At the same time, specialised models can execute individual functions with substantially less capacity, but typically require task-specific training, fine-tuning, or distillation [Hsieh et al., 2023, Wang et al., 2020, Tang et al., 2019]. This suggests an alternative form of inference-time adaptation: rather than retaining the inferred behaviour within a general-purpose model, it could be instantiated as a lightweight task-specific model and executed independently. This perspective also extends the paradigm of tool use in language-model agents [Schick et al., 2023, Jin et al., 2025, Qian et al., 2025] by allowing a model to construct a specialised model rather than select a predefined tool or generate code. This leads to our central question: can compact task-specific models be constructed on-the-fly from only a few demonstrations? More generally, this can be viewed as a problem of few-shot function induction. A few demonstrations specify a function that must be inferred and applied to new inputs. While this function can remain implicit in context or be expressed through programs or latent representations, we instead consider representing it directly as model parameters. Hypernetworks provide one mechanism for doing so by mapping contextual information to model weights [Ha et al., 2017, Chauhan et al., 2024]. We study whether demonstrations can be used to instantiate the complete parameters of a compact standalone network, separating task inference from task execution. By representing functions as model parameters, we can study whether weight space reflects structure in the functions themselves. Recent work has modelled neural-network weights as a generative domain [Han et al., 2026, Zeng et al., 2026, Wang et al., 2026], while meta-learning and abstract reasoning have explored structured parameter, program, and function spaces that support adaptation and composition [Rusu et al., 2019, Macfarlane and Bonnet, 2025, Shirakami and Henderson, 2026]. A related challenge is whether this can be achieved without an explicit task identifier: tiny-model Preprint. arXiv:2610.00820v1 [cs.LG] 30 Sep 2026 Figure 1: Hypernetwork weight generation. The support setZ E = (x i ,y i ) K i=1 is encoded by the task encoderφ ω into an episode representationz E , which the weight decoderψ ρ maps to the complete parametersw E of the episode-specific predictorf (·;w E ). Once generated, the predictor operates independently of the support set, receiving only a new input x ′ to produce its output. approaches to ARC-AGI often use task-specific conditioning [Wang et al., 2025a, Jolicoeur-Martineau, 2025, Gao et al., 2025], while larger-scale text-to-LoRA methods condition generated adaptations on explicit task descriptions [Charakorn et al., 2025]. These considerations lead to two questions: do related functions occupy related regions of weight space that support reuse and generalisation, and can the underlying function be inferred from support examples alone, without an explicit task identifier? We study these questions on ARC-1D [Xu et al., 2024], a controlled setting at the intersection of abstract reasoning and sequence modelling. Like ARC-AGI [Chollet, 2019, Chollet et al., 2026, ARC Prize Foundation, 2026], ARC-1D requires inferring an unknown transformation from a few input–output demonstrations, but operates over one-dimensional discrete sequences that can be modelled with compact sequence architectures. This makes ARC-1D a useful testbed for studying few-shot function induction and the generation, reuse, and generalisation of compact executable models. Our goal is not to build a specialised ARC-1D solver, but to study whether demonstrations can be converted on-the-fly into compact model parameters. Our contributions are as follows: •Specialisation reduces the capacity required for task execution. Tiny models accurately represent individual ARC-1D transformations, while the same architectures degrade when trained jointly across transformations, even with access to task identifier (Section 4.2). •Hypernetworks can generate compact specialists directly from demonstrations. The generated models separate task inference from execution and form a structured weight space organised by transformation, with related functions occupying nearby regions (Section 4.3). •Generated specialists encode reusable functions and generalise beyond known trans- formations. Once generated from a support set, a specialist can be decoupled from the hypernetwork and reused on new instances of the same transformation. Removing task identity further improves transfer to novel compositions of known transformations and to transformations not seen during training (Section 4.4). 2 Related Work Small and Specialised Models. Large general-purpose models provide broad capabilities, while smaller task-specific models can achieve strong performance on narrower tasks with substantially less capacity [Hsieh et al., 2023]. Such specialists are commonly obtained through fine-tuning or distillation [Wang et al., 2020, Tang et al., 2019], while lottery-ticket results suggest that much smaller subnetworks can retain the functionality of larger networks [Malach et al., 2020]. Recent abstract-reasoning models, including HRM [Wang et al., 2025a], TRM [Jolicoeur-Martineau, 2025], and URM [Gao et al., 2025], further show that small networks can solve complex reasoning tasks through iterative computation, extended through stochastic exploration [Sghaier et al., 2026] and 2 adaptive fixed-point iteration [Movahedi et al., 2026]. Similarly, neural algorithmic reasoning has progressed from specialist models executing individual algorithms towards generalist models learning across collections of algorithms [Ibarz et al., 2022]. Rather than training each specialist independently or representing all functions within shared parameters, we ask whether a general model can infer a function from a few demonstrations and directly generate the corresponding compact specialist. Hypernetworks and Weight Generation. Hypernetworks generate the parameters of a target network with a forward pass, enabling rapid instantiation of task-specific weights [Ha et al., 2017, Chauhan et al., 2024]. Subsequent work has improved scalability through weight chunking [Zhao et al., 2020], low-rank factorisations [Bertinetto et al., 2016, Zhmoginov et al., 2022], and parameter- efficient generation, including adapters and LoRA updates [Karimi Mahabadi et al., 2021, Ruiz et al., 2024, Charakorn et al., 2025, Liang et al., 2025]. In parallel, generative weight-space models learn distributions over neural-network parameters [Han et al., 2026, Zeng et al., 2026, Wang et al., 2026], using conditional diffusion to generate task- or domain-specific parameters [Jin et al., 2024, Xie et al., 2024], recurrent diffusion to scale to complete large networks [Wang et al., 2025b], and flow matching for efficient full-network generation [Gupta et al., 2026]. Rather than generating from collections of trained parameters or explicit task conditions, we infer a function from a few input–output demonstrations and instantiate it as the complete weights of a small executable model. We further ask whether generating many such models induces a structured weight space that supports generalisation between functions. Abstract Reasoning and Function Induction. ARC-AGI [Chollet, 2019] frames abstract rea- soning as few-shot function induction, where an unknown transformation must be inferred from input–output demonstrations and applied to a held-out query. ARC-1D translates this setting to one-dimensional discrete sequences, providing a controlled setting for studying function induction with sequence models [Xu et al., 2024]. Approaches to ARC style problems differ in how this inferred function is represented and recovered, ranging from direct prediction to explicit or latent program induction [Li et al., 2025]. Latent Program Networks, for example, learn a latent program space and search within it at test time [Macfarlane and Bonnet, 2025], while recent work on ARC-1D learns structured latent function spaces that support compositional generalisation [Shirakami and Henderson, 2026]. Related work has proposed puzzle-specific weight perturbations and suggests that hypernetworks mapping puzzle embeddings to model parameters as a direction for ARC [Liao and Gu, 2025]. Similarly, HRM conditions predictions on learned puzzle-ID embeddings rather than few-shot context; subsequent analysis identifies this as a major limitation because inference is restricted to puzzle identities observed during training, motivating approaches that infer task identity directly from demonstrations [Schürholt, 2025]. We instead amortise function induction into a hypernetwork that maps demonstrations directly to the complete parameters of a reusable specialist, and ask whether task identity can be inferred from these demonstrations without an explicit puzzle embedding. We further study whether the resulting weight space supports generalisation to novel compositions and transformations not seen during training. Meta-Learning and Few-Shot Adaptation. Our formulation is closely related to meta-learning, where models learn across tasks to adapt from few examples [Hospedales et al., 2022]. Gradient- based methods such as MAML [Finn et al., 2017] and Reptile [Nichol et al., 2018] learn shared parameters that can be adapted to each task through a small number of optimisation steps, while LEO [Rusu et al., 2019] performs this task-specific optimisation in a learned low-dimensional latent parameter space. Other approaches avoid task-specific optimisation through differentiable solvers [Bertinetto et al., 2019], fast weights [Munkhdalai and Yu, 2017], or direct prediction of task-specific parameters [Bertinetto et al., 2016, Nava et al., 2022]. Our setting differs from conventional few-shot meta-learning because augmentation provides many training episodes for each transformation; the few-shot constraint applies within each episode, where a small support set must specify the function and the parameters that execute it. This follows the data-augmented training regime common to small ARC models. 3 3 Methodology 3.1 Episodic Meta-Learning Formulation We formulate weight generation as episodic meta-learning. LetEdenote a collection of episodes, where eachE ∈ E contains a support set and a query set, Z E =(x i ,y i ) K i=1 , Z ′ E =(x ′ i ,y ′ i ) Q i=1 . The support and query examples share the same underlying transformation. For ARC-1D,K = 3and Q = 1, with inputs and outputs given by variable-length sequencesx,y ∈V ∗ overV =0,..., 9. At test time, the model observes Z E and the query input x ′ , and must predict its output y ′ . At the level of an individual episode, a base learnerΛmaps the support examples to episode-specific parameters w E ∈ R p , w E = Λ(Z E ),ˆy ′ = f (x ′ ;w E ),(1) wheref (·;w E )is the episode-specific predictor. Meta-learning learns across episodes how to construct these predictors. In our formulation,Λis a learned feed-forward weight generator that produces the complete parameters offdirectly from the support examples, without task-specific optimisation. The query is therefore held out from the construction ofw E , while its target is used to supervise the weight generator during meta-training. 3.2 Hypernetwork Weight Generation We implement the base learnerΛas an encoder–decoder hypernetwork (Figure 1). A task encoder φ ω compresses the support set into an episode representationz E , and a weight decoderψ ρ maps this representation to the complete parameters of the episode-specific predictor: z E = φ ω (Z E ), w E = ψ ρ (z E ),Λ ω,ρ = ψ ρ ◦ φ ω .(2) Task inference and task execution are therefore separated: the support examples are processed by the hypernetwork, while the generated parametersw E provide the interface to the predictor that executes the inferred transformation. Task encoder. The encoderφ ω is a Transformer operating on theKsupport pairs. Each token is represented as e(t) = e tok (t) + e ex (i) + e role (t), combining token, support-example, and input/output-role embeddings. RoPE [Su et al., 2024] provides positional information, with positions reset for each input and output sequence. Learned- query attention pooling reduces the resulting support representations to a single episode representation z E ∈ R d h . Weight decoder.In the task-ID-conditioned variant, we augment the support-derived representation as ̃z E = z E W id c E , wherec E is a one-hot transformation identifier andW id ∈ R d h ×14 is randomly initialised and kept fixed during training. Without task-ID conditioning, no additional information is provided and ̃z E = z E . The decoder ψ ρ is a two-layer MLP without a bias term, z ′ E = GELU(W 1 ̃z E ), w E = W 2 z ′ E ,(3) withW 1 ∈ R d ′ ×d h ,W 2 ∈ R p×d ′ , andd ′ ≪ p. The intermediate representationz ′ E therefore provides a low-dimensional parameter-space bottleneck: althoughw E ∈ R p specifies the complete target model, all generated weight vectors lie in the learned subspace spanned byW 2 . Thus, the transformation inferred from the demonstrations must be expressed through a constrained region of the target model’s parameter space. Episode-specific predictor. The generated vectorw E ∈ R p contains all parameters off (·;w E ), a three-layer Canon Transformer [Allen-Zhu, 2025] with RoPE. This architecture is fixed across episodes, while its parameters are generated anew from each support set 1 . Oncew E has been generated,f (·;w E )is a standalone model: it can be separated from the hypernetwork and applied to new inputs using only its generated parameters, without access to either Z E or the encoder–decoder that produced them. 1 Indeed, generating the architecture and weights jointly would be more expressive, we leave this for future work. 4 3.3 Meta-Training Objective We train the hypernetwork end-to-end across episodes. For each episode, only the support setZ E is provided to the hypernetwork, which encodes it into an episode representation and decodes it into the target-model parametersw E . The generated predictor must then reconstruct the demonstrated input–output mapping through these parameters. This gives an autoencoder-like reconstruction task in which task information is compressed through a low-dimensional bottleneck and re-expressed as an executable model. The episode loss is L E (ω,ρ) = 1 |Z E ∪ Z ′ E | X (x,y)∈Z E ∪Z ′ E L pred (f (x;w E ),y),(4) where w E = ψ ρ (φ ω (Z E )). The meta-training objective averagesL E across training episodes. Unlike the held-out-query objective commonly used in episodic meta-learning, we supervise on both the support and query examplesZ E ∪ Z ′ E . The support loss requires the generated parameters to capture the demonstrated function, while the query loss requires that function to generalise beyond the demonstrations. Because the entire computation fromZ E throughw E to the predictions offis differentiable, these losses jointly train the task encoder φ ω and weight decoder ψ ρ . 4 Experiments 4.1 Data ARC-1D. We evaluate on ARC-1D 2 [Xu et al., 2024], a one-dimensional variant of ARC-AGI for studying abstract reasoning from few demonstrations with sequence models. As illustrated in Figure 2, inputs and outputs are sequences of coloured cells, with transformations involving concepts such as objects, counting, and spatial relations. Each episodeEcontains three support pairs, Z E = (x 1 ,y 1 ), (x 2 ,y 2 ), (x 3 ,y 3 ), and one query pair,Z ′ E = (x ′ ,y ′ ) , corresponding toK = 3 and Q = 1 in Section 3. x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????????? ̂ y 0 Figure 2: Flip transformation: Example ARC-1D episode The support setZ E contains three input–output pairs specifying the transformation, which is applied to the query input inZ ′ E . See Appendix G for more examples. Transformation subset. We use14of the18ARC-1D transformation categories, retaining those that can be executed by a standalone predictor without continued access to the support set. We additionally excludePadded_Fillbecause of its substantially different sequence-length distribution. ARC-1D provides50episodes for each transformation, which we split into40training,5validation, and5test episodes. The same transformation categories therefore occur across all three splits, while the episodes themselves are disjoint. In the task-ID-conditioned variant, the transformation category provides c E . Augmentation. Following ARC-style augmentation [Wang et al., 2025a], we generate1Kaug- mentations per base training episode using colour remapping and sequence shifts that preserve the underlying transformation. Colour mappings are sampled independently across examples, except for transformations requiring a globally consistent marker colour, and augmented training episodes matching held-out examples are discarded to prevent leakage. This yields561Kaugmented training episodes. Validation and test episodes are augmented separately using colour remapping only, with 20 remappings per held-out episode, yielding 100 evaluation episodes per category. 2 https://github.com/khalil-research/1D-ARC 5 Experimental Setup and Evaluation.Full architecture, optimisation, and implementation details are provided in Appendix A. We additionally ablate the use of Canon layers [Allen-Zhu, 2025] and the Muon optimiser [Liu et al., 2025] in Appendix B, finding that both improve downstream performance. We evaluate query predictions using exact-match accuracy as our primary metric, which requires the complete predicted sequence to match the target. We additionally report token (per-cell) accuracy when analysing partial solutions. 4.2 Specialised Models Require Less Capacity 1.4K2.4K5.4K9.5K Model parameters (log scale) 0% 20% 40% 60% 80% 100% Test exact match accuracy Individual models Joint model, Task ID Joint model, w/o Task ID Figure 3: Specialisation requires less capacity. Individual models saturate earlier than joint models, even when provided with transformation identity. Bands show±1 s.d. across 5 seeds. We first ask how much capacity is required to represent an individual ARC-1D transformation. Recent results show that small recurrent models can solve complex ARC tasks [Wang et al., 2025a, Jolicoeur-Martineau, 2025, Gao et al., 2025], motivating us to use a deliberately small target ar- chitecture. We progressively simplify a Transformer with RoPE [Su et al., 2024] to the three-layer Canon Trans- former [Allen-Zhu, 2025] used throughout our subsequent experiments. Our aim is not to identify an optimal archi- tecture, but to obtain a simple target model whose capacity can be varied directly. Full architecture configurations are provided in Appendix Table 3. Figure 3 reveals a clear cost to sharing parameters. Individ- ual models saturate at 98.9% accuracy with only 2.4K pa- rameters, while the same sized architecture reaches 74.0% when trained jointly, even when given the transformation identity. Increasing capacity closes this gap, suggesting that individual transformations are cheap to represent but require additional capacity when stored in shared parame- ters. This penalty is concentrated in a few transformations (Figure 4); most remain near their individual ceiling, while Flip, Mirror and Moves1,3fall from 100%. We provide figures of the alternative model sizes in Appendix C. Move 1Move 2 Move 2 Towards Move 3 Move Dynamic Scaling Fill Hollow Flip Mirror Denoise Denoise MC Pattern Copy Pattern Copy MC 0% 20% 40% 60% 80% 100% Validation exact match accuracy Individual modelsJoint model, Task IDJoint model, w/o Task ID Figure 4: The joint-training penalty is concentrated in a few transformations. Per-transformation exact-match validation accuracy at 2.4K parameters. Error bars show±1 s.d. across 5 seeds. These results separate the capacity required to represent multiple functions in shared parameters from that required to execute an individual function. This motivates separating task inference from execution: rather than storing all transformations in shared target-model parameters, a hypernetwork can infer the transformation from the support set and instantiate a small standalone predictor to execute it. This leads to our central question: given that a tiny network can represent each solution, can a hypernetwork learn to generate those networks from examples? 6 4.3 Generating Specialised Models with Hypernetworks The previous experiment showed that representing multiple transformations in shared target-model parameters incurs a capacity cost. We therefore ask whether a hypernetwork can instead generate a small, specialised model for each episode. We train a 10K-parameter hypernetworkΛ ω,ρ = ψ ρ ◦ φ ω jointly across all 14 transformations to map the support setZ E to the complete parametersw E of the same 1.4K-parameter target architecture used above. We compare generation from the support set alone with a task-ID-conditioned variant that additionally receives a frozen transformation embedding; alternative embedding placements and learned embeddings are considered in Appendix B. Full hypernetwork configurations are given in Appendix Table 3. Move 1Move 2 Move 2 Towards Move 3 Move Dynamic Scaling Fill Hollow Flip Mirror Denoise Denoise MC Pattern Copy Pattern Copy MC 0% 20% 40% 60% 80% 100% Validation exact match accuracy Individual modelHypernetwork, Task IDHypernetwork, w/o Task ID Figure 5: Generated models recover specialist performance. Per-task validation exact-match accuracy for individually trained and hypernetwork-generated models, with and without task IDs, using the smallest 1.4K-parameter target model. Error bars show±1 s.d. across 5 seeds. We find that hypernetwork-generated models largely recover the performance of individually trained specialists (Figure 5). We also test whether the hypernetwork improves data efficiency by varying the amount of augmented training data, but find no gain over individual models at this scale (Appendix D). We next consider whether transformations are reflected in the structure of the generated weight space in Figure 6. With a frozen task embedding, transformations are perfectly separated (100% linear-probe accuracy). Without task identity, structure still emerges from the support examples alone (60.9% linear-probe accuracy), with overlap concentrated among related transformations: the Move variants overlap with one another, as do the Pattern Copy variants. This suggests that related functions occupy nearby regions of weight space. PCA and UMAP show the same qualitative structure (Appendix Figures 17 & 18). −6−4−20 dim 1 −4 −2 0 2 4 6 dim 2 Task ID 3456789 dim 1 −4 −2 0 2 4 dim 2 w/o Task ID Denoise Denoise MC Fill Flip Hollow Mirror Move 1 Move 2 Move 2 Towards Move 3 Move Dynamic Pattern Copy Pattern Copy MC Scaling Figure 6: Generated parameters form a task-structured weight space. t-SNE projection of generated target-model parameters, coloured by transformation. Left: With a frozen task embedding, tasks are fully separable (100% linear probe accuracy). Right: Without task identity, structure emerges from the support examples alone (60.9% linear probe accuracy). 7 Together, these results show that a hypernetwork can generate small task-specific models while learning a structured weight space across transformations. However, all transformations considered so far were observed during training, and the partial overlap between related functions suggests that shared structure may be useful rather than a limitation. We therefore consider whether the hypernetwork can move beyond generating known functions: first by reusing generated models across examples of the same transformation, then by composing related transformations, and finally by generating models for entirely unseen tasks. 4.4 Towards Generalisation with Hypernetworks Figure 7: Generated specialists gen- eralise across episodes. Exact-match accuracy on the query paired with the generating support set and on queries from other episodes of the same transfor- mation. Bars show the macro-average over 14 transformations with±1 s.d. The previous section showed that hypernetworks can gen- erate effective specialists from demonstrations and that the resulting parameters exhibit structure across transforma- tions. We next ask whether these generated parameters encode functions that generalise beyond the support sets from which they were produced. We consider three set- tings: (1) reusing generated weights across new episodes of the same transformation, (2) generating weights for novel compositions of transformations observed indepen- dently during meta-training, and (3) generating weights for transformations not observed during meta-training. Generate Once, Execute Many. We next ask whether generated parameters encode reusable functions rather than support-specific solutions. For each transformation, we generatew E from one validation support set and hold it fixed while evaluatingf (·;w E )on all remaining validation and test queries, with no further support context, training, or adaptation. As shown in Figure 7, macro-averaged accuracy remains close to performance on the query paired with the generating support set, both with and without task-ID conditioning. Thus, a generated specialist can be instantiated once and reused across unseen inputs from the same transformation. Compositional Generalisation. We next test whether the generated specialists can transfer to combinations of functions learned independently during meta-training. We construct 10 held-out compositions of known transformations, such that each constituent transformation is observed during training but the composition itself is never seen (Figure 8). We denote a composition asB◦A, where Ais applied first andBsecond. We evaluate the same models in-distribution on the 14 transformations used during training and zero-shot on the held-out compositions (Table 1). Task-ID conditioning improves in-distribution performance, but this advantage reverses on the compositions: removing the task ID improves transfer, particularly at the token level. Exact-match performance remains near zero for both variants, with the few exact solutions concentrated in compositions involvingMirrorand Denoise MC(Appendix Table 6). Thus, explicit task identity improves in-distribution specialisation but appears to constrain the composition of known transformations into new functions. x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????????????????????? ̂ y 0 Figure 8: Fill◦Mirror: Example compositional ARC-1D episode The support pairs specify the composition of Mirror followed by Fill, where both transformations are observed independently during training but their composition is held out. See Appendix H for more examples. 8 Table 1: Task-ID conditioning improves in-distribution specialisation but hinders generalisation. Results for compositional and unseen transformations, reported as mean± s.d. over 5 seeds. In-DistributionGeneralisation Exact Match↑Token Accuracy↑Exact Match↑ Compositional Task ID 93.0± 2.467.7± 1.30.05± 0.10 w/o Task ID57.4± 5.1 (-35.6) 75.5± 1.1 (+7.8) 1.00± 0.71 (+0.95) Unseen Task ID 97.0± 0.568.5± 4.80.29± 0.57 w/o Task ID64.7± 3.2 (-32.3) 81.9± 0.4 (+13.4) 9.14± 1.14 (+8.85) Unseen Generalisation. We finally test transfer to transformations entirely absent from meta- training using leave-one-category-out evaluation: each of the 14 transformations is withheld in turn, with the hypernetwork trained on the remaining 13 and results macro-averaged across held-out transformations (Table 1). As with compositional generalisation, task-ID conditioning improves in- distribution performance but hinders transfer, with the support-only model achieving higher token and exact-match accuracy on unseen transformations. Unlike the compositional setting, exact solutions emerge more clearly, suggesting that support examples can provide a transferable description of an unseen function rather than relying solely on task identity. Per-transformation results are reported in Appendix Table 7. 5 Discussion From Task Identification to Model Construction. Our results suggest that few-shot function induction need not remain within a general-purpose model: a function inferred from demonstrations can instead be instantiated as a compact standalone specialist. Task-ID conditioning improves specialisation to known transformations but hinders generalisation beyond them, suggesting that functions can be inferred directly from demonstrations without explicit task identity. This extends the idea of tool use from selecting predefined tools to generating a specialised executable that can be reused independently. More broadly, a pretrained model could act as a model generator, using demonstrations to construct task-specific specialists on the fly. Towards Local and Decentralised Execution. Generated weights could provide a form of task- specific pretraining, giving a compact model a useful initialisation from only a few demonstrations. Rather than being the final model, this specialist could then be further fine-tuned or personalised using local data before being executed independently. This could reduce reliance on remote inference and allow task- or user-specific data to remain local, with potential benefits for privacy and security. More broadly, a large pretrained model could generate initial specialists that are subsequently refined and deployed on resource-constrained hardware. Our experiments do not yet establish these benefits in latency, memory, compute, privacy, or security, nor how the approach scales beyond tiny target models. 6 Conclusion We study whether few-shot function induction can be realised by generating task-specific model parameters directly from demonstrations. We show that individual ARC-1D transformations require little capacity but incur a substantial cost when represented in shared parameters (Section 4.2); that a hypernetwork can generate compact specialists whose parameters form a structured weight space (Section 4.3); and that these specialists can be reused across new instances and partially generalise to novel compositions and transformations not seen during training (Section 4.4). In both generalisation settings, removing explicit task identity improves transfer beyond the training transformations. Together, these results provide a proof of concept that few-shot task context can be compiled on-the-fly into compact, executable model parameters, separating task inference from execution and enabling reusable specialists that generalise beyond known functions. 9 Future Directions. This proof of concept suggests two directions. First, extending weight gen- eration beyond ARC-1D to richer reasoning settings such as ARC-AGI, including small looped or recursive specialists generated directly from demonstrations without explicit task identifiers. Second, scaling weight generation to larger models through structured or constrained parameter spaces, such as adapters or LoRA updates. Generated parameters could then provide a form of task-specific initialisation, transferring capabilities from a large pretrained model to a compact specialist that can be further refined using downstream or local data. This could enable specialists to be generated on demand, personalised, and executed independently. Limitations Our evaluation is deliberately controlled: ARC-1D contains a limited set of transformations, so it remains unclear whether weight generation scales to richer reasoning settings such as ARC-AGI or language modelling. Although specialists are generated from only a few demonstrations at inference time, the hypernetwork is trained with substantial data augmentation, so our results demonstrate few- shot task inference rather than learning the weight generator from few examples. We also generate complete parameters for small target networks; scaling to larger models may require structured or constrained parameter spaces or parameter-efficient updates such as adapters or LoRAs. Finally, while generated specialists can be reused across inputs, we do not measure end-to-end efficiency gains in latency, memory, or compute relative to alternative inference-time adaptation methods. Broader impacts Generating compact specialised models on-the-fly could enable cheaper inference and local de- ployment, reducing reliance on large-scale GPU infrastructure and remote API services. This may improve accessibility and privacy while reducing computational and energy costs. However, we do not measure these efficiency benefits directly, and establishing them remains future work. 10 References Zeyuan Allen-Zhu. Physics of language models: Part 4.1, architecture design and the magic of canon layers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps: //openreview.net/forum?id=kxv0M6I7Ud. ARC Prize Foundation. Arc-agi-3: A new challenge for frontier agentic intelligence. arXiv preprint arXiv:2603.24621, 2026. Luca Bertinetto, João F. Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed-forward one-shot learners. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URLhttps://proceedings. neurips.c/paper_files/paper/2016/file/839ab46820b524afda05122893c2fe8e-Paper.pdf. Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed- form solvers. In International Conference on Learning Representations, 2019. URLhttps://openreview. net/forum?id=HyxnZh0ct7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Pro- cessing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020. URLhttps://proceedings. neurips.c/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, and Robert Tjarko Lange. Text-to-loRA: Instant trans- former adaption. In Forty-second International Conference on Machine Learning, 2025. URLhttps: //openreview.net/forum?id=zWskCdu3QA. Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. A brief review of hypernetworks in deep learning. Artificial Intelligence Review, 57(9), August 2024. ISSN 1573-7462. doi: 10.1007/s10462-024-10862-8. URL http://dx.doi.org/10.1007/s10462-024-10862-8. François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. François Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2025: Technical report, 2026. URL https://arxiv.org/abs/2601.10904. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.64. URLhttps://aclanthology.org/ 2024.emnlp-main.64/. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126–1135. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/finn17a.html. Zitian Gao, Lynx Chen, Yihao Xiao, He Xing, Ran Tao, Haoming Luo, Joey Zhou, and Bryan Dai. Universal reasoning model, 2025. URL https://arxiv.org/abs/2512.14693. Saumya Gupta, Scott Biggs, Moritz Laber, Zohair Shafi, Robin Walters, and Ayan Paul. Deepweightflow: Re-basined flow matching for generating neural network weights. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=fOwsr1VTi8. David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rkpACe1lx. Xiaolong Han, Zehong Wang, Bo Zhao, Binchi Zhang, Jundong Li, Damian Borth, Rose Yu, Haggai Maron, Yanfang Ye, Lu Yin, and Ferrante Neri. A survey of weight space learning: Understanding, representation, and generation, 2026. URL https://arxiv.org/abs/2603.10090. Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-Learning in Neural Networks: A Survey . IEEE Transactions on Pattern Analysis & Machine Intelligence, 44(09):5149–5169, September 2022. ISSN 1939-3539. doi: 10.1109/TPAMI.2021.3079209. URLhttps://doi.ieeecomputersociety. org/10.1109/TPAMI.2021.3079209. 11 Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 8003–8017, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.507. URL https://aclanthology.org/2023.findings-acl.507/. Borja Ibarz, Vitaly Kurin, George Papamakarios, Kyriacos Nikiforou, Mehdi Bennani, Róbert Csordás, An- drew Joseph Dudzik, Matko Bošnjak, Alex Vitvitskyi, Yulia Rubanova, Andreea Deac, Beatrice Bevilacqua, Yaroslav Ganin, Charles Blundell, and Petar Veli ˇ ckovi ́ c. A generalist neural algorithmic learner. In The First Learning on Graphs Conference, 2022. URL https://openreview.net/forum?id=FebadKZf6Gd. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=Rwhi91ideu. Xiaolong Jin, Kai Wang, Dongwen Tang, Wangbo Zhao, Yukun Zhou, Junshu Tang, and Yang You. Conditional lora parameter generation, 2024. URL https://arxiv.org/abs/2408.01415. Alexia Jolicoeur-Martineau.Less is more: Recursive reasoning with tiny networks.arXiv preprint arXiv:2510.04871, 2025. Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URLhttps://kellerjordan.github. io/posts/muon/. Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 565–576, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.acl-long.47. URL https://aclanthology.org/2021.acl-long.47/. Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn, Hao Tang, Wei-Long Zheng, Yewen Pu, and Kevin Ellis. Combining induction and transduction for abstract reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps: //openreview.net/forum?id=UmdotAAVDe. Zhiyuan Liang, Dongwen Tang, Yuhao Zhou, Xuanlei Zhao, Mingjia Shi, Wangbo Zhao, Zekai Li, Peihao Wang, Konstantin Schürholt, Damian Borth, Michael M. Bronstein, Yang You, Zhangyang Wang, and Kai Wang. Drag-and-drop LLMs: Zero-shot prompt-to-weights. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=fTkBZLxBzV. Isaac Liao and Albert Gu. Arc-agi without pretraining, 2025. URL https://arxiv.org/abs/2512.06104. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. Matthew Macfarlane and Clément Bonnet. Searching latent program spaces. In ICML 2025 Workshop on Programmatic Representations for Agent Learning, 2025. URLhttps://openreview.net/forum?id= Va937YZ9it. Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In Hal Daumé I and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6682– 6691. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/malach20a.html. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.759. URLhttps://aclanthology. org/2022.emnlp-main.759/. 12 Sajad Movahedi, Vera Milovanovi ́ c, Shlomo Libo Feigin, Alexander Theus, Thomas Hofmann, Valentina Boeva, T. Konstantin Rusch, and Antonio Orvieto. Fixed-point reasoners: Stable and adaptive deep looped transform- ers. In ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning, 2026. URL https://openreview.net/forum?id=EpkmPWS4Tm. Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2554–2563. PMLR, 06–11 Aug 2017. URLhttps://proceedings.mlr.press/v70/ munkhdalai17a.html. Elvis Nava, Seijin Kobayashi, Yifei Yin, Robert K. Katzschmann, and Benjamin F Grewe. Meta-learning via classifier(-free) guidance. In Sixth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=fY5xpJszW2-. Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018. URL https://arxiv.org/abs/1803.02999. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru WANG, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. ToolRL: Reward is all tool learning needs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=eOLdGbXT6t. Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6527– 6536, Los Alamitos, CA, USA, June 2024. IEEE Computer Society. doi: 10.1109/CVPR52733.2024.00624. URL https://doi.ieeecomputersociety.org/10.1109/CVPR52733.2024.00624. Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=BJgklhAcK7. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach them- selves to use tools. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, ed- itors, Advances in Neural Information Processing Systems, volume 36, pages 68539–68551. Curran Associates, Inc., 2023. URLhttps://proceedings.neurips.c/paper_files/paper/2023/file/ d842425e4bf79ba039352da0f658a906-Paper-Conference.pdf. Konstantin Schürholt. The hidden drivers of HRM’s performance on ARC-AGI, 2025. URLhttps:// arcprize.org/blog/hrm-analysis. Amin Sghaier, Ali Parviz, and Alexia Jolicoeur-Martineau. Probabilistic tiny recursive model, 2026. URL https://arxiv.org/abs/2605.19943. Haruki Shirakami and James Henderson. Compositional generalization through gradient search in nonparametric latent space. In The Fourteenth International Conference on Learning Representations, 2026. URLhttps: //openreview.net/forum?id=RNTWTJe4x6. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. ISSN 0925-2312. doi: https: //doi.org/10.1016/j.neucom.2023.127063. URLhttps://w.sciencedirect.com/science/article/ pii/S0925231223011864. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from bert into simple neural networks, 2019. URL https://arxiv.org/abs/1903.12136. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URLhttps://proceedings.neurips.c/paper_files/paper/2017/ file/3f5e243547dee91fbd053c1c4a845a-Paper.pdf. Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model, 2025a. URL https://arxiv.org/abs/2506.21734. Kai Wang, Dongwen Tang, Wangbo Zhao, Konstantin Schürholt, Zhangyang Wang, and Yang You. Scaling up parameter generation: A recurrent diffusion approach. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URL https://openreview.net/forum?id=65llKR17s4. 13 Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distil- lation for task-agnostic compression of pre-trained transformers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5776–5788. Curran Associates, Inc., 2020. URLhttps://proceedings.neurips.c/paper_files/ paper/2020/file/3f5e243547dee91fbd053c1c4a845a-Paper.pdf. Zhangyang Wang, Peihao Wang, and Kai Wang. Position: Weight space should be a first-class generative AI modality. In Forty-third International Conference on Machine Learning Position Paper Track, 2026. URL https://openreview.net/forum?id=vu0ousWLde. Mixue Xie, Shuang Li, Binhui Xie, Chi Harold Liu, Jian Liang, Zixun Sun, Ke Feng, and Chengwei Zhu. Weight diffusion for future: Learn to generalize in non-stationary environments. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum? id=2cFUYnNL1m. Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias Boutros Khalil. LLMs and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URLhttps://openreview.net/forum?id= E8m8oySvPJ. Boya Zeng, Yida Yin, Zhiqiu Xu, and Zhuang Liu. Generative modeling of weights: Generalization or memorization? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026. Dominic Zhao, Seijin Kobayashi, João Sacramento, and Johannes von Oswald. Meta-Learning via Hypernet- works. IEEE, Dec 2020. Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov. HyperTransformer: Model generation for supervised and semi-supervised few-shot learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27075–27098. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/zhmoginov22a.html. 14 A Training Details Training.All experiments are implemented in PyTorch Lightning and trained on a single NVIDIA A40 GPU for 8,000 steps, using cosine learning-rate decay with linear warmup. We use Muon [Jordan et al., 2024, Liu et al., 2025] for hidden 2-D linear weights and AdamW [Loshchilov and Hutter, 2019] for embeddings, input/output layers, biases, normalisation, and convolution parameters, with optimisation hyperparameters given in Table 2. Muon uses momentum 0.95 with Nesterov momentum and Newton–Schulz orthogonalisation. Directly trained models use batches of 256 input–output pairs in FP32 with TF32 matrix multiplications, while hypernetworks use batches of 512 episodes with BF16 mixed precision. All Transformers use dropout 0.1, and validation is performed once per epoch. We train five seeds per configuration and report mean±standard error. Models selected by validation use the checkpoint with the lowest validation loss. Table 2: Optimisation hyperparameters for directly trained models and hypernetworks. Direct modelsHypernetworks Muon learning rate0.0050.005 Muon momentum0.950.95 AdamW learning rate0.00050.001 Weight decay0.010.01 Warmup steps200800 Cosine minimum LR010 −4 Gradient clippingNone10.0 Model comparison. We evaluate the three-layer RoPE [Su et al., 2024] Canon Transformer [Allen-Zhu, 2025, Vaswani et al., 2017] at hidden dimensions4,6, and10, corresponding to approximately 1.4K, 2.4K, and 5.4K parameters. Individual models are trained separately for each transformation using five seeds per model size. Hypernetworks are trained jointly across all 14 transformations to generate the same target architectures, either from the support examples alone or with a frozen task embedding, with results averaged over five seeds. All experiments use the augmented ARC-1D dataset described in Section 4.1. Canon layers use SiLU activations, while MLP blocks and the hypernetwork decoder use GELU. Table 3: Model configurations. Direct models are the three-layer RoPE Canon Transformer at four widths. The hypernetwork generates every weight of the target except the token embedder, which it shares with its encoder. Only the dimension-4 hypernetwork is trained; grey values marked with †show the corresponding parameter counts for wider targets, where only the decoder output layer grows. Hidden dimension461014 Direct model Layers / heads3 / 13 / 13 / 13 / 1 Token embedding dim.10101010 Canon kernel5555 Parameters1,3982,4445,4009,508 Hypernetwork Token embedding dim.4444 Encoder dim. / layers4 / 14 / 14 / 14 / 1 Decoder bottleneck (d ′ )8888 Generated target weights1,2042,238 † 5,170 † 9,254 † Trainable parameters10,15618,428 † 41,884 † 74,556 † 15 Compute.All runs use a single NVIDIA A40 GPU (48 GB). Table 4 reports the number of runs, wall-clock time per run including data setup and evaluation, and total GPU-hours for each experiment. Directly trained models take approximately 3–9 minutes per run and hypernetworks approximately 24 minutes for 8,000 training steps, although the data-efficiency experiments include substantially longer runs. In total, the experiments reported in this paper comprise 2,000 runs and approximately 460 GPU-hours. Table 4: Compute per experiment. Wall-clock time per run and total compute on a single NVIDIA A40 GPU unless otherwise indicated. Times include data setup and evaluation. RunsMin./runGPU-h 1Capacity and ablations9603–964 2Hypernetwork, in distribution10244 3Weight reuse (evaluation only)100.4 < 1 4Compositional generalisation (CPU evaluation)100.70 5Leave-one-out1402354 6Data efficiency8707–109340 Total2,000461 16 B Architecture & Optimiser Ablations Task-Identity Conditioning. We ask where task identity should enter the hypernetwork and whether its representation should be learned. Our main model adds a frozen random projection of the one-hot task ID to the pooled task representationz. We compare this latent conditioning with adding the task embedding to every support token before the encoder, using either frozen random or learned embeddings, alongside a model without task identity. Training follows Section 4.3, while generalisation follows Section 4.4: held-out categories receive a zero task vector, and unseen compositions receive the sum of their constituent task vectors. Table 5: Task-ID conditioning improves in-distribution specialisation but hinders generalisation. Results across task-ID placements and embedding types, reported as mean±s.d. over 5 seeds. Parentheses show the change relative to no task-ID conditioning. In-DistributionGeneralisation Exact Match↑UnseenCompositional Token Accuracy↑Exact Match↑Token Accuracy↑ None57.4± 5.7 81.6± 0.9 10.2± 3.6 75.5± 1.2 Latent conditioning Frozen93.0± 2.6 (+35.6)79.1± 1.6 (-2.5)0.6± 0.5 (-9.6)63.2± 8.3 (-12.3) Learned 96.5± 0.8 (+39.1)77.3± 2.9 (-4.3)1.1± 1.9 (-9.1)66.4± 4.8 (-9.1) Input conditioning Frozen95.1± 2.2 (+37.7)74.1± 4.2 (-7.5)0.0± 0.0 (-10.2)61.0± 3.0 (-14.5) Learned96.2± 1.3 (+38.8)73.1± 6.2 (-8.5)0.8± 0.8 (-9.4)68.8± 2.9 (-6.7) All task-ID variants substantially improve in-distribution performance, but generalise less well than the model without task identity (Table 5). Learning the task embedding provides little additional benefit over a frozen random embedding. Placement has a clearer effect: latent conditioning generally retains better generalisation than conditioning the encoder input, while preserving similarly strong in-distribution performance. The repre- sentations in Figure 9 are consistent with this difference: latent conditioning produces a clearer task-aligned organisation, whereas input conditioning yields a coarser structure. Overall, explicit task identity improves specialisation to known transformations, but neither its placement nor learning the embedding recovers the generalisation obtained without task-ID conditioning. w/o Task IDFrozen, latentLearned, latentFrozen, inputLearned, input Denoise Denoise MC Fill Flip Hollow Mirror Move 1 Move 2 Move 2 Towards Move 3 Move Dynamic Pattern Copy Pattern Copy MC Scaling Figure 9: Task representations depend on task-ID placement. t-SNE of the task representationz for validation examples from each transformation (seed 1; axes are arbitrary). Latent conditioning produces clearer task-aligned structure, while input conditioning forms coarser groups. Canon ablation. We ablate the Canon layers across all Section 4.2 configurations, covering individual and joint models with and without task identity at all four model sizes, using five seeds per configuration. Canon improves test exact-match accuracy across all settings, with the largest gains at low capacity (Figure 10). These results indicate that the Canon layers are particularly important when capacity is limited or shared across transformations. 17 1.4K2.4K5.4K9.5K Model parameters (log scale) 0% 20% 40% 60% 80% 100% Test exact match accuracy Individual models (canon) Individual models (no canon) Joint model, Task ID (canon) Joint model, Task ID (no canon) Joint model, w/o Task ID (canon) Joint model, w/o Task ID (no canon) Figure 10: Canon layers improve performance, particularly at low capacity. Test exact-match accuracy with and without Canon layers for individual and joint models across model sizes. The gain diminishes for larger individual models but remains substantial for jointly trained models. Bands show±1 s.d. across 5 seeds. Optimiser ablation. We replace Muon with AdamW in every configuration of Section 4.2, using five seeds per configuration. Muon uses a learning rate of 0.005 with Nesterov momentum 0.95 and an auxiliary AdamW group (learning rate 0.0005,β = (0.9, 0.95)) for embeddings, the output layer, normalisation, and Canon parameters. AdamW instead optimises all parameters with a learning rate of 0.001 andβ = (0.9, 0.999). Both use weight decay 0.01 and the same warmup and cosine schedule. Muon matches or exceeds AdamW in every setting (Figure 11). Note: the AdamW learning rate was not independently tuned, this ablation does not isolate optimiser choice from hyperparameter selection. We look at this a bit more closely for the hypernetwork architectures. 1.4K2.4K5.4K9.5K Model parameters (log scale) 0% 20% 40% 60% 80% 100% Test exact match accuracy Individual models (Muon) Individual models (AdamW) Joint model, Task ID (Muon) Joint model, Task ID (AdamW) Joint model, w/o Task ID (Muon) Joint model, w/o Task ID (AdamW) Figure 11: Muon improves marginally over AdamW Test exact-match accuracy with Muon (solid) and AdamW (dashed) across model sizes. Bands show±1 s.d. across 5 seeds. Hyperparameter search for hypernetworks. We tried tuning both optimisers on the hypernetwork without task-identity conditioning, using three seeds per configuration. For Muon, we search the Muon learning rate (0.002, 0.005, 0.01, 0.02, 0.04), auxiliary AdamW learning rate (10 −4 , 3×10 −4 , 5×10 −4 , 10 −3 ), and weight decay (0, 0.01, 0.1), giving 60 configurations. We fix momentum to 0.95 with Nesterov momentum, use five Newton–Schulz steps, and set the auxiliary AdamW betas to(0.9, 0.95). For AdamW, we search the learning rate (10 −4 , 3×10 −4 , 10 −3 , 3×10 −3 , 10 −2 ) and weight decay (0, 0.01, 0.1), giving 15 configurations, with betas(0.9, 0.95). All configurations are trained for 8,000 steps with 800 warmup steps and cosine decay to zero, and are selected by validation loss. Muon is relatively insensitive to the searched hyperparameters: its best configurations achieve the highest test exact-match accuracy, reaching66.7± 11.0%, but with high variation across seeds. AdamW achieves a lower best validation loss and a test exact-match accuracy of 60.0± 0.9%. 18 C Task Performance Individual vs Joint Move 1Move 2 Move 2 Towards Move 3 Move Dynamic Scaling Fill Hollow Flip Mirror Denoise Denoise MC Pattern Copy Pattern Copy MC 0% 20% 40% 60% 80% 100% Validation exact match accuracy Individual modelsJoint model, Task IDJoint model, w/o Task ID Figure 12: The joint-training penalty is concentrated in a few transformations. Per-transformation exact-match validation accuracy at 1.4K parameters. Error bars show±1 s.d. across 5 seeds. Move 1Move 2 Move 2 Towards Move 3 Move Dynamic Scaling Fill Hollow Flip Mirror Denoise Denoise MC Pattern Copy Pattern Copy MC 0% 20% 40% 60% 80% 100% Validation exact match accuracy Individual modelsJoint model, Task IDJoint model, w/o Task ID Figure 13: The joint-training penalty is concentrated in a few transformations. Per-transformation exact-match validation accuracy at 2.4K parameters. Error bars show±1 s.d. across 5 seeds. Move 1Move 2 Move 2 Towards Move 3 Move Dynamic Scaling Fill Hollow Flip Mirror Denoise Denoise MC Pattern Copy Pattern Copy MC 0% 20% 40% 60% 80% 100% Validation exact match accuracy Individual modelsJoint model, Task IDJoint model, w/o Task ID Figure 14: The joint-training penalty is concentrated in a few transformations. Per-transformation exact-match validation accuracy at 5.4K parameters. Error bars show±1 s.d. across 5 seeds. Move 1Move 2 Move 2 Towards Move 3 Move Dynamic Scaling Fill Hollow Flip Mirror Denoise Denoise MC Pattern Copy Pattern Copy MC 0% 20% 40% 60% 80% 100% Validation exact match accuracy Individual modelsJoint model, Task IDJoint model, w/o Task ID Figure 15: The joint-training penalty is concentrated in a few transformations. Per-transformation exact-match validation accuracy at 9.5K parameters. Error bars show±1 s.d. across 5 seeds. 19 D Data Efficiency We ask whether sharing across transformations improves data efficiency. ARC-1D provides 50 episodes per transformation, of which 40 are used for training, and we increase the training set through augmentation. We compare individual models, hypernetworks, and joint models, with and without task-ID conditioning (Figure 16). Individual models and task-ID-conditioned hypernetworks follow similar data-efficiency curves, showing no material benefit from hypernetwork weight sharing at this scale. The task-ID-conditioned joint model performs better with less data, but stores all transformations directly in its 10K parameters rather than generating decoupled 1.4K-parameter specialists. Without task IDs, the joint model and hypernetwork perform similarly across training- set sizes. Across models, augmentation provides diminishing gains beyond roughly 100–200 training episodes per transformation. Similarly, Schürholt [2025] find that small HRMs on ARC-AGI-1 benefit from augmentation but require substantially fewer than 1K augmentations. We retain 1K augmentations in our main experiments for consistency. 005201000 Augmentations 1104020080040K Training episodes (log scale) 0% 20% 40% 60% 80% 100% Test exact match accuracy Individual models Hypernetwork, Task ID Hypernetwork, w/o Task ID Joint model, Task ID Joint model, w/o Task ID Figure 16: Data efficiency across models. Individual models and task-ID-conditioned hypernetworks require similar amounts of training data, while the larger task-ID-conditioned joint model learns with fewer episodes. Without task IDs, joint models and hypernetworks follow similar curves. Bands show±1 s.d. across 5 seeds. 20 E Generalisation Details Table 6: Performance on held-out compositions. Each composition combines two transformations observed independently during training, while the composition itself is held out.B ◦ Adenotes applyingAfirst, followed byB. Token accuracy is macro-averaged over the 10 compositions and 5 seeds; exact match is pooled over all 2,000 held-out instances. Task IDw/o Task ID CompositionToken Acc.EMToken Acc.EM Mirror◦ Denoise MC47.70.085.6 (+37.9)8.0 Copy◦ Denoise MC69.60.067.3 (-2.3)0.0 Denoise 1C◦ Denoise MC76.70.068.3 (-8.3)0.5 Mirror◦ Fill61.30.073.1 (+11.7)1.5 Shift 3◦ Denoise 1C75.50.577.0 (+1.5)0.0 Shift 3◦ Hollow78.20.085.2 (+7.0)0.0 Hollow◦ Move Dynamic79.10.082.4 (+3.2)0.0 Shift 3◦ Fill68.80.073.9 (+5.1)0.0 Move Dynamic◦ Fill62.70.071.6 (+9.0)0.0 Copy◦ Shift 357.20.070.3 (+13.1)0.0 Overall67.70.0575.5 (+7.8)1.00 Table 7: Performance on held-out task categories. Each row holds out one of the 14 base transformations entirely during training and evaluates it zero-shot. Token accuracy and exact match are each the mean across 5 seeds for that category’s own held-out score. Task IDw/o Task ID CategoryToken Acc.EMToken Acc.EM Denoise Multicolor47.90.0081.1 (+33.2)4.00 Scaling57.10.0086.9 (+29.8)0.00 Pattern Copy Multicolor57.60.0084.7 (+27.2)0.00 Move 3 Pixels68.40.0091.8 (+23.4)0.00 Pattern Copy77.30.0099.5 (+22.2)96.00 Move 2 Pixels Towards74.90.0095.5 (+20.6)20.00 Mirror59.00.0068.1 (+9.1)0.00 Move Dynamic77.04.0086.1 (+9.1)8.00 Move 1 Pixel84.00.0092.0 (+8.0)0.00 Move 2 Pixels84.60.0091.2 (+6.5)0.00 Fill62.80.0069.2 (+6.4)0.00 Flip78.20.0083.6 (+5.5)0.00 Hollow66.10.0069.9 (+3.9)0.00 Denoise64.60.0046.9 (-17.7)0.00 Overall68.50.2981.9 (+13.4)9.14 21 F Task structured Weight-Space −1.5−1.0−0.50.00.5 dim 1 −1.0 −0.5 0.0 0.5 dim 2 Task ID −3−2−10123 dim 1 −3 −2 −1 0 1 2 dim 2 w/o Task ID Denoise Denoise MC Fill Flip Hollow Mirror Move 1 Move 2 Move 2 Towards Move 3 Move Dynamic Pattern Copy Pattern Copy MC Scaling Figure 17: Generated parameters form a task-structured weight space. PCA projection of generated target-model parameters, coloured by transformation. Left: With a frozen task embedding, tasks are fully separable (100% linear probe accuracy). Right: Without task identity, structure emerges from the support examples alone (60.9% linear probe accuracy). 2.55.07.510.012.515.0 dim 1 0 2 4 6 8 10 12 dim 2 Task ID −10−505 dim 1 −5.0 −2.5 0.0 2.5 5.0 7.5 10.0 dim 2 w/o Task ID Denoise Denoise MC Fill Flip Hollow Mirror Move 1 Move 2 Move 2 Towards Move 3 Move Dynamic Pattern Copy Pattern Copy MC Scaling Figure 18: Generated parameters form a task-structured weight space. UMAP projection of generated target-model parameters, coloured by transformation. Left: With a frozen task embedding, tasks are fully separable (100% linear probe accuracy). Right: Without task identity, structure emerges from the support examples alone (60.9% linear probe accuracy). 22 G Task Examples x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????? ̂ y 0 Move 1 x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????? ̂ y 0 Move 2 x 1 y 1 x 2 y 2 x 3 y 3 x 0 ??????? ̂ y 0 Move 2 Towards x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????? ̂ y 0 Move 3 x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????? ̂ y 0 Move Dynamic x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????? ̂ y 0 Scaling Figure 19: ARC-1D movement and scaling tasks. Examples of the movement and scaling task families, including one-point (1P), two-point (2P), two-point dynamic-position (2P-DP), three-point (3P), dynamic-position (DP), and scaling variants. 23 x 1 y 1 x 2 y 2 x 3 y 3 x 0 ???????? ̂ y 0 Fill x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????? ̂ y 0 Hollow x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????????? ̂ y 0 Flip x 1 y 1 x 2 y 2 x 3 y 3 x 0 ?????????????? ̂ y 0 Mirror Figure 20: ARC-1D object-transformation tasks. Examples of the fill, hollow, flip, and mirror task families. 24 x 1 y 1 x 2 y 2 x 3 y 3 x 0 ???????????????????????????????? ̂ y 0 Denoise x 1 y 1 x 2 y 2 x 3 y 3 x 0 ???????????????????????????????? ̂ y 0 Denoise MC x 1 y 1 x 2 y 2 x 3 y 3 x 0 ???????????????????????????????? ̂ y 0 Pattern Copy x 1 y 1 x 2 y 2 x 3 y 3 x 0 ???????????????????????????????? ̂ y 0 Pattern Copy MC Figure 21: ARC-1D denoising and pattern-copy tasks. Examples of the denoising and pattern-copy task families, shown for both the single-color (1C) and multicolor (MC) variants. 25 H Compositional Generalisation Examples x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????????????????????? ̂ y 0 Denoise MC ∘ Mirror x 1 y 1 x 2 y 2 x 3 y 3 x 0 ????????????????????????????????? ̂ y 0 Fill ∘ Mirror x 1 y 1 x 2 y 2 x 3 y 3 x 0 ??????????????????????????? ̂ y 0 Fill ∘ Shift 3 x 1 y 1 x 2 y 2 x 3 y 3 x 0 ??????????????????????????????? ̂ y 0 Hollow ∘ Shift 3 Figure 22: ARC-1D compositional task examples. Examples of held-out compositions formed by combining two transformations observed independently during training. 26 x 1 y 1 x 2 y 2 x 3 y 3 x 0 ?????????????????????? ̂ y 0 Denoise MC ∘ Pattern Copy x 1 y 1 x 2 y 2 x 3 y 3 x 0 ?????????????????? ̂ y 0 Shift 3 ∘ Pattern Copy x 1 y 1 x 2 y 2 x 3 y 3 x 0 ?????????????????????? ̂ y 0 Denoise MC ∘ Denoise x 1 y 1 x 2 y 2 x 3 y 3 x 0 ??????????????????????????? ̂ y 0 Fill ∘ Move Dynamic x 1 y 1 x 2 y 2 x 3 y 3 x 0 ??????????????????????????????? ̂ y 0 Move Dynamic ∘ Hollow Figure 23: ARC-1D compositional task examples. Further examples of held-out compositions formed from transformations observed independently during training. 27