Paper deep dive
One Model, Many Minds: Unlocking Multi-Agent Synergy in a Single Agent via Mixture of Roles
Zhichen Zeng, Huiyuan Chen, Jingru Cheng, Juan Zha, Ming Liu, Ying Chen, Xiyuan Yang, Chaosheng Dong, Haiyang Zhang, Hanghang Tong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/28/2026, 2:55:56 AM
Summary
The paper introduces Mixture of Roles (MoRe), a framework that enables single Large Language Models (LLMs) to achieve dynamic, multi-perspective specialization within a single-turn inference. MoRe utilizes a learnable codebook of steering vectors representing latent roles and a query-aware router to fuse these vectors into a composite steering vector. This approach outperforms single-agent baselines by 2.2% on average and matches the performance of Multi-Agent Systems (MAS) while reducing token costs by 20x. Training involves a three-stage Supervised Fine-Tuning (SFT) curriculum and Group Relative Policy Optimization (GRPO) post-training, keeping the backbone LLM frozen.
Entities (12)
Relation Signals (9)
MoRe → reducescostby → 20x
confidence 95% · reducing token cost by 20x.
MoRe → uses → Codebook
confidence 95% · MoRe learns a diversified codebook of steering vectors, each of which encodes a latent role.
MoRe → uses → Router
confidence 95% · A query-aware router dynamically fuses the codebook into a steering vector that encompasses multiple roles.
MoRe → outperforms → Single-agent baselines
confidence 92% · MoRe outperforms single-agent baselines by 2.2% on average
MoRe → matchesperformanceof → Multi-Agent Systems
confidence 90% · achieves performance on par with MAS while reducing token cost by 20x.
MoRe → trainswith → GRPO
confidence 90% · ...and GRPO post-training, while the backbone LLM remains frozen.
MoRe → trainswith → SFT
confidence 90% · The proposed MoRe can be efficiently trained via a three-stage SFT curriculum...
MoRe → evaluatedon → Llama-3.1-8B-Instruct
confidence 85% · We consider two LLMs: Llama-3.1-8B-Instruct... Experiments... show that MoRe outperforms...
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Specializing Large Language Models (LLMs) toward distinct abilities underpins successes ranging from personalized assistants to multi-agent systems (MAS). Single-agent paradigms rely on pre-defined personas or steering vectors to induce specialization, yet they impose a single fixed specialization that fails to adapt to diverse queries. Conversely, MAS achieves dynamic multi-perspective problem solving by orchestrating agents with distinct text-based roles, but fusing these specializations requires multi-turn interactions that inflate context length and inference cost. To address these limitations, we propose Mixture of Roles (MoRe), which adaptively composes multiple specializations into a single steering vector for single-turn inference. Specifically, MoRe learns a diversified codeboox of steering vectors, each of which encodes a latent role. A query-aware router dynamically fuses the codebook into a steering vector that encompasses multiple roles. By steering the backbone LLM with the composed vector, MoRe enables multi-perspective specialization in a single-agent, single-turn inference process. The proposed MoRe can be efficiently trained via a three-stage SFT curriculum and GRPO post-training, while the backbone LLM remains frozen. Experiments across reasoning and personality benchmarks show that MoRe outperforms single-agent baselines by 2.2% on average, and achieves performance on par with MAS while reducing token cost by 20x.
Tags
Links
- Source: https://arxiv.org/abs/2608.27338v1
- Canonical: https://arxiv.org/abs/2608.27338v1
Trouble viewing inline? Open PDF directly →
Full Text
75,956 characters extracted from source content.
Expand or collapse full text
One Model, Many Minds: Unlocking Multi-Agent Synergy in a Single Agent via Mixture of Roles Zhichen Zeng Affiliation: University of Illinois Urbana-Champaign Email: zhichenz@illinois.edu Huiyuan Chen Affiliation: Amazon Jingru Cheng Affiliation: Amazon Juan Zha Affiliation: Amazon Ming Liu Affiliation: Amazon Ying Chen Affiliation: Amazon Xiyuan Yang Affiliation: University of Illinois Urbana-Champaign Chaosheng Dong Affiliation: Amazon Haiyang Zhang Affiliation: Amazon Hanghang Tong Affiliation: University of Illinois Urbana-Champaign Abstract Specializing Large Language Models (LLMs) toward distinct abilities underpins successes ranging from personalized assistants to multi-agent systems (MAS). Single-agent paradigms rely on pre-defined personas or steering vectors to induce specialization, yet they impose a single fixed specialization that fails to adapt to diverse queries. Conversely, MAS achieves dynamic multi-perspective problem solving by orchestrating agents with distinct text-based roles, but fusing these specializations requires multi-turn interactions that inflate context length and inference cost. To address these limitations, we propose Mixture of Roles (MoRe), which adaptively composes multiple specializations into a single steering vector for single-turn inference. Specifically, MoRe learns a diversified codebook of steering vectors, each of which encodes a latent role. A query-aware router dynamically fuses the codebook into a steering vector that encompasses multiple roles. By steering the backbone LLM with the composed vector, MoRe enables multi-perspective specialization in a single-agent, single-turn inference process. The proposed MoRe can be efficiently trained via a three-stage SFT curriculum and GRPO post-training, while the backbone LLM remains frozen. Experiments across reasoning and personality benchmarks show that MoRe outperforms single-agent baselines by 2.2% on average, and achieves performance on par with MAS while reducing token cost by 20×20×. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language tasks Achiam et al. (2023); Grattafiori et al. (2024); Comanici et al. (2025); Guo et al. (2025). Beyond general-purpose competence, recent studies show that LLMs benefit substantially from being specialized toward distinct behaviors, cognitive modes, or expert roles Chen et al. (2025b); Wei et al. (2026c); Sun et al. (2025). Conditioning a model on a specialization elicits targeted expertise and reasoning styles that a generic prompt fails to evoke, powering applications from role-conditioned reasoning to personalized assistance. The single-agent paradigm induces LLM specialization primarily through explicit role-playing Chen et al. (2024a); Shanahan et al. (2023); Shao et al. (2023) or implicit activation steering Turner et al. (2024); Li et al. (2023b). Specifically, role-playing assigns textual role descriptions, whereas activation steering directly injects a pre-computed steering vector into the hidden layers to elicit targeted behaviors. While computationally efficient, both approaches rely on pre-defined specializations, i.e., textual personas or pre-computed steering vectors, that enforce a single static perspective Wei et al. (2026c); Sun et al. (2025). Consequently, the single-agent paradigms lack the capacity for query-specific adaptivity or multi-perspective reasoning. In contrast, multi-agent systems (MAS) dynamically integrate multi-perspective specialization via multi-turn interactions Liang et al. (2024); Du et al. (2023); Liu et al. (2024c); Wu et al. (2023). By instantiating agents with distinct roles via explicit text-based prompts across agents, agents exchange and refine their perspectives over multi-turn interactions. Yet this superiority comes at a steep price: MAS involve multiple agents and multi-turn interactions that induces heavy computation and long context Kim et al. (2025); Tran and Kiela (2026). Figure 1: Comparison between single agent, MAS and MoRe. MoRe achieves dynamic multi-perspective specialization within single-turn inference, outperforming single-agent methods while achieving higher efficiency than MAS. The two paradigms thus present a dilemma between computational cost and reasoning capability. As the example shown in Figure 1, we are required to solve a complex task that requires mathematical derivation, algorithmic coding, and rigorous testing. A single agent with the fixed mathematician specialization may fall short on the remaining capabilities. Conversely, MAS allows agents with different specializations to collaborate, but the collaboration pipeline requires multi-turn inference with high computation. Therefore, we ask the following research question: Can a single agent acquire dynamic multi-perspective specialization within single-turn inference? To answer this question, we introduce Mixture of Roles (MoRe) to dynamically compose multiple roles into a single steering vector for single-turn inference. To move beyond static steering, MoRe parameterizes steering vectors into a learnable codebook, where each entry corresponds to a candidate specialization optimized end-to-end. A lightweight router selects and combines these candidates into a query-specific composed steering vector. By steering with the composed vector, we equip the single agent with a dynamic multi-perspective specialization, eliminating the textual communication overhead of MAS entirely. Our proposed MoRe is a lightweight module that can be efficiently trained via a three-stage SFT and GRPO post-training with the backbone LLM completely frozen. Our contributions are summarized as follows: • Framework. We enable dynamic multi-perspective specialization within single-agent inference by replacing multi-turn textual coordination with activation-space composition. • Method. We propose a lightweight module named MoRe to generate query-specific steering vectors with minimal computational overhead. MoRe is efficiently trained via curriculum SFT followed by GRPO post-training with the backbone LLM entirely frozen. • Evaluation. Extensive experiments show that MoRe outperforms single-agent specialization baselines by 2.2% on average and achieves performance competitive with MAS baselines while reducing the cost cost by 20×20×. 2 Related Works 2.1 Specialized Agent Specializing LLMs toward targeted behaviors, domain expertise, or cognitive modes has become a foundational paradigm for solving complex tasks. Early approaches predominantly rely on textual prompting Shanahan et al. (2023); Shao et al. (2023); Chen et al. (2024a), prepending hand-crafted role descriptions to condition the generation trajectory. Recent works have shifted from explicit prompt engineering to implicit representation engineering, where activation steering has emerged as an efficient alternative, directly modulating the model’s hidden activations at inference time. Foundational works Rimsky et al. (2024); Turner et al. (2024); Li et al. (2023b) derive steering vectors from contrastive activations to guide the residual stream toward desired traits. This paradigm has been extended to persona control Chen et al. (2025b), with recent efforts targeting persona-related neurons Sun et al. (2025); Wei et al. (2026c) or attention heads Izawa et al. (2026). More recent studies attempt to refine vector construction via preference learning Cao et al. (2024) or select predefined vectors based on situational contexts Wei et al. (2026c). However, these methods share common limitations: either the specialization is pre-defined, or only a single fixed role is applied uniformly. In contrast, MoRe learns a codebook of steering vector and composes them per query via a lightweight router, synthesizing hybrid specializations that no single fixed vector can express. 2.2 Multi-Agent System Multi-agent system (MAS) tackle complex tasks by orchestrating multiple LLM agents with distinct, prompt-defined roles. CAMEL Li et al. (2023a) pioneered role-playing between communicative agents, and AutoGen Wu et al. (2023) provides a general conversational programming framework for composing agents. AgentVerse Chen et al. (2024b) dynamically recruits collaborator agents and studies emergent group behaviors, and DyLAN Liu et al. (2024c) adaptively selects agent teams and interaction structures on the fly. Multi-agent debate frameworks Du et al. (2023); Liang et al. (2024) improve factuality and reasoning by letting agents critique and revise others answers. Across these designs, the gains stem from fusing multiple specialized perspectives, yet the fusion is realized via expensive multi-turn text communication, where inference cost grows with the number of agents and rounds. Besides, the inflated context may degrade reasoning quality Liu et al. (2024a). In contrast, MoRe pursues the same multi-perspective principle but in a single-turn inference: specializations live as steering directions and are fused in activation space by a query-aware router. 3 Methodology 3.1 Problem Formulation Consider a frozen LLM with L layers and hidden dimension d. Given a query Q=(q1,…,qT)Q=(q_1,…,q_T), where qiq_i is the i-th token, LLM produces hidden states t(ℓ)∈ℝd h^( )_t ^d for token position t at layer ℓ . Activation steering intervenes on the residual stream of an intermediate layer ℓ⋆ by adding a steering vector ∈ℝd v ^d Rimsky et al. (2024): ~t(ℓ⋆)=t(ℓ⋆)+α‖t(ℓ⋆)‖2‖2, h^( )_t= h^( )_t+α \| h^( )_t\|_2\| v\|_2 v, (1) where α is the steering strength. To construct the steering vector, prior works Rimsky et al. (2024); Turner et al. (2024) utilize contrastive pairs that do and do not exhibit the target behavior, e.g., persona-conditioned vs. plain queries, and set the steering vector as the mean difference of their activations, that is =Q[¯(ℓ⋆)(p+⊕Q)−¯(ℓ⋆)(p−⊕Q)] v=E_Q [ h^( )(p_+ Q)- h^( )(p_- Q) ] (2) where p+,p−p_+,p_- denote positive and negative prompts, respectively. ¯(ℓ⋆)(⋅) h^( )(·) is the hidden state of layer-ℓ⋆ mean-pooled over answer tokens. The expectation is taken over all queries Q. However, the generated vector is heuristically pre-defined and uniformly applied regardless of the input query. To overcome these drawbacks, we propose shifting from static single steering to dynamic multi-perspective composition. We aim to learn a composer ϕC_φ, which maps an input query Q to a dynamic, query-specific steering vector Q v_Q via mixture of roles, that is, ϕ:Q↦Q=∑n=1Nwn(Q)⋅n,Cφ:Q v_Q= _n=1^Nw_n(Q)· e_n, where nn=1N\ e_n\_n=1^N denotes a learnable codebook of latent roles, and wn(Q)w_n(Q) is the routing weight. This formulation enables a single frozen LLM to acquire learnable, query-adaptive, multi-perspective capabilities on the fly. 3.2 Composer Design Our proposed MoRe includes two key modules: a specialization codebook consisting of candidate vectors, and a query-aware router that selects and fuses the candidates. Specialization codebook. To move beyond pre-defined specialization, we maintain a learnable codebook nn=1N\ e_n\_n=1^N, where each entry n∈ℝd e_n ^d is a candidate vector that encodes a specialization as a direction in the residual stream. Rather than random initialization, we warm-start the codebook with contrastive activation directions. Specifically, we adopt Eq. (2) to extract M different vectors as the initialization for nn=1M\ e_n\_n=1^M. Details on the construction of CAA prompts are provided in Appendix B.1. For the remaining N−MN-M entries, we initialize them as the convex mixtures of nn=1M\ e_n\_n=1^M. This anchors the codebook in behaviorally meaningful regions of activation space, which subsequent training refines end-to-end. Query-aware router. To dynamically compose multiple roles, a router is designed score candidate vectors accordingly given the input query. Specifically, we associate each expert n e_n with a learnable probe vector n∈ℝd u_n ^d, concatenating which formulates the probe matrix ∈ℝN×d U ^N× d. Given the token embeddings of the input query =Emb(Q)∈ℝT×d X=Emb(Q) ^T× d, a multi-head cross-attention module utilizes the expert probe U as the query to attend the token embeddings X that serve as the keys and values, that is =MHA(Q,K,V), Z=MHA ( U W^Q, X W^K, X W^V ), (3) where Q,K,V∈ℝd×d W^Q, W^K, W^V ^d× d are the projection matrices. Such design enables each candidate vector to attend the query tokens, forming a query-aware representation ∈ℝN×d Z ^N× d that is further processed by a MLP layer fθ:ℝN×d→ℝNf_θ:R^N× d ^N to generate a routing logit =fθ()∈ℝN g=f_θ( Z) ^N. We further adopt Top-K sparse routing to generate a composed vector Q v_Q, that is Q=∑n∈Kwnn,wn=exp(gn)∑j∈Kexp(gj), v_Q= _n _Kw_n e_n,~~w_n= (g_n) _j _K (g_j), (4) where KT_K is the index set of the Top-K logits in g. Note that for unselected experts (n∉Kn _K), the routing weight is strictly zero. By restricting the fusion to K≪NK N experts, we compose a few distinct roles per query, rather than averaging the whole codebook into an indiscriminate direction. Substituting the composed vector Q v_Q into Eq. (1) gives our final steering operation: ~t(ℓ⋆)=t(ℓ⋆)+α‖t(ℓ⋆)‖2‖Q‖2Q, h^( )_t= h^( )_t+α \| h^( )_t\|_2\| v_Q\|_2 v_Q, In MoRe, we steer the last token of the input query, i.e., t=Tt=T, at the middle layer ℓ∗=⌊L/2⌋ ^*= L/2 . 3.3 Curriculum Supervised Fine-Tuning We first optimize the composer via supervised fine-tuning (SFT) with the backbone LLM frozen. However, direct joint optimization of the codebook and router can be suboptimal. First, an untrained router distributes noisy, near-random routing weights, causing query-specific gradients to be propagated to arbitrary candidates. Second, an untrained codebook lacks stable and distinct semantic anchors, making it difficult for the router to identify appropriate candidates for each query. To avoid these, we propose a three-stage curriculum for SFT training. In stage 1, we only optimize the codebook to ensure that each candidate is individually effective before composition. Specifically, for each query, we uniformly sample a single candidate n e_n from the codebook and optimize the selected candidate with the next-token prediction loss ℒCEL_CE. In stage 2, we freeze the codebook and train the router to select and compose candidates via Top-K routing. To prevent the router from collapsing onto a few candidates, an auxiliary load-balancing loss is adopted Fedus et al. (2022), that is ℒlb=N∑n=1Nfnpn,fn=1BK∑b[n∈K(Qb)]pn=1B∑bSoftmax(Qb)nL_lb=N _n=1^Nf_np_n,~~ \ aligned &f_n= 1BK _b 1[n _K( g_Q_b)]\\ &p_n= 1B _bSoftmax( g_Q_b)_n aligned . where B is the batch size, and (n∈K(Q)) 1(n _K( g_Q)) is the indicator function denoting whether candidate n e_n is among the Top-K for query Q. The stage-2 objective is ℒCE+λlbℒlbL_CE+ _lbL_lb. In stage 3, after both codebook and router are warmed up, we jointly optimize both, allowing the candidate vectors to adapt to the query-dependent compositions learned by the router. 3.4 GRPO Post-Training Figure 2: Comparison between (a) conventional GRPO and (b) GRPO for MoRe. Our policy is defined on the candidate role space, and the rollout only depends on the composed vector generated from the composer, making the credit assignment clear and optimization efficient. Although SFT equips the composer with basic routing capabilities via teacher forcing, such static imitation fails to discover optimal persona synergies. Therefore, we utilize Group Relative Policy Optimization (GRPO) Shao et al. (2024) to post-train the router, encouraging the explorations of better reasoning trajectories. Conventional GRPO optimizes an LLM’s token-level policy by sampling stochastic text trajectories and reinforcing those with high group advantages. However, this standard protocol can not be directly applied to our setting. To train the lightweight router while keeping the backbone LLM frozen, the trajectory advantage must be attributed solely to the generated steering vector, rather than the LLM’s decoding randomness. Policy. To decouple the reward from the LLM and fully attribute it to the router, we formulate the frozen LLM as a black-box environment. Consequently, we shift the RL action space from the token space to the persona space. As defined in Eq. (4), the composed steering vector relies on the candidates and their softmax fusion. Therefore, we define the action space as the space of all possible unordered Top-K-subset =S⊂1,…,N∣|S|=KS=\S⊂\ e_1,…, e_N\ |S|=K\. To enable exploration for GRPO rollout, we adopt a stochastic relaxation by defining the routing policy as a stochastic mapping from query space Q to the action space S, i.e., π:↦π:Q . Instead of taking the top-k logits, we randomly draw K experts based on corresponding Softmax probabilities without replacement. As the composed vector is order-invariant, the probability of sampling a specific unordered subset S given a query Q marginalized over all possible permutations, that is π(S∣Q)=∑s∈perm(S)Pr(s|Q) π(S Q)= _s (S)Pr(s|Q) (5) =∑s∈perm(S)∏n=1Ksoftmax(τ)n∑j∉s1,…,sn−1softmax(τ)j, =\!\!\! _s (S) _n=1^K softmax( gτ)_n _j∉\s_1,…,s_n-1\softmax( gτ)_j, where g is the routing logit, τ is the temperature, and perm(⋅)perm(·) is the set of all possible permutations. Note that the policy π(S∣Q)π(S Q) is governed entirely by the lightweight router and remains independent of the backbone LLM. Therefore, the downstream reward is uniquely attributable to the router’s action, yielding precise credit assignment uncontaminated by LLM sampling noise. Besides, such independence offers substantial computational efficiency as rollouts are executed entirely without gradient tracking. GRPO only evaluates the router on the query representations, thereby eliminating the massive memory and computational overhead associated with LLM. Group-relative update. For each query Q, we sample G steering vector sets Qjj=1G\ v_Q^j\_j=1^G, and steer the frozen LLM with each vector to produce the responses. We adopt verifiable task reward to score each response, and collect the rewards r1,…,rG\r_1,…,r_G\ for all steering vectors. Details on reward designs are provided in Appendix B.2. With the group-normalized advantages Aj=(rj−r¯)/(std(r)+ε)A_j=(r_j- r)/(std(r)+ ), together with the policy probability in Eq. (5), the router is updated by the clipped surrogate ℒGRPOL_GRPO Shao et al. (2024) ℒGRPO=−1G∑j=1Gmin(ρjAj,clip(ρj,1−ϵ,1+ϵ)Aj)+ℛKL _GRPO\!=\!- 1G\! _j=1^G\! (\! _jA_j,\,\!clip( _j,\!1-ε,\!1+ε)A_j\! )\!+\!R_KL ρj=πnew(Sj∣Q)πold(Sj∣Q),ℛKL=βKL(πnew∥πref). _j= _new(S_j Q) _old(S_j Q), _KL= \! ( _new\| _ref ). 4 Experiments 4.1 Experimental Setup Models and baselines. We consider two LLMs: Llama-3.1-8B-Instruct Grattafiori et al. (2024) and Qwen3-8B Yang et al. (2025). We consider single agent baselines including Vanilla LLM, Text Role, Contrastive Activation Addition (CAA) Rimsky et al. (2024), ActAdd Turner et al. (2024), Prompt FT Lester et al. (2021), Billy Pai et al. (2026), NPTI Deng et al. (2025), and IRIS Wei et al. (2026c). We also evaluate MAS baselines including LLM Debate Du et al. (2023), AutoGen Wu et al. (2023), AgentVerse Chen et al. (2024b), and DyLAN Liu et al. (2024c). More details on baselines are provided in Appendix B.3. Table 1: Benchmark results on reasoning tasks. We highlight Top-1, Top-2, and Top-3 performance for each dataset. Method MMLU TriviaQA MATH GSM8K MedQA AvgAcc AvgRank Llama3.1-8B-Instruct Vanilla 61.80 74.80 52.60 80.40 60.00 65.92 8.80 Text 61.00 75.57 56.09 80.06 60.31 66.61 8.60 CAA 61.34 73.77 56.31 82.69 61.06 67.03 7.60 ActAdd 61.74 79.00 56.46 82.89 61.71 68.36 5.00 Prompt FT 62.80 77.60 58.60 78.00 63.20 68.04 4.60 Billy 61.00 74.20 57.00 81.20 59.80 66.64 8.40 LLM Debate 62.20 74.60 58.40 85.80 62.20 68.64 4.20 AutoGen 63.80 73.60 54.80 81.20 61.40 66.96 6.80 AgentVerse 62.40 79.40 59.00 83.80 61.80 69.28 2.60 DyLAN 63.00 76.00 54.40 82.40 61.00 67.36 6.40 MoRe 63.20 79.00 59.20 83.40 61.80 69.32 2.20 Qwen3-8B Vanilla 68.80 61.60 70.20 88.80 59.20 69.72 8.40 Text 67.74 61.43 70.43 90.11 59.26 69.79 7.80 CAA 69.57 62.23 69.83 89.09 59.54 70.05 7.40 ActAdd 69.74 62.63 70.11 90.00 60.09 70.51 6.00 Prompt FT 70.40 60.00 69.80 87.80 58.80 69.96 8.00 Billy 70.40 63.40 67.80 88.20 60.00 69.96 5.40 LLM Debate 69.80 63.00 72.00 92.40 60.80 71.60 3.20 AutoGen 70.20 58.80 72.60 86.60 60.00 69.64 6.40 AgentVerse 69.80 63.00 75.40 91.80 62.00 72.40 2.40 DyLAN 69.40 62.00 70.80 82.20 61.60 69.20 6.80 MoRe 70.20 63.20 72.40 90.80 60.80 71.48 2.80 Table 2: Benchmark results on PersonalityBench. We highlight Top-1 and Top-2 performance for each dataset. Method O C E A N AvgScore AvgRank Llama3.1-8B-Instruct Text 8.12±1.90 7.06±1.25 9.21±0.77 8.15±1.68 9.00±1.76 8.31±0.85 4.6 CAA 8.79±0.47 7.84±1.02 9.56±0.42 7.40±1.15 8.77±0.88 8.47±0.86 4.4 NPTI 9.15±0.71 9.86±0.13 9.72±0.20 9.29±0.40 9.84±0.00 9.57±0.33 2.6 IRIS 9.29±0.36 9.84±0.13 9.76±0.27 9.06±0.66 9.92±0.00 9.57±0.38 2.4 MoRe 9.47±0.44 9.87±0.14 9.81±0.15 9.88±0.11 9.98±0.02 9.80±0.20 1.0 Qwen3-8B Text 8.31±1.68 6.70±1.01 9.39±0.44 8.05±1.6 9.12±0.94 8.31±1.06 3.4 CAA 7.89±1.38 8.47±1.28 9.47±0.93 8.60±1.24 9.81±0.17 8.85±0.78 2.6 NPTI 8.00±1.25 7.98±0.94 7.89±1.53 7.88±0.78 9.17±0.68 8.18±0.55 3.8 IRIS 8.20±1.13 7.87±0.93 8.24±1.73 7.86±0.76 8.87±0.83 8.21±0.55 4.2 MoRe 8.92±1.43 9.49±0.54 9.51±0.32 9.47±0.36 9.91±0.00 9.46±0.35 1.0 Datasets. We assess task-oriented specialization on five reasoning benchmarks spanning general knowledge (MMLU Hendrycks et al. (2020),TriviaQA Joshi et al. (2017)), mathematics (MATH Hendrycks et al. (2021),GSM8K Cobbe et al. (2021)), and domain expertise (MedQA Jin et al. (2021)). We adopt LLM-as-judge Chen et al. (2025a) for evaluation. We also evaluate personality-oriented specialization on PersonalityBench Deng et al. (2025), covering Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. For each personality trait, methods are steered to activate (high pole) and deactivate (low poles) the trait on open-ended questions. Following the standard evaluation protocol, we adopt Claude Anthropic (2024) to score each response on two 1–5 scales, and report the sum and standard deviation of both high and low poles. Experiment pipeline. We adopt 7 candidate roles, including mathematician, software engineer, data scientist, logician, teacher, skeptic, and doctor. Details on text role design are provided in Appendix B. Candidate roles are utilized as the “System Role” for Text; for steering vector extraction in CAA, ActAdd and Billy, and codebook initialization for MoRe. For Text, CAA and ActAdd, we report the average performance over all roles, per-role performance is reported in Appendix B. All experiments are conducted on 8 NVIDIA 40G A100 GPU. 4.2 Reasoning Results Figure 3: Performance vs tokens. Optimal value lies at the upper left corner. MoRe matches MAS performance while only consuming tokens of single-agent methods. Table 1 summarizes results on reasoning benchmarks. A comparison of model performance and token consumption is shown in Figure 3. Overall, MoRe ranks Top-2 on two backbone LLMs with significantly fewer token consumption, demonstrating great balance between performance and efficiency. Specifically, we make the following observations. Specialization improves LLM performance. Comparing with Vanilla, specialization helps LLM achieve higher average accuracy and better average rank. Moreover, the per-role results in Tables 5–5 reveal that different roles affect reasoning performance in distinct ways. For example, mathematician performs well on math datasets (MATH and GSM8K), but is less effective on general knowledge tasks such as TriviaQA. These results suggest that no single role is universally optimal and that selecting appropriate specializations is important for downstream tasks. MoRe outperforms single-agent baselines. On average, MoRe outperforms the best single-agent baselines ActAdd, by 0.96 and 0.97 on Llama and Qwen, respectively. Such improvement is more pronounced on tasks requiring longer reasoning chains. Since longer reasoning involves more intermediate decisions, this provides more opportunities for specializations to influence the reasoning trajectory. In contrast, tasks dominated by short-form recall provide fewer stages at which such specialized reasoning patterns can take effect. MoRe matches MAS performance with significantly lower inference cost. Despite using only single-agent, single-turn inference, MoRe achieves performance comparable to MAS, ranking the first and second on two backbone LLMs, respectively. Unlike MAS approaches, which rely on repeated model calls, iterative discussions, and communication among multiple agents, MoRe only requires a single model call. As shown in Figures 1 and 3, MoRe uses only approximately 1/20 as many tokens as the MAS baselines on average. It therefore provides a favorable balancing between performance and efficiency, matching the reasoning quality of substantially more expensive multi-agent pipelines while avoiding their large token overhead. 4.3 Personality Results Table 2 reports the results on PersonalityBench, covering the Big Five personality traits. Overall, MoRe achieves state-of-the-art performance across backbone LLMs and different personalities. Comparing with the best competitor, MoRe achieves an average outperformance of 0.23 on Llama and 0.61 on Qwen, and consistently ranks the first. In contrast, existing methods exhibit pronounced preferences across personality traits and model families. For example, Text roles perform well on linguistically salient traits, e.g., Extraversion and Neuroticism, while scores lower on Conscientiousness and Agreeableness. IRIS performs well on Llama (AvgRank 2.4) but degrades substantially on Qwen (AvgRank 4.2). Such cross-model degradation suggests that the personality directions identified by existing methods may be closely tied to the internal representation geometry of a particular model. In contrast, MoRe consistently outperforms all baselines across both backbones and exhibits the lowest variation in average performance. These results demonstrate that MoRe induces the desired personalities both effectively and consistently, with stronger robustness to model-specific representations and better generalization across personality dimensions and LLM families. (a) Last token embedding. (b) Codebook similarity. Figure 4: Studies on the learned codebook. 4.4 Studies 4.4.1 On the Learned Codebook We first investigate the learned codebook. Figure 4(a) visualizes the hidden states of the final prompt token after applying different candidate steering vectors. The embeddings form compact clusters, indicating that different candidate vectors redirect model representations towards distinct role-specific regions. We further report the pairwise cosine similarities among the learned candidates in Figure 4(b). Most off-diagonal entries exhibit low similarity, showing that the candidates learn diverse steering directions even without an explicit diversity regularizer. This suggests that the end-to-end training objective naturally encourages functional differentiation among the latent experts. Moreover, the high dimensionality of the steering space provides sufficient capacity for different candidates to occupy distinct directions, thereby reducing redundancy within the codebook. 4.5 On the Learned Router (a) Avg. routing weight (b) Routing weight per dataset Figure 5: Studies on the learned router. Figure 5(a) visualizes the routing weights averaged across all datasets. In general, weights are distributed across most experts rather than collapsing onto a single one, while still exhibiting meaningful preferences. In particular, Experts 0 and 4, initialized from the high-performing mathematician and teacher roles (see Table 5), receive noticeably higher weights, suggesting that the router effectively exploits the useful inductive biases introduced by role-based initialization. Meanwhile, expert 7 remains nearly inactive, potentially indicating an under-utilized or overlapping specialization. Figure 5(b) presents the routing weights for each dataset. Different datasets exhibit distinct routing preferences, whereas related datasets share similar patterns. For example, the two math benchmarks (MATH and GSM8K), heavily utilize Experts 0 initialized from the mathematician. MedQA instead combines Experts 4–6, initialized from the teacher, skeptic, and doctor. In contrast, the broad disciplinary coverage of MMLU leads to a more dispersed routing pattern involving Experts 5, 6, 8, 9. These results indicate that the router learns transferable specializations shared across related tasks and dataset-specific expert compositions, thereby adaptively matching different reasoning requirements with complementary roles. 4.5.1 Ablation Study (a) Router and Codebook (b) Steering Position Figure 6: Ablation studies. (a) Performance gain with base, +router, +codebook, and MoRe; (b) Performance gain with different steering positions. Both figures use Vanilla performance as the baseline. We first study the benefits of two key modules: the learnable codebook and the router. We consider four variants, including: (1) base, which ablates both modules and uniformly fuse fixed CAA vectors, (2) +router, which ablates the codebook and uses the learned router to fuse the fixed CAA vector, (3) +codebook, which ablates the router and uniformly fuse the learned vectors, and (4) MoRe, which jointly employs both learned modules. The results are shown in Figure 6(a). First, even uniform fusion of fixed CAA vectors (base) outperforms Vanilla, which validates the necessity of adopt multi-perspective specialization to enhance LLM reasoning. Second, both modules contribute to the final performance: +codebook achieves more significant improvement than +router, suggesting that learning task-adaptive steering is more effective than dynamic selection among pre-defined ones. Third, MoRe significantly outperforms both ablated variants: the codebook provides diverse steering directions, while the router adaptively selects and combines them according to each input. We further examine the effects of steering position using four variants, including: (1) Resp that steers the response tokens; (2) Prmt that steers all prompt tokens, (3) Prmt-First that steers the first prompt token, and Prmt-Last that steers the last prompt token. The results are shown in Figure 6(b). First, we observe that steering one single prompt token is more effective than steering all prompt or response tokens. This suggests that a local intervention is sufficient to redirect model behaviors, whereas repeated perturbations may interfere with the original semantic representations. Second, steering prompt is more effective than steering response, which indicates that steering before generation can better propagate throughout the entire reasoning process. Third, the optimal position to steer is the final prompt token, as it summarizes the preceding context and serves as the starting point for subsequent generation. Therefore, the final prompt token acts as an effective bridging point for controlling the subsequent reasoning trajectory. 4.5.2 Hyperparameter Study Figure 7: Hyperparameter study on Llama+GSM8K. Figure 7 studies the sensitivity of MoRe to the steering strength α, codebook size N, router top-K, and load-balancing weight λlb _lb. Overall, MoRe remains stable across a broad range of configurations. For the steering strength α, a small α is sufficient to redirect model behavior, whereas an excessively strong α over-perturbs the original representations. Similarly, a moderate codebook size N provides sufficient diversity, while an overly small codebook limits its diversity and a larger one introduces redundancy. Moreover, we observe that a moderate routing K generally performs well: too sparse routing (e.g., Top-1) loses the power of multi-perspective fusion, while too dense routing (e.g., Top-10) fails to achieve adaptivity to queries hence diluting expert specialization. Finally, a moderate λlb _lb yields the best result, suggesting a favorable trade-off between preventing routing collapse and preserving expert specialization. 5 Conclusion In this work, we introduced MoRe, a lightweight module to enable dynamic multi-perspective specialization within a single-agent, single-turn inference process. MoRe learns a diversified codebook of specializations and employs a router to adaptively compose them into a multi-perspective steering vector for each input. This design combines the efficiency of single-agent steering with the adaptivity of multi-agent collaboration. Experiments on reasoning and personality benchmarks demonstrate that MoRe consistently outperforms existing single-agent specialization methods and reduces token cost by 20×20× compared to MAS. Limitations While MoRe achieves consistent gains on reasoning quality and efficiency, we acknowledge a few limitations. First, the proposed MoRe requires access to the internal hidden states of the backbone LLM and is therefore not directly applicable to black-box API models. Second, although the codebook is initialized from interpretable text prompts, the learned candidate vectors are latent. Improving the interpretability of the learned vectors could provide a better understanding of how different candidates contribute to model behavior. Third, MoRe composes a query-level persona and keeps it fixed throughout generation. More fine-grained routing at the token or reasoning-step level may enable additional adaptivity, although it could also introduce extra computational overhead. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: Appendix C, §1. Ai et al. (2025) M. Ai, T. Wei, Y. Chen, Z. Zeng, R. Zhao, G. Varatkar, B. D. Rouhani, X. Tang, H. Tong, and J. He Resmoe: space-efficient compression of mixture of experts llms via residual restoration. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 1–12. Cited by: Appendix C. Anthropic (2024) Anthropic The claude 3 model family: opus, sonnet, haiku. Technical report External Links: Link Cited by: §4.1. Bommasani et al. (2021) R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Cited by: Appendix C. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: Appendix C. Cao et al. (2024) Y. Cao, T. Zhang, B. Cao, Z. Yin, L. Lin, F. Ma, and J. Chen Personalized steering of large language models: versatile steering vectors through bi-directional preference optimization. Advances in Neural Information Processing Systems 37, p. 49519–49551. Cited by: Appendix C, §2.1. Chen et al. (2025a) D. Chen, Q. Yu, P. Wang, M. Hu, W. Zhang, Z. Wang, B. Tang, F. Xiong, X. Li, C. Wang, et al. Xverify: efficient answer verifier for reasoning model evaluations. arXiv preprint arXiv:2504.10481. Cited by: §4.1. Chen et al. (2024a) J. Chen, X. Wang, R. Xu, S. Yuan, Y. Zhang, W. Shi, J. Xie, S. Li, R. Yang, T. Zhu, A. Chen, N. Li, L. Chen, C. Hu, S. Wu, S. Ren, Z. Fu, and Y. Xiao From persona to personalization: a survey on role-playing language agents. Transactions on Machine Learning Research. Note: Survey Certification External Links: ISSN 2835-8856, Link Cited by: Appendix C, §1, §2.1. Chen et al. (2025b) R. Chen, A. Arditi, H. Sleight, O. Evans, and J. Lindsey Persona vectors: monitoring and controlling character traits in language models. arXiv preprint arXiv:2507.21509. Cited by: Appendix C, §1, §2.1. Chen et al. (2024b) W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C. Chan, H. Yu, Y. Lu, Y. Hung, C. Qian, et al. Agentverse: facilitating multi-agent collaboration and exploring emergent behaviors. In International Conference on Learning Representations, Vol. 2024, p. 20094–20136. Cited by: 3rd item, §E.1, §2.2, §4.1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: 2nd item, §E.1, §4.1. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: Appendix C, §1. Cui et al. (2026) C. Cui, T. Wei, Z. Chen, R. Qiu, Z. Zeng, Z. Liu, X. Ning, D. Zhou, and J. He AdaFuse: adaptive ensemble decoding for large language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 42644–42657. Cited by: Appendix C. Dai et al. (2024) D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. Deepseekmoe: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), p. 1280–1297. Cited by: Appendix C. Deng et al. (2025) J. Deng, T. Tang, Y. Yin, X. Zhao, J. Wen, et al. Neuron based personality trait induction in large language models. In International Conference on Learning Representations, Vol. 2025, p. 85059–85083. Cited by: 7th item, §B.4, Appendix C, §E.1, §E.1, §4.1, §4.1. Du et al. (2022) N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al. Glam: efficient scaling of language models with mixture-of-experts. In International conference on machine learning, p. 5547–5569. Cited by: Appendix C. Du et al. (2023) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325. Cited by: 1st item, §E.1, §1, §2.2, §4.1. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: Appendix C, §3.3. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: Appendix C, §E.1, §1, §4.1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: Appendix C, §1. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: 1st item, §E.1, §4.1. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: 3rd item, §E.1, §4.1. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: Appendix C. Izawa et al. (2026) Y. Izawa, G. Minegishi, K. Eguchi, S. Hosokawa, and K. Taura Steering at the source: style modulation heads for robust persona control. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §2.1. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: Appendix C. Jin et al. (2021) D. Jin, E. Pan, N. Oufattole, W. Weng, H. Fang, and P. Szolovits What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11 (14), p. 6421. Cited by: 5th item, §E.1, §4.1. Joshi et al. (2017) M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551. Cited by: 4th item, §E.1, §4.1. Kim et al. (2025) Y. Kim, K. Gu, C. Park, C. Park, S. Schmidgall, A. A. Heydari, Y. Yan, Z. Zhang, Y. Zhuang, Y. Liu, et al. Towards a science of scaling agent systems. arXiv preprint arXiv:2512.08296. Cited by: §1. Lepikhin et al. (2020) D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen Gshard: scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. Cited by: Appendix C. Lester et al. (2021) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 3045–3059. Cited by: 5th item, Appendix C, §E.1, §4.1. Li et al. (2024) D. Li, Y. Ma, N. Wang, Z. Ye, Z. Cheng, Y. Tang, Y. Zhang, L. Duan, J. Zuo, C. Yang, et al. Mixlora: enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint arXiv:2404.15159. Cited by: Appendix C. Li et al. (2023a) G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem Camel: communicative agents for" mind" exploration of large language model society. Advances in neural information processing systems 36, p. 51991–52008. Cited by: §2.2. Li et al. (2023b) K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg Inference-time intervention: eliciting truthful answers from a language model. Advances in Neural Information Processing Systems 36, p. 41451–41530. Cited by: Appendix C, §1, §2.1. Li et al. (2026) Z. Li, Z. Zeng, X. Lin, F. Fang, Y. Qu, Z. Xu, Z. Liu, X. Ning, T. Wei, G. Liu, et al. Flow matching meets biology and life science: a survey. npj Artificial Intelligence 2 (1), p. 17. Cited by: Appendix C. Liang et al. (2025) M. Liang, X. Liu, R. Jin, B. Liu, Q. Suo, Q. Zhou, S. Zhou, L. Chen, H. Zheng, Z. Li, et al. External large foundation model: how to efficiently serve trillions of parameters for online ads recommendation. In Companion Proceedings of the ACM on Web Conference 2025, p. 344–353. Cited by: Appendix C. Liang et al. (2024) T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, S. Shi, and Z. Tu Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 conference on empirical methods in natural language processing, p. 17889–17904. Cited by: §1, §2.2. Lin et al. (2026a) X. Lin, P. Li, Z. Zeng, T. Li, T. Wei, X. Ning, G. Li, Y. Chen, and H. Tong Alert: zero-shot llm jailbreak detection via internal discrepancy amplification. arXiv preprint arXiv:2601.03600. Cited by: Appendix C. Lin et al. (2026b) X. Lin, Z. Tang, W. Cong, M. Hang, K. Wang, Y. Wang, Z. Zeng, T. Li, H. Yoo, Z. Liu, et al. Mixture of sequence: theme-aware mixture-of-experts for long-sequence recommendation. In Proceedings of the ACM Web Conference 2026, p. 6469–6480. Cited by: Appendix C. Liu et al. (2024a) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the association for computational linguistics 12, p. 157–173. Cited by: §2.2. Liu et al. (2024b) Q. Liu, X. Wu, X. Zhao, Y. Zhu, D. Xu, F. Tian, and Y. Zheng When moe meets llms: parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, p. 1104–1114. Cited by: Appendix C. Liu et al. (2024c) Z. Liu, Y. Zhang, P. Li, Y. Liu, and D. Yang A dynamic llm-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling, Cited by: 4th item, §E.1, §1, §2.2, §4.1. Ning et al. (2026) X. Ning, K. Tieu, D. Fu, T. Wei, Z. Li, Y. Bei, J. Zou, M. Ai, Z. Liu, T. Li, et al. Code as agent harness. arXiv preprint arXiv:2605.18747. Cited by: Appendix C. Pai et al. (2026) T. Pai, J. Wang, L. Lu, S. Sun, H. Lee, and K. Chang Billy: steering large language models via merging persona vectors for creative generation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7870–7915. Cited by: 6th item, §E.1, §4.1. Rimsky et al. (2024) N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15504–15522. Cited by: 3rd item, Appendix C, §E.1, §2.1, §3.1, §3.1, §4.1. Shanahan et al. (2023) M. Shanahan, K. McDonell, and L. Reynolds Role play with large language models. Nature 623 (7987), p. 493–498. Cited by: Appendix C, §1, §2.1. Shao et al. (2023) Y. Shao, L. Li, J. Dai, and X. Qiu Character-llm: a trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 13153–13187. Cited by: Appendix C, §1, §2.1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §3.4, §3.4. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. Cited by: Appendix C. Sun et al. (2025) S. Sun, S. Y. Baek, and J. H. Kim Personality vector: modulating personality of large language models by model merging. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 24667–24688. Cited by: Appendix C, §1, §1, §2.1. Tran and Kiela (2026) D. Tran and D. Kiela Single-agent llms outperform multi-agent systems on multi-hop reasoning under equal thinking token budgets. arXiv preprint arXiv:2604.02460. Cited by: §1. Turner et al. (2024) A. M. Turner, L. Thiergart, G. Leech, D. Udell, U. Mini, and M. MacDiarmid Activation addition: steering language models without optimization. Cited by: 4th item, Appendix C, §E.1, §1, §2.1, §3.1, §4.1. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: Appendix C. Wei et al. (2026a) T. Wei, T. Li, Z. Liu, X. Ning, Z. Yang, J. Zou, Z. Zeng, R. Qiu, X. Lin, D. Fu, et al. Agentic reasoning for large language models. arXiv preprint arXiv:2601.12538. Cited by: Appendix C. Wei et al. (2026b) T. Wei, Z. Zeng, R. Qiu, Z. Liu, X. Ning, X. He, W. Bao, X. Lin, Q. He, X. Tang, H. Tong, and J. He Inference scaling of LLM ensembling: bridging token spaces with token translation. In Third Conference on Language Modeling, External Links: Link Cited by: Appendix C. Wei et al. (2026c) Z. Wei, M. Li, Z. Wang, and Y. Deng Beyond static personas: situational personality steering for large language models. In Findings of the Association for Computational Linguistics: ACL 2026, p. 19185–19210. Cited by: 8th item, Appendix C, §E.1, §1, §1, §2.1, §4.1. Wu et al. (2023) Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. Autogen: enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155. Cited by: 2nd item, §E.1, §1, §2.2, §4.1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Appendix C, §E.1, §4.1. Zeng et al. (2026a) H. Zeng, Y. Xia, Z. Zhao, C. Jiang, Q. Zhang, J. Liu, Q. Zhang, L. Zhang, X. Fan, and B. Zhang S’more: structural mixture of residual experts for parameter-efficient llm fine-tuning. Advances in Neural Information Processing Systems 38, p. 81441–81478. Cited by: Appendix C. Zeng et al. (2026b) Z. Zeng, W. Bao, X. Lin, R. Qiu, T. Wei, X. Ning, Y. Yan, C. Luo, M. X. Cheng, J. He, et al. Subspace alignment for vision-language model test-time adaptation. arXiv preprint arXiv:2601.08139. Cited by: Appendix C. Zeng et al. (2025a) Z. Zeng, M. Hang, X. Liu, X. Liu, X. Lin, R. Qiu, T. Wei, Z. Liu, S. Yuan, C. Yang, et al. Hierarchical lora moe for efficient ctr model scaling. arXiv preprint arXiv:2510.10432. Cited by: Appendix C. Zeng et al. (2025b) Z. Zeng, X. Liu, M. Hang, X. Liu, Q. Zhou, C. Yang, Y. Liu, Y. Ruan, L. Chen, Y. Chen, et al. Interformer: effective heterogeneous interaction learning for click-through rate prediction. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 6225–6233. Cited by: Appendix C. Zeng et al. (2026c) Z. Zeng, Q. Yu, X. Lin, R. Qiu, X. Ning, T. Wei, Y. Yan, J. He, and H. Tong Harnessing consistency for robust test-time llm ensemble. In Findings of the Association for Computational Linguistics: EACL 2026, p. 3528–3545. Cited by: Appendix C. Zhang et al. (2026) Y. Zhang, D. Yu, T. Ge, L. Song, Z. Zeng, H. Mi, N. Jiang, and D. Yu Improving llm general preference alignment via optimistic online mirror descent. Advances in Neural Information Processing Systems 38, p. 160165–160187. Cited by: Appendix C. Appendix Figure 8: Per-role performance ranges compared with MoRe. For each method, the blue bar spans the lowest to the highest score achieved across different roles, while the red dashed line denotes the performance of MoRe. Figure 9: Performance vs #token on different models and datasets. Left upper corner represents the Pareto front with the best performance and minimal inference cost. Appendix A Additional Experiment Results A.1 Per-role Performance To better understand the effect of roles on performance, we report the per-role results in Tables 5–5 and summarize their performance in Figure 8. Table 3: Per-role performance on Text. Role MMLU TriviaQA MATH GSM8K MedQA AvgAcc Llama-8B-Instruct Math. 64.00 73.60 58.40 81.40 61.00 67.68 SDE 59.60 75.60 55.00 79.60 54.00 64.76 DataSci. 60.20 75.40 56.40 81.60 61.40 67.00 Logic. 61.00 72.40 54.40 77.20 55.80 64.16 Teacher 61.80 78.20 58.20 79.80 61.00 67.80 Skeptic 60.20 77.60 51.40 80.40 64.20 66.76 Doctor 60.20 76.20 58.80 80.40 64.80 68.08 Qwen3-8B Math. 70.40 61.60 70.40 91.80 59.00 70.64 SDE 67.40 61.80 71.40 91.00 59.80 70.28 DataSci. 66.60 61.00 69.80 90.20 58.40 69.20 Logic. 67.20 61.80 70.60 89.80 57.80 69.44 Teacher 67.00 60.60 69.80 90.20 60.20 69.56 Skeptic 68.00 62.20 70.00 89.60 58.80 69.72 Doctor 67.60 61.00 71.00 88.20 60.80 69.72 Table 4: Per-role performance on CAA. Role MMLU TriviaQA MATH GSM8K MedQA AvgAcc Llama-8B-Instruct Math. 60.60 74.40 58.00 84.40 60.40 67.56 SDE 60.60 72.00 56.20 80.80 61.20 66.16 DataSci. 62.00 76.20 56.80 80.80 59.80 67.12 Logic. 61.00 74.60 54.00 83.60 61.40 66.92 Teacher 61.60 72.60 56.80 82.00 61.80 66.96 Skeptic 62.40 73.60 55.60 82.80 62.00 67.28 Doctor 61.20 73.00 56.80 84.40 60.80 67.24 Qwen3-8B Math. 70.40 62.00 69.80 90.40 59.60 70.44 SDE 68.80 62.40 70.40 89.80 59.60 70.20 DataSci. 69.40 61.60 67.60 88.00 59.40 69.20 Logic. 69.60 62.20 70.20 89.40 59.20 70.12 Teacher 69.00 62.80 70.20 88.80 60.00 70.16 Skeptic 69.20 62.20 70.80 89.60 59.60 70.28 Doctor 70.60 62.40 69.80 87.60 59.40 69.96 Table 5: Per-role performance on ActAdd. Role MMLU TriviaQA MATH GSM8K MedQA AvgAcc Llama-8B-Instruct Math. 61.80 79.20 55.20 82.40 61.60 68.04 SDE 61.60 78.80 55.00 83.40 61.60 68.08 DataSci. 62.00 79.40 56.60 83.40 61.60 68.60 Logic. 61.80 78.80 57.80 82.40 62.00 68.56 Teacher 61.60 79.00 57.80 83.80 61.80 68.80 Skeptic 61.40 78.80 56.20 82.20 61.60 68.04 Doctor 62.00 79.00 56.60 82.60 61.80 68.40 Qwen3-8B Math. 69.60 62.20 70.80 89.00 59.80 70.28 SDE 69.60 63.00 70.40 89.60 60.00 70.52 DataSci. 70.00 62.60 68.60 89.60 60.00 70.16 Logic. 69.80 63.00 69.40 90.40 60.20 70.56 Teacher 69.60 62.60 69.60 91.00 60.20 70.60 Skeptic 70.00 62.40 71.60 90.60 60.20 70.96 Doctor 69.60 62.60 70.40 89.80 60.20 70.52 We first observe that role effectiveness is highly task-dependent. Some roles exhibit intuitive alignment with particular tasks: for example, the mathematician role performs strongly on mathematical reasoning benchmarks, while the doctor role achieves the best Text performance on MedQA for both backbones. However, such alignment is not universal. Roles emphasizing general reasoning strategies, e.g., teacher, skeptic, and logician, can outperform domain-matched roles on several tasks, and the best role often changes across backbone LLMs and steering methods. These results suggest that roles induce complementary reasoning biases rather than fixed, task-specific capabilities, and that no single predefined role is consistently optimal. In contrast, MoRe generally performs near the upper bound of the per-role ranges or even surpasses the best predefined role. Such improvements can not be solely achieved by selecting a globally strong role, but instead validates that MoRe benefits from query-dependent selection and composition of complementary candidate steering vectors. Consequently, MoRe produces composed specializations that are not restricted to any single predefined role. A.2 Balancing Performance–Efficiency Figure 9 compares model performance against the number of tokens consumed. Across different models and datasets, MoRe consistently occupies the upper-left region, indicating a favorable balance between accuracy and inference cost. For example, MoRe achieves the best performance on MATH with Llama while using substantially fewer tokens than all MAS baselines, and remains highly competitive on TriviaQA and GSM8K across both backbones. Compared with single-agent methods, MoRe consumes a similar number of tokens but generally achieves stronger performance, with particularly clear gains on MATH and GSM8K. This suggests that query-dependent composition is especially beneficial for tasks involving longer reasoning trajectories and multiple intermediate decisions. In contrast, fixed textual roles or steering vectors may provide useful but limited specialization. MAS baselines occasionally achieve higher accuracy, such as on GSM8K, but typically require hundreds or thousands of completion tokens per sample. In comparison, MoRe approaches their performance with one to two orders of magnitude lower token cost. Furthermore, several MAS methods remain less accurate than MoRe despite their substantially larger inference budgets, showing that additional agent interactions do not necessarily produce proportional performance gains. Overall, MoRe combines the efficiency of single-agent inference with performance competitive with substantially more expensive MAS pipelines. A.3 On the Steering Effect Figure 10 measures the angular change of the last token embedding with and without steering. The left figure shows the average angular change and std steered by different expert. We observe a consistent increase from approximately 12∘12 at the steering layer to over 40∘40 at the final layer, indicating that the injected perturbation is not attenuated by subsequent computation but is progressively propagated and amplified through the network. The right figure shows the angular change per expert along the layers. Although all candidates exhibit the similar increasing trend, their layer-wise trajectories differ noticeably, with the variance across candidates increasing in deeper layers. This suggests that the learned candidate vectors induce distinct patterns of representational transformation rather than merely applying perturbations of different magnitudes. Together, these results show that steering produces persistent and candidate-specific effects that accumulate toward the model output. Figure 10: Angular change between last token embeddings with and without steering. Appendix B Experiment Details B.1 Text Role Design We describe our text role design for Text, CAA, Billy, and the codebook initialization for MoRe. In general, we consider seven persons covering different roles as follows Mathematician "You are a Mathematician. Work from precise definitions and exact quantities. Manipulate the problem symbolically, derive each result formally, and justify that every line follows necessarily from the one before." Software Engineer "You are a Software Engineer. Decompose the task into a concrete procedure, then mentally execute it like code: track the intermediate state, cover boundary and edge cases, and run the inputs back through your answer to test it before finalizing." Data Scientist "You are a Data Scientist. Reason quantitatively and empirically. Estimate magnitudes, weigh base rates and likelihoods, sanity-check every number against a plausible range, and let the evidence arbitrate between competing answers." Logician "You are a Logician. List the premises explicitly, then chain strict deductive inferences. Discard any option that forces a contradiction, and surface every hidden assumption and logical fallacy." Teacher "You are a Teacher. Build intuition in plain language. Motivate why each move makes sense, ground it with a simple analogy or concrete example, and keep the explanation clear enough for a beginner to follow." Skeptic "You are a Skeptic. Distrust the first answer that comes to mind. Hunt for counterexamples, watch for traps and misleading wording, and re-derive the result from a different angle before you commit to it." Doctor "You are a Medical Doctor. Gather the salient findings, form a differential of candidate explanations, then rule them out one by one by weighing the evidence for and against each, until the most likely answer remains." B.2 Reward Design We utilize GRPO to post train the composer, where the training signal originates from the group-relative advantages of a group of rollouts. A binary reward based on answer correctness can be problematic in this setting: on samples where rollouts consistently provide correct or wrong answers, the group’s reward variance collapses to zero, and the resulting advantage vanishes. Therefore, we design soft, task-aligned rewards that (1) preserve a strict correct-over-incorrect ordering, and (2) inject within-group variance so that rollouts sharing the same final-answer verdict are still separable by reasoning quality. Our designs are as follows • Multi-choice questions (MMLU, MedQA). We assign a stepped reward: 1.01.0 for an exact match to the groundtruth, 0.20.2 when the model emits a wrong but syntactically valid choice (A–D) as a format reward, and 0.00.0 when no valid choice can be parsed. • Free-form QA questions (TriviaQA). We assign a soft reward: an exact match scores 1.01.0; otherwise the reward is the SBERT similarity in [0,1][0,1] between the normalized prediction and groundtruth. • Long-chain reasoning questions (GSM8K, MATH). We assign a soft reward. For correct rollouts, we assign a soft reward of τ+(1−τ)⋅sim(c,r)τ+(1-τ)·sim(c,r), where sim(⋅,⋅)sim(·,·) is the SBERT cosine similarity between the generated chain-of-thought c and the reference rationale r. For incorrect rollouts, we assign a continuous partial-credit score min(τ,;λ1⋅cov+λ2⋅sim) (τ,; _1·cov+ _2·sim ), where cov is the fraction of the reference’s intermediate numeric values that also appear in the rollout’s reasoning. B.3 Baseline Methods We consider the following single LLM baselines • Vanilla utilizes backbone LLM without any specialization. • Text prepends text persona before the query. • CAA Rimsky et al. (2024) extracts persona steering vector from contrastive pairs. • ActAdd Turner et al. (2024) extracts token-wise steering vector from contrastive pairs. • Prompt FT Lester et al. (2021) fine-tunes the soft prompt tokens. • Billy Pai et al. (2026) merges multiple contrastive vectors into a single averaged steering vector. • NPTI Deng et al. (2025) modulates neurons for each persona traits in the FFN activations. • IRIS Wei et al. (2026c) extends NPTI by retrieving topic-specific neuron sets for adaptive neuron modulation. We also consider the following MAS baselines • LLM Debate Du et al. (2023) orchestrates multi-agent to iteratively debate each other’s reasoning to reach a consensus. • AutoGen Wu et al. (2023) enables multi-agent collaboration by assigning distinct roles and facilitating multi-turn conversations. • AgentVerse Chen et al. (2024b) coordinates multi-agent via expert recruitment, joint brainstorming, and voting. • DyLAN Liu et al. (2024c) dynamically constructs an agent network that evaluates contributions and routes information among specialized agents across multiple reasoning layers. B.4 Datasets We evaluate on five reasoning benchmarks spanning general knowledge (MMLU, TriviaQA), mathematics (MATH,GSM8K), and domain expertise (MedQA), including • MMLU Hendrycks et al. (2020) consists of multiple-choice questions across 57 subjects for assessing general knowledge and problem-solving abilities. • GSM8K Cobbe et al. (2021) covers grade-school math word problems that require multi-step reasoning. • MATH Hendrycks et al. (2021) covers mathematics problems ranging from grade-school difficulty to advanced competition level. • TriviaQA Joshi et al. (2017) covers open-domain factoid question answering and reading comprehension questions. • MedQA Jin et al. (2021) consists of multiple-choice questions collected from professional medical board exams. Besides, we adopt PersonalityBench Deng et al. (2025) to evaluate whether LLM responses exhibit specified personality traits. It is grounded in the Big Five personality model, covering Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, including the opposing aspects of each trait. PersonalityBench utilizes open-ended situational questions derived from real-world behaviors, requiring models to express the target personality through natural-language responses. We evaluate on the evaluation set containing 90 questions per personality, and employ an LLM-as-a-judge to assess the extent to which each response reflects the intended personality. Appendix C More on Related Works We provide more related works on mixture of experts and pre-trained foundation models. Mixture of Experts Mixture-of-Experts (MoE) provides a general framework for conditional computation by dynamically routing different inputs to specialized expert modules Shazeer et al. (2017); Lepikhin et al. (2020). Modern sparse MoE architectures substantially scale this principle by activating only a small subset of experts for each input Fedus et al. (2022); Du et al. (2022); Ai et al. (2025); Lin et al. (2026b). This paradigm has become increasingly prevalent in LLMs, where models such as Mixtral Jiang et al. (2024) and DeepSeekMoE Dai et al. (2024) increase model capacity while maintaining a relatively small number of activated parameters. In particular, recent designs increasingly emphasize fine-grained expert specialization and flexible expert composition, which decomposes experts into finer-grained modules and introduces shared experts to reduce redundancy among routed experts Dai et al. (2024). Beyond full-parameter experts, the MoE principle has also been extended to parameter-efficient specialization, where lightweight LoRA adapters serve as experts and are dynamically selected or composed according to the input Liu et al. (2024b); Li et al. (2024); Zeng et al. (2025a); Zeng et al. (2026a). MoRe extends the principle of MoE from parameter-space computation to activation-space behavioral specialization, enabling dynamic multi-perspective composition with minimal additional inference overhead. Pre-trained Foundational Models Foundation models have reshaped the field by pretraining models on vast amounts of web-scale data Brown et al. (2020); Bommasani et al. (2021); Achiam et al. (2023); Grattafiori et al. (2024). Large Language Models (LLMs) built upon this paradigm have demonstrated remarkable performance across diverse tasks including natural language understanding Comanici et al. (2025); Yang et al. (2025); Wei et al. (2026b), knowledge-intensive question answering Achiam et al. (2023); Zeng et al. (2026c); Cui et al. (2026); Lin et al. (2026a), multi-modal reasoning Zeng et al. (2026b); Zeng et al. (2025b); Liang et al. (2025), and domain-specific problem solving Wei et al. (2022); Guo et al. (2025); Li et al. (2026); Zhang et al. (2026). Beyond general-purpose capabilities, an emerging line of research seeks to specialize pretrained LLMs toward particular domains, behaviors, personalities, or reasoning styles Wei et al. (2026a); Ning et al. (2026). Such specialization can be induced through parameter-efficient adaptation, including prompt tuning and lightweight parameter updates Lester et al. (2021); Hu et al. (2021), or through role-conditioned prompting that instructs LLMs to behave as particular experts or personas Shanahan et al. (2023); Shao et al. (2023); Chen et al. (2024a). More recently, representation-level approaches directly manipulate internal activations to induce targeted behaviors without modifying the backbone parameters Li et al. (2023b); Turner et al. (2024); Rimsky et al. (2024). These techniques have been further extended to persona and behavioral control through learned steering vectors, neuron-level interventions, and context-dependent specialization Cao et al. (2024); Chen et al. (2025b); Deng et al. (2025); Sun et al. (2025); Wei et al. (2026c). Together, these studies demonstrate that pretrained foundation models contain rich latent capabilities that can be selectively elicited through lightweight specialization. However, most existing specialized-agent approaches rely on a predefined or individually selected specialization, motivating methods that can dynamically compose multiple complementary specializations according to the input query. Appendix D Potential Risks MoRe modifies the internal representations of LLMs to induce query-dependent specialization. Although our experiments focus on established reasoning and personality benchmarks, the learned steering vectors may amplify undesirable biases or produce unintended behavioral changes on inputs outside the training distribution. Besides, the personality evaluation relies on LLM-as-a-judge, which may inherit biases, preferences, or inconsistencies from the evaluator model. We mitigate this by following the standard evaluation protocol of PersonalityBench and applying the same scoring criteria to all methods. Appendix E Use Or Create Scientific Artifacts Our work builds upon publicly available datasets, pre-trained language models, and existing specialization and multi-agent baselines. We do not collect new data from human participants. E.1 Cite Creators Of Artifacts All external artifacts are properly credited to their original publications and repositories. The benchmarks used in this work, including MMLU Hendrycks et al. (2020), TriviaQA Joshi et al. (2017), MATH Hendrycks et al. (2021), GSM8K Cobbe et al. (2021)), MedQA Jin et al. (2021) and PersonalityBench Deng et al. (2025), are credited to their respective authors. The backbones used in this work, including Llama3.1-8B-Instruct Grattafiori et al. (2024) and Qwen3-8B Yang et al. (2025), are referenced to its official publication. We additionally cite the original publications for all single-agent and multi-agent baselines, including CAA Rimsky et al. (2024), ActAdd Turner et al. (2024), Prompt FT Lester et al. (2021), Billy Pai et al. (2026), NPTI Deng et al. (2025), IRIS Wei et al. (2026c), LLM Debate Du et al. (2023), AutoGen Wu et al. (2023), AgentVerse Chen et al. (2024b), and DyLAN Liu et al. (2024c). E.2 Discuss The License For Artifacts We comply with the licenses of all artifacts used in this work. The datasets and backbone models are available for research purposes under their respective licenses. E.3 Data Contains Personally Identifying Info Or Offensive Content We do not collect or introduce new personally identifying information. The reasoning benchmarks mainly contain general knowledge, mathematical, and professional examination questions. PersonalityBench contains synthetically constructed situational questions based on the Big Five personality framework. These questions may describe interpersonal conflicts, emotional states, or other sensitive situations necessary for evaluating personality expression. To our knowledge, they do not contain private information about identifiable individuals. E.4 Documentation Of Artifacts Appendix B documents the evaluated backbone models, datasets, baselines, evaluation protocols, and hardware. Dataset statistics are summarized in Appendix F. These materials specify how each artifact is used and support reproducibility of the experiments. Appendix F Statistics For Data We summarize the dataset statistics as follows F.1 MMLU • Domain: Multiple-choice knowledge and reasoning questions covering 57 subjects in the humanities, social sciences, natural sciences, and professional domains. Each question has four candidate answers. • Size: Approximately 16,000 questions. • Metric: exact match F.2 TriviaQA • Domain: Open-domain factoid question answering covering diverse topics such as history, geography, science, entertainment, and popular culture. Questions require free-form textual answers. • Size: Approximately 95,000 question–answer pairs. • Metric: accuracy given by LLM-as-a-judge. F.3 MATH • Domain: Competition-level mathematical problem solving across algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus. Each problem requires a free-form solution and final answer. • Size: 7,500 training problems and 5,000 test problems. • Metric: accuracy given by LLM-as-a-judge. F.4 GSM8K • Domain: Multi-step grade-school mathematical word problems that require arithmetic reasoning and the generation of a numerical answer. • Size: 7,473 training problems and 1,319 test problems. • Metric: accuracy given by LLM-as-a-judge. F.5 MedQA • Domain: Multiple-choice medical question answering derived from professional medical licensing examinations. We use the English subset, which primarily contains questions based on the United States Medical Licensing Examination. • Size: 12,723 questions across the official training, development, and test splits. • Metric: exact match F.6 PersonalityBench • Domain: Open-ended situational question answering for evaluating personality expression. • Size: 180,000 training instances, with approximately 36,000 instances for each personality trait. Its evaluation set contains approximately 90 situational questions for each trait. • Metric: LLM-as-a-judge to score each response on a scale from 1 to 5. We report the sum of the scores for the high and low poles of each trait, as well as the standard deviation across traits. Appendix G Computational Experiments All computational experiments in this work are fully reproducible, with details provided in Section 4.1. G.1 Model Size And Budget We evaluate MoRe on two open-weight instruction-following LLMs: • Llama-3.1-8B-Instruct: approximately 8 billion parameters. • Qwen3-8B: approximately 8 billion parameters. The parameters of both backbone LLMs remain frozen throughout training. Only the codebook of candidate steering vectors and the lightweight query-aware composer are optimized. The resulting number of parameters 0.04M for codebook and 1.32M for router, sum of which constitutes 0.017% of the backbone model. All experiments are conducted on 8 NVIDIA A100 GPUs with 40GB memory. G.2 Experimental Setup And Hyper-params We describe experimental settings in Section 4.1. Our default configuration for key hyperparameters studied includes: • Steering strength α=0.2α=0.2. • Codebook size N=10N=10. • Sparse routing Top-K with K=3K=3. • Load-balancing weight λlb=0.1 _lb=0.1. We study the sensitivity to the following hyperparameters in Section 4.5.2, including Steering strength α∈0.1,0.2,0.4,0.6,0.8α∈\0.1,0.2,0.4,0.6,0.8\; Codebook size N∈4,7,10,13,16N∈\4,7,10,13,16\; Router sparsity K∈1,3,5,7,10K∈\1,3,5,7,10\; and Load-balancing weight λlb∈0,10−3,10−2,10−1,1 _lb∈\0,10^-3,10^-2,10^-1,1\. G.3 Descriptive Statistics For the five reasoning benchmarks, we report accuracy on each dataset, average accuracy across datasets, and average rank across methods. For PersonalityBench, we report the LLM-judge score for each Big Five trait, the average score across traits, and the standard deviation between the high- and low-pole evaluations. G.4 Parameters For Packages The existing packages used are specified as follows, including Python: 3.10.20, PyTorch: 2.9.1 (CUDA 12.8), transformers: 5.10.2, datasets: 5.0.0, sentence-transformers: 5.5.1 and vllm 0.6.2. Appendix H AI Assistants In Research Or Writing In this paper, AI assistant tool is used to edit and improve the quality of the text, including checking the spelling, grammar, punctuation and clarity.