Paper deep dive
Consilience: Conformally Calibrated Communication Control for Hidden-Profile Multi-Agent Reasoning
Abhijith Babu, Ramneet Kaur, Vishal Pramanik, Olivera Kotevska, Nathaniel D. Bastian, Susmit Jha, Sunny Raj, Yanzhao Wu, Sumit Kumar Jha, Anirban Roy
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/24/2026, 4:45:13 AM
Summary
The paper introduces Consilience, an inference-time orchestration framework for multi-agent LLM systems designed to improve reasoning in hidden-profile settings where information is distributed among agents. Consilience uses a compact state representation to summarize discussion dynamics (uncertainty, disagreement, redundancy, etc.) and employs a round-wise conformal calibration procedure to certify communication interventions (challenge, clarify, seek evidence, route, vote). This provides distribution-free, finite-sample guarantees on the regret of the selected actions, ensuring that communication is adaptive and statistically reliable. Experiments on HiddenBench-style tasks show that Consilience improves decision accuracy and communication efficiency over fixed protocols and unstructured debates, sometimes surpassing full-information baselines.
Entities (10)
Relation Signals (9)
Consilience → uses → Conformal Calibration
confidence 95% · Its central contribution is a round-wise conformal calibration procedure that provides a distribution-free, finite-sample guarantee...
Consilience → evaluatedon → HiddenBench
confidence 92% · On HiddenBench-style hidden-profile tasks... Consilience improves decision accuracy...
Conformal Calibration → optimizes → One-step Regret
confidence 90% · We propose one-step regret as the non-conformity score... round-wise conformal acceptance retains or replaces it using calibrated one-step regret.
Consilience → targets → Hidden-Profile
confidence 90% · ...steers and certifies multi-agent communication under distributed private information... particularly in hidden-profile settings...
Consilience → employsactions → Clarify
confidence 85% · Consilience uses four communication interventions: Challenge, Clarify, SeekEvidence, Route.
Consilience → employsactions → SeekEvidence
confidence 85% · Consilience uses four communication interventions: Challenge, Clarify, SeekEvidence, Route.
Consilience → employsactions → Route
confidence 85% · Consilience uses four communication interventions: Challenge, Clarify, SeekEvidence, Route.
Consilience → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-agent LLM systems can improve reasoning by pooling diverse perspectives, but their effectiveness depends on coordinating communication, particularly in hidden-profile settings where each agent holds only part of the evidence required for a correct decision. Existing protocols, including fixed schedules, round-robin exchange, and unstructured debate, provide no guarantee that a conversational action is appropriate. We propose Consilience, an inference-time orchestration framework that both steers and certifies multi-agent communication under distributed private information. At each turn, Consilience summarizes the discussion using a compact state capturing uncertainty, disagreement, evidence gain, redundancy, and premature consensus, then selects both a communication intervention (challenge, clarify, seek evidence, or route) and an appropriate speaker. Its central contribution is a round-wise conformal calibration procedure that provides a distribution-free, finite-sample guarantee: at each discussion round, conditional on reaching that round, the one-step regret of a controller's proposed action is bounded by a calibrated threshold with marginal probability at least 1 - alpha; an acceptance mechanism enforces the same guarantee for the executed action by replacing inadmissible proposals. On HiddenBench-style hidden-profile tasks spanning 12 open and closed weight language models, Consilience improves decision accuracy and communication efficiency over fixed and unstructured discussion protocols, sometimes surpassing a full-information baseline where every agent observes all evidence. These results demonstrate that certified adaptive communication control can be more valuable than increasing information availability, providing a practical mechanism for reliable multi-agent LLM coordination.
Tags
Links
- Source: https://arxiv.org/abs/2608.20564v1
- Canonical: https://arxiv.org/abs/2608.20564v1
Trouble viewing inline? Open PDF directly →
Full Text
99,693 characters extracted from source content.
Expand or collapse full text
Consilience: Conformally Calibrated Communication Control for Hidden-Profile Multi-Agent Reasoning Abhijith Babu1, Ramneet Kaur2, Vishal Pramanik3, Olivera Kotevska4, Nathaniel D. Bastian5 Susmit Jha2, Sunny Raj6, Yanzhao Wu1, Sumit Kumar Jha3,*, Anirban Roy2 1Knight Foundation School of Computing and Information Sciences, Florida International University 2SRI International 3Department of Computer & Information Science & Engineering, University of Florida 4Oak Ridge National Laboratory 5Army Cyber Institute, United States Military Academy 6Department of Computer Science and Engineering, Oakland University *Corresponding author: sumit.jha@ufl.edu Abstract Multi-agent LLM systems can improve reasoning by pooling diverse perspectives, but their effectiveness depends on coordinating communication, particularly in hidden-profile settings where each agent holds only part of the evidence required for a correct decision. Existing protocols, including fixed schedules, round-robin exchange, and unstructured debate, provide no guarantee that a conversational action is appropriate. We propose Consilience, an inference-time orchestration framework that both steers and certifies multi-agent communication under distributed private information. At each turn, Consilience summarizes the discussion using a compact state capturing uncertainty, disagreement, evidence gain, redundancy, and premature consensus, then selects both a communication intervention (challenge, clarify, seek evidence, or route) and an appropriate speaker. Its central contribution is a round-wise conformal calibration procedure that provides a distribution-free, finite-sample guarantee: at each discussion round, conditional on reaching that round, the one-step regret of a controller’s proposed action is bounded by a calibrated threshold with marginal probability at least 1−α1-α; an acceptance mechanism enforces the same guarantee for the executed action by replacing inadmissible proposals. On HiddenBench-style hidden-profile tasks spanning 12 open and closed weight language models, Consilience improves decision accuracy and communication efficiency over fixed and unstructured discussion protocols, sometimes surpassing a full-information baseline where every agent observes all evidence. These results demonstrate that certified adaptive communication control can be more valuable than increasing information availability, providing a practical mechanism for reliable multi-agent LLM coordination. 1 Introduction Large language models (LLMs) are increasingly deployed as multi-agent systems in which several agents propose solutions, exchange arguments, critique one another, and jointly produce a decision 6; 12; 23; 22; 10. Such systems have shown promise in mathematical reasoning, question answering, software development, planning, and tool-assisted problem solving. Their appeal stems from the idea that multiple agents can contribute complementary reasoning, identify errors overlooked by individual models, and produce more reliable collective decisions. However, recent studies question whether discussion alone consistently delivers these benefits. Multi-agent deliberation can duplicate the capabilities of a well-prompted single agent, amplify correlated errors, or converge prematurely on persuasive but incorrect conclusions 21; 12. Simply increasing the number of agents or discussion rounds therefore does not guarantee better collective reasoning. This limitation becomes especially consequential when task-relevant information is distributed across agents. Many collaborative decisions in medicine, intelligence analysis, and organizational planning involve participants who observe different pieces of evidence. No individual has access to the complete information required for a correct decision; the group succeeds only if its members communicate and integrate their complementary knowledge. This setting is captured by the hidden-profile paradigm, originally introduced in social psychology to study information pooling in group decision making 17; 18. In a hidden profile, information shared by all group members supports a plausible but suboptimal alternative, whereas uniquely held information, when pooled across members, reveals the correct choice 17; 14; 15. Although discussion should theoretically recover distributed evidence, human groups consistently exhibit a common-information bias: information shared by several members is more likely to be mentioned, repeated, and reinforced than information known to only one member 17; 16; 7. As a result, discussion often reinforces initial preferences instead of correcting them. A meta-analysis of hidden-profile studies found that groups discussed substantially more common than unique information and were significantly less likely to identify the correct solution under hidden- than full-information conditions 14. Effective hidden-profile reasoning therefore requires more than allowing participants to speak: groups must surface unshared evidence, consider dissenting perspectives, revise premature beliefs, and determine when consensus is sufficiently supported to justify termination 14; 15. Recent work shows that LLM-based groups reproduce many of these collective reasoning failures. HiddenBench (11) formalizes the hidden-profile paradigm for multi-agent LLM evaluation using 65 tasks derived from custom scenarios, prior human studies, and automatically generated decision problems. Each agent receives shared task information together with an asymmetric private clue, and the agents must deliberate before making a collective decision. Across 15 models from four language-model families, HiddenBench finds persistent failures to integrate distributed evidence: agents can collectively possess sufficient information yet still converge on an incorrect answer, and model scale or general reasoning strength does not reliably predict successful information pooling 11. HiddenBench, therefore, exposes a gap not directly addressed by the conventional reasoning benchmarks. The bottleneck is not only whether an LLM can reason from the evidence it observes, but whether a group can control its communication so that the right evidence is elicited, routed, and incorporated before consensus forms. Existing multi-agent LLM methods provide only partial solutions. Debate frameworks improve reasoning through critique and competing arguments 6; 12; 23; role-based frameworks coordinate specialized agents 10; 22; 8; and coordination methods rely on predefined interaction topologies, voting, judges, or aggregation mechanisms 3; 4; 25. These approaches work well when agents share most of the available information. Hidden-profile tasks, however, require adaptive communication control: the system must decide whether to elicit missing evidence, clarify disagreement, challenge unsupported conclusions, route the discussion, or terminate deliberation. Fixed round-robin protocols (11) cannot adapt to the evolving discussion state, while unconstrained LLM orchestrators may make unreliable control decisions. Moreover, existing debate and orchestration methods generally provide no statistical guarantee that the communication action selected at a given state is appropriate. Such guarantees matter because communication decisions shape the information available to the group: selecting the wrong speaker, repeating shared evidence, or voting too early can reinforce an incorrect consensus, while unnecessary discussion increases cost and risks distraction or context degradation. Communication control should therefore be treated as a sequential decision problem rather than a fixed prompting protocol. At each round, the controller must use the agents’ evolving beliefs and public transcript to select an intervention that improves the collective decision state while avoiding unnecessary communication. Our contributions are: • Closed-loop communication control. We pose hidden-profile reasoning as a sequential communication-control problem and introduce Consilience, a framework that uses the evolving discussion state to select, conformally certify, and route communication interventions, enabling closed-loop coordination among LLM agents. • Conformal reliability. We propose a round-wise conformal calibration framework that certifies communication actions using one-step regret, providing finite-sample, distribution-free regret guarantees and replacing inadmissible proposals with conformally certified alternatives. • Comprehensive evaluation. We evaluate Consilience across 12 open- and closed-weight language models on HiddenBench 11 and an LLM-generated benchmark based on the GroupTravelBench task design 5. We compare against hidden-information, round-robin, and full-information baselines, while analyzing communication efficiency, conformal calibration, speaker routing, controller generalization, heterogeneous agents, and key ablations. Figure 1: Proposed framework Consilience: (1) Each agent receives shared task information and a private clue. (2) Agents report beliefs conditioned on the public transcript, which Consilience aggregates with transcript-level signals into the collective state sts_t. (3) A controller proposes a communication action, and round-wise conformal acceptance retains or replaces it using calibrated one-step regret. (4) An action-conditional router selects the speaker, whose message updates the transcript and agent beliefs. The loop repeats until termination, followed by plurality voting with confidence-based tie-breaking. 2 Problem Formulation We consider hidden-profile multi-agent decision tasks in which the information needed to identify the correct answer is distributed across multiple agents. A task is a tuple (x,,Ishared,Iii=1N)(x,Y,I_shared,\I_i\_i=1^N), where x is the task description, =y1,…,yKY=\y_1,…,y_K\ is a finite answer set, IsharedI_shared is visible to every agent, and IiI_i is visible only to agent i. Thus, agent i initially observes (x,,Ishared,Ii)(x,Y,I_shared,I_i), but not IjI_j for j≠ij≠ i. The agents communicate through a shared public transcript ℋt=(m1,…,mt)H_t=(m_1,…,m_t), with ℋ0=∅H_0= . Private evidence becomes available to the group only when an agent communicates it on the public transcript. The goal is to produce a collective answer y^∈ y while eliciting and integrating the relevant private evidence with limited communication. Consilience treats this process as closed-loop communication control: the controller chooses what kind of action intervention is needed, and a separate action-conditional router chooses which agent should carry it out. Neither the controller nor the router answers the task itself or generates evidence on an agent’s behalf. The final answer is produced from the quorum among the LLM agents from their beliefs formed from the information on the public transcript. 3 Consilience: Conformally Calibrated Communication for Multi-Agent Systems 3.1 Method Overview Figure 1 summarizes the Consilience loop. Design principle. Let Y∈Y denote the latent correct answer. An ideal controller would choose the intervention that maximizes the expected quality of the next collective belief while penalizing communication: at⋆=argmaxa∈[U(bt+1)−λC(ℋt+1)∣st,a],a_t = _a E\! [U(b_t+1)-λ C(H_t+1) s_t,a ], (1) where bt+1b_t+1 is the collective posterior induced after the next message, U(b)=−H(b)U(b)=-H(b) is negative posterior entropy, and C(ℋ)C(H) is communication cost. Exact optimization of Equation (1) is intractable because each action can induce many possible language-model responses and future discussion trajectories. Consilience, therefore, uses an observable collective state and a tractable one-step objective to propose an intervention, round-wise conformal acceptance to screen that proposal, and an action-conditional router to select its speaker. The resulting message updates the public transcript and agent beliefs, closing the loop. The following subsections formalize state construction, action proposal, conformal acceptance, speaker routing, and termination. 3.1.1 Stage 1: Belief Elicitation and State Construction After observing ℋtH_t, each agent reports pi(t)=[pi(t)(y1),…,pi(t)(yK)],∑k=1Kpi(t)(yk)=1.p_i^(t)= [p_i^(t)(y_1),…,p_i^(t)(y_K) ], _k=1^Kp_i^(t)(y_k)=1. This is the agent’s reported belief conditional on (x,Ishared,Ii,ℋt)(x,I_shared,I_i,H_t); it is not assumed to be a calibrated Bayesian posterior. The aggregate belief is the arithmetic mean p¯(t)=1N∑i=1Npi(t). p^(t)= 1N _i=1^Np_i^(t). (2) Consilience represents the current discussion by st=[Ht,Dt,Rt,Gt,Ct,Pt],s_t= [H_t,D_t,R_t,G_t,C_t,P_t ], (3) whose components are defined as: Group uncertainty (HtH_t) We use the normalized entropy of the aggregate belief: Ht=−∑k=1Kp¯(t)(yk)logp¯(t)(yk)logK.H_t=- _k=1^K p^(t)(y_k) p^(t)(y_k) K. Thus, larger values of HtH_t indicate a less decisive group belief. Inter-agent disagreement (DtD_t) Aggregate confidence can conceal incompatible individual beliefs. We measure disagreement by Dt=1N∑i=1NJSD(pi(t)∥p¯(t)),D_t= 1N _i=1^NJSD\! (p_i^(t) p^(t) ), where JSDJSD is the Jensen–Shannon divergence 13. A large DtD_t indicates that some agents interpret the available evidence differently from the group. Message redundancy (RtR_t) and evidence gain (GtG_t) Let Rt=ϕR(mt,ℋt−1)andGt=ϕG(mt,ℋt−1,),R_t= _R(m_t,H_t-1) G_t= _G(m_t,H_t-1,Y), where the fixed estimators ϕR,ϕG∈[0,1] _R, _G∈[0,1] are applied identically across controllers. The redundancy estimator ϕR _R combines lexical overlap with an LLM assessment of whether mtm_t repeats previously public information. The evidence-gain estimator ϕG _G assesses whether mtm_t contributes a previously unshared, decision-relevant fact that supports or eliminates candidate answers. Higher RtR_t indicates more repetition, whereas higher GtG_t indicates a more informative contribution. We set R0=G0=0R_0=G_0=0. Communication cost (Ct)(C_t) We use normalized transcript length: Ct=Words(ℋt)1000.C_t= Words(H_t)1000. Premature consensus (OPENPt)P_t) Let pt⋆=maxy∈p¯(t)(y)p_t = _y p^(t)(y). We distinguish supported agreement from early anchoring as Pt=pt⋆(1−Gt)[pt⋆>0.70,Gt<0.30,t<K+2].P_t=p_t (1-G_t)\,1\! [p_t >0.70,\;G_t<0.30,\;t<K+2 ]. The score is nonzero when the group becomes highly confident early in the discussion despite weak recent evidence. Surrogate discussion loss. The state variables define J(st)=αHHt+αDDt+αRRt−αGGt+αCCt+αPPt.J(s_t)= _HH_t+ _DD_t+ _RR_t- _GG_t+ _CC_t+ _PP_t. Lower values are preferred. The coefficients are fixed design hyperparameters, selected before evaluation and held constant across controller variants and test tasks. The objective is a local control signal, not a substitute for answer correctness. To summarize, Stage 1: the current transcript and private agent contexts produce the observable state sts_t and its scalar discussion loss J(st)J(s_t). Stage 2 uses these to propose the next intervention. 3.1.2 Stage 2: Communication-Action Proposal Communication-action space Consilience uses four communication interventions: comm=Challenge,Clarify,SeekEvidence,Route.A_comm=\ Challenge, Clarify, SeekEvidence, Route\. Controllers that can terminate adaptively additionally include Vote. All controller variants use the same action definitions and message templates. • Challenge asks the speaker to examine the current leading answer, identify unsupported assumptions, and provide potentially contradictory evidence. • Clarify asks an agent with a divergent belief to explain the private evidence/interpretation for that disagreement. • SeekEvidence requests one previously unshared, decision-relevant fact and an explanation of how it supports or eliminates candidate answers. • Route is the evidence-sharing intervention. It asks an under-participating agent for concise discriminative facts. • Vote, when available, terminates deliberation and invokes the final voting rule. Controller variants. Every controller maps sts_t from Stage 1 to a proposal action at,0=πm(st)a_t,0= _m(s_t). The routing and message-generation procedures remain fixed, so controller comparisons isolate action proposal. Rule-based controller The interpretable controller applies fixed thresholds to the state variables. It selects Challenge when confidence is high but recent evidence is weak, Clarify when disagreement is high, SeekEvidence when communication is redundant or uninformative, and Route otherwise. It permits voting only after a minimum number of rounds, when the leading belief is sufficiently strong, disagreement is low, and premature consensus is absent. Learned controller Exhaustive training rollouts evaluate every admissible action from each encountered state using the Stage-3 counterfactual reward and record the vector [rt(st,a1),…,rt(st,a|comm|)]. [r_t(s_t,a_1),…,r_t(s_t,a_|A_comm|) ]. A multilayer perceptron fθ:ℝ6→ℝ|comm|f_θ:R^6 ^|A_comm| with one 12-unit ReLU hidden layer is trained to predict this vector. At inference time, it proposes πMLP(st)=argmaxa∈commfθ(st)a. _MLP(s_t)= _a _commf_θ(s_t)_a. LLM-based controller The LLM controller receives the task, candidate answers, public transcript, state vector, agent beliefs, and speaker counts, and selects one action from the admissible action space. A second variant includes Vote, enabling adaptive termination. Speaker selection remains separate. Regardless of controller type, Stage 2 outputs only the proposed action at,0a_t,0. Stage 3 evaluates that proposal against the other admissible actions and determines the action ata_t that may actually be executed. 3.1.3 Stage 3: Counterfactual Evaluation and Conformal Acceptance Conformal prediction 20 provide distribution-free, finite-sample marginal coverage for exchangeable non-conformity scores on calibration datapoints (Q1,…,Qn)(Q_1,…,Q_n) and the test input QtestQ_test. Specifically, if Q1,…,Qn,QtestQ_1,…,Q_n,Q_test are exchangeable and are produced by the same fixed scoring procedure, then, for k=⌈(n+1)(1−α)⌉k= (n+1)(1-α) , and ϵε as the k-th smallest calibration score, we have the following probabilistic guarantees: ℙ(Qtest≤ϵ)≥1−α.P\! (Q_test≤ε )≥ 1-α. Counterfactual action outcomes For a nonterminal action a applied at state sts_t, the fixed Stage-4 router selects a speaker, the speaker generates a message, all agents update their beliefs, and Consilience obtains a candidate next state st+1as_t+1^a. All candidate actions are evaluated independently from the same frozen pre-action state; only the finally accepted branch is committed to the public transcript. The one-step improvement is rt(st,a)=J(st)−J(st+1a).r_t(s_t,a)=J(s_t)-J(s_t+1^a). We propose one-step regret as the non-conformity score: ℛt(st,a)=maxa′∈tm(st)rt(st,a′)−rt(st,a) _t(s_t,a)= _a _t^m(s_t)r_t(s_t,a )-r_t(s_t,a) =J(st+1a)−mina′∈tm(st)J(st+1a′), =J(s_t+1^a)- _a _t^m(s_t)J(s_t+1^a ), where tm(st)A_t^m(s_t) is the set of actions admissible for controller m at round t. Hence ℛt(st,a)≥0R_t(s_t,a)≥ 0, and at least one admissible action has regret zero. Terminal actions, when available, are evaluated using the same fixed terminal-state scoring rule in calibration and testing. Offline round-wise calibration For a fixed controller method m, define the non-conformity score of its proposal at round t as Qtm(s)=ℛt(s,πm(s)).Q_t^m(s)=R_t\! (s, _m(s) ). (4) Calibration uses complete trajectories generated by the fixed controller, routing policy, model, prompts, and score-generating procedure. Because some trajectories terminate early, only calibration trajectories that reach round t contribute to the round-t multiset calm,(t)=Qtm(stτ):τ∈cal reaches round t.Q_cal^m,(t)= \Q_t^m(s_t^τ):τ _cal reaches round t \. Let nt=|calm,(t)|n_t=|Q_cal^m,(t)| and kt=⌈(nt+1)(1−α)⌉.k_t= (n_t+1)(1-α) . The threshold ϵtm _t^m is the ktk_t-th smallest calibration score. Online conformal acceptance At test time, non-conformity score (or regret) for the controller’s proposed action is computed, and if it is within the conformal threshold ϵtm _t^m, then it is passed to the router for stage 4. Otherwise, the near-optimal conformal action set Γtm(s) _t^m(s) is generated by executing all possible actions: Γtm(s)=a∈tm(s):ℛt(s,a)≤ϵtm. _t^m(s)= \a _t^m(s):R_t(s,a)≤ _t^m \. (5) Here, Consilience applies a fixed fallback SelectSelect, which returns a minimum-regret member of Γtm(s) _t^m(s), with ties resolved by a fixed action ordering: Select(Γtm(s))∈argmina∈Γtm(s)ℛt(s,a).Select( _t^m(s))∈ _a∈ _t^m(s)R_t(s,a). (6) The set is nonempty because a minimum-regret action has regret zero and ϵtm≥0 _t^m≥ 0. Constructing either the calibration score in Equation (4) or the test-time set in Equation (5) requires evaluating every admissible action from the same state. These evaluations are counterfactual branches: they are used to calculate regret, and only the accepted branch updates the actual transcript. Stage 3 therefore maps the proposal at,0a_t,0 to an accepted action ata_t. The action has not yet changed the real transcript: Stage 4 first chooses the agent who will execute it. 3.1.4 Stage 4: Action-Conditioned Routing and Transcript Update After conformal acceptance chooses ata_t, the deterministic router selects speaker it=ρ(at,st)i_t=ρ(a_t,s_t). For disagreement-focused actions, ρ(at,st) ρ(a_t,s_t) =argmaxiJSD(pi(t)∥p¯(t)), = _iJSD\! (p_i^(t) p^(t) ), (7) at a_t ∈Challenge,Clarify. ∈\ Challenge, Clarify\. The most disagreeing agent is the most likely to hold evidence or an interpretation not yet incorporated into the group belief. For evidence-acquisition actions, ρ(at,st)=argminini(t),at∈SeekEvidence,Route,ρ(a_t,s_t)= _in_i^(t),\\ a_t∈\ SeekEvidence, Route\, (8) where ni(t)n_i^(t) is the number of messages previously contributed by agent i. The least-heard agent is used as a proxy for the agent most likely to retain unshared evidence. Ties in Equations (7)–(8) are resolved by a fixed agent ordering. The executed control is therefore ut=(at,it),u_t=(a_t,i_t), where conformal communication control determines ata_t and the router determines iti_t. The selected agent generates one action-conditioned message mt+1∼LLMi(x,,Ishared,Ii,ℋt,at),m_t+1 _i\! (x,Y,I_shared,I_i,H_t,a_t ), which is appended to the public transcript. All agents subsequently report pi(t+1)p_i^(t+1), and Consilience recomputes st+1s_t+1. Thus, if no stopping condition is met, Stage 4 closes the feedback loop by returning the updated transcript to Stage 1. 3.1.5 Stage 5: Termination and Collective Prediction Deliberation terminates when Vote is selected by a controller that admits it, a fixed early-stopping condition is met, or the maximum number of rounds T is reached. Each agent then independently reports an answer y^i∈ y_i and confidence ci∈[0,1]c_i∈[0,1]. We define N(y)=∑i=1N[y^i=y], andN(y)= _i=1^N1[ y_i=y], and tie=y:N(y)=maxy′∈N(y′).Y_tie= \y:N(y)= _y N(y ) \. tieY_tie contains all the plurality winners. The final prediction is y^final=argmaxy∈tie∑i:y^i=yci. y_final= _y _tie _i: y_i=yc_i. 3.2 End-to-End Procedure and Theoretical Guarantee The five stages above specify one online control round: (ℋt,Iii=1N)⏟current information→Stage 1st→Stage 2at,0→Stage 3at→Stage 4ℋt+1, (H_t,\I_i\_i=1^N)_current information Stage 1s_t Stage 2a_t,0 Stage 3a_t Stage 4H_t+1, followed by either another round or the Stage-5 prediction. The calibration phase precedes online deliberation and supplies the round-specific thresholds used in Stage 3. Round-wise marginal guarantee. Here, we formalize the statistical guarantee for the Stage-3 acceptance rule based on the theory of conformal prediction framework. Proposition 1 specializes this construction to the round-specific one-step regret scores of a fixed Consilience controller and gives the sharper finite-sample result for almost surely distinct scores. Proposition 1 (Round-Wise Marginal Regret Coverage). Fix a controller m and round t. Let ntn_t calibration trajectories reach round t, with states s1,…,snts_1,…,s_n_t, and let stests_test be the round-t state of a new trajectory conditional on that trajectory reaching round t. Assume that, conditional on the realized calibration-cohort size and on the test trajectory reaching round t, Qtm(s1),…,Qtm(snt),Qtm(stest)Q_t^m(s_1),…,Q_t^m(s_n_t),Q_t^m(s_test) are exchangeable, and that πm _m and all components of the score-generating procedure were fixed independently of the calibration sample. With ktk_t and ϵtm _t^m defined in Stage 3, ℙ(Qtm(stest)≤ϵtm|test reaches t,Nt=nt)≥1−α.P\! (Q_t^m(s_test)≤ _t^m\; |\;T_test reaches t,\,N_t=n_t )≥ 1-α. (9) Equivalently, ℙ(πm(stest)∈Γtm(stest)|test reaches t,Nt=nt)≥1−α.P\! ( _m(s_test)∈ _t^m(s_test)\; |\;T_test reaches t,\,N_t=n_t )≥ 1-α. If kt≤ntk_t≤ n_t and the pooled calibration and test scores are almost surely distinct, the probability in Equation (9) equals kt/(nt+1)k_t/(n_t+1), and therefore lies in [1−α, 1−α+1nt+1). [1-α,\;1-α+ 1n_t+1 ). The interpretation and scope of Proposition 1 are in the Appendix. 4 Experiments Table 1: Task and vote accuracy on HiddenBench 11 across the evaluated language models after conformal acceptance. Each entry reports task accuracy / vote accuracy. Hidden Pre, Hidden Post, and Full Info are reproduced from the baseline. Higher values are better. The best values are shown in bold. Model Hidden Pre Hidden Post Full Info Rules (Ours) MLP (Ours) LLM (Ours) LLM+Vote (Ours) Qwen3-0.6B .159/.202 .185/.203 .359/.364 .289/.309 .267/.280 .333/.331 .222/.234 Qwen3-1.7B .062/.095 .092/.092 .267/.285 .644/.629 .578/.583 .556/.571 .556/.560 Qwen3-4B .092/.095 .097/.124 .369/.406 .733/.743 .778/.800 .778/.783 .844/.846 Qwen3-8B .062/.126 .149/.175 .631/.596 .911/.926 .867/.880 .911/.903 .867/.880 Qwen3-14B .123/.161 .267/.266 .774/.752 .889/.891 .778/.794 .867/.863 .933/.931 Qwen3-32B .108/.141 .282/.287 .826/.817 .933/.937 .889/.897 .889/.886 .867/.869 Qwen2.5-32B .046/.087 .297/.290 .908/.881 .911/.897 .867/.863 .956/.960 .933/.914 Llama-3.1-8B .077/.177 .354/.354 .610/.599 .800/.743 .822/.811 .711/.680 .622/.623 Phi-4 .077/.150 .195/.221 .877/.819 .933/.914 .822/.823 .911/.897 .911/.914 Mistral-24B .072/.170 .467/.485 .908/.859 .956/.943 .867/.863 .889/.880 .867/.891 DeepSeek-v4-Flash .092/.185 .384/.395 .907/.881 .956/.947 .844/.840 .867/.863 .955/.931 DeepSeek-v4-Pro .077/.158 .431/.474 .954/.933 1.00/1.00 .867/.874 .956/.954 1.00/1.00 Table 2: Task accuracy and vote accuracy for the benchmark generated from GroupTravelBench 5 averaged across 12 models. Method Task Accuracy Vote Accuracy Hidden Pre 0.2537 0.2475 Hidden Post 0.4982 0.4925 Full Info 0.3657 0.3358 Rules (Ours) 0.6544 0.6542 MLP (Ours) 0.5071 0.5025 LLM (Ours) 0.6150 0.6125 LLM+Vote (Ours) 0.6682 0.6642 We evaluate Consilience through four research questions: RQ1 whether adaptive communication improves accuracy over no-discussion and round-robin baselines; RQ2 whether distributed deliberation can match full-information reasoning; RQ3 how controller choice, communication actions, routing, and termination affect performance; and RQ4 whether learned coordination generalizes to unseen tasks and heterogeneous agent groups. Benchmarks and models. We evaluate Consilience on the 65 hidden-profile decision problems in HiddenBench (11) and on a separately generated benchmark following the task structure of GroupTravelBench (5). Each task distributes the evidence required for the correct answer across agents, which communicate only through the public transcript. Because the original GroupTravelBench data are unavailable, our generated benchmark is not a reproduction and its results are not directly comparable; generation and validation details are in the Appendix. We test 12 instruction-tuned open-weight models spanning multiple families and scales 19; 2; 1; 24; 9; model and serving details, together with details on supplementary closed-model experiments, are also provided in the Appendix. 4.1 Compared Discussion Protocols We compare four Consilience controllers (Rules, MLP, LLM, and LLM+Vote) with three references: Hidden-Pre, in which agents vote without communicating; Hidden-Post, which uses fixed round-robin discussion; and Full-Info, in which agents receive all evidence but do not discuss. Full-Info is an information-rich empirical reference, not a theoretical upper bound. 4.2 Evaluation Metrics We report two complementary evaluation metrics computed from the agents’ final votes. Vote accuracy. Vote accuracy measures the proportion of individual agent votes that match the ground-truth answer: VoteAcc=∑d∈∑i=1Nd[vd,i=yd]∑d∈Nd,VoteAcc= _d _i=1^N_dI[v_d,i=y_d] _d N_d, where D is the evaluation set, NdN_d is the number of agents for task d, vd,iv_d,i is the final vote of agent i, and ydy_d is the correct answer. Task accuracy. Task accuracy measures whether the collective decision is correct. For each task, the final prediction is obtained by plurality voting, y^d=argmaxa∈d∑i=1Nd[vd,i=a], y_d= _a _d _i=1^N_dI[v_d,i=a], with ties broken using the summed confidence of supporting agents when available, and otherwise by the benchmark’s deterministic ordering. Task accuracy is then TaskAcc=1||∑d∈[y^d=yd].TaskAcc= 1|D| _d I[ y_d=y_d]. Task accuracy is our primary evaluation metric, while vote accuracy provides a complementary measure of how consistently individual agents converge to the correct answer. 5 Results and Discussion We provide the results of two benchmarks along with their analysis in this section. Detailed results are in the appendix. 5.1 Overall Performance Table 1 reports the accuracy of each discussion protocol across the evaluated models. The hidden pre-discussion condition provides a lower-information baseline in which agents cannot exchange their private evidence. Fixed round-robin discussion improves information availability, but does not explicitly control which evidence should be requested, which agent should provide it, or when the discussion should terminate. Across the evaluated models, Consilience consistently improves over these uncontrolled hidden-information protocols. The strongest controller obtains an average accuracy of 0.83, compared with 0.26 for fixed round-robin discussion, corresponding to an absolute improvement of 0.57. This result supports RQ1: the observed gains arise from adapting the communication intervention to the evolving collective state rather than from allowing agents to communicate. 5.2 Can Communication Compensate for Distributed Information? We next compare controlled hidden-information deliberation with the full-information, no-discussion condition. This comparison separates two potential sources of performance: access to evidence and the ability to integrate that evidence effectively. The best Consilience policy reaches or exceeds the full-information condition on 11 of the 12 models. Averaged across models, its accuracy bettered the full-information condition by 0.13 . In almost all cases, controlled deliberation performs better despite no individual agent having direct access to all hidden facts. This result should not be interpreted as exceeding a theoretical upper bound. The full-information condition is an information-rich empirical baseline: language models may still overlook evidence, confuse relationships among facts, or commit prematurely to an answer. The result instead suggests that staged evidence sharing and explicit disagreement resolution can facilitate evidence integration more effectively than presenting all facts in a single context. 5.3 Effect of Controller Design All controller variants share the same actions, routing rules, agents, and prompts, differing only in action selection. The rule-based controller achieves the highest average accuracy 0.83, followed by the LLM 0.80, LLM+Vote 0.80, and MLP 0.77, showing that the collective-state variables support effective coordination even without learned control. The two LLM controllers behave differently across model scales as seen in Tab 1: adaptive voting benefits smaller models by stopping before repeated or misleading discussion causes conversational drift, whereas larger models benefit from longer deliberation, which helps uncover additional evidence without degrading context. 5.4 Generalization to Unseen Tasks To evaluate held-out generalization, we train the MLP controller on five-task subsets and test it on the remaining tasks; the all-task MLP is retained only as an in-domain oracle-imitation diagnostic. The unseen-task MLP achieves 0.73 average accuracy, compared with 0.77 for the all-task model, indicating that most of the learned coordination behavior transfers to unseen discussion states. This suggests that the compact state captures recurring conditions such as disagreement, redundancy, and unsupported consensus rather than task-specific content. The remaining gap may reflect task-distribution shift, variation in answer-set size, or imperfect counterfactual reward targets. Full training and evaluation details are provided in the appendix. 5.5 Ablation Studies We conduct five ablations that independently examine: (1) random sampling from conformal prediction sets, (2) action-conditional speaker routing, (3) the contribution of each communication action, (4) heterogeneous agent compositions, and (5) controller-state features. In each case, all other experimental components remain fixed. All detailed results are provided in appendix. 5.5.1 Conformal Prediction-Set Sampling tests whether conformally admissible actions also support successful deliberation. At each discussion round, we uniformly sample an action from the conformal prediction set rather than executing the controller’s preferred action, and follow the resulting trajectory until termination. Randomly sampled actions maintain high task success across all controller variants, with only small differences between them. Thus, the calibrated sets generally contain multiple effective actions rather than a single fragile choice, providing useful alternative deliberation paths in addition to their finite-sample statistical guarantee. 5.5.2 Speaker-Routing Ablation compares action-conditional routing with generic policies while holding the discussion state and controller-selected action fixed. We measure average one-step objective improvement and normalized regret relative to the counterfactual optimal speaker. Action-conditional routing achieves the largest mean objective improvement while maintaining low regret. Random, most-disagreeing, and most-spoken routing produce negative mean improvements. Round-robin obtains marginally lower normalized regret, but substantially lower improvement, partly because near-optimal speakers can have similar outcomes when the counterfactual performance range is small. These results show that no single generic notion of speaker importance is sufficient. Matching disagreement-oriented actions to dissenting agents and evidence-seeking actions to less-participating agents yields more informative messages without jointly optimizing every action–speaker pair. 5.5.3 Action Leave-One-Out Ablation We remove each communication action for an entire discussion and measure the change in task accuracy. Removing Route causes the largest average drop, highlighting the importance of engaging underrepresented agents, while removing SeekNewEvidence also substantially hurts performance, especially for LLM controllers. Challenge has a modest effect, consistent with its targeted role in preventing premature consensus, whereas ClarifyDisagreement has the smallest effect and can slightly reduce accuracy, suggesting that routing and evidence seeking often resolve disagreement indirectly. 5.5.4 Heterogeneous-Agent Ablation We replace each agent with a different language model while keeping the controller fixed, introducing variation in reasoning ability, calibration, and instruction following. Results shows that Consilience remains effective across heterogeneous compositions. Strong and cross-family agent groups generally benefit most from adaptive coordination, whereas weaker compositions remain limited by their ability to extract evidence. This suggests that Consilience can exploit complementary reasoning styles and primarily relies on observable discussion dynamics rather than model-family-specific interactions. 5.5.5 Controller-State Feature Ablation compares MLP controllers trained with different subsets of the controller-state representation. The complete state achieves the highest task accuracy and lowest discussion objective. Information-related features (redundancy, evidence gain, and premature consensus) produce comparable accuracy, showing that they provide the most direct signals for action selection, but their higher objective indicates less efficient or controlled discussions. Removing communication cost from the combined belief-and-information state reduces accuracy and produces the worst objective. Conversely, belief-only features yield the shortest discussions but the lowest accuracy because uncertainty and disagreement do not reveal whether new evidence has been introduced or whether consensus is sufficiently supported. The feature groups are therefore complementary: information features guide intervention selection, belief features characterize collective reasoning, and cost discourages unnecessarily long discussions. Combining all three provides the most reliable balance between task success and discussion quality. 6 Conclusion We introduced Consilience, a closed-loop framework for adaptively selecting, certifying, and routing communication in hidden-profile multi-agent reasoning. Across two benchmarks and 12 language models, Consilience consistently improves over uncontrolled discussion and often matches or exceeds full-information baselines, demonstrating that structured communication can be as important as information access itself. While our evaluation focuses on hidden-profile tasks, future work will explore broader collaborative reasoning settings. References Abdin et al. (2024) M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905. Cited by: §4. Bai et al. (2023) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: §4. Chan et al. (2023) C. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu Chateval: towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201. Cited by: §1. Chen et al. (2024) W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C. Chan, H. Yu, Y. Lu, Y. Hung, C. Qian, et al. Agentverse: facilitating multi-agent collaboration and exploring emergent behaviors. In International Conference on Learning Representations, Vol. 2024, p. 20094–20136. Cited by: §1. Cheng et al. (2026) X. Cheng, Y. Hu, L. Zheng, Z. Pan, X. Li, and Y. Liu Grouptravelbench: benchmarking llm agents on multi-person travel planning. arXiv preprint arXiv:2605.25200. Cited by: 3rd item, §4, Table 2. Du et al. (2023) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325. Cited by: §1, §1. Gigone and Hastie (1993) D. Gigone and R. Hastie The common knowledge effect: information sharing and group judgment.. Journal of Personality and social Psychology 65 (5), p. 959. Cited by: §1. Hong et al. (2024) S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhou, et al. MetaGPT: meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, Vol. 2024, p. 23247–23275. Cited by: §1. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: §4. Li et al. (2023) G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem Camel: communicative agents for" mind" exploration of large language model society. arXiv preprint arXiv:2303.17760. Cited by: §1, §1. Li et al. (2025) Y. Li, A. Naito, and H. Shirado HiddenBench: assessing collective reasoning in multi-agent llms via hidden profile tasks. arXiv preprint arXiv:2505.11556. Cited by: 3rd item, §1, §1, §4, Table 1. Liang et al. (2024) T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, S. Shi, and Z. Tu Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 conference on empirical methods in natural language processing, p. 17889–17904. Cited by: §1, §1. Lin (2002) J. Lin Divergence measures based on the shannon entropy. IEEE Transactions on Information theory 37 (1), p. 145–151. Cited by: §3.1.1. Lu et al. (2012) L. Lu, Y. C. Yuan, and P. L. McLeod Twenty-five years of hidden profiles in group decision making: a meta-analysis. Personality and Social Psychology Review 16 (1), p. 54–75. Cited by: §1, §1. Schulz-Hardt and Mojzisch (2012) S. Schulz-Hardt and A. Mojzisch How to achieve synergy in group decision making: lessons to be learned from the hidden profile paradigm. European Review of Social Psychology 23 (1), p. 305–343. Cited by: §1, §1. Stasser et al. (1989) G. Stasser, L. A. Taylor, and C. Hanna Information sampling in structured and unstructured discussions of three-and six-person groups.. Journal of personality and social psychology 57 (1), p. 67. Cited by: §1. Stasser and Titus (1985) G. Stasser and W. Titus Pooling of unshared information in group decision making: biased information sampling during discussion.. Journal of personality and social psychology 48 (6), p. 1467. Cited by: §1, §1. Stasser (1988) G. Stasser Computer simulation as a research tool: the discuss model of group decision making. Journal of experimental social psychology 24 (5), p. 393–422. Cited by: §1. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §4. Vovk et al. (2005) V. Vovk, A. Gammerman, and G. Shafer Algorithmic learning in a random world. Springer. Cited by: §3.1.3. Wang et al. (2024) Q. Wang, Z. Wang, Y. Su, H. Tong, and Y. Song Rethinking the bounds of llm reasoning: are multi-agent discussions the key?. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6106–6131. Cited by: §1. Wu et al. (2024) Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. Autogen: enabling next-gen llm applications via multi-agent conversations. In First conference on language modeling, Cited by: §1, §1. Xiong et al. (2023) K. Xiong, X. Ding, Y. Cao, T. Liu, and B. Qin Examining inter-consistency of large language models collaboration: an in-depth analysis via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 7572–7590. Cited by: §1, §1. Xu et al. (2026) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §4. Zhu et al. (2025) K. Zhu, H. Du, Z. Hong, X. Yang, S. Guo, D. Z. Wang, Z. Wang, C. Qian, R. Tang, H. Ji, et al. Multiagentbench: evaluating the collaboration and competition of llm agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8580–8622. Cited by: §1. Appendix Supplementary material: a worked hidden-profile example, the interpretation and full proof of Proposition 1, expanded experimental results, ablations, and the complete prompt templates. We first provide an example of this failure mode, and how Consilience corrects it by surfacing complementary private evidence, as illustrated in Figure 2. Then we clarify the interpretation, assumptions, and scope of the conformal coverage guarantee in Proposition 1, and provide its complete proof. We then present additional experimental details and expanded non-conformal results on HiddenBench and the generated travel-planning benchmark, followed by analyses of task-level generalization and heterogeneous-agent groups. Next, we report ablations of the collective-state features, communication actions, speaker routing strategy, and conformal action-sampling procedure. Finally, we provide the complete prompt templates used for belief estimation, message generation, controller action selection, and message-level evaluation, together with the generation, validation, repair, and semantic-review prompts used to construct the synthetic benchmark. Figure 2: Three agents choose between Apartments A and B. Shared evidence initially favors Apartment A, but complementary private clues collectively identify Apartment B as the correct choice. Round-robin discussion repeats common evidence further suppresses support for B (as Agent 1 has no clue about any utility surcharges for Apartment B in Round 1), and the final decision converges on A. Consilience instead adaptively selects interventions and speakers to surface unshared evidence, reverse premature consensus, and recover B. Appendix A Interpretation, Scope, and Proof of Proposition 1 Coverage of the controller proposal. Proposition 1 provides a round-wise marginal coverage statement for the one-step regret of the controller’s proposed action. Let NtN_t denote the random number of calibration trajectories that reach discussion round t, and let ntn_t denote its realized value. For a fixed controller m and round t, the proposition states that Pr(CLOSE \! ( ℛt(sttest,πm(sttest))≤ϵtm _t\! (s_t^test, _m(s_t^test) )≤ _t^m (10) |test reaches round t,Nt=nt)≥1−α. |\;T_test reaches round t,\,N_t=n_t )≥ 1-α. Equivalently, with probability at least 1−α1-α, the controller’s proposal belongs to the round-specific conformal action set: πm(sttest)∈Γtm(sttest). _m(s_t^test)∈ _t^m(s_t^test). (11) The probability in Equation (10) is marginal over the calibration trajectories and the new test trajectory, conditional on the test trajectory reaching round t and on Nt=ntN_t=n_t. It is not, in general, a coverage guarantee conditional on one fixed realized calibration sample. Deterministic property of the executed action. The probabilistic coverage statement concerns the controller’s raw proposal rather than the action ultimately executed by Consilience. The executed action is at=πm(st),ℛt(st,πm(st))≤ϵtm,Select(Γtm(st)),ℛt(st,πm(st))>ϵtm.a_t= cases _m(s_t),&R_t(s_t, _m(s_t))≤ _t^m,\\[3.0pt] Select\! ( _t^m(s_t) ),&R_t(s_t, _m(s_t))> _t^m. cases (12) Provided that the conformal action set is evaluated using all admissible counterfactual branches and that Select(Γtm(st))∈Γtm(st),Select\! ( _t^m(s_t) )∈ _t^m(s_t), the executed action satisfies ℛt(st,at)≤ϵtmR_t(s_t,a_t)≤ _t^m (13) for every evaluated state sts_t. Equation (13) is an algorithmic consequence of the acceptance rule, rather than an additional probabilistic conformal guarantee. If the proposal is accepted, it satisfies the threshold by definition. If it is rejected, the fallback returns an element of Γtm(st) _t^m(s_t), every member of which has regret at most ϵtm _t^m. The conformal set is nonempty because at least one admissible action minimizes the realized counterfactual next-state loss and therefore has zero one-step regret. This deterministic statement assumes that the counterfactual branch used to evaluate ℛt(st,at)R_t(s_t,a_t) is the branch committed to the public trajectory. In particular, the selected speaker message and the resulting belief updates must not be resampled after acceptance. Otherwise, the realized next state may differ from the evaluated counterfactual state, and Equation (13) need not hold for the resampled outcome. Scope of the guarantee. The proposition bounds one-step regret relative to the admissible action producing the lowest realized counterfactual next-state loss under the surrogate discussion objective J. It does not identify a unique ground-truth communication action, guarantee that the surrogate-optimal action improves answer correctness, establish global policy optimality, or guarantee that the final collective answer is correct. The proposition also does not provide simultaneous coverage over all rounds of a discussion trajectory. In particular, the collection of round-wise guarantees Pr(Qtm(sttest)≤ϵtm|test reaches round t,Nt=nt)≥1−α \! (Q_t^m(s_t^test)≤ _t^m\, |\,T_test reaches round t,\,N_t=n_t )≥ 1-α does not imply Pr(Qtm(sttest)≤ϵtm for every reached round t)≥1−α. \! (Q_t^m(s_t^test)≤ _t^m for every reached round t )≥ 1-α. A trajectory-level guarantee would require an additional simultaneous or sequential calibration argument, which is outside the scope of Proposition 1. Round-specific exchangeability. The marginal coverage statement requires exchangeability of the calibration and test proposal scores at the round under consideration. More precisely, conditional on Nt=ntN_t=n_t and on the test trajectory reaching round t, the scores Qtm(st1),…,Qtm(stnt),Qtm(sttest)Q_t^m(s_t^1),…,Q_t^m(s_t^n_t),Q_t^m(s_t^test) must be exchangeable. This condition holds under the proposition’s assumptions when calibration and test trajectories are generated by the same fixed, policy-induced trajectory distribution and when the controller and all components of the score-generating procedure are fixed independently of the calibration sample. Because communication control is sequential, exchangeability at a later round does not follow automatically from exchangeability at the initial round. In particular, if an earlier conformal replacement changes the distribution of later test states relative to the calibration trajectories, exchangeability of the later-round scores is an additional assumption. The guarantee should therefore be understood as conditional on round-specific exchangeability under the deployed policy. Accordingly, calibration is performed separately for every configuration for which coverage is claimed, including the controller, agent model, prompting configuration, admissible action space, routing policy, termination rule, decoding procedure, and score-generating components. Any learned component must be fixed before calibration or trained using data independent of the calibration sample. Finite-sample quantile convention. Let kt=⌈(nt+1)(1−α)⌉.k_t= (n_t+1)(1-α) . When kt≤ntk_t≤ n_t, the threshold ϵtm _t^m is the ktk_t-th smallest calibration score. When kt=nt+1k_t=n_t+1, including the case nt=0n_t=0, we use the standard convention ϵtm=+∞. _t^m=+∞. This convention yields valid but uninformative coverage because every admissible action belongs to the resulting conformal set. When ties occur among the pooled calibration and test scores, the nonrandomized procedure may be conservative. The exact coverage value kt/(nt+1)k_t/(n_t+1) stated in Proposition 1 applies when kt≤ntk_t≤ n_t and the pooled scores are almost surely distinct. Proof of Proposition 1. Proof. Condition on the event ℰt=test reaches round t,Nt=nt.E_t= \T_test reaches round t,\,N_t=n_t \. For notational convenience, define Qi=Qtm(sti),i=1,…,nt,Q_i=Q_t^m(s_t^i), i=1,…,n_t, and Qnt+1=Qtm(sttest).Q_n_t+1=Q_t^m(s_t^test). By assumption, Q1,…,Qnt,Qnt+1Q_1,…,Q_n_t,Q_n_t+1 are exchangeable conditional on ℰtE_t. By the definitions of QtmQ_t^m and Γtm _t^m, πm(sttest)∈Γtm(sttest)⟺Qnt+1≤ϵtm. _m(s_t^test)∈ _t^m(s_t^test) Q_n_t+1≤ _t^m. (14) First suppose that kt=nt+1k_t=n_t+1. By convention, ϵtm=+∞ _t^m=+∞, and hence Pr(Qnt+1≤ϵtm|ℰt)=1=ktnt+1. \! (Q_n_t+1≤ _t^m\, |\,E_t )=1= k_tn_t+1. Now suppose that kt≤ntk_t≤ n_t, so that ϵtm _t^m is the ktk_t-th smallest calibration score. Under conditional exchangeability, the standard split-conformal order-statistic argument gives Pr(Qnt+1≤ϵtm|ℰt)≥ktnt+1. \! (Q_n_t+1≤ _t^m\, |\,E_t )≥ k_tn_t+1. (15) This inequality remains valid in the presence of ties; ties may make the nonrandomized conformal procedure conservative. From the definition of ktk_t, ktnt+1=⌈(nt+1)(1−α)⌉nt+1≥1−α. k_tn_t+1= (n_t+1)(1-α) n_t+1≥ 1-α. (16) Combining this inequality with Equation (15) proves the round-wise lower coverage bound. Equation (14) then gives the equivalent action-set statement. Finally, suppose that kt≤ntk_t≤ n_t and that the nt+1n_t+1 pooled calibration and test scores are almost surely distinct. Conditional exchangeability implies that the rank of the test score among the pooled scores is uniformly distributed over 1,…,nt+1.\1,…,n_t+1\. Under distinctness, Qnt+1≤ϵtmQ_n_t+1≤ _t^m holds if and only if the pooled rank of Qnt+1Q_n_t+1 is at most ktk_t. Consequently, Pr(Qnt+1≤ϵtm|ℰt)=ktnt+1. \! (Q_n_t+1≤ _t^m\, |\,E_t )= k_tn_t+1. (17) Using ⌈x⌉<x+1, x <x+1, we obtain ktnt+1 k_tn_t+1 =⌈(nt+1)(1−α)⌉nt+1 = (n_t+1)(1-α) n_t+1 (18) <1−α+1nt+1. <1-α+ 1n_t+1. (19) Together with the lower bound, this proves 1−α≤ktnt+1<1−α+1nt+1,1-α≤ k_tn_t+1<1-α+ 1n_t+1, completing the proof. ∎ Appendix B Experimental Details Models. We evaluate Consilience on 13 instruction-tuned open-weight language models spanning multiple families and scales, including Qwen3 (0.6B, 1.7B, 4B, 8B, 14B, 32B, and 235B), Qwen2.5-32B, Mistral-24B, Phi-4, Llama-3.1-8B, Llama-3.3-70B, and Gemma-3-27B. We further include GPT-4.1-mini and Claude-3-Haiku in a supplementary closed-model experiment comparing the full-information baseline against LLM-based orchestration with adaptive voting to assess whether the observed trends generalize beyond open-weight models. Unless otherwise specified, all agents within a discussion instantiate the same underlying language model. For LLM-based controller, the same model additionally serves as the controller responsible for selecting communication actions, whereas the learned-controller variants replace this decision process with a lightweight MLP while retaining the evaluated LLM for belief estimation, message generation, transcript analysis, and final voting. Appendix C Non-Conformal Results Table 3: Task and vote accuracy for Hiddenbench across the evaluated language models. Each entry reports task accuracy / vote accuracy. Higher values are better. Model Hidden Pre Hidden Post Full Info Rules MLP LLM LLM+Vote Qwen3-0.6B .159/.202 .185/.203 .359/.364 .374/.391 .287/.295 .313/.306 .297/.314 Qwen3-1.7B .062/.095 .092/.092 .267/.285 .559/.563 .600/.599 .554/.561 .564/.561 Qwen3-4B .092/.095 .097/.124 .369/.406 .826/.829 .831/.829 .826/.831 .800/.789 Qwen3-8B .062/.126 .149/.175 .631/.596 .908/.913 .856/.871 .877/.879 .805/.808 Qwen3-14B .123/.161 .267/.266 .774/.752 .872/.879 .774/.783 .821/.841 .928/.928 Qwen3-32B .108/.141 .282/.287 .826/.817 .918/.900 .841/.856 .913/.903 .923/.918 Qwen3-235B .062/.123 .292/.306 .862/.841 .928/.943 .856/.846 .887/.879 .836/.848 Qwen2.5-32B .046/.087 .297/.290 .908/.881 .897/.903 .867/.874 .923/.926 .918/.924 Llama-3.1-8B .077/.177 .354/.354 .610/.599 .733/.682 .667/.655 .595/.572 .600/.590 Llama-3.3-70B .046/.123 .436/.460 .862/.877 .928/.935 .928/.926 .949/.953 .908/.914 Phi-4 .077/.150 .195/.221 .877/.819 .923/.924 .826/.831 .903/.895 .944/.937 Mistral-24B .072/.170 .467/.485 .908/.859 .923/.924 .841/.848 .903/.914 .923/.925 Gemma-3-27B .097/.184 .349/.352 .846/.821 .903/.883 .872/.863 .856/.848 .897/.904 DeepSeek-V4-Flash .092/.185 .384/.395 .907/.881 .953/.957 .784/.779 .953/.937 .908/.889 DeepSeek-V4-Pro .077/.158 .431/.474 .954/.933 1.00/1.00 .939/.921 .969/.968 .969/.972 Table 4: Task and vote accuracy for the generated benchmark across the evaluated language models. Each entry reports task accuracy / vote accuracy. Higher values are better. Model Hidden Pre Hidden Post Full Info Rules MLP LLM LLM+Vote Qwen3-0.6B .228/.240 .236/.240 .204/.230 .228/.230 .220/.220 .209/.210 .248/.240 Qwen3-1.7B .204/.210 .272/.270 .248/.240 .312/.310 .264/.260 .252/.250 .264/.260 Qwen3-4B .224/.230 .292/.290 .292/.270 .456/.440 .428/.430 .480/.480 .476/.470 Llama-3.1-8B .224/.230 .312/.310 .280/.270 .509/.530 .384/.370 .524/.520 .504/.490 Qwen3-8B .248/.230 .324/.300 .288/.270 .684/.690 .528/.540 .652/.650 .716/.710 Qwen3-14B .256/.240 .400/.400 .336/.310 .696/.690 .544/.540 .684/.690 .744/.750 Mistral-24B .292/.280 .576/.570 .432/.370 .872/.870 .496/.480 .624/.620 .832/.840 Gemma-3-27B .244/.250 .730/.720 .444/.400 .806/.800 .549/.560 .750/.740 .858/.850 Qwen3-32B .256/.240 .668/.660 .376/.360 .660/.650 .384/.370 .592/.580 .716/.710 Qwen2.5-32B .284/.280 .684/.670 .496/.440 .884/.890 .752/.740 .880/.880 .900/.900 Llama-3.3-70B .276/.270 .728/.720 .520/.440 .889/.890 .848/.830 .924/.920 .912/.910 Qwen3-235B .308/.270 .756/.760 .472/.430 .857/.860 .688/.690 .809/.810 .848/.840 HiddenBench. Table 3 shows that access to distributed evidence alone is insufficient without effective coordination. Averaged across the 15 models, hidden pre-discussion achieves only 0.0830.083 task accuracy, while uncontrolled round-robin discussion improves this to 0.2850.285. Providing all evidence directly increases average accuracy to 0.7310.731, confirming that the main difficulty arises from recovering and integrating information distributed across agents. All four Consilience controllers exceed the full-information reference on average: Rules achieves 0.8430.843 task accuracy, followed by LLM at 0.8160.816, LLM+Vote at 0.8150.815, and MLP at 0.7850.785. The rule-based controller therefore provides the strongest average performance, although the best controller varies across models. In particular, the LLM-based controllers are competitive for stronger models, and adaptive voting produces the best results for models such as Qwen3-14B and Phi-4. Task and vote accuracy are also closely aligned for all controllers, indicating that improvements in the final collective answer generally correspond to broader agreement among the individual agents rather than being produced only by plurality aggregation. Supplementary experiments with frontier API models show the same trend: Consilience with LLM+Vote matches or exceeds full-information reasoning, achieving 0.9380.938 versus 0.8770.877 on openai’s gpt models and 0.8620.862 versus 0.6460.646 on Claude-3-Haiku. Generated benchmark. Table 4 presents a different performance profile. Hidden pre-discussion obtains an average task accuracy of 0.2540.254, while uncontrolled discussion increases it to 0.4980.498. Interestingly, hidden post-discussion outperforms the full-information condition, which achieves only 0.3660.366. This suggests that merely placing all evidence in one context does not guarantee that the model will identify and combine the relevant facts; interaction can help expose, repeat, and reconcile evidence that may otherwise be overlooked. Among the controlled methods, LLM+Vote achieves the highest average task accuracy at 0.6680.668, followed closely by Rules at 0.6540.654, LLM at 0.6150.615, and MLP at 0.5070.507. Relative to uncontrolled discussion, LLM+Vote improves task accuracy by 0.1700.170, whereas the MLP improves it by only 0.0090.009. Adaptive termination is therefore especially useful on this benchmark, where continued discussion may reinforce an incorrect answer after the relevant evidence has already been surfaced. Performance on the generated benchmark also depends strongly on the capability of the participating model. The smallest Qwen3 models receive limited benefit from communication control, whereas Qwen2.5-32B and Llama-3.3-70B exceed 0.880.88 task accuracy under the strongest controllers. The MLP controller is less consistent under this setting, performing substantially below Rules and the LLM-based controllers for models such as Mistral-24B and Qwen3-32B. This pattern suggests that the fixed learned mapping from collective-state features to actions is more sensitive to changes in benchmark structure or discussion dynamics, while an LLM controller can adapt its action choice using the current transcript and beliefs. Comparison across benchmarks. Across the 12 models shared by both tables, controller performance is generally lower on the generated benchmark. Relative to HiddenBench, average task accuracy decreases by 0.1600.160 points for Rules, 0.2610.261 points for MLP, and 0.1700.170 points for LLM, but by only 0.1150.115 points for LLM+Vote. Thus, LLM+Vote exhibits the smallest cross-benchmark degradation, while the MLP exhibits the largest. Together, these results indicate that explicit communication control consistently improves hidden-profile reasoning, but the most effective controller depends on the discussion distribution: simple state-dependent rules are particularly effective on HiddenBench, whereas transcript-aware control with adaptive voting transfers more robustly to the generated benchmark. C.1 Generalization to Held-Out Tasks The standard MLP controller is trained using exhaustive action outcomes collected from all benchmark tasks. It therefore measures how accurately a lightweight learned controller can imitate the one-step action oracle within the task distribution used to generate its supervision. However, because the same tasks contribute to both controller training and downstream evaluation, this condition should be interpreted primarily as an in-domain oracle-imitation diagnostic rather than as a strict test of task-level generalization. We therefore construct a separate held-out evaluation. HiddenBench is partitioned into a five-task training subset and a disjoint 60-task test set. For each training task, the exhaustive action explorer follows the current discussion trajectory and, at every reached state, executes each of the four available communication actions: Challenge, ClarifyDisagreement, SeekNewEvidence, and Route. The resulting training examples pair the six-dimensional collective-state representation with a four-dimensional vector containing the observed reward of every communication action. The MLP is trained only on examples generated from the five training tasks. Its feature-normalization statistics, reward-normalization statistics, and network parameters are therefore estimated exclusively from the training split. After training, the controller is frozen and evaluated on the remaining 60 tasks without any additional fitting or access to their counterfactual action outcomes. At each test-time discussion state, the MLP predicts the expected immediate reward of the four actions and executes the action with the highest predicted value. Agent prompting, speaker routing, belief estimation, and final voting remain unchanged. This evaluation measures whether the mapping learned from abstract collective-state variables to communication actions transfers across decision problems. The controller does not observe task identifiers or raw private clues directly; it acts on quantities such as group-belief entropy, inter-agent disagreement, message redundancy, communication cost, evidence gain, and premature consensus. Consequently, successful transfer would indicate that these variables capture recurring coordination conditions rather than merely memorizing the content or trajectory of individual benchmark tasks. Across the 13 models for which held-out results are available, the held-out MLP obtains an average task accuracy of 0.73. The corresponding all-task MLP average is 0.77. Thus, withholding the evaluation tasks during controller training is associated with a descriptive decrease of approximately 0.04. The relatively small aggregate gap suggests that a substantial portion of the learned action-selection behavior transfers to discussion trajectories that were not observed during training. Transfer is particularly strong for medium and large language models. 8 of the 13 evaluated models achieve at least 0.83 held-out task accuracy. Qwen3-32B, Qwen3-235B, Llama-3.3-70B, and Mistral-24B each reach 0.86 task accuracy, while Qwen3-8B, Qwen3-14B, Qwen2.5-32B, and Gemma-3-27B reach 0.83. These results indicate that the controller trained on only a small subset of tasks can still select useful communication actions when paired with agents capable of producing stable beliefs and informative responses. The largest degradations relative to the all-task MLP occur for Qwen3-1.7B and Phi-4, whose task accuracies decrease by 0.17 and 0.16 respectively. Qwen3-0.6B also decreases by 0.09 points. This pattern suggests that held-out transfer is less reliable when the participating agents produce noisier belief distributions or when their discussion states differ substantially from those represented in the controller-training trajectories. Because the MLP observes only the collective-state summary, changes in the quality or calibration of agent beliefs can produce a state distribution that is difficult to interpret using a controller trained from a small number of tasks. Several models show comparable or slightly higher accuracy under the held-out controller. For example, held-out task accuracy increases by 0.06 points for Qwen3-14B, 0.07 points for Llama-3.1-8B, and 0.03 points for both Qwen3-32B and Mistral-24B. These increases should not be interpreted as evidence that using less training data improves the controller. The two conditions are evaluated over different task sets, and the deliberations themselves involve stochastic language-model outputs. Small positive differences can therefore arise from variation in task difficulty, generated messages, belief estimates, or action trajectories. Task and vote accuracy remain closely aligned under held-out evaluation. Their mean absolute difference across models is approximately 0.02 accuracy points, and for Qwen3-8B, Qwen3-14B, and Llama-3.3-70B the two values are identical. This agreement indicates that held-out performance is generally supported by the individual agents’ final judgments rather than arising only from plurality aggregation or confidence-based tie-breaking. Larger discrepancies for a few models nevertheless show that communication control can affect both the correctness of the group decision and the distribution of support among agents. Overall, the held-out results provide evidence that the controller learns reusable coordination behavior from the collective-state representation. The performance decrease relative to the in-domain MLP is modest on average, and high accuracy is retained across most medium and large models despite training on only five tasks. At the same time, the larger losses for some weaker or behaviorally distinct models show that transfer is not uniform. The learned controller generalizes most reliably when the test-time discussion dynamics resemble the state distributions encountered during training. For a strictly controlled estimate of the generalization gap, the all-task MLP should additionally be evaluated on the same 60 held-out tasks. The current comparison uses the 65-task result for the standard MLP and the 60-task result for the held-out MLP; consequently, the reported 0.03 and 0.04 point differences combine controller-training effects with a small difference in the evaluation task set. Evaluating both checkpoints on the identical 60-task test split would isolate the effect of withholding task-level training supervision. Appendix D Heterogeneous-Agent Analysis Table 5 evaluates whether Consilience remains effective when the participating agents use different underlying language models. Overall, heterogeneous groups can achieve performance comparable to strong homogeneous groups, but their results are substantially more sensitive to model composition, agent assignment, and controller choice. Among the heterogeneous four-agent configurations, the rule-based controller reaches its highest task accuracy of 0.9660.966 on both Position Rot-4 and Ladder L2, while the LLM and LLM+Voting controllers reach 0.9480.948 on Ladder L4. These results show that model heterogeneity does not inherently prevent successful coordination when the group contains sufficiently capable and complementary agents. The position-rotation experiments reveal that assigning the same four models to different agent positions can materially alter performance. For example, rule-based accuracy varies from 0.8790.879 in the documented assignment to 0.9660.966 in Position Rot-4, while LLM+Voting varies from 0.8280.828 to 0.9310.931 across the rotations. Thus, performance depends not only on which models are present, but also on which private evidence each model receives. Using the stronger Qwen3-32B controller further improves the documented heterogeneous group, raising MLP accuracy from 0.7760.776 to 0.9310.931, LLM accuracy from 0.8450.845 to 0.8970.897, and LLM+Voting accuracy from 0.8790.879 to 0.9310.931. This indicates that controller capability can compensate for some of the variability introduced by heterogeneous participants. Group composition also has a clear effect. Configurations composed primarily of capable models, such as Family-Qwen, Spread-Low, and Ladder L4, remain strong across several controllers. In contrast, groups containing several small models perform substantially worse: Star and Ladder L1 obtain only 0.5860.586 task accuracy under LLM+Voting. The Weak-Link configuration shows that introducing a single weak agent does not necessarily cause failure, particularly for Rules, but can reduce the effectiveness of adaptive voting. Among homogeneous groups, Qwen3-32B with LLM+Voting achieves the highest overall task accuracy of 0.9830.983, while the other strong homogeneous groups also remain consistently competitive. Taken together, the results suggest that Consilience generalizes to heterogeneous teams, but reliable performance depends on both the capability distribution within the group and the controller’s ability to route communication across differently capable agents. Table 5: Performance of heterogeneous and homogeneous multi-agent groups on the corresponding HiddenBench agent-count partitions. Four-agent configurations are evaluated on the 58-task four-agent partition, while Documented-3Agent is evaluated on the seven-task three-agent partition. Each controller entry reports task accuracy / vote accuracy. The orchestration controller is Qwen3-8B unless marked with † , where Qwen3-32B is used. Model abbreviations are: Q0.6, Q1.7, Q4, Q8, Q14, and Q32 for Qwen3 at the corresponding parameter scale; Q2.5-32 for Qwen2.5-32B; P4 for Phi-4; M24 for Mistral-24B; G27 for Gemma-3-27B; L8 for Llama-3.1-8B; and L70 for Llama-3.3-70B. Configuration Agent Models Rules MLP LLM LLM+Voting Heterogeneous groups Documented-4Agent Q8, Q32, P4, M24 .879/.888 .776/.780 .845/.866 .879/.888 Documented-3Agent Q8, Q32, P4 .714/.714 .571/.571 .714/.667 .714/.667 Position Rot-2 Q32, P4, M24, Q8 .897/.897 .897/.897 .828/.853 .931/.909 Position Rot-3 P4, M24, Q8, Q32 .914/.931 .897/.888 .879/.875 .897/.914 Position Rot-4 M24, Q8, Q32, P4 .966/.961 .879/.879 .897/.897 .828/.836 Controller-Strong† Q8, Q32, P4, M24 .879/.897 .931/.892 .897/.901 .931/.927 Family-Cross L8, P4, M24, G27 .914/.897 .776/.772 .845/.815 .914/.871 Family-Qwen Q8, Q14, Q32, Q2.5-32 .897/.892 .862/.853 .914/.914 .914/.905 Spread-Low P4, M24, G27, Q2.5-32 .948/.940 .793/.797 .845/.853 .948/.927 Spread-High Q0.6, Q4, Q14, Q32 .793/.746 .845/.810 .741/.724 .724/.685 Weak-Link Q0.6, Q32, P4, M24 .897/.853 .828/.802 .845/.819 .759/.759 Star Q32, Q0.6, Q1.7, Q4 .759/.694 .655/.634 .621/.621 .586/.556 Ladder L1 Q0.6, Q1.7, Q4, Q8 .776/.746 .621/.616 .724/.711 .586/.603 Ladder L2 Q4, Q8, Q14, P4 .966/.966 .897/.888 .845/.853 .879/.871 Ladder L3 Q14, P4, M24, Q32 .897/.918 .828/.828 .793/.780 .879/.858 Ladder L4 Q32, Q2.5-32, G27, L70 .931/.927 .897/.892 .948/.948 .948/.935 Homogeneous groups Qwen3-32B 4×Q32 .948/.927 .862/.862 .897/.905 .983/.957 Mistral-24B 4×M24 .931/.927 .879/.879 .914/.905 .948/.948 Phi-4 4×P4 .914/.918 .879/.879 .931/.931 .879/.871 Qwen3-8B 4×Q8 .948/.944 .810/.841 .914/.922 .810/.819 Appendix E Ablations State-feature ablation. Table 6 evaluates how different subsets of the collective-state representation affect the learned MLP controller. Using the complete feature set achieves the best overall trade-off, with 82.22% task success and the lowest final objective value of 0.0239. The information-only controller matches this success rate, but incurs slightly higher communication cost and a substantially worse final objective, indicating that the remaining state variables improve the quality and efficiency of the learned policy even when they do not change the final task accuracy. Removing information-related features reduces performance. Both the belief-plus-information and belief-plus-cost variants achieve 77.78% success, while the belief-only controller performs worst at 75.56%. Although the belief-only representation produces the lowest communication cost, this reduction is accompanied by a 6.66 percentage-point drop in success relative to the full model. Among the reduced representations, belief plus cost yields a lower final objective than belief plus information, suggesting that explicit cost awareness helps the controller avoid inefficient deliberation. Overall, the results show that belief statistics alone are insufficient, while combining belief, information, and cost-related signals produces the most balanced controller. Table 6: State-feature ablation for Qwen3-8B. Each row retrains the MLP using only the retained feature groups. Higher success is better; lower communication cost and final objective are better. Features retained Success ↑ Communication cost ↓ Final objective ↓ All features 82.22% 0.5228 0.0239 Information only 82.22% 0.5254 0.2277 Belief + information 77.78% 0.5285 0.3509 Belief + cost 77.78% 0.5253 0.1582 Belief only 75.56% 0.4625 0.2212 Action ablation. Table 7 evaluates the contribution of each communication action by removing it while keeping the remaining controller unchanged. Across controllers, routing and new-evidence solicitation are the most consequential components. Removing routing produces the largest average drop, reducing task accuracy by 4.6 percentage points, while removing SeekNewEvidence decreases average accuracy by 3.8 points. These effects are especially pronounced for the LLM+Voting controller, where removing routing and new-evidence solicitation reduces accuracy by 13.8 and 12.3 points, respectively. This suggests that adaptive termination is particularly dependent on directing the discussion toward underrepresented agents and unshared evidence before voting occurs. The effects vary across controller types. The rule-based controller depends most strongly on routing, whereas the standard LLM controller is most affected by removing SeekNewEvidence. Removing Challenge causes a small but consistent degradation for Rules, MLP, and LLM, indicating that explicitly testing the leading hypothesis provides a modest benefit. In contrast, removing ClarifyDisagreement has no average effect, and removing voting leaves the Rules and LLM+Voting results unchanged in this ablation. Several removals improve performance, including the removal of SeekNewEvidence from the MLP controller, showing that individual actions are not uniformly useful and may interact with the controller’s selection policy. Overall, the results indicate that the full action set is most valuable for providing complementary interventions, with routing and new-evidence acquisition contributing the strongest aggregate gains. Table 7: Action ablation study on HiddenBench. Each row removes one communication action from the full controller. Entries report task accuracy (%), with the change relative to the corresponding full action set shown in parentheses. Negative values indicate performance degradation. Configuration Rules MLP LLM LLM+Vote Average Full 92.3 87.7 89.2 83.1 88.1 – Challenge 90.8 (−1.5-1.5) 86.2 (−1.5-1.5) 87.7 (−1.5-1.5) 83.1 (0.0) 86.9 (−1.2-1.2) – Clarify 93.8 (+1.5) 87.7 (0.0) 89.2 (0.0) 81.5 (−1.5-1.5) 88.1 (0.0) – New Evidence 92.3 (0.0) 92.3 (+4.6) 81.5 (−7.7-7.7) 70.8 (−12.3-12.3) 84.2 (−3.8-3.8) – Routing 87.7 (−4.6-4.6) 89.2 (+1.5) 87.7 (−1.5-1.5) 69.2 (−13.8-13.8) 83.5 (−4.6-4.6) – Voting 92.3 (0.0) NA NA 83.1 (0.0) 87.7 (0.0) Speaker-routing ablation. Table 8 isolates the effect of speaker selection by holding the controller-selected communication action fixed. Consilience’s action-conditional router achieves the lowest mean objective change, ΔJ=−0.0079 J=-0.0079, indicating that its selected speakers produce the strongest average immediate improvement in the discussion state. Round-robin is the closest alternative, with ΔJ=−0.0017 J=-0.0017, while random routing slightly worsens the objective on average (ΔJ=0.0145 J=0.0145). The normalized-regret results are less decisive. Round-robin obtains the lowest regret at 0.48390.4839, followed closely by action-conditional routing at 0.48680.4868 and random routing at 0.49300.4930. Thus, although the proposed router performs best under the mean objective-change metric, its advantage over simple round-robin and random selection is small, and it does not achieve the lowest normalized regret. In contrast, selecting the most-disagreeing or most-spoken agent performs substantially worse under both metrics, suggesting that disagreement or participation frequency alone is insufficient for identifying the most useful speaker. Overall, the results support action-aware routing over these stronger heuristic alternatives, but provide only limited evidence of a clear advantage over round-robin routing. Conformal-sampling ablation. Table 9 compares the four controller variants over 400 trials with Qwen3-8B. The LLM and MLP controllers achieve the highest mean success rate, both reaching 82.75%, despite using different action-selection mechanisms. Their discussion lengths are also similar, averaging 9.54 and 9.87 rounds per trial, respectively. This indicates that the lightweight learned controller can match the success of the LLM controller, although it requires slightly more communication. Table 8: Counterfactual speaker-routing ablation with Qwen3-8B. The controller-selected action is held fixed and only the speaker-selection policy is changed. Lower objective change is better, while lower normalized regret indicates routing decisions closer to the counterfactual optimum. Routing policy Mean ΔJ J ↓ Normalized regret ↓ Consilience (action-conditional) -0.0079 0.4868 Random 0.0145 0.4930 Round-robin -0.0017 0.4839 Most-disagreeing 0.1205 0.5735 Most-spoken 0.2815 0.7184 The rule-based controller terminates substantially earlier, after only 3.24 rounds on average, but its success decreases to 64.50%. The LLM+Voting variant also conducts shorter discussions, averaging 5.43 rounds, yet attains the lowest success rate of 63.00%. These results reveal a clear trade-off between communication length and task performance: controllers that allow longer deliberation are substantially more successful, whereas aggressive or adaptive termination can stop the discussion before complementary private evidence has been sufficiently integrated. Overall, conformal sampling preserves comparable performance for the LLM and MLP controllers, while the weaker results of the early-terminating variants suggest that calibrated action selection alone does not compensate for premature voting. Table 9: Comparison of controller methods over 400 trials. Using Qwen3-8B model. Method Mean Success (%) Mean Rounds/Trial LLM 82.75 9.54 MLP 82.75 9.87 Rules 64.50 3.24 LLM+Voting 63.00 5.43 Appendix F Complete Prompt Templates This section reports the complete prompt templates used in our experiments. Text enclosed in angle brackets denotes a runtime placeholder populated separately for each task or deliberation state. ⬇ 1 You are <AGENT_NAME>, one participant in a multi-agent group decision task. 2 3 Task description: 4 <TASK_DESCRIPTION> 5 6 Possible answers: 7 <POSSIBLE_ANSWERS> 8 9 Shared information visible to everyone: 10 - <SHARED_INFORMATION_ITEM_1> 11 - <SHARED_INFORMATION_ITEM_2> 12 - … 13 14 Information visible only to you: 15 - <PRIVATE_INFORMATION_ITEM_1> 16 - <PRIVATE_INFORMATION_ITEM_2> 17 - … 18 19 Rules: 20 - You may use the information visible only to you. 21 - Share decision-relevant evidence when it helps the group. 22 - Do not invent facts. 23 - Be concise. 24 25 Public transcript so far: 26 (empty transcript) 27 28 Give your current belief over the possible answers. Return JSON only: 29 30 "belief": "OPTION": 0.5, 31 "best_answer": "one option exactly as written", 32 "confidence": 0.0, 33 "one_sentence_evidence": "short evidence summary", 34 "needs_more_information": true 35 36 37 Include every possible answer exactly and make the probabilities sum to 1. 38 Possible answers: <POSSIBLE_ANSWERS> ⬇ 1 You are <AGENT_NAME>, one participant in a multi-agent group decision task. 2 3 Task description: 4 <TASK_DESCRIPTION> 5 6 Possible answers: 7 <POSSIBLE_ANSWERS> 8 9 Shared information visible to everyone: 10 - <SHARED_INFORMATION_ITEM_1> 11 - <SHARED_INFORMATION_ITEM_2> 12 - … 13 14 Information visible only to you: 15 - <PRIVATE_INFORMATION_ITEM_1> 16 - <PRIVATE_INFORMATION_ITEM_2> 17 - … 18 19 Rules: 20 - You may use the information visible only to you. 21 - Share decision-relevant evidence when it helps the group. 22 - Do not invent facts. 23 - Be concise. 24 25 Public transcript so far: 26 <PUBLIC_TRANSCRIPT> 27 28 Controller instruction: 29 <ACTION_SPECIFIC_INSTRUCTION> 30 31 Write your next message to the group. Share relevant evidence, clearly state 32 if it rules out an option, communicate naturally, and do not refer to it as 33 private information. ⬇ 1 The current leading option is <LEADING_OPTION>. State any evidence that could 2 support or challenge <LEADING_OPTION>. If your information rules out another 3 option, say so clearly. ⬇ 1 Your belief differs from the group. Explain the evidence behind your view, 2 especially any fact that rules out an option. ⬇ 1 Share one new decision-relevant fact. Focus on facts that eliminate or support an option. Avoid repeating the transcript. ⬇ 1 Contribute one concise piece of decision-relevant evidence. If possible, 2 explain which option it supports or rules out. ⬇ 1 You are the consilience controller. Choose the next controller action for a 2 multi-agent decision task. You must choose exactly one existing action type; do not 3 choose a speaker and do not write the agent’s message. 4 5 Available actions: 6 - challenge: ask an agent to test the current leading option and surface contrary evidence. 7 - clarify_disagreement: ask the agent whose belief differs most from the group to explain why. 8 - seek_new_evidence: ask the least-heard agent for one new decision-relevant fact. 9 - route: ask the least-heard agent for concise evidence that supports or rules out an option. 10 11 Task: 12 <TASK_DESCRIPTION> 13 14 Possible answers: 15 <POSSIBLE_ANSWERS> 16 17 Public transcript so far: 18 <PUBLIC_TRANSCRIPT> 19 20 Current state: 21 <COLLECTIVE_STATE_JSON> 22 23 Current round: 24 <CURRENT_ROUND> 25 26 Current agent beliefs: 27 <AGENT_BELIEFS_JSON> 28 29 Times each agent has spoken: 30 <SPEAKER_COUNTS_JSON> 31 32 Return JSON only: 33 34 "action": "one of: challenge, clarify_disagreement, seek_new_evidence, route", 35 "reason": "brief reason for choosing this action", 36 "expected_effect": "brief description of what this should improve" 37 ⬇ 1 You are the consilience controller. Choose the next controller action for a 2 multi-agent decision task. You must choose exactly one existing action type; do not 3 choose a speaker and do not write the agent’s message. 4 5 Available actions: 6 - challenge: ask an agent to test the current leading option and surface contrary evidence. 7 - clarify_disagreement: ask the agent whose belief differs most from the group to explain why. 8 - seek_new_evidence: ask the least-heard agent for one new decision-relevant fact. 9 - route: ask the least-heard agent for concise evidence that supports or rules out an option. 10 - vote: end the discussion now and collect every agent’s final vote. 11 12 Task: 13 <TASK_DESCRIPTION> 14 15 Possible answers: 16 <POSSIBLE_ANSWERS> 17 18 Public transcript so far: 19 <PUBLIC_TRANSCRIPT> 20 21 Current state: 22 <COLLECTIVE_STATE_JSON> 23 24 Current round: 25 <CURRENT_ROUND> 26 27 Current agent beliefs: 28 <AGENT_BELIEFS_JSON> 29 30 Times each agent has spoken: 31 <SPEAKER_COUNTS_JSON> 32 33 Return JSON only: 34 35 "action": "one of: challenge, clarify_disagreement, seek_new_evidence, route, vote", 36 "reason": "brief reason for choosing this action", 37 "expected_effect": "brief description of what this should improve" 38 ⬇ 1 Analyze a message in a multi-agent decision conversation. 2 3 Options: <POSSIBLE_ANSWERS> 4 Transcript before this message: 5 <PUBLIC_TRANSCRIPT_BEFORE_MESSAGE> 6 7 New message: 8 <NEW_MESSAGE> 9 10 Return JSON only: 11 12 "new_evidence_score": 0.0, 13 "redundancy_score": 0.0, 14 "supports_options": ["option names exactly"], 15 "attacks_or_rules_out_options": ["option names exactly"], 16 "is_clarification": false, 17 "short_explanation": "one sentence" 18 19 20 Use scores between 0 and 1. New evidence adds task-relevant information; redundancy 21 means the message mostly repeats the earlier transcript. F.1 Synthetic Benchmark Generation Prompts The additional travel-planning benchmark was generated using structured prompting followed by deterministic validation, automatic repair, and LLM-based semantic quality review. Text enclosed in angle brackets denotes a value populated at runtime. ⬇ 1 You design adversarial, realistic multi-agent travel-planning 2 benchmarks. Return exactly one valid JSON object and no markdown. Never reveal or hint at the 3 correct answer outside the declared correct_answer field. Use only the schema and operators 4 specified by the user. ⬇ 1 Create one HiddenBench-style group travel decision task. 2 3 Variation seed: <VARIATION_SEED> 4 Dataset position: <TASK_INDEX> 5 Avoid duplicating these accepted task signatures: <ACCEPTED_TASK_SIGNATURES> 6 Use exactly <NUMBER_OF_TRAVELERS> travelers. 7 Make option <TARGET_CORRECT_OPTION> the declared correct answer. 8 Use HiddenBench-style weighted preference aggregation: 9 - satisfied must = +2 10 - satisfied prefer = +1 11 - matched avoid = -1 12 - matched reject = -2 13 The declared answer must be the unique highest-scoring option. It may violate a strong preference 14 if it remains the best group compromise. Distractors do not need artificial, unique violation 15 patterns. 16 17 The task must: 18 - involve exactly 3 or 4 travelers with plausible roles and a concrete group-trip context; 19 - expose exactly four fully specified candidate plans (A, B, C, D) in public information; 20 - distribute private structured preferences across travelers using all tiers: must, prefer, 21 avoid, reject; 22 - make the declared answer uniquely best under the validation profile above; 23 - require evidence from multiple travelers to identify the answer; 24 - remain solvable by literal comparison only: no outside travel facts, arithmetic ambiguity, 25 subjective interpretation, compromise, or unstated assumptions; 26 - make every candidate feasible in ordinary real-world terms; 27 - give every traveler at least one preference that matches some but not all candidates; 28 - avoid unsafe, discriminatory, medically implausible, or internally contradictory content. 29 30 Tier semantics: 31 - must: matching is a strong positive preference worth +2. 32 - reject: matching is a strong negative preference worth -2. 33 - prefer: matching adds +1 utility. 34 - avoid: matching subtracts 1 utility. 35 36 Allowed operators: eq, neq, in, not_in, contains, contains_all, contains_any, excludes, lte, gte. 37 For contains/excludes operators, candidate attributes must be arrays. ‘in‘ means a scalar or any 38 array item belongs to the preference value list; ‘not_in‘ means no scalar/array item belongs to 39 that list. For lte/gte, attributes must be numbers. Every preference must reference an attribute 40 present in every option. 41 42 Every option attributes object must include at least these exact keys: 43 transport_mode, departure_window, duration_days, price_per_person_usd, hotel_category, 44 room_arrangement, step_free, daily_active_hours, max_walk_km_per_day, meal_options, activities. 45 You may add a small number of concrete keys such as arrival_time, layovers, or neighborhood. 46 All four options must use the same keys and comparable units. 47 48 Return this exact shape: 49 50 "name": "short_unique_snake_case_name", 51 "description": "shared scenario, dates or season, origin, destination, group relationship", 52 "public_information": [ 53 "2-5 public facts that do not encode any preference or answer" 54 ], 55 "options": [ 56 57 "id": "A", 58 "title": "concise plan label", 59 "attributes": 60 "transport_mode": "train", 61 "departure_window": "morning", 62 "duration_days": 4, 63 "price_per_person_usd": 900, 64 "hotel_category": "midrange", 65 "room_arrangement": "two twin rooms", 66 "step_free": true, 67 "daily_active_hours": 7, 68 "max_walk_km_per_day": 5, 69 "meal_options": ["vegetarian", "seafood"], 70 "activities": ["museum", "market"] 71 72 73 ], 74 "agents": [ 75 76 "id": "traveler_1", 77 "role": "realistic role in this group", 78 "context": "one sentence explaining why the preferences are plausible", 79 "preferences": [ 80 81 "id": "t1_must_1", 82 "tier": "must", 83 "attribute": "step_free", 84 "operator": "eq", 85 "value": true, 86 "statement": "A natural first-person statement with clear non-negotiable force." 87 , 88 89 "id": "t1_reject_1", 90 "tier": "reject", 91 "attribute": "transport_mode", 92 "operator": "eq", 93 "value": "flight", 94 "statement": "A natural first-person statement clearly refusing flights." 95 96 ] 97 98 ], 99 "correct_answer": "one of A/B/C/D" 100 101 102 Use 2-4 preferences per traveler. Individual travelers may have only hard or only soft items, but 103 the complete task must use each of must, prefer, avoid, and reject. NEVER omit the "tier" key: 104 copy its lowercase value from the preference id. Before responding, evaluate every predicate 105 against A/B/C/D, calculate the aggregate score for all options, and verify that each traveler has 106 at least one predicate matching 1-3 options. It is acceptable for an individual traveler to rank 107 the declared answer first; at least two profiles must still contribute non-constant evidence to 108 the aggregate decision. The statement must faithfully express its predicate and tier. Do not add 109 evaluation, rationale, violation summaries, or extra top-level fields. ⬇ 1 A previous blueprint was abandoned after failed repairs. Create a genuinely new blueprint and 2 avoid these failures: 3 <PRIOR_VALIDATION_ISSUES> F.2 Deterministic Validation and Repair Generated tasks are first evaluated by a deterministic validator. Under the HiddenBench validation profile, a matched must contributes +2+2, a matched prefer contributes +1+1, a matched avoid contributes −1-1, and a matched reject contributes −2-2. The declared answer must be the unique highest-scoring candidate. Instances that fail validation are repaired using the following prompt. ⬇ 1 Repair the rejected HiddenBench travel blueprint below in place. 2 3 Dataset position: <TASK_INDEX> 4 Do not duplicate accepted signatures: <ACCEPTED_TASK_SIGNATURES> 5 6 Change the minimum option attributes or preferences needed to fix every validator issue. Preserve 7 the scenario, traveler roles, four option ids, and all already-valid content. Return the complete 8 repaired blueprint as one JSON object, not a patch. 9 10 For an "expected exactly one hard-feasible option, found []" error in strict mode, repair in this order: 11 1. Read the detailed violation list for the declared correct_answer. 12 2. Change its public attributes, or the conflicting preference predicate and matching statement, 13 until it violates zero MUST/REJECT preferences. 14 3. Re-evaluate every hard predicate literally against all four options. 15 4. Ensure each other option retains a different non-empty violation set. 16 Do not merely repeat the rejected blueprint. 17 18 Required invariants after repair: 19 - exactly four comparable plans and exactly 3 or 4 travelers; 20 - every preference contains id, tier, attribute, operator, value, and statement; 21 - tier is exactly one lowercase value: must, prefer, avoid, or reject; 22 - each traveler has at least two preferences and at least one predicate that matches some but not 23 all plans; 24 - the task collectively uses all four tiers; 25 - using must=+2, prefer=+1, avoid=-1, reject=-2, correct_answer is the unique 26 highest-scoring option; 27 - an individual traveler may rank correct_answer first, but at least two profiles must contribute 28 non-constant option scores; 29 - plans may violate MUST or REJECT items; do not force a fully hard-feasible plan; 30 - every preference statement exactly matches its structured predicate. 31 32 Validator issues: 33 <VALIDATOR_ISSUES_JSON> 34 35 Rejected blueprint: 36 <REJECTED_BLUEPRINT_JSON> ⬇ 1 You are a strict dataset reviewer. Check a travel hidden-profile 2 benchmark for semantic realism and consistency. Return exactly one valid JSON object with keys 3 accepted (boolean), issues (array of concise strings), and checks (object). Do not repair the task. ⬇ 1 Review this candidate benchmark after deterministic validation. 2 3 Reject it if any preference statement disagrees with its structured predicate; a plan attribute 4 is vague, incomparable, or implausible; a private fact leaks into public text; the declared answer 5 is hinted by names/order/wording; the scenario needs outside knowledge; constraints are contrived 6 instead of realistic; an entire traveler profile has no decision-relevant information; two options 7 are semantically indistinguishable; or the task could be solved from public information or one 8 profile alone. A realistic preference shared by all plans is allowed when that same traveler has 9 another preference that distinguishes plans. 10 11 Accept only if the exact four plans, realistic profiles, a unique weighted-utility answer under 12 must=+2, prefer=+1, avoid=-1, reject=-2, and multi-agent dependency are clear to a careful human. 13 Do not require every strong preference to be satisfied in HiddenBench mode; realistic compromise 14 plans are allowed. 15 16 The deterministic audit is authoritative for predicate evaluation, scoring, winner uniqueness, 17 and information dependency. Do NOT reject a task merely because any option fails a MUST, matches a 18 REJECT, or because one traveler locally ranks the final answer first. Do not reinterpret operators 19 or report option-level score violations already handled by the audit. Review only semantic 20 fidelity of statements to predicates, realism, ambiguity, leakage, and unsupported outside facts. 21 22 Deterministic audit: 23 <DETERMINISTIC_AUDIT_JSON> 24 25 Blueprint: 26 <CANDIDATE_BLUEPRINT_JSON>