Paper deep dive
MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin, Kai-Wei Chang, Siddhant Arora, Shu-wen Yang, Abdelrahman Mohamed, Shinji Watanabe, Hung-yi Lee, David Harwath
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/20/2026, 2:56:06 AM
Summary
The paper introduces MP-Bench, the first benchmark designed to evaluate conversational voice agents as active participants in multi-party conversations. It assesses agents on turn-taking awareness and response appropriateness across two scenarios: Discussion and Turn-based Game. The study benchmarks 12 voice agents, revealing that current real-time models perform poorly (near chance) on multi-party turn-taking and have low comprehension scores, highlighting significant challenges in integrating voice agents into group dynamics.
Entities (10)
Relation Signals (8)
MP-Bench â evaluates â Voice Agents
confidence 95% · MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness.
MP-Bench â measures â Response Appropriateness
confidence 95% · MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness.
MP-Bench â measures â Turn-taking Awareness
confidence 95% · MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness.
Real-time Voice Agents â performspoorlyon â Multi-party Turn-taking
confidence 93% · realtime voice agents... remain near chance on multiparty turn-taking
MP-Bench â hasscenario â Discussion Scenario
confidence 92% · Specifically, we design two scenarios: Discussion... and Turn-based Game...
MP-Bench â hasscenario â Turn-based Game Scenario
confidence 92% · Specifically, we design two scenarios: Discussion... and Turn-based Game...
GPT-Realtime â isevaluatedby â MP-Bench
confidence 90% · By benchmarking 12 voice agents... GPT-Realtime (OpenAI, 2025b)...
Moshi â isevaluatedby â MP-Bench
confidence 90% · open-source full-duplex streaming models (Moshi...)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conversational voice agents have advanced significantly, offering increasingly natural human-machine interactions through both cascaded and end-to-end architectures. However, while recent benchmarks extensively evaluate dyadic interactions and passive audio comprehension, they largely overlook a prevalent real-world scenario: multi-party conversations. Evaluating agents in these settings is fundamentally more challenging than in dyadic interactions due to the exponentially greater conversational complexity. For voice agents to integrate seamlessly into human group dynamics, they must not only generate contextually appropriate responses but also demonstrate a nuanced understanding of open turn-taking. To address this gap, we introduce Multiparty Bench (MP-Bench), the first benchmark specifically designed to objectively evaluate conversational speech systems as active participants within multi-party contexts. MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness. Additionally, we incorporate comprehension-based question-answering tasks as a complementary evaluation. By benchmarking 12 voice agents, we find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking, exposing an open challenge for real-time voice agents under multiparty scenario.
Tags
Links
- Source: https://arxiv.org/abs/2609.13076v1
- Canonical: https://arxiv.org/abs/2609.13076v1
Trouble viewing inline? Open PDF directly â
Full Text
77,535 characters extracted from source content.
Expand or collapse full text
MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant Yi-Jen Shih 1 * , Shih-Yun Shan Kuan 2 * , Guan-Ting Lin 2 * , Kai-Wei Chang 3 , Siddhant Arora 4 , Shu-wen Yang 2 , Abdelrahman Mohamed 5 , Shinji Watanabe 4 , Hung-yi Lee 2 , David Harwath 1 1 The University of Texas at Austin, 2 National Taiwan University, 3 Massachusetts Institute of Technology, 4 Carnegie Mellon University, 5 Meta AI Correspondence: yjshih@utexas.edu, r14942094@ntu.edu.tw Abstract Conversational voice agents have advanced sig- nificantly, offering increasingly natural human- machine interactions through both cascaded and end-to-end architectures. However, while recent benchmarks extensively evaluate dyadic interactions and passive audio comprehension, they largely overlook a prevalent real-world scenario: multiparty conversations. Evaluat- ing agents in these settings is fundamentally more challenging than in dyadic interactions due to the dramatically greater conversational complexity. For voice agents to integrate seam- lessly into human group dynamics, they must not only generate contextually appropriate re- sponses but also demonstrate a nuanced un- derstanding of open turn-taking. To address this gap, we introduce Multiparty Bench (MP- Bench), the first benchmark specifically de- signed to objectively evaluate conversational speech systems as active participants within multiparty contexts. MP-Bench assesses agent behavior along two primary dimensions: turn- taking awareness and response appropriateness. Additionally, we incorporate comprehension- based question-answering tasks as a comple- mentary evaluation. By benchmarking 12 voice agents, we find that realtime voice agents stay at or below33% on multiparty comprehension and remain near chance on multiparty turn- taking, exposing an open challenge for realtime voice agents under multiparty scenario. 1 Introduction Conversational voice agents have gained consid- erable attention in both academia (Wang et al., 2024, 2026a) and industry (Google, 2025; Ope- nAI, 2025b) in recent years, as speech serves as the most natural and intuitive modality for human- machine interaction. Furthermore, compared to text, the speech modality conveys rich paralinguis- tic information, such as speaker identity, emotion, * Equal contribution. Multiparty Spoken Conversation Recording Leo "Does anyone actually know how tall the Eiffel Tower is with the antenna?" Emma "I think it's exactly 300 meters tall." Karen "No, with the new antenna, I'm pretty sure it's 330 meters." ... Leo "AI Assistant, what is your answer and who got the height right?" * Example: Appropriateness evaluation Voice Agent Evaluation Understanding Does the agent understand the context? Behavior Turn-Taking Decision Does the agent know when to speak? Appropriateness (if speaking) Does the agent know what to say? Figure 1: Overview of the evaluation aspects in MP- Bench. Given a prepared multiparty spoken conversa- tion, we evaluate the voice agentâs understanding and behavioral abilities. and prosody. These systems are designed to pro- cess human speech and generate responses in the same audio modality. Based on their interactive behavior, conversa- tional agents can be broadly classified as either non-realtime or realtime models. Non-realtime agents (Xu et al., 2025) can comprehend spoken ut- terances and synthesize audio responses; however, they exhibit considerable latency, precluding seam- less, human-like conversations. Real-time models, conversely, are explicitly optimized for interactive dialogue. To implement these agents, there are two primary architectural approaches: cascaded and end-to-end. The cascaded approach relies on a pipeline comprising a front-end Automatic Speech Recognition (ASR) module, a text-based LLM, and a Text-to-Speech (TTS) synthesizer. The end-to- end approach, on the other hand, employs a single unified model that directly processes speech input and generates speech output, bypassing the need to convert speech to intermediate text representations. To evaluate the performance of these systems, recent studies have introduced various bench- arXiv:2609.13076v1 [eess.AS] 11 Sep 2026 marks (Arora et al., 2025a). Though multiple di- mensions of conversational behavior have been evaluated, existing benchmarks largely overlook multiparty conversations, which are a prevalent sce- nario in daily human communication. Ultimately, we envision integrating voice agents into human group interactions, enabling them to participate and contribute to discussions as naturally as humans. To this end, we introduce Multiparty Bench (MP- Bench) to evaluate voice agentsâ turn-taking and response behavior in controlled multiparty scenar- ios spanning explicit, implicit, and negative turn- taking conditions, as a first step toward full partic- ipation in natural multiparty conversation (Fig 1). Our benchmark assesses agent performance along two primary dimensions: turn-taking awareness (knowing when to speak or remain silent) and con- textual appropriateness. Unlike dyadic interactions where the end of an utterance typically serves as an implicit cue to respond, multiparty turn-taking introduces more complexity. To evaluate these dy- namics objectively, we utilize synthetically gener- ated recordings, enabling precise control over the conversational state. Specifically, we design two scenarios: Discus- sion, where the agent mediates a multiparty dispute, and Turn-based Game, which requires the agent to adhere to game rules and speak only on its des- ignated turn. These scenarios span three distinct turn-taking conditions: Explicit (the agent is ad- dressed by name), Implicit (contextual cues imply it is the agentâs turn without explicit naming), and Negative (the agent must correctly remain silent while not their turn). Furthermore, we incorporate comprehension tasks to test accurate speaker at- tribution. While our evaluation focuses primarily on realtime voice agents, we also benchmark non- realtime systems and text LLMs for comparison. Our evaluation reveals that current state-of-the- art realtime voice agents lag behind non-realtime counterparts in multiparty conversation and both of them perform near-random performance on im- plicit and negative turn-taking, highlighting a crit- ical limitation in current models. Notably, these findings are robustly supported by our human eval- uation, which exhibits a high correlation with the LLM judge. Our main contributions are as follows: 1.We introduce MP-Bench 1 , the first step toward benchmarking voice agents as active partici- pants in multiparty conversations, spanning 1 https://github.com/atosystem/MP-Bench both understanding and behavior aspects. 2. We evaluate 12 state-of-the-art voice agents, revealing that current models struggle signifi- cantly with appropriate multiparty interaction. 3.Through ablation studies, we demonstrate that these shortcomings are non-trivial and stem directly from the inherent complexities of mul- tiparty dynamics. 2 Related Works Recent voice agent benchmarks have targeted vari- ous dimensions, including paralinguistics (Ao et al., 2024; Yang et al., 2026; Liu et al., 2025), full- duplex turn-taking (Ge et al., 2025; Arora et al., 2025b; Chang et al., 2025; Peng et al., 2025; Lin et al., 2025b,a, 2026b), reasoning (Cui et al., 2025; Lin et al., 2025c; Fang et al., 2025), multi-turn coherence (Du et al., 2025; Gosai et al., 2025), tool-use (Jain et al., 2026; Lin et al., 2026a), or a combination thereof (Chen et al., 2024; Liu et al., 2026; Yan et al., 2025; Li et al., 2026). Existing multiparty benchmarks primarily assess passive understanding rather than active participation (In- oue et al., 2025; Wang et al., 2025; Kwon et al., 2025), and while concurrent work evaluates active turn-taking, it is strictly confined to text modal- ity (Bhagtani et al., 2026; Nama et al., 2026). In contrast, our benchmark evaluates understanding, turn-taking, and response appropriateness directly from audio, forcing agents to jointly process acous- tic and contextual cues, more closely mirroring real-world scenarios. 3 MP-Bench Overview MP-Bench consists of927tasks created from two scenarios, Discussion and Turn-based Game, with each task belonging to one of two task families: understanding tasks or behavioral tasks. The Dis- cussion scenario contributes tasks of both task fam- ilies, whereas the Turn-based Game scenario con- tributes only behavioral-task tasks. We provide the overview and evaluation description in this section and leave the construction process in Sec 4. 3.1 Scenario Both scenarios place the voice agent as the fourth participant in a four-way spoken interaction with three simulated humans. Representative tasks from each scenario are shown in Table 1. Spk.Content Discussion Scenario MiaWhich continent has the highest number of coun- tries in it? KarenI read somewhere that itâs Africa. JackReally? I feel like Europe has a ton of tiny coun- tries. MiaThat is a good point, Jack. KarenBut Africa is massive. JackMaybe, but Asia has a lot too. What is your final answer, Karen? KarenSticking with Africa. MiaI am going with Europe. JackMia is probably wrong. Mia Rude! AI Assistant, please tell us the answer. And judge who is correct. AgentItâs Africa, so Mia is correct.â AgentItâs Africa, so Karen is correct.â Turn-based Game Scenario RoseMia, Leo, are you two ready for a mental work- out? I call it the Rival Response. LeoAlways ready, Rose. How do we play? RoseTwo teams. When a Group Leader says a word, the Group Follower from the other side has to respond randomly. MiaSounds fun. How are we dividing up? Rose Mia, you take the helm of Group 1 as Leader, and Leo, youâl be the Follower for Group 1. LeoSolid copy. Iâm following Mia. Rose Iâl take the lead for Group 2, and the AI Assis- tant will be my Follower. Mia Okay, so if Rose speaks, Leo answers. If I speak, the AI answers. RoseExactly. Focus up, everyone. Game start. RosePenguin LeoLaser RoseBook AgentBalloonâ Agent[silence]â Table 1: Examples behavior tasks generated from the two scenarios. In the top example, the agent must settle the discussion while on the bottom example the agent should remain silent since it is not its turn. We also provide appropriate and non-appropriate agent response for each example. More examples in Appendix A.5. Discussion. This scenario simulates a multiparty debate centered on a single question. After the question is posed, the three human participants exchange differing viewpoints; the conversation concludes with one of them explicitly addressing the voice agent by name and inviting it to evaluate the arguments and declare which speaker is correct. This setup is designed to evaluate whether the agent can comprehend a contested multiparty discussion and act on an explicit invitation to take its turn. Turn-based Game. The Turn-based Game sce- nario evaluates the agentâs behavior within a struc- tured, turn-based game. A human speaker intro- duces the rules and leads a preliminary exchange that establishes the vocal characteristics of the four participants, who are split into two opposing teams; each team has a leader and a follower. The core rule dictates that whenever a team leader speaks, the opposing teamâs follower must immediately re- spond. These role assignments are introduced pre- cisely to make turn-taking clear and deterministic: at any moment in the game, the voice agentâs role within its team unambiguously specifies whether it is the participant expected to speak next. The required spoken content depends on the game vari- ant being played: Word Chain (saying a random word), Increasing Number, or Decreasing Num- ber (incrementing or decrementing the opposing leaderâs number). We intentionally minimize the cognitive load of the required spoken content as the goal is to test multiparty turn-taking ability rather than complex reasoning and instruction following. 3.2 Task Each task in MP-Bench belongs to one of two fam- ilies: understanding tasks or behavioral tasks. Understanding tasks. In an understanding task, the audio clip ends with one speaker explicitly ask- ing the voice agent an understanding question about the preceding conversation, and the agentâs spo- ken answer is compared against a ground-truth an- swer. These tasks are constructed from the Discus- sion scenario and are designed to probe the agentâs ability to comprehend multiparty conversations, in- cluding identifying speakers, tracking who said what, and resolving references across speakers. Behavioral tasks. In a behavioral task, we assess the agentâs action immediately following the audio segment: whether it appropriately takes a turn or remains silent, and, when it does speak, whether its response is contextually appropriate. Each be- havioral task carries a ground-truth label indicating whether the agent is expected to speak. Behav- ioral tasks drawn from Discussion target explicit turn-taking, in which the agent is directly invited to respond; behavioral tasks drawn from Turn-based Game target implicit and negative turn-taking, in which the agent must infer from context whether to speak or remain silent and, when it does speak, comply with the established game rule. 3.3 Evaluation 3.3.1 Evaluation Pipeline Following the protocol of Full-Duplex-Bench (Lin et al., 2025b), we stream the input audio to the voice agents and transcribe their spoken out- put using the NeMo Parakeet streaming ASR model (Rekesh et al., 2023). We leverage ASR word-level timestamps to detect response onsets; an agent is flagged as having responded if any gen- erated word starts after the conclusion of the input audio, successfully bypassing the need for an ex- plicit voice activity detection (VAD) module. The generated spoken content is then assessed using an LLM-as-a-judge framework (Chiang and Lee, 2023) powered by GPT-5 (OpenAI, 2025a). 3.3.2 Understanding Task Metrics Overall Accuracy. This metric measures whether the voice agent correctly answers the prompted understanding questions. Speaker Name Accuracy. Additionally, we re- port accuracy for a specific subset of tasks where the ground-truth answer corresponds to one of the speakersâ names in the recording. We isolate this metric to enable direct comparison against a33.3% random-guessing baseline, as every task involves exactly 3 human speakers. 3.3.3 Behavioral Task Metrics Turn-Taking Accuracy. This metric evaluates the agentâs ability to respond when required and remain silent otherwise. In the Discussion scenario, the ground truth always requires a response (evaluating explicit turn-taking). In the Turn-based Game sce- nario, the ground-truth labels are evenly balanced between speaking and remaining silent, establish- ing a 50% random-guessing baseline. Appropriateness Accuracy. Finally, we report an appropriateness accuracy where a sample is consid- ered correct only if the agent exhibits both correct turn-taking behavior and generates a valid response. For the Discussion scenario, a response is deemed valid if it shows logical consistency within the de- bate, regardless of factuality. For the Turn-based Game scenario, validity is defined as compliance with the assigned game rules. 3.3.4 Response Latency For each task, we report the mean response latency for realtime voice agents. This is defined as the du- ration between the end of the input audio (i.e., the conclusion of the last input word) and the start of the agentâs first generated word. These timings are extracted from the ASR timestamps results same as turn-taking detection. 4 MP-Bench Construction The construction of MP-Bench is guided by two core principles: (1) tasks must be objective and automatically evaluable, and (2) tasks must necessi- tate auditory cues, ensuring they cannot be trivially solved by text-only LLMs using transcripts alone. To satisfy these principles, we require precise con- trol over the spoken dialogue. Therefore, we em- ploy LLMs to generate the conversation content and ElevenLabs 2 â conversation API to synthesize high-quality conversational audio. Fig. 2 shows our benchmark generation pipeline. We begin by curating a pool of speaker combina- tions (Sec. 4.1) and defining the two scenarios (Sec. 3.1, 4.2). Leveraging these components, we prompt an LLM to generate diverse dialogue in- stances. Finally, for each instance, we construct specific evaluation tasks (Sec. 3.2, 4.3). To ensure high quality, all tasks undergo rigorous human ver- ification before being finalized for the benchmark. Detailed statistics in Appendix A.1. 4.1 Speaker Attribution We evaluate voice agents in a four-participant set- ting to balance multiparty complexity with man- ageable audio lengths. This choice is deliberate: 3 speakers (2 pre-recorded plus the agent itself) simplifies the task to a near-dyadic baseline easily solved by standard ASR and LLM pipelines, while more than 4 introduces long-context memory fail- ures that confound evaluation. 4 participants cap- tures true multiparty dynamics without triggering context-length bottlenecks, though varying speaker counts remains a natural direction for future work as long-context robustness improves. To ensure objective evaluation, we carefully cu- rate voice sets with highly distinct acoustic char- acteristics, yielding high speaker separability (Ap- pendix A.3). These curated sets are designated as âvalid speaker combinationsâ and used during dia- logue generation (See Appendix A.2). To evaluate benchmark difficulty under harder acoustic condi- tions, we also conduct an ablation using deliber- ately similar voices across all pre-recorded audio. Further details are provided in Appendix D. 4.2 Scenario Generation 4.2.1 Discussion Scenario We curate a list of 83 topics (Appendix A.4) and design a corresponding prompt template for the 2 https://elevenlabs.io Scenario Generation (Sec. 4.2) Topic List (83 topics) Valid Speaker Combinations Game Types Discussion (Sec. 4.2.1) Dialogue Generation Template Turn-based Game (Sec. 4.2.2) Dialogue Generation Template LLM Dialogue Instance Filtering (Sec 4.2.3) w/o Spk. Names w/ Anonymized Spk. Keep Failure Keep Success Human Check #256 Dialogue Instances (128/scenario) Task Generation (Sec. 4.3) Understanding Task (Sec. 4.3.1) QA Generation Template (4 QA types) LLM Keep Failure Keep Success Human Check #543 Understanding Tasks Behavior Task (Sec. 4.3.2) Explicit TurnTaking: Address voice agent to settle the discussion Implicit/Negative TurnTaking: Create several turns of game play Human Check #384 Behavior Tasks Dialogue Instances Ask LLM to recover Spk. Names Turn-based Game Dialogue Instances Discussion Dialogue Instances w/o Spk. Names w/ Spk. Names Task FilteringAsk LLM to answer the questions QA Pairs Figure 2: The overall pipeline for creating MP-Bench. We first generate dialogue contents from the two scenarios and then generate Understanding and Behavior tasks for each generated dialogue instance. LLM. To generate each dialogue instance, we ran- domly sample a topic from the list alongside a valid speaker combination to populate the tem- plate, which is then fed into an LLM for gener- ation. To maximize diversity and prevent duplica- tion, we maintain a strict record of all generated topic-speaker tuples. Furthermore, the LLM generates two additional types of content-related metadata for each spoken turn to facilitate subsequent question generation: addressees and coreferences. Addressees indicate whom the speaker is addressing during a given utterance; notably, this is not necessarily the subse- quent speaker. Coreferences are pronouns or noun phrases in utterances that refer back to the conver- sational participants or previously discussed topics. 4.2.2 Turn-based Game Scenario Similar to Scenario 1, we design a prompt template and randomly sample valid speaker combinations to guide the LLM in generating the dialogue tran- scripts. For this scenario, we do not ask LLM to generate the coreferences and the addressee as this scenario are not suitable due to the game setup. 4.2.3 Dialogue Instance Filtering For both scenarios, to ensure that the generated dialogues cannot be trivially solved by a text-only LLM, we must guarantee that speaker identities cannot be fully recovered solely from the transcript. Conversely, to make the task solvable via the au- dio modality, every participantâs name must be ex- plicitly mentioned at least once during the conver- #InquiryBased onExample 1Spk.Cont.Whatâs the speakerâs name who think Paris is a capital? 2Cont.Spk.What is Jack certain about in the conversation? 3Cont.Cont.Whatâs the claim of the speaker who started the question? 4Spk.Spk.Whatâs the speakerâs name that disagree with Jackâs answer? Table 2: Examples for the 4 types of QA in our bench- mark. âSpk.â and âCont.â indicates speaker attributes and content-related information respectively. sation; otherwise, it would be impossible for the voice agent to map the acoustic profiles to their respective identities. In practice, we apply a rule-based filter to ver- ify that each participant is addressed or mentioned at least once. Furthermore, we employ an LLM to validate the generated dialogue under two dis- tinct prompting conditions: (a) providing only the raw spoken content without any speaker informa- tion, and (b) providing the spoken content paired with anonymized speaker identifiers (Speaker1, Speaker2...) for each turn. We then filter the gener- ated dialogues, retaining only those instances that the LLM successfully solves all speaker names for each spoken turn under condition (b) but fails to solve under condition (a). Notably, condition (b) effectively simulates the theoretical upper bound of a perfect cascaded voice agent equipped with flawless ASR and speaker diarization systems. 4.3 Task Generation 4.3.1 Understanding Tasks We employ Discussion dialogue instances to cre- ate question-answering tasks. To generate QA pairs systematically, we first categorize the infor- mation in the generated dialogue instances into two categories: speaker-related attributes (e.g., name, gender, voice characteristics) and content-related details (e.g., spoken utterances, addressees, and coreferences). Based on these two types of infor- mation, we construct four distinct categories of questions (Tab. 2). We explicitly prompt an LLM to generate QA pairs corresponding to one of these four defined types. Subsequently, we apply an LLM-based filter- ing mechanism, similar to the protocol employed during the scenario generation. Specifically, we evaluate the generated questions under two prompt- ing conditions: (a) providing a text-only LLM with the raw spoken transcript, and (b) providing the LLM with the speaker names. We retain only those QA pairs that the LLM successfully resolves under condition (b) but fails under condition (a). This en- sures that the questions are inherently logical and solvable with full context, yet cannot be trivially answered using solely the text transcript, thereby necessitating true multiparty spoken conversation comprehension. 4.3.2 Behavior Tasks For the Discussion scenario dialogue instances, we append an additional spoken turn at the end of the dialogue instance that explicitly addresses the voice agent (Table 1), asking which participant it agrees with. For the Turn-based Game scenario, we gen- erate two distinct sets of game-playing turns and append them following the dialogue instances (i.e., the rules introduction and team assignment). 5 Benchmark Results 5.1 Evaluated Models Realtime Audio Models.We evaluate 10 systems designed for realtime spoken interac- tion, comprising proprietary full-duplex APIs (GPT-Realtime (OpenAI, 2025b), Gemini-2.5 Live (Google DeepMind, 2025a) 3 , Gemini-3.1 Live (Google, 2025)), open-source full-duplex streaming models (Moshi (DĂ©fossez et al., 2024), PersonaPlex (Roy et al., 2026), Freeze- Omni (Wang et al., 2024), and Ultravox (Fixie 3 We refer to gemini-2.5-native-dialogue. AI, 2024)), and half-duplex end-to-end speech LMs (GLM-4-Voice (Zeng et al., 2024) and Covo- Audio (Wang et al., 2026b)). Non-real-time Audio Models. Unsuited for low- latency interaction, this group comprises non- streaming omni models. We evaluate two systems: the non-streaming Qwen3-Omni (Xu et al., 2025) and Gemini-3.1-Pro (Google DeepMind, 2025b), which process full audio inputs to produce text. These models serve as a strong top line for audio- native multiparty comprehension when the realtime interaction constraint is relaxed. Text LLMs. As references, we evaluate Gemini- 3.1-Pro (Google DeepMind, 2025b) under two con- figurations. In the ASR-only setting, the model receives the ground truth ASR transcript of the in- put audio without any speaker label; this captures the performance ceiling of a cascaded system that lacks speaker diarization. In the SpkD-ASR setting, the model additionally receives the ground-truth anonymized speaker label for every turn, emulat- ing the upper bound of a cascaded system equipped with perfect speaker diarization. For both settings, the response is generated as text and compared to the reference answer directly. 5.2 Main Results Table 3 summarizes the MP-Bench results. We highlight several notable trends in the following. Behavioral Tasks are Less Challenging than Un- derstanding under Explicit Multiparty Turn- Taking. We observe divergent performance trends between behavioral and understanding tasks within the Discussion scenario. Across nearly all settings, models achieve significantly higher scores on be- havioral tasks than on understanding. This discrep- ancy arises because understanding tasks require the model to accurately understand fine-grained acoustic and contextual relationships. In contrast, behavioral tasks allow powerful LLMs to rely on their strong linguistic priors to generate reasonable responses, even when they fail to fully resolve the multiparty dynamics. Notably, the performance gap between the text LLM ASR-Only and SpkD- ASR settings shrinks dramatically from66.67% (93.94 - 27.27) on understanding to just14.06% (89.06 - 75.00) on behavioral appropriateness accu- racy. This confirms that highly capable LLMs can effectively infer or hallucinate reasonable conversa- tional continuations even in the absence of explicit speaker identification. Voice Agents Struggle with Implicit and Neg- Model Understanding: DiscussionBehavior: DiscussionBehavior: Turn-based Overall Acc. Spk. Name Acc. Lat. (s) T Acc. App Acc. Lat. (s) T Acc. App Acc. Lat. (s) Text LLMs Gemini-3.1-Pro (ASR-only)4.7927.27â10075.00â49.2249.22â Gemini-3.1-Pro (SpkD-ASR)87.4893.94â10089.06â98.4498.44â Non-realtime Voice Agents Qwen3-Omni34.4436.11â10060.16â52.7346.87â Gemini-3.1-Pro73.3078.28â10090.63â52.7351.95â Realtime Voice Agents Freeze-Omni0.180.001.817.710.001.049.6148.440.6 Moshi1.660.000.975.001.560.552.7328.121.5 PersonaPlex3.147.320.696.090.000.551.174.690.7 Ultravox21.3625.004.599.2239.063.646.0931.254.0 GLM-4-Voice11.6012.632.4100.0034.382.349.1445.352.6 Covo-Audio32.6037.371.999.2242.191.748.8348.441.3 GPT-Realtime14.0016.675.9100.0053.136.048.4428.136.7 DuplexCascade6.459.232.7100.003.912.148.6348.6317.2 Gemini-2.5 Live10.5811.878.578.1336.726.548.0020.008.6 Gemini-3.1 Live4.905.818.799.2257.037.452.5738.998.2 Table 3: Main MP-Bench results. T. Acc. and Lat. denote turn-taking accuracy (%) and latency (seconds), respectively. App Acc. indicates appropriateness accuracy. Random-guessing baselines are 33.3% for Spk. Name Acc. and 50% for Turn-Based T. Acc. For Discussion T. the ground truth is always respond, please read jointly with the balanced Turn-based metrics. Latency is measured exclusively for real-time voice agents and averaged over non-silent responses. ative Turn-Taking Relative to Explicit Cues. Comparing turn-taking accuracy between the Turn- based Game and Discussion scenarios reveals a substantial performance gap. Nearly all voice agents perform at or below the random-guessing baseline in the Turn-based Game, whereas multi- ple realtime agents achieve over90% accuracy in the Discussion scenario. In contrast, the text LLM baseline equipped with SpkD-ASR achieves near- perfect turn-taking accuracy across both. These results indicate that a primary challenge for current voice agents lies in identifying implicit turns and appropriately remaining silent when they are not addressed. Furthermore, the near-perfect perfor- mance of the SpkD-ASR baseline confirms that the task is fundamentally solvable, provided the model can accurately extract and leverage speaker- attributed information during prediction. Non-Realtime Agents Outperform Realtime Models, Particularly on Understanding.In understanding tasks, non-realtime agents (led by Gemini-3.1-Pro) significantly outperform their re- altime counterparts. Even the best realtime system, Covo-Audio, lags far behind Gemini-3.1-Pro in both overall and speaker name accuracy. Notably, most realtime models fail to clear the33.3% ran- dom baseline for speaker identification, whereas both non-realtime systems exceed this threshold. Within the realtime group, Covo-Audio stands out as the lone strong performer; Moshi and Freeze- Omni produce mostly generic backchannels at the bottom. This gap indicates that low-latency ar- chitectural trade-offs, such as streaming decoding, severely penalize multiparty understanding. In Discussion behavioral tasks, the realtime ver- sus non-realtime performance gap persists but nar- rows. However, in Turn-Based Games, this delta vanishes entirely, with all models performing at near-random levels. These findings suggest that while multiparty comprehension can be improved by sacrificing latency, the inability to navigate im- plicit and negative turn-taking is a fundamental deficit orthogonal to latency constraints. 6 Ablation Results 6.1 Disentangling Game-Rule Following from Multiparty Capabilities The multiparty Turn-Based Games scenario de- mands two concurrent capabilities from the agent: adhering to game rules to generate appropriate con- tent, and utilizing multi-party context to determine when it is appropriate to speak. To test whether the observed failures stem from complex game rules, we construct a single-speaker variant. In this setup, one human speaker introduces the rules, assigns roles, and plays directly with the agent, while the core mechanics (Word Chain, Increasing Number, Decreasing Number) remain identical to the multi- ModelPrecision Recall T Acc. GPT-Realtime78.270.076.0 Gemini-2.5 Live64.090.068.0 Table 4: Single-speaker Turn-Based game ablation (%). 50ElevenLabs-synthesized dialogues featuring a single human speaker who explains the rules and plays the game directly with the agent. Both models are clearly above the50% random baseline, unlike the multiparty Turn-Based game, where every realtime model is within ±3 points of chance. party version. Evaluating50ElevenLabs-synthesized single- speaker dialogues (Table 4) reveals that both GPT- Realtime (76.0%T Acc.) and Gemini-2.5 Live (68.0%) perform significantly above the50%ran- dom baseline (exact binomial,p = 3.1 Ă 10 â4 andp = 0.015;95%intervals[62.6, 85.7]and [54.2, 79.2], both excluding chance), and their ap- propriateness accuracy (74.0%and63.0%) far ex- ceeds their multiparty counterparts (28.13%and 20.00%). In stark contrast, their performance on the multiparty equivalent hovers within±3points of chance (Table 3). These results demonstrate that the multiparty failure is not a failure of game-rule instruction following. We note, however, that the single-speaker variant also removes the speaker- identification burden, so this ablation alone is not aim to separate turn-taking from speaker attribu- tion. That separation is instead provided by the text-LLM references in Table 3. Turn-Based turn- taking is at chance given transcripts alone (ASR- only, 49.22) but near-ceiling with oracle speaker labels (SpkD-ASR, 98.44), indicating the deficit lies in recovering speaker attribution from audio rather than in reasoning about turns. 6.2 Multiparty-aware Prompting Given the generally poor performance of voice agents in multiparty settings, we investigate whether this limitation can be trivially resolved through prompting. In our primary evaluation (Sec. 5.1), we employ the default system prompt for all applicable voice agents. For this ablation, we design multiparty-aware system prompts that explicitly inform the agent that âthere may be mul- tiple speakers in the conversation.â Both prompts are provided verbatim in Appendix E.1. Table 5 reports the comparison for the five real- time models evaluated under both prompts; the default-prompt rows match those in Table 3. The Model Und. Disc. Turn-Based OAAppTTApp GPT-Realtime (default)14.00 53.13 48.44 28.13 GPT-Realtime (MP-aware)16.39 50.02 49.61 26.17 Gemini-2.5 Live (default)10.58 36.72 48.00 20.00 Gemini-2.5 Live (MP-aware) 13.63 85.16 48.39 18.95 Gemini-3.1 Live (default)4.90 57.03 52.57 38.99 Gemini-3.1 Live (MP-aware)7.82 57.03 47.41 25.96 Ultravox (default)21.36 39.06 46.09 31.25 Ultravox (MP-aware)19.44 39.06 51.95 12.50 PersonaPlex (default)3.140.00 51.174.69 PersonaPlex (MP-aware)3.130.00 51.955.47 Table 5: Effect of inference-time system prompts (de- fault vs. multi-party-aware) on MP-Bench scores (%). Abbreviations: OA (Overall Accuracy), App (Appro- priateness Accuracy), and T (Turn-Taking Accuracy). Und. and Disc. indicates understanding and behavior tasks from Discussion scenario respectively. most significant improvements are concentrated in Discussion scenario, where explicitly remind- ing agents of multiple speakers yields substantive gains for several systems. In contrast, overall per- formance either stagnates or degrades in the Turn- Based Games scenario. Since T Acc. remains near the 50% random baseline under both prompts, the drop in Appropriateness Accuracy cannot be attributed to a change in turn-taking behavior. In- stead, it reflects weaker rule compliance in the re- sponses agents produce when they do choose to speak: under the multiparty-aware prompt, the con- tent of their utterances departs more often from the game rules. 7 Human Evaluation In addition to LLM Judge results, we also provide a small scale of human evaluation to make sure the alignment between human and LLM Judge is high. We recruited human annotators to independently score a subset of 280 evaluation data entries equally sampled across all 14 settings from behavior tasks from discussion scenario, given that the appropri- ateness accuracy of this task is the most subjective compared to other metrics that all have a specific ground truth. Human annotators are given the same transcripts and also an instruction simplified from the LLM judge prompt. More details of human evaluation setup is provided in Appendix B. We measure Krippendorffâsαand CohenâsÎș. The former is an inter-annotator agreement co- efficient computed over a unified rater pool con- sisting of the three human annotators. The latter is the agreement between the LLM judgeâs pre- dicted label and the per-item majority vote of the three human annotators. The results show reason- ably strong alignment across the metrics (Krippen- dorffâsα= 0.745, CohenâsÎș= 0.646), suggesting that the LLM-as-a-judge evaluation is reliable and substantially aligned with human evaluation. 8 Conclusion We present MP-Bench, the first step toward bench- marking voice agents as active participants within multiparty interactions. Our evaluations reveal a stark capability gap: while non-realtime mod- els demonstrate robust comprehension, their per- formance collapses to near-random levels during active participation. Meanwhile, realtime agents struggle broadly across both understanding and behavioral metrics. Crucially, our ablation stud- ies confirm that these failures stem fundamentally from the inherent complexities of multiparty dy- namics rather than task-specific nuances or gen- eral instruction-following deficits. Furthermore, we demonstrate that these severe shortcomings can- not be circumvented via trivial interventions like explicit system prompting. Ultimately, MP-Bench exposes multiparty awareness as a foundational missing capability, establishing a clear mandate for the next generation of conversational AI. 9 Limitation Scope.MP-Bench is a controlled first step rather than a full test of open-ended participation. Be- cause each task evaluates a single decision point at the end of a pre-recorded clip, we do not assess multi-turn engagement, barge-in, or backchannel- ing. Additionally, an agentâs response is only reg- istered if ASR detects a word onset after the input audio concludes. Any utterances by full-duplex models during the clip are ignored by our metrics. Consequently, MP-Bench evaluates voice agents under a conservative setting where competence is a necessary, but not sufficient, condition for natural multiparty interaction. Synthetic Audio. Controlled speech synthesis enables an objective ground truth for turn-taking that spontaneous, real-world recordings lack. How- ever, this introduces a domain gap (Table 6). Our benchmark features no overlapping speech (0% vs. AMIâs24.5%), background noise, reverbera- tion, long pauses, or disfluencies, though inter- turn timing for active transitions remains natural- istic (0.40s vs.0.43s). Because existing agents already struggle on these cleaned inputs, our sim- plified setup provides a conservative lower bound on model limitations. Incorporating real or actor- voiced recordings remains an important direction for future work. References Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, and Zhizheng Wu. 2024. Sd-eval: a benchmark dataset for spoken dialogue understanding beyond words. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS â24, Red Hook, NY, USA. Curran Associates Inc. Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung yi Lee, Karen Livescu, and Shinji Watanabe. 2025a. On the landscape of spoken lan- guage models: A comprehensive survey. Transac- tions on Machine Learning Research. Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruom- ing Pang, and Shinji Watanabe. 2025b. Talking turns: Benchmarking audio foundation models on turn-taking dynamics. In The Thirteenth Interna- tional Conference on Learning Representations. Kratika Bhagtani, Mrinal Anand, Yu Chen Xu, and Amit Kumar Singh Yadav. 2026. Speak or stay silent: Context-aware turn-taking in multi-party dialogue. Preprint, arXiv:2603.11409. Kai-Wei Chang, En-Pei Hu, Chun-Yi Kuan, Wenze Ren, Wei-Chih Chen, Guan-Ting Lin, Yu Tsao, Shao-Hua Sun, Hung yi Lee, and James Glass. 2025. Game- time: Evaluating temporal dynamics in spoken lan- guage models. arXiv preprint arXiv:2509.26388. Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. 2024. Voicebench: Benchmarking llm-based voice assistants. arXiv preprint arXiv:2410.17196. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evalua- tions? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 15607â15631, Toronto, Canada. Association for Computational Linguistics. Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King. 2025. VoxEval: Benchmarking the knowledge under- standing capabilities of end-to-end spoken language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 16735â16753, Vienna, Austria. Association for Computational Linguistics. Alexandre DĂ©fossez, Laurent MazarĂ©, Manu Orsini, AmĂ©lie Royer, Patrick PĂ©rez, HervĂ© JĂ©gou, Edouard Grave, and Neil Zeghidour. 2024.Moshi: A speech-text foundation model for real-time dialogue. Preprint, arXiv:2410.00037. Yuhao Du, Qianwei Huang, Guo Zhu, Zhanchen Dai, Shunian Chen, Qiming Zhu, Yuhao Zhang, Li Zhou, Benyou Wang, and Haizhou Li. 2025.Mtalk- bench: Evaluating speech-to-speech models in multi- turn dialogues via arena-style and rubrics protocols. Preprint, arXiv:2508.18240. Yuanbo Fang, Haoze Sun, Jun Liu, Tao Zhang, Zenan Zhou, Weipeng Chen, Xiaofen Xing, and Xiangmin Xu. 2025. S2sbench: A benchmark for quantifying intelligence degradation in speech-to-speech large language models. Preprint, arXiv:2505.14438. Fixie AI. 2024.Ultravox Realtime: Build voice ai agents that work anywhere.https://docs. ultravox.ai/overview. Yuan Ge, Saihan Chen, Jingqi Xiao, Xiaoqian Liu, Tong Xiao, Yan Xiang, Zhengtao Yu, and Jingbo Zhu. 2025. Flexi: Benchmarking full-duplex human-llm speech interaction. arXiv preprint arXiv:2509.22243. Google. 2025. Gemini Live API overview.https: //ai.google.dev/gemini-api/docs/live. Google DeepMind. 2025a.Gemini 2.5:Push- ingthefrontierwithadvancedreason- ing,multimodality,long context,and next generationagenticcapabilities.https: //storage.googleapis.com/deepmind-media/ gemini/gemini_v2_5_report.pdf. Google DeepMind. 2025b.Gemini 3 Pro model card.https://deepmind.google/models/ model-cards/gemini-3-pro/. Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, and Yunzhong He. 2025. Audio multichallenge: A multi-turn evaluation of spoken dialogue systems on natural human interac- tion. Preprint, arXiv:2512.14865. Koji Inoue, Divesh Lala, Mikey Elmers, Keiko Ochi, and Tatsuya Kawahara. 2025. An LLM benchmark for addressee recognition in multi-modal multi-party dialogue. In Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology, pages 330â334, Bilbao, Spain. Association for Com- putational Linguistics. Dhruv Jain, Harshit Shukla, Gautam Rajeev, Ashish Kulkarni, Chandra Khatri, and Shubham Agarwal. 2026. Voiceagentbench: Are voice assistants ready for agentic tasks? Preprint, arXiv:2510.07978. Yejin Kwon, Taewoo Kang, Hyunsoo Yoon, and Chang- ouk Kim. 2025.M3-slu: Evaluating speaker- attributed reasoning in multimodal large language models. Preprint, arXiv:2510.19358. Yangzhuo Li, Shengpeng Ji, Yifu Chen, Tianle Liang, Haorong Ying, Yule Wang, Junbo Li, Jun Fang, and Zhou Zhao. 2026.Wavbench: Benchmark- ing reasoning, colloquialism, and paralinguistics for end-to-end spoken dialogue models. Preprint, arXiv:2602.12135. Guan-Ting Lin, Chen Chen, Zhehuai Chen, and Hung yi Lee. 2026a. Full-duplex-bench-v3: Benchmarking tool use for full-duplex voice agents under real-world disfluency. Preprint, arXiv:2604.04847. Guan-Ting Lin, Shih-Yun Shan Kuan, Jiatong Shi, Kai- Wei Chang, Siddhant Arora, Shinji Watanabe, and Hung-yi Lee. 2026b. Full-duplex-bench-v2: A multi- turn evaluation framework for duplex dialogue sys- tems with an automated examiner. arXiv preprint arXiv:2510.07838. Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, and Hung-yi Lee. 2025a. Full-duplex-bench v1. 5: Evaluating overlap han- dling for full-duplex speech models. arXiv preprint arXiv:2507.23159. Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H Liu, and Hung-yi Lee. 2025b. Full-duplex-bench: A bench- mark to evaluate full-duplex spoken dialogue mod- els on turn-taking capabilities. arXiv preprint arXiv:2503.04721. Yueqian Lin, Zhengmian Hu, Qinsi Wang, Yudong Liu, Hengfan Zhang, Jayakumar Subramanian, Nikos Vlassis, Hai Helen Li, and Yiran Chen. 2025c. Voice evaluation of reasoning ability: Diagnosing the modality-induced performance gap. Preprint, arXiv:2509.26542. Heyang Liu, Yuhao Wang, Ziyang Cheng, Hongcheng Liu, Yiqi Li, Yixuan Hou, Ronghua Wu, Qunshan Gu, Yanfeng Wang, and Yu Wang. 2026.Vo- calbench: Benchmarking the vocal conversational abilities for speech interaction models. Preprint, arXiv:2505.15727. Jingwen Liu, Kan Jen Cheng, Jiachen Lian, Akshay Anand, Rishi Jain, Faith Qiao, Robin Netzorg, Huang- Cheng Chou, Tingle Li, Guan-Ting Lin, and Gopala Anumanchipalli. 2025. Emo-reasoning: Benchmark- ing emotional reasoning capabilities in spoken di- alogue systems. In 2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). Vihaan Nama, Shreya Mendi, Zian Ye, and Brinnae Bent. 2026. When2speak: A dataset for tempo- ral participation and turn-taking in multi-party con- versations for large language models. Preprint, arXiv:2605.05626. OpenAI. 2025a.GPT-5 system card. Preprint, arXiv:2601.03267. OpenAI.2025b.Introducinggpt-realtime andRealtimeAPIupdatesforproduction voice agents.https://openai.com/index/ introducing-gpt-realtime/. Yizhou Peng, Yi-Wen Chao, Dianwen Ng, Yukun Ma, Chongjia Ni, Bin Ma, and Eng Siong Chng. 2025. Fd-bench: A full-duplex benchmarking pipeline designed for full duplex spoken dialogue systems. Preprint, arXiv:2507.19040. Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam, and Boris Ginsburg. 2023. Fast conformer with linearly scalable attention for effi- cient speech recognition. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1â8. Rajarshi Roy, Jonathan Raiman, Sang-gil Lee, Teodor- Dumitru Ene, Robert Kirby, Sungwon Kim, Jae- hyeon Kim, and Bryan Catanzaro. 2026. Person- aPlex: Voice and role control for full duplex conver- sational speech models. Preprint, arXiv:2602.06053. Chengyou Wang, Hongfei Xue, Guojian Li, Zhixian Zhao, Shuiyuan Wang, Shuai Wang, Xin Xu, Hui Bu, and Lei Xie. 2026a. Full-duplex interaction in spoken dialogue systems: A comprehensive study from the icassp 2026 humdial challenge. Preprint, arXiv:2604.21406. Shuai Wang, Zhaokai Sun, Zhennan Lin, Chengyou Wang, Zhou Pan, and Lei Xie. 2025. Msu-bench: To- wards understanding the conversational multi-talker scenarios. Preprint, arXiv:2508.08155. Wenfu Wang, Chenxing Li, Liqiang Zhang, Yiyang Zhao, Yuxiang Zou, Hanzhao Li, Mingyu Cui, Hao Zhang, Kun Wei, Le Xu, Zikang Huang, Jiajun Xu, Jiliang Hu, Xiang He, Zeyu Xie, Jiawen Kang, Youjun Chen, Meng Yu, Dong Yu, and 7 others. 2026b.Covo-audio technical report. Preprint, arXiv:2602.09823. Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. 2024. Freeze-omni: A smart and low latency speech-to- speech dialogue model with frozen llm. Preprint, arXiv:2411.00774. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025.Qwen3-Omni technical report. Preprint, arXiv:2509.17765. Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, and Xie Chen. 2025. URO- bench: Towards comprehensive evaluation for end- to-end spoken dialogue models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17211â17242, Suzhou, China. Associa- tion for Computational Linguistics. Shuwen Yang, Ming Tu, Andy T. Liu, Xinghua Qu, Hung yi Lee, Lu Lu, Yuxuan Wang, and Yonghui Wu. 2026. Paras2s: Benchmarking and aligning spoken language models for paralinguistic-aware speech-to- speech interaction. In The Fourteenth International Conference on Learning Representations. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. GLM-4-Voice: Towards intelligent and human-like end-to-end spoken chatbot. Preprint, arXiv:2412.02612. A Details of MP-Bench 2030405060708090 Duration (seconds) 0 20 40 60 80 100 Count Task Audio Duration Distribution Figure 3: Task audio duration distribution. Rose David Mia Emma Ben Lily Karen Luke Adam Leo Anna Jack Speaker 0 2 4 6 8 10 12 Share of all turns (%) Speaker Name Distribution Over Entire Benchmark Figure 4: Overall Speaker Name Distribution. 468101214 Turns in Task 0 25 50 75 100 125 150 175 200 Count Number of Turns per Task Figure 5: Number of Turns per Task. A.1 Statistics We present the core structural statistics of the MP- Bench dataset. On average, each task recording Rose Mia Adam Leo Anna Ben David Luke Jack Emma Lily Karen Speaker 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Turns per task (mean ± std) Turns per Speaker (Top 20 by Mean) Figure 6: Number of Turns per Speaker for each Task. lasts45.6± 11.5seconds and comprises8.4± 2.1 conversational turns. Within individual tasks, each participant contributes an average of2.8±1.1turns. Across the entire dataset, speaker participation is well-balanced, with each unique speaker account- ing for a mean of647.3± 124.7total turns, repre- senting approximately an8.3%±1.6%share of the overall dialogue. Comprehensive visualizations of these metricsâincluding the audio duration distri- bution (Fig. 3), overall speaker distribution (Fig. 4), and per-task turn frequencies (Figs. 5 and 6)âare provided for further detail. A.2 Speaker Attribution The initial step in constructing the benchmark is to establish the speaker profiles for the dialogue. Each dialogue instance in our benchmark com- prises 3 simulated human speakers, resulting in a four-participant conversation when the voice agent is included. To minimize subjectivity in our evalu- ation tasks, we ensure that the three speakers sam- pled for any given instance possess highly distinct vocal characteristics. Additionally, we provide an explicit textual description of each speakerâs acous- tic profile. In practice, we curate 12 distinct voices from the ElevenLabs library and categorize them along two primary axes: 1) gender (male or female) and 2) pitch (high, medium, or low). To maximize the difference between the speakers within the same conversation, we do not allow speakers with high medium and low pitch of the same gender to be selected, as it might be tricky for evaluation. As a result, to sample a valid speaker combination, we either select 2 male 1 female or 1 male 2 female speakers. For each gender, we select 1 or 2 random distinct pitch. For example, (male high pitched, male medium pitched, female medium pitch) is a valid combination. Other than the two primary axes, we also include several special description for each speaker voice. By concatenating all of them, we create voice characteristic description for each voice. This description will be used as a information when we create tasks in the following sections. A.3 Audio Quality Analysis To provide readers with audio quality of MP-Bench, we calculated several statistics of our generated audio and the comparison with AMI corpus (as a real Multiparty recording reference). We read Table 6 in two ways, and we are careful not to overclaim: âąCleanliness / intelligibility. The audio is clean and highly intelligible: WER is 6.6% vs. 21.7% on real AMI recordings, and UT- MOS is high. We do not interpret UTMOS >AMI as our audio being âmore natural than real conversationâ. Clean synthesized speech is expected to score above spontaneous, noisy meeting audio on UTMOS. We read these two numbers only as evidence that the audio is free of TTS artifacts, so model failures cannot be attributed to audio quality. Speaker embed- dings also show large, consistent separation (inter-speaker 0.159 vs. intra 0.571; separation 0.412, larger than AMIâs 0.299), confirming speakers are clearly distinguishable. âą Controlled simplifications. Our stimuli dif- fer from real meetings in three deliberate ways: (i) the complete absence of overlap- ping speech (0% vs. AMIâs 24.5%); (i) the absence of long âthinkingâ pauses (see the gap-duration rows below); and (i) the ab- sence of background noise and reverberation (part of why AMIâs WER is far higher than ours). Our inter-turn timing for active transi- tions is itself naturalistic. We report gaps<1 s separately because AMI, as a real meeting corpus, contains a large fraction (56.6%) of long silences (participants pausing to think) that inflate its overall mean gap to 1.8 s. Restricting to gaps<1 s isolates comparable active- turn transitions: there MP-Bench (0.40 s) and AMI (0.43 s) are closely matched, indicating that the overall difference reflects our deliberate omission of long pauses rather than unnaturally fast turn- taking. MetricMP-Bench (ours)AMI (real) UTMOS (quality, 1â5)3.88± 0.556 (4.05)2.83± 0.725 (2.97) ASR WER (%)6.6± 6.0 (5.0)21.7± 5.4 (21.5) Intra-speaker similarity0.571± 0.022 (0.567)0.387± 0.067 (0.391) Inter-speaker similarity0.159± 0.024 (0.153)0.088± 0.031 (0.083) Conversational turn gap duration (s)0.412± 0.235 (0.405)1.796± 2.634 (1.170) Conversational turn gap duration, gaps < 1 s (s)0.401± 0.177 (0.405)0.434± 0.278 (0.400) Overlap time (%)0.0± 0.0 (0.0)24.5± 10.4 (23.3) Mean turn duration (s)2.91± 0.937 (2.81)2.65± 0.480 (2.62) Table 6: Comparison of metrics between MP-Bench and AMI. A.4 List of Topics for Discussion Scenario Topic List: Capital Cities, National Currencies, U.S. State Capitals, Planets of the Solar System, Animal Group Names, Animal Baby Names, Hu- man Anatomy Basics, Simple Arithmetic, Colors of the Rainbow, Days of the Week, Months of the Year, Zodiac Signs, Birthstones, National Flags, Fa- mous Landmarks, Continents, Oceans of the World, Major Rivers, Mountain Ranges, US Presidents, In- ventions and Inventors, Famous Explorers, World War I Dates, Greek Mythology Gods, Geomet- ric Shapes, Units of Measurement, Freezing and Boiling Points, States of Matter, Primary and Sec- ondary Colors, Musical Instruments, Famous Clas- sical Composers, Popular Nursery Rhymes, Fairy Tales, Disney Movies, Superheroes and Powers, Harry Potter Houses, Star Wars Characters, Game of Thrones Houses, Oscar-Winning Movies, Sit- com Characters, Famous Movie Quotes, Celebrity Real Names, Pop Music Artists, The Beatles Mem- bers, Musical Bands and Lead Singers, Sports Team Mascots, Olympic Host Cities, Rules of Soc- cer, Rules of Basketball, Rules of Baseball, Tennis Scoring, Golf Terms, Bowling Terms, Chess Pieces, Video Game Characters, Pokemon Types, Ingredi- ents in Common Cocktails, Coffee Types, Tea Vari- eties, Fruit and Vegetable Identification, Spices and Herbs, Types of Pasta, Breakfast Foods, Fast Food Chains, Candy Bars, Soda Brands, Car Brands and Logos, Compass Directions, Time Zones, So- cial Media Platforms, Internet Slang, Common Id- ioms, Synonyms and Antonyms, Collective Nouns, Common Phobias, Holidays and Dates, Seasons, Weather Phenomena, Natural Disasters, Types of Clouds, Tree Types, Dog Breeds, Cat Breeds, Farm Animal A.5 Tasks Examples A.5.1 Understanding Task for Discussion Scenario Speakers: Adam (male) - voice: relatively low male pitch, male, slow, relaxed David (male) - voice: relatively high male pitch, male, nosy Rose (female) - voice: female, clear, clean Transcripts: Adam:Does anyone know the actual difference between cilantro and coriander? Rose:I am pretty sure they are entirely different species, Adam. Like, one is an herb and one is a spice seed. David:No, I cook with them all the time. They are the same plant. Cilantro is the leaves and coriander is the seeds. Rose:That sounds too simple. I think they are cousins, not the same plant. David:I am telling you, Rose, it is the Coriandrum sativum plant. Adam:Okay, David is sounding very confident with the Latin names there, but I am still confused. Rose:Hey AI Assistant. Please answer the question. What is the name of the speaker with a relatively low male pitch ? A.5.2 Behavior Task for Discussion Scenario Speakers: Adam (male) - voice: relatively low male pitch, male, slow, relaxed David (male) - voice: relatively high male pitch, male, nosy Rose (female) - voice: female, clear, clean Transcripts: Adam:Does anyone know the actual difference between cilantro and coriander? Rose:I am pretty sure they are entirely different species, Adam. Like, one is an herb and one is a spice seed. David:No, I cook with them all the time. They are the same plant. Cilantro is the leaves and coriander is the seeds. Rose:That sounds too simple. I think they are cousins, not the same plant. David:I am telling you, Rose, it is the Coriandrum sativum plant. Adam:Okay, David is sounding very confident with the Latin names there, but I am still confused. Rose:Let's settle this. AI Assistant, what is the answer and who has the right information here? A.5.3 Behavior Task for Turn-Based Game Voice Agent should take turn Speakers Leo (male) - voice: relatively low pitch, male, resonant throaty David (male) - voice: relatively high pitch, male, nosy Emma (female) - voice: low pitch, female, intonation-rich Transcripts: Leo:Hey everyone, I've got a fun mental exercise for us to try. It's a number response game involving two teams. Emma:Oh? How does it work, Leo? Leo:It's simple. We have Group 1 and Group 2. Each has a Leader and a Follower. If the Leader of Group 1 says a number, the Follower of Group 2 has to say the next integer. David:And if the Leader of Group 2 speaks? Leo:Then the Follower of Group 1 has to respond with the next integer. It's criss-crossed. David:Okay, sounds easy enough if we pay attention. Leo:Great. So, I will take the role of Group 1 Leader, and the AI Assistant will be the Group 1 Follower. Emma, you are the Group 2 Leader. Emma:Understood. That means David is my Follower in Group 2? Leo:Exactly, David is Group 2 Follower. Everyone ready? Game start! Leo:5 David:6 Leo:12 Voice Agent shouldnât take turn Speakers Leo (male) - voice: relatively low pitch, male, resonant throaty David (male) - voice: relatively high pitch, male, nosy Emma (female) - voice: low pitch, female, intonation-rich Transcripts: Leo:Hey everyone, I've got a fun mental exercise for us to try. It's a number response game involving two teams. Emma:Oh? How does it work, Leo? Leo:It's simple. We have Group 1 and Group 2. Each has a Leader and a Follower. If the Leader of Group 1 says a number, the Follower of Group 2 has to say the next integer. David:And if the Leader of Group 2 speaks? Leo:Then the Follower of Group 1 has to respond with the next integer. It's criss-crossed. David:Okay, sounds easy enough if we pay attention. Leo:Great. So, I will take the role of Group 1 Leader, and the AI Assistant will be the Group 1 Follower. Emma, you are the Group 2 Leader. Emma:Understood. That means David is my Follower in Group 2? Leo:Exactly, David is Group 2 Follower. Everyone ready? Game start! Leo:5 David:6 Emma:12 B Details of Human Evaluation We recruit 13 graduate-student annotators from two universities with backgrounds in speech and lan- guage processing. All annotators agree to partici- pate in this human evaluation study and they agree their annotation data will be used for this study. Each annotator works on a batch of 30 items per session. The annotation guideline instructs that a response is considered consistent only if it (1) ex- plicitly states the modelâs own claim or answer and (2) explicitly identifies the speaker whose claim it agrees with; if either element is missing the re- sponse is judged inconsistent. C Robustness of LLM Judge across model families To test whether our GPT-5 labels are idiosyn- cratic, we re-judged a stratified subsample of 720 items (UnderstandingâDiscussion: 14 settingsĂ 30; BehaviorâDiscussion: 6 settingsĂ50; seed 20260712) with three additional judges from dif- ferent families. (Other metrics like Turn-Taking is judged by timestamps so is independent of the LLM judge.) By scenario, Understanding is especially stable (Îș= 0.83 / 0.85 / 0.90 for Flash-Lite / Pro / Claude) and Behavior remains substantial (Îș= 0.74 / 0.71 / 0.78). All three judges are only mildly more SettingUnd. Overall Acc.Spk. Name Acc.Disc. App Acc.TB T Acc. Distinct voices40.0% (20/50) [27.6, 53.8]45.0% (18/40) [30.7, 60.2]75.0% (15/20) [53.1, 88.8]50.0% (15/30) [33.2, 66.8] Similar voices4.0% (2/50) [1.1, 13.5]0.0% (0/40) [0.0, 8.8]45.0% (9/20) [25.8, 65.8]43.3% (13/30) [27.4, 60.8] Table 7: Covo-Audio, distinct- vs. similar-voice subsets (95% Wilson intervals in %). Disc. App = Discussion appropriateness; TB T = Turn-based turn-taking. JudgenExact matchCohenâs Îș Gemini-3.1-Flash-Lite72088.9%0.78 Gemini-3.1-Pro72088.8%0.77 Claude Opus 4.672091.7%0.83 Table 8: Agreement across LLM judges. lenient than GPT-5, with no systematic flips that would reorder model rankings. D Ablation for high voice similarity speakers for MP-Bench In our benchmark, we chose distinct voices delib- erately, for two reasons: 1.As a first step, MP-Bench targets the core multiparty ability, so we disentangle it from speaker-diarization difficulty; distinct voices keep diarization from being the bottleneck. 2.It enables objective, unambiguous question design (e.g., âwhat did the higher-pitched speaker suggest?â), which is hard to grade objectively when voices are too similar. As a ablation, we additionally ran a similar- voice ablation on the strongest realtime under- standing model (Covo-Audio), excluding questions that explicitly reference pitch to avoid ambiguity. As shown in Table 7, when voice separability is reduced, the abilities that depend on acoustic attri- bution collapse: Understanding drops 40% to 4%, speaker-name accuracy drops 45% to 0%, and Dis- cussion appropriateness drops 75%â45%. Turn- based turn-taking remains near/below the 50% chance baseline throughout. Thus our negative find- ings of weak multiparty understanding and near- chance implicit/negative turn-taking persist, and for understanding become more severe, as voices become harder to tell apart. E LLM Prompt Templates E.1 Inference-Time System Prompts for Realtime Models The realtime audio models in our experiments are driven by one of the two short voice-assistant sys- tem prompts shown below. The default prompt is used in all main results (Table 3); the multi-party- aware variantâwhich differs only in a single added sentence (âBe aware that there may be multiple speakers in the conversation.â)âis used solely for the prompt ablation in Table 5. Default System Prompt You are a helpful voice AI assistant. Keep your responses concise and conversational since they will be spoken aloud. Respond naturally to whatever the user says or asks. Multi-Party-Aware System Prompt You are a helpful voice AI assistant. Be aware that there may be multiple speakers in the conversation. Keep your responses concise and conversational since they will be spoken aloud. Respond naturally to whatever the users say or ask. E.2 LLM Prompt for Scenario Generation Discussion Scenario Generation You are a helpful assistant. Please generate num_diags_per_generation naturally spoken 3 people conversation. The names of the 3 people are: speaker_names_str In the beginning of the conversation, the first person ask a factual question based on the topic of "topic_list_str ". Then, all people give their answers in random order. (Speaker turns interleaved each other) At the end, one person redirect the question to "AI Assistant" and ask it to judge who is correct. The name of the AI assistant should be called "AI Assistant". Please regard "AI Assistant" as a human. Make sure that all speaker names are only mentioned once in one of the speaker's spoken content during the conversation. The same speaker can speaks multiple turns. The output is a list of generated conversations. Each conversation contains: 1. Topic of the factual question asked by the first person. 2. conversation (a list of speaker names together with utterances) The output should follow the JSON format. ["topic": ..., conversation": ["speaker": ..., "content": ... , "speaker": ..., "content": ... , ...], "topic ": ..., conversation": ["speaker": ..., "content": ... , "speaker": ..., "content": ... , ...] ...] Turn-based Game Scenario Generation You are a helpful assistant. Please generate num_diags_per_generation naturally spoken 4 people conversation. The speakers are: speaker_names_str and " AI Assistant". Please regard "AI Assistant" as a human participant too. In the conversation, people are playing a game. In the game, two people formed a group so we have Group1 and Group2. In each group, one person is the leader and the other is the follower. game_rules In the beginning of the conversation, the first person(should not be "AI Assistant ") initiate to play the game and also introduce the rules. Afterwards, the first person assign people into Group1, Group2 and also leaders and followers. During the introduction of game rule, the two other people (excluding "AI Assistant") could also interact with the first person. After that, the first person anounces game start without reveal anything related to who start the game first. Throughout the generated conversation, "AI Assistant" should not speak. Given the following group and role assignment: Group1 Leader: group1_leader Group1 Follower: group1_follower Group2 Leader: group2_leader Group2 Follower: group2_follower Please generate the conversation until the first person anounces game start. game_pairs_instruction Make sure that all speakers (excluding "AI Assistant")'s name are being addressed in the game rule introduction phase. The same speaker can speaks multiple turns. The output contains a list of generated spoken conversations and each example contains the list of speaker names, answer pairs and a list of utterances of the conversation. The output should follow the JSON format. ["speaker_names:" ["...", "...", "..."], " answer_pairs": [["...", "..."], ...] " conversation": ["speaker": ..., " content"... , "speaker": ..., " content"... , ...], "speaker_names :" ["...", "...", "..."], "answer_pairs ": [["...", "..."], ...] "conversation": ["speaker": ..., "content"... , " speaker": ..., "content"... , ...],...] E.3 LLM Prompt for Validating Generated Main Dialogues Validation without Speaker Names You are a helpful assistant. You are given a multiparty spoken conversation transcript. Please predicted the speaker names of each utterances. Input format (JSON): ["content": ..., "content": ..., ...] Output format (JSON): ["speaker": ..., "content": ..., " speaker": ..., "content": ..., ...] Now given: Input: conversation_tranwo_spks Output: Validation with Anonymized Speaker Names You are a helpful assistant. You are given a multiparty spoken conversation transcript. For each utterances, the speaker's names are anonymized as Speaker1, Speaker2, Speaker3. Please predicted the real speaker names of Speaker1, Speaker2, Speaker3 based on the given information. Input format (JSON): ["speaker_anonymized": "...", content": ..., "speaker_anonymized": "...", " content": ..., ...] Output format (JSON): "Speaker1": "...", "Speaker2": "...", " Speaker3": "..." Now given: Input: conversation_tranw_anonymized_spks Output: E.4 LLM-as-a-Judge Prompts This subsection lists the four LLM-as-a-judge prompts that produce the numbers reported in the main table and ablations. All four are run with GPT-5 as the judge. The first prompt grades the Understanding task, the next two grade the two Behavior sub-tasks (Discussion consistency and Turn-Based rule compliance), and the last is the dedicated fuzzy-matching judge used to compute Spk. Name Acc. on the speaker-name slice of the Understanding QA pool. Understanding QA Judge You are a helpful assistant. You are given a meta-transcript of a multiparty spoken conversation together with a question, ground truth answer and a predicted answer. The question is related to the spoken conversation. The meta-transcript includes the speakers ( name,gender,voice_characteristics) that are in the conversation. The meta-transcript also includes each spoken turn in the conversation. Each turn contains information for speaker, content, addressee and coreferences. Please judge the correctness of the predicted answer. If the predicted answer says something like there is not enough information or refuse to give an answer, please consider it as incorrect. The input is given in the following format: Metatranscripts for Multiparty Spoken Conversation: "speakers": ["name": "...", "gender": "...", "voice_characteristics": "...", ...], "metatranscript": ["speaker": "...", "content": "...", "addressee": ["...", ...], "coreferences": "...": "...", ..., ...] Question: ... Ground Truth Answer: ... Predicted Answer: ... Please output with the following format: "correct": true/false Now given Metatranscripts for Multiparty Spoken Conversation: metainformation Question: question_text Ground Truth Answer: answer_text Predicted Answer: llm_predicted_answer Output: Discussion Consistency Judge You are a helpful assistant. You are given a meta-transcript of a multiparty spoken conversation together with a model response The model's name is "AI Assistant" and is one of the participant in the spoken conversation. The meta-transcript includes the speakers ( name,gender,voice_characteristics) that are in the conversation. The meta-transcript also includes each spoken turn in the conversation. Each turn contains information for speaker, content, addressee and coreferences. Please judge whether the model's response is consistent with the spoken conversation . The model's response is considered as consistent if the person it identified as correct also have the same claim as its own answer. Do not judge the consistency of the model's response based on factual correctness. The input is given in the following format: Metatranscripts for Multiparty Spoken Conversation: "speakers": ["name": "...", "gender": "...", "voice_characteristics": "...", ...], "metatranscript": ["speaker": "...", "content": "...", "addressee": ["...", ...], "coreferences": "...": "...", ..., ...] Model's response: ... Please output with the following format: "consistent": true/false Now given Metatranscripts for Multiparty Spoken Conversation: metatranscript Model's response: llm_predicted_answer Output: Turn-Based Game Judge (with chain-of- thought reasoning) You are a helpful assistant. You are given a transcript of a multiparty spoken conversation together with a model response. The model's name is "AI Assistant" and is one of the participants in the spoken conversation. The people in the conversation are playing a turn-based game. The game rules are described within the conversation itself . The transcript includes the speakers (name, gender, voice_characteristics) and each spoken turn (speaker, content). Your task is to judge whether the model's response is consistent with the game being played. You MUST follow these steps in order and include your reasoning: Step 1: Identify the game and its rules from the conversation. Step 2: Identify ALL group/team assignments -- specifically, who is the Leader and who is the Follower of each group. Step 3: Determine which Leader's speech triggers the AI Assistant to respond. In a cross-group reaction game, Group X's Leader triggers Group Y's Follower (NOT Group X's own Follower). Step 4: Look at the LAST speaker before the model's response. Determine whether that speaker is the Leader who triggers the AI Assistant. Step 5: Based on Step 4: - If the last speaker IS the Leader who triggers the AI Assistant -> the AI SHOULD respond. Check if the response follows the game rules (e.g., says a random word in a word game). - If the last speaker is NOT the Leader who triggers the AI Assistant -> the AI should remain SILENT. Any non-empty response means it acted out of turn. - An empty response means the model remained silent. Focus ONLY on the model's final response content. Even if the model's reasoning or explanation contains errors, as long as the final action follows the game rules, consider it consistent. Do not judge the model's response based on factual correctness outside of the game context. === EXAMPLE 1 (consistent = true) === Transcript: "speakers": ["name": "Alice", "gender": "female", "voice_characteristics": " clear", "name": "Bob", "gender": " male", "voice_characteristics": "deep "], "metatranscript": ["speaker": " Alice", "content": "OK new plan. Two groups. Group 1: Leader = Alice, Follower = Bob. Group 2: Leader = Bob, Follower = AI Assistant. When Group 1's Leader speaks a word, Group 2's Follower must respond with a random word. When Group 2's Leader speaks, Group 1's Follower responds. Ready? Go!", " speaker": "Alice", "content": "Sun"] Model's response: Moon Reasoning: - Game: Cross-group word reaction game. - Group 1: Leader = Alice, Follower = Bob. - Group 2: Leader = Bob, Follower = AI Assistant. - Trigger rule: Group 1's Leader (Alice) triggers Group 2's Follower (AI Assistant). - Last speaker before model's response: Alice (Group 1's Leader). - Alice IS the Leader who triggers the AI Assistant -> the AI SHOULD respond. - The AI responded with "Moon", which is a valid random word. - Verdict: consistent = true. === EXAMPLE 2 (consistent = false) === Transcript: "speakers": ["name": "Lily", "gender": " female", "voice_characteristics": "clear ", "name": "David", "gender": "male ", "voice_characteristics": "warm", "name": "Jack", "gender": "male", " voice_characteristics": "deep"], " metatranscript": ["speaker": "Lily", " content": "Let's play a reaction game! Group 1: Leader = Lily, Follower = David . Group 2: Leader = Jack, Follower = AI Assistant. If Group 1's Leader speaks, Group 2's Follower responds. If Group 2' s Leader speaks, Group 1's Follower responds. Go!", "speaker": "Jack", " content": "Apple"] Model's response: Banana Reasoning: - Game: Cross-group word reaction game. - Group 1: Leader = Lily, Follower = David. - Group 2: Leader = Jack, Follower = AI Assistant. - Trigger rule: Group 1's Leader (Lily) triggers Group 2's Follower (AI Assistant). Group 2's Leader (Jack) triggers Group 1's Follower (David). - Last speaker before model's response: Jack (Group 2's Leader). - Jack is the AI Assistant's OWN group's Leader. Jack's speech triggers David ( Group 1's Follower), NOT the AI Assistant. - The AI should have remained silent, but it responded with "Banana". - Verdict: consistent = false. === END OF EXAMPLES === The input is given in the following format: Transcript for Multiparty Spoken Conversation: "speakers": ["name": "...", "gender": "...", "voice_characteristics": "...", ...], "metatranscript": ["speaker": "...", "content": "...", ...] Model's response: ... Please output a JSON object with your reasoning and judgment: "reasoning": "your step-by-step analysis here", "consistent": true/false Now given Transcript for Multiparty Spoken Conversation: metatranscript Model's response: llm_predicted_answer Output: Speaker-Name Fuzzy Match Judge (system prompt) You are a strict evaluator for a speaker- name identification task. The model was asked a question about a multiparty spoken conversation (e.g., " What is the name of the speaker with a warm voice?"). The ground truth answer is a single speaker name. The model heard the conversation as audio and responded via speech, which was then transcribed by ASR. The ASR transcription may contain errors: misspelled names (e.g., "Karen" -> " Caring", "Cairn"; "Leo" -> "Lioh"; "Lily " -> "Leely"), foreign-language greetings, filler words, or garbled text . Your task: 1. Read the model's transcribed response carefully. 2. Determine what speaker name the model intended as its answer. - Ignore greetings, hedging, filler, or refusal disclaimers. - If the model mentions ANY valid speaker name anywhere in the response -- even if surrounded by refusal language like "I can't identify speakers" -- extract that name. - If the model mentions multiple names, take the one most likely intended as the answer to the question (often the last or most confidently stated). - Account for phonetic / ASR spelling errors (e.g., "Caring" = "Karen", " Lioh" = "Leo"). 3. Compare the extracted name to the ground truth. IMPORTANT: A response like "I can't identify speakers, but it sounds like it might be Karen" should extract "Karen" -- the refusal disclaimer does NOT prevent name extraction. Only set extracted_name to null if absolutely NO valid speaker name (or phonetic variant) appears anywhere in the response. Valid speaker names: Adam, Anna, Ben, David, Emma, Jack, Karen, Leo, Lily, Luke, Mia , Rose. Output EXACTLY one JSON object (no markdown): "extracted_name": "<name or null>", " correct": true/false, "reasoning": "< brief>" Rules: - "extracted_name" must be one of the 12 valid names above, or null if the model gave no identifiable name guess anywhere in the response. - "correct" is true only when extracted_name matches the ground truth (case- insensitive). - Only set extracted_name to null if NO valid name or phonetic variant appears in the response at all. Speaker-Name Fuzzy Match Judge (user message) Question: question Ground truth answer: gt_name Model's transcribed response (may contain ASR errors): response F Use of Generative AI Tools We used generative AI assistants in three support- ing roles: polishing the writing, assisting with im- plementation and debugging of our evaluation and analysis code, and assisting with figure and table preparation. AI tools were not used to originate the research idea, design the benchmark, or interpret the results. All AI-assisted text, code, and figures were reviewed by the authors, who take full respon- sibility for the motivation, methodology, results, and any errors in this paper.