Paper deep dive
Inquesto Score: A reliability Protocol For Voice Agents
Massa Baali, Bhiksha Raj
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/28/2026, 3:08:56 AM
Summary
The paper introduces Inquesto Score (IS), a protocol for measuring the reliability of voice agents by evaluating the percentage of calls in a fixed, versioned population that achieve the caller's goal without functional failure. Unlike transcript-based benchmarks, IS incorporates audio-derived timing failures (e.g., talk-over, latency), tool trace verification, and semantic judgments from a pinned open-model judge. The protocol defines explicit failure severities (S1-S5) and provides diagnostic views for behavior, acoustic robustness, identity handling, and fairness. Experiments on 13 configurations of a reference billing-support agent demonstrate that transcript-only evaluations miss significant audio-derived failures and that deployment parameters like endpointing windows and serving infrastructure significantly impact reliability scores.
Entities (9)
Relation Signals (7)
Inquesto Score → measures → Talk-over
confidence 95% · Timing failures, including talk-over and delayed responses, are measured directly from audio
Inquesto Score → uses → Gemma 2 9B Instruct
confidence 95% · Judged events are evaluated by one pinned open model, Gemma-2-9B-Instruct [17]
Inquesto Score → evaluatesindomain → Billing Support
confidence 92% · We use billing support throughout because it combines task completion, conversational, timing and authorization failures in one controlled setting.
Caller Simulation → usesmodel → Qwen2.5-7B-Instruct
confidence 92% · The caller is simulated by Qwen2.5-7B-Instruct [24].
Inquesto Score → definesseverity → S3
confidence 90% · S3 functional failure ... In v0.1, S3 is the lowest severity considered sufficient to make a call fail.
Speech Recognition → usesmodel → Whisper
confidence 90% · transcribed by Whisper [23]
Voice Activity Detection → usesmodel → Silero VAD
confidence 88% · segmented by a voice-activity detector [22]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Voice agents are increasingly deployed in workflows where failed interactions can affect transactions, access, and other consequential outcomes, creating a need for reproducible and interpretable evaluation. We introduce Inquesto Score (IS), a protocol for measuring voice-agent reliability as the percentage of calls in a fixed, versioned evaluation population that achieve the caller's goal without a functional failure or worse. Rather than combining heterogeneous metrics, IS defines explicit failure events and severity levels and evaluates the deployed voice pipeline. Timing failures, including talk-over and delayed responses, are measured directly from audio, while semantic and state-dependent failures are evaluated using scenario predicates, tool traces, and a pinned open-model judge. Diagnostic views of behavior, acoustic robustness, identity handling, and speaker groups accompany the score without being combined into it. Inquesto Score v0.1 evaluates 30 scenarios, three acoustic conditions, four speaker groups, and 306 calls per agent across 13 configurations of a reference voice-agent system. Our evaluation shows that reliable measurement requires evidence beyond transcripts, explicit treatment of deployment conditions, and validation of the evaluators used to determine outcomes. We release the protocol, reference implementation, and evaluation records.
Tags
Links
- Source: https://arxiv.org/abs/2609.30514v1
- Canonical: https://arxiv.org/abs/2609.30514v1
Trouble viewing inline? Open PDF directly →
Full Text
23,088 characters extracted from source content.
Expand or collapse full text
INQUESTO SCORE: A RELIABILITY PROTOCOL FOR VOICE AGENTS Massa Baali Bhiksha Raj Abstract Voice agents are increasingly deployed in workflows where failed interactions can affect transactions, access, and other consequential outcomes, creating a need for reproducible and interpretable evaluation. We introduce Inquesto Score (IS), a protocol for measuring voice-agent reliability as the percentage of calls in a fixed, versioned evaluation population that achieve the caller’s goal without a functional failure or worse. Rather than combining heterogeneous metrics, IS defines explicit failure events and severity levels and evaluates the deployed voice pipeline. Timing failures, including talk-over and delayed responses, are measured directly from audio, while semantic and state-dependent failures are evaluated using scenario predicates, tool traces, and a pinned open-model judge. Diagnostic views of behavior, acoustic robustness, identity handling, and speaker groups accompany the score without being combined into it. Inquesto Score v0.1 evaluates 30 scenarios, three acoustic conditions, four speaker groups, and 306 calls per agent across 13 configurations of a reference voice-agent system. Our evaluation shows that reliable measurement requires evidence beyond transcripts, explicit treatment of deployment conditions, and validation of the evaluators used to determine outcomes. We release the protocol, reference implementation, and evaluation records. Index Terms: voice agents, evaluation protocol, spoken dialogue systems, reliability, fairness †address: Carnegie Mellon University 1 Introduction Voice agents can fail in ways that text agents cannot: they can talk over a caller, respond several seconds too late [1], lose a correction that arrived while they were still speaking, or take an action that should have been refused. Some failures are semantic, such as providing incorrect information; others depend on the audio and timing of the interaction itself, where human conversation operates on gaps of only a few hundred milliseconds [2]. A narrowband telephone connection, for example, can obscure acoustic cues needed by a verifier [3, 4], causing an agent to act for a caller whom it should have refused. Such failures can determine whether an interaction actually succeeds, yet many disappear when a call is reduced to its transcript [1]. As voice agents move into workflows involving transactions, access, and other consequential actions [5], evaluation therefore has to answer a more basic question than whether a model produced a good response: did the interaction actually succeed? This is increasingly a measurement problem rather than only a benchmarking one. The NIST AI Risk Management Framework asks for rigorous testing with stated uncertainty and documented results [6]; ISO/IEC 42001 builds AI governance on documented, traceable risk processes [7]; and the EU AI Act requires providers of high-risk systems to test against predefined metrics and to document and disclose results [8]. None of these prescribes a metric; all of them make the meaning of a reported number matter. Recent benchmarks evaluate voice agents end to end with simulated callers. EVA-Bench runs bot-to-bot audio conversations and reports composite accuracy and experience metrics under controlled acoustic variation [9]; VAmoS Bench drives complete stacks through simulated phone calls against a seeded backend and grades transcripts, tool calls and backend state with per-scenario assertions [10]; MTVA-Bench isolates the language model inside a cascade by exposing it to simulated recognition errors and utterance splits [11]; VoiceAgentEval and τ-bench extend the tool-agent tradition [12, 5]. What these leave open is a measurement question: what should a single reported score mean? Combining heterogeneous dimensions requires assumptions about how outcomes trade off [13], and those assumptions can hide a severe functional failure behind strong performance elsewhere [14]; and an evaluation can be internally consistent while measuring the wrong thing [15]. We introduce the Inquesto Score, a protocol for measuring voice-agent reliability at the level of the complete interaction. The protocol is scenario-agnostic: it fixes how an evaluation population is defined, how success and failure are decided, and how evidence from audio, transcripts, tool traces and semantic judgments becomes one pass rate; a use case supplies the scenarios and predicates. The score is the percentage of protocol-defined calls that achieve their goal without a functional failure or worse (severity S3 and above in v0.1), so that under a fixed protocol version a score of 73 means exactly that 73 % of the evaluated calls met the clean-success criterion. The unit of evaluation is the deployed pipeline, from speech recognition through reasoning, tool use and synthesis to interaction timing, and each failure type is checked against the evidence that can show it, timing against the audio timeline rather than a transcript. Our contributions are: (i) a versioned protocol whose headline score is a rate over a fixed population with a direct interpretation and a confidence interval; (i) a cross-modal failure taxonomy with observable definitions and fixed severities, in which timing failures are measured from audio and semantic ones by a pinned judge whose influence on the score is measured; (i) diagnostic population slices for behavior, acoustic robustness, identity handling and speaker groups that explain the rate without competing with it; (iv) controlled experiments on 13 configurations of one reference system probing audio-grounded failures, deployment-level changes, alternative severity thresholds and the dependence on the judge. 2 The Inquesto Protocol v0.1 Inquesto is a scoring procedure that a use case instantiates: the use case supplies the scenarios and their goal and state predicates, while the population construction, scoring rule, failure semantics, evidence requirements and reporting stay fixed. We use billing support throughout because it combines task completion, conversational, timing and authorization failures in one controlled setting. 2.1 Evaluation Population For the experiments in this paper, we instantiate Inquesto v0.1 on a billing-support use case. The population contains 30 scenarios in one billing-support domain: 24 goal-directed scenarios in four caller styles (neutral, fast, hesitant, and correcting), plus 3 legitimate and 3 impostor identity scenarios. We apply three acoustic conditions to the caller’s line before the agent’s speech recognizer (clean 16 kHz; telephone, 300–3400 Hz with G.711 μ-law quantization [16]; and babble at 10 dB SNR), and four speaker groups realized with open TTS voices (US/UK × female/male). Identity scenarios use fixed voices, with the enrolled account holder representing legitimate callers and other voices representing impostors. This yields 24×3×4+6×3=30624× 3× 4+6× 3=306 calls per agent, each with weight one. Replacing the scenario set and its predicates defines a new protocol instance (a new version, once released), with the semantics of the score unchanged; absolute scores are comparable only under the same version and population. 2.2 Clean Success and Severity For each call c, the protocol obtains a goal predicate g(c)∈0,1g(c)∈\0,1\ and a set of failure events E(c)E(c), each with a severity σ(e)∈1,…,5σ(e)∈\1,…,5\ fixed by its event type. The severity scale separates failures by their operational consequence: S1 denotes cosmetic issues, S2 degraded experience, S3 functional failure, S4 material risk, and S5 critical failure. In v0.1, S3 is the lowest severity considered sufficient to make a call fail. The goal predicate is state first, judge second: the action required by the scenario must appear in the tool trace (a refund the agent promised but never issued is not a refund), and a judge confirms the semantic outcome; for an impostor, success requires that nothing was changed and nothing was disclosed. A call is a clean success when pass(c)=g(c)∧maxe∈E(c)σ(e)<3,pass(c)=g(c)\, \, _e∈ E(c)σ(e)<3, (1) and for a protocol-defined population C the Inquesto Score is is(C)=100⋅1|C|∑c∈Cpass(c). is(C)=100· 1|C| _c∈ Cpass(c). (2) S1 and S2 events are counted and reported but do not fail a call in v0.1, whereas S3, S4, and S5 events do. This threshold is an explicit protocol parameter rather than a universal definition of failure; Section 4 tests the sensitivity of the score to alternative thresholds. 2.3 Failure Taxonomy Table 1 defines every v0.1 failure event and its observable criterion. Timing events are computed from the audio timeline and voice-activity timestamps rather than inferred from transcripts. In particular, talk-over and delayed responses are measured from the actual speech signals. Trace events are obtained directly from tool calls. Judged events are evaluated by one pinned open model, Gemma-2-9B-Instruct [17], at temperature 0 using a fixed rubric containing the account record available to the agent. The judge differs in model family from both the caller simulator and all agents under test; Section 4 measures how much the score depends on it. Table 1: Failure events in protocol v0.1. S3 and above fail the call. Event Observable definition Source Sev. verbose mean agent turn >> 60 words transcript S1 slow median median reply latency >> 1.5 s audio S2 talk-over agent audio starts ≥ 0.5 s before caller audio ends audio S2 slow turn a reply >> 3 s after the caller stops, or no reply audio S3 repeated talk-over talk-over on ≥ 1/3 of agent turns (≥ 2) audio S3 context loss corrected detail later used in its original form judge S3 false reject verified account holder refused service judge S3 wrong information stated account fact or policy contradicts the record judge S4 promised, not done judge reads success, required tool never called trace × judge S4 unverified action account change before lookup and voice verification trace S5 impostor served impostor obtains a change or account details trace, judge S5 2.4 Diagnostic Views There is one score. Four diagnostic views show where it comes from; each is the same pass rate on a defined slice, reported with its own interval and size, and never combined into the score. Behavior: clean audio, reference speaker group, non-identity scenarios (n=24n=24). Robustness: degraded conditions (n=204n=204), with the gap to clean audio. Identity handling: the 18 identity calls, where verified callers must be served and impostors must obtain neither a change nor account details. Fairness: the pass rate of the worst speaker group (n = 75–81 per group), with every group’s difference to the reference group. 2.5 Identity Evaluation and Reporting The authentication tool uses an open speaker-embedding model [18] enrolled on the account holder’s voice. It scores the caller’s first utterance as delivered through the acoustic condition. Its threshold (τ=0.68τ=0.68) is calibrated on the v0.1 population to accept the enrolled voice under all tested conditions and reject all impostors. Consequently, the identity view evaluates how the agent uses an authentication signal rather than the intrinsic accuracy of the speaker verifier. Every score is reported with a 95 % Wilson interval [19] and n, in the form Inquesto v0.1 = 44.8% (95% CI 39.3–50.4; n = 306), followed by the views. A run emits a numeric record (score, interval, n, views, failures by severity and type, protocol version, agent fingerprint) without audio or transcripts; a protocol version never changes after release. The reference implementation, protocol files and all records are at github.com/massabaali7/inquesto-score. 3 Experimental setup 3.1 Pipeline Calls run through an open audio pipeline [20]: the caller’s turns are synthesized in the scenario’s style with an open TTS model [21], passed through the acoustic condition, segmented by a voice-activity detector [22] using the agent’s endpointing window, transcribed by Whisper [23] and answered by the agent’s language model; the reply is synthesized and its start time measured, so latency and talk-over are properties of real audio. The caller is simulated by Qwen2.5-7B-Instruct [24]. 3.2 Agents as instruments The experiments test the scoring protocol, not the agents. One fixed billing-support program (prompt, six tools, constraints) is run in 13 configurations that differ only in what practitioners change: the language model behind it (open Qwen2.5 3B–32B and Llama-3.1-8B served locally, and hosted models behind an OpenAI-compatible gateway) and the endpointing window (400, 700, 1100 ms). Speech recognition, synthesis, caller simulation, population, taxonomy and judge are identical across rows, so a difference between rows is a difference in the agent’s behavior and timing, and each configuration acts as a controlled probe: if endpointing changes interruption timing, or two deployments of the same weights behave differently, the score should show it. 4 Results Table 2: Inquesto v0.1 for every configuration of the reference agent: score, 95 % interval, diagnostic views B/R/I/F, pass rate per speaker group (US/UK × female/male; F is their minimum), events by severity (S3/S4/S5), the score with audio-derived events removed (tx-only) and the number of failed calls whose only S3+ events were audio-derived. h = language model behind a hosted gateway, everything else local. Agent (LLM / endpointing) IS 95 % CI B R I F US-F US-M UK-F UK-M S3/4/5 tx-only audio-only Qwen2.5-7B / 1100 ms 45 [39, 50] 42 46 39 41 42 41 53 43 80/75/11 58 39 Qwen2.5-7B / 700 ms 39 [34, 44] 46 36 44 32 41 32 37 45 104/78/16 56 51 Llama-3.1-8B / 700 ms 36 [31, 42] 46 36 39 27 42 33 27 41 122/18/54 58 68 Qwen2.5-32B / 700 ms 33 [28, 39] 38 30 33 27 27 29 35 43 94/12/80 56 70 Qwen2.5-14B / 700 ms 33 [28, 38] 38 32 39 29 35 29 31 36 103/22/89 54 64 Claude-Haiku-4.5h / 700 ms 26 [21, 31] 17 27 22 24 24 25 28 27 263/7/0 66 122 Gemini-2.5-Flashh / 700 ms 25 [20, 30] 25 26 22 21 22 21 25 31 187/14/8 53 87 Qwen2.5-7B / 400 ms 24 [20, 29] 25 24 22 16 28 16 27 25 183/56/13 61 114 Llama-3.1-8Bh / 700 ms 20 [16, 25] 12 22 22 17 21 17 20 23 137/19/69 37 52 GPT-4.1-minih / 700 ms 19 [15, 24] 21 18 44 13 21 13 23 19 257/12/55 50 95 Claude-Sonnet-4.6h / 700 ms 19 [15, 24] 25 18 28 12 21 12 25 17 265/3/0 65 140 Qwen2.5-3B / 700 ms 14 [11, 18] 17 13 0 10 10 21 13 12 94/47/60 18 12 GPT-5.4-minih / 700 ms 6 [4, 10] 4 7 6 5 5 9 7 5 236/7/1 24 53 4.1 Audio Reveals Failures Transcript Evaluation Misses Across the 13 configurations (Table 2), 33 % of failed calls failed only on audio-derived events; a transcript-only evaluation of the same conversations would have passed them and reported a score 24 points higher on average. The agent that reads best on a transcript, Claude-Haiku-4.5h / 700 ms (66 with audio events removed, no policy violation, no promise it did not keep), scores 26 on the phone, because most of its calls contain a reply longer than 3 s. 4.2 Deployment Parameters Change Reliability For a fixed Qwen2.5-7B language model, changing only the endpointing window moves the score from 24 at 400 ms to 39 at 700 ms and 45 at 1100 ms. The shorter window produces more interruptions and responses to incomplete requests. The same Llama-3.1-8B weights also produce different results when served locally versus through a hosted gateway: scores are 36 and 20, respectively, with median reply latencies of 1.3 s and 1.6 s; the difference is not only timing, since the hosted serving stack is terser and skips verification more often, so even the transcript-only readings differ (58 versus 37). Inquesto scores the deployed agent, serving and timing included, not the model in isolation. 4.3 Diagnostic Views Expose Hidden Differences The worst speaker group falls as much as 15.3 points below the reference voice (per-group columns of Table 2). The fairness view therefore exposes subgroup differences that the aggregate score can conceal. Across agents impostors obtained a change or account details 40 times (S5) and verified callers were refused 2 times (S3); the identity-handling view ranges from 0 to 44, roughly independent of the score, and its n=18n=18 makes its interval wide, as the table shows. Combining the views (mean, geometric mean, minimum) would shift the agents by -0.6, -1.6 and -6.9 points on average; v0.1 adopts none. Table 3: Score under each clean-success threshold: a call fails on any event of the given severity or worse. v0.1 uses S3 (functional failure). Agent S2+ S3+ (v0.1) S4+ S5 Qwen2.5-7B / 1100 ms 44 45 58 58 Qwen2.5-7B / 700 ms 34 39 56 56 Llama-3.1-8B / 700 ms 18 36 58 60 Qwen2.5-32B / 700 ms 27 33 56 57 Qwen2.5-14B / 700 ms 22 33 54 55 Claude-Haiku-4.5h / 700 ms 3 26 66 67 Gemini-2.5-Flashh / 700 ms 0 25 53 54 Qwen2.5-7B / 400 ms 8 24 61 61 Llama-3.1-8Bh / 700 ms 8 20 38 38 GPT-4.1-minih / 700 ms 0 19 51 52 Claude-Sonnet-4.6h / 700 ms 0 19 65 65 Qwen2.5-3B / 700 ms 10 14 18 19 GPT-5.4-minih / 700 ms 0 6 24 24 4.4 Threshold Sensitivity and the Judge The clean-success threshold is a stated parameter, and Table 3 shows what moving it does. Requiring that a call be free even of degraded-experience events (S2) lowers the score by 12.8 points on average; counting only material-risk (S4) or critical (S5) events raises it by 24.5 and 25.0 points. (S4 and S5 nearly coincide: almost every S4 event sits in a call whose goal also failed.) The ordering of configurations is stable across thresholds: the parameter sets how strict the contract is, not who passes it. The semantic judge is the one component of the protocol that is itself a model, so we measured how much rests on it. Re-reading every transcript with the caller’s own Qwen2.5-7B instead of the pinned Gemma-2-9B agrees with it on 82 % of goal decisions and 91 % of wrong-information decisions but only 68 % of context-loss decisions, which is why the protocol pins one judge and one rubric per version rather than averaging judges. The headline results do not hinge on it: a judge-free variant of the score, using only state predicates and audio and trace events, ranks the 13 configurations almost identically (Spearman ρ=0.90ρ=0.90) while sitting 12 points higher, because the judge adds failures (context loss, wrong information, unkept promises) that state and timing alone cannot see. Timing events need no such check: they are measured, not judged. 5 Limitations and conclusion Inquesto Score is a protocol that measures the reliability of a deployed voice agent as an interpretable rate over a fixed population, with success and failure defined explicitly, interaction failures measured from audio, and diagnostics kept separate from the score. v0.1 is scoped to one domain, synthesized callers, one judge and one verifier, and its scores are comparable only under the same protocol version; its purpose is a reproducible contract, not a real-world success probability. Three limitations shape v0.2: the hosted rows describe one deployment path, an institutional gateway whose latency is part of what was measured; the semantic judge is a model whose context-loss decisions vary with the judge chosen, so a human-calibrated rubric comes first; and the population is one domain with synthesized callers, to be widened with a second domain, human callers and a cloning attack on the verifier. Protocol, implementation and records are released. References [1] Guan-Ting Lin et al., “Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities,” arXiv preprint arXiv:2503.04721, 2025. [2] Tanya Stivers et al., “Universals and cultural variation in turn-taking in conversation,” Proceedings of the National Academy of Sciences, vol. 106, no. 26, p. 10587–10592, 2009. [3] Xuechen Liu et al., “ASVspoof 2021: Towards spoofed and deepfake speech detection in the wild,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, p. 2507–2522, 2023. [4] Massa Baali et al., “Sveritas: Benchmark for robust speaker verification under diverse conditions.,” in EMNLP (Findings), 2025, p. 9714–9731. [5] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan, “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024. [6] National Institute of Standards and Technology, “Artificial intelligence risk management framework (AI RMF 1.0),” Tech. Rep. NIST AI 100-1, NIST, 2023. [7] ISO/IEC, “ISO/IEC 42001:2023 information technology — artificial intelligence — management system,” 2023. [8] European Parliament and Council, “Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI act),” 2024. [9] ServiceNow AI, “EVA-Bench: A new end-to-end framework for evaluating voice agents,” arXiv preprint arXiv:2605.13841, 2026. [10] Joshua Meyer et al., “VAmoS bench: Voice agent simulation bench,” arXiv preprint arXiv:2607.27453, 2026. [11] Pritish Mishra, Ishaan Kumar, Akshat Mandoli, and Sudarshan Kamath, “MTVA-Bench: Evaluating the language model inside cascaded voice agents,” arXiv preprint arXiv:2609.20152, 2026. [12] Xbench, “VoiceAgentEval: A dual-dimensional benchmark for expert-level intelligent voice-agent evaluation,” arXiv preprint arXiv:2510.21244, 2025. [13] Kawin Ethayarajh and Dan Jurafsky, “Utility is in the eye of the user: A critique of NLP leaderboards,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, p. 4846–4853. [14] Ryan Burnell et al., “Rethink reporting of evaluation results in AI,” Science, vol. 380, no. 6641, p. 136–138, 2023. [15] Hanna Wallach et al., “Position: Evaluating generative AI systems is a social science measurement challenge,” in Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. [16] ITU-T, “Recommendation G.711: Pulse code modulation (PCM) of voice frequencies,” 1988. [17] Gemma Team, “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024. [18] Hongji Wang et al., “Wespeaker: A research and production oriented speaker embedding learning toolkit,” in Proc. ICASSP, 2023. [19] Edwin B. Wilson, “Probable inference, the law of succession, and statistical inference,” Journal of the American Statistical Association, vol. 22, no. 158, p. 209–212, 1927. [20] Daily, “Pipecat: Open source framework for voice and multimodal conversational AI,” https://github.com/pipecat-ai/pipecat, 2024. [21] hexgrad, “Kokoro-82m: an open-weight text-to-speech model,” https://huggingface.co/hexgrad/Kokoro-82M, 2025. [22] Silero Team, “Silero VAD: pre-trained enterprise-grade voice activity detector,” https://github.com/snakers4/silero-vad, 2021. [23] Alec Radford et al., “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023. [24] Qwen Team, “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024.