Paper deep dive

FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India

Aman Dalmia, Sanskriti Midha, Jigar Doshi

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 75

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/8/2026, 4:43:17 AM

Summary

The paper introduces FormBharo, a voice-based conversational agent designed to assist low-literacy, Hindi-speaking mothers in rural India in filling out enrollment forms for maternal and child health benefits. The system combines Large Language Models (LLMs) for natural language understanding and generation with deterministic, rule-based validation and flow control to handle latency, cost, and accuracy constraints. The authors evaluate the system using FormVoiceAgentBench, a new benchmark comprising 3,760 multi-turn conversation tests across 960 simulated calls with real acoustic variations. Key findings indicate that component-level performance does not predict end-to-end form completion, as errors propagate and cancel across the pipeline. Rule-based controls significantly improve form completion rates, allowing smaller, cheaper models to match or surpass frontier models. The optimal deployment configuration is selected using Pareto-based weighted-sum scalarization to balance accuracy, cost, and latency.

Entities (10)

FormBharo · system · 98%FormVoiceAgentBench · dataset · 97%ARMMAN · organization · 95%Hindi · language · 95%GPT-5.5 · model · 92%Chirp 3 · model · 90%Maharashtra · location · 90%Nova-3 · model · 90%Scribe v2 · model · 90%LLM-WER · metric · 85%

Relation Signals (9)

FormVoiceAgentBench → contains → Hindi audio

confidence 95% · ...a benchmark pairing human-recorded Hindi audio with 3,760 multi-turn conversation tests...

FormBharo → pilotedwith → ARMMAN

confidence 95% · It is being piloted with ARMMAN, an NGO running large-scale maternal and child mobile-health programs in India...

FormBharo → targetspopulation → Low-literacy users

confidence 95% · ...frame the call-based conversational form-filling task for low-literacy users in LMICs...

FormBharo → usescomponent → LLMs

confidence 95% · ...pairing Large Language Models (LLMs) with deterministic, rule-based validation and flow control.

FormBharo → usescomponent → Rule-based validation

confidence 95% · ...pairing Large Language Models (LLMs) with deterministic, rule-based validation and flow control.

FormVoiceAgentBench → createdfor → FormBharo

confidence 90% · We openly release FormVoiceAgentBench... to evaluate our agent's components...

FormBharo → developedby → Aman Dalmia

confidence 90% · FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India Aman Dalmia...

Rule-based controls → improves → Form completion

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:In India, almost every social benefit starts with a form, yet the people who need these benefits most are often unable to read or write. Reaching them requires a spoken conversation. Today that work falls to frontline health workers who enroll beneficiaries one at a time, a poor use of stretched capacity. We built FormBharo ("fill the form" in Hindi), a voice agent that fills a structured form over a phone call under tight latency and cost budgets by pairing Large Language Models (LLMs) with deterministic, rule-based validation and flow control. It is being piloted with ARMMAN, an NGO running large-scale maternal and child mobile-health programs in India, to enroll low-income, Hindi-speaking mothers in antenatal and postnatal care. To our knowledge, it is the first voice agent piloted to fill an enrollment form for this population. We openly release FormVoiceAgentBench, a benchmark pairing human-recorded Hindi audio with 3,760 multi-turn conversation tests across 960 simulated calls, to evaluate our agent's components (transcription, extraction, reply generation) and end-to-end form completion under real acoustic variations. Form completion drops by up to ~41 points when LLMs receive error-prone real-speech transcripts instead of reference ones. The rule-based controls recover many turn-level extraction errors, helping smaller, cheaper models match or surpass frontier models on form completion. Component performance does not predict end-to-end performance: GPT-5.5 leads turn-level extraction accuracy on reference transcripts (99.8%) but ranks lower on form completion. Since errors both propagate and cancel across the pipeline, the optimal model choice of models emerges only through end-to-end evaluation. Finally, no single model is best across accuracy, cost, and latency at once, so we use a Pareto-based weighted-sum scalarization to select a deployable configuration balancing the three.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

73,814 characters extracted from source content.

Expand or collapse full text

FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India Aman Dalmia, Sanskriti Midha, Jigar Doshi Abstract In India, almost every social benefit starts with a form, yet the people who need these benefits most are often unable to read or write. Reaching them requires a spoken conversation. Today that work falls to frontline health workers who enroll beneficiaries one at a time, a poor use of their stretched capacity. We built FormBharo (“fill the form” in Hindi), a voice agent that fills a structured form over a phone call under tight latency and cost budgets by pairing Large Language Models (LLMs) with deterministic, rule-based validation and flow control. It is being piloted with ARMMAN, an NGO running large-scale maternal and child mobile-health programs in India, to enroll low-income, Hindi-speaking mothers in antenatal and postnatal care. To our knowledge, it is the first conversational voice agent piloted to fill an enrollment form for this population. We openly release FormVoiceAgentBench, a new benchmark pairing human-recorded Hindi audio with 3,760 multi-turn conversation tests across 960 simulated calls, to evaluate our agent’s components (transcription, extraction, reply generation) and end-to-end form completion under real acoustic variations. Form completion drops by up to ∼ 41 percentage points when LLMs receive error-prone real-speech transcripts instead of reference transcripts. The rule-based controls recover many turn-level extraction errors, helping smaller, cheaper models match or surpass frontier models on form completion. Component-level performance does not predict end-to-end performance: GPT-5.5 leads turn-level extraction accuracy on reference transcripts (99.8%) but ranks lower on form completion. Since errors both propagate and cancel across the pipeline, the optimal choice of models emerges only through end-to-end evaluation. Finally, no single model is the best across accuracy, cost, and latency at once, so we use a Pareto-based weighted-sum scalarization for selecting a deployable configuration that balances the three. Figure 1: FormBharo’s architecture. EXTRACT extracts the form values from the transcribed text. The rule-based layer validates the extracted values, updates the form state, skips inactive branches, and selects the next question to ask. REPLY phrases the next question which is spoken back to the user through a TTS model. The “Anganwadi name” is the public clinic’s name. 1 Introduction Access to nearly every social benefit in India runs through a form a citizen must complete to enroll (Indus Action 2026). Yet, the people with the greatest need for these programs are often disproportionately low-income and less literate, leaving them least equipped to access the benefits: more than half the women in the lowest wealth quintile cannot read at all (International Institute for Population Sciences and ICF(2021) (IIPS)). Enrollment therefore falls to frontline health workers who sign up beneficiaries one conversation at a time (Drèze and Khera 2017). Documentation already consumes much of their day (Khandre, Jakasania, and Raut 2023) and dozens of overlapping reporting systems add to this burden, requiring the same data to be re-entered across apps and paper registers (Nongrum et al. 2025). This caps enrollment at worker capacity rather than need. Additionally, these records are the administrator’s source of truth for allocating resources, so errors and delays in enrollment slow the system’s response. Voice-based services have already demonstrated population-scale reach in low- and middle-income countries (LMICs). For example, the outbound prerecorded-call service Kilkari reached more than 10 million subscribers (LeFevre et al. 2022) and delivered ∼ 1.2 million calls per day (Bashingwa et al. 2021). Early interactive voice response (IVR) systems commonly relied on touch-tone input, which users preferred over low-quality speech recognition (Patel et al. 2010). However, Sherwani et al. (2009) found that a carefully designed speech interface achieved significantly higher task-completion rates than an equivalent touch-tone interface among both low-literate and literate users. LLM-powered voice agents are now making real-time spoken interaction increasingly practical (Mukherjee 2026), creating new opportunities for service delivery. A mother can call a number and enroll by speaking naturally with an agent in her own words and language, with nothing to read, type, or install. Doing this reliably is hard. Voice agents work well with a cooperative speaker in a quiet room, but performance degrades with background noise, telephone-channel distortion, underrepresented regional accents, unusual speaking rates, mispronunciations, and disfluencies (Chen et al. 2026; Bhanushali et al. 2022). Hindi-English code-mixing introduces another challenge (Diwan et al. 2021), as do structured values such as phone numbers and dates, which can be spoken in many ways but must resolve to a single value (Mohammadi et al. 2026). These problems compound in form-filling calls: callers hesitate, they correct themselves mid-answer, and the form branches on earlier answers, so an incorrectly captured field can send the agent down the wrong path. We present FormBharo, a hybrid voice agent that fills a structured form over a phone call under tight latency and cost constraints (Figure 1). A speech-to-text (STT) model transcribes the caller’s speech. An LLM, EXTRACT, extracts the relevant form values. A rule-based layer then validates them, updates the form state and picks the next question to ask. A second LLM, REPLY, phrases the question naturally, which a text-to-speech (TTS) model speaks back to the caller. If all the fields have been answered, the LLM decides to end the call instead. FormBharo is being piloted in rural Maharashtra, India, enrolling low-income, Hindi-speaking mothers in an antenatal and postnatal care program. The pilot runs in collaboration with ARMMAN (ARMMAN 2008), a nonprofit in India that operates large-scale mobile-health programs for maternal and child health among underserved communities. To evaluate our agent, we release FormVoiceAgentBench, a benchmark pairing human-recorded Hindi audio with 3,760 multi-turn conversation tests across 960 simulated calls, built on the enrollment form from our pilot without exposing any real caller’s data111Code and data will be made available here: https://github.com/dalmia/AAAI-FormBharo-final/tree/main/code. Unit tests are used to evaluate the quality of transcription, data extraction, and reply generation individually. Integration tests chain them to measure end-to-end form completion with real-speech input. Our experiments show that component-level performance does not predict form completion: LLMs that lead turn-level extraction accuracy with reference transcripts as inputs rank lower once the components are chained. The rule-based layer recovers many extraction errors, helping smaller, cheaper models match or surpass frontier models on form completion. Since errors both propagate and cancel across the pipeline, the best configuration emerges only through end-to-end evaluation. Finally, no single model is the best across accuracy, cost, and latency at once. To pick the best combination of models, we discard those that fail to satisfy our deployment constraints, keep the Pareto-optimal ones, and rank the rest by weighted-sum scalarization (Marler and Arora 2010), with the weights reflecting our deployment’s priorities. Our contributions are summarized below: • We frame the call-based conversational form-filling task for low-literacy users in LMICs and characterize the challenges that make it hard. • We present FormBharo, to our knowledge the first conversational voice agent piloted to fill an enrollment form for this population. • We share our evaluation design methodology and openly release FormVoiceAgentBench, a new benchmark in Hindi that implements it. • We compare different model choices and share our findings from component-level and end-to-end evaluations across accuracy, latency, and cost. 2 System Architecture FormBharo is a voice agent that fills a structured form over a phone call by asking one question at a time. The design principle is to use LLMs only where they add value: interpreting user inputs, turning unstructured answers into structured fields, phrasing the next question naturally to be spoken back to the user, and deciding when the form is complete. Everything else stays rule-based: validation, retries, branching, and choosing the next question. Figure 1 shows the call flow. Once the Voice Activity Detection (VAD) model detects that the caller has stopped speaking, an STT model transcribes the audio. The EXTRACT LLM reads the transcript together with the conversation history and extracts every field answered in that turn, so the agent does not re-ask questions already answered. The form has fields of several types: free text, number, date, boolean, and categorical. For categorical fields, EXTRACT returns the index of the chosen option rather than the option text. In parallel, EXTRACT generates a short acknowledgement that the TTS model speaks back to reduce perceived latency. The rule-based layer validates each extracted value against field-specific built-in guards, such as a specified minimum length, 10 digits for a phone number, or a date restricted to the past. When a value is missing or fails validation, it checks whether the maximum number of retries for that field is reached. If retries remain, the validation error is passed to REPLY LLM along with the conversation history, which asks the question again. Once retries are exhausted for a required field, the rule-based layer ends the call. For an optional field, it either ends the call or skips it, depending on the field-specific setting. EXTRACT also detects when the caller declines or does not know the answer to an optional field (the skip flag). That field is then skipped without triggering a retry. When validation passes, the values are written to the form state, the single source of truth for which fields have been answered. The form has several branches conditioned on the values of earlier fields. For example, gestational age is asked only if the caller says she is pregnant. The rule-based flow-control evaluates whether any of the form’s branches have been activated or disabled, and decides the next question to ask. This decision is injected into the conversation history of REPLY LLM as a tool call. It phrases the question naturally for the TTS model to speak back to the caller, or ends the call if the form is complete. Finally, if the user starts speaking while the agent’s inference is running, the current turn is terminated immediately and the agent goes back to waiting for the user to stop speaking, so interruptions are handled gracefully. Field Example answer On fail Name Sunita Devi skip District Mayurbhanj end Clinic name City Clinic skip Pregnant? yes / no end yes: Gestational age 5 months end no: Child’s name Aarav end no: Child’s DOB 15-04-2025 end Phone linked to clinic? yes / no skip no: Number linked 9876543210 skip Phone linked to WhatsApp? yes / no skip no: WhatsApp number 9876543210 skip Aadhaar last 4 digits 4321 skip Table 1: The enrollment form for testing FormBharo. When a caller fails to give a valid answer after a few retries, the agent either skips to the next field (skip) or ends the call (end). Indented fields are conditional: whether they are asked depends on the parent field’s response. Aadhaar is India’s national ID. 3 FormVoiceAgentBench Statistic Count Simulated users 5 Form fields 12 Acoustic conditions per utterance 4 Unique audio recordings 380 Calls (unique call scripts) 240 Calls (with acoustic variations) 960 Unit tests (total) 3,760 EXTRACT 1,880 REPLY 1,880 Table 2: FormVoiceAgentBench statistics. FormVoiceAgentBench is a Hindi benchmark, grounded in the maternal and child health enrollment form from our pilot, that pairs 380 audio recordings with 3,760 multi-turn conversation tests across 960 simulated calls (Table 2) without exposing any Personally Identifiable Information (PII) from real callers. It evaluates the agent at two levels. Unit tests score each component in isolation: predicted transcripts against reference transcripts, extracted form data against the expected values, reply quality and ability to end the call given that the extraction was perfectly accurate. Integration tests chain the components to measure end-to-end form completion. 3.1 Form Structure The form has 12 fields (Table 1) which follow a fixed order including conditional branching. If the caller is pregnant, the agent asks her gestational age. Otherwise, it asks her recently born child’s name and date of birth. If her calling number is not linked to the clinic, the agent asks for the linked number; otherwise it skips ahead. Similarly, the agent asks for her WhatsApp number if it differs from her calling number. The other fields are always asked. 3.2 Simulated Users and Transcripts We define five simulated user profiles, each with fixed personal details, corresponding to the fields we intend to capture. Each form field has three representations: the value, the desired form entry and the ground truth for measuring form completion (“Sunita Devi”); the reference transcript, a naturally spoken rendering of the value, phrased as a caller would say it in a real conversation (“My name is Sunita Devi”); and the recording of that transcript. The value and the transcript are generated sequentially by an LLM, under field-specific constraints defined with ARMMAN: names and district names with phonetically hard spellings, and numbers with many spoken variations that make them hard to transcribe. District names were deliberately selected from outside the pilot state for adequate stress-testing. For optional fields, we generate two linguistic variants of the transcript: one where the caller responds with the value and the other where she says she does not know it. We use OpenAI’s GPT-5.5 (OpenAI 2026) for the generation. 3.3 Audio Data Collection Each transcript was recorded to mimic the conditions in a public clinic across three acoustic variations: background noise (ambient chatter and nearby speakers), microphone distance (close or far, chosen at random), and speaking pace (fast or slow, chosen at random). Combined with the ideal acoustic condition, this gives four recordings per transcript. Five native Hindi speakers were selected to match the target demographic: all female, aged 18–35, drawn from two states (Uttar Pradesh and Maharashtra) to cover differences in accent and colloquialisms. Annotators were trained with sample recordings and recorded each answer under specific directives. For example, “record in a noisy environment, or keep the mic at least 25 cm away, or speak slowly”. A separate set of human supervisors listened to every clip to ensure the recordings met the requirements. 3.4 From Transcripts to Calls We assemble each call by stitching together a simulated user’s transcripts with one recording per field. Since the form branches, a simulated user can follow several paths, and each path becomes its own call. For example, the same simulated user produces one call in which she is pregnant and another in which she is not. Calls never mix values from different simulated users. Each call uses a single acoustic condition throughout (e.g. noisy environment or speaking slowly), since a caller’s environment does not change mid-call. Each simulated user follows 2×3×4×2=482× 3× 4× 2=48 distinct paths through the form, one call per path: two pregnancy branches (pregnant or not), three branches for whether the calling number is linked to the clinic (linked, not linked and correct value provided, or not linked but not able to remember the linked number), four choices for whether the WhatsApp number is the same as the calling number (same, not same and answered, not same but unable to recall the WhatsApp number, cannot recall if they are the same), and two branches for the Aadhaar digits (answered or not known). Across the five simulated users this gives 5×48=2405× 48=240 calls, and recording each under the four acoustic conditions yields 240×4=960240× 4=960. 4 Evaluation Design 4.1 Unit Tests Speech-to-Text Word Error Rate (WER) is commonly used for comparing STT models. However, it is a poor metric for transcription quality on Indic languages (Sarvam AI 2026) and for agents (Pipecat AI 2026b): it counts every surface difference as an error, even when the meaning is unchanged. We therefore also report LLM-WER (Sarvam AI 2026), which discards mismatches that an LLM classifies as semantically equivalent or phonetically similar and recomputes WER over the genuine errors. EXTRACT LLM To evaluate data extraction, we prepare unit tests from every call. For each turn, we create a unit test using the latest user response paired with the preceding conversation history as input and the expected form values as the ground truth. So, one call produces many tests. Deduplicating tests with identical inputs across all 960 calls yields 1,880 unit tests. The extracted values for closed-ended fields like numbers, dates, and booleans are scored by computing the exact match with the expected values. The skip flag is scored the same way. Outputs for open-ended fields like names are harder to rate since the expected value can have many phonetic forms (“Lakshmi” versus “Laxmi”), so we use a calibrated binary LLM judge (details are in the Appendix) to evaluate them. Since EXTRACT also generates a short open-ended acknowledgement, we use the LLM judge to evaluate its quality against a rubric. An example is provided in the Appendix. An accurate extraction passes both the exact-match checks and the LLM judgments. We report the mean extraction accuracy. REPLY LLM We similarly prepare unit tests from the calls to measure response accuracy, with two key differences. Along with the latest user response and the preceding conversation history, REPLY additionally receives a tool call carrying the decision of the rule-based layer as an input too (Section 2). For the unit tests, we construct this tool call by passing the expected extraction values through the rule-based validation and flow-control step, so that measuring response accuracy can be isolated from extraction errors. Secondly, REPLY either phrases the next question or ends the call when the form is complete. The end-call decision is a tool call, evaluated by exact match. Since the generated reply can be phrased in many equally correct ways, we again rely on calibrated binary LLM judges to rate them. However, unlike what we did for EXTRACT, each reply is graded across five independent dimensions: 1) asking the right question, 2) adherence to Hindi, 3) single line response, 4) no acknowledgment (since EXTRACT handles that), and 5) not echoing the caller’s answer back. The LLM judge calibration details and an example are in the Appendix. An accurate response either passes all the LLM judgments or ends the call at the right time. We report the mean response accuracy. As with EXTRACT, deduplicating the tests across the 960 simulated calls, each call producing many tests, yields 1,880 unit tests: 920 require a reply, while the remaining 960 check whether REPLY ends the call correctly. 4.2 Integration Tests Integration tests chain the components, so the output of one feeds into the next, and the agent is judged on completing the form, not on any single turn. First, we replace the reference transcripts in the unit-test inputs with transcripts produced by the STT models, so transcription errors propagate to the LLMs. Next, we evaluate each call as a whole. We start from an empty form and go through the call’s turns in order. At each turn, we write the values extracted for that turn’s unit test into the form, as the agent would in a live call. The form after the last turn is the predicted final form. Comparing this against the expected values gives the form-completion accuracy: the fraction of form fields captured correctly at the end of the call. We compute it using both reference transcripts and the transcripts produced by each STT model as inputs. The gap between them quantifies the impact of transcription errors on form completion. Finally, we chain EXTRACT and REPLY. To build the tool call for REPLY, we pass EXTRACT’s actual output through the rule-based validation and flow control, instead of the expected extraction values used in the unit tests. Each EXTRACT model drives its own set of REPLY runs. Because each stage feeds the next, extraction accuracy is measured for every STT × EXTRACT pair, and response accuracy for every STT × EXTRACT × REPLY combination. 5 Experiments & Analysis Model WER ↓ LLM-WER ↓ Cost (USD) ↓ Scribe v2 0.8350.835 0.0620.062 0.1720.172 Chirp 3 0.9810.981 0.0550.055 0.4240.424 Saaras v3 1.0071.007 0.0740.074 0.1370.137 GPT-4o-transcribe 1.0121.012 0.1270.127 0.1590.159 Nova-3 0.8260.826 0.1040.104 0.1270.127 Table 3: Comparison of STT models. Since models have different billing units, the total cost of transcribing the entire benchmark is reported. 5.1 Setup We benchmark five STT models and 11 LLMs across accuracy, latency (p95), and cost. The temperature is set to 0 for non-reasoning models. The reasoning models use “medium” reasoning effort for EXTRACT and “low” for REPLY. Full implementation details and 95% confidence intervals can be found in the appendix. 5.2 Speech-to-Text Table 3 shows that the two metrics disagree: Nova-3 has the best WER yet the second-worst LLM-WER. We therefore rank models by LLM-WER, which was built to address the shortcomings of WER. Chirp 3 has the best LLM-WER but costs at least twice as much as any other model, while Scribe v2 performs close to Chirp 3 at a fraction of the cost. GPT-4o-transcribe is the least accurate. Eliminating these two leaves Scribe v2, Saaras v3, and Nova-3, which we carry into the integration tests. Model Reference Saaras v3 Scribe v2 Nova-3 GPT-5.5 99.7999.79 95.5395.53 98.6298.62 95.1695.16 Gemini 3.5 Flash 99.3699.36 95.6995.69 98.9498.94 96.3896.38 Claude Opus 4.8 99.1599.15 96.2296.22 98.4098.40 96.0196.01 Claude Sonnet 4.6 99.1599.15 96.0196.01 98.9498.94 95.9695.96 GLM-5.1 98.7298.72 94.5294.52 63.4063.40 95.4395.43 Gemini Pro 97.2397.23 94.2694.26 97.2997.29 94.6394.63 Gemini 2.5 Flash 96.1796.17 95.2795.27 98.0398.03 95.4395.43 Gemini 3 Flash 95.9695.96 84.7984.79 88.8388.83 86.2886.28 GPT-5.4-mini 92.3492.34 90.0590.05 96.7696.76 91.5491.54 Mistral Medium 3.5 89.3689.36 87.8287.82 92.3492.34 87.7787.77 GPT-4.1 85.1185.11 75.5375.53 78.9478.94 79.8479.84 Table 4: Per-turn extraction accuracy using reference transcripts and the transcripts from the three STT models as inputs. Model Reference Saaras v3 Scribe v2 Nova-3 Gemini 3 Flash 100.00100.00 85.2485.24 89.4889.48 83.9283.92 Gemini 3.5 Flash 100.00100.00 90.7490.74 92.5092.50 87.0987.09 GPT-5.5 99.9599.95 89.9789.97 89.3789.37 84.7384.73 Claude Sonnet 4.6 99.6699.66 91.5591.55 93.0193.01 84.5484.54 GLM-5.1 99.5699.56 90.9290.92 58.6658.66 86.0986.09 Claude Opus 4.8 99.4199.41 92.2292.22 92.0392.03 85.8385.83 Gemini 2.5 Flash 99.0699.06 91.7691.76 92.3992.39 85.5085.50 Gemini Pro 98.9098.90 92.5592.55 93.4793.47 86.5386.53 GPT-5.4-mini 98.3598.35 90.6890.68 90.1090.10 84.4084.40 GPT-4.1 95.9995.99 88.8288.82 91.5091.50 86.3386.33 Mistral Medium 3.5 92.8392.83 90.3990.39 91.8491.84 83.3783.37 Table 5: Form-completion accuracy using reference transcripts and the outputs of the three STT models as inputs. Extraction accuracy Form completion Model Median (%) Δ (p) Median (%) Δ (p) Saaras v3 94.5294.52 −3.14-3.14 90.7490.74 −7.67-7.67 Scribe v2 97.2997.29 −0.42-0.42 91.8491.84 −7.38-7.38 Nova-3 95.1695.16 −3.14-3.14 85.5085.50 −13.56-13.56 Table 6: Median extraction and form-completion accuracy with real-speech transcripts as inputs. Δ denotes the median change relative to reference transcripts. Both degrade under transcription noise. 5.3 Data Extraction Table 4 compares various LLMs on per-turn extraction accuracy, computed using reference transcripts and the transcripts of the three STT models as inputs. Frontier models saturate extraction accuracy on reference transcripts as inputs with the top models scoring almost perfectly: GPT-5.5 leads at 99.79%, with Gemini 3.5 Flash (99.36%) and the two Claude models (99.15%) just behind. Robustness to STT errors varies across LLMs: the median drop is modest, from 0.42 percentage points to 3.14 (Table 6). The four most accurate models lose at most 4.63 points regardless of the STT model. But weaker models degrade much more: GLM-5.1 collapses by 35 points with Scribe v2 transcripts. The next-worst drop is still 11.2 points. The extraction accuracy leaderboard changes when real transcripts are used: GPT-5.5 is no longer the winner. The best extraction accuracy under real-speech is 98.94% (Gemini 3.5 Flash and Claude Sonnet 4.6) with transcripts from Scribe v2. Claude Opus 4.8 leads under Saaras v3 (96.22%) and Gemini 3.5 Flash leads under Nova-3 transcripts (96.38%). 5.4 Form Completion Similar to extraction accuracy, Table 5 reports form-completion accuracy across all the LLMs being tested. The rule-based layer recovers extraction errors. On reference transcripts, Gemini 3 Flash achieves 95.96% per-turn extraction accuracy but 100% form-completion accuracy. Per-turn extraction evaluates the output of EXTRACT LLM, whereas form completion evaluates the values ultimately stored in the form after the rule-based layer processes it. All the extraction errors for this model arise from a type mismatch: a numeric field being returned as a string. The rule-based layer normalizes it before storing the value, preserving 100% form-completion accuracy. This highlights a benefit of our hybrid design which enables smaller models to perform better end-to-end even if their per-turn inference is not perfect. The best model end-to-end differs from the best model per-turn. Under reference transcripts, GPT-5.5 leads extraction accuracy (99.79%), whereas Gemini 3 Flash and Gemini 3.5 Flash tie for the highest form-completion accuracy at 100%. Form completion degrades more than per-turn extraction with error-prone real-speech transcripts: median form-completion accuracy drops by 7.67 percentage points with Saaras v3, 7.38 points with Scribe v2, and 13.56 points with Nova-3 (Table 6), compared with median per-turn extraction drops of only 0.42–3.14 points. The largest model-specific decline is ∼ 41 points for GLM-5.1 with Scribe v2. Although the rule-based layer recovers some extraction errors, uncorrected errors can accumulate across turns to produce a larger degradation end-to-end. 5.5 Model Selection We select the STT, EXTRACT, and REPLY models in the pipeline (Figure 1) sequentially because each downstream component consumes the outputs of the components before it. We first select the STT model, then identify the best LLM for EXTRACT using that STT model’s transcripts, and finally evaluate REPLY with the selected STT and EXTRACT models fixed. The deployment objective is to balance task performance, p95 latency, and cost, subject to component-specific deployment constraints. STT selection. Scribe v2 provides the strongest downstream performance. It achieves the highest median extraction accuracy (97.29%) and form-completion accuracy (91.84%), while producing the smallest median drops relative to the model performance on reference transcripts (Table 6). We therefore select Scribe v2 as the STT model. EXTRACT selection. With Scribe v2 fixed as the STT model, we compare EXTRACT models using form-completion accuracy on its transcripts, together with latency and cost. We discard models that fail our deployment constraints: p95 latency below 55 s and form-completion accuracy above 90%90\%. This excludes Gemini Pro, despite its leading form-completion accuracy, and Gemini 2.5 Flash, leaving behind six candidates. Among those, Claude Sonnet 4.6 achieves the highest form-completion accuracy, Mistral Medium 3.5 has the lowest p95 latency (1.75 s), and GPT-5.4-mini is the cheapest ($0.0011). No model leads all three axes, and all six candidates lie on the Pareto frontier (Figure 2), so selecting a deployable model requires balancing these competing objectives. First, we min–max normalize the three axes within the frontier. Since lower latency and cost are better, we reverse their scales so that higher values are always preferred: a~i=ai−aminamax−amin,ℓ~i=ℓmax−ℓiℓmax−ℓmin,c~i=cmax−cicmax−cmin. gathered a_i= a_i-a_ a_ -a_ , _i= _ - _i _ - _ ,\\ c_i= c_ -c_ic_ -c_ . gathered Next, we rank the models using weighted-sum scalarization (Marler and Arora 2010): Ui=waa~i+wℓℓ~i+wcc~i,i⋆=arg⁡maxi⁡Ui,U_i=w_a a_i+w_ _i+w_c c_i, i = _iU_i, where wa+wℓ+wc=1w_a+w_ +w_c=1. These weights can be tuned to reflect the deployment priorities. As a baseline, under equal weights, Mistral Medium 3.5 ranks highest (U=0.81U=0.81), followed by GPT-4.1 (U=0.78U=0.78). Mistral combines the lowest latency with 91.84% form-completion accuracy and a cost of $0.0080 per turn. Claude Sonnet 4.6, the most accurate candidate, scores lower (U=0.51U=0.51) because it is slower and more expensive. The hard latency constraint already excludes models that are too slow for deployment, so we assign the remaining latency differences a lower weight, wℓ=0.1w_ =0.1. Because accuracy remains the primary objective, we sweep waw_a from 0.50.5 to 0.90.9, with wc=1−wa−wℓw_c=1-w_a-w_ . Only two models lead across this range: Gemini 3.5 Flash for 0.50≤wa≤0.610.50≤ w_a≤ 0.61, and Claude Sonnet 4.6 for 0.62≤wa≤0.900.62≤ w_a≤ 0.90 (full table in the Appendix). We select Gemini 3.5 Flash for EXTRACT: it trails Claude Sonnet 4.6 by only 0.51 percentage points on form-completion accuracy while responding faster and costing less than half as much per turn. Figure 2: Cost–quality–latency trade-off among EXTRACT models satisfying the deployment constraints. The logarithmic x-axis shows cost per turn, the y-axis shows form-completion accuracy using Scribe v2 transcripts, and marker area encodes p95 latency per turn (bigger is slower). All six models are Pareto-optimal across the three objectives. The model selected for deployment is highlighted. 5.6 Response Generation Model Reference Scribe v2 Claude Sonnet 4.6 100.00100.00 97.7797.77 GPT-4.1 100.00100.00 96.6096.60 GPT-5.4-mini 98.0998.09 96.7096.70 Gemini 3 Flash 97.2397.23 96.1796.17 Gemini 3.5 Flash 97.0297.02 94.7394.73 Table 7: REPLY response accuracy with Gemini 3.5 Flash as the EXTRACT LLM. REPLY has the narrowest role in the pipeline (Section 2): it either phrases the selected question naturally or ends the call. So, a smaller, faster model may suffice. With Scribe v2 and Gemini 3.5 Flash for transcription and extraction, we compare five LLMs for REPLY. The selected EXTRACT model’s actual outputs on Scribe v2 transcripts are passed through the rule-based layer to construct the decision passed to REPLY. Table 7 reports the resulting response accuracy. The Reference column is computed using the expected extraction values with reference transcripts to prepare the inputs, whereas for Scribe v2, the outputs of EXTRACT on Scribe v2 transcripts are used to prepare the tests (Section 4). Errors propagate through the pipeline. Response accuracy decreases for all five models when errors from transcription and extraction flow through to REPLY. The decline ranges from 1.06 to 3.40 percentage points, with a median of 2.23 points, capturing the combined effect of transcription and extraction errors. No single model is the best across accuracy, latency, and cost. Claude Sonnet 4.6 is the most accurate on Scribe v2 transcripts (97.77%), Gemini 3 Flash has the lowest p95 latency (2.66 s), and GPT-5.4-mini is the cheapest ($0.0008 per turn). Our deployment constraints of p95 latency below 44 s and response accuracy above 95%95\% exclude Claude Sonnet 4.6 on latency and Gemini 3.5 Flash on accuracy, leaving three candidates. Pareto filtering leaves GPT-5.4-mini and Gemini 3 Flash. Using the same weighted-sum scalarization method defined earlier with wℓ=0.1w_ =0.1, GPT-5.4-mini ranks highest throughout 0.5≤wa≤0.90.5≤ w_a≤ 0.9 (details are in the Appendix). Therefore, we select GPT-5.4-mini for REPLY. 6 Related Work Deployed AI for public services. A growing body of AI-for-social-impact research studies how AI reshapes access to services for under-served users. Jo et al. (2025) show LLM assistants can lower administrative burdens while adding new compliance and trust costs, and studies of digital welfare systems document similar transfers of burden to claimants (Watson, Parnaby, and Kharrufa 2024). Closest to our design, Kothari et al. (2026) decompose a clinical LLM-summarization task into semi-structured attributes that can be validated separately rather than trusting one end-to-end prompt. This matters for equity: Poole-Dayan, Roy, and Kabbara (2026) found that the LLM’s response quality drops for users with lower English proficiency and literacy. Our callers fit that profile, so any non-English system needs to be evaluated rigorously. Spoken understanding and Indic speech. Spoken slot filling and dialogue state tracking often use an STT-to-LLM cascade that transcribes and extracts values, where recognition errors propagate into slot and state errors (Yoon et al. 2023; Jacqmin et al. 2023; Ganesan et al. 2021; Sun et al. 2024); Si et al. (2023) show that a low WER does not guarantee task accuracy. Indic and code-mixed resources supply realistic acoustic and linguistic variation, including multilingual, code-switched, spontaneous, telephonic, geographically diverse, and low-resource speech (Diwan et al. 2021; Bhanushali et al. 2022; Bhogale et al. 2026; Javed et al. 2023, 2024; Pulikodan et al. 2026; Joshi et al. 2025). However, these resources evaluate transcriptions or other component-level speech tasks such as speaker identification. In our paper, in addition to transcriptions, we score per-field correctness on a real Indian enrollment form. Voice agents and form-filling. Recent benchmarks such as VoiceBench (Chen et al. 2026) and VoiceAgentBench (Jain et al. 2025) evaluate voice systems on outcomes beyond transcription such as spoken question answering, instruction following and tool selection, using predominantly synthetic speech. Closest to us, EVA-Bench evaluates task accuracy and interaction quality over simulated multi-turn enterprise calls (Bogavelli et al. 2026). None of them, however, target a constrained, structured task like form completion. Related application systems assess latency and conversational quality, form usability, clinician-reviewed speech to EMR generation, or sampled production records (Cuadra et al. 2024; Mustafa et al. 2026; Mukherjee et al. 2026). In contrast, FormVoiceAgentBench scores many models on form completion using noisy audio in a low-resource language. 7 Conclusion We presented FormBharo, a phone-call-based conversational form-filling voice agent being piloted in a live maternal and child health enrollment program in rural Maharashtra, India, and described FormVoiceAgentBench, a benchmark for evaluating it, along with our findings. Our results show that component-level accuracy does not predict end-to-end form completion, with errors both propagating and canceling across the pipeline. The rule-based layer recovers many model errors, helping smaller, cheaper models meet deployment constraints. This is critical in LMICs, where cost and latency constrain deployment at scale. The current benchmark is limited in several ways. It captures scripted, well-formed answers, but real callers also give wrong, partial, or self-corrected values. Each call applies only a single acoustic variation at a time, so combinations within the same call, such as a distant microphone in a noisy room, remain untested. The benchmark is based on five simulated users, with audio recorded one turn at a time by five annotators from two states rather than through full live calls, limiting its conversational, linguistic, and demographic diversity. The dataset is also limited to Hindi, though our intended users are multilingual. Finally, we did not evaluate TTS output quality. Ethical Statement No real user data was used to build the dataset. The spoken scripts were recorded by paid annotators who are native Hindi speakers. The broader intended impact of FormBharo is to widen access to care for an under-served population, but this also increases the risks. Since incorrect data capture could instead delay or deny access, deployment at scale requires more rigorous testing with adequate guardrails and fallback mechanisms in place to confirm or correct captured information when required. Acknowledgments We thank Amrita Mahale, Parina Anand and Hetvi Lodaya at ARMMAN for designing the enrollment form, testing the agent through successive iterations, and sharing the insights from the field that guided its design. We thank the Vaani team at ARTPARK for their help with data collection and annotation. Finally, we are grateful to the frontline health workers and mothers who tested FormBharo and shared their feedback. References ARMMAN (2008) ARMMAN. 2008. ARMMAN — Advancing Reduction in Mortality and Morbidity of Mothers, Children and Neonates. https://armman.org. Bashingwa et al. (2021) Bashingwa, J. J. H.; Mohan, D.; Chamberlain, S.; Arora, S.; Mendiratta, J.; Rahul, S.; Chauhan, V.; Scott, K.; Shah, N.; Ummer, O.; Ved, R.; Mulder, N.; and LeFevre, A. E. 2021. Assessing exposure to Kilkari: a big data analysis of a large maternal mobile messaging service across 13 states in India. BMJ Global Health, 6(Suppl 4): e005213. Bhanushali et al. (2022) Bhanushali, A.; Bridgman, G.; G, D.; Ghosh, P. K.; Kumar, P.; Kumar, S.; Kolladath, A. R.; Ravi, N.; Seth, A.; Seth, A.; Singh, A.; Sukhadia, V. N.; Umesh, S.; Udupa, S.; and Durga Prasad, L. V. S. V. 2022. Gram Vaani ASR Challenge on Spontaneous Telephone Speech Recordings in Regional Variations of Hindi. In Proceedings of Interspeech 2022, 3548–3552. ISCA. Bhogale et al. (2026) Bhogale, K.; Dhir, M.; Walecha, A.; Kaur, M.; Chhabra, V.; Pareek, A.; Sidh, H.; Manik, M.; Jain, S.; Singh, B.; Singh, U.; Javed, T.; Banga, S.; and Khapra, M. M. 2026. Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India. arXiv:2604.19151. Bogavelli et al. (2026) Bogavelli, T.; Gauthier Melançon, G.; Stankiewicz, K.; Bamgbose, O.; Riols, F.; Nguyen, H. H.; Mehndiratta, R.; Brin, L. D.; Marinier, J.; Subramani, H.; Madamala, A.; Nemala, S. K.; and Sunkara, S. 2026. EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents. arXiv:2605.13841. Chen et al. (2026) Chen, Y.; Yue, X.; Zhang, C.; Gao, X.; Tan, R. T.; and Li, H. 2026. VoiceBench: Benchmarking LLM-Based Voice Assistants. Transactions of the Association for Computational Linguistics, 14: 378–398. Cuadra et al. (2024) Cuadra, A.; Breuch, J.; Estrada, S.; Ihim, D.; Hung, I.; Askaryar, D.; Hassanien, M.; Fessele, K. L.; and Landay, J. A. 2024. Digital Forms for All: A Holistic Multimodal Large Language Model Agent for Health Data Entry. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(2): 1–39. Dalmia and Doshi (2025) Dalmia, A.; and Doshi, J. 2025. Calibrate: An Open-Source Evaluation Platform for AI Agents. https://calibrate.artpark.ai/. ARTPARK, Indian Institute of Science. Code: https://github.com/artpark-sahai-org/calibrate. Diwan et al. (2021) Diwan, A.; Vaideeswaran, R.; Shah, S.; Singh, A.; Srinivasa Raghavan, K. M.; Khare, S.; Unni, V.; Vyas, S.; Rajpuria, A.; Yarra, C.; Mittal, A. R.; Ghosh, P. K.; Jyothi, P.; Bali, K.; Seshadri, V.; Sitaram, S.; Bharadwaj, S.; Nanavati, J.; Nanavati, R.; and Sankaranarayanan, K. 2021. MUCS 2021: Multilingual and Code-Switching ASR Challenges for Low Resource Indian Languages. In Proceedings of Interspeech 2021, 2446–2450. ISCA. Drèze and Khera (2017) Drèze, J.; and Khera, R. 2017. Recent Social Security Initiatives in India. World Development, 98: 555–572. Exotel (2026) Exotel. 2026. Exotel: Cloud Telephony and Contact Center Platform. https://exotel.com. Accessed: 2026-08-01. Ganesan et al. (2021) Ganesan, K.; Bamdev, P.; B, J.; Venugopal, A.; and Tushar, A. 2021. N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 93–98. Association for Computational Linguistics. Google Cloud (2026) Google Cloud. 2026. Chirp 3: HD Voices — Text-to-Speech. https://cloud.google.com/text-to-speech/docs/chirp3-hd. Accessed: 2026-08-01. Indus Action (2026) Indus Action. 2026. Administrative Burden in India’s Welfare System: Examining the Learning, Compliance and Psychological Costs Faced by Vulnerable Citizens in Accessing Social Protection Programs. https://indusaction.org/case-study/. Accessed: 2026-07-23. International Institute for Population Sciences and ICF(2021) (IIPS) International Institute for Population Sciences (IIPS); and ICF. 2021. National Family Health Survey (NFHS-5), 2019–21: India Report. Technical Report FR375, International Institute for Population Sciences, Mumbai. Jacqmin et al. (2023) Jacqmin, L.; Druart, L.; Estève, Y.; Favre, B.; M Rojas, L.; and Vielzeuf, V. 2023. OLISIA: a Cascade System for Spoken Dialogue State Tracking. In Proceedings of the Eleventh Dialog System Technology Challenge, 95–104. Prague, Czech Republic: Association for Computational Linguistics. Jain et al. (2025) Jain, D.; Shukla, H.; Rajeev, G.; Kulkarni, A.; Khatri, C.; and Agarwal, S. 2025. VoiceAgentBench: Are Voice Assistants Ready for Agentic Tasks? arXiv:2510.07978. Javed et al. (2023) Javed, T.; Bhogale, K.; Raman, A.; Kumar, P.; Kunchukuttan, A.; and Khapra, M. M. 2023. IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian Languages. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11): 12942–12950. Javed et al. (2024) Javed, T.; Nawale, J.; George, E.; Joshi, S.; Bhogale, K.; Mehendale, D.; Sethi, I.; Ananthanarayanan, A.; Faquih, H.; Palit, P.; Ravishankar, S.; Sukumaran, S.; Panchagnula, T.; Murali, S.; Gandhi, K.; R, A.; M, M.; Vaijayanthi, C.; Karunganni, K.; Kumar, P.; and Khapra, M. 2024. IndicVoices: Towards Building an Inclusive Multilingual Speech Dataset for Indian Languages. In Findings of the Association for Computational Linguistics: ACL 2024, 10740–10782. Bangkok, Thailand: Association for Computational Linguistics. Jo et al. (2025) Jo, J.; Zhang, H.; Cai, J.; and Goyal, N. 2025. AI Trust Reshaping Administrative Burdens: Understanding Trust-Burden Dynamics in LLM-Assisted Benefits Systems. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT), 1172–1183. New York, NY, USA: ACM. Joshi et al. (2025) Joshi, S.; George, E. I.; Javed, T.; Bhogale, K.; Narasimhan, N.; and Khapra, M. M. 2025. Recognizing Every Voice: Towards Inclusive ASR for Rural Bhojpuri Women. In Proceedings of Interspeech 2025, 4243–4247. ISCA. Khandre, Jakasania, and Raut (2023) Khandre, R. R.; Jakasania, A.; and Raut, A. 2023. “We are working for seven days a week”: Time motion study of accredited social health activists from central India. Medical Journal Armed Forces India, 79(Suppl 1): S142–S149. Kothari et al. (2026) Kothari, A.; Vossler, P.; Digitale, J.; Forouzannia, M.; Rosenberg, E.; Lee, M.; Bryant, J.; Molina, M.; Marks, J.; Zier, L.; and Feng, J. 2026. When the Domain Expert Has No Time and the LLM Developer Has No Clinical Expertise: Real-World Lessons from LLM Co-Design in a Safety-Net Hospital. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 38754–38762. Kunchukuttan (2020) Kunchukuttan, A. 2020. The IndicNLP Library. https://github.com/anoopkunchukuttan/indic_nlp_library. Accessed: 2026-08-01. LeFevre et al. (2022) LeFevre, A. E.; Shah, N.; Scott, K.; Chamberlain, S.; Ummer, O.; Bashingwa, J. J. H.; Chakraborty, A.; Godfrey, A.; Dutt, P.; Ved, R.; and Mohan, D. 2022. The impact of a direct to beneficiary mobile communication program on reproductive and child health outcomes: a randomised controlled trial in India. BMJ Global Health, 6(Suppl 5): e008838. Marler and Arora (2010) Marler, R. T.; and Arora, J. S. 2010. The weighted sum method for multi-objective optimization: new insights. Structural and Multidisciplinary Optimization, 41(6): 853–862. Mohammadi et al. (2026) Mohammadi, S.; Paldhe, M.; Chhabra, A.; Son, Y.; and Seshagiri, V. 2026. LingVarBench: Benchmarking LLMs on Entity Recognitions and Linguistic Verbalization Patterns in Phone-Call Transcripts. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), 545–561. Rabat, Morocco: Association for Computational Linguistics. Mukherjee (2026) Mukherjee, R. 2026. What exactly is an AI voice agent? And why does it matter in enterprise communication? https://w.techradar.com/pro/what-exactly-is-an-ai-voice-agent. Accessed: 2026-07-24. Mukherjee et al. (2026) Mukherjee, S.; Sanz Ausin, M.; Aggarwal, K.; Datta, D.; Puri, S.; Jin, W.; Laud, T.; Manjunath, N.; Ding, J.; Paudel, B.; Schellenberger, J.; Huo, Z. F.; Shen, W.; Shirazian, N.; Potter, N.; Perkari, S.; Filippova, D.; Morozov, A.; Mease, A.; Muppalla, V.; Shakir, G.; Miller, A.; Ghukasyan, J.; Raglow-Defranco, M.; Taylor, M.; Mahal, H.; and Agnew, J. 2026. Perfecting Human-AI Interaction at Clinical Scale: Turning Production Signals into Safer, More Human Conversations. arXiv:2603.29893. Mustafa et al. (2026) Mustafa, M.; Shahnawaz, A.; Ammara, U.; Abrar, M.; Ahtisham, B.; Qureshi, F. U.; Shahin, M.; and Ahmed, B. 2026. Awaaz-e-Sehat: A Mobile Voice-based AI System for EMR Generation and Clinical Decision Support in Low-resource Maternal Healthcare. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 10(1): 16:1–16:37. Nongrum et al. (2025) Nongrum, M.; Dhaliwal, B.; Na, Y.; Jamir, T.; Shekhawat, S.; Rao, K. D.; Ramani, S.; Albert, S.; and Closser, S. 2025. Disconnected data: mHealth data systems and challenges for primary health care workers in India. SSM - Health Systems, 5: 100124. OpenAI (2026) OpenAI. 2026. GPT-5.5 System Card. https://openai.com/index/gpt-5-5-system-card/. Accessed: 2026-07-23. OpenRouter (2026) OpenRouter. 2026. OpenRouter: A Unified API for Large Language Models. https://openrouter.ai. Accessed: 2026-07-31. Patel et al. (2010) Patel, N.; Chittamuru, D.; Jain, A.; Dave, P.; and Parikh, T. S. 2010. Avaaj Otalo: A Field Study of an Interactive Voice Forum for Small Farmers in Rural India. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, 733–742. ACM. Pipecat AI (2026a) Pipecat AI. 2026a. Pipecat: Open Source Framework for Voice and Multimodal Conversational AI. https://github.com/pipecat-ai/pipecat. Accessed: 2026-07-31. Pipecat AI (2026b) Pipecat AI. 2026b. stt-benchmark: Benchmarking Speech-to-Text with Semantic WER and TTFS Latency. https://github.com/pipecat-ai/stt-benchmark. Accessed: 2026-07-23. Poole-Dayan, Roy, and Kabbara (2026) Poole-Dayan, E.; Roy, D.; and Kabbara, J. 2026. LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 39116–39124. Pulikodan et al. (2026) Pulikodan, S.; Singh, A.; Basu, A.; Desai, N.; J, P. K.; Bhat, P. D.; Dharmaraju, R.; Gupta, R.; Udupa, S.; Kumar, S.; Sharma, S.; Sanka, V.; Tewari, D.; Dhand, H.; Kamat, A.; Singh, S.; Vashishth, S.; Talukdar, P.; Acharya, R.; and Ghosh, P. K. 2026. VAANI: Capturing the Language Landscape for an Inclusive Digital India. arXiv:2603.28714. Sarvam AI (2026) Sarvam AI. 2026. Evaluating Indian Language ASR. https://w.sarvam.ai/blogs/evaluating-indian-language-asr. Accessed: 2026-06-29. Sherwani et al. (2009) Sherwani, J.; Palijo, S.; Mirza, S.; Ahmed, T.; Ali, N.; and Rosenfeld, R. 2009. Speech vs. Touch-tone: Telephony Interfaces for Information Access by Low Literate Users. In Proceedings of the 3rd International Conference on Information and Communication Technologies and Development (ICTD), 447–457. IEEE. Si et al. (2023) Si, S.; Ma, W.; Gao, H.; Wu, Y.; Lin, T.-E.; Dai, Y.; Li, H.; Yan, R.; Huang, F.; and Li, Y. 2023. SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track, 39088–39118. Silero Team (2024) Silero Team. 2024. Silero VAD: Pre-Trained Enterprise-Grade Voice Activity Detector. https://github.com/snakers4/silero-vad. Accessed: 2026-07-31. Sun et al. (2024) Sun, G.; Feng, S.; Jiang, D.; Zhang, C.; Gasic, M.; and Woodland, P. 2024. Speech-based Slot Filling using Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, 6351–6362. Bangkok, Thailand: Association for Computational Linguistics. Vaessen (2024) Vaessen, N. 2024. jiwer: Evaluate Automatic Speech Recognition Systems. https://github.com/jitsi/jiwer. Accessed: 2026-08-01. Watson, Parnaby, and Kharrufa (2024) Watson, C.; Parnaby, A. W.; and Kharrufa, A. 2024. Precarious Experiences: Citizens’ Frustrations, Anxieties and Burdens of an Online Welfare Benefit System. arXiv:2405.08515. Yoon et al. (2023) Yoon, J.; Hwang, S.; Ran, H.; Bang, J.-U.; and Kim, K.-E. 2023. Adapting Text-based Dialogue State Tracker for Spoken Dialogues. In Proceedings of the Eleventh Dialog System Technology Challenge, 81–88. Prague, Czech Republic: Association for Computational Linguistics. Zheng et al. (2023) Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS). Appendix A Enrollment Form The full call flow of the enrollment form is shown in Figure 3. Figure 3: Call flow of the enrollment form used to build FormVoiceAgentBench. The agent speaks first and asks one question at a time, in the order shown. The answers to the branching questions decide which of the conditional fields are asked next. If a caller gives no valid answer, the rule-based layer re-prompts up to the retry limit shown for that field, and then either skips to the next question or ends the call marking it incomplete. Aadhaar is India’s national ID, of which only the last four digits are collected. Appendix B LLM Judges All LLM judges receive the conversation history, together with the agent output being graded, as the input, and produce a boolean score indicating whether the agent’s output adheres to the judge criteria, following the LLM-as-a-judge paradigm (Zheng et al. 2023). The temperature is set to 0 for every LLM judge inference. B.1 EXTRACT Judges Each extracted field is scored either by exact match or by an LLM judge (Table 8), as is the acknowledgement the agent emits alongside the extraction. openai/gpt-5.4-mini is used as the judge model. The following instructions are added as the system prompt for the LLM judge, in which criteria is replaced by the criteria for the field being scored: You are a highly accurate evaluator checking whether the value an agent produced for a single tool-call argument satisfies a given criteria. You will be given the tool name, the argument name, and the actual value the agent produced for that argument. Mark match true only if the actual value satisfies the following criteria, and false otherwise: criteria An example is given below. Field Scoring Name LLM judge District exact Clinic name LLM judge Child’s name LLM judge Pregnant? exact Gestational age exact Child’s DOB exact Calling number linked to clinic? exact Number linked to clinic exact WhatsApp = calling number? exact WhatsApp number exact Aadhaar last 4 digits exact Table 8: How each form field is scored during EXTRACT evaluation. Open-ended text fields are evaluated using an LLM judge, whereas every closed-ended field is scored by exact match. Conversation history assistant: Namaste! Yah ek automated call hai, ek nishulk maatru evam shishu swasthya seva ki taraf se. Aapne hamari seva se judne ke liye Anganwadi se sampark kiya hai. Yah ek muft seva hai, jismein aapko maa aur bachche ke swasthya se judi upyogi jaankari milegi. Hamari seva se judne ke liye hum aapse kuch sawal poochhenge. Kripya unke jawab bolkar dein. Aapki di gayi saari jaankari hamare saath surakshit rahegi. Apna poora naam bataiye (Namaste! This is an automated call from a free maternal and child health service. You contacted the Anganwadi to join our service. This is a free service through which you will receive useful information about mother and child health. To enroll you in our service we will ask you a few questions. Please answer them by speaking. All the information you give will stay safe with us. Please tell me your full name.) user: Mera poora naam Mrinmayee Kshirsagar hai (My full name is Mrinmayee Kshirsagar.) Evaluation. The turn is scored on the extraction call the agent makes. One field, the caller’s name, is open-ended and so is sent to an LLM judge with the criteria below; the acknowledgement is judged against a rubric, since it has no single correct answer. Both criteria follow. Acknowledgement criteria. A brief acknowledgement of the user’s previous answer, made of real words (not a non-lexical sound, grunt, or filler). This is encompassing and NOT restricted to any specific words: accept a receipt, back-channel, reassurance, praise, or transition phrase in any language, for example (but not strictly limited to) thik hai, achcha, ji haan, samajh gaya, koi baat nahi, “okay”, “yes”, “alright”, “got it”, “thank you”, “no problem”. Do not require any particular phrase, a ‘hearing-only’ tone, or extra brevity; phrases that imply the agent understood or accepted the answer (e.g. samajh gaya, “got it”) are acceptable. Only a missing acknowledgement, or a non-lexical sound/filler, should fail this criterion. Name criteria. The captured value refers to the SAME name as “Mrinmayee Kshirsagar” when read aloud. Treat them as EQUAL when they differ only by: letter casing; leading/trailing or internal word spacing; or a reasonable alternative romanisation / transliteration of the same spoken Hindi name, for example ‘ksh’ vs ‘ksha’, doubled vs single consonants, an inserted or dropped short ‘a’/schwa vowel, ‘v’ vs ‘w’, or ‘s’ vs ‘sh’ (so ‘Ghorakshpalli’, ‘Ghorakshapalli’ and ‘Goraksha Palli’ are all equal). Mark it NOT equal only if it is a genuinely different name or refers to a different place/word, not for spelling or casing variants of the same name. The field scored above, the caller’s name, is open-ended: the same spoken answer can be written many ways (for example, “Lakshmi” or “Laxmi”). An LLM judge therefore decides whether the extracted value matches a given rubric for that field. “Anganwadi” refers to the local clinic. A closed-ended field has one correct value and the extracted value is compared directly to it. The example below illustrates this. Conversation history assistant: Namaste! Yah ek automated call hai, ek nishulk maatru evam shishu swasthya seva ki taraf se. Aapne hamari seva se judne ke liye Anganwadi se sampark kiya hai. Yah ek muft seva hai, jismein aapko maa aur bachche ke swasthya se judi upyogi jaankari milegi. Hamari seva se judne ke liye hum aapse kuch sawal poochhenge. Kripya unke jawab bolkar dein. Aapki di gayi saari jaankari hamare saath surakshit rahegi. Apna poora naam bataiye (Namaste! This is an automated call from a free maternal and child health service. You contacted the Anganwadi to join our service. This is a free service through which you will receive useful information about mother and child health. To enroll you in our service we will ask you a few questions. Please answer them by speaking. All the information you give will stay safe with us. Please tell me your full name.) user: Mera poora naam Mrinmayee Kshirsagar hai (My full name is Mrinmayee Kshirsagar.) assistant: Theek hai. Apne zile ka naam bataayein (All right. Please tell me the name of your district.) user: Form ke liye, mera zila Kamrup Metropolitan hai. (For the form, my district is Kamrup Metropolitan.) assistant: Theek hai. Apne Anganwadi ka naam bataayein (All right. Please tell me the name of your Anganwadi.) user: Mere aanganwadi ka naam Ghorakshpalli hai (The name of my Anganwadi is Ghorakshpalli.) assistant: Theek hai. Kya aap abhi garbhvati hain? (All right. Are you currently pregnant?) user: Haan main abhi garbhvati hoon. (Yes, I am currently pregnant.) assistant: Theek hai. Aapko garbhvati hue kitne mahine hue hain? (All right. How many months pregnant are you?) user: Abhi meri pregnancy teen mahine ki hai. (My pregnancy is three months along right now.) Evaluation. The extracted value for how long the user has been pregnant must be 33 (months). The acknowledgement is judged against the same rubric as before. B.2 REPLY Judges Each reply is graded independently on five dimensions, each using a separate LLM judge: Correctness (openai/gpt-5.4-mini): does the reply ask the question the rule-based layer selected? Hindi Language adherence (openai/gpt-5.5): is the reply in Hindi? Reply Conciseness (openai/gpt-5.4-mini): is it a single question (with an optional informational note on the form progress)? No Acknowledgement (openai/gpt-5.5): does it avoid generating any form of acknowledgement for the previous user answer? No Value Echo (openai/gpt-5.5): does it avoid reading the captured value back to the caller? The full prompt for each is given in the example below. Only the “Correctness” prompt varies by test case; the other four are the same for every reply. Example. A REPLY test case is shown below, with the same conventions. Conversation history assistant: Namaste! Yah ek automated call hai, ek nishulk maatru evam shishu swasthya seva ki taraf se. Aapne hamari seva se judne ke liye Anganwadi se sampark kiya hai. Yah ek muft seva hai, jismein aapko maa aur bachche ke swasthya se judi upyogi jaankari milegi. Hamari seva se judne ke liye hum aapse kuch sawal poochhenge. Kripya unke jawab bolkar dein. Aapki di gayi saari jaankari hamare saath surakshit rahegi. Apna poora naam bataiye (Namaste! This is an automated call from a free maternal and child health service. You contacted the Anganwadi to join our service. This is a free service through which you will receive useful information about mother and child health. To enroll you in our service we will ask you a few questions. Please answer them by speaking. All the information you give will stay safe with us. Please tell me your full name.) user: Mera poora naam Mrinmayee Kshirsagar hai (My full name is Mrinmayee Kshirsagar.) assistant: Theek hai. Apne zile ka naam bataayein (All right. Please tell me the name of your district.) user: Form ke liye, mera zila Kamrup Metropolitan hai. (For the form, my district is Kamrup Metropolitan.) assistant: Theek hai. Apne Anganwadi ka naam bataayein (All right. Please tell me the name of your Anganwadi.) user: Mere aanganwadi ka naam Ghorakshpalli hai (The name of my Anganwadi is Ghorakshpalli.) assistant: Theek hai. Kya aap abhi garbhvati hain? (All right. Are you currently pregnant?) user: Haan main abhi garbhvati hoon. (Yes, I am currently pregnant.) tool: validate_user_turn tool output: Correct value True captured for the pregnancy field. Pending questions: (1) gestational age in months, optional, integer, script “Aapko garbhvati hue kitne mahine hue hain?” (How many months pregnant are you?); the answer is a number of months. Evaluation. The reply the agent produces next is sent to all five judges. Each judge receives the conversation above and their respective system prompts. Correctness. You are a highly accurate evaluator evaluating the response of an agent to a user’s message. You will be given a conversation between a user and an agent along with the response of the agent to the final user message. You need to evaluate if the response adheres to the evaluation criteria: The reply advances the interview to the correct next question: the one whose script is “Aapko garbhvati hue kitne mahine hue hain?” (How many months pregnant are you?). It asks that specific question (not an earlier, later, or invented one), does not re-ask an already-answered question, and does not end the call. Hindi Language adherence. You are a highly accurate evaluator checking the language of an agent’s reply. You will be given a conversation between a user and an agent along with the agent’s response to the final user message. You need to evaluate if the last agent response adheres to the evaluation criteria: The last agent reply should be largely written in Hindi (Devanagari script). It does not switch to roman alphabets or english words except for using proper nouns and everyday common-use english words in devnagari (e.g. saying “Whatsapp” as its Devanagari transliteration is fine) or digits in roman numerals are fine. If you think there has been any violation, give concrete examples of the violation in your reasoning. Reply Conciseness. You are a highly accurate evaluator checking whether an agent’s reply is concise. You will be given a conversation between a user and an agent along with the agent’s response to the final user message. You need to evaluate if the response adheres to the evaluation criteria: The reply is concise: it asks exactly ONE question, optionally preceded by the ONE informational preamble the script requires for this field. It does not bundle multiple questions, enumerate pending fields, summarise progress, or add chit-chat beyond what the script requires. No Acknowledgement. You are a highly accurate evaluator checking whether an agent’s reply is free of any acknowledgement of the user’s previous answer. You will be given a conversation between a user and an agent along with the agent’s response to the final user message. You need to evaluate if the response adheres to the evaluation criteria: The reply begins directly with its substantive line. Examples: the question, the re-ask, or a required informational/skip script like stating the progress update of the form completion so far, etc. It contains NO acknowledgement of the user’s previous answer like one or more of the following but not strictly limited to them: no receipt, back-channel, reassurance, praise, or transition phrase in any language (e.g. “it’s okay”, “okay”, “yes”, “alright”, “got it”, “thank you”, “no problem”). No Value Echo. You are a highly accurate evaluator checking whether an agent’s reply avoids echoing the user’s just-provided value. You will be given a conversation between a user and an agent along with the agent’s response to the final user message. You need to evaluate if the response adheres to the evaluation criteria: The reply does not read back, repeat, or confirm the specific value the user just provided for the field that was just captured (e.g. echoing their name/number/DOB etc. back at them as confirmation). Asking the next scripted question, or speaking a required verbatim re-ask line, even one containing a quoted example token, is NOT an echo and must pass. The chat history will contain a tool call prior to the last agent message. You are not supposed to check the agent message against that tool call message. You need to compare the last agent message with only the user messages prior to it. B.3 Calibration Procedure Every judge was calibrated against human labels before being used to score the benchmark. We drew a batch of 50 unit tests, ran each judge over them, and had the authors independently label every judgment. The initial prompts were not fully aligned: each judge disagreed with the human label on some cases. We revised the prompts over several iterations until every judge agreed with the human labels on all 50 tests. We then applied the final prompts unchanged to a held-out batch of 50 tests that had played no part in the iteration, on which the LLM judge outputs matched the human labels on every case. We acknowledge that the size of the calibration dataset is small and plan to expand it in future work. Appendix C Full Results with Confidence Intervals Tables 9–12 show the model comparison results for speech-to-text (STT), extraction accuracy, form completion and response accuracy with 95% confidence intervals. Model WER ↓ LLM-WER ↓ Cost ($) ↓ Scribe v2 0.835±0.0150.835± 0.015 0.062±0.0240.062± 0.024 0.1720.172 Chirp 3 0.981±0.0070.981± 0.007 0.055±0.0250.055± 0.025 0.4240.424 Saaras v3 1.007±0.0041.007± 0.004 0.074±0.0280.074± 0.028 0.1370.137 GPT-4o-transcr. 1.012±0.0041.012± 0.004 0.127±0.0330.127± 0.033 0.1590.159 Nova-3 0.826±0.0150.826± 0.015 0.104±0.0310.104± 0.031 0.1270.127 Table 9: Comparison of STT models. Since models have different billing units, the total cost (USD) of transcribing the entire benchmark is reported. Model Reference Saaras v3 Scribe v2 Nova-3 GPT-5.5 99.79±0.9999.79± 0.99 95.53±1.0395.53± 1.03 98.62±0.6498.62± 0.64 95.16±1.0795.16± 1.07 Gemini 3.5 Flash 99.36±1.2299.36± 1.22 95.69±1.0195.69± 1.01 98.94±0.5898.94± 0.58 96.38±0.9496.38± 0.94 Claude Opus 4.8 99.15±1.3299.15± 1.32 96.22±0.9696.22± 0.96 98.40±0.6798.40± 0.67 96.01±0.9896.01± 0.98 Claude Sonnet 4.6 99.15±1.3299.15± 1.32 96.01±0.9896.01± 0.98 98.94±0.5898.94± 0.58 95.96±0.9995.96± 0.99 GLM-5.1 98.72±1.4898.72± 1.48 94.52±1.1294.52± 1.12 63.40±2.2063.40± 2.20 95.43±1.0595.43± 1.05 Gemini Pro 97.23±1.9097.23± 1.90 94.26±1.1594.26± 1.15 97.29±0.8497.29± 0.84 94.63±1.1294.63± 1.12 Gemini 2.5 Flash 96.17±2.1496.17± 2.14 95.27±1.0695.27± 1.06 98.03±0.7398.03± 0.73 95.43±1.0595.43± 1.05 Gemini 3 Flash 95.96±2.1995.96± 2.19 84.79±1.7084.79± 1.70 88.83±1.5088.83± 1.50 86.28±1.6386.28± 1.63 GPT-5.4-mini 92.34±2.7692.34± 2.76 90.05±1.4390.05± 1.43 96.76±0.9196.76± 0.91 91.54±1.3491.54± 1.34 Mistral Medium 3.5 89.36±3.1189.36± 3.11 87.82±1.5687.82± 1.56 92.34±1.2992.34± 1.29 87.77±1.5687.77± 1.56 GPT-4.1 85.11±3.5185.11± 3.51 75.53±1.9975.53± 1.99 78.94±1.9178.94± 1.91 79.84±1.8779.84± 1.87 Table 10: Per-turn extraction accuracy using reference transcripts and the outputs of the three STT models as inputs. Model Reference Saaras v3 Scribe v2 Nova-3 Gemini 3 Flash 100.00±0.00100.00± 0.00 85.24±0.7785.24± 0.77 89.48±0.6089.48± 0.60 83.92±1.0183.92± 1.01 Gemini 3.5 Flash 100.00±0.00100.00± 0.00 90.74±0.6190.74± 0.61 92.50±0.5392.50± 0.53 87.09±0.7687.09± 0.76 GPT-5.5 99.95±0.0999.95± 0.09 89.97±0.7389.97± 0.73 89.37±0.6489.37± 0.64 84.73±0.8084.73± 0.80 Claude Sonnet 4.6 99.66±0.2399.66± 0.23 91.55±0.7191.55± 0.71 93.01±0.5093.01± 0.50 84.54±0.7784.54± 0.77 GLM-5.1 99.56±0.2799.56± 0.27 90.92±0.7590.92± 0.75 58.66±1.4658.66± 1.46 86.09±0.7586.09± 0.75 Claude Opus 4.8 99.41±0.3199.41± 0.31 92.22±0.6192.22± 0.61 92.03±0.5692.03± 0.56 85.83±0.8885.83± 0.88 Gemini 2.5 Flash 99.06±0.3899.06± 0.38 91.76±0.7091.76± 0.70 92.39±0.5292.39± 0.52 85.50±0.7385.50± 0.73 Gemini Pro 98.90±0.4298.90± 0.42 92.55±0.5292.55± 0.52 93.47±0.4593.47± 0.45 86.53±0.7086.53± 0.70 GPT-5.4-mini 98.35±0.4898.35± 0.48 90.68±0.7590.68± 0.75 90.10±0.5590.10± 0.55 84.40±0.7984.40± 0.79 GPT-4.1 95.99±0.7095.99± 0.70 88.82±0.7488.82± 0.74 91.50±0.5391.50± 0.53 86.33±0.7486.33± 0.74 Mistral Medium 3.5 92.83±0.6892.83± 0.68 90.39±0.6890.39± 0.68 91.84±0.4991.84± 0.49 83.37±0.7683.37± 0.76 Table 11: End-to-end form-completion accuracy using reference transcripts and the outputs of the three STT models as inputs. Model Response accuracy Latency (ms) Cost ($/turn) Reference Scribe v2 Claude Sonnet 4.6 100.00±0.00100.00± 0.00 97.77±0.7597.77± 0.75 5148±3925148± 392 0.0146±0.00000.0146± 0.0000 GPT-4.1 100.00±0.00100.00± 0.00 96.60±0.9196.60± 0.91 3647±6563647± 656 0.0025±0.00010.0025± 0.0001 GPT-5.4-mini 98.09±1.2898.09± 1.28 96.70±0.8596.70± 0.85 3356±5233356± 523 0.0008±0.00000.0008± 0.0000 Gemini 3 Flash 97.23±1.4997.23± 1.49 96.17±0.9696.17± 0.96 ±2661± 239 0.0017±0.00000.0017± 0.0000 Gemini 3.5 Flash 97.02±1.2897.02± 1.28 94.73±1.1194.73± 1.11 3496±13223496± 1322 0.0052±0.00010.0052± 0.0001 Table 12: REPLY response accuracy using reference transcripts and transcripts from Scribe v2 as inputs. Appendix D Model Selection D.1 EXTRACT Selection Table 13 gives the optimal EXTRACT model using the weighted-sum scalarization method for 0.50≤wa≤0.900.50≤ w_a≤ 0.90 with wℓ=0.1w_ =0.1. waw_a wcw_c Model U waw_a wcw_c Model U waw_a wcw_c Model U 0.50 0.40 Gemini 3.5 Flash 0.763 0.64 0.26 Sonnet 4.6 0.776 0.78 0.12 Sonnet 4.6 0.843 0.51 0.39 Gemini 3.5 Flash 0.763 0.65 0.25 Sonnet 4.6 0.781 0.79 0.11 Sonnet 4.6 0.848 0.52 0.38 Gemini 3.5 Flash 0.763 0.66 0.24 Sonnet 4.6 0.786 0.80 0.10 Sonnet 4.6 0.852 0.53 0.37 Gemini 3.5 Flash 0.763 0.67 0.23 Sonnet 4.6 0.791 0.81 0.09 Sonnet 4.6 0.857 0.54 0.36 Gemini 3.5 Flash 0.763 0.68 0.22 Sonnet 4.6 0.795 0.82 0.08 Sonnet 4.6 0.862 0.55 0.35 Gemini 3.5 Flash 0.763 0.69 0.21 Sonnet 4.6 0.800 0.83 0.07 Sonnet 4.6 0.867 0.56 0.34 Gemini 3.5 Flash 0.764 0.70 0.20 Sonnet 4.6 0.805 0.84 0.06 Sonnet 4.6 0.871 0.57 0.33 Gemini 3.5 Flash 0.764 0.71 0.19 Sonnet 4.6 0.810 0.85 0.05 Sonnet 4.6 0.876 0.58 0.32 Gemini 3.5 Flash 0.764 0.72 0.18 Sonnet 4.6 0.814 0.86 0.04 Sonnet 4.6 0.881 0.59 0.31 Gemini 3.5 Flash 0.764 0.73 0.17 Sonnet 4.6 0.819 0.87 0.03 Sonnet 4.6 0.886 0.60 0.30 Gemini 3.5 Flash 0.764 0.74 0.16 Sonnet 4.6 0.824 0.88 0.02 Sonnet 4.6 0.890 0.61 0.29 Gemini 3.5 Flash 0.764 0.75 0.15 Sonnet 4.6 0.829 0.89 0.01 Sonnet 4.6 0.895 0.62 0.28 Sonnet 4.6 0.767 0.76 0.14 Sonnet 4.6 0.833 0.90 0.00 Sonnet 4.6 0.900 0.63 0.27 Sonnet 4.6 0.772 0.77 0.13 Sonnet 4.6 0.838 Table 13: Optimal EXTRACT model under weighted-sum scalarization for 0.50≤wa≤0.900.50≤ w_a≤ 0.90, with wℓ=0.1w_ =0.1 and wc=1−wa−wℓw_c=1-w_a-w_ , and its composite score U. D.2 REPLY Selection GPT-5.4-mini is optimal throughout 0.50≤wa≤0.900.50≤ w_a≤ 0.90 (wℓ=0.1w_ =0.1), among the models on the Pareto frontier (Figure 4), with a consistent composite score of U=0.9U=0.9 across the sweep. Figure 4: Cost–quality–latency trade-off among REPLY models satisfying the deployment constraints. The logarithmic x-axis shows cost per turn, the y-axis shows response accuracy using Scribe v2 transcripts, and marker area encodes p95 latency per turn (bigger is slower). Both models are Pareto-optimal across the three objectives. The model selected for deployment is highlighted. Appendix E Implementation Details Orchestration and telephony. We use Pipecat (Pipecat AI 2026a) for orchestration and Exotel (Exotel 2026) for telephony, which delivers 8 kHz μ-law audio over a WebSocket. For VAD, we use Silero (Silero Team 2024) at the same 8 kHz sample rate. It marks the start of speech after 0.10.1 s, the end after 0.20.2 s of silence, and ignores any audio that scores below 0.70.7 confidence or 0.60.6 loudness. The agent waits a further 0.4 s after that before marking the turn as completed, so a caller pausing mid-answer is not cut off. If the caller has not spoken for more than 3.0 s since the agent stopped speaking, we re-prompt. Each audio file in the benchmark is 16 kHz mono 16-bit PCM. Models. All LLM calls are served through OpenRouter (OpenRouter 2026). We set the temperature to 0 for non-reasoning models, and the reasoning effort to “medium” for EXTRACT and “low” for REPLY. For both, we cap the output at 16,00016,000 tokens and pass the most recent 200 turns of the conversation as input. For TTS, we use Google Cloud Chirp 3 HD (Google Cloud 2026) with the female Hindi voice Achernar, slowed to 0.9×0.9× the default speed so the questions are easier to follow. Evaluation harness. All STT and LLM evaluations were run using Calibrate (Dalmia and Doshi 2025). Environment. Experiments were run from a MacBook Pro (Apple M4 Pro, 2424 GB) on macOS 15.7 with Python 3.11, using pipecat-ai 1.2.1, openai 2.15.0, instructor 1.13.0, pydantic 2.12.3, jiwer (Vaessen 2024) 4.0.0, indic-nlp-library (Kunchukuttan 2020) 0.92, pydub 0.25.1 and numpy 2.2.6. Determinism. No random seeds were set: every model is served by a hosted API that offers no determinism guarantee, so identical settings can still produce different outputs. Each configuration was evaluated once over the full test suite, so the confidence intervals in Section C reflect variation across test items rather than across repeated runs.

Instant research discovery