New Dialpad research offers a reproducible way to test whether tool-calling agents can understand and act from speech—not just clean text.
Every business should truly know its customers. As AI agents increasingly meet customers by voice, that understanding must survive the gap between what is written and what is spoken.
Today, many widely used tool-calling benchmarks are text based. They can tell us whether a model chooses the correct API call from a clean written prompt. They do not necessarily tell us whether a voice agent can do the same when a request arrives as audio, with differences in speaker, delivery, and environmental noise.
In From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Omni-Modal Tool-Calling LLM Agents, accepted to the REALM workshop at EMNLP 2026, we introduce a way to extend verified text benchmarks into controlled audio evaluations, without changing the original tool schema or gold labels.
A controlled way to measure the text-to-voice gap
Our framework converts text-based tool-calling benchmarks into paired text–audio instances using text-to-speech, speaker variation, and environmental noise. Because the original annotations remain intact, researchers can compare a model’s text and voice performance on the same underlying task.
We evaluated seven omni-modal models on audio-converted versions of the Confetti and When2Call benchmarks. The results show that voice performance is strongly model- and task-dependent. Gemini-3.1-Flash-Live achieved the highest Confetti score (70.4), while GPT-Realtime-1.5 performed best on When2Call (71.9).
There was no universal voice penalty. On Confetti, the text-to-voice gap ranged from 1.8 points for Qwen3-Omni to 4.8 points for GPT-Realtime-1.5. In other words, a model’s performance on text alone does not tell the full story about how it will perform in voice.
Most failures were not simply the wrong tool
The paired evaluation also helps diagnose what changes when input moves from text to audio. In a targeted analysis of failure cases, degradations most often reflected misunderstandings of argument values in speech.
That is a useful distinction. An agent may select the right tool but still fail to execute successfully because it misunderstands the details needed to make the call. For example, the problem might be a date, location, number, or other field that has been degraded or interpreted differently in audio.
This is why voice-agent evaluation needs more than a single leaderboard score. Teams need visibility into whether the issue is high-level decision making, tool selection, argument schema, or the values supplied to a correct tool.
Evaluation has to reflect the environment of use
The framework also includes an ambiguity-based stress test, text-only comparisons, and a reference-free LLM-as-judge protocol for settings where gold labels are unavailable. In our experiments, open-source Qwen3 judges with at least 8B parameters exceeded 80% agreement with proprietary judges, pointing to a privacy-preserving option for some evaluation workflows.
The broader takeaway is simple: voice agents should be evaluated in voice. Clean text remains a useful starting point, but it cannot stand in for the acoustic and conversational conditions an agent will encounter in production.
Dialpad is the AI platform for customer experience. Our research is focused on helping organizations understand real conversations and use that intelligence to take the right action. For voice agents, that means measuring performance at the point where customer intent reaches the system—not only after it has been reduced to an ideal text prompt.
This framework is a first-stage diagnostic, not a replacement for evaluation on naturally spoken customer conversations. The study uses text-to-speech-generated audio and a finite set of current models and benchmarks. But by preserving tool schemas and gold labels, it gives researchers and builders a reproducible way to make more informed cascade-versus-omni deployment decisions before moving into real-world testing.
A Reproducible and Verifiable Framework for Evaluating Omni-Modal Tool-Calling LLM Agents was accepted to REALM at EMNLP 2026. The paper is authored by Md Tahmid Rahman, Xue-Yong Fu, Seyyed Saeed Sarfjoo, Quinten McNamara, Jonas Robertson, and Shashi Bhushan TN of Dialpad.