"Voice AI" is really Text AI with a voice overlay

Martijn Bartelds, Federico Bianchi, James Zou, "Real-Time Voice AI Hears but Does Not Listen", 6/24/2026:

Speech conveys information through both words and vocal delivery. We evaluate four leading production realtime voice systems – OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash – on tasks where the words and the delivery patterns both convey meaningful information. Across three consequential scenarios, all four systems act on the words rather than the voice. They end calls with crying callers who insist nothing is wrong, approve wire transfers authorized in frightened voices, and enroll callers whose agreement is clearly sarcastic. Surprisingly, this is often not a failure of perception. When asked directly, three of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions. We observe a similar pattern when these realtime voice systems estimate accent and age, as their responses frequently follow the biases of the words rather than the acoustic properties of the speaker. We term this disconnect between perception and action the emotional intelligence gap of voice AI. Prompting systems to explicitly attend to vocal delivery improves performance only partially and inconsistently. Our findings show that current realtime voice AI systems often behave as if speech had been reduced to a transcript, suggesting that they should be used with caution in settings where the tone and emotion of delivery convey important information.

An html version is here, with in-line voice demos like this one:

When a caller's words and delivery disagree, real-time voice systems go with the words. A 911 caller is crying but insists everything is fine, and GPT Realtime 2 ended the call.

Your browser does not support the audio element.

When specifically asked, the systems can mostly recognize when the speaker sounds distressed, frightened, or sarcastic — but they don't perform that recognition when not specifically asked, and don't act on it. And accent and age are "only partially perceived" — when older adult voices read a child's script, most of the systems guess that a child is speaking.

The code and data used in the paper are available here.

I'll be interested to see whether the eventual journal home of this paper — if any — manages to offer similar in-line audio, or just sends readers off to a data page.