Sleepless in Samsung?

I'm spending a couple of days at the DARPA AI Colloquium — about which more later —  and during yesterday's afternoon session, I experienced an amusing conjunction of events. Pedro Szekeley gave a nice presentation on "Advances in Natural Language Understanding", after which one of the questions from the audience was "Hasn't Google solved all these problems?" Meanwhile, during the session, I got a quasi-spam cell-phone call trying to recruit me for a medical study, and since my (Google Fi) phone was turned off, it went to voicemail, and Google helpfully offered me a text as well as audio version of the call.

The result illustrates one of the key ways that modern technology, including Google's, fails to solve all the problems of natural language understanding.

Here's Google's text version of the voicemail:

Hello, we're contacting you from the University of Pennsylvania's behavioral sleep medicine program about a study for people who have trouble falling asleep or staying asleep. If this is true for you, please visit www.samsung.com. If not, perhaps you could share this information with appropriate friends or family. The website again is w w w. Sleepless in philly.com. If you have any questions, please feel free to contact us by phone at XXX-XXX-XXXX. Thank you for your time.

Since I don't have any problems falling or staying asleep, I wasn't tempted to click on the links. But still, I wondered what Samsung could possibly have to do with it, and so I listened to the audio version (presented below with the phone number noisified):

Your browser does not support the audio element.

In this example, Google's speech-to-text accuracy is extremely good in terms of Word Error Rate — depending on how you do the division into "words", it's somewhere in the low single digits.

But as I suspected, the caller was not trying to send me to www.samsung.com to enroll in the study — Google somehow mistranscribed "sleepless in philly" as "samsung".

Pedro joined most of the other DARPA AI speakers in noting that a key failure of contemporary "artificial intelligence" is that it's actually pretty stupid, in the sense of having no common sense — and this failure illustrates the point. Any sentient human inhabitant of the modern world knows that you probably can't enroll in a sleep study at www.samsung.com. Another simple clue that Google missed is provided by the wording "The website again is …", which tells us that the two URLs should not be as wildly different as they are in the transcript.

And finally, while it's good that Google correctly recognized  "w w w dot sleepless in philly dot com"  in the second rendition of the website name, it's a failure of modern-era common sense not to join that sequence into a coherent URL. The transcription of the first version of the website sends me to an electronics company, and the second one sends me to a local news aggregator — neither of which lets me sign up for the sleep study.

In fact this voicemail transcription was good enough for me to figure out what's going on, even without listening. But if we were using this stuff to populate a knowledge base …