There's a lot of hype around the application of generative AI to the voice channel. Listen in as Nikola and Michelle discuss who's doing it and how, what pitfalls organizations need to be aware of, and what you can expect from the next generation of models like GPT 4.0 and beyond. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
NikolaThere's a lot that you can do, for example, with untuned GPT 4, where previously you would have had to tune a smaller LLM to get to some level of performance, I think if we can get these extremely large, more advanced LLMs to do all the work, then We might not even have to have private instances
MichelleHi everyone. And welcome to the Poly AI podcast, where we'll be exploring topics like generative AI, voice AI, and the whole conversational landscape. We'll have interesting topics and interesting guests for you each week. And I'm one of your hosts, Michelle Schroeder. And it's my job today to ask questions to our expert co host, Nicola Mierczyk who has a PhD from Cambridge in machine learning and natural language understanding, was selected for Forbes 30 under 30, and who founded Poly AI. In 2017, so welcome.
MichelleYeah, I'm excited to get into this conversation. Cool. Yeah, so I figured we'd start with a primer for outsiders. I think a lot of people are hearing a lot about generative AI, and this is after just a huge wave of news and buzz around large language models. And while it's pretty safe to assume most people don't understand either. I think maybe if we just start by understanding both together, understanding what makes them different. And I was given this semi helpful model. And I want to know from you if it's correct, if this is a way that people can think about this.
MichelleSo not all generative AI tools are LLMs or rely on LLMs, but all LLMs are a form of generative AI. Is that correct?
NikolaOkay. Okay. To put my purpose for your hat on generative models are basically one family. Machine learning models, statistical models, really, which model what is known as the joint distribution of the data that they observe, right? And that broadly means that they're they're called generative because once they model the data that they're trying to learn, they're able to produce data of that kind or to generate it, right? That data could be text, right? And that is what language models do, large or not.
NikolaAnd it could be images. It could be something else entirely. It could be medical data. It could be all sorts of things, right? And data of any kind. The other family of models is called discriminative. And that's basically where you're not really trying to understand the substance, essence of that data, but rather you're trying to... So if you want to learn how to classify it, so say medical data, you'd be like, is this an example of this virus? Where is it? Not if is it a, in the case of text, it could be, is it an article about sports or about politics?
NikolaGenerative models are really learning the structure of that data, right? For example, with text, they would often be trained on things like Wikipedia or a data set known as common crawl, which is basically crawling the internet and just trying to and just hold that data and then reproduce data. Of a similar kind back so language models are models that are trained through observing text and they can produce text that can continue a sentence if you started or in the case of large language models and what we've all recently been excited about is you can really create even autonomous agents.
NikolaFrom them where basically you instruct the language model to behave in a certain kind of, because it's seen so much data and the case of things like GPT 3 and similar models, a lot of synthetic data that followed a certain pattern of conversation. Then you're able to have a system that produces text in a conversation with you that is very reminiscent of an actual human. That you could be having that conversation with. So LLMs are an instance of generative models. There are different kinds.
NikolaSo when you look at a lot of the stuff that was done mid journey and stability and other models like those were models for images, right? You could have models that are jointly model images and text. You could have models for music, right? It doesn't have to be just text, but when we're dealing with text, then we're using a language model. And. In the case of what we're talking about here, we call them large language models simply because they're far larger and more impressive than anything we have trained so far.
MichelleYeah. Tell me this. When you were getting your PhD and you were studying, in and. Learning about these large language models, creating these large language models did you ever expect that one day you'd be able to give it some agency, tell it what to do and have it sound human, have it actually generate a plausibly human response?
NikolaYeah, so the problem operates at several levels, right? No one really called them large language models until very recently, right? A language model is one of the first things you learn about when you do a natural language processing course, right? And I think one of the favorite gimmicks in most courses is, you could start something that sounds a bit Shakespearean. And you run the language model to predict the continuation, and you will get a few verses that really do sound Shakespearean, right? And, in the simplest form, what a large language model does now is it's able to continue that a lot better, a lot more intelligently, holding like dependencies, say in the poem that it's writing, referring to something that it generated.
NikolaSo it's all really impressive, right? Now, in terms of Telling it to behave autonomously and stuff, that was really a matter of the data that was used to train these LLMs, right? So people at OpenAI worked longer than hard to create that kind of data and to, create training procedures that were conducive to getting that kind of behavior. Like always with machine learning. Big breakthroughs are not purely scientific. They're also feats of great engineering. And I'll say that, what they've done with building GPT 3.
Nikola5 and GPT 4 is, probably. An effort comparable in in scale to, I don't know, building the pyramids or something like that. It's really, it's not all that easy to reproduce, which from the other models and the fact that they haven't really yet in match that performance easily. So I think there's a lot of good hard work that went into creating the data and the training procedures that they're using. It's not just that there's been a scientific Shift if you were to be a bit cynical about it You could even say that there isn't really a major scientific breakthrough and that the only real engineering improvement that happened is the use of transformers rather than say LSTM cells Cool.
NikolaAgain, these are building blocks of neural networks, right? So it's saying you have a better building block in creating the whole thing. But the whole thing, we would have probably gotten there even without transformers. But it might have taken us longer, maybe substantially longer, maybe a decade or two. Maybe even more. It's hard to know at this point. Yeah.
MichelleI'm glad Transformers came about. Given all of the progress that we've made just as a company, but also as a society with what's happening with generative AI, I think it's a lot of really important developments. One thing I did want to ask you, it's, you talked about OpenAI and their sort of pioneering work. Unless you look at it cynically and give a lot of that
Nikolacredit to transformers. It is, I only say it to put it in context. It is groundbreaking. It is amazing what they've done and the beliefs that they've had in in, putting the whole thing together and building like bigger, better things, improving the models, data, training, hardware. They've done an incredible thing. Yeah,
Michelleand I think I agree with you from a layman's perspective, but just given sort of recent developments with open AI, I think it was only a matter of time before they put a voice to GPT before they allowed you to interact with it from like a voice interface perspective. And I'm just curious as a person who's done a lot of pioneering work in the voice side of things, what you think about that you mentioned on a previous podcast. That you felt that voice was the most intuitive interface. Is this confirming or affirming of that to you?
MichelleI
Nikolamean, look, I think plugging speech recognition on one end and text to speech on another does not make it does not complete the process of building a really good voice assistant, right? I think that, OpenAI has done work on speech recognition. They recently announced that they're doing work on text to speech, and I have no doubt that they will be one of the, main players building the best versions of these things, right? Thank you. When you put it all together, it's still not the whole story, right? So for example what you'll often see when you interact with.
NikolaLike simple speech recognition plus an LLM versions of voice assistance is that speech recognition errors that happen will just be ignored and there will be absolute confidence in how these models continue conversation. If I said, Hey, I referred to Michelle, but actually it's transcribed as Michael or something similar, like the model will just continue as if, like I typed out. What they had recognized. So that kind of like composite motion of knowing that the input is not perfect. That's something that the LLM is not yet primed for.
NikolaSo when you build enterprise applications there's still a lot of room to improve in just dealing with speech recognition and everything that goes wrong on that front, because even prior to the advent of LLMs, the things that we have been building with voice assistance did not really. Suffer from like, whenever we misunderstand something, it's not really due to our natural language understanding models failing. It's mostly to do with the fact that the speech recognizer just does not.
NikolaHas not captured the right information, and we're unable to parse that from its output. So that's the kind of like big deal with voice. The other part, obviously, is like that whole thing, like what does it sound like? Is that generate, generated voice on text to speech model? Is it something that's good, right? And there we like internally at Poly AI for a long time have had two tests, right? One is, can I tell that it's a human or not? That's like your Turing test equivalent. And then the second thing is is this a really good voice, right? Does this sound like my best agent or someone as good as my best customer service rep?
NikolaAnd there are two very different levels, right? A lot of companies out there at this point have text to speech models where you can't really tell. At least from a few words whether it's human or not, and they're getting really good. But often, like when you hear them there is still quite a there's a lot of room for improvement compared to just a human with good diction, prosody and good expression. Right?
MichelleYeah. Yeah. No, it's interesting that you mentioned the Turing test because I think there's some controversy over the past few years about, putting these models to the test. Are they actually able to pass a Turing test? And like you say, voice is a lot harder. Not just because of being able to pick up what's being said and then respond to it in an adequate amount of time in a human amount of time. Yeah. It's also something else, and actually this is a bit of a strange kind of aside, but I remember reading about some research a long time ago about like humans ability to tell if they're speaking with a psychopath, and it is relevant, I promise, but just the amount of emotion that the person puts into their voice, the inflections, like where they choose to put in emotion and how, and I'm just wondering, Even with the best sounding voices today, even with the lowest latency sort of conversations, and the best, all of the things that need to go into a stack to make a good voice and to make something conversational.
MichelleDo you think that there is voice out AI out there that can pass a Turing test that can make people feel like they're speaking to a human?
NikolaYeah. So again, the original Turing test isn't really phrased in terms of voice, right? It's really about the technical interface and language. And there are algorithms really, not even like heavily machine learning based ones that have passed it, right? These conditions are those conditions. And what I think the academic community will tell you is that like the test is really not worth a whole lot because it's not really a measure of intelligence, just a game to be gained. And once you give clear conditions, then gaming, anything's quite easy, right? Like a law or a game or anything like that.
NikolaLike whether, like the Turing test is the best thing to talk about is the beta, right? But while we're at it, when it comes to voice, I think what you would notice now. With generative approaches is that the voice itself, depending on what you use, would vary and, it would be enough to, the way you could call it out if you really wanted to port the test to voice is you could say, Hey would you mind, saying my name is Nicola flat, not Nicola, because I'm not a British woman, I am a Serbian man, hence the style is that like the model will fall flat on his face because that level of control say over stress.
NikolaIt's just not there and that's something that to us as humans it's pretty intuitive. And, my last name, we could go around in circles around trying to say, if you haven't grown up in, with a Slavic language as one of your bases, you're probably going to struggle for quite some time, we could have that conversation. I tried. Yeah, but it would be something that we could iterate on as humans, and I would know that you're human, right? Whereas if a system just goes and misunderstands it, and, typical speech recognizer will.
NikolaRecognize exotic things like muffin, mattress. I once called Brexit recognized as just from the speech recognition. It's just not a word in the vocabulary of the language, right? And as humans, we still have far more expressive models of how we represent things and how we store them in our mind. Like playback audio in a way that these things have just not been built to do yet. So there's a lot of depth in there. If we were to. People imagine a world where like these deep fakes are so convincing that people never stop and tell.
NikolaAnd I think it's an important topic because people are not expecting it. Then, they might never think to check, right? If it's a scamming, a scammer's call telling you that, I don't know, a loved one is in trouble and you're calling from a hospital or some, nefarious scenario like that, then you might not just think to check. But I think as these things proliferate and I don't know about you, but I get spam calls all the time in English and other languages. And I think it's just people. We'll very quickly be able to throw these things off and technology will catch up.
NikolaThings like stress on words there's a lot that can be done if you're expecting it.
MichelleYeah, you ask them to pronounce your name, I think. Instantly that it's not real. I know what's interesting just from a poly perspective is that you get these turns and these inflections and this expressiveness. That does feel very human. It feels so human. That a lot of. Our customers are saying things like, thank you. Have a nice day. We're asking about the weather. You're like, really trying to have a 2 way conversation as though we're human because, most likely they're believing that they're speaking with a human and I'm curious about.
NikolaYeah, sometimes they change their mind, right? We've had all sorts of... Here, I'll tell you an example.
DemoHi, thanks for calling Bella Italia Telford. How can I help you today? Book a table. When would you like to come in? Today. What time do you think you'll be here? 6. 15. And how many people? Seven. Let me just have a look at what space we have. It looks like we've got nothing within two hours of 6. 15pm. Is there another day or time I can check for you?
DemoNow, I'm so sorry, I thought I was talking to a robot, so I was being really rude, I do apologize. Not really. nO, that's no problem at all, thanks ever so much.
NikolaA very British reaction. Those things happen, and they get away, right? But it's it's very rewarding when you see something like that. Not that we designed it for you our philosophy is not to trick people into thinking that they're speaking with... A human, we want them to just not feel like they're speaking with a machine.
MichelleAbsolutely. Yeah, that's such a cool example. And I'm just curious about the generative version of those types of calls. I know we're still, it feels like on the early frontier of a generative bot being able to take that entire conversation. And make it sound as wonderful as what you just showed, like as human, as expressive as, that, that sort of pause that, let me just look that, that sort of feeling that you got from that. How far off are we from seeing generative be able to do a call that sounds exactly like that?
NikolaI think we're pretty close. I think that there's one piece just around like the speech and how you do it. What do we use is an approach, which basically. Gets a voice actor to record a lot of things and then we synthesize things that we don't have in a very seamless way We've found that to be the best thing we can do better than a fully neural approach. Neural approaches are getting better and better So there is more and more that can be done and at some point probably some point very soon It's likely that we'll have convergence there, right?
NikolaThe other issues with generative are, I think people talk about it a lot It's basically just the controllability. We won't... Actually get something very weird out of the model of hallucination, right? An answer to a question that the model might be just freestyling rather than pulling from data that you've actually provided. Because these things are trained on so much data that say, if you were building I don't know, concierge bought for the Rosewood here in London like you would probably know the answer.
NikolaBut the hotel's history, not probably, definitely, but you can play with that and see it, right? You could ask it about the history of any famous landmark and it would know, right? So the question then becomes like if it's remembered something about, I don't know, the parking information that I saw on a website. That's not up to date, but the model doesn't know and you've given it like the universe of things that it knows about, but equally you've given it like completely, you've told it to not supply answers to questions that it doesn't know the answer to, but it might not just listen to you.
NikolaIt's a pretty like uncontrollable, wild. Energy that it has, and it might do what you expect. It might do something else. So it's just something that we need to well, then you need to keep in mind when you design things with generative models. Yeah. Yeah.
MichelleAnd then just, I guess on that exact note, how far off are we from seeing companies, especially enterprise companies with safety concerns and, like brands to protect and all of that, being able to incorporate generative into their workflows in a trustworthy way. Or what considerations would you give companies thinking about it? Yeah.
NikolaI think that you see a lot of companies now, like investing a lot in AI into using LLMs. There are different ways to use it. And one is to just use it as an assistive tool but you have the human there as a necessary filter. But when you look at it autonomously, say in the context of voice assistance or chatbots it's up to the risk appetite. You can build very engaging, interesting prototypes very quickly. But equally, like putting something out there that you'll have absolute faith and confidence in is at the moment a bit difficult, right?
NikolaThere are other considerations again, engineering related about latency, quality of voice all sorts of things, right? The safety around like hallucinations, potentially offensive language like people tricking the assistant into saying different things and then like calling your company out on it. A lot to think about and unpack and to use it like safe and responsibly within a broader kind of like a system framework. Is the thing that we spend a lot of time working on it has huge potential, but it's like the alternating current at at the very start.
NikolaAnd you can be an optimist about it and say, Hey, this has the potential to be transmitted over hundreds of kilometers. It's how we light up the world, which. That's how it happened, right? If you're on the Tesla side of the equation, or if you're Thomas Edison, you can go around New York electrocuting sheep like he did to prove that Tesla's wrong. And if you want to do that with LLMs, you absolutely can, right? You can go and show that it will generate output that you might not want to generate right now, but very quickly will build the right kind of control mechanisms around it.
NikolaAnd we'll just. Learn to use them in a more responsible way. And the more important thing, is these models are improving at an astonishing rate. If you look at the jump from a GPT 3 to 4 in terms of just like things that could go wrong, it has reduced drastically. So I think it's safe to assume they will continue to make progress on the foundational model side in equal increments or maybe even higher. And if that happens, then you can just assume that alternating current becomes a lot more safe, a lot more quickly.
MichelleYeah, love it. So I think you may have just invalidated my last question for you, which was around like how people who are listening to demos or they're interacting with bots that companies are saying are based on generative AI, how they are able to tell if it truly is if there are ways to test that, but just given, what you just said, And feel free to hop in here, but like about the astonishing rate of improvement. It might be possible that the things that I get, that get flagged for me to listen to. And it's just is this actually generative?
MichelleIt doesn't, I'm not hearing the latency or it sounds too good to be generative. Is it, that might be a question that gets harder and harder to answer. Even for folks like you who have been very close to the development.
NikolaYeah, I think that there are like very specific ways of looking at the output of a voice assistant and understanding whether it's contextualized things around the way you ask the question, right? Are you open on Tuesdays? Yes. Versus we are open. Monday to Friday a nine to five, right? That's most definitely not right. If the answer is a flat out, yes. You're probably dealing with a more sophisticated model. So I think, if you think about just the levels of autonomy, say in the autonomous driving or something like that, we can probably define a similar thing where.
NikolaThese, we, we can probably, we wish to think about it actually releasing just a set of different kind of like questions to ask around, like, how competent is this model? What can it do? And how does it deal with, yes or no kind of like questions? More complicated things or just questions that have references in them that are not that obvious. But yeah, like I guess the more important thing is just like, is the end product good? Are people getting stuff done? But yeah, generative will be a big part of advancing these overall capabilities.
MichelleExactly. I love it. So it's almost like a business. You've got Turing tests and other types of tests of machine intelligence that are maybe. Not exactly like we'd test human intelligence. Like maybe that world is developing alongside, LLMs and generative and all of that stuff. But then from a business perspective, you've got another sort of test of the quality of a generative bot. And that is, does it help you? Is it profitable for you?
NikolaYeah. I think like for enterprises a lot of people are investing a lot into, often these days we'll hear like about a midsize financial institution developing their own LLM and frankly. That's a pretty big investment in a world that's changing so fast that, the jury is still out on whether they will actually have to fine tune models for their own use cases. I think there's a large impetus by cloud providers for them to do that because it will be an expensive operation that they can spend a lot of money on cloud compute doing.
NikolaAnd, they'll get better models. Do they need those models? Do we absolutely have to do it that way? Do they need faster yet smaller models? That's a big trend, which it's definitely a common sense logic and, if you built like a fine tune, smaller model will run faster and cheaper. Absolutely. Yes, it will. The question there goes back into the build versus buy. And at what level are you operating? Do you really need to do that? My intuition is that as we get towards better and better general models, and from all of the experimentation we've done it's increasingly, and I'll, not just react, people write about this very openly.
NikolaThere's a lot that you can do, for example, with untuned GPT 4, where previously you would have had to tune a smaller LLM to get to some level of performance, I think if we can get these extremely large, more advanced LLMs to do all the work, then We might not even have to have private instances and stuff, but there are other sets of enterprise considerations around like data governance and, do you want your data accessed by open AI, Microsoft or Google, or do you want a private cloud? I think there are many buyers with different kind of considerations.
NikolaAnd some will have to develop those capabilities themselves. Some won't. I think regulated versus not regulated will be. Will be very important,
MichelleI think you just teased our episode topic for next week, which is a lot around the way that enterprises think about generative, the way that we think about building and incorporating generative into our stacks and to our customer experience. But yeah, I really appreciate the conversation today. Looking forward to the next one. Thanks, Nicola.
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...