50th episode! Damien welocmes newcomer to the pod Oliver Shoulson, PolyAI's senior dialogue designer, about what dialogue design entails and why it's crucial in the era of advanced language models (LLMs). Oliver explains that dialogue design goes beyond just scripting responses; it's about making AI interactions feel natural to humans through understanding subtleties like turn-taking, back channeling, and processing human language cues. He notes that good dialogue designers often come from backgrounds in linguistics, human-computer interaction, and scriptwriting, and they blend skills from product design, software engineering, and data analysis. Despite the advancements in LLMs, Oliver insists that these technologies still need careful structuring to avoid reverting to generic, less human-like interactions. An interesting topic everyone can appreciate! Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
OliverTypically, one of the things that we're trying to get voice actors to do is to not override their impulse to have ums and ahs, not override their impulse to elongate particular syllables. When they're processing or thinking that sort of elongation that is just another one of these cues that I'm not done talking that I'm still processing that I'm going to say something else.
DamienHello and welcome back to another riveting episode of deep learning with PolyAI. I'm your host, Damien. And I am here with a brand new guest. His name is Oliver Shoulson, and he is the senior dialogue designer here at PolyAI. Oliver, welcome to the pod.
OliverThank you so much for having me. Super happy to be here.
DamienOf course. It's exhilarating. Oliver was one of the first voices that I heard from PolyAI after I joined. A little over a year and a half ago. And I'm so excited to have you on the show because one, you are just absolutely persona plus, and to your field. Dialogue design is such a hot topic right now from across industry basis. It's really in demand. Good dialogue designers are harder and harder to find. And the question is, what the heck is dialogue design and why is it on the tip of everybody's tongues these days?
DamienGive us a little brief about what you do.
OliverSure. So I guess the one sentence summary of dialogue design, or you'll sometimes see it called conversation design across the industry is we're the owners of the user experience for a conversational dialogue system. And particularly in spoken dialogue systems, like the ones we do here at poly AI, there are so many layers to the user experience and so many places things can go wrong. We'd have to start with. Turning the user's voice into text and then processing that text to understand it in some way, and then use some kind of internal logic to decide what to do next and then produce an utterance from the system to respond to what the user said.
OliverAnd any of those layers. There are things that can go well things that can go wrong. And things that can make for a great user experience or a super friction full and not frustrating user experience. And so dialogue designers have a role to play basically at every single 1 of those layers. But overall, I would, I like to say that we're. A third product designer, third software engineer, and a third data analyst. So we play a role in the sort of high level design of the dialogue system, the implementation and the building of it, and then analyzing and figuring out insights afterwards to find areas for improvement.
OliverLook at trends and metrics and come up with ways to make the experience better.
DamienInteresting data analysis, product design, dialogue, design, obviously this, these are some pretty varied skillsets. So what kind of backgrounds do you tend to see with really strong dialogue designers? And maybe. What are some consistent characteristics or traits that you find no matter the background that makes for a really effective dialogue designer? You already alluded to it, right? Having those three functionalities are these people coming from backgrounds in product design? Are they coming from backgrounds in our application?
DamienIt makes a lot of sense, right? But what about in the broader industry? What kind of backgrounds do the folks who make really good dialogue designers come from?
OliverYeah. So actually every dialogue designer at poly AI has an academic background in linguistics, which is crazy because it's not a huge field academically. There aren't, there aren't a ton of linguists out there. We have
Damienall of you worldwide. We have every linguist now works for poly AI.
OliverYeah. And I actually came to the field of conversation design. through a like career bootcamp that was run through the Linguistic Society of America. So I came to it very much through my linguistics academic community. Oh, that's cool. And linguists are, people who study human language per se. And I think one of the things that makes. Linguistics, such a powerful background to have in this field is that it merges this kind of deeply intuitive design oriented aspect of the job where you're being asked to reflect deeply on yourself and your own experiences, interacting with these kinds of systems and what makes it feel natural.
OliverWhat would I do if I were in the shoes of the customer service representative that I'm trying to embody when I design a voice agent. But then also on the other side. We're trained in this very programmatic kind of style of thinking that is based around rules and constraints that really lends itself toward operating within computational systems in general. Linguists love rules and constraints. I, that's just like where we live. I know you wear that
Damient shirt all the time. It just says linguists love rules and constraints. It's like, get a new shirt, Oliver.
OliverExactly. That's that's just where we live. And basically, your job as a linguist is to think about this thing, language, that is so intuitive and so integral to the human experience and something that is built into our brains and that no one thinks about, but does every day and figure out what are the rules and constraints that model and, And systematize this process. And so that's basically exactly what you're doing when you are creating a computational dialogue system or an AI dialogue system is you are teaching a computer that doesn't have any of the kind of cognitive biases or architecture that we have, how to do this thing that we do so intuitively every day.
DamienBut, and not only that, but you have to make it so it operates as autonomously or not autonomously, like, well, yeah, I guess a little bit and effectively. So that it doesn't register as something that's non intuitive to the corresponding party, which seems to be a task unto itself. Not only to train technology, to do this, to train a bot, to do this, but to do it and have it be received as something that's just as intuitive as it is for human participants.
OliverThat's exactly right. And I think you asked why conversation designers are in such high demand right now. And I think the reason for that is that because everyone has such deep Built in intuitions about language. It's very easy to tell when something is not good When an interaction doesn't feel natural or doesn't sure human and so I think everyone has this sense We've had this sense for as long as there have been Like the notion of a robot that it would be cool to interact with robots in natural language look at star wars talking to c 3po or whatever, right?
OliverThese are two is a
Damiendifferent story, we'll get to that. Well, they talked to, they
Olivertalked to him enough in natural language, but I guess that's true. I say him, I don't know the gender. Yeah. I think everyone has always felt like, wouldn't it be cool if we could talk to computers, talk to these automatic systems in natural language but what you realize really quickly as you start to try and make that happen, implement that is Think weird things happen and you have these interactions that feel so unnatural. So uncanny. But it's actually really hard to put your finger on what the problem is and to affirmatively design a solution to work around that issue.
OliverAnd so I think that's where the linguistics expertise really comes into play. There are also people who have, you can't
Damiendefine bad art, but you know it when you see it. And it's the same thing with bad dialogue design, right? It's like, if something sounds off, you're like, Oh, but I can't put my finger on it. And even if you could. How do I correct for it? How do I code it out? That's a totally different story. That's really interesting. All right. So we obviously have at poly we have a lot of linguists that come into play, but what if you're not an academic linguist? Like what are some other folks like our authors, our traditional creative writers, our arts or screenwriters any, anything that involves high amount of dialogue that.
DamienJust forces you to commit to pen and paper almost what we would do intuitively and naturally in the course of a day to day who are some, what are some other skillsets that I think are universal among really solid dialogue designers?
OliverYeah. I think, copywriting or script writing or whatever is super huge. I think that the ability to. Be able to articulate in the abstract, what makes an utterance align natural is different from being able to actually, as you said, commit, put pen to paper, produce something that is going to sound natural and human to someone.
DamienYeah.
OliverThere are people who have degrees in human computer interaction. That is like a field that is. specifically focused on this question. And typically those people have experience both in the natural language side, the sort of spoken or written dialogue design, but also in visual design, graphical user interfaces, like understanding how people interact intuitively with computers. So those are some common backgrounds that we see.
DamienWell, what's interesting about those sort of interfaces is that we always talk about how anybody can theoretically build, like a chatbot these days built on a strongly effective and widely available LLM. Right.
OliverYeah.
DamienBut you lose a lot of the nuances that makes voice such an interesting. Communication medium. And so perhaps like the same skill sets that would apply to somebody who has that machine human interaction through various channels, probably a little bit easier than voice because voice has certain layers of complexity that come into play that would trigger that uncanny Valley sensor a lot easier than something over chat. Or even as well over visual medium.
DamienVisual is probably on par with audio, but that's really interesting. So speaking of LLMs and the availability of them, it seems like right now, some of the key players who are establishing some foundational generative models. Are looking at voice and saying, this is important. Obviously we want to do this and they're releasing, at least, in, in marketing materials, some really like high quality synthesized voices that seem to embrace fairly realistic ability to engage in dialogue.
DamienWhy can't like, let's say businesses out there that are interested in folding this into their, in an enterprise tech stack, why can't they just go out and grab an LLM and say, okay, let's make it happen. Is this something that theoretically they could do? Why would dialogue design be necessary in addition to that?
OliverSo I think you foreshadowed what I would say to this before, when you talked about how like text versus voice. I think that the fact that so many of these LLMs are almost entirely trained on text data. And specifically like mostly long form text data, like they're getting data from books and from articles and from things like that makes them very good at producing text of that kind, which differs in some really fundamental ways from. Conversational dialogue.
OliverAnd so one of the things that you'll notice if you take one of these LLMs and you particularly try to get it to perform a complex multi turn task or transaction is that LLMs love to combine steps and overwhelm human processing abilities. They'll ask you to do Let's Way more things at once than you can possibly remember to do. And that's because LLMs are fundamentally not constrained by human language processing capabilities or human cognitive processing capabilities. And so they don't have any of these intuitions about like, what's a logical or rational way to guide someone through this complex task that we're trying to perform together.
OliverAnd so, and so an LLM, they just,
Damienthey just like data dump everything all at once at you. This is a very common thing. Yeah. So you'll say,
OliverI, We're trying to do a hotel reservation, so we need to collect the date they're arriving, the date they're departing, any room preferences they have, the size of their party their name, payment information, and whatever. And as many times as you tell an LLM, take your time, do this step by step. It has a really hard time following that instruction. And I think this is a bias that's built into a lot of these LLMs. Based around the kind of data that it was trained on. But also based around the fact that fundamentally, I think neural networks and the way these LLMs are built differ from the way that humans think.
OliverAnd so as much as dialogue design is about producing individual interactions or individual turns in a conversation that feel natural, it's also about structuring the entire conversation in a way that works for the human brain and makes us and makes sense to us and feels Systematic in a way that accords with human thinking.
DamienWell, that makes a lot of sense. And to your point, these LLMs are typically developed, not so much with a voice application in mind, although it is an easy or not easy, it is a plausible offshoot, but fundamentally it's a text to text interface. Right. It's looking to give you as much information as possible. You go to chat GPT and you say, Hey, build me an outline for hosting a podcast with the Oliver Schultz and here's his LinkedIn bio. And then, it'll give you a comprehensive framework, but that's not the way that we absorb information when, when being like audibly guided or whatever, and so that's a really interesting concept is taking into consideration how the brain processes like spoken information.
DamienI dig that a lot. What are some of the other like little nuanced facets of human language that nobody really thinks about? We touched on intuition early on, and I want to go back to that a little bit. What are some of those indicators and components of human language and speech that nobody really considers that dialogue designers have to keep front in mind or front of mind? When developing these complex systems, especially a top, neural networks that process information and deliver information a little bit differently.
OliverI think the first one that comes to mind has to do with turn taking and the way that we as humans cue each other in spoken conversations that either our turn speaking is not over or it is over and it's time for you to join in. There are also other things. I'm still
Damienterrible at that. That's why I interrupt everybody on this show.
OliverWell illustrated. There, there are little, just very subtle cues that we give both lexical cues, so things like in the actual words that we say yeah. Either to acknowledge to the person that we've understood what they said to try and interrupt and take over our turn from that, from the other person. Yeah, these little conformational markers that we give to people saying got it. Yes. Yeah, and, continuing into the next piece of the conversation these sort of transition words that make something into a turn taking routine as opposed to just what can feel like a very stilted one question, one answer, one question, one answer.
OliverAnother huge thing that we try We tackled a long time ago. And this is one of the first sort of user experience tasks that I really delved deeply into at Poly AI is this problem of back channeling. This is something that we've been doing throughout this entire conversation very intuitively, where if I'm speaking and you want to let me know that you're with me, you're listening to what I'm saying, you're not trying to take over the conversation, but but that you're with me, there you go, you'll give me these little things called back channels, which are things like, yep. Even there are visual back channels, like head nods, things like that.
OliverInteresting. And this is something that is so intuitive that people will actually have trouble if you don't give them a back channel at crucial points in an interaction. So that, so
Damienthat can be a key flag. If you're giving a tremendous amount of information to an automated system, and it doesn't, at some point acknowledge that it's still listening to that information that you're giving and still processing it, by giving that.
OliverMm hmm. Exactly.
DamienThen, then we that maybe sends off one of those uncanny valley triggers.
OliverThat's right. So a place that we saw this like all the time is when people are reciting their phone numbers. We have this sort of intuitive choreography that we do when I'm giving you my phone number where I will. Say the area code, then you'll give me an, then I'll say the next three digits and you'll give me a, yep. And then I'll say the next four, the last four digits. And you'll be like, all right, so I've got, and that's for
Damienus based phone numbers, obviously phone structure and the condensation of that phone structure in other countries is going to vary as well.
OliverExactly. So what we would see so often is people, we would say, could you give me your phone number? The system would say, could you give me your phone number? People would say, okay, it's 1, 2, 3, and then pause
Damiena bad system would say 1, 2, 3 is not a valid phone number.
OliverExactly. Exactly. Interesting. So being able to give them that back channel and be like, yep, I got that. I'm still listening. Keep going. Is a crucial and completely Ubiquitous part of human conversation that a computer does not intuitively know to do and that you have to build in very deliberate.
DamienAnd that's a complex building too, because sometimes they are done talking or sometimes they're not done and don't want to be interrupted. So the eponymous barge in, when is it appropriate? When isn't it, do you need some more information or do you just want to indicate through back channel that you are. Still processing that information or that request or that turn. Wow. That's all right. Anything else besides that back channel indicator that really comes to mind. How about some of the things I know I've listened to a few call samples that we've done and this is particular when it comes to non synthesized voice, but I am seeing it in more in more in.
DamienOpportunities with synthesized voice, things where certain phonemes are dragged out the and the, well, let me see about that. Or, and I think back to traditional IVRs, I think it's nuanced that, that introduced the typing in the background, the clackety clack. And it was like, Oh, that is about as inorganic as possible, but those sort of add ons are really nice. Is that something that you consider? Do you actually like. I hate to break it down to such a rudimentary process, but do you actually read a script out loud and see how you process?
DamienOh,
Oliveryeah. Oh, yeah, absolutely. That's a key part of making, I think, designing any natural dialogue is like putting it in your own mouth and seeing, like, Role playing with yourself or with your roommate or whatever, like how would this conversation actually go? It's a fun day in the
Damienoffice. You guys are just exchanging different ways to read a scene
OliverYeah, and I and you know when we work with actual voice actors, as opposed to Synthesized voices which we do both in different cases. This is the kind of direction that we give them is Typically, one of the things that we're trying to get voice actors to do is to not override their impulse to have ums and ahs, not override their impulse to elongate particular syllables. When they're processing or thinking that sort of elongation that you were talking about is just another one of these cues that I'm not done talking that I'm still processing that I'm going to say something else.
OliverAnd and you need to wait for me to, as opposed to if I just like, if I just said, it's like, what is that? But if I say, so I think that that's that sort of upward intonation, that extent that's telling you that I'm about to go into something else. I'm working on something. Hold, please. While I get back to you about whatever I'm about to say.
DamienYeah, because contextually, you could use. So in other circumstances that's That's so true. And you don't want to be like, that's so true. It could be off brand. So that's case of each individual word. Yeah. Or yeah that's really interesting. All right. So we got a couple of flags to keep in mind when it comes to, you want to be, so you want to be a dialogue designer here, the things you need to think of upfront, where is, what does the future of this industry look like, especially with the advent of such aggressive and functional language systems that exist in LLMs and other applications of artificial intelligence, like where, what is the future of dialogue design mean with these new tools and technologies in play?
OliverI think like on the one side, it's changed everything. And on the other side, it's changed. Nothing. what I mean by that is, on the implementation side, implementing a conversational AI system with an LLM is so different from implementing this more traditional Conversational systems where the AI was all on the understanding side and everything that was produced was prescripted or there was internal logic behind it. Where in the former in with LLMs, you are basically trying to constrain behavior and make sure that it only does the things that you tell it to do and doesn't do.
OliverWhereas in the, on the ladder in the more legacy style of conversational AI system, every new bit of functionality had to be explicitly designed and built in.
DamienRight.
OliverHowever, in both of those cases, again, we're coming back to the rules and constraints. The key task is to make sure that the behavior aligns with the rules and constraints of human interaction. So those intuitions that dialogue designers have that linguists have that human computer interaction people or writers or copywriters have are applicable in every case one one thing that you'll notice is that If you spend time interacting with an llm is that although you can tell it to Have a certain personality or a way of talking it will almost always particularly when it runs into a some kind of challenging problem revert back to its kind of baked in personality and it makes it really hard for companies to differentiate themselves on a branding from the branding side.
OliverAnd actually we still find ourselves, even when we're working with LLMs, having to give them a lot of very specific instructions for, here's how I want you to say this, because otherwise it's going to sound like. GPT or whatever LLM it happens to be. Interesting.
DamienSo in that migration from being able to tell something, what to do, to being able to tell something effectively, what not to do, which is what the era of LLMs is ushered us into that even with LLMs, there is still a ton of opportunity as the technology sits right now. To be able to tell it what to do, because often when it gets to certain situations, it will revert to that baseline templated information. And when you're buying an on brand solution that's facing customers, you need it to stay on brand throughout the entire interaction.
DamienSo
Oliveryeah,
Damienthat's a fascinating component. I didn't think, I didn't think that when it came to like pros and cons of LLMs versus like structured dialogue design.
OliverYeah, it will revert either, it will revert in that stylistic way back to that baked in personality, or it will revert back to its sort of long form text based biases that we were talking about before. Again, there's like a, there's a huge role that dialogue designers still play in structuring a conversation, even when you're working with an LLM. Particularly when you're doing these kind of complex multi turn tasks,
Damiena little bit of language police and that you're trying to avoid unintentional prompt injection injections. I think that's, that's that's very wild. I I, it just harkens to the Twitter X, whatever it is. And threads meme now about trying to catch automated bots by having them ignore all previous instruction. It's just very funny. You don't want to get those. You don't want to get your voice spot back to back to square one, back to root personality. That would be disastrous. This has been really, really interesting. And I appreciate you taking some time out of your day to pop on the show and tell us a little bit about this utterly fascinating field.
DamienYou're one of the best. It's always a pleasure to chat with you. And thanks for giving us a little bit of a deeper glimpse into the world of dialogue design. I really appreciate it.
OliverWell, thank you so much for having me. This was really fun.
DamienOf course. And for the rest of you, we'll catch you next week for something probably not as interesting. I gotta be honest. Anyway, take care. And thanks again, Oliver. Have a good one.
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...