Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Deep Learning with PolyAI

Deep Learning with PolyAI artwork

061: An early look at OpenAI's Operator

Jan 24, 202520 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Kylie takes the helm for this episode of Deep Learning with Poly AI, where she and PolyAI CEO Nikola Mrkšić chat(no GPT) about OpenAI's latest innovation: the Operator agent. From the concept of AI performing tasks such as ordering groceries, booking travel, and making restaurant reservations, to live demos of agentic requests, Kylie and Nikola butt heads on overall impact and explore the broader implications of AI in increasing productivity, interface interaction, and transforming enterprise applications. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

KylieYou've spent your career thinking about this kind of AI, me, less and I look at this and I'm a bit like. Why do I need AI to do something on my screen that I could just be doing myself? And it's a bit slower, right? hello and welcome to another episode of deep learning with Poly AI. I'm your host, Kylie Whitehead, and I'm joined today by Poly AI CEO, Nicola Merkzich and today, Nicola, I want to talk about OpenAI, who's just released their first AI agent operator.

KylieAn operator performs tasks for users on the web, right? So some of the examples that they've given are ordering groceries or booking travel or ordering takeout, online shopping, all these kinds of things. So the first question I want to ask is like, what is your gut reaction to this product release?

NikolaI can't wait. And I've been playing with it. I love it. I think that like coupling, this is the moment that. Like, and the thing that's always been the most exciting to me, cause like, if you, when you think about like an autonomous AI, like whatever it decides to do, once it's reached, like superhuman AGI this is the moment where it like breaks free and goes and like interacts with the internet, the world, et cetera. And the internet is the first. Instance of our world that it can interact with programmatically. So it should be doing it liberally. Right. And it already is in all sorts of ways to LLM applications.

NikolaBut I think that kind of like to visualize and understand the future and explain it to a layman, someone who's not like in the weeds of it, this is it. Right. Like it will do it. It will show people how it's doing it. It's a great way to make chat GPT a lot more useful to people. And just to break new ground. And I think that also to overcome a lot of the like stale stuff where you might not be able to get an API call in, but you can go and do it online the way a human would.

KylieYeah. You've been in this field for a long time, right? You've spent your career thinking about this kind of AI, me, less and I look at this and I'm a bit like. Why do I need AI to do something on my screen that I could just be doing myself? And it's a bit slower, right? Like I watched that demo and I was like, this seems like a really cool technical, Yeah, but like, do I actually want it? I dunno.

NikolaYeah, look, I think like in the interest of just explaining to the audience here, I've I've got it open. They've deliberately excluded Europe, I think. I don't know why the UK got hit. We're a pariah state that has not implemented the EU AI act. So Sam, please get your act together, but I'm not lazy. The VPN is all I need. So if we tell it to like book the best Georgian restaurant in NYC for tomorrow night, and we leave it to chug along.

NikolaRight. Like, I think it's really good. I think, like getting a whole representation of what the system is doing, right. How it's doing it and how it goes about doing it. It'll take time. It's a bit slow.

KylieIn the demo they showed in the release the person who's using it is using, it's like use OpenTable to book me a restaurant. Are you just going straight through that and telling it?

NikolaBook the best Georgian restaurant in New York for tomorrow night. It is currently, as you can see on a website called. Eater. com. It's looking at a classical Georgian dish called Khajapuri. I recommend it to everyone. It's called a messy egg infused pizza like thing. Still looking at dishes and reading the articles, which is awesome. And it's looking at Chama Mama at the moment, which actually is located right in front of our New York office. Can't claim to know every restaurant in New York, but this is uncanny. We'll see if it picks that one.

NikolaThere seemed to be a few anyways, anyways, we'll let it chug along and see where it gets stuck. If it gets stuck, it's a bit slow, but this is exactly what we mean. When we say agentic, please go away and do something on my behalf. Right. And think that's a real instance of like multiplying productivity, et cetera. And will your life be profoundly shaken because AI can not book a restaurant for you? Hell no. Right. We had the first like, AI to AI interaction with Google duplex back in 2020. Right. When their duplex system used to update Google maps, opening hours spoke to our system and both were hilariously incompetent.

NikolaThey got it done. Roughly. We have a great blog post. I think it got like 10 million hits. But we're not looking at this reservation. Here, right. It's found a Georgian restaurant called Oda house. Okay. I do not know how to house. This is something in Brooklyn. It can book a table for tomorrow night at their Brooklyn location. Default reservations for two people at seven. Okay. So how about something closer? Let's really prove the value closer.

KylieYeah, I would, yeah, Georgian restaurant. You'd have probably found something you wanted by now.

NikolaRight. The claim here is not that you will do it faster. The claim is that you don't have to do it yourself. Right. And let's really put it to the test, but I think that like this mission of grappling with RPA to do it. And like, when we talk about multimodality, sure, voice or text is one thing, but the ability to interact with an online interface is a whole new, different form of multimodality. Right. And it's currently on the PolyEye website. It is there. It is looking for our address. Right. And here you have to start thinking like, would you trust just anyone to do it?

NikolaLike, yeah, there's like you in this, right. If it can apply it well, right. Now I'm hell bent on getting it to pick that Georgian restaurant. That's across from our office. I've not been to it. I don't know if it's good. Maybe it did me a favor with picking the other one. But

Kylieno, that's like, at what point does this become? Life changing because like, okay, this is pretty cool. I'm more skeptical. You're more excited, but ultimately we both agree that this isn't going to change our lives. At what point does it change our lives or does it, are we expecting? I

Nikolathink that we've already taken a pill that changes our lives forever by having LLMs reach the point where they are. The rest are like implementation details, right? So does this particular moment change how I feel about, mid long term future with advanced. And AGI no, it doesn't, but I'm excited that they've decided to tackle the song because I've spent a lot of time back when we even looked at buying contact center companies. Right. I've spent a lot of time looking at how we would do RPA for clients who do not have APIs into their backends.

NikolaRight. So very practically making this work well is how you would make the dream of what's a UI path, blue prism, the RPA companies were. Where if you could say to a client on our behalf, I don't need your APIs. I don't need anything I'm going to be your agent Like I'm going to do the same thing as your agent that reduces our addressable market from those companies that have undergone a lot of digital transformation to just like, Hey, let me control your screen that comes with a different set of info set constraints and all sorts of trepidation at the moment.

NikolaThe reason you couldn't put this in. Okay, I'm running back trauma. Mama highly rated Georgian restaurant is near the poly office. Yes, please book. It is two people. So it's slow right on the phone here. The main issue is not like it's circumventing APIs, right? It's doing it online and that. I think it happens to have a website. It would be interesting to see it call. I don't think that's created yet, but that journey will come. No doubt. Right. Google duplex has been doing that for some time now. So we know it's doable. It's just about chaining it together.

NikolaThe real issue here, it's a bit slow, so it's not saving me time. And if it was a contact center agent or like a virtual EA doing this for me, it wouldn't be good because it's very slow, but it knew enough about me or just contacted me asynchronously. It would already be capable of doing it and it will do what is just depends on the use case, but it's time to start working because by working on it, like we're going to get to a point where it does get faster. And once it does, eventually it'll be faster than a human. Cause when it opens the poly I website and looks for the address, it finds it faster than you or I could.

KylieYeah.

NikolaSo yeah, superhuman capabilities, AGI, this is exactly what it is.

KylieI'm really interested in this concept of circumventing APIs, right? Because, so we recently did some research, we ran a survey to a bunch of CX and contact center leaders, right? And about, I think it's about 50%, don't quote me on that, look at the research when it comes out, but I think about 50 percent of them said that they're, contact center tech stack is not really in fit state to implement AI. And I think some of that is maybe a little bit of just a negative attitude and like a misunderstanding of what's possible with what, but I think you have to rely on your it team to build out these APIs to connect a, AI agents to be able to do things.

KylieAnd so does this kind of technology that operators using. Will that apply to contact centers in the future so that they can, move around those APIs and do more?

NikolaAbsolutely. I think that a lot of the RPA wins, implementations of UiPath and other RPA companies have exactly done that, right? Like it may be a backend process. And like we, what we do is a lot of like front end people talking to you. And then. I was just changing a flight and had to get back from Seattle. Right. So they said, we'll be in touch in 48 hours because our systems are not live. We can't wait with you. And they got in touch like a day later. Right. So that kind of stuff in the backend probably requires some, someone looking somewhere, going to someone else, et cetera.

NikolaIf you join it up so that something's not no longer like 77 clicks. By the script that just runs programmatically. That's a lot of what say UiPath did to automate that API. And now you can treat it like it's an API, like it's a, a web request. You can send and get something back, but it's not, it's actually doing it the way that a human would, because it may be simpler and faster and easier to implement it and follow that than to build like the underlying mechanics of the whole thing. So yeah, I think that. Right now we're stuck here and it's given me full control of the thing.

NikolaAnd it's asking me to log into the website of this of this thing. Like through that, it's accessing a widget to what looks like open table it's got some ways to go, right. It's not solved. And the reason I'm excited is exactly because, like opening, I. Publicly directing humanity towards it means that in three years from now, there will be 200 Y condominator companies. Working on the problem and bit by bit, they'll trip away at it and we'll get it done. Right. And working on things, technology, there's many ways to do many different things.

NikolaAnd they'll have their specializations and they'll get to something really good, I've been longing for it because the exact reason we didn't go and buy a contact center company and then, proceed with automating the voice assistant part, which is something we can do very well is that digital transformation in the backend is, was too much. It still is too much. Right. But. Choosing to make this one of the problems to work on. I think that's really good.

KylieYeah, that's exciting. I guess chat GPT kind of took the front end, which is the way that poly AI is, you know, approaching a lot of these problems and then operator is taking the backend and that will change. How we use this kind of technology forever.

NikolaYeah. And look, it's just explaining to people what a gen tech means in a very visual way, which I think is really good. And, like with initial LLMs, people will now come up with a gazillion innovative ways to do it. And, even if we do a fast follow and just read Reddit for two days, you'll find people thinking of different ways to do it. I think it's really exciting.

KylieSo we can't see the interface. You're doing this on your phone as we've been talking, we can't see the interface, but I believe that operator. Tells you what it's doing at each stage, right? So it opens up Like the old one model,

Nikolait has a high level of like what it's doing right now. So at the moment, if you can see, I have control over the whole thing, which is a bit boring. But yeah, at each point it goes and just tells you what it's doing at a moment. Right. I think like, it's it's crazy that it's not available in Europe. Okay. I woke up to a meme from a friend who probably wouldn't mind me sharing the story, but it's two Godzilla like creatures fighting for the AI, for AI supremacy being US and China. And a kid playing with the thing where it fits like triangles or circles into the right kind of like, slots in the machine and labeled as Europe.

NikolaI think it's hilarious. We'll see where it lands, but they should make it available. I don't know how many people. In Europe, paid 200 bucks a month, they're missing out on everything.

KylieYeah, that's true. Maybe there, there isn't like a huge reason to go through all of those hoops in the AI regulation. Right. If users aren't using it, I, yeah, I think it would be really interesting in enterprise applications as well, having that level of visibility, right. Being able to understand, like in an enterprise setting. You, you need to know why every single decision is being made. And when you have people making those decisions you trust them, but if AI is making those decisions and you can't see how that is, I think where the Yeah, this

Nikolais exactly where this gets interesting, right? Cause it has a drill. So the old one model did reasoning and it didn't output the full. Steps towards it for a myriad of reasons, but for example, I just asked this thing to tell me more about what's happening with Serbian student protests where today there's a big rally in Belgrade. Right. And at the moment it is going through MSN and like reading articles about it and trying to figure things out. We'll see where it gets to by the end of the podcast, but I think, one gripe that we have with AI being.

NikolaEverything that it should be right now is that you ask and you get that answer. I only have information where a cutoff date of X, right? What you want to do is real time, right? Tell me different media writing. And then you have things like either ground news, which tell you whether it's like a right wing, left wing source of information, how biased or non biased. I would love to get like a summary that tells me, Hey, I've read like right wing media, left wing media. I've gotten a summary where, the truth looks like something like this. And here's my workflow. If you want to audit it. Right.

KylieYes. Yes, I love that being able to understand because still when when I'm Googling stuff, I don't really know if I don't trust, if I don't know the source and have an opinion on how that source views the world, I feel weird about the information on it.

NikolaOh yeah. That's like all the rage. Right. What I got back here is a pretty neutral statement from an article from Vulcan Insight from a few days ago. So it's still a lot better than like. Previous chad GPT models that would just not access more recent information. yeah, from what I'm seeing right now here, it's not really blowing me away, but it'll get there, right? You gotta take the first step. So I think that's really exciting.

KylieWhat do you think the general public is going to feel about it? Because I feel like seeing something. Moving the mouse and doing all this stuff in front of me, it feels like something out of a spy film and it's a bit scary.

NikolaYeah, I think at this point I asked what's happening today. It's forgotten the context and telling me that it's national girl child day in India, celebrated in India to honor girls. Like there's a ways to go. So I think what you'll get. From a lot of people, it's probably mockery, right? And that's, I think, one one of my favorite kind of like, people to test anything we do with is my wife who, as we worked on a lot of these like restaurant examples early on, I would give her a system and be like, Hey, like this talks about restaurants.

NikolaRight. And while everyone found things that no one else could do, she would do a very simple thing. Like, Hey, what kind of food do you want? Not Indian. And you would just get recommendations for Indian food, right? Because the negation and old school Engram based NLP was a pretty hard thing to do. And if you didn't explicitly build for it, it just didn't work. So I think, here, it would be very easy for us to make a video and be like, ha, look, it doesn't work. Of course it doesn't work because it's really hard to do something like this, but it got really far. Right. And.

NikolaWhat this means is we got to start collecting data. We got to start collecting examples. You can I'd bet you a lot of money that You know, the large annotation companies like scale AI currently have thousands, if not tens of thousands of people working on behalf of open AI and doing these tasks so that they can record the right kind of information that the next version of the model can be trained on. Right. But what we're doing now as consumers, paying them 200 bucks a month is providing that data as well. I'm sure it's getting stuck, but it's telling them where to look.

NikolaAnd if enough people get stuck looking for. A restaurant near the office then you can bet that there's 10, 000 people annotators getting similar tasks and going through it themselves. And I think then, over time we'll get there. The general public I dunno, I think your viewpoint of kind of like, mild intrusion, slight discomfort, we have maybe slightly different worldviews, although you are a very long tenured head of content for a leading AI company. So I'm not sure you'd qualify in the eyes of Luddites when they come for us as someone who's a long time.

KylieI'm much more open probably than. Many people but I said, do you

Nikolaremember when we did like initial guerrilla testing of our systems? Right? I distinctly remember a woman in central London telling me that she doesn't want to feed AI. I was like, Hey, we're testing them. And most people were like, ha, let me see. And, worked or not, people were like, Oh, it was good. Or it sounds like a like terminator, whatever. This one was adamant. I'm not helping you bring about our demise. Right. And hell, she may be right. I don't know, right. Maybe we are completely unwitting participants and all that. I don't think so.

NikolaBut yeah.

KylieYeah. I don't think at that point intent based system helping you book a restaurant table was too life threatening for anybody,

Nikolabut I think our future generative overlords may look at us as fond builders of the antiquated version of its ancestor, the Neanderthal version of our current agentic LLM powered applications, right?

KylieYeah. Okay. Depending on what the future looks like we'll either take that as a compliment or we'll deny it. Well, I, yeah, I think what I've got out of this conversation is that for me, I saw this release and I thought, yeah, whatever, who needs this? I'm, I'm sneering. You know what I'm like anyway, that's my personality. I'm always a bit suspicious. Exactly. But I guess what you said that, that has. I think it's the real takeaway of this episode is like, this is a new way of doing things. This is going to open so many doors for the way that we are able to think about conversational systems and agentic systems in particular.

KylieAnd yeah, at the moment, maybe most of us don't need to see a cursor moving around on the screen on its own to book us a restaurant when it takes much longer than it would do to do ourselves, but what could that look like in the future? It could look like. A lot of things that really do create value. And I think that I admire your excitement about it. Because I think, yeah, there's probably a lot more that can be done.

NikolaNo, there will be a lot. Right. And if you just like, look at Reddit right now, I'm just looking at, people look at like different, like real world tasks, then it's interesting, right? Like we can probably do a fast follow up and see, um, what happens. I think the main comment, the main negative commentary right now is just about it being slow. That's, that will be solved quickly. Right.

KylieYeah. I

Nikolamean, if we look at this target announcement and stuff, like, Hey, go build those data centers, please. I think around the world, nations are mobilizing for unbelievable resources in this. So stay put, give it a year or two. We'll get there fast.

KylieLet's catch up again in, in a few months and see where this has gone.

NikolaSounds like a plan.

KylieCool. Thanks so much for joining me, Nicola. And thanks everyone for listening. We'll see you next time.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. What if your contact center became a product team?

    Nov 17, 202635 minAudio

    Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...

    Credit the people on this episode

  2. What happens if your AI agents get 1% better every day?

    Sep 10, 202639 minAudio

    An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...

    Credit the people on this episode

  3. Who's coordinating your army of AI agents?

    Sep 3, 202644 minAudio

    Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...

    Credit the people on this episode

  4. Why should CX leaders care about MCP?

    Aug 20, 202636 minAudio

    When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...

    Credit the people on this episode

  5. Can AI really hear a call the way a person does?

    Aug 6, 202622 minAudio

    PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...

    Credit the people on this episode