MIT reports 95% of enterprise AI pilots fail. Why? In this episode of AI Outcomes, Rickard Hansson (CEO of Weavy) returns to unpack the real reasons so many AI initiatives crash and burn - and what to do instead. We break down MIT’s State of AI in Business 2025 report and explore what’s going wrong in enterprise POCs: lack of integration, poor workflows, unclear outcomes, and vendors over-promising on “magic” AI agents. Rickard also shares how he built a community intelligence tool in under 14 hours using vibe coding and retrieval-augmented generation (RAG). It’s a live case study in getting AI right - from problem framing to workflow fit. Chapters: 00:00 Introduction 02:02 Building a Community Intelligence Tool 12:41 MIT Report Deep Dive 17:46 Why POCs Fail (and How to Fix Them) 26:24 Real-World Example: Sarah in HR 28:24 The Right Way to Use RAG 38:54 AI vs. Human Tasks 39:59 What Executives Want From AI 50:56 Final Thoughts If you’re tired of AI hype and want to hear what actually works - this episode is for you. Connect with Rickard: https://www.linkedin.com/in/rickardh/ #AIOutcomes #EnterpriseAI #RickardHansson #GenerativeAI #AIAgents #RAG #AIWorkflow #AIFails #MITAIReport #ProofOfConcept #DigitalTransformation #TechLeadership #AIPodcast #VibeCoding #BusinessAI
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
SPEAKER_01The legend that is and the myth, the legend that is Ricard Hanson and your founder and CEO, Weavy. I don't think there's anything around software development that you don't know. So um, so I was really keen to get you back on again just to talk about the realities that customers are facing who don't have JP Morgan's budget in actually executing and adopting AI. And there's an interesting report that came out a few weeks ago from MIT, which I've got some snippets of to share.
SPEAKER_01And in the last episode you appeared on, uh Ricard, you were live vibe coding, building AI, digital assets, just live, right? So anyone who's not seen episode 21, definitely go back and watch Ricard at work. Uh it's phenomenal. Um, so it's brilliant to have you back on today, Ricard. And first of all, how are you doing? How's everything going with you?
SPEAKER_02Good. And like it feels like a lifetime since the last episode since last we talked, but at the same time, it's been just leaping forward. Anything, you know, vibe coding in itself, I think, has been uh degrading, to be honest, because the platforms are like struggling and people are realizing it's great to do the MVPs and whatnot. And then when you actually want to move into production, it's a different story. And then I've been leaning in way more into personally in Agentic AI and how that can help our businesses.
SPEAKER_02And what we've done there is basically then actually combining vibe coding to create our own stuff at speed. Because vibe coding is great when you know what you're doing, it just like gives you like superpowers. But then on top of that, I've been playing around a lot with AI to then analyze data to our advantage to give us insights and speed on the other end. In this to for so we built our internal tool for community intelligence. So crawl, ingest, analyze, task, get active in the community intelligence.
SPEAKER_02So instead of spending$3,000 on a super expensive enterprise solution, we just decided to build ourselves.
SPEAKER_01And a couple of days later, we did. Wow. So what is that actually delivering? Just go, and I'm I'm gonna say again, go civilian audiences civilian, not developer. What are the outcomes? What's the problem that you're solving with that tool, Ricardo? This is yeah, this is great because this is like the reality of everyone.
SPEAKER_02So like I was sitting in my own world and like I was like, okay, what am I missing as a CEO and founder and owner of this company? What tool am I missing? It's like I need to understand what the community is talking about in general. And then, okay, that's one thing. And then I want to engage with that incredibly fast. So then I started to look around. So, what do we have? A lot of expensive tools. And I looked internally, it's like, oh, I have a lot of talent. Uh so I sat down with the guys, my my people, and said, like, we do not need a new version of Weeb.
SPEAKER_02No one cares, but we do need to understand what people are talking about. So what I need is you guys first to create, or like, okay, so what I want in the end is like I want to have like an agent that can just say, what are people talking about? And tell me what to where to engage and how to engage. That's like my end product. And then I said to them, to get to that point, I need to first extract what's going on. So then I tasked every engineer to like, you create a crawler for LinkedIn, you create a crawler for Twitter, you create a crawler for Reddit, and so on and so forth.
SPEAKER_02So each one started creating crawlers to extract information from the communities with specific keywords, like you know, vibe coding and so on and so forth. And then I created a middleware where I took all this information they're crawling and then ingesting this into something called a vector database. And why is that important? A vector database means you can do semantic searches. You can do not like absolute searches, you do like semantically, okay, I'm looking for this and that. And then you can find correlations between two different platforms, in my case, Replit and Lovable, maybe, and then on completely different uh sources, LinkedIn and Reddit.
SPEAKER_02And then that is already populated, like 30, 40,000 records. And then on top of that, then I vibe coded a community intelligence platform together with Cloud Code, where I now can ask this data, like, what do people love about lovable? Give me a deep research, uh, analyze patterns. And this is the thing people think is just easy, but it's not, because what you have to do there is like first, you have to actually come up with a search strategy. It's like, okay, so now I have 30,000 records, and the user is asking for what do people love about lovable?
SPEAKER_02You need to broaden that a little bit to find the scope, right? So then I ask Claude to come up with a search strategy, and then it will come up with 20 different queries, and then it will start querying the data. And then the end, it's gonna I ask it to, are you happy with this result? No. Okay, ask for some more. And then it generates a report. And then on top of that, I then have another agent say, now I want you to create tasks based on our value proposition. Our value proposition is this and this is this. Map this to everything you found and see if we can uh add value.
SPEAKER_02If you have and I had a scoring between zero and 12, if you score six or higher, create a task, tell me why, and also uh come up with a response to that sort of that post. And then it will generate maybe out of a hundred, maybe twenty-three tasks, and then I can review them and then ask them, add them as tasks, and then delegate them to my my people. So, and this was built end-to-end within roughly with 12 to 14 hours. And it's an enterprise grade kind of outcome in that sense.
SPEAKER_02So the the interesting thing here is like this is like a mix of vibe coding and agentic AI. I needed a way to engage with the community rapidly, understand what they were talking about, find the pain points, and then be helpful. Uh and obviously doing that manually in all these different sources with all these different posts happening every hour, every day, is not feasible. So I combined the best of two worlds: vibe coded a solution and identicay on top.
SPEAKER_01Whoa. Um so if I had the problem, which I guess I do in the sales kind of enablement world of okay, I want to know what's the pain point now. How do I help? How do I refine my services for the pain right now in the sales community? I could use the the AI tool that you've created to just just go on LinkedIn, find out what everyone's saying, go on X, find out wherever the platform is that's available for the tool to go and look at, where the community is, you can then go and see what they're saying in real time.
SPEAKER_01And is that the um the concept and therefore the outcome? Is that you then then the AI can say, well, look, this is what you should be doing slightly differently. The markets change for you. This is what people need in terms of problems that that they they have, that they need a resolution to.
SPEAKER_02Yeah, exactly. Uh in the end, it's like how you uh instruct your agent, right? What do you what is your purpose when you engage with this LinkedIn post, for example? And how I uh approach that is basically saying, okay, here's here's the post, and this is my value proposition. And like, you know, but also clear instructions, like always be helpful, never sell, you know, you always go in with like a like an informative knowledge sharing kind of approach. And then it just generates like amazing responses, and it's also resonating.
SPEAKER_02Like, you know, if if you listen to episode 21, what we do at Weavy, we're adding collaboration features to an application. And then obviously, I'm not just looking for people asking for that specifics, but so the agent of reasoning is like, oh, here's a person who built a dating app using Lovable. They do not explicitly ask for chat or something, but I would suggest that chat is a great feature for this uh dating app. So here's uh uh suggested response to this post saying, oh, great work for this dating app.
SPEAKER_02Have you discovered or have you contemplated adding chat to it, or do you have it already? So it's reasoning around it, create like a conversation around it, and it just automated the whole thing. Like I think it's like wonderful when I saw it like this. I'm incredibly proud of the creation I created.
SPEAKER_01Yeah, that's that's absolutely awesome. And the fact that that was done in like two days, yeah. Yeah, but uh and we'll come on to now. I think some of the challenges businesses are facing implementing AI, is you came at this from a clear, right? I had this vision, we didn't need Weavy2, Weavy's uh Ricard's business. We didn't, and you had a very clear vision on the outcome that you wanted and the problem that you want that you wanted to solve, and then you got building the technology.
SPEAKER_01So you might have built the technology in a couple of days, but actually the process that you went through beforehand with your team, if you hadn't got that right or hadn't done that and raced to just the technical solution, then you may not have been clear on what the problem statement was that you were solving for. And if I just go on to my share and anyone who's just listening, uh then I will I'll definitely just say all of this as well. So if you can't see the screen share, don't don't worry. So segue now to the state of AI in business 2025 report from MIT.
SPEAKER_01So this came out, I don't know, uh a few weeks ago, over the summer at some point, and a lot of people criticized the report for a couple of reasons, mainly proof of concept size. Um, so the actual number of customers, sorry, the the number of sample customers they they use, not proof of concept size. So they the sample size was only 52 organizations, but I still see value in knowing what 52 organizations have to say about the realities for them.
SPEAKER_01So I'm not gonna leap on and criticize, but but I guess all of these reports kind of need the need the big headline. And the big headline in MIT's report was that 95% of AI POCs fail, so proof of concepts uh are failing. Um so I think we should pay attention to the data, and just to give you another perspective of one of the AI kind of like service providers who criticized uh the report said that released their data, and they said, we've done 250 POCs, and this is what we learnt.
SPEAKER_0130% of the POCs were learning exercises anyway. So effectively they achieved their goal of just learning. Um, I'm like, gosh, it's a lot of work to learn, isn't it? To go through a whole POC. But any anyway, uh, but so but these are their numbers, not mine. 25% of the 250 POCs that this uh this AI provider, I I should cred them. I'll have to go and find a link. 25% of vendor evaluations to test to see if the tool fits before committing.
SPEAKER_01Okay, got it, seems sensible. 20% of the 250 POCs they've done work perf perfectly, but revealed that the problem wasn't worth solving. So it's like you going through all of that work and then going, actually, it wasn't that much of a problem in the first place. So a stunningly high number of uh uh in that category. And then the final category where 25% are actual failures, and they called they define that as a wrong approach, bad implementation, or solving for the wrong problem statement.
SPEAKER_01So headline, I'll I've got a few other key points from the report just to walk through and Rickard get your perspective on. But that what's your just initial perspective around some of the headlines there, around these proof of concepts just failing or not going anywhere?
SPEAKER_02I think it's you know the classic saying shit in shit out, right? Pardon my French, but some of that data was like revealing, like, you know, it didn't solve the problem or like we were focused on the wrong thing. Uh and there's a big part that is technology, and the other one is actually outcomes, right? In terms of what AI can do for you and whatnot. So the PVC are approving, okay, AI can do this, what we told it to do, it's doing, but we don't like the outcome. I mean, that's two different things. So I think like this is the biggest challenge right now for enterprises, is uh there's a lot of vendors out there, crew AI and whatnot, who claims like you can create these different agentic AIs and agents internally for the enterprises and can do all this magical stuff for you.
SPEAKER_02But it's very, very, I think, still abstract in that sense. Like, what is what does that mean? And it all comes down to like you you allegedly put it yourself, it's like you know, the problem statement. Okay, what am I trying to solve? Right now, everyone it's like the holy grail of AI is going to solve everything, right? And that's a pretty bad problem statement, everything. It's like, okay, like I did, like it was a very niche specific task, actually. It's you know, very uh cumbersome and very time-consuming task to do manually to go through all those different communities, right?
SPEAKER_02To find the right posts and so on and so forth, and then engage with those posts. It's incredibly time consuming. So, what I wanted to my problem statement was speed, right? I wanted to instead of spend weeks of doing something, I want to spend an hour. The innovation and what I created was pretty low, value high, innovation low, right? So I essentially, you know, hired a bunch of teenagers in a basement going through it for me by doing through AI, right?
SPEAKER_02So I think these numbers are pretty accurate. Like it's very and like you know, the AI is still Chat GPT helping you to write an email or figure out how to respond to the teacher conference, you know, if you're using it privately or personally, right? Uh, and you just think you can transform this into the business. And you do that still, but it's just emails basically, where it's like we are creating values. What I do, for example, in in this my use case is like I I I harness information from community in an enterprise setting.
SPEAKER_02You might you want to maybe harness information from a bunch of different business systems you have internally. And then what do you want to what do you want to do with that data? I wanna do you wanna have a reactive uh system that will ping you saying something is really off? Or do you want to be just having a your own chapity where you can ask the data stuff? And then you have to obviously you know deal down into what that is. And also the gentic AI is like, you know, there is this, you know, the autonomous piece of it, which I think is pretty scary.
SPEAKER_02Uh as like you just tell an idiot to do something and then we'll produce something, and then you're gonna trust that. No, maybe not. I still gonna have humans involved in the loop, and that's why I think still like uh any enterprise should focus on scrap the agentic AI. The agentic AI can be like collecting data, but still have a chat GPT UI in the end where you can ask it because then you can see is it providing what I expected? An expectation is one thing, and then value is another thing, and then play around with it before you turn it truly agentic.
SPEAKER_02But people rush in to make it agentic and think it's just going to be magically working in the background.
SPEAKER_01Yes, and that I think that's compounded by the AI agent companies. And I was just on GPT-5 asking it, give me approximately how many AI agents are being marketed to enterprise businesses today. And it's it can't answer the question exactly, but it came up with probably about 5,000, and it's gonna increase to probably around 10,000 by this time next year.
SPEAKER_01So all of these businesses, AI agent businesses, are funded, and they are privately funded generally, and that means those businesses need to grow, they need to get pipeline, and they need to book revenue, right? Um and so, in that pressure, when you work for one of those software companies, you're gonna start saying, Hey, Ricard, don't worry, it'll it'll integrate with your legacy systems. And you you just mentioned legacy systems, and uh, of course, everybody it looks great in the demo, and everyone rushes to implement and it doesn't deliver the outcome.
SPEAKER_01Probably going back to because the outcome wasn't clear in the first place, and then guess what? The magic AI agent didn't do it. So, so then let's look at take a look at some of the I just picked out some of the kind of what customers were saying out of this report. So we've got enterprises are piloting. Oh, sorry, these aren't the customer comments, they're they're I think on the next slide. These are some of the interesting things that I gleaned from the report. So enterprises are piloting Gen AI tools, but very few reach deployment, hence the POC fail rate.
SPEAKER_01Generic tools like ChatGPT are widely used, but custom solutions stall due to integration complexity and lack of fit with existing workflows. That's exactly what you were just talking about, Ricard, wasn't it, in terms of in integration complexity. And we all agree that Chat GPT, Grok, Claude, these are super powerful GPTs that we're all getting value from, but if left generic without your data, without customization, you've got a real limit on the value you're going to get out of it.
SPEAKER_01And then, of course, how much can you integrate a public kind of consumer chat GPT type product with all of your commercial information, for example? Um, so you you mentioned um integration, lack of fit with existing workflows, two more. Next one, enterprise users reported consistently positive experiences with consumer grade tools like ChatGPT and Copilot. These systems were praised for flexibility, familiarity, and immediate utility. There, they're just super useful, aren't they?
SPEAKER_01At your fingertips, but they're not driving the type of outcomes that an AI agent's meant to drive. And then the same users were overwhelmingly skeptical of customer vendor-pitched AI tools, describing them as brittle, over-engineered, or misaligned with actual workflows. And that's that piece around the industry having a role in why these things are are failing by the rush to get to an order. What's your thoughts um on some of those comments, Ricard?
SPEAKER_02No, but this is aligned exactly what I said about like the enterprise user report and like you know, positive consumer grade tools, right? It's helping them in their everyday life. And that's true, right? Like they're helping them to respond to emails, you know, to sort emails, or they can take this MIT report and throw it into ChatGPT and get the talking points and whatnot, right? They can do all that stuff as an individual, right? But the IDNT AI promise is this is going to help the enterprise accelerate, right? And then we come into another thing, like stall due to integration complexity in existing workflows.
SPEAKER_02So, like, you know, okay, take a step back. What is the workflow? Number one, there's probably a lot of manual touch points, whatnot. And if an AI can help this workflow, how will it work? That workflow. The workflow I described for myself was a non-existing workflow we had. It was just a frustration out of my head. It's like we really needed to lean in and engage with the community rapidly and uh often and on volume and so on and so forth, but we're only so few people. So, how can I accelerate that? Yeah. So it wasn't in the workflow, but it was a problem statement, right?
SPEAKER_02And we solved the problem statement. When I when I was trying to, when I've been trying to solve existing workflows we have, very difficult, to be honest. And we're using HopSpot for some and Stripe and QuickBooks. You know, I want to accelerate though, like uh, like uh, how does a funnel look like? What is people doing? How can we see where it's leaking in the funnel? Blah, blah, blah. Super hard to apply it to an existing workflow.
SPEAKER_01Creating something new, easy. Whoa, that's a really big point, that isn't it? That existing workflows can be really, really tricky to drive the outcome. If I, again, we're not talking about the JP Morgan's of this world now, right? We're talking about big, big businesses, but not megacat. Uh, most of the CEOs that I've I've spoken to, and some of them have been on this podcast, have said that their ambition is not to shrink their workforce with AI.
SPEAKER_01Initially, their ambition is to make, so if you checked, check out the episode with Graham Donahue, CEO at Forge Holiday Group, his ambition was to make everybody in the business 30% more effective. And then the net for that will be we're going to deliver a 30% better experience to our customers and we'll win market share that way. And also lots of CEOs saying they just they want to take out the The workflow that's really causing a horrible experience for some of their employees.
SPEAKER_01So Sarah in HR constantly has to wade through all of these manual reports. How can we and she hates it and she has to come in on the weekend to do it? How do we make her life better?
SPEAKER_00Yeah.
SPEAKER_01And so I'd encourage any customers, hey, Gore, first of all, before you try and automate the workflow, can you just fix it just by going through the process, get Sarah in the room with the team and say, right, let's understand the workflow. And then can we find some quick wins here to either eliminate it completely or speed it up somehow? Do that step, then look to automate that step. But I think you need someone in the room who can tell you the art of the possible around automation.
SPEAKER_02Yeah. I totally agree with that because like that people are speeding into the automation piece, right? And that is also whether you know you asked how many agents are out there. Agents, like per se, is like automation. But in reality, what people want to have is a specialized chat GPT experience, right? They want to have the exact same conservative experience, but it will respond with your knowledge and whatnot. So I would encourage any CEO who wants to lean into AI is to focus only on one thing and one thing only, and that is rag.
SPEAKER_02And that is basically fill an AI agent with your knowledge and have it only answer to that knowledge or use that knowledge as uh its knowledge, so to speak, not the web, nothing else, just that knowledge. And then you couple that with instructions. And the instructions is basically a prompt, like you probably a lot of people have what they call base prompts when they start a conversation with Chat GPT to have the right tone and da-da-da-da-da. Same thing here. You can have you know instructions saying you have knowledge, which is a hundred PDFs or whatever.
SPEAKER_02That knowledge is a sales playbook, and use that to help our sales reps do this and that and that. Doesn't have to be long, maybe 20, 30 rows. That's not it's not a lot. And then you you couple that with a chat GPT UI, and that's your first focus on applied AI in your organization. Period. No automation, no nothing else. If you can give that to Sarah, you know, going for manuals, whatever, and give her two weeks, and then also keep her in the loop in terms of oh, it didn't respond correctly here.
SPEAKER_02That's just an instruction set. You can even give her the ability to change the instructions in live in real time. So next time, don't respond with this. Actually, prioritize this and that, because she can have a conversation with the AI to respond better and train it. Because people think AI trains itself. It doesn't. Like you train the AI, you have to tell her what to do, and you have to do that progressively and get better, better outcomes, right? But if you can do that, then or you should not could you should do that. That will alone with the other person there will definitely increase your organization's you know productivity with 30%, like hands down.
SPEAKER_02And implementing a RAG AI agent today is incredibly easy. And the problem statement is do we have knowledge that we can pour into a bucket? And can we give that specific like uh like a uh a kind of a base prompt? And can we give that to a team of people and have them ask questions and increase productivity? That's the problem statement end-to-end. And if the answer is yes after two weeks, yeah, their productivity increased, okay, that's good. We have an we have something here, and then you can lean into automation.
SPEAKER_01Yeah, and uh uh uh Ricard used the term rag a couple of times. That's retrieval augmented generation, which is essentially when I first heard that term last year, I was like, what the hell does that mean? What? And um, here's getting your data, right?
SPEAKER_02Yeah, here's a follow-up file to tell AI look in there before you answer any questions.
SPEAKER_01Yeah, so you're gonna go retrieve my own data about my business as you're answering the questions and and generate your responses using my data. So I think that's a really, really cool way of getting started. Make Sarah's life easier, so she's happier. And then you're in POC as well, so you're keeping her involved, she's training the AI with her prompts. And I saw one of my customers actually is now running a training course for their employees on how to prompt their own internal AI, which I I thought was really interesting.
SPEAKER_01And then and then Sarah might say, Hey, do you know what? The only thing is it was quite slow this week, and it's like, ah, right, we have maybe a latency issue. How do and then you might be into how do we increase the performance? And she might say, Yeah, but actually, I asked about this data, and it just said it it wasn't there. Oh, right, we need to go and get another folder and put that in, or this data came out, but it was the old data and not the current data. And you're gonna start getting all of these little tweaks that are necessary to make to make this thing to make this thing work properly.
SPEAKER_01Um, but you're at the races then, aren't you? You're learning your you're into execution mode with AI rather than we threw this AI agent that we bought at this problem, it didn't work, or a load of data came up in it, like with those early co-pilot uh rollouts last year when it was coming up with the CEO's salary, pension details, and all this sort of thing. But why why was that? It was because there was a uh a chat in a Teams group somewhere that the AI got hold of.
SPEAKER_01So you got to be you got to keep it really narrow rather than just going, right, let's just give it this thing absolutely everything, yeah, and and maybe not make it customer-facing to kick things off. Hey. True.
SPEAKER_02And I also think what I've not uh realized myself, I've been playing around with it, is like as soon as you get into this groove, right? Sarah's groove and with his AI and the people on the team, is that's when you are starting to realize what it can do. It's like you you have to kind of experience and say, oh, this is cool, this is interesting, and what if we could, you know, will then already pop up as a question. Uh if you just like you know theorize around it and whiteboard it and think it's gonna fix it, and like and as I said, shit in, shit out, that's gonna be the outcome.
SPEAKER_02And then it's gonna be that 25% failed because you didn't do it like in the right order, in that sense.
SPEAKER_01Yeah, in indeed. So let's let's take a look at some more of from the MIT report. Excuse me. This chart is showing why Jenna AI pilots fail. Top barriers to scaling AI in the enterprise. Users were asked to rate each issue on a scale of one to ten. I guess ten being being much more of a problem than a one. Um uh at a I would call it a six and a half out of ten is challenging change management.
SPEAKER_01You need to kind of unpack that a bit, but yeah, hey, change is difficult within a company. We already knew that, didn't we, guys, before we went into this. Uh, so you probably got some communication, you've got some governance around that, and it hey, someone with their hand on the rudder. Uh, next at the same level, so that's six and a half out of ten as a challenge, was lack of executive sponsorship. Yeah, look, even Alexandra Musa Visa Day, who is CEO of Evident, who is in the room with the top CEOs in the banking industry, said the people leading on AI adoption in the likes of the JP Morgan's and the Barclays and the HSBC worlds, are where you've got executives who are really like big AI users outside of work as well as inside.
SPEAKER_01So I think the CEO's got to be in the room for this one personally and got to be excited about this. So, so yeah, I I would agree. And then we have a what have we got now? A getting higher, a seven out of ten on the scaling barriers to it to for AI and the enterprise is model output quality concerns. That's your shit out, isn't it, Ricard? It's your fault. Yeah, yeah. If the output's not what you're expecting, then uh uh yeah, guess what?
SPEAKER_01Go take a look at what you put into it. Yeah, and then uh and then the the final one's the highest one, nine out of ten. Unwillingness to adopt new tools. How interesting that people don't want to adopt new tools. They adopt things that make their lives easier, don't they? They don't adopt things that don't. But that's kind of that point kind of linked to challenging change management for me. What's what's your thought on some of these, Rick, Ricard?
SPEAKER_02No, but I think uh yeah, exactly. If it, you know, adopt new tools is easy if it's helping your everyday life, right? And I think everything here comes down to the sponsorship, right, in terms of tools as well and change management, I would agree. But I think the most in interesting here is the model output quality concerns. It's because you're asking the wrong questions. You need to, like if you're leaning into AI, which everyone should be like, I'm a firm believer, like you're not going to be replaced by AI, but you're going to be replaced by the people using AI.
SPEAKER_02That means like you need to lean into this. But if you hire a new person, don't you have onboarding programs, you have training sessions, you understand the company, you know, here's the manuals, here's the playbook, here's the HR manual. You're making sure then you have follow-up questions with this person every, especially in the beginning, you're fine, you're good. Then you have uh performance reviews. Uh, is this person doing the right thing? You need to treat AI exactly the same. Like it's not that there is no silver bullet, period. But if it but you can hire as many AI agents as you want, and they will do exactly what you tell them.
SPEAKER_02You know what I mean? So it's like also your fault if they're doing wrong, because they're not making their own decision in a sense, right? They're making the decision you told them to do. Uh a person you hire will definitely take their own decision, sometimes good, sometimes bad. But it's the same thing here. There is no silver bullet. You need to train it, you need to work with it, you have to do performance reviews, and so on and so forth. But if you do that correctly, you will definitely accelerate your business, whatever you're doing.
SPEAKER_01Yeah, and and look, the two points I'd make on that. First of all, on adopt unwillingness to adopt a new tool. There was no problem with Chat GPT adoption when it came out in November 2022. If you remember that chart, it beat hands down, it beat TikTok, it beat the internet, it beat Google search. Hands down to get to that first, what was it, a the first five, five million users?
SPEAKER_01It was like this vertical line of adoption upright. And no one was training on us on how to use it, right? We were just all like, flipping act, this thing's amazing. And so, and so now they their challenge is we can't get enough processors. So we know if the tool's done right, and if you give your employees an actual useful chat GPT-like experience, they're gonna use it. But the first wrong answer, and they're gonna they're gonna be like, oh, this thing's this thing's not working properly, right?
SPEAKER_02And that's a good point because like when you do approach, it's like you need to uh tell everyone like you're part of the training. This is like you're part of the loop. You can't just like be angry about the output. Tell me, you know, thumbs up, thumbs down. You have to have a feedback function. Like now, when we are implementing our community thingy, right? I had my problem stated in my head. I'm not running it now. Mark Jennings is my head of growth. He's running that tool now. And what he's doing right now is training it because he's like, okay, here's the outcome.
SPEAKER_02And what like I don't want this outcome, skip and tell me why. This I don't want, skip, tell me why. And this response I like, but I tweak it a little. And we're gonna do that for a couple of days, and then I can give all that information back to the agent and say, these are the things we want you to do better, and then it's gonna do that, so he's gonna help me train it, make it better. So I'm then in bringing him into the loop and creating like a super simple UI for him to basically say good, bad, and tell me why you didn't like it.
SPEAKER_01Yes, in indeed. And so he's constantly in the loop, which is partly governance and it's partly having the the team in place, and the person with the actual challenge, no matter how junior they are in your business, they're the one they're the people you want in the room and you want to get them on board. Over at Forge Holiday Group, such a great example. I think they called them their AI advocates. Yeah, so they picked smart people who were tech savvy, who in different parts of the business they could bring on and say, Hey, you recommend what we POC.
SPEAKER_01What was yeah, what AI would you like? Yeah, and um uh and so and so that's that's how they're doing it. The second point uh that you prompted me to make, Ricard, was yeah, I like that whole and I believe that whole mantra around AI's not gonna take you job, but someone using AI will. I think it's probably the same for the enterprise, isn't it? Is if if if you don't adopt AI in your business, you will lose to a business that has who does a similar thing to you who has adopted AI.
SPEAKER_01Yeah, because they're gonna get to 30% better, then probably 130% better, and that's what's going to happen. And I think that's maybe the new danger is once a customer has had a bad experience on a failed AI POC, and everyone's now rolling their eyes at AI, it's not that great after all, is it? Yeah, doesn't bloody well work, and then you were like, Well, we're not doing that again, because that was all a bit embarrassing, wasn't it? And cost us a bit of money.
SPEAKER_01And now all of a sudden we've decided to let's just wait another year. And what a what a strategic mistake that that would be. And maybe that's what's happening in a whole bunch of these businesses where they they've had the fail, the failed POC.
SPEAKER_02Yeah, just go back to the dot com boom, you know, and like, you know, let's go everyone on his website and they create a website. It's like, did it generate any sales? No. Ah, fuck it. Sorry. Or like, you know, we need to do a white paper. Did it generate more leads? No. Okay, scrap it. Oh, do we have a LinkedIn page now? Do we have a LinkedIn profile? Is it generating more leads? Nah. You know, but you know, if you do it correctly, yes, you will lose like to a competitor who does it better.
SPEAKER_02Um but the tools are going to stay, for sure.
SPEAKER_01Yeah, absolutely. Okay, let's uh let's move to the next screenshot I took of this report. Uh, what are we saying here? Exhibit this is the shadow AI economy. Totally real, that isn't it? And that is employee usage far outpaces official adoption usage of basically consumer GPTs, right? And so it's saying that for companies who have purchased an LLM subscription is at 40%, but employees who use LLMs regularly 90%.
SPEAKER_01And all my customers are saying, Oh, yeah, we've kind of like banned the use of chat GPT, and it's like, no, you haven't. No, you haven't. And also like, no, you haven't. I'm sorry, you might think you have, but you haven't. And if they're not using your internal tool, take a look at the internal tool you've given them. But I be so this seems to make perfect sense to me, right?
SPEAKER_02Yeah, and this is just like a signal that just bright red telling you, like, just imagine how many documents that would never supposed to be uploaded to ChatGPT, or who has been uploaded to ChatGPT for a meeting, for a business plan, where it says like confidential everywhere, blah blah blah. Yeah, I mean, yeah, yeah, it's you know, but this is like you know the Dropbox era when Dropbox was new. All of a sudden people were starting to share Dropbox files because it was easier.
SPEAKER_02I want to share a file with Ben, he's outside my organization, and my email doesn't allow big attachments. I gotta do it for Dropbox. And then the IT, the CIO is like, shoot, like, wow, this is a security bridge, the information is everywhere. And you don't care because you want to take the least friction way of sharing something. And then obviously Dropbox when you know leaned into enterprise and box and so forth. Same here now. Like this is gonna happen whether the enterprise like it or not. Make sure you provide the tools you want them to use.
SPEAKER_01Yeah, okay, let's go to the next slide. So, what was this one that I grabbed? Would it's quite a funny one actually? Perceived fitness for high-stakes work. Would you assign this task to AI or a junior colleague? And uh one's AI preferred and one's human preferred. And so for a complex project, uh like a multi-week work or client management, 90% is human preferred.
SPEAKER_01And then, of course, for your quick tasks, emails, summaries, bake basic analysis, 70% preferred AI. Yep, that's the point, isn't it? That we're trying to so the complex project, the strategic stuff, how I look after my client and manage that relationship, that's what I want to spend my time doing and listening to them and learning, and hopefully winning new work with them. The quick stuff, I want AI to do that.
SPEAKER_01So I wasn't kind of surprised with this, but what I would say is your example that you started this podcast episode with, Ricard, was quite a complex task that you're now built an AI agent to actually deliver.
SPEAKER_02Yeah. The thing is, like multi-wick work, he says here, like client management, whatnot. Of course, you don't want something to be automated where you have no control, right? You need to have an approval process, and that's what I created. Like, I have an agent. You know, we have the scrapers ingesting it, and then we have an agent analyzing it and then creating tasks about it. But I would never let the AI just go rogue and posting things on my name on Reddit without me reviewing it, right? But I don't want to do the heavy work. The only concern I have is like, am I missing out on something? Am I looking at the right things or blah, blah, blah, blah.
SPEAKER_02And that's the training process. Like, I need to focus and do some manual testing as well. So, like, I have my agent in the background, but I also have my own Chat GPT experience where I can ask it very pointy questions and then see, did my background, my agent, find the same thing as I did? If it didn't, then I trained it wrongly, right? I kind of align that. So, I mean, not anytime soon. You're gonna give a complex project to an AI alone and then say, go off. But in that complex project, there's a lot of tasks you're doing that you can alone speed up, which I'm doing right now, right?
SPEAKER_02So instead of doing all this scraping on my own and then putting in something kind of document and then task it, and that I shortened that.
SPEAKER_01Yeah, look, and uh I think this is this is the goal is make a start, get this technology useful somewhere in your business to take on the as they're calling them, these quick tasks. Yeah, then build on it to go go for the more complex. Let's look at the next one. How how executives select gen AI vendors? This is derived from interviews and coded by these categories, it's saying uh so 50% of the reason why executives selected gen AI vendors is down to flexibility when things change.
SPEAKER_01Next one is 60% was down to the ability to improve over time. Then 65 clear data boundaries, 65%, 70% of the reason was minimal disruption to current tools. Nine uh sorry, a 75% was down to deep understanding of our workflow, and then 90% was a vendor we trust. A few things in there. One of the, if you actually read the report, what they were saying was over and over again is the AI is not learning.
SPEAKER_01It's just still it's not learning and it's not adapting. And maybe that's that that piece around the ability to improve over time and sort of grow and learn with you. But what some of these make sense, right, Ricard? But what's your perspective?
SPEAKER_02Depends on the definition with Gen AI, right? Gen AI, there's not that many. You have OpenAI, Anthropic, Gemini, Grok, basically, right? And then you have people implementing these and working with them by demo and Accenture and whatnot. I don't know what like exactly how this question is asked, because this is like how SecTV select an operative system is basically what they're asking. It's like is a macOS line, it's a Windows. There's not that many to choose from. And this is kind of a I would think like kind of a shit thing shit out again, like it's a pretty bad question.
SPEAKER_02It's like, okay, yeah, yeah, yeah. Generative AI is per definition generative. It will answer you. If you ask the population of Sweden, it will ask answer, give me that answer. But if you're gonna ask me what is Bendrake's university doing and how should they improve business based on the data I have in my CRM, it will say I have. Yes. So that's where agentic AI comes in. And like the question should be like how to exactly select agentic AI vendors, maybe which should be the question. And there's not that many to choose from.
SPEAKER_02And you can build agentic AI with any generative AI. Now if you're going to throw around worlds words. So because I I used like for our thing, we're using Claude, our fabric. So that's a generate the AI engine, so to speak, the LLM, the vendor is anthropic. And but then we train that specifically for our use case. And now we have our own rag. The RAG is the community data, right? That's the RAID. Yes, right. I don't ask you to answer ask anything for internet. And I train it to say, I don't give a crap about your training you have in your memory.
SPEAKER_02Just care about what we have. Don't taint it with your opinions, is basically where I'm telling you to. So it's very like focusing on mine.
SPEAKER_01It's that focus on workflow that we've discussed. Data. Where's the data? What states the data are in? Is the data accurate and has it got gaps in it? Because if that's the shit in, if it is which is going to give you the shit out and your your parlance, uh Ricard, which I love. And then never more so did you need a partner who really understands your business and doesn't just have this generic view that the technology is going to just magically work.
SPEAKER_01Oh, yeah. Here, here's some of the quotes. These are direct quotes. So we're on vendors, uh, we're more likely to wait for our existing partner to add AI than gamble on a startup. Lots of what we didn't discuss is lots of these big SaaS. You've got agent force, all the big SaaS providers have got all their AI agents now. And it's don't you worry about those new AI agents. You stick with us. Uh you'll be just fine. So make just to confuse the matters even more.
SPEAKER_01Next quote was most vendors don't get how our approvals or data flows work. Yep, that was ever thus. It was ever thus in technology. I've been delivering these programs for years and years and years, and that's always the problem. Uh, next one, if it doesn't plug into Salesforce or our internal systems, no one's going to use it. Yep, completely. Again, it doesn't have access to the critical data. As Ricard just outlined, then you're not gonna you have no rag, you have nothing to retrieve, and it's it's just gonna be too generic for you.
SPEAKER_01Next one is I can't risk client data mixing with someone else's model, even if the vendor says it's fine. I love that. And the vendor said it's absolutely fine for us to drop all of our commercial information into that AI. They said it'd be cool, don't you worry about that. Of course they did. Uh, two more, and then I'll get your take, Ricard. It's useful the first week, but then it just repeats the same mistakes. Why would I use that? And then finally, our process evolves every quarter.
SPEAKER_01If the AI can't adapt, we're back to spreadsheets.
SPEAKER_02Oh man. You know, drizzled in misconceptions and you know, and like, you know, buying you know the silver bullet promise from a lot of the vendors. But a lot of this is back to the analogy I did before. Like when you hire a new person, you have to train them, you have to follow up, you have to do performance reviews, you have to change. And these like, you know, it's like if it doesn't adapt, we're back to spreadsheets. Yeah, but if you're just gonna deploy an AI and then think it's just gonna figure it out itself, that's not what's gonna happen. AI is like not that smart.
SPEAKER_02AI is dumb, right? In the end, it's just making very qualified guesses. That's what AI is, if we're gonna be honest with each other. But so these are like, and also like again, like you know, crap in, crap out analogy is like, yes, data is something, but also if you tell it the wrong instructions around the data and how you massage the data through that workflow, and you get something out in the other end for the end user, all those things are crap in, crap out. And people think it's like a chat GPT experience. Question in, something comes out, and that's the workflow.
SPEAKER_02But in reality, as I said, like my agent I created is like a multiple step agent that is I said, okay, someone is asking for something. First, I have to analyze what they're asking for, so I can come up with a search strategy, and that's gonna create like 10 different queries. And then when it's done with that and extracted all the information, I ask the agent again, are you happy with this? You like this, or do you think you need more? I do think I need more. So I can I create three more questions, and then I'm happy with this result. Now I'm gonna analyze this result, and then when you analyze the result, it's like you can ask the A agents again, are you happy with this one if you're actually gonna be critical with yourself?
SPEAKER_02And then you can actually mix that around. Like you can have Claude in one phase, and then you can have Gemini in another one just to get like a second opinion kind of approach. And then you can ask ChatGPT in the end or OpenAI in the end and say, like, take the best of those two reports and deliver that to the end user. All this is input to create an output. But people think it's just input-output. So data absolutely, but the the magic with data is like AI is worth nothing for an enterprise if it's not your data. And when it comes to data, you should actually approach it like in the in the like a decade ago, there was this big trend called big data.
SPEAKER_02Dump all you have in one big data set, don't categorize, don't tag it, don't care. Because if you put something on top of that, you will see the thread, you will see the patterns, and that is correct. And that is even more so now, because rag is basically dump it all into some kind of a rag, and then the AI will find the patterns together with your instructions. So the gaps are fine because you might find the gaps, and don't over-engineer the data collection, just take it and dump it.
SPEAKER_02Because that's like the conversation, the first conversation I have with my engineers when we're gonna do this, it's like, but how do we filter it? Like, and how do we think about like should I should I value the LinkedIn post I'm looking at before I ingest it? No, just dump it. Just dump it because in that LinkedIn post there is an intention, there is a story that you can analyze and you can see it's about something, it's about lovable, it's about replit, it's about XYZ. The AI will figure that out. But it needs to be able to retrieve it effectively, right?
SPEAKER_02And that is a rag. Yes, yes, together with instructions. If you say it's financial data, it will fight figure it out. If you say it's community data, it will have a different approach. So that process of collecting data, do not over-engineer, just dump it, and then on top then you train the ADI agent around that particular data set.
SPEAKER_01Yeah, yeah. It absolutely and and such great advice. And the whole point of this podcast is we we believe in AI. Everyone listening to this podcast, if you've especially got all the way through this podcast, you believe in AI. Of course you do. We know it's super powerful. It's just helping helping businesses uh execute the their initial footsteps in using AI. And don't worry every everybody, we are still early, right? This is super, super early in this industry for sure.
SPEAKER_01Now, if you'd like any help on that, and you want to speak to myself and Ricard, because you just want to chat through where you're at in terms of your AI initiatives. Maybe it's maybe you're in the bucket of people who've having problems with POCs. You know, just reach out. You've got our contact details at the on the in the show notes of this podcast, but and I know we've got a hard stop now, Ricard. So so thank you for joining again today. I thought that was a really useful, useful episode, and I hope you enjoyed it too, Ricard.
SPEAKER_01So we will definitely be having you back on definitely before Christmas, hey. Let's do let's say the world will change again. Yeah, it will. It will. But um, okay, brilliant, awesome. Well, look, thanks so much, and let's wrap it up there. I'll see you on the next one.
Data centers are at the heart of everything in AI right now and the backlash is growing. Ben Drakes is joined by regular contributor Thomas Odenwald, a senior executive with HPE and SAP and former head of North America...
Jensen Huang stepped down as Nvidia CEO about an hour and a half to two hours before this recording and Colette Kress, Nvidia's CFO, is stepping in as interim CEO. The news was not on the docket and it took up the bulk...
Protesters interrupted OpenAI's kickoff keynote at Seattle AI Week and Ben Pickett caught it on his phone. He describes more than half a dozen people repeatedly disrupting the session before security and police...
Two Harvard researchers looked at 718 companies. When AI coding agents came in, the code went up 30%, but finished software didn't measurably move. Where did it go? Into review: the time to review the code went up by...
Frontier models are turning into a commodity. When one model goes off the boil, enterprises switch, and a multimillion-dollar contract signed with one provider can look foolish in six weeks. Ben Drakes and Ben Pickett...