Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Rogue Startups

Rogue Startups artwork

Is the SaaSpocalypse Over?

Oct 22, 202644 minAudio
ShareXLinkedInFacebookWhatsAppEmail

In this episode Craig talks with Aaron, who built DocsBot, an AI customer support agent that's almost four years old, about what AI is really doing to SaaS. Aaron shares DocsBot's numbers: slower growth and fewer new customers, but more revenue per customer and lower churn after narrowing in on the right customers. They also get into what it costs to run an AI product, how to choose and switch models, and the AI tools Aaron is excited about next. Topics discussed Growth in the AI era: new customer acquisition is down year over year, but revenue per customer is up 20% from last year Finding the ideal customer: talking to customers, cancellation surveys, and pointing AI agents at all of the product and Stripe data, after reading Jason Cohen's book Hidden Multipliers Signal density: why founders with only a handful of customers struggle to draw conclusions Where customers come from now: organic search, AI search, referrals, conferences like WordCamp, and an agency partner program Why big companies move slowly: and why that makes larger customers stickier Churn to DIY: churned customers citing AI or their own custom solution went from 1.6% to 13%, and most weren't a good fit anyway Pricing around token costs: from bring-your-own-key to included usage to AI credits Evaluating new models: why letting customers edit their system prompt makes evals hard, and a classifier that behaved differently on GPT-5.4 Choosing models: balancing cost, speed and quality instead of reaching for the newest model Local and specialised models: embedding models, Cohere for other languages, and Mistral for PDF extraction Privacy: a tiny model in the chat widget that redacts personal information before it leaves the user's browser What's next: using a new model to re-rank search results at a fraction of the cost Key takeaways The biggest cheat code for finding your ideal customer, in Aaron's words, is to "talk to your customers". Don't pick your ideal customer by revenue alone: DocsBot's biggest enterprise customer uses the product completely differently from almost everyone else. Customers who leave to build their own alternative are often not best-fit customers: too price sensitive, a tiny team, or not enough support tickets. Bring-your-own-key let a fully bootstrapped company launch an AI product without paying for inference for about the first two years. For an AI support agent, speed matters, so the newest model on its highest reasoning setting isn't always the right choice. Model upgrades can cause regressions that evals don't necessarily show. Chapters (00:00:00) - Intro: Is the SaaS apocalypse here? (00:00:59) - Growth metrics and MRR trends over 12 months (00:02:08) - Focusing on ICP to boost revenue per customer (00:03:11) - Churn rates and pricing restructuring (00:04:49) - Talking to customers to define the ideal customer profile (00:07:32) - Signal density and sample size in customer data (00:09:13) - Customer acquisition channels and what's working now (00:10:47) - Events, conferences, and agency partnerships (00:13:05) - Enterprise inertia vs. consumer switching risk (00:15:19) - Enterprise adoption pace and pilot deployment friction (00:17:03) - How AI changes customer support buying behavior (00:19:07) - DIY/AI churn: are customers building their own solutions? (00:20:45) - Price increases and churn reasons breakdown (00:21:55) - Token costs, pricing strategy, and bring-your-own-key history (00:26:06) - Evaluating and switching AI models internally (00:29:24) - Personal model stack and balancing cost, speed, quality (00:34:00) - Local models, embeddings, and specialized extraction tools (00:36:25) - Small/local models and future of specialized AI (00:38:56) - PII redaction with tiny on-device models (00:40:16) - What's exciting next: re-ranking, Gemini, and new paradigms (00:43:19) - Wrap-up and where to find DocSpot

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Speaker AWell, I think we can say the SaaS apocalypse has passed and we're all still here for now. I guess you sit at an interesting place as you're an AI tool which has a strong vibe, coding, replacement risk, but obviously doing really well. We're customers. It does a better job than I think I could do by myself and I'm reasonably technical. But how, how are you feeling today about like the business and, and the viability, I guess.

Speaker BYeah, good questions. I feel good actually. That's the numbers. If you want me to share, I can share a sure slide and that might actually be very visual and cool. Um, so this is like actually going in deep. I shared this on another podcast the other day, just kind of looking at our stats over the last 12 months to kind of see what's been going on, you know, and that really ties into kind of the patterns, how we're seeing AI affecting SaaS. So we're still growing, but it's a lot slower than it used to be. So if you look at like our all time kind of MRR graph and you can see it's pretty steady, somewhat steep, and then over the last 12 months it's somewhat leveled out and I've heard similar stories from a lot of other founders as well.

Speaker AOur graph looks very similar. Just rewind of years and the flat part is longer.

Speaker BOkay.

Speaker AAnd just for folks who are listening only, like Aaron showed the first graph here was like MRR up 5% but ARPU and LTV up quite a bit. So, you know, not getting more customers, but customers paying more. And so that, that's like I always say, one of the best levers in SaaS is expansion revenue. And so you have that, you have that nailed in somehow.

Speaker BYeah, exactly. Yeah. So what we've really done, like, like if I jump ahead and you can see like really acquisition, I mean that's been the biggest challenge for many different reasons, but you can see year over year that's a pretty big drop. And so what we put a whole lot of focus on this year is getting the right customers and better customers, you know, really narrowing down like on our ICP and our targeting and what we spend on marketing and all those kinds of things, landing pages, all that, even restructuring our plans a bit to target like our ideal customer profile. And so what that's led to is good. We have more revenue per customer, you know, and that's up 20% from last year. And lifetime value especially is up a lot higher because with this we've also been able to decrease churn A bit. So that's kind of been our focus and that's kept us afloat.

Speaker AYeah. Are you all kind of a high or low churn business? You don't have to give exact numbers but like, yeah, good.

Speaker BOr I got a slide for that so you can tell me if it's high or low. There's a subscriber turn rate, so we have dropped that 27% this year.

Speaker AYep.

Speaker BSo that's good.

Speaker ASo from like 8% to 4 or 5%.

Speaker BYeah, yeah, yeah. 8 to 10 to, to that. So we're getting those better fit customers. And probably more importantly is revenue churn. So that's factored in, you know, upgrades and plan changes and pricing changes and all that kind of stuff. So that's down as well a bit. So that's been helpful in offsetting kind of like the decrease on the front end of new people coming in. So we're getting more of the better people, which is helping a lot. And I think the other big thing for us for growth is we did do a price increase or more of a restructuring. But probably our most popular targeted plan, like the one that says recommended on the pricing page, you know, or most popular, that one, we did have a substantial price increase to bring us kind of in line with competitors. And alongside that was when we released a whole lot of new like agentic kind of features and stuff which increased a lot of costs on the back end as well, you know. So there was some cost increase but mainly it's just like trying to get figure out okay, what's the best price range for the average best ICP person, you know. And so that's really helped us as well, like keeping up with growth slightly and not being in the negative.

Speaker AYeah, yeah. Do, do you think that like the focus on certain ICPs is, is like a product thing or like a marketing thing because like customer acquisition is, is gone down. So I think you could argue that like, oh, this focus on this particular icp.

Speaker BYeah.

Speaker AIsn't successful if you have less trial, like relatively less trials, but you're getting higher paying customers. So like do you call it a success?

Speaker BYeah, I mean it is both, obviously. Like, I mean probably the biggest cheat code is talk to your customers, right? Yeah, just talk to them and figure out what they want. So that may be surveys that you're collecting during signup or during churn cancellation, you know, to figure out what's actually going on their head. Most importantly, just get on calls with them, even if that's not necessarily like scalable to have Like a whole sales process. But the, the insights that it reveals are huge. Like on, on. And then figuring out, like, I just sent. So I read the Hidden Multipliers, new book by Jason Cohen. I know you're a friend of his. And so I just sent, you know, our agents that all. All our data, you know, our whole data lake for the whole product and stripe and everything say, all right, what is our best icp, which in that book they call the Carol, you know, essentially. So it's narrowing down much more than just like, oh, it's women over 35 or something, you know, it's narrowing down to like, what's important to that person, you know, and where the money and budget comes from for that person, etc. You know.

Speaker AYeah.

Speaker BSo really narrowing down to that is super helpful. And we do that based not by just. Not just like the money that they bring in because we have, like, probably our biggest customer that pays us, like a big enterprise deal. They use the product in a completely different way than almost everyone else, you know, So I can't just like decide by revenue alone because that's, that's really not. I mean, that's great that we have that revenue from that customer or a handful of customers that use it in a strange way. But that's not our ideal profile, you know. So essentially we just narrowed it down, looking at every single customer who's using it, how they're using it, what they're using it for, what they've trained their AI agents for, on and all that kind of stuff, you know, to kind of get a much better general ICP that's not tied to a specific industry or anything like that, but really tied to what is important to that person.

Speaker AYeah, yeah. How many, like, orders of magnitude your hunt? Hundreds or thousands of customers?

Speaker BYeah, I think it's somewhere between five hundred and a thousand. Okay.

Speaker AI just, I think one of the biggest, like, risks for folks early on, and you're not at the early on stage, but like, for folks early on is like, I call it signal density. It's like they don't have enough people saying the same. To draw the conclusions that you have, like, you need a reason. Like, we have, we have a lot of customers, so like three or four thousand customers that all pay us a little bit of money. But it's easier for us to say, to draw those conclusions. We have a lot more volume. You have enough volume to be able to say, like, oh, we have five or five hundred or a thousand customers. There are these bands of customers that are that are really good. But if you have 20 customers, I think the likelihood that all 20 of them or enough of those 20 are similar for you to be able to draw conclusions is like really hard.

Speaker BYeah.

Speaker ASo it's kind of a chicken or the egg thing.

Speaker BNo, that's very true. I feel like our first. So we're almost four years old now, so we're one of the first in the space for AI customer support agents, you know, using RAG and technology. And we, we actually released like the same week that chat GPT APIs came out, you know, so we're one of the first there, which is great building brand and SEO and things like that. But it was completely new industry, completely new everything, you know, so we had no idea like what the best use case was, what our ideal customer was, you know, and we're really relying a lot for our initial growth on kind of just like the whole viral like, oh, AI is like amazing. We need to try it, you know.

Speaker AYeah.

Speaker BEvery, every company out there figuring out how they can use AI and how are you.

Speaker AHow are you acquiring customers now?

Speaker BYeah. So I wish I knew. No, still, I mean, why.

Speaker ALet me, let me frame that. I talk to a lot of founders either in like my work with Tiny Seed or I do some coaching and like this podcast. Get. Get to chat with a lot of smart folks. Not a lot of people have a great answer for that these days. I'll say they know what marketing they do. They have less of an idea of what works well and even less what they should be doing more of.

Speaker BYeah. So I can share kind of what I have too here. This is just something I post on X recently. This is purely going just on what data we have. You know, as you can see, there's like a huge chunk that's direct, which basically just means we have no idea.

Speaker AYeah, yeah.

Speaker BWhere they're coming from. But search was pretty big and that has dropped this year, definitely. But not massively so. Organic still is our biggest source. And even like last year, like AI, like referrals and stuff from ChatGPT and perplexity and things like that. Like that definitely we get, when we do like sales calls, like for more kind of like higher tier potential customers, we do a lot of like demo calls and things like that with them. And I always ask them, you know, how they found out about us. So AI search or AEO or whatever you may call it, that's definitely been a pretty decent source. And then of course we have just referrals, you know, word of mouth, what we've really been pushing into this year. A whole lot is events and in person stuff. So gone to a handful of conferences and just really not necessarily like did the whole booth thing, but just hardcore like hit the floor. The whole team talking to every single person out there, you know, like giving them demos just on the fly or whatever. Trying to be pre, pre booked meetings however we could. And we found that that is a huge leverage. I mean it's a lot more work, it's slower. Well, kind of differences. Yeah. So we went to work Camp Europe and US Press conf and we went to one that's basically for agency owners that was actually great. So that's another push that we've been doing this year is really building out like an agency partner program and let them sell our product to small businesses, you know, small medium businesses, their own clients and help them win in the process, you know.

Speaker AYeah, that's always the like gold mine. Right. Of every founder's dream is like let me get channel partners in place to sell this stuff for me and I can sell this one agency and then they go sell 10 clients a month on it or something like that. And my experience, that's like a lot easier to say than to do. So if you guys are cracking that, like that's great. That must have been a significant amount of work.

Speaker BYeah, I feel like we're just starting to like we have this huge list of leads and people are talking to and then you're trying to get them. Okay. Figure out, identify one client that might be able to use this, you know, to get kind of get that going. So we've definitely had a handful of growth and we haven't seen like an exponential lift from that. Yeah, but that is the idea. I mean you, you do the work once and then they do the work for you. And yeah, I think as we're talking just more general about like AI and SAS as you're talking. I think that's the pattern that I'm seeing and hearing. Especially if you're. If it's B2B, this is like where it ties in is yes, we have like all these new AI tools and technology and probably competing products and things like that and even like people vibe coding stuff. But for the most part like your normal like SMB is not gonna like vibe code their own solution or they wouldn't even, they don't even know how to like identify and purchase software like for a certain use case, you know. And so that's kind of where when I talk to these agencies, that's kind of the same thing that they're telling me. You know, it's like they need someone to look, to go into the business, figure out, okay, what are the workflows they're doing, what are the pain points they're experiencing, where are the inefficiencies, and interpret that into a solution that AI may or may not help with, you know, and that really hasn't changed since it was just SaaS and not AI. You know, it's not really any different. It's maybe even more complex because AI feels even more like, overwhelming to people because of the speed that it's moving and all that kind of thing.

Speaker AYeah, yeah, yeah, yeah. I mean, I do a bit of consulting for companies around like, AI, like strategy and implementation. I think one thing that has been illuminating for me is like the pace of change with AI is so rapid, but the ability for even moderate sized companies, you know, kind of doing a half billion dollars a year up is like really slow.

Speaker BYeah.

Speaker AYou know, I, I have direct line to vice president of marketing at a big old, huge company and they're like, it's going to be six months before we can do that.

Speaker BRight.

Speaker AI'm like, you're shitting me. Like, like, who knows what this is going to look like in six months. And so I think, I think like, with respect to the SaaS apocalypse or people vibe coding their stuff, like, if you're selling at big companies, the ability for them to change is very, very, very small. If you're selling to you and me and like, I could just pick up, you know, Help Scouts Tool tomorrow or Doc Spot or, you know, ATO or HubSpot or whatever. And no one's going to tell me what to do. I think that's where the risk comes in. And so like, probably what you found is like, oh, our best customers are sticky because they have this like, degree of enterprise ness where, oh, getting them over the hill is hard, but once you do, they're going to stay for a long time because just their ability to change is much less.

Speaker BYeah, I would say probably around about at least five employees. Like once you're talking to an employee and like not the boss who does everything, you know,

Speaker Athat's a good litmus test.

Speaker BYeah, yeah. Who are assigned to like, have a role, you know, and this is what they do and they have to get approval, you know. Yeah, it's harder. Like we have, we have people that we're like doing pilots with and I'm like, oh, this contract is not even that big and it's taken them months, you know, and they're saying, how many months will this take to deploy? And it's like, honestly, like, we've built it. Like, there's literally an agent built into the platform. You just say, build it for me, and it does it, you know.

Speaker AYeah, yeah.

Speaker BAnd it tests everything for you and trains your bot and sets up all the escalation rules and everything. You know, like, we can get this working for you in a day. Like, oh, we have to go through this process approvals. The whole support team has to test the bot for a year, you know, before they feel okay using it.

Speaker AYou know, I mean, that was like, so we're customers. That was our, that was our process, I think, is we graduated into it. We had it running for a while and, and we evaluated its responses or we evaluated. We tested it internally, I think, for a while.

Speaker BYeah.

Speaker AAnd then we turned it on. And it always has had like a good escape hatch. Like people can talk to a customer and then we load the Help Scout widget if they need to send us a message. Yeah, yeah. I think we saw initially like a pretty good drop, like 30 plus percent reduction in tickets, which is good. I think the most, interestingly, the most helpful thing for us has been we now have like, in our own internal process where we have an agent that monitors all conversations in Help Scout and has us go updating our documentation every week. So, like, hey, someone asked this question. There's not a doc about this thing which is probably just making your tool better because you're trained on like our website and on our help docs. So like, I think that's, that's like a little microcosm of, I think overall what's happening with AI and like the, the change in dynamic of the need that people have. Yeah, because that's really what's happening is like nobody's buying a help desk solution anymore. You know, like the way. The way they used to. They're buying a different thing. And that's probably like what we all can't get our head around because it's changing maybe, and it's changing for each customer. Like kind of where they are in this, in this journey is like, how, how. How much other stuff do they have going on that's helping their business? And how. How can you fit in on that is a moving target. And you're probably like right in the, like, we're not in that really at all as podcast hosting software, but you're probably kind of right in the middle of that because this is A a place where people like, rightfully I think, say, hey, we can use AI to make this a lot better and faster and cheaper. But you're also not the, you're not the only solution they're deploying around this. There's a fair amount of people involved, probably for all of your customers.

Speaker BYeah. And I think one thing also would be kind of like the whole. I could build it myself now, you know, and in the last year since kind of agentic coding, you know, Claude code and codecs have become, become like much more accessible to normies, I guess. And they're trying to actually build things themselves and play around with it and stuff. I know a lot of people have been saying that's the SAS apocalypse where anyone can build their own software. So I actually dug into kind of our churn to kind of get an idea what that looks like. This is looking at like when. So we kind of do like a. We do force a little mini survey when they're canceling, which is super helpful. And you can see kind of like how that has increased to where people mention like either using AI or just creating their own custom solutions. So that's definitely been a big increase, but it's still not like a massive amount of the churn reasons like it's 13% in the last quarter and this isn't like a massive amount of data statistically. But I thought it was kind of interesting to see how that is changing and how that is affecting. What is cool though is when I dig into these people like they were not a best fit customer, you know, so.

Speaker ARight.

Speaker BSo either they're too price sensitive or they just like it's a tiny team that was able to do a solution, you know, or they just didn't have enough support tickets or something like that. You know, it's not stockspot. Even without making their own alternative was. Was not enough value for them, you know, to pay our monthly rate. So it's not like a massive loss in that regard. Even though it's scary when you see those like alerts come in that says, oh, I created my own alternative. Yeah, yeah, yeah.

Speaker AI mean, yeah, I would, I would look at this graph and just for folks who are just listening, kind of the initial cohort here is this like fourth quarter last year, 1.6% of churned customers cited using like DIY or custom alternative increased pretty progressively to now it's 13% of churn customers are rolling their own solution. Yeah, yeah, this feels right. This feels right. I'm sure the biggest one is I'M not using it enough and it's too expensive. Which. Those are kind of the same thing, you know, those are flavors of the same thing.

Speaker BYeah, exactly. And they both kind of identify like, not a good fit. Um, yeah. So in our case.

Speaker ARight.

Speaker BIn our case, we did have a price increase last year.

Speaker AOkay.

Speaker BSo you can kind of see how that's gone, you know, from 18 up to 27. That's like a 10 points kind of increase, which makes sense. And that's fine. We expected, you know, that. And that's still. Overall, it increased our, our revenue and profit, you know, and we just filtered out kind of the, the less of a good fit users and attracted more of the good fit users.

Speaker AYeah, yeah. I want to ask you about the technical side of things because I think one thing that is glossed over a lot and not with you, I'm sure, because you're an AI tool company. But we all think of like, oh, SaaS has great margins and it's kind of a lot of fixed costs and then everything's gravy on top of that. But you're paying for tokens for every customer. How do you. I guess, first question is like, how do you think about that in terms of how you price things? How do you think about. Oh, we expect our token cost to be this. And so we need to bake this in to have the kind of margin we're looking for. Maybe there's something else you do.

Speaker BYeah, it is an interesting world. So when we first launched, it was GPT 3.5 and GPT 4. And since then, the cost of intelligence, like for the same amount of intelligence as that has dropped. I think it's something like a thousand extras. It's high. You know, it's in the hundreds of how much like smarts you get for the same amount of money. Right.

Speaker AOkay.

Speaker BAnd so that's been great for us. So first of all, when that launched, I mean, it was expensive, like GPT4. It would be like, it's like $0.05 per, like customer support question that would come in. Right. And so I was fully bootstrapped, had no cash to like put into this. You know, it wasn't trying to run like an unprofitable company. So we actually started bring your own key. So we never actually paid the AI cost ourselves for the first two years or so of, of the product. And so businesses, they sign up, they had the software and they had a place to paste in their OpenAI key, and we just used that for all the inference. So that was great. For us. And surprisingly it didn't. Like people didn't complain. Obviously it added friction to the sign up and onboarding process. But at that point in time, AI was still so exciting and fresh that no one cares. They do that no problem. Create their own OpenAI account and add that key. So that was really helpful for us. Once we started getting like faster, better, cheaper models like GPT4O I think is when we started doing this is where we started building in like a certain amount of usage, you know, and we would charge. We kind of just based on the plans, we say, okay, you get this many user messages or whatever, right. Or this many support questions that's tied to the plan level. And that worked for a while. And then like in the last year as everything started getting more agentic and more complex, we were able to use like cheaper, smarter models still like when the five family came out and stuff and reasoning started being released. But at the same time like you're starting to add tool calls and it's like a full agentic chain where it's calling this tool and coming back and calling another tool coming back, you know, and all that keeps adding to the token cost. So we kind of had to reformat our thing and instead of tying like those limits to like per user question, because user questions could vary like in the cost substantially depending on everything from the tools they have enabled to the training data, to the type of question, et cetera, to a more of an AI credit based. We just renamed it AI Credits and we said okay, before one question was equal to one credit. Right. And then we just looked at all our customers on average, how many tokens or how much token cost do they use per user message or whatever. And then we just kind of set that as like the baseline so that when we made that transition for most users, or at least on average across all users, like the message credits and AI credits were kind of similar. The same unless they added new features or change the model or something like that. Yeah, that's kind of how we approach that.

Speaker AYeah, yeah. Do you. One of the things like I'm looking at right now with like just my own internal work and we have a, we have a bit of AI both in terms of like our own internal process, like how we do development and how we do, you know, PR reviews and stuff like that. But then, but like so we have AI in the product and we have our own stuff is how we're evaluating new models. That must be a massive thing for you when you change your own internal models. We Have a pretty shit set of evals. It's basically like, oh, Opus 5.5 is awesome, let's just start using that. That's not smart. As you get more entrenched in agentic systems and your product and your internal processes are relying on this stuff more and more, how do you look at evaluating, I guess two questions. How do you look at evaluating new models and how do you think about when you should look at new models? If the other one ain't broke, don't fix it maybe. Or is that wrong?

Speaker BYeah, that's the other tough side of doing AI agents. So it's kind of been a mixed bag. What kind of happened is users would be able to choose which model and so new one would come out, we'd add it. We may make some, just do some testing with a new bot in our default prompt templates. One of the downsides with evals and stuff is we let users change a large amount of like their system prompt essentially. So that gives them a lot of power. But it means that changing anything like a model can completely change the behavior of their bot. That's a bit of a nightmare because it's fully open ended. So there are like internals have some

Speaker Alike fixed evals with like this is the system prompt, this is the input, this is the output. And you can compare different models?

Speaker BYeah, yeah, so we can compare. So like for example, like we may need to adjust like our internal like tool descriptions or things like that, classifiers, stuff like that, that the customers don't see. So we'll run evals against that and testing. But yeah, it's hard and sometimes you have like regressions. So for example we have like an internal classifier that classifies if the bot was able to answer the user or not. And that kind of goes into public or into stats that they see if the AI is able to respond or not. And that had a fully different meeting in the GPT4 family. And then the GPT like once GPT5.4 came out, it just took those instructions a lot more literally. And we had about probably like a quarter of our customers where suddenly it just looks like when they upgrade that model it just looks like their bot is performing horribly. You know, when the answers really haven't changed. In fact the answer quality probably went up. It's just classifying it differently more so stuff like that can be a nightmare to deal with. And a lot of times you're just kind of dealing with it when it's discovered like it doesn't evals and stuff don't show that necessarily.

Speaker ARight, right. What is your, like, your current, like, personal stack for, like, I guess like development work, but also like, what, what kinds of internal, I'll say agents. That's such a bad term. But like, what kind of AI systems do, do y' all run?

Speaker BYeah. So as far as models, like, I kind of love this topic because it makes me mad a little bit. Like, everyone on X and stuff, they're excited about the new release and they gotta use Astra for everything. Or Opus 5.5 just came out and everyone's going nuts about that. Right. But when you're actually like, building a product that like, real customers are using, it's not, you don't just always just throw the latest model at it. Like, there's a, there's a balance between like, costs, performance for specific tasks. Speed is a huge one. Like, we're doing like AI agents. Like, you have to respond fast to users so you can't turn on reasoning max or whatever. And it's going to take five minutes behind the scenes before they ever get a response. So while all these benchmarks maybe are tied to whatever the max reasoning setting is for that model, and that's what people are looking at internally, we're trying to figure out, okay, what are the actual our internal benchmarks on? None or low reasoning or medium, Both speed and answer quality. So we're balancing cost, speed, quality, all those things kind of in one thing. So it's tough, it's very different. And the models that get me excited was the GPT6, Luna and Sol release the other day.

Speaker AYeah.

Speaker BAnd meanwhile, everyone's like talking about whatever the next biggest one, and that's what they're trying in their coding agent. But for me, I'm like, Luna was already, that was like the miraculous model

Speaker Afor us, like, because it was fast and cheap.

Speaker BYeah. So fast, so cheap. They had just dropped the price 80%, I think, last month. And we actually passed that on to our customers, which is amazing that we could do that. So their bots all got smarter and using the same amount of credits or whatever. And then just the other day they did Luna, you know, distillation of astra, essentially. So GPT6, Luna, it's substantially better and it's half the price again, you know, so that's really our workhorse. So we're defaulting everyone to that, even internally making some decisions where we just secretly route things through that, you know, or force upgrade people from more expensive poor models.

Speaker AInteresting. Interesting. Yeah, I, I, you know, I, I think we all for myself at least try to be pretty agnostic. And the tools that I use, the harness and the model, I think it's important because even though I don't do a ton of development, I do a bit, but I do mostly marketing and knowledge work and stuff. It changes dramatically. I was a huge Opus fan for a long time up until three, four, five months ago. And I was just like, I just like I can't, I can't get what I want out of this. This is crazy for me. A long time. Codex was really, really good. I still think Codex is really good. The newest model is. The newest Opus model is very, very good. But it's crazy that it's just such a moving target. Like you just can't stay married to it. Even like as a tool I use when I'm sitting at the computer to do stuff, like I just can't, I can't, I can't be married to one tool because like tomorrow it could be, it could be very different. And I do think there's this like degradation over time. Like it comes out and it's amazing. And then it gets worse as they're I think ramping up to release a new model and that. It happens for both, both companies, I think.

Speaker BYeah, it could be that. And sometimes older ones like you mentioned are, are better. So internally we still use like some of our kind of bulk things. Behind the scenes we're using Gemini like 2.5 flashlight, you know, because still like the quality for what we're doing with it like extraction and the price, like there isn't anything comparable in newer model families, you know. Yeah, they just have gone up in cost and power. But there's not those super dirt cheap like ones that are great for those kind of bulk deals, you know. So even like we have some that are still using I think GPT5 nano, you know.

Speaker AAre you guys using any local models?

Speaker BNo, I mean I toy around with them on my Mac.

Speaker AYeah.

Speaker BBut we haven't found any like practical use case. Yeah, we do use some specialized stuff. So we're doing RAG essentially which is retrieval augmented generations. So users upload their docs and then we're extracting that, like finding the best way to extract that, insert it into vector databases, generate like sparse and deep embeddings, figure out all the search logic and things like that. So there, there's surprisingly been very little progress in embedding models over the last few years. Yeah, some of them are actually quite old now like in AI terms. So that's kind of Interesting that there hasn't been improvements there there. We do use a various suite of embedding models. So for example, if a bot is like foreign language then we'll use like a different embedding model than like OpenAI. We'll use like Cohere because that has like better for other languages and things like that. So we're trying a lot of different things and even like extractions. So we recently added obviously your bot learning from your documents is a huge part of our product. So PDFs was always a struggle. So we've always had our own internal extractor but that would fail. Images or complex PDFs are the absolute worst data source for AI there is because that's fully visual. Fully visual layout dependent. You have tables, you have weird things. You know, it's, it can be really tough. So we're using Mistral for that now.

Speaker AOh, interesting. Okay.

Speaker BYeah, because they have like the best and they have their own small model that's trained purely on document extraction. You know, it's not cheap, it's, it's pretty expensive. You know you're paying per page of PDF. So we pass that on as like higher as using their AI credits, you know, if people choose to that they need that for like more advanced PDFs and spec sheets and things like that. Yeah, yeah, definitely use everything out there.

Speaker AYeah, that's. I mean that. No, I think, I think that's smart. I mean like we were talking about this in our little team meeting. Like we have a huddle on Thursdays today is like I play around with local models a little bit and I was like, hey, yeah, like Quinn, Quinn 3.8 is really good for like a lot of stuff that we all probably do. Like it's probably plenty good. I was like, I think there's probably a world pretty soon where there'll be like much smaller, much more specialized models for, for marketing for hey, we write PHP code. Like there's. Is there a model just for php? Because like it doesn't need to know Rust and it doesn't need to know who the Queen of England was in 1935. And you know, it doesn't need to know all that. It just needs to know how to write really good PHP code. And I think that's probably where local will like really take hold is like what if it can run on like, you know, my 8 year old M1 MacBook Air, like do everything I want it to do. That's probably what's going to happen.

Speaker BYeah. Our problem, like experiment to that like Obviously local means it's running local. That doesn't really work like for software that's being consumed by other users, you know.

Speaker AYeah, right.

Speaker BSo that's like a challenge. Yes. If it's an open model, that means that it could be hosted in the cloud somewhere. But from all our research, like in general, like where those models, those open models are hosted for the most part, like they're not as reliable. Their costs often end up still being more than just using like OpenAI or Cloud or something like that. Just because they don't have the same economies of scale and because those models are changing so often as well.

Speaker ASo interesting.

Speaker BI mean, there's going to be Quinn 3.9 soon enough. Right. And they're going to stop serving 3.8.

Speaker AYeah.

Speaker BSo that's kind of a challenge there, I think, where, for like a B2B, like SAS, where that comes and useful is if you have like security. Privacy conscious, like companies are like, oh, we want to host our own inference, you know, and we haven't figured out how to do that and let them connect DocSpot to that. It's just no one's been at that complexity yet of our customers. A lot of them ask for it if that's an option. It's like, well, do you really Want to pay 20 grand to rent your own private GPU cluster or whatever? I mean, sure, you can do that. But

Speaker Ayeah, I think those barriers will continue to come down for the regulated industries. They'll just. People will accept, it's just like email and there'll be a good secure way to do it.

Speaker BYeah.

Speaker AOver time, I think.

Speaker BYeah. Speaking of small models too, we do use one. This is kind of cool use case. So for privacy, there's one that's open one released I think by the US Government actually, or some division or something. And it's basically a tiny model that's trained on PII redaction. So we actually have that in our chat widget because a common worry is like privacy, like how do you control what users enter in to the chat widget? And so we actually have that embedded in our chat widget and if you turn that feature on, it loads that tiny model in the user's browser and that tiny model looks at whatever the user types in and if it finds like a name or a phone number or a Social Security number or credit

Speaker Acard number, it sends it.

Speaker BYeah. Before it even leaves their browser, which is really.

Speaker AThat's cool.

Speaker BYeah. And we do that transparently. So it sends like a placeholder, like credit Card number one. And that's all you see on our side. But the AI is smart enough to know, like, if they ask, oh, what is my credit card number? It'll reply with that placeholder, which gets dynamically replaced in their browser. So they don't even. They don't even see that it was redacted. You know, that's cool.

Speaker AThat's cool. Last question. Like, what. What are you excited about? Like, I think we're all in uncertain times, right? There's, like, a lot with, like, you know, customer acquisition and, you know, demand for software with AI and all this kind of stuff. What are you looking at? And you're like, whoa, that's awesome. I can't wait for this thing to come to fruition.

Speaker BOh, gosh, this. I feel like we're always behind, right? We're cutting edge probably compared to most of the population, but there's so many things I want to try. So, yeah, like, I know JEV was all the rage. The new Jev.

Speaker AYeah.

Speaker BClassifier model. Last week, so I finally ran some benchmarks. Like, in our use case, the best idea that I have for it was using it for re ranking. So in retrieval augmented generation, essentially, we're doing a search based on the user question. Or in our case, we do anywhere from two to five parallel searches of different phrasings, and then we combine all those results and then we try to figure out what's most relevant and we cut it off. And then we only return, like, the top six chunks to the AI to answer from. Right. And that can be hard when you have, like, really complex questions that go across multiple chunks. So JEV could be super useful to where we search when we pull in the top 100 results. And then Jeff looks at them all and says, okay, it gives them each a score. Which one is the most relevant to this query, and then that way we can return only the most relevant things and hopefully get better results. So I was just benchmarking that last night, and there's been the RE ranking models before. Like, Cohere has one, but it's super expensive. Like, it would add like a few cents per per request to us, which hasn't been enough of a quality improvement to, like, try to pass that cost on to users. But this is at least a 10x factor cheaper. In fact, it may save money in that because you're returning less context to the model, so you're using less tokens on that side.

Speaker AYeah, yeah, yeah. I think JEV was the first time in a long time there's been like, a fundamental paradigm shift in the models. It's not like, oh, it's smarter or better, faster. It's like, whoa. It's like, super different in just, like, what it works and how it returns. Yeah. So I think, yeah, that kind of thing will be. Will be interesting. Different paradigm shifts like that. And. And I think we just. We. There will be people who are able to think outside the box like that going forward, and probably not the frontier model companies. Like, probably. It. It has to be somebody from kind of outside the box, I think.

Speaker BYeah, yeah, yeah. So that's one thing I was excited about. And then I mentioned Luna and Saul 6.

Speaker AYeah.

Speaker BI feel nerdy being excited about these things. I know Jev went viral, but I still feel like no one knows, like, what it's actually for, you know, it's for very specific use cases, you know?

Speaker ARight.

Speaker BAnd it's a great paradigm shift for those use cases, but yeah. Yeah.

Speaker ACool, buddy. Well, it's good to catch up. Really, like, really love the tool. Thank you. Thank you for building it. We'll have a link to Dogspot in the description below and on Twitter. Your. Your Twitter handle is.

Speaker BIs at Ugly robot. So.

Speaker AOkay. Love it. All one word.

Speaker BUgly robot. Yeah.

Speaker AOkay. Awesome. Aaron, thanks so much for joining. Appreciate it.

Speaker BYes, thank you.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Can You Vibe Code a Secure App?

    Nov 19, 202655 minAudio

    Craig Hewitt and Colin catch up after a stressful week. Castos was taken offline on purpose for about half a day while Craig's team implemented security patches for two large PHP vulnerabilities, and the two talk about...

    Credit the people on this episode

  2. Why Enterprise Deals Die w/ Marc Thomas

    Nov 11, 202648 minAudio

    Craig Hewitt sits down with Marc Thomas, a consultant who works on lifecycle marketing, to talk about what it feels like to run a business while AI changes the ground under everyone's feet. Both swing between optimism...

    Credit the people on this episode

  3. Your Product Is Now a Wrapper Around Go-To-Market | Hiten Shah

    Oct 13, 202651 minAudio

    Hiten Shah has built Crazy Egg, KISSmetrics, and a long list of products - and right now he’s shipping a new batch of AI tools while stepping back into Crazy Egg after nearly 20 years. In this conversation we get into...

    Credit the people on this episode

  4. From Audience To Income

    Sep 29, 202657 minAudio

    What does it take to turn an audience into a business - and how do you do it without losing the enjoyment that got you started? This week on Rogue Startups, Craig and Colin compare notes on building creator businesses....

    Credit the people on this episode

  5. RS368: Grok Bot & Hermes

    Sep 22, 202650 minAudio

    AI agents are getting a lot more capable, but are they actually making work easier yet? Craig and Colin compare their latest experiments with GrokBot, Hermes, OpenClaw, Claude Code, Codex, and other AI tools, including...

    Credit the people on this episode