Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Semi Doped

Semi Doped artwork

Every Muse User Gets 2 CPUs and 8GB in Meta's Cloud. Your Phone Already Has That

Oct 1, 202651 minAudio
ShareXLinkedInFacebookWhatsAppEmail

The push to run AI agents on edge devices will not be driven by model labs or hardware vendors, but by service providers like Meta whose ad-based business models create a powerful financial incentive to offload massive cloud compute costs onto the user's own powerful-but-idle hardware. Austin Lyons and Vik Sekar break down why the failed "AI PC" push has given way to a new, economically-driven strategy for on-device AI. They discuss how Qualcomm's acquisition of Modular provides the key software layer and why the ultimate vision is an "ecosystem of you" where the device acts as a context-aware orchestrator between local and cloud compute. This episode is presented by Crusoe. Try Crusoe's managed inference for open models with $5 in free credits: https://semidoped.com/crusoe "[Model Labs] business model is we make money when we do inference in the cloud. And yet we are saying, actually, there's a world where we would really like to just run this stuff locally... Meta's like, bro, I want to help you at the edge." - Austin Lyons, Chipstrat Key Takeaways: - The driver for edge AI is a business model conflict. Model labs like OpenAI sell cloud inference; service providers like Meta see it as a massive cost center for their ad-supported agents like Muse. - Your phone is an underutilized edge server. Its specs - like 8GB of RAM and 2 vCPUs of power - are comparable to the dedicated cloud VMs that services like Meta's Muse provision for each user. - The "AIPC" push failed because it was a hardware-first strategy without a use case. Today's edge AI push is a software-first strategy driven by the need to cut cloud OpEx for mass-market agents. - The role of the edge device is becoming a "context translation layer." It uses private data like location and biometrics to intelligently decide which AI tasks run locally versus in the cloud. - Qualcomm's acquisition of Modular is a key strategic move. It positions the open-source Mojo language as the "write once, run anywhere" software layer for deploying agents across diverse hardware. - Personal agents will lead to the "end of apps." Fixed UIs, which force a designer's paradigm on the user, will be replaced by interfaces generated on demand via natural language. - Google is missing its opportunity to lead in on-device AI. Despite vertical integration from TPUs to Pixel hardware and a leading ad business, it lacks the product execution of players like Meta. Chapters: 0:00 Who Makes Local AI Happen? 1:25 Making the Intro with AI 4:43 Qualcomm's Agent Vision 6:54 Why AIPCs Failed 8:45 The Phone as a Portal 13:52 Meta's Incentive for Edge AI 24:05 The End of Apps & UIs? 32:31 The Ideal Personal Agent 34:25 The Edge as a Context Layer 38:17 Qualcomm's Modular Acquisition 43:16 The Ecosystem of You 48:01 Google's Missed Opportunity Follow Chipstrat: Newsletter: https://www.chipstrat.com X: https://x.com/chipstrat Follow Vik: Newsletter: https://www.viksnewsletter.com/ X: https://x.com/vikramskr Follow Semi Doped: Get more of Austin and Vik daily, free: https://daily.semidoped.com/

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_00Last week, Qualcomm said agents will run on your phone, your glasses, and your earbuds. But chip companies don't decide where inference runs, app companies do. And the biggest app companies of our time right now, OpenAI and Anthropic, well, they only get paid when it runs in their cloud. So who will actually make local AI happen? Maybe the company that makes its money on ads and already ships Snapdragon in its glasses, like Meta. Or Google, but do they have the Moxie?

SPEAKER_00Let's get into it. Hello everyone, I'm Austin Lyons. Welcome to another semiduck episode. I write Chipstrat, a semiconductor, an AI infrastructure newsletter for investors and tech executives.

SPEAKER_01Hey, I'm Vic Shaker. I've worked in the semiconductor industry for about 20 years, mostly on engineering. I'm the founder of Semi-Exponent, where I do independent semiconductor research, mostly focused on AI and infrastructure. And I also write Vic's newsletter on Substack, where I just explain how things work and what matters to the markets. So that's what we bring to the newsletter. Check us out on our Substacks where we write like a whole lot of stuff every week.

SPEAKER_00Yes, totally. I know you wrote some stuff last week, but let's not talk about that. Let's talk about our new intro. Tell me about our sick new intro.

SPEAKER_01Yeah, that was uh by the way, uh all done with um Opus 5.5. And you know, uh I we have our podcast brand document, and all I did was I like I gave it a brand document uh PDF and was like, just make me a motion graphics intro. And our branding document has basically the theme it's based off of, like uh the field and the single point of singularity. And our designer put in these like like phrases in there, so it just used that and built the whole thing out, and I was like, ah, this is insane.

SPEAKER_01And I told it like a few other things, like make it six seconds, make it like 30 seconds, and we I tried different things, and I was like, wait, wait, the logo stays for too long, make the logo stay for lesser time, make the animation slower. It's insane. That's that's what it came up with. That's pretty crazy. And then uh I used uh uh you know Suno AI, which is like a music generation thing. Uh, and if you get the paid subscription uh to the tool, you can actually generate commercial uh music, you can use that in podcasts and you know other things to make money off of.

SPEAKER_01Uh, not as not a sponsor, but they could very well be, I suppose. Uh, really, like it's just so so cool. Like, I keep kept trying different things. I said, hey, make some like jazz fusion. I'm like, nah, this sounds like elevator music. And I'm like, I don't know, like I want something exciting. And I told it, like, build it up and like give it something in the beginning that's repetitive, but like I want a climax finish and then goes to slower things where you know we can talk over it and things like that. I don't know, it all works just fine. And then our editor took it takes it and yeah, you know, runs with it or uh with with the magic there.

SPEAKER_01So all this is really nice. AI.

SPEAKER_00Pro AI. Dude, so so good, so amazing. I love that the the Suno, like you still had to have taste. Like you're able to use Suno to generate the music, which is incredible, um, and and really speed things up as opposed to, you know, what would you have to do previously? Contact someone who knows how to do this, or figure out how to make music yourself, you know, with your computer. Um, but you had the taste to say, oh, you know, here's what I'm listening for. I need it to build, I want there to be a hook, I need it to be catchy.

SPEAKER_01Yeah, so yeah, clearly I am a musician and like I have recorded music and I kind of know what to say to it. So that little bit of domain expertise actually helps uh to tell it what to do. Otherwise, it's very impressive. I think most people can do something with it.

SPEAKER_00Awesome. I love it. Okay, so let's talk about what else is new in this episode. Um, so we have a presenting sponsor. Yes, awesome.

SPEAKER_01We have Crusoe. This is like our first episode with a presenting sponsor. We're very excited.

SPEAKER_00Very excited, really uh stoked to partner with them and hang tight, and you'll hear our first ad read and you'll get to learn more about them. But uh, it's gonna be fun and you're gonna learn a lot about them. There's so much to learn about Crusoe, and um we're gonna do fun ad reads every single week, and they'll be different. So stay tuned, listen to them, and check them out. And you know, they sat down with us and we spent a lot of time talking about all their value props and their interesting technology from top to bottom. So you'll get to learn about that too over the next couple months. Um, but with that, let's let's get into it.

SPEAKER_00So I was at Snapdragon Summit last week in Maui, in Hawaii. It was awesome. There was a tropical storm incoming, which I didn't even know, but some other analysts are like, hey, my like my mom or my wife called and said they saw on the news that there was a tropical storm that's gonna hit Hawaii. I was like, oh man, really, I'm just too busy enjoying the beach to even notice. But we got we got out before it it was a problem. Um, but yeah, let's talk about Qualcomm. So I think that, you know, of course, everyone may have seen uh I did an interview with Durgo, which was really interesting and talked about high bandwidth compute.

SPEAKER_00And that was mostly because Durgo was there and I was super excited to talk about it. Um, but the name of the game for Snapdragon Summit, my takeaway was Qualcomm really talking about a vision for agents running anywhere, and the fact that um their Snapdragon SOC, they have different sort of flavors of it that can run in watches, in augmented reality glasses, uh on your phone, on your car, in your laptop. And so naturally, Qualcomm as a hardware vendor, their vision is agents that can run anywhere.

SPEAKER_01Some time ago, if you had told me like agents are going to run on the edge, I think the perception I would have had is LLMs are too big. Like, how are you going to run a frontier like LLM that is intelligent on your phone? And clearly we will never have, I don't know, like a terabyte of memory on the phone to run uh a trillion parameter model. And clearly scaling laws still hold, which means the bigger you make a model in terms of its parameters, the smarter it seems to get.

SPEAKER_01And it still haven't, we still haven't seen the end of that. So I would have laughed at the idea of what's no way we can have that running on our phones, right? But in talking with you just before the show, and also uh reading a little bit about this stuff and the event uh that Qualcomm just had, it seems like there is a different way to think about it. Um, agents could be a portal into our world of AI, and we'll clarify what that means. It's very interesting.

SPEAKER_00Yeah. You know, you make a really interesting point, which like go back a couple years. Uh, remember like Intel, uh Pat Gelsinger, he talked a lot about AI PCs, and there's this big push. Yeah, we're gonna have these AI PCs. They've got NPUs, and everyone's like, oh, how many tops does your NPU have? Mine has 35, mine has 45, mine has 55. And it was all about like um the idea was running AI locally. But at the time, it was LLMs. And first of all, there were barely any open source LLMs when that started. Um, and then you know, when they came out, it's a 70 billion parameter and it barely fits on your laptop, and your NPU really can't chug along and do too much.

SPEAKER_00And so everyone's like, dude, what are we doing here? Like this, this is gonna be slow. What's the use case? Um, and ultimately, AI PCs were kind of a dud. Everyone just ran their AI in the cloud. Um, and I think now we are getting closer to a world where there will be useful AI and we can get into this that could run at the edge, and it doesn't have to be an LLM, it doesn't have to be a frontier trillion parameter LLM. Um, but we can get into that. I will say uh, you know, we're we're still in this transitory world where, and and we'll get into this, where you can see a world that will run AI at the edge, and yet there's still talk about like, oh yeah, we we can run all of this, uh agentic AI, for example, cloud and code codex and you know, open cloud, perplexity, you uh, you know, and a lot of that is still running in the cloud.

SPEAKER_00So there's there's still a little bit of, you know, we're envisioning a world to come, but we're still talking about apps that actually run in the cloud. But um, you know, obviously I think Muse comes to mind uh as a truly agentic AI, and and then where that could go is pretty interesting. But yeah, I'll hand it over to you.

SPEAKER_01The app thing is very interesting. So recently I have gotten uh very much, let's say, AI billed to say, not that I wasn't already, but the reason is given the Substack, and now I'm you know running, I've been talking to prospective clients about semi-exponent and institutional research for financial firms and things like that, and then we have the podcast, right? And then we have so many like podcast briefings and recordings and back and forth of documents. It's a large universe that we deal with, really, you know, and so I have been thinking this needs to be more agentic AI for me.

SPEAKER_01I I really need the help. Like uh, I could go off and hire people and all that, but I was like, okay, let me see what parts of this, it's not like human taste isn't important, it is, and we do have people who work with us on the podcast. We do need people. I'm not saying we're gonna replace everything, but I'm like, just let's what can I what can AI take care of for me? Because there's like a minimum requirement for some things, the MVP in my life. What can it do? So the app itself, the phone has become a very big part of my day.

SPEAKER_01And the reason is I have Chat GPT, I have I have Claude too, and this happens both places really. But in ChatGPT, I have now scheduled tasks. So what it does is like every morning it it reads all the emails, and because I'm on a different time zone, it gives me a briefing as soon as I wake up in the morning. So when I'm making coffee, uh literally chat GPT reads out to me like whatever has come to me, or or I read it out or whatever. It doesn't matter, but it's all in one place. It's become an integral part of my morning. And so, yes, AI doesn't run on my device, but it's become a portal for me to handle tasks during the day.

SPEAKER_01And this is kind of what Qualcomm has been saying, uh, except that their view is that a lot more inference runs on the device. Maybe we are not just there yet, but I can see where we are headed because it's become so important for me that every hour, right, I don't want to miss emails and all that, because some emails require me to respond immediately. So I have another schedule task that uh you know reads my emails and tells me every hour. So a notification comes up on my phone. Hey, somebody is asking you to do this. Here's another great example.

SPEAKER_01I messed up and I always get time zones wrong. And anybody who's like worked with me on this, I always get rescheduling invites full of the calendars, and I always get, I can't do math in my head, I guess. Um it it sent me a notification. It says, hey, your email says that you're gonna meet with this person at this time, but your scheduled uh you know meeting is one hour late. Are you sure this is what you want to do? Like, or do you want to move it? I'm like, oh my god. Like I told ChatGPT, like, remember to put that on my to-do list and put it on my task calendar because it's hooked up to Google, right?

SPEAKER_01And so it made it uh, you know, it's put on my to-do list and I, you know, I just have to go and send, you know, reschedule it. I could ask it to do that too, but I'm just not interfacing with other people yet because I don't want it to like keep sending stuff to other people. Could be done uh automatically, but you see where this is going, right? Like I know inference doesn't run on the device yet, but on the cloud, it's still useful. Edge devices are being useful to me already.

SPEAKER_00Yeah, yeah, I love that use case. Good job, ChatGPT, by the way, for keeping things straight and figuring it out.

SPEAKER_01I wouldn't even have showed up to this podcast on time. I would have been one hour late. That's hilarious. This episode of Semidoke is presented by Crusoe. Crusoe builds AI infrastructure from the ground up. Power, data centers, and the cloud on top. Crusoe Intelligence Foundry is their managed AI platform. And a nice feature in there is serverless inference. Now, if you're going what in Jensen's name is that, it means that there are no GPUs for you to provision and no clusters to manage.

SPEAKER_01You simply pay per token. If you're running open models like GLM, Deep Seek, Kimi, or Quen, Crusoe's managed inference is fast, nearly 10x faster, time to first token, and 5x the throughput of VLLM. Their memory alloy caching layer routes requests to maximize cache hits. Since it works with OpenAI SDK, switching over to Crusoe's API is quick and easy to get it running. All you have to do is pick a model, grab an API key, and start sending requests.

SPEAKER_01If you sign up with the link in the show notes, you get$5 in free credits to try Crusoe Intelligence Foundry. Think of it, five whole bucks, so you can just burn some tokens just for fun. And some of these open models are quite inexpensive, so you'll actually have a ton of fun with it. Check out Crusoe and see their serverless inference on their managed AI platform. You really got to find out this for yourself. There are a whole lot of more features in this that I want to try out for myself, but serverless inference I've found is the best place to get started.

SPEAKER_01It's a great way to support the show and try out Crusoe's Intelligence Foundry. The link is in the show notes, and now back to the show.

SPEAKER_00Previously, when everything was just running ChatGPT, chatting with ChatGPT, or chatting with Codex, still everyone is still focused on like, oh, it's gotta be Fable, it's gotta be the Frontier model. Um, what I think has been really interesting lately is Muse and Instinct, like we've talked about, Grok by, and these agents where you realize one, I could see this going to actual like normal consumer adoption because it's so simple to use. You know, you can there's billions of people that have um Meta's family of apps installed, and it's very simple for them to get up and running with Muse.

SPEAKER_00And um, by the way, when they when they sign up for Muse, they get a dedicated uh, what is it, like two CPUs, a dedicated cloud VPM. It's like two CPUs, eight gigabytes of RAM. Um, I think I even saw maybe like a hundred gigabyte of storage, or uh, maybe that was Tom's hardware speculating or something, but clearly like quite a beefy computer for every single person. And and so on the one hand, this is like crazy powerful that all these people can spin up agents that can actually go do useful stuff. And by the way, um, my use case for Muse, because I was like, okay, I want to try this out, what am I gonna do?

SPEAKER_00And I've got all this like electronics and old recording equipment and stuff like this sitting just next to me in my office, and it kills my wife because she's kind of a minimalist, and she's like, this junk is sitting here, do something about it. I'm like, literally, I she's like, it has so much value, it's just sitting there, just selling on Facebook Marketplace, you know? I'm like, I just don't have time. I just don't have time to take pictures, write out all the listings, look up how much everything should cost. And then I was like, wait a minute, this is perfect for muse. I'm just gonna take pictures of everything and tell it, bro, just post this on Facebook Marketplace.

SPEAKER_00I don't care what the price is, you pick a good price, you you make a nice listing, and I just took, gave it a bunch of pictures, and it actually did an awesome job. It like looked up what everything was called and wrote like really great descriptions, figured out the prices and posted it and everything. And so I'm like, dude, that is like truly useful work for me. And I even may make some money out of it. And but even more importantly, of course, I'm gonna make my wife happy. Uh, so you know, happy wife, happy life. Um, but you know, here's the interesting thing. The more so I was like, oh man, I'm I'm using those uh, you know, all this stuff that was provisioned for me in the cloud, but then I stopped using it.

SPEAKER_00And so I'm like, is it just sitting there? Are they sharing with other people? But I was thinking, like, man, if I get this just cranking all the time for me, Muse is so handy, so convenient because it's on my phone. And so I was actually checking it when I was in Hawaii, you know, people are responding, like, hey, can I buy this? So I'll tell Muse, like, tell everyone, like, no, I'll get back to you when I'm back from vacation, but don't tell them I'm gone. I don't want them to rob my house, you know. So just tell them, like, I'll get back to you this weekend, you know. And so Muse was doing that for me. And I just thought, like, again, hey, some of that stuff, it's here's my phone right here.

SPEAKER_00And it's pretty simple. Like, they're paying a lot of money in the cloud to reserve this infrastructure and use it for me. But like, couldn't some of this very simple stuff, like sending someone a message, like, couldn't that actually happen on my phone? So it just got me thinking, like, hey, maybe app companies who end up spending a ton of money in the cloud, like maybe Meta, maybe they will be the ones who are incentivized to actually push a model down to my phone and try to orchestrate and have some of it run locally when it can.

SPEAKER_01Yeah, that's very interesting. That the fact that, you know, you mentioned that they everybody gets a virtual machine in some cloud instance somewhere. Actually, I sat down uh the Substack post this week was all about estimating how many CPUs are going to be deployed, assuming, I don't know, a billion uh VMs being in the cloud. Like, like how many CPUs do you need? Because I think there's like a lot of like excitement about CPU requirements. So I calculated all this out. It's very interesting actually, because people can share CPUs.

SPEAKER_01You know, everybody doesn't need to get a dedicated VM. Like, you know, when it's not posting your uh Facebook marketplace ad, like it could go and uh help Vic like get his calendar right, right? Like the same CPUs. Why should it like just chill out? So this kind of stuff is there. So the second thing is like what we don't have yet, I think, is uh we've got 8 GB of RAM on this thing, right? And we've got uh some computing power, like two V CPUs worth of computing powers right here.

SPEAKER_01We've got some storage. I mean, I don't know how much is in this thing, maybe not a hundred gigabyte, like I need space for apps, or do I? We'll get to that soon. Do I really need apps anymore? Is a big question. So, yeah, like although there's like all the CPU deployments on the cloud and everybody's excited about the prospect of CPU, I'm like, look, my phone is already my portal to the AI world. It's happening right now, it keeps buzzing about all the scheduled tasks I have. You're putting up stuff for sale, everybody's happy, things are getting done, you're making money.

SPEAKER_01But then what's my CPU doing here? Nothing. So I'm like, why don't we now start using this as our edge computing device where your two VMs and 8 GB of RAM live so that you know you can just talk to this device and it has everything, no, it knows everything about you. It knows where you are, it knows how much how much you walked in a day, it could even listen to you, it knows who you're talking to. Uh, all your devices connect to this, like your watch can connect to it, your you know, your your app airpods can connect to it.

SPEAKER_01Everything is phone-centric. So we have to reimagine the AI on the edge world in this framing.

SPEAKER_00Yes, yes, I totally agree. It and it's all about all that context, too. And that's kind of the point you're making. It's like, uh you you've got your watch, it's got your health data, your steps data. Maybe for me it logs my Strava, whatever. Um, you've got your microphones, you've got your phone, which has not only, of course, your calendar and whatever, but it's got like the pictures that you've taken. So there's all this just local context on your watch and your glasses and your earbuds and your phone. And so again, um, the idea, let's let's say it's just Claude and OpenAI, the idea that you would take everything local, send it up to the cloud, have it compute in the cloud, send it back down, feels a little crazy.

SPEAKER_00And of course, in their world, and remember, uh, OpenAI and Claude, or OpenAI and Anthropic, these are like two of the fastest growing companies ever, great companies, uh, amazing app companies, whether they meant to become that or not in the first place, because they're both kind of research companies. Um, they their business model is we make money when we do inference in the cloud. And yet we are saying, actually, there's a world where we would really like to just run this stuff locally because all of our context is here. You could have, of course, there's privacy if you can just filter through things and take only what you need here and not don't send up all my photos to the cloud, don't send all this information to the cloud just so that you can compute it there and send it back down.

SPEAKER_00Like, why not do it here? And what's interesting again is Meta stands out as a company who their business model is ads. It's they just want to help me and have the opportunity to show me ads and just be part of my life, essentially. Now, whether you like ads or not is a whole nother conversation. I think it can be useful. Um, but there's a disincentive, of course, for the model labs to want to come help me at the edge. But Meta's like, bro, I want to help you at the edge. You've already got our apps installed. Like, let me help you.

SPEAKER_00And so Meta has already made open models. Meta has um, I think they released Muse Glimmer, which is in this uh direction of running things locally, orchestrating, running on the edge. So it's very natural actually for Meta to say, hey, I've made Muse. It's super helpful for you, it's really expensive. You've got this um cloud or this this little VM here with these CPUs and the eight gigs of memory, which is a very great point. It's just sitting idle. We can give you more Muse if we can just run some of it locally.

SPEAKER_00Like imagine I hit my token limit and they're like, yeah, didn't that suck? We can make your tokens go farther. Let us run some, will you give us permission to run it on your phone when possible? And I would say, totally. And this doesn't even necessarily mean that it has to be the Stuff like you said that requires that sort of system to deep thinking, deep research. You need a frontier model. There's lots of stuff that can be done at the edge that aligns with just helping my little agent making some quick decisions and helping me with my life, looking through my context or whatever, before deciding what needs to go to the cloud.

SPEAKER_01Yeah. So that's the that's the whole point. Because look at the like the app, like you mentioned the Strava thing. Um, and you you run a lot and uh you keep track of your steps. You even posted on X and all that, and you were like, on when you were in Hawaii, you were like, look at this, I have 20,000 steps today, you know, all that stuff. Clearly, it all runs on an app, but do you really need that app? You know, there's a fitness app sitting on your home screen occupying like the space, and you have to find where it is, or you have to type fitness, and then you have to click on the app and it shows you the steps.

SPEAKER_01Like, why do you need that? All you have to do is ask it how many steps, and it it has a screen. That's all it needs. It's just a screen. Everything that happens on a phone can just happen. You don't need an app for it. Like, and this is a point that Ben Thompson recently made on Strategicry, uh, which if we are both big fans of that newsletter, he's like the OG newsletter guy. I think at some point everybody who started a newsletter wanted to be like Ben. So that's exactly so he made the point that uh there is no need for apps anymore because all everything can be built and destroyed on demand.

SPEAKER_01It's very simple to do. Anybody can like build an app for their own use cases and uh change it out if they don't want to. Some things may not even need an app, like this fitness example, right? So, where are we going in in this world of like devices? Like, what do devices need to have? Do we need to think of them as computers? Or what is its function in the world of AI? Are really deep questions that we have to think about because that defines what edge AI is.

SPEAKER_01Another point that Ben made, I want to mention this and I'll ask you what you think because it's a very important point. He's like, look, now there is no reason for anybody to code. There's like literally no purpose. It is a syntax that nobody understands with this like braces and like colons and like indenting or whatever. We don't need to deal with that mess, okay? Like you just ask it, it writes the code, it gives you, you don't worry what happens in the background. Now we have news and we have these CPUs and all that. These agents can even control your computer.

SPEAKER_01Like Chat GPT has a computer use. So now agents control your computer too. Like, so you don't need to even use your computer. The computer is meant for your AI to use, not for you. Right? So you see where I'm going with this. Yeah, yeah, yeah. So, what is the device supposed to do now?

SPEAKER_00Oh, good. Okay, so this is a great setup. I have lots of thought. So the idea that we don't need to code, I would say baloney to that, although many people don't need to learn. But and and by the way, I got that. Um, Ben lives in Wisconsin right now, and I was listening to this guy, Roz how I built this with the guy who started Culvers, which is like a, you know, if anyone knows, it's like a fast food restaurant. It started in Wisconsin, and it's so funny because the guy's so Wisconsin, and instead of swearing on the episode when he was like really angry, he said, baloney to that. And I was like, Oh, that's so good.

SPEAKER_00I have to use that. I wanted to Yeah, thank you. So thank you. I wanted to swear, but I, you know, I don't want to get that explicit rating on our podcast with our sponsor. Um, but okay, so as you said earlier, why could you make such a great little hook and little jingle with AI using Suno? It's because you had domain expertise, because you've made music manually the hard way, therefore you knew how to prompt the AI. I think obviously, with the best coding, um it's still going to be people who have written code and understand how to uh write good code and what good code looks like.

SPEAKER_00And actually, and we'll get into this a little bit later. There's this new model that came out recently, Jev. And I won't get into it right now because then I'll get too distracted from the point that you were trying to make. Um but Jev, what's really cool is it can you can use it to take in like the world's input. Um so you could give it freeform text, or you could give it a chatbot conversation or a phone call or whatever, and then you can have it do simple things like classification, like, oh, here's what I'm a bank, here's what the person said.

SPEAKER_00Should it go to fraud department? Should it go to the payment department, whatever? It's literally just like a little, these are little like classifiers or little switch statements or if statements, if you will. Um, but it can work crazy fast, and it's trained like an LLM on the on the world's um the internet's corpus. Um, so it's very intelligent. And anyway, these these are like new programming constructs that will actually go into code. And if you like zoom out and you think like, wait a minute, today we're in a world where the LLM just writes all the code end to end. But if you actually try to stick an LLM call in your code, it gets really funky because these LLM calls are like, they take a long time, potentially, they're not very bounded, they're not typed, so you don't quite know what's going to come back.

SPEAKER_00It's just a string that comes back and you have to parse it and figure out. And if you even if you ask for structured output, and so it comes back as like a JSON or something, it's still like very slow and it's like inefficient. So, anyway, I say all this to say there's going to actually be new ways of programming, which the models aren't even trained on yet. So I think there's all sorts of reasons why having domain experience as a programmer, sure, for the Ben Thompsons, they don't need to learn how to code. Um, but for folks like you and I, like knowing how to write RTL or knowing how to code will help you obviously be able to prompt it better.

SPEAKER_00And then again, as we continue to even invent new ways of programming, someone's got to prompt the AI to do that, to train it on it. But to the question of do we even need apps? So, yes, we need to program. We need people who have domain experience in programming so they can they can do it better and more efficiently. Do we need apps? Okay, this is actually really interesting, and I won't bear the lead. I agree with you that we don't need apps, and let's think from first principles. Why do we have user interfaces in the first place? At the end of the day, let's take Excel for an example. Dude, at the end of the day, you could just have a database and a prompt and like store data in a database and prompt use SQL and just say, like, hey, uh whatever, I'm trying to um calculate my mortgage.

SPEAKER_00Like you could put, you could either write all this stuff in the command line or you know, put a bunch of data in a database and pull extract it out and make calculations, and everything could be the command line. But obviously, that's like a very difficult thing for people to learn. So we make these user interfaces like Excel, where it's like, oh, it's very visual and you can, you know, um have little if statements and summing and stuff like that. And it's super simple. But the problem with all user interfaces is someone has to decide what the rest of us are going to be forced to use.

SPEAKER_00And that's why um user interface designers are amazing because they're almost like psychologists, they can think really well about like what's the best experience and what are people going to try to do and what are they gonna expect? Are they gonna expect this button and that button? But still, there's that new level, yes, you didn't have to learn how to code, but you had to learn how to use the UI. So you still have to learn this paradigm that's forced upon you. And we've all been in UIs that are terrible where you're just like, dude, what the heck? This why is this button here, or why or whatever? And it just sucks. And ultimately, what that user interface is a slightly simpler way for us to tell a machine what to do, right?

SPEAKER_00And so if it's not telling it with programming, it's telling it with some user interface that someone created. And by the way, it's never perfect. Like, um, for me, I I've got this like um very niche thing, right? I have hearing aids and I want and I can connect them via Bluetooth, but it's not normal Bluetooth, it's this other thing. And for me to access it, I have to go into the accessibility menu on Mac and scroll down and click on this thing and do this pairing, and then there's not like an easy disconnect if I want to switch the headphones. I just have to forget it every time. Super cumbersome. But Apple has given me no other simpler way, and they don't give me the ability to make my own little like app on top of it where I could just, I literally just want to push a button and say connect or disconnect, right?

SPEAKER_00So I'm forced to live inside the paradigm that Apple made. And I hey, I'm glad they made it a couple clicks instead of having to write code, but it still sucks. I would love to just use human language and say, disconnect my hearing aids, right? And I think that's the point you're making, and that Ben Thompson is making too, which is like at the end of the day, we all just want to talk to the machine, to the agent, to the computer, to the phone, tell it what we want, have it give it back. And of course, the beautiful thing, and Ben Thompson talks about this about like generating UI too, is like if it gives it back to you in text or in uh speech, but you're like, I'm visual, create a nice visualization for me, it can just on demand create a visualization for you.

SPEAKER_00And so I, you know, I definitely think that's the world where apps are going, but I think to your point, that can impact like how we interact with things at the edge and make the edge much more interesting if we sort of expand our mind to say, how could we interact with AI? How could we control computers? How could we re-visualize and rethink things?

SPEAKER_01Yeah, so the edge device is inherently a multimodal device, right? It does video, it does text, it does audio, everything. And so technically, you could uh use it in any way that is most useful to you. And you could ask AI to give you the format that suits you best as well. And one thing I'll tell you, I tried to do for this episode. So I was looking through some briefing notes for preparing for this podcast, and I was like, wait, Austin just got back from Hawaii.

SPEAKER_01I'm gonna see if I can make him a MP3 file that you can use while you go on your run in the morning so that you go come back in the morning from your run to the podcast, and you're like, oh, I'm briefed, right? So if you look in the Google Drive folder, you'll actually find an MP3 file there. It didn't really work out, it's like a little bit robotics. I have to figure out how to fix that. But you see, what I was trying to do here is give you a format that will suit you better. So you don't have to sit at that, you know, computer and like read the text like PDF file I created for this podcast, right?

SPEAKER_01You could just go on a run and get your information. And how did that happen with AI, right? I mean, yes, you could you could read out, obviously. You could you have dictation apps that just read stuff to you and they do pretty well too. But like that's another app, okay? Like I just want AI to give you the format you wanted best.

SPEAKER_00Dude, Vic, this is such a great example. Vic is my ideal agent. He knows all about me, he has the context, he knows that I run, he knows that I like to listen to information, he knows that there's something on my calendar called our podcast recording that I need to prepare for. And he went so far as to put everything together in MP3, which I totally didn't even see. My apologies, but I love the heart behind it. Um, but of course, this is the future. And then, of course, maybe someday it's not you doing this for me, but it's an AI, knowing so much about me and then preparing me in the way that best suits me.

SPEAKER_01Yeah, and it's your edge devices that know you the best, right? Your watch knows how long you run. I don't know how long you run, I don't know where you run. Right? That's true. I don't know so many things about it. Like uh I don't have entire context, I have some context because like I generally know what you're up to. You're like, hey, I'm I just got back from my run, I'll see you in a bit, is what you're like texted me before the show. So I generally know what's going on, but your devices know way more than me about you.

SPEAKER_00Yeah, yeah, yeah. That's true. And and that doesn't necessarily mean that it all even has to run at the edge, but they obviously have all that context and could recognize, like, oh, generating this piece of actually generating the MP3, that's big. Go run that in the cloud somewhere. Or, or who knows, man, my phone could probably do it just fine if it's by the way, I'm just running, so it's just chilling, not doing anything. It's fine if it's slow, you know, to process it. Yeah, exactly.

SPEAKER_01So it knows that, hey, the you know, what your phone is not going to be really useful right now. You're not like browsing on it or anything, you're not gaming on it. Uh, maybe you do before you go to bed, or when you want to relax, you just like game on your phone, and that time you're like, okay, let's not run some agentic stuff now. He's gaming, okay? Like Austin usually plays on his phone from like 7 to 7:30 p.m. And maybe then he grabs dinner and goes to bed or whatever. So that information it knows. So the role of the edge device now is a context translation layer.

SPEAKER_01So it takes the inputs it knows about you when you run, what you do, when you play, when you sleep, and then it manages its own workloads and decides what can happen locally. Maybe a simple translation to an MP3 file from text can happen locally. Why does it need a frontier model? It's a it's a rather simple task, right? Or maybe it needs to be in a different language to somebody else. Maybe that translation of a document can happen easily.

SPEAKER_01Those are simple tasks, just run them on the phone. It's not like it's something frontier. I'm not asking you to, you know, you know, find some hard problem to solve. Like maybe it can run on the device and the rest of it, send it to the cloud and decide when to send it to the cloud. Maybe some answers you don't need. Like I have a podcast recording, uh, you know, then in the next few days, I've actually lined up a couple of them. Um, you know, so my agent already knows that this podcast is only three days away. So it doesn't need to make make the briefing document now.

SPEAKER_01It it just needs to handle that in two days. So my edge device can find out when's the best time to get this done. Because all the context lives on it. So that's that's the reimagination of the edge in a sense.

SPEAKER_00Yeah, yeah, I love it. And uh so I'm gonna take it back to that. I mentioned Jev, the sort of fast thinking system one model from TypeSafe AI, which people, if you haven't heard about, you should go look it up, check it out. I wrote an article about it as well this past week. And this type of model is really good for just making fast decisions. And what is a lot of what we're talking about? A lot of it's just little fast decisions, like looking at context, deciding if something needs to be done, looking at the time of day, deciding what needs to be done.

SPEAKER_00Large language models are great at prose. They're writing natural language, they're doing that decode loop over and over and predicting the next token, and it takes a long time. Sometimes you just need a fast decision. Now, the ironic thing is Jev launched their fast thinking model as a cloud offering. I think that's the wrong place. I think it should live on the edge. I think it, but of course, I don't know how they would make money that way. So I think that this might be one of those use cases where they made really good technology and now everyone's gonna take it and go make edge versions. So I hope they get acquired and have, you know, get some sort of upside out of all the good work that they did.

SPEAKER_00Um, but I don't think, again, just making the same point over and over that like it has to be the big um, I take in English and I write back out English, massive models. I think there could be lots of little models that help with all of these little decisions at the edge.

SPEAKER_01Yeah, there are so many modes of operating this edge device that I feel like we haven't actually explored all of them yet. In my own mind, what I'm doing with all these things is very limited context. Like I'm not doing anything dramatically different from what I have done in the past. I am only adapting my existing way of working somehow into the world of AI. But it doesn't have to be that way. There could be a complete reimagination of how we do all of this.

SPEAKER_01Before the show, I was thinking, what do I really need in my laptop? Really? Is it all that I let's assume the bulk of the inference happens in the cloud, just for a minute. Do I even need a keyboard? Like, is a screen with a speaker enough as a device? Like, what is the simplest device that could be useful for an edge device? Like, literally, like a very few inference chips, a screen and a microphone. That's it. And a radio communication device. Don't forget, like it has to be wireless. Nobody wants to be connected with wires.

SPEAKER_01So a radio device, a microphone, a speaker, and a screen. That should take care of everything for you.

SPEAKER_00So good, man. Um, I totally agree. And okay, so that makes me want to go to wearables and to how do you orchestrate across all this, and we'll get there in a second, but I do want to go back. You were kind of breaking things down and going up from first principles, which is like, what is just the basic things that I need? And so you were talking about you're forcing AI into your existing workflow, and you're saying, wait a minute, no, no, I should build my workflow around AI. And that reminds me of two things. One, it reminds me of that transition from the horse-drawn carriage to the automobile, where the automobile was literally a horse-drawn carriage with an engine, basically.

SPEAKER_00It's like it didn't have, it was like open air and it had the wooden wheels and everything. And clearly they kind of like just bolted on an engine. Only much later do we really, you know, reimagine. I mean, you could argue that even it's really like the robo taxis also reimagining things of like, guys, what if it's just a car and you don't even have to drive it and should even have a steering wheel? You know, so there's obviously ways to reimagine something once the technology is there. And then of course it reminds me even also of like when electricity first happened, you know, it's like taking existing um manufacturing workflows and saying, like, oh, how do we stick like some sort of engine or you know, conveyor belt or something in here, instead of reimagining the whole manufacturing line around electricity, right?

SPEAKER_00And so this is what we need to do with AI. Um, this is what we're trying to do with our uh, you know, podcast too, and everything you've talked about, about thinking how do we prep and how do we manage this? You know, is like how do we put AI at the center and build our podcast around of it instead of our the way we've been doing our podcast and inserting AI. Hopefully we'll have more anecdotes to share with people over time, uh, you know, victory stories there. But here's one question. Okay, so you built up um saying, like, hey, really, all I need is like con, I just need context.

SPEAKER_00That's all I really need is be able to give context to the AI and then to somehow receive context back. Maybe it's I hear it, maybe it's I see it on a screen. And you're even saying, maybe I don't need a keyboard because keyboard's a slow way to give context. I just want to speak it now. Um, okay, so then the question is like that's obviously very promising for what Qualcomm was saying, because they're literally saying we've taken our Snapdragon processor and we've made it a little so like different little form factors so you can have Snapdragon wear and make wearables, or you know, make earbuds or make glasses or watches or whatever.

SPEAKER_00All that's very promising. And of course, you could do it on the phone, you could do it in the car, that kind of thing. Um, so then the next natural question is like, and and and then I posited that maybe someone like Meta who has the correct business model and is incentivized to today both subsidize agentecai in the cloud, but then eventually optimize and bring those costs down and make it more affordable by pushing more and more stuff to the edge. So then the question is, oh man, this could be lots of different hardware, lots of different little interfaces.

SPEAKER_00Yes, lots of it's Snapdragon, but lots of it isn't. So, how do you ideally make sure that stuff can run everywhere? Let's say you're meta. I ideally you don't you're not gonna want to write an Android app and an iPhone app and all these different things everywhere. And so then where my head went is um when I was at Snapdragon Summit, I had the fortune of having dinner uh with Chris Lattner, uh CEO of Modular, and he's famous for creating the Swift programming language, creating VLLM. Um, you know, he worked at Google on their TPUs.

SPEAKER_00I think he had a stint um at Tesla even. And I got to sit across from him and ask him a lot about Modular and his vision. And Mojo is their programming language, and it's like Python, but better. It it's statically typed, um, it has like safe memory access and stuff like that. And he using Mojo, and then they have Max, which is like an LLM serving layer. Um, and they had already they actually started by showing that using Mojo, you could run models on CPUs.

SPEAKER_00It could be an Intel CPU, it could be an AMD CPU. Now they've also showed that um, you know, they can you can use their software and their technology to run models on Nvidia and on AMD and maybe even like Google TPUs, I can't remember, but like on lots of different accelerators. And so naturally, where my head goes is like, oh wow, Qualcomm bot modular, mojo could be a great way for someone like Meta to say, let's use one programming language and one tool chain to the extent possible and write kind of orchestration software to make sure that these models could run anywhere.

SPEAKER_00And so I'll be very interested to see if that actually happens, but it's quite promising for Qualcomm that not only do they have the hardware platform, but they have the right software, which is being open sourced because as a hardware merchant vendor, they don't need to make money, they don't need to like try to keep it as a proprietary thing like CUDA, where it's like, no, no, we want you to get accustomed to our software so that you buy our hardware, but instead they're just saying, hey, our hardware's here, let it compete on its own merits. But by the way, here's this open source software that we have bought and are providing, and you could use it to write it and run it anywhere.

SPEAKER_01So your point is like, I like how you're bringing all these things together. You're bringing, look, Coalcom is always has been at the edge. They've had their moments of trying to go into servers and all that, they may still do it, but their strength really lies at the edge. Then they have Modular, which is a software platform, for lack of a better word, to run literally different pieces of hardware. Uh, and this software platform is the glue that puts all of that together, and you can run like inference based on that.

SPEAKER_01And then you've got somebody like Meta who is incentivized to bring all of that into the edge device. So now you've you see all these things. Kind of align. And uh in my own simplification of what a fundamental AI interface device should look like, it seems like Qualcomm's radios and their modems for their wireless connectivity, their NPUs, um, their CPUs, because like you know, you could this could be your agentic VM, your little iPad-like device.

SPEAKER_01That's all you need. And now you have your microphone and your Bluetooth connections and all of this stuff. All of that ecosystem fits into Qualcomm's product line today, right? And I think that is their vision. And I was reading their blog post about this, and they said this is the ecosystem of you. The idea is that you are in the center. It's not like the device is in the center, no. It's not like the cloud server is in the center of all of this AI. It's you and like what you interact with your outside world and what gets taken care of by the edge device is not your problem.

SPEAKER_01When you ask a question, it is not your problem whether it's handled within the device on the edge or it goes to the cloud. You don't need to know, right? That's where we want to get to eventually with this edge AI stuff. That's the vision I think that's coming out of what Qualcomm is saying. Um I think I'm kind of starting to warm up to the idea that this is interesting. Uh, but the question is like what I'm waiting for, what I want to see next, is a clear definition of how any kind of computing and what kind of computing gives benefit at the edge.

SPEAKER_01I don't think I can answer that clearly. I want to see that happen, right? Yes, yes.

SPEAKER_00Yeah. And so I think you you captured correctly that uh Qualcomm is making the hardware that involves the radios and the CPUs and the NPUs and GPUs available everywhere. And it used to feel a little self-serving because they came as a radio company already at the edge. And so, like in the AI PC world, right, when they're like everything's gonna run the edge, and we're like, dudes, right now only everything runs in the cloud. It feels like you're just saying everything's gonna run the edge because you're a radio company and an SOC company and you know, a smartphone company.

SPEAKER_00You want it to run at the edge. But we've seen so much progression with first reasoning models, which open the door to agentic AI, and finally the consumerification, there's probably a better word, um, of agentic AI with Muse, with Grockbot with Instinct, Muse definitely leading the way, to all of a sudden it's like, whoa, I can actually see a lot of people wanting to run this at the edge. And then the question is, well, why Qualcomm? Why not someone else at the edge? Uh there's lots of other CPU companies, right? And I think um interestingly, Qualcomm's advantage is having the full stack.

SPEAKER_00So having the radio and the compute is is gonna be an important part. Obviously, other people can put it together, um, but it it's pretty advantageous for Qualcomm. Meta is uh, yeah, you're gonna say something. I have a I have a counter to that. What about Huawei? Huawei, there you go. Right, they have a whole stack. Okay, so Huawei has the whole stack, and I'd love to talk to Huawei sometime. I think this is totally exactly what they can do. You know, they will probably be banned from doing this in the United States.

SPEAKER_00Um, so that's probably the only other advantage that American companies have. But yeah, Huawei totally has the whole stack and could do this too. Media tech. Yeah, media tech, yeah. No, no, media tech, no, yeah, exactly. Yeah, media techniques.

SPEAKER_01I guess these names are just coming to me. I'm like, look, I think a bunch of these companies can actually benefit out of this movement because I think a lot of them have this. Think of all the companies that make like phones. All these companies now open to this. Like if they have a CPU-like device and a radio-like device, you're basically set.

SPEAKER_00Exactly. And microphones and screens. Yeah, totally. And this is why also listeners, we try to differentiate from other podcasts because I'm very American egocentric, and Vic is uh, you know, a world traveler, and uh, so he's thinking about all the companies in the rest of the world. So I appreciate that. Um so yes, phone companies will do well. Um, you know who else has a phone and who could potentially play in all of this space? Google, Google Pixel. Google has a phone, Google has a watch, Google's the original Google Glasses, um, they've got the business model.

SPEAKER_00Of course, the question is do they have the Moxie? Clearly, um Zuck and Alex Wang are just going after it and taking advantage of this world that we live in and making Muse happen. Um, it just doesn't, I don't see. Google could, they've got everything. They've got the business model, they've got the hardware from the TPUs in the cloud, so they can, you know, be vertically integrated and have cost efficiencies there when they run things in the cloud, all the way to the edge. They've got all the brilliant people, all the models, they've got everything.

SPEAKER_00They just don't seem to have the product wherewithal and the courage to just get out there and do it. Please tell me I'm wrong. But um, Google, this is your space to play in too. You can make an amazing agent, you can make it run everywhere, you can cost optimize it and bring it all to the edge. But right now, I say meta's in the lead. Yeah, so many things can happen.

SPEAKER_01So many things. Like this is an exciting time because AI, with agents and with these personal assistance that we are seeing, AI has finally come to the user. And we've said this in the podcast before, like the regular person can now use it because everybody uses messaging apps. If you can message somebody and it gives you an answer, which is what a lot of these apps can do, you are now in the masses. Like the everybody has access to AI now. Don't give somebody a chatbot, they don't know what to ask it.

SPEAKER_01Like, literally, I'm not joking. Like, there's like a barrier to this. Like, it's like the Google search bar when it first came out. They're like, okay, what do I type in it? Whatever you want. I don't want anything. Like, so that's not the way to do it, right? And that was a problem even in the Google era when it came out. But like, chatbot is the initial. Like, like this, this whole like open cloud was like, is totally for the hacker types, okay? Like, who the hell knows how to spin up like a VM and you know install, give API keys and do some command line stuff.

SPEAKER_01Nobody knows that. Now we are in the consumer layer, and what's going to happen is very exciting.

SPEAKER_00Indeed. Indeed. And it will have all sorts of implications for the AI infrastructure trade, which I know lots of you listeners are here for. Um, but we are at the end of today's episode, so that's your homework is to go think about this. And then, you know, write comments on YouTube, send us emails, follow us, uh, daily.semidope.com. We have a daily newsletter, and uh go check out our sponsors. So thank you for listening, guys.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Google Built Two New TPUs Without New Silicon: The 8i Max and 8TX

    Nov 17, 202654 minAudio

    Austin Lyons and Vik Sekar break down Google's two new TPU systems, the 8i Max and 8TX. These are not new silicon but new system configurations of the existing V8 chip, designed for specific workloads. The conversation...

    Credit the people on this episode

  2. Jensen Huang Resigns

    Nov 12, 202632 minAudio

    Jensen Huang resigned as Nvidia CEO this morning, and Austin and Vik recorded two hours after the news broke. They argue that a board must prioritize corporate integrity over short-term shareholder value, even for a...

    Credit the people on this episode

  3. Intel's Earnings Disaster: Bad Luck, Bad Planning, or Both?

    Nov 9, 202657 minAudio

    Intel just had its worst day on the stock market in 55 years, with shares falling nearly 30%. But here's the strange part: Intel actually reported a strong quarter. Revenue grew, its data center business surged, and the...

    Credit the people on this episode

  4. 3D Stacked DRAM for Inference: d-Matrix's Sudeep Bhoja

    Nov 5, 20261 hr 16 minAudio

    The fundamental physics of data movement are forcing a shift to new memory architectures for AI inference. Sudeep Bhoja, co-founder and CTO of d-Matrix, joins Vik to detail the company's journey from a non-consensus bet...

    Credit the people on this episode

  5. Why Marvell Wants Lumentum: Vertical Integration in AI Optics With Cignal AI

    Oct 30, 202640 minAudio

    Marvell's potential acquisition of Lumentum is a strategic vertical integration play to capture the massive value optical components now unlock in the AI era. Andrew Schmitt and Scott Wilkinson of market research firm...

    Credit the people on this episode