Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Semi Doped

Semi Doped artwork

Qualcomm's Durga Malladi: HBC vs HBM, Dragonfly AI 250, and Winning on TCO

Sep 24, 202621 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Qualcomm re-entered the data center inference market late - competitive GPUs, XPUs, and custom accelerators are already out there. Durga Malladi (EVP/GM, Technology Planning, Edge & Data Center) explains why they didn't build another me-too accelerator and instead placed a differentiating technical bet: High Bandwidth Compute (HBC). The difference between HBC and HBM: HBM is stacked DRAM that shuttles data across wide buses to a separate accelerator - very high bandwidth, but expensive and power-hungry. HBC bonds stacked DRAM directly on top of a custom logic die using wafer-on-wafer techniques, so a lot of the compute runs literally next to memory. The result is lower latency and lower power per bit - and 18x effective bandwidth on the AI 250 versus the AI 200 at the SAME 768 GB LPDDR capacity and the SAME 160 kW per rack. Why the late-mover bet works: Qualcomm's differentiation isn't just architectural. Their foundry and memory-vendor relationships let them credibly promise the multi-megawatt-to-gigawatt supply hyperscalers need. Memory vendors are developing their own HBC-equivalent variants as partners rather than pure competitors, and Qualcomm expects the two technologies to co-evolve. Silicon is back in the lab; proof points land in Q1/Q2 2027. Chapters: 0:00 Introduction 2:49 The Memory Wall Problem 8:05 HBC Physical Architecture 9:33 HBC vs. Custom HBM 11:50 Silicon Proof Points Coming 12:08 Managing Thermals 13:34 Dragonfly AI 250 14:48 Trillion-Parameter Model on One Card 16:56 Tokenomics and TCO 18:26 Why TCO is Key 19:26 Manufacturing at Scale Follow Chipstrat: Newsletter: https://www.chipstrat.com X: https://x.com/chipstrat Follow Vik: Newsletter: https://www.viksnewsletter.com/ X: https://x.com/vikramskr Follow Semi Doped: Get more of Austin and Vik daily, free: https://daily.semidoped.com/

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_00Aloha listeners. Austin Lyons here with Chipstrat. And with me is Durga Malati, IEEE Fellow, which is pretty cool, and EVP and GM of tech planning for Edge and Data Center. So we're here to talk high bandwidth compute. But first, before we get into that, which is a great topic, I want to know more about like, tell me what your job title means. Tell me what you do at Qualcomm.

SPEAKER_01So I run all the IP and the technology planning for what we need to do across all the products, ranging from data center all the way down to Ed Solutions, including devices. So that means the IP, if you kind of think about it, is what should we be doing in our CPU portfolio in, let's say, three, four years downstream, in 29, in 30, in data center, versus what do we need to do in devices, in handsets, and in new agent tech devices.

SPEAKER_01And then rinse and repeat the same exercise for what do we need on GPU? What about NPU? What about modem? What about sensing hub and so on? So these are all the different IPs. So that's the planning exercise, which kind of runs the roadmap behind the scenes, but it is across all the business units.

SPEAKER_00Nice. That's pretty cool. I like that you see across all the different technologies so that it's not like CPU running, not together with GPU or not together with the different products. Yeah, that's pretty cool. Okay, so let's talk high bandwidth compute HBC. So the way I see this is obviously Qualcomm is trying to enter and is in a competitive space with inference solutions in the data center. You've got GPUs, there's XPUs, and Qualcomm has the Dragonfly lineup, AI 200, 250, 300.

SPEAKER_00It feels to me like HBC is Qualcomm's big technical bet to differentiate and bring something different to market as compared to what everyone else is doing. So, you know, of course, I want your thoughts on that, but also then tell us what is this HBC and how is it different from what others are doing?

SPEAKER_01Absolutely, yeah. So when we re-entered the business, uh data center business last year, uh, and uh I actually ran it for one year, like last year. One of the things that we really focused on is you're absolutely right. Uh there's a lot of players already in the market, and they have some pretty incredible solutions already. So it wasn't one of those moments where we said, okay, let's also jump into the fray, and it's it's not like a Me Too moment or anything of that sort, but it was like, what is it that is going to be differentiating us from uh uh the rest of the ecosystem there?

SPEAKER_01And what is the really uh what is a really good tough problem that we really want to solve because that actually gives us the benefit of okay, that is different, let's take a look at Qualcomm. And we latched on to the fact that um and and we had an RD project that was running internally for a while on what is called as near memory computing. And the principles of that are very simple, but before that, what is the problem that we are trying to solve? And it turns out that if you kind of go back in time, maybe over the last six years or so, you will see that uh the compute capability, like if you just take any of the GPUs and say generation over generation, how much is it improving?

SPEAKER_01It improves quite a lot, actually. It's like uh at least a factor of two, sometimes a factor of three. But the memory bandwidth wasn't actually improving at the same league. And so the natural next question to ask is so what? What does that do? Well, you know, if you kind of go back to some of the basic principles of how do you do compute, well, you have all your data sitting somewhere and you have to fetch that data to do the compute processing, but at some point in time, what happens is that your compute is so good that you've uh kind of tilted the scales quite a bit.

SPEAKER_01So you might be just idling, you're waiting for the data to actually show up. So memory bandwidth becomes a bottleneck, and it's called the memory wall. And you see that a lot more with AI inference, especially in the decode portion of it. And so what we did is our HBC architecture is based upon what is called as near memory compute. Instead of treating memory and compute as two distinct concepts, what if we actually do a lot of the compute right next to memory, like literally right next to it? It does two things.

SPEAKER_01One, the latency comes down quite a bit. And second, uh, we can uh reduce the power consumption quite a lot. So today in in data centers, HBM, you know, it's the the bandwidth is very high, but uh it's expensive and it's a power hog. So uh we actually wanted to make sure that we can solve it in a different way. And one thing led to another, and as we started talking about this with uh a lot of the hyperscalers and a lot of other partners, they loved the idea.

SPEAKER_01And uh uh as we announced it, we actually announced uh uh a couple of names in terms of some of the partners that we have. So it's been a great journey on HPC. But uh as we briefly mentioned in our investor day in June and today at Snapdragon Summit, in fact, Cristiano left a teaser out there. Uh but let me just tell you uh, you know, what is the point of this? HPC for us is a technology, but it's not just for data center.

SPEAKER_01It actually applies to everything, all the way from data center to servers and accelerator cards and automotive in PCs and XR and yes in smartphones as well. Uh but it actually works in a different way in some of these things. But the principle remains the same, which is you're doing a lot of computation right next to memory.

SPEAKER_00That's good, that's a great explanation. And and I like how you mentioned that HPC is something that can be applied across from the data center all the way to the edge, and that's a good organizational design, as you're talking earlier about how your job is to see IP across all that. So that's pretty cool. Okay, so what I heard from you was the problem is compute is increasing quickly, memory bandwidth is not increasing fast enough, so this like all this extra compute can't get fed quickly enough. And maybe instead of trying to just increase the bandwidth of the memory we use today, you guys stopped and said, Well, wait a minute, why have to shuttle all this data?

SPEAKER_00Why why shuttle even more like wider buses or even faster speeds? Why not just bring it really close to the compute? That's correct, exactly.

SPEAKER_01And uh keep in mind that it's uh uh it's a principle that you know as you start bringing down the power consumption quite a lot, uh just uh going one step further on devices, power consumption matters quite a lot in devices, especially as you think of all the agentic AI-based devices that you see these days. And they're still evolving, by the way. There's a lot of new players who are coming to the market. In fact, quite a few of them have very little background in manufacturing devices or making devices. They come in from an AI-first mentality and they're coming out with these devices.

SPEAKER_01And the job of agentic AI devices is to be kind of being constantly on because they're working all the time. So that power consumption benefit that we talked about is going to be key, uh, we believe, uh, in unlocking the full potential of uh agentic AI devices.

SPEAKER_00Yeah, that makes sense. So it's not just the memory bandwidth problem, but the also added benefit of because you're not shuttling this data so far horizontally, now it's just going vertically, which we'll get into. There's also big uh power savings. Yes. And of course we know that power is the big constraint. Um, so then you know, to someone who's listening, this is the first time they've heard of high bandwidth compute. Are you sticking memory on top of an accelerator? Or tell us more about like the stacking and the separation. What is being sent kind of across the wire horizontally versus vertically?

SPEAKER_01Yeah. Uh by the way, there's a really cool animation that we showed at Invest today. I wish we had that, but I'll just use my hands as props. Okay. So the way it works is that uh the memory is like uh, you know, you have these individual, you know, the DRAM, uh it's called as a stack DRAM. You're basically stacking DRAM one on top of the other. And there's a lot of wafer-on-wafer bonding techniques that are used on that. So this is this is a hardcore process technology, by the way. This is, in fact, it has nothing to do with compute at this point in time. But then right underneath that stack DRAM is what is called as the logic die.

SPEAKER_01That's where you put in all the compute. And depending on the use case, you might actually say, I want some specific logic to actually decide over there. There might be yet another processor that is still sitting, which is away from it, but you've brought in a lot of the heavy computational needs, like the ones which really need that memory bandwidth. You put all of that right next to memory in that base die. And that HBC, therefore, is a combination of both: the stack DRAM on top plus the logic die that is underneath, or the compute die that is underneath. And now you've kind of done a bulk of the work just with that alone.

SPEAKER_01There might still be some residual compute that you want to do, and that could occur in a separate processor over here. In some sense, HPC kind of is like it's a bit of like a coprocessor because it's doing a lot of the compute, but it's not the only thing. You might still have another main processor that's running out there, but it removes that complexity quite a bit. So uh in short, it is literally right underneath that stacked DRAM that we're talking of. That's where all the computers being.

SPEAKER_00Okay, okay, that that's helpful. So anyone who's familiar with HBM will say, well, this sounds a lot like HBM 4E with a custom-based dye and stacked DRAM on top. So tell us the differences. First of all, is it a different class of DRAM that's being stacked? And and then how should we think about the difference between the accelerator at the bottom versus a custom die?

SPEAKER_01Yeah, that's a in fact, this is exactly one of the things uh when we first came out with the concept and when when we talked about it, uh you know, I went to Korea and we and some of the other memory vendors in the US and in other parts of the world. And to be honest, I half expected people to say, now wait a minute, now we're doing exactly the same thing. I don't know what you guys are talking about. Uh, you know, we have a pretty robust roadmap coming in. Actually, what I heard is the diametric opposite of that, which is how can we work with you closer on this?

SPEAKER_01Because yes, this is indeed the right way of actually doing it. Because one thing that happens is there might still be ways of, you know, the basic principles and the concepts look similar. But that wafer on wafer bonding that we are talking of, and we still use a lot of LPTR-based uh uh uh interfaces towards the main processor which is sitting out there, which means we still keep the power consumption extremely low. Uh, there are evolving things that are coming up in HPM as we move forwards, but at this point in time we see that if you're a memory vendor, uh even if you are working on HPM and you're looking at customizing HPM a little bit further, going towards 4E and beyond, I think Stack DRAM is here to stay.

SPEAKER_01And you might have noticed a couple of announcements that have been made even by the memory vendors. They have different terminology for that. It is their equivalent of HPC, but actually they are partners of ours. So uh we expect that uh these two technologies will co-evolve for a period of time, and we'll see what happens after that.

SPEAKER_00Interesting, interesting. Okay, so then um which by the way, have you guys disclosed anything about the layers, uh, number of layers that you're stacking?

SPEAKER_01Uh we have not yet provided all the numbers. In fact, stay tuned on that front because uh, you know, you haven't asked the question, how's the silicon coming along and so on? Because we've talked a lot about it in terms of uh PowerPoint presentations and and uh you know, most people are actually taking our numbers for what they are worth. But we will be talking, we will be showing some proof points over the next quarter or two as the silicon is back in the lab and it's coming along very well. So stay tuned.

SPEAKER_00Okay, awesome. I'd love to hear that. So you'll have silicon coming back soon. Looking forward to that, looking forward to see the benchmarks. That really turns it real you know, from just PowerPoint. Um, back to the the stacking of the uh memory, talk just about the thermals. Is is does that become a constraint for that base dye?

SPEAKER_01I think it is uh one of those things that uh ask Qualcomm, when we come in with uh uh, you know, we are not when we talk about IP, for instance, it's not just about what we are doing uh uh uh for our CPU or our NPU logic and so on. We have a rich heritage of uh both uh packaging technologies and and 3D stacking, and a lot of these things that we do, we work very closely with the foundries in terms of like how these techniques need to come together. That has played in our favor as we brought out uh HPC.

SPEAKER_01And uh one thing to mention is that uh in this space, uh you know you have to think through carefully as to which product are you designing this for, and based upon that, you have to think about the right kind of packaging so that you don't start hitting other constraints, like it's not the power maybe, but then you start hitting thermal constraints. So we do have a lot of experience in that space. So different, depending upon which product it is, we have dimensioned the compute in such a way, compute plus the memory, which is dimensioned in such a way that we avoid the thermal problems.

SPEAKER_01And then so that is something that we've paid some careful attention to.

SPEAKER_00Nice, nice, very promising. Okay, so the Dragonfly AI 250 is the first product that will have HPC in it. That's good. And that's coming in 2027. That's correct. Um I saw that compared to the AI 200, they both have the same amount of LPDDR in the card. It's like 768 gigs. 768 gigs. Um and yet the 250, which is the first to have the HPC, has 18x the effective bandwidth, I believe is what the what the website said. So again, just for everyone, can you uh remind us again why that uh HBC in the stacking is 18x the bandwidth as compared to in the 200, where presumably the LPTDR is a little bit further away?

SPEAKER_01So, first of all, the raw memory is a completely different concept compared to the actual memory bandwidth in itself. So this is LPDDR, which means we can we can actually pack, it's a very dense card that we have, so it's 768 gigabytes. You know, a lot of the other things that we see in the industry, they have like 288 gigabytes per card. This is like 768 gigabytes. Which, by the way, if you kind of think through that, 768 gigabytes per card means if you take a typical order of magnitude, trillion parameter model, and it's all like at mixed precision FP4, you can fit a trillion parameter model in just one card.

SPEAKER_01That is like seriously mind-boggling. And we have shown that. We have shown some proof points associated with that. But that's just the memory capacity. The memory bandwidth is where as we bring in HPC, then you have this combination of the underlying logic plus uh plus DRAM stack. So this actually is a slightly different concept. This is not about the capacity anymore. Using the same capacity, you've just boosted the memory bandwidth significantly. That's why they are two independent concepts. But another way of looking at it is if you take an AI 200 card and it has a certain performance, certain uh uh exaflops per rack or uh exaflops uh uh in a given rack, and of course, we have disclosed that you know our racks are like 160 kilowatts.

SPEAKER_01When you go from AI 200 to AI 250, the amount of memory inside it's like 41 or 43 terabytes or so, uh that doesn't change. But because of the fact that there's a huge memory bandwidth increase, the number of exaflops in the card uh in the in the rack actually goes up substantially because of that, because the decode performance improves quite a lot. So you see it's the the the total number of like the amount of capacity that you have in the rack is the same. But now you've dismantled that memory wall with HPC, and that's the reason why our overall performance goes up significantly with AI 250.

SPEAKER_00Okay, so let me reflect it back to you, and there's a part where I'm a little fuzzy on. So we've got more bandwidth in the AI 250, which of course would mean better decode performance. So just like more tokens per second. That's correct. How does that that unlocks more flops because the flops are not underutilized? They're not like sitting waiting. Is that what it is?

SPEAKER_01That's right. In fact, because of the fact that you're uh running it that way, your TCO also becomes better in the sense that uh if you just take in because the tokens per second per watt goes up significantly, because your tokens per second has gone up, it's the same power consumption, by the way, same 160 watt uh uh kilowatts. Uh but then in addition to that, then when you take a look at the total, it's called the tokenomics, where you also normalize it by what's the cost associated with that, it comes out to be even more favorable compared to some of the other things that you see in the market today.

SPEAKER_01So, in that sense, every time we have talked about the benefits of AI 250, we have talked about not just tokens per second, but tokens per second, tokens per second per watt, and tokens per second per watt per dollar, or sometimes even tokens per second per dollar. When you put the entire arithmetic together, then you see why it becomes actually far more favorable to go with this.

SPEAKER_00Nice. Okay, so I was gonna ask you then. So when you have talked to hyperscalar partners, ultimately, is there one of those that's most compelling? Is it is it about the TCO or about the tokens per watt or kind of all of it?

SPEAKER_01I think increasingly these days, it's about the TCO.

SPEAKER_00Increasingly.

SPEAKER_01It used to be about raw tokens per second and tokens per second were per watt.

SPEAKER_00Yeah.

SPEAKER_01But I would argue that TCO already kind of captures that. Because if you kind of you know think about it, what are all the things that you need to worry about? Well, okay, if I buy one of these racks, first of all, how much am I paying for it? So that's like one variable. Second, what's the power consumption? And then you have to figure out for that power consumption what's my utility rate, and there is some number associated with that. The third one, of course, is uh how many tokens per second does it really generate? Like how effective is it? And you put all of these three variables together, and I'm discounting things like water that you need uh uh into the uh for cooling and so on.

SPEAKER_01Uh it's the TCO that uh is something that is emerging as uh the most important, but it does take into account like these three big variables.

SPEAKER_00Yeah, yeah. Yeah, no, I think that makes a lot of sense. And just like the simplest way for me to think about it is like, okay, if I have a billion dollars to spend and I have like a gig, not even a gigawatt, a hundred megawatts, if I can buy X racks from some provider that fit within the power envelope and my budget, or I can buy two X racks from someone else, well, even if the throughput's a little bit different, you know, generally if you have two X of the racks, you're gonna get more throughput. Um so that makes a lot of sense to me. So I guess um, let's see, maybe last question.

SPEAKER_00When so there are uh other AI ASIC startups who also, because they're coming in and needing to do something different, they're also making different technical bets. And usually like the skepticism that people get with them is they say, okay, that's great. Um can you actually manufacture it? Can you actually get allocation? Now, obviously, Qualcomm has is one of TSMC's largest customers, and you have wafer allocation, but I also know that you have many different business units from data center to edge who all want the same wafers. So, like talk us through like how does that work?

SPEAKER_00And I I assume that you know you do have the advantage there because you are manufacturing things in production at scale.

SPEAKER_01So uh, you know, that's one of the places where it is true that it's not enough to just have the just the technology in itself. That's why we've been placing careful bits in terms of making sure that uh you know our wafer allocations, we have like an excellent set of relationships with all the foundries and also with the memory makers. And so, as they have come up as extremely valuable partners in across the entire portfolio, not just in one domain, uh those relationships do matter. When we go in front of a customer and say, uh, yeah, we are in a position to supply you.

SPEAKER_01How much do you want? You want, okay, uh, these many megawatts or maybe a gigawatt, uh, this is what you want. Yeah, we have the supply chain lined up for that because we do have those relationships across all the different businesses. And uh that actually helps us quite a lot.

SPEAKER_00Yeah, sure, totally. That makes a lot of sense. All right, well, that's it for today. All right. Um, Durga, this was great. We will have to talk again once you guys have more to share about the actual measured performance of HPC. All right, sounds good. Thank you. Thanks, thanks again.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Google Built Two New TPUs Without New Silicon: The 8i Max and 8TX

    Nov 17, 202654 minAudio

    Austin Lyons and Vik Sekar break down Google's two new TPU systems, the 8i Max and 8TX. These are not new silicon but new system configurations of the existing V8 chip, designed for specific workloads. The conversation...

    Credit the people on this episode

  2. Jensen Huang Resigns

    Nov 12, 202632 minAudio

    Jensen Huang resigned as Nvidia CEO this morning, and Austin and Vik recorded two hours after the news broke. They argue that a board must prioritize corporate integrity over short-term shareholder value, even for a...

    Credit the people on this episode

  3. Intel's Earnings Disaster: Bad Luck, Bad Planning, or Both?

    Nov 9, 202657 minAudio

    Intel just had its worst day on the stock market in 55 years, with shares falling nearly 30%. But here's the strange part: Intel actually reported a strong quarter. Revenue grew, its data center business surged, and the...

    Credit the people on this episode

  4. 3D Stacked DRAM for Inference: d-Matrix's Sudeep Bhoja

    Nov 5, 20261 hr 16 minAudio

    The fundamental physics of data movement are forcing a shift to new memory architectures for AI inference. Sudeep Bhoja, co-founder and CTO of d-Matrix, joins Vik to detail the company's journey from a non-consensus bet...

    Credit the people on this episode

  5. Why Marvell Wants Lumentum: Vertical Integration in AI Optics With Cignal AI

    Oct 30, 202640 minAudio

    Marvell's potential acquisition of Lumentum is a strategic vertical integration play to capture the massive value optical components now unlock in the AI era. Andrew Schmitt and Scott Wilkinson of market research firm...

    Credit the people on this episode